[
https://issues.apache.org/jira/browse/ORC-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16572426#comment-16572426
]
ASF GitHub Bot commented on ORC-203:
------------------------------------
Github user omalley commented on a diff in the pull request:
https://github.com/apache/orc/pull/299#discussion_r208398373
--- Diff: java/core/src/java/org/apache/orc/impl/ColumnStatisticsImpl.java
---
@@ -683,6 +787,150 @@ public int hashCode() {
result = 31 * result + (int) (sum ^ (sum >>> 32));
return result;
}
+
+ /**
+ * A helper function that truncates the {@link Text} input
+ * based on {@link #MAX_BYTES_RECORDED} and increments
+ * the last codepoint by 1.
+ * @param text
+ * @return truncated Text value
+ */
+ private static Text truncateUpperBound(final Text text) {
+
+ if(text.getBytes().length > MAX_BYTES_RECORDED) {
+ return truncateUpperBound(text.getBytes());
+ } else {
+ return text;
+ }
+
+ }
+
+ /**
+ * A helper function that truncates the {@link byte[]} input
+ * based on {@link #MAX_BYTES_RECORDED} and increments
+ * the last codepoint by 1.
+ * @param text
+ * @return truncated Text value
+ */
+ private static Text truncateUpperBound(final byte[] text) {
--- End diff --
You need the length passed in here.
> Modify the StringStatistics to trim minimum and maximum values
> --------------------------------------------------------------
>
> Key: ORC-203
> URL: https://issues.apache.org/jira/browse/ORC-203
> Project: ORC
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Sandeep More
> Priority: Major
>
> Currently the StringStatistics will record the entire value for minimum or
> maximum. It creates large protobuf objects and serves very little value. I
> think we should trim long strings to 1024 characters and record the fact that
> they were trimmed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
