Github user omalley commented on a diff in the pull request:
https://github.com/apache/orc/pull/292#discussion_r205181230
--- Diff: java/core/src/java/org/apache/orc/impl/ColumnStatisticsImpl.java
---
@@ -543,35 +549,87 @@ public void reset() {
super.reset();
minimum = null;
maximum = null;
+ isLowerBoundSet = false;
+ isUpperBoundSet = false;
sum = 0;
}
@Override
public void updateString(Text value) {
if (minimum == null) {
- maximum = minimum = new Text(value);
+ if(value.toString().length() > MAX_STRING_LENGTH_RECORDED) {
+ minimum = new Text(
--- End diff --
We can even use the structure of UTF-8 to do truncateLowerBound without
forming a string. In particular you want to search from 1024 to find the first
byte of a character (0xxx xxxx or 11xx xxxx) and truncate just before it.
---
