spancer opened a new issue, #10333: URL: https://github.com/apache/paimon/issues/10333
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version 2.0.0 (paimon-bundle-2.0.0) ### Compute Engine None — reproduced with the Java API, local filesystem catalog, Java 21. Default `metadata.stats-mode` (`truncate(16)`). ### Minimal reproduce step Write a string longer than 16 characters whose 16th character is U+D7FF, then inspect the file max statistic and query for that value: ```java Identifier id = Identifier.create("db", "t_trunc"); catalog.createTable( id, Schema.newBuilder().column("id", DataTypes.INT()).column("c", DataTypes.STRING()).build(), false); Table table = catalog.getTable(id); String longValue = "0123456789abcdezz"; write( table, List.of( GenericRow.of(1, BinaryString.fromString(longValue)), GenericRow.of(2, BinaryString.fromString("0")))); for (Split split : table.newReadBuilder().newScan().plan().splits()) { for (DataFileMeta file : ((DataSplit) split).dataFiles()) { System.out.println(file.valueStats().maxValues().getString(1)); // 0123456789abcde? } } PredicateBuilder builder = new PredicateBuilder(table.rowType()); System.out.println(count(table, builder.equal(1, BinaryString.fromString(longValue)))); // 0 System.out.println(count(table, builder.greaterOrEqual(1, BinaryString.fromString(longValue)))); // 0 System.out.println(count(table, null)); // 2 ``` Helper methods: ```java static void write(Table table, List<GenericRow> rows) throws Exception { BatchWriteBuilder builder = table.newBatchWriteBuilder(); try (BatchTableWrite write = builder.newWrite(); BatchTableCommit commit = builder.newCommit()) { for (GenericRow row : rows) write.write(row); commit.commit(write.prepareCommit()); } } static long count(Table table, Predicate filter) throws Exception { ReadBuilder builder = table.newReadBuilder(); if (filter != null) builder = builder.withFilter(filter); long[] count = new long[1]; try (RecordReader<InternalRow> reader = builder.newRead().createReader(builder.newScan().plan())) { reader.forEachRemaining(row -> count[0]++); } return count[0]; } ``` ### What doesn't meet your expectations? The max statistic for `c` is recorded as `"0123456789abcde?"`, which is **smaller** than the real maximum `"0123456789abcdezz"` (`?` is U+003F). The statistic is therefore not an upper bound, and the file is skipped for predicates that the row satisfies: `c = <that value>` and `c >= <that value>` both return **0** rows instead of 1. ### Anything else? Root cause is in `org.apache.paimon.statistics.TruncateSimpleColStatsCollector#truncateMax`. After truncating to N characters it increments the last code point and accepts the result when: ```java nextCodePoint != 0 && Character.isValidCodePoint(nextCodePoint) ``` U+D7FF + 1 is U+D800, a surrogate. `Character.isValidCodePoint` is true for surrogates (it only checks `[0, 0x10FFFF]`), so the lone surrogate is appended, and `BinaryString.fromString` encodes it to UTF-8 as `?`. U+D7FF is the only code point affected: U+FFFF + 1 = U+10000, while U+10FFFF is not incrementable and the previous character is incremented instead. Suggested fix: do not produce a surrogate. Either skip the surrogate block (use U+E000 when the incremented value falls in `[0xD800, 0xDFFF]`), or treat that position as not incrementable and move to the previous character, as is already done for U+10FFFF. The Avro writer path has a second effect of the same bug. With rows in this order: ```text 0123456789abcdezz 0123456789abcdeA ``` the broken truncated max (`...e?`) is replaced by the later value, producing `0123456789abcdeA`, which is still below the real max. Equality on the first value again returns 0 rows. ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
