zhangyue19921010 opened a new issue, #11125: URL: https://github.com/apache/paimon/issues/11125
### Search before asking - [x] I searched existing issues and PRs. #8140 and #8312 fix related metadata/NPE cases; the legacy empty-key-plus-NULL case below still loses rows. ### Paimon version Current master, `31f3db767` (2.2-SNAPSHOT), reading BTree metadata written before #8140. ### Compute Engine Java table API, JDK 11. No Spark or Flink job was run. ### Minimal reproduce step 1. Create a Data Evolution append table with `row-tracking.enabled=true`, `data-evolution.enabled=true`, no primary key and `bucket=-1`. Write `(id, value)` rows `(0, '')`, `(1, NULL)`, `(2, '')`. 2. Build a STRING BTree index using the legacy metadata format, then read it with current master. 3. Filter with `value = ''`. The expected IDs are `0, 2`, but the result is empty. If a later indexed range contains `(3, 'a')`, ascending TopN with NULLS LAST can retain only ID `3` instead of an empty-string row. The executable regression uses the current V1 BTree writer and replaces its manifest metadata with the original encoding, omitting newer source metadata. It does not run the full historical writer. For both an all-NULL file and an empty-string-plus-NULL file, the old serializer writes the same nine bytes: ```java MemorySliceOutput out = new MemorySliceOutput(9); out.writeInt(0); // first serialized key length out.writeInt(0); // last serialized key length out.writeByte(1); // contains NULLs ``` ### What doesn't meet your expectations? Upgraded reads should retain matching empty strings. `SortedIndexFileMeta.deserialize` currently interprets these ambiguous bytes as proof of all-NULL data, so predicate selection and TopN can discard matching rows. Residual filtering cannot restore discarded row IDs. ### Anything else? Preserve the ambiguity during metadata pruning, recover the actual empty/nonexistent key from the BTree file, and retain all TopN candidate files when metadata alone cannot rank them safely. Modern metadata and true all-NULL files remain covered by regressions. Legacy ambiguous files can require additional index reads. AI-generated with OpenAI Codex. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
