zhangyue19921010 opened a new issue, #11125:
URL: https://github.com/apache/paimon/issues/11125

   ### Search before asking
   
   - [x] I searched existing issues and PRs. #8140 and #8312 fix related 
metadata/NPE cases; the legacy empty-key-plus-NULL case below still loses rows.
   
   ### Paimon version
   
   Current master, `31f3db767` (2.2-SNAPSHOT), reading BTree metadata written 
before #8140.
   
   ### Compute Engine
   
   Java table API, JDK 11. No Spark or Flink job was run.
   
   ### Minimal reproduce step
   
   1. Create a Data Evolution append table with `row-tracking.enabled=true`, 
`data-evolution.enabled=true`, no primary key and `bucket=-1`. Write `(id, 
value)` rows `(0, '')`, `(1, NULL)`, `(2, '')`.
   2. Build a STRING BTree index using the legacy metadata format, then read it 
with current master.
   3. Filter with `value = ''`. The expected IDs are `0, 2`, but the result is 
empty. If a later indexed range contains `(3, 'a')`, ascending TopN with NULLS 
LAST can retain only ID `3` instead of an empty-string row.
   
   The executable regression uses the current V1 BTree writer and replaces its 
manifest metadata with the original encoding, omitting newer source metadata. 
It does not run the full historical writer. For both an all-NULL file and an 
empty-string-plus-NULL file, the old serializer writes the same nine bytes:
   
   ```java
   MemorySliceOutput out = new MemorySliceOutput(9);
   out.writeInt(0); // first serialized key length
   out.writeInt(0); // last serialized key length
   out.writeByte(1); // contains NULLs
   ```
   
   ### What doesn't meet your expectations?
   
   Upgraded reads should retain matching empty strings. 
`SortedIndexFileMeta.deserialize` currently interprets these ambiguous bytes as 
proof of all-NULL data, so predicate selection and TopN can discard matching 
rows. Residual filtering cannot restore discarded row IDs.
   
   ### Anything else?
   
   Preserve the ambiguity during metadata pruning, recover the actual 
empty/nonexistent key from the BTree file, and retain all TopN candidate files 
when metadata alone cannot rank them safely. Modern metadata and true all-NULL 
files remain covered by regressions. Legacy ambiguous files can require 
additional index reads.
   
   AI-generated with OpenAI Codex.
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to