LuciferYang opened a new pull request, #11030:
URL: https://github.com/apache/paimon/pull/11030

   ### Purpose
   
   close #11029
   
   PyPaimon gave every file of a bucketed append table the bucket's restored 
max sequence number as its whole range. Java's compaction picks a bucket's 
files in sequence order, so once a Java writer compacts the bucket the rows 
lose their append order. The issue has the repro.
   
   Files of bucketed append tables now get the range of their own rows, 
starting at the bucket's max sequence number + 1, like Java's 
`AppendOnlyWriter`. Files of unaware-bucket tables keep the zero range; on 
row-tracking tables, which require `bucket = -1`, commit replaces it with the 
snapshot id. Primary-key tables are unchanged.
   
   ### Tests
   
   - `append_sequence_test.py`: three commits to a bucketed table get the 
ranges (2, 2), (3, 4) and (5, 5); on master all three are (1, 1). On 
unaware-bucket row-tracking and data-evolution tables the files keep (0, 0) and 
`_SEQUENCE_NUMBER` reads the snapshot id; this one also passes on master.
   - `test_py_write_bucketed_append_sequence` with 
`JavaPyE2ETest#testReadPythonBucketedAppendOrder`, added to 
`run_mixed_tests.sh`: PyPaimon commits six rows to one bucket, a Java write 
compacts the bucket, and Java reads the rows in append order. On master it 
reads `[2, 0, 1, 5, 4, 3, 6]`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to