JingsongLi opened a new pull request, #11142:
URL: https://github.com/apache/paimon/pull/11142

   ### Purpose
   
   A Data Evolution rewrite can keep picking up concurrent writes after its 
scan has already produced the last batch, because each successful commit 
triggers planning against the committed snapshot.
   
   Return planned tasks together with a scan-finished flag and stop after 
committing the final batch. Determine completion from iterator exhaustion 
rather than the batch size, since an underfilled batch can still have more 
complete row-ID ranges to process. Apply this to both compaction and 
deletion-vector materialization, retaining commit conflict retries and 
empty-batch handling.
   
   ### Tests
   
   - Passed 65 tests in `DataEvolutionCompactCoordinatorTest`, 
`DataEvolutionCompactRangePlannerTest`, and 
`DataEvolutionDeletionVectorMaterializeCoordinatorTest`.
   - Passed 19 Data Evolution tests from Spark 3.5's `CompactProcedureTest`, 
including concurrent appends at the final batch, batch-size boundaries, 
underfilled nonfinal batches, empty final batches, conflict retries, and 
snapshot expiration.
   - Maven compilation, Checkstyle, Spotless, and Enforcer passed without 
`fast-build`, using JDK 8 and Scala 2.12.
   - An isolated mutation that ignores the completion flag made both new 
termination regression tests fail as expected.
   
   The full Spark suite was blocked by an unrelated `NoSuchMethodError` in 
Spark's Parquet footer reader (`ParquetFileReader.open(InputFile, 
ParquetReadOptions)`). The 19 related tests were run directly with ScalaTest 
using the compiled reactor classpath.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to