JingsongLi opened a new pull request, #11142: URL: https://github.com/apache/paimon/pull/11142
### Purpose A Data Evolution rewrite can keep picking up concurrent writes after its scan has already produced the last batch, because each successful commit triggers planning against the committed snapshot. Return planned tasks together with a scan-finished flag and stop after committing the final batch. Determine completion from iterator exhaustion rather than the batch size, since an underfilled batch can still have more complete row-ID ranges to process. Apply this to both compaction and deletion-vector materialization, retaining commit conflict retries and empty-batch handling. ### Tests - Passed 65 tests in `DataEvolutionCompactCoordinatorTest`, `DataEvolutionCompactRangePlannerTest`, and `DataEvolutionDeletionVectorMaterializeCoordinatorTest`. - Passed 19 Data Evolution tests from Spark 3.5's `CompactProcedureTest`, including concurrent appends at the final batch, batch-size boundaries, underfilled nonfinal batches, empty final batches, conflict retries, and snapshot expiration. - Maven compilation, Checkstyle, Spotless, and Enforcer passed without `fast-build`, using JDK 8 and Scala 2.12. - An isolated mutation that ignores the completion flag made both new termination regression tests fail as expected. The full Spark suite was blocked by an unrelated `NoSuchMethodError` in Spark's Parquet footer reader (`ParquetFileReader.open(InputFile, ParquetReadOptions)`). The 19 related tests were run directly with ScalaTest using the compiled reactor classpath. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
