Zouxxyy opened a new pull request, #8802: URL: https://github.com/apache/paimon/pull/8802
### Purpose Postpone bucket tables buffer batch-written records in the postpone bucket, so these records are not visible to ordinary queries until postpone compaction finishes. This change introduces opt-in merge-on-read support through `spark.paimon.postpone.merge-on-read=true`. * Add an engine-neutral Core plan and read API for planning real-bucket splits and physical postpone-file tasks from one snapshot. * Preserve postpone-file replay order with deterministic sequence ranges, route postpone records to their target buckets, and reuse Paimon's existing merge engine for each bucket. * Add a Spark DSv2 scan and physical execution path which clusters real split metadata and postpone records by partition and target bucket before delegating the merge to Core. * Fall back to the ordinary Paimon scan when no postpone files are present, keeping the existing pushdowns and execution path. * Share bucket assignment and postpone-file ordering between merge-on-read and the Spark postpone compact procedure. The option is disabled by default, so existing delayed-visibility behavior is unchanged. ### Tests * Core targeted tests: `PostponeUtilsTest` and `TableScanTest#testPostponeMergeReadBuilderAndPushDown` — 11 tests passed. * Spark `PostponeBucketTableTest` — 14 tests passed. * Spark `PrimaryKeyVectorSearchTest` — 12 tests passed. * Compiled the Spark 3.2, 3.5, and 4.1 modules. * Ran Spotless, regenerated the Core configuration documentation, and verified `git diff --check`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
