tub opened a new issue, #9690: URL: https://github.com/apache/paimon/issues/9690
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Motivation ## Problem When combining batch generated & streaming data, a single large snapshot in a Paimon table can stall Flink checkpointing for a long time. ```mermaid flowchart TD Spark --> BatchData[(Batch Data)] BatchData --> FlinkJoin EventData[(Event Data)] --> FlinkJoin FlinkJoin --> JoinedData[(Joined Data)] ``` For example, if we rewrite every row in `Batch Data` in a single snapshot from Spark, the realtime `Event Data` stalls until the full table is scanned. ## Solution Using Paimon's branching support, we can split the Spark changes up into smaller commits on a staging branch, then fast-forward the main branch at the end of the batch. This allows us to have smaller, incremental snapshots while keeping the batch update atomic & avoiding partial data if the batch fails half way through. ### Solution This is achievable today, but it would be great to have a similar feature to Iceberg, where you can specify `write.wap.enabled` and the branch is created & merged automatically. Prior art from Iceberg: https://iceberg.apache.org/docs/latest/branching/#audit-branch ### Anything else? _No response_ ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
