plusplusjiajia opened a new pull request, #10948:
URL: https://github.com/apache/paimon/pull/10948

   ### Purpose
   
   A Format Table has no partition mark done today: 
`PartitionMarkDoneAction.createActions` and the `mark_partition_done` 
procedures only accept a `FileStoreTable`. Users moving Hive-style tables to 
Format Tables often have downstream jobs that wait for a `_SUCCESS` file in the 
partition directory.
   
   This PR supports the `success-file` action for Format Tables:
   
   - `FormatTableCommit#committedPartitions()` returns the partitions the last 
successful commit wrote, including the full static partition of an `INSERT 
OVERWRITE` that wrote no rows.
   - `FormatTableSuccessFileMarkDoneAction` writes `_SUCCESS` in the partition 
directory, following `format-table.partition-path-only-value`. With 
catalog-managed partitions, a partition registered at a custom location, or not 
registered, is skipped with a warning, so the file is only written inside the 
table directory.
   - `PartitionMarkDoneAction#createActions` gets a `FormatTable` overload. 
Only `success-file` is supported; other actions fail with a clear message.
   - Spark: with `partition.end-input-to-done = true`, a Format Table write 
marks the partitions it committed. Once the commit succeeds, a later abort (for 
example after marking fails) no longer deletes the committed data.
   - Spark and Flink `mark_partition_done` (procedure and Flink action) support 
Format Tables.
   
   Flink writes to a Format Table do not mark partitions done in this PR: each 
sink subtask commits on its own in `close()`, so there is no point where every 
subtask has finished. That needs a global committer and is left for a follow-up.
   
   ### Tests
   
   - `FormatTableSuccessFileMarkDoneActionTest`: key-value and value-only 
directories, escaped values, re-marking keeps the creation time, 
catalog-managed partitions at the default location / a custom location / not 
registered, unsupported actions, unpartitioned tables.
   - `FormatTableCommitCommittedPartitionsTest`: append, empty static 
overwrite, prefix overwrite, failed commit, unpartitioned table.
   - `FormatTablePartitionMarkDoneTest` (Spark): writes with and without 
catalog-managed partitions, empty static overwrite, value-only layout, disabled 
`end-input-to-done`, the procedure, and an unsupported action.
   - `FormatTableITCase#testMarkPartitionDoneProcedure` (Flink).
   - Existing `FormatTableCommit*Test`, `FormatTableScanTest`, 
`MarkPartitionDoneProcedureTest`, `PaimonFormatTableTest`, 
`MarkPartitionDoneActionITCase` pass.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to