beryllw opened a new issue, #3314:
URL: https://github.com/apache/fluss/issues/3314

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/fluss/issues) and 
found nothing similar.
   
   
   ### Motivation
   
   For partitioned tables with Paimon lake tiering enabled, downstream batch 
jobs need a signal that a partition's data is "ready" (i.e., fully tiered). 
This is the **Mark Done** mechanism — when a partition has been idle (no new 
data tiered) for a configurable duration, Fluss should execute Paimon's 
mark-done actions (e.g., write `_SUCCESS` file, notify metastore) so downstream 
schedulers can safely begin processing.
   
   Currently, Fluss tiering operates at the table level with no partition-level 
idle tracking or mark-done capability. Paimon already has a complete mark-done 
action framework (`PartitionMarkDoneAction`), but its trigger mechanism is 
tightly coupled to Flink checkpoint lifecycle, which doesn't fit Fluss's 
tiering model.
   
   ### Solution
   
   Extend the Fluss offset file (V2) with:
   - `partition_tiered_times`: map of partitionId -> epoch millis of last 
tiering commit (wall-clock time, unaffected by compaction)
   
   ### Anything else?
   
   _No response_
   
   ### Willingness to contribute
   
   - [x] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to