plusplusjiajia commented on PR #10948:
URL: https://github.com/apache/paimon/pull/10948#issuecomment-6649148355

   Thanks for the questions. I went through the end-to-end flow again.
   
   The use case was Hive-style scheduling: a downstream job waits for 
`_SUCCESS` in a partition directory. That happens either after a Spark batch 
write, or once a Flink job has finished an hourly partition. With a REST 
catalog, the table's storage can only be reached with credentials the catalog 
issues. A scheduler could therefore read the file only through PVFS or the 
Paimon SDK; a plain file sensor cannot see it.
   
   In that setup the catalog is the better place for the signal. The Flink job 
now writes a Paimon append table with `partition.mark-done-action = 
mark-event`. The scheduler waits until `listPartitions` reports the hour as 
done, and then a Spark job writes the result into the Format Table. Nothing 
downstream waits on the Format Table, so it needs no marker.
   
   So this PR isn't needed and I'm closing it. Thanks for the review!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to