plusplusjiajia commented on PR #10948: URL: https://github.com/apache/paimon/pull/10948#issuecomment-6649148355
Thanks for the questions. I went through the end-to-end flow again. The use case was Hive-style scheduling: a downstream job waits for `_SUCCESS` in a partition directory. That happens either after a Spark batch write, or once a Flink job has finished an hourly partition. With a REST catalog, the table's storage can only be reached with credentials the catalog issues. A scheduler could therefore read the file only through PVFS or the Paimon SDK; a plain file sensor cannot see it. In that setup the catalog is the better place for the signal. The Flink job now writes a Paimon append table with `partition.mark-done-action = mark-event`. The scheduler waits until `listPartitions` reports the hour as done, and then a Spark job writes the result into the Format Table. Nothing downstream waits on the Format Table, so it needs no marker. So this PR isn't needed and I'm closing it. Thanks for the review! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
