nsivabalan commented on issue #17848: URL: https://github.com/apache/hudi/issues/17848#issuecomment-3766359317
hey @kbuci : I might need some clarification on the ask. Which writer are we talking about, is it spark batch writer or deltastreamer. for spark batch writer, I see we have support for "hoodie.datasource.write.commitmeta.key.prefix" where users can add additional metadata to commit metadata. But I don't think this was designed to capture lineage and I don't see any documentation calling out such support. For HoodieStreamer (deltastreamer), the checkpoint is used for internal purposes. To withstand failure and restarts, we store checkpoint from source for each batch in the commit metadata. Can you add more clarity on which one are we talking about. Or is it more of a custom implementation where you chose to store some additional metadata in commit metadata. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
