Hi all, I'd like to bring up this discussion around handling errors in Hudi write paths. https://issues.apache.org/jira/browse/HUDI-648
Trying to gather some feedbacks about the implementation details 1. Error location I'm thinking of writing the failed records to `.hoodie/errors/` for a) encapsulate data within the Hudi table for ease of management b) make use of existing dedicated directory 2. Write path org.apache.hudi.client.HoodieWriteClient#postWrite org.apache.hudi.client.HoodieWriteClient#completeCompaction These 2 methods should be the places to persist failed records in `org.apache.hudi.table.action.HoodieWriteMetadata#writeStatuses` to the designated location 3. Format Records should be written as logs (avro) 4. Metric Post writing failed records, it should send a metric of basic count of errors written. Easier for monitoring system to pick up and send alert. Foreseeably, some details may need to be adjusted throughout the development. To begin with, we may agree on a feasible plan at high level. Please kindly share thoughts and feedbacks. Thank you. Regards, Raymond
