nsivabalan commented on pull request #3590: URL: https://github.com/apache/hudi/pull/3590#issuecomment-922983380
@vinothchandar : I have addressed most of the feedback from you except for few. Have responded in the other patch for those which is yet to be resolved. wrt multi-writer and compaction, here is what is happening in this patch. (have covered this in the description of the patch, but anyways). All writes to metadata table happens within the datatable lock. So, implicitly there are no multi-writers (including compaction, cleaning happening in datatable) to metadata table. There will always be only one writer. For regular writes to data table, we do conflict resolution, where as for compaction and clustering (data table), we don't do any conflict resolution. Remember taking a lock while committing compaction and clustering in data table is being added in this patch. So, please pay attention if there is something I am missing. So, in summary, we have a constraint only single writer model to metadata table. May be we need to revisit this when we integrate record level index. But for now, this is our current state of things. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
