deepakpanda93 commented on issue #15653: URL: https://github.com/apache/hudi/issues/15653#issuecomment-4891403256
This is implemented on current 1.x. Data-table rollbacks are no longer applied to the metadata table as delta commits that append large numbers of delete records. Instead (HoodieTableMetadataUtil.convertMetadataToRecords for rollback metadata): - We only need to handle the FILES partition here (a MOR rollback may add a new log file that must be recorded). All other partitions — including RECORD_INDEX (RLI), column_stats, etc. — are handled by an actual rollback of the deltacommit that added those records. - So RLI rollbacks are done by rolling back the MDT deltacommit rather than resolving 1000s of delete records, which is exactly what this ticket requested and why RLI scales in current releases. The mechanism continues to be hardened (e.g. #18160 post-commit rollback of MDT deltacommits in 1.2.0, #18033, #18279). Regarding the earlier question about a failed DT instant that was never written to the MDT: there's simply no MDT deltacommit to roll back in that case, and the FILES-partition record path covers any log file added by the rollback. Closing as resolved on current 1.x. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
