zhang-arvin commented on PR #19806: URL: https://github.com/apache/hudi/pull/19806#issuecomment-5616751807
Thanks @danny0405 for the triage - agreed that insert-overwrite partition (and likely other partition-replacement paths, e.g. clustering/compaction rewriting a partition) hit the same stale-index-records problem, so a per-operation fix in the delete-partition caller alone isn't the right shape. Before I widen the change, I'd like to align on the general solution level: 1. **Metadata-level fix (my leaning)**: when a partition's file set is rewritten (delete/insert-overwrite), invalidate or regenerate the partition's record-level index entries atomically with the commit, rather than having each call site remember to clean up. This covers MOR/COW and NBCC uniformly. 2. **Operation-level family**: keep the fix in the operation paths that rewrite partitions (delete_partition, insert_overwrite), sharing one helper for index-entry invalidation. @vinothchandar - would the metadata-level approach fit the team's direction for the record-level index, or do you prefer keeping it scoped to the affected write paths for now? I'm happy to draft the metadata-level design in the issue if that's useful for triage. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
