danny0405 commented on PR #19806: URL: https://github.com/apache/hudi/pull/19806#issuecomment-5628749858
> Thanks @danny0405 for the triage - agreed that insert-overwrite partition (and likely other partition-replacement paths, e.g. clustering/compaction rewriting a partition) hit the same stale-index-records problem, so a per-operation fix in the delete-partition caller alone isn't the right shape. > > Before I widen the change, I'd like to align on the general solution level: > > 1. **Metadata-level fix (my leaning)**: when a partition's file set is rewritten (delete/insert-overwrite), invalidate or regenerate the partition's record-level index entries atomically with the commit, rather than having each call site remember to clean up. This covers MOR/COW and NBCC uniformly. > 2. **Operation-level family**: keep the fix in the operation paths that rewrite partitions (delete_partition, insert_overwrite), sharing one helper for index-entry invalidation. > > @vinothchandar - would the metadata-level approach fit the team's direction for the record-level index, or do you prefer keeping it scoped to the affected write paths for now? I'm happy to draft the metadata-level design in the issue if that's useful for triage. Thanks, the NBCC does not work with record-level-index and only works for simple bucket index, the possible solution that comes to my mind: 1. stop using the hard coding constant file group id suffix which is prefixed with the bucket num; 2. in order to impl 1, we need a atomic creation of the file group id with a given bucket num to resolve the conflicts from creation of multiple tasks from multiple jobs; 3. for Flink, the bucket write function also caches the bucket_num -> file group id mapping, we need a way to invalidate the cache immediately once a replace commit occurs for its corresponding partition. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
