prathik-onehouse opened a new pull request, #20572: URL: https://github.com/apache/hudi/pull/20572
### Describe the issue this Pull Request addresses closes #19770 A row-writer bulk insert `INSERT OVERWRITE` in dynamic overwrite mode on a table with `hoodie.populate.meta.fields=false` skips the pending clustering guard. `prepareForBulkInsert` stubs the meta columns as null in that case, so `DatasetBulkInsertOverwriteCommitActionExecutor#resolveTargetPartitions` reads no real partitions from `_hoodie_partition_path`, finds no file groups to replace, and the overwrite replaces file groups that a pending clustering plan owns. Rows can be lost once that plan commits. ### Summary and Changelog When meta fields are not populated, the dynamic arm of `resolveTargetPartitions` now derives the partitions with the key generator the row writer uses to route each row (new `HoodieDatasetBulkInsertHelper#getDistinctPartitionPaths`), so the guard rejects an overwrite that overlaps pending clustering. `BaseDatasetBulkInsertCommitActionExecutor#execute` reads `writeConfig.populateMetaFields()` instead of the raw config key, matching `prepareForBulkInsert` and the writer. The pending clustering SQL test in `TestInsertTable2` now runs both overwrite modes with and without meta fields; without the fix, the dynamic case with meta fields off lets the overlapping overwrite through. ### Impact Overwrites that overlap pending clustering on tables without meta fields now fail with the same "Not allowed to update the clustering file group" error as tables with meta fields. When clustering is pending, the guard runs one extra Spark job over the prepared rows to collect their partitions. ### Risk Level low. The new path only runs for dynamic overwrite with meta fields off while clustering is pending, and it uses the same key generator the writer already uses. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
