ODriscollBoys opened a new pull request, #20280: URL: https://github.com/apache/hudi/pull/20280
### Describe the issue this Pull Request addresses Closes #19770 On a table with `hoodie.populate.meta.fields=false`, a row-writer `INSERT OVERWRITE` in dynamic partition mode went through while a clustering plan was pending on the file groups it replaced. The same statement is rejected when meta fields are populated (#18829). `rejectIfOverlappingPendingClustering` resolves the target partitions through `DatasetBulkInsertOverwriteCommitActionExecutor.resolveTargetPartitions`, whose dynamic arm read `_hoodie_partition_path` off the prepared dataset. Without meta fields, `HoodieDatasetBulkInsertHelper.prepareForBulkInsert` stubs every meta column with `null`. The arm therefore resolved `[null]`, the file system view returned no file slices for it, and the check returned without consulting the clustering update strategy. ### Summary and Changelog A row-writer dynamic `INSERT OVERWRITE` that overlaps pending clustering is now rejected whether or not meta fields are populated. With `SparkAllowUpdateStrategy`, it is handled by that strategy instead. - `DatasetBulkInsertOverwriteCommitActionExecutor.resolveTargetPartitions`: when meta fields are not populated (`writeConfig.populateMetaFields()`, the same condition `prepareForBulkInsert` stubs the columns on), derive the distinct partition paths with the configured key generator. This matches how the bulk-insert writer computes them (`BulkInsertDataInternalWriterHelper#extractPartitionPath`). It only runs when some file group is in pending clustering, since the base class returns early otherwise. - Javadoc updated to stop claiming that `_hoodie_partition_path` is always populated. - `TestInsertTable2`: the row-writer pending-clustering test now runs with and without meta fields for both static and dynamic overwrite modes. The dynamic mode without meta fields failed before this change. The issue also notes that `execute()` picks the partitioner from the raw `hoodie.populate.meta.fields` boolean rather than `writeConfig.populateMetaFields()`. That changes partitioner selection for selective meta-field modes, so it is left for a separate change. ### Impact Closes a gap in the pending-clustering guard for row-writer insert overwrite on tables without meta fields. No public API, config or storage format changes. ### Risk Level Low. The new code path only runs for dynamic overwrite on partitioned tables without populated meta fields, and only when clustering is pending. Validated with `TestInsertTable2` on the Spark 3.5 profile: the extended pending-clustering test fails without the fix (the dynamic overwrite of the clustered partition is accepted) and the whole suite passes with it. ### Documentation Update None. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
