bhat-vinay opened a new pull request, #10876: URL: https://github.com/apache/hudi/pull/10876
### Change Logs Sort the input records based on new configs for INSERT operation.The records are globally sorted (within an input batch). This allows for efficient pruning of files (based on column stats) on the query path. BaseSparkCommitActionExecutor sorts the input records (if the configs are set) based on partition-path + sort-column. The default sort-column is the record-key. A new UpsertSortPartitioner is implemented which assigns a contiguous chunk of sorted input records to a given bucket (or spark RDD partition). The follwoing changes are made in this PR: HoodieWriteConfig.java: Two new configs to enable/diable sorting for insert operation AND to specify custom sort columns. BaseSparkCommitActionExecutor.java: The input records are sorted (using JavaRDD::sortBy(...) lazy evaluator) here and subsequently indexed (using JavaRDD::zipWithIndex(...) lazy evaluator). The sorted records are subsequently fed to the new UpsertSortPartitioner to assign contiguous chunk of sorted records to a specific bucket (or Spark RDD partition). UpsertSortPartitioner.java: New partitioner that extends from UpsertPartitioner. There are two major changes (as compared to UpsertPartitioner). 1. The partitionPath in the workload profile is extracted as a sorted list before UpsertPartitioner can assign bucket numbers to Hudi partition. Since the records are sorted on the prefix of partition-path, this ensures that a contiguous chunk of sorted input records can be assigned to specific buckets. 2. getPartition(...) method is overridden with logic so that successive indexed and sorted input records can be assigned to the same bucket i.e a contiguous chunk of input records are assigned to the same bucket. ### Impact Allows the input records to be globally sorted for insert operation. This means that the records will be bin-packed into new files in a sorted manner (allowing for efficient pruning of files in the query path). The existing small files are bin-packed first and hence those existing small files may not be efficiently pruned. ### Risk level (write none, low medium or high below) Low. Existing tests and new tests should suffice. the new functionality is guarded behind a config (which is default disabled) ### Documentation Update New config op[tions need to be documented. ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Change Logs and Impact were stated clearly - [ ] Adequate tests were added if applicable - [ ] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
