KnightChess commented on code in PR #11463:
URL: https://github.com/apache/hudi/pull/11463#discussion_r1643781381
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java:
##########
@@ -157,6 +157,9 @@ public HoodieWriteMetadata<HoodieData<WriteStatus>>
execute(HoodieData<HoodieRec
// Handle records update with clustering
HoodieData<HoodieRecord<T>> inputRecordsWithClusteringUpdate =
clusteringHandleUpdate(inputRecords);
+ if (config.getBuildWorkloadParallelism() > 0) {
+ inputRecordsWithClusteringUpdate =
inputRecordsWithClusteringUpdate.repartition(config.getBuildWorkloadParallelism());
Review Comment:
> Refer to the initial file number which is to upsert to hudi table, we can
change the number by change source file number but the way is not convenient
@xuzifu666 got it, and this is an existing question in other logical, add
new param is not friendly to user, I have a pr will optimizing the inference
problem of partitions and it can also solve your problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
