github-actions[bot] commented on code in PR #70311:
URL: https://github.com/apache/doris/pull/70311#discussion_r4547081429
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/source/HiveScanNode.java:
##########
@@ -263,18 +268,54 @@ public void startSplit(int numBackends) throws
UserException {
}
});
}
- if (prunedPartitions.isEmpty()) {
+ int partitionCount = partitionInit ? prunedPartitions.size() : (int)
selectedPartitionNum;
+ if (partitionCount == 0) {
splitAssignment.finishSchedule();
return;
}
HiveExternalMetaCache cache = Env.getCurrentEnv().getExtMetaCacheMgr()
.hive(hmsTable.getCatalog().getId());
Executor executor = getSplitTaskExecutor();
- String broker = hmsTable.getCatalog().bindBrokerName();
+ HMSExternalCatalog catalog = (HMSExternalCatalog)
hmsTable.getCatalog();
+ String broker = catalog.bindBrokerName();
// Without the file cache, the eager path listed partitions
concurrently. Keep that overlap.
boolean parallelListing = !fileCountBatch ||
Config.max_external_file_cache_num <= 0;
+ int partitionsPerProducer = partitionInit ? 1 :
sessionVariable.splitPartitionBatchSize;
int producers = parallelListing
- ? Math.min(prunedPartitions.size(),
Config.max_external_cache_loader_thread_pool_size) : 1;
+ ? Math.min((partitionCount - 1) / partitionsPerProducer + 1,
Review Comment:
[P2] Preserve directory listing parallelism for small partition sets
With the defaults `num_partitions_in_batch_mode=1` and
`split_partition_batch_size=16`, a real partitioned scan selecting 2-16
partitions takes the new lazy path (`partitionInit=false`), so this expression
creates only one producer. That producer claims the whole slice and lists each
partition through a singleton `getFilesByPartitions` call (`concurrent=false`),
serializing previously concurrent remote directory reads even when the executor
is idle. Eight equally slow listings now form eight sequential waits instead of
overlapping; COUNT prefetch uses the same reduced worker count. Keep listing
concurrency independent of claim size and distribute claims across available
readers. The existing overlap test calls an overridden eager `getPartitions`,
so add overlap coverage on the new lazy path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
