924060929 commented on code in PR #69991:
URL: https://github.com/apache/doris/pull/69991#discussion_r4445707092
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/source/HiveScanNode.java:
##########
@@ -256,7 +261,7 @@ public void startSplit(int numBackends) {
try {
List<Split> allFiles = Lists.newArrayList();
getFileSplitByPartitions(
- cache,
Collections.singletonList(partition), allFiles, bindBrokerName,
Review Comment:
Fixed in b76e3987f5e88de9bee95cabbd60a7d104b7f1c4. Hive now counts the
produced splits by their actual partition values within each scheduling batch,
then atomically keeps the largest per-partition count. The existing minimum of
ten per partition remains intact. This also covers TABLESAMPLE and
unpartitioned scans, without repeating metadata reads or file splitting. The
real asynchronous split-generation tests reproduced 360 instead of 60 (and 116
instead of 100) before the repair; all four cases now pass across batch sizes
1/2/16, including unequal/empty partitions. All 209 related FE tests and full
FE packaging/Checkstyle pass locally.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/SplitAssignment.java:
##########
@@ -113,15 +126,24 @@ private void appendBatch(Multimap<Backend, Split> batch)
throws UserException {
locations.add(splitToScanRange.getScanRange(backend,
locationProperties, split, pathPartitionKeys,
fileCacheAdmission));
}
- while (needMoreSplit()) {
- BlockingQueue<Collection<TScanRangeLocations>> queue =
- assignment.computeIfAbsent(backend, be -> new
LinkedBlockingQueue<>(10000));
- try {
- if (queue.offer(locations, 100, TimeUnit.MILLISECONDS)) {
- break;
+ BlockingQueue<TScanRangeLocations> queue =
assignment.computeIfAbsent(
+ backend.getId(), be -> new
LinkedBlockingQueue<>(queueCapacity));
+ for (TScanRangeLocations location : locations) {
+ while (needMoreSplit()) {
+ try {
Review Comment:
I rechecked the carried-forward rolling-upgrade concern against the base
release. It already has a [10,000-batch producer
queue](https://github.com/apache/doris/blob/a623df21d75683c61a39ae830822850e6be0a7f0/fe/fe-core/src/main/java/org/apache/doris/datasource/SplitAssignment.java#L118)
and [stops scanners at a local
LIMIT](https://github.com/apache/doris/blob/a623df21d75683c61a39ae830822850e6be0a7f0/be/src/exec/operator/scan_operator.cpp#L1285).
Its live empty wait [throws
Timeout](https://github.com/apache/doris/blob/a623df21d75683c61a39ae830822850e6be0a7f0/fe/fe-core/src/main/java/org/apache/doris/datasource/SplitSource.java#L102),
which FrontendService maps to INTERNAL_ERROR. Thus the old FE's per-backend
termination gap predates this PR. The new BE still propagates that error after
one fetch; the added OldFrontendTimeoutRemainsAnErrorWithoutRetry test verifies
one fetch and no unsupported close RPC with capability absent. I corrected the
close-path comment that had incorrectly
claimed older FEs had no bounded-queue dependency. With the supported
BE-first upgrade, the new FE retry/close protocol is used only after BEs
support it; an old FE retains its existing protocol and behavior. The new-FE
close/queue-release fix and capability gate remain unchanged. The clarification
and regression test are in b76e3987f5e88de9bee95cabbd60a7d104b7f1c4.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
