924060929 commented on code in PR #70664:
URL: https://github.com/apache/doris/pull/70664#discussion_r4624591663


##########
fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java:
##########
@@ -543,6 +545,57 @@ private TScanRangeLocations splitToScanRange(
         return curLocations;
     }
 
+    protected boolean supportsBeSplitRefinement(FileSplit split) {
+        return true;
+    }
+
+    protected long coarseSplitSizeForBeRefinement() {
+        return isCoarseSplitPlanningEnabled(sessionVariable) && 
!wouldRunSerialOnBe()

Review Comment:
   [P2] Preserve scan parallelism across BEs. In synchronous Hive planning with 
two eligible BEs and round-robin assignment, one 400 MiB Parquet file currently 
produces two ranges at the default 256 MiB target. At 512 MiB it produces one 
range, so all physical children run on the same BE. Can we account for 
available BEs and expected range count before coarsening, and add a multi-BE 
test that checks placement?



##########
fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java:
##########
@@ -1948,6 +1948,7 @@ public List<Split> getSplits(int numBackends) throws 
UserException {
                 .requiredPartitions(requiredPartitions)
                 .partitionsPrunedToEmpty(partitionsPrunedToEmpty)
                 .countPushdown(countPushdown)
+                .coarseSplitSize(coarseSplitSizeForBeRefinement())

Review Comment:
   [P2] Preserve Hive `TABLESAMPLE` granularity. Hive samples the ranges 
returned by the connector. With `file_split_size=16 MiB`, a 4% sample of one 
400 MiB Parquet file selects 16 MiB under the existing policy. The 512 MiB 
coarse target produces one range and the same sample selects the whole file. 
Can we use `applySample` to keep legacy FE sizing for sampled scans? The BE can 
still refine the selected ranges.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to