github-actions[bot] commented on code in PR #70746:
URL: https://github.com/apache/doris/pull/70746#discussion_r4625789929


##########
fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java:
##########
@@ -549,6 +552,46 @@ protected boolean supportsBeSplitRefinement(FileSplit 
split) {
         return true;
     }
 
+    protected boolean supportsFeSplitRefinement(FileSplit split) {
+        return isColumnarFormat(params.getFormatType()) && 
supportsBeSplitRefinement(split);
+    }
+
+    List<Split> splitForBackendParallelism(List<Split> splits, int 
numBackends) throws UserException {
+        int desiredSplits = sessionVariable.getMaxFileSplitNum() > 0
+                ? Math.min(numBackends, sessionVariable.getMaxFileSplitNum()) 
: numBackends;
+        if (splits.size() >= desiredSplits || coarseSplitSizeForBeRefinement() 
<= 0
+                || isTableLevelCountStarPushdown()) {
+            return splits;
+        }
+
+        // BE physical children stay on their source BE. Retain a bounded 
number of FE ranges
+        // for cross-BE placement without listing files again or making 
already ample plans finer.
+        PriorityQueue<FileSplit> candidates = new PriorityQueue<>(
+                Comparator.comparingLong(FileSplit::getLength).reversed());
+        List<Split> result = new ArrayList<>();
+        for (Split split : splits) {
+            FileSplit fileSplit = (FileSplit) split;
+            if (supportsFeSplitRefinement(fileSplit) && 
getFileCompressType(fileSplit) == TFileCompressType.PLAIN) {
+                candidates.add(fileSplit);
+            } else {
+                result.add(split);
+            }
+        }
+        while (!candidates.isEmpty() && result.size() + candidates.size() < 
desiredSplits) {
+            FileSplit largest = candidates.poll();
+            long half = largest.getLength() / 2;
+            // Avoid replacing one coarse range with tiny FE tasks merely to 
reach every BE.
+            if (half < sessionVariable.getFileSplitSizeOnBe()) {
+                result.add(largest);
+                break;
+            }
+            candidates.add(largest.withRange(largest.getStart(), half));

Review Comment:
   [P2] Keep root-only Parquet ranges whole
   
   This can turn a supported partition-only scan into a corruption error. For 
example, with two BEs, synchronous Hive planning and `file_split_size_on_be=1` 
(an accepted session value), `SELECT p FROM t` over a positive-row, root-only 
Parquet file used to receive one full-file range. The partition projection 
becomes a constant, so the native reader can return the footer row count with 
no physical columns (see 
`PositiveRowRootOnlyFileKeepsInitializedReaderForCount`). These copies instead 
produce two partial ranges; `select_native_row_groups_by_scan_range()` rejects 
both with `Parquet row group ... has no column chunks`. BE physical refinement 
deliberately retains the initialized parent for this case because it has no 
byte envelope. Preserve that whole-file behavior here, or teach partial-range 
ownership to handle such row groups, and cover the partition-only query through 
FE subdivision.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to