github-actions[bot] commented on code in PR #70746:
URL: https://github.com/apache/doris/pull/70746#discussion_r4625789929
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java:
##########
@@ -549,6 +552,46 @@ protected boolean supportsBeSplitRefinement(FileSplit
split) {
return true;
}
+ protected boolean supportsFeSplitRefinement(FileSplit split) {
+ return isColumnarFormat(params.getFormatType()) &&
supportsBeSplitRefinement(split);
+ }
+
+ List<Split> splitForBackendParallelism(List<Split> splits, int
numBackends) throws UserException {
+ int desiredSplits = sessionVariable.getMaxFileSplitNum() > 0
+ ? Math.min(numBackends, sessionVariable.getMaxFileSplitNum())
: numBackends;
+ if (splits.size() >= desiredSplits || coarseSplitSizeForBeRefinement()
<= 0
+ || isTableLevelCountStarPushdown()) {
+ return splits;
+ }
+
+ // BE physical children stay on their source BE. Retain a bounded
number of FE ranges
+ // for cross-BE placement without listing files again or making
already ample plans finer.
+ PriorityQueue<FileSplit> candidates = new PriorityQueue<>(
+ Comparator.comparingLong(FileSplit::getLength).reversed());
+ List<Split> result = new ArrayList<>();
+ for (Split split : splits) {
+ FileSplit fileSplit = (FileSplit) split;
+ if (supportsFeSplitRefinement(fileSplit) &&
getFileCompressType(fileSplit) == TFileCompressType.PLAIN) {
+ candidates.add(fileSplit);
+ } else {
+ result.add(split);
+ }
+ }
+ while (!candidates.isEmpty() && result.size() + candidates.size() <
desiredSplits) {
+ FileSplit largest = candidates.poll();
+ long half = largest.getLength() / 2;
+ // Avoid replacing one coarse range with tiny FE tasks merely to
reach every BE.
+ if (half < sessionVariable.getFileSplitSizeOnBe()) {
+ result.add(largest);
+ break;
+ }
+ candidates.add(largest.withRange(largest.getStart(), half));
Review Comment:
[P2] Keep root-only Parquet ranges whole
This can turn a supported partition-only scan into a corruption error. For
example, with two BEs, synchronous Hive planning and `file_split_size_on_be=1`
(an accepted session value), `SELECT p FROM t` over a positive-row, root-only
Parquet file used to receive one full-file range. The partition projection
becomes a constant, so the native reader can return the footer row count with
no physical columns (see
`PositiveRowRootOnlyFileKeepsInitializedReaderForCount`). These copies instead
produce two partial ranges; `select_native_row_groups_by_scan_range()` rejects
both with `Parquet row group ... has no column chunks`. BE physical refinement
deliberately retains the initialized parent for this case because it has no
byte envelope. Preserve that whole-file behavior here, or teach partial-range
ownership to handle such row groups, and cover the partition-only query through
FE subdivision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]