github-actions[bot] commented on code in PR #66818:
URL: https://github.com/apache/doris/pull/66818#discussion_r3793594620
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java:
##########
@@ -664,6 +665,11 @@ private List<ConnectorScanRange> planScanInternal(
if (projected.length > 0) {
readBuilder.withProjection(projected);
}
+ if (limit > 0 && limit <= Integer.MAX_VALUE) {
Review Comment:
[P2] Preserve limit pushdown for file-creation scans
This attaches the limit only to the `TableScan` created below, but the
supported pinned file-creation-time branch later discards that scan and calls
`planFileCreationTimeSplits()`, which creates a fresh `SnapshotReader` and
returns all of `read().splits()`. Consequently an
`@options("scan.file-creation-time-millis"=...) ... LIMIT 1` query still
enumerates every qualifying file, so this PR's split-planning optimization is
silently absent on that path. Please either carry a conservatively safe limit
into the direct snapshot-reader path (after applying the filter and fallback
safety gates in the other comments), or explicitly scope the optimization and
cover this branch in the test.
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java:
##########
@@ -664,6 +665,11 @@ private List<ConnectorScanRange> planScanInternal(
if (projected.length > 0) {
readBuilder.withProjection(projected);
}
+ if (limit > 0 && limit <= Integer.MAX_VALUE) {
+ // Paimon's limit is an int and may prune whole splits; never
narrow a larger Doris limit,
+ // because doing so could omit rows before the engine applies its
authoritative long limit.
+ readBuilder.withLimit((int) limit);
Review Comment:
[P1] Do not prune splits before residual filters
Paimon 1.3.1's
[`applyPushDownLimit()`](https://github.com/apache/paimon/blob/release-1.3.1/paimon-core/src/main/java/org/apache/paimon/table/source/DataTableBatchScan.java#L123-L154)
stops after summing each raw-convertible split's pre-filter
`partialMergedRowCount`, while Paimon filtering is best-effort and Doris
rechecks rows later. For example, if the first split contains `{1,3}` and a
later split contains `{2}`, `WHERE id = 2 LIMIT 1` can retain only the first
split (its min/max still admits 2), then filter it to zero and never read the
match. Paimon 1.4.2 added a [`hasNonPartitionFilter()`
guard](https://github.com/apache/paimon/blob/release-1.4.2/paimon-core/src/main/java/org/apache/paimon/table/source/DataTableBatchScan.java#L118-L121),
which the pinned version lacks. Please push this limit only when the original
request filter is absent (not merely when converted `predicates` is empty,
because conversion can decline expressions), and add a result-bearing filtered
regressi
on.
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java:
##########
@@ -664,6 +665,11 @@ private List<ConnectorScanRange> planScanInternal(
if (projected.length > 0) {
readBuilder.withProjection(projected);
}
+ if (limit > 0 && limit <= Integer.MAX_VALUE) {
+ // Paimon's limit is an int and may prune whole splits; never
narrow a larger Doris limit,
+ // because doing so could omit rows before the engine applies its
authoritative long limit.
Review Comment:
[P1] Exclude fallback-backed scans before applying this limit
In the pinned Paimon 1.3.1,
[`FallbackReadScan.withLimit`](https://github.com/apache/paimon/blob/release-1.3.1/paimon-core/src/main/java/org/apache/paimon/table/FallbackReadFileStoreTable.java#L314-L318)
limits both children, then
[`plan()`](https://github.com/apache/paimon/blob/release-1.3.1/paimon-core/src/main/java/org/apache/paimon/table/FallbackReadFileStoreTable.java#L384-L404)
derives main-owned `completePartitions` only from the already-truncated main
plan. If main has partitions p1 and p2 but LIMIT 1 stops after p1, p2 is
misclassified as absent and its older fallback copy is emitted; later unordered
scheduling can therefore return stale p2 data. [Paimon
1.4.2](https://github.com/apache/paimon/blob/release-1.4.2/paimon-core/src/main/java/org/apache/paimon/table/FallbackReadFileStoreTable.java#L463-L483)
fixes this by listing main partitions with a separate unlimited scan. Please
exclude direct and decorator/system-hidden fallback sources while Doris is on
1.3.1 (checking
only the visible table class is insufficient), and add a result-bearing
fallback regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
