morningman opened a new pull request, #68901:
URL: https://github.com/apache/doris/pull/68901
### What problem does this PR solve?
Issue Number: None
Related PR: #66473, #66530
Problem Summary:
**In short.** With `enable_external_scan_task_reuse` on (the default), the
Hive, Hudi, Iceberg and Paimon connectors keep the planned range list of every
scan in the statement scope until the statement ends, even when no other scan
of the statement can reuse it. Reuse can only happen while the statement is
being planned, so this PR drops the memo when `NereidsPlanner.plan()` returns
or throws. branch-4.x (#66473) bounds the same reuse with per-statement budgets
instead; master had no bound.
**Background.** #66530 lets the identical scans of one statement (a self
join, a UNION of the same table, a CTE read twice) plan once: each of the four
scan providers memoizes the final `List<ConnectorScanRange>` of a synchronous
`planScan` in a map hung in the statement's `ConnectorStatementScope`, keyed by
everything the ranges depend on. The scope lives until the statement's
query-finish callback closes it. Without the memo the ranges are garbage as
soon as the scan node has turned them into `TScanRangeLocations`, because the
scan node keeps no reference to them.
**The problem, and what it cost.** The memo is the only long-lived reference
to the ranges, and it keeps them for the whole execution:
- A statement with a single relation, which can never reuse anything, still
keeps all of its ranges until it ends.
- Paimon and Hudi have no batch path, so every scan of them keeps its whole
range list; a Paimon JNI range carries the Base64-serialized `DataSplit`, which
grows with the number of files. Hive keeps it for an unpartitioned table or
below the batch threshold, and Iceberg until a scan switches to streaming.
- An Arrow Flight statement keeps it until the client has fetched the whole
result.
Reuse needs none of this. Scans are planned only inside
`NereidsPlanner.plan()` (translation finalizes each scan node, which calls
`planScan`); the coordinator, Arrow Flight's DoGet and `handleQueryWithRetry`
never plan a scan again, and the batch / streaming paths that produce splits
during execution do not use the memo. A statement planned more than once (cloud
replan, INSERT retry, the probe of INSERT OVERWRITE) gets a new query id or a
reset scope each time, so no plan reuses the memo of another plan today either.
**How this PR fixes it.**
- `ConnectorStatementScope` (SPI) gains `computeIfAbsentForPlanning`, for a
value only planning reads, and `endPlanning`, which drops those values. The
defaults keep today's behavior (`computeIfAbsent` and a no-op), and `NONE`
still memoizes nothing. A value registered for planning must hold nothing to
close, since it is dropped without being closed. The connector SPI has not
shipped in a release, so `connector.plugin.api.version` stays as it is;
`ConnectorStatementScope` is not part of the frozen plugin surface baselines.
- `ConnectorStatementScopeImpl` records the keys registered for planning,
and `endPlanning` removes them; the other values of the statement (metadata,
tables, transactions) stay until `closeAll`.
- `NereidsPlanner.plan` ends connector planning in its `finally` block,
through `StatementContext.endConnectorPlanning`, which never creates a scope
for a statement no connector has touched.
- The Hive, Hudi, Iceberg and Paimon providers register their range memo for
planning. The Hudi and Paimon table-resolution maps, which
`getScanNodeProperties` also reads, stay for the statement. Fluss reads its
lake table through the Paimon provider, so it follows.
A scan planned during execution through the default
`planScanForPartitionBatch` (only MaxCompute uses it, and MaxCompute keeps no
memo) would register the memo again and keep it until the statement ends, as
today; the SPI javadoc says so.
**Results.**
| Statement | Before: the planned ranges are held until | After |
|---|---|---|
| SELECT, INSERT ... SELECT | the statement ends | `plan()` returns |
| Arrow Flight query | the client has fetched the result | `plan()` returns |
| EXPLAIN | the statement ends | `plan()` returns |
| Planning fails | the statement ends | `plan()` throws |
Identical scans of one plan still share their ranges. On a local cluster
with a filesystem Paimon catalog, a self join, a UNION ALL of the same table, a
self join filtered on a partition and a repeated statement return the expected
rows, and EXPLAIN shows both scans of the self join with the same splits.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- `ConnectorStatementScopeTest`: `endPlanning` drops only the
planning values, and a later plan of the statement registers and drops them
again; `NONE` never memoizes them; a planning value must hold nothing to close;
`StatementContext.endConnectorPlanning` ends the planning of its scope.
- `NereidsPlannerConnectorPlanningTest`: `plan()` ends connector
planning for a statement it plans (`select 1`) and for one that fails analysis.
- Hive, Hudi, Iceberg and Paimon
`statementReuseLastsOnlyWhilePlanning`: once planning ends the memo is gone and
an identical scan plans again; Paimon keeps the table resolved for the
statement.
- Mutations: not ending planning in `plan()`, an `endPlanning` that
drops nothing, dropping the closeable check, or registering a provider's memo
with `computeIfAbsent` each makes exactly the matching new cases fail.
- The fe-core scope and planner tests (49), fe-connector-spi (146),
Hive (492), Hudi (252), Iceberg (1500), Paimon (696) and Fluss (290) pass.
- The branch-4.x budget tests
(`testOversizedIcebergPlanIsNotRetained`,
`testStatementCacheStopsSerializingEntryOverByteBudget`,
`HudiScanNodeTest.testOversizedSnapshotPlanIsNotRetained` and the like) are not
ported: there is no budget to test.
- [x] Manual test (add detailed scripts or steps below)
- A local FE + BE whose FE is this change on master `038e2274ef6`,
with a filesystem Paimon catalog: `SELECT count(*) FROM t a JOIN t b ON a.id =
b.id`, a `UNION ALL` of the same table, a self join filtered on a partition,
and the first statement again return the expected rows; EXPLAIN shows both
scans of the self join with `inputSplitNum=3`.
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [x] No. Identical scans of one plan still share their ranges; the
ranges are no longer held after planning.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
