yujun777 opened a new pull request, #69850: URL: https://github.com/apache/doris/pull/69850
### What problem does this PR solve? Issue Number: None Related issue: https://github.com/apache/doris/issues/65418 Problem Summary: **In short.** A scan that reads a table stream is planned over a lightweight wrapper of the base table: `OlapTableWrapper` for the base-table reads (`stream@reset`, and the base part of `stream@snapshot`), `OlapChangelogTableWrapper` for the changelog part. The wrapper delegates partition lookups but holds no partitions of its own, while `OlapTable` derives a row count by iterating that map. Every scan over a wrapper was therefore estimated at zero rows, which `StatsCalculator.computeOlapScan` raises to one: a read of a four-billion-row table looked like a one-row scan. In cloud mode `NormalizeOlapTableStreamScan` wraps the base table for every read once the cloud read states are installed, so both `stream@reset()` and `stream@snapshot()` were estimated at one row. An incremental view maintenance refresh plans its reads the same way. With the wrong estimate the optimizer picks the real four-hundred-million-row side as the broadcast build, and the refresh exhausts the task memory limit. **Reproduction.** One duplicate-key table with four rows and one stream (`show_initial_rows=false`, the fourth row written after the stream). Before this change the plans estimated the base table at 4 rows, `stream@reset()` at 1 row in cloud (4 rows in shared-nothing, where the scan uses the real table), and `stream@snapshot()` at 1 row in both. **How this PR fixes it.** - `OlapTableWrapper` delegates `getRowCountForIndex`, `getRowCountForPartitionIndex` and `getPartitionNum` to the table it wraps. A reset read, and a snapshot read whose offsets already reached the end, see exactly that table's content; a snapshot read of an older offset is bounded by it. - `OlapChangelogTableWrapper` deliberately does not inherit that count: a changelog scan reads the change rows of its offset range, not the table, so the table's size would overestimate a delta by orders of magnitude. It keeps the previous conservative estimate until change-row statistics exist. **Results** (local clusters, the same table and stream; `stats=` is the estimate printed in the query profile): | Read | Before (cloud) | After (cloud) | Before (shared-nothing) | | --- | --- | --- | --- | | base table | 4 | 4 | 4 | | `stream@reset()` | 1 | 4 | 4 | | `stream@snapshot()` | 1 | 4 | 1 | `stream@snapshot()` actually reads 3 of the 4 rows: the fourth was written after the stream's offset, so its estimate is an upper bound by design. ### Release note Fix stream reads (including incremental view maintenance refreshes) estimating every scan at one row, which could make the optimizer broadcast a very large build side and exhaust memory during a refresh. ### Check List (For Author) - Test: Unit test / Manual test - Unit test: `OlapTableWrapperTest`, two new cases (row count delegation; the changelog wrapper keeps the conservative estimate instead of inheriting the base table's count). - Manual test: the reproduction above, verified through query profiles on a local cloud cluster before and after the change. - Behavior changed: Yes. Scans over a stream read are now estimated from the table they read instead of one row. - Does this need documentation: No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
