matanper opened a new issue, #69924: URL: https://github.com/apache/doris/issues/69924
### Search before asking - [x] I had searched in the issues and found no similar issues. ### Version 4.1.x (also present on master) ### What's Wrong? `ZoneMapIndexReader` keeps every page's `ZoneMapPB` in memory (`std::vector<ZoneMapPB> _page_zone_maps`). For a string column whose values are longer than `MAX_ZONE_MAP_INDEX_SIZE` (512 bytes), the writer cuts the page bounds, so each page holds two 513-byte strings. After allocation that is about 1.4 KB per page. These readers are loaded lazily per predicate column (`ColumnReader::get_row_ranges_by_zone_map`, `_apply_expr_zonemap_to_row_ranges`) and kept by the column reader cache inside each `Segment`. That memory is not charged to `SegmentCache`, so cache eviction does not see it. #68304 charges only primary key index roots. A wide scan with a predicate on a long string column, across many segments, therefore keeps tens of GB in zone map readers (`ZoneMapIndexReaderMemBytes`). Queries then fail with `MEM_LIMIT_EXCEEDED` once the process limit is reached. The memory comes back when segments are evicted, but normal query load rebuilds it within minutes. #67642 makes cut bounds safe (the carry on a 0xFF last byte, and no MIN/MAX answers from cut bounds), but page pruning still keeps and uses the full 512-byte bounds. ### What You Expected? Page zone map memory for long string columns stays small, or is bounded by a cache. ### How to Reproduce? 1. Create a table with a VARCHAR/STRING column whose values are longer than 512 bytes. Load enough rows for many pages and segments. 2. Run a query with a range or equality predicate on that column, so zone map pruning loads the page zone maps. 3. Watch `ZoneMapIndexReaderMemBytes` (bvar / mem tracker). It grows by about 1.4 KB per page scanned and is not charged to `SegmentCache`. In a small test (200k rows of 700-character strings), the reader memory after one scan was about 2.2 MB. With cut pages compacted to `pass_all`, it fell to about 13 KB with identical query results. ### Anything Else? Possible fixes: - (a) Keep only a short prefix of a cut bound in memory, e.g. 16 to 32 bytes. A prefix of min is still a lower bound. A prefix of max, raised with the carry from #67642, is still an upper bound. Pruning keeps nearly all of its power at a fraction of the memory. - (b) Treat a page with a cut bound as `pass_all` when loading (keep `has_null` / `has_not_null`). This is simplest, but it gives up page pruning on those pages, which is often of little value for long free-text values anyway. - (c) Charge lazily loaded index readers to `SegmentCache`, as #68304 does for primary key roots. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's Code of Conduct -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
