GitHub user roseduan created a discussion: [Ideas] Better parallel scans for AO and AOCS tables
### Description Hi all, I'm working on making parallel scan of append-optimized tables (AO and AOCS) more useful, and I'd like your feedback. **The problem** Today each parallel worker reads a whole segment file. A table with few segment files, often just one after a bulk load, gets little or no parallelism(if GUC `gp_appendonly_insert_files` is not set), and one big file is always read by a single worker. **The idea** Split each segment file into many small chunks of rows, and let workers take chunks one at a time from a shared counter. Every worker stays busy until the scan ends, however many segment files the table has. No index or extra metadata is needed. **Status** - AOCS: implemented. Results match serial scans, the related regression tests pass, and large tables scan several times faster in my tests. The planner can now choose a parallel plan for AOCS tables with only one segment file. - AO row: planned next, using the same approach. **Questions for the community** - Does this direction make sense to you? - Anything else I should cover or test? Thanks! ### Use case/motivation _No response_ ### Related issues _No response_ ### Are you willing to submit a PR? - [X] Yes I am willing to submit a PR! GitHub link: https://github.com/apache/cloudberry/discussions/2186 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
