GitHub user roseduan edited a comment on the discussion: [Ideas] Better parallel scans for AO and AOCS tables
> This looks promising, especially for tables with a single large segment file. > Could you explain how a worker reaches its assigned row chunk without an > index or additional metadata? Does it retain its position and skip block > headers, and can multiple workers end up reading or decompressing the same > blocks? Could you also share the patch and benchmark setup—table size, > compression, projected columns, worker count, and whether the data was > cached? That would help assess the tradeoff between better balancing and > extra scan work. Thanks for your attention. Well, may be the saying 'without additional metadata' is not very accurate. The whole mechanism is roughly like: * get how many tuples of an aocs table * calculate a number of rows that a single worker will scan (this is called a chunk, a chunk contains multiple rows) * each worker scan a chunk at a time, and then claim another chunk to scan * since we don't have a block directory or any other information to find the start row quickly, so right now we only scan the block header to find out if the start row is in the current block. (but we only scan the block header from last position instead of the begining, so maybe it's not an expensive cost, but I will test it anyway) In addition, I will do some benchmarks later, so you can run the new implementation in your env. Thanks again, feel free to give me any feedback. GitHub link: https://github.com/apache/cloudberry/discussions/2186#discussioncomment-19213258 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
