adriangb commented on issue #6946: URL: https://github.com/apache/arrow-rs/issues/6946#issuecomment-5061172503
I was looking into something similar to this in our production system the other day. There's two problems that this is creating for us: 1. Because our columns are very wide / have large values (e.g. 1MB LLM prompts) buffering at the row group level causes high memory usage. Even if we could set some row limit that's still not optimal since depending on what columns the query reads, the shape of the data, etc. the optimal number varies widely. 2. Since fetching data off of object storage is slow there's latency because every time we need more data we have to wait to buffer an entire row group before we start decoding / processing it. I started looking into a design that has a stream per column. This results in more object store GETs but allows each column to stream at whatever pace it needs to to feed decoding and minimizes how much data has to be buffered in memory to keep the CPU fully fed. I initially got mixed results, but I'm posting here to bump the issue and share the pain / solution ideas. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
