alamb commented on issue #24704: URL: https://github.com/apache/datafusion/issues/24704#issuecomment-5679617604
> > Having nested Vec harm performance due to the extra indirection and add cost for the low cardinality case > > Two things that might address the low-cardinality concern: > > 1. **Flat until the first block fills.** State is a plain Vec until it reaches block capacity B. At that point the Vec becomes block 0 as-is (no copy, since B is a power of two and doubling lands on it exactly), and further growth appends new blocks. Below B nothing changes from today: same Vec, same values[idx], same emit. I think this is the only realistic way we will avoid a prerformance regression I also agree with @jayzhan211 and @2010YOUY01 that mmap is probably not a valid solution for us as not all DataFusion users even hav a local disk they could use for mmap to overcommit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
