alamb commented on PR #15591: URL: https://github.com/apache/datafusion/pull/15591#issuecomment-5333397682
> In the very hot loop of doing aggregation and jumping in memory you are adding another memory jump I think this is the key observation -- I think there is no way we can add the extra indirection and not slow down performance for small aggregates. For higher cardinality, I think it may not be as bad as we are going to be doing lots of random memory acesses anyways as the table doesn't fit in the processor caches. So I am guessing what we will need is some way to use direct indexing for small hash tables, but when the table grows above a certain size switch to 2 part indexes But that is just a theory and I have not tested it yet -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
