sunchao opened a new issue, #25392: URL: https://github.com/apache/datafusion/issues/25392
### Describe the problem Generic hash joins reserve hash-bucket capacity from the number of build rows, even when most rows have the same join key. The lookup table stores one entry per distinct hash; duplicate rows are already retained in its separate row-index chain. Sizing both structures by total rows therefore wastes bucket memory and can reject a join whose payload, duplicate chain and useful lookup index fit in the memory pool. For example, a build with many rows keyed by a small set of state codes still needs every row for join multiplicity, but does not need a hash bucket for every row. This is distinct from deduplicating semi-join input: ordinary joins must preserve all matching rows. ### Expected behavior Keep the row-index chain sized for every build row, but start the generic lookup index small and grow it only when needed. Preserve a fast construction path for mostly unique keys. Account for scratch and old/new index allocations during resizing so the optimization also works under a bounded memory pool. ### Scope The generic hash lookup path in `HashJoinExec`. The existing perfect-hash `ArrayMap` is a separate path. This does not add spilling or address build-payload concatenation accounting. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
