dwsmith1983 opened a new pull request, #26793: URL: https://github.com/apache/datafusion/pull/26793
> [!NOTE] > Stacked on #26228. Until that merges, this branch also carries its commits; the change here is the last commit. Opened as a draft for that reason. ## Which issue does this PR close? - Closes #26792. ## Rationale for this change `build_in_memory` had two build paths that only shared a concat. The `ArrayMap` path concatenated first. The `JoinHashMap` path hashed each buffered batch with its own offset and concatenated afterwards, and #26226 came from those offsets and the concat order disagreeing. After #26228 both paths concatenate the same way, so they can share one concat (as @jayzhan211 suggested and @comphead pointed out on #26228). Building after the concat also lowers the build-side peak on the non-prepared path. The inputs are released once their copy exists, so the hash map is no longer allocated while the inputs and the copy are both alive: the peak goes from about `inputs + map + copy` to `max(2 * inputs, copy + map)`. ## What changes are included in this PR? - `build_in_memory` concatenates once, then picks the map. `array_map_key_range` gets the single concatenated batch, and the key arrays are evaluated once for both paths. - The hash map is filled by walking zero-copy slices of the concatenated batch from the last slice to the first, `max_batch_rows` rows at a time. Rows are still inserted from the highest index down, so the chains, and therefore the join output, are identical. `hashes_buffer` and the prepared scratch reservation stay bounded by `max_batch_rows`. An empty build is guarded (`max_batch_rows` is 0 there). - The `HashJoinExec` doc and diagram describe the new build. ## What is the testing strategy for this PR? Existing order tests still pass unchanged (`join_left_anti_keeps_build_order_across_batches`, `join_inner_one_two_parts_left_randomly_ordered`, `prepared_build_matches_collect_left`). New tests: - `join_keeps_build_order_across_uneven_batches`: build batches of 2, 3 and 2 rows, so the slices do not line up with the batches, with duplicate and NULL keys, Inner and LeftAnti, under both null equalities, over the hash map and array map configs. - `join_build_side_without_batches`: an empty build on the hash map path. - `join_build_concat_is_reserved` now checks the lower peak per map type: limits that fit the new order but not the old one, plus a case that fails at the copy (array map) or at the map once the inputs are gone (hash map). These fail on the previous code. ## Are there any user-facing changes? No. Lower build-side peak memory for in-memory hash join builds. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
