mbutrovich commented on PR #27399:
URL: https://github.com/apache/datafusion/pull/27399#issuecomment-6961888717

   I ran property-based tests from 
[arrow-pbt](https://github.com/mbutrovich/arrow-pbt) (mbutrovich/arrow-pbt#5) 
against this PR at dbc4514bb, with arrow-rs `main` at d65329d3e7. Every join 
test passed seeds 1 to 6 at 1,000 cases each, including [one that compares 
`enable_migration_hash_join` on and 
off](https://github.com/mbutrovich/arrow-pbt/blob/2c4a66ab8d645443a5157e44936430cb632c54e4/arrow-pbt-datafusion/tests/joins.rs#L147-L158).
   
   That test runs the same `HashJoinExec` with the option on and off, and 
[checks](https://github.com/mbutrovich/arrow-pbt/blob/2c4a66ab8d645443a5157e44936430cb632c54e4/arrow-pbt-datafusion/src/join.rs#L490-L558)
 that both fail, or both return the same output types and the same rows in any 
order, with every output column valid under the Arrow format. Two more tests 
compare the hash join with a `NestedLoopJoinExec` whose filter is the residual 
filter and one `=` or `IS NOT DISTINCT FROM` per key, and the perfect hash join 
(`ArrayMap`) with the general hash map.
   
   The 
[cases](https://github.com/mbutrovich/arrow-pbt/blob/2c4a66ab8d645443a5157e44936430cb632c54e4/arrow-pbt-datafusion/src/join.rs#L82-L162)
 cover every join type, both `NullEquality` values, a residual filter in half 
of them, and `CollectLeft` and `Partitioned` with one partition per side. Keys 
are one or two columns of 24 types, including dictionaries, views, structs, 
lists, and floats whose values include NaN, -0.0, and 0.0. Both sides draw each 
key column from one small pool of values, and `batch_size` is 1 to 8, 16, 63 to 
65, 1024, or 8192. In 31% of cases a probe row matches more build rows than 
`batch_size`, so its matches span chunks and reach 
[`last_probe_row_with_on_match`](https://github.com/apache/datafusion/blob/dbc4514bbd8710e384316282606d17a949989a50/datafusion/physical-plan/src/joins/hash_join/batched_join_processor.rs#L423-L435)
 for the joins that keep unmatched probe rows. That share depends only on the 
inputs. On `main` at e2dc8b048b, 4.2% of cases built an `Arr
 ayMap`.
   
   The comparison with the nested loop join also discarded 175 cases on three 
open arrow-rs bugs that the joins' non-key columns reach: 
apache/arrow-rs#11607, apache/arrow-rs#10925, and apache/arrow-rs#11835. In the 
`enable_migration_hash_join` test, a case where both paths fail on one of those 
bugs counts as agreement.
   
   To rerun it, check out the `datafusion-27399` branch of arrow-pbt, whose 
`Cargo.toml` points at this PR, and run:
   
   ```sh
   ARROW_PBT_SEEDS=1-6 ARROW_PBT_CASES=1000 cargo test -p arrow-pbt-datafusion 
--test joins
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to