sunchao commented on PR #7361: URL: https://github.com/apache/datafusion-comet/pull/7361#issuecomment-6546598612
Thanks @andygrove. I addressed all three points in [13fd925bb5](https://github.com/apache/datafusion-comet/pull/7361/commits/13fd925bb51a73f88ea3d301fe390dd44241fb89). 1. The native planner test now builds `coalesce(k, 0)` and checks that the right sort reads from the reducer, the reducer reads from the projection, and the sort and join both use the projected `membership_key`. A computed Boolean key keeps the original sort over the scan. 2. The new Spark test joins against `coalesce(r._1, r._2)` on two nullable `INT` columns. Only left keys 5 and 7 match. All 24,576 right rows reach the reducer, including 8,192 with both columns NULL. The left-side zero also catches an accidental NULL-to-zero match. 3. If the planner cannot resolve the computed key’s type, it now skips the reduction. The ordinary native join planner still validates the join. Both native planner tests and all four membership reducer tests passed. All five focused join tests passed on Spark 3.5.9 and 4.1.3. I also refreshed the benchmark table in the PR description. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
