adriangb opened a new pull request, #27092: URL: https://github.com/apache/datafusion/pull/27092
## Which issue does this PR close? - This PR does not close an issue. It follows up on https://github.com/apache/datafusion/pull/21666, which added canceled partitions to the hash join dynamic filter to fix https://github.com/apache/datafusion/issues/21625. > [!NOTE] > Stacked on https://github.com/apache/datafusion/pull/27090 and https://github.com/apache/datafusion/pull/27091. Review only the last three commits, starting at `test: cover signed zero and composite nullable keys in RangeRouter`. ## Rationale for this change A `Partitioned` hash join cancels one of its partitions when a parent operator drops that partition's stream before it reports its build side. For example, a parent join partition whose own build side is empty drops its probe-side stream unpolled. The dynamic filter keeps every probe row routed to a canceled partition, because that partition's build keys are unknown. Keeping those rows buys nothing. A stream starts to probe only after its report is delivered, so a canceled stream has emitted no row and never will. The shared probe input still reads the rows routed to it, and `RepartitionExec` then discards them. Keeping them also costs something: - A canceled partition forces the routed `CASE` fallback instead of the merged lookup. After https://github.com/apache/datafusion/pull/27091 it is the only thing that does. - Under `NullEqualsNull`, a canceled partition makes every live partition keep probe-side NULL rows. ## What changes are included in this PR? Three commits: 1. `test:` Two dynamic filter tests checked range routing for `-0.0` and for composite nullable keys through a canceled partition's `CASE` branch. The commit moves both cases into `RangeRouter` tests, before the next commit removes that branch. 2. `fix:` A canceled partition counts as empty, so the filter rejects the rows routed to it. The commit rewrites the tests that asserted the old behavior. 3. `refactor:` The `CASE` fallback has no input left. The commit removes it, the per-partition filters it used, and the test assertions that a filter is not a `CASE`. No behavior changes. Over https://github.com/apache/datafusion/pull/27091, this is +76 / −325 lines. ## What is the testing strategy for this PR? - The new `RangeRouter` tests keep the routing cases that the removed tests covered. - `partitioned_canceled_and_empty_partitions_reject_every_row` and `partitioned_canceled_partition_does_not_widen_filter` cover the new behavior. - `test_partitioned_dynamic_filter_reports_empty_canceled_partitions`, the regression test for https://github.com/apache/datafusion/issues/21625, still passes. - All `datafusion-physical-plan` tests, all sqllogictests, and `core_integration` pass. I also checked the behavior on a query that cancels partitions. In `SELECT x.k FROM (SELECT big.k FROM mid JOIN big ON mid.k = big.k) x WHERE x.k IN (SELECT k FROM tiny)` with 4 partitions, the parent `RightSemi` join cancels two child partitions. The result is the same with and without this PR. ## Are there any user-facing changes? No. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
