milenkovicm opened a new pull request, #2776:
URL: https://github.com/apache/datafusion-ballista/pull/2776
# Which issue does this PR close?
Closes #.
# Rationale for this change
`ballista.planner.propagate_empty.enabled` is documented in the user guide
(default `true`) as the switch for the AQE propagate-empty-relation rule, but
`PropagateEmptyExecRule` never read it. Setting it to `false` had no effect.
This PR also reverts the workaround from #2062. It wrapped every
multi-partition `EmptyExec` in a `RepartitionExec: RoundRobinBatch(N)` before a
stage plan was serialized, because `datafusion-proto` used to drop the
partition count (apache/datafusion#23642). `EmptyExecNode` now has a
`partitions` field, and `empty_exec_partition_count_survives_datafusion_proto`
checks that the count survives the round trip.
# What changes are included in this PR?
- `PropagateEmptyExecRule::optimize` returns the plan unchanged when
`propagate_empty_enabled()` is `false`.
- Removed `make_empty_exec_serde_safe` from the scheduler's `planner.rs`,
along with its call in `create_shuffle_writer_with_config` and its unit tests.
- New tests in `state/aqe/test/alter_stages.rs`:
- `should_not_propagate_empty_stage_when_disabled` checks the stage plan
with the option set to `false` and then to `true`.
- `should_run_empty_stage_query_with_propagation_disabled` runs the query
end to end with the rule disabled and checks that the job succeeds.
- Updated the empty-stage snapshots, which are now `ShuffleWriterExec`
directly over `EmptyExec` without the round-robin wrapper.
# Are there any user-facing changes?
`ballista.planner.propagate_empty.enabled=false` now disables the rule, as
documented. The default is unchanged, and there are no public API or `.proto`
changes.
Executors built before `EmptyExecNode` gained its `partitions` field should
not be used with this scheduler. They would decode a multi-partition
`EmptyExec` as one partition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
