ozankabak commented on code in PR #5322:
URL: https://github.com/apache/arrow-datafusion/pull/5322#discussion_r1120399841
##########
datafusion/core/src/physical_optimizer/pipeline_fixer.rs:
##########
@@ -182,13 +289,46 @@ fn apply_subrules_and_check_finiteness_requirements(
physical_optimizer_subrules: &Vec<Box<PipelineFixerSubrule>>,
) -> Result<Option<PipelineStatePropagator>> {
for sub_rule in physical_optimizer_subrules {
- if let Some(value) = sub_rule(&input).transpose()? {
+ if let Some(value) = sub_rule(input.clone()).transpose()? {
input = value;
}
}
check_finiteness_requirements(input)
}
Review Comment:
We have made significant progress in foundational streaming execution
support on the Datafusion side over the last few months. For example, you can
already mark sources as infinite/unbounded (check out the tests with FIFO
files). Every `ExecutionPlan` then propagates this information upwards, so we
make adjustments to the plan in an intelligent way depending on whether we are
dealing with a stream.
We will write a blog post about this soon, I will share with you when it is
done so you can get up to speed on what we've done already. I think it would be
also good to have a meeting with everyone interested in this to discuss future
plans and designs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
