tschwarzinger commented on PR #27126:
URL: https://github.com/apache/datafusion/pull/27126#issuecomment-6690203291
Thanks @devanbenz !
I think you're observing what I meant with
> Possibly higher latency due to the delayed opening of files but I could
not measure a significant impact locally.
I've only tested tpch sf10 with filter pushdown, so thanks for pointing out
the regressions in the other benchmarks! It definitely makes sense as execution
is now more sequentially (as you nice diagram shows :) ). Maybe also doing a
run on TPC-DS with dynamic filters on would make sense.
The questions I guess is whether this is an intended design or just a
something that grew organically.
Also if there is only a single input partition we don't poll the stream
eagerly.
```rust
1 => {
// single-partition path: execute child directly, but ensure fetch is
respected
// (wrap with ObservedStream only if fetch is present so we don't add
overhead otherwise)
let child_stream = self.input.execute(0, context)?;
if self.fetch.is_some() {
let baseline_metrics = BaselineMetrics::new(&self.metrics,
partition);
return Ok(Box::pin(ObservedStream::new(
child_stream,
baseline_metrics,
self.fetch,
)));
}
Ok(child_stream)
}
```
It kind of seems to me that this is an unintended effect (I wasn't aware
that `CoalescePartitionsExec` does eager polling). Anyway, if we land something
like this we should figure out a way without regressing the queries.
Personally, I think it's kind of a surprise that `CoalescePartitionsExec`
does eager polling. Maybe the parallelization should happen in a different
operator? Maybe the hash joins should execute their build side in a task?
Any takes on this?
I will need to look at the queries though next week to see if there is a
good solution. Marking this as draft for now so we don't merge it by accident.
Thanks again!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
