adriangb commented on code in PR #22698:
URL: https://github.com/apache/datafusion/pull/22698#discussion_r4026806646
##########
datafusion/physical-plan/src/filter.rs:
##########
@@ -98,6 +99,10 @@ pub struct FilterExec {
batch_size: usize,
/// Number of rows to fetch
fetch: Option<usize>,
+ /// Measurements and trial verdicts shared by all partition streams,
+ /// used by adaptive conjunct reordering (see [`AdaptiveConjunction`]) so
+ /// the streams learn as one. Fresh per plan node; never affects the plan.
+ adaptive_stats: Arc<AdaptiveFilterShared>,
Review Comment:
What do you mean by "ordinary or repeated concurrent executions"? If
multiple partitions then yes, that is by design: `FilterExec::execute()` gets
called once per partition, and we _want_ to track stats across partitions, not
just within one partition.
The whole point of the `reset_state()` API is that any repeated execution
that wants to be isolated should call it. If it does not, that's a bug with
that execution, not our implementation. If you have a concrete case that we
should test I'm happy to investigate more and add tests.
Finally, unlike dynamic filters and some other execution plan states, there
is no correctness risk here. Even if we got it _completely wrong_ the worst
thing we can cause is suboptimal evaluation order for filters, not incorrect
results.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
