ctsk commented on code in PR #15301:
URL: https://github.com/apache/datafusion/pull/15301#discussion_r2022274853
##########
datafusion/physical-plan/src/topk/mod.rs:
##########
@@ -186,6 +235,90 @@ impl TopK {
Ok(())
}
+ fn calculate_dynamic_filters(
+ thresholds: Vec<ColumnThreshold>,
+ ) -> Result<Option<Arc<dyn PhysicalExpr>>> {
+ // Create filter expressions for each threshold
+ let mut filters: Vec<Arc<dyn PhysicalExpr>> =
Review Comment:
Been thinking about this too :D.
One could add another term to the generated expression:
```
passive OR <cur output of calculate_dynamic_filters>
```
`passive` itself is a dynamic literal that starts out as `true` and gets set
to `false` by the TopK Heap once the heap is full.
A downside of the DynamicLiteral approach is that it requires one mutex per
field (vs 1 mutex for the whole expression previously). For native types, this
*feels* like something that could be backed by an atomic, but I don't see an
easy way of achieving that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
