asolimando commented on code in PR #19957:
URL: https://github.com/apache/datafusion/pull/19957#discussion_r2739145784
##########
datafusion/physical-expr/src/projection.rs:
##########
@@ -660,9 +660,25 @@ impl ProjectionExprs {
}
}
} else {
- // TODO stats: estimate more statistics from expressions
- // (expressions should compute their statistics themselves)
- ColumnStatistics::new_unknown()
+ // TODO: expressions should compute their own statistics
Review Comment:
I have done a POC for ExpressionAnalyzer, commit
[3d229b6f5](https://github.com/asolimando/datafusion/commit/3d229b6f5)
introduces it, it is a pluggable chain-of-responsibility for expression-level
statistics (selectivity, NDV, min/max). It includes default Selinger-style
estimation and built-in analyzers for string/math/datetime functions (that's
just an example to see how you could do it in practice, but it's also sound).
Commit
[d725cc08c](https://github.com/asolimando/datafusion/commit/d725cc08c)
demonstrates its use by integrating it into projection NDV estimation (where
the present discussion started, basically).
This is a non-breaking change, purely additive. The `StatisticsRegistry`
approach for operator-level statistics would require modifying
`partition_statistics` signatures and that would be breaking.
As I was saying, I think it's inevitable to adopt something similar in the
long run, but at what time and under what form can be discussed, if the
community is interested in exploring this direction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
