Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/21319
@cloud-fan, what about adding support for v2 pushdown in the stats visitor
instead?
Here's the idea: when the visitor hits a `Filter` or a `Project`, it tries
to match the plan using `PhysicalOperation`. If that works and the underlying
relation is a `DataSourceV2Relation`, it does the pushdown to configure a
reader and return stats from it.
That would give us the ability to do pushdown on conversion to physical
plan, but we'd get correct stats. The only drawback is that we would
temporarily make the stats code a bit larger, but at least it is separate so we
can get the logical plans right and fix stats on a separate schedule.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
