Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/20476
@gatorsmile, Do you mean this?
> Extensibility is not good and operator push-down capabilities are limited.
If so, that's very open to interpretation. I would assume it means that the
V2 interfaces should support more than just projection and filter push-down,
but not a redesign of how push-down happens in the optimizer. Even if it is
called out as a goal, I now see it as a misguided choice.
But either way, you make a good point about changing things for a release.
I'll defer to your judgement about what should be done for the release. But for
the long term, I think this issue underscores my point about reusing code that
already works. Let's separate DSv2 from a push-down redesign and get it working
reliably without introducing more risk and unknown problems.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
