Github user mallman commented on the issue:
https://github.com/apache/spark/pull/16578
> I'm reluctant to generalize this PR without practical experience applying
it to other column-oriented file formats. The only format I'm familiar with and
have production experience with is Parquet.
I want to expand on what I wrote. A lot of this patch is already
generalized, e.g. the catalyst code. The tricky part is porting and testing the
file access code. While columnar formats operate under the same principles, the
devil is in the details, so to speak. Hence my reluctance to sign off on a
broad generalization of this patch to other file formats.
BTW, one thing that's occurred to me is the possibility of putting this
functionality behind a configuration setting for the first one or two releases
in which it exists. In the case of a bug we've overlooked, the end user can
disable the optimization. What do you think?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
