Github user rdblue commented on the issue:
https://github.com/apache/spark/pull/16281
I don't think a fork is a good idea, nor do I think there is a reasonable
need for one.
@gatorsmile brought up that the Parquet community refused to build a patch
release: "The problem is the Parquet community will not create a branch 1.8.2+
for us." I don't remember this happening, and as far as I can tell from both
google and my inbox, the Parquet community never rejected the idea of patch
releases. If there was a conversation that I don't know about, then I apologize
that you were given the impression that patch releases aren't possible. That
isn't the case.
I'm happy to work with the community to put out patch releases, especially
if that's needed for Spark. To demonstrate, look at PARQUET-389 and
PARQUET-654. @rxin asked the Parquet dev list about predicate push-down
features and within a week and a half, both of those issues were resolved.
(PARQUET-389 is the fix for SPARK-18539, cited as motivation to fork.)
As for the other motivating issue, PARQUET-686, a fork can't help solve
this problem. This is an issue that requires updating the Parquet format spec
so you couldn't simply fix your own fork without abandoning compatibility. The
Parquet community put out a release that gives the user a choice between
correctness and performance, which is a good compromise until this can be fixed.
It is fair to point out that Parquet has not had a regular release cadence
for minor releases (1.8.1 to 1.9.0), which is something that the Parquet
community knows about and has discussed. We have recently committed to
quarterly releases to fix this, with patch releases whenever they are needed.
I'd encourage anyone interested to get involved.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
