Github user nsyca commented on the issue:
https://github.com/apache/spark/pull/16281
My two cents:
- Do we have a Parquet specific test suite **with sufficient coverage** to
run and back us up that this upgrade won't cause any regressions? I think
simply moving up the version of the jar files is a risky act. This practice of
doing (sort of) integration test will gain the confidence of our user community
that they can count on Spark to exercise its due diligence when it changes the
versions of any third party modules Spark runs on. Yes, the activity comes with
a cost. We can always define how much we can test and we want to test.
- On the topic of forking, it is a judgment call. It's a balance of having
a full control on the dependent third party modules but deviating from their
origin versus doing little work on our end but risking any contamination. In
the world of interdependence and interconnect, my opinion leans towards "good
fences make good neighbours." That comes back to my first point, we need to
have a good test coverage to gauge the impact of an upgrade to Spark.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
