Thank you for the responses and guidance in this discussion. Based on the feedback so far it sounds like my next steps should be the following:
*Drop Spark 1 Support:* I have a patch in progress here <https://gerrit.cloudera.org/#/c/7690/> and we can pull it in when we think its the right time. Note that whenever we drop Spark 1 support, maintenance releases can still provide updates and bug fixes. *Deprecate Java 7 Support: * It sounds like we should have at least 1 more minor release where Java 7 support is documented as deprecated. After that we need to decide when to drop Java 7 (likely 2.0). *Upgrade to Spark 2.2:* This will include changes to use/require Java 8 for only kudu-spark and kudu-spark-tools modules. Thank you, Grant On Wed, Aug 16, 2017 at 10:40 AM, Jean-Daniel Cryans <[email protected]> wrote: > Right, and I think this is where the disagreement is, and where SemVer > isn't helping us much, in that the change is source compatible but not > binary compatible for JDK 7. The more I think about it the more I'm fine > with it, I guess it's a quirk of using the JVM. > > J-D > > On Wed, Aug 16, 2017 at 8:25 AM, Mark Hamstra <[email protected]> > wrote: > > > Java 7 support was deprecated in Spark 2.0.0 and documented as such in > the > > release notes. The removal of Java 7 support does not introduce a > > source-level backwards incompatibility in the public Spark API. > > > > On Wed, Aug 16, 2017 at 8:20 AM, Jean-Daniel Cryans <[email protected] > > > > wrote: > > > > > Hi Mark, > > > > > > Thank you for your insight in Spark, we're obviously missing such > > expertise > > > and this has led us to make some mistakes. > > > > > > Perusing the documentation, I only see obvious deprecation notices in > > 2.1.0 > > > after SPARK-18138 was pushed. > > > > > > Nevertheless, I think Dan's interpretation is that the major version > must > > > be incremented if a backward incompatible change is made. Spark 2.1.0 > > could > > > run on JDK 7, 2.2.0 requires JDK 8, so what used to work doesn't > anymore. > > > We're far from public APIs, but I can relate to his point of view. > > > > > > On the bright side, I don't think this is putting Kudu in a bad spot. > If > > we > > > upgrade kudu-spark's jar to require JDK 8 then we can also limit this > > > requirement to that module so that users of other modules (like > > > kudu-client) can still use it with JDK 7. This means kudu-spark users > > have > > > to make sure they're on JDK 8, but they'd have to do that regardless if > > > they want to use Spark 2.2.0. All we need is some more lines in our pom > > > files. > > > > > > Thanks, > > > > > > J-D > > > > > > On Wed, Aug 16, 2017 at 7:50 AM, Mark Hamstra <[email protected] > > > > > wrote: > > > > > > > And? > > > > > > > > Not only was the change documented, but there was more than one minor > > > > release with the deprecation in place before the removal of Java 7 > and > > > > Scala 2.10 support in the new major release. Java 7 and Scala 2.10 > have > > > > never been anything but deprecated functionality in the Spark 2 API. > It > > > is > > > > just not the Spark PMC's fault if you chose not to follow that > > > deprecation > > > > guidance. > > > > > > > > > On Aug 15, 2017, at 8:27 PM, Dan Burkert <[email protected]> > > > wrote: > > > > > > > > > > Hi Mark, > > > > > > > > > > On Tue, Aug 15, 2017 at 6:49 PM, Mark Hamstra < > > [email protected] > > > > > > > > > wrote: > > > > > > > > > >> You are badly mistaken > > > > > > > > > > > > > > > My interpretation of SemVer above is based on the definition at > > > > SemVer.org, > > > > > which has this to say about when it's appropriate to remove > > deprecated > > > > > functionality: > > > > > > > > > >> When you deprecate part of your public API, you should do two > > things: > > > > (1) > > > > > update your documentation to let users know about the change, (2) > > > issue a > > > > > new minor release with the deprecation in place. Before you > > completely > > > > > remove the functionality in a new major release there should be at > > > least > > > > > one minor release that contains the deprecation so that users can > > > > smoothly > > > > > transition to the new API. > > > > > > > > > > Also relevant, from the same source: > > > > > > > > > >> Major version X (X.y.z | X > 0) MUST be incremented if any > backwards > > > > > incompatible changes are introduced to the public API. > > > > > > > > > > - Dan > > > > > > > > > > > > > > >> > > > > >> On Tue, Aug 15, 2017 at 2:18 PM, Dan Burkert < > [email protected] > > > > > > > >> wrote: > > > > >> > > > > >>> I'll preface my response by saying I don't think there are any > hard > > > and > > > > >>> fast rules here, but I'd like us to try > > > > >>> and continue following SemVer rules as much as possible. > > > > >>> > > > > >>> On Tue, Aug 15, 2017 at 2:03 PM, Grant Henke < > [email protected]> > > > > >> wrote: > > > > >>>> > > > > >>>> > > > > >>>> - Should/can we drop Spark 1 support in the next minor > release? > > > > >>>> > > > > >>> > > > > >>> My interpretation is that it's permissible to stop shipping > > releases > > > of > > > > >> an > > > > >>> artifact at any point (in this case kudu-spark1_2.10), > > > > >>> so I'm all for dropping Spark 1 as soon as we feel there are a > > > > >> sufficiently > > > > >>> low number of users. > > > > >>> > > > > >>> > > > > >>>> - Should/can we drop Java 7 support in the next minor release? > > > Does > > > > >> it > > > > >>>> need to be a major release? > > > > >>>> > > > > >>> > > > > >>> My interpretation of SemVer is that we can't drop JRE 7 support > > > > without a > > > > >>> major version bump. That being said, > > > > >>> I do think we're quickly approaching the time in which it would > be > > > > >>> appropriate to make this step. > > > > >>> > > > > >>> > > > > >>>> - How should we support Spark 2.2.0 if we don't drop Java 7? > > > Should > > > > >> we > > > > >>>> only require Java 1.8 for the Spark 2 modules? > > > > >>>> > > > > >>> > > > > >>> Spark has put us in a difficult position here - either > > > kudu-spark2_2.11 > > > > >>> remains JRE 7 compatible > > > > >>> and is capped at Spark 2.1, or we make an exception for > > > > kudu-spark2_2.11, > > > > >>> drop > > > > >>> JRE 7 compatibility, and continue floating the Spark version > > against > > > > the > > > > >>> latest 2.x release. I think given the > > > > >>> velocity of the Spark project and the fact that Spark itself > > doesn't > > > > seem > > > > >>> to have any qualms about > > > > >>> breaking SemVer, we should do the latter. > > > > >>> > > > > >>> > > > > >>>> -- > > > > >>>> Grant Henke > > > > >>>> Software Engineer | Cloudera > > > > >>>> [email protected] | twitter.com/gchenke | > > > linkedin.com/in/granthenke > > > > >>>> > > > > >>> > > > > >> > > > > > > > > > > -- Grant Henke Software Engineer | Cloudera [email protected] | twitter.com/gchenke | linkedin.com/in/granthenke
