Should this be considered a backwards-incompatible change? If so, do we need to wait until Beam 3.0.0 is released?
On Tue, Oct 17, 2017 at 9:11 AM, Ismaël Mejía <ieme...@gmail.com> wrote: > I am bringing the subject to the user mailing list to get some > feedback because it makes sense anyway to discuss this there. But I > also agree with Kenneth about the fact that runner authors must weight > in on this subject. > > > On Tue, Oct 17, 2017 at 5:24 PM, Kenneth Knowles <k...@google.com.invalid> > wrote: > > +1 to having runner maintainers weigh in as proxies. Added a few in case > > they haven't followed this thread. > > > > On Mon, Oct 16, 2017 at 11:38 PM, Eugene Kirpichov < > > kirpic...@google.com.invalid> wrote: > > > >> Agreed that polling Dataflow users makes sense, though I think they are > >> very unlikely to have concerns, because unlike Spark/Flink users they > >> wouldn't have a "cluster" that they need to migrate to a new JVM: they'd > >> only need to recompile their pipelines with JDK 8. > >> > >> On Mon, Oct 16, 2017 at 11:21 PM Reuven Lax <re...@google.com.invalid> > >> wrote: > >> > >> > I think the Flink and Spark runner maintainers can weigh in here; > given > >> > that both of those systems are moving to Java 8, I doubt they will > have > >> > concerns. Same is true for the Dataflow runner - we should probably > poll > >> > Dataflow users to make sure this is not a problem for any of them. > >> > > >> > On Mon, Oct 16, 2017 at 11:05 PM, Eugene Kirpichov < > >> > kirpic...@google.com.invalid> wrote: > >> > > >> > > Reuven - do you mean e.g. a poll on the Flink mailing list asking > >> whether > >> > > there are Flink users who use Beam with Java 7? Or just people who > use > >> > > Flink with Java 7? (the latter question I'd assume was settled by > the > >> > poll > >> > > about making Flink itself Java8-only?) > >> > > > >> > > On Mon, Oct 16, 2017 at 10:32 PM Reuven Lax > <re...@google.com.invalid> > >> > > wrote: > >> > > > >> > > > I don't know if a vote in @user is sufficient, as it's > unfortunately > >> > not > >> > > > representative of all Beam users. I think the runner communities > >> should > >> > > be > >> > > > polled as well (though I suspect the answer will be the same, > that we > >> > can > >> > > > go ahead and deprecate Java 7 support). > >> > > > > >> > > > On Mon, Oct 16, 2017 at 4:50 PM, Eugene Kirpichov < > >> > > > kirpic...@google.com.invalid> wrote: > >> > > > > >> > > > > Yeah, a vote on user@ sounds like a good idea. Ismaël, would > you > >> be > >> > > > > interested in driving this process, since you're already > working on > >> > > Java9 > >> > > > > support and hence you have a good understanding of what's > involved > >> > in a > >> > > > JDK > >> > > > > version migration for a large project? > >> > > > > > >> > > > > As due diligence, we can look at how the other data processing > >> > systems > >> > > > > handled dropping Java7. > >> > > > > > >> > > > > Flink: > >> > > > > JIRA https://issues.apache.org/jira/browse/FLINK-7242 > >> > > > > Discussion > >> > > > > http://apache-flink-user-mailing-list-archive.2336050. > >> > > > > n4.nabble.com/POLL-Who-still-uses-Java-7-with-Flink- > td12216.html > >> > > > > > >> > > > > They also did a Twitter poll in addition to the mailing list > poll, > >> > > which > >> > > > > seems like a good idea. > >> > > > > Note that Flink had a number of strong reasons for dropping > Java7 > >> > that > >> > > do > >> > > > > not apply in Beam. > >> > > > > > >> > > > > Spark: > >> > > > > JIRA https://issues.apache.org/jira/browse/SPARK-19493 > >> > > > > Discussion > >> > > > > > >> > > > http://apache-spark-developers-list.1001551.n3. > >> > > nabble.com/discuss-ending- > >> > > > > support-for-Java-7-in-Spark-2-0-td16808.html > >> > > > > (I > >> > > > > couldn't find a formal poll of the user list rather than > developer > >> > > list) > >> > > > > > >> > > > > Hadoop: > >> > > > > Hadoop 3.0 is Java8-only, but I couldn't quickly find a > discussion > >> of > >> > > > where > >> > > > > that decision was made. > >> > > > > https://lists.apache.org/thread.html/e5c8085ada2cca47027b63f > >> 5439839 > >> > > > > 731a392335770386e10895be06@1444091751@%3Cmapreduce-dev. > >> > > > > hadoop.apache.org%3E > >> > > > > might > >> > > > > be it. > >> > > > > > >> > > > > Kafka is considering it: > >> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > >> > > > > 118%3A+Drop+Support+for+Java+7+in+Kafka+0.11 > >> > > > > and > >> > > > > quotes a number of other open-source projects that have switched > >> > > > > http://markmail.org/message/l7s276y3xkga2eqf > >> > > > > > >> > > > > So basically these projects all did a mailing list poll, and one > >> did > >> > > > also a > >> > > > > twitter poll. > >> > > > > > >> > > > > Beam has the advantage of being a relatively young project with > >> > > perhaps a > >> > > > > smaller base of users entrenched in using old versions of Java; > >> > > moreover, > >> > > > > Java version would matter only for the smaller subset of users > who > >> > use > >> > > > Beam > >> > > > > Spark/Flink/Apex/.. runners (as opposed to Cloud Dataflow), > which > >> is > >> > > > likely > >> > > > > an even more "early adopter"-ish group of users, as these > runners > >> > > > generally > >> > > > > receive less support. > >> > > > > > >> > > > > It may be a good idea to have at least 1 release pass between > >> > > announcing > >> > > > > the intention to drop Java8 and actually dropping it (e.g. if we > >> > > decided > >> > > > it > >> > > > > now, then 2.4 would drop Java7). Also, we could start by > switching > >> > > tests > >> > > > to > >> > > > > compile/run with java8 (Maven allows this). This is, I think, > >> pretty > >> > > much > >> > > > > safe to do immediately. > >> > > > > > >> > > > > On Mon, Oct 16, 2017 at 7:35 AM Ismaël Mejía <ieme...@gmail.com > > > >> > > wrote: > >> > > > > > >> > > > > > Any progress on this? What is the proposed way to validate if > >> users > >> > > > > > are still interested on Java 7? A vote on user or something > >> > > different? > >> > > > > > > >> > > > > > > >> > > > > > On Wed, Sep 27, 2017 at 7:59 PM, Kenneth Knowles > >> > > > <k...@google.com.invalid > >> > > > > > > >> > > > > > wrote: > >> > > > > > > Agree with polling Beam users as well as each runner's > >> community > >> > in > >> > > > > > > aggregate. > >> > > > > > > > >> > > > > > > On Wed, Sep 27, 2017 at 9:44 AM, Jean-Baptiste Onofré < > >> > > > j...@nanthrax.net > >> > > > > > > >> > > > > > > wrote: > >> > > > > > > > >> > > > > > >> Definitely agree > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> On 09/27/2017 06:00 PM, Robert Bradshaw wrote: > >> > > > > > >> > >> > > > > > >>> I also think that it's time to seriously consider dropping > >> > > support > >> > > > > for > >> > > > > > >>> Java 7. > >> > > > > > >>> > >> > > > > > >>> On Tue, Sep 26, 2017 at 9:14 PM, Daniel Oliveira > >> > > > > > >>> <danolive...@google.com.invalid> wrote: > >> > > > > > >>> > >> > > > > > >>>> Yes, just as Ismaël said it's a compilation blocker right > >> now > >> > > > > despite > >> > > > > > >>>> that > >> > > > > > >>>> (I believe) we don't use the extension that's breaking. > >> > > > > > >>>> > >> > > > > > >>>> As for other ways to solve this, if there is a way to > avoid > >> > > > > compiling > >> > > > > > the > >> > > > > > >>>> advanced features of AutoValue that might be worth a > try. We > >> > > could > >> > > > > > also > >> > > > > > >>>> try > >> > > > > > >>>> to get a release of AutoValue with the fix that works in > >> Java > >> > 7. > >> > > > > > However > >> > > > > > >>>> I > >> > > > > > >>>> feel that slowly moving over to Java 8 is the most > >> > future-proof > >> > > > > > solution > >> > > > > > >>>> if > >> > > > > > >>>> it's possible. > >> > > > > > >>>> > >> > > > > > >>>> On Tue, Sep 26, 2017 at 2:47 PM, Ismaël Mejía < > >> > > ieme...@gmail.com> > >> > > > > > wrote: > >> > > > > > >>>> > >> > > > > > >>>> The current issue is that compilation fails on master > >> because > >> > > > beam's > >> > > > > > >>>>> parent pom is configured to fail if it finds warnings): > >> > > > > > >>>>> > >> > > > > > >>>>> <compiler.error.flag>-Werror</compiler.error.flag> > >> > > > > > >>>>> > >> > > > > > >>>>> However if you remove that line from the parent pom the > >> > > > compilation > >> > > > > > >>>>> passes. > >> > > > > > >>>>> > >> > > > > > >>>>> Of course this does not mean that everything is solved > for > >> > Java > >> > > > 9, > >> > > > > > >>>>> there are some tests that break and other issues > because of > >> > > other > >> > > > > > >>>>> plugins and dependencies (e.g. bytebuddy), but those are > >> not > >> > > part > >> > > > > of > >> > > > > > >>>>> this discussion. > >> > > > > > >>>>> > >> > > > > > >>>>> On Tue, Sep 26, 2017 at 11:38 PM, Eugene Kirpichov > >> > > > > > >>>>> <kirpic...@google.com.invalid> wrote: > >> > > > > > >>>>> > >> > > > > > >>>>>> AFAIK we don't use any advanced capabilities of > AutoValue. > >> > > Does > >> > > > > that > >> > > > > > >>>>>> mean > >> > > > > > >>>>>> this issue is moot? I didn't quite understand from your > >> > email > >> > > > > > whether > >> > > > > > >>>>>> it > >> > > > > > >>>>>> > >> > > > > > >>>>> is > >> > > > > > >>>>> > >> > > > > > >>>>>> a compilation blocker for Beam or not. > >> > > > > > >>>>>> > >> > > > > > >>>>>> On Tue, Sep 26, 2017 at 2:32 PM Ismaël Mejía < > >> > > ieme...@gmail.com > >> > > > > > >> > > > > > >>>>>> wrote: > >> > > > > > >>>>>> > >> > > > > > >>>>>> Great that you are also working on this too Daniel and > >> > thanks > >> > > > for > >> > > > > > >>>>>>> bringing this subject to the mailing list, I was > waiting > >> to > >> > > my > >> > > > > > return > >> > > > > > >>>>>>> to office next week, but you did it first :) > >> > > > > > >>>>>>> > >> > > > > > >>>>>>> Eugene for reference (This is the issue on the > migration > >> to > >> > > > Java > >> > > > > > 9), > >> > > > > > >>>>>>> notice that here the goal is first that beam passes > mvn > >> > clean > >> > > > > > install > >> > > > > > >>>>>>> with pure Java 9 (and also add this to jenkins), not > to > >> > > rewrite > >> > > > > > >>>>>>> anything to use the new stuff (e.g. modules): > >> > > > > > >>>>>>> https://issues.apache.org/jira/browse/BEAM-2530 > >> > > > > > >>>>>>> > >> > > > > > >>>>>>> Eugene can you also PTAL at the AutoValue issue, more > >> > details > >> > > > on > >> > > > > > the > >> > > > > > >>>>>>> issue, this is a warning so I don't know if it is > really > >> > > > critical > >> > > > > > in > >> > > > > > >>>>>>> particular because we are not using Memoization (do > we?). > >> > > > > > >>>>>>> https://github.com/google/auto/issues/503 > >> > > > > > >>>>>>> > >> > > > > > >>>>>>> > >> > > > > > > >> > https://github.com/google/auto/commit/71514081f2ca6fb4ead2b7f0a25f5d > >> > > > > > >>>>>>> > >> > > > > > >>>>>> 02247b8532 > >> > > > > > >>>>> > >> > > > > > >>>>>> > >> > > > > > >>>>>>> Wouldn't the easiest way be that you guys convince the > >> > > > > google.auto > >> > > > > > >>>>>>> guys to generate that simple fix in a Java 7 > compatible > >> way > >> > > and > >> > > > > > >>>>>>> 'voila' ? > >> > > > > > >>>>>>> > >> > > > > > >>>>>>> However I agree that moving to Java 8 is an excellent > >> idea > >> > > and > >> > > > as > >> > > > > > >>>>>>> Eugene mentions there is less friction now since most > >> > > projects > >> > > > > are > >> > > > > > >>>>>>> moving, only pending issue are existing clusters on > java > >> 7 > >> > in > >> > > > the > >> > > > > > >>>>>>> hadoop world, but those are less frequent now. Anyway > >> this > >> > > > > > discussion > >> > > > > > >>>>>>> is really important (maybe even worth a vote). Because > >> > moving > >> > > > to > >> > > > > > Java > >> > > > > > >>>>>>> 8 will allow us also to move some of the dependencies > >> that > >> > we > >> > > > are > >> > > > > > >>>>>>> keeping in old versions and in general to move > forward. > >> > > > > > >>>>>>> > >> > > > > > >>>>>>> What do the others think ? > >> > > > > > >>>>>>> > >> > > > > > >>>>>>> > >> > > > > > >>>>>>> > >> > > > > > >>>>>>> On Tue, Sep 26, 2017 at 11:09 PM, Eugene Kirpichov > >> > > > > > >>>>>>> <kirpic...@google.com.invalid> wrote: > >> > > > > > >>>>>>> > >> > > > > > >>>>>>>> Very excited to hear that there's work on JDK9 > support - > >> > is > >> > > > > there > >> > > > > > a > >> > > > > > >>>>>>>> > >> > > > > > >>>>>>> public > >> > > > > > >>>>>>> > >> > > > > > >>>>>>>> description of the plans for this work somewhere? > >> > > > > > >>>>>>>> > >> > > > > > >>>>>>>> In general, Beam could probably drop Java7 support > >> > > altogether > >> > > > at > >> > > > > > some > >> > > > > > >>>>>>>> > >> > > > > > >>>>>>> point > >> > > > > > >>>>>>> > >> > > > > > >>>>>>>> soon: Java7 has reached end-of-life (i.e. it's not > >> > receiving > >> > > > > even > >> > > > > > >>>>>>>> > >> > > > > > >>>>>>> security > >> > > > > > >>>>>>> > >> > > > > > >>>>>>>> patches) 2 years ago, and all major players in the > data > >> > > > > processing > >> > > > > > >>>>>>>> ecosystem have dropped Java7 support (Spark, Flink, > >> > Hadoop), > >> > > > so > >> > > > > I > >> > > > > > >>>>>>>> > >> > > > > > >>>>>>> presume > >> > > > > > >>>>> > >> > > > > > >>>>>> the demand for Java7 support in the data processing > >> industry > >> > > is > >> > > > > low. > >> > > > > > >>>>>>>> > >> > > > > > >>>>>>> By > >> > > > > > >>>>> > >> > > > > > >>>>>> the > >> > > > > > >>>>>>> > >> > > > > > >>>>>>>> way: would a Java8 migration be in the scope of your > >> work > >> > in > >> > > > > > general? > >> > > > > > >>>>>>>> > >> > > > > > >>>>>>>> However, until we say that Beam requires Java8, what > >> would > >> > > be > >> > > > > the > >> > > > > > >>>>>>>> implications of using a version of AutoValue that can > >> only > >> > > be > >> > > > > > >>>>>>>> compiled > >> > > > > > >>>>>>>> > >> > > > > > >>>>>>> with > >> > > > > > >>>>>>> > >> > > > > > >>>>>>>> Java8? Are you saying that this is simply a matter > of a > >> > > > compiler > >> > > > > > bug, > >> > > > > > >>>>>>>> > >> > > > > > >>>>>>> and > >> > > > > > >>>>> > >> > > > > > >>>>>> if we use a Java8 compiler but configured to use source > >> and > >> > > > target > >> > > > > > >>>>>>>> > >> > > > > > >>>>>>> versions > >> > > > > > >>>>>>> > >> > > > > > >>>>>>>> of 1.7 and using bootclasspath of rt.jar from 1.7, > then > >> > the > >> > > > > > resulting > >> > > > > > >>>>>>>> > >> > > > > > >>>>>>> Beam > >> > > > > > >>>>>>> > >> > > > > > >>>>>>>> artifacts will be usable by people who don't have > Java8? > >> > > > > > >>>>>>>> > >> > > > > > >>>>>>>> On Tue, Sep 26, 2017 at 1:53 PM Daniel Oliveira > >> > > > > > >>>>>>>> <danolive...@google.com.invalid> wrote: > >> > > > > > >>>>>>>> > >> > > > > > >>>>>>>> So I've been working on JDK 9 support for Beam, and I > >> > have a > >> > > > bug > >> > > > > > in > >> > > > > > >>>>>>>>> AutoValue that can be fixed by updating our > AutoValue > >> > > > > dependency > >> > > > > > to > >> > > > > > >>>>>>>>> > >> > > > > > >>>>>>>> the > >> > > > > > >>>>> > >> > > > > > >>>>>> latest. The problem is that AutoValue from 1.5+ seems > to > >> be > >> > > > banned > >> > > > > > >>>>>>>>> > >> > > > > > >>>>>>>> for > >> > > > > > >>>>> > >> > > > > > >>>>>> Beam > >> > > > > > >>>>>>> > >> > > > > > >>>>>>>> due to requiring Java 8 compilers. However, it should > >> > still > >> > > be > >> > > > > > >>>>>>>>> > >> > > > > > >>>>>>>> possible > >> > > > > > >>>>> > >> > > > > > >>>>>> to > >> > > > > > >>>>>>> > >> > > > > > >>>>>>>> compile and execute Java 7 code from the Java 8 > compiler > >> > by > >> > > > > > building > >> > > > > > >>>>>>>>> > >> > > > > > >>>>>>>> with > >> > > > > > >>>>>>> > >> > > > > > >>>>>>>> the correct arguments. So the fix to this bug would > >> > > > essentially > >> > > > > > >>>>>>>>> > >> > > > > > >>>>>>>> require > >> > > > > > >>>>> > >> > > > > > >>>>>> Java 8 compilers even for compiling Java 7 code. > >> > > > > > >>>>>>>>> > >> > > > > > >>>>>>>>> Does anyone need to use Java 7 compilers? Because if > >> not > >> > I > >> > > > > would > >> > > > > > >>>>>>>>> > >> > > > > > >>>>>>>> like to > >> > > > > > >>>>> > >> > > > > > >>>>>> continue with this fix. > >> > > > > > >>>>>>>>> > >> > > > > > >>>>>>>>> > >> > > > > > >>>>>>> > >> > > > > > >>>>> > >> > > > > > >> -- > >> > > > > > >> Jean-Baptiste Onofré > >> > > > > > >> jbono...@apache.org > >> > > > > > >> http://blog.nanthrax.net > >> > > > > > >> Talend - http://www.talend.com > >> > > > > > >> > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> >
