Re: [DISCUSS] Moving branch-2 to java 8

[Jonathan Hung]

Hi Anu, we will configure precommit jobs to continue compiling on openjdk7.
If there's incompatible source changes then the precommit job will catch
this. The change proposed here is only for the *test* phase of branch-2
precommit executions (and branch-2 nightly job) to run on openjdk8 only.

Jonathan Hung


On Mon, Feb 4, 2019 at 10:45 AM Anu Engineer 
wrote:

> Konstantin,
>
> Just a nitpicky thought, if we move this branch to Java-8 on Jenkins, but
> still hope to release code that can run on Java 7, how will we detect
> Java 8 only changes? I am asking because till now whenever I checked in
> Java 8 features in branch-2 Jenkins would catch that issue.
>
> With this approach, we might not find it out the issues till the release
> time when the release manager decides to compile with Java 7.
> It might be more pragmatic to say that your Java 7 mileage may vary once
> this goes in, since we will have no visibility to Java 7 compatibility
> until it is too late.
>
> Another approach could be that we create a read-only 2.x branch, then we
> know that code will work with Java 7 since the last snapshot was known to
> work with Java 7.
>
>
> Thanks
> Anu
>
>
>
> On 2/1/19, 5:04 PM, "Konstantin Shvachko"  wrote:
>
> Just to make sure we are on the same page, as the subject of this
> thread is
> too generic and confusing.
> *The proposal is to move branch-2 Jenkins builds such as precommit to
> run
> tests on openJDK-8.*
> We do not want to break Java 7 source compatibility. The sources and
> releases will still depend on Java 7.
> We don't see test failures discussed in HADOOP-15711 when we run them
> locally with Oracle Java 7.
>
> Thanks,
> --Konst
>
> On Fri, Feb 1, 2019 at 12:44 PM Jonathan Hung 
> wrote:
>
> > Thanks Vinod and Steve, agreed about java7 compile compatibility. At
> least
> > for now, we should be able to maintain java7 source compatibility
> and run
> > tests on java8. There's a test run here:
> >
> https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86-jhung/46/
> > which calls a java8 specific API, installs both openjdk7/openjdk8 in
> the
> > dockerfile, compiles on both versions, and tests on just java8 (via
> >
> >
> --multijdkdirs=/usr/lib/jvm/java-7-openjdk-amd64,/usr/lib/jvm/java-8-openjdk-amd64
> > and --multijdktests=compile). If we eventually decide it's too much
> of a
> > pain to maintain java7 source compatibility we can do that at a later
> > point.
> >
> > Also based on discussion with others in the community at the
> contributors
> > meetup this past Wednesday, seems we are generally in favor of
> testing
> > against java8. I'll start a vote soon.
> >
> > Jonathan Hung
> >
> >
> > On Tue, Jan 29, 2019 at 4:11 AM Steve Loughran <
> ste...@hortonworks.com>
> > wrote:
> >
> > > branch-2 is the JDK 7 branch, but for a long time I (and presumably
> > > others) have relied on jenkins to keep us honest by doing that
> build and
> > > test
> > >
> > > right now, we can't do that any more, due to jdk7 bugs which will
> never
> > be
> > > fixed by oracle, or at least, not in a public release.
> > >
> > > If we can still do the compile in java 7 language and link to java
> 7 JDK,
> > > then that bit of the release is good -then java 8 can be used for
> that
> > test
> > >
> > > Ultimately, we're going to be forced onto java 8 just because all
> our
> > > dependencies have moved onto it, and some CVE will force us to
> move.
> > >
> > > At which point, I think its time to declare branch-2 dead. It's
> had a
> > > great life, but trying to keep java 7 support alive isn't
> sustainable.
> > Not
> > > just in this testing, but
> > > cherrypicking patches back gets more and more difficult -branch-3
> has
> > > moved on in both use of java 8 language, and in the codebase in
> general.
> > >
> > > > On 28 Jan 2019, at 20:18, Vinod Kumar Vavilapalli <
> vino...@apache.org>
> > > wrote:
> > > >
> > > > The community made a decision long time ago that we'd like to
> keep the
> > > compatibility & so tie branch-2 to Java 7, but do Java 8+ only
> work on
> > 3.x.
> > > >
> > > > I always assumed that most (all?) downstream users build
> branch-2 on
> > JDK
> > > 7 only, can anyone confirm? If so, there may be an easier way to
> address
> > > these test issues.
> > > >
> > > > +Vinod
> > > >
> > > >> On Jan 28, 2019, at 11:24 AM, Jonathan Hung <
> jyhung2...@gmail.com>
> > > wrote:
> > > >>
> > > >> Hi folks,
> > > >>
> > > >> Forking a discussion based on HADOOP-15711. To summarize, there
> are
> > > issues
> > > >> with branch-2 tests running on java 7 (openjdk) which don't
> exist on
> > > java
> > > >> 8. From our testing, the build can pass with openjdk 8.
> > > >>
>   

Re: [DISCUSS] Moving branch-2 to java 8

[Anu Engineer]

Konstantin, 

Just a nitpicky thought, if we move this branch to Java-8 on Jenkins, but still 
hope to release code that can run on Java 7, how will we detect
Java 8 only changes? I am asking because till now whenever I checked in Java 8 
features in branch-2 Jenkins would catch that issue.

With this approach, we might not find it out the issues till the release time 
when the release manager decides to compile with Java 7.
It might be more pragmatic to say that your Java 7 mileage may vary once this 
goes in, since we will have no visibility to Java 7 compatibility until it is 
too late.

Another approach could be that we create a read-only 2.x branch, then we know 
that code will work with Java 7 since the last snapshot was known to work with 
Java 7.


Thanks
Anu



On 2/1/19, 5:04 PM, "Konstantin Shvachko"  wrote:

Just to make sure we are on the same page, as the subject of this thread is
too generic and confusing.
*The proposal is to move branch-2 Jenkins builds such as precommit to run
tests on openJDK-8.*
We do not want to break Java 7 source compatibility. The sources and
releases will still depend on Java 7.
We don't see test failures discussed in HADOOP-15711 when we run them
locally with Oracle Java 7.

Thanks,
--Konst

On Fri, Feb 1, 2019 at 12:44 PM Jonathan Hung  wrote:

> Thanks Vinod and Steve, agreed about java7 compile compatibility. At least
> for now, we should be able to maintain java7 source compatibility and run
> tests on java8. There's a test run here:
> https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86-jhung/46/
> which calls a java8 specific API, installs both openjdk7/openjdk8 in the
> dockerfile, compiles on both versions, and tests on just java8 (via
>
> 
--multijdkdirs=/usr/lib/jvm/java-7-openjdk-amd64,/usr/lib/jvm/java-8-openjdk-amd64
> and --multijdktests=compile). If we eventually decide it's too much of a
> pain to maintain java7 source compatibility we can do that at a later
> point.
>
> Also based on discussion with others in the community at the contributors
> meetup this past Wednesday, seems we are generally in favor of testing
> against java8. I'll start a vote soon.
>
> Jonathan Hung
>
>
> On Tue, Jan 29, 2019 at 4:11 AM Steve Loughran 
> wrote:
>
> > branch-2 is the JDK 7 branch, but for a long time I (and presumably
> > others) have relied on jenkins to keep us honest by doing that build and
> > test
> >
> > right now, we can't do that any more, due to jdk7 bugs which will never
> be
> > fixed by oracle, or at least, not in a public release.
> >
> > If we can still do the compile in java 7 language and link to java 7 
JDK,
> > then that bit of the release is good -then java 8 can be used for that
> test
> >
> > Ultimately, we're going to be forced onto java 8 just because all our
> > dependencies have moved onto it, and some CVE will force us to move.
> >
> > At which point, I think its time to declare branch-2 dead. It's had a
> > great life, but trying to keep java 7 support alive isn't sustainable.
> Not
> > just in this testing, but
> > cherrypicking patches back gets more and more difficult -branch-3 has
> > moved on in both use of java 8 language, and in the codebase in general.
> >
> > > On 28 Jan 2019, at 20:18, Vinod Kumar Vavilapalli 
> > wrote:
> > >
> > > The community made a decision long time ago that we'd like to keep the
> > compatibility & so tie branch-2 to Java 7, but do Java 8+ only work on
> 3.x.
> > >
> > > I always assumed that most (all?) downstream users build branch-2 on
> JDK
> > 7 only, can anyone confirm? If so, there may be an easier way to address
> > these test issues.
> > >
> > > +Vinod
> > >
> > >> On Jan 28, 2019, at 11:24 AM, Jonathan Hung 
> > wrote:
> > >>
> > >> Hi folks,
> > >>
> > >> Forking a discussion based on HADOOP-15711. To summarize, there are
> > issues
> > >> with branch-2 tests running on java 7 (openjdk) which don't exist on
> > java
> > >> 8. From our testing, the build can pass with openjdk 8.
> > >>
> > >> For branch-3, the work to move the build to use java 8 was done in
> > >> HADOOP-14816 as part of the Dockerfile OS version change. 
HADOOP-16053
> > was
> > >> filed to backport this OS version change to branch-2 (but without the
> > java
> > >> 7 -> java 8 change). So my proposal is to also make the java 7 ->
> java 8
> > >> version change in branch-2.
> > >>
> > >> As mentioned in HADOOP-15711, the main issue is around source and
> binary
> > >> compatibility. I don't currently have a great answer, but one initial
> > >> thought is to build source/binary against java 7 to ensure
> compatibility
> > >> 

Re: [DISCUSS] Moving branch-2 to java 8

[Jonathan Hung]

Yeah, it's possible with yetus, there's one example here

which
runs compilation on openjdk7 (and openjdk8), and runs tests on openjdk8
only.

Jonathan Hung


On Mon, Feb 4, 2019 at 10:11 AM Steve Loughran 
wrote:

>
>
> On 2 Feb 2019, at 00:57, Konstantin Shvachko  wrote:
>
> Just to make sure we are on the same page, as the subject of this thread
> is too generic and confusing.
> *The proposal is to move branch-2 Jenkins builds such as precommit to run
> tests on openJDK-8.*
> We do not want to break Java 7 source compatibility. The sources and
> releases will still depend on Java 7.
> We don't see test failures discussed in HADOOP-15711 when we run them
> locally with Oracle Java 7.
>
> Thanks,
> --Konst
>
>
> Given the tests aren't working today, the risk that an openjdk 8 test run
> hides a problem which would show up on openjdk 7 has to consider that at
> least openjdk8 will run the tests.
>
> One thing I would like to be confident is that at least the compile phase
> of all the source (including generated source) is on jdk7, and its only the
> test run which switches JVM. Can we do that?
>

Re: [DISCUSS] Moving branch-2 to java 8

[Steve Loughran]

On 2 Feb 2019, at 00:57, Konstantin Shvachko 
mailto:shv.had...@gmail.com>> wrote:

Just to make sure we are on the same page, as the subject of this thread is too 
generic and confusing.
The proposal is to move branch-2 Jenkins builds such as precommit to run tests 
on openJDK-8.
We do not want to break Java 7 source compatibility. The sources and releases 
will still depend on Java 7.
We don't see test failures discussed in HADOOP-15711 when we run them locally 
with Oracle Java 7.

Thanks,
--Konst

Given the tests aren't working today, the risk that an openjdk 8 test run hides 
a problem which would show up on openjdk 7 has to consider that at least 
openjdk8 will run the tests.

One thing I would like to be confident is that at least the compile phase of 
all the source (including generated source) is on jdk7, and its only the test 
run which switches JVM. Can we do that?

Re: [DISCUSS] Moving branch-2 to java 8

[Konstantin Shvachko]

Just to make sure we are on the same page, as the subject of this thread is
too generic and confusing.
*The proposal is to move branch-2 Jenkins builds such as precommit to run
tests on openJDK-8.*
We do not want to break Java 7 source compatibility. The sources and
releases will still depend on Java 7.
We don't see test failures discussed in HADOOP-15711 when we run them
locally with Oracle Java 7.

Thanks,
--Konst

On Fri, Feb 1, 2019 at 12:44 PM Jonathan Hung  wrote:

> Thanks Vinod and Steve, agreed about java7 compile compatibility. At least
> for now, we should be able to maintain java7 source compatibility and run
> tests on java8. There's a test run here:
> https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86-jhung/46/
> which calls a java8 specific API, installs both openjdk7/openjdk8 in the
> dockerfile, compiles on both versions, and tests on just java8 (via
>
> --multijdkdirs=/usr/lib/jvm/java-7-openjdk-amd64,/usr/lib/jvm/java-8-openjdk-amd64
> and --multijdktests=compile). If we eventually decide it's too much of a
> pain to maintain java7 source compatibility we can do that at a later
> point.
>
> Also based on discussion with others in the community at the contributors
> meetup this past Wednesday, seems we are generally in favor of testing
> against java8. I'll start a vote soon.
>
> Jonathan Hung
>
>
> On Tue, Jan 29, 2019 at 4:11 AM Steve Loughran 
> wrote:
>
> > branch-2 is the JDK 7 branch, but for a long time I (and presumably
> > others) have relied on jenkins to keep us honest by doing that build and
> > test
> >
> > right now, we can't do that any more, due to jdk7 bugs which will never
> be
> > fixed by oracle, or at least, not in a public release.
> >
> > If we can still do the compile in java 7 language and link to java 7 JDK,
> > then that bit of the release is good -then java 8 can be used for that
> test
> >
> > Ultimately, we're going to be forced onto java 8 just because all our
> > dependencies have moved onto it, and some CVE will force us to move.
> >
> > At which point, I think its time to declare branch-2 dead. It's had a
> > great life, but trying to keep java 7 support alive isn't sustainable.
> Not
> > just in this testing, but
> > cherrypicking patches back gets more and more difficult -branch-3 has
> > moved on in both use of java 8 language, and in the codebase in general.
> >
> > > On 28 Jan 2019, at 20:18, Vinod Kumar Vavilapalli 
> > wrote:
> > >
> > > The community made a decision long time ago that we'd like to keep the
> > compatibility & so tie branch-2 to Java 7, but do Java 8+ only work on
> 3.x.
> > >
> > > I always assumed that most (all?) downstream users build branch-2 on
> JDK
> > 7 only, can anyone confirm? If so, there may be an easier way to address
> > these test issues.
> > >
> > > +Vinod
> > >
> > >> On Jan 28, 2019, at 11:24 AM, Jonathan Hung 
> > wrote:
> > >>
> > >> Hi folks,
> > >>
> > >> Forking a discussion based on HADOOP-15711. To summarize, there are
> > issues
> > >> with branch-2 tests running on java 7 (openjdk) which don't exist on
> > java
> > >> 8. From our testing, the build can pass with openjdk 8.
> > >>
> > >> For branch-3, the work to move the build to use java 8 was done in
> > >> HADOOP-14816 as part of the Dockerfile OS version change. HADOOP-16053
> > was
> > >> filed to backport this OS version change to branch-2 (but without the
> > java
> > >> 7 -> java 8 change). So my proposal is to also make the java 7 ->
> java 8
> > >> version change in branch-2.
> > >>
> > >> As mentioned in HADOOP-15711, the main issue is around source and
> binary
> > >> compatibility. I don't currently have a great answer, but one initial
> > >> thought is to build source/binary against java 7 to ensure
> compatibility
> > >> and run the rest of the build as java 8.
> > >>
> > >> Thoughts?
> > >>
> > >> Jonathan Hung
> > >
> > >
> > > -
> > > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> > >
> >
> >
>

Re: [DISCUSS] Moving branch-2 to java 8

[Jonathan Hung]

Thanks Vinod and Steve, agreed about java7 compile compatibility. At least
for now, we should be able to maintain java7 source compatibility and run
tests on java8. There's a test run here:
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86-jhung/46/
which calls a java8 specific API, installs both openjdk7/openjdk8 in the
dockerfile, compiles on both versions, and tests on just java8 (via
--multijdkdirs=/usr/lib/jvm/java-7-openjdk-amd64,/usr/lib/jvm/java-8-openjdk-amd64
and --multijdktests=compile). If we eventually decide it's too much of a
pain to maintain java7 source compatibility we can do that at a later point.

Also based on discussion with others in the community at the contributors
meetup this past Wednesday, seems we are generally in favor of testing
against java8. I'll start a vote soon.

Jonathan Hung


On Tue, Jan 29, 2019 at 4:11 AM Steve Loughran 
wrote:

> branch-2 is the JDK 7 branch, but for a long time I (and presumably
> others) have relied on jenkins to keep us honest by doing that build and
> test
>
> right now, we can't do that any more, due to jdk7 bugs which will never be
> fixed by oracle, or at least, not in a public release.
>
> If we can still do the compile in java 7 language and link to java 7 JDK,
> then that bit of the release is good -then java 8 can be used for that test
>
> Ultimately, we're going to be forced onto java 8 just because all our
> dependencies have moved onto it, and some CVE will force us to move.
>
> At which point, I think its time to declare branch-2 dead. It's had a
> great life, but trying to keep java 7 support alive isn't sustainable. Not
> just in this testing, but
> cherrypicking patches back gets more and more difficult -branch-3 has
> moved on in both use of java 8 language, and in the codebase in general.
>
> > On 28 Jan 2019, at 20:18, Vinod Kumar Vavilapalli 
> wrote:
> >
> > The community made a decision long time ago that we'd like to keep the
> compatibility & so tie branch-2 to Java 7, but do Java 8+ only work on 3.x.
> >
> > I always assumed that most (all?) downstream users build branch-2 on JDK
> 7 only, can anyone confirm? If so, there may be an easier way to address
> these test issues.
> >
> > +Vinod
> >
> >> On Jan 28, 2019, at 11:24 AM, Jonathan Hung 
> wrote:
> >>
> >> Hi folks,
> >>
> >> Forking a discussion based on HADOOP-15711. To summarize, there are
> issues
> >> with branch-2 tests running on java 7 (openjdk) which don't exist on
> java
> >> 8. From our testing, the build can pass with openjdk 8.
> >>
> >> For branch-3, the work to move the build to use java 8 was done in
> >> HADOOP-14816 as part of the Dockerfile OS version change. HADOOP-16053
> was
> >> filed to backport this OS version change to branch-2 (but without the
> java
> >> 7 -> java 8 change). So my proposal is to also make the java 7 -> java 8
> >> version change in branch-2.
> >>
> >> As mentioned in HADOOP-15711, the main issue is around source and binary
> >> compatibility. I don't currently have a great answer, but one initial
> >> thought is to build source/binary against java 7 to ensure compatibility
> >> and run the rest of the build as java 8.
> >>
> >> Thoughts?
> >>
> >> Jonathan Hung
> >
> >
> > -
> > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> >
>
>

Re: [DISCUSS] Moving branch-2 to java 8

[Steve Loughran]

branch-2 is the JDK 7 branch, but for a long time I (and presumably others) 
have relied on jenkins to keep us honest by doing that build and test

right now, we can't do that any more, due to jdk7 bugs which will never be 
fixed by oracle, or at least, not in a public release.

If we can still do the compile in java 7 language and link to java 7 JDK, then 
that bit of the release is good -then java 8 can be used for that test

Ultimately, we're going to be forced onto java 8 just because all our 
dependencies have moved onto it, and some CVE will force us to move.

At which point, I think its time to declare branch-2 dead. It's had a great 
life, but trying to keep java 7 support alive isn't sustainable. Not just in 
this testing, but
cherrypicking patches back gets more and more difficult -branch-3 has moved on 
in both use of java 8 language, and in the codebase in general. 

> On 28 Jan 2019, at 20:18, Vinod Kumar Vavilapalli  wrote:
> 
> The community made a decision long time ago that we'd like to keep the 
> compatibility & so tie branch-2 to Java 7, but do Java 8+ only work on 3.x.
> 
> I always assumed that most (all?) downstream users build branch-2 on JDK 7 
> only, can anyone confirm? If so, there may be an easier way to address these 
> test issues.
> 
> +Vinod
> 
>> On Jan 28, 2019, at 11:24 AM, Jonathan Hung  wrote:
>> 
>> Hi folks,
>> 
>> Forking a discussion based on HADOOP-15711. To summarize, there are issues
>> with branch-2 tests running on java 7 (openjdk) which don't exist on java
>> 8. From our testing, the build can pass with openjdk 8.
>> 
>> For branch-3, the work to move the build to use java 8 was done in
>> HADOOP-14816 as part of the Dockerfile OS version change. HADOOP-16053 was
>> filed to backport this OS version change to branch-2 (but without the java
>> 7 -> java 8 change). So my proposal is to also make the java 7 -> java 8
>> version change in branch-2.
>> 
>> As mentioned in HADOOP-15711, the main issue is around source and binary
>> compatibility. I don't currently have a great answer, but one initial
>> thought is to build source/binary against java 7 to ensure compatibility
>> and run the rest of the build as java 8.
>> 
>> Thoughts?
>> 
>> Jonathan Hung
> 
> 
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> 


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [DISCUSS] Moving branch-2 to java 8

[Vinod Kumar Vavilapalli]

The community made a decision long time ago that we'd like to keep the 
compatibility & so tie branch-2 to Java 7, but do Java 8+ only work on 3.x.

I always assumed that most (all?) downstream users build branch-2 on JDK 7 
only, can anyone confirm? If so, there may be an easier way to address these 
test issues.

+Vinod

> On Jan 28, 2019, at 11:24 AM, Jonathan Hung  wrote:
> 
> Hi folks,
> 
> Forking a discussion based on HADOOP-15711. To summarize, there are issues
> with branch-2 tests running on java 7 (openjdk) which don't exist on java
> 8. From our testing, the build can pass with openjdk 8.
> 
> For branch-3, the work to move the build to use java 8 was done in
> HADOOP-14816 as part of the Dockerfile OS version change. HADOOP-16053 was
> filed to backport this OS version change to branch-2 (but without the java
> 7 -> java 8 change). So my proposal is to also make the java 7 -> java 8
> version change in branch-2.
> 
> As mentioned in HADOOP-15711, the main issue is around source and binary
> compatibility. I don't currently have a great answer, but one initial
> thought is to build source/binary against java 7 to ensure compatibility
> and run the rest of the build as java 8.
> 
> Thoughts?
> 
> Jonathan Hung


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[DISCUSS] Moving branch-2 to java 8

[Jonathan Hung]

Hi folks,

Forking a discussion based on HADOOP-15711. To summarize, there are issues
with branch-2 tests running on java 7 (openjdk) which don't exist on java
8. From our testing, the build can pass with openjdk 8.

For branch-3, the work to move the build to use java 8 was done in
HADOOP-14816 as part of the Dockerfile OS version change. HADOOP-16053 was
filed to backport this OS version change to branch-2 (but without the java
7 -> java 8 change). So my proposal is to also make the java 7 -> java 8
version change in branch-2.

As mentioned in HADOOP-15711, the main issue is around source and binary
compatibility. I don't currently have a great answer, but one initial
thought is to build source/binary against java 7 to ensure compatibility
and run the rest of the build as java 8.

Thoughts?

Jonathan Hung