Actually, a lot of our customers are still on JDK6, so if anything, its
popularity hasn't significantly decreased. We still test and support JDK6
for CDH4 and CDH5. The claim that branch-2 is effectively JDK7 because no
one supports JDK6 is untrue.
One issue with your proposal is that java 7+ libraries can have
incompatible APIs compared to their java 6 versions. Guava moves very
quickly with regard to the deprecate+remove cycle. This means branch-2 and
trunk divergence, as we're stuck using different Guava APIs to do the same
thing.
No one's arguing against moving to Java 7+ in trunk eventually, but there
isn't a clear plan for a trunk-based release. I don't see any point to
switching trunk over until that's true, for the aforementioned reasons.
Best,
Andrew
On Wed, Jun 18, 2014 at 12:08 PM, Steve Loughran <ste...@hortonworks.com>
wrote:
> I also think we need to recognise that its been three months since that
> last discussion, and Java 6 has not suddenly burst back into popularity
>
>
> - nobody providing commercial support for Hadoop is offering branch-2
> support on Java 6 AFAIK
> - therefore, nobody is testing it at scale except privately, and they
> aren't reporting bugs if they are
> - if someone actually did file a bug on something on branch-2 which
> didn't work on Java 6 but went away on Java7+, we'd probably close it
> as a
> WORKSFORME
>
>
> whether we acknowledge it or not, Hadoop 2.x is now really Java 7+.
>
> We do all agree that hadoop 3 will not be java 6, so the only issue is
> "when and how to make that transition".
>
> That patch of mine just makes it possible to do today.
>
> I have actually jumped to Java7 in the slider project, and actually being
> using Java 8 and twill; the new language features there are significant and
> would be great to use in Hadoop *at some point in the future*
>
> For Java 7 though, based on that experience, the language changes are
> convenient but not essential
>
> - try-with-resources simply swallows close failures without the log
> integration we have with IOUtils.closeStream(), so shoudn't be used in
> hadoop core anyway.
> - string based switching: convenient, but not critical
> - type inference on template constructors. Modern IDEs handle the pain
> anyway
>
> The only feature I like is multi-catch and typed rethrow
>
> catch(IOException | ExitException e) {
> log.warn(e.toString();
> throw e;
> }
>
> this would make "e" look like Exception, but when rethrown go back to its
> original type.
>
> This reduces duplicate work, and is the bit l actually value. Is it enough
> to justify making code incompatible across branches? No.
>
> So i'm going to propose this, and would like to start a vote on it soon
>
>
> 1. we parameterize java versions in the POMs on all branches, with
> separate JDK versions and Java language
> 2. branch-2: java-6-language and JDK-6 minimum JDK
> 3. trunk: java-6-language and JDK-7 minimum JDK
>
> This would guarantee that none of the java 7 language features went in, but
> we could move trunk up to java 7+ only libraries (jersey, guava). Adopting
> JDK7 features then becomes no more different from adopting java7+
> libraries: those bits of code that have moved can't be backported.
>
> -Steve
>
>
>
>
>
> On 17 June 2014 22:08, Andrew Wang <andrew.w...@cloudera.com> wrote:
>
> > Reviving this thread, I noticed there's been a patch and +1 on
> > HADOOP-10530, and I don't think we actually reached a conclusion.
> >
> > I (and others) have expressed concerns about moving to JDK7 for trunk.
> > Summarizing a few points:
> >
> > - We can't move to JDK7 in branch-2 because of compatibility
> > - branch-2 is currently the only Hadoop release vehicle, there are no
> plans
> > for a trunk-based Hadoop 3
> > - Introducing JDK7-only APIs in trunk will increase divergence with
> > branch-2 and make backports harder
> > - Almost all developers care only about branch-2, since it is the only
> > release vehicle
> >
> > With this in mind, I struggle to see any upsides to introducing JDK7-only
> > APIs to trunk. Please let's not do anything on HADOOP-10530 or related
> > until we agree on this.
> >
> > Thanks,
> > Andrew
> >
> >
> > On Mon, Apr 14, 2014 at 3:31 PM, Steve Loughran <ste...@hortonworks.com>
> > wrote:
> >
> > > On 14 April 2014 17:46, Andrew Purtell <apurt...@apache.org> wrote:
> > >
> > > > How well is trunk tested? Does anyone deploy it with real
> applications
> > > > running on top? When will the trunk codebase next be the basis for a
> > > > production release? An impromptu diff of hadoop-common trunk against
> > > > branch-2 as of today is 38,625 lines. Can they be said to be the same
> > > > animal? I ask because any disincentive toward putting code in trunk
> is
> > > > beside the point, if the only target worth pursuing today is branch-2
> > > > unless one doesn't care if the code is released for production use.
> > > > Questions on whither JDK6 or JDK7+ (or JRE6 versus JRE7+) only matter
> > for
> > > > the vast majority of Hadoopers if talking about branch-2.
> > > >
> > > >
> > > I think its partly a timescale issue; its also because the 1-2
> transition
> > > was so significant, especially at the YARN layer, that it's still
> taking
> > > time to trickle through.
> > >
> > > If you do want code to ship this year, branch-2 is where you are going
> to
> > > try and get it in -and like you say, that's where things get tried in
> the
> > > field. At the same time, the constraints of stability are holding us
> back
> > > -already-.
> > >
> > > I don't see why we should have such another major 1-2 transition in
> > future;
> > > the rate that Arun is pushing out 2.x releases its almost back to the
> > 0.1x
> > > timescale -though at that point most people were fending for themselves
> > and
> > > expectations of stability were less. We do want smaller version
> > increments
> > > in future, which branch-2 is -mostly- delivering.
> > >
> > > While Java 7 doesn't have some must-have features, Java 8 is a
> > significant
> > > improvement in the language, and we should be looking ahead to that,
> > maybe
> > > even doing some leading-edge work on the side, so the same discussion
> > > doesn't come up in two years time when java 7 goes EOL.
> > >
> > >
> > > -steve
> > >
> > > (personal opinions only, etc, )
> > >
> > >
> > > >
> > > > On Mon, Apr 14, 2014 at 9:22 AM, Colin McCabe <
> cmcc...@alumni.cmu.edu
> > > > >wrote:
> > > >
> > > > > I think the bottom line here is that as long as our stable release
> > > > > uses JDK6, there is going to be a very, very strong disincentive to
> > > > > put any code which can't run on JDK6 into trunk.
> > > > >
> > > > > Like I said earlier, the traditional reason for putting something
> in
> > > > > trunk but not the stable release is that it needs more testing.
> If a
> > > > > stable release that drops support for JDK6 is more than a year
> away,
> > > > > does it make sense to put anything in trunk like that? What might
> > > > > need more than a year of testing? Certainly not changes to
> > > > > LocalFileSystem to use the new APIs. I also don't think an upgrade
> > to
> > > > > various libraries qualifies.
> > > > >
> > > > > It might be best to shelve this for now, like we've done in the
> past,
> > > > > until we're ready to talk about a stable release that requires
> JDK7+.
> > > > > At least that's my feeling.
> > > > >
> > > > > If we're really desperate for the new file APIs JDK7 provides, we
> > > > > could consider using loadable modules for it in branch-2. This is
> > > > > similar to how we provide JNI versions of certain things on certain
> > > > > platforms, without dropping support for the other platforms.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > On Sun, Apr 13, 2014 at 10:39 AM, Raymie Stata <
> rst...@altiscale.com
> > >
> > > > > wrote:
> > > > > > There's an outstanding question addressed to me: "Are there
> > > particular
> > > > > > features or new dependencies that you would like to contribute
> (or
> > > see
> > > > > > contributed) that require using the Java 1.7 APIs?" The question
> > > > > > misses the point: We'd figure out how to write something we
> wanted
> > to
> > > > > > contribute to Hadoop against the APIs of Java4 if that's what it
> > took
> > > > > > to get them into a stable release. And at current course and
> > speed,
> > > > > > that's how ridiculous things could get.
> > > > > >
> > > > > > To summarize, it seems like there's a vague consensus that it
> might
> > > be
> > > > > > okay to eventually allow the use of Java7 in trunk, but there's
> no
> > > > > > decision. And there's been no answer to the concern that even if
> > > such
> > > > > > dependencies were allowed in Java7, the only people using them
> > would
> > > > > > be people who uninterested in getting their patches into a stable
> > > > > > release of Hadoop on any knowable timeframe, which doesn't bode
> > well
> > > > > > for the ability to stabilize that Java7 code when it comes time
> to
> > > > > > attempt to.
> > > > > >
> > > > > > I don't have more to add, so I'll go back to lurking. It'll be
> > > > > > interesting to see where we'll be standing a year from now.
> > > > > >
> > > > > > On Sun, Apr 13, 2014 at 2:09 AM, Tsuyoshi OZAWA
> > > > > > <ozawa.tsuyo...@gmail.com> wrote:
> > > > > >> Hi,
> > > > > >>
> > > > > >> +1 for Karthik's idea(non-binding).
> > > > > >>
> > > > > >> IMO, we should keep the compatibility between JDK 6 and JDK 7 on
> > > both
> > > > > branch-1
> > > > > >> and branch-2, because users can be using them. For future
> releases
> > > > that
> > > > > we can
> > > > > >> declare breaking compatibility(e.g. 3.0.0 release), we can use
> > JDK 7
> > > > > >> features if we
> > > > > >> can get benefits. However, it can increase maintenance costs and
> > > > > distributes the
> > > > > >> efforts of contributions to maintain branches. Then, I think it
> is
> > > > > >> reasonable approach
> > > > > >> that we use limited and minimum JDK-7 APIs when we have reasons
> we
> > > > need
> > > > > to use
> > > > > >> the features.
> > > > > >> By the way, if we start to use JDK 7 APIs, we should declare the
> > > basis
> > > > > >> when to use
> > > > > >> JDK 7 APIs on Wiki not to confuse contributors.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> - Tsuyoshi
> > > > > >>
> > > > > >> On Wed, Apr 9, 2014 at 11:44 AM, Raymie Stata <
> > rst...@altiscale.com
> > > >
> > > > > wrote:
> > > > > >>>> It might make sense to try to enumerate the benefits of
> > switching
> > > to
> > > > > >>>> Java7 APIs and dependencies.
> > > > > >>>
> > > > > >>> - Java7 introduced a huge number of language, byte-code, API,
> > and
> > > > > >>> tooling enhancements! Just to name a few: try-with-resources,
> > > newer
> > > > > >>> and stronger encyrption methods, more scalable concurrency
> > > > primitives.
> > > > > >>> See http://www.slideshare.net/boulderjug/55-things-in-java-7
> > > > > >>>
> > > > > >>> - We can't update current dependencies, and we can't add cool
> > new
> > > > > ones.
> > > > > >>>
> > > > > >>> - Putting language/APIs aside, don't forget that a huge
> amount
> > of
> > > > > effort
> > > > > >>> goes into qualifying for Java6 (at least, I hope the folks
> > claiming
> > > > to
> > > > > >>> support Java6 are putting in such an effort :-). Wouldn't
> Hadoop
> > > > > >>> users/customers be better served if qualification effort went
> > into
> > > > > >>> Java7/8 versus Java6/7?
> > > > > >>>
> > > > > >>> Getting to Java7 as a development env (and Java8 as a runtime
> > env)
> > > > > >>> seems like a no-brainer. Question is: How?
> > > > > >>>
> > > > > >>> On Tue, Apr 8, 2014 at 10:21 AM, Sandy Ryza <
> > > sandy.r...@cloudera.com
> > > > >
> > > > > wrote:
> > > > > >>>> It might make sense to try to enumerate the benefits of
> > switching
> > > to
> > > > > Java7
> > > > > >>>> APIs and dependencies. IMO, the ones listed so far on this
> > thread
> > > > > don't
> > > > > >>>> make a compelling enough case to drop Java6 in branch-2 on any
> > > time
> > > > > frame,
> > > > > >>>> even if this means supporting Java6 through 2015. For
> example,
> > > the
> > > > > change
> > > > > >>>> in RawLocalFileSystem semantics might be an incompatible
> change
> > > for
> > > > > >>>> branch-2 any way.
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> On Tue, Apr 8, 2014 at 10:05 AM, Karthik Kambatla <
> > > > ka...@cloudera.com
> > > > > >wrote:
> > > > > >>>>
> > > > > >>>>> +1 to NOT breaking compatibility in branch-2.
> > > > > >>>>>
> > > > > >>>>> I think it is reasonable to require JDK7 for trunk, if we
> limit
> > > use
> > > > > of
> > > > > >>>>> JDK7-only API to security fixes etc. If we make other
> > > optimizations
> > > > > (like
> > > > > >>>>> IO), it would be a pain to backport things to branch-2. I
> guess
> > > > this
> > > > > all
> > > > > >>>>> depends on when we see ourselves shipping Hadoop-3. Any ideas
> > on
> > > > > that?
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>> On Tue, Apr 8, 2014 at 9:19 AM, Eli Collins <
> e...@cloudera.com>
> > > > > wrote:
> > > > > >>>>>
> > > > > >>>>> > On Tue, Apr 8, 2014 at 2:00 AM, Ottenheimer, Davi
> > > > > >>>>> > <davi.ottenhei...@emc.com> wrote:
> > > > > >>>>> > >> From: Eli Collins [mailto:e...@cloudera.com]
> > > > > >>>>> > >> Sent: Monday, April 07, 2014 11:54 AM
> > > > > >>>>> > >>
> > > > > >>>>> > >>
> > > > > >>>>> > >> IMO we should not drop support for Java 6 in a minor
> > update
> > > > of a
> > > > > >>>>> stable
> > > > > >>>>> > >> release (v2). I don't think the larger Hadoop user base
> > > would
> > > > > find it
> > > > > >>>>> > >> acceptable that upgrading to a minor update caused their
> > > > > systems to
> > > > > >>>>> stop
> > > > > >>>>> > >> working because they didn't upgrade Java. There are
> people
> > > > still
> > > > > >>>>> getting
> > > > > >>>>> > >> support for Java 6. ...
> > > > > >>>>> > >>
> > > > > >>>>> > >> Thanks,
> > > > > >>>>> > >> Eli
> > > > > >>>>> > >
> > > > > >>>>> > > Hi Eli,
> > > > > >>>>> > >
> > > > > >>>>> > > Technically you are correct those with extended support
> get
> > > > > critical
> > > > > >>>>> > security fixes for 6 until the end of 2016. I am curious
> > > whether
> > > > > many of
> > > > > >>>>> > those are in the Hadoop user base. Do you know? My guess is
> > the
> > > > > vast
> > > > > >>>>> > majority are within Oracle's official public end of life,
> > which
> > > > > was over
> > > > > >>>>> 12
> > > > > >>>>> > months ago. Even Premier support ended Dec 2013:
> > > > > >>>>> > >
> > > > > >>>>> > > http://www.oracle.com/technetwork/java/eol-135779.html
> > > > > >>>>> > >
> > > > > >>>>> > > The end of Java 6 support carries much risk. It has to be
> > > > > considered in
> > > > > >>>>> > terms of serious security vulnerabilities such as
> > CVE-2013-2465
> > > > > with CVSS
> > > > > >>>>> > score 10.0.
> > > > > >>>>> > >
> > > > > >>>>> > > http://www.cvedetails.com/cve/CVE-2013-2465/
> > > > > >>>>> > >
> > > > > >>>>> > > Since you mentioned "caused systems to stop" as an
> example
> > of
> > > > > what
> > > > > >>>>> would
> > > > > >>>>> > be a concern to Hadoop users, please note the CVE-2013-2465
> > > > > availability
> > > > > >>>>> > impact:
> > > > > >>>>> > >
> > > > > >>>>> > > "Complete (There is a total shutdown of the affected
> > > resource.
> > > > > The
> > > > > >>>>> > attacker can render the resource completely unavailable.)"
> > > > > >>>>> > >
> > > > > >>>>> > > This vulnerability was patched in Java 6 Update 51, but
> > post
> > > > end
> > > > > of
> > > > > >>>>> > life. Apple pushed out the update specifically because of
> > this
> > > > > >>>>> > vulnerability (http://support.apple.com/kb/HT5717) as did
> > some
> > > > > other
> > > > > >>>>> > vendors privately, but for the majority of people using
> Java
> > 6
> > > > > means they
> > > > > >>>>> > have a ticking time bomb.
> > > > > >>>>> > >
> > > > > >>>>> > > Allowing it to stay should be considered in terms of
> > > accepting
> > > > > the
> > > > > >>>>> whole
> > > > > >>>>> > risk posture.
> > > > > >>>>> > >
> > > > > >>>>> >
> > > > > >>>>> > There are some who get extended support, but I suspect many
> > > just
> > > > > have
> > > > > >>>>> > a if-it's-not-broke mentality when it comes to production
> > > > > deployments.
> > > > > >>>>> > The current code supports both java6 and java7 and so
> allows
> > > > these
> > > > > >>>>> > people to remain compatible, while enabling others to
> upgrade
> > > to
> > > > > the
> > > > > >>>>> > java7 runtime. This seems like the right compromise for a
> > > stable
> > > > > >>>>> > release series. Again, absolutely makes sense for trunk (ie
> > v3)
> > > > to
> > > > > >>>>> > require java7 or greater.
> > > > > >>>>> >
> > > > > >>>>>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> - Tsuyoshi
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > >
> > > > - Andy
> > > >
> > > > Problems worthy of attack prove their worth by hitting back. - Piet
> > Hein
> > > > (via Tom White)
> > > >
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
