Re: Raise Java dependency from 6 to 7

[Matei Zaharia]

So from my point of view, I'd do it maybe 1-2 years after all the major Hadoop 
vendors have stopped supporting Java 6. We're not there yet, but we will be 
soon. The reason is that the cost of staying on Java 6 is much smaller to us 
(as developers) than the cost of fragmenting the Spark community by having a 
big chunk of users unable to upgrade past a certain version of Spark (or 
requiring them to use a modified third-party version, which is a similar 
thing). There's very little stuff in Java 7 or 8 that would make the project 
much better if we dropped support for 6 -- this Jetty issue might be one, but 
we can certainly work around it.

We've done a lot of stuff to reach the broadest set of users, including pushing 
back the versions of Python and NumPy we support, and in my experience it's 
been well worth it. In surveys I've seen the majority of users (something like 
75%) updating to each new Spark release within a few months of it coming out, 
which is awesome for keeping the community healthy.

Matei


> On Oct 19, 2014, at 9:48 AM, Corey Nolet  wrote:
> 
> A concrete plan and a definite version upon which the upgrade would be 
> applied sounds like it would benefit the community. If you plan far enough 
> out (as Hadoop has done) and give the community enough of a notice, I can't 
> see it being a problem as they would have ample time upgrade. 
> 
> 
> 
> On Sat, Oct 18, 2014 at 9:20 PM, Marcelo Vanzin  > wrote:
> Hadoop, for better or worse, depends on an ancient version of Jetty
> (6), that is even on a different package. So Spark (or anyone trying
> to use a newer Jetty) is lucky on that front...
> 
> IIRC Hadoop is planning to move to Java 7-only starting with 2.7. Java
> 7 is also supposed to be EOL some time next year, so a plan to move to
> Java 7 and, eventually, Java 8 would be nice.
> 
> On Sat, Oct 18, 2014 at 5:44 PM, Matei Zaharia  > wrote:
> > I'd also wait a bit until these are gone. Jetty is unfortunately a much 
> > hairier topic by the way, because the Hadoop libraries also depend on 
> > Jetty. I think it will be hard to update. However, a patch that shades 
> > Jetty might be nice to have, if that doesn't require shading a lot of other 
> > stuff.
> >
> > Matei
> >
> >> On Oct 18, 2014, at 4:37 PM, Koert Kuipers  >> > wrote:
> >>
> >> my experience is that there are still a lot of java 6 clusters out there.
> >> also distros that bundle spark still support java 6
> >> On Oct 17, 2014 8:01 PM, "Andrew Ash"  >> > wrote:
> >>
> >>> Hi Spark devs,
> >>>
> >>> I've heard a few times that keeping support for Java 6 is a priority for
> >>> Apache Spark.  Given that Java 6 has been publicly EOL'd since Feb 2013
> >>>  >>> > and the last
> >>> public update was Apr 2013
> >>>  >>> >, why
> >>> are we still maintaing support for 6?  The only people using it now must 
> >>> be
> >>> paying for the extended support to continue receiving security fixes.
> >>>
> >>> Bumping the lower bound of Java versions up to Java 7 would allow us to
> >>> upgrade from Jetty 8 to 9, which is currently a conflict with the
> >>> Dropwizard framework and a personal pain point.
> >>>
> >>> Java 6 vs 7 for Spark links:
> >>> Try with resources
> >>>  >>> > for
> >>> SparkContext et al
> >>> Upgrade to Jetty 9
> >>>  >>> >
> >>> Warn when not compiling with Java6
> >>>  >>> >
> >>>
> >>>
> >>> Who are the people out there that still need Java 6 support?
> >>>
> >>> Thanks!
> >>> Andrew
> >>>
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org 
> > 
> > For additional commands, e-mail: dev-h...@spark.apache.org 
> > 
> >
> 
> 
> 
> --
> Marcelo
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org 
> 
> For additional commands, e-mail: dev-h...@spark.apache.org 
> 
> 
> 

Re: Raise Java dependency from 6 to 7

[Corey Nolet]

A concrete plan and a definite version upon which the upgrade would be
applied sounds like it would benefit the community. If you plan far enough
out (as Hadoop has done) and give the community enough of a notice, I can't
see it being a problem as they would have ample time upgrade.



On Sat, Oct 18, 2014 at 9:20 PM, Marcelo Vanzin  wrote:

> Hadoop, for better or worse, depends on an ancient version of Jetty
> (6), that is even on a different package. So Spark (or anyone trying
> to use a newer Jetty) is lucky on that front...
>
> IIRC Hadoop is planning to move to Java 7-only starting with 2.7. Java
> 7 is also supposed to be EOL some time next year, so a plan to move to
> Java 7 and, eventually, Java 8 would be nice.
>
> On Sat, Oct 18, 2014 at 5:44 PM, Matei Zaharia 
> wrote:
> > I'd also wait a bit until these are gone. Jetty is unfortunately a much
> hairier topic by the way, because the Hadoop libraries also depend on
> Jetty. I think it will be hard to update. However, a patch that shades
> Jetty might be nice to have, if that doesn't require shading a lot of other
> stuff.
> >
> > Matei
> >
> >> On Oct 18, 2014, at 4:37 PM, Koert Kuipers  wrote:
> >>
> >> my experience is that there are still a lot of java 6 clusters out
> there.
> >> also distros that bundle spark still support java 6
> >> On Oct 17, 2014 8:01 PM, "Andrew Ash"  wrote:
> >>
> >>> Hi Spark devs,
> >>>
> >>> I've heard a few times that keeping support for Java 6 is a priority
> for
> >>> Apache Spark.  Given that Java 6 has been publicly EOL'd since Feb 2013
> >>>  and the last
> >>> public update was Apr 2013
> >>> ,
> why
> >>> are we still maintaing support for 6?  The only people using it now
> must be
> >>> paying for the extended support to continue receiving security fixes.
> >>>
> >>> Bumping the lower bound of Java versions up to Java 7 would allow us to
> >>> upgrade from Jetty 8 to 9, which is currently a conflict with the
> >>> Dropwizard framework and a personal pain point.
> >>>
> >>> Java 6 vs 7 for Spark links:
> >>> Try with resources
> >>>  for
> >>> SparkContext et al
> >>> Upgrade to Jetty 9
> >>> 
> >>> Warn when not compiling with Java6
> >>> 
> >>>
> >>>
> >>> Who are the people out there that still need Java 6 support?
> >>>
> >>> Thanks!
> >>> Andrew
> >>>
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> > For additional commands, e-mail: dev-h...@spark.apache.org
> >
>
>
>
> --
> Marcelo
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> For additional commands, e-mail: dev-h...@spark.apache.org
>
>

Re: Raise Java dependency from 6 to 7

[Marcelo Vanzin]

Hadoop, for better or worse, depends on an ancient version of Jetty
(6), that is even on a different package. So Spark (or anyone trying
to use a newer Jetty) is lucky on that front...

IIRC Hadoop is planning to move to Java 7-only starting with 2.7. Java
7 is also supposed to be EOL some time next year, so a plan to move to
Java 7 and, eventually, Java 8 would be nice.

On Sat, Oct 18, 2014 at 5:44 PM, Matei Zaharia  wrote:
> I'd also wait a bit until these are gone. Jetty is unfortunately a much 
> hairier topic by the way, because the Hadoop libraries also depend on Jetty. 
> I think it will be hard to update. However, a patch that shades Jetty might 
> be nice to have, if that doesn't require shading a lot of other stuff.
>
> Matei
>
>> On Oct 18, 2014, at 4:37 PM, Koert Kuipers  wrote:
>>
>> my experience is that there are still a lot of java 6 clusters out there.
>> also distros that bundle spark still support java 6
>> On Oct 17, 2014 8:01 PM, "Andrew Ash"  wrote:
>>
>>> Hi Spark devs,
>>>
>>> I've heard a few times that keeping support for Java 6 is a priority for
>>> Apache Spark.  Given that Java 6 has been publicly EOL'd since Feb 2013
>>>  and the last
>>> public update was Apr 2013
>>> , why
>>> are we still maintaing support for 6?  The only people using it now must be
>>> paying for the extended support to continue receiving security fixes.
>>>
>>> Bumping the lower bound of Java versions up to Java 7 would allow us to
>>> upgrade from Jetty 8 to 9, which is currently a conflict with the
>>> Dropwizard framework and a personal pain point.
>>>
>>> Java 6 vs 7 for Spark links:
>>> Try with resources
>>>  for
>>> SparkContext et al
>>> Upgrade to Jetty 9
>>> 
>>> Warn when not compiling with Java6
>>> 
>>>
>>>
>>> Who are the people out there that still need Java 6 support?
>>>
>>> Thanks!
>>> Andrew
>>>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> For additional commands, e-mail: dev-h...@spark.apache.org
>



-- 
Marcelo

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: Raise Java dependency from 6 to 7

[Matei Zaharia]

I'd also wait a bit until these are gone. Jetty is unfortunately a much hairier 
topic by the way, because the Hadoop libraries also depend on Jetty. I think it 
will be hard to update. However, a patch that shades Jetty might be nice to 
have, if that doesn't require shading a lot of other stuff.

Matei

> On Oct 18, 2014, at 4:37 PM, Koert Kuipers  wrote:
> 
> my experience is that there are still a lot of java 6 clusters out there.
> also distros that bundle spark still support java 6
> On Oct 17, 2014 8:01 PM, "Andrew Ash"  wrote:
> 
>> Hi Spark devs,
>> 
>> I've heard a few times that keeping support for Java 6 is a priority for
>> Apache Spark.  Given that Java 6 has been publicly EOL'd since Feb 2013
>>  and the last
>> public update was Apr 2013
>> , why
>> are we still maintaing support for 6?  The only people using it now must be
>> paying for the extended support to continue receiving security fixes.
>> 
>> Bumping the lower bound of Java versions up to Java 7 would allow us to
>> upgrade from Jetty 8 to 9, which is currently a conflict with the
>> Dropwizard framework and a personal pain point.
>> 
>> Java 6 vs 7 for Spark links:
>> Try with resources
>>  for
>> SparkContext et al
>> Upgrade to Jetty 9
>> 
>> Warn when not compiling with Java6
>> 
>> 
>> 
>> Who are the people out there that still need Java 6 support?
>> 
>> Thanks!
>> Andrew
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: Raise Java dependency from 6 to 7

[Koert Kuipers]

my experience is that there are still a lot of java 6 clusters out there.
also distros that bundle spark still support java 6
On Oct 17, 2014 8:01 PM, "Andrew Ash"  wrote:

> Hi Spark devs,
>
> I've heard a few times that keeping support for Java 6 is a priority for
> Apache Spark.  Given that Java 6 has been publicly EOL'd since Feb 2013
>  and the last
> public update was Apr 2013
> , why
> are we still maintaing support for 6?  The only people using it now must be
> paying for the extended support to continue receiving security fixes.
>
> Bumping the lower bound of Java versions up to Java 7 would allow us to
> upgrade from Jetty 8 to 9, which is currently a conflict with the
> Dropwizard framework and a personal pain point.
>
> Java 6 vs 7 for Spark links:
> Try with resources
>  for
> SparkContext et al
> Upgrade to Jetty 9
> 
> Warn when not compiling with Java6
> 
>
>
> Who are the people out there that still need Java 6 support?
>
> Thanks!
> Andrew
>

Raise Java dependency from 6 to 7

[Andrew Ash]

Hi Spark devs,

I've heard a few times that keeping support for Java 6 is a priority for
Apache Spark.  Given that Java 6 has been publicly EOL'd since Feb 2013
 and the last
public update was Apr 2013
, why
are we still maintaing support for 6?  The only people using it now must be
paying for the extended support to continue receiving security fixes.

Bumping the lower bound of Java versions up to Java 7 would allow us to
upgrade from Jetty 8 to 9, which is currently a conflict with the
Dropwizard framework and a personal pain point.

Java 6 vs 7 for Spark links:
Try with resources
 for
SparkContext et al
Upgrade to Jetty 9

Warn when not compiling with Java6



Who are the people out there that still need Java 6 support?

Thanks!
Andrew