Re: Dropping Apache Spark Hadoop2 Binary Distribution?

2022-10-06 Thread Steve Loughran
On Wed, 5 Oct 2022 at 21:59, Chao Sun wrote: > +1 > > > and specifically may allow us to finally move off of the ancient version > of Guava (?) > > I think the Guava issue comes from Hive 2.3 dependency, not Hadoop. > hadoop branch-2 has guava dependencies; not sure which one A key lesson

Re: Dropping Apache Spark Hadoop2 Binary Distribution?

2022-10-05 Thread Dongjoon Hyun
Thank you all. SPARK-40651 is merged to Apache Spark master branch for Apache Spark 3.4.0 now. Dongjoon. On Wed, Oct 5, 2022 at 3:24 PM L. C. Hsieh wrote: > +1 > > Thanks Dongjoon. > > On Wed, Oct 5, 2022 at 3:11 PM Jungtaek Lim > wrote: > > > > +1 > > > > On Thu, Oct 6, 2022 at 5:59 AM Chao

Re: Dropping Apache Spark Hadoop2 Binary Distribution?

2022-10-05 Thread L. C. Hsieh
+1 Thanks Dongjoon. On Wed, Oct 5, 2022 at 3:11 PM Jungtaek Lim wrote: > > +1 > > On Thu, Oct 6, 2022 at 5:59 AM Chao Sun wrote: >> >> +1 >> >> > and specifically may allow us to finally move off of the ancient version >> > of Guava (?) >> >> I think the Guava issue comes from Hive 2.3

Re: Dropping Apache Spark Hadoop2 Binary Distribution?

2022-10-05 Thread Jungtaek Lim
+1 On Thu, Oct 6, 2022 at 5:59 AM Chao Sun wrote: > +1 > > > and specifically may allow us to finally move off of the ancient version > of Guava (?) > > I think the Guava issue comes from Hive 2.3 dependency, not Hadoop. > > On Wed, Oct 5, 2022 at 1:55 PM Xinrong Meng > wrote: > >> +1. >> >>

Re: Dropping Apache Spark Hadoop2 Binary Distribution?

2022-10-05 Thread Chao Sun
+1 > and specifically may allow us to finally move off of the ancient version of Guava (?) I think the Guava issue comes from Hive 2.3 dependency, not Hadoop. On Wed, Oct 5, 2022 at 1:55 PM Xinrong Meng wrote: > +1. > > On Wed, Oct 5, 2022 at 1:53 PM Xiao Li > wrote: > >> +1. >> >> Xiao >>

Re: Dropping Apache Spark Hadoop2 Binary Distribution?

2022-10-05 Thread Xinrong Meng
+1. On Wed, Oct 5, 2022 at 1:53 PM Xiao Li wrote: > +1. > > Xiao > > On Wed, Oct 5, 2022 at 12:49 PM Sean Owen wrote: > >> I'm OK with this. It simplifies maintenance a bit, and specifically may >> allow us to finally move off of the ancient version of Guava (?) >> >> On Mon, Oct 3, 2022 at

Re: Dropping Apache Spark Hadoop2 Binary Distribution?

2022-10-05 Thread Xiao Li
+1. Xiao On Wed, Oct 5, 2022 at 12:49 PM Sean Owen wrote: > I'm OK with this. It simplifies maintenance a bit, and specifically may > allow us to finally move off of the ancient version of Guava (?) > > On Mon, Oct 3, 2022 at 10:16 PM Dongjoon Hyun > wrote: > >> Hi, All. >> >> I'm wondering

Re: Dropping Apache Spark Hadoop2 Binary Distribution?

2022-10-05 Thread Sean Owen
I'm OK with this. It simplifies maintenance a bit, and specifically may allow us to finally move off of the ancient version of Guava (?) On Mon, Oct 3, 2022 at 10:16 PM Dongjoon Hyun wrote: > Hi, All. > > I'm wondering if the following Apache Spark Hadoop2 Binary Distribution > is still used by

Re: Dropping Apache Spark Hadoop2 Binary Distribution?

2022-10-04 Thread Dongjoon Hyun
Thank you for your feedback and support, YangJie and Steve. For the internally-built Hadoop clusters, I believe internally-built Spark distribution with the corresponding custom Hadoop will be the best solution instead of Apache Spark with Apache Hadoop 2.7.4 client to have the full internal

Re: Dropping Apache Spark Hadoop2 Binary Distribution?

2022-10-04 Thread Dongjoon Hyun
Yes, it's yours. I added you (Steve Loughran ) as BCC at the first email, Steve. :) Dongjoon. On Tue, Oct 4, 2022 at 6:24 AM Steve Loughran wrote: > > that sounds suspiciously like something I'd write :) > > the move to java8 happened in HADOOP-11858; 3.0.0 > > HADOOP-16219, "[JDK8] Set

Re: Dropping Apache Spark Hadoop2 Binary Distribution?

2022-10-04 Thread Steve Loughran
that sounds suspiciously like something I'd write :) the move to java8 happened in HADOOP-11858; 3.0.0 HADOOP-16219, "[JDK8] Set minimum version of Hadoop 2 to JDK 8" has been open since 2019 and I just closed as WONTFIX. Most of the big production hadoop 2 clusters use java7, because that is

Re: Dropping Apache Spark Hadoop2 Binary Distribution?

2022-10-03 Thread Yang,Jie(INF)
. Regards, YangJie 发件人: Dongjoon Hyun 日期: 2022年10月4日 星期二 11:16 收件人: dev 主题: Dropping Apache Spark Hadoop2 Binary Distribution? Hi, All. I'm wondering if the following Apache Spark Hadoop2 Binary Distribution is still used by someone in the community or not. If it's not used or not useful, we may

Dropping Apache Spark Hadoop2 Binary Distribution?

2022-10-03 Thread Dongjoon Hyun
Hi, All. I'm wondering if the following Apache Spark Hadoop2 Binary Distribution is still used by someone in the community or not. If it's not used or not useful, we may remove it from Apache Spark 3.4.0 release. https://downloads.apache.org/spark/spark-3.3.0/spark-3.3.0-bin-hadoop2.tgz Here