Thank you for the feedback, Sean and Xiao. Bests, Dongjoon.
On Mon, Oct 28, 2019 at 12:52 PM Xiao Li <lix...@databricks.com> wrote: > The stability and quality of Hadoop 3.2 profile are unknown. The changes > are massive, including Hive execution and a new version of Hive > thriftserver. > > To reduce the risk, I would like to keep the current default version > unchanged. When it becomes stable, we can change the default profile to > Hadoop-3.2. > > Cheers, > > Xiao > > On Mon, Oct 28, 2019 at 12:51 PM Sean Owen <sro...@gmail.com> wrote: > >> I'm OK with that, but don't have a strong opinion nor info about the >> implications. >> That said my guess is we're close to the point where we don't need to >> support Hadoop 2.x anyway, so, yeah. >> >> On Mon, Oct 28, 2019 at 2:33 PM Dongjoon Hyun <dongjoon.h...@gmail.com> >> wrote: >> > >> > Hi, All. >> > >> > There was a discussion on publishing artifacts built with Hadoop 3 . >> > But, we are still publishing with Hadoop 2.7.3 and `3.0-preview` will >> be the same because we didn't change anything yet. >> > >> > Technically, we need to change two places for publishing. >> > >> > 1. Jenkins Snapshot Publishing >> > >> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-master-maven-snapshots/ >> > >> > 2. Release Snapshot/Release Publishing >> > >> https://github.com/apache/spark/blob/master/dev/create-release/release-build.sh >> > >> > To minimize the change, we need to switch our default Hadoop profile. >> > >> > Currently, the default is `hadoop-2.7 (2.7.4)` profile and `hadoop-3.2 >> (3.2.0)` is optional. >> > We had better use `hadoop-3.2` profile by default and `hadoop-2.7` >> optionally. >> > >> > Note that this means we use Hive 2.3.6 by default. Only `hadoop-2.7` >> distribution will use `Hive 1.2.1` like Apache Spark 2.4.x. >> > >> > Bests, >> > Dongjoon. >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >> > > -- > [image: Databricks Summit - Watch the talks] > <https://databricks.com/sparkaisummit/north-america> >
