Yes, the main blocking issue is the hive version used in Spark (1.2.1.spark) doesn't support run on Hadoop 3. Hive will check the Hadoop version in the runtime [1]. Besides this I think some pom changes should be enough to support Hadoop 3.
If we want to use Hadoop 3 shaded client jar, then the pom requires lots of changes, but this is not necessary. [1] https://github.com/apache/hive/blob/6751225a5cde4c40839df8b46e8d241fdda5cd34/shims/common/src/main/java/org/apache/hadoop/hive/shims/ShimLoader.java#L144 2018-04-03 4:57 GMT+08:00 Marcelo Vanzin <van...@cloudera.com>: > Saisai filed SPARK-23534, but the main blocking issue is really > SPARK-18673. > > > On Mon, Apr 2, 2018 at 1:00 PM, Reynold Xin <r...@databricks.com> wrote: > > Does anybody know what needs to be done in order for Spark to support > Hadoop > > 3? > > > > > > -- > Marcelo > > --------------------------------------------------------------------- > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >
