I solved this issue by using spark 2.3.1 jars copied from the HDP3.0
cluster. Thanks.
On Thu, Aug 30, 2018 at 10:18 AM Lian Jiang wrote:
> Per https://spark.apache.org/docs/latest/building-spark.html, spark 2.3.1
> is built with hadoop 2.6.X by default. This is why I see my fat jar
> includes ha
Per https://spark.apache.org/docs/latest/building-spark.html, spark 2.3.1
is built with hadoop 2.6.X by default. This is why I see my fat jar
includes hadoop 2.6.5 (instead of 3.1.0) jars. HftpFileSystem has been
removed in hadoop 3.
On https://spark.apache.org/downloads.html, I only see spark 2.3
Hi,
I am using HDP3.0 which uses HADOOP3.1.0 and Spark 2.3.1. My spark
streaming jobs running fine in HDP2.6.4 (HADOOP2.7.3, spark 2.2.0) fails in
HDP3:
java.lang.IllegalAccessError: class
org.apache.hadoop.hdfs.web.HftpFileSystem cannot access its superinterface
org.apache.hadoop.hdfs.web.TokenA