[
https://issues.apache.org/jira/browse/HIVE-18436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16321701#comment-16321701
]
Sahil Takiar commented on HIVE-18436:
-------------------------------------
Diff to compile is pretty small:
{code}
diff --git a/pom.xml b/pom.xml
index cf5ad5e3a6..b6c373c49b 100644
--- a/pom.xml
+++ b/pom.xml
@@ -197,7 +197,7 @@
<tez.version>0.9.1</tez.version>
<slider.version>0.92.0-incubating</slider.version>
<super-csv.version>2.2.0</super-csv.version>
- <spark.version>2.2.0</spark.version>
+ <spark.version>2.3.0-SNAPSHOT</spark.version>
<scala.binary.version>2.11</scala.binary.version>
<scala.version>2.11.8</scala.version>
<tempus-fugit.version>1.1</tempus-fugit.version>
diff --git
a/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/LocalHiveSparkClient.java
b/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/LocalHiveSparkClient.java
index 8956a92bb2..4ebddc7e93 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/LocalHiveSparkClient.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/LocalHiveSparkClient.java
@@ -102,7 +102,7 @@ private LocalHiveSparkClient(SparkConf sparkConf, HiveConf
hiveConf)
sc.addJar(regJar);
}
jobMetricsListener = new JobMetricsListener();
- sc.sc().listenerBus().addListener(jobMetricsListener);
+ sc.sc().addSparkListener(jobMetricsListener);
}
@Override
{code}
However, I'm getting a weird error when running {{TestMiniSparkOnYarnCliDriver}}
{code}
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/hive/conf/HiveConf
at
org.apache.spark.deploy.security.HadoopDelegationTokenManager.getDelegationTokenProviders(HadoopDelegationTokenManager.scala:68)
at
org.apache.spark.deploy.security.HadoopDelegationTokenManager.<init>(HadoopDelegationTokenManager.scala:54)
at
org.apache.spark.deploy.yarn.security.YARNHadoopDelegationTokenManager.<init>(YARNHadoopDelegationTokenManager.scala:44)
at org.apache.spark.deploy.yarn.Client.<init>(Client.scala:123)
at
org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1502)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.hive.conf.HiveConf
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 10 more
{code}
> Upgrade to Spark 2.3.0
> ----------------------
>
> Key: HIVE-18436
> URL: https://issues.apache.org/jira/browse/HIVE-18436
> Project: Hive
> Issue Type: Task
> Components: Spark
> Reporter: Sahil Takiar
> Assignee: Sahil Takiar
>
> Branching has been completed. Release candidates should be published soon.
> Might be a while before the actual release, but at least we get to identify
> any issues early.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
