[
https://issues.apache.org/jira/browse/SPARK-8142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14577108#comment-14577108
]
Dev Lakhani commented on SPARK-8142:
------------------------------------
I am not bundling spark, again, I have pre-built binaries downloaded from the
spark website, deployed on my cluster: version 1.3.1 hadoop 2.6. I am using
java maven client dependencies for org.apache.spark spark core_2.10 version
1.3.1 in my application- added as a maven dependency. I package my application
using mvn shade, this builds me a jar with my applications code, glassfish
jersey 2.7 deps and spark 1.3.1 core deps.
I then spark-submit my job jar using spark submit (from spark 1.3.1) with my
jar and the classcast error occurs if I have the userClasspath set to true as
described above.
If I don't my REST client tries to do a get operation and uses the glassfish
jersey 2.7 api but this conflicts with com.sun.jersey 1.9 which comes with
Spark.
I am using 1 version of spark 1.3.1-hadoop.2.6 on my cluster, I have no other
versions of spark on that cluster. The RELEASE file states 1.3.1 (git revision
908a0bf) build for Hadoop 2.6.0. Confirmed on all nodes.
I did a maven dependency tree on my application code and the only spark version
is 1.3.1 in all of the maven dependencies that I use. Spark SQL 1.3.1, Spark
Core 1.3.1, Spark Network Common 1.3.1. I've been using this version fine for
all other non REST based operations and other spark operations, its only when I
use userClasspath I get this error.
> Spark Job Fails with ResultTask ClassCastException
> --------------------------------------------------
>
> Key: SPARK-8142
> URL: https://issues.apache.org/jira/browse/SPARK-8142
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.3.1
> Reporter: Dev Lakhani
>
> When running a Spark Job, I get no failures in the application code
> whatsoever but a weird ResultTask Class exception. In my job, I create a RDD
> from HBase and for each partition do a REST call on an API, using a REST
> client. This has worked in IntelliJ but when I deploy to a cluster using
> spark-submit.sh I get :
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in
> stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0
> (TID 3, host): java.lang.ClassCastException:
> org.apache.spark.scheduler.ResultTask cannot be cast to
> org.apache.spark.scheduler.Task
> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:185)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> These are the configs I set to override the spark classpath because I want to
> use my own glassfish jersey version:
>
> sparkConf.set("spark.driver.userClassPathFirst","true");
> sparkConf.set("spark.executor.userClassPathFirst","true");
> I see no other warnings or errors in any of the logs.
> Unfortunately I cannot post my code, but please ask me questions that will
> help debug the issue. Using spark 1.3.1 hadoop 2.6.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
