[
https://issues.apache.org/jira/browse/SPARK-7819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14572255#comment-14572255
]
Fi edited comment on SPARK-7819 at 6/4/15 6:31 AM:
---------------------------------------------------
Actually, maybe the InvalidClassCastException might be a little too flaky.
I am running a spark job which queries an ORC table daily partition via the
HiveContext.
A single context is created, and I spin up about ten threads (one for each day).
I am seeing plenty of the same errors in various tasks, enough to kill the job
for that day.
Curiously, the exact same serialVersionUID are logged in each failure:
org.apache.spark.sql.hive.MetastoreRelation; local class incompatible:
stream classdesc serialVersionUID = 2590680563934099718, local class
serialVersionUID = -8650941563091306200
So the interesting thing is that some jobs (for a particular day) work
perfectly fine, but others fail.
I tried running this multi-threaded job again, and the same error occurs, but
in different places.
It looks like it usually works fine when Spark automatically schedules the task
to be rerun.
So a workaround might be to bump up the number of allowable failures before
giving up on the job.
This job works perfectly fine on our Spark 1.3 builds, unfortunately, this
issue is occurring too often in a larger job in Spark 1.4 :(
was (Author: coderfi):
Actually, maybe the InvalidClassCastException might be a little too flaky.
I am running a spark job which queries an ORC table daily partition via the
HiveContext.
A single context is created, and I spin up about ten threads (one for each day).
I am seeing plenty of the same errors in various tasks, enough to kill the job
for that day.
Curiously, the exact same serialVersionUID are logged in each failure:
org.apache.spark.sql.hive.MetastoreRelation; local class incompatible:
stream classdesc serialVersionUID = 2590680563934099718, local class
serialVersionUID = -8650941563091306200
So the interesting thing is that some jobs (for a particular day) work
perfectly fine, but others fail.
I tried running this multi-threaded job again, and the same error occurs, but
in different places.
This job works perfectly fine on our Spark 1.3 builds, unfortunately, this
issue is occurring too often in a larger job in Spark 1.4 :(
> Isolated Hive Client Loader appears to cause Native Library
> libMapRClient.4.0.2-mapr.so already loaded in another classloader error
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-7819
> URL: https://issues.apache.org/jira/browse/SPARK-7819
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.4.0
> Reporter: Fi
> Priority: Critical
> Attachments: invalidClassException.log, stacktrace.txt, test.py
>
>
> In reference to the pull request: https://github.com/apache/spark/pull/5876
> I have been running the Spark 1.3 branch for some time with no major hiccups,
> and recently switched to the Spark 1.4 branch.
> I build my spark distribution with the following build command:
> {noformat}
> make-distribution.sh --tgz --skip-java-test --with-tachyon -Phive
> -Phive-0.13.1 -Pmapr4 -Pspark-ganglia-lgpl -Pkinesis-asl -Phive-thriftserver
> {noformat}
> When running a python script containing a series of smoke tests I use to
> validate the build, I encountered an error under the following conditions:
> * start a spark context
> * start a hive context
> * run any hive query
> * stop the spark context
> * start a second spark context
> * run any hive query
> ** ERROR
> From what I can tell, the Isolated Class Loader is hitting a MapR class that
> is loading its native library (presumedly as part of a static initializer).
> Unfortunately, the JVM prohibits this the second time around.
> I would think that shutting down the SparkContext would clear out any
> vestigials of the JVM, so I'm surprised that this would even be a problem.
> Note: all other smoke tests we are running passes fine.
> I will attach the stacktrace and a python script reproducing the issue (at
> least for my environment and build).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
