[
https://issues.apache.org/jira/browse/SPARK-2018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14027753#comment-14027753
]
Gireesh Punathil commented on SPARK-2018:
-----------------------------------------
I face this same issue for a different class:
org.apache.spark.SerializableWritable; local class incompatible: stream
classdesc serialVersionUID = 6301214776158303468, local class serialVersionUID
= -7785455416944904980
All the entities are in the same system, using same JVM version.
Bad value in hex: 57726974 225BB4EC
Good value in hex: 93F4818C 225BB4EC
Please note that the lower word (in big endian) is same while the higher word
is different.
With little debugging, I extracted the byte buffer from which the
de-serialization was being performed, and it looks like below. Apparently,
offsets 48,49,50,51 contain the right value for the lower word (in big endian)
of the SUID, while offsets 41,42,43,44 seem to be containing portion of the
class name ('Writ', which is part of 'SerializableWritable'). Looks like data
corruption in the serializer?
Is there any configurable option through which I can disable custom
serializations, and fall back (compltely) to Java serialization?
In a spark configuration which runs HdfsWordCount example, which process is
responsible for serializing this object? With my little understanding of the
whole framework, I can see a master, a worker, an execute runner (spawned by
worker) and the client who issues the test case.
The client reports the execption, but I believe it just prints whatever the
execute runner reports to it. ExecutorRunner actually invokes a
org.apache.spark.executor.CoarseGrainedExecutorBackend to perform its task.
This process never serialized any objects. is the serialization performed by
the master? Or is the example exercising a pre-serialized disc file? How do we
know that?
Is there any debug option which reports messages on step-by-step serialization
/ de-serialization process? I have all the tools and debug capabilities at the
JVM / native level, but lack of understanding of the spark topology makes it
hard to move further.
Finally, if I modify a scala file, how do I effect the changes into the class
file and the parent jar file? when I performed sbt assembly or sbt package it
indeed compiled the scala file, but the .class file (
org/apache/spark/SerializableWritable.class) or the .jar file
(spark-core_2.10-0.9.1.jar) do not seem to have updated. Is there any other way
of doing it?
[0] = 172, 0xac
[1] = 116, 0x74, 't'
[2] = 31, 0x1f
[3] = 172, 0xac
[4] = 237, 0xed
[5] = 0, 0x00
[6] = 5, 0x05
[7] = 115, 0x73, 's'
[8] = 114, 0x72, 'r'
[9] = 0, 0x00
[10] = 37, 0x25, '%' // indicates next 37 bytes are for the class name?
then probably multiple over-writes happened: i) the classname over the SUID,
ii) something else on the classname (bytes 45 through 48)
[11] = 111, 0x6f, 'o'
[12] = 114, 0x72, 'r'
[13] = 103, 0x67, 'g'
[14] = 46, 0x2e, '.'
[15] = 97, 0x61, 'a'
[16] = 112, 0x70, 'p'
[17] = 97, 0x61, 'a'
[18] = 99, 0x63, 'c'
[19] = 104, 0x68, 'h'
[20] = 101, 0x65, 'e'
[21] = 46, 0x2e, '.'
[22] = 115, 0x73, 's'
[23] = 112, 0x70, 'p'
[24] = 97, 0x61, 'a'
[25] = 114, 0x72, 'r'
[26] = 107, 0x6b, 'k'
[27] = 46, 0x2e, '.'
[28] = 83, 0x53, 'S'
[29] = 101, 0x65, 'e'
[30] = 114, 0x72, 'r'
[31] = 105, 0x69, 'i'
[32] = 97, 0x61, 'a'
[33] = 108, 0x6c, 'l'
[34] = 105, 0x69, 'i'
[35] = 8, 0x08
[36] = 122, 0x7a, 'z'
[37] = 97, 0x61, 'a'
[38] = 98, 0x62, 'b'
[39] = 108, 0x6c, 'l'
[40] = 101, 0x65, 'e'
[41] = 87, 0x57, 'W'
[42] = 114, 0x72, 'r'
[43] = 105, 0x69, 'i'
[44] = 116, 0x74, 't'
[45] = 192, 0xc0
[46] = 7, 0x07
[47] = 15, 0x0f
[48] = 34, 0x22, '"'
[49] = 91, 0x5b, '['
[50] = 180, 0xb4
[51] = 236, 0xec
[52] = 3, 0x03
[53] = 0, 0x00
[54] = 0, 0x00
[55] = 120, 0x78, 'x'
[56] = 112, 0x70, 'p'
[57] = 122, 0x7a, 'z'
[58] = 0, 0x00
[59] = 0, 0x00
[60] = 4, 0x04
[61] = 0, 0x00
[62] = 0, 0x00
[63] = 32, 0x20, ' '
[64] = 224, 0xe0
...
> Big-Endian (IBM Power7) Spark Serialization issue
> --------------------------------------------------
>
> Key: SPARK-2018
> URL: https://issues.apache.org/jira/browse/SPARK-2018
> Project: Spark
> Issue Type: Bug
> Affects Versions: 1.0.0
> Environment: hardware : IBM Power7
> OS:Linux version 2.6.32-358.el6.ppc64
> (mockbu...@ppc-017.build.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red
> Hat 4.4.7-3) (GCC) ) #1 SMP Tue Jan 29 11:43:27 EST 2013
> JDK: Java(TM) SE Runtime Environment (build pxp6470sr5-20130619_01(SR5))
> IBM J9 VM (build 2.6, JRE 1.7.0 Linux ppc64-64 Compressed References
> 20130617_152572 (JIT enabled, AOT enabled)
> Hadoop:Hadoop-0.2.3-CDH5.0
> Spark:Spark-1.0.0 or Spark-0.9.1
> spark-env.sh:
> export JAVA_HOME=/opt/ibm/java-ppc64-70/
> export SPARK_MASTER_IP=9.114.34.69
> export SPARK_WORKER_MEMORY=10000m
> export SPARK_CLASSPATH=/home/test1/spark-1.0.0-bin-hadoop2/lib
> export STANDALONE_SPARK_MASTER_HOST=9.114.34.69
> #export SPARK_JAVA_OPTS=' -Xdebug
> -Xrunjdwp:transport=dt_socket,address=99999,server=y,suspend=n '
> Reporter: Yanjie Gao
>
> We have an application run on Spark on Power7 System .
> But we meet an important issue about serialization.
> The example HdfsWordCount can meet the problem.
> ./bin/run-example org.apache.spark.examples.streaming.HdfsWordCount
> localdir
> We used Power7 (Big-Endian arch) and Redhat 6.4.
> Big-Endian is the main cause since the example ran successfully in another
> Power-based Little Endian setup.
> here is the exception stack and log:
> Spark Executor Command: "/opt/ibm/java-ppc64-70//bin/java" "-cp"
> "/home/test1/spark-1.0.0-bin-hadoop2/lib::/home/test1/src/spark-1.0.0-bin-hadoop2/conf:/home/test1/src/spark-1.0.0-bin-hadoop2/lib/spark-assembly-1.0.0-hadoop2.2.0.jar:/home/test1/src/spark-1.0.0-bin-hadoop2/lib/datanucleus-rdbms-3.2.1.jar:/home/test1/src/spark-1.0.0-bin-hadoop2/lib/datanucleus-api-jdo-3.2.1.jar:/home/test1/src/spark-1.0.0-bin-hadoop2/lib/datanucleus-core-3.2.2.jar:/home/test1/src/hadoop-2.3.0-cdh5.0.0/etc/hadoop/:/home/test1/src/hadoop-2.3.0-cdh5.0.0/etc/hadoop/"
> "-XX:MaxPermSize=128m" "-Xdebug"
> "-Xrunjdwp:transport=dt_socket,address=99999,server=y,suspend=n" "-Xms512M"
> "-Xmx512M" "org.apache.spark.executor.CoarseGrainedExecutorBackend"
> "akka.tcp://spark@9.186.105.141:60253/user/CoarseGrainedScheduler" "2"
> "p7hvs7br16" "4" "akka.tcp://sparkWorker@p7hvs7br16:59240/user/Worker"
> "app-20140604023054-0000"
> ========================================
> 14/06/04 02:31:20 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/06/04 02:31:21 INFO spark.SecurityManager: Changing view acls to:
> test1,yifeng
> 14/06/04 02:31:21 INFO spark.SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(test1, yifeng)
> 14/06/04 02:31:22 INFO slf4j.Slf4jLogger: Slf4jLogger started
> 14/06/04 02:31:22 INFO Remoting: Starting remoting
> 14/06/04 02:31:22 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://sparkExecutor@p7hvs7br16:39658]
> 14/06/04 02:31:22 INFO Remoting: Remoting now listens on addresses:
> [akka.tcp://sparkExecutor@p7hvs7br16:39658]
> 14/06/04 02:31:22 INFO executor.CoarseGrainedExecutorBackend: Connecting to
> driver: akka.tcp://spark@9.186.105.141:60253/user/CoarseGrainedScheduler
> 14/06/04 02:31:22 INFO worker.WorkerWatcher: Connecting to worker
> akka.tcp://sparkWorker@p7hvs7br16:59240/user/Worker
> 14/06/04 02:31:23 INFO worker.WorkerWatcher: Successfully connected to
> akka.tcp://sparkWorker@p7hvs7br16:59240/user/Worker
> 14/06/04 02:31:24 INFO executor.CoarseGrainedExecutorBackend: Successfully
> registered with driver
> 14/06/04 02:31:24 INFO spark.SecurityManager: Changing view acls to:
> test1,yifeng
> 14/06/04 02:31:24 INFO spark.SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(test1, yifeng)
> 14/06/04 02:31:24 INFO slf4j.Slf4jLogger: Slf4jLogger started
> 14/06/04 02:31:24 INFO Remoting: Starting remoting
> 14/06/04 02:31:24 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://spark@p7hvs7br16:58990]
> 14/06/04 02:31:24 INFO Remoting: Remoting now listens on addresses:
> [akka.tcp://spark@p7hvs7br16:58990]
> 14/06/04 02:31:24 INFO spark.SparkEnv: Connecting to MapOutputTracker:
> akka.tcp://spark@9.186.105.141:60253/user/MapOutputTracker
> 14/06/04 02:31:25 INFO spark.SparkEnv: Connecting to BlockManagerMaster:
> akka.tcp://spark@9.186.105.141:60253/user/BlockManagerMaster
> 14/06/04 02:31:25 INFO storage.DiskBlockManager: Created local directory at
> /tmp/spark-local-20140604023125-3f61
> 14/06/04 02:31:25 INFO storage.MemoryStore: MemoryStore started with capacity
> 307.2 MB.
> 14/06/04 02:31:25 INFO network.ConnectionManager: Bound socket to port 39041
> with id = ConnectionManagerId(p7hvs7br16,39041)
> 14/06/04 02:31:25 INFO storage.BlockManagerMaster: Trying to register
> BlockManager
> 14/06/04 02:31:25 INFO storage.BlockManagerMaster: Registered BlockManager
> 14/06/04 02:31:25 INFO spark.HttpFileServer: HTTP File server directory is
> /tmp/spark-7bce4e43-2833-4666-93af-bd97c327497b
> 14/06/04 02:31:25 INFO spark.HttpServer: Starting HTTP Server
> 14/06/04 02:31:25 INFO server.Server: jetty-8.y.z-SNAPSHOT
> 14/06/04 02:31:26 INFO server.AbstractConnector: Started
> SocketConnector@0.0.0.0:39958
> 14/06/04 02:31:26 INFO executor.CoarseGrainedExecutorBackend: Got assigned
> task 2
> 14/06/04 02:31:26 INFO executor.Executor: Running task ID 2
> 14/06/04 02:31:26 ERROR executor.Executor: Exception in task ID 2
> java.io.InvalidClassException: scala.reflect.ClassTag$$anon$1; local class
> incompatible: stream classdesc serialVersionUID = -8102093212602380348, local
> class serialVersionUID = -4937928798201944954
> at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:678)
> at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1678)
> at
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1573)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1827)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1406)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2047)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1971)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1854)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1406)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2047)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1971)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1854)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1406)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:409)
> at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:76)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:607)
> at
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1078)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1949)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1854)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1406)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2047)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1971)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1854)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1406)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:409)
> at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:76)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:607)
> at
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1078)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1949)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1854)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1406)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2047)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1971)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1854)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1406)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2047)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1971)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1854)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1406)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:409)
> at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:76)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:607)
> at
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1078)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1949)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1854)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1406)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2047)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1971)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1854)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1406)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2047)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1971)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1854)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1406)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:409)
> at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:76)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:607)
> at
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1078)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1949)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1854)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1406)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2047)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1971)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1854)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1406)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2047)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1971)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1854)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1406)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:409)
> at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:76)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:607)
> at java.lang.reflect.Method.invoke(Method.java:607)
> at
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1078)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1949)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1854)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1406)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2047)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1971)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1854)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1406)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:409)
> at
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:63)
> at
> org.apache.spark.scheduler.ResultTask$.deserializeInfo(ResultTask.scala:61)
> at
> org.apache.spark.scheduler.ResultTask.readExternal(ResultTask.scala:141)
> at
> java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1893)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1852)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1406)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:409)
> at
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:63)
> at
> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:85)
> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:169)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:781)
> 14/06/04 02:31:26 ERROR executor.CoarseGrainedExecutorBackend: Driver
> Disassociated [akka.tcp://sparkExecutor@p7hvs7br16:39658] ->
> [akka.tcp://spark@9.186.105.141:60253] disassociated! Shutting down.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
