[
https://issues.apache.org/jira/browse/SPARK-4133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antonio Jesus Navarro updated SPARK-4133:
-----------------------------------------
Description:
Snappy related problems found when trying to upgrade existing Spark Streaming
App from 1.0.2 to 1.1.0.
We can not run an existing 1.0.2 spark app if upgraded to 1.1.0
> IOException is thrown by snappy (parsing_error(2))
{code}
Executor task launch worker-0 DEBUG storage.BlockManager - Getting local block
broadcast_0
Executor task launch worker-0 DEBUG storage.BlockManager - Level for block
broadcast_0 is StorageLevel(true, true, false, true, 1)
Executor task launch worker-0 DEBUG storage.BlockManager - Getting block
broadcast_0 from memory
Executor task launch worker-0 DEBUG storage.BlockManager - Getting local block
broadcast_0
Executor task launch worker-0 DEBUG executor.Executor - Task 0's epoch is 0
Executor task launch worker-0 DEBUG storage.BlockManager - Block broadcast_0
not registered locally
Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Started
reading broadcast variable 0
sparkDriver-akka.actor.default-dispatcher-4 INFO
receiver.ReceiverSupervisorImpl - Registered receiver 0
Executor task launch worker-0 INFO util.RecurringTimer - Started timer for
BlockGenerator at time 1414656492400
Executor task launch worker-0 INFO receiver.BlockGenerator - Started
BlockGenerator
Thread-87 INFO receiver.BlockGenerator - Started block pushing thread
Executor task launch worker-0 INFO receiver.ReceiverSupervisorImpl - Starting
receiver
sparkDriver-akka.actor.default-dispatcher-5 INFO scheduler.ReceiverTracker -
Registered receiver for stream 0 from akka://sparkDriver
Executor task launch worker-0 INFO kafka.KafkaReceiver - Starting Kafka
Consumer Stream with group: stratioStreaming
Executor task launch worker-0 INFO kafka.KafkaReceiver - Connecting to
Zookeeper: node.stratio.com:2181
sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor]
received message StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0
cap=0]) from Actor[akka://sparkDriver/deadLetters]
sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor]
received message StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0
cap=0]) from Actor[akka://sparkDriver/deadLetters]
sparkDriver-akka.actor.default-dispatcher-6 DEBUG local.LocalActor - [actor]
received message StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0
cap=0]) from Actor[akka://sparkDriver/deadLetters]
sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor]
handled message (8.442354 ms)
StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from
Actor[akka://sparkDriver/deadLetters]
sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor]
handled message (8.412421 ms)
StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from
Actor[akka://sparkDriver/deadLetters]
sparkDriver-akka.actor.default-dispatcher-6 DEBUG local.LocalActor - [actor]
handled message (8.385471 ms)
StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from
Actor[akka://sparkDriver/deadLetters]
Executor task launch worker-0 INFO utils.VerifiableProperties - Verifying
properties
Executor task launch worker-0 INFO utils.VerifiableProperties - Property
group.id is overridden to stratioStreaming
Executor task launch worker-0 INFO utils.VerifiableProperties - Property
zookeeper.connect is overridden to node.stratio.com:2181
Executor task launch worker-0 INFO utils.VerifiableProperties - Property
zookeeper.connection.timeout.ms is overridden to 10000
Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Reading
broadcast variable 0 took 0.033998997 s
Executor task launch worker-0 INFO consumer.ZookeeperConsumerConnector -
[stratioStreaming_ajn-stratio-1414656492293-8ecb3e3a], Connecting to zookeeper
instance at node.stratio.com:2181
Executor task launch worker-0 DEBUG zkclient.ZkConnection - Creating new
ZookKeeper instance to connect to node.stratio.com:2181.
ZkClient-EventThread-169-node.stratio.com:2181 INFO zkclient.ZkEventThread -
Starting ZkClient event thread.
Executor task launch worker-0 INFO zookeeper.ZooKeeper - Initiating client
connection, connectString=node.stratio.com:2181 sessionTimeout=6000
watcher=org.I0Itec.zkclient.ZkClient@5b4bdc81
Executor task launch worker-0 DEBUG zkclient.ZkClient - Awaiting connection to
Zookeeper server
Executor task launch worker-0 DEBUG zkclient.ZkClient - Waiting for keeper
state SyncConnected
Executor task launch worker-0-SendThread(node.stratio.com:2181) INFO
zookeeper.ClientCnxn - Opening socket connection to server
node.stratio.com/172.19.0.96:2181. Will not attempt to authenticate using SASL
(unknown error)
Executor task launch worker-0-SendThread(node.stratio.com:2181) INFO
zookeeper.ClientCnxn - Socket connection established to
node.stratio.com/172.19.0.96:2181, initiating session
Executor task launch worker-0-SendThread(node.stratio.com:2181) DEBUG
zookeeper.ClientCnxn - Session establishment request sent on
node.stratio.com/172.19.0.96:2181
Executor task launch worker-0-SendThread(node.stratio.com:2181) INFO
zookeeper.ClientCnxn - Session establishment complete on server
node.stratio.com/172.19.0.96:2181, sessionid = 0x1496007e6710002, negotiated
timeout = 6000
Executor task launch worker-0-EventThread DEBUG zkclient.ZkClient - Received
event: WatchedEvent state:SyncConnected type:None path:null
Executor task launch worker-0-EventThread INFO zkclient.ZkClient - zookeeper
state changed (SyncConnected)
Executor task launch worker-0-EventThread DEBUG zkclient.ZkClient - Leaving
process event
Executor task launch worker-0 DEBUG zkclient.ZkClient - State is SyncConnected
RecurringTimer - BlockGenerator DEBUG util.RecurringTimer - Callback for
BlockGenerator called at time 1414656492400
Executor task launch worker-0 DEBUG utils.KafkaScheduler - Initializing task
scheduler.
Executor task launch worker-0 INFO consumer.ZookeeperConsumerConnector -
[stratioStreaming_ajn-stratio-1414656492293-8ecb3e3a], starting auto committer
every 60000 ms
Executor task launch worker-0 DEBUG utils.KafkaScheduler - Scheduling task
kafka-consumer-autocommit with initial delay 60000 ms and period 60000 ms.
Executor task launch worker-0 INFO kafka.KafkaReceiver - Connected to
node.stratio.com:2181
Executor task launch worker-0 DEBUG consumer.ZookeeperConsumerConnector -
[stratioStreaming_ajn-stratio-1414656492293-8ecb3e3a], entering consume
Executor task launch worker-0 INFO consumer.ZookeeperConsumerConnector -
[stratioStreaming_ajn-stratio-1414656492293-8ecb3e3a], begin registering
consumer stratioStreaming_ajn-stratio-1414656492293-8ecb3e3a in ZK
Executor task launch worker-0 DEBUG storage.BlockManager - Getting local block
broadcast_0
Executor task launch worker-0 DEBUG storage.BlockManager - Block broadcast_0
not registered locally
Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Started
reading broadcast variable 0
Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Reading
broadcast variable 0 took 5.5676E-5 s
Executor task launch worker-0 ERROR executor.Executor - Exception in task 0.0
in stage 0.0 (TID 0)
java.io.IOException: PARSING_ERROR(2)
at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
at org.xerial.snappy.SnappyNative.uncompressedLength(Native Method)
at org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:545)
at
org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:125)
at
org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:88)
at org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:58)
at
org.apache.spark.io.SnappyCompressionCodec.compressedInputStream(CompressionCodec.scala:128)
at
org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:232)
at
org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
at
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Executor task launch worker-0 ERROR executor.Executor - Exception in task 0.0
in stage 0.0 (TID 0)
java.io.IOException: PARSING_ERROR(2)
at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
at org.xerial.snappy.SnappyNative.uncompressedLength(Native Method)
at org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:545)
at
org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:125)
at
org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:88)
at org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:58)
at
org.apache.spark.io.SnappyCompressionCodec.compressedInputStream(CompressionCodec.scala:128)
at
org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:232)
at
org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
at
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor]
received message StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=2144
cap=2144]) from Actor[akka://sparkDriver/deadLetters]
sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor]
received message StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=2144
cap=2144]) from Actor[akka://sparkDriver/deadLetters]
sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.TaskSchedulerImpl -
parentName: , name: TaskSet_0, runningTasks: 0
sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.TaskSchedulerImpl -
parentName: , name: TaskSet_0, runningTasks: 0
sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor]
handled message (1.213476 ms)
StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=2144 cap=2144]) from
Actor[akka://sparkDriver/deadLetters]
sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor]
handled message (1.543991 ms)
StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=2144 cap=2144]) from
Actor[akka://sparkDriver/deadLetters]
Result resolver thread-0 WARN scheduler.TaskSetManager - Lost task 0.0 in
stage 0.0 (TID 0, localhost): java.io.IOException: PARSING_ERROR(2)
org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
org.xerial.snappy.SnappyNative.uncompressedLength(Native Method)
org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:545)
org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:125)
org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:88)
org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:58)
org.apache.spark.io.SnappyCompressionCodec.compressedInputStream(CompressionCodec.scala:128)
org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:232)
org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:606)
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
Result resolver thread-0 WARN scheduler.TaskSetManager - Lost task 0.0 in
stage 0.0 (TID 0, localhost): java.io.IOException: PARSING_ERROR(2)
org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
org.xerial.snappy.SnappyNative.uncompressedLength(Native Method)
org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:545)
org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:125)
org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:88)
org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:58)
org.apache.spark.io.SnappyCompressionCodec.compressedInputStream(CompressionCodec.scala:128)
org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:232)
org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:606)
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
Result resolver thread-0 ERROR scheduler.TaskSetManager - Task 0 in stage 0.0
failed 1 times; aborting job
Result resolver thread-0 ERROR scheduler.TaskSetManager - Task 0 in stage 0.0
failed 1 times; aborting job
Result resolver thread-0 INFO scheduler.TaskSchedulerImpl - Removed TaskSet
0.0, whose tasks have all completed, from pool
Result resolver thread-0 INFO scheduler.TaskSchedulerImpl - Removed TaskSet
0.0, whose tasks have all completed, from pool
sparkDriver-akka.actor.default-dispatcher-2 INFO scheduler.TaskSchedulerImpl -
Cancelling stage 0
sparkDriver-akka.actor.default-dispatcher-2 INFO scheduler.TaskSchedulerImpl -
Cancelling stage 0
Thread-84 INFO scheduler.DAGScheduler - Failed to run runJob at
ReceiverTracker.scala:275
Thread-85 INFO scheduler.DAGScheduler - Failed to run runJob at
ReceiverTracker.scala:275
sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.DAGScheduler -
Removing running stage 0
sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.DAGScheduler -
Removing running stage 0
sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.DAGScheduler -
After removal of stage 0, remaining stages = 0
sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.DAGScheduler -
After removal of stage 0, remaining stages = 0
Executor task launch worker-0-SendThread(node.stratio.com:2181) DEBUG
zookeeper.ClientCnxn - Reading reply sessionid:0x1496007e6710002, packet::
clientPath:null serverPath:null finished:false header:: 1,1 replyHeader::
1,25,-101 request::
'/consumers/stratioStreaming/ids/stratioStreaming_ajn-stratio-1414656492293-8ecb3e3a,#7b2276657273696f6e223a312c22737562736372697074696f6e223a7b227374726174696f5f73747265616d696e675f616374696f6e223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343134363536343932343737227d,v{s{31,s{'world,'anyone}}},1
response::
{code}
> Only spark version changed
As far as we have checked, snappy will throw this error when dealing with zero
bytes length arrays.
We have tried:
> Changing from snappy to LZF,
> Changing spark.broadcast.compress false
{code}
Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Reading
broadcast variable 0 took 0.240869283 s
sparkDriver-akka.actor.default-dispatcher-4 DEBUG kafka.KafkaInputDStream -
Cleared 1 RDDs that were older than 1414657342000 ms: 1414657342000 ms
sparkDriver-akka.actor.default-dispatcher-4 DEBUG kafka.KafkaInputDStream -
Cleared 1 RDDs that were older than 1414657342000 ms: 1414657342000 ms
sparkDriver-akka.actor.default-dispatcher-4 DEBUG streaming.DStreamGraph -
Cleared old metadata for time 1414657344000 ms
sparkDriver-akka.actor.default-dispatcher-13 DEBUG
storage.BlockManagerSlaveActor - removing RDD 3
Executor task launch worker-1 DEBUG storage.BlockManager - Getting local block
broadcast_1
sparkDriver-akka.actor.default-dispatcher-4 DEBUG dstream.MappedDStream - Time
1414657344000 ms is valid
sparkDriver-akka.actor.default-dispatcher-3 DEBUG dstream.FilteredDStream -
Time 1414657344000 ms is valid
sparkDriver-akka.actor.default-dispatcher-4 DEBUG dstream.FilteredDStream -
Time 1414657344000 ms is valid
sparkDriver-akka.actor.default-dispatcher-13 INFO storage.BlockManager -
Removing RDD 3
sparkDriver-akka.actor.default-dispatcher-2 DEBUG
storage.BlockManagerSlaveActor - [actor] handled message (134.08408 ms)
RemoveRdd(3) from Actor[akka://sparkDriver/temp/$f]
sparkDriver-akka.actor.default-dispatcher-2 DEBUG
storage.BlockManagerSlaveActor - [actor] received message RemoveRdd(2) from
Actor[akka://sparkDriver/temp/$i]
sparkDriver-akka.actor.default-dispatcher-4 DEBUG
storage.BlockManagerSlaveActor - removing RDD 2
sparkDriver-akka.actor.default-dispatcher-4 INFO storage.BlockManager -
Removing RDD 2
sparkDriver-akka.actor.default-dispatcher-2 DEBUG
storage.BlockManagerSlaveActor - [actor] handled message (0.050955 ms)
RemoveRdd(2) from Actor[akka://sparkDriver/temp/$i]
sparkDriver-akka.actor.default-dispatcher-2 DEBUG
storage.BlockManagerSlaveActor - [actor] received message RemoveRdd(1) from
Actor[akka://sparkDriver/temp/$j]
sparkDriver-akka.actor.default-dispatcher-5 DEBUG
storage.BlockManagerSlaveActor - removing RDD 1
sparkDriver-akka.actor.default-dispatcher-5 INFO storage.BlockManager -
Removing RDD 1
sparkDriver-akka.actor.default-dispatcher-2 DEBUG
storage.BlockManagerSlaveActor - [actor] handled message (0.037738 ms)
RemoveRdd(1) from Actor[akka://sparkDriver/temp/$j]
Executor task launch worker-0 ERROR executor.Executor - Exception in task 0.0
in stage 0.0 (TID 0)
java.io.EOFException
at
java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2325)
at
java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2794)
at
java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
at
org.apache.spark.serializer.JavaDeserializationStream$$anon$1.<init>(JavaSerializer.scala:57)
at
org.apache.spark.serializer.JavaDeserializationStream.<init>(JavaSerializer.scala:57)
at
org.apache.spark.serializer.JavaSerializerInstance.deserializeStream(JavaSerializer.scala:95)
at
org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:235)
at
org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
at
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
sparkDriver-akka.actor.default-dispatcher-5 DEBUG
storage.BlockManagerSlaveActor - Done removing RDD 1, response is 0
sparkDriver-akka.actor.default-dispatcher-2 DEBUG
storage.BlockManagerSlaveActor - Done removing RDD 2, response is 0
sparkDriver-akka.actor.default-dispatcher-3 DEBUG
storage.BlockManagerSlaveActor - Done removing RDD 3, response is 0
Executor task launch worker-1 DEBUG storage.BlockManager - Level for block
broadcast_1 is StorageLevel(true, true, false, true, 1)
sparkDriver-akka.actor.default-dispatcher-3 DEBUG dstream.FilteredDStream -
Time 1414657344000 ms is valid
sparkDriver-akka.actor.default-dispatcher-4 DEBUG dstream.MappedDStream - Time
1414657344000 ms is valid
sparkDriver-akka.actor.default-dispatcher-4 DEBUG dstream.FilteredDStream -
Time 1414657344000 ms is valid
sparkDriver-akka.actor.default-dispatcher-3 DEBUG streaming.DStreamGraph -
Generated 4 jobs for time 1414657344000 ms
sparkDriver-akka.actor.default-dispatcher-3 INFO scheduler.JobScheduler -
Added jobs for time 1414657344000 ms
sparkDriver-akka.actor.default-dispatcher-3 DEBUG scheduler.JobGenerator - Got
event DoCheckpoint(1414657344000 ms)
sparkDriver-akka.actor.default-dispatcher-5 DEBUG
storage.BlockManagerSlaveActor - Sent response: 0 to
Actor[akka://sparkDriver/temp/$j]
sparkDriver-akka.actor.default-dispatcher-2 DEBUG
storage.BlockManagerSlaveActor - Sent response: 0 to
Actor[akka://sparkDriver/temp/$i]
Executor task launch worker-1 DEBUG storage.BlockManager - Getting block
broadcast_1 from memory
sparkDriver-akka.actor.default-dispatcher-3 DEBUG
storage.BlockManagerSlaveActor - Sent response: 0 to
Actor[akka://sparkDriver/temp/$f]
Executor task launch worker-0 DEBUG storage.BlockManager - Getting local block
broadcast_0
Executor task launch worker-0 DEBUG storage.BlockManager - Level for block
broadcast_0 is StorageLevel(true, true, false, true, 1)
Executor task launch worker-0 DEBUG storage.BlockManager - Getting block
broadcast_0 from memory
Executor task launch worker-0 DEBUG executor.Executor - Task 0's epoch is 0
Executor task launch worker-1 DEBUG executor.Executor - Task 1's epoch is 0
Executor task launch worker-0 DEBUG storage.BlockManager - Getting local block
broadcast_0
Executor task launch worker-0 DEBUG storage.BlockManager - Block broadcast_0
not registered locally
Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Started
reading broadcast variable 0
sparkDriver-akka.actor.default-dispatcher-4 DEBUG dstream.MappedDStream - Time
1414657344000 ms is valid
sparkDriver-akka.actor.default-dispatcher-4 DEBUG dstream.FilteredDStream -
Time 1414657344000 ms is valid
Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Reading
broadcast variable 0 took 7.0321E-5 s
Executor task launch worker-0 ERROR executor.Executor - Exception in task 0.0
in stage 0.0 (TID 0)
java.io.EOFException
at
java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2325)
at
java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2794)
at
java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
at
org.apache.spark.serializer.JavaDeserializationStream$$anon$1.<init>(JavaSerializer.scala:57)
at
org.apache.spark.serializer.JavaDeserializationStream.<init>(JavaSerializer.scala:57)
at
org.apache.spark.serializer.JavaSerializerInstance.deserializeStream(JavaSerializer.scala:95)
at
org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:235)
at
org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
at
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
sparkDriver-akka.actor.default-dispatcher-5 DEBUG local.LocalActor - [actor]
received message StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=2066
cap=2066]) from Actor[akka://sparkDriver/deadLetters]
sparkDriver-akka.actor.default-dispatcher-3 DEBUG local.LocalActor - [actor]
received message StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=2066
cap=2066]) from Actor[akka://sparkDriver/deadLetters]
sparkDriver-akka.actor.default-dispatcher-5 DEBUG scheduler.TaskSchedulerImpl -
parentName: , name: TaskSet_0, runningTasks: 0
sparkDriver-akka.actor.default-dispatcher-3 DEBUG scheduler.TaskSchedulerImpl -
parentName: , name: TaskSet_0, runningTasks: 0
sparkDriver-akka.actor.default-dispatcher-5 DEBUG local.LocalActor - [actor]
handled message (1.681797 ms)
StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=2066 cap=2066]) from
Actor[akka://sparkDriver/deadLetters]
sparkDriver-akka.actor.default-dispatcher-3 DEBUG local.LocalActor - [actor]
handled message (0.688875 ms)
StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=2066 cap=2066]) from
Actor[akka://sparkDriver/deadLetters]
sparkDriver-akka.actor.default-dispatcher-4 DEBUG dstream.MappedDStream - Time
1414657344000 ms is valid
sparkDriver-akka.actor.default-dispatcher-4 DEBUG dstream.FilteredDStream -
Time 1414657344000 ms is valid
sparkDriver-akka.actor.default-dispatcher-5 DEBUG spark.HeartbeatReceiver -
[actor] received message
Heartbeat(localhost,[Lscala.Tuple2;@1a94802,BlockManagerId(<driver>,
ajn-stratio.local, 53377, 0)) from Actor[akka://sparkDriver/temp/$c]
sparkDriver-akka.actor.default-dispatcher-3 INFO
receiver.ReceiverSupervisorImpl - Registered receiver 0
Executor task launch worker-0 INFO util.RecurringTimer - Started timer for
BlockGenerator at time 1414657344800
Executor task launch worker-0 INFO receiver.BlockGenerator - Started
BlockGenerator
Executor task launch worker-0 INFO receiver.ReceiverSupervisorImpl - Starting
receiver
Thread-87 INFO receiver.BlockGenerator - Started block pushing thread
Result resolver thread-0 WARN scheduler.TaskSetManager - Lost task 0.0 in
stage 0.0 (TID 0, localhost): java.io.EOFException:
java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2325)
java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2794)
java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801)
java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
org.apache.spark.serializer.JavaDeserializationStream$$anon$1.<init>(JavaSerializer.scala:57)
org.apache.spark.serializer.JavaDeserializationStream.<init>(JavaSerializer.scala:57)
org.apache.spark.serializer.JavaSerializerInstance.deserializeStream(JavaSerializer.scala:95)
org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:235)
org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:606)
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
Result resolver thread-0 WARN scheduler.TaskSetManager - Lost task 0.0 in
stage 0.0 (TID 0, localhost): java.io.EOFException:
java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2325)
java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2794)
java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801)
java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
org.apache.spark.serializer.JavaDeserializationStream$$anon$1.<init>(JavaSerializer.scala:57)
org.apache.spark.serializer.JavaDeserializationStream.<init>(JavaSerializer.scala:57)
org.apache.spark.serializer.JavaSerializerInstance.deserializeStream(JavaSerializer.scala:95)
org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:235)
org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:606)
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
Executor task launch worker-0 INFO kafka.KafkaReceiver - Starting Kafka
Consumer Stream with group: stratioStreaming
Executor task launch worker-0 INFO kafka.KafkaReceiver - Connecting to
Zookeeper: node.stratio.com:2181
sparkDriver-akka.actor.default-dispatcher-4 DEBUG dstream.MappedDStream - Time
1414657344000 ms is valid
sparkDriver-akka.actor.default-dispatcher-4 DEBUG dstream.FilteredDStream -
Time 1414657344000 ms is valid
Result resolver thread-0 ERROR scheduler.TaskSetManager - Task 0 in stage 0.0
failed 1 times; aborting job
Result resolver thread-0 ERROR scheduler.TaskSetManager - Task 0 in stage 0.0
failed 1 times; aborting job
sparkDriver-akka.actor.default-dispatcher-2 INFO scheduler.ReceiverTracker -
Registered receiver for stream 0 from akka://sparkDriver
Result resolver thread-0 INFO scheduler.TaskSchedulerImpl - Removed TaskSet
0.0, whose tasks have all completed, from pool
Result resolver thread-0 INFO scheduler.TaskSchedulerImpl - Removed TaskSet
0.0, whose tasks have all completed, from pool
sparkDriver-akka.actor.default-dispatcher-3 DEBUG
storage.BlockManagerMasterActor - [actor] received message
BlockManagerHeartbeat(BlockManagerId(<driver>, ajn-stratio.local, 53377, 0))
from Actor[akka://sparkDriver/temp/$d]
sparkDriver-akka.actor.default-dispatcher-3 DEBUG
storage.BlockManagerMasterActor - [actor] handled message (0.197908 ms)
BlockManagerHeartbeat(BlockManagerId(<driver>, ajn-stratio.local, 53377, 0))
from Actor[akka://sparkDriver/temp/$d]
sparkDriver-akka.actor.default-dispatcher-5 DEBUG spark.HeartbeatReceiver -
[actor] handled message (169.804965 ms)
Heartbeat(localhost,[Lscala.Tuple2;@1a94802,BlockManagerId(<driver>,
ajn-stratio.local, 53377, 0)) from Actor[akka://sparkDriver/temp/$c]
{code}
> Changing from TorrentBroadcast to HTTPBroadcast.
but with no luck for the moment.
was:
Snappy related problems found when trying to upgrade existing Spark Streaming
App from 1.0.2 to 1.1.0.
We can not run an existing 1.0.2 spark app if upgraded to 1.1.0
> IOException is thrown by snappy (parsing_error(2))
> Only spark version changed
As far as we have checked, snappy will throw this error when dealing with zero
bytes length arrays.
We have tried:
> Changing from snappy to LZF,
> Changing broadcast.compression false
> Changing from TorrentBroadcast to HTTPBroadcast.
but with no luck for the moment.
{code}
Executor task launch worker-0 DEBUG storage.BlockManager - Getting local block
broadcast_0
Executor task launch worker-0 DEBUG storage.BlockManager - Level for block
broadcast_0 is StorageLevel(true, true, false, true, 1)
Executor task launch worker-0 DEBUG storage.BlockManager - Getting block
broadcast_0 from memory
Executor task launch worker-0 DEBUG storage.BlockManager - Getting local block
broadcast_0
Executor task launch worker-0 DEBUG executor.Executor - Task 0's epoch is 0
Executor task launch worker-0 DEBUG storage.BlockManager - Block broadcast_0
not registered locally
Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Started
reading broadcast variable 0
sparkDriver-akka.actor.default-dispatcher-4 INFO
receiver.ReceiverSupervisorImpl - Registered receiver 0
Executor task launch worker-0 INFO util.RecurringTimer - Started timer for
BlockGenerator at time 1414656492400
Executor task launch worker-0 INFO receiver.BlockGenerator - Started
BlockGenerator
Thread-87 INFO receiver.BlockGenerator - Started block pushing thread
Executor task launch worker-0 INFO receiver.ReceiverSupervisorImpl - Starting
receiver
sparkDriver-akka.actor.default-dispatcher-5 INFO scheduler.ReceiverTracker -
Registered receiver for stream 0 from akka://sparkDriver
Executor task launch worker-0 INFO kafka.KafkaReceiver - Starting Kafka
Consumer Stream with group: stratioStreaming
Executor task launch worker-0 INFO kafka.KafkaReceiver - Connecting to
Zookeeper: node.stratio.com:2181
sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor]
received message StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0
cap=0]) from Actor[akka://sparkDriver/deadLetters]
sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor]
received message StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0
cap=0]) from Actor[akka://sparkDriver/deadLetters]
sparkDriver-akka.actor.default-dispatcher-6 DEBUG local.LocalActor - [actor]
received message StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0
cap=0]) from Actor[akka://sparkDriver/deadLetters]
sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor]
handled message (8.442354 ms)
StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from
Actor[akka://sparkDriver/deadLetters]
sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor]
handled message (8.412421 ms)
StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from
Actor[akka://sparkDriver/deadLetters]
sparkDriver-akka.actor.default-dispatcher-6 DEBUG local.LocalActor - [actor]
handled message (8.385471 ms)
StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from
Actor[akka://sparkDriver/deadLetters]
Executor task launch worker-0 INFO utils.VerifiableProperties - Verifying
properties
Executor task launch worker-0 INFO utils.VerifiableProperties - Property
group.id is overridden to stratioStreaming
Executor task launch worker-0 INFO utils.VerifiableProperties - Property
zookeeper.connect is overridden to node.stratio.com:2181
Executor task launch worker-0 INFO utils.VerifiableProperties - Property
zookeeper.connection.timeout.ms is overridden to 10000
Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Reading
broadcast variable 0 took 0.033998997 s
Executor task launch worker-0 INFO consumer.ZookeeperConsumerConnector -
[stratioStreaming_ajn-stratio-1414656492293-8ecb3e3a], Connecting to zookeeper
instance at node.stratio.com:2181
Executor task launch worker-0 DEBUG zkclient.ZkConnection - Creating new
ZookKeeper instance to connect to node.stratio.com:2181.
ZkClient-EventThread-169-node.stratio.com:2181 INFO zkclient.ZkEventThread -
Starting ZkClient event thread.
Executor task launch worker-0 INFO zookeeper.ZooKeeper - Initiating client
connection, connectString=node.stratio.com:2181 sessionTimeout=6000
watcher=org.I0Itec.zkclient.ZkClient@5b4bdc81
Executor task launch worker-0 DEBUG zkclient.ZkClient - Awaiting connection to
Zookeeper server
Executor task launch worker-0 DEBUG zkclient.ZkClient - Waiting for keeper
state SyncConnected
Executor task launch worker-0-SendThread(node.stratio.com:2181) INFO
zookeeper.ClientCnxn - Opening socket connection to server
node.stratio.com/172.19.0.96:2181. Will not attempt to authenticate using SASL
(unknown error)
Executor task launch worker-0-SendThread(node.stratio.com:2181) INFO
zookeeper.ClientCnxn - Socket connection established to
node.stratio.com/172.19.0.96:2181, initiating session
Executor task launch worker-0-SendThread(node.stratio.com:2181) DEBUG
zookeeper.ClientCnxn - Session establishment request sent on
node.stratio.com/172.19.0.96:2181
Executor task launch worker-0-SendThread(node.stratio.com:2181) INFO
zookeeper.ClientCnxn - Session establishment complete on server
node.stratio.com/172.19.0.96:2181, sessionid = 0x1496007e6710002, negotiated
timeout = 6000
Executor task launch worker-0-EventThread DEBUG zkclient.ZkClient - Received
event: WatchedEvent state:SyncConnected type:None path:null
Executor task launch worker-0-EventThread INFO zkclient.ZkClient - zookeeper
state changed (SyncConnected)
Executor task launch worker-0-EventThread DEBUG zkclient.ZkClient - Leaving
process event
Executor task launch worker-0 DEBUG zkclient.ZkClient - State is SyncConnected
RecurringTimer - BlockGenerator DEBUG util.RecurringTimer - Callback for
BlockGenerator called at time 1414656492400
Executor task launch worker-0 DEBUG utils.KafkaScheduler - Initializing task
scheduler.
Executor task launch worker-0 INFO consumer.ZookeeperConsumerConnector -
[stratioStreaming_ajn-stratio-1414656492293-8ecb3e3a], starting auto committer
every 60000 ms
Executor task launch worker-0 DEBUG utils.KafkaScheduler - Scheduling task
kafka-consumer-autocommit with initial delay 60000 ms and period 60000 ms.
Executor task launch worker-0 INFO kafka.KafkaReceiver - Connected to
node.stratio.com:2181
Executor task launch worker-0 DEBUG consumer.ZookeeperConsumerConnector -
[stratioStreaming_ajn-stratio-1414656492293-8ecb3e3a], entering consume
Executor task launch worker-0 INFO consumer.ZookeeperConsumerConnector -
[stratioStreaming_ajn-stratio-1414656492293-8ecb3e3a], begin registering
consumer stratioStreaming_ajn-stratio-1414656492293-8ecb3e3a in ZK
Executor task launch worker-0 DEBUG storage.BlockManager - Getting local block
broadcast_0
Executor task launch worker-0 DEBUG storage.BlockManager - Block broadcast_0
not registered locally
Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Started
reading broadcast variable 0
Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Reading
broadcast variable 0 took 5.5676E-5 s
Executor task launch worker-0 ERROR executor.Executor - Exception in task 0.0
in stage 0.0 (TID 0)
java.io.IOException: PARSING_ERROR(2)
at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
at org.xerial.snappy.SnappyNative.uncompressedLength(Native Method)
at org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:545)
at
org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:125)
at
org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:88)
at org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:58)
at
org.apache.spark.io.SnappyCompressionCodec.compressedInputStream(CompressionCodec.scala:128)
at
org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:232)
at
org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
at
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Executor task launch worker-0 ERROR executor.Executor - Exception in task 0.0
in stage 0.0 (TID 0)
java.io.IOException: PARSING_ERROR(2)
at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
at org.xerial.snappy.SnappyNative.uncompressedLength(Native Method)
at org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:545)
at
org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:125)
at
org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:88)
at org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:58)
at
org.apache.spark.io.SnappyCompressionCodec.compressedInputStream(CompressionCodec.scala:128)
at
org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:232)
at
org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
at
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor]
received message StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=2144
cap=2144]) from Actor[akka://sparkDriver/deadLetters]
sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor]
received message StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=2144
cap=2144]) from Actor[akka://sparkDriver/deadLetters]
sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.TaskSchedulerImpl -
parentName: , name: TaskSet_0, runningTasks: 0
sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.TaskSchedulerImpl -
parentName: , name: TaskSet_0, runningTasks: 0
sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor]
handled message (1.213476 ms)
StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=2144 cap=2144]) from
Actor[akka://sparkDriver/deadLetters]
sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor]
handled message (1.543991 ms)
StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=2144 cap=2144]) from
Actor[akka://sparkDriver/deadLetters]
Result resolver thread-0 WARN scheduler.TaskSetManager - Lost task 0.0 in
stage 0.0 (TID 0, localhost): java.io.IOException: PARSING_ERROR(2)
org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
org.xerial.snappy.SnappyNative.uncompressedLength(Native Method)
org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:545)
org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:125)
org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:88)
org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:58)
org.apache.spark.io.SnappyCompressionCodec.compressedInputStream(CompressionCodec.scala:128)
org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:232)
org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:606)
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
Result resolver thread-0 WARN scheduler.TaskSetManager - Lost task 0.0 in
stage 0.0 (TID 0, localhost): java.io.IOException: PARSING_ERROR(2)
org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
org.xerial.snappy.SnappyNative.uncompressedLength(Native Method)
org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:545)
org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:125)
org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:88)
org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:58)
org.apache.spark.io.SnappyCompressionCodec.compressedInputStream(CompressionCodec.scala:128)
org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:232)
org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:606)
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
Result resolver thread-0 ERROR scheduler.TaskSetManager - Task 0 in stage 0.0
failed 1 times; aborting job
Result resolver thread-0 ERROR scheduler.TaskSetManager - Task 0 in stage 0.0
failed 1 times; aborting job
Result resolver thread-0 INFO scheduler.TaskSchedulerImpl - Removed TaskSet
0.0, whose tasks have all completed, from pool
Result resolver thread-0 INFO scheduler.TaskSchedulerImpl - Removed TaskSet
0.0, whose tasks have all completed, from pool
sparkDriver-akka.actor.default-dispatcher-2 INFO scheduler.TaskSchedulerImpl -
Cancelling stage 0
sparkDriver-akka.actor.default-dispatcher-2 INFO scheduler.TaskSchedulerImpl -
Cancelling stage 0
Thread-84 INFO scheduler.DAGScheduler - Failed to run runJob at
ReceiverTracker.scala:275
Thread-85 INFO scheduler.DAGScheduler - Failed to run runJob at
ReceiverTracker.scala:275
sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.DAGScheduler -
Removing running stage 0
sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.DAGScheduler -
Removing running stage 0
sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.DAGScheduler -
After removal of stage 0, remaining stages = 0
sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.DAGScheduler -
After removal of stage 0, remaining stages = 0
Executor task launch worker-0-SendThread(node.stratio.com:2181) DEBUG
zookeeper.ClientCnxn - Reading reply sessionid:0x1496007e6710002, packet::
clientPath:null serverPath:null finished:false header:: 1,1 replyHeader::
1,25,-101 request::
'/consumers/stratioStreaming/ids/stratioStreaming_ajn-stratio-1414656492293-8ecb3e3a,#7b2276657273696f6e223a312c22737562736372697074696f6e223a7b227374726174696f5f73747265616d696e675f616374696f6e223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343134363536343932343737227d,v{s{31,s{'world,'anyone}}},1
response::
{code}
> PARSING_ERROR(2) when upgrading issues from 1.0.2 to 1.1.0
> ----------------------------------------------------------
>
> Key: SPARK-4133
> URL: https://issues.apache.org/jira/browse/SPARK-4133
> Project: Spark
> Issue Type: Bug
> Components: Streaming
> Affects Versions: 1.1.0
> Reporter: Antonio Jesus Navarro
> Priority: Blocker
>
> Snappy related problems found when trying to upgrade existing Spark Streaming
> App from 1.0.2 to 1.1.0.
> We can not run an existing 1.0.2 spark app if upgraded to 1.1.0
> > IOException is thrown by snappy (parsing_error(2))
> {code}
> Executor task launch worker-0 DEBUG storage.BlockManager - Getting local
> block broadcast_0
> Executor task launch worker-0 DEBUG storage.BlockManager - Level for block
> broadcast_0 is StorageLevel(true, true, false, true, 1)
> Executor task launch worker-0 DEBUG storage.BlockManager - Getting block
> broadcast_0 from memory
> Executor task launch worker-0 DEBUG storage.BlockManager - Getting local
> block broadcast_0
> Executor task launch worker-0 DEBUG executor.Executor - Task 0's epoch is 0
> Executor task launch worker-0 DEBUG storage.BlockManager - Block broadcast_0
> not registered locally
> Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Started
> reading broadcast variable 0
> sparkDriver-akka.actor.default-dispatcher-4 INFO
> receiver.ReceiverSupervisorImpl - Registered receiver 0
> Executor task launch worker-0 INFO util.RecurringTimer - Started timer for
> BlockGenerator at time 1414656492400
> Executor task launch worker-0 INFO receiver.BlockGenerator - Started
> BlockGenerator
> Thread-87 INFO receiver.BlockGenerator - Started block pushing thread
> Executor task launch worker-0 INFO receiver.ReceiverSupervisorImpl -
> Starting receiver
> sparkDriver-akka.actor.default-dispatcher-5 INFO scheduler.ReceiverTracker -
> Registered receiver for stream 0 from akka://sparkDriver
> Executor task launch worker-0 INFO kafka.KafkaReceiver - Starting Kafka
> Consumer Stream with group: stratioStreaming
> Executor task launch worker-0 INFO kafka.KafkaReceiver - Connecting to
> Zookeeper: node.stratio.com:2181
> sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor]
> received message StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0
> cap=0]) from Actor[akka://sparkDriver/deadLetters]
> sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor]
> received message StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0
> cap=0]) from Actor[akka://sparkDriver/deadLetters]
> sparkDriver-akka.actor.default-dispatcher-6 DEBUG local.LocalActor - [actor]
> received message StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0
> cap=0]) from Actor[akka://sparkDriver/deadLetters]
> sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor]
> handled message (8.442354 ms)
> StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from
> Actor[akka://sparkDriver/deadLetters]
> sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor]
> handled message (8.412421 ms)
> StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from
> Actor[akka://sparkDriver/deadLetters]
> sparkDriver-akka.actor.default-dispatcher-6 DEBUG local.LocalActor - [actor]
> handled message (8.385471 ms)
> StatusUpdate(0,RUNNING,java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]) from
> Actor[akka://sparkDriver/deadLetters]
> Executor task launch worker-0 INFO utils.VerifiableProperties - Verifying
> properties
> Executor task launch worker-0 INFO utils.VerifiableProperties - Property
> group.id is overridden to stratioStreaming
> Executor task launch worker-0 INFO utils.VerifiableProperties - Property
> zookeeper.connect is overridden to node.stratio.com:2181
> Executor task launch worker-0 INFO utils.VerifiableProperties - Property
> zookeeper.connection.timeout.ms is overridden to 10000
> Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Reading
> broadcast variable 0 took 0.033998997 s
> Executor task launch worker-0 INFO consumer.ZookeeperConsumerConnector -
> [stratioStreaming_ajn-stratio-1414656492293-8ecb3e3a], Connecting to
> zookeeper instance at node.stratio.com:2181
> Executor task launch worker-0 DEBUG zkclient.ZkConnection - Creating new
> ZookKeeper instance to connect to node.stratio.com:2181.
> ZkClient-EventThread-169-node.stratio.com:2181 INFO zkclient.ZkEventThread -
> Starting ZkClient event thread.
> Executor task launch worker-0 INFO zookeeper.ZooKeeper - Initiating client
> connection, connectString=node.stratio.com:2181 sessionTimeout=6000
> watcher=org.I0Itec.zkclient.ZkClient@5b4bdc81
> Executor task launch worker-0 DEBUG zkclient.ZkClient - Awaiting connection
> to Zookeeper server
> Executor task launch worker-0 DEBUG zkclient.ZkClient - Waiting for keeper
> state SyncConnected
> Executor task launch worker-0-SendThread(node.stratio.com:2181) INFO
> zookeeper.ClientCnxn - Opening socket connection to server
> node.stratio.com/172.19.0.96:2181. Will not attempt to authenticate using
> SASL (unknown error)
> Executor task launch worker-0-SendThread(node.stratio.com:2181) INFO
> zookeeper.ClientCnxn - Socket connection established to
> node.stratio.com/172.19.0.96:2181, initiating session
> Executor task launch worker-0-SendThread(node.stratio.com:2181) DEBUG
> zookeeper.ClientCnxn - Session establishment request sent on
> node.stratio.com/172.19.0.96:2181
> Executor task launch worker-0-SendThread(node.stratio.com:2181) INFO
> zookeeper.ClientCnxn - Session establishment complete on server
> node.stratio.com/172.19.0.96:2181, sessionid = 0x1496007e6710002, negotiated
> timeout = 6000
> Executor task launch worker-0-EventThread DEBUG zkclient.ZkClient - Received
> event: WatchedEvent state:SyncConnected type:None path:null
> Executor task launch worker-0-EventThread INFO zkclient.ZkClient - zookeeper
> state changed (SyncConnected)
> Executor task launch worker-0-EventThread DEBUG zkclient.ZkClient - Leaving
> process event
> Executor task launch worker-0 DEBUG zkclient.ZkClient - State is SyncConnected
> RecurringTimer - BlockGenerator DEBUG util.RecurringTimer - Callback for
> BlockGenerator called at time 1414656492400
> Executor task launch worker-0 DEBUG utils.KafkaScheduler - Initializing task
> scheduler.
> Executor task launch worker-0 INFO consumer.ZookeeperConsumerConnector -
> [stratioStreaming_ajn-stratio-1414656492293-8ecb3e3a], starting auto
> committer every 60000 ms
> Executor task launch worker-0 DEBUG utils.KafkaScheduler - Scheduling task
> kafka-consumer-autocommit with initial delay 60000 ms and period 60000 ms.
> Executor task launch worker-0 INFO kafka.KafkaReceiver - Connected to
> node.stratio.com:2181
> Executor task launch worker-0 DEBUG consumer.ZookeeperConsumerConnector -
> [stratioStreaming_ajn-stratio-1414656492293-8ecb3e3a], entering consume
> Executor task launch worker-0 INFO consumer.ZookeeperConsumerConnector -
> [stratioStreaming_ajn-stratio-1414656492293-8ecb3e3a], begin registering
> consumer stratioStreaming_ajn-stratio-1414656492293-8ecb3e3a in ZK
> Executor task launch worker-0 DEBUG storage.BlockManager - Getting local
> block broadcast_0
> Executor task launch worker-0 DEBUG storage.BlockManager - Block broadcast_0
> not registered locally
> Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Started
> reading broadcast variable 0
> Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Reading
> broadcast variable 0 took 5.5676E-5 s
> Executor task launch worker-0 ERROR executor.Executor - Exception in task 0.0
> in stage 0.0 (TID 0)
> java.io.IOException: PARSING_ERROR(2)
> at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
> at org.xerial.snappy.SnappyNative.uncompressedLength(Native Method)
> at org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:545)
> at
> org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:125)
> at
> org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:88)
> at org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:58)
> at
> org.apache.spark.io.SnappyCompressionCodec.compressedInputStream(CompressionCodec.scala:128)
> at
> org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:232)
> at
> org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
> at
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
> at
> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Executor task launch worker-0 ERROR executor.Executor - Exception in task 0.0
> in stage 0.0 (TID 0)
> java.io.IOException: PARSING_ERROR(2)
> at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
> at org.xerial.snappy.SnappyNative.uncompressedLength(Native Method)
> at org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:545)
> at
> org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:125)
> at
> org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:88)
> at org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:58)
> at
> org.apache.spark.io.SnappyCompressionCodec.compressedInputStream(CompressionCodec.scala:128)
> at
> org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:232)
> at
> org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
> at
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
> at
> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor]
> received message StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=2144
> cap=2144]) from Actor[akka://sparkDriver/deadLetters]
> sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor]
> received message StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=2144
> cap=2144]) from Actor[akka://sparkDriver/deadLetters]
> sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.TaskSchedulerImpl
> - parentName: , name: TaskSet_0, runningTasks: 0
> sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.TaskSchedulerImpl
> - parentName: , name: TaskSet_0, runningTasks: 0
> sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor]
> handled message (1.213476 ms)
> StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=2144 cap=2144]) from
> Actor[akka://sparkDriver/deadLetters]
> sparkDriver-akka.actor.default-dispatcher-2 DEBUG local.LocalActor - [actor]
> handled message (1.543991 ms)
> StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=2144 cap=2144]) from
> Actor[akka://sparkDriver/deadLetters]
> Result resolver thread-0 WARN scheduler.TaskSetManager - Lost task 0.0 in
> stage 0.0 (TID 0, localhost): java.io.IOException: PARSING_ERROR(2)
> org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
> org.xerial.snappy.SnappyNative.uncompressedLength(Native Method)
> org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:545)
>
> org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:125)
>
> org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:88)
> org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:58)
>
> org.apache.spark.io.SnappyCompressionCodec.compressedInputStream(CompressionCodec.scala:128)
>
> org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:232)
>
> org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke(Method.java:606)
>
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
>
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
>
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
>
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
>
> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> java.lang.Thread.run(Thread.java:745)
> Result resolver thread-0 WARN scheduler.TaskSetManager - Lost task 0.0 in
> stage 0.0 (TID 0, localhost): java.io.IOException: PARSING_ERROR(2)
> org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
> org.xerial.snappy.SnappyNative.uncompressedLength(Native Method)
> org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:545)
>
> org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:125)
>
> org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:88)
> org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:58)
>
> org.apache.spark.io.SnappyCompressionCodec.compressedInputStream(CompressionCodec.scala:128)
>
> org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:232)
>
> org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke(Method.java:606)
>
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
>
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
>
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
>
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
>
> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> java.lang.Thread.run(Thread.java:745)
> Result resolver thread-0 ERROR scheduler.TaskSetManager - Task 0 in stage 0.0
> failed 1 times; aborting job
> Result resolver thread-0 ERROR scheduler.TaskSetManager - Task 0 in stage 0.0
> failed 1 times; aborting job
> Result resolver thread-0 INFO scheduler.TaskSchedulerImpl - Removed TaskSet
> 0.0, whose tasks have all completed, from pool
> Result resolver thread-0 INFO scheduler.TaskSchedulerImpl - Removed TaskSet
> 0.0, whose tasks have all completed, from pool
> sparkDriver-akka.actor.default-dispatcher-2 INFO scheduler.TaskSchedulerImpl
> - Cancelling stage 0
> sparkDriver-akka.actor.default-dispatcher-2 INFO scheduler.TaskSchedulerImpl
> - Cancelling stage 0
> Thread-84 INFO scheduler.DAGScheduler - Failed to run runJob at
> ReceiverTracker.scala:275
> Thread-85 INFO scheduler.DAGScheduler - Failed to run runJob at
> ReceiverTracker.scala:275
> sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.DAGScheduler -
> Removing running stage 0
> sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.DAGScheduler -
> Removing running stage 0
> sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.DAGScheduler -
> After removal of stage 0, remaining stages = 0
> sparkDriver-akka.actor.default-dispatcher-2 DEBUG scheduler.DAGScheduler -
> After removal of stage 0, remaining stages = 0
> Executor task launch worker-0-SendThread(node.stratio.com:2181) DEBUG
> zookeeper.ClientCnxn - Reading reply sessionid:0x1496007e6710002, packet::
> clientPath:null serverPath:null finished:false header:: 1,1 replyHeader::
> 1,25,-101 request::
> '/consumers/stratioStreaming/ids/stratioStreaming_ajn-stratio-1414656492293-8ecb3e3a,#7b2276657273696f6e223a312c22737562736372697074696f6e223a7b227374726174696f5f73747265616d696e675f616374696f6e223a317d2c227061747465726e223a22737461746963222c2274696d657374616d70223a2231343134363536343932343737227d,v{s{31,s{'world,'anyone}}},1
> response::
> {code}
> > Only spark version changed
> As far as we have checked, snappy will throw this error when dealing with
> zero bytes length arrays.
> We have tried:
> > Changing from snappy to LZF,
> > Changing spark.broadcast.compress false
> {code}
> Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Reading
> broadcast variable 0 took 0.240869283 s
> sparkDriver-akka.actor.default-dispatcher-4 DEBUG kafka.KafkaInputDStream -
> Cleared 1 RDDs that were older than 1414657342000 ms: 1414657342000 ms
> sparkDriver-akka.actor.default-dispatcher-4 DEBUG kafka.KafkaInputDStream -
> Cleared 1 RDDs that were older than 1414657342000 ms: 1414657342000 ms
> sparkDriver-akka.actor.default-dispatcher-4 DEBUG streaming.DStreamGraph -
> Cleared old metadata for time 1414657344000 ms
> sparkDriver-akka.actor.default-dispatcher-13 DEBUG
> storage.BlockManagerSlaveActor - removing RDD 3
> Executor task launch worker-1 DEBUG storage.BlockManager - Getting local
> block broadcast_1
> sparkDriver-akka.actor.default-dispatcher-4 DEBUG dstream.MappedDStream -
> Time 1414657344000 ms is valid
> sparkDriver-akka.actor.default-dispatcher-3 DEBUG dstream.FilteredDStream -
> Time 1414657344000 ms is valid
> sparkDriver-akka.actor.default-dispatcher-4 DEBUG dstream.FilteredDStream -
> Time 1414657344000 ms is valid
> sparkDriver-akka.actor.default-dispatcher-13 INFO storage.BlockManager -
> Removing RDD 3
> sparkDriver-akka.actor.default-dispatcher-2 DEBUG
> storage.BlockManagerSlaveActor - [actor] handled message (134.08408 ms)
> RemoveRdd(3) from Actor[akka://sparkDriver/temp/$f]
> sparkDriver-akka.actor.default-dispatcher-2 DEBUG
> storage.BlockManagerSlaveActor - [actor] received message RemoveRdd(2) from
> Actor[akka://sparkDriver/temp/$i]
> sparkDriver-akka.actor.default-dispatcher-4 DEBUG
> storage.BlockManagerSlaveActor - removing RDD 2
> sparkDriver-akka.actor.default-dispatcher-4 INFO storage.BlockManager -
> Removing RDD 2
> sparkDriver-akka.actor.default-dispatcher-2 DEBUG
> storage.BlockManagerSlaveActor - [actor] handled message (0.050955 ms)
> RemoveRdd(2) from Actor[akka://sparkDriver/temp/$i]
> sparkDriver-akka.actor.default-dispatcher-2 DEBUG
> storage.BlockManagerSlaveActor - [actor] received message RemoveRdd(1) from
> Actor[akka://sparkDriver/temp/$j]
> sparkDriver-akka.actor.default-dispatcher-5 DEBUG
> storage.BlockManagerSlaveActor - removing RDD 1
> sparkDriver-akka.actor.default-dispatcher-5 INFO storage.BlockManager -
> Removing RDD 1
> sparkDriver-akka.actor.default-dispatcher-2 DEBUG
> storage.BlockManagerSlaveActor - [actor] handled message (0.037738 ms)
> RemoveRdd(1) from Actor[akka://sparkDriver/temp/$j]
> Executor task launch worker-0 ERROR executor.Executor - Exception in task 0.0
> in stage 0.0 (TID 0)
> java.io.EOFException
> at
> java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2325)
> at
> java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2794)
> at
> java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801)
> at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
> at
> org.apache.spark.serializer.JavaDeserializationStream$$anon$1.<init>(JavaSerializer.scala:57)
> at
> org.apache.spark.serializer.JavaDeserializationStream.<init>(JavaSerializer.scala:57)
> at
> org.apache.spark.serializer.JavaSerializerInstance.deserializeStream(JavaSerializer.scala:95)
> at
> org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:235)
> at
> org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
> at
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
> at
> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> sparkDriver-akka.actor.default-dispatcher-5 DEBUG
> storage.BlockManagerSlaveActor - Done removing RDD 1, response is 0
> sparkDriver-akka.actor.default-dispatcher-2 DEBUG
> storage.BlockManagerSlaveActor - Done removing RDD 2, response is 0
> sparkDriver-akka.actor.default-dispatcher-3 DEBUG
> storage.BlockManagerSlaveActor - Done removing RDD 3, response is 0
> Executor task launch worker-1 DEBUG storage.BlockManager - Level for block
> broadcast_1 is StorageLevel(true, true, false, true, 1)
> sparkDriver-akka.actor.default-dispatcher-3 DEBUG dstream.FilteredDStream -
> Time 1414657344000 ms is valid
> sparkDriver-akka.actor.default-dispatcher-4 DEBUG dstream.MappedDStream -
> Time 1414657344000 ms is valid
> sparkDriver-akka.actor.default-dispatcher-4 DEBUG dstream.FilteredDStream -
> Time 1414657344000 ms is valid
> sparkDriver-akka.actor.default-dispatcher-3 DEBUG streaming.DStreamGraph -
> Generated 4 jobs for time 1414657344000 ms
> sparkDriver-akka.actor.default-dispatcher-3 INFO scheduler.JobScheduler -
> Added jobs for time 1414657344000 ms
> sparkDriver-akka.actor.default-dispatcher-3 DEBUG scheduler.JobGenerator -
> Got event DoCheckpoint(1414657344000 ms)
> sparkDriver-akka.actor.default-dispatcher-5 DEBUG
> storage.BlockManagerSlaveActor - Sent response: 0 to
> Actor[akka://sparkDriver/temp/$j]
> sparkDriver-akka.actor.default-dispatcher-2 DEBUG
> storage.BlockManagerSlaveActor - Sent response: 0 to
> Actor[akka://sparkDriver/temp/$i]
> Executor task launch worker-1 DEBUG storage.BlockManager - Getting block
> broadcast_1 from memory
> sparkDriver-akka.actor.default-dispatcher-3 DEBUG
> storage.BlockManagerSlaveActor - Sent response: 0 to
> Actor[akka://sparkDriver/temp/$f]
> Executor task launch worker-0 DEBUG storage.BlockManager - Getting local
> block broadcast_0
> Executor task launch worker-0 DEBUG storage.BlockManager - Level for block
> broadcast_0 is StorageLevel(true, true, false, true, 1)
> Executor task launch worker-0 DEBUG storage.BlockManager - Getting block
> broadcast_0 from memory
> Executor task launch worker-0 DEBUG executor.Executor - Task 0's epoch is 0
> Executor task launch worker-1 DEBUG executor.Executor - Task 1's epoch is 0
> Executor task launch worker-0 DEBUG storage.BlockManager - Getting local
> block broadcast_0
> Executor task launch worker-0 DEBUG storage.BlockManager - Block broadcast_0
> not registered locally
> Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Started
> reading broadcast variable 0
> sparkDriver-akka.actor.default-dispatcher-4 DEBUG dstream.MappedDStream -
> Time 1414657344000 ms is valid
> sparkDriver-akka.actor.default-dispatcher-4 DEBUG dstream.FilteredDStream -
> Time 1414657344000 ms is valid
> Executor task launch worker-0 INFO broadcast.TorrentBroadcast - Reading
> broadcast variable 0 took 7.0321E-5 s
> Executor task launch worker-0 ERROR executor.Executor - Exception in task 0.0
> in stage 0.0 (TID 0)
> java.io.EOFException
> at
> java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2325)
> at
> java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2794)
> at
> java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801)
> at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
> at
> org.apache.spark.serializer.JavaDeserializationStream$$anon$1.<init>(JavaSerializer.scala:57)
> at
> org.apache.spark.serializer.JavaDeserializationStream.<init>(JavaSerializer.scala:57)
> at
> org.apache.spark.serializer.JavaSerializerInstance.deserializeStream(JavaSerializer.scala:95)
> at
> org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:235)
> at
> org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
> at
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
> at
> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> sparkDriver-akka.actor.default-dispatcher-5 DEBUG local.LocalActor - [actor]
> received message StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=2066
> cap=2066]) from Actor[akka://sparkDriver/deadLetters]
> sparkDriver-akka.actor.default-dispatcher-3 DEBUG local.LocalActor - [actor]
> received message StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=2066
> cap=2066]) from Actor[akka://sparkDriver/deadLetters]
> sparkDriver-akka.actor.default-dispatcher-5 DEBUG scheduler.TaskSchedulerImpl
> - parentName: , name: TaskSet_0, runningTasks: 0
> sparkDriver-akka.actor.default-dispatcher-3 DEBUG scheduler.TaskSchedulerImpl
> - parentName: , name: TaskSet_0, runningTasks: 0
> sparkDriver-akka.actor.default-dispatcher-5 DEBUG local.LocalActor - [actor]
> handled message (1.681797 ms)
> StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=2066 cap=2066]) from
> Actor[akka://sparkDriver/deadLetters]
> sparkDriver-akka.actor.default-dispatcher-3 DEBUG local.LocalActor - [actor]
> handled message (0.688875 ms)
> StatusUpdate(0,FAILED,java.nio.HeapByteBuffer[pos=0 lim=2066 cap=2066]) from
> Actor[akka://sparkDriver/deadLetters]
> sparkDriver-akka.actor.default-dispatcher-4 DEBUG dstream.MappedDStream -
> Time 1414657344000 ms is valid
> sparkDriver-akka.actor.default-dispatcher-4 DEBUG dstream.FilteredDStream -
> Time 1414657344000 ms is valid
> sparkDriver-akka.actor.default-dispatcher-5 DEBUG spark.HeartbeatReceiver -
> [actor] received message
> Heartbeat(localhost,[Lscala.Tuple2;@1a94802,BlockManagerId(<driver>,
> ajn-stratio.local, 53377, 0)) from Actor[akka://sparkDriver/temp/$c]
> sparkDriver-akka.actor.default-dispatcher-3 INFO
> receiver.ReceiverSupervisorImpl - Registered receiver 0
> Executor task launch worker-0 INFO util.RecurringTimer - Started timer for
> BlockGenerator at time 1414657344800
> Executor task launch worker-0 INFO receiver.BlockGenerator - Started
> BlockGenerator
> Executor task launch worker-0 INFO receiver.ReceiverSupervisorImpl -
> Starting receiver
> Thread-87 INFO receiver.BlockGenerator - Started block pushing thread
> Result resolver thread-0 WARN scheduler.TaskSetManager - Lost task 0.0 in
> stage 0.0 (TID 0, localhost): java.io.EOFException:
>
> java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2325)
>
> java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2794)
> java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801)
> java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
>
> org.apache.spark.serializer.JavaDeserializationStream$$anon$1.<init>(JavaSerializer.scala:57)
>
> org.apache.spark.serializer.JavaDeserializationStream.<init>(JavaSerializer.scala:57)
>
> org.apache.spark.serializer.JavaSerializerInstance.deserializeStream(JavaSerializer.scala:95)
>
> org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:235)
>
> org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke(Method.java:606)
>
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
>
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
>
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
>
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
>
> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> java.lang.Thread.run(Thread.java:745)
> Result resolver thread-0 WARN scheduler.TaskSetManager - Lost task 0.0 in
> stage 0.0 (TID 0, localhost): java.io.EOFException:
>
> java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2325)
>
> java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2794)
> java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801)
> java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
>
> org.apache.spark.serializer.JavaDeserializationStream$$anon$1.<init>(JavaSerializer.scala:57)
>
> org.apache.spark.serializer.JavaDeserializationStream.<init>(JavaSerializer.scala:57)
>
> org.apache.spark.serializer.JavaSerializerInstance.deserializeStream(JavaSerializer.scala:95)
>
> org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:235)
>
> org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke(Method.java:606)
>
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
>
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
>
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
>
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
>
> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> java.lang.Thread.run(Thread.java:745)
> Executor task launch worker-0 INFO kafka.KafkaReceiver - Starting Kafka
> Consumer Stream with group: stratioStreaming
> Executor task launch worker-0 INFO kafka.KafkaReceiver - Connecting to
> Zookeeper: node.stratio.com:2181
> sparkDriver-akka.actor.default-dispatcher-4 DEBUG dstream.MappedDStream -
> Time 1414657344000 ms is valid
> sparkDriver-akka.actor.default-dispatcher-4 DEBUG dstream.FilteredDStream -
> Time 1414657344000 ms is valid
> Result resolver thread-0 ERROR scheduler.TaskSetManager - Task 0 in stage 0.0
> failed 1 times; aborting job
> Result resolver thread-0 ERROR scheduler.TaskSetManager - Task 0 in stage 0.0
> failed 1 times; aborting job
> sparkDriver-akka.actor.default-dispatcher-2 INFO scheduler.ReceiverTracker -
> Registered receiver for stream 0 from akka://sparkDriver
> Result resolver thread-0 INFO scheduler.TaskSchedulerImpl - Removed TaskSet
> 0.0, whose tasks have all completed, from pool
> Result resolver thread-0 INFO scheduler.TaskSchedulerImpl - Removed TaskSet
> 0.0, whose tasks have all completed, from pool
> sparkDriver-akka.actor.default-dispatcher-3 DEBUG
> storage.BlockManagerMasterActor - [actor] received message
> BlockManagerHeartbeat(BlockManagerId(<driver>, ajn-stratio.local, 53377, 0))
> from Actor[akka://sparkDriver/temp/$d]
> sparkDriver-akka.actor.default-dispatcher-3 DEBUG
> storage.BlockManagerMasterActor - [actor] handled message (0.197908 ms)
> BlockManagerHeartbeat(BlockManagerId(<driver>, ajn-stratio.local, 53377, 0))
> from Actor[akka://sparkDriver/temp/$d]
> sparkDriver-akka.actor.default-dispatcher-5 DEBUG spark.HeartbeatReceiver -
> [actor] handled message (169.804965 ms)
> Heartbeat(localhost,[Lscala.Tuple2;@1a94802,BlockManagerId(<driver>,
> ajn-stratio.local, 53377, 0)) from Actor[akka://sparkDriver/temp/$c]
> {code}
> > Changing from TorrentBroadcast to HTTPBroadcast.
> but with no luck for the moment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
