[
https://issues.apache.org/jira/browse/SPARK-11751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-11751.
-------------------------------
Resolution: Fixed
Fix Version/s: 1.6.0
Issue resolved by pull request 9734
[https://github.com/apache/spark/pull/9734]
> Doc describe error in the "Spark Streaming Programming Guide" page
> ------------------------------------------------------------------
>
> Key: SPARK-11751
> URL: https://issues.apache.org/jira/browse/SPARK-11751
> Project: Spark
> Issue Type: Documentation
> Components: Documentation
> Affects Versions: 1.4.1, 1.5.0, 1.5.1, 1.5.2
> Reporter: yangping wu
> Priority: Trivial
> Fix For: 1.6.0
>
>
> In the *Task Launching Overheads* section,
> {quote}*Task Serialization*: Using Kryo serialization for serializing tasks
> can reduce the task sizes, and therefore reduce the time taken to send them
> to the slaves.{quote}
> As we known *Task Serialization* is configuration by
> *spark.closure.serializer* parameter, but currently only the Java serializer
> is supported. If we set *spark.closure.serializer* to
> *org.apache.spark.serializer.KryoSerializer*, then this will throw a
> exception as follow:
> {code}
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 516
> in stage 0.0 failed 4 times, most recent failure: Lost task 516.3 in stage
> 0.0 (TID 21, spark-cluster.data.com): java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at
> org.apache.spark.scheduler.Task$.deserializeWithDependencies(Task.scala:188)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:192)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Driver stacktrace:
> at
> org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1273)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1264)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1263)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
> at
> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1263)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:730)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:730)
> at scala.Option.foreach(Option.scala:236)
> at
> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:730)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1457)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1418)
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
