[
https://issues.apache.org/jira/browse/SPARK-14540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16180658#comment-16180658
]
Sean Owen commented on SPARK-14540:
-----------------------------------
I might have spoken too soon. After solving some other 2.12 issues, I am now
facing this:
{code}
[ERROR] Tests run: 18, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.165
s <<< FAILURE! - in test.org.apache.spark.Java8RDDAPISuite
[ERROR] foldByKey(test.org.apache.spark.Java8RDDAPISuite) Time elapsed: 0.084
s <<< ERROR!
org.apache.spark.SparkException:
Job aborted due to stage failure: Task not serializable:
java.io.NotSerializableException: scala.runtime.LazyRef
Serialization stack:
- object not serializable (class: scala.runtime.LazyRef, value: LazyRef
thunk)
- element of array (index: 2)
- array (class [Ljava.lang.Object;, size 3)
- field (class: java.lang.invoke.SerializedLambda, name: capturedArgs,
type: class [Ljava.lang.Object;)
- object (class java.lang.invoke.SerializedLambda,
SerializedLambda[capturingClass=class org.apache.spark.rdd.PairRDDFunctions,
functionalInterfaceMethod=scala/Function0.apply:()Ljava/lang/Object;,
implementation=invokeStatic
org/apache/spark/rdd/PairRDDFunctions.$anonfun$foldByKey$2:(Lorg/apache/spark/rdd/PairRDDFunctions;[BLscala/runtime/LazyRef;)Ljava/lang/Object;,
instantiatedMethodType=()Ljava/lang/Object;, numCaptured=3])
- writeReplace data (class: java.lang.invoke.SerializedLambda)
- object (class
org.apache.spark.rdd.PairRDDFunctions$$Lambda$1249/2053647669,
org.apache.spark.rdd.PairRDDFunctions$$Lambda$1249/2053647669@2bf19860)
- element of array (index: 0)
- array (class [Ljava.lang.Object;, size 2)
- field (class: java.lang.invoke.SerializedLambda, name: capturedArgs,
type: class [Ljava.lang.Object;)
- object (class java.lang.invoke.SerializedLambda,
SerializedLambda[capturingClass=class org.apache.spark.rdd.PairRDDFunctions,
functionalInterfaceMethod=scala/Function1.apply:(Ljava/lang/Object;)Ljava/lang/Object;,
implementation=invokeStatic
org/apache/spark/rdd/PairRDDFunctions.$anonfun$foldByKey$3:(Lscala/Function0;Lscala/Function2;Ljava/lang/Object;)Ljava/lang/Object;,
instantiatedMethodType=(Ljava/lang/Object;)Ljava/lang/Object;, numCaptured=2])
- writeReplace data (class: java.lang.invoke.SerializedLambda)
- object (class
org.apache.spark.rdd.PairRDDFunctions$$Lambda$1250/250144767,
org.apache.spark.rdd.PairRDDFunctions$$Lambda$1250/250144767@36d4186f)
- field (class: org.apache.spark.Aggregator, name: createCombiner,
type: interface scala.Function1)
- object (class org.apache.spark.Aggregator,
Aggregator(org.apache.spark.rdd.PairRDDFunctions$$Lambda$1250/250144767@36d4186f,org.apache.spark.api.java.JavaPairRDD$$$Lambda$832/1799521220@551d5576,org.apache.spark.api.java.JavaPairRDD$$$Lambda$832/1799521220@551d5576))
- field (class: scala.Some, name: value, type: class java.lang.Object)
- object (class scala.Some,
Some(Aggregator(org.apache.spark.rdd.PairRDDFunctions$$Lambda$1250/250144767@36d4186f,org.apache.spark.api.java.JavaPairRDD$$$Lambda$832/1799521220@551d5576,org.apache.spark.api.java.JavaPairRDD$$$Lambda$832/1799521220@551d5576)))
- field (class: org.apache.spark.ShuffleDependency, name: aggregator,
type: class scala.Option)
- object (class org.apache.spark.ShuffleDependency,
org.apache.spark.ShuffleDependency@aa278a4)
- field (class: scala.Tuple2, name: _2, type: class java.lang.Object)
- object (class scala.Tuple2, (ParallelCollectionRDD[0] at
parallelizePairs at
Java8RDDAPISuite.java:137,org.apache.spark.ShuffleDependency@aa278a4))
at
test.org.apache.spark.Java8RDDAPISuite.foldByKey(Java8RDDAPISuite.java:139)
{code}
This might be the current manifestation of the same problem, not sure.
> Support Scala 2.12 closures and Java 8 lambdas in ClosureCleaner
> ----------------------------------------------------------------
>
> Key: SPARK-14540
> URL: https://issues.apache.org/jira/browse/SPARK-14540
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Reporter: Josh Rosen
>
> Using https://github.com/JoshRosen/spark/tree/build-for-2.12, I tried running
> ClosureCleanerSuite with Scala 2.12 and ran into two bad test failures:
> {code}
> [info] - toplevel return statements in closures are identified at cleaning
> time *** FAILED *** (32 milliseconds)
> [info] Expected exception
> org.apache.spark.util.ReturnStatementInClosureException to be thrown, but no
> exception was thrown. (ClosureCleanerSuite.scala:57)
> {code}
> and
> {code}
> [info] - user provided closures are actually cleaned *** FAILED *** (56
> milliseconds)
> [info] Expected ReturnStatementInClosureException, but got
> org.apache.spark.SparkException: Job aborted due to stage failure: Task not
> serializable: java.io.NotSerializableException: java.lang.Object
> [info] - element of array (index: 0)
> [info] - array (class "[Ljava.lang.Object;", size: 1)
> [info] - field (class "java.lang.invoke.SerializedLambda", name:
> "capturedArgs", type: "class [Ljava.lang.Object;")
> [info] - object (class "java.lang.invoke.SerializedLambda",
> SerializedLambda[capturingClass=class
> org.apache.spark.util.TestUserClosuresActuallyCleaned$,
> functionalInterfaceMethod=scala/runtime/java8/JFunction1$mcII$sp.apply$mcII$sp:(I)I,
> implementation=invokeStatic
> org/apache/spark/util/TestUserClosuresActuallyCleaned$.org$apache$spark$util$TestUserClosuresActuallyCleaned$$$anonfun$69:(Ljava/lang/Object;I)I,
> instantiatedMethodType=(I)I, numCaptured=1])
> [info] - element of array (index: 0)
> [info] - array (class "[Ljava.lang.Object;", size: 1)
> [info] - field (class "java.lang.invoke.SerializedLambda", name:
> "capturedArgs", type: "class [Ljava.lang.Object;")
> [info] - object (class "java.lang.invoke.SerializedLambda",
> SerializedLambda[capturingClass=class org.apache.spark.rdd.RDD,
> functionalInterfaceMethod=scala/Function3.apply:(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;,
> implementation=invokeStatic
> org/apache/spark/rdd/RDD.org$apache$spark$rdd$RDD$$$anonfun$20$adapted:(Lscala/Function1;Lorg/apache/spark/TaskContext;Ljava/lang/Object;Lscala/collection/Iterator;)Lscala/collection/Iterator;,
>
> instantiatedMethodType=(Lorg/apache/spark/TaskContext;Ljava/lang/Object;Lscala/collection/Iterator;)Lscala/collection/Iterator;,
> numCaptured=1])
> [info] - field (class "org.apache.spark.rdd.MapPartitionsRDD", name:
> "f", type: "interface scala.Function3")
> [info] - object (class "org.apache.spark.rdd.MapPartitionsRDD",
> MapPartitionsRDD[2] at apply at Transformer.scala:22)
> [info] - field (class "scala.Tuple2", name: "_1", type: "class
> java.lang.Object")
> [info] - root object (class "scala.Tuple2", (MapPartitionsRDD[2] at
> apply at
> Transformer.scala:22,org.apache.spark.SparkContext$$Lambda$957/431842435@6e803685)).
> [info] This means the closure provided by user is not actually cleaned.
> (ClosureCleanerSuite.scala:78)
> {code}
> We'll need to figure out a closure cleaning strategy which works for 2.12
> lambdas.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
