[
https://issues.apache.org/jira/browse/SPARK-32964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Jie updated SPARK-32964:
-----------------------------
Description:
There is only one failed case of `streaming` module in Scala 2.13:
* `start with non-serializable DStream checkpoint ` in StreamingContextSuite
StackOverflowError is thrown here when SerializationDebugger#visit method is
called.
The error stack as follow:
{code:java}
Expected exception java.io.NotSerializableException to be thrown, but
java.lang.StackOverflowError was thrownExpected exception
java.io.NotSerializableException to be thrown, but java.lang.StackOverflowError
was thrownScalaTestFailureLocation:
org.apache.spark.streaming.StreamingContextSuite at
(StreamingContextSuite.scala:159)org.scalatest.exceptions.TestFailedException:
Expected exception java.io.NotSerializableException to be thrown, but
java.lang.StackOverflowError was thrown at
org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472) at
org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471) at
org.scalatest.funsuite.AnyFunSuite.newAssertionFailedException(AnyFunSuite.scala:1562)
at org.scalatest.Assertions.intercept(Assertions.scala:756) at
org.scalatest.Assertions.intercept$(Assertions.scala:746) at
org.scalatest.funsuite.AnyFunSuite.intercept(AnyFunSuite.scala:1562) at
org.apache.spark.streaming.StreamingContextSuite.$anonfun$new$13(StreamingContextSuite.scala:159)
...Caused by: java.lang.StackOverflowError at
org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1397) at
org.apache.spark.streaming.dstream.DStream.writeObject(DStream.scala:513) at
sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1140) at
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496) at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) at
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at
java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) at
scala.collection.generic.DefaultSerializationProxy.$anonfun$writeObject$1(DefaultSerializationProxy.scala:38)
at
scala.collection.generic.DefaultSerializationProxy.$anonfun$writeObject$1$adapted(DefaultSerializationProxy.scala:37)
at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:553) at
scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:551) at
scala.collection.AbstractIterable.foreach(Iterable.scala:920) at
scala.collection.generic.DefaultSerializationProxy.writeObject(DefaultSerializationProxy.scala:37)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1140) at
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496) at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) at
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at
java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) at
org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visitSerializableWithWriteObjectMethod(SerializationDebugger.scala:230)
at
org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visitSerializable(SerializationDebugger.scala:189)
at
org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visit(SerializationDebugger.scala:108)
at
org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visitSerializableWithWriteObjectMethod(SerializationDebugger.scala:243)
at
org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visitSerializable(SerializationDebugger.scala:189)
at
org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visit(SerializationDebugger.scala:108)
at
org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visitSerializableWithWriteObjectMethod(SerializationDebugger.scala:243)
at
org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visitSerializable(SerializationDebugger.scala:189)
at
org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visit(SerializationDebugger.scala:108)
{code}
was:
There is only one failed case of `streaming` module in Scala 2.13:
* `start with non-serializable DStream checkpoint ` in StreamingContextSuite
StackOverflowError is thrown here when SerializationDebugger#visit method is
called.
The error msg as follow:
{code:java}
Expected exception java.io.NotSerializableException to be thrown, but
java.lang.StackOverflowError was thrownExpected exception
java.io.NotSerializableException to be thrown, but java.lang.StackOverflowError
was thrownScalaTestFailureLocation:
org.apache.spark.streaming.StreamingContextSuite at
(StreamingContextSuite.scala:159)org.scalatest.exceptions.TestFailedException:
Expected exception java.io.NotSerializableException to be thrown, but
java.lang.StackOverflowError was thrown at
org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472) at
org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471) at
org.scalatest.funsuite.AnyFunSuite.newAssertionFailedException(AnyFunSuite.scala:1562)
at org.scalatest.Assertions.intercept(Assertions.scala:756) at
org.scalatest.Assertions.intercept$(Assertions.scala:746) at
org.scalatest.funsuite.AnyFunSuite.intercept(AnyFunSuite.scala:1562) at
org.apache.spark.streaming.StreamingContextSuite.$anonfun$new$13(StreamingContextSuite.scala:159)
...Caused by: java.lang.StackOverflowError at
org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1397) at
org.apache.spark.streaming.dstream.DStream.writeObject(DStream.scala:513) at
sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1140) at
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496) at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) at
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at
java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) at
scala.collection.generic.DefaultSerializationProxy.$anonfun$writeObject$1(DefaultSerializationProxy.scala:38)
at
scala.collection.generic.DefaultSerializationProxy.$anonfun$writeObject$1$adapted(DefaultSerializationProxy.scala:37)
at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:553) at
scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:551) at
scala.collection.AbstractIterable.foreach(Iterable.scala:920) at
scala.collection.generic.DefaultSerializationProxy.writeObject(DefaultSerializationProxy.scala:37)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1140) at
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496) at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) at
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at
java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) at
org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visitSerializableWithWriteObjectMethod(SerializationDebugger.scala:230)
at
org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visitSerializable(SerializationDebugger.scala:189)
at
org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visit(SerializationDebugger.scala:108)
at
org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visitSerializableWithWriteObjectMethod(SerializationDebugger.scala:243)
at
org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visitSerializable(SerializationDebugger.scala:189)
at
org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visit(SerializationDebugger.scala:108)
at
org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visitSerializableWithWriteObjectMethod(SerializationDebugger.scala:243)
at
org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visitSerializable(SerializationDebugger.scala:189)
at
org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visit(SerializationDebugger.scala:108)
at
org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visitSerializableWithWriteObjectMethod(SerializationDebugger.scala:243)
at
org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visitSerializable(SerializationDebugger.scala:189)
{code}
> Pass all `streaming` module UTs in Scala 2.13
> ---------------------------------------------
>
> Key: SPARK-32964
> URL: https://issues.apache.org/jira/browse/SPARK-32964
> Project: Spark
> Issue Type: Sub-task
> Components: DStreams, Spark Core
> Affects Versions: 3.1.0
> Reporter: Yang Jie
> Priority: Minor
>
> There is only one failed case of `streaming` module in Scala 2.13:
> * `start with non-serializable DStream checkpoint ` in StreamingContextSuite
> StackOverflowError is thrown here when SerializationDebugger#visit method is
> called.
> The error stack as follow:
> {code:java}
> Expected exception java.io.NotSerializableException to be thrown, but
> java.lang.StackOverflowError was thrownExpected exception
> java.io.NotSerializableException to be thrown, but
> java.lang.StackOverflowError was thrownScalaTestFailureLocation:
> org.apache.spark.streaming.StreamingContextSuite at
> (StreamingContextSuite.scala:159)org.scalatest.exceptions.TestFailedException:
> Expected exception java.io.NotSerializableException to be thrown, but
> java.lang.StackOverflowError was thrown at
> org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472) at
> org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
> at
> org.scalatest.funsuite.AnyFunSuite.newAssertionFailedException(AnyFunSuite.scala:1562)
> at org.scalatest.Assertions.intercept(Assertions.scala:756) at
> org.scalatest.Assertions.intercept$(Assertions.scala:746) at
> org.scalatest.funsuite.AnyFunSuite.intercept(AnyFunSuite.scala:1562) at
> org.apache.spark.streaming.StreamingContextSuite.$anonfun$new$13(StreamingContextSuite.scala:159)
> ...Caused by: java.lang.StackOverflowError at
> org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1397) at
> org.apache.spark.streaming.dstream.DStream.writeObject(DStream.scala:513) at
> sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1140) at
> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496) at
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at
> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) at
> scala.collection.generic.DefaultSerializationProxy.$anonfun$writeObject$1(DefaultSerializationProxy.scala:38)
> at
> scala.collection.generic.DefaultSerializationProxy.$anonfun$writeObject$1$adapted(DefaultSerializationProxy.scala:37)
> at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:553) at
> scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:551) at
> scala.collection.AbstractIterable.foreach(Iterable.scala:920) at
> scala.collection.generic.DefaultSerializationProxy.writeObject(DefaultSerializationProxy.scala:37)
> at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source) at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1140) at
> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496) at
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at
> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) at
> org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visitSerializableWithWriteObjectMethod(SerializationDebugger.scala:230)
> at
> org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visitSerializable(SerializationDebugger.scala:189)
> at
> org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visit(SerializationDebugger.scala:108)
> at
> org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visitSerializableWithWriteObjectMethod(SerializationDebugger.scala:243)
> at
> org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visitSerializable(SerializationDebugger.scala:189)
> at
> org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visit(SerializationDebugger.scala:108)
> at
> org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visitSerializableWithWriteObjectMethod(SerializationDebugger.scala:243)
> at
> org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visitSerializable(SerializationDebugger.scala:189)
> at
> org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visit(SerializationDebugger.scala:108)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
