Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/6625#discussion_r31777046
--- Diff:
core/src/main/scala/org/apache/spark/serializer/SerializationDebugger.scala ---
@@ -169,18 +208,53 @@ private[spark] object SerializationDebugger extends
Logging {
}
j += 1
}
-
}
i += 1
}
return List.empty
}
+
+ /**
+ * Visit a serializable object which has the writeObject() defined.
+ * Since writeObject() can choose add arbitrary objects at the time of
serialization,
+ * the only way to capture all the objects it will serialize is by
using a
+ * dummy ObjectOutputStream that collects all the relevant fields for
further testing.
+ */
+ private def visitSerializableWithWriteObjectMethod(
+ o: Object, stack: List[String]): List[String] = {
+ val innerObjectsCatcher = new ListObjectOutputStream
+ var notSerializableFound = false
+ try {
+ innerObjectsCatcher.writeObject(o)
+ } catch {
+ case io: IOException =>
+ notSerializableFound = true
+ }
+
+ // If something was not serializable, then visit the captured
objects.
+ // Otherwise, all the captured objects are safely serializable, so
no need to visit them.
+ // As an optimization, just added them to the visited list.
--- End diff --
then down there in L248 you can optionally add the optimization comment
(though I don't think it's necessary)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
