Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/6034#discussion_r30095833
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/dstream/DStream.scala ---
@@ -109,6 +109,18 @@ abstract class DStream[T: ClassTag] (
/* Set the creation call site */
private[streaming] val creationSite = DStream.getCreationSite()
+ /**
+ * The scope associated with the operation that created this DStream.
+ *
+ * This is the medium through which we pass the DStream operation name
(e.g. updatedStateByKey)
+ * to the RDDs created by this DStream. Note that we never directly use
this scope in RDDs.
+ * Instead, every time we call `compute` we instantiate a new scope
using the same name as this
+ * one. Otherwise, all RDDs ever created by this DStream will be in the
same scope.
+ */
+ private val scope: Option[RDDOperationScope] = {
--- End diff --
Can you store only the name in as a DStream field? I does not make sense to
save this arbitrary mostly unnecessary RDDOperationScope object to be
serialized with DStreams as part of DStream checkpoints.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
