Github user dragos commented on a diff in the pull request:
https://github.com/apache/spark/pull/7600#discussion_r35629147
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/scheduler/ReceiverTrackerSuite.scala
---
@@ -110,8 +110,15 @@ class ReceiverTrackerSuite extends TestSuiteBase {
}
}
-/** An input DStream with a hard-coded receiver that gives access to
internals for testing. */
-private class RateLimitInputDStream(@transient ssc_ : StreamingContext)
+/**
+ * An input DStream with a hard-coded receiver that gives access to
internals for testing.
+ *
+ * @note Make sure to call {{{SingletonDummyReceiver.reset()}}} before
using this in a test,
+ * or otherwise you may get {{{NotSerializableException}}} when
trying to serialize
+ * the receiver.
--- End diff --
Because after being `started` once, it has a reference to the
ReceiverSupervisor, which is not serializable. So the second time the receiver
is used, it can't be serialized and shipped to executors. I thought the
one-line method is self-explanatory, but I can add a few more lines of comments
around that one line :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
