Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21057#discussion_r181632575 --- Diff: python/pyspark/streaming/listener.py --- @@ -22,6 +22,10 @@ class StreamingListener(object): def __init__(self): pass + + def onStreamingStarted(self, streamingStarted): --- End diff -- I first thought it's doc only change but I realised it's actually not after taking a close look. Implementing `onStreamingStarted` looks actually required: > Add missing method to StreamingListener, which is invoked by proxy from the Java/Scala side, and throws a strange exception when not found. This wasn't there at the first - https://github.com/apache/spark/blob/ace0db47141ffd457c2091751038fc291f6d5a8b/python/pyspark/streaming/listener.py / https://github.com/apache/spark/blob/ace0db47141ffd457c2091751038fc291f6d5a8b/streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingListener.scala ; however, this method was added from https://github.com/apache/spark/commit/ce99f51d2e8cbcb1565c9e7a729183bd74a0c2bc but seems Python's implementation is missed. Strictly it sounds better to have an explicit test since @aviv-ebates has a reproducer (assuming from the description) and should be easy to add.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org