[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25439: [SPARK-28709][DSTREAMS] - Fix StreamingContext leak through Streaming…

2019-08-13 Thread GitBox
dongjoon-hyun commented on a change in pull request #25439: 
[SPARK-28709][DSTREAMS] - Fix StreamingContext leak through Streaming…
URL: https://github.com/apache/spark/pull/25439#discussion_r313625277
 
 

 ##
 File path: 
streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala
 ##
 @@ -52,8 +52,6 @@ class InputStreamsSuite extends TestSuiteBase with 
BeforeAndAfter {
 
   // Set up the streaming context and input streams
   withStreamingContext(new StreamingContext(conf, batchDuration)) { ssc =>
-ssc.addStreamingListener(ssc.progressListener)
-
 
 Review comment:
   If you remove the behavior change here( 
https://github.com/apache/spark/pull/25439/files#r313624949 ), it seems that we 
don't need to change this, right?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25439: [SPARK-28709][DSTREAMS] - Fix StreamingContext leak through Streaming…

2019-08-13 Thread GitBox
dongjoon-hyun commented on a change in pull request #25439: 
[SPARK-28709][DSTREAMS] - Fix StreamingContext leak through Streaming…
URL: https://github.com/apache/spark/pull/25439#discussion_r313625277
 
 

 ##
 File path: 
streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala
 ##
 @@ -52,8 +52,6 @@ class InputStreamsSuite extends TestSuiteBase with 
BeforeAndAfter {
 
   // Set up the streaming context and input streams
   withStreamingContext(new StreamingContext(conf, batchDuration)) { ssc =>
-ssc.addStreamingListener(ssc.progressListener)
-
 
 Review comment:
   If you remove the behavior change here( 
https://github.com/apache/spark/pull/25439/files#r313624949 ), we don't need to 
change this, right?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25439: [SPARK-28709][DSTREAMS] - Fix StreamingContext leak through Streaming…

2019-08-13 Thread GitBox
dongjoon-hyun commented on a change in pull request #25439: 
[SPARK-28709][DSTREAMS] - Fix StreamingContext leak through Streaming…
URL: https://github.com/apache/spark/pull/25439#discussion_r313624949
 
 

 ##
 File path: 
streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala
 ##
 @@ -575,6 +577,8 @@ class StreamingContext private[streaming] (
   try {
 validate()
 
+registerProgressListener()
 
 Review comment:
   I understand you are suggesting that the unregister is important, but do we 
need this? This seems to be a behavior change.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25439: [SPARK-28709][DSTREAMS] - Fix StreamingContext leak through Streaming…

2019-08-13 Thread GitBox
dongjoon-hyun commented on a change in pull request #25439: 
[SPARK-28709][DSTREAMS] - Fix StreamingContext leak through Streaming…
URL: https://github.com/apache/spark/pull/25439#discussion_r313623047
 
 

 ##
 File path: core/src/main/scala/org/apache/spark/ui/SparkUI.scala
 ##
 @@ -138,6 +138,10 @@ private[spark] class SparkUI private (
 streamingJobProgressListener = Option(sparkListener)
   }
 
+  def clearStreamingJobProgressListener(): Unit = {
+streamingJobProgressListener = None
+  }
+
 
 Review comment:
   nit. redundant empty line.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25439: [SPARK-28709][DSTREAMS] - Fix StreamingContext leak through Streaming…

2019-08-13 Thread GitBox
dongjoon-hyun commented on a change in pull request #25439: 
[SPARK-28709][DSTREAMS] - Fix StreamingContext leak through Streaming…
URL: https://github.com/apache/spark/pull/25439#discussion_r313622771
 
 

 ##
 File path: 
streaming/src/test/scala/org/apache/spark/streaming/StreamingContextSuite.scala
 ##
 @@ -373,7 +374,7 @@ class StreamingContextSuite extends SparkFunSuite with 
BeforeAndAfter with TimeL
 Thread.sleep(100)
   }
 
-  test ("registering and de-registering of streamingSource") {
+  test("registering and de-registering of streamingSource") {
 
 Review comment:
   Usually, we don't touch the other test cases because it makes backporting 
this PR difficult.
   Please revert this and focus on your test case.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25439: [SPARK-28709][DSTREAMS] - Fix StreamingContext leak through Streaming…

2019-08-13 Thread GitBox
dongjoon-hyun commented on a change in pull request #25439: 
[SPARK-28709][DSTREAMS] - Fix StreamingContext leak through Streaming…
URL: https://github.com/apache/spark/pull/25439#discussion_r313622430
 
 

 ##
 File path: 
streaming/src/test/scala/org/apache/spark/streaming/StreamingContextSuite.scala
 ##
 @@ -392,6 +393,29 @@ class StreamingContextSuite extends SparkFunSuite with 
BeforeAndAfter with TimeL
 assert(!sourcesAfterStop.contains(streamingSourceAfterStop))
   }
 
+  test("registering and de-registering of progressListener") {
 
 Review comment:
   For a bug fix, we use JIRA ID prefix. Could you update like this?
   ```scala
   - test("registering and de-registering of progressListener") {
   + test("SPARK-28709 registering and de-registering of progressListener") {
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org