[jira] [Commented] (SPARK-21547) Spark cleaner cost too many time
[ https://issues.apache.org/jira/browse/SPARK-21547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106292#comment-16106292 ] DjvuLee commented on SPARK-21547: - Ok, I will try and posted the result later. > Spark cleaner cost too many time > > > Key: SPARK-21547 > URL: https://issues.apache.org/jira/browse/SPARK-21547 > Project: Spark > Issue Type: Bug > Components: DStreams >Affects Versions: 2.0.0 >Reporter: DjvuLee > > Spark Streaming sometime cost so many time deal with cleaning, and this can > become worse when enable the dynamic allocation. > I post the Driver's Log in the following comments, we can find that the > cleaner costs more than 2min. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21547) Spark cleaner cost too many time
[ https://issues.apache.org/jira/browse/SPARK-21547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106262#comment-16106262 ] Shixiong Zhu commented on SPARK-21547: -- Could you try 2.1.1 or 2.2.0? This may be just SPARK-18991 > Spark cleaner cost too many time > > > Key: SPARK-21547 > URL: https://issues.apache.org/jira/browse/SPARK-21547 > Project: Spark > Issue Type: Bug > Components: DStreams >Affects Versions: 2.0.0 >Reporter: DjvuLee > > Spark Streaming sometime cost so many time deal with cleaning, and this can > become worse when enable the dynamic allocation. > I post the Driver's Log in the following comments, we can find that the > cleaner costs more than 2min. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21547) Spark cleaner cost too many time
[ https://issues.apache.org/jira/browse/SPARK-21547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16104532#comment-16104532 ] Sean Owen commented on SPARK-21547: --- It depends on your app, what's in your closure, etc. I'm not sure what problem this causes you. "Look into X" isn't suitable as a JIRA. I think this would have to be paired with some hint about what the issue is or how it could be addressed. > Spark cleaner cost too many time > > > Key: SPARK-21547 > URL: https://issues.apache.org/jira/browse/SPARK-21547 > Project: Spark > Issue Type: Bug > Components: DStreams >Affects Versions: 2.0.0 >Reporter: DjvuLee > > Spark Streaming sometime cost so many time deal with cleaning, and this can > become worse when enable the dynamic allocation. > I post the Driver's Log in the following comments, we can find that the > cleaner costs more than 2min. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21547) Spark cleaner cost too many time
[ https://issues.apache.org/jira/browse/SPARK-21547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103459#comment-16103459 ] DjvuLee commented on SPARK-21547: - Yes, I agree that this has a relationship with the work, but doing nothing about 3min is too long for a Streaming Application. My proposal is try to let us to inspect whether the current cleaner strategy is good enough. > Spark cleaner cost too many time > > > Key: SPARK-21547 > URL: https://issues.apache.org/jira/browse/SPARK-21547 > Project: Spark > Issue Type: Bug > Components: DStreams >Affects Versions: 2.0.0 >Reporter: DjvuLee > > Spark Streaming sometime cost so many time deal with cleaning, and this can > become worse when enable the dynamic allocation. > I post the Driver's Log in the following comments, we can find that the > cleaner costs more than 2min. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21547) Spark cleaner cost too many time
[ https://issues.apache.org/jira/browse/SPARK-21547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103021#comment-16103021 ] Sean Owen commented on SPARK-21547: --- It's not clear that this is "too slow", relative to whatever work it's doing. Why is it unnecessarily slow, or what change are you proposing? > Spark cleaner cost too many time > > > Key: SPARK-21547 > URL: https://issues.apache.org/jira/browse/SPARK-21547 > Project: Spark > Issue Type: Bug > Components: DStreams >Affects Versions: 2.0.0 >Reporter: DjvuLee > > Spark Streaming sometime cost so many time deal with cleaning, and this can > become worse when enable the dynamic allocation. > I post the Driver's Log in the following comments, we can find that the > cleaner costs more than 2min. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21547) Spark cleaner cost too many time
[ https://issues.apache.org/jira/browse/SPARK-21547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103005#comment-16103005 ] DjvuLee commented on SPARK-21547: - 17/07/27 11:29:51 INFO TaskSetManager: Finished task 169.0 in stage 1504.0 (TID 1504369) in 43975 ms on n6-195-137.byted.org (999/1000) 17/07/27 11:29:55 INFO TaskSetManager: Finished task 882.0 in stage 1504.0 (TID 1504905) in 44153 ms on n6-195-137.byted.org (1000/1000) 17/07/27 11:29:55 INFO YarnScheduler: Removed TaskSet 1504.0, whose tasks have all completed, from pool 17/07/27 11:29:55 INFO DAGScheduler: ResultStage 1504 (call at /spark2/python/lib/py4j-0.10.3-src.zip/py4j/java_gateway.py:2230) finished in 457.863 s 17/07/27 11:29:55 INFO DAGScheduler: Job 1504 finished: call at /spark2/python/lib/py4j-0.10.3-src.zip/py4j/java_gateway.py:2230, took 457.877969 s 17/07/27 11:30:02 INFO JobScheduler: Added jobs for time 150112620 ms 17/07/27 11:30:32 INFO JobScheduler: Added jobs for time 150112623 ms 17/07/27 11:31:02 INFO JobScheduler: Added jobs for time 150112626 ms 17/07/27 11:31:32 INFO JobScheduler: Added jobs for time 150112629 ms 17/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 10906391 17/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 10906392 17/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 10906396 17/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 10906402 17/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 10906404 17/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 12492509 17/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 12492508 17/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 12492507 17/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 12492506 17/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 12492505 17/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 12492504 17/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 12492503 17/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 12492502 ... 7/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 10906397 17/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 10906398 17/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 10906395 17/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 10906399 17/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 10906403 17/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 10906400 17/07/27 11:31:53 INFO ContextCleaner: Cleaned accumulator 10906401 17/07/27 11:31:53 INFO BlockManagerInfo: Removed broadcast_1504_piece0 on 10.6.131.75:23734 in memory (size: 35.9 KB, free: 2.4 GB) 17/07/27 11:31:53 INFO BlockManagerInfo: Removed broadcast_1504_piece0 on n8-157-227.byted.org:13090 in memory (size: 35.9 KB, free: 9.4 GB) 17/07/27 11:31:53 INFO BlockManagerInfo: Removed broadcast_1504_piece0 on n8-157-158.byted.org:21120 in memory (size: 35.9 KB, free: 9.4 GB) 17/07/27 11:31:53 INFO BlockManagerInfo: Removed broadcast_1504_piece0 on n6-195-150.byted.org:13277 in memory (size: 35.9 KB, free: 9.4 GB) 17/07/27 11:31:53 INFO BlockManagerInfo: Removed broadcast_1504_piece0 on n8-156-165.byted.org:35355 in memory (size: 35.9 KB, free: 9.4 GB) 17/07/27 11:31:53 INFO BlockManagerInfo: Removed broadcast_1504_piece0 on n6-132-023.byted.org:52521 in memory (size: 35.9 KB, free: 9.4 GB) 17/07/27 11:31:53 INFO BlockManagerInfo: Removed broadcast_1504_piece0 on n8-136-133.byted.org:25696 in memory (size: 35.9 KB, free: 9.4 GB) 17/07/27 11:31:53 INFO BlockManagerInfo: Removed broadcast_1504_piece0 on n8-150-029.byted.org:34673 in memory (size: 35.9 KB, free: 9.4 GB) 17/07/27 11:31:53 INFO BlockManagerInfo: Removed broadcast_1504_piece0 on n8-148-038.byted.org:22503 in memory (size: 35.9 KB, free: 9.4 GB) 17/07/27 11:31:53 INFO BlockManagerInfo: Removed broadcast_1504_piece0 on n8-150-038.byted.org:28209 in memory (size: 35.9 KB, free: 9.4 GB) ... 17/07/27 11:32:01 INFO BlockManagerInfo: Removed broadcast_1442_piece0 on n8-163-151.byted.org:33703 in memory (size: 35.9 KB, free: 9.4 GB) 17/07/27 11:32:01 INFO BlockManagerInfo: Removed broadcast_1442_piece0 on n8-148-028.byted.org:36086 in memory (size: 35.9 KB, free: 9.4 GB) 17/07/27 11:32:01 INFO BlockManagerInfo: Removed broadcast_1442_piece0 on n8-151-039.byted.org:21081 in memory (size: 35.9 KB, free: 9.4 GB) 17/07/27 11:32:01 INFO BlockManagerInfo: Removed broadcast_1442_piece0 on n8-157-167.byted.org:29370 in memory (size: 35.9 KB, free: 9.4 GB) 17/07/27 11:32:02 INFO JobScheduler: Added jobs for time 150112632 ms 17/07/27 11:32:32 INFO JobScheduler: Added jobs for time 150112635 ms 17/07/27 11:32:45 INFO JobScheduler: Finished job streaming job 150111696 ms.0 from job set of time 150111696 ms 17/07/27 11:32:45 INFO JobScheduler: Total delay: 9405.183 s for time 150111696 ms (execution: 1169.595 s) 17/07/27 11:32:45 INFO JobScheduler: Starting job streaming
