[
https://issues.apache.org/jira/browse/SPARK-6270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15053223#comment-15053223
]
Shivaram Venkataraman commented on SPARK-6270:
----------------------------------------------
I looked into our log file more closely and in our case the size blow up was
from the storage information saved in SparkListenerJobStart. From what I can
see we added RDD scope, parentIds to be logged either per block or per RDD in
1.5.0 as a part of
https://github.com/apache/spark/commit/fc8b58195afa67fbb75b4c8303e022f703cbf007#diff-4f6ba18259eb4c31ac930e18f1ba6f88
-- I'm guessing this is the cause, but I haven't run the same application with
1.4.x to verify yet.
FWIW I noticed that [~andrewor14] checked in a change to avoid a similar issue
in 1.6 in
https://github.com/apache/spark/commit/688e521c2833a00069272a6749153d721a0996f6
-- It would be cool to have some regression tests which measure the event log
size for common benchmarks across releases.
> Standalone Master hangs when streaming job completes and event logging is
> enabled
> ---------------------------------------------------------------------------------
>
> Key: SPARK-6270
> URL: https://issues.apache.org/jira/browse/SPARK-6270
> Project: Spark
> Issue Type: Bug
> Components: Deploy, Streaming
> Affects Versions: 1.2.0, 1.2.1, 1.3.0, 1.5.1
> Reporter: Tathagata Das
> Priority: Critical
>
> If the event logging is enabled, the Spark Standalone Master tries to
> recreate the web UI of a completed Spark application from its event logs.
> However if this event log is huge (e.g. for a Spark Streaming application),
> then the master hangs in its attempt to read and recreate the web ui. This
> hang causes the whole standalone cluster to be unusable.
> Workaround is to disable the event logging.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
