[
https://issues.apache.org/jira/browse/SPARK-19316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15840348#comment-15840348
]
Jisoo Kim commented on SPARK-19316:
-----------------------------------
Found the duplicate and made a PR to resolve the issue
https://github.com/apache/spark/pull/16714. I wasn't sure if I needed to
include this JIRA ticket to the name so I only mentioned the old one.
> Spark event logs are huge compared to 1.5.2
> -------------------------------------------
>
> Key: SPARK-19316
> URL: https://issues.apache.org/jira/browse/SPARK-19316
> Project: Spark
> Issue Type: Bug
> Affects Versions: 2.0.0
> Reporter: Jisoo Kim
>
> I have a Spark application with many tasks (more than 40k). The event logs
> for such application used to be around 2g when I was using Spark 1.5.2
> standalone cluster. Now that I am using Spark 2.0 with Mesos, the size of the
> event log of such application drastically increased from 2g to 60g with a
> similar number of tasks. This is affecting Spark History Server since it is
> having trouble reading such huge event log. I wonder the increase in a size
> of an event log is expected in Spark 2.0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
