[
https://issues.apache.org/jira/browse/SPARK-17231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15435767#comment-15435767
]
Michael Allman commented on SPARK-17231:
----------------------------------------
Note that in the attached screenshots, all stats are the same except task and
gc time.
> Avoid building debug or trace log messages unless the respective log level is
> enabled
> -------------------------------------------------------------------------------------
>
> Key: SPARK-17231
> URL: https://issues.apache.org/jira/browse/SPARK-17231
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.0.0
> Environment: Spark cluster with 8 r3.8xl EC2 worker instances
> Reporter: Michael Allman
> Priority: Minor
> Attachments: logging_perf_improvements.jpg, master.jpg
>
>
> While debugging the performance of a large GraphX connected components
> computation, I found several places in the `network-common` and
> `network-shuffle` code bases where trace or debug log messages are
> constructed even if the respective log level is disabled. Refactoring the
> respective code to avoid these constructions except where necessary led to a
> modest but measurable reduction in task time, GC time and the ratio thereof.
> (Before and after executor stats to follow in screenshots.)
> (PR to follow.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
