[
https://issues.apache.org/jira/browse/SPARK-17231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Allman updated SPARK-17231:
-----------------------------------
Description: While debugging the performance of a large GraphX connected
components computation, I found several places in the `network-common` and
`network-shuffle` code bases where trace or debug log messages are constructed
even if the respective log level is disabled. Refactoring the respective code
to avoid these constructions except where necessary led to a modest but
measurable reduction in task time, GC time and the ratio thereof. (was: While
debugging the performance of a large GraphX connected components computation, I
found several places in the `network-common` and `network-shuffle` code bases
where trace or debug log messages are constructed even if the respective log
level is disabled. Refactoring the respective code to avoid these constructions
except where necessary led to a modest but measurable reduction in task time,
GC time and the ratio thereof.
(PR to follow.))
> Avoid building debug or trace log messages unless the respective log level is
> enabled
> -------------------------------------------------------------------------------------
>
> Key: SPARK-17231
> URL: https://issues.apache.org/jira/browse/SPARK-17231
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.0.0
> Environment: Spark cluster with 8 r3.8xl EC2 worker instances
> Reporter: Michael Allman
> Priority: Minor
> Attachments: logging_perf_improvements.jpg, master.jpg
>
>
> While debugging the performance of a large GraphX connected components
> computation, I found several places in the `network-common` and
> `network-shuffle` code bases where trace or debug log messages are
> constructed even if the respective log level is disabled. Refactoring the
> respective code to avoid these constructions except where necessary led to a
> modest but measurable reduction in task time, GC time and the ratio thereof.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
