[
https://issues.apache.org/jira/browse/HADOOP-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16294241#comment-16294241
]
Igor Dvorzhak commented on HADOOP-15124:
----------------------------------------
I have created PR (https://github.com/apache/hadoop/pull/316) with Draft
changes that I have tested and posted results in the description.
Please, take a look if it looks reasonable and I will clean up it (especially
tests) and made necessary changes so it could be submitted upstream.
> Slow FileSystem.Statistics counters
> ------------------------------------
>
> Key: HADOOP-15124
> URL: https://issues.apache.org/jira/browse/HADOOP-15124
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
> Affects Versions: 2.9.0, 2.8.3, 2.7.5, 3.0.0
> Reporter: Igor Dvorzhak
>
> While profiling 1TB TeraGen on Hadoop 2.8.2 cluster (Google Dataproc, 2
> workers, GCS connector) I saw that FileSystem.Statistics code paths Wall time
> is 5.58% and CPU time is 26.5% of total execution time.
> After switching FileSystem.Statistics implementation to LongAdder consumed
> Wall time decreased to 0.006% and CPU time to 0.104% of total execution time.
> Total runtime decreased from 66 mins to 61 mins.
> These results are not conclusive, because I didn't benchmark multiple times
> to average results, but regardless of performance gains switching to
> LongAdder simplifies code and reduces its complexity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
