[jira] [Updated] (HADOOP-13747) Use LongAdder for more efficient metrics tracking
[
https://issues.apache.org/jira/browse/HADOOP-13747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated HADOOP-13747:
Target Version/s: 3.5.0 (was: 3.4.0)
> Use LongAdder for more efficient metrics tracking
> -
>
> Key: HADOOP-13747
> URL: https://issues.apache.org/jira/browse/HADOOP-13747
> Project: Hadoop Common
> Issue Type: Improvement
> Components: metrics
>Reporter: Zhe Zhang
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HADOOP-13747.patch, benchmark_results
>
>
> Currently many metrics, including {{RpcMetrics}} and {{RpcDetailedMetrics}},
> use a synchronized counter to be updated by all handler threads (multiple
> hundreds in large production clusters). As [~andrew.wang] suggested, it'd be
> more efficient to use the [LongAdder |
> http://gee.cs.oswego.edu/cgi-bin/viewcvs.cgi/jsr166/src/jsr166e/LongAdder.java?view=co]
> library which dynamically create intermediate-result variables.
> Assigning to [~xkrogen] who has already done some investigation on this.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13747) Use LongAdder for more efficient metrics tracking
[
https://issues.apache.org/jira/browse/HADOOP-13747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HADOOP-13747:
--
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> Use LongAdder for more efficient metrics tracking
> -
>
> Key: HADOOP-13747
> URL: https://issues.apache.org/jira/browse/HADOOP-13747
> Project: Hadoop Common
> Issue Type: Improvement
> Components: metrics
>Reporter: Zhe Zhang
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HADOOP-13747.patch, benchmark_results
>
>
> Currently many metrics, including {{RpcMetrics}} and {{RpcDetailedMetrics}},
> use a synchronized counter to be updated by all handler threads (multiple
> hundreds in large production clusters). As [~andrew.wang] suggested, it'd be
> more efficient to use the [LongAdder |
> http://gee.cs.oswego.edu/cgi-bin/viewcvs.cgi/jsr166/src/jsr166e/LongAdder.java?view=co]
> library which dynamically create intermediate-result variables.
> Assigning to [~xkrogen] who has already done some investigation on this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13747) Use LongAdder for more efficient metrics tracking
[
https://issues.apache.org/jira/browse/HADOOP-13747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated HADOOP-13747:
Target Version/s: 3.0.0-alpha2 (was: 2.8.0, 3.0.0-alpha2)
> Use LongAdder for more efficient metrics tracking
> -
>
> Key: HADOOP-13747
> URL: https://issues.apache.org/jira/browse/HADOOP-13747
> Project: Hadoop Common
> Issue Type: Improvement
> Components: metrics
>Reporter: Zhe Zhang
>Assignee: Erik Krogen
> Attachments: HADOOP-13747.patch, benchmark_results
>
>
> Currently many metrics, including {{RpcMetrics}} and {{RpcDetailedMetrics}},
> use a synchronized counter to be updated by all handler threads (multiple
> hundreds in large production clusters). As [~andrew.wang] suggested, it'd be
> more efficient to use the [LongAdder |
> http://gee.cs.oswego.edu/cgi-bin/viewcvs.cgi/jsr166/src/jsr166e/LongAdder.java?view=co]
> library which dynamically create intermediate-result variables.
> Assigning to [~xkrogen] who has already done some investigation on this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13747) Use LongAdder for more efficient metrics tracking
[
https://issues.apache.org/jira/browse/HADOOP-13747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Krogen updated HADOOP-13747:
-
Attachment: HADOOP-13747.patch
benchmark_results
As discussed in the comments of HDFS-10872, doing aggregation-on-read would be
ideal. I'm attaching some relevant benchmark numbers supporting this. The
{{MetricBenchmark.increment}} numbers show the overhead of contesting on a
single metric (as discussed further in HDFS-10872, though this also includes an
implementation which uses an {{AtomicLong}} for updating). The
{{MetricGroupBenchmark.increment}} numbers show the overhead of having 50
different metrics with 100 threads trying to update them all as fast as
possible (each individual metric is stored as an {{AtomicReferenceAdder}} from
the upper benchmarks). {{Synchronized}} is essentially the current
implementation with {{synchronized}} methods; {{RegularHashMap}} stores the
metrics in a {{HashMap}} which does not provide any synchronization but serves
as a baseline; {{ConcurrentHashMapComputeIfAbsent}} which uses the
{{computeIfAbsent}} method to insert a new metric if necessary;
{{HashMapRWLock}} which wraps a {{HashMap}} in a {{ReadWriteLock}}; and
{{LocalUpdateAggregateWeakRef}} in which each thread stores a local copy of the
metrics and, on snapshot, aggregates them all. This is achieved by keeping one
{{ConcurrentLinkedDequeue}} of {{WeakReference}} to a map of metrics stored in
a {{ThreadLocal}}. Each thread has its local map which it inserts into the
{{ConcurrentLinkedDequeue}}, and upon snapshot the snapshotting thread
traverses this queue to read metrics from each thread's map. If the
{{WeakReference}} is unresolvable (i.e., the thread has died), the snapshotting
thread will remove the reference from the queue.
I'm attaching a patch implementing the {{LocalUpdateAggregateWeakRef}} but not
marking patch available. I just want to put up the ideas in code for
discussion; it needs a little cleaning up with naming and such before it's
ready to go.
> Use LongAdder for more efficient metrics tracking
> -
>
> Key: HADOOP-13747
> URL: https://issues.apache.org/jira/browse/HADOOP-13747
> Project: Hadoop Common
> Issue Type: Improvement
> Components: metrics
>Reporter: Zhe Zhang
>Assignee: Erik Krogen
> Attachments: HADOOP-13747.patch, benchmark_results
>
>
> Currently many metrics, including {{RpcMetrics}} and {{RpcDetailedMetrics}},
> use a synchronized counter to be updated by all handler threads (multiple
> hundreds in large production clusters). As [~andrew.wang] suggested, it'd be
> more efficient to use the [LongAdder |
> http://gee.cs.oswego.edu/cgi-bin/viewcvs.cgi/jsr166/src/jsr166e/LongAdder.java?view=co]
> library which dynamically create intermediate-result variables.
> Assigning to [~xkrogen] who has already done some investigation on this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13747) Use LongAdder for more efficient metrics tracking
[
https://issues.apache.org/jira/browse/HADOOP-13747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HADOOP-13747:
-
Target Version/s: 2.8.0, 3.0.0-alpha2
> Use LongAdder for more efficient metrics tracking
> -
>
> Key: HADOOP-13747
> URL: https://issues.apache.org/jira/browse/HADOOP-13747
> Project: Hadoop Common
> Issue Type: Improvement
> Components: metrics
>Reporter: Zhe Zhang
>Assignee: Erik Krogen
>
> Currently many metrics, including {{RpcMetrics}} and {{RpcDetailedMetrics}},
> use a synchronized counter to be updated by all handler threads (multiple
> hundreds in large production clusters). As [~andrew.wang] suggested, it'd be
> more efficient to use the [LongAdder |
> http://gee.cs.oswego.edu/cgi-bin/viewcvs.cgi/jsr166/src/jsr166e/LongAdder.java?view=co]
> library which dynamically create intermediate-result variables.
> Assigning to [~xkrogen] who has already done some investigation on this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13747) Use LongAdder for more efficient metrics tracking
[
https://issues.apache.org/jira/browse/HADOOP-13747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HADOOP-13747:
---
Description:
Currently many metrics, including {{RpcMetrics}} and {{RpcDetailedMetrics}},
use a synchronized counter to be updated by all handler threads (multiple
hundreds in large production clusters). As [~andrew.wang] suggested, it'd be
more efficient to use the [LongAdder |
http://gee.cs.oswego.edu/cgi-bin/viewcvs.cgi/jsr166/src/jsr166e/LongAdder.java?view=co]
library which dynamically create intermediate-result variables.
Assigning to [~xkrogen] who has already done some investigation on this.
was:
Currently most metrics, including {{RpcMetrics}} and {{RpcDetailedMetrics}},
use a synchronized counter to be updated by all handler threads (multiple
hundreds in large production clusters). As [~andrew.wang] suggested, it'd be
more efficient to use the [LongAdder |
http://gee.cs.oswego.edu/cgi-bin/viewcvs.cgi/jsr166/src/jsr166e/LongAdder.java?view=co]
library which dynamically create intermediate-result variables.
Assigning to [~xkrogen] who has already done some investigation on this.
> Use LongAdder for more efficient metrics tracking
> -
>
> Key: HADOOP-13747
> URL: https://issues.apache.org/jira/browse/HADOOP-13747
> Project: Hadoop Common
> Issue Type: Improvement
> Components: metrics
>Reporter: Zhe Zhang
>Assignee: Erik Krogen
>
> Currently many metrics, including {{RpcMetrics}} and {{RpcDetailedMetrics}},
> use a synchronized counter to be updated by all handler threads (multiple
> hundreds in large production clusters). As [~andrew.wang] suggested, it'd be
> more efficient to use the [LongAdder |
> http://gee.cs.oswego.edu/cgi-bin/viewcvs.cgi/jsr166/src/jsr166e/LongAdder.java?view=co]
> library which dynamically create intermediate-result variables.
> Assigning to [~xkrogen] who has already done some investigation on this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
