[
https://issues.apache.org/jira/browse/IGNITE-6923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16304502#comment-16304502
]
Aleksey Plekhanov edited comment on IGNITE-6923 at 12/27/17 12:36 PM:
----------------------------------------------------------------------
There were 4 fixes made:
# Each cluster metrics calculation invoked the
{{DiscoveryMetricsProvider#cacheMetrics}} method, which creates metrics
snapshots for all caches and is heavy, when a large number of caches are
started. But only one cache metric is needed for cluster metrics calculation
({{getOffHeapAllocatedSize}}), so invocation of {{cacheMetrics}} is replaced
with a direct call to {{getOffHeapAllocatedSize}}.
# Each invocation of {{DiscoveryMetricsProvider#metrics}} creates cluster
metrics snapshot. It’s ok for discovery update message (because all metrics are
serialized and calculation of all metrics is needed), but sometimes this is
redundant. For example, call to {{locNode.metrics().getTotalCpus()}} calculates
each cluster metric, but uses only a total CPU count, which can be calculated
at a lower cost. To solve this problem {{DiscoveryMetricsProvider#metrics}}
method does not calculate snapshot of all metrics now, but returns a
{{ClusterMetricsImpl}} class instance, which calculates each metric on demand.
# Some cache metrics (entities count, partitions count) iterates through local
partitions to get the result. There are 9 such methods which uses this
iteration per each call. But it is possible to calculate this 9 metrics using
one iteration over partitions. The new method was implemented to calculate all
this metrics at once, in {{CacheMetricsSnapshot}} individual calculations of
this metrics were replaced by the new method.
# If there are a lot of nodes and a lot of caches in the cluster, size of
{{TcpDiscoveryMetricsUpdateMessage}} can be rather large (up to hundreds of
megabytes), because it contains information about each metric for each cache on
each node. Sending cache metrics in the discovery message can be turned off by
disabling statistics for caches, but it also makes unavailable cache metrics
via JMX. New system property was added to disable cache metrics update in the
discovery message without disabling statistics.
Some benchmarks:
Environment: 2 nodes, 200 caches (with statistics enabled), 1024 partitions per
cache, 10000 job metrics snapshots.
* First optimization (direct call to {{getOffHeapAllocatedSize}})
Subject: {{DiscoveryMetricsProvider.metrics()}} method
Before optimization: 17 operations per second
After optimization: 8000 operations per second
* Second optimization ({{ClusterMetricsImpl}} instead of
{{ClusterMetricsSnapshot}})
Subject: {{DiscoveryMetricsProvider.metrics().getTotalCpus()}}
Before optimization: 8000 operations per second
After optimization: 2000000 operations per second
But individual call to {{getTotalCpus()}} is relatively rare, in most cases
{{DiscoveryMetricsProvider.metrics()}} used for sending
{{TcpDiscoveryMetricsUpdateMessage}} and performance of
{{ClusterMetricsSnapshot.serialize(DiscoveryMetricsProvider.metrics())}} left
the same (8000 operations per second). Perhaps after the first optimization
this (second) optimization is no longer needed?
* Third optimization (one iteration over partitions)
Subject: {{DiscoveryMetricsProvider.cacheMetrics()}}
Before optimization: 17 operations per second
After optimization: 75 operations per second
was (Author: alex_pl):
There were 4 fixes made:
# Each cluster metrics calculation invoked the
{DiscoveryMetricsProvider#cacheMetrics} method, which creates metrics snapshots
for all caches and is heavy, when a large number of caches are started. But
only one cache metric is needed for cluster metrics calculation
({getOffHeapAllocatedSize}), so invocation of {cacheMetrics} is replaced with a
direct call to {getOffHeapAllocatedSize}.
# Each invocation of {DiscoveryMetricsProvider#metrics} creates cluster metrics
snapshot. It’s ok for discovery update message (because all metrics are
serialized and calculation of all metrics is needed), but sometimes this is
redundant. For example, call to {locNode.metrics().getTotalCpus()} calculates
each cluster metric, but uses only a total CPU count, which can be calculated
at a lower cost. To solve this problem {DiscoveryMetricsProvider#metrics}
method does not calculate snapshot of all metrics now, but returns a
{ClusterMetricsImpl} class instance, which calculates each metric on demand.
# Some cache metrics (entities count, partitions count) iterates through local
partitions to get the result. There are 9 such methods which uses this
iteration per each call. But it is possible to calculate this 9 metrics using
one iteration over partitions. The new method was implemented to calculate all
this metrics at once, in {CacheMetricsSnapshot} individual calculations of this
metrics were replaced by the new method.
# If there are a lot of nodes and a lot of caches in the cluster, size of
{TcpDiscoveryMetricsUpdateMessage} can be rather large (up to hundreds of
megabytes), because it contains information about each metric for each cache on
each node. Sending cache metrics in the discovery message can be turned off by
disabling statistics for caches, but it also makes unavailable cache metrics
via JMX. New system property was added to disable cache metrics update in the
discovery message without disabling statistics.
Some benchmarks:
Environment: 2 nodes, 200 caches (with statistics enabled), 1024 partitions per
cache, 10000 job metrics snapshots.
* First optimization (direct call to {getOffHeapAllocatedSize})
Subject: {DiscoveryMetricsProvider.metrics()} method
Before optimization: 17 operations per second
After optimization: 8000 operations per second
* Second optimization ({ClusterMetricsImpl} instead of {ClusterMetricsSnapshot})
Subject: {DiscoveryMetricsProvider.metrics().getTotalCpus()}
Before optimization: 8000 operations per second
After optimization: 2000000 operations per second
But individual call to {getTotalCpus()} is relatively rare, in most cases
{DiscoveryMetricsProvider.metrics()} used for sending
{TcpDiscoveryMetricsUpdateMessage} and performance of
{ClusterMetricsSnapshot.serialize(DiscoveryMetricsProvider.metrics())} left the
same (8000 operations per second). Perhaps after the first optimization this
(second) optimization is no longer needed?
* Third optimization (one iteration over partitions)
Subject: {DiscoveryMetricsProvider.cacheMetrics()}
Before optimization: 17 operations per second
After optimization: 75 operations per second
> Cache metrics are updated in timeout-worker potentially delaying critical
> code execution due to current implementation issues.
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: IGNITE-6923
> URL: https://issues.apache.org/jira/browse/IGNITE-6923
> Project: Ignite
> Issue Type: Improvement
> Affects Versions: 2.3
> Reporter: Alexei Scherbakov
> Assignee: Aleksey Plekhanov
> Labels: iep-6
> Fix For: 2.4
>
>
> Some metrics are using cache iteration for calculation. If number of caches
> rather large this can be slow.
> Similar code is running in discovery thread.
> See stack trace for example.
> {noformat}
> "grid-timeout-worker-#39%DPL_GRID%DplGridNodeName%" #152 prio=5 os_prio=0
> tid=0x00007f1009a03000 nid=0x5caa runnable [0x00007f0f059d9000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.containsKey(HashMap.java:595)
> at java.util.HashSet.contains(HashSet.java:203)
> at
> java.util.Collections$UnmodifiableCollection.contains(Collections.java:1032)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$3.apply(IgniteCacheOffheapManagerImpl.java:339)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$3.apply(IgniteCacheOffheapManagerImpl.java:337)
> at
> org.apache.ignite.internal.util.lang.gridfunc.TransformFilteringIterator.hasNext:@TransformFilteringIterator.java:90)
> at
> org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
>
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.cacheEntriesCount(IgniteCacheOffheapManagerImpl.java:293)
> at
> org.apache.ignite.internal.processors.cache.CacheMetricsImpl.getOffHeapPrimaryEntriesCount(CacheMetricsImpl.java:240)
> at
> org.apache.ignite.internal.processors.cache.CacheMetricsSnapshot.<init>(CacheMetricsSnapshot.java:271)
>
> at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter.localMetrics(GridCacheAdapter.java:3217)
>
> at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$7.cacheMetrics(GridDiscoveryManager.java:1151)
> at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$7.nonHeapMemoryUsed(GridDiscoveryManager.java:1121)
> at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$7.metrics(GridDiscoveryManager.java:1087)
>
> at
> org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.metrics(TcpDiscoveryNode.java:269)
>
> at
> org.apache.ignite.internal.IgniteKernal$3.run(IgniteKernal.java:1175)
> at
> org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$CancelableTask.onTimeout(GridTimeoutProcessor.java:256)
> - locked <0x00007f115f5bf890> (a
> org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$CancelableTask)
> at
> org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:158)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
