[
https://issues.apache.org/jira/browse/IGNITE-10920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16757648#comment-16757648
]
Alexei Scherbakov edited comment on IGNITE-10920 at 1/31/19 7:20 PM:
---------------------------------------------------------------------
[~kbolyandra],
I've played a bit with JOL benchmark and got following results for a single
history assignment with 32 and 128 nodes in topology, results are impressive:
{noformat}
Heap usage [optimized=false, parts=32768, nodeCnt=32, backups=2, footprint:
COUNT AVG SUM DESCRIPTION
33714 39 1340960 [Ljava.lang.Object;
33714 24 809136 java.util.ArrayList
2 24 48 java.util.Collections$UnmodifiableRandomAccessList
1 24 24 org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion
1 40 40 org.apache.ignite.internal.processors.affinity.HistoryAffinityAssignment
67432 2150208 (total)
]
Heap usage [optimized=true, parts=32768, nodeCnt=32, backups=2, footprint:
COUNT AVG SUM DESCRIPTION
945 232 219280 [C
1 8208 8208 [Ljava.util.HashMap$Node;
1 144 144 [Lorg.apache.ignite.cluster.ClusterNode;
944 16 15104 java.lang.Integer
1 48 48 java.util.HashMap
944 32 30208 java.util.HashMap$Node
1 24 24 org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion
1 40 40 org.apache.ignite.internal.processors.affinity.HistoryAffinityAssignment
1 32 32
org.apache.ignite.internal.processors.affinity.HistoryAffinityAssignment$1
1 32 32
org.apache.ignite.internal.processors.affinity.HistoryAffinityAssignment$2
2840 273120 (total)
]
Optimization: optimized=273120, deoptimized=2150208 rate: 7.872{noformat}
{noformat}
Heap usage [optimized=false, parts=32768, nodeCnt=128, backups=2, footprint:
COUNT AVG SUM DESCRIPTION
33066 39 1320224 [Ljava.lang.Object;
33066 24 793584 java.util.ArrayList
2 24 48 java.util.Collections$UnmodifiableRandomAccessList
1 24 24 org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion
1 40 40 org.apache.ignite.internal.processors.affinity.HistoryAffinityAssignment
66136 2113920 (total)
]
Heap usage [optimized=true, parts=32768, nodeCnt=128, backups=2, footprint:
COUNT AVG SUM DESCRIPTION
297 685 203728 [C
1 2064 2064 [Ljava.util.HashMap$Node;
1 528 528 [Lorg.apache.ignite.cluster.ClusterNode;
296 16 4736 java.lang.Integer
1 48 48 java.util.HashMap
296 32 9472 java.util.HashMap$Node
1 24 24 org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion
1 40 40 org.apache.ignite.internal.processors.affinity.HistoryAffinityAssignment
1 32 32
org.apache.ignite.internal.processors.affinity.HistoryAffinityAssignment$1
1 32 32
org.apache.ignite.internal.processors.affinity.HistoryAffinityAssignment$2
896 220704 (total)
Optimization: optimized=220704, deoptimized=2113920 rate: 9.578{noformat}
No objections from my side.
I think somebody of the commiters will pass by shortly and finish your
contribution.
was (Author: ascherbakov):
[~kbolyandra],
I've played a bit with JOL benchmark and got following results for a single
history assignment with 32 and 128 nodes in topology, results are impressive:
{{Heap usage [optimized=false, parts=32768, nodeCnt=32, backups=2, footprint:
COUNT AVG SUM DESCRIPTION
33714 39 1340960 [Ljava.lang.Object;
33714 24 809136 java.util.ArrayList
2 24 48 java.util.Collections$UnmodifiableRandomAccessList
1 24 24 org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion
1 40 40
org.apache.ignite.internal.processors.affinity.HistoryAffinityAssignment
67432 2150208 (total)
]
Heap usage [optimized=true, parts=32768, nodeCnt=32, backups=2, footprint:
COUNT AVG SUM DESCRIPTION
945 232 219280 [C
1 8208 8208 [Ljava.util.HashMap$Node;
1 144 144 [Lorg.apache.ignite.cluster.ClusterNode;
944 16 15104 java.lang.Integer
1 48 48 java.util.HashMap
944 32 30208 java.util.HashMap$Node
1 24 24 org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion
1 40 40
org.apache.ignite.internal.processors.affinity.HistoryAffinityAssignment
1 32 32
org.apache.ignite.internal.processors.affinity.HistoryAffinityAssignment$1
1 32 32
org.apache.ignite.internal.processors.affinity.HistoryAffinityAssignment$2
2840 273120 (total)
]
Optimization: optimized=273120, deoptimized=2150208 rate: 7.872}}
{{Heap usage [optimized=false, parts=32768, nodeCnt=128, backups=2, footprint:
COUNT AVG SUM DESCRIPTION
33066 39 1320224 [Ljava.lang.Object;
33066 24 793584 java.util.ArrayList
2 24 48 java.util.Collections$UnmodifiableRandomAccessList
1 24 24 org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion
1 40 40
org.apache.ignite.internal.processors.affinity.HistoryAffinityAssignment
66136 2113920 (total)
]
Heap usage [optimized=true, parts=32768, nodeCnt=128, backups=2, footprint:
COUNT AVG SUM DESCRIPTION
297 685 203728 [C
1 2064 2064 [Ljava.util.HashMap$Node;
1 528 528 [Lorg.apache.ignite.cluster.ClusterNode;
296 16 4736 java.lang.Integer
1 48 48 java.util.HashMap
296 32 9472 java.util.HashMap$Node
1 24 24 org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion
1 40 40
org.apache.ignite.internal.processors.affinity.HistoryAffinityAssignment
1 32 32
org.apache.ignite.internal.processors.affinity.HistoryAffinityAssignment$1
1 32 32
org.apache.ignite.internal.processors.affinity.HistoryAffinityAssignment$2
896 220704 (total)
Optimization: optimized=220704, deoptimized=2113920 rate: 9.578}}
No objections from my side.
I think somebody of the commiters will pass by shortly and finish your
contribution.
> Optimize HistoryAffinityAssignment heap usage.
> ----------------------------------------------
>
> Key: IGNITE-10920
> URL: https://issues.apache.org/jira/browse/IGNITE-10920
> Project: Ignite
> Issue Type: Improvement
> Reporter: Alexei Scherbakov
> Assignee: Konstantin Bolyandra
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> With large topology and large amount of caches/partitions many server
> discovery events may quickly produce large affinity history, eating gigabytes
> of heap.
> Solution: implement some kind of a compression for affinity cache map.
> On example, affinity history could be stored as delta to some previous
> version.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
