[
https://issues.apache.org/jira/browse/IGNITE-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16916855#comment-16916855
]
Alexei Scherbakov commented on IGNITE-3195:
-------------------------------------------
[~avinogradov]
Overall fix looks good, but I think we could improve it.
1. Looks like it's safe to remove ordering for historical rebalance because
after IGNITE-10078 rmvQeue for partition is no longer cleared during rebalance
and removals cannot be lost.
Given what, we could use single thread pool for historical and full rebalance
and parallelize historical rebalance on supplier side same as full.
This is right thing to do because from user side of view there is no difference
between full and historical rebalance and they can happen simultaneously.
Note, proper fix for writing tombstones is on the way [1]
2. Looks like current implementation for detecting partition completion on
concurrent processing using *queued *and *processed *is flawed.
Consider the scenario:
Demander sends initial demand request for single partition.
Supplier replies with 2 total supply messages which are starting to process in
parallel.
2-d message is last.
2-d message started to process first, increments *queued *to N (number of
entries in message)
2-d message finished processing, incrementing *processed *to N.
Because this is last message partition will be owned before other messages are
applied.
[1] https://issues.apache.org/jira/browse/IGNITE-11704
> Rebalancing: IgniteConfiguration.rebalanceThreadPoolSize is wrongly treated
> ---------------------------------------------------------------------------
>
> Key: IGNITE-3195
> URL: https://issues.apache.org/jira/browse/IGNITE-3195
> Project: Ignite
> Issue Type: Bug
> Components: cache
> Reporter: Denis Magda
> Assignee: Anton Vinogradov
> Priority: Major
> Labels: iep-16
> Fix For: 2.8
>
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> Presently it's considered that the maximum number of threads that has to
> process all demand and supply messages coming from all the nodes must not be
> bigger than {{IgniteConfiguration.rebalanceThreadPoolSize}}.
> Current implementation relies on ordered messages functionality creating a
> number of topics equal to {{IgniteConfiguration.rebalanceThreadPoolSize}}.
> However, the implementation doesn't take into account that ordered messages,
> that correspond to a particular topic, are processed in parallel for
> different nodes. Refer to the implementation of
> {{GridIoManager.processOrderedMessage}} to see that for every topic there
> will be a unique {{GridCommunicationMessageSet}} for every node.
> Also to prove that this is true you can refer to this execution stack
> {noformat}
> java.lang.RuntimeException: HAPPENED DEMAND
> at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$5.apply(GridCachePartitionExchangeManager.java:378)
> at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$5.apply(GridCachePartitionExchangeManager.java:364)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:622)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:320)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$300(GridCacheIoManager.java:81)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager$OrderedMessageListener.onMessage(GridCacheIoManager.java:1125)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1219)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.access$1600(GridIoManager.java:105)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager$GridCommunicationMessageSet.unwind(GridIoManager.java:2456)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.unwindMessageSet(GridIoManager.java:1179)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.access$1900(GridIoManager.java:105)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager$6.run(GridIoManager.java:1148)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> All this means that in fact the number of threads that will be busy with
> replication activity will be equal to
> {{IgniteConfiguration.rebalanceThreadPoolSize}} x
> number_of_nodes_participated_in_rebalancing
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
