[
https://issues.apache.org/jira/browse/IGNITE-19104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17705856#comment-17705856
]
Sergey Uttsel edited comment on IGNITE-19104 at 3/28/23 1:09 PM:
-----------------------------------------------------------------
Actually we have a race between async 'logicalTopology' initialization in
initDataNodesFromVaultManager() and 'logicalTopology' updating in
DistributionZoneManager#watchListener. So we need to initialize
'logicalTopology' sync in DistributionZoneManager#start().
Another issue caused by uninitialized 'logicalTopology' if zoneManager is
started and recieved event about new zone creation before 'logicalTopology'
initialization. In this case possible that new zone will be created with wrong
data nodes (see ZonesConfigurationListener#onCreate).
Some additional remarks.
Another issues is a race in updating metastorage data nodes between
DistributionZoneManager#start() and DistributionZoneManager#watchListener. It
is possible that DistributionZoneManager#watchListener updates
'logicalTopology' and schedules timers with some revision. Then async logic in
start() read this revision and updates metastorage data nodes emmediately
before timers are expired. This race will be fixed in
https://ggsystems.atlassian.net/browse/IGN-21354.
was (Author: sergey uttsel):
Actually we have a race between async 'logicalTopology' initialization in
initDataNodesFromVaultManager() and 'logicalTopology' updating in
DistributionZoneManager#watchListener. So we need to initialize
'logicalTopology' sync in DistributionZoneManager#start().
Another issue caused by uninitialized 'logicalTopology' if zoneManager is
started and recieved event about new zone creation before 'logicalTopology'
initialization. In this case possible that new zone will be created with wrong
data nodes (see ZonesConfigurationListener#onCreate).
Some additional remarks.
Another issues is a race in updating metastorage data nodes between
DistributionZoneManager#start() and DistributionZoneManager#watchListener. It
is possible that DistributionZoneManager#watchListener updates
'logicalTopology' and schedules timers with some revision. Then async logic in
start() read this revision and updates metastorage data nodes emmediately
before timers are expired. This race will be fixed in
https://ggsystems.atlassian.net/browse/IGN-21354.
> Late logicalTopology initialization in DistributionZoneManager
> --------------------------------------------------------------
>
> Key: IGNITE-19104
> URL: https://issues.apache.org/jira/browse/IGNITE-19104
> Project: Ignite
> Issue Type: Bug
> Reporter: Andrey Mashenkov
> Priority: Major
> Labels: ignite-3
> Fix For: 3.0.0-beta2
>
>
> DistributionZoneManager run next methods on start
> {code:java}
> initDataNodesFromVaultManager();
> initLogicalTopologyAndVersionInMetaStorageOnStart();
> {code}
> The first method gets logicalTopology from Vault and try to put it into
> MetaStorage.
> The second one gets logicalTopology from CMG and try to put it into
> MetaStorage.
> Both methods actually asynchronous, because Vault.get() and
> TopologyService.logicalTopologyOnLeader() are async.
> There are 2 issues:
> * these methods may run concurrently in separate threads
> * we unconditionally rewrite local volatile field 'logicalTopology' in
> initDataNodesFromVaultManager()
> Thus, we may see initial value (empty topology) after
> DistributionZoneManager.start() finish.
> Also, seems, there is a chance to see stale value from Vault, however a new
> value was got from config, then rewritten by stale value.
> DistributionZoneManagerConfigurationChangesTest passes, because test
> Metastorage initialization happens before the
> DistributionZoneManagerConfigurationChangesTest starts (in reality, they
> start in different order),
> and because test initialization seems a bit slower than
> DistributionZoneManagerConfigurationChangesTest.start().
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
