date:20210122

[jira] [Commented] (IGNITE-14039) Add warnings about WAL enable/disable requiring topology stability

2021-01-22 Thread Ignite TC Bot (Jira)



[ 
https://issues.apache.org/jira/browse/IGNITE-14039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17270397#comment-17270397
 ] 

Ignite TC Bot commented on IGNITE-14039:


{panel:title=Branch: [pull/8688/head] Base: [master] : Possible Blockers 
(2)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#F7D6C1}
{color:#d04437}PDS (Compatibility)*{color} [[tests 
2|https://ci.ignite.apache.org/viewLog.html?buildId=5838926]]
* IgniteCompatibilityBasicTestSuite: 
PersistenceBasicCompatibilityTest.testNodeStartByOldVersionPersistenceData_2_8 
- Test has low fail rate in base branch 0,0% and is not flaky
* IgniteCompatibilityBasicTestSuite: 
PersistenceBasicCompatibilityTest.testNodeStartByOldVersionPersistenceData_2_8_1
 - Test has low fail rate in base branch 0,0% and is not flaky

{panel}
{panel:title=Branch: [pull/8688/head] Base: [master] : New Tests 
(3)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}
{color:#8b}PDS 1{color} [[tests 
3|https://ci.ignite.apache.org/viewLog.html?buildId=5838512]]
* {color:#013220}IgnitePdsTestSuite: 
WalEnableDisableWithNodeShutdownTest.testOffWhileNodeOffline - PASSED{color}
* {color:#013220}IgnitePdsTestSuite: 
WalEnableDisableWithNodeShutdownTest.testDisableWhileNodeOffline - PASSED{color}
* {color:#013220}IgnitePdsTestSuite: 
WalEnableDisableWithNodeShutdownTest.testEnableWhileNodeOffline - PASSED{color}

{panel}
[TeamCity *-- Run :: All* 
Results|https://ci.ignite.apache.org/viewLog.html?buildId=5838541buildTypeId=IgniteTests24Java8_RunAll]

> Add warnings about WAL enable/disable requiring topology stability
> --
>
> Key: IGNITE-14039
> URL: https://issues.apache.org/jira/browse/IGNITE-14039
> Project: Ignite
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 2.9.1
>Reporter: Ilya Kasnacheev
>Assignee: Ilya Kasnacheev
>Priority: Blocker
> Fix For: 2.10
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Due to IGNITE-13976 WAL enable/disable is not suitable for any kind of 
> changing topology and will cause cache to go into inconsistent state. We need 
> to add warnings to javadoc, runtime and documentation, and provide tests.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Issue Comment Deleted] (IGNITE-14039) Add warnings about WAL enable/disable requiring topology stability

2021-01-22 Thread Ilya Kasnacheev (Jira)



 [ 
https://issues.apache.org/jira/browse/IGNITE-14039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ilya Kasnacheev updated IGNITE-14039:
-
Comment: was deleted

(was: {panel:title=Branch: [pull/8688/head] Base: [master] : Possible Blockers 
(5)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#F7D6C1}
{color:#d04437}Java Client{color} [[tests 0 TIMEOUT , Exit Code 
|https://ci.ignite.apache.org/viewLog.html?buildId=5838460]]

{color:#d04437}PDS (Compatibility)*{color} [[tests 
4|https://ci.ignite.apache.org/viewLog.html?buildId=5838509]]
* IgniteCompatibilityBasicTestSuite: 
PersistenceBasicCompatibilityTest.testNodeStartByOldVersionPersistenceData_2_2 
- Test has low fail rate in base branch 0,0% and is not flaky
* IgniteCompatibilityBasicTestSuite: 
PersistenceBasicCompatibilityTest.testNodeStartByOldVersionPersistenceData_2_1 
- Test has low fail rate in base branch 0,0% and is not flaky
* IgniteCompatibilityBasicTestSuite: 
PersistenceBasicCompatibilityTest.testNodeStartByOldVersionPersistenceData_2_8_1
 - Test has low fail rate in base branch 0,0% and is not flaky
* IgniteCompatibilityBasicTestSuite: 
PersistenceBasicCompatibilityTest.testNodeStartByOldVersionPersistenceData_2_7_6
 - Test has low fail rate in base branch 0,0% and is not flaky

{panel}
{panel:title=Branch: [pull/8688/head] Base: [master] : New Tests 
(3)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}
{color:#8b}PDS 1{color} [[tests 
3|https://ci.ignite.apache.org/viewLog.html?buildId=5838512]]
* {color:#013220}IgnitePdsTestSuite: 
WalEnableDisableWithNodeShutdownTest.testOffWhileNodeOffline - PASSED{color}
* {color:#013220}IgnitePdsTestSuite: 
WalEnableDisableWithNodeShutdownTest.testDisableWhileNodeOffline - PASSED{color}
* {color:#013220}IgnitePdsTestSuite: 
WalEnableDisableWithNodeShutdownTest.testEnableWhileNodeOffline - PASSED{color}

{panel}
[TeamCity *-- Run :: All* 
Results|https://ci.ignite.apache.org/viewLog.html?buildId=5838541buildTypeId=IgniteTests24Java8_RunAll])

> Add warnings about WAL enable/disable requiring topology stability
> --
>
> Key: IGNITE-14039
> URL: https://issues.apache.org/jira/browse/IGNITE-14039
> Project: Ignite
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 2.9.1
>Reporter: Ilya Kasnacheev
>Assignee: Ilya Kasnacheev
>Priority: Blocker
> Fix For: 2.10
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Due to IGNITE-13976 WAL enable/disable is not suitable for any kind of 
> changing topology and will cause cache to go into inconsistent state. We need 
> to add warnings to javadoc, runtime and documentation, and provide tests.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (IGNITE-14039) Add warnings about WAL enable/disable requiring topology stability

2021-01-22 Thread Ignite TC Bot (Jira)



[ 
https://issues.apache.org/jira/browse/IGNITE-14039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17270301#comment-17270301
 ] 

Ignite TC Bot commented on IGNITE-14039:


{panel:title=Branch: [pull/8688/head] Base: [master] : Possible Blockers 
(5)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#F7D6C1}
{color:#d04437}Java Client{color} [[tests 0 TIMEOUT , Exit Code 
|https://ci.ignite.apache.org/viewLog.html?buildId=5838460]]

{color:#d04437}PDS (Compatibility)*{color} [[tests 
4|https://ci.ignite.apache.org/viewLog.html?buildId=5838509]]
* IgniteCompatibilityBasicTestSuite: 
PersistenceBasicCompatibilityTest.testNodeStartByOldVersionPersistenceData_2_2 
- Test has low fail rate in base branch 0,0% and is not flaky
* IgniteCompatibilityBasicTestSuite: 
PersistenceBasicCompatibilityTest.testNodeStartByOldVersionPersistenceData_2_1 
- Test has low fail rate in base branch 0,0% and is not flaky
* IgniteCompatibilityBasicTestSuite: 
PersistenceBasicCompatibilityTest.testNodeStartByOldVersionPersistenceData_2_8_1
 - Test has low fail rate in base branch 0,0% and is not flaky
* IgniteCompatibilityBasicTestSuite: 
PersistenceBasicCompatibilityTest.testNodeStartByOldVersionPersistenceData_2_7_6
 - Test has low fail rate in base branch 0,0% and is not flaky

{panel}
{panel:title=Branch: [pull/8688/head] Base: [master] : New Tests 
(3)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}
{color:#8b}PDS 1{color} [[tests 
3|https://ci.ignite.apache.org/viewLog.html?buildId=5838512]]
* {color:#013220}IgnitePdsTestSuite: 
WalEnableDisableWithNodeShutdownTest.testOffWhileNodeOffline - PASSED{color}
* {color:#013220}IgnitePdsTestSuite: 
WalEnableDisableWithNodeShutdownTest.testDisableWhileNodeOffline - PASSED{color}
* {color:#013220}IgnitePdsTestSuite: 
WalEnableDisableWithNodeShutdownTest.testEnableWhileNodeOffline - PASSED{color}

{panel}
[TeamCity *-- Run :: All* 
Results|https://ci.ignite.apache.org/viewLog.html?buildId=5838541buildTypeId=IgniteTests24Java8_RunAll]

> Add warnings about WAL enable/disable requiring topology stability
> --
>
> Key: IGNITE-14039
> URL: https://issues.apache.org/jira/browse/IGNITE-14039
> Project: Ignite
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 2.9.1
>Reporter: Ilya Kasnacheev
>Assignee: Ilya Kasnacheev
>Priority: Blocker
> Fix For: 2.10
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Due to IGNITE-13976 WAL enable/disable is not suitable for any kind of 
> changing topology and will cause cache to go into inconsistent state. We need 
> to add warnings to javadoc, runtime and documentation, and provide tests.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (IGNITE-13976) WAL disable/enable with node restarts results in mismatching state, data loss

2021-01-22 Thread Ilya Kasnacheev (Jira)



 [ 
https://issues.apache.org/jira/browse/IGNITE-13976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ilya Kasnacheev reassigned IGNITE-13976:


Assignee: Anton Kalashnikov  (was: Ilya Kasnacheev)

> WAL disable/enable with node restarts results in mismatching state, data loss
> -
>
> Key: IGNITE-13976
> URL: https://issues.apache.org/jira/browse/IGNITE-13976
> Project: Ignite
>  Issue Type: Bug
>  Components: cache
>Affects Versions: 2.9.1
>Reporter: Ilya Kasnacheev
>Assignee: Anton Kalashnikov
>Priority: Critical
>
> If you try to enable/disable WAL on unstable topology, you will get to state 
> when WAL status is undefined, nodes might have different wall status and the 
> only way to fix it is to restart the cluster, which will lead to data loss 
> because ignite removes data if WAL is disabled on restart.
> See the reproducer in PR.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (IGNITE-14027) Switching Auto-adjust on does not trigger including existent nodes(which are not in BLT) to the BLT

2021-01-22 Thread Sergey Chugunov (Jira)



 [ 
https://issues.apache.org/jira/browse/IGNITE-14027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Chugunov updated IGNITE-14027:
-
Fix Version/s: 2.11

> Switching Auto-adjust on does not trigger including existent nodes(which are 
> not in BLT) to the BLT
> ---
>
> Key: IGNITE-14027
> URL: https://issues.apache.org/jira/browse/IGNITE-14027
> Project: Ignite
>  Issue Type: Bug
>  Components: general
>Affects Versions: 2.9.1
>Reporter: Semyon Danilov
>Assignee: Semyon Danilov
>Priority: Major
> Fix For: 2.11
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Steps to reproduce:
>  * Start cluster with AA on
>  * Disable AA
>  * Start new node - it joins topology, not the BL
>  * Enable AA
> Expected:
>  * As _softTimeout_ passes, the node joins the BLT
> Actual:
>  * The node remains out of BLT
> With that new note won't join BLT until either another new node started or AA 
> disabled and the node is included in the BLT manually



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (IGNITE-13976) WAL disable/enable with node restarts results in mismatching state, data loss

2021-01-22 Thread Ilya Kasnacheev (Jira)



 [ 
https://issues.apache.org/jira/browse/IGNITE-13976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ilya Kasnacheev updated IGNITE-13976:
-
Priority: Critical  (was: Blocker)

> WAL disable/enable with node restarts results in mismatching state, data loss
> -
>
> Key: IGNITE-13976
> URL: https://issues.apache.org/jira/browse/IGNITE-13976
> Project: Ignite
>  Issue Type: Bug
>  Components: cache
>Affects Versions: 2.9.1
>Reporter: Ilya Kasnacheev
>Assignee: Ilya Kasnacheev
>Priority: Critical
>
> If you try to enable/disable WAL on unstable topology, you will get to state 
> when WAL status is undefined, nodes might have different wall status and the 
> only way to fix it is to restart the cluster, which will lead to data loss 
> because ignite removes data if WAL is disabled on restart.
> See the reproducer in PR.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (IGNITE-13976) WAL disable/enable with node restarts results in mismatching state, data loss

2021-01-22 Thread Ilya Kasnacheev (Jira)

[
https://issues.apache.org/jira/browse/IGNITE-13976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17270245#comment-17270245
]

Ilya Kasnacheev commented on IGNITE-13976:
--

I will document a list of open questions and problems with the implementation
of WAL enable/disable:

* CacheGroupDescriptor has volatile walEnabled field. CacheGroupDescriptor
denotes configuration of cache group at some moment of time, such as, on node
join. It is not topology versioned. Therefore, if you have a
CacheGroupDescriptor.walEnabled "false", you have no idea in which topology
version it is enabled, and when you are asked to change it, you have no
guarantee that it is in sync with the rest of the cluster.
* CacheGroupContext also has localWalEnabled and globalWalEnabled. I'm not sure
what's the difference. Anyway, CacheGroupDescriptor.walEnabled is changed in a
different place and different thread than CacheGroupContext.globalWalEnabled -
former is set when it is decided that WAL change request should be handled,
latter when relevant checkpoint finishes. This means that node could already
have set CacheGroupDescriptor.walEnabled to false, already sending this value
to new nodes which try to join, but it did not set
CacheGroupContext.globalWalEnabled to false - it is still waiting on checkpoint
to do so.
* Enabling/disabling WAL will trigger checkpoint _on client nodes_. Despite
that fact, it has separate logic for server nodes. I'm not sure what does it do.
* When persistent node starts, it will read its current WAL disabled/enabled
status into maps initiallyLocalWalDisabledGrps/initiallyGlobalWalDisabledGrps.
It will, however, not clear values from these maps when cache is destroyed.
Meaning, when you will recreate such cache, node will still re-use old value of
WAL disabled for that cache. You may get cache that is in inconsistent state
from the start. The very existence of these maps outside of CacheGroupContext
or CacheGroupDescriptor is a liability. I have tried sidestepping it in
IGNITE-14039
* CacheGroupDescriptor has a list of WAL change requests. I don't understand
why we need to have a list of requests per cache, since every one of them
involves a PME, and therefore at any moment you need to do no more than one WAL
change for a cache - either you need to toggle it, or leave as it is.
* #onCachesInfoCollected will try to process the last WAL change request. I'm
not sure why it will discard the rest. For this last WAL change request it will
issue a "changed true" or "changed false" verdict, in order to presumably
notify the rest of nodes in the cluster. However, it does not make sense to
fail operation here because a newcomer node has ignored some of WAL change
requests and responded false to the last one only.
* Did I already mentioned that it does not make sense to have more than one WAL
change request per cache, or change any flag outside of PME blocking operation?
If you do this inside PME lock you may rely that all nodes do this in sync.
* On node startup, CacheGroupDescriptor.walEnabled is received from peer nodes,
representing WAL status at some unknown moment of time. Already bad, it is
compounded by the fact this walEnabled flag is never propagated to
cacheGroupContext. So the cache will start in "WAL enabled" mode regardless of
current cluster WAL mode for this cache.
* CacheGroupDescriptor.walEnabled and CacheGroupContext.globalWalEnabled are
checked in some random different places while WAL state change request is
handled. This is never documented.
* WalStateManager will produce the following warnings: "Received finish message
for unknown operation (will ignore)". However, it will not actually ignore
anything after this warning and continue this operation as usual.
* There's code which blindly toggles the value of "WAL enabled" without
checking that in this specific place it will be set to desired value
{code}
if (msg.changed())
grpDesc.walEnabled(!grpDesc.walEnabled());
{code}

I observe at least four failure scenarios for this code:
- Enabling or disabling WAL state when a baseline node was down. When node
comes back it is in incorrect WAL status. This is checked by sequential test
and always fails embarrasingly (see mailing list thread in IGNITE-14039). I
have tried fixing it in the PR.
- Unknown reason why on the client node WAL state will be different from the
server nodes, after WAL change is requested from this node.
- If server node is joining when WAL state change is underway, it is going to
get incorrect value in its CacheGroupDescriptor for that cache (too old or too
new compared to what the rest of nodes consider recent) and will get out of
sync.
- When cache is destroyed and recreated, nodes will take the wal enabled flag
from their maps populated at startup. See above. Leading to WAL state
inconsistent on the cache from the first

55 matches

Mail list logo