date:20230407



 [ 
https://issues.apache.org/jira/browse/IGNITE-18496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

YuJue Li updated IGNITE-18496:
--
Component/s: documentation

> Handle documentation feedback
> -
>
> Key: IGNITE-18496
> URL: https://issues.apache.org/jira/browse/IGNITE-18496
> Project: Ignite
>  Issue Type: Task
>  Components: documentation
>Reporter: Igor Gusev
>Assignee: Igor Gusev
>Priority: Major
> Fix For: 2.15
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> We have had bugyard for a while, and there is a lot of useful feedback on 
> documentation. Its time to go through it and fix all issues.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-18496) Handle documentation feedback



 [ 
https://issues.apache.org/jira/browse/IGNITE-18496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

YuJue Li updated IGNITE-18496:
--
Fix Version/s: 2.15

> Handle documentation feedback
> -
>
> Key: IGNITE-18496
> URL: https://issues.apache.org/jira/browse/IGNITE-18496
> Project: Ignite
>  Issue Type: Task
>Reporter: Igor Gusev
>Assignee: Igor Gusev
>Priority: Major
> Fix For: 2.15
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> We have had bugyard for a while, and there is a lot of useful feedback on 
> documentation. Its time to go through it and fix all issues.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (IGNITE-19259) [IEP-94] Reimplement cluster restart command to control.sh

YuJue Li created IGNITE-19259:
-

 Summary: [IEP-94] Reimplement cluster restart command to control.sh
 Key: IGNITE-19259
 URL: https://issues.apache.org/jira/browse/IGNITE-19259
 Project: Ignite
  Issue Type: Improvement
  Components: control.sh
Affects Versions: 2.14
Reporter: YuJue Li


To decomission ignitevisorcmd.sh we need to move all useful commands to control 
script.

 

Cluster restart command is used by users to restart the whole cluster so we 
must provide it via control.sh
h4.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (IGNITE-19258) [IEP-94] Reimplement cluster stop command to control.sh

YuJue Li created IGNITE-19258:
-

 Summary: [IEP-94] Reimplement cluster stop command to control.sh
 Key: IGNITE-19258
 URL: https://issues.apache.org/jira/browse/IGNITE-19258
 Project: Ignite
  Issue Type: Improvement
  Components: control.sh
Affects Versions: 2.14
Reporter: YuJue Li


To decomission ignitevisorcmd.sh we need to move all useful commands to control 
script.

 

Cluster stop command is used by users to stop the whole cluster so we must 
provide it via control.sh
h4.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (IGNITE-19257) The cache is not destroyed if snapshot restore start cache stage failed.

2023-04-07 Thread Nikita Amelchev (Jira)



 [ 
https://issues.apache.org/jira/browse/IGNITE-19257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikita Amelchev reassigned IGNITE-19257:


Assignee: Nikita Amelchev

> The cache is not destroyed if snapshot restore start cache stage failed.
> 
>
> Key: IGNITE-19257
> URL: https://issues.apache.org/jira/browse/IGNITE-19257
> Project: Ignite
>  Issue Type: Bug
>Reporter: Nikita Amelchev
>Assignee: Nikita Amelchev
>Priority: Major
>  Labels: ise
>
> Add the {{RESTORE_CACHE_GROUP_SNAPSHOT_START}} stage to the 
> {{IncrementalSnapshotTest#testStagesFail}} test to reproduce the fail.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (IGNITE-19178) Add Sonar to Ignite project to analyze sources

2023-04-07 Thread Maxim Muzafarov (Jira)



[ 
https://issues.apache.org/jira/browse/IGNITE-19178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17709727#comment-17709727
 ] 

Maxim Muzafarov commented on IGNITE-19178:
--

[~NSAmelchev] thank you for the review. Merged to the master branch.

Examples of Sonar Analysis:
https://sonarcloud.io/summary/overall?id=apache_ignite

A PR Analysis:
https://github.com/apache/ignite/pull/10629


> Add Sonar to Ignite project to analyze sources
> --
>
> Key: IGNITE-19178
> URL: https://issues.apache.org/jira/browse/IGNITE-19178
> Project: Ignite
>  Issue Type: Task
>  Components: build
>Reporter: Maxim Muzafarov
>Assignee: Maxim Muzafarov
>Priority: Major
> Fix For: 2.16
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Sonar is a code quality and security tool that is free to open-source 
> projects and recommended by the INFRA team the link for documentation:
> https://cwiki.apache.org/confluence/display/INFRA/SonarCloud+for+ASF+projects
> Despite in commonly used by many of the ASF projects, it can have the 
> following benefits for us:
> - visualise simple problems for newcomers to work on;
> - see the trends in the source code;
> - add an extra layer of static code analysis;
> The INFRA ticket:
> https://issues.apache.org/jira/browse/INFRA-24415



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (IGNITE-19254) Move ignite-ssh to extensions



 [ 
https://issues.apache.org/jira/browse/IGNITE-19254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikolay Izhikov resolved IGNITE-19254.
--
Resolution: Fixed

> Move ignite-ssh to extensions
> -
>
> Key: IGNITE-19254
> URL: https://issues.apache.org/jira/browse/IGNITE-19254
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Nikolay Izhikov
>Assignee: Nikolay Izhikov
>Priority: Major
> Fix For: 2.15
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-19254) Move ignite-ssh to extensions



 [ 
https://issues.apache.org/jira/browse/IGNITE-19254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikolay Izhikov updated IGNITE-19254:
-
Fix Version/s: 2.15

> Move ignite-ssh to extensions
> -
>
> Key: IGNITE-19254
> URL: https://issues.apache.org/jira/browse/IGNITE-19254
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Nikolay Izhikov
>Assignee: Nikolay Izhikov
>Priority: Major
> Fix For: 2.15
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (IGNITE-19254) Move ignite-ssh to extensions



 [ 
https://issues.apache.org/jira/browse/IGNITE-19254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikolay Izhikov reassigned IGNITE-19254:


Assignee: Nikolay Izhikov

> Move ignite-ssh to extensions
> -
>
> Key: IGNITE-19254
> URL: https://issues.apache.org/jira/browse/IGNITE-19254
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Nikolay Izhikov
>Assignee: Nikolay Izhikov
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (IGNITE-19254) Move ignite-ssh to extensions

2023-04-07 Thread Ignite TC Bot (Jira)



[ 
https://issues.apache.org/jira/browse/IGNITE-19254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17709723#comment-17709723
 ] 

Ignite TC Bot commented on IGNITE-19254:


{panel:title=Branch: [pull/10635/head] Base: [master] : Possible Blockers 
(5)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#F7D6C1}
{color:#d04437}Cache 2{color} [[tests 
1|https://ci2.ignite.apache.org/viewLog.html?buildId=7127653]]
* IgniteCacheTestSuite2: 
GridCachePartitionedTxMultiThreadedSelfTest.testOptimisticReadCommittedCommitMultithreaded
 - Test has low fail rate in base branch 0,0% and is not flaky

{color:#d04437}Start Nodes{color} [[tests 0 Exit Code , Failure on metric 
|https://ci2.ignite.apache.org/viewLog.html?buildId=7127739]]

{color:#d04437}Snapshots{color} [[tests 
1|https://ci2.ignite.apache.org/viewLog.html?buildId=7127733]]
* IgniteSnapshotTestSuite: 
EncryptedSnapshotTest.testSnapshotRestoringFailsWithOtherMasterKey[encryption=true,
 onlyPrimay=false] - Test has low fail rate in base branch 0,0% and is not flaky

{color:#d04437}PDS 4{color} [[tests 
1|https://ci2.ignite.apache.org/viewLog.html?buildId=7127709]]
* IgnitePdsTestSuite4: 
IgniteClusterActivateDeactivateTestWithPersistenceAndMemoryReuse.testActivateInReadOnlySimple_5_Servers
 - Test has low fail rate in base branch 0,0% and is not flaky

{color:#d04437}Queries 3 (lazy=true){color} [[tests 
1|https://ci2.ignite.apache.org/viewLog.html?buildId=7127727]]
* IgniteBinaryCacheQueryLazyTestSuite3: 
CacheEventsCdcTest.testCreateDropSQLTable[persistence=false] - Test has low 
fail rate in base branch 0,0% and is not flaky

{panel}
{panel:title=Branch: [pull/10635/head] Base: [master] : No new tests 
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#F7D6C1}{panel}
[TeamCity *-- Run :: All* 
Results|https://ci2.ignite.apache.org/viewLog.html?buildId=7127755buildTypeId=IgniteTests24Java8_RunAll]

> Move ignite-ssh to extensions
> -
>
> Key: IGNITE-19254
> URL: https://issues.apache.org/jira/browse/IGNITE-19254
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Nikolay Izhikov
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (IGNITE-19257) The cache is not destroyed if snapshot restore start cache stage failed.

2023-04-07 Thread Nikita Amelchev (Jira)

Nikita Amelchev created IGNITE-19257:


 Summary: The cache is not destroyed if snapshot restore start 
cache stage failed.
 Key: IGNITE-19257
 URL: https://issues.apache.org/jira/browse/IGNITE-19257
 Project: Ignite
  Issue Type: Bug
Reporter: Nikita Amelchev


Add the {{RESTORE_CACHE_GROUP_SNAPSHOT_START}} stage to the 
{{IncrementalSnapshotTest#testStagesFail}} test to reproduce the fail.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (IGNITE-19256) .NET: Thin 3.0: Document MemberInit projections in LINQ

Pavel Tupitsyn created IGNITE-19256:
---

 Summary: .NET: Thin 3.0: Document MemberInit projections in LINQ
 Key: IGNITE-19256
 URL: https://issues.apache.org/jira/browse/IGNITE-19256
 Project: Ignite
  Issue Type: Improvement
  Components: platforms
Affects Versions: 3.0.0-beta1
Reporter: Pavel Tupitsyn
Assignee: Sergey Stronchinskiy
 Fix For: 3.0.0-beta2






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-19256) .NET: Thin 3.0: Document MemberInit projections in LINQ



 [ 
https://issues.apache.org/jira/browse/IGNITE-19256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pavel Tupitsyn updated IGNITE-19256:

Description: Update 
https://github.com/apache/ignite-3/blob/main/modules/platforms/dotnet/Apache.Ignite/Internal/Linq/README.md
 with changes made in IGNITE-18120 

> .NET: Thin 3.0: Document MemberInit projections in LINQ
> ---
>
> Key: IGNITE-19256
> URL: https://issues.apache.org/jira/browse/IGNITE-19256
> Project: Ignite
>  Issue Type: Improvement
>  Components: platforms
>Affects Versions: 3.0.0-beta1
>Reporter: Pavel Tupitsyn
>Assignee: Sergey Stronchinskiy
>Priority: Major
>  Labels: .NET, LINQ, ignite-3
> Fix For: 3.0.0-beta2
>
>
> Update 
> https://github.com/apache/ignite-3/blob/main/modules/platforms/dotnet/Apache.Ignite/Internal/Linq/README.md
>  with changes made in IGNITE-18120 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-18120) .NET: Thin 3.0: Allow arbitrary MemberInit projections in LINQ



 [ 
https://issues.apache.org/jira/browse/IGNITE-18120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pavel Tupitsyn updated IGNITE-18120:

Ignite Flags:   (was: Docs Required,Release Notes Required)

> .NET: Thin 3.0: Allow arbitrary MemberInit projections in LINQ
> --
>
> Key: IGNITE-18120
> URL: https://issues.apache.org/jira/browse/IGNITE-18120
> Project: Ignite
>  Issue Type: Improvement
>  Components: platforms
>Reporter: Pavel Tupitsyn
>Assignee: Sergey Stronchinskiy
>Priority: Major
>  Labels: .NET, LINQ, ignite-3
> Fix For: 3.0.0-beta2
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Ignite LINQ provider allows anonymous type projections:
> {code}
> query.Select(emp => new {Id = emp.Key, Name = emp.Value.Name});
> {code}
> However, it does not work with a custom class:
> {code}
> query.Select(emp => new Foo {Id = emp.Key, Name = emp.Value.Name});
> {code}
> throws exception:
> {code}
> System.NotSupportedException : The expression 'new Foo() {Id = [x].Key}' 
> (type: System.Linq.Expressions.MemberInitExpression) is not supported.
> {code}
> Add VisitMemberInit overload to CacheQueryExpressionVisitor to support this 
> scenario. See linked SO page for more details - there is a proposed fix as 
> well.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (IGNITE-18120) .NET: Thin 3.0: Allow arbitrary MemberInit projections in LINQ



 [ 
https://issues.apache.org/jira/browse/IGNITE-18120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pavel Tupitsyn resolved IGNITE-18120.
-
Fix Version/s: 3.0.0-beta2
   Resolution: Fixed

> .NET: Thin 3.0: Allow arbitrary MemberInit projections in LINQ
> --
>
> Key: IGNITE-18120
> URL: https://issues.apache.org/jira/browse/IGNITE-18120
> Project: Ignite
>  Issue Type: Improvement
>  Components: platforms
>Reporter: Pavel Tupitsyn
>Assignee: Sergey Stronchinskiy
>Priority: Major
>  Labels: .NET, LINQ, ignite-3
> Fix For: 3.0.0-beta2
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Ignite LINQ provider allows anonymous type projections:
> {code}
> query.Select(emp => new {Id = emp.Key, Name = emp.Value.Name});
> {code}
> However, it does not work with a custom class:
> {code}
> query.Select(emp => new Foo {Id = emp.Key, Name = emp.Value.Name});
> {code}
> throws exception:
> {code}
> System.NotSupportedException : The expression 'new Foo() {Id = [x].Key}' 
> (type: System.Linq.Expressions.MemberInitExpression) is not supported.
> {code}
> Add VisitMemberInit overload to CacheQueryExpressionVisitor to support this 
> scenario. See linked SO page for more details - there is a proposed fix as 
> well.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (IGNITE-18120) .NET: Thin 3.0: Allow arbitrary MemberInit projections in LINQ



[ 
https://issues.apache.org/jira/browse/IGNITE-18120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17709707#comment-17709707
 ] 

Pavel Tupitsyn commented on IGNITE-18120:
-

Looks good. Merged to main: 54781ef08e6b66fefaca13259fc888daef5ffe8f

> .NET: Thin 3.0: Allow arbitrary MemberInit projections in LINQ
> --
>
> Key: IGNITE-18120
> URL: https://issues.apache.org/jira/browse/IGNITE-18120
> Project: Ignite
>  Issue Type: Improvement
>  Components: platforms
>Reporter: Pavel Tupitsyn
>Assignee: Sergey Stronchinskiy
>Priority: Major
>  Labels: .NET, LINQ, ignite-3
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Ignite LINQ provider allows anonymous type projections:
> {code}
> query.Select(emp => new {Id = emp.Key, Name = emp.Value.Name});
> {code}
> However, it does not work with a custom class:
> {code}
> query.Select(emp => new Foo {Id = emp.Key, Name = emp.Value.Name});
> {code}
> throws exception:
> {code}
> System.NotSupportedException : The expression 'new Foo() {Id = [x].Key}' 
> (type: System.Linq.Expressions.MemberInitExpression) is not supported.
> {code}
> Add VisitMemberInit overload to CacheQueryExpressionVisitor to support this 
> scenario. See linked SO page for more details - there is a proposed fix as 
> well.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-19255) Fix broken unit tests in distribution-zones module



 [ 
https://issues.apache.org/jira/browse/IGNITE-19255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksandr Polovtcev updated IGNITE-19255:
-
Description: 
In IGNITE-19105 I've changed some internal shenanigans of the 
MetaStorageManager (without affecting its API in any way). After that, nearly 
all unit tests in the {{distribution-zones}} module started to fail. Turns out 
it happened because of extensive mock usages that emulate behavior of the Meta 
Storage. So I decided to replace it with the {{StandaloneMetaStorageManager}} 
implementation and all hell broke loose: many tests emulate Meta Storage 
incorrectly, a lot of races appeared, because many methods became truly 
asynchronous.

This situation is very frustrating: a different component internals were 
changed with no API changes and a completely unrelated module is not longer 
able to pass its tests. Though I fixed most of the failures, some tests are 
still failing and I'm going to try to describe, what's wrong with them:

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationAfterScaleUpTriggeredOnNewCluster}}*
 - this test tests a scenario when we start a node after logical topology was 
updated. I don't know how realistic is this scenario, but the problem is that 
"data nodes" don't get populated with the logical topology nodes on 
{{distributionZoneManager}} start, because {{scheduleTimers}} method, that 
get's invoked from the Meta Storage Watch, doesn't go inside the {{if 
(!addedNodes.isEmpty() && autoAdjustScaleUp != INFINITE_TIMER_VALUE)}} branch.

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationForDefaultZoneAfterScaleUpTriggered}}*
 - same issue as above.

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationForDefaultZoneAfterScaleDownTriggered}}*
 - same issue as above.

*{{DistributionZoneManagerScaleUpTest#testUpdateZoneScaleUpTriggersDataNodePropagation}}*
 - this test fails with the following assertion error: _Expected revision that 
is greater or equal to already seen meta storage events._. This is because 
TestConfigurationStorage does not use the same revision as the Meta Storage, 
therefore their revisions can't be compared directly. This should either be 
converted to an integration test or it should use 
`DistributedConfigurationStrorage` instead.

*{{DistributionZoneManagerScaleUpTest#testUpdateZoneScaleDownTriggersDataNodePropagation}}*
 - same issue as above.

*{{DistributionZoneManagerScaleUpTest#testScaleUpDidNotChangeDataNodesWhenTriggerKeyWasConcurrentlyChanged}}*
 - fails, because I had to comment the part which emulates a concurrent Meta 
Storage update. Nothing is wrong with this test, we simply need to invent a 
different way to emulate concurrent Meta Storage updates.

*{{DistributionZoneManagerScaleUpTest#testScaleDownDidNotChangeDataNodesWhenTriggerKeyWasConcurrentlyChanged}}*
  - same issue as above.

*{{DistributionZoneManagerWatchListenerTest#testDataNodesOfDefaultZoneUpdatedOnWatchListenerEvent}}*
 - this test is flaky, probably due to some races between Watch and 
Configuration Listener execution (sometimes a retry on {{invoke}} happens and 
{{Mockito#verify}} fails).

  was:
In IGNITE-19105 I've changed some internal shenanigans of the 
MetaStorageManager (without affecting its API in any way). After that, nearly 
all unit tests in the {{distribution-zones}} module started to fail. Turns out 
it happened because of extensive mock usages that emulate behavior of the Meta 
Storage. So I decided to replace it with the {{StandaloneMetaStorageManager}} 
implementation and all hell broke loose: many tests emulate Meta Storage 
incorrectly, a lot of races appeared, because many methods became truly 
asynchronous.

This situation is very frustrating: a different component internals were 
changed with no API changes and a completely unrelated module is not longer 
able to pass its tests. Though I fixed most of the failures, some tests are 
still failing and I'm going to try to describe, what's wrong with them:

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationAfterScaleUpTriggeredOnNewCluster}}*
 - this test tests a scenario when we start a node after logical topology was 
updated. I don't know how realistic is this scenario, but the problem is that 
"data nodes" don't get populated with the logical topology nodes on 
{{distributionZoneManager}} start, because {{scheduleTimers}} method, that 
get's invoked from the Meta Storage Watch, doesn't go inside the {{if 
(!addedNodes.isEmpty() && autoAdjustScaleUp != INFINITE_TIMER_VALUE)}} branch.

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationForDefaultZoneAfterScaleUpTriggered}}*
 - same issue as above.

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationForDefaultZoneAfterScaleDownTriggered}}*
 - same issue as above.

*{{DistributionZoneManagerScaleUpTest#testUpdateZoneScaleUpTriggersDataNodePropagation}}*
 - this test

[jira] [Assigned] (IGNITE-19255) Fix broken unit tests in distribution-zones module



 [ 
https://issues.apache.org/jira/browse/IGNITE-19255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksandr Polovtcev reassigned IGNITE-19255:


Assignee: Mirza Aliev

> Fix broken unit tests in distribution-zones module
> --
>
> Key: IGNITE-19255
> URL: https://issues.apache.org/jira/browse/IGNITE-19255
> Project: Ignite
>  Issue Type: Bug
>Reporter: Aleksandr Polovtcev
>Assignee: Mirza Aliev
>Priority: Blocker
>  Labels: ignite-3
>
> In IGNITE-19105 I've changed some internal shenanigans of the 
> MetaStorageManager (without affecting its API in any way). After that, nearly 
> all unit tests in the {{distribution-zones}} module started to fail. Turns 
> out it happened because of extensive mock usages that emulate behavior of the 
> Meta Storage. So I decided to replace it with the 
> {{StandaloneMetaStorageManager}} implementation and all hell broke loose: 
> many tests emulate Meta Storage incorrectly, a lot of races appeared, because 
> many methods became truly asynchronous.
> This situation is very frustrating: a different component internals were 
> changed with no API changes and a completely unrelated module is not longer 
> able to pass its tests. Though I fixed most of the failures, some tests are 
> still failing and I'm going to try to describe, what's wrong with them:
> *{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationAfterScaleUpTriggeredOnNewCluster}}*
>  - this test tests a scenario when we start a node after logical topology was 
> updated. I don't know how realistic is this scenario, but the problem is that 
> "data nodes" don't get populated with the logical topology nodes on 
> {{distributionZoneManager}} start, because {{scheduleTimers}} method, that 
> get's invoked from the Meta Storage Watch, doesn't go inside the {{if 
> (!addedNodes.isEmpty() && autoAdjustScaleUp != INFINITE_TIMER_VALUE)}} branch.
> *{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationForDefaultZoneAfterScaleUpTriggered}}*
>  - same issue as above.
> *{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationForDefaultZoneAfterScaleDownTriggered}}*
>  - same issue as above.
> *{{DistributionZoneManagerScaleUpTest#testUpdateZoneScaleUpTriggersDataNodePropagation}}*
>  - this test fails with the following assertion error: _Expected revision 
> that is greater or equal to already seen meta storage events._. This is 
> because TestConfigurationStorage does not use the same revision as the Meta 
> Storage, therefore their revisions can't be compared directly. This should 
> either be converted to an integration test or it should use 
> `DistributedConfigurationStrorage` instead.
> *{{DistributionZoneManagerScaleUpTest#testUpdateZoneScaleDownTriggersDataNodePropagation}}*
>  - same issue as above.
> *{{DistributionZoneManagerScaleUpTest#testScaleUpDidNotChangeDataNodesWhenTriggerKeyWasConcurrentlyChanged}}*
>  - fails, because I had to comment the part which emulates a concurrent Meta 
> Storage update. Nothing is wrong with this test, we simply need to invent a 
> different way to emulate concurrent Meta Storage updates.
> *{{DistributionZoneManagerScaleUpTest#testScaleDownDidNotChangeDataNodesWhenTriggerKeyWasConcurrentlyChanged}}*
>   - same issue as above.
> *{{DistributionZoneManagerWatchListenerTest#testDataNodesOfDefaultZoneUpdatedOnWatchListenerEvent}}*
>  - this test is flaky, probably due to some races between Watch and 
> Configuration Listener execution (sometimes a retry on {{invoke}} happens and 
> {{Mockito#verify}} fails).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-19255) Fix broken unit tests in distribution-zones module



 [ 
https://issues.apache.org/jira/browse/IGNITE-19255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksandr Polovtcev updated IGNITE-19255:
-
Description: 
In IGNITE-19105 I've changed some internal shenanigans of the 
MetaStorageManager (without affecting its API in any way). After that, nearly 
all unit tests in the {{distribution-zones}} module started to fail. Turns out 
it happened because of extensive mock usages that emulate behavior of the Meta 
Storage. So I decided to replace it with the {{StandaloneMetaStorageManager}} 
implementation and all hell broke loose: many tests emulate Meta Storage 
incorrectly, a lot of races appeared, because many methods became truly 
asynchronous.

This situation is very frustrating: a different component internals were 
changed with no API changes and a completely unrelated module is not longer 
able to pass its tests. Though I fixed most of the failures, some tests are 
still failing and I'm going to try to describe, what's wrong with them:

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationAfterScaleUpTriggeredOnNewCluster}}*
 - this test tests a scenario when we start a node after logical topology was 
updated. I don't know how realistic is this scenario, but the problem is that 
"data nodes" don't get populated with the logical topology nodes on 
{{distributionZoneManager}} start, because {{scheduleTimers}} method, that 
get's invoked from the Meta Storage Watch, doesn't go inside the {{if 
(!addedNodes.isEmpty() && autoAdjustScaleUp != INFINITE_TIMER_VALUE)}} branch.

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationForDefaultZoneAfterScaleUpTriggered}}*
 - same issue as above.

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationForDefaultZoneAfterScaleDownTriggered}}*
 - same issue as above.

*{{DistributionZoneManagerScaleUpTest#testUpdateZoneScaleUpTriggersDataNodePropagation}}*
 - this test fails with the following assertion error: _Expected revision that 
is greater or equal to already seen meta storage events._. This is because 
TestConfigurationStorage does not use the same revision as the Meta Storage, 
therefore their revisions can't be compared directly. This should either be 
converted to an integration test or it should use 
`DistributedConfigurationStrorage` instead.

*{{DistributionZoneManagerScaleUpTest#testUpdateZoneScaleDownTriggersDataNodePropagation}}*
 - same issue as above.

*{{DistributionZoneManagerWatchListenerTest#testDataNodesOfDefaultZoneUpdatedOnWatchListenerEvent}}*
 - this test is flaky, probably due to some races between Watch and 
Configuration Listener execution (sometimes a retry on {{invoke}} happens and 
{{Mockito#verify}} fails).

  was:
In IGNITE-19105 I've changed some internal shenanigans of the 
MetaStorageManager (without affecting its API in any way). After that, nearly 
all unit tests in the {{distribution-zones}} module started to fail. Turns out 
it happened because of extensive mock usages that emulate behavior of the Meta 
Storage. So I decided to replace it with the {{StandaloneMetaStorageManager}} 
implementation and all hell broke loose: many tests emulate Meta Storage 
incorrectly, a lot of races appeared, because many methods became truly 
asynchronous.

This situation is very frustrating: a different component internals were 
changed with no API changes and a completely unrelated module is not longer 
able to pass its tests. Though I fixed most of the failures, some tests are 
still failing and I'm going to try to describe, what's wrong with them:

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationAfterScaleUpTriggeredOnNewCluster}}*
 - this test tests a scenario when we start a node after logical topology was 
updated. I don't know how realistic is this scenario, but the problem is that 
"data nodes" don't get populated with the logical topology nodes on 
{{distributionZoneManager}} start, because {{scheduleTimers}} method, that 
get's invoked from the Meta Storage Watch, doesn't go inside the {{if 
(!addedNodes.isEmpty() && autoAdjustScaleUp != INFINITE_TIMER_VALUE)}} branch.

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationForDefaultZoneAfterScaleUpTriggered}}*
 - same issue as above.

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationForDefaultZoneAfterScaleDownTriggered}}*
 - same issue as above.

*{{DistributionZoneManagerScaleUpTest#testUpdateZoneScaleUpTriggersDataNodePropagation}}*
 - this test fails with the following assertion error: _Expected revision that 
is greater or equal to already seen meta storage events._. This is because 
TestConfigurationStorage does not use the same revision as the Meta Storage, 
therefore their revisions can't be compared directly. This should either be 
converted to an integration test or it should use 
`DistributedConfigurationStrorage` instead.

[jira] [Updated] (IGNITE-19255) Fix broken unit tests in distribution-zones module



 [ 
https://issues.apache.org/jira/browse/IGNITE-19255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksandr Polovtcev updated IGNITE-19255:
-
Description: 
In IGNITE-19105 I've changed some internal shenanigans of the 
MetaStorageManager (without affecting its API in any way). After that, nearly 
all unit tests in the {{distribution-zones}} module started to fail. Turns out 
it happened because of extensive mock usages that emulate behavior of the Meta 
Storage. So I decided to replace it with the {{StandaloneMetaStorageManager}} 
implementation and all hell broke loose: many tests emulate Meta Storage 
incorrectly, a lot of races appeared, because many methods became truly 
asynchronous.

This situation is very frustrating: a different component internals were 
changed with no API changes and a completely unrelated module is not longer 
able to pass its tests. Though I fixed most of the failures, some tests are 
still failing and I'm going to try to describe, what's wrong with them:

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationAfterScaleUpTriggeredOnNewCluster}}*
 - this test tests a scenario when we start a node after logical topology was 
updated. I don't know how realistic is this scenario, but the problem is that 
"data nodes" don't get populated with the logical topology nodes on 
{{distributionZoneManager}} start, because {{scheduleTimers}} method, that 
get's invoked from the Meta Storage Watch, doesn't go inside the {{if 
(!addedNodes.isEmpty() && autoAdjustScaleUp != INFINITE_TIMER_VALUE)}} branch.

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationForDefaultZoneAfterScaleUpTriggered}}*
 - same issue as above.

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationForDefaultZoneAfterScaleDownTriggered}}*
 - same issue as above.

*{{DistributionZoneManagerScaleUpTest#testUpdateZoneScaleUpTriggersDataNodePropagation}}*
 - this test fails with the following assertion error: _Expected revision that 
is greater or equal to already seen meta storage events._. This is because 
TestConfigurationStorage does not use the same revision as the Meta Storage, 
therefore their revisions can't be compared directly. This should either be 
converted to an integration test or it should use 
`DistributedConfigurationStrorage` instead.

*{{DistributionZoneManagerWatchListenerTest#testDataNodesOfDefaultZoneUpdatedOnWatchListenerEvent}}*
 - this test is flaky, probably due to some races between Watch and 
Configuration Listener execution (sometimes a retry on {{invoke}} happens and 
{{Mockito#verify}} fails).

  was:
In IGNITE-19105 I've changed some internal shenanigans of the 
MetaStorageManager (without affecting its API in any way). After that, nearly 
all unit tests in the {{distribution-zones}} module started to fail. Turns out 
it happened because of extensive mock usages that emulate behavior of the Meta 
Storage. So I decided to replace it with the {{StandaloneMetaStorageManager}} 
implementation and all hell broke loose: many tests emulate Meta Storage 
incorrectly, a lot of races appeared, because many methods became truly 
asynchronous.

This situation is very frustrating: a different component internals were 
changed with no API changes and a completely unrelated module is not longer 
able to pass its tests. Though I fixed most of the failures, some tests are 
still failing and I'm going to try to describe, what's wrong with them:

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationAfterScaleUpTriggeredOnNewCluster}}*
 - this test tests a scenario when we start a node after logical topology was 
updated. I don't know how realistic is this scenario, but the problem is that 
"data nodes" don't get populated with the logical topology nodes on 
{{distributionZoneManager}} start, because {{scheduleTimers}} method, that 
get's invoked from the Meta Storage Watch, doesn't go inside the {{if 
(!addedNodes.isEmpty() && autoAdjustScaleUp != INFINITE_TIMER_VALUE)}} branch.

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationForDefaultZoneAfterScaleUpTriggered}}*
 - same issue as above.

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationForDefaultZoneAfterScaleDownTriggered}}*
 - same issue as above.

*{{DistributionZoneManagerScaleUpTest#testUpdateZoneScaleUpTriggersDataNodePropagation}}
 - this test fails with the following assertion error: _Expected revision that 
is greater or equal to already seen meta storage events._. This is because 
TestConfigurationStorage does not use the same revision as the Meta Storage, 
therefore their revisions can't be compared directly. This should either be 
converted to an integration test or it should use 
`DistributedConfigurationStrorage` instead.

*{{DistributionZoneManagerWatchListenerTest#testDataNodesOfDefaultZoneUpdatedOnWatchListenerEvent}}*
 - this test is flaky, probably due to some races between Watch and 
Configuration

[jira] [Updated] (IGNITE-19255) Fix broken unit tests in distribution-zones module



 [ 
https://issues.apache.org/jira/browse/IGNITE-19255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksandr Polovtcev updated IGNITE-19255:
-
Description: 
In IGNITE-19105 I've changed some internal shenanigans of the 
MetaStorageManager (without affecting its API in any way). After that, nearly 
all unit tests in the {{distribution-zones}} module started to fail. Turns out 
it happened because of extensive mock usages that emulate behavior of the Meta 
Storage. So I decided to replace it with the {{StandaloneMetaStorageManager}} 
implementation and all hell broke loose: many tests emulate Meta Storage 
incorrectly, a lot of races appeared, because many methods became truly 
asynchronous.

This situation is very frustrating: a different component internals were 
changed with no API changes and a completely unrelated module is not longer 
able to pass its tests. Though I fixed most of the failures, some tests are 
still failing and I'm going to try to describe, what's wrong with them:

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationAfterScaleUpTriggeredOnNewCluster}}*
 - this test tests a scenario when we start a node after logical topology was 
updated. I don't know how realistic is this scenario, but the problem is that 
"data nodes" don't get populated with the logical topology nodes on 
{{distributionZoneManager}} start, because {{scheduleTimers}} method, that 
get's invoked from the Meta Storage Watch, doesn't go inside the {{if 
(!addedNodes.isEmpty() && autoAdjustScaleUp != INFINITE_TIMER_VALUE)}} branch.

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationForDefaultZoneAfterScaleUpTriggered}}*
 - same issue as above.

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationForDefaultZoneAfterScaleDownTriggered}}*
 - same issue as above.

*{{DistributionZoneManagerScaleUpTest#testUpdateZoneScaleUpTriggersDataNodePropagation}}
 - this test fails with the following assertion error: _Expected revision that 
is greater or equal to already seen meta storage events._. This is because 
TestConfigurationStorage does not use the same revision as the Meta Storage, 
therefore their revisions can't be compared directly. This should either be 
converted to an integration test or it should use 
`DistributedConfigurationStrorage` instead.

*{{DistributionZoneManagerWatchListenerTest#testDataNodesOfDefaultZoneUpdatedOnWatchListenerEvent}}*
 - this test is flaky, probably due to some races between Watch and 
Configuration Listener execution (sometimes a retry on {{invoke}} happens and 
{{Mockito#verify}} fails).

  was:
In IGNITE-19105 I've changed some internal shenanigans of the 
MetaStorageManager (without affecting its API in any way). After that, nearly 
all unit tests in the {{distribution-zones}} module started to fail. Turns out 
it happened because of extensive mock usages that emulate behavior of the Meta 
Storage. So I decided to replace it with the {{StandaloneMetaStorageManager}} 
implementation and all hell broke loose: many tests emulate Meta Storage 
incorrectly, a lot of races appeared, because many methods became truly 
asynchronous.

This situation is very frustrating: a different component internals were 
changed with no API changes and a completely unrelated module is not longer 
able to pass its tests. Though I fixed most of the failures, some tests are 
still failing and I'm going to try to describe, what's wrong with them:

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationAfterScaleUpTriggeredOnNewCluster}}*
 - this test tests a scenario when we start a node after logical topology was 
updated. I don't know how realistic is this scenario, but the problem is that 
"data nodes" don't get populated with the logical topology nodes on 
{{distributionZoneManager}} start, because {{scheduleTimers}} method, that 
get's invoked from the Meta Storage Watch, doesn't go inside the {{if 
(!addedNodes.isEmpty() && autoAdjustScaleUp != INFINITE_TIMER_VALUE)}} branch.

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationForDefaultZoneAfterScaleUpTriggered}}*
 - same issue as above.

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationForDefaultZoneAfterScaleDownTriggered}}*
 - same issue as above.

*{{DistributionZoneManagerWatchListenerTest#testDataNodesOfDefaultZoneUpdatedOnWatchListenerEvent}}*
 - this test is flaky, probably due to some races between Watch and 
Configuration Listener execution (sometimes a retry on {{invoke}} happens and 
{{Mockito#verify}} fails).


> Fix broken unit tests in distribution-zones module
> --
>
> Key: IGNITE-19255
> URL: https://issues.apache.org/jira/browse/IGNITE-19255
> Project: Ignite
>  Issue Type: Bug
>Reporter: Aleksandr Polovtcev
>Priority: Blocker
>  Labels: ignite-3
>
> In IGNITE-19105 I've

[jira] [Updated] (IGNITE-17842) .NET: LINQ GroupBy with anonymous type produces invalid SQL



 [ 
https://issues.apache.org/jira/browse/IGNITE-17842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pavel Tupitsyn updated IGNITE-17842:

Fix Version/s: (was: 2.15)

> .NET: LINQ GroupBy with anonymous type produces invalid SQL
> ---
>
> Key: IGNITE-17842
> URL: https://issues.apache.org/jira/browse/IGNITE-17842
> Project: Ignite
>  Issue Type: Bug
>  Components: platforms
>Reporter: Pavel Tupitsyn
>Assignee: Pavel Tupitsyn
>Priority: Major
>  Labels: .NET
>
> To reproduce, change *TestGroupBy* like this:
> {code}
> CollectionAssert.AreEquivalent(new[] { 1000, 1001 },
> persons.GroupBy(x => new { I0 = x.Value.OrganizationId 
> }).Select(x => x.Key.I0).ToArray());
> {code}
> Result:
> {code}
> Apache.Ignite.Core.Common.IgniteException : Failed to parse query. Column 
> "_T0.I0" not found; SQL statement:
> select _T0.I0 from PERSON_ORG_SCHEMA.Person as _T0 group by 
> (_T0.ORGANIZATIONID)
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17550) .NET: SslStreamFactory does not dispose X509Certificate2



 [ 
https://issues.apache.org/jira/browse/IGNITE-17550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pavel Tupitsyn updated IGNITE-17550:

Fix Version/s: (was: 2.15)

> .NET: SslStreamFactory does not dispose X509Certificate2
> 
>
> Key: IGNITE-17550
> URL: https://issues.apache.org/jira/browse/IGNITE-17550
> Project: Ignite
>  Issue Type: Bug
>  Components: platforms, thin client
>Reporter: Pavel Tupitsyn
>Assignee: Pavel Tupitsyn
>Priority: Major
>
> See 
> https://ayende.com/blog/198081-A/managing-the-most-dangerous-constructor-ever,
>  https://snede.net/the-most-dangerous-constructor-in-net/



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-19255) Fix broken unit tests in distribution-zones module



 [ 
https://issues.apache.org/jira/browse/IGNITE-19255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksandr Polovtcev updated IGNITE-19255:
-
Description: 
In IGNITE-19105 I've changed some internal shenanigans of the 
MetaStorageManager (without affecting its API in any way). After that, nearly 
all unit tests in the {{distribution-zones}} module started to fail. Turns out 
it happened because of extensive mock usages that emulate behavior of the Meta 
Storage. So I decided to replace it with the {{StandaloneMetaStorageManager}} 
implementation and all hell broke loose: many tests emulate Meta Storage 
incorrectly, a lot of races appeared, because many methods became truly 
asynchronous.

This situation is very frustrating: a different component internals were 
changed with no API changes and a completely unrelated module is not longer 
able to pass its tests. Though I fixed most of the failures, some tests are 
still failing and I'm going to try to describe, what's wrong with them:

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationAfterScaleUpTriggeredOnNewCluster}}*
 - this test tests a scenario when we start a node after logical topology was 
updated. I don't know how realistic is this scenario, but the problem is that 
"data nodes" don't get populated with the logical topology nodes on 
{{distributionZoneManager}} start, because {{scheduleTimers}} method, that 
get's invoked from the Meta Storage Watch, doesn't go inside the {{if 
(!addedNodes.isEmpty() && autoAdjustScaleUp != INFINITE_TIMER_VALUE)}} branch.

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationForDefaultZoneAfterScaleUpTriggered}}*
 - same issue as above.

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationForDefaultZoneAfterScaleDownTriggered}}*
 - same issue as above.

*{{DistributionZoneManagerWatchListenerTest#testDataNodesOfDefaultZoneUpdatedOnWatchListenerEvent}}*
 - this test is flaky, probably due to some races between Watch and 
Configuration Listener execution (sometimes a retry on {{invoke}} happens and 
{{Mockito#verify}} fails).

  was:
In IGNITE-19105 I've changed some internal shenanigans of the 
MetaStorageManager (without affecting its API in any way). After that, nearly 
all unit tests in the {{distribution-zones}} module started to fail. Turns out 
it happened because of extensive mock usages that emulate behavior of the Meta 
Storage. So I decided to replace it with the {{StandaloneMetaStorageManager}} 
implementation and all hell broke loose: many tests emulate Meta Storage 
incorrectly, a lot of races appeared, because many methods became truly 
asynchronous.

This situation is very frustrating: a different component internals were 
changed with no API changes and a completely unrelated module is not longer 
able to pass its tests. Though I fixed most of the failures, some tests are 
still failing and I'm going to try to describe, what's wrong with them:

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationAfterScaleUpTriggeredOnNewCluster}}*
 - this test tests a scenario when we start a node after logical topology was 
updated. I don't know how realistic is this scenario, but the problem is that 
"data nodes" don't get populated with the logical topology nodes on 
{{distributionZoneManager}} start, because {{scheduleTimers}} method, that 
get's invoked from the Meta Storage Watch, doesn't go inside the {{if 
(!addedNodes.isEmpty() && autoAdjustScaleUp != INFINITE_TIMER_VALUE)}} branch.

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationForDefaultZoneAfterScaleUpTriggered}}*
 - same issue as above.

*{{DistributionZoneManagerWatchListenerTest#testDataNodesOfDefaultZoneUpdatedOnWatchListenerEvent}}*
 - this test is flaky, probably due to some races between Watch and 
Configuration Listener execution (sometimes a retry on {{invoke}} happens and 
{{Mockito#verify}} fails).


> Fix broken unit tests in distribution-zones module
> --
>
> Key: IGNITE-19255
> URL: https://issues.apache.org/jira/browse/IGNITE-19255
> Project: Ignite
>  Issue Type: Bug
>Reporter: Aleksandr Polovtcev
>Priority: Blocker
>  Labels: ignite-3
>
> In IGNITE-19105 I've changed some internal shenanigans of the 
> MetaStorageManager (without affecting its API in any way). After that, nearly 
> all unit tests in the {{distribution-zones}} module started to fail. Turns 
> out it happened because of extensive mock usages that emulate behavior of the 
> Meta Storage. So I decided to replace it with the 
> {{StandaloneMetaStorageManager}} implementation and all hell broke loose: 
> many tests emulate Meta Storage incorrectly, a lot of races appeared, because 
> many methods became truly asynchronous.
> This situation is very frustrating: a different component internals were 
> changed

[jira] [Updated] (IGNITE-19255) Fix broken unit tests in distribution-zones module



 [ 
https://issues.apache.org/jira/browse/IGNITE-19255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksandr Polovtcev updated IGNITE-19255:
-
Description: 
In IGNITE-19105 I've changed some internal shenanigans of the 
MetaStorageManager (without affecting its API in any way). After that, nearly 
all unit tests in the {{distribution-zones}} module started to fail. Turns out 
it happened because of extensive mock usages that emulate behavior of the Meta 
Storage. So I decided to replace it with the {{StandaloneMetaStorageManager}} 
implementation and all hell broke loose: many tests emulate Meta Storage 
incorrectly, a lot of races appeared, because many methods became truly 
asynchronous.

This situation is very frustrating: a different component internals were 
changed with no API changes and a completely unrelated module is not longer 
able to pass its tests. Though I fixed most of the failures, some tests are 
still failing and I'm going to try to describe, what's wrong with them:

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationAfterScaleUpTriggeredOnNewCluster}}*
 - this test tests a scenario when we start a node after logical topology was 
updated. I don't know how realistic is this scenario, but the problem is that 
"data nodes" don't get populated with the logical topology nodes on 
{{distributionZoneManager}} start, because {{scheduleTimers}} method, that 
get's invoked from the Meta Storage Watch, doesn't go inside the {{if 
(!addedNodes.isEmpty() && autoAdjustScaleUp != INFINITE_TIMER_VALUE)}} branch.

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationForDefaultZoneAfterScaleUpTriggered}}*
 - same issue as above.

*{{DistributionZoneManagerWatchListenerTest#testDataNodesOfDefaultZoneUpdatedOnWatchListenerEvent}}*
 - this test is flaky, probably due to some races between Watch and 
Configuration Listener execution (sometimes a retry on {{invoke}} happens and 
{{Mockito#verify}} fails).

  was:
In IGNITE-19105 I've changed some internal shenanigans of the 
MetaStorageManager (without affecting its API in any way). After that, nearly 
all unit tests in the {{distribution-zones}} module started to fail. Turns out 
it happened because of extensive mock usages that emulate behavior of the Meta 
Storage. So I decided to replace it with the {{StandaloneMetaStorageManager}} 
implementation and all hell broke loose: many tests emulate Meta Storage 
incorrectly, a lot of races appeared, because many methods became truly 
asynchronous.

This situation is very frustrating: a different component internals were 
changed with no API changes and a completely unrelated module is not longer 
able to pass its tests. Though I fixed most of the failures, some tests are 
still failing and I'm going to try to describe, what's wrong with them:

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationAfterScaleUpTriggeredOnNewCluster}}*
 - this test tests a scenario when we start a node after logical topology was 
updated. I don't know how realistic is this scenario, but the problem is that 
"data nodes" don't get populated with the logical topology nodes on 
{{distributionZoneManager}} start, because {{scheduleTimers}} method, that 
get's invoked from the Meta Storage Watch, doesn't go inside the {{if 
(!addedNodes.isEmpty() && autoAdjustScaleUp != INFINITE_TIMER_VALUE)}} branch.

*{{DistributionZoneManagerWatchListenerTest#testDataNodesOfDefaultZoneUpdatedOnWatchListenerEvent}}*
 - this test is flaky, probably due to some races between Watch and 
Configuration Listener execution (sometimes a retry on {{invoke}} happens and 
{{Mockito#verify}} fails).


> Fix broken unit tests in distribution-zones module
> --
>
> Key: IGNITE-19255
> URL: https://issues.apache.org/jira/browse/IGNITE-19255
> Project: Ignite
>  Issue Type: Bug
>Reporter: Aleksandr Polovtcev
>Priority: Blocker
>  Labels: ignite-3
>
> In IGNITE-19105 I've changed some internal shenanigans of the 
> MetaStorageManager (without affecting its API in any way). After that, nearly 
> all unit tests in the {{distribution-zones}} module started to fail. Turns 
> out it happened because of extensive mock usages that emulate behavior of the 
> Meta Storage. So I decided to replace it with the 
> {{StandaloneMetaStorageManager}} implementation and all hell broke loose: 
> many tests emulate Meta Storage incorrectly, a lot of races appeared, because 
> many methods became truly asynchronous.
> This situation is very frustrating: a different component internals were 
> changed with no API changes and a completely unrelated module is not longer 
> able to pass its tests. Though I fixed most of the failures, some tests are 
> still failing and I'm going to try to describe, what's wrong with them:
>

[jira] [Commented] (IGNITE-19252) The incremental snapshot restore operation fails if there is a node not from the baseline.

2023-04-07 Thread Ignite TC Bot (Jira)



[ 
https://issues.apache.org/jira/browse/IGNITE-19252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17709689#comment-17709689
 ] 

Ignite TC Bot commented on IGNITE-19252:


{panel:title=Branch: [pull/10630/head] Base: [master] : No blockers 
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}{panel}
{panel:title=Branch: [pull/10630/head] Base: [master] : New Tests 
(2)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}
{color:#8b}Disk Page Compressions 1{color} [[tests 
1|https://ci2.ignite.apache.org/viewLog.html?buildId=7126467]]
* {color:#013220}IgnitePdsCompressionTestSuite: 
IncrementalSnapshotRestoreTest.testRecoveryWithNotBaselineNode - PASSED{color}

{color:#8b}Snapshots{color} [[tests 
1|https://ci2.ignite.apache.org/viewLog.html?buildId=7127632]]
* {color:#013220}IgniteSnapshotTestSuite: 
IncrementalSnapshotRestoreTest.testRecoveryWithNotBaselineNode - PASSED{color}

{panel}
[TeamCity *-- Run :: All* 
Results|https://ci2.ignite.apache.org/viewLog.html?buildId=7126470buildTypeId=IgniteTests24Java8_RunAll]

> The incremental snapshot restore operation fails if there is a node not from 
> the baseline.
> --
>
> Key: IGNITE-19252
> URL: https://issues.apache.org/jira/browse/IGNITE-19252
> Project: Ignite
>  Issue Type: Bug
>Reporter: Nikita Amelchev
>Assignee: Nikita Amelchev
>Priority: Major
>  Labels: ise
> Fix For: 2.15
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> The incremental snapshot restore operation fails if there is a node not from 
> the baseline:
> {noformat}
> 21:20:40.324 [disco-notifier-worker-#147%server-1%] ERROR 
> org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotRestoreProcess
>  - Failed to restore snapshot cache groups 
> [reqId=55eead09-4da7-4232-8e98-976dba117d91].
> org.apache.ignite.IgniteCheckedException: Snapshot metafile cannot be read 
> due to it doesn't exist: 
> /work/snapshots/snp1/increments/0001/server_3.smf
>   at 
> org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.readFromFile(IgniteSnapshotManager.java:2001)
>  ~[ignite-core-15.0.0-SNAPSHOT.jar:15.0.0-SNAPSHOT]
>   at 
> org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.readIncrementalSnapshotMetadata(IgniteSnapshotManager.java:1098)
>  ~[ignite-core-15.0.0-SNAPSHOT.jar:15.0.0-SNAPSHOT]
>   at 
> org.apache.ignite.internal.processors.cache.persistence.snapshot.IncrementalSnapshotProcessor.process(IncrementalSnapshotProcessor.java:94)
>  ~[ignite-core-15.0.0-SNAPSHOT.jar:15.0.0-SNAPSHOT]
>   at 
> org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotRestoreProcess.restoreIncrementalSnapshot(SnapshotRestoreProcess.java:1466)
>  ~[ignite-core-15.0.0-SNAPSHOT.jar:15.0.0-SNAPSHOT]
>   at 
> org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotRestoreProcess.lambda$incrementalSnapshotRestore$35(SnapshotRestoreProcess.java:1417)
>  ~[ignite-core-15.0.0-SNAPSHOT.jar:15.0.0-SNAPSHOT]
>   at 
> org.apache.ignite.internal.processors.security.thread.SecurityAwareRunnable.run(SecurityAwareRunnable.java:51)
>  ~[ignite-core-15.0.0-SNAPSHOT.jar:15.0.0-SNAPSHOT]
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> ~[?:1.8.0_201]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> ~[?:1.8.0_201]
>   at 
> org.apache.ignite.internal.processors.security.thread.SecurityAwareRunnable.run(SecurityAwareRunnable.java:51)
>  ~[ignite-core-15.0.0-SNAPSHOT.jar:15.0.0-SNAPSHOT]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>  ~[?:1.8.0_201]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  ~[?:1.8.0_201]
>   at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_201]
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-19255) Fix broken unit tests in distribution-zones module



 [ 
https://issues.apache.org/jira/browse/IGNITE-19255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksandr Polovtcev updated IGNITE-19255:
-
Description: 
In IGNITE-19105 I've changed some internal shenanigans of the 
MetaStorageManager (without affecting its API in any way). After that, nearly 
all unit tests in the {{distribution-zones}} module started to fail. Turns out 
it happened because of extensive mock usages that emulate behavior of the Meta 
Storage. So I decided to replace it with the {{StandaloneMetaStorageManager}} 
implementation and all hell broke loose: many tests emulate Meta Storage 
incorrectly, a lot of races appeared, because many methods became truly 
asynchronous.

This situation is very frustrating: a different component internals were 
changed with no API changes and a completely unrelated module is not longer 
able to pass its tests. Though I fixed most of the failures, some tests are 
still failing and I'm going to try to describe, what's wrong with them:

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationAfterScaleUpTriggeredOnNewCluster}}*
 - this test tests a scenario when we start a node after logical topology was 
updated. I don't know how realistic is this scenario, but the problem is that 
"data nodes" don't get populated with the logical topology nodes on 
{{distributionZoneManager}} start, because {{scheduleTimers}} method, that 
get's invoked from the Meta Storage Watch, doesn't go inside the {{if 
(!addedNodes.isEmpty() && autoAdjustScaleUp != INFINITE_TIMER_VALUE)}} branch,

*{{DistributionZoneManagerWatchListenerTest#testDataNodesOfDefaultZoneUpdatedOnWatchListenerEvent}}*
 - this test is flaky, probably due to some races between Watch and 
Configuration Listener execution (sometimes a retry on {{invoke}} happens and 
{{Mockito#verify}} fails)

  was:
In IGNITE-19105 I've changed some internal shenanigans of the 
MetaStorageManager (without affecting its API in any way). After that, nearly 
all unit tests in the {{distribution-zones}} module started to fail. Turns out 
it happened because of extensive mock usages that emulate behavior of the Meta 
Storage. So I decided to replace it with the {{StandaloneMetaStorageManager}} 
implementation and all hell broke loose: many tests emulate Meta Storage 
incorrectly, a lot of races appeared, because many methods became truly 
asynchronous.

This situation is very frustrating: a different component internals were 
changed with no API changes and a completely unrelated module is not longer 
able to pass its tests. Though I fixed most of the failures, some tests are 
still failing and I'm going to try to describe, what's wrong with them:

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationAfterScaleUpTriggeredOnNewCluster}}*
 - this test tests a scenario when we start a node after logical topology was 
updated. I don't know how realistic is this scenario, but the problem is that 
"data nodes" don't get populated with the logical topology nodes on 
{{distributionZoneManager}} start, because {{scheduleTimers}} method, that 
get's invoked from the Meta Storage Watch, doesn't go inside the {{if 
(!addedNodes.isEmpty() && autoAdjustScaleUp != INFINITE_TIMER_VALUE)}} branch,

*{{DistributionZoneManagerWatchListenerTest#testDataNodesOfDefaultZoneUpdatedOnWatchListenerEvent}}*
 - this test is flaky, probably due to some races between Watch and 
Configuration Listener execution. 


> Fix broken unit tests in distribution-zones module
> --
>
> Key: IGNITE-19255
> URL: https://issues.apache.org/jira/browse/IGNITE-19255
> Project: Ignite
>  Issue Type: Bug
>Reporter: Aleksandr Polovtcev
>Priority: Blocker
>  Labels: ignite-3
>
> In IGNITE-19105 I've changed some internal shenanigans of the 
> MetaStorageManager (without affecting its API in any way). After that, nearly 
> all unit tests in the {{distribution-zones}} module started to fail. Turns 
> out it happened because of extensive mock usages that emulate behavior of the 
> Meta Storage. So I decided to replace it with the 
> {{StandaloneMetaStorageManager}} implementation and all hell broke loose: 
> many tests emulate Meta Storage incorrectly, a lot of races appeared, because 
> many methods became truly asynchronous.
> This situation is very frustrating: a different component internals were 
> changed with no API changes and a completely unrelated module is not longer 
> able to pass its tests. Though I fixed most of the failures, some tests are 
> still failing and I'm going to try to describe, what's wrong with them:
> *{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationAfterScaleUpTriggeredOnNewCluster}}*
>  - this test tests a scenario when we start a node after logical topology was 
> updated. I don't know how realistic is this

[jira] [Updated] (IGNITE-19255) Fix broken unit tests in distribution-zones module



 [ 
https://issues.apache.org/jira/browse/IGNITE-19255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksandr Polovtcev updated IGNITE-19255:
-
Description: 
In IGNITE-19105 I've changed some internal shenanigans of the 
MetaStorageManager (without affecting its API in any way). After that, nearly 
all unit tests in the {{distribution-zones}} module started to fail. Turns out 
it happened because of extensive mock usages that emulate behavior of the Meta 
Storage. So I decided to replace it with the {{StandaloneMetaStorageManager}} 
implementation and all hell broke loose: many tests emulate Meta Storage 
incorrectly, a lot of races appeared, because many methods became truly 
asynchronous.

This situation is very frustrating: a different component internals were 
changed with no API changes and a completely unrelated module is not longer 
able to pass its tests. Though I fixed most of the failures, some tests are 
still failing and I'm going to try to describe, what's wrong with them:

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationAfterScaleUpTriggeredOnNewCluster}}*
 - this test tests a scenario when we start a node after logical topology was 
updated. I don't know how realistic is this scenario, but the problem is that 
"data nodes" don't get populated with the logical topology nodes on 
{{distributionZoneManager}} start, because {{scheduleTimers}} method, that 
get's invoked from the Meta Storage Watch, doesn't go inside the {{if 
(!addedNodes.isEmpty() && autoAdjustScaleUp != INFINITE_TIMER_VALUE)}} branch.

*{{DistributionZoneManagerWatchListenerTest#testDataNodesOfDefaultZoneUpdatedOnWatchListenerEvent}}*
 - this test is flaky, probably due to some races between Watch and 
Configuration Listener execution (sometimes a retry on {{invoke}} happens and 
{{Mockito#verify}} fails).

  was:
In IGNITE-19105 I've changed some internal shenanigans of the 
MetaStorageManager (without affecting its API in any way). After that, nearly 
all unit tests in the {{distribution-zones}} module started to fail. Turns out 
it happened because of extensive mock usages that emulate behavior of the Meta 
Storage. So I decided to replace it with the {{StandaloneMetaStorageManager}} 
implementation and all hell broke loose: many tests emulate Meta Storage 
incorrectly, a lot of races appeared, because many methods became truly 
asynchronous.

This situation is very frustrating: a different component internals were 
changed with no API changes and a completely unrelated module is not longer 
able to pass its tests. Though I fixed most of the failures, some tests are 
still failing and I'm going to try to describe, what's wrong with them:

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationAfterScaleUpTriggeredOnNewCluster}}*
 - this test tests a scenario when we start a node after logical topology was 
updated. I don't know how realistic is this scenario, but the problem is that 
"data nodes" don't get populated with the logical topology nodes on 
{{distributionZoneManager}} start, because {{scheduleTimers}} method, that 
get's invoked from the Meta Storage Watch, doesn't go inside the {{if 
(!addedNodes.isEmpty() && autoAdjustScaleUp != INFINITE_TIMER_VALUE)}} branch,

*{{DistributionZoneManagerWatchListenerTest#testDataNodesOfDefaultZoneUpdatedOnWatchListenerEvent}}*
 - this test is flaky, probably due to some races between Watch and 
Configuration Listener execution (sometimes a retry on {{invoke}} happens and 
{{Mockito#verify}} fails)


> Fix broken unit tests in distribution-zones module
> --
>
> Key: IGNITE-19255
> URL: https://issues.apache.org/jira/browse/IGNITE-19255
> Project: Ignite
>  Issue Type: Bug
>Reporter: Aleksandr Polovtcev
>Priority: Blocker
>  Labels: ignite-3
>
> In IGNITE-19105 I've changed some internal shenanigans of the 
> MetaStorageManager (without affecting its API in any way). After that, nearly 
> all unit tests in the {{distribution-zones}} module started to fail. Turns 
> out it happened because of extensive mock usages that emulate behavior of the 
> Meta Storage. So I decided to replace it with the 
> {{StandaloneMetaStorageManager}} implementation and all hell broke loose: 
> many tests emulate Meta Storage incorrectly, a lot of races appeared, because 
> many methods became truly asynchronous.
> This situation is very frustrating: a different component internals were 
> changed with no API changes and a completely unrelated module is not longer 
> able to pass its tests. Though I fixed most of the failures, some tests are 
> still failing and I'm going to try to describe, what's wrong with them:
> *{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationAfterScaleUpTriggeredOnNewCluster}}*
>  - this test tests a scenario when we start a node after

[jira] [Updated] (IGNITE-19255) Fix broken unit tests in distribution-zones module



 [ 
https://issues.apache.org/jira/browse/IGNITE-19255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksandr Polovtcev updated IGNITE-19255:
-
Description: 
In IGNITE-19105 I've changed some internal shenanigans of the 
MetaStorageManager (without affecting its API in any way). After that, nearly 
all unit tests in the {{distribution-zones}} module started to fail. Turns out 
it happened because of extensive mock usages that emulate behavior of the Meta 
Storage. So I decided to replace it with the {{StandaloneMetaStorageManager}} 
implementation and all hell broke loose: many tests emulate Meta Storage 
incorrectly, a lot of races appeared, because many methods became truly 
asynchronous.

This situation is very frustrating: a different component internals were 
changed with no API changes and a completely unrelated module is not longer 
able to pass its tests. Though I fixed most of the failures, some tests are 
still failing and I'm going to try to describe, what's wrong with them:

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationAfterScaleUpTriggeredOnNewCluster}}*
 - this test tests a scenario when we start a node after logical topology was 
updated. I don't know how realistic is this scenario, but the problem is that 
"data nodes" don't get populated with the logical topology nodes on 
{{distributionZoneManager}} start, because {{scheduleTimers}} method, that 
get's invoked from the Meta Storage Watch, doesn't go inside the {{if 
(!addedNodes.isEmpty() && autoAdjustScaleUp != INFINITE_TIMER_VALUE)}} branch,

*{{DistributionZoneManagerWatchListenerTest#testDataNodesOfDefaultZoneUpdatedOnWatchListenerEvent}}*
 - this test is flaky, probably due to some races between Watch and 
Configuration Listener execution. 

  was:
In IGNITE-19105 I've changed some internal shenanigans of the 
MetaStorageManager (without affecting its API in any way). After that, nearly 
all unit tests in the {{distribution-zones}} module started to fail. Turns out 
it happened because of extensive mock usages that emulate behavior of the Meta 
Storage. So I decided to replace it with the {{StandaloneMetaStorageManager}} 
implementation and all hell broke loose: many tests emulate Meta Storage 
incorrectly, a lot of races appeared, because many methods became truly 
asynchronous.

This situation is very frustrating: a different component internals were 
changed with no API changes and a completely unrelated module is not longer 
able to pass its tests. Though I fixed most of the failures, some tests are 
still failing and I'm going to try to describe, what's wrong with them:

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationAfterScaleUpTriggeredOnNewCluster}}*
 - this test tests a scenario when we start a node after logical topology was 
updated. I don't know how realistic is this scenario, but the problem is that 
"data nodes" don't get populated with the logical topology nodes on 
{{distributionZoneManager}} start, because {{scheduleTimers}} method, that 
get's invoked from the Meta Storage Watch, doesn't go inside the {{if 
(!addedNodes.isEmpty() && autoAdjustScaleUp != INFINITE_TIMER_VALUE)}} branch,


> Fix broken unit tests in distribution-zones module
> --
>
> Key: IGNITE-19255
> URL: https://issues.apache.org/jira/browse/IGNITE-19255
> Project: Ignite
>  Issue Type: Bug
>Reporter: Aleksandr Polovtcev
>Priority: Blocker
>  Labels: ignite-3
>
> In IGNITE-19105 I've changed some internal shenanigans of the 
> MetaStorageManager (without affecting its API in any way). After that, nearly 
> all unit tests in the {{distribution-zones}} module started to fail. Turns 
> out it happened because of extensive mock usages that emulate behavior of the 
> Meta Storage. So I decided to replace it with the 
> {{StandaloneMetaStorageManager}} implementation and all hell broke loose: 
> many tests emulate Meta Storage incorrectly, a lot of races appeared, because 
> many methods became truly asynchronous.
> This situation is very frustrating: a different component internals were 
> changed with no API changes and a completely unrelated module is not longer 
> able to pass its tests. Though I fixed most of the failures, some tests are 
> still failing and I'm going to try to describe, what's wrong with them:
> *{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationAfterScaleUpTriggeredOnNewCluster}}*
>  - this test tests a scenario when we start a node after logical topology was 
> updated. I don't know how realistic is this scenario, but the problem is that 
> "data nodes" don't get populated with the logical topology nodes on 
> {{distributionZoneManager}} start, because {{scheduleTimers}} method, that 
> get's invoked from the Meta Storage Watch, doesn't go inside the {{if 
>

[jira] [Updated] (IGNITE-15147) Possible leak in metrics in PageLockTracker

2023-04-07 Thread Aleksey Plekhanov (Jira)



 [ 
https://issues.apache.org/jira/browse/IGNITE-15147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Plekhanov updated IGNITE-15147:
---
Fix Version/s: 2.16
   (was: 2.15)

> Possible leak in metrics in PageLockTracker
> ---
>
> Key: IGNITE-15147
> URL: https://issues.apache.org/jira/browse/IGNITE-15147
> Project: Ignite
>  Issue Type: Bug
>  Components: persistence
>Affects Versions: 2.10
>Reporter: Sergey Chugunov
>Priority: Major
> Fix For: 2.16
>
>
> In one of PageHandler#readPage methods there is the following code:
> {code:java}
> long pageAddr = readLock(pageMem, cacheId, pageId, page, lsnr);
> if (pageAddr == 0L)
> return lockFailed;
> try {
> PageIO io = pageIoRslvr.resolve(pageAddr);
> return h.run(cacheId, pageId, page, pageAddr, io, null, arg, intArg, 
> statHolder);
> }
> finally {
> readUnlock(pageMem, cacheId, pageId, page, pageAddr, lsnr);
> }
> {code}
> Here we obtain a read lock on a page by calling {{readLock}} method, its 
> implementation is as following:
> {code:java}
> lsnr.onBeforeReadLock(cacheId, pageId, page);
> long pageAddr = pageMem.readLock(cacheId, pageId, page);
> lsnr.onReadLock(cacheId, pageId, page, pageAddr);
> return pageAddr;
> {code}
> And here is a problem: in {{readLock}} we always call {{onReadLock}} for the 
> page but {{onReadUnlock}} is called *only if lock was acquired successfully*.
> Otherwise lock counters end up in incorrect state: {{onReadLock}} called 
> although no lock was actually acqured.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-18471) Integer overflow in LRU page eviction trackers

2023-04-07 Thread Aleksey Plekhanov (Jira)



 [ 
https://issues.apache.org/jira/browse/IGNITE-18471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Plekhanov updated IGNITE-18471:
---
Fix Version/s: 2.16
   (was: 2.15)

> Integer overflow in LRU page eviction trackers
> --
>
> Key: IGNITE-18471
> URL: https://issues.apache.org/jira/browse/IGNITE-18471
> Project: Ignite
>  Issue Type: Bug
>Reporter: Roman Puchkovskiy
>Priority: Major
> Fix For: 2.16
>
>
> In IGNITE-16866, an integer overflow was fixed in 
> RandomLruPageEvictionTracker and Random2LruPageEvictionTracker (namely, 
> start() methods). But there are other integer overflows in these classes that 
> need to be fixed.
> Also, testLargeRegionsWithRandomLRU() and testLargeRegionsWithRandom2LRU() 
> were added to CacheDataRegionConfigurationTest to test the fix, but these 
> tests allocate correspondingly 4Gb and 8Gb of memory which is too much for 
> our CI. It seems reasonable to either reduce region sizes (to 2060 and 1030 
> Gb, correspondingly) to make the tests only allocate 2Gb (which seems 
> bearable), or completely remove the tests and test this differently.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-19255) Fix broken unit tests in distribution-zones module



 [ 
https://issues.apache.org/jira/browse/IGNITE-19255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksandr Polovtcev updated IGNITE-19255:
-
Description: 
In IGNITE-19105 I've changed some internal shenanigans of the 
MetaStorageManager (without affecting its API in any way). After that, nearly 
all unit tests in the {{distribution-zones}} module started to fail. Turns out 
it happened because of extensive mock usages that emulate behavior of the Meta 
Storage. So I decided to replace it with the {{StandaloneMetaStorageManager}} 
implementation and all hell broke loose: many tests emulate Meta Storage 
incorrectly, a lot of races appeared, because many methods became truly 
asynchronous.

This situation is very frustrating: a different component internals were 
changed with no API changes and a completely unrelated module is not longer 
able to pass its tests. Though I fixed most of the failures, some tests are 
still failing and I'm going to try to describe, what's wrong with them:

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationAfterScaleUpTriggeredOnNewCluster}}*
 - this test tests a scenario when we start a node after logical topology was 
updated. I don't know how realistic is this scenario, but the problem is that 
"data nodes" don't get populated with the logical topology nodes on 
{{distributionZoneManager}} start, because {{scheduleTimers}} method, that 
get's invoked from the Meta Storage Watch, doesn't go inside the {{if 
(!addedNodes.isEmpty() && autoAdjustScaleUp != INFINITE_TIMER_VALUE)}} branch,

  was:
In IGNITE-19105 I've changed some internal shenanigans of the 
MetaStorageManager (without affecting its API in any way). After that, nearly 
all unit tests in the {{distribution-zones}} module started to fail. Turns out 
it happened because of extensive mock usages that emulate behavior of the Meta 
Storage. So I decided to replace it with the {{StandaloneMetaStorageManager}} 
implementation and all hell broke loose: many tests emulate Meta Storage 
incorrectly, a lot of races appeared, because many methods became truly 
asynchronous.

This situation is very frustrating: a different component internals were 
changed with no API changes and a completely unrelated module is not longer 
able to pass its tests. Though I fixed most of the failures, some tests are 
still failing and I'm going to try to describe, what's wrong with them:

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationAfterScaleUpTriggeredOnNewCluster}}*
 - this test tests a scenario when we start a node after logical topology was 
updated. I don't know how realistic is this scenario, but the problem is that 
"data nodes" don't get populated with the logical topology nodes on 
{{distributionZoneManager}} start, because {{scheduleTimers}} method, that 
get's invoked from the Meta Storage Watch doesn't go inside the {{if 
(!addedNodes.isEmpty() && autoAdjustScaleUp != INFINITE_TIMER_VALUE)}} branch,


> Fix broken unit tests in distribution-zones module
> --
>
> Key: IGNITE-19255
> URL: https://issues.apache.org/jira/browse/IGNITE-19255
> Project: Ignite
>  Issue Type: Bug
>Reporter: Aleksandr Polovtcev
>Priority: Blocker
>  Labels: ignite-3
>
> In IGNITE-19105 I've changed some internal shenanigans of the 
> MetaStorageManager (without affecting its API in any way). After that, nearly 
> all unit tests in the {{distribution-zones}} module started to fail. Turns 
> out it happened because of extensive mock usages that emulate behavior of the 
> Meta Storage. So I decided to replace it with the 
> {{StandaloneMetaStorageManager}} implementation and all hell broke loose: 
> many tests emulate Meta Storage incorrectly, a lot of races appeared, because 
> many methods became truly asynchronous.
> This situation is very frustrating: a different component internals were 
> changed with no API changes and a completely unrelated module is not longer 
> able to pass its tests. Though I fixed most of the failures, some tests are 
> still failing and I'm going to try to describe, what's wrong with them:
> *{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationAfterScaleUpTriggeredOnNewCluster}}*
>  - this test tests a scenario when we start a node after logical topology was 
> updated. I don't know how realistic is this scenario, but the problem is that 
> "data nodes" don't get populated with the logical topology nodes on 
> {{distributionZoneManager}} start, because {{scheduleTimers}} method, that 
> get's invoked from the Meta Storage Watch, doesn't go inside the {{if 
> (!addedNodes.isEmpty() && autoAdjustScaleUp != INFINITE_TIMER_VALUE)}} branch,



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-19255) Fix broken unit tests in distribution-zones module



 [ 
https://issues.apache.org/jira/browse/IGNITE-19255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksandr Polovtcev updated IGNITE-19255:
-
Description: 
In IGNITE-19105 I've changed some internal shenanigans of the 
MetaStorageManager (without affecting its API in any way). After that, nearly 
all unit tests in the {{distribution-zones}} module started to fail. Turns out 
it happened because of extensive mock usages that emulate behavior of the Meta 
Storage. So I decided to replace it with the {{StandaloneMetaStorageManager}} 
implementation and all hell broke loose: many tests emulate Meta Storage 
incorrectly, a lot of races appeared, because many methods became truly 
asynchronous.

This situation is very frustrating: a different component internals were 
changed with no API changes and a completely unrelated module is not longer 
able to pass its tests. Though I fixed most of the failures, some tests are 
still failing and I'm going to try to describe, what's wrong with them:

*{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationAfterScaleUpTriggeredOnNewCluster}}*
 - this test tests a scenario when we start a node after logical topology was 
updated. I don't know how realistic is this scenario, but the problem is that 
"data nodes" don't get populated with the logical topology nodes on 
{{distributionZoneManager}} start, because {{scheduleTimers}} method, that 
get's invoked from the Meta Storage Watch doesn't go inside the {{if 
(!addedNodes.isEmpty() && autoAdjustScaleUp != INFINITE_TIMER_VALUE)}} branch,

  was:
In IGNITE-19105 I've changed some internal shenanigans of the 
MetaStorageManager (without affecting its API in any way). After that, nearly 
all unit tests in the {{distribution-zones}} module started to fail. Turns out 
it happened because of extensive mock usages that emulate behavior of the Meta 
Storage. So I decided to replace it with the {{StandaloneMetaStorageManager}} 
implementation and all hell broke loose: many tests emulate Meta Storage 
incorrectly, a lot of races appeared, because many methods became truly 
asynchronous.

This situation is very frustrating: a different component internals were 
changed with no API changes and a completely unrelated module is not longer 
able to pass its tests. Though I fixed most of the failures, some tests are 
still failing and I'm going to try to describe, what's wrong with them:

{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationAfterScaleUpTriggeredOnNewCluster}}
 - this test tests a scenario when we start a node after logical topology was 
updated. I don't know how realistic is this scenario, but the problem is that 
"data nodes" don't get populated with the logical topology nodes on 
{{distributionZoneManager}} start, because {{scheduleTimers}} method, that 
get's invoked from the Meta Storage Watch doesn't go inside the {{if 
(!addedNodes.isEmpty() && autoAdjustScaleUp != INFINITE_TIMER_VALUE)}} branch,


> Fix broken unit tests in distribution-zones module
> --
>
> Key: IGNITE-19255
> URL: https://issues.apache.org/jira/browse/IGNITE-19255
> Project: Ignite
>  Issue Type: Bug
>Reporter: Aleksandr Polovtcev
>Priority: Blocker
>  Labels: ignite-3
>
> In IGNITE-19105 I've changed some internal shenanigans of the 
> MetaStorageManager (without affecting its API in any way). After that, nearly 
> all unit tests in the {{distribution-zones}} module started to fail. Turns 
> out it happened because of extensive mock usages that emulate behavior of the 
> Meta Storage. So I decided to replace it with the 
> {{StandaloneMetaStorageManager}} implementation and all hell broke loose: 
> many tests emulate Meta Storage incorrectly, a lot of races appeared, because 
> many methods became truly asynchronous.
> This situation is very frustrating: a different component internals were 
> changed with no API changes and a completely unrelated module is not longer 
> able to pass its tests. Though I fixed most of the failures, some tests are 
> still failing and I'm going to try to describe, what's wrong with them:
> *{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationAfterScaleUpTriggeredOnNewCluster}}*
>  - this test tests a scenario when we start a node after logical topology was 
> updated. I don't know how realistic is this scenario, but the problem is that 
> "data nodes" don't get populated with the logical topology nodes on 
> {{distributionZoneManager}} start, because {{scheduleTimers}} method, that 
> get's invoked from the Meta Storage Watch doesn't go inside the {{if 
> (!addedNodes.isEmpty() && autoAdjustScaleUp != INFINITE_TIMER_VALUE)}} branch,



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-19255) Fix broken unit tests in distribution-zones module



 [ 
https://issues.apache.org/jira/browse/IGNITE-19255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksandr Polovtcev updated IGNITE-19255:
-
Description: 
In IGNITE-19105 I've changed some internal shenanigans of the 
MetaStorageManager (without affecting its API in any way). After that, nearly 
all unit tests in the {{distribution-zones}} module started to fail. Turns out 
it happened because of extensive mock usages that emulate behavior of the Meta 
Storage. So I decided to replace it with the {{StandaloneMetaStorageManager}} 
implementation and all hell broke loose: many tests emulate Meta Storage 
incorrectly, a lot of races appeared, because many methods became truly 
asynchronous.

This situation is very frustrating: a different component internals were 
changed with no API changes and a completely unrelated module is not longer 
able to pass its tests. Though I fixed most of the failures, some tests are 
still failing and I'm going to try to describe, what's wrong with them:

{{DistributionZoneManagerScaleUpTest#testDataNodesPropagationAfterScaleUpTriggeredOnNewCluster}}
 - this test tests a scenario when we start a node after logical topology was 
updated. I don't know how realistic is this scenario, but the problem is that 
"data nodes" don't get populated with the logical topology nodes on 
{{distributionZoneManager}} start, because {{scheduleTimers}} method, that 
get's invoked from the Meta Storage Watch doesn't go inside the {{if 
(!addedNodes.isEmpty() && autoAdjustScaleUp != INFINITE_TIMER_VALUE)}} branch,

> Fix broken unit tests in distribution-zones module
> --
>
> Key: IGNITE-19255
> URL: https://issues.apache.org/jira/browse/IGNITE-19255
> Project: Ignite
>  Issue Type: Bug
>Reporter: Aleksandr Polovtcev
>Priority: Blocker
>  Labels: ignite-3
>
> In IGNITE-19105 I've changed some internal shenanigans of the 
> MetaStorageManager (without affecting its API in any way). After that, nearly 
> all unit tests in the {{distribution-zones}} module started to fail. Turns 
> out it happened because of extensive mock usages that emulate behavior of the 
> Meta Storage. So I decided to replace it with the 
> {{StandaloneMetaStorageManager}} implementation and all hell broke loose: 
> many tests emulate Meta Storage incorrectly, a lot of races appeared, because 
> many methods became truly asynchronous.
> This situation is very frustrating: a different component internals were 
> changed with no API changes and a completely unrelated module is not longer 
> able to pass its tests. Though I fixed most of the failures, some tests are 
> still failing and I'm going to try to describe, what's wrong with them:
> {{DistributionZoneManagerScaleUpTest#testDataNodesPropagationAfterScaleUpTriggeredOnNewCluster}}
>  - this test tests a scenario when we start a node after logical topology was 
> updated. I don't know how realistic is this scenario, but the problem is that 
> "data nodes" don't get populated with the logical topology nodes on 
> {{distributionZoneManager}} start, because {{scheduleTimers}} method, that 
> get's invoked from the Meta Storage Watch doesn't go inside the {{if 
> (!addedNodes.isEmpty() && autoAdjustScaleUp != INFINITE_TIMER_VALUE)}} branch,



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (IGNITE-19255) Fix broken unit tests in distribution-zones module

Aleksandr Polovtcev created IGNITE-19255:


 Summary: Fix broken unit tests in distribution-zones module
 Key: IGNITE-19255
 URL: https://issues.apache.org/jira/browse/IGNITE-19255
 Project: Ignite
  Issue Type: Bug
Reporter: Aleksandr Polovtcev






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (IGNITE-19253) [Missing Tests] check fails on windows agents

2023-04-07 Thread Anton Vinogradov (Jira)



[ 
https://issues.apache.org/jira/browse/IGNITE-19253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17709676#comment-17709676
 ] 

Anton Vinogradov commented on IGNITE-19253:
---

Merged to the master.
[~alexpl], thanks for the review!

> [Missing Tests] check fails on windows agents
> -
>
> Key: IGNITE-19253
> URL: https://issues.apache.org/jira/browse/IGNITE-19253
> Project: Ignite
>  Issue Type: Bug
>Reporter: Anton Vinogradov
>Assignee: Anton Vinogradov
>Priority: Major
> Fix For: 2.15
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> {noformat}
> org.apache.ignite.tools.surefire.testsuites.CheckAllTestsInSuites
> 12:53:08 check
> 12:53:08   java.nio.file.InvalidPathException: Illegal char <:> at index 
> 2: /C:/BuildAgent/work/6429fa5c3148cb5c/modules/tools/target/classes/
> 12:53:08   java.nio.file.InvalidPathException: Illegal char <:> at index 
> 2: /C:/BuildAgent/work/6429fa5c3148cb5c/modules/tools/target/classes/
> at 
> org.apache.ignite.tools.surefire.testsuites.CheckAllTestsInSuites.check(CheckAllTestsInSuites.java:79)
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Comment Edited] (IGNITE-19253) [Missing Tests] check fails on windows agents

2023-04-07 Thread Anton Vinogradov (Jira)



[ 
https://issues.apache.org/jira/browse/IGNITE-19253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17709669#comment-17709669
 ] 

Anton Vinogradov edited comment on IGNITE-19253 at 4/7/23 12:59 PM:


Fixed according to the hints from 
https://stackoverflow.com/questions/9834776/java-nio-file-path-issue (to use 
toURI() instead of getPath())


was (Author: av):
Fixed according to the hints from 
https://stackoverflow.com/questions/9834776/java-nio-file-path-issue

> [Missing Tests] check fails on windows agents
> -
>
> Key: IGNITE-19253
> URL: https://issues.apache.org/jira/browse/IGNITE-19253
> Project: Ignite
>  Issue Type: Bug
>Reporter: Anton Vinogradov
>Assignee: Anton Vinogradov
>Priority: Major
> Fix For: 2.15
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> {noformat}
> org.apache.ignite.tools.surefire.testsuites.CheckAllTestsInSuites
> 12:53:08 check
> 12:53:08   java.nio.file.InvalidPathException: Illegal char <:> at index 
> 2: /C:/BuildAgent/work/6429fa5c3148cb5c/modules/tools/target/classes/
> 12:53:08   java.nio.file.InvalidPathException: Illegal char <:> at index 
> 2: /C:/BuildAgent/work/6429fa5c3148cb5c/modules/tools/target/classes/
> at 
> org.apache.ignite.tools.surefire.testsuites.CheckAllTestsInSuites.check(CheckAllTestsInSuites.java:79)
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (IGNITE-19253) [Missing Tests] check fails on windows agents

2023-04-07 Thread Anton Vinogradov (Jira)



[ 
https://issues.apache.org/jira/browse/IGNITE-19253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17709669#comment-17709669
 ] 

Anton Vinogradov commented on IGNITE-19253:
---

Fixed according to the hints from 
https://stackoverflow.com/questions/9834776/java-nio-file-path-issue

> [Missing Tests] check fails on windows agents
> -
>
> Key: IGNITE-19253
> URL: https://issues.apache.org/jira/browse/IGNITE-19253
> Project: Ignite
>  Issue Type: Bug
>Reporter: Anton Vinogradov
>Assignee: Anton Vinogradov
>Priority: Major
> Fix For: 2.15
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> {noformat}
> org.apache.ignite.tools.surefire.testsuites.CheckAllTestsInSuites
> 12:53:08 check
> 12:53:08   java.nio.file.InvalidPathException: Illegal char <:> at index 
> 2: /C:/BuildAgent/work/6429fa5c3148cb5c/modules/tools/target/classes/
> 12:53:08   java.nio.file.InvalidPathException: Illegal char <:> at index 
> 2: /C:/BuildAgent/work/6429fa5c3148cb5c/modules/tools/target/classes/
> at 
> org.apache.ignite.tools.surefire.testsuites.CheckAllTestsInSuites.check(CheckAllTestsInSuites.java:79)
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-18242) Update ignite dependency: kafka

2023-04-07 Thread Dmitry Pavlov (Jira)



 [ 
https://issues.apache.org/jira/browse/IGNITE-18242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dmitry Pavlov updated IGNITE-18242:
---
Fix Version/s: 2.16
   (was: 2.15)

> Update ignite dependency: kafka
> ---
>
> Key: IGNITE-18242
> URL: https://issues.apache.org/jira/browse/IGNITE-18242
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Aleksandr Nikolaev
>Assignee: Aleksandr Nikolaev
>Priority: Major
>  Labels: ise
> Fix For: 2.16
>
>
> Update Ignite dependency: kafka 2.0.0 to 3.3.1



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-19133) Increase partitions count upper bound

2023-04-07 Thread Kirill Tkalenko (Jira)

[
https://issues.apache.org/jira/browse/IGNITE-19133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Kirill Tkalenko updated IGNITE-19133:
-
Description:
h3. Problem

Data partitioning is used to distribute data (hopefully) evenly across the
cluster and to provide necessary operation parallelism for the end user. As a
rule of thumb, one may consider allocating 256 partitions per Ignite node, in
order to achieve that.

This rule only scales up to a certain point. Imagine a cluster of 1000 nodes,
with a table that has 3 replicas of each partition (ability to lose 1 backup).
With current limit of 65500 partitions, the maximal number of partitions per
node would be {{{}65500*3/1000 ~= 196{}}}. This is the limit of our
scalability, according to aforementioned rule. To provide 256 partitions per
node, the user would have to:
* either increase the number of backups, which proportionally increases
required storage space (affects cost),
* or increase the total number of partitions up to about 85 thousands. This is
not possible right now.

h3. What's the reason of current limit

Disclaimer: I'm not the one who designed it, so my thoughts may be speculative
in some sense.

Short answer is: we need a number of partitions to fit into 2 bytes.

Long answer: in current implementation we have 1 to 1 correspondence between
logical partition id and physical partition id. We use the same value both in
affinity and in physical file name. This makes system simpler, and I believe
that simplicity is the real explanation of the restriction.

Why does it have to be 2 bytes, and not 3, for example. The key is the
structure of page identifiers in data regions:
{code:java}
+---++-+-+
| rotation/item id (1 byte) | flags (1 byte) | partition (2 bytes) | index (4
bytes) |
+---++-+-+{code}
The idea was to fit it into 8 bytes. Good idea, in my opinion. Making it bigger
doesn't feel right.
h3. Proposed solution

As mentioned, there are to components to the problem:
# One to one correspondence between partition ids.
# Hard limit in a single data region, caused by the page id layout.

There's not much we can do with component #2, because the implications are
unpredictable, and the amount of code we would need to fix is astonishing.
h4. More reasonable restrictions

This leads us to the following problem: every single Ignite node can't have
more than 65500 partitions for a table (or distribution zone). So, imagine the
situation:
* user has a cluster with 3 nodes
* user tries to create distribution zone with 3 nodes, 3 replicas for each
partitions and 10 partitions

While this is absurd, the configuration is still "valid", but it leads to 100k
partitions per node, which is impossible.

Such zone configurations must be banned. Such restriction doesn't seem
unreasonable. If a user wants to start so many partitions for such a small
cluster, they really don't understand what they're doing.

This naturally gives us a minimal number of nodes per the number of partitions,
as the following formula (assuming that you can't have 2 replicas of the same
partition on the same Ignite node):
{code:java}
nodes >= min(replicas, ceil(partitions * replicas / 65500))
{code}
This estimation is imprecise, because it assumes perfect distribution. In
reality,
rendezvous affinity is uneven, so the real value must be checked when user
configures the number of nodes for specific distribution zone.
h4. Ties to rebalance

For this question I would probably need an assistance. While affinity
reassignment, each node may store more partition then it's stated in every
single distribution. What do I mean by this:
* imagine node having partitions 1, 2, and 3
* after the reassignment, the node has partitions 3, 4 and 5

Each individual distribution states that node only has 3 partitions, while
during the rebalance, it may store all 5: sending 1 and 2 to some node, and
receiving 4 and 5 from some different node.

Multiply that by a big factor, and it is possible to have situation, where
local number of partitions exceeds 65500. The only way to beat it, in my
opinion, is to lower the hard limit in affinity function to 32xxx per node,
leaving a space for partitions in a MOVING state.
h4. Mapping partition ids

With that being said, all that's left is to map logical partition ids from the
range 0..N (where N is unlimited) to physical ids from the range 0..65500.

Such mapping is a local entity, encapsulated deep inside of the storage engine.
Simplest way to do so is to have a HashMap \{ logical -> physical } and to
increase physical partition id by 1 every time you insert a new value. If the
{{values()}} set is not continuous, one may occupy the gap, it's not too hard
to implement.

Of course, this

[jira] [Updated] (IGNITE-19133) Increase partitions count upper bound

2023-04-07 Thread Ivan Bessonov (Jira)

[
https://issues.apache.org/jira/browse/IGNITE-19133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ivan Bessonov updated IGNITE-19133:
---
Description:
h3. Problem

h3. What's the reason of current limit

Disclaimer: I'm not the one who designed it, so my thoughts may be speculative
in some sense.

Short answer is: we need a number of partitions to fit into 2 bytes.

As mentioned, there are to components to the problem:
# One to one correspondence between partition ids.
# Hard limit in a single data region, caused by the page id layout.

There's not much we can do with component #2, because the implications are
unpredictable, and the amount of code we would need to fix is astonishing.
h4. More reasonable restrictions

While this is absurd, the configuration is still "valid", but it leads to 100k
partitions per node, which is impossible.

This naturally gives us a minimal number of nodes per the number of partitions,
as the following formula (assuming that you can't have 2 replicas of the same
partition on the same Ignite node):
{code:java}
nodes >= min(replicas, ceil(partitions * replicas / 65500))
{code}
This estimation is imprecise, because it assumes perfect distribution. In
reality, rendezvous affinity is uneven, so the real value must be checked when
user configures the number of nodes for specific distribution zone.
h4. Ties to rebalance

Each individual distribution states that node only has 3 partitions, while
during the rebalance, it may store all 5: sending 1 and 2 to some node, and
receiving 4 and 5 from some different node.

With that being said, all that's left is to map logical partition ids from the
range 0..N (where N is unlimited) to physical ids from the range 0..65500.

Such mapping is a local entity, encapsulated deep inside of the storage engine.
Simplest way to do so is to have a HashMap \{ logical -> physical } and to
increase physical partition id by 1 every time you add a new partition to the
node. If the {{values()}} set is not continuous, one may occupy the gap, it's
not too hard to implement.

Of course, this

[jira] [Created] (IGNITE-19254) Move ignite-ssh to extensions