[jira] [Commented] (IGNITE-5553) Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16621716#comment-16621716
 ] 

ASF GitHub Bot commented on IGNITE-5553:


Github user xtern closed the pull request at:

https://github.com/apache/ignite/pull/3587


> Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error
> -
>
> Key: IGNITE-5553
> URL: https://issues.apache.org/jira/browse/IGNITE-5553
> Project: Ignite
>  Issue Type: Bug
>  Components: data structures, persistence
>Affects Versions: 2.1
>Reporter: Dmitriy Pavlov
>Assignee: Pavel Pereslegin
>Priority: Major
>  Labels: MakeTeamcityGreenAgain, Muted_test, test-fail
> Fix For: 2.7
>
>
> h2. Notes-4435
> When IgniteSet is restored from persistence, size of set is always 0, [link 
> to test 
> history|http://ci.ignite.apache.org/project.html?projectId=Ignite20Tests=-7043871603266099589=testDetails].
> h2. Detailed description
> Unlike *IgniteQueue* which uses separate cache key to store its size 
> *IgniteSet* stores it in a field of some class.
> Test from the link above shows very clearly that after restoring memory state 
> from PDS all set values are restored correctly but size is lost.
> h2. Proposed solution
> One possible solution might be to do the same thing as *IgniteQueue* does: 
> size of *IgniteSet* must be stored is cache instead of volatile in-memory 
> fields of random classes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-5553) Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error

[Ryabov Dmitrii (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617613#comment-16617613
 ] 

Ryabov Dmitrii commented on IGNITE-5553:


{panel:title=No blockers 
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}{panel}
[TeamCity Run 
All|http://ci.ignite.apache.org/viewLog.html?buildId=1880784buildTypeId=IgniteTests24Java8_RunAll]

> Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error
> -
>
> Key: IGNITE-5553
> URL: https://issues.apache.org/jira/browse/IGNITE-5553
> Project: Ignite
>  Issue Type: Bug
>  Components: data structures, persistence
>Affects Versions: 2.1
>Reporter: Dmitriy Pavlov
>Assignee: Pavel Pereslegin
>Priority: Major
>  Labels: MakeTeamcityGreenAgain, Muted_test, test-fail
> Fix For: 2.7
>
>
> h2. Notes-4435
> When IgniteSet is restored from persistence, size of set is always 0, [link 
> to test 
> history|http://ci.ignite.apache.org/project.html?projectId=Ignite20Tests=-7043871603266099589=testDetails].
> h2. Detailed description
> Unlike *IgniteQueue* which uses separate cache key to store its size 
> *IgniteSet* stores it in a field of some class.
> Test from the link above shows very clearly that after restoring memory state 
> from PDS all set values are restored correctly but size is lost.
> h2. Proposed solution
> One possible solution might be to do the same thing as *IgniteQueue* does: 
> size of *IgniteSet* must be stored is cache instead of volatile in-memory 
> fields of random classes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-5553) Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error

[Pavel Pereslegin (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617165#comment-16617165
 ] 

Pavel Pereslegin commented on IGNITE-5553:
--

[~avinogradov],
merged with last master.
TC results looks good. 

> Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error
> -
>
> Key: IGNITE-5553
> URL: https://issues.apache.org/jira/browse/IGNITE-5553
> Project: Ignite
>  Issue Type: Bug
>  Components: data structures, persistence
>Affects Versions: 2.1
>Reporter: Dmitriy Pavlov
>Assignee: Pavel Pereslegin
>Priority: Major
>  Labels: MakeTeamcityGreenAgain, Muted_test, test-fail
> Fix For: 2.7
>
>
> h2. Notes-4435
> When IgniteSet is restored from persistence, size of set is always 0, [link 
> to test 
> history|http://ci.ignite.apache.org/project.html?projectId=Ignite20Tests=-7043871603266099589=testDetails].
> h2. Detailed description
> Unlike *IgniteQueue* which uses separate cache key to store its size 
> *IgniteSet* stores it in a field of some class.
> Test from the link above shows very clearly that after restoring memory state 
> from PDS all set values are restored correctly but size is lost.
> h2. Proposed solution
> One possible solution might be to do the same thing as *IgniteQueue* does: 
> size of *IgniteSet* must be stored is cache instead of volatile in-memory 
> fields of random classes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-5553) Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error

[Anton Vinogradov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613692#comment-16613692
 ] 

Anton Vinogradov commented on IGNITE-5553:
--

[~Mmuzaf] , [~xtern] ,

I've checked the changes. 

Initially, looks good to me.

Let's check TC and I will recheck the fix.

> Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error
> -
>
> Key: IGNITE-5553
> URL: https://issues.apache.org/jira/browse/IGNITE-5553
> Project: Ignite
>  Issue Type: Bug
>  Components: data structures, persistence
>Affects Versions: 2.1
>Reporter: Dmitriy Pavlov
>Assignee: Pavel Pereslegin
>Priority: Major
>  Labels: MakeTeamcityGreenAgain, Muted_test, test-fail
> Fix For: 2.7
>
>
> h2. Notes-4435
> When IgniteSet is restored from persistence, size of set is always 0, [link 
> to test 
> history|http://ci.ignite.apache.org/project.html?projectId=Ignite20Tests=-7043871603266099589=testDetails].
> h2. Detailed description
> Unlike *IgniteQueue* which uses separate cache key to store its size 
> *IgniteSet* stores it in a field of some class.
> Test from the link above shows very clearly that after restoring memory state 
> from PDS all set values are restored correctly but size is lost.
> h2. Proposed solution
> One possible solution might be to do the same thing as *IgniteQueue* does: 
> size of *IgniteSet* must be stored is cache instead of volatile in-memory 
> fields of random classes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-5553) Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error

[Maxim Muzafarov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603153#comment-16603153
 ] 

Maxim Muzafarov commented on IGNITE-5553:
-

[~xtern], [~avinogradov]

I've read the whole discussion on dev.list according to this change and 
removing {{setDataMap}} seems to be a bit less risky and easier to implement 
without huge {{IgniteSet}} slowdown impact.

I think prepared PR is ready for final review, whole its changes looks good to 
me.

Anton, can you look at this PR as an expert in data structures an provide your 
thoughts? 
If everything is OK, I think we can start checking TC carefully.

> Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error
> -
>
> Key: IGNITE-5553
> URL: https://issues.apache.org/jira/browse/IGNITE-5553
> Project: Ignite
>  Issue Type: Bug
>  Components: data structures, persistence
>Affects Versions: 2.1
>Reporter: Dmitriy Pavlov
>Assignee: Pavel Pereslegin
>Priority: Major
>  Labels: MakeTeamcityGreenAgain, Muted_test, test-fail
> Fix For: 2.7
>
>
> h2. Notes-4435
> When IgniteSet is restored from persistence, size of set is always 0, [link 
> to test 
> history|http://ci.ignite.apache.org/project.html?projectId=Ignite20Tests=-7043871603266099589=testDetails].
> h2. Detailed description
> Unlike *IgniteQueue* which uses separate cache key to store its size 
> *IgniteSet* stores it in a field of some class.
> Test from the link above shows very clearly that after restoring memory state 
> from PDS all set values are restored correctly but size is lost.
> h2. Proposed solution
> One possible solution might be to do the same thing as *IgniteQueue* does: 
> size of *IgniteSet* must be stored is cache instead of volatile in-memory 
> fields of random classes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-5553) Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error

[Pavel Pereslegin (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603018#comment-16603018
 ] 

Pavel Pereslegin commented on IGNITE-5553:
--

As described above, collocated IgniteSet uses on-heap caching known as 
"setDataMap" to support iterator, and I didn't find the optimal way to recover 
heap data after node restart ([details 
here|http://apache-ignite-developers.2346864.n4.nabble.com/IgniteSet-implementation-changes-required-tp23783p31923.html]).
On devlist we decided to remove this optimization and the changes are ready 
according to this discussion.

[~Mmuzaf], take a look at this patch please.

> Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error
> -
>
> Key: IGNITE-5553
> URL: https://issues.apache.org/jira/browse/IGNITE-5553
> Project: Ignite
>  Issue Type: Bug
>  Components: data structures, persistence
>Affects Versions: 2.1
>Reporter: Dmitriy Pavlov
>Assignee: Pavel Pereslegin
>Priority: Major
>  Labels: MakeTeamcityGreenAgain, Muted_test, test-fail
> Fix For: 2.7
>
>
> h2. Notes-4435
> When IgniteSet is restored from persistence, size of set is always 0, [link 
> to test 
> history|http://ci.ignite.apache.org/project.html?projectId=Ignite20Tests=-7043871603266099589=testDetails].
> h2. Detailed description
> Unlike *IgniteQueue* which uses separate cache key to store its size 
> *IgniteSet* stores it in a field of some class.
> Test from the link above shows very clearly that after restoring memory state 
> from PDS all set values are restored correctly but size is lost.
> h2. Proposed solution
> One possible solution might be to do the same thing as *IgniteQueue* does: 
> size of *IgniteSet* must be stored is cache instead of volatile in-memory 
> fields of random classes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-5553) Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error

[Anton Vinogradov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462525#comment-16462525
 ] 

Anton Vinogradov commented on IGNITE-5553:
--

[~ilantukh], [~xtern]
It looks like main problem is that 
{{org.apache.ignite.internal.processors.cache.datastructures.CacheDataStructuresManager#onEntryUpdated}}
 is not called on memory restore, so 
{{org.apache.ignite.internal.processors.cache.datastructures.CacheDataStructuresManager#setDataMap}}
 is not filled on restart.

We have to call 
{{org.apache.ignite.internal.processors.cache.datastructures.CacheDataStructuresManager#onEntryUpdated}}
 on memory restore finished like we call it on each cacheEntry update, I think.
Thoughts?

> Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error
> -
>
> Key: IGNITE-5553
> URL: https://issues.apache.org/jira/browse/IGNITE-5553
> Project: Ignite
>  Issue Type: Bug
>  Components: data structures, persistence
>Affects Versions: 2.1
>Reporter: Dmitriy Pavlov
>Assignee: Pavel Pereslegin
>Priority: Major
>  Labels: MakeTeamcityGreenAgain, Muted_test, test-fail
> Fix For: 2.6
>
>
> h2. Notes-4435
> When IgniteSet is restored from persistence, size of set is always 0, [link 
> to test 
> history|http://ci.ignite.apache.org/project.html?projectId=Ignite20Tests=-7043871603266099589=testDetails].
> h2. Detailed description
> Unlike *IgniteQueue* which uses separate cache key to store its size 
> *IgniteSet* stores it in a field of some class.
> Test from the link above shows very clearly that after restoring memory state 
> from PDS all set values are restored correctly but size is lost.
> h2. Proposed solution
> One possible solution might be to do the same thing as *IgniteQueue* does: 
> size of *IgniteSet* must be stored is cache instead of volatile in-memory 
> fields of random classes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-5553) Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error

[Ilya Lantukh (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16444049#comment-16444049
 ] 

Ilya Lantukh commented on IGNITE-5553:
--

[~xtern], sorry for late reply.
New solution also have problems (explained in Upsource). We need to think about 
other possible approaches. I'll think for a while and summarize my thoughts on 
dev list. Feel free to share your ideas as well.

> Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error
> -
>
> Key: IGNITE-5553
> URL: https://issues.apache.org/jira/browse/IGNITE-5553
> Project: Ignite
>  Issue Type: Bug
>  Components: data structures, persistence
>Affects Versions: 2.1
>Reporter: Dmitriy Pavlov
>Assignee: Pavel Pereslegin
>Priority: Major
>  Labels: MakeTeamcityGreenAgain, Muted_test, test-fail
> Fix For: 2.6
>
>
> h2. Notes-4435
> When IgniteSet is restored from persistence, size of set is always 0, [link 
> to test 
> history|http://ci.ignite.apache.org/project.html?projectId=Ignite20Tests=-7043871603266099589=testDetails].
> h2. Detailed description
> Unlike *IgniteQueue* which uses separate cache key to store its size 
> *IgniteSet* stores it in a field of some class.
> Test from the link above shows very clearly that after restoring memory state 
> from PDS all set values are restored correctly but size is lost.
> h2. Proposed solution
> One possible solution might be to do the same thing as *IgniteQueue* does: 
> size of *IgniteSet* must be stored is cache instead of volatile in-memory 
> fields of random classes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-5553) Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error

[Pavel Pereslegin (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426821#comment-16426821
 ] 

Pavel Pereslegin commented on IGNITE-5553:
--

Hello [~ilantukh],
I updated this patch and test results, could you look again, please?

> Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error
> -
>
> Key: IGNITE-5553
> URL: https://issues.apache.org/jira/browse/IGNITE-5553
> Project: Ignite
>  Issue Type: Bug
>  Components: data structures, persistence
>Affects Versions: 2.1
>Reporter: Dmitriy Pavlov
>Assignee: Pavel Pereslegin
>Priority: Major
>  Labels: MakeTeamcityGreenAgain, Muted_test, test-fail
> Fix For: 2.5
>
>
> h2. Notes-4435
> When IgniteSet is restored from persistence, size of set is always 0, [link 
> to test 
> history|http://ci.ignite.apache.org/project.html?projectId=Ignite20Tests=-7043871603266099589=testDetails].
> h2. Detailed description
> Unlike *IgniteQueue* which uses separate cache key to store its size 
> *IgniteSet* stores it in a field of some class.
> Test from the link above shows very clearly that after restoring memory state 
> from PDS all set values are restored correctly but size is lost.
> h2. Proposed solution
> One possible solution might be to do the same thing as *IgniteQueue* does: 
> size of *IgniteSet* must be stored is cache instead of volatile in-memory 
> fields of random classes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-5553) Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error

[Dmitriy Pavlov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411433#comment-16411433
 ] 

Dmitriy Pavlov commented on IGNITE-5553:


Hi [~ilantukh], 

I've checked changes but stil need advice if it safe to initialize set this way
https://reviews.ignite.apache.org/ignite/review/IGNT-CR-508

Could you please take a look to CacheDataStructuresManager and its 
initialization stages.

> Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error
> -
>
> Key: IGNITE-5553
> URL: https://issues.apache.org/jira/browse/IGNITE-5553
> Project: Ignite
>  Issue Type: Bug
>  Components: data structures, persistence
>Affects Versions: 2.1
>Reporter: Dmitriy Pavlov
>Assignee: Pavel Pereslegin
>Priority: Major
>  Labels: MakeTeamcityGreenAgain, Muted_test, test-fail
> Fix For: 2.5
>
>
> h2. Notes-4435
> When IgniteSet is restored from persistence, size of set is always 0, [link 
> to test 
> history|http://ci.ignite.apache.org/project.html?projectId=Ignite20Tests=-7043871603266099589=testDetails].
> h2. Detailed description
> Unlike *IgniteQueue* which uses separate cache key to store its size 
> *IgniteSet* stores it in a field of some class.
> Test from the link above shows very clearly that after restoring memory state 
> from PDS all set values are restored correctly but size is lost.
> h2. Proposed solution
> One possible solution might be to do the same thing as *IgniteQueue* does: 
> size of *IgniteSet* must be stored is cache instead of volatile in-memory 
> fields of random classes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-5553) Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error

[Pavel Pereslegin (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408110#comment-16408110
 ] 

Pavel Pereslegin commented on IGNITE-5553:
--

Hello [~dpavlov],

thank you and Andrey K for the review.
TeamCity looks overloaded nowadays, so the tests have been run in several 
attempts, but cumulative results are good.

> Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error
> -
>
> Key: IGNITE-5553
> URL: https://issues.apache.org/jira/browse/IGNITE-5553
> Project: Ignite
>  Issue Type: Bug
>  Components: data structures, persistence
>Affects Versions: 2.1
>Reporter: Dmitriy Pavlov
>Assignee: Pavel Pereslegin
>Priority: Major
>  Labels: MakeTeamcityGreenAgain, Muted_test, test-fail
> Fix For: 2.5
>
>
> h2. Notes-4435
> When IgniteSet is restored from persistence, size of set is always 0, [link 
> to test 
> history|http://ci.ignite.apache.org/project.html?projectId=Ignite20Tests=-7043871603266099589=testDetails].
> h2. Detailed description
> Unlike *IgniteQueue* which uses separate cache key to store its size 
> *IgniteSet* stores it in a field of some class.
> Test from the link above shows very clearly that after restoring memory state 
> from PDS all set values are restored correctly but size is lost.
> h2. Proposed solution
> One possible solution might be to do the same thing as *IgniteQueue* does: 
> size of *IgniteSet* must be stored is cache instead of volatile in-memory 
> fields of random classes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-5553) Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error

[Andrey Kuznetsov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406427#comment-16406427
 ] 

Andrey Kuznetsov commented on IGNITE-5553:
--

[~dpavlov], we discussed the PR with [~xtern] and replaced two init flags with 
single state field (that is, single mean of synchronization). Now I like the 
changes. 

> Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error
> -
>
> Key: IGNITE-5553
> URL: https://issues.apache.org/jira/browse/IGNITE-5553
> Project: Ignite
>  Issue Type: Bug
>  Components: data structures, persistence
>Affects Versions: 2.1
>Reporter: Dmitriy Pavlov
>Assignee: Pavel Pereslegin
>Priority: Major
>  Labels: MakeTeamcityGreenAgain, Muted_test, test-fail
> Fix For: 2.5
>
>
> h2. Notes-4435
> When IgniteSet is restored from persistence, size of set is always 0, [link 
> to test 
> history|http://ci.ignite.apache.org/project.html?projectId=Ignite20Tests=-7043871603266099589=testDetails].
> h2. Detailed description
> Unlike *IgniteQueue* which uses separate cache key to store its size 
> *IgniteSet* stores it in a field of some class.
> Test from the link above shows very clearly that after restoring memory state 
> from PDS all set values are restored correctly but size is lost.
> h2. Proposed solution
> One possible solution might be to do the same thing as *IgniteQueue* does: 
> size of *IgniteSet* must be stored is cache instead of volatile in-memory 
> fields of random classes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-5553) Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error

[Dmitriy Pavlov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405109#comment-16405109
 ] 

Dmitriy Pavlov commented on IGNITE-5553:


Can I ask [~andrey-kuznetsov] to look to this change first?

TC seems ok, I left several comments in 
https://reviews.ignite.apache.org/ignite/review/IGNT-CR-508 , mostly confused 
by boolean init flags there.

> Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error
> -
>
> Key: IGNITE-5553
> URL: https://issues.apache.org/jira/browse/IGNITE-5553
> Project: Ignite
>  Issue Type: Bug
>  Components: data structures, persistence
>Affects Versions: 2.1
>Reporter: Dmitriy Pavlov
>Assignee: Pavel Pereslegin
>Priority: Major
>  Labels: MakeTeamcityGreenAgain, Muted_test, test-fail
> Fix For: 2.5
>
>
> h2. Notes-4435
> When IgniteSet is restored from persistence, size of set is always 0, [link 
> to test 
> history|http://ci.ignite.apache.org/project.html?projectId=Ignite20Tests=-7043871603266099589=testDetails].
> h2. Detailed description
> Unlike *IgniteQueue* which uses separate cache key to store its size 
> *IgniteSet* stores it in a field of some class.
> Test from the link above shows very clearly that after restoring memory state 
> from PDS all set values are restored correctly but size is lost.
> h2. Proposed solution
> One possible solution might be to do the same thing as *IgniteQueue* does: 
> size of *IgniteSet* must be stored is cache instead of volatile in-memory 
> fields of random classes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-5553) Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error

[Pavel Pereslegin (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16404782#comment-16404782
 ] 

Pavel Pereslegin commented on IGNITE-5553:
--

Hello [~dpavlov],

could you suggest someone who can review this fix?

> Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error
> -
>
> Key: IGNITE-5553
> URL: https://issues.apache.org/jira/browse/IGNITE-5553
> Project: Ignite
>  Issue Type: Bug
>  Components: data structures, persistence
>Affects Versions: 2.1
>Reporter: Dmitriy Pavlov
>Assignee: Pavel Pereslegin
>Priority: Major
>  Labels: MakeTeamcityGreenAgain, Muted_test, test-fail
> Fix For: 2.5
>
>
> h2. Notes-4435
> When IgniteSet is restored from persistence, size of set is always 0, [link 
> to test 
> history|http://ci.ignite.apache.org/project.html?projectId=Ignite20Tests=-7043871603266099589=testDetails].
> h2. Detailed description
> Unlike *IgniteQueue* which uses separate cache key to store its size 
> *IgniteSet* stores it in a field of some class.
> Test from the link above shows very clearly that after restoring memory state 
> from PDS all set values are restored correctly but size is lost.
> h2. Proposed solution
> One possible solution might be to do the same thing as *IgniteQueue* does: 
> size of *IgniteSet* must be stored is cache instead of volatile in-memory 
> fields of random classes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-5553) Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error

[Dmitriy Pavlov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382232#comment-16382232
 ] 

Dmitriy Pavlov commented on IGNITE-5553:


TC looks good, thank you. 

> Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error
> -
>
> Key: IGNITE-5553
> URL: https://issues.apache.org/jira/browse/IGNITE-5553
> Project: Ignite
>  Issue Type: Bug
>  Components: data structures, persistence
>Affects Versions: 2.1
>Reporter: Dmitriy Pavlov
>Assignee: Pavel Pereslegin
>Priority: Major
>  Labels: MakeTeamcityGreenAgain, Muted_test, test-fail
> Fix For: 2.5
>
>
> h2. Notes-4435
> When IgniteSet is restored from persistence, size of set is always 0, [link 
> to test 
> history|http://ci.ignite.apache.org/project.html?projectId=Ignite20Tests=-7043871603266099589=testDetails].
> h2. Detailed description
> Unlike *IgniteQueue* which uses separate cache key to store its size 
> *IgniteSet* stores it in a field of some class.
> Test from the link above shows very clearly that after restoring memory state 
> from PDS all set values are restored correctly but size is lost.
> h2. Proposed solution
> One possible solution might be to do the same thing as *IgniteQueue* does: 
> size of *IgniteSet* must be stored is cache instead of volatile in-memory 
> fields of random classes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-5553) Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error

[Pavel Pereslegin (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382183#comment-16382183
 ] 

Pavel Pereslegin commented on IGNITE-5553:
--

[~dpavlov], I updated TC test results link.

> Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error
> -
>
> Key: IGNITE-5553
> URL: https://issues.apache.org/jira/browse/IGNITE-5553
> Project: Ignite
>  Issue Type: Bug
>  Components: data structures, persistence
>Affects Versions: 2.1
>Reporter: Dmitriy Pavlov
>Assignee: Pavel Pereslegin
>Priority: Major
>  Labels: MakeTeamcityGreenAgain, Muted_test, test-fail
> Fix For: 2.5
>
>
> h2. Notes-4435
> When IgniteSet is restored from persistence, size of set is always 0, [link 
> to test 
> history|http://ci.ignite.apache.org/project.html?projectId=Ignite20Tests=-7043871603266099589=testDetails].
> h2. Detailed description
> Unlike *IgniteQueue* which uses separate cache key to store its size 
> *IgniteSet* stores it in a field of some class.
> Test from the link above shows very clearly that after restoring memory state 
> from PDS all set values are restored correctly but size is lost.
> h2. Proposed solution
> One possible solution might be to do the same thing as *IgniteQueue* does: 
> size of *IgniteSet* must be stored is cache instead of volatile in-memory 
> fields of random classes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-5553) Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error

[Dmitriy Pavlov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382143#comment-16382143
 ] 

Dmitriy Pavlov commented on IGNITE-5553:


Hi [~xtern], could you provide link to TC run all?

> Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error
> -
>
> Key: IGNITE-5553
> URL: https://issues.apache.org/jira/browse/IGNITE-5553
> Project: Ignite
>  Issue Type: Bug
>  Components: data structures, persistence
>Affects Versions: 2.1
>Reporter: Dmitriy Pavlov
>Assignee: Pavel Pereslegin
>Priority: Major
>  Labels: MakeTeamcityGreenAgain, Muted_test, test-fail
> Fix For: 2.5
>
>
> h2. Notes-4435
> When IgniteSet is restored from persistence, size of set is always 0, [link 
> to test 
> history|http://ci.ignite.apache.org/project.html?projectId=Ignite20Tests=-7043871603266099589=testDetails].
> h2. Detailed description
> Unlike *IgniteQueue* which uses separate cache key to store its size 
> *IgniteSet* stores it in a field of some class.
> Test from the link above shows very clearly that after restoring memory state 
> from PDS all set values are restored correctly but size is lost.
> h2. Proposed solution
> One possible solution might be to do the same thing as *IgniteQueue* does: 
> size of *IgniteSet* must be stored is cache instead of volatile in-memory 
> fields of random classes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-5553) Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16380325#comment-16380325
 ] 

ASF GitHub Bot commented on IGNITE-5553:


GitHub user xtern opened a pull request:

https://github.com/apache/ignite/pull/3587

IGNITE-5553 Restore IgniteSet local data when PDS enabled.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xtern/ignite IGNITE-5553

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3587.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3587


commit 8e01cf250d4f4cd06d2b3c44612b439410745219
Author: pereslegin-pa 
Date:   2018-02-28T09:27:57Z

IGNITE-5553 PDS restore local set data.




> Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error
> -
>
> Key: IGNITE-5553
> URL: https://issues.apache.org/jira/browse/IGNITE-5553
> Project: Ignite
>  Issue Type: Bug
>  Components: data structures, persistence
>Affects Versions: 2.1
>Reporter: Dmitriy Pavlov
>Assignee: Pavel Pereslegin
>Priority: Major
>  Labels: MakeTeamcityGreenAgain, Muted_test, test-fail
> Fix For: 2.5
>
>
> h2. Notes-4435
> When IgniteSet is restored from persistence, size of set is always 0, [link 
> to test 
> history|http://ci.ignite.apache.org/project.html?projectId=Ignite20Tests=-7043871603266099589=testDetails].
> h2. Detailed description
> Unlike *IgniteQueue* which uses separate cache key to store its size 
> *IgniteSet* stores it in a field of some class.
> Test from the link above shows very clearly that after restoring memory state 
> from PDS all set values are restored correctly but size is lost.
> h2. Proposed solution
> One possible solution might be to do the same thing as *IgniteQueue* does: 
> size of *IgniteSet* must be stored is cache instead of volatile in-memory 
> fields of random classes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-5553) Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error

[Pavel Pereslegin (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364043#comment-16364043
 ] 

Pavel Pereslegin commented on IGNITE-5553:
--

Hello.

In addition to [discussion on 
devlist|http://apache-ignite-developers.2346864.n4.nabble.com/IgniteSet-implementation-changes-required-td23783.html],
 I want to describe in more detail another way to implement IgniteSet, which 
will not use JVM heap for duplicate values (this approach was not discussed 
yet).

Introduce GridCacheSetIndexKey as follows:
{code:java}
IgniteUuid id;
long idx;
{code}

For each IgniteSet element put two key-value pairs.
First:
{noformat}
   key - GridCacheSetItemKey
 value - GridCacheSetIndexKey
{noformat}
Second:
{noformat}
   key - GridCacheSetIndexKey
 value - GridCacheSetItemKey
{noformat}

Add HEAD and TAIL indexes to the GridCacheSetHeader.

The main purpose for indexes is to hold the data in such way that will allow 
iterate through items from head to tail (similarly to IgniteQueue).

For clarity, I'll give an example of *IgniteSet with elements A, B, C*.
Datastructures cache
||key||value||
|A|0|
|B|1|
|C|2|
|.|.|
|0|A|
|1|B|
|2|C|
|.|.|
{noformat}
head = 0, tail = 3
{noformat}

*Addition of the element "D"*:
||key||value||
|A|0|
|B|1|
|C|2|
|D|3|
|.|.|
|0|A|
|1|B|
|2|C|
|3|D|
|.|.|
{noformat}
 head = 0, tail = 4
{noformat}
 
*To remove element "B"* we can move the tail element ("D") to the location of 
"B":
||key||value||
|A|0|
|C|2|
|D|1|
|.|.|
|0|A|
|1|D|
|2|C|
|-- tail --|
|3|D|
|.|.|
{noformat}
head = 0, tail = 3.
{noformat}

Operation "contains" will work as before, iterator will work as current 
IgniteQueue iterator.

> Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error
> -
>
> Key: IGNITE-5553
> URL: https://issues.apache.org/jira/browse/IGNITE-5553
> Project: Ignite
>  Issue Type: Bug
>  Components: data structures, persistence
>Affects Versions: 2.1
>Reporter: Dmitriy Pavlov
>Assignee: Andrey Kuznetsov
>Priority: Major
>  Labels: MakeTeamcityGreenAgain, Muted_test, test-fail
> Fix For: 2.5
>
>
> h2. Notes-4435
> When IgniteSet is restored from persistence, size of set is always 0, [link 
> to test 
> history|http://ci.ignite.apache.org/project.html?projectId=Ignite20Tests=-7043871603266099589=testDetails].
> h2. Detailed description
> Unlike *IgniteQueue* which uses separate cache key to store its size 
> *IgniteSet* stores it in a field of some class.
> Test from the link above shows very clearly that after restoring memory state 
> from PDS all set values are restored correctly but size is lost.
> h2. Proposed solution
> One possible solution might be to do the same thing as *IgniteQueue* does: 
> size of *IgniteSet* must be stored is cache instead of volatile in-memory 
> fields of random classes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (IGNITE-5553) Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error

[Sergey Chugunov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16224481#comment-16224481
 ] 

Sergey Chugunov commented on IGNITE-5553:
-

[~andrey-kuznetsov],
Options you've proposed don't look like a simple fix and may impact other 
aspects of data structures like performance or memory consumption.
So it is better to discuss them on dev-list to keep community aware of the 
changes.

> Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error
> -
>
> Key: IGNITE-5553
> URL: https://issues.apache.org/jira/browse/IGNITE-5553
> Project: Ignite
>  Issue Type: Bug
>  Components: data structures, persistence
>Affects Versions: 2.1
>Reporter: Dmitriy Pavlov
>Assignee: Andrey Kuznetsov
>Priority: Critical
>  Labels: MakeTeamcityGreenAgain, Muted_test, test-fail
>
> h2. Notes-4435
> When IgniteSet is restored from persistence, size of set is always 0, [link 
> to test 
> history|http://ci.ignite.apache.org/project.html?projectId=Ignite20Tests=-7043871603266099589=testDetails].
> h2. Detailed description
> Unlike *IgniteQueue* which uses separate cache key to store its size 
> *IgniteSet* stores it in a field of some class.
> Test from the link above shows very clearly that after restoring memory state 
> from PDS all set values are restored correctly but size is lost.
> h2. Proposed solution
> One possible solution might be to do the same thing as *IgniteQueue* does: 
> size of *IgniteSet* must be stored is cache instead of volatile in-memory 
> fields of random classes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (IGNITE-5553) Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error

[Andrey Kuznetsov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-5553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16216890#comment-16216890
 ] 

Andrey Kuznetsov commented on IGNITE-5553:
--

Indeed, we can put current set size into set header cache entry. This will fix 
size(), but we have broken iterator() implementation as well.

Currently, set implementation maintains plain Java sets on every node, see 
CacheDataStructuresManager.setDataMap. These sets duplicate backing-cache 
entries, both primary and backup. size() and iterator() calls issue distributed 
queries to collect/filter data from all setDataMap's. And setDataMaps remain 
empty after cluster is recovered from checkpoint.

Now I see the following options to fix the issue.

#1 - Naive. Iterate over all datastructure-backing caches entries during 
recover from checkpoint procedure, filter set-related entries and refill 
setDataMap's.
Pros: easy to implement
Cons: inpredictable time/memory overhead.

#2 - More realistic. Avoid node-local copies of cache data. Maintain linked 
list in datastructure-backing cache: key is set item, value is next set item. 
List head is stored in set header cache entry (this set item is youngest one). 
Iterators build on top of this structure are fail-fast.
Pros: less memory overhead, no need to maintain node-local mirrors of cache data
Cons: iterators are not fail-safe.

#3 - Option #2 modified. We can store reference counter and 'removed' flag 
along with next item reference. This allows to make iterators fail safe.
Pros: iterators are fail-safe
Cons: slightly more complicated implementation, may affect performance, also I 
see no way to handle active iterators on remote nodes failures.

[~sergey-chugunov], any thoughts? Or maybe we should discuss this on devlist?

> Ignite PDS 2: IgnitePersistentStoreDataStructuresTest testSet assertion error
> -
>
> Key: IGNITE-5553
> URL: https://issues.apache.org/jira/browse/IGNITE-5553
> Project: Ignite
>  Issue Type: Bug
>  Components: data structures, persistence
>Affects Versions: 2.1
>Reporter: Dmitriy Pavlov
>Assignee: Andrey Kuznetsov
>Priority: Critical
>  Labels: MakeTeamcityGreenAgain, Muted_test, test-fail
>
> h2. Notes-4435
> When IgniteSet is restored from persistence, size of set is always 0, [link 
> to test 
> history|http://ci.ignite.apache.org/project.html?projectId=Ignite20Tests=-7043871603266099589=testDetails].
> h2. Detailed description
> Unlike *IgniteQueue* which uses separate cache key to store its size 
> *IgniteSet* stores it in a field of some class.
> Test from the link above shows very clearly that after restoring memory state 
> from PDS all set values are restored correctly but size is lost.
> h2. Proposed solution
> One possible solution might be to do the same thing as *IgniteQueue* does: 
> size of *IgniteSet* must be stored is cache instead of volatile in-memory 
> fields of random classes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)