[jira] [Commented] (IGNITE-4157) Use discovery custom messages instead of marshaller cache

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15874409#comment-15874409
 ] 

ASF GitHub Bot commented on IGNITE-4157:


Github user sergey-chugunov-1985 closed the pull request at:

https://github.com/apache/ignite/pull/1447


> Use discovery custom messages instead of marshaller cache
> -
>
> Key: IGNITE-4157
> URL: https://issues.apache.org/jira/browse/IGNITE-4157
> Project: Ignite
>  Issue Type: Improvement
>  Components: cache
>Reporter: Alexey Goncharuk
>Assignee: Sergey Chugunov
> Fix For: 2.0
>
>
> Currently we use system caches for keeping classname to class ID mapping and 
> for storing binary metadata
> This has several serious disadvantages:
> 1) We need to introduce at least two additional thread pools for each of 
> these caches
> 2) Since cache operations require stable topology, registering a class ID or 
> updating metadata inside a transaction or another cache operation is tricky 
> and deadlock-prone.
> 3) It may be beneficial in some cases to have nodes with no caches at all, 
> currently this is impossible because system caches are always present.
> 4) Reading binary metadata leads to huge local contention, caching metadata 
> values in a local map doubles memory consumption
> I suggest we use discovery custom events for these purposes. Each node will 
> have a corresponding local map (state) which will be updated inside custom 
> event handler. From the first point of view, this should remove all the 
> disadvantages above.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-4157) Use discovery custom messages instead of marshaller cache

[Sergey Chugunov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860931#comment-15860931
 ] 

Sergey Chugunov commented on IGNITE-4157:
-

Replaced direct usage of system pool in *MarshallerContextImpl* with 
*GridClosureProcessor* which is a conventional way to submit Runnables into 
platform.

> Use discovery custom messages instead of marshaller cache
> -
>
> Key: IGNITE-4157
> URL: https://issues.apache.org/jira/browse/IGNITE-4157
> Project: Ignite
>  Issue Type: Improvement
>  Components: cache
>Reporter: Alexey Goncharuk
>Assignee: Sergey Chugunov
> Fix For: 2.0
>
>
> Currently we use system caches for keeping classname to class ID mapping and 
> for storing binary metadata
> This has several serious disadvantages:
> 1) We need to introduce at least two additional thread pools for each of 
> these caches
> 2) Since cache operations require stable topology, registering a class ID or 
> updating metadata inside a transaction or another cache operation is tricky 
> and deadlock-prone.
> 3) It may be beneficial in some cases to have nodes with no caches at all, 
> currently this is impossible because system caches are always present.
> 4) Reading binary metadata leads to huge local contention, caching metadata 
> values in a local map doubles memory consumption
> I suggest we use discovery custom events for these purposes. Each node will 
> have a corresponding local map (state) which will be updated inside custom 
> event handler. From the first point of view, this should remove all the 
> disadvantages above.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (IGNITE-4157) Use discovery custom messages instead of marshaller cache

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15839735#comment-15839735
 ] 

ASF GitHub Bot commented on IGNITE-4157:


Github user sergey-chugunov-1985 closed the pull request at:

https://github.com/apache/ignite/pull/1463


> Use discovery custom messages instead of marshaller cache
> -
>
> Key: IGNITE-4157
> URL: https://issues.apache.org/jira/browse/IGNITE-4157
> Project: Ignite
>  Issue Type: Improvement
>  Components: cache
>Reporter: Alexey Goncharuk
>Assignee: Sergey Chugunov
> Fix For: 2.0
>
>
> Currently we use system caches for keeping classname to class ID mapping and 
> for storing binary metadata
> This has several serious disadvantages:
> 1) We need to introduce at least two additional thread pools for each of 
> these caches
> 2) Since cache operations require stable topology, registering a class ID or 
> updating metadata inside a transaction or another cache operation is tricky 
> and deadlock-prone.
> 3) It may be beneficial in some cases to have nodes with no caches at all, 
> currently this is impossible because system caches are always present.
> 4) Reading binary metadata leads to huge local contention, caching metadata 
> values in a local map doubles memory consumption
> I suggest we use discovery custom events for these purposes. Each node will 
> have a corresponding local map (state) which will be updated inside custom 
> event handler. From the first point of view, this should remove all the 
> disadvantages above.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4157) Use discovery custom messages instead of marshaller cache

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15839420#comment-15839420
 ] 

ASF GitHub Bot commented on IGNITE-4157:


GitHub user sergey-chugunov-1985 opened a pull request:

https://github.com/apache/ignite/pull/1463

IGNITE-4157 DO NOT MERGE

This pull request was created to run tests on public TC only, it is not 
intended to be merged.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite 
ignite-4157-disco_msg_loss

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1463.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1463


commit 8eaf184702f88cbb389d2d7ac6be81f7e9ccb191
Author: Sergey Chugunov 
Date:   2017-01-26T08:32:36Z

IGNITE-4157 check discovery messages loss case




> Use discovery custom messages instead of marshaller cache
> -
>
> Key: IGNITE-4157
> URL: https://issues.apache.org/jira/browse/IGNITE-4157
> Project: Ignite
>  Issue Type: Improvement
>  Components: cache
>Reporter: Alexey Goncharuk
>Assignee: Sergey Chugunov
> Fix For: 2.0
>
>
> Currently we use system caches for keeping classname to class ID mapping and 
> for storing binary metadata
> This has several serious disadvantages:
> 1) We need to introduce at least two additional thread pools for each of 
> these caches
> 2) Since cache operations require stable topology, registering a class ID or 
> updating metadata inside a transaction or another cache operation is tricky 
> and deadlock-prone.
> 3) It may be beneficial in some cases to have nodes with no caches at all, 
> currently this is impossible because system caches are always present.
> 4) Reading binary metadata leads to huge local contention, caching metadata 
> values in a local map doubles memory consumption
> I suggest we use discovery custom events for these purposes. Each node will 
> have a corresponding local map (state) which will be updated inside custom 
> event handler. From the first point of view, this should remove all the 
> disadvantages above.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4157) Use discovery custom messages instead of marshaller cache

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831847#comment-15831847
 ] 

ASF GitHub Bot commented on IGNITE-4157:


GitHub user sergey-chugunov-1985 opened a pull request:

https://github.com/apache/ignite/pull/1447

IGNITE-4157 mapping updates notification hook



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-4157

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1447.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1447






> Use discovery custom messages instead of marshaller cache
> -
>
> Key: IGNITE-4157
> URL: https://issues.apache.org/jira/browse/IGNITE-4157
> Project: Ignite
>  Issue Type: Improvement
>  Components: cache
>Reporter: Alexey Goncharuk
>Assignee: Sergey Chugunov
> Fix For: 2.0
>
>
> Currently we use system caches for keeping classname to class ID mapping and 
> for storing binary metadata
> This has several serious disadvantages:
> 1) We need to introduce at least two additional thread pools for each of 
> these caches
> 2) Since cache operations require stable topology, registering a class ID or 
> updating metadata inside a transaction or another cache operation is tricky 
> and deadlock-prone.
> 3) It may be beneficial in some cases to have nodes with no caches at all, 
> currently this is impossible because system caches are always present.
> 4) Reading binary metadata leads to huge local contention, caching metadata 
> values in a local map doubles memory consumption
> I suggest we use discovery custom events for these purposes. Each node will 
> have a corresponding local map (state) which will be updated inside custom 
> event handler. From the first point of view, this should remove all the 
> disadvantages above.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4157) Use discovery custom messages instead of marshaller cache

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831712#comment-15831712
 ] 

ASF GitHub Bot commented on IGNITE-4157:


Github user sergey-chugunov-1985 closed the pull request at:

https://github.com/apache/ignite/pull/1271


> Use discovery custom messages instead of marshaller cache
> -
>
> Key: IGNITE-4157
> URL: https://issues.apache.org/jira/browse/IGNITE-4157
> Project: Ignite
>  Issue Type: Improvement
>  Components: cache
>Reporter: Alexey Goncharuk
>Assignee: Sergey Chugunov
> Fix For: 2.0
>
>
> Currently we use system caches for keeping classname to class ID mapping and 
> for storing binary metadata
> This has several serious disadvantages:
> 1) We need to introduce at least two additional thread pools for each of 
> these caches
> 2) Since cache operations require stable topology, registering a class ID or 
> updating metadata inside a transaction or another cache operation is tricky 
> and deadlock-prone.
> 3) It may be beneficial in some cases to have nodes with no caches at all, 
> currently this is impossible because system caches are always present.
> 4) Reading binary metadata leads to huge local contention, caching metadata 
> values in a local map doubles memory consumption
> I suggest we use discovery custom events for these purposes. Each node will 
> have a corresponding local map (state) which will be updated inside custom 
> event handler. From the first point of view, this should remove all the 
> disadvantages above.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4157) Use discovery custom messages instead of marshaller cache

[Sergey Chugunov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824126#comment-15824126
 ] 

Sergey Chugunov commented on IGNITE-4157:
-

[~ptupitsyn], [~vozerov] found numerous code style violations, I've already 
addressed them. Also I addressed issue I found recently. It is related to 
unnecessary service reassignments. 

Build is already scheduled in TC, I don't expect tests to fail because of code 
style fixes.

Tomorrow I expect having this PR merged.

> Use discovery custom messages instead of marshaller cache
> -
>
> Key: IGNITE-4157
> URL: https://issues.apache.org/jira/browse/IGNITE-4157
> Project: Ignite
>  Issue Type: Improvement
>  Components: cache
>Reporter: Alexey Goncharuk
>Assignee: Sergey Chugunov
> Fix For: 2.0
>
>
> Currently we use system caches for keeping classname to class ID mapping and 
> for storing binary metadata
> This has several serious disadvantages:
> 1) We need to introduce at least two additional thread pools for each of 
> these caches
> 2) Since cache operations require stable topology, registering a class ID or 
> updating metadata inside a transaction or another cache operation is tricky 
> and deadlock-prone.
> 3) It may be beneficial in some cases to have nodes with no caches at all, 
> currently this is impossible because system caches are always present.
> 4) Reading binary metadata leads to huge local contention, caching metadata 
> values in a local map doubles memory consumption
> I suggest we use discovery custom events for these purposes. Each node will 
> have a corresponding local map (state) which will be updated inside custom 
> event handler. From the first point of view, this should remove all the 
> disadvantages above.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4157) Use discovery custom messages instead of marshaller cache

[Pavel Tupitsyn (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824067#comment-15824067
 ] 

Pavel Tupitsyn commented on IGNITE-4157:


Hey guys, any update on this ticket? Who is the reviewer?

> Use discovery custom messages instead of marshaller cache
> -
>
> Key: IGNITE-4157
> URL: https://issues.apache.org/jira/browse/IGNITE-4157
> Project: Ignite
>  Issue Type: Improvement
>  Components: cache
>Reporter: Alexey Goncharuk
>Assignee: Sergey Chugunov
> Fix For: 2.0
>
>
> Currently we use system caches for keeping classname to class ID mapping and 
> for storing binary metadata
> This has several serious disadvantages:
> 1) We need to introduce at least two additional thread pools for each of 
> these caches
> 2) Since cache operations require stable topology, registering a class ID or 
> updating metadata inside a transaction or another cache operation is tricky 
> and deadlock-prone.
> 3) It may be beneficial in some cases to have nodes with no caches at all, 
> currently this is impossible because system caches are always present.
> 4) Reading binary metadata leads to huge local contention, caching metadata 
> values in a local map doubles memory consumption
> I suggest we use discovery custom events for these purposes. Each node will 
> have a corresponding local map (state) which will be updated inside custom 
> event handler. From the first point of view, this should remove all the 
> disadvantages above.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4157) Use discovery custom messages instead of marshaller cache

[Sergey Chugunov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15783251#comment-15783251
 ] 

Sergey Chugunov commented on IGNITE-4157:
-

Reviewed code before reopening pull request, made several improvements and 
added more documentation comments.

> Use discovery custom messages instead of marshaller cache
> -
>
> Key: IGNITE-4157
> URL: https://issues.apache.org/jira/browse/IGNITE-4157
> Project: Ignite
>  Issue Type: Improvement
>  Components: cache
>Reporter: Alexey Goncharuk
>Assignee: Sergey Chugunov
> Fix For: 2.0
>
>
> Currently we use system caches for keeping classname to class ID mapping and 
> for storing binary metadata
> This has several serious disadvantages:
> 1) We need to introduce at least two additional thread pools for each of 
> these caches
> 2) Since cache operations require stable topology, registering a class ID or 
> updating metadata inside a transaction or another cache operation is tricky 
> and deadlock-prone.
> 3) It may be beneficial in some cases to have nodes with no caches at all, 
> currently this is impossible because system caches are always present.
> 4) Reading binary metadata leads to huge local contention, caching metadata 
> values in a local map doubles memory consumption
> I suggest we use discovery custom events for these purposes. Each node will 
> have a corresponding local map (state) which will be updated inside custom 
> event handler. From the first point of view, this should remove all the 
> disadvantages above.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4157) Use discovery custom messages instead of marshaller cache

[Sergey Chugunov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15780669#comment-15780669
 ] 

Sergey Chugunov commented on IGNITE-4157:
-

Finished with moving functionality to two distinct classes: *DiscoveryDataBag* 
from public API and *DiscoveryDataPacket* in internal implementation.
If all tests pass all comments to initial PR will be addressed.

> Use discovery custom messages instead of marshaller cache
> -
>
> Key: IGNITE-4157
> URL: https://issues.apache.org/jira/browse/IGNITE-4157
> Project: Ignite
>  Issue Type: Improvement
>  Components: cache
>Reporter: Alexey Goncharuk
>Assignee: Sergey Chugunov
> Fix For: 2.0
>
>
> Currently we use system caches for keeping classname to class ID mapping and 
> for storing binary metadata
> This has several serious disadvantages:
> 1) We need to introduce at least two additional thread pools for each of 
> these caches
> 2) Since cache operations require stable topology, registering a class ID or 
> updating metadata inside a transaction or another cache operation is tricky 
> and deadlock-prone.
> 3) It may be beneficial in some cases to have nodes with no caches at all, 
> currently this is impossible because system caches are always present.
> 4) Reading binary metadata leads to huge local contention, caching metadata 
> values in a local map doubles memory consumption
> I suggest we use discovery custom events for these purposes. Each node will 
> have a corresponding local map (state) which will be updated inside custom 
> event handler. From the first point of view, this should remove all the 
> disadvantages above.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4157) Use discovery custom messages instead of marshaller cache

[Sergey Chugunov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778838#comment-15778838
 ] 

Sergey Chugunov commented on IGNITE-4157:
-

Splitted *DiscoveryDataContainer* class into two parts: one for public API, 
another for internal implementation. Working on moving functionality.

> Use discovery custom messages instead of marshaller cache
> -
>
> Key: IGNITE-4157
> URL: https://issues.apache.org/jira/browse/IGNITE-4157
> Project: Ignite
>  Issue Type: Improvement
>  Components: cache
>Reporter: Alexey Goncharuk
>Assignee: Sergey Chugunov
> Fix For: 2.0
>
>
> Currently we use system caches for keeping classname to class ID mapping and 
> for storing binary metadata
> This has several serious disadvantages:
> 1) We need to introduce at least two additional thread pools for each of 
> these caches
> 2) Since cache operations require stable topology, registering a class ID or 
> updating metadata inside a transaction or another cache operation is tricky 
> and deadlock-prone.
> 3) It may be beneficial in some cases to have nodes with no caches at all, 
> currently this is impossible because system caches are always present.
> 4) Reading binary metadata leads to huge local contention, caching metadata 
> values in a local map doubles memory consumption
> I suggest we use discovery custom events for these purposes. Each node will 
> have a corresponding local map (state) which will be updated inside custom 
> event handler. From the first point of view, this should remove all the 
> disadvantages above.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4157) Use discovery custom messages instead of marshaller cache

[Sergey Chugunov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15773222#comment-15773222
 ] 

Sergey Chugunov commented on IGNITE-4157:
-

Optimized discovery data collection procedure (no filtering of system classes 
is applied now), changed collection used for holding cache from Map to List.

> Use discovery custom messages instead of marshaller cache
> -
>
> Key: IGNITE-4157
> URL: https://issues.apache.org/jira/browse/IGNITE-4157
> Project: Ignite
>  Issue Type: Improvement
>  Components: cache
>Reporter: Alexey Goncharuk
>Assignee: Sergey Chugunov
> Fix For: 2.0
>
>
> Currently we use system caches for keeping classname to class ID mapping and 
> for storing binary metadata
> This has several serious disadvantages:
> 1) We need to introduce at least two additional thread pools for each of 
> these caches
> 2) Since cache operations require stable topology, registering a class ID or 
> updating metadata inside a transaction or another cache operation is tricky 
> and deadlock-prone.
> 3) It may be beneficial in some cases to have nodes with no caches at all, 
> currently this is impossible because system caches are always present.
> 4) Reading binary metadata leads to huge local contention, caching metadata 
> values in a local map doubles memory consumption
> I suggest we use discovery custom events for these purposes. Each node will 
> have a corresponding local map (state) which will be updated inside custom 
> event handler. From the first point of view, this should remove all the 
> disadvantages above.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4157) Use discovery custom messages instead of marshaller cache

[Sergey Chugunov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15770271#comment-15770271
 ] 

Sergey Chugunov commented on IGNITE-4157:
-

Created test for server nodes' failures scenario for #4, fixed several minor 
issues along the way.

> Use discovery custom messages instead of marshaller cache
> -
>
> Key: IGNITE-4157
> URL: https://issues.apache.org/jira/browse/IGNITE-4157
> Project: Ignite
>  Issue Type: Improvement
>  Components: cache
>Reporter: Alexey Goncharuk
>Assignee: Sergey Chugunov
> Fix For: 2.0
>
>
> Currently we use system caches for keeping classname to class ID mapping and 
> for storing binary metadata
> This has several serious disadvantages:
> 1) We need to introduce at least two additional thread pools for each of 
> these caches
> 2) Since cache operations require stable topology, registering a class ID or 
> updating metadata inside a transaction or another cache operation is tricky 
> and deadlock-prone.
> 3) It may be beneficial in some cases to have nodes with no caches at all, 
> currently this is impossible because system caches are always present.
> 4) Reading binary metadata leads to huge local contention, caching metadata 
> values in a local map doubles memory consumption
> I suggest we use discovery custom events for these purposes. Each node will 
> have a corresponding local map (state) which will be updated inside custom 
> event handler. From the first point of view, this should remove all the 
> disadvantages above.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4157) Use discovery custom messages instead of marshaller cache

[Sergey Chugunov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15769059#comment-15769059
 ] 

Sergey Chugunov commented on IGNITE-4157:
-

Implemented server node failure safe case for #4.

> Use discovery custom messages instead of marshaller cache
> -
>
> Key: IGNITE-4157
> URL: https://issues.apache.org/jira/browse/IGNITE-4157
> Project: Ignite
>  Issue Type: Improvement
>  Components: cache
>Reporter: Alexey Goncharuk
>Assignee: Sergey Chugunov
> Fix For: 2.0
>
>
> Currently we use system caches for keeping classname to class ID mapping and 
> for storing binary metadata
> This has several serious disadvantages:
> 1) We need to introduce at least two additional thread pools for each of 
> these caches
> 2) Since cache operations require stable topology, registering a class ID or 
> updating metadata inside a transaction or another cache operation is tricky 
> and deadlock-prone.
> 3) It may be beneficial in some cases to have nodes with no caches at all, 
> currently this is impossible because system caches are always present.
> 4) Reading binary metadata leads to huge local contention, caching metadata 
> values in a local map doubles memory consumption
> I suggest we use discovery custom events for these purposes. Each node will 
> have a corresponding local map (state) which will be updated inside custom 
> event handler. From the first point of view, this should remove all the 
> disadvantages above.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4157) Use discovery custom messages instead of marshaller cache

[Sergey Chugunov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15764641#comment-15764641
 ] 

Sergey Chugunov commented on IGNITE-4157:
-

Implemented happy-path for #4.

> Use discovery custom messages instead of marshaller cache
> -
>
> Key: IGNITE-4157
> URL: https://issues.apache.org/jira/browse/IGNITE-4157
> Project: Ignite
>  Issue Type: Improvement
>  Components: cache
>Reporter: Alexey Goncharuk
>Assignee: Sergey Chugunov
> Fix For: 2.0
>
>
> Currently we use system caches for keeping classname to class ID mapping and 
> for storing binary metadata
> This has several serious disadvantages:
> 1) We need to introduce at least two additional thread pools for each of 
> these caches
> 2) Since cache operations require stable topology, registering a class ID or 
> updating metadata inside a transaction or another cache operation is tricky 
> and deadlock-prone.
> 3) It may be beneficial in some cases to have nodes with no caches at all, 
> currently this is impossible because system caches are always present.
> 4) Reading binary metadata leads to huge local contention, caching metadata 
> values in a local map doubles memory consumption
> I suggest we use discovery custom events for these purposes. Each node will 
> have a corresponding local map (state) which will be updated inside custom 
> event handler. From the first point of view, this should remove all the 
> disadvantages above.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4157) Use discovery custom messages instead of marshaller cache

[Sergey Chugunov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15754660#comment-15754660
 ] 

Sergey Chugunov commented on IGNITE-4157:
-

Received some comments on review including (from major to minor):
* DiscoveryDataContainer is part of public API but expresses some internal 
concepts through its interface;
* Mappings should be persisted outside of discovery thread;
* Managing map used for synchronization of mappings distribution should be done 
in GridFutureAdapter specialized subclass;
* It is more efficient for client to request missed mappings 
(*MissingMappingRequest/ResponseMessage*) using communication instead of 
discovery messages;
* *MessageRejectedMessage* may not be necessary, figuring out if I can get rid 
of additional pass of this message;

> Use discovery custom messages instead of marshaller cache
> -
>
> Key: IGNITE-4157
> URL: https://issues.apache.org/jira/browse/IGNITE-4157
> Project: Ignite
>  Issue Type: Improvement
>  Components: cache
>Reporter: Alexey Goncharuk
>Assignee: Sergey Chugunov
> Fix For: 2.0
>
>
> Currently we use system caches for keeping classname to class ID mapping and 
> for storing binary metadata
> This has several serious disadvantages:
> 1) We need to introduce at least two additional thread pools for each of 
> these caches
> 2) Since cache operations require stable topology, registering a class ID or 
> updating metadata inside a transaction or another cache operation is tricky 
> and deadlock-prone.
> 3) It may be beneficial in some cases to have nodes with no caches at all, 
> currently this is impossible because system caches are always present.
> 4) Reading binary metadata leads to huge local contention, caching metadata 
> values in a local map doubles memory consumption
> I suggest we use discovery custom events for these purposes. Each node will 
> have a corresponding local map (state) which will be updated inside custom 
> event handler. From the first point of view, this should remove all the 
> disadvantages above.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4157) Use discovery custom messages instead of marshaller cache

[Sergey Chugunov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751474#comment-15751474
 ] 

Sergey Chugunov commented on IGNITE-4157:
-

Fixed failures of newly added *IgniteMarshallerCacheClassNameConflictTest* (it 
was added to wrong TestSuite so was called with wrong parameters).
Fixed somewhat major issue with marshaller cache working directory which shown 
up as a failure of .NET platform *IgniteConfigurationTest.TestWorkDirectory* 
test.

Also created an issue 
[IGNITE-4435|https://issues.apache.org/jira/browse/IGNITE-4435] to address 
discovery collecting optimizations later.

> Use discovery custom messages instead of marshaller cache
> -
>
> Key: IGNITE-4157
> URL: https://issues.apache.org/jira/browse/IGNITE-4157
> Project: Ignite
>  Issue Type: Improvement
>  Components: cache
>Reporter: Alexey Goncharuk
>Assignee: Sergey Chugunov
> Fix For: 2.0
>
>
> Currently we use system caches for keeping classname to class ID mapping and 
> for storing binary metadata
> This has several serious disadvantages:
> 1) We need to introduce at least two additional thread pools for each of 
> these caches
> 2) Since cache operations require stable topology, registering a class ID or 
> updating metadata inside a transaction or another cache operation is tricky 
> and deadlock-prone.
> 3) It may be beneficial in some cases to have nodes with no caches at all, 
> currently this is impossible because system caches are always present.
> 4) Reading binary metadata leads to huge local contention, caching metadata 
> values in a local map doubles memory consumption
> I suggest we use discovery custom events for these purposes. Each node will 
> have a corresponding local map (state) which will be updated inside custom 
> event handler. From the first point of view, this should remove all the 
> disadvantages above.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4157) Use discovery custom messages instead of marshaller cache

[Sergey Chugunov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15748724#comment-15748724
 ] 

Sergey Chugunov commented on IGNITE-4157:
-

Found and fixed issue with validating topology upon executing transaction when 
there are only client nodes in topology. 
*GridCachePartitionedClientOnlyNoPrimaryFullApiSelfTest* and 
*GridCachePartitionedNearOnlyNoPrimaryFullApiSelfTest* tests are fixed.

Fixed some comments on open pull request.

> Use discovery custom messages instead of marshaller cache
> -
>
> Key: IGNITE-4157
> URL: https://issues.apache.org/jira/browse/IGNITE-4157
> Project: Ignite
>  Issue Type: Improvement
>  Components: cache
>Reporter: Alexey Goncharuk
>Assignee: Sergey Chugunov
> Fix For: 2.0
>
>
> Currently we use system caches for keeping classname to class ID mapping and 
> for storing binary metadata
> This has several serious disadvantages:
> 1) We need to introduce at least two additional thread pools for each of 
> these caches
> 2) Since cache operations require stable topology, registering a class ID or 
> updating metadata inside a transaction or another cache operation is tricky 
> and deadlock-prone.
> 3) It may be beneficial in some cases to have nodes with no caches at all, 
> currently this is impossible because system caches are always present.
> 4) Reading binary metadata leads to huge local contention, caching metadata 
> values in a local map doubles memory consumption
> I suggest we use discovery custom events for these purposes. Each node will 
> have a corresponding local map (state) which will be updated inside custom 
> event handler. From the first point of view, this should remove all the 
> disadvantages above.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4157) Use discovery custom messages instead of marshaller cache

[Sergey Chugunov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15745597#comment-15745597
 ] 

Sergey Chugunov commented on IGNITE-4157:
-

Fixed issue with newly added test for *MappingRejected* messages, fixed issue 
with existing *OptimizedMarshallerNodeFailoverTest* shown up after latest merge 
from master branch.

> Use discovery custom messages instead of marshaller cache
> -
>
> Key: IGNITE-4157
> URL: https://issues.apache.org/jira/browse/IGNITE-4157
> Project: Ignite
>  Issue Type: Improvement
>  Components: cache
>Reporter: Alexey Goncharuk
>Assignee: Sergey Chugunov
> Fix For: 2.0
>
>
> Currently we use system caches for keeping classname to class ID mapping and 
> for storing binary metadata
> This has several serious disadvantages:
> 1) We need to introduce at least two additional thread pools for each of 
> these caches
> 2) Since cache operations require stable topology, registering a class ID or 
> updating metadata inside a transaction or another cache operation is tricky 
> and deadlock-prone.
> 3) It may be beneficial in some cases to have nodes with no caches at all, 
> currently this is impossible because system caches are always present.
> 4) Reading binary metadata leads to huge local contention, caching metadata 
> values in a local map doubles memory consumption
> I suggest we use discovery custom events for these purposes. Each node will 
> have a corresponding local map (state) which will be updated inside custom 
> event handler. From the first point of view, this should remove all the 
> disadvantages above.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4157) Use discovery custom messages instead of marshaller cache

[Sergey Chugunov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15742188#comment-15742188
 ] 

Sergey Chugunov commented on IGNITE-4157:
-

Alexey, I covered this improvement in *Other improvements* section. Yet be 
aware that this improvement was used only for new MarshallerMappingProcessor 
component.

Other components like GridCacheProcessor still don't use all advantages of 
DiscoveryDataContainer entity and collect discovery data on each node of 
cluster.

I didn't convert them to speed up development and reduce number of changes to 
merge. This should be done later as part of a separate story.

> Use discovery custom messages instead of marshaller cache
> -
>
> Key: IGNITE-4157
> URL: https://issues.apache.org/jira/browse/IGNITE-4157
> Project: Ignite
>  Issue Type: Improvement
>  Components: cache
>Reporter: Alexey Goncharuk
>Assignee: Sergey Chugunov
> Fix For: 2.0
>
>
> Currently we use system caches for keeping classname to class ID mapping and 
> for storing binary metadata
> This has several serious disadvantages:
> 1) We need to introduce at least two additional thread pools for each of 
> these caches
> 2) Since cache operations require stable topology, registering a class ID or 
> updating metadata inside a transaction or another cache operation is tricky 
> and deadlock-prone.
> 3) It may be beneficial in some cases to have nodes with no caches at all, 
> currently this is impossible because system caches are always present.
> 4) Reading binary metadata leads to huge local contention, caching metadata 
> values in a local map doubles memory consumption
> I suggest we use discovery custom events for these purposes. Each node will 
> have a corresponding local map (state) which will be updated inside custom 
> event handler. From the first point of view, this should remove all the 
> disadvantages above.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4157) Use discovery custom messages instead of marshaller cache

[Alexey Goncharuk (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15741949#comment-15741949
 ] 

Alexey Goncharuk commented on IGNITE-4157:
--

Sergey, the design looks good to me. Do I understand correctly that you also 
reduced the amount of data being transferred on discovery stage for this 
use-case because each node returns identical id <-> classname mapping? If so, 
could you also add this to the design note above, and we will include this to 
the public wiki?

> Use discovery custom messages instead of marshaller cache
> -
>
> Key: IGNITE-4157
> URL: https://issues.apache.org/jira/browse/IGNITE-4157
> Project: Ignite
>  Issue Type: Improvement
>  Components: cache
>Reporter: Alexey Goncharuk
>Assignee: Sergey Chugunov
> Fix For: 2.0
>
>
> Currently we use system caches for keeping classname to class ID mapping and 
> for storing binary metadata
> This has several serious disadvantages:
> 1) We need to introduce at least two additional thread pools for each of 
> these caches
> 2) Since cache operations require stable topology, registering a class ID or 
> updating metadata inside a transaction or another cache operation is tricky 
> and deadlock-prone.
> 3) It may be beneficial in some cases to have nodes with no caches at all, 
> currently this is impossible because system caches are always present.
> 4) Reading binary metadata leads to huge local contention, caching metadata 
> values in a local map doubles memory consumption
> I suggest we use discovery custom events for these purposes. Each node will 
> have a corresponding local map (state) which will be updated inside custom 
> event handler. From the first point of view, this should remove all the 
> disadvantages above.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4157) Use discovery custom messages instead of marshaller cache

[Sergey Chugunov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15741923#comment-15741923
 ] 

Sergey Chugunov commented on IGNITE-4157:
-

h4. Design notes for final version

# Each node holds a local cache of typeId->mappedName mappings. MappedName 
contains typeName and a status of mappedName (*proposed* or *accepted*).
# Node is allowed to use proposed mappedName only to unmarshal incoming 
requests. In order to use it for marshalling it should wait until mappedName 
gets transferred to accepted status.
# Requesting new mapping is two-phase process and requires two messages to pass 
across the cluster: *MappingProposedMessage* and *MappingAcceptedMessage* as a 
successful acknowledge message for the first one.
# Requesting new mapping is synchronous process: when node requests new 
mapping, requesting thread is blocked until mapping is accepted or rejected.
# If two nodes simultaneously propose mappings for classes with conflicting 
names, only the mapping will be accepted which MappingProposedMessage reaches 
coordinator node first. Other mapping proposed request will be rejected,  
*MappingRejectedMessage* will be sent to requesting node as a failure 
acknowledge.
# If two nodes simultaneously propose mapping for exactly the same class, this 
situation will be treated by coordinator node as a duplicate request, no 
acknowledge message will be sent. Node that sent duplicate request will be 
unblocked from waiting on mapping accepted when accept for the first request 
passes across the ring.
# Functionality of persisting accepted mappings in filesystem is preserved, 
file name format slightly changed as support for mappings on different 
platforms was added. Pattern looks like this now: 
*.classname*, e.g. *1033.classname0*

> Use discovery custom messages instead of marshaller cache
> -
>
> Key: IGNITE-4157
> URL: https://issues.apache.org/jira/browse/IGNITE-4157
> Project: Ignite
>  Issue Type: Improvement
>  Components: cache
>Reporter: Alexey Goncharuk
>Assignee: Sergey Chugunov
> Fix For: 2.0
>
>
> Currently we use system caches for keeping classname to class ID mapping and 
> for storing binary metadata
> This has several serious disadvantages:
> 1) We need to introduce at least two additional thread pools for each of 
> these caches
> 2) Since cache operations require stable topology, registering a class ID or 
> updating metadata inside a transaction or another cache operation is tricky 
> and deadlock-prone.
> 3) It may be beneficial in some cases to have nodes with no caches at all, 
> currently this is impossible because system caches are always present.
> 4) Reading binary metadata leads to huge local contention, caching metadata 
> values in a local map doubles memory consumption
> I suggest we use discovery custom events for these purposes. Each node will 
> have a corresponding local map (state) which will be updated inside custom 
> event handler. From the first point of view, this should remove all the 
> disadvantages above.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (IGNITE-4157) Use discovery custom messages instead of marshaller cache

[Sergey Chugunov (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15711292#comment-15711292
 ] 

Sergey Chugunov commented on IGNITE-4157:
-

Marshaller mapping exchange was rewritten to work synchronously: when some node 
wants to add new mapping it gets blocked until grid accepts the mapping or 
rejects it.

> Use discovery custom messages instead of marshaller cache
> -
>
> Key: IGNITE-4157
> URL: https://issues.apache.org/jira/browse/IGNITE-4157
> Project: Ignite
>  Issue Type: Improvement
>  Components: cache
>Reporter: Alexey Goncharuk
>Assignee: Sergey Chugunov
> Fix For: 2.0
>
>
> Currently we use system caches for keeping classname to class ID mapping and 
> for storing binary metadata
> This has several serious disadvantages:
> 1) We need to introduce at least two additional thread pools for each of 
> these caches
> 2) Since cache operations require stable topology, registering a class ID or 
> updating metadata inside a transaction or another cache operation is tricky 
> and deadlock-prone.
> 3) It may be beneficial in some cases to have nodes with no caches at all, 
> currently this is impossible because system caches are always present.
> 4) Reading binary metadata leads to huge local contention, caching metadata 
> values in a local map doubles memory consumption
> I suggest we use discovery custom events for these purposes. Each node will 
> have a corresponding local map (state) which will be updated inside custom 
> event handler. From the first point of view, this should remove all the 
> disadvantages above.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)