[jira] [Commented] (IGNITE-7918) Huge memory leak when data streamer used together with local cache
[ https://issues.apache.org/jira/browse/IGNITE-7918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16462103#comment-16462103 ] Andrey Aleksandrov commented on IGNITE-7918: [~dpavlov] Thank you. Changes that you saw are results of fixing code style issues and merging with master that I did to re-run failed suites. > Huge memory leak when data streamer used together with local cache > -- > > Key: IGNITE-7918 > URL: https://issues.apache.org/jira/browse/IGNITE-7918 > Project: Ignite > Issue Type: Bug > Components: cache >Affects Versions: 2.3 >Reporter: Zbyszek B >Assignee: Andrey Aleksandrov >Priority: Major > Fix For: 2.6 > > Attachments: Demo.java, MemLeak-Ignite.png, MemLeak-Ignite.txt > > > Dear Igniters, > We observe huge memory leak when data streamer used together with local cache. > In the attached demo producer produces local cache with single binary object > and passes this to the queue. Consumer picks up the cache from the queue, > constructs different binary object from it, adds it to global partitioned > cache and destroys local cache. > This design causes a significant leak - the whole heap is used within minutes > (no matter if this is 4G or 24G). > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7918) Huge memory leak when data streamer used together with local cache
[ https://issues.apache.org/jira/browse/IGNITE-7918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16457680#comment-16457680 ] ASF GitHub Bot commented on IGNITE-7918: Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/3778 > Huge memory leak when data streamer used together with local cache > -- > > Key: IGNITE-7918 > URL: https://issues.apache.org/jira/browse/IGNITE-7918 > Project: Ignite > Issue Type: Bug > Components: cache >Affects Versions: 2.3 >Reporter: Zbyszek B >Assignee: Andrey Aleksandrov >Priority: Major > Fix For: 2.6 > > Attachments: Demo.java, MemLeak-Ignite.png, MemLeak-Ignite.txt > > > Dear Igniters, > We observe huge memory leak when data streamer used together with local cache. > In the attached demo producer produces local cache with single binary object > and passes this to the queue. Consumer picks up the cache from the queue, > constructs different binary object from it, adds it to global partitioned > cache and destroys local cache. > This design causes a significant leak - the whole heap is used within minutes > (no matter if this is 4G or 24G). > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7918) Huge memory leak when data streamer used together with local cache
[ https://issues.apache.org/jira/browse/IGNITE-7918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16457668#comment-16457668 ] Dmitriy Pavlov commented on IGNITE-7918: [~aealeksandrov] it seems run-all is not actual, TC shows 2-3 changes after run. Why did you decided to use old run-all? > Huge memory leak when data streamer used together with local cache > -- > > Key: IGNITE-7918 > URL: https://issues.apache.org/jira/browse/IGNITE-7918 > Project: Ignite > Issue Type: Bug > Components: cache >Affects Versions: 2.3 >Reporter: Zbyszek B >Assignee: Andrey Aleksandrov >Priority: Major > Fix For: 2.6 > > Attachments: Demo.java, MemLeak-Ignite.png, MemLeak-Ignite.txt > > > Dear Igniters, > We observe huge memory leak when data streamer used together with local cache. > In the attached demo producer produces local cache with single binary object > and passes this to the queue. Consumer picks up the cache from the queue, > constructs different binary object from it, adds it to global partitioned > cache and destroys local cache. > This design causes a significant leak - the whole heap is used within minutes > (no matter if this is 4G or 24G). > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7918) Huge memory leak when data streamer used together with local cache
[ https://issues.apache.org/jira/browse/IGNITE-7918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16456291#comment-16456291 ] Andrey Aleksandrov commented on IGNITE-7918: [~dpavlov] results of tests - [https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8&branch_IgniteTests24Java8=pull%2F3778%2Fhead] Could you please take a look and merge if everything is ok? > Huge memory leak when data streamer used together with local cache > -- > > Key: IGNITE-7918 > URL: https://issues.apache.org/jira/browse/IGNITE-7918 > Project: Ignite > Issue Type: Bug > Components: cache >Affects Versions: 2.3 >Reporter: Zbyszek B >Assignee: Andrey Aleksandrov >Priority: Major > Fix For: 2.6 > > Attachments: Demo.java, MemLeak-Ignite.png, MemLeak-Ignite.txt > > > Dear Igniters, > We observe huge memory leak when data streamer used together with local cache. > In the attached demo producer produces local cache with single binary object > and passes this to the queue. Consumer picks up the cache from the queue, > constructs different binary object from it, adds it to global partitioned > cache and destroys local cache. > This design causes a significant leak - the whole heap is used within minutes > (no matter if this is 4G or 24G). > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7918) Huge memory leak when data streamer used together with local cache
[ https://issues.apache.org/jira/browse/IGNITE-7918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16453758#comment-16453758 ] Dmitriy Govorukhin commented on IGNITE-7918: [~aealeksandrov] Looks good for me, please merge. > Huge memory leak when data streamer used together with local cache > -- > > Key: IGNITE-7918 > URL: https://issues.apache.org/jira/browse/IGNITE-7918 > Project: Ignite > Issue Type: Bug > Components: cache >Affects Versions: 2.3 >Reporter: Zbyszek B >Assignee: Andrey Aleksandrov >Priority: Major > Fix For: 2.6 > > Attachments: Demo.java, MemLeak-Ignite.png, MemLeak-Ignite.txt > > > Dear Igniters, > We observe huge memory leak when data streamer used together with local cache. > In the attached demo producer produces local cache with single binary object > and passes this to the queue. Consumer picks up the cache from the queue, > constructs different binary object from it, adds it to global partitioned > cache and destroys local cache. > This design causes a significant leak - the whole heap is used within minutes > (no matter if this is 4G or 24G). > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7918) Huge memory leak when data streamer used together with local cache
[ https://issues.apache.org/jira/browse/IGNITE-7918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16447963#comment-16447963 ] Andrey Aleksandrov commented on IGNITE-7918: [~agoncharuk] fixed. Now null cache name is used. Key with null cache name now must be less than any key with not null cache name for the same topVer. > Huge memory leak when data streamer used together with local cache > -- > > Key: IGNITE-7918 > URL: https://issues.apache.org/jira/browse/IGNITE-7918 > Project: Ignite > Issue Type: Bug > Components: cache >Affects Versions: 2.3 >Reporter: Zbyszek B >Assignee: Andrey Aleksandrov >Priority: Major > Fix For: 2.6 > > Attachments: Demo.java, MemLeak-Ignite.png, MemLeak-Ignite.txt > > > Dear Igniters, > We observe huge memory leak when data streamer used together with local cache. > In the attached demo producer produces local cache with single binary object > and passes this to the queue. Consumer picks up the cache from the queue, > constructs different binary object from it, adds it to global partitioned > cache and destroys local cache. > This design causes a significant leak - the whole heap is used within minutes > (no matter if this is 4G or 24G). > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7918) Huge memory leak when data streamer used together with local cache
[
https://issues.apache.org/jira/browse/IGNITE-7918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16445896#comment-16445896
]
Alexey Goncharuk commented on IGNITE-7918:
--
[~aealeksandrov], {{new AffinityAssignmentKey("topVerRmv", topVerRmv)}} does
not look right - what if a cache starts with a name that is smaller than
"topVer"? We need to have a special key here.
Otherwise looks good.
> Huge memory leak when data streamer used together with local cache
> --
>
> Key: IGNITE-7918
> URL: https://issues.apache.org/jira/browse/IGNITE-7918
> Project: Ignite
> Issue Type: Bug
> Components: cache
>Affects Versions: 2.3
>Reporter: Zbyszek B
>Assignee: Andrey Aleksandrov
>Priority: Major
> Fix For: 2.6
>
> Attachments: Demo.java, MemLeak-Ignite.png, MemLeak-Ignite.txt
>
>
> Dear Igniters,
> We observe huge memory leak when data streamer used together with local cache.
> In the attached demo producer produces local cache with single binary object
> and passes this to the queue. Consumer picks up the cache from the queue,
> constructs different binary object from it, adds it to global partitioned
> cache and destroys local cache.
> This design causes a significant leak - the whole heap is used within minutes
> (no matter if this is 4G or 24G).
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (IGNITE-7918) Huge memory leak when data streamer used together with local cache
[ https://issues.apache.org/jira/browse/IGNITE-7918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16440580#comment-16440580 ] Andrey Aleksandrov commented on IGNITE-7918: [~agoncharuk] fix is ready. Tests run: https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8&branch_IgniteTests24Java8=pull%2F3778%2Fhead > Huge memory leak when data streamer used together with local cache > -- > > Key: IGNITE-7918 > URL: https://issues.apache.org/jira/browse/IGNITE-7918 > Project: Ignite > Issue Type: Bug > Components: cache >Affects Versions: 2.3 >Reporter: Zbyszek B >Assignee: Andrey Aleksandrov >Priority: Major > Fix For: 2.5 > > Attachments: Demo.java, MemLeak-Ignite.png, MemLeak-Ignite.txt > > > Dear Igniters, > We observe huge memory leak when data streamer used together with local cache. > In the attached demo producer produces local cache with single binary object > and passes this to the queue. Consumer picks up the cache from the queue, > constructs different binary object from it, adds it to global partitioned > cache and destroys local cache. > This design causes a significant leak - the whole heap is used within minutes > (no matter if this is 4G or 24G). > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7918) Huge memory leak when data streamer used together with local cache
[
https://issues.apache.org/jira/browse/IGNITE-7918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16439486#comment-16439486
]
Alexey Goncharuk commented on IGNITE-7918:
--
[~aealeksandrov], in your updated implementation {{compareTo}} is not
consistent with {{equals}}, so it looks like in affinity history map will
always be only one entry per topology version.
> Huge memory leak when data streamer used together with local cache
> --
>
> Key: IGNITE-7918
> URL: https://issues.apache.org/jira/browse/IGNITE-7918
> Project: Ignite
> Issue Type: Bug
> Components: cache
>Affects Versions: 2.3
>Reporter: Zbyszek B
>Assignee: Andrey Aleksandrov
>Priority: Major
> Fix For: 2.5
>
> Attachments: Demo.java, MemLeak-Ignite.png, MemLeak-Ignite.txt
>
>
> Dear Igniters,
> We observe huge memory leak when data streamer used together with local cache.
> In the attached demo producer produces local cache with single binary object
> and passes this to the queue. Consumer picks up the cache from the queue,
> constructs different binary object from it, adds it to global partitioned
> cache and destroys local cache.
> This design causes a significant leak - the whole heap is used within minutes
> (no matter if this is 4G or 24G).
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (IGNITE-7918) Huge memory leak when data streamer used together with local cache
[ https://issues.apache.org/jira/browse/IGNITE-7918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16439413#comment-16439413 ] Andrey Aleksandrov commented on IGNITE-7918: [~dpavlov], [~agoncharuk] ready for review. Could you please take a look? Also I am going to rerun TS. > Huge memory leak when data streamer used together with local cache > -- > > Key: IGNITE-7918 > URL: https://issues.apache.org/jira/browse/IGNITE-7918 > Project: Ignite > Issue Type: Bug > Components: cache >Affects Versions: 2.3 >Reporter: Zbyszek B >Assignee: Andrey Aleksandrov >Priority: Major > Fix For: 2.5 > > Attachments: Demo.java, MemLeak-Ignite.png, MemLeak-Ignite.txt > > > Dear Igniters, > We observe huge memory leak when data streamer used together with local cache. > In the attached demo producer produces local cache with single binary object > and passes this to the queue. Consumer picks up the cache from the queue, > constructs different binary object from it, adds it to global partitioned > cache and destroys local cache. > This design causes a significant leak - the whole heap is used within minutes > (no matter if this is 4G or 24G). > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7918) Huge memory leak when data streamer used together with local cache
[ https://issues.apache.org/jira/browse/IGNITE-7918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437281#comment-16437281 ] Andrey Aleksandrov commented on IGNITE-7918: [~agoncharuk] In current case with this fix the size of this map should not be bigger then IGNITE_AFFINITY_HISTORY_SIZE value. Yes it could be changed to ConcurrentSkipListMap as you mentioned for avoiding of the full iteration over the affMap. Thank you! > Huge memory leak when data streamer used together with local cache > -- > > Key: IGNITE-7918 > URL: https://issues.apache.org/jira/browse/IGNITE-7918 > Project: Ignite > Issue Type: Bug > Components: cache >Affects Versions: 2.3 >Reporter: Zbyszek B >Assignee: Andrey Aleksandrov >Priority: Major > Fix For: 2.5 > > Attachments: Demo.java, MemLeak-Ignite.png, MemLeak-Ignite.txt > > > Dear Igniters, > We observe huge memory leak when data streamer used together with local cache. > In the attached demo producer produces local cache with single binary object > and passes this to the queue. Consumer picks up the cache from the queue, > constructs different binary object from it, adds it to global partitioned > cache and destroys local cache. > This design causes a significant leak - the whole heap is used within minutes > (no matter if this is 4G or 24G). > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7918) Huge memory leak when data streamer used together with local cache
[ https://issues.apache.org/jira/browse/IGNITE-7918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437239#comment-16437239 ] Dmitriy Pavlov commented on IGNITE-7918: [~aealeksandrov], if you're agree with notes, please set InProgress status. > Huge memory leak when data streamer used together with local cache > -- > > Key: IGNITE-7918 > URL: https://issues.apache.org/jira/browse/IGNITE-7918 > Project: Ignite > Issue Type: Bug > Components: cache >Affects Versions: 2.3 >Reporter: Zbyszek B >Assignee: Andrey Aleksandrov >Priority: Major > Fix For: 2.5 > > Attachments: Demo.java, MemLeak-Ignite.png, MemLeak-Ignite.txt > > > Dear Igniters, > We observe huge memory leak when data streamer used together with local cache. > In the attached demo producer produces local cache with single binary object > and passes this to the queue. Consumer picks up the cache from the queue, > constructs different binary object from it, adds it to global partitioned > cache and destroys local cache. > This design causes a significant leak - the whole heap is used within minutes > (no matter if this is 4G or 24G). > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7918) Huge memory leak when data streamer used together with local cache
[
https://issues.apache.org/jira/browse/IGNITE-7918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437178#comment-16437178
]
Alexey Goncharuk commented on IGNITE-7918:
--
[~aealeksandrov] I am a bit concerned that cleanup is done using a full
iteration over the {{affMap}}. What is the expected and max size of this map in
the use-case you were debugging? Do you think it makes sense to change the map
type to ConcurrentSkipListMap (make AffinityAssignmentKey comparable with
topVer being compared first)?
> Huge memory leak when data streamer used together with local cache
> --
>
> Key: IGNITE-7918
> URL: https://issues.apache.org/jira/browse/IGNITE-7918
> Project: Ignite
> Issue Type: Bug
> Components: cache
>Affects Versions: 2.3
>Reporter: Zbyszek B
>Assignee: Andrey Aleksandrov
>Priority: Major
> Fix For: 2.5
>
> Attachments: Demo.java, MemLeak-Ignite.png, MemLeak-Ignite.txt
>
>
> Dear Igniters,
> We observe huge memory leak when data streamer used together with local cache.
> In the attached demo producer produces local cache with single binary object
> and passes this to the queue. Consumer picks up the cache from the queue,
> constructs different binary object from it, adds it to global partitioned
> cache and destroys local cache.
> This design causes a significant leak - the whole heap is used within minutes
> (no matter if this is 4G or 24G).
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (IGNITE-7918) Huge memory leak when data streamer used together with local cache
[ https://issues.apache.org/jira/browse/IGNITE-7918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437124#comment-16437124 ] Andrey Aleksandrov commented on IGNITE-7918: Thank you! I will fix them soon and retun to you. > Huge memory leak when data streamer used together with local cache > -- > > Key: IGNITE-7918 > URL: https://issues.apache.org/jira/browse/IGNITE-7918 > Project: Ignite > Issue Type: Bug > Components: cache >Affects Versions: 2.3 >Reporter: Zbyszek B >Assignee: Andrey Aleksandrov >Priority: Major > Fix For: 2.5 > > Attachments: Demo.java, MemLeak-Ignite.png, MemLeak-Ignite.txt > > > Dear Igniters, > We observe huge memory leak when data streamer used together with local cache. > In the attached demo producer produces local cache with single binary object > and passes this to the queue. Consumer picks up the cache from the queue, > constructs different binary object from it, adds it to global partitioned > cache and destroys local cache. > This design causes a significant leak - the whole heap is used within minutes > (no matter if this is 4G or 24G). > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7918) Huge memory leak when data streamer used together with local cache
[ https://issues.apache.org/jira/browse/IGNITE-7918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437117#comment-16437117 ] Dmitriy Pavlov commented on IGNITE-7918: [~aealeksandrov] I left several code style-related proposals in PR comments. Could you please address? > Huge memory leak when data streamer used together with local cache > -- > > Key: IGNITE-7918 > URL: https://issues.apache.org/jira/browse/IGNITE-7918 > Project: Ignite > Issue Type: Bug > Components: cache >Affects Versions: 2.3 >Reporter: Zbyszek B >Assignee: Andrey Aleksandrov >Priority: Major > Fix For: 2.5 > > Attachments: Demo.java, MemLeak-Ignite.png, MemLeak-Ignite.txt > > > Dear Igniters, > We observe huge memory leak when data streamer used together with local cache. > In the attached demo producer produces local cache with single binary object > and passes this to the queue. Consumer picks up the cache from the queue, > constructs different binary object from it, adds it to global partitioned > cache and destroys local cache. > This design causes a significant leak - the whole heap is used within minutes > (no matter if this is 4G or 24G). > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7918) Huge memory leak when data streamer used together with local cache
[ https://issues.apache.org/jira/browse/IGNITE-7918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16435481#comment-16435481 ] Andrey Aleksandrov commented on IGNITE-7918: [~dpavlov] Could you please take a look and commit if everything is ok? > Huge memory leak when data streamer used together with local cache > -- > > Key: IGNITE-7918 > URL: https://issues.apache.org/jira/browse/IGNITE-7918 > Project: Ignite > Issue Type: Bug > Components: cache >Affects Versions: 2.3 >Reporter: Zbyszek B >Assignee: Andrey Aleksandrov >Priority: Blocker > Fix For: 2.5 > > Attachments: Demo.java, MemLeak-Ignite.png, MemLeak-Ignite.txt > > > Dear Igniters, > We observe huge memory leak when data streamer used together with local cache. > In the attached demo producer produces local cache with single binary object > and passes this to the queue. Consumer picks up the cache from the queue, > constructs different binary object from it, adds it to global partitioned > cache and destroys local cache. > This design causes a significant leak - the whole heap is used within minutes > (no matter if this is 4G or 24G). > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7918) Huge memory leak when data streamer used together with local cache
[ https://issues.apache.org/jira/browse/IGNITE-7918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16435402#comment-16435402 ] Ilya Lantukh commented on IGNITE-7918: -- Looks good. > Huge memory leak when data streamer used together with local cache > -- > > Key: IGNITE-7918 > URL: https://issues.apache.org/jira/browse/IGNITE-7918 > Project: Ignite > Issue Type: Bug > Components: cache >Affects Versions: 2.3 >Reporter: Zbyszek B >Assignee: Andrey Aleksandrov >Priority: Blocker > Fix For: 2.5 > > Attachments: Demo.java, MemLeak-Ignite.png, MemLeak-Ignite.txt > > > Dear Igniters, > We observe huge memory leak when data streamer used together with local cache. > In the attached demo producer produces local cache with single binary object > and passes this to the queue. Consumer picks up the cache from the queue, > constructs different binary object from it, adds it to global partitioned > cache and destroys local cache. > This design causes a significant leak - the whole heap is used within minutes > (no matter if this is 4G or 24G). > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7918) Huge memory leak when data streamer used together with local cache
[ https://issues.apache.org/jira/browse/IGNITE-7918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16435123#comment-16435123 ] Andrey Aleksandrov commented on IGNITE-7918: Team city test run: https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8&branch_IgniteTests24Java8=pull%2F3778%2Fhead > Huge memory leak when data streamer used together with local cache > -- > > Key: IGNITE-7918 > URL: https://issues.apache.org/jira/browse/IGNITE-7918 > Project: Ignite > Issue Type: Bug > Components: cache >Affects Versions: 2.3 >Reporter: Zbyszek B >Assignee: Andrey Aleksandrov >Priority: Blocker > Fix For: 2.5 > > Attachments: Demo.java, MemLeak-Ignite.png, MemLeak-Ignite.txt > > > Dear Igniters, > We observe huge memory leak when data streamer used together with local cache. > In the attached demo producer produces local cache with single binary object > and passes this to the queue. Consumer picks up the cache from the queue, > constructs different binary object from it, adds it to global partitioned > cache and destroys local cache. > This design causes a significant leak - the whole heap is used within minutes > (no matter if this is 4G or 24G). > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7918) Huge memory leak when data streamer used together with local cache
[ https://issues.apache.org/jira/browse/IGNITE-7918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16430718#comment-16430718 ] Andrey Aleksandrov commented on IGNITE-7918: [~ilantukh] Could you please also take a look at this PR? > Huge memory leak when data streamer used together with local cache > -- > > Key: IGNITE-7918 > URL: https://issues.apache.org/jira/browse/IGNITE-7918 > Project: Ignite > Issue Type: Bug > Components: cache >Affects Versions: 2.3 >Reporter: Zbyszek B >Assignee: Andrey Aleksandrov >Priority: Blocker > Fix For: 2.5 > > Attachments: Demo.java, MemLeak-Ignite.png, MemLeak-Ignite.txt > > > Dear Igniters, > We observe huge memory leak when data streamer used together with local cache. > In the attached demo producer produces local cache with single binary object > and passes this to the queue. Consumer picks up the cache from the queue, > constructs different binary object from it, adds it to global partitioned > cache and destroys local cache. > This design causes a significant leak - the whole heap is used within minutes > (no matter if this is 4G or 24G). > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7918) Huge memory leak when data streamer used together with local cache
[ https://issues.apache.org/jira/browse/IGNITE-7918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16430546#comment-16430546 ] Andrey Aleksandrov commented on IGNITE-7918: [~amashenkov] Could you please talke a look at the pull request? > Huge memory leak when data streamer used together with local cache > -- > > Key: IGNITE-7918 > URL: https://issues.apache.org/jira/browse/IGNITE-7918 > Project: Ignite > Issue Type: Bug > Components: cache >Affects Versions: 2.3 >Reporter: Zbyszek B >Assignee: Andrey Aleksandrov >Priority: Blocker > Fix For: 2.5 > > Attachments: Demo.java, MemLeak-Ignite.png, MemLeak-Ignite.txt > > > Dear Igniters, > We observe huge memory leak when data streamer used together with local cache. > In the attached demo producer produces local cache with single binary object > and passes this to the queue. Consumer picks up the cache from the queue, > constructs different binary object from it, adds it to global partitioned > cache and destroys local cache. > This design causes a significant leak - the whole heap is used within minutes > (no matter if this is 4G or 24G). > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7918) Huge memory leak when data streamer used together with local cache
[ https://issues.apache.org/jira/browse/IGNITE-7918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16430534#comment-16430534 ] ASF GitHub Bot commented on IGNITE-7918: GitHub user aealeksandrov opened a pull request: https://github.com/apache/ignite/pull/3778 IGNITE-7918: memory leak was fixed. Unit test was added. Some caches didn't take into account that topology minor version could be changed during creating/destroying of the caches dynamically. These places were fixed. You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-7918 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/3778.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3778 commit 15b8623c684c6250c1c700459299e7986fdd566f Author: Andrei Aleksandrov Date: 2018-04-09T13:32:45Z IGNITE-7918: memory leak was fixed. Unit test was added. > Huge memory leak when data streamer used together with local cache > -- > > Key: IGNITE-7918 > URL: https://issues.apache.org/jira/browse/IGNITE-7918 > Project: Ignite > Issue Type: Bug > Components: cache >Affects Versions: 2.3 >Reporter: Zbyszek B >Assignee: Andrey Aleksandrov >Priority: Blocker > Fix For: 2.5 > > Attachments: Demo.java, MemLeak-Ignite.png, MemLeak-Ignite.txt > > > Dear Igniters, > We observe huge memory leak when data streamer used together with local cache. > In the attached demo producer produces local cache with single binary object > and passes this to the queue. Consumer picks up the cache from the queue, > constructs different binary object from it, adds it to global partitioned > cache and destroys local cache. > This design causes a significant leak - the whole heap is used within minutes > (no matter if this is 4G or 24G). > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (IGNITE-7918) Huge memory leak when data streamer used together with local cache
[
https://issues.apache.org/jira/browse/IGNITE-7918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395087#comment-16395087
]
Zbyszek B commented on IGNITE-7918:
---
{noformat}
C:\Dev>"C:\Program Files\Java\jdk1.8.0_111\bin\java" -Xmx4G -XX:+UseG1GC
"-javaagent:C:\Program Files\JetBrains\IntelliJ IDEA Community Edition
2017.2.5\lib\ide
a_rt.jar=52512:C:\Program Files\JetBrains\IntelliJ IDEA Community Edition
2017.2.5\bin" -Dfile.encoding=UTF-8 -classpath "C:\Program
Files\Java\jdk1.8.0_111\jre
\lib\charsets.jar;C:\Program
Files\Java\jdk1.8.0_111\jre\lib\deploy.jar;C:\Program
Files\Java\jdk1.8.0_111\jre\lib\ext\access-bridge-64.jar;C:\Program Files\Jav
a\jdk1.8.0_111\jre\lib\ext\cldrdata.jar;C:\Program
Files\Java\jdk1.8.0_111\jre\lib\ext\dnsns.jar;C:\Program
Files\Java\jdk1.8.0_111\jre\lib\ext\jaccess.jar;C:\P
rogram Files\Java\jdk1.8.0_111\jre\lib\ext\jfxrt.jar;C:\Program
Files\Java\jdk1.8.0_111\jre\lib\ext\localedata.jar;C:\Program
Files\Java\jdk1.8.0_111\jre\lib\ex
t\nashorn.jar;C:\Program
Files\Java\jdk1.8.0_111\jre\lib\ext\sunec.jar;C:\Program
Files\Java\jdk1.8.0_111\jre\lib\ext\sunjce_provider.jar;C:\Program Files\Java\
jdk1.8.0_111\jre\lib\ext\sunmscapi.jar;C:\Program
Files\Java\jdk1.8.0_111\jre\lib\ext\sunpkcs11.jar;C:\Program
Files\Java\jdk1.8.0_111\jre\lib\ext\zipfs.jar;C:\
Program Files\Java\jdk1.8.0_111\jre\lib\javaws.jar;C:\Program
Files\Java\jdk1.8.0_111\jre\lib\jce.jar;C:\Program
Files\Java\jdk1.8.0_111\jre\lib\jfr.jar;C:\Prog
ram Files\Java\jdk1.8.0_111\jre\lib\jfxswt.jar;C:\Program
Files\Java\jdk1.8.0_111\jre\lib\jsse.jar;C:\Program
Files\Java\jdk1.8.0_111\jre\lib\management-agent.j
ar;C:\Program Files\Java\jdk1.8.0_111\jre\lib\plugin.jar;C:\Program
Files\Java\jdk1.8.0_111\jre\lib\resources.jar;C:\Program
Files\Java\jdk1.8.0_111\jre\lib\rt.
jar;C:\Dev\IgniteMemLeak4\out\production\IgniteDemo;C:\Dev\IgniteMemLeak4\lib\2.3\cache-api-1.0.0.jar;C:\Dev\IgniteMemLeak4\lib\2.3\commons-codec-1.6.jar;C:\Dev
\IgniteMemLeak4\lib\2.3\h2-1.4.195.jar;C:\Dev\IgniteMemLeak4\lib\2.3\ignite-core-2.3.0.jar;C:\Dev\IgniteMemLeak4\lib\2.3\ignite-indexing-2.3.0.jar;C:\Dev\Ignite
MemLeak4\lib\2.3\lucene-analyzers-common-5.5.2.jar;C:\Dev\IgniteMemLeak4\lib\2.3\lucene-core-5.5.2.jar;C:\Dev\IgniteMemLeak4\lib\2.3\lucene-queryparser-5.5.2.ja
r" leak.Demo
>>> Starting Ignite on MyLeakingNode; work directory C:\Users\user1\ignite ...
Mar 11, 2018 5:24:06 PM java.util.logging.LogManager$RootLogger log
SEVERE: Failed to resolve default logging config file:
config/java.util.logging.properties
[17:24:06] __
[17:24:06] / _/ ___/ |/ / _/_ __/ __/
[17:24:06] _/ // (7 7 // / / / / _/
[17:24:06] /___/\___/_/|_/___/ /_/ /___/
[17:24:06]
[17:24:06] ver. 2.3.0#20171028-sha1:8add7fd5
[17:24:06] 2017 Copyright(C) Apache Software Foundation
[17:24:06]
[17:24:06] Ignite documentation: http://ignite.apache.org
[17:24:06]
[17:24:06] Quiet mode.
[17:24:06] ^-- To see **FULL** console log here add -DIGNITE_QUIET=false or
"-v" to ignite.{sh|bat}
[17:24:06]
[17:24:06] OS: Windows 7 6.1 amd64
[17:24:06] VM information: Java(TM) SE Runtime Environment 1.8.0_111-b14 Oracle
Corporation Java HotSpot(TM) 64-Bit Server VM 25.111-b14
[17:24:06] Initial heap size is 256MB (should be no less than 512MB, use
-Xms512m -Xmx512m).
[17:24:06] Configured plugins:
[17:24:06] ^-- None
[17:24:06]
[17:24:06] Security status [authentication=off, tls/ssl=off]
[17:24:08] Performance suggestions for grid (fix if possible)
[17:24:08] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
[17:24:08] ^-- Set max direct memory size if getting 'OOME: Direct buffer
memory' (add '-XX:MaxDirectMemorySize=[g|G|m|M|k|K]' to JVM options)
[17:24:08] ^-- Disable processing of calls to System.gc() (add
'-XX:+DisableExplicitGC' to JVM options)
[17:24:08] Refer to this page for more performance suggestions:
https://apacheignite.readme.io/docs/jvm-and-system-tuning
[17:24:08]
[17:24:08] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat}
[17:24:08]
[17:24:08] Ignite node started OK (id=991d7791)
[17:24:08] Topology snapshot [ver=1, servers=1, clients=0, CPUs=8, heap=4.0GB]
>>> Ignite started on MyLeakingNode (991d7791-5cc8-4765-82a4-6aff1a6e74c6)
>>> successfully!
Mar 11, 2018 11:35:28 PM org.apache.ignite.logger.java.JavaLogger error
SEVERE: TcpDiscoverSpi's message worker thread failed abnormally. Stopping the
node in order to prevent cluster wide instability.
java.lang.OutOfMemoryError: Java heap space
at
org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNodesRing.serverNodes(TcpDiscoveryNodesRing.java:608)
at
org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNodesRing.coordinator(TcpDiscoveryNodesRing.java:439)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl.resolveCoordinator(ServerImpl.java:1491)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl.resolveCoordinator(ServerImpl.java:1472)
at
org
[jira] [Commented] (IGNITE-7918) Huge memory leak when data streamer used together with local cache
[
https://issues.apache.org/jira/browse/IGNITE-7918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395086#comment-16395086
]
Zbyszek B commented on IGNITE-7918:
---
{code:java}
package leak;
import org.apache.ignite.*;
import org.apache.ignite.binary.BinaryObject;
import org.apache.ignite.binary.BinaryObjectBuilder;
import org.apache.ignite.cache.CacheMode;
import org.apache.ignite.cache.CachePeekMode;
import org.apache.ignite.cache.QueryEntity;
import org.apache.ignite.cluster.ClusterNode;
import org.apache.ignite.configuration.CacheConfiguration;
import org.apache.ignite.configuration.IgniteConfiguration;
import org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi;
import java.io.BufferedReader;
import java.io.File;
import java.io.InputStreamReader;
import java.text.SimpleDateFormat;
import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicLong;
import java.util.stream.Collectors;
import java.util.stream.IntStream;
import java.util.stream.Stream;
public class Demo {
private final static Ignite ignite = createIgnite();
private final static AtomicLong producedCount = new AtomicLong();
private final static AtomicLong consumedCount = new AtomicLong();
private final static int numOfLocalFields = 10;
private static BinaryObject localPrototype = createLocalPrototype(); //
prototype of local cache entity with all fields set to null
private final static int numOfGlobalFields = 5;
private final static List globalFields = globalFields();
private static BinaryObject globalPrototype = createGlobalPrototype(); //
prototype of cluster-wide cache entity with all fields set to null
private static final IgniteCache globalCache =
getOrCreateGlobalCache();
private final static String ID = "id";
private final static boolean useLocalCacheToCauseMemLeak = true;
private final static BlockingQueue queue = new
LinkedBlockingQueue<>(10);
public static void main(String[] args) throws Exception {
CompletableFuture.runAsync(Demo::runProducer);
CompletableFuture.runAsync(Demo::runConsumer);
BufferedReader br = new BufferedReader(new InputStreamReader(System.in));
do {
String input = br.readLine();
if ("q".equals(input)) {
System.out.println("Exit!");
System.exit(0);
} else {
String timestamp = new SimpleDateFormat("HH:mm:ss").format(new Date());
System.out.println(String.format("[%s] Queue size: %d", timestamp,
queue.size()));
System.out.println(String.format("[%s] Local caches produced: %d", timestamp,
producedCount.longValue()));
System.out.println(String.format("[%s] Local caches consumed %d", timestamp,
consumedCount.longValue()));
System.out.println(String.format("[%s] Global cache size: %d", timestamp,
globalCache.size(CachePeekMode.ALL)));
}
} while (true);
}
private static void runProducer() {
do {
long cacheId = producedCount.incrementAndGet();
try {
BinaryObject bObj = createLocalObject(1);
if (useLocalCacheToCauseMemLeak) {
IgniteCache cache = createLocalCache(cacheId);
cache.put(bObj.field(ID), bObj);
queue.put(cache);
} else {
Map cache = new HashMap<>();
cache.put(bObj.field(ID), bObj);
queue.put(cache);
}
} catch (Exception e) {
System.out.println(e.toString());
System.exit(1);
}
} while (true);
}
private static List getSQLSelectRow(BinaryObject localObject) {
List res = new ArrayList<>();
for (String f : globalFields) {
res.add(localObject.field(f));
}
return res;
}
@SuppressWarnings("unchecked")
private static void runConsumer() {
do {
try {
final Object taken = queue.take();
final List> globalSQLRows = new ArrayList<>(); // Note: In practice
rows are generated by executing SqlFieldsQuery against local cache; here
omitted for sake of simplicity.
if (taken instanceof IgniteCache) {
final IgniteCache cache = IgniteCache.class.cast(taken);
cache.forEach(e -> {
globalSQLRows.add(getSQLSelectRow(e.getValue()));
});
cache.destroy();
} else if (taken instanceof Map) {
final Map cache = Map.class.cast(taken);
cache.forEach((k, v) -> {
globalSQLRows.add(getSQLSelectRow(v));
});
}
consumedCount.incrementAndGet();
final IgniteDataStreamer globalStreamer =
createGlobalStreamer(globalCache);
for (List row : globalSQLRows) {
final BinaryObjectBuilder globalBuilder = globalPrototype.toBuilder();
for (int i = 0; i < globalFields.size(); i++) {
globalBuilder.setField(globalFields.get(i), row.get(i));
}
final BinaryObject gObj = globalBuilder.build();
globalStreamer.addData(gObj.field(ID), gObj);
}
globalStreamer.flush();
globalStreamer.close();
} catch (Exception e) {
System.out.println(e.toString());
System.exit(1);
}
} while (true);
}
private static Ignite createIgnite() {
IgniteConfiguration iCfg = new IgniteConfiguration();
TcpCommunicationSpi tcpCommunication = new TcpCommunicationSpi();
tcpCommunication.setMessageQueueLimit(1024); // to get rid of the warning
iCfg.setCommunicationSpi(tcpCommunication);
String workDirectory = System.getProperty("user.home") + File.separator
