[jira] [Commented] (IGNITE-10959) Memory leaks in continuous query handlers
[ https://issues.apache.org/jira/browse/IGNITE-10959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17156525#comment-17156525 ] Maxim Muzafarov commented on IGNITE-10959: -- Changes merged to 2.9. > Memory leaks in continuous query handlers > - > > Key: IGNITE-10959 > URL: https://issues.apache.org/jira/browse/IGNITE-10959 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.7 >Reporter: Denis Mekhanikov >Assignee: Maxim Muzafarov >Priority: Critical > Fix For: 2.9 > > Attachments: CacheContinuousQueryMemoryUsageTest.java, > CacheContinuousQueryMemoryUsageTest.result, > CacheContinuousQueryMemoryUsageTest2.java, > Memory_blowup_in_Ignite_CacheContinuousQueryHandler.txt, > Memory_blowup_in_Ignite_CacheContinuousQueryHandler.txt, > Memory_blowup_in_Ignite_CacheContinuousQueryHandler.txt, > continuousquery_leak_profile.png, referencepath.PNG > > Time Spent: 4.5h > Remaining Estimate: 0h > > Continuous query handlers don't clear internal data structures after cache > events are processed. > A test, that reproduces the problem, is attached. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-10959) Memory leaks in continuous query handlers
[ https://issues.apache.org/jira/browse/IGNITE-10959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155617#comment-17155617 ] Maxim Muzafarov commented on IGNITE-10959: -- [~alex_pl] Thank you for the review. I've merged changes to the master branch. I'll cherry-pick changes to the 2.9 branch at the beginning of the next week. > Memory leaks in continuous query handlers > - > > Key: IGNITE-10959 > URL: https://issues.apache.org/jira/browse/IGNITE-10959 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.7 >Reporter: Denis Mekhanikov >Assignee: Maxim Muzafarov >Priority: Critical > Fix For: 2.9 > > Attachments: CacheContinuousQueryMemoryUsageTest.java, > CacheContinuousQueryMemoryUsageTest.result, > CacheContinuousQueryMemoryUsageTest2.java, > Memory_blowup_in_Ignite_CacheContinuousQueryHandler.txt, > Memory_blowup_in_Ignite_CacheContinuousQueryHandler.txt, > Memory_blowup_in_Ignite_CacheContinuousQueryHandler.txt, > continuousquery_leak_profile.png, referencepath.PNG > > Time Spent: 4.5h > Remaining Estimate: 0h > > Continuous query handlers don't clear internal data structures after cache > events are processed. > A test, that reproduces the problem, is attached. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-10959) Memory leaks in continuous query handlers
[ https://issues.apache.org/jira/browse/IGNITE-10959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17154287#comment-17154287 ] Aleksey Plekhanov commented on IGNITE-10959: [~mmuzaf] looks good to me. > Memory leaks in continuous query handlers > - > > Key: IGNITE-10959 > URL: https://issues.apache.org/jira/browse/IGNITE-10959 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.7 >Reporter: Denis Mekhanikov >Assignee: Maxim Muzafarov >Priority: Critical > Fix For: 2.9 > > Attachments: CacheContinuousQueryMemoryUsageTest.java, > CacheContinuousQueryMemoryUsageTest.result, > CacheContinuousQueryMemoryUsageTest2.java, > Memory_blowup_in_Ignite_CacheContinuousQueryHandler.txt, > Memory_blowup_in_Ignite_CacheContinuousQueryHandler.txt, > Memory_blowup_in_Ignite_CacheContinuousQueryHandler.txt, > continuousquery_leak_profile.png, referencepath.PNG > > Time Spent: 4h 20m > Remaining Estimate: 0h > > Continuous query handlers don't clear internal data structures after cache > events are processed. > A test, that reproduces the problem, is attached. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-10959) Memory leaks in continuous query handlers
[
https://issues.apache.org/jira/browse/IGNITE-10959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152579#comment-17152579
]
Ignite TC Bot commented on IGNITE-10959:
{panel:title=Branch: [pull/7881/head] Base: [master] : No blockers
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}{panel}
{panel:title=Branch: [pull/7881/head] Base: [master] : New Tests
(20)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}
{color:#8b}Service Grid{color} [tests 4]
* {color:#013220}IgniteServiceGridTestSuite:
ServiceDeploymentProcessIdSelfTest.topologyVersion[Test event=IgniteBiTuple
[val1=DiscoveryEvent [evtNode=44bf7bae-2206-4ed1-82a1-7a7db33b43f7, topVer=0,
nodeId8=5a2eebcf, msg=, type=NODE_JOINED, tstamp=1594066169735],
val2=AffinityTopologyVersion [topVer=6482430669873845675, minorTopVer=0]]] -
PASSED{color}
* {color:#013220}IgniteServiceGridTestSuite:
ServiceDeploymentProcessIdSelfTest.requestId[Test event=IgniteBiTuple
[val1=DiscoveryEvent [evtNode=44bf7bae-2206-4ed1-82a1-7a7db33b43f7, topVer=0,
nodeId8=5a2eebcf, msg=, type=NODE_JOINED, tstamp=1594066169735],
val2=AffinityTopologyVersion [topVer=6482430669873845675, minorTopVer=0]]] -
PASSED{color}
* {color:#013220}IgniteServiceGridTestSuite:
ServiceDeploymentProcessIdSelfTest.topologyVersion[Test event=IgniteBiTuple
[val1=DiscoveryCustomEvent [customMsg=ServiceChangeBatchRequest
[id=dc8a9c52371-a2b5ad3a-a360-4a0d-a45e-eabd4760e8a1, reqs=SingletonList
[ServiceUndeploymentRequest []]], affTopVer=null, super=DiscoveryEvent
[evtNode=674b2a47-fbf0-43bf-873e-8aea010c6927, topVer=0, nodeId8=674b2a47,
msg=null, type=DISCOVERY_CUSTOM_EVT, tstamp=1594066169735]],
val2=AffinityTopologyVersion [topVer=7429373169187075070, minorTopVer=0]]] -
PASSED{color}
* {color:#013220}IgniteServiceGridTestSuite:
ServiceDeploymentProcessIdSelfTest.requestId[Test event=IgniteBiTuple
[val1=DiscoveryCustomEvent [customMsg=ServiceChangeBatchRequest
[id=dc8a9c52371-a2b5ad3a-a360-4a0d-a45e-eabd4760e8a1, reqs=SingletonList
[ServiceUndeploymentRequest []]], affTopVer=null, super=DiscoveryEvent
[evtNode=674b2a47-fbf0-43bf-873e-8aea010c6927, topVer=0, nodeId8=674b2a47,
msg=null, type=DISCOVERY_CUSTOM_EVT, tstamp=1594066169735]],
val2=AffinityTopologyVersion [topVer=7429373169187075070, minorTopVer=0]]] -
PASSED{color}
{color:#8b}Service Grid (legacy mode){color} [tests 4]
* {color:#013220}IgniteServiceGridTestSuite:
ServiceDeploymentProcessIdSelfTest.topologyVersion[Test event=IgniteBiTuple
[val1=DiscoveryEvent [evtNode=b7f0b615-1cc4-4550-af55-c95af8d47ac7, topVer=0,
nodeId8=4aff7685, msg=, type=NODE_JOINED, tstamp=1594066260935],
val2=AffinityTopologyVersion [topVer=-8678113640998879127, minorTopVer=0]]] -
PASSED{color}
* {color:#013220}IgniteServiceGridTestSuite:
ServiceDeploymentProcessIdSelfTest.requestId[Test event=IgniteBiTuple
[val1=DiscoveryEvent [evtNode=b7f0b615-1cc4-4550-af55-c95af8d47ac7, topVer=0,
nodeId8=4aff7685, msg=, type=NODE_JOINED, tstamp=1594066260935],
val2=AffinityTopologyVersion [topVer=-8678113640998879127, minorTopVer=0]]] -
PASSED{color}
* {color:#013220}IgniteServiceGridTestSuite:
ServiceDeploymentProcessIdSelfTest.topologyVersion[Test event=IgniteBiTuple
[val1=DiscoveryCustomEvent [customMsg=ServiceChangeBatchRequest
[id=bc3d0c52371-12c4d2a2-2bff-47c6-9df2-45f29305345f, reqs=SingletonList
[ServiceUndeploymentRequest []]], affTopVer=null, super=DiscoveryEvent
[evtNode=6e1cb0a6-cb58-4e86-99df-280864fd66fd, topVer=0, nodeId8=6e1cb0a6,
msg=null, type=DISCOVERY_CUSTOM_EVT, tstamp=1594066260935]],
val2=AffinityTopologyVersion [topVer=4148470245918002258, minorTopVer=0]]] -
PASSED{color}
* {color:#013220}IgniteServiceGridTestSuite:
ServiceDeploymentProcessIdSelfTest.requestId[Test event=IgniteBiTuple
[val1=DiscoveryCustomEvent [customMsg=ServiceChangeBatchRequest
[id=bc3d0c52371-12c4d2a2-2bff-47c6-9df2-45f29305345f, reqs=SingletonList
[ServiceUndeploymentRequest []]], affTopVer=null, super=DiscoveryEvent
[evtNode=6e1cb0a6-cb58-4e86-99df-280864fd66fd, topVer=0, nodeId8=6e1cb0a6,
msg=null, type=DISCOVERY_CUSTOM_EVT, tstamp=1594066260935]],
val2=AffinityTopologyVersion [topVer=4148470245918002258, minorTopVer=0]]] -
PASSED{color}
{color:#8b}Continuous Query 1{color} [tests 12]
* {color:#013220}IgniteCacheQuerySelfTestSuite3:
CacheContinuousQueryBufferLimitTest.testContinuousQueryBatchSwitchOnAck[cacheMode=PARTITIONED,
atomicityMode=TRANSACTIONAL] - PASSED{color}
* {color:#013220}IgniteCacheQuerySelfTestSuite3:
CacheContinuousQueryBufferLimitTest.testContinuousQueryPendingBufferLimit[cacheMode=PARTITIONED,
atomicityMode=TRANSACTIONAL] - PASSED{color}
* {color:#013220}IgniteCacheQuerySelfTestSuite3:
CacheContinuousQueryBufferLimitTest.testContinuousQueryBatchSwitchOnAck[cacheMode=PARTITIONED,
atomicityMode=ATOMIC] - PASSED{color}
* {color:#013220}IgniteCacheQuerySelfTestSuite3:
[jira] [Commented] (IGNITE-10959) Memory leaks in continuous query handlers
[ https://issues.apache.org/jira/browse/IGNITE-10959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126689#comment-17126689 ] Maxim Muzafarov commented on IGNITE-10959: -- [~alex_pl], can you review my changes, please? > Memory leaks in continuous query handlers > - > > Key: IGNITE-10959 > URL: https://issues.apache.org/jira/browse/IGNITE-10959 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.7 >Reporter: Denis Mekhanikov >Assignee: Maxim Muzafarov >Priority: Critical > Fix For: 2.9 > > Attachments: CacheContinuousQueryMemoryUsageTest.java, > CacheContinuousQueryMemoryUsageTest.result, > CacheContinuousQueryMemoryUsageTest2.java, > Memory_blowup_in_Ignite_CacheContinuousQueryHandler.txt, > Memory_blowup_in_Ignite_CacheContinuousQueryHandler.txt, > Memory_blowup_in_Ignite_CacheContinuousQueryHandler.txt, > continuousquery_leak_profile.png, referencepath.PNG > > Time Spent: 10m > Remaining Estimate: 0h > > Continuous query handlers don't clear internal data structures after cache > events are processed. > A test, that reproduces the problem, is attached. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-10959) Memory leaks in continuous query handlers
[
https://issues.apache.org/jira/browse/IGNITE-10959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126687#comment-17126687
]
Ignite TC Bot commented on IGNITE-10959:
{panel:title=Branch: [pull/7881/head] Base: [master] : No blockers
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}{panel}
[TeamCity *-- Run :: All*
Results|https://ci.ignite.apache.org/viewLog.html?buildId=5364933buildTypeId=IgniteTests24Java8_RunAll]
> Memory leaks in continuous query handlers
> -
>
> Key: IGNITE-10959
> URL: https://issues.apache.org/jira/browse/IGNITE-10959
> Project: Ignite
> Issue Type: Bug
>Affects Versions: 2.7
>Reporter: Denis Mekhanikov
>Assignee: Maxim Muzafarov
>Priority: Critical
> Fix For: 2.9
>
> Attachments: CacheContinuousQueryMemoryUsageTest.java,
> CacheContinuousQueryMemoryUsageTest.result,
> CacheContinuousQueryMemoryUsageTest2.java,
> Memory_blowup_in_Ignite_CacheContinuousQueryHandler.txt,
> Memory_blowup_in_Ignite_CacheContinuousQueryHandler.txt,
> Memory_blowup_in_Ignite_CacheContinuousQueryHandler.txt,
> continuousquery_leak_profile.png, referencepath.PNG
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Continuous query handlers don't clear internal data structures after cache
> events are processed.
> A test, that reproduces the problem, is attached.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IGNITE-10959) Memory leaks in continuous query handlers
[ https://issues.apache.org/jira/browse/IGNITE-10959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17121138#comment-17121138 ] Maxim Muzafarov commented on IGNITE-10959: -- Folks, I can confirm the issue with the `pending` map overflow mentioned above. PR with reproducer (PR #7881) attached. I will try to fix it. > Memory leaks in continuous query handlers > - > > Key: IGNITE-10959 > URL: https://issues.apache.org/jira/browse/IGNITE-10959 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.7 >Reporter: Denis Mekhanikov >Assignee: Maxim Muzafarov >Priority: Critical > Attachments: CacheContinuousQueryMemoryUsageTest.java, > CacheContinuousQueryMemoryUsageTest.result, > CacheContinuousQueryMemoryUsageTest2.java, > Memory_blowup_in_Ignite_CacheContinuousQueryHandler.txt, > Memory_blowup_in_Ignite_CacheContinuousQueryHandler.txt, > Memory_blowup_in_Ignite_CacheContinuousQueryHandler.txt, > continuousquery_leak_profile.png, referencepath.PNG > > Time Spent: 10m > Remaining Estimate: 0h > > Continuous query handlers don't clear internal data structures after cache > events are processed. > A test, that reproduces the problem, is attached. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-10959) Memory leaks in continuous query handlers
[ https://issues.apache.org/jira/browse/IGNITE-10959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118854#comment-17118854 ] Maxim Muzafarov commented on IGNITE-10959: -- [~cguo1] Yes, not all issues resolved by IGNITE-11970 which were mention above. I'll check the rest of them. Thank you for the details. > Memory leaks in continuous query handlers > - > > Key: IGNITE-10959 > URL: https://issues.apache.org/jira/browse/IGNITE-10959 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.7 >Reporter: Denis Mekhanikov >Assignee: Maxim Muzafarov >Priority: Critical > Attachments: CacheContinuousQueryMemoryUsageTest.java, > CacheContinuousQueryMemoryUsageTest.result, > CacheContinuousQueryMemoryUsageTest2.java, > Memory_blowup_in_Ignite_CacheContinuousQueryHandler.txt, > Memory_blowup_in_Ignite_CacheContinuousQueryHandler.txt, > Memory_blowup_in_Ignite_CacheContinuousQueryHandler.txt, > continuousquery_leak_profile.png, referencepath.PNG > > > Continuous query handlers don't clear internal data structures after cache > events are processed. > A test, that reproduces the problem, is attached. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-10959) Memory leaks in continuous query handlers
[ https://issues.apache.org/jira/browse/IGNITE-10959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118803#comment-17118803 ] Cong Guo commented on IGNITE-10959: --- Hi, Maxim In my case, the fix under IGNITE-11970 does not solve the problem. The entries are stored in the pending buffer because the current batch does not move on. It is not a waste like in IGNITE-11970. It is a memory leak. There is no limit on the pending buffer. According to another comment above, this issue is relevant to the sync mode and the backup number. I did not test all the cases, but you can easily reproduce this issue using three Ignite node with Transactional + Partition + 2 backups. # Start a 3-node Ignite cluster, and each node has a continuous query to watch updates in cache. # Send updates to three nodes continuously. Simple puts are OK. # Restart one Ignite node, and then memory leak can be seen on two other nodes. I attach the figure showing the accumulation point and references in my experiment here. !referencepath.PNG! > Memory leaks in continuous query handlers > - > > Key: IGNITE-10959 > URL: https://issues.apache.org/jira/browse/IGNITE-10959 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.7 >Reporter: Denis Mekhanikov >Assignee: Maxim Muzafarov >Priority: Critical > Attachments: CacheContinuousQueryMemoryUsageTest.java, > CacheContinuousQueryMemoryUsageTest.result, > CacheContinuousQueryMemoryUsageTest2.java, > Memory_blowup_in_Ignite_CacheContinuousQueryHandler.txt, > Memory_blowup_in_Ignite_CacheContinuousQueryHandler.txt, > Memory_blowup_in_Ignite_CacheContinuousQueryHandler.txt, > continuousquery_leak_profile.png, referencepath.PNG > > > Continuous query handlers don't clear internal data structures after cache > events are processed. > A test, that reproduces the problem, is attached. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-10959) Memory leaks in continuous query handlers
[ https://issues.apache.org/jira/browse/IGNITE-10959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118550#comment-17118550 ] Maxim Muzafarov commented on IGNITE-10959: -- Folks, Some of the memory consumption issues were already fixed under IGNITE-10959 by [~ibelyakov] (don't be confused with the `open` status, changes already exist). I hope those changes will be merged on the next week. I'll take care of the left cases right after IGNITE-10959 will be merged. > Memory leaks in continuous query handlers > - > > Key: IGNITE-10959 > URL: https://issues.apache.org/jira/browse/IGNITE-10959 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.7 >Reporter: Denis Mekhanikov >Assignee: Maxim Muzafarov >Priority: Critical > Attachments: CacheContinuousQueryMemoryUsageTest.java, > CacheContinuousQueryMemoryUsageTest.result, > CacheContinuousQueryMemoryUsageTest2.java, > Memory_blowup_in_Ignite_CacheContinuousQueryHandler.txt, > Memory_blowup_in_Ignite_CacheContinuousQueryHandler.txt, > Memory_blowup_in_Ignite_CacheContinuousQueryHandler.txt, > continuousquery_leak_profile.png > > > Continuous query handlers don't clear internal data structures after cache > events are processed. > A test, that reproduces the problem, is attached. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-10959) Memory leaks in continuous query handlers
[ https://issues.apache.org/jira/browse/IGNITE-10959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118447#comment-17118447 ] Ivan Pavlukhin commented on IGNITE-10959: - [~mmuzaf], unfortunately I cannot help with GG tickets nowadays. [~jooger], [~tledkov-gridgain], [~amashenkov] possibly can help here. > Memory leaks in continuous query handlers > - > > Key: IGNITE-10959 > URL: https://issues.apache.org/jira/browse/IGNITE-10959 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.7 >Reporter: Denis Mekhanikov >Assignee: Maxim Muzafarov >Priority: Critical > Attachments: CacheContinuousQueryMemoryUsageTest.java, > CacheContinuousQueryMemoryUsageTest.result, > CacheContinuousQueryMemoryUsageTest2.java, > Memory_blowup_in_Ignite_CacheContinuousQueryHandler.txt, > Memory_blowup_in_Ignite_CacheContinuousQueryHandler.txt, > Memory_blowup_in_Ignite_CacheContinuousQueryHandler.txt, > continuousquery_leak_profile.png > > > Continuous query handlers don't clear internal data structures after cache > events are processed. > A test, that reproduces the problem, is attached. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-10959) Memory leaks in continuous query handlers
[
https://issues.apache.org/jira/browse/IGNITE-10959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117908#comment-17117908
]
Maxim Muzafarov commented on IGNITE-10959:
--
[~Pavlukhin]
Hello, I've found some code changes probably related to the current issue in
GGCE (suppose associated with GG-17432). At first glance, they related to
memory consumption.
I'm not diving deep into those details yet, but can you please share some
details at those fix? I'm going to continue the investigation, should I?
Are there any corner cases left uncovered? I've applied those changes and
checked the attached reproducer {{CacheContinuousQueryMemoryUsageTest}}
locally, it seems it still fails (the \{{backupQ}} remains uncleared).
> Memory leaks in continuous query handlers
> -
>
> Key: IGNITE-10959
> URL: https://issues.apache.org/jira/browse/IGNITE-10959
> Project: Ignite
> Issue Type: Bug
>Affects Versions: 2.7
>Reporter: Denis Mekhanikov
>Assignee: Maxim Muzafarov
>Priority: Critical
> Attachments: CacheContinuousQueryMemoryUsageTest.java,
> CacheContinuousQueryMemoryUsageTest.result,
> CacheContinuousQueryMemoryUsageTest2.java,
> Memory_blowup_in_Ignite_CacheContinuousQueryHandler.txt,
> Memory_blowup_in_Ignite_CacheContinuousQueryHandler.txt,
> Memory_blowup_in_Ignite_CacheContinuousQueryHandler.txt,
> continuousquery_leak_profile.png
>
>
> Continuous query handlers don't clear internal data structures after cache
> events are processed.
> A test, that reproduces the problem, is attached.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IGNITE-10959) Memory leaks in continuous query handlers
[ https://issues.apache.org/jira/browse/IGNITE-10959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17114194#comment-17114194 ] Zane Hu commented on IGNITE-10959: -- As I commented before, there are two confirmed cases of memory blowup in Ignite, which are caused by too many cache update events in pending buffers since an older update event than the buffered pending events has not arrived yet. # One is per-partition TreeMap CacheContinuousQueryPartitionRecovery.pendingEvts in CacheContinuousQueryHandler.rcvs. It has a upper-bound prevention of MAX_BUFF_SIZE. # Another is per-partition ConcurrentSkipListMap CacheContinuousQueryEventBuffer.pending. It has no upper-bound prevention on CacheContinuousQueryEventBuffer.pending. Attached below is a pseudo code of the main logic flow of how they are processed in Ignite. Hope it can help people to fix the problem. They all started in CacheContinuousQueryHandler.CacheContinuousQueryListener.onEntryUpdated() when a cache entry is updated. [^Memory_blowup_in_Ignite_CacheContinuousQueryHandler.txt] > Memory leaks in continuous query handlers > - > > Key: IGNITE-10959 > URL: https://issues.apache.org/jira/browse/IGNITE-10959 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.7 >Reporter: Denis Mekhanikov >Assignee: Maxim Muzafarov >Priority: Critical > Attachments: CacheContinuousQueryMemoryUsageTest.java, > CacheContinuousQueryMemoryUsageTest.result, > CacheContinuousQueryMemoryUsageTest2.java, > Memory_blowup_in_Ignite_CacheContinuousQueryHandler.txt, > Memory_blowup_in_Ignite_CacheContinuousQueryHandler.txt, > Memory_blowup_in_Ignite_CacheContinuousQueryHandler.txt, > continuousquery_leak_profile.png > > > Continuous query handlers don't clear internal data structures after cache > events are processed. > A test, that reproduces the problem, is attached. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-10959) Memory leaks in continuous query handlers
[ https://issues.apache.org/jira/browse/IGNITE-10959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17114107#comment-17114107 ] Grey Guo commented on IGNITE-10959: --- Similar as [~chinhodado] , we are very to close to remove Ignite from our product technology stack > Memory leaks in continuous query handlers > - > > Key: IGNITE-10959 > URL: https://issues.apache.org/jira/browse/IGNITE-10959 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.7 >Reporter: Denis Mekhanikov >Assignee: Maxim Muzafarov >Priority: Critical > Attachments: CacheContinuousQueryMemoryUsageTest.java, > CacheContinuousQueryMemoryUsageTest.result, > CacheContinuousQueryMemoryUsageTest2.java, continuousquery_leak_profile.png > > > Continuous query handlers don't clear internal data structures after cache > events are processed. > A test, that reproduces the problem, is attached. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-10959) Memory leaks in continuous query handlers
[ https://issues.apache.org/jira/browse/IGNITE-10959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113831#comment-17113831 ] Maxim Muzafarov commented on IGNITE-10959: -- Folks, This issue was completely out of our radar. It's sad to hear that it affects so use cases. I'll try to investigate the issue shortly, right after finishing [IEP-43: Cluster snapshots|https://cwiki.apache.org/confluence/display/IGNITE/IEP-43%3A+Cluster+snapshots] retated tasks. Will keep you it touch. > Memory leaks in continuous query handlers > - > > Key: IGNITE-10959 > URL: https://issues.apache.org/jira/browse/IGNITE-10959 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.7 >Reporter: Denis Mekhanikov >Assignee: Maxim Muzafarov >Priority: Critical > Attachments: CacheContinuousQueryMemoryUsageTest.java, > CacheContinuousQueryMemoryUsageTest.result, > CacheContinuousQueryMemoryUsageTest2.java, continuousquery_leak_profile.png > > > Continuous query handlers don't clear internal data structures after cache > events are processed. > A test, that reproduces the problem, is attached. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-10959) Memory leaks in continuous query handlers
[ https://issues.apache.org/jira/browse/IGNITE-10959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113502#comment-17113502 ] chin commented on IGNITE-10959: --- I just want to say that we ran into OOM with ContinuousQuery constantly, likely because of this bug, with no way around it, so in the end we were forced to abandon Ignite and moved to Hazelcast instead, which is a shame. > Memory leaks in continuous query handlers > - > > Key: IGNITE-10959 > URL: https://issues.apache.org/jira/browse/IGNITE-10959 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.7 >Reporter: Denis Mekhanikov >Assignee: Maxim Muzafarov >Priority: Critical > Attachments: CacheContinuousQueryMemoryUsageTest.java, > CacheContinuousQueryMemoryUsageTest.result, > CacheContinuousQueryMemoryUsageTest2.java, continuousquery_leak_profile.png > > > Continuous query handlers don't clear internal data structures after cache > events are processed. > A test, that reproduces the problem, is attached. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-10959) Memory leaks in continuous query handlers
[ https://issues.apache.org/jira/browse/IGNITE-10959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113213#comment-17113213 ] Cong Guo commented on IGNITE-10959: --- Hi, Is this issue still open? Is there any patch to fix this issue? I can reproduce this issue by restarting one node in a cluster. # Start a 3-node Ignite cluster, and each node has a continuous query to watch updates in cache. # Send updates to three nodes continuously. # Restart one Ignite node, and then memory leak can be seen on two other nodes. Based on the heap dump, the memory issue happens in entryBufs of CacheContinuousQueryHandler. I already close the continuous query when I stop the Ignite node. However, the pending buffer on two other nodes still increases when the node starts again. Is there any method to discard the pending buffer when one node (the master node of a continuous query) leaves the cluster? !ObjectRef.PNG! > Memory leaks in continuous query handlers > - > > Key: IGNITE-10959 > URL: https://issues.apache.org/jira/browse/IGNITE-10959 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.7 >Reporter: Denis Mekhanikov >Priority: Major > Fix For: 2.9 > > Attachments: CacheContinuousQueryMemoryUsageTest.java, > CacheContinuousQueryMemoryUsageTest.result, > CacheContinuousQueryMemoryUsageTest2.java, continuousquery_leak_profile.png > > > Continuous query handlers don't clear internal data structures after cache > events are processed. > A test, that reproduces the problem, is attached. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-10959) Memory leaks in continuous query handlers
[ https://issues.apache.org/jira/browse/IGNITE-10959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973903#comment-16973903 ] Zane Hu commented on IGNITE-10959: -- In addition to having an upper-bound limit to flush and remove 10% events out from CacheContinuousQueryEventBuffer.pending if the limit is reached, it would be nice to inform the app somehow that at least one of the earlier events than the flushed 10% of pending events has dropped because it has not arrived in time. This way, the app may have a chance to handle such exception afterwards, for example, by doing a full scan of the partition to which the dropped event belongs if possible. We would like to handle such exception for both CacheContinuousQueryPartitionRecovery.pendingEvts and CacheContinuousQueryEventBuffer.pending. > Memory leaks in continuous query handlers > - > > Key: IGNITE-10959 > URL: https://issues.apache.org/jira/browse/IGNITE-10959 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.7 >Reporter: Denis Mekhanikov >Priority: Major > Fix For: 2.9 > > Attachments: CacheContinuousQueryMemoryUsageTest.java, > CacheContinuousQueryMemoryUsageTest.result, > CacheContinuousQueryMemoryUsageTest2.java, continuousquery_leak_profile.png > > > Continuous query handlers don't clear internal data structures after cache > events are processed. > A test, that reproduces the problem, is attached. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-10959) Memory leaks in continuous query handlers
[ https://issues.apache.org/jira/browse/IGNITE-10959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967866#comment-16967866 ] Zane Hu commented on IGNITE-10959: -- We have observed two cases of using huge amount of memory in Ignite Continuous Query, which both are caused by too many pending cache-update events since an earlier event than the pending events has not arrived yet. BTW, we use Ignite 2.7.0. * One is CacheContinuousQueryHandler.rcvs growing to 7.7 GB Retained Heap, seen in Jmap/Memory Analyzer. Also we saw "Pending events reached max of buffer size" in Ignite log file. According to [https://github.com/apache/ignite/blob/ignite-2.7/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/query/continuous/CacheContinuousQueryPartitionRecovery.java#L196], it is because the size of CacheContinuousQueryPartitionRecovery.pendingEvts >= MAX_BUFF_SIZE, (default 10,000). And Ignite will flush and remove 10% of the entries in the pendingEvts, regardless some unarrived early events are dropped without notifying the listener. This upper-bound limit of MAX_BUFF_SIZE prevents the memory from further growing to OOM. * Another is CacheContinuousQueryEventBuffer.pending growing to 22 GB Retained Heap, seen in Jmap/Memory Analyzer. According to [https://github.com/apache/ignite/blob/ignite-2.7/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/query/continuous/CacheContinuousQueryEventBuffer.java#L168], the cache-update events are processed in batch of CacheContinuousQueryEventBuffer.Batch.entries[BUF_SIZE] (default BUF_SIZE is 1,000). If an event entry is within the current batch (e.updateCounter() <= batch.endCntr), it is processed by batch.processEntry0(). Otherwise it is put in CacheContinuousQueryEventBuffer.pending. However, according to [https://github.com/apache/ignite/blob/ignite-2.7/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/query/continuous/CacheContinuousQueryEventBuffer.java#L425], if any event within the current batch has not arrived, Ignite will not move to the next batch to process entries in CacheContinuousQueryEventBuffer.pending. Differently from processing in CacheContinuousQueryPartitionRecovery.pendingEvts and MAX_BUFF_SIZE, there is NO upper-bound limit on CacheContinuousQueryEventBuffer.pending. It means if an earlier event than the events in CacheContinuousQueryEventBuffer.pending never comes for some reason (high frequency of lots of events, high concurrency, timeout, ...), CacheContinuousQueryEventBuffer.pending will grow to OOM. To prevent this, I think Ignite at least needs to add an upper-bound limit and some processing here, to flush and remove 10% events out from CacheContinuousQueryEventBuffer.pending, similarly to CacheContinuousQueryPartitionRecovery.pendingEvts. In terms of exception handling, I think dropping some events is better than OOM. > Memory leaks in continuous query handlers > - > > Key: IGNITE-10959 > URL: https://issues.apache.org/jira/browse/IGNITE-10959 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.7 >Reporter: Denis Mekhanikov >Priority: Major > Fix For: 2.9 > > Attachments: CacheContinuousQueryMemoryUsageTest.java, > CacheContinuousQueryMemoryUsageTest.result, > CacheContinuousQueryMemoryUsageTest2.java, continuousquery_leak_profile.png > > > Continuous query handlers don't clear internal data structures after cache > events are processed. > A test, that reproduces the problem, is attached. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-10959) Memory leaks in continuous query handlers
[ https://issues.apache.org/jira/browse/IGNITE-10959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16958202#comment-16958202 ] Zane Hu commented on IGNITE-10959: -- The test result of [^CacheContinuousQueryMemoryUsageTest.result] if done using the following program with Ignite 2.7.0. [^CacheContinuousQueryMemoryUsageTest2.java] > Memory leaks in continuous query handlers > - > > Key: IGNITE-10959 > URL: https://issues.apache.org/jira/browse/IGNITE-10959 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.7 >Reporter: Denis Mekhanikov >Priority: Major > Fix For: 2.9 > > Attachments: CacheContinuousQueryMemoryUsageTest.java, > CacheContinuousQueryMemoryUsageTest.result, > CacheContinuousQueryMemoryUsageTest2.java, continuousquery_leak_profile.png > > > Continuous query handlers don't clear internal data structures after cache > events are processed. > A test, that reproduces the problem, is attached. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-10959) Memory leaks in continuous query handlers
[ https://issues.apache.org/jira/browse/IGNITE-10959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16957176#comment-16957176 ] Zane Hu commented on IGNITE-10959: -- Here is the test result. [^CacheContinuousQueryMemoryUsageTest.result] > Memory leaks in continuous query handlers > - > > Key: IGNITE-10959 > URL: https://issues.apache.org/jira/browse/IGNITE-10959 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.7 >Reporter: Denis Mekhanikov >Priority: Major > Fix For: 2.9 > > Attachments: CacheContinuousQueryMemoryUsageTest.java, > CacheContinuousQueryMemoryUsageTest.result, continuousquery_leak_profile.png > > > Continuous query handlers don't clear internal data structures after cache > events are processed. > A test, that reproduces the problem, is attached. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-10959) Memory leaks in continuous query handlers
[ https://issues.apache.org/jira/browse/IGNITE-10959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956149#comment-16956149 ] Grey Guo commented on IGNITE-10959: --- |CacheConfig|NoBackup|OneBackup|TwoBackup|Replicated| |AtomicPartitionedPrimarySync|Passed|Failed|Failed|NA| |AtomicPartitionedFullSync|Passed|Failed|Failed|NA| |AtomicReplicatedPrimarySync|NA|NA|NA|Failed| |AtomicReplicateFullSync|NA|NA|NA|Failed| |LocalAtomicPartitionedPrimarySync|Passed|Failed|Failed|NA| |LocalAtomicPartitionedFullSync|Passed|Failed|Failed|NA| |LocalAtomicReplicatedPrimarySync|NA|NA|NA|Passed| |LocalAtomicReplicateFullSync|NA|NA|NA|Passed| |TransactionPartitionedPrimarySync|Passed|Passed|Failed|NA| |TransactionPartitionedFullSync|Passed|Passed|Failed|NA| |TransactionReplicatedPrimarySync|NA|NA|NA|Failed| |TransactionReplicatedFullSync|NA|NA|NA|Failed| |LocalTransactionPartitionedPrimarySync|Passed|Passed|Failed|NA| |LocalTransactionPartitionedFullSync|Passed|Passed|Failed|NA| |LocalTransactionReplicatedPrimarySync|NA|NA|NA|Passed| |LocalTransactionReplicatedFullSync|NA|NA|NA|Passed| This is our testing > Memory leaks in continuous query handlers > - > > Key: IGNITE-10959 > URL: https://issues.apache.org/jira/browse/IGNITE-10959 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.7 >Reporter: Denis Mekhanikov >Priority: Major > Fix For: 2.9 > > Attachments: CacheContinuousQueryMemoryUsageTest.java, > continuousquery_leak_profile.png > > > Continuous query handlers don't clear internal data structures after cache > events are processed. > A test, that reproduces the problem, is attached. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-10959) Memory leaks in continuous query handlers
[ https://issues.apache.org/jira/browse/IGNITE-10959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956101#comment-16956101 ] Trung commented on IGNITE-10959: I don't think the problem is related to backups. I have tested with * single member grid, and * multi-member grid in PARTITIONED mode with no backup In both cases, the leak was still there. > Memory leaks in continuous query handlers > - > > Key: IGNITE-10959 > URL: https://issues.apache.org/jira/browse/IGNITE-10959 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.7 >Reporter: Denis Mekhanikov >Priority: Major > Fix For: 2.9 > > Attachments: CacheContinuousQueryMemoryUsageTest.java, > continuousquery_leak_profile.png > > > Continuous query handlers don't clear internal data structures after cache > events are processed. > A test, that reproduces the problem, is attached. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-10959) Memory leaks in continuous query handlers
[
https://issues.apache.org/jira/browse/IGNITE-10959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955345#comment-16955345
]
Zane Hu commented on IGNITE-10959:
--
An error case TransactionalPartitionedTwoBackupFullSync is as the following log
we got from a slightly modified CacheContinuousQueryMemoryUsageTest.java:
[ERROR]
CacheContinuousQueryMemoryUsageTest>GridAbstractTest.access$000:143->GridAbstractTest.runTestInternal:2177->testTransactionalPartitionedTwoBackupFullSync:235->testContinuousQuery:355->assertEntriesReleased:423->assertEntriesReleased:435->checkEntryBuffers:466
Backup queue is not empty. Node:
continuous.CacheContinuousQueryMemoryUsageTest0; cache: test-cache.
expected:<0> but was:<1>
But we don't see such error for TransactionalReplicatedTwoBackupFullSync or
TransactionalPartitionedOneBackupFullSync
Looked at Ignite code, the following snip of onEntryUpdated() in
CacheContinuousQueryHandler.java
{code:java}
if (primary || skipPrimaryCheck) //
TransactionalReplicatedTwoBackupFullSync goes here
onEntryUpdate(evt, notify, loc, recordIgniteEvt); // Notify the query
client without putting evt.entry() into backupQ.
else // A backup node of TransactionalPartitionedTwoBackupFullSync goes here
handleBackupEntry(cctx, evt.entry()); // This will put evt.entry() into
backupQ
{code}
After notifying the query client, there seems an ack msg
CacheContinuousQueryBatchAck sent to the CQ server side on backup nodes to
clean up the entries in backupQ. And there is even a periodic BackupCleaner
task every 5 seconds to clean up backupQ. The actual cleanup code is as below:
{code:java}
/**
* @param updateCntr Acknowledged counter.
*/
void cleanupBackupQueue(Long updateCntr) {
Iterator it = backupQ.iterator();
while (it.hasNext()) {
CacheContinuousQueryEntry backupEntry = it.next();
if (backupEntry.updateCounter() <= updateCntr) // Remove
backupEntry if its updateCounter <= Ack updateCntr
it.remove();
}
}
{code}
So some questions are
# Why is a backupEntry still left over in backupQ after all these?
# Is it possible that the updateCounter and Ack updateCntr are mis-calculated?
# Is it possible that the ack msg is sent to only one of the two backup nodes?
The load of 1000 updates of 3 nodes in a stable network, so there shouldn't be
a msg somehow dropped in the middle.
Please help to look into more, especially from Ignite experts or developers.
Thanks,
> Memory leaks in continuous query handlers
> -
>
> Key: IGNITE-10959
> URL: https://issues.apache.org/jira/browse/IGNITE-10959
> Project: Ignite
> Issue Type: Bug
>Affects Versions: 2.7
>Reporter: Denis Mekhanikov
>Priority: Major
> Fix For: 2.9
>
> Attachments: CacheContinuousQueryMemoryUsageTest.java,
> continuousquery_leak_profile.png
>
>
> Continuous query handlers don't clear internal data structures after cache
> events are processed.
> A test, that reproduces the problem, is attached.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IGNITE-10959) Memory leaks in continuous query handlers
[ https://issues.apache.org/jira/browse/IGNITE-10959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16954721#comment-16954721 ] Zane Hu commented on IGNITE-10959: -- We hit this issue too. Is it possible to have a quick fix patch soon? Thanks! > Memory leaks in continuous query handlers > - > > Key: IGNITE-10959 > URL: https://issues.apache.org/jira/browse/IGNITE-10959 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.7 >Reporter: Denis Mekhanikov >Priority: Major > Fix For: 2.9 > > Attachments: CacheContinuousQueryMemoryUsageTest.java, > continuousquery_leak_profile.png > > > Continuous query handlers don't clear internal data structures after cache > events are processed. > A test, that reproduces the problem, is attached. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-10959) Memory leaks in continuous query handlers
[ https://issues.apache.org/jira/browse/IGNITE-10959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16954690#comment-16954690 ] Trung commented on IGNITE-10959: We're affected by this issue as well. Memory is leaked at an alarming rate. See the attached continuousquery_leak_profile.png for the profile result of where the leaks are, They're all in byte[] buffers of BinaryObjectImpl inside CacheContinuousQueryEntry, > Memory leaks in continuous query handlers > - > > Key: IGNITE-10959 > URL: https://issues.apache.org/jira/browse/IGNITE-10959 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.7 >Reporter: Denis Mekhanikov >Priority: Major > Fix For: 2.9 > > Attachments: CacheContinuousQueryMemoryUsageTest.java, > continuousquery_leak_profile.png > > > Continuous query handlers don't clear internal data structures after cache > events are processed. > A test, that reproduces the problem, is attached. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-10959) Memory leaks in continuous query handlers
[ https://issues.apache.org/jira/browse/IGNITE-10959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16953706#comment-16953706 ] Grey Guo commented on IGNITE-10959: --- This is critical, it will trigger major GC everyday in our scenario, if there is no quick fix, please give some recommendation on how to workaround this > Memory leaks in continuous query handlers > - > > Key: IGNITE-10959 > URL: https://issues.apache.org/jira/browse/IGNITE-10959 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.7 >Reporter: Denis Mekhanikov >Priority: Major > Fix For: 2.9 > > Attachments: CacheContinuousQueryMemoryUsageTest.java > > > Continuous query handlers don't clear internal data structures after cache > events are processed. > A test, that reproduces the problem, is attached. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-10959) Memory leaks in continuous query handlers
[ https://issues.apache.org/jira/browse/IGNITE-10959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948484#comment-16948484 ] Denis Mekhanikov commented on IGNITE-10959: --- [~mmuzaf] I haven't been working on this issue lately. Sorry for the misleading "in progress" status. I don't have a patch for it, so let's move it to the next release unless somebody wants to fix it before 2.8 is released. > Memory leaks in continuous query handlers > - > > Key: IGNITE-10959 > URL: https://issues.apache.org/jira/browse/IGNITE-10959 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.7 >Reporter: Denis Mekhanikov >Assignee: Denis Mekhanikov >Priority: Major > Fix For: 2.8 > > Attachments: CacheContinuousQueryMemoryUsageTest.java > > > Continuous query handlers don't clear internal data structures after cache > events are processed. > A test, that reproduces the problem, is attached. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-10959) Memory leaks in continuous query handlers
[ https://issues.apache.org/jira/browse/IGNITE-10959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948456#comment-16948456 ] Maxim Muzafarov commented on IGNITE-10959: -- [~dmekhanikov] Hello, any updates here? Can we move this issue to the next release? Sounds like a `critical` issue! > Memory leaks in continuous query handlers > - > > Key: IGNITE-10959 > URL: https://issues.apache.org/jira/browse/IGNITE-10959 > Project: Ignite > Issue Type: Bug >Affects Versions: 2.7 >Reporter: Denis Mekhanikov >Assignee: Denis Mekhanikov >Priority: Major > Fix For: 2.8 > > Attachments: CacheContinuousQueryMemoryUsageTest.java > > > Continuous query handlers don't clear internal data structures after cache > events are processed. > A test, that reproduces the problem, is attached. -- This message was sent by Atlassian Jira (v8.3.4#803005)
