Re: Wrong off-heap size is reported for a node
Fixed, now the fix is in the ignite-2.7 branch. ср, 26 сент. 2018 г. в 23:18, Dmitriy Pavlov : > Oops, my fault. It seems I forgot to cherry-pick to 2.7, but I resolved > ticket with 2.7 assigned. > > I will fix it tomorrow. > > ср, 26 сент. 2018 г. в 21:52, Denis Magda : > >> Thanks, Pavel and the rest of the Igniters involved. >> >> That simple usability improvement is a big deal for those who use Ignite >> in >> production. >> >> Are we getting it in 2.7? >> >> -- >> Denis >> >> On Wed, Sep 26, 2018 at 10:11 AM Dmitriy Pavlov >> wrote: >> >> > Hi Igniters, >> > >> > Thanks to everyone, who has participated in this discussion and shared >> > their view and ideas. >> > >> > I've merged fix of changes related to logging only. Fixing of cluster >> > metrics can be done in a separate ticket/discussion. >> > >> > Pavel, thank you for your contribution and for answering my questions. >> > >> > Sincerely, >> > Dmitriy Pavlov >> > >> > пн, 24 сент. 2018 г. в 18:52, Pavel Pereslegin : >> > >> > > Andrei, >> > > I totally agree with you and I think that "ClusterMetrics" should also >> > > be fixed, I'm just not sure that we should include this change in the >> > > same ticket. >> > > пн, 24 сент. 2018 г. в 18:43, aealexsandrov > >: >> > > > >> > > > Hi, >> > > > >> > > > OK, the user can use it to calculate the off-heap. But I think that >> the >> > > > reason for your changes to fix the calculation of the nonHeap used >> in >> > > Ignite >> > > > now. For example now REST return "-1" for nonHeapMemoryMaximum. I >> think >> > > that >> > > > it can't be used somehow. So REST possible should be updated as you >> did >> > > for >> > > > log metrics and it will require for the same logic. >> > > > >> > > > BR, >> > > > Andrei >> > > > >> > > > >> > > > >> > > > -- >> > > > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/ >> > > >> > >> >
Re: Wrong off-heap size is reported for a node
Oops, my fault. It seems I forgot to cherry-pick to 2.7, but I resolved ticket with 2.7 assigned. I will fix it tomorrow. ср, 26 сент. 2018 г. в 21:52, Denis Magda : > Thanks, Pavel and the rest of the Igniters involved. > > That simple usability improvement is a big deal for those who use Ignite in > production. > > Are we getting it in 2.7? > > -- > Denis > > On Wed, Sep 26, 2018 at 10:11 AM Dmitriy Pavlov > wrote: > > > Hi Igniters, > > > > Thanks to everyone, who has participated in this discussion and shared > > their view and ideas. > > > > I've merged fix of changes related to logging only. Fixing of cluster > > metrics can be done in a separate ticket/discussion. > > > > Pavel, thank you for your contribution and for answering my questions. > > > > Sincerely, > > Dmitriy Pavlov > > > > пн, 24 сент. 2018 г. в 18:52, Pavel Pereslegin : > > > > > Andrei, > > > I totally agree with you and I think that "ClusterMetrics" should also > > > be fixed, I'm just not sure that we should include this change in the > > > same ticket. > > > пн, 24 сент. 2018 г. в 18:43, aealexsandrov : > > > > > > > > Hi, > > > > > > > > OK, the user can use it to calculate the off-heap. But I think that > the > > > > reason for your changes to fix the calculation of the nonHeap used in > > > Ignite > > > > now. For example now REST return "-1" for nonHeapMemoryMaximum. I > think > > > that > > > > it can't be used somehow. So REST possible should be updated as you > did > > > for > > > > log metrics and it will require for the same logic. > > > > > > > > BR, > > > > Andrei > > > > > > > > > > > > > > > > -- > > > > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/ > > > > > >
Re: Wrong off-heap size is reported for a node
Thanks, Pavel and the rest of the Igniters involved. That simple usability improvement is a big deal for those who use Ignite in production. Are we getting it in 2.7? -- Denis On Wed, Sep 26, 2018 at 10:11 AM Dmitriy Pavlov wrote: > Hi Igniters, > > Thanks to everyone, who has participated in this discussion and shared > their view and ideas. > > I've merged fix of changes related to logging only. Fixing of cluster > metrics can be done in a separate ticket/discussion. > > Pavel, thank you for your contribution and for answering my questions. > > Sincerely, > Dmitriy Pavlov > > пн, 24 сент. 2018 г. в 18:52, Pavel Pereslegin : > > > Andrei, > > I totally agree with you and I think that "ClusterMetrics" should also > > be fixed, I'm just not sure that we should include this change in the > > same ticket. > > пн, 24 сент. 2018 г. в 18:43, aealexsandrov : > > > > > > Hi, > > > > > > OK, the user can use it to calculate the off-heap. But I think that the > > > reason for your changes to fix the calculation of the nonHeap used in > > Ignite > > > now. For example now REST return "-1" for nonHeapMemoryMaximum. I think > > that > > > it can't be used somehow. So REST possible should be updated as you did > > for > > > log metrics and it will require for the same logic. > > > > > > BR, > > > Andrei > > > > > > > > > > > > -- > > > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/ > > >
Re: Wrong off-heap size is reported for a node
Hi Igniters, Thanks to everyone, who has participated in this discussion and shared their view and ideas. I've merged fix of changes related to logging only. Fixing of cluster metrics can be done in a separate ticket/discussion. Pavel, thank you for your contribution and for answering my questions. Sincerely, Dmitriy Pavlov пн, 24 сент. 2018 г. в 18:52, Pavel Pereslegin : > Andrei, > I totally agree with you and I think that "ClusterMetrics" should also > be fixed, I'm just not sure that we should include this change in the > same ticket. > пн, 24 сент. 2018 г. в 18:43, aealexsandrov : > > > > Hi, > > > > OK, the user can use it to calculate the off-heap. But I think that the > > reason for your changes to fix the calculation of the nonHeap used in > Ignite > > now. For example now REST return "-1" for nonHeapMemoryMaximum. I think > that > > it can't be used somehow. So REST possible should be updated as you did > for > > log metrics and it will require for the same logic. > > > > BR, > > Andrei > > > > > > > > -- > > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/ >
Re: Wrong off-heap size is reported for a node
Andrei, I totally agree with you and I think that "ClusterMetrics" should also be fixed, I'm just not sure that we should include this change in the same ticket. пн, 24 сент. 2018 г. в 18:43, aealexsandrov : > > Hi, > > OK, the user can use it to calculate the off-heap. But I think that the > reason for your changes to fix the calculation of the nonHeap used in Ignite > now. For example now REST return "-1" for nonHeapMemoryMaximum. I think that > it can't be used somehow. So REST possible should be updated as you did for > log metrics and it will require for the same logic. > > BR, > Andrei > > > > -- > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Wrong off-heap size is reported for a node
Hi, OK, the user can use it to calculate the off-heap. But I think that the reason for your changes to fix the calculation of the nonHeap used in Ignite now. For example now REST return "-1" for nonHeapMemoryMaximum. I think that it can't be used somehow. So REST possible should be updated as you did for log metrics and it will require for the same logic. BR, Andrei -- Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Wrong off-heap size is reported for a node
Hello Andrei, All these metrics available through JMX (see "DataRegionMetrics" group) [1]. [1] https://apacheignite.readme.io/docs/memory-metrics пн, 24 сент. 2018 г. в 17:58, aealexsandrov : > > Hi Igniters, > > Small notes according to these fix. > > As I see that all logic of calculation off-neap max size at the moment > located in the ackNodeMetrics method in IgniteKernal.java. I think that it > isn't ok because this logic should be added to other functionality too. I > think that will be better to move this logic for example somewhere inside > ClusterMetrics and update existed metrics. > > Also in case if you will push this change in current view then, for example, > the user could be confused because he will see different off-heap values in > the log and rest http://127.0.0.1:8080/ignite?cmd=node=true=127.0.0.1 > > Log: > > Metrics for local node (to disable set 'metricsLogFrequency' to 0) > ^-- Node [id=*3229aa83*, name=ignite1, uptime=00:00:41.202] > ... > ^-- Off-heap [used=8MB, free=99,82%, comm=1580MB] > ... > > Rest: > > "nodeId":"*3229aa83*-bfcb-45b3-af05-be33da50aa2f" > "nonHeapMemoryMaximum":-1 > "nonHeapMemoryUsed":58390208 > "nonHeapMemoryCommitted":59572224 > > I understand that REST could be implemented later but looks like at the > moment there is no way to get these metrics somehow except parsing the log > files. > > I suggest creating the way to get it by the user. > > BR, > Andrei > > > > > > -- > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Wrong off-heap size is reported for a node
Hi Igniters, Small notes according to these fix. As I see that all logic of calculation off-neap max size at the moment located in the ackNodeMetrics method in IgniteKernal.java. I think that it isn't ok because this logic should be added to other functionality too. I think that will be better to move this logic for example somewhere inside ClusterMetrics and update existed metrics. Also in case if you will push this change in current view then, for example, the user could be confused because he will see different off-heap values in the log and rest http://127.0.0.1:8080/ignite?cmd=node=true=127.0.0.1 Log: Metrics for local node (to disable set 'metricsLogFrequency' to 0) ^-- Node [id=*3229aa83*, name=ignite1, uptime=00:00:41.202] ... ^-- Off-heap [used=8MB, free=99,82%, comm=1580MB] ... Rest: "nodeId":"*3229aa83*-bfcb-45b3-af05-be33da50aa2f" "nonHeapMemoryMaximum":-1 "nonHeapMemoryUsed":58390208 "nonHeapMemoryCommitted":59572224 I understand that REST could be implemented later but looks like at the moment there is no way to get these metrics somehow except parsing the log files. I suggest creating the way to get it by the user. BR, Andrei -- Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Wrong off-heap size is reported for a node
Hi Dmitriy, Yes, I will finish review tomorrow. ср, 12 сент. 2018 г. в 20:37, Dmitriy Pavlov : > Hi Igniters, > > It seems the issue is a blocker of 2.7. > > We need a volunteer here to run review. > > Alexey P. will you step in? > > Sincerely, > Dmitriy Pavlov > > ср, 29 авг. 2018 г. в 16:16, Nikita Amelchev : > > > Pavel, > > > > I think that point 1 is the correct way to calculate the committed > > size for a log. It is already calculated regardless of the > > metricsEnabled flag. > > > > In addition, I suggest more readable log format for data regions in > > issue comments [1]. > > > > [1] https://issues.apache.org/jira/browse/IGNITE-9305 > > > > вт, 21 авг. 2018 г. в 12:15, Pavel Pereslegin : > > > > > > Hello, Igniters. > > > > > > I assigned ticket [1] created by Denis and want to clarify how to log > > > committed size. > > > The metric offHeapSize (in DataRegionMetricsImpl) is always > > > calculated, but getOffHeapSize returns zero if memory metrics are > > > disabled for this data region. > > > > > > So I see the following options: > > > 1. Modify method getOffHeapSize so that it always returns actual value > > > offHeapSize. > > > 2. Add another offHeapSize() method. > > > 3. Output to log max size instead of committed (change "comm" to "max" > > > in log output). > > > 4. Don't bother about disabling metrics and output to log value > > > returned by getOffHeapSize. > > > > > > Any thoughts? > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-9305 > > > сб, 18 авг. 2018 г. в 3:17, Denis Magda : > > > > > > > > Vova, the things are even simpler - we have this > > > > > > > > ignite.dataRegionMetrics().getPhysicalMemorySize() that returns the > > > > number equal/comparabel to pageNumber X pageSize. > > > > > > > > > > > > Igniters, if you believe that we need to do more work here then let's > > > > do it iteratively. Let's fix the off-heap occupied size the way above > > > > (just print out getPhysicalMemorySize() for every data region). Then > > > > do the rest. This needs to be fixed in 2.7. > > > > > > > > > > > > -- > > > > > > > > Denis > > > > > > > > > > > > On Fri, Aug 17, 2018 at 10:20 AM Vladimir Ozerov < > voze...@gridgain.com > > > > > > > wrote: > > > > > > > > > Folks, > > > > > > > > > > We already have this: > > > > > >>> PageMemory [pages=6997377] > > > > > > > > > > Then we can multiply it by page size and get occupied memory. Am I > > wrong? > > > > > > > > > > On Fri, Aug 17, 2018 at 12:56 PM Dmitriy Pavlov < > > dpavlov@gmail.com> > > > > > wrote: > > > > > > > > > > > Hi Maxim, > > > > > > > > > > > > thank you for stepping in and for finding these issues. Yes, > these > > > > > tickets > > > > > > are correct. > > > > > > > > > > > > I can move https://issues.apache.org/jira/browse/IGNITE-5583 to > > > > > unassigned > > > > > > if someone would like to implement this change. I will not have > > enough > > > > > time > > > > > > to complete it in 1 month (before 2.7 release). > > > > > > > > > > > > Sincerely, > > > > > > Dmitriy Pavlov > > > > > > > > > > > > пт, 17 авг. 2018 г. в 11:04, Maxim Muzafarov >: > > > > > > > > > > > > > Igniters, > > > > > > > > > > > > > > Suppose, Dmitry is talking about IGNITE-5583 [1] - `Switch > > non-heap > > > > > > memory > > > > > > > metrics > > > > > > > to new page memory semantics` and related previous disscustions > > to it > > > > > > [4]. > > > > > > > > > > > > > > Also we have some additional improvements to CacheMetrics: > > > > > > > IGNITE-5490 [2] - `Implement replacement for obsolete > > > > > > > CacheMetrics#getOffHeapAllocatedSize` > > > > > > > IGNITE-5765 [3] - `CacheMetrics interface cleanup, > documentation > > and > > > > > > fixes` > > > > > > > > > > > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-5583 > > > > > > > [2] https://issues.apache.org/jira/browse/IGNITE-5490 > > > > > > > [3] https://issues.apache.org/jira/browse/IGNITE-5765 > > > > > > > [4] > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://apache-ignite-developers.2346864.n4.nabble.com/Negative-non-heap-memory-maximum-td17990.html > > > > > > > > > > > > > > On Fri, 17 Aug 2018 at 10:14 Dmitriy Pavlov < > > dpavlov@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > > > Hi Igniters, > > > > > > > > > > > > > > > > It is not an easy fix, so I'm not sure it is possible to do > in > > 2.7. > > > > > > > > > > > > > > > > Offheap size is not reported by VM (it returns -1). To > > implement it > > > > > we > > > > > > > need > > > > > > > > totally migrate off-heap memory metrics to durable memory > data. > > > > > > > > > > > > > > > > I think this issue was reported and I'll find the duplicate. > > > > > > > > > > > > > > > > Sincerely, > > > > > > > > Dmitriy Pavlov > > > > > > > > > > > > > > > > пт, 17 авг. 2018 г. в 6:10, Denis Magda : > > > > > > > > > > > > > > > > > Yes, it was at the end of my wordy email :) > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-9305 > > > >
Re: Wrong off-heap size is reported for a node
Pavel, I think that point 1 is the correct way to calculate the committed size for a log. It is already calculated regardless of the metricsEnabled flag. In addition, I suggest more readable log format for data regions in issue comments [1]. [1] https://issues.apache.org/jira/browse/IGNITE-9305 вт, 21 авг. 2018 г. в 12:15, Pavel Pereslegin : > > Hello, Igniters. > > I assigned ticket [1] created by Denis and want to clarify how to log > committed size. > The metric offHeapSize (in DataRegionMetricsImpl) is always > calculated, but getOffHeapSize returns zero if memory metrics are > disabled for this data region. > > So I see the following options: > 1. Modify method getOffHeapSize so that it always returns actual value > offHeapSize. > 2. Add another offHeapSize() method. > 3. Output to log max size instead of committed (change "comm" to "max" > in log output). > 4. Don't bother about disabling metrics and output to log value > returned by getOffHeapSize. > > Any thoughts? > > [1] https://issues.apache.org/jira/browse/IGNITE-9305 > сб, 18 авг. 2018 г. в 3:17, Denis Magda : > > > > Vova, the things are even simpler - we have this > > > > ignite.dataRegionMetrics().getPhysicalMemorySize() that returns the > > number equal/comparabel to pageNumber X pageSize. > > > > > > Igniters, if you believe that we need to do more work here then let's > > do it iteratively. Let's fix the off-heap occupied size the way above > > (just print out getPhysicalMemorySize() for every data region). Then > > do the rest. This needs to be fixed in 2.7. > > > > > > -- > > > > Denis > > > > > > On Fri, Aug 17, 2018 at 10:20 AM Vladimir Ozerov > > wrote: > > > > > Folks, > > > > > > We already have this: > > > >>> PageMemory [pages=6997377] > > > > > > Then we can multiply it by page size and get occupied memory. Am I wrong? > > > > > > On Fri, Aug 17, 2018 at 12:56 PM Dmitriy Pavlov > > > wrote: > > > > > > > Hi Maxim, > > > > > > > > thank you for stepping in and for finding these issues. Yes, these > > > tickets > > > > are correct. > > > > > > > > I can move https://issues.apache.org/jira/browse/IGNITE-5583 to > > > unassigned > > > > if someone would like to implement this change. I will not have enough > > > time > > > > to complete it in 1 month (before 2.7 release). > > > > > > > > Sincerely, > > > > Dmitriy Pavlov > > > > > > > > пт, 17 авг. 2018 г. в 11:04, Maxim Muzafarov : > > > > > > > > > Igniters, > > > > > > > > > > Suppose, Dmitry is talking about IGNITE-5583 [1] - `Switch non-heap > > > > memory > > > > > metrics > > > > > to new page memory semantics` and related previous disscustions to it > > > > [4]. > > > > > > > > > > Also we have some additional improvements to CacheMetrics: > > > > > IGNITE-5490 [2] - `Implement replacement for obsolete > > > > > CacheMetrics#getOffHeapAllocatedSize` > > > > > IGNITE-5765 [3] - `CacheMetrics interface cleanup, documentation and > > > > fixes` > > > > > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-5583 > > > > > [2] https://issues.apache.org/jira/browse/IGNITE-5490 > > > > > [3] https://issues.apache.org/jira/browse/IGNITE-5765 > > > > > [4] > > > > > > > > > > > > > > > > > http://apache-ignite-developers.2346864.n4.nabble.com/Negative-non-heap-memory-maximum-td17990.html > > > > > > > > > > On Fri, 17 Aug 2018 at 10:14 Dmitriy Pavlov > > > > wrote: > > > > > > > > > > > Hi Igniters, > > > > > > > > > > > > It is not an easy fix, so I'm not sure it is possible to do in 2.7. > > > > > > > > > > > > Offheap size is not reported by VM (it returns -1). To implement it > > > we > > > > > need > > > > > > totally migrate off-heap memory metrics to durable memory data. > > > > > > > > > > > > I think this issue was reported and I'll find the duplicate. > > > > > > > > > > > > Sincerely, > > > > > > Dmitriy Pavlov > > > > > > > > > > > > пт, 17 авг. 2018 г. в 6:10, Denis Magda : > > > > > > > > > > > > > Yes, it was at the end of my wordy email :) > > > > > > > https://issues.apache.org/jira/browse/IGNITE-9305 > > > > > > > > > > > > > > -- > > > > > > > Denis > > > > > > > > > > > > > > On Thu, Aug 16, 2018 at 11:03 PM Dmitriy Setrakyan < > > > > > > dsetrak...@apache.org> > > > > > > > wrote: > > > > > > > > > > > > > > > Is there a blocker ticket for 2.7? > > > > > > > > > > > > > > > > On Thu, Aug 16, 2018, 19:59 Denis Magda > > > wrote: > > > > > > > > > > > > > > > > > Igniters, > > > > > > > > > > > > > > > > > > Was troubleshooting an Ignite deployment today and couldn't > > > find > > > > > out > > > > > > > from > > > > > > > > > the logs what was the actual off-heap space used. > > > > > > > > > > > > > > > > > > Those were the given memory resoures (Ignite 2.6): > > > > > > > > > > > > > > > > > > [2018-08-16 15:07:49,961][INFO ][main][GridDiscoveryManager] > > > > > Topology > > > > > > > > > snapshot [ver=1, servers=1, clients=0, CPUs=64, > > > *offheap=30.0GB*, > > > > > > > > > heap=24.0GB] > > > > > > > > > > > > > > > > > > And that
Re: Wrong off-heap size is reported for a node
Hello, Igniters. I assigned ticket [1] created by Denis and want to clarify how to log committed size. The metric offHeapSize (in DataRegionMetricsImpl) is always calculated, but getOffHeapSize returns zero if memory metrics are disabled for this data region. So I see the following options: 1. Modify method getOffHeapSize so that it always returns actual value offHeapSize. 2. Add another offHeapSize() method. 3. Output to log max size instead of committed (change "comm" to "max" in log output). 4. Don't bother about disabling metrics and output to log value returned by getOffHeapSize. Any thoughts? [1] https://issues.apache.org/jira/browse/IGNITE-9305 сб, 18 авг. 2018 г. в 3:17, Denis Magda : > > Vova, the things are even simpler - we have this > > ignite.dataRegionMetrics().getPhysicalMemorySize() that returns the > number equal/comparabel to pageNumber X pageSize. > > > Igniters, if you believe that we need to do more work here then let's > do it iteratively. Let's fix the off-heap occupied size the way above > (just print out getPhysicalMemorySize() for every data region). Then > do the rest. This needs to be fixed in 2.7. > > > -- > > Denis > > > On Fri, Aug 17, 2018 at 10:20 AM Vladimir Ozerov > wrote: > > > Folks, > > > > We already have this: > > >>> PageMemory [pages=6997377] > > > > Then we can multiply it by page size and get occupied memory. Am I wrong? > > > > On Fri, Aug 17, 2018 at 12:56 PM Dmitriy Pavlov > > wrote: > > > > > Hi Maxim, > > > > > > thank you for stepping in and for finding these issues. Yes, these > > tickets > > > are correct. > > > > > > I can move https://issues.apache.org/jira/browse/IGNITE-5583 to > > unassigned > > > if someone would like to implement this change. I will not have enough > > time > > > to complete it in 1 month (before 2.7 release). > > > > > > Sincerely, > > > Dmitriy Pavlov > > > > > > пт, 17 авг. 2018 г. в 11:04, Maxim Muzafarov : > > > > > > > Igniters, > > > > > > > > Suppose, Dmitry is talking about IGNITE-5583 [1] - `Switch non-heap > > > memory > > > > metrics > > > > to new page memory semantics` and related previous disscustions to it > > > [4]. > > > > > > > > Also we have some additional improvements to CacheMetrics: > > > > IGNITE-5490 [2] - `Implement replacement for obsolete > > > > CacheMetrics#getOffHeapAllocatedSize` > > > > IGNITE-5765 [3] - `CacheMetrics interface cleanup, documentation and > > > fixes` > > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-5583 > > > > [2] https://issues.apache.org/jira/browse/IGNITE-5490 > > > > [3] https://issues.apache.org/jira/browse/IGNITE-5765 > > > > [4] > > > > > > > > > > > > > http://apache-ignite-developers.2346864.n4.nabble.com/Negative-non-heap-memory-maximum-td17990.html > > > > > > > > On Fri, 17 Aug 2018 at 10:14 Dmitriy Pavlov > > > wrote: > > > > > > > > > Hi Igniters, > > > > > > > > > > It is not an easy fix, so I'm not sure it is possible to do in 2.7. > > > > > > > > > > Offheap size is not reported by VM (it returns -1). To implement it > > we > > > > need > > > > > totally migrate off-heap memory metrics to durable memory data. > > > > > > > > > > I think this issue was reported and I'll find the duplicate. > > > > > > > > > > Sincerely, > > > > > Dmitriy Pavlov > > > > > > > > > > пт, 17 авг. 2018 г. в 6:10, Denis Magda : > > > > > > > > > > > Yes, it was at the end of my wordy email :) > > > > > > https://issues.apache.org/jira/browse/IGNITE-9305 > > > > > > > > > > > > -- > > > > > > Denis > > > > > > > > > > > > On Thu, Aug 16, 2018 at 11:03 PM Dmitriy Setrakyan < > > > > > dsetrak...@apache.org> > > > > > > wrote: > > > > > > > > > > > > > Is there a blocker ticket for 2.7? > > > > > > > > > > > > > > On Thu, Aug 16, 2018, 19:59 Denis Magda > > wrote: > > > > > > > > > > > > > > > Igniters, > > > > > > > > > > > > > > > > Was troubleshooting an Ignite deployment today and couldn't > > find > > > > out > > > > > > from > > > > > > > > the logs what was the actual off-heap space used. > > > > > > > > > > > > > > > > Those were the given memory resoures (Ignite 2.6): > > > > > > > > > > > > > > > > [2018-08-16 15:07:49,961][INFO ][main][GridDiscoveryManager] > > > > Topology > > > > > > > > snapshot [ver=1, servers=1, clients=0, CPUs=64, > > *offheap=30.0GB*, > > > > > > > > heap=24.0GB] > > > > > > > > > > > > > > > > And that weird stuff was reported by the node (pay attention to > > > the > > > > > > last > > > > > > > > line): > > > > > > > > > > > > > > > > [2018-08-16 15:45:50,211][INFO > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ][grid-timeout-worker-#135%cluster_31-Dec-2017%][IgniteKernal%cluster_31-Dec-2017] > > > > > > > > Metrics for local node (to disable set 'metricsLogFrequency' to > > > 0) > > > > > > > > ^-- Node [id=c033026e, name=cluster_31-Dec-2017, > > > > > > uptime=00:38:00.257] > > > > > > > > ^-- H/N/C [hosts=1, nodes=1, CPUs=64] > > > > > > > > ^-- CPU [cur=0.03%,
Re: Wrong off-heap size is reported for a node
Vova, the things are even simpler - we have this ignite.dataRegionMetrics().getPhysicalMemorySize() that returns the number equal/comparabel to pageNumber X pageSize. Igniters, if you believe that we need to do more work here then let's do it iteratively. Let's fix the off-heap occupied size the way above (just print out getPhysicalMemorySize() for every data region). Then do the rest. This needs to be fixed in 2.7. -- Denis On Fri, Aug 17, 2018 at 10:20 AM Vladimir Ozerov wrote: > Folks, > > We already have this: > >>> PageMemory [pages=6997377] > > Then we can multiply it by page size and get occupied memory. Am I wrong? > > On Fri, Aug 17, 2018 at 12:56 PM Dmitriy Pavlov > wrote: > > > Hi Maxim, > > > > thank you for stepping in and for finding these issues. Yes, these > tickets > > are correct. > > > > I can move https://issues.apache.org/jira/browse/IGNITE-5583 to > unassigned > > if someone would like to implement this change. I will not have enough > time > > to complete it in 1 month (before 2.7 release). > > > > Sincerely, > > Dmitriy Pavlov > > > > пт, 17 авг. 2018 г. в 11:04, Maxim Muzafarov : > > > > > Igniters, > > > > > > Suppose, Dmitry is talking about IGNITE-5583 [1] - `Switch non-heap > > memory > > > metrics > > > to new page memory semantics` and related previous disscustions to it > > [4]. > > > > > > Also we have some additional improvements to CacheMetrics: > > > IGNITE-5490 [2] - `Implement replacement for obsolete > > > CacheMetrics#getOffHeapAllocatedSize` > > > IGNITE-5765 [3] - `CacheMetrics interface cleanup, documentation and > > fixes` > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-5583 > > > [2] https://issues.apache.org/jira/browse/IGNITE-5490 > > > [3] https://issues.apache.org/jira/browse/IGNITE-5765 > > > [4] > > > > > > > > > http://apache-ignite-developers.2346864.n4.nabble.com/Negative-non-heap-memory-maximum-td17990.html > > > > > > On Fri, 17 Aug 2018 at 10:14 Dmitriy Pavlov > > wrote: > > > > > > > Hi Igniters, > > > > > > > > It is not an easy fix, so I'm not sure it is possible to do in 2.7. > > > > > > > > Offheap size is not reported by VM (it returns -1). To implement it > we > > > need > > > > totally migrate off-heap memory metrics to durable memory data. > > > > > > > > I think this issue was reported and I'll find the duplicate. > > > > > > > > Sincerely, > > > > Dmitriy Pavlov > > > > > > > > пт, 17 авг. 2018 г. в 6:10, Denis Magda : > > > > > > > > > Yes, it was at the end of my wordy email :) > > > > > https://issues.apache.org/jira/browse/IGNITE-9305 > > > > > > > > > > -- > > > > > Denis > > > > > > > > > > On Thu, Aug 16, 2018 at 11:03 PM Dmitriy Setrakyan < > > > > dsetrak...@apache.org> > > > > > wrote: > > > > > > > > > > > Is there a blocker ticket for 2.7? > > > > > > > > > > > > On Thu, Aug 16, 2018, 19:59 Denis Magda > wrote: > > > > > > > > > > > > > Igniters, > > > > > > > > > > > > > > Was troubleshooting an Ignite deployment today and couldn't > find > > > out > > > > > from > > > > > > > the logs what was the actual off-heap space used. > > > > > > > > > > > > > > Those were the given memory resoures (Ignite 2.6): > > > > > > > > > > > > > > [2018-08-16 15:07:49,961][INFO ][main][GridDiscoveryManager] > > > Topology > > > > > > > snapshot [ver=1, servers=1, clients=0, CPUs=64, > *offheap=30.0GB*, > > > > > > > heap=24.0GB] > > > > > > > > > > > > > > And that weird stuff was reported by the node (pay attention to > > the > > > > > last > > > > > > > line): > > > > > > > > > > > > > > [2018-08-16 15:45:50,211][INFO > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ][grid-timeout-worker-#135%cluster_31-Dec-2017%][IgniteKernal%cluster_31-Dec-2017] > > > > > > > Metrics for local node (to disable set 'metricsLogFrequency' to > > 0) > > > > > > > ^-- Node [id=c033026e, name=cluster_31-Dec-2017, > > > > > uptime=00:38:00.257] > > > > > > > ^-- H/N/C [hosts=1, nodes=1, CPUs=64] > > > > > > > ^-- CPU [cur=0.03%, avg=5.54%, GC=0%] > > > > > > > ^-- PageMemory [pages=6997377] > > > > > > > ^-- Heap [used=9706MB, free=61.18%, comm=22384MB] > > > > > > >* ^-- Non heap [used=144MB, free=-1%, comm=148MB] - this > line > > is > > > > > > always > > > > > > > the same!* > > > > > > > > > > > > > > Had to change the code by using > > dataRegion.getPhysicalMemoryPages() > > > > to > > > > > > find > > > > > > > out that actual off-heap usage size was > > > > > > > >>> Physical Memory Size: 28651614208 => 27324 MB, 26 GB > > > > > > > > > > > > > > Let's fix this issue in 2.7, I proposed a new format. Please > > review > > > > and > > > > > > > share your thoughts: > > > > > > > https://issues.apache.org/jira/browse/IGNITE-9305 > > > > > > > > > > > > > > -- > > > > > > > Denis > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > -- > > > Maxim Muzafarov > > > > > >
Re: Wrong off-heap size is reported for a node
Yes, I agree. To calculate free (currently -1) we need to know total & used. Used=sysPageSize*usedPages, but overall should be obtained from all segments and chunks. So it would be a remarkable part of all mem.metrics migration (required by IGNITE-5583). пт, 17 авг. 2018 г. в 23:55, Alex Plehanov : > To be more precise we need to multiply it by page size with system overhead > (systemPageSize). If we want to print only used offheap memory, this will > solve the problem. If we want to print, for example, currently allocated > (commited) offheap memory (if persistence is disabled for data region then > offheap is allocated by chunks) we need to do more complex calculations and > there is no public API for this now. > > 2018-08-17 17:19 GMT+03:00 Vladimir Ozerov : > > > Folks, > > > > We already have this: > > >>> PageMemory [pages=6997377] > > > > Then we can multiply it by page size and get occupied memory. Am I wrong? > > > > On Fri, Aug 17, 2018 at 12:56 PM Dmitriy Pavlov > > wrote: > > > > > Hi Maxim, > > > > > > thank you for stepping in and for finding these issues. Yes, these > > tickets > > > are correct. > > > > > > I can move https://issues.apache.org/jira/browse/IGNITE-5583 to > > unassigned > > > if someone would like to implement this change. I will not have enough > > time > > > to complete it in 1 month (before 2.7 release). > > > > > > Sincerely, > > > Dmitriy Pavlov > > > > > > пт, 17 авг. 2018 г. в 11:04, Maxim Muzafarov : > > > > > > > Igniters, > > > > > > > > Suppose, Dmitry is talking about IGNITE-5583 [1] - `Switch non-heap > > > memory > > > > metrics > > > > to new page memory semantics` and related previous disscustions to it > > > [4]. > > > > > > > > Also we have some additional improvements to CacheMetrics: > > > > IGNITE-5490 [2] - `Implement replacement for obsolete > > > > CacheMetrics#getOffHeapAllocatedSize` > > > > IGNITE-5765 [3] - `CacheMetrics interface cleanup, documentation and > > > fixes` > > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-5583 > > > > [2] https://issues.apache.org/jira/browse/IGNITE-5490 > > > > [3] https://issues.apache.org/jira/browse/IGNITE-5765 > > > > [4] > > > > > > > > > > > http://apache-ignite-developers.2346864.n4.nabble. > > com/Negative-non-heap-memory-maximum-td17990.html > > > > > > > > On Fri, 17 Aug 2018 at 10:14 Dmitriy Pavlov > > > wrote: > > > > > > > > > Hi Igniters, > > > > > > > > > > It is not an easy fix, so I'm not sure it is possible to do in 2.7. > > > > > > > > > > Offheap size is not reported by VM (it returns -1). To implement it > > we > > > > need > > > > > totally migrate off-heap memory metrics to durable memory data. > > > > > > > > > > I think this issue was reported and I'll find the duplicate. > > > > > > > > > > Sincerely, > > > > > Dmitriy Pavlov > > > > > > > > > > пт, 17 авг. 2018 г. в 6:10, Denis Magda : > > > > > > > > > > > Yes, it was at the end of my wordy email :) > > > > > > https://issues.apache.org/jira/browse/IGNITE-9305 > > > > > > > > > > > > -- > > > > > > Denis > > > > > > > > > > > > On Thu, Aug 16, 2018 at 11:03 PM Dmitriy Setrakyan < > > > > > dsetrak...@apache.org> > > > > > > wrote: > > > > > > > > > > > > > Is there a blocker ticket for 2.7? > > > > > > > > > > > > > > On Thu, Aug 16, 2018, 19:59 Denis Magda > > wrote: > > > > > > > > > > > > > > > Igniters, > > > > > > > > > > > > > > > > Was troubleshooting an Ignite deployment today and couldn't > > find > > > > out > > > > > > from > > > > > > > > the logs what was the actual off-heap space used. > > > > > > > > > > > > > > > > Those were the given memory resoures (Ignite 2.6): > > > > > > > > > > > > > > > > [2018-08-16 15:07:49,961][INFO ][main][GridDiscoveryManager] > > > > Topology > > > > > > > > snapshot [ver=1, servers=1, clients=0, CPUs=64, > > *offheap=30.0GB*, > > > > > > > > heap=24.0GB] > > > > > > > > > > > > > > > > And that weird stuff was reported by the node (pay attention > to > > > the > > > > > > last > > > > > > > > line): > > > > > > > > > > > > > > > > [2018-08-16 15:45:50,211][INFO > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ][grid-timeout-worker-#135%cluster_31-Dec-2017%][ > > IgniteKernal%cluster_31-Dec-2017] > > > > > > > > Metrics for local node (to disable set 'metricsLogFrequency' > to > > > 0) > > > > > > > > ^-- Node [id=c033026e, name=cluster_31-Dec-2017, > > > > > > uptime=00:38:00.257] > > > > > > > > ^-- H/N/C [hosts=1, nodes=1, CPUs=64] > > > > > > > > ^-- CPU [cur=0.03%, avg=5.54%, GC=0%] > > > > > > > > ^-- PageMemory [pages=6997377] > > > > > > > > ^-- Heap [used=9706MB, free=61.18%, comm=22384MB] > > > > > > > >* ^-- Non heap [used=144MB, free=-1%, comm=148MB] - this > > line > > > is > > > > > > > always > > > > > > > > the same!* > > > > > > > > > > > > > > > > Had to change the code by using > > > dataRegion.getPhysicalMemoryPages() > > > > > to > > > > > > > find > > > > > > > > out that actual
Re: Wrong off-heap size is reported for a node
To be more precise we need to multiply it by page size with system overhead (systemPageSize). If we want to print only used offheap memory, this will solve the problem. If we want to print, for example, currently allocated (commited) offheap memory (if persistence is disabled for data region then offheap is allocated by chunks) we need to do more complex calculations and there is no public API for this now. 2018-08-17 17:19 GMT+03:00 Vladimir Ozerov : > Folks, > > We already have this: > >>> PageMemory [pages=6997377] > > Then we can multiply it by page size and get occupied memory. Am I wrong? > > On Fri, Aug 17, 2018 at 12:56 PM Dmitriy Pavlov > wrote: > > > Hi Maxim, > > > > thank you for stepping in and for finding these issues. Yes, these > tickets > > are correct. > > > > I can move https://issues.apache.org/jira/browse/IGNITE-5583 to > unassigned > > if someone would like to implement this change. I will not have enough > time > > to complete it in 1 month (before 2.7 release). > > > > Sincerely, > > Dmitriy Pavlov > > > > пт, 17 авг. 2018 г. в 11:04, Maxim Muzafarov : > > > > > Igniters, > > > > > > Suppose, Dmitry is talking about IGNITE-5583 [1] - `Switch non-heap > > memory > > > metrics > > > to new page memory semantics` and related previous disscustions to it > > [4]. > > > > > > Also we have some additional improvements to CacheMetrics: > > > IGNITE-5490 [2] - `Implement replacement for obsolete > > > CacheMetrics#getOffHeapAllocatedSize` > > > IGNITE-5765 [3] - `CacheMetrics interface cleanup, documentation and > > fixes` > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-5583 > > > [2] https://issues.apache.org/jira/browse/IGNITE-5490 > > > [3] https://issues.apache.org/jira/browse/IGNITE-5765 > > > [4] > > > > > > > > http://apache-ignite-developers.2346864.n4.nabble. > com/Negative-non-heap-memory-maximum-td17990.html > > > > > > On Fri, 17 Aug 2018 at 10:14 Dmitriy Pavlov > > wrote: > > > > > > > Hi Igniters, > > > > > > > > It is not an easy fix, so I'm not sure it is possible to do in 2.7. > > > > > > > > Offheap size is not reported by VM (it returns -1). To implement it > we > > > need > > > > totally migrate off-heap memory metrics to durable memory data. > > > > > > > > I think this issue was reported and I'll find the duplicate. > > > > > > > > Sincerely, > > > > Dmitriy Pavlov > > > > > > > > пт, 17 авг. 2018 г. в 6:10, Denis Magda : > > > > > > > > > Yes, it was at the end of my wordy email :) > > > > > https://issues.apache.org/jira/browse/IGNITE-9305 > > > > > > > > > > -- > > > > > Denis > > > > > > > > > > On Thu, Aug 16, 2018 at 11:03 PM Dmitriy Setrakyan < > > > > dsetrak...@apache.org> > > > > > wrote: > > > > > > > > > > > Is there a blocker ticket for 2.7? > > > > > > > > > > > > On Thu, Aug 16, 2018, 19:59 Denis Magda > wrote: > > > > > > > > > > > > > Igniters, > > > > > > > > > > > > > > Was troubleshooting an Ignite deployment today and couldn't > find > > > out > > > > > from > > > > > > > the logs what was the actual off-heap space used. > > > > > > > > > > > > > > Those were the given memory resoures (Ignite 2.6): > > > > > > > > > > > > > > [2018-08-16 15:07:49,961][INFO ][main][GridDiscoveryManager] > > > Topology > > > > > > > snapshot [ver=1, servers=1, clients=0, CPUs=64, > *offheap=30.0GB*, > > > > > > > heap=24.0GB] > > > > > > > > > > > > > > And that weird stuff was reported by the node (pay attention to > > the > > > > > last > > > > > > > line): > > > > > > > > > > > > > > [2018-08-16 15:45:50,211][INFO > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ][grid-timeout-worker-#135%cluster_31-Dec-2017%][ > IgniteKernal%cluster_31-Dec-2017] > > > > > > > Metrics for local node (to disable set 'metricsLogFrequency' to > > 0) > > > > > > > ^-- Node [id=c033026e, name=cluster_31-Dec-2017, > > > > > uptime=00:38:00.257] > > > > > > > ^-- H/N/C [hosts=1, nodes=1, CPUs=64] > > > > > > > ^-- CPU [cur=0.03%, avg=5.54%, GC=0%] > > > > > > > ^-- PageMemory [pages=6997377] > > > > > > > ^-- Heap [used=9706MB, free=61.18%, comm=22384MB] > > > > > > >* ^-- Non heap [used=144MB, free=-1%, comm=148MB] - this > line > > is > > > > > > always > > > > > > > the same!* > > > > > > > > > > > > > > Had to change the code by using > > dataRegion.getPhysicalMemoryPages() > > > > to > > > > > > find > > > > > > > out that actual off-heap usage size was > > > > > > > >>> Physical Memory Size: 28651614208 => 27324 MB, 26 GB > > > > > > > > > > > > > > Let's fix this issue in 2.7, I proposed a new format. Please > > review > > > > and > > > > > > > share your thoughts: > > > > > > > https://issues.apache.org/jira/browse/IGNITE-9305 > > > > > > > > > > > > > > -- > > > > > > > Denis > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > -- > > > Maxim Muzafarov > > > > > >
Re: Wrong off-heap size is reported for a node
Folks, We already have this: >>> PageMemory [pages=6997377] Then we can multiply it by page size and get occupied memory. Am I wrong? On Fri, Aug 17, 2018 at 12:56 PM Dmitriy Pavlov wrote: > Hi Maxim, > > thank you for stepping in and for finding these issues. Yes, these tickets > are correct. > > I can move https://issues.apache.org/jira/browse/IGNITE-5583 to unassigned > if someone would like to implement this change. I will not have enough time > to complete it in 1 month (before 2.7 release). > > Sincerely, > Dmitriy Pavlov > > пт, 17 авг. 2018 г. в 11:04, Maxim Muzafarov : > > > Igniters, > > > > Suppose, Dmitry is talking about IGNITE-5583 [1] - `Switch non-heap > memory > > metrics > > to new page memory semantics` and related previous disscustions to it > [4]. > > > > Also we have some additional improvements to CacheMetrics: > > IGNITE-5490 [2] - `Implement replacement for obsolete > > CacheMetrics#getOffHeapAllocatedSize` > > IGNITE-5765 [3] - `CacheMetrics interface cleanup, documentation and > fixes` > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-5583 > > [2] https://issues.apache.org/jira/browse/IGNITE-5490 > > [3] https://issues.apache.org/jira/browse/IGNITE-5765 > > [4] > > > > > http://apache-ignite-developers.2346864.n4.nabble.com/Negative-non-heap-memory-maximum-td17990.html > > > > On Fri, 17 Aug 2018 at 10:14 Dmitriy Pavlov > wrote: > > > > > Hi Igniters, > > > > > > It is not an easy fix, so I'm not sure it is possible to do in 2.7. > > > > > > Offheap size is not reported by VM (it returns -1). To implement it we > > need > > > totally migrate off-heap memory metrics to durable memory data. > > > > > > I think this issue was reported and I'll find the duplicate. > > > > > > Sincerely, > > > Dmitriy Pavlov > > > > > > пт, 17 авг. 2018 г. в 6:10, Denis Magda : > > > > > > > Yes, it was at the end of my wordy email :) > > > > https://issues.apache.org/jira/browse/IGNITE-9305 > > > > > > > > -- > > > > Denis > > > > > > > > On Thu, Aug 16, 2018 at 11:03 PM Dmitriy Setrakyan < > > > dsetrak...@apache.org> > > > > wrote: > > > > > > > > > Is there a blocker ticket for 2.7? > > > > > > > > > > On Thu, Aug 16, 2018, 19:59 Denis Magda wrote: > > > > > > > > > > > Igniters, > > > > > > > > > > > > Was troubleshooting an Ignite deployment today and couldn't find > > out > > > > from > > > > > > the logs what was the actual off-heap space used. > > > > > > > > > > > > Those were the given memory resoures (Ignite 2.6): > > > > > > > > > > > > [2018-08-16 15:07:49,961][INFO ][main][GridDiscoveryManager] > > Topology > > > > > > snapshot [ver=1, servers=1, clients=0, CPUs=64, *offheap=30.0GB*, > > > > > > heap=24.0GB] > > > > > > > > > > > > And that weird stuff was reported by the node (pay attention to > the > > > > last > > > > > > line): > > > > > > > > > > > > [2018-08-16 15:45:50,211][INFO > > > > > > > > > > > > > > > > > > > > > > > > > > > ][grid-timeout-worker-#135%cluster_31-Dec-2017%][IgniteKernal%cluster_31-Dec-2017] > > > > > > Metrics for local node (to disable set 'metricsLogFrequency' to > 0) > > > > > > ^-- Node [id=c033026e, name=cluster_31-Dec-2017, > > > > uptime=00:38:00.257] > > > > > > ^-- H/N/C [hosts=1, nodes=1, CPUs=64] > > > > > > ^-- CPU [cur=0.03%, avg=5.54%, GC=0%] > > > > > > ^-- PageMemory [pages=6997377] > > > > > > ^-- Heap [used=9706MB, free=61.18%, comm=22384MB] > > > > > >* ^-- Non heap [used=144MB, free=-1%, comm=148MB] - this line > is > > > > > always > > > > > > the same!* > > > > > > > > > > > > Had to change the code by using > dataRegion.getPhysicalMemoryPages() > > > to > > > > > find > > > > > > out that actual off-heap usage size was > > > > > > >>> Physical Memory Size: 28651614208 => 27324 MB, 26 GB > > > > > > > > > > > > Let's fix this issue in 2.7, I proposed a new format. Please > review > > > and > > > > > > share your thoughts: > > > > > > https://issues.apache.org/jira/browse/IGNITE-9305 > > > > > > > > > > > > -- > > > > > > Denis > > > > > > > > > > > > > > > > > > > > -- > > -- > > Maxim Muzafarov > > >
Re: Wrong off-heap size is reported for a node
Hi Maxim, thank you for stepping in and for finding these issues. Yes, these tickets are correct. I can move https://issues.apache.org/jira/browse/IGNITE-5583 to unassigned if someone would like to implement this change. I will not have enough time to complete it in 1 month (before 2.7 release). Sincerely, Dmitriy Pavlov пт, 17 авг. 2018 г. в 11:04, Maxim Muzafarov : > Igniters, > > Suppose, Dmitry is talking about IGNITE-5583 [1] - `Switch non-heap memory > metrics > to new page memory semantics` and related previous disscustions to it [4]. > > Also we have some additional improvements to CacheMetrics: > IGNITE-5490 [2] - `Implement replacement for obsolete > CacheMetrics#getOffHeapAllocatedSize` > IGNITE-5765 [3] - `CacheMetrics interface cleanup, documentation and fixes` > > > [1] https://issues.apache.org/jira/browse/IGNITE-5583 > [2] https://issues.apache.org/jira/browse/IGNITE-5490 > [3] https://issues.apache.org/jira/browse/IGNITE-5765 > [4] > > http://apache-ignite-developers.2346864.n4.nabble.com/Negative-non-heap-memory-maximum-td17990.html > > On Fri, 17 Aug 2018 at 10:14 Dmitriy Pavlov wrote: > > > Hi Igniters, > > > > It is not an easy fix, so I'm not sure it is possible to do in 2.7. > > > > Offheap size is not reported by VM (it returns -1). To implement it we > need > > totally migrate off-heap memory metrics to durable memory data. > > > > I think this issue was reported and I'll find the duplicate. > > > > Sincerely, > > Dmitriy Pavlov > > > > пт, 17 авг. 2018 г. в 6:10, Denis Magda : > > > > > Yes, it was at the end of my wordy email :) > > > https://issues.apache.org/jira/browse/IGNITE-9305 > > > > > > -- > > > Denis > > > > > > On Thu, Aug 16, 2018 at 11:03 PM Dmitriy Setrakyan < > > dsetrak...@apache.org> > > > wrote: > > > > > > > Is there a blocker ticket for 2.7? > > > > > > > > On Thu, Aug 16, 2018, 19:59 Denis Magda wrote: > > > > > > > > > Igniters, > > > > > > > > > > Was troubleshooting an Ignite deployment today and couldn't find > out > > > from > > > > > the logs what was the actual off-heap space used. > > > > > > > > > > Those were the given memory resoures (Ignite 2.6): > > > > > > > > > > [2018-08-16 15:07:49,961][INFO ][main][GridDiscoveryManager] > Topology > > > > > snapshot [ver=1, servers=1, clients=0, CPUs=64, *offheap=30.0GB*, > > > > > heap=24.0GB] > > > > > > > > > > And that weird stuff was reported by the node (pay attention to the > > > last > > > > > line): > > > > > > > > > > [2018-08-16 15:45:50,211][INFO > > > > > > > > > > > > > > > > > > > > ][grid-timeout-worker-#135%cluster_31-Dec-2017%][IgniteKernal%cluster_31-Dec-2017] > > > > > Metrics for local node (to disable set 'metricsLogFrequency' to 0) > > > > > ^-- Node [id=c033026e, name=cluster_31-Dec-2017, > > > uptime=00:38:00.257] > > > > > ^-- H/N/C [hosts=1, nodes=1, CPUs=64] > > > > > ^-- CPU [cur=0.03%, avg=5.54%, GC=0%] > > > > > ^-- PageMemory [pages=6997377] > > > > > ^-- Heap [used=9706MB, free=61.18%, comm=22384MB] > > > > >* ^-- Non heap [used=144MB, free=-1%, comm=148MB] - this line is > > > > always > > > > > the same!* > > > > > > > > > > Had to change the code by using dataRegion.getPhysicalMemoryPages() > > to > > > > find > > > > > out that actual off-heap usage size was > > > > > >>> Physical Memory Size: 28651614208 => 27324 MB, 26 GB > > > > > > > > > > Let's fix this issue in 2.7, I proposed a new format. Please review > > and > > > > > share your thoughts: > > > > > https://issues.apache.org/jira/browse/IGNITE-9305 > > > > > > > > > > -- > > > > > Denis > > > > > > > > > > > > > > > -- > -- > Maxim Muzafarov >
Re: Wrong off-heap size is reported for a node
Igniters, Suppose, Dmitry is talking about IGNITE-5583 [1] - `Switch non-heap memory metrics to new page memory semantics` and related previous disscustions to it [4]. Also we have some additional improvements to CacheMetrics: IGNITE-5490 [2] - `Implement replacement for obsolete CacheMetrics#getOffHeapAllocatedSize` IGNITE-5765 [3] - `CacheMetrics interface cleanup, documentation and fixes` [1] https://issues.apache.org/jira/browse/IGNITE-5583 [2] https://issues.apache.org/jira/browse/IGNITE-5490 [3] https://issues.apache.org/jira/browse/IGNITE-5765 [4] http://apache-ignite-developers.2346864.n4.nabble.com/Negative-non-heap-memory-maximum-td17990.html On Fri, 17 Aug 2018 at 10:14 Dmitriy Pavlov wrote: > Hi Igniters, > > It is not an easy fix, so I'm not sure it is possible to do in 2.7. > > Offheap size is not reported by VM (it returns -1). To implement it we need > totally migrate off-heap memory metrics to durable memory data. > > I think this issue was reported and I'll find the duplicate. > > Sincerely, > Dmitriy Pavlov > > пт, 17 авг. 2018 г. в 6:10, Denis Magda : > > > Yes, it was at the end of my wordy email :) > > https://issues.apache.org/jira/browse/IGNITE-9305 > > > > -- > > Denis > > > > On Thu, Aug 16, 2018 at 11:03 PM Dmitriy Setrakyan < > dsetrak...@apache.org> > > wrote: > > > > > Is there a blocker ticket for 2.7? > > > > > > On Thu, Aug 16, 2018, 19:59 Denis Magda wrote: > > > > > > > Igniters, > > > > > > > > Was troubleshooting an Ignite deployment today and couldn't find out > > from > > > > the logs what was the actual off-heap space used. > > > > > > > > Those were the given memory resoures (Ignite 2.6): > > > > > > > > [2018-08-16 15:07:49,961][INFO ][main][GridDiscoveryManager] Topology > > > > snapshot [ver=1, servers=1, clients=0, CPUs=64, *offheap=30.0GB*, > > > > heap=24.0GB] > > > > > > > > And that weird stuff was reported by the node (pay attention to the > > last > > > > line): > > > > > > > > [2018-08-16 15:45:50,211][INFO > > > > > > > > > > > > > > ][grid-timeout-worker-#135%cluster_31-Dec-2017%][IgniteKernal%cluster_31-Dec-2017] > > > > Metrics for local node (to disable set 'metricsLogFrequency' to 0) > > > > ^-- Node [id=c033026e, name=cluster_31-Dec-2017, > > uptime=00:38:00.257] > > > > ^-- H/N/C [hosts=1, nodes=1, CPUs=64] > > > > ^-- CPU [cur=0.03%, avg=5.54%, GC=0%] > > > > ^-- PageMemory [pages=6997377] > > > > ^-- Heap [used=9706MB, free=61.18%, comm=22384MB] > > > >* ^-- Non heap [used=144MB, free=-1%, comm=148MB] - this line is > > > always > > > > the same!* > > > > > > > > Had to change the code by using dataRegion.getPhysicalMemoryPages() > to > > > find > > > > out that actual off-heap usage size was > > > > >>> Physical Memory Size: 28651614208 => 27324 MB, 26 GB > > > > > > > > Let's fix this issue in 2.7, I proposed a new format. Please review > and > > > > share your thoughts: > > > > https://issues.apache.org/jira/browse/IGNITE-9305 > > > > > > > > -- > > > > Denis > > > > > > > > > > -- -- Maxim Muzafarov
Re: Wrong off-heap size is reported for a node
Hi Igniters, It is not an easy fix, so I'm not sure it is possible to do in 2.7. Offheap size is not reported by VM (it returns -1). To implement it we need totally migrate off-heap memory metrics to durable memory data. I think this issue was reported and I'll find the duplicate. Sincerely, Dmitriy Pavlov пт, 17 авг. 2018 г. в 6:10, Denis Magda : > Yes, it was at the end of my wordy email :) > https://issues.apache.org/jira/browse/IGNITE-9305 > > -- > Denis > > On Thu, Aug 16, 2018 at 11:03 PM Dmitriy Setrakyan > wrote: > > > Is there a blocker ticket for 2.7? > > > > On Thu, Aug 16, 2018, 19:59 Denis Magda wrote: > > > > > Igniters, > > > > > > Was troubleshooting an Ignite deployment today and couldn't find out > from > > > the logs what was the actual off-heap space used. > > > > > > Those were the given memory resoures (Ignite 2.6): > > > > > > [2018-08-16 15:07:49,961][INFO ][main][GridDiscoveryManager] Topology > > > snapshot [ver=1, servers=1, clients=0, CPUs=64, *offheap=30.0GB*, > > > heap=24.0GB] > > > > > > And that weird stuff was reported by the node (pay attention to the > last > > > line): > > > > > > [2018-08-16 15:45:50,211][INFO > > > > > > > > > ][grid-timeout-worker-#135%cluster_31-Dec-2017%][IgniteKernal%cluster_31-Dec-2017] > > > Metrics for local node (to disable set 'metricsLogFrequency' to 0) > > > ^-- Node [id=c033026e, name=cluster_31-Dec-2017, > uptime=00:38:00.257] > > > ^-- H/N/C [hosts=1, nodes=1, CPUs=64] > > > ^-- CPU [cur=0.03%, avg=5.54%, GC=0%] > > > ^-- PageMemory [pages=6997377] > > > ^-- Heap [used=9706MB, free=61.18%, comm=22384MB] > > >* ^-- Non heap [used=144MB, free=-1%, comm=148MB] - this line is > > always > > > the same!* > > > > > > Had to change the code by using dataRegion.getPhysicalMemoryPages() to > > find > > > out that actual off-heap usage size was > > > >>> Physical Memory Size: 28651614208 => 27324 MB, 26 GB > > > > > > Let's fix this issue in 2.7, I proposed a new format. Please review and > > > share your thoughts: > > > https://issues.apache.org/jira/browse/IGNITE-9305 > > > > > > -- > > > Denis > > > > > >
Re: Wrong off-heap size is reported for a node
Is there a blocker ticket for 2.7? On Thu, Aug 16, 2018, 19:59 Denis Magda wrote: > Igniters, > > Was troubleshooting an Ignite deployment today and couldn't find out from > the logs what was the actual off-heap space used. > > Those were the given memory resoures (Ignite 2.6): > > [2018-08-16 15:07:49,961][INFO ][main][GridDiscoveryManager] Topology > snapshot [ver=1, servers=1, clients=0, CPUs=64, *offheap=30.0GB*, > heap=24.0GB] > > And that weird stuff was reported by the node (pay attention to the last > line): > > [2018-08-16 15:45:50,211][INFO > > ][grid-timeout-worker-#135%cluster_31-Dec-2017%][IgniteKernal%cluster_31-Dec-2017] > Metrics for local node (to disable set 'metricsLogFrequency' to 0) > ^-- Node [id=c033026e, name=cluster_31-Dec-2017, uptime=00:38:00.257] > ^-- H/N/C [hosts=1, nodes=1, CPUs=64] > ^-- CPU [cur=0.03%, avg=5.54%, GC=0%] > ^-- PageMemory [pages=6997377] > ^-- Heap [used=9706MB, free=61.18%, comm=22384MB] >* ^-- Non heap [used=144MB, free=-1%, comm=148MB] - this line is always > the same!* > > Had to change the code by using dataRegion.getPhysicalMemoryPages() to find > out that actual off-heap usage size was > >>> Physical Memory Size: 28651614208 => 27324 MB, 26 GB > > Let's fix this issue in 2.7, I proposed a new format. Please review and > share your thoughts: > https://issues.apache.org/jira/browse/IGNITE-9305 > > -- > Denis >
Wrong off-heap size is reported for a node
Igniters, Was troubleshooting an Ignite deployment today and couldn't find out from the logs what was the actual off-heap space used. Those were the given memory resoures (Ignite 2.6): [2018-08-16 15:07:49,961][INFO ][main][GridDiscoveryManager] Topology snapshot [ver=1, servers=1, clients=0, CPUs=64, *offheap=30.0GB*, heap=24.0GB] And that weird stuff was reported by the node (pay attention to the last line): [2018-08-16 15:45:50,211][INFO ][grid-timeout-worker-#135%cluster_31-Dec-2017%][IgniteKernal%cluster_31-Dec-2017] Metrics for local node (to disable set 'metricsLogFrequency' to 0) ^-- Node [id=c033026e, name=cluster_31-Dec-2017, uptime=00:38:00.257] ^-- H/N/C [hosts=1, nodes=1, CPUs=64] ^-- CPU [cur=0.03%, avg=5.54%, GC=0%] ^-- PageMemory [pages=6997377] ^-- Heap [used=9706MB, free=61.18%, comm=22384MB] * ^-- Non heap [used=144MB, free=-1%, comm=148MB] - this line is always the same!* Had to change the code by using dataRegion.getPhysicalMemoryPages() to find out that actual off-heap usage size was >>> Physical Memory Size: 28651614208 => 27324 MB, 26 GB Let's fix this issue in 2.7, I proposed a new format. Please review and share your thoughts: https://issues.apache.org/jira/browse/IGNITE-9305 -- Denis
[jira] [Created] (IGNITE-9305) Wrong off-heap size is reported for a node
Denis Magda created IGNITE-9305:
---
Summary: Wrong off-heap size is reported for a node
Key: IGNITE-9305
URL: https://issues.apache.org/jira/browse/IGNITE-9305
Project: Ignite
Issue Type: Task
Affects Versions: 2.6
Reporter: Denis Magda
Fix For: 2.7
Was troubleshooting an Ignite deployment today and couldn't find out from the
logs what was the actual off-heap space used.
Those were the given memory resoures (Ignite 2.6):
{code}
[2018-08-16 15:07:49,961][INFO ][main][GridDiscoveryManager] Topology snapshot
[ver=1, servers=1, clients=0, CPUs=64, offheap=30.0GB, heap=24.0GB]
{code}
And that weird stuff was reported by the node (pay attention to the last line):
{code}
[2018-08-16 15:45:50,211][INFO
][grid-timeout-worker-#135%cluster_31-Dec-2017%][IgniteKernal%cluster_31-Dec-2017]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=c033026e, name=cluster_31-Dec-2017, uptime=00:38:00.257]
^-- H/N/C [hosts=1, nodes=1, CPUs=64]
^-- CPU [cur=0.03%, avg=5.54%, GC=0%]
^-- PageMemory [pages=6997377]
^-- Heap [used=9706MB, free=61.18%, comm=22384MB]
^-- Non heap [used=144MB, free=-1%, comm=148MB] - this line is always the
same!
{code}
Had to change the code by using {code}dataRegion.getPhysicalMemoryPages(){code}
to find out that actual off-heap usage size was
{code}
>>> Physical Memory Size: 28651614208 => 27324 MB, 26 GB
{code}
The logs have to report the following instead:
{code}
^-- Off-heap {Data Region 1} [used={dataRegion1.getPhysicalMemorySize()},
free=X%, comm=dataRegion1.maxSize()]
^-- Off-heap {Data Region 2} [used={dataRegion2.getPhysicalMemorySize()},
free=X%, comm=dataRegion2.maxSize()]
{code}
If Ignite persistence is enabled then the following extra lines have to be
added to see the disk used space:
{code}
^-- Off-heap {Data Region 1}: used={dataRegion1.getTotalAllocatedSize() -
dataRegion1.getPhysicalMemorySize()}
^-- Off-heap {Data Region 2} [used={dataRegion2.getTotalAllocatedSize() -
dataRegion2.getPhysicalMemorySize()}]
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
