Re: [ntp:questions] Ntpq.exe memory issue with windows 2019
On Thursday, October 29, 2020 at 8:28:37 PM UTC+5:30, Chris wrote: > On 10/29/20 12:24, David Woolley wrote: > > On 28/10/2020 18:01, Uwe Klein wrote: > >> That does not jibe with "creeping increase of memory taken". > >> > >> If Win keeps something for reuse ... it should not grow but be reused > >> later again kept for ...:-) > > > > As I said, I'm not familiar with the way that Windows shows memory > > statistics, but I do know that both on Windows and Linux, people often > > get confused because the OS aims only to keep a small amount of memory > > free. I was assuming that memory used was total - free, rather than > > reflecting that which could not be released without fatally harming a > > running process. > Sounds like a classic memory leak. Not familiar with ws2019, but you > need to track down what's causing it. For example, does the problem > go away if you disable ntpq.exe and is that part of the os, or third > party utility ?. Are you sure ntpq.exe terminates after execution, > or is it staying resident in memory ?... > > Regards, > > Chris Hi, Yes The issue goes away if I disable the execution of ntpq.exe, Please check the link I have shared in my earlier comment. I'm tracking the problem down and so far I have found that the issue doesn't occur if you have the windows time service disabled. ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Ntpq.exe memory issue with windows 2019
On 10/29/20 12:24, David Woolley wrote: On 28/10/2020 18:01, Uwe Klein wrote: That does not jibe with "creeping increase of memory taken". If Win keeps something for reuse ... it should not grow but be reused later again kept for ...:-) As I said, I'm not familiar with the way that Windows shows memory statistics, but I do know that both on Windows and Linux, people often get confused because the OS aims only to keep a small amount of memory free. I was assuming that memory used was total - free, rather than reflecting that which could not be released without fatally harming a running process. Sounds like a classic memory leak. Not familiar with ws2019, but you need to track down what's causing it. For example, does the problem go away if you disable ntpq.exe and is that part of the os, or third party utility ?. Are you sure ntpq.exe terminates after execution, or is it staying resident in memory ?... Regards, Chris ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Ntpq.exe memory issue with windows 2019
On 28/10/2020 18:01, Uwe Klein wrote: That does not jibe with "creeping increase of memory taken". If Win keeps something for reuse ... it should not grow but be reused later again kept for ...:-) As I said, I'm not familiar with the way that Windows shows memory statistics, but I do know that both on Windows and Linux, people often get confused because the OS aims only to keep a small amount of memory free. I was assuming that memory used was total - free, rather than reflecting that which could not be released without fatally harming a running process. ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Ntpq.exe memory issue with windows 2019
Am 21.10.20 um 22:45 schrieb David Woolley: > One wild thought: is ntpq or one of its DLLs not position independent > code? I could speculate as to why Windows might keep relocated pages > around in case the code is reloaded at the same address before the page > gets reused for other reasons. However, I don't actually know if > Windows does something like that. That does not jibe with "creeping increase of memory taken". If Win keeps something for reuse ... it should not grow but be reused later again kept for ... :-) Windows has always been a rather buggy and unrealiable OS. Uwe ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Ntpq.exe memory issue with windows 2019
-Original Message- From: Sadique Urf Arbaz Sayyed We started with a brand new windows server 2019 datacenter edition and installed an infrastructure monitoring agent on it and strictly no other program. The machine had 8 GB of memory. As part of monitoring NTP offset from sync'd host we scheduled a ntpqexe. The problem started after 4-5 days, the memory utilisation had increased to significant level >80%. On analysis we found it was a gradual increase and using RAMMAP we saw every time the ntpq.exe will run it will leave behind 24k of memory in PAGE Table with 0 B in Private. Moreover this issue is specific to windows server 2019 we tried following same steps on windows server 2012 machine and it worked perfectly fine with no memory creeping issues. Any help or pointer are appreciated = Thanks for your report. On a newly installed PC running Windows-10 Pro, using standard NTP I see a loss of "Available" memory of about 16 kB per ntpq invocation. Numbers: running ntpq in a Perl script. This Perl script is called (with different parameters) four times for each node. I'm monitoring ~25 nodes, so that's about 100 calls for each MRTG check, which is every five minutes, making 28,800 calls per day. Loss (by visual inspection) is 8 GB in 17 days. So approximately 16 kB per ntpq invocation. From other monitoring, ntpd is completely constant in memory usage (35-40 MB). I'll see whether entering an NTP bug still worksit does so I'll add evidence to your 3695. Cheers, David -- SatSignal Software - Quality software for you Web: https://www.satsignal.eu Email: david-tay...@blueyonder.co.uk Twitter: @gm8arv ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Ntpq.exe memory issue with windows 2019
On Wednesday, October 21, 2020 at 6:19:21 PM UTC+5:30, Jakob Bohm wrote: > On 2020-10-21 12:45, Sadique Urf Arbaz Sayyed wrote: > > On Wednesday, October 21, 2020 at 4:14:05 PM UTC+5:30, Sadique Urf Arbaz > > Sayyed wrote: > >> We started with a brand new windows server 2019 datacenter edition and > >> installed an infrastructure monitoring agent on it and strictly no other > >> program. The machine had 8 GB of memory. As part of monitoring NTP offset > >> from sync'd host we scheduled a ntpqexe. The problem started after 4-5 > >> days, the memory utilisation had increased to significant level >80%. On > >> analysis we found it was a gradual increase and using RAMMAP we saw every > >> time the ntpq.exe will run it will leave behind 24k of memory in PAGE > >> Table with 0 B in Private. Moreover this issue is specific to windows > >> server 2019 we tried following same steps on windows server 2012 machine > >> and it worked perfectly fine with no memory creeping issues. > >> > >> Any help or pointer are appreciated > > > > Full problem is mentioned here > > https://social.technet.microsoft.com/Forums/en-US/d45ba91a-cc43-4010-9a2c-d65b1a4cc33a/windows-2019-server-ntp-increases-memory-utilisation-by-acquiring-page-table?forum=ws2019 > > > > > From what you have posted above (I have not checked the MS forum) it > sounds like the leak is of "PAGE"-d memory consumption, not page tables > (also known in the NT world as hyperspace). > > Question is what is actually registered as owning that page-able memory. > Maybe you have found a memory leak in the NTP server itself, maybe > something in NT 10.??.2019?? (server 2019) leaks memory every time ntpq > makes an UDP socket connection to the NTP service to query it. > > There are some advanced tools for checking what owns outstanding global > (kernel) memory allocations on NT systems. However first you should > check the "commit charge" of the NTP service process, as that is a > direct measure of how much pageable virtual memory is allocated in the > user mode part of that process. > > > Enjoy > > Jakob > -- > Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com > Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10 > This public discussion message is non-binding and may contain errors. > WiseMo - Remote Service Management for PCs, Phones and Embedded Ok so this NT server is used by our customers but in house I do not have any server. I'm simply running ntpq.exe -nc peers 127.0.0.1, here I am not expecting the output but why is it hogging onto memory. As you pointed out ntpq uses the UDP port 123 which is by default used by Windows Time service. And this issue does not occur if I have the windows time service disabled. So my question here is why does the process hogs onto memory if I have the Windows time service running (In windows 2019 it's running by default). ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Ntpq.exe memory issue with windows 2019
On Wednesday, October 21, 2020 at 8:02:33 PM UTC+5:30, Jakob Bohm wrote: > On 2020-10-21 15:16, Sadique Urf Arbaz Sayyed wrote: > > On Wednesday, October 21, 2020 at 6:19:21 PM UTC+5:30, Jakob Bohm wrote: > >> On 2020-10-21 12:45, Sadique Urf Arbaz Sayyed wrote: > >>> On Wednesday, October 21, 2020 at 4:14:05 PM UTC+5:30, Sadique Urf Arbaz > >>> Sayyed wrote: > We started with a brand new windows server 2019 datacenter edition and > installed an infrastructure monitoring agent on it and strictly no other > program. The machine had 8 GB of memory. As part of monitoring NTP > offset from sync'd host we scheduled a ntpqexe. The problem started > after 4-5 days, the memory utilisation had increased to significant > level >80%. On analysis we found it was a gradual increase and using > RAMMAP we saw every time the ntpq.exe will run it will leave behind 24k > of memory in PAGE Table with 0 B in Private. Moreover this issue is > specific to windows server 2019 we tried following same steps on windows > server 2012 machine and it worked perfectly fine with no memory creeping > issues. > > Any help or pointer are appreciated > >>> > >>> Full problem is mentioned here > >>> https://social.technet.microsoft.com/Forums/en-US/d45ba91a-cc43-4010-9a2c-d65b1a4cc33a/windows-2019-server-ntp-increases-memory-utilisation-by-acquiring-page-table?forum=ws2019 > >>> > >>> > >> From what you have posted above (I have not checked the MS forum) it > >> sounds like the leak is of "PAGE"-d memory consumption, not page tables > >> (also known in the NT world as hyperspace). > >> > >> Question is what is actually registered as owning that page-able memory. > >> Maybe you have found a memory leak in the NTP server itself, maybe > >> something in NT 10.??.2019?? (server 2019) leaks memory every time ntpq > >> makes an UDP socket connection to the NTP service to query it. > >> > >> There are some advanced tools for checking what owns outstanding global > >> (kernel) memory allocations on NT systems. However first you should > >> check the "commit charge" of the NTP service process, as that is a > >> direct measure of how much pageable virtual memory is allocated in the > >> user mode part of that process. > >> > >> > >> Enjoy > >> > >> Jakob > >> -- > >> Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com > >> Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10 > >> This public discussion message is non-binding and may contain errors. > >> WiseMo - Remote Service Management for PCs, Phones and Embedded > > > > Ok so this NT server is used by our customers but in house I do not have > > any server. I'm simply running ntpq.exe -nc peers 127.0.0.1, here I am not > > expecting the output but why is it hogging onto memory. As you pointed out > > ntpq uses the UDP port 123 which is by default used by Windows Time > > service. And this issue does not occur if I have the windows time service > > disabled. So my question here is why does the process hogs onto memory if I > > have the Windows time service running (In windows 2019 it's running by > > default). > > > ntpq.exe is designed to talk to ntpd.exe from the same source > code bundle. Microsoft's NTP service (win32time) probably > completely lacks the ability to correctly handle that query. > Enjoy > > Jakob > -- > Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com > Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10 > This public discussion message is non-binding and may contain errors. > WiseMo - Remote Service Management for PCs, Phones and Embedded Yes Correct, But John here suppose my ntpd is running on a server and from the client side (Suppose windows time service is running) I run ntpq.exe with above command I do see the issue. The process will hold the 24Kb but actually the process is not visible in Task manager or anywhere but in RamMap we see that the executed process still occupied the Page table memory. So each query fired will consume 24KB and server will eventually see the 100% memory utilisation. I even had a call with Microsoft support they confirmed that this would be the application specific issue where caching is not cleared properly. I had the similar memory issue with IPMI utils as well and they provided me with the Hotfix. ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
[ntp:questions] Ntpq.exe memory issue with windows 2019
We started with a brand new windows server 2019 datacenter edition and installed an infrastructure monitoring agent on it and strictly no other program. The machine had 8 GB of memory. As part of monitoring NTP offset from sync'd host we scheduled a ntpqexe. The problem started after 4-5 days, the memory utilisation had increased to significant level >80%. On analysis we found it was a gradual increase and using RAMMAP we saw every time the ntpq.exe will run it will leave behind 24k of memory in PAGE Table with 0 B in Private. Moreover this issue is specific to windows server 2019 we tried following same steps on windows server 2012 machine and it worked perfectly fine with no memory creeping issues. Any help or pointer are appreciated ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Ntpq.exe memory issue with windows 2019
On Wednesday, October 21, 2020 at 4:14:05 PM UTC+5:30, Sadique Urf Arbaz Sayyed wrote: > We started with a brand new windows server 2019 datacenter edition and > installed an infrastructure monitoring agent on it and strictly no other > program. The machine had 8 GB of memory. As part of monitoring NTP offset > from sync'd host we scheduled a ntpqexe. The problem started after 4-5 days, > the memory utilisation had increased to significant level >80%. On analysis > we found it was a gradual increase and using RAMMAP we saw every time the > ntpq.exe will run it will leave behind 24k of memory in PAGE Table with 0 B > in Private. Moreover this issue is specific to windows server 2019 we tried > following same steps on windows server 2012 machine and it worked perfectly > fine with no memory creeping issues. > > Any help or pointer are appreciated Full problem is mentioned here https://social.technet.microsoft.com/Forums/en-US/d45ba91a-cc43-4010-9a2c-d65b1a4cc33a/windows-2019-server-ntp-increases-memory-utilisation-by-acquiring-page-table?forum=ws2019 ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Ntpq.exe memory issue with windows 2019
On 21/10/2020 15:44, Sadique Urf Arbaz Sayyed wrote: The process will hold the 24Kb but actually the process is not visible in Task manager or anywhere but in RamMap we see that the executed process still occupied the Page table memory. Did you mean 24KB, or is that really 3KB? Any memory leakage that persists after the process is no longer visible in the process table is down to the OS. As already pointed out page table allocations are unlikely to be what that memory is. I'm not familiar with the details of Window NT family memory use reporting, but the only legitimate carry over would be for cached pages. Such pages are optimisations and don't deny memory to other processes. Both Linux and Windows try and allocate nearly all their memory to something. In the case of the Linux "free" command, it computes an available figure which is much larger than the free figure, by including memory that can safely be discarded. If there isn't an OS bug, I wonder if you are seeing that effect. One wild thought: is ntpq or one of its DLLs not position independent code? I could speculate as to why Windows might keep relocated pages around in case the code is reloaded at the same address before the page gets reused for other reasons. However, I don't actually know if Windows does something like that. ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Ntpq.exe memory issue with windows 2019
On 2020-10-21 15:16, Sadique Urf Arbaz Sayyed wrote: On Wednesday, October 21, 2020 at 6:19:21 PM UTC+5:30, Jakob Bohm wrote: On 2020-10-21 12:45, Sadique Urf Arbaz Sayyed wrote: On Wednesday, October 21, 2020 at 4:14:05 PM UTC+5:30, Sadique Urf Arbaz Sayyed wrote: We started with a brand new windows server 2019 datacenter edition and installed an infrastructure monitoring agent on it and strictly no other program. The machine had 8 GB of memory. As part of monitoring NTP offset from sync'd host we scheduled a ntpqexe. The problem started after 4-5 days, the memory utilisation had increased to significant level >80%. On analysis we found it was a gradual increase and using RAMMAP we saw every time the ntpq.exe will run it will leave behind 24k of memory in PAGE Table with 0 B in Private. Moreover this issue is specific to windows server 2019 we tried following same steps on windows server 2012 machine and it worked perfectly fine with no memory creeping issues. Any help or pointer are appreciated Full problem is mentioned here https://social.technet.microsoft.com/Forums/en-US/d45ba91a-cc43-4010-9a2c-d65b1a4cc33a/windows-2019-server-ntp-increases-memory-utilisation-by-acquiring-page-table?forum=ws2019 From what you have posted above (I have not checked the MS forum) it sounds like the leak is of "PAGE"-d memory consumption, not page tables (also known in the NT world as hyperspace). Question is what is actually registered as owning that page-able memory. Maybe you have found a memory leak in the NTP server itself, maybe something in NT 10.??.2019?? (server 2019) leaks memory every time ntpq makes an UDP socket connection to the NTP service to query it. There are some advanced tools for checking what owns outstanding global (kernel) memory allocations on NT systems. However first you should check the "commit charge" of the NTP service process, as that is a direct measure of how much pageable virtual memory is allocated in the user mode part of that process. Enjoy Jakob -- Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10 This public discussion message is non-binding and may contain errors. WiseMo - Remote Service Management for PCs, Phones and Embedded Ok so this NT server is used by our customers but in house I do not have any server. I'm simply running ntpq.exe -nc peers 127.0.0.1, here I am not expecting the output but why is it hogging onto memory. As you pointed out ntpq uses the UDP port 123 which is by default used by Windows Time service. And this issue does not occur if I have the windows time service disabled. So my question here is why does the process hogs onto memory if I have the Windows time service running (In windows 2019 it's running by default). ntpq.exe is designed to talk to ntpd.exe from the same source code bundle. Microsoft's NTP service (win32time) probably completely lacks the ability to correctly handle that query. Enjoy Jakob -- Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10 This public discussion message is non-binding and may contain errors. WiseMo - Remote Service Management for PCs, Phones and Embedded ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
Re: [ntp:questions] Ntpq.exe memory issue with windows 2019
On 2020-10-21 12:45, Sadique Urf Arbaz Sayyed wrote: On Wednesday, October 21, 2020 at 4:14:05 PM UTC+5:30, Sadique Urf Arbaz Sayyed wrote: We started with a brand new windows server 2019 datacenter edition and installed an infrastructure monitoring agent on it and strictly no other program. The machine had 8 GB of memory. As part of monitoring NTP offset from sync'd host we scheduled a ntpqexe. The problem started after 4-5 days, the memory utilisation had increased to significant level >80%. On analysis we found it was a gradual increase and using RAMMAP we saw every time the ntpq.exe will run it will leave behind 24k of memory in PAGE Table with 0 B in Private. Moreover this issue is specific to windows server 2019 we tried following same steps on windows server 2012 machine and it worked perfectly fine with no memory creeping issues. Any help or pointer are appreciated Full problem is mentioned here https://social.technet.microsoft.com/Forums/en-US/d45ba91a-cc43-4010-9a2c-d65b1a4cc33a/windows-2019-server-ntp-increases-memory-utilisation-by-acquiring-page-table?forum=ws2019 From what you have posted above (I have not checked the MS forum) it sounds like the leak is of "PAGE"-d memory consumption, not page tables (also known in the NT world as hyperspace). Question is what is actually registered as owning that page-able memory. Maybe you have found a memory leak in the NTP server itself, maybe something in NT 10.??.2019?? (server 2019) leaks memory every time ntpq makes an UDP socket connection to the NTP service to query it. There are some advanced tools for checking what owns outstanding global (kernel) memory allocations on NT systems. However first you should check the "commit charge" of the NTP service process, as that is a direct measure of how much pageable virtual memory is allocated in the user mode part of that process. Enjoy Jakob -- Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10 This public discussion message is non-binding and may contain errors. WiseMo - Remote Service Management for PCs, Phones and Embedded ___ questions mailing list questions@lists.ntp.org http://lists.ntp.org/listinfo/questions
