Re: [OmniOS-discuss] ILB memory leak?

> On Nov 10, 2015, at 2:50 AM, Al Slater wrote: > > On 10/11/2015 07:40, Al Slater wrote: >> It seems to me that ilbd_run_probe just needs to call >> posix_spawn_file_actions_destroy appropriately. > > And probably posix_spawnattr_destroy as well? Wow! Great catch. I'll

Re: [OmniOS-discuss] ILB memory leak?

On 10/11/15 15:26, Dan McDonald wrote: > >> On Nov 10, 2015, at 2:50 AM, Al Slater wrote: >> >> On 10/11/2015 07:40, Al Slater wrote: >>> It seems to me that ilbd_run_probe just needs to call >>> posix_spawn_file_actions_destroy appropriately. >> >> And probably

Re: [OmniOS-discuss] ILB memory leak?

Hi Dan, On 06/11/2015 18:31, Dan McDonald wrote: You said you had a test box, right? Yes. Can you: - Disable UMEM_DEBUG - RESTART the service. - IMMEDIATELY after restart do pmap, and do pmap once per (sec, 10 sec, something) to see how it grows? Attached is a compressed file with 5hrs

Re: [OmniOS-discuss] ILB memory leak?

On 09/11/15 15:43, Dan McDonald wrote: > >> On Nov 9, 2015, at 8:39 AM, Al Slater wrote: >> >> Attached is a compressed file with 5hrs or so of 10s pmaps. >> Hopefully not too big for the list. > > It compressed nicely. I'm noticing a pattern: > > Mon Nov 9 08:21:45 UTC

Re: [OmniOS-discuss] ILB memory leak?

On 05/11/2015 14:57, Dan McDonald wrote: On Nov 5, 2015, at 6:38 AM, Al Slater wrote: I have the 4Gb core file. Is there anything useful I can extract from it to try and spot where the problem is? Your one ::findleaks showed nothing. Did your 4GB corefile have

Re: [OmniOS-discuss] ILB memory leak?

> On Nov 6, 2015, at 3:11 AM, Al Slater wrote: > > On 05/11/2015 14:57, Dan McDonald wrote: >> >>> On Nov 5, 2015, at 6:38 AM, Al Slater wrote: >>> >>> I have the 4Gb core file. Is there anything useful I can extract from >>> it to try and spot

Re: [OmniOS-discuss] ILB memory leak?

> On Nov 6, 2015, at 9:39 AM, Dan McDonald wrote: > > Lots of LARGE anonymous mappings. I wonder why that happened? I'll dig into > that a bit more. pmap(1) works even better on running processes. Could you run, say "pmap -xa `pgrep ilbd`" on your running machine? Dan

Re: [OmniOS-discuss] ILB memory leak?

On 06/11/15 14:51, Dan McDonald wrote: > >> On Nov 6, 2015, at 9:39 AM, Dan McDonald wrote: >> >> Lots of LARGE anonymous mappings. I wonder why that happened? I'll dig into >> that a bit more. > > pmap(1) works even better on running processes. Could you run, say "pmap

Re: [OmniOS-discuss] ILB memory leak?

> On Nov 6, 2015, at 10:57 AM, Al Slater wrote: > > > 7D80 1048576 1048576 1048576 - rwx--[ anon ] > BDA0 524288 524288 524288 - rwx--[ anon ] > DDC0 262144 262144 262144 - rwx--[ anon ] > EDE0 131072 131072 131072

Re: [OmniOS-discuss] ILB memory leak?

> On Nov 6, 2015, at 11:25 AM, Dan McDonald wrote: > >> On Nov 6, 2015, at 10:57 AM, Al Slater wrote: >> >> >> 7D80 1048576 1048576 1048576 - rwx--[ anon ] >> BDA0 524288 524288 524288 - rwx--[ anon ] >> DDC0 262144

Re: [OmniOS-discuss] ILB memory leak?

On Fri, 6 Nov 2015, Dan McDonald wrote: More huge anonymous mappings (1G, 512MB, 256MB, 128MB). I don't know pmap as well as I should. I don't see anything in the man page to give me further insight into why these chunks of memory are being eaten. It is pretty common for memory allocators

Re: [OmniOS-discuss] ILB memory leak?

To the mailing list as well... On 22/10/2015 09:43, Al Slater wrote: > On 21/10/2015 17:35, Dan McDonald wrote: >> >>> On Oct 21, 2015, at 6:08 AM, Al Slater >>> wrote: >>> >>> Hi, >>> >>> I am running omnios r151014 on a couple of machines with a couple >>> of zones each.

Re: [OmniOS-discuss] ILB memory leak?

Hi Dan, On 05/11/2015 14:57, Dan McDonald wrote: On Nov 5, 2015, at 6:38 AM, Al Slater wrote: I have the 4Gb core file. Is there anything useful I can extract from it to try and spot where the problem is? Your one ::findleaks showed nothing. Did your 4GB corefile

Re: [OmniOS-discuss] ILB memory leak?

> On Nov 5, 2015, at 6:38 AM, Al Slater wrote: > > I have the 4Gb core file. Is there anything useful I can extract from > it to try and spot where the problem is? Your one ::findleaks showed nothing. Did your 4GB corefile have ::findleaks show nothing as well?

Re: [OmniOS-discuss] ILB memory leak?

Le 22/10/15 10:43, Al Slater a écrit : > I am seeing kernel memory consumption increasing as well, but that may > be a different issue. The ilbd process memory is definitely growing. > this is indeed probably a different issue, but it would be useful to create a thread on illumos discuss as

Re: [OmniOS-discuss] ILB memory leak?

On 21/10/2015 17:35, Dan McDonald wrote: On Oct 21, 2015, at 6:08 AM, Al Slater wrote: Hi, I am running omnios r151014 on a couple of machines with a couple of zones each. 1 zone runs apache as an SSL reverse proxy, the other runs ILB for load balancing web to app tier

Re: [OmniOS-discuss] ILB memory leak?

Al Slater writes: > On 21/10/2015 17:35, Dan McDonald wrote: >> >> That should enable user-level memory debugging. If you get a >> coredump, save it and share it. If you don't and the ilb daemon >> keeps running, eventually please: >> >> gcore `pgrep ilbd` >> >> and share THAT corefile. You

Re: [OmniOS-discuss] ILB memory leak?

> On Oct 21, 2015, at 6:08 AM, Al Slater wrote: > > Hi, > > I am running omnios r151014 on a couple of machines with a couple of zones > each. 1 zone runs apache as an SSL reverse proxy, the other runs ILB for > load balancing web to app tier connections. > > I noticed

Re: [OmniOS-discuss] ILB memory leak?

On Wed, 21 Oct 2015, Dan McDonald wrote: You can use svccfg(1M) to enable user-level memory debugging on ilb. It may cause the ilb daemon to dump core. (And you're just noticing this in the process, not kernel memory consumption, correct?) As root: svcadm disable -t ilb

[OmniOS-discuss] ILB memory leak?

Hi, I am running omnios r151014 on a couple of machines with a couple of zones each. 1 zone runs apache as an SSL reverse proxy, the other runs ILB for load balancing web to app tier connections. I noticed that in the ILB zone, the ilbd process memory grows to about 2Gb. Restarting ILB