Hi przemol,
Bellow output of plockstat for malloc and libumem. Both many locks.
Why changing to libumem I didn't get less locks ?
********** Plockstat using malloc (many many locks):
Mutex hold
Count nsec Lock Caller
----------------------------------------------------------------------------
---
4036 14391 libc.so.1`libc_malloc_lock
libjava.so`JNU_GetStringPlatformChars+0x290
2118 7385 libc.so.1`libc_malloc_lock
libjava.so`JNU_ReleaseStringPlatformChars+0x18
3174 4700 libc.so.1`libc_malloc_lock
libjvm.so`__1cCosGmalloc6FI_pv_+0x29
181 63407 libc.so.1`_uberdata+0x40
libjvm.so`__1cCosRpd_suspend_thread6FpnGThread_i_i_+0x46
3170 3407 libc.so.1`libc_malloc_lock
libjvm.so`__1cCosEfree6Fpv_v_+0x18
588 12443 libc.so.1`libc_malloc_lock
libnet.so`Java_java_net_SocketOutputStream_socketWrite0+0x7f
172 37572 libc.so.1`_uberdata+0x40
libjvm.so`__1cCosQpd_resume_thread6FpnGThread__i_+0x1f
176 26701 libc.so.1`libc_malloc_lock
libjvm.so`__1cCosGmalloc6FI_pv_+0x29
596 7124 libc.so.1`libc_malloc_lock
libnet.so`Java_java_net_SocketOutputStream_socketWrite0+0x1e0
450 7254 libc.so.1`libc_malloc_lock 0x858b2167
(...)
********** Plockstat using libumem (many many locks too):
Mutex hold
Count nsec Lock Caller
----------------------------------------------------------------------------
---
4 1450455 libumem.so.1`umem_cache_lock
libumem.so.1`umem_cache_applyall+0x51
100 46388 libc.so.1`_uberdata+0x40
libjvm.so`__1cCosRpd_suspend_thread6FpnGThread_i_i_+0x46
100 23226 libc.so.1`_uberdata+0x40
libjvm.so`__1cCosQpd_resume_thread6FpnGThread__i_+0x1f
486 4314 0x807b680
libumem.so.1`umem_cache_alloc+0xc1
488 4236 0x807b680
libumem.so.1`umem_cache_free+0x194
356 4859 0x807b300
libumem.so.1`umem_cache_alloc+0xc1
150 11499 0x8073030 libumem.so.1`vmem_xfree+0xfe
115 14473 0x8073030 libumem.so.1`vmem_alloc+0x13c
297 5374 0x807ad80
libumem.so.1`umem_cache_alloc+0xc1
362 4258 0x807b300
libumem.so.1`umem_cache_free+0x194
297 4304 0x807ad80
libumem.so.1`umem_cache_free+0x194
48 21805 0x807b6c0
libumem.so.1`umem_cache_free+0x194
150 6635 libumem.so.1`vmem0+0x30 libumem.so.1`vmem_alloc+0x126
(...)
Mutex hold
----------------------------------------------------------------------------
---
Count nsec Lock Caller
543 13110 0x807b680
libumem.so.1`umem_cache_alloc+0xc1
nsec ---- Time Distribution --- count Stack
8192 |@@@@@@@@@@@@ | 289
libumem.so.1`umem_cache_alloc+0xc1
16384 |@@@@@@@@@ | 224 libumem.so.1`umem_alloc+0x3f
32768 |@ | 27 libumem.so.1`malloc+0x23
65536 | | 3
libjava.so`JNU_GetStringPlatformChars+0x290
0x20ac
----------------------------------------------------------------------------
---
Count nsec Lock Caller
78 89901 libc.so.1`_uberdata+0x40
libjvm.so`__1cCosRpd_suspend_thread6FpnGThread_i_i_+0x46
nsec ---- Time Distribution --- count Stack
65536 |@@@@@@@@@@@@@@@@@ | 57 libc.so.1`fork_lock_exit+0x2f
131072 |@@@@@ | 19 libc.so.1`_thrp_suspend+0x22c
262144 | | 1 libc.so.1`thr_suspend+0x1a
524288 | | 1
libjvm.so`__1cCosRpd_suspend_thread6FpnGThread_i_i_+0x46
libjvm.so`__1cGThreadNdo_vm_suspend6M_i_+0x42
libjvm.so`__1cGThreadKvm_suspend6M_i_+0x33
libjvm.so`__1cUThreadSafepointStateXexamine_state_of_thread6Mi_v_+0x101
libjvm.so`__1cUSafepointSynchronizeFbegin6F_v_+0x12b
libjvm.so`__1cIVMThreadEloop6M_v_+0x1b4
(...)
Regards.
Kleyson Rios.
-----Mensagem original-----
De: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] Em nome de
[EMAIL PROTECTED]
Enviada em: sexta-feira, 18 de abril de 2008 06:49
Para: [email protected]
Assunto: Re: [dtrace-discuss] Process in LCK / SLP (Please)
On Wed, Apr 16, 2008 at 12:35:53PM -0300, Kleyson Rios wrote:
> Hi,
>
>
>
> I really need help.
>
>
>
> How can i identify why my process are 100% in LCK and SLP ?
Could you please use 'plockstat' and show us the output ?
E.g.: plockstat -e 30 -s 10 -A -p `<pid of java proc>'
We had such case in the past. I don't think it is the same reason but might
give you some ideas.
We had a lot of locks. By using plockstat it turned out that those locks
comes from malloc. Using mtmalloc or libumem reduced a number of locks but
it was still a huge number (and eating our CPU). By using dtrace it turned
out that those locks comes from threads which were ... supposed to be
killed. Our programmers found a bug in the application (already unnecessary
threads were still alive and all of the them going in the endless loop),
changed the code and ... all locks went away.
As Jim said: start with watching who is using mallocs (which thread(s)).
Regards
przemol
--
http://przemol.blogspot.com/
----------------------------------------------------------------------
Poprowadz swoj klub do zwyciestwa!
Sprawdz >>> http://link.interia.pl/f1d76
_______________________________________________
dtrace-discuss mailing list
[email protected]
_______________________________________________
dtrace-discuss mailing list
[email protected]
