Re: [dtrace-discuss] Hotspot provider probes not getting cleaned up
On Jan 4, 2008 8:49 AM, Keith McGuigan <[EMAIL PROTECTED]> wrote: > Matty wrote: > > > > I am not 100% sure what the PID was, but I am certain there were no > > Java processes running (at least according to ps). After I rebooted > > the server, everything worked flawlessly. Unless anyone has any ideas, > > I will open up a ticket with Sun support. This appears to be some sort > > of bug. > > > The hotspot probes you referenced aren't tied to the 'java' executable. > They're created when the 'libjvm.so' library is loaded (they are > traditional USDT probes in that library). The 'java' executable isn't > the only thing that loads libjvm.so. So not having 'ps' list a 'java' > process, while dtrace still shows some 'hotspot*' probes doesn't > necessarily indicate a problem. Your reboot may have fixed things > because it terminated some other unrelated process that may have loaded > libjvm.so (perhaps a browser?) I am 100% certain that the 'java' executable is the only thing that makes use of libjvm.so on this specific server. Since my script didn't work prior to me rebooting the box, and it started working after I rebooted the box, it would appear to me that some type of bug reared its head. > Just make sure to take a close look at the pid of the java process > you're starting (and stopping), and make sure that the hotspot > provider of the leftover probes matches that pid. What were you doing > that caused this situation to occur? We have bumped into a couple bugs in the JVM CMS collector that cause our Java processes to fail hard. This is the only thing I can think of that might lead to this behavior. Thanks for the feedback, - Ryan -- UNIX Administrator http://prefetch.net ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] Getting total memory allocated so far
Thanks for the update. You probably already know this, but... > > 2. I'm trying to track the memory allocations/frees relative to > specific events in the code (e.g. someone getting hit by a projectile, > etc). The granularity for this stuff is pretty tight. Some state > variables linked to the place in the code where the program is (single > threaded) when the malloc/free occurs tells me a lot. Makes it easy > to tell which portions of the program are responsible for most of the > memory allocations. Grabbing a ustack() in the pid:libc:malloc:entry and pid:libc:free:entry probes is your friend here... Thanks, /jim ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
[dtrace-discuss] How do I print wide char string arguments!
Hopefully this is a straight forward issue and I'm missing
something obvious.
I'm trying to print a wchar_t [] argument from a simple example
program like
#include
#include
#include
void
prout(wchar_t *wstr)
{
}
int
main()
{
wchar_t name[]=L"Angelo Rajadurai";
int i;
for(i=0; i<1000 ; i++)
{
prout(name);
sleep(1);
}
}
With a D script like
#!/usr/sbin/dtrace -s
pid$target::prout:entry{
printf("Argument of %s is %s\n",probefunc,copyinstr(arg0));
}
but it just prints the first char of the string "A"
I'm aware of the %ls and %ws format options in printf() but for
the life of me I'm unable to typecast copyin() into a whcar_t[]
So help! I think I need a copyinwstr() or wstringof() equiv!
Thanks
Angelo
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] Obtaining file attributes from open()
On Fri, Jan 04, 2008 at 07:08:55PM +0100, Martin Carpenter wrote: > > Hello all, > > I'm trying to determine the file attributes (mode, owner, group) from an > open() via the syscall probe. I have found fds[] and fileinfo_t and also > > curthread->t_procp->p_user.u_finfo.fi_list[arg0].uf_file->f_vnode > > which gets me as far as the vnode. But I don't see how to get to struct > vattr from there, or if there is a function that will enable me to access > these in the predicate. > > Many thanks for any illumination, > > Martin. There is no way to statically examine that data because it's up to each filesystem to implement a function, VOP_GETATTR(), that fills that structure in, and there is no requirement everything in sitting in an in-core vattr_t already. And if the object is not in core, that function will need to read it in from disk etc. So you can't generically use DTrace to examine in-memory data and be assured of getting this data. You'll need to look at the code for the underlying filesystem of your open() and, assuming all the data is in-memory, write a function for that filesystem to get what you need. As one example if you look at ufs_getattr(), you will see that you can do a relatively straightforward conversion from f_vnode to the ufs inode structure, and from there get at i_uid, i_gid, i_mode, etc. ZFS is more complicated due to the use of FUIDs, but in the common case you should be able to get what you need from the znode. -Mike -- Mike Shapiro, Solaris Kernel Development. blogs.sun.com/mws/ ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
[dtrace-discuss] Obtaining file attributes from open()
Hello all, I'm trying to determine the file attributes (mode, owner, group) from an open() via the syscall probe. I have found fds[] and fileinfo_t and also curthread->t_procp->p_user.u_finfo.fi_list[arg0].uf_file->f_vnode which gets me as far as the vnode. But I don't see how to get to struct vattr from there, or if there is a function that will enable me to access these in the predicate. Many thanks for any illumination, Martin. ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] Hotspot provider probes not getting cleaned up
On Jan 3, 2008 10:25 PM, Matty <[EMAIL PROTECTED]> wrote: > I am not 100% sure what the PID was, but I am certain there were no > Java processes running (at least according to ps). After I rebooted > the server, everything worked flawlessly. Unless anyone has any ideas, > I will open up a ticket with Sun support. This appears to be some sort > of bug. C programs that have USDT probes get linked with drti.o. Notice that to register and unregister the USDT probes, drti.c[1] does the following in ini and fini functions. fd = open(/dev/dtrace/helper, ...) ioctl(fd, ...) close(fd) Assuming that Java is using this same mechanism, it suggests to me that the fini function was not called or failed. Would it be called if the java process was killed in a bad way (e.g. SIGSEGV, SIGKILL)? Is it possible that the process could be out of file descriptors when the open() call happens in fini? Perhaps there is a need (maybe already exists) a hook in the kernel code called during process cleanup to do the equivalent of ioctl(fd, DTRACEHIOC_REMOVE, gen) from within the kernel just in case the user space code fails. 1.http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/lib/libdtrace/common/drti.c -- Mike Gerdts http://mgerdts.blogspot.com/ ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] Hotspot provider probes not getting cleaned up
Matty wrote: > > I am not 100% sure what the PID was, but I am certain there were no > Java processes running (at least according to ps). After I rebooted > the server, everything worked flawlessly. Unless anyone has any ideas, > I will open up a ticket with Sun support. This appears to be some sort > of bug. > The hotspot probes you referenced aren't tied to the 'java' executable. They're created when the 'libjvm.so' library is loaded (they are traditional USDT probes in that library). The 'java' executable isn't the only thing that loads libjvm.so. So not having 'ps' list a 'java' process, while dtrace still shows some 'hotspot*' probes doesn't necessarily indicate a problem. Your reboot may have fixed things because it terminated some other unrelated process that may have loaded libjvm.so (perhaps a browser?) Just make sure to take a close look at the pid of the java process you're starting (and stopping), and make sure that the hotspot provider of the leftover probes matches that pid. What were you doing that caused this situation to occur? -- - Keith ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] Please comment on this
Matty wrote: > On Nov 25, 2007 4:42 PM, Adam Leventhal <[EMAIL PROTECTED]> wrote: > >> The dvm provider can be quite intrusive. You might try upgrading to the >> 1.6 JVM which has native support for the hotspot provider. That said, it's >> still quite expensive. >> > > Are there any efforts ongoing to decrease the probe effect ? We have > attempted to > use the hotspot provider in production in the past, but the overhead has > proven > to be a bit too much. On some of out mildly busy applicaiton servers we saw > the CPU load jump from 10% utilization to 100% utilization when we enabled > the DTraceMethodProbes probes. We would _LOVE_ be able to use the > hotspot provider in production, but the runtime overhead currently prevents > us from doing so. > We have some initial plans for doing something more intelligent than turning on all of the methods all of the time, and turning them on dynamically as needed, but we're limited by our available resources. If you (or anyone on this list) would be interested in helping us out in the effort, please speak up! -- - Keith ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
