Re: [dtrace-discuss] Is the nfs dtrace script right (from nfsv3 provider wiki)?
Marcelo Leal wrote: Marcelo Leal wrote: Hello all... Thanks a lot for the answers! I think the problem is almost fixed. Every dtrace documentation says to use predicates to guarantee the relation between the start/done probes... Max was the only one paying attention reading the docs. ;-) But i'm still getting weird numbers: Which numbers don't look right? 3 of your reads took between 2 and 4 milliseconds, most were between 8 and 16 nanoseconds. 21 writes took between 2 and 4 milliseconds, the most amount of time spent doing read/write by host is about 1.2 seconds, and teste/file10 took about 1.1 seconds. Looks pretty good to me.(?). I'm curious about what you were expecting to see. The problem is the total numbers... 1267135728 and 1126991407, for example. 21 and 19 minutes in a ten minutes trace. Or am i missing something? When I do the arithmetic, I get about 1.2 seconds for the first number, and 1.1 seconds for the second number. These numbers are in nanoseconds, no? So, 1267135728/(1000*1000*1000) = 1.267... seconds. max Leal [http://www.eall.com.br/blog] Wed Dec 10 08:36:33 BRST 2008 Wed Dec 10 08:46:55 BRST 2008 cut here - Tracing... Hit Ctrl-C to end. ^C NFSv3 read/write distributions (us): read value - Distribution - count 2 | 0 631 @@@ 145603 16 |@ 155926 15970 6111 942 372 883 1649 1090 8278 24605 8868 1694 304 63 27 31 43 3 0 value - Distribution - count 128 | 0 1083 @@@ 32622 1024 |@ 70353 70851 47906 44898 20481 5633 1605 1339 957 380 143 21 0 otal us): x.16.0.x 647019 x.16.0.x 734488 x.16.0.x 0890034 x.16.0.x 8852624 x.16.0.x 0407241 x.16.0.x 9028592 x.16.0.x 3013688 x.16.0.x 04045281 x.16.0.x 05245138 x.16.0.x 24286383 x.16.0.x 54526695 x.16.0.x 94419023 x.16.0.x 21794650 x.16.0.x 59302970 x.16.0.x 89694542 x.16.0.x 90207418 x.16.0.x 87983050 x.16.0.x 267135728 NFSv3 read/write top 10 files (total us): /teste/file1 95870303 /teste/file2 104212948 /teste/file3 104311607 /teste/file4 121076447 /teste/file5 137687236 /teste/file6 160895273 /teste/file7 180765880 /teste/file8 198827114 /teste/file9 372380414 /teste/file10 1126991407 -- cut here -- Max, will be difficult disable processors on that machine (production). Yes. I understand. Regards, max Thanks again! Leal [http://www.eall.com.br/blog] ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] Is the nfs dtrace script right (from nfsv3 provider wiki)?
Marcelo Leal wrote: Hello all... Thanks a lot for the answers! I think the problem is almost fixed. Every dtrace documentation says to use predicates to guarantee the relation between the start/done probes... Max was the only one paying attention reading the docs. ;-) Actually, this is not from reading the docs. It is from being burnt a few times by getting impossible time values. By the way, I am replying to you and to the mailing list, but messages to you are getting bounced. But i'm still getting weird numbers: Which numbers don't look right? 3 of your reads took between 2 and 4 milliseconds, most were between 8 and 16 nanoseconds. 21 writes took between 2 and 4 milliseconds, the most amount of time spent doing read/write by host is about 1.2 seconds, and teste/file10 took about 1.1 seconds. Looks pretty good to me.(?). I'm curious about what you were expecting to see. Wed Dec 10 08:36:33 BRST 2008 Wed Dec 10 08:46:55 BRST 2008 cut here - Tracing... Hit Ctrl-C to end. ^C NFSv3 read/write distributions (us): read value - Distribution - count 2 | 0 4 | 631 8 | 145603 16 |@155926 32 |@@ 15970 64 |@6111 128 | 942 256 | 372 512 | 883 1024 | 1649 2048 | 1090 4096 |@8278 8192 |@@@ 24605 16384 |@8868 32768 | 1694 65536 | 304 131072 | 63 262144 | 27 524288 | 31 1048576 | 43 2097152 | 3 4194304 | 0 write value - Distribution - count 128 | 0 256 | 1083 512 | 32622 1024 |@70353 2048 |@@ 70851 4096 |@@ 47906 8192 |@@ 44898 16384 |@@@ 20481 32768 |@5633 65536 | 1605 131072 | 1339 262144 | 957 524288 | 380 1048576 | 143 2097152 | 21 4194304 | 0 NFSv3 read/write by host (total us): x.16.0.x 3647019 x.16.0.x 8734488 x.16.0.x 50890034 x.16.0.x 68852624 x.16.0.x 70407241 x.16.0.x 79028592 x.16.0.x 83013688 x.16.0.x 104045281 x.16.0.x 105245138 x.16.0.x 124286383 x.16.0.x154526695 x.16.0.x 194419023 x.16.0.x 221794650 x.16.0.x 259302970 x.16.0.x 289694542 x.16.0.x 290207418 x.16.0.x 487983050 x.16.0.x
Re: [dtrace-discuss] Is the nfs dtrace script right (from nfsv3 provider wiki)?
Hi Marcelo, Marcelo Leal wrote: I think (us) is microseconds. There is one division by 1000 on the source code... Oops. You're right. I did not see that. (That might explain the 4-8 nanosecond I/Os, which I did think seemed pretty fast. They are actually 4-8 microsecond). So, you want to know how you can spend 19 or 20 minutes in a 10 minute trace? You have multiple cpu's, so each cpu can be working in parallel on different I/O requests. If you have 8 cpu's, the average time spent by each cpu would be about 2.5 minutes. This does sound a little high to me, but not extreme. If you got 80 minutes, I would be concerned that all cpus are working on requests all the time. It might be difficult to correlate per cpu, as I suspect the start and done probe for a given I/O could fire on different cpus. max Leal [http://www.eall.com.br/blog] ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] Is the nfs dtrace script right (from nfsv3 provider wiki)?
Hi Marcelo, Marcelo Leal wrote: Ok, but that is a bug, or should work like that? We can not use dtrace on multiple processors systems? Sorry, but i don't get it... I don't consider this a bug. I think it depends on what you are trying to measure. The script you are using measures latency for read/write operations across all cpus. There is nothing wrong with the sum of the times being longer than the total time you are tracing, if there are multiple cpus. If you wanted to measure latency per cpu, the script would need to be changed. So, what are you trying to measure? I think more interesting is to find out why some I/O operations take longer than others. For this, you can use speculative tracing. Trace all function entries and returns, but only commit if the time spent is longer than the longest time found thus far. I'm sure others have ideas about this... max __ Leal [http://www.eall.com.br/blog] ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] Is the nfs dtrace script right (from nfsv3 provider wiki)?
Hi,
I have looked at the script, and there is no correspondence between
start and done.
So, I am not sure how this script is supposed to work.
I think there should be a predicate in the done probes...
The way the script is written, it assumes that for any start, the done
that fires after it is for the same noi_xid.
Current script:
nfsv3:::op-read-start,
nfsv3:::op-write-start
{
start[args[1]-noi_xid] = timestamp;
}
nfsv3:::op-read-done,
nfsv3:::op-write-done
{
...
Changed script:
nfsv3:::op-read-start,
nfsv3:::op-write-start
{
start[args[1]-noi_xid] = timestamp;
}
nfsv3:::op-read-done,
nfsv3:::op-write-done
/start[args[1]-noi_xid] != 0/
{
That way, you don't have the done probe clause executing
for id's where the start has not fired first. (This still does not
match start/done for a given xid).
But what do I know...
max
Jim Mauro wrote:
Also (I meant to ask) - are you having performance problems, or
just monitoring with the NFS provider scripts?
Thanks,
/jim
Marcelo Leal wrote:
Hello Jim, this is not a benchmark. The filenames i did change for privacy...
This is a NFS server, yes.
# uname -a
SunOS test 5.11 snv_89 i86pc i386 i86pc
# cat /etc/release
Solaris Express Community Edition snv_89 X86
Copyright 2008 Sun Microsystems, Inc. All Rights Reserved.
Use is subject to license terms.
Assembled 06 May 2008
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] tracemem question
Hi Chip,
Chip Bennett wrote:
First, the length on tracemem is a constant because the output from one
clause must be the same length every time it executes. With ASCII
strings, you can fake it, because you can set the maximum length of
strings to whatever you want. When the used length of a string varies
each time the same clause is executed, this isn't a problem because D
always records the full length of the string in the switch buffer, not
just the used length. The full length never varies.
However, the problem with this for binary data, is that all of the
string format items in printf stop when a null character ('\0') is
reached: not a good thing for binary data.
I have a possible solution, but I don't think you're going to like it.
I've thought about this before, and considered how I might solve it. If
you know the buffer you want to trace is between say 1 and 8000 bytes,
include enough additional probe specs and clauses for the same function
over and over to display the whole thing, but trace it 100 bytes at a
time. So for a maximum of 8000 bytes, you'd need 80 clauses. Then use
a counter in a predicate to limit the number of clauses executed for
each pass. It's crude, I know.
Yes, I thought of that. The problem is that I can not assume
that a given dump has, say 100 bytes in it. I would have to have a separate
probe for each byte (ugh). I think I'll end up dumping the maximum size
using tracemem, and then let an application whittle it down based
on a length I can place into the output.
I considered a profile probe that fires over and over with a predicate
that stops the output when the end of the buffer is reached, but the
buffer would likely be modified before you'd get a chance to get all of
the data traced.
We need a tracemem that has two parameters: buffer len, a variable, a
max length, a constant. Tracemem would then always record the full
length in the switch buffer, but only the actual data would be
displayed, along with the length.
Yes! That would be nice...
Thanks much,
max
Good luck!
Chip
-Original Message-
From: [EMAIL PROTECTED] [mailto:dtrace-discuss-
[EMAIL PROTECTED] On Behalf Of [EMAIL PROTECTED]
Sent: Friday, November 21, 2008 7:10 AM
To: dtrace-discuss@opensolaris.org
Subject: [dtrace-discuss] tracemem question
Hi All,
I am sure this has been asked before (in fact, I think I asked it over
2
years ago).
I am snooping tcp traffic on the e1000g using dtrace. Almost
everything
works (I print mac header,
ip header, and tcp header). I would like to use something like
tracemem() to dump the payload.
However, tracemem() won't let me specify anything but a constant as
the
length. Has anyone
succeeded in dumping an arbitrary number of hex bytes? What I want is
something
like:
tracemem(mp-b_rptr+offset, mp-b_wptr-(mp-b_rptr+offset));
Or maybe there is a way to do this with printf???
thanks,
max
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
[dtrace-discuss] Getting error ld: fatal: symbol `__SUNW_dof' is multiply-defined:
I have 2 .cpp files in two different directories in which I need to put DTrace
probes. They are compiled into two .a libraries one after another. In the end
they are combined to a single .so file. This sequence I can not change. I get
an error ld: fatal: symbol `__SUNW_dof' is multiply-defined:. What is the
solution for this?
Here is my simulation of real world problem in sample files.
$gmake
CC -mt -xtarget=ultra -g -xs -c -o a.o a.cpp
dtrace -G -32 -s 1.d a.o -o 1.o
CC -xar -o liba.a a.o 1.o
CC -mt -xtarget=ultra -g -xs -c -o b.o b.cpp
dtrace -G -32 -s 2.d b.o -o 2.o
CC -xar -o libb.a b.o 2.o
CC -G -o libc.so a.o b.o 1.o 2.o -mt -norunpath
ld: fatal: symbol `__SUNW_dof' is multiply-defined:
(file 1.o type=OBJT; file 2.o type=OBJT);
ld: fatal: File processing errors. No output written to libc.so
gmake: *** [all] Error 1
$cat Makefile
all::
CC -mt -xtarget=ultra -g -xs -c -o a.o a.cpp
dtrace -G -32 -s 1.d a.o -o 1.o
CC -xar -o liba.a a.o 1.o
CC -mt -xtarget=ultra -g -xs -c -o b.o b.cpp
dtrace -G -32 -s 2.d b.o -o 2.o
CC -xar -o libb.a b.o 2.o
CC -G -o libc.so a.o b.o 1.o 2.o -mt -norunpath
clean::
rm *.o *.a
$cat 1.d
provider sjsws1 {
probe reached1(const char *);
};
$cat 2.d
provider sjsws2 {
probe reached2(const char *);
};
$cat a.cpp
#include stdio.h
#include sjsws.h
main()
{
DTRACE_REACHED1(one);
}
$cat b.cpp
#include stdio.h
#include sjsws.h
void myfunc(void)
{
DTRACE_REACHED2(two);
}
$cat sjsws.h
#ifndef SJSWS
#define SJSWS 1
#include sys/sdt.h
#ifdef __cplusplus
extern C {
#endif
#define DTRACE_REACHED1(arg0) \
__dtrace_sjsws1___reached1(arg0)
#define DTRACE_REACHED2(arg0) \
__dtrace_sjsws2___reached2(arg0)
extern void __dtrace_sjsws1___reached1(const char *);
extern void __dtrace_sjsws2___reached2(const char *);
#ifdef __cplusplus
}
#endif
#endif
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] truss -fall equivalent in DTrace
Thanx a lot. Can I get pstack equivalent script using DTrace? One problem is if we call java from C (using JNI) such stacks are not shown in pstack. One more problem is running instance is hang thread lock can we have a script which shows which all threads are causing deadlock? --- On Fri, 11/7/08, Mark Plotnick [EMAIL PROTECTED] wrote: Check out Brendan Gregg's dtruss shell script (available from his web site or as part of MacOSX Leopard). It emulates truss using dtrace, including truss's -f option. ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] truss -fall equivalent in DTrace
What I mean to ask is will PID provider trace child processes forked by the
parent process?
--- On Wed, 11/5/08, Angelo Rajadurai [EMAIL PROTECTED] wrote:
From: Angelo Rajadurai [EMAIL PROTECTED]
Subject: Re: [dtrace-discuss] truss -fall equivalent in DTrace
To: [EMAIL PROTECTED]
Cc: dtrace-discuss@opensolaris.org
Date: Wednesday, November 5, 2008, 2:52 PM
Change $1 to $target!
(ie)
./sample.d -c a.out
pid$target::somefunctionname:entry
{
printf(%s is called by %d,probefunc, tid);
}
In case you want to find the funcstions of a running
process (say pid 1234)
you have two options.
./sample.d --p 1234
pid$target::somefunctionname:entry
{
printf(%s is called by %d,probefunc, tid);
}
or
./sample.d 1234
pid$1::somefunctionname:entry
{
printf(%s is called by %d,probefunc, tid);
}
HTHs
Angelo
On Nov 5, 2008, at 5:21 AM, [EMAIL PROTECTED] wrote:
I have an executable. When I attach a truss, we have
to give $truss -o truss.out -fall ./a.out. It shows all
system calls made by this program and all the child
processes it forks.
Where as if I am using a DTrace script I am not able
to do it.
./sample.d -c a.out
pid$1::somefunctionname:entry
{
printf(%s is called by %d,probefunc,
tid);
}
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
[dtrace-discuss] truss -fall equivalent in DTrace
I have an executable. When I attach a truss, we have to give $truss -o
truss.out -fall ./a.out. It shows all system calls made by this program and all
the child processes it forks.
Where as if I am using a DTrace script I am not able to do it.
./sample.d -c a.out
pid$1::somefunctionname:entry
{
printf(%s is called by %d,probefunc, tid);
}
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
[dtrace-discuss] Printing a const char *
I have a C program,
$cat sample.cpp
#include stdio.h
#include memory.h
#include stdlib.h
struct xxx
{
int yyy;
int zzz;
const char *name;
};
void sub1 (struct xxx *p)
{
printf (CProgram: %d %d %s\n, p-yyy, p-zzz, p-name);
}
main()
{
char * key = (char *)malloc(5);
key[0] = 'A';
key[1] = 'B';
key[2] = 'C';
key[3] = 'D';
key[4] = '\0';
struct xxx *t1 = new struct xxx;
t1-yyy = 20;
t1-zzz = 30;
t1-name = key;
sub1 (t1);
}
and a DTrace script :
$cat sample.d
struct xxx
{
int yyy;
int zzz;
const char *name;
};
pid$target:a.out:*sub1*:entry
{
sp = (struct xxx *) copyin (arg0, sizeof (struct xxx));
printf (DTrace: %d %d \n, sp-yyy, sp-zzz);
printf (DTrace: name=%s\n, stringof(sp-name));
exit (0);
}
$CC sample.cpp
$dtrace: script 'sample.d' matched 1 probe
CProgram: 20 30 ABCD
dtrace: pid 2624 has exited
dtrace: error on enabled probe ID 1 (ID 47665:
pid2624:a.out:__1cEsub16FpnDxxx__v_:entry): invalid address (0x28728)
in action #4
What is this error ? I get this error when I compile the program in 64 bit as
well.
$CC -xarch=v9 sample.cpp
$dtrace -s sample.d -c ./a.out
dtrace: script 'sample.d' matched 1 probe
CProgram: 20 30 ABCD
dtrace: pid 2629 has exited
dtrace: error on enabled probe ID 1 (ID 47665:
pid2629:a.out:__1cEsub16FpnDxxx__v_:entry): invalid address (0x10010a000) in
action #4
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] Printing a const char *
$dtrace -s sample.d -c ./a.out
dtrace: failed to compile script sample.d: line 12: copyinstr( ) argument #1 is
incompatible with prototype:
prototype: uintptr_t
argument: char *
$cat sample.d
struct xxx
{
int yyy;
int zzz;
const char *name;
};
pid$target:a.out:*sub1*:entry
{
sp = (struct xxx *) copyin (arg0, sizeof (struct xxx));
printf (DTrace: %d %d \n, sp-yyy, sp-zzz);
printf (DTrace: name=%s\n, copyinstr(sp-name));
exit (0);
}
You've correctly copied the structure into DTrace's
address space, but you
didn't copy in the const char * (string). Rather than
doing stringof() on
sp-name, use the copyinstr() subroutine.
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] Printing a const char *
Casting it explicitly as uintptr_t works for 64 bit program and not for 32
bit program.
$CC -xarch=v9 sample.cpp
$dtrace -s sample.d -c ./a.out
dtrace: script 'sample.d' matched 1 probe
CProgram: 20 30 ABCD
dtrace: pid 2974 has exited
CPU IDFUNCTION:NAME
0 47856 __1cEsub16FpnDxxx__v_:entry DTrace: 20 30
DTrace: name=ABCD
$CC sample.cpp
$dtrace -s sample.d -c ./a.out
dtrace: script 'sample.d' matched 1 probe
CProgram: 20 30 ABCD
dtrace: error on enabled probe ID 1 (ID 47856:
pid2979:a.out:__1cEsub16FpnDxxx__v_:entry): invalid address (0x28728)
in action #4 at DIF offset 28
dtrace: pid 2979 has exited
$cat sample.d
struct xxx
{
int yyy;
int zzz;
const char *name;
};
pid$target:a.out:*sub1*:entry
{
sp = (struct xxx *) copyin (arg0, sizeof (struct xxx));
printf (DTrace: %d %d \n, sp-yyy, sp-zzz);
printf (DTrace: name=%s\n, copyinstr((uintptr_t)sp-name));
exit (0);
}
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] Why could I get function names from a stripped exec?
Hi, Yasufumi CHIDA wrote: Thanks. To tell the truth, I'm an instructor, so I have to reply to my trainees. Of course I'm curious about the internal working of DTrace. BTW, you said dynamic info, what is this on earth? The names are needed for dynamic linking. They are in the .dynsym section of the file. You might want to look at elfdump output, then http://blogs.sun.com/ali/entry/inside_elf_symbol_tables, where Ali Bahrami has a decent description. (Or google for .dynsym, which is how I found Ali's blog). max Yasufumi. -- This message posted from opensolaris.org ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] Solaris Dynamic Tracing Guide gets Wikified!
Hi All, Now that the dtrace guide has been wikified, what happens with the guide on docs.sun.com? Should I look at the wikified guide, or the (dated) guide on docs.sun.com? I have the same question about the Modular Debugger guide. Also, any chance of changing the dashboard page to include MDB with the Solaris Modular Debugger line? I went to wikis.sun.com and searched the page for mdb and didn't find it. Then I searched for debug and there it was. thanks, max Adam Leventhal wrote: Agreed. We were thinking of doing exactly this. The provider chapters would be a great place to start. There should be a table in each describing the support on various platforms as well as any usage notes which might apply. Adam On Thu, Dec 13, 2007 at 10:37:01AM -0500, Jim Mauro wrote: Great question. As a dtrace user and documentation reader, I would not want to need to flip to another chapter, or another section, to read about platform differences for a particular provider, function, etc. I'm not saying you suggested that, I'm just thinking out loud... I think a reasonable way to do this, and maintain consistency, is to add a Platform Notes heading to each section - or perhaps at a Chapter granularity. Within Platform Notes, we have subsections for Mac OS X, FreeBSD, etc, and add the platform specific notes there. It may also be useful to add a chapter that discusses platform differences at a high level. Just some thoughts...if the platform differences are not vast enough to warrant adding format changes to the document, perhaps they can be handled in footnotes. (keeping in mind footnotes are considered poor writing style). Thanks, /jim Quinn wrote: At 10:23 -0800 28/11/07, Adam Leventhal wrote: On Tue, Nov 27, 2007 at 07:46:37PM +, Jon Haslam wrote: To gain access to the DTrace wiki space contact Adam ([EMAIL PROTECTED]). If you are working on a bug from the bug list make sure you update the owner column of the bug to reflect the fact. When you've finished please update the column in the bug list to reflect that fact. Actually, there's no need to contact me or anyone: the wiki is open for all registered users to modify. We'll tighted up security if we have any problems. How do you want to handle platform-specific changes? As I stumble across Mac OS X specific stuff, I'd like to add it to the book. To me that makes a lot more sense that maintaining a separate DTrace on Mac OS X document. For example, the discussion of the stop action says that you can start a process using prun, but that's not available on Mac OS X. On Mac OS X you can use kill -CONT pid. I'd love to just go in there and add a note to that effect, but I figured I'd ask about formatting conventions first. S+E ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] dtrace wired issue
Aubrey Li wrote: On Jan 15, 2008 2:45 PM, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: Hi Aubrey, ---snip-- setrlimit(RLIMIT_NOFILE, 0xFD7FFFDFEE80)= 0 openat(AT_FDCWD, /dev/dtrace/provider, O_RDONLY|O_NDELAY|O_LARGEFILE) = 3 fcntl(3, F_SETFD, 0x0001) = 0 fstat(3, 0xFD7FFFDFE8E0)= 0 getdents(3, 0xFD7FFF174000, 8192) (sleeping...) ---here, dtrace sleep almost 4 minutes ...and continue... How long does an ls -l /dev/dtrace/provider take? Yes, it costs almost 4 minutes. So, what does ls -l /dev/dtrace/provider show? max I doubt you are swapping with 2G of memory. You can run vmstat -S 2 to see if there is any swapping. I'll look a bit more... # vmstat -S 2 kthr memorypagedisk faults cpu r b w swap free si so pi po fr de sr cd -- -- -- in sy cs us sy id 0 0 0 1601116 1352560 0 0 717 2 53 0 2502 71 0 0 0 718 7222 2813 4 3 93 0 0 0 1516012 1236776 0 0 0 0 0 0 0 0 0 0 0 388 269 347 0 0 99 0 0 0 1516012 1236808 0 0 0 0 0 0 0 0 0 0 0 394 262 354 0 0 100 0 0 0 1516012 1236808 0 0 0 0 0 0 0 0 0 0 0 392 255 356 0 0 100 0 0 0 1516016 1236812 0 0 0 0 0 0 0 23 0 0 0 425 380 396 1 0 99 0 0 0 1516016 1236812 0 0 0 0 0 0 0 0 0 0 0 389 294 332 0 0 100 0 0 0 1516016 1236812 0 0 0 0 0 0 0 0 0 0 0 382 466 444 0 0 99 0 0 0 1516016 1236812 0 0 0 0 0 0 0 0 0 0 0 394 273 346 0 0 100 0 0 0 1516016 1236812 0 0 0 0 0 0 0 0 0 0 0 387 250 328 0 0 99 0 0 0 1516016 1236812 0 0 0 0 0 0 0 0 0 0 0 384 264 339 0 0 100 By the way, are you running on Indiana (just curious...). No, I'm using nevada. [EMAIL PROTECTED] /etc/release Solaris Express Community Edition snv_77 X86 Copyright 2007 Sun Microsystems, Inc. All Rights Reserved. Use is subject to license terms. Assembled 06 November 2007 Thanks, -Aubrey ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] dtrace wired issue
Hi James, James C. McPherson wrote: Aubrey Li wrote: On Jan 14, 2008 8:52 PM, Sean McGrath - Sun Microsystems Ireland [EMAIL PROTECTED] wrote: Aubrey Li stated: Every first time to run dtrace command after the system boot up, It takes a very long time to get response. But the second time is OK, as follows: # time dtrace -l /dev/null real4m8.011s user0m0.116s sys 0m2.420s This first time is probably when the kernel is loading the dtrace modules. Though still seems slow, 4 minutes. What kind of system (cpu speed etc) is the machine ? # psrinfo -vp The physical processor has 2 virtual processors (0 1) x86 (GenuineIntel 10674 family 6 model 23 step 4 clock 2400 MHz) Intel(r) CPU @ 2.40GHz So, I failed to understand the modules loading needs 4 minutes. If you run dtrace -l with no args, *every* single loadable module on the system will be loaded, interrogated by dtrace and then unloaded if possible. Are you sure of this? Why should it load everything? (And then unload them if they're not being used). # dtrace -l | wc -l 88939 # modload dedump # dtrace -l | wc -l 88963 I get different counts. I think dtrace only looks at what is currently on the system. But 4 minutes??? max All those attach()es and detach()es need time, as does the probe collation. James C. McPherson -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] dtrace wired issue
Hi James, James C. McPherson wrote: Aubrey Li wrote: On Jan 14, 2008 8:52 PM, Sean McGrath - Sun Microsystems Ireland [EMAIL PROTECTED] wrote: Aubrey Li stated: Every first time to run dtrace command after the system boot up, It takes a very long time to get response. But the second time is OK, as follows: # time dtrace -l /dev/null real4m8.011s user0m0.116s sys 0m2.420s This first time is probably when the kernel is loading the dtrace modules. Though still seems slow, 4 minutes. What kind of system (cpu speed etc) is the machine ? # psrinfo -vp The physical processor has 2 virtual processors (0 1) x86 (GenuineIntel 10674 family 6 model 23 step 4 clock 2400 MHz) Intel(r) CPU @ 2.40GHz So, I failed to understand the modules loading needs 4 minutes. If you run dtrace -l with no args, *every* single loadable module on the system will be loaded, interrogated by dtrace and then unloaded if possible. You mean every kernel module currently loaded will be interrogated. Not every kernel module on the system. So _init/_info/_fini/attach/detach should not enter into this. max All those attach()es and detach()es need time, as does the probe collation. James C. McPherson -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] Dtrace probes for voluntary and involuntary context switches
Hi Neelam, Neelam wrote: Hi, I am profiling some workloads for the voluntary and involuntary context switches. I am interested in finding out the reasons causing these two types of context switches. As far as I understand, involuntary context switch happens on expiration of time slice or when a higher priority process comes in. While the voluntary switch generally happens when a process is waiting for I/O etc. So to find out what system calls are causing voluntary context switches in my workloads, I printed whenever a system calls is invoked and whenever a context switch happens. I am profiling the system calls and context switched inside critical sections (while some lock is being held). But I see something unexpected. I see * Voluntary context switches occur almost every time due to doorfs() system call. They do occur for a few times due to lwp_park() and very few times due to yield(). * Involuntary happens anytime. (lwp_park(), read(), fstat(), putmsg(), gtime() and sometime without any system call!!) Does anyone have any idea, what could be the reason for this unexpected behavior? This behavior is not unexpected. In general, threads should not be sleeping while holding locks. What do you think is unexpected? max Thanks, Neelam -- This message posted from opensolaris.org ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
