[dtrace-discuss] dtrace scripts not working
Hello, I have solaris 10 machine and I was trying out dtrace. I downloaded some scripts (iotop, errorinfo, prustat) and they all show this error: dtrace: invalid probe specifier along with other errors such as: probe description dtrace:::BEGIN does not match any probes (iotop) probe description syscall:::return does not match any probes (errorinfo) I'm running dtrace from inside a zone and SUNWdtrc is properly installed. Any ideas as to what is going on? Thank you! Diego Lima ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] How to dig deeper
Start with iostat. It's simple, and provides an average of service times
for disk IOs (iostat -xnz 1, the asvc_t column is average service times
in milliseconds). As Jim Litchfield pointed out in a previous thread,
keep in mind it is an average, so you won't see nasty peaks, but if the
average is bad, it's time to talk to the SAN/Storage folks.
Use DTrace with the IO provider to get more precise measurements.
You can borrow from existing scripts in /usr/demo/dtrace to add
time stamps, and use aggregations to capture IO times.
If you need assistance with such a script, please let us know and
we'll be glad to post one (it looks like you're already pretty comfortable
with DTrace...).
Thanks,
/jim
Hans-Peter wrote:
In order to get more information on IO performance problems I created the
script below:
#!/usr/sbin/dtrace -s
#pragma D option flowindent
syscall::*write*:entry
/pid == $1 guard++ == 0/
{
self - ts = timestamp;
self-traceme = 1;
printf(fd: %d, arg0);
}
fbt:::
/self-traceme/
{
/* elapsd =timestamp - self - ts;
printf( elapsed : %d , elapsd); */
printf( timestamp : %d , timestamp);
}
syscall::*write*:return
/self-traceme/
{
self-traceme = 0;
elapsed=timestamp - self - ts;
printf( timestamp : %d , timestamp);
printf(\telapsed : %d , elapsed);
exit(0);
}
I gives me the timestamp for every fbt call during a write system call.
A snippet is here below
8 - schedctl_save timestamp : 1627201334052600
8- savectx timestamp : 1627201334053000
0 - restorectx timestamp : 1627202741110300
difference = 1.407.057.300
0- schedctl_restore timestamp : 1627202741115100
0- schedctl_restore timestamp : 1627202741116100
Visible is that the thread of for 1.4 s off cpu.
Storage is on SAN with fibers between the system and the storage.
Is it possible to dig deeper with dtrace to see how the HBA's are performing.
Other suggestions are welcome too.
Regards Hans-Peter
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] DIF content is invalid?
How does one make the compiler generate runnable code with such D input?
Adam Leventhal wrote:
Hey Jim,
I was able to reduce this to a simpler case:
# dtrace -n BEGIN'{ foo[1, 1, 1, 1, 1, 1] = 1; }'
dtrace: failed to enable 'BEGIN': DIF program content is invalid
This failed because the D compiler generated invalid DIF in that it used
too many virutal registers.
If you try one more key, the error is properly identified.
# dtrace -n BEGIN'{ foo[1, 1, 1, 1, 1, 1, 1] = 1; }'
dtrace: invalid probe specifier BEGIN{ foo[1, 1, 1, 1, 1, 1, 1] = 1; }:
Insufficient registers to generate code
It looks like libdtrace is happily allocating a 9th register even
though there are only 8:
# dtrace -c '/usr/sbin/i86/dtrace -s /var/tmp/foo.d' -n
'pid$target::dt_regset_alloc:return{ trace(arg1); }'
dtrace: description 'pid$target::dt_regset_alloc:return' matched 1 probe
dtrace: failed to compile script /var/tmp/foo.d: Insufficient registers to
generate code
dtrace: pid 101053 has exited
CPU IDFUNCTION:NAME
0 73111 dt_regset_alloc:return 0
0 73111 dt_regset_alloc:return 1
0 73111 dt_regset_alloc:return 2
0 73111 dt_regset_alloc:return 3
0 73111 dt_regset_alloc:return 4
0 73111 dt_regset_alloc:return 5
0 73111 dt_regset_alloc:return 6
0 73111 dt_regset_alloc:return 7
0 73111 dt_regset_alloc:return 8
0 73111 dt_regset_alloc:return4294967295
It looks like there's a simple fix in dt_regset_create() which is not to add
one to the number of registers we've been told exist. I've filed this bug
for you:
6778459 libdtrace will allocate one more register than there exists
Adam
On Sun, Nov 30, 2008 at 08:36:50PM -0800, James Litchfield wrote:
What's going on?
# dtrace -s iotime_all.d 100
dtrace: failed to enable 'iotime_all.d': DIF program content is invalid
The errant script
#pragma D option quiet
BEGIN
{
stime = timestamp;
io_count = 0;
}
io:::start
/args[2]-fi_pathname != none/
{
start[pid, args[2]-fi_pathname, args[0]-b_edev, args[0]-b_blkno,
args[0]-b_bcount] = timestamp;
self-pid = pid;
self-name = args[2]-fi_pathname;
self-size = args[0]-b_bcount;
}
io:::start
/args[2]-fi_pathname != none/
{
start[pid, args[1]-dev_pathname, args[0]-b_edev,
args[0]-b_blkno, args[0]-b_bcount] = timestamp;
self-pid = pid;
self-name = args[1]-dev_pathname;
self-size = args[0]-b_bcount;
}
io:::done
/start[self-pid, self-name, args[0]-b_edev, args[0]-b_blkno,
self-size]/
{
this-elapsed = timestamp - start[self-pid, self-name,
args[0]-b_edev, args[0]-b_blkno, self-size];
printf(%5u %10s %58s %2s %8u %8u %3d.%03d\n, self-pid,
args[1]-dev_statname,
self-name, args[0]-b_flags B_READ ? R : W,
args[0]-b_bcount, self-size,
this-elapsed / 100, (this-elapsed / 1000) % 1000);
start[self-pid, self-name, args[0]-b_edev, args[0]-b_blkno,
self-size] = 0;
self-pid = 0;
self-name = 0;
self-size = 0;
io_count++;
}
io:::done
/io_count $1/
{
printf(Elapsed Time: %u seconds\n\n, (timestamp - stime) /
10);
exit (0);
}
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] DIF content is invalid?
You can't. Right now this is a limitation of the compiler. If it's a
common use case we can likely find a solution.
Adam
On Dec 2, 2008, at 10:25 PM, Dan Mick wrote:
How does one make the compiler generate runnable code with such D
input?
Adam Leventhal wrote:
Hey Jim,
I was able to reduce this to a simpler case:
# dtrace -n BEGIN'{ foo[1, 1, 1, 1, 1, 1] = 1; }'
dtrace: failed to enable 'BEGIN': DIF program content is invalid
This failed because the D compiler generated invalid DIF in that it
used
too many virutal registers.
If you try one more key, the error is properly identified.
# dtrace -n BEGIN'{ foo[1, 1, 1, 1, 1, 1, 1] = 1; }'
dtrace: invalid probe specifier BEGIN{ foo[1, 1, 1, 1, 1, 1, 1] =
1; }:
Insufficient registers to generate code
It looks like libdtrace is happily allocating a 9th register even
though there are only 8:
# dtrace -c '/usr/sbin/i86/dtrace -s /var/tmp/foo.d' -n 'pid
$target::dt_regset_alloc:return{ trace(arg1); }'
dtrace: description 'pid$target::dt_regset_alloc:return' matched 1
probe
dtrace: failed to compile script /var/tmp/foo.d: Insufficient
registers to
generate code
dtrace: pid 101053 has exited
CPU IDFUNCTION:NAME
0 73111 dt_regset_alloc:return 0
0 73111 dt_regset_alloc:return 1
0 73111 dt_regset_alloc:return 2
0 73111 dt_regset_alloc:return 3
0 73111 dt_regset_alloc:return 4
0 73111 dt_regset_alloc:return 5
0 73111 dt_regset_alloc:return 6
0 73111 dt_regset_alloc:return 7
0 73111 dt_regset_alloc:return 8
0 73111 dt_regset_alloc:return4294967295
It looks like there's a simple fix in dt_regset_create() which is
not to add
one to the number of registers we've been told exist. I've filed
this bug
for you:
6778459 libdtrace will allocate one more register than there exists
Adam
On Sun, Nov 30, 2008 at 08:36:50PM -0800, James Litchfield wrote:
What's going on?
# dtrace -s iotime_all.d 100
dtrace: failed to enable 'iotime_all.d': DIF program content is
invalid
The errant script
#pragma D option quiet
BEGIN
{
stime = timestamp;
io_count = 0;
}
io:::start
/args[2]-fi_pathname != none/
{
start[pid, args[2]-fi_pathname, args[0]-b_edev, args[0]-
b_blkno, args[0]-b_bcount] = timestamp;
self-pid = pid;
self-name = args[2]-fi_pathname;
self-size = args[0]-b_bcount;
}
io:::start
/args[2]-fi_pathname != none/
{
start[pid, args[1]-dev_pathname, args[0]-b_edev, args[0]-
b_blkno, args[0]-b_bcount] = timestamp;
self-pid = pid;
self-name = args[1]-dev_pathname;
self-size = args[0]-b_bcount;
}
io:::done
/start[self-pid, self-name, args[0]-b_edev, args[0]-b_blkno,
self-size]/
{
this-elapsed = timestamp - start[self-pid, self-name,
args[0]-b_edev, args[0]-b_blkno, self-size];
printf(%5u %10s %58s %2s %8u %8u %3d.%03d\n, self-pid,
args[1]-dev_statname,
self-name, args[0]-b_flags B_READ ? R : W,
args[0]-b_bcount, self-size,
this-elapsed / 100, (this-elapsed / 1000) % 1000);
start[self-pid, self-name, args[0]-b_edev, args[0]-b_blkno,
self-size] = 0;
self-pid = 0;
self-name = 0;
self-size = 0;
io_count++;
}
io:::done
/io_count $1/
{
printf(Elapsed Time: %u seconds\n\n, (timestamp - stime) /
10);
exit (0);
}
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
--
Adam Leventhal, Fishworkshttp://blogs.sun.com/ahl
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] Tracing application-specific data structures
Hi Matthew,
Your choice are to unroll a loop and use predicates to know when
you've reached the end condition or to save state in global variables
and use a tick probe to iterate through the list. If you need more
guidance, feel free to ask, but you may be able to find an example of
this in the discussion list archives as similar questions have been
asked.
Adam
On Dec 2, 2008, at 4:01 PM, Matthew Barker wrote:
I have a provider that allows me to look at some application-
specific data structs for SJS Web Server 7. The problem I'm finding
is that some of the data is stored in a short linked list and I
don't see any way, without the unimplemented for or if
constructs in dtrace, to trace those data structures.
I've included an excerpt of the structure definitions from my script
(copied from the provider). I'm trying to look at the contents of
the series of pb_param structs, each referred to by a linked list of
pb_entry structs which are hanging off of the pblock struct pointed
to by Request.vars.
Can anyone provide some insight as to how to go about it?
typedef struct pblock {
int hsize;
struct pb_entry **ht;
} pblock;
typedef struct pb_param {
char *name,*value;
} pb_param;
typedef struct pb_entry {
pb_param *param;
struct pb_entry *next;
} pb_entry;
typedef struct Request {
pblock *vars;
pblock *reqpb;
int loadhdrs;
pblock *headers;
int senthdrs;
pblock *srvhdrs;
...
uint32_t orig_rq; /* was Request * */
} Request;
sjsws$1::*:saf-entry
{
self-i=0;
r=copyin(arg3,sizeof(struct Request));
v=copyin( ((uint32_t)((Request *)r)-vars), sizeof(pblock));
self-last=((pblock*)v)-hsize;
printf(T(%d): Calling saf %s:vars(%d)\n, tid, copyinstr(arg0),
self-last );
}
sjsws$1:::saf-exit
{
printf(T(%d):%s--%d\n\n, tid, copyinstr(arg0), args[1]);
}
The provider declares these functions:
provider sjsws {
probe saf__entry(const char *, pblock *, Session *, Request *);
probe saf__exit(const char *, int, pblock *, Session *, Request *);
};
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
--
Adam Leventhal, Fishworkshttp://blogs.sun.com/ahl
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] DIF content is invalid?
OK. I don't know if it is or not, but Jim ran into it on what seems like a
valid D script. I'm guessing his problem was because of the multi-dimensional
aggregation index? Can the error message indicate the D line that caused the
problem?
Adam Leventhal wrote:
You can't. Right now this is a limitation of the compiler. If it's a
common use case we can likely find a solution.
Adam
On Dec 2, 2008, at 10:25 PM, Dan Mick wrote:
How does one make the compiler generate runnable code with such D
input?
Adam Leventhal wrote:
Hey Jim,
I was able to reduce this to a simpler case:
# dtrace -n BEGIN'{ foo[1, 1, 1, 1, 1, 1] = 1; }'
dtrace: failed to enable 'BEGIN': DIF program content is invalid
This failed because the D compiler generated invalid DIF in that it
used
too many virutal registers.
If you try one more key, the error is properly identified.
# dtrace -n BEGIN'{ foo[1, 1, 1, 1, 1, 1, 1] = 1; }'
dtrace: invalid probe specifier BEGIN{ foo[1, 1, 1, 1, 1, 1, 1] =
1; }:
Insufficient registers to generate code
It looks like libdtrace is happily allocating a 9th register even
though there are only 8:
# dtrace -c '/usr/sbin/i86/dtrace -s /var/tmp/foo.d' -n 'pid
$target::dt_regset_alloc:return{ trace(arg1); }'
dtrace: description 'pid$target::dt_regset_alloc:return' matched 1
probe
dtrace: failed to compile script /var/tmp/foo.d: Insufficient
registers to
generate code
dtrace: pid 101053 has exited
CPU IDFUNCTION:NAME
0 73111 dt_regset_alloc:return 0
0 73111 dt_regset_alloc:return 1
0 73111 dt_regset_alloc:return 2
0 73111 dt_regset_alloc:return 3
0 73111 dt_regset_alloc:return 4
0 73111 dt_regset_alloc:return 5
0 73111 dt_regset_alloc:return 6
0 73111 dt_regset_alloc:return 7
0 73111 dt_regset_alloc:return 8
0 73111 dt_regset_alloc:return4294967295
It looks like there's a simple fix in dt_regset_create() which is
not to add
one to the number of registers we've been told exist. I've filed
this bug
for you:
6778459 libdtrace will allocate one more register than there exists
Adam
On Sun, Nov 30, 2008 at 08:36:50PM -0800, James Litchfield wrote:
What's going on?
# dtrace -s iotime_all.d 100
dtrace: failed to enable 'iotime_all.d': DIF program content is
invalid
The errant script
#pragma D option quiet
BEGIN
{
stime = timestamp;
io_count = 0;
}
io:::start
/args[2]-fi_pathname != none/
{
start[pid, args[2]-fi_pathname, args[0]-b_edev, args[0]-
b_blkno, args[0]-b_bcount] = timestamp;
self-pid = pid;
self-name = args[2]-fi_pathname;
self-size = args[0]-b_bcount;
}
io:::start
/args[2]-fi_pathname != none/
{
start[pid, args[1]-dev_pathname, args[0]-b_edev, args[0]-
b_blkno, args[0]-b_bcount] = timestamp;
self-pid = pid;
self-name = args[1]-dev_pathname;
self-size = args[0]-b_bcount;
}
io:::done
/start[self-pid, self-name, args[0]-b_edev, args[0]-b_blkno,
self-size]/
{
this-elapsed = timestamp - start[self-pid, self-name,
args[0]-b_edev, args[0]-b_blkno, self-size];
printf(%5u %10s %58s %2s %8u %8u %3d.%03d\n, self-pid,
args[1]-dev_statname,
self-name, args[0]-b_flags B_READ ? R : W,
args[0]-b_bcount, self-size,
this-elapsed / 100, (this-elapsed / 1000) % 1000);
start[self-pid, self-name, args[0]-b_edev, args[0]-b_blkno,
self-size] = 0;
self-pid = 0;
self-name = 0;
self-size = 0;
io_count++;
}
io:::done
/io_count $1/
{
printf(Elapsed Time: %u seconds\n\n, (timestamp - stime) /
10);
exit (0);
}
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
--
Adam Leventhal, Fishworkshttp://blogs.sun.com/ahl
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] DIF content is invalid?
The error message without this bug would have identified the DIF
offset which could then be used to manually determine the line of the
source file. Doing it programmatically is a long-standing RFE.
Adam
On Dec 2, 2008, at 10:43 PM, Dan Mick wrote:
OK. I don't know if it is or not, but Jim ran into it on what seems
like a valid D script. I'm guessing his problem was because of the
multi-dimensional aggregation index? Can the error message indicate
the D line that caused the problem?
Adam Leventhal wrote:
You can't. Right now this is a limitation of the compiler. If it's
a common use case we can likely find a solution.
Adam
On Dec 2, 2008, at 10:25 PM, Dan Mick wrote:
How does one make the compiler generate runnable code with such D
input?
Adam Leventhal wrote:
Hey Jim,
I was able to reduce this to a simpler case:
# dtrace -n BEGIN'{ foo[1, 1, 1, 1, 1, 1] = 1; }'
dtrace: failed to enable 'BEGIN': DIF program content is invalid
This failed because the D compiler generated invalid DIF in that
it used
too many virutal registers.
If you try one more key, the error is properly identified.
# dtrace -n BEGIN'{ foo[1, 1, 1, 1, 1, 1, 1] = 1; }'
dtrace: invalid probe specifier BEGIN{ foo[1, 1, 1, 1, 1, 1, 1]
= 1; }:
Insufficient registers to generate code
It looks like libdtrace is happily allocating a 9th register even
though there are only 8:
# dtrace -c '/usr/sbin/i86/dtrace -s /var/tmp/foo.d' -n 'pid
$target::dt_regset_alloc:return{ trace(arg1); }'
dtrace: description 'pid$target::dt_regset_alloc:return' matched
1 probe
dtrace: failed to compile script /var/tmp/foo.d: Insufficient
registers to
generate code
dtrace: pid 101053 has exited
CPU IDFUNCTION:NAME
0 73111 dt_regset_alloc:return 0
0 73111 dt_regset_alloc:return 1
0 73111 dt_regset_alloc:return 2
0 73111 dt_regset_alloc:return 3
0 73111 dt_regset_alloc:return 4
0 73111 dt_regset_alloc:return 5
0 73111 dt_regset_alloc:return 6
0 73111 dt_regset_alloc:return 7
0 73111 dt_regset_alloc:return 8
0 73111 dt_regset_alloc:return4294967295
It looks like there's a simple fix in dt_regset_create() which
is not to add
one to the number of registers we've been told exist. I've filed
this bug
for you:
6778459 libdtrace will allocate one more register than there exists
Adam
On Sun, Nov 30, 2008 at 08:36:50PM -0800, James Litchfield wrote:
What's going on?
# dtrace -s iotime_all.d 100
dtrace: failed to enable 'iotime_all.d': DIF program content is
invalid
The errant script
#pragma D option quiet
BEGIN
{
stime = timestamp;
io_count = 0;
}
io:::start
/args[2]-fi_pathname != none/
{
start[pid, args[2]-fi_pathname, args[0]-b_edev, args[0]-
b_blkno, args[0]-b_bcount] = timestamp;
self-pid = pid;
self-name = args[2]-fi_pathname;
self-size = args[0]-b_bcount;
}
io:::start
/args[2]-fi_pathname != none/
{
start[pid, args[1]-dev_pathname, args[0]-b_edev, args[0]-
b_blkno, args[0]-b_bcount] = timestamp;
self-pid = pid;
self-name = args[1]-dev_pathname;
self-size = args[0]-b_bcount;
}
io:::done
/start[self-pid, self-name, args[0]-b_edev, args[0]-
b_blkno, self-size]/
{
this-elapsed = timestamp - start[self-pid, self-name,
args[0]-b_edev, args[0]-b_blkno, self-size];
printf(%5u %10s %58s %2s %8u %8u %3d.%03d\n, self-pid,
args[1]-dev_statname,
self-name, args[0]-b_flags B_READ ? R : W,
args[0]-b_bcount, self-size,
this-elapsed / 100, (this-elapsed / 1000) % 1000);
start[self-pid, self-name, args[0]-b_edev, args[0]-
b_blkno, self-size] = 0;
self-pid = 0;
self-name = 0;
self-size = 0;
io_count++;
}
io:::done
/io_count $1/
{
printf(Elapsed Time: %u seconds\n\n, (timestamp - stime) /
10);
exit (0);
}
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
--
Adam Leventhal, Fishworkshttp://blogs.sun.com/ahl
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
--
Adam Leventhal, Fishworkshttp://blogs.sun.com/ahl
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
