Hi,Thanks for the comment for my nfsv3client probe implementation! I have made changes accordingly. Webrev: http://cr.opensolaris.org/~danhua/webrev/
To reduce the overhead, I use a local variable to save XID, rather than alloc memory space with kmem_zalloc().
According to the overhead caused by tsd_get() and tsd_set(), I did an experiment to measure it.
In this experiment, I run a dtrace script to enable nfsv3client probes and measure time consumed by each nfsv3 operation and tsd_get() and tsd_get(). Then I run some workloads in *filebench* to perform some file operations in a nfs mounted folder. I use the ratio of time spent on sd_get()+tsd_get() to the time spent on the whole operation to evaluate the overhead.
The workload I selected are: randomrw, filemicro_rwrite, filemicro_rread, randomread, randomwrite.
Summary of the result: 1. for every operation, most of the distribution of ratio is less than 1%. 2. for every operation, the average of the ratio is less than 1%.3. the time spent on tsd_get() and tsd_set() are very similar. so the ratio mostly depends on the time spend on operation.
4. In worst case, the ratio is very different from operation to operation. For operations like commit, the max ratio is only about 0.05% For operations like read the max ratio goes up to 10% For operations like write, the max ratio is even more than 40% (Detail of the result in attachment)To make improvement, I have following approaches. I would like to hear your feedback to them.
1. add an argument to rfscall() and rfs3call(), pass XID as an parameter pro: little performance overheadcon: change all the code that calls rfs3call() and rfscall(), even they are not related to nfs3 client
2. insert XID into some argument of rfscall() and rfs3call() pro: little performance overheadcon: change all the code that calls rfs3call() and rfscall(), even they are not related to nfs3 client because rfscall() is not just for nfs3
3. move the dtrace probe declaration into rfscall(),pro: little performance overhead, each to catch XID, no need to change other parts
con: lost information about current path and file 4. add a function to check if a nfsv3client probe is enabled.implementation: during nfsv3 client initialization, setup a table. Each probe has an entry which point to the code that will be modified when a nfsv3client probe is enabled.
pro: no need to change on current code. con: platform dependent (It is different on x86 and sparc).These are the approaches I am thinking about, welcome to comment them or suggest new one!
Regards, Danhua
Experiment: measure overhead of nfsv3client probes
method: running under workloads of filebench
result: for each operation, record overhead time and operation time.
overhead time = time elapse of tsd_get() and tsd_set()
operation time = time elapse for an operation (from request sent out to
reply received)
calculate the ratio of overhead-time/operation-time (in 1/10000)
following figure shows the min, max, average, and distribution of ratio
for each operation
workload randomrw
ratio_min
op-getattr-reply 3
op-write-reply 3
op-commit-reply 5
op-read-reply 22
ratio_max
op-commit-reply 5
op-read-reply 686
op-write-reply 4284
op-getattr-reply 4479
ratio_avg
op-commit-reply 5
op-write-reply 42
op-read-reply 63
op-getattr-reply 69
op-commit-reply
value ------------- Distribution ------------- count
2 | 0
4 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1
8 | 0
op-read-reply
value ------------- Distribution ------------- count
8 | 0
16 |@@@@ 5
32 |@@@@@@@@@@@@@@@@@@@@@@@@@@@ 37
64 |@@@@@@@@@ 12
128 | 0
256 | 0
512 |@ 1
1024 | 0
op-getattr-reply
value ------------- Distribution ------------- count
1 | 0
2 | 3
4 | 18
8 |@@@@ 750
16 |@@@@@@@@@@ 1777
32 |@@@@@@@@@@ 1770
64 |@@@@@@@@@@@@@@@ 2727
128 |@@ 293
256 | 33
512 | 29
1024 | 29
2048 | 6
4096 | 2
8192 | 0
op-write-reply
value ------------- Distribution ------------- count
1 | 0
2 | 1
4 | 84
8 |@@@@ 2307
16 |@@@@@@@@@@@@@@@@@ 10731
32 |@@@@@@@@@@@@@@@@ 10202
64 |@@@ 1788
128 | 72
256 | 94
512 | 71
1024 | 45
2048 | 16
4096 | 1
8192 | 0
workload: filemicro_rwrite
ratio_min
op-commit-reply 0
op-read-reply 2
op-write-reply 3
op-getattr-reply 4
ratio_max
op-commit-reply 0
op-getattr-reply 189
op-read-reply 959
op-write-reply 4483
ratio_avg
op-commit-reply 0
op-write-reply 37
op-read-reply 53
op-getattr-reply 65
op-commit-reply
value ------------- Distribution ------------- count
-1 | 0
0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1
1 | 0
op-getattr-reply
value ------------- Distribution ------------- count
2 | 0
4 | 1
8 | 1
16 |@@@ 7
32 |@@@@@@@@@@@@@@@@@@ 50
64 |@@@@@@@@@@@@@@@@@@ 49
128 |@ 3
256 | 0
op-write-reply
value ------------- Distribution ------------- count
1 | 0
2 | 2
4 | 9
8 |@ 332
16 |@@@@@@@@@@@@@@@@@@@@@@@@ 14470
32 |@@@@@@@@@@@@@@@ 8938
64 |@ 508
128 | 10
256 | 44
512 | 73
1024 | 37
2048 | 13
4096 | 1
8192 | 0
op-read-reply
value ------------- Distribution ------------- count
1 | 0
2 | 2
4 | 3
8 | 278
16 | 11
32 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 19845
64 |@@@@@@@@@ 5639
128 | 2
256 | 0
512 | 2
1024 | 0
workload: filemicro_rread
ratio_min
op-read-reply 22
op-getattr-reply 66
ratio_max
op-getattr-reply 190
op-read-reply 789
ratio_avg
op-read-reply 56
op-getattr-reply 107
op-getattr-reply
value ------------- Distribution ------------- count
32 | 0
64 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 5
128 |@@@@@@@@@@@ 2
256 | 0
op-read-reply
value ------------- Distribution ------------- count
8 | 0
16 | 2
32 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 23654
64 |@@@@ 2746
128 | 26
256 | 0
512 | 5
1024 | 0
workload: randomread
ratio_min
op-read-reply 19
op-getattr-reply 69
ratio_max
op-read-reply 139
op-getattr-reply 247
ratio_avg
op-read-reply 49
op-getattr-reply 133
op-getattr-reply
value ------------- Distribution ------------- count
32 | 0
64 |@@@@@@@@@@@@@@@@@@@@@@@ 4
128 |@@@@@@@@@@@@@@@@@ 3
256 | 0
op-read-reply
value ------------- Distribution ------------- count
8 | 0
16 |@@@@@@@ 22
32 |@@@@@@@@@@@@@@@@@@@@@@@@@@@ 83
64 |@@@@@ 14
128 |@ 2
256 | 0
workload: randomwrite
ratio_min
op-write-reply 1
op-commit-reply 3
op-getattr-reply 53
ratio_max
op-commit-reply 3
op-getattr-reply 253
op-write-reply 4142
ratio_avg
op-commit-reply 3
op-write-reply 34
op-getattr-reply 138
op-commit-reply
value ------------- Distribution ------------- count
1 | 0
2 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 1
4 | 0
op-getattr-reply
value ------------- Distribution ------------- count
16 | 0
32 |@@@@@ 1
64 |@@@@@@@@@@@@@@@ 3
128 |@@@@@@@@@@@@@@@@@@@@ 4
256 | 0
op-write-reply
value ------------- Distribution ------------- count
0 | 0
1 | 1
2 | 1
4 | 191
8 |@@@@@@@@@@@@@ 12918
16 |@@@@@@@@@@@@@@@@@@@ 18630
32 |@@@@@@@ 6979
64 |@ 776
128 | 95
256 | 125
512 | 116
1024 | 156
2048 | 32
4096 | 1
8192 | 0
_______________________________________________ dtrace-discuss mailing list [email protected]
