On 08/07/09 08:19, Marion Hakanson wrote:
[email protected] said:
[email protected] said:
3 properties can be edited in mpt.conf to tune the performance:
. . .
mpt_doneq_length_threshold_prop - default 8
Thanks for the suggestion, I'll give this a try today. I expect that since
the V245 has only two single-core CPU's, there's not a lot of help to be had
with this approach, but it still sounds encouraging.
Problem is resolved, without applying any settings of any kind.
Turns out the V245's SERVICE light was lit, but there were no hardware
faults to be found. Only the occasional entry in /var/adm/messages:
Aug 4 08:22:44 myslowhost rmclomv: [ID 647266 kern.warning] WARNING: SC is not
responding
Aug 4 09:07:44 myslowhost rmclomv: [ID 647266 kern.warning] WARNING: SC is not
responding
Aug 4 11:56:47 myslowhost rmclomv: [ID 714237 kern.notice] NOTICE: SC
recovered
I shutdown the OS, removed power from the chassis, then re-powered it,
which allowed the SC to cold-start. After rebooting the system, the
slow I/O was gone. Fixed by rebooting -- kind of embarrassing.
Note that while the problem was present, I/O on both the SAS HBA (mpt0) and
the internal SAS controller (mpt1) were affected, and it's possible that other
drivers may have been involved as well. I can only guess that while handling
mpt interrupts, the kernel went off in some SC-related direction; perhaps
the rmc_comm driver was nonresponsive and slowing everything else down.
If I'd been better versed in DTrace, I'm sure it would've been obvious.
You can use the attached dtrace script to see what the interrupt CPU is
doing.
While the problem is present, run like this:
#./kp1cpu cpuid > tmp.out <<<<< where the cpuid is the id of the CPU
where mpt interrupt is registered or whichever you are interested in.
#./inclexcl tmp.out > ie.out
then you can check the text file ie.out to see which function consume
most of the cpu cycles.
Oh, by the way, I did try my simple "dd" write tests both with and without
these "mpt_doneq_*" settings in place:
mpt_doneq_thread_n_prop = 8;
mpt_doneq_length_threshold_prop = 8;
mpt_doneq_thread_threshold_prop = 10;
I couldn't measure any difference in performance. It's possible that the
simple "dd" test did not load the system enough for the I/O completion
fanout feature to kick in, and/or that two CPU cores is not enough to
really take advantage of it.
If the load is not heavy enough, that is, the only one interrupt CPU can
handle all of the loads, then the interrupt fanout feature will not be
triggered. You can run tens of 'dd' to make sure the load is heavy enough.
One thing should be pointed out is the firmware version on the hba has
to be higher than LSI1.26.00, otherwise, the new feature will no yield
higher IOPs than the original one.
The new feature(interrupt fanout) + new firmware version can get maxmium
IOPs from 70k to ~120k.
Thanks,
Brian
#!/bin/sh
TOT=`nawk '
$1=="!" {
if (bang++) s+= last;
}
{ last = $1;
}
END { print s;
}
' $*`
cat<<EOF
Inclusive Exclusive
Function nsamples pct nsamples pct
EOF
nawk '
$1=="!" {
if (bang) {
# consolidate the data
for (i=1; i<nf; i++) {
r = fr[i];
gsub(/^.*`/,"",r);
gsub(/\+.*$/,"",r);
fnc[r]++;
if (i==1) excl[r] += fr[nf];
}
for (r in fnc)
incl[r] += fr[nf];
for (r in fnc)
delete fnc[r];
# cleanout the callstack
nf=0;
for (i in fr) delete fr[i];
}
bang++;
next;
}
bang != 0 {
fr[++nf] = $1; # indexed 1..nf
}
END {
for (r in incl)
printf("%-36s %8d %5.2f%% %8d %5.2f%%\n",r,
incl[r], 100*incl[r]/'$TOT', excl[r],
100*excl[r]/'$TOT');
}
' $* | sort -rn +1
#!/usr/sbin/dtrace -qs
BEGIN {
start = timestamp;
}
profile:::profile-479
/cpu == $1 && arg0/
{
@a["!",stack(50)] = count();
}
profile:::profile-479
/cpu == $1 && arg1/
{
usermode++;
}
END {
elapsed = timestamp - start;
printf("%d.%09d seconds elapsed\n", elapsed/1000000000, elapsed%1000000000);
printa(@a);
printf("!\n");
printf("(usermode)\n%d\n", usermode);
printf("!\n");
}
_______________________________________________
storage-discuss mailing list
[email protected]
http://mail.opensolaris.org/mailman/listinfo/storage-discuss
