Darren Reed writes:
> Andrew Gallatin wrote:
> > Hi,
> >
> > When testing single-stream 10GbE netperf receive performance on both
> > S10U4 and S11b84, I found an odd performance quirk. As I increased
> > the receiver's socket buffer size, I noticed a decrease in performance
> > as the socket buffer sizes approached 512KB, with a crash in
> > performance above 512KB. I also noticed that mpstat was reporting an
> > increasing number of xcals as I increased the socket buffer size.
> > ...
> > Looking at the source of kstrgetmsg(), there is a heuristic which
> > tries to do a pullup(mp, -1) if the first mblk has a length of less
> > than mblk_pull_len, or 64 bytes. In this case, it is pulling up about
> > 460KB.
> >
> > If I disable this by setting mblk_pull_len to 0, then the decrease in
> > performance with an increase in socket buffer size goes away on S10U4
> > (haven't looked at b84 yet).
> >
> > Can somebody explain the "performance degredation from copyout
> > overhead" talked about by the comment near the declaration of
> > mblk_pull_len in common/os/streamio.c. I thought copyout was the
> > normal copy routine plus a trap handler, and should cost the same as
> > the copy in the pullup.
> >
>
> For reference, the two bugs that tie in with this are:
> 4226443
> 4347047
>
> As 4347047 introduced this behaviour, I've copied a section from the
> Comments to the
> Description (nothing confidential or going to cause anyone pain) so you
> should be able
> to check that out when the bug database syncs up. As for 4226443, I
> need to review
> that one more carefully, but I'll add that the behaviour you're seeing
> was introduced by
> '047, while the general problem it was solving goes back to '443.
>
> Since you're experimenting with this, can you try changing the if() from
> if (MBLKL(bp) < mblk_pull_len) {
> to
> if ((bp->b_cont != NULL) && (MBLKL(bp) < mblk_pull_len) &&
> (MBLKL(bp->b_cont) < mblk_pull_len)) {
>
> It's not going to be as good as the "0" case, but it should be better
> than what it
> is today for your case.
>
> This is one of those times when it would be nice if there was a packet
> structure
> that "owned" the mblks and where you track the number of chained mblks,
> thus changing this "guessing game" heuristic into some simple math, e.g:
> if (pkt->dsize / pkt->mblkcnt < mblk_pull_len)
Indeed, this seems like the best sort of fix, but it is one which
requires probably requires lots of changes in the stack.
In my case, the dblock size of the tiny mblock indicates that it was
originally much larger, as 32 is less than the size at which the
driver will copy receieves rather than pass up its receive pool by
reference. So I suspect the tiny mblk that triggered the behavior was
the result of the benchmark reading X bytes when there were X+32 bytes
left in the soecket. Then TCP/IP kept appending 1460 more bytes to
the end of the chain while the application was thinking.
Maybe the "best" fix for this would be to apply a similar heuristic to
where data is appended to the socketbuffer. If there is a single mblk
in the socket buffer at the time of the append, and the mblk len is <
mblk_pull_len, then do the pullup on the append. This would keep
things from getting out of control. Heck, reduce mblk_pull_len
54 bytes, and there should always be room to pull up the short
head into the leadingspace of the mblk you're appending.
Drew
_______________________________________________
networking-discuss mailing list
[email protected]
