Cathy Zhou wrote:
> A follow up on this. I did some test to see how much issue 2 could affect
> the performance. I changed dld_tx_enqueue() code, removed the message
> counting and the code to walk the message chain to get the "tail" message
> pointer (which can be passed in as an input argument). After that, the TCP
> throughput improves about %10.
>
does that mean you no longer check for ds_tx_cnt but you do check
ds_tx_msgcnt?
the problem with this is you could always end up with dld_max_q_count
(currently 16M) messages queued up. if messages are large, you may tie
up lots of memory. even though TCP does flow control, there could be
multiple connections sending though the same nic, so 16M messages could
get queued fairly quickly. for UDP, this could happen with just one sender.
eric
> - Cathy
>
>> Hi,
>>
>> I had another look at the er_kernel data (which seems have some problem when
>> showing the disassembly code, which I will ask Roch), and below is what I
>> found, and I hope I can get some of your suggestions:
>>
>> 1. The first problem I noticed is what I've mentioned in another email, that
>> a lot of CPU cycles are on the dld_tx_enqueue()->q_enable(), and which again
>> causes dld_wsrv()->dld_tx_enqueue() code paths. That also seems to cause the
>> hot lock dsp->ds_tx_list_lock, which is hold by both dld_tx_enqueue() and
>> dld_wsrv() function.
>>
>> But as Eric points out, we cannot remove the q_enable() call from
>> dld_tx_enqueu() function. I don't see anything we can do about the above
>> issue.
>>
>> 2. The data also shows that the get_mpsize() functions takes the second most
>> exclusive KCPU cycles. get_mpsize() function is added by UV to count the
>> message size of the message chains, so that dld_tx_enqueue() function can
>> determine whether or not to free the enqueued message:
>>
>> static void
>> dld_tx_enqueue(dld_str_t *dsp, mblk_t *mp, boolean_t head_insert)
>> {
>> ...
>> /* Calculate total size and count of the packet(s) */
>> for (tail = mp, cnt = get_mpsize(mp), msgcnt = 1;
>> tail->b_next != NULL; tail = tail->b_next) {
>> cnt += get_mpsize(tail->b_next);
>> msgcnt++;
>> }
>>
>> mutex_enter(&dsp->ds_tx_list_lock);
>> /*
>> * If the queue depth would exceed the allowed threshold, drop
>> * new packet(s) and drain those already in the queue.
>> */
>> tot_cnt = dsp->ds_tx_cnt + cnt;
>> tot_msgcnt = dsp->ds_tx_msgcnt + msgcnt;
>>
>> if (!head_insert &&
>> (tot_cnt >= dld_max_q_count || tot_msgcnt >= dld_max_q_count)) {
>> ASSERT(dsp->ds_tx_qbusy);
>> mutex_exit(&dsp->ds_tx_list_lock);
>> freemsgchain(mp);
>> goto done;
>> }
>>
>> /* Update the queue size parameters */
>> dsp->ds_tx_cnt = tot_cnt;
>> dsp->ds_tx_msgcnt = tot_msgcnt;
>> ...
>> }
>>
>>
>> My question is, whether the above message-counting-check step is needed.
>> Usually, the stream layer above dld should call canputnext() before it calls
>> the dld tx function, and if the dld layer is in the state that the messages
>> need to be enqueued, canputnext() should return FALSE. In this particular
>> test, it is the TCP TX test, and because TCP does not call canputnext() to
>> check whether dld can handle more tx packets, therefore, it causes lots of
>> CPU cycles spend on the dld_tx_enqueue() and get_mpsize() functions. But
>> since TCP has its own flow control, I don't think there will be lots of
>> messages to flood into dld queues and use up the memory.
>>
>> Thanks
>> - Cathy
>>
>>
>> _________________________________
>> clearview-discuss mailing list
>> clearview-discuss at opensolaris.org
>
>
> _________________________________
> clearview-discuss mailing list
> clearview-discuss at opensolaris.org
