A follow up on this. I did some test to see how much issue 2 could affect
the performance. I changed dld_tx_enqueue() code, removed the message
counting and the code to walk the message chain to get the "tail" message
pointer (which can be passed in as an input argument). After that, the TCP
throughput improves about %10.
- Cathy
> Hi,
>
> I had another look at the er_kernel data (which seems have some problem when
> showing the disassembly code, which I will ask Roch), and below is what I
> found, and I hope I can get some of your suggestions:
>
> 1. The first problem I noticed is what I've mentioned in another email, that
> a lot of CPU cycles are on the dld_tx_enqueue()->q_enable(), and which again
> causes dld_wsrv()->dld_tx_enqueue() code paths. That also seems to cause the
> hot lock dsp->ds_tx_list_lock, which is hold by both dld_tx_enqueue() and
> dld_wsrv() function.
>
> But as Eric points out, we cannot remove the q_enable() call from
> dld_tx_enqueu() function. I don't see anything we can do about the above
> issue.
>
> 2. The data also shows that the get_mpsize() functions takes the second most
> exclusive KCPU cycles. get_mpsize() function is added by UV to count the
> message size of the message chains, so that dld_tx_enqueue() function can
> determine whether or not to free the enqueued message:
>
> static void
> dld_tx_enqueue(dld_str_t *dsp, mblk_t *mp, boolean_t head_insert)
> {
> ...
> /* Calculate total size and count of the packet(s) */
> for (tail = mp, cnt = get_mpsize(mp), msgcnt = 1;
> tail->b_next != NULL; tail = tail->b_next) {
> cnt += get_mpsize(tail->b_next);
> msgcnt++;
> }
>
> mutex_enter(&dsp->ds_tx_list_lock);
> /*
> * If the queue depth would exceed the allowed threshold, drop
> * new packet(s) and drain those already in the queue.
> */
> tot_cnt = dsp->ds_tx_cnt + cnt;
> tot_msgcnt = dsp->ds_tx_msgcnt + msgcnt;
>
> if (!head_insert &&
> (tot_cnt >= dld_max_q_count || tot_msgcnt >= dld_max_q_count)) {
> ASSERT(dsp->ds_tx_qbusy);
> mutex_exit(&dsp->ds_tx_list_lock);
> freemsgchain(mp);
> goto done;
> }
>
> /* Update the queue size parameters */
> dsp->ds_tx_cnt = tot_cnt;
> dsp->ds_tx_msgcnt = tot_msgcnt;
> ...
> }
>
>
> My question is, whether the above message-counting-check step is needed.
> Usually, the stream layer above dld should call canputnext() before it calls
> the dld tx function, and if the dld layer is in the state that the messages
> need to be enqueued, canputnext() should return FALSE. In this particular
> test, it is the TCP TX test, and because TCP does not call canputnext() to
> check whether dld can handle more tx packets, therefore, it causes lots of
> CPU cycles spend on the dld_tx_enqueue() and get_mpsize() functions. But
> since TCP has its own flow control, I don't think there will be lots of
> messages to flood into dld queues and use up the memory.
>
> Thanks
> - Cathy
>
>
> _________________________________
> clearview-discuss mailing list
> clearview-discuss at opensolaris.org
