[EMAIL PROTECTED] wrote on Tue, 21 Feb 2006 16:56 -0600:
> According to the profiling, we're looking at roughly 50% of total
> cpu-time being spent inside check_cq() in a given run.
> ( src/io/bmi/bmi_ib/ib.c:check_cq() )
> --I'll further test this on some longer tests tomorrow to get more
> accurate readings.. however the initial results show check_cq being
> overwhelmingly called.
> the polling function inside check_cq() is << 1% so I doesnt appear as
> if that is blocking it, which was one of my original thoughts.
>
> Is this normal for the BMI/IB interface to be calling the check_cq()
> function so often?
> Can we aggregate some of these calls?
It's normal. To get maximum performance out of IB, you have to poll
the device to see if anything interesting happened. The function
construct_poll_plan() decides whether we should be busy polling or
fall back to periodic (every 10ms or so) polling. It will busy poll
for up to 1 sec after something has happened on the network.
It is possible to get interrupt generation, but that adds quite a
bit of overhead. A blocking API would rely on this feature, then
we'd have to allocate a separate thread to block, to support
multiple network uses and to look for new incoming TCP connection
requests. And I'd still want the busy poll for some circumstances
else we eat 30 us or so for every network event.
I'm not worried about the overhead in check_cq() because
pvfs2-server only ends up in here via BMI_testcontext(), i.e.
because it is looking for work to complete or for incoming work
requests. So it's not really overhead in the sense of wasted
cycles. Other threads should be off paying attention to the disks.
-- Pete
[..]
> ---128MB test via 'pvfs2-cp -t /tmp/bigfile /mnt/pvfs2' ---
> initial Results:
>
> samples % symbol name
> 2151 51.5210 check_cq
> 318 7.6168 construct_poll_plan
> 303 7.2575 BMI_testcontext
> 257 6.1557 BMI_ib_testcontext
> 257 6.1557 bmi_thread_function
[..]
_______________________________________________
Pvfs2-developers mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
