Bharath Ramesh, if you wish you can send the source to me and i will
review it.
Dotan
Rick Jones wrote:
Gleb Natapov wrote:
On Wed, Oct 17, 2007 at 09:44:04AM +0200, Dotan Barak wrote:
Hi.
Bharath Ramesh wrote:
I wrote a simple test program to actual time it takes for RDMA read
over
IB. I find a huge difference in the numbers returned by timing. I was
wondering if someone could help me in finding what I might be doing
wrong in the way I am measuring the time.
Steps I do for timing is as follows.
1) Create the send WR for RDMA Read.
2) call gettimeofday ()
3) ibv_post_send () the WR
4) Loop around ibv_poll_cq () till I get the completion event.
5) call gettimeofday ();
The difference in time would give me the time it takes to perform RDMA
read over IB. I constantly get around 35 microsecs as the timing which
seems to be really large considering the latency of IB. I am measuring
the time for transferring 4K bytes of data. If anyone wants I can send
the code that I have written. I am not subscribed to the list, if you
could please cc me in the reply.
I don't familiar with the implementation of gettimeofday, but i
believe that this function do a context switch
(and/or spend some time in the function to fill the struct that you
supply to it)
Here:
struct timeval tv_s, tv_e;
gettimeofday(&tv_s, NULL);
gettimeofday(&tv_e, NULL);
printf("%d\n", tv_e.tv_usec - tv_s.tv_usec);
Compile and run it. The overhead of two calls to gettimeofday is at most
1 microsecond.
Unless there is contention with other gettimeofday() calls on the
system - on SMP etc there are locks involved in making sure that each
call to gettimeofday() does not go backwards and the like, and on some
systems, with enough callers to gettimeofday() one can run into lock
contention. So, while 99 times out of ten gettimeofday() may be
"cheap" it really isn't a good idea to ass-u-me it will always be cheap.
And besides, the most efficient call is the one which is never made,
so the suggestion to perform N operations between the calls is
probably still a good one. Even for measuring the overhead of
gettimeofday() :)
Also, while it may not be so much the case these days, certainly in
the past there were "gettimeofday()" implementations which may have
rather coarse granularity.
Now, some CPUs offer interval timer/registers/whatever - for example
the ITC on Itanium or CR16 on PA-RISC, I'm sure there are other
examples - which can be used for measuring very short things. Under
some OSes - HP-UX and Solaris are two with which I am familiar - there
is a "gethrtime()" interface which uses those without the user having
to deal with inline assembly. That should have lower overhead than
gettimeofday() although even then it would probably be best, if one is
indeed going for the average, to use those to measure the time to
perform N operations.
If one does use gethrtime(), it should only be for measuring short
things, and those "timestamps" should not be interspersed with those
from gettimeofday(). The two are really separate "timespaces" if you
will. Gethrtime() does not get tick adjustment like gettimeofday()
does/can.
rick jones
FWIW, netperf uses gettimeofday() to measure the overall runtime of a
netperf test, and gethrtime() (when available) to measure the
individual times for "transactions" such as the exchange of a
request/response, or time spend in send() or recv() or whatnot.
--
Gleb.
_______________________________________________
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit
http://openib.org/mailman/listinfo/openib-general
_______________________________________________
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
