On Jan 11, 2007, at 2:39 PM, Scott Atchley wrote:
On Jan 11, 2007, at 2:51 PM, Julian Martin Kunkel wrote:
Hi,
These results are quite interesting and the possible throughput
you get with
IB is amazing.
Do you mean Pete's IB 4x numbers or my MX-10G numbers? Or both?
However, I think you get into trouble to find the bottleneck of
the operation and the reason for the observed gaps due to the
complexity of the software
stack.
Maybe you could benefit from using TAS instead of DBPF so that you
could avoid
network activity at all. (Also you might use only one buffer to
already fill
the network).
I am not sure that I follow you here. Ideally, I only want to
measure network activity and PVFS2 overhead. I would prefer to
avoid measuring disk activity but these old nodes do not have
enough memory to use ramfs well.
I think Julian meant 'avoid _disk_ activity'. TAS is a replacement
he wrote for our disk layer, and would be similar to using tmpfs. I
should try to merge the TAS stuff into trunk (sorry Julian!). Its
been something I've been meaning to do for a while now.
-sam
My MX-10G results are from newer nodes that have enough memory to
use ramfs effectively. I can't keep them tied up for bmi_mx
development as long as I can these older nodes. :-)
On the other hand you could use the pvfs2-hint-branch which
provides you with
better MPE logging on the server side, we have some tools to
convert and
merge client and pvfs2-server logs and show the results. I could
upgrade the
current pvfs2-hint-branch with your patches (which is currently
somewhere at
release 2.5). The reason we need parts of the advanced logging is
that logs
have a problem on the server side if multiple start events occur
before the
end events happen, for example if you use multiple flow streams.
@Scott
I have seen you solved it by using events and wonder which tool
you have used
to create states out of the events.
You said you have problems with the MPE log on the server, maybe
we could help
you if you give details ?
Best regards,
Julian
I am using MPE. Since I am not using MPI, I compile mpich2 with
CLFAGS="-DCLOG_NOMPI". I then add "-lmpe_nompi" to my LIBS and the
path to mpich2 to my LDFLAGS.
In my initialization function, I have:
#if BMX_LOGGING
MPE_Init_log();
send_start = MPE_Log_get_event_number();
send_finish = MPE_Log_get_event_number();
recv_start = MPE_Log_get_event_number();
recv_finish = MPE_Log_get_event_number();
sendunex_start = MPE_Log_get_event_number();
sendunex_finish = MPE_Log_get_event_number();
recvunex_start = MPE_Log_get_event_number();
recvunex_finish = MPE_Log_get_event_number();
MPE_Describe_state(send_start, send_finish, "Send", "red");
MPE_Describe_state(recv_start, recv_finish, "Recv", "blue");
MPE_Describe_state(sendunex_start, sendunex_finish,
"SendUnex", "orange");
MPE_Describe_state(recvunex_start, recvunex_finish,
"RecvUnex", "green");
#endif
I then insert MPE_Log_event() in my functions that send, recv and
complete messages.
I can now get server logs. My SERVER_LDFLAGS were wrong. Also, on
the server, I had to specify an absolute path (I did not on the
client).
I would be interested in merging the logs if you can provide some
tools or insight.
Thanks,
Scott
_______________________________________________
Pvfs2-developers mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
_______________________________________________
Pvfs2-developers mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
