[EMAIL PROTECTED] wrote on Wed, 05 Sep 2007 15:10 -0500:
> > Looks good. By the way, once this starts working, you may want to
> > grab CVS head rather than 2.6.3 as it has the important configure
> > option "--disable-tcp" that is required to get good IB performance.
> >
>
> Ok I will give CVS a try. I am assuming I only need the "--disable-tcp"
> option on the client side rigtht? I need to have both tcp and IB on the
> server.
Ach, no. Only the server side. You do need both and will have to
think about a bigger fix. It's not impossible, just nobody has
gotten around to it yet. The fd polling in TCP and IB must be
merged into a single location. If you feel like going off to fix
this.
> Here is the output from ifconfig on the client:
>
> ib0 Link encap:UNSPEC HWaddr
> 80-00-04-04-FE-80-00-00-00-00-00-00-00-00-00-00
> inet6 addr: fe80::208:f104:398:2991/64 Scope:Link
> inet addr:10.1.1.2 Bcast:10.1.255.255 Mask:255.255.0.0
> UP BROADCAST RUNNING MULTICAST MTU:2044 Metric:1
>
> ib1 Link encap:UNSPEC HWaddr
> 80-00-04-04-FE-80-00-00-00-00-00-00-00-00-00-00
> inet addr:10.2.1.2 Bcast:10.2.255.255 Mask:255.255.0.0
> inet6 addr: fe80::208:f104:398:2aed/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:2044 Metric:1
>
> ib2 Link encap:UNSPEC HWaddr
> 80-00-04-04-FE-80-00-00-00-00-00-00-00-00-00-00
> inet addr:10.3.1.2 Bcast:10.3.255.255 Mask:255.255.0.0
> inet6 addr: fe80::208:f104:398:dc5/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:2044 Metric:1
>
>
> The pvfs server is at IPoIB 10.1.1.106 so the client should be
> connecting via ib0 or mthca0 or the first device as expected. Here is
> the output for the ibv_devices and ibv_devinfo:
>
>
>
> hpca4000:/test # ibv_devices
> device node GUID
> ------ ----------------
> mthca0 0008f10403982990
> mthca1 0008f10403982aec
> mthca2 0008f10403980dc4
>
> hpca4000:/test # ibv_devinfo
> hca_id: mthca0
> fw_ver: 1.0.800
> node_guid: 0008:f104:0398:2990
> sys_image_guid: 0008:f104:0398:2993
> vendor_id: 0x08f1
> vendor_part_id: 25204
> hw_ver: 0xA0
> board_id: VLT0050010001
> phys_port_cnt: 1
> port: 1
> state: PORT_ACTIVE (4)
> max_mtu: 2048 (4)
> active_mtu: 2048 (4)
> sm_lid: 1
> port_lid: 2
> port_lmc: 0x00
>
> hca_id: mthca1
> fw_ver: 1.0.800
> node_guid: 0008:f104:0398:2aec
> sys_image_guid: 0008:f104:0398:2aef
> vendor_id: 0x08f1
> vendor_part_id: 25204
> hw_ver: 0xA0
> board_id: VLT0050010001
> phys_port_cnt: 1
> port: 1
> state: PORT_ACTIVE (4)
> max_mtu: 2048 (4)
> active_mtu: 2048 (4)
> sm_lid: 1
> port_lid: 13
> port_lmc: 0x00
>
> hca_id: mthca2
> fw_ver: 1.0.800
> node_guid: 0008:f104:0398:0dc4
> sys_image_guid: 0008:f104:0398:0dc7
> vendor_id: 0x08f1
> vendor_part_id: 25204
> hw_ver: 0xA0
> board_id: VLT0050010001
> phys_port_cnt: 1
> port: 1
> state: PORT_ACTIVE (4)
> max_mtu: 2048 (4)
> active_mtu: 2048 (4)
> sm_lid: 1
> port_lid: 14
> port_lmc: 0x00
[..]
> This only happens when I try to use pvfs2-cp. If i run an application,
> cp, or even a dd on the pvfs2 file system things seem to behave as
> expected.
That is certainly unexpected. There is truly no difference as far
as the OpenIB library in BMI can tell.
The firmware is a bit old, but I would not be willing to say that
was the problem.
I'm at a loss here. At least normal cp works?
-- Pete
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
