Hello Hal. Thank you for your information.
I'll feedback to Mellanox. 2011/10/27 Hal Rosenstock <[email protected]>: > > > On Thu, Oct 27, 2011 at 7:48 AM, Hiroyuki Sato <[email protected]> wrote: >> >> Hello Hal >> >> Thank you for your information. >> >> I read this documents. PP 45. >> >> http://www.mellanox.com/related-docs/prod_software/Mellanox_OFED_Linux_user_manual_1_5_1.pdf >> >> Decide on the PKey to be used in the subnet. >> Valid values are 0-255. > > > Valid values for pkey are 1-32767 (1-0x7ffff). > >> >> The actual PKey used is a 16-bit number with >> the most significant bit set. >> For example, a value of 0 will give a >> PKey with the value 0x8000. >> >> Is this wrong?? > > > Pkey 0/0x8000 is invalid so this is wrong. What it's showing about turning > on the full membership bit in the pkey is correct. A better example would be > showing a value of 1 being transformed into 0x8001 pkey. > > -- Hal > >> >> 2011/10/27 Hal Rosenstock <[email protected]>: >> > >> > >> > On Thu, Oct 27, 2011 at 7:05 AM, Hiroyuki Sato <[email protected]> >> > wrote: >> >> >> >> Hello Hal and Richard. >> >> >> >> I found what is the problem. >> >> >> >> Q: Can I create subinterface with 0 (0x8000)??? >> > >> > >> > That uses an invalid pkey so this is not a valid configuration. >> > >> > -- Hal >> > >> >> >> >> My Test result. >> >> >> >> partitions.conf >> >> >> >> Default=0x7fff, : ALL=full ; >> >> Net0=0x0000, ipoib : ALL=full ; >> >> Net1=0x0001, ipoib : ALL=full ; >> >> Net2=0x0002, ipoib : ALL=full ; >> >> >> >> 1) Server1 ib.8000 -> Server1 ib.8000 >> >> Ping NG >> >> >> >> 2) Server1 ib.8001 -> Server1 ib.8001 >> >> >> >> The differece is subinterfae number. >> >> >> >> Yesterday, I always tested subinterface 0x8000. >> >> >> >> >> >> >> >> And about previous post, >> >> >> >> The partitions.conf which I posted few hous ago, >> >> was semi colon missing. >> >> >> >> > * partitions.conf is the following >> >> > Default=0x7fff, ipoib : ALL=full >> >> > Net0=0x0001, ipoib : ALL=full >> >> >> >> Should be >> >> >> >> Default=0x7fff, ipoib : ALL=full ; >> >> Net0=0x0001, ipoib : ALL=full ; >> >> >> >> Thank you again. >> >> >> >> >> >> 2011/10/27 Hiroyuki Sato <[email protected]>: >> >> > Hello Hal >> >> > >> >> > Thank you for your information. >> >> > >> >> > I simplified my test environment >> >> > The environment is the folloing. >> >> > >> >> > I'm not sure what is wrong Test2 (Partition test) >> >> > And Could you please tell me how to check the problem?? >> >> > (tool, logfile, etc.. ) >> >> > >> >> > * Environment >> >> > >> >> > OS: Scientific Linux6.1 >> >> > OFED: 1.5.3. >> >> > >> >> > * Diagram >> >> > >> >> > +----------+ib0 ib0+----------+ >> >> > | Server1 |------------------| Server2 | >> >> > +----------+ +----------+ >> >> > >> >> > >> >> > >> >> > >> >> > 1) Test1 (Simple IPoIB no partition Ping test) >> >> > >> >> > Server1 >> >> > >> >> > a) /sbin/ifconfig ib0 inet 192.168.1.1/24 >> >> > b) /sbn/service opensmd start >> >> > 3) /sbin/service opensmd start >> >> > * no /etc/opensm/partitions.conf >> >> > >> >> > Server2 >> >> > /sbin/ifconfig ib0 inet 192.168.1.1/24 >> >> > >> >> > Test from server1(192.168.1.1) to server2(192.168.1.2) >> >> > ping 192.168.1.2 >> >> > The result was OK. >> >> > >> >> > 2) Test2 (Add partition by hand and Ping test) >> >> > >> >> > reboot Server 1 and Server 2 >> >> > >> >> > Server 1 >> >> > 1) echo 0x8001 > /sys/class/net/ib0/create_child >> >> > 2) /sbin/ifconfig ib0.8001 inet 192.168.0.1/24 >> >> > 3) /sbin/service opensmd start >> >> > >> >> > * partitions.conf is the following >> >> > Default=0x7fff, ipoib : ALL=full >> >> > Net0=0x0001, ipoib : ALL=full >> >> > >> >> > Server2 >> >> > 1) echo 0x8001 > /sys/class/net/ib0/create_child >> >> > 2) /sbin/ifconfig ib0.8001 inet 192.168.0.2/24 >> >> > >> >> > Test from server1(ib0.8001/192.168.0.1) to >> >> > server2(ib0.8001/192.168.0.2) >> >> > >> >> > ping 192.168.0.2 >> >> > no response. >> >> > >> >> > 3) Test2 Log >> >> > >> >> > I enabled debug_log parameter in ib_ipoib modules. >> >> > >> >> > cd /sys/module/ib_ipoib/parameters >> >> > # cat debug_level >> >> > 1 >> >> > >> >> > Could you please tell me what is wrong?? >> >> > >> >> > ib0: enabling connected mode will cause multicast packet drops >> >> > ib0: mtu > 2044 will cause multicast packet drops. >> >> > ib0: mtu > 2044 will cause multicast packet drops. >> >> > ib1: enabling connected mode will cause multicast packet drops >> >> > ib1: mtu > 2044 will cause multicast packet drops. >> >> > ib1: mtu > 2044 will cause multicast packet drops. >> >> > ib0: Event 17 on device mthca0 port 1 >> >> > ib0: Not flushing - IPOIB_FLAG_INITIALIZED not set. >> >> > ib0: Event 11 on device mthca0 port 1 >> >> > ib0: Not flushing - IPOIB_FLAG_INITIALIZED not set. >> >> > ib0: Event 9 on device mthca0 port 1 >> >> > ib0: Not flushing - IPOIB_FLAG_INITIALIZED not set. >> >> > ib0.8001: max_srq_sge=27 >> >> > ib0.8001: max_cm_mtu = 0xfff0, num_frags=16 >> >> > ib0.8001: bringing up interface >> >> > ib0.8001: IPOIB_FLAG_OPER_UP not set >> >> > ib0.8001: IPOIB_FLAG_OPER_UP not set >> >> > ADDRCONF(NETDEV_UP): ib0.8001: link is not ready >> >> > ib0.8001: IPOIB_FLAG_OPER_UP not set >> >> > >> >> > regards. >> >> > >> >> > 2011/10/27 Hal Rosenstock <[email protected]>: >> >> >> >> >> >> >> >> >> On Wed, Oct 26, 2011 at 10:03 AM, Hiroyuki Sato >> >> >> <[email protected]> >> >> >> wrote: >> >> >>> >> >> >>> Dear members. >> >> >>> >> >> >>> I have some question about Infiniband Partitions. >> >> >>> >> >> >>> I would like to build Linux box with IPoIB Router. >> >> >>> >> >> >>> * Questions >> >> >>> >> >> >>> My Question is the following. >> >> >>> >> >> >>> a) Can I create the following IPoIB network ?? >> >> >>> >> >> >>> b) If so, how to setup /etc/opensm/partitions.conf?? >> >> >>> >> >> >>> * Problem. >> >> >>> >> >> >>> Ping IB Client to Router : I got the followig error. >> >> >>> >> >> >>> Network is unreachable. >> >> >>> >> >> >>> * Environment >> >> >>> >> >> >>> OFED: OFED-1.5.3.2 >> >> >>> OS: Scientific Linux 6.1 >> >> >>> >> >> >>> * Linux box (Router) >> >> >>> >> >> >>> 1 ethernet interface (eth0) >> >> >>> 1 Infiniband interfae (ib0) >> >> >>> Run subnet manager. >> >> >>> >> >> >>> * Ethernet setup >> >> >>> >> >> >>> a) add VLAN ID 120 eth0.120 (192.168.120.0/24) >> >> >>> b) add VLAN ID 130 eth0.130 (192.168.130.0/24) >> >> >>> >> >> >>> * Infiniband Setup >> >> >>> a) add subinterface 0 ib0.8000 (192.168.0.0/24) >> >> >>> b) add subinterface 1 ib0.8001 (192.168.1.0/24) >> >> >>> c) Run opensm >> >> >>> >> >> >>> >> >> >>> I crated subinterface with the following command. >> >> >>> >> >> >>> echo 0 > /sys/class/net/ib0/create_child >> >> >>> echo 1 > /sys/class/net/ib0/create_child >> >> >> >> >> >> >> >> >> echo 0x8001 > /sys/class/net/ib0/create_child >> >> >> echo 0x8002 > /sys/class/net/ib0/create_child >> >> >> >> >> >>> >> >> >>> and assign ip address with ifconfig >> >> >>> >> >> >>> * Logical Diagram >> >> >>> >> >> >>> +----------- 192.168.120.0/24 --- >> >> >>> | >> >> >>> | +---- 192.168.130.0/24 >> >> >>> eth0.120| | eth0.130 >> >> >>> +----------+ >> >> >>> | Router | >> >> >>> +----------+ >> >> >>> ib0.8000| |ib0.8001 >> >> >>> 192.168.0.1 | +---- 192.168.1.0/24 >> >> >>> | >> >> >>> +----------- 192.168.0.0/24 --- IB Cllient. >> >> >>> (ib0.8000) >> >> >>> (192.168.0.2) >> >> >>> >> >> >>> * /etc/opensm/partitions.conf >> >> >>> >> >> >>> Default=0x7fff : ALL=full >> >> >>> Net0=0x00, ipoib : ALL=full >> >> >>> Net1=0x01, ipoib : ALL=full >> >> >> >> >> >> >> >> >> Default=0x7fff: ALL=full >> >> >> Net0=0x0001, ipoib: ALL=full >> >> >> Net1=0x0002, ipoib: ALL=full >> >> >> >> >> >> >> >> >>> >> >> >>> Coule you please tell me what is wrong?? >> >> >> >> >> >> >> >> >> The one thing in the above I'm not sure of is whether the default >> >> >> partition >> >> >> also needs the ipoib flag above so you might need to add that too if >> >> >> the >> >> >> above doesn't work. >> >> >> >> >> >> -- Hal >> >> >> >> >> >>> >> >> >>> Sincerely. >> >> >>> >> >> >>> -- >> >> >>> Hiroyuki Sato. >> >> >>> _______________________________________________ >> >> >>> ewg mailing list >> >> >>> [email protected] >> >> >>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg >> >> >> >> >> >> >> >> > >> >> > >> >> > >> >> > -- >> >> > Hiroyuki Sato >> >> > >> >> >> >> >> >> >> >> -- >> >> Hiroyuki Sato >> > >> > >> >> >> >> -- >> Hiroyuki Sato > > -- Hiroyuki Sato _______________________________________________ ewg mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
