Re: [ofa-general] Combined DR path with empty DR path, what is the expected behavior?
On Wed, 26 Aug 2009 10:55:41 -0400 Hal Rosenstock wrote: > On 8/25/09, Ira Weiny wrote: > > > > On Tue, 25 Aug 2009 19:15:19 -0400 > > Hal Rosenstock wrote: > > > > > On 8/24/09, Ira Weiny wrote: > > > [snip] > > > > > > > > > Not all 4 combinations are supported/known to work. When this was added > > for > > > ibportstate, the only combined routing form that was important was LID > > > routed part followed by a DR part. > > > > > > > When you say "known to work" you mean implemented with the diags? Or known > > to > > work in all hardware? > > > The former with most hardware up to some time ago. Note there is no > compliance testing of combined routing and heavy reliance on this makes some > a little nervous. Ok, Good to know. With this, and the rest of your response, in mind I went ahead and created a patch to libibnetdisc which will go back to LID routing when the Hop Count is returned to 0. Patch to follow. > > > > > > > On the other hand I think strictly this should be supported. > > > > > > > > > In an ideal world yes but are they all required or is it just the one > > form > > > most heavily used ? > > > > That is what I am unclear on. Does the spec require that all 8 > > combinations > > are required to work? I don't see a specific compliance which says that > > and I > > am not sure if C14-9 and C14-13 cover all 8 combinations. > > > I don't think there's any compliance on this. It all appears to be > informative text. Perhaps a shortcoming of the spec. So there's nothing > definitive. It just says there are 8 combinations (2**3 as there are 3 parts > with 2 possibilities in each part) and that only 4 are really useful. Well I agree that only 4 are "useful". It is just the algorithm which libibnetdisc used which resulted in this "weird" case. [snip] > > > > > > If so, what's the initial path at this point (or more specifically index > > 1 > > > of the initial path) ? I think that needs to be port 0 (if a switch) but > > > this is a little weird as I would think it should be handed to the SMA > > which > > > is different cases in the spec. > > > > Yes I think I was wrong on the case. But still wouldn't the SMI detect > > that > > this is the end of the DRPath and simply hand it to the SMA. > > > Yes, that's what should happen. I am going to take this up with the switch vendors and see what their interpretation is. For the time being I think my patch will fix libibnetdisc (iblinkinfo). Thanks again! Ira > > > > > > > > > > Then after processing > > > > > > > > > by the SMA and doing the required returning initialization > > > > > > the SMI should return the packet as specified in C14-13 > > > > item 3 on line 9 page 812. > > > > > > > > > I'm not sure it would use this case in the case of an empty DR pafh on > > > return. > > > > Actually I think it will use this. C14-9 item 3) states "the Hop Pointer > > shall be incremented by 1" Therefore when the response is handed back to > > the > > SMI the Hop pointer will be 1 and the hop count 0. And the SMI uses the > > DRSLID to send the packet back to the requester. > > > It goes up to the SMA and then when the response is to be made it goes > through returning SMI initialization and handling. > > -- Hal > > > > > > Am I wrong? In the end it does not matter as I have to make the software > > > > work > > > > for all the hardware I have; so I will change the software. > > > > > > > > > IMO it does matter as to where the problem lies (SMI or otherwise) and > > how > > > the layers are comprised in the implementation. > > > > Agreed. I am mainly confused because I have 2 different implementations of > > this. My "old" switches seem to handle this case just fine. My "new" > > switches do not. So I am really wondering what is going on. > > > > Here is the above output for the same query which works with an "old" > > switch. > > > > 17:28:04 > ./smpquery -e -c portinfo 7 0 1 > > ... > > trid 1a4329de; HopCount 0; HopPointer 0; slid 2; dlid 65535; 0, drpath->cnt > > 0 > > ... > > > > Aug 25 17:46:40 woprjr0 Madeye:sent SMP > > Aug 25 17:46:40 woprjr0 MAD version0x1 > > Aug 25 17:46:40 woprjr0 Class..0x81 (Directed route SMP) > > Aug 25 17:46:40 woprjr0 Class version..0x1 > > Aug 25 17:46:40 woprjr0 Method.0x1 (Get) > > Aug 25 17:46:40 woprjr0 Status.0x00 > > Aug 25 17:46:40 woprjr0 Hop pointer0x0 > > Aug 25 17:46:40 woprjr0 Hop counter0x0 > > Aug 25 17:46:40 woprjr0 Trans ID...0x1ba01a4329de > > Aug 25 17:46:40 woprjr0 Attr ID0x15 (port info) > > Aug 25 17:46:40 woprjr0 Attr modifier..0x0001 > > Aug 25 17:46:40 woprjr0 Mkey...0x0 > > Aug 25 17:46:40 woprjr0 DR SLID0x02 > > Aug 25 17:46:40 woprjr0 DR DLID0x > > Aug 25 17:46:40 woprjr0 Madeye:recv SMP > > Aug 25 17:46:40 woprjr0 MAD version0x1 > > Aug 25 17:46:40 woprjr0 Class..0x81 (Directed route SMP) > > Aug 25 17:46:40 woprjr0 Class version..0x1 > > Aug 25 17:46:40 woprjr0 Method
Re: [ofa-general] Combined DR path with empty DR path, what is the expected behavior?
On 8/25/09, Ira Weiny wrote: > > On Tue, 25 Aug 2009 19:15:19 -0400 > Hal Rosenstock wrote: > > > On 8/24/09, Ira Weiny wrote: > > > > > If I send a combined DR path with a start lid but an empty (0 length) > DR > > > path. > > > > > > Hop Count 0 ? > > Yes > > > > > > > > What is the expected behavior? > > > > > > Not sure what you mean by expected here. Are you referring to expectation > > based on the spec ? > > > > yes > > > > > > I know this could be specified with LID routing, but I don't see > anywhere > > > in > > > the specification which says this is an error. > > > > > > I don't think it should be an error (certainly not for the form you are > > using LID routed part followed by a DR part) but a null DR part is a > little > > funny/odd. > > Yea I know. It turns out that the new iblinkinfo issues queries like this > when it is removes recurses back from the last DR portion of the combined > route path. It only showed up as an error when using the -S option > of > iblinkinfo with this new switch I have. Works fine with the old switches. > > > > > > I do however seem to have 2 > > > different implementations on 2 different switches. For example: > > > > > > I have Switch A (Lid 1) and Switch B (Lid 7). I attempt to query > PortInfo > > > of > > > Port 1 of each switch using the LID followed by an empty DR path. > > > > > > 17:55:22 > ./smpquery -c portinfo 1 0 1 > > > ibwarn: [21005] mad_rpc: _do_madrpc failed; dport (Lid 1) > > > ./smpquery: iberror: failed: operation portinfo: port info query failed > > > > > > Is this a timeout ? > > yes > > 16:26:25 > ./smpquery -e -c portinfo 1 0 1 > ibwarn: [27150] _do_madrpc: retry 1 (timeout 1000 ms) > ibwarn: [27150] _do_madrpc: retry 2 (timeout 1000 ms) > ibwarn: [27150] _do_madrpc: timeout after 3 retries, 3000 ms > ibwarn: [27150] mad_rpc: _do_madrpc failed; dport (Lid 1) > ./smpquery: iberror: failed: operation portinfo: port info query failed > > > > > > > > > 17:55:31 > ./smpquery -c portinfo 7 0 1 > > > # Port info: Lid 7 port 1 > > > Mkey:0x > > > GidPrefix:...0x > > > ... > > > > > > > > > Detecting this special case in libibmad and turning the packet into a > LID > > > routed one > > > > > > Ugh... Is this special case really needed ? I don't think the underlying > > issue is understood sufficiently yet. > > Well I just did it to prove that what I was doing would work with a > "simple" > lid routed packet. Like I said it might be that this portid which is being > specified to libibmad by libibnetdisc is not valid. If that is true then > libibnetdisc should detect when the DR path is empty and go back to LID > routed > requests. That is a valid fix in my mind. Sure; there's no real need for combined route when the DR path is empty but it should work (at least with switches). > > > > succeeds but I wonder if this is an error in the SMI? > > > > > > Switch SMI ? Is this a proprietary implementation ? > > > > Yes I see the bug with 2 different vendors switches. One is managed and > the > other is not. My "old" switches (3 different vendors) do not show this > behavior. (Just to be clear I now I have 5 switches in my 5 node cluster! > ;-) > > > > > > > > I also notice this is an error on the HCA I am running from (lid 2). > > > > > > Is this HCA node OpenIB based ? > > yes If I recall correctly, there is something in the spec that makes combined routing not be allowed on HCA (and router) ports so this seems correct. I can dig this out if really needed. > > > 17:57:42 > ./smpquery -c portinfo 2 0 1 > > > ibwarn: [21008] mad_rpc: _do_madrpc failed; dport (Lid 2) > > > ./smpquery: iberror: failed: operation portinfo: port info query failed > > > > > > Is this also a timeout ? > > yes > > > > > Also, does the result differ based on where you source these from matter > > (locally v. remotely)? > > Same result local and remote. > > > > > > > > > > Running with a simple DR path works, > > > > > > You're referring to the same DR path here that fails in the combined > route > > examples above, right ? > > > > No. the example below is a DR path with Hop Count == 0 but without the > initial > LID routing. > > > > > > I guess because this is the loopback case mentioned on page 805. > > > > > > Yes but that's the high level requirement rather than the SMI rules which > > make that work. > > > > > > > > > 17:58:16 > ./smpquery -D portinfo 0 1 > > > # Port info: DR path slid 65535; dlid 65535; 0 port 1 > > > Mkey:0x > > > GidPrefix:...0x2007 > > > ... > > > > > > > > > It guess that the comment "Since each part may be empty, there are > eight > > > combinations, although only four are really useful:" on line 36 Page > 805 > > > can > > > be interpreted to mean that only those 4 combinations need to be > supported. > > > Is this true? > > > > > > Not all 4 combinations are supported/known
Re: [ofa-general] Combined DR path with empty DR path, what is the expected behavior?
On Tue, 25 Aug 2009 19:15:19 -0400 Hal Rosenstock wrote: > On 8/24/09, Ira Weiny wrote: > > > If I send a combined DR path with a start lid but an empty (0 length) DR > > path. > > > Hop Count 0 ? Yes > > > > What is the expected behavior? > > > Not sure what you mean by expected here. Are you referring to expectation > based on the spec ? > yes > > > I know this could be specified with LID routing, but I don't see anywhere > > in > > the specification which says this is an error. > > > I don't think it should be an error (certainly not for the form you are > using LID routed part followed by a DR part) but a null DR part is a little > funny/odd. Yea I know. It turns out that the new iblinkinfo issues queries like this when it is removes recurses back from the last DR portion of the combined route path. It only showed up as an error when using the -S option of iblinkinfo with this new switch I have. Works fine with the old switches. > > > I do however seem to have 2 > > different implementations on 2 different switches. For example: > > > > I have Switch A (Lid 1) and Switch B (Lid 7). I attempt to query PortInfo > > of > > Port 1 of each switch using the LID followed by an empty DR path. > > > > 17:55:22 > ./smpquery -c portinfo 1 0 1 > > ibwarn: [21005] mad_rpc: _do_madrpc failed; dport (Lid 1) > > ./smpquery: iberror: failed: operation portinfo: port info query failed > > > Is this a timeout ? yes 16:26:25 > ./smpquery -e -c portinfo 1 0 1 ibwarn: [27150] _do_madrpc: retry 1 (timeout 1000 ms) ibwarn: [27150] _do_madrpc: retry 2 (timeout 1000 ms) ibwarn: [27150] _do_madrpc: timeout after 3 retries, 3000 ms ibwarn: [27150] mad_rpc: _do_madrpc failed; dport (Lid 1) ./smpquery: iberror: failed: operation portinfo: port info query failed > > > > 17:55:31 > ./smpquery -c portinfo 7 0 1 > > # Port info: Lid 7 port 1 > > Mkey:0x > > GidPrefix:...0x > > ... > > > > > > Detecting this special case in libibmad and turning the packet into a LID > > routed one > > > Ugh... Is this special case really needed ? I don't think the underlying > issue is understood sufficiently yet. Well I just did it to prove that what I was doing would work with a "simple" lid routed packet. Like I said it might be that this portid which is being specified to libibmad by libibnetdisc is not valid. If that is true then libibnetdisc should detect when the DR path is empty and go back to LID routed requests. That is a valid fix in my mind. > > > succeeds but I wonder if this is an error in the SMI? > > > Switch SMI ? Is this a proprietary implementation ? > Yes I see the bug with 2 different vendors switches. One is managed and the other is not. My "old" switches (3 different vendors) do not show this behavior. (Just to be clear I now I have 5 switches in my 5 node cluster! ;-) > > > > I also notice this is an error on the HCA I am running from (lid 2). > > > Is this HCA node OpenIB based ? yes > > 17:57:42 > ./smpquery -c portinfo 2 0 1 > > ibwarn: [21008] mad_rpc: _do_madrpc failed; dport (Lid 2) > > ./smpquery: iberror: failed: operation portinfo: port info query failed > > > Is this also a timeout ? yes > > Also, does the result differ based on where you source these from matter > (locally v. remotely)? Same result local and remote. > > > > > Running with a simple DR path works, > > > You're referring to the same DR path here that fails in the combined route > examples above, right ? > No. the example below is a DR path with Hop Count == 0 but without the initial LID routing. > > > I guess because this is the loopback case mentioned on page 805. > > > Yes but that's the high level requirement rather than the SMI rules which > make that work. > > > > > 17:58:16 > ./smpquery -D portinfo 0 1 > > # Port info: DR path slid 65535; dlid 65535; 0 port 1 > > Mkey:0x > > GidPrefix:...0x2007 > > ... > > > > > > It guess that the comment "Since each part may be empty, there are eight > > combinations, although only four are really useful:" on line 36 Page 805 > > can > > be interpreted to mean that only those 4 combinations need to be supported. > > Is this true? > > > Not all 4 combinations are supported/known to work. When this was added for > ibportstate, the only combined routing form that was important was LID > routed part followed by a DR part. > When you say "known to work" you mean implemented with the diags? Or known to work in all hardware? > > > On the other hand I think strictly this should be supported. > > > In an ideal world yes but are they all required or is it just the one form > most heavily used ? That is what I am unclear on. Does the spec require that all 8 combinations are required to work? I don't see a specific compliance which says that and I am not sure if C14-9 and C14-13
Re: [ofa-general] Combined DR path with empty DR path, what is the expected behavior?
On 8/24/09, Ira Weiny wrote: > If I send a combined DR path with a start lid but an empty (0 length) DR > path. Hop Count 0 ? > What is the expected behavior? Not sure what you mean by expected here. Are you referring to expectation based on the spec ? > I know this could be specified with LID routing, but I don't see anywhere > in > the specification which says this is an error. I don't think it should be an error (certainly not for the form you are using LID routed part followed by a DR part) but a null DR part is a little funny/odd. > I do however seem to have 2 > different implementations on 2 different switches. For example: > > I have Switch A (Lid 1) and Switch B (Lid 7). I attempt to query PortInfo > of > Port 1 of each switch using the LID followed by an empty DR path. > > 17:55:22 > ./smpquery -c portinfo 1 0 1 > ibwarn: [21005] mad_rpc: _do_madrpc failed; dport (Lid 1) > ./smpquery: iberror: failed: operation portinfo: port info query failed Is this a timeout ? > 17:55:31 > ./smpquery -c portinfo 7 0 1 > # Port info: Lid 7 port 1 > Mkey:0x > GidPrefix:...0x > ... > > > Detecting this special case in libibmad and turning the packet into a LID > routed one Ugh... Is this special case really needed ? I don't think the underlying issue is understood sufficiently yet. > succeeds but I wonder if this is an error in the SMI? Switch SMI ? Is this a proprietary implementation ? > I also notice this is an error on the HCA I am running from (lid 2). Is this HCA node OpenIB based ? 17:57:42 > ./smpquery -c portinfo 2 0 1 > ibwarn: [21008] mad_rpc: _do_madrpc failed; dport (Lid 2) > ./smpquery: iberror: failed: operation portinfo: port info query failed Is this also a timeout ? Also, does the result differ based on where you source these from matter (locally v. remotely)? > Running with a simple DR path works, You're referring to the same DR path here that fails in the combined route examples above, right ? > I guess because this is the loopback case mentioned on page 805. Yes but that's the high level requirement rather than the SMI rules which make that work. > 17:58:16 > ./smpquery -D portinfo 0 1 > # Port info: DR path slid 65535; dlid 65535; 0 port 1 > Mkey:0x > GidPrefix:...0x2007 > ... > > > It guess that the comment "Since each part may be empty, there are eight > combinations, although only four are really useful:" on line 36 Page 805 > can > be interpreted to mean that only those 4 combinations need to be supported. > Is this true? Not all 4 combinations are supported/known to work. When this was added for ibportstate, the only combined routing form that was important was LID routed part followed by a DR part. > On the other hand I think strictly this should be supported. In an ideal world yes but are they all required or is it just the one form most heavily used ? > Item 4 of C14-9 > (line 24 page 810) requires the SMI to handle the packet if the HopPointer > equals HopCount +1, which it is in my case (HopCount == 0, HopPointer == 1) By handle, this means "The SMI *shall *output the packet on the port whose number is in the entry indexed by Hop Pointer in the Initial Path. If that port number is invalid, the SMI *shall *discard the SMP." Are you sure the Hop Pointer is 1 ? Where do you see this ? If so, what's the initial path at this point (or more specifically index 1 of the initial path) ? I think that needs to be port 0 (if a switch) but this is a little weird as I would think it should be handed to the SMA which is different cases in the spec. > Then after processing by the SMA and doing the required returning initialization the SMI should return the packet as specified in C14-13 > item 3 on line 9 page 812. I'm not sure it would use this case in the case of an empty DR pafh on return. Am I wrong? In the end it does not matter as I have to make the software > work > for all the hardware I have; so I will change the software. IMO it does matter as to where the problem lies (SMI or otherwise) and how the layers are comprised in the implementation. However, I wonder > where exactly the spec falls on this, because I think it will influence > where > the fix resides. If the spec does not allow this then I think it is fine > to > have libibmad return an error since the user specified an invalid combined > DR > path. However, if this should be legal I think libibmad should work around > the bad hardware out there. Is it hardware or firmware that needs fixing ? I think it may depend on the specific workaround for this as to whether it is acceptable as it might harm something else or might violate the spec. -- Hal Thoughts? > Ira > > -- > Ira Weiny > Math Programmer/Computer Scientist > Lawrence Livermore National Lab > 925-423-8008 > [email protected] > __
