Re: [openib-general] Problem is routing CM REQ

2007-02-14 Thread Sean Hefty
I agree with what was in your response, however, this is how I interpret your answers: >> Active side QP - DLID 2 >> Passive side QP - DLID 94 >> CM REQ Primary Local Port LID no answer given > - CM creates a REQ and populates the global information to identify the > remote endnode. The LRH

Re: [openib-general] Problem is routing CM REQ

2007-02-14 Thread Michael Krause
At 01:36 PM 2/14/2007, Sean Hefty wrote: >Assume that the active and passive sides of a connection request are on >different subnets and: > >Active side - LID 1 >Active side router - LID 2 >Passive side - LID 93 >Passive side router - LID 94 > >What values are you suggesting are used for: > >Activ

Re: [openib-general] Problem is routing CM REQ

2007-02-14 Thread Sean Hefty
Assume that the active and passive sides of a connection request are on different subnets and: Active side - LID 1 Active side router - LID 2 Passive side - LID 93 Passive side router - LID 94 What values are you suggesting are used for: Active side QP - DLID Passive side QP - DLID CM REQ Prima

Re: [openib-general] Problem is routing CM REQ

2007-02-13 Thread Sean Hefty
>A LID is subnet local on that we can all agree. The CM Req contains >either the LID of a local subnet CA or the LID a local router which will >move the packet to the next hop to the destination. 12.7.11 is basically >saying that the remote LID is the router's LID of the local subnet's router >

Re: [openib-general] Problem is routing CM REQ

2007-02-13 Thread Michael Krause
At 02:02 PM 2/13/2007, Jason Gunthorpe wrote: >On Tue, Feb 13, 2007 at 12:49:57PM -0800, Michael Krause wrote: > > > >Translated into a network with routers this means that for a RC flow > > >to successfully work both the *forward* and *reverse* direction must > > >traverse the same router *LID* no

Re: [openib-general] Problem is routing CM REQ

2007-02-13 Thread Michael Krause
At 01:14 PM 2/13/2007, Sean Hefty wrote: >>It does not need to comprehend the remote subnet(s) LID. >>That is the router protocol to determine. CM also must understand the >>GIDs involved which the router will process to figure out its LID mapping >>to the next hop. > >The CM REQ carries the rem

Re: [openib-general] Problem is routing CM REQ

2007-02-13 Thread Sean Hefty
> It does not need to comprehend the remote subnet(s) LID. > That is the router protocol to determine. CM also must understand the > GIDs involved which the router will process to figure out its LID > mapping to the next hop. The CM REQ carries the remote router LID (primary local port lid -

Re: [openib-general] Problem is routing CM REQ

2007-02-13 Thread Michael Krause
At 04:10 PM 2/12/2007, Jason Gunthorpe wrote: >On Mon, Feb 12, 2007 at 03:31:15PM -0800, Michael Krause wrote: > > > TClass is intended to communicate the end-to-end QoS desired. TClass is > > then mapped to a SL that is local to each subnet. A flow label is > > intended to much the same as in

Re: [openib-general] Problem is routing CM REQ

2007-02-13 Thread Michael Krause
At 03:48 PM 2/12/2007, Sean Hefty wrote: >>An endnode look up should be to find the address vector to the >>remote. A look up may return multiple vectors. The SLID would >>correspond to each local subnet router port that acts as a first-hop >>destination to the remote subnet.I don't see

Re: [openib-general] Problem is routing CM REQ

2007-02-13 Thread Sean Hefty
>What your #4 and #5 are talking about is not just that, but also PR >queries that can unambigously identify the LID selections of the >router in advance. That is hugely different! IMHO, just because a >reversible path exists and will be used by the router shouldn't be >taken to mean that the it is

Re: [openib-general] Problem is routing CM REQ

2007-02-12 Thread Jason Gunthorpe
On Mon, Feb 12, 2007 at 04:45:33PM -0800, Sean Hefty wrote: > >>4. A PR from the local SA with reversible=1 indicates that data sent from > >>the remote GID to the local GID using the PR TC and FL will route locally > >>using the specified LID pair. This holds whether the PR SGID is local or >

Re: [openib-general] Problem is routing CM REQ

2007-02-12 Thread Sean Hefty
>>4. A PR from the local SA with reversible=1 indicates that data sent from >>the remote GID to the local GID using the PR TC and FL will route locally >>using the specified LID pair. This holds whether the PR SGID is local or >>remote. > >>5. A PR from a remote SA with reversible=1 indicates

Re: [openib-general] Problem is routing CM REQ

2007-02-12 Thread Jason Gunthorpe
On Mon, Feb 12, 2007 at 03:31:15PM -0800, Michael Krause wrote: > TClass is intended to communicate the end-to-end QoS desired. TClass is > then mapped to a SL that is local to each subnet. A flow label is > intended to much the same as in the IP world and is left, in essence, to > routers

Re: [openib-general] Problem is routing CM REQ

2007-02-12 Thread Jason Gunthorpe
On Mon, Feb 12, 2007 at 02:47:42PM -0800, Sean Hefty wrote: > Maybe it would help if we can agree on a set of expectations. These are > what I am thinking: > > 1. An SA should be able to respond to a valid PR query if at least one of > the GIDs in the path record is local. > > 2. The LIDs in

Re: [openib-general] Problem is routing CM REQ

2007-02-12 Thread Michael Krause
At 02:47 PM 2/12/2007, Sean Hefty wrote: > > 1) What does the TClass and FlowLabel returned from SGID=local > >DGID=remote mean? > >Do you use it in the Node1 -> Node2 direction or the Node2 -> Node1 > direction > >or both? > >Maybe it would help if we can agree on a set of expectation

Re: [openib-general] Problem is routing CM REQ

2007-02-12 Thread Sean Hefty
> An endnode look up should be to find the address > vector to the remote. A look up may return multiple vectors. The > SLID would correspond to each local subnet router port that acts as a > first-hop destination to the remote subnet.I don't see why the > router protocol would not simp

Re: [openib-general] Problem is routing CM REQ

2007-02-12 Thread Michael Krause
At 12:56 PM 2/12/2007, Jason Gunthorpe wrote: >On Mon, Feb 12, 2007 at 09:23:06AM -0800, Sean Hefty wrote: > > >Ah, I think I missed the key step in your scheme.. You plan to query > > >the local SM for SGID=remote DGID=local? (ie reversed from 'normal'. I > > >was thinking only about the SGID=loca

Re: [openib-general] Problem is routing CM REQ

2007-02-12 Thread Sean Hefty
> 1) What does the TClass and FlowLabel returned from SGID=local >DGID=remote mean? >Do you use it in the Node1 -> Node2 direction or the Node2 -> Node1 > direction >or both? Maybe it would help if we can agree on a set of expectations. These are what I am thinking: 1. An SA should

Re: [openib-general] Problem is routing CM REQ

2007-02-12 Thread Jason Gunthorpe
On Mon, Feb 12, 2007 at 09:23:06AM -0800, Sean Hefty wrote: > >Ah, I think I missed the key step in your scheme.. You plan to query > >the local SM for SGID=remote DGID=local? (ie reversed from 'normal'. I > >was thinking only about the SGID=local DGID=remote query direction) > > I'm not sure that

Re: [openib-general] Problem is routing CM REQ

2007-02-12 Thread Todd Rimmer
> From: Sean Hefty > Sent: Monday, February 12, 2007 12:23 PM > To: Jason Gunthorpe; Hal Rosenstock > Cc: openib-general@openib.org > Subject: Re: [openib-general] Problem is routing CM REQ There has been a lot of good discussion and proposed designs for this solution. I think i

Re: [openib-general] Problem is routing CM REQ

2007-02-12 Thread Sean Hefty
> Ah, I think I missed the key step in your scheme.. You plan to query > the local SM for SGID=remote DGID=local? (ie reversed from 'normal'. I > was thinking only about the SGID=local DGID=remote query direction) I'm not sure that the query needs the GIDs reversed, as long as the path is reversi

Re: [openib-general] Problem is routing CM REQ

2007-02-11 Thread Jason Gunthorpe
On Fri, Feb 09, 2007 at 06:08:34PM -0800, Sean Hefty wrote: > >So basically what you are saying is that the TClass and FlowLabel act > >as some kind of global dis-ambiguation that lets all SAs know that the > >tuple MUST be matched with > >on each side. > > Sort of... My reasoning is that if yo

Re: [openib-general] Problem is routing CM REQ

2007-02-09 Thread Sean Hefty
>So basically what you are saying is that the TClass and FlowLabel act >as some kind of global dis-ambiguation that lets all SAs know that the >tuple MUST be matched with >on each side. Sort of... My reasoning is that if you look at a packet traveling from the source QP to the destination QP, a

Re: [openib-general] Problem is routing CM REQ

2007-02-09 Thread Jason Gunthorpe
On Fri, Feb 09, 2007 at 03:08:12PM -0800, Sean Hefty wrote: > The route itself is determined using the SGID, DGID, TClass, FlowLabel. > So, as long as the two queries match on these fields, I would think that it > would work. So basically what you are saying is that the TClass and FlowLabel ac

Re: [openib-general] Problem is routing CM REQ

2007-02-09 Thread Sean Hefty
> The hard part is the global distribution of this information. The best idea I can come up with for locating remote SAs is to have the SAs assign themselves a specific Unicast Global GID Assigned Value. So, each SA gives themselves a GID similar to: 64-bit subnet prefix :: 1. Hosts on remote

Re: [openib-general] Problem is routing CM REQ

2007-02-09 Thread Sean Hefty
> Sean: Even if you can query both SA's there isn't enough information > to force things to use the same router path in each direction. My assumption is that the remote SA contains the necessary information about how a packet coming from the local SGID to the remote DGID would be routed on the

Re: [openib-general] Problem is routing CM REQ

2007-02-09 Thread Jason Gunthorpe
On Fri, Feb 09, 2007 at 04:45:29PM -0500, Hal Rosenstock wrote: > >Off hand I don't see that the existing path record query structure > >has enough information to do this.. Particularly, in cases > >where each subnet has more than 1 router port there is no real > >guarentee that qu

Re: [openib-general] Problem is routing CM REQ

2007-02-09 Thread Hal Rosenstock
On Fri, 2007-02-09 at 15:34, Sean Hefty wrote: > > the /missing part (right now) is locating the SA on that > > remote subnet if this is a needed function. > > Maybe we can expose this to SA clients through a ServiceRecord? That might be one way if there were a standardized service name for SA an

Re: [openib-general] Problem is routing CM REQ

2007-02-09 Thread Hal Rosenstock
On Fri, 2007-02-09 at 14:20, Jason Gunthorpe wrote: > On Fri, Feb 09, 2007 at 12:58:51PM -0500, Hal Rosenstock wrote: > > > For simplicity, assume a single path. My assumption in this case was > > > that the > > > SLID/DLID values would be reversed. That is, the LIDs are relative to > > > the

Re: [openib-general] Problem is routing CM REQ

2007-02-09 Thread Sean Hefty
> the /missing part (right now) is locating the SA on that > remote subnet if this is a needed function. Maybe we can expose this to SA clients through a ServiceRecord? This doesn't solve how the two SAs find each other (or any of the other difficult stuff), but with this and the path record q

Re: [openib-general] Problem is routing CM REQ

2007-02-09 Thread Sean Hefty
> - A kind of inter-subnet path record query is needed that can > return a local and remote GRH and LRH. These four structures need to > be *linked* so that: >- Side A GRH.SGID = active side's Port GID >- Side A GRH.DGID = passive side's Port GID >- Side A LRH.SLID = any active side

Re: [openib-general] Problem is routing CM REQ

2007-02-09 Thread Hal Rosenstock
On Fri, 2007-02-09 at 14:56, Sean Hefty wrote: > I don't see a way to issue the SA query to the remote subnet though. Even though SA queries can go intersubnet as they are GMPs and can contain a GRH, the /missing part (right now) is locating the SA on that remote subnet if this is a needed functio

Re: [openib-general] Problem is routing CM REQ

2007-02-09 Thread Jason Gunthorpe
On Fri, Feb 09, 2007 at 12:58:51PM -0500, Hal Rosenstock wrote: > > For simplicity, assume a single path. My assumption in this case was that > > the > > SLID/DLID values would be reversed. That is, the LIDs are relative to the > > local > > subnet, not the SGID. But if I set the SGID = DGID

Re: [openib-general] Problem is routing CM REQ

2007-02-09 Thread Hal Rosenstock
On Fri, 2007-02-09 at 12:22, Sean Hefty wrote: > > SLID corresponding to SGID and a DLID for some IB router on the subnet > > which can route to the remote DGID. > > This was my assumption as well. > > > An SM is free to choose SLID and DLID to supply to if there are multiple > > LIDs for the por

Re: [openib-general] Problem is routing CM REQ

2007-02-09 Thread Sean Hefty
> SLID corresponding to SGID and a DLID for some IB router on the subnet > which can route to the remote DGID. This was my assumption as well. > An SM is free to choose SLID and DLID to supply to if there are multiple > LIDs for the ports in question it can choose alternates. The key here is > wh

Re: [openib-general] Problem is routing CM REQ

2007-02-09 Thread Sean Hefty
> I have a follow up question to this.. With CM how is the SL for each > side determined? I'm looking through the code here and it looks like > the SL of the active side is passed in the REQ to the passive side (ie > both sides are the same) But cma_query_ib_route does not set the > reversible bit

Re: [openib-general] Problem is routing CM REQ

2007-02-09 Thread Hal Rosenstock
On Thu, 2007-02-08 at 23:37, Jason Gunthorpe wrote: > On Thu, Feb 08, 2007 at 03:43:24PM -0800, Sean Hefty wrote: > > > Looking at the problem more, I think that the issue extends to the remote > > > port > > > LID as well. My expectation with a local path record query is that the > > > SLID is

Re: [openib-general] Problem is routing CM REQ

2007-02-09 Thread Hal Rosenstock
On Thu, 2007-02-08 at 18:43, Sean Hefty wrote: > > Looking at the problem more, I think that the issue extends to the remote > > port > > LID as well. My expectation with a local path record query is that the > > SLID is > > the local port, and the DLID is the local router. This should be >

Re: [openib-general] Problem is routing CM REQ

2007-02-08 Thread Jason Gunthorpe
On Thu, Feb 08, 2007 at 03:43:24PM -0800, Sean Hefty wrote: > > Looking at the problem more, I think that the issue extends to the remote > > port > > LID as well. My expectation with a local path record query is that the > > SLID is > > the local port, and the DLID is the local router. This

Re: [openib-general] Problem is routing CM REQ was: Use a GRH when appropriate for unicast packets

2007-02-08 Thread Hal Rosenstock
On Thu, 2007-02-08 at 17:02, Sean Hefty wrote: > >>This requires that the passive side be able to issue path record queries, > >>but I > >>think that it could work for static routes. A point was made to me that the > >>remote side could be a TCA without query capabilities. > > > > Are you referr

Re: [openib-general] Problem is routing CM REQ

2007-02-08 Thread Sean Hefty
> Looking at the problem more, I think that the issue extends to the remote > port > LID as well. My expectation with a local path record query is that the SLID > is > the local port, and the DLID is the local router. This should be sufficient > for > one-way UD traffic, but for connected t

Re: [openib-general] Problem is routing CM REQ was: Use a GRH when appropriate for unicast packets

2007-02-08 Thread Sean Hefty
>>This requires that the passive side be able to issue path record queries, but >>I >>think that it could work for static routes. A point was made to me that the >>remote side could be a TCA without query capabilities. > > Are you referring to SA query capabilities ? Would such a device just be

Re: [openib-general] Problem is routing CM REQ was: Use a GRH when appropriate for unicast packets

2007-02-08 Thread Michael Krause
At 12:39 PM 2/8/2007, Hal Rosenstock wrote: >On Thu, 2007-02-08 at 14:54, Sean Hefty wrote: > > >Hum, you mean to meet the LID validation rules of 9.6.1.5? That is a > > >huge PITA.. > > > > > >[IMHO, 9.6.1.5 C9-54 is a mistake, if there is a GRH then the LRH.SLID > > > should not be validated agai

Re: [openib-general] Problem is routing CM REQ was: Use a GRH when appropriate for unicast packets

2007-02-08 Thread Hal Rosenstock
On Thu, 2007-02-08 at 14:54, Sean Hefty wrote: > >Hum, you mean to meet the LID validation rules of 9.6.1.5? That is a > >huge PITA.. > > > >[IMHO, 9.6.1.5 C9-54 is a mistake, if there is a GRH then the LRH.SLID > > should not be validated against the QP context since it makes it > > extra hard for

Re: [openib-general] Problem is routing CM REQ was: Use a GRH when appropriate for unicast packets

2007-02-08 Thread Sean Hefty
>Hum, you mean to meet the LID validation rules of 9.6.1.5? That is a >huge PITA.. > >[IMHO, 9.6.1.5 C9-54 is a mistake, if there is a GRH then the LRH.SLID > should not be validated against the QP context since it makes it > extra hard for multipath routing and QoS to work...] Yes - this gets mes

Re: [openib-general] Problem is routing CM REQ was: Use a GRH when appropriate for unicast packets

2007-02-08 Thread Jason Gunthorpe
On Thu, Feb 08, 2007 at 10:23:11AM -0800, Sean Hefty wrote: > >>The active side clearly cannot learn what the SLID of the passive > >>side's router should be. > >> > >>We don't want to have the routers snoop and alter CM GMPs. > >> > >>The passive side cannot use information from the LRH to get the

[openib-general] Problem is routing CM REQ was: Use a GRH when appropriate for unicast packets

2007-02-08 Thread Sean Hefty
>>The active side clearly cannot learn what the SLID of the passive >>side's router should be. >> >>We don't want to have the routers snoop and alter CM GMPs. >> >>The passive side cannot use information from the LRH to get the router >>LID since the LRH may not be reversible. >> >>The only option