Re: [ofa-general] question about partitioning IB networks

2009-09-11 Thread Hal Rosenstock 
On Fri, Sep 11, 2009 at 11:18 AM, Sasha Khapyorsky wrote:

> On 09:17 Thu 03 Sep , Hal Rosenstock wrote:
> > >  Also it says the default partition will be created “unconditionally
> even
> > > when partition configuration file does not exist or cannot be
> accessed.”
> > >  Will it also be created if the partition configuration file exists but
> does
> > > not have a default partition defined?
> > >
> > No.
>
> AFAIR OpenSM will create the default partition even before partitions
> config file parsing, when the file exists it will be:
>
> Default=0x7fff: ALL=limited, SELF=full;
>
> (no IPoIB there). And this can be overwritten by configuration.
>

 Indeed it does (since the default partition is always required). The man
page (and partition doc) should be updated to clarify this.

-- Hal



>
> Sasha
>
___
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [ofa-general] question about partitioning IB networks

2009-09-11 Thread Sasha Khapyorsky 
On 16:43 Thu 03 Sep , Yevgeny Kliteynik wrote:
> > 
> > I've never done it this way but it does look like the partition create 
> > code will detect the duplicated partitions (0x111 and 0x112) and merge 
> > ports from rack2 with rack1 and rack4 with rack3. 
> 
> It will.
> Note that partition names are meaningless in terms of IB management.
> Basically they are used just for logging.

Not only for logging, OpenSM will also try preserve same PKey value for
partition name over reconfiguration, and guess it is all.

Sasha
___
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [ofa-general] question about partitioning IB networks

2009-09-11 Thread Sasha Khapyorsky 
On 09:17 Thu 03 Sep , Hal Rosenstock wrote:
> >  Also it says the default partition will be created “unconditionally even
> > when partition configuration file does not exist or cannot be accessed.”
> >  Will it also be created if the partition configuration file exists but does
> > not have a default partition defined?
> >
> No.

AFAIR OpenSM will create the default partition even before partitions
config file parsing, when the file exists it will be:

Default=0x7fff: ALL=limited, SELF=full;

(no IPoIB there). And this can be overwritten by configuration.

Sasha
___
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [ofa-general] question about partitioning IB networks

2009-09-03 Thread Hal Rosenstock 
Don,

On Thu, Sep 3, 2009 at 1:12 PM, Meyer, Donald J wrote:

>  Hal,
>
>
>
> If you would like to use my example configuration in the man page (the one
> with realistic GUID’s) please feel free to do so.  The GUID’s are all
> imaginary but realistic.
>
>
>
> Are you sure the default partition should be
> “Default=0x7fff,ipoib,rate=7:ALL,SELF=full;” and not
> “Default=0x7fff:SELF=full,ALL=limited;”?
>
   The second version forces both known and unknown CA’s to be unable to
> reach any CA but the sm except via their own partition.
>
I thought that was what you wanted. I thought you only wanted the CAs to be
able to talk with each other on the designated non default partitions.


>It also seems to me that the first version bypasses partitioning by
> allowing CA’s to use the default partition to reach other CA’s not in the
> same partition.
>
You mean the since they are all full members of the default partition they
can talk to each other on that partition despite that not being allowed on
some other partition. If so, yes.

>
>
> Also, if you would like, I would be happy to work on a version of the man
> page where I would try to possibly explain a bit more and have more complete
> examples.
>
Sure; if you want you are welcome to post patches to the list for review,
comment, etc.

-- Hal

>
>
> *Thanks,*
>
> *Don Meyer*
>
> *Senior Network/System Engineer/Programmer*
>
> US+ (253) 371-9532 iNet 8-371-9532
>
> **Other names and brands may be claimed as the property of others*
>  --
>
> *From:* Hal Rosenstock [mailto:[email protected]]
> *Sent:* Thursday, September 03, 2009 6:46 AM
> *To:* [email protected]
> *Cc:* Meyer, Donald J; [email protected]
> *Subject:* Re: [ofa-general] question about partitioning IB networks
>
>
>
>
>
> On Thu, Sep 3, 2009 at 9:43 AM, Yevgeny Kliteynik <
> [email protected]> wrote:
>
> Hal Rosenstock wrote:
>
>
>  On Mon, Aug 31, 2009 at 3:29 PM, Meyer, Donald J <
> [email protected] <mailto:[email protected]>> wrote:
> ...
> ...
>
>
>Just as a sanity check, my interpretation so far is that my network
>should have a partition configuration file similar to the following.
> Can anyone tell me if I have this correct?  In this example
>configuration, I am trying to create two partitions.  One with rack
>one and two, the other with rack three and four:
>
>
>#Default partition (for SM control of the CA’s)
>
>Default=0x7fff,ipoib,rate=7:ALL=limited;
>
> Default=0x7fff,ipoib,rate=7:ALL,SELF=full;
>
>#rack1
>
>rack1=0x111,ipoib,rate=7,defmember=full:;
>
>#rack2
>
>rack2=0x111,ipoib,rate=7,defmember=full:;
>
>#rack3
>
>rack3=0x112,ipoib,rate=7,defmember=full:;
>
>#rack4
>
>rack4=0x112,ipoib,rate=7,defmember=full:;
>
> I've never done it this way but it does look like the partition create code
> will detect the duplicated partitions (0x111 and 0x112) and merge ports from
> rack2 with rack1 and rack4 with rack3.
>
>
> It will.
> Note that partition names are meaningless in terms of IB management.
> Basically they are used just for logging. The only real partition ID
> is its pkey.
>
>
>
> The low 7 bits (without membership bit) of pkey denotes partition.
>
>
>
> -- Hal
>
>
>
>
> -- Yevgeny
>
> -- Hal
>
>
>*Thanks,*
>
>*Don Meyer*
>
>/Senior Network/System Engineer/Programmer/
>
>US+ (253) 371-9532 iNet 8-371-9532
>
>/*Other names and brands may be claimed as the property of others/
>
>
>
>___
>general mailing list
>
>[email protected] <mailto:[email protected]>
>
>
>http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
>To unsubscribe, please visit
>http://openib.org/mailman/listinfo/openib-general
>
>
> 
>
>
>
> ___
> general mailing list
> [email protected]
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
> To unsubscribe, please visit
> http://openib.org/mailman/listinfo/openib-general
>
>
>
>
>
___
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

RE: [ofa-general] question about partitioning IB networks

2009-09-03 Thread Meyer, Donald J 
Hal,

If you would like to use my example configuration in the man page (the one with 
realistic GUID's) please feel free to do so.  The GUID's are all imaginary but 
realistic.

Are you sure the default partition should be 
"Default=0x7fff,ipoib,rate=7:ALL,SELF=full;" and not 
"Default=0x7fff:SELF=full,ALL=limited;"?

The second version forces both known and unknown CA's to be unable to reach any 
CA but the sm except via their own partition.  It also seems to me that the 
first version bypasses partitioning by allowing CA's to use the default 
partition to reach other CA's not in the same partition.

Also, if you would like, I would be happy to work on a version of the man page 
where I would try to possibly explain a bit more and have more complete 
examples.

Thanks,
Don Meyer
Senior Network/System Engineer/Programmer
US+ (253) 371-9532 iNet 8-371-9532
*Other names and brands may be claimed as the property of others

From: Hal Rosenstock [mailto:[email protected]]
Sent: Thursday, September 03, 2009 6:46 AM
To: [email protected]
Cc: Meyer, Donald J; [email protected]
Subject: Re: [ofa-general] question about partitioning IB networks


On Thu, Sep 3, 2009 at 9:43 AM, Yevgeny Kliteynik 
mailto:[email protected]>> wrote:
Hal Rosenstock wrote:

 On Mon, Aug 31, 2009 at 3:29 PM, Meyer, Donald J 
mailto:[email protected]> 
<mailto:[email protected]<mailto:[email protected]>>> wrote:
...
...

   Just as a sanity check, my interpretation so far is that my network
   should have a partition configuration file similar to the following.
Can anyone tell me if I have this correct?  In this example
   configuration, I am trying to create two partitions.  One with rack
   one and two, the other with rack three and four:


   #Default partition (for SM control of the CA's)

   Default=0x7fff,ipoib,rate=7:ALL=limited;

Default=0x7fff,ipoib,rate=7:ALL,SELF=full;

   #rack1

   rack1=0x111,ipoib,rate=7,defmember=full:;

   #rack2

   rack2=0x111,ipoib,rate=7,defmember=full:;

   #rack3

   rack3=0x112,ipoib,rate=7,defmember=full:;

   #rack4

   rack4=0x112,ipoib,rate=7,defmember=full:;

I've never done it this way but it does look like the partition create code 
will detect the duplicated partitions (0x111 and 0x112) and merge ports from 
rack2 with rack1 and rack4 with rack3.

It will.
Note that partition names are meaningless in terms of IB management.
Basically they are used just for logging. The only real partition ID
is its pkey.

The low 7 bits (without membership bit) of pkey denotes partition.

-- Hal


-- Yevgeny
-- Hal


   *Thanks,*

   *Don Meyer*

   /Senior Network/System Engineer/Programmer/

   US+ (253) 371-9532 iNet 8-371-9532

   /*Other names and brands may be claimed as the property of others/



   ___
   general mailing list
   [email protected]<mailto:[email protected]> 
<mailto:[email protected]<mailto:[email protected]>>

   http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general

   To unsubscribe, please visit
   http://openib.org/mailman/listinfo/openib-general





___
general mailing list
[email protected]<mailto:[email protected]>
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general


___
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [ofa-general] question about partitioning IB networks

2009-09-03 Thread Hal Rosenstock 
On Thu, Sep 3, 2009 at 9:43 AM, Yevgeny Kliteynik <
[email protected]> wrote:

> Hal Rosenstock wrote:
>
>>
>>  On Mon, Aug 31, 2009 at 3:29 PM, Meyer, Donald J <
>> [email protected] > wrote:
>> ...
>> ...
>>Just as a sanity check, my interpretation so far is that my network
>>should have a partition configuration file similar to the following.
>> Can anyone tell me if I have this correct?  In this example
>>configuration, I am trying to create two partitions.  One with rack
>>one and two, the other with rack three and four:
>>
>>
>>#Default partition (for SM control of the CA’s)
>>
>>Default=0x7fff,ipoib,rate=7:ALL=limited;
>>
>> Default=0x7fff,ipoib,rate=7:ALL,SELF=full;
>>
>>#rack1
>>
>>rack1=0x111,ipoib,rate=7,defmember=full:;
>>
>>#rack2
>>
>>rack2=0x111,ipoib,rate=7,defmember=full:;
>>
>>#rack3
>>
>>rack3=0x112,ipoib,rate=7,defmember=full:;
>>
>>#rack4
>>
>>rack4=0x112,ipoib,rate=7,defmember=full:;
>>
>> I've never done it this way but it does look like the partition create
>> code will detect the duplicated partitions (0x111 and 0x112) and merge ports
>> from rack2 with rack1 and rack4 with rack3.
>>
>
> It will.
> Note that partition names are meaningless in terms of IB management.
> Basically they are used just for logging. The only real partition ID
> is its pkey.
>

The low 7 bits (without membership bit) of pkey denotes partition.

-- Hal


>
> -- Yevgeny
>
>  -- Hal
>>
>>
>>*Thanks,*
>>
>>*Don Meyer*
>>
>>/Senior Network/System Engineer/Programmer/
>>
>>US+ (253) 371-9532 iNet 8-371-9532
>>
>>/*Other names and brands may be claimed as the property of others/
>>
>>
>>
>>___
>>general mailing list
>>[email protected] 
>>http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>
>>To unsubscribe, please visit
>>http://openib.org/mailman/listinfo/openib-general
>>
>>
>>
>> 
>>
>> ___
>> general mailing list
>> [email protected]
>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>
>> To unsubscribe, please visit
>> http://openib.org/mailman/listinfo/openib-general
>>
>
>
___
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [ofa-general] question about partitioning IB networks

2009-09-03 Thread Yevgeny Kliteynik 

Hal Rosenstock wrote:


 
On Mon, Aug 31, 2009 at 3:29 PM, Meyer, Donald J 
mailto:[email protected]>> wrote:

...
...
Just as a sanity check, my interpretation so far is that my network
should have a partition configuration file similar to the following.
 Can anyone tell me if I have this correct?  In this example
configuration, I am trying to create two partitions.  One with rack
one and two, the other with rack three and four:

 


#Default partition (for SM control of the CA’s)

Default=0x7fff,ipoib,rate=7:ALL=limited;

Default=0x7fff,ipoib,rate=7:ALL,SELF=full;

#rack1

rack1=0x111,ipoib,rate=7,defmember=full:;

#rack2

rack2=0x111,ipoib,rate=7,defmember=full:;

#rack3

rack3=0x112,ipoib,rate=7,defmember=full:;

#rack4

rack4=0x112,ipoib,rate=7,defmember=full:;

I've never done it this way but it does look like the partition create 
code will detect the duplicated partitions (0x111 and 0x112) and merge 
ports from rack2 with rack1 and rack4 with rack3. 


It will.
Note that partition names are meaningless in terms of IB management.
Basically they are used just for logging. The only real partition ID
is its pkey.

-- Yevgeny


-- Hal
 

 


*Thanks,*

*Don Meyer*

/Senior Network/System Engineer/Programmer/

US+ (253) 371-9532 iNet 8-371-9532

/*Other names and brands may be claimed as the property of others/

 



___
general mailing list
[email protected] 
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general

To unsubscribe, please visit
http://openib.org/mailman/listinfo/openib-general





___
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general


___
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [ofa-general] question about partitioning IB networks

2009-09-03 Thread Hal Rosenstock 
On Mon, Aug 31, 2009 at 3:29 PM, Meyer, Donald J
wrote:

>  I am trying to partition my IB network but I don’t seem to be able to
> understand the opensm man page.
>
>
>
> First it says “The default partition has P_Key value 0x7fff. OpenSM´s port
> will have full membership in default partition. All other end ports will
> have partial membership.” but I don’t see the difference defined between
> full and partial membership anywhere.  Is it possible the reference was to
> full and limited membership instead?
>
Yes, partial == limited. I've just sent a patch to change that word in the
man page and doc.

>  Does this partition have to exist on all CA’s so the SM can “talk”
> them?
>
Yes, this is an IBA requirement.

>  Also it says the default partition will be created “unconditionally even
> when partition configuration file does not exist or cannot be accessed.”
>  Will it also be created if the partition configuration file exists but does
> not have a default partition defined?
>
No.

>   Second, I see where CA’s can be members of multiple partitions (have
> multiple P_keys).  If a CA is in multiple partitions (has multiple P_Keys
> assigned to it), which partition does it “send” on when the CA has packets
> to send if more than one partition can reach the destination CA?
>
That's up to the application/ULP to set the proper PKey index. The
application/ULP needs to ensure the destination is reachable via a common
PKey. It does that via some sort of PathRecord request to the SA.

>  Also do switches (or any non CA’s) have to have P_Keys assigned for any
> reason?
>
Yes, but with OpenSM they do not need configuration. OpenSM detects which
switches are leaf switches with peer CA ports and sets up their partition
tables appropriately.

>
>
> Just as a sanity check, my interpretation so far is that my network should
> have a partition configuration file similar to the following.  Can anyone
> tell me if I have this correct?  In this example configuration, I am trying
> to create two partitions.  One with rack one and two, the other with rack
> three and four:
>
>
>
> #Default partition (for SM control of the CA’s)
>
> Default=0x7fff,ipoib,rate=7:ALL=limited;
>
Default=0x7fff,ipoib,rate=7:ALL,SELF=full;

>   #rack1
>
> rack1=0x111,ipoib,rate=7,defmember=full:;
>
> #rack2
>
> rack2=0x111,ipoib,rate=7,defmember=full:;
>
> #rack3
>
> rack3=0x112,ipoib,rate=7,defmember=full:;
>
> #rack4
>
> rack4=0x112,ipoib,rate=7,defmember=full:;
>
I've never done it this way but it does look like the partition create code
will detect the duplicated partitions (0x111 and 0x112) and merge ports
from rack2 with rack1 and rack4 with rack3.

-- Hal


>
>
> *Thanks,*
>
> *Don Meyer*
>
> *Senior Network/System Engineer/Programmer*
>
> US+ (253) 371-9532 iNet 8-371-9532
>
> **Other names and brands may be claimed as the property of others*
>
>
>
> ___
> general mailing list
> [email protected]
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
> To unsubscribe, please visit
> http://openib.org/mailman/listinfo/openib-general
>
___
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general