No difference, still the same error
[root@master xCAT]# diff IPMI.pm.2.8.4.orig IPMI.pm
689c689
< push
@payload,unpack("C*",hmac_sha1(pack("C*",@{$self->{remoterandomnumber}},@{$self->{sidm}},4,scalar
@user,@user),$self->{password}));
---
> push
@payload,unpack("C*",hmac_sha1(pack("C*",@{$self->{remoterandomnumber}},@{$self->{sidm}},20,scalar
@user,@user),$self->{password}));
703c703
< push @payload,(4,0,0); # request admin
---
> push @payload,(20,0,0); # request admin
843c843
< my $hmacdata =
pack("C*",(@{$self->{sidm}},@{$self->{pendingsessionid}},@{$self->{randomnumber}},@{$self->{remoterandomnumber}},@{$self->{remoteguid}},4,$ulength,@user));
---
> my $hmacdata =
pack("C*",(@{$self->{sidm}},@{$self->{pendingsessionid}},@{$self->{randomnumber}},@{$self->{remoterandomnumber}},@{$self->{remoteguid}},20,$ulength,@user));
852c852
< $self->{sik} =
hmac_sha1(pack("C*",@{$self->{randomnumber}},@{$self->{remoterandomnumber}},4,$ulength,@user),$self->{password});
---
> $self->{sik} =
hmac_sha1(pack("C*",@{$self->{randomnumber}},@{$self->{remoterandomnumber}},20,$ulength,@user),$self->{password});
Lanae Neild
Systems Programmer I
HPC, CCIT, Clemson University
(864)505-4293
lne...@clemson.edu
On Wed, Aug 6, 2014 at 10:29 AM, Jarrod B Johnson <jbjoh...@us.ibm.com>
wrote:
> I checked out the tag.
>
> Are all four of the following changes in place?
>
> $ git diff -U1
> diff --git a/xCAT-server/lib/perl/xCAT/IPMI.pm
> b/xCAT-server/lib/perl/xCAT/IPMI.pm
> index 3afb337..c7089ee 100644
> --- a/xCAT-server/lib/perl/xCAT/IPMI.pm
> +++ b/xCAT-server/lib/perl/xCAT/IPMI.pm
> @@ -688,3 +688,3 @@ sub send_rakp3 {
>
> my @user = unpack("C*",$self->{userid});
> - push
> @payload,unpack("C*",hmac_sha1(pack("C*",@{$self->{remoterandomnumber}},@{$self->{sidm}},4,scalar
> @user,@user),$self->{password}));
> + push
> @payload,unpack("C*",hmac_sha1(pack("C*",@{$self->{remoterandomnumber}},@{$self->{sidm}},20,scalar
> @user,@user),$self->{password}));
>
> $self->sendpayload(payload=>\@payload,type=>$payload_types{'rakp3'});
> @@ -702,3 +702,3 @@ sub send_rakp1 {
>
> push @payload, @{$self->{randomnumber}};
> - push @payload,(4,0,0); # request admin
> + push @payload,(20,0,0); # request admin
>
> my @user = unpack("C*",$self->{userid});
> @@ -842,3 +842,3 @@ sub got_rakp2 {
>
> my $ulength = scalar @user;
> - my $hmacdata =
> pack("C*",(@{$self->{sidm}},@{$self->{pendingsessionid}},@{$self->{randomnumber}},@{$self->{remoterandomnumber}},@{$self->{remoteguid}},4,$ulength,@user));
> + my $hmacdata =
> pack("C*",(@{$self->{sidm}},@{$self->{pendingsessionid}},@{$self->{randomnumber}},@{$self->{remoterandomnumber}},@{$self->{remoteguid}},20,$ulength,@user));
>
> my @expectedhash =
> (unpack("C*",hmac_sha1($hmacdata,$self->{password})));
> @@ -851,3 +851,3 @@ sub got_rakp2 {
> }
> - $self->{sik} =
> hmac_sha1(pack("C*",@{$self->{randomnumber}},@{$self->{remoterandomnumber}},4,$ulength,@user),$self->{password});
> + $self->{sik} =
> hmac_sha1(pack("C*",@{$self->{randomnumber}},@{$self->{remoterandomnumber}},20,$ulength,@user),$self->{password});
>
> $self->{k1} =
> hmac_sha1(pack("C*",1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1),$self->{sik});
>
> [image: Inactive hide details for Lanae Neild ---08/06/2014 10:18:40
> AM---Thanks Jarrod, yes mine has a 4. I replaced the 4's with 20 i]Lanae
> Neild ---08/06/2014 10:18:40 AM---Thanks Jarrod, yes mine has a 4. I
> replaced the 4's with 20 in IPMI.pm like below, and still getting
>
>
> From: Lanae Neild <lne...@clemson.edu>
> To: xCAT Users Mailing list <xcat-user@lists.sourceforge.net>
> Date: 08/06/2014 10:18 AM
>
> Subject: Re: [xcat-user] xCAT 2.8.4 and HP IPMI
> ------------------------------
>
>
>
> Thanks Jarrod, yes mine has a 4. I replaced the 4's with 20 in IPMI.pm
> like below, and still getting the same problem:
>
> [root@master xCAT]# diff IPMI.pm.2.8.4.orig IPMI.pm
> 689c689
> < push
> @payload,unpack("C*",hmac_sha1(pack("C*",@{$self->{remoterandomnumber}},@{$self->{sidm}},4,scalar
> @user,@user),$self->{password}));
> ---
> > push
> @payload,unpack("C*",hmac_sha1(pack("C*",@{$self->{remoterandomnumber}},@{$self->{sidm}},20,scalar
> @user,@user),$self->{password}));
> 703c703
> < push @payload,(4,0,0); # request admin
> ---
> > push @payload,(20,0,0); # request admin
>
>
>
> [root@master xCAT]# rsetboot node1903 net
> node1903: Error: Unauthorized name
> node1903: Error: Unauthorized name
> node1903: Error: Insufficient resources to create new session (wait for
> existing sessions to timeout)
> node1903: Error: Insufficient resources to create new session (wait for
> existing sessions to timeout)
> node1903: Error: Insufficient resources to create new session (wait for
> existing sessions to timeout)
> ^C[root@master xCAT]#
>
>
> Lanae Neild
> Systems Programmer I
> HPC, CCIT, Clemson University
> (864)505-4293
> *lne...@clemson.edu* <lne...@clemson.edu>
>
>
>
>
>
> On Tue, Aug 5, 2014 at 4:33 PM, Jarrod B Johnson <*jbjoh...@us.ibm.com*
> <jbjoh...@us.ibm.com>> wrote:
>
> hmm, instead of privlevel do you see the number '4'? could replace
> the 4 with 20 and see what happens. If there is a 4 every place you see
> 'privlevel', then making it 20 would be exactly equivalent.
>
> The version I patched was a version that could negotiate down to
> operator if needed. I might should checkout the 2.8.4 tag in the future
> before making patches.
>
> [image: Inactive hide details for Lanae Neild ---08/05/2014 04:26:46
> PM---Thanks Jarrod, and sorry I didn't answer your question: yes I]Lanae
> Neild ---08/05/2014 04:26:46 PM---Thanks Jarrod, and sorry I didn't answer
> your question: yes I'm using only original 2.8.4 files now.
>
>
>
> From: Lanae Neild <*lne...@clemson.edu* <lne...@clemson.edu>>
> To: xCAT Users Mailing list <*xcat-user@lists.sourceforge.net*
> <xcat-user@lists.sourceforge.net>>
> Date: 08/05/2014 04:26 PM
>
> Subject: Re: [xcat-user] xCAT 2.8.4 and HP IPMI
> ------------------------------
>
>
>
> Thanks Jarrod, and sorry I didn't answer your question: yes I'm using
> only original 2.8.4 files now.
>
> What version of IPMI.pm are you modifying here? It doesn't match my
> copy of either 2.7.8 or 2.8.4. Here is my 2.8.4 below. I'm especially not
> sure how to change sub send_rakp1 because the line containing
> "privlevel" doesn't exist in my copy.
>
> To make your changes, should I download some other, specific version
> first, and if so, should I replace all my xcat code with a new version, or
> just IPMI.pm?
>
>
> sub send_rakp3 {
> #TODO: this is the point where OPEN RMCP SESSION REQUEST should
> have retry stopped, not send_rakp
> 1
> my $self = shift;
> $self->{rmcptag}+=1;
> my @payload =
> ($self->{rmcptag},0,0,0,@{$self->{pendingsessionid}});
> my @user = unpack("C*",$self->{userid});
> push
>
> @payload,unpack("C*",hmac_sha1(pack("C*",@{$self->{remoterandomnumber}},@{$self->{sidm}},4,s
> calar @user,@user),$self->{password}));
>
> $self->sendpayload(payload=>\@payload,type=>$payload_types{'rakp3'});
> }
>
> sub send_rakp1 {
> my $self = shift;
> $self->{rmcptag}+=1;
> my @payload =
> ($self->{rmcptag},0,0,0,@{$self->{pendingsessionid}});
> $self->{randomnumber}=[];
> foreach (1..16) {
> my $randomnumber = int(rand(255));
> push @{$self->{randomnumber}},$randomnumber;
> }
> push @payload, @{$self->{randomnumber}};
> push @payload,(4,0,0); # request admin
> my @user = unpack("C*",$self->{userid});
> push @payload,scalar @user;
> push @payload,@user;
> $self->{sessionestablishmentcontext} = STATE_EXPECTINGRAKP2;
>
> $self->sendpayload(payload=>\@payload,type=>$payload_types{'rakp1'});
> }
>
>
> Lanae Neild
> Systems Programmer I
> HPC, CCIT, Clemson University
> *(864)505-4293* <%28864%29505-4293>
> *lne...@clemson.edu* <lne...@clemson.edu>
>
>
>
>
>
> On Tue, Aug 5, 2014 at 1:52 PM, Jarrod B Johnson <*jbjoh...@us.ibm.com*
> <jbjoh...@us.ibm.com>> wrote:
> Ok, so if ipmi2support is true, then the following changes to IPMI.pm
> might be interesting:
> diff --git a/xCAT-server/lib/perl/xCAT/IPMI.pm
> b/xCAT-server/lib/perl/xCAT/IPMI.pm
> index 7d7541b..0ac9b35 100644
> --- a/xCAT-server/lib/perl/xCAT/IPMI.pm
> +++ b/xCAT-server/lib/perl/xCAT/IPMI.pm
> @@ -693,7 +693,7 @@ sub send_rakp3 {
> $self->{rmcptag}+=1;
> my @payload =
> ($self->{rmcptag},0,0,0,@{$self->{pendingsessionid}});
> my @user = unpack("C*",$self->{userid});
> - push
>
> @payload,unpack("C*",hmac_sha1(pack("C*",@{$self->{remoterandomnumber}},@{$self->{sidm}},$self->{privlevel},scalar
> @user,@user),$self->{password}));
> + push
>
> @payload,unpack("C*",hmac_sha1(pack("C*",@{$self->{remoterandomnumber}},@{$self->{sidm}},16
> | $self->{privlevel},scalar @user,@user),$self->{password}));
>
> $self->sendpayload(payload=>\@payload,type=>$payload_types{'rakp3'});
> }
>
> @@ -707,7 +707,7 @@ sub send_rakp1 {
> push @{$self->{randomnumber}},$randomnumber;
> }
> push @payload, @{$self->{randomnumber}};
> - push @payload,($self->{privlevel},0,0); # request priv
> + push @payload,(16 | $self->{privlevel},0,0); # request priv
> my @user = unpack("C*",$self->{userid});
> push @payload,scalar @user;
> push @payload,@user;
> @@ -838,7 +838,7 @@ sub got_rakp2 {
> return;
> }
> if ($rmcp_codes{$byte}) {
> - $self->{onlogon}->("ERROR:
> ".$rmcp_codes{$byte},$self->{onlogon_args}); #TODO: errors
> + $self->{onlogon}->("ERROR:
> t".$rmcp_codes{$byte},$self->{onlogon_args}); #TODO: errors
> } else {
> $self->{onlogon}->("ERROR: $byte code on opening
> RMCP+ session",$self->{onlogon_args}); #TODO: errors
> }
> @@ -856,7 +856,7 @@ sub got_rakp2 {
> #Data now represents authcode.. sha1 only..
> my @user = unpack("C*",$self->{userid});
> my $ulength = scalar @user;
> - my $hmacdata =
>
> pack("C*",(@{$self->{sidm}},@{$self->{pendingsessionid}},@{$self->{randomnumber}},@{$self->{remoterandomnumber}},@{$self->{remoteguid}},$self->{privlevel},$ulength,@user));
> + my $hmacdata =
>
> pack("C*",(@{$self->{sidm}},@{$self->{pendingsessionid}},@{$self->{randomnumber}},@{$self->{remoterandomnumber}},@{$self->{remoteguid}},16
> | $self->{privlevel},$ulength,@user));
> my @expectedhash =
> (unpack("C*",hmac_sha1($hmacdata,$self->{password})));
> foreach (0..(scalar(@expectedhash)-1)) {
> if ($expectedhash[$_] != $data[$_]) {
> @@ -865,7 +865,7 @@ sub got_rakp2 {
> return 9;
> }
> }
> - $self->{sik} =
>
> hmac_sha1(pack("C*",@{$self->{randomnumber}},@{$self->{remoterandomnumber}},$self->{privlevel},$ulength,@user),$self->{password});
> + $self->{sik} =
>
> hmac_sha1(pack("C*",@{$self->{randomnumber}},@{$self->{remoterandomnumber}},16
> | $self->{privlevel},$ulength,@user),$self->{password});
> $self->{k1} =
>
> hmac_sha1(pack("C*",1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1),$self->{sik});
> $self->{k2} =
>
> hmac_sha1(pack("C*",2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2),$self->{sik});
> my @aeskey = unpack("C*",$self->{k2});
>
> Was using a method that was able to cope with some vintages of
> Intel BMC code. I could be convinced that the world has moved on from
> those and default to this strategy. If and when the same conflict
> arises
> in the future, then I'd detect and failover to a compatible
> authentication.
> [image: Inactive hide details for Lanae Neild ---08/05/2014
> 12:17:52 PM---[root@master autobackups]# ipmitool -I lan -U xxxx -P
> xxxx -H]Lanae
> Neild ---08/05/2014 12:17:52 PM---[root@master autobackups]#
> ipmitool -I lan -U xxxx -P xxxx -H node1903-man0 power status
>
>
>
> From: Lanae Neild <*lne...@clemson.edu* <lne...@clemson.edu>>
> To: xCAT Users Mailing list <*xcat-user@lists.sourceforge.net*
> <xcat-user@lists.sourceforge.net>>
> Date: 08/05/2014 12:17 PM
>
> Subject: Re: [xcat-user] xCAT 2.8.4 and HP IPMI
>
> ------------------------------
>
>
>
> [root@master autobackups]# ipmitool -I lan -U xxxx -P xxxx -H
> node1903-man0 power status
> Authentication type NONE not supported
> Authentication type NONE not supported
> Error: Unable to establish LAN session
> Unable to get Chassis Power Status
>
> [root@master autobackups]# ipmitool -I lan -U xxxx -P xxxx -L
> ADMINISTRATOR+ -H node1903-man0 power status
> Authentication type NONE not supported
> Authentication type NONE not supported
> Error: Unable to establish LAN session
> Unable to get Chassis Power Status
>
>
> Lanae Neild
> Systems Programmer I
> HPC, CCIT, Clemson University
> *(864)505-4293* <%28864%29505-4293>
> *lne...@clemson.edu* <lne...@clemson.edu>
>
>
>
>
>
> On Tue, Aug 5, 2014 at 11:53 AM, Jarrod B Johnson <
> *jbjoh...@us.ibm.com* <jbjoh...@us.ibm.com>> wrote:
> Previously, cipher suite 2 would be attempted if Crypt::Rijndael
> were not installed. Now it always requires Rijndael...
>
> If it is -I lan instead of lanplus does it work?
>
> If you do '-L ADMINISTRATOR+' in ipmitool, does it affect
> ipmitool?
>
> [image: Inactive hide details for Lanae Neild ---08/05/2014
> 11:49:49 AM---So it was not locked down before? Our 'ipmitool lan
> print 2']Lanae
> Neild ---08/05/2014 11:49:49 AM---So it was not locked down before?
> Our
> 'ipmitool lan print 2' output has not changed so I think we're
>
>
>
> From: Lanae Neild <*lne...@clemson.edu* <lne...@clemson.edu>>
> To: xCAT Users Mailing list <*xcat-user@lists.sourceforge.net*
> <xcat-user@lists.sourceforge.net>>
> Date: 08/05/2014 11:49 AM
>
> Subject: Re: [xcat-user] xCAT 2.8.4 and HP IPMI
>
> ------------------------------
>
>
>
> So it was not locked down before? Our 'ipmitool lan print 2'
> output has not changed so I think we're on to something...
>
> Yes this works now that we fixed our user/password:
>
> (xxxx = our custom user/passwd)
>
> [root@master xCAT]# ipmitool -I lanplus -U xxxx -P xxxx -H
> node1903-man0 power status
>
> Chassis Power is on
>
>
> [root@node1903 tmp]# ipmitool user list 2
> ID Name Callin Link Auth IPMI Msg Channel Priv Limit
> 1 Administrator true false true ADMINISTRATOR
> 2 xxxx true false true ADMINISTRATOR
> 3 (Empty User) true false false NO ACCESS
> 4 (Empty User) true false false NO ACCESS
> 5 (Empty User) true false false NO ACCESS
> 6 (Empty User) true false false NO ACCESS
> 7 (Empty User) true false false NO ACCESS
> 8 (Empty User) true false false NO ACCESS
> 9 (Empty User) true false false NO ACCESS
> 10 (Empty User) true false false NO ACCESS
> 11 (Empty User) true false false NO ACCESS
> 12 (Empty User) true false false NO ACCESS
>
> [root@node1903 tmp]# ipmitool lan print 2
> Set in Progress : Set Complete
> Auth Type Support :
> IP Address Source : Static Address
> IP Address : 10.125.58.99
> Subnet Mask : 255.255.0.0
> MAC Address : 28:92:4a:38:d7:e8
> BMC ARP Control : ARP Responses Enabled, Gratuitous ARP
> Disabled
> Default Gateway IP : 10.125.100.1
> 802.1q VLAN ID : Disabled
> 802.1q VLAN Priority : 0
> Cipher Suite Priv Max : Not Available
>
>
> I'm looking into trying to enter settings.
>
> Lanae Neild
> Systems Programmer I
> HPC, CCIT, Clemson University
> *(864)505-4293* <%28864%29505-4293>
> *lne...@clemson.edu* <lne...@clemson.edu>
>
>
>
>
>
> On Tue, Aug 5, 2014 at 11:09 AM, Jarrod B Johnson <
> *jbjoh...@us.ibm.com* <jbjoh...@us.ibm.com>> wrote:
>
>
>
> Did you restore ipmi2support after modifying IPMI.pm to not
> have it?
>
> Does ipmitool -I lan -U xxx -P xxx -H node11903-man0 power
> status work?
>
> The 2.8.4 change should lock things down to 'cipher suite 3',
> which is the same as ipmitool -I lanplus in behavior.
>
> ipmitool user list 2 and ipmitool lan print 2 might be
> interesting...
>
>
> [image: Inactive hide details for Lanae Neild ---08/05/2014
> 09:39:32 AM---I have some not such good news...we found that
> ipmitool didn']Lanae
> Neild ---08/05/2014 09:39:32 AM---I have some not such good
> news...we found
> that ipmitool didn't connect remotely because another admi
>
>
>
> From: Lanae Neild <*lne...@clemson.edu* <lne...@clemson.edu>>
> To: xCAT Users Mailing list <*xcat-user@lists.sourceforge.net*
> <xcat-user@lists.sourceforge.net>>
> Date: 08/05/2014 09:39 AM
>
> Subject: Re: [xcat-user] xCAT 2.8.4 and HP IPMI
>
> ------------------------------
>
>
>
> I have some not such good news...we found that ipmitool
> didn't connect remotely because another admin had changed the
> user/password
> by accident. Apparently it affected ipmitool but not the web ILO.
> Now that
> I've fixed the user/password, ipmitool works:
>
> [root@master xCAT]# ipmitool -I lanplus -U xxxx -P xxxx -H
> node11903-man0 power status
> Chassis Power is on
>
> Now I don't have anything to take up with HP :( Even after
> ipmitool works remotely now, that still didn't fix the xcat/ipmi
> error.
>
> My only lead now is the cipher change David mentioned earlier
> in this thread. So there is some cipher xcat no longer supports
> as of
> 2.8.4? Can anyone elaborate on this, perhaps its the clue about
> our BMC
> configuration? If I know what cipher needs to be supported
> perhaps I can
> find out what I need from HP.
>
>
> Lanae Neild
> Systems Programmer I
> HPC, CCIT, Clemson University
> *(864)505-4293* <%28864%29505-4293>
> *lne...@clemson.edu* <lne...@clemson.edu>
>
>
>
>
>
> On Fri, Aug 1, 2014 at 11:53 AM, Jarrod B Johnson <
> *jbjoh...@us.ibm.com* <jbjoh...@us.ibm.com>> wrote:
>
>
>
>
> Regretably, I think you have to take up the issue with HP
> support at this point. ipmitool is a pretty widely known tool
> so they
> shouldn't balk at that method of reproducing it like they
> might balk at
> over xCAT.
>
> [image: Inactive hide details for Lanae Neild
> ---08/01/2014 11:31:53 AM---Weird. we have a couple different
> IPMI firmware
> versions in t]Lanae Neild ---08/01/2014 11:31:53
> AM---Weird. we have a couple different IPMI firmware versions
> in the
> sl250's and both behave the same way
>
>
>
> From: Lanae Neild <*lne...@clemson.edu*
> <lne...@clemson.edu>>
> To: xCAT Users Mailing list <
> *xcat-user@lists.sourceforge.net*
> <xcat-user@lists.sourceforge.net>>
> Date: 08/01/2014 11:31 AM
> Subject: Re: [xcat-user] xCAT 2.8.4 and HP IPMI
>
> ------------------------------
>
>
>
> Weird. we have a couple different IPMI firmware versions
> in the sl250's and both behave the same way. What could be
> different in
> xcat 2.7.8 that it would work when this command doesn't?
>
> sl250 from remote:
>
> [root@master xCAT]# ipmitool -I lanplus -U xxxx -P xxxx -H
> node1903-man0 power status
> Error: Unable to establish IPMI v2 / RMCP+ session
> Unable to get Chassis Power Status
>
> dl165 from remote:
>
> [root@master xCAT]# ipmitool -I lanplus -U xxxx -P xxxx -H
> node1572-man0 power status
> Chassis Power is on
>
>
> Lanae Neild
> Systems Programmer I
> HPC, CCIT, Clemson University
> *(864)505-4293* <%28864%29505-4293>
> *lne...@clemson.edu* <lne...@clemson.edu>
>
>
>
>
>
> On Fri, Aug 1, 2014 at 11:05 AM, Jarrod B Johnson <
> *jbjoh...@us.ibm.com* <jbjoh...@us.ibm.com>> wrote:
>
>
>
>
>
> Hmm, that looks bizarre. If you try to do 'ipmitool -I
> lanplus -U <username> -P <password> -H <ilo> power state'
> from a remote
> node, does ipmitool also fail? This might be some oddity
> with your DL160
> systems. Sorry I don't know off hand, I only know our
> system x products
> directly.
>
> [image: Inactive hide details for Lanae Neild
> ---08/01/2014 10:50:56 AM---The channel 1 isn't set up, but
> either
> 'ipmitool lan print' o]Lanae Neild ---08/01/2014
> 10:50:56 AM---The channel 1 isn't set up, but either
> 'ipmitool lan print'
> or 'ipmitool lan print 2' return the sam
>
>
>
> From: Lanae Neild <*lne...@clemson.edu*
> <lne...@clemson.edu>>
> To: xCAT Users Mailing list <
> *xcat-user@lists.sourceforge.net*
> <xcat-user@lists.sourceforge.net>>
> Date: 08/01/2014 10:50 AM
> Subject: Re: [xcat-user] xCAT 2.8.4 and HP IPMI
>
> ------------------------------
>
>
>
>
> The channel 1 isn't set up, but either 'ipmitool lan
> print' or 'ipmitool lan print 2' return the same thing:
>
> [root@node1903 ~]# ipmitool lan print 1
> Channel 1 is not a LAN channel
> [root@node1903 ~]# ipmitool lan print
> Set in Progress : Set Complete
> Auth Type Support :
> IP Address Source : Static Address
> IP Address : 10.125.58.99
> Subnet Mask : 255.255.0.0
> MAC Address : 28:92:4a:38:d7:e8
> BMC ARP Control : ARP Responses Enabled,
> Gratuitous ARP Disabled
> Default Gateway IP : 10.125.100.1
> 802.1q VLAN ID : Disabled
> 802.1q VLAN Priority : 0
> Cipher Suite Priv Max : Not Available
>
> Maybe the channel is important? However, on a HP DL165
> where xcat rinstalls are working fine, its also channel 2,
> so much for
> that. But there's a lot more information from this command
> on a node that's
> working. I would think this might be a clue except that we
> can't find
> anything changed about the sl250s except for xcat being
> upgraded, when the
> issue started. Also, I can still log in to the web mgmt
> interface on the
> sl250s, using the same interface and credentials that xcat
> is configured to
> use for ipmi.
>
> output from a HP DL165 where xcat with ipmi works:
>
> [root@node1572 ~]# ipmitool lan print 2
> Set in Progress : Set Complete
> Auth Type Support : NONE MD5 PASSWORD
> Auth Type Enable : Callback : NONE MD5 PASSWORD
> : User : NONE MD5 PASSWORD
> : Operator : NONE MD5 PASSWORD
> : Admin : NONE MD5 PASSWORD
> : OEM : NONE MD5 PASSWORD
> IP Address Source : Static Address
> IP Address : 10.125.56.136
> Subnet Mask : 255.255.0.0
> MAC Address : 98:4b:e1:60:94:71
> SNMP Community String : public
> IP Header : TTL=0x40 Flags=0x40
> Precedence=0x00 TOS=0x10
> BMC ARP Control : ARP Responses Enabled,
> Gratuitous ARP Disabled
> Gratituous ARP Intrvl : 2.0 seconds
> Default Gateway IP : 10.125.100.1
> 802.1q VLAN ID : Disabled
> 802.1q VLAN Priority : 0
> RMCP+ Cipher Suites : 0,1,2,3
> Cipher Suite Priv Max : OOOOXXXXXXXXXXX
> : X=Cipher Suite Unused
> : c=CALLBACK
> : u=USER
> : o=OPERATOR
> : a=ADMIN
> : O=OEM
>
>
> Lanae Neild
> Systems Programmer I
> HPC, CCIT, Clemson University
> *(864)505-4293* <%28864%29505-4293>
> *lne...@clemson.edu* <lne...@clemson.edu>
>
>
>
>
>
> On Fri, Aug 1, 2014 at 7:05 AM, Jarrod Johnson <
> *jarrod.b.john...@gmail.com*
> <jarrod.b.john...@gmail.com>> wrote:
> How does ipmitool fare? Ipmitool lan print 1 if it
> does work and ipmitool cmd syntax that did work
>
> On Jul 31, 2014 10:42 AM, "Lanae Neild" <
> *lne...@clemson.edu* <lne...@clemson.edu>> wrote:
>
>
>
>
>
>
> Forcing ipmi2support to 0 didn't make any
> difference
>
> [root@master xCAT]# nodeset node1903
> osimage=SL-x86_64-install-compute
> node1903: install SL-x86_64-compute
> [root@master xCAT]# rsetboot node1903 net
> node1903: Error: Invalid role
> node1903: Error: Invalid role
> node1903: Error: Insufficient resources to create
> new session (wait for existing sessions to timeout)
> node1903: Error: Insufficient resources to create
> new session (wait for existing sessions to timeout)
> node1903: Error: Insufficient resources to create
> new session (wait for existing sessions to timeout)
> node1903: Error: Invalid role
>
>
> Lanae Neild
> Systems Programmer I
> HPC, CCIT, Clemson University
> *(864)505-4293* <%28864%29505-4293>
> *lne...@clemson.edu* <lne...@clemson.edu>
>
>
>
>
>
> On Wed, Jul 30, 2014 at 8:09 PM, Jarrod Johnson <
> *jarrod.b.john...@gmail.com*
> <jarrod.b.john...@gmail.com>> wrote:
> Hmmm... If you edit IPMI.pm and force
> ipmi2support to 0 do things change?
>
> On Jul 30, 2014 2:43 PM, "Lanae Neild" <
> *lne...@clemson.edu* <lne...@clemson.edu>>
> wrote:
>
>
>
>
>
>
> Correction - replaced the correct file this
> time, /opt/xcat/lib/perl/xCAT/IPMI.pm with the
> one from 2.7.8 gives this -
> at least it is a different error this time!
>
> [root@master xCAT]# nodeset node1903
> osimage=SL-x86_64-install-compute
> node1903: install SL-x86_64-compute
> [root@master xCAT]# rsetboot node1903 net
> node1903: Error: 9 code on opening RMCP+
> session
> node1903: Error: 1 code on opening RMCP+
> session
> node1903: Error: 1 code on opening RMCP+
> session
> node1903: Error: 1 code on opening RMCP+
> session
> node1903: Error: 9 code on opening RMCP+
> session
> node1903: Error: 9 code on opening RMCP+
> session
> node1903: Error: 1 code on opening RMCP+
> session
> node1903: Error: 1 code on opening RMCP+
> session
>
> Lanae Neild
> Systems Programmer I
> HPC, CCIT, Clemson University
> *(864)505-4293* <%28864%29505-4293>
> *lne...@clemson.edu* <lne...@clemson.edu>
>
>
>
>
>
> On Wed, Jul 30, 2014 at 2:12 PM, Lanae
> Neild <*lne...@clemson.edu*
> <lne...@clemson.edu>> wrote:
> I tried *ipmi.pm* <http://ipmi.pm/> from
> 2.7.8, that's what we had last before
> upgrading to 2.8.4. It seems module
> files have been moved around since then
>
> [root@master xCAT_plugin]# nodeset
> node1903 osimage=SL-x86_64-install-compute
> node1903: install SL-x86_64-compute
> [root@master xCAT_plugin]# rsetboot
> node1903 net
> Error: ipmi plugin bug, pid 14688,
> process description: 'xCATd SSL: rsetboot to
> node1903 for
> root@localhost.localdomain: ipmi
> instance' with error 'Can't locate object
> method "get_ServiceNode" via
> package "xCAT::Utils" at
> /opt/xcat/lib/perl/xCAT_plugin/
> *ipmi.pm* <http://ipmi.pm/> line 5865.
> ' while trying to fulfill request for
> the following nodes: node1903
>
>
> Lanae Neild
> Systems Programmer I
> HPC, CCIT, Clemson University
> *(864)505-4293* <%28864%29505-4293>
> *lne...@clemson.edu* <lne...@clemson.edu>
>
>
>
>
>
> On Wed, Jul 30, 2014 at 8:51 AM, Jarrod
> B Johnson <*jbjoh...@us.ibm.com*
> <jbjoh...@us.ibm.com>> wrote:
> Can you try IPMI.pm from 2.7? If 2.7
> still works, that means the measures to
> try to work even if Administrator
> is forbidden messed something up...
>
>
> >
> > From: Lanae Neild <
> *lne...@clemson.edu*
> <lne...@clemson.edu>>
> > To: xCAT Users Mailing list <
> *xcat-user@lists.sourceforge.net*
> <xcat-user@lists.sourceforge.net>
> >
> > Date: 2014-07-29 11:20
> > Subject: [xcat-user] xCAT
> 2.8.4 and HP IPMI
> >
> >
> >
> > Does anyone else have a similar
> issue or know how to work around it?
> >
> > We have HP SL250s some with their
> IPMI firmware rev 1.20 some with
> > 1.40. Confirmed problem with both.
> They have their ILO settings
> > configured the same way, and xCAT
> rboot and rinstall worked before
> > we upgraded from 2.7x to 2.8.4. Now
> we're getting this error, and
> > have to manually reboot them, as it
> no longer works with xCAT:
> >
> > [root@master ~]# rboot node1903
> > node1903: boot
> > node1903: Error: Invalid role
> > node1903: Error: Invalid role
> > node1903: Error: I nsufficient
> resources to create new session (wait
> > for existing sessions to timeout)
> > node1903: Error: Insufficient
> resources to create new session (wait
> > for existing sessions to timeout)
> > node1903: Error: Insufficient
> resources to create new session (wait
> > for existing sessions to timeout)
> > node1903: Error: Invalid role
> > node1903: Error: Invalid role
> > node1903: Error: Insufficient
> resources to create new session (wait
> > for existing sessions to timeout)
> > node1903: Error: Insufficient
> resources to create new session (wait
> > for existing sessions to timeout)
> > node1903: Error: Invalid role
> > node1903: Error: Invalid role
> > node1903: Error: Insufficient
> resources to create new session (wait
> > for existing sessions to timeout)
> > node1903: Error: Insufficient
> resources to create new session (wait
> > for existing sessions to timeout)
> > node1903: Error: Invalid role
> > node1903: Error: Invalid role
> > node1903: Error: Insufficient
> resources to create new session (wait
> > for existing sessions to timeout)
> > node1903: Error: Insufficient
> resources to create new session (wait
> > for existing sessions to timeout)
> > node1903: Error: Invalid role
> > node1903: Error: Invalid role
> > node1903: Error: Insufficient
> resources to create new session (wait
> > for existing sessions to timeout)
> >
> >
> > Here is the node definition for
> this one:
> >
> > [root@master ~]# lsdef node1903
> > Object name: node1903
> > addkcmdline=vga=0x303
> >
>
> rdblacklist=nouveau,mlx4_ib,mlx4_en,mlx4_core nouveau.modeset=0
> > arch=x86_64
> > bmc=node1903-man0
> > chain=runcmd=bmcsetup,standby
> > cons=ipmi
> > currchain=boot
> > currstate=boot
> >
>
> groups=phase09,all,hp,compute,SL250,gpu,k20
> >
> initrd=xcat/SL/x86_64/initrd.img
> > interface=eth2
> > kcmdline=quiet repo=
>
> *http://10.125.40.6/install/SL/x86_64/*
> <http://10.125.40.6/install/SL/x86_64/>
> ks=
> >
>
> *http://10.125.40.6/install/autoinst/node1903*
>
> <http://10.125.40.6/install/autoinst/node1903> ksdevice=eth2
> cmdline
> > console=tty0 console=ttyS0,19200n8r
> > kernel=xcat/SL/x86_64/vmlinuz
> > mac=2c:44:fd:97:18:88
> > mgt=ipmi
> > mpa=hpsl250chassis33
> > mtm=SL250
> > netboot=pxe
> > nfsdir=/install
> > nfsserver=10.125.40.6
> > ondiscover=nodediscover
> > os=SL
> >
>
> postbootscripts=otherpkgs,palmetto-ipmi,serialconsole
> >
>
> postscripts=syslog,remoteshell,syncfiles,resyslog,palmetto-
> >
>
> mountxcat,palmetto-yumpackages,palmetto-nfsmounts,palmetto-
> >
>
> syncfiles,palmetto-services,palmetto-bios,palmetto-
> >
>
> ethernet,palmetto-umountxcat,palmetto-puppet
> > power=ipmi
> > primarynic=eth2
> > profile=compute
> >
> provmethod=SL-x86_64-install-compute
> > rack=BM-27
> > room=ITC
> > serial=USE341HCLF
> > serialflow=hard
> > serialport=0
> > serialspeed=19200
> > slotid=3
> > status=booted
> > statustime=10-24-2013 20:29:25
> > supportedarchs=x86,x86_64
> > switch=h-itc-bm27-d4810-117
> > switchinterface=0/1
> > switchport=1
> > tftpserver=10.125.40.6
> > unit=8-11
> > xcatmaster=10.125.40.6
> > [root@master ~]#
> >
> > Lanae Neild
> > Systems Programmer I
> > HPC, CCIT, Clemson University
> >
> *(864)505-4293* <%28864%29505-4293>
>
> > *lne...@clemson.edu*
> <lne...@clemson.edu>
> >
> >
> >
>
> ------------------------------------------------------------------------------
> > Infragistics Professional
> > Build stunning WinForms apps today!
> > Reboot your WinForms applications
> with our WinForms controls.
> > Build a bridge from your legacy
> apps to the future.
> >
>
> *http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk*
>
> <http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk>
> >
>
> _______________________________________________
> > xCAT-user mailing list
> > *xCAT-user@lists.sourceforge.net*
> <xCAT-user@lists.sourceforge.net>
> >
>
> *https://lists.sourceforge.net/lists/listinfo/xcat-user*
>
> <https://lists.sourceforge.net/lists/listinfo/xcat-user>
> >
>
> ------------------------------------------------------------------------------
> > Infragistics Professional
> > Build stunning WinForms apps today!
> > Reboot your WinForms applications
> with our WinForms controls.
> > Build a bridge from your legacy
> apps to the future.
> >
>
> *http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk*
>
> <http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk>
> >
>
> _______________________________________________
> > xCAT-user mailing list
> > *xCAT-user@lists.sourceforge.net*
> <xCAT-user@lists.sourceforge.net>
> >
>
> *https://lists.sourceforge.net/lists/listinfo/xcat-user*
>
> <https://lists.sourceforge.net/lists/listinfo/xcat-user>
>
>
>
> ------------------------------------------------------------------------------
> Infragistics Professional
> Build stunning WinForms apps today!
> Reboot your WinForms applications
> with our WinForms controls.
> Build a bridge from your legacy apps
> to the future.
>
>
> *http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk*
>
> <http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk>
>
>
> _______________________________________________
> xCAT-user mailing list
> *xCAT-user@lists.sourceforge.net* <xCAT-user@lists.sourceforge.net>
> *https://lists.sourceforge.net/lists/listinfo/xcat-user*
>
> <https://lists.sourceforge.net/lists/listinfo/xcat-user>
>
>
>
>
>
> ------------------------------------------------------------------------------
> Infragistics Professional
> Build stunning WinForms apps today!
> Reboot your WinForms applications with our
> WinForms controls.
> Build a bridge from your legacy apps to the
> future.
>
>
> *http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk*
>
> <http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk>
>
> _______________________________________________
> xCAT-user mailing list
> *xCAT-user@lists.sourceforge.net* <xCAT-user@lists.sourceforge.net>
> *https://lists.sourceforge.net/lists/listinfo/xcat-user*
>
> <https://lists.sourceforge.net/lists/listinfo/xcat-user>
>
>
>
> ------------------------------------------------------------------------------
> Infragistics Professional
> Build stunning WinForms apps today!
> Reboot your WinForms applications with our
> WinForms controls.
> Build a bridge from your legacy apps to the
> future.
>
>
> *http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk*
>
> <http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk>
> _______________________________________________
> xCAT-user mailing list
> *xCAT-user@lists.sourceforge.net* <xCAT-user@lists.sourceforge.net>
> *https://lists.sourceforge.net/lists/listinfo/xcat-user*
>
> <https://lists.sourceforge.net/lists/listinfo/xcat-user>
>
>
>
>
>
>
>
>
>
>
>
> ------------------------------------------------------------------------------
> Infragistics Professional
> Build stunning WinForms apps today!
> Reboot your WinForms applications with our
> WinForms controls.
> Build a bridge from your legacy apps to the
> future.
>
>
> *http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk*
>
> <http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk>
> _______________________________________________
> xCAT-user mailing list
> *xCAT-user@lists.sourceforge.net* <xCAT-user@lists.sourceforge.net>
> *https://lists.sourceforge.net/lists/listinfo/xcat-user*
>
> <https://lists.sourceforge.net/lists/listinfo/xcat-user>
>
>
>
> ------------------------------------------------------------------------------
> Want fast and easy access to all the code in your
> enterprise? Index and
> search up to 200,000 lines of code with a free copy
> of Black Duck
> Code Sight - the same software that powers the
> world's largest code
> search on Ohloh, the Black Duck Open Hub! Try it now.
> *http://p.sf.net/sfu/bds* <http://p.sf.net/sfu/bds>
> _______________________________________________
> xCAT-user mailing list
> *xCAT-user@lists.sourceforge.net* <xCAT-user@lists.sourceforge.net>
> *https://lists.sourceforge.net/lists/listinfo/xcat-user*
> <https://lists.sourceforge.net/lists/listinfo/xcat-user>
>
>
>
>
>
>
>
>
>
> ------------------------------------------------------------------------------
> Want fast and easy access to all the code in your
> enterprise? Index and
> search up to 200,000 lines of code with a free copy of
> Black Duck
> Code Sight - the same software that powers the world's
> largest code
> search on Ohloh, the Black Duck Open Hub! Try it now.
> *http://p.sf.net/sfu/bds* <http://p.sf.net/sfu/bds>
> _______________________________________________
> xCAT-user mailing list
> *xCAT-user@lists.sourceforge.net* <xCAT-user@lists.sourceforge.net>
> *https://lists.sourceforge.net/lists/listinfo/xcat-user*
> <https://lists.sourceforge.net/lists/listinfo/xcat-user>
>
>
>
>
> ------------------------------------------------------------------------------
> Want fast and easy access to all the code in your
> enterprise? Index and
> search up to 200,000 lines of code with a free copy of
> Black Duck
> Code Sight - the same software that powers the world's
> largest code
> search on Ohloh, the Black Duck Open Hub! Try it now.
> *http://p.sf.net/sfu/bds* <http://p.sf.net/sfu/bds>
> _______________________________________________
> xCAT-user mailing list
> *xCAT-user@lists.sourceforge.net* <xCAT-user@lists.sourceforge.net>
> *https://lists.sourceforge.net/lists/listinfo/xcat-user*
> <https://lists.sourceforge.net/lists/listinfo/xcat-user>
>
>
>
>
>
>
>
>
> ------------------------------------------------------------------------------
> Want fast and easy access to all the code in your
> enterprise? Index and
> search up to 200,000 lines of code with a free copy of
> Black Duck
> Code Sight - the same software that powers the world's
> largest code
> search on Ohloh, the Black Duck Open Hub! Try it now.
> *http://p.sf.net/sfu/bds* <http://p.sf.net/sfu/bds>
> _______________________________________________
> xCAT-user mailing list
> *xCAT-user@lists.sourceforge.net* <xCAT-user@lists.sourceforge.net>
> *https://lists.sourceforge.net/lists/listinfo/xcat-user*
> <https://lists.sourceforge.net/lists/listinfo/xcat-user>
>
>
>
>
> ------------------------------------------------------------------------------
> Want fast and easy access to all the code in your
> enterprise? Index and
> search up to 200,000 lines of code with a free copy of
> Black Duck
> Code Sight - the same software that powers the world's
> largest code
> search on Ohloh, the Black Duck Open Hub! Try it now.
> *http://p.sf.net/sfu/bds* <http://p.sf.net/sfu/bds>
> _______________________________________________
> xCAT-user mailing list
> *xCAT-user@lists.sourceforge.net* <xCAT-user@lists.sourceforge.net>
> *https://lists.sourceforge.net/lists/listinfo/xcat-user*
> <https://lists.sourceforge.net/lists/listinfo/xcat-user>
>
>
>
>
>
>
>
> ------------------------------------------------------------------------------
> Infragistics Professional
> Build stunning WinForms apps today!
> Reboot your WinForms applications with our WinForms controls.
> Build a bridge from your legacy apps to the future.
>
>
> *http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk*
>
> <http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk>
> _______________________________________________
> xCAT-user mailing list
> *xCAT-user@lists.sourceforge.net* <xCAT-user@lists.sourceforge.net>
> *https://lists.sourceforge.net/lists/listinfo/xcat-user*
> <https://lists.sourceforge.net/lists/listinfo/xcat-user>
>
>
>
>
> ------------------------------------------------------------------------------
> Infragistics Professional
> Build stunning WinForms apps today!
> Reboot your WinForms applications with our WinForms controls.
> Build a bridge from your legacy apps to the future.
>
>
> *http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk*
>
> <http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk>
> _______________________________________________
> xCAT-user mailing list
> *xCAT-user@lists.sourceforge.net* <xCAT-user@lists.sourceforge.net>
> *https://lists.sourceforge.net/lists/listinfo/xcat-user*
> <https://lists.sourceforge.net/lists/listinfo/xcat-user>
>
>
>
>
>
>
> ------------------------------------------------------------------------------
> Infragistics Professional
> Build stunning WinForms apps today!
> Reboot your WinForms applications with our WinForms controls.
> Build a bridge from your legacy apps to the future.
>
>
> *http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk*
>
> <http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk>
> _______________________________________________
> xCAT-user mailing list
> *xCAT-user@lists.sourceforge.net* <xCAT-user@lists.sourceforge.net>
> *https://lists.sourceforge.net/lists/listinfo/xcat-user*
> <https://lists.sourceforge.net/lists/listinfo/xcat-user>
>
>
>
>
> ------------------------------------------------------------------------------
> Infragistics Professional
> Build stunning WinForms apps today!
> Reboot your WinForms applications with our WinForms controls.
> Build a bridge from your legacy apps to the future.
>
>
> *http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk*
>
> <http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk>
> _______________________________________________
> xCAT-user mailing list
> *xCAT-user@lists.sourceforge.net* <xCAT-user@lists.sourceforge.net>
> *https://lists.sourceforge.net/lists/listinfo/xcat-user*
> <https://lists.sourceforge.net/lists/listinfo/xcat-user>
>
>
>
>
>
> ------------------------------------------------------------------------------
> Infragistics Professional
> Build stunning WinForms apps today!
> Reboot your WinForms applications with our WinForms controls.
> Build a bridge from your legacy apps to the future.
>
>
> *http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk*
>
> <http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk>
> _______________________________________________
> xCAT-user mailing list
> *xCAT-user@lists.sourceforge.net* <xCAT-user@lists.sourceforge.net>
> *https://lists.sourceforge.net/lists/listinfo/xcat-user*
> <https://lists.sourceforge.net/lists/listinfo/xcat-user>
>
>
>
>
> ------------------------------------------------------------------------------
> Infragistics Professional
> Build stunning WinForms apps today!
> Reboot your WinForms applications with our WinForms controls.
> Build a bridge from your legacy apps to the future.
>
>
> *http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk*
>
> <http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk>
> _______________________________________________
> xCAT-user mailing list
> *xCAT-user@lists.sourceforge.net* <xCAT-user@lists.sourceforge.net>
> *https://lists.sourceforge.net/lists/listinfo/xcat-user*
> <https://lists.sourceforge.net/lists/listinfo/xcat-user>
>
>
>
>
> ------------------------------------------------------------------------------
> Infragistics Professional
> Build stunning WinForms apps today!
> Reboot your WinForms applications with our WinForms controls.
> Build a bridge from your legacy apps to the future.
>
>
> *http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk*
>
> <http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk>
> _______________________________________________
> xCAT-user mailing list
> *xCAT-user@lists.sourceforge.net* <xCAT-user@lists.sourceforge.net>
> *https://lists.sourceforge.net/lists/listinfo/xcat-user*
> <https://lists.sourceforge.net/lists/listinfo/xcat-user>
>
>
>
>
> ------------------------------------------------------------------------------
> Infragistics Professional
> Build stunning WinForms apps today!
> Reboot your WinForms applications with our WinForms controls.
> Build a bridge from your legacy apps to the future.
>
>
> *http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk*
>
> <http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk>
> _______________________________________________
> xCAT-user mailing list
> *xCAT-user@lists.sourceforge.net* <xCAT-user@lists.sourceforge.net>
> *https://lists.sourceforge.net/lists/listinfo/xcat-user*
> <https://lists.sourceforge.net/lists/listinfo/xcat-user>
>
>
>
> ------------------------------------------------------------------------------
> Infragistics Professional
> Build stunning WinForms apps today!
> Reboot your WinForms applications with our WinForms controls.
> Build a bridge from your legacy apps to the future.
>
> http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk
> _______________________________________________
> xCAT-user mailing list
> xCAT-user@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xcat-user
>
>
>
> ------------------------------------------------------------------------------
> Infragistics Professional
> Build stunning WinForms apps today!
> Reboot your WinForms applications with our WinForms controls.
> Build a bridge from your legacy apps to the future.
>
> http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk
> _______________________________________________
> xCAT-user mailing list
> xCAT-user@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xcat-user
>
>
------------------------------------------------------------------------------
Infragistics Professional
Build stunning WinForms apps today!
Reboot your WinForms applications with our WinForms controls.
Build a bridge from your legacy apps to the future.
http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk
_______________________________________________
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user
