RE: [Freeipmi-devel] FW: [bug #24300] ipmiconsole cannot connect to various Intel servers
Hey Andrew, For fi4.log I see a return code of CCh = Invalid data field in Request for the Activate Payload command. Comparing FreeIPMI's code to your code in ipmiutil, I see: if (mfg == VENDOR_INTEL) ibuf[2] |= SOL_BMC_ASSERTS_CTS_MASK_TRUE; else ibuf[2] |= SOL_BMC_ASSERTS_CTS_MASK_FALSE; So I would assume this is yet another intel specific workaround? I can't (code auditting wise) see another difference in the packets. For fi3.log I definitely got a cycle in my state machine. I should (at minimum) monitor and exit after some number of failed attempts for this state- machine cycle. It seems that the get payload activation status request always returns that SOL is not active. But whenever the activate payload command is executed, it responds w/ 80h = payload already active. The current state machine moves back to get payload activation status to (hopefully) verify that the session is now active, so that it can then move on to deactivate payload. Since one of these commands is continuously returning bad data, the cycle results. So I assume there is a bug in the Intel firmware somewhere in here. Perhaps the fix to the activate payload command up above (from fi4.log) would be enough to make the activate payload command work? I'll put a fix in for these and e-mail you a beta link. Thanks for all the help. Al On Mon, 2008-12-08 at 10:35 -0800, Al Chu wrote: Hey Andrew, Cool. These logs show that we're fully authenticating, so the hard Intel IPMI 2.0 workarounds are over. So there is something in the libipmiconsole state machine that it is stuck on. Likely some corner case b/c I do things differently than ipmitool/ipmiutil. I'll take a look into it. Al On Mon, 2008-12-08 at 10:27 -0800, Cress, Andrew R wrote: Al, Oops. I forgot the -W intel20 option. Here are logs with that included, and debug. The problem seems to be with the Trailer now. It flags a BMC Error message to one system and repeats the cycle forever on another system. Both of these can activate SOL sessions via ipmitool and ipmiutil. It may be just one more tweak is needed. Andy -Original Message- From: Al Chu [mailto:[EMAIL PROTECTED] Sent: Monday, December 08, 2008 1:13 PM To: Cress, Andrew R Cc: freeipmi-devel@gnu.org Subject: RE: [Freeipmi-devel] FW: [bug #24300] ipmiconsole cannot connect to various Intel servers Hey Andrew, Hmmm, the BMC Busy error seems justified. telco64: -- telco64: [ 93h] = message_tag[ 8b] telco64: [ 1h] = rmcpplus_status_code[ 8b] 1h = Insufficient resources to create a session So I think this error is legit given the response. Any idea why the board would respond with this type of error code? Could it be to many failed connection attempts? I remember an Intel IPMI 1.5 board I played with would not be very responsive if there were many failed connections in a row. telco64: = telco64: IPMI 2.0 Open Session Request telco64: = snip telco64: IPMI Command Data: telco64: -- telco64: [ C5h] = message_tag[ 8b] telco64: [ 0h] = requested_maximum_privilege_level[ 4b] telco64: = telco64: IPMI 2.0 Open Session Response telco64: = snip telco64: IPMI Command Data: telco64: -- telco64: [ C5h] = message_tag[ 8b] telco64: [ 0h] = rmcpplus_status_code[ 8b] telco64: [ 0h] = maximum_privilege_level[ 4b] (ipmiconsole_checks.c, ipmiconsole_check_open_session_response_privilege, 610): hostname=telco64; protocol_state=0x2: open session response privilege check failed; p = 3^M [error received]: privilege level cannot be obtained for this user I'll have to look into this. 0h = IPMI_PRIVILEGE_LEVEL_HIGHEST_LEVEL, which should not be done for the Intel boards when the intel workaround is specified. /* IPMI Workaround * * Intel IPMI 2.0 implementations don't support the highest level privilege. */ if (c-config.workaround_flags IPMICONSOLE_WORKAROUND_INTEL_2_0_SESSION) privilege_level = c-config.privilege_level; else privilege_level = IPMI_PRIVILEGE_LEVEL_HIGHEST_LEVEL; Al On Mon, 2008-12-08 at 06:38 -0800, Cress, Andrew R wrote: Al, Attached are the debug logs of the two errors I get. Same errors for any Intel server, the BMC Busy error in fi.log is the most common. Andy -Original Message- From: Al Chu [mailto:[EMAIL PROTECTED] Sent: Friday, December 05, 2008 5:41 PM To: Cress, Andrew R Subject: RE: [Freeipmi-devel] FW: [bug #24300] ipmiconsole
RE: [Freeipmi-devel] FW: [bug #24300] ipmiconsole cannot connect to various Intel servers
Al, For the CTS_MASK, I'm not where I can pull this up, but I thought that this option was something that could be detected. IIRC, I think all BMCs will work with this FALSE, but only some support it with TRUE. For the status loop, I'd first assume a software bug, before pointing to the time-tested and fully validated firmware. The point of the get activation payload status is to prevent multiple activate payload commands from being issued from one client. Andy -Original Message- From: Al Chu [mailto:[EMAIL PROTECTED] Sent: Monday, December 08, 2008 2:34 PM To: Cress, Andrew R Cc: freeipmi-devel@gnu.org Subject: RE: [Freeipmi-devel] FW: [bug #24300] ipmiconsole cannot connect to various Intel servers Hey Andrew, For fi4.log I see a return code of CCh = Invalid data field in Request for the Activate Payload command. Comparing FreeIPMI's code to your code in ipmiutil, I see: if (mfg == VENDOR_INTEL) ibuf[2] |= SOL_BMC_ASSERTS_CTS_MASK_TRUE; else ibuf[2] |= SOL_BMC_ASSERTS_CTS_MASK_FALSE; So I would assume this is yet another intel specific workaround? I can't (code auditting wise) see another difference in the packets. For fi3.log I definitely got a cycle in my state machine. I should (at minimum) monitor and exit after some number of failed attempts for this state- machine cycle. It seems that the get payload activation status request always returns that SOL is not active. But whenever the activate payload command is executed, it responds w/ 80h = payload already active. The current state machine moves back to get payload activation status to (hopefully) verify that the session is now active, so that it can then move on to deactivate payload. Since one of these commands is continuously returning bad data, the cycle results. So I assume there is a bug in the Intel firmware somewhere in here. Perhaps the fix to the activate payload command up above (from fi4.log) would be enough to make the activate payload command work? I'll put a fix in for these and e-mail you a beta link. Thanks for all the help. Al On Mon, 2008-12-08 at 10:35 -0800, Al Chu wrote: Hey Andrew, Cool. These logs show that we're fully authenticating, so the hard Intel IPMI 2.0 workarounds are over. So there is something in the libipmiconsole state machine that it is stuck on. Likely some corner case b/c I do things differently than ipmitool/ipmiutil. I'll take a look into it. Al On Mon, 2008-12-08 at 10:27 -0800, Cress, Andrew R wrote: Al, Oops. I forgot the -W intel20 option. Here are logs with that included, and debug. The problem seems to be with the Trailer now. It flags a BMC Error message to one system and repeats the cycle forever on another system. Both of these can activate SOL sessions via ipmitool and ipmiutil. It may be just one more tweak is needed. Andy -Original Message- From: Al Chu [mailto:[EMAIL PROTECTED] Sent: Monday, December 08, 2008 1:13 PM To: Cress, Andrew R Cc: freeipmi-devel@gnu.org Subject: RE: [Freeipmi-devel] FW: [bug #24300] ipmiconsole cannot connect to various Intel servers Hey Andrew, Hmmm, the BMC Busy error seems justified. telco64: -- telco64: [ 93h] = message_tag[ 8b] telco64: [ 1h] = rmcpplus_status_code[ 8b] 1h = Insufficient resources to create a session So I think this error is legit given the response. Any idea why the board would respond with this type of error code? Could it be to many failed connection attempts? I remember an Intel IPMI 1.5 board I played with would not be very responsive if there were many failed connections in a row. telco64: = telco64: IPMI 2.0 Open Session Request telco64: = snip telco64: IPMI Command Data: telco64: -- telco64: [ C5h] = message_tag[ 8b] telco64: [ 0h] = requested_maximum_privilege_level[ 4b] telco64: = telco64: IPMI 2.0 Open Session Response telco64: = snip telco64: IPMI Command Data: telco64: -- telco64: [ C5h] = message_tag[ 8b] telco64: [ 0h] = rmcpplus_status_code[ 8b] telco64: [ 0h] = maximum_privilege_level[ 4b] (ipmiconsole_checks.c, ipmiconsole_check_open_session_response_privilege, 610): hostname=telco64; protocol_state=0x2: open session response privilege check failed; p = 3^M [error received]: privilege level cannot be obtained for this user I'll have to look into this. 0h = IPMI_PRIVILEGE_LEVEL_HIGHEST_LEVEL, which should not be done for the Intel boards when the intel workaround is specified. /* IPMI Workaround * * Intel IPMI 2.0 implementations don't
[Freeipmi-devel] FW: [bug #24300] ipmiconsole cannot connect to various Intel servers
Retry, first send failed. -Original Message- From: Cress, Andrew R [mailto:[EMAIL PROTECTED] Sent: Friday, December 05, 2008 8:43 AM To: Albert Chu; Bryan Henderson; Andy Cress; freeipmi-devel@gnu.org Subject: RE: [bug #24300] ipmiconsole cannot connect to various Intel servers Al, This was the first time I had tried ipmiconsole, so I don't know if it worked before or what changed. For an example of what is different with Intel boards, you can view the source to ipmitool or ipmiutil under the 'lanplus' protocol. It boils down to some different assumptions about defaults or special conditions. In ipmitool, the syntax requires specifying -o intelplus, but ipmiutil detects the manufacturer/product id first and doesn't need those options. From ipmiutil: lib/lanplus/lanplus.c:is_sol_partial_ack() has an intelplus special case, which probably should apply to all other boards too (?) lib/lanplus/lanplus.c:ipmi_lanplus_open_session() has an intelplus condition for privilege defaults lib/lanplus/lanplus_crypt.c:lanplus_rakp4_hmac_matches() has two intelplus cases lib/lanplus/lanplus_crypt.c:lanplus_generate_rakp3_authcode() has an intelplus case for privilege defaults That's all that is different. Andy -Original Message- From: Albert Chu [mailto:[EMAIL PROTECTED] Sent: Thursday, December 04, 2008 5:47 PM To: Albert Chu; Bryan Henderson; Andy Cress; freeipmi-devel@gnu.org Subject: [bug #24300] ipmiconsole cannot connect to various Intel servers Follow-up Comment #2, bug #24300 (project freeipmi): Sorry I didn't see these posts earlier. Hopefully I've fixed the config on Savannah so that bugs actually send out e-mails to the mailing list. I implemented the Intel workarounds a long time ago, but no longer have an Intel motherboard. So I've been forward porting the patches since then and praying they still work and I didn't mess anything up along the way. I guess something is messed up or there is something new to workaround. Hopefully I can find an Intel mobo to try and fix this on. I'm going through the code right now visually and can't see a workaround issue. Al P.S. Bryan, I can see how the wording of the manpage was misinterpreted to make you think I note the manual mentions this can happen with --workaround=intel20, but it doesn't mention anything to do about it. . I'm going to fix up the manpage to instead say: There are a number of Intel IPMI 2.0 bugs. These problems may cause username invalid, password invalid, or k_g invalid errors to occur. They can be worked around by specifying the intel20 workaround. ___ Reply to this item at: http://savannah.gnu.org/bugs/?24300 ___ Message sent via/by Savannah http://savannah.gnu.org/ ___ Freeipmi-devel mailing list Freeipmi-devel@gnu.org http://lists.gnu.org/mailman/listinfo/freeipmi-devel
Re: [Freeipmi-devel] FW: [bug #24300] ipmiconsole cannot connect to various Intel servers
Hey Andrew,
On Fri, 2008-12-05 at 09:14 -0800, Cress, Andrew R wrote:
Retry, first send failed.
-Original Message-
From: Cress, Andrew R [mailto:[EMAIL PROTECTED]
Sent: Friday, December 05, 2008 8:43 AM
To: Albert Chu; Bryan Henderson; Andy Cress; freeipmi-devel@gnu.org
Subject: RE: [bug #24300] ipmiconsole cannot connect to various Intel servers
Al,
This was the first time I had tried ipmiconsole, so I don't know if it worked
before or what changed.
For an example of what is different with Intel boards, you can view the
source to ipmitool or ipmiutil under the 'lanplus' protocol. It boils down
to some different assumptions about defaults or special conditions.
In ipmitool, the syntax requires specifying -o intelplus, but ipmiutil
detects the manufacturer/product id first and doesn't need those options.
From ipmiutil:
lib/lanplus/lanplus.c:is_sol_partial_ack() has an intelplus special case,
which probably should apply to all other boards too (?)
Yeah, I'm not quite sure why this would be an intel corner case. Does
intel return 0 for the accepted_character_count even if it accepted all
the data? So a return of 0 is just assumed to be equivalent to all
data accepted?? I don't currently handle this case in ipmiconsole.
The assumption is if you didn't accept any data, then you resend data
just as if there was a partial acceptance.
lib/lanplus/lanplus.c:ipmi_lanplus_open_session() has an intelplus condition
for privilege defaults
I handle this one:
/* IPMI Workaround
*
* Intel IPMI 2.0 implementations don't support the highest level
privilege.
*/
if (c-config.workaround_flags IPMICONSOLE_WORKAROUND_INTEL_2_0_SESSION)
privilege_level = c-config.privilege_level;
else
privilege_level = IPMI_PRIVILEGE_LEVEL_HIGHEST_LEVEL;
lib/lanplus/lanplus_crypt.c:lanplus_rakp4_hmac_matches() has two intelplus
cases
handle this one:
/* IPMI Workaround
*
* Intel IPMI 2.0 implementations respond with the integrity check
* value based on the integrity algorithm rather than the
* authentication algorithm.
*/
if (c-config.workaround_flags IPMICONSOLE_WORKAROUND_INTEL_2_0_SESSION)
{
if (c-config.integrity_algorithm == IPMI_INTEGRITY_ALGORITHM_NONE)
authentication_algorithm = IPMI_AUTHENTICATION_ALGORITHM_RAKP_NONE;
else if (c-config.integrity_algorithm ==
IPMI_INTEGRITY_ALGORITHM_HMAC_SHA1_96)
authentication_algorithm = IPMI_AUTHENTICATION_ALGORITHM_RAKP_HMAC_SHA1;
else if (c-config.integrity_algorithm ==
IPMI_INTEGRITY_ALGORITHM_HMAC_MD5_128)
authentication_algorithm = IPMI_AUTHENTICATION_ALGORITHM_RAKP_HMAC_MD5;
else if (c-config.integrity_algorithm ==
IPMI_INTEGRITY_ALGORITHM_MD5_128)
/* achu: I have not been able to reverse engineer this. So accept it */
return 1;
}
else
authentication_algorithm = c-config.authentication_algorithm;
lib/lanplus/lanplus_crypt.c:lanplus_generate_rakp3_authcode() has an
intelplus case for privilege defaults
Ahh, this might be it. In ipmipower and libfreeipmi's ipmi 2.0 code I
handle this properly. But in ipmiconsole I seem to have accidentally
put this code in the RAKP1 section. That might be the reason that I'm
checking the return value from the RAKP2 incorrectly.
I'll do some more auditing to see if I can find why the other fellow's
example isn't working on the INTELs. I noticed he is using a NULL
username/password. You mind given my next tar.gz a test run to see if I
caught everything?
BTW, it seems as though ipmiutil does not implement all of the Intel
workarounds I found. There were a number of corner cases for
username/password lengths. Here's what I have in my comments.
/* IPMI Workaround (achu)
*
* Discovered on SE7520AF2 with Intel Server Management Module
* (Professional Edition)
*
* The username must be padded despite explicitly not being
* allowed. No Null characters (00h) are allowed in the name.
* Table 13-11 in the IPMI 2.0 spec.
*/
/* IPMI Workaround (achu)
*
* Discovered on SE7520AF2 with Intel Server Management Module
* (Professional Edition)
*
* When the authentication algorithm is HMAC-MD5-128 and the
* password is greater than 16 bytes, the Intel BMC truncates the
* password to 16 bytes when generating keys, hashes, etc. So we
* have to do the same when generating keys, hashes, etc.
*/
Does ipmiutil handle these too? Or is it possible Intel fixed some
issues but not others in newer firmware?
Al
That's all that is different.
Andy
-Original Message-
From: Albert Chu [mailto:[EMAIL PROTECTED]
Sent: Thursday, December 04, 2008 5:47 PM
To: Albert Chu; Bryan Henderson; Andy Cress; freeipmi-devel@gnu.org
Subject: [bug #24300] ipmiconsole cannot connect to various Intel servers
Follow-up
RE: [Freeipmi-devel] FW: [bug #24300] ipmiconsole cannot connect to various Intel servers
Al,
/* IPMI Workaround (achu)
*
* Discovered on SE7520AF2 with Intel Server Management Module
* (Professional Edition)
*
* The username must be padded despite explicitly not being
* allowed. No Null characters (00h) are allowed in the name.
* Table 13-11 in the IPMI 2.0 spec.
*/
Hmmm. This is because when the set username command is issued, the command
data is padded with nulls to 16 bytes, as shown in secion 22.28, and that is
common across most of the spec. But Table 13-11 is an exception to the rest of
IPMI username handling, in that it is not a fixed-length 16-byte entity. This
is implemented the same in most Intel BMCs that are currently available. In
ipmiutil, this RAKP1 stuff is handled the same way that ipmitool does, by ...
/* IPMI Workaround (achu)
*
* Discovered on SE7520AF2 with Intel Server Management Module
* (Professional Edition)
*
* When the authentication algorithm is HMAC-MD5-128 and the
* password is greater than 16 bytes, the Intel BMC truncates the
* password to 16 bytes when generating keys, hashes, etc. So we
* have to do the same when generating keys, hashes, etc.
*/
I doubt that the IMM firmware supported 20-byte passwords (the first Intel BMC
with IPMI 2.0), but some other Intel BMCs do, I believe. Ipmiutil currently
does not handle 20-byte passwords, and that could be done conditionally on
certain systems, but if we are talking about managing passwords for hundreds of
servers where some support 16 bytes and some support 20 bytes, all of the
passwords would be restricted to 16-bytes anyway, and the IPMI 2.0 spec
requires support/compatibility for 16-byte passwords for this reason. We don't
have any customers who have a use-case where 20-byte passwords would be used,
so I haven't implemented the logic for it.
-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Al Chu
Sent: Friday, December 05, 2008 1:27 PM
To: Cress, Andrew R
Cc: freeipmi-devel@gnu.org
Subject: Re: [Freeipmi-devel] FW: [bug #24300] ipmiconsole cannot connect to
various Intel servers
Hey Andrew,
On Fri, 2008-12-05 at 09:14 -0800, Cress, Andrew R wrote:
Retry, first send failed.
-Original Message-
From: Cress, Andrew R [mailto:[EMAIL PROTECTED]
Sent: Friday, December 05, 2008 8:43 AM
To: Albert Chu; Bryan Henderson; Andy Cress; freeipmi-devel@gnu.org
Subject: RE: [bug #24300] ipmiconsole cannot connect to various Intel servers
Al,
This was the first time I had tried ipmiconsole, so I don't know if it worked
before or what changed.
For an example of what is different with Intel boards, you can view the
source to ipmitool or ipmiutil under the 'lanplus' protocol. It boils down
to some different assumptions about defaults or special conditions.
In ipmitool, the syntax requires specifying -o intelplus, but ipmiutil
detects the manufacturer/product id first and doesn't need those options.
From ipmiutil:
lib/lanplus/lanplus.c:is_sol_partial_ack() has an intelplus special case,
which probably should apply to all other boards too (?)
Yeah, I'm not quite sure why this would be an intel corner case. Does
intel return 0 for the accepted_character_count even if it accepted all
the data? So a return of 0 is just assumed to be equivalent to all
data accepted?? I don't currently handle this case in ipmiconsole.
The assumption is if you didn't accept any data, then you resend data
just as if there was a partial acceptance.
lib/lanplus/lanplus.c:ipmi_lanplus_open_session() has an intelplus condition
for privilege defaults
I handle this one:
/* IPMI Workaround
*
* Intel IPMI 2.0 implementations don't support the highest level
privilege.
*/
if (c-config.workaround_flags IPMICONSOLE_WORKAROUND_INTEL_2_0_SESSION)
privilege_level = c-config.privilege_level;
else
privilege_level = IPMI_PRIVILEGE_LEVEL_HIGHEST_LEVEL;
lib/lanplus/lanplus_crypt.c:lanplus_rakp4_hmac_matches() has two intelplus
cases
handle this one:
/* IPMI Workaround
*
* Intel IPMI 2.0 implementations respond with the integrity check
* value based on the integrity algorithm rather than the
* authentication algorithm.
*/
if (c-config.workaround_flags IPMICONSOLE_WORKAROUND_INTEL_2_0_SESSION)
{
if (c-config.integrity_algorithm == IPMI_INTEGRITY_ALGORITHM_NONE)
authentication_algorithm = IPMI_AUTHENTICATION_ALGORITHM_RAKP_NONE;
else if (c-config.integrity_algorithm ==
IPMI_INTEGRITY_ALGORITHM_HMAC_SHA1_96)
authentication_algorithm = IPMI_AUTHENTICATION_ALGORITHM_RAKP_HMAC_SHA1;
else if (c-config.integrity_algorithm ==
IPMI_INTEGRITY_ALGORITHM_HMAC_MD5_128)
authentication_algorithm = IPMI_AUTHENTICATION_ALGORITHM_RAKP_HMAC_MD5;
else if (c-config.integrity_algorithm
