Corey, I'll have to run the test again overnight to get it to occur again.
1) I have been able to unload the ipmi_msghandler module after this happens. 2) For /proc/ipmi/0/si_stats and /proc/ipmi/0/stats: I need to get this once it occurs again. 3) For newest driver: I could try this in the next pass after (2) I have the ipmi driver source from RHEL5.3 that I'll send you offline for comparison. RE: off-by-one Yes, the replies are coming back, but seem to match up to the request from the other session, when this occurs. Andy -----Original Message----- From: Corey Minyard [mailto:[email protected]] Sent: Monday, July 27, 2009 12:21 PM To: Andy Cress Cc: Mathieu Alorent; [email protected]; Gemini TRAN Subject: Re: [Openipmi-developer] BMC returned incorrect response Also, if possible, can you unload the ipmi_msghandler module? That will tell if any messages were leaked, which would be another clue. Also, is it possible to try the newest version of the driver? I'm not sure how easy that is, though. I don't have ready access to RHEL5.3. The all_cmd_rcvr() thing is really strange. Looking at that, it makes no sense. Also, Andy, the one you have below seems to be a "off by one" problem, if you look the expected value is always in the next values actual received. This is not the case with the other errors, from what I can tell. -corey Andy Cress wrote: > Corey, > > I've seen the same thing on the RHEL5.3 v39.1 driver with Intel BMCs. > Below is a long sequence of mismatched request response pairs. > To reproduce this we had a script running doing 'ipmitool sensor' every 15 > seconds, with another script doing a few other IPMI commands in a loop > without any delays, so I think that it must be that the responses can get > confused when two or more simultaneous local sessions are in play. It occurs > usually after 4-8 hours of testing like that. > > One other clue is that if the ipmi_msghandler.c has the all_cmd_rcvr() > routine included from the ipmi_emu patch, these errors do not occur. Note > that both scripts still use /dev/ipmi0 and /dev/imb emulation is not being > used in this test case. > This leads me to believe that this can be resolved in the ipmi_msghandler.c. > > Andy > > Jul 21 13:00:37 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 2d cmd 0, got netfn 5 cmd 12 > Jul 21 13:00:37 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 5 cmd 12, got netfn 2d cmd 0 > Jul 21 13:00:37 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 7 cmd 1, got netfn 5 cmd 12 > Jul 21 13:00:37 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 5 cmd 12, got netfn 7 cmd 1 > Jul 21 13:00:37 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 7 cmd 35, got netfn 5 cmd 12 > Jul 21 13:00:37 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 5 cmd 12, got netfn 7 cmd 35 > Jul 21 13:00:37 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn d cmd 2, got netfn 5 cmd 12 > Jul 21 13:01:47 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 7 cmd 35, got netfn 5 cmd 12 > Jul 21 13:01:47 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 5 cmd 12, got netfn 7 cmd 35 > Jul 21 13:01:47 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn d cmd 2, got netfn 5 cmd 12 > Jul 21 13:02:56 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn b cmd 23, got netfn 5 cmd 12 > Jul 21 13:02:56 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 5 cmd 12, got netfn b cmd 23 > Jul 21 13:02:56 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn b cmd 23, got netfn 5 cmd 12 > Jul 21 13:02:56 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 5 cmd 12, got netfn b cmd 23 > Jul 21 13:02:56 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn b cmd 23, got netfn 5 cmd 12 > Jul 21 13:02:56 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 5 cmd 12, got netfn b cmd 23 > Jul 21 13:02:56 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn b cmd 23, got netfn 5 cmd 12 > Jul 21 13:02:56 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 5 cmd 12, got netfn b cmd 23 > Jul 21 13:02:56 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn b cmd 23, got netfn 5 cmd 12 > Jul 21 13:02:56 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 5 cmd 12, got netfn b cmd 23 > Jul 21 13:02:56 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn b cmd 23, got netfn 5 cmd 12 > Jul 21 13:02:56 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 5 cmd 12, got netfn b cmd 23 > Jul 21 13:02:56 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 7 cmd 35, got netfn 5 cmd 12 > Jul 21 13:02:56 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 5 cmd 12, got netfn 7 cmd 35 > Jul 21 13:02:56 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn d cmd 2, got netfn 5 cmd 12 > Jul 21 13:04:03 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 7 cmd 35, got netfn 5 cmd 12 > Jul 21 13:04:03 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 5 cmd 12, got netfn 7 cmd 35 > Jul 21 13:04:03 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn d cmd 2, got netfn 5 cmd 12 > Jul 21 13:05:13 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn b cmd 23, got netfn 5 cmd 12 > Jul 21 13:05:13 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 5 cmd 12, got netfn b cmd 23 > Jul 21 13:05:13 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn b cmd 23, got netfn 5 cmd 12 > Jul 21 13:05:13 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 5 cmd 12, got netfn b cmd 23 > Jul 21 13:05:13 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn b cmd 23, got netfn 5 cmd 12 > Jul 21 13:05:13 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 5 cmd 12, got netfn b cmd 23 > Jul 21 13:05:13 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn b cmd 23, got netfn 5 cmd 12 > Jul 21 13:05:13 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 5 cmd 12, got netfn b cmd 23 > Jul 21 13:05:13 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn b cmd 23, got netfn 5 cmd 12 > Jul 21 13:05:13 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 5 cmd 12, got netfn b cmd 23 > Jul 21 13:05:13 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn b cmd 23, got netfn 5 cmd 12 > Jul 21 13:05:13 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 5 cmd 12, got netfn b cmd 23 > Jul 21 13:05:13 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 7 cmd 35, got netfn 5 cmd 12 > Jul 21 13:05:13 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 5 cmd 12, got netfn 7 cmd 35 > Jul 21 13:05:13 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn d cmd 2, got netfn 5 cmd 12 > Jul 21 13:07:00 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn b cmd 23, got netfn 5 cmd 12 > Jul 21 13:07:00 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 5 cmd 12, got netfn b cmd 23 > Jul 21 13:07:00 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn b cmd 23, got netfn 5 cmd 12 > Jul 21 13:07:00 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 5 cmd 12, got netfn b cmd 23 > Jul 21 13:07:00 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn b cmd 23, got netfn 5 cmd 12 > Jul 21 13:07:00 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 5 cmd 12, got netfn b cmd 23 > Jul 21 13:07:00 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn b cmd 23, got netfn 5 cmd 12 > Jul 21 13:07:00 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn d cmd 2, got netfn b cmd 23 > Jul 21 13:07:00 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn b cmd 23, got netfn d cmd 2 > Jul 21 13:07:00 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn d cmd 2, got netfn b cmd 23 > Jul 21 13:07:00 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn b cmd 23, got netfn d cmd 2 > Jul 21 13:07:00 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn d cmd 2, got netfn b cmd 23 > Jul 21 13:07:00 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 7 cmd 35, got netfn d cmd 2 > Jul 21 13:07:00 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn d cmd 2, got netfn 7 cmd 35 > Jul 21 13:08:08 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 2d cmd 0, got netfn 5 cmd 12 > Jul 21 13:08:08 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 5 cmd 12, got netfn 2d cmd 0 > Jul 21 13:08:08 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 7 cmd 1, got netfn 5 cmd 12 > Jul 21 13:08:08 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 5 cmd 12, got netfn 7 cmd 1 > Jul 21 13:08:08 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 7 cmd 35, got netfn 5 cmd 12 > Jul 21 13:08:08 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 5 cmd 12, got netfn 7 cmd 35 > Jul 21 13:08:08 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn d cmd 2, got netfn 5 cmd 12 > Jul 21 13:09:17 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 7 cmd 35, got netfn 5 cmd 12 > Jul 21 13:09:17 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 5 cmd 12, got netfn 7 cmd 35 > Jul 21 13:09:17 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 7 cmd 35, got netfn 5 cmd 12 > Jul 21 13:09:17 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 5 cmd 12, got netfn 7 cmd 35 > Jul 21 13:09:17 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn d cmd 2, got netfn 5 cmd 12 > Jul 21 13:10:23 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 2d cmd 0, got netfn 5 cmd 12 > Jul 21 13:10:23 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 5 cmd 12, got netfn 2d cmd 0 > Jul 21 13:10:23 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 7 cmd 1, got netfn 5 cmd 12 > Jul 21 13:10:23 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 5 cmd 12, got netfn 7 cmd 1 > Jul 21 13:10:23 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 7 cmd 35, got netfn 5 cmd 12 > Jul 21 13:10:23 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 5 cmd 12, got netfn 7 cmd 35 > Jul 21 13:10:23 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn d cmd 2, got netfn 5 cmd 12 > Jul 21 13:11:33 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 7 cmd 35, got netfn 5 cmd 12 > Jul 21 13:11:33 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn 5 cmd 12, got netfn 7 cmd 35 > Jul 21 13:11:34 tigw1u-ht kernel: IPMI message handler: BMC returned > incorrect response, expected netfn d cmd 2, got netfn 5 cmd 12 > > -----Original Message----- > From: Corey Minyard [mailto:[email protected]] > Sent: Saturday, July 25, 2009 3:57 PM > To: Mathieu Alorent > Cc: [email protected]; Gemini TRAN > Subject: [SPAM] - Re: [Openipmi-developer] BMC returned incorrect response - > Email found in subject > > Mathieu Alorent wrote: > >> Le mercredi 11 mars 2009 à 08:50 -0500, Corey Minyard a écrit : >> >> >>> Mathieu ALORENT wrote: >>> >>> >>>> Hi, >>>> >>>> we've got this error on all our SUN V40Z systems. I don't know what does >>>> it mean.. >>>> >>>> [kern.warning] kernel IPMI message handler: BMC returned incorrect >>>> response, expected netfn 3b cmd 3c, got netfn 3a cmd 3c >>>> >>>> Can someone give me an explanation ? >>>> >>>> >>>> >>> It appears to be a bug in the BMC. On responses, the netfn is supposed >>> to have the 1 set in the lowest bit. So if it sends netfn 0x2c, it >>> should get back netfn 0x2d. >>> >>> These appear to be for error responses, for instance the 2C/2D one is an >>> ATCA probe (which your box won't support, since I don't think it's >>> ATCA). So it appears to be returning incorrect responses for errors. >>> You should report this to Sun, though if everything else is working >>> correctly and it's not spewing out these errors it shouldn't affect >>> normal operations very much. >>> >>> The last one is a little strange, it's doing a "Get SDR" request and >>> that shouldn't fail. >>> >>> -corey >>> >>> >>> >>>> Regards >>>> Mât >>>> >>>> --------------------------------------- >>>> >>>> # uname -a >>>> Linux xxxxxxxxxxxxxx 2.6.24-standard #2 SMP Fri Jul 11 15:04:57 UTC 2008 >>>> x86_64 GNU/Linux >>>> >>>> >>>> # dmesg |grep -i IPMI >>>> ipmi message handler version 39.1 >>>> ipmi device interface >>>> IPMI System Interface driver. >>>> ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca2, >>>> slave address 0x20, irq 0 >>>> ipmi: Found new BMC (man_id: 0x00002a, prod_id: 0x09ef, dev_id: 0x20) >>>> IPMI kcs interface initialized >>>> Copyright (C) 2004 MontaVista Software - IPMI Powerdown via sys_reboot. >>>> IPMI message handler: BMC returned incorrect response, expected netfn 2d >>>> cmd 1, got netfn 2c cmd 1 >>>> IPMI poweroff: ATCA Detect mfg 0x2A prod 0x9EF >>>> IPMI poweroff: Found a chassis style poweroff function >>>> IPMI message handler: BMC returned incorrect response, expected netfn 3b >>>> cmd 3c, got netfn 3a cmd 3c >>>> IPMI message handler: BMC returned incorrect response, expected netfn b >>>> cmd 23, got netfn 5 cmd 2d >>>> >>>> >>>> >> I've contacted SUN about this, a ticket is open... >> >> Today, I've found similar logs on another system "HP DL360 G6"... >> here are the logs... >> >> IPMI message handler: BMC returned incorrect response, expected netfn 5 >> cmd 27, got netfn 5 cmd 35 >> IPMI message handler: BMC returned incorrect response, expected netfn 5 >> cmd 27, got netfn 5 cmd 35 >> IPMI message handler: BMC returned incorrect response, expected netfn 5 >> cmd 27, got netfn 5 cmd 35 >> >> >> what I've done is : >> >> $ watch -n1 "ipmitool sensor" >> >> >> So this let me think that it's maybe not a bug in BMC... >> >> What is your opinion? >> >> >> > I've let this run for a while on one of my systems and haven't seen the > problem. Does this happen constantly, or is this a rare occurrence? > > If the driver is getting confused, it's possible this is due to a > conflict with a read event buffer, but that would not happen very > often. If it's happening all the time, then it's probably a bug in the > BMC. Also, if it's the same netfn and command getting messed up every > time, then it's not likely to be the driver, as it doesn't care about > those values beyond just verifying that the send and receive match. > That would point to a problem in the BMC. If the it's different > netfn/commands, then that might be the driver. > > -corey > > > ------------------------------------------------------------------------------ > _______________________________________________ > Openipmi-developer mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/openipmi-developer > G2Ɨ7FfV֒FWfVW > > > ------------------------------------------------------------------------------ > _______________________________________________ > Openipmi-developer mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/openipmi-developer > ------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ Openipmi-developer mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/openipmi-developer
