Hey Guys, Ran into an issue with hardware raid on the Sun x4100 that we run zenoss on. Zenoss did not survive it unfortunetly, upon rebooting i am seeing some missing .fm files and the more typical iteration over non-sequence error.
I've followed this post: http://community.zenoss.com/forums/viewtopic.php?t=4434&highlight=unclean+shutdown which was kind of similar. Deets: Zenoss: 2.1.3 OS: SLES 10.2 sp1 Hardware: Sun x4100 Phython: 2.4.4 The main error from zenhub is this: ZenBackendFailure: (1017, "Can't find file: './events/status.frm' (errno: 13)") The zenmodler is saying this: OperationalError: (1017, "Can't find file: './events/heartbeat.frm' (errno:") And the majority of the other daemons return the non-sequence error. So using the method Cluther posted in the post i linked.. here is the debug info from zenhub as it starts: Code: [EMAIL PROTECTED]:/usr/zenoss/bin> ./zenhub run -v 10 DEBUG:zen.Events:looking up ip hector.aliant.icn DEBUG:zen.Events:EventClassInst=Start DEBUG:zen.DbConnectionPool:Creating a new connection; Pool size: 0 DEBUG:zen.Events:update status set clearid = '8eb1026136aed40affff2da' where device='hector.aliant.icn' and component='zenhub' and eventKey='' and (eventClass='/App/Stop' or eventClass='/App/Start') ERROR:zen.Events:(1017, "Can't find file: './events/status.frm' (errno: 13)") Traceback (most recent call last): File "/usr/zenoss/Products/ZenEvents/MySqlSendEvent.py", line 31, in execute result = cursor.execute(statement) File "usr/zenoss/lib/python/MySQLdb/cursors.py", line 137, in execute File "usr/zenoss/lib/python/MySQLdb/connections.py", line 33, in defaulterrorhandler OperationalError: (1017, "Can't find file: './events/status.frm' (errno: 13)") DEBUG:zen.DbConnectionPool:Returned a connection; Pool size: 1 ERROR:zen.Events:(1017, "Can't find file: './events/status.frm' (errno: 13)") Traceback (most recent call last): File "/usr/zenoss/Products/ZenEvents/MySqlSendEvent.py", line 98, in sendEvent evid = self.doSendEvent(event) File "/usr/zenoss/Products/ZenEvents/MySqlSendEvent.py", line 121, in doSendEvent rows = execute(curs, self.buildClearUpdate(event, clearcls)) File "/usr/zenoss/Products/ZenEvents/MySqlSendEvent.py", line 36, in execute raise ex OperationalError: (1017, "Can't find file: './events/status.frm' (errno: 13)") ERROR:zen.ZenHub:Unable to send an event Traceback (most recent call last): File "/usr/zenoss/Products/ZenHub/zenhub.py", line 282, in sendEvent self.zem.sendEvent(Event(**kw)) File "/usr/zenoss/Products/ZenEvents/MySqlSendEvent.py", line 103, in sendEvent raise ZenBackendFailure(str(e)) ZenBackendFailure: (1017, "Can't find file: './events/status.frm' (errno: 13)") Here is the output from zenperf: Code: [EMAIL PROTECTED]:/usr/zenoss/bin> ./zenperfsnmp run -v 10 DEBUG:zen.zenperfsnmp:run DEBUG:zen.zenperfsnmp:Connecting to localhost DEBUG:zen.zenperfsnmp:Logging in as admin WARNING:zen.zenperfsnmp:Reconnected to ZenHub DEBUG:zen.zenperfsnmp:setting up services EventService, SnmpPerfConfig DEBUG:zen.zenperfsnmp:chaining getInitialServices with d2 DEBUG:zen.zenperfsnmp:callback after getting service EventService DEBUG:zen.zenperfsnmp:callback after getting service SnmpPerfConfig DEBUG:zen.zenperfsnmp:Calling connected. DEBUG:zen.zenperfsnmp:connected DEBUG:zen.zenperfsnmp:Sending event {'severity': 0, 'component': 'zenperfsnmp', 'agent': 'zenperfsnmp', 'summary': 'started', 'manager': 'localhost', 'device': 'hector.aliant.icn', 'eventClass': '/App/Start'} INFO:zen.zenperfsnmp:fetching property items DEBUG:zen.zenperfsnmp:Updated configCycleInterval config to 360 INFO:zen.zenperfsnmp:getting threshold classes DEBUG:zen.zenperfsnmp:Loading classes ['Products.ZenModel.MinMaxThreshold'] INFO:zen.zenperfsnmp:checking for outdated configs INFO:zen.zenperfsnmp:fetching configs for []... INFO:zen.zenperfsnmp:fetching default RRDCreateCommand INFO:zen.zenperfsnmp:fetching snmp status ERROR:zen.zenperfsnmp:[Failure instance: Traceback (failure with no frames): exceptions.TypeError: iteration over non-sequence ] Traceback (most recent call last): File "/usr/zenoss/Products/ZenUtils/Driver.py", line 46, in _next self.iter.next().addBoth(self._finish) File "/usr/zenoss/Products/ZenRRD/zenperfsnmp.py", line 325, in startUpdateConfig self.updateSnmpStatus(driver.next()) File "/usr/zenoss/Products/ZenRRD/zenperfsnmp.py", line 401, in updateSnmpStatus countMap = dict(status) TypeError: iteration over non-sequence DEBUG:zen.zenperfsnmp:Sending event {'severity': 3, 'component': 'zenperfsnmp', 'agent': 'zenperfsnmp', 'summary': 'stopped', 'manager': 'localhost', 'device': 'hector.aliant.icn', 'eventClass': '/App/Stop'} DEBUG:zen.zenperfsnmp:removing service EventService DEBUG:zen.zenperfsnmp:removing service SnmpPerfConfig INFO:zen.zenperfsnmp:zenperfsnmp shutting down And finally here is what zenhub said when i kicked off the zenperf: Code: DEBUG:zen.DbConnectionPool:Retrieved a connection; Pool size: 0 DEBUG:zen.DbConnectionPool:Returned a connection; Pool size: 1 ERROR:zenhub:Unable to get Snmp Status Traceback (most recent call last): File "/usr/zenoss/Products/ZenHub/services/PerformanceConfig.py", line 135, in remote_getSnmpStatus curs.execute(cmd); File "usr/zenoss/lib/python/MySQLdb/cursors.py", line 137, in execute File "usr/zenoss/lib/python/MySQLdb/connections.py", line 33, in defaulterrorhandler OperationalError: (1017, "Can't find file: './events/status.frm' (errno: 13)") Peer will receive following PB traceback: Traceback (most recent call last): File "usr/zenoss/lib/python/twisted/spread/banana.py", line 148, in gotItem File "usr/zenoss/lib/python/twisted/spread/banana.py", line 113, in callExpressionReceived File "usr/zenoss/lib/python/twisted/spread/pb.py", line 522, in expressionReceived File "usr/zenoss/lib/python/twisted/spread/pb.py", line 833, in proto_message --- <exception caught here> --- File "usr/zenoss/lib/python/twisted/spread/pb.py", line 847, in _recvMessage File "usr/zenoss/lib/python/twisted/spread/flavors.py", line 119, in remoteMessageReceived File "/usr/zenoss/Products/ZenHub/services/PerformanceConfig.py", line 135, in remote_getSnmpStatus curs.execute(cmd); File "usr/zenoss/lib/python/MySQLdb/cursors.py", line 137, in execute File "usr/zenoss/lib/python/MySQLdb/connections.py", line 33, in defaulterrorhandler _mysql_exceptions.OperationalError: (1017, "Can't find file: './events/status.frm' (errno: 13)") DEBUG:zen.Events:looking up ip hector.aliant.icn DEBUG:zen.Events:EventClassInst=Stop DEBUG:zen.DbConnectionPool:Retrieved a connection; Pool size: 0 DEBUG:zen.Events:insert into status set firstTime=1209481558.594,severity=2,component='zenperfsnmp',agent='zenperfsn mp',summary='stopped',dedupid='hector.aliant.icn|zenperfsnmp|/App/Stop||2|st opped',manager='localhost',eventKey='',device='hector.aliant.icn',eventClass ='/App/Stop',lastTime=1209481558.594,message='stopped',evid='8eb1026136aed45 6fffd2da' on duplicate key update summary='stopped',count=count+1,lastTime=1209481558.594 ERROR:zen.Events:(1017, "Can't find file: './events/status.frm' (errno: 13)") Traceback (most recent call last): File "/usr/zenoss/Products/ZenEvents/MySqlSendEvent.py", line 31, in execute result = cursor.execute(statement) File "usr/zenoss/lib/python/MySQLdb/cursors.py", line 137, in execute File "usr/zenoss/lib/python/MySQLdb/connections.py", line 33, in defaulterrorhandler OperationalError: (1017, "Can't find file: './events/status.frm' (errno: 13)") DEBUG:zen.DbConnectionPool:Returned a connection; Pool size: 1 ERROR:zen.Events:(1017, "Can't find file: './events/status.frm' (errno: 13)") Traceback (most recent call last): File "/usr/zenoss/Products/ZenEvents/MySqlSendEvent.py", line 98, in sendEvent evid = self.doSendEvent(event) File "/usr/zenoss/Products/ZenEvents/MySqlSendEvent.py", line 132, in doSendEvent rescount = execute(curs, stmt) File "/usr/zenoss/Products/ZenEvents/MySqlSendEvent.py", line 36, in execute raise ex OperationalError: (1017, "Can't find file: './events/status.frm' (errno: 13)") ERROR:log:(1017, "Can't find file: './events/status.frm' (errno: 13)") Traceback (most recent call last): File "/usr/zenoss/Products/ZenHub/services/EventService.py", line 26, in remote_sendEvent return self.zem.sendEvent(evt) File "/usr/zenoss/Products/ZenEvents/MySqlSendEvent.py", line 103, in sendEvent raise ZenBackendFailure(str(e)) ZenBackendFailure: (1017, "Can't find file: './events/status.frm' (errno: 13)") WARNING:zenhub:removing listener WARNING:zenhub:removing listener I see another post that suggests dropping the events table, which i could try. There isn't much data in this particular zenoss so i'm not worried about retaining any data gladly. But the scenario in that post was slightly different. So if anyone has some suggestions about how i can fix this it would be great. I'm not opposed to a reinstall i guess. I'm going to continue digging around now. -------------------- m2f -------------------- Read this topic online here: http://community.zenoss.com/forums/viewtopic.php?p=19124#19124 -------------------- m2f -------------------- _______________________________________________ zenoss-users mailing list [email protected] http://lists.zenoss.org/mailman/listinfo/zenoss-users
