Here you go: [root@xcp-hoppe ~]# ps axf | grep xapi 7598 ? S 32:46 /opt/xensource/libexec/cdrommon /dev/xapi/cd/sr0 6840 pts/8 S+ 0:00 | \_ grep xapi 24767 ? Z<l 88:38 [xapi] <defunct> <<<<<<<<<<<< defunct?????? 6757 ? S<s 0:00 /usr/sbin/stunnel /var/xapi/xapi-ssl.conf
[root@xcp-hoppe ~]# ps axf | grep stunnel 6677 pts/8 S+ 0:00 \_ grep stunnel 6397 ? S<s 0:00 /usr/sbin/stunnel /var/xapi/xapi-ssl.conf There is something listening on port 443, though my skills don't allow me to ascertain what. Same goes for 80: [root@xcp-hoppe ~]# netstat -an Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 127.0.0.1:199 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:5900 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:5901 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:5902 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:5903 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:111 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:600 0.0.0.0:* LISTEN tcp 0 0 192.168.1.10:443 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:443 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:3551 0.0.0.0:* LISTEN tcp 0 0 192.168.1.10:22 192.168.1.1:4930 ESTABLISHED tcp 0 0 192.168.1.10:51388 192.168.1.1:445 ESTABLISHED tcp 0 0 192.168.1.10:22 10.168.1.9:61784 ESTABLISHED tcp 1 0 127.0.0.1:80 127.0.0.1:48318 CLOSE_WAIT tcp 0 52 192.168.1.10:22 192.168.1.1:1093 ESTABLISHED udp 0 0 0.0.0.0:58882 0.0.0.0:* udp 0 0 0.0.0.0:161 0.0.0.0:* udp 0 0 0.0.0.0:600 0.0.0.0:* udp 0 0 0.0.0.0:601 0.0.0.0:* udp 0 0 127.0.0.1:111 0.0.0.0:* udp 0 0 192.168.1.10:123 0.0.0.0:* udp 0 0 127.0.0.1:123 0.0.0.0:* udp 0 0 0.0.0.0:123 0.0.0.0:* netstat -an | grep stunnel returns nothing. [root@xcp-hoppe ~]# netstat -an | grep xapi unix 2 [ ACC ] STREAM LISTENING 708872842 /var/run/openvswitch/xapi1.snoop unix 2 [ ACC ] STREAM LISTENING 708872844 /var/run/openvswitch/xapi1.mgmt unix 2 [ ACC ] STREAM LISTENING 834141817 /var/xapi/forker/main unix 2 [ ACC ] STREAM LISTENING 834141947 /var/xapi/v6 unix 2 [ ] STREAM CONNECTED 831867057 /var/xapi/xapi unix 2 [ ] STREAM CONNECTED 831832667 /var/xapi/xapi unix 2 [ ] STREAM CONNECTED 831831438 /var/xapi/xapi unix 2 [ ] STREAM CONNECTED 831825575 /var/xapi/xapi unix 2 [ ] STREAM CONNECTED 827814254 /var/xapi/forker/main unix 2 [ ] STREAM CONNECTED 806731261 /var/xapi/forker/main unix 2 [ ] STREAM CONNECTED 797242460 /var/xapi/forker/main New pastebin is up, with 1749 lines. Enjoy :p http://pastebin.com/gHTeNqXb Thanks! On 20 January 2012 16:21, Dave Scott <[email protected]> wrote: > Hi,**** > > ** ** > > Thanks for the info.**** > > ** ** > > Could you check whether stunnel is running in ‘ps axf’? Also check in > ‘netstat –an’ to see whether something is listening on port 443 (should be > stunnel) and port 80 (should be xapi).**** > > ** ** > > The pastebin text stops at line 310, which looks inoccuous. Is there > anything after that?**** > > ** ** > > Cheers,**** > > Dave**** > > ** ** > > ** ** > > ** ** > > *From:* Lars Seeliger [mailto:[email protected]] > *Sent:* 20 January 2012 14:50 > > *To:* Dave Scott > *Cc:* [email protected] > *Subject:* Re: [Xen-API] xapi will no longer start - what are my options?* > *** > > ** ** > > It's a full install from an XCP CD. > > # service xapissl status > xapissl (pid 2323) is running... > > I performed a restart of xapissl anyway, and xe-toolstack-restart still > fails starting the xapi service. Any other ideas? What would happen if I > were to restart the system or perform a reinstall? > > Our customer and us are pretty nervous now, as they have not had a > successful backup since last week Friday. So what else can I try? > > Thanks, Dave!**** > > On 20 January 2012 15:41, Dave Scott <[email protected]> wrote:**** > > Hi,**** > > **** > > I should have asked earlier: is this a host installed via the XCP CD, or > is this a Debian system running the xcp- packages?**** > > **** > > It looks like xapi can’t find a running stunnel, and it looks like the > xe-toolstack-restart failed to run “/sbin/service xapissl restart”. Do you > have stunnel running, and listening on port 443? On a system installed via > the XCP CD, “service xapissl restart” should start stunnel. On a Debian > system – I believe – the xapi init.d script itself starts stunnel.**** > > **** > > Dave**** > > **** > > *From:* Lars Seeliger [mailto:[email protected]] > *Sent:* 20 January 2012 12:15 > *To:* Dave Scott > *Cc:* [email protected] > *Subject:* Re: [Xen-API] xapi will no longer start - what are my options?* > *** > > **** > > Hey, Dave > > Thanks for the prompt response. For fear of not including enough info, > I've pasted the entire xensource.log contents here: > http://pastebin.com/AW12gfM0 > > If you need anything else, just shout; this problem has ruined my day! :p* > *** > > On 20 January 2012 12:53, Dave Scott <[email protected]> wrote:**** > > Hi Lars,**** > > **** > > Have a look in the main xapi logs (in the confusingly-named file > /var/log/xensource.log). Start from the bottom and reverse-search to the > string “XAPI SERVER STARTING”. The lines after that will show how far the > startup sequence got.**** > > **** > > Cheers,**** > > Dave**** > > **** > > **** > > **** > > *From:* [email protected] [mailto: > [email protected]] *On Behalf Of *Lars Seeliger > *Sent:* 20 January 2012 10:23 > *To:* [email protected] > *Subject:* [Xen-API] xapi will no longer start - what are my options?**** > > **** > > Hi there > > A scripted backup running on one of our XCP installations failed a few > days ago, while exporting a snapshot. After the failure I tried to delete > the snapshot in question, to no avail (something about the VDI being in > use). > > Anyway, I thought an xe-toolstack-restart would reset any lock on that > file, allowing me to delete the no longer needed snapshot. Unfortunately, > xapi is now unable to start. > > I've just tried again, and this appears in /var/log/messages: > > Jan 20 11:01:21 xcp-hoppe xapi: [ info|xcp-hoppe|0 thread_zero||watchdog] > (Re)starting xapi... > Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.850Z||1172|About to bind > to /var/xapi/forker/fd_e8f89481-9aae-05a5-1d73-fbb713f58ea3 > Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.850Z||1172|bound, > listening > Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.851Z||2300|Child here! > Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.851Z||2301|Grandchild > here! > Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.851Z||2301|Started: > state.cmdargs = [/sbin/service;xapissl;restart] > Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.851Z||2301|Started: > state.env = [PATH=/sbin:/usr/sbin:/bin:/usr/bin] > Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Selecting in > handle_comms_no_fd_sock2 > Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Done > Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|fd sock > Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Selecting in > handle_comms_with_fd_sock2 > Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Done > Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|fd sock2 > Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Received fd > named: ed350b47-3eb6-63e0-38c5-3beaaefb65dd - duping to 1 (from 6) > Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Selecting in > handle_comms_with_fd_sock2 > Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Done > Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|fd sock2 > Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Received fd > named: d810a903-961d-bb1e-aeb7-b39c98e5eefa - duping to 2 (from 6) > Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Selecting in > handle_comms_with_fd_sock2 > Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Done > Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|comms sock > Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Exec > Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Finished... > Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Args after > replacement = [/sbin/service;xapissl;restart] > Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|I've received > the following fds: [2;1] > Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:26.942Z||2301|Caught > unexpected exception: Unix.Unix_error(31, "write", "") > Jan 20 10:05:41 xcp-hoppe HVM5652[12007]: XENNET: WARNING: No handler > for oid 0xffda0014. > Jan 20 10:05:41 xcp-hoppe HVM5652[12007]: XENNET: WARNING: No handler > for oid 0xffa0ffa1. > Jan 20 10:05:41 xcp-hoppe HVM5652[12007]: XENNET: WARNING: Bad tcp task > offload header? > Jan 20 10:05:41 xcp-hoppe HVM5652[12007]: XENNET: WARNING: Bad tcp task > offload header? > Jan 20 11:06:16 xcp-hoppe snmpd[15173]: Received SNMP packet(s) from UDP: > [192.168.1.1]:2227 > Jan 20 11:06:16 xcp-hoppe snmpd[15173]: Received SNMP packet(s) from UDP: > [192.168.1.1]:2228 > Jan 20 11:06:17 xcp-hoppe snmpd[15173]: Received SNMP packet(s) from UDP: > [192.168.1.1]:2229 > Jan 20 11:06:17 xcp-hoppe snmpd[15173]: Received SNMP packet(s) from UDP: > [192.168.1.1]:2230 > Jan 20 11:06:25 xcp-hoppe python: PERFMON: caught socket.error: (111 > Connection refused) - restarting XAPI session > Jan 20 10:08:36 xcp-hoppe HVM5652[12007]: Time offset set 3569, added > offset -1 > Jan 20 11:08:54 xcp-hoppe python: PERFMON: Caught signal 15 - exiting > Jan 20 11:08:54 xcp-hoppe python: PERFMON: 11 Resource temporarily > unavailable > Jan 20 11:08:54 xcp-hoppe python: PERFMON: Traceback (most recent call > last): > Jan 20 11:08:54 xcp-hoppe python: PERFMON: File > "/opt/xensource/bin/perfmon", line 930, in ? rc = main() > Jan 20 11:08:54 xcp-hoppe python: PERFMON: File > "/opt/xensource/bin/perfmon", line 880, in main cmd = > cmdsock.recv(cmdmaxlen) > Jan 20 11:08:54 xcp-hoppe python: PERFMON: error: (11, 'Resource > temporarily unavailable') > Jan 20 11:08:54 xcp-hoppe python: PERFMON: caught socket.error: (111 > Connection refused) - restarting XAPI session > Jan 20 11:08:55 xcp-hoppe v6d: [ info|xcp-hoppe|0||watchdog] (Re)starting > v6d... > Jan 20 11:08:55 xcp-hoppe xapi: [ info|xcp-hoppe|0 thread_zero||watchdog] > (Re)starting xapi... > Jan 20 10:10:45 xcp-hoppe HVM5641[28792]: Time offset set 3563, added > offset -1 > Jan 20 11:13:45 xcp-hoppe python: PERFMON: caught socket.error: (111 > Connection refused) - restarting XAPI session > > > /var/log/SMI contains: > > [2598] 2012-01-20 11:08:55.277830 VASSR run > ['/opt/xensource/sm/VASSR', > '<methodCall><methodName>sr_get_driver_info</methodName><params><param><value><struct><member><name>host_ref</name><value>OpaqueRef:NULL</value></member><member><name>command</name><value>sr_get_driver_info</value></member><member><name>args</name><value><array><data/></array></value></member><member><name>device_config</name><value><struct/></value></member></struct></value></param></params></methodCall>'] > [2598] 2012-01-20 11:08:55.278332 Warning: vdi_[de]activate present > for vastsky > [2619] 2012-01-20 11:08:55.858537 Warning: vdi_[de]activate present > for dummy > > Not sure there's anything of value in those logs... > > I'm somewhat desperate, as I'm unable to perform any xe commands and am > worried a reboot of the server will not magically bring xapi back online, > meaning the VMs will not start. This XCP host is critical to one of our > customer's infrastructure. It's Friday and I could possibly go there this > evening and perform tasks necessary to bring everything back online, I just > don't quite know what my options are, aside from reboot and perhaps an > in-place install of XCP. > > Does anyone have any bright ideas? I'm all ears!**** > > **** > > ** ** >
_______________________________________________ xen-api mailing list [email protected] http://lists.xensource.com/mailman/listinfo/xen-api
