Bumping this thread. Adding in users to see if anyone else has seen this. I am running into the exact same issue. XenServer 6.0.2, Basic networking with bridging with CSP installed. However I am using CS 4.0.1.
My issues arised after I rebooted my XS host outside of CS (I believe this was due to inode exhaustion although i didn't realize this until later) Upon start CS seemingly connects to the host for an indefinite amount of time. Unfortunately nothing in logs explains why its behaving like this (CS management-server.log) So far I've tried setting the networking from bridge->ovs->bridge and reinstalling the CSP with no success. On Wed, Dec 5, 2012 at 6:08 PM, Jason Bausewein (JIRA) <j...@apache.org>wrote: > > [ > https://issues.apache.org/jira/browse/CLOUDSTACK-105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510950#comment-13510950] > > Jason Bausewein edited comment on CLOUDSTACK-105 at 12/6/12 12:07 AM: > ---------------------------------------------------------------------- > > I am able to reproduce this issue consistently. I have a single xen host > in a basic zone. I tracked it down to the following process creating the > stream-unix.****.**** files about every 10 seconds. > > root 8237 8223 0 09:59 ? 00:00:00 ovs-vsctl add-br xapi0 > > Dec 5 09:59:15 xenserver1 ovs-vsctl: 00001|vsctl|INFO|Called as ovs-vsctl > add-br xapi0 > Dec 5 09:59:15 xenserver1 ovs-vsctl: > 00002|stream_unix|ERR|/tmp/stream-unix.8237.0: connection to > /var/run/openvswitch/db.sock failed: No such file or directory > Dec 5 09:59:15 xenserver1 ovs-vsctl: > 00003|reconnect|WARN|unix:/var/run/openvswitch/db.sock: connection attempt > failed (No such file or directory) > Dec 5 09:59:16 xenserver1 ovs-vsctl: > 00004|stream_unix|ERR|/tmp/stream-unix.8237.1: connection to > /var/run/openvswitch/db.sock failed: No such file or directory > Dec 5 09:59:16 xenserver1 ovs-vsctl: > 00005|reconnect|WARN|unix:/var/run/openvswitch/db.sock: connection attempt > failed (No such file or directory) > Dec 5 09:59:18 xenserver1 ovs-vsctl: > 00006|stream_unix|ERR|/tmp/stream-unix.8237.2: connection to > /var/run/openvswitch/db.sock failed: No such file or directory > Dec 5 09:59:18 xenserver1 ovs-vsctl: > 00007|reconnect|WARN|unix:/var/run/openvswitch/db.sock: connection attempt > failed (No such file or directory) > > These messages start immediately on boot. I attached my messages log file. > > I am not using openvswitch. Is there something I need to turn off? > > > was (Author: jbausewein): > I am able to reproduce this issue consistently. I have a single xen > host in a basic zone. I tracked it down to the following process creating > the stream-unix.****.**** files about every 10 seconds. > > root 8237 8223 0 09:59 ? 00:00:00 ovs-vsctl add-br xapi0 > > Dec 5 09:59:15 xenserver1 ovs-vsctl: 00001|vsctl|INFO|Called as ovs-vsctl > add-br xapi0 > Dec 5 09:59:15 xenserver1 ovs-vsctl: > 00002|stream_unix|ERR|/tmp/stream-unix.8237.0: connection to > /var/run/openvswitch/db.sock failed: No such file or directory > Dec 5 09:59:15 xenserver1 ovs-vsctl: > 00003|reconnect|WARN|unix:/var/run/openvswitch/db.sock: connection attempt > failed (No such file or directory) > Dec 5 09:59:16 xenserver1 ovs-vsctl: > 00004|stream_unix|ERR|/tmp/stream-unix.8237.1: connection to > /var/run/openvswitch/db.sock failed: No such file or directory > Dec 5 09:59:16 xenserver1 ovs-vsctl: > 00005|reconnect|WARN|unix:/var/run/openvswitch/db.sock: connection attempt > failed (No such file or directory) > Dec 5 09:59:18 xenserver1 ovs-vsctl: > 00006|stream_unix|ERR|/tmp/stream-unix.8237.2: connection to > /var/run/openvswitch/db.sock failed: No such file or directory > Dec 5 09:59:18 xenserver1 ovs-vsctl: > 00007|reconnect|WARN|unix:/var/run/openvswitch/db.sock: connection attempt > failed (No such file or directory) > > I am not using openvswitch. Is there something I need to turn off? > > > > /tmp/stream-unix.####.###### stale sockets causing inodes to run out on > Xenserver > > > --------------------------------------------------------------------------------- > > > > Key: CLOUDSTACK-105 > > URL: > https://issues.apache.org/jira/browse/CLOUDSTACK-105 > > Project: CloudStack > > Issue Type: Bug > > Security Level: Public(Anyone can view this level - this is the > default.) > > Components: XenServer > > Affects Versions: pre-4.0.0 > > Environment: Xenserver 6.0.2 > > Cloudstack 3.0.2 > > Reporter: Caleb Call > > Assignee: Devdeep Singh > > Fix For: 4.1.0 > > > > Attachments: messages > > > > > > We came across an interesting issue in one of our clusters. We ran out > of inodes on all of our cluster members (since when does this happen in > 2012?). When this happened, it in turn made the / filesystem a read-only > filesystem which in turn made all the hosts go in to emergency maintenance > mode and as a result get marked down by Cloudstack. We found that it was > caused by hundreds of thousands of stale socket files in /tmp named > "stream-unix.####.######". To resolve the issue, we had to delete those > stale socket files (find /tmp -name "*stream*" -mtime +7 -exec rm -v {} > \;), then kill and restart xapi, then correct the emergency maintenance > mode. These hosts had only been up for 45 days before this issue occurred. > > In our scouring of the interwebs, the only other instance we've been > able to find of this (or similar) happening is in the same setup we are > currently running. Xenserver 6.0.2 with CS 3.0.2. Do these stream-unix > sockets have anything to do with Cloudstack? I would think if this was a > Xenserver issue (bug), there would be a lot more on the internet about this > happening. For a temporary workaround, we've added a cronjob to cleanup > these files but we'd really like to address the actual issue that's causing > these sockets to become stale and not get cleaned-up. > > -- > This message is automatically generated by JIRA. > If you think it was sent incorrectly, please contact your JIRA > administrators > For more information on JIRA, see: http://www.atlassian.com/software/jira >
