Public bug reported:
We have a Ubuntu Openstack HA cluster running on 5 OpenPower nodes
(POWER8).
This is on 14.04 with Openstack Liberty
It was all working just peachy, when node 5 starting having issues and became
inaccessible.
It seems the file system on the local disk (ext4) has some errors and requires
a fsck/e2fsck.
However, the erros in the filesystem seems to have affected the SSH keys so I
cannot get into the system.
I've attached the boot log of the system that I've retrieved from the
console. Note that the juju-agent cannot start, but some Openstack
services do.
When this system is up, we get errors in the Horizon gui and VMs cannot be
provisioned.
Below is the bundle.yaml for the HA environment model.
So this may be a potential bug - if we have a system in a partially
failed/untrusted state, should Openstack/Horizon/Nova still work as
expected?
A couple notes:
As you can see in the bundler we are not using clustered mysql at this point,
but the mysql db is not on this node.
This node is one of two Ceph OSD nodes.
I've not tried redeploying the system yet. I may be able to use Petitboot to do
the fsck, or use it to modify the /etc/fstab so it will happen at boot time.
The main goal here is not necessarily to fix this particularly node, but to
explore what should happen with Openstack in this situation.
Let me know if more information is needed.
bundle.yaml
=============
machines:
'0':
constraints: arch=ppc64el
series: trusty
'1':
constraints: { arch=ppc64el, tags=compute }
series: trusty
'2':
constraints: { arch=ppc64el, tags=compute }
series: trusty
'3':
constraints: { arch=ppc64el, tags=compute }
series: trusty
'4':
constraints: { arch=ppc64el, tags=compute }
series: trusty
'5':
constraints: { arch=ppc64el, tags=compute }
series: trusty
'6':
constraints: { arch=ppc64el, tags=neutron-gateway }
series: trusty
relations:
- - nova-compute:amqp
- rabbitmq-server:amqp
- - neutron-gateway:amqp
- rabbitmq-server:amqp
- - keystone:shared-db
- mysql:shared-db
- - nova-cloud-controller:identity-service
- keystone:identity-service
- - glance:identity-service
- keystone:identity-service
- - neutron-api:identity-service
- keystone:identity-service
- - neutron-openvswitch:neutron-plugin-api
- neutron-api:neutron-plugin-api
- - neutron-api:shared-db
- mysql:shared-db
- - neutron-api:amqp
- rabbitmq-server:amqp
- - neutron-gateway:neutron-plugin-api
- neutron-api:neutron-plugin-api
- - glance:shared-db
- mysql:shared-db
- - glance:amqp
- rabbitmq-server:amqp
- - nova-cloud-controller:image-service
- glance:image-service
- - nova-compute:image-service
- glance:image-service
- - nova-cloud-controller:cloud-compute
- nova-compute:cloud-compute
- - nova-cloud-controller:amqp
- rabbitmq-server:amqp
- - nova-cloud-controller:quantum-network-service
- neutron-gateway:quantum-network-service
- - nova-compute:neutron-plugin
- neutron-openvswitch:neutron-plugin
- - neutron-openvswitch:amqp
- rabbitmq-server:amqp
- - openstack-dashboard:identity-service
- keystone:identity-service
- - nova-cloud-controller:shared-db
- mysql:shared-db
- - nova-cloud-controller:neutron-api
- neutron-api:neutron-api
- - cinder:image-service
- glance:image-service
- - cinder:amqp
- rabbitmq-server:amqp
- - cinder:identity-service
- keystone:identity-service
- - cinder:cinder-volume-service
- nova-cloud-controller:cinder-volume-service
- - cinder-ceph:storage-backend
- cinder:storage-backend
- - ceph:client
- nova-compute:ceph
- - cinder:shared-db
- mysql:shared-db
- - ceph:client
- cinder-ceph:ceph
- - ceph:client
- glance:ceph
- - ceph-osd:mon
- ceph:osd
- - ntp:juju-info
- nova-compute:juju-info
- - ntp:juju-info
- neutron-gateway:juju-info
- - ceilometer:amqp
- rabbitmq-server:amqp
- - ceilometer-agent:ceilometer-service
- ceilometer:ceilometer-service
- - ceilometer:identity-service
- keystone:identity-service
- - ceilometer:identity-notifications
- keystone:identity-notifications
- - ceilometer-agent:nova-ceilometer
- nova-compute:nova-ceilometer
- - ceilometer:shared-db
- mongodb:database
- - memcached
- nova-cloud-controller
- - hacluster-ceilometer
- ceilometer
- - hacluster-cinder
- cinder
- - hacluster-glance
- glance
- - hacluster-keystone
- keystone
#- - hacluster-mysql
# - mysql
- - hacluster-neutron-api
- neutron-api
- - hacluster-nova-cloud-controller
- nova-cloud-controller
- - hacluster-openstack-dashboard
- openstack-dashboard
series: trusty
services:
ceilometer:
annotations:
gui-x: '1288.8744298356794'
gui-y: '0.7040786325134718'
charm: cs:trusty/ceilometer
num_units: 3
options:
vip: 10.0.84.19
openstack-origin: cloud:trusty-liberty
to:
- lxc:2
- lxc:3
- lxc:4
ceilometer-agent:
annotations:
gui-x: '1288.9999389648438'
gui-y: '503'
charm: cs:trusty/ceilometer-agent
ceph:
annotations:
gui-x: '750'
gui-y: '500'
charm: cs:trusty/ceph
num_units: 3
options:
fsid: cbd8508e-d726-4785-bff9-2fbf4af2df61
monitor-secret: AQDxeg9XUMRsKRAAyfClczi5hEV/3j0CuIN8dA==
osd-devices: '/dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:0:0
/dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:1:0
/dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:2:0
/dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:3:0
/dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:4:0
/dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:5:0
/dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:6:0
/dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:7:0'
osd-reformat: 'yes'
source: cloud:trusty-liberty
to:
- '1'
- '2'
- '3'
ceph-osd:
annotations:
gui-x: '1000'
gui-y: '500'
charm: cs:trusty/ceph-osd
num_units: 2
options:
osd-devices: '/dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:0:0
/dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:1:0
/dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:2:0
/dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:3:0
/dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:4:0
/dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:5:0
/dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:6:0
/dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:7:0'
osd-reformat: 'yes'
source: cloud:trusty-liberty
to:
- '4'
- '5'
cinder:
annotations:
gui-x: '750'
gui-y: '0'
charm: cs:trusty/cinder
num_units: 3
options:
vip: 10.0.84.15
block-device: None
glance-api-version: 2
openstack-origin: cloud:trusty-liberty
to:
- lxc:4
- lxc:5
- lxc:1
cinder-ceph:
annotations:
gui-x: '750'
gui-y: '250'
charm: cs:trusty/cinder-ceph
num_units: 0
glance:
annotations:
gui-x: '250'
gui-y: '0'
charm: cs:trusty/glance
num_units: 3
options:
vip: 10.0.84.14
openstack-origin: cloud:trusty-liberty
to:
- lxc:5
- lxc:1
- lxc:2
keystone:
annotations:
gui-x: '500'
gui-y: '0'
charm: cs:trusty/keystone
num_units: 3
options:
vip: 10.0.84.13
admin-password: openstack
openstack-origin: cloud:trusty-liberty
to:
- lxc:1
- lxc:2
- lxc:3
mongodb:
annotations:
gui-x: '1287.9999389648438'
gui-y: '251.24996948242188'
charm: cs:trusty/mongodb
num_units: 3
to:
- lxc:2
- lxc:3
- lxc:4
mysql:
annotations:
gui-x: '0'
gui-y: '250'
#charm: cs:trusty/percona-cluster
charm: cs:trusty/mysql
num_units: 1
options:
#vip: 10.0.84.12
max-connections: 20000
#innodb-buffer-pool-size: 4G
dataset-size: 4G
#root-password: NHYEGOYnHg9qJ2ByutPrI5P5UmQQklaG
#sst-password: c78alhN2tG6hpQ46dh1T6rHV19nax86M
#min-cluster-size: 3
#lp1366997-workaround: true
to:
- lxc:1
neutron-api:
annotations:
gui-x: '500'
gui-y: '500'
charm: cs:trusty/neutron-api
num_units: 3
options:
vip: 10.0.84.17
neutron-security-groups: true
openstack-origin: cloud:trusty-liberty
dhcp-agents-per-network: 2
to:
- lxc:4
- lxc:5
- lxc:1
neutron-gateway:
annotations:
gui-x: '500'
gui-y: '500'
charm: cs:trusty/neutron-gateway
num_units: 1
options:
ext-port: eth1
openstack-origin: cloud:trusty-liberty
instance-mtu: 1400
to:
- 6
neutron-openvswitch:
annotations:
gui-x: '250'
gui-y: '500'
charm: cs:trusty/neutron-openvswitch
num_units: 0
options:
enable-local-dhcp-and-metadata: true
nova-cloud-controller:
annotations:
gui-x: '0'
gui-y: '500'
charm: cs:trusty/nova-cloud-controller
num_units: 3
options:
vip: 10.0.84.16
network-manager: Neutron
openstack-origin: cloud:trusty-liberty
# console-access-protocol: novnc
console-access-protocol: None
ram-allocation-ratio: 1.0
single-nova-consoleauth: false
to:
- lxc:5
- lxc:1
- lxc:2
memcached:
charm: cs:trusty/memcached
num_units: 3
options:
allow-ufw-ip6-softfail: true
to:
- lxc:1
- lxc:2
- lxc:3
nova-compute:
annotations:
gui-x: '250'
gui-y: '250'
charm: cs:trusty/nova-compute
num_units: 5
options:
enable-live-migration: true
enable-resize: true
manage-neutron-plugin-legacy-mode: false
migration-auth-type: ssh
openstack-origin: cloud:trusty-liberty
to:
- '1'
- '2'
- '3'
- '4'
- '5'
ntp:
annotations:
gui-x: '1000'
gui-y: '0'
charm: cs:trusty/ntp
num_units: 0
openstack-dashboard:
annotations:
gui-x: '500'
gui-y: '-250'
charm: cs:trusty/openstack-dashboard
num_units: 3
options:
vip: 10.0.84.11
openstack-origin: cloud:trusty-liberty
webroot: /
secret: OPWEY1P8wN029oTH4ZhWGy5JCcnMBn0I
to:
- lxc:2
- lxc:3
- lxc:4
rabbitmq-server:
annotations:
gui-x: '500'
gui-y: '250'
charm: cs:trusty/rabbitmq-server
num_units: 3
options:
source: cloud:trusty-liberty
cluster-partition-handling: pause_minority
min-cluster-size: 3
to:
- lxc:3
- lxc:4
- lxc:5
hacluster-ceilometer:
charm: cs:trusty/hacluster
num_units: 0
options:
corosync_key:
'1j6nBSeLpNIYz0yUqALCFWDcivmWxCyp/reUuzfeQrsZJIUlyb7i1krQolucBCuTN37/vmWT/bC5qqBIE+17JKcoo/RfkOtSXF5DK6bXCkvUoLHQBRTKL755Yycp7HiZKea7R29oGnqml3A3vpMcrEVyjT7Ax6Nf+oZ2GwpGYi8='
cluster_count: 3
corosync_transport: unicast
hacluster-cinder:
charm: cs:trusty/hacluster
num_units: 0
options:
corosync_key:
'1j6nBSeLpNIYz0yUqALCFWDcivmWxCyp/reUuzfeQrsZJIUlyb7i1krQolucBCuTN37/vmWT/bC5qqBIE+17JKcoo/RfkOtSXF5DK6bXCkvUoLHQBRTKL755Yycp7HiZKea7R29oGnqml3A3vpMcrEVyjT7Ax6Nf+oZ2GwpGYi8='
cluster_count: 3
corosync_transport: unicast
hacluster-glance:
charm: cs:trusty/hacluster
num_units: 0
options:
corosync_key:
'1j6nBSeLpNIYz0yUqALCFWDcivmWxCyp/reUuzfeQrsZJIUlyb7i1krQolucBCuTN37/vmWT/bC5qqBIE+17JKcoo/RfkOtSXF5DK6bXCkvUoLHQBRTKL755Yycp7HiZKea7R29oGnqml3A3vpMcrEVyjT7Ax6Nf+oZ2GwpGYi8='
cluster_count: 3
corosync_transport: unicast
hacluster-keystone:
charm: cs:trusty/hacluster
num_units: 0
options:
corosync_key:
'1j6nBSeLpNIYz0yUqALCFWDcivmWxCyp/reUuzfeQrsZJIUlyb7i1krQolucBCuTN37/vmWT/bC5qqBIE+17JKcoo/RfkOtSXF5DK6bXCkvUoLHQBRTKL755Yycp7HiZKea7R29oGnqml3A3vpMcrEVyjT7Ax6Nf+oZ2GwpGYi8='
cluster_count: 3
corosync_transport: unicast
# hacluster-mysql:
# charm: cs:trusty/hacluster
# num_units: 0
# options:
# corosync_key:
'1j6nBSeLpNIYz0yUqALCFWDcivmWxCyp/reUuzfeQrsZJIUlyb7i1krQolucBCuTN37/vmWT/bC5qqBIE+17JKcoo/RfkOtSXF5DK6bXCkvUoLHQBRTKL755Yycp7HiZKea7R29oGnqml3A3vpMcrEVyjT7Ax6Nf+oZ2GwpGYi8='
# cluster_count: 3
# corosync_transport: unicast
hacluster-neutron-api:
charm: cs:trusty/hacluster
num_units: 0
options:
corosync_key:
'1j6nBSeLpNIYz0yUqALCFWDcivmWxCyp/reUuzfeQrsZJIUlyb7i1krQolucBCuTN37/vmWT/bC5qqBIE+17JKcoo/RfkOtSXF5DK6bXCkvUoLHQBRTKL755Yycp7HiZKea7R29oGnqml3A3vpMcrEVyjT7Ax6Nf+oZ2GwpGYi8='
cluster_count: 3
corosync_transport: unicast
hacluster-nova-cloud-controller:
charm: cs:trusty/hacluster
num_units: 0
options:
corosync_key:
'1j6nBSeLpNIYz0yUqALCFWDcivmWxCyp/reUuzfeQrsZJIUlyb7i1krQolucBCuTN37/vmWT/bC5qqBIE+17JKcoo/RfkOtSXF5DK6bXCkvUoLHQBRTKL755Yycp7HiZKea7R29oGnqml3A3vpMcrEVyjT7Ax6Nf+oZ2GwpGYi8='
cluster_count: 3
corosync_transport: unicast
hacluster-openstack-dashboard:
charm: cs:trusty/hacluster
num_units: 0
options:
corosync_key:
'1j6nBSeLpNIYz0yUqALCFWDcivmWxCyp/reUuzfeQrsZJIUlyb7i1krQolucBCuTN37/vmWT/bC5qqBIE+17JKcoo/RfkOtSXF5DK6bXCkvUoLHQBRTKL755Yycp7HiZKea7R29oGnqml3A3vpMcrEVyjT7Ax6Nf+oZ2GwpGYi8='
cluster_count: 3
corosync_transport: unicast
** Affects: horizon (Ubuntu)
Importance: Undecided
Status: New
** Attachment added: "boot log attached"
https://bugs.launchpad.net/bugs/1578471/+attachment/4656072/+files/bootlog.txt
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1578471
Title:
Ubuntu Openstack HA becomes unusable with a partially failed node
OpenPower
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/horizon/+bug/1578471/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
