Public bug reported:
We have a Ubuntu Openstack HA cluster running on 5 OpenPower nodes
(POWER8).
This is on 14.04 with Openstack Liberty
It was all working just peachy, when node 5 starting having issues and became
inaccessible.
It seems the file system on the local disk (ext4) has some errors and requires
a fsck/e2fsck.
The system does booot, however, the erros in the filesystem seems to have
affected the SSH keys so I cannot get into the system. The juju agent does not
start.
This is related to bug 1578471.
This bug is to determine if the corrupt filesystem was just from a failing
disk, or if the software is somehow in fault.
As this is an Openstack HA environment, this node is running containers with:
Cinder
glance
horizon
neutron
rabbitmq
ceph-osd and nova are running on the host OS itself.
I've not attempted to "fix" this node yet, by running fsck somehow, or
just doing a redeploy. I can do either of those, if it's recommended.
bundle.yaml
==========
machines:
'0':
constraints: arch=ppc64el
series: trusty
'1':
constraints: { arch=ppc64el, tags=compute }
series: trusty
'2':
constraints: { arch=ppc64el, tags=compute }
series: trusty
'3':
constraints: { arch=ppc64el, tags=compute }
series: trusty
'4':
constraints: { arch=ppc64el, tags=compute }
series: trusty
'5':
constraints: { arch=ppc64el, tags=compute }
series: trusty
'6':
constraints: { arch=ppc64el, tags=neutron-gateway }
series: trusty
relations:
- - nova-compute:amqp
- rabbitmq-server:amqp
- - neutron-gateway:amqp
- rabbitmq-server:amqp
- - keystone:shared-db
- mysql:shared-db
- - nova-cloud-controller:identity-service
- keystone:identity-service
- - glance:identity-service
- keystone:identity-service
- - neutron-api:identity-service
- keystone:identity-service
- - neutron-openvswitch:neutron-plugin-api
- neutron-api:neutron-plugin-api
- - neutron-api:shared-db
- mysql:shared-db
- - neutron-api:amqp
- rabbitmq-server:amqp
- - neutron-gateway:neutron-plugin-api
- neutron-api:neutron-plugin-api
- - glance:shared-db
- mysql:shared-db
- - glance:amqp
- rabbitmq-server:amqp
- - nova-cloud-controller:image-service
- glance:image-service
- - nova-compute:image-service
- glance:image-service
- - nova-cloud-controller:cloud-compute
- nova-compute:cloud-compute
- - nova-cloud-controller:amqp
- rabbitmq-server:amqp
- - nova-cloud-controller:quantum-network-service
- neutron-gateway:quantum-network-service
- - nova-compute:neutron-plugin
- neutron-openvswitch:neutron-plugin
- - neutron-openvswitch:amqp
- rabbitmq-server:amqp
- - openstack-dashboard:identity-service
- keystone:identity-service
- - nova-cloud-controller:shared-db
- mysql:shared-db
- - nova-cloud-controller:neutron-api
- neutron-api:neutron-api
- - cinder:image-service
- glance:image-service
- - cinder:amqp
- rabbitmq-server:amqp
- - cinder:identity-service
- keystone:identity-service
- - cinder:cinder-volume-service
- nova-cloud-controller:cinder-volume-service
- - cinder-ceph:storage-backend
- cinder:storage-backend
- - ceph:client
- nova-compute:ceph
- - cinder:shared-db
- mysql:shared-db
- - ceph:client
- cinder-ceph:ceph
- - ceph:client
- glance:ceph
- - ceph-osd:mon
- ceph:osd
- - ntp:juju-info
- nova-compute:juju-info
- - ntp:juju-info
- neutron-gateway:juju-info
- - ceilometer:amqp
- rabbitmq-server:amqp
- - ceilometer-agent:ceilometer-service
- ceilometer:ceilometer-service
- - ceilometer:identity-service
- keystone:identity-service
- - ceilometer:identity-notifications
- keystone:identity-notifications
- - ceilometer-agent:nova-ceilometer
- nova-compute:nova-ceilometer
- - ceilometer:shared-db
- mongodb:database
- - memcached
- nova-cloud-controller
- - hacluster-ceilometer
- ceilometer
- - hacluster-cinder
- cinder
- - hacluster-glance
- glance
- - hacluster-keystone
- keystone
#- - hacluster-mysql
# - mysql
- - hacluster-neutron-api
- neutron-api
- - hacluster-nova-cloud-controller
- nova-cloud-controller
- - hacluster-openstack-dashboard
- openstack-dashboard
series: trusty
services:
ceilometer:
annotations:
gui-x: '1288.8744298356794'
gui-y: '0.7040786325134718'
charm: cs:trusty/ceilometer
num_units: 3
options:
vip: 10.0.84.19
openstack-origin: cloud:trusty-liberty
to:
- lxc:2
- lxc:3
- lxc:4
ceilometer-agent:
annotations:
gui-x: '1288.9999389648438'
gui-y: '503'
charm: cs:trusty/ceilometer-agent
ceph:
annotations:
gui-x: '750'
gui-y: '500'
charm: cs:trusty/ceph
num_units: 3
options:
fsid: cbd8508e-d726-4785-bff9-2fbf4af2df61
monitor-secret: AQDxeg9XUMRsKRAAyfClczi5hEV/3j0CuIN8dA==
osd-devices: '/dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:0:0
/dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:1:0
/dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:2:0
/dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:3:0
/dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:4:0
/dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:5:0
/dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:6:0
/dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:7:0'
osd-reformat: 'yes'
source: cloud:trusty-liberty
to:
- '1'
- '2'
- '3'
ceph-osd:
annotations:
gui-x: '1000'
gui-y: '500'
charm: cs:trusty/ceph-osd
num_units: 2
options:
osd-devices: '/dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:0:0
/dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:1:0
/dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:2:0
/dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:3:0
/dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:4:0
/dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:5:0
/dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:6:0
/dev/disk/by-path/pci-0002:01:00.0-scsi-0:1:7:0'
osd-reformat: 'yes'
source: cloud:trusty-liberty
to:
- '4'
- '5'
cinder:
annotations:
gui-x: '750'
gui-y: '0'
charm: cs:trusty/cinder
num_units: 3
options:
vip: 10.0.84.15
block-device: None
glance-api-version: 2
openstack-origin: cloud:trusty-liberty
to:
- lxc:4
- lxc:5
- lxc:1
cinder-ceph:
annotations:
gui-x: '750'
gui-y: '250'
charm: cs:trusty/cinder-ceph
num_units: 0
glance:
annotations:
gui-x: '250'
gui-y: '0'
charm: cs:trusty/glance
num_units: 3
options:
vip: 10.0.84.14
openstack-origin: cloud:trusty-liberty
to:
- lxc:5
- lxc:1
- lxc:2
keystone:
annotations:
gui-x: '500'
gui-y: '0'
charm: cs:trusty/keystone
num_units: 3
options:
vip: 10.0.84.13
admin-password: openstack
openstack-origin: cloud:trusty-liberty
to:
- lxc:1
- lxc:2
- lxc:3
mongodb:
annotations:
gui-x: '1287.9999389648438'
gui-y: '251.24996948242188'
charm: cs:trusty/mongodb
num_units: 3
to:
- lxc:2
- lxc:3
- lxc:4
mysql:
annotations:
gui-x: '0'
gui-y: '250'
#charm: cs:trusty/percona-cluster
charm: cs:trusty/mysql
num_units: 1
options:
#vip: 10.0.84.12
max-connections: 20000
#innodb-buffer-pool-size: 4G
dataset-size: 4G
#root-password: NHYEGOYnHg9qJ2ByutPrI5P5UmQQklaG
#sst-password: c78alhN2tG6hpQ46dh1T6rHV19nax86M
#min-cluster-size: 3
#lp1366997-workaround: true
to:
- lxc:1
neutron-api:
annotations:
gui-x: '500'
gui-y: '500'
charm: cs:trusty/neutron-api
num_units: 3
options:
vip: 10.0.84.17
neutron-security-groups: true
openstack-origin: cloud:trusty-liberty
dhcp-agents-per-network: 2
to:
- lxc:4
- lxc:5
- lxc:1
neutron-gateway:
annotations:
gui-x: '500'
gui-y: '500'
charm: cs:trusty/neutron-gateway
num_units: 1
options:
ext-port: eth1
openstack-origin: cloud:trusty-liberty
instance-mtu: 1400
to:
- 6
neutron-openvswitch:
annotations:
gui-x: '250'
gui-y: '500'
charm: cs:trusty/neutron-openvswitch
num_units: 0
options:
enable-local-dhcp-and-metadata: true
nova-cloud-controller:
annotations:
gui-x: '0'
gui-y: '500'
charm: cs:trusty/nova-cloud-controller
num_units: 3
options:
vip: 10.0.84.16
network-manager: Neutron
openstack-origin: cloud:trusty-liberty
# console-access-protocol: novnc
console-access-protocol: None
ram-allocation-ratio: 1.0
single-nova-consoleauth: false
to:
- lxc:5
- lxc:1
- lxc:2
memcached:
charm: cs:trusty/memcached
num_units: 3
options:
allow-ufw-ip6-softfail: true
to:
- lxc:1
- lxc:2
- lxc:3
nova-compute:
annotations:
gui-x: '250'
gui-y: '250'
charm: cs:trusty/nova-compute
num_units: 5
options:
enable-live-migration: true
enable-resize: true
manage-neutron-plugin-legacy-mode: false
migration-auth-type: ssh
openstack-origin: cloud:trusty-liberty
to:
- '1'
- '2'
- '3'
- '4'
- '5'
ntp:
annotations:
gui-x: '1000'
gui-y: '0'
charm: cs:trusty/ntp
num_units: 0
openstack-dashboard:
annotations:
gui-x: '500'
gui-y: '-250'
charm: cs:trusty/openstack-dashboard
num_units: 3
options:
vip: 10.0.84.11
openstack-origin: cloud:trusty-liberty
webroot: /
secret: OPWEY1P8wN029oTH4ZhWGy5JCcnMBn0I
to:
- lxc:2
- lxc:3
- lxc:4
rabbitmq-server:
annotations:
gui-x: '500'
gui-y: '250'
charm: cs:trusty/rabbitmq-server
num_units: 3
options:
source: cloud:trusty-liberty
cluster-partition-handling: pause_minority
min-cluster-size: 3
to:
- lxc:3
- lxc:4
- lxc:5
hacluster-ceilometer:
charm: cs:trusty/hacluster
num_units: 0
options:
corosync_key:
'1j6nBSeLpNIYz0yUqALCFWDcivmWxCyp/reUuzfeQrsZJIUlyb7i1krQolucBCuTN37/vmWT/bC5qqBIE+17JKcoo/RfkOtSXF5DK6bXCkvUoLHQBRTKL755Yycp7HiZKea7R29oGnqml3A3vpMcrEVyjT7Ax6Nf+oZ2GwpGYi8='
cluster_count: 3
corosync_transport: unicast
hacluster-cinder:
charm: cs:trusty/hacluster
num_units: 0
options:
corosync_key:
'1j6nBSeLpNIYz0yUqALCFWDcivmWxCyp/reUuzfeQrsZJIUlyb7i1krQolucBCuTN37/vmWT/bC5qqBIE+17JKcoo/RfkOtSXF5DK6bXCkvUoLHQBRTKL755Yycp7HiZKea7R29oGnqml3A3vpMcrEVyjT7Ax6Nf+oZ2GwpGYi8='
cluster_count: 3
corosync_transport: unicast
hacluster-glance:
charm: cs:trusty/hacluster
num_units: 0
options:
corosync_key:
'1j6nBSeLpNIYz0yUqALCFWDcivmWxCyp/reUuzfeQrsZJIUlyb7i1krQolucBCuTN37/vmWT/bC5qqBIE+17JKcoo/RfkOtSXF5DK6bXCkvUoLHQBRTKL755Yycp7HiZKea7R29oGnqml3A3vpMcrEVyjT7Ax6Nf+oZ2GwpGYi8='
cluster_count: 3
corosync_transport: unicast
hacluster-keystone:
charm: cs:trusty/hacluster
num_units: 0
options:
corosync_key:
'1j6nBSeLpNIYz0yUqALCFWDcivmWxCyp/reUuzfeQrsZJIUlyb7i1krQolucBCuTN37/vmWT/bC5qqBIE+17JKcoo/RfkOtSXF5DK6bXCkvUoLHQBRTKL755Yycp7HiZKea7R29oGnqml3A3vpMcrEVyjT7Ax6Nf+oZ2GwpGYi8='
cluster_count: 3
corosync_transport: unicast
# hacluster-mysql:
# charm: cs:trusty/hacluster
# num_units: 0
# options:
# corosync_key:
'1j6nBSeLpNIYz0yUqALCFWDcivmWxCyp/reUuzfeQrsZJIUlyb7i1krQolucBCuTN37/vmWT/bC5qqBIE+17JKcoo/RfkOtSXF5DK6bXCkvUoLHQBRTKL755Yycp7HiZKea7R29oGnqml3A3vpMcrEVyjT7Ax6Nf+oZ2GwpGYi8='
# cluster_count: 3
# corosync_transport: unicast
hacluster-neutron-api:
charm: cs:trusty/hacluster
num_units: 0
options:
corosync_key:
'1j6nBSeLpNIYz0yUqALCFWDcivmWxCyp/reUuzfeQrsZJIUlyb7i1krQolucBCuTN37/vmWT/bC5qqBIE+17JKcoo/RfkOtSXF5DK6bXCkvUoLHQBRTKL755Yycp7HiZKea7R29oGnqml3A3vpMcrEVyjT7Ax6Nf+oZ2GwpGYi8='
cluster_count: 3
corosync_transport: unicast
hacluster-nova-cloud-controller:
charm: cs:trusty/hacluster
num_units: 0
options:
corosync_key:
'1j6nBSeLpNIYz0yUqALCFWDcivmWxCyp/reUuzfeQrsZJIUlyb7i1krQolucBCuTN37/vmWT/bC5qqBIE+17JKcoo/RfkOtSXF5DK6bXCkvUoLHQBRTKL755Yycp7HiZKea7R29oGnqml3A3vpMcrEVyjT7Ax6Nf+oZ2GwpGYi8='
cluster_count: 3
corosync_transport: unicast
hacluster-openstack-dashboard:
charm: cs:trusty/hacluster
num_units: 0
options:
corosync_key:
'1j6nBSeLpNIYz0yUqALCFWDcivmWxCyp/reUuzfeQrsZJIUlyb7i1krQolucBCuTN37/vmWT/bC5qqBIE+17JKcoo/RfkOtSXF5DK6bXCkvUoLHQBRTKL755Yycp7HiZKea7R29oGnqml3A3vpMcrEVyjT7Ax6Nf+oZ2GwpGYi8='
cluster_count: 3
corosync_transport: unicast
** Affects: ubuntu
Importance: Undecided
Status: New
** Attachment added: "bootlog attached"
https://bugs.launchpad.net/bugs/1578475/+attachment/4656077/+files/bootlog.txt
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1578475
Title:
An OpenPower node within an Ubuntu Openstack HA cluster has it's base
ext4 filesystem corrupted
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+bug/1578475/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
