So now I ignored all log entries that are Nov 4th or before - well, there are actually only a few logs that have log entries with timestamp Nov 5th or later.
The interesting ones (with Nov 5 ff entries) are syslog and cloud-init - auth has Nov 5 entries, too but is not really relevant. cloud-init- output also has Nov 5 entries but is fine. First of all I couldn't find a cloud-init issue with timestamp Nov 5th or later. The following block tells me that cloud-init was able to find a config drive and was able to read it (even if it didn't had a user_data entry - which is fine): 2019-11-06 06:04:27,346 - util.py[DEBUG]: Reading from /run/cloud-init/tmp/tmpswvqg6v_/openstack/2018-08-27/network_data.json (quiet=False) 2019-11-06 06:04:27,347 - util.py[DEBUG]: Read 463 bytes from /run/cloud-init/tmp/tmpswvqg6v_/openstack/2018-08-27/network_data.json 2019-11-06 06:04:27,347 - util.py[DEBUG]: Reading from /run/cloud-init/tmp/tmpswvqg6v_/openstack/2018-08-27/vendor_data.json (quiet=False) 2019-11-06 06:04:27,347 - util.py[DEBUG]: Read 2 bytes from /run/cloud-init/tmp/tmpswvqg6v_/openstack/2018-08-27/vendor_data.json 2019-11-06 06:04:27,347 - util.py[DEBUG]: Reading from /run/cloud-init/tmp/tmpswvqg6v_/openstack/2018-08-27/meta_data.json (quiet=False) 2019-11-06 06:04:27,348 - util.py[DEBUG]: Read 1054 bytes from /run/cloud-init/tmp/tmpswvqg6v_/openstack/2018-08-27/meta_data.json 2019-11-06 06:04:27,348 - util.py[DEBUG]: Reading from /run/cloud-init/tmp/tmpswvqg6v_/openstack/2018-08-27/user_data (quiet=False) 2019-11-06 06:04:27,348 - openstack.py[DEBUG]: Failed reading optional path /run/cloud-init/tmp/tmpswvqg6v_/openstack/2018-08-27/user_data due to: [Errno 2] No such file or directory: '/run/cloud-init/tmp/tmpswvqg6v_/openstack/2018-08-27/user_data' 2019-11-06 06:04:27,348 - util.py[DEBUG]: Reading from /run/cloud-init/tmp/tmpswvqg6v_/openstack/content/0000 (quiet=False) 2019-11-06 06:04:27,349 - util.py[DEBUG]: Read 606 bytes from /run/cloud-init/tmp/tmpswvqg6v_/openstack/content/0000 2019-11-06 06:04:27,349 - util.py[DEBUG]: Reading from /run/cloud-init/tmp/tmpswvqg6v_/ec2/latest/meta-data.json (quiet=False) 2019-11-06 06:04:27,350 - util.py[DEBUG]: Read 581 bytes from /run/cloud-init/tmp/tmpswvqg6v_/ec2/latest/meta-data.json 2019-11-06 06:04:27,350 - util.py[DEBUG]: Running command ['umount', '/run/cloud-init/tmp/tmpswvqg6v_'] with allowed return codes [0] (shell=False, capture=True) 2019-11-06 06:04:27,364 - util.py[DEBUG]: Reading from /var/lib/cloud/data/instance-id (quiet=False) 2019-11-06 06:04:27,371 - util.py[DEBUG]: Read 20 bytes from /var/lib/cloud/data/instance-id 2019-11-06 06:04:27,373 - atomic_helper.py[DEBUG]: Atomically writing to file /run/cloud-init/instance-data.json (via temporary file /run/cloud-init/tmpuulu3ae2) - w: [644] 3272 bytes/chars 2019-11-06 06:04:27,373 - atomic_helper.py[DEBUG]: Atomically writing to file /run/cloud-init/instance-data-sensitive.json (via temporary file /run/cloud-init/tmpbft3zc1b) - w: [600] 3272 bytes/chars 2019-11-06 06:04:27,373 - handlers.py[DEBUG]: finish: init-local/search-ConfigDrive: SUCCESS: found local data from DataSourceConfigDrive Then I had a look at syslog and came across some of these entires: Nov 5 01:14:36 ComputeAutomation3 RMCdaemon[19028]: (Recorded using libct_ffdc.a cv 2):::Error ID: 824....Q3GkR/XAT/7wGGi....................:::Reference ID: :::Template ID: 0:::Details File: :::Location: RSCT,rmcd.c,1.137.1.1,1203 :::RMCD_INFO_1_ST#012The daemon is stopped.#012Number of command that stopped the daemon#0123 Nov 5 01:14:40 ComputeAutomation3 RMCdaemon[4407]: (Recorded using libct_ffdc.a cv 2):::Error ID: 824....U3GkR/dlU/7wGGi....................:::Reference ID: :::Template ID: 0:::Details File: :::Location: RSCT,rmcd.c,1.137.1.1,255 :::RMCD_INFO_0_ST#012The daemon is started. This tells me that there is an issue with the IBM RCST layer "IBM Reliable Scaleable Cluster Technology" that is used for example in Tivoli System Automation (TSM) or GPFS. This error can lead to severe problems (shutdown, reboots, etc.) like indicated here: https://access.redhat.com/solutions/894513 https://access.redhat.com/solutions/664783 Since there are no other issues found in the logs (again after Nov the 5th), I think that this is causing the problems on your system. Please solve this first - maybe by involving the support team of the product that you are using (that provides rsct) - or recreate the system w/o rsct to make sure that this can be excluded. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1852057 Title: The system is moving to emergency mode while booting up. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1852057/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
