Installed another Xenial and Bionic in vmware to take a deper look. - Xenial (with backported open-vm-tools): affected - Bionic (with the interim fix reverted): no hit in several retries, explanation below
Systemd fixed it (via our assumed implicit dependency). In Bionic the PrivateTmp gives it a dependency on systemd-tmpfile-setup.service (seen in systemd analyze, there might be more but not on crit path). This is configured by default to include /var/tmp in /usr/lib/tmpfiles.d/tmp.conf. In regard to your thoughts about later on changing cloud-init ordering that won't help you, as the dependency is there (implicit or explicit doesn't matter). For the xenial case where I reliably hit the issue instead of stracing I cut things short. A service with the following exposes exactly the same error: [Unit] Description=foo DefaultDependencies=no [Service] PrivateTmp=yes ExecStart=/bin/true [Install] WantedBy=multi-user.target So back on Xenial it is privateTmp + too early that breaks it. Xenial vs Bionic critical-chain according to "systemd-analyze critical- chain open.vm-tools.service" Xenial with fix: open-vm-tools.service @3.482s └─local-fs.target @3.460s └─local-fs-pre.target @3.460s └─systemd-remount-fs.service @3.442s +9ms └─system.slice @220ms └─-.slice @204m Xenial without fix: └─run-vmblock\x2dfuse.mount @6.076s +390ms └─sys-fs-fuse-connections.mount @5.510s +375ms └─systemd-modules-load.service @1.996s +75ms └─system.slice @1.984s └─-.slice @1.966s Bionic open-vm-tools.service @3.566s └─systemd-tmpfiles-setup.service @3.421s +100ms └─systemd-journal-flush.service @3.054s +342ms └─systemd-journald.service @825ms +2.219s └─syslog.socket @808ms └─system.slice @621ms └─-.slice @613ms To Summarize, we can: - revert the fix for Bionic (or later) - just make it a sync when convenient down the road, it doesn't hurt for now as it is (almost) the same as the implicit dependency) - add a xenials systemd bug task (probably too complex to fix as -upstream) - until said systemd bug is fixed a backport of open-vm-tools needs this fix ** Also affects: systemd (Ubuntu) Importance: Undecided Status: New ** Also affects: open-vm-tools (Ubuntu Xenial) Importance: Undecided Status: New ** Also affects: systemd (Ubuntu Xenial) Importance: Undecided Status: New ** Changed in: open-vm-tools (Ubuntu Xenial) Status: New => Triaged ** Changed in: systemd (Ubuntu) Status: New => Fix Released -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1750780 Title: Race with local file systems can make open-vm-tools fail to start Status in cloud-init: Invalid Status in open-vm-tools package in Ubuntu: Fix Released Status in systemd package in Ubuntu: Fix Released Status in open-vm-tools source package in Xenial: Triaged Status in systemd source package in Xenial: New Status in open-vm-tools package in Debian: Incomplete Bug description: Since the change in [1] open-vm-tools-service starts very (very) early. Not so much due to the Before=cloud-init-local.service But much more by DefaultDependencies=no That can trigger an issue that looks like root@ubuntuguest:~# systemctl status -l open-vm-tools.service ● open-vm-tools.service - Service for virtual machines hosted on VMware Loaded: loaded (/lib/systemd/system/open-vm-tools.service; enabled; vendor preset: enabled) Active: failed (Result: resources) As it is right now open-vm-tools can race with the other early start and then fail. In detail one can find a message like: open-vm-tools.service: Failed to run 'start' task: Read-only file system" This is due to privtaeTmp=yes which is also set needing a writable /var/tmp [2] To ensure this works PrivateTmp would have to be removed (not good) or some after dependencies added that make this work reliably. I added After=local-fs.target which made it work for me in 3/3 tests. I' like to have an ack by the cloud-init Team that this does not totally kill the originally intended Before=cloud-init-local.service I think it does not as local-fs can complete before cloud-init-local, then open-vm-tools can initialize and finally cloud-init-local can pick up the data. To summarize: # cloud-init-local # DefaultDependencies=no Wants=network-pre.target After=systemd-remount-fs.service Before=NetworkManager.service Before=network-pre.target Before=shutdown.target Before=sysinit.target Conflicts=shutdown.target RequiresMountsFor=/var/lib/cloud # open-vm-tools # DefaultDependencies=no Before=cloud-init-local.service Proposed is to add to the latter: After=local-fs.target [1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859677 [2]: https://github.com/systemd/systemd/issues/5610 To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1750780/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp