[Touch-packages] [Bug 1750780] Re: Race with local file systems can make open-vm-tools fail to start
Tracked in Github Issues as https://github.com/canonical/cloud- init/issues/3122 ** Bug watch added: github.com/canonical/cloud-init/issues #3122 https://github.com/canonical/cloud-init/issues/3122 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1750780 Title: Race with local file systems can make open-vm-tools fail to start Status in cloud-init: Invalid Status in open-vm-tools package in Ubuntu: Invalid Status in systemd package in Ubuntu: Fix Released Status in open-vm-tools source package in Xenial: Fix Released Status in systemd source package in Xenial: Incomplete Status in open-vm-tools source package in Artful: Fix Released Status in open-vm-tools package in Debian: Fix Released Bug description: Since the change in [1] open-vm-tools-service starts very (very) early. Not so much due to the Before=cloud-init-local.service But much more by DefaultDependencies=no That can trigger an issue that looks like root@ubuntuguest:~# systemctl status -l open-vm-tools.service ● open-vm-tools.service - Service for virtual machines hosted on VMware Loaded: loaded (/lib/systemd/system/open-vm-tools.service; enabled; vendor preset: enabled) Active: failed (Result: resources) As it is right now open-vm-tools can race with the other early start and then fail. In detail one can find a message like: open-vm-tools.service: Failed to run 'start' task: Read-only file system" This is due to privtaeTmp=yes which is also set needing a writable /var/tmp [2] To ensure this works PrivateTmp would have to be removed (not good) or some after dependencies added that make this work reliably. I added After=local-fs.target which made it work for me in 3/3 tests. I' like to have an ack by the cloud-init Team that this does not totally kill the originally intended Before=cloud-init-local.service I think it does not as local-fs can complete before cloud-init-local, then open-vm-tools can initialize and finally cloud-init-local can pick up the data. To summarize: # cloud-init-local # DefaultDependencies=no Wants=network-pre.target After=systemd-remount-fs.service Before=NetworkManager.service Before=network-pre.target Before=shutdown.target Before=sysinit.target Conflicts=shutdown.target RequiresMountsFor=/var/lib/cloud # open-vm-tools # DefaultDependencies=no Before=cloud-init-local.service Proposed is to add to the latter: After=local-fs.target [1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859677 [2]: https://github.com/systemd/systemd/issues/5610 To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1750780/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1750780] Re: Race with local file systems can make open-vm-tools fail to start
No, unless you were afraid that what ticked open-vm-tools into having issues here being a wider more general issue. But then given that 16.04 is used widely for quite some time I think we can assume it is a corner case only. incomplete on the systemd task is ok. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1750780 Title: Race with local file systems can make open-vm-tools fail to start Status in cloud-init: Invalid Status in open-vm-tools package in Ubuntu: Invalid Status in systemd package in Ubuntu: Fix Released Status in open-vm-tools source package in Xenial: Fix Released Status in systemd source package in Xenial: Incomplete Status in open-vm-tools source package in Artful: Fix Released Status in open-vm-tools package in Debian: Fix Released Bug description: Since the change in [1] open-vm-tools-service starts very (very) early. Not so much due to the Before=cloud-init-local.service But much more by DefaultDependencies=no That can trigger an issue that looks like root@ubuntuguest:~# systemctl status -l open-vm-tools.service ● open-vm-tools.service - Service for virtual machines hosted on VMware Loaded: loaded (/lib/systemd/system/open-vm-tools.service; enabled; vendor preset: enabled) Active: failed (Result: resources) As it is right now open-vm-tools can race with the other early start and then fail. In detail one can find a message like: open-vm-tools.service: Failed to run 'start' task: Read-only file system" This is due to privtaeTmp=yes which is also set needing a writable /var/tmp [2] To ensure this works PrivateTmp would have to be removed (not good) or some after dependencies added that make this work reliably. I added After=local-fs.target which made it work for me in 3/3 tests. I' like to have an ack by the cloud-init Team that this does not totally kill the originally intended Before=cloud-init-local.service I think it does not as local-fs can complete before cloud-init-local, then open-vm-tools can initialize and finally cloud-init-local can pick up the data. To summarize: # cloud-init-local # DefaultDependencies=no Wants=network-pre.target After=systemd-remount-fs.service Before=NetworkManager.service Before=network-pre.target Before=shutdown.target Before=sysinit.target Conflicts=shutdown.target RequiresMountsFor=/var/lib/cloud # open-vm-tools # DefaultDependencies=no Before=cloud-init-local.service Proposed is to add to the latter: After=local-fs.target [1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859677 [2]: https://github.com/systemd/systemd/issues/5610 To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1750780/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1750780] Re: Race with local file systems can make open-vm-tools fail to start
So, is there anything requested to be done with Xenial's systemd in this bug? ** Changed in: systemd (Ubuntu Xenial) Status: New => Incomplete -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1750780 Title: Race with local file systems can make open-vm-tools fail to start Status in cloud-init: Invalid Status in open-vm-tools package in Ubuntu: Invalid Status in systemd package in Ubuntu: Fix Released Status in open-vm-tools source package in Xenial: Fix Released Status in systemd source package in Xenial: Incomplete Status in open-vm-tools source package in Artful: Fix Released Status in open-vm-tools package in Debian: Fix Released Bug description: Since the change in [1] open-vm-tools-service starts very (very) early. Not so much due to the Before=cloud-init-local.service But much more by DefaultDependencies=no That can trigger an issue that looks like root@ubuntuguest:~# systemctl status -l open-vm-tools.service ● open-vm-tools.service - Service for virtual machines hosted on VMware Loaded: loaded (/lib/systemd/system/open-vm-tools.service; enabled; vendor preset: enabled) Active: failed (Result: resources) As it is right now open-vm-tools can race with the other early start and then fail. In detail one can find a message like: open-vm-tools.service: Failed to run 'start' task: Read-only file system" This is due to privtaeTmp=yes which is also set needing a writable /var/tmp [2] To ensure this works PrivateTmp would have to be removed (not good) or some after dependencies added that make this work reliably. I added After=local-fs.target which made it work for me in 3/3 tests. I' like to have an ack by the cloud-init Team that this does not totally kill the originally intended Before=cloud-init-local.service I think it does not as local-fs can complete before cloud-init-local, then open-vm-tools can initialize and finally cloud-init-local can pick up the data. To summarize: # cloud-init-local # DefaultDependencies=no Wants=network-pre.target After=systemd-remount-fs.service Before=NetworkManager.service Before=network-pre.target Before=shutdown.target Before=sysinit.target Conflicts=shutdown.target RequiresMountsFor=/var/lib/cloud # open-vm-tools # DefaultDependencies=no Before=cloud-init-local.service Proposed is to add to the latter: After=local-fs.target [1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859677 [2]: https://github.com/systemd/systemd/issues/5610 To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1750780/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1750780] Re: Race with local file systems can make open-vm-tools fail to start
This bug was fixed in the package open-vm-tools - 2:10.2.0-3~ubuntu0.17.10.1 --- open-vm-tools (2:10.2.0-3~ubuntu0.17.10.1) artful; urgency=medium * backport Bionic open-vm-tools to Artful (LP: #1741390) - d/control: B-D for dh-autoreconf and dh-systemd - d/rules: re-add autoreconf and systemd - d/control: B-D to debhelper version of xenial supporting compat 10 - d/compat: compat level 10 is latest xenial supported level - d/rules: go back from override_dh_installsystemd to override_dh_systemd_enable and override_dh_systemd_start as needed on xenials debhelper version. - d/rules: drop --no-restart-after-upgrade which only exists in debhelper 10, the version 9 behavior defaults to what we need. - debian/rules: dh_systemd_start in xenial has issues with the escaping of the run-vmblock\\x2dfuse.mount job, so drop this call - debian/open-vm-tools-desktop.postinst: add a fixed version of what dh_systemd_start would have added - d/open-vm-tools-desktop.prerm, d/open-vm-tools-desktop.postrm: add what dh_systemd_start would have added (as-is since those sections worked) - d/control: update maintainers * d/open-vm-tools.service: Add After=local-fs.target dependency ensuring filesystems are ready to fix a race on startup (LP: #1750780) open-vm-tools (2:10.2.0-3ubuntu3) bionic; urgency=medium * Disable PrivateTmp for the open-vm-tools.service as it triggers issues when triggering processes that need tmp through VMOMI API (LP: #1758428) open-vm-tools (2:10.2.0-3ubuntu2) bionic; urgency=medium * Revert change in d/open-vm-tools.service that added After=local-fs.target. It turned out that the systemd in bionic already implicitly fixes this issue (the change is still needed for backports) (LP: 1750780) open-vm-tools (2:10.2.0-3ubuntu1) bionic; urgency=medium * d/open-vm-tools.service: Add After=local-fs.target dependency ensuring filesystems are ready to fix a race on startup (LP: #1750780) open-vm-tools (2:10.2.0-3) unstable; urgency=medium * [47e50a1] Fix debhelper dep for backports * [34538a5] Make tools.conf useful. Thanks to Dariusz Gadomski (Closes: #889884) open-vm-tools (2:10.2.0-2) unstable; urgency=medium * [249d54c] Fix wayland segfault. Adding a patch from Fedora to fix a wayland/gnome related segfault. Thanks to Oliver Kurth (Closes: #887755) open-vm-tools (2:10.2.0-1) unstable; urgency=medium * [f0bf956] Add .travis.yml which was removed by gbp. * [892e2f6] Build with gtk3. * [03a655b] Check if debhelper handles \ in systemd units. Thanks to Oliver Kurth (Closes: #886191) * [5bf9301] Drop -dkms package. Thanks to Christian Ehrhardt (Closes: #884656) * [236cdba] Update upstream source from tag 'upstream/10.2.0' Update to upstream version '10.2.0' with Debian dir d5190e486b6beb65ee7ed31c0c23a789b8f60cab (Closes: #884496) * [692beff] snapshot changelog. * [45aa743] Add .travis.yml which was removed by gbp. * [aabeded] Fix dpkg --compare-versions call * [0697425] Better debhelper version parsing. * [390ec09] fix even more makefile bugs. * [6e5fa38] Refreshing patches. Dropping kernel-module related patches. * [cbfec05] Drop dh_autoreconf, not needed anymore. * [d5fef50] autotools-dev is done by dh now. * [b66ab14] use dh_installsystemd open-vm-tools (2:10.1.15-1) unstable; urgency=medium * [78a17f1] Remove fixed CXX std setting. * [f96f479] Updated version 10.1.15 from 'upstream/10.1.15' with Debian dir c44394c71e055f4cfd3a15ee578fc9895d64ebb1 * [682790b] Refreshing patches. -- Christian Ehrhardt Thu, 15 Feb 2018 09:36:20 +0100 ** Changed in: open-vm-tools (Ubuntu Artful) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1750780 Title: Race with local file systems can make open-vm-tools fail to start Status in cloud-init: Invalid Status in open-vm-tools package in Ubuntu: Invalid Status in systemd package in Ubuntu: Fix Released Status in open-vm-tools source package in Xenial: Fix Released Status in systemd source package in Xenial: New Status in open-vm-tools source package in Artful: Fix Released Status in open-vm-tools package in Debian: Fix Released Bug description: Since the change in [1] open-vm-tools-service starts very (very) early. Not so much due to the Before=cloud-init-local.service But much more by DefaultDependencies=no That can trigger an issue that looks like root@ubuntuguest:~# systemctl status -l open-vm-tools.service ● open-vm-tools.service - Service for virtual machines hosted on VMware Loaded: loaded (/lib/systemd/system/open-vm-tools.service; enabled; vendor preset: enabled) Active: failed (Result: resources) As it is right now open-vm-tools can race with t
[Touch-packages] [Bug 1750780] Re: Race with local file systems can make open-vm-tools fail to start
This bug was fixed in the package open-vm-tools - 2:10.2.0-3~ubuntu0.16.04.1 --- open-vm-tools (2:10.2.0-3~ubuntu0.16.04.1) xenial; urgency=medium * backport Bionic open-vm-tools to Xenial (LP: #1741390) - d/control: B-D for dh-autoreconf and dh-systemd - d/rules: re-add autoreconf and systemd - d/control: B-D to debhelper version of xenial supporting compat 10 - d/compat: compat level 10 is latest xenial supported level - d/rules: go back from override_dh_installsystemd to override_dh_systemd_enable and override_dh_systemd_start as needed on xenials debhelper version. - d/rules: drop --no-restart-after-upgrade which only exists in debhelper 10, the version 9 behavior defaults to what we need. - debian/rules: dh_systemd_start in xenial has issues with the escaping of the run-vmblock\\x2dfuse.mount job, so drop this call - debian/open-vm-tools-desktop.postinst: add a fixed version of what dh_systemd_start would have added - d/open-vm-tools-desktop.prerm, d/open-vm-tools-desktop.postrm: add what dh_systemd_start would have added (as-is since those sections worked) - d/control: update maintainers * d/open-vm-tools.service: Add After=local-fs.target dependency ensuring filesystems are ready to fix a race on startup (LP: #1750780) open-vm-tools (2:10.2.0-3ubuntu3) bionic; urgency=medium * Disable PrivateTmp for the open-vm-tools.service as it triggers issues when triggering processes that need tmp through VMOMI API (LP: #1758428) open-vm-tools (2:10.2.0-3ubuntu2) bionic; urgency=medium * Revert change in d/open-vm-tools.service that added After=local-fs.target. It turned out that the systemd in bionic already implicitly fixes this issue (the change is still needed for backports) (LP: 1750780) open-vm-tools (2:10.2.0-3ubuntu1) bionic; urgency=medium * d/open-vm-tools.service: Add After=local-fs.target dependency ensuring filesystems are ready to fix a race on startup (LP: #1750780) open-vm-tools (2:10.2.0-3) unstable; urgency=medium * [47e50a1] Fix debhelper dep for backports * [34538a5] Make tools.conf useful. Thanks to Dariusz Gadomski (Closes: #889884) open-vm-tools (2:10.2.0-2) unstable; urgency=medium * [249d54c] Fix wayland segfault. Adding a patch from Fedora to fix a wayland/gnome related segfault. Thanks to Oliver Kurth (Closes: #887755) open-vm-tools (2:10.2.0-1) unstable; urgency=medium * [f0bf956] Add .travis.yml which was removed by gbp. * [892e2f6] Build with gtk3. * [03a655b] Check if debhelper handles \ in systemd units. Thanks to Oliver Kurth (Closes: #886191) * [5bf9301] Drop -dkms package. Thanks to Christian Ehrhardt (Closes: #884656) * [236cdba] Update upstream source from tag 'upstream/10.2.0' Update to upstream version '10.2.0' with Debian dir d5190e486b6beb65ee7ed31c0c23a789b8f60cab (Closes: #884496) * [692beff] snapshot changelog. * [45aa743] Add .travis.yml which was removed by gbp. * [aabeded] Fix dpkg --compare-versions call * [0697425] Better debhelper version parsing. * [390ec09] fix even more makefile bugs. * [6e5fa38] Refreshing patches. Dropping kernel-module related patches. * [cbfec05] Drop dh_autoreconf, not needed anymore. * [d5fef50] autotools-dev is done by dh now. * [b66ab14] use dh_installsystemd open-vm-tools (2:10.1.15-1) unstable; urgency=medium * [78a17f1] Remove fixed CXX std setting. * [f96f479] Updated version 10.1.15 from 'upstream/10.1.15' with Debian dir c44394c71e055f4cfd3a15ee578fc9895d64ebb1 * [682790b] Refreshing patches. open-vm-tools (2:10.1.10-3) unstable; urgency=medium * [00bc9bb] Build with CXXFLAGS=-std=c++14. Thanks to Rene Engelhard and Oliver Kurth (Closes: #876121) * [6b61376] Re-add .travis.yml. Seems it went missing with the last merge. * [bf07ae8] Work around dh_systemd escaping bugs again. Seems the unit escaping changed with a new dh version. Work around the known and reported fails again. Thanks to Christian Ehrhardt (Closes: #875657) open-vm-tools (2:10.1.10-2) unstable; urgency=medium [ Shota Aratono ] * [5ab87bd] Fix scsi timeout setting error on debian stretch * [c0847eb] Fix attempting change setting to unintentional target [ Raphaël Hertzog ] * [dc2e27f] Add patch to support resolution switching with KMS. This is needed for proper support of Wayland sessions. (Closes: #872779) open-vm-tools (2:10.1.10-1) unstable; urgency=medium * Drop the extra part of the version string. Easier to handle in the short variant. * [6be6a49] Updating watch file. Use tags from github. * [8c25235] Updated version 10.1.10 from 'upstream/10.1.10' with Debian dir 3dddea7985f21457b294b5f554d5ecdf32aabfff * [195cead] Refreshing patches. Removing cve-2015-5191.patch as it is part of the upstream release. * [e6b3fd5] Build using libssl-dev and libxmlsec1-dev. (Closes: #859416) ope
[Touch-packages] [Bug 1750780] Re: Race with local file systems can make open-vm-tools fail to start
1. Upgrade from proposed - this is the same for all associated bugs, so I only documented details in bug 1741390 2. This bug in particular Running a few restarts and checking $ systemctl status -l open-vm-tools.service This checks if the service rules avoid the issue on these systems with older systemd (older than Bionic where these extra constraints are not needed) Artful: did not expose the issue in 5/5 retries Xenial: did not expose the issue in 5/5 retries Per above (and the extensive precheck on the content-equal ppa by VMWare) - setting to verified ** Tags removed: verification-needed verification-needed-artful verification-needed-xenial ** Tags added: verification-done verification-done-artful verification-done-xenial -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1750780 Title: Race with local file systems can make open-vm-tools fail to start Status in cloud-init: Invalid Status in open-vm-tools package in Ubuntu: Invalid Status in systemd package in Ubuntu: Fix Released Status in open-vm-tools source package in Xenial: Fix Committed Status in systemd source package in Xenial: New Status in open-vm-tools source package in Artful: Fix Committed Status in open-vm-tools package in Debian: Fix Released Bug description: Since the change in [1] open-vm-tools-service starts very (very) early. Not so much due to the Before=cloud-init-local.service But much more by DefaultDependencies=no That can trigger an issue that looks like root@ubuntuguest:~# systemctl status -l open-vm-tools.service ● open-vm-tools.service - Service for virtual machines hosted on VMware Loaded: loaded (/lib/systemd/system/open-vm-tools.service; enabled; vendor preset: enabled) Active: failed (Result: resources) As it is right now open-vm-tools can race with the other early start and then fail. In detail one can find a message like: open-vm-tools.service: Failed to run 'start' task: Read-only file system" This is due to privtaeTmp=yes which is also set needing a writable /var/tmp [2] To ensure this works PrivateTmp would have to be removed (not good) or some after dependencies added that make this work reliably. I added After=local-fs.target which made it work for me in 3/3 tests. I' like to have an ack by the cloud-init Team that this does not totally kill the originally intended Before=cloud-init-local.service I think it does not as local-fs can complete before cloud-init-local, then open-vm-tools can initialize and finally cloud-init-local can pick up the data. To summarize: # cloud-init-local # DefaultDependencies=no Wants=network-pre.target After=systemd-remount-fs.service Before=NetworkManager.service Before=network-pre.target Before=shutdown.target Before=sysinit.target Conflicts=shutdown.target RequiresMountsFor=/var/lib/cloud # open-vm-tools # DefaultDependencies=no Before=cloud-init-local.service Proposed is to add to the latter: After=local-fs.target [1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859677 [2]: https://github.com/systemd/systemd/issues/5610 To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1750780/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1750780] Re: Race with local file systems can make open-vm-tools fail to start
Hello ChristianEhrhardt, or anyone else affected, Accepted open-vm-tools into artful-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/open-vm- tools/2:10.2.0-3~ubuntu0.17.10.1 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-artful to verification-done-artful. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-artful. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance! ** Changed in: open-vm-tools (Ubuntu Artful) Status: Triaged => Fix Committed ** Tags added: verification-needed-artful -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1750780 Title: Race with local file systems can make open-vm-tools fail to start Status in cloud-init: Invalid Status in open-vm-tools package in Ubuntu: Invalid Status in systemd package in Ubuntu: Fix Released Status in open-vm-tools source package in Xenial: Fix Committed Status in systemd source package in Xenial: New Status in open-vm-tools source package in Artful: Fix Committed Status in open-vm-tools package in Debian: Fix Released Bug description: Since the change in [1] open-vm-tools-service starts very (very) early. Not so much due to the Before=cloud-init-local.service But much more by DefaultDependencies=no That can trigger an issue that looks like root@ubuntuguest:~# systemctl status -l open-vm-tools.service ● open-vm-tools.service - Service for virtual machines hosted on VMware Loaded: loaded (/lib/systemd/system/open-vm-tools.service; enabled; vendor preset: enabled) Active: failed (Result: resources) As it is right now open-vm-tools can race with the other early start and then fail. In detail one can find a message like: open-vm-tools.service: Failed to run 'start' task: Read-only file system" This is due to privtaeTmp=yes which is also set needing a writable /var/tmp [2] To ensure this works PrivateTmp would have to be removed (not good) or some after dependencies added that make this work reliably. I added After=local-fs.target which made it work for me in 3/3 tests. I' like to have an ack by the cloud-init Team that this does not totally kill the originally intended Before=cloud-init-local.service I think it does not as local-fs can complete before cloud-init-local, then open-vm-tools can initialize and finally cloud-init-local can pick up the data. To summarize: # cloud-init-local # DefaultDependencies=no Wants=network-pre.target After=systemd-remount-fs.service Before=NetworkManager.service Before=network-pre.target Before=shutdown.target Before=sysinit.target Conflicts=shutdown.target RequiresMountsFor=/var/lib/cloud # open-vm-tools # DefaultDependencies=no Before=cloud-init-local.service Proposed is to add to the latter: After=local-fs.target [1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859677 [2]: https://github.com/systemd/systemd/issues/5610 To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1750780/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1750780] Re: Race with local file systems can make open-vm-tools fail to start
Hello ChristianEhrhardt, or anyone else affected, Accepted open-vm-tools into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/open-vm- tools/2:10.2.0-3~ubuntu0.16.04.1 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance! ** Changed in: open-vm-tools (Ubuntu Xenial) Status: Triaged => Fix Committed ** Tags added: verification-needed verification-needed-xenial -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1750780 Title: Race with local file systems can make open-vm-tools fail to start Status in cloud-init: Invalid Status in open-vm-tools package in Ubuntu: Invalid Status in systemd package in Ubuntu: Fix Released Status in open-vm-tools source package in Xenial: Fix Committed Status in systemd source package in Xenial: New Status in open-vm-tools source package in Artful: Triaged Status in open-vm-tools package in Debian: Fix Released Bug description: Since the change in [1] open-vm-tools-service starts very (very) early. Not so much due to the Before=cloud-init-local.service But much more by DefaultDependencies=no That can trigger an issue that looks like root@ubuntuguest:~# systemctl status -l open-vm-tools.service ● open-vm-tools.service - Service for virtual machines hosted on VMware Loaded: loaded (/lib/systemd/system/open-vm-tools.service; enabled; vendor preset: enabled) Active: failed (Result: resources) As it is right now open-vm-tools can race with the other early start and then fail. In detail one can find a message like: open-vm-tools.service: Failed to run 'start' task: Read-only file system" This is due to privtaeTmp=yes which is also set needing a writable /var/tmp [2] To ensure this works PrivateTmp would have to be removed (not good) or some after dependencies added that make this work reliably. I added After=local-fs.target which made it work for me in 3/3 tests. I' like to have an ack by the cloud-init Team that this does not totally kill the originally intended Before=cloud-init-local.service I think it does not as local-fs can complete before cloud-init-local, then open-vm-tools can initialize and finally cloud-init-local can pick up the data. To summarize: # cloud-init-local # DefaultDependencies=no Wants=network-pre.target After=systemd-remount-fs.service Before=NetworkManager.service Before=network-pre.target Before=shutdown.target Before=sysinit.target Conflicts=shutdown.target RequiresMountsFor=/var/lib/cloud # open-vm-tools # DefaultDependencies=no Before=cloud-init-local.service Proposed is to add to the latter: After=local-fs.target [1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859677 [2]: https://github.com/systemd/systemd/issues/5610 To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1750780/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1750780] Re: Race with local file systems can make open-vm-tools fail to start
This will only "become" an issue for Xenial/Artful with the backport. But lets do it right for tracking - so I added/modified tasks for these releases which allows me to refer changes and changelog to here. That way with the backport it will "be an issue" for the former releases, but also instantly be closed for them. For SRU consideration, due to the newer systemd in >=Bionic it is not affected - so it is "invalid" there. But for SRU considerations this is ok, as the "most recent release has to be fixed" is covered, there won't be an upgrade regression as when going e.g. Xenial->Bionic. In the queue for SRU review now - main bug 1741390 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1750780 Title: Race with local file systems can make open-vm-tools fail to start Status in cloud-init: Invalid Status in open-vm-tools package in Ubuntu: Invalid Status in systemd package in Ubuntu: Fix Released Status in open-vm-tools source package in Xenial: Triaged Status in systemd source package in Xenial: New Status in open-vm-tools source package in Artful: Triaged Status in open-vm-tools package in Debian: Fix Released Bug description: Since the change in [1] open-vm-tools-service starts very (very) early. Not so much due to the Before=cloud-init-local.service But much more by DefaultDependencies=no That can trigger an issue that looks like root@ubuntuguest:~# systemctl status -l open-vm-tools.service ● open-vm-tools.service - Service for virtual machines hosted on VMware Loaded: loaded (/lib/systemd/system/open-vm-tools.service; enabled; vendor preset: enabled) Active: failed (Result: resources) As it is right now open-vm-tools can race with the other early start and then fail. In detail one can find a message like: open-vm-tools.service: Failed to run 'start' task: Read-only file system" This is due to privtaeTmp=yes which is also set needing a writable /var/tmp [2] To ensure this works PrivateTmp would have to be removed (not good) or some after dependencies added that make this work reliably. I added After=local-fs.target which made it work for me in 3/3 tests. I' like to have an ack by the cloud-init Team that this does not totally kill the originally intended Before=cloud-init-local.service I think it does not as local-fs can complete before cloud-init-local, then open-vm-tools can initialize and finally cloud-init-local can pick up the data. To summarize: # cloud-init-local # DefaultDependencies=no Wants=network-pre.target After=systemd-remount-fs.service Before=NetworkManager.service Before=network-pre.target Before=shutdown.target Before=sysinit.target Conflicts=shutdown.target RequiresMountsFor=/var/lib/cloud # open-vm-tools # DefaultDependencies=no Before=cloud-init-local.service Proposed is to add to the latter: After=local-fs.target [1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859677 [2]: https://github.com/systemd/systemd/issues/5610 To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1750780/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1750780] Re: Race with local file systems can make open-vm-tools fail to start
** Also affects: open-vm-tools (Ubuntu Artful) Importance: Undecided Status: New ** Also affects: systemd (Ubuntu Artful) Importance: Undecided Status: New ** No longer affects: systemd (Ubuntu Artful) ** Changed in: open-vm-tools (Ubuntu Xenial) Status: Invalid => Triaged ** Changed in: open-vm-tools (Ubuntu Artful) Status: New => Triaged ** Changed in: open-vm-tools (Ubuntu) Status: Fix Released => Invalid -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1750780 Title: Race with local file systems can make open-vm-tools fail to start Status in cloud-init: Invalid Status in open-vm-tools package in Ubuntu: Invalid Status in systemd package in Ubuntu: Fix Released Status in open-vm-tools source package in Xenial: Triaged Status in systemd source package in Xenial: New Status in open-vm-tools source package in Artful: Triaged Status in open-vm-tools package in Debian: Fix Released Bug description: Since the change in [1] open-vm-tools-service starts very (very) early. Not so much due to the Before=cloud-init-local.service But much more by DefaultDependencies=no That can trigger an issue that looks like root@ubuntuguest:~# systemctl status -l open-vm-tools.service ● open-vm-tools.service - Service for virtual machines hosted on VMware Loaded: loaded (/lib/systemd/system/open-vm-tools.service; enabled; vendor preset: enabled) Active: failed (Result: resources) As it is right now open-vm-tools can race with the other early start and then fail. In detail one can find a message like: open-vm-tools.service: Failed to run 'start' task: Read-only file system" This is due to privtaeTmp=yes which is also set needing a writable /var/tmp [2] To ensure this works PrivateTmp would have to be removed (not good) or some after dependencies added that make this work reliably. I added After=local-fs.target which made it work for me in 3/3 tests. I' like to have an ack by the cloud-init Team that this does not totally kill the originally intended Before=cloud-init-local.service I think it does not as local-fs can complete before cloud-init-local, then open-vm-tools can initialize and finally cloud-init-local can pick up the data. To summarize: # cloud-init-local # DefaultDependencies=no Wants=network-pre.target After=systemd-remount-fs.service Before=NetworkManager.service Before=network-pre.target Before=shutdown.target Before=sysinit.target Conflicts=shutdown.target RequiresMountsFor=/var/lib/cloud # open-vm-tools # DefaultDependencies=no Before=cloud-init-local.service Proposed is to add to the latter: After=local-fs.target [1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859677 [2]: https://github.com/systemd/systemd/issues/5610 To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1750780/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1750780] Re: Race with local file systems can make open-vm-tools fail to start
** Merge proposal unlinked: https://code.launchpad.net/~paelzer/ubuntu/+source/open-vm-tools/+git/open-vm-tools/+merge/341797 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1750780 Title: Race with local file systems can make open-vm-tools fail to start Status in cloud-init: Invalid Status in open-vm-tools package in Ubuntu: Fix Released Status in systemd package in Ubuntu: Fix Released Status in open-vm-tools source package in Xenial: Invalid Status in systemd source package in Xenial: New Status in open-vm-tools package in Debian: Fix Released Bug description: Since the change in [1] open-vm-tools-service starts very (very) early. Not so much due to the Before=cloud-init-local.service But much more by DefaultDependencies=no That can trigger an issue that looks like root@ubuntuguest:~# systemctl status -l open-vm-tools.service ● open-vm-tools.service - Service for virtual machines hosted on VMware Loaded: loaded (/lib/systemd/system/open-vm-tools.service; enabled; vendor preset: enabled) Active: failed (Result: resources) As it is right now open-vm-tools can race with the other early start and then fail. In detail one can find a message like: open-vm-tools.service: Failed to run 'start' task: Read-only file system" This is due to privtaeTmp=yes which is also set needing a writable /var/tmp [2] To ensure this works PrivateTmp would have to be removed (not good) or some after dependencies added that make this work reliably. I added After=local-fs.target which made it work for me in 3/3 tests. I' like to have an ack by the cloud-init Team that this does not totally kill the originally intended Before=cloud-init-local.service I think it does not as local-fs can complete before cloud-init-local, then open-vm-tools can initialize and finally cloud-init-local can pick up the data. To summarize: # cloud-init-local # DefaultDependencies=no Wants=network-pre.target After=systemd-remount-fs.service Before=NetworkManager.service Before=network-pre.target Before=shutdown.target Before=sysinit.target Conflicts=shutdown.target RequiresMountsFor=/var/lib/cloud # open-vm-tools # DefaultDependencies=no Before=cloud-init-local.service Proposed is to add to the latter: After=local-fs.target [1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859677 [2]: https://github.com/systemd/systemd/issues/5610 To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1750780/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1750780] Re: Race with local file systems can make open-vm-tools fail to start
Note: When we do the Xenial backport of the new version of open-vm-tools (which we plan to do) this becomes an issue. In the same upload I intend to fix it right away, so it should never effectively exist in the field (keep invalid, but there might be a fix-released update to open-vm-tools here at some point). -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1750780 Title: Race with local file systems can make open-vm-tools fail to start Status in cloud-init: Invalid Status in open-vm-tools package in Ubuntu: Fix Released Status in systemd package in Ubuntu: Fix Released Status in open-vm-tools source package in Xenial: Invalid Status in systemd source package in Xenial: New Status in open-vm-tools package in Debian: Fix Released Bug description: Since the change in [1] open-vm-tools-service starts very (very) early. Not so much due to the Before=cloud-init-local.service But much more by DefaultDependencies=no That can trigger an issue that looks like root@ubuntuguest:~# systemctl status -l open-vm-tools.service ● open-vm-tools.service - Service for virtual machines hosted on VMware Loaded: loaded (/lib/systemd/system/open-vm-tools.service; enabled; vendor preset: enabled) Active: failed (Result: resources) As it is right now open-vm-tools can race with the other early start and then fail. In detail one can find a message like: open-vm-tools.service: Failed to run 'start' task: Read-only file system" This is due to privtaeTmp=yes which is also set needing a writable /var/tmp [2] To ensure this works PrivateTmp would have to be removed (not good) or some after dependencies added that make this work reliably. I added After=local-fs.target which made it work for me in 3/3 tests. I' like to have an ack by the cloud-init Team that this does not totally kill the originally intended Before=cloud-init-local.service I think it does not as local-fs can complete before cloud-init-local, then open-vm-tools can initialize and finally cloud-init-local can pick up the data. To summarize: # cloud-init-local # DefaultDependencies=no Wants=network-pre.target After=systemd-remount-fs.service Before=NetworkManager.service Before=network-pre.target Before=shutdown.target Before=sysinit.target Conflicts=shutdown.target RequiresMountsFor=/var/lib/cloud # open-vm-tools # DefaultDependencies=no Before=cloud-init-local.service Proposed is to add to the latter: After=local-fs.target [1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859677 [2]: https://github.com/systemd/systemd/issues/5610 To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1750780/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1750780] Re: Race with local file systems can make open-vm-tools fail to start
** Merge proposal linked: https://code.launchpad.net/~paelzer/ubuntu/+source/open-vm-tools/+git/open-vm-tools/+merge/341796 ** Merge proposal linked: https://code.launchpad.net/~paelzer/ubuntu/+source/open-vm-tools/+git/open-vm-tools/+merge/341797 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1750780 Title: Race with local file systems can make open-vm-tools fail to start Status in cloud-init: Invalid Status in open-vm-tools package in Ubuntu: Fix Released Status in systemd package in Ubuntu: Fix Released Status in open-vm-tools source package in Xenial: Invalid Status in systemd source package in Xenial: New Status in open-vm-tools package in Debian: Fix Released Bug description: Since the change in [1] open-vm-tools-service starts very (very) early. Not so much due to the Before=cloud-init-local.service But much more by DefaultDependencies=no That can trigger an issue that looks like root@ubuntuguest:~# systemctl status -l open-vm-tools.service ● open-vm-tools.service - Service for virtual machines hosted on VMware Loaded: loaded (/lib/systemd/system/open-vm-tools.service; enabled; vendor preset: enabled) Active: failed (Result: resources) As it is right now open-vm-tools can race with the other early start and then fail. In detail one can find a message like: open-vm-tools.service: Failed to run 'start' task: Read-only file system" This is due to privtaeTmp=yes which is also set needing a writable /var/tmp [2] To ensure this works PrivateTmp would have to be removed (not good) or some after dependencies added that make this work reliably. I added After=local-fs.target which made it work for me in 3/3 tests. I' like to have an ack by the cloud-init Team that this does not totally kill the originally intended Before=cloud-init-local.service I think it does not as local-fs can complete before cloud-init-local, then open-vm-tools can initialize and finally cloud-init-local can pick up the data. To summarize: # cloud-init-local # DefaultDependencies=no Wants=network-pre.target After=systemd-remount-fs.service Before=NetworkManager.service Before=network-pre.target Before=shutdown.target Before=sysinit.target Conflicts=shutdown.target RequiresMountsFor=/var/lib/cloud # open-vm-tools # DefaultDependencies=no Before=cloud-init-local.service Proposed is to add to the latter: After=local-fs.target [1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859677 [2]: https://github.com/systemd/systemd/issues/5610 To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1750780/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1750780] Re: Race with local file systems can make open-vm-tools fail to start
** Changed in: open-vm-tools (Debian) Status: Incomplete => Fix Released -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1750780 Title: Race with local file systems can make open-vm-tools fail to start Status in cloud-init: Invalid Status in open-vm-tools package in Ubuntu: Fix Released Status in systemd package in Ubuntu: Fix Released Status in open-vm-tools source package in Xenial: Invalid Status in systemd source package in Xenial: New Status in open-vm-tools package in Debian: Fix Released Bug description: Since the change in [1] open-vm-tools-service starts very (very) early. Not so much due to the Before=cloud-init-local.service But much more by DefaultDependencies=no That can trigger an issue that looks like root@ubuntuguest:~# systemctl status -l open-vm-tools.service ● open-vm-tools.service - Service for virtual machines hosted on VMware Loaded: loaded (/lib/systemd/system/open-vm-tools.service; enabled; vendor preset: enabled) Active: failed (Result: resources) As it is right now open-vm-tools can race with the other early start and then fail. In detail one can find a message like: open-vm-tools.service: Failed to run 'start' task: Read-only file system" This is due to privtaeTmp=yes which is also set needing a writable /var/tmp [2] To ensure this works PrivateTmp would have to be removed (not good) or some after dependencies added that make this work reliably. I added After=local-fs.target which made it work for me in 3/3 tests. I' like to have an ack by the cloud-init Team that this does not totally kill the originally intended Before=cloud-init-local.service I think it does not as local-fs can complete before cloud-init-local, then open-vm-tools can initialize and finally cloud-init-local can pick up the data. To summarize: # cloud-init-local # DefaultDependencies=no Wants=network-pre.target After=systemd-remount-fs.service Before=NetworkManager.service Before=network-pre.target Before=shutdown.target Before=sysinit.target Conflicts=shutdown.target RequiresMountsFor=/var/lib/cloud # open-vm-tools # DefaultDependencies=no Before=cloud-init-local.service Proposed is to add to the latter: After=local-fs.target [1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859677 [2]: https://github.com/systemd/systemd/issues/5610 To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1750780/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1750780] Re: Race with local file systems can make open-vm-tools fail to start
For open-vm-tools this issue will only exist with the planned backport of the newer version. Since we will not ship the broken backport as we found it in pre-checks the correct state for open-vm-tools in xenial is invalid. ** Changed in: open-vm-tools (Ubuntu Xenial) Status: Triaged => Invalid -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1750780 Title: Race with local file systems can make open-vm-tools fail to start Status in cloud-init: Invalid Status in open-vm-tools package in Ubuntu: Fix Released Status in systemd package in Ubuntu: Fix Released Status in open-vm-tools source package in Xenial: Invalid Status in systemd source package in Xenial: New Status in open-vm-tools package in Debian: Incomplete Bug description: Since the change in [1] open-vm-tools-service starts very (very) early. Not so much due to the Before=cloud-init-local.service But much more by DefaultDependencies=no That can trigger an issue that looks like root@ubuntuguest:~# systemctl status -l open-vm-tools.service ● open-vm-tools.service - Service for virtual machines hosted on VMware Loaded: loaded (/lib/systemd/system/open-vm-tools.service; enabled; vendor preset: enabled) Active: failed (Result: resources) As it is right now open-vm-tools can race with the other early start and then fail. In detail one can find a message like: open-vm-tools.service: Failed to run 'start' task: Read-only file system" This is due to privtaeTmp=yes which is also set needing a writable /var/tmp [2] To ensure this works PrivateTmp would have to be removed (not good) or some after dependencies added that make this work reliably. I added After=local-fs.target which made it work for me in 3/3 tests. I' like to have an ack by the cloud-init Team that this does not totally kill the originally intended Before=cloud-init-local.service I think it does not as local-fs can complete before cloud-init-local, then open-vm-tools can initialize and finally cloud-init-local can pick up the data. To summarize: # cloud-init-local # DefaultDependencies=no Wants=network-pre.target After=systemd-remount-fs.service Before=NetworkManager.service Before=network-pre.target Before=shutdown.target Before=sysinit.target Conflicts=shutdown.target RequiresMountsFor=/var/lib/cloud # open-vm-tools # DefaultDependencies=no Before=cloud-init-local.service Proposed is to add to the latter: After=local-fs.target [1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859677 [2]: https://github.com/systemd/systemd/issues/5610 To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1750780/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1750780] Re: Race with local file systems can make open-vm-tools fail to start
With the former update in mind I retried Xenial/Bionic again. All of it is racy (as we knew), but it never triggered for Bionic. Xenial (19/33 fails) Bionic (0/37 fails) So for now we continue to assume that it is fixed there (by systemd) and revert our added dependency. Note: as with the simple job in comment #12 and comment #14 this is Hipervisor agnostic. You can even test the open-vm-tools service in KVM by removing the condition on vmware. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1750780 Title: Race with local file systems can make open-vm-tools fail to start Status in cloud-init: Invalid Status in open-vm-tools package in Ubuntu: Fix Released Status in systemd package in Ubuntu: Fix Released Status in open-vm-tools source package in Xenial: Triaged Status in systemd source package in Xenial: New Status in open-vm-tools package in Debian: Incomplete Bug description: Since the change in [1] open-vm-tools-service starts very (very) early. Not so much due to the Before=cloud-init-local.service But much more by DefaultDependencies=no That can trigger an issue that looks like root@ubuntuguest:~# systemctl status -l open-vm-tools.service ● open-vm-tools.service - Service for virtual machines hosted on VMware Loaded: loaded (/lib/systemd/system/open-vm-tools.service; enabled; vendor preset: enabled) Active: failed (Result: resources) As it is right now open-vm-tools can race with the other early start and then fail. In detail one can find a message like: open-vm-tools.service: Failed to run 'start' task: Read-only file system" This is due to privtaeTmp=yes which is also set needing a writable /var/tmp [2] To ensure this works PrivateTmp would have to be removed (not good) or some after dependencies added that make this work reliably. I added After=local-fs.target which made it work for me in 3/3 tests. I' like to have an ack by the cloud-init Team that this does not totally kill the originally intended Before=cloud-init-local.service I think it does not as local-fs can complete before cloud-init-local, then open-vm-tools can initialize and finally cloud-init-local can pick up the data. To summarize: # cloud-init-local # DefaultDependencies=no Wants=network-pre.target After=systemd-remount-fs.service Before=NetworkManager.service Before=network-pre.target Before=shutdown.target Before=sysinit.target Conflicts=shutdown.target RequiresMountsFor=/var/lib/cloud # open-vm-tools # DefaultDependencies=no Before=cloud-init-local.service Proposed is to add to the latter: After=local-fs.target [1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859677 [2]: https://github.com/systemd/systemd/issues/5610 To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1750780/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1750780] Re: Race with local file systems can make open-vm-tools fail to start
Thanks Scott for your cross check. I wonder why my former test failed on each of my tests without writing, but never the less your extended example is great for the systemd issue that remains. Although all of this is still a race, for example with the job above on a Xenial container I could not trigger it in several retries (5/5 worked). Doing the same in a Xenial KVM guest instead triggered it right away, but still not on every reboot (3/5 failed). I hope that helps everybody trying to reproduce this. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1750780 Title: Race with local file systems can make open-vm-tools fail to start Status in cloud-init: Invalid Status in open-vm-tools package in Ubuntu: Fix Released Status in systemd package in Ubuntu: Fix Released Status in open-vm-tools source package in Xenial: Triaged Status in systemd source package in Xenial: New Status in open-vm-tools package in Debian: Incomplete Bug description: Since the change in [1] open-vm-tools-service starts very (very) early. Not so much due to the Before=cloud-init-local.service But much more by DefaultDependencies=no That can trigger an issue that looks like root@ubuntuguest:~# systemctl status -l open-vm-tools.service ● open-vm-tools.service - Service for virtual machines hosted on VMware Loaded: loaded (/lib/systemd/system/open-vm-tools.service; enabled; vendor preset: enabled) Active: failed (Result: resources) As it is right now open-vm-tools can race with the other early start and then fail. In detail one can find a message like: open-vm-tools.service: Failed to run 'start' task: Read-only file system" This is due to privtaeTmp=yes which is also set needing a writable /var/tmp [2] To ensure this works PrivateTmp would have to be removed (not good) or some after dependencies added that make this work reliably. I added After=local-fs.target which made it work for me in 3/3 tests. I' like to have an ack by the cloud-init Team that this does not totally kill the originally intended Before=cloud-init-local.service I think it does not as local-fs can complete before cloud-init-local, then open-vm-tools can initialize and finally cloud-init-local can pick up the data. To summarize: # cloud-init-local # DefaultDependencies=no Wants=network-pre.target After=systemd-remount-fs.service Before=NetworkManager.service Before=network-pre.target Before=shutdown.target Before=sysinit.target Conflicts=shutdown.target RequiresMountsFor=/var/lib/cloud # open-vm-tools # DefaultDependencies=no Before=cloud-init-local.service Proposed is to add to the latter: After=local-fs.target [1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859677 [2]: https://github.com/systemd/systemd/issues/5610 To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1750780/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1750780] Re: Race with local file systems can make open-vm-tools fail to start
I agree with reverting change in bionic, and that xenial still needs some fix. I recreated your failure in xenial and agree with your change there. I used this job as it actually tries to write to the private tmp. If I change 'ExecStart' below to be just '/bin/true' like in your job, it does not fail (status says 'code=exited, status=0/SUCCESS). But the command below seems to never get exited (it produces no output). So it seems to me that the 'Read only filesystem' error is not even from the service itself, but rather from systemd, and also dependent on the content in ExecStart. $ cat /lib/systemd/system/my-ptmp-test.service [Unit] Description=foo DefaultDependencies=no [Service] Type=oneshot PrivateTmp=yes ExecStart=/bin/sh -ec 'f=/tmp/my.txt; uptime > $f; echo ==; cat $f; echo ==; cat /proc/mounts' [Install] WantedBy=multi-user.target Then we can see the same failure: $ journalctl -o short-monotonic --no-pager | grep --context=2 my-ptmp-test [7.715220] xenial-20180223-142555 systemd[1]: Created slice System Slice. [7.718923] xenial-20180223-142555 systemd[1]: Created slice system-systemd\x2dfsck.slice. [7.740250] xenial-20180223-142555 systemd[1]: my-ptmp-test.service: Failed to run 'start' task: Read-only file system [7.749135] xenial-20180223-142555 systemd[1]: Failed to start foo. [7.754114] xenial-20180223-142555 systemd[1]: my-ptmp-test.service: Unit entered failed state. [7.760217] xenial-20180223-142555 systemd[1]: my-ptmp-test.service: Failed with result 'resources'. [7.764170] xenial-20180223-142555 systemd[1]: Starting Load Kernel Modules... [7.775558] xenial-20180223-142555 systemd[1]: Starting Journal Service... $ systemctl status my-ptmp-test.service --no-pager --full ● my-ptmp-test.service - foo Loaded: loaded (/lib/systemd/system/my-ptmp-test.service; enabled; vendor preset: enabled) Active: failed (Result: resources) -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1750780 Title: Race with local file systems can make open-vm-tools fail to start Status in cloud-init: Invalid Status in open-vm-tools package in Ubuntu: Fix Released Status in systemd package in Ubuntu: Fix Released Status in open-vm-tools source package in Xenial: Triaged Status in systemd source package in Xenial: New Status in open-vm-tools package in Debian: Incomplete Bug description: Since the change in [1] open-vm-tools-service starts very (very) early. Not so much due to the Before=cloud-init-local.service But much more by DefaultDependencies=no That can trigger an issue that looks like root@ubuntuguest:~# systemctl status -l open-vm-tools.service ● open-vm-tools.service - Service for virtual machines hosted on VMware Loaded: loaded (/lib/systemd/system/open-vm-tools.service; enabled; vendor preset: enabled) Active: failed (Result: resources) As it is right now open-vm-tools can race with the other early start and then fail. In detail one can find a message like: open-vm-tools.service: Failed to run 'start' task: Read-only file system" This is due to privtaeTmp=yes which is also set needing a writable /var/tmp [2] To ensure this works PrivateTmp would have to be removed (not good) or some after dependencies added that make this work reliably. I added After=local-fs.target which made it work for me in 3/3 tests. I' like to have an ack by the cloud-init Team that this does not totally kill the originally intended Before=cloud-init-local.service I think it does not as local-fs can complete before cloud-init-local, then open-vm-tools can initialize and finally cloud-init-local can pick up the data. To summarize: # cloud-init-local # DefaultDependencies=no Wants=network-pre.target After=systemd-remount-fs.service Before=NetworkManager.service Before=network-pre.target Before=shutdown.target Before=sysinit.target Conflicts=shutdown.target RequiresMountsFor=/var/lib/cloud # open-vm-tools # DefaultDependencies=no Before=cloud-init-local.service Proposed is to add to the latter: After=local-fs.target [1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859677 [2]: https://github.com/systemd/systemd/issues/5610 To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1750780/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1750780] Re: Race with local file systems can make open-vm-tools fail to start
I'Ll likely revert the Binonic change tmrw morning as we have discussed. local-fs.target is actually >> the implicit dependency. But that does not solve the Xenial issue outlined in the former comment. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1750780 Title: Race with local file systems can make open-vm-tools fail to start Status in cloud-init: Invalid Status in open-vm-tools package in Ubuntu: Fix Released Status in systemd package in Ubuntu: Fix Released Status in open-vm-tools source package in Xenial: Triaged Status in systemd source package in Xenial: New Status in open-vm-tools package in Debian: Incomplete Bug description: Since the change in [1] open-vm-tools-service starts very (very) early. Not so much due to the Before=cloud-init-local.service But much more by DefaultDependencies=no That can trigger an issue that looks like root@ubuntuguest:~# systemctl status -l open-vm-tools.service ● open-vm-tools.service - Service for virtual machines hosted on VMware Loaded: loaded (/lib/systemd/system/open-vm-tools.service; enabled; vendor preset: enabled) Active: failed (Result: resources) As it is right now open-vm-tools can race with the other early start and then fail. In detail one can find a message like: open-vm-tools.service: Failed to run 'start' task: Read-only file system" This is due to privtaeTmp=yes which is also set needing a writable /var/tmp [2] To ensure this works PrivateTmp would have to be removed (not good) or some after dependencies added that make this work reliably. I added After=local-fs.target which made it work for me in 3/3 tests. I' like to have an ack by the cloud-init Team that this does not totally kill the originally intended Before=cloud-init-local.service I think it does not as local-fs can complete before cloud-init-local, then open-vm-tools can initialize and finally cloud-init-local can pick up the data. To summarize: # cloud-init-local # DefaultDependencies=no Wants=network-pre.target After=systemd-remount-fs.service Before=NetworkManager.service Before=network-pre.target Before=shutdown.target Before=sysinit.target Conflicts=shutdown.target RequiresMountsFor=/var/lib/cloud # open-vm-tools # DefaultDependencies=no Before=cloud-init-local.service Proposed is to add to the latter: After=local-fs.target [1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859677 [2]: https://github.com/systemd/systemd/issues/5610 To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1750780/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1750780] Re: Race with local file systems can make open-vm-tools fail to start
Installed another Xenial and Bionic in vmware to take a deper look. - Xenial (with backported open-vm-tools): affected - Bionic (with the interim fix reverted): no hit in several retries, explanation below Systemd fixed it (via our assumed implicit dependency). In Bionic the PrivateTmp gives it a dependency on systemd-tmpfile-setup.service (seen in systemd analyze, there might be more but not on crit path). This is configured by default to include /var/tmp in /usr/lib/tmpfiles.d/tmp.conf. In regard to your thoughts about later on changing cloud-init ordering that won't help you, as the dependency is there (implicit or explicit doesn't matter). For the xenial case where I reliably hit the issue instead of stracing I cut things short. A service with the following exposes exactly the same error: [Unit] Description=foo DefaultDependencies=no [Service] PrivateTmp=yes ExecStart=/bin/true [Install] WantedBy=multi-user.target So back on Xenial it is privateTmp + too early that breaks it. Xenial vs Bionic critical-chain according to "systemd-analyze critical- chain open.vm-tools.service" Xenial with fix: open-vm-tools.service @3.482s └─local-fs.target @3.460s └─local-fs-pre.target @3.460s └─systemd-remount-fs.service @3.442s +9ms └─system.slice @220ms └─-.slice @204m Xenial without fix: └─run-vmblock\x2dfuse.mount @6.076s +390ms └─sys-fs-fuse-connections.mount @5.510s +375ms └─systemd-modules-load.service @1.996s +75ms └─system.slice @1.984s └─-.slice @1.966s Bionic open-vm-tools.service @3.566s └─systemd-tmpfiles-setup.service @3.421s +100ms └─systemd-journal-flush.service @3.054s +342ms └─systemd-journald.service @825ms +2.219s └─syslog.socket @808ms └─system.slice @621ms └─-.slice @613ms To Summarize, we can: - revert the fix for Bionic (or later) - just make it a sync when convenient down the road, it doesn't hurt for now as it is (almost) the same as the implicit dependency) - add a xenials systemd bug task (probably too complex to fix as -upstream) - until said systemd bug is fixed a backport of open-vm-tools needs this fix ** Also affects: systemd (Ubuntu) Importance: Undecided Status: New ** Also affects: open-vm-tools (Ubuntu Xenial) Importance: Undecided Status: New ** Also affects: systemd (Ubuntu Xenial) Importance: Undecided Status: New ** Changed in: open-vm-tools (Ubuntu Xenial) Status: New => Triaged ** Changed in: systemd (Ubuntu) Status: New => Fix Released -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1750780 Title: Race with local file systems can make open-vm-tools fail to start Status in cloud-init: Invalid Status in open-vm-tools package in Ubuntu: Fix Released Status in systemd package in Ubuntu: Fix Released Status in open-vm-tools source package in Xenial: Triaged Status in systemd source package in Xenial: New Status in open-vm-tools package in Debian: Incomplete Bug description: Since the change in [1] open-vm-tools-service starts very (very) early. Not so much due to the Before=cloud-init-local.service But much more by DefaultDependencies=no That can trigger an issue that looks like root@ubuntuguest:~# systemctl status -l open-vm-tools.service ● open-vm-tools.service - Service for virtual machines hosted on VMware Loaded: loaded (/lib/systemd/system/open-vm-tools.service; enabled; vendor preset: enabled) Active: failed (Result: resources) As it is right now open-vm-tools can race with the other early start and then fail. In detail one can find a message like: open-vm-tools.service: Failed to run 'start' task: Read-only file system" This is due to privtaeTmp=yes which is also set needing a writable /var/tmp [2] To ensure this works PrivateTmp would have to be removed (not good) or some after dependencies added that make this work reliably. I added After=local-fs.target which made it work for me in 3/3 tests. I' like to have an ack by the cloud-init Team that this does not totally kill the originally intended Before=cloud-init-local.service I think it does not as local-fs can complete before cloud-init-local, then open-vm-tools can initialize and finally cloud-init-local can pick up the data. To summarize: # cloud-init-local # DefaultDependencies=no Wants=network-pre.target After=systemd-remount-fs.service Before=NetworkManager.service Before=network-pre.target Before=shutdown.target Before=sysinit.target Conflicts=shutdown.target RequiresMountsFor=/var/lib/cloud # open-vm-tools # DefaultDependencies=no Before=cloud-init-local.service Proposed is to add to the latter: After=local-fs.target [1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859677 [2]: https://github.com/systemd/systemd/issues/5610 To manage
