I agree with reverting change in bionic, and that xenial still needs some fix.
I recreated your failure in xenial and agree with your change there. I used this job as it actually tries to write to the private tmp. If I change 'ExecStart' below to be just '/bin/true' like in your job, it does not fail (status says 'code=exited, status=0/SUCCESS). But the command below seems to never get exited (it produces no output). So it seems to me that the 'Read only filesystem' error is not even from the service itself, but rather from systemd, and also dependent on the content in ExecStart. $ cat /lib/systemd/system/my-ptmp-test.service [Unit] Description=foo DefaultDependencies=no [Service] Type=oneshot PrivateTmp=yes ExecStart=/bin/sh -ec 'f=/tmp/my.txt; uptime > $f; echo ==; cat $f; echo ==; cat /proc/mounts' [Install] WantedBy=multi-user.target Then we can see the same failure: $ journalctl -o short-monotonic --no-pager | grep --context=2 my-ptmp-test [ 7.715220] xenial-20180223-142555 systemd[1]: Created slice System Slice. [ 7.718923] xenial-20180223-142555 systemd[1]: Created slice system-systemd\x2dfsck.slice. [ 7.740250] xenial-20180223-142555 systemd[1]: my-ptmp-test.service: Failed to run 'start' task: Read-only file system [ 7.749135] xenial-20180223-142555 systemd[1]: Failed to start foo. [ 7.754114] xenial-20180223-142555 systemd[1]: my-ptmp-test.service: Unit entered failed state. [ 7.760217] xenial-20180223-142555 systemd[1]: my-ptmp-test.service: Failed with result 'resources'. [ 7.764170] xenial-20180223-142555 systemd[1]: Starting Load Kernel Modules... [ 7.775558] xenial-20180223-142555 systemd[1]: Starting Journal Service... $ systemctl status my-ptmp-test.service --no-pager --full ● my-ptmp-test.service - foo Loaded: loaded (/lib/systemd/system/my-ptmp-test.service; enabled; vendor preset: enabled) Active: failed (Result: resources) -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1750780 Title: Race with local file systems can make open-vm-tools fail to start Status in cloud-init: Invalid Status in open-vm-tools package in Ubuntu: Fix Released Status in systemd package in Ubuntu: Fix Released Status in open-vm-tools source package in Xenial: Triaged Status in systemd source package in Xenial: New Status in open-vm-tools package in Debian: Incomplete Bug description: Since the change in [1] open-vm-tools-service starts very (very) early. Not so much due to the Before=cloud-init-local.service But much more by DefaultDependencies=no That can trigger an issue that looks like root@ubuntuguest:~# systemctl status -l open-vm-tools.service ● open-vm-tools.service - Service for virtual machines hosted on VMware Loaded: loaded (/lib/systemd/system/open-vm-tools.service; enabled; vendor preset: enabled) Active: failed (Result: resources) As it is right now open-vm-tools can race with the other early start and then fail. In detail one can find a message like: open-vm-tools.service: Failed to run 'start' task: Read-only file system" This is due to privtaeTmp=yes which is also set needing a writable /var/tmp [2] To ensure this works PrivateTmp would have to be removed (not good) or some after dependencies added that make this work reliably. I added After=local-fs.target which made it work for me in 3/3 tests. I' like to have an ack by the cloud-init Team that this does not totally kill the originally intended Before=cloud-init-local.service I think it does not as local-fs can complete before cloud-init-local, then open-vm-tools can initialize and finally cloud-init-local can pick up the data. To summarize: # cloud-init-local # DefaultDependencies=no Wants=network-pre.target After=systemd-remount-fs.service Before=NetworkManager.service Before=network-pre.target Before=shutdown.target Before=sysinit.target Conflicts=shutdown.target RequiresMountsFor=/var/lib/cloud # open-vm-tools # DefaultDependencies=no Before=cloud-init-local.service Proposed is to add to the latter: After=local-fs.target [1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859677 [2]: https://github.com/systemd/systemd/issues/5610 To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1750780/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
