On Fri, 2015-05-22 at 10:06 -0400, Robert Rati wrote: > I've reproduced this issue pretty easily. We have symlinks in > /etc/systemd/system that point to common unit files on an NFS share. > The unit files in the NFS share are usable and functioning on 7.1.0. > Then I do: > > ostree remote add --set=gpg-verify=false atomic7.1.1 <url> > rpm-ostree rebase atomic7.1.1:rhel-atomic-host/7/x86_64/standard > systemctl reboot > > After the system boots, the kubelets and kube-proxy are not running. > Attempting to start the kubelet manually produces a failure because > our > configuration causes issues with the kube shipped in 7.1.1 (unit file > > ${VAR} issue). Doing: > > systemctl daemon-reload; systemctl restart kubelet > > gets me the kubelet from our custom unit files > (v0.17.1-629-gd9d12fd3f7036c). > > Then I upgrade to 7.1.2: > ostree remote add --set=gpg-verify=false atomic7.1.2 <url> > rpm-ostree rebase atomic7.1.1:rhel-atomic-host/7/x86_64/standard > systemctl reboot > > After the system boots, the kubelet is not running. Manually > starting > the kubelet results in the kubelet version from atomic 7.1.2 running > (v0.15.0-123-g0ea87e48640729) > > Throughout all upgrades the files in /etc/systemd/system are > untouched. > The symlinks are still pointing to the unit files in the shared > storage and the links are still good, the NFS share is mounted, etc. > > systemctl daemon-reload; systemctl restart kubelet > > The above gets me the kubelet from our custom unit files > (v0.17.1-629-gd9d12fd3f7036c). > > In each upgrade case, "atomic host upgrade" never did anything but > report an error or that no upgrades available. I tried running it > after > ostree and rpm-ostree steps, and before and after the reboot. > > It should also be noted that in each upgrade case daemons that were > enabled didn't end up starting on the reboot. From 7.1.0->7.1.1 > docker, > kubelet, and kube-proxy did not start after the reboot. From > 7.1.1->7.1.2 docker and the kubelet did not start after the reboot.
Did this every work after a reboot? I think the key is that you don't actually have unit files in etc, you have them in nfs! /etc is "supposed" to be local I don't see a good way to fix your setup. Don't use NFS for /etc Add a daemon-reload after the mounting of remote filesystems, but that's hard, as you'll need to add After= stuff to your unit files you have on NFS... Nothing about this seems feasible really. Would we really expect to put links to NFS shares in /etc/init.d/rc3.d/ and them to work correctly? > > Rob > > On 05/21/2015 04:07 PM, Tim St Clair wrote: > > Hey Folks - > > > > We recently upgraded our cluster 7.1.0->7.1.1->7.1.2 and we > > uncovered that our systemd.unit files did not hold across upgrades. > > > > Is/was this a known issue? > > >