Bug#775541: tests of new packages
On Mon, Sep 12, 2016 at 02:38:32PM +1000, Vincent McIntyre wrote: A brief note on the client side of things. At step 3, after installing your packages but before tweaking nfs-server.service, the critical chain for the client mount is usr-local.mount +1.145s └─remote-fs-pre.target @7.493s └─nfs-client.target @2.362s After (step 4.x) it is somewhat different usr-local.mount +1.141s └─remote-fs-pre.target @10.207s └─nfs-server.service @10.193s +12ms └─rpc-statd.service @10.342s +18ms └─nss-lookup.target @10.340s The 'list-dependencies' output is the same for both Dependencies for usr-local.mount usr-local.mount ● ├─-.mount ● ├─system.slice ● ├─usr.mount ● └─network-online.target Not sure why nfs-client has dopped out of the picture. Nonetheless, the mount works fine. Cheers Vince
Bug#775541: tests of new packages
On Fri, Sep 09, 2016 at 10:22:25AM +0200, Andreas Henriksson wrote: > Hello Vincent McIntyre. > > Thanks for your thurough testing and useful feedback. > > Let me start with a disclaimer: I'm not maintaining nfs (and I'm > not even using it myself so my knowledge is very limited). My only > involvement here is trying to squash some RC bugs and unblocking > work elsewhere by importing native systemd units. That is fine with me and highly appreciated. > On Fri, Sep 09, 2016 at 11:57:34AM +1000, Vincent McIntyre wrote: > > > > Thanks Andreas for those new packages. > > I did some testing of the 1.2.8-9.2 packages on a clean jessie install. > > They are pretty close but I found an issue with NFS exports in one case. > > > > I used the attached check.sh script to show the state of various targets > > as I changed things. The attached results.tar shows the output from it. > > Between each major stage below, the system is rebooted. > > Then I log in and run check.sh before doing anything else. > > Throughout, /etc/network/interfaces contained: > > source /etc/network/interfaces.d/* > > auto lo > > iface lo inet loopback > > allow-hotplug eth0 > > I think this is problematic and unfortunately spoils my interest > in the rest of your testing. > > From what I've been told one of the main differences between "auto" > and "allow-hotplug" is that allow-hotplug will not block your bootup > so services will run wild and start up ASAP. This means they can and > likely will start up before your network connection is up. Specially > if using dhcp which takes time to configure the interface and extra > so if you're using a network device (eg. usb dongle) which takes a > long time to initialize. > > I assume eth0 was involved in your tests, and if so it would be > very useful to know what difference it makes if you use auto > instead. > Thanks for that. I've repeated with a slightly different numbering. The short version is that the client mounts and exports all work with your packages and 'auto eth0' BUT one change is needed to make exports work at boot time: /lib/systemd/system/nfs-mountd.service - After=network.target local-fs.target + After=rpcbind.target network.target local-fs.target There might be a more elegant way to this but this seems to work. I did a quick check of your packages with 'auto eth0' and a static address setup in /etc/network/interfaces. That worked also but to get the exports working at boot the change above was required. Detailed results: step 1 - jessie system with nfs packages 1.2.8-9 one fstab mount, no exports. no changes to /etc/default/nfs-* The fstab mount works ok; it politely waits for dhcp to finish before trying to mount. It works in all the other tests so I won't mention it further. step 2 - as above but export one filesystem. Tried a few different ways to do the exports - wildcard host (step2.1) - point the export at a single host (step2.2) - point the export at a netgroup (step2.3) All of these work with no problem. step 3 - jessie system with nfs packages 1.2.8-9.2 one fstab mount, one exported directory. rpc.mountd fails to start because it wants rpcbind to be there, but rpcbind has not been started yet. showmount -e fails: # showmount -e clnt_create: RPC: Program not registered step3.1 - modify /lib/systemd/system/nfs-mountd.service, - After=network.target local-fs.target + After=rpcbind.target network.target local-fs.target run systemd daemon-reload but don't reboot. The exports are still not there # showmount -e clnt_create: RPC: Program not registered re run exports, it fails # exportfs -rav exporting @all_hosts:/data/INSTALL_1 # showmount -e clnt_create: RPC: Program not registered # step 4 - reboot after step3.1: All of the exports work now. I don't need to insert any 30s waits anywhere, yay. Without the tweak to nfs-mountd.service, the critical chain is: nfs-mountd.service +33ms └─network.target @7.453s └─networking.service @2.349s +5.103s └─local-fs.target @2.345s └─data-INSTALL_1.mount @2.212s +132ms └─systemd-fsck@dev-mapper-libra\x2ddata.service @2.193s +16ms └─dev-mapper-libra\x2ddata.device @2.192s With the tweak in place nfs-mountd.service +19ms └─rpcbind.target @10.169s └─rpcbind.service @10.126s +42ms └─network-online.target @10.123s └─network.target @10.123s └─networking.service @1.289s +8.833s └─local-fs.target @1.285s └─data-INSTALL_1.mount @1.061s +223ms └─systemd-fsck@dev-mapper-libra\x2ddata.service @1.043s +14ms └─dev-mapper-libra\x2ddata.device @1.042s Active config in
Bug#775541: tests of new packages
Hello Vincent McIntyre. Thanks for your thurough testing and useful feedback. Let me start with a disclaimer: I'm not maintaining nfs (and I'm not even using it myself so my knowledge is very limited). My only involvement here is trying to squash some RC bugs and unblocking work elsewhere by importing native systemd units. On Fri, Sep 09, 2016 at 11:57:34AM +1000, Vincent McIntyre wrote: > > Thanks Andreas for those new packages. > I did some testing of the 1.2.8-9.2 packages on a clean jessie install. > They are pretty close but I found an issue with NFS exports in one case. > > I used the attached check.sh script to show the state of various targets > as I changed things. The attached results.tar shows the output from it. > Between each major stage below, the system is rebooted. > Then I log in and run check.sh before doing anything else. > Throughout, /etc/network/interfaces contained: > source /etc/network/interfaces.d/* > auto lo > iface lo inet loopback > allow-hotplug eth0 I think this is problematic and unfortunately spoils my interest in the rest of your testing. >From what I've been told one of the main differences between "auto" and "allow-hotplug" is that allow-hotplug will not block your bootup so services will run wild and start up ASAP. This means they can and likely will start up before your network connection is up. Specially if using dhcp which takes time to configure the interface and extra so if you're using a network device (eg. usb dongle) which takes a long time to initialize. I assume eth0 was involved in your tests, and if so it would be very useful to know what difference it makes if you use auto instead. > iface eth0 inet dhcp > and /etc/network/interfaces.d/ was empty. > > step 1 - jessie system with nfs packages 1.2.8-9 > one fstab mount, no exports. no changes to /etc/default/nfs-*. > The fstab mount works ok; it tries to mount it way too early > but we invoke it with the 'bg' option so eventually it works. > > step 2 - as above but with the _netdev option added to the fstab entry. > System still tries to mount too early, before dhclient is done, > but it works in the end. > > step 3 - as above but export one filesystem. > The fstab mount works, as above. > Tried a few different ways to do the exports >- wildcard host (step3.1) > This works ok. >- point the export at a single host (step3.2) > This tries to start mountd to early, before it can resolve > the hostname given in the exports file. > As a result we end up with an empty export list: > # showmount -e > Export list for install: > # > Restarting the service once the system is fully booted works > # systemctl restart nfs-kernel-server.service > # showmount -e > Export list for install: > /data/INSTALL_1 mayhem.atnf.CSIRO.AU > # >- point the export at a netgroup (step3.3) > This works, even though the netgroup cannot be resolved > (there's no definition for it). > # showmount -e > Export list for install: > /data/INSTALL_1 @all_hosts > # > > NB nfs-kernel-server starts before dhclient > Sep 08 17:04:46 install nfs-kernel-server[657]: Exporting directories for NFS > kernel daemon > Sep 08 17:04:46 install kernel: NFSD: Using /var/lib/nfs/v4recovery as the > NFSv4 state recovery directory > Sep 08 17:04:46 install kernel: NFSD: starting 90-second grace period (net > 818ba280) > ... > Sep 08 17:04:46 install rpc.mountd[738]: Version 1.2.8 starting > Sep 08 17:04:46 install nfs-kernel-server[657]: Starting NFS kernel daemon: > nfsd mountd. > Sep 08 17:04:46 install acpid[734]: starting up with netlink and the input > layer > Sep 08 17:04:46 install acpid[734]: 1 rule loaded > Sep 08 17:04:46 install acpid[734]: waiting for events: event logging is off > ... > Sep 08 17:04:49 install kernel: e1000e: eth0 NIC Link is Up 1000 Mbps Full > Duplex, Flow Control: None > Sep 08 17:04:49 install kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link > becomes ready > Sep 08 17:04:50 install dhclient[532]: DHCPDISCOVER on eth0 to > 255.255.255.255 port 67 interval 6 > ... > Sep 08 17:04:56 install ifup[522]: Starting rpcbind (via systemctl): > rpcbind.service. > Sep 08 17:04:56 install ifup[522]: Starting nfs-common (via systemctl): > nfs-common.service. > Sep 08 17:04:56 install kernel: Key type dns_resolver registered > Sep 08 17:04:56 install kernel: NFS: Registering the id_resolver key type > Sep 08 17:04:56 install kernel: Key type id_resolver registered > Sep 08 17:04:56 install kernel: Key type id_legacy registered > > > step3.4 - install new nfs packages but don't reboot yet. > I didn't analyse this step carefully > > step 4 -
Bug#775541: tests of new packages
I found one further issue which is related to #738063. I wanted to limit the exports to supporting version 2 and 3. I first set RPCMOUNTDOPTS="--manage-gids --no-nfs-version 4" as explained in /etc/default/nfs-kernel-server. But I found I also needed add the following to that file: RPCNFSDPRIORITY=0 + # other options for nfsd + # To disable NFSv4 on the server, specify '--no-nfs-version 4' here also + RPCNFSDOPTS="--no-nfs-version 4" I discovered this by reading the source of the new script /usr/lib/systemd/scripts/nfs-utils_env.sh which constructs the RPCNFSDARGS variable used in the /lib/systemd/system/nfs-server.service file. If you upload a new version of this package, could you add an empty definition of RPCNFSDOPTS to /etc/default/nfs-kernel-server, so people can find it? Users of the init script would also benefit from this change. Thanks Vince
Bug#775541: tests of new packages
Thanks Andreas for those new packages. I did some testing of the 1.2.8-9.2 packages on a clean jessie install. They are pretty close but I found an issue with NFS exports in one case. I used the attached check.sh script to show the state of various targets as I changed things. The attached results.tar shows the output from it. Between each major stage below, the system is rebooted. Then I log in and run check.sh before doing anything else. Throughout, /etc/network/interfaces contained: source /etc/network/interfaces.d/* auto lo iface lo inet loopback allow-hotplug eth0 iface eth0 inet dhcp and /etc/network/interfaces.d/ was empty. step 1 - jessie system with nfs packages 1.2.8-9 one fstab mount, no exports. no changes to /etc/default/nfs-*. The fstab mount works ok; it tries to mount it way too early but we invoke it with the 'bg' option so eventually it works. step 2 - as above but with the _netdev option added to the fstab entry. System still tries to mount too early, before dhclient is done, but it works in the end. step 3 - as above but export one filesystem. The fstab mount works, as above. Tried a few different ways to do the exports - wildcard host (step3.1) This works ok. - point the export at a single host (step3.2) This tries to start mountd to early, before it can resolve the hostname given in the exports file. As a result we end up with an empty export list: # showmount -e Export list for install: # Restarting the service once the system is fully booted works # systemctl restart nfs-kernel-server.service # showmount -e Export list for install: /data/INSTALL_1 mayhem.atnf.CSIRO.AU # - point the export at a netgroup (step3.3) This works, even though the netgroup cannot be resolved (there's no definition for it). # showmount -e Export list for install: /data/INSTALL_1 @all_hosts # NB nfs-kernel-server starts before dhclient Sep 08 17:04:46 install nfs-kernel-server[657]: Exporting directories for NFS kernel daemon Sep 08 17:04:46 install kernel: NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory Sep 08 17:04:46 install kernel: NFSD: starting 90-second grace period (net 818ba280) ... Sep 08 17:04:46 install rpc.mountd[738]: Version 1.2.8 starting Sep 08 17:04:46 install nfs-kernel-server[657]: Starting NFS kernel daemon: nfsd mountd. Sep 08 17:04:46 install acpid[734]: starting up with netlink and the input layer Sep 08 17:04:46 install acpid[734]: 1 rule loaded Sep 08 17:04:46 install acpid[734]: waiting for events: event logging is off ... Sep 08 17:04:49 install kernel: e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None Sep 08 17:04:49 install kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Sep 08 17:04:50 install dhclient[532]: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 6 ... Sep 08 17:04:56 install ifup[522]: Starting rpcbind (via systemctl): rpcbind.service. Sep 08 17:04:56 install ifup[522]: Starting nfs-common (via systemctl): nfs-common.service. Sep 08 17:04:56 install kernel: Key type dns_resolver registered Sep 08 17:04:56 install kernel: NFS: Registering the id_resolver key type Sep 08 17:04:56 install kernel: Key type id_resolver registered Sep 08 17:04:56 install kernel: Key type id_legacy registered step3.4 - install new nfs packages but don't reboot yet. I didn't analyse this step carefully step 4 - jessie system with nfs packages 1.2.8-9.2 one fstab mount, one exported directory. rpc.mountd fails to start because it wants rpcbind to be there, but rpcbind has not been started yet. showmount -e fails: # showmount -e clnt_create: RPC: Program not registered step4.1 modify /lib/systemd/system/nfs-mountd.service, - After=network.target local-fs.target + After=rpcbind.target network.target local-fs.target run systemd daemon-reload but don't reboot. The exports are still not there # showmount -e clnt_create: RPC: Program not registered re run exports, it fails # exportfs -rav exporting @all_hosts:/data/INSTALL_1 # showmount -e clnt_create: RPC: Program not registered # step 5 - reboot after step4.1: usr-local.mount starts a little (1-2s) later. It starts just after ifup, but before dhclient has gotten an answer so it can't resolve the server hostname, but eventually the mount does work. Exports also work, in some cases: - wildcard case is OK (step5.1) # showmount -e Export list for install:
