[Bug 2054616] Re: The systemd service for NBD Client doesn't come up on boot
Thanks for the additional details and for the improved reproducer. I can reproduce the issue and looks like the steps you identified are right. I assume this worked at some point, but the issue is also present in Focal and Bionic. I don't think this is working as expected, OTOH the RequiredBy= (and, in earlier releases, Before=) systemd dependencies come from the upstream project [1], where I see no bug reports about this issue (not even closed ones). Same goes for Debian: no bugs about this. I am going to triage this as a bug, however to speed things up I suggest filing an upstream bug [2]. Given that the nbd@. units come from there, the upstream devs may be able to help. @Thiago would you be willing to do so? Thanks! [1] https://github.com/NetworkBlockDevice/nbd/ [2] https://github.com/NetworkBlockDevice/nbd/issues [3] https://github.com/NetworkBlockDevice/nbd/tree/master/systemd ** Changed in: nbd (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2054616 Title: The systemd service for NBD Client doesn't come up on boot To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nbd/+bug/2054616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2054616] Re: The systemd service for NBD Client doesn't come up on boot
** Description changed: The `systemd` service for starting the NBD Clients on boot isn't working. ### Steps to Reproduce: 0. Install NBD - apt install nbd-server nbd-client - - 1. Prepare an NBD server with a disk image to share. For instance: - truncate -s 1T /vol-0.img - - - 2. Configure the NBD server by adding the following to `/etc/nbd-server/conf.d/exports.conf`: + 2. Configure the NBD server by adding the following to `/etc/nbd- + server/conf.d/exports.conf`: - [vol0] exportname = /vol-0.img - - 3. Configuring the client connection in `/etc/nbdtab`: Contents of `/etc/nbdtab`: - nbd0 localhost vol0 - - 4. Enable it on `systemd`: - systemctl enable nbd@nbd0 - Here's what I see: - # systemctl enable nbd@nbd0 Created symlink /etc/systemd/system/dev-nbd0.device.requires/nbd@nbd0.service → /lib/systemd/system/nbd@.service. Unit /lib/systemd/system/nbd@.service is added as a dependency to a non-existent unit dev-nbd0.device. - - 5. Add `nbd` auto-load to system'd boot: echo nbd > /etc/modules-load.d/nbd.conf - 6. Reboot - sudo reboot - - 7. The service not coming up on the system's boot. Look: After rebooting the system, the NBD clien for `nbd0` isn't running (but it's enabled on systemd). - - root@cephao-1-ceph-osd-1:~# systemctl status nbd@nbd0 + root@demo-nbd-test:~# systemctl status nbd@nbd0 ○ nbd@nbd0.service - NBD client connection for nbd0 Loaded: loaded (/lib/systemd/system/nbd@.service; enabled; vendor preset: enabled) Active: inactive (dead) Docs: man:nbd-client - Feb 21 22:54:22 cephao-1-ceph-osd-1 systemd[1]: nbd@nbd0.service: Dependency Before=dev-nbd0.device ignored (.device units cannot be delayed) + Feb 21 22:54:22 demo-nbd-test systemd[1]: nbd@nbd0.service: Dependency Before=dev-nbd0.device ignored (.device units cannot be delayed) - But if you try to start it again. It'll work: - - root@cephao-1-ceph-osd-1:~# systemctl start nbd@nbd0 - root@cephao-1-ceph-osd-1:~# systemctl status nbd@nbd0 + root@demo-nbd-test:~# systemctl start nbd@nbd0 + root@demo-nbd-test:~# systemctl status nbd@nbd0 ● nbd@nbd0.service - NBD client connection for nbd0 Loaded: loaded (/lib/systemd/system/nbd@.service; enabled; vendor preset: enabled) Active: active (exited) since Wed 2024-02-21 22:57:37 UTC; 2s ago Docs: man:nbd-client Process: 1096 ExecStart=//sbin/nbd-client nbd0 (code=exited, status=0/SUCCESS) Main PID: 1096 (code=exited, status=0/SUCCESS) CPU: 4ms - Feb 21 22:57:37 cephao-1-ceph-osd-1 systemd[1]: Starting NBD client connection for nbd0... - Feb 21 22:57:37 cephao-1-ceph-osd-1 nbd-client[1096]: Negotiation: ..size = 3145728MB - Feb 21 22:57:37 cephao-1-ceph-osd-1 nbd-client[1096]: Connected /dev/nbd0 - Feb 21 22:57:37 cephao-1-ceph-osd-1 systemd[1]: Finished NBD client connection for nbd0. + Feb 21 22:57:37 demo-nbd-test systemd[1]: Starting NBD client connection for nbd0... + Feb 21 22:57:37 demo-nbd-test nbd-client[1096]: Negotiation: ..size = 3145728MB + Feb 21 22:57:37 demo-nbd-test nbd-client[1096]: Connected /dev/nbd0 + Feb 21 22:57:37 demo-nbd-test systemd[1]: Finished NBD client connection for nbd0. - So this forces me to have the following `/etc/rc.local` file as a workaround: - #!/bin/bash sleep 6 systemctl start nbd@nbd0 - Make it executable: - chmod +x /etc/rc.local - Reboot again, it'll be started on next boot! I don't think that the `/etc/rc.local` should be part of this procedure... =P NOTE: Don't need to run `update-initramfs -k all -u`. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2054616 Title: The systemd service for NBD Client doesn't come up on boot To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nbd/+bug/2054616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2054616] Re: The systemd service for NBD Client doesn't come up on boot
** Description changed: The `systemd` service for starting the NBD Clients on boot isn't working. ### Steps to Reproduce: 0. Install NBD -``` -apt install nbd-server nbd-client -``` + - + apt install nbd-server nbd-client + - + 1. Prepare an NBD server with a disk image to share. For instance: -```bash -truncate -s 1T /vol-0.img -``` + + - + truncate -s 1T /vol-0.img + - + 2. Configure the NBD server by adding the following to `/etc/nbd-server/conf.d/exports.conf`: -``` -[vol0] -exportname = /vol-0.img -``` + + - + [vol0] + exportname = /vol-0.img + - + 3. Configuring the client connection in `/etc/nbdtab`: Contents of `/etc/nbdtab`: -``` -nbd0 localhost vol0 -``` + - + nbd0 localhost vol0 + - + 4. Enable it on `systemd`: -```bash -systemctl enable nbd@nbd0 -``` + - + systemctl enable nbd@nbd0 + - Here's what I see: - ``` + - # systemctl enable nbd@nbd0 Created symlink /etc/systemd/system/dev-nbd0.device.requires/nbd@nbd0.service → /lib/systemd/system/nbd@.service. Unit /lib/systemd/system/nbd@.service is added as a dependency to a non-existent unit dev-nbd0.device. - ``` + - + 5. Add `nbd` auto-load to system'd boot: - ``` - echo nbd > /etc/modules-load.d/nbd.conf - ``` + echo nbd > /etc/modules-load.d/nbd.conf + 6. Reboot -``` -sudo reboot -``` + - + sudo reboot + - + 7. The service not coming up on the system's boot. Look: After rebooting the system, the NBD clien for `nbd0` isn't running (but it's enabled on systemd). - ``` + - root@cephao-1-ceph-osd-1:~# systemctl status nbd@nbd0 ○ nbd@nbd0.service - NBD client connection for nbd0 - Loaded: loaded (/lib/systemd/system/nbd@.service; enabled; vendor preset: enabled) - Active: inactive (dead) -Docs: man:nbd-client + Loaded: loaded (/lib/systemd/system/nbd@.service; enabled; vendor preset: enabled) + Active: inactive (dead) + Docs: man:nbd-client Feb 21 22:54:22 cephao-1-ceph-osd-1 systemd[1]: nbd@nbd0.service: Dependency Before=dev-nbd0.device ignored (.device units cannot be delayed) - ``` + - But if you try to start it again. It'll work: - ``` + - root@cephao-1-ceph-osd-1:~# systemctl start nbd@nbd0 root@cephao-1-ceph-osd-1:~# systemctl status nbd@nbd0 ● nbd@nbd0.service - NBD client connection for nbd0 - Loaded: loaded (/lib/systemd/system/nbd@.service; enabled; vendor preset: enabled) - Active: active (exited) since Wed 2024-02-21 22:57:37 UTC; 2s ago -Docs: man:nbd-client - Process: 1096 ExecStart=//sbin/nbd-client nbd0 (code=exited, status=0/SUCCESS) -Main PID: 1096 (code=exited, status=0/SUCCESS) - CPU: 4ms + Loaded: loaded (/lib/systemd/system/nbd@.service; enabled; vendor preset: enabled) + Active: active (exited) since Wed 2024-02-21 22:57:37 UTC; 2s ago + Docs: man:nbd-client + Process: 1096 ExecStart=//sbin/nbd-client nbd0 (code=exited, status=0/SUCCESS) + Main PID: 1096 (code=exited, status=0/SUCCESS) + CPU: 4ms Feb 21 22:57:37 cephao-1-ceph-osd-1 systemd[1]: Starting NBD client connection for nbd0... Feb 21 22:57:37 cephao-1-ceph-osd-1 nbd-client[1096]: Negotiation: ..size = 3145728MB Feb 21 22:57:37 cephao-1-ceph-osd-1 nbd-client[1096]: Connected /dev/nbd0 Feb 21 22:57:37 cephao-1-ceph-osd-1 systemd[1]: Finished NBD client connection for nbd0. - ``` + - So this forces me to have the following `/etc/rc.local` file as a workaround: - ``` + - #!/bin/bash sleep 6 systemctl start nbd@nbd0 - ``` + - Make it executable: - ``` + - chmod +x /etc/rc.local - ``` + - Reboot again, it'll be started on next boot! I don't think that the `/etc/rc.local` should be part of this procedure... =P NOTE: Don't need to run `update-initramfs -k all -u`. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2054616 Title: The systemd service for NBD Client doesn't come up on boot To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nbd/+bug/2054616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2054616] Re: The systemd service for NBD Client doesn't come up on boot
I've updated the description to include all the steps necessary to reproduce the issue on Ubuntu 22.04 (Cloud Image - Server Minimal install). This seems to be a `systemd` thing! lol ** Description changed: The `systemd` service for starting the NBD Clients on boot isn't working. ### Steps to Reproduce: + + 0. Install NBD + +``` +apt install nbd-server nbd-client +``` 1. Prepare an NBD server with a disk image to share. For instance: ```bash truncate -s 1T /vol-0.img ``` 2. Configure the NBD server by adding the following to `/etc/nbd-server/conf.d/exports.conf`: ``` [vol0] exportname = /vol-0.img ``` 3. Configuring the client connection in `/etc/nbdtab`: Contents of `/etc/nbdtab`: ``` nbd0 localhost vol0 ``` 4. Enable it on `systemd`: ```bash systemctl enable nbd@nbd0 ``` - 5. The service not coming up on the system's boot. Look: + Here's what I see: + + ``` + # systemctl enable nbd@nbd0 + Created symlink /etc/systemd/system/dev-nbd0.device.requires/nbd@nbd0.service → /lib/systemd/system/nbd@.service. + Unit /lib/systemd/system/nbd@.service is added as a dependency to a non-existent unit dev-nbd0.device. + ``` + + 5. Add `nbd` auto-load to system'd boot: + + ``` + echo nbd > /etc/modules-load.d/nbd.conf + ``` + + 6. Reboot + +``` +sudo reboot +``` + + 7. The service not coming up on the system's boot. Look: After rebooting the system, the NBD clien for `nbd0` isn't running (but it's enabled on systemd). ``` root@cephao-1-ceph-osd-1:~# systemctl status nbd@nbd0 ○ nbd@nbd0.service - NBD client connection for nbd0 - Loaded: loaded (/lib/systemd/system/nbd@.service; enabled; vendor preset: enabled) - Active: inactive (dead) - Docs: man:nbd-client + Loaded: loaded (/lib/systemd/system/nbd@.service; enabled; vendor preset: enabled) + Active: inactive (dead) +Docs: man:nbd-client Feb 21 22:54:22 cephao-1-ceph-osd-1 systemd[1]: nbd@nbd0.service: Dependency Before=dev-nbd0.device ignored (.device units cannot be delayed) ``` - Then, after a restart, it works: + But if you try to start it again. It'll work: ``` - root@cephao-1-ceph-osd-1:~# systemctl restart nbd@nbd0 + root@cephao-1-ceph-osd-1:~# systemctl start nbd@nbd0 root@cephao-1-ceph-osd-1:~# systemctl status nbd@nbd0 ● nbd@nbd0.service - NBD client connection for nbd0 - Loaded: loaded (/lib/systemd/system/nbd@.service; enabled; vendor preset: enabled) - Active: active (exited) since Wed 2024-02-21 22:57:37 UTC; 2s ago - Docs: man:nbd-client - Process: 1096 ExecStart=//sbin/nbd-client nbd0 (code=exited, status=0/SUCCESS) - Main PID: 1096 (code=exited, status=0/SUCCESS) - CPU: 4ms + Loaded: loaded (/lib/systemd/system/nbd@.service; enabled; vendor preset: enabled) + Active: active (exited) since Wed 2024-02-21 22:57:37 UTC; 2s ago +Docs: man:nbd-client + Process: 1096 ExecStart=//sbin/nbd-client nbd0 (code=exited, status=0/SUCCESS) +Main PID: 1096 (code=exited, status=0/SUCCESS) + CPU: 4ms Feb 21 22:57:37 cephao-1-ceph-osd-1 systemd[1]: Starting NBD client connection for nbd0... Feb 21 22:57:37 cephao-1-ceph-osd-1 nbd-client[1096]: Negotiation: ..size = 3145728MB Feb 21 22:57:37 cephao-1-ceph-osd-1 nbd-client[1096]: Connected /dev/nbd0 Feb 21 22:57:37 cephao-1-ceph-osd-1 systemd[1]: Finished NBD client connection for nbd0. ``` So this forces me to have the following `/etc/rc.local` file as a workaround: ``` #!/bin/bash - systemctl restart nbd@nbd0 + sleep 6 + systemctl start nbd@nbd0 ``` - If this is not the proper way of automatically starting a `nbd0` NBD - client, please, let me know! Perhaps documentation should be more clear. + Make it executable: - Interesting reference: - https://bugs.launchpad.net/ubuntu/+source/nbd/+bug/2054470 + ``` + chmod +x /etc/rc.local + ``` + + Reboot again, it'll be started on next boot! + + I don't think that the `/etc/rc.local` should be part of this + procedure... =P + + NOTE: Don't need to run `update-initramfs -k all -u`. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2054616 Title: The systemd service for NBD Client doesn't come up on boot To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nbd/+bug/2054616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2054616] Re: The systemd service for NBD Client doesn't come up on boot
I'm testing on Ubuntu 22.04.4 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2054616 Title: The systemd service for NBD Client doesn't come up on boot To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nbd/+bug/2054616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2054616] Re: The systemd service for NBD Client doesn't come up on boot
Hi Mitchell, No worries at all! I forgot to mention another step I took: ``` echo nbd > /etc/modules-load.d/nbd.conf ``` Otherwise it won't load the module on boot. Perhaps there's also a need to `update-initramfs -k all -u`, not 100% sure. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2054616 Title: The systemd service for NBD Client doesn't come up on boot To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nbd/+bug/2054616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2054616] Re: The systemd service for NBD Client doesn't come up on boot
Wouter, is my assumption correct that ideally we should see the devices available on boot if we enable nbd@nbd0? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2054616 Title: The systemd service for NBD Client doesn't come up on boot To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nbd/+bug/2054616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2054616] Re: The systemd service for NBD Client doesn't come up on boot
I've been playing around with noble and I've noticed that sometimes if I try to restart the service right after boot it will fail root@n-vm:~# systemctl restart nbd@nbd0 Job for nbd@nbd0.service failed because the control process exited with error code. See "systemctl status nbd@nbd0.service" and "journalctl -xeu nbd@nbd0.service" for details. root@n-vm:~# systemctl status nbd@nbd0 × nbd@nbd0.service - NBD client connection for nbd0 Loaded: loaded (/usr/lib/systemd/system/nbd@.service; enabled; preset: enabled) Active: failed (Result: exit-code) since Tue 2024-02-27 23:44:35 UTC; 2s ago Docs: man:nbd-client Process: 871 ExecStart=//sbin/nbd-client nbd0 (code=exited, status=1/FAILURE) Main PID: 871 (code=exited, status=1/FAILURE) CPU: 5ms Feb 27 23:44:35 n-vm systemd[1]: Starting nbd@nbd0.service - NBD client connection for nbd0... Feb 27 23:44:35 n-vm nbd-client[871]: Negotiation: ..size = 1024MB Feb 27 23:44:35 n-vm nbd_client[871]: Failed to setup device, check dmesg Feb 27 23:44:35 n-vm nbd-client[871]: Error: Failed to setup device, check dmesg Feb 27 23:44:35 n-vm nbd-client[871]: Exiting. Feb 27 23:44:35 n-vm nbd_client[871]: Exiting. Feb 27 23:44:35 n-vm systemd[1]: nbd@nbd0.service: Main process exited, code=exited, status=1/FAILURE Feb 27 23:44:35 n-vm systemd[1]: nbd@nbd0.service: Failed with result 'exit-code'. Feb 27 23:44:35 n-vm systemd[1]: Failed to start nbd@nbd0.service - NBD client connection for nbd0. root@n-vm:~# dmesg | tail -n 1 [ 12.032858] nbd: nbd0 already in use But yeah, it just seems that systemd ignores the start because it doesn't want to wait for the devices: Feb 27 23:44:31 n-vm systemd[1]: nbd@nbd0.service: Dependency Before=dev-nbd0.device ignored (.device units cannot be delayed) You can force a start (We should not do this, but just showing for debug) by doing something such as: # echo "WantedBy=multi-user.target" >> /usr/lib/systemd/system/nbd@.service && systemctl enable nbd@nbd0 But then you see the client fails to start, because the devices are not available at that point. root@n-vm:~# systemctl status nbd@nbd0 × nbd@nbd0.service - NBD client connection for nbd0 Loaded: loaded (/usr/lib/systemd/system/nbd@.service; enabled; preset: enabled) Active: failed (Result: exit-code) since Tue 2024-02-27 23:48:22 UTC; 1s ago Docs: man:nbd-client Process: 663 ExecStart=//sbin/nbd-client nbd0 (code=exited, status=1/FAILURE) Main PID: 663 (code=exited, status=1/FAILURE) CPU: 5ms Feb 27 23:48:22 n-vm systemd[1]: Starting nbd@nbd0.service - NBD client connection for nbd0... Feb 27 23:48:22 n-vm nbd_client[663]: Socket failed: Connection refused I suppose the expectation here is that when you run `systemctl enable nbd@nbd0` that the /dev/nbd* devices should be instantiated at boot and the client should run and use those devices, but the devices are not coming up at boot. root@n-vm:~# ll /dev/nbd* ls: cannot access '/dev/nbd*': No such file or directory The interesting thing I am noticing is that the devices are instantiated after the first restart, so even though you see a failure the first time, restarting it a second time makes everything seem happy. root@n-vm:~# systemctl restart nbd@nbd0 Job for nbd@nbd0.service failed because the control process exited with error code. See "systemctl status nbd@nbd0.service" and "journalctl -xeu nbd@nbd0.service" for details. root@n-vm:~# ll /dev/nbd* | head -n 2 brw-rw 1 root disk 43, 0 Feb 27 23:55 /dev/nbd0 brw-rw 1 root disk 43, 16 Feb 27 23:54 /dev/nbd1 root@n-vm:~# systemctl restart nbd@nbd0 root@n-vm:~# systemctl status nbd@nbd0 ● nbd@nbd0.service - NBD client connection for nbd0 Loaded: loaded (/usr/lib/systemd/system/nbd@.service; enabled; preset: enabled) Active: active (exited) since Tue 2024-02-27 23:55:01 UTC; 4s ago Docs: man:nbd-client Process: 933 ExecStart=//sbin/nbd-client nbd0 (code=exited, status=0/SUCCESS) Main PID: 933 (code=exited, status=0/SUCCESS) CPU: 4ms -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2054616 Title: The systemd service for NBD Client doesn't come up on boot To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nbd/+bug/2054616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2054616] Re: The systemd service for NBD Client doesn't come up on boot
Thiago, Apologies if I missed it somewhere, but which Ubuntu release are you using for your tests? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2054616 Title: The systemd service for NBD Client doesn't come up on boot To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nbd/+bug/2054616/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
