** Description changed: - I noticed that `workshop stop` has been taking a long time recently, due - to systemd waiting for snapd.service to stop. It doesn't happen for - stock LXD containers, but Workshop modifies the stock containers. One - thing it does is disable cloud-init after the first boot, which seems to - cause this issue. + [Impact] + + * + + [Test Plan] + + * + + [Where problems could occur] + + * + + [Other Info] + + * + + ---original--- + I noticed that `workshop stop` has been taking a long time recently, due to systemd waiting for snapd.service to stop. It doesn't happen for stock LXD containers, but Workshop modifies the stock containers. One thing it does is disable cloud-init after the first boot, which seems to cause this issue. I narrowed it down to the attached reproducer. I also collected logs for `snapd.service` during a slow shutdown. It seems to me that disabling cloud-init removes the implicit dependency that `snapd.service` has on `network.target`. This causes the network to be stopped before snapd has finished shutting down. During the shutdown process it makes some HTTP requests, which time out after several retries. One fix would be to make snapd.service depend on network.target explicitly, but maybe there's a way to completely remove the HTTP requests from the shutdown process. Logs from `journalctl -b -1 -u snapd.service` after a slow `lxc stop`: ``` Sep 01 05:53:34 c1 systemd[1]: Stopping snapd.service - Snap Daemon... Sep 01 05:53:34 c1 snapd[262]: main.go:197: Exiting on terminated signal. Sep 01 05:53:37 c1 snapd[262]: standby.go:121: standby monitoring stop requested Sep 01 05:53:44 c1 snapd[262]: retry.go:122: DEBUG: Retrying because of: net/http: request canceled (Client.Timeout exceeded while awaiting headers) Sep 01 05:53:44 c1 snapd[262]: retry.go:114: DEBUG: ShouldRetryError: net/http: request canceled (Client.Timeout exceeded while awaiting headers) *http.httpError -> true Sep 01 05:53:44 c1 snapd[262]: retry.go:51: DEBUG: Retrying https://api.snapcraft.io/api/v1/snaps/names?confinement=strict%2Cclassic, attempt 2, elapsed time=10.006350989s Sep 01 05:53:44 c1 snapd[262]: retry.go:165: DEBUG: Retrying because of temporary net error: &net.OpError{Op:"dial", Net:"tcp", Source:net.Addr(nil), Addr:net.Addr(nil), Err:(*net.DNSError)(0xc00052d740)} Sep 01 05:53:44 c1 snapd[262]: retry.go:114: DEBUG: ShouldRetryError: dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving *net.OpError -> true Sep 01 05:53:45 c1 snapd[262]: retry.go:51: DEBUG: Retrying https://api.snapcraft.io/api/v1/snaps/names?confinement=strict%2Cclassic, attempt 3, elapsed time=11.257236692s Sep 01 05:53:45 c1 snapd[262]: retry.go:165: DEBUG: Retrying because of temporary net error: &net.OpError{Op:"dial", Net:"tcp", Source:net.Addr(nil), Addr:net.Addr(nil), Err:(*net.DNSError)(0xc00052d7c0)} Sep 01 05:53:45 c1 snapd[262]: retry.go:114: DEBUG: ShouldRetryError: dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving *net.OpError -> true Sep 01 05:53:48 c1 snapd[262]: retry.go:51: DEBUG: Retrying https://api.snapcraft.io/api/v1/snaps/names?confinement=strict%2Cclassic, attempt 4, elapsed time=14.381609704s Sep 01 05:53:48 c1 snapd[262]: retry.go:165: DEBUG: Retrying because of temporary net error: &net.OpError{Op:"dial", Net:"tcp", Source:net.Addr(nil), Addr:net.Addr(nil), Err:(*net.DNSError)(0xc0008dc900)} Sep 01 05:53:48 c1 snapd[262]: retry.go:114: DEBUG: ShouldRetryError: dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving *net.OpError -> true Sep 01 05:53:56 c1 snapd[262]: retry.go:51: DEBUG: Retrying https://api.snapcraft.io/api/v1/snaps/names?confinement=strict%2Cclassic, attempt 5, elapsed time=22.196202407s Sep 01 05:53:56 c1 snapd[262]: retry.go:63: DEBUG: The retry loop for https://api.snapcraft.io/api/v1/snaps/names?confinement=strict%2Cclassic finished after 5 retries, elapsed time=22.197780386s, status: Get "https://api.snapcraft.io/api/v1/snaps/names?confinement=strict%2Cclassic": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving Sep 01 05:53:56 c1 snapd[262]: catalogrefresh.go:144: DEBUG: Catalog refresh failed: Get "https://api.snapcraft.io/api/v1/snaps/names?confinement=strict%2Cclassic": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving. Sep 01 05:53:56 c1 snapd[262]: snapmgr.go:1675: performing periodic snap downloads cache cleanup Sep 01 05:53:56 c1 snapd[262]: snapmgr.go:1685: cannot clean store downloads cache: open /var/lib/snapd/cache: no such file or directory Sep 01 05:53:56 c1 snapd[262]: stateengine.go:161: state ensure error: Get "https://api.snapcraft.io/api/v1/snaps/names?confinement=strict%2Cclassic": dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving Sep 01 05:53:56 c1 snapd[262]: autorepair.go:177: WARNING: the system booted with an old initrd without unlocked status reporting Sep 01 05:53:56 c1 snapd[262]: certmgr.go:72: DEBUG: ca-certificate database has already been generated, skipping generation Sep 01 05:53:56 c1 snapd[262]: overlord.go:568: Released state lock file Sep 01 05:53:56 c1 snapd[262]: seclog.go:189: security logger disabled Sep 01 05:53:56 c1 systemd[1]: snapd.service: Deactivated successfully. Sep 01 05:53:56 c1 systemd[1]: Stopped snapd.service - Snap Daemon. ``` Logs from reproducer script: ``` Creating c1 Baseline: takes around 4s for me, consistently real 0m3.964s user 0m0.044s sys 0m0.023s real 0m3.948s user 0m0.033s sys 0m0.022s real 0m3.952s user 0m0.042s sys 0m0.021s real 0m3.959s user 0m0.028s sys 0m0.032s Bug: sometimes takes more than 20s real 0m22.510s user 0m0.050s sys 0m0.026s real 0m23.008s user 0m0.033s sys 0m0.031s real 0m22.525s user 0m0.049s sys 0m0.026s real 0m22.443s user 0m0.040s sys 0m0.028s real 0m23.050s user 0m0.041s sys 0m0.024s real 0m22.584s user 0m0.050s sys 0m0.020s real 0m22.435s user 0m0.032s sys 0m0.021s real 0m22.607s user 0m0.052s sys 0m0.026s Workaround: back to baseline real 0m3.963s user 0m0.048s sys 0m0.027s real 0m3.977s user 0m0.048s sys 0m0.027s real 0m3.993s user 0m0.045s sys 0m0.028s real 0m3.948s user 0m0.045s sys 0m0.020s ```
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2165972 Title: snapd network requests slow down LXD container stop To manage notifications about this bug go to: https://bugs.launchpad.net/snapd/+bug/2165972/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
