** Description changed:

- I noticed that `workshop stop` has been taking a long time recently, due
- to systemd waiting for snapd.service to stop. It doesn't happen for
- stock LXD containers, but Workshop modifies the stock containers. One
- thing it does is disable cloud-init after the first boot, which seems to
- cause this issue.
+ [Impact]
+ 
+  *
+ 
+ [Test Plan]
+ 
+  *
+ 
+ [Where problems could occur]
+ 
+  *
+ 
+ [Other Info]
+ 
+  *
+ 
+ ---original---
+ I noticed that `workshop stop` has been taking a long time recently, due to 
systemd waiting for snapd.service to stop. It doesn't happen for stock LXD 
containers, but Workshop modifies the stock containers. One thing it does is 
disable cloud-init after the first boot, which seems to cause this issue.
  
  I narrowed it down to the attached reproducer. I also collected logs for
  `snapd.service` during a slow shutdown. It seems to me that disabling
  cloud-init removes the implicit dependency that `snapd.service` has on
  `network.target`. This causes the network to be stopped before snapd has
  finished shutting down. During the shutdown process it makes some HTTP
  requests, which time out after several retries.
  
  One fix would be to make snapd.service depend on network.target
  explicitly, but maybe there's a way to completely remove the HTTP
  requests from the shutdown process.
  
  
  Logs from `journalctl -b -1 -u snapd.service` after a slow `lxc stop`:
  ```
  Sep 01 05:53:34 c1 systemd[1]: Stopping snapd.service - Snap Daemon...
  Sep 01 05:53:34 c1 snapd[262]: main.go:197: Exiting on terminated signal.
  Sep 01 05:53:37 c1 snapd[262]: standby.go:121: standby monitoring stop 
requested
  Sep 01 05:53:44 c1 snapd[262]: retry.go:122: DEBUG: Retrying because of: 
net/http: request canceled (Client.Timeout exceeded while awaiting headers)
  Sep 01 05:53:44 c1 snapd[262]: retry.go:114: DEBUG: ShouldRetryError: 
net/http: request canceled (Client.Timeout exceeded while awaiting headers) 
*http.httpError -> true
  Sep 01 05:53:44 c1 snapd[262]: retry.go:51: DEBUG: Retrying 
https://api.snapcraft.io/api/v1/snaps/names?confinement=strict%2Cclassic, 
attempt 2, elapsed time=10.006350989s
  Sep 01 05:53:44 c1 snapd[262]: retry.go:165: DEBUG: Retrying because of 
temporary net error: &net.OpError{Op:"dial", Net:"tcp", Source:net.Addr(nil), 
Addr:net.Addr(nil), Err:(*net.DNSError)(0xc00052d740)}
  Sep 01 05:53:44 c1 snapd[262]: retry.go:114: DEBUG: ShouldRetryError: dial 
tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving *net.OpError 
-> true
  Sep 01 05:53:45 c1 snapd[262]: retry.go:51: DEBUG: Retrying 
https://api.snapcraft.io/api/v1/snaps/names?confinement=strict%2Cclassic, 
attempt 3, elapsed time=11.257236692s
  Sep 01 05:53:45 c1 snapd[262]: retry.go:165: DEBUG: Retrying because of 
temporary net error: &net.OpError{Op:"dial", Net:"tcp", Source:net.Addr(nil), 
Addr:net.Addr(nil), Err:(*net.DNSError)(0xc00052d7c0)}
  Sep 01 05:53:45 c1 snapd[262]: retry.go:114: DEBUG: ShouldRetryError: dial 
tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving *net.OpError 
-> true
  Sep 01 05:53:48 c1 snapd[262]: retry.go:51: DEBUG: Retrying 
https://api.snapcraft.io/api/v1/snaps/names?confinement=strict%2Cclassic, 
attempt 4, elapsed time=14.381609704s
  Sep 01 05:53:48 c1 snapd[262]: retry.go:165: DEBUG: Retrying because of 
temporary net error: &net.OpError{Op:"dial", Net:"tcp", Source:net.Addr(nil), 
Addr:net.Addr(nil), Err:(*net.DNSError)(0xc0008dc900)}
  Sep 01 05:53:48 c1 snapd[262]: retry.go:114: DEBUG: ShouldRetryError: dial 
tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving *net.OpError 
-> true
  Sep 01 05:53:56 c1 snapd[262]: retry.go:51: DEBUG: Retrying 
https://api.snapcraft.io/api/v1/snaps/names?confinement=strict%2Cclassic, 
attempt 5, elapsed time=22.196202407s
  Sep 01 05:53:56 c1 snapd[262]: retry.go:63: DEBUG: The retry loop for 
https://api.snapcraft.io/api/v1/snaps/names?confinement=strict%2Cclassic 
finished after 5 retries, elapsed time=22.197780386s, status: Get 
"https://api.snapcraft.io/api/v1/snaps/names?confinement=strict%2Cclassic": 
dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
  Sep 01 05:53:56 c1 snapd[262]: catalogrefresh.go:144: DEBUG: Catalog refresh 
failed: Get 
"https://api.snapcraft.io/api/v1/snaps/names?confinement=strict%2Cclassic": 
dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving.
  Sep 01 05:53:56 c1 snapd[262]: snapmgr.go:1675: performing periodic snap 
downloads cache cleanup
  Sep 01 05:53:56 c1 snapd[262]: snapmgr.go:1685: cannot clean store downloads 
cache: open /var/lib/snapd/cache: no such file or directory
  Sep 01 05:53:56 c1 snapd[262]: stateengine.go:161: state ensure error: Get 
"https://api.snapcraft.io/api/v1/snaps/names?confinement=strict%2Cclassic": 
dial tcp: lookup api.snapcraft.io on 127.0.0.53:53: server misbehaving
  Sep 01 05:53:56 c1 snapd[262]: autorepair.go:177: WARNING: the system booted 
with an old initrd without unlocked status reporting
  Sep 01 05:53:56 c1 snapd[262]: certmgr.go:72: DEBUG: ca-certificate database 
has already been generated, skipping generation
  Sep 01 05:53:56 c1 snapd[262]: overlord.go:568: Released state lock file
  Sep 01 05:53:56 c1 snapd[262]: seclog.go:189: security logger disabled
  Sep 01 05:53:56 c1 systemd[1]: snapd.service: Deactivated successfully.
  Sep 01 05:53:56 c1 systemd[1]: Stopped snapd.service - Snap Daemon.
  ```
  
  Logs from reproducer script:
  ```
  Creating c1
  Baseline: takes around 4s for me, consistently
  
  real  0m3.964s
  user  0m0.044s
  sys   0m0.023s
  
  real  0m3.948s
  user  0m0.033s
  sys   0m0.022s
  
  real  0m3.952s
  user  0m0.042s
  sys   0m0.021s
  
  real  0m3.959s
  user  0m0.028s
  sys   0m0.032s
  Bug: sometimes takes more than 20s
  
  real  0m22.510s
  user  0m0.050s
  sys   0m0.026s
  
  real  0m23.008s
  user  0m0.033s
  sys   0m0.031s
  
  real  0m22.525s
  user  0m0.049s
  sys   0m0.026s
  
  real  0m22.443s
  user  0m0.040s
  sys   0m0.028s
  
  real  0m23.050s
  user  0m0.041s
  sys   0m0.024s
  
  real  0m22.584s
  user  0m0.050s
  sys   0m0.020s
  
  real  0m22.435s
  user  0m0.032s
  sys   0m0.021s
  
  real  0m22.607s
  user  0m0.052s
  sys   0m0.026s
  Workaround: back to baseline
  
  real  0m3.963s
  user  0m0.048s
  sys   0m0.027s
  
  real  0m3.977s
  user  0m0.048s
  sys   0m0.027s
  
  real  0m3.993s
  user  0m0.045s
  sys   0m0.028s
  
  real  0m3.948s
  user  0m0.045s
  sys   0m0.020s
  ```

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2165972

Title:
  snapd network requests slow down LXD container stop

To manage notifications about this bug go to:
https://bugs.launchpad.net/snapd/+bug/2165972/+subscriptions


-- 
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

Reply via email to