Bug#821037: adt-virt-lxc: leaves used containers behind

[Paul Gevers]

Hi all,

On Sun, 22 Mar 2020 12:17:08 +0100 Paul Gevers  wrote:
> We're seeing this quite a lot currently on our ci.d.n infrastructure,
> and maybe also on Ubuntu's infra, as reported in bug #908193 (I'm not
> sure if that is actually a duplicate or different.)

Another example:

https://ci.debian.net/data/autopkgtest/testing/amd64/o/openjdk-15/7742200/log.gz

jaxp SKIP exit status 77 and marked as skippable
autopkgtest [19:39:42]: ERROR: "rm -rf
/tmp/autopkgtest-lxc.9db11r52/downtmp/jaxp-artifacts
/tmp/autopkgtest-lxc.9db11r52/downtmp/autopkgtest_tmp" failed with
stderr "rm: cannot remove
'/tmp/autopkgtest-lxc.9db11r52/downtmp/autopkgtest_tmp/hotspot/JTwork':
Directory not empty
"

I checked the worker, it had a tree in /tmp with (however, this is
probably from another run):
admin@ci-worker05:/tmp/autopkgtest-lxc._w5283nq/downtmp/autopkgtest_tmp/hotspot/JTwork$
ls -al
total 24
drwxr-xr-x  6 admin admin 4096 Oct 25 15:55 .
drwxr-xr-x  3 admin admin 4096 Oct 25 15:55 ..
drwxr-xr-x  3 admin admin 4096 Oct 25 15:55 classes
drwxr-xr-x 11 admin admin 4096 Oct 25 15:55 runtime
drwxr-xr-x  3 admin admin 4096 Oct 25 15:55 scratch
drwxr-xr-x 12 admin admin 4096 Oct 25 15:55 serviceability

There was one lxc present:
admin@ci-worker05:~$ sudo lxc-ls -f
NAMESTATE   AUTOSTART GROUPS IPV4 IPV6 UNPRIVILEGED
autopkgtest-oldstable-amd64 STOPPED 0 -  --false
autopkgtest-stable-amd64STOPPED 0 -  --false
autopkgtest-testing-amd64   STOPPED 0 -  --false
autopkgtest-unstable-amd64  STOPPED 0 -  --false
ci-298-ba5278af STOPPED 0 -  --false

But I could not start it:admin@ci-worker05:~$ sudo lxc-start ci-298-ba5278af
lxc-start: ci-298-ba5278af: lxccontainer.c: wait_on_daemonized_start:
851 Received container state "ABORTING" instead of "RUNNING"
lxc-start: ci-298-ba5278af: tools/lxc_start.c: main: 329 The container
failed to start
lxc-start: ci-298-ba5278af: tools/lxc_start.c: main: 332 To get more
details, run the container in foreground mode
lxc-start: ci-298-ba5278af: tools/lxc_start.c: main: 335 Additional
information can be obtained by setting the --logfile and --logpriority
options
admin@ci-worker05:~$ sudo lxc-start ci-298-ba5278af --foreground
lxc-start: ci-298-ba5278af: start.c: proc_pidfd_open: 1619 Function not
implemented - Failed to send signal through pidfd

 lxc-start:
ci-298-ba5278af: utils.c: safe_mount: 1225 Permission denied - Failed to
mount "proc" onto "/proc"

  lxc-start: ci-298-ba5278af: conf.c:
lxc_mount_auto_mounts: 728 Permission denied - Failed to mount "proc" on
"/proc" with flags 14

lxc-start: ci-298-ba5278af: conf.c: lxc_setup:
3561 Failed to setup first automatic mounts
  lxc-start:
ci-298-ba5278af: start.c: do_start: 1311 Failed to setup container
"ci-298-ba5278af"
 lxc-start: ci-298-ba5278af: sync.c: __sync_wait: 62 An
error occurred in another process (expected sequence number 5)
  lxc-start: ci-298-ba5278af: start.c: __lxc_start: 2031 Failed to spawn
container "ci-298-ba5278af"

lxc-start: ci-298-ba5278af:
tools/lxc_start.c: main: 329 The container failed to start
lxc-start: ci-298-ba5278af: tools/lxc_start.c: main: 335 Additional
information can be obtained by setting the --logfile and --logpriority
options


Paul




signature.asc
Description: OpenPGP digital signature

Bug#821037: adt-virt-lxc: leaves used containers behind

[Paul Gevers]

tag 821037 - moreinfo
thanks

Hi,

On Tue, 26 Apr 2016 08:57:13 +0200 Martin Pitt  wrote:
> Antonio Terceiro [2016-04-14 18:06 -0300]:
> > When using adt-virt-lxc, sometimes containers will not be properly shut 
> > down,
> > or not properly destroyed.
> 
> Do you have the corresponding adt-run logs? I'd like to check what
> happened there, either the cleanup doesn't happen in some exit path,
> or the cleanup actually fails.

We're seeing this quite a lot currently on our ci.d.n infrastructure,
and maybe also on Ubuntu's infra, as reported in bug #908193 (I'm not
sure if that is actually a duplicate or different.)

E.g. a recent log [1] ends like this:
autopkgtest [21:21:18]: test compile-policy:  - - - - - - - - - -
results - - - - - - - - - -
compile-policy   PASS
autopkgtest [21:21:18]: test test-installed: preparing testbed
: failure: ['sudo', 'timeout', '600', 'lxc-stop',
'--quiet', '--kill', '--name', 'ci-081-36080d53'] failed (exit status
124, stderr '')
autopkgtest [21:31:18]: ERROR: testbed failure: cannot send to testbed:
[Errno 32] Broken pipe

The container is still running.

Paul

[1]
https://ci.debian.net/data/autopkgtest/testing/arm64/a/apparmor/4616701/log.gz



signature.asc
Description: OpenPGP digital signature

Bug#821037: adt-virt-lxc: leaves used containers behind

[Martin Pitt]

Control: tag -1 moreinfo

Hello Antonio,

Antonio Terceiro [2016-04-14 18:06 -0300]:
> When using adt-virt-lxc, sometimes containers will not be properly shut down,
> or not properly destroyed.

Do you have the corresponding adt-run logs? I'd like to check what
happened there, either the cleanup doesn't happen in some exit path,
or the cleanup actually fails.

Thanks,

Martin


-- 
Martin Pitt| http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)


signature.asc
Description: PGP signature

Bug#821037: adt-virt-lxc: leaves used containers behind

[Antonio Terceiro]

Package: autopkgtest
Version: 3.20.3
Severity: normal

When using adt-virt-lxc, sometimes containers will not be properly shut down,
or not properly destroyed. For example, on one of the Debian CI workers I have
this:

root@ci-arm64-01:~# lxc-ls --fancy
NAME STATEIPV4 IPV6  AUTOSTART
--
adt-sid-arm64STOPPED  -- NO
adt-virt-lxc-faoppo  STOPPED  -- NO
adt-virt-lxc-fuwahc  RUNNING  -- NO
adt-virt-lxc-kargpd  STOPPED  -- NO
adt-virt-lxc-lmtqqi  STOPPED  -- NO
adt-virt-lxc-moatnl  RUNNING  192.168.122.188  - NO

on all the others the situation is similar:

ci-worker10.debian.net: $ sudo lxc-ls --fancy | wc -l
ci-worker10.debian.net: 7
ci-worker09.debian.net: $ sudo lxc-ls --fancy | wc -l
ci-worker09.debian.net: 7
ci-worker08.debian.net: $ sudo lxc-ls --fancy | wc -l
ci-worker08.debian.net: 7
ci-worker07.debian.net: $ sudo lxc-ls --fancy | wc -l
ci-worker07.debian.net: 6
ci-worker06.debian.net: $ sudo lxc-ls --fancy | wc -l
ci-worker06.debian.net: 7
ci-worker05.debian.net: $ sudo lxc-ls --fancy | wc -l
ci-worker05.debian.net: 7
ci-worker04.debian.net: $ sudo lxc-ls --fancy | wc -l
ci-worker04.debian.net: 7
ci-worker03.debian.net: $ sudo lxc-ls --fancy | wc -l
ci-worker03.debian.net: 7
ci-worker02.debian.net: $ sudo lxc-ls --fancy | wc -l
ci-worker02.debian.net: 10
ci-worker01.debian.net: $ sudo lxc-ls --fancy | wc -l
ci-worker01.debian.net: 6
ci-arm64-02.debian.net: $ sudo lxc-ls --fancy | wc -l
ci-arm64-02.debian.net: 4
ci-arm64-01.debian.net: $ sudo lxc-ls --fancy | wc -l
ci-arm64-01.debian.net: 8

I have also noticed that on my workstation, after running a few adt-run calls.

-- System Information:
Debian Release: stretch/sid
  APT prefers unstable
  APT policy: (500, 'unstable'), (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 4.4.0-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=pt_BR.UTF-8, LC_CTYPE=pt_BR.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages autopkgtest depends on:
ii  apt-utils   1.2.10
ii  libdpkg-perl1.18.4
ii  procps  2:3.3.11-3
ii  python3 3.5.1-3
ii  python3-debian  0.1.27

Versions of packages autopkgtest recommends:
ii  autodep8  0.3

Versions of packages autopkgtest suggests:
ii  lxc  1:2.0.0-2
pn  lxd-client   
ii  qemu-system  1:2.5+dfsg-5+b1
ii  qemu-utils   1:2.5+dfsg-5+b1
ii  schroot  1.6.10-2

-- no debconf information

--
Antonio Terceiro 


signature.asc
Description: PGP signature