On Wed, Mar 8, 2023 at 10:49 PM Xiangyu Chen
<[email protected]> wrote:
>
> Hi Bruce,
>
>
> Sorry for being late,
>
> On 3/8/23 12:00, Bruce Ashfield wrote:
> > CAUTION: This email comes from a non Wind River email account!
> > Do not click links or open attachments unless you recognize the sender and
> > know the content is safe.
> >
> > On Thu, Mar 2, 2023 at 8:17 PM Xiangyu Chen
> > <[email protected]> wrote:
> >> Hi Bruce,
> >>
> >>
> >> On 3/3/23 05:24, Bruce Ashfield wrote:
> >>> CAUTION: This email comes from a non Wind River email account!
> >>> Do not click links or open attachments unless you recognize the sender
> >>> and know the content is safe.
> >>>
> >>> In message: [meta-virtualization][PATCH 0/1] lxc:
> >>> templates/lxc-busybox.in: if busybox contains init then use it
> >>> on 01/03/2023 Xiangyu Chen wrote:
> >>>
> >>>> From: Xiangyu Chen <[email protected]>
> >>>>
> >>>> Hi Bruce,
> >>>>
> >>>> Recently we found that the lxc ptest has lots of failure cases as below
> >>>> log-1, after checking the
> >>>> code, some cases failed due to related the init progess. For example,
> >>>> lxc-test-exit-code need to
> >>>> start container as daemon, but if using bash as init, the container
> >>>> cannot start correctly.
> >>> Is there an indication of what busybox is providing that bash isn't ?
> >> In my local setup, when using bash as container init, the container
> >> doesn't support "reboot" and cannot start correctly in a daemon mode,
> >> test step as below:
> >>
> >> lxc-create -t busybox -n t
> >>
> >> lxc-start -n t -d
> >>
> >> lxc-ls -f
> >>
> >> the container "t" status still in "STOPPED", but when we use a busybox
> >> init instead of bash, the container status is correct in daemon mode.
> >>
> > I'm setting up to test this myself, but generally speaking we should
> > include this detail in the commit log.
>
> Thanks, if we find the final root cause, I'll add our discussion
> information into the commit and send a v2 patch :p
>
>
> >
> >>> I don't like to force this in the ptest, while for actual lxc containers
> >>> we still allow bash, which means it may not be functional.
> >> This is a common template for lxc, but I am not sure whether others
> >> still need bash as lxc container init, so I based the patch
> >>
> >> "template-make-busybox-template-compatible-with-core-.patch" to add the
> >> busybox back.
> >>
> > That's the part that concerns me. Why does our bash behave differently
> > than in other lxc integrations and other distros ?
> Currently, the behavior is when using lxc with busybox template in
> daemon mode, the status still stay in "STOPPED", but it's working well
> in foreground mode.
>
> > Do you have the ability to run the same simple tests you have above on
> > a desktop distro ?
>
> I have setup a virtualbox today and did some test with trace, here is
> something I was found:
>
> As above mentioned, lxc working well in foreground mode but something
> wrong with daemon mode, according to manual of lxc-start, the foreground
> mode attach the tty to /dev/console, but daemon mode doesn't.
>
And to confirm, this is running on something like ubuntu, using the
ubuntu bash ?
> I was enable the lxc debug trace as below when start a container with
> daemon mode (part of init related log and remove timestamp and full
> source code path)
>
> ##### lxc-start -n t -o /tmp/log.txt -l TRACE -d #####
>
> start - /src/lxc/start.c:post_start:2205 - Started "/sbin/init" with pid
> "871"
> start - /src/lxc/start.c:lxc_serve_state_clients:483 - Set container
> state to RUNNING
>
> <<<<<<<<< we can see the bash as init was starting, and lxc update mode
>
> start - /src/lxc/start.c:lxc_serve_state_clients:486 - No state clients
> registered
> mainloop - /src/lxc/mainloop.c:__epoll_open:493 - Created epoll instance
> mainloop - /mainloop.c:__epoll_open:493 - Created epoll instance
> start - /src/lxc/start.c:lxc_poll:626 - Mainloop is ready
> start - /src/lxc/start.c:signal_handler:396 - Received signal
> ssi_signo(17) for ssi_pid(871), si_signo(17), si_pid(871)
> start - /src/lxc/start.c:signal_handler:464 - Container init process 871
> exited
>
> <<<<<<<<<< seems something wrong with the init, it exited and lxc got
> the exit signal.
>
> start - /src/lxc/start.c:lxc_poll:643 - Closed console mainloop
> start - /src/lxc/start.c:lxc_poll:648 - Closed mainloop
> start - /src/lxc/start.c:lxc_poll:651 - Closed signal file descriptor 7
> ..... removed some networking teminating related trace .....
> start - /src/lxc/start.c:lxc_serve_state_clients:483 - Set container
> state to STOPPING
>
> <<<<<<<<<<< now the lxc set container mode back to stop mode.
>
> start - /src/lxc/start.c:lxc_serve_state_clients:486 - No state clients
> registered
> ##### end of lxc-start -n t -o /tmp/log.txt -l TRACE -d #####
>
>
> Let's use strace to see what happens in container(part of init related log):
>
> #####strace -s 1024 -f lxc-start -n t -d #####
>
> [pid 1211] execve("/sbin/init", ["/sbin/init"], 0x55a07c90eb30 /* 1 var
> */ <unfinished ...>
> ......
>
> [pid 1211] ioctl(2, TIOCGPGRP, 0x7fffe212610c) = -1 ENOTTY
> (Inappropriate ioctl for device)
> [pid 1211] rt_sigaction(SIGCHLD, {sa_handler=0x5632e07dcec0,
> sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART,
> sa_restorer=0x7f1237db3190}, {sa_handler=SIG_DFL, sa_mask=[],
> sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f1237db3190}, 8) = 0
> [pid 1211] ioctl(2, TIOCGPGRP, 0x7fffe21260ec) = -1 ENOTTY
> (Inappropriate ioctl for device)
> [pid 1211] prlimit64(0, RLIMIT_NPROC, NULL, {rlim_cur=3818,
> rlim_max=3818}) = 0
> [pid 1211] rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
> [pid 1211] fcntl(0, F_GETFL) = 0x8002 (flags O_RDWR|O_LARGEFILE)
> [pid 1211] newfstatat(0, "", {st_mode=S_IFCHR|0666,
> st_rdev=makedev(0x1, 0x3), ...}, AT_EMPTY_PATH) = 0
> [pid 1211] lseek(0, 0, SEEK_CUR) = 0
> [pid 1211] read(0, "", 1) = 0
> [pid 1211] rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
> [pid 1211] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
> [pid 1211] exit_group(0) = ?
> [pid 1211] +++ exited with 0 +++
> #####end of strace -s 1024 -f lxc-start -n t -d #####
>
> it looks that seems bash stdi/o/err cannot attach the available caused
> this issue, right?
It does look like that. When busybox is used as init, do you have a
similar strace ? I'd like to do that comparison.
Bruce
>
>
> > I don't want to force this switch to busybox, without understanding if
> > we are the only ones seeing this issue .. since that means we are
> > simply hiding an issue, versus fixing it.
>
> Yes indeed, if someone is using systemd as init, they need to add extra
> configurations to setup busybox and keep default systemd init cannot be
> replaced by busybox in local.conf .
>
>
> Thanks,
>
> Xiangyu
>
> > Bruce
> >>> There are other init options in meta-virt, like the docker tini, if we
> >>> had to enforce something, I'd rather that than busybox.
> >> Good to hear the tini :)
> >>
> >> Indeed, enable the busybox-init might need lots of additional effort to
> >> take care of system which using systemd, otherwise, /sbin/init always to
> >> be covered by busybox.
> >>
> >>> Bruce
> >>
> >> Br,
> >>
> >> Xiangyu
> >>
> >>
> >>>> So added a busybox init utils checking in lxc-busybox template, if
> >>>> current system busybox contains
> >>>> init then use it, after applying this patch, the ptest result as log-2.
> >>>>
> >>>>
> >>>> ######## 1og-1: ptest without patch #######
> >>>>
> >>>> Starting LXC ptest ###
> >>>> FAIL: lxc-test-api-reboot
> >>>> SKIPPED: lxc-test-apparmor
> >>>> PASS: lxc-test-apparmor-generated
> >>>> FAIL: lxc-test-apparmor-mount
> >>>> PASS: lxc-test-arch-parse
> >>>> FAIL: lxc-test-attach
> >>>> PASS: lxc-test-automount
> >>>> FAIL: lxc-test-autostart
> >>>> PASS: lxc-test-basic
> >>>> FAIL: lxc-test-capabilities
> >>>> FAIL: lxc-test-cgpath
> >>>> PASS: lxc-test-checkpoint-restore
> >>>> FAIL: lxc-test-cloneconfig
> >>>> FAIL: lxc-test-clonetest
> >>>> FAIL: lxc-test-concurrent
> >>>> PASS: lxc-test-config-jump-table
> >>>> FAIL: lxc-test-console
> >>>> FAIL: lxc-test-console-log
> >>>> FAIL: lxc-test-containertests
> >>>> FAIL: lxc-test-createconfig
> >>>> FAIL: lxc-test-createtest
> >>>> PASS: lxc-test-criu-check-feature
> >>>> FAIL: lxc-test-cve-2019-5736
> >>>> FAIL: lxc-test-destroytest
> >>>> FAIL: lxc-test-device-add-remove
> >>>> FAIL: lxc-test-exit-code
> >>>> FAIL: lxc-test-get_item
> >>>> PASS: lxc-test-getkeys
> >>>> PASS: lxc-test-list
> >>>> PASS: lxc-test-locktests
> >>>> FAIL: lxc-test-lxc-attach
> >>>> PASS: lxc-test-lxcpath
> >>>> PASS: lxc-test-may-control
> >>>> FAIL: lxc-test-mount-injection
> >>>> FAIL: lxc-test-no-new-privs
> >>>> PASS: lxc-test-parse-config-file
> >>>> FAIL: lxc-test-proc-pid
> >>>> FAIL: lxc-test-procsys
> >>>> PASS: lxc-test-raw-clone
> >>>> PASS: lxc-test-reboot
> >>>> FAIL: lxc-test-rootfs
> >>>> FAIL: lxc-test-rootfs-options
> >>>> FAIL: lxc-test-saveconfig
> >>>> FAIL: lxc-test-share-ns
> >>>> FAIL: lxc-test-shortlived
> >>>> SKIPPED: lxc-test-shutdowntest
> >>>> FAIL: lxc-test-snapdeps
> >>>> FAIL: lxc-test-snapshot
> >>>> FAIL: lxc-test-startone
> >>>> SKIPPED: lxc-test-state-server
> >>>> FAIL: lxc-test-symlink
> >>>> FAIL: lxc-test-sys-mixed
> >>>> FAIL: lxc-test-sysctls
> >>>> FAIL: lxc-test-unpriv
> >>>> FAIL: lxc-test-usernic
> >>>> PASS: lxc-test-usernsexec
> >>>> PASS: lxc-test-utils
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> Results:
> >>>> PASSED = 17
> >>>> FAILED = 37
> >>>> SKIPPED = 3
> >>>> (for details check individual test log in ./logs directory)
> >>>>
> >>>>
> >>>> ###########log-2: ptest with patch ###################
> >>>> root@intel-x86-64:/usr/lib64/lxc/ptest# ./run-ptest
> >>>> ### Starting LXC ptest ###
> >>>> PASS: lxc-test-api-reboot
> >>>> SKIPPED: lxc-test-apparmor
> >>>> PASS: lxc-test-apparmor-generated
> >>>> FAIL: lxc-test-apparmor-mount
> >>>> PASS: lxc-test-arch-parse
> >>>> PASS: lxc-test-attach
> >>>> PASS: lxc-test-automount
> >>>> PASS: lxc-test-autostart
> >>>> PASS: lxc-test-basic
> >>>> PASS: lxc-test-capabilities
> >>>> PASS: lxc-test-cgpath
> >>>> PASS: lxc-test-checkpoint-restore
> >>>> FAIL: lxc-test-cloneconfig
> >>>> PASS: lxc-test-clonetest
> >>>> PASS: lxc-test-concurrent
> >>>> PASS: lxc-test-config-jump-table
> >>>> PASS: lxc-test-console
> >>>> PASS: lxc-test-console-log
> >>>> PASS: lxc-test-containertests
> >>>> PASS: lxc-test-createconfig
> >>>> PASS: lxc-test-createtest
> >>>> PASS: lxc-test-criu-check-feature
> >>>> PASS: lxc-test-cve-2019-5736
> >>>> PASS: lxc-test-destroytest
> >>>> PASS: lxc-test-device-add-remove
> >>>> PASS: lxc-test-exit-code
> >>>> FAIL: lxc-test-get_item
> >>>> PASS: lxc-test-getkeys
> >>>> PASS: lxc-test-list
> >>>> PASS: lxc-test-locktests
> >>>> PASS: lxc-test-lxc-attach
> >>>> PASS: lxc-test-lxcpath
> >>>> PASS: lxc-test-may-control
> >>>> PASS: lxc-test-mount-injection
> >>>> FAIL: lxc-test-no-new-privs
> >>>> PASS: lxc-test-parse-config-file
> >>>> PASS: lxc-test-proc-pid
> >>>> PASS: lxc-test-procsys
> >>>> PASS: lxc-test-raw-clone
> >>>> PASS: lxc-test-reboot
> >>>> PASS: lxc-test-rootfs
> >>>> PASS: lxc-test-rootfs-options
> >>>> PASS: lxc-test-saveconfig
> >>>> PASS: lxc-test-share-ns
> >>>> PASS: lxc-test-shortlived
> >>>> SKIPPED: lxc-test-shutdowntest
> >>>> FAIL: lxc-test-snapdeps
> >>>> PASS: lxc-test-snapshot
> >>>> PASS: lxc-test-startone
> >>>> SKIPPED: lxc-test-state-server
> >>>> PASS: lxc-test-symlink
> >>>> PASS: lxc-test-sys-mixed
> >>>> PASS: lxc-test-sysctls
> >>>> FAIL: lxc-test-unpriv
> >>>> FAIL: lxc-test-usernic
> >>>> PASS: lxc-test-usernsexec
> >>>> PASS: lxc-test-utils
> >>>>
> >>>> Results:
> >>>> PASSED = 47
> >>>> FAILED = 7
> >>>> SKIPPED = 3
> >>>> (for details check individual test log in ./logs directory)
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> Xiangyu Chen (1):
> >>>> lxc: templates/lxc-busybox.in: if busybox contains init then use it
> >>>>
> >>>> ...box-contains-init-use-it-in-containe.patch | 45 +++++++++++++++++++
> >>>> recipes-containers/lxc/lxc_git.bb | 1 +
> >>>> 2 files changed, 46 insertions(+)
> >>>> create mode 100644
> >>>> recipes-containers/lxc/files/0001-template-if-busybox-contains-init-use-it-in-containe.patch
> >>>>
> >>>> --
> >>>> 2.34.1
> >>>>
> >>>>
> >>>>
> >
> >
> > --
> > - Thou shalt not follow the NULL pointer, for chaos and madness await
> > thee at its end
> > - "Use the force Harry" - Gandalf, Star Trek II
--
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#7940):
https://lists.yoctoproject.org/g/meta-virtualization/message/7940
Mute This Topic: https://lists.yoctoproject.org/mt/97306618/21656
Group Owner: [email protected]
Unsubscribe:
https://lists.yoctoproject.org/g/meta-virtualization/leave/6693005/21656/1014668956/xyzzy
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
