Re: [Qemu-devel] [PATCH] Don't enable networking as a side-effect of DEBUG=1

[Daniel P. Berrange]

On Wed, Jul 12, 2017 at 06:55:45PM -0300, Philippe Mathieu-Daudé wrote:
> Hi Daniel,
> 
> On 07/12/2017 01:25 PM, Daniel P. Berrange wrote:
> > When trying to debug problems with tests it is natural to set
> > DEBUG=1 when starting the docker environment. Unfortunately
> > this has a side-effect of enabling an eth0 network interface
> > in the container, which changes the operating environment of
> > the test suite. IOW tests with fail may suddenly start
> > working again if DEBUG=1 is set, due to changed network setup.
> > 
> > Add a separate NETWORK=1 option to allow enablement of
> > networking separately from DEBUG=1, since common debugging
> > tasks probably don't require networking anyway.
> > 
> > Signed-off-by: Daniel P. Berrange 
> > ---
> >   tests/docker/Makefile.include | 4 +++-
> >   1 file changed, 3 insertions(+), 1 deletion(-)
> > 
> > diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> > index 037cb9e..a8c4b82 100644
> > --- a/tests/docker/Makefile.include
> > +++ b/tests/docker/Makefile.include
> > @@ -106,6 +106,7 @@ docker:
> > @echo ' (default is 1)'
> > @echo 'DEBUG=1  Stop and drop to shell in the created 
> > container'
> > @echo ' before running the command.'
> > +   @echo 'NETWORK=1Enable eth0 virtual network interface.'
> 
> "eth0" is not always true...
> 
> This patch could be more generic, maybe documented as:
> 
>   NETWORK=host Use full host network stack (default no network).'
> 
> > @echo 'NOUSER   Define to disable adding current user 
> > to containers passwd.'
> > @echo 'NOCACHE=1Ignore cache when build images.'
> > @echo 'EXECUTABLE=Include executable in image.'
> > @@ -132,7 +133,8 @@ docker-run: docker-qemu-src
> > $(SRC_PATH)/tests/docker/docker.py run  \
> > $(if $(NOUSER),,-u $(shell id -u)) -t   \
> > $(if $V,,--rm)  \
> > -   $(if $(DEBUG),-i,--net=none)\
> > +   $(if $(DEBUG),-i,)  \
> > +   $(if $(NETWORK),,--net=none)\
> 
> and here use directly:  --net=${NETWORK:-none}
> 
> so an experimented docker user could even run tests as:
> 
>   make docker-test-quick@centos6 NETWORK=container:qemu
> 
> (or NETWORK=bridge)

This is a nice idea, though slightly more complicated. It would be good
to support NETWORK=1 as a short-cut for enabling the default docker
network backend, as well as being able to give an explicit backend for
those who really care about the flexibility.


Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: [Qemu-devel] [PATCH] Don't enable networking as a side-effect of DEBUG=1

[Alex Bennée]

Fam Zheng  writes:

> On Wed, 07/12 17:25, Daniel P. Berrange wrote:
>> When trying to debug problems with tests it is natural to set
>> DEBUG=1 when starting the docker environment. Unfortunately
>> this has a side-effect of enabling an eth0 network interface
>> in the container, which changes the operating environment of
>> the test suite. IOW tests with fail may suddenly start
>> working again if DEBUG=1 is set, due to changed network setup.
>
> Makes sense.
>
>>
>> Add a separate NETWORK=1 option to allow enablement of
>> networking separately from DEBUG=1, since common debugging
>> tasks probably don't require networking anyway.
>
> Not uncommon because fiddling with the package manager is often needed when
> working on the test coverage. But that doesn't mean it's bad to control 
> network
> explicitly.
>
>>
>> Signed-off-by: Daniel P. Berrange 
>> ---
>>  tests/docker/Makefile.include | 4 +++-
>>  1 file changed, 3 insertions(+), 1 deletion(-)
>>
>> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
>> index 037cb9e..a8c4b82 100644
>> --- a/tests/docker/Makefile.include
>> +++ b/tests/docker/Makefile.include
>> @@ -106,6 +106,7 @@ docker:
>>  @echo ' (default is 1)'
>>  @echo 'DEBUG=1  Stop and drop to shell in the created 
>> container'
>>  @echo ' before running the command.'
>> +@echo 'NETWORK=1Enable eth0 virtual network interface.'
>
> As pointed out by Philippe, I'd just document it as "enable network".
>
>>  @echo 'NOUSER   Define to disable adding current user 
>> to containers passwd.'
>>  @echo 'NOCACHE=1Ignore cache when build images.'
>>  @echo 'EXECUTABLE=Include executable in image.'
>> @@ -132,7 +133,8 @@ docker-run: docker-qemu-src
>>  $(SRC_PATH)/tests/docker/docker.py run  \
>>  $(if $(NOUSER),,-u $(shell id -u)) -t   \
>>  $(if $V,,--rm)  \
>> -$(if $(DEBUG),-i,--net=none)\
>> +$(if $(DEBUG),-i,)  \
>> +$(if $(NETWORK),,--net=none)\
>>  -e TARGET_LIST=$(TARGET_LIST)   \
>>  -e EXTRA_CONFIGURE_OPTS="$(EXTRA_CONFIGURE_OPTS)" \
>>  -e V=$V -e J=$J -e DEBUG=$(DEBUG)   \
>> --
>> 2.9.4
>>
>
> If others don't come up with objections, I'll apply this.

Acked-by: Alex Bennée 


--
Alex Bennée

Re: [Qemu-devel] [PATCH] Don't enable networking as a side-effect of DEBUG=1

[Fam Zheng]

On Wed, 07/12 17:25, Daniel P. Berrange wrote:
> When trying to debug problems with tests it is natural to set
> DEBUG=1 when starting the docker environment. Unfortunately
> this has a side-effect of enabling an eth0 network interface
> in the container, which changes the operating environment of
> the test suite. IOW tests with fail may suddenly start
> working again if DEBUG=1 is set, due to changed network setup.

Makes sense.

> 
> Add a separate NETWORK=1 option to allow enablement of
> networking separately from DEBUG=1, since common debugging
> tasks probably don't require networking anyway.

Not uncommon because fiddling with the package manager is often needed when
working on the test coverage. But that doesn't mean it's bad to control network
explicitly.

> 
> Signed-off-by: Daniel P. Berrange 
> ---
>  tests/docker/Makefile.include | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
> index 037cb9e..a8c4b82 100644
> --- a/tests/docker/Makefile.include
> +++ b/tests/docker/Makefile.include
> @@ -106,6 +106,7 @@ docker:
>   @echo ' (default is 1)'
>   @echo 'DEBUG=1  Stop and drop to shell in the created 
> container'
>   @echo ' before running the command.'
> + @echo 'NETWORK=1Enable eth0 virtual network interface.'

As pointed out by Philippe, I'd just document it as "enable network".

>   @echo 'NOUSER   Define to disable adding current user 
> to containers passwd.'
>   @echo 'NOCACHE=1Ignore cache when build images.'
>   @echo 'EXECUTABLE=Include executable in image.'
> @@ -132,7 +133,8 @@ docker-run: docker-qemu-src
>   $(SRC_PATH)/tests/docker/docker.py run  \
>   $(if $(NOUSER),,-u $(shell id -u)) -t   \
>   $(if $V,,--rm)  \
> - $(if $(DEBUG),-i,--net=none)\
> + $(if $(DEBUG),-i,)  \
> + $(if $(NETWORK),,--net=none)\
>   -e TARGET_LIST=$(TARGET_LIST)   \
>   -e EXTRA_CONFIGURE_OPTS="$(EXTRA_CONFIGURE_OPTS)" \
>   -e V=$V -e J=$J -e DEBUG=$(DEBUG)   \
> -- 
> 2.9.4
> 

If others don't come up with objections, I'll apply this.

Thanks,
Fam

Re: [Qemu-devel] [PATCH] Don't enable networking as a side-effect of DEBUG=1

[Philippe Mathieu-Daudé]

On 07/12/2017 06:46 PM, Philippe Mathieu-Daudé wrote:

now trying old debian release:

$ docker run --rm -it debian:wheezy sh -c "cat /etc/debian_version"; 
echo $?

7.11
0

$ docker run --rm -it debian:wheezy bash -c "cat /etc/debian_version"; 
echo $?

139


Indeed using debian:wheezy based dockerfile:

$ make docker-test-quick@debian7
[...]
Step 4/14 : RUN apt-get update
 ---> Running in 305758a09ca4
E: Method http has died unexpectedly!
E: Sub-process http received a segmentation fault.
The command '/bin/sh -c apt-get update' returned a non-zero code: 100

$ dmesg
sh[25336] vsyscall attempted with vsyscall=none ip:ff600400 
cs:33 sp:7fffa210e208 ax:ff600400 si:7fffa210ef60 di:0
sh[25336]: segfault at ff600400 ip ff600400 sp 
7fffa210e208 error 15


note, this does test Fam's "docker.py: Improve subprocess exit code 
handling" :P

Re: [Qemu-devel] [PATCH] Don't enable networking as a side-effect of DEBUG=1

[Philippe Mathieu-Daudé]

Hi Daniel,

On 07/12/2017 01:25 PM, Daniel P. Berrange wrote:

When trying to debug problems with tests it is natural to set
DEBUG=1 when starting the docker environment. Unfortunately
this has a side-effect of enabling an eth0 network interface
in the container, which changes the operating environment of
the test suite. IOW tests with fail may suddenly start
working again if DEBUG=1 is set, due to changed network setup.

Add a separate NETWORK=1 option to allow enablement of
networking separately from DEBUG=1, since common debugging
tasks probably don't require networking anyway.

Signed-off-by: Daniel P. Berrange 
---
  tests/docker/Makefile.include | 4 +++-
  1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 037cb9e..a8c4b82 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -106,6 +106,7 @@ docker:
@echo ' (default is 1)'
@echo 'DEBUG=1  Stop and drop to shell in the created 
container'
@echo ' before running the command.'
+   @echo 'NETWORK=1Enable eth0 virtual network interface.'


"eth0" is not always true...

This patch could be more generic, maybe documented as:

  NETWORK=host Use full host network stack (default no network).'


@echo 'NOUSER   Define to disable adding current user 
to containers passwd.'
@echo 'NOCACHE=1Ignore cache when build images.'
@echo 'EXECUTABLE=Include executable in image.'
@@ -132,7 +133,8 @@ docker-run: docker-qemu-src
$(SRC_PATH)/tests/docker/docker.py run  \
$(if $(NOUSER),,-u $(shell id -u)) -t   \
$(if $V,,--rm)  \
-   $(if $(DEBUG),-i,--net=none)\
+   $(if $(DEBUG),-i,)  \
+   $(if $(NETWORK),,--net=none)\


and here use directly:  --net=${NETWORK:-none}

so an experimented docker user could even run tests as:

  make docker-test-quick@centos6 NETWORK=container:qemu

(or NETWORK=bridge)


-e TARGET_LIST=$(TARGET_LIST)   \
-e EXTRA_CONFIGURE_OPTS="$(EXTRA_CONFIGURE_OPTS)" \
-e V=$V -e J=$J -e DEBUG=$(DEBUG)   \



Regards,

Phil.

Re: [Qemu-devel] [PATCH] Don't enable networking as a side-effect of DEBUG=1

[Philippe Mathieu-Daudé]

Hi Alex, Fam,

I wanted to try this patch but got:

$ make docker-test-quick@centos6 NETWORK=1
  BUILD   centos6
The command '/bin/sh -c yum install -y epel-release' returned a non-zero 
code: 139

Traceback (most recent call last):
  File "./tests/docker/docker.py", line 382, in 
sys.exit(main())
  File "./tests/docker/docker.py", line 379, in main
return args.cmdobj.run(args, argv)
  File "./tests/docker/docker.py", line 301, in run
extra_files_cksum=cksum)
  File "./tests/docker/docker.py", line 185, in build_image
quiet=quiet)
  File "./tests/docker/docker.py", line 123, in _do_check
return subprocess.check_call(self._command + cmd, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 186, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['docker', 'build', '-t', 
'qemu:centos6', '-f', '/tmp/docker_buildIrIR2w/tmpxMPjZu.docker', 
'--build-arg=http_proxy=http://172.17.0.1:3142/', 
'/tmp/docker_buildIrIR2w']' returned non-zero exit status 139
tests/docker/Makefile.include:47: recipe for target 
'docker-image-centos6' failed

make: *** [docker-image-centos6] Error 1

looking further:

$ docker run --rm centos:6 cat /etc/redhat-release; echo $?
CentOS release 6.9 (Final)
0

$ docker run --rm centos:6 sh -c "cat /etc/redhat-release"; echo $?
139

uh?

$ docker run --rm centos:7 sh -c "cat /etc/redhat-release"; echo $?
CentOS Linux release 7.3.1611 (Core)
0

now trying old debian release:

$ docker run --rm -it debian:wheezy sh -c "cat /etc/debian_version"; echo $?
7.11
0

$ docker run --rm -it debian:wheezy bash -c "cat /etc/debian_version"; 
echo $?

139

hmmm

$ docker run --rm -it debian:jessie bash -c "cat /etc/debian_version"; 
echo $?

8.7
0

$ docker info
Server Version: 17.05.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
Cgroup Driver: cgroupfs
Default Runtime: runc
Init Binary: docker-init
containerd version: 9048e5e50717ea4497b757314bad98ea3763c145
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Kernel Version: 4.11.0-1-amd64
Operating System: Debian GNU/Linux buster/sid
Architecture: x86_64

$ sudo journalctl -kb -l -o json-pretty
{
"PRIORITY" : "6",
"_TRANSPORT" : "kernel",
"SYSLOG_FACILITY" : "0",
"SYSLOG_IDENTIFIER" : "kernel",
"MESSAGE" : "sh[23389] vsyscall attempted with vsyscall=none 
ip:ff600400 cs:33 sp:7ffcfd21a6c8 ax:ff600400 
si:7ffcfd21af6f di:0"

}
{
"_TRANSPORT" : "kernel",
"SYSLOG_FACILITY" : "0",
"SYSLOG_IDENTIFIER" : "kernel",
"MESSAGE" : "sh[23389]: segfault at ff600400 ip 
ff600400 sp 7ffcfd21a6c8 error 15"

}

is it time to upgrade the docker image to centos:7 ?