Re: Background processes in GNU Autotest

[Olaf Mandel]

Hello Mike,

Am 23.06.2016 um 07:19 schrieb Mike Frysinger:
> On 22 Jun 2016 22:52, Olaf Mandel wrote:
>> On 22.06.2016 21:12, Mike Frysinger wrote:
-Snipp-
>>> wouldn't you want the test itself to spin up/down the server as need
>>> be ?  that way you can write multiple end-to-end tests and have them
>>> all run in parallel.
>>>
>> You mean my "client" program starting the server itself? [...]
>> I will have to try if
>> this gives suitable debugging information (I am mostly interested in
>> memory debugging the real server, not the client that was only written
>> for testing).
>>
I finally got it working exactly as I want: thank you for the solution.

>>> i'm guessing your method described above also doesn't work when you
>>> try to run all the tests in parallel.
-Snipp-
> 
> probing for an open port would be one way to handle it.  if you can
> conditionally rely on Linux namespace features, you could create a
> new network namespace for each test which would give you a completely
> blank slate with localhost available.
> $ unshare -Urn
-Snipp-

Decided not to do that because I didn't need the parallel testing on one
machine and because my default user does not have the required
permissions. But now that I learned of this I will keep it in mind
should the need arise: thanks for pointing it out.

Best regards,
Olaf



signature.asc
Description: OpenPGP digital signature
___
Autoconf mailing list
Autoconf@gnu.org
https://lists.gnu.org/mailman/listinfo/autoconf

Re: Background processes in GNU Autotest

[Mike Frysinger]

On 22 Jun 2016 22:52, Olaf Mandel wrote:
> On 22.06.2016 21:12, Mike Frysinger wrote:
> > On 22 Jun 2016 11:03, Olaf Mandel wrote:
> >> I am trying to use GNU Autotest (via AX_GNU_AUTOTEST()) to run
> >> end-to-end tests on a network server. [...]
> >>
> >> AT_CHECK([server&],, [ignore])
> >> AT_CHECK([client --cmd]),  , [expected-output])
> >> AT_CHECK([killall server], , [ignore])
> >>
> -Snipp-
> >> Now I want to combine this with valgrind checking [...]
> >>
> >> valgrind --error-exitcode=1 --quiet ./server &
> >>
> -Snipp-
> > 
> > wouldn't you want the test itself to spin up/down the server as need
> > be ?  that way you can write multiple end-to-end tests and have them
> > all run in parallel.
> > 
> You mean my "client" program starting the server itself? Didn't think of
> that... it would solve the cleanup question during testing. I see that
> valgrind provides the --trace-children=yes option: I will have to try if
> this gives suitable debugging information (I am mostly interested in
> memory debugging the real server, not the client that was only written
> for testing).
> 
> > i'm guessing your method described above also doesn't work when you
> > try to run all the tests in parallel.
> 
> Right. This would already fail because the TCP port would be blocked and
> there is no reliable way to feed back the port number from the server to
> the client (and no: I can't use Unix sockets instead of TCP ports
> without extending some external library). But here the suggestion of
> forking the server from the client may help as well: the client can try
> starting the server on different ports until one works.

probing for an open port would be one way to handle it.  if you can
conditionally rely on Linux namespace features, you could create a
new network namespace for each test which would give you a completely
blank slate with localhost available.
$ unshare -Urn
# ifconfig lo up
# ip a s
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group 
default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
   valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
   valid_lft forever preferred_lft forever
2: sit0@NONE:  mtu 1480 qdisc noop state DOWN group default qlen 1
link/sit 0.0.0.0 brd 0.0.0.0
3: ip6tnl0@NONE:  mtu 1452 qdisc noop state DOWN group default qlen 1
link/tunnel6 :: brd ::
# netstat -nat
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address   Foreign Address State
-mike


signature.asc
Description: Digital signature
___
Autoconf mailing list
Autoconf@gnu.org
https://lists.gnu.org/mailman/listinfo/autoconf

Re: Background processes in GNU Autotest

[Olaf Mandel]

Hello Mike,

On 22.06.2016 21:12, Mike Frysinger wrote:
> On 22 Jun 2016 11:03, Olaf Mandel wrote:
>> I am trying to use GNU Autotest (via AX_GNU_AUTOTEST()) to run
>> end-to-end tests on a network server. [...]
>>
>> AT_CHECK([server&],, [ignore])
>> AT_CHECK([client --cmd]),  , [expected-output])
>> AT_CHECK([killall server], , [ignore])
>>
-Snipp-
>> Now I want to combine this with valgrind checking [...]
>>
>> valgrind --error-exitcode=1 --quiet ./server &
>>
-Snipp-
> 
> wouldn't you want the test itself to spin up/down the server as need
> be ?  that way you can write multiple end-to-end tests and have them
> all run in parallel.
> 
You mean my "client" program starting the server itself? Didn't think of
that... it would solve the cleanup question during testing. I see that
valgrind provides the --trace-children=yes option: I will have to try if
this gives suitable debugging information (I am mostly interested in
memory debugging the real server, not the client that was only written
for testing).

> i'm guessing your method described above also doesn't work when you
> try to run all the tests in parallel.

Right. This would already fail because the TCP port would be blocked and
there is no reliable way to feed back the port number from the server to
the client (and no: I can't use Unix sockets instead of TCP ports
without extending some external library). But here the suggestion of
forking the server from the client may help as well: the client can try
starting the server on different ports until one works.

Thank you for the suggestion,
Olaf



signature.asc
Description: OpenPGP digital signature
___
Autoconf mailing list
Autoconf@gnu.org
https://lists.gnu.org/mailman/listinfo/autoconf

Re: Background processes in GNU Autotest

[Mike Frysinger]

On 22 Jun 2016 11:03, Olaf Mandel wrote:
> I am trying to use GNU Autotest (via AX_GNU_AUTOTEST()) to run
> end-to-end tests on a network server. My current test script looks
> somewhat like this:
> 
> AT_CHECK([server&],, [ignore])
> AT_CHECK([client --cmd]),  , [expected-output])
> AT_CHECK([killall server], , [ignore])
> 
> This is far from perfect, however: there is no cleanup if the client
> check fails or if the executable is called differently (see below).
> 
> Now I want to combine this with valgrind checking (from
> AX_VALGRIND_CHECK()): I defined the check-TESTS target in the Makefile
> so it calls "make check-autotest" and renames the log files to the
> expected names and I added a macro to the test script to use an
> environment variable RUNNER:
> 
> AC_DEFUN([WRAP_CMD], [${RUNNER} $(which $1) $2])
> AT_CHECK(WRAP_CMD([server], [&]),   , [ignore])
> AT_CHECK(WRAP_CMD([client], [--cmd]),   , [expected-output])
> AT_CHECK(WRAP_CMD([killall], [server]), , [ignore])
> 
> But this fails to kill the server simply because the actual command
> being executed by the first AT_CHECK with make check-valgrind is:
> 
> valgrind --error-exitcode=1 --quiet ./server &
> 
> So the simple killall call is no longer feasible.
> 
> Now my question: is there a standard and approved way to start a
> background process in a GNU Autotest script and to kill it at or before
> the AT_CLEANUP ?

wouldn't you want the test itself to spin up/down the server as need
be ?  that way you can write multiple end-to-end tests and have them
all run in parallel.

i'm guessing your method described above also doesn't work when you
try to run all the tests in parallel.
-mike


signature.asc
Description: Digital signature
___
Autoconf mailing list
Autoconf@gnu.org
https://lists.gnu.org/mailman/listinfo/autoconf