Bug#867361: sitesummary: Fix autopkgtest to work in environments with http_proxy set

[Wolfgang Schweer]

On Fri, Jul 07, 2017 at 01:19:17PM +0200, Petter Reinholdtsen wrote:
> What about a loop looking for a file every 1 second, and exiting after X
> seconds if no file is found.  It should handle more load while exiting
> as quickly as possible.
 
I guess something like this should work:

--- a/test-server-client2017-07-07 11:16:36.0 +0200
+++ b/test-server-client2017-07-07 15:21:45.881225852 +0200
@@ -29,9 +29,13 @@
 
 find /var/lib/sitesummary
 
-if find /var/lib/sitesummary | grep ether- ; then
-echo success: found entry
-else
-echo error: did not find entry
-exit 1
-fi
+for i in $(seq 0 10) ; do
+if find /var/lib/sitesummary | grep ether- ; then
+   echo success: found entry
+   exit 0
+fi
+sleep 1
+done
+
+echo error: did not find entry after waiting for 10 seconds.
+exit 1

Wolfgang


signature.asc
Description: PGP signature

Bug#867361: sitesummary: Fix autopkgtest to work in environments with http_proxy set

[Petter Reinholdtsen]

[Wolfgang Schweer]
> The below one should work (even sleep 1 seems to be enough).

What about a loop looking for a file every 1 second, and exiting after X
seconds if no file is found.  It should handle more load while exiting
as quickly as possible.

-- 
Happy hacking
Petter Reinholdtsen

Bug#867361: sitesummary: Fix autopkgtest to work in environments with http_proxy set

[Wolfgang Schweer]

On Thu, Jul 06, 2017 at 07:26:19AM +0200, Petter Reinholdtsen wrote:
> 
> Hm, look like a race condition:
> 
> autopkgtest [01:08:03]: test test-server-client: [---
> /var/lib/sitesummary
> /var/lib/sitesummary/tmpstorage
> /var/lib/sitesummary/tmpstorage/127.0.0.1-2017-07-06T01:08:03-2349-sitesummary
> /var/lib/sitesummary/tmpstorage/127.0.0.1-2017-07-06T01:08:03-2349-sitesummary.tar.gz
> /var/lib/sitesummary/www
> /var/lib/sitesummary/www/index.html
> /var/lib/sitesummary/entries
> error: did not find entry
> info: terminating script
> autopkgtest [01:08:04]: test test-server-client: ---]
> 
> It look like the server is still busy unpacking the submitted entry, and
> have not yet had time to move the relevant files under entries/.
> 
> Anyone got any idea how we can best avoid the race?  'sleep 10' seem
> like such a crude solution...
 
The below one should work (even sleep 1 seems to be enough).

--- a/test-server-client2017-07-07 11:16:36.0 +0200
+++ b/test-server-client2017-07-07 12:16:17.406332996 +0200
@@ -26,6 +26,7 @@
 service apache2 restart
 
 sitesummary-client
+sleep 2
 
 find /var/lib/sitesummary
 

Wolfgang


signature.asc
Description: PGP signature

Bug#867361: sitesummary: Fix autopkgtest to work in environments with http_proxy set

[Wolfgang Schweer]

On Thu, Jul 06, 2017 at 11:58:13AM -0700, Steve Langasek wrote:
> On Thu, Jul 06, 2017 at 07:26:19AM +0200, Petter Reinholdtsen wrote:
> > [Steve Langasek]
> > > FWIW this does fix the problem with the test talking to the wrong server,
> > > but the test still fails as shown at
> > > .
> 
> > Hm, look like a race condition:
> 
> > autopkgtest [01:08:03]: test test-server-client: [---
> > /var/lib/sitesummary
> > /var/lib/sitesummary/tmpstorage
> > /var/lib/sitesummary/tmpstorage/127.0.0.1-2017-07-06T01:08:03-2349-sitesummary
> > /var/lib/sitesummary/tmpstorage/127.0.0.1-2017-07-06T01:08:03-2349-sitesummary.tar.gz
> > /var/lib/sitesummary/www
> > /var/lib/sitesummary/www/index.html
> > /var/lib/sitesummary/entries
> > error: did not find entry
> > info: terminating script
> > autopkgtest [01:08:04]: test test-server-client: ---]
> > 
> > It look like the server is still busy unpacking the submitted entry, and
> > have not yet had time to move the relevant files under entries/.
> 
> > Anyone got any idea how we can best avoid the race?  'sleep 10' seem
> > like such a crude solution...
> 
> Ah.  Seems to me like either a sleep or a poll are your only options.  I
> don't know how much processing sitesummary has to do, but maybe a one minute
> timeout on a poll would be reasonable?

There's not much to do, it's rather seconds than a minute; the overview 
page shows the test failing only randomly, times differ as well:

https://ci.debian.net/packages/s/sitesummary/unstable/amd64/

Wolfgang


signature.asc
Description: PGP signature

Bug#867361: sitesummary: Fix autopkgtest to work in environments with http_proxy set

[Steve Langasek]

On Thu, Jul 06, 2017 at 07:26:19AM +0200, Petter Reinholdtsen wrote:

> Thank you Steve, for looking into autopkgtest failures. :)

> [Steve Langasek]
> > FWIW this does fix the problem with the test talking to the wrong server,
> > but the test still fails as shown at
> > .

> Hm, look like a race condition:

> autopkgtest [01:08:03]: test test-server-client: [---
> /var/lib/sitesummary
> /var/lib/sitesummary/tmpstorage
> /var/lib/sitesummary/tmpstorage/127.0.0.1-2017-07-06T01:08:03-2349-sitesummary
> /var/lib/sitesummary/tmpstorage/127.0.0.1-2017-07-06T01:08:03-2349-sitesummary.tar.gz
> /var/lib/sitesummary/www
> /var/lib/sitesummary/www/index.html
> /var/lib/sitesummary/entries
> error: did not find entry
> info: terminating script
> autopkgtest [01:08:04]: test test-server-client: ---]
> 
> It look like the server is still busy unpacking the submitted entry, and
> have not yet had time to move the relevant files under entries/.

> Anyone got any idea how we can best avoid the race?  'sleep 10' seem
> like such a crude solution...

Ah.  Seems to me like either a sleep or a poll are your only options.  I
don't know how much processing sitesummary has to do, but maybe a one minute
timeout on a poll would be reasonable?

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developerhttp://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


signature.asc
Description: PGP signature

Bug#867361: sitesummary: Fix autopkgtest to work in environments with http_proxy set

[Petter Reinholdtsen]

Thank you Steve, for looking into autopkgtest failures. :)

[Steve Langasek]
> FWIW this does fix the problem with the test talking to the wrong server,
> but the test still fails as shown at
> .

Hm, look like a race condition:

autopkgtest [01:08:03]: test test-server-client: [---
/var/lib/sitesummary
/var/lib/sitesummary/tmpstorage
/var/lib/sitesummary/tmpstorage/127.0.0.1-2017-07-06T01:08:03-2349-sitesummary
/var/lib/sitesummary/tmpstorage/127.0.0.1-2017-07-06T01:08:03-2349-sitesummary.tar.gz
/var/lib/sitesummary/www
/var/lib/sitesummary/www/index.html
/var/lib/sitesummary/entries
error: did not find entry
info: terminating script
autopkgtest [01:08:04]: test test-server-client: ---]

It look like the server is still busy unpacking the submitted entry, and
have not yet had time to move the relevant files under entries/.

Anyone got any idea how we can best avoid the race?  'sleep 10' seem
like such a crude solution...

-- 
Happy hacking
Petter Reinholdtsen

Bug#867361: sitesummary: Fix autopkgtest to work in environments with http_proxy set

[Steve Langasek]

On Wed, Jul 05, 2017 at 05:02:43PM -0700, Steve Langasek wrote:
> Package: sitesummary
> Version: 0.1.28
> Severity: normal
> Tags: patch
> User: ubuntu-de...@lists.ubuntu.com
> Usertags: origin-ubuntu artful ubuntu-patch

> Dear maintainers,

> Thanks for the work on improving the autopkgtest for sitesummary in recent
> uploads.

> In Ubuntu, sitesummary has been stuck at version 0.1.21 because this is the
> last version of the package with an autopkgtest that didn't confirm the
> results on disk; and the autopkgtest does not actually correctly test
> sitesummary in an Ubuntu environment because http_proxy is set on our test
> runners and so the test case is talking to a squid proxy instead of to the
> local apache.

> I have just uploaded the attached patch to Ubuntu, which should make the
> autopkgtest work in the face of http_proxy (results will soon be visible at
> http://autopkgtest.ubuntu.com/packages/s/sitesummary/artful/amd64).

> Please consider including the patch in Debian as well.

FWIW this does fix the problem with the test talking to the wrong server,
but the test still fails as shown at
.

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developerhttp://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


signature.asc
Description: PGP signature

Bug#867361: sitesummary: Fix autopkgtest to work in environments with http_proxy set

[Steve Langasek]

Package: sitesummary
Version: 0.1.28
Severity: normal
Tags: patch
User: ubuntu-de...@lists.ubuntu.com
Usertags: origin-ubuntu artful ubuntu-patch

Dear maintainers,

Thanks for the work on improving the autopkgtest for sitesummary in recent
uploads.

In Ubuntu, sitesummary has been stuck at version 0.1.21 because this is the
last version of the package with an autopkgtest that didn't confirm the
results on disk; and the autopkgtest does not actually correctly test
sitesummary in an Ubuntu environment because http_proxy is set on our test
runners and so the test case is talking to a squid proxy instead of to the
local apache.

I have just uploaded the attached patch to Ubuntu, which should make the
autopkgtest work in the face of http_proxy (results will soon be visible at
http://autopkgtest.ubuntu.com/packages/s/sitesummary/artful/amd64).

Please consider including the patch in Debian as well.

Thanks,
-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developerhttp://www.debian.org/
slanga...@ubuntu.com vor...@debian.org
diff -Nru sitesummary-0.1.28/debian/tests/test-server-client 
sitesummary-0.1.28ubuntu1/debian/tests/test-server-client
--- sitesummary-0.1.28/debian/tests/test-server-client  2017-01-07 
04:36:53.0 -0800
+++ sitesummary-0.1.28ubuntu1/debian/tests/test-server-client   2017-07-05 
16:56:28.0 -0700
@@ -14,6 +14,8 @@
 LC_ALL=C
 export LC_ALL
 
+unset http_proxy
+
 # Installing the binary packages should set up apache on localhost and
 # the cgi script for the sitesummary collector.  But some times the
 # install does not do this, because the apt system is sorting the