#17924: Failure in doctest framework when running in docker
-------------------------------------+-------------------------------------
Reporter: Vincent.Neri | Owner:
Type: defect | Status: needs_info
Priority: major | Milestone: sage-6.6
Component: doctest framework | Resolution:
Keywords: docker | Merged in:
Authors: Vincent Neri | Reviewers:
Report Upstream: N/A | Work issues:
Branch: | Commit:
u/Vincent.Neri/failure_in_doctest_framework_when_running_in_docker|
add3e53a81dafd966b07727d87c3d441e1e2468f
Dependencies: | Stopgaps:
-------------------------------------+-------------------------------------
Description changed by nthiery:
Old description:
> We are investigating the usage of docker to run multiple instances of
> the sage patchbot within sandboxes on a cluster. This works smoothly,
> except that the doctesting framework fails on three files:
>
> {{{
> sage -t --warn-long 19.5 src/sage/interfaces/qsieve.py # Bad exit: 1
> sage -t --warn-long 19.5 local/lib/python2.7/site-
> packages/sagenb-0.11.4-py2.7.egg/sagenb/notebook/worksheet.py
> # Bad exit: 1
> sage -t --warn-long 19.5 local/lib/python2.7/site-
> packages/sagenb-0.11.4-py2.7.egg/sagenb/notebook/cell.py
> # Bad exit: 1
> }}}
>
> Minimal steps to reproduce and complete log:
> {{{
> # Install docker (here for Debian / Ubuntu):
> > sudo apt-get install docker.io
>
> # Run a docker container with Sage:
> > sudo docker run -t -i sagemath/sage su - sage
> Last login: Thu Nov 27 15:28:05 GMT 2014
> [sage@63182b1b1561 ~]$ uname -a
> Linux 63182b1b1561 3.13.0-43-generic #72-Ubuntu SMP Mon Dec 8 19:35:06
> UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
>
> [sage@63182b1b1561 ~]$ cd sage
> [sage@63182b1b1561 sage-6.4.1]$ ./sage -v
> Sage Version 6.4.1, Release Date: 2014-11-23
>
> [sage@63182b1b1561 sage-6.4.1]$ cat > bla.py << EOF
> > r"""
> > sage: from sage.interfaces.qsieve import qsieve
> > sage: q = qsieve(next_prime(10^20)*next_prime(10^21),
> block=False)
> > """
> > EOF
>
> [sage@63182b1b1561 sage-6.4.1]$ ./sage -t --verbose bla.py
> init.sage does not exist ... creating
> no stored timings available
> Running doctests with ID 2015-03-04-07-53-04-191d76e2.
> Doctesting 1 file.
> sage -t bla.py
> Trying (line 2): from sage.interfaces.qsieve import qsieve
> Expecting nothing
> ok [0.00 s]
> Trying (line 3): k = 19; n = next_prime(10^k)*next_prime(10^(k+1))
> Expecting nothing
> ok [0.00 s]
> Trying (line 5): q = qsieve(next_prime(10^20)*next_prime(10^21),
> block=False)
> Expecting nothing
> ok [0.23 s]
> Trying (line 6): sig_on_count()
> Expecting:
> 0
> ok [0.00 s]
> 1 item passed all tests:
> 4 tests in bla
> 4 tests in 1 item.
> 4 passed and 0 failed.
> Test passed.
> Process DocTestWorker-1:
> Traceback (most recent call last):
> File
> "/home/sage/sage-6.4.1/local/lib/python/multiprocessing/process.py", line
> 258, in _bootstrap
> self.run()
> File "/home/sage/sage-6.4.1/local/lib/python2.7/site-
> packages/sage/doctest/forker.py", line 1839, in run
> task(self.options, self.outtmpfile, msgpipe, self.result_queue)
> File "/home/sage/sage-6.4.1/local/lib/python2.7/site-
> packages/sage/doctest/forker.py", line 2159, in __call__
> result_queue.put(result, False)
> File
> "/home/sage/sage-6.4.1/local/lib/python/multiprocessing/queues.py", line
> 102, in put
> raise Full
> Full
> Bad exit: 1
> **********************************************************************
> Tests run before process (pid=108) failed:
> sage: from sage.interfaces.qsieve import qsieve ## line 2 ##
> sage: k = 19; n = next_prime(10^k)*next_prime(10^(k+1)) ## line 3 ##
> sage: q = qsieve(next_prime(10^20)*next_prime(10^21), block=False) ##
> line 5 ##
> sage: sig_on_count() ## line 6 ##
> 0
>
> **********************************************************************
> ----------------------------------------------------------------------
> sage -t bla.py # Bad exit: 1
> ----------------------------------------------------------------------
> Total time for all tests: 0.4 seconds
> cpu time: 0.0 seconds
> cumulative wall time: 0.0 seconds
> [sage@63182b1b1561 sage-6.4.1]$ logout
> }}}
>
> This was tested with the centos-based image built by Volker:
>
> https://github.com/sagemath/docker
>
> and a variety of homemade docker images built on several other linux
> distributions (debian) and various recent versions of Sage (including
> 6.4.1 and 6.5). The failure is systematic.
>
> Investing this further, it's the doctesting framework that fails in
> forker.py. There seems to be a race condition at the complete end,
> after collecting and reporting the results of the slaves: on line
> 2159, forker wants to put back something in the queue which is
> reported as Full. If one tests just before whether the queue is empty,
> the answer is positive, and the error disappears (isn't that a nice
> heisenbug!!!). The error disappears as well if one inserts a little
> sleep (no less than 0.3s!).
>
> It's very frustrating not to have a better understanding of the cause
> of the problem, and why it appears specifically under docker and those
> files. If someone understands precisely the problem and can fix the
> problem now, that's great. Otherwise, as this is a blocker for moving
> further with deploying patchbots, we propose in the attached branch to
> insert the emptyness test as a temporary workaround.
>
> Thread on sage-devel: https://groups.google.com/d/msg/sage-
> devel/8hWBzBMjKXI/5DfW8H3nKbAJ
New description:
We are investigating the usage of docker to run multiple instances of
the sage patchbot within sandboxes on a cluster. This works smoothly,
except that the doctesting framework fails on three files:
{{{
sage -t --warn-long 19.5 src/sage/interfaces/qsieve.py # Bad exit: 1
sage -t --warn-long 19.5 local/lib/python2.7/site-
packages/sagenb-0.11.4-py2.7.egg/sagenb/notebook/worksheet.py
# Bad exit: 1
sage -t --warn-long 19.5 local/lib/python2.7/site-
packages/sagenb-0.11.4-py2.7.egg/sagenb/notebook/cell.py
# Bad exit: 1
}}}
Minimal steps to reproduce and complete log:
{{{
# Install docker (here for Debian / Ubuntu):
> sudo apt-get install docker.io
# Run a docker container with Sage:
> docker run -t -i --entrypoint "sudo" sagemath/sage su - sage
Last login: Thu Nov 27 15:28:05 GMT 2014
[sage@63182b1b1561 ~]$ uname -a
Linux 63182b1b1561 3.13.0-43-generic #72-Ubuntu SMP Mon Dec 8 19:35:06 UTC
2014 x86_64 x86_64 x86_64 GNU/Linux
[sage@63182b1b1561 ~]$ cd /opt/sage
[sage@63182b1b1561 sage-6.4.1]$ ./sage -v
SageMath Version 6.7, Release Date: 2015-05-17
[sage@63182b1b1561 sage-6.4.1]$ cat > bla.py << EOF
r"""
sage: from sage.interfaces.qsieve import qsieve
sage: q = qsieve(next_prime(10^20)*next_prime(10^21), block=False)
"""
EOF
[sage@63182b1b1561 sage]$ ./sage -t --verbose bla.py
init.sage does not exist ... creating
no stored timings available
Running doctests with ID 2015-03-04-07-53-04-191d76e2.
Doctesting 1 file.
sage -t bla.py
Trying (line 2): from sage.interfaces.qsieve import qsieve
Expecting nothing
ok [0.00 s]
Trying (line 3): k = 19; n = next_prime(10^k)*next_prime(10^(k+1))
Expecting nothing
ok [0.00 s]
Trying (line 5): q = qsieve(next_prime(10^20)*next_prime(10^21),
block=False)
Expecting nothing
ok [0.23 s]
Trying (line 6): sig_on_count()
Expecting:
0
ok [0.00 s]
1 item passed all tests:
4 tests in bla
4 tests in 1 item.
4 passed and 0 failed.
Test passed.
Process DocTestWorker-1:
Traceback (most recent call last):
File
"/home/sage/sage-6.4.1/local/lib/python/multiprocessing/process.py", line
258, in _bootstrap
self.run()
File "/home/sage/sage-6.4.1/local/lib/python2.7/site-
packages/sage/doctest/forker.py", line 1839, in run
task(self.options, self.outtmpfile, msgpipe, self.result_queue)
File "/home/sage/sage-6.4.1/local/lib/python2.7/site-
packages/sage/doctest/forker.py", line 2159, in __call__
result_queue.put(result, False)
File "/home/sage/sage-6.4.1/local/lib/python/multiprocessing/queues.py",
line 102, in put
raise Full
Full
Bad exit: 1
**********************************************************************
Tests run before process (pid=108) failed:
sage: from sage.interfaces.qsieve import qsieve ## line 2 ##
sage: k = 19; n = next_prime(10^k)*next_prime(10^(k+1)) ## line 3 ##
sage: q = qsieve(next_prime(10^20)*next_prime(10^21), block=False) ## line
5 ##
sage: sig_on_count() ## line 6 ##
0
**********************************************************************
----------------------------------------------------------------------
sage -t bla.py # Bad exit: 1
----------------------------------------------------------------------
Total time for all tests: 0.4 seconds
cpu time: 0.0 seconds
cumulative wall time: 0.0 seconds
[sage@63182b1b1561 sage-6.4.1]$ logout
}}}
This was tested with the centos-based image built by Volker:
https://github.com/sagemath/docker
and a variety of homemade docker images built on several other linux
distributions (debian) and various recent versions of Sage (including
6.4.1, 6.5, 6.7). The failure is systematic.
Investing this further, it's the doctesting framework that fails in
forker.py. There seems to be a race condition at the complete end,
after collecting and reporting the results of the slaves: on line
2159, forker wants to put back something in the queue which is
reported as Full. If one tests just before whether the queue is empty,
the answer is positive, and the error disappears (isn't that a nice
heisenbug!!!). The error disappears as well if one inserts a little
sleep (no less than 0.3s!).
It's very frustrating not to have a better understanding of the cause
of the problem, and why it appears specifically under docker and those
files. If someone understands precisely the problem and can fix the
problem now, that's great. Otherwise, as this is a blocker for moving
further with deploying patchbots, we propose in the attached branch to
insert the emptyness test as a temporary workaround.
Thread on sage-devel: https://groups.google.com/d/msg/sage-
devel/8hWBzBMjKXI/5DfW8H3nKbAJ
--
--
Ticket URL: <http://trac.sagemath.org/ticket/17924#comment:26>
Sage <http://www.sagemath.org>
Sage: Creating a Viable Open Source Alternative to Magma, Maple, Mathematica,
and MATLAB
--
You received this message because you are subscribed to the Google Groups
"sage-trac" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/sage-trac.
For more options, visit https://groups.google.com/d/optout.
