#17924: Failure in doctest framework when running in docker
-------------------------------------+------------------------
Reporter: Vincent.Neri | Owner:
Type: defect | Status: new
Priority: major | Milestone: sage-6.6
Component: doctest framework | Resolution:
Keywords: docker | Merged in:
Authors: Vincent Neri | Reviewers:
Report Upstream: N/A | Work issues:
Branch: | Commit:
Dependencies: | Stopgaps:
-------------------------------------+------------------------
Changes (by nthiery):
* cc: roed, robertwb (removed)
Old description:
> In all cases, all tests pass smoothly, except for three files:
>
> sage -t --warn-long 19.5 src/sage/interfaces/qsieve.py # Bad exit: 1
> sage -t --warn-long 19.5 local/lib/python2.7/site-
> packages/sagenb-0.11.4-py2.7.egg/sagenb/notebook/worksheet.py
> # Bad exit: 1
> sage -t --warn-long 19.5 local/lib/python2.7/site-
> packages/sagenb-0.11.4-py2.7.egg/sagenb/notebook/cell.py
> # Bad exit: 1
>
> In all three cases, it's actually the doctesting framework that fails,
> at the end of forker.py, where it seems to be a race condition.
>
> Traceback (most recent call last):
> File
> "/home/sage/sage-6.4.1/local/lib/python/multiprocessing/process.py", line
> 258, in _bootstrap
> self.run()
> File "/home/sage/sage-6.4.1/local/lib/python2.7/site-
> packages/sage/doctest/forker.py", line 1839, in run
> task(self.options, self.outtmpfile, msgpipe, self.result_queue)
> File "/home/sage/sage-6.4.1/local/lib/python2.7/site-
> packages/sage/doctest/forker.py", line 2159, in __call__
> result_queue.put(result, False)
> File
> "/home/sage/sage-6.4.1/local/lib/python/multiprocessing/queues.py", line
> 102, in put
> raise Full
> Full
> Bad exit: 1
New description:
We are investigating the usage of docker to run multiple instances of
the sage patchbot within sandboxes on a cluster. This works smoothly,
except that the doctesting framework fails on three files:
{{{
sage -t --warn-long 19.5 src/sage/interfaces/qsieve.py # Bad exit: 1
sage -t --warn-long 19.5 local/lib/python2.7/site-
packages/sagenb-0.11.4-py2.7.egg/sagenb/notebook/worksheet.py
# Bad exit: 1
sage -t --warn-long 19.5 local/lib/python2.7/site-
packages/sagenb-0.11.4-py2.7.egg/sagenb/notebook/cell.py
# Bad exit: 1
}}}
Minimal steps to reproduce and complete log:
{{{
root@xxx:~# docker run -t -i sagemath/sage su - sage
Last login: Thu Nov 27 15:28:05 GMT 2014
[sage@63182b1b1561 ~]$ uname -a
Linux 63182b1b1561 3.13.0-43-generic #72-Ubuntu SMP Mon Dec 8 19:35:06 UTC
2014 x86_64 x86_64 x86_64 GNU/Linux
[sage@63182b1b1561 ~]$ cd sage-6.4.1/
[sage@63182b1b1561 sage-6.4.1]$ ./sage -v
Sage Version 6.4.1, Release Date: 2014-11-23
[sage@63182b1b1561 sage-6.4.1]$ cat > bla.py << EOF
> r"""
> sage: from sage.interfaces.qsieve import qsieve
> sage: k = 19; n = next_prime(10^k)*next_prime(10^(k+1))
> sage: v, t = qsieve(n, time=True) # uses qsieve; optional -
time
> sage: q = qsieve(next_prime(10^20)*next_prime(10^21),
block=False)
> """
> EOF
[sage@63182b1b1561 sage-6.4.1]$ ./sage -t --verbose bla.py
init.sage does not exist ... creating
no stored timings available
Running doctests with ID 2015-03-04-07-53-04-191d76e2.
Doctesting 1 file.
sage -t bla.py
Trying (line 2): from sage.interfaces.qsieve import qsieve
Expecting nothing
ok [0.00 s]
Trying (line 3): k = 19; n = next_prime(10^k)*next_prime(10^(k+1))
Expecting nothing
ok [0.00 s]
Trying (line 5): q = qsieve(next_prime(10^20)*next_prime(10^21),
block=False)
Expecting nothing
ok [0.23 s]
Trying (line 6): sig_on_count()
Expecting:
0
ok [0.00 s]
1 item passed all tests:
4 tests in bla
4 tests in 1 item.
4 passed and 0 failed.
Test passed.
Process DocTestWorker-1:
Traceback (most recent call last):
File
"/home/sage/sage-6.4.1/local/lib/python/multiprocessing/process.py", line
258, in _bootstrap
self.run()
File "/home/sage/sage-6.4.1/local/lib/python2.7/site-
packages/sage/doctest/forker.py", line 1839, in run
task(self.options, self.outtmpfile, msgpipe, self.result_queue)
File "/home/sage/sage-6.4.1/local/lib/python2.7/site-
packages/sage/doctest/forker.py", line 2159, in __call__
result_queue.put(result, False)
File "/home/sage/sage-6.4.1/local/lib/python/multiprocessing/queues.py",
line 102, in put
raise Full
Full
Bad exit: 1
**********************************************************************
Tests run before process (pid=108) failed:
sage: from sage.interfaces.qsieve import qsieve ## line 2 ##
sage: k = 19; n = next_prime(10^k)*next_prime(10^(k+1)) ## line 3 ##
sage: q = qsieve(next_prime(10^20)*next_prime(10^21), block=False) ## line
5 ##
sage: sig_on_count() ## line 6 ##
0
**********************************************************************
----------------------------------------------------------------------
sage -t bla.py # Bad exit: 1
----------------------------------------------------------------------
Total time for all tests: 0.4 seconds
cpu time: 0.0 seconds
cumulative wall time: 0.0 seconds
[sage@63182b1b1561 sage-6.4.1]$ logout
}}}
This was tested with the centos-based image built by Volker:
https://github.com/sagemath/docker
and a variety of homemade docker images built on several other linux
distributions (debian) and various recent versions of Sage (including
6.4.1 and 6.5). The failure is systematic.
Investing this further, it's the doctesting framework that fails in
forker.py. There seems to be a race condition at the complete end,
after collecting and reporting the results of the slaves: on line
2159, forker wants to put back something in the queue which is
reported as Full. If one tests just before whether the queue is empty,
the answer is positive, and the error disappears (isn't that a nice
heisenbug!!!). The error disappears as well if one inserts a little
sleep (no less than 0.3s!).
It's very frustrating not to have a better understanding of the cause
of the problem, and why it appears specifically under docker and those
files. If someone understands precisely the problem and can fix the
problem now, that's great. Otherwise, as this is a blocker for moving
further with deploying patchbots, we propose in the attached branch to
insert the emptyness test as a temporary workaround.
--
--
Ticket URL: <http://trac.sagemath.org/ticket/17924#comment:4>
Sage <http://www.sagemath.org>
Sage: Creating a Viable Open Source Alternative to Magma, Maple, Mathematica,
and MATLAB
--
You received this message because you are subscribed to the Google Groups
"sage-trac" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/sage-trac.
For more options, visit https://groups.google.com/d/optout.
