This is easily and repeatably duplicatable on in my lxc lucid container.
The error for the last test (with the multiprocessing SemLock OSError
seems the most likely to be a shared root cause to me. Interestingly,
the OSError was slightly different for me, with a newer lxc: "OSError:
[Errno 38] Function not implemented". I checked the syslog and some
random other log files, without really knowing where to look, and saw
nothing of interest.
As you'd expect from the traceback, you can repeat this error with a
single Python call to the standard library.
>>> from multiprocessing import synchronize
>>> synchronize.Lock()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.6/multiprocessing/synchronize.py", line 117, in
__init__
SemLock.__init__(self, SEMAPHORE, 1, 1)
File "/usr/lib/python2.6/multiprocessing/synchronize.py", line 49, in __init__
sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue)
OSError: [Errno 38] Function not implemented
38 is ENOSYS
I talked with hallyn and this is an lxc issue. Adding this line to the
container's fstab will fix the problem:
none dev/shm tmpfs defaults 0 0
He is updating the ubuntu template with this change.
** Changed in: launchpad
Status: Triaged => Invalid
** Also affects: lxc (Ubuntu)
Importance: Undecided
Status: New
** Summary changed:
- celery tests fail on parallel testing instances
+ Semaphores cannot be created in lxc container
** Description changed:
- The same three tests failed consistently over several runs:
+ >>> from multiprocessing import synchronize
+ >>> synchronize.Lock()
+ Traceback (most recent call last):
+ File "<stdin>", line 1, in <module>
+ File "/usr/lib/python2.6/multiprocessing/synchronize.py", line 117, in
__init__
+ SemLock.__init__(self, SEMAPHORE, 1, 1)
+ File "/usr/lib/python2.6/multiprocessing/synchronize.py", line 49, in
__init__
+ sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue)
+ OSError: [Errno 38] Function not implemented
+ Thanks to hallyn, there's a workaround and a fix: adding this line to the
container's fstab will fix the problem:
+ none dev/shm tmpfs defaults 0 0
- ======================================================================
- FAIL:
lp.code.model.tests.test_branch.TestBranchJobViaCelery.test_destroySelf_via_celery
- ----------------------------------------------------------------------
- Traceback (most recent call last):
- _StringException: librarian-log: {{{
- 2012-04-05 19:56:24+0530 [-] Log opened.
- 2012-04-05 19:56:24+0530 [-] twistd 11.1.0 (/usr/bin/python2.6 2.6.5)
starting up.
- 2012-04-05 19:56:24+0530 [-] reactor class:
twisted.internet.pollreactor.PollReactor.
- 2012-04-05 19:56:24+0530 [-] FileUploadFactory starting on 58125
- 2012-04-05 19:56:24+0530 [-] Starting factory
<lp.services.librarianserver.libraryprotocol.FileUploadFactory instance at
0xa2e90ec>
- 2012-04-05 19:56:24+0530 [-] Site starting on 51003
- 2012-04-05 19:56:24+0530 [-] Starting factory <twisted.web.server.Site
instance at 0xa2e94ac>
- 2012-04-05 19:56:24+0530 [-] FileUploadFactory starting on 51219
- 2012-04-05 19:56:24+0530 [-] Starting factory
<lp.services.librarianserver.libraryprotocol.FileUploadFactory instance at
0xa2e978c>
- 2012-04-05 19:56:24+0530 [-] Site starting on 35098
- 2012-04-05 19:56:24+0530 [-] Starting factory <twisted.web.server.Site
instance at 0xa2e96ac>
- 2012-04-05 19:56:24+0530 [-] Not using upstream librarian
- 2012-04-05 19:56:24+0530 [-] daemon ready!
- }}}
+ He is updating the ubuntu template with this change.
- Traceback (most recent call last):
- File
"/var/lib/buildbot/slaves/slave/lucid-devel/build/lib/lp/code/model/tests/test_branch.py",
line 364, in test_destroySelf_via_celery
- self.assertThat(branch_path, Not(PathExists()))
- MismatchError: '/var/tmp/bazaar.launchpad.dev/mirrors/00/00/00/4d' matches
MatchesPredicate(<function exists at 0xf73b5df4>, '%s does not exist.')
- ======================================================================
- ERROR:
lp.code.model.tests.test_branch.TestBranchUpgrade.test_requestUpgradeUsesCelery
- ----------------------------------------------------------------------
- Traceback (most recent call last):
- _StringException: librarian-log: {{{
- 2012-04-05 19:55:15+0530 [-] Log opened.
- 2012-04-05 19:55:15+0530 [-] twistd 11.1.0 (/usr/bin/python2.6 2.6.5)
starting up.
- 2012-04-05 19:55:15+0530 [-] reactor class:
twisted.internet.pollreactor.PollReactor.
- 2012-04-05 19:55:15+0530 [-] FileUploadFactory starting on 39820
- 2012-04-05 19:55:15+0530 [-] Starting factory
<lp.services.librarianserver.libraryprotocol.FileUploadFactory instance at
0xa6d10ec>
- 2012-04-05 19:55:15+0530 [-] Site starting on 53904
- 2012-04-05 19:55:15+0530 [-] Starting factory <twisted.web.server.Site
instance at 0xa6d14ac>
- 2012-04-05 19:55:15+0530 [-] FileUploadFactory starting on 51169
- 2012-04-05 19:55:15+0530 [-] Starting factory
<lp.services.librarianserver.libraryprotocol.FileUploadFactory instance at
0xa6d178c>
- 2012-04-05 19:55:15+0530 [-] Site starting on 52093
- 2012-04-05 19:55:15+0530 [-] Starting factory <twisted.web.server.Site
instance at 0xa6d16ac>
- 2012-04-05 19:55:15+0530 [-] Not using upstream librarian
- 2012-04-05 19:55:15+0530 [-] daemon ready!
- }}}
-
- Traceback (most recent call last):
- File
"/var/lib/buildbot/slaves/slave/lucid-devel/build/lib/lp/code/model/tests/test_branch.py",
line 833, in test_requestUpgradeUsesCelery
- responses[-1].wait(30)
- File
"/var/lib/buildbot/dependencies/eggs/celery-2.5.1-py2.6.egg/celery/result.py",
line 95, in get
- interval=interval)
- File
"/var/lib/buildbot/dependencies/eggs/celery-2.5.1-py2.6.egg/celery/backends/amqp.py",
line 137, in wait_for
- raise TimeoutError("The operation timed out.")
- TimeoutError: The operation timed out.
- ======================================================================
- FAIL:
lp.code.model.tests.test_branch.TestBranchJobViaCelery.test_branchChanged_via_celery
- ----------------------------------------------------------------------
- Traceback (most recent call last):
- _StringException: librarian-log: {{{
- 2012-04-05 19:55:15+0530 [-] Log opened.
- 2012-04-05 19:55:15+0530 [-] twistd 11.1.0 (/usr/bin/python2.6 2.6.5)
starting up.
- 2012-04-05 19:55:15+0530 [-] reactor class:
twisted.internet.pollreactor.PollReactor.
- 2012-04-05 19:55:15+0530 [-] FileUploadFactory starting on 39820
- 2012-04-05 19:55:15+0530 [-] Starting factory
<lp.services.librarianserver.libraryprotocol.FileUploadFactory instance at
0xa6d10ec>
- 2012-04-05 19:55:15+0530 [-] Site starting on 53904
- 2012-04-05 19:55:15+0530 [-] Starting factory <twisted.web.server.Site
instance at 0xa6d14ac>
- 2012-04-05 19:55:15+0530 [-] FileUploadFactory starting on 51169
- 2012-04-05 19:55:15+0530 [-] Starting factory
<lp.services.librarianserver.libraryprotocol.FileUploadFactory instance at
0xa6d178c>
- 2012-04-05 19:55:15+0530 [-] Site starting on 52093
- 2012-04-05 19:55:15+0530 [-] Starting factory <twisted.web.server.Site
instance at 0xa6d16ac>
- 2012-04-05 19:55:15+0530 [-] Not using upstream librarian
- 2012-04-05 19:55:15+0530 [-] daemon ready!
- }}}
-
- Traceback (most recent call last):
- File
"/var/lib/buildbot/slaves/slave/lucid-devel/build/lib/lp/code/model/tests/test_branch.py",
line 333, in test_branchChanged_via_celery
- 'Updating branch scanner status: 1 revs', proc.stderr.read())
- MismatchError: 'Updating branch scanner status: 1 revs' not in
'/var/lib/buildbot/dependencies/eggs/ampoule-0.2.0-py2.6.egg/ampoule/main.py:4:
DeprecationWarning: the sets module is deprecated\n import sets\n[2012-04-05
19:56:01,602: WARNING/MainProcess] -------------- celery@lptests-temp-7AmBFVK
v2.5.1\n---- **** -----\n--- * *** * -- [Configuration]\n-- * - **** --- .
broker: amqp://guest@localhost:60868//\n- ** ---------- . loader:
celery.loaders.default.Loader\n- ** ---------- . logfile:
[stderr]@INFO\n- ** ---------- . concurrency: 1\n- ** ---------- . events:
OFF\n- *** --- * --- . beat: OFF\n-- ******* ----\n--- ***** -----
[Queues]\n -------------- . standard: exchange:standard (direct)
binding:standard\n \n\n[Tasks]\n .
lp.services.job.celeryjob.CeleryRunJob\n[2012-04-05 19:56:01,742:
ERROR/MainProcess] Unrecoverable error: OSError(13, \'Permission
denied\')\nTraceback (most recent call last):\n File
"/var/lib/buildbot/dependencies/eggs/celery-2.5.1-py2.6.egg/celery/worker/__init__.py",
line 230, in start\n component.start()\n File
"/var/lib/buildbot/dependencies/eggs/celery-2.5.1-py2.6.egg/celery/concurrency/base.py",
line 83, in start\n self.on_start()\n File
"/var/lib/buildbot/dependencies/eggs/celery-2.5.1-py2.6.egg/celery/concurrency/processes/__init__.py",
line 67, in on_start\n **self.options)\n File
"/var/lib/buildbot/dependencies/eggs/celery-2.5.1-py2.6.egg/celery/concurrency/processes/pool.py",
line 537, in __init__\n self._setup_queues()\n File
"/var/lib/buildbot/dependencies/eggs/celery-2.5.1-py2.6.egg/celery/concurrency/processes/pool.py",
line 720, in _setup_queues\n self._inqueue = SimpleQueue()\n File
"/usr/lib/python2.6/multiprocessing/queues.py", line 328, in __init__\n
self._rlock = Lock()\n File
"/usr/lib/python2.6/multiprocessing/synchronize.py", line 117, in __init__\n
SemLock.__init__(self, SEMAPHORE, 1, 1)\n File
"/usr/lib/python2.6/multiprocessing/synchronize.py", line 49, in __init__\n
sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue)\nOSError:
[Errno 13] Permission denied\n[2012-04-05 19:56:01,744: INFO/MainProcess]
process shutting down\n'
+ Original bug report description was Launchpad related (celery tests fail
+ on parallel testing instances)
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to lxc in Ubuntu.
https://bugs.launchpad.net/bugs/974584
Title:
Semaphores cannot be created in lxc container
To manage notifications about this bug go to:
https://bugs.launchpad.net/launchpad/+bug/974584/+subscriptions
--
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
