Package: src:joblib
Version: 0.13.0-1
Severity: serious
Tags: ftbfs patch
Dear maintainer:
I tried to build this package in buster but it failed:
--------------------------------------------------------------------------------
[...]
debian/rules build-indep
dh build-indep --buildsystem=python_distutils --with=python2,python3
dh_update_autotools_config -i -O--buildsystem=python_distutils
dh_auto_configure -i -O--buildsystem=python_distutils
dh_auto_configure: Please use the third-party "pybuild" build system instead of
python-distutils
dh_auto_configure: This feature will be removed in compat 12.
dh_auto_build -i -O--buildsystem=python_distutils
dh_auto_build: Please use the third-party "pybuild" build system instead of
python-distutils
dh_auto_build: This feature will be removed in compat 12.
python setup.py build --force
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
[... snipped ...]
joblib/test/test_testing.py::test_check_subprocess_call PASSED
joblib/test/test_testing.py::test_check_subprocess_call_non_matching_regex
PASSED
joblib/test/test_testing.py::test_check_subprocess_call_wrong_command PASSED
joblib/test/test_testing.py::test_check_subprocess_call_non_zero_return_code
PASSED
joblib/test/test_testing.py::test_check_subprocess_call_timeout
/<<PKGBUILDDIR>>/joblib/testing.py:50: UserWarning: Timeout running
['/usr/bin/python2.7', '-c', 'import time\nimport sys\nprint("before sleep on
stdout")\nsys.stdout.flush()\nsys.stderr.write("before sleep on
stderr")\nsys.stderr.flush()\ntime.sleep(1.1)\nprint("process should have be
killed before")\nsys.stdout.flush()']
warnings.warn("Timeout running {}".format(cmd))
PASSED
joblib/test/data/create_numpy_pickle.py::joblib.test.data.create_numpy_pickle.get_joblib_version
PASSED
=================================== FAILURES ===================================
_____________________ test_nested_parallelism_limit[loky] ______________________
backend = 'loky'
@with_multiprocessing
@parametrize('backend', ['loky', 'threading'])
def test_nested_parallelism_limit(backend):
with parallel_backend(backend, n_jobs=2):
backend_types_and_levels = _recursive_backend_info()
if cpu_count() == 1:
second_level_backend_type = 'SequentialBackend'
max_level = 1
else:
second_level_backend_type = 'ThreadingBackend'
max_level = 2
top_level_backend_type = backend.title() + 'Backend'
expected_types_and_levels = [
(top_level_backend_type, 0),
(second_level_backend_type, 1),
('SequentialBackend', max_level),
('SequentialBackend', max_level)
]
> assert backend_types_and_levels == expected_types_and_levels
E AssertionError: assert [('LokyBacken...lBackend', 2)] ==
[('LokyBackend...lBackend', 1)]
E At index 1 diff: ('ThreadingBackend', 1) != ('SequentialBackend', 1)
E Full diff:
E [('LokyBackend', 0),
E - ('ThreadingBackend', 1),
E - ('SequentialBackend', 2),
E ? ^
E + ('SequentialBackend', 1),...
E
E ...Full output truncated (7 lines hidden), use '-vv' to show
joblib/test/test_parallel.py:1452: AssertionError
___________________ test_nested_parallelism_limit[threading] ___________________
backend = 'threading'
@with_multiprocessing
@parametrize('backend', ['loky', 'threading'])
def test_nested_parallelism_limit(backend):
with parallel_backend(backend, n_jobs=2):
backend_types_and_levels = _recursive_backend_info()
if cpu_count() == 1:
second_level_backend_type = 'SequentialBackend'
max_level = 1
else:
second_level_backend_type = 'ThreadingBackend'
max_level = 2
top_level_backend_type = backend.title() + 'Backend'
expected_types_and_levels = [
(top_level_backend_type, 0),
(second_level_backend_type, 1),
('SequentialBackend', max_level),
('SequentialBackend', max_level)
]
> assert backend_types_and_levels == expected_types_and_levels
E AssertionError: assert [('ThreadingB...lBackend', 2)] ==
[('ThreadingBa...lBackend', 1)]
E At index 1 diff: ('ThreadingBackend', 1) != ('SequentialBackend', 1)
E Full diff:
E [('ThreadingBackend', 0),
E - ('ThreadingBackend', 1),
E - ('SequentialBackend', 2),
E ? ^
E + ('SequentialBackend', 1),...
E
E ...Full output truncated (7 lines hidden), use '-vv' to show
joblib/test/test_parallel.py:1452: AssertionError
============== 2 failed, 1099 passed, 19 skipped in 70.93 seconds ==============
[INFO:MainProcess:Dummy-463] process shutting down
[DEBUG:MainProcess:Dummy-463] running all "atexit" finalizers with priority >= 0
[DEBUG:MainProcess:Dummy-463] Interpreter shutting down. Waking up
queue_manager_threads [(<Thread(QueueManagerThread, started daemon
139974192133888)>, <joblib.externals.loky.process_executor._ThreadWakeup
instance at 0x7f4e544197a0>)]
[DEBUG:MainProcess:QueueManagerThread] queue management thread shutting down
[DEBUG:MainProcess:QueueManagerThread] closing call_queue
[DEBUG:MainProcess:QueueManagerThread] telling queue thread to quit
[DEBUG:MainProcess:QueueManagerThread] joining processes
[DEBUG:MainProcess:QueueFeederThread] feeder thread got sentinel -- exiting
[INFO:LokyProcess-239:MainThread] Shutting down worker on sentinel
[INFO:LokyProcess-239:MainThread] process shutting down
[DEBUG:LokyProcess-239:MainThread] running all "atexit" finalizers with
priority >= 0
[DEBUG:LokyProcess-239:MainThread] running the remaining "atexit" finalizers
[INFO:LokyProcess-239:MainThread] process exiting with exitcode 0
[INFO:LokyProcess-239:Dummy-1] process shutting down
[DEBUG:LokyProcess-239:Dummy-1] running all "atexit" finalizers with priority
>= 0
[DEBUG:LokyProcess-239:Dummy-1] running the remaining "atexit" finalizers
[INFO:LokyProcess-240:MainThread] Shutting down worker on sentinel
[INFO:LokyProcess-240:MainThread] process shutting down
[DEBUG:LokyProcess-240:MainThread] running all "atexit" finalizers with
priority >= 0
[DEBUG:LokyProcess-240:MainThread] running the remaining "atexit" finalizers
[INFO:LokyProcess-240:MainThread] process exiting with exitcode 0
[INFO:LokyProcess-240:Dummy-1] process shutting down
[DEBUG:LokyProcess-240:Dummy-1] running all "atexit" finalizers with priority
>= 0
[DEBUG:LokyProcess-240:Dummy-1] running the remaining "atexit" finalizers
[DEBUG:MainProcess:QueueManagerThread] queue management thread clean shutdown
of worker processes: []
[DEBUG:MainProcess:Dummy-463] finalizing pool
[DEBUG:MainProcess:Dummy-463] helping task handler/workers to finish
[DEBUG:MainProcess:Dummy-463] removing tasks from inqueue until task handler
finished
[DEBUG:MainProcess:Thread-40] worker handler exiting
[DEBUG:MainProcess:Thread-41] task handler got sentinel
[DEBUG:MainProcess:Thread-41] task handler sending sentinel to result handler
[DEBUG:MainProcess:Thread-41] task handler sending sentinel to workers
[DEBUG:MainProcess:Thread-41] task handler exiting
[DEBUG:MainProcess:Thread-42] result handler got sentinel
[DEBUG:MainProcess:Thread-42] ensuring that outqueue is not full
[DEBUG:MainProcess:Thread-42] result handler exiting: len(cache)=0,
thread._state=0
[DEBUG:PoolWorker-78:MainThread] worker got sentinel -- exiting
[DEBUG:PoolWorker-78:MainThread] worker exiting after 1 tasks
[INFO:PoolWorker-78:MainThread] process shutting down
[DEBUG:PoolWorker-78:MainThread] running all "atexit" finalizers with priority
>= 0
[DEBUG:PoolWorker-78:MainThread] running the remaining "atexit" finalizers
[INFO:PoolWorker-78:MainThread] process exiting with exitcode 0
[DEBUG:PoolWorker-79:MainThread] worker got sentinel -- exiting
[DEBUG:PoolWorker-79:MainThread] worker exiting after 0 tasks
[INFO:PoolWorker-79:MainThread] process shutting down
[DEBUG:PoolWorker-79:MainThread] running all "atexit" finalizers with priority
>= 0
[DEBUG:PoolWorker-79:MainThread] running the remaining "atexit" finalizers
[INFO:PoolWorker-79:MainThread] process exiting with exitcode 0
[DEBUG:MainProcess:Dummy-463] joining worker handler
[DEBUG:MainProcess:Dummy-463] terminating workers
[DEBUG:MainProcess:Dummy-463] joining task handler
[DEBUG:MainProcess:Dummy-463] joining result handler
[DEBUG:MainProcess:Dummy-463] joining pool workers
[DEBUG:MainProcess:Dummy-463] running the remaining "atexit" finalizers
joblib/externals/loky/backend/semaphore_tracker.py:198: UserWarning:
semaphore_tracker: There appear to be 16 leaked semaphores to clean up at
shutdown
len(cache))
make[1]: *** [debian/rules:25: python-test2.7] Error 1
make[1]: Leaving directory '/<<PKGBUILDDIR>>'
make: *** [debian/rules:10: build-indep] Error 2
dpkg-buildpackage: error: debian/rules build-indep subprocess returned exit
status 2
--------------------------------------------------------------------------------
Version 0.12.5-1 used to work ok in my autobuilders. I see that there
is some code to detect the number of CPUs, so this is clearly intended
to work everywhere, but apparently fails when there is only one CPU.
I've put a bunch of build logs here:
https://people.debian.org/~sanvila/build-logs/joblib/
Assuming the package is useful "as is" on every system and this is
really a bug in the test suite, the patch below works for me.
Note: I've checked and this failure may be reproduced easily on any
system by doing "taskset -c 0 dpkg-buildpackage", but if for whatever
reason you need a single-CPU system to test, please contact me
privately and I will gladly provide one.
Thanks.
--- a/joblib/test/test_parallel.py
+++ b/joblib/test/test_parallel.py
@@ -1429,6 +1429,7 @@ def _recursive_backend_info(limit=3, **kwargs):
return this_level + results[0]
+@pytest.mark.xfail
@with_multiprocessing
@parametrize('backend', ['loky', 'threading'])
def test_nested_parallelism_limit(backend):
