areusch commented on issue #4953:
URL: https://github.com/apache/incubator-tvm/issues/4953#issuecomment-617491801
I see this reliably with a virtualenv on bionic on AWS.
environment:
ami:
`099720109477/ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20200408`
python version
```$ python --version
Python 3.7.5
```
installed python packages:
```
antlr4-python3-runtime==4.8
Cython==0.29.16
decorator==4.4.2
psutil==5.7.0
pylint==2.4.4
- astroid [required: >=2.3.0,<2.4, installed: 2.3.3]
- lazy-object-proxy [required: ==1.4.*, installed: 1.4.3]
- six [required: ~=1.12, installed: 1.14.0]
- typed-ast [required: >=1.4.0,<1.5, installed: 1.4.1]
- wrapt [required: ==1.11.*, installed: 1.11.2]
- isort [required: >=4.2.5,<5, installed: 4.3.21]
- mccabe [required: >=0.6,<0.7, installed: 0.6.1]
pytest==5.4.1
- attrs [required: >=17.4.0, installed: 19.3.0]
- importlib-metadata [required: >=0.12, installed: 1.6.0]
- zipp [required: >=0.5, installed: 3.1.0]
- more-itertools [required: >=4.0.0, installed: 8.2.0]
- packaging [required: Any, installed: 20.3]
- pyparsing [required: >=2.0.2, installed: 2.4.7]
- six [required: Any, installed: 1.14.0]
- pluggy [required: >=0.12,<1.0, installed: 0.13.1]
- importlib-metadata [required: >=0.12, installed: 1.6.0]
- zipp [required: >=0.5, installed: 3.1.0]
- py [required: >=1.5.0, installed: 1.8.1]
- wcwidth [required: Any, installed: 0.1.9]
tornado==6.0.4
xgboost==1.0.2
- numpy [required: Any, installed: 1.18.3]
- scipy [required: Any, installed: 1.4.1]
- numpy [required: >=1.13.3, installed: 1.18.3]
```
backtrace:
```
~/ws/tvm$ gdb python
GNU gdb (Ubuntu 8.1-0ubuntu3.2) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from python...(no debugging symbols found)...done.
(gdb) run tests/python/unittest/test_autotvm_xgboost_model.py
Starting program:
/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/bin/python
tests/python/unittest/test_autotvm_xgboost_model.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff4104700 (LWP 11905)]
[New Thread 0x7ffff3903700 (LWP 11906)]
[New Thread 0x7fffef102700 (LWP 11907)]
[New Thread 0x7fffec901700 (LWP 11908)]
[New Thread 0x7fffea100700 (LWP 11909)]
[New Thread 0x7fffe78ff700 (LWP 11910)]
[New Thread 0x7fffe50fe700 (LWP 11911)]
[New Thread 0x7fffe28fd700 (LWP 11912)]
[New Thread 0x7fffe00fc700 (LWP 11913)]
[New Thread 0x7fffdd8fb700 (LWP 11914)]
[New Thread 0x7fffdb0fa700 (LWP 11915)]
[New Thread 0x7fffd88f9700 (LWP 11916)]
[New Thread 0x7fffd60f8700 (LWP 11917)]
[New Thread 0x7fffd38f7700 (LWP 11918)]
[New Thread 0x7fffd10f6700 (LWP 11919)]
[Thread 0x7fffe50fe700 (LWP 11911) exited]
[Thread 0x7fffd88f9700 (LWP 11916) exited]
[Thread 0x7fffd10f6700 (LWP 11919) exited]
[Thread 0x7fffd60f8700 (LWP 11917) exited]
[Thread 0x7fffdb0fa700 (LWP 11915) exited]
[Thread 0x7fffdd8fb700 (LWP 11914) exited]
[Thread 0x7fffe00fc700 (LWP 11913) exited]
[Thread 0x7fffe28fd700 (LWP 11912) exited]
[Thread 0x7fffe78ff700 (LWP 11910) exited]
[Thread 0x7fffea100700 (LWP 11909) exited]
[Thread 0x7fffec901700 (LWP 11908) exited]
[Thread 0x7fffef102700 (LWP 11907) exited]
[Thread 0x7ffff3903700 (LWP 11906) exited]
[Thread 0x7ffff4104700 (LWP 11905) exited]
[Thread 0x7fffd38f7700 (LWP 11918) exited]
[New Thread 0x7fffd10f6700 (LWP 11936)]
[New Thread 0x7fffd38f7700 (LWP 11937)]
[New Thread 0x7fffd60f8700 (LWP 11938)]
[Thread 0x7fffd10f6700 (LWP 11936) exited]
[Thread 0x7fffd60f8700 (LWP 11938) exited]
[Thread 0x7fffd38f7700 (LWP 11937) exited]
[New Thread 0x7fffd38f7700 (LWP 11955)]
[New Thread 0x7fffd60f8700 (LWP 11956)]
[New Thread 0x7fffd10f6700 (LWP 11957)]
Thread 1 "python" received signal SIGSEGV, Segmentation fault.
0x00007fffb5ba9e37 in std::vector<std::pair<std::string, std::string>,
std::allocator<std::pair<std::string, std::string> > >
xgboost::XGBoostParameter<xgboost::GenericParameter>::UpdateAllowUnknown<std::vector<std::pair<std::string,
std::string>, std::allocator<std::pair<std::string, std::string> > >
>(std::vector<std::pair<std::string, std::string>,
std::allocator<std::pair<std::string, std::string> > > const&, bool*) ()
from
/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/xgboost/./lib/libxgboost.so
(gdb) bt
#0 0x00007fffb5ba9e37 in std::vector<std::pair<std::string, std::string>,
std::allocator<std::pair<std::string, std::string> > >
xgboost::XGBoostParameter<xgboost::GenericParameter>::UpdateAllowUnknown<std::vector<std::pair<std::string,
std::string>, std::allocator<std::pair<std::string, std::string> > >
>(std::vector<std::pair<std::string, std::string>,
std::allocator<std::pair<std::string, std::string> > > const&, bool*) ()
from
/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/xgboost/./lib/libxgboost.so
#1 0x00007fffb5b970b7 in xgboost::GenericParameter::ConfigureGpuId(bool) ()
from
/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/xgboost/./lib/libxgboost.so
#2 0x00007fffb5bb2cd1 in xgboost::LearnerImpl::Configure() ()
from
/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/xgboost/./lib/libxgboost.so
#3 0x00007fffb5bad69d in xgboost::LearnerImpl::UpdateOneIter(int,
xgboost::DMatrix*) ()
from
/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/xgboost/./lib/libxgboost.so
#4 0x00007fffb5ab0639 in XGBoosterUpdateOneIter ()
from
/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/xgboost/./lib/libxgboost.so
#5 0x00007fffce06adae in ffi_call_unix64 () from
/usr/lib/x86_64-linux-gnu/libffi.so.6
#6 0x00007fffce06a71f in ffi_call () from
/usr/lib/x86_64-linux-gnu/libffi.so.6
#7 0x00007fffce27e944 in _ctypes_callproc () from
/usr/lib/python3.7/lib-dynload/_ctypes.cpython-37m-x86_64-linux-gnu.so
#8 0x00007fffce27efb3 in ?? () from
/usr/lib/python3.7/lib-dynload/_ctypes.cpython-37m-x86_64-linux-gnu.so
#9 0x00000000005d96db in _PyObject_FastCallKeywords ()
#10 0x000000000054aa51 in ?? ()
#11 0x0000000000551c08 in _PyEval_EvalFrameDefault ()
#12 0x000000000054b302 in _PyEval_EvalCodeWithName ()
#13 0x00000000005d8bd2 in _PyFunction_FastCallKeywords ()
#14 0x000000000054de9e in _PyEval_EvalFrameDefault ()
#15 0x000000000054b302 in _PyEval_EvalCodeWithName ()
#16 0x00000000005d8bd2 in _PyFunction_FastCallKeywords ()
#17 0x000000000054a880 in ?? ()
#18 0x000000000054ebbd in _PyEval_EvalFrameDefault ()
#19 0x000000000054b302 in _PyEval_EvalCodeWithName ()
#20 0x00000000005d8bd2 in _PyFunction_FastCallKeywords ()
#21 0x000000000054a880 in ?? ()
#22 0x000000000054ebbd in _PyEval_EvalFrameDefault ()
#23 0x000000000054b302 in _PyEval_EvalCodeWithName ()
#24 0x00000000005d8bd2 in _PyFunction_FastCallKeywords ()
#25 0x000000000054a880 in ?? ()
#26 0x000000000054ebbd in _PyEval_EvalFrameDefault ()
#27 0x00000000005d88dc in _PyFunction_FastCallKeywords ()
#28 0x000000000054dd08 in _PyEval_EvalFrameDefault ()
#29 0x000000000054b302 in _PyEval_EvalCodeWithName ()
#30 0x000000000054d803 in PyEval_EvalCode ()
#31 0x00000000006308e2 in ?? ()
#32 0x0000000000630997 in PyRun_FileExFlags ()
#33 0x000000000063160f in PyRun_SimpleFileExFlags ()
#34 0x000000000065450e in ?? ()
#35 0x000000000065486e in _Py_UnixMain ()
#36 0x00007ffff7a05b97 in __libc_start_main (main=0x4b84d0 <main>, argc=2,
argv=0x7fffffffdfc8, init=<optimized out>, fini=<optimized out>,
rtld_fini=<optimized out>, stack_end=0x7fffffffdfb8) at
../csu/libc-start.c:310
#37 0x00000000005df80a in _start ()
(gdb)
```
pytest log:
```
~/ws/tvm$ TVM_FFI=ctypes python3 -m pytest -s -v tests/python/unittest -k
'test_autotvm_xgboost_model'
================================================================ test
session starts =================================================================
platform linux -- Python 3.7.5, pytest-5.4.1, py-1.8.1, pluggy-0.13.1 --
/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/bin/python3
cachedir: .pytest_cache
rootdir: /home/ubuntu/ws/tvm
collecting 88 items
Testing
using contexts: [cpu(0)]
collected 562 items / 560 deselected / 2 selected
tests/python/unittest/test_autotvm_xgboost_model.py::test_fit Fatal Python
error: Segmentation fault
Thread 0x00007f6c59dd9700 (most recent call first):
File "/usr/lib/python3.7/multiprocessing/connection.py", line 379 in _recv
File "/usr/lib/python3.7/multiprocessing/connection.py", line 407 in
_recv_bytes
File "/usr/lib/python3.7/multiprocessing/connection.py", line 250 in recv
File "/usr/lib/python3.7/multiprocessing/pool.py", line 470 in
_handle_results
File "/usr/lib/python3.7/threading.py", line 870 in run
File "/usr/lib/python3.7/threading.py", line 926 in _bootstrap_inner
File "/usr/lib/python3.7/threading.py", line 890 in _bootstrap
Thread 0x00007f6c5cddb700 (most recent call first):
File "/usr/lib/python3.7/multiprocessing/pool.py", line 422 in
_handle_tasks
File "/usr/lib/python3.7/threading.py", line 870 in run
File "/usr/lib/python3.7/threading.py", line 926 in _bootstrap_inner
File "/usr/lib/python3.7/threading.py", line 890 in _bootstrap
Thread 0x00007f6c5c5da700 (most recent call first):
File "/usr/lib/python3.7/multiprocessing/pool.py", line 413 in
_handle_workers
File "/usr/lib/python3.7/threading.py", line 870 in run
File "/usr/lib/python3.7/threading.py", line 926 in _bootstrap_inner
File "/usr/lib/python3.7/threading.py", line 890 in _bootstrap
Current thread 0x00007f6c7f460740 (most recent call first):
File
"/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/xgboost/core.py",
line 1249 in update
File
"/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/xgboost/training.py",
line 74 in _train_internal
File
"/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/xgboost/training.py",
line 209 in train
File "/home/ubuntu/ws/tvm/python/tvm/autotvm/tuner/xgboost_cost_model.py",
line 272 in fit_log
File
"/home/ubuntu/ws/tvm/tests/python/unittest/test_autotvm_xgboost_model.py", line
35 in test_fit
File
"/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/_pytest/python.py",
line 184 in pytest_pyfunc_call
File
"/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/callers.py",
line 187 in _multicall
File
"/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/manager.py",
line 87 in <lambda>
File
"/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/manager.py",
line 93 in _hookexec
File
"/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/hooks.py",
line 286 in __call__
File
"/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/_pytest/python.py",
line 1479 in runtest
File
"/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/_pytest/runner.py",
line 135 in pytest_runtest_call
File
"/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/callers.py",
line 187 in _multicall
File
"/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/manager.py",
line 87 in <lambda>
File
"/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/manager.py",
line 93 in _hookexec
File
"/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/hooks.py",
line 286 in __call__
File
"/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/_pytest/runner.py",
line 217 in <lambda>
File
"/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/_pytest/runner.py",
line 244 in from_call
File
"/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/_pytest/runner.py",
line 217 in call_runtest_hook
File
"/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/_pytest/runner.py",
line 186 in call_and_report
File
"/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/_pytest/runner.py",
line 100 in runtestprotocol
File
"/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/_pytest/runner.py",
line 85 in pytest_runtest_protocol
File
"/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/callers.py",
line 187 in _multicall
File
"/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/manager.py",
line 87 in <lambda>
File
"/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/manager.py",
line 93 in _hookexec
File
"/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/hooks.py",
line 286 in __call__
File
"/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/_pytest/main.py",
line 272 in pytest_runtestloop
File
"/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/callers.py",
line 187 in _multicall
File
"/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/manager.py",
line 87 in <lambda>
File
"/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/manager.py",
line 93 in _hookexec
File
"/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/hooks.py",
line 286 in __call__
File
"/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/_pytest/main.py",
line 247 in _main
File
"/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/_pytest/main.py",
line 191 in wrap_session
File
"/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/_pytest/main.py",
line 240 in pytest_cmdline_main
File
"/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/callers.py",
line 187 in _multicall
File
"/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/manager.py",
line 87 in <lambda>
File
"/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/manager.py",
line 93 in _hookexec
File
"/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pluggy/hooks.py",
line 286 in __call__
File
"/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/_pytest/config/__init__.py",
line 125 in main
File
"/home/ubuntu/.local/share/virtualenvs/tvm-FxJJpK7X/lib/python3.7/site-packages/pytest/__main__.py",
line 7 in <module>
File "/usr/lib/python3.7/runpy.py", line 85 in _run_code
File "/usr/lib/python3.7/runpy.py", line 193 in _run_module_as_main
Segmentation fault (core dumped)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
