[
https://issues.apache.org/jira/browse/ARROW-12241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17316063#comment-17316063
]
Antoine Pitrou commented on ARROW-12241:
----------------------------------------
> , looks the keyboard interrupt is generated deliberately to test the
> cancellation? But why it stops the pytest main process?
Hmm... probably because CSV parsing is too fast on your machine (especially in
debug mode). We can try to tweak the timings and/or make the workload harder.
> [Python] Parallel csv reader cancellation test kills pytest
> -----------------------------------------------------------
>
> Key: ARROW-12241
> URL: https://issues.apache.org/jira/browse/ARROW-12241
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Reporter: Yibo Cai
> Priority: Major
>
> CI job is okay, but it failed from my side. Tested on x86 skylake server with
> 32 cores, and Apple M1 with 8 cores.
> Maybe I missed something?
> Test steps:
> {code:bash}
> $ cmake -GNinja -DCMAKE_BUILD_TYPE=Release -DARROW_COMPUTE=ON
> -DARROW_PARQUET=ON -DARROW_BUILD_TESTS=ON
> -DCMAKE_INSTALL_PREFIX=$(pwd)/_install -DCMAKE_INSTALL_LIBDIR=lib
> -DARROW_PYTHON=ON -DCMAKE_CXX_COMPILER=/usr/bin/clang++-9
> -DCMAKE_C_COMPILER=/usr/bin/clang-9 ..
> $ ninja install
> $ cd ~/arrow/python
> # set LD_LIBRARY_PATH, ARROW_HOME to newly built binaries
> $ python setup.py build_ext --inplace
> $ pytest pyarrow
> {code}
> Error log:
> {code:bash}
> =======================================================================================
> test session starts
> ========================================================================================
> platform linux -- Python 3.6.9, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
> rootdir: /home/cyb/arrow/python, configfile: setup.cfg
> plugins: lazy-fixture-0.6.3, hypothesis-5.41.2
> collected 3802 items / 3 skipped / 3799 selected
>
>
> pyarrow/tests/test_adhoc_memory_leak.py s
>
> [ 0%]
> pyarrow/tests/test_array.py
> ......................s...........................................................................................s.............................................
> [ 4%]
> ............................ss.........
>
> [ 5%]
> pyarrow/tests/test_builder.py ....
>
> [ 5%]
> pyarrow/tests/test_cffi.py ......
>
> [ 5%]
> pyarrow/tests/test_compute.py
> ....................................................................................................................................................
> [ 9%]
> pyarrow/tests/test_convert_builtin.py
> ......................................................................................................................................................
> [ 13%]
> ................................................................................................................................x..................................................x........
> [ 18%]
> ...............ssssss.....................................................................................................sssssss
> [ 21%]
> pyarrow/tests/test_csv.py
> ....................................F.......................
> =============================================================================================
> FAILURES
> =============================================================================================
> ______________________________________________________________________________
> TestParallelCSVRead.test_cancellation
> _______________________________________________________________________________
> self = <pyarrow.tests.test_csv.TestParallelCSVRead
> testMethod=test_cancellation>
> def test_cancellation(self):
> if (threading.current_thread().ident !=
> threading.main_thread().ident):
> pytest.skip("test only works from main Python thread")
> if sys.version_info >= (3, 8):
> raise_signal = signal.raise_signal
> elif os.name == 'nt':
> # On Windows, os.kill() doesn't actually send a signal,
> # it just terminates the process with the given exit code.
> pytest.skip("test requires Python 3.8+ on Windows")
> else:
> # On Unix, emulate raise_signal() with os.kill().
> def raise_signal(signum):
> os.kill(os.getpid(), signum)
> large_csv = b"a,b,c\n" + b"1,2,3\n" * 30000000
> def signal_from_thread():
> time.sleep(0.2)
> raise_signal(signal.SIGINT)
> t1 = time.time()
> with pytest.raises(KeyboardInterrupt) as exc_info:
> threading.Thread(target=signal_from_thread).start()
> > self.read_bytes(large_csv)
> E Failed: DID NOT RAISE <class 'KeyboardInterrupt'>
> pyarrow/tests/test_csv.py:927: Failed
> =========================================================================================
> warnings summary
> =========================================================================================
> ../../archery/lib/python3.6/distutils/__init__.py:4
> /home/cyb/archery/lib/python3.6/distutils/__init__.py:4:
> DeprecationWarning: the imp module is deprecated in favour of importlib; see
> the module's documentation for alternative uses
> import imp
> -- Docs: https://docs.pytest.org/en/stable/warnings.html
> =====================================================================================
> short test summary info
> ======================================================================================
> FAILED pyarrow/tests/test_csv.py::TestParallelCSVRead::test_cancellation -
> Failed: DID NOT RAISE <class 'KeyboardInterrupt'>
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> KeyboardInterrupt
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> pyarrow/error.pxi:221: KeyboardInterrupt
> (to show a full traceback on KeyboardInterrupt use --full-trace)
> ================================================================= 1 failed,
> 864 passed, 21 skipped, 2 xfailed, 1 warning in 4.19s
> ==================================================================
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
