llama90 commented on issue #41312: URL: https://github.com/apache/arrow/issues/41312#issuecomment-2068024528
I checked a few things: I was using Python `3.10` on my environment, but the CI system was using Python `3.11`. So, I set up Python `3.11` with conda and ran the tests again. I found that some tests in `pyarrow/tests/test_dataset.py` were skipped when I tested. But it can be executed with `export PYARROW_TEST_DATASET=ON`. Even though I built it by referring to the contents of ci , an error appears... - https://github.com/apache/arrow/blob/16e20b712ab8dafa9d1ec6154415fe5748fe6135/.github/workflows/python.yml#L137-L157 #### python 3.11 <details><summary>test result</summary> ```bash (pyarrow-dev-311) python -m pytest -k "test_make_write_options_error" ========================================================================================================== test session starts =========================================================================================================== platform darwin -- Python 3.11.9, pytest-8.1.1, pluggy-1.5.0 rootdir: /Users/lama/workspace/arrow-new/python configfile: setup.cfg plugins: hypothesis-6.100.1 collected 7667 items / 7666 deselected / 4 skipped / 1 selected pyarrow/tests/test_dataset.py F [100%] ================================================================================================================ FAILURES ================================================================================================================ _____________________________________________________________________________________________________ test_make_write_options_error ______________________________________________________________________________________________________ def test_make_write_options_error(): # GH-39440: calling make_write_options as a static class method msg_1 = ("make_write_options() should be called on an " "instance of ParquetFileFormat") # GH-41043: In Cython2 all Cython methods were "regular" C extension methods # see: https://github.com/cython/cython/issues/6127#issuecomment-2038153359 msg_2 = ("descriptor 'make_write_options' for " "'pyarrow._dataset_parquet.ParquetFileFormat' objects " "doesn't apply to a 'int'") with pytest.raises(TypeError) as excinfo: > pa.dataset.ParquetFileFormat.make_write_options(43) E AttributeError: module 'pyarrow' has no attribute 'dataset' pyarrow/tests/test_dataset.py:5645: AttributeError ======================================================================================================== short test summary info ========================================================================================================= FAILED pyarrow/tests/test_dataset.py::test_make_write_options_error - AttributeError: module 'pyarrow' has no attribute 'dataset' ============================================================================================= 1 failed, 4 skipped, 7666 deselected in 1.05s ============================================================================================== ``` </details> #### Python 3.10 <details><summary>test result</summary> ``` (pyarrow-dev-310) python -m pytest -k "test_make_write_options_error" ============================================================================================================ test session starts ============================================================================================================= platform darwin -- Python 3.10.13, pytest-7.4.4, pluggy-1.4.0 rootdir: /Users/lama/workspace/arrow-new/python configfile: setup.cfg plugins: hypothesis-6.99.9 collected 7667 items / 7666 deselected / 4 skipped / 1 selected pyarrow/tests/test_dataset.py Fatal Python error: Segmentation fault Current thread 0x000000020330bac0 (most recent call first): File "/Users/lama/workspace/arrow-new/python/pyarrow/tests/test_dataset.py", line 5645 in test_make_write_options_error File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_callers.py", line 102 in _multicall File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_manager.py", line 119 in _hookexec File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_hooks.py", line 501 in __call__ File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/python.py", line 1792 in runtest File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_callers.py", line 102 in _multicall File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_manager.py", line 119 in _hookexec File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_hooks.py", line 501 in __call__ File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/runner.py", line 262 in <lambda> File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/runner.py", line 341 in from_call File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/runner.py", line 261 in call_runtest_hook File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/runner.py", line 222 in call_and_report File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/runner.py", line 133 in runtestprotocol File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_callers.py", line 102 in _multicall File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_manager.py", line 119 in _hookexec File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_hooks.py", line 501 in __call__ File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/main.py", line 350 in pytest_runtestloop File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_callers.py", line 102 in _multicall File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_manager.py", line 119 in _hookexec File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_hooks.py", line 501 in __call__ File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/main.py", line 325 in _main File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/main.py", line 271 in wrap_session File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/main.py", line 318 in pytest_cmdline_main File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_callers.py", line 102 in _multicall File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_manager.py", line 119 in _hookexec File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pluggy/_hooks.py", line 501 in __call__ File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/config/__init__.py", line 169 in main File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/_pytest/config/__init__.py", line 192 in console_main File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/site-packages/pytest/__main__.py", line 5 in <module> File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/runpy.py", line 86 in _run_code File "/Users/lama/anaconda3/envs/pyarrow-dev-310/lib/python3.10/runpy.py", line 196 in _run_module_as_main Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, pyarrow.lib, pyarrow._fs, cython.cimports.libc.math, pyarrow._compute, pyarrow._acero, pyarrow._csv, pyarrow._json, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_ dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pandas._libs.ops, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, pyarrow._dataset, pyarrow._parquet, pyarrow._dataset_parquet, _cffi_backend, pyarrow._pyarrow_cpp_tests, pyarrow._feather (total: 66) [1] 77852 segmentation fault python -m pytest -k "test_make_write_options_error" ``` </details> Additionally, by enabling detailed output for the tests, it has been confirmed that the test is being passed when triggered. - `tests/test_dataset.py::test_make_write_options_error <- ../../../../../Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pyarrow/tests/test_dataset.py PASSED ` - https://github.com/apache/arrow/actions/runs/8772308641/job/24071088984?pr=41313#step:6:90 - https://github.com/apache/arrow/actions/runs/8772308641/job/24071089064?pr=41313#step:6:90 ## To summarize - The test appears not to have been run even though `PYARROW_TEST_DATASET` is `ON`. - Why are tests treated as passed in CI? - If running the test locally, different errors appear depending on the python version. - 3.10: segmentation fault - 3.11: AttributeError: module 'pyarrow' has no attribute 'dataset' -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
