powersj opened a new issue, #40100: URL: https://github.com/apache/arrow/issues/40100
### Describe the bug, including details regarding any error messages, version, and platform. Hi, When using the pyarrow flight client, I have a user who occasionally sees a Windows fatal exception error. This involves a query with multiple subqueries across many fields. I do have access to the environment and can reproduce. We have found that there is some sort of correlation between the number of fields and the exception occurring. As we decrease the number of fields the issue can occur less and less consistently. I realize that getting an issue without exact steps to reproduce is unhelpful. However, I am more than willing to try out test builds or build a customer version to gather more details if I can get some guidance. I was able to easily build a custom version on Linux [per the dev docs](https://github.com/apache/arrow/blob/main/docs/source/developers/python.rst), but I tried building a custom pyarrow on Windows and ran into issues right away with detection of the compiler. I have my steps and logs below. ## Observations 1. This only occurs on Windows 10 or 11; the same query runs fun on Linux/macOS 2. This only occurs when running as a Python notebook, running as a script works 3. It reproduces with both Python 3.11 and 3.12 4. Issues occurs with both a pip-only or conda environment 5. Disabling all virus or Windows security detection does not help 6. A windows event occurs calling out `arrow_flight.dll` ## Windows Event Log Message ```s Faulting application name: python3.12.exe, version: 3.12.1150.1013, time stamp: 0x6572422a Faulting module name: arrow_flight.dll, version: 0.0.0.0, time stamp: 0x65a69ccb Exception code: 0xc0000005 Fault offset: 0x00000000002dc6b0 Faulting process id: 0x0x4F8 Faulting application start time: 0x0x1DA55FAF308D836 Faulting application path: C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.496.0_x64__qbz5n2kfra8p0\python3.12.exe Faulting module path: C:\Users\powersj\v3-ear\.venv\Lib\site-packages\pyarrow\arrow_flight.dll Report Id: f8313105-2c59-4f1a-a8a6-a4227a8ae7d9 Faulting package full name: PythonSoftwareFoundation.Python.3.12_3.12.496.0_x64__qbz5n2kfra8p0 Faulting package-relative application ID: Python ``` ## Code ```python import json import certifi from pyarrow.flight import FlightClient, Ticket, FlightCallOptions import faulthandler faulthandler.enable() host = "host" token = "token" database = "db" with open(certifi.where(), "r", encoding="utf-8") as f_cert: cert = f_cert.read() with open("kernel-crash.sql", "r", encoding="utf-8") as f_sql: query = f_sql.read() options = FlightCallOptions(**{ "headers": [(b"authorization", f"Bearer {token}".encode('utf-8'))], "timeout": 300 }) ticket_data = { "database": database, "sql_query": query, "query_type": "sql", } ticket = Ticket(json.dumps(ticket_data).encode('utf-8')) with FlightClient(f"grpc+tls://{host}:443", tls_root_certs=cert) as client: reader = client.do_get(ticket, options) print(reader.read_all()) ``` ## Traceback ```s Windows fatal exception: access violation Thread 0x000026a8 (most recent call first): File "C:\Users\powersj\v3-ear\.venv\Lib\site-packages\ipykernel\parentpoller.py", line 93 in run File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.752.0_x64__qbz5n2kfra8p0\Lib\threading.py", line 1073 in _bootstrap_inner File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.752.0_x64__qbz5n2kfra8p0\Lib\threading.py", line 1030 in _bootstrap Thread 0x00002700 (most recent call first): File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.752.0_x64__qbz5n2kfra8p0\Lib\threading.py", line 355 in wait File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.752.0_x64__qbz5n2kfra8p0\Lib\threading.py", line 655 in wait File "C:\Users\powersj\v3-ear\.venv\Lib\site-packages\IPython\core\history.py", line 894 in run File "C:\Users\powersj\v3-ear\.venv\Lib\site-packages\IPython\core\history.py", line 60 in only_when_enabled File "C:\Users\powersj\v3-ear\.venv\Lib\site-packages\decorator.py", line 232 in fun File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.752.0_x64__qbz5n2kfra8p0\Lib\threading.py", line 1073 in _bootstrap_inner File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.752.0_x64__qbz5n2kfra8p0\Lib\threading.py", line 1030 in _bootstrap Thread 0x00002620 (most recent call first): File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.752.0_x64__qbz5n2kfra8p0\Lib\selectors.py", line 314 in _select File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.752.0_x64__qbz5n2kfra8p0\Lib\selectors.py", line 323 in select File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.752.0_x64__qbz5n2kfra8p0\Lib\asyncio\base_events.py", line 1947 in _run_once File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.752.0_x64__qbz5n2kfra8p0\Lib\asyncio\base_events.py", line 639 in run_forever File "C:\Users\powersj\v3-ear\.venv\Lib\site-packages\tornado\platform\asyncio.py", line 205 in start File "C:\Users\powersj\v3-ear\.venv\Lib\site-packages\ipykernel\control.py", line 23 in run File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.752.0_x64__qbz5n2kfra8p0\Lib\threading.py", line 1073 in _bootstrap_inner File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.752.0_x64__qbz5n2kfra8p0\Lib\threading.py", line 1030 in _bootstrap Thread 0x00001ba8 (most recent call first): File "C:\Users\powersj\v3-ear\.venv\Lib\site-packages\ipykernel\heartbeat.py", line 106 in run File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.752.0_x64__qbz5n2kfra8p0\Lib\threading.py", line 1073 in _bootstrap_inner File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.752.0_x64__qbz5n2kfra8p0\Lib\threading.py", line 1030 in _bootstrap Thread 0x00001d80 (most recent call first): File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.752.0_x64__qbz5n2kfra8p0\Lib\selectors.py", line 314 in _select File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.752.0_x64__qbz5n2kfra8p0\Lib\selectors.py", line 323 in select File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.752.0_x64__qbz5n2kfra8p0\Lib\asyncio\base_events.py", line 1947 in _run_once File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.752.0_x64__qbz5n2kfra8p0\Lib\asyncio\base_events.py", line 639 in run_forever File "C:\Users\powersj\v3-ear\.venv\Lib\site-packages\tornado\platform\asyncio.py", line 205 in start File "C:\Users\powersj\v3-ear\.venv\Lib\site-packages\ipykernel\iostream.py", line 92 in _thread_main File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.752.0_x64__qbz5n2kfra8p0\Lib\threading.py", line 1010 in run File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.752.0_x64__qbz5n2kfra8p0\Lib\threading.py", line 1073 in _bootstrap_inner File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.752.0_x64__qbz5n2kfra8p0\Lib\threading.py", line 1030 in _bootstrap Current thread 0x000025e0 (most recent call first): File "C:\Users\powersj\AppData\Local\Temp\ipykernel_9720\769077188.py", line 26 in <module> File "C:\Users\powersj\v3-ear\.venv\Lib\site-packages\IPython\core\interactiveshell.py", line 3553 in run_code File "C:\Users\powersj\v3-ear\.venv\Lib\site-packages\IPython\core\interactiveshell.py", line 3493 in run_ast_nodes File "C:\Users\powersj\v3-ear\.venv\Lib\site-packages\IPython\core\interactiveshell.py", line 3311 in run_cell_async File "C:\Users\powersj\v3-ear\.venv\Lib\site-packages\IPython\core\async_helpers.py", line 129 in _pseudo_sync_runner File "C:\Users\powersj\v3-ear\.venv\Lib\site-packages\IPython\core\interactiveshell.py", line 3106 in _run_cell File "C:\Users\powersj\v3-ear\.venv\Lib\site-packages\IPython\core\interactiveshell.py", line 3051 in run_cell File "C:\Users\powersj\v3-ear\.venv\Lib\site-packages\ipykernel\zmqshell.py", line 549 in run_cell File "C:\Users\powersj\v3-ear\.venv\Lib\site-packages\ipykernel\ipkernel.py", line 446 in do_execute File "C:\Users\powersj\v3-ear\.venv\Lib\site-packages\ipykernel\kernelbase.py", line 775 in execute_request File "C:\Users\powersj\v3-ear\.venv\Lib\site-packages\ipykernel\ipkernel.py", line 359 in execute_request File "C:\Users\powersj\v3-ear\.venv\Lib\site-packages\ipykernel\kernelbase.py", line 437 in dispatch_shell File "C:\Users\powersj\v3-ear\.venv\Lib\site-packages\ipykernel\kernelbase.py", line 531 in process_one File "C:\Users\powersj\v3-ear\.venv\Lib\site-packages\ipykernel\kernelbase.py", line 542 in dispatch_queue File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.752.0_x64__qbz5n2kfra8p0\Lib\asyncio\events.py", line 88 in _run File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.752.0_x64__qbz5n2kfra8p0\Lib\asyncio\base_events.py", line 1985 in _run_once File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.752.0_x64__qbz5n2kfra8p0\Lib\asyncio\base_events.py", line 639 in run_forever File "C:\Users\powersj\v3-ear\.venv\Lib\site-packages\tornado\platform\asyncio.py", line 205 in start File "C:\Users\powersj\v3-ear\.venv\Lib\site-packages\ipykernel\kernelapp.py", line 739 in start File "C:\Users\powersj\v3-ear\.venv\Lib\site-packages\traitlets\config\application.py", line 1075 in launch_instance File "C:\Users\powersj\v3-ear\.venv\Lib\site-packages\ipykernel_launcher.py", line 17 in <module> File "<frozen runpy>", line 88 in _run_code File "<frozen runpy>", line 198 in _run_module_as_main ``` ### System Information ```s $ python --version Python 3.11.8 (venv) $ pip list Package Version ----------------- -------- asttokens 2.4.1 certifi 2024.2.2 colorama 0.4.6 comm 0.2.1 debugpy 1.8.1 decorator 5.1.1 executing 2.0.1 ipdb 0.13.13 ipykernel 6.29.2 ipython 8.21.0 jedi 0.19.1 jupyter_client 8.6.0 jupyter_core 5.7.1 matplotlib-inline 0.1.6 nest-asyncio 1.6.0 numpy 1.26.4 packaging 23.2 parso 0.8.3 pip 23.3.1 platformdirs 4.2.0 prompt-toolkit 3.0.43 psutil 5.9.8 pure-eval 0.2.2 pyarrow 15.0.0 Pygments 2.17.2 python-dateutil 2.8.2 pywin32 306 pyzmq 25.1.2 setuptools 69.0.2 six 1.16.0 stack-data 0.6.3 tornado 6.4 traitlets 5.14.1 wcwidth 0.2.13 wheel 0.42.0 ``` When using conda: ``` C:\Users\powersj>conda info active environment : None user config file : C:\Users\powersj\.condarc populated config files : conda version : 23.11.0 conda-build version : not installed python version : 3.11.5.final.0 solver : libmamba (default) virtual packages : __archspec=1=x86_64 __conda=23.11.0=0 __win=0=0 base environment : C:\ProgramData\miniconda3 (read only) conda av data dir : C:\ProgramData\miniconda3\etc\conda conda av metadata url : None channel URLs : https://repo.anaconda.com/pkgs/main/win-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/r/win-64 https://repo.anaconda.com/pkgs/r/noarch https://repo.anaconda.com/pkgs/msys2/win-64 https://repo.anaconda.com/pkgs/msys2/noarch package cache : C:\ProgramData\miniconda3\pkgs C:\Users\powersj\.conda\pkgs C:\Users\powersj\AppData\Local\conda\conda\pkgs envs directories : C:\Users\powersj\.conda\envs C:\ProgramData\miniconda3\envs C:\Users\powersj\AppData\Local\conda\conda\envs platform : win-64 user-agent : conda/23.11.0 requests/2.31.0 CPython/3.11.5 Windows/10 Windows/10.0.22621 solver/libmamba conda-libmamba-solver/23.12.0 libmambapy/1.5.3 administrator : False netrc file : None offline mode : False ``` ## Build Attempt ```cmd C:\Users\powersj>conda create -y -n pyarrow-dev -c conda-forge ^ More? --file arrow\ci\conda_env_cpp.txt ^ More? --file arrow\ci\conda_env_python.txt ^ More? --file arrow\ci\conda_env_gandiva.txt ^ More? python=3.11 <snip> C:\Users\powersj>conda activate pyarrow-dev (pyarrow-dev) C:\Users\powersj>set ARROW_HOME=%CONDA_PREFIX%\Library (pyarrow-dev) C:\Users\powersj>mkdir arrow\cpp\build (pyarrow-dev) C:\Users\powersj>pushd arrow\cpp\build (pyarrow-dev) C:\Users\powersj\arrow\cpp\build>cmake -G "Ninja" ^ More? -DCMAKE_INSTALL_PREFIX=%ARROW_HOME% ^ More? -DCMAKE_UNITY_BUILD=ON ^ More? -DARROW_COMPUTE=ON ^ More? -DARROW_CSV=ON ^ More? -DARROW_CXXFLAGS="/WX /MP" ^ More? -DARROW_DATASET=ON ^ More? -DARROW_FILESYSTEM=ON ^ More? -DARROW_HDFS=ON ^ More? -DARROW_JSON=ON ^ More? -DARROW_PARQUET=ON ^ More? -DARROW_WITH_LZ4=ON ^ More? -DARROW_WITH_SNAPPY=ON ^ More? -DARROW_WITH_ZLIB=ON ^ More? -DARROW_WITH_ZSTD=ON ^ More? -DARROW_FLIGHT=ON ^ More? .. -- Building using CMake version: 3.28.3 -- The C compiler identification is Clang 17.0.6 with GNU-like command-line -- The CXX compiler identification is unknown CMake Error at C:/Users/powersj/.conda/envs/pyarrow-dev/Library/share/cmake-3.28/Modules/Platform/Windows-Clang.cmake:170 (message): The current configuration mixes Clang and MSVC or some other CL compatible compiler tool. This is not supported. Use either clang or MSVC as both C, C++ and/or HIP compilers. Call Stack (most recent call first): C:/Users/powersj/.conda/envs/pyarrow-dev/Library/share/cmake-3.28/Modules/Platform/Windows-Clang.cmake:180 (__verify_same_language_values) C:/Users/powersj/.conda/envs/pyarrow-dev/Library/share/cmake-3.28/Modules/Platform/Windows-Clang-C.cmake:1 (include) C:/Users/powersj/.conda/envs/pyarrow-dev/Library/share/cmake-3.28/Modules/CMakeCInformation.cmake:48 (include) CMakeLists.txt:95 (project) CMake Error at CMakeLists.txt:95 (project): No CMAKE_CXX_COMPILER could be found. Tell CMake where to find the compiler by setting either the environment variable "CXX" or the CMake cache entry CMAKE_CXX_COMPILER to the full path to the compiler, or to the compiler name if it is in the PATH. -- Configuring incomplete, errors occurred! (pyarrow-dev) C:\Users\powersj\arrow\cpp\build> ``` It is not clear to me what compiler I am suppose to use, either something from the conda environment or the locally installed one? If I try setting via the CC and CXX env variables I get: ``` set CC=C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\bin\Hostx64\x64\cl.exe set CXX=C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\bin\Hostx64\x64\cl.exe <snip> -- Building using CMake version: 3.28.3 -- The C compiler identification is MSVC 19.39.33519.0 -- The CXX compiler identification is MSVC 19.39.33519.0 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - failed -- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/cl.exe -- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/cl.exe - broken CMake Error at C:/Users/powersj/.conda/envs/pyarrow-dev/Library/share/cmake-3.28/Modules/CMakeTestCCompiler.cmake:67 (message): The C compiler "C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/cl.exe" is not able to compile a simple test program. It fails with the following output: Change Dir: 'C:/Users/powersj/arrow/cpp/build/CMakeFiles/CMakeScratch/TryCompile-j51cjy' Run Build Command(s): C:/Users/powersj/.conda/envs/pyarrow-dev/Library/bin/ninja.exe -v cmTC_f4d4d [1/2] C:\PROGRA~1\MICROS~2\2022\COMMUN~1\VC\Tools\MSVC\1439~1.335\bin\Hostx64\x64\cl.exe /nologo /DWIN32 /D_WINDOWS /Zi /Ob0 /Od /RTC1 -MDd /showIncludes /FoCMakeFiles\cmTC_f4d4d.dir\testCCompiler.c.obj /FdCMakeFiles\cmTC_f4d4d.dir\ /FS -c C:\Users\powersj\arrow\cpp\build\CMakeFiles\CMakeScratch\TryCompile-j51cjy\testCCompiler.c [2/2] C:\WINDOWS\system32\cmd.exe /C "cd . && C:\Users\powersj\.conda\envs\pyarrow-dev\Library\bin\cmake.exe -E vs_link_exe --intdir=CMakeFiles\cmTC_f4d4d.dir --rc=rc --mt=CMAKE_MT-NOTFOUND --manifests -- C:\PROGRA~1\MICROS~2\2022\COMMUN~1\VC\Tools\MSVC\1439~1.335\bin\Hostx64\x64\link.exe /nologo CMakeFiles\cmTC_f4d4d.dir\testCCompiler.c.obj /out:cmTC_f4d4d.exe /implib:cmTC_f4d4d.lib /pdb:cmTC_f4d4d.pdb /version:0.0 /machine:x64 /debug /INCREMENTAL /subsystem:console kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib && cd ." FAILED: cmTC_f4d4d.exe C:\WINDOWS\system32\cmd.exe /C "cd . && C:\Users\powersj\.conda\envs\pyarrow-dev\Library\bin\cmake.exe -E vs_link_exe --intdir=CMakeFiles\cmTC_f4d4d.dir --rc=rc --mt=CMAKE_MT-NOTFOUND --manifests -- C:\PROGRA~1\MICROS~2\2022\COMMUN~1\VC\Tools\MSVC\1439~1.335\bin\Hostx64\x64\link.exe /nologo CMakeFiles\cmTC_f4d4d.dir\testCCompiler.c.obj /out:cmTC_f4d4d.exe /implib:cmTC_f4d4d.lib /pdb:cmTC_f4d4d.pdb /version:0.0 /machine:x64 /debug /INCREMENTAL /subsystem:console kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib && cd ." RC Pass 1: command "rc /fo CMakeFiles\cmTC_f4d4d.dir/manifest.res CMakeFiles\cmTC_f4d4d.dir/manifest.rc" failed (exit code 0) with the following output: The system cannot find the file specified ninja: build stopped: subcommand failed. CMake will not be able to correctly generate this project. Call Stack (most recent call first): CMakeLists.txt:95 (project) -- Configuring incomplete, errors occurred! (pyarrow-dev) C:\Users\powersj\arrow\cpp\build> ``` ### Component(s) Python -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
