Mohit-25-tech opened a new pull request, #53034: URL: https://github.com/apache/arrow/pull/53034
## What changes are included in this PR? Fixes #53033 This PR fixes a bug in `pyarrow.dataset.dataset()` where mixing `FileInfo` objects and path-like sources produces inconsistent behavior depending on their order. Previously: - `[path, path]` — works. - `[FileInfo, FileInfo]` — works. - `[path, FileInfo]` — raises `TypeError`. - `[FileInfo, path]` — causes a native segmentation fault. The issue occurs because the dataset construction logic determines how to process the entire source list based on its first element, even when subsequent elements have different types. ### Root cause The public `dataset()` API accepts lists containing path-like objects or `FileInfo` objects. However, `_filesystem_dataset()` previously dispatched based on the first element alone. When the first element was `FileInfo`, the mixed list could reach `FileSystemDatasetFactory`, where the Cython `unwrap_finfos()` implementation performed an unchecked cast of subsequent elements to `FileInfo`. This could terminate the Python interpreter instead of raising a normal exception. ### Changes **1. Support mixed sources in the Python wrapper** - Normalize path-like entries in mixed lists into `FileInfo` objects using the selected filesystem. - Preserve existing `FileInfo` objects and their metadata. - Preserve source ordering and duplicate file entries. - Keep homogeneous path and `FileInfo` lists on their existing processing paths. **2. Add defensive Cython validation** - Validate element types before casting them to `FileInfo`. - Raise a descriptive `TypeError` for unsupported mixed inputs passed directly to the native factory. - Prevent an unchecked cast from causing a process crash. **3. Add regression tests** Add coverage for: - All four path/`FileInfo` combinations. - List and tuple inputs. - Duplicate sources and source ordering. - Subtree filesystem handling. - Preservation of supplied `FileInfo` objects. - Missing files and directory paths. - Direct `FileSystemDatasetFactory` invalid-input handling. ### Behavior after the fix For valid files, all four input combinations are supported: ```python ds.dataset([path, path], format="parquet") ds.dataset([info, info], format="parquet") ds.dataset([path, info], format="parquet") ds.dataset([info, path], format="parquet") ``` Each should construct a dataset successfully without losing source ordering or duplicate entries. Invalid or missing paths continue to raise appropriate Python exceptions rather than being silently ignored. ### Testing **Independent reproduction** - Reproduced the original bug on Kaggle/Linux using PyArrow 23.0.1 and 26.0.0. - Confirmed that `[FileInfo, path]` terminates the original process with `SIGSEGV` (exit code `-11`). - Previously reproduced the same issue locally on Windows using PyArrow 16.1.0 and 24.0.0. **Patched implementation verification** - Loaded the actual modified `dataset.py` wrapper with the installed PyArrow 26.0.0 native runtime in Kaggle. - All four input combinations passed and returned the expected data. - 25 focused local checks passed against the modified Python wrapper using the installed native PyArrow runtime. - Python syntax checks passed. - Flake8 passed. - autopep8 diff checks passed. - `git diff --check` passed. **Testing limitations** - The modified Cython extension has not been compiled or executed locally. - Full checkout pytest and source-build verification remain pending. - Remote filesystem behavior has not been independently verified against an actual remote service. The combined Python and Cython changes require repository CI validation. ### Compatibility and performance considerations The change extends the public dataset API to support mixed path and `FileInfo` lists. Existing homogeneous-source behavior is preserved. For mixed lists, resolving path entries to `FileInfo` objects can introduce an additional filesystem metadata lookup, including on remote filesystems. ### AI assistance OpenAI Codex assisted with investigating the issue, implementing the changes, creating regression tests, and reviewing the patch. I independently reproduced the original crash, tested the modified Python implementation using Kaggle, and reviewed the proposed behavior and verification results. ### Related issue Fixes #53033 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
