Mohit-25-tech opened a new issue, #53033:
URL: https://github.com/apache/arrow/issues/53033
### Describe the bug, including details regarding any error messages,
version, and platform.
### Describe the bug
`pyarrow.dataset.dataset()` behaves inconsistently when its source list
contains both filesystem paths and `pyarrow.fs.FileInfo` objects.
Both homogeneous input types work correctly. However, mixed inputs behave
differently depending on their order:
- `[path, FileInfo]` raises a normal `TypeError`.
- `[FileInfo, path]` terminates the Python process.
The crash occurs instead of a safe Python exception.
### Reproduction
```python
import pathlib
import subprocess
import sys
import tempfile
import textwrap
import pyarrow as pa
import pyarrow.parquet as pq
child_code = textwrap.dedent("""
import sys
import pyarrow.dataset as ds
import pyarrow.fs as fs
path, case = sys.argv[1:3]
info = fs.LocalFileSystem().get_file_info(path)
sources = {
"paths_only": [path, path],
"infos_only": [info, info],
"path_first": [path, info],
"info_first": [info, path],
}[case]
print("Starting:", case, flush=True)
try:
result = ds.dataset(
sources, format="parquet"
).to_table().to_pydict()
print("Result:", result, flush=True)
except Exception as exc:
print("Exception:", type(exc).__name__, str(exc), flush=True)
""")
print("Python:", sys.version.split()[0])
print("PyArrow:", pa.__version__)
print("Platform:", sys.platform)
with tempfile.TemporaryDirectory() as directory:
path = pathlib.Path(directory) / "tiny.parquet"
pq.write_table(pa.table({"x": [1]}), path)
for case in ("paths_only", "infos_only", "path_first", "info_first"):
run = subprocess.run(
[sys.executable, "-X", "faulthandler", "-c",
child_code, str(path), case],
capture_output=True,
text=True,
timeout=30,
)
print(case, "returncode:", run.returncode)
print("stdout:", run.stdout.strip())
print("stderr:", run.stderr.strip())
```
### Actual behavior
Observed with PyArrow 23.0.1 on Kaggle/Linux:
```text
paths_only: returncode 0
infos_only: returncode 0
path_first: TypeError: not a path-like object
info_first: returncode -11
Fatal Python error: Segmentation fault
```
The same four-case assessment was repeated with PyArrow 26.0.0, where both
homogeneous cases succeeded, path-first raised `TypeError`, and FileInfo-first
terminated the subprocess.
The behavior was also reproduced locally on Windows with PyArrow 16.1.0 and
24.0.0.
### Expected behavior
Mixing file paths and `FileInfo` objects should either:
1. Be supported consistently in both orders; or
2. Raise a clear Python exception in both orders.
Invalid or unsupported source combinations should not terminate the
interpreter.
### Possible root cause
In `python/pyarrow/dataset.py`, the public `dataset()` function accepts
lists containing either paths or `FileInfo` objects, but the downstream
`_filesystem_dataset()` logic dispatches based on the first element.
When the first element is `FileInfo`, the native `FileSystemDatasetFactory`
receives a mixed list. Its Cython path assumes the remaining entries are also
`FileInfo` objects, which appears to cause the native crash.
A possible minimal fix is to reject mixed lists safely before entering
native code. Alternatively, mixed inputs could be normalized if supporting them
is the intended API contract.
I'd appreciate maintainer guidance on the preferred behavior.
### Environment
- Kaggle/Linux, Python 3.13.15, PyArrow 23.0.1
- Kaggle/Linux, PyArrow 26.0.0
- Windows, PyArrow 16.1.0 and 24.0.0
The investigation and initial report drafting were AI-assisted. I
independently ran and reviewed the Kaggle reproductions.
### Component(s)
Python, Parquet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
