gforsyth opened a new issue, #34683:
URL: https://github.com/apache/arrow/issues/34683
### Describe the bug, including details regarding any error messages,
version, and platform.
Using a recent nightly on Ubuntu 22.04.2.
`pyarrow.dataset.dataset` can load a parquet directory from an `s3`-prefixed
URI but throws an error if you pass it a list of individual parquet files that
share the same schema.
```python
[ins] In [1]: import pyarrow
[ins] In [2]: pyarrow.__version__
Out[2]: '12.0.0.dev266'
[ins] In [3]: import pyarrow.dataset as ds
/home/gil/mambaforge/envs/pyarrow_nightly/lib/python3.10/site-packages/pyarrow/compute.py:206:
RuntimeWarning: Python binding for RunEndEncodeOptions not exposed
warnings.warn("Python binding for {} not exposed"
[ins] In [4]: files = [
...:
f"s3://gbif-open-data-us-east-1/occurrence/2023-02-01/occurrence.parquet/00000{i}"
...: for i in range(3)
...: ]
[ins] In [5]: ds.dataset(files)
---------------------------------------------------------------------------
ArrowInvalid Traceback (most recent call last)
Cell In[5], line 1
----> 1 ds.dataset(files)
File
~/mambaforge/envs/pyarrow_nightly/lib/python3.10/site-packages/pyarrow/dataset.py:765,
in dataset(source, schema, format, filesystem, partitioning,
partition_base_dir, exclude_invalid_files, ignore_prefixes)
763 elif isinstance(source, (tuple, list)):
764 if all(_is_path_like(elem) for elem in source):
--> 765 return _filesystem_dataset(source, **kwargs)
766 elif all(isinstance(elem, Dataset) for elem in source):
767 return _union_dataset(source, **kwargs)
File
~/mambaforge/envs/pyarrow_nightly/lib/python3.10/site-packages/pyarrow/dataset.py:443,
in _filesystem_dataset(source, schema, filesystem, partitioning, format,
partition_base_dir, exclude_invalid_files, selector_ignore_prefixes)
440 partitioning = _ensure_partitioning(partitioning)
442 if isinstance(source, (list, tuple)):
--> 443 fs, paths_or_selector = _ensure_multiple_sources(source,
filesystem)
444 else:
445 fs, paths_or_selector = _ensure_single_source(source, filesystem)
File
~/mambaforge/envs/pyarrow_nightly/lib/python3.10/site-packages/pyarrow/dataset.py:351,
in _ensure_multiple_sources(paths, filesystem)
344 is_local = (
345 isinstance(filesystem, (LocalFileSystem, _MockFileSystem)) or
346 (isinstance(filesystem, SubTreeFileSystem) and
347 isinstance(filesystem.base_fs, LocalFileSystem))
348 )
350 # allow normalizing irregular paths such as Windows local paths
--> 351 paths = [filesystem.normalize_path(_stringify_path(p)) for p in
paths]
353 # validate that all of the paths are pointing to existing *files*
354 # possible improvement is to group the file_infos by type and raise
for
355 # multiple paths per error category
356 if is_local:
File
~/mambaforge/envs/pyarrow_nightly/lib/python3.10/site-packages/pyarrow/dataset.py:351,
in <listcomp>(.0)
344 is_local = (
345 isinstance(filesystem, (LocalFileSystem, _MockFileSystem)) or
346 (isinstance(filesystem, SubTreeFileSystem) and
347 isinstance(filesystem.base_fs, LocalFileSystem))
348 )
350 # allow normalizing irregular paths such as Windows local paths
--> 351 paths = [filesystem.normalize_path(_stringify_path(p)) for p in
paths]
353 # validate that all of the paths are pointing to existing *files*
354 # possible improvement is to group the file_infos by type and raise
for
355 # multiple paths per error category
356 if is_local:
File
~/mambaforge/envs/pyarrow_nightly/lib/python3.10/site-packages/pyarrow/_fs.pyx:967,
in pyarrow._fs.FileSystem.normalize_path()
File
~/mambaforge/envs/pyarrow_nightly/lib/python3.10/site-packages/pyarrow/error.pxi:144,
in pyarrow.lib.pyarrow_internal_check_status()
File
~/mambaforge/envs/pyarrow_nightly/lib/python3.10/site-packages/pyarrow/error.pxi:100,
in pyarrow.lib.check_status()
ArrowInvalid: Expected a local filesystem path, got a URI:
's3://gbif-open-data-us-east-1/occurrence/2023-02-01/occurrence.parquet/000000'
```
You can work around this with union dataset, but this seems like it should
work?
workaround:
```
[ins] In [6]: ds.dataset(list(map(ds.dataset, files)))
```
The performance of `ds.dataset` loading files from s3 is _much_ improved
compared with Arrow 11 -- for the files above I was seeing a pretty consistent
~4s per-file and that's dropped down by at least 5x on this nightly!
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]