gforsyth opened a new issue, #34683:
URL: https://github.com/apache/arrow/issues/34683

   ### Describe the bug, including details regarding any error messages, 
version, and platform.
   
   Using a recent nightly on Ubuntu 22.04.2.
   
   `pyarrow.dataset.dataset` can load a parquet directory from an `s3`-prefixed 
URI but throws an error if you pass it a list of individual parquet files that 
share the same schema.
   
   ```python
   [ins] In [1]: import pyarrow
   
   [ins] In [2]: pyarrow.__version__
   Out[2]: '12.0.0.dev266'
   
   [ins] In [3]: import pyarrow.dataset as ds
   
/home/gil/mambaforge/envs/pyarrow_nightly/lib/python3.10/site-packages/pyarrow/compute.py:206:
 RuntimeWarning: Python binding for RunEndEncodeOptions not exposed
     warnings.warn("Python binding for {} not exposed"
   
   [ins] In [4]: files = [
            ...:     
f"s3://gbif-open-data-us-east-1/occurrence/2023-02-01/occurrence.parquet/00000{i}"
            ...:     for i in range(3)
            ...: ]
   
   [ins] In [5]: ds.dataset(files)
   ---------------------------------------------------------------------------
   ArrowInvalid                              Traceback (most recent call last)
   Cell In[5], line 1
   ----> 1 ds.dataset(files)
   
   File 
~/mambaforge/envs/pyarrow_nightly/lib/python3.10/site-packages/pyarrow/dataset.py:765,
 in dataset(source, schema, format, filesystem, partitioning, 
partition_base_dir, exclude_invalid_files, ignore_prefixes)
       763 elif isinstance(source, (tuple, list)):
       764     if all(_is_path_like(elem) for elem in source):
   --> 765         return _filesystem_dataset(source, **kwargs)
       766     elif all(isinstance(elem, Dataset) for elem in source):
       767         return _union_dataset(source, **kwargs)
   
   File 
~/mambaforge/envs/pyarrow_nightly/lib/python3.10/site-packages/pyarrow/dataset.py:443,
 in _filesystem_dataset(source, schema, filesystem, partitioning, format, 
partition_base_dir, exclude_invalid_files, selector_ignore_prefixes)
       440 partitioning = _ensure_partitioning(partitioning)
       442 if isinstance(source, (list, tuple)):
   --> 443     fs, paths_or_selector = _ensure_multiple_sources(source, 
filesystem)
       444 else:
       445     fs, paths_or_selector = _ensure_single_source(source, filesystem)
   
   File 
~/mambaforge/envs/pyarrow_nightly/lib/python3.10/site-packages/pyarrow/dataset.py:351,
 in _ensure_multiple_sources(paths, filesystem)
       344 is_local = (
       345     isinstance(filesystem, (LocalFileSystem, _MockFileSystem)) or
       346     (isinstance(filesystem, SubTreeFileSystem) and
       347      isinstance(filesystem.base_fs, LocalFileSystem))
       348 )
       350 # allow normalizing irregular paths such as Windows local paths
   --> 351 paths = [filesystem.normalize_path(_stringify_path(p)) for p in 
paths]
       353 # validate that all of the paths are pointing to existing *files*
       354 # possible improvement is to group the file_infos by type and raise 
for
       355 # multiple paths per error category
       356 if is_local:
   
   File 
~/mambaforge/envs/pyarrow_nightly/lib/python3.10/site-packages/pyarrow/dataset.py:351,
 in <listcomp>(.0)
       344 is_local = (
       345     isinstance(filesystem, (LocalFileSystem, _MockFileSystem)) or
       346     (isinstance(filesystem, SubTreeFileSystem) and
       347      isinstance(filesystem.base_fs, LocalFileSystem))
       348 )
       350 # allow normalizing irregular paths such as Windows local paths
   --> 351 paths = [filesystem.normalize_path(_stringify_path(p)) for p in 
paths]
       353 # validate that all of the paths are pointing to existing *files*
       354 # possible improvement is to group the file_infos by type and raise 
for
       355 # multiple paths per error category
       356 if is_local:
   
   File 
~/mambaforge/envs/pyarrow_nightly/lib/python3.10/site-packages/pyarrow/_fs.pyx:967,
 in pyarrow._fs.FileSystem.normalize_path()
   
   File 
~/mambaforge/envs/pyarrow_nightly/lib/python3.10/site-packages/pyarrow/error.pxi:144,
 in pyarrow.lib.pyarrow_internal_check_status()
   
   File 
~/mambaforge/envs/pyarrow_nightly/lib/python3.10/site-packages/pyarrow/error.pxi:100,
 in pyarrow.lib.check_status()
   
   ArrowInvalid: Expected a local filesystem path, got a URI: 
's3://gbif-open-data-us-east-1/occurrence/2023-02-01/occurrence.parquet/000000'
   ```
   
   You can work around this with union dataset, but this seems like it should 
work?
   
   workaround:
   ```
   [ins] In [6]: ds.dataset(list(map(ds.dataset, files)))
   ```
   
   The performance of  `ds.dataset` loading files from s3 is _much_ improved 
compared with Arrow 11 -- for the files above I was seeing a pretty consistent 
~4s per-file and that's dropped down by at least 5x on this nightly!
   
   ### Component(s)
   
   Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to