bveeramani opened a new issue, #36278:
URL: https://github.com/apache/arrow/issues/36278
### Describe the enhancement requested
Add a `exclude_invalid_files` parameter to `ParquetDatasource`.
I want to read Parquet files from a bucket that contains a JSON metadata
file. Because `ParquetDatasource` doesn't expose `exclude_invalid_files`, I get
an error:
```
File
"/Users/balaji/Documents/GitHub/ray/python/ray/data/datasource/parquet_datasource.py",
line 204, in __init__
pq_ds = pq.ParquetDataset(
^^^^^^^^^^^^^^^^^^
File
"/Users/balaji/Documents/GitHub/ray/.venv/lib/python3.11/site-packages/pyarrow/parquet/core.py",
line 1776, in __new__
return _ParquetDatasetV2(
^^^^^^^^^^^^^^^^^^
File
"/Users/balaji/Documents/GitHub/ray/.venv/lib/python3.11/site-packages/pyarrow/parquet/core.py",
line 2490, in __init__
self._dataset = ds.dataset(path_or_paths, filesystem=filesystem,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/Users/balaji/Documents/GitHub/ray/.venv/lib/python3.11/site-packages/pyarrow/dataset.py",
line 763, in dataset
return _filesystem_dataset(source, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/Users/balaji/Documents/GitHub/ray/.venv/lib/python3.11/site-packages/pyarrow/dataset.py",
line 456, in _filesystem_dataset
return factory.finish(schema)
^^^^^^^^^^^^^^^^^^^^^^
File "pyarrow/_dataset.pyx", line 2752, in
pyarrow._dataset.DatasetFactory.finish
File "pyarrow/error.pxi", line 144, in
pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Error creating dataset. Could not read schema from
'ray-example-data/iris.json'. Is this a 'parquet' file?: Could not open Parquet
input source 'ray-example-data/iris.json': Parquet magic bytes not found in
footer. Either the file is corrupted or this is not a parquet file.
```
This issue is motivated by https://github.com/ray-project/ray/issues/36753.
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
