swyatt7 opened a new issue, #37667:
URL: https://github.com/apache/arrow/issues/37667
### Describe the bug, including details regarding any error messages,
version, and platform.
Hi all, I am trying to read a hive partitioned parquet dataset from cloud
storage and am running into some issues.
The dataset is partitioned with the structure:
```
/path/to/dataset/ ->
metadata_file.json
metadata_file.csv
metadata_file.fits
partition_var1=X/ ->
partition_var2=Y/ ->
partition_var3=Z.parquet
```
Locally, loading this with `pyarrow.dataset.dataset(...)` gives zero issues.
Once I uploaded it to our `abfs` storage container, and attempted to read it
with that filesystem, I began receiving the error:
```
pyarrow.lib.ArrowInvalid: GetFileInfo() yielded path
'/path/to/dataset/metadata_file.json', which is outside base dir
'/path/to/dataset/'
```
I've even tried loading it without the metadata files in the base dir, but
then it descends into the partitions and begins complaining about the parquet
files themselves being outside the base dir.
Here is the code I am using to read in the files:
```python
import os
import fsspec
import pyarrow.dataset as pds
storage_options = {
"account_key" : os.environ.get("ABFS_LINCCDATA_ACCOUNT_KEY"),
"account_name" : os.environ.get("ABFS_LINCCDATA_ACCOUNT_NAME")
}
file_system = fsspec.filesystem("abfs", **storage_options)
dataset_dir = "/path/to/dataset"
ignore_prefixes = [
"metadata_file"
]
dataset = pds.dataset(
dataset_dir,
filesystem=file_system,
partitioning="hive",
exclude_invalid_files=True,
format="parquet",
ignore_prefixes=ignore_prefixes,
)
```
Any help would be appreciated.
pyarrow version: 12.0.1
### Component(s)
Parquet, Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]