swyatt7 opened a new issue, #37667:
URL: https://github.com/apache/arrow/issues/37667

   ### Describe the bug, including details regarding any error messages, 
version, and platform.
   
   Hi all, I am trying to read a hive partitioned parquet dataset from cloud 
storage and am running into some issues.
   
   The dataset is partitioned with the structure:
   ```
   /path/to/dataset/ ->
      metadata_file.json
      metadata_file.csv
      metadata_file.fits
      partition_var1=X/ -> 
         partition_var2=Y/ ->
            partition_var3=Z.parquet
   ```
   
   Locally, loading this with `pyarrow.dataset.dataset(...)` gives zero issues. 
Once I uploaded it to our `abfs` storage container, and attempted to read it 
with that filesystem, I began receiving the error: 
   ```
   pyarrow.lib.ArrowInvalid: GetFileInfo() yielded path 
'/path/to/dataset/metadata_file.json', which is outside base dir 
'/path/to/dataset/'
   ```
   
   I've even tried loading it without the metadata files in the base dir, but 
then it descends into the partitions and begins complaining about the parquet 
files themselves being outside the base dir. 
   
   Here is the code I am using to read in the files:
   
   ```python
   import os
   import fsspec
   import pyarrow.dataset as pds
   
   storage_options = {
         "account_key" : os.environ.get("ABFS_LINCCDATA_ACCOUNT_KEY"),
         "account_name" : os.environ.get("ABFS_LINCCDATA_ACCOUNT_NAME")
     }
   
   file_system = fsspec.filesystem("abfs", **storage_options)
   dataset_dir = "/path/to/dataset"
   ignore_prefixes = [
       "metadata_file"
   ]
   
   dataset = pds.dataset(
       dataset_dir,
       filesystem=file_system,
       partitioning="hive",
       exclude_invalid_files=True,
       format="parquet",
       ignore_prefixes=ignore_prefixes,
   )
   ```
   
   Any help would be appreciated. 
   
   pyarrow version: 12.0.1
   
   ### Component(s)
   
   Parquet, Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to