swyatt7 opened a new issue, #34403:
URL: https://github.com/apache/arrow/issues/34403

   ### Describe the usage question you have. Please include as many useful 
details as  possible.
   
   
   Hello,
   
   I am working with an already partitioned dataset that exists in the hive 
format as such:
   
   ```
   dataset/
      partition_col1=XX/
         partition_col2=YY/
            file.parquet
   ```
   
   We have already designed methods to construct this partitioned format, and 
are wondering if it is possible to create the `_metadata` and 
`_common_metadata` files _post-facto_ with pyarrow. It seems the only way to 
create these `_metadata` files so that it retains the partitioning schema is 
through the `pyarrow.parquet.write_to_dataset()` and passing in 
`metadata_collector` kwargg, and populating it that way. 
   
   Since our datasets are already partitioned in the manner we desire, is there 
a way to ONLY create the `metadata_collector` object to write the files WITHOUT 
using the `write_to_dataset` function? These datasets are large, and on 
abstract file systems (abfls, s3... etc) it takes around 4-5 minutes to load 
our datasets (around 4000 partitions) lazily with 
`dask.dataframe.read_parquet(abfls://dataset/partition_col1=*/partition_col2=*/file.parquet)`
   
   Any help would be appreciated :) 
   
   ### Component(s)
   
   Parquet, Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to