swyatt7 opened a new issue, #34403:
URL: https://github.com/apache/arrow/issues/34403
### Describe the usage question you have. Please include as many useful
details as possible.
Hello,
I am working with an already partitioned dataset that exists in the hive
format as such:
```
dataset/
partition_col1=XX/
partition_col2=YY/
file.parquet
```
We have already designed methods to construct this partitioned format, and
are wondering if it is possible to create the `_metadata` and
`_common_metadata` files _post-facto_ with pyarrow. It seems the only way to
create these `_metadata` files so that it retains the partitioning schema is
through the `pyarrow.parquet.write_to_dataset()` and passing in
`metadata_collector` kwargg, and populating it that way.
Since our datasets are already partitioned in the manner we desire, is there
a way to ONLY create the `metadata_collector` object to write the files WITHOUT
using the `write_to_dataset` function? These datasets are large, and on
abstract file systems (abfls, s3... etc) it takes around 4-5 minutes to load
our datasets (around 4000 partitions) lazily with
`dask.dataframe.read_parquet(abfls://dataset/partition_col1=*/partition_col2=*/file.parquet)`
Any help would be appreciated :)
### Component(s)
Parquet, Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]