johnseekins opened a new issue, #34688: URL: https://github.com/apache/arrow/issues/34688
### Describe the usage question you have. Please include as many useful details as possible. This is a bit of a follow-on from https://github.com/apache/arrow/issues/34546 So a rather long example: Initial table keys: ["key1", "key2", "key3", "partition1", "partition2"] Load to dataset with ``` partitions = ["partition1", "partition2"] data_set = ds.dataset( f"gs://bucket1/tables/parts", format="parquet", partitioning=partitions, ) ``` Then say ``` >>> data_set.to_table().schema ["key1", "key2", "key3"] ``` Clearly we're relying on the data layout to get the partitions. Which is fine and expected from how the docs describe it. Write that data out: ``` table = polars.from_arrow(data_set.to_table()) if table.height > 1: keys = [k for k, v in table.schema.items() if v in POLARS_UNIQUE_TYPES] table = table.unique(subset=keys) table = table.to_arrow() ds.write_dataset( table, f"gs://{bucket}/{config['prefix']}/", format="parquet", file_options=file_options, partitioning=partitions, schema=table.schema, existing_data_behavior="overwrite_or_ignore", ) ``` We should have a file structure like: ``` bucket1 - tables - parts - partition1_value - partition2_value part-0.parquet ``` Load that data: ``` data_set = ds.dataset( f"gs://bucket1/tables/parts", format="parquet", partitioning=partitions, ) ``` Follow the same pattern: ``` table = polars.from_arrow(data_set.to_table()) if table.height > 1: keys = [k for k, v in table.schema.items() if v in POLARS_UNIQUE_TYPES] table = table.unique(subset=keys) table = table.to_arrow() ds.write_dataset( table, f"gs://{bucket}/{config['prefix']}/", format="parquet", file_options=file_options, partitioning=partitions, schema=table.schema, existing_data_behavior="overwrite_or_ignore", ) ``` Now we have: ``` bucket1 - tables - parts - part-0.parquet - partition1_value - partition2_value part-0.parquet ``` And we've lost the partition columns. I _assume_ this is due to the conversion to polars and back. So my questions are: 1. Is there a better way to do duplicate filtering in arrows than converting to polars and back? 2. If we _are_ converting back and forth, how do we preserve the partitions from a dataset? ### Component(s) Parquet, Python -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
