johnseekins opened a new issue, #34688:
URL: https://github.com/apache/arrow/issues/34688

   ### Describe the usage question you have. Please include as many useful 
details as  possible.
   
   
   This is a bit of a follow-on from 
https://github.com/apache/arrow/issues/34546
   
   So a rather long example:
   
   Initial table keys:
   ["key1", "key2", "key3", "partition1", "partition2"]
   
   Load to dataset with 
   ```
   partitions = ["partition1", "partition2"]
   data_set = ds.dataset(
       f"gs://bucket1/tables/parts",
       format="parquet",
       partitioning=partitions,
   )
   ```
   Then say
   ```
   >>> data_set.to_table().schema
   ["key1", "key2", "key3"]
   ```
   
   Clearly we're relying on the data layout to get the partitions. Which is 
fine and expected from how the docs describe it.
   
   Write that data out:
   ```
   table = polars.from_arrow(data_set.to_table())
   if table.height > 1:
       keys = [k for k, v in table.schema.items() if v in POLARS_UNIQUE_TYPES]
       table = table.unique(subset=keys)
   table = table.to_arrow()
   ds.write_dataset(
       table,
       f"gs://{bucket}/{config['prefix']}/",
       format="parquet",
       file_options=file_options,
       partitioning=partitions,
       schema=table.schema,
       existing_data_behavior="overwrite_or_ignore",
   )
   ```
   
   We should have a file structure like:
   ```
   bucket1
    - tables
      - parts
        - partition1_value
          - partition2_value
            part-0.parquet
   ```
   
   Load that data:
   ```
   data_set = ds.dataset(
       f"gs://bucket1/tables/parts",
       format="parquet",
       partitioning=partitions,
   )
   ```
   
   Follow the same pattern:
   ```
   table = polars.from_arrow(data_set.to_table())
   if table.height > 1:
       keys = [k for k, v in table.schema.items() if v in POLARS_UNIQUE_TYPES]
       table = table.unique(subset=keys)
   table = table.to_arrow()
   ds.write_dataset(
       table,
       f"gs://{bucket}/{config['prefix']}/",
       format="parquet",
       file_options=file_options,
       partitioning=partitions,
       schema=table.schema,
       existing_data_behavior="overwrite_or_ignore",
   )
   ```
   Now we have:
   ```
   bucket1
    - tables
      - parts
        - part-0.parquet
        - partition1_value
          - partition2_value
            part-0.parquet
   ```
   
   And we've lost the partition columns. I _assume_ this is due to the 
conversion to polars and back.
   
   So my questions are:
   1. Is there a better way to do duplicate filtering in arrows than converting 
to polars and back?
   2. If we _are_ converting back and forth, how do we preserve the partitions 
from a dataset?
   
   ### Component(s)
   
   Parquet, Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to