[
https://issues.apache.org/jira/browse/ARROW-9476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Szucs resolved ARROW-9476.
------------------------------------
Resolution: Fixed
> [C++][Dataset] HivePartitioning discovery with dictionary types fails for
> multiple fields
> -----------------------------------------------------------------------------------------
>
> Key: ARROW-9476
> URL: https://issues.apache.org/jira/browse/ARROW-9476
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
> Affects Versions: 0.17.1
> Reporter: Joris Van den Bossche
> Assignee: Ben Kietzman
> Priority: Major
> Labels: dataset, pull-request-available
> Fix For: 1.0.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Apparently, ARROW-9288 was not fully / correctly fixing the issue. With a
> single string partition field, it now works fine. But once you have multiple
> string fields, you get parsing errors.
> A reproducible example:
> {code}
> import numpy as np
> import pyarrow as pa
> import pyarrow.parquet as pq
> import pyarrow.dataset as ds
> foo_keys = np.array(['a', 'b', 'c'], dtype=object)
> bar_keys = np.array(['d', 'e', 'f'], dtype=object)
> N = 30
> table = pa.table({
> 'foo': foo_keys.repeat(10),
> 'bar': np.tile(np.tile(bar_keys, 5), 2),
> 'values': np.random.randn(N)
> })
> base_path = "test_partition_directories3"
> pq.write_to_dataset(table, base_path, partition_cols=["bar", "foo"])
> # works
> ds.dataset(base_path, partitioning="hive")
> # fails
> part = ds.HivePartitioning.discover(max_partition_dictionary_size=-1)
> ds.dataset(base_path, partitioning=part)
> {code}
> cc [~bkietz]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
