[
https://issues.apache.org/jira/browse/ARROW-18269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17632043#comment-17632043
]
Vibhatha Lakmal Abeykoon commented on ARROW-18269:
--------------------------------------------------
[~dytyniak] Imagine the following situation
{code:python}
import pandas as pd
import pyarrow as pa
from pyarrow import dataset as ds
path = tmpdir / "slash-writer-x"
# pathx = "/Users/vibhatha/data/parquet/slash-writer-x"
df = pd.DataFrame({
'exp_id': [1, 2, 1, 3, 6],
'exp_meta': ["experiment/A/f.csv", "experiment/B/f.csv",
"experiment/A/d.csv", "experiment/C/k.csv",
"experiment/M/i.csv"],
})
dt_table = pa.Table.from_pandas(df)
ds.write_dataset(
data=dt_table,
base_dir=path,
format='parquet',
partitioning=['exp_meta'],
partitioning_flavor='hive',
)
table = ds.dataset(
source=path,
format='parquet',
partitioning='hive',
schema = pa.schema([pa.field("exp_id", pa.int32()),
pa.field("exp_meta", pa.utf8())])
).to_table()
print(table)
df = pa.concat_tables([table]).to_pandas()
print(df.head())
{code}
If we go for option2, the users won't be able to handle such situation. But we
could suggest them to do it in a different way. But it would require them to
encode and decode the URIs. If this is billions of raws, and that would be
really expensive.
WDYT?
cc [~westonpace]
> [C++] Slash character in partition value handling
> -------------------------------------------------
>
> Key: ARROW-18269
> URL: https://issues.apache.org/jira/browse/ARROW-18269
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++, Python
> Affects Versions: 10.0.0
> Reporter: Vadym Dytyniak
> Assignee: Vibhatha Lakmal Abeykoon
> Priority: Major
> Labels: good-first-issue
>
>
> Provided example shows that pyarrow does not handle partition value that
> contains '/' correctly:
> {code:java}
> import pandas as pd
> import pyarrow as pa
> from pyarrow import dataset as ds
> df = pd.DataFrame({
> 'value': [1, 2],
> 'instrument_id': ['A/Z', 'B'],
> })
> ds.write_dataset(
> data=pa.Table.from_pandas(df),
> base_dir='data',
> format='parquet',
> partitioning=['instrument_id'],
> partitioning_flavor='hive',
> )
> table = ds.dataset(
> source='data',
> format='parquet',
> partitioning='hive',
> ).to_table()
> tables = [table]
> df = pa.concat_tables(tables).to_pandas() tables = [table]
> df = pa.concat_tables(tables).to_pandas()
> print(df.head()){code}
> Result:
> {code:java}
> value instrument_id
> 0 1 A
> 1 2 B {code}
> Expected behaviour:
> Option 1: Result should be:
> {code:java}
> value instrument_id
> 0 1 A/Z
> 1 2 B {code}
> Option 2: Error should be raised to avoid '/' in partition value.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
