[
https://issues.apache.org/jira/browse/ARROW-16302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sanjiban Sengupta updated ARROW-16302:
--------------------------------------
Description:
While using FilenamePartitioning, currently when we read a dataset, say from
PyArrow, the partitioning field only has Null values.
The issue can be reproduced with the following code
{code:python}
table = pa.table([
pa.array(range(20)), pa.array(np.random.randn(20)),
pa.array(np.repeat(['a', 'b'], 10))],
names=["f1", "f2", "part"]
)
part = ds.partitioning(pa.schema([("part", pa.string())]), flavor="filename")
# test is the directory where partitions are written
ds.write_dataset(
table, "test",
format="parquet", partitioning=part
)
result = ds.dataset(
"test", format="parquet", partitioning=part,
).to_table(){code}
This results in something this:
{code:python}
pyarrow.Table
f1: int64
f2: double
part: string
----
f1: [[0,1,2,3,4,5,6,7,8,9],[10,11,12,13,14,15,16,17,18,19]]
f2:
[[-1.1753280347394899,-0.9640239222827617,0.7907247451009602,1.3667778347936321,0.005079832420686733,0.9024313772071855,-1.01618656608383,-1.1459911861999188,-0.7407261867306765,-0.012823499364722428],[0.29893685698088185,1.3907720928021299,-0.48826416913435605,-1.3436821154932153,-0.5492388164165941,-0.07093280675027104,0.009918818541272493,-1.05561750529359,-2.0209000426858927,-0.28081085330210676]]
part:
[[null,null,null,null,null,null,null,null,null,null],[null,null,null,null,null,null,null,null,null,null]]
{code}
was:
While using {{FilenamePartitioning, }}currently when we read a dataset, say
from PyArrow, the partitioning field only has Null values.
The issue can be reproduced with the following code
{code:python}
table = pa.table([
pa.array(range(20)), pa.array(np.random.randn(20)),
pa.array(np.repeat(['a', 'b'], 10))],
names=["f1", "f2", "part"]
)
part = ds.partitioning(pa.schema([("part", pa.string())]), flavor="filename")
# test is the directory where partitions are written
ds.write_dataset(
table, "test",
format="parquet", partitioning=part
)
result = ds.dataset(
"test", format="parquet", partitioning=part,
).to_table(){code}
This results in something this:
{code:python}
pyarrow.Table
f1: int64
f2: double
part: string
----
f1: [[0,1,2,3,4,5,6,7,8,9],[10,11,12,13,14,15,16,17,18,19]]
f2:
[[-1.1753280347394899,-0.9640239222827617,0.7907247451009602,1.3667778347936321,0.005079832420686733,0.9024313772071855,-1.01618656608383,-1.1459911861999188,-0.7407261867306765,-0.012823499364722428],[0.29893685698088185,1.3907720928021299,-0.48826416913435605,-1.3436821154932153,-0.5492388164165941,-0.07093280675027104,0.009918818541272493,-1.05561750529359,-2.0209000426858927,-0.28081085330210676]]
part:
[[null,null,null,null,null,null,null,null,null,null],[null,null,null,null,null,null,null,null,null,null]]
{code}
> [C++] Null values in Partitioning field for FilenamePartitioning
> ----------------------------------------------------------------
>
> Key: ARROW-16302
> URL: https://issues.apache.org/jira/browse/ARROW-16302
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
> Affects Versions: 8.0.0
> Reporter: Sanjiban Sengupta
> Assignee: Sanjiban Sengupta
> Priority: Major
> Labels: bug, partitioning
>
> While using FilenamePartitioning, currently when we read a dataset, say from
> PyArrow, the partitioning field only has Null values.
>
> The issue can be reproduced with the following code
>
> {code:python}
> table = pa.table([
> pa.array(range(20)), pa.array(np.random.randn(20)),
> pa.array(np.repeat(['a', 'b'], 10))],
> names=["f1", "f2", "part"]
> )
> part = ds.partitioning(pa.schema([("part", pa.string())]), flavor="filename")
> # test is the directory where partitions are written
> ds.write_dataset(
> table, "test",
> format="parquet", partitioning=part
> )
> result = ds.dataset(
> "test", format="parquet", partitioning=part,
> ).to_table(){code}
> This results in something this:
> {code:python}
> pyarrow.Table
> f1: int64
> f2: double
> part: string
> ----
> f1: [[0,1,2,3,4,5,6,7,8,9],[10,11,12,13,14,15,16,17,18,19]]
> f2:
> [[-1.1753280347394899,-0.9640239222827617,0.7907247451009602,1.3667778347936321,0.005079832420686733,0.9024313772071855,-1.01618656608383,-1.1459911861999188,-0.7407261867306765,-0.012823499364722428],[0.29893685698088185,1.3907720928021299,-0.48826416913435605,-1.3436821154932153,-0.5492388164165941,-0.07093280675027104,0.009918818541272493,-1.05561750529359,-2.0209000426858927,-0.28081085330210676]]
> part:
> [[null,null,null,null,null,null,null,null,null,null],[null,null,null,null,null,null,null,null,null,null]]
> {code}
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
