[
https://issues.apache.org/jira/browse/ARROW-10145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17204845#comment-17204845
]
Joris Van den Bossche commented on ARROW-10145:
-----------------------------------------------
Note that reading it with pyarrow.dataset interface and _with_ specifying a
schema manually, it seems to work correctly:
{code}
In [38]: import pyarrow.dataset as ds
In [40]: dataset = ds.dataset("test_int64_partition/",
schema=pa.schema([("part", pa.int64()), ("col", pa.int64())]),
partitioning="hive")
In [42]: dataset.to_table()
Out[42]:
pyarrow.Table
part: int64
col: int64
In [43]: dataset.to_table().to_pandas()
Out[43]:
part col
0 3760212050 0
1 3760212050 1
2 3760212050 2
3 3760212050 3
4 3760212050 4
5 3760212050 5
6 3760212050 6
7 3760212050 7
8 3760212050 8
9 3760212050 9
{code}
At the moment, this {{schema}} keyword can not yet be passed to the
{{pq.read_table}} function (something we should add!)
> [C++][Dataset] Integer-like partition field values outside int32 range error
> on reading
> ---------------------------------------------------------------------------------------
>
> Key: ARROW-10145
> URL: https://issues.apache.org/jira/browse/ARROW-10145
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
> Reporter: Joris Van den Bossche
> Priority: Major
> Labels: dataset
>
> From
> https://stackoverflow.com/questions/64137664/how-to-override-type-inference-for-partition-columns-in-hive-partitioned-dataset
> Small reproducer:
> {code}
> import pyarrow as pa
> import pyarrow.parquet as pq
> table = pa.table({'part': [3760212050]*10, 'col': range(10)})
> pq.write_to_dataset(table, "test_int64_partition", partition_cols=['part'])
> In [35]: pq.read_table("test_int64_partition/")
> ...
> ArrowInvalid: error parsing '3760212050' as scalar of type int32
> In ../src/arrow/scalar.cc, line 333, code: VisitTypeInline(*type_, this)
> In ../src/arrow/dataset/partition.cc, line 218, code:
> (_error_or_value26).status()
> In ../src/arrow/dataset/partition.cc, line 229, code:
> (_error_or_value27).status()
> In ../src/arrow/dataset/discovery.cc, line 256, code:
> (_error_or_value17).status()
> In [36]: pq.read_table("test_int64_partition/", use_legacy_dataset=True)
> Out[36]:
> pyarrow.Table
> col: int64
> part: dictionary<values=int64, indices=int32, ordered=0>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
