[
https://issues.apache.org/jira/browse/ARROW-1982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wes McKinney updated ARROW-1982:
--------------------------------
Fix Version/s: 0.9.0
> [Python] Return parquet statistics min/max as values instead of strings
> -----------------------------------------------------------------------
>
> Key: ARROW-1982
> URL: https://issues.apache.org/jira/browse/ARROW-1982
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Reporter: Jim Crist
> Priority: Major
> Fix For: 0.9.0
>
>
> Currently `min` and `max` column statistics are returned as formatted strings
> of the _physical type_. This makes using them in python a bit tricky, as the
> strings need to be parsed as the proper _logical type_. Observe:
> {code}
> In [20]: import pandas as pd
> In [21]: df = pd.DataFrame({'a': [1, 2, 3],
> ...: 'b': ['a', 'b', 'c'],
> ...: 'c': [pd.Timestamp('1991-01-01')]*3})
> ...:
> In [22]: df.to_parquet('temp.parquet', engine='pyarrow')
> In [23]: from pyarrow import parquet as pq
> In [24]: f = pq.ParquetFile('temp.parquet')
> In [25]: rg = f.metadata.row_group(0)
> In [26]: rg.column(0).statistics.min # string instead of integer
> Out[26]: '1'
> In [27]: rg.column(1).statistics.min # weird space added after value due to
> formatter
> Out[27]: 'a '
> In [28]: rg.column(2).statistics.min # formatted as physical type (int)
> instead of logical (datetime)
> Out[28]: '662688000000'
> {code}
> Since the type information is known, it should be possible to convert these
> to arrow values instead of strings.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
