[ https://issues.apache.org/jira/browse/ARROW-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16175077#comment-16175077 ]
Wes McKinney commented on ARROW-1593: ------------------------------------- That seems reasonable. We could also address the range index by augmenting the metadata in http://pandas-docs.github.io/pandas-docs-travis/developer.html I have spent significantly less time optimizing {{RecordBatch.from_pandas}} than {{RecordBatch/Table.to_pandas}}, so there's likely some perf improvements that could be made on the ingest to Arrow path (e.g. parallel column conversions) > [PYTHON] serialize_pandas should pass through the preserve_index keyword > ------------------------------------------------------------------------ > > Key: ARROW-1593 > URL: https://issues.apache.org/jira/browse/ARROW-1593 > Project: Apache Arrow > Issue Type: Improvement > Components: Python > Affects Versions: 0.7.0 > Reporter: Tom Augspurger > Assignee: Tom Augspurger > Priority: Minor > Fix For: 0.8.0 > > > I'm doing some benchmarking of Arrow serialization for dask.distributed to > serialize dataframes. > Overall things look good compared to the current implementation (using > pickle). The biggest difference was pickle's ability to use pandas' > RangeIndex to avoid serializing the entire Index of values when possible. > I suspect that a "range type" isn't in scope for arrow, but in the meantime > applications using Arrow could detect the `RangeIndex`, and pass {{ > pyarrow.serialize_pandas(df, preserve_index=False) }} -- This message was sent by Atlassian JIRA (v6.4.14#64029)