[
https://issues.apache.org/jira/browse/ARROW-7596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17023156#comment-17023156
]
Bryan Cutler commented on ARROW-7596:
-------------------------------------
Linking some discussion on the mailing list regarding pyspark behavior with
this option: [https://www.mail-archive.com/[email protected]/msg17008.html]
{noformat}
Joris Van den Bossche
Fri, 24 Jan 2020 02:11:05 -0800
Hi Bryan,
For the case that the column is no timestamp and was not modified: I don't
think it will take copies of the full dataframe by assigning columns in a
loop like that. But it is still doing work (it will copy data for that
column into the array holding those data for 2D blocks), and which can
easily be avoided I think by only assigning back when the column was
actually modified (eg by moving the is_datetime64tz_dtype inline in the
loop iterating through all columns, so you can only write back if actually
having tz-aware data).Further, even if you do the above to avoid writing back
to the dataframe
when not needed, I am not sure you should directly try to use the new
zero-copy feature of the Table.to_pandas conversion (with
split_blocks=True). It depends very much on what further happens with the
converted dataframe. Once you do some operations in pandas, those splitted
blocks will get combined (resulting in a memory copy then), and it also
means you can't modify the dataframe (if this dataframe is used in python
UDFs, it might limit what can be done in those UDFs. Just guessing here, I
don't know the pyspark code well enough).
Joris
On Thu, 23 Jan 2020 at 21:03, Bryan Cutler <[email protected]> wrote:
> Thanks for investigating this and the quick fix Joris and Wes! I just have
> a couple questions about the behavior observed here. The pyspark code
> assigns either the same series back to the pandas.DataFrame or makes some
> modifications if it is a timestamp. In the case there are no timestamps, is
> this potentially making extra copies or will it be unable to take advantage
> of new zero-copy features in pyarrow? For the case of having timestamp
> columns that need to be modified, is there a more efficient way to create a
> new dataframe with only copies of the modified series? Thanks!
>
> Bryan {noformat}
> [Python] Only apply zero-copy DataFrame block optimizations when
> split_blocks=True
> ----------------------------------------------------------------------------------
>
> Key: ARROW-7596
> URL: https://issues.apache.org/jira/browse/ARROW-7596
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Reporter: Wes McKinney
> Assignee: Wes McKinney
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 0.16.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Follow up to ARROW-3789 since there is downstream code that assumes that the
> DataFrame produced always has all mutable blocks
--
This message was sent by Atlassian Jira
(v8.3.4#803005)