BryanCutler commented on a change in pull request #27358:
[SPARK-30640][PYTHON][SQL] Prevent unnecessary copies of data during Arrow to
Pandas conversion
URL: https://github.com/apache/spark/pull/27358#discussion_r370884090
##########
File path: python/pyspark/sql/pandas/types.py
##########
@@ -165,22 +165,6 @@ def _check_series_localize_timestamps(s, timezone):
return s
-def _check_dataframe_localize_timestamps(pdf, timezone):
- """
- Convert timezone aware timestamps to timezone-naive in the specified
timezone or local timezone
-
- :param pdf: pandas.DataFrame
- :param timezone: the timezone to convert. if None then use local timezone
- :return pandas.DataFrame where any timezone aware columns have been
converted to tz-naive
- """
- from pyspark.sql.pandas.utils import require_minimum_pandas_version
- require_minimum_pandas_version()
-
- for column, series in pdf.iteritems():
- pdf[column] = _check_series_localize_timestamps(series, timezone)
Review comment:
The problem is pyarrow stores the DataFrame data in blocks internally, and
assigning series back to the DataFrame would cause the blocks to be reallocated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
