pralabhkumar commented on a change in pull request #33980:
URL: https://github.com/apache/spark/pull/33980#discussion_r715799109
##########
File path: python/pyspark/sql/tests/test_arrow.py
##########
@@ -118,32 +117,50 @@ def create_pandas_data_frame(self):
data_dict["4_float_t"] = np.float32(data_dict["4_float_t"])
return pd.DataFrame(data=data_dict)
- def test_toPandas_fallback_enabled(self):
+ # def test_toPandas_fallback_enabled(self):
Review comment:
done. However i have changed the expected failure type to
Map<TimeStamp,TimeStamp> instead of ArrayType<TimeStamp> . Since
ArrayType<TimeStamp> will not throw exception
##########
File path: python/pyspark/sql/pandas/types.py
##########
@@ -190,10 +191,19 @@ def _check_series_localize_timestamps(s, timezone):
"""
from pyspark.sql.pandas.utils import require_minimum_pandas_version
require_minimum_pandas_version()
-
+ from pandas import Series, DatetimeTZDtype
+ from pandas.api.types import is_datetime64_dtype
from pandas.api.types import is_datetime64tz_dtype
tz = timezone or _get_local_timezone()
# TODO: handle nested timestamps, such as ArrayType(TimestampType())?
+ # Check if its Array of TimeStamp
Review comment:
done
##########
File path: python/pyspark/sql/pandas/types.py
##########
@@ -190,10 +191,19 @@ def _check_series_localize_timestamps(s, timezone):
"""
from pyspark.sql.pandas.utils import require_minimum_pandas_version
require_minimum_pandas_version()
-
+ from pandas import Series, DatetimeTZDtype
+ from pandas.api.types import is_datetime64_dtype
from pandas.api.types import is_datetime64tz_dtype
tz = timezone or _get_local_timezone()
# TODO: handle nested timestamps, such as ArrayType(TimestampType())?
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
