HyukjinKwon commented on a change in pull request #33876:
URL: https://github.com/apache/spark/pull/33876#discussion_r699814247
##########
File path: python/pyspark/sql/pandas/conversion.py
##########
@@ -336,10 +338,12 @@ def createDataFrame(self, data, schema=None,
samplingRatio=None, verifySchema=Tr
"has been set to false.\n %s" % str(e))
warnings.warn(msg)
raise
- data = self._convert_from_pandas(data, schema, timezone)
+
+ should_localize = not self._is_timestamp_ntz_preferred()
Review comment:
I think I called it here to make `_convert_from_pandas` indenependent
from `SparkSession`. `_convert_from_pandas` doesn't have any reference of
`self` ... but looks like I overthought too much ... let me call it there.
##########
File path: python/pyspark/sql/tests/test_arrow.py
##########
@@ -167,6 +167,18 @@ def test_toPandas_arrow_toggle(self):
assert_frame_equal(expected, pdf)
assert_frame_equal(expected, pdf_arrow)
+ def test_create_data_frame_to_pandas_timestamp_ntz(self):
+ # SPARK-36608: Test TimestampNTZ in createDataFrame and toPandas
Review comment:
```suggestion
# SPARK-36626: Test TimestampNTZ in createDataFrame and toPandas
```
##########
File path: python/pyspark/sql/tests/test_pandas_udf.py
##########
@@ -239,6 +240,23 @@ def udf(column):
with
self.sql_conf({"spark.sql.execution.pandas.convertToArrowArraySafely": False}):
df.withColumn('udf', udf('id')).collect()
+ def test_pandas_udf_timestamp_ntz(self):
+ # SPARK-36608: Test TimestampNTZ in pandas UDF
Review comment:
```suggestion
# SPARK-36626: Test TimestampNTZ in pandas UDF
```
##########
File path: python/pyspark/sql/tests/test_udf.py
##########
@@ -552,6 +553,23 @@ def __call__(self, x):
self.assertEqual(f, f_.func)
self.assertEqual(return_type, f_.returnType)
+ def test_udf_timestamp_ntz(self):
+ # SPARK-36608: Test TimestampNTZ in Python UDF
Review comment:
```suggestion
# SPARK-36626: Test TimestampNTZ in Python UDF
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
