[
https://issues.apache.org/jira/browse/SPARK-42027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17743485#comment-17743485
]
Hyukjin Kwon commented on SPARK-42027:
--------------------------------------
I think it will require a pretty large fix to support this correctly from my
cursory look (might be wrong). Please go ahead for a PR [~gdhuper]
> CreateDataframe from Pandas with Struct and Timestamp
> -----------------------------------------------------
>
> Key: SPARK-42027
> URL: https://issues.apache.org/jira/browse/SPARK-42027
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 3.4.0
> Reporter: Martin Grund
> Priority: Major
>
> The following should be supported and correctly truncate the nanosecond
> timestamps.
> {code:python}
> from datetime import datetime, timezone, timedelta
> from pandas import Timestamp
> ts=Timestamp(year=2019, month=1, day=1, nanosecond=500,
> tz=timezone(timedelta(hours=-8)))
> d = pd.DataFrame({"col1": [1], "col2": [{"a":1, "b":2.32, "c":ts}]})
> spark.createDataFrame(d).collect()
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
