Phillip Cloud created SPARK-43194:
-------------------------------------
Summary: PySpark 3.4.0 cannot convert timestamp-typed objects to
pandas with pandas 2.0
Key: SPARK-43194
URL: https://issues.apache.org/jira/browse/SPARK-43194
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 3.4.0
Environment: {code}
In [4]: import pandas as pd
In [5]: pd.__version__
Out[5]: '2.0.0'
In [6]: import pyspark as ps
In [7]: ps.__version__
Out[7]: '3.4.0'
{code}
Reporter: Phillip Cloud
{code}
In [1]: from pyspark.sql import SparkSession
In [2]: session = SparkSession.builder.appName("test").getOrCreate()
23/04/19 09:21:42 WARN Utils: Your hostname, albatross resolves to a loopback
address: 127.0.0.2; using 192.168.1.170 instead (on interface enp5s0)
23/04/19 09:21:42 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another
address
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
setLogLevel(newLevel).
23/04/19 09:21:42 WARN NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
In [3]: session.sql("select now()").toPandas()
{code}
Results in:
{code}
...
TypeError: Casting to unit-less dtype 'datetime64' is not supported. Pass e.g.
'datetime64[ns]' instead.
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
