Gabriel Henrique created SPARK-61937:
----------------------------------------
Summary: Fix DataFrame.select_dtypes failing on datetime columns
with pandas 3
Key: SPARK-61937
URL: https://issues.apache.org/jira/browse/SPARK-61937
Project: Spark
Issue Type: Sub-task
Components: Pandas API on Spark
Affects Versions: 4.4.0, 5.0.0
Reporter: Gabriel Henrique
With pandas 3, DataFrame.select_dtypes raises "ValueError: 'datetime64[us]' is
too specific of a frequency" for any DataFrame with a datetime column, because
pandas 3 creates datetime64[us] columns by default and the per-column loop
resolves each type from dtype.name.
Map datetime64 and timedelta64 column dtypes of any unit to their generic numpy
type, as pandas does, and keep the existing path for other dtypes.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]