Gabriel Henrique created SPARK-61937:
----------------------------------------

             Summary: Fix DataFrame.select_dtypes failing on datetime columns 
with pandas 3
                 Key: SPARK-61937
                 URL: https://issues.apache.org/jira/browse/SPARK-61937
             Project: Spark
          Issue Type: Sub-task
          Components: Pandas API on Spark
    Affects Versions: 4.4.0, 5.0.0
            Reporter: Gabriel Henrique


With pandas 3, DataFrame.select_dtypes raises "ValueError: 'datetime64[us]' is 
too specific of a frequency" for any DataFrame with a datetime column, because 
pandas 3 creates datetime64[us] columns by default and the per-column loop 
resolves each type from dtype.name.

Map datetime64 and timedelta64 column dtypes of any unit to their generic numpy 
type, as pandas does, and keep the existing path for other dtypes.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to