nicolasazrak opened a new pull request #34509: URL: https://github.com/apache/spark/pull/34509
### What changes were proposed in this pull request? This change fixes `SPARK-34521`. It allows creating a spark DataFrame from a pandas DataFrame that is using a `StringDtype` column. ### Why are the changes needed? Pandas stores string columns in two different ways: using a numpy `ndarray` or using a custom `StringArray`. The `StringArray` version is used when specifing the `dtype=string`. When that happens, spark cannot serialize the column to arrow. Converting the `Series` before fixes this problem. However, due to the different ways to handle string columns, doing `spark.createDataFrame(pandas_dataframe).toPandas()` might not equal to `pandas_dataframe`. The column dtype could be different. More info: https://pandas.pydata.org/docs/user_guide/text.html ### Does this PR introduce _any_ user-facing change? No, it only fixes a use case that broken. ### How was this patch tested? Using the `test_createDataFrame_with_string_dtype` test. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
