aimtsou commented on PR #37817: URL: https://github.com/apache/spark/pull/37817#issuecomment-1444515612
@srowen: Although this is causing an issue: If you try to build your own docker image of Spark including pyspark while trying to be compliant with Databricks you will observe that Databricks Runtime 12.1 and 12.2(which is currently in beta), both support officially until Spark 3.3.1 (while current version is 3.3.2). Actually all of the LTS versions in the [support matrix](https://docs.databricks.com/release-notes/runtime/releases.html) are not EOLed and since numpy 1.20.0 was released in 01/2021, which means that most spark compliant versions carry this bug. If you try to use Pandas by using toPandas() you end up with the numpy error, consequently being blocked from upgrading your spark versions. Is there any chance of back-porting this commit into previous pyspark versions? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
