HyukjinKwon commented on PR #59551: URL: https://github.com/apache/spark/pull/59551#issuecomment-6243756358
@sarutak Thank you, you're right: the configuration did not apply to the Arrow cache. Fixed in a37bb497eed5: - `ArrowEvalPythonExec.doExecute` now checks `supportsColumnar`, which includes the configuration, instead of `child.supportsColumnar`, so that a child that also returns rows, such as the scan of the Arrow cache, returns rows when columnar input is disabled. - A new suite, `ArrowCachedPythonUDFSuite`, runs a UDF over 200 batches of 1000 rows of the Arrow cache with the default configuration. It fails with the `IndexOutOfBoundsException` without the change to `doExecute`, and passes with it. It is a suite of its own, since the serializer of the cache is kept for the JVM. - The Scaladoc of the evaluator and of `evalArrowColumnar` no longer says that the references are safe. - The migration guide has the new default in "Upgrading from Spark SQL 4.2.0 to 4.2.1" and in "Upgrading from Spark SQL 4.3 to 4.4", where a PR against master first ships. As 4.2.1 and 4.3.0 are at their release candidates, the sections may need to follow the branches that it is backported to. - I fixed the repro in the description: 200 batches of 10000 rows, by `spark.sql.execution.arrow.maxRecordsPerBatch`. - The `ArrayDeque` race in pipelined mode: agreed, both settings are opt-in now. @viirya, could the follow-up that enables the Arrow columnar path again cover it too? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
