HyukjinKwon commented on PR #59551: URL: https://github.com/apache/spark/pull/59551#issuecomment-6228436683
@viirya Thanks, that design makes sense to me: respecting the signal of the batch itself, as `closeIfFreeable` does, is better than copying every pass-through column, and the producers that reuse or release Arrow vectors without saying so are the other half of it. I switched this PR to 4 in 91aa31b209fb: it disables `spark.sql.execution.arrow.pythonUDF.columnarInput.enabled` by default, and the evaluator is back to what it is on master. I kept it disabled on master too, so that master and the maintenance branches are correct with the default configuration until path 1 respects the marker, and your PR can enable it again there; tell me if you'd rather leave master as it is. The tests of the columnar input path enable it, and a new test reads the source that refills its vectors with the default configuration, which fails with columnar input enabled. I updated the title and the description. Please go ahead with the marker interface; I'm happy to review it. `ReusedArrowColumnVector` in #59424 can implement the marker, whichever of the two lands first. I also re-scoped SPARK-60306 to the consumers that do not call `closeIfFreeable`, as you suggested. @sarutak The copy, the guard and the checks that you reviewed are gone with this change. Thanks for reviewing them so carefully: the list views and the pipelined mode were part of why we moved away from the copy. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
