viirya commented on PR #58978: URL: https://github.com/apache/spark/pull/58978#issuecomment-5808926315
Thanks for the follow-up review. The previous revision left gaps in physical-plan integration and Arrow validation, and I missed the configured mypy check. I've pushed another revision in ab01d81b15c. The in-process implementation now uses an `EvalPythonEvaluatorFactory` selected by `ArrowEvalPythonExec`, sharing argument projection, row buffering, result joining, and partition-evaluator execution. I removed the separate physical operator and strategy branch, and added named-argument handling, metrics, and JVM-side result checks. The revision also handles map entries offsets, normalizes declared field names/metadata while preserving null-parent semantics, and makes interpreter shutdown and cleanup explicit. The API now defers serialization until first use, requires matching value types, rejects zero-argument scalar functions, and rejects unsupported SQL registration early. Site-package initialization processes `.pth` files, and closure transfer uses a direct ByteBuffer instead of per-byte JNI iteration. I corrected the remaining concurrency and MapType documentation contradictions too. Local validation passed: the Hive-enabled build, 38 Scala tests, 220 Python tests, the full `dev/lint-python --mypy` check, Ruff, and Scalastyle. The Python total includes 50 in-process integration tests, 22 runtime tests, and the existing Python/scalar Arrow UDF suites. Ordinary SQL and worker UDFs also passed a smoke check without JEP/CDI jars. I have not rerun the performance benchmarks for this revision. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
