haiyangsun-db commented on PR #58988: URL: https://github.com/apache/spark/pull/58988#issuecomment-5829676558
@viirya Thanks for the detailed review. I addressed the points in the latest revision: - Added ValueVectorUtility.validate and documented that responses must match output_schema; same-layout logical mismatches remain part of the worker contract. - Clarified finish handling: GrpcWorkerSession inspects terminal responses and surfaces finish errors while exhausting the result iterator. session.close() is cleanup-only for these UDF sessions and does not determine task success. - Documented application-level flow control as a prerequisite for enabling a production dispatcher. - Documented that Arrow compression is self-describing. VectorLoader uses Arrow’s codec factory, and an LZ4 round-trip test now covers compressed responses. - Moved named-argument rejection into logical planning, tracked by SPARK-59745. - Documented the current always-nullable result behavior and removed the redundant physical assertion. - Shared the byte-limit loop in ArrowBatchIterator, using Long while preserving the existing API. - Renamed the payload format to pyspark-udf-experimental. - Kept _workerSpec intentionally reserved for the temporary Python adapter, pending SPARK-59364. - Rejected barrier task contexts before worker-session creation and added a regression test. Thanks again for catching these. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
