haiyangsun-db commented on PR #58988: URL: https://github.com/apache/spark/pull/58988#issuecomment-5843912466
> One follow-up outside this diff: WorkerSession.close()'s Scaladoc still says a finish/close failure after the data is drained "is reported only here, so the caller must inspect the [[Termination]]", which is now the opposite of what withUDFWorkerSession relies on. Could we align the base contract, either by stating that the result iterator surfaces finish errors or by noting that this is a GrpcWorkerSession guarantee? A follow-up PR is fine. Thank you! I will followup on this. I think the statement here "so the caller must inspect the [[Termination]]" is not fully accurate. Overall. session.close() is to ensure all inputs and results are exhausted. In the spark exeuction here, exhausting the whole input or output is not required to complete a task successfully (e.g., due to limit), and spark needs to use session.close() to finish the contract for worker lifecycle management. I will think of how to update the documentation to reflect the special handling in spark. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
