BryanCutler commented on issue #24834: [WIP][SPARK-27992][PYTHON] Synchronize with Python connection thread to propagate errors URL: https://github.com/apache/spark/pull/24834#issuecomment-500618047 I think might be a better way to propagate exceptions from the Python connection serving thread for the cases of `toPandas()` with Arrow and `toLocalIterator()`. This way, any exception in the serving thread will be raised in Python when making the call `getResult()` in Python, which will join the thread and evaluate the thread future and raise any exception that occurred. Here I duplicated the `serveToStream()` code path so that if we want to backport this fix to branch-2.4, it will be a lot easier. If we don't want to backport, I can clean this up quite a bit. I still think it is probably better to not backport due to the risk, but I'll leave this as is if we want to discuss the possibility.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
