Davies Liu created SPARK-3842:
---------------------------------
Summary: Remove the hacks for Python callback server in py4j
Key: SPARK-3842
URL: https://issues.apache.org/jira/browse/SPARK-3842
Project: Spark
Issue Type: Improvement
Components: PySpark, Streaming
Reporter: Davies Liu
Priority: Minor
There are three hacks while create Python API for Streaming
(https://github.com/apache/spark/pull/2538) :
1. daemonize the callback server thread, by 'thread.daemon = True' before start
it https://github.com/bartdag/py4j/issues/147
2. let callback server bind to random port, then update the Java callback
client with real port. https://github.com/bartdag/py4j/issues/148
3. start the callback server later. https://github.com/bartdag/py4j/issues/149
These hacks should be removed after py4j has fix these issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
