Tian Gao created SPARK-54640:
--------------------------------
Summary: Replace select.select with select.poll in worker
Key: SPARK-54640
URL: https://issues.apache.org/jira/browse/SPARK-54640
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 4.2.0
Reporter: Tian Gao
We use select.select in worker.py to communicate with daemon. However,
select.select has a limit that it can't deal with fd > 1024. This might be an
issue on a heavy load system. We should use the more modern select.poll to do
this.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
