Adam Binford created SPARK-51223:
------------------------------------
Summary: Spark Connect should use ephemeral port for local mode
Key: SPARK-51223
URL: https://issues.apache.org/jira/browse/SPARK-51223
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: 3.5.4, 4.0.0
Reporter: Adam Binford
Currently Spark Connect will always use the port 15002 by default when starting
a connect server, no matter how the connect server was started. However, for
locally started servers, either through --remote local, or the new
spark.api.mode=connect, a connect server is started purely for the local job
that is kicked off. If you use --remote local, you can't start more than one
job without having port conflicts.
With the connect API mode, this gets worse, as you could kick off two jobs in
cluster mode in Yarn, and end up with port conflicts on whatever node ends up
running your driver.
Locally launched Spark Connect servers like this should just always use
ephemeral ports to avoid these port conflicts.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
