Kimahriman commented on PR #49107:
URL: https://github.com/apache/spark/pull/49107#issuecomment-2638350996
> > If so, how would this work on YARN? It looks like it will only use an
ephemeral port when running tests, I think YARN would also need to use an
ephemeral port?
>
> Yes, it will launch a Spark Connect server locally.
>
But if it uses the default 15002 port, it will hit issues when multiple
drivers are running on the same node, unless it uses an ephemeral port.
> This is the same as Py4J server. It does not enable the auth feature by
default. We could have the implementation in Spark Connect to but I would
prefer to run this separately.
Where is auth not enabled for Py4J? It looks like it always generates a
secret that the client uses to connect to it
https://github.com/apache/spark/blob/e2ef5a482604059197c79f1f7c305df4ce57213a/core/src/main/scala/org/apache/spark/api/python/Py4JServer.scala#L32
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
