nchammas commented on PR #56907:
URL: https://github.com/apache/spark/pull/56907#issuecomment-4861070058
> I think the idea of a new CLI interface is interesting, but you probably
need more than my support.
Do you think I need to create an SPIP, or shall I just open a PR and share
it on the dev list? I wasn't sure if an SPIP would be overkill so I stuck to
the dev list.
> Anyway, I believe @ericm-db is also very interested in this project and
you could work together on this!
Happy to collaborate. I think a good sketch of what a collaboration might
look like, given @ericm-db's work on this PR, is to figure out how automatic
local discovery of an existing Spark Connect server would work.
The unified CLI would be a relatively standalone piece of work. But as we
make it easier for users to manage Connect servers via the new CLI, Eric can
lead the effort to make leveraging that more seamless.
In other words:
1. I will lead the effort for the unified CLI. This will include a new
`spark connect` subcommand.
2. Eric will lead the effort to enhance local sessions to automatically
discover and use an existing local Connect server.
3. Eric and I will collaborate on the local Connect server discovery
mechanism. I think of it as the interface between `spark connect start` and
`SparkSession.builder.remote("local[*]").getOrCreate()`.
What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
