Hi devs, I'd like to start a discussion on adding capability discovery and negotiation to the Spark Connect protocol.
Spark Connect currently has no way for a client to discover which operators, functions, data types, or features a server supports. Multiple server implementations now implement the protocol, but clients discover unsupported features only through runtime failures. This can also happen when the client and server are on different versions (e.g. client running 4.1.x, server running 3.5.x). This proposal adds a GetServerCapabilities RPC that lets Spark Connect clients query the server's supported protocol surface at connection time, following the pattern of JDBC's DatabaseMetaData and Flight SQL's GetSqlInfo. JIRA: https://issues.apache.org/jira/browse/SPARK-56398 SPIP Doc: https://docs.google.com/document/d/1lYPCJqh9ftOsDmM617SqtVpuSPaf5fNSD8uz1FHl_5A/edit?usp=sharing Thanks in advance for your comments and feedback. Devin
