Hi all, The Spark Connect entry-point umbrella (SPARK-49194) has landed Connect support for spark-shell (SPARK-48936), spark-submit (SPARK-48960), and pyspark. The one interactive entry point still missing it is the spark-sql CLI: there is currently no way to run spark-sql against a Connect server (spark-sql --remote sc://...).
Before writing any code, two questions for the Connect maintainers: 1. Tracking: should this be a sub-task under SPARK-49194, or a new ticket? The umbrella lists pyspark/spark-shell/spark-submit and its sub-tasks are all resolved, so I want to confirm the right home. 2. Process: does this warrant a SPIP, or is it an incremental improvement completing the already-accepted Connect pattern? It is user-facing but adds no new public API (details below). Proposed approach (kept minimal): - A new SparkConnectSQLCLIDriver in sql/connect/client/jvm, selected on --remote / spark.api.mode=connect, mirroring how ConnectRepl backs spark-shell --remote (SPARK-48936). - Builds a remote session via SparkSession.builder().client(...), runs statements via spark.sql(...), and formats results client-side. - No change to the existing SparkSQLCLIDriver / sql/hive-thriftserver path; the legacy driver stays the default. No existing public API changes. - MVP first (-e/-f + interactive loop + basic output); follow-ups for full Hive-style output parity. I have a short design write-up I can share/attach to the JIRA. This is complementary to the draft "Add Unified Spark CLI" PR (#57313, cc @nchammas): that unifies the bin/ scripts and its sql subcommand currently wraps the existing spark-sql shell, so a Connect-native spark-sql would slot under it cleanly. Happy to coordinate to avoid duplicate CLI churn. Thanks, Dhruv Pratap (Netflix) Apache JIRA: dhruvpratap [email protected] ยท [email protected]
