Dhruv Pratap created SPARK-58227:
------------------------------------
Summary: Support Spark Connect in the spark-sql CLI (spark-sql
--remote)
Key: SPARK-58227
URL: https://issues.apache.org/jira/browse/SPARK-58227
Project: Spark
Issue Type: New Feature
Components: Connect
Affects Versions: 5.0.0
Reporter: Dhruv Pratap
h2. Problem
The spark-sql CLI (bin/spark-sql -> SparkSQLCLIDriver, in
sql/hive-thriftserver) runs only against a local, in-process SparkSession.
Unlike spark-shell (SPARK-48936) and spark-submit (SPARK-48960), it cannot
target a Spark Connect server; there is no "spark-sql --remote sc://...". It is
the last interactive entry point without Connect support and the only one still
requiring a local Catalyst driver and the Hive Thrift Server module.
h2. Approach
Add a new SparkConnectSQLCLIDriver in sql/connect/client/jvm, selected when
--remote / spark.api.mode=connect is set (the same dispatch spark-shell uses),
mirroring ConnectRepl (SPARK-48936). It builds a remote SparkSession via the
Connect JVM client, executes statements with spark.sql(...), and renders
tab-separated output client-side. The existing SparkSQLCLIDriver (Catalyst /
hive-thriftserver-coupled) is unchanged and remains the default; no existing
public API changes and no dependency on sql/hive-thriftserver.
h2. Scope (MVP first)
MVP: remote session + -e / -f + an interactive loop + basic tabular output.
Follow-ups: full hiveResultString output parity (types, nulls, complex/nested,
timestamps), the remaining CLI flags, and init files.
Relates to SPARK-49194.
Design:
https://docs.google.com/document/d/14iOZAZFt5BN9whbbjCzZX1mVuhaKq2LEkfP6Ts4prHo/edit
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
