roland created SPARK-43585:
------------------------------
Summary: Spark Connect client cannot read from Hive metastore
Key: SPARK-43585
URL: https://issues.apache.org/jira/browse/SPARK-43585
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 3.4.0
Reporter: roland
I created a Spark Connect shell in a pod using the following yaml.
{code:java}
apiVersion: v1
kind: Service
metadata:
name: spark-connect-svc
namespace: MY_NAMESPACE
spec:
clusterIP: None
selector:
app: spark-connect-pod
podType: spark-connect-driver---------------------
apiVersion: v1
kind: Pod
metadata:
name: spark-connect-pod
namespace: realtime-streaming
labels:
app: spark-connect-pod
podType: spark-connect-driver
spec:
restartPolicy: Never
containers:
- command:
- sh
- -c
- /opt/spark/sbin/start-connect-server.sh --master
k8s://https://MY_API_SERVER:443 --packages
org.apache.spark:spark-connect_2.12:3.4.0 --conf
spark.kubernetes.executor.limit.cores=1.0 --conf
spark.kubernetes.executor.request.cores=1.0 --conf spark.executor.cores=1
--conf spark.executor.memory=6G --conf
spark.kubernetes.container.image=MY_ECR_REPO/spark:3.4-prd --conf
spark.kubernetes.executor.podNamePrefix=spark-connect --num-executors=10 --conf
spark.kubernetes.driver.pod.name=spark-connect-pod --conf
spark.kubernetes.namespace=MY_NAMESPACE && tail -100f
/opt/spark/logs/spark--org.apache.spark.sql.connect.service.SparkConnectServer-1-spark-connect-pod.out
image: MY_ECR_REPO/spark-py:3.4-prd
name: spark-connect-pod
{code}
The Spark Connect server was successfully launched and I can connect to it
using pyspark.
But when I want to add a Hive metastore config , it won't work.
{code:java}
>>> spark =
>>> SparkSession.builder.remote("sc://spark-connect-svc").config("spark.hive.metastore.uris",
>>> "thrift://hive-metastore:9083").getOrCreate()
>>> spark.sql("show databases").show()
+---------+
|namespace|
+---------+
| default|
+---------+{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
