[
https://issues.apache.org/jira/browse/SPARK-28245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16878953#comment-16878953
]
Liu Runzhong edited comment on SPARK-28245 at 7/5/19 3:46 AM:
--------------------------------------------------------------
You should ensure you have put the jar like spark-kubernetes_2.11-2.3.0.jar
into your classpath when you develop your Spark application in Scala/Java.
For example, you can try it by spark-shell, if you download the
spark-2.4.0-bin-hadoop2.7.tgz file.
```
➜ spark-2.4.0-bin-hadoop2.7 ll jars/spark-kubernetes_2.11-2.4.0.jar
-rw-r--r--@ 1 runzhliu staff 523K 10 29 2018
jars/spark-kubernetes_2.11-2.4.0.jar
```
K8S as the external cluster manager, you need to load the related class by the
jar above.
!image-2019-07-05-11-45-21-609.png!
was (Author: runzhliu):
You should put the jar like spark-kubernetes_2.11-2.3.0.jar into your classpath
when you develop your Spark application.
For example, you can try it by spark-shell, if you download the
spark-2.4.0-bin-hadoop2.7.tgz file.
```
➜ spark-2.4.0-bin-hadoop2.7 ll jars/spark-kubernetes_2.11-2.4.0.jar
-rw-r--r--@ 1 runzhliu staff 523K 10 29 2018
jars/spark-kubernetes_2.11-2.4.0.jar
```
K8S as the external cluster manager, you need to load the related class by the
jar above.
!image-2019-07-05-11-45-21-609.png!
> Problem to recognize Kubernetes cluster address from Scala
> ----------------------------------------------------------
>
> Key: SPARK-28245
> URL: https://issues.apache.org/jira/browse/SPARK-28245
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
> Affects Versions: 2.4.0
> Reporter: Alberto Andreotti
> Priority: Major
> Attachments: image-2019-07-05-11-45-21-609.png
>
>
> This works in PySpark,
>
> from pyspark.sql import SparkSession
> spark = SparkSession.builder \
> .master('k8s://https://kubernetes.default') \
> But in Scala,
> val spark = SparkSession.builder()
> .master("k8s://https://kubernetes.default";)
> I get the following exception,
> Caused by: org.apache.spark.SparkException: Could not parse Master URL:
> 'k8s://https://kubernetes.default'
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
