[
https://issues.apache.org/jira/browse/SPARK-41496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yanzhe Xu updated SPARK-41496:
------------------------------
Description:
When defining multiple documents in the pod template file:
{code:java}
apiVersion: v1
kind: Pod
...# the first document to create a PVC
...
--- # use tripple-dash to split multiple documents in one yaml file
apiVersion: v1
kind: Pod
...
... # the second document to create the Spark pod.
{code}
Then set the podTemplate file in spark-submit command:
{code:java}
--conf
spark.kubernetes.executor.podTemplateFile=s3a://***/executor_pod_template.yaml
\{code}
Then Spark throws:
{code:java}
22/12/09 17:17:41 ERROR KubernetesUtils: Encountered exception while attempting
to load initial pod spec from file
expected a single document in the stream
in 'reader', line 1, column 1:
apiVersion: v1
^
but found another document
in 'reader', line 27, column 1:
---
^
at org.yaml.snakeyaml.composer.Composer.getSingleNode(Composer.java:125)
at
org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:150)
at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:490)
at org.yaml.snakeyaml.Yaml.load(Yaml.java:429)
at
io.fabric8.kubernetes.client.utils.Serialization.unmarshalYaml(Serialization.java:304)
at
io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:245)
at
io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:201)
at
io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:189)
at
io.fabric8.kubernetes.client.dsl.base.OperationSupport.unmarshal(OperationSupport.java:656)
at
io.fabric8.kubernetes.client.dsl.base.BaseOperation.load(BaseOperation.java:315)
at
io.fabric8.kubernetes.client.dsl.base.BaseOperation.load(BaseOperation.java:330)
at
io.fabric8.kubernetes.client.dsl.base.BaseOperation.load(BaseOperation.java:86)
at
org.apache.spark.deploy.k8s.KubernetesUtils$.loadPodFromTemplate(KubernetesUtils.scala:105)
{code}
The entrypoint code:
[https://github.com/apache/spark/blame/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala#L105]
was:
When defining multiple documents in the pod template file:
{code:java}
apiVersion: v1
kind: Pod
...# the first document to create a PVC
...
— # use tripple-dash to split multiple documents in one yaml file
apiVersion: v1
kind: Pod
...
... # the second document to create the Spark pod.
{code}
Then set the podTemplate file in spark-submit command:
{code:java}
--conf
spark.kubernetes.executor.podTemplateFile=s3a://***/executor_pod_template.yaml
\{code}
Then Spark throws:
{code:java}
22/12/09 17:17:41 ERROR KubernetesUtils: Encountered exception while attempting
to load initial pod spec from file
expected a single document in the stream
in 'reader', line 1, column 1:
apiVersion: v1
^
but found another document
in 'reader', line 27, column 1:
---
^
at org.yaml.snakeyaml.composer.Composer.getSingleNode(Composer.java:125)
at
org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:150)
at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:490)
at org.yaml.snakeyaml.Yaml.load(Yaml.java:429)
at
io.fabric8.kubernetes.client.utils.Serialization.unmarshalYaml(Serialization.java:304)
at
io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:245)
at
io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:201)
at
io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:189)
at
io.fabric8.kubernetes.client.dsl.base.OperationSupport.unmarshal(OperationSupport.java:656)
at
io.fabric8.kubernetes.client.dsl.base.BaseOperation.load(BaseOperation.java:315)
at
io.fabric8.kubernetes.client.dsl.base.BaseOperation.load(BaseOperation.java:330)
at
io.fabric8.kubernetes.client.dsl.base.BaseOperation.load(BaseOperation.java:86)
at
org.apache.spark.deploy.k8s.KubernetesUtils$.loadPodFromTemplate(KubernetesUtils.scala:105)
{code}
The entrypoint code:
[https://github.com/apache/spark/blame/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala#L105]
> Spark doesn't support multiple documents in one pod template file
> -----------------------------------------------------------------
>
> Key: SPARK-41496
> URL: https://issues.apache.org/jira/browse/SPARK-41496
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
> Affects Versions: 3.2.2
> Reporter: Yanzhe Xu
> Priority: Major
>
> When defining multiple documents in the pod template file:
> {code:java}
> apiVersion: v1
> kind: Pod
> ...# the first document to create a PVC
> ...
> --- # use tripple-dash to split multiple documents in one yaml file
> apiVersion: v1
> kind: Pod
> ...
> ... # the second document to create the Spark pod.
> {code}
>
> Then set the podTemplate file in spark-submit command:
>
> {code:java}
> --conf
> spark.kubernetes.executor.podTemplateFile=s3a://***/executor_pod_template.yaml
> \{code}
>
> Then Spark throws:
> {code:java}
> 22/12/09 17:17:41 ERROR KubernetesUtils: Encountered exception while
> attempting to load initial pod spec from file
> expected a single document in the stream
> in 'reader', line 1, column 1:
> apiVersion: v1
> ^
> but found another document
> in 'reader', line 27, column 1:
> ---
> ^
> at org.yaml.snakeyaml.composer.Composer.getSingleNode(Composer.java:125)
> at
> org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:150)
> at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:490)
> at org.yaml.snakeyaml.Yaml.load(Yaml.java:429)
> at
> io.fabric8.kubernetes.client.utils.Serialization.unmarshalYaml(Serialization.java:304)
> at
> io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:245)
> at
> io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:201)
> at
> io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:189)
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.unmarshal(OperationSupport.java:656)
> at
> io.fabric8.kubernetes.client.dsl.base.BaseOperation.load(BaseOperation.java:315)
> at
> io.fabric8.kubernetes.client.dsl.base.BaseOperation.load(BaseOperation.java:330)
> at
> io.fabric8.kubernetes.client.dsl.base.BaseOperation.load(BaseOperation.java:86)
> at
> org.apache.spark.deploy.k8s.KubernetesUtils$.loadPodFromTemplate(KubernetesUtils.scala:105)
> {code}
>
> The entrypoint code:
> [https://github.com/apache/spark/blame/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala#L105]
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
