[
https://issues.apache.org/jira/browse/SPARK-32259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prakash Rajendran updated SPARK-32259:
--------------------------------------
Description:
In Spark-Submit, I have these config
"{color:#4c9aff}*spark.kubernetes.local.dirs.tmpfs=true*{color}", still spark
is not pointing its spill data to SPARK_LOCAL_DIRS path.
K8s is evicting the pod due to error "{color:#de350b}*Pod ephemeral local
storage usage exceeds the total limit of containers.*{color}"
We use Spark launcher to do spark submit in k8s. Since it is evicted, the pod
logs for stack trace is not available. we have only pod events given in
attachment
was:
In Spark-Submit, I have these config
"{color:#4c9aff}*spark.kubernetes.local.dirs.tmpfs=true*{color}", still spark
is not pointing its spill data to SPARK_LOCAL_DIRS path.
K8s is evicting the pod due to error "{color:#de350b}*Pod ephemeral local
storage usage exceeds the total limit of containers.*{color}"
We use Spark launcher to do spark submit in k8s. Since it is evicted, the pod
logs for stack trace is not available. we have only pod events as below.
> tmpfs=true, not pointing to SPARK_LOCAL_DIRS in k8s
> ---------------------------------------------------
>
> Key: SPARK-32259
> URL: https://issues.apache.org/jira/browse/SPARK-32259
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
> Affects Versions: 3.0.0
> Reporter: Prakash Rajendran
> Priority: Blocker
> Attachments: Capture.PNG
>
>
> In Spark-Submit, I have these config
> "{color:#4c9aff}*spark.kubernetes.local.dirs.tmpfs=true*{color}", still spark
> is not pointing its spill data to SPARK_LOCAL_DIRS path.
> K8s is evicting the pod due to error "{color:#de350b}*Pod ephemeral local
> storage usage exceeds the total limit of containers.*{color}"
>
> We use Spark launcher to do spark submit in k8s. Since it is evicted, the pod
> logs for stack trace is not available. we have only pod events given in
> attachment
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
