[
https://issues.apache.org/jira/browse/SPARK-34293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manohar Chamaraju updated SPARK-34293:
--------------------------------------
Description:
Steps to reproduce
# Configure secure HDFS cluster running as containers in kubernetes.
# Configure KDC on centos and create keytab for user principal hdfs, in
hdfsuser.keytab.
# Genearte spark image(v3.0.1), to spwan as container out of spark image.
# Inside spark container, run export HADOOP_CONF_DIR=/etc/hadoop/conf/ with
core-site.xml configuration as below !image-2021-01-30-00-03-41-506.png!
# Create configmap kbr-conf !image-2021-01-30-00-08-52-903.png!
# Run the command /opt/spark/bin/spark-submit \
--deploy-mode client \
--executor-memory 1g\
--executor-memory 1g\
--executor-cores 1\
--class org.apache.spark.examples.HdfsTest \
--conf spark.kubernetes.namespace=arcsight-installer-lh7fm\
--master k8s://[https://172.17.17.1:443|https://172.17.17.1/] \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--conf spark.app.name=spark-hdfs \
--conf spark.executer.instances=1 \
--conf spark.kubernetes.node.selector.spark=yes\
--conf spark.kubernetes.node.selector.Worker=label\
--conf spark.kubernetes.container.image=manohar/spark:v3.0.1 \
--conf spark.kubernetes.kerberos.enabled=true \
--conf spark.kubernetes.kerberos.krb5.configMapName=krb5-conf \
--conf spark.kerberos.keytab=/data/hdfsuser.keytab \
--conf [email protected] \
local:///opt/spark/examples/jars/spark-examples_2.12-3.0.1.jar \
hdfs://hdfs-namenode:30820/staging-directory.
# On running this command driver is able to connect hdfs with kerberos but
execurtor fails to connect to secure hdfs and below is the logs
!image-2021-01-30-00-11-22-401.png!
was:
Steps to reproduce
# Configure secure HDFS cluster running as containers in kubernetes.
# Configure KDC on centos and create keytab for user principal hdfs, in
hdfsuser.keytab.
# Genearte spark image(v3.0.1), to spwan as container out of spark image.
# Inside spark container, run export HADOOP_CONF_DIR=/etc/hadoop/conf/ with
core-site.xml configuration as below !image-2021-01-30-00-03-41-506.png!
# Create configmap kbr-conf !image-2021-01-30-00-08-52-903.png!
# Run the command /opt/spark/bin/spark-submit \
--deploy-mode client \
--executor-memory 1g\
--executor-memory 1g\
--executor-cores 1\
--class org.apache.spark.examples.HdfsTest \
--conf spark.kubernetes.namespace=arcsight-installer-lh7fm\
--master k8s://https://172.17.17.1:443 \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--conf spark.app.name=spark-hdfs \
--conf spark.executer.instances=1 \
--conf spark.kubernetes.node.selector.spark=yes\
--conf spark.kubernetes.node.selector.Worker=label\
--conf spark.kubernetes.container.image=manohar/spark:v3.0.1 \
--conf spark.kubernetes.kerberos.enabled=true \
--conf spark.kubernetes.kerberos.krb5.configMapName=krb5-conf \
--conf spark.kerberos.keytab=/data/hdfsuser.keytab \
--conf [email protected] \
local:///opt/spark/examples/jars/spark-examples_2.12-3.0.1.jar \
hdfs://hdfs-namenode:30820/staging-directory.
# On running this command driver is able to connect hdfs with kerberos but
execurtor fails to connect to secure hdfs and below is the logs
!image-2021-01-30-00-11-22-401.png!
> kubernetes executor pod unable to access secure hdfs
> ----------------------------------------------------
>
> Key: SPARK-34293
> URL: https://issues.apache.org/jira/browse/SPARK-34293
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
> Affects Versions: 3.0.1
> Reporter: Manohar Chamaraju
> Priority: Major
>
> Steps to reproduce
> # Configure secure HDFS cluster running as containers in kubernetes.
> # Configure KDC on centos and create keytab for user principal hdfs, in
> hdfsuser.keytab.
> # Genearte spark image(v3.0.1), to spwan as container out of spark image.
> # Inside spark container, run export HADOOP_CONF_DIR=/etc/hadoop/conf/ with
> core-site.xml configuration as below !image-2021-01-30-00-03-41-506.png!
> # Create configmap kbr-conf !image-2021-01-30-00-08-52-903.png!
> # Run the command /opt/spark/bin/spark-submit \
> --deploy-mode client \
> --executor-memory 1g\
> --executor-memory 1g\
> --executor-cores 1\
> --class org.apache.spark.examples.HdfsTest \
> --conf spark.kubernetes.namespace=arcsight-installer-lh7fm\
> --master k8s://[https://172.17.17.1:443|https://172.17.17.1/] \
> --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
> --conf spark.app.name=spark-hdfs \
> --conf spark.executer.instances=1 \
> --conf spark.kubernetes.node.selector.spark=yes\
> --conf spark.kubernetes.node.selector.Worker=label\
> --conf spark.kubernetes.container.image=manohar/spark:v3.0.1 \
> --conf spark.kubernetes.kerberos.enabled=true \
> --conf spark.kubernetes.kerberos.krb5.configMapName=krb5-conf \
> --conf spark.kerberos.keytab=/data/hdfsuser.keytab \
> --conf [email protected] \
> local:///opt/spark/examples/jars/spark-examples_2.12-3.0.1.jar \
> hdfs://hdfs-namenode:30820/staging-directory.
> # On running this command driver is able to connect hdfs with kerberos but
> execurtor fails to connect to secure hdfs and below is the logs
> !image-2021-01-30-00-11-22-401.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
