Manohar Chamaraju created SPARK-34293:
-----------------------------------------
Summary: kubernetes executor pod unable to access secure hdfs
Key: SPARK-34293
URL: https://issues.apache.org/jira/browse/SPARK-34293
Project: Spark
Issue Type: Bug
Components: Kubernetes
Affects Versions: 3.0.1
Reporter: Manohar Chamaraju
Steps to reproduce
# Configure secure HDFS cluster running as containers in kubernetes.
# Configure KDC on centos and create keytab for user principal hdfs, in
hdfsuser.keytab.
# Genearte spark image(v3.0.1), to spwan as container out of spark image.
# Inside spark container, run export HADOOP_CONF_DIR=/etc/hadoop/conf/ with
core-site.xml configuration as below !image-2021-01-30-00-03-41-506.png!
# Create configmap kbr-conf !image-2021-01-30-00-08-52-903.png!
# Run the command /opt/spark/bin/spark-submit \
--deploy-mode client \
--executor-memory 1g\
--executor-memory 1g\
--executor-cores 1\
--class org.apache.spark.examples.HdfsTest \
--conf spark.kubernetes.namespace=arcsight-installer-lh7fm\
--master k8s://https://172.17.17.1:443 \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--conf spark.app.name=spark-hdfs \
--conf spark.executer.instances=1 \
--conf spark.kubernetes.node.selector.spark=yes\
--conf spark.kubernetes.node.selector.Worker=label\
--conf spark.kubernetes.container.image=manohar/spark:v3.0.1 \
--conf spark.kubernetes.kerberos.enabled=true \
--conf spark.kubernetes.kerberos.krb5.configMapName=krb5-conf \
--conf spark.kerberos.keytab=/data/hdfsuser.keytab \
--conf [email protected] \
local:///opt/spark/examples/jars/spark-examples_2.12-3.0.1.jar \
hdfs://hdfs-namenode:30820/staging-directory.
# On running this command driver is able to connect hdfs with kerberos but
execurtor fails to connect to secure hdfs and below is the logs
!image-2021-01-30-00-11-22-401.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
