GitHub user ifilonenko opened a pull request:
https://github.com/apache/spark/pull/21669
[SPARK-23257][K8S][WIP] Kerberos Support for Spark on K8S
## What changes were proposed in this pull request?
This is the work on setting up Secure HDFS interaction with Spark-on-K8S.
The architecture is discussed in this community-wide google
[doc](https://docs.google.com/document/d/1RBnXD9jMDjGonOdKJ2bA1lN4AAV_1RwpU_ewFuCNWKg)
This initiative can be broken down into 4 Stages
**STAGE 1**
- [x] Detecting `HADOOP_CONF_DIR` environmental variable and using Config
Maps to store all Hadoop config files locally, while also setting
`HADOOP_CONF_DIR` locally in the driver / executors
**STAGE 2**
- [x] Grabbing `TGT` from `LTC` or using keytabs+principle and creating a
`DT` that will be mounted as a secret or using a pre-populated secret
**STAGE 3**
- [x] Driver
**STAGE 3**
- [ ] Executor
## How was this patch tested?
- [ ] E2E Integration tests
- [ ] Unit tests
## Docs and Error Handling?
- [ ] Docs
- [ ] Error Handling
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ifilonenko/spark secure-hdfs
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21669.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21669
----
commit 73f2777c7ef1b925d323599952c792b5605d5c72
Author: Ilan Filonenko <if56@...>
Date: 2018-06-29T08:41:18Z
initial Driver logic for Hadoop and Kerberos Support
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
