GitHub user liyinan926 opened a pull request:
https://github.com/apache/spark/pull/19954
[SPARK-22757][Kubernetes] add init-container bootstrapping and secret
mounting steps
## What changes were proposed in this pull request?
This PR expands the Kubernetes mode to be able to use remote dependencies
on http/https endpoints, GCS, S3, etc. It adds steps for configuring and
appending the Kubernetes init-container into the driver and executor pods for
downloading remote dependencies.
[Init-containers](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/),
as the name suggests, are containers that are run to completion before the
main containers start, and are often used to perform initialization tasks prior
to starting the main containers. We use init-containers to localize remote
application dependencies before the driver/executors start running. The code
that the init-container runs is also included. This PR also adds a step to the
driver and executors for mounting user-specified secrets that may store
credentials for accessing data storage, e.g., S3 and Google Cloud Storage
(GCS), into the driver and executors.
## How was this patch tested?
* The patch contains unit tests which are passing.
* Manual testing: `./build/mvn -Pkubernetes clean package` succeeded.
* It is a subset of the entire changelist hosted at
http://github.com/apache-spark-on-k8s/spark which is in active use in several
organizations.
* There is integration testing enabled in the fork currently hosted by
PepperData which is being moved over to RiseLAB CI.
* Detailed documentation on trying out the patch in its entirety is in:
https://apache-spark-on-k8s.github.io/userdocs/running-on-kubernetes.html
cc @rxin @felixcheung @mateiz (shepherd)
k8s-big-data SIG members & contributors: @mccheah @foxish @ash211 @ssuchter
@varunkatta @kimoonkim @erikerlandson @tnachen @ifilonenko @liyinan926
reviewers: @vanzin @felixcheung @jiangxb1987 @mridulm
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/liyinan926/spark-k8s init-container
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19954.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19954
----
commit 8aa875906ad11780587543f87634c21d3ca94677
Author: Yinan Li <[email protected]>
Date: 2017-12-12T18:24:27Z
[SPARK-22757][Kubernetes] add init-container bootstrapping and secret
mounting steps
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
