dungdm93 commented on pull request #30738:
URL: https://github.com/apache/spark/pull/30738#issuecomment-748566945
@tgravescs here is an example of my deployment.
```yaml
apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
name: pyspark-pi
namespace: default
spec:
type: Python
sparkVersion: "3.0.0"
pythonVersion: "3"
mode: cluster
image: hub.acme.corp/library/pyspark:v3.0.0
imagePullPolicy: Always
mainApplicationFile: local://path/to/python/main.py
hadoopConf:
fs.s3a.impl: org.apache.hadoop.fs.s3a.S3AFileSystem
fs.s3a.endpoint: https://minio.acme.corp:443/
fs.s3a.path.style.access: "true"
fs.s3a.access.key: sample-access-key
fs.s3a.secret.key: sample-secret-key
# other config here
sparkConfigMap: spark-conf
driver:
cores: 1
coreLimit: "1200m"
memory: "512m"
serviceAccount: spark
executor:
instances: 1
cores: 1
memory: "512m"
---
apiVersion: v1
kind: ConfigMap
metadata:
name: spark-conf
namespace: default
data:
spark-env.sh: |
keytool -import -alias "acme.corp" -noprompt -file
$SPARK_CONF_DIR/ca.crt -keystore $JAVA_HOME/lib/security/cacerts -storepass
changeit
ca.crt: |
-----BEGIN CERTIFICATE-----
....
-----END CERTIFICATE-----
```
A few points worth noting:
* We use PySpark instead of Scala/Java, so write an ExecutorPlugin seem
overkilled.
* Spark configs, spark-env as well as ca.crt are mounted into spark pods at
runtime instead of bundle it into docker image. Is that your concern?
@holdenk thanks for you suggestion. I used to use that feature but it fails
in some situations because Kubernetes don't guarantee that the hook will
execute before the container entrypoint. You could checkout [k8s documentation
here](https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
