This is an automated email from the ASF dual-hosted git repository.
yao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kyuubi.git
The following commit(s) were added to refs/heads/master by this push:
new b0036fdf92 [KYUUBI #6521] [K8S][HELM] Implement new configuration
approach
b0036fdf92 is described below
commit b0036fdf927e8dfdb0c28c855f07aa607612187e
Author: dnskr <[email protected]>
AuthorDate: Fri Nov 22 17:47:07 2024 +0800
[KYUUBI #6521] [K8S][HELM] Implement new configuration approach
# :mag: Description
## Issue References ๐
This pull request changes Helm chart configuration approach as discussed in
https://github.com/apache/kyuubi/issues/6123
## Describe Your Solution ๐ง
Suggested implementation makes chart configuration more general and more
flexible. It allows to configure Kyuubi (and its engines) by setting
configuration file contents to chart properties or by providing configuration
files through existing ConfigMaps and Secrets. Also users are not limited by
predefined number of files and can put any files to configuration directories.
## Types of changes :bookmark:
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Breaking change (fix or feature that would cause existing
functionality to change)
## Test Plan ๐งช
### 1. Test suite "Kyuubi configuration"
Property file `values-kyuubi-files.yaml`
```yaml
kyuubiConf:
dir: /opt/kyuubi/conf
files:
'kyuubi-env.sh': |
#!/usr/bin/env bash
export KYUUBI_TEST=true
'kyuubi-custom.properties': |
kyuubi.custom=true
filesFrom:
- configMap:
name: kyuubi-configs
```
ConfigMap `kyuubi-configs` from `configmap-kyuubi-configs.yaml`
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: kyuubi-configs
data:
'kyuubi-test.properties': |
kyuubi.config.test=true
```
#### Rendered templates are correct
```shell
$ helm template charts/kyuubi -f values-kyuubi-files.yaml -s
templates/kyuubi-configmap.yaml -s templates/kyuubi-statefulset.yaml
```
#### Configuration files are in place
```shell
$ kubectl create -f configmap-kyuubi-configs.yaml
$ helm install kyuubi charts/kyuubi -f values-kyuubi-files.yaml
$ kubectl exec kyuubi-0 -- ls conf
kyuubi-custom.properties
kyuubi-env.sh
kyuubi-test.properties
```
### 2. Test suite "Spark configuration"
Property file `values-spark-files.yaml`
```yaml
sparkConf:
dir: /opt/spark/conf
files:
'spark-env.sh': |
#!/usr/bin/env bash
export SPARK_TEST=true
'spark-custom.properties': |
spark.custom=true
filesFrom:
- configMap:
name: spark-configs
```
ConfigMap `spark-configs` from `configmap-spark-configs.yaml`
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: spark-configs
data:
'spark-test.properties': |
spark.config.test=true
```
#### Rendered templates are correct
```shell
$ helm template charts/kyuubi -f values-spark-files.yaml -s
templates/kyuubi-spark-configmap.yaml -s templates/kyuubi-statefulset.yaml
```
#### Configuration files are in place
```shell
$ kubectl create -f configmap-spark-configs.yaml
$ helm install kyuubi charts/kyuubi -f values-spark-files.yaml
$ kubectl exec kyuubi-0 -- ls ../spark/conf
spark-custom.properties
spark-env.sh
spark-test.properties
```
3. Test suite "Custom kyuubi-defaults.conf from existing ConfigMap
overwrites kyuubi-defaults.conf from values"
Property file `values-kyuubi-defaults.yaml`
```yaml
kyuubiConf:
dir: /opt/kyuubi/conf
files:
'kyuubi-defaults.conf': |
custom.from.values=true
filesFrom:
- configMap:
name: kyuubi-defaults-config
```
ConfigMap `kyuubi-defaults-config` from `configmap-kyuubi-defaults.yaml`
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: kyuubi-defaults-config
data:
'kyuubi-defaults.conf': |
custom.from.configmap=true
```
#### Rendered templates are correct
```shell
$ helm template charts/kyuubi -f values-kyuubi-defaults.yaml -s
templates/kyuubi-configmap.yaml -s templates/kyuubi-statefulset.yaml
```
#### Content of `kyuubi-defaults.conf` comes from ConfigMap
```shell
$ kubectl create -f configmap-kyuubi-defaults.yaml
$ helm install kyuubi charts/kyuubi -f values-kyuubi-defaults.yaml
$ kubectl exec kyuubi-0 -- ls conf
kyuubi-defaults.conf
$ kubectl exec kyuubi-0 -- cat conf/kyuubi-defaults.conf
custom.from.configmap=true
```
---
# Checklist ๐
- [x] This patch was not authored or co-authored using [Generative
Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6521 from dnskr/implement-new-helm-chart-configuration-approach.
Closes #6521
452dca32f [dnskr] Fix empty value type
14829f342 [dnskr] Revert "[REVERT BEFORE MERGE] Use 'master-snapshot' image
tag"
8d90f424c [dnskr] Move default properties from 'kyuubi-defaults.conf' to
--conf args
6b3c77f94 [dnskr] [REVERT BEFORE MERGE] Use 'master-snapshot' image tag
fe7c17a68 [dnskr] [K8S][HELM] Implement new configuration approach
Authored-by: dnskr <[email protected]>
Signed-off-by: Kent Yao <[email protected]>
---
charts/kyuubi/templates/kyuubi-configmap.yaml | 28 +------
.../kyuubi/templates/kyuubi-spark-configmap.yaml | 17 +---
charts/kyuubi/templates/kyuubi-statefulset.yaml | 42 +++++++---
charts/kyuubi/values.yaml | 90 ++++++++++------------
4 files changed, 79 insertions(+), 98 deletions(-)
diff --git a/charts/kyuubi/templates/kyuubi-configmap.yaml
b/charts/kyuubi/templates/kyuubi-configmap.yaml
index a171d5855f..6465ec79df 100644
--- a/charts/kyuubi/templates/kyuubi-configmap.yaml
+++ b/charts/kyuubi/templates/kyuubi-configmap.yaml
@@ -22,30 +22,6 @@ metadata:
labels:
{{- include "kyuubi.labels" . | nindent 4 }}
data:
- {{- with .Values.kyuubiConf.kyuubiEnv }}
- kyuubi-env.sh: |
- {{- tpl . $ | nindent 4 }}
- {{- end }}
- kyuubi-defaults.conf: |
- ## Helm chart provided Kyuubi configurations
- kyuubi.kubernetes.namespace={{ .Release.Namespace }}
- kyuubi.frontend.connection.url.use.hostname=false
- kyuubi.frontend.thrift.binary.bind.port={{
.Values.server.thriftBinary.port }}
- kyuubi.frontend.thrift.http.bind.port={{ .Values.server.thriftHttp.port }}
- kyuubi.frontend.rest.bind.port={{ .Values.server.rest.port }}
- kyuubi.frontend.mysql.bind.port={{ .Values.server.mysql.port }}
- kyuubi.frontend.protocols={{ include "kyuubi.frontend.protocols" . }}
-
- # Kyuubi Metrics
- kyuubi.metrics.enabled={{ .Values.metrics.enabled }}
- kyuubi.metrics.reporters={{ .Values.metrics.reporters }}
- kyuubi.metrics.prometheus.port={{ .Values.metrics.prometheusPort }}
-
- ## User provided Kyuubi configurations
- {{- with .Values.kyuubiConf.kyuubiDefaults }}
- {{- tpl . $ | nindent 4 }}
- {{- end }}
- {{- with .Values.kyuubiConf.log4j2 }}
- log4j2.xml: |
- {{- tpl . $ | nindent 4 }}
+ {{- with .Values.kyuubiConf.files }}
+ {{- tpl (toYaml .) $ | nindent 2 }}
{{- end }}
diff --git a/charts/kyuubi/templates/kyuubi-spark-configmap.yaml
b/charts/kyuubi/templates/kyuubi-spark-configmap.yaml
index 5794c429f5..9b2e9784c6 100644
--- a/charts/kyuubi/templates/kyuubi-spark-configmap.yaml
+++ b/charts/kyuubi/templates/kyuubi-spark-configmap.yaml
@@ -22,19 +22,6 @@ metadata:
labels:
{{- include "kyuubi.labels" . | nindent 4 }}
data:
- {{- with .Values.sparkConf.sparkEnv }}
- spark-env.sh: |
- {{- tpl . $ | nindent 4 }}
- {{- end }}
- {{- with .Values.sparkConf.sparkDefaults }}
- spark-defaults.conf: |
- {{- tpl . $ | nindent 4 }}
- {{- end }}
- {{- with .Values.sparkConf.log4j2 }}
- log4j2.properties: |
- {{- tpl . $ | nindent 4 }}
- {{- end }}
- {{- with .Values.sparkConf.metrics }}
- metrics.properties: |
- {{- tpl . $ | nindent 4 }}
+ {{- with .Values.sparkConf.files }}
+ {{- tpl (toYaml .) $ | nindent 2 }}
{{- end }}
diff --git a/charts/kyuubi/templates/kyuubi-statefulset.yaml
b/charts/kyuubi/templates/kyuubi-statefulset.yaml
index caea7d251b..57601826f3 100644
--- a/charts/kyuubi/templates/kyuubi-statefulset.yaml
+++ b/charts/kyuubi/templates/kyuubi-statefulset.yaml
@@ -59,14 +59,28 @@ spec:
{{- with .Values.command }}
command: {{- tpl (toYaml .) $ | nindent 12 }}
{{- end }}
- {{- with .Values.args }}
+ {{- if .Values.args }}
args: {{- tpl (toYaml .) $ | nindent 12 }}
+ {{- else }}
+ args:
+ - ./bin/kyuubi
+ - run
+ - --conf kyuubi.kubernetes.namespace={{ .Release.Namespace }}
+ - --conf kyuubi.frontend.connection.url.use.hostname=false
+ - --conf kyuubi.frontend.thrift.binary.bind.port={{
.Values.server.thriftBinary.port }}
+ - --conf kyuubi.frontend.thrift.http.bind.port={{
.Values.server.thriftHttp.port }}
+ - --conf kyuubi.frontend.rest.bind.port={{
.Values.server.rest.port }}
+ - --conf kyuubi.frontend.mysql.bind.port={{
.Values.server.mysql.port }}
+ - --conf kyuubi.frontend.protocols={{ include
"kyuubi.frontend.protocols" . }}
+ - --conf kyuubi.metrics.enabled={{ .Values.metrics.enabled }}
+ - --conf kyuubi.metrics.reporters={{ .Values.metrics.reporters }}
+ - --conf kyuubi.metrics.prometheus.port={{
.Values.metrics.prometheusPort }}
{{- end }}
env:
- name: KYUUBI_CONF_DIR
- value: {{ .Values.kyuubiConfDir }}
+ value: {{ .Values.kyuubiConf.dir }}
- name: SPARK_CONF_DIR
- value: {{ .Values.sparkConfDir }}
+ value: {{ .Values.sparkConf.dir }}
{{- with .Values.env }}
{{- tpl (toYaml .) $ | nindent 12 }}
{{- end }}
@@ -109,9 +123,9 @@ spec:
{{- end }}
volumeMounts:
- name: conf
- mountPath: {{ .Values.kyuubiConfDir }}
+ mountPath: {{ .Values.kyuubiConf.dir }}
- name: conf-spark
- mountPath: {{ .Values.sparkConfDir }}
+ mountPath: {{ .Values.sparkConf.dir }}
{{- with .Values.volumeMounts }}
{{- tpl (toYaml .) $ | nindent 12 }}
{{- end }}
@@ -120,11 +134,21 @@ spec:
{{- end }}
volumes:
- name: conf
- configMap:
- name: {{ .Release.Name }}
+ projected:
+ sources:
+ - configMap:
+ name: {{ .Release.Name }}
+ {{- with .Values.kyuubiConf.filesFrom }}
+ {{- tpl (toYaml .) $ | nindent 14 }}
+ {{- end }}
- name: conf-spark
- configMap:
- name: {{ .Release.Name }}-spark
+ projected:
+ sources:
+ - configMap:
+ name: {{ .Release.Name }}-spark
+ {{- with .Values.sparkConf.filesFrom }}
+ {{- tpl (toYaml .) $ | nindent 14 }}
+ {{- end }}
{{- with .Values.volumes }}
{{- tpl (toYaml .) $ | nindent 8 }}
{{- end }}
diff --git a/charts/kyuubi/values.yaml b/charts/kyuubi/values.yaml
index 23e5e7fdc4..1f35c9ba87 100644
--- a/charts/kyuubi/values.yaml
+++ b/charts/kyuubi/values.yaml
@@ -145,53 +145,40 @@ server:
# clientIP:
# timeoutSeconds: 10800
-# $KYUUBI_CONF_DIR directory
-kyuubiConfDir: /opt/kyuubi/conf
# Kyuubi configuration files
kyuubiConf:
- # The value (templated string) is used for kyuubi-env.sh file
- # See example at conf/kyuubi-env.sh.template and
https://kyuubi.readthedocs.io/en/master/configuration/settings.html#environments
for more details
- kyuubiEnv: ~
- # kyuubiEnv: |
- # #!/usr/bin/env bash
- # export JAVA_HOME=/usr/jdk64/jdk1.8.0_152
- # export SPARK_HOME=/opt/spark
- # export FLINK_HOME=/opt/flink
- # export HIVE_HOME=/opt/hive
-
- # The value (templated string) is used for kyuubi-defaults.conf file
- # See
https://kyuubi.readthedocs.io/en/master/configuration/settings.html#kyuubi-configurations
for more details
- kyuubiDefaults: ~
- # kyuubiDefaults: |
+ # $KYUUBI_CONF_DIR directory
+ dir: /opt/kyuubi/conf
+ # Configuration files from the specified keys (file name) and values (file
content)
+ files: ~
+ #files:
+ # 'kyuubi-defaults.conf': |
# kyuubi.authentication=NONE
- # kyuubi.frontend.bind.host=10.0.0.1
- # kyuubi.engine.type=SPARK_SQL
# kyuubi.engine.share.level=USER
- # kyuubi.session.engine.initialize.timeout=PT3M
- # kyuubi.ha.addresses=zk1:2181,zk2:2181,zk3:2181
- # kyuubi.ha.namespace=kyuubi
- # The value (templated string) is used for log4j2.xml file
- # See example at conf/log4j2.xml.template
https://kyuubi.readthedocs.io/en/master/configuration/settings.html#logging for
more details
- log4j2: ~
-
-# $SPARK_CONF_DIR directory
-sparkConfDir: /opt/spark/conf
-# Spark configuration files
+ # Configuration files from the list of existing ConfigMaps and Secrets
+ filesFrom: []
+ #filesFrom:
+ #- configMap:
+ # name: kyuubi-configs
+ #- secret:
+ # name: kyuubi-secrets
+ #- secret:
+ # name: ssl-secrets
+ # items:
+ # - key: key-store
+ # path: certs/keystore.jks
+ # - key: trust-store
+ # path: certs/truststore.jks
+
+# Spark configuration, see https://github.com/apache/spark/tree/master/conf
and Spark documentation for more details
sparkConf:
- # The value (templated string) is used for spark-env.sh file
- # See example at
https://github.com/apache/spark/blob/master/conf/spark-env.sh.template and
Spark documentation for more details
- sparkEnv: ~
- # sparkEnv: |
- # #!/usr/bin/env bash
- # export JAVA_HOME=/usr/jdk64/jdk1.8.0_152
- # export SPARK_LOG_DIR=/opt/spark/logs
- # export SPARK_LOG_MAX_FILES=5
-
- # The value (templated string) is used for spark-defaults.conf file
- # See example at
https://github.com/apache/spark/blob/master/conf/spark-defaults.conf.template
and Spark documentation for more details
- sparkDefaults: ~
- # sparkDefaults: |
+ # $SPARK_CONF_DIR directory
+ dir: /opt/spark/conf
+ # Configuration files from the specified keys (file name) and values (file
content)
+ files: ~
+ #files:
+ # 'spark-defaults.conf': |
# spark.submit.deployMode=cluster
# spark.kubernetes.container.image=apache/spark:3.5.0
# spark.kubernetes.authenticate.driver.serviceAccountName=spark
@@ -207,13 +194,20 @@ sparkConf:
# spark.hadoop.fs.s3a.path.style.access=true
# spark.hadoop.fs.s3a.fast.upload=true
- # The value (templated string) is used for log4j2.properties file
- # See example at
https://github.com/apache/spark/blob/master/conf/log4j2.properties.template and
Spark documentation for more details
- log4j2: ~
-
- # The value (templated string) is used for metrics.properties file
- # See example at
https://github.com/apache/spark/blob/master/conf/metrics.properties.template
and Spark documentation for more details
- metrics: ~
+ # Configuration files from the list of existing ConfigMaps and Secrets
+ filesFrom: []
+ #filesFrom:
+ #- configMap:
+ # name: spark-configs
+ #- secret:
+ # name: spark-secrets
+ #- secret:
+ # name: ssl-secrets
+ # items:
+ # - key: key-store
+ # path: certs/keystore.jks
+ # - key: trust-store
+ # path: certs/truststore.jks
# Command to launch Kyuubi server (templated)
command: ~
