This is an automated email from the ASF dual-hosted git repository.
zhongqiangchen pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-celeborn.git
The following commit(s) were added to refs/heads/main by this push:
new 8b4fe73d4 [CELEBORN-972][HELM] Enhance workingdirDiskCapacity unit
parsing and fix ConfigMap is not effected for workerStatefuleSet
8b4fe73d4 is described below
commit 8b4fe73d4ff49dc2bd98a44b519b5ca98285b0d9
Author: zhongqiang.czq <[email protected]>
AuthorDate: Thu Sep 14 11:58:26 2023 +0800
[CELEBORN-972][HELM] Enhance workingdirDiskCapacity unit parsing and fix
ConfigMap is not effected for workerStatefuleSet
### What changes were proposed in this pull request?
1. fix the issue with the configmap not being mounted for worker
2. fix compatability with different workingdir's capacity byte unit types,
e.g. Gi, Ti.
### Why are the changes needed?
1. in previous pr the configmap is removed from value.yaml, but
worker-statefulset.yaml still use this config, so worker pod can't mount the
/opt/celeborn/conf to configmap volume.
2. in previous pr capacity is appended for workingdir, but unit type Gi is
not suppored by byteStringTransformer
``` java
java.lang.NumberFormatException: Size must be specified as bytes (b),
kibibytes (k), mebibytes (m), gibibytes (g), tebibytes (t), or pebibytes(p).
E.g. 50b, 100k, or 250m.Invalid suffix: "gi"
celeborn.worker.storage.dirs=/mnt/disk1:disktype=SSD:capacity=100Gi,/mnt/disk2:disktype=SSD:capacity=100Gi,/mnt/disk3:disktype=SSD:capacity=100Gi,/mnt/disk4:disktype=SSD:capacity=100Gi
```
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
manual on k8s
1. the issue of configMap not being mounted
- befor this pr
```shell
kubectl get pod celeborn-worker-0 -o yaml |grep conf
- configMap:
name: celeborn-conf
- configMap:
```
- after this pr
``` shell
kubectl get pod celeborn-worker-0 -o yaml |grep conf
- mountPath: /opt/celeborn/conf
- configMap:
name: celeborn-conf
- configMap:
```
2. compatilbiy
the NumberFormatException is not thrown after this pr.
Closes #1906 from zhongqiangczq/helm-fix.
Authored-by: zhongqiang.czq <[email protected]>
Signed-off-by: zhongqiang.czq <[email protected]>
---
charts/celeborn/templates/worker-statefulset.yaml | 2 --
.../main/java/org/apache/celeborn/common/util/JavaUtils.java | 10 ++++++++++
2 files changed, 10 insertions(+), 2 deletions(-)
diff --git a/charts/celeborn/templates/worker-statefulset.yaml
b/charts/celeborn/templates/worker-statefulset.yaml
index 19df75577..8c01dd5ad 100644
--- a/charts/celeborn/templates/worker-statefulset.yaml
+++ b/charts/celeborn/templates/worker-statefulset.yaml
@@ -114,11 +114,9 @@ spec:
name: metrics
protocol: TCP
volumeMounts:
- {{- if .Values.configmap }}
- mountPath: /opt/celeborn/conf
name: {{ include "celeborn.fullname" . }}-volume
readOnly: true
- {{- end }}
{{- range $index, $volume := .Values.volumes.worker }}
- name: {{ $.Release.Name }}-worker-vol-{{ $index }}
mountPath: {{ .mountPath }}
diff --git
a/common/src/main/java/org/apache/celeborn/common/util/JavaUtils.java
b/common/src/main/java/org/apache/celeborn/common/util/JavaUtils.java
index 493ea4369..2580ac300 100644
--- a/common/src/main/java/org/apache/celeborn/common/util/JavaUtils.java
+++ b/common/src/main/java/org/apache/celeborn/common/util/JavaUtils.java
@@ -226,14 +226,24 @@ public class JavaUtils {
.put("b", ByteUnit.BYTE)
.put("k", ByteUnit.KiB)
.put("kb", ByteUnit.KiB)
+ .put("kib", ByteUnit.KiB)
+ .put("ki", ByteUnit.KiB)
.put("m", ByteUnit.MiB)
.put("mb", ByteUnit.MiB)
+ .put("mib", ByteUnit.MiB)
+ .put("mi", ByteUnit.MiB)
.put("g", ByteUnit.GiB)
.put("gb", ByteUnit.GiB)
+ .put("gib", ByteUnit.GiB)
+ .put("gi", ByteUnit.GiB)
.put("t", ByteUnit.TiB)
.put("tb", ByteUnit.TiB)
+ .put("tib", ByteUnit.TiB)
+ .put("ti", ByteUnit.TiB)
.put("p", ByteUnit.PiB)
.put("pb", ByteUnit.PiB)
+ .put("pib", ByteUnit.PiB)
+ .put("pi", ByteUnit.PiB)
.build();
/**
