ChenYi015 commented on PR #2499: URL: https://github.com/apache/celeborn/pull/2499#issuecomment-2105512774
> Nice pr, I have some comments left. In addition, can you compare the actual YAML files before and after the modifications to see what differences there are? ```shell $ git checkout main Switched to branch 'main' $ helm template celeborn charts/celeborn >original.yaml (base) $ git checkout helm-unittest Switched to branch 'helm-unittest' Your branch is up to date with 'origin/helm-unittest'. (base) $ helm template celeborn charts/celeborn >current.yaml (base) $ diff original.yaml current.yaml 3,19d2 < # < # Licensed to the Apache Software Foundation (ASF) under one or more < # contributor license agreements. See the NOTICE file distributed with < # this work for additional information regarding copyright ownership. < # The ASF licenses this file to You under the Apache License, Version 2.0 < # (the "License"); you may not use this file except in compliance with < # the License. You may obtain a copy of the License at < # < # http://www.apache.org/licenses/LICENSE-2.0 < # < # Unless required by applicable law or agreed to in writing, software < # distributed under the License is distributed on an "AS IS" BASIS, < # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. < # See the License for the specific language governing permissions and < # limitations under the License. < # < 132,149c115 < # Source: celeborn/templates/master-service.yaml < # < # Licensed to the Apache Software Foundation (ASF) under one or more < # contributor license agreements. See the NOTICE file distributed with < # this work for additional information regarding copyright ownership. < # The ASF licenses this file to You under the Apache License, Version 2.0 < # (the "License"); you may not use this file except in compliance with < # the License. You may obtain a copy of the License at < # < # http://www.apache.org/licenses/LICENSE-2.0 < # < # Unless required by applicable law or agreed to in writing, software < # distributed under the License is distributed on an "AS IS" BASIS, < # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. < # See the License for the specific language governing permissions and < # limitations under the License. < # < --- > # Source: celeborn/templates/master/service.yaml 161,167d126 < type: ClusterIP < ports: < - port: 9097 < targetPort: 9097 < protocol: TCP < name: celeborn-master < clusterIP: None 171a131,137 > ports: > - port: 9097 > targetPort: 9097 > protocol: TCP > name: celeborn-master > type: ClusterIP > clusterIP: None 173,190c139 < # Source: celeborn/templates/worker-service.yaml < # < # Licensed to the Apache Software Foundation (ASF) under one or more < # contributor license agreements. See the NOTICE file distributed with < # this work for additional information regarding copyright ownership. < # The ASF licenses this file to You under the Apache License, Version 2.0 < # (the "License"); you may not use this file except in compliance with < # the License. You may obtain a copy of the License at < # < # http://www.apache.org/licenses/LICENSE-2.0 < # < # Unless required by applicable law or agreed to in writing, software < # distributed under the License is distributed on an "AS IS" BASIS, < # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. < # See the License for the specific language governing permissions and < # limitations under the License. < # < --- > # Source: celeborn/templates/worker/service.yaml 202,203d150 < type: ClusterIP < clusterIP: None 207a155,156 > type: ClusterIP > clusterIP: None 209,226c158 < # Source: celeborn/templates/master-statefulset.yaml < # < # Licensed to the Apache Software Foundation (ASF) under one or more < # contributor license agreements. See the NOTICE file distributed with < # this work for additional information regarding copyright ownership. < # The ASF licenses this file to You under the Apache License, Version 2.0 < # (the "License"); you may not use this file except in compliance with < # the License. You may obtain a copy of the License at < # < # http://www.apache.org/licenses/LICENSE-2.0 < # < # Unless required by applicable law or agreed to in writing, software < # distributed under the License is distributed on an "AS IS" BASIS, < # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. < # See the License for the specific language governing permissions and < # limitations under the License. < # < --- > # Source: celeborn/templates/master/statefulset.yaml 238a171 > serviceName: celeborn-master-svc 244,245d176 < serviceName: celeborn-master-svc < replicas: 3 253a185,241 > initContainers: > - name: chown-celeborn-master-volume > image: alpine:3.18 > imagePullPolicy: Always > command: > - chown > - 10006:10006 > - /mnt/celeborn_ratis > volumeMounts: > - name: celeborn-master-vol-0 > mountPath: /mnt/celeborn_ratis > securityContext: > runAsUser: 0 > containers: > - name: celeborn > image: aliyunemr/remote-shuffle-service:0.1.1-6badd20 > imagePullPolicy: Always > command: > - /usr/bin/tini > - -- > - /bin/sh > - -c > - "until true; do echo waiting for master; sleep 2; done && exec /opt/celeborn/sbin/start-master.sh" > ports: > - containerPort: 9097 > - containerPort: 9098 > name: metrics > protocol: TCP > env: > - name: CELEBORN_MASTER_JAVA_OPTS > value: "-XX:-PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:gc-master.out -Dio.netty.leakDetectionLevel=advanced" > - name: CELEBORN_MASTER_MEMORY > value: "2g" > - name: CELEBORN_NO_DAEMONIZE > value: "1" > - name: CELEBORN_WORKER_JAVA_OPTS > value: "-XX:-PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:gc-worker.out -Dio.netty.leakDetectionLevel=advanced" > - name: CELEBORN_WORKER_MEMORY > value: "2g" > - name: CELEBORN_WORKER_OFFHEAP_MEMORY > value: "12g" > - name: TZ > value: "Asia/Shanghai" > volumeMounts: > - name: celeborn-volume > mountPath: /opt/celeborn/conf > readOnly: true > - name: celeborn-master-vol-0 > mountPath: /mnt/celeborn_ratis > volumes: > - name: celeborn-volume > configMap: > name: celeborn-conf > - name: celeborn-master-vol-0 > hostPath: > path: /mnt/celeborn_ratis/master > type: DirectoryOrCreate 267a256 > dnsPolicy: ClusterFirst 272,326d260 < hostNetwork: false < dnsPolicy: ClusterFirst < initContainers: < - name: chown-celeborn-master-volume < image: alpine:3.18 < imagePullPolicy: Always < securityContext: < runAsUser: 0 < command: < - chown < - 10006:10006 < - /mnt/celeborn_ratis < resources: < null < volumeMounts: < - name: celeborn-master-vol-0 < mountPath: /mnt/celeborn_ratis < containers: < - name: celeborn < image: "aliyunemr/remote-shuffle-service:0.1.1-6badd20" < imagePullPolicy: Always < command: < - "/usr/bin/tini" < - "--" < - "/bin/sh" < - '-c' < - "until nslookup celeborn-master-0.celeborn-master-svc.default.svc.cluster.local && nslookup celeborn-master-1.celeborn-master-svc.default.svc.cluster.local && nslookup celeborn-master-2.celeborn-master-svc.default.svc.cluster.local && true; do echo waiting for master; sleep 2; done && exec /opt/celeborn/sbin/start-master.sh" < resources: < null < ports: < - containerPort: 9097 < - containerPort: 9098 < name: metrics < protocol: TCP < volumeMounts: < - mountPath: /opt/celeborn/conf < name: celeborn-volume < readOnly: true < - name: celeborn-master-vol-0 < mountPath: /mnt/celeborn_ratis < env: < - name: CELEBORN_MASTER_JAVA_OPTS < value: "-XX:-PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:gc-master.out -Dio.netty.leakDetectionLevel=advanced" < - name: CELEBORN_MASTER_MEMORY < value: "2g" < - name: CELEBORN_NO_DAEMONIZE < value: "1" < - name: CELEBORN_WORKER_JAVA_OPTS < value: "-XX:-PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:gc-worker.out -Dio.netty.leakDetectionLevel=advanced" < - name: CELEBORN_WORKER_MEMORY < value: "2g" < - name: CELEBORN_WORKER_OFFHEAP_MEMORY < value: "12g" < - name: TZ < value: "Asia/Shanghai" 328,335c262 < volumes: < - configMap: < name: celeborn-conf < name: celeborn-volume < - name: celeborn-master-vol-0 < hostPath: < path: /mnt/celeborn_ratis/master < type: DirectoryOrCreate --- > replicas: 3 337,354c264 < # Source: celeborn/templates/worker-statefulset.yaml < # < # Licensed to the Apache Software Foundation (ASF) under one or more < # contributor license agreements. See the NOTICE file distributed with < # this work for additional information regarding copyright ownership. < # The ASF licenses this file to You under the Apache License, Version 2.0 < # (the "License"); you may not use this file except in compliance with < # the License. You may obtain a copy of the License at < # < # http://www.apache.org/licenses/LICENSE-2.0 < # < # Unless required by applicable law or agreed to in writing, software < # distributed under the License is distributed on an "AS IS" BASIS, < # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. < # See the License for the specific language governing permissions and < # limitations under the License. < # < --- > # Source: celeborn/templates/worker/statefulset.yaml 366a277 > serviceName: celeborn-worker-svc 372,373d282 < serviceName: celeborn-worker < replicas: 5 382,401d290 < affinity: < podAntiAffinity: < requiredDuringSchedulingIgnoredDuringExecution: < - labelSelector: < matchExpressions: < - key: app.kubernetes.io/name < operator: In < values: < - celeborn < - key: app.kubernetes.io/role < operator: In < values: < - worker < topologyKey: kubernetes.io/hostname < securityContext: < fsGroup: 10006 < runAsGroup: 10006 < runAsUser: 10006 < hostNetwork: false < dnsPolicy: ClusterFirst 406,407d294 < securityContext: < runAsUser: 0 415,416d301 < resources: < null 425a311,312 > securityContext: > runAsUser: 0 428c315 < image: "aliyunemr/remote-shuffle-service:0.1.1-6badd20" --- > image: aliyunemr/remote-shuffle-service:0.1.1-6badd20 431,437c318,322 < - "/usr/bin/tini" < - "--" < - "/bin/sh" < - '-c' < - "until nslookup celeborn-master-0.celeborn-master-svc.default.svc.cluster.local && nslookup celeborn-master-1.celeborn-master-svc.default.svc.cluster.local && nslookup celeborn-master-2.celeborn-master-svc.default.svc.cluster.local && true; do echo waiting for master; sleep 2; done && exec /opt/celeborn/sbin/start-worker.sh" < resources: < null --- > - /usr/bin/tini > - -- > - /bin/sh > - -c > - "until true; do echo waiting for master; sleep 2; done && exec /opt/celeborn/sbin/start-worker.sh" 439,453c324,326 < - containerPort: 9096 < name: metrics < protocol: TCP < volumeMounts: < - mountPath: /opt/celeborn/conf < name: celeborn-volume < readOnly: true < - name: celeborn-worker-vol-0 < mountPath: /mnt/disk1 < - name: celeborn-worker-vol-1 < mountPath: /mnt/disk2 < - name: celeborn-worker-vol-2 < mountPath: /mnt/disk3 < - name: celeborn-worker-vol-3 < mountPath: /mnt/disk4 --- > - containerPort: 9096 > name: metrics > protocol: TCP 455,472c328,343 < - name: CELEBORN_MASTER_JAVA_OPTS < value: "-XX:-PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:gc-master.out -Dio.netty.leakDetectionLevel=advanced" < - name: CELEBORN_MASTER_MEMORY < value: "2g" < - name: CELEBORN_NO_DAEMONIZE < value: "1" < - name: CELEBORN_WORKER_JAVA_OPTS < value: "-XX:-PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:gc-worker.out -Dio.netty.leakDetectionLevel=advanced" < - name: CELEBORN_WORKER_MEMORY < value: "2g" < - name: CELEBORN_WORKER_OFFHEAP_MEMORY < value: "12g" < - name: TZ < value: "Asia/Shanghai" < terminationGracePeriodSeconds: 30 < volumes: < - configMap: < name: celeborn-conf --- > - name: CELEBORN_MASTER_JAVA_OPTS > value: "-XX:-PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:gc-master.out -Dio.netty.leakDetectionLevel=advanced" > - name: CELEBORN_MASTER_MEMORY > value: "2g" > - name: CELEBORN_NO_DAEMONIZE > value: "1" > - name: CELEBORN_WORKER_JAVA_OPTS > value: "-XX:-PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:gc-worker.out -Dio.netty.leakDetectionLevel=advanced" > - name: CELEBORN_WORKER_MEMORY > value: "2g" > - name: CELEBORN_WORKER_OFFHEAP_MEMORY > value: "12g" > - name: TZ > value: "Asia/Shanghai" > volumeMounts: > - mountPath: /opt/celeborn/conf 473a345 > readOnly: true 475,477c347 < hostPath: < path: /mnt/disk1/worker < type: DirectoryOrCreate --- > mountPath: /mnt/disk1 479,481c349 < hostPath: < path: /mnt/disk2/worker < type: DirectoryOrCreate --- > mountPath: /mnt/disk2 483,485c351 < hostPath: < path: /mnt/disk3/worker < type: DirectoryOrCreate --- > mountPath: /mnt/disk3 487,543c353,394 < hostPath: < path: /mnt/disk4/worker < type: DirectoryOrCreate < --- < # Source: celeborn/templates/master-priorityclass.yaml < # < # Licensed to the Apache Software Foundation (ASF) under one or more < # contributor license agreements. See the NOTICE file distributed with < # this work for additional information regarding copyright ownership. < # The ASF licenses this file to You under the Apache License, Version 2.0 < # (the "License"); you may not use this file except in compliance with < # the License. You may obtain a copy of the License at < # < # http://www.apache.org/licenses/LICENSE-2.0 < # < # Unless required by applicable law or agreed to in writing, software < # distributed under the License is distributed on an "AS IS" BASIS, < # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. < # See the License for the specific language governing permissions and < # limitations under the License. < # < --- < # Source: celeborn/templates/prometheus-podmonitor.yaml < # < # Licensed to the Apache Software Foundation (ASF) under one or more < # contributor license agreements. See the NOTICE file distributed with < # this work for additional information regarding copyright ownership. < # The ASF licenses this file to You under the Apache License, Version 2.0 < # (the "License"); you may not use this file except in compliance with < # the License. You may obtain a copy of the License at < # < # http://www.apache.org/licenses/LICENSE-2.0 < # < # Unless required by applicable law or agreed to in writing, software < # distributed under the License is distributed on an "AS IS" BASIS, < # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. < # See the License for the specific language governing permissions and < # limitations under the License. < # < --- < # Source: celeborn/templates/worker-priorityclass.yaml < # < # Licensed to the Apache Software Foundation (ASF) under one or more < # contributor license agreements. See the NOTICE file distributed with < # this work for additional information regarding copyright ownership. < # The ASF licenses this file to You under the Apache License, Version 2.0 < # (the "License"); you may not use this file except in compliance with < # the License. You may obtain a copy of the License at < # < # http://www.apache.org/licenses/LICENSE-2.0 < # < # Unless required by applicable law or agreed to in writing, software < # distributed under the License is distributed on an "AS IS" BASIS, < # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. < # See the License for the specific language governing permissions and < # limitations under the License. < # --- > mountPath: /mnt/disk4 > volumes: > - name: celeborn-volume > configMap: > name: celeborn-conf > - name: celeborn-worker-vol-0 > hostPath: > path: /mnt/disk1/worker > type: DirectoryOrCreate > - name: celeborn-worker-vol-1 > hostPath: > path: /mnt/disk2/worker > type: DirectoryOrCreate > - name: celeborn-worker-vol-2 > hostPath: > path: /mnt/disk3/worker > type: DirectoryOrCreate > - name: celeborn-worker-vol-3 > hostPath: > path: /mnt/disk4/worker > type: DirectoryOrCreate > affinity: > podAntiAffinity: > requiredDuringSchedulingIgnoredDuringExecution: > - labelSelector: > matchExpressions: > - key: app.kubernetes.io/name > operator: In > values: > - celeborn > - key: app.kubernetes.io/role > operator: In > values: > - worker > topologyKey: kubernetes.io/hostname > dnsPolicy: ClusterFirst > securityContext: > fsGroup: 10006 > runAsGroup: 10006 > runAsUser: 10006 > terminationGracePeriodSeconds: 30 > replicas: 5 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
