[submarine] branch master updated: SUBMARINE-854. Avoid making the same bucket twice

[kaihsun]

This is an automated email from the ASF dual-hosted git repository.

kaihsun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/submarine.git


The following commit(s) were added to refs/heads/master by this push:
     new 292b759  SUBMARINE-854. Avoid making the same bucket twice
292b759 is described below

commit 292b759a962d718a1672dfdb37b00b879010079f
Author: Kai-Hsun Chen <b03901...@ntu.edu.tw>
AuthorDate: Thu Jun 17 23:32:22 2021 +0800

    SUBMARINE-854. Avoid making the same bucket twice
    
    ### What is this PR for?
    1. How to reproduce the bug?
    ```
    # Step1: Launch minikube
    minikube start --vm-driver=docker  --kubernetes-version v1.15.11
    
    # Step2: Install Submarine chart
    helm install submarine ./helm-charts/submarine
    
    # Step3: port-forwarding
    kubectl port-forward --address 0.0.0.0 service/submarine-traefik 32080:80
    
    # Step4: Check workbench 127.0.0.1:32080, and MLflow works well
    
    # Step5: Uninstall Submarine chart
    helm uninstall submarine
    
    # Step6: Wait until all resources related to the Submarine chart are deleted
    # Step7: Install Submarine chart again
    helm install submarine ./helm-charts/submarine
    
    # Step8: MLflow POD crash
    ```
    2. Root cause:
    * In Step2, the MLflow POD will execute "[./mc mb 
minio/mlflow](https://github.com/apache/submarine/blob/master/dev-support/docker-images/mlflow/start.sh#L35)"
 to make a bucket "minio/mlflow".
    * In Step5, the persistent volume `submarine-minio-pv` is deleted. However, 
the bucket "minio/mlflow" still exists in `hostPath`.
    * In Step7, the MLflow POD will execute "[./mc mb 
minio/mlflow](https://github.com/apache/submarine/blob/master/dev-support/docker-images/mlflow/start.sh#L35)"
 again. Because the bucket "minio/mlflow" is made twice, the command will cause 
error. Hence, the MLflow POD crashes.
    3. Reference:
    * 
https://github.com/bitnami/bitnami-docker-minio/commit/23109dcd6f0cc1b0d8f9700d6689909e3e613628
    
    ### What type of PR is it?
    [Bug Fix]
    
    ### Todos
    * The MLflow deployment should be created after the Minio POD is ready.
    
    ### What is the Jira issue?
    https://issues.apache.org/jira/browse/SUBMARINE-854
    
    ### How should this be tested?
    ```
    # Step1: Launch minikube & build image
    minikube start --vm-driver=docker  --kubernetes-version v1.15.11
    eval $(minikube docker-env)
    ./dev-support/docker-images/mlflow/build.sh
    
    # Step2: Install Submarine chart
    helm install submarine ./helm-charts/submarine
    
    # Step3: port-forwarding
    kubectl port-forward --address 0.0.0.0 service/submarine-traefik 32080:80
    
    # Step4: Check the log of MLFlow POD (See screenshot)
    # Step5: Check workbench 127.0.0.1:32080, and MLflow works well
    
    # Step6: Uninstall Submarine chart
    helm uninstall submarine
    
    # Step7: Wait until all resources related to the Submarine chart are deleted
    # Step8: Install Submarine chart again
    helm install submarine ./helm-charts/submarine
    
    # Step9: Check the log of MLFlow POD (See screenshot)
    # Step10: Check workbench 127.0.0.1:32080, and MLflow works well
    ```
    
    ### Screenshots (if appropriate)
    * Step4 screenshot
      * LOG: "Bucket created successfully minio/mlflow"
    <img width="1440" alt="截圖 2021-06-17 下午9 59 00" 
src="https://user-images.githubusercontent.com/20109646/122431323-ef49d680-cfc6-11eb-99a7-176bf48f2c76.png";>
    
    * Step10 screenshot
       * LOG: "Bucket minio/mlflow already exists, skipping creation."
    <img width="1440" alt="截圖 2021-06-17 下午11 30 57" 
src="https://user-images.githubusercontent.com/20109646/122431370-f96bd500-cfc6-11eb-8082-3365b8bb3293.png";>
    
    ### Questions:
    * Do the license files need updating? No
    * Are there breaking changes for older versions? No
    * Does this need new documentation? No
    
    Author: Kai-Hsun Chen <b03901...@ntu.edu.tw>
    
    Signed-off-by: Kai-Hsun Chen <kaih...@apache.org>
    
    Closes #607 from kevin85421/SUBMARINE-854 and squashes the following 
commits:
    
    6432a6e5 [Kai-Hsun Chen] SUBMARINE-854. Avoid making the same bucket twice
---
 dev-support/docker-images/mlflow/start.sh | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/dev-support/docker-images/mlflow/start.sh 
b/dev-support/docker-images/mlflow/start.sh
index 26d43cb..9849052 100755
--- a/dev-support/docker-images/mlflow/start.sh
+++ b/dev-support/docker-images/mlflow/start.sh
@@ -21,6 +21,15 @@
 
 set -euo pipefail
 
+# Check if the bucket "minio/mlflow" already exists
+check_minio_mlflow_bucket_exists() {
+    if /bin/bash -c "./mc ls minio/mlflow" >/dev/null 2>&1; then
+       true
+    else
+       false
+    fi
+}
+
 MLFLOW_S3_ENDPOINT_URL="http://10.96.0.4:9000";
 AWS_ACCESS_KEY_ID="submarine_minio"
 AWS_SECRET_ACCESS_KEY="submarine_minio"
@@ -32,6 +41,10 @@ STATIC_PREFIX="/mlflow"
 
 /bin/bash -c "sleep 60; ./mc config host add minio ${MLFLOW_S3_ENDPOINT_URL} 
${AWS_ACCESS_KEY_ID} ${AWS_SECRET_ACCESS_KEY}"
 
-/bin/bash -c "./mc mb minio/mlflow"
+if ! check_minio_mlflow_bucket_exists; then
+       /bin/bash -c "./mc mb minio/mlflow"
+else
+       echo "Bucket minio/mlflow already exists, skipping creation."
+fi
 
 /bin/bash -c "mlflow server --host 0.0.0.0 --backend-store-uri ${BACKEND_URI} 
--default-artifact-root ${DEFAULT_ARTIFACT_ROOT} --static-prefix 
${STATIC_PREFIX}"
\ No newline at end of file

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org