cdmikechen created SUBMARINE-1124:
-------------------------------------
Summary: The automatic deletion of notebook does not delete the
related PVC and IngressRoute
Key: SUBMARINE-1124
URL: https://issues.apache.org/jira/browse/SUBMARINE-1124
Project: Apache Submarine
Issue Type: Bug
Components: K8s Submitter
Reporter: cdmikechen
Attachments: image-2021-12-10-09-56-32-491.png
I use newest submarine server to create a notebook called *test*. Notebook pod
can not start by this error with image
*apache/submarine:jupyter-notebook-0.7.0-SNAPSHOT*
!image-2021-12-10-09-56-32-491.png!
After a long wait, submarine will automatically delete the notebook row in the
database. I see the corresponding log information in the background.
{code:log}
[INFO ] 2021-12-10 09:23:25,015
method:org.apache.submarine.server.notebook.database.service.NotebookService.selectAll(NotebookService.java:43)
Notebook selectAll
Opening JDBC Connection
Checked out connection 1073981532 from pool.
Testing connection 1073981532 ...
Connection 1073981532 is GOOD!
Setting autocommit to false on JDBC Connection
[com.mysql.jdbc.JDBC4Connection@4003a85c]
==> Preparing: select id, notebook_spec, create_by, create_time, update_by,
update_time from notebook
==> Parameters:
<== Columns: id, notebook_spec, create_by, create_time, update_by,
update_time
<== Row: notebook_1639041890391_0002, <<BLOB>>, null, 2021-12-09
09:33:27.0, null, 2021-12-09 09:33:27.0
<== Total: 1
Resetting autocommit to true on JDBC Connection
[com.mysql.jdbc.JDBC4Connection@4003a85c]
Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@4003a85c]
Testing connection 1073981532 ...
Connection 1073981532 is GOOD!
Returned connection 1073981532 to pool.
[INFO ] 2021-12-10 09:23:25,593
method:org.apache.submarine.server.submitter.k8s.util.NotebookUtils.parseObject(NotebookUtils.java:62)
Upstream response JSON:
{"apiVersion":"kubeflow.org/v1","kind":"Notebook","metadata":{"creationTimestamp":"2021-12-09T09:33:27Z","generation":2.0,"labels":{"notebook-id":"notebook_1639041890391_0002","notebook-owner-id":"4291d7da9005377ec9aec4a71ea837f"},"managedFields":[{"apiVersion":"kubeflow.org/v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:notebook-id":{},"f:notebook-owner-id":{}}},"f:spec":{".":{},"f:template":{".":{},"f:spec":{".":{},"f:volumes":{}}}}},"manager":"Swagger-Codegen","operation":"Update","time":"2021-12-09T09:33:27Z"},{"apiVersion":"kubeflow.org/v1beta1","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:template":{"f:spec":{"f:containers":{}}}},"f:status":{".":{},"f:conditions":{},"f:containerState":{".":{},"f:waiting":{".":{},"f:message":{},"f:reason":{}}},"f:readyReplicas":{}}},"manager":"manager","operation":"Update","time":"2021-12-10T01:20:54Z"}],"name":"test","namespace":"submarine","resourceVersion":"274119717","selfLink":"/apis/kubeflow.org/v1/namespaces/submarine/notebooks/test","uid":"ddf1bc27-67e7-4df6-8259-649f3c8deb68"},"spec":{"template":{"spec":{"containers":[{"env":[{"name":"LANG","value":"zh_CN.UTF8"},{"name":"SUBMARINE_SERVER_DNS_NAME"},{"name":"SUBMARINE_SERVER_PORT","value":"8080"},{"name":"INSTALL_ENVIRONMENT_COMMAND","value":"minVersion\u003d\"4.0.1\";maxVersion\u003d\"4.10.10\";currentVersion\u003d$(conda
-V | cut -f2 -d\u0027 \u0027);if [ \"$(printf \u0027%s\\n\u0027
\"$minVersion\" \"$maxVersion\" \"$currentVersion\" | sort -V | head -n2 | tail
-1 )\" !\u003d \"$currentVersion\" ]; then echo \"Conda version should be
between minVersion\u003d\"4.0.1\"; and maxVersion\u003d\"4.10.10\";\"; exit 1;
else echo \"Conda current version is currentVersion\u003d$(conda -V | cut -f2
-d\u0027 \u0027);. Moving forward with env creation and activation.\";
fi"}],"image":"syharbor.sypesco.com:5000/apache/submarine:jupyter-notebook-0.7.0-SNAPSHOT","name":"test","resources":{"limits":{"cpu":"1","memory":"1024M"}},"volumeMounts":[{"mountPath":"/home/jovyan/workspace","name":"notebook-storage-test"}]}],"volumes":[{"name":"notebook-storage-test","persistentVolumeClaim":{"claimName":"notebook-pvc-test"}}]}}},"status":{"conditions":[{"lastProbeTime":"2021-12-10T01:20:54Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:20:41Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:20:38Z","type":"Running"},{"lastProbeTime":"2021-12-10T01:15:47Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:15:35Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:10:43Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:10:28Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:05:30Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:05:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:22Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:00:11Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:08Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:55:12Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:55:02Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:54:59Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:50:08Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:49:55Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:49:54Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:45:03Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:44:50Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:11Z","message":"back-off
2m40s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:42:00Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:00Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:40:44Z","message":"back-off
1m20s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:40:29Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:40:00Z","message":"back-off
40s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:47Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:46Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:39:30Z","message":"back-off
20s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:15Z","message":"back-off
10s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:03Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:38:57Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:26:49Z","reason":"ContainerCreating","type":"Waiting"},{"lastProbeTime":"2021-12-09T09:33:32Z","type":"Running"},{"lastProbeTime":"2021-12-09T09:33:29Z","reason":"ContainerCreating","type":"Waiting"}],"containerState":{"waiting":{"message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff"}},"readyReplicas":0.0}}
[INFO ] 2021-12-10 09:23:25,602
method:org.apache.submarine.server.submitter.k8s.K8sSubmitter.findNotebook(K8sSubmitter.java:457)
notebook status: waiting; check the pods in namespace:[submarine] to ensure is
the waiting caused by image pulling
[INFO ] 2021-12-10 09:23:25,904
method:org.apache.submarine.server.submitter.k8s.util.NotebookUtils.parseObject(NotebookUtils.java:62)
Upstream response JSON:
{"apiVersion":"kubeflow.org/v1","kind":"Notebook","metadata":{"creationTimestamp":"2021-12-09T09:33:27Z","generation":2.0,"labels":{"notebook-id":"notebook_1639041890391_0002","notebook-owner-id":"4291d7da9005377ec9aec4a71ea837f"},"managedFields":[{"apiVersion":"kubeflow.org/v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:notebook-id":{},"f:notebook-owner-id":{}}},"f:spec":{".":{},"f:template":{".":{},"f:spec":{".":{},"f:volumes":{}}}}},"manager":"Swagger-Codegen","operation":"Update","time":"2021-12-09T09:33:27Z"},{"apiVersion":"kubeflow.org/v1beta1","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:template":{"f:spec":{"f:containers":{}}}},"f:status":{".":{},"f:conditions":{},"f:containerState":{".":{},"f:waiting":{".":{},"f:message":{},"f:reason":{}}},"f:readyReplicas":{}}},"manager":"manager","operation":"Update","time":"2021-12-10T01:20:54Z"}],"name":"test","namespace":"submarine","resourceVersion":"274119717","selfLink":"/apis/kubeflow.org/v1/namespaces/submarine/notebooks/test","uid":"ddf1bc27-67e7-4df6-8259-649f3c8deb68"},"spec":{"template":{"spec":{"containers":[{"env":[{"name":"LANG","value":"zh_CN.UTF8"},{"name":"SUBMARINE_SERVER_DNS_NAME"},{"name":"SUBMARINE_SERVER_PORT","value":"8080"},{"name":"INSTALL_ENVIRONMENT_COMMAND","value":"minVersion\u003d\"4.0.1\";maxVersion\u003d\"4.10.10\";currentVersion\u003d$(conda
-V | cut -f2 -d\u0027 \u0027);if [ \"$(printf \u0027%s\\n\u0027
\"$minVersion\" \"$maxVersion\" \"$currentVersion\" | sort -V | head -n2 | tail
-1 )\" !\u003d \"$currentVersion\" ]; then echo \"Conda version should be
between minVersion\u003d\"4.0.1\"; and maxVersion\u003d\"4.10.10\";\"; exit 1;
else echo \"Conda current version is currentVersion\u003d$(conda -V | cut -f2
-d\u0027 \u0027);. Moving forward with env creation and activation.\";
fi"}],"image":"syharbor.sypesco.com:5000/apache/submarine:jupyter-notebook-0.7.0-SNAPSHOT","name":"test","resources":{"limits":{"cpu":"1","memory":"1024M"}},"volumeMounts":[{"mountPath":"/home/jovyan/workspace","name":"notebook-storage-test"}]}],"volumes":[{"name":"notebook-storage-test","persistentVolumeClaim":{"claimName":"notebook-pvc-test"}}]}}},"status":{"conditions":[{"lastProbeTime":"2021-12-10T01:20:54Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:20:41Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:20:38Z","type":"Running"},{"lastProbeTime":"2021-12-10T01:15:47Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:15:35Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:10:43Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:10:28Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:05:30Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:05:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:22Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:00:11Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:08Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:55:12Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:55:02Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:54:59Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:50:08Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:49:55Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:49:54Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:45:03Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:44:50Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:11Z","message":"back-off
2m40s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:42:00Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:00Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:40:44Z","message":"back-off
1m20s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:40:29Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:40:00Z","message":"back-off
40s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:47Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:46Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:39:30Z","message":"back-off
20s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:15Z","message":"back-off
10s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:03Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:38:57Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:26:49Z","reason":"ContainerCreating","type":"Waiting"},{"lastProbeTime":"2021-12-09T09:33:32Z","type":"Running"},{"lastProbeTime":"2021-12-09T09:33:29Z","reason":"ContainerCreating","type":"Waiting"}],"containerState":{"waiting":{"message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff"}},"readyReplicas":0.0}}
[INFO ] 2021-12-10 09:23:25,918
method:org.apache.submarine.server.submitter.k8s.K8sSubmitter.findNotebook(K8sSubmitter.java:457)
notebook status: waiting; check the pods in namespace:[submarine] to ensure is
the waiting caused by image pulling
[WARN ] 2021-12-10 09:23:27,257
method:org.apache.submarine.server.response.DictAnnotation.parseDictAnnotation(DictAnnotation.java:176)
Unsupported parse class java.util.ArrayList Dict Annotation!
[INFO ] 2021-12-10 09:23:27,279
method:org.apache.submarine.server.notebook.database.service.NotebookService.selectAll(NotebookService.java:43)
Notebook selectAll
Opening JDBC Connection
Checked out connection 1073981532 from pool.
Testing connection 1073981532 ...
[INFO ] 2021-12-10 09:23:27,283
method:org.apache.submarine.server.notebook.database.service.NotebookService.selectAll(NotebookService.java:43)
Notebook selectAll
Opening JDBC Connection
Connection 1073981532 is GOOD!
Setting autocommit to false on JDBC Connection
[com.mysql.jdbc.JDBC4Connection@4003a85c]
==> Preparing: select id, notebook_spec, create_by, create_time, update_by,
update_time from notebook
==> Parameters:
<== Columns: id, notebook_spec, create_by, create_time, update_by,
update_time
<== Row: notebook_1639041890391_0002, <<BLOB>>, null, 2021-12-09
09:33:27.0, null, 2021-12-09 09:33:27.0
<== Total: 1
Resetting autocommit to true on JDBC Connection
[com.mysql.jdbc.JDBC4Connection@4003a85c]
Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@4003a85c]
[WARN ] 2021-12-10 09:23:28,972
method:org.apache.submarine.server.response.DictAnnotation.parseDictAnnotation(DictAnnotation.java:176)
Unsupported parse class java.util.ArrayList Dict Annotation!
[INFO ] 2021-12-10 09:23:29,221
method:org.apache.submarine.server.notebook.database.service.NotebookService.selectAll(NotebookService.java:43)
Notebook selectAll
Opening JDBC Connection
[INFO ] 2021-12-10 09:23:30,257
method:org.apache.submarine.server.notebook.database.service.NotebookService.selectAll(NotebookService.java:43)
Notebook selectAll
Opening JDBC Connection
[INFO ] 2021-12-10 09:23:31,273
method:org.apache.submarine.server.notebook.database.service.NotebookService.selectAll(NotebookService.java:43)
Notebook selectAll
Opening JDBC Connection
[INFO ] 2021-12-10 09:23:32,683
method:org.apache.submarine.server.notebook.database.service.NotebookService.selectAll(NotebookService.java:43)
Notebook selectAll
Opening JDBC Connection
Created connection 1441218336.
Setting autocommit to false on JDBC Connection
[com.mysql.jdbc.JDBC4Connection@55e73f20]
==> Preparing: select id, notebook_spec, create_by, create_time, update_by,
update_time from notebook
==> Parameters:
Created connection 875758614.
Setting autocommit to false on JDBC Connection
[com.mysql.jdbc.JDBC4Connection@34330416]
==> Preparing: select id, notebook_spec, create_by, create_time, update_by,
update_time from notebook
==> Parameters:
Created connection 616832400.
Setting autocommit to false on JDBC Connection
[com.mysql.jdbc.JDBC4Connection@24c41d90]
==> Preparing: select id, notebook_spec, create_by, create_time, update_by,
update_time from notebook
==> Parameters:
Created connection 670559153.
Setting autocommit to false on JDBC Connection
[com.mysql.jdbc.JDBC4Connection@27f7ebb1]
==> Preparing: select id, notebook_spec, create_by, create_time, update_by,
update_time from notebook
==> Parameters:
Created connection 2081921914.
Setting autocommit to false on JDBC Connection
[com.mysql.jdbc.JDBC4Connection@7c179b7a]
Testing connection 1073981532 ...
==> Preparing: select id, notebook_spec, create_by, create_time, update_by,
update_time from notebook
==> Parameters:
Connection 1073981532 is GOOD!
Returned connection 1073981532 to pool.
<== Columns: id, notebook_spec, create_by, create_time, update_by,
update_time
<== Row: notebook_1639041890391_0002, <<BLOB>>, null, 2021-12-09
09:33:27.0, null, 2021-12-09 09:33:27.0
<== Total: 1
Resetting autocommit to true on JDBC Connection
[com.mysql.jdbc.JDBC4Connection@34330416]
Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@34330416]
Testing connection 875758614 ...
<== Columns: id, notebook_spec, create_by, create_time, update_by,
update_time
<== Row: notebook_1639041890391_0002, <<BLOB>>, null, 2021-12-09
09:33:27.0, null, 2021-12-09 09:33:27.0
<== Total: 1
Resetting autocommit to true on JDBC Connection
[com.mysql.jdbc.JDBC4Connection@24c41d90]
Connection 875758614 is GOOD!
Returned connection 875758614 to pool.
<== Columns: id, notebook_spec, create_by, create_time, update_by,
update_time
<== Row: notebook_1639041890391_0002, <<BLOB>>, null, 2021-12-09
09:33:27.0, null, 2021-12-09 09:33:27.0
<== Total: 1
Resetting autocommit to true on JDBC Connection
[com.mysql.jdbc.JDBC4Connection@27f7ebb1]
Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@24c41d90]
Testing connection 616832400 ...
Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@27f7ebb1]
<== Columns: id, notebook_spec, create_by, create_time, update_by,
update_time
<== Row: notebook_1639041890391_0002, <<BLOB>>, null, 2021-12-09
09:33:27.0, null, 2021-12-09 09:33:27.0
<== Total: 1
Resetting autocommit to true on JDBC Connection
[com.mysql.jdbc.JDBC4Connection@7c179b7a]
<== Columns: id, notebook_spec, create_by, create_time, update_by,
update_time
<== Row: notebook_1639041890391_0002, <<BLOB>>, null, 2021-12-09
09:33:27.0, null, 2021-12-09 09:33:27.0
Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@7c179b7a]
Connection 616832400 is GOOD!
Returned connection 616832400 to pool.
Testing connection 2081921914 ...
<== Total: 1
Resetting autocommit to true on JDBC Connection
[com.mysql.jdbc.JDBC4Connection@55e73f20]
Connection 2081921914 is GOOD!
Returned connection 2081921914 to pool.
Testing connection 670559153 ...
Connection 670559153 is GOOD!
Returned connection 670559153 to pool.
Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@55e73f20]
Testing connection 1441218336 ...
<== Updates: 1
Committing JDBC Connection [com.mysql.jdbc.JDBC4Connection@41c15145]
<== Updates: 0
Committing JDBC Connection [com.mysql.jdbc.JDBC4Connection@124f309f]
Connection 1441218336 is GOOD!
Resetting autocommit to true on JDBC Connection
[com.mysql.jdbc.JDBC4Connection@41c15145]
Resetting autocommit to true on JDBC Connection
[com.mysql.jdbc.JDBC4Connection@124f309f]
Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@124f309f]
Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@41c15145]
Closed connection 1441218336.
Testing connection 1103188293 ...
Connection 1103188293 is GOOD!
Closed connection 1103188293.
Testing connection 307179679 ...
[WARN ] 2021-12-10 09:23:35,350
method:org.apache.submarine.server.response.DictAnnotation.parseDictAnnotation(DictAnnotation.java:176)
Unsupported parse class java.util.ArrayList Dict Annotation!
Connection 307179679 is GOOD!
Closed connection 307179679.
[WARN ] 2021-12-10 09:23:35,352
method:org.apache.submarine.server.response.DictAnnotation.parseDictAnnotation(DictAnnotation.java:176)
Unsupported parse class java.util.ArrayList Dict Annotation!
[INFO ] 2021-12-10 09:23:38,326
method:org.apache.submarine.server.submitter.k8s.util.NotebookUtils.parseObject(NotebookUtils.java:62)
Upstream response JSON:
{"apiVersion":"kubeflow.org/v1","kind":"Notebook","metadata":{"creationTimestamp":"2021-12-09T09:33:27Z","generation":2.0,"labels":{"notebook-id":"notebook_1639041890391_0002","notebook-owner-id":"4291d7da9005377ec9aec4a71ea837f"},"managedFields":[{"apiVersion":"kubeflow.org/v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:notebook-id":{},"f:notebook-owner-id":{}}},"f:spec":{".":{},"f:template":{".":{},"f:spec":{".":{},"f:volumes":{}}}}},"manager":"Swagger-Codegen","operation":"Update","time":"2021-12-09T09:33:27Z"},{"apiVersion":"kubeflow.org/v1beta1","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:template":{"f:spec":{"f:containers":{}}}},"f:status":{".":{},"f:conditions":{},"f:containerState":{".":{},"f:waiting":{".":{},"f:message":{},"f:reason":{}}},"f:readyReplicas":{}}},"manager":"manager","operation":"Update","time":"2021-12-10T01:20:54Z"}],"name":"test","namespace":"submarine","resourceVersion":"274119717","selfLink":"/apis/kubeflow.org/v1/namespaces/submarine/notebooks/test","uid":"ddf1bc27-67e7-4df6-8259-649f3c8deb68"},"spec":{"template":{"spec":{"containers":[{"env":[{"name":"LANG","value":"zh_CN.UTF8"},{"name":"SUBMARINE_SERVER_DNS_NAME"},{"name":"SUBMARINE_SERVER_PORT","value":"8080"},{"name":"INSTALL_ENVIRONMENT_COMMAND","value":"minVersion\u003d\"4.0.1\";maxVersion\u003d\"4.10.10\";currentVersion\u003d$(conda
-V | cut -f2 -d\u0027 \u0027);if [ \"$(printf \u0027%s\\n\u0027
\"$minVersion\" \"$maxVersion\" \"$currentVersion\" | sort -V | head -n2 | tail
-1 )\" !\u003d \"$currentVersion\" ]; then echo \"Conda version should be
between minVersion\u003d\"4.0.1\"; and maxVersion\u003d\"4.10.10\";\"; exit 1;
else echo \"Conda current version is currentVersion\u003d$(conda -V | cut -f2
-d\u0027 \u0027);. Moving forward with env creation and activation.\";
fi"}],"image":"syharbor.sypesco.com:5000/apache/submarine:jupyter-notebook-0.7.0-SNAPSHOT","name":"test","resources":{"limits":{"cpu":"1","memory":"1024M"}},"volumeMounts":[{"mountPath":"/home/jovyan/workspace","name":"notebook-storage-test"}]}],"volumes":[{"name":"notebook-storage-test","persistentVolumeClaim":{"claimName":"notebook-pvc-test"}}]}}},"status":{"conditions":[{"lastProbeTime":"2021-12-10T01:20:54Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:20:41Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:20:38Z","type":"Running"},{"lastProbeTime":"2021-12-10T01:15:47Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:15:35Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:10:43Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:10:28Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:05:30Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:05:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:22Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:00:11Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:08Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:55:12Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:55:02Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:54:59Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:50:08Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:49:55Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:49:54Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:45:03Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:44:50Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:11Z","message":"back-off
2m40s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:42:00Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:00Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:40:44Z","message":"back-off
1m20s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:40:29Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:40:00Z","message":"back-off
40s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:47Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:46Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:39:30Z","message":"back-off
20s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:15Z","message":"back-off
10s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:03Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:38:57Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:26:49Z","reason":"ContainerCreating","type":"Waiting"},{"lastProbeTime":"2021-12-09T09:33:32Z","type":"Running"},{"lastProbeTime":"2021-12-09T09:33:29Z","reason":"ContainerCreating","type":"Waiting"}],"containerState":{"waiting":{"message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff"}},"readyReplicas":0.0}}
[INFO ] 2021-12-10 09:23:38,334
method:org.apache.submarine.server.submitter.k8s.K8sSubmitter.findNotebook(K8sSubmitter.java:457)
notebook status: waiting; check the pods in namespace:[submarine] to ensure is
the waiting caused by image pulling
[INFO ] 2021-12-10 09:23:38,338
method:org.apache.submarine.server.submitter.k8s.util.NotebookUtils.parseObject(NotebookUtils.java:62)
Upstream response JSON:
{"apiVersion":"kubeflow.org/v1","kind":"Notebook","metadata":{"creationTimestamp":"2021-12-09T09:33:27Z","generation":2.0,"labels":{"notebook-id":"notebook_1639041890391_0002","notebook-owner-id":"4291d7da9005377ec9aec4a71ea837f"},"managedFields":[{"apiVersion":"kubeflow.org/v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:notebook-id":{},"f:notebook-owner-id":{}}},"f:spec":{".":{},"f:template":{".":{},"f:spec":{".":{},"f:volumes":{}}}}},"manager":"Swagger-Codegen","operation":"Update","time":"2021-12-09T09:33:27Z"},{"apiVersion":"kubeflow.org/v1beta1","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:template":{"f:spec":{"f:containers":{}}}},"f:status":{".":{},"f:conditions":{},"f:containerState":{".":{},"f:waiting":{".":{},"f:message":{},"f:reason":{}}},"f:readyReplicas":{}}},"manager":"manager","operation":"Update","time":"2021-12-10T01:20:54Z"}],"name":"test","namespace":"submarine","resourceVersion":"274119717","selfLink":"/apis/kubeflow.org/v1/namespaces/submarine/notebooks/test","uid":"ddf1bc27-67e7-4df6-8259-649f3c8deb68"},"spec":{"template":{"spec":{"containers":[{"env":[{"name":"LANG","value":"zh_CN.UTF8"},{"name":"SUBMARINE_SERVER_DNS_NAME"},{"name":"SUBMARINE_SERVER_PORT","value":"8080"},{"name":"INSTALL_ENVIRONMENT_COMMAND","value":"minVersion\u003d\"4.0.1\";maxVersion\u003d\"4.10.10\";currentVersion\u003d$(conda
-V | cut -f2 -d\u0027 \u0027);if [ \"$(printf \u0027%s\\n\u0027
\"$minVersion\" \"$maxVersion\" \"$currentVersion\" | sort -V | head -n2 | tail
-1 )\" !\u003d \"$currentVersion\" ]; then echo \"Conda version should be
between minVersion\u003d\"4.0.1\"; and maxVersion\u003d\"4.10.10\";\"; exit 1;
else echo \"Conda current version is currentVersion\u003d$(conda -V | cut -f2
-d\u0027 \u0027);. Moving forward with env creation and activation.\";
fi"}],"image":"syharbor.sypesco.com:5000/apache/submarine:jupyter-notebook-0.7.0-SNAPSHOT","name":"test","resources":{"limits":{"cpu":"1","memory":"1024M"}},"volumeMounts":[{"mountPath":"/home/jovyan/workspace","name":"notebook-storage-test"}]}],"volumes":[{"name":"notebook-storage-test","persistentVolumeClaim":{"claimName":"notebook-pvc-test"}}]}}},"status":{"conditions":[{"lastProbeTime":"2021-12-10T01:20:54Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:20:41Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:20:38Z","type":"Running"},{"lastProbeTime":"2021-12-10T01:15:47Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:15:35Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:10:43Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:10:28Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:05:30Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:05:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:22Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:00:11Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:08Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:55:12Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:55:02Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:54:59Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:50:08Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:49:55Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:49:54Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:45:03Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:44:50Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:11Z","message":"back-off
2m40s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:42:00Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:00Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:40:44Z","message":"back-off
1m20s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:40:29Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:40:00Z","message":"back-off
40s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:47Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:46Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:39:30Z","message":"back-off
20s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:15Z","message":"back-off
10s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:03Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:38:57Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:26:49Z","reason":"ContainerCreating","type":"Waiting"},{"lastProbeTime":"2021-12-09T09:33:32Z","type":"Running"},{"lastProbeTime":"2021-12-09T09:33:29Z","reason":"ContainerCreating","type":"Waiting"}],"containerState":{"waiting":{"message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff"}},"readyReplicas":0.0}}
[INFO ] 2021-12-10 09:23:38,342
method:org.apache.submarine.server.submitter.k8s.K8sSubmitter.findNotebook(K8sSubmitter.java:457)
notebook status: waiting; check the pods in namespace:[submarine] to ensure is
the waiting caused by image pulling
[INFO ] 2021-12-10 09:23:40,057
method:org.apache.submarine.server.submitter.k8s.util.NotebookUtils.parseObject(NotebookUtils.java:62)
Upstream response JSON:
{"apiVersion":"kubeflow.org/v1","kind":"Notebook","metadata":{"creationTimestamp":"2021-12-09T09:33:27Z","generation":2.0,"labels":{"notebook-id":"notebook_1639041890391_0002","notebook-owner-id":"4291d7da9005377ec9aec4a71ea837f"},"managedFields":[{"apiVersion":"kubeflow.org/v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:notebook-id":{},"f:notebook-owner-id":{}}},"f:spec":{".":{},"f:template":{".":{},"f:spec":{".":{},"f:volumes":{}}}}},"manager":"Swagger-Codegen","operation":"Update","time":"2021-12-09T09:33:27Z"},{"apiVersion":"kubeflow.org/v1beta1","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:template":{"f:spec":{"f:containers":{}}}},"f:status":{".":{},"f:conditions":{},"f:containerState":{".":{},"f:waiting":{".":{},"f:message":{},"f:reason":{}}},"f:readyReplicas":{}}},"manager":"manager","operation":"Update","time":"2021-12-10T01:20:54Z"}],"name":"test","namespace":"submarine","resourceVersion":"274119717","selfLink":"/apis/kubeflow.org/v1/namespaces/submarine/notebooks/test","uid":"ddf1bc27-67e7-4df6-8259-649f3c8deb68"},"spec":{"template":{"spec":{"containers":[{"env":[{"name":"LANG","value":"zh_CN.UTF8"},{"name":"SUBMARINE_SERVER_DNS_NAME"},{"name":"SUBMARINE_SERVER_PORT","value":"8080"},{"name":"INSTALL_ENVIRONMENT_COMMAND","value":"minVersion\u003d\"4.0.1\";maxVersion\u003d\"4.10.10\";currentVersion\u003d$(conda
-V | cut -f2 -d\u0027 \u0027);if [ \"$(printf \u0027%s\\n\u0027
\"$minVersion\" \"$maxVersion\" \"$currentVersion\" | sort -V | head -n2 | tail
-1 )\" !\u003d \"$currentVersion\" ]; then echo \"Conda version should be
between minVersion\u003d\"4.0.1\"; and maxVersion\u003d\"4.10.10\";\"; exit 1;
else echo \"Conda current version is currentVersion\u003d$(conda -V | cut -f2
-d\u0027 \u0027);. Moving forward with env creation and activation.\";
fi"}],"image":"syharbor.sypesco.com:5000/apache/submarine:jupyter-notebook-0.7.0-SNAPSHOT","name":"test","resources":{"limits":{"cpu":"1","memory":"1024M"}},"volumeMounts":[{"mountPath":"/home/jovyan/workspace","name":"notebook-storage-test"}]}],"volumes":[{"name":"notebook-storage-test","persistentVolumeClaim":{"claimName":"notebook-pvc-test"}}]}}},"status":{"conditions":[{"lastProbeTime":"2021-12-10T01:20:54Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:20:41Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:20:38Z","type":"Running"},{"lastProbeTime":"2021-12-10T01:15:47Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:15:35Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:10:43Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:10:28Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:05:30Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:05:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:22Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:00:11Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:08Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:55:12Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:55:02Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:54:59Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:50:08Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:49:55Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:49:54Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:45:03Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:44:50Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:11Z","message":"back-off
2m40s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:42:00Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:00Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:40:44Z","message":"back-off
1m20s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:40:29Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:40:00Z","message":"back-off
40s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:47Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:46Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:39:30Z","message":"back-off
20s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:15Z","message":"back-off
10s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:03Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:38:57Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:26:49Z","reason":"ContainerCreating","type":"Waiting"},{"lastProbeTime":"2021-12-09T09:33:32Z","type":"Running"},{"lastProbeTime":"2021-12-09T09:33:29Z","reason":"ContainerCreating","type":"Waiting"}],"containerState":{"waiting":{"message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff"}},"readyReplicas":0.0}}
[INFO ] 2021-12-10 09:23:40,062
method:org.apache.submarine.server.submitter.k8s.K8sSubmitter.findNotebook(K8sSubmitter.java:457)
notebook status: waiting; check the pods in namespace:[submarine] to ensure is
the waiting caused by image pulling
[INFO ] 2021-12-10 09:23:42,452
method:org.apache.submarine.server.submitter.k8s.util.NotebookUtils.parseObject(NotebookUtils.java:62)
Upstream response JSON:
{"apiVersion":"kubeflow.org/v1","kind":"Notebook","metadata":{"creationTimestamp":"2021-12-09T09:33:27Z","generation":2.0,"labels":{"notebook-id":"notebook_1639041890391_0002","notebook-owner-id":"4291d7da9005377ec9aec4a71ea837f"},"managedFields":[{"apiVersion":"kubeflow.org/v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:notebook-id":{},"f:notebook-owner-id":{}}},"f:spec":{".":{},"f:template":{".":{},"f:spec":{".":{},"f:volumes":{}}}}},"manager":"Swagger-Codegen","operation":"Update","time":"2021-12-09T09:33:27Z"},{"apiVersion":"kubeflow.org/v1beta1","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:template":{"f:spec":{"f:containers":{}}}},"f:status":{".":{},"f:conditions":{},"f:containerState":{".":{},"f:waiting":{".":{},"f:message":{},"f:reason":{}}},"f:readyReplicas":{}}},"manager":"manager","operation":"Update","time":"2021-12-10T01:20:54Z"}],"name":"test","namespace":"submarine","resourceVersion":"274119717","selfLink":"/apis/kubeflow.org/v1/namespaces/submarine/notebooks/test","uid":"ddf1bc27-67e7-4df6-8259-649f3c8deb68"},"spec":{"template":{"spec":{"containers":[{"env":[{"name":"LANG","value":"zh_CN.UTF8"},{"name":"SUBMARINE_SERVER_DNS_NAME"},{"name":"SUBMARINE_SERVER_PORT","value":"8080"},{"name":"INSTALL_ENVIRONMENT_COMMAND","value":"minVersion\u003d\"4.0.1\";maxVersion\u003d\"4.10.10\";currentVersion\u003d$(conda
-V | cut -f2 -d\u0027 \u0027);if [ \"$(printf \u0027%s\\n\u0027
\"$minVersion\" \"$maxVersion\" \"$currentVersion\" | sort -V | head -n2 | tail
-1 )\" !\u003d \"$currentVersion\" ]; then echo \"Conda version should be
between minVersion\u003d\"4.0.1\"; and maxVersion\u003d\"4.10.10\";\"; exit 1;
else echo \"Conda current version is currentVersion\u003d$(conda -V | cut -f2
-d\u0027 \u0027);. Moving forward with env creation and activation.\";
fi"}],"image":"syharbor.sypesco.com:5000/apache/submarine:jupyter-notebook-0.7.0-SNAPSHOT","name":"test","resources":{"limits":{"cpu":"1","memory":"1024M"}},"volumeMounts":[{"mountPath":"/home/jovyan/workspace","name":"notebook-storage-test"}]}],"volumes":[{"name":"notebook-storage-test","persistentVolumeClaim":{"claimName":"notebook-pvc-test"}}]}}},"status":{"conditions":[{"lastProbeTime":"2021-12-10T01:20:54Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:20:41Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:20:38Z","type":"Running"},{"lastProbeTime":"2021-12-10T01:15:47Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:15:35Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:10:43Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:10:28Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:05:30Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:05:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:22Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:00:11Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:08Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:55:12Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:55:02Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:54:59Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:50:08Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:49:55Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:49:54Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:45:03Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:44:50Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:11Z","message":"back-off
2m40s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:42:00Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:00Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:40:44Z","message":"back-off
1m20s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:40:29Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:40:00Z","message":"back-off
40s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:47Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:46Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:39:30Z","message":"back-off
20s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:15Z","message":"back-off
10s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:03Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:38:57Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:26:49Z","reason":"ContainerCreating","type":"Waiting"},{"lastProbeTime":"2021-12-09T09:33:32Z","type":"Running"},{"lastProbeTime":"2021-12-09T09:33:29Z","reason":"ContainerCreating","type":"Waiting"}],"containerState":{"waiting":{"message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff"}},"readyReplicas":0.0}}
[INFO ] 2021-12-10 09:23:42,458
method:org.apache.submarine.server.submitter.k8s.K8sSubmitter.findNotebook(K8sSubmitter.java:457)
notebook status: waiting; check the pods in namespace:[submarine] to ensure is
the waiting caused by image pulling
[INFO ] 2021-12-10 09:23:42,705
method:org.apache.submarine.server.submitter.k8s.util.NotebookUtils.parseObject(NotebookUtils.java:62)
Upstream response JSON:
{"apiVersion":"kubeflow.org/v1","kind":"Notebook","metadata":{"creationTimestamp":"2021-12-09T09:33:27Z","generation":2.0,"labels":{"notebook-id":"notebook_1639041890391_0002","notebook-owner-id":"4291d7da9005377ec9aec4a71ea837f"},"managedFields":[{"apiVersion":"kubeflow.org/v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:notebook-id":{},"f:notebook-owner-id":{}}},"f:spec":{".":{},"f:template":{".":{},"f:spec":{".":{},"f:volumes":{}}}}},"manager":"Swagger-Codegen","operation":"Update","time":"2021-12-09T09:33:27Z"},{"apiVersion":"kubeflow.org/v1beta1","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:template":{"f:spec":{"f:containers":{}}}},"f:status":{".":{},"f:conditions":{},"f:containerState":{".":{},"f:waiting":{".":{},"f:message":{},"f:reason":{}}},"f:readyReplicas":{}}},"manager":"manager","operation":"Update","time":"2021-12-10T01:20:54Z"}],"name":"test","namespace":"submarine","resourceVersion":"274119717","selfLink":"/apis/kubeflow.org/v1/namespaces/submarine/notebooks/test","uid":"ddf1bc27-67e7-4df6-8259-649f3c8deb68"},"spec":{"template":{"spec":{"containers":[{"env":[{"name":"LANG","value":"zh_CN.UTF8"},{"name":"SUBMARINE_SERVER_DNS_NAME"},{"name":"SUBMARINE_SERVER_PORT","value":"8080"},{"name":"INSTALL_ENVIRONMENT_COMMAND","value":"minVersion\u003d\"4.0.1\";maxVersion\u003d\"4.10.10\";currentVersion\u003d$(conda
-V | cut -f2 -d\u0027 \u0027);if [ \"$(printf \u0027%s\\n\u0027
\"$minVersion\" \"$maxVersion\" \"$currentVersion\" | sort -V | head -n2 | tail
-1 )\" !\u003d \"$currentVersion\" ]; then echo \"Conda version should be
between minVersion\u003d\"4.0.1\"; and maxVersion\u003d\"4.10.10\";\"; exit 1;
else echo \"Conda current version is currentVersion\u003d$(conda -V | cut -f2
-d\u0027 \u0027);. Moving forward with env creation and activation.\";
fi"}],"image":"syharbor.sypesco.com:5000/apache/submarine:jupyter-notebook-0.7.0-SNAPSHOT","name":"test","resources":{"limits":{"cpu":"1","memory":"1024M"}},"volumeMounts":[{"mountPath":"/home/jovyan/workspace","name":"notebook-storage-test"}]}],"volumes":[{"name":"notebook-storage-test","persistentVolumeClaim":{"claimName":"notebook-pvc-test"}}]}}},"status":{"conditions":[{"lastProbeTime":"2021-12-10T01:20:54Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:20:41Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:20:38Z","type":"Running"},{"lastProbeTime":"2021-12-10T01:15:47Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:15:35Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:10:43Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:10:28Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:05:30Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:05:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:22Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T01:00:11Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T01:00:08Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:55:12Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:55:02Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:54:59Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:50:08Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:49:55Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:49:54Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:45:03Z","message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:44:50Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:11Z","message":"back-off
2m40s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:42:00Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:42:00Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:40:44Z","message":"back-off
1m20s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:40:29Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:40:00Z","message":"back-off
40s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:47Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:46Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:39:30Z","message":"back-off
20s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:18Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:39:15Z","message":"back-off
10s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff","type":"Waiting"},{"lastProbeTime":"2021-12-10T00:39:03Z","message":"Error","reason":"Error","type":"Terminated"},{"lastProbeTime":"2021-12-10T00:38:57Z","type":"Running"},{"lastProbeTime":"2021-12-10T00:26:49Z","reason":"ContainerCreating","type":"Waiting"},{"lastProbeTime":"2021-12-09T09:33:32Z","type":"Running"},{"lastProbeTime":"2021-12-09T09:33:29Z","reason":"ContainerCreating","type":"Waiting"}],"containerState":{"waiting":{"message":"back-off
5m0s restarting failed container\u003dtest
pod\u003dtest-0_submarine(fbf3e763-d87d-4d93-98b0-03f69e82cdee)","reason":"CrashLoopBackOff"}},"readyReplicas":0.0}}
[INFO ] 2021-12-10 09:23:42,711
method:org.apache.submarine.server.submitter.k8s.K8sSubmitter.findNotebook(K8sSubmitter.java:457)
notebook status: waiting; check the pods in namespace:[submarine] to ensure is
the waiting caused by image pulling
[WARN ] 2021-12-10 09:23:43,055
method:org.apache.submarine.server.response.DictAnnotation.parseDictAnnotation(DictAnnotation.java:176)
Unsupported parse class java.util.ArrayList Dict Annotation!
[WARN ] 2021-12-10 09:23:43,065
method:org.apache.submarine.server.response.DictAnnotation.parseDictAnnotation(DictAnnotation.java:176)
Unsupported parse class java.util.ArrayList Dict Annotation!
[INFO ] 2021-12-10 09:23:43,078
method:org.apache.submarine.server.notebook.database.service.NotebookService.selectAll(NotebookService.java:43)
Notebook selectAll
Opening JDBC Connection
Checked out connection 1073981532 from pool.
Testing connection 1073981532 ...
Connection 1073981532 is GOOD!
Setting autocommit to false on JDBC Connection
[com.mysql.jdbc.JDBC4Connection@4003a85c]
==> Preparing: select id, notebook_spec, create_by, create_time, update_by,
update_time from notebook
==> Parameters:
<== Total: 0
Resetting autocommit to true on JDBC Connection
[com.mysql.jdbc.JDBC4Connection@4003a85c]
Closing JDBC Connection [com.mysql.jdbc.JDBC4Connection@4003a85c]
Testing connection 1073981532 ...
Connection 1073981532 is GOOD!
Returned connection 1073981532 to pool.
[WARN ] 2021-12-10 09:23:43,083
method:org.apache.submarine.server.response.DictAnnotation.parseDictAnnotation(DictAnnotation.java:176)
Unsupported parse class java.util.ArrayList Dict Annotation!
[INFO ] 2021-12-10 09:23:43,085
method:org.apache.submarine.server.notebook.database.service.NotebookService.selectAll(NotebookService.java:43)
Notebook selectAll
Opening JDBC Connection
Checked out connection 875758614 from pool.
{code}
I checked codes and found that submarine just delete database row but it does
not delete the resource like CRD, PVC and IngressRoute. So that if I recreate
with a duplicate name, submarine tell me I can not create this.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
