[
https://issues.apache.org/jira/browse/SUBMARINE-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
cdmikechen reassigned SUBMARINE-1154:
-------------------------------------
Assignee: cdmikechen
> tf-job-operator can not create experiment pod in openshift
> ----------------------------------------------------------
>
> Key: SUBMARINE-1154
> URL: https://issues.apache.org/jira/browse/SUBMARINE-1154
> Project: Apache Submarine
> Issue Type: Bug
> Components: experiment
> Reporter: cdmikechen
> Assignee: cdmikechen
> Priority: Critical
> Labels: pull-request-available
> Fix For: 0.7.0
>
> Attachments: image-2021-12-21-16-42-30-133.png
>
>
> I use OKD 4.7 to run submarine. When I created an experiment, I can not find
> pod started successfully.
> I checked tf-job-operator log and found that some problems were encountered
> when operator creating pod.
>
> {code}
> {"filename":"app/server.go:76","level":"info","msg":"EnvKubeflowNamespace not
> set, use default namespace","time":"2021-12-21T08:17:42Z"}
> {"filename":"app/server.go:80","level":"info","msg":"Using cluster scoped
> operator","time":"2021-12-21T08:17:42Z"}
> {"filename":"app/server.go:86","level":"info","msg":"[API Version: v1
> Version: v0.1.0-alpha Git SHA: Not provided. Go Version: go1.13.5 Go OS/Arch:
> linux/amd64]","time":"2021-12-21T08:17:42Z"}
> W1221 08:17:42.318664 1 client_config.go:548] Neither --kubeconfig nor
> --master was specified. Using the inClusterConfig. This might not work.
> {"filename":"tf-operator.v1/main.go:42","level":"info","msg":"Setting up
> client for monitoring on port: 8443","time":"2021-12-21T08:17:42Z"}
> {"filename":"tensorflow/controller.go:120","level":"info","msg":"Creating
> TFJob controller","time":"2021-12-21T08:17:42Z"}
> {"filename":"tensorflow/controller.go:127","level":"info","msg":"Creating Job
> controller","time":"2021-12-21T08:17:42Z"}
> I1221 08:17:42.377440 1 leaderelection.go:187] attempting to acquire
> leader lease default/tf-operator...
> {"filename":"tensorflow/job.go:94","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"TFJob
> experiment-1640073634042-0001 is
> created.","time":"2021-12-21T08:17:42Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> I1221 08:18:08.422211 1 leaderelection.go:196] successfully acquired
> lease default/tf-operator
> {"filename":"tensorflow/controller.go:187","level":"info","msg":"Starting
> TFJob controller","time":"2021-12-21T08:18:08Z"}
> {"filename":"tensorflow/controller.go:190","level":"info","msg":"Waiting for
> informer caches to sync","time":"2021-12-21T08:18:08Z"}
> {"filename":"tensorflow/controller.go:196","level":"info","msg":"Starting 1
> workers","time":"2021-12-21T08:18:08Z"}
> {"filename":"tensorflow/controller.go:202","level":"info","msg":"Started
> workers","time":"2021-12-21T08:18:08Z"}
> {"filename":"tensorflow/controller.go:339","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Reconcile
> TFJobs
> experiment-1640073634042-0001","time":"2021-12-21T08:18:08Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> {"filename":"tensorflow/pod.go:80","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Need
> to create new pod:
> worker-0","replica-type":"worker","time":"2021-12-21T08:18:08Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> {"filename":"tensorflow/controller.go:470","job":"submarine.experiment-1640073634042-0001","level":"warning","msg":"reconcilePods
> error pods \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot
> set blockOwnerDeletion if an ownerReference refers to a resource you can't
> set finalizers on: ,
> \u003cnil\u003e","time":"2021-12-21T08:18:09Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> {"filename":"tensorflow/controller.go:290","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Finished
> syncing tfjob \"submarine/experiment-1640073634042-0001\"
> (859.732664ms)","time":"2021-12-21T08:18:09Z"}
> E1221 08:18:09.383188 1 controller.go:266] error syncing tfjob: pods
> "experiment-1640073634042-0001-worker-0" is forbidden: cannot set
> blockOwnerDeletion if an ownerReference refers to a resource you can't set
> finalizers on: , <nil>
> {"filename":"record/event.go:221","level":"info","msg":"Event(v1.ObjectReference{Kind:\"TFJob\",
> Namespace:\"submarine\", Name:\"experiment-1640073634042-0001\",
> UID:\"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4\", APIVersion:\"kubeflow.org/v1\",
> ResourceVersion:\"287590271\", FieldPath:\"\"}): type: 'Warning' reason:
> 'FailedCreatePod' Error creating: pods
> \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot set
> blockOwnerDeletion if an ownerReference refers to a resource you can't set
> finalizers on: , \u003cnil\u003e","time":"2021-12-21T08:18:09Z"}
> {"filename":"tensorflow/controller.go:339","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Reconcile
> TFJobs
> experiment-1640073634042-0001","time":"2021-12-21T08:18:09Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> {"filename":"tensorflow/pod.go:80","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Need
> to create new pod:
> worker-0","replica-type":"worker","time":"2021-12-21T08:18:09Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> {"filename":"tensorflow/controller.go:470","job":"submarine.experiment-1640073634042-0001","level":"warning","msg":"reconcilePods
> error pods \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot
> set blockOwnerDeletion if an ownerReference refers to a resource you can't
> set finalizers on: ,
> \u003cnil\u003e","time":"2021-12-21T08:18:09Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> {"filename":"tensorflow/controller.go:290","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Finished
> syncing tfjob \"submarine/experiment-1640073634042-0001\"
> (271.610488ms)","time":"2021-12-21T08:18:09Z"}
> E1221 08:18:09.664630 1 controller.go:266] error syncing tfjob: pods
> "experiment-1640073634042-0001-worker-0" is forbidden: cannot set
> blockOwnerDeletion if an ownerReference refers to a resource you can't set
> finalizers on: , <nil>
> {"filename":"record/event.go:221","level":"info","msg":"Event(v1.ObjectReference{Kind:\"TFJob\",
> Namespace:\"submarine\", Name:\"experiment-1640073634042-0001\",
> UID:\"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4\", APIVersion:\"kubeflow.org/v1\",
> ResourceVersion:\"287590271\", FieldPath:\"\"}): type: 'Warning' reason:
> 'FailedCreatePod' Error creating: pods
> \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot set
> blockOwnerDeletion if an ownerReference refers to a resource you can't set
> finalizers on: , \u003cnil\u003e","time":"2021-12-21T08:18:09Z"}
> {"filename":"tensorflow/controller.go:339","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Reconcile
> TFJobs
> experiment-1640073634042-0001","time":"2021-12-21T08:18:09Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> {"filename":"tensorflow/pod.go:80","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Need
> to create new pod:
> worker-0","replica-type":"worker","time":"2021-12-21T08:18:09Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> {"filename":"tensorflow/controller.go:470","job":"submarine.experiment-1640073634042-0001","level":"warning","msg":"reconcilePods
> error pods \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot
> set blockOwnerDeletion if an ownerReference refers to a resource you can't
> set finalizers on: ,
> \u003cnil\u003e","time":"2021-12-21T08:18:09Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> {"filename":"tensorflow/controller.go:290","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Finished
> syncing tfjob \"submarine/experiment-1640073634042-0001\"
> (16.622312ms)","time":"2021-12-21T08:18:09Z"}
> E1221 08:18:09.695125 1 controller.go:266] error syncing tfjob: pods
> "experiment-1640073634042-0001-worker-0" is forbidden: cannot set
> blockOwnerDeletion if an ownerReference refers to a resource you can't set
> finalizers on: , <nil>
> {"filename":"record/event.go:221","level":"info","msg":"Event(v1.ObjectReference{Kind:\"TFJob\",
> Namespace:\"submarine\", Name:\"experiment-1640073634042-0001\",
> UID:\"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4\", APIVersion:\"kubeflow.org/v1\",
> ResourceVersion:\"287590271\", FieldPath:\"\"}): type: 'Warning' reason:
> 'FailedCreatePod' Error creating: pods
> \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot set
> blockOwnerDeletion if an ownerReference refers to a resource you can't set
> finalizers on: , \u003cnil\u003e","time":"2021-12-21T08:18:09Z"}
> {"filename":"tensorflow/controller.go:339","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Reconcile
> TFJobs
> experiment-1640073634042-0001","time":"2021-12-21T08:18:09Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> {"filename":"tensorflow/pod.go:80","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Need
> to create new pod:
> worker-0","replica-type":"worker","time":"2021-12-21T08:18:09Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> {"filename":"tensorflow/controller.go:470","job":"submarine.experiment-1640073634042-0001","level":"warning","msg":"reconcilePods
> error pods \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot
> set blockOwnerDeletion if an ownerReference refers to a resource you can't
> set finalizers on: ,
> \u003cnil\u003e","time":"2021-12-21T08:18:09Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> {"filename":"tensorflow/controller.go:290","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Finished
> syncing tfjob \"submarine/experiment-1640073634042-0001\"
> (20.935691ms)","time":"2021-12-21T08:18:09Z"}
> E1221 08:18:09.742111 1 controller.go:266] error syncing tfjob: pods
> "experiment-1640073634042-0001-worker-0" is forbidden: cannot set
> blockOwnerDeletion if an ownerReference refers to a resource you can't set
> finalizers on: , <nil>
> {"filename":"record/event.go:221","level":"info","msg":"Event(v1.ObjectReference{Kind:\"TFJob\",
> Namespace:\"submarine\", Name:\"experiment-1640073634042-0001\",
> UID:\"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4\", APIVersion:\"kubeflow.org/v1\",
> ResourceVersion:\"287590271\", FieldPath:\"\"}): type: 'Warning' reason:
> 'FailedCreatePod' Error creating: pods
> \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot set
> blockOwnerDeletion if an ownerReference refers to a resource you can't set
> finalizers on: , \u003cnil\u003e","time":"2021-12-21T08:18:09Z"}
> {"filename":"tensorflow/controller.go:339","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Reconcile
> TFJobs
> experiment-1640073634042-0001","time":"2021-12-21T08:18:09Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> {"filename":"tensorflow/pod.go:80","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Need
> to create new pod:
> worker-0","replica-type":"worker","time":"2021-12-21T08:18:09Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> {"filename":"tensorflow/controller.go:470","job":"submarine.experiment-1640073634042-0001","level":"warning","msg":"reconcilePods
> error pods \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot
> set blockOwnerDeletion if an ownerReference refers to a resource you can't
> set finalizers on: ,
> \u003cnil\u003e","time":"2021-12-21T08:18:09Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> {"filename":"tensorflow/controller.go:290","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Finished
> syncing tfjob \"submarine/experiment-1640073634042-0001\"
> (126.441784ms)","time":"2021-12-21T08:18:09Z"}
> E1221 08:18:09.909122 1 controller.go:266] error syncing tfjob: pods
> "experiment-1640073634042-0001-worker-0" is forbidden: cannot set
> blockOwnerDeletion if an ownerReference refers to a resource you can't set
> finalizers on: , <nil>
> {"filename":"record/event.go:221","level":"info","msg":"Event(v1.ObjectReference{Kind:\"TFJob\",
> Namespace:\"submarine\", Name:\"experiment-1640073634042-0001\",
> UID:\"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4\", APIVersion:\"kubeflow.org/v1\",
> ResourceVersion:\"287590271\", FieldPath:\"\"}): type: 'Warning' reason:
> 'FailedCreatePod' Error creating: pods
> \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot set
> blockOwnerDeletion if an ownerReference refers to a resource you can't set
> finalizers on: , \u003cnil\u003e","time":"2021-12-21T08:18:09Z"}
> {"filename":"tensorflow/controller.go:339","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Reconcile
> TFJobs
> experiment-1640073634042-0001","time":"2021-12-21T08:18:09Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> {"filename":"tensorflow/pod.go:80","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Need
> to create new pod:
> worker-0","replica-type":"worker","time":"2021-12-21T08:18:09Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> {"filename":"tensorflow/controller.go:470","job":"submarine.experiment-1640073634042-0001","level":"warning","msg":"reconcilePods
> error pods \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot
> set blockOwnerDeletion if an ownerReference refers to a resource you can't
> set finalizers on: ,
> \u003cnil\u003e","time":"2021-12-21T08:18:10Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> {"filename":"tensorflow/controller.go:290","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Finished
> syncing tfjob \"submarine/experiment-1640073634042-0001\"
> (37.721013ms)","time":"2021-12-21T08:18:10Z"}
> E1221 08:18:10.027525 1 controller.go:266] error syncing tfjob: pods
> "experiment-1640073634042-0001-worker-0" is forbidden: cannot set
> blockOwnerDeletion if an ownerReference refers to a resource you can't set
> finalizers on: , <nil>
> {"filename":"record/event.go:221","level":"info","msg":"Event(v1.ObjectReference{Kind:\"TFJob\",
> Namespace:\"submarine\", Name:\"experiment-1640073634042-0001\",
> UID:\"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4\", APIVersion:\"kubeflow.org/v1\",
> ResourceVersion:\"287590271\", FieldPath:\"\"}): type: 'Warning' reason:
> 'FailedCreatePod' Error creating: pods
> \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot set
> blockOwnerDeletion if an ownerReference refers to a resource you can't set
> finalizers on: , \u003cnil\u003e","time":"2021-12-21T08:18:10Z"}
> {"filename":"tensorflow/controller.go:339","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Reconcile
> TFJobs
> experiment-1640073634042-0001","time":"2021-12-21T08:18:10Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> {"filename":"tensorflow/pod.go:80","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Need
> to create new pod:
> worker-0","replica-type":"worker","time":"2021-12-21T08:18:10Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> {"filename":"tensorflow/controller.go:470","job":"submarine.experiment-1640073634042-0001","level":"warning","msg":"reconcilePods
> error pods \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot
> set blockOwnerDeletion if an ownerReference refers to a resource you can't
> set finalizers on: ,
> \u003cnil\u003e","time":"2021-12-21T08:18:10Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> {"filename":"tensorflow/controller.go:290","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Finished
> syncing tfjob \"submarine/experiment-1640073634042-0001\"
> (36.386638ms)","time":"2021-12-21T08:18:10Z"}
> E1221 08:18:10.224908 1 controller.go:266] error syncing tfjob: pods
> "experiment-1640073634042-0001-worker-0" is forbidden: cannot set
> blockOwnerDeletion if an ownerReference refers to a resource you can't set
> finalizers on: , <nil>
> {"filename":"record/event.go:221","level":"info","msg":"Event(v1.ObjectReference{Kind:\"TFJob\",
> Namespace:\"submarine\", Name:\"experiment-1640073634042-0001\",
> UID:\"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4\", APIVersion:\"kubeflow.org/v1\",
> ResourceVersion:\"287590271\", FieldPath:\"\"}): type: 'Warning' reason:
> 'FailedCreatePod' Error creating: pods
> \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot set
> blockOwnerDeletion if an ownerReference refers to a resource you can't set
> finalizers on: , \u003cnil\u003e","time":"2021-12-21T08:18:10Z"}
> {"filename":"tensorflow/controller.go:339","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Reconcile
> TFJobs
> experiment-1640073634042-0001","time":"2021-12-21T08:18:10Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> {"filename":"tensorflow/pod.go:80","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Need
> to create new pod:
> worker-0","replica-type":"worker","time":"2021-12-21T08:18:10Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> {"filename":"tensorflow/controller.go:470","job":"submarine.experiment-1640073634042-0001","level":"warning","msg":"reconcilePods
> error pods \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot
> set blockOwnerDeletion if an ownerReference refers to a resource you can't
> set finalizers on: ,
> \u003cnil\u003e","time":"2021-12-21T08:18:10Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> {"filename":"tensorflow/controller.go:290","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Finished
> syncing tfjob \"submarine/experiment-1640073634042-0001\"
> (172.821236ms)","time":"2021-12-21T08:18:10Z"}
> E1221 08:18:10.753307 1 controller.go:266] error syncing tfjob: pods
> "experiment-1640073634042-0001-worker-0" is forbidden: cannot set
> blockOwnerDeletion if an ownerReference refers to a resource you can't set
> finalizers on: , <nil>
> {"filename":"record/event.go:221","level":"info","msg":"Event(v1.ObjectReference{Kind:\"TFJob\",
> Namespace:\"submarine\", Name:\"experiment-1640073634042-0001\",
> UID:\"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4\", APIVersion:\"kubeflow.org/v1\",
> ResourceVersion:\"287590271\", FieldPath:\"\"}): type: 'Warning' reason:
> 'FailedCreatePod' Error creating: pods
> \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot set
> blockOwnerDeletion if an ownerReference refers to a resource you can't set
> finalizers on: , \u003cnil\u003e","time":"2021-12-21T08:18:10Z"}
> {"filename":"tensorflow/controller.go:339","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Reconcile
> TFJobs
> experiment-1640073634042-0001","time":"2021-12-21T08:18:11Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> {"filename":"tensorflow/pod.go:80","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Need
> to create new pod:
> worker-0","replica-type":"worker","time":"2021-12-21T08:18:11Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> {"filename":"tensorflow/controller.go:470","job":"submarine.experiment-1640073634042-0001","level":"warning","msg":"reconcilePods
> error pods \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot
> set blockOwnerDeletion if an ownerReference refers to a resource you can't
> set finalizers on: ,
> \u003cnil\u003e","time":"2021-12-21T08:18:11Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> {"filename":"tensorflow/controller.go:290","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Finished
> syncing tfjob \"submarine/experiment-1640073634042-0001\"
> (73.171981ms)","time":"2021-12-21T08:18:11Z"}
> E1221 08:18:11.469567 1 controller.go:266] error syncing tfjob: pods
> "experiment-1640073634042-0001-worker-0" is forbidden: cannot set
> blockOwnerDeletion if an ownerReference refers to a resource you can't set
> finalizers on: , <nil>
> {"filename":"record/event.go:221","level":"info","msg":"Event(v1.ObjectReference{Kind:\"TFJob\",
> Namespace:\"submarine\", Name:\"experiment-1640073634042-0001\",
> UID:\"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4\", APIVersion:\"kubeflow.org/v1\",
> ResourceVersion:\"287590271\", FieldPath:\"\"}): type: 'Warning' reason:
> 'FailedCreatePod' Error creating: pods
> \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot set
> blockOwnerDeletion if an ownerReference refers to a resource you can't set
> finalizers on: , \u003cnil\u003e","time":"2021-12-21T08:18:11Z"}
> {"filename":"tensorflow/job.go:130","level":"info","msg":"Updating tfjob:
> experiment-1640073634042-0001","time":"2021-12-21T08:18:12Z"}
> {"filename":"tensorflow/controller.go:339","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Reconcile
> TFJobs
> experiment-1640073634042-0001","time":"2021-12-21T08:18:12Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> {"filename":"tensorflow/pod.go:80","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Need
> to create new pod:
> worker-0","replica-type":"worker","time":"2021-12-21T08:18:12Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> {"filename":"tensorflow/controller.go:470","job":"submarine.experiment-1640073634042-0001","level":"warning","msg":"reconcilePods
> error pods \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot
> set blockOwnerDeletion if an ownerReference refers to a resource you can't
> set finalizers on: ,
> \u003cnil\u003e","time":"2021-12-21T08:18:12Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> {"filename":"tensorflow/controller.go:290","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Finished
> syncing tfjob \"submarine/experiment-1640073634042-0001\"
> (36.756625ms)","time":"2021-12-21T08:18:12Z"}
> E1221 08:18:12.423556 1 controller.go:266] error syncing tfjob: pods
> "experiment-1640073634042-0001-worker-0" is forbidden: cannot set
> blockOwnerDeletion if an ownerReference refers to a resource you can't set
> finalizers on: , <nil>
> {"filename":"record/event.go:221","level":"info","msg":"Event(v1.ObjectReference{Kind:\"TFJob\",
> Namespace:\"submarine\", Name:\"experiment-1640073634042-0001\",
> UID:\"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4\", APIVersion:\"kubeflow.org/v1\",
> ResourceVersion:\"287590271\", FieldPath:\"\"}): type: 'Warning' reason:
> 'FailedCreatePod' Error creating: pods
> \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot set
> blockOwnerDeletion if an ownerReference refers to a resource you can't set
> finalizers on: , \u003cnil\u003e","time":"2021-12-21T08:18:12Z"}
> {"filename":"tensorflow/controller.go:339","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Reconcile
> TFJobs
> experiment-1640073634042-0001","time":"2021-12-21T08:18:12Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> {"filename":"tensorflow/pod.go:80","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Need
> to create new pod:
> worker-0","replica-type":"worker","time":"2021-12-21T08:18:12Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> {"filename":"tensorflow/controller.go:470","job":"submarine.experiment-1640073634042-0001","level":"warning","msg":"reconcilePods
> error pods \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot
> set blockOwnerDeletion if an ownerReference refers to a resource you can't
> set finalizers on: ,
> \u003cnil\u003e","time":"2021-12-21T08:18:12Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> {"filename":"tensorflow/controller.go:290","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Finished
> syncing tfjob \"submarine/experiment-1640073634042-0001\"
> (24.93148ms)","time":"2021-12-21T08:18:12Z"}
> E1221 08:18:12.785557 1 controller.go:266] error syncing tfjob: pods
> "experiment-1640073634042-0001-worker-0" is forbidden: cannot set
> blockOwnerDeletion if an ownerReference refers to a resource you can't set
> finalizers on: , <nil>
> {"filename":"record/event.go:221","level":"info","msg":"Event(v1.ObjectReference{Kind:\"TFJob\",
> Namespace:\"submarine\", Name:\"experiment-1640073634042-0001\",
> UID:\"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4\", APIVersion:\"kubeflow.org/v1\",
> ResourceVersion:\"287590271\", FieldPath:\"\"}): type: 'Warning' reason:
> 'FailedCreatePod' Error creating: pods
> \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot set
> blockOwnerDeletion if an ownerReference refers to a resource you can't set
> finalizers on: , \u003cnil\u003e","time":"2021-12-21T08:18:12Z"}
> {"filename":"tensorflow/controller.go:339","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Reconcile
> TFJobs
> experiment-1640073634042-0001","time":"2021-12-21T08:18:17Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> {"filename":"tensorflow/pod.go:80","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Need
> to create new pod:
> worker-0","replica-type":"worker","time":"2021-12-21T08:18:17Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> {"filename":"tensorflow/controller.go:470","job":"submarine.experiment-1640073634042-0001","level":"warning","msg":"reconcilePods
> error pods \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot
> set blockOwnerDeletion if an ownerReference refers to a resource you can't
> set finalizers on: ,
> \u003cnil\u003e","time":"2021-12-21T08:18:17Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> {"filename":"tensorflow/controller.go:290","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Finished
> syncing tfjob \"submarine/experiment-1640073634042-0001\"
> (22.903888ms)","time":"2021-12-21T08:18:17Z"}
> E1221 08:18:17.945442 1 controller.go:266] error syncing tfjob: pods
> "experiment-1640073634042-0001-worker-0" is forbidden: cannot set
> blockOwnerDeletion if an ownerReference refers to a resource you can't set
> finalizers on: , <nil>
> {"filename":"record/event.go:221","level":"info","msg":"Event(v1.ObjectReference{Kind:\"TFJob\",
> Namespace:\"submarine\", Name:\"experiment-1640073634042-0001\",
> UID:\"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4\", APIVersion:\"kubeflow.org/v1\",
> ResourceVersion:\"287590271\", FieldPath:\"\"}): type: 'Warning' reason:
> 'FailedCreatePod' Error creating: pods
> \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot set
> blockOwnerDeletion if an ownerReference refers to a resource you can't set
> finalizers on: , \u003cnil\u003e","time":"2021-12-21T08:18:17Z"}
> {"filename":"tensorflow/controller.go:339","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Reconcile
> TFJobs
> experiment-1640073634042-0001","time":"2021-12-21T08:18:28Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> {"filename":"tensorflow/pod.go:80","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Need
> to create new pod:
> worker-0","replica-type":"worker","time":"2021-12-21T08:18:28Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> {"filename":"tensorflow/controller.go:470","job":"submarine.experiment-1640073634042-0001","level":"warning","msg":"reconcilePods
> error pods \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot
> set blockOwnerDeletion if an ownerReference refers to a resource you can't
> set finalizers on: ,
> \u003cnil\u003e","time":"2021-12-21T08:18:28Z","uid":"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4"}
> {"filename":"tensorflow/controller.go:290","job":"submarine.experiment-1640073634042-0001","level":"info","msg":"Finished
> syncing tfjob \"submarine/experiment-1640073634042-0001\"
> (183.401302ms)","time":"2021-12-21T08:18:28Z"}
> E1221 08:18:28.381347 1 controller.go:266] error syncing tfjob: pods
> "experiment-1640073634042-0001-worker-0" is forbidden: cannot set
> blockOwnerDeletion if an ownerReference refers to a resource you can't set
> finalizers on: , <nil>
> {"filename":"record/event.go:221","level":"info","msg":"Event(v1.ObjectReference{Kind:\"TFJob\",
> Namespace:\"submarine\", Name:\"experiment-1640073634042-0001\",
> UID:\"cf89ad4f-bf54-42fa-b8ce-e30fc1c9b3d4\", APIVersion:\"kubeflow.org/v1\",
> ResourceVersion:\"287590271\", FieldPath:\"\"}): type: 'Warning' reason:
> 'FailedCreatePod' Error creating: pods
> \"experiment-1640073634042-0001-worker-0\" is forbidden: cannot set
> blockOwnerDeletion if an ownerReference refers to a resource you can't set
> finalizers on: , \u003cnil\u003e","time":"2021-12-21T08:18:28Z"}
> {code}
> I checked training-operator and search *finalizers*
> https://github.com/kubeflow/training-operator/search?q=finalizers
> !image-2021-12-21-16-42-30-133.png!
> The screenshot should prove that we need to supplement relevant resources
> when RBAC is created.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
