Aaaand it's closed :-) This was a typical case of Layer 7 problem.
I messed up the name of the secret that's used to mount the volume as an actual secret was called ceph-user-secret but the storageclass was referencing ceph-secret-user. *DOH* Be healthy oh mighty devs. Let you well beings no longer be disturbed by silly users. Best regards P. > Od: "Piotr Baranowski" <[email protected]> > Do: "dev" <[email protected]> > Wysłane: sobota, 16 września, 2017 19:38:27 > Temat: OpenShift Origin 3.6 + Ceph 10.2 - Integration fails > Hello dear devs. > I've tried searching for help in users list but no one seems to be interested > or > didn't try my setup before. > Forgive me for taking your precious time but that thing seems like a bug. (Or > i > missed something) > HEre's the setup: > Origin 3.6 installed using ansible-installer. > Three masters, three infra nodes, three app nodes. > It works like a charm. > I also have a running ceph cluster. > I'm trying to integrate them using either access.redhat.com or > docs.openshift.com and I'm halfway there. > - I've created a pool and granted the permissions to a dedicated cephx user. > - I piped the admin secret through base64 as instructed and got the > ceph-secret.yaml > - I created ceph-user-secret with dedicated users' secret in it. (also piped > via > base64) > Created the secret, created second secret, created a dynamicstorageclass. > I _CAN_ create PVC and that creates PV my ceph backend. > I _CAN_ see the resulting rbd images on my ceph storage > (kubernetes-dynamic-pvc-6d23efc6-9b02-11e7-b1fe-525400b47591) > I modified the default project template to reference the ceph-user-secret and > I > _CAN_ see that secret. > All seems to be good. > But. When I try to attach such storage do an existing pod or trying to create > a > new pod with attached storage it fails. > In the logs of the node that is running a pod i can see lots of entries like > this: > Sep 16 19:20:24 node1.osecloud.eu origin-node[25141]: E0916 19:20:24.436094 > 25141 rbd.go:459] failed to get secret from ["foo"/"ceph-secret-user"] > Sep 16 19:20:24 node1.osecloud.eu origin-node[25141]: E0916 19:20:24.436132 > 25141 rbd.go:111] Couldn't get secret from > foo/&LocalObjectReference{Name:ceph-secret-user,} > Sep 16 19:20:24 node1.osecloud.eu origin-node[25141]: E0916 19:20:24.436176 > 25141 reconciler.go:308] operationExecutor.MountVolume failed for volume > "kubernetes.io/rbd/3c06ed1a-9b03-11e7-9af0-525400b47591-pvc-6d1f1461-9b02-11e7-b52b-525400addd4a" > (spec.Name: "pvc-6d1f1461-9b02-11e7-b52b-525400addd4a") pod > "3c06ed1a-9b03-11e7-9af0-525400b47591" (UID: > "3c06ed1a-9b03-11e7-9af0-525400b47591") controllerAttachDetachEnabled: true > with err: MountVolume.NewMounter failed for volume > "kubernetes.io/rbd/3c06ed1a-9b03-11e7-9af0-525400b47591-pvc-6d1f1461-9b02-11e7-b52b-525400addd4a" > (spec.Name: "pvc-6d1f1461-9b02-11e7-b52b-525400addd4a") pod > "3c06ed1a-9b03-11e7-9af0-525400b47591" (UID: > "3c06ed1a-9b03-11e7-9af0-525400b47591") with: failed to get secret from > ["foo"/"ceph-secret-user"] > Sep 16 19:20:24 node1.osecloud.eu origin-node[25141]: E0916 19:20:24.540636 > 25141 rbd.go:459] failed to get secret from ["foo"/"ceph-secret-user"] > Sep 16 19:20:24 node1.osecloud.eu origin-node[25141]: E0916 19:20:24.540665 > 25141 rbd.go:111] Couldn't get secret from > foo/&LocalObjectReference{Name:ceph-secret-user,} > Sep 16 19:20:24 node1.osecloud.eu origin-node[25141]: E0916 19:20:24.540709 > 25141 reconciler.go:308] operationExecutor.MountVolume failed for volume > "kubernetes.io/rbd/3c06ed1a-9b03-11e7-9af0-525400b47591-pvc-6d1f1461-9b02-11e7-b52b-525400addd4a" > (spec.Name: "pvc-6d1f1461-9b02-11e7-b52b-525400addd4a") pod > "3c06ed1a-9b03-11e7-9af0-525400b47591" (UID: > "3c06ed1a-9b03-11e7-9af0-525400b47591") controllerAttachDetachEnabled: true > with err: MountVolume.NewMounter failed for volume > "kubernetes.io/rbd/3c06ed1a-9b03-11e7-9af0-525400b47591-pvc-6d1f1461-9b02-11e7-b52b-525400addd4a" > (spec.Name: "pvc-6d1f1461-9b02-11e7-b52b-525400addd4a") pod > "3c06ed1a-9b03-11e7-9af0-525400b47591" (UID: > "3c06ed1a-9b03-11e7-9af0-525400b47591") with: failed to get secret from > ["foo"/"ceph-secret-user"] > Sep 16 19:20:24 node1.osecloud.eu origin-node[25141]: E0916 19:20:24.644485 > 25141 rbd.go:459] failed to get secret from ["foo"/"ceph-secret-user"] > Sep 16 19:20:24 node1.osecloud.eu origin-node[25141]: E0916 19:20:24.644523 > 25141 rbd.go:111] Couldn't get secret from > foo/&LocalObjectReference{Name:ceph-secret-user,} > Sep 16 19:20:24 node1.osecloud.eu origin-node[25141]: E0916 19:20:24.644583 > 25141 reconciler.go:308] operationExecutor.MountVolume failed for volume > "kubernetes.io/rbd/3c06ed1a-9b03-11e7-9af0-525400b47591-pvc-6d1f1461-9b02-11e7-b52b-525400addd4a" > (spec.Name: "pvc-6d1f1461-9b02-11e7-b52b-525400addd4a") pod > "3c06ed1a-9b03-11e7-9af0-525400b47591" (UID: > "3c06ed1a-9b03-11e7-9af0-525400b47591") controllerAttachDetachEnabled: true > with err: MountVolume.NewMounter failed for volume > "kubernetes.io/rbd/3c06ed1a-9b03-11e7-9af0-525400b47591-pvc-6d1f1461-9b02-11e7-b52b-525400addd4a" > (spec.Name: "pvc-6d1f1461-9b02-11e7-b52b-525400addd4a") pod > "3c06ed1a-9b03-11e7-9af0-525400b47591" (UID: > "3c06ed1a-9b03-11e7-9af0-525400b47591") with: failed to get secret from > ["foo"/"ceph-secret-user"] > And it keeps on flooding the journal with such entries. > What am I doing wrong? > I'm trying to find a problem. Maybe the documentation is not precise enough? > Maybe there is some bug. > Aby help will be very much appreciated. > I'm more than willing to send you any info you need about the setup. > -- > Piotr Baranowski > _______________________________________________ > dev mailing list > [email protected] > http://lists.openshift.redhat.com/openshiftmm/listinfo/dev -- Piotr Baranowski CTO/VP/Chief Instructor@OSEC mob://0048504242337 Dlaczego informatycy mylą Halloween z Bożym narodzeniem? bo 31oct == 25dec
_______________________________________________ dev mailing list [email protected] http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
