Hello all!
I have setup a cluster with 3 glusterFS nodes for disk persistence just
as specified in the docs. I have configured the inventory file to
install the containerized version to be used by Openshift's integrated
registry. This works fine.
Now I wanted to install the metrics component and I followed the
procedure described here:
https://docs.openshift.org/latest/install_config/persistent_storage/persistent_storage_glusterfs.html#install-example-infra
I end up with openshift-infra project set up, but with 3 pods failing to
start and I think this has to do with the PVC for cassandra that fails
to create.
oc get pvc metrics-cassandra-1 -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
control-plane.alpha.kubernetes.io/leader:
'{"holderIdentity":"8ef584d1-5923-11e8-8730-0a580a830040","leaseDurationSeconds":15,"acquireTime":"2018-05-17T00:38:34Z","renewTime":"2018-05-17T00:55:33Z","leaderTransitions":0}'
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{"volume.beta.kubernetes.io/storage-provisioner":"gluster.org/glusterblock"},"labels":{"metrics-infra":"hawkular-cassandra"},"name":"metrics-cassandra-1","namespace":"openshift-infra"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"6Gi"}},"storageClassName":"glusterfs-registry-block"}}
volume.beta.kubernetes.io/storage-provisioner: gluster.org/glusterblock
creationTimestamp: 2018-05-17T00:38:34Z
labels:
metrics-infra: hawkular-cassandra
name: metrics-cassandra-1
namespace: openshift-infra
resourceVersion: "1204482"
selfLink:
/api/v1/namespaces/openshift-infra/persistentvolumeclaims/metrics-cassandra-1
uid: a18b8c20-596a-11e8-8a63-fa163ed601cb
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 6Gi
storageClassName: glusterfs-registry-block
status:
phase: Pending
oc describe pvc metrics-cassandra-1 shows these warnings:
36m 23m 13 gluster.org/glusterblock
glusterblock-registry-provisioner-dc-1-tljbb
8ef584d1-5923-11e8-8730-0a580a830040 Warning
ProvisioningFailed Failed to provision volume with StorageClass
"glusterfs-registry-block": failed to create volume: [heketi] failed to
create volume: Failed to allocate new block volume: No space
36m 21m 14 gluster.org/glusterblock
glusterblock-registry-provisioner-dc-1-tljbb
8ef584d1-5923-11e8-8730-0a580a830040 Normal Provisioning
External provisioner is provisioning volume for claim
"openshift-infra/metrics-cassandra-1"
21m 21m 1 gluster.org/glusterblock
glusterblock-registry-provisioner-dc-1-tljbb
8ef584d1-5923-11e8-8730-0a580a830040 Warning
ProvisioningFailed Failed to provision volume with StorageClass
"glusterfs-registry-block": failed to create volume: [heketi] failed to
create volume: Post
http://heketi-registry-default.apps.my.net/blockvolumes: dial tcp:
lookup heketi-registry-default.apps.my.net on 192.168.150.16:53: no such
host
In the default project, if I check the logs for heketi-registry, I get a
lot of
[heketi] ERROR 2018/05/17 00:46:47
/src/github.com/heketi/heketi/apps/glusterfs/operations.go:909: Create
Block Volume Build Failed: No space
[negroni] Started POST /blockvolumes
[heketi] INFO 2018/05/17 00:49:02 Loaded simple allocator
[heketi] INFO 2018/05/17 00:49:02 brick_num: 0
[heketi] INFO 2018/05/17 00:49:02 brick_num: 0
[heketi] INFO 2018/05/17 00:49:02 brick_num: 0
[heketi] INFO 2018/05/17 00:49:02 brick_num: 0
[heketi] INFO 2018/05/17 00:49:02 brick_num: 1
[negroni] Completed 500 Internal Server Error in 7.091238ms
For the other glusterFS-related pod, I see the same errors reported by
the pvc creation
oc logs -f glusterblock-registry-provisioner-dc-1-tljbb -n default
I0516 22:38:49.136388 1 controller.go:1167]
scheduleOperation[lock-provision-openshift-infra/metrics-cassandra-1[1191fb8d-5959-11e8-94c9-fa163e1cba7f]]
I0516 22:38:49.166658 1 leaderelection.go:156] attempting to
acquire leader lease...
I0516 22:38:49.197051 1 leaderelection.go:178] successfully
acquired lease to provision for pvc openshift-infra/metrics-cassandra-1
I0516 22:38:49.197122 1 controller.go:1167]
scheduleOperation[provision-openshift-infra/metrics-cassandra-1[1191fb8d-5959-11e8-94c9-fa163e1cba7f]]
E0516 22:38:49.207257 1 glusterblock-provisioner.go:441] BLOCK
VOLUME NAME I RECEIEVED:
E0516 22:38:49.207288 1 glusterblock-provisioner.go:449] BLOCK
VOLUME CREATE REQUEST: &{Size:6 Clusters:[] Name: Hacount:3 Auth:true}
E0516 22:38:49.355122 1 glusterblock-provisioner.go:451] BLOCK
VOLUME RESPONSE: <nil>
E0516 22:38:49.355204 1 glusterblock-provisioner.go:453] [heketi]
failed to create volume: Failed to allocate new block volume: No space
E0516 22:38:49.355262 1 controller.go:895] Failed to provision
volume for claim "openshift-infra/metrics-cassandra-1" with StorageClass
"glusterfs-registry-block": failed to create volume: [heketi] failed to
create volume: Failed to allocate new block volume: No space
E0516 22:38:49.355365 1 goroutinemap.go:165] Operation for
"provision-openshift-infra/metrics-cassandra-1[1191fb8d-5959-11e8-94c9-fa163e1cba7f]"
failed. No retries permitted until 2018-05-16 22:40:51.355301022 +0000
UTC m=+23465.283195247 (durationBeforeRetry 2m2s). Error: "failed to
create volume: [heketi] failed to create volume: Failed to allocate new
block volume: No space"
I0516 22:38:51.241605 1 leaderelection.go:198] stopped trying to
renew lease to provision for pvc openshift-infra/metrics-cassandra-1,
task failed
Regarding the no space message, I am certain that there is space on the
device (if there isn't some glusterFS config that's done on the servers
which prevents them to extend/create the volumes). All disks have the
same 26GB capacity and lvs on one of the machines shows:
LV VG Attr LSize
Pool Origin Data% Meta% Move Log
Cpy%Sync Convert
docker-pool rootvg twi-aot---
<4,16g 52,37 2,62
home rootvg -wi-ao---- 1,00g
root rootvg -wi-ao---- 2,00g
swap rootvg -wi-a----- 2,00g
tmp rootvg -wi-ao---- 1,17g
usr rootvg -wi-ao---- 4,00g
var rootvg -wi-ao---- 4,00g
brick_7aa3a789badd1ae620a2bbefe51b8c73
vg_bd61a1e6f317bb9decade964449c12e8 Vwi-aotz-- 2,00g
tp_7aa3a789badd1ae620a2bbefe51b8c73 0,71
brick_8818ffee7ab2244ca721b7d15ea1e514
vg_bd61a1e6f317bb9decade964449c12e8 Vwi-aotz-- 5,00g
tp_8818ffee7ab2244ca721b7d15ea1e514 7,57
tp_7aa3a789badd1ae620a2bbefe51b8c73
vg_bd61a1e6f317bb9decade964449c12e8 twi-aotz--
2,00g 0,71 0,33
tp_8818ffee7ab2244ca721b7d15ea1e514
vg_bd61a1e6f317bb9decade964449c12e8 twi-aotz--
5,00g 7,57 0,29
Any ideas where to look for misconfigurations?
Thank you,
Dan Pungă
_______________________________________________
users mailing list
[email protected]
http://lists.openshift.redhat.com/openshiftmm/listinfo/users
