Hello Rodrigo, I appreciate your answer!
In the meantime I had reached for the heketi-cli related support(chat)
and I got the same reference. There's a config map generated by the
installer for the heketi-registry pod that has the default for
block-hosting volumes size set at 100GB.
What I thought was that the "block hosting volume" would be an
equivalent of a logical volume and it(heketi-cli) tries to create a lv
of size 100GB inside the already created
vg_bd61a1e6f317bb9decade964449c12e8(which has 26GB).
I've actually modified the encrypted json config and tried to restart
the heketi-registry pod, which failed. So I ended up with some unmanaged
glusterFS storage, but since I'm on a test envionment, it's fine.
Otherwise, good to know for the future.
Now what I also don't understand is how did the initial volume group for
the registry got created with just 26GB of storage if the default is for
100GB? Is there a rule such as: "create block-hosting volume of default
size=100GB or max available"?
The integrated registry's persistence is set to 5GB. This is, I believe,
a default value, as I haven't set anything related to it in my inventory
file when installing Openshift Origin. How can I use the remaining
storage in my vg with glusterFS and Openshift?
Thank you!
On 19.05.2018 02:43, Rodrigo Bersa wrote:
Hi Dan,
The Gluster Block volumes works with the concept of block-hosting
volume, and these ones are created with 100GB by default.
To clarify, the block volumes will be provisioned over the block
hosting volumes.
Let's say you need a 10GB block volume, it will create a block hosting
volume with 100GB and then the 10GB block volume over it, as the next
block volumes requested until it reaches the 100GB. After that a new
block hosting volume will be created and so on.
So, if you have just 26GB available in each server, it's not enough to
create the block hosting volume. You may need to add more devices to
your CNS Cluster to grow your free space.
Kind regards,
Rodrigo Bersa
Cloud Consultant, RHCVA, RHCE
Red Hat Brasil <https://www.redhat.com>
[email protected] <mailto:[email protected]> M: +55-11-99557-5841
<tel:+55-11-99557-5841>
<https://red.ht/sig>
TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
Red Hat é reconhecida entre as melhores empresas para trabalhar no
Brasil pelo *Great Place to Work*.
On Wed, May 16, 2018 at 10:35 PM, Dan Pungă <[email protected]
<mailto:[email protected]>> wrote:
Hello all!
I have setup a cluster with 3 glusterFS nodes for disk persistence
just as specified in the docs. I have configured the inventory
file to install the containerized version to be used by
Openshift's integrated registry. This works fine.
Now I wanted to install the metrics component and I followed the
procedure described here:
https://docs.openshift.org/latest/install_config/persistent_storage/persistent_storage_glusterfs.html#install-example-infra
<https://docs.openshift.org/latest/install_config/persistent_storage/persistent_storage_glusterfs.html#install-example-infra>
I end up with openshift-infra project set up, but with 3 pods
failing to start and I think this has to do with the PVC for
cassandra that fails to create.
oc get pvc metrics-cassandra-1 -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
control-plane.alpha.kubernetes.io/leader
<http://control-plane.alpha.kubernetes.io/leader>:
'{"holderIdentity":"8ef584d1-5923-11e8-8730-0a580a830040","leaseDurationSeconds":15,"acquireTime":"2018-05-17T00:38:34Z","renewTime":"2018-05-17T00:55:33Z","leaderTransitions":0}'
kubectl.kubernetes.io/last-applied-configuration
<http://kubectl.kubernetes.io/last-applied-configuration>: |
{"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{"volume.beta.kubernetes.io/storage-provisioner
<http://volume.beta.kubernetes.io/storage-provisioner>":"gluster.org/glusterblock
<http://gluster.org/glusterblock>"},"labels":{"metrics-infra":"hawkular-cassandra"},"name":"metrics-cassandra-1","namespace":"openshift-infra"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"6Gi"}},"storageClassName":"glusterfs-registry-block"}}
volume.beta.kubernetes.io/storage-provisioner
<http://volume.beta.kubernetes.io/storage-provisioner>:
gluster.org/glusterblock <http://gluster.org/glusterblock>
creationTimestamp: 2018-05-17T00:38:34Z
labels:
metrics-infra: hawkular-cassandra
name: metrics-cassandra-1
namespace: openshift-infra
resourceVersion: "1204482"
selfLink:
/api/v1/namespaces/openshift-infra/persistentvolumeclaims/metrics-cassandra-1
uid: a18b8c20-596a-11e8-8a63-fa163ed601cb
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 6Gi
storageClassName: glusterfs-registry-block
status:
phase: Pending
oc describe pvc metrics-cassandra-1 shows these warnings:
36m 23m 13 gluster.org/glusterblock
<http://gluster.org/glusterblock>
glusterblock-registry-provisioner-dc-1-tljbb
8ef584d1-5923-11e8-8730-0a580a830040 Warning
ProvisioningFailed Failed to provision volume with StorageClass
"glusterfs-registry-block": failed to create volume: [heketi]
failed to create volume: Failed to allocate new block volume: No space
36m 21m 14 gluster.org/glusterblock
<http://gluster.org/glusterblock>
glusterblock-registry-provisioner-dc-1-tljbb
8ef584d1-5923-11e8-8730-0a580a830040 Normal Provisioning
External provisioner is provisioning volume for claim
"openshift-infra/metrics-cassandra-1"
21m 21m 1 gluster.org/glusterblock
<http://gluster.org/glusterblock>
glusterblock-registry-provisioner-dc-1-tljbb
8ef584d1-5923-11e8-8730-0a580a830040 Warning
ProvisioningFailed Failed to provision volume with StorageClass
"glusterfs-registry-block": failed to create volume: [heketi]
failed to create volume: Post
http://heketi-registry-default.apps.my.net/blockvolumes
<http://heketi-registry-default.apps.my.net/blockvolumes>: dial
tcp: lookup heketi-registry-default.apps.my.net
<http://heketi-registry-default.apps.my.net> on 192.168.150.16:53
<http://192.168.150.16:53>: no such host
In the default project, if I check the logs for heketi-registry, I
get a lot of
[heketi] ERROR 2018/05/17 00:46:47
/src/github.com/heketi/heketi/apps/glusterfs/operations.go:909
<http://github.com/heketi/heketi/apps/glusterfs/operations.go:909>:
Create Block Volume Build Failed: No space
[negroni] Started POST /blockvolumes
[heketi] INFO 2018/05/17 00:49:02 Loaded simple allocator
[heketi] INFO 2018/05/17 00:49:02 brick_num: 0
[heketi] INFO 2018/05/17 00:49:02 brick_num: 0
[heketi] INFO 2018/05/17 00:49:02 brick_num: 0
[heketi] INFO 2018/05/17 00:49:02 brick_num: 0
[heketi] INFO 2018/05/17 00:49:02 brick_num: 1
[negroni] Completed 500 Internal Server Error in 7.091238ms
For the other glusterFS-related pod, I see the same errors
reported by the pvc creation
oc logs -f glusterblock-registry-provisioner-dc-1-tljbb -n default
I0516 22:38:49.136388 1 controller.go:1167]
scheduleOperation[lock-provision-openshift-infra/metrics-cassandra-1[1191fb8d-5959-11e8-94c9-fa163e1cba7f]]
I0516 22:38:49.166658 1 leaderelection.go:156] attempting to
acquire leader lease...
I0516 22:38:49.197051 1 leaderelection.go:178] successfully
acquired lease to provision for pvc
openshift-infra/metrics-cassandra-1
I0516 22:38:49.197122 1 controller.go:1167]
scheduleOperation[provision-openshift-infra/metrics-cassandra-1[1191fb8d-5959-11e8-94c9-fa163e1cba7f]]
E0516 22:38:49.207257 1 glusterblock-provisioner.go:441]
BLOCK VOLUME NAME I RECEIEVED:
E0516 22:38:49.207288 1 glusterblock-provisioner.go:449]
BLOCK VOLUME CREATE REQUEST: &{Size:6 Clusters:[] Name: Hacount:3
Auth:true}
E0516 22:38:49.355122 1 glusterblock-provisioner.go:451]
BLOCK VOLUME RESPONSE: <nil>
E0516 22:38:49.355204 1 glusterblock-provisioner.go:453]
[heketi] failed to create volume: Failed to allocate new block
volume: No space
E0516 22:38:49.355262 1 controller.go:895] Failed to
provision volume for claim "openshift-infra/metrics-cassandra-1"
with StorageClass "glusterfs-registry-block": failed to create
volume: [heketi] failed to create volume: Failed to allocate new
block volume: No space
E0516 22:38:49.355365 1 goroutinemap.go:165] Operation for
"provision-openshift-infra/metrics-cassandra-1[1191fb8d-5959-11e8-94c9-fa163e1cba7f]"
failed. No retries permitted until 2018-05-16 22:40:51.355301022
+0000 UTC m=+23465.283195247 (durationBeforeRetry 2m2s). Error:
"failed to create volume: [heketi] failed to create volume: Failed
to allocate new block volume: No space"
I0516 22:38:51.241605 1 leaderelection.go:198] stopped
trying to renew lease to provision for pvc
openshift-infra/metrics-cassandra-1, task failed
Regarding the no space message, I am certain that there is space
on the device (if there isn't some glusterFS config that's done on
the servers which prevents them to extend/create the volumes). All
disks have the same 26GB capacity and lvs on one of the machines
shows:
LV VG Attr LSize Pool Origin Data% Meta% Move Log
Cpy%Sync Convert
docker-pool rootvg twi-aot--- <4,16g 52,37 2,62
home rootvg -wi-ao---- 1,00g
root rootvg -wi-ao---- 2,00g
swap rootvg -wi-a----- 2,00g
tmp rootvg -wi-ao---- 1,17g
usr rootvg -wi-ao---- 4,00g
var rootvg -wi-ao---- 4,00g
brick_7aa3a789badd1ae620a2bbefe51b8c73
vg_bd61a1e6f317bb9decade964449c12e8 Vwi-aotz-- 2,00g
tp_7aa3a789badd1ae620a2bbefe51b8c73 0,71
brick_8818ffee7ab2244ca721b7d15ea1e514
vg_bd61a1e6f317bb9decade964449c12e8 Vwi-aotz-- 5,00g
tp_8818ffee7ab2244ca721b7d15ea1e514 7,57
tp_7aa3a789badd1ae620a2bbefe51b8c73
vg_bd61a1e6f317bb9decade964449c12e8 twi-aotz-- 2,00g 0,71 0,33
tp_8818ffee7ab2244ca721b7d15ea1e514
vg_bd61a1e6f317bb9decade964449c12e8 twi-aotz-- 5,00g 7,57 0,29
Any ideas where to look for misconfigurations?
Thank you,
Dan Pungă
_______________________________________________
users mailing list
[email protected]
<mailto:[email protected]>
http://lists.openshift.redhat.com/openshiftmm/listinfo/users
<http://lists.openshift.redhat.com/openshiftmm/listinfo/users>
_______________________________________________
users mailing list
[email protected]
http://lists.openshift.redhat.com/openshiftmm/listinfo/users
