Hi, you mentioned that the arbiter bricks run out of inodes.Are you using XFS ?Can you provide the xfs_info of each brick ? Best Regards,Strahil Nikolov On Sat, Jul 1, 2023 at 19:41, Liam Smith<liam.sm...@ek.co> wrote: Hi, We're running a cluster with two data nodes and one arbiter, and have sharding enabled. We had an issue a while back where one of the server's crashed, we got the server back up and running and ensured that all healing entries cleared, and also increased the server spec (CPU/Mem) as this seemed to be the potential cause. Since then however, we've seen some strange behaviour, whereby a lot of 'remove_me' files are building up under `/data/glusterfs/gv1/brick2/brick/.shard/.remove_me/` and `/data/glusterfs/gv1/brick3/brick/.shard/.remove_me/`. This is causing the arbiter to run out of space on brick2 and brick3, as the remove_me files are constantly increasing. brick1 appears to be fine, the disk usage increases throughout the day and drops down in line with the trend of the brick on the data nodes. We see the disk usage increase and drop throughout the day on the data nodes for brick2 and brick3 as well, but while the arbiter follows the same trend of the disk usage increasing, it doesn't drop at any point. This is the output of some gluster commands, occasional heal entries come and go:
root@uk3-prod-gfs-arb-01:~# gluster volume info gv1 Volume Name: gv1Type: Distributed-ReplicateVolume ID: d3d1fdec-7df9-4f71-b9fc-660d12c2a046Status: StartedSnapshot Count: 0Number of Bricks: 3 x (2 + 1) = 9Transport-type: tcpBricks:Brick1: uk1-prod-gfs-01:/data/glusterfs/gv1/brick1/brickBrick2: uk2-prod-gfs-01:/data/glusterfs/gv1/brick1/brickBrick3: uk3-prod-gfs-arb-01:/data/glusterfs/gv1/brick1/brick (arbiter)Brick4: uk1-prod-gfs-01:/data/glusterfs/gv1/brick3/brickBrick5: uk2-prod-gfs-01:/data/glusterfs/gv1/brick3/brickBrick6: uk3-prod-gfs-arb-01:/data/glusterfs/gv1/brick3/brick (arbiter)Brick7: uk1-prod-gfs-01:/data/glusterfs/gv1/brick2/brickBrick8: uk2-prod-gfs-01:/data/glusterfs/gv1/brick2/brickBrick9: uk3-prod-gfs-arb-01:/data/glusterfs/gv1/brick2/brick (arbiter)Options Reconfigured:cluster.entry-self-heal: oncluster.metadata-self-heal: oncluster.data-self-heal: onperformance.client-io-threads: offstorage.fips-mode-rchecksum: ontransport.address-family: inetcluster.lookup-optimize: offperformance.readdir-ahead: offcluster.readdir-optimize: offcluster.self-heal-daemon: enablefeatures.shard: enablefeatures.shard-block-size: 512MBcluster.min-free-disk: 10%cluster.use-anonymous-inode: yes root@uk3-prod-gfs-arb-01:~# gluster peer status Number of Peers: 2 Hostname: uk2-prod-gfs-01Uuid: 2fdfa4a2-195d-4cc5-937c-f48466e76149State: Peer in Cluster (Connected) Hostname: uk1-prod-gfs-01Uuid: 43ec93d1-ad83-4103-aea3-80ded0903d88State: Peer in Cluster (Connected) root@uk3-prod-gfs-arb-01:~# gluster volume heal gv1 info Brick uk1-prod-gfs-01:/data/glusterfs/gv1/brick1/brick<gfid:5b57e1f6-3e3d-4334-a0db-b2560adae6d1>Status: ConnectedNumber of entries: 1 Brick uk2-prod-gfs-01:/data/glusterfs/gv1/brick1/brickStatus: ConnectedNumber of entries: 0 Brick uk3-prod-gfs-arb-01:/data/glusterfs/gv1/brick1/brickStatus: ConnectedNumber of entries: 0 Brick uk1-prod-gfs-01:/data/glusterfs/gv1/brick3/brickStatus: ConnectedNumber of entries: 0 Brick uk2-prod-gfs-01:/data/glusterfs/gv1/brick3/brickStatus: ConnectedNumber of entries: 0 Brick uk3-prod-gfs-arb-01:/data/glusterfs/gv1/brick3/brickStatus: ConnectedNumber of entries: 0 Brick uk1-prod-gfs-01:/data/glusterfs/gv1/brick2/brickStatus: ConnectedNumber of entries: 0 Brick uk2-prod-gfs-01:/data/glusterfs/gv1/brick2/brick<gfid:6ba9c472-9232-4b45-b12f-a1232d6f4627>/.shard/.remove_me<gfid:0f042518-248d-426a-93f4-cfaa92b6ef3e>Status: ConnectedNumber of entries: 3 Brick uk3-prod-gfs-arb-01:/data/glusterfs/gv1/brick2/brick<gfid:6ba9c472-9232-4b45-b12f-a1232d6f4627>/.shard/.remove_me<gfid:0f042518-248d-426a-93f4-cfaa92b6ef3e>Status: ConnectedNumber of entries: 3 root@uk3-prod-gfs-arb-01:~# gluster volume get all cluster.op-versionOption Value------ -----cluster.op-version 100000 We're not sure if this is a potential bug or if something's corrupted that we don't have visibility of, so any pointers/suggestions about how to approach this would be appreciated. Thanks,Liam | | The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure. ________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
