Re: [Gluster-users] Disk use with GlusterFS

[Hu Bert]

Hi,

just a guess and easy to test/try: inodes? df -i?

regards,
Hubert

Am Fr., 6. März 2020 um 04:42 Uhr schrieb David Cunningham
:
>
> Hi Aravinda,
>
> That's what was reporting 54% used, at the same time that GlusterFS was 
> giving no space left on device errors. It's a bit worrying that they're not 
> reporting the same thing.
>
> Thank you.
>
>
> On Fri, 6 Mar 2020 at 16:33, Aravinda VK  wrote:
>>
>> Hi David,
>>
>> What is it reporting for brick’s `df` output?
>>
>> ```
>> df /nodirectwritedata/gluster/gvol0
>> ```
>>
>> —
>> regards
>> Aravinda Vishwanathapura
>> https://kadalu.io
>>
>> On 06-Mar-2020, at 2:52 AM, David Cunningham  
>> wrote:
>>
>> Hello,
>>
>> A major concern we have is that "df" was reporting only 54% used and yet 
>> GlusterFS was giving "No space left on device" errors. We rely on "df" to 
>> report the correct result to monitor the system and ensure stability. Does 
>> anyone know what might have been going on here?
>>
>> Thanks in advance.
>>
>>
>> On Thu, 5 Mar 2020 at 21:35, David Cunningham  
>> wrote:
>>>
>>> Hi Aravinda,
>>>
>>> Thanks for the reply. This test server is indeed the master server for 
>>> geo-replication to a slave.
>>>
>>> I'm really surprised that geo-replication simply keeps writing logs until 
>>> all space is consumed, without cleaning them up itself. I didn't see any 
>>> warning about it in the geo-replication install documentation which is 
>>> unfortunate. We'll come up with a solution to delete log files older than 
>>> the LAST_SYNCED time in the geo-replication status. Is anyone aware of any 
>>> other potential gotchas like this?
>>>
>>> Does anyone have an idea why in my previous note some space in the 2GB 
>>> GlusterFS partition apparently went missing? We had 0.47GB of data, 1GB 
>>> reported used by .glusterfs, which even if they were separate files would 
>>> only add up to 1.47GB used, meaning 0.53GB should have been left in the 
>>> partition. If less space is actually being used because of the hard links 
>>> then it's even harder to understand where the other 1.53GB went. So why 
>>> would GlusterFS report "No space left on device"?
>>>
>>> Thanks again for any assistance.
>>>
>>>
>>> On Thu, 5 Mar 2020 at 17:31, Aravinda VK  wrote:

 Hi David,

 Is this Volume is uses Geo-replication? Geo-replication feature enables 
 Changelog to identify the latest changes happening in the GlusterFS volume.

 Content of .glusterfs directory also includes hardlinks to the actual 
 data, so the size shown in .glusterfs is including data. Please refer the 
 comment by Xavi 
 https://github.com/gluster/glusterfs/issues/833#issuecomment-594436009

 If Changelogs files are causing issue, you can use archival tool to remove 
 processed changelogs.
 https://github.com/aravindavk/archive_gluster_changelogs

 —
 regards
 Aravinda Vishwanathapura
 https://kadalu.io


 On 05-Mar-2020, at 9:02 AM, David Cunningham  
 wrote:

 Hello,

 We are looking for some advice on disk use. This is on a single node 
 GlusterFS test server.

 There's a 2GB partition for GlusterFS. Of that, 470MB is used for actual 
 data, and 1GB is used by the .glusterfs directory. The .glusterfs 
 directory is mostly used by the two-character directories and the 
 "changelogs" directory. Why is so much used by .glusterfs, and can we 
 reduce that overhead?

 We also have a problem with this test system where GlusterFS is giving "No 
 space left on device" errors. That's despite "df" reporting only 54% used, 
 and even if we add the 470MB to 1GB used above, that still comes out to 
 less than the 2GB available, so there should be some spare.

 Would anyone be able to advise on these please? Thank you in advance.

 The GlusterFS version is 5.11 and here is the volume information:

 Volume Name: gvol0
 Type: Distribute
 Volume ID: 33ed309b-0e63-4f9a-8132-ab1b0fdcbc36
 Status: Started
 Snapshot Count: 0
 Number of Bricks: 1
 Transport-type: tcp
 Bricks:
 Brick1: myhost:/nodirectwritedata/gluster/gvol0
 Options Reconfigured:
 transport.address-family: inet
 nfs.disable: on
 geo-replication.indexing: on
 geo-replication.ignore-pid-check: on
 changelog.changelog: on

 --
 David Cunningham, Voisonics Limited
 http://voisonics.com/
 USA: +1 213 221 1092
 New Zealand: +64 (0)28 2558 3782
 



 Community Meeting Calendar:

 Schedule -
 Every Tuesday at 14:30 IST / 09:00 UTC
 Bridge: https://bluejeans.com/441850968

 Gluster-users mailing list
 Gluster-users@gluster.org
 https://lists.gluster.org/mailman/listinfo/gluster-users






>>>
>>>
>>> --
>>> David Cunningham, Voisonics Limited
>>> http://voisonics.com/
>>> USA: +1 213 221 1092
>>> New Zealand: +64 

Re: [Gluster-users] What command can check the default value of all options?

[gil han Choi]

Thank you
This is what I was looking for.

> 2020. 3. 6. 12:47, Aravinda VK  작성:
> 
> gluster volume set help?
> 
> —
> regards
> Aravinda Vishwanathapura
> https://kadalu.io
> 
>> On 06-Mar-2020, at 6:39 AM, gil han Choi  wrote:
>> 
>> Hello
>> 
>> I used a command to print out the default values and descriptions of all 
>> options.
>> But I can't remember what command I used and can't find it.
>> What command can check its contents?
>> 
>> Option: cluster.lookup-unhashed
>> Default Value: on
>> Description: This option if set to ON, does a lookup through all the 
>> sub-volumes, in case a lookup didn't return any result from the hash 
>> subvolume. If set to OFF, it does not do a lookup on the remaining 
>> subvolumes.
>> 
>> Option: cluster.lookup-optimize
>> Default Value: on
>> Description: This option if set to ON enables the optimization of -ve 
>> lookups, by not doing a lookup on non-hashed subvolumes for files, in case 
>> the hashed subvolume does not return any result. This option disregards the 
>> lookup-unhashed setting, when enabled.
>> 
>> Option: cluster.min-free-disk
>> Default Value: 10%
>> Description: Percentage/Size of disk space, after which the process starts 
>> balancing out the cluster, and logs will appear in log files
>> 
>> Option: cluster.min-free-inodes
>> Default Value: 5%
>> Description: after system has only N% of inodes, warnings starts to appear 
>> in log files
>> 
>> Option: cluster.rebalance-stats
>> Default Value: off
>> Description: This option if set to ON displays and logs the  time taken for 
>> migration of each file, during the rebalance process. If set to OFF, the 
>> rebalance logs will only display the time spent in each directory.
>> 
>> Option: cluster.subvols-per-directory
>> Default Value: (null)
>> Description: Specifies the directory layout spread. Takes number of 
>> subvolumes as default value.
>> 
>> Option: cluster.readdir-optimize
>> Default Value: off
>> Description: This option if set to ON enables the optimization that allows 
>> DHT to requests non-first subvolumes to filter out directory entries.
>> 
>> Option: cluster.rebal-throttle
>> Default Value: normal
>> Description:  Sets the maximum number of parallel file migrations allowed on 
>> a node during the rebalance operation. The default value is normal and 
>> allows a max of [($(processing units) - 4) / 2), 2]  files to be migrated at 
>> a time. Lazy will allow only one file to be migrated at a time and 
>> aggressive will allow max of [($(processing units) - 4) / 2), 4]
>> 
>> Option: cluster.lock-migration
>> Default Value: off
>> Description:  If enabled this feature will migrate the posix locks 
>> associated with a file during rebalance
>> 
>> Option: cluster.force-migration
>> Default Value: off
>> Description: If disabled, rebalance will not migrate files that are being 
>> written to by an application
>> 
>> Option: cluster.weighted-rebalance
>> Default Value: on
>> Description: When enabled, files will be allocated to bricks with a 
>> probability proportional to their size.  Otherwise, all bricks will have the 
>> same probability (legacy behavior).
>> 
>> Option: cluster.entry-change-log
>> Default Value: on
>> Description: This option exists only for backward compatibility and 
>> configuring it doesn't have any effect
>> 
>> Option: cluster.read-subvolume
>> Default Value: (null)
>> Description: inode-read fops happen only on one of the bricks in replicate. 
>> Afr will prefer the one specified using this option if it is not stale. 
>> Option value must be one of the xlator names of the children. Ex: 
>> -client-0 till -client-
>> 
>> Option: cluster.read-subvolume-index
>> Default Value: -1
>> Description: inode-read fops happen only on one of the bricks in replicate. 
>> AFR will prefer the one specified using this option if it is not stale. 
>> allowed options include -1 till replica-count - 1
>> 
>> Option: cluster.read-hash-mode
>> Default Value: 1
>> Description: inode-read fops happen only on one of the bricks in replicate. 
>> AFR will prefer the one computed using the method specified using this 
>> option.
>> 0 = first readable child of AFR, starting from 1st child.
>> 1 = hash by GFID of file (all clients use same subvolume).
>> 2 = hash by GFID of file and client PID.
>> 3 = brick having the least outstanding read requests.
>> 
>> Option: cluster.background-self-heal-count
>> Default Value: 8
>> Description: This specifies the number of per client self-heal jobs that can 
>> perform parallel heals in the background.
>> 
>> Option: cluster.metadata-self-heal
>> Default Value: off
>> Description: Using this option we can enable/disable metadata i.e. 
>> Permissions, ownerships, xattrs self-heal on the file/directory.
>> 
>> Option: cluster.data-self-heal
>> Default Value: off
>> Description: Using this option we can enable/disable data self-heal on the 
>> file. "open" means data self-heal action will only be triggered by file open 
>> operations.
>> 
>> 

Re: [Gluster-users] Disk use with GlusterFS

[Aravinda VK]

@Amar/Mohit, Do you see any issues with Posix reserve feature?


> On 06-Mar-2020, at 9:11 AM, David Cunningham  
> wrote:
> 
> Hi Aravinda,
> 
> That's what was reporting 54% used, at the same time that GlusterFS was 
> giving no space left on device errors. It's a bit worrying that they're not 
> reporting the same thing.
> 
> Thank you.
> 
> 
> On Fri, 6 Mar 2020 at 16:33, Aravinda VK  > wrote:
> Hi David,
> 
> What is it reporting for brick’s `df` output?
> 
> ```
> df /nodirectwritedata/gluster/gvol0
> ```
> 
> —
> regards
> Aravinda Vishwanathapura
> https://kadalu.io 
> 
>> On 06-Mar-2020, at 2:52 AM, David Cunningham > > wrote:
>> 
>> Hello,
>> 
>> A major concern we have is that "df" was reporting only 54% used and yet 
>> GlusterFS was giving "No space left on device" errors. We rely on "df" to 
>> report the correct result to monitor the system and ensure stability. Does 
>> anyone know what might have been going on here?
>> 
>> Thanks in advance.
>> 
>> 
>> On Thu, 5 Mar 2020 at 21:35, David Cunningham > > wrote:
>> Hi Aravinda,
>> 
>> Thanks for the reply. This test server is indeed the master server for 
>> geo-replication to a slave.
>> 
>> I'm really surprised that geo-replication simply keeps writing logs until 
>> all space is consumed, without cleaning them up itself. I didn't see any 
>> warning about it in the geo-replication install documentation which is 
>> unfortunate. We'll come up with a solution to delete log files older than 
>> the LAST_SYNCED time in the geo-replication status. Is anyone aware of any 
>> other potential gotchas like this?
>> 
>> Does anyone have an idea why in my previous note some space in the 2GB 
>> GlusterFS partition apparently went missing? We had 0.47GB of data, 1GB 
>> reported used by .glusterfs, which even if they were separate files would 
>> only add up to 1.47GB used, meaning 0.53GB should have been left in the 
>> partition. If less space is actually being used because of the hard links 
>> then it's even harder to understand where the other 1.53GB went. So why 
>> would GlusterFS report "No space left on device"?
>> 
>> Thanks again for any assistance.
>> 
>> 
>> On Thu, 5 Mar 2020 at 17:31, Aravinda VK > > wrote:
>> Hi David,
>> 
>> Is this Volume is uses Geo-replication? Geo-replication feature enables 
>> Changelog to identify the latest changes happening in the GlusterFS volume. 
>> 
>> Content of .glusterfs directory also includes hardlinks to the actual data, 
>> so the size shown in .glusterfs is including data. Please refer the comment 
>> by Xavi 
>> https://github.com/gluster/glusterfs/issues/833#issuecomment-594436009 
>> 
>> 
>> If Changelogs files are causing issue, you can use archival tool to remove 
>> processed changelogs.
>> https://github.com/aravindavk/archive_gluster_changelogs 
>> 
>> 
>> —
>> regards
>> Aravinda Vishwanathapura
>> https://kadalu.io 
>> 
>> 
>>> On 05-Mar-2020, at 9:02 AM, David Cunningham >> > wrote:
>>> 
>>> Hello,
>>> 
>>> We are looking for some advice on disk use. This is on a single node 
>>> GlusterFS test server.
>>> 
>>> There's a 2GB partition for GlusterFS. Of that, 470MB is used for actual 
>>> data, and 1GB is used by the .glusterfs directory. The .glusterfs directory 
>>> is mostly used by the two-character directories and the "changelogs" 
>>> directory. Why is so much used by .glusterfs, and can we reduce that 
>>> overhead?
>>> 
>>> We also have a problem with this test system where GlusterFS is giving "No 
>>> space left on device" errors. That's despite "df" reporting only 54% used, 
>>> and even if we add the 470MB to 1GB used above, that still comes out to 
>>> less than the 2GB available, so there should be some spare.
>>> 
>>> Would anyone be able to advise on these please? Thank you in advance.
>>> 
>>> The GlusterFS version is 5.11 and here is the volume information:
>>> 
>>> Volume Name: gvol0
>>> Type: Distribute
>>> Volume ID: 33ed309b-0e63-4f9a-8132-ab1b0fdcbc36
>>> Status: Started
>>> Snapshot Count: 0
>>> Number of Bricks: 1
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: myhost:/nodirectwritedata/gluster/gvol0
>>> Options Reconfigured:
>>> transport.address-family: inet
>>> nfs.disable: on
>>> geo-replication.indexing: on
>>> geo-replication.ignore-pid-check: on
>>> changelog.changelog: on
>>> 
>>> -- 
>>> David Cunningham, Voisonics Limited
>>> http://voisonics.com/ 
>>> USA: +1 213 221 1092
>>> New Zealand: +64 (0)28 2558 3782
>>> 
>>> 
>>> 
>>> 
>>> Community Meeting Calendar:
>>> 
>>> Schedule -
>>> Every Tuesday at 14:30 IST / 09:00 UTC
>>> Bridge: https://bluejeans.com/441850968 
>>> 

Re: [Gluster-users] What command can check the default value of all options?

[Aravinda VK]

gluster volume set help?

—
regards
Aravinda Vishwanathapura
https://kadalu.io

> On 06-Mar-2020, at 6:39 AM, gil han Choi  wrote:
> 
> Hello
> 
> I used a command to print out the default values and descriptions of all 
> options.
> But I can't remember what command I used and can't find it.
> What command can check its contents?
> 
> Option: cluster.lookup-unhashed
> Default Value: on
> Description: This option if set to ON, does a lookup through all the 
> sub-volumes, in case a lookup didn't return any result from the hash 
> subvolume. If set to OFF, it does not do a lookup on the remaining subvolumes.
> 
> Option: cluster.lookup-optimize
> Default Value: on
> Description: This option if set to ON enables the optimization of -ve 
> lookups, by not doing a lookup on non-hashed subvolumes for files, in case 
> the hashed subvolume does not return any result. This option disregards the 
> lookup-unhashed setting, when enabled.
> 
> Option: cluster.min-free-disk
> Default Value: 10%
> Description: Percentage/Size of disk space, after which the process starts 
> balancing out the cluster, and logs will appear in log files
> 
> Option: cluster.min-free-inodes
> Default Value: 5%
> Description: after system has only N% of inodes, warnings starts to appear in 
> log files
> 
> Option: cluster.rebalance-stats
> Default Value: off
> Description: This option if set to ON displays and logs the  time taken for 
> migration of each file, during the rebalance process. If set to OFF, the 
> rebalance logs will only display the time spent in each directory.
> 
> Option: cluster.subvols-per-directory
> Default Value: (null)
> Description: Specifies the directory layout spread. Takes number of 
> subvolumes as default value.
> 
> Option: cluster.readdir-optimize
> Default Value: off
> Description: This option if set to ON enables the optimization that allows 
> DHT to requests non-first subvolumes to filter out directory entries.
> 
> Option: cluster.rebal-throttle
> Default Value: normal
> Description:  Sets the maximum number of parallel file migrations allowed on 
> a node during the rebalance operation. The default value is normal and allows 
> a max of [($(processing units) - 4) / 2), 2]  files to be migrated at a time. 
> Lazy will allow only one file to be migrated at a time and aggressive will 
> allow max of [($(processing units) - 4) / 2), 4]
> 
> Option: cluster.lock-migration
> Default Value: off
> Description:  If enabled this feature will migrate the posix locks associated 
> with a file during rebalance
> 
> Option: cluster.force-migration
> Default Value: off
> Description: If disabled, rebalance will not migrate files that are being 
> written to by an application
> 
> Option: cluster.weighted-rebalance
> Default Value: on
> Description: When enabled, files will be allocated to bricks with a 
> probability proportional to their size.  Otherwise, all bricks will have the 
> same probability (legacy behavior).
> 
> Option: cluster.entry-change-log
> Default Value: on
> Description: This option exists only for backward compatibility and 
> configuring it doesn't have any effect
> 
> Option: cluster.read-subvolume
> Default Value: (null)
> Description: inode-read fops happen only on one of the bricks in replicate. 
> Afr will prefer the one specified using this option if it is not stale. 
> Option value must be one of the xlator names of the children. Ex: 
> -client-0 till -client-
> 
> Option: cluster.read-subvolume-index
> Default Value: -1
> Description: inode-read fops happen only on one of the bricks in replicate. 
> AFR will prefer the one specified using this option if it is not stale. 
> allowed options include -1 till replica-count - 1
> 
> Option: cluster.read-hash-mode
> Default Value: 1
> Description: inode-read fops happen only on one of the bricks in replicate. 
> AFR will prefer the one computed using the method specified using this option.
> 0 = first readable child of AFR, starting from 1st child.
> 1 = hash by GFID of file (all clients use same subvolume).
> 2 = hash by GFID of file and client PID.
> 3 = brick having the least outstanding read requests.
> 
> Option: cluster.background-self-heal-count
> Default Value: 8
> Description: This specifies the number of per client self-heal jobs that can 
> perform parallel heals in the background.
> 
> Option: cluster.metadata-self-heal
> Default Value: off
> Description: Using this option we can enable/disable metadata i.e. 
> Permissions, ownerships, xattrs self-heal on the file/directory.
> 
> Option: cluster.data-self-heal
> Default Value: off
> Description: Using this option we can enable/disable data self-heal on the 
> file. "open" means data self-heal action will only be triggered by file open 
> operations.
> 
> Option: cluster.entry-self-heal
> Default Value: off
> Description: Using this option we can enable/disable entry self-heal on the 
> directory.
> 
> Option: cluster.self-heal-daemon
> Default Value: on
> Description: This 

Re: [Gluster-users] Disk use with GlusterFS

[David Cunningham]

Hi Aravinda,

That's what was reporting 54% used, at the same time that GlusterFS was
giving no space left on device errors. It's a bit worrying that they're not
reporting the same thing.

Thank you.


On Fri, 6 Mar 2020 at 16:33, Aravinda VK  wrote:

> Hi David,
>
> What is it reporting for brick’s `df` output?
>
> ```
> df /nodirectwritedata/gluster/gvol0
> ```
>
> —
> regards
> Aravinda Vishwanathapura
> https://kadalu.io
>
> On 06-Mar-2020, at 2:52 AM, David Cunningham 
> wrote:
>
> Hello,
>
> A major concern we have is that "df" was reporting only 54% used and yet
> GlusterFS was giving "No space left on device" errors. We rely on "df" to
> report the correct result to monitor the system and ensure stability. Does
> anyone know what might have been going on here?
>
> Thanks in advance.
>
>
> On Thu, 5 Mar 2020 at 21:35, David Cunningham 
> wrote:
>
>> Hi Aravinda,
>>
>> Thanks for the reply. This test server is indeed the master server for
>> geo-replication to a slave.
>>
>> I'm really surprised that geo-replication simply keeps writing logs until
>> all space is consumed, without cleaning them up itself. I didn't see any
>> warning about it in the geo-replication install documentation which is
>> unfortunate. We'll come up with a solution to delete log files older than
>> the LAST_SYNCED time in the geo-replication status. Is anyone aware of any
>> other potential gotchas like this?
>>
>> Does anyone have an idea why in my previous note some space in the 2GB
>> GlusterFS partition apparently went missing? We had 0.47GB of data, 1GB
>> reported used by .glusterfs, which even if they were separate files would
>> only add up to 1.47GB used, meaning 0.53GB should have been left in the
>> partition. If less space is actually being used because of the hard links
>> then it's even harder to understand where the other 1.53GB went. So why
>> would GlusterFS report "No space left on device"?
>>
>> Thanks again for any assistance.
>>
>>
>> On Thu, 5 Mar 2020 at 17:31, Aravinda VK  wrote:
>>
>>> Hi David,
>>>
>>> Is this Volume is uses Geo-replication? Geo-replication feature enables
>>> Changelog to identify the latest changes happening in the GlusterFS volume.
>>>
>>> Content of .glusterfs directory also includes hardlinks to the actual
>>> data, so the size shown in .glusterfs is including data. Please refer the
>>> comment by Xavi
>>> https://github.com/gluster/glusterfs/issues/833#issuecomment-594436009
>>>
>>> If Changelogs files are causing issue, you can use archival tool to
>>> remove processed changelogs.
>>> https://github.com/aravindavk/archive_gluster_changelogs
>>>
>>> —
>>> regards
>>> Aravinda Vishwanathapura
>>> https://kadalu.io
>>>
>>>
>>> On 05-Mar-2020, at 9:02 AM, David Cunningham 
>>> wrote:
>>>
>>> Hello,
>>>
>>> We are looking for some advice on disk use. This is on a single node
>>> GlusterFS test server.
>>>
>>> There's a 2GB partition for GlusterFS. Of that, 470MB is used for actual
>>> data, and 1GB is used by the .glusterfs directory. The .glusterfs directory
>>> is mostly used by the two-character directories and the "changelogs"
>>> directory. Why is so much used by .glusterfs, and can we reduce that
>>> overhead?
>>>
>>> We also have a problem with this test system where GlusterFS is giving
>>> "No space left on device" errors. That's despite "df" reporting only 54%
>>> used, and even if we add the 470MB to 1GB used above, that still comes out
>>> to less than the 2GB available, so there should be some spare.
>>>
>>> Would anyone be able to advise on these please? Thank you in advance.
>>>
>>> The GlusterFS version is 5.11 and here is the volume information:
>>>
>>> Volume Name: gvol0
>>> Type: Distribute
>>> Volume ID: 33ed309b-0e63-4f9a-8132-ab1b0fdcbc36
>>> Status: Started
>>> Snapshot Count: 0
>>> Number of Bricks: 1
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: myhost:/nodirectwritedata/gluster/gvol0
>>> Options Reconfigured:
>>> transport.address-family: inet
>>> nfs.disable: on
>>> geo-replication.indexing: on
>>> geo-replication.ignore-pid-check: on
>>> changelog.changelog: on
>>>
>>> --
>>> David Cunningham, Voisonics Limited
>>> http://voisonics.com/
>>> USA: +1 213 221 1092
>>> New Zealand: +64 (0)28 2558 3782
>>> 
>>>
>>>
>>>
>>> Community Meeting Calendar:
>>>
>>> Schedule -
>>> Every Tuesday at 14:30 IST / 09:00 UTC
>>> Bridge: https://bluejeans.com/441850968
>>>
>>> Gluster-users mailing list
>>> Gluster-users@gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>> --
>> David Cunningham, Voisonics Limited
>> http://voisonics.com/
>> USA: +1 213 221 1092
>> New Zealand: +64 (0)28 2558 3782
>>
>
>
> --
> David Cunningham, Voisonics Limited
> http://voisonics.com/
> USA: +1 213 221 1092
> New Zealand: +64 (0)28 2558 3782
>
>
>
>
>
>
>

-- 
David Cunningham, Voisonics Limited
http://voisonics.com/
USA: +1 213 221 1092
New Zealand: +64 (0)28 2558 3782




Community Meeting Calendar:

Schedule -

Re: [Gluster-users] Disk use with GlusterFS

[Aravinda VK]

Hi David,

What is it reporting for brick’s `df` output?

```
df /nodirectwritedata/gluster/gvol0
```

—
regards
Aravinda Vishwanathapura
https://kadalu.io

> On 06-Mar-2020, at 2:52 AM, David Cunningham  
> wrote:
> 
> Hello,
> 
> A major concern we have is that "df" was reporting only 54% used and yet 
> GlusterFS was giving "No space left on device" errors. We rely on "df" to 
> report the correct result to monitor the system and ensure stability. Does 
> anyone know what might have been going on here?
> 
> Thanks in advance.
> 
> 
> On Thu, 5 Mar 2020 at 21:35, David Cunningham  > wrote:
> Hi Aravinda,
> 
> Thanks for the reply. This test server is indeed the master server for 
> geo-replication to a slave.
> 
> I'm really surprised that geo-replication simply keeps writing logs until all 
> space is consumed, without cleaning them up itself. I didn't see any warning 
> about it in the geo-replication install documentation which is unfortunate. 
> We'll come up with a solution to delete log files older than the LAST_SYNCED 
> time in the geo-replication status. Is anyone aware of any other potential 
> gotchas like this?
> 
> Does anyone have an idea why in my previous note some space in the 2GB 
> GlusterFS partition apparently went missing? We had 0.47GB of data, 1GB 
> reported used by .glusterfs, which even if they were separate files would 
> only add up to 1.47GB used, meaning 0.53GB should have been left in the 
> partition. If less space is actually being used because of the hard links 
> then it's even harder to understand where the other 1.53GB went. So why would 
> GlusterFS report "No space left on device"?
> 
> Thanks again for any assistance.
> 
> 
> On Thu, 5 Mar 2020 at 17:31, Aravinda VK  > wrote:
> Hi David,
> 
> Is this Volume is uses Geo-replication? Geo-replication feature enables 
> Changelog to identify the latest changes happening in the GlusterFS volume. 
> 
> Content of .glusterfs directory also includes hardlinks to the actual data, 
> so the size shown in .glusterfs is including data. Please refer the comment 
> by Xavi 
> https://github.com/gluster/glusterfs/issues/833#issuecomment-594436009 
> 
> 
> If Changelogs files are causing issue, you can use archival tool to remove 
> processed changelogs.
> https://github.com/aravindavk/archive_gluster_changelogs 
> 
> 
> —
> regards
> Aravinda Vishwanathapura
> https://kadalu.io 
> 
> 
>> On 05-Mar-2020, at 9:02 AM, David Cunningham > > wrote:
>> 
>> Hello,
>> 
>> We are looking for some advice on disk use. This is on a single node 
>> GlusterFS test server.
>> 
>> There's a 2GB partition for GlusterFS. Of that, 470MB is used for actual 
>> data, and 1GB is used by the .glusterfs directory. The .glusterfs directory 
>> is mostly used by the two-character directories and the "changelogs" 
>> directory. Why is so much used by .glusterfs, and can we reduce that 
>> overhead?
>> 
>> We also have a problem with this test system where GlusterFS is giving "No 
>> space left on device" errors. That's despite "df" reporting only 54% used, 
>> and even if we add the 470MB to 1GB used above, that still comes out to less 
>> than the 2GB available, so there should be some spare.
>> 
>> Would anyone be able to advise on these please? Thank you in advance.
>> 
>> The GlusterFS version is 5.11 and here is the volume information:
>> 
>> Volume Name: gvol0
>> Type: Distribute
>> Volume ID: 33ed309b-0e63-4f9a-8132-ab1b0fdcbc36
>> Status: Started
>> Snapshot Count: 0
>> Number of Bricks: 1
>> Transport-type: tcp
>> Bricks:
>> Brick1: myhost:/nodirectwritedata/gluster/gvol0
>> Options Reconfigured:
>> transport.address-family: inet
>> nfs.disable: on
>> geo-replication.indexing: on
>> geo-replication.ignore-pid-check: on
>> changelog.changelog: on
>> 
>> -- 
>> David Cunningham, Voisonics Limited
>> http://voisonics.com/ 
>> USA: +1 213 221 1092
>> New Zealand: +64 (0)28 2558 3782
>> 
>> 
>> 
>> 
>> Community Meeting Calendar:
>> 
>> Schedule -
>> Every Tuesday at 14:30 IST / 09:00 UTC
>> Bridge: https://bluejeans.com/441850968 
>> 
>> Gluster-users mailing list
>> Gluster-users@gluster.org 
>> https://lists.gluster.org/mailman/listinfo/gluster-users 
>> 
> 
> 
> 
> 
> 
> 
> 
> -- 
> David Cunningham, Voisonics Limited
> http://voisonics.com/ 
> USA: +1 213 221 1092
> New Zealand: +64 (0)28 2558 3782
> 
> 
> -- 
> David Cunningham, Voisonics Limited
> http://voisonics.com/ 
> USA: +1 213 221 1092
> New Zealand: +64 (0)28 2558 3782









Community Meeting Calendar:

Schedule -
Every Tuesday at 14:30 

[Gluster-users] What command can check the default value of all options?

[gil han Choi]

Hello

I used a command to print out the default values and descriptions of all 
options.
But I can't remember what command I used and can't find it.
What command can check its contents?

Option: cluster.lookup-unhashed
Default Value: on
Description: This option if set to ON, does a lookup through all the 
sub-volumes, in case a lookup didn't return any result from the hash subvolume. 
If set to OFF, it does not do a lookup on the remaining subvolumes.

Option: cluster.lookup-optimize
Default Value: on
Description: This option if set to ON enables the optimization of -ve lookups, 
by not doing a lookup on non-hashed subvolumes for files, in case the hashed 
subvolume does not return any result. This option disregards the 
lookup-unhashed setting, when enabled.

Option: cluster.min-free-disk
Default Value: 10%
Description: Percentage/Size of disk space, after which the process starts 
balancing out the cluster, and logs will appear in log files

Option: cluster.min-free-inodes
Default Value: 5%
Description: after system has only N% of inodes, warnings starts to appear in 
log files

Option: cluster.rebalance-stats
Default Value: off
Description: This option if set to ON displays and logs the  time taken for 
migration of each file, during the rebalance process. If set to OFF, the 
rebalance logs will only display the time spent in each directory.

Option: cluster.subvols-per-directory
Default Value: (null)
Description: Specifies the directory layout spread. Takes number of subvolumes 
as default value.

Option: cluster.readdir-optimize
Default Value: off
Description: This option if set to ON enables the optimization that allows DHT 
to requests non-first subvolumes to filter out directory entries.

Option: cluster.rebal-throttle
Default Value: normal
Description:  Sets the maximum number of parallel file migrations allowed on a 
node during the rebalance operation. The default value is normal and allows a 
max of [($(processing units) - 4) / 2), 2]  files to be migrated at a time. 
Lazy will allow only one file to be migrated at a time and aggressive will 
allow max of [($(processing units) - 4) / 2), 4]

Option: cluster.lock-migration
Default Value: off
Description:  If enabled this feature will migrate the posix locks associated 
with a file during rebalance

Option: cluster.force-migration
Default Value: off
Description: If disabled, rebalance will not migrate files that are being 
written to by an application

Option: cluster.weighted-rebalance
Default Value: on
Description: When enabled, files will be allocated to bricks with a probability 
proportional to their size.  Otherwise, all bricks will have the same 
probability (legacy behavior).

Option: cluster.entry-change-log
Default Value: on
Description: This option exists only for backward compatibility and configuring 
it doesn't have any effect

Option: cluster.read-subvolume
Default Value: (null)
Description: inode-read fops happen only on one of the bricks in replicate. Afr 
will prefer the one specified using this option if it is not stale. Option 
value must be one of the xlator names of the children. Ex: -client-0 
till -client-

Option: cluster.read-subvolume-index
Default Value: -1
Description: inode-read fops happen only on one of the bricks in replicate. AFR 
will prefer the one specified using this option if it is not stale. allowed 
options include -1 till replica-count - 1

Option: cluster.read-hash-mode
Default Value: 1
Description: inode-read fops happen only on one of the bricks in replicate. AFR 
will prefer the one computed using the method specified using this option.
0 = first readable child of AFR, starting from 1st child.
1 = hash by GFID of file (all clients use same subvolume).
2 = hash by GFID of file and client PID.
3 = brick having the least outstanding read requests.

Option: cluster.background-self-heal-count
Default Value: 8
Description: This specifies the number of per client self-heal jobs that can 
perform parallel heals in the background.

Option: cluster.metadata-self-heal
Default Value: off
Description: Using this option we can enable/disable metadata i.e. Permissions, 
ownerships, xattrs self-heal on the file/directory.

Option: cluster.data-self-heal
Default Value: off
Description: Using this option we can enable/disable data self-heal on the 
file. "open" means data self-heal action will only be triggered by file open 
operations.

Option: cluster.entry-self-heal
Default Value: off
Description: Using this option we can enable/disable entry self-heal on the 
directory.

Option: cluster.self-heal-daemon
Default Value: on
Description: This option applies to only self-heal-daemon. Index directory 
crawl and automatic healing of files will not be performed if this option is 
turned off.

Option: cluster.heal-timeout
Default Value: 600
Description: time interval for checking the need to self-heal in 
self-heal-daemon

Option: cluster.self-heal-window-size
Default Value: 1
Description: Maximum number blocks per file 

Re: [Gluster-users] Disk use with GlusterFS

[David Cunningham]

Hello,

A major concern we have is that "df" was reporting only 54% used and yet
GlusterFS was giving "No space left on device" errors. We rely on "df" to
report the correct result to monitor the system and ensure stability. Does
anyone know what might have been going on here?

Thanks in advance.


On Thu, 5 Mar 2020 at 21:35, David Cunningham 
wrote:

> Hi Aravinda,
>
> Thanks for the reply. This test server is indeed the master server for
> geo-replication to a slave.
>
> I'm really surprised that geo-replication simply keeps writing logs until
> all space is consumed, without cleaning them up itself. I didn't see any
> warning about it in the geo-replication install documentation which is
> unfortunate. We'll come up with a solution to delete log files older than
> the LAST_SYNCED time in the geo-replication status. Is anyone aware of any
> other potential gotchas like this?
>
> Does anyone have an idea why in my previous note some space in the 2GB
> GlusterFS partition apparently went missing? We had 0.47GB of data, 1GB
> reported used by .glusterfs, which even if they were separate files would
> only add up to 1.47GB used, meaning 0.53GB should have been left in the
> partition. If less space is actually being used because of the hard links
> then it's even harder to understand where the other 1.53GB went. So why
> would GlusterFS report "No space left on device"?
>
> Thanks again for any assistance.
>
>
> On Thu, 5 Mar 2020 at 17:31, Aravinda VK  wrote:
>
>> Hi David,
>>
>> Is this Volume is uses Geo-replication? Geo-replication feature enables
>> Changelog to identify the latest changes happening in the GlusterFS volume.
>>
>> Content of .glusterfs directory also includes hardlinks to the actual
>> data, so the size shown in .glusterfs is including data. Please refer the
>> comment by Xavi
>> https://github.com/gluster/glusterfs/issues/833#issuecomment-594436009
>>
>> If Changelogs files are causing issue, you can use archival tool to
>> remove processed changelogs.
>> https://github.com/aravindavk/archive_gluster_changelogs
>>
>> —
>> regards
>> Aravinda Vishwanathapura
>> https://kadalu.io
>>
>>
>> On 05-Mar-2020, at 9:02 AM, David Cunningham 
>> wrote:
>>
>> Hello,
>>
>> We are looking for some advice on disk use. This is on a single node
>> GlusterFS test server.
>>
>> There's a 2GB partition for GlusterFS. Of that, 470MB is used for actual
>> data, and 1GB is used by the .glusterfs directory. The .glusterfs directory
>> is mostly used by the two-character directories and the "changelogs"
>> directory. Why is so much used by .glusterfs, and can we reduce that
>> overhead?
>>
>> We also have a problem with this test system where GlusterFS is giving
>> "No space left on device" errors. That's despite "df" reporting only 54%
>> used, and even if we add the 470MB to 1GB used above, that still comes out
>> to less than the 2GB available, so there should be some spare.
>>
>> Would anyone be able to advise on these please? Thank you in advance.
>>
>> The GlusterFS version is 5.11 and here is the volume information:
>>
>> Volume Name: gvol0
>> Type: Distribute
>> Volume ID: 33ed309b-0e63-4f9a-8132-ab1b0fdcbc36
>> Status: Started
>> Snapshot Count: 0
>> Number of Bricks: 1
>> Transport-type: tcp
>> Bricks:
>> Brick1: myhost:/nodirectwritedata/gluster/gvol0
>> Options Reconfigured:
>> transport.address-family: inet
>> nfs.disable: on
>> geo-replication.indexing: on
>> geo-replication.ignore-pid-check: on
>> changelog.changelog: on
>>
>> --
>> David Cunningham, Voisonics Limited
>> http://voisonics.com/
>> USA: +1 213 221 1092
>> New Zealand: +64 (0)28 2558 3782
>> 
>>
>>
>>
>> Community Meeting Calendar:
>>
>> Schedule -
>> Every Tuesday at 14:30 IST / 09:00 UTC
>> Bridge: https://bluejeans.com/441850968
>>
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>>
>>
>>
>>
>>
>>
>
> --
> David Cunningham, Voisonics Limited
> http://voisonics.com/
> USA: +1 213 221 1092
> New Zealand: +64 (0)28 2558 3782
>


-- 
David Cunningham, Voisonics Limited
http://voisonics.com/
USA: +1 213 221 1092
New Zealand: +64 (0)28 2558 3782




Community Meeting Calendar:

Schedule -
Every Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] volume add-brick: failed: Pre Validation failed

[DUCARROZ Birgit]

Hi,

Thank you for your message.

Since I had no response within one day, I uninstalled glusterFS on all 
nodes and installed version 7.2


I re-created all volumes using the 4 servers in peer and now I rsync all 
the data from the "old" bricks to the new volumes. This avoids me also 
to do a re-balance.


I know this is not the way you normally should do but at least it works. 
And it is just a backup cluster, so users are not affected.


To respond to your questions:

- Yes, I did peer probe, as said in my first message, the output was 
"Peer in Cluster (Connected)" and the log file was posted.


- No, it is not a heterogeneous cluster. There are 4 identical machines 
with the same OS. Just the glusterFS version was different, since the 2 
new servers were added afterwards.


Anyway, I managed the issue a bit differently and now they are clustered.

Kind regards,
Birgit

On 05/03/20 09:01, Strahil Nikolov wrote:

On March 5, 2020 7:52:48 AM GMT+02:00, Sanju Rakonde  
wrote:

Not sure whether I understood it correctly, are you using an
heterogeneous cluster? If yes, why?

Are the new servers are in part of the cluster? You need to peer probe
the
servers before adding bricks. Copying the configurations of volume will
not
help.

scp -r /var/lib/glusterd/vols/* root@diufnas22:/var/lib/glusterd/vols/
scp -r /var/lib/glusterd/vols/* root@diufnas23:/var/lib/glusterd/vols/

Please paste the peer status output and glusterd logs from peers around
the
time when add-brick is performed.

On Tue, Mar 3, 2020 at 2:56 PM DUCARROZ Birgit

wrote:



Just some additional info:

On diufnas26 and diufnas27:
glusterfs 4.1.8

On diufnas22 and diufnas23:
glusterfs 7.2


setfattr -x --> does not help

On 03/03/20 10:01, DUCARROZ Birgit wrote:

Hi,

Having 2 servers diufnas26 and diufnas27 with 3 volumes

(Type=Distribute):

vol-backup
vol-restore
vol-veeam

diufnas26
/bigdisk/brick1

diufnas27
/bigdisk/brick2


I would like to add 2 new servers: diufnas22 and diufnas23.

diufnas22
/bigdisk/brick3

diufnas23
/bigdisk/brick4

gluster peer status on all servers shows State: Peer in Cluster

(Connected)


Wanna add the 2 new bricks:

root@diufnas26:/bigdisk/brick1# gluster volume add-brick vol-backup
diufnas22:/bigdisk/brick3 diufnas23:/bigdisk/brick4 force
*volume add-brick: failed: Pre Validation failed on diufnas23.

Please

check log file for details.
Pre Validation failed on diufnas22. Please check log file for

details.*




I tried this, but it does not help:
   scp -r /var/lib/glusterd/vols/*

root@diufnas22:/var/lib/glusterd/vols/

   scp -r /var/lib/glusterd/vols/*

root@diufnas23:/var/lib/glusterd/vols/



root@diufnas26:/bigdisk/brick1# tail /var/log/glusterfs/*log -f |

grep E


[2020-03-03 08:37:52.544881]  : volume add-brick vol-backup
diufnas22:/bigdisk/brick3 diufnas23:/bigdisk/brick4 force : FAILED

: Pre

Validation failed on diufnas22. Please check log file for details.
[2020-03-03 08:39:35.991559]  : volume add-brick vol-backup
diufnas22:/bigdisk/brick3 diufnas23:/bigdisk/brick4 force : FAILED

: Pre

Validation failed on diufnas23. Please check log file for details.
[2020-03-03 08:42:40.800060]  : volume add-brick vol-backup
diufnas22:/bigdisk/brick3 diufnas23:/bigdisk/brick4 force : FAILED

: Pre

Validation failed on diufnas23. Please check log file for details.
[2020-03-03 08:50:15.699112]  : volume add-brick vol-backup
diufnas22:/bigdisk/brick3 diufnas23:/bigdisk/brick4 force : FAILED

: Pre

Validation failed on diufnas22. Please check log file for details.
[2020-03-03 08:54:56.472727]  : volume add-brick vol-backup
diufnas22:/bigdisk/brick3 diufnas23:/bigdisk/brick4 force : FAILED

: Pre

Validation failed on diufnas23. Please check log file for details.
[2020-03-03 08:50:15.698286] E [MSGID: 106121]
[glusterd-mgmt.c:1148:glusterd_mgmt_v3_pre_validate] 0-management:

Pre

Validation failed on peers
[2020-03-03 08:50:15.698315] E [MSGID: 106121]
[glusterd-mgmt.c:2242:glusterd_mgmt_v3_initiate_all_phases]
0-management: Pre Validation Failed
[2020-03-03 08:54:56.471628] E [MSGID: 106115]
[glusterd-mgmt.c:124:gd_mgmt_v3_collate_errors] 0-management: Pre
Validation failed on diufnas23. Please check log file for details.
[2020-03-03 08:54:56.471687] E [MSGID: 106115]
[glusterd-mgmt.c:124:gd_mgmt_v3_collate_errors] 0-management: Pre
Validation failed on diufnas22. Please check log file for details.
[2020-03-03 08:54:56.471908] E [MSGID: 106121]
[glusterd-mgmt.c:1148:glusterd_mgmt_v3_pre_validate] 0-management:

Pre

Validation failed on peers
[2020-03-03 08:54:56.471935] E [MSGID: 106121]
[glusterd-mgmt.c:2242:glusterd_mgmt_v3_initiate_all_phases]
0-management: Pre Validation Failed


What am I missing?
Thank you for any help.
Regards, Birgit





Community Meeting Calendar:

Schedule -
Every Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users





Community 

Re: [Gluster-users] Disk use with GlusterFS

[David Cunningham]

Hi Aravinda,

Thanks for the reply. This test server is indeed the master server for
geo-replication to a slave.

I'm really surprised that geo-replication simply keeps writing logs until
all space is consumed, without cleaning them up itself. I didn't see any
warning about it in the geo-replication install documentation which is
unfortunate. We'll come up with a solution to delete log files older than
the LAST_SYNCED time in the geo-replication status. Is anyone aware of any
other potential gotchas like this?

Does anyone have an idea why in my previous note some space in the 2GB
GlusterFS partition apparently went missing? We had 0.47GB of data, 1GB
reported used by .glusterfs, which even if they were separate files would
only add up to 1.47GB used, meaning 0.53GB should have been left in the
partition. If less space is actually being used because of the hard links
then it's even harder to understand where the other 1.53GB went. So why
would GlusterFS report "No space left on device"?

Thanks again for any assistance.


On Thu, 5 Mar 2020 at 17:31, Aravinda VK  wrote:

> Hi David,
>
> Is this Volume is uses Geo-replication? Geo-replication feature enables
> Changelog to identify the latest changes happening in the GlusterFS volume.
>
> Content of .glusterfs directory also includes hardlinks to the actual
> data, so the size shown in .glusterfs is including data. Please refer the
> comment by Xavi
> https://github.com/gluster/glusterfs/issues/833#issuecomment-594436009
>
> If Changelogs files are causing issue, you can use archival tool to remove
> processed changelogs.
> https://github.com/aravindavk/archive_gluster_changelogs
>
> —
> regards
> Aravinda Vishwanathapura
> https://kadalu.io
>
>
> On 05-Mar-2020, at 9:02 AM, David Cunningham 
> wrote:
>
> Hello,
>
> We are looking for some advice on disk use. This is on a single node
> GlusterFS test server.
>
> There's a 2GB partition for GlusterFS. Of that, 470MB is used for actual
> data, and 1GB is used by the .glusterfs directory. The .glusterfs directory
> is mostly used by the two-character directories and the "changelogs"
> directory. Why is so much used by .glusterfs, and can we reduce that
> overhead?
>
> We also have a problem with this test system where GlusterFS is giving "No
> space left on device" errors. That's despite "df" reporting only 54% used,
> and even if we add the 470MB to 1GB used above, that still comes out to
> less than the 2GB available, so there should be some spare.
>
> Would anyone be able to advise on these please? Thank you in advance.
>
> The GlusterFS version is 5.11 and here is the volume information:
>
> Volume Name: gvol0
> Type: Distribute
> Volume ID: 33ed309b-0e63-4f9a-8132-ab1b0fdcbc36
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1
> Transport-type: tcp
> Bricks:
> Brick1: myhost:/nodirectwritedata/gluster/gvol0
> Options Reconfigured:
> transport.address-family: inet
> nfs.disable: on
> geo-replication.indexing: on
> geo-replication.ignore-pid-check: on
> changelog.changelog: on
>
> --
> David Cunningham, Voisonics Limited
> http://voisonics.com/
> USA: +1 213 221 1092
> New Zealand: +64 (0)28 2558 3782
> 
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
>
>
>
>
>
>

-- 
David Cunningham, Voisonics Limited
http://voisonics.com/
USA: +1 213 221 1092
New Zealand: +64 (0)28 2558 3782




Community Meeting Calendar:

Schedule -
Every Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users