subject:"\[Gluster\-users\] df reports wrong full capacity for distributed volumes \(Glusterfs 3.12.6\-1\)"

Re: [Gluster-users] df reports wrong full capacity for distributed volumes (Glusterfs 3.12.6-1)

2018-03-01 Thread Nithya Balachandran

On 1 March 2018 at 15:25, Jose V. Carrión  wrote:

> I'm sorry for my last incomplete message.
>
> Below the output of both volumes:
>
> [root@stor1t ~]# gluster volume rebalance volumedisk1 status
> Node Rebalanced-files  size
> scanned  failures   skipped   status  run time in
> h:m:s
>-  ---   ---
> ---   ---   --- 
> --
>localhost   703964 16384.0PB
> 1475983 0 0completed   64:37:55
>stor2data   704610 16384.0PB
> 1475199 0 0completed   64:31:30
>stor3data   703964 16384.0PB
> 1475983 0 0completed   64:37:55
> volume rebalance: volumedisk1: success
>
> [root@stor1 ~]# gluster volume rebalance volumedisk0 status
> Node Rebalanced-files  size
> scanned  failures   skipped   status  run time in
> h:m:s
>-  ---   ---
> ---   ---   --- 
> --
>localhost   411919 1.1GB
>  718044 0 0completed2:28:52
>stor2data   435340 16384.0PB
>  741287 0 0completed2:26:01
>stor3data   411919 1.1GB
>  718044 0 0completed2:28:52
> volume rebalance: volumedisk0: success
>
> And  volumedisk1 rebalance logs finished saying:
> [2018-02-13 03:47:48.703311] I [MSGID: 109028]
> [dht-rebalance.c:5053:gf_defrag_status_get] 0-volumedisk1-dht: Rebalance
> is completed. Time taken is 232675.00 secs
> [2018-02-13 03:47:48.703351] I [MSGID: 109028]
> [dht-rebalance.c:5057:gf_defrag_status_get] 0-volumedisk1-dht: Files
> migrated: 703964, size: 14046969178073, lookups: 1475983, failures: 0,
> skipped: 0
>
> Checking my logs the new stor3node and the rebalance task was executed on
>  2018-02-10 . From this date to now I have been storing new files.
> The exact sequence of commands to add the new node was:
>
> gluster peer probe stor3data
>
> gluster volume add-brick volumedisk0 stor3data:/mnt/disk_b1/glusterfs/vol0
>
> gluster volume add-brick volumedisk0 stor3data:/mnt/disk_b2/glusterfs/vol0
>
> gluster volume add-brick volumedisk1 stor3data:/mnt/disk_c/glusterfs/vol1
>
> gluster volume add-brick volumedisk1 stor3data:/mnt/disk_d/glusterfs/vol1
>
> gluster volume rebalance volumedisk0 start force
>
> gluster volume rebalance volumedisk1 start force
>
> For some reason , could be unbalanced the assigned range of DHT for
> stor3data bricks ? Could be minor than stor1data and stor2data ? ,
>
> Any way to verify it ?
>
> Any way to modify/rebalance the DHT range between bricks  in order to
> unify the DHT range per brick ?.
>
> Thanks a lot,
>
> Greetings.
>
> Jose V.
>
>
> 2018-03-01 10:39 GMT+01:00 Jose V. Carrión :
>
>> Hi Nithya,
>> Below the output of both volumes:
>>
>> [root@stor1t ~]# gluster volume rebalance volumedisk1 status
>> Node Rebalanced-files  size
>> scanned  failures   skipped   status  run time in
>> h:m:s
>>-  ---   ---
>> ---   ---   --- 
>> --
>>localhost   703964 16384.0PB
>> 1475983 0 0completed   64:37:55
>>stor2data   704610 16384.0PB
>> 1475199 0 0completed   64:31:30
>>stor3data   703964 16384.0PB
>> 1475983 0 0completed   64:37:55
>> volume rebalance: volumedisk1: success
>>
>> [root@stor1 ~]# gluster volume rebalance volumedisk0 status
>> Node Rebalanced-files  size
>> scanned  failures   skipped   status  run time in
>> h:m:s
>>-  ---   ---
>> ---   ---   --- 
>> --
>>localhost   411919 1.1GB
>>  718044 0 0completed2:28:52
>>stor2data   435340 16384.0PB
>>  741287 0 0completed2:26:01
>>stor3data

Re: [Gluster-users] df reports wrong full capacity for distributed volumes (Glusterfs 3.12.6-1)

2018-03-01 Thread Jose V . Carrión

I'm sorry for my last incomplete message.

Below the output of both volumes:

[root@stor1t ~]# gluster volume rebalance volumedisk1 status
Node Rebalanced-files  size
  scanned  failures   skipped   status  run time in
h:m:s
   -  ---   ---
---   ---   --- 
--
   localhost   703964 16384.0PB
  1475983 0 0completed   64:37:55
   stor2data   704610 16384.0PB
1475199 0 0completed   64:31:30
   stor3data   703964 16384.0PB
1475983 0 0completed   64:37:55
volume rebalance: volumedisk1: success

[root@stor1 ~]# gluster volume rebalance volumedisk0 status
Node Rebalanced-files  size
  scanned  failures   skipped   status  run time in
h:m:s
   -  ---   ---
---   ---   --- 
--
   localhost   411919 1.1GB
   718044 0 0completed2:28:52
   stor2data   435340 16384.0PB
 741287 0 0completed2:26:01
   stor3data   411919 1.1GB
 718044 0 0completed2:28:52
volume rebalance: volumedisk0: success

And  volumedisk1 rebalance logs finished saying:
[2018-02-13 03:47:48.703311] I [MSGID: 109028]
[dht-rebalance.c:5053:gf_defrag_status_get]
0-volumedisk1-dht: Rebalance is completed. Time taken is 232675.00 secs
[2018-02-13 03:47:48.703351] I [MSGID: 109028]
[dht-rebalance.c:5057:gf_defrag_status_get]
0-volumedisk1-dht: Files migrated: 703964, size: 14046969178073, lookups:
1475983, failures: 0, skipped: 0

Checking my logs the new stor3node and the rebalance task was executed on
 2018-02-10 . From this date to now I have been storing new files.
The exact sequence of commands to add the new node was:

gluster peer probe stor3data

gluster volume add-brick volumedisk0 stor3data:/mnt/disk_b1/glusterfs/vol0

gluster volume add-brick volumedisk0 stor3data:/mnt/disk_b2/glusterfs/vol0

gluster volume add-brick volumedisk1 stor3data:/mnt/disk_c/glusterfs/vol1

gluster volume add-brick volumedisk1 stor3data:/mnt/disk_d/glusterfs/vol1

gluster volume rebalance volumedisk0 start force

gluster volume rebalance volumedisk1 start force

For some reason , could be unbalanced the assigned range of DHT for
stor3data bricks ? Could be minor than stor1data and stor2data ? ,

Any way to verify it ?

Any way to modify/rebalance the DHT range between bricks  in order to unify
the DHT range per brick ?.

Thanks a lot,

Greetings.

Jose V.


2018-03-01 10:39 GMT+01:00 Jose V. Carrión :

> Hi Nithya,
> Below the output of both volumes:
>
> [root@stor1t ~]# gluster volume rebalance volumedisk1 status
> Node Rebalanced-files  size
> scanned  failures   skipped   status  run time in
> h:m:s
>-  ---   ---
> ---   ---   --- 
> --
>localhost   703964 16384.0PB
> 1475983 0 0completed   64:37:55
>stor2data   704610 16384.0PB
> 1475199 0 0completed   64:31:30
>stor3data   703964 16384.0PB
> 1475983 0 0completed   64:37:55
> volume rebalance: volumedisk1: success
>
> [root@stor1 ~]# gluster volume rebalance volumedisk0 status
> Node Rebalanced-files  size
> scanned  failures   skipped   status  run time in
> h:m:s
>-  ---   ---
> ---   ---   --- 
> --
>localhost   411919 1.1GB
>  718044 0 0completed2:28:52
>stor2data   435340 16384.0PB
>  741287 0 0completed2:26:01
>stor3data   411919 1.1GB
>  718044 0 0completed2:28:52
> volume rebalance: volumedisk0: success
>
> And  volumedisk1 rebalance logs finished saying:
> [2018-02-13 03:47:48.703311] I [MSGID:

Re: [Gluster-users] df reports wrong full capacity for distributed volumes (Glusterfs 3.12.6-1)

2018-03-01 Thread Jose V . Carrión

Hi Nithya,
Below the output of both volumes:

[root@stor1t ~]# gluster volume rebalance volumedisk1 status
Node Rebalanced-files  size
  scanned  failures   skipped   status  run time in
h:m:s
   -  ---   ---
---   ---   --- 
--
   localhost   703964 16384.0PB
  1475983 0 0completed   64:37:55
   stor2data   704610 16384.0PB
1475199 0 0completed   64:31:30
   stor3data   703964 16384.0PB
1475983 0 0completed   64:37:55
volume rebalance: volumedisk1: success

[root@stor1 ~]# gluster volume rebalance volumedisk0 status
Node Rebalanced-files  size
  scanned  failures   skipped   status  run time in
h:m:s
   -  ---   ---
---   ---   --- 
--
   localhost   411919 1.1GB
   718044 0 0completed2:28:52
   stor2data   435340 16384.0PB
 741287 0 0completed2:26:01
   stor3data   411919 1.1GB
 718044 0 0completed2:28:52
volume rebalance: volumedisk0: success

And  volumedisk1 rebalance logs finished saying:
[2018-02-13 03:47:48.703311] I [MSGID: 109028]
[dht-rebalance.c:5053:gf_defrag_status_get] 0-volumedisk1-dht: Rebalance is
completed. Time taken is 232675.00 secs
[2018-02-13 03:47:48.703351] I [MSGID: 109028]
[dht-rebalance.c:5057:gf_defrag_status_get] 0-volumedisk1-dht: Files
migrated: 703964, size: 14046969178073, lookups: 1475983, failures: 0,
skipped: 0

Checking my logs the new stor3node and the rebalance task was executed on
 2018-02-10 . From this date to now I have been storing new files.
The sequence of commands to add the node was:

gluster peer probe stor3data

gluster volume add-brick volumedisk0 stor3data:/mnt/disk_b1/glusterfs/vol0

gluster volume add-brick volumedisk0 stor3data:/mnt/disk_b1/glusterfs/vol0





2018-03-01 6:32 GMT+01:00 Nithya Balachandran :

> Hi Jose,
>
> On 28 February 2018 at 22:31, Jose V. Carrión  wrote:
>
>> Hi Nithya,
>>
>> My initial setup was composed of 2 similar nodes: stor1data and
>> stor2data. A month ago I expanded both volumes with a new node: stor3data
>> (2 bricks per volume).
>> Of course, then to add the new peer with the bricks I did the 'balance
>> force' operation. This task finished successfully (you can see info below)
>> and number of files on the 3 nodes were very similar .
>>
>> For volumedisk1 I only have files of 500MB and they are continuosly
>> written in sequential mode. The filename pattern of written files is:
>>
>> run.node1..rd
>> run.node2..rd
>> run.node1.0001.rd
>> run.node2.0001.rd
>> run.node1.0002.rd
>> run.node2.0002.rd
>> ...
>> ...
>> run.node1.X.rd
>> run.node2.X.rd
>>
>> (  X ranging from  to infinite )
>>
>> Curiously stor1data and stor2data maintain similar ratios in bytes:
>>
>> Filesystem  1K-blocksUsed   Available
>> Use% Mounted on
>> /dev/sdc1 52737613824 17079174264 <(707)%20917-4264>
>>  35658439560  33% /mnt/glusterfs/vol1   -> stor1data
>> /dev/sdc1 52737613824 17118810848  35618802976  33%
>> /mnt/glusterfs/vol1  ->  stor2data
>>
>> However the ratio on som3data differs too much (1TB):
>> Filesystem   1K-blocksUsedAvailable
>> Use% Mounted on
>> /dev/sdc1 52737613824 15479191748  37258422076  30%
>> /mnt/disk_c/glusterfs/vol1 -> stor3data
>> /dev/sdd1 52737613824 15566398604  37171215220  30%
>> /mnt/disk_d/glusterfs/vol1 -> stor3data
>>
>> Thinking in  inodes:
>>
>> FilesystemInodes   IUsed   IFree  IUse%
>> Mounted on
>> /dev/sdc1 5273970048  851053  52731189951%
>> /mnt/glusterfs/vol1 ->  stor1data
>> /dev/sdc1 5273970048  849388  52731206601%
>> /mnt/glusterfs/vol1 ->  stor2data
>>
>> /dev/sdc1 5273970048  846877  52731231711%
>> /mnt/disk_c/glusterfs/vol1 -> stor3data
>> /dev/sdd1 5273970048  845250  52731247981%
>> /mnt/disk_d/glusterfs/vol1 -> stor3data
>>
>> 851053 (stor1) - 845250 (stor3) = 5803 files of difference !
>>
>
> The inode numbers are a little misleading here - gluster uses some to
> create its own internal files and directory structures. Based on the
> average file size, I think this would actually work out to a

Re: [Gluster-users] df reports wrong full capacity for distributed volumes (Glusterfs 3.12.6-1)

2018-02-28 Thread Nithya Balachandran

Hi Jose,

On 28 February 2018 at 22:31, Jose V. Carrión  wrote:

> Hi Nithya,
>
> My initial setup was composed of 2 similar nodes: stor1data and stor2data.
> A month ago I expanded both volumes with a new node: stor3data (2 bricks
> per volume).
> Of course, then to add the new peer with the bricks I did the 'balance
> force' operation. This task finished successfully (you can see info below)
> and number of files on the 3 nodes were very similar .
>
> For volumedisk1 I only have files of 500MB and they are continuosly
> written in sequential mode. The filename pattern of written files is:
>
> run.node1..rd
> run.node2..rd
> run.node1.0001.rd
> run.node2.0001.rd
> run.node1.0002.rd
> run.node2.0002.rd
> ...
> ...
> run.node1.X.rd
> run.node2.X.rd
>
> (  X ranging from  to infinite )
>
> Curiously stor1data and stor2data maintain similar ratios in bytes:
>
> Filesystem  1K-blocksUsed   Available
> Use% Mounted on
> /dev/sdc1 52737613824 17079174264  35658439560  33%
> /mnt/glusterfs/vol1   -> stor1data
> /dev/sdc1 52737613824 17118810848  35618802976  33%
> /mnt/glusterfs/vol1  ->  stor2data
>
> However the ratio on som3data differs too much (1TB):
> Filesystem   1K-blocksUsedAvailable
> Use% Mounted on
> /dev/sdc1 52737613824 15479191748  37258422076  30%
> /mnt/disk_c/glusterfs/vol1 -> stor3data
> /dev/sdd1 52737613824 15566398604  37171215220  30%
> /mnt/disk_d/glusterfs/vol1 -> stor3data
>
> Thinking in  inodes:
>
> FilesystemInodes   IUsed   IFree  IUse%
> Mounted on
> /dev/sdc1 5273970048  851053  52731189951%
> /mnt/glusterfs/vol1 ->  stor1data
> /dev/sdc1 5273970048  849388  52731206601%
> /mnt/glusterfs/vol1 ->  stor2data
>
> /dev/sdc1 5273970048  846877  52731231711%
> /mnt/disk_c/glusterfs/vol1 -> stor3data
> /dev/sdd1 5273970048  845250  52731247981%
> /mnt/disk_d/glusterfs/vol1 -> stor3data
>
> 851053 (stor1) - 845250 (stor3) = 5803 files of difference !
>

The inode numbers are a little misleading here - gluster uses some to
create its own internal files and directory structures. Based on the
average file size, I think this would actually work out to a difference of
around 2000 files.


>
> In adition, correct me if I'm wrong,  stor3data should have 50% of
> probability to store a new file (even taking into account the algorithm of
> DHT with filename patterns)
>
> Theoretically yes , but again, it depends on the filenames and their hash
distribution.

Please send us the output of :
gluster volume rebalance  status

for the volume.

Regards,
Nithya


> Thanks,
> Greetings.
>
> Jose V.
>
> Status of volume: volumedisk0
> Gluster process TCP Port  RDMA Port  Online
>  Pid
> 
> --
> Brick stor1data:/mnt/glusterfs/vol0/bri
> ck1 49152 0  Y
> 13533
> Brick stor2data:/mnt/glusterfs/vol0/bri
> ck1 49152 0  Y
> 13302
> Brick stor3data:/mnt/disk_b1/glusterfs/
> vol0/brick1 49152 0  Y
> 17371
> Brick stor3data:/mnt/disk_b2/glusterfs/
> vol0/brick1 49153 0  Y
> 17391
> NFS Server on localhost N/A   N/AN
> N/A
> NFS Server on stor3data N/A   N/AN   N/A
> NFS Server on stor2data N/A   N/AN   N/A
>
> Task Status of Volume volumedisk0
> 
> --
> Task : Rebalance
> ID   : 7f5328cb-ed25-4627-9196-fb3e29e0e4ca
> Status   : completed
>
> Status of volume: volumedisk1
> Gluster process TCP Port  RDMA Port  Online
>  Pid
> 
> --
> Brick stor1data:/mnt/glusterfs/vol1/bri
> ck1 49153 0  Y
> 13579
> Brick stor2data:/mnt/glusterfs/vol1/bri
> ck1 49153 0  Y
> 13344
> Brick stor3data:/mnt/disk_c/glusterfs/v
> ol1/brick1  49154 0  Y
> 17439
> Brick stor3data:/mnt/disk_d/glusterfs/v
> ol1/brick1  49155 0  Y
> 17459
> NFS Server on localhost N/A   N/AN
> N/A
> NFS Server on stor3data N/A   N/AN   N/A
> NFS Server on stor2data N/A   N/AN   N/A
>
> Task Status of Volume volumedisk1
> 
> --
> Task

Re: [Gluster-users] df reports wrong full capacity for distributed volumes (Glusterfs 3.12.6-1)

2018-02-28 Thread Jose V . Carrión

Hi Nithya,

My initial setup was composed of 2 similar nodes: stor1data and stor2data.
A month ago I expanded both volumes with a new node: stor3data (2 bricks
per volume).
Of course, then to add the new peer with the bricks I did the 'balance
force' operation. This task finished successfully (you can see info below)
and number of files on the 3 nodes were very similar .

For volumedisk1 I only have files of 500MB and they are continuosly written
in sequential mode. The filename pattern of written files is:

run.node1..rd
run.node2..rd
run.node1.0001.rd
run.node2.0001.rd
run.node1.0002.rd
run.node2.0002.rd
...
...
run.node1.X.rd
run.node2.X.rd

(  X ranging from  to infinite )

Curiously stor1data and stor2data maintain similar ratios in bytes:

Filesystem  1K-blocksUsed   Available
Use% Mounted on
/dev/sdc1 52737613824 17079174264  35658439560  33%
/mnt/glusterfs/vol1   -> stor1data
/dev/sdc1 52737613824 17118810848  35618802976  33%
/mnt/glusterfs/vol1  ->  stor2data

However the ratio on som3data differs too much (1TB):
Filesystem   1K-blocksUsedAvailable
Use% Mounted on
/dev/sdc1 52737613824 15479191748  37258422076  30%
/mnt/disk_c/glusterfs/vol1 -> stor3data
/dev/sdd1 52737613824 15566398604  37171215220  30%
/mnt/disk_d/glusterfs/vol1 -> stor3data

Thinking in  inodes:

FilesystemInodes   IUsed   IFree  IUse%
Mounted on
/dev/sdc1 5273970048  851053  52731189951%
/mnt/glusterfs/vol1 ->  stor1data
/dev/sdc1 5273970048  849388  52731206601%
/mnt/glusterfs/vol1 ->  stor2data

/dev/sdc1 5273970048  846877  52731231711%
/mnt/disk_c/glusterfs/vol1 -> stor3data
/dev/sdd1 5273970048  845250  52731247981%
/mnt/disk_d/glusterfs/vol1 -> stor3data

851053 (stor1) - 845250 (stor3) = 5803 files of difference !

In adition, correct me if I'm wrong,  stor3data should have 50% of
probability to store a new file (even taking into account the algorithm of
DHT with filename patterns)

Thanks,
Greetings.

Jose V.

Status of volume: volumedisk0
Gluster process TCP Port  RDMA Port  Online  Pid
--
Brick stor1data:/mnt/glusterfs/vol0/bri
ck1 49152 0  Y
13533
Brick stor2data:/mnt/glusterfs/vol0/bri
ck1 49152 0  Y
13302
Brick stor3data:/mnt/disk_b1/glusterfs/
vol0/brick1 49152 0  Y
17371
Brick stor3data:/mnt/disk_b2/glusterfs/
vol0/brick1 49153 0  Y
17391
NFS Server on localhost N/A   N/AN
N/A
NFS Server on stor3data N/A   N/AN   N/A
NFS Server on stor2data N/A   N/AN   N/A

Task Status of Volume volumedisk0
--
Task : Rebalance
ID   : 7f5328cb-ed25-4627-9196-fb3e29e0e4ca
Status   : completed

Status of volume: volumedisk1
Gluster process TCP Port  RDMA Port  Online  Pid
--
Brick stor1data:/mnt/glusterfs/vol1/bri
ck1 49153 0  Y
13579
Brick stor2data:/mnt/glusterfs/vol1/bri
ck1 49153 0  Y
13344
Brick stor3data:/mnt/disk_c/glusterfs/v
ol1/brick1  49154 0  Y
17439
Brick stor3data:/mnt/disk_d/glusterfs/v
ol1/brick1  49155 0  Y
17459
NFS Server on localhost N/A   N/AN
N/A
NFS Server on stor3data N/A   N/AN   N/A
NFS Server on stor2data N/A   N/AN   N/A

Task Status of Volume volumedisk1
--
Task : Rebalance
ID   : d0048704-beeb-4a6a-ae94-7e7916423fd3
Status   : completed


2018-02-28 15:40 GMT+01:00 Nithya Balachandran :

> Hi Jose,
>
> On 28 February 2018 at 18:28, Jose V. Carrión  wrote:
>
>> Hi Nithya,
>>
>> I applied the workarround for this bug and now df shows the right size:
>>
>> That is good to hear.
>
>
>
>> [root@stor1 ~]# df -h
>> FilesystemSize  Used Avail Use% Mounted on
>> /dev/sdb1  26T  1,1T   25T   4% /mnt/glusterfs/vol0
>> /dev/sdc1  50T   16T   34T  33% /mnt/glusterfs/vol1
>> stor1data:/volumedisk0
>>   101T  3,3T   97T   4% /volumedisk0
>> stor1data:/volumedisk1
>>

Re: [Gluster-users] df reports wrong full capacity for distributed volumes (Glusterfs 3.12.6-1)

2018-02-28 Thread Nithya Balachandran

Hi Jose,

On 28 February 2018 at 18:28, Jose V. Carrión  wrote:

> Hi Nithya,
>
> I applied the workarround for this bug and now df shows the right size:
>
> That is good to hear.



> [root@stor1 ~]# df -h
> FilesystemSize  Used Avail Use% Mounted on
> /dev/sdb1  26T  1,1T   25T   4% /mnt/glusterfs/vol0
> /dev/sdc1  50T   16T   34T  33% /mnt/glusterfs/vol1
> stor1data:/volumedisk0
>   101T  3,3T   97T   4% /volumedisk0
> stor1data:/volumedisk1
>   197T   61T  136T  31% /volumedisk1
>
>
> [root@stor2 ~]# df -h
> FilesystemSize  Used Avail Use% Mounted on
> /dev/sdb1  26T  1,1T   25T   4% /mnt/glusterfs/vol0
> /dev/sdc1  50T   16T   34T  33% /mnt/glusterfs/vol1
> stor2data:/volumedisk0
>   101T  3,3T   97T   4% /volumedisk0
> stor2data:/volumedisk1
>   197T   61T  136T  31% /volumedisk1
>
>
> [root@stor3 ~]# df -h
> FilesystemSize  Used Avail Use% Mounted on
> /dev/sdb1  25T  638G   24T   3% /mnt/disk_b1/glusterfs/vol0
> /dev/sdb2  25T  654G   24T   3% /mnt/disk_b2/glusterfs/vol0
> /dev/sdc1  50T   15T   35T  30% /mnt/disk_c/glusterfs/vol1
> /dev/sdd1  50T   15T   35T  30% /mnt/disk_d/glusterfs/vol1
> stor3data:/volumedisk0
>   101T  3,3T   97T   4% /volumedisk0
> stor3data:/volumedisk1
>   197T   61T  136T  31% /volumedisk1
>
>
> However I'm concerned because, as you can see, the volumedisk0 on
> stor3data is composed by 2 bricks on thesame disk but on different
> partitions (/dev/sdb1 and /dev/sdb2).
> After to aplly the workarround, the  shared-brick-count parameter was
> setted to 1 in all the bricks and all the servers (see below). Could be
> this an issue ?
>
> No, this is correct. The shared-brick-count will be > 1 only if multiple
bricks share the same partition.



> Also, I can check that stor3data is now unbalanced respect stor1data and
> stor2data. The three nodes have the same size of brick but stor3data bricks
> have used 1TB less than stor1data and stor2data:
>


This does not necessarily indicate a problem. The distribution need not be
exactly equal and depends on the filenames. Can you provide more
information on the kind of dataset (how many files, sizes etc) on this
volume? Did you create the volume with all 4 bricks or add some later?

Regards,
Nithya

>
> stor1data:
> /dev/sdb1  26T  1,1T   25T   4% /mnt/glusterfs/vol0
> /dev/sdc1  50T   16T   34T  33% /mnt/glusterfs/vol1
>
> stor2data bricks:
> /dev/sdb1  26T  1,1T   25T   4% /mnt/glusterfs/vol0
> /dev/sdc1  50T   16T   34T  33% /mnt/glusterfs/vol1
>
> stor3data bricks:
>   /dev/sdb1  25T  638G   24T   3% /mnt/disk_b1/glusterfs/vol0
>   /dev/sdb2  25T  654G   24T   3% /mnt/disk_b2/glusterfs/vol0
>dev/sdc1  50T   15T   35T  30% /mnt/disk_c/glusterfs/vol1
>/dev/sdd1 50T   15T   35T  30% /mnt/disk_d/glusterfs/vol1
>
>
> [root@stor1 ~]# grep -n "share" /var/lib/glusterd/vols/volumedisk1/*
> /var/lib/glusterd/vols/volumedisk1/volumedisk1.
> stor1data.mnt-glusterfs-vol1-brick1.vol:3:option shared-brick-count 1
> /var/lib/glusterd/vols/volumedisk1/volumedisk1.
> stor1data.mnt-glusterfs-vol1-brick1.vol.rpmsave:3:option
> shared-brick-count 1
> /var/lib/glusterd/vols/volumedisk1/volumedisk1.
> stor2data.mnt-glusterfs-vol1-brick1.vol:3:option shared-brick-count 1
> /var/lib/glusterd/vols/volumedisk1/volumedisk1.
> stor2data.mnt-glusterfs-vol1-brick1.vol.rpmsave:3:option
> shared-brick-count 0
> /var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_c-glusterfs-vol1-brick1.vol:3:
>option shared-brick-count 1
> /var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_c-
> glusterfs-vol1-brick1.vol.rpmsave:3:option shared-brick-count 0
> /var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_d-glusterfs-vol1-brick1.vol:3:
>option shared-brick-count 1
> /var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_d-
> glusterfs-vol1-brick1.vol.rpmsave:3:option shared-brick-count 0
>
> [root@stor2 ~]# grep -n "share" /var/lib/glusterd/vols/volumedisk1/*
> /var/lib/glusterd/vols/volumedisk1/volumedisk1.
> stor1data.mnt-glusterfs-vol1-brick1.vol:3:option shared-brick-count 1
> /var/lib/glusterd/vols/volumedisk1/volumedisk1.
> stor1data.mnt-glusterfs-vol1-brick1.vol.rpmsave:3:option
> shared-brick-count 0
> /var/lib/glusterd/vols/volumedisk1/volumedisk1.
> stor2data.mnt-glusterfs-vol1-brick1.vol:3:option shared-brick-count 1
> /var/lib/glusterd/vols/volumedisk1/volumedisk1.
> stor2data.mnt-glusterfs-vol1-brick1.vol.rpmsave:3:option
> shared-brick-count 1
> /var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_c-glusterfs-vol1-brick1.vol:3:
>option shared-brick-count 1
>

Re: [Gluster-users] df reports wrong full capacity for distributed volumes (Glusterfs 3.12.6-1)

2018-02-28 Thread Jose V . Carrión

Hi Nithya,

I applied the workarround for this bug and now df shows the right size:

[root@stor1 ~]# df -h
FilesystemSize  Used Avail Use% Mounted on
/dev/sdb1  26T  1,1T   25T   4% /mnt/glusterfs/vol0
/dev/sdc1  50T   16T   34T  33% /mnt/glusterfs/vol1
stor1data:/volumedisk0
  101T  3,3T   97T   4% /volumedisk0
stor1data:/volumedisk1
  197T   61T  136T  31% /volumedisk1


[root@stor2 ~]# df -h
FilesystemSize  Used Avail Use% Mounted on
/dev/sdb1  26T  1,1T   25T   4% /mnt/glusterfs/vol0
/dev/sdc1  50T   16T   34T  33% /mnt/glusterfs/vol1
stor2data:/volumedisk0
  101T  3,3T   97T   4% /volumedisk0
stor2data:/volumedisk1
  197T   61T  136T  31% /volumedisk1


[root@stor3 ~]# df -h
FilesystemSize  Used Avail Use% Mounted on
/dev/sdb1  25T  638G   24T   3% /mnt/disk_b1/glusterfs/vol0
/dev/sdb2  25T  654G   24T   3% /mnt/disk_b2/glusterfs/vol0
/dev/sdc1  50T   15T   35T  30% /mnt/disk_c/glusterfs/vol1
/dev/sdd1  50T   15T   35T  30% /mnt/disk_d/glusterfs/vol1
stor3data:/volumedisk0
  101T  3,3T   97T   4% /volumedisk0
stor3data:/volumedisk1
  197T   61T  136T  31% /volumedisk1


However I'm concerned because, as you can see, the volumedisk0 on stor3data
is composed by 2 bricks on thesame disk but on different partitions
(/dev/sdb1 and /dev/sdb2).
After to aplly the workarround, the  shared-brick-count parameter was
setted to 1 in all the bricks and all the servers (see below). Could be
this an issue ?

Also, I can check that stor3data is now unbalanced respect stor1data and
stor2data. The three nodes have the same size of brick but stor3data bricks
have used 1TB less than stor1data and stor2data:

stor1data:
/dev/sdb1  26T  1,1T   25T   4% /mnt/glusterfs/vol0
/dev/sdc1  50T   16T   34T  33% /mnt/glusterfs/vol1

stor2data bricks:
/dev/sdb1  26T  1,1T   25T   4% /mnt/glusterfs/vol0
/dev/sdc1  50T   16T   34T  33% /mnt/glusterfs/vol1

stor3data bricks:
  /dev/sdb1  25T  638G   24T   3% /mnt/disk_b1/glusterfs/vol0
  /dev/sdb2  25T  654G   24T   3% /mnt/disk_b2/glusterfs/vol0
   dev/sdc1  50T   15T   35T  30% /mnt/disk_c/glusterfs/vol1
   /dev/sdd1 50T   15T   35T  30% /mnt/disk_d/glusterfs/vol1


[root@stor1 ~]# grep -n "share" /var/lib/glusterd/vols/volumedisk1/*
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor1data.mnt-glusterfs-vol1-brick1.vol:3:
   option shared-brick-count 1
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor1data.mnt-glusterfs-vol1-brick1.vol.rpmsave:3:
   option shared-brick-count 1
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor2data.mnt-glusterfs-vol1-brick1.vol:3:
   option shared-brick-count 1
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor2data.mnt-glusterfs-vol1-brick1.vol.rpmsave:3:
   option shared-brick-count 0
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_c-glusterfs-vol1-brick1.vol:3:
   option shared-brick-count 1
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_c-glusterfs-vol1-brick1.vol.rpmsave:3:
   option shared-brick-count 0
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_d-glusterfs-vol1-brick1.vol:3:
   option shared-brick-count 1
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_d-glusterfs-vol1-brick1.vol.rpmsave:3:
   option shared-brick-count 0

[root@stor2 ~]# grep -n "share" /var/lib/glusterd/vols/volumedisk1/*
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor1data.mnt-glusterfs-vol1-brick1.vol:3:
   option shared-brick-count 1
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor1data.mnt-glusterfs-vol1-brick1.vol.rpmsave:3:
   option shared-brick-count 0
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor2data.mnt-glusterfs-vol1-brick1.vol:3:
   option shared-brick-count 1
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor2data.mnt-glusterfs-vol1-brick1.vol.rpmsave:3:
   option shared-brick-count 1
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_c-glusterfs-vol1-brick1.vol:3:
   option shared-brick-count 1
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_c-glusterfs-vol1-brick1.vol.rpmsave:3:
   option shared-brick-count 0
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_d-glusterfs-vol1-brick1.vol:3:
   option shared-brick-count 1
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_d-glusterfs-vol1-brick1.vol.rpmsave:3:
   option shared-brick-count 0

[root@stor3t ~]# grep -n "share" /var/lib/glusterd/vols/volumedisk1/*
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor1data.mnt-glusterfs-vol1-brick1.vol:3:
   option shared-brick-count 1
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor1data.mnt-glusterfs-vol1-brick1.vol.rpmsave:3:
   option shared-brick-count 1

Re: [Gluster-users] df reports wrong full capacity for distributed volumes (Glusterfs 3.12.6-1)

2018-02-27 Thread Nithya Balachandran

Hi Jose,

There is a known issue with gluster 3.12.x builds (see [1]) so you may be
running into this.

The "shared-brick-count" values seem fine on stor1. Please send us "grep -n
"share" /var/lib/glusterd/vols/volumedisk1/*" results for the other nodes
so we can check if they are the cause.


Regards,
Nithya



[1] https://bugzilla.redhat.com/show_bug.cgi?id=1517260

On 28 February 2018 at 03:03, Jose V. Carrión  wrote:

> Hi,
>
> Some days ago all my glusterfs configuration was working fine. Today I
> realized that the total size reported by df command was changed and is
> smaller than the aggregated capacity of all the bricks in the volume.
>
> I checked that all the volumes status are fine, all the glusterd daemons
> are running, there is no error in logs,  however df shows a bad total size.
>
> My configuration for one volume: volumedisk1
> [root@stor1 ~]# gluster volume status volumedisk1  detail
>
> Status of volume: volumedisk1
> 
> --
> Brick: Brick stor1data:/mnt/glusterfs/vol1/brick1
> TCP Port : 49153
> RDMA Port: 0
> Online   : Y
> Pid  : 13579
> File System  : xfs
> Device   : /dev/sdc1
> Mount Options: rw,noatime
> Inode Size   : 512
> Disk Space Free  : 35.0TB
> Total Disk Space : 49.1TB
> Inode Count  : 5273970048
> Free Inodes  : 5273123069
> 
> --
> Brick: Brick stor2data:/mnt/glusterfs/vol1/brick1
> TCP Port : 49153
> RDMA Port: 0
> Online   : Y
> Pid  : 13344
> File System  : xfs
> Device   : /dev/sdc1
> Mount Options: rw,noatime
> Inode Size   : 512
> Disk Space Free  : 35.0TB
> Total Disk Space : 49.1TB
> Inode Count  : 5273970048
> Free Inodes  : 5273124718
> 
> --
> Brick: Brick stor3data:/mnt/disk_c/glusterfs/vol1/brick1
> TCP Port : 49154
> RDMA Port: 0
> Online   : Y
> Pid  : 17439
> File System  : xfs
> Device   : /dev/sdc1
> Mount Options: rw,noatime
> Inode Size   : 512
> Disk Space Free  : 35.7TB
> Total Disk Space : 49.1TB
> Inode Count  : 5273970048
> Free Inodes  : 5273125437
> 
> --
> Brick: Brick stor3data:/mnt/disk_d/glusterfs/vol1/brick1
> TCP Port : 49155
> RDMA Port: 0
> Online   : Y
> Pid  : 17459
> File System  : xfs
> Device   : /dev/sdd1
> Mount Options: rw,noatime
> Inode Size   : 512
> Disk Space Free  : 35.6TB
> Total Disk Space : 49.1TB
> Inode Count  : 5273970048
> Free Inodes  : 5273127036
>
>
> Then full size for volumedisk1 should be: 49.1TB + 49.1TB + 49.1TB +49.1TB
> = *196,4 TB  *but df shows:
>
> [root@stor1 ~]# df -h
> FilesystemSize  Used Avail Use% Mounted on
> /dev/sda2  48G   21G   25G  46% /
> tmpfs  32G   80K   32G   1% /dev/shm
> /dev/sda1 190M   62M  119M  35% /boot
> /dev/sda4 395G  251G  124G  68% /data
> /dev/sdb1  26T  601G   25T   3% /mnt/glusterfs/vol0
> /dev/sdc1  50T   15T   36T  29% /mnt/glusterfs/vol1
> stor1data:/volumedisk0
>76T  1,6T   74T   3% /volumedisk0
> stor1data:/volumedisk1
>   *148T*   42T  106T  29% /volumedisk1
>
> Exactly 1 brick minus: 196,4 TB - 49,1TB = 148TB
>
> It's a production system so I hope you can help me.
>
> Thanks in advance.
>
> Jose V.
>
>
> Below some other data of my configuration:
>
> [root@stor1 ~]# gluster volume info
>
> Volume Name: volumedisk0
> Type: Distribute
> Volume ID: 0ee52d94-1131-4061-bcef-bd8cf898da10
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 4
> Transport-type: tcp
> Bricks:
> Brick1: stor1data:/mnt/glusterfs/vol0/brick1
> Brick2: stor2data:/mnt/glusterfs/vol0/brick1
> Brick3: stor3data:/mnt/disk_b1/glusterfs/vol0/brick1
> Brick4: stor3data:/mnt/disk_b2/glusterfs/vol0/brick1
> Options Reconfigured:
> performance.cache-size: 4GB
> cluster.min-free-disk: 1%
> performance.io-thread-count: 16
> performance.readdir-ahead: on
>
> Volume Name: volumedisk1
> Type: Distribute
> Volume ID: 591b7098-800e-4954-82a9-6b6d81c9e0a2
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 4
> Transport-type: tcp
> Bricks:
> Brick1: stor1data:/mnt/glusterfs/vol1/brick1
> Brick2: stor2data:/mnt/glusterfs/vol1/brick1
> Brick3: stor3data:/mnt/disk_c/glusterfs/vol1/brick1
> Brick4: stor3data:/mnt/disk_d/glusterfs/vol1/brick1
> Options

[Gluster-users] df reports wrong full capacity for distributed volumes (Glusterfs 3.12.6-1)

2018-02-27 Thread Jose V . Carrión

Hi,

Some days ago all my glusterfs configuration was working fine. Today I
realized that the total size reported by df command was changed and is
smaller than the aggregated capacity of all the bricks in the volume.

I checked that all the volumes status are fine, all the glusterd daemons
are running, there is no error in logs,  however df shows a bad total size.

My configuration for one volume: volumedisk1
[root@stor1 ~]# gluster volume status volumedisk1  detail

Status of volume: volumedisk1
--
Brick: Brick stor1data:/mnt/glusterfs/vol1/brick1
TCP Port : 49153
RDMA Port: 0
Online   : Y
Pid  : 13579
File System  : xfs
Device   : /dev/sdc1
Mount Options: rw,noatime
Inode Size   : 512
Disk Space Free  : 35.0TB
Total Disk Space : 49.1TB
Inode Count  : 5273970048
Free Inodes  : 5273123069
--
Brick: Brick stor2data:/mnt/glusterfs/vol1/brick1
TCP Port : 49153
RDMA Port: 0
Online   : Y
Pid  : 13344
File System  : xfs
Device   : /dev/sdc1
Mount Options: rw,noatime
Inode Size   : 512
Disk Space Free  : 35.0TB
Total Disk Space : 49.1TB
Inode Count  : 5273970048
Free Inodes  : 5273124718
--
Brick: Brick stor3data:/mnt/disk_c/glusterfs/vol1/brick1
TCP Port : 49154
RDMA Port: 0
Online   : Y
Pid  : 17439
File System  : xfs
Device   : /dev/sdc1
Mount Options: rw,noatime
Inode Size   : 512
Disk Space Free  : 35.7TB
Total Disk Space : 49.1TB
Inode Count  : 5273970048
Free Inodes  : 5273125437
--
Brick: Brick stor3data:/mnt/disk_d/glusterfs/vol1/brick1
TCP Port : 49155
RDMA Port: 0
Online   : Y
Pid  : 17459
File System  : xfs
Device   : /dev/sdd1
Mount Options: rw,noatime
Inode Size   : 512
Disk Space Free  : 35.6TB
Total Disk Space : 49.1TB
Inode Count  : 5273970048
Free Inodes  : 5273127036


Then full size for volumedisk1 should be: 49.1TB + 49.1TB + 49.1TB +49.1TB
= *196,4 TB  *but df shows:

[root@stor1 ~]# df -h
FilesystemSize  Used Avail Use% Mounted on
/dev/sda2  48G   21G   25G  46% /
tmpfs  32G   80K   32G   1% /dev/shm
/dev/sda1 190M   62M  119M  35% /boot
/dev/sda4 395G  251G  124G  68% /data
/dev/sdb1  26T  601G   25T   3% /mnt/glusterfs/vol0
/dev/sdc1  50T   15T   36T  29% /mnt/glusterfs/vol1
stor1data:/volumedisk0
   76T  1,6T   74T   3% /volumedisk0
stor1data:/volumedisk1
  *148T*   42T  106T  29% /volumedisk1

Exactly 1 brick minus: 196,4 TB - 49,1TB = 148TB

It's a production system so I hope you can help me.

Thanks in advance.

Jose V.


Below some other data of my configuration:

[root@stor1 ~]# gluster volume info

Volume Name: volumedisk0
Type: Distribute
Volume ID: 0ee52d94-1131-4061-bcef-bd8cf898da10
Status: Started
Snapshot Count: 0
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: stor1data:/mnt/glusterfs/vol0/brick1
Brick2: stor2data:/mnt/glusterfs/vol0/brick1
Brick3: stor3data:/mnt/disk_b1/glusterfs/vol0/brick1
Brick4: stor3data:/mnt/disk_b2/glusterfs/vol0/brick1
Options Reconfigured:
performance.cache-size: 4GB
cluster.min-free-disk: 1%
performance.io-thread-count: 16
performance.readdir-ahead: on

Volume Name: volumedisk1
Type: Distribute
Volume ID: 591b7098-800e-4954-82a9-6b6d81c9e0a2
Status: Started
Snapshot Count: 0
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: stor1data:/mnt/glusterfs/vol1/brick1
Brick2: stor2data:/mnt/glusterfs/vol1/brick1
Brick3: stor3data:/mnt/disk_c/glusterfs/vol1/brick1
Brick4: stor3data:/mnt/disk_d/glusterfs/vol1/brick1
Options Reconfigured:
cluster.min-free-inodes: 6%
performance.cache-size: 4GB
cluster.min-free-disk: 1%
performance.io-thread-count: 16
performance.readdir-ahead: on

[root@stor1 ~]# grep -n "share" /var/lib/glusterd/vols/volumedisk1/*
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor1data.mnt-glusterfs-vol1-brick1.vol:3:
option shared-brick-count 1
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor1data.mnt-glusterfs-vol1-brick1.vol.rpmsave:3:
option shared-brick-count 1
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor2data.mnt-glusterfs-vol1-brick1.vol:3:
option shared-brick-count 0
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor2data.mnt-glusterfs-vol1-brick1.vol.rpmsave:3:
option

Re: [Gluster-users] df reports wrong full capacity for distributed volumes (Glusterfs 3.12.6-1)

Re: [Gluster-users] df reports wrong full capacity for distributed volumes (Glusterfs 3.12.6-1)

Re: [Gluster-users] df reports wrong full capacity for distributed volumes (Glusterfs 3.12.6-1)

Re: [Gluster-users] df reports wrong full capacity for distributed volumes (Glusterfs 3.12.6-1)

Re: [Gluster-users] df reports wrong full capacity for distributed volumes (Glusterfs 3.12.6-1)

Re: [Gluster-users] df reports wrong full capacity for distributed volumes (Glusterfs 3.12.6-1)

Re: [Gluster-users] df reports wrong full capacity for distributed volumes (Glusterfs 3.12.6-1)

Re: [Gluster-users] df reports wrong full capacity for distributed volumes (Glusterfs 3.12.6-1)

[Gluster-users] df reports wrong full capacity for distributed volumes (Glusterfs 3.12.6-1)

9 matches

Site Navigation

Mail list logo

Footer information