[Gluster-users] Bitrot strange behavior

[Cedric Lemarchand]

Hello,

I am playing around with the bitrot feature and have some questions:

1. when a file is created, the "trusted.bit-rot.signature” attribute
seems only created approximatively 120 seconds after its creations
(the cluster is idle and there is only one file living on it). Why ?
Is there a way to make this attribute generated at the same time of
the file creation ?

2. corrupting a file (adding a 0 locally on a brick) before the
creation of the "trusted.bit-rot.signature” do not provide any
warning: its signature is different than the 2 others copies on other
bricks. Starting a scrub did not show up anything. I would think that
Gluster compares signature between bricks for this particular use
cases, but it seems the check is only local, so a file corrupted
before it’s bitrot signature creation stay corrupted, and thus could
be served to clients whith bad data ?

Gluster 3.12.8 on Debian Stretch, bricks on ext4.

Volume Name: vol1
Type: Replicate
Volume ID: 85ccfaf2-5793-46f2-bd20-3f823b0a2232
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: gluster-01:/data/brick1
Brick2: gluster-02:/data/brick2
Brick3: gluster-03:/data/brick3
Options Reconfigured:
storage.build-pgfid: on
performance.client-io-threads: off
nfs.disable: on
transport.address-family: inet
features.bitrot: on
features.scrub: Active
features.scrub-throttle: aggressive
features.scrub-freq: hourly

Cheers,

Cédric
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] lstat & readlink calls during glusterfsd process startup

[Serkan Çoban]

Hi all,

I am on gluster 3.10.5 with one EC volume 16+4.
One of the machines go down previous night and I just fixed it and powered on.
When glusterfsd processes started they consume all CPU on the server.
strace shows every process goes over in bricks directory and do a
lstat & readlink calls.
Each brick directory is 8TB, %60 full. I waited for 24 hours for it to
finish but it did not.
I stopped glusterd and restarted it but same thing happens again. Why
on startup glusterfsd processes traverse brick directory? Is it
related to self heal?

This happened one time before and I somehow prevent it happening with
glusterd stop or some other way I cannot remember right now.

Any thoughts how to solve this issue?
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Fwd: lstat & readlink calls during glusterfsd process startup

[Serkan Çoban]

This is an example from one of the glusterfsd processes, strace -f -c
-p pid_of_glusterfsd

%time seconds usecs/call calls   errors syscall
68   36.2   2131 17002 4758  futex
137   5783 1206   epoll_wait
115.4360545   15  select
...

-- Forwarded message --
From: Serkan Çoban 
Date: Mon, Apr 16, 2018 at 9:20 AM
Subject: lstat & readlink calls during glusterfsd process startup
To: Gluster Users 


Hi all,

I am on gluster 3.10.5 with one EC volume 16+4.
One of the machines go down previous night and I just fixed it and powered on.
When glusterfsd processes started they consume all CPU on the server.
strace shows every process goes over in bricks directory and do a
lstat & readlink calls.
Each brick directory is 8TB, %60 full. I waited for 24 hours for it to
finish but it did not.
I stopped glusterd and restarted it but same thing happens again. Why
on startup glusterfsd processes traverse brick directory? Is it
related to self heal?

This happened one time before and I somehow prevent it happening with
glusterd stop or some other way I cannot remember right now.

Any thoughts how to solve this issue?
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Gluster FUSE mount sometimes reports that files do not exist until ls is performed on parent directory

[Niels Hendriks]

Hi,

We have a 3-node gluster setup where gluster is both the server and the
client.
Every few days we have some $random file or directory that does not exist
according to the FUSE mountpoint. When we try to access the file (stat,
cat, etc...) the filesystem reports that the file/directory does not exist,
even though it does. When we try to create the file/directory we receive
the following error which is also logged in
/var/log/glusterfs/bricks/$brick.log:

[2018-04-10 12:51:26.755928] E [MSGID: 113027] [posix.c:1779:posix_mkdir]
0-www-posix: mkdir of /storage/gluster/path/to/dir failed [File exists]

We don't see this issue on all of the servers, but only on the servers that
did not create the file/directory (so 2 of the 3 gluster nodes).

We found that this issue does not resolve itself automatically. However,
when we perform an ls command on the parent directory the issue will be
resolved for the other nodes.

We are running glusterfs 3.12.6 on debian 8

Mount-options in /etc/fstab:
/dev/storage-gluster/gluster /storage/gluster xfs rw,inode64,noatime,nouuid
0 2
localhost:/www /var/www glusterfs
backup-volfile-servers=10.0.0.2:10.0.0.3,log-level=WARNING
0 0

gluster volume info www

Volume Name: www
Type: Replicate
Volume ID: e0579d53-f671-4868-863b-ba85c4cfacb3
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: n01c01-gluster:/storage/gluster/www
Brick2: n02c01-gluster:/storage/gluster/www
Brick3: n03c01-gluster:/storage/gluster/www
Options Reconfigured:
performance.read-ahead: on
performance.client-io-threads: on
nfs.disable: on
transport.address-family: inet
performance.md-cache-timeout: 600
diagnostics.brick-log-level: WARNING
network.ping-timeout: 3
features.cache-invalidation: on
server.event-threads: 4
performance.cache-invalidation: on
performance.quick-read: on
features.cache-invalidation-timeout: 600
network.inode-lru-limit: 9
performance.cache-priority: *.php:3,*.temp:3,*:1
performance.nl-cache: on
performance.cache-size: 1GB
performance.readdir-ahead: on
performance.write-behind: on
cluster.readdir-optimize: on
performance.io-thread-count: 64
client.event-threads: 4
cluster.lookup-optimize: on
performance.parallel-readdir: off
performance.write-behind-window-size: 4MB
performance.flush-behind: on
features.bitrot: on
features.scrub: Active
performance.io-cache: off
performance.stat-prefetch: on

We suspected that the md-cache could be the cause, but it does have a
timeout of 600 seconds so this would be strange since the issue can be
present for hours (at which point we did an ls to fix it).

Does anyone have an idea of what could be the cause of this?

Thanks!
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster FUSE mount sometimes reports that files do not exist until ls is performed on parent directory

[Raghavendra Gowdappa]

On Mon, Apr 16, 2018 at 1:54 PM, Niels Hendriks  wrote:

> Hi,
>
> We have a 3-node gluster setup where gluster is both the server and the
> client.
> Every few days we have some $random file or directory that does not exist
> according to the FUSE mountpoint. When we try to access the file (stat,
> cat, etc...) the filesystem reports that the file/directory does not exist,
> even though it does. When we try to create the file/directory we receive
> the following error which is also logged in
> /var/log/glusterfs/bricks/$brick.log:
>
> [2018-04-10 12:51:26.755928] E [MSGID: 113027] [posix.c:1779:posix_mkdir]
> 0-www-posix: mkdir of /storage/gluster/path/to/dir failed [File exists]
>
> We don't see this issue on all of the servers, but only on the servers that
> did not create the file/directory (so 2 of the 3 gluster nodes).
>
> We found that this issue does not resolve itself automatically. However,
> when we perform an ls command on the parent directory the issue will be
> resolved for the other nodes.
>
> We are running glusterfs 3.12.6 on debian 8
>
> Mount-options in /etc/fstab:
> /dev/storage-gluster/gluster /storage/gluster xfs rw,inode64,noatime,nouuid
> 0 2
> localhost:/www /var/www glusterfs
> backup-volfile-servers=10.0.0.2:10.0.0.3,log-level=WARNING
> 0 0
>
> gluster volume info www
>
> Volume Name: www
> Type: Replicate
> Volume ID: e0579d53-f671-4868-863b-ba85c4cfacb3
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x 3 = 3
> Transport-type: tcp
> Bricks:
> Brick1: n01c01-gluster:/storage/gluster/www
> Brick2: n02c01-gluster:/storage/gluster/www
> Brick3: n03c01-gluster:/storage/gluster/www
> Options Reconfigured:
> performance.read-ahead: on
> performance.client-io-threads: on
> nfs.disable: on
> transport.address-family: inet
> performance.md-cache-timeout: 600
> diagnostics.brick-log-level: WARNING
> network.ping-timeout: 3
> features.cache-invalidation: on
> server.event-threads: 4
> performance.cache-invalidation: on
> performance.quick-read: on
> features.cache-invalidation-timeout: 600
> network.inode-lru-limit: 9
> performance.cache-priority: *.php:3,*.temp:3,*:1
> performance.nl-cache: on
> performance.cache-size: 1GB
> performance.readdir-ahead: on
> performance.write-behind: on
> cluster.readdir-optimize: on
> performance.io-thread-count: 64
> client.event-threads: 4
> cluster.lookup-optimize: on
> performance.parallel-readdir: off
> performance.write-behind-window-size: 4MB
> performance.flush-behind: on
> features.bitrot: on
> features.scrub: Active
> performance.io-cache: off
> performance.stat-prefetch: on
>
> We suspected that the md-cache could be the cause, but it does have a
> timeout of 600 seconds so this would be strange since the issue can be
> present for hours (at which point we did an ls to fix it).
>
> Does anyone have an idea of what could be the cause of this?
>

For files, it could be because of:
* cluster.lookup-optimize set to on
* datafile is present on non hashed subvol, but linkto file is absent on
hashed subvol

I see that lookup-optimize is on. Can you get the following information
from problematic file?

* Name of the file
* all xattrs on parent directory from all bricks
* stat of file from all bricks where it is present.
* all xattrs on file from all bricks where it is present.

If you are seeing the problem on directory,
* Name of directory
* xattr of directory and its parent from all bricks

regards,
Raghavendra


> Thanks!
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Getting glusterfs to expand volume size to brick size

[Nithya Balachandran]

Hi Artem,

Was the volume size correct before the bricks were expanded?

This sounds like [1] but that should have been fixed in 4.0.0. Can you let
us know the values of shared-brick-count in the files in
/var/lib/glusterd/vols/dev_apkmirror_data/ ?

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1541880

On 17 April 2018 at 05:17, Artem Russakovskii  wrote:

> Hi Nithya,
>
> I'm on Gluster 4.0.1.
>
> I don't think the bricks were smaller before - if they were, maybe 20GB
> because Linode's minimum is 20GB, then I extended them to 25GB, resized
> with resize2fs as instructed, and rebooted many times over since. Yet,
> gluster refuses to see the full disk size.
>
> Here's the status detail output:
>
> gluster volume status dev_apkmirror_data detail
> Status of volume: dev_apkmirror_data
> 
> --
> Brick: Brick pylon:/mnt/pylon_block1/dev_apkmirror_data
> TCP Port : 49152
> RDMA Port: 0
> Online   : Y
> Pid  : 1263
> File System  : ext4
> Device   : /dev/sdd
> Mount Options: rw,relatime,data=ordered
> Inode Size   : 256
> Disk Space Free  : 23.0GB
> Total Disk Space : 24.5GB
> Inode Count  : 1638400
> Free Inodes  : 1625429
> 
> --
> Brick: Brick pylon:/mnt/pylon_block2/dev_apkmirror_data
> TCP Port : 49153
> RDMA Port: 0
> Online   : Y
> Pid  : 1288
> File System  : ext4
> Device   : /dev/sdc
> Mount Options: rw,relatime,data=ordered
> Inode Size   : 256
> Disk Space Free  : 24.0GB
> Total Disk Space : 25.5GB
> Inode Count  : 1703936
> Free Inodes  : 1690965
> 
> --
> Brick: Brick pylon:/mnt/pylon_block3/dev_apkmirror_data
> TCP Port : 49154
> RDMA Port: 0
> Online   : Y
> Pid  : 1313
> File System  : ext4
> Device   : /dev/sde
> Mount Options: rw,relatime,data=ordered
> Inode Size   : 256
> Disk Space Free  : 23.0GB
> Total Disk Space : 24.5GB
> Inode Count  : 1638400
> Free Inodes  : 1625433
>
>
>
> What's interesting here is that the gluster volume size is exactly 1/3 of
> the total (8357M * 3 = 25071M). Yet, each block device is separate, and the
> total storage available is 25071M on each brick.
>
> The fstab is as follows:
> /dev/disk/by-id/scsi-0Linode_Volume_pylon_block1 /mnt/pylon_block1 ext4
> defaults 0 2
> /dev/disk/by-id/scsi-0Linode_Volume_pylon_block2 /mnt/pylon_block2 ext4
> defaults 0 2
> /dev/disk/by-id/scsi-0Linode_Volume_pylon_block3 /mnt/pylon_block3 ext4
> defaults 0 2
>
> localhost:/dev_apkmirror_data/mnt/dev_apkmirror_data1   glusterfs
> defaults,_netdev,fopen-keep-cache,direct-io-mode=enable 0 0
> localhost:/dev_apkmirror_data/mnt/dev_apkmirror_data2   glusterfs
> defaults,_netdev,fopen-keep-cache,direct-io-mode=enable 0 0
> localhost:/dev_apkmirror_data/mnt/dev_apkmirror_data3   glusterfs
> defaults,_netdev,fopen-keep-cache,direct-io-mode=enable 0 0
> localhost:/dev_apkmirror_data/mnt/dev_apkmirror_data_ganesha   nfs4
> defaults,_netdev,bg,intr,soft,timeo=5,retrans=5,actimeo=10,retry=5 0 0
>
> The latter entry is for an nfs ganesha test, in case it matters (which,
> btw, fails miserably with all kinds of stability issues about broken pipes).
>
> Note: this is a test server, so all 3 bricks are attached and mounted on
> the same server.
>
>
> Sincerely,
> Artem
>
> --
> Founder, Android Police , APK Mirror
> , Illogical Robot LLC
> beerpla.net | +ArtemRussakovskii
>  | @ArtemR
> 
>
> On Sun, Apr 15, 2018 at 10:56 PM, Nithya Balachandran  > wrote:
>
>> What version of Gluster are you running? Were the bricks smaller earlier?
>>
>> Regards,
>> Nithya
>>
>> On 15 April 2018 at 00:09, Artem Russakovskii 
>> wrote:
>>
>>> Hi,
>>>
>>> I have a 3-brick replicate volume, but for some reason I can't get it to
>>> expand to the size of the bricks. The bricks are 25GB, but even after
>>> multiple gluster restarts and remounts, the volume is only about 8GB.
>>>
>>> I believed I could always extend the bricks (we're using Linode block
>>> storage, which allows extending block devices after they're created), and
>>> gluster would see the newly available space and extend to use it.
>>>
>>> Multiple Google searches, and I'm still nowhere. Any ideas?
>>>
>>> df | ack "block|data"
>>> Filesystem   1M-blocks
>>>Used Available Use% Mounted on
>>> 

Re: [Gluster-users] Getting glusterfs to expand volume size to brick size

[Artem Russakovskii]

pylon:/var/lib/glusterd/vols/dev_apkmirror_data # ack shared-brick-count
dev_apkmirror_data.pylon.mnt-pylon_block3-dev_apkmirror_data.vol
3:option shared-brick-count 3

dev_apkmirror_data.pylon.mnt-pylon_block2-dev_apkmirror_data.vol
3:option shared-brick-count 3

dev_apkmirror_data.pylon.mnt-pylon_block1-dev_apkmirror_data.vol
3:option shared-brick-count 3


Sincerely,
Artem

--
Founder, Android Police , APK Mirror
, Illogical Robot LLC
beerpla.net | +ArtemRussakovskii
 | @ArtemR


On Mon, Apr 16, 2018 at 9:22 PM, Nithya Balachandran 
wrote:

> Hi Artem,
>
> Was the volume size correct before the bricks were expanded?
>
> This sounds like [1] but that should have been fixed in 4.0.0. Can you let
> us know the values of shared-brick-count in the files in
> /var/lib/glusterd/vols/dev_apkmirror_data/ ?
>
> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1541880
>
> On 17 April 2018 at 05:17, Artem Russakovskii  wrote:
>
>> Hi Nithya,
>>
>> I'm on Gluster 4.0.1.
>>
>> I don't think the bricks were smaller before - if they were, maybe 20GB
>> because Linode's minimum is 20GB, then I extended them to 25GB, resized
>> with resize2fs as instructed, and rebooted many times over since. Yet,
>> gluster refuses to see the full disk size.
>>
>> Here's the status detail output:
>>
>> gluster volume status dev_apkmirror_data detail
>> Status of volume: dev_apkmirror_data
>> 
>> --
>> Brick: Brick pylon:/mnt/pylon_block1/dev_apkmirror_data
>> TCP Port : 49152
>> RDMA Port: 0
>> Online   : Y
>> Pid  : 1263
>> File System  : ext4
>> Device   : /dev/sdd
>> Mount Options: rw,relatime,data=ordered
>> Inode Size   : 256
>> Disk Space Free  : 23.0GB
>> Total Disk Space : 24.5GB
>> Inode Count  : 1638400
>> Free Inodes  : 1625429
>> 
>> --
>> Brick: Brick pylon:/mnt/pylon_block2/dev_apkmirror_data
>> TCP Port : 49153
>> RDMA Port: 0
>> Online   : Y
>> Pid  : 1288
>> File System  : ext4
>> Device   : /dev/sdc
>> Mount Options: rw,relatime,data=ordered
>> Inode Size   : 256
>> Disk Space Free  : 24.0GB
>> Total Disk Space : 25.5GB
>> Inode Count  : 1703936
>> Free Inodes  : 1690965
>> 
>> --
>> Brick: Brick pylon:/mnt/pylon_block3/dev_apkmirror_data
>> TCP Port : 49154
>> RDMA Port: 0
>> Online   : Y
>> Pid  : 1313
>> File System  : ext4
>> Device   : /dev/sde
>> Mount Options: rw,relatime,data=ordered
>> Inode Size   : 256
>> Disk Space Free  : 23.0GB
>> Total Disk Space : 24.5GB
>> Inode Count  : 1638400
>> Free Inodes  : 1625433
>>
>>
>>
>> What's interesting here is that the gluster volume size is exactly 1/3 of
>> the total (8357M * 3 = 25071M). Yet, each block device is separate, and the
>> total storage available is 25071M on each brick.
>>
>> The fstab is as follows:
>> /dev/disk/by-id/scsi-0Linode_Volume_pylon_block1 /mnt/pylon_block1 ext4
>> defaults 0 2
>> /dev/disk/by-id/scsi-0Linode_Volume_pylon_block2 /mnt/pylon_block2 ext4
>> defaults 0 2
>> /dev/disk/by-id/scsi-0Linode_Volume_pylon_block3 /mnt/pylon_block3 ext4
>> defaults 0 2
>>
>> localhost:/dev_apkmirror_data/mnt/dev_apkmirror_data1   glusterfs
>> defaults,_netdev,fopen-keep-cache,direct-io-mode=enable 0 0
>> localhost:/dev_apkmirror_data/mnt/dev_apkmirror_data2   glusterfs
>> defaults,_netdev,fopen-keep-cache,direct-io-mode=enable 0 0
>> localhost:/dev_apkmirror_data/mnt/dev_apkmirror_data3   glusterfs
>> defaults,_netdev,fopen-keep-cache,direct-io-mode=enable 0 0
>> localhost:/dev_apkmirror_data/mnt/dev_apkmirror_data_ganesha   nfs4
>> defaults,_netdev,bg,intr,soft,timeo=5,retrans=5,actimeo=10,retry=5 0 0
>>
>> The latter entry is for an nfs ganesha test, in case it matters (which,
>> btw, fails miserably with all kinds of stability issues about broken pipes).
>>
>> Note: this is a test server, so all 3 bricks are attached and mounted on
>> the same server.
>>
>>
>> Sincerely,
>> Artem
>>
>> --
>> Founder, Android Police , APK Mirror
>> , Illogical Robot LLC
>> beerpla.net | +ArtemRussakovskii
>>  | @ArtemR
>> 
>>
>> On Sun, Apr 15, 2018 at 10:56 PM, Nithya Balachandran <
>> nbala...@redhat.com> wrote:
>>
>>> What version of Gluster are 

Re: [Gluster-users] Getting glusterfs to expand volume size to brick size

[Artem Russakovskii]

To clarify, I was on 3.13.2 previously, recently updated to 4.0.1, and the
bug seems to persist in 4.0.1.


Sincerely,
Artem

--
Founder, Android Police , APK Mirror
, Illogical Robot LLC
beerpla.net | +ArtemRussakovskii
 | @ArtemR


On Mon, Apr 16, 2018 at 9:27 PM, Artem Russakovskii 
wrote:

> pylon:/var/lib/glusterd/vols/dev_apkmirror_data # ack shared-brick-count
> dev_apkmirror_data.pylon.mnt-pylon_block3-dev_apkmirror_data.vol
> 3:option shared-brick-count 3
>
> dev_apkmirror_data.pylon.mnt-pylon_block2-dev_apkmirror_data.vol
> 3:option shared-brick-count 3
>
> dev_apkmirror_data.pylon.mnt-pylon_block1-dev_apkmirror_data.vol
> 3:option shared-brick-count 3
>
>
> Sincerely,
> Artem
>
> --
> Founder, Android Police , APK Mirror
> , Illogical Robot LLC
> beerpla.net | +ArtemRussakovskii
>  | @ArtemR
> 
>
> On Mon, Apr 16, 2018 at 9:22 PM, Nithya Balachandran 
> wrote:
>
>> Hi Artem,
>>
>> Was the volume size correct before the bricks were expanded?
>>
>> This sounds like [1] but that should have been fixed in 4.0.0. Can you
>> let us know the values of shared-brick-count in the files in
>> /var/lib/glusterd/vols/dev_apkmirror_data/ ?
>>
>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1541880
>>
>> On 17 April 2018 at 05:17, Artem Russakovskii 
>> wrote:
>>
>>> Hi Nithya,
>>>
>>> I'm on Gluster 4.0.1.
>>>
>>> I don't think the bricks were smaller before - if they were, maybe 20GB
>>> because Linode's minimum is 20GB, then I extended them to 25GB, resized
>>> with resize2fs as instructed, and rebooted many times over since. Yet,
>>> gluster refuses to see the full disk size.
>>>
>>> Here's the status detail output:
>>>
>>> gluster volume status dev_apkmirror_data detail
>>> Status of volume: dev_apkmirror_data
>>> 
>>> --
>>> Brick: Brick pylon:/mnt/pylon_block1/dev_apkmirror_data
>>> TCP Port : 49152
>>> RDMA Port: 0
>>> Online   : Y
>>> Pid  : 1263
>>> File System  : ext4
>>> Device   : /dev/sdd
>>> Mount Options: rw,relatime,data=ordered
>>> Inode Size   : 256
>>> Disk Space Free  : 23.0GB
>>> Total Disk Space : 24.5GB
>>> Inode Count  : 1638400
>>> Free Inodes  : 1625429
>>> 
>>> --
>>> Brick: Brick pylon:/mnt/pylon_block2/dev_apkmirror_data
>>> TCP Port : 49153
>>> RDMA Port: 0
>>> Online   : Y
>>> Pid  : 1288
>>> File System  : ext4
>>> Device   : /dev/sdc
>>> Mount Options: rw,relatime,data=ordered
>>> Inode Size   : 256
>>> Disk Space Free  : 24.0GB
>>> Total Disk Space : 25.5GB
>>> Inode Count  : 1703936
>>> Free Inodes  : 1690965
>>> 
>>> --
>>> Brick: Brick pylon:/mnt/pylon_block3/dev_apkmirror_data
>>> TCP Port : 49154
>>> RDMA Port: 0
>>> Online   : Y
>>> Pid  : 1313
>>> File System  : ext4
>>> Device   : /dev/sde
>>> Mount Options: rw,relatime,data=ordered
>>> Inode Size   : 256
>>> Disk Space Free  : 23.0GB
>>> Total Disk Space : 24.5GB
>>> Inode Count  : 1638400
>>> Free Inodes  : 1625433
>>>
>>>
>>>
>>> What's interesting here is that the gluster volume size is exactly 1/3
>>> of the total (8357M * 3 = 25071M). Yet, each block device is separate, and
>>> the total storage available is 25071M on each brick.
>>>
>>> The fstab is as follows:
>>> /dev/disk/by-id/scsi-0Linode_Volume_pylon_block1 /mnt/pylon_block1 ext4
>>> defaults 0 2
>>> /dev/disk/by-id/scsi-0Linode_Volume_pylon_block2 /mnt/pylon_block2 ext4
>>> defaults 0 2
>>> /dev/disk/by-id/scsi-0Linode_Volume_pylon_block3 /mnt/pylon_block3 ext4
>>> defaults 0 2
>>>
>>> localhost:/dev_apkmirror_data/mnt/dev_apkmirror_data1   glusterfs
>>> defaults,_netdev,fopen-keep-cache,direct-io-mode=enable 0 0
>>> localhost:/dev_apkmirror_data/mnt/dev_apkmirror_data2   glusterfs
>>> defaults,_netdev,fopen-keep-cache,direct-io-mode=enable 0 0
>>> localhost:/dev_apkmirror_data/mnt/dev_apkmirror_data3   glusterfs
>>> defaults,_netdev,fopen-keep-cache,direct-io-mode=enable 0 0
>>> localhost:/dev_apkmirror_data/mnt/dev_apkmirror_data_ganesha   nfs4
>>> defaults,_netdev,bg,intr,soft,timeo=5,retrans=5,actimeo=10,retry=5 0 0
>>>
>>> The latter entry is for an nfs ganesha test, in case it matters (which,
>>> btw, 

Re: [Gluster-users] Getting glusterfs to expand volume size to brick size

[Nithya Balachandran]

Ok, it looks like the same problem.


@Amar, this fix is supposed to be in 4.0.1. Is it possible to regenerate
the volfiles to fix this?

Regards,
Nithya

On 17 April 2018 at 09:57, Artem Russakovskii  wrote:

> pylon:/var/lib/glusterd/vols/dev_apkmirror_data # ack shared-brick-count
> dev_apkmirror_data.pylon.mnt-pylon_block3-dev_apkmirror_data.vol
> 3:option shared-brick-count 3
>
> dev_apkmirror_data.pylon.mnt-pylon_block2-dev_apkmirror_data.vol
> 3:option shared-brick-count 3
>
> dev_apkmirror_data.pylon.mnt-pylon_block1-dev_apkmirror_data.vol
> 3:option shared-brick-count 3
>
>
> Sincerely,
> Artem
>
> --
> Founder, Android Police , APK Mirror
> , Illogical Robot LLC
> beerpla.net | +ArtemRussakovskii
>  | @ArtemR
> 
>
> On Mon, Apr 16, 2018 at 9:22 PM, Nithya Balachandran 
> wrote:
>
>> Hi Artem,
>>
>> Was the volume size correct before the bricks were expanded?
>>
>> This sounds like [1] but that should have been fixed in 4.0.0. Can you
>> let us know the values of shared-brick-count in the files in
>> /var/lib/glusterd/vols/dev_apkmirror_data/ ?
>>
>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1541880
>>
>> On 17 April 2018 at 05:17, Artem Russakovskii 
>> wrote:
>>
>>> Hi Nithya,
>>>
>>> I'm on Gluster 4.0.1.
>>>
>>> I don't think the bricks were smaller before - if they were, maybe 20GB
>>> because Linode's minimum is 20GB, then I extended them to 25GB, resized
>>> with resize2fs as instructed, and rebooted many times over since. Yet,
>>> gluster refuses to see the full disk size.
>>>
>>> Here's the status detail output:
>>>
>>> gluster volume status dev_apkmirror_data detail
>>> Status of volume: dev_apkmirror_data
>>> 
>>> --
>>> Brick: Brick pylon:/mnt/pylon_block1/dev_apkmirror_data
>>> TCP Port : 49152
>>> RDMA Port: 0
>>> Online   : Y
>>> Pid  : 1263
>>> File System  : ext4
>>> Device   : /dev/sdd
>>> Mount Options: rw,relatime,data=ordered
>>> Inode Size   : 256
>>> Disk Space Free  : 23.0GB
>>> Total Disk Space : 24.5GB
>>> Inode Count  : 1638400
>>> Free Inodes  : 1625429
>>> 
>>> --
>>> Brick: Brick pylon:/mnt/pylon_block2/dev_apkmirror_data
>>> TCP Port : 49153
>>> RDMA Port: 0
>>> Online   : Y
>>> Pid  : 1288
>>> File System  : ext4
>>> Device   : /dev/sdc
>>> Mount Options: rw,relatime,data=ordered
>>> Inode Size   : 256
>>> Disk Space Free  : 24.0GB
>>> Total Disk Space : 25.5GB
>>> Inode Count  : 1703936
>>> Free Inodes  : 1690965
>>> 
>>> --
>>> Brick: Brick pylon:/mnt/pylon_block3/dev_apkmirror_data
>>> TCP Port : 49154
>>> RDMA Port: 0
>>> Online   : Y
>>> Pid  : 1313
>>> File System  : ext4
>>> Device   : /dev/sde
>>> Mount Options: rw,relatime,data=ordered
>>> Inode Size   : 256
>>> Disk Space Free  : 23.0GB
>>> Total Disk Space : 24.5GB
>>> Inode Count  : 1638400
>>> Free Inodes  : 1625433
>>>
>>>
>>>
>>> What's interesting here is that the gluster volume size is exactly 1/3
>>> of the total (8357M * 3 = 25071M). Yet, each block device is separate, and
>>> the total storage available is 25071M on each brick.
>>>
>>> The fstab is as follows:
>>> /dev/disk/by-id/scsi-0Linode_Volume_pylon_block1 /mnt/pylon_block1 ext4
>>> defaults 0 2
>>> /dev/disk/by-id/scsi-0Linode_Volume_pylon_block2 /mnt/pylon_block2 ext4
>>> defaults 0 2
>>> /dev/disk/by-id/scsi-0Linode_Volume_pylon_block3 /mnt/pylon_block3 ext4
>>> defaults 0 2
>>>
>>> localhost:/dev_apkmirror_data/mnt/dev_apkmirror_data1   glusterfs
>>> defaults,_netdev,fopen-keep-cache,direct-io-mode=enable 0 0
>>> localhost:/dev_apkmirror_data/mnt/dev_apkmirror_data2   glusterfs
>>> defaults,_netdev,fopen-keep-cache,direct-io-mode=enable 0 0
>>> localhost:/dev_apkmirror_data/mnt/dev_apkmirror_data3   glusterfs
>>> defaults,_netdev,fopen-keep-cache,direct-io-mode=enable 0 0
>>> localhost:/dev_apkmirror_data/mnt/dev_apkmirror_data_ganesha   nfs4
>>> defaults,_netdev,bg,intr,soft,timeo=5,retrans=5,actimeo=10,retry=5 0 0
>>>
>>> The latter entry is for an nfs ganesha test, in case it matters (which,
>>> btw, fails miserably with all kinds of stability issues about broken pipes).
>>>
>>> Note: this is a test server, so all 3 bricks are attached and mounted on
>>> the same server.
>>>
>>>
>>> Sincerely,
>>> Artem
>>>
>>> 

Re: [Gluster-users] Getting glusterfs to expand volume size to brick size

[Nithya Balachandran]

That might be the reason. Perhaps the volfiles were not regenerated after
upgrading to the version with the fix.


There is a workaround detailed in [2] for the time being (you will need to
copy the shell script into the correct directory for your Gluster release).


[2] https://bugzilla.redhat.com/show_bug.cgi?id=1517260#c19



On 17 April 2018 at 09:58, Artem Russakovskii  wrote:

> To clarify, I was on 3.13.2 previously, recently updated to 4.0.1, and the
> bug seems to persist in 4.0.1.
>
>
> Sincerely,
> Artem
>
> --
> Founder, Android Police , APK Mirror
> , Illogical Robot LLC
> beerpla.net | +ArtemRussakovskii
>  | @ArtemR
> 
>
> On Mon, Apr 16, 2018 at 9:27 PM, Artem Russakovskii 
> wrote:
>
>> pylon:/var/lib/glusterd/vols/dev_apkmirror_data # ack shared-brick-count
>> dev_apkmirror_data.pylon.mnt-pylon_block3-dev_apkmirror_data.vol
>> 3:option shared-brick-count 3
>>
>> dev_apkmirror_data.pylon.mnt-pylon_block2-dev_apkmirror_data.vol
>> 3:option shared-brick-count 3
>>
>> dev_apkmirror_data.pylon.mnt-pylon_block1-dev_apkmirror_data.vol
>> 3:option shared-brick-count 3
>>
>>
>> Sincerely,
>> Artem
>>
>> --
>> Founder, Android Police , APK Mirror
>> , Illogical Robot LLC
>> beerpla.net | +ArtemRussakovskii
>>  | @ArtemR
>> 
>>
>> On Mon, Apr 16, 2018 at 9:22 PM, Nithya Balachandran > > wrote:
>>
>>> Hi Artem,
>>>
>>> Was the volume size correct before the bricks were expanded?
>>>
>>> This sounds like [1] but that should have been fixed in 4.0.0. Can you
>>> let us know the values of shared-brick-count in the files in
>>> /var/lib/glusterd/vols/dev_apkmirror_data/ ?
>>>
>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1541880
>>>
>>> On 17 April 2018 at 05:17, Artem Russakovskii 
>>> wrote:
>>>
 Hi Nithya,

 I'm on Gluster 4.0.1.

 I don't think the bricks were smaller before - if they were, maybe 20GB
 because Linode's minimum is 20GB, then I extended them to 25GB, resized
 with resize2fs as instructed, and rebooted many times over since. Yet,
 gluster refuses to see the full disk size.

 Here's the status detail output:

 gluster volume status dev_apkmirror_data detail
 Status of volume: dev_apkmirror_data
 
 --
 Brick: Brick pylon:/mnt/pylon_block1/dev_apkmirror_data
 TCP Port : 49152
 RDMA Port: 0
 Online   : Y
 Pid  : 1263
 File System  : ext4
 Device   : /dev/sdd
 Mount Options: rw,relatime,data=ordered
 Inode Size   : 256
 Disk Space Free  : 23.0GB
 Total Disk Space : 24.5GB
 Inode Count  : 1638400
 Free Inodes  : 1625429
 
 --
 Brick: Brick pylon:/mnt/pylon_block2/dev_apkmirror_data
 TCP Port : 49153
 RDMA Port: 0
 Online   : Y
 Pid  : 1288
 File System  : ext4
 Device   : /dev/sdc
 Mount Options: rw,relatime,data=ordered
 Inode Size   : 256
 Disk Space Free  : 24.0GB
 Total Disk Space : 25.5GB
 Inode Count  : 1703936
 Free Inodes  : 1690965
 
 --
 Brick: Brick pylon:/mnt/pylon_block3/dev_apkmirror_data
 TCP Port : 49154
 RDMA Port: 0
 Online   : Y
 Pid  : 1313
 File System  : ext4
 Device   : /dev/sde
 Mount Options: rw,relatime,data=ordered
 Inode Size   : 256
 Disk Space Free  : 23.0GB
 Total Disk Space : 24.5GB
 Inode Count  : 1638400
 Free Inodes  : 1625433



 What's interesting here is that the gluster volume size is exactly 1/3
 of the total (8357M * 3 = 25071M). Yet, each block device is separate, and
 the total storage available is 25071M on each brick.

 The fstab is as follows:
 /dev/disk/by-id/scsi-0Linode_Volume_pylon_block1 /mnt/pylon_block1
 ext4 defaults 0 2
 /dev/disk/by-id/scsi-0Linode_Volume_pylon_block2 /mnt/pylon_block2
 ext4 defaults 0 2
 /dev/disk/by-id/scsi-0Linode_Volume_pylon_block3 /mnt/pylon_block3
 ext4 defaults 0 2

 localhost:/dev_apkmirror_data/mnt/dev_apkmirror_data1   glusterfs
 

Re: [Gluster-users] Getting glusterfs to expand volume size to brick size

[Amar Tumballi]

On Tue, Apr 17, 2018 at 9:59 AM, Nithya Balachandran 
wrote:

> Ok, it looks like the same problem.
>
>
> @Amar, this fix is supposed to be in 4.0.1. Is it possible to regenerate
> the volfiles to fix this?
>

Yes, regenerating volfiles should fix it. Should we try a volume set/reset
of any option?


>
> On 17 April 2018 at 09:57, Artem Russakovskii  wrote:
>
>> pylon:/var/lib/glusterd/vols/dev_apkmirror_data # ack shared-brick-count
>> dev_apkmirror_data.pylon.mnt-pylon_block3-dev_apkmirror_data.vol
>> 3:option shared-brick-count 3
>>
>> dev_apkmirror_data.pylon.mnt-pylon_block2-dev_apkmirror_data.vol
>> 3:option shared-brick-count 3
>>
>> dev_apkmirror_data.pylon.mnt-pylon_block1-dev_apkmirror_data.vol
>> 3:option shared-brick-count 3
>>
>>
>> Sincerely,
>> Artem
>>
>> --
>> Founder, Android Police , APK Mirror
>> , Illogical Robot LLC
>> beerpla.net | +ArtemRussakovskii
>>  | @ArtemR
>> 
>>
>> On Mon, Apr 16, 2018 at 9:22 PM, Nithya Balachandran > > wrote:
>>
>>> Hi Artem,
>>>
>>> Was the volume size correct before the bricks were expanded?
>>>
>>> This sounds like [1] but that should have been fixed in 4.0.0. Can you
>>> let us know the values of shared-brick-count in the files in
>>> /var/lib/glusterd/vols/dev_apkmirror_data/ ?
>>>
>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1541880
>>>
>>> On 17 April 2018 at 05:17, Artem Russakovskii 
>>> wrote:
>>>
 Hi Nithya,

 I'm on Gluster 4.0.1.

 I don't think the bricks were smaller before - if they were, maybe 20GB
 because Linode's minimum is 20GB, then I extended them to 25GB, resized
 with resize2fs as instructed, and rebooted many times over since. Yet,
 gluster refuses to see the full disk size.

 Here's the status detail output:

 gluster volume status dev_apkmirror_data detail
 Status of volume: dev_apkmirror_data
 
 --
 Brick: Brick pylon:/mnt/pylon_block1/dev_apkmirror_data
 TCP Port : 49152
 RDMA Port: 0
 Online   : Y
 Pid  : 1263
 File System  : ext4
 Device   : /dev/sdd
 Mount Options: rw,relatime,data=ordered
 Inode Size   : 256
 Disk Space Free  : 23.0GB
 Total Disk Space : 24.5GB
 Inode Count  : 1638400
 Free Inodes  : 1625429
 
 --
 Brick: Brick pylon:/mnt/pylon_block2/dev_apkmirror_data
 TCP Port : 49153
 RDMA Port: 0
 Online   : Y
 Pid  : 1288
 File System  : ext4
 Device   : /dev/sdc
 Mount Options: rw,relatime,data=ordered
 Inode Size   : 256
 Disk Space Free  : 24.0GB
 Total Disk Space : 25.5GB
 Inode Count  : 1703936
 Free Inodes  : 1690965
 
 --
 Brick: Brick pylon:/mnt/pylon_block3/dev_apkmirror_data
 TCP Port : 49154
 RDMA Port: 0
 Online   : Y
 Pid  : 1313
 File System  : ext4
 Device   : /dev/sde
 Mount Options: rw,relatime,data=ordered
 Inode Size   : 256
 Disk Space Free  : 23.0GB
 Total Disk Space : 24.5GB
 Inode Count  : 1638400
 Free Inodes  : 1625433



 What's interesting here is that the gluster volume size is exactly 1/3
 of the total (8357M * 3 = 25071M). Yet, each block device is separate, and
 the total storage available is 25071M on each brick.

 The fstab is as follows:
 /dev/disk/by-id/scsi-0Linode_Volume_pylon_block1 /mnt/pylon_block1
 ext4 defaults 0 2
 /dev/disk/by-id/scsi-0Linode_Volume_pylon_block2 /mnt/pylon_block2
 ext4 defaults 0 2
 /dev/disk/by-id/scsi-0Linode_Volume_pylon_block3 /mnt/pylon_block3
 ext4 defaults 0 2

 localhost:/dev_apkmirror_data/mnt/dev_apkmirror_data1   glusterfs
 defaults,_netdev,fopen-keep-cache,direct-io-mode=enable 0 0
 localhost:/dev_apkmirror_data/mnt/dev_apkmirror_data2   glusterfs
 defaults,_netdev,fopen-keep-cache,direct-io-mode=enable 0 0
 localhost:/dev_apkmirror_data/mnt/dev_apkmirror_data3   glusterfs
 defaults,_netdev,fopen-keep-cache,direct-io-mode=enable 0 0
 localhost:/dev_apkmirror_data/mnt/dev_apkmirror_data_ganesha
  nfs4 defaults,_netdev,bg,intr,soft,timeo=5,retrans=5,actimeo=10,retry=5
 0 0

Re: [Gluster-users] Getting glusterfs to expand volume size to brick size

[Artem Russakovskii]

I just remembered that I didn't run
https://docs.gluster.org/en/v3/Upgrade-Guide/op_version/ for this test
volume/box like I did for the main production gluster, and one of these ops
- either heal or the op-version, resolved the issue.

I'm now seeing:
pylon:/var/lib/glusterd/vols/dev_apkmirror_data # ack shared-brick-count
dev_apkmirror_data.pylon.mnt-pylon_block3-dev_apkmirror_data.vol
3:option shared-brick-count 1

dev_apkmirror_data.pylon.mnt-pylon_block2-dev_apkmirror_data.vol
3:option shared-brick-count 1

dev_apkmirror_data.pylon.mnt-pylon_block1-dev_apkmirror_data.vol
3:option shared-brick-count 1



/dev/sdd25071M
1491M22284M   7% /mnt/pylon_block1
/dev/sdc26079M
1491M23241M   7% /mnt/pylon_block2
/dev/sde25071M
1491M22315M   7% /mnt/pylon_block3

localhost:/dev_apkmirror_data   25071M
1742M22284M   8% /mnt/dev_apkmirror_data1
localhost:/dev_apkmirror_data   25071M
1742M22284M   8% /mnt/dev_apkmirror_data2
localhost:/dev_apkmirror_data   25071M
1742M22284M   8% /mnt/dev_apkmirror_data3
localhost:/dev_apkmirror_data   25071M
1742M22284M   8% /mnt/dev_apkmirror_data_ganesha


Problem is solved!


Sincerely,
Artem

--
Founder, Android Police , APK Mirror
, Illogical Robot LLC
beerpla.net | +ArtemRussakovskii
 | @ArtemR


On Mon, Apr 16, 2018 at 9:29 PM, Nithya Balachandran 
wrote:

> Ok, it looks like the same problem.
>
>
> @Amar, this fix is supposed to be in 4.0.1. Is it possible to regenerate
> the volfiles to fix this?
>
> Regards,
> Nithya
>
> On 17 April 2018 at 09:57, Artem Russakovskii  wrote:
>
>> pylon:/var/lib/glusterd/vols/dev_apkmirror_data # ack shared-brick-count
>> dev_apkmirror_data.pylon.mnt-pylon_block3-dev_apkmirror_data.vol
>> 3:option shared-brick-count 3
>>
>> dev_apkmirror_data.pylon.mnt-pylon_block2-dev_apkmirror_data.vol
>> 3:option shared-brick-count 3
>>
>> dev_apkmirror_data.pylon.mnt-pylon_block1-dev_apkmirror_data.vol
>> 3:option shared-brick-count 3
>>
>>
>> Sincerely,
>> Artem
>>
>> --
>> Founder, Android Police , APK Mirror
>> , Illogical Robot LLC
>> beerpla.net | +ArtemRussakovskii
>>  | @ArtemR
>> 
>>
>> On Mon, Apr 16, 2018 at 9:22 PM, Nithya Balachandran > > wrote:
>>
>>> Hi Artem,
>>>
>>> Was the volume size correct before the bricks were expanded?
>>>
>>> This sounds like [1] but that should have been fixed in 4.0.0. Can you
>>> let us know the values of shared-brick-count in the files in
>>> /var/lib/glusterd/vols/dev_apkmirror_data/ ?
>>>
>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1541880
>>>
>>> On 17 April 2018 at 05:17, Artem Russakovskii 
>>> wrote:
>>>
 Hi Nithya,

 I'm on Gluster 4.0.1.

 I don't think the bricks were smaller before - if they were, maybe 20GB
 because Linode's minimum is 20GB, then I extended them to 25GB, resized
 with resize2fs as instructed, and rebooted many times over since. Yet,
 gluster refuses to see the full disk size.

 Here's the status detail output:

 gluster volume status dev_apkmirror_data detail
 Status of volume: dev_apkmirror_data
 
 --
 Brick: Brick pylon:/mnt/pylon_block1/dev_apkmirror_data
 TCP Port : 49152
 RDMA Port: 0
 Online   : Y
 Pid  : 1263
 File System  : ext4
 Device   : /dev/sdd
 Mount Options: rw,relatime,data=ordered
 Inode Size   : 256
 Disk Space Free  : 23.0GB
 Total Disk Space : 24.5GB
 Inode Count  : 1638400
 Free Inodes  : 1625429
 
 --
 Brick: Brick pylon:/mnt/pylon_block2/dev_apkmirror_data
 TCP Port : 49153
 RDMA Port: 0
 Online   : Y
 Pid  : 1288
 File System  : ext4
 Device   : /dev/sdc
 Mount Options: rw,relatime,data=ordered
 Inode Size   : 256
 Disk Space Free  : 24.0GB
 Total Disk Space : 25.5GB
 Inode Count  : 1703936
 Free Inodes  : 1690965
 
 --

Re: [Gluster-users] Getting glusterfs to expand volume size to brick size

[Nithya Balachandran]

On 17 April 2018 at 10:03, Artem Russakovskii  wrote:

> I just remembered that I didn't run https://docs.gluster.org/
> en/v3/Upgrade-Guide/op_version/ for this test volume/box like I did for
> the main production gluster, and one of these ops - either heal or the
> op-version, resolved the issue.
>
> I'm now seeing:
> pylon:/var/lib/glusterd/vols/dev_apkmirror_data # ack shared-brick-count
> dev_apkmirror_data.pylon.mnt-pylon_block3-dev_apkmirror_data.vol
> 3:option shared-brick-count 1
>
> dev_apkmirror_data.pylon.mnt-pylon_block2-dev_apkmirror_data.vol
> 3:option shared-brick-count 1
>
> dev_apkmirror_data.pylon.mnt-pylon_block1-dev_apkmirror_data.vol
> 3:option shared-brick-count 1
>
>
>
> /dev/sdd25071M
> 1491M22284M   7% /mnt/pylon_block1
> /dev/sdc26079M
> 1491M23241M   7% /mnt/pylon_block2
> /dev/sde25071M
> 1491M22315M   7% /mnt/pylon_block3
>
> localhost:/dev_apkmirror_data   25071M
> 1742M22284M   8% /mnt/dev_apkmirror_data1
> localhost:/dev_apkmirror_data   25071M
> 1742M22284M   8% /mnt/dev_apkmirror_data2
> localhost:/dev_apkmirror_data   25071M
> 1742M22284M   8% /mnt/dev_apkmirror_data3
> localhost:/dev_apkmirror_data   25071M
> 1742M22284M   8% /mnt/dev_apkmirror_data_ganesha
>
>
> Problem is solved!
>
>
>
Excellent!



> Sincerely,
> Artem
>
> --
> Founder, Android Police , APK Mirror
> , Illogical Robot LLC
> beerpla.net | +ArtemRussakovskii
>  | @ArtemR
> 
>
> On Mon, Apr 16, 2018 at 9:29 PM, Nithya Balachandran 
> wrote:
>
>> Ok, it looks like the same problem.
>>
>>
>> @Amar, this fix is supposed to be in 4.0.1. Is it possible to regenerate
>> the volfiles to fix this?
>>
>> Regards,
>> Nithya
>>
>> On 17 April 2018 at 09:57, Artem Russakovskii 
>> wrote:
>>
>>> pylon:/var/lib/glusterd/vols/dev_apkmirror_data # ack shared-brick-count
>>> dev_apkmirror_data.pylon.mnt-pylon_block3-dev_apkmirror_data.vol
>>> 3:option shared-brick-count 3
>>>
>>> dev_apkmirror_data.pylon.mnt-pylon_block2-dev_apkmirror_data.vol
>>> 3:option shared-brick-count 3
>>>
>>> dev_apkmirror_data.pylon.mnt-pylon_block1-dev_apkmirror_data.vol
>>> 3:option shared-brick-count 3
>>>
>>>
>>> Sincerely,
>>> Artem
>>>
>>> --
>>> Founder, Android Police , APK Mirror
>>> , Illogical Robot LLC
>>> beerpla.net | +ArtemRussakovskii
>>>  | @ArtemR
>>> 
>>>
>>> On Mon, Apr 16, 2018 at 9:22 PM, Nithya Balachandran <
>>> nbala...@redhat.com> wrote:
>>>
 Hi Artem,

 Was the volume size correct before the bricks were expanded?

 This sounds like [1] but that should have been fixed in 4.0.0. Can you
 let us know the values of shared-brick-count in the files in
 /var/lib/glusterd/vols/dev_apkmirror_data/ ?

 [1] https://bugzilla.redhat.com/show_bug.cgi?id=1541880

 On 17 April 2018 at 05:17, Artem Russakovskii 
 wrote:

> Hi Nithya,
>
> I'm on Gluster 4.0.1.
>
> I don't think the bricks were smaller before - if they were, maybe
> 20GB because Linode's minimum is 20GB, then I extended them to 25GB,
> resized with resize2fs as instructed, and rebooted many times over since.
> Yet, gluster refuses to see the full disk size.
>
> Here's the status detail output:
>
> gluster volume status dev_apkmirror_data detail
> Status of volume: dev_apkmirror_data
> 
> --
> Brick: Brick pylon:/mnt/pylon_block1/dev_ap
> kmirror_data
> TCP Port : 49152
> RDMA Port: 0
> Online   : Y
> Pid  : 1263
> File System  : ext4
> Device   : /dev/sdd
> Mount Options: rw,relatime,data=ordered
> Inode Size   : 256
> Disk Space Free  : 23.0GB
> Total Disk Space : 24.5GB
> Inode Count  : 1638400
> Free Inodes  : 1625429
> 
> --
> Brick: Brick pylon:/mnt/pylon_block2/dev_ap
> kmirror_data
> TCP Port : 49153
> RDMA Port: 0
> Online   : Y
> Pid  : 1288
> File System  : ext4
> Device   : /dev/sdc
> Mount Options: 

Re: [Gluster-users] Getting glusterfs to expand volume size to brick size

[Artem Russakovskii]

Hi Nithya,

I'm on Gluster 4.0.1.

I don't think the bricks were smaller before - if they were, maybe 20GB
because Linode's minimum is 20GB, then I extended them to 25GB, resized
with resize2fs as instructed, and rebooted many times over since. Yet,
gluster refuses to see the full disk size.

Here's the status detail output:

gluster volume status dev_apkmirror_data detail
Status of volume: dev_apkmirror_data

--
Brick: Brick pylon:/mnt/pylon_block1/dev_apkmirror_data
TCP Port : 49152
RDMA Port: 0
Online   : Y
Pid  : 1263
File System  : ext4
Device   : /dev/sdd
Mount Options: rw,relatime,data=ordered
Inode Size   : 256
Disk Space Free  : 23.0GB
Total Disk Space : 24.5GB
Inode Count  : 1638400
Free Inodes  : 1625429

--
Brick: Brick pylon:/mnt/pylon_block2/dev_apkmirror_data
TCP Port : 49153
RDMA Port: 0
Online   : Y
Pid  : 1288
File System  : ext4
Device   : /dev/sdc
Mount Options: rw,relatime,data=ordered
Inode Size   : 256
Disk Space Free  : 24.0GB
Total Disk Space : 25.5GB
Inode Count  : 1703936
Free Inodes  : 1690965

--
Brick: Brick pylon:/mnt/pylon_block3/dev_apkmirror_data
TCP Port : 49154
RDMA Port: 0
Online   : Y
Pid  : 1313
File System  : ext4
Device   : /dev/sde
Mount Options: rw,relatime,data=ordered
Inode Size   : 256
Disk Space Free  : 23.0GB
Total Disk Space : 24.5GB
Inode Count  : 1638400
Free Inodes  : 1625433



What's interesting here is that the gluster volume size is exactly 1/3 of
the total (8357M * 3 = 25071M). Yet, each block device is separate, and the
total storage available is 25071M on each brick.

The fstab is as follows:
/dev/disk/by-id/scsi-0Linode_Volume_pylon_block1 /mnt/pylon_block1 ext4
defaults 0 2
/dev/disk/by-id/scsi-0Linode_Volume_pylon_block2 /mnt/pylon_block2 ext4
defaults 0 2
/dev/disk/by-id/scsi-0Linode_Volume_pylon_block3 /mnt/pylon_block3 ext4
defaults 0 2

localhost:/dev_apkmirror_data/mnt/dev_apkmirror_data1   glusterfs
defaults,_netdev,fopen-keep-cache,direct-io-mode=enable 0 0
localhost:/dev_apkmirror_data/mnt/dev_apkmirror_data2   glusterfs
defaults,_netdev,fopen-keep-cache,direct-io-mode=enable 0 0
localhost:/dev_apkmirror_data/mnt/dev_apkmirror_data3   glusterfs
defaults,_netdev,fopen-keep-cache,direct-io-mode=enable 0 0
localhost:/dev_apkmirror_data/mnt/dev_apkmirror_data_ganesha   nfs4
defaults,_netdev,bg,intr,soft,timeo=5,retrans=5,actimeo=10,retry=5 0 0

The latter entry is for an nfs ganesha test, in case it matters (which,
btw, fails miserably with all kinds of stability issues about broken pipes).

Note: this is a test server, so all 3 bricks are attached and mounted on
the same server.


Sincerely,
Artem

--
Founder, Android Police , APK Mirror
, Illogical Robot LLC
beerpla.net | +ArtemRussakovskii
 | @ArtemR


On Sun, Apr 15, 2018 at 10:56 PM, Nithya Balachandran 
wrote:

> What version of Gluster are you running? Were the bricks smaller earlier?
>
> Regards,
> Nithya
>
> On 15 April 2018 at 00:09, Artem Russakovskii  wrote:
>
>> Hi,
>>
>> I have a 3-brick replicate volume, but for some reason I can't get it to
>> expand to the size of the bricks. The bricks are 25GB, but even after
>> multiple gluster restarts and remounts, the volume is only about 8GB.
>>
>> I believed I could always extend the bricks (we're using Linode block
>> storage, which allows extending block devices after they're created), and
>> gluster would see the newly available space and extend to use it.
>>
>> Multiple Google searches, and I'm still nowhere. Any ideas?
>>
>> df | ack "block|data"
>> Filesystem   1M-blocks
>>  Used Available Use% Mounted on
>> /dev/sdd25071M
>> 1491M22284M   7% /mnt/pylon_block1
>> /dev/sdc26079M
>> 1491M23241M   7% /mnt/pylon_block2
>> /dev/sde25071M
>> 1491M22315M   7% /mnt/pylon_block3
>> localhost:/dev_apkmirror_data8357M
>>  581M 7428M   8% /mnt/dev_apkmirror_data1
>> localhost:/dev_apkmirror_data8357M
>>  581M 7428M   8% /mnt/dev_apkmirror_data2
>> 

Re: [Gluster-users] Gluster FUSE mount sometimes reports that files do not exist until ls is performed on parent directory

[Nithya Balachandran]

On 16 April 2018 at 14:07, Raghavendra Gowdappa  wrote:

>
>
> On Mon, Apr 16, 2018 at 1:54 PM, Niels Hendriks  wrote:
>
>> Hi,
>>
>> We have a 3-node gluster setup where gluster is both the server and the
>> client.
>> Every few days we have some $random file or directory that does not exist
>> according to the FUSE mountpoint. When we try to access the file (stat,
>> cat, etc...) the filesystem reports that the file/directory does not
>> exist,
>> even though it does. When we try to create the file/directory we receive
>> the following error which is also logged in
>> /var/log/glusterfs/bricks/$brick.log:
>>
>> [2018-04-10 12:51:26.755928] E [MSGID: 113027] [posix.c:1779:posix_mkdir]
>> 0-www-posix: mkdir of /storage/gluster/path/to/dir failed [File exists]
>>
>> We don't see this issue on all of the servers, but only on the servers
>> that
>> did not create the file/directory (so 2 of the 3 gluster nodes).
>>
>> We found that this issue does not resolve itself automatically. However,
>> when we perform an ls command on the parent directory the issue will be
>> resolved for the other nodes.
>>
>> We are running glusterfs 3.12.6 on debian 8
>>
>> Mount-options in /etc/fstab:
>> /dev/storage-gluster/gluster /storage/gluster xfs
>> rw,inode64,noatime,nouuid
>> 0 2
>> localhost:/www /var/www glusterfs
>> backup-volfile-servers=10.0.0.2:10.0.0.3,log-level=WARNING
>> 0 0
>>
>> gluster volume info www
>>
>> Volume Name: www
>> Type: Replicate
>> Volume ID: e0579d53-f671-4868-863b-ba85c4cfacb3
>> Status: Started
>> Snapshot Count: 0
>> Number of Bricks: 1 x 3 = 3
>> Transport-type: tcp
>> Bricks:
>> Brick1: n01c01-gluster:/storage/gluster/www
>> Brick2: n02c01-gluster:/storage/gluster/www
>> Brick3: n03c01-gluster:/storage/gluster/www
>> Options Reconfigured:
>> performance.read-ahead: on
>> performance.client-io-threads: on
>> nfs.disable: on
>> transport.address-family: inet
>> performance.md-cache-timeout: 600
>> diagnostics.brick-log-level: WARNING
>> network.ping-timeout: 3
>> features.cache-invalidation: on
>> server.event-threads: 4
>> performance.cache-invalidation: on
>> performance.quick-read: on
>> features.cache-invalidation-timeout: 600
>> network.inode-lru-limit: 9
>> performance.cache-priority: *.php:3,*.temp:3,*:1
>> performance.nl-cache: on
>> performance.cache-size: 1GB
>> performance.readdir-ahead: on
>> performance.write-behind: on
>> cluster.readdir-optimize: on
>> performance.io-thread-count: 64
>> client.event-threads: 4
>> cluster.lookup-optimize: on
>> performance.parallel-readdir: off
>> performance.write-behind-window-size: 4MB
>> performance.flush-behind: on
>> features.bitrot: on
>> features.scrub: Active
>> performance.io-cache: off
>> performance.stat-prefetch: on
>>
>> We suspected that the md-cache could be the cause, but it does have a
>> timeout of 600 seconds so this would be strange since the issue can be
>> present for hours (at which point we did an ls to fix it).
>>
>> Does anyone have an idea of what could be the cause of this?
>>
>
> For files, it could be because of:
> * cluster.lookup-optimize set to on
> * datafile is present on non hashed subvol, but linkto file is absent on
> hashed subvol
>
>
This is a pure replicate volume:

Type: Replicate
Volume ID: e0579d53-f671-4868-863b-ba85c4cfacb3
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3



So unlikely to be a lookup-optimize problem.



> I see that lookup-optimize is on. Can you get the following information
> from problematic file?
>
> * Name of the file
> * all xattrs on parent directory from all bricks
> * stat of file from all bricks where it is present.
> * all xattrs on file from all bricks where it is present.
>
> If you are seeing the problem on directory,
> * Name of directory
> * xattr of directory and its parent from all bricks
>
> regards,
> Raghavendra
>
>
>> Thanks!
>>
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster FUSE mount sometimes reports that files do not exist until ls is performed on parent directory

[Nithya Balachandran]

Hi Niels,

As this is a pure replicate volume, lookup-optimize is not going to be much
use so you can turn it off if you wish.

Do you see any error messages in the FUSE mount logs when this happens?
If it happens again, a tcpdump of the fuse mount would help.


Regards,
Nithya

On 16 April 2018 at 14:59, Nithya Balachandran  wrote:

>
>
> On 16 April 2018 at 14:07, Raghavendra Gowdappa 
> wrote:
>
>>
>>
>> On Mon, Apr 16, 2018 at 1:54 PM, Niels Hendriks  wrote:
>>
>>> Hi,
>>>
>>> We have a 3-node gluster setup where gluster is both the server and the
>>> client.
>>> Every few days we have some $random file or directory that does not exist
>>> according to the FUSE mountpoint. When we try to access the file (stat,
>>> cat, etc...) the filesystem reports that the file/directory does not
>>> exist,
>>> even though it does. When we try to create the file/directory we receive
>>> the following error which is also logged in
>>> /var/log/glusterfs/bricks/$brick.log:
>>>
>>> [2018-04-10 12:51:26.755928] E [MSGID: 113027] [posix.c:1779:posix_mkdir]
>>> 0-www-posix: mkdir of /storage/gluster/path/to/dir failed [File exists]
>>>
>>> We don't see this issue on all of the servers, but only on the servers
>>> that
>>> did not create the file/directory (so 2 of the 3 gluster nodes).
>>>
>>> We found that this issue does not resolve itself automatically. However,
>>> when we perform an ls command on the parent directory the issue will be
>>> resolved for the other nodes.
>>>
>>> We are running glusterfs 3.12.6 on debian 8
>>>
>>> Mount-options in /etc/fstab:
>>> /dev/storage-gluster/gluster /storage/gluster xfs
>>> rw,inode64,noatime,nouuid
>>> 0 2
>>> localhost:/www /var/www glusterfs
>>> backup-volfile-servers=10.0.0.2:10.0.0.3,log-level=WARNING
>>> 0 0
>>>
>>> gluster volume info www
>>>
>>> Volume Name: www
>>> Type: Replicate
>>> Volume ID: e0579d53-f671-4868-863b-ba85c4cfacb3
>>> Status: Started
>>> Snapshot Count: 0
>>> Number of Bricks: 1 x 3 = 3
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: n01c01-gluster:/storage/gluster/www
>>> Brick2: n02c01-gluster:/storage/gluster/www
>>> Brick3: n03c01-gluster:/storage/gluster/www
>>> Options Reconfigured:
>>> performance.read-ahead: on
>>> performance.client-io-threads: on
>>> nfs.disable: on
>>> transport.address-family: inet
>>> performance.md-cache-timeout: 600
>>> diagnostics.brick-log-level: WARNING
>>> network.ping-timeout: 3
>>> features.cache-invalidation: on
>>> server.event-threads: 4
>>> performance.cache-invalidation: on
>>> performance.quick-read: on
>>> features.cache-invalidation-timeout: 600
>>> network.inode-lru-limit: 9
>>> performance.cache-priority: *.php:3,*.temp:3,*:1
>>> performance.nl-cache: on
>>> performance.cache-size: 1GB
>>> performance.readdir-ahead: on
>>> performance.write-behind: on
>>> cluster.readdir-optimize: on
>>> performance.io-thread-count: 64
>>> client.event-threads: 4
>>> cluster.lookup-optimize: on
>>> performance.parallel-readdir: off
>>> performance.write-behind-window-size: 4MB
>>> performance.flush-behind: on
>>> features.bitrot: on
>>> features.scrub: Active
>>> performance.io-cache: off
>>> performance.stat-prefetch: on
>>>
>>> We suspected that the md-cache could be the cause, but it does have a
>>> timeout of 600 seconds so this would be strange since the issue can be
>>> present for hours (at which point we did an ls to fix it).
>>>
>>> Does anyone have an idea of what could be the cause of this?
>>>
>>
>> For files, it could be because of:
>> * cluster.lookup-optimize set to on
>> * datafile is present on non hashed subvol, but linkto file is absent on
>> hashed subvol
>>
>>
> This is a pure replicate volume:
>
> Type: Replicate
> Volume ID: e0579d53-f671-4868-863b-ba85c4cfacb3
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x 3 = 3
>
>
>
> So unlikely to be a lookup-optimize problem.
>
>
>
>> I see that lookup-optimize is on. Can you get the following information
>> from problematic file?
>>
>> * Name of the file
>> * all xattrs on parent directory from all bricks
>> * stat of file from all bricks where it is present.
>> * all xattrs on file from all bricks where it is present.
>>
>> If you are seeing the problem on directory,
>> * Name of directory
>> * xattr of directory and its parent from all bricks
>>
>> regards,
>> Raghavendra
>>
>>
>>> Thanks!
>>>
>>>
>>> ___
>>> Gluster-users mailing list
>>> Gluster-users@gluster.org
>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users