On 17 April 2018 at 10:03, Artem Russakovskii <[email protected]> wrote:
> I just remembered that I didn't run https://docs.gluster.org/ > en/v3/Upgrade-Guide/op_version/ for this test volume/box like I did for > the main production gluster, and one of these ops - either heal or the > op-version, resolved the issue. > > I'm now seeing: > pylon:/var/lib/glusterd/vols/dev_apkmirror_data # ack shared-brick-count > dev_apkmirror_data.pylon.mnt-pylon_block3-dev_apkmirror_data.vol > 3: option shared-brick-count 1 > > dev_apkmirror_data.pylon.mnt-pylon_block2-dev_apkmirror_data.vol > 3: option shared-brick-count 1 > > dev_apkmirror_data.pylon.mnt-pylon_block1-dev_apkmirror_data.vol > 3: option shared-brick-count 1 > > > > /dev/sdd 25071M > 1491M 22284M 7% /mnt/pylon_block1 > /dev/sdc 26079M > 1491M 23241M 7% /mnt/pylon_block2 > /dev/sde 25071M > 1491M 22315M 7% /mnt/pylon_block3 > > localhost:/dev_apkmirror_data 25071M > 1742M 22284M 8% /mnt/dev_apkmirror_data1 > localhost:/dev_apkmirror_data 25071M > 1742M 22284M 8% /mnt/dev_apkmirror_data2 > localhost:/dev_apkmirror_data 25071M > 1742M 22284M 8% /mnt/dev_apkmirror_data3 > localhost:/dev_apkmirror_data 25071M > 1742M 22284M 8% /mnt/dev_apkmirror_data_ganesha > > > Problem is solved! > > > Excellent! > Sincerely, > Artem > > -- > Founder, Android Police <http://www.androidpolice.com>, APK Mirror > <http://www.apkmirror.com/>, Illogical Robot LLC > beerpla.net | +ArtemRussakovskii > <https://plus.google.com/+ArtemRussakovskii> | @ArtemR > <http://twitter.com/ArtemR> > > On Mon, Apr 16, 2018 at 9:29 PM, Nithya Balachandran <[email protected]> > wrote: > >> Ok, it looks like the same problem. >> >> >> @Amar, this fix is supposed to be in 4.0.1. Is it possible to regenerate >> the volfiles to fix this? >> >> Regards, >> Nithya >> >> On 17 April 2018 at 09:57, Artem Russakovskii <[email protected]> >> wrote: >> >>> pylon:/var/lib/glusterd/vols/dev_apkmirror_data # ack shared-brick-count >>> dev_apkmirror_data.pylon.mnt-pylon_block3-dev_apkmirror_data.vol >>> 3: option shared-brick-count 3 >>> >>> dev_apkmirror_data.pylon.mnt-pylon_block2-dev_apkmirror_data.vol >>> 3: option shared-brick-count 3 >>> >>> dev_apkmirror_data.pylon.mnt-pylon_block1-dev_apkmirror_data.vol >>> 3: option shared-brick-count 3 >>> >>> >>> Sincerely, >>> Artem >>> >>> -- >>> Founder, Android Police <http://www.androidpolice.com>, APK Mirror >>> <http://www.apkmirror.com/>, Illogical Robot LLC >>> beerpla.net | +ArtemRussakovskii >>> <https://plus.google.com/+ArtemRussakovskii> | @ArtemR >>> <http://twitter.com/ArtemR> >>> >>> On Mon, Apr 16, 2018 at 9:22 PM, Nithya Balachandran < >>> [email protected]> wrote: >>> >>>> Hi Artem, >>>> >>>> Was the volume size correct before the bricks were expanded? >>>> >>>> This sounds like [1] but that should have been fixed in 4.0.0. Can you >>>> let us know the values of shared-brick-count in the files in >>>> /var/lib/glusterd/vols/dev_apkmirror_data/ ? >>>> >>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1541880 >>>> >>>> On 17 April 2018 at 05:17, Artem Russakovskii <[email protected]> >>>> wrote: >>>> >>>>> Hi Nithya, >>>>> >>>>> I'm on Gluster 4.0.1. >>>>> >>>>> I don't think the bricks were smaller before - if they were, maybe >>>>> 20GB because Linode's minimum is 20GB, then I extended them to 25GB, >>>>> resized with resize2fs as instructed, and rebooted many times over since. >>>>> Yet, gluster refuses to see the full disk size. >>>>> >>>>> Here's the status detail output: >>>>> >>>>> gluster volume status dev_apkmirror_data detail >>>>> Status of volume: dev_apkmirror_data >>>>> ------------------------------------------------------------ >>>>> ------------------ >>>>> Brick : Brick pylon:/mnt/pylon_block1/dev_ap >>>>> kmirror_data >>>>> TCP Port : 49152 >>>>> RDMA Port : 0 >>>>> Online : Y >>>>> Pid : 1263 >>>>> File System : ext4 >>>>> Device : /dev/sdd >>>>> Mount Options : rw,relatime,data=ordered >>>>> Inode Size : 256 >>>>> Disk Space Free : 23.0GB >>>>> Total Disk Space : 24.5GB >>>>> Inode Count : 1638400 >>>>> Free Inodes : 1625429 >>>>> ------------------------------------------------------------ >>>>> ------------------ >>>>> Brick : Brick pylon:/mnt/pylon_block2/dev_ap >>>>> kmirror_data >>>>> TCP Port : 49153 >>>>> RDMA Port : 0 >>>>> Online : Y >>>>> Pid : 1288 >>>>> File System : ext4 >>>>> Device : /dev/sdc >>>>> Mount Options : rw,relatime,data=ordered >>>>> Inode Size : 256 >>>>> Disk Space Free : 24.0GB >>>>> Total Disk Space : 25.5GB >>>>> Inode Count : 1703936 >>>>> Free Inodes : 1690965 >>>>> ------------------------------------------------------------ >>>>> ------------------ >>>>> Brick : Brick pylon:/mnt/pylon_block3/dev_ap >>>>> kmirror_data >>>>> TCP Port : 49154 >>>>> RDMA Port : 0 >>>>> Online : Y >>>>> Pid : 1313 >>>>> File System : ext4 >>>>> Device : /dev/sde >>>>> Mount Options : rw,relatime,data=ordered >>>>> Inode Size : 256 >>>>> Disk Space Free : 23.0GB >>>>> Total Disk Space : 24.5GB >>>>> Inode Count : 1638400 >>>>> Free Inodes : 1625433 >>>>> >>>>> >>>>> >>>>> What's interesting here is that the gluster volume size is exactly 1/3 >>>>> of the total (8357M * 3 = 25071M). Yet, each block device is separate, and >>>>> the total storage available is 25071M on each brick. >>>>> >>>>> The fstab is as follows: >>>>> /dev/disk/by-id/scsi-0Linode_Volume_pylon_block1 /mnt/pylon_block1 >>>>> ext4 defaults 0 2 >>>>> /dev/disk/by-id/scsi-0Linode_Volume_pylon_block2 /mnt/pylon_block2 >>>>> ext4 defaults 0 2 >>>>> /dev/disk/by-id/scsi-0Linode_Volume_pylon_block3 /mnt/pylon_block3 >>>>> ext4 defaults 0 2 >>>>> >>>>> localhost:/dev_apkmirror_data /mnt/dev_apkmirror_data1 glusterfs >>>>> defaults,_netdev,fopen-keep-cache,direct-io-mode=enable 0 0 >>>>> localhost:/dev_apkmirror_data /mnt/dev_apkmirror_data2 glusterfs >>>>> defaults,_netdev,fopen-keep-cache,direct-io-mode=enable 0 0 >>>>> localhost:/dev_apkmirror_data /mnt/dev_apkmirror_data3 glusterfs >>>>> defaults,_netdev,fopen-keep-cache,direct-io-mode=enable 0 0 >>>>> localhost:/dev_apkmirror_data /mnt/dev_apkmirror_data_ganesha >>>>> nfs4 defaults,_netdev,bg,intr,soft,timeo=5,retrans=5,actimeo=10,retry=5 >>>>> 0 0 >>>>> >>>>> The latter entry is for an nfs ganesha test, in case it matters >>>>> (which, btw, fails miserably with all kinds of stability issues about >>>>> broken pipes). >>>>> >>>>> Note: this is a test server, so all 3 bricks are attached and mounted >>>>> on the same server. >>>>> >>>>> >>>>> Sincerely, >>>>> Artem >>>>> >>>>> -- >>>>> Founder, Android Police <http://www.androidpolice.com>, APK Mirror >>>>> <http://www.apkmirror.com/>, Illogical Robot LLC >>>>> beerpla.net | +ArtemRussakovskii >>>>> <https://plus.google.com/+ArtemRussakovskii> | @ArtemR >>>>> <http://twitter.com/ArtemR> >>>>> >>>>> On Sun, Apr 15, 2018 at 10:56 PM, Nithya Balachandran < >>>>> [email protected]> wrote: >>>>> >>>>>> What version of Gluster are you running? Were the bricks smaller >>>>>> earlier? >>>>>> >>>>>> Regards, >>>>>> Nithya >>>>>> >>>>>> On 15 April 2018 at 00:09, Artem Russakovskii <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I have a 3-brick replicate volume, but for some reason I can't get >>>>>>> it to expand to the size of the bricks. The bricks are 25GB, but even >>>>>>> after >>>>>>> multiple gluster restarts and remounts, the volume is only about 8GB. >>>>>>> >>>>>>> I believed I could always extend the bricks (we're using Linode >>>>>>> block storage, which allows extending block devices after they're >>>>>>> created), >>>>>>> and gluster would see the newly available space and extend to use it. >>>>>>> >>>>>>> Multiple Google searches, and I'm still nowhere. Any ideas? >>>>>>> >>>>>>> df | ack "block|data" >>>>>>> Filesystem >>>>>>> 1M-blocks Used Available Use% Mounted on >>>>>>> /dev/sdd >>>>>>> 25071M 1491M 22284M 7% /mnt/pylon_block1 >>>>>>> /dev/sdc >>>>>>> 26079M 1491M 23241M 7% /mnt/pylon_block2 >>>>>>> /dev/sde >>>>>>> 25071M 1491M 22315M 7% /mnt/pylon_block3 >>>>>>> localhost:/dev_apkmirror_data >>>>>>> 8357M 581M 7428M 8% /mnt/dev_apkmirror_data1 >>>>>>> localhost:/dev_apkmirror_data >>>>>>> 8357M 581M 7428M 8% /mnt/dev_apkmirror_data2 >>>>>>> localhost:/dev_apkmirror_data >>>>>>> 8357M 581M 7428M 8% /mnt/dev_apkmirror_data3 >>>>>>> >>>>>>> >>>>>>> >>>>>>> gluster volume info >>>>>>> >>>>>>> Volume Name: dev_apkmirror_data >>>>>>> Type: Replicate >>>>>>> Volume ID: cd5621ee-7fab-401b-b720-08863717ed56 >>>>>>> Status: Started >>>>>>> Snapshot Count: 0 >>>>>>> Number of Bricks: 1 x 3 = 3 >>>>>>> Transport-type: tcp >>>>>>> Bricks: >>>>>>> Brick1: pylon:/mnt/pylon_block1/dev_apkmirror_data >>>>>>> Brick2: pylon:/mnt/pylon_block2/dev_apkmirror_data >>>>>>> Brick3: pylon:/mnt/pylon_block3/dev_apkmirror_data >>>>>>> Options Reconfigured: >>>>>>> disperse.eager-lock: off >>>>>>> cluster.lookup-unhashed: auto >>>>>>> cluster.read-hash-mode: 0 >>>>>>> performance.strict-o-direct: on >>>>>>> cluster.shd-max-threads: 12 >>>>>>> performance.nl-cache-timeout: 600 >>>>>>> performance.nl-cache: on >>>>>>> cluster.quorum-count: 1 >>>>>>> cluster.quorum-type: fixed >>>>>>> network.ping-timeout: 5 >>>>>>> network.remote-dio: enable >>>>>>> performance.rda-cache-limit: 256MB >>>>>>> performance.parallel-readdir: on >>>>>>> network.inode-lru-limit: 500000 >>>>>>> performance.md-cache-timeout: 600 >>>>>>> performance.cache-invalidation: on >>>>>>> performance.stat-prefetch: on >>>>>>> features.cache-invalidation-timeout: 600 >>>>>>> features.cache-invalidation: on >>>>>>> performance.io-thread-count: 32 >>>>>>> server.event-threads: 4 >>>>>>> client.event-threads: 4 >>>>>>> performance.read-ahead: off >>>>>>> cluster.lookup-optimize: on >>>>>>> performance.client-io-threads: on >>>>>>> performance.cache-size: 1GB >>>>>>> transport.address-family: inet >>>>>>> performance.readdir-ahead: on >>>>>>> nfs.disable: on >>>>>>> cluster.readdir-optimize: on >>>>>>> >>>>>>> >>>>>>> Thank you. >>>>>>> >>>>>>> Sincerely, >>>>>>> Artem >>>>>>> >>>>>>> -- >>>>>>> Founder, Android Police <http://www.androidpolice.com>, APK Mirror >>>>>>> <http://www.apkmirror.com/>, Illogical Robot LLC >>>>>>> beerpla.net | +ArtemRussakovskii >>>>>>> <https://plus.google.com/+ArtemRussakovskii> | @ArtemR >>>>>>> <http://twitter.com/ArtemR> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Gluster-users mailing list >>>>>>> [email protected] >>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >
_______________________________________________ Gluster-users mailing list [email protected] http://lists.gluster.org/mailman/listinfo/gluster-users
