Re: [Gluster-users] Client disconnections, memory use

[Nithya Balachandran]

Hi,

For the memory increase, please capture statedumps of the process at
intervals of an hour and send it across.
https://docs.gluster.org/en/latest/Troubleshooting/statedump/ describes how
to generate a statedump for the client process.

Regards,
Nithya

On Wed, 13 Nov 2019 at 05:18, Jamie Lawrence 
wrote:

> Glusternauts,
>
> I have a 3x3 cluster running 5.9 under Ubuntu 16.04. We migrated clients
> from a different, much older, cluster. Those clients are running 5.9
> clients, and spontaneously disconnect. It was signal 15, but no user killed
> it, and I can't imagine why another daemon would have.
>
>
> [2019-11-12 22:52:42.790687] I [fuse-bridge.c:5144:fuse_thread_proc]
> 0-fuse: initating unmount of /mnt/informatica/sftp/dectools
> [2019-11-12 22:52:42.791414] W [glusterfsd.c:1500:cleanup_and_exit]
> (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f141e4466ba]
> -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xed) [0x55711c79994d]
> -->/usr/sbin/glusterfs(cleanup_and_exit+0x54) [0x55711c7997b4] ) 0-:
> received signum (15), shutting down
> [2019-11-12 22:52:42.791435] I [fuse-bridge.c:5914:fini] 0-fuse:
> Unmounting '/mnt/informatica/sftp/dectools'.
> [2019-11-12 22:52:42.791444] I [fuse-bridge.c:5919:fini] 0-fuse: Closing
> fuse connection to '/mnt/informatica/sftp/dectools'.
>
> Nothing in the log for about 12 minutes previously.
>
> Volume info:
>
> Volume Name: sc5_informatica_prod_shared
> Type: Distributed-Replicate
> Volume ID: db5d2693-59e1-40e0-9c28-7a2385b2524f
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 3 x 3 = 9
> Transport-type: tcp
> Bricks:
> Brick1: sc5-storage-1:/gluster-bricks/pool-1/sc5_informatica_prod_shared
> Brick2: sc5-storage-2:/gluster-bricks/pool-1/sc5_informatica_prod_shared
> Brick3: sc5-storage-3:/gluster-bricks/pool-1/sc5_informatica_prod_shared
> Brick4: sc5-storage-4:/gluster-bricks/pool-1/sc5_informatica_prod_shared
> Brick5: sc5-storage-5:/gluster-bricks/pool-1/sc5_informatica_prod_shared
> Brick6: sc5-storage-6:/gluster-bricks/pool-1/sc5_informatica_prod_shared
> Brick7: sc5-storage-7:/gluster-bricks/pool-1/sc5_informatica_prod_shared
> Brick8: sc5-storage-8:/gluster-bricks/pool-1/sc5_informatica_prod_shared
> Brick9: sc5-storage-9:/gluster-bricks/pool-1/sc5_informatica_prod_shared
> Options Reconfigured:
> performance.readdir-ahead: disable
> performance.quick-read: disable
> features.quota-deem-statfs: on
> features.inode-quota: on
> features.quota: on
> transport.address-family: inet
> nfs.disable: on
> performance.client-io-threads: off
>
>
> One very disturbing thing I'm noticing is that memory use on the client
> seems to be growing at rate of about 1MB/10 minutes of active use. One
> glusterfs process I'm looking at is consuming about 2.4G right now and
> growing. Does 5.9 have a memory leak, too?
>
>
> -j
>
> Community Meeting Calendar:
>
> APAC Schedule -
> Every 2nd and 4th Tuesday at 11:30 AM IST
> Bridge: https://bluejeans.com/118564314
>
> NA/EMEA Schedule -
> Every 1st and 3rd Tuesday at 01:00 PM EDT
> Bridge: https://bluejeans.com/118564314
>
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Symbolic Links under .glusterfs deleted

[Nithya Balachandran]

On Thu, 7 Nov 2019 at 15:15, Shreyansh Shah 
wrote:

> Hi,
> Running distributed gluster 5.10 with 6 node and 2 bricks on each node (12
> in total)
> Due to some reason the files under /.glusterfs were deleted.
> Post that we have loads of broken symlinks, but the data on the disks exist
> Due to this, even though the file is present, ls on the folder does not
> show the file, but I can access the file using the complete file path.
>

This is due to the way the ls is implemented on the brick as well as
because dht strips out files with invalid stats in the listing.
How did the symlinks go missing? Were the contents of the .glusterfs
directory deleted?

Regards,
Nithya


>
> Steps to reproduce:
> Stop gluster
> Umount brick
> Run `rm -rf /.glusterfs/
> Mount brick
> Start gluster
>
> The total storage space remains the same, we can access the data using
> absolute path, but
> we are not able to see the files in the folder.
> This is very critical setup for us, we cannot afford any data loss and are
> heavily dependent.
> Kindly revert back with the solution at the earliest.
>
> --
> Regards,
> Shreyansh Shah
> 
>
> Community Meeting Calendar:
>
> APAC Schedule -
> Every 2nd and 4th Tuesday at 11:30 AM IST
> Bridge: https://bluejeans.com/118564314
>
> NA/EMEA Schedule -
> Every 1st and 3rd Tuesday at 01:00 PM EDT
> Bridge: https://bluejeans.com/118564314
>
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Disappearing files on gluster mount

[Nithya Balachandran]

Good to hear. Thanks for the update.

Regards,
Nithya

On Fri, 20 Sep 2019 at 22:01, Patrick Riehecky  wrote:

> I upgraded to gluster 5 and the problem seems to have vanished.  I'd say
> I'm fixed (though not sure how I was broken)
>
> Pat
>
> --
> Pat Riehecky
>
> Fermi National Accelerator Laboratory
> www.fnal.gov
> www.scientificlinux.org
>
>
> ________
> From: Nithya Balachandran 
> Sent: Thursday, September 19, 2019 10:14 PM
> To: Patrick Riehecky
> Cc: gluster-users
> Subject: Re: [Gluster-users] Disappearing files on gluster mount
>
> Hi Pat,
>
> Do you still see the problem of missing files? If yes please provide the
> following :
>
> 1. gluster volume info
> 2. ls -l of the directory containing the missing files from the mount
> point and from the individual bricks.
>
>
> Regards,
> Niyhya
>
>
> On Thu, 29 Aug 2019 at 18:57, Pat Riehecky  riehe...@fnal.gov>> wrote:
> I moved my wife's photo archive to a mirrored gluster volume. Lately,
> she's noticed that a number of files are missing.  I'm pretty sure they
> are still in the .glusterfs dir as no one deleted them, but they simply
> don't display
>
> Any ideas how to get the files to reappear?
>
> Glusterfs 3.14
>
> --
> Pat Riehecky
>
> Fermi National Accelerator Laboratory
> www.fnal.gov<http://www.fnal.gov>
> www.scientificlinux.org<
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.scientificlinux.org=DwMFaQ=gRgGjJ3BkIsb5y6s49QqsA=OAMtP0DWou0nlXG7Kmxo2enjXJfwb1DXS9fwcaESuTE=ri4Ypm0YWvxU1BH81WaUOvlm6GJ71tWjG-z5Xz-St7g=imhcObrvaSWONJNfdbENWVdOPXGOIMzG2mL25zDHgTc=
> >
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org<mailto:Gluster-users@gluster.org>
> https://lists.gluster.org/mailman/listinfo/gluster-users<
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.gluster.org_mailman_listinfo_gluster-2Dusers=DwMFaQ=gRgGjJ3BkIsb5y6s49QqsA=OAMtP0DWou0nlXG7Kmxo2enjXJfwb1DXS9fwcaESuTE=ri4Ypm0YWvxU1BH81WaUOvlm6GJ71tWjG-z5Xz-St7g=4x41YyeQIQbelJJkPO4cqC8dZEcv5deWO-EfBKpLF5Q=
> >
>


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] File visible in listing but inaccessible over libgfapi

[Nithya Balachandran]

On Thu, 19 Sep 2019 at 15:40, Milewski Daniel 
wrote:

> I've observed an interesting behavior in Gluster 5.6. I had a file
> which was placed on incorrect subvolume (aparrently by the rebalancing
> process). I could stat and read the file just fine over FUSE mount
> point, with this entry appearing in log file:
>
> [2019-09-18 13:00:04.484683] D [MSGID: 0]
> [dht-common.c:3030:dht_lookup_cbk] 0-portal-shared-dht: Entry
> /9d33b830-6fc9-4190-a33a-19940a3a8589/images/8130a8db-e3b0-4345-b040-72df1252faaf/d486dfc8-0ad5-4253-a586-e
> 18f47139a31 missing on subvol portal-shared-replicate-2
>
> However, when using libgfapi the file was visible in directory listing,
> but I could neither stat nor read it. I fixed this by running
> rebalance.
>
> I have two questions:
>
> 1. Why the behavior is different between these two access methods? From
> what I know, when a file can't be find on a subvolume it belongs to
> (according to file name's hash value) Gluster looks for file on any
> other subvolume.
>


We recently enabled lookup-optimize  for volumes by default. In this case,
gluster will not look on other subvols if it doesn't find it on the hashed
subvol.
You can disable this option to get rid of the issue permanently.

2. Could replace-brick operation trigger rebalance by itself? As far as
> I know, the only operation that can trigger rebalace is remove-brick,
> but I've found out that rebalance process is running in oVirt events
> tab after I've run replace-brick. The message said:
>
> Replace-brick should not trigger rebalance.

Regards,
Nithya


> Detected start of rebalance on volume portal-shared of Cluster Portal from
> CLI.
>
> However, I'm pretty sure I didn't run it manually, nor could I find any
> trace of this in cmd_history.log.
>
>
> Spółki Grupy Wirtualna Polska:
>
> Wirtualna Polska Holding Spółka Akcyjna z siedzibą w Warszawie, ul.
> Jutrzenki 137A, 02-231 Warszawa, wpisana do Krajowego Rejestru Sądowego -
> Rejestru Przedsiębiorców prowadzonego przez Sąd Rejonowy dla m.st.
> Warszawy w Warszawie pod nr KRS: 407130, kapitał zakładowy: 1 445
> 199,00 zł (w całości wpłacony), Numer Identyfikacji Podatkowej (NIP):
> 521-31-11-513
>
> Wirtualna Polska Media Spółka Akcyjna z siedzibą w Warszawie, ul.
> Jutrzenki 137A, 02-231 Warszawa, wpisana do Krajowego Rejestru Sądowego -
> Rejestru Przedsiębiorców prowadzonego przez Sąd Rejonowy dla m.st.
> Warszawy w Warszawie pod nr KRS: 580004, kapitał zakładowy: 320 005
> 950,00 zł (w całości wpłacony), Numer Identyfikacji Podatkowej (NIP):
> 527-26-45-593
>
> Administratorem udostępnionych danych osobowych jest Wirtualna Polska
> Media S.A. z siedzibą w Warszawie (dalej „WPM”). WPM przetwarza Twoje dane
> osobowe, które zostały podane przez Ciebie dobrowolnie w trakcie
> dotychczasowej współpracy, w związku z zawarciem umowy lub zostały zebrane
> ze źródeł powszechnie dostępnych, w szczególności: imię i nazwisko, adres
> email, numer telefonu. Przetwarzamy te dane w celach opisanych w polityce
> prywatności, między innymi w celu
> realizacji współpracy, realizacji obowiązków przewidzianych prawem, w
> celach marketingowych WP. Podstawą prawną przetwarzania Twoich danych
> osobowych w celach marketingowych jest prawnie uzasadniony interes jakim
> jest m.in. przesyłanie informacji marketingowych o usługach WP, w tym
> zaproszeń na konferencje branżowe, informacje o publikacjach. Twoje dane
> możemy przekazywać podmiotom przetwarzającym je na nasze zlecenie oraz
> podmiotom uprawnionym do uzyskania danych na podstawie obowiązującego
> prawa. Masz prawo m.in. do żądania dostępu do danych, sprostowania,
> usunięcia lub ograniczenia ich przetwarzania, jak również prawo do
> zgłoszenia sprzeciwu w przewidzianych w prawie sytuacjach. Prawa te oraz
> sposób ich realizacji opisaliśmy w polityce prywatności<
> https://onas.wp.pl/poufnosc.html>. Tam też znajdziesz informacje jak
> zakomunikować nam Twoją wolę skorzystania z tych praw.
> 
>
> Community Meeting Calendar:
>
> APAC Schedule -
> Every 2nd and 4th Tuesday at 11:30 AM IST
> Bridge: https://bluejeans.com/118564314
>
> NA/EMEA Schedule -
> Every 1st and 3rd Tuesday at 01:00 PM EDT
> Bridge: https://bluejeans.com/118564314
>
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Disappearing files on gluster mount

[Nithya Balachandran]

Hi Pat,

Do you still see the problem of missing files? If yes please provide the
following :

1. gluster volume info
2. ls -l of the directory containing the missing files from the mount point
and from the individual bricks.


Regards,
Niyhya


On Thu, 29 Aug 2019 at 18:57, Pat Riehecky  wrote:

> I moved my wife's photo archive to a mirrored gluster volume. Lately,
> she's noticed that a number of files are missing.  I'm pretty sure they
> are still in the .glusterfs dir as no one deleted them, but they simply
> don't display
>
> Any ideas how to get the files to reappear?
>
> Glusterfs 3.14
>
> --
> Pat Riehecky
>
> Fermi National Accelerator Laboratory
> www.fnal.gov
> www.scientificlinux.org
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Rebalancing newly added bricks

[Nithya Balachandran]

On Sat, 14 Sep 2019 at 01:25, Herb Burnswell 
wrote:

> Hi,
>
> Well our rebalance seems to have failed.  Here is the output:
>

Hi,

Rebalance will abort itself if it cannot reach any of the nodes. Are all
the bricks still up and reachable?

Regards,
Nithya




>
> # gluster vol rebalance tank status
> Node Rebalanced-files  size
> scanned  failures   skipped   status  run time in
> h:m:s
>-  ---   ---
> ---   ---   --- 
> --
>localhost  134870657.8TB
> 2234439 9 6   failed  190:24:3
>serverB 0
>  0Bytes 7 0 0completed
>   63:47:55
> volume rebalance: tank: success
>
> # gluster vol status tank
> Status of volume: tank
> Gluster process TCP Port  RDMA Port  Online
>  Pid
>
> --
> Brick serverA:/gluster_bricks/data1   49162 0  Y
> 20318
> Brick serverB:/gluster_bricks/data1   49166 0  Y
> 3432
> Brick serverA:/gluster_bricks/data2   49163 0  Y
> 20323
> Brick serverB:/gluster_bricks/data2   49167 0  Y
> 3435
> Brick serverA:/gluster_bricks/data3   49164 0  Y
> 4625
> Brick serverA:/gluster_bricks/data4   49165 0  Y
> 4644
> Brick serverA:/gluster_bricks/data5   49166 0  Y
> 5088
> Brick serverA:/gluster_bricks/data6   49167 0  Y
> 5128
> Brick serverB:/gluster_bricks/data3   49168 0  Y
> 22314
> Brick serverB:/gluster_bricks/data4   49169 0  Y
> 22345
> Brick serverB:/gluster_bricks/data5   49170 0  Y
> 22889
> Brick serverB:/gluster_bricks/data6   49171 0  Y
> 22932
> Self-heal Daemon on localhost   N/A   N/AY
> 6202
> Self-heal Daemon on serverB   N/A   N/AY
> 22981
>
> Task Status of Volume tank
>
> --
> Task : Rebalance
> ID   : eec64343-8e0d-4523-ad05-5678f9eb9eb2
> Status   : failed
>
> # df -hP |grep data
> /dev/mapper/gluster_vg-gluster_lv1_data   60T   31T   29T  52%
> /gluster_bricks/data1
> /dev/mapper/gluster_vg-gluster_lv2_data   60T   31T   29T  51%
> /gluster_bricks/data2
> /dev/mapper/gluster_vg-gluster_lv3_data   60T   15T   46T  24%
> /gluster_bricks/data3
> /dev/mapper/gluster_vg-gluster_lv4_data   60T   15T   46T  24%
> /gluster_bricks/data4
> /dev/mapper/gluster_vg-gluster_lv5_data   60T   15T   45T  25%
> /gluster_bricks/data5
> /dev/mapper/gluster_vg-gluster_lv6_data   60T   15T   45T  25%
> /gluster_bricks/data6
>
>
> The rebalance log on serverA shows a disconnect from serverB
>
> [2019-09-08 15:41:44.285591] C
> [rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired] 0-tank-client-10: server
> :49170 has not responded in the last 42 seconds, disconnecting.
> [2019-09-08 15:41:44.285739] I [MSGID: 114018]
> [client.c:2280:client_rpc_notify] 0-tank-client-10: disconnected from
> tank-client-10. Client process will keep trying to connect to glusterd
> until brick's port is available
> [2019-09-08 15:41:44.286023] E [rpc-clnt.c:365:saved_frames_unwind] (-->
> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7ff986e8b132] (-->
> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7ff986c5299e] (-->
> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff986c52aae] (-->
> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7ff986c54220] (-->
> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x7ff986c54ce0] )
> 0-tank-client-10: forced unwinding frame type(GlusterFS 3.3)
> op(FXATTROP(34)) called at 2019-09-08 15:40:44.040333 (xid=0x7f8cfac)
>
> Does this type of failure cause data corruption?  What is the best course
> of action at this point?
>
> Thanks,
>
> HB
>
> On Wed, Sep 11, 2019 at 11:58 PM Strahil  wrote:
>
>> Hi Nithya,
>>
>> Thanks for the detailed explanation.
>> It makes sense.
>>
>> Best Regards,
>> Strahil Nikolov
>> On Sep 12, 2019 08:18, Nithya Balachandran  wrote:
>>
>>
>>
>> On Wed, 11 Sep 2019 at 09:47, Strahil  wrote:
>>
>> Hi Nithya,
>>
>> I just reminded about your previous  e-mail  which left me with the
>> im

Re: [Gluster-users] Rebalancing newly added bricks

[Nithya Balachandran]

On Wed, 11 Sep 2019 at 09:47, Strahil  wrote:

> Hi Nithya,
>
> I just reminded about your previous  e-mail  which left me with the
> impression that old volumes need that.
> This is the one 1 mean:
>
> >It looks like this is a replicate volume. If >that is the case then yes,
> you are >running an old version of Gluster for >which this was the default
>

Hi Strahil,

I'm providing a little more detail here which I hope will explain things.
Rebalance was always a volume wide operation - a *rebalance start*
operation will start rebalance processes on all nodes of the volume.
However, different processes would behave differently. In earlier releases,
all nodes would crawl the bricks and update the directory layouts. However,
only one node in each replica/disperse set would actually migrate files,so
the rebalance status would only show one node doing any "work" (scanning,
rebalancing etc). However, this one node will process all the files in its
replica sets. Rerunning rebalance on other nodes would make no difference
as it will always be the same node that ends up migrating files.
So for instance, for a replicate volume with server1:/brick1,
server2:/brick2 and server3:/brick3 in that order, only the rebalance
process on server1 would migrate files. In newer releases, all 3 nodes
would migrate files.

The rebalance status does not capture the directory operations of fixing
layouts which is why it looks like the other nodes are not doing anything.

Hope this helps.

Regards,
Nithya

> behaviour.
>
> >
> >
>
> >Regards,
>
> >
>
> >Nithya
>
>
> Best Regards,
> Strahil Nikolov
> On Sep 9, 2019 06:36, Nithya Balachandran  wrote:
>
>
>
> On Sat, 7 Sep 2019 at 00:03, Strahil Nikolov 
> wrote:
>
> As it was mentioned, you might have to run rebalance on the other node -
> but it is better to wait this node is over.
>
>
> Hi Strahil,
>
> Rebalance does not need to be run on the other node - the operation is a
> volume wide one . Only a single node per replica set would migrate files in
> the version used in this case .
>
> Regards,
> Nithya
>
> Best Regards,
> Strahil Nikolov
>
> В петък, 6 септември 2019 г., 15:29:20 ч. Гринуич+3, Herb Burnswell <
> herbert.burnsw...@gmail.com> написа:
>
>
>
>
> On Thu, Sep 5, 2019 at 9:56 PM Nithya Balachandran 
> wrote:
>
>
>
> On Thu, 5 Sep 2019 at 02:41, Herb Burnswell 
> wrote:
>
> Thanks for the replies.  The rebalance is running and the brick
> percentages are not adjusting as expected:
>
> # df -hP |grep data
> /dev/mapper/gluster_vg-gluster_lv1_data   60T   49T   11T  83%
> /gluster_bricks/data1
> /dev/mapper/gluster_vg-gluster_lv2_data   60T   49T   11T  83%
> /gluster_bricks/data2
> /dev/mapper/gluster_vg-gluster_lv3_data   60T  4.6T   55T   8%
> /gluster_bricks/data3
> /dev/mapper/gluster_vg-gluster_lv4_data   60T  4.6T   55T   8%
> /gluster_bricks/data4
> /dev/mapper/gluster_vg-gluster_lv5_data   60T  4.6T   55T   8%
> /gluster_bricks/data5
> /dev/mapper/gluster_vg-gluster_lv6_data   60T  4.6T   55T   8%
> /gluster_bricks/data6
>
> At the current pace it looks like this will continue to run for another
> 5-6 days.
>
> I appreciate the guidance..
>
>
> What is the output of the rebalance status command?
> Can you check if there are any errors in the rebalance logs on the node
> on which you see rebalance activity?
> If there are a lot of small files on the volume, the rebalance is expected
> to take time.
>
> Regards,
> Nithya
>
>
> My apologies, that was a typo.  I meant to say:
>
> "The rebalance is running and the brick percentages are NOW adjusting as
> expected"
>
> I did expect the rebalance to take several days.  The rebalance log is not
> showing any errors.  Status output:
>
> # gluster vol rebalance tank status
> Node Rebalanced-files  size
> scanned  failures   skipped   status  run time in
> h:m:s
>-  ---   ---
> ---   ---   --- 
> --
>localhost      1251320    35.5TB
> 2079527 0 0  in progress  139:9:46
>serverB 0
>  0Bytes 7 0 0completed
>   63:47:55
> volume rebalance: tank: success
>
> Thanks again for the guidance.
>
> HB
>
>
>
>
>
> On Mon, Sep 2, 2019 at 9:

Re: [Gluster-users] Rebalancing newly added bricks

[Nithya Balachandran]

On Sat, 7 Sep 2019 at 00:03, Strahil Nikolov  wrote:

> As it was mentioned, you might have to run rebalance on the other node -
> but it is better to wait this node is over.
>
>
Hi Strahil,

Rebalance does not need to be run on the other node - the operation is a
volume wide one . Only a single node per replica set would migrate files in
the version used in this case .

Regards,
Nithya

Best Regards,
> Strahil Nikolov
>
> В петък, 6 септември 2019 г., 15:29:20 ч. Гринуич+3, Herb Burnswell <
> herbert.burnsw...@gmail.com> написа:
>
>
>
>
> On Thu, Sep 5, 2019 at 9:56 PM Nithya Balachandran 
> wrote:
>
>
>
> On Thu, 5 Sep 2019 at 02:41, Herb Burnswell 
> wrote:
>
> Thanks for the replies.  The rebalance is running and the brick
> percentages are not adjusting as expected:
>
> # df -hP |grep data
> /dev/mapper/gluster_vg-gluster_lv1_data   60T   49T   11T  83%
> /gluster_bricks/data1
> /dev/mapper/gluster_vg-gluster_lv2_data   60T   49T   11T  83%
> /gluster_bricks/data2
> /dev/mapper/gluster_vg-gluster_lv3_data   60T  4.6T   55T   8%
> /gluster_bricks/data3
> /dev/mapper/gluster_vg-gluster_lv4_data   60T  4.6T   55T   8%
> /gluster_bricks/data4
> /dev/mapper/gluster_vg-gluster_lv5_data   60T  4.6T   55T   8%
> /gluster_bricks/data5
> /dev/mapper/gluster_vg-gluster_lv6_data   60T  4.6T   55T   8%
> /gluster_bricks/data6
>
> At the current pace it looks like this will continue to run for another
> 5-6 days.
>
> I appreciate the guidance..
>
>
> What is the output of the rebalance status command?
> Can you check if there are any errors in the rebalance logs on the node
> on which you see rebalance activity?
> If there are a lot of small files on the volume, the rebalance is expected
> to take time.
>
> Regards,
> Nithya
>
>
> My apologies, that was a typo.  I meant to say:
>
> "The rebalance is running and the brick percentages are NOW adjusting as
> expected"
>
> I did expect the rebalance to take several days.  The rebalance log is not
> showing any errors.  Status output:
>
> # gluster vol rebalance tank status
> Node Rebalanced-files  size
> scanned  failures   skipped   status  run time in
> h:m:s
>-  ---   ---
> ---   ---   --- 
> --
>localhost  125132035.5TB
> 2079527 0 0  in progress  139:9:46
>serverB         0
>  0Bytes 7 0 0completed
>   63:47:55
> volume rebalance: tank: success
>
> Thanks again for the guidance.
>
> HB
>
>
>
>
>
> On Mon, Sep 2, 2019 at 9:08 PM Nithya Balachandran 
> wrote:
>
>
>
> On Sat, 31 Aug 2019 at 22:59, Herb Burnswell 
> wrote:
>
> Thank you for the reply.
>
> I started a rebalance with force on serverA as suggested.  Now I see
> 'activity' on that node:
>
> # gluster vol rebalance tank status
> Node Rebalanced-files  size
> scanned  failures   skipped   status  run time in
> h:m:s
>-  ---   ---
> ---   ---   --- 
> --
>localhost 6143 6.1GB
>9542 0 0  in progress0:4:5
>serverB  00Bytes
>   7 0 0  in progress0:4:5
> volume rebalance: tank: success
>
> But I am not seeing any activity on serverB.  Is this expected?  Does the
> rebalance need to run on each node even though it says both nodes are 'in
> progress'?
>
>
> It looks like this is a replicate volume. If that is the case then yes,
> you are running an old version of Gluster for which this was the default
> behaviour.
>
> Regards,
> Nithya
>
> Thanks,
>
> HB
>
> On Sat, Aug 31, 2019 at 4:18 AM Strahil  wrote:
>
> The rebalance status show 0 Bytes.
>
> Maybe you should try with the 'gluster volume rebalance  start
> force' ?
>
> Best Regards,
> Strahil Nikolov
>
> Source:
> https://docs.gluster.org/en/latest/Administrator%20Guide/Managing%20Volumes/#rebalancing-volumes
> On Aug 30, 2019 20:04, Herb Burnswell  wrote:
>
> All,
>
> RHEL 7.5
> Gluster 3.8.15
> 2 Nodes: serverA & serverB
>
> I am not deeply knowledgeable about Glust

Re: [Gluster-users] Rebalancing newly added bricks

[Nithya Balachandran]

On Thu, 5 Sep 2019 at 02:41, Herb Burnswell 
wrote:

> Thanks for the replies.  The rebalance is running and the brick
> percentages are not adjusting as expected:
>
> # df -hP |grep data
> /dev/mapper/gluster_vg-gluster_lv1_data   60T   49T   11T  83%
> /gluster_bricks/data1
> /dev/mapper/gluster_vg-gluster_lv2_data   60T   49T   11T  83%
> /gluster_bricks/data2
> /dev/mapper/gluster_vg-gluster_lv3_data   60T  4.6T   55T   8%
> /gluster_bricks/data3
> /dev/mapper/gluster_vg-gluster_lv4_data   60T  4.6T   55T   8%
> /gluster_bricks/data4
> /dev/mapper/gluster_vg-gluster_lv5_data   60T  4.6T   55T   8%
> /gluster_bricks/data5
> /dev/mapper/gluster_vg-gluster_lv6_data   60T  4.6T   55T   8%
> /gluster_bricks/data6
>
> At the current pace it looks like this will continue to run for another
> 5-6 days.
>
> I appreciate the guidance..
>
>
What is the output of the rebalance status command?
Can you check if there are any errors in the rebalance logs on the node  on
which you see rebalance activity?
If there are a lot of small files on the volume, the rebalance is expected
to take time.

Regards,
Nithya

> HB
>
> On Mon, Sep 2, 2019 at 9:08 PM Nithya Balachandran 
> wrote:
>
>>
>>
>> On Sat, 31 Aug 2019 at 22:59, Herb Burnswell 
>> wrote:
>>
>>> Thank you for the reply.
>>>
>>> I started a rebalance with force on serverA as suggested.  Now I see
>>> 'activity' on that node:
>>>
>>> # gluster vol rebalance tank status
>>> Node Rebalanced-files  size
>>>   scanned  failures   skipped   status  run time in
>>> h:m:s
>>>-  ---   ---
>>>   ---   ---   --- 
>>> --
>>>localhost 6143 6.1GB
>>>  9542 0 0  in progress0:4:5
>>>serverB  00Bytes
>>> 7 0 0  in progress0:4:5
>>> volume rebalance: tank: success
>>>
>>> But I am not seeing any activity on serverB.  Is this expected?  Does
>>> the rebalance need to run on each node even though it says both nodes are
>>> 'in progress'?
>>>
>>>
>> It looks like this is a replicate volume. If that is the case then yes,
>> you are running an old version of Gluster for which this was the default
>> behaviour.
>>
>> Regards,
>> Nithya
>>
>> Thanks,
>>>
>>> HB
>>>
>>> On Sat, Aug 31, 2019 at 4:18 AM Strahil  wrote:
>>>
>>>> The rebalance status show 0 Bytes.
>>>>
>>>> Maybe you should try with the 'gluster volume rebalance  start
>>>> force' ?
>>>>
>>>> Best Regards,
>>>> Strahil Nikolov
>>>>
>>>> Source:
>>>> https://docs.gluster.org/en/latest/Administrator%20Guide/Managing%20Volumes/#rebalancing-volumes
>>>> On Aug 30, 2019 20:04, Herb Burnswell 
>>>> wrote:
>>>>
>>>> All,
>>>>
>>>> RHEL 7.5
>>>> Gluster 3.8.15
>>>> 2 Nodes: serverA & serverB
>>>>
>>>> I am not deeply knowledgeable about Gluster and it's administration but
>>>> we have a 2 node cluster that's been running for about a year and a half.
>>>> All has worked fine to date.  Our main volume has consisted of two 60TB
>>>> bricks on each of the cluster nodes.  As we reached capacity on the volume
>>>> we needed to expand.  So, we've added four new 60TB bricks to each of the
>>>> cluster nodes.  The bricks are now seen, and the total size of the volume
>>>> is as expected:
>>>>
>>>> # gluster vol status tank
>>>> Status of volume: tank
>>>> Gluster process TCP Port  RDMA Port  Online
>>>>  Pid
>>>>
>>>> --
>>>> Brick serverA:/gluster_bricks/data1   49162 0  Y
>>>> 20318
>>>> Brick serverB:/gluster_bricks/data1   49166 0  Y
>>>> 3432
>>>> Brick serverA:/gluster_bricks/data2   49163 0  Y
>>>> 20323
>>>> Brick serverB:/gluster_bricks/data2   49167 0  Y
>>>> 3435
>>>

Re: [Gluster-users] Rebalancing newly added bricks

[Nithya Balachandran]

On Sat, 31 Aug 2019 at 22:59, Herb Burnswell 
wrote:

> Thank you for the reply.
>
> I started a rebalance with force on serverA as suggested.  Now I see
> 'activity' on that node:
>
> # gluster vol rebalance tank status
> Node Rebalanced-files  size
> scanned  failures   skipped   status  run time in
> h:m:s
>-  ---   ---
> ---   ---   --- 
> --
>localhost 6143 6.1GB
>9542 0 0  in progress0:4:5
>serverB  00Bytes
>   7 0 0  in progress0:4:5
> volume rebalance: tank: success
>
> But I am not seeing any activity on serverB.  Is this expected?  Does the
> rebalance need to run on each node even though it says both nodes are 'in
> progress'?
>
>
It looks like this is a replicate volume. If that is the case then yes, you
are running an old version of Gluster for which this was the default
behaviour.

Regards,
Nithya

Thanks,
>
> HB
>
> On Sat, Aug 31, 2019 at 4:18 AM Strahil  wrote:
>
>> The rebalance status show 0 Bytes.
>>
>> Maybe you should try with the 'gluster volume rebalance  start
>> force' ?
>>
>> Best Regards,
>> Strahil Nikolov
>>
>> Source:
>> https://docs.gluster.org/en/latest/Administrator%20Guide/Managing%20Volumes/#rebalancing-volumes
>> On Aug 30, 2019 20:04, Herb Burnswell 
>> wrote:
>>
>> All,
>>
>> RHEL 7.5
>> Gluster 3.8.15
>> 2 Nodes: serverA & serverB
>>
>> I am not deeply knowledgeable about Gluster and it's administration but
>> we have a 2 node cluster that's been running for about a year and a half.
>> All has worked fine to date.  Our main volume has consisted of two 60TB
>> bricks on each of the cluster nodes.  As we reached capacity on the volume
>> we needed to expand.  So, we've added four new 60TB bricks to each of the
>> cluster nodes.  The bricks are now seen, and the total size of the volume
>> is as expected:
>>
>> # gluster vol status tank
>> Status of volume: tank
>> Gluster process TCP Port  RDMA Port  Online
>>  Pid
>>
>> --
>> Brick serverA:/gluster_bricks/data1   49162 0  Y
>> 20318
>> Brick serverB:/gluster_bricks/data1   49166 0  Y
>> 3432
>> Brick serverA:/gluster_bricks/data2   49163 0  Y
>> 20323
>> Brick serverB:/gluster_bricks/data2   49167 0  Y
>> 3435
>> Brick serverA:/gluster_bricks/data3   49164 0  Y
>> 4625
>> Brick serverA:/gluster_bricks/data4   49165 0  Y
>> 4644
>> Brick serverA:/gluster_bricks/data5   49166 0  Y
>> 5088
>> Brick serverA:/gluster_bricks/data6   49167 0  Y
>> 5128
>> Brick serverB:/gluster_bricks/data3   49168 0  Y
>> 22314
>> Brick serverB:/gluster_bricks/data4   49169 0  Y
>> 22345
>> Brick serverB:/gluster_bricks/data5   49170 0  Y
>> 22889
>> Brick serverB:/gluster_bricks/data6   49171 0  Y
>> 22932
>> Self-heal Daemon on localhost N/A   N/AY
>> 22981
>> Self-heal Daemon on serverA.example.com   N/A   N/AY
>> 6202
>>
>> After adding the bricks we ran a rebalance from serverA as:
>>
>> # gluster volume rebalance tank start
>>
>> The rebalance completed:
>>
>> # gluster volume rebalance tank status
>> Node Rebalanced-files  size
>> scanned  failures   skipped   status  run time in
>> h:m:s
>>-  ---   ---
>> ---   ---   --- 
>> --
>>localhost00Bytes
>>   0 0 0completed3:7:10
>>  serverA.example.com00Bytes
>> 0 0 0completed0:0:0
>> volume rebalance: tank: success
>>
>> However, when I run a df, the two original bricks still show all of the
>> consumed space (this is the same on both nodes):
>>
>> # df -hP
>> Filesystem   Size  Used Avail Use% Mounted on
>> /dev/mapper/vg0-root 5.0G  625M  4.4G  13% /
>> devtmpfs  32G 0   32G   0% /dev
>> tmpfs 32G 0   32G   0% /dev/shm
>> tmpfs 32G   67M   32G   1% /run
>> tmpfs 32G 0   32G   0%
>> /sys/fs/cgroup
>> /dev/mapper/vg0-usr   20G  3.6G   17G  18% /usr
>> /dev/md126 

Re: [Gluster-users] Continue to work in "degraded mode" (missing brick)

[Nithya Balachandran]

Hi,

This is the expected behaviour for a distribute volume. Files that hash to
a brick that is down will not be created. This is to prevent issues in case
the file already exists on that brick.

To prevent this, please use distribute-replicate volumes.

Regards,
Nithya

On Thu, 8 Aug 2019 at 13:17, Nux!  wrote:

> Sorry, I meant to say distributed, not replicated!
> I'm on 6.4 from CentOs7 SIG.
>
> I was hoping the volume might still be fully usable write-wise, with
> files going on the remaining bricks, but it doesn't seem to be the case.
>
> ---
> Sent from the Delta quadrant using Borg technology!
>
> On 2019-08-08 06:54, Ravishankar N wrote:
> > On 07/08/19 9:53 PM, Nux! wrote:
> >> Hello,
> >>
> >> I'm testing a replicated volume with 3 bricks. I've killed a brick,
> >> but the volume is still mounted and can see the files from the bricks
> >> that are still online and can do operations on them.
> >> What I cannot do is create new files in the volume, e.g.:
> >>
> >> dd: failed to open ‘test1000’: Transport endpoint is not connected
> >>
> >>
> >> Is there a way to make this volume continue to work while one of the
> >> bricks is offline? There is still space available in the remaining
> >> bricks, shouldn't it try to use it?
> >
> > If 2 bricks are online and the clients are connected to them, writes
> > should work.  Unless the brick that was down was the only good copy,
> > i.e. the only one that successfully witnessed all previous writes.
> > What version of gluster are you using? Check the mount log for more
> > details.
> >
> >>
> >> Regards
> >>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster eating up a lot of ram

[Nithya Balachandran]

On Tue, 30 Jul 2019 at 16:37, Diego Remolina  wrote:

> This option is enabled. In which version has this been patched? This is a
> file server and disabling readdir-ahead will have a hard impact on
> performance.
>

This was fixed in 5.3 (https://bugzilla.redhat.com/show_bug.cgi?id=1659676
).
This bug is only relevant if the gluster fuse client is the one that is
using up memory.

The first thing to do would be to determine which process is using up the
memory and to get a statedump.

ps  should give you the details of the gluster process .

Regards,
Nithya



>
> [root@ysmha01 ~]# gluster v get export readdir-ahead
> Option  Value
>
> --  -
>
> performance.readdir-ahead   on
>
> The guide recommends enabling the setting:
>
>
> https://docs.gluster.org/en/latest/Administrator%20Guide/Accessing%20Gluster%20from%20Windows/
>
> Diego
>
>
>
> On Mon, Jul 29, 2019 at 11:52 PM Nithya Balachandran 
> wrote:
>
>>
>> Hi Diego,
>>
>> Please do the following:
>>
>> gluster v get  readdir-ahead
>>
>> If this is enabled, please disable it and see if it helps. There was a
>> leak in the opendir codpath that was fixed in later released.
>>
>> Regards,
>> Nithya
>>
>>
>> On Tue, 30 Jul 2019 at 09:04, Diego Remolina  wrote:
>>
>>> Will this kill the actual process or simply trigger the dump? Which
>>> process should I kill? The brick process in the system or the fuse mount?
>>>
>>> Diego
>>>
>>> On Mon, Jul 29, 2019, 23:27 Nithya Balachandran 
>>> wrote:
>>>
>>>>
>>>>
>>>> On Tue, 30 Jul 2019 at 05:44, Diego Remolina 
>>>> wrote:
>>>>
>>>>> Unfortunately statedump crashes on both machines, even freshly
>>>>> rebooted.
>>>>>
>>>>
>>>> Do you see any statedump files in /var/run/gluster?  This looks more
>>>> like the gluster cli crashed.
>>>>
>>>>>
>>>>> [root@ysmha01 ~]# gluster --print-statedumpdir
>>>>> /var/run/gluster
>>>>> [root@ysmha01 ~]# gluster v statedump export
>>>>> Segmentation fault (core dumped)
>>>>>
>>>>> [root@ysmha02 ~]# uptime
>>>>>  20:12:20 up 6 min,  1 user,  load average: 0.72, 0.52, 0.24
>>>>> [root@ysmha02 ~]# gluster --print-statedumpdir
>>>>> /var/run/gluster
>>>>> [root@ysmha02 ~]# gluster v statedump export
>>>>> Segmentation fault (core dumped)
>>>>>
>>>>> I rebooted today after 40 days. Gluster was eating up shy of 40GB of
>>>>> RAM out of 64.
>>>>>
>>>>> What would you recommend to be the next step?
>>>>>
>>>>> Diego
>>>>>
>>>>> On Mon, Mar 4, 2019 at 5:07 AM Poornima Gurusiddaiah <
>>>>> pguru...@redhat.com> wrote:
>>>>>
>>>>>> Could you also provide the statedump of the gluster process consuming
>>>>>> 44G ram [1]. Please make sure the statedump is taken when the memory
>>>>>> consumption is very high, like 10s of GBs, otherwise we may not be able 
>>>>>> to
>>>>>> identify the issue. Also i see that the cache size is 10G is that 
>>>>>> something
>>>>>> you arrived at, after doing some tests? Its relatively higher than 
>>>>>> normal.
>>>>>>
>>>>>> [1]
>>>>>> https://docs.gluster.org/en/v3/Troubleshooting/statedump/#generate-a-statedump
>>>>>>
>>>>>> On Mon, Mar 4, 2019 at 12:23 AM Diego Remolina 
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I will not be able to test gluster-6rc because this is a production
>>>>>>> environment and it takes several days for memory to grow a lot.
>>>>>>>
>>>>>>> The Samba server is hosting all types of files, small and large from
>>>>>>> small roaming profile type files to bigger files like adobe suite, 
>>>>>>> autodesk
>>>>>>> Revit (file sizes in the hundreds of megabytes).
>>>>>>>
>>>>>>> As I stated before, this same issue was present back with 3.8.x
>>>>>>> which I was running before.
>>>>

Re: [Gluster-users] Gluster eating up a lot of ram

[Nithya Balachandran]

Hi Diego,

Please do the following:

gluster v get  readdir-ahead

If this is enabled, please disable it and see if it helps. There was a leak
in the opendir codpath that was fixed in later released.

Regards,
Nithya


On Tue, 30 Jul 2019 at 09:04, Diego Remolina  wrote:

> Will this kill the actual process or simply trigger the dump? Which
> process should I kill? The brick process in the system or the fuse mount?
>
> Diego
>
> On Mon, Jul 29, 2019, 23:27 Nithya Balachandran 
> wrote:
>
>>
>>
>> On Tue, 30 Jul 2019 at 05:44, Diego Remolina  wrote:
>>
>>> Unfortunately statedump crashes on both machines, even freshly rebooted.
>>>
>>
>> Do you see any statedump files in /var/run/gluster?  This looks more like
>> the gluster cli crashed.
>>
>>>
>>> [root@ysmha01 ~]# gluster --print-statedumpdir
>>> /var/run/gluster
>>> [root@ysmha01 ~]# gluster v statedump export
>>> Segmentation fault (core dumped)
>>>
>>> [root@ysmha02 ~]# uptime
>>>  20:12:20 up 6 min,  1 user,  load average: 0.72, 0.52, 0.24
>>> [root@ysmha02 ~]# gluster --print-statedumpdir
>>> /var/run/gluster
>>> [root@ysmha02 ~]# gluster v statedump export
>>> Segmentation fault (core dumped)
>>>
>>> I rebooted today after 40 days. Gluster was eating up shy of 40GB of RAM
>>> out of 64.
>>>
>>> What would you recommend to be the next step?
>>>
>>> Diego
>>>
>>> On Mon, Mar 4, 2019 at 5:07 AM Poornima Gurusiddaiah <
>>> pguru...@redhat.com> wrote:
>>>
>>>> Could you also provide the statedump of the gluster process consuming
>>>> 44G ram [1]. Please make sure the statedump is taken when the memory
>>>> consumption is very high, like 10s of GBs, otherwise we may not be able to
>>>> identify the issue. Also i see that the cache size is 10G is that something
>>>> you arrived at, after doing some tests? Its relatively higher than normal.
>>>>
>>>> [1]
>>>> https://docs.gluster.org/en/v3/Troubleshooting/statedump/#generate-a-statedump
>>>>
>>>> On Mon, Mar 4, 2019 at 12:23 AM Diego Remolina 
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I will not be able to test gluster-6rc because this is a production
>>>>> environment and it takes several days for memory to grow a lot.
>>>>>
>>>>> The Samba server is hosting all types of files, small and large from
>>>>> small roaming profile type files to bigger files like adobe suite, 
>>>>> autodesk
>>>>> Revit (file sizes in the hundreds of megabytes).
>>>>>
>>>>> As I stated before, this same issue was present back with 3.8.x which
>>>>> I was running before.
>>>>>
>>>>> The information you requested:
>>>>>
>>>>> [root@ysmha02 ~]# gluster v info export
>>>>>
>>>>> Volume Name: export
>>>>> Type: Replicate
>>>>> Volume ID: b4353b3f-6ef6-4813-819a-8e85e5a95cff
>>>>> Status: Started
>>>>> Snapshot Count: 0
>>>>> Number of Bricks: 1 x 2 = 2
>>>>> Transport-type: tcp
>>>>> Bricks:
>>>>> Brick1: 10.0.1.7:/bricks/hdds/brick
>>>>> Brick2: 10.0.1.6:/bricks/hdds/brick
>>>>> Options Reconfigured:
>>>>> performance.stat-prefetch: on
>>>>> performance.cache-min-file-size: 0
>>>>> network.inode-lru-limit: 65536
>>>>> performance.cache-invalidation: on
>>>>> features.cache-invalidation: on
>>>>> performance.md-cache-timeout: 600
>>>>> features.cache-invalidation-timeout: 600
>>>>> performance.cache-samba-metadata: on
>>>>> transport.address-family: inet
>>>>> server.allow-insecure: on
>>>>> performance.cache-size: 10GB
>>>>> cluster.server-quorum-type: server
>>>>> nfs.disable: on
>>>>> performance.io-thread-count: 64
>>>>> performance.io-cache: on
>>>>> cluster.lookup-optimize: on
>>>>> cluster.readdir-optimize: on
>>>>> server.event-threads: 5
>>>>> client.event-threads: 5
>>>>> performance.cache-max-file-size: 256MB
>>>>> diagnostics.client-log-level: INFO
>>>>> diagnostics.brick-log-level: INFO
>>>>> cluster.server-quorum-ratio: 51%
>

Re: [Gluster-users] Gluster eating up a lot of ram

[Nithya Balachandran]

On Tue, 30 Jul 2019 at 05:44, Diego Remolina  wrote:

> Unfortunately statedump crashes on both machines, even freshly rebooted.
>

Do you see any statedump files in /var/run/gluster?  This looks more like
the gluster cli crashed.

>
> [root@ysmha01 ~]# gluster --print-statedumpdir
> /var/run/gluster
> [root@ysmha01 ~]# gluster v statedump export
> Segmentation fault (core dumped)
>
> [root@ysmha02 ~]# uptime
>  20:12:20 up 6 min,  1 user,  load average: 0.72, 0.52, 0.24
> [root@ysmha02 ~]# gluster --print-statedumpdir
> /var/run/gluster
> [root@ysmha02 ~]# gluster v statedump export
> Segmentation fault (core dumped)
>
> I rebooted today after 40 days. Gluster was eating up shy of 40GB of RAM
> out of 64.
>
> What would you recommend to be the next step?
>
> Diego
>
> On Mon, Mar 4, 2019 at 5:07 AM Poornima Gurusiddaiah 
> wrote:
>
>> Could you also provide the statedump of the gluster process consuming 44G
>> ram [1]. Please make sure the statedump is taken when the memory
>> consumption is very high, like 10s of GBs, otherwise we may not be able to
>> identify the issue. Also i see that the cache size is 10G is that something
>> you arrived at, after doing some tests? Its relatively higher than normal.
>>
>> [1]
>> https://docs.gluster.org/en/v3/Troubleshooting/statedump/#generate-a-statedump
>>
>> On Mon, Mar 4, 2019 at 12:23 AM Diego Remolina 
>> wrote:
>>
>>> Hi,
>>>
>>> I will not be able to test gluster-6rc because this is a production
>>> environment and it takes several days for memory to grow a lot.
>>>
>>> The Samba server is hosting all types of files, small and large from
>>> small roaming profile type files to bigger files like adobe suite, autodesk
>>> Revit (file sizes in the hundreds of megabytes).
>>>
>>> As I stated before, this same issue was present back with 3.8.x which I
>>> was running before.
>>>
>>> The information you requested:
>>>
>>> [root@ysmha02 ~]# gluster v info export
>>>
>>> Volume Name: export
>>> Type: Replicate
>>> Volume ID: b4353b3f-6ef6-4813-819a-8e85e5a95cff
>>> Status: Started
>>> Snapshot Count: 0
>>> Number of Bricks: 1 x 2 = 2
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: 10.0.1.7:/bricks/hdds/brick
>>> Brick2: 10.0.1.6:/bricks/hdds/brick
>>> Options Reconfigured:
>>> performance.stat-prefetch: on
>>> performance.cache-min-file-size: 0
>>> network.inode-lru-limit: 65536
>>> performance.cache-invalidation: on
>>> features.cache-invalidation: on
>>> performance.md-cache-timeout: 600
>>> features.cache-invalidation-timeout: 600
>>> performance.cache-samba-metadata: on
>>> transport.address-family: inet
>>> server.allow-insecure: on
>>> performance.cache-size: 10GB
>>> cluster.server-quorum-type: server
>>> nfs.disable: on
>>> performance.io-thread-count: 64
>>> performance.io-cache: on
>>> cluster.lookup-optimize: on
>>> cluster.readdir-optimize: on
>>> server.event-threads: 5
>>> client.event-threads: 5
>>> performance.cache-max-file-size: 256MB
>>> diagnostics.client-log-level: INFO
>>> diagnostics.brick-log-level: INFO
>>> cluster.server-quorum-ratio: 51%
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> 
>>>  Virus-free.
>>> www.avast.com
>>> 
>>> <#m_-4519402017059013283_m_-1483290904248086332_m_-4429654867678350131_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>>
>>> On Fri, Mar 1, 2019 at 11:07 PM Poornima Gurusiddaiah <
>>> pguru...@redhat.com> wrote:
>>>
 This high memory consumption is not normal. Looks like it's a memory
 leak. Is it possible to try it on test setup with gluster-6rc? What is the
 kind of workload that goes into fuse mount? Large files or small files? We
 need the following information to debug further:
 - Gluster volume info output
 - Statedump of the Gluster fuse mount process consuming 44G ram.

 Regards,
 Poornima


 On Sat, Mar 2, 2019, 3:40 AM Diego Remolina  wrote:

> I am using glusterfs with two servers as a file server sharing files
> via samba and ctdb. I cannot use samba vfs gluster plugin, due to bug in
> current Centos version of samba. So I am mounting via fuse and exporting
> the volume to samba from the mount point.
>
> Upon initial boot, the server where samba is exporting files climbs up
> to ~10GB RAM within a couple hours of use. From then on, it is a constant
> slow memory increase. In the past with gluster 3.8.x we had to reboot the
> servers at around 30 days . With gluster 4.1.6 we are getting up to 48
> days, but RAM use is at 48GB out of 64GB. Is this normal?
>
> The particular versions are below,
>
> [root@ysmha01 home]# uptime
> 16:59:39 up 48 days,  9:56,  1 user,  load average: 3.75, 3.17, 3.00
> [root@ysmha01 home]# rpm -qa | grep gluster
> centos-release-gluster41-1.0-3.el7.centos.noarch
> 

Re: [Gluster-users] Brick missing trusted.glusterfs.dht xattr

[Nithya Balachandran]

On Sat, 27 Jul 2019 at 02:31, Matthew Benstead  wrote:

> Ok thank-you for explaining everything - that makes sense.
>
> Currently the brick file systems are pretty evenly distributed so I
> probably won't run the fix-layout right now.
>
> Would this state have any impact on geo-replication? I'm trying to
> geo-replicate this volume, but am getting a weird error: "Changelog
> register failed error=[Errno 21] Is a directory"
>

It should not. Sunny, can you comment on this?

Regards,
Nithya

>
> I assume this is related to something else, but I wasn't sure.
>
> Thanks,
>  -Matthew
>
> --
> Matthew Benstead
> System Administrator
> Pacific Climate Impacts Consortium <https://pacificclimate.org/>
> University of Victoria, UH1
> PO Box 1800, STN CSC
> Victoria, BC, V8W 2Y2
> Phone: +1-250-721-8432
> Email: matth...@uvic.ca
> On 7/26/19 12:02 AM, Nithya Balachandran wrote:
>
>
>
> On Fri, 26 Jul 2019 at 01:56, Matthew Benstead  wrote:
>
>> Hi Nithya,
>>
>> Hmm... I don't remember if I did, but based on what I'm seeing it sounds
>> like I probably didn't run rebalance or fix-layout.
>>
>> It looks like folders that haven't had any new files created have a dht
>> of 0, while other folders have non-zero values.
>>
>> [root@gluster07 ~]# getfattr --absolute-names -m . -d -e hex
>> /mnt/raid6-storage/storage/ | grep dht
>> [root@gluster07 ~]# getfattr --absolute-names -m . -d -e hex
>> /mnt/raid6-storage/storage/home | grep dht
>> trusted.glusterfs.dht=0x
>> [root@gluster07 ~]# getfattr --absolute-names -m . -d -e hex
>> /mnt/raid6-storage/storage/home/matthewb | grep dht
>> trusted.glusterfs.dht=0x00014924921a6db6dbc7
>>
>> If I just run the fix-layout command will it re-create all of the dht
>> values or just the missing ones?
>>
>
> A fix-layout will recalculate the layouts entirely so files all the values
> will change. No files will be moved.
> A rebalance will recalculate the layouts like the fix-layout but will also
> move files to their new locations based on the new layout ranges. This
> could take a lot of time depending on the number of files/directories on
> the volume. If you do this, I would recommend that you turn off
> lookup-optimize until the rebalance is over.
>
>
>> Since the brick is already fairly size balanced could I get away with
>> running fix-layout but not rebalance? Or would the new dht layout mean
>> slower accesses since the files may be expected on different bricks?
>>
>
> The first access for a file will be slower. The next one will be faster as
> the location will be cached in the client's in-memory structures.
> You may not need to run either a fix-layout or a rebalance if new file
> creations will be in directories created after the add-brick. Gluster will
> automatically include all 7 bricks for those directories.
>
> Regards,
> Nithya
>
>
>> Thanks,
>>  -Matthew
>>
>> --
>> Matthew Benstead
>> System Administrator
>> Pacific Climate Impacts Consortium <https://pacificclimate.org/>
>> University of Victoria, UH1
>> PO Box 1800, STN CSC
>> Victoria, BC, V8W 2Y2
>> Phone: +1-250-721-8432
>> Email: matth...@uvic.ca
>> On 7/24/19 9:30 PM, Nithya Balachandran wrote:
>>
>>
>>
>> On Wed, 24 Jul 2019 at 22:12, Matthew Benstead  wrote:
>>
>>> So looking more closely at the trusted.glusterfs.dht attributes from the
>>> bricks it looks like they cover the entire range... and there is no range
>>> left for gluster07.
>>>
>>> The first 6 bricks range from 0x to 0x - so... is there
>>> a way to re-calculate what the dht values should be? Each of the bricks
>>> should have a gap
>>>
>>> Gluster05  -> 2aa9
>>> Gluster06 2aaa -> 5553
>>> Gluster01 5554 -> 7ffd
>>> Gluster02 7ffe -> aaa7
>>> Gluster03 aaa8 -> d551
>>> Gluster04 d552 -> 
>>> Gluster07 None
>>>
>>> If we split the range into 7 servers that would be a gap of about
>>> 0x24924924 for each server.
>>>
>>> Now in terms of the gluster07 brick, about 2 years ago the RAID array
>>> the brick was stored on became corrupted. I ran the remove-brick force
>>> command, then provisioned a new server, ran the add-brick command and then
>>> restored the missing files from backup by copying them back to the main
>>> gluster mount (not the brick).
>>>
>>>
>> 

Re: [Gluster-users] Brick missing trusted.glusterfs.dht xattr

[Nithya Balachandran]

On Fri, 26 Jul 2019 at 01:56, Matthew Benstead  wrote:

> Hi Nithya,
>
> Hmm... I don't remember if I did, but based on what I'm seeing it sounds
> like I probably didn't run rebalance or fix-layout.
>
> It looks like folders that haven't had any new files created have a dht of
> 0, while other folders have non-zero values.
>
> [root@gluster07 ~]# getfattr --absolute-names -m . -d -e hex
> /mnt/raid6-storage/storage/ | grep dht
> [root@gluster07 ~]# getfattr --absolute-names -m . -d -e hex
> /mnt/raid6-storage/storage/home | grep dht
> trusted.glusterfs.dht=0x
> [root@gluster07 ~]# getfattr --absolute-names -m . -d -e hex
> /mnt/raid6-storage/storage/home/matthewb | grep dht
> trusted.glusterfs.dht=0x00014924921a6db6dbc7
>
> If I just run the fix-layout command will it re-create all of the dht
> values or just the missing ones?
>

A fix-layout will recalculate the layouts entirely so files all the values
will change. No files will be moved.
A rebalance will recalculate the layouts like the fix-layout but will also
move files to their new locations based on the new layout ranges. This
could take a lot of time depending on the number of files/directories on
the volume. If you do this, I would recommend that you turn off
lookup-optimize until the rebalance is over.


> Since the brick is already fairly size balanced could I get away with
> running fix-layout but not rebalance? Or would the new dht layout mean
> slower accesses since the files may be expected on different bricks?
>

The first access for a file will be slower. The next one will be faster as
the location will be cached in the client's in-memory structures.
You may not need to run either a fix-layout or a rebalance if new file
creations will be in directories created after the add-brick. Gluster will
automatically include all 7 bricks for those directories.

Regards,
Nithya


> Thanks,
>  -Matthew
>
> --
> Matthew Benstead
> System Administrator
> Pacific Climate Impacts Consortium <https://pacificclimate.org/>
> University of Victoria, UH1
> PO Box 1800, STN CSC
> Victoria, BC, V8W 2Y2
> Phone: +1-250-721-8432
> Email: matth...@uvic.ca
> On 7/24/19 9:30 PM, Nithya Balachandran wrote:
>
>
>
> On Wed, 24 Jul 2019 at 22:12, Matthew Benstead  wrote:
>
>> So looking more closely at the trusted.glusterfs.dht attributes from the
>> bricks it looks like they cover the entire range... and there is no range
>> left for gluster07.
>>
>> The first 6 bricks range from 0x to 0x - so... is there a
>> way to re-calculate what the dht values should be? Each of the bricks
>> should have a gap
>>
>> Gluster05  -> 2aa9
>> Gluster06 2aaa -> 5553
>> Gluster01 5554 -> 7ffd
>> Gluster02 7ffe -> aaa7
>> Gluster03 aaa8 -> d551
>> Gluster04 d552 -> 
>> Gluster07 None
>>
>> If we split the range into 7 servers that would be a gap of about
>> 0x24924924 for each server.
>>
>> Now in terms of the gluster07 brick, about 2 years ago the RAID array the
>> brick was stored on became corrupted. I ran the remove-brick force command,
>> then provisioned a new server, ran the add-brick command and then restored
>> the missing files from backup by copying them back to the main gluster
>> mount (not the brick).
>>
>>
> Did you run a rebalance after performing the add-brick? Without a
> rebalance/fix-layout , the layout for existing directories on the volume
> will not  be updated to use the new brick as well.
>
> That the layout does not include the new brick in the root dir is in
> itself is not a problem. Do you create a lot of files directly in the root
> of the volume? If yes, you might want to run a rebalance. Otherwise, if you
> mostly create files in newly added directories, you can probably ignore
> this. You can check the layout for directories on the volume and see if
> they incorporate the brick7.
>
> I would expect a lookup on the root to have set an xattr on the brick with
> an empty layout range . The fact that the xattr does not exist at all on
> the brick is what I am looking into.
>
>
> It looks like prior to that event this was the layout - which would make
>> sense given the equal size of the 7 bricks:
>>
>> gluster02.pcic.uvic.ca | SUCCESS | rc=0 >>
>> # file: /mnt/raid6-storage/storage
>> trusted.glusterfs.dht=0x000148bfff206d1ffe5f
>>
>> gluster05.pcic.uvic.ca | SUCCESS | rc=0 >>
>> # file: /mnt/raid6-storage/storage
>> trusted.glusterfs.dht=0x0001b5dffce0da3ffc1f
>>
>> gluster04.pcic.uvic.ca | SUC

Re: [Gluster-users] Brick missing trusted.glusterfs.dht xattr

[Nithya Balachandran]

On Wed, 24 Jul 2019 at 22:12, Matthew Benstead  wrote:

> So looking more closely at the trusted.glusterfs.dht attributes from the
> bricks it looks like they cover the entire range... and there is no range
> left for gluster07.
>
> The first 6 bricks range from 0x to 0x - so... is there a
> way to re-calculate what the dht values should be? Each of the bricks
> should have a gap
>
> Gluster05  -> 2aa9
> Gluster06 2aaa -> 5553
> Gluster01 5554 -> 7ffd
> Gluster02 7ffe -> aaa7
> Gluster03 aaa8 -> d551
> Gluster04 d552 -> 
> Gluster07 None
>
> If we split the range into 7 servers that would be a gap of about
> 0x24924924 for each server.
>
> Now in terms of the gluster07 brick, about 2 years ago the RAID array the
> brick was stored on became corrupted. I ran the remove-brick force command,
> then provisioned a new server, ran the add-brick command and then restored
> the missing files from backup by copying them back to the main gluster
> mount (not the brick).
>
>
Did you run a rebalance after performing the add-brick? Without a
rebalance/fix-layout , the layout for existing directories on the volume
will not  be updated to use the new brick as well.

That the layout does not include the new brick in the root dir is in itself
is not a problem. Do you create a lot of files directly in the root of the
volume? If yes, you might want to run a rebalance. Otherwise, if you mostly
create files in newly added directories, you can probably ignore this. You
can check the layout for directories on the volume and see if they
incorporate the brick7.

I would expect a lookup on the root to have set an xattr on the brick with
an empty layout range . The fact that the xattr does not exist at all on
the brick is what I am looking into.


It looks like prior to that event this was the layout - which would make
> sense given the equal size of the 7 bricks:
>
> gluster02.pcic.uvic.ca | SUCCESS | rc=0 >>
> # file: /mnt/raid6-storage/storage
> trusted.glusterfs.dht=0x000148bfff206d1ffe5f
>
> gluster05.pcic.uvic.ca | SUCCESS | rc=0 >>
> # file: /mnt/raid6-storage/storage
> trusted.glusterfs.dht=0x0001b5dffce0da3ffc1f
>
> gluster04.pcic.uvic.ca | SUCCESS | rc=0 >>
> # file: /mnt/raid6-storage/storage
> trusted.glusterfs.dht=0x0001917ffda0b5dffcdf
>
> gluster03.pcic.uvic.ca | SUCCESS | rc=0 >>
> # file: /mnt/raid6-storage/storage
> trusted.glusterfs.dht=0x00016d1ffe60917ffd9f
>
> gluster01.pcic.uvic.ca | SUCCESS | rc=0 >>
> # file: /mnt/raid6-storage/storage
> trusted.glusterfs.dht=0x0001245fffe048bfff1f
>
> gluster07.pcic.uvic.ca | SUCCESS | rc=0 >>
> # file: /mnt/raid6-storage/storage
> trusted.glusterfs.dht=0x0001245fffdf
>
> gluster06.pcic.uvic.ca | SUCCESS | rc=0 >>
> # file: /mnt/raid6-storage/storage
> trusted.glusterfs.dht=0x0001da3ffc20
>
> Which yields the following:
>
>  -> 245fffdfGluster07
> 245fffe0 -> 48bfff1fGluster01
> 48bfff20 -> 6d1ffe5fGluster02
> 6d1ffe60 -> 917ffd9fGluster03
> 917ffda0 -> b5dffcdfGluster04
> b5dffce0 -> da3ffc1fGluster05
> da3ffc20 -> Gluster06
>
> Is there some way to get back to this?
>
> Thanks,
>  -Matthew
>
> --
> Matthew Benstead
> System Administrator
> Pacific Climate Impacts Consortium <https://pacificclimate.org/>
> University of Victoria, UH1
> PO Box 1800, STN CSC
> Victoria, BC, V8W 2Y2
> Phone: +1-250-721-8432
> Email: matth...@uvic.ca
> On 7/18/19 7:20 AM, Matthew Benstead wrote:
>
> Hi Nithya,
>
> No - it was added about a year and a half ago. I have tried re-mounting
> the volume on the server, but it didn't add the attr:
>
> [root@gluster07 ~]# umount /storage/
> [root@gluster07 ~]# cat /etc/fstab | grep "/storage"
> 10.0.231.56:/storage /storage glusterfs
> defaults,log-level=WARNING,backupvolfile-server=10.0.231.51 0 0
> [root@gluster07 ~]# mount /storage/
> [root@gluster07 ~]# df -h /storage/
> FilesystemSize  Used Avail Use% Mounted on
> 10.0.231.56:/storage  255T  194T   62T  77% /storage
> [root@gluster07 ~]# getfattr --absolute-names -m . -d -e hex
> /mnt/raid6-storage/storage/
> # file: /mnt/raid6-storage/storage/
>
> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000
> trusted.gfid=0x0001
>
> trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime=0x5d307baa00023ec0
> trusted.glusterfs.quota.dirty=0x3000
>
> trusted.glusterfs.quota.size.2=0x1b71d5279e0

Re: [Gluster-users] Brick missing trusted.glusterfs.dht xattr

[Nithya Balachandran]

Hi Mathew,

Could you send me the log file for this mount point after doing the
following:


   1. gluster v set  client-log-level DEBUG
   2. unmount and remount the volume
   3. stat 
   4. gluster v set  client-log-level INFO



Regards,
Nithya

On Thu, 18 Jul 2019 at 19:49, Matthew Benstead  wrote:

> Hi Nithya,
>
> No - it was added about a year and a half ago. I have tried re-mounting
> the volume on the server, but it didn't add the attr:
>
> [root@gluster07 ~]# umount /storage/
> [root@gluster07 ~]# cat /etc/fstab | grep "/storage"
> 10.0.231.56:/storage /storage glusterfs
> defaults,log-level=WARNING,backupvolfile-server=10.0.231.51 0 0
> [root@gluster07 ~]# mount /storage/
> [root@gluster07 ~]# df -h /storage/
> FilesystemSize  Used Avail Use% Mounted on
> 10.0.231.56:/storage  255T  194T   62T  77% /storage
> [root@gluster07 ~]# getfattr --absolute-names -m . -d -e hex
> /mnt/raid6-storage/storage/
> # file: /mnt/raid6-storage/storage/
>
> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000
> trusted.gfid=0x0001
>
> trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime=0x5d307baa00023ec0
> trusted.glusterfs.quota.dirty=0x3000
>
> trusted.glusterfs.quota.size.2=0x1b71d5279e763e320005cd53
> trusted.glusterfs.volume-id=0x6f95525a94d74174bac4e1a18fe010a2
>
> Thanks,
>  -Matthew
>
> On 7/17/19 10:04 PM, Nithya Balachandran wrote:
>
> Hi Matthew,
>
> Was this node/brick added to the volume recently? If yes, try mounting the
> volume on a fresh mount point - that should create the xattr on this as
> well.
>
> Regards,
> Nithya
>
> On Wed, 17 Jul 2019 at 21:01, Matthew Benstead  wrote:
>
>> Hello,
>>
>> I've just noticed one brick in my 7 node distribute volume is missing
>> the trusted.glusterfs.dht xattr...? How can I fix this?
>>
>> I'm running glusterfs-5.3-2.el7.x86_64 on CentOS 7.
>>
>> All of the other nodes are fine, but gluster07 from the list below does
>> not have the attribute.
>>
>> $ ansible -i hosts gluster-servers[0:6] ... -m shell -a "getfattr -m .
>> --absolute-names -n trusted.glusterfs.dht -e hex
>> /mnt/raid6-storage/storage"
>> ...
>> gluster05 | SUCCESS | rc=0 >>
>> # file: /mnt/raid6-storage/storage
>> trusted.glusterfs.dht=0x00012aa9
>>
>> gluster03 | SUCCESS | rc=0 >>
>> # file: /mnt/raid6-storage/storage
>> trusted.glusterfs.dht=0x0001aaa8d551
>>
>> gluster04 | SUCCESS | rc=0 >>
>> # file: /mnt/raid6-storage/storage
>> trusted.glusterfs.dht=0x0001d552
>>
>> gluster06 | SUCCESS | rc=0 >>
>> # file: /mnt/raid6-storage/storage
>> trusted.glusterfs.dht=0x00012aaa5553
>>
>> gluster02 | SUCCESS | rc=0 >>
>> # file: /mnt/raid6-storage/storage
>> trusted.glusterfs.dht=0x00017ffeaaa7
>>
>> gluster07 | FAILED | rc=1 >>
>> /mnt/raid6-storage/storage: trusted.glusterfs.dht: No such
>> attributenon-zero return code
>>
>> gluster01 | SUCCESS | rc=0 >>
>> # file: /mnt/raid6-storage/storage
>> trusted.glusterfs.dht=0x000155547ffd
>>
>> Here are all of the attr's from the brick:
>>
>> [root@gluster07 ~]# getfattr --absolute-names -m . -d -e hex
>> /mnt/raid6-storage/storage/
>> # file: /mnt/raid6-storage/storage/
>>
>> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000
>> trusted.gfid=0x0001
>>
>> trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime=0x5d2dee81fdf9
>> trusted.glusterfs.quota.dirty=0x3000
>>
>> trusted.glusterfs.quota.size.2=0x1b69498a1476332e0005cd03
>> trusted.glusterfs.volume-id=0x6f95525a94d74174bac4e1a18fe010a2
>>
>>
>> And here is the volume information:
>>
>> [root@gluster07 ~]# gluster volume info storage
>>
>> Volume Name: storage
>> Type: Distribute
>> Volume ID: 6f95525a-94d7-4174-bac4-e1a18fe010a2
>> Status: Started
>> Snapshot Count: 0
>> Number of Bricks: 7
>> Transport-type: tcp
>> Bricks:
>> Brick1: 10.0.231.50:/mnt/raid6-storage/storage
>> Brick2: 10.0.231.51:/mnt/raid6-storage/storage
>> Brick3: 10.0.231.52:/mnt/raid6-storage/storage
>> Brick4: 10.0.231.53:/mnt/raid6-storage/storage
>> Brick5: 10.0.231.54:/mnt/raid6-storage/storage
>> Brick6: 10.0.231.55:/mnt/raid6-storage/storage
>&

Re: [Gluster-users] Parallel process hang on gluster volume

[Nithya Balachandran]

Did you see this behaviour with previous Gluster versions?

Regards,
Nithya

On Wed, 3 Jul 2019 at 21:41,  wrote:

> Am I alone having this problem ?
>
> - Mail original -
> De: n...@furyweb.fr
> À: "gluster-users" 
> Envoyé: Vendredi 21 Juin 2019 09:48:47
> Objet: [Gluster-users] Parallel process hang on gluster volume
>
> I encounterd an issue on production servers using GlusterFS servers 5.1
> and clients 4.1.5 when several process write at the same time on a gluster
> volume.
>
> With more than 48 process writes on the volume at the same time, they are
> blocked in D state (uninterruptible sleep), I guess some volume settings
> have to be tuned but can't figure out which.
>
> The client is using op-version 40100 on this volume
> Below are volume info, volume settings and ps output on blocked processes.
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Removing subvolume from dist/rep volume

[Nithya Balachandran]

Hi Dave,

Yes, files in split brain are not migrated as we cannot figure out which is
the good copy. Adding Ravi to look at this and see what can be done.
Also adding Krutika as this is a sharded volume.

The files with the "-T" permissions are internal files and can be
ignored. Ravi and Krutika, please take a look at the other files.

Regards,
Nithya


On Fri, 28 Jun 2019 at 19:56, Dave Sherohman  wrote:

> On Thu, Jun 27, 2019 at 12:17:10PM +0530, Nithya Balachandran wrote:
> > There are some edge cases that may prevent a file from being migrated
> > during a remove-brick. Please do the following after this:
> >
> >1. Check the remove-brick status for any failures.  If there are any,
> >check the rebalance log file for errors.
> >2. Even if there are no failures, check the removed bricks to see if
> any
> >files have not been migrated. If there are any, please check that
> they are
> >valid files on the brick and copy them to the volume from the brick
> to the
> >mount point.
>
> Well, looks like I hit one of those edge cases.  Probably because of
> some issues around a reboot last September which left a handful of files
> in a state where self-heal identified them as needing to be healed, but
> incapable of actually healing them.  (Check the list archives for
> "Kicking a stuck heal", posted on Sept 4, if you want more details.)
>
> So I'm getting 9 failures on the arbiter (merlin), 8 on one data brick
> (gandalf), and 3 on the other (saruman).  Looking in
> /var/log/gluster/palantir-rebalance.log, I see those numbers of
>
> migrate file failed: /.shard/291e9749-2d1b-47af-ad53-3a09ad4e64c6.229:
> failed to lock file on palantir-replicate-1 [Stale file handle]
>
> errors.
>
> Also, merlin has four errors, and gandalf has one, of the form:
>
> Gfid mismatch detected for
> /0f500288-ff62-4f0b-9574-53f510b4159f.2898>,
> 9f00c0fe-58c3-457e-a2e6-f6a006d1cfc6 on palantir-client-7 and
> 08bb7cdc-172b-4c21-916a-2a244c095a3e on palantir-client-1.
>
> There are no gfid mismatches recorded on saruman.  All of the gfid
> mismatches are for  and (on
> saruman) appear to correspond to 0-byte files (e.g.,
> .shard/0f500288-ff62-4f0b-9574-53f510b4159f.2898, in the case of the
> gfid mismatch quoted above).
>
> For both types of errors, all affected files are in .shard/ and have
> UUID-style names, so I have no idea which actual files they belong to.
> File sizes are generally either 0 bytes or 4M (exactly), although one of
> them has a size slightly larger than 3M.  So I'm assuming they're chunks
> of larger files (which would be almost all the files on the volume -
> it's primarily holding disk image files for kvm servers).
>
> Web searches generally seem to consider gfid mismatches to be a form of
> split-brain, but `gluster volume heal palantir info split-brain` shows
> "Number of entries in split-brain: 0" for all bricks, including those
> bricks which are reporting gfid mismatches.
>
>
> Given all that, how do I proceed with cleaning up the stale handle
> issues?  I would guess that this will involve somehow converting the
> shard filename to a "real" filename, then shutting down the
> corresponding VM and maybe doing some additional cleanup.
>
> And then there's the gfid mismatches.  Since they're for 0-byte files,
> is it safe to just ignore them on the assumption that they only hold
> metadata?  Or do I need to do some kind of split-brain resolution on
> them (even though gluster says no files are in split-brain)?
>
>
> Finally, a listing of /var/local/brick0/data/.shard on saruman, in case
> any of the information it contains (like file sizes/permissions) might
> provide clues to resolving the errors:
>
> --- cut here ---
> root@saruman:/var/local/brick0/data/.shard# ls -l
> total 63996
> -rw-rw 2 root libvirt-qemu   0 Sep 17  2018
> 0f500288-ff62-4f0b-9574-53f510b4159f.2864
> -rw-rw 2 root libvirt-qemu   0 Sep 17  2018
> 0f500288-ff62-4f0b-9574-53f510b4159f.2868
> -rw-rw 2 root libvirt-qemu   0 Sep 17  2018
> 0f500288-ff62-4f0b-9574-53f510b4159f.2879
> -rw-rw 2 root libvirt-qemu   0 Sep 17  2018
> 0f500288-ff62-4f0b-9574-53f510b4159f.2898
> -rw--- 2 root libvirt-qemu 4194304 May 17 14:42
> 291e9749-2d1b-47af-ad53-3a09ad4e64c6.229
> -rw--- 2 root libvirt-qemu 4194304 Jun 24 09:10
> 291e9749-2d1b-47af-ad53-3a09ad4e64c6.925
> -rw-rw-r-- 2 root libvirt-qemu 4194304 Jun 26 12:54
> 2df12cb0-6cf4-44ae-8b0a-4a554791187e.266
> -rw-rw-r-- 2 root libvirt-qemu 4194304 Jun 26 16:30
> 2df12cb0-6cf4-44ae-8b0a-4a554791187e.820
> -rw-r--r-- 2 root libvirt-qemu 4194304 Jun 17 20:22
> 323186b1-6296-4cbe-8275-b940cc9d65cf

Re: [Gluster-users] Removing subvolume from dist/rep volume

[Nithya Balachandran]

On Fri, 28 Jun 2019 at 14:34, Dave Sherohman  wrote:

> On Thu, Jun 27, 2019 at 12:17:10PM +0530, Nithya Balachandran wrote:
> > On Tue, 25 Jun 2019 at 15:26, Dave Sherohman  wrote:
> > > My objective is to remove nodes B and C entirely.
> > >
> > > First up is to pull their bricks from the volume:
> > >
> > > # gluster volume remove-brick myvol B:/data C:/data A:/arb1 start
> > > (wait for data to be migrated)
> > > # gluster volume remove-brick myvol B:/data C:/data A:/arb1 commit
> > >
> > >
> > There are some edge cases that may prevent a file from being migrated
> > during a remove-brick. Please do the following after this:
> >
> >1. Check the remove-brick status for any failures.  If there are any,
> >check the rebalance log file for errors.
> >2. Even if there are no failures, check the removed bricks to see if
> any
> >files have not been migrated. If there are any, please check that
> they are
> >valid files on the brick and copy them to the volume from the brick
> to the
> >mount point.
> >
> > The rest of the steps look good.
>
> Apparently, they weren't quite right.  I tried it and it just gives me
> the usage notes in return.  Transcript of the commands and output is below.
>
> Any insight on how I got the syntax wrong?
>
> --- cut here ---
> root@merlin:/# gluster volume status
> Status of volume: palantir
> Gluster process TCP Port  RDMA Port  Online
> Pid
>
> --
> Brick saruman:/var/local/brick0/data49153 0  Y
>  17995
> Brick gandalf:/var/local/brick0/data49153 0  Y
>  9415
> Brick merlin:/var/local/arbiter1/data   49170 0  Y
>  35034
> Brick azathoth:/var/local/brick0/data   49153 0  Y
>  25312
> Brick yog-sothoth:/var/local/brick0/data49152 0  Y
>  10671
> Brick merlin:/var/local/arbiter2/data   49171 0  Y
>  35043
> Brick cthulhu:/var/local/brick0/data49153 0  Y
>  21925
> Brick mordiggian:/var/local/brick0/data 49152 0  Y
>  12368
> Brick merlin:/var/local/arbiter3/data   49172 0  Y
>  35050
> Self-heal Daemon on localhost   N/A   N/AY
>  1209
> Self-heal Daemon on saruman.lub.lu.se   N/A   N/AY
>  23253
> Self-heal Daemon on gandalf.lub.lu.se   N/A   N/AY
>  9542
> Self-heal Daemon on mordiggian.lub.lu.seN/A   N/AY
>  11016
> Self-heal Daemon on yog-sothoth.lub.lu.se   N/A   N/AY
>  8126
> Self-heal Daemon on cthulhu.lub.lu.se   N/A   N/AY
>  30998
> Self-heal Daemon on azathoth.lub.lu.se  N/A   N/AY
>  34399
>
> Task Status of Volume palantir
>
> --
> Task : Rebalance
> ID   : e58bc091-5809-4364-af83-2b89bc5c7106
> Status   : completed
>
> root@merlin:/# gluster volume remove-brick palantir
> saruman:/var/local/brick0/data gandalf:/var/local/brick0/data
> merlin:/var/local/arbiter1/data
>
>

You had it  right in the first email.

 gluster volume remove-brick palantir replica 3 arbiter 1
saruman:/var/local/brick0/data gandalf:/var/local/brick0/data
merlin:/var/local/arbiter1/data *start*


Usage:
> volume remove-brick  [replica ]  ...
> 
>
> root@merlin:/# gluster volume remove-brick palantir replica 3 arbiter 1
> saruman:/var/local/brick0/data gandalf:/var/local/brick0/data
> merlin:/var/local/arbiter1/data
>
> Usage:
> volume remove-brick  [replica ]  ...
> 
>
> root@merlin:/# gluster volume remove-brick palantir replica 3
> saruman:/var/local/brick0/data gandalf:/var/local/brick0/data
> merlin:/var/local/arbiter1/data
>
> Usage:
> volume remove-brick  [replica ]  ...
> 
> --- cut here ---
>
> --
> Dave Sherohman
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Removing subvolume from dist/rep volume

[Nithya Balachandran]

On Thu, 27 Jun 2019 at 12:17, Nithya Balachandran 
wrote:

> Hi,
>
>
> On Tue, 25 Jun 2019 at 15:26, Dave Sherohman  wrote:
>
>> I have a 9-brick, replica 2+A cluster and plan to (permanently) remove
>> one of the three subvolumes.  I think I've worked out how to do it, but
>> want to verify first that I've got it right, since downtime or data loss
>> would be Bad Things.
>>
>> The current configuration has six data bricks across six hosts (B
>> through G), and all three arbiter bricks on the same host (A), such as
>> one might create with
>>
>> # gluster volume create myvol replica 3 arbiter 1 B:/data C:/data A:/arb1
>> D:/data E:/data A:/arb2 F:/data G:/data A:/arb3
>>
>>
>> My objective is to remove nodes B and C entirely.
>>
>> First up is to pull their bricks from the volume:
>>
>> # gluster volume remove-brick myvol B:/data C:/data A:/arb1 start
>> (wait for data to be migrated)
>> # gluster volume remove-brick myvol B:/data C:/data A:/arb1 commit
>>
>>
> There are some edge cases that may prevent a file from being migrated
> during a remove-brick. Please do the following after this:
>
>1. Check the remove-brick status for any failures.  If there are any,
>check the rebalance log file for errors.
>2. Even if there are no failures, check the removed bricks to see if
>any files have not been migrated. If there are any, please check that they
>are valid files on the brick and that they match on both bricks (files are
>not in split brain) and copy them to the volume from the brick to the mount
>point.
>
> You can run the following at the root of the brick to find any files that
have not been migrated:

find . -not \( -path ./.glusterfs -prune \) -type f -not -perm 01000


> The rest of the steps look good.
>
> Regards,
> Nithya
>
>> And then remove the nodes with:
>>
>> # gluster peer detach B
>> # gluster peer detach C
>>
>>
>> Is this correct, or did I forget any steps and/or mangle the syntax on
>> any commands?
>>
>> Also, for the remove-brick command, is there any way to throttle the
>> amount of bandwidth which will be used for the data migration?
>> Unfortunately, I was not able to provision a dedicated VLAN for the
>> gluster servers to communicate among themselves, so I don't want it
>> hogging all available capacity if that can be avoided.
>>
>>
>> If it makes a difference, my gluster version is 3.12.15-1, running on
>> Debian and installed from the debs at
>>
>> deb
>> https://download.gluster.org/pub/gluster/glusterfs/3.12/LATEST/Debian/9/amd64/apt
>> stretch main
>>
>> --
>> Dave Sherohman
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Removing subvolume from dist/rep volume

[Nithya Balachandran]

Hi,


On Tue, 25 Jun 2019 at 15:26, Dave Sherohman  wrote:

> I have a 9-brick, replica 2+A cluster and plan to (permanently) remove
> one of the three subvolumes.  I think I've worked out how to do it, but
> want to verify first that I've got it right, since downtime or data loss
> would be Bad Things.
>
> The current configuration has six data bricks across six hosts (B
> through G), and all three arbiter bricks on the same host (A), such as
> one might create with
>
> # gluster volume create myvol replica 3 arbiter 1 B:/data C:/data A:/arb1
> D:/data E:/data A:/arb2 F:/data G:/data A:/arb3
>
>
> My objective is to remove nodes B and C entirely.
>
> First up is to pull their bricks from the volume:
>
> # gluster volume remove-brick myvol B:/data C:/data A:/arb1 start
> (wait for data to be migrated)
> # gluster volume remove-brick myvol B:/data C:/data A:/arb1 commit
>
>
There are some edge cases that may prevent a file from being migrated
during a remove-brick. Please do the following after this:

   1. Check the remove-brick status for any failures.  If there are any,
   check the rebalance log file for errors.
   2. Even if there are no failures, check the removed bricks to see if any
   files have not been migrated. If there are any, please check that they are
   valid files on the brick and copy them to the volume from the brick to the
   mount point.

The rest of the steps look good.

Regards,
Nithya

> And then remove the nodes with:
>
> # gluster peer detach B
> # gluster peer detach C
>
>
> Is this correct, or did I forget any steps and/or mangle the syntax on
> any commands?
>
> Also, for the remove-brick command, is there any way to throttle the
> amount of bandwidth which will be used for the data migration?
> Unfortunately, I was not able to provision a dedicated VLAN for the
> gluster servers to communicate among themselves, so I don't want it
> hogging all available capacity if that can be avoided.
>
>
> If it makes a difference, my gluster version is 3.12.15-1, running on
> Debian and installed from the debs at
>
> deb
> https://download.gluster.org/pub/gluster/glusterfs/3.12/LATEST/Debian/9/amd64/apt
> stretch main
>
> --
> Dave Sherohman
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Does replace-brick migrate data?

[Nithya Balachandran]

On Sat, 8 Jun 2019 at 01:29, Alan Orth  wrote:

> Dear Ravi,
>
> In the last week I have completed a fix-layout and a full INDEX heal on
> this volume. Now I've started a rebalance and I see a few terabytes of data
> going around on different bricks since yesterday, which I'm sure is good.
>
> While I wait for the rebalance to finish, I'm wondering if you know what
> would cause directories to be missing from the FUSE mount point? If I list
> the directories explicitly I can see their contents, but they do not appear
> in their parent directories' listing. In the case of duplicated files it is
> always because the files are not on the correct bricks (according to the
> Dynamo/Elastic Hash algorithm), and I can fix it by copying the file to the
> correct brick(s) and removing it from the others (along with their
> .glusterfs hard links). So what could cause directories to be missing?
>
> Hi Alan,

The directories that don't show up in the parent directory listing is
probably because they do not exist on the hashed subvol. Please check the
backend bricks to see if they are missing on any of them.

Regards,
Nithya

Thank you,
>
> Thank you,
>
> On Wed, Jun 5, 2019 at 1:08 AM Alan Orth  wrote:
>
>> Hi Ravi,
>>
>> You're right that I had mentioned using rsync to copy the brick content
>> to a new host, but in the end I actually decided not to bring it up on a
>> new brick. Instead I added the original brick back into the volume. So the
>> xattrs and symlinks to .glusterfs on the original brick are fine. I think
>> the problem probably lies with a remove-brick that got interrupted. A few
>> weeks ago during the maintenance I had tried to remove a brick and then
>> after twenty minutes and no obvious progress I stopped it—after that the
>> bricks were still part of the volume.
>>
>> In the last few days I have run a fix-layout that took 26 hours and
>> finished successfully. Then I started a full index heal and it has healed
>> about 3.3 million files in a few days and I see a clear increase of network
>> traffic from old brick host to new brick host over that time. Once the full
>> index heal completes I will try to do a rebalance.
>>
>> Thank you,
>>
>>
>> On Mon, Jun 3, 2019 at 7:40 PM Ravishankar N 
>> wrote:
>>
>>>
>>> On 01/06/19 9:37 PM, Alan Orth wrote:
>>>
>>> Dear Ravi,
>>>
>>> The .glusterfs hardlinks/symlinks should be fine. I'm not sure how I
>>> could verify them for six bricks and millions of files, though... :\
>>>
>>> Hi Alan,
>>>
>>> The reason I asked this is because you had mentioned in one of your
>>> earlier emails that when you moved content from the old brick to the new
>>> one, you had skipped the .glusterfs directory. So I was assuming that when
>>> you added back this new brick to the cluster, it might have been missing
>>> the .glusterfs entries. If that is the cae, one way to verify could be to
>>> check using a script if all files on the brick have a link-count of at
>>> least 2 and all dirs have valid symlinks inside .glusterfs pointing to
>>> themselves.
>>>
>>>
>>> I had a small success in fixing some issues with duplicated files on the
>>> FUSE mount point yesterday. I read quite a bit about the elastic hashing
>>> algorithm that determines which files get placed on which bricks based on
>>> the hash of their filename and the trusted.glusterfs.dht xattr on brick
>>> directories (thanks to Joe Julian's blog post and Python script for showing
>>> how it works¹). With that knowledge I looked closer at one of the files
>>> that was appearing as duplicated on the FUSE mount and found that it was
>>> also duplicated on more than `replica 2` bricks. For this particular file I
>>> found two "real" files and several zero-size files with
>>> trusted.glusterfs.dht.linkto xattrs. Neither of the "real" files were on
>>> the correct brick as far as the DHT layout is concerned, so I copied one of
>>> them to the correct brick, deleted the others and their hard links, and did
>>> a `stat` on the file from the FUSE mount point and it fixed itself. Yay!
>>>
>>> Could this have been caused by a replace-brick that got interrupted and
>>> didn't finish re-labeling the xattrs?
>>>
>>> No, replace-brick only initiates AFR self-heal, which just copies the
>>> contents from the other brick(s) of the *same* replica pair into the
>>> replaced brick.  The link-to files are created by DHT when you rename a
>>> file from the client. If the new name hashes to a different  brick, DHT
>>> does not move the entire file there. It instead creates the link-to file
>>> (the one with the dht.linkto xattrs) on the hashed subvol. The value of
>>> this xattr points to the brick where the actual data is there (`getfattr -e
>>> text` to see it for yourself).  Perhaps you had attempted a rebalance or
>>> remove-brick earlier and interrupted that?
>>>
>>> Should I be thinking of some heuristics to identify and fix these issues
>>> with a script (incorrect brick placement), or is this something a fix
>>> layout or repeated 

Re: [Gluster-users] Memory leak in glusterfs

[Nithya Balachandran]

Hi Abhishek,

Please use statedumps taken at intervals to determine where the memory is
increasing. See [1] for details.

Regards,
Nithya

[1] https://docs.gluster.org/en/latest/Troubleshooting/statedump/


On Fri, 7 Jun 2019 at 08:13, ABHISHEK PALIWAL 
wrote:

> Hi Nithya,
>
> We are having the setup where copying the file to and deleting it from
> gluster mount point to update the latest file. We noticed due to this
> having some memory increase in glusterfsd process.
>
> To find the memory leak we are using valgrind but didn't get any help.
>
> That's why contacted to glusterfs community.
>
> Regards,
> Abhishek
>
> On Thu, Jun 6, 2019, 16:08 Nithya Balachandran 
> wrote:
>
>> Hi Abhishek,
>>
>> I am still not clear as to the purpose of the tests. Can you clarify why
>> you are using valgrind and why you think there is a memory leak?
>>
>> Regards,
>> Nithya
>>
>> On Thu, 6 Jun 2019 at 12:09, ABHISHEK PALIWAL 
>> wrote:
>>
>>> Hi Nithya,
>>>
>>> Here is the Setup details and test which we are doing as below:
>>>
>>>
>>> One client, two gluster Server.
>>> The client is writing and deleting one file each 15 minutes by script
>>> test_v4.15.sh.
>>>
>>> IP
>>> Server side:
>>> 128.224.98.157 /gluster/gv0/
>>> 128.224.98.159 /gluster/gv0/
>>>
>>> Client side:
>>> 128.224.98.160 /gluster_mount/
>>>
>>> Server side:
>>> gluster volume create gv0 replica 2 128.224.98.157:/gluster/gv0/
>>> 128.224.98.159:/gluster/gv0/ force
>>> gluster volume start gv0
>>>
>>> root@128:/tmp/brick/gv0# gluster volume info
>>>
>>> Volume Name: gv0
>>> Type: Replicate
>>> Volume ID: 7105a475-5929-4d60-ba23-be57445d97b5
>>> Status: Started
>>> Snapshot Count: 0
>>> Number of Bricks: 1 x 2 = 2
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: 128.224.98.157:/gluster/gv0
>>> Brick2: 128.224.98.159:/gluster/gv0
>>> Options Reconfigured:
>>> transport.address-family: inet
>>> nfs.disable: on
>>> performance.client-io-threads: off
>>>
>>> exec script: ./ps_mem.py -p 605 -w 61 > log
>>> root@128:/# ./ps_mem.py -p 605
>>> Private + Shared = RAM used Program
>>> 23668.0 KiB + 1188.0 KiB = 24856.0 KiB glusterfsd
>>> -
>>> 24856.0 KiB
>>> =====
>>>
>>>
>>> Client side:
>>> mount -t glusterfs -o acl -o resolve-gids 128.224.98.157:gv0
>>> /gluster_mount
>>>
>>>
>>> We are using the below script write and delete the file.
>>>
>>> *test_v4.15.sh <http://test_v4.15.sh>*
>>>
>>> Also the below script to see the memory increase whihle the script is
>>> above script is running in background.
>>>
>>> *ps_mem.py*
>>>
>>> I am attaching the script files as well as the result got after testing
>>> the scenario.
>>>
>>> On Wed, Jun 5, 2019 at 7:23 PM Nithya Balachandran 
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> Writing to a volume should not affect glusterd. The stack you have
>>>> shown in the valgrind looks like the memory used to initialise the
>>>> structures glusterd uses and will free only when it is stopped.
>>>>
>>>> Can you provide more details to what it is you are trying to test?
>>>>
>>>> Regards,
>>>> Nithya
>>>>
>>>>
>>>> On Tue, 4 Jun 2019 at 15:41, ABHISHEK PALIWAL 
>>>> wrote:
>>>>
>>>>> Hi Team,
>>>>>
>>>>> Please respond on the issue which I raised.
>>>>>
>>>>> Regards,
>>>>> Abhishek
>>>>>
>>>>> On Fri, May 17, 2019 at 2:46 PM ABHISHEK PALIWAL <
>>>>> abhishpali...@gmail.com> wrote:
>>>>>
>>>>>> Anyone please reply
>>>>>>
>>>>>> On Thu, May 16, 2019, 10:49 ABHISHEK PALIWAL 
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Team,
>>>>>>>
>>>>>>> I upload some valgrind logs from my gluster 5.4 setup. This is
>>>>>>> writing to the volume every 15 minutes. I stopped glusterd and then copy
>>>>>>> away the logs.  The test was running for some simulated days. 

Re: [Gluster-users] Memory leak in glusterfs

[Nithya Balachandran]

Hi Abhishek,

I am still not clear as to the purpose of the tests. Can you clarify why
you are using valgrind and why you think there is a memory leak?

Regards,
Nithya

On Thu, 6 Jun 2019 at 12:09, ABHISHEK PALIWAL 
wrote:

> Hi Nithya,
>
> Here is the Setup details and test which we are doing as below:
>
>
> One client, two gluster Server.
> The client is writing and deleting one file each 15 minutes by script
> test_v4.15.sh.
>
> IP
> Server side:
> 128.224.98.157 /gluster/gv0/
> 128.224.98.159 /gluster/gv0/
>
> Client side:
> 128.224.98.160 /gluster_mount/
>
> Server side:
> gluster volume create gv0 replica 2 128.224.98.157:/gluster/gv0/
> 128.224.98.159:/gluster/gv0/ force
> gluster volume start gv0
>
> root@128:/tmp/brick/gv0# gluster volume info
>
> Volume Name: gv0
> Type: Replicate
> Volume ID: 7105a475-5929-4d60-ba23-be57445d97b5
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x 2 = 2
> Transport-type: tcp
> Bricks:
> Brick1: 128.224.98.157:/gluster/gv0
> Brick2: 128.224.98.159:/gluster/gv0
> Options Reconfigured:
> transport.address-family: inet
> nfs.disable: on
> performance.client-io-threads: off
>
> exec script: ./ps_mem.py -p 605 -w 61 > log
> root@128:/# ./ps_mem.py -p 605
> Private + Shared = RAM used Program
> 23668.0 KiB + 1188.0 KiB = 24856.0 KiB glusterfsd
> -
> 24856.0 KiB
> =
>
>
> Client side:
> mount -t glusterfs -o acl -o resolve-gids 128.224.98.157:gv0
> /gluster_mount
>
>
> We are using the below script write and delete the file.
>
> *test_v4.15.sh <http://test_v4.15.sh>*
>
> Also the below script to see the memory increase whihle the script is
> above script is running in background.
>
> *ps_mem.py*
>
> I am attaching the script files as well as the result got after testing
> the scenario.
>
> On Wed, Jun 5, 2019 at 7:23 PM Nithya Balachandran 
> wrote:
>
>> Hi,
>>
>> Writing to a volume should not affect glusterd. The stack you have shown
>> in the valgrind looks like the memory used to initialise the structures
>> glusterd uses and will free only when it is stopped.
>>
>> Can you provide more details to what it is you are trying to test?
>>
>> Regards,
>> Nithya
>>
>>
>> On Tue, 4 Jun 2019 at 15:41, ABHISHEK PALIWAL 
>> wrote:
>>
>>> Hi Team,
>>>
>>> Please respond on the issue which I raised.
>>>
>>> Regards,
>>> Abhishek
>>>
>>> On Fri, May 17, 2019 at 2:46 PM ABHISHEK PALIWAL <
>>> abhishpali...@gmail.com> wrote:
>>>
>>>> Anyone please reply
>>>>
>>>> On Thu, May 16, 2019, 10:49 ABHISHEK PALIWAL 
>>>> wrote:
>>>>
>>>>> Hi Team,
>>>>>
>>>>> I upload some valgrind logs from my gluster 5.4 setup. This is writing
>>>>> to the volume every 15 minutes. I stopped glusterd and then copy away the
>>>>> logs.  The test was running for some simulated days. They are zipped in
>>>>> valgrind-54.zip.
>>>>>
>>>>> Lots of info in valgrind-2730.log. Lots of possibly lost bytes in
>>>>> glusterfs and even some definitely lost bytes.
>>>>>
>>>>> ==2737== 1,572,880 bytes in 1 blocks are possibly lost in loss record
>>>>> 391 of 391
>>>>> ==2737== at 0x4C29C25: calloc (in
>>>>> /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
>>>>> ==2737== by 0xA22485E: ??? (in
>>>>> /usr/lib64/glusterfs/5.4/xlator/mgmt/glusterd.so)
>>>>> ==2737== by 0xA217C94: ??? (in
>>>>> /usr/lib64/glusterfs/5.4/xlator/mgmt/glusterd.so)
>>>>> ==2737== by 0xA21D9F8: ??? (in
>>>>> /usr/lib64/glusterfs/5.4/xlator/mgmt/glusterd.so)
>>>>> ==2737== by 0xA21DED9: ??? (in
>>>>> /usr/lib64/glusterfs/5.4/xlator/mgmt/glusterd.so)
>>>>> ==2737== by 0xA21E685: ??? (in
>>>>> /usr/lib64/glusterfs/5.4/xlator/mgmt/glusterd.so)
>>>>> ==2737== by 0xA1B9D8C: init (in
>>>>> /usr/lib64/glusterfs/5.4/xlator/mgmt/glusterd.so)
>>>>> ==2737== by 0x4E511CE: xlator_init (in
>>>>> /usr/lib64/libglusterfs.so.0.0.1)
>>>>> ==2737== by 0x4E8A2B8: ??? (in /usr/lib64/libglusterfs.so.0.0.1)
>>>>> ==2737== by 0x4E8AAB3: glusterfs_graph_activate (in
>>>>> /usr/lib64/libglusterfs.so.0.0.1)
>>>>> ==2737== by 0x409C35: glusterfs_process_volfp (in /usr/

Re: [Gluster-users] Memory leak in glusterfs

[Nithya Balachandran]

Hi,

Writing to a volume should not affect glusterd. The stack you have shown in
the valgrind looks like the memory used to initialise the structures
glusterd uses and will free only when it is stopped.

Can you provide more details to what it is you are trying to test?

Regards,
Nithya


On Tue, 4 Jun 2019 at 15:41, ABHISHEK PALIWAL 
wrote:

> Hi Team,
>
> Please respond on the issue which I raised.
>
> Regards,
> Abhishek
>
> On Fri, May 17, 2019 at 2:46 PM ABHISHEK PALIWAL 
> wrote:
>
>> Anyone please reply
>>
>> On Thu, May 16, 2019, 10:49 ABHISHEK PALIWAL 
>> wrote:
>>
>>> Hi Team,
>>>
>>> I upload some valgrind logs from my gluster 5.4 setup. This is writing
>>> to the volume every 15 minutes. I stopped glusterd and then copy away the
>>> logs.  The test was running for some simulated days. They are zipped in
>>> valgrind-54.zip.
>>>
>>> Lots of info in valgrind-2730.log. Lots of possibly lost bytes in
>>> glusterfs and even some definitely lost bytes.
>>>
>>> ==2737== 1,572,880 bytes in 1 blocks are possibly lost in loss record
>>> 391 of 391
>>> ==2737== at 0x4C29C25: calloc (in
>>> /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
>>> ==2737== by 0xA22485E: ??? (in
>>> /usr/lib64/glusterfs/5.4/xlator/mgmt/glusterd.so)
>>> ==2737== by 0xA217C94: ??? (in
>>> /usr/lib64/glusterfs/5.4/xlator/mgmt/glusterd.so)
>>> ==2737== by 0xA21D9F8: ??? (in
>>> /usr/lib64/glusterfs/5.4/xlator/mgmt/glusterd.so)
>>> ==2737== by 0xA21DED9: ??? (in
>>> /usr/lib64/glusterfs/5.4/xlator/mgmt/glusterd.so)
>>> ==2737== by 0xA21E685: ??? (in
>>> /usr/lib64/glusterfs/5.4/xlator/mgmt/glusterd.so)
>>> ==2737== by 0xA1B9D8C: init (in
>>> /usr/lib64/glusterfs/5.4/xlator/mgmt/glusterd.so)
>>> ==2737== by 0x4E511CE: xlator_init (in /usr/lib64/libglusterfs.so.0.0.1)
>>> ==2737== by 0x4E8A2B8: ??? (in /usr/lib64/libglusterfs.so.0.0.1)
>>> ==2737== by 0x4E8AAB3: glusterfs_graph_activate (in
>>> /usr/lib64/libglusterfs.so.0.0.1)
>>> ==2737== by 0x409C35: glusterfs_process_volfp (in /usr/sbin/glusterfsd)
>>> ==2737== by 0x409D99: glusterfs_volumes_init (in /usr/sbin/glusterfsd)
>>> ==2737==
>>> ==2737== LEAK SUMMARY:
>>> ==2737== definitely lost: 1,053 bytes in 10 blocks
>>> ==2737== indirectly lost: 317 bytes in 3 blocks
>>> ==2737== possibly lost: 2,374,971 bytes in 524 blocks
>>> ==2737== still reachable: 53,277 bytes in 201 blocks
>>> ==2737== suppressed: 0 bytes in 0 blocks
>>>
>>> --
>>>
>>>
>>>
>>>
>>> Regards
>>> Abhishek Paliwal
>>>
>>
>
> --
>
>
>
>
> Regards
> Abhishek Paliwal
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] remove-brick failure on distributed with 5.6

[Nithya Balachandran]

Hi Brandon,

Please send the following:

1. the gluster volume info
2. Information about which brick was removed
3. The rebalance log file for all nodes hosting removed bricks.

Regards,
Nithya


On Fri, 24 May 2019 at 19:33, Ravishankar N  wrote:

> Adding a few DHT folks for some possible suggestions.
>
> -Ravi
> On 23/05/19 11:15 PM, bran...@thinkhuge.net wrote:
>
> Does anyone know what should be done on a glusterfs v5.6 "gluster volume
> remove-brick" operation that fails?  I'm trying to remove 1 of 8
> distributed smaller nodes for replacement with larger node.
>
>
>
> The "gluster volume remove-brick ... status" command reports status failed
> and failures = "3"
>
>
>
> cat /var/log/glusterfs/volbackups-rebalance.log
>
> ...
>
> [2019-05-23 16:43:37.442283] I [MSGID: 109028]
> [dht-rebalance.c:5070:gf_defrag_status_get] 0-volbackups-dht: Rebalance is
> failed. Time taken is 545.00 secs
>
>
>
> All servers are confirmed in good communications and updated and freshly
> rebooted and retried the remove-brick few times with fail each time
>
>
>
> ___
> Gluster-users mailing 
> listGluster-users@gluster.orghttps://lists.gluster.org/mailman/listinfo/gluster-users
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] add-brick: failed: Commit failed

[Nithya Balachandran]

On Fri, 17 May 2019 at 06:01, David Cunningham 
wrote:

> Hello,
>
> We're adding an arbiter node to an existing volume and having an issue.
> Can anyone help? The root cause error appears to be
> "----0001: failed to resolve (Transport
> endpoint is not connected)", as below.
>
> We are running glusterfs 5.6.1. Thanks in advance for any assistance!
>
> On existing node gfs1, trying to add new arbiter node gfs3:
>
> # gluster volume add-brick gvol0 replica 3 arbiter 1
> gfs3:/nodirectwritedata/gluster/gvol0
> volume add-brick: failed: Commit failed on gfs3. Please check log file for
> details.
>

This looks like a glusterd issue. Please check the glusterd logs for more
info.
Adding the glusterd dev to this thread. Sanju, can you take a look?

Regards,
Nithya

>
> On new node gfs3 in gvol0-add-brick-mount.log:
>
> [2019-05-17 01:20:22.689721] I [fuse-bridge.c:4267:fuse_init]
> 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel
> 7.22
> [2019-05-17 01:20:22.689778] I [fuse-bridge.c:4878:fuse_graph_sync]
> 0-fuse: switched to graph 0
> [2019-05-17 01:20:22.694897] E [fuse-bridge.c:4336:fuse_first_lookup]
> 0-fuse: first lookup on root failed (Transport endpoint is not connected)
> [2019-05-17 01:20:22.699770] W [fuse-resolve.c:127:fuse_resolve_gfid_cbk]
> 0-fuse: ----0001: failed to resolve (Transport
> endpoint is not connected)
> [2019-05-17 01:20:22.699834] W [fuse-bridge.c:3294:fuse_setxattr_resume]
> 0-glusterfs-fuse: 2: SETXATTR ----0001/1
> (trusted.add-brick) resolution failed
> [2019-05-17 01:20:22.715656] I [fuse-bridge.c:5144:fuse_thread_proc]
> 0-fuse: initating unmount of /tmp/mntQAtu3f
> [2019-05-17 01:20:22.715865] W [glusterfsd.c:1500:cleanup_and_exit]
> (-->/lib64/libpthread.so.0(+0x7dd5) [0x7fb223bf6dd5]
> -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x560886581e75]
> -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x560886581ceb] ) 0-:
> received signum (15), shutting down
> [2019-05-17 01:20:22.715926] I [fuse-bridge.c:5914:fini] 0-fuse:
> Unmounting '/tmp/mntQAtu3f'.
> [2019-05-17 01:20:22.715953] I [fuse-bridge.c:5919:fini] 0-fuse: Closing
> fuse connection to '/tmp/mntQAtu3f'.
>
> Processes running on new node gfs3:
>
> # ps -ef | grep gluster
> root  6832 1  0 20:17 ?00:00:00 /usr/sbin/glusterd -p
> /var/run/glusterd.pid --log-level INFO
> root 15799 1  0 20:17 ?00:00:00 /usr/sbin/glusterfs -s
> localhost --volfile-id gluster/glustershd -p
> /var/run/gluster/glustershd/glustershd.pid -l
> /var/log/glusterfs/glustershd.log -S
> /var/run/gluster/24c12b09f93eec8e.socket --xlator-option
> *replicate*.node-uuid=2069cfb3-c798-47e3-8cf8-3c584cf7c412 --process-name
> glustershd
> root 16856 16735  0 21:21 pts/000:00:00 grep --color=auto gluster
>
> --
> David Cunningham, Voisonics Limited
> http://voisonics.com/
> USA: +1 213 221 1092
> New Zealand: +64 (0)28 2558 3782
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Cannot see all data in mount

[Nithya Balachandran]

On Thu, 16 May 2019 at 14:17, Paul van der Vlis  wrote:

> Op 16-05-19 om 05:43 schreef Nithya Balachandran:
> >
> >
> > On Thu, 16 May 2019 at 03:05, Paul van der Vlis  > <mailto:p...@vandervlis.nl>> wrote:
> >
> > Op 15-05-19 om 15:45 schreef Nithya Balachandran:
> > > Hi Paul,
> > >
> > > A few questions:
> > > Which version of gluster are you using?
> >
> > On the server and some clients: glusterfs 4.1.2
> > On a new client: glusterfs 5.5
> >
> > Is the same behaviour seen on both client versions?
>
> Yes.
>
> > > Did this behaviour start recently? As in were the contents of that
> > > directory visible earlier?
> >
> > This directory was normally used in the headoffice, and there is
> direct
> > access to the files without Glusterfs. So I don't know.
> >
> >
> > Do you mean that they access the files on the gluster volume without
> > using the client or that these files were stored elsewhere
> > earlier (not on gluster)? Files on a gluster volume should never be
> > accessed directly.
>
> The central server (this is the only gluster-brick) is a thin-client
> server, people are working directly on the server using LTSP terminals:
> http://ltsp.org/).
>
> The data is exported using Gluster to some other machines in smaller
> offices.
>
> And to a new thin-client server what I am making (using X2go). The goal
> is that this server will replace all of the excisting machines in the
> future. X2go is something like "Citrix for Linux", you can use it over
> the internet.
>
> I did not setup Gluster and I have never met the old sysadmin. I guess
> it's also very strange to use Gluster with only one brick. So when I
> understand you right, the whole setup is wrong, and you may not access
> the files without client?
>
>
That is correct - any files on a gluster volume should be accessed only via
a gluster client (if using fuse).


> > To debug this further, please send the following:
> >
> >  1. The directory contents when the listing is performed directly on the
> > brick.
> >  2. The tcpdump of the gluster client when listing the directory using
> > the following command:
> >
> > tcpdump -i any -s 0 -w /var/tmp/dirls.pcap tcp and not port 22
> >
> >
> > You can send these directly to me in case you want to keep the
> > information private.
>
> I have just heard (during writing this message) that the owner of the
> firm where I make this for, is in hospital in very critical condition.
> They've asked me to stop with the work at the moment.
>
> I did also hear that there where more problems with the filesystem.
> Especially when a directory was renamed.
> And this directory was renamed in the past.
>
>
Let me know when you plan to continue with this . We can take a look.

Regards,
Nithya


> With regards,
> Paul van der Vlis
>
> > Regards,
> > Nithya
> >
> >
> >
> > With regards,
> > Paul van der Vlis
> >
> > > Regards,
> > > Nithya
> > >
> > >
> > > On Wed, 15 May 2019 at 18:55, Paul van der Vlis
> > mailto:p...@vandervlis.nl>
> > > <mailto:p...@vandervlis.nl <mailto:p...@vandervlis.nl>>> wrote:
> > >
> > > Hello Strahil,
> > >
> > > Thanks for your answer. I don't find the word "sharding" in the
> > > configfiles. There is not much shared data (24GB), and only 1
> > brick:
> > > ---
> > > root@xxx:/etc/glusterfs# gluster volume info DATA
> > >
> > > Volume Name: DATA
> > > Type: Distribute
> > > Volume ID: db53ece1-5def-4f7c-b59d-3a230824032a
> > > Status: Started
> > > Snapshot Count: 0
> > > Number of Bricks: 1
> > > Transport-type: tcp
> > > Bricks:
> > > Brick1: xxx-vpn:/DATA
> > > Options Reconfigured:
> > > transport.address-family: inet
> > > nfs.disable: on
> > > 
> > > (I have edited this a bit for privacy of my customer).
> > >
> > > I think they have used glusterfs because it can do ACLs.
> > >
> > > With regards,
> > > Paul van der Vlis
> > >
> > >
> > > Op 15-05-19 om 14:59 schreef Strahil Nikolov:
> > >

Re: [Gluster-users] Cannot see all data in mount

[Nithya Balachandran]

On Thu, 16 May 2019 at 03:05, Paul van der Vlis  wrote:

> Op 15-05-19 om 15:45 schreef Nithya Balachandran:
> > Hi Paul,
> >
> > A few questions:
> > Which version of gluster are you using?
>
> On the server and some clients: glusterfs 4.1.2
> On a new client: glusterfs 5.5
>
> Is the same behaviour seen on both client versions?


> > Did this behaviour start recently? As in were the contents of that
> > directory visible earlier?
>
> This directory was normally used in the headoffice, and there is direct
> access to the files without Glusterfs. So I don't know.
>

Do you mean that they access the files on the gluster volume without using
the client or that these files were stored elsewhere
earlier (not on gluster)? Files on a gluster volume should never be
accessed directly.

To debug this further, please send the following:

   1. The directory contents when the listing is performed directly on the
   brick.
   2. The tcpdump of the gluster client when listing the directory using
   the following command:

tcpdump -i any -s 0 -w /var/tmp/dirls.pcap tcp and not port 22


You can send these directly to me in case you want to keep the information
private.

Regards,
Nithya


>
> With regards,
> Paul van der Vlis
>
> > Regards,
> > Nithya
> >
> >
> > On Wed, 15 May 2019 at 18:55, Paul van der Vlis  > <mailto:p...@vandervlis.nl>> wrote:
> >
> > Hello Strahil,
> >
> > Thanks for your answer. I don't find the word "sharding" in the
> > configfiles. There is not much shared data (24GB), and only 1 brick:
> > ---
> > root@xxx:/etc/glusterfs# gluster volume info DATA
> >
> > Volume Name: DATA
> > Type: Distribute
> > Volume ID: db53ece1-5def-4f7c-b59d-3a230824032a
> > Status: Started
> > Snapshot Count: 0
> > Number of Bricks: 1
> > Transport-type: tcp
> > Bricks:
> > Brick1: xxx-vpn:/DATA
> > Options Reconfigured:
> > transport.address-family: inet
> > nfs.disable: on
> > 
> > (I have edited this a bit for privacy of my customer).
> >
> > I think they have used glusterfs because it can do ACLs.
> >
> > With regards,
> > Paul van der Vlis
> >
> >
> > Op 15-05-19 om 14:59 schreef Strahil Nikolov:
> > > Most probably you use sharding , which splits the files into
> smaller
> > > chunks so you can fit a 1TB file into gluster nodes with bricks of
> > > smaller size.
> > > So if you have 2 dispersed servers each having 500Gb brick->
> without
> > > sharding you won't be able to store files larger than the brick
> size -
> > > no matter you have free space on the other server.
> > >
> > > When sharding is enabled - you will see on the brick the first
> > shard as
> > > a file and the rest is in a hidden folder called ".shards" (or
> > something
> > > like that).
> > >
> > > The benefit is also viewable when you need to do some maintenance
> on a
> > > gluster node, as you will need to heal only the shards containing
> > > modified by the customers' data.
> > >
> > > Best Regards,
> > > Strahil Nikolov
> > >
> > >
> > > В сряда, 15 май 2019 г., 7:31:39 ч. Гринуич-4, Paul van der Vlis
> > > mailto:p...@vandervlis.nl>> написа:
> > >
> > >
> > > Hello,
> > >
> > > I am the new sysadmin of an organization what uses Glusterfs.
> > > I did not set it up, and I don't know much about Glusterfs.
> > >
> > > What I do not understand is that I do not see all data in the
> mount.
> > > Not as root, not as a normal user who has privileges.
> > >
> > > When I do "ls" in one of the subdirectories I don't see any data,
> but
> > > this data exists at the server!
> > >
> > > In another subdirectory I see everything fine, the rights of the
> > > directories and files inside are the same.
> > >
> > > I mount with something like:
> > > /bin/mount -t glusterfs -o acl 10.8.0.1:/data /data
> > > I see data in /data/VOORBEELD/, and I don't see any data in
> > /data/ALGEMEEN/.
> > >
> > > I don't see something special in /etc/exports or in /etc/glusterfs
> on
> > > the server.
> > >
> >

Re: [Gluster-users] Cannot see all data in mount

[Nithya Balachandran]

Hi Paul,

A few questions:
Which version of gluster are you using?
Did this behaviour start recently? As in were the contents of that
directory visible earlier?

Regards,
Nithya


On Wed, 15 May 2019 at 18:55, Paul van der Vlis  wrote:

> Hello Strahil,
>
> Thanks for your answer. I don't find the word "sharding" in the
> configfiles. There is not much shared data (24GB), and only 1 brick:
> ---
> root@xxx:/etc/glusterfs# gluster volume info DATA
>
> Volume Name: DATA
> Type: Distribute
> Volume ID: db53ece1-5def-4f7c-b59d-3a230824032a
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1
> Transport-type: tcp
> Bricks:
> Brick1: xxx-vpn:/DATA
> Options Reconfigured:
> transport.address-family: inet
> nfs.disable: on
> 
> (I have edited this a bit for privacy of my customer).
>
> I think they have used glusterfs because it can do ACLs.
>
> With regards,
> Paul van der Vlis
>
>
> Op 15-05-19 om 14:59 schreef Strahil Nikolov:
> > Most probably you use sharding , which splits the files into smaller
> > chunks so you can fit a 1TB file into gluster nodes with bricks of
> > smaller size.
> > So if you have 2 dispersed servers each having 500Gb brick->  without
> > sharding you won't be able to store files larger than the brick size -
> > no matter you have free space on the other server.
> >
> > When sharding is enabled - you will see on the brick the first shard as
> > a file and the rest is in a hidden folder called ".shards" (or something
> > like that).
> >
> > The benefit is also viewable when you need to do some maintenance on a
> > gluster node, as you will need to heal only the shards containing
> > modified by the customers' data.
> >
> > Best Regards,
> > Strahil Nikolov
> >
> >
> > В сряда, 15 май 2019 г., 7:31:39 ч. Гринуич-4, Paul van der Vlis
> >  написа:
> >
> >
> > Hello,
> >
> > I am the new sysadmin of an organization what uses Glusterfs.
> > I did not set it up, and I don't know much about Glusterfs.
> >
> > What I do not understand is that I do not see all data in the mount.
> > Not as root, not as a normal user who has privileges.
> >
> > When I do "ls" in one of the subdirectories I don't see any data, but
> > this data exists at the server!
> >
> > In another subdirectory I see everything fine, the rights of the
> > directories and files inside are the same.
> >
> > I mount with something like:
> > /bin/mount -t glusterfs -o acl 10.8.0.1:/data /data
> > I see data in /data/VOORBEELD/, and I don't see any data in
> /data/ALGEMEEN/.
> >
> > I don't see something special in /etc/exports or in /etc/glusterfs on
> > the server.
> >
> > Is there maybe a mechanism in Glusterfs what can exclude data from
> > export?  Or is there a way to debug this problem?
> >
> > With regards,
> > Paul van der Vlis
> >
> > 
> > # file: VOORBEELD
> > # owner: root
> > # group: secretariaat
> > # flags: -s-
> > user::rwx
> > group::rwx
> > group:medewerkers:r-x
> > mask::rwx
> > other::---
> > default:user::rwx
> > default:group::rwx
> > default:group:medewerkers:r-x
> > default:mask::rwx
> > default:other::---
> >
> > # file: ALGEMEEN
> > # owner: root
> > # group: secretariaat
> > # flags: -s-
> > user::rwx
> > group::rwx
> > group:medewerkers:r-x
> > mask::rwx
> > other::---
> > default:user::rwx
> > default:group::rwx
> > default:group:medewerkers:r-x
> > default:mask::rwx
> > default:other::---
> > --
> >
> >
> >
> >
> >
> > --
> > Paul van der Vlis Linux systeembeheer Groningen
> > https://www.vandervlis.nl/
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org 
> > https://lists.gluster.org/mailman/listinfo/gluster-users
>
>
>
> --
> Paul van der Vlis Linux systeembeheer Groningen
> https://www.vandervlis.nl/
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Extremely slow Gluster performance

[Nithya Balachandran]

Hi Patrick,

Did this start only after the upgrade?
How do you determine which brick process to kill?
Are there a lot of files to be healed on the volume?

Can you provide a tcpdump of the slow listing from a separate test client
mount ?

   1. Mount the gluster volume on a different mount point than the one
   being used by your users.
   2. Start capturing packets on the system where you have mounted the
   volume in (1).
   - tcpdump -i any -s 0 -w /var/tmp/dirls.pcap tcp and not port 22
  3. List the directory that is slow from the fuse client
   4. Stop the capture (after a couple of minutes or after the listing
   returns, whichever is earlier)
   5. Send us the pcap and the listing of the same directory from one of
   the bricks in order to compare the entries.


We may need more information post looking at the tcpdump.

Regards,
Nithya

On Tue, 23 Apr 2019 at 23:39, Patrick Rennie 
wrote:

> Hello Gluster Users,
>
> I am hoping someone can help me with resolving an ongoing issue I've been
> having, I'm new to mailing lists so forgive me if I have gotten anything
> wrong. We have noticed our performance deteriorating over the last few
> weeks, easily measured by trying to do an ls on one of our top-level
> folders, and timing it, which usually would take 2-5 seconds, and now takes
> up to 20 minutes, which obviously renders our cluster basically unusable.
> This has been intermittent in the past but is now almost constant and I am
> not sure how to work out the exact cause. We have noticed some errors in
> the brick logs, and have noticed that if we kill the right brick process,
> performance instantly returns back to normal, this is not always the same
> brick, but it indicates to me something in the brick processes or
> background tasks may be causing extreme latency. Due to this ability to fix
> it by killing the right brick process off, I think it's a specific file, or
> folder, or operation which may be hanging and causing the increased
> latency, but I am not sure how to work it out. One last thing to add is
> that our bricks are getting quite full (~95% full), we are trying to
> migrate data off to new storage but that is going slowly, not helped by
> this issue. I am currently trying to run a full heal as there appear to be
> many files needing healing, and I have all brick processes running so they
> have an opportunity to heal, but this means performance is very poor. It
> currently takes over 15-20 minutes to do an ls of one of our top-level
> folders, which just contains 60-80 other folders, this should take 2-5
> seconds. This is all being checked by FUSE mount locally on the storage
> node itself, but it is the same for other clients and VMs accessing the
> cluster. Initially it seemed our NFS mounts were not affected and operated
> at normal speed, but testing over the last day has shown that our NFS
> clients are also extremely slow, so it doesn't seem specific to FUSE as I
> first thought it might be.
>
> I am not sure how to proceed from here, I am fairly new to gluster having
> inherited this setup from my predecessor and trying to keep it going. I
> have included some info below to try and help with diagnosis, please let me
> know if any further info would be helpful. I would really appreciate any
> advice on what I could try to work out the cause. Thank you in advance for
> reading this, and any suggestions you might be able to offer.
>
> - Patrick
>
> This is an example of the main error I see in our brick logs, there have
> been others, I can post them when I see them again too:
> [2019-04-20 04:54:43.055680] E [MSGID: 113001]
> [posix.c:4940:posix_getxattr] 0-gvAA01-posix: getxattr failed on
> /brick1/ library: system.posix_acl_default  [Operation not
> supported]
> [2019-04-20 05:01:29.476313] W [posix.c:4929:posix_getxattr]
> 0-gvAA01-posix: Extended attributes not supported (try remounting brick
> with 'user_xattr' flag)
>
> Our setup consists of 2 storage nodes and an arbiter node. I have noticed
> our nodes are on slightly different versions, I'm not sure if this could be
> an issue. We have 9 bricks on each node, made up of ZFS RAIDZ2 pools -
> total capacity is around 560TB.
> We have bonded 10gbps NICS on each node, and I have tested bandwidth with
> iperf and found that it's what would be expected from this config.
> Individual brick performance seems ok, I've tested several bricks using dd
> and can write a 10GB files at 1.7GB/s.
>
> # dd if=/dev/zero of=/brick1/test/test.file bs=1M count=1
> 1+0 records in
> 1+0 records out
> 1048576 bytes (10 GB, 9.8 GiB) copied, 6.20303 s, 1.7 GB/s
>
> Node 1:
> # glusterfs --version
> glusterfs 3.12.15
>
> Node 2:
> # glusterfs --version
> glusterfs 3.12.14
>
> Arbiter:
> # glusterfs --version
> glusterfs 3.12.14
>
> Here is our gluster volume status:
>
> # gluster volume status
> Status of volume: gvAA01
> Gluster process TCP Port  RDMA Port  Online
> Pid
>
> 

Re: [Gluster-users] Inconsistent issues with a client

[Nithya Balachandran]

Hi,

If you know which directories are problematic, please check and see if the
permissions on them are correct on the individual bricks.
Please also provide the following:

   - *gluster volume info* for the volume
   - The gluster version you are running


regards,
Nithya

On Wed, 27 Mar 2019 at 19:10, Tami Greene  wrote:

> The system is a 5 server, 20 brick distributed system with a hardware
> configured RAID 6 underneath with xfs as filesystem.  This client is a data
> collection node which transfers data to specific directories within one of
> the gluster volumes.
>
>
>
> I have a client with submounted directories (glustervolume/project) rather
> than the entire volume.  Some files can be transferred no problem, but
> others send an error about transport endpoint not connected.  The transfer
> is handed by a rsync script triggered as a cron job.
>
>
>
> When remotely connected to this client, user access to these files does
> not always behave as they are set – 2770 for directories and 440.  Owners
> are not always able to move the files, processes ran as the owners are not
> always able to move files; root is not always allowed to move or delete
> these file.
>
>
>
> This process seemed to worked smoothly before adding another server and 4
> storage bricks to the volume, logs indicate there were intermittent issues
> at least a month before the last server was added.  While a new collection
> device has been streaming to this one machine, the issue started the day
> before.
>
>
>
> Is there another level for permissions and ownership that I am not aware
> of that needs to be sync’d?
>
>
> --
> Tami
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Prioritise local bricks for IO?

[Nithya Balachandran]

On Wed, 27 Mar 2019 at 20:27, Poornima Gurusiddaiah 
wrote:

> This feature is not under active development as it was not used widely.
> AFAIK its not supported feature.
> +Nithya +Raghavendra for further clarifications.
>

This is not actively supported  - there has been no work done on this
feature for a long time.

Regards,
Nithya

>
> Regards,
> Poornima
>
> On Wed, Mar 27, 2019 at 12:33 PM Lucian  wrote:
>
>> Oh, that's just what the doctor ordered!
>> Hope it works, thanks
>>
>> On 27 March 2019 03:15:57 GMT, Vlad Kopylov  wrote:
>>>
>>> I don't remember if it still in works
>>> NUFA
>>>
>>> https://github.com/gluster/glusterfs-specs/blob/master/done/Features/nufa.md
>>>
>>> v
>>>
>>> On Tue, Mar 26, 2019 at 7:27 AM Nux!  wrote:
>>>
 Hello,

 I'm trying to set up a distributed backup storage (no replicas), but
 I'd like to prioritise the local bricks for any IO done on the volume.
 This will be a backup stor, so in other words, I'd like the files to be
 written locally if there is space, so as to save the NICs for other 
 traffic.

 Anyone knows how this might be achievable, if at all?

 --
 Sent from the Delta quadrant using Borg technology!

 Nux!
 www.nux.ro
 ___
 Gluster-users mailing list
 Gluster-users@gluster.org
 https://lists.gluster.org/mailman/listinfo/gluster-users

>>>
>> --
>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Transport endpoint is not connected failures in

[Nithya Balachandran]

On Wed, 27 Mar 2019 at 21:47,  wrote:

> Hello Amar and list,
>
>
>
> I wanted to follow-up to confirm that upgrading to 5.5 seem to fix the
> “Transport endpoint is not connected failures” for us.
>
>
>
> We did not have any of these failures in this past weekend backups cycle.
>
>
>
> Thank you very much for fixing whatever was the problem.
>
>
>
> I also removed some volume config options.  One or more of the settings
> was contributing to the slow directory listing.
>

Hi Brandon,

Which options were removed?

Thanks,
Nithya

>
>
> Here is our current volume info.
>
>
>
> [root@lonbaknode3 ~]# gluster volume info
>
>
>
> Volume Name: volbackups
>
> Type: Distribute
>
> Volume ID: 32bf4fe9-5450-49f8-b6aa-05471d3bdffa
>
> Status: Started
>
> Snapshot Count: 0
>
> Number of Bricks: 8
>
> Transport-type: tcp
>
> Bricks:
>
> Brick1: lonbaknode3.domain.net:/lvbackups/brick
>
> Brick2: lonbaknode4.domain.net:/lvbackups/brick
>
> Brick3: lonbaknode5.domain.net:/lvbackups/brick
>
> Brick4: lonbaknode6.domain.net:/lvbackups/brick
>
> Brick5: lonbaknode7.domain.net:/lvbackups/brick
>
> Brick6: lonbaknode8.domain.net:/lvbackups/brick
>
> Brick7: lonbaknode9.domain.net:/lvbackups/brick
>
> Brick8: lonbaknode10.domain.net:/lvbackups/brick
>
> Options Reconfigured:
>
> performance.io-thread-count: 32
>
> performance.client-io-threads: on
>
> client.event-threads: 8
>
> diagnostics.brick-sys-log-level: WARNING
>
> diagnostics.brick-log-level: WARNING
>
> performance.cache-max-file-size: 2MB
>
> performance.cache-size: 256MB
>
> cluster.min-free-disk: 1%
>
> nfs.disable: on
>
> transport.address-family: inet
>
> server.event-threads: 8
>
> [root@lonbaknode3 ~]#
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] .glusterfs GFID links

[Nithya Balachandran]

On Wed, 20 Mar 2019 at 22:59, Jim Kinney  wrote:

> I have half a zillion broken symlinks in the .glusterfs folder on 3 of 11
> volumes. It doesn't make sense to me that a GFID should linklike some of
> the ones below:
>
> /data/glusterfs/home/brick/brick/.glusterfs/9e/75/9e75a16f-fe4f-411e-937d-1a6c4758fd0e
> -> ../../c7/6f/c76ff719-dde6-41f5-a327-7e13fdf6ec4b/bundle2
> /data/glusterfs/home/brick/brick/.glusterfs/9e/75/9e7594de-c68a-44d0-a959-46ceb628c28b
> -> SSL_CTX_set0_CA_list.html
> /data/glusterfs/home/brick/brick/.glusterfs/9e/75/9e75a40b-990f-4dd5-8aaa-9996da0fbdf4
> -> ../../46/93/4693face-affb-4647-bcd0-919bccc82c42/labeled_tensor
> /data/glusterfs/home/brick/brick/.glusterfs/9e/75/9e75eb33-2941-461f-aa50-394a8e9cbba1
> -> libtiff.so.5.2.6
>
> The links are pointing to file names in the .glusterfs directories.
> Shouldn't all of these be symlinks to other GFID entries and not contain
> text like "bundle2" and "labeled_tensor"?
>
>
> Hi,

These bundle2 and labeled_tensor links are fine - they are gfids of
directories, not files, so the symlinks created will point to the parent
handle+dirname.
The  ones pointing to SSL_CTX_set0_CA_list.html and  libtiff.so.5.2.6 look
odd. They should not be symlinks . File handles should be hardlinks.

Regards,
Nithya


-- 
>
> James P. Kinney III Every time you stop a school, you will have to build a
> jail. What you gain at one end you lose at the other. It's like feeding a
> dog on his own tail. It won't fatten the dog. - Speech 11/23/1900 Mark
> Twain http://heretothereideas.blogspot.com/
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Upgrade 5.3 -> 5.4 on debian: public IP is used instead of LAN IP

[Nithya Balachandran]

Hi Artem,

I think you are running into a different crash. The ones reported which
were prevented by turning off write-behind are now fixed.
We will need to look into the one you are seeing to see why it is happening.

Regards,
Nithya


On Tue, 19 Mar 2019 at 20:25, Artem Russakovskii 
wrote:

> The flood is indeed fixed for us on 5.5. However, the crashes are not.
>
> Sincerely,
> Artem
>
> --
> Founder, Android Police , APK Mirror
> , Illogical Robot LLC
> beerpla.net | +ArtemRussakovskii
>  | @ArtemR
> 
>
>
> On Mon, Mar 18, 2019 at 5:41 AM Hu Bert  wrote:
>
>> Hi Amar,
>>
>> if you refer to this bug:
>> https://bugzilla.redhat.com/show_bug.cgi?id=1674225 : in the test
>> setup i haven't seen those entries, while copying & deleting a few GBs
>> of data. For a final statement we have to wait until i updated our
>> live gluster servers - could take place on tuesday or wednesday.
>>
>> Maybe other users can do an update to 5.4 as well and report back here.
>>
>>
>> Hubert
>>
>>
>>
>> Am Mo., 18. März 2019 um 11:36 Uhr schrieb Amar Tumballi Suryanarayan
>> :
>> >
>> > Hi Hu Bert,
>> >
>> > Appreciate the feedback. Also are the other boiling issues related to
>> logs fixed now?
>> >
>> > -Amar
>> >
>> > On Mon, Mar 18, 2019 at 3:54 PM Hu Bert  wrote:
>> >>
>> >> update: upgrade from 5.3 -> 5.5 in a replicate 3 test setup with 2
>> >> volumes done. In 'gluster peer status' the peers stay connected during
>> >> the upgrade, no 'peer rejected' messages. No cksum mismatches in the
>> >> logs. Looks good :-)
>> >>
>> >> Am Mo., 18. März 2019 um 09:54 Uhr schrieb Hu Bert <
>> revi...@googlemail.com>:
>> >> >
>> >> > Good morning :-)
>> >> >
>> >> > for debian the packages are there:
>> >> >
>> https://download.gluster.org/pub/gluster/glusterfs/5/5.5/Debian/stretch/amd64/apt/pool/main/g/glusterfs/
>> >> >
>> >> > I'll do an upgrade of a test installation 5.3 -> 5.5 and see if there
>> >> > are some errors etc. and report back.
>> >> >
>> >> > btw: no release notes for 5.4 and 5.5 so far?
>> >> > https://docs.gluster.org/en/latest/release-notes/ ?
>> >> >
>> >> > Am Fr., 15. März 2019 um 14:28 Uhr schrieb Shyam Ranganathan
>> >> > :
>> >> > >
>> >> > > We created a 5.5 release tag, and it is under packaging now. It
>> should
>> >> > > be packaged and ready for testing early next week and should be
>> released
>> >> > > close to mid-week next week.
>> >> > >
>> >> > > Thanks,
>> >> > > Shyam
>> >> > > On 3/13/19 12:34 PM, Artem Russakovskii wrote:
>> >> > > > Wednesday now with no update :-/
>> >> > > >
>> >> > > > Sincerely,
>> >> > > > Artem
>> >> > > >
>> >> > > > --
>> >> > > > Founder, Android Police , APK
>> Mirror
>> >> > > > , Illogical Robot LLC
>> >> > > > beerpla.net  | +ArtemRussakovskii
>> >> > > >  | @ArtemR
>> >> > > > 
>> >> > > >
>> >> > > >
>> >> > > > On Tue, Mar 12, 2019 at 10:28 AM Artem Russakovskii <
>> archon...@gmail.com
>> >> > > > > wrote:
>> >> > > >
>> >> > > > Hi Amar,
>> >> > > >
>> >> > > > Any updates on this? I'm still not seeing it in OpenSUSE
>> build
>> >> > > > repos. Maybe later today?
>> >> > > >
>> >> > > > Thanks.
>> >> > > >
>> >> > > > Sincerely,
>> >> > > > Artem
>> >> > > >
>> >> > > > --
>> >> > > > Founder, Android Police , APK
>> Mirror
>> >> > > > , Illogical Robot LLC
>> >> > > > beerpla.net  | +ArtemRussakovskii
>> >> > > >  | @ArtemR
>> >> > > > 
>> >> > > >
>> >> > > >
>> >> > > > On Wed, Mar 6, 2019 at 10:30 PM Amar Tumballi Suryanarayan
>> >> > > > mailto:atumb...@redhat.com>> wrote:
>> >> > > >
>> >> > > > We are talking days. Not weeks. Considering already it is
>> >> > > > Thursday here. 1 more day for tagging, and packaging.
>> May be ok
>> >> > > > to expect it on Monday.
>> >> > > >
>> >> > > > -Amar
>> >> > > >
>> >> > > > On Thu, Mar 7, 2019 at 11:54 AM Artem Russakovskii
>> >> > > > mailto:archon...@gmail.com>>
>> wrote:
>> >> > > >
>> >> > > > Is the next release going to be an imminent hotfix,
>> i.e.
>> >> > > > something like today/tomorrow, or are we talking
>> weeks?
>> >> > > >
>> >> > > > Sincerely,
>> >> > > > Artem
>> >> > > >
>> >> > > > --
>> >> > > > Founder, Android Police <
>> http://www.androidpolice.com>, APK
>> >> > > > Mirror , Illogical Robot
>> LLC
>> >> > > > beerpla.net  |
>> +ArtemRussakovskii
>> >> > > > 

Re: [Gluster-users] / - is in split-brain

[Nithya Balachandran]

Hi,

What is the output of the gluster volume info ?

Thanks,
Nithya

On Wed, 20 Mar 2019 at 01:58, Pablo Schandin  wrote:

> Hello all!
>
> I had a volume with only a local brick running vms and recently added a
> second (remote) brick to the volume. After adding the brick, the heal
> command reported the following:
>
> root@gluster-gu1:~# gluster volume heal gv1 info
>> Brick gluster-gu1:/mnt/gv_gu1/brick
>> / - Is in split-brain
>> Status: Connected
>> Number of entries: 1
>> Brick gluster-gu2:/mnt/gv_gu1/brick
>> Status: Connected
>> Number of entries: 0
>
>
> All other files healed correctly. I noticed that in the xfs of the brick I
> see a directory named localadmin but when I ls the gluster volume
> mountpoint I got an error and a lot of ???
>
> root@gluster-gu1:/var/lib/vmImages_gu1# ll
>> ls: cannot access 'localadmin': No data available
>> d?  ? ??   ?? localadmin/
>
>
> This goes for both servers that have that volume gv1 mounted. Both see
> that directory like that. While in the xfs brick
> /mnt/gv_gu1/brick/localadmin is an accessible directory.
>
> root@gluster-gu1:/mnt/gv_gu1/brick/localadmin# ll
>> total 4
>> drwxr-xr-x 2 localadmin root6 Mar  7 09:40 ./
>> drwxr-xr-x 6 root   root 4096 Mar  7 09:40 ../
>
>
> When I added the second brick to the volume, this localadmin folder was
> not replicated there I imagine because of this strange behavior.
>
> Can someone help me with this?
> Thanks!
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] gluster 5.3: file or directory not read-/writeable, although it exists - cache?

[Nithya Balachandran]

On Tue, 19 Feb 2019 at 15:18, Hu Bert  wrote:

> Hello @ll,
>
> one of our backend developers told me that, in the tomcat logs, he
> sees errors that directories on a glusterfs mount aren't readable.
> Within tomcat the errors look like this:
>
> 2019-02-19 07:39:27,124 WARN  Path
> /data/repository/shared/public/staticmap/370/626 is existed but it is
> not directory
> java.nio.file.FileAlreadyExistsException:
> /data/repository/shared/public/staticmap/370/626
>

Do you know which operation failed here?
regards,
Nithya

>
> But the basic directory does exist, has been created on 2019-02-18
> (and is readable on other clients):
>
> ls -lah /data/repository/shared/public/staticmap/370/626/
> total 36K
> drwxr-xr-x   9 tomcat8 tomcat8 4.0K Feb 18 12:15 .
> drwxr-xr-x 522 tomcat8 tomcat8 4.0K Feb 19 10:29 ..
> drwxr-xr-x   2 tomcat8 tomcat8 4.0K Feb 18 11:45 37062632
> drwxr-xr-x   2 tomcat8 tomcat8 4.0K Feb 19 09:29 37062647
> drwxr-xr-x   2 tomcat8 tomcat8 4.0K Feb 18 12:15 37062663
> drwxr-xr-x   2 tomcat8 tomcat8 4.0K Feb 18 11:18 37062668
> drwxr-xr-x   2 tomcat8 tomcat8 4.0K Feb 18 11:36 37062681
> drwxr-xr-x   2 tomcat8 tomcat8 4.0K Feb 18 16:53 37062682
> drwxr-xr-x   2 tomcat8 tomcat8 4.0K Feb 19 08:19 37062688
>
> gluster v5.3, debian stretch.
> gluster volume info: https://pastebin.com/UBVWSUex
> gluster volume status: https://pastebin.com/3guxFq5m
>
> mount options on client:
> gluster1:/workdata /data/repository/shared/public glusterfs
> defaults,_netdev,lru-limit=0,backup-volfile-servers=gluster2:gluster3
> 0 0
>
> brick mount options:
> /dev/md/4  /gluster/md4  xfs  inode64,noatime,nodiratime  0 0
>
> Hmm... problem with mount options? Or is some cache involved?
>
>
> Best regards,
> Hubert
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Files on Brick not showing up in ls command

[Nithya Balachandran]

Let me know if you still see problems.

Thanks,
Nithya

On Thu, 14 Feb 2019 at 09:05, Patrick Nixon  wrote:

> Thanks for the follow up.  After reviewing the logs Vijay mentioned,
> nothing useful was found.
>
> I wiped removed and wiped the brick tonight.   I'm in the process of
> balancing the new brick and will resync the files onto the full gluster
> volume when that completes
>
> On Wed, Feb 13, 2019, 10:28 PM Nithya Balachandran 
> wrote:
>
>>
>>
>> On Tue, 12 Feb 2019 at 08:30, Patrick Nixon  wrote:
>>
>>> The files are being written to via the glusterfs mount (and read on the
>>> same client and a different client). I try not to do anything on the nodes
>>> directly because I understand that can cause weirdness.   As far as I can
>>> tell, there haven't been any network disconnections, but I'll review the
>>> client log to see if there any indication.   I don't recall any issues last
>>> time I was in there.
>>>
>>>
>> If I understand correctly, the files are written to the volume from the
>> client , but when the same client tries to list them again, those entries
>> are not listed. Is that right?
>>
>> Do the files exist on the bricks?
>> Would you be willing to provide a tcpdump of the client when doing this?
>> If yes, please do the following:
>>
>> On the client system:
>>
>>- tcpdump -i any -s 0 -w /var/tmp/dirls.pcap tcp and not port 22
>>- Copy the files to the volume using the client
>>- List the contents of the directory in which the files should exist
>>- Stop the tcpdump capture and send it to us.
>>
>>
>> Also provide the name of the directory and the missing files.
>>
>> Regards,
>> NIthya
>>
>>
>>
>>
>>
>>> Thanks for the response!
>>>
>>> On Mon, Feb 11, 2019 at 7:35 PM Vijay Bellur  wrote:
>>>
>>>>
>>>>
>>>> On Sun, Feb 10, 2019 at 5:20 PM Patrick Nixon  wrote:
>>>>
>>>>> Hello!
>>>>>
>>>>> I have an 8 node distribute volume setup.   I have one node that
>>>>> accept files and stores them on disk, but when doing an ls, none of the
>>>>> files on that specific node are being returned.
>>>>>
>>>>>  Can someone give some guidance on what should be the best place to
>>>>> start troubleshooting this?
>>>>>
>>>>
>>>>
>>>> Are the files being written from a glusterfs mount? If so, it might be
>>>> worth checking if the network connectivity is fine between the client (that
>>>> does ls) and the server/brick that contains these files. You could look up
>>>> the client log file to check if there are any messages related to
>>>> rpc disconnections.
>>>>
>>>> Regards,
>>>> Vijay
>>>>
>>>>
>>>>> # gluster volume info
>>>>>
>>>>> Volume Name: gfs
>>>>> Type: Distribute
>>>>> Volume ID: 44c8c4f1-2dfb-4c03-9bca-d1ae4f314a78
>>>>> Status: Started
>>>>> Snapshot Count: 0
>>>>> Number of Bricks: 8
>>>>> Transport-type: tcp
>>>>> Bricks:
>>>>> Brick1: gfs01:/data/brick1/gv0
>>>>> Brick2: gfs02:/data/brick1/gv0
>>>>> Brick3: gfs03:/data/brick1/gv0
>>>>> Brick4: gfs05:/data/brick1/gv0
>>>>> Brick5: gfs06:/data/brick1/gv0
>>>>> Brick6: gfs07:/data/brick1/gv0
>>>>> Brick7: gfs08:/data/brick1/gv0
>>>>> Brick8: gfs04:/data/brick1/gv0
>>>>> Options Reconfigured:
>>>>> cluster.min-free-disk: 10%
>>>>> nfs.disable: on
>>>>> performance.readdir-ahead: on
>>>>>
>>>>> # gluster peer status
>>>>> Number of Peers: 7
>>>>> Hostname: gfs03
>>>>> Uuid: 4a2d4deb-f8dd-49fc-a2ab-74e39dc25e20
>>>>> State: Peer in Cluster (Connected)
>>>>> Hostname: gfs08
>>>>> Uuid: 17705b3a-ed6f-4123-8e2e-4dc5ab6d807d
>>>>> State: Peer in Cluster (Connected)
>>>>> Hostname: gfs07
>>>>> Uuid: dd699f55-1a27-4e51-b864-b4600d630732
>>>>> State: Peer in Cluster (Connected)
>>>>> Hostname: gfs06
>>>>> Uuid: 8eb2a965-2c1e-4a64-b5b5-b7b7136ddede
>>>>> State: Peer in Cluster (Connected)
>>>>> Hostname: gfs04
>>>>> Uuid: cd866191-f767-40d0-bf7b-81ca0bc032b7
>>>>> State: Peer in Cluster (Connected)
>>>>> Hostname: gfs02
>>>>> Uuid: 6864c6ac-6ff4-423a-ae3c-f5fd25621851
>>>>> State: Peer in Cluster (Connected)
>>>>> Hostname: gfs05
>>>>> Uuid: dcecb55a-87b8-4441-ab09-b52e485e5f62
>>>>> State: Peer in Cluster (Connected)
>>>>>
>>>>> All gluster nodes are running glusterfs 4.0.2
>>>>> The clients accessing the files are also running glusterfs 4.0.2
>>>>> Both are Ubuntu
>>>>>
>>>>> Thanks!
>>>>> ___
>>>>> Gluster-users mailing list
>>>>> Gluster-users@gluster.org
>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>>
>>>> ___
>>> Gluster-users mailing list
>>> Gluster-users@gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Files on Brick not showing up in ls command

[Nithya Balachandran]

On Tue, 12 Feb 2019 at 08:30, Patrick Nixon  wrote:

> The files are being written to via the glusterfs mount (and read on the
> same client and a different client). I try not to do anything on the nodes
> directly because I understand that can cause weirdness.   As far as I can
> tell, there haven't been any network disconnections, but I'll review the
> client log to see if there any indication.   I don't recall any issues last
> time I was in there.
>
>
If I understand correctly, the files are written to the volume from the
client , but when the same client tries to list them again, those entries
are not listed. Is that right?

Do the files exist on the bricks?
Would you be willing to provide a tcpdump of the client when doing this? If
yes, please do the following:

On the client system:

   - tcpdump -i any -s 0 -w /var/tmp/dirls.pcap tcp and not port 22
   - Copy the files to the volume using the client
   - List the contents of the directory in which the files should exist
   - Stop the tcpdump capture and send it to us.


Also provide the name of the directory and the missing files.

Regards,
NIthya





> Thanks for the response!
>
> On Mon, Feb 11, 2019 at 7:35 PM Vijay Bellur  wrote:
>
>>
>>
>> On Sun, Feb 10, 2019 at 5:20 PM Patrick Nixon  wrote:
>>
>>> Hello!
>>>
>>> I have an 8 node distribute volume setup.   I have one node that accept
>>> files and stores them on disk, but when doing an ls, none of the files on
>>> that specific node are being returned.
>>>
>>>  Can someone give some guidance on what should be the best place to
>>> start troubleshooting this?
>>>
>>
>>
>> Are the files being written from a glusterfs mount? If so, it might be
>> worth checking if the network connectivity is fine between the client (that
>> does ls) and the server/brick that contains these files. You could look up
>> the client log file to check if there are any messages related to
>> rpc disconnections.
>>
>> Regards,
>> Vijay
>>
>>
>>> # gluster volume info
>>>
>>> Volume Name: gfs
>>> Type: Distribute
>>> Volume ID: 44c8c4f1-2dfb-4c03-9bca-d1ae4f314a78
>>> Status: Started
>>> Snapshot Count: 0
>>> Number of Bricks: 8
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: gfs01:/data/brick1/gv0
>>> Brick2: gfs02:/data/brick1/gv0
>>> Brick3: gfs03:/data/brick1/gv0
>>> Brick4: gfs05:/data/brick1/gv0
>>> Brick5: gfs06:/data/brick1/gv0
>>> Brick6: gfs07:/data/brick1/gv0
>>> Brick7: gfs08:/data/brick1/gv0
>>> Brick8: gfs04:/data/brick1/gv0
>>> Options Reconfigured:
>>> cluster.min-free-disk: 10%
>>> nfs.disable: on
>>> performance.readdir-ahead: on
>>>
>>> # gluster peer status
>>> Number of Peers: 7
>>> Hostname: gfs03
>>> Uuid: 4a2d4deb-f8dd-49fc-a2ab-74e39dc25e20
>>> State: Peer in Cluster (Connected)
>>> Hostname: gfs08
>>> Uuid: 17705b3a-ed6f-4123-8e2e-4dc5ab6d807d
>>> State: Peer in Cluster (Connected)
>>> Hostname: gfs07
>>> Uuid: dd699f55-1a27-4e51-b864-b4600d630732
>>> State: Peer in Cluster (Connected)
>>> Hostname: gfs06
>>> Uuid: 8eb2a965-2c1e-4a64-b5b5-b7b7136ddede
>>> State: Peer in Cluster (Connected)
>>> Hostname: gfs04
>>> Uuid: cd866191-f767-40d0-bf7b-81ca0bc032b7
>>> State: Peer in Cluster (Connected)
>>> Hostname: gfs02
>>> Uuid: 6864c6ac-6ff4-423a-ae3c-f5fd25621851
>>> State: Peer in Cluster (Connected)
>>> Hostname: gfs05
>>> Uuid: dcecb55a-87b8-4441-ab09-b52e485e5f62
>>> State: Peer in Cluster (Connected)
>>>
>>> All gluster nodes are running glusterfs 4.0.2
>>> The clients accessing the files are also running glusterfs 4.0.2
>>> Both are Ubuntu
>>>
>>> Thanks!
>>> ___
>>> Gluster-users mailing list
>>> Gluster-users@gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Message repeated over and over after upgrade from 4.1 to 5.3: W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fd966fcd329] -->/us

[Nithya Balachandran]

Not yet but we are discussing an interim release. It is going to take a
couple of days to review the fixes so not before then. We will update on
the list with dates once we decide.


On Tue, 12 Feb 2019 at 11:46, Artem Russakovskii 
wrote:

> Awesome. But is there a release schedule and an ETA for when these will be
> out in the repos?
>
> On Mon, Feb 11, 2019, 9:34 PM Raghavendra Gowdappa 
> wrote:
>
>>
>>
>> On Tue, Feb 12, 2019 at 10:24 AM Artem Russakovskii 
>> wrote:
>>
>>> Great job identifying the issue!
>>>
>>> Any ETA on the next release with the logging and crash fixes in it?
>>>
>>
>> I've marked write-behind corruption as a blocker for release-6. Logging
>> fixes are already in codebase.
>>
>>
>>> On Mon, Feb 11, 2019, 7:19 PM Raghavendra Gowdappa 
>>> wrote:
>>>
>>>>
>>>>
>>>> On Mon, Feb 11, 2019 at 3:49 PM João Baúto <
>>>> joao.ba...@neuro.fchampalimaud.org> wrote:
>>>>
>>>>> Although I don't have these error messages, I'm having fuse crashes as
>>>>> frequent as you. I have disabled write-behind and the mount has been
>>>>> running over the weekend with heavy usage and no issues.
>>>>>
>>>>
>>>> The issue you are facing will likely be fixed by patch [1]. Me, Xavi
>>>> and Nithya were able to identify the corruption in write-behind.
>>>>
>>>> [1] https://review.gluster.org/22189
>>>>
>>>>
>>>>> I can provide coredumps before disabling write-behind if needed. I
>>>>> opened a BZ report
>>>>> <https://bugzilla.redhat.com/show_bug.cgi?id=1671014> with the
>>>>> crashes that I was having.
>>>>>
>>>>> *João Baúto*
>>>>> ---
>>>>>
>>>>> *Scientific Computing and Software Platform*
>>>>> Champalimaud Research
>>>>> Champalimaud Center for the Unknown
>>>>> Av. Brasília, Doca de Pedrouços
>>>>> 1400-038 Lisbon, Portugal
>>>>> fchampalimaud.org <https://www.fchampalimaud.org/>
>>>>>
>>>>>
>>>>> Artem Russakovskii  escreveu no dia sábado,
>>>>> 9/02/2019 à(s) 22:18:
>>>>>
>>>>>> Alright. I've enabled core-dumping (hopefully), so now I'm waiting
>>>>>> for the next crash to see if it dumps a core for you guys to remotely 
>>>>>> debug.
>>>>>>
>>>>>> Then I can consider setting performance.write-behind to off and
>>>>>> monitoring for further crashes.
>>>>>>
>>>>>> Sincerely,
>>>>>> Artem
>>>>>>
>>>>>> --
>>>>>> Founder, Android Police <http://www.androidpolice.com>, APK Mirror
>>>>>> <http://www.apkmirror.com/>, Illogical Robot LLC
>>>>>> beerpla.net | +ArtemRussakovskii
>>>>>> <https://plus.google.com/+ArtemRussakovskii> | @ArtemR
>>>>>> <http://twitter.com/ArtemR>
>>>>>>
>>>>>>
>>>>>> On Fri, Feb 8, 2019 at 7:22 PM Raghavendra Gowdappa <
>>>>>> rgowd...@redhat.com> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Sat, Feb 9, 2019 at 12:53 AM Artem Russakovskii <
>>>>>>> archon...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi Nithya,
>>>>>>>>
>>>>>>>> I can try to disable write-behind as long as it doesn't heavily
>>>>>>>> impact performance for us. Which option is it exactly? I don't see it 
>>>>>>>> set
>>>>>>>> in my list of changed volume variables that I sent you guys earlier.
>>>>>>>>
>>>>>>>
>>>>>>> The option is performance.write-behind
>>>>>>>
>>>>>>>
>>>>>>>> Sincerely,
>>>>>>>> Artem
>>>>>>>>
>>>>>>>> --
>>>>>>>> Founder, Android Police <http://www.androidpolice.com>, APK Mirror
>>>>>>>> <http://www.apkmirror.com/>, Illogical Robot LLC
>>>>>>>> beerpla.net | +ArtemRussakovskii
>>>>>>>> <https

Re: [Gluster-users] Message repeated over and over after upgrade from 4.1 to 5.3: W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fd966fcd329] -->/us

[Nithya Balachandran]

ter.com/ArtemR>
>
>
> On Thu, Feb 7, 2019 at 1:28 PM Artem Russakovskii 
> wrote:
>
>> I've added the lru-limit=0 parameter to the mounts, and I see it's taken
>> effect correctly:
>> "/usr/sbin/glusterfs --lru-limit=0 --process-name fuse
>> --volfile-server=localhost --volfile-id=/  /mnt/"
>>
>> Let's see if it stops crashing or not.
>>
>> Sincerely,
>> Artem
>>
>> --
>> Founder, Android Police <http://www.androidpolice.com>, APK Mirror
>> <http://www.apkmirror.com/>, Illogical Robot LLC
>> beerpla.net | +ArtemRussakovskii
>> <https://plus.google.com/+ArtemRussakovskii> | @ArtemR
>> <http://twitter.com/ArtemR>
>>
>>
>> On Wed, Feb 6, 2019 at 10:48 AM Artem Russakovskii 
>> wrote:
>>
>>> Hi Nithya,
>>>
>>> Indeed, I upgraded from 4.1 to 5.3, at which point I started seeing
>>> crashes, and no further releases have been made yet.
>>>
>>> volume info:
>>> Type: Replicate
>>> Volume ID: SNIP
>>> Status: Started
>>> Snapshot Count: 0
>>> Number of Bricks: 1 x 4 = 4
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: SNIP
>>> Brick2: SNIP
>>> Brick3: SNIP
>>> Brick4: SNIP
>>> Options Reconfigured:
>>> cluster.quorum-count: 1
>>> cluster.quorum-type: fixed
>>> network.ping-timeout: 5
>>> network.remote-dio: enable
>>> performance.rda-cache-limit: 256MB
>>> performance.readdir-ahead: on
>>> performance.parallel-readdir: on
>>> network.inode-lru-limit: 50
>>> performance.md-cache-timeout: 600
>>> performance.cache-invalidation: on
>>> performance.stat-prefetch: on
>>> features.cache-invalidation-timeout: 600
>>> features.cache-invalidation: on
>>> cluster.readdir-optimize: on
>>> performance.io-thread-count: 32
>>> server.event-threads: 4
>>> client.event-threads: 4
>>> performance.read-ahead: off
>>> cluster.lookup-optimize: on
>>> performance.cache-size: 1GB
>>> cluster.self-heal-daemon: enable
>>> transport.address-family: inet
>>> nfs.disable: on
>>> performance.client-io-threads: on
>>> cluster.granular-entry-heal: enable
>>> cluster.data-self-heal-algorithm: full
>>>
>>> Sincerely,
>>> Artem
>>>
>>> --
>>> Founder, Android Police <http://www.androidpolice.com>, APK Mirror
>>> <http://www.apkmirror.com/>, Illogical Robot LLC
>>> beerpla.net | +ArtemRussakovskii
>>> <https://plus.google.com/+ArtemRussakovskii> | @ArtemR
>>> <http://twitter.com/ArtemR>
>>>
>>>
>>> On Wed, Feb 6, 2019 at 12:20 AM Nithya Balachandran 
>>> wrote:
>>>
>>>> Hi Artem,
>>>>
>>>> Do you still see the crashes with 5.3? If yes, please try mount the
>>>> volume using the mount option lru-limit=0 and see if that helps. We are
>>>> looking into the crashes and will update when have a fix.
>>>>
>>>> Also, please provide the gluster volume info for the volume in question.
>>>>
>>>>
>>>> regards,
>>>> Nithya
>>>>
>>>> On Tue, 5 Feb 2019 at 05:31, Artem Russakovskii 
>>>> wrote:
>>>>
>>>>> The fuse crash happened two more times, but this time monit helped
>>>>> recover within 1 minute, so it's a great workaround for now.
>>>>>
>>>>> What's odd is that the crashes are only happening on one of 4 servers,
>>>>> and I don't know why.
>>>>>
>>>>> Sincerely,
>>>>> Artem
>>>>>
>>>>> --
>>>>> Founder, Android Police <http://www.androidpolice.com>, APK Mirror
>>>>> <http://www.apkmirror.com/>, Illogical Robot LLC
>>>>> beerpla.net | +ArtemRussakovskii
>>>>> <https://plus.google.com/+ArtemRussakovskii> | @ArtemR
>>>>> <http://twitter.com/ArtemR>
>>>>>
>>>>>
>>>>> On Sat, Feb 2, 2019 at 12:14 PM Artem Russakovskii <
>>>>> archon...@gmail.com> wrote:
>>>>>
>>>>>> The fuse crash happened again yesterday, to another volume. Are there
>>>>>> any mount options that could help mitigate this?
>>>>>>
>>>>>> In the meantime, I set up a

Re: [Gluster-users] Message repeated over and over after upgrade from 4.1 to 5.3: W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fd966fcd329] -->/us

[Nithya Balachandran]

fect correctly:
>> "/usr/sbin/glusterfs --lru-limit=0 --process-name fuse
>> --volfile-server=localhost --volfile-id=/  /mnt/"
>>
>> Let's see if it stops crashing or not.
>>
>> Sincerely,
>> Artem
>>
>> --
>> Founder, Android Police <http://www.androidpolice.com>, APK Mirror
>> <http://www.apkmirror.com/>, Illogical Robot LLC
>> beerpla.net | +ArtemRussakovskii
>> <https://plus.google.com/+ArtemRussakovskii> | @ArtemR
>> <http://twitter.com/ArtemR>
>>
>>
>> On Wed, Feb 6, 2019 at 10:48 AM Artem Russakovskii 
>> wrote:
>>
>>> Hi Nithya,
>>>
>>> Indeed, I upgraded from 4.1 to 5.3, at which point I started seeing
>>> crashes, and no further releases have been made yet.
>>>
>>> volume info:
>>> Type: Replicate
>>> Volume ID: SNIP
>>> Status: Started
>>> Snapshot Count: 0
>>> Number of Bricks: 1 x 4 = 4
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: SNIP
>>> Brick2: SNIP
>>> Brick3: SNIP
>>> Brick4: SNIP
>>> Options Reconfigured:
>>> cluster.quorum-count: 1
>>> cluster.quorum-type: fixed
>>> network.ping-timeout: 5
>>> network.remote-dio: enable
>>> performance.rda-cache-limit: 256MB
>>> performance.readdir-ahead: on
>>> performance.parallel-readdir: on
>>> network.inode-lru-limit: 50
>>> performance.md-cache-timeout: 600
>>> performance.cache-invalidation: on
>>> performance.stat-prefetch: on
>>> features.cache-invalidation-timeout: 600
>>> features.cache-invalidation: on
>>> cluster.readdir-optimize: on
>>> performance.io-thread-count: 32
>>> server.event-threads: 4
>>> client.event-threads: 4
>>> performance.read-ahead: off
>>> cluster.lookup-optimize: on
>>> performance.cache-size: 1GB
>>> cluster.self-heal-daemon: enable
>>> transport.address-family: inet
>>> nfs.disable: on
>>> performance.client-io-threads: on
>>> cluster.granular-entry-heal: enable
>>> cluster.data-self-heal-algorithm: full
>>>
>>> Sincerely,
>>> Artem
>>>
>>> --
>>> Founder, Android Police <http://www.androidpolice.com>, APK Mirror
>>> <http://www.apkmirror.com/>, Illogical Robot LLC
>>> beerpla.net | +ArtemRussakovskii
>>> <https://plus.google.com/+ArtemRussakovskii> | @ArtemR
>>> <http://twitter.com/ArtemR>
>>>
>>>
>>> On Wed, Feb 6, 2019 at 12:20 AM Nithya Balachandran 
>>> wrote:
>>>
>>>> Hi Artem,
>>>>
>>>> Do you still see the crashes with 5.3? If yes, please try mount the
>>>> volume using the mount option lru-limit=0 and see if that helps. We are
>>>> looking into the crashes and will update when have a fix.
>>>>
>>>> Also, please provide the gluster volume info for the volume in question.
>>>>
>>>>
>>>> regards,
>>>> Nithya
>>>>
>>>> On Tue, 5 Feb 2019 at 05:31, Artem Russakovskii 
>>>> wrote:
>>>>
>>>>> The fuse crash happened two more times, but this time monit helped
>>>>> recover within 1 minute, so it's a great workaround for now.
>>>>>
>>>>> What's odd is that the crashes are only happening on one of 4 servers,
>>>>> and I don't know why.
>>>>>
>>>>> Sincerely,
>>>>> Artem
>>>>>
>>>>> --
>>>>> Founder, Android Police <http://www.androidpolice.com>, APK Mirror
>>>>> <http://www.apkmirror.com/>, Illogical Robot LLC
>>>>> beerpla.net | +ArtemRussakovskii
>>>>> <https://plus.google.com/+ArtemRussakovskii> | @ArtemR
>>>>> <http://twitter.com/ArtemR>
>>>>>
>>>>>
>>>>> On Sat, Feb 2, 2019 at 12:14 PM Artem Russakovskii <
>>>>> archon...@gmail.com> wrote:
>>>>>
>>>>>> The fuse crash happened again yesterday, to another volume. Are there
>>>>>> any mount options that could help mitigate this?
>>>>>>
>>>>>> In the meantime, I set up a monit (https://mmonit.com/monit/) task
>>>>>> to watch and restart the mount, which works and recovers the mount point
>>>>>> within a minute. Not ideal, but a temporary wo

Re: [Gluster-users] gluster 5.3: transport endpoint gets disconnected - Assertion failed: GF_MEM_TRAILER_MAGIC

[Nithya Balachandran]

On Wed, 6 Feb 2019 at 14:34, Hu Bert  wrote:

> Hi there,
>
> just curious - from man mount.glusterfs:
>
>lru-limit=N
>  Set fuse module's limit for number of inodes kept in LRU
> list to N [default: 0]
>

Sorry, that is a bug in the man page and we will fix that. The current
default is 131072:
{

.key = {"lru-limit"},

.type = GF_OPTION_TYPE_INT,

.default_value = "131072",

.min = 0,

.description = "makes glusterfs invalidate kernel inodes after "

   "reaching this limit (0 means 'unlimited')",

},




>
> This seems to be the default already? Set it explicitly?
>
> Regards,
> Hubert
>
> Am Mi., 6. Feb. 2019 um 09:26 Uhr schrieb Nithya Balachandran
> :
> >
> > Hi,
> >
> > The client logs indicates that the mount process has crashed.
> > Please try mounting the volume with the volume option lru-limit=0 and
> see if it still crashes.
> >
> > Thanks,
> > Nithya
> >
> > On Thu, 24 Jan 2019 at 12:47, Hu Bert  wrote:
> >>
> >> Good morning,
> >>
> >> we currently transfer some data to a new glusterfs volume; to check
> >> the throughput of the new volume/setup while the transfer is running i
> >> decided to create some files on one of the gluster servers with dd in
> >> loop:
> >>
> >> while true; do dd if=/dev/urandom of=/shared/private/1G.file bs=1M
> >> count=1024; rm /shared/private/1G.file; done
> >>
> >> /shared/private is the mount point of the glusterfs volume. The dd
> >> should run for about an hour. But now it happened twice that during
> >> this loop the transport endpoint gets disconnected:
> >>
> >> dd: failed to open '/shared/private/1G.file': Transport endpoint is
> >> not connected
> >> rm: cannot remove '/shared/private/1G.file': Transport endpoint is not
> connected
> >>
> >> In the /var/log/glusterfs/shared-private.log i see:
> >>
> >> [2019-01-24 07:03:28.938745] W [MSGID: 108001]
> >> [afr-transaction.c:1062:afr_handle_quorum] 0-persistent-replicate-0:
> >> 7212652e-c437-426c-a0a9-a47f5972fffe: Failing WRITE as quorum i
> >> s not met [Transport endpoint is not connected]
> >> [2019-01-24 07:03:28.939280] E [mem-pool.c:331:__gf_free]
> >>
> (-->/usr/lib/x86_64-linux-gnu/glusterfs/5.3/xlator/cluster/replicate.so(+0x5be8c)
> >> [0x7eff84248e8c] -->/usr/lib/x86_64-lin
> >> ux-gnu/glusterfs/5.3/xlator/cluster/replicate.so(+0x5be18)
> >> [0x7eff84248e18]
> >> -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(__gf_free+0xf6)
> >> [0x7eff8a9485a6] ) 0-: Assertion failed:
> >> GF_MEM_TRAILER_MAGIC == *(uint32_t *)((char *)free_ptr + header->size)
> >> [snip]
> >>
> >> The whole output can be found here: https://pastebin.com/qTMmFxx0
> >> gluster volume info here: https://pastebin.com/ENTWZ7j3
> >>
> >> After umount + mount the transport endpoint is connected again - until
> >> the next disconnect. A /core file gets generated. Maybe someone wants
> >> to have a look at this file?
> >> ___
> >> Gluster-users mailing list
> >> Gluster-users@gluster.org
> >> https://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] gluster 5.3: transport endpoint gets disconnected - Assertion failed: GF_MEM_TRAILER_MAGIC

[Nithya Balachandran]

Hi,

The client logs indicates that the mount process has crashed.
Please try mounting the volume with the volume option lru-limit=0 and see
if it still crashes.

Thanks,
Nithya

On Thu, 24 Jan 2019 at 12:47, Hu Bert  wrote:

> Good morning,
>
> we currently transfer some data to a new glusterfs volume; to check
> the throughput of the new volume/setup while the transfer is running i
> decided to create some files on one of the gluster servers with dd in
> loop:
>
> while true; do dd if=/dev/urandom of=/shared/private/1G.file bs=1M
> count=1024; rm /shared/private/1G.file; done
>
> /shared/private is the mount point of the glusterfs volume. The dd
> should run for about an hour. But now it happened twice that during
> this loop the transport endpoint gets disconnected:
>
> dd: failed to open '/shared/private/1G.file': Transport endpoint is
> not connected
> rm: cannot remove '/shared/private/1G.file': Transport endpoint is not
> connected
>
> In the /var/log/glusterfs/shared-private.log i see:
>
> [2019-01-24 07:03:28.938745] W [MSGID: 108001]
> [afr-transaction.c:1062:afr_handle_quorum] 0-persistent-replicate-0:
> 7212652e-c437-426c-a0a9-a47f5972fffe: Failing WRITE as quorum i
> s not met [Transport endpoint is not connected]
> [2019-01-24 07:03:28.939280] E [mem-pool.c:331:__gf_free]
>
> (-->/usr/lib/x86_64-linux-gnu/glusterfs/5.3/xlator/cluster/replicate.so(+0x5be8c)
> [0x7eff84248e8c] -->/usr/lib/x86_64-lin
> ux-gnu/glusterfs/5.3/xlator/cluster/replicate.so(+0x5be18)
> [0x7eff84248e18]
> -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(__gf_free+0xf6)
> [0x7eff8a9485a6] ) 0-: Assertion failed:
> GF_MEM_TRAILER_MAGIC == *(uint32_t *)((char *)free_ptr + header->size)
> [snip]
>
> The whole output can be found here: https://pastebin.com/qTMmFxx0
> gluster volume info here: https://pastebin.com/ENTWZ7j3
>
> After umount + mount the transport endpoint is connected again - until
> the next disconnect. A /core file gets generated. Maybe someone wants
> to have a look at this file?
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Message repeated over and over after upgrade from 4.1 to 5.3: W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fd966fcd329] -->/us

[Nithya Balachandran]

Hi Artem,

Do you still see the crashes with 5.3? If yes, please try mount the volume
using the mount option lru-limit=0 and see if that helps. We are looking
into the crashes and will update when have a fix.

Also, please provide the gluster volume info for the volume in question.


regards,
Nithya

On Tue, 5 Feb 2019 at 05:31, Artem Russakovskii  wrote:

> The fuse crash happened two more times, but this time monit helped recover
> within 1 minute, so it's a great workaround for now.
>
> What's odd is that the crashes are only happening on one of 4 servers, and
> I don't know why.
>
> Sincerely,
> Artem
>
> --
> Founder, Android Police , APK Mirror
> , Illogical Robot LLC
> beerpla.net | +ArtemRussakovskii
>  | @ArtemR
> 
>
>
> On Sat, Feb 2, 2019 at 12:14 PM Artem Russakovskii 
> wrote:
>
>> The fuse crash happened again yesterday, to another volume. Are there any
>> mount options that could help mitigate this?
>>
>> In the meantime, I set up a monit (https://mmonit.com/monit/) task to
>> watch and restart the mount, which works and recovers the mount point
>> within a minute. Not ideal, but a temporary workaround.
>>
>> By the way, the way to reproduce this "Transport endpoint is not
>> connected" condition for testing purposes is to kill -9 the right
>> "glusterfs --process-name fuse" process.
>>
>>
>> monit check:
>> check filesystem glusterfs_data1 with path /mnt/glusterfs_data1
>>   start program  = "/bin/mount  /mnt/glusterfs_data1"
>>   stop program  = "/bin/umount /mnt/glusterfs_data1"
>>   if space usage > 90% for 5 times within 15 cycles
>> then alert else if succeeded for 10 cycles then alert
>>
>>
>> stack trace:
>> [2019-02-01 23:22:00.312894] W [dict.c:761:dict_ref]
>> (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329)
>> [0x7fa0249e4329]
>> -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5)
>> [0x7fa024bf5af5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58)
>> [0x7fa02cf5b218] ) 0-dict: dict is NULL [Invalid argument]
>> [2019-02-01 23:22:00.314051] W [dict.c:761:dict_ref]
>> (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329)
>> [0x7fa0249e4329]
>> -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5)
>> [0x7fa024bf5af5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58)
>> [0x7fa02cf5b218] ) 0-dict: dict is NULL [Invalid argument]
>> The message "E [MSGID: 101191]
>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
>> handler" repeated 26 times between [2019-02-01 23:21:20.857333] and
>> [2019-02-01 23:21:56.164427]
>> The message "I [MSGID: 108031]
>> [afr-common.c:2543:afr_local_discovery_cbk] 0-SITE_data3-replicate-0:
>> selecting local read_child SITE_data3-client-3" repeated 27 times between
>> [2019-02-01 23:21:11.142467] and [2019-02-01 23:22:03.474036]
>> pending frames:
>> frame : type(1) op(LOOKUP)
>> frame : type(0) op(0)
>> patchset: git://git.gluster.org/glusterfs.git
>> signal received: 6
>> time of crash:
>> 2019-02-01 23:22:03
>> configuration details:
>> argp 1
>> backtrace 1
>> dlfcn 1
>> libpthread 1
>> llistxattr 1
>> setfsid 1
>> spinlock 1
>> epoll.h 1
>> xattr.h 1
>> st_atim.tv_nsec 1
>> package-string: glusterfs 5.3
>> /usr/lib64/libglusterfs.so.0(+0x2764c)[0x7fa02cf6664c]
>> /usr/lib64/libglusterfs.so.0(gf_print_trace+0x306)[0x7fa02cf70cb6]
>> /lib64/libc.so.6(+0x36160)[0x7fa02c12d160]
>> /lib64/libc.so.6(gsignal+0x110)[0x7fa02c12d0e0]
>> /lib64/libc.so.6(abort+0x151)[0x7fa02c12e6c1]
>> /lib64/libc.so.6(+0x2e6fa)[0x7fa02c1256fa]
>> /lib64/libc.so.6(+0x2e772)[0x7fa02c125772]
>> /lib64/libpthread.so.0(pthread_mutex_lock+0x228)[0x7fa02c4bb0b8]
>>
>> /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x5dc9d)[0x7fa025543c9d]
>>
>> /usr/lib64/glusterfs/5.3/xlator/cluster/replicate.so(+0x70ba1)[0x7fa025556ba1]
>>
>> /usr/lib64/glusterfs/5.3/xlator/protocol/client.so(+0x58f3f)[0x7fa0257dbf3f]
>> /usr/lib64/libgfrpc.so.0(+0xe820)[0x7fa02cd31820]
>> /usr/lib64/libgfrpc.so.0(+0xeb6f)[0x7fa02cd31b6f]
>> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7fa02cd2e063]
>> /usr/lib64/glusterfs/5.3/rpc-transport/socket.so(+0xa0b2)[0x7fa02694e0b2]
>> /usr/lib64/libglusterfs.so.0(+0x854c3)[0x7fa02cfc44c3]
>> /lib64/libpthread.so.0(+0x7559)[0x7fa02c4b8559]
>> /lib64/libc.so.6(clone+0x3f)[0x7fa02c1ef81f]
>>
>> Sincerely,
>> Artem
>>
>> --
>> Founder, Android Police , APK Mirror
>> , Illogical Robot LLC
>> beerpla.net | +ArtemRussakovskii
>>  | @ArtemR
>> 
>>
>>
>> On Fri, Feb 1, 2019 at 9:03 AM Artem Russakovskii 
>> wrote:
>>
>>> Hi,
>>>
>>> The first (and so far only) crash happened at 2am the next day after we
>>> upgraded, on only one of four servers and only to one of two mounts.
>>>
>>> I have no idea what caused it, but 

Re: [Gluster-users] Getting timedout error while rebalancing

[Nithya Balachandran]

On Tue, 5 Feb 2019 at 17:26, deepu srinivasan  wrote:

> HI Nithya
> We have a test gluster setup.We are testing the rebalancing option of
> gluster. So we started the volume which have 1x3 brick with some data on it
> .
> command : gluster volume create test-volume replica 3
> 192.168.xxx.xx1:/home/data/repl 192.168.xxx.xx2:/home/data/repl
> 192.168.xxx.xx3:/home/data/repl.
>
> Now we tried to expand the cluster storage by adding three more bricks.
> command : gluster volume add-brick test-volume 192.168.xxx.xx4:/home/data/repl
> 192.168.xxx.xx5:/home/data/repl 192.168.xxx.xx6:/home/data/repl
>
> So after the brick addition we tried to rebalance the layout and the data.
> command : gluster volume rebalance test-volume fix-layout start.
> The command exited with status "Error : Request timed out".
>

This sounds like an error in the cli or glusterd. Can you send the
glusterd.log from the node on which you ran the command?

regards,
Nithya

>
> After the failure of the command, we tried to view the status of the
> command and it is something like this :
>
> Node Rebalanced-files  size
> scanned  failures   skipped   status  run time in
> h:m:s
>
>-  ---   ---   
> ---
>   ---   ---  --
>
>localhost   4141.0MB
> 8200 0 0completed
> 0:00:09
>
>  192.168.xxx.xx4   7979.0MB
> 8231 0 0completed
> 0:00:12
>
>  192.168.xxx.xx6   5858.0MB
> 8281 0 0completed
> 0:00:10
>
>  192.168.xxx.xx2  136   136.0MB
> 8566 0   136completed
> 0:00:07
>
>  192.168.xxx.xx4  129   129.0MB
> 8566 0   129completed
> 0:00:07
>
>  192.168.xxx.xx6  201   201.0MB
> 8566 0   201completed
> 0:00:08
>
> Is the rebalancing option working fine? Why did gluster  throw the error
> saying that "Error : Request timed out"?
> .On Tue, Feb 5, 2019 at 4:23 PM Nithya Balachandran 
> wrote:
>
>> Hi,
>> Please provide the exact step at which you are seeing the error. It would
>> be ideal if you could copy-paste the command and the error.
>>
>> Regards,
>> Nithya
>>
>>
>>
>> On Tue, 5 Feb 2019 at 15:24, deepu srinivasan  wrote:
>>
>>> HI everyone. I am getting "Error : Request timed out " while doing
>>> rebalance . I have aded new bricks to my replicated volume.i.e. First it
>>> was 1x3 volume and added three more bricks to make it
>>> distributed-replicated volume(2x3) . What should i do for the timeout error
>>> ?
>>> ___
>>> Gluster-users mailing list
>>> Gluster-users@gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Getting timedout error while rebalancing

[Nithya Balachandran]

Hi,
Please provide the exact step at which you are seeing the error. It would
be ideal if you could copy-paste the command and the error.

Regards,
Nithya



On Tue, 5 Feb 2019 at 15:24, deepu srinivasan  wrote:

> HI everyone. I am getting "Error : Request timed out " while doing
> rebalance . I have aded new bricks to my replicated volume.i.e. First it
> was 1x3 volume and added three more bricks to make it
> distributed-replicated volume(2x3) . What should i do for the timeout error
> ?
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] gluster remove-brick

[Nithya Balachandran]

Hi,


On Mon, 4 Feb 2019 at 16:39, mohammad kashif  wrote:

> Hi Nithya
>
> Thanks for replying so quickly. It is very much appreciated.
>
> There are lots if  " [No space left on device] " errors which I can not
> understand as there are much space on all of the nodes.
>

This means that Gluster could not find sufficient space for the file. Would
you be willing to share your rebalance log file?
Please provide the following information:

   - The gluster version
   - The gluster volume info for the volume
   - How full are the individual bricks for the volume?



> A little bit of background will be useful in this case. I had cluster of
> seven nodes of varying capacity(73, 73, 73, 46, 46, 46,46 TB) .  The
> cluster was almost 90% full so every node has almost 8 to 15 TB free
> space.  I added two new nodes with 100TB each and ran fix-layout which
> completed successfully.
>
> After that I started remove-brick operation.  I don't think that any point
> , any of the nodes were 100% full. Looking at my ganglia graph, there is
> minimum 5TB always available at every node.
>
> I was keeping an eye on remove-brick status and for very long time there
> was no failures and then at some point these 17000 failures appeared and it
> stayed like that.
>
>  Thanks
>
> Kashif
>
>
>
>
>
> Let me explain a little bit of background.
>
>
> On Mon, Feb 4, 2019 at 5:09 AM Nithya Balachandran 
> wrote:
>
>> Hi,
>>
>> The status shows quite a few failures. Please check the rebalance logs to
>> see why that happened. We can decide what to do based on the errors.
>> Once you run a commit, the brick will no longer be part of the volume and
>> you will not be able to access those files via the client.
>> Do you have sufficient space on the remaining bricks for the files on the
>> removed brick?
>>
>> Regards,
>> Nithya
>>
>> On Mon, 4 Feb 2019 at 03:50, mohammad kashif 
>> wrote:
>>
>>> Hi
>>>
>>> I have a pure distributed gluster volume with nine nodes and trying to
>>> remove one node, I ran
>>> gluster volume remove-brick atlasglust
>>> nodename:/glusteratlas/brick007/gv0 start
>>>
>>> It completed but with around 17000 failures
>>>
>>>   Node Rebalanced-files  size   scanned  failures
>>>skipped   status  run time in h:m:s
>>>-  ---   ---
>>>  ---   ---   --- 
>>>  --
>>>   nodename  418585827.5TB   6746030
>>>  17488 0completed  405:15:34
>>>
>>> I can see that there is still 1.5 TB of data on the node which I was
>>> trying to remove.
>>>
>>> I am not sure what to do now?  Should I run remove-brick command again
>>> so the files which has been failed can be tried again?
>>>
>>> or should I run commit first and then try to remove node again?
>>>
>>> Please advise as I don't want to remove files.
>>>
>>> Thanks
>>>
>>> Kashif
>>>
>>>
>>>
>>> ___
>>> Gluster-users mailing list
>>> Gluster-users@gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] gluster remove-brick

[Nithya Balachandran]

Hi,

The status shows quite a few failures. Please check the rebalance logs to
see why that happened. We can decide what to do based on the errors.
Once you run a commit, the brick will no longer be part of the volume and
you will not be able to access those files via the client.
Do you have sufficient space on the remaining bricks for the files on the
removed brick?

Regards,
Nithya

On Mon, 4 Feb 2019 at 03:50, mohammad kashif  wrote:

> Hi
>
> I have a pure distributed gluster volume with nine nodes and trying to
> remove one node, I ran
> gluster volume remove-brick atlasglust nodename:/glusteratlas/brick007/gv0
> start
>
> It completed but with around 17000 failures
>
>   Node Rebalanced-files  size   scanned  failures
>  skipped   status  run time in h:m:s
>-  ---   ---
>  ---   ---   --- 
>  --
>   nodename  418585827.5TB   6746030
>  17488 0completed  405:15:34
>
> I can see that there is still 1.5 TB of data on the node which I was
> trying to remove.
>
> I am not sure what to do now?  Should I run remove-brick command again so
> the files which has been failed can be tried again?
>
> or should I run commit first and then try to remove node again?
>
> Please advise as I don't want to remove files.
>
> Thanks
>
> Kashif
>
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Files losing permissions in GlusterFS 3.12

[Nithya Balachandran]

On Wed, 30 Jan 2019 at 19:12, Gudrun Mareike Amedick <
g.amed...@uni-luebeck.de> wrote:

> Hi,
>
> a bit additional info inlineAm Montag, den 28.01.2019, 10:23 +0100 schrieb
> Frank Ruehlemann:
> > Am Montag, den 28.01.2019, 09:50 +0530 schrieb Nithya Balachandran:
> > >
> > > On Fri, 25 Jan 2019 at 20:51, Gudrun Mareike Amedick <
> > > g.amed...@uni-luebeck.de> wrote:
> > >
> > > >
> > > > Hi all,
> > > >
> > > > we have a problem with a distributed dispersed volume (GlusterFS
> 3.12). We
> > > > have files that lost their permissions or gained sticky bits. The
> files
> > > > themselves seem to be okay.
> > > >
> > > > It looks like this:
> > > >
> > > > # ls -lah $file1
> > > > -- 1 www-data www-data 45M Jan 12 07:01 $file1
> > > >
> > > > # ls -lah $file2
> > > > -rw-rwS--T 1 $user $group 11K Jan  9 11:48 $file2
> > > >
> > > > # ls -lah $file3
> > > > -T 1 $user $group 6.8M Jan 12 08:17 $file3
> > > >
> > > > These are linkto files (internal dht files) and should not be
> visible on
> > > the mount point. Are they consistently visible like this or do they
> revert
> > > to the proper permissions after some time?
> > They didn't heal yet, even after more than 4 weeks. Therefore we decided
> > to recommend our users to fix their files by setting the correct
> > permissions again, which worked without problems. But for analysis
> > reasons we still have some broken files nobody touched yet.
> >
> > We know these linkto files but they were never visible to clients. We
> > did these ls-commands on a client, not on a brick.
>
> They have linkfile permissions but on brick side, it looks like this:
>
> root@gluster06:~# ls -lah /$brick/$file3
> -T 2 $user $group 1.7M Jan 12 08:17 /$brick/$file3
>
> That seems to be too big for a linkfile. Also, there is no file it could
> link to. There's no other file with that name at that path on any other
> subvolume.
>

This sounds like the rebalance failed to transition the file from a linkto
to a data file once the migration was complete. Please check the rebalance
logs on all nodes for any messages that refer to this file.
If you still see any such files, please check the its xattrs directly on
the brick. You should see one called trusted.glusterfs.dht.linkto. Let me
know if that is missing.

Regards,
Nithya

>
>
> >
> > >
> > > >
> > > > This is not what the permissions are supposed to look. They were 644
> or
> > > > 660 before. And they definitely had no sticky bits.
> > > > The permissions on the bricks match what I see on client side. So I
> think
> > > > the original permissions are lost without a chance to recover them,
> right?
> > > >
> > > >
> > > > With some files with weird looking permissions (but not with all of
> them),
> > > > I can do this:
> > > > # ls -lah $path/$file4
> > > > -rw-r--r-- 1 $user $group 6.0G Oct 11 09:34 $path/$file4
> > > > ls -lah $path | grep $file4
> > > > -rw-r-Sr-T  1 $user$group 6.0G Oct 11 09:34 $file4
> > >
> > > >
> > > > So, the permissions I see depend on how I'm querying them. The
> permissions
> > > > on brick side agree with the ladder result, stat sees the former.
> I'm not
> > > > sure how that works.
> > > >
> > > The S and T bits indicate that a file is being migrated. The difference
> > > seems to be because of the way lookup versus readdirp handle this  -
> this
> > > looks like a bug. Lookup will strip out the internal permissions set. I
> > > don't think readdirp does. This is happening because a rebalance is in
> > > progress.
> > There is no active rebalance. At least in "gluster volume rebalance
> > $VOLUME status" is none visible.
> >
> > And in the rebalance log file of this volume is the last line:
> > "[2019-01-11 02:14:50.101944] W … received signum (15), shutting down"
> >
> > >
> > > >
> > > > We know for at least a part of those files that they were okay at
> December
> > > > 19th. We got the first reports of weird-looking permissions at
> January
> > > > 12th. Between that, there was a rebalance running (January 7th to
> January
> > > > 11th). During that rebalance, a node was offline for a longer period
> of time
> > > > due to h

Re: [Gluster-users] Files losing permissions in GlusterFS 3.12

[Nithya Balachandran]

On Fri, 25 Jan 2019 at 20:51, Gudrun Mareike Amedick <
g.amed...@uni-luebeck.de> wrote:

> Hi all,
>
> we have a problem with a distributed dispersed volume (GlusterFS 3.12). We
> have files that lost their permissions or gained sticky bits. The files
> themselves seem to be okay.
>
> It looks like this:
>
> # ls -lah $file1
> -- 1 www-data www-data 45M Jan 12 07:01 $file1
>
> # ls -lah $file2
> -rw-rwS--T 1 $user $group 11K Jan  9 11:48 $file2
>
> # ls -lah $file3
> -T 1 $user $group 6.8M Jan 12 08:17 $file3
>
> These are linkto files (internal dht files) and should not be visible on
the mount point. Are they consistently visible like this or do they revert
to the proper permissions after some time?



> This is not what the permissions are supposed to look. They were 644 or
> 660 before. And they definitely had no sticky bits.
> The permissions on the bricks match what I see on client side. So I think
> the original permissions are lost without a chance to recover them, right?
>
>
> With some files with weird looking permissions (but not with all of them),
> I can do this:
> # ls -lah $path/$file4
> -rw-r--r-- 1 $user $group 6.0G Oct 11 09:34 $path/$file4
> ls -lah $path | grep $file4
> -rw-r-Sr-T  1 $user$group 6.0G Oct 11 09:34 $file4


> So, the permissions I see depend on how I'm querying them. The permissions
> on brick side agree with the ladder result, stat sees the former. I'm not
> sure how that works.
>
The S and T bits indicate that a file is being migrated. The difference
seems to be because of the way lookup versus readdirp handle this  - this
looks like a bug. Lookup will strip out the internal permissions set. I
don't think readdirp does. This is happening because a rebalance is in
progress.


> We know for at least a part of those files that they were okay at December
> 19th. We got the first reports of weird-looking permissions at January
> 12th. Between that, there was a rebalance running (January 7th to January
> 11th). During that rebalance, a node was offline for a longer period of time
> due to hardware issues. The output of "gluster volume heal $VOLUME info"
> shows no files though.
>
> For all files with broken permissions we found so far, the following lines
> are in the rebalance log:
>
> [2019-01-07 09:31:11.004802] I [MSGID: 109045]
> [dht-common.c:2456:dht_lookup_cbk] 0-$VOLUME-dht: linkfile not having link
> subvol for $file5
> [2019-01-07 09:31:11.262273] I [MSGID: 109069]
> [dht-common.c:1410:dht_lookup_unlink_of_false_linkto_cbk] 0-$VOLUME-dht:
> lookup_unlink returned with
> op_ret -> 0 and op-errno -> 0 for $file5
> [2019-01-07 09:31:11.266014] I [dht-rebalance.c:1570:dht_migrate_file]
> 0-$VOLUME-dht: $file5: attempting to move from $VOLUME-readdir-ahead-0 to
> $VOLUME-readdir-ahead-5
> [2019-01-07 09:31:11.278120] I [dht-rebalance.c:1570:dht_migrate_file]
> 0-$VOLUME-dht: $file5: attempting to move from $VOLUME-readdir-ahead-0 to
> $VOLUME-readdir-ahead-5
> [2019-01-07 09:31:11.732175] W [dht-rebalance.c:2159:dht_migrate_file]
> 0-$VOLUME-dht: $file5: failed to perform removexattr on
> $VOLUME-readdir-ahead-0
> (No data available)
> [2019-01-07 09:31:11.737319] W [MSGID: 109023]
> [dht-rebalance.c:2179:dht_migrate_file] 0-$VOLUME-dht: $file5: failed to do
> a stat on $VOLUME-readdir-
> ahead-0 [No such file or directory]
> [2019-01-07 09:31:11.744382] I [MSGID: 109022]
> [dht-rebalance.c:2218:dht_migrate_file] 0-$VOLUME-dht: completed migration
> of $file5 from subvolume
> $VOLUME-readdir-ahead-0 to $VOLUME-readdir-ahead-5
> [2019-01-07 09:31:11.744676] I [MSGID: 109022]
> [dht-rebalance.c:2218:dht_migrate_file] 0-$VOLUME-dht: completed migration
> of $file5 from subvolume
> $VOLUME-readdir-ahead-0 to $VOLUME-readdir-ahead-5
>
>
>
> I've searched the brick logs for $file5 with broken permissions and found
> this on all bricks from (I think) the subvolume $VOLUME-readdir-ahead-5:
>
> [2019-01-07 09:32:13.821545] I [MSGID: 113030] [posix.c:2171:posix_unlink]
> 0-$VOLUME-posix: open-fd-key-status: 0 for $file5
> [2019-01-07 09:32:13.821609] I [MSGID: 113031]
> [posix.c:2084:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0
> for $file5
>
>
>
> Also, we noticed that many directories got their modification time
> updated. It was set to the rebalance date. Is that supposed to happen?
>
>
> We had parallel-readdir enabled during the rebalance. We disabled it since
> we had empty directories that couldn't be deleted. I was able to delete
> those dirs after that.
>

Was this disabled during the rebalance? parallel-readdirp changes the
volume graph for clients but not for the rebalance process causing it to
fail to find the linkto subvols.


>
> Also, we have directories who lost their GFID on some bricks. Again.


Is this the missing symlink problem that was reported earlier?

Regards,
Nithya

>
>

> What happened? Can we do something to fix this? And could that happen
> again?
>
> We want to upgrade to 4.1 soon. Is it safe to do that or could it 

Re: [Gluster-users] usage of harddisks: each hdd a brick? raid?

[Nithya Balachandran]

On Tue, 22 Jan 2019 at 11:42, Amar Tumballi Suryanarayan <
atumb...@redhat.com> wrote:

>
>
> On Thu, Jan 10, 2019 at 1:56 PM Hu Bert  wrote:
>
>> Hi,
>>
>> > > We ara also using 10TB disks, heal takes 7-8 days.
>> > > You can play with "cluster.shd-max-threads" setting. It is default 1 I
>> > > think. I am using it with 4.
>> > > Below you can find more info:
>> > > https://access.redhat.com/solutions/882233
>> > cluster.shd-max-threads: 8
>> > cluster.shd-wait-qlength: 1
>>
>> Our setup:
>> cluster.shd-max-threads: 2
>> cluster.shd-wait-qlength: 1
>>
>> > >> Volume Name: shared
>> > >> Type: Distributed-Replicate
>> > A, you have distributed-replicated volume, but I choose only replicated
>> > (for beginning simplicity :)
>> > May be replicated volume are healing faster?
>>
>> Well, maybe our setup with 3 servers and 4 disks=bricks == 12 bricks,
>> resulting in a distributed-replicate volume (all /dev/sd{a,b,c,d}
>> identical) , isn't optimal? And it would be better to create a
>> replicate 3 volume with only 1 (big) brick per server (with 4 disks:
>> either a logical volume or sw/hw raid)?
>>
>> But it would be interesting to know if a replicate volume is healing
>> faster than a distributed-replicate volume - even if there was only 1
>> faulty brick.
>>
>>
> We don't have any data point to agree to this, but it may be true.
> Specially, as the crawling when DHT (ie, distribute) is involved can get
> little slower, which means, the healing would get slower too.
>

If the healing is being done by the Self heal daemon, the slowdown is not
due to dht (shd does not load dht).

>
> We are trying to experiment few performance enhancement patches (like
> https://review.gluster.org/20636), would be great to see how things work
> with newer base. Will keep the list updated about performance numbers once
> we have some more data on them.
>
> -Amar
>
>
>>
>> Thx
>> Hubert
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>>
>>
>
> --
> Amar Tumballi (amarts)
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] invisible files in some directory

[Nithya Balachandran]

On Fri, 18 Jan 2019 at 14:25, Mauro Tridici  wrote:

> Dear Users,
>
> I’m facing with a new problem on our gluster volume (v. 3.12.14).
> Sometime it happen that “ls” command execution, in a specified directory,
> return empty output.
> “ls” command output is empty, but I know that the involved directory
> contains some files and subdirectories.
> In fact, if I try to execute “ls” command against a specified file (in the
> same folder) I can see that the file is there.
>
> In a few words:
>
> “ls" command output executed in a particular folder is empty;
> "ls filename” command output executed in the same folder is ok.
>
> There is something I can do in order to identify the cause of this issue?
>
>
Yes, please take a tcpdump of the client when running the ls on the
problematic directory and send it across.

tcpdump -i any -s 0 -w /var/tmp/dirls.pcap tcp and not port 22


We have seen such issues when the gfid handle for the directory is missing
on the bricks.
Regards,
Nithya




> You can find below some information about the volume.
> Thank you in advance,
> Mauro Tridici
>
> [root@s01 ~]# gluster volume info
>
>
> Volume Name: tier2
> Type: Distributed-Disperse
> Volume ID: a28d88c5-3295-4e35-98d4-210b3af9358c
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 12 x (4 + 2) = 72
> Transport-type: tcp
> Bricks:
> Brick1: s01-stg:/gluster/mnt1/brick
> Brick2: s02-stg:/gluster/mnt1/brick
> Brick3: s03-stg:/gluster/mnt1/brick
> Brick4: s01-stg:/gluster/mnt2/brick
> Brick5: s02-stg:/gluster/mnt2/brick
> Brick6: s03-stg:/gluster/mnt2/brick
> Brick7: s01-stg:/gluster/mnt3/brick
> Brick8: s02-stg:/gluster/mnt3/brick
> Brick9: s03-stg:/gluster/mnt3/brick
> Brick10: s01-stg:/gluster/mnt4/brick
> Brick11: s02-stg:/gluster/mnt4/brick
> Brick12: s03-stg:/gluster/mnt4/brick
> Brick13: s01-stg:/gluster/mnt5/brick
> Brick14: s02-stg:/gluster/mnt5/brick
> Brick15: s03-stg:/gluster/mnt5/brick
> Brick16: s01-stg:/gluster/mnt6/brick
> Brick17: s02-stg:/gluster/mnt6/brick
> Brick18: s03-stg:/gluster/mnt6/brick
> Brick19: s01-stg:/gluster/mnt7/brick
> Brick20: s02-stg:/gluster/mnt7/brick
> Brick21: s03-stg:/gluster/mnt7/brick
> Brick22: s01-stg:/gluster/mnt8/brick
> Brick23: s02-stg:/gluster/mnt8/brick
> Brick24: s03-stg:/gluster/mnt8/brick
> Brick25: s01-stg:/gluster/mnt9/brick
> Brick26: s02-stg:/gluster/mnt9/brick
> Brick27: s03-stg:/gluster/mnt9/brick
> Brick28: s01-stg:/gluster/mnt10/brick
> Brick29: s02-stg:/gluster/mnt10/brick
> Brick30: s03-stg:/gluster/mnt10/brick
> Brick31: s01-stg:/gluster/mnt11/brick
> Brick32: s02-stg:/gluster/mnt11/brick
> Brick33: s03-stg:/gluster/mnt11/brick
> Brick34: s01-stg:/gluster/mnt12/brick
> Brick35: s02-stg:/gluster/mnt12/brick
> Brick36: s03-stg:/gluster/mnt12/brick
> Brick37: s04-stg:/gluster/mnt1/brick
> Brick38: s05-stg:/gluster/mnt1/brick
> Brick39: s06-stg:/gluster/mnt1/brick
> Brick40: s04-stg:/gluster/mnt2/brick
> Brick41: s05-stg:/gluster/mnt2/brick
> Brick42: s06-stg:/gluster/mnt2/brick
> Brick43: s04-stg:/gluster/mnt3/brick
> Brick44: s05-stg:/gluster/mnt3/brick
> Brick45: s06-stg:/gluster/mnt3/brick
> Brick46: s04-stg:/gluster/mnt4/brick
> Brick47: s05-stg:/gluster/mnt4/brick
> Brick48: s06-stg:/gluster/mnt4/brick
> Brick49: s04-stg:/gluster/mnt5/brick
> Brick50: s05-stg:/gluster/mnt5/brick
> Brick51: s06-stg:/gluster/mnt5/brick
> Brick52: s04-stg:/gluster/mnt6/brick
> Brick53: s05-stg:/gluster/mnt6/brick
> Brick54: s06-stg:/gluster/mnt6/brick
> Brick55: s04-stg:/gluster/mnt7/brick
> Brick56: s05-stg:/gluster/mnt7/brick
> Brick57: s06-stg:/gluster/mnt7/brick
> Brick58: s04-stg:/gluster/mnt8/brick
> Brick59: s05-stg:/gluster/mnt8/brick
> Brick60: s06-stg:/gluster/mnt8/brick
> Brick61: s04-stg:/gluster/mnt9/brick
> Brick62: s05-stg:/gluster/mnt9/brick
> Brick63: s06-stg:/gluster/mnt9/brick
> Brick64: s04-stg:/gluster/mnt10/brick
> Brick65: s05-stg:/gluster/mnt10/brick
> Brick66: s06-stg:/gluster/mnt10/brick
> Brick67: s04-stg:/gluster/mnt11/brick
> Brick68: s05-stg:/gluster/mnt11/brick
> Brick69: s06-stg:/gluster/mnt11/brick
> Brick70: s04-stg:/gluster/mnt12/brick
> Brick71: s05-stg:/gluster/mnt12/brick
> Brick72: s06-stg:/gluster/mnt12/brick
> Options Reconfigured:
> disperse.eager-lock: off
> diagnostics.count-fop-hits: on
> diagnostics.latency-measurement: on
> cluster.server-quorum-type: server
> features.default-soft-limit: 90
> features.quota-deem-statfs: on
> performance.io-thread-count: 16
> disperse.cpu-extensions: auto
> performance.io-cache: off
> network.inode-lru-limit: 5
> performance.md-cache-timeout: 600
> performance.cache-invalidation: on
> performance.stat-prefetch: on
> features.cache-invalidation-timeout: 600
> features.cache-invalidation: on
> cluster.readdir-optimize: on
> performance.parallel-readdir: off
> performance.readdir-ahead: on
> cluster.lookup-optimize: on
> client.event-threads: 4
> server.event-threads: 4
> nfs.disable: on
> transport.address-family: inet
> cluster.quorum-type: none
> cluster.min-free-disk: 10
> 

Re: [Gluster-users] Input/output error on FUSE log

[Nithya Balachandran]

I don't see write failures in the log but I do see fallocate failing with
EIO.

[2019-01-07 19:16:44.846187] W [MSGID: 109011]
[dht-layout.c:163:dht_layout_search] 0-gv1-dht: no subvolume for hash
(value) = 1285124113
[2019-01-07 19:16:44.846194] D [MSGID: 0]
[dht-helper.c:969:dht_subvol_get_hashed] 0-gv1-dht: No hashed subvolume for
path=/.shard/aa3ef10e-95e0-40d3-9464-133d72fa8a95.185
[2019-01-07 19:16:44.846200] D [MSGID: 0] [dht-common.c:7631:dht_mknod]
0-gv1-dht: no subvolume in layout for
path=/.shard/aa3ef10e-95e0-40d3-9464-133d72fa8a95.185   <---  *** *DHT
failed to find a hashed subvol* ***

[2019-01-07 19:16:44.846207] D [MSGID: 0] [dht-common.c:7712:dht_mknod]
0-stack-trace: stack-address: 0x7f6748006778, gv1-dht returned -1 error:
Input/output error [Input/output error]
[2019-01-07 19:16:44.846215] D [MSGID: 0]
[shard.c:3645:shard_common_mknod_cbk] 0-gv1-shard: mknod of shard 185
failed: Input/output error
[2019-01-07 19:16:44.846223] D [MSGID: 0]
[shard.c:720:shard_common_inode_write_failure_unwind] 0-stack-trace:
stack-address: 0x7f6748006778, gv1-shard returned -1 error: Input/output
error [Input/output error]
[2019-01-07 19:16:44.846234] D [MSGID: 0]
[defaults.c:1352:default_fallocate_cbk] 0-stack-trace: stack-address:
0x7f6748006778, gv1-quick-read returned -1 error: Input/output error
[Input/output error]
[2019-01-07 19:16:44.846244] D [MSGID: 0]
[defaults.c:1352:default_fallocate_cbk] 0-stack-trace: stack-address:
0x7f6748006778, gv1-open-behind returned -1 error: Input/output error
[Input/output error]
[2019-01-07 19:16:44.846254] D [MSGID: 0]
[md-cache.c:2715:mdc_fallocate_cbk] 0-stack-trace: stack-address:
0x7f6748006778, gv1-md-cache returned -1 error: Input/output error
[Input/output error]
[2019-01-07 19:16:44.846264] D [MSGID: 0]
[defaults.c:1352:default_fallocate_cbk] 0-stack-trace: stack-address:
0x7f6748006778, gv1-io-threads returned -1 error: Input/output error
[Input/output error]
[2019-01-07 19:16:44.846274] D [MSGID: 0]
[io-stats.c:2528:io_stats_fallocate_cbk] 0-stack-trace: stack-address:
0x7f6748006778, gv1 returned -1 error: Input/output error [Input/output
error]
[2019-01-07 19:16:44.846284] W [fuse-bridge.c:1441:fuse_err_cbk]
0-glusterfs-fuse: 1373: FALLOCATE() ERR => -1 (Input/output error)
[2019-01-07 19:16:44.846298] T [fuse-bridge.c:278:send_fuse_iov]
0-glusterfs-fuse: writev() result 16/16

Please get the xattrs on the .shard directory on each brick of the volume
so we can check if the layout is complete:

getfattr -e hex -m . -d /.shard

Thanks,
Nithya

On Thu, 10 Jan 2019 at 02:25, Matt Waymack  wrote:

> Has anyone any other ideas where to look?  This is only affecting FUSE
> clients.  SMB clients are unaffected by this problem.
>
>
>
> Thanks!
>
>
>
> *From:* gluster-users-boun...@gluster.org <
> gluster-users-boun...@gluster.org> *On Behalf Of *Matt Waymack
> *Sent:* Monday, January 7, 2019 1:19 PM
> *To:* Raghavendra Gowdappa 
> *Cc:* gluster-users@gluster.org List 
> *Subject:* Re: [Gluster-users] Input/output error on FUSE log
>
>
>
> Attached are the logs from when a failure occurred with diagnostics set to
> trace.
>
>
>
> Thank you!
>
>
>
> *From:* Raghavendra Gowdappa 
> *Sent:* Saturday, January 5, 2019 8:32 PM
> *To:* Matt Waymack 
> *Cc:* gluster-users@gluster.org List 
> *Subject:* Re: [Gluster-users] Input/output error on FUSE log
>
>
>
>
>
>
>
> On Sun, Jan 6, 2019 at 7:58 AM Raghavendra Gowdappa 
> wrote:
>
>
>
>
>
> On Sun, Jan 6, 2019 at 4:19 AM Matt Waymack  wrote:
>
> Hi all,
>
>
>
> I'm having a problem writing to our volume.  When writing files larger
> than about 2GB, I get an intermittent issue where the write will fail and
> return Input/Output error.  This is also shown in the FUSE log of the
> client (this is affecting all clients).  A snip of a client log is below:
>
> [2019-01-05 22:39:44.581371] W [fuse-bridge.c:2474:fuse_writev_cbk]
> 0-glusterfs-fuse: 51040978: WRITE => -1
> gfid=82a0b5c4-7ef3-43c2-ad86-41e16673d7c2 fd=0x7f949839a368 (Input/output
> error)
>
> [2019-01-05 22:39:44.598392] W [fuse-bridge.c:1441:fuse_err_cbk]
> 0-glusterfs-fuse: 51040979: FLUSH() ERR => -1 (Input/output error)
>
> [2019-01-05 22:39:47.420920] W [fuse-bridge.c:2474:fuse_writev_cbk]
> 0-glusterfs-fuse: 51041266: WRITE => -1
> gfid=0e8e1e13-97a5-478a-bc58-e81ddf3698a3 fd=0x7f949809b7f8 (Input/output
> error)
>
> [2019-01-05 22:39:47.433377] W [fuse-bridge.c:1441:fuse_err_cbk]
> 0-glusterfs-fuse: 51041267: FLUSH() ERR => -1 (Input/output error)
>
> [2019-01-05 22:39:50.441531] W [fuse-bridge.c:2474:fuse_writev_cbk]
> 0-glusterfs-fuse: 51041548: WRITE => -1
> gfid=0e8e1e13-97a5-478a-bc58-e81ddf3698a3 fd=0x7f949839a368 (Input/output
> error)
>
> [2019-01-05 22:39:50.451914] W [fuse-bridge.c:1441:fuse_err_cbk]
> 0-glusterfs-fuse: 51041549: FLUSH() ERR => -1 (Input/output error)
>
> The message "W [MSGID: 109011] [dht-layout.c:163:dht_layout_search]
> 0-gv1-dht: no subvolume for hash (value) = 1311504267" repeated 1721 times
> between [2019-01-05 

Re: [Gluster-users] A broken file that can not be deleted

[Nithya Balachandran]

On Wed, 9 Jan 2019 at 19:49, Dmitry Isakbayev  wrote:

> I am seeing a broken file that exists on 2 out of 3 nodes.  The
> application trying to use the file throws file permissions error.  ls, rm,
> mv, touch all throw "Input/output error"
>
> $ ls -la
> ls: cannot access .download_suspensions.memo: Input/output error
> drwxrwxr-x. 2 ossadmin ossadmin  4096 Jan  9 08:06 .
> drwxrwxr-x. 5 ossadmin ossadmin  4096 Jan  3 11:36 ..
> -?? ? ????
> .download_suspensions.memo
>
> $ rm ".download_suspensions.memo"
> rm: cannot remove ‘.download_suspensions.memo’: Input/output error
>
> Do you see any errors in the mount log?
Regards,
Nithya

>
>
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] update to 4.1.6-1 and fix-layout failing

[Nithya Balachandran]

On Fri, 4 Jan 2019 at 17:10, mohammad kashif  wrote:

> Hi Nithya
>
> rebalance logs has only these warnings
> 2019-01-04 09:59:20.826261] W [rpc-clnt.c:1753:rpc_clnt_submit]
> 0-atlasglust-client-5: error returned while attempting to connect to
> host:(null), port:0 [2019-01-04 09:59:20.828113] W
> [rpc-clnt.c:1753:rpc_clnt_submit] 0-atlasglust-client-6: error returned
> while attempting to connect to host:(null), port:0 [2019-01-04
> 09:59:20.832017] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-atlasglust-client-4:
> error returned while attempting to connect to host:(null), port:0
>

Please send me the rebalance logs if possible. Are 08 and 09 the newly
added nodes?  Are no directories being created on those ?

>
> gluster volume rebalance atlasglust status
>Node
> status   run time in h:m:s
>   -
> ---
>   localhost fix-layout
> in progress1:0:59
>  pplxgluster02.physics.ox.ac.uk
> fix-layout in progress1:0:59
>  pplxgluster03.physics.ox.ac.uk
> fix-layout in progress1:0:59
>  pplxgluster04.physics.ox.ac.uk
> fix-layout in progress1:0:59
>  pplxgluster05.physics.ox.ac.uk
> fix-layout in progress1:0:59
>  pplxgluster06.physics.ox.ac.uk
> fix-layout in progress1:0:59
>  pplxgluster07.physics.ox.ac.uk
> fix-layout in progress1:0:59
>  pplxgluster08.physics.ox.ac.uk
> fix-layout in progress1:0:59
>  pplxgluster09.physics.ox.ac.uk
> fix-layout in progress1:0:59
>
> But there is no new entry in logs for last one hour and I can't see any
> new directories being created.
>
> Thanks
>
> Kashif
>
>
> On Fri, Jan 4, 2019 at 10:42 AM Nithya Balachandran 
> wrote:
>
>>
>>
>> On Fri, 4 Jan 2019 at 15:48, mohammad kashif 
>> wrote:
>>
>>> Hi
>>>
>>> I have updated our distributed gluster storage from 3.12.9-1 to 4.1.6-1.
>>> The existing cluster had seven servers totalling in around 450 TB. OS is
>>> Centos7.  The update went OK and I could access files.
>>> Then I added two more servers of 90TB each to cluster and started
>>> fix-layout
>>>
>>> gluster volume rebalance atlasglust fix-layout start
>>>
>>> Some directories were created at new servers and then stopped although
>>> rebalance status was showing that it is still running. I think it stopped
>>> creating new directories after this error
>>>
>>> E [MSGID: 106061]
>>> [glusterd-utils.c:10697:glusterd_volume_rebalance_use_rsp_dict] 0-glusterd:
>>> failed to get index The message "E [MSGID: 106061]
>>> [glusterd-utils.c:10697:glusterd_volume_rebalance_use_rsp_dict] 0-glusterd:
>>> failed to get index" repeated 7 times between [2019-01-03 13:16:31.146779]
>>> and [2019-01-03 13:16:31.158612]
>>>
>>>
>> There are also many warning like this
>>> [2019-01-03 16:04:34.120777] I [MSGID: 106499]
>>> [glusterd-handler.c:4314:__glusterd_handle_status_volume] 0-management:
>>> Received status volume req for volume atlasglust [2019-01-03
>>> 17:04:28.541805] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-management: error
>>> returned while attempting to connect to host:(null), port:0
>>>
>>> These are the glusterd logs. Do you see any errors in the rebalance logs
>> for this volume?
>>
>>
>>> I waited for around 12 hours and then stopped fix-layout and started
>>> again
>>> I can see the same error again
>>>
>>> [2019-01-04 09:59:20.825930] E [MSGID: 106061]
>>> [glusterd-utils.c:10697:glusterd_volume_rebalance_use_rsp_dict] 0-glusterd:
>>> failed to get index The message "E [MSGID: 106061]
>>> [glusterd-utils.c:10697:glusterd_volume_rebalance_use_rsp_dict] 0-glusterd:
>>> failed to get index" repeated 7 times between [2019-01-04 09:59:20.825930]
>>> and [2019-01-04 09:59:20.837068]
>>>
>>> Please suggest as it is our production service.
>>>
>>> At the moment, I have stopped clients from using file system. Would it
>>> be OK if I allow clients to access file system while fix-layout is still
>>> going.
>>>
>>> Thanks
>>>
>>> Kashif
>>>
>>>
>>>
>>> ___
>>> Gluster-users mailing list
>>> Gluster-users@gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] update to 4.1.6-1 and fix-layout failing

[Nithya Balachandran]

On Fri, 4 Jan 2019 at 15:48, mohammad kashif  wrote:

> Hi
>
> I have updated our distributed gluster storage from 3.12.9-1 to 4.1.6-1.
> The existing cluster had seven servers totalling in around 450 TB. OS is
> Centos7.  The update went OK and I could access files.
> Then I added two more servers of 90TB each to cluster and started
> fix-layout
>
> gluster volume rebalance atlasglust fix-layout start
>
> Some directories were created at new servers and then stopped although
> rebalance status was showing that it is still running. I think it stopped
> creating new directories after this error
>
> E [MSGID: 106061]
> [glusterd-utils.c:10697:glusterd_volume_rebalance_use_rsp_dict] 0-glusterd:
> failed to get index The message "E [MSGID: 106061]
> [glusterd-utils.c:10697:glusterd_volume_rebalance_use_rsp_dict] 0-glusterd:
> failed to get index" repeated 7 times between [2019-01-03 13:16:31.146779]
> and [2019-01-03 13:16:31.158612]
>
>
There are also many warning like this
> [2019-01-03 16:04:34.120777] I [MSGID: 106499]
> [glusterd-handler.c:4314:__glusterd_handle_status_volume] 0-management:
> Received status volume req for volume atlasglust [2019-01-03
> 17:04:28.541805] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-management: error
> returned while attempting to connect to host:(null), port:0
>
> These are the glusterd logs. Do you see any errors in the rebalance logs
for this volume?


> I waited for around 12 hours and then stopped fix-layout and started again
> I can see the same error again
>
> [2019-01-04 09:59:20.825930] E [MSGID: 106061]
> [glusterd-utils.c:10697:glusterd_volume_rebalance_use_rsp_dict] 0-glusterd:
> failed to get index The message "E [MSGID: 106061]
> [glusterd-utils.c:10697:glusterd_volume_rebalance_use_rsp_dict] 0-glusterd:
> failed to get index" repeated 7 times between [2019-01-04 09:59:20.825930]
> and [2019-01-04 09:59:20.837068]
>
> Please suggest as it is our production service.
>
> At the moment, I have stopped clients from using file system. Would it be
> OK if I allow clients to access file system while fix-layout is still going.
>
> Thanks
>
> Kashif
>
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [Stale file handle] in shard volume

[Nithya Balachandran]

Adding Krutika.

On Wed, 2 Jan 2019 at 20:56, Olaf Buitelaar 
wrote:

> Hi Nithya,
>
> Thank you for your reply.
>
> the VM's using the gluster volumes keeps on getting paused/stopped on
> errors like these;
> [2019-01-02 02:33:44.469132] E [MSGID: 133010]
> [shard.c:1724:shard_common_lookup_shards_cbk] 0-ovirt-kube-shard: Lookup on
> shard 101487 failed. Base file gfid = a38d64bc-a28b-4ee1-a0bb-f919e7a1022c
> [Stale file handle]
> [2019-01-02 02:33:44.563288] E [MSGID: 133010]
> [shard.c:1724:shard_common_lookup_shards_cbk] 0-ovirt-kube-shard: Lookup on
> shard 101488 failed. Base file gfid = a38d64bc-a28b-4ee1-a0bb-f919e7a1022c
> [Stale file handle]
>
> Krutika, Can you take a look at this?


>
> What i'm trying to find out, if i can purge all gluster volumes from all
> possible stale file handles (and hopefully find a method to prevent this in
> the future), so the VM's can start running stable again.
> For this i need to know when the "shard_common_lookup_shards_cbk" function
> considers a file as stale.
> The statement; "Stale file handle errors show up when a file with a
> specified gfid is not found." doesn't seem to cover it all, as i've shown
> in earlier mails the shard file and glusterfs/xx/xx/uuid file do both
> exist, and have the same inode.
> If the criteria i'm using aren't correct, could you please tell me which
> criteria i should use to determine if a file is stale or not?
> these criteria are just based observations i made, moving the stale files
> manually. After removing them i was able to start the VM again..until some
> time later it hangs on another stale shard file unfortunate.
>
> Thanks Olaf
>
> Op wo 2 jan. 2019 om 14:20 schreef Nithya Balachandran <
> nbala...@redhat.com>:
>
>>
>>
>> On Mon, 31 Dec 2018 at 01:27, Olaf Buitelaar 
>> wrote:
>>
>>> Dear All,
>>>
>>> till now a selected group of VM's still seem to produce new stale file's
>>> and getting paused due to this.
>>> I've not updated gluster recently, however i did change the op version
>>> from 31200 to 31202 about a week before this issue arose.
>>> Looking at the .shard directory, i've 100.000+ files sharing the same
>>> characteristics as a stale file. which are found till now,
>>> they all have the sticky bit set, e.g. file permissions; -T. are
>>> 0kb in size, and have the trusted.glusterfs.dht.linkto attribute.
>>>
>>
>> These are internal files used by gluster and do not necessarily mean they
>> are stale. They "point" to data files which may be on different bricks
>> (same name, gfid etc but no linkto xattr and no T permissions).
>>
>>
>>> These files range from long a go (beginning of the year) till now. Which
>>> makes me suspect this was laying dormant for some time now..and somehow
>>> recently surfaced.
>>> Checking other sub-volumes they contain also 0kb files in the .shard
>>> directory, but don't have the sticky bit and the linkto attribute.
>>>
>>> Does anybody else experience this issue? Could this be a bug or an
>>> environmental issue?
>>>
>> These are most likely valid files- please do not delete them without
>> double-checking.
>>
>> Stale file handle errors show up when a file with a specified gfid is not
>> found. You will need to debug the files for which you see this error by
>> checking the bricks to see if they actually exist.
>>
>>>
>>> Also i wonder if there is any tool or gluster command to clean all stale
>>> file handles?
>>> Otherwise i'm planning to make a simple bash script, which iterates over
>>> the .shard dir, checks each file for the above mentioned criteria, and
>>> (re)moves the file and the corresponding .glusterfs file.
>>> If there are other criteria needed to identify a stale file handle, i
>>> would like to hear that.
>>> If this is a viable and safe operation to do of course.
>>>
>>> Thanks Olaf
>>>
>>>
>>>
>>> Op do 20 dec. 2018 om 13:43 schreef Olaf Buitelaar <
>>> olaf.buitel...@gmail.com>:
>>>
>>>> Dear All,
>>>>
>>>> I figured it out, it appeared to be the exact same issue as described
>>>> here;
>>>> https://lists.gluster.org/pipermail/gluster-users/2018-March/033785.html
>>>> Another subvolume also had the shard file, only were all 0 bytes and
>>>> had the dht.linkto
>>>>
>>>> for reference;
>>>> [root@lease-04 ovirt-backbone-2]# getfattr -d -

Re: [Gluster-users] [Stale file handle] in shard volume

[Nithya Balachandran]

On Mon, 31 Dec 2018 at 01:27, Olaf Buitelaar 
wrote:

> Dear All,
>
> till now a selected group of VM's still seem to produce new stale file's
> and getting paused due to this.
> I've not updated gluster recently, however i did change the op version
> from 31200 to 31202 about a week before this issue arose.
> Looking at the .shard directory, i've 100.000+ files sharing the same
> characteristics as a stale file. which are found till now,
> they all have the sticky bit set, e.g. file permissions; -T. are
> 0kb in size, and have the trusted.glusterfs.dht.linkto attribute.
>

These are internal files used by gluster and do not necessarily mean they
are stale. They "point" to data files which may be on different bricks
(same name, gfid etc but no linkto xattr and no T permissions).


> These files range from long a go (beginning of the year) till now. Which
> makes me suspect this was laying dormant for some time now..and somehow
> recently surfaced.
> Checking other sub-volumes they contain also 0kb files in the .shard
> directory, but don't have the sticky bit and the linkto attribute.
>
> Does anybody else experience this issue? Could this be a bug or an
> environmental issue?
>
These are most likely valid files- please do not delete them without
double-checking.

Stale file handle errors show up when a file with a specified gfid is not
found. You will need to debug the files for which you see this error by
checking the bricks to see if they actually exist.

>
> Also i wonder if there is any tool or gluster command to clean all stale
> file handles?
> Otherwise i'm planning to make a simple bash script, which iterates over
> the .shard dir, checks each file for the above mentioned criteria, and
> (re)moves the file and the corresponding .glusterfs file.
> If there are other criteria needed to identify a stale file handle, i
> would like to hear that.
> If this is a viable and safe operation to do of course.
>
> Thanks Olaf
>
>
>
> Op do 20 dec. 2018 om 13:43 schreef Olaf Buitelaar <
> olaf.buitel...@gmail.com>:
>
>> Dear All,
>>
>> I figured it out, it appeared to be the exact same issue as described
>> here;
>> https://lists.gluster.org/pipermail/gluster-users/2018-March/033785.html
>> Another subvolume also had the shard file, only were all 0 bytes and had
>> the dht.linkto
>>
>> for reference;
>> [root@lease-04 ovirt-backbone-2]# getfattr -d -m . -e hex
>> .shard/f28cabcb-d169-41fc-a633-9bef4c4a8e40.51500
>> # file: .shard/f28cabcb-d169-41fc-a633-9bef4c4a8e40.51500
>>
>> security.selinux=0x73797374656d5f753a6f626a6563745f723a6574635f72756e74696d655f743a733000
>> trusted.gfid=0x298147e49f9748b2baf1c8fff897244d
>>
>> trusted.gfid2path.b48064c78d7a85c9=0x62653331383633382d653861302d346336642d393737642d3761393337616138343830362f66323863616263622d643136392d343166632d61362d3962656634633461386534302e3531353030
>>
>> trusted.glusterfs.dht.linkto=0x6f766972742d6261636b626f6e652d322d7265706c69636174652d3100
>>
>> [root@lease-04 ovirt-backbone-2]# getfattr -d -m . -e hex
>> .glusterfs/29/81/298147e4-9f97-48b2-baf1-c8fff897244d
>> # file: .glusterfs/29/81/298147e4-9f97-48b2-baf1-c8fff897244d
>>
>> security.selinux=0x73797374656d5f753a6f626a6563745f723a6574635f72756e74696d655f743a733000
>> trusted.gfid=0x298147e49f9748b2baf1c8fff897244d
>>
>> trusted.gfid2path.b48064c78d7a85c9=0x62653331383633382d653861302d346336642d393737642d3761393337616138343830362f66323863616263622d643136392d343166632d61362d3962656634633461386534302e3531353030
>>
>> trusted.glusterfs.dht.linkto=0x6f766972742d6261636b626f6e652d322d7265706c69636174652d3100
>>
>> [root@lease-04 ovirt-backbone-2]# stat
>> .glusterfs/29/81/298147e4-9f97-48b2-baf1-c8fff897244d
>>   File: ‘.glusterfs/29/81/298147e4-9f97-48b2-baf1-c8fff897244d’
>>   Size: 0   Blocks: 0  IO Block: 4096   regular empty
>> file
>> Device: fd01h/64769dInode: 1918631406  Links: 2
>> Access: (1000/-T)  Uid: (0/root)   Gid: (0/root)
>> Context: system_u:object_r:etc_runtime_t:s0
>> Access: 2018-12-17 21:43:36.405735296 +
>> Modify: 2018-12-17 21:43:36.405735296 +
>> Change: 2018-12-17 21:43:36.405735296 +
>>  Birth: -
>>
>> removing the shard file and glusterfs file from each node resolved the
>> issue.
>>
>> I also found this thread;
>> https://lists.gluster.org/pipermail/gluster-users/2018-December/035460.html
>> Maybe he suffers from the same issue.
>>
>> Best Olaf
>>
>>
>> Op wo 19 dec. 2018 om 21:56 schreef Olaf Buitelaar <
>> olaf.buitel...@gmail.com>:
>>
>>> Dear All,
>>>
>>> It appears i've a stale file in one of the volumes, on 2 files. These
>>> files are qemu images (1 raw and 1 qcow2).
>>> I'll just focus on 1 file since the situation on the other seems the
>>> same.
>>>
>>> The VM get's paused more or less directly after being booted with error;
>>> [2018-12-18 14:05:05.275713] E [MSGID: 133010]
>>> [shard.c:1724:shard_common_lookup_shards_cbk] 0-ovirt-backbone-2-shard:
>>> Lookup on shard 51500 

Re: [Gluster-users] distribute remove-brick has started migrating the wrong brick (glusterfs 3.8.13)

[Nithya Balachandran]

On Tue, 18 Dec 2018 at 21:11, Stephen Remde 
wrote:

> Nithya,
>
> You are correct, but as you stated earlier, it also has to migrate data
> from other bricks on the same host, so another 74TB on dc4-03 /dev/md0
> needs to be migrated?
>

Ah, ok. Let me clarify - it will migrate files to and from dc4-03 /dev/md0
to use the new directory layouts (which is correct behaviour for a
rebalance but unnecessary in this case as it can potentially cause the
remove-brick operation to take longer). It should not migrate all the data
from this brick so you should still see lots of files on it. Only the data
on the removed-brick will be moved off completely.

I recommend that you let the remove-brick proceed and keep an eye on the
disk usage of dc4-03 /dev/md0 as well. Let me know if you see if falling
drastically.

If you can afford some downtime, we could probably try alternate methods
but those would mean that users would not have access to some files on the
volume.


Regards,
Nithya






> > This is the current behaviour of rebalance and nothing to be concerned
> about - it will migrate data on all bricks on the nodes which host the
> bricks being removed
>
>
> Steve
>
> On Tue, 18 Dec 2018 at 15:37, Nithya Balachandran 
> wrote:
>
>>
>>
>> On Tue, 18 Dec 2018 at 14:56, Stephen Remde 
>> wrote:
>>
>>> Nithya,
>>>
>>> I've realised, I will not have enough space on the other bricks in my
>>> cluster to migrate data off the server so I can remove the single brick -
>>> is there a work around?
>>>
>>> As you can see below, the new brick was created with the wrong raid
>>> configuration, so I want to remove it recreate the raid, and re add it.
>>>
>>> xx Filesystem  Size  Used Avail Use% Mounted on
>>> dc4-01 /dev/md0 95T   87T  8.0T  92% /export/md0
>>> dc4-01 /dev/md1 95T   87T  8.4T  92% /export/md1
>>> dc4-01 /dev/md2 95T   86T  9.3T  91% /export/md2
>>> dc4-01 /dev/md3 95T   86T  8.9T  91% /export/md3
>>> dc4-02 /dev/md0 95T   89T  6.5T  94% /export/md0
>>> dc4-02 /dev/md1 95T   87T  8.4T  92% /export/md1
>>> dc4-02 /dev/md2 95T   87T  8.6T  91% /export/md2
>>> dc4-02 /dev/md3 95T   86T  8.8T  91% /export/md3
>>> dc4-03 /dev/md0 95T   74T   21T  78% /export/md0
>>> dc4-03 /dev/md1102T  519G  102T   1% /export/md1
>>>
>>>
>> I believe this is the brick being removed - the one that has about 519G
>> of data? If I have understood the scenario properly, there seems to be
>> plenty of free space on the other bricks (most seem to have terabytes free)
>> . Is there something I am missing ?
>>
>> Regards,
>> Nithya
>>
>>
>>> This is the backup storage, so if I HAVE to lose the 519GB and resync,
>>> that's an acceptable worst-case.
>>>
>>> gluster> v info video-backup
>>>
>>> Volume Name: video-backup
>>> Type: Distribute
>>> Volume ID: 887bdc2a-ca5e-4ca2-b30d-86831839ed04
>>> Status: Started
>>> Snapshot Count: 0
>>> Number of Bricks: 10
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: 10.0.0.41:/export/md0/brick
>>> Brick2: 10.0.0.42:/export/md0/brick
>>> Brick3: 10.0.0.43:/export/md0/brick
>>> Brick4: 10.0.0.41:/export/md1/brick
>>> Brick5: 10.0.0.42:/export/md1/brick
>>> Brick6: 10.0.0.41:/export/md2/brick
>>> Brick7: 10.0.0.42:/export/md2/brick
>>> Brick8: 10.0.0.41:/export/md3/brick
>>> Brick9: 10.0.0.42:/export/md3/brick
>>> Brick10: 10.0.0.43:/export/md1/brick
>>> Options Reconfigured:
>>> cluster.rebal-throttle: aggressive
>>> cluster.min-free-disk: 1%
>>> transport.address-family: inet
>>> performance.readdir-ahead: on
>>> nfs.disable: on
>>>
>>>
>>> Best,
>>>
>>> Steve
>>>
>>>
>>> On Wed, 12 Dec 2018 at 03:07, Nithya Balachandran 
>>> wrote:
>>>
>>>>
>>>> This is the current behaviour of rebalance and nothing to be concerned
>>>> about - it will migrate data on all bricks on the nodes which host the
>>>> bricks being removed. The data on the removed bricks will be moved to other
>>>> bricks, some of the  data on the other bricks on the node will just be
>>>> moved to other bricks based on the new directory layouts.
>>>> I will fix this in the near future but you don't need to to stop the
>>>> remove-brick operation.
>>>>
>>>> Regards

Re: [Gluster-users] distribute remove-brick has started migrating the wrong brick (glusterfs 3.8.13)

[Nithya Balachandran]

On Tue, 18 Dec 2018 at 14:56, Stephen Remde 
wrote:

> Nithya,
>
> I've realised, I will not have enough space on the other bricks in my
> cluster to migrate data off the server so I can remove the single brick -
> is there a work around?
>
> As you can see below, the new brick was created with the wrong raid
> configuration, so I want to remove it recreate the raid, and re add it.
>
> xx Filesystem  Size  Used Avail Use% Mounted on
> dc4-01 /dev/md0 95T   87T  8.0T  92% /export/md0
> dc4-01 /dev/md1 95T   87T  8.4T  92% /export/md1
> dc4-01 /dev/md2 95T   86T  9.3T  91% /export/md2
> dc4-01 /dev/md3 95T   86T  8.9T  91% /export/md3
> dc4-02 /dev/md0 95T   89T  6.5T  94% /export/md0
> dc4-02 /dev/md1 95T   87T  8.4T  92% /export/md1
> dc4-02 /dev/md2 95T   87T  8.6T  91% /export/md2
> dc4-02 /dev/md3 95T   86T  8.8T  91% /export/md3
> dc4-03 /dev/md0 95T   74T   21T  78% /export/md0
> dc4-03 /dev/md1102T  519G  102T   1% /export/md1
>
>
I believe this is the brick being removed - the one that has about 519G of
data? If I have understood the scenario properly, there seems to be plenty
of free space on the other bricks (most seem to have terabytes free) . Is
there something I am missing ?

Regards,
Nithya


> This is the backup storage, so if I HAVE to lose the 519GB and resync,
> that's an acceptable worst-case.
>
> gluster> v info video-backup
>
> Volume Name: video-backup
> Type: Distribute
> Volume ID: 887bdc2a-ca5e-4ca2-b30d-86831839ed04
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 10
> Transport-type: tcp
> Bricks:
> Brick1: 10.0.0.41:/export/md0/brick
> Brick2: 10.0.0.42:/export/md0/brick
> Brick3: 10.0.0.43:/export/md0/brick
> Brick4: 10.0.0.41:/export/md1/brick
> Brick5: 10.0.0.42:/export/md1/brick
> Brick6: 10.0.0.41:/export/md2/brick
> Brick7: 10.0.0.42:/export/md2/brick
> Brick8: 10.0.0.41:/export/md3/brick
> Brick9: 10.0.0.42:/export/md3/brick
> Brick10: 10.0.0.43:/export/md1/brick
> Options Reconfigured:
> cluster.rebal-throttle: aggressive
> cluster.min-free-disk: 1%
> transport.address-family: inet
> performance.readdir-ahead: on
> nfs.disable: on
>
>
> Best,
>
> Steve
>
>
> On Wed, 12 Dec 2018 at 03:07, Nithya Balachandran 
> wrote:
>
>>
>> This is the current behaviour of rebalance and nothing to be concerned
>> about - it will migrate data on all bricks on the nodes which host the
>> bricks being removed. The data on the removed bricks will be moved to other
>> bricks, some of the  data on the other bricks on the node will just be
>> moved to other bricks based on the new directory layouts.
>> I will fix this in the near future but you don't need to to stop the
>> remove-brick operation.
>>
>> Regards,
>> Nithya
>>
>> On Wed, 12 Dec 2018 at 06:36, Stephen Remde 
>> wrote:
>>
>>> I requested a brick be removed from a distribute only volume and it seems 
>>> to be migrating data from the wrong brick... unless I am reading this wrong 
>>> which I doubt because the disk usage is definitely decreasing on the wrong 
>>> brick.
>>>
>>> gluster> volume status
>>> Status of volume: video-backup
>>> Gluster process TCP Port  RDMA Port  Online  Pid
>>> --
>>> Brick 10.0.0.41:/export/md0/brick   49172 0  Y   
>>> 5306
>>> Brick 10.0.0.42:/export/md0/brick   49172 0  Y   
>>> 3651
>>> Brick 10.0.0.43:/export/md0/brick   49155 0  Y   
>>> 2826
>>> Brick 10.0.0.41:/export/md1/brick   49173 0  Y   
>>> 5311
>>> Brick 10.0.0.42:/export/md1/brick   49173 0  Y   
>>> 3656
>>> Brick 10.0.0.41:/export/md2/brick   49174 0  Y   
>>> 5316
>>> Brick 10.0.0.42:/export/md2/brick   49174 0  Y   
>>> 3662
>>> Brick 10.0.0.41:/export/md3/brick   49175 0  Y   
>>> 5322
>>> Brick 10.0.0.42:/export/md3/brick   49175 0  Y   
>>> 3667
>>> Brick 10.0.0.43:/export/md1/brick   49156 0  Y   
>>> 4836
>>>
>>> Task Status of Volume video-backup
>>> --
>>> Task : Rebalance
>>> ID   : 7895be7c-4ab9-440d-a301-c11

Re: [Gluster-users] Invisible files

[Nithya Balachandran]

On Fri, 14 Dec 2018 at 19:10, Raghavendra Gowdappa 
wrote:

>
>
> On Fri, Dec 14, 2018 at 6:38 PM Lindolfo Meira 
> wrote:
>
>> It happened to me using gluster 5.0, on OpenSUSE Leap 15, during a
>> benchmark with IOR: the volume would seem normally mounted, but I was
>> unable to overwrite files, and ls would show the volume as totally empty.
>> I could write new files without any problems. But still ls would not show
>> anything.
>>
>> Does anyone know anything about that?
>>
>
> can you get following information from the problematic directory on
> backend bricks?
>
> * ls -l 
> * getfattr -e hex -d -m. 
>

We have seen similar behaviour if the gfid handle is missing for the parent
directory on the bricks.
Once you get the gfid please check if the handle exists on the backend as
well.

>
>
>>
>> Lindolfo Meira, MSc
>> Diretor Geral, Centro Nacional de Supercomputação
>> Universidade Federal do Rio Grande do Sul
>> +55 (51) 3308-3122___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Should I be using gluster 3 or gluster 4?

[Nithya Balachandran]

On 6 November 2018 at 12:24, Jeevan Patnaik  wrote:

> Hi Vlad,
>
> I'm still confused of gluster releases. :(
> Is 3.13 an official gluster release? It's not mentioned in
> www.gluster.org/release-schedule
>
>
3.13 is EOL. It was a short term release.

Which is more stable 3.13.2 or 3.12.6 or 4.1.5?
>
> 3.12 is also EOL now that 5 is out.


> 3.13.2 was released in Jan and no minor releases since then..So, I expect
> it's a stable release. or maybe noone has used it enough to report bugs?
>
> I understand we may see bugs even in the most stable version while using
> it. I'm looking for a version thats safe to use with least chance of
> corrupting or loosing our files.
>
> I'm set to test 3.13.2 tiering feature, but have my thoughts about if
> 3.12.6 or 4.1.5 should be tested instead.
>



>
> Regards,
> Jeevan.
>
>
> On Nov 4, 2018 5:11 AM, "Vlad Kopylov"  wrote:
>
> If you doing replica - start with 5, run your load on it. You can always
> fallback to 3.12 or 4. It is not like your files will be gone.
> With distributed might be harder, files will still be on the bricks but
> you will have to consolidate them or copy in to new volume after downgrade.
> Documentation is all the same.
>
>
> v
>
> On Wed, Oct 31, 2018 at 2:54 AM Jeevan Patnaik 
> wrote:
>
>> Hi Vlad,
>>
>> Can gluster 4.1.5 too be used for production? There's no documentation
>> for gluster 4.
>>
>> Regards,
>> Jeevan.
>>
>> On Wed, Oct 31, 2018, 9:37 AM Vlad Kopylov  wrote:
>>
>>> 3.12.14 working fine in production for file access
>>> you can find vol and mount settings in mailing list archive
>>>
>>> On Tue, Oct 30, 2018 at 11:05 AM Jeevan Patnaik 
>>> wrote:
>>>
 Hi All,

 I see gluster 3 has reached end of life and gluster 5 has just been
 introduced.

 Is gluster 4.1.5 stable enough for production deployment? I see by
 default gluster docs point  to v3  only  and there  are no gluster docs
 for 4 or 5.  Why so? And I'm mainly looking for a stable gluster tiering
 feature and Kernek NFS support. I faced few issues with tiering in 3.14 and
 so thinking if I should switch to 4.1.5, as it will be a production
 deployment.

 Thank you.

 Regards,
 Jeevan.
 ___
 Gluster-users mailing list
 Gluster-users@gluster.org
 https://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Wrong volume size for distributed dispersed volume on 4.1.5

[Nithya Balachandran]

On 16 October 2018 at 20:04,  wrote:

> Hi,
>
> > > So we did a quick grep shared-brick-count 
> > > /var/lib/glusterd/vols/data_vol1/*
> on all boxes and found that on 5 out of 6 boxes this was
> shared-brick-count=0 for all bricks on remote boxes and 1 for local bricks.
> > >
> > > Is this the expected result or should we have all 1 everywhere (as the
> quick fix script from the case sets it)?
> >
> > No , this is fine. The shared-brick-count only needs to be 1 for the
> local bricks. The value for the remote bricks can be 0.
> >
> > > Also on one box (the one where we created the volume from, btw) we
> have shared-brick-count=0 for all remote bricks and 10 for the local bricks.
> >
> > This is a problem. The shared-brick-count should be 1 for the local
> bricks here as well.
> >
> > > Is it possible that the bug from 3.4 still exists in 4.1.5 and should
> we try the filter script which sets shared-brick-count=1 for all bricks?
> > >
> >
> > Can you try
> > 1. restarting glusterd on all the nodes one after another (not at the
> same time)
> > 2. Setting a volume option (say gluster volume set 
> cluster.min-free-disk 11%)
> >
> > and see if it fixes the issue?
>
> Hi,
>
> ok, this was a quick fix - volume size is correct again and the
> shared-brick-count is correct everywhere.
>
> We'll duly note this in our wiki.
>
> Thanks a lot!
>

If there were any directories created on the volume when the sizes were
wrong, the layouts sets on them are probably incorrect. You might want to
do a fix-layout on the volume.

Regards,
Nithya

>
> Joachim
> 
> -
> FreeMail powered by mail.de - MEHR SICHERHEIT, SERIOSITÄT UND KOMFORT
>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Wrong volume size for distributed dispersed volume on 4.1.5

[Nithya Balachandran]

Hi,



On 16 October 2018 at 18:20,  wrote:

> Hi everybody,
>
> I have created a distributed dispersed volume on 4.1.5 under centos7 like
> this a few days ago:
>
> gluster volume create data_vol1 disperse-data 4 redundancy 2 transport tcp
> \
> \
> gf-p-d-01.isec.foobar.com:/bricks/brick1/brick \
> gf-p-d-03.isec.foobar.com:/bricks/brick1/brick \
> gf-p-d-04.isec.foobar.com:/bricks/brick1/brick \
> gf-p-k-01.isec.foobar.com:/bricks/brick1/brick \
> gf-p-k-03.isec.foobar.com:/bricks/brick1/brick \
> gf-p-k-04.isec.foobar.com:/bricks/brick1/brick \
> \
> gf-p-d-01.isec.foobar.com:/bricks/brick2/brick \
> gf-p-d-03.isec.foobar.com:/bricks/brick2/brick \
> gf-p-d-04.isec.foobar.com:/bricks/brick2/brick \
> gf-p-k-01.isec.foobar.com:/bricks/brick2/brick \
> gf-p-k-03.isec.foobar.com:/bricks/brick2/brick \
> gf-p-k-04.isec.foobar.com:/bricks/brick2/brick \
> \
> ... same for brick3 to brick9...
> \
> gf-p-d-01.isec.foobar.com:/bricks/brick10/brick \
> gf-p-d-03.isec.foobar.com:/bricks/brick10/brick \
> gf-p-d-04.isec.foobar.com:/bricks/brick10/brick \
> gf-p-k-01.isec.foobar.com:/bricks/brick10/brick \
> gf-p-k-03.isec.foobar.com:/bricks/brick10/brick \
> gf-p-k-04.isec.foobar.com:/bricks/brick10/brick
>
> This worked nicely and resulted in the following filesystem:
> [root@gf-p-d-01 ~]# df -h /data/
> Dateisystem Größe Benutzt Verf. Verw% Eingehängt auf
> gf-p-d-01.isec.foobar.com:/data_vol1 219T 2,2T 217T 2% /data
>
> Each of the bricks resides on its own 6TB disk with 1 big partition
> formated with xfs.
>
> Yesterday a colleague looked at the filesystem and found some space
> missing...
> [root@gf-p-d-01 ~]# df -h /data/
> Filesystem Size Used Avail Use% Mounted on
> gf-p-d-01.isec.foobar.com:/data_vol1 22T 272G 22T 2% /data
>
> Some googling brought the following bug report against 3.4 which looks
> familiar:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1541830
>
> So we did a quick grep shared-brick-count /var/lib/glusterd/vols/data_vol1/*
> on all boxes and found that on 5 out of 6 boxes this was
> shared-brick-count=0 for all bricks on remote boxes and 1 for local bricks.
>
> Is this the expected result or should we have all 1 everywhere (as the
> quick fix script from the case sets it)?
>

No , this is fine. The shared-brick-count only needs to be 1 for the local
bricks. The value for the remote bricks can be 0.


>
> Also on one box (the one where we created the volume from, btw) we have
> shared-brick-count=0 for all remote bricks and 10 for the local bricks.
>

This is a problem. The shared-brick-count should be 1 for the local bricks
here as well.


> Is it possible that the bug from 3.4 still exists in 4.1.5 and should we
> try the filter script which sets shared-brick-count=1 for all bricks?
>
>
Can you try
1. restarting glusterd on all the nodes one after another (not at the same
time)
2. Setting a volume option (say gluster volume set 
cluster.min-free-disk 11%)

and see if it fixes the issue?

Regards,
Nithya



> The volume is not currently in production so now would be the time to play
> around and find the problem...
>
> TIA and regards,
>
> Joachim
>
>
> 
> -
> FreeMail powered by mail.de - MEHR SICHERHEIT, SERIOSITÄT UND KOMFORT
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Rebalance failed on Distributed Disperse volume based on 3.12.14 version

[Nithya Balachandran]

_select]
> 0-tier2-disperse-10: Insufficient available children for this request (have
> 0, need 4)
> [2018-10-05 23:48:44.736574] E [MSGID: 122034] 
> [ec-common.c:613:ec_child_select]
> 0-tier2-disperse-9: Insufficient available children for this request (have
> 0, need 4)
> [2018-10-05 23:48:44.736604] E [MSGID: 122037] 
> [ec-common.c:2040:ec_update_size_version_done]
> 0-tier2-disperse-9: Failed to update version and size [Input/output error]
> [2018-10-05 23:48:44.736604] E [MSGID: 122037] 
> [ec-common.c:2040:ec_update_size_version_done]
> 0-tier2-disperse-10: Failed to update version and size [Input/output error]
> [2018-10-05 23:48:44.736827] W [MSGID: 122040]
> [ec-common.c:1097:ec_prepare_update_cbk] 0-tier2-disperse-11: Failed to
> get size and version [Input/output error]
> [2018-10-05 23:48:44.736887] E [MSGID: 122034] 
> [ec-common.c:613:ec_child_select]
> 0-tier2-disperse-11: Insufficient available children for this request (have
> 0, need 4)
> [2018-10-05 23:48:44.736904] E [MSGID: 122037] 
> [ec-common.c:2040:ec_update_size_version_done]
> 0-tier2-disperse-11: Failed to update version and size [Input/output error]
> [2018-10-05 23:48:44.740337] W [MSGID: 122040]
> [ec-common.c:1097:ec_prepare_update_cbk] 0-tier2-disperse-6: Failed to
> get size and version [Input/output error]
> [2018-10-05 23:48:44.740381] E [MSGID: 122034] 
> [ec-common.c:613:ec_child_select]
> 0-tier2-disperse-6: Insufficient available children for this request (have
> 0, need 4)
> [2018-10-05 23:48:44.740394] E [MSGID: 122037] 
> [ec-common.c:2040:ec_update_size_version_done]
> 0-tier2-disperse-6: Failed to update version and size [Input/output error]
> [2018-10-05 23:48:50.066103] I [MSGID: 109081] 
> [dht-common.c:4379:dht_setxattr]
> 0-tier2-dht: fixing the layout of /
>
> In attachment you can find the first logs captured during the rebalance
> execution.
> In your opinion, is there a way to restore the gluster storage or all the
> data have been lost?
>
> Thank you in advance,
> Mauro
>
> 
>
>
>
> Il giorno 04 ott 2018, alle ore 15:31, Mauro Tridici <
> mauro.trid...@cmcc.it> ha scritto:
>
>
> Hi Nithya,
>
> thank you very much.
> This is the current “gluster volume info” output after removing bricks
> (and after peer detach command).
>
> [root@s01 ~]# gluster volume info
>
> Volume Name: tier2
> Type: Distributed-Disperse
> Volume ID: a28d88c5-3295-4e35-98d4-210b3af9358c
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 6 x (4 + 2) = 36
> Transport-type: tcp
> Bricks:
> Brick1: s01-stg:/gluster/mnt1/brick
> Brick2: s02-stg:/gluster/mnt1/brick
> Brick3: s03-stg:/gluster/mnt1/brick
> Brick4: s01-stg:/gluster/mnt2/brick
> Brick5: s02-stg:/gluster/mnt2/brick
> Brick6: s03-stg:/gluster/mnt2/brick
> Brick7: s01-stg:/gluster/mnt3/brick
> Brick8: s02-stg:/gluster/mnt3/brick
> Brick9: s03-stg:/gluster/mnt3/brick
> Brick10: s01-stg:/gluster/mnt4/brick
> Brick11: s02-stg:/gluster/mnt4/brick
> Brick12: s03-stg:/gluster/mnt4/brick
> Brick13: s01-stg:/gluster/mnt5/brick
> Brick14: s02-stg:/gluster/mnt5/brick
> Brick15: s03-stg:/gluster/mnt5/brick
> Brick16: s01-stg:/gluster/mnt6/brick
> Brick17: s02-stg:/gluster/mnt6/brick
> Brick18: s03-stg:/gluster/mnt6/brick
> Brick19: s01-stg:/gluster/mnt7/brick
> Brick20: s02-stg:/gluster/mnt7/brick
> Brick21: s03-stg:/gluster/mnt7/brick
> Brick22: s01-stg:/gluster/mnt8/brick
> Brick23: s02-stg:/gluster/mnt8/brick
> Brick24: s03-stg:/gluster/mnt8/brick
> Brick25: s01-stg:/gluster/mnt9/brick
> Brick26: s02-stg:/gluster/mnt9/brick
> Brick27: s03-stg:/gluster/mnt9/brick
> Brick28: s01-stg:/gluster/mnt10/brick
> Brick29: s02-stg:/gluster/mnt10/brick
> Brick30: s03-stg:/gluster/mnt10/brick
> Brick31: s01-stg:/gluster/mnt11/brick
> Brick32: s02-stg:/gluster/mnt11/brick
> Brick33: s03-stg:/gluster/mnt11/brick
> Brick34: s01-stg:/gluster/mnt12/brick
> Brick35: s02-stg:/gluster/mnt12/brick
> Brick36: s03-stg:/gluster/mnt12/brick
> Options Reconfigured:
> network.ping-timeout: 0
> features.scrub: Active
> features.bitrot: on
> features.inode-quota: on
> features.quota: on
> performance.client-io-threads: on
> cluster.min-free-disk: 10
> cluster.quorum-type: auto
> transport.address-family: inet
> nfs.disable: on
> server.event-threads: 4
> client.event-threads: 4
> cluster.lookup-optimize: on
> performance.readdir-ahead: on
> performance.parallel-readdir: off
> cluster.readdir-optimize: on
> features.cache-invalidation: on
> features.cache-invalidation-timeout: 600
> performance.stat-prefetch: on
> performance.cache-invalidation: on
> performance.md-cache-timeout: 600
> network.inode-lru-limit: 5
> perform

Re: [Gluster-users] Found anomalies in ganesha-gfapi.log

[Nithya Balachandran]

Is it always the same directory (the gfid in the message)? Please provide
more details as to what the client is doing and the log file if possible.

Thanks,
Nithya

On 4 October 2018 at 20:48, Renaud Fortier 
wrote:

> In the last 24 hour I’ve got this 65000 times (5 at night during the
> backup from only one client) I find it a little worrying even if it’s an
> INFO log level.
>
>
>
> *De :* Nithya Balachandran [mailto:nbala...@redhat.com]
> *Envoyé :* 4 octobre 2018 09:34
> *À :* Renaud Fortier 
> *Cc :* gluster-users@gluster.org
>
> *Objet :* Re: [Gluster-users] Found anomalies in ganesha-gfapi.log
>
>
>
>
>
>
>
> On 4 October 2018 at 17:39, Renaud Fortier 
> wrote:
>
> Yes !
>
> 2 clients using the same export connected to the same IP. Do you see
> something wrong with that ?
>
> Thank you
>
>
>
> Not necessarily wrong. This message shows up if DHT does not find a
> complete layout set on the directory when it does a lookup. In this case,
> it may be because the directory creation from the other client is still not
> complete.
>
>
>
>
>
>
>
>
>
> *De :* Jiffin Tony Thottan [mailto:jthot...@redhat.com]
> *Envoyé :* 4 octobre 2018 01:09
> *À :* Renaud Fortier ;
> gluster-users@gluster.org
> *Objet :* Re: [Gluster-users] Found anomalies in ganesha-gfapi.log
>
>
>
> Are u performing lookups or mkdir in parallel via two different clients ?
>
> --
>
> Jiffin
>
>
>
> On Friday 28 September 2018 08:13 PM, Renaud Fortier wrote:
>
> Hi,
>
> I have a lot of these lines in ganesha-gfapi.log. What is it and should I
> worried about it ?
>
> [2018-09-28 14:26:46.296375] I [MSGID: 109063]
> [dht-layout.c:693:dht_layout_normalize] 0-testing-dht: Found anomalies in
> (null) (gfid = 4efad4fd-fc7f-4c06-90e0-f882ca74b9a5). Holes=1 overlaps=0
>
>
>
> OS : Debian stretch
>
> Gluster : v4.1.5 type : replicated 3 briks
>
> Ganesha : 2.6.0
>
>
>
> Thank you
>
>
>
> ___
>
> Gluster-users mailing list
>
> Gluster-users@gluster.org
>
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
>
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Found anomalies in ganesha-gfapi.log

[Nithya Balachandran]

On 4 October 2018 at 17:39, Renaud Fortier 
wrote:

> Yes !
>
> 2 clients using the same export connected to the same IP. Do you see
> something wrong with that ?
>
> Thank you
>

Not necessarily wrong. This message shows up if DHT does not find a
complete layout set on the directory when it does a lookup. In this case,
it may be because the directory creation from the other client is still not
complete.




>
> *De :* Jiffin Tony Thottan [mailto:jthot...@redhat.com]
> *Envoyé :* 4 octobre 2018 01:09
> *À :* Renaud Fortier ;
> gluster-users@gluster.org
> *Objet :* Re: [Gluster-users] Found anomalies in ganesha-gfapi.log
>
>
>
> Are u performing lookups or mkdir in parallel via two different clients ?
>
> --
>
> Jiffin
>
>
>
> On Friday 28 September 2018 08:13 PM, Renaud Fortier wrote:
>
> Hi,
>
> I have a lot of these lines in ganesha-gfapi.log. What is it and should I
> worried about it ?
>
> [2018-09-28 14:26:46.296375] I [MSGID: 109063]
> [dht-layout.c:693:dht_layout_normalize] 0-testing-dht: Found anomalies in
> (null) (gfid = 4efad4fd-fc7f-4c06-90e0-f882ca74b9a5). Holes=1 overlaps=0
>
>
>
> OS : Debian stretch
>
> Gluster : v4.1.5 type : replicated 3 briks
>
> Ganesha : 2.6.0
>
>
>
> Thank you
>
>
>
> ___
>
> Gluster-users mailing list
>
> Gluster-users@gluster.org
>
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Rebalance failed on Distributed Disperse volume based on 3.12.14 version

[Nithya Balachandran]

Hi Mauro,


The files on s04 and s05 can be deleted safely as long as those bricks have
been removed from the volume and their brick processes are not running.


.glusterfs/indices/xattrop/xattrop-* are links to files that need to be healed.
.glusterfs/quarantine/stub-----0008 links
to files that bitrot (if enabled)says are corrupted. (none in this
case)



I will get back to you on s06. Can you please provide the output of
gluster volume info again?


Regards,
Nithya



On 4 October 2018 at 13:47, Mauro Tridici  wrote:

>
> Dear Ashish, Dear Nithya,
>
> I’m writing this message only to summarize and simplify the information
> about the "not migrated” files left on removed bricks on server s04, s05
> and s06.
> In attachment, you can find 3 files (1 file for each server) containing
> the “not migrated” files lists and related brick number.
>
> In particular:
>
>- s04 and s05 bricks contain only not migrated files in hidden
>directories “/gluster/mnt#/brick/.glusterfs" (I could delete them,
>doesn’t it?)
>- s06 bricks contain
>   - not migrated files in hidden directories “/gluster/mnt#/
>   brick/.glusterfs”;
>   - not migrated files with size equal to 0;
>   - not migrated files with size greater than 0.
>
>
> I think it was necessary to collect and summarize information to simplify
> your analysis.
> Thank you very much,
> Mauro
>
>
>
>
>
>
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Rebalance failed on Distributed Disperse volume based on 3.12.14 version

[Nithya Balachandran]

Hi Mauro,

It looks like all of these are actual files that were not migrated. Please
send me the rebalance logs for this node so I can check for any migration
errors.

As this is a disperse volume, copying the files to the mount will be
difficult. Ashish, how would we go about this?


Regards,
Nithya

On 3 October 2018 at 21:18, Mauro Tridici  wrote:

>
> Hi Nithya,
>
> in order to give an answer to your question as soon as possible, I just
> considered only the content of one brick of server s06 (in attachment you
> can find the content of /gluster/mnt1/brick).
>
> [root@s06 ~]# df -h
> File system  Dim. Usati Dispon. Uso% Montato su
> /dev/mapper/cl_s06-root  100G  2,1G 98G   3% /
> devtmpfs  32G 0 32G   0% /dev
> tmpfs 32G  4,0K 32G   1% /dev/shm
> tmpfs 32G  106M 32G   1% /run
> tmpfs 32G 0 32G   0% /sys/fs/cgroup
> /dev/mapper/cl_s06-var   100G  3,0G 97G   3% /var
> /dev/mapper/cl_s06-gluster   100G   33M100G   1% /gluster
> /dev/sda1   1014M  152M863M  15% /boot
> /dev/mapper/gluster_vgd-gluster_lvd  9,0T   12G9,0T   1% /gluster/mnt3
> /dev/mapper/gluster_vgg-gluster_lvg  9,0T   12G9,0T   1% /gluster/mnt6
> /dev/mapper/gluster_vgc-gluster_lvc  9,0T   12G9,0T   1% /gluster/mnt2
> /dev/mapper/gluster_vge-gluster_lve  9,0T   12G9,0T   1% /gluster/mnt4
> /dev/mapper/gluster_vgj-gluster_lvj  9,0T  1,4T7,7T  16% /gluster/mnt9
> */dev/mapper/gluster_vgb-gluster_lvb  9,0T   12G9,0T   1%
> /gluster/mnt1*
> /dev/mapper/gluster_vgh-gluster_lvh  9,0T  1,4T7,7T  16% /gluster/mnt7
> /dev/mapper/gluster_vgf-gluster_lvf  9,0T   12G9,0T   1% /gluster/mnt5
> /dev/mapper/gluster_vgi-gluster_lvi  9,0T  1,4T7,7T  16% /gluster/mnt8
> /dev/mapper/gluster_vgl-gluster_lvl  9,0T  1,4T7,7T  16%
> /gluster/mnt11
> /dev/mapper/gluster_vgk-gluster_lvk  9,0T  1,4T7,7T  16%
> /gluster/mnt10
> /dev/mapper/gluster_vgm-gluster_lvm  9,0T  1,4T7,7T  16%
> /gluster/mnt12
>
> The scenario is almost the same for all the bricks removed from server
> s04, s05 and s06.
> In the next hours, I will check every files on each removed bricks.
>
> So, if I understand, I can proceed with deletion of directories and files
> left on the bricks only if each file have T tag, right?
>
> Thank you in advance,
> Mauro
>
>
> Il giorno 03 ott 2018, alle ore 16:49, Nithya Balachandran <
> nbala...@redhat.com> ha scritto:
>
>
>
> On 1 October 2018 at 15:35, Mauro Tridici  wrote:
>
>> Good morning Ashish,
>>
>> your explanations are always very useful, thank you very much: I will
>> remember these suggestions for any future needs.
>> Anyway, during the week-end, the remove-brick procedures ended
>> successfully and we were able to free up all bricks defined on server s04,
>> s05 and 6 bricks of 12 on server s06.
>> So, we can say that, thanks to your suggestions, we are about to complete
>> this first phase (removing of all bricks defined on s04, s05 and s06
>> servers).
>>
>> I really appreciated your support.
>> Now I have a last question (I hope): after remove-brick commit I noticed
>> that some data remain on each brick (about 1.2GB of data).
>> Please, take a look to the “df-h_on_s04_s05_s06.txt”.
>> The situation is almost the same on all 3 servers mentioned above: a long
>> list of directories names and some files that are still on the brick, but
>> respective size is 0.
>>
>> Examples:
>>
>> a lot of empty directories on /gluster/mnt*/brick/.glusterfs
>>
>> 8 /gluster/mnt2/brick/.glusterfs/b7/1b
>> 0 /gluster/mnt2/brick/.glusterfs/b7/ee/b7ee94a5-a77c-4c02-
>> 85a5-085992840c83
>> 0 /gluster/mnt2/brick/.glusterfs/b7/ee/b7ee85d4-ce48-43a7-
>> a89a-69c728ee8273
>>
>> some empty files in directories in /gluster/mnt*/brick/*
>>
>> [root@s04 ~]# cd /gluster/mnt1/brick/
>> [root@s04 brick]# ls -l
>> totale 32
>> drwxr-xr-x 7 root  root  100 11 set 22.14 *archive_calypso*
>>
>> [root@s04 brick]# cd archive_calypso/
>> [root@s04 archive_calypso]# ll
>> totale 0
>> drwxr-x--- 3 root 5200 29 11 set 22.13 *ans002*
>> drwxr-x--- 3 5104 5100 32 11 set 22.14 *ans004*
>> drwxr-x--- 3 4506 4500 31 11 set 22.14 *ans006*
>> drwxr-x--- 3 4515 4500 28 11 set 22.14 *ans015*
>> drwxr-x--- 4 4321 4300 54 11 set 22.14 *ans021*
>> [root@s04 archive_calypso]# du -a *
>> 0 ans002/archive/ans002/HINDCASTS/RUN_ATMWANG_LANSENS/19810501
>> .0/echam5/echam_

Re: [Gluster-users] Rebalance failed on Distributed Disperse volume based on 3.12.14 version

[Nithya Balachandran]

On 1 October 2018 at 15:35, Mauro Tridici  wrote:

> Good morning Ashish,
>
> your explanations are always very useful, thank you very much: I will
> remember these suggestions for any future needs.
> Anyway, during the week-end, the remove-brick procedures ended
> successfully and we were able to free up all bricks defined on server s04,
> s05 and 6 bricks of 12 on server s06.
> So, we can say that, thanks to your suggestions, we are about to complete
> this first phase (removing of all bricks defined on s04, s05 and s06
> servers).
>
> I really appreciated your support.
> Now I have a last question (I hope): after remove-brick commit I noticed
> that some data remain on each brick (about 1.2GB of data).
> Please, take a look to the “df-h_on_s04_s05_s06.txt”.
> The situation is almost the same on all 3 servers mentioned above: a long
> list of directories names and some files that are still on the brick, but
> respective size is 0.
>
> Examples:
>
> a lot of empty directories on /gluster/mnt*/brick/.glusterfs
>
> 8 /gluster/mnt2/brick/.glusterfs/b7/1b
> 0 /gluster/mnt2/brick/.glusterfs/b7/ee/b7ee94a5-a77c-
> 4c02-85a5-085992840c83
> 0 /gluster/mnt2/brick/.glusterfs/b7/ee/b7ee85d4-ce48-
> 43a7-a89a-69c728ee8273
>
> some empty files in directories in /gluster/mnt*/brick/*
>
> [root@s04 ~]# cd /gluster/mnt1/brick/
> [root@s04 brick]# ls -l
> totale 32
> drwxr-xr-x 7 root  root  100 11 set 22.14 *archive_calypso*
>
> [root@s04 brick]# cd archive_calypso/
> [root@s04 archive_calypso]# ll
> totale 0
> drwxr-x--- 3 root 5200 29 11 set 22.13 *ans002*
> drwxr-x--- 3 5104 5100 32 11 set 22.14 *ans004*
> drwxr-x--- 3 4506 4500 31 11 set 22.14 *ans006*
> drwxr-x--- 3 4515 4500 28 11 set 22.14 *ans015*
> drwxr-x--- 4 4321 4300 54 11 set 22.14 *ans021*
> [root@s04 archive_calypso]# du -a *
> 0 ans002/archive/ans002/HINDCASTS/RUN_ATMWANG_LANSENS/
> 19810501.0/echam5/echam_sf006_198110.01.gz
> 0 ans002/archive/ans002/HINDCASTS/RUN_ATMWANG_LANSENS/19810501.0/echam5
> 0 ans002/archive/ans002/HINDCASTS/RUN_ATMWANG_LANSENS/19810501.0
> 0 ans002/archive/ans002/HINDCASTS/RUN_ATMWANG_LANSENS/
> 19810501.1/echam5/echam_sf006_198105.01.gz
> 0 ans002/archive/ans002/HINDCASTS/RUN_ATMWANG_LANSENS/
> 19810501.1/echam5/echam_sf006_198109.01.gz
> 8 ans002/archive/ans002/HINDCASTS/RUN_ATMWANG_LANSENS/19810501.1/echam5
>
> What we have to do with this data? Should I backup this “empty” dirs and
> files on a different storage before deleting them?
>

Hi Mauro,

Are you sure these files and directories are empty? Please provide the ls
-l output for the files. If they are 'T' files , they can be ignored.

Regards,
Nithya

>
> As soon as all the bricks will be empty, I plan to re-add the new bricks
> using the following commands:
>
> *gluster peer detach s04*
> *gluster peer detach s05*
> *gluster peer detach s06*
>
> *gluster peer probe s04*
> *gluster peer probe s05*
> *gluster peer probe s06*
>
> *gluster volume add-brick tier2 s04-stg:/gluster/mnt1/brick
> s05-stg:/gluster/mnt1/brick s06-stg:/gluster/mnt1/brick
> s04-stg:/gluster/mnt2/brick s05-stg:/gluster/mnt2/brick
> s06-stg:/gluster/mnt2/brick s04-stg:/gluster/mnt3/brick
> s05-stg:/gluster/mnt3/brick s06-stg:/gluster/mnt3/brick
> s04-stg:/gluster/mnt4/brick s05-stg:/gluster/mnt4/brick
> s06-stg:/gluster/mnt4/brick s04-stg:/gluster/mnt5/brick
> s05-stg:/gluster/mnt5/brick s06-stg:/gluster/mnt5/brick
> s04-stg:/gluster/mnt6/brick s05-stg:/gluster/mnt6/brick
> s06-stg:/gluster/mnt6/brick s04-stg:/gluster/mnt7/brick
> s05-stg:/gluster/mnt7/brick s06-stg:/gluster/mnt7/brick
> s04-stg:/gluster/mnt8/brick s05-stg:/gluster/mnt8/brick
> s06-stg:/gluster/mnt8/brick s04-stg:/gluster/mnt9/brick
> s05-stg:/gluster/mnt9/brick s06-stg:/gluster/mnt9/brick
> s04-stg:/gluster/mnt10/brick s05-stg:/gluster/mnt10/brick
> s06-stg:/gluster/mnt10/brick s04-stg:/gluster/mnt11/brick
> s05-stg:/gluster/mnt11/brick s06-stg:/gluster/mnt11/brick
> s04-stg:/gluster/mnt12/brick s05-stg:/gluster/mnt12/brick
> s06-stg:/gluster/mnt12/brick force*
>
> *gluster volume rebalance tier2 fix-layout start*
>
> *gluster volume rebalance tier2 start*
>
> From your point of view, are they the right commands to close this
> repairing task?
>
> Thank you very much for your help.
> Regards,
> Mauro
>
>
>
>
>
> Il giorno 01 ott 2018, alle ore 09:17, Ashish Pandey 
> ha scritto:
>
>
> Ohh!! It is because brick-multiplexing is "ON" on your setup. Not sure if
> it is by default ON for 3.12.14  or not.
>
> See "cluster.brick-multiplex: on" in gluster v  info
> If brick multiplexing is ON, you will see only one process running for all
> the bricks on a Node.
>
> So we have to do following step to  kill any one brick on a node.
>
> *Steps to kill a brick when multiplex is on  -*
>
> *Step - 1 *
> Find *unix domain_socket* of the process on a node.
> Run "ps -aef | grep glusterfsd" on a node. Example :
>
> This is on my machine when I have all the bricks on same machine
>
> [root@apandey glusterfs]# ps -aef | grep glusterfsd | grep -v 

Re: [Gluster-users] Rebalance failed on Distributed Disperse volume based on 3.12.14 version

[Nithya Balachandran]

Hi Mauro,


Please send the rebalance logs from s04-stg. I will take a look and get
back.


Regards,
Nithya

On 28 September 2018 at 16:21, Mauro Tridici  wrote:

>
> Hi Ashish,
>
> as I said in my previous message, we adopted the first approach you
> suggested (setting network.ping-timeout option to 0).
> This choice was due to the absence of empty brick to be used as indicated
> in the second approach.
>
> So, we launched remove-brick command on the first subvolume (V1, bricks
> 1,2,3,4,5,6 on server s04).
> Rebalance started moving the data across the other bricks, but, after
> about 3TB of moved data, rebalance speed slowed down and some transfer
> errors appeared in the rebalance.log of server s04.
> At this point, since remaining 1,8TB need to be moved in order to complete
> the step, we decided to stop the remove-brick execution and start it again
> (I hope it doesn’t stop again before complete the rebalance)
>
> Now rebalance is not moving data, it’s only scanning files (please, take a
> look to the following output)
>
> [root@s01 ~]# gluster volume remove-brick tier2
> s04-stg:/gluster/mnt1/brick s04-stg:/gluster/mnt2/brick
> s04-stg:/gluster/mnt3/brick s04-stg:/gluster/mnt4/brick
> s04-stg:/gluster/mnt5/brick s04-stg:/gluster/mnt6/brick status
> Node Rebalanced-files  size
> scanned  failures   skipped   status  run time in
> h:m:s
>-  ---   ---
> ---   ---   --- 
> --
>  s04-stg00Bytes
> 182008 0 0  in progress3:08:09
> Estimated time left for rebalance to complete :  442:45:06
>
> If I’m not wrong, remove-brick rebalances entire cluster each time it
> start.
> Is there a way to speed up this procedure? Do you have some other
> suggestion that, in this particular case, could be useful to reduce errors
> (I know that they are related to the current volume configuration) and
> improve rebalance performance avoiding to rebalance the entire cluster?
>
> Thank you in advance,
> Mauro
>
> Il giorno 27 set 2018, alle ore 13:14, Ashish Pandey 
> ha scritto:
>
>
> Yes, you can.
> If not me others may also reply.
>
> ---
> Ashish
>
> --
> *From: *"Mauro Tridici" 
> *To: *"Ashish Pandey" 
> *Cc: *"gluster-users" 
> *Sent: *Thursday, September 27, 2018 4:24:12 PM
> *Subject: *Re: [Gluster-users] Rebalance failed on Distributed Disperse
> volumebased on 3.12.14 version
>
>
> Dear Ashish,
>
> I can not thank you enough!
> Your procedure and description is very detailed.
> I think to follow the first approach after setting network.ping-timeout
> option to 0 (If I’m not wrong “0" means “infinite”...I noticed that this
> value reduced rebalance errors).
> After the fix I will set network.ping-timeout option to default value.
>
> Could I contact you again if I need some kind of suggestion?
>
> Thank you very much again.
> Have a good day,
> Mauro
>
>
> Il giorno 27 set 2018, alle ore 12:38, Ashish Pandey 
> ha scritto:
>
>
> Hi Mauro,
>
> We can divide the 36 newly added bricks into 6 set of 6 bricks each
> starting from brick37.
> That means, there are 6 ec subvolumes and we have to deal with one sub
> volume at a time.
> I have named it V1 to V6.
>
> Problem:
> Take the case of V1.
> The best configuration/setup would be to have all the 6 bricks of V1 on 6
> different nodes.
> However, in your case you have added 3 new nodes. So, at least we should
> have 2 bricks on 3 different newly added nodes.
> This way, in 4+2 EC configuration, even if one node goes down you will
> have 4 other bricks of that volume and the data on that volume would be
> accessible.
> In current setup if s04-stg goes down, you will loose all the data on V1
> and V2 as all the bricks will be down. We want to avoid and correct it.
>
> Now, we can have two approach to correct/modify this setup.
>
> *Approach 1*
> We have to remove all the newly added bricks in a set of 6 bricks. This
> will trigger re- balance and move whole data to other sub volumes.
> Repeat the above step and then once all the bricks are removed, add those
> bricks again in a set of 6 bricks, this time have 2 bricks from each of the
> 3 newly added Nodes.
>
> While this is a valid and working approach, I personally think that this
> will take long time and also require lot of movement of data.
>
> *Approach 2*
>
> In this approach we can use the heal process. We have to deal with all the
> volumes (V1 to V6) one by one. Following are the steps for V1-
>
> *Step 1 - *
> Use replace-brick command to move following bricks on *s05-stg* node *one
> by one (heal should be completed after every replace brick command)*
>
>
> *Brick39: s04-stg:/gluster/mnt3/brick to s05-stg/*
>
> *Brick40: s04-stg:/gluster/mnt4/brick to s05-stg/ free>*
>
> Command :
> gluster v 

Re: [Gluster-users] Gluster tier in progress after attach

[Nithya Balachandran]

Please check the rebalance log to see why those files were not migrated.

Regards,
Nithya

On 24 September 2018 at 21:21, Jeevan Patnaik  wrote:

> Hi,
>
> Yes, I did. The status shows all as completed.
>
> Regards,
> Jeevan.
>
> On Mon 24 Sep, 2018, 9:20 PM Nithya Balachandran, 
> wrote:
>
>>
>>
>> On 24 September 2018 at 21:01, Jeevan Patnaik 
>> wrote:
>>
>>> Hi,
>>>
>>> What do you mean by expanding volume?
>>>
>>> I mean adding new bricks to extend storage. It is mentioned that
>>> detaching tier is must for adding or removing bricks.
>>>
>>> What command did you use to detach the tier? And how did you check that
>>> the files still exist on the hot tier?
>>>
>>> gluster volume tier volume_name detach start
>>>
>>> I check the files directly from backend filesystem on disk and there are
>>> still left over files on all hot tier disks.
>>>
>>> Anyhow I proceeded with detach commit to see what would happen..so after
>>> commit, I can't access those files from client filesystems..so they're gone
>>> with the hot tier and were not moved to cold tier.
>>>
>>
>> Did you check that the detach tier operation had completed before running
>> detach commit?
>>
>>>
>>> And even if I try to attach the hot tier bricks again to bring back the
>>> files, attach won't work with existing files.
>>>
>>> So, it seems we shouldn't detach hot tier without making sure all files
>>> are moved to cold tier..but in my case, some files just won't move, even
>>> though they aren't being accessed anywhere.
>>>
>>
>>
>>
>>>
>>> There's a statement saying POSIX locks can cause files to be still left
>>> in hot tier and in that case, either the application causing posix lock has
>>> to be closed or files should be moved manually (but where?)... however, I
>>> don't think in my case there's any posix locks as the files are not
>>> accessed by any application except gluster itself.
>>>
>>> Regards,
>>> Jeevan.
>>>
>>>
>>> On Mon 24 Sep, 2018, 8:45 PM Nithya Balachandran, 
>>> wrote:
>>>
>>>> It has been a while since I worked on tier but here is what I remember:
>>>>
>>>>
>>>> On 24 September 2018 at 19:15, Jeevan Patnaik 
>>>> wrote:
>>>>
>>>>> Hi Nithya,
>>>>>
>>>>> No..the files are not being accessed. And tiering mode is in cache
>>>>> mode and what I understood is in cache mode every file is moved to hot 
>>>>> tier
>>>>> until it reaches low watermark, then it only promotes highly accessed 
>>>>> files.
>>>>>
>>>>> Not quite . All files will initially exist on the cold tier. Any new
>>>> files will be created on the hot tier. Any old files that are accessed will
>>>> be promoted ot the hot tier.
>>>>
>>>>
>>>> And do you mean everytime we expand volume,  we are virtually flushing
>>>>> the tier and it regenerates only when a files are accessed?
>>>>>
>>>>> What do you mean by expanding the volume?
>>>>
>>>>
>>>>> Also I see one more issue now, I hav generated another 1000 files
>>>>> after tiering is started and I see all 1000 files are now created on hot
>>>>> tier. Now, I detached again..in detach status it shows completed..but when
>>>>> I checked, around 260 files are still located on hot tier..those files are
>>>>> not being used anywhere..ca you tell why are they still not moved to cold
>>>>> tier and why the detach status is showing a wrong status.
>>>>>
>>>>
>>>> What command did you use to detach the tier? And how did you check that
>>>> the files still exist on the hot tier?
>>>>
>>>> Regards,
>>>> Nithya
>>>>
>>>>>
>>>>> I am using gluster 3.12.3 and server hosts include RHEL 6.7 and 7.2
>>>>> hosts also.
>>>>>
>>>>> Regards,
>>>>> Jeevan.
>>>>>
>>>>> On Mon 24 Sep, 2018, 7:00 PM Nithya Balachandran, 
>>>>> wrote:
>>>>>
>>>>>> Are those files being accessed? Tiering will only promote those that
>>>>>> have been accessed recently.
>>>>>&

Re: [Gluster-users] Gluster tier in progress after attach

[Nithya Balachandran]

On 24 September 2018 at 21:01, Jeevan Patnaik  wrote:

> Hi,
>
> What do you mean by expanding volume?
>
> I mean adding new bricks to extend storage. It is mentioned that detaching
> tier is must for adding or removing bricks.
>
> What command did you use to detach the tier? And how did you check that
> the files still exist on the hot tier?
>
> gluster volume tier volume_name detach start
>
> I check the files directly from backend filesystem on disk and there are
> still left over files on all hot tier disks.
>
> Anyhow I proceeded with detach commit to see what would happen..so after
> commit, I can't access those files from client filesystems..so they're gone
> with the hot tier and were not moved to cold tier.
>

Did you check that the detach tier operation had completed before running
detach commit?

>
> And even if I try to attach the hot tier bricks again to bring back the
> files, attach won't work with existing files.
>
> So, it seems we shouldn't detach hot tier without making sure all files
> are moved to cold tier..but in my case, some files just won't move, even
> though they aren't being accessed anywhere.
>



>
> There's a statement saying POSIX locks can cause files to be still left in
> hot tier and in that case, either the application causing posix lock has to
> be closed or files should be moved manually (but where?)... however, I
> don't think in my case there's any posix locks as the files are not
> accessed by any application except gluster itself.
>
> Regards,
> Jeevan.
>
>
> On Mon 24 Sep, 2018, 8:45 PM Nithya Balachandran, 
> wrote:
>
>> It has been a while since I worked on tier but here is what I remember:
>>
>>
>> On 24 September 2018 at 19:15, Jeevan Patnaik 
>> wrote:
>>
>>> Hi Nithya,
>>>
>>> No..the files are not being accessed. And tiering mode is in cache mode
>>> and what I understood is in cache mode every file is moved to hot tier
>>> until it reaches low watermark, then it only promotes highly accessed files.
>>>
>>> Not quite . All files will initially exist on the cold tier. Any new
>> files will be created on the hot tier. Any old files that are accessed will
>> be promoted ot the hot tier.
>>
>>
>> And do you mean everytime we expand volume,  we are virtually flushing
>>> the tier and it regenerates only when a files are accessed?
>>>
>>> What do you mean by expanding the volume?
>>
>>
>>> Also I see one more issue now, I hav generated another 1000 files after
>>> tiering is started and I see all 1000 files are now created on hot tier.
>>> Now, I detached again..in detach status it shows completed..but when I
>>> checked, around 260 files are still located on hot tier..those files are
>>> not being used anywhere..ca you tell why are they still not moved to cold
>>> tier and why the detach status is showing a wrong status.
>>>
>>
>> What command did you use to detach the tier? And how did you check that
>> the files still exist on the hot tier?
>>
>> Regards,
>> Nithya
>>
>>>
>>> I am using gluster 3.12.3 and server hosts include RHEL 6.7 and 7.2
>>> hosts also.
>>>
>>> Regards,
>>> Jeevan.
>>>
>>> On Mon 24 Sep, 2018, 7:00 PM Nithya Balachandran, 
>>> wrote:
>>>
>>>> Are those files being accessed? Tiering will only promote those that
>>>> have been accessed recently.
>>>>
>>>> Regards,
>>>> Nithya
>>>>
>>>> On 24 September 2018 at 18:32, Jeevan Patnaik 
>>>> wrote:
>>>>
>>>>> I see it still not promoting any files. Do we need to run any command
>>>>> to force the movement?
>>>>>
>>>>> This would be an issue if we need to expand disk as then we also need
>>>>> to detach existing tier and attach again and expect the data to be 
>>>>> promoted
>>>>> to the hot tier.
>>>>>
>>>>> Regards,
>>>>> Jeevan.
>>>>>
>>>>> On Mon 24 Sep, 2018, 6:17 PM Jeevan Patnaik, 
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I have created a 18 disk replica 3 volume and created 1000 random
>>>>>> files each of 10M.
>>>>>>
>>>>>> Later I have attached 15 disk replica 3 tier with each hot tier disk
>>>>>> coming from cold tier host, except on 3 (as we don't have hot tier disk)
>>>>>>
>>>>>> After attach, I see no files are promoted at even after 15 minutes.
>>>>>> Status is showing as In progress.
>>>>>>
>>>>>> Is it always this much slow? For 10G data, 15 minutes is too much
>>>>>> Also will the data try to be promoted to a hot tier coming from the
>>>>>> same host to reduce the time taken to move the files?
>>>>>>
>>>>>> Cold tier is made of RAID supported SSD and hot tier is made of NVMe
>>>>>> SSDs.
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>> Jeevan.
>>>>>>
>>>>>
>>>>> ___
>>>>> Gluster-users mailing list
>>>>> Gluster-users@gluster.org
>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>
>>>>
>>>>
>>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster tier in progress after attach

[Nithya Balachandran]

It has been a while since I worked on tier but here is what I remember:


On 24 September 2018 at 19:15, Jeevan Patnaik  wrote:

> Hi Nithya,
>
> No..the files are not being accessed. And tiering mode is in cache mode
> and what I understood is in cache mode every file is moved to hot tier
> until it reaches low watermark, then it only promotes highly accessed files.
>
> Not quite . All files will initially exist on the cold tier. Any new files
will be created on the hot tier. Any old files that are accessed will be
promoted ot the hot tier.


And do you mean everytime we expand volume,  we are virtually flushing the
> tier and it regenerates only when a files are accessed?
>
> What do you mean by expanding the volume?


> Also I see one more issue now, I hav generated another 1000 files after
> tiering is started and I see all 1000 files are now created on hot tier.
> Now, I detached again..in detach status it shows completed..but when I
> checked, around 260 files are still located on hot tier..those files are
> not being used anywhere..ca you tell why are they still not moved to cold
> tier and why the detach status is showing a wrong status.
>

What command did you use to detach the tier? And how did you check that the
files still exist on the hot tier?

Regards,
Nithya

>
> I am using gluster 3.12.3 and server hosts include RHEL 6.7 and 7.2 hosts
> also.
>
> Regards,
> Jeevan.
>
> On Mon 24 Sep, 2018, 7:00 PM Nithya Balachandran, 
> wrote:
>
>> Are those files being accessed? Tiering will only promote those that have
>> been accessed recently.
>>
>> Regards,
>> Nithya
>>
>> On 24 September 2018 at 18:32, Jeevan Patnaik 
>> wrote:
>>
>>> I see it still not promoting any files. Do we need to run any command to
>>> force the movement?
>>>
>>> This would be an issue if we need to expand disk as then we also need to
>>> detach existing tier and attach again and expect the data to be promoted to
>>> the hot tier.
>>>
>>> Regards,
>>> Jeevan.
>>>
>>> On Mon 24 Sep, 2018, 6:17 PM Jeevan Patnaik, 
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I have created a 18 disk replica 3 volume and created 1000 random files
>>>> each of 10M.
>>>>
>>>> Later I have attached 15 disk replica 3 tier with each hot tier disk
>>>> coming from cold tier host, except on 3 (as we don't have hot tier disk)
>>>>
>>>> After attach, I see no files are promoted at even after 15 minutes.
>>>> Status is showing as In progress.
>>>>
>>>> Is it always this much slow? For 10G data, 15 minutes is too much
>>>> Also will the data try to be promoted to a hot tier coming from the
>>>> same host to reduce the time taken to move the files?
>>>>
>>>> Cold tier is made of RAID supported SSD and hot tier is made of NVMe
>>>> SSDs.
>>>>
>>>>
>>>> Regards,
>>>> Jeevan.
>>>>
>>>
>>> ___
>>> Gluster-users mailing list
>>> Gluster-users@gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>
>>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster tier in progress after attach

[Nithya Balachandran]

Are those files being accessed? Tiering will only promote those that have
been accessed recently.

Regards,
Nithya

On 24 September 2018 at 18:32, Jeevan Patnaik  wrote:

> I see it still not promoting any files. Do we need to run any command to
> force the movement?
>
> This would be an issue if we need to expand disk as then we also need to
> detach existing tier and attach again and expect the data to be promoted to
> the hot tier.
>
> Regards,
> Jeevan.
>
> On Mon 24 Sep, 2018, 6:17 PM Jeevan Patnaik,  wrote:
>
>> Hi,
>>
>> I have created a 18 disk replica 3 volume and created 1000 random files
>> each of 10M.
>>
>> Later I have attached 15 disk replica 3 tier with each hot tier disk
>> coming from cold tier host, except on 3 (as we don't have hot tier disk)
>>
>> After attach, I see no files are promoted at even after 15 minutes.
>> Status is showing as In progress.
>>
>> Is it always this much slow? For 10G data, 15 minutes is too much
>> Also will the data try to be promoted to a hot tier coming from the same
>> host to reduce the time taken to move the files?
>>
>> Cold tier is made of RAID supported SSD and hot tier is made of NVMe SSDs.
>>
>>
>> Regards,
>> Jeevan.
>>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Failures during rebalance on gluster distributed disperse volume

[Nithya Balachandran]

Hi Mauro,

Please stop the rebalance before you upgrade.
Thanks,
Nithya

On 15 September 2018 at 22:55, Mauro Tridici  wrote:

>
> Hi Sunil,
>
> many thanks to you too.
> I will follow your suggestions and the guide for upgrading to 3.12
>
> Crossing fingers :-)
> Regards,
> Mauro
>
> Il giorno 15 set 2018, alle ore 11:57, Sunil Kumar Heggodu Gopala Acharya <
> shegg...@redhat.com> ha scritto:
>
> Hi Mauro,
>
> As Nithya highlighted FALLOCATE support for EC volumes went in 3.11 as
> part of https://bugzilla.redhat.com/show_bug.cgi?id=1454686. Hence,
> upgrading to 3.12 as suggested before would be a right move.
>
> Here is the documentation for upgrading to 3.12:
> https://docs.gluster.org/en/latest/Upgrade-Guide/upgrade_to_3.12/
>
> Regards,
> Sunil kumar Acharya
>
> Senior Software Engineer
> Red Hat
>
> <https://www.redhat.com/>
>
> T: +91-8067935170 <http://redhatemailsignature-marketing.itos.redhat.com/>
>
> <https://red.ht/sig>
> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
>
>
> On Sat, Sep 15, 2018 at 3:42 AM, Mauro Tridici 
> wrote:
>
>>
>> Hi Nithya,
>>
>> thank you very much for your answer.
>> I will wait for @Sunil opinion too before starting the upgrade procedure.
>>
>> Since it will be the first upgrade of our Gluster cluster, I would like
>> to know if it could be a “virtually dangerous" procedure and if it will be
>> the risk of losing data :-)
>> Unfortunately, I can’t do a preventive copy of the volume data in another
>> location.
>> If it is possible, could you please illustrate the right steps needed to
>> complete the upgrade procedure from the 3.10.5 to the 3.12 version?
>>
>> Thank you again, Nithya.
>> Thank you to all of you for the help!
>>
>> Regards,
>> Mauro
>>
>> Il giorno 14 set 2018, alle ore 16:59, Nithya Balachandran <
>> nbala...@redhat.com> ha scritto:
>>
>> Hi Mauro,
>>
>>
>> The rebalance code started using fallocate in 3.10.5 (
>> https://bugzilla.redhat.com/show_bug.cgi?id=1473132) which works fine on
>> replicated volumes. However, we neglected to test this with EC volumes on
>> 3.10. Once we discovered the issue, the EC fallocate implementation was
>> made available in 3.11.
>>
>> At this point, I'm afraid the only option I see is to upgrade to at least
>> 3.12.
>>
>> @Sunil, do you have anything to add?
>>
>> Regards,
>> Nithya
>>
>> On 13 September 2018 at 18:34, Mauro Tridici 
>> wrote:
>>
>>>
>>> Hi Nithya,
>>>
>>> thank you for involving EC group.
>>> I will wait for your suggestions.
>>>
>>> Regards,
>>> Mauro
>>>
>>> Il giorno 13 set 2018, alle ore 13:38, Nithya Balachandran <
>>> nbala...@redhat.com> ha scritto:
>>>
>>> This looks like an issue because rebalance switched to using fallocate
>>> which EC did not have implemented at that point.
>>>
>>> @Pranith, @Ashish, which version of gluster had support for fallocate in
>>> EC?
>>>
>>>
>>> Regards,
>>> Nithya
>>>
>>> On 12 September 2018 at 19:24, Mauro Tridici 
>>> wrote:
>>>
>>>> Dear All,
>>>>
>>>> I recently added 3 servers (each one with 12 bricks) to an existing
>>>> Gluster Distributed Disperse Volume.
>>>> Volume extension has been completed without error and I already
>>>> executed the rebalance procedure with fix-layout option with no problem.
>>>> I just launched the rebalance procedure without fix-layout option, but,
>>>> as you can see in the output below, I noticed that some failures have been
>>>> detected.
>>>>
>>>> [root@s01 glusterfs]# gluster v rebalance tier2 status
>>>> Node Rebalanced-files  size
>>>>   scanned  failures   skipped   status  run time in
>>>> h:m:s
>>>>-  ---   ---
>>>>   ---   ---   --- 
>>>> --
>>>>localhost71176 3.2MB
>>>>   2137557   1530391  8128  in progress
>>>> 13:59:05
>>>>  s02-stg00Bytes
>>>> 0 0 0 

Re: [Gluster-users] Failures during rebalance on gluster distributed disperse volume

[Nithya Balachandran]

Hi Mauro,


The rebalance code started using fallocate in 3.10.5 (
https://bugzilla.redhat.com/show_bug.cgi?id=1473132) which works fine on
replicated volumes. However, we neglected to test this with EC volumes on
3.10. Once we discovered the issue, the EC fallocate implementation was
made available in 3.11.

At this point, I'm afraid the only option I see is to upgrade to at least
3.12.

@Sunil, do you have anything to add?

Regards,
Nithya

On 13 September 2018 at 18:34, Mauro Tridici  wrote:

>
> Hi Nithya,
>
> thank you for involving EC group.
> I will wait for your suggestions.
>
> Regards,
> Mauro
>
> Il giorno 13 set 2018, alle ore 13:38, Nithya Balachandran <
> nbala...@redhat.com> ha scritto:
>
> This looks like an issue because rebalance switched to using fallocate
> which EC did not have implemented at that point.
>
> @Pranith, @Ashish, which version of gluster had support for fallocate in
> EC?
>
>
> Regards,
> Nithya
>
> On 12 September 2018 at 19:24, Mauro Tridici 
> wrote:
>
>> Dear All,
>>
>> I recently added 3 servers (each one with 12 bricks) to an existing
>> Gluster Distributed Disperse Volume.
>> Volume extension has been completed without error and I already executed
>> the rebalance procedure with fix-layout option with no problem.
>> I just launched the rebalance procedure without fix-layout option, but,
>> as you can see in the output below, I noticed that some failures have been
>> detected.
>>
>> [root@s01 glusterfs]# gluster v rebalance tier2 status
>> Node Rebalanced-files  size
>> scanned  failures   skipped   status  run time in
>> h:m:s
>>-  ---   ---
>> ---   ---   --- 
>> --
>>localhost71176 3.2MB
>> 2137557   1530391  8128  in progress   13:59:05
>>  s02-stg00Bytes
>>   0 0 0completed   11:53:28
>>  s03-stg00Bytes
>>   0 0 0completed   11:53:32
>>  s04-stg00Bytes
>>   0 0 0completed0:00:06
>>  s05-stg   150Bytes
>>   17055 018completed   10:48:01
>>  s06-stg00Bytes
>>   0 0 0completed0:00:06
>> Estimated time left for rebalance to complete :0:46:53
>> volume rebalance: tier2: success
>>
>> In the volume rebalance log file, I detected a lot of error messages
>> similar to the following ones:
>>
>> [2018-09-12 13:15:50.756703] E [MSGID: 0] 
>> [dht-rebalance.c:1696:dht_migrate_file]
>> 0-tier2-dht: Create dst failed on - tier2-disperse-6 for file -
>> /CSP/sp1/CESM/archive/sps_200508_003/atm/hist/postproc/sps_
>> 200508_003.cam.h0.2005-12_grid.nc
>> [2018-09-12 13:15:50.757025] E [MSGID: 109023]
>> [dht-rebalance.c:2733:gf_defrag_migrate_single_file] 0-tier2-dht:
>> migrate-data failed for /CSP/sp1/CESM/archive/sps_2005
>> 08_003/atm/hist/postproc/sps_200508_003.cam.h0.2005-12_grid.nc
>> [2018-09-12 13:15:50.759183] E [MSGID: 109023]
>> [dht-rebalance.c:844:__dht_rebalance_create_dst_file] 0-tier2-dht:
>> fallocate failed for /CSP/sp1/CESM/archive/sps_2005
>> 08_003/atm/hist/postproc/sps_200508_003.cam.h0.2005-09_grid.nc on
>> tier2-disperse-9 (Operation not supported)
>> [2018-09-12 13:15:50.759206] E [MSGID: 0] 
>> [dht-rebalance.c:1696:dht_migrate_file]
>> 0-tier2-dht: Create dst failed on - tier2-disperse-9 for file -
>> /CSP/sp1/CESM/archive/sps_200508_003/atm/hist/postproc/sps_
>> 200508_003.cam.h0.2005-09_grid.nc
>> [2018-09-12 13:15:50.759536] E [MSGID: 109023]
>> [dht-rebalance.c:2733:gf_defrag_migrate_single_file] 0-tier2-dht:
>> migrate-data failed for /CSP/sp1/CESM/archive/sps_2005
>> 08_003/atm/hist/postproc/sps_200508_003.cam.h0.2005-09_grid.nc
>> [2018-09-12 13:15:50.777219] E [MSGID: 109023]
>> [dht-rebalance.c:844:__dht_rebalance_create_dst_file] 0-tier2-dht:
>> fallocate failed for /CSP/sp1/CESM/archive/sps_2005
>> 08_003/atm/hist/postproc/sps_200508_003.cam.h0.2006-01_grid.nc on
>> tier2-disperse-10 (Operation not supported)
&g

Re: [Gluster-users] Failures during rebalance on gluster distributed disperse volume

[Nithya Balachandran]

This looks like an issue because rebalance switched to using fallocate
which EC did not have implemented at that point.

@Pranith, @Ashish, which version of gluster had support for fallocate in EC?


Regards,
Nithya

On 12 September 2018 at 19:24, Mauro Tridici  wrote:

> Dear All,
>
> I recently added 3 servers (each one with 12 bricks) to an existing
> Gluster Distributed Disperse Volume.
> Volume extension has been completed without error and I already executed
> the rebalance procedure with fix-layout option with no problem.
> I just launched the rebalance procedure without fix-layout option, but, as
> you can see in the output below, I noticed that some failures have been
> detected.
>
> [root@s01 glusterfs]# gluster v rebalance tier2 status
> Node Rebalanced-files  size
> scanned  failures   skipped   status  run time in
> h:m:s
>-  ---   ---
> ---   ---   --- 
> --
>localhost71176 3.2MB
> 2137557   1530391  8128  in progress   13:59:05
>  s02-stg00Bytes
>   0 0 0completed   11:53:28
>  s03-stg00Bytes
>   0 0 0completed   11:53:32
>  s04-stg00Bytes
>   0 0 0completed0:00:06
>  s05-stg   150Bytes
>   17055 018completed   10:48:01
>  s06-stg00Bytes
>   0 0 0completed0:00:06
> Estimated time left for rebalance to complete :0:46:53
> volume rebalance: tier2: success
>
> In the volume rebalance log file, I detected a lot of error messages
> similar to the following ones:
>
> [2018-09-12 13:15:50.756703] E [MSGID: 0] 
> [dht-rebalance.c:1696:dht_migrate_file]
> 0-tier2-dht: Create dst failed on - tier2-disperse-6 for file -
> /CSP/sp1/CESM/archive/sps_200508_003/atm/hist/postproc/s
> ps_200508_003.cam.h0.2005-12_grid.nc
> [2018-09-12 13:15:50.757025] E [MSGID: 109023] 
> [dht-rebalance.c:2733:gf_defrag_migrate_single_file]
> 0-tier2-dht: migrate-data failed for /CSP/sp1/CESM/archive/sps_
> 200508_003/atm/hist/postproc/sps_200508_003.cam.h0.2005-12_grid.nc
> [2018-09-12 13:15:50.759183] E [MSGID: 109023] 
> [dht-rebalance.c:844:__dht_rebalance_create_dst_file]
> 0-tier2-dht: fallocate failed for /CSP/sp1/CESM/archive/sps_
> 200508_003/atm/hist/postproc/sps_200508_003.cam.h0.2005-09_grid.nc on
> tier2-disperse-9 (Operation not supported)
> [2018-09-12 13:15:50.759206] E [MSGID: 0] 
> [dht-rebalance.c:1696:dht_migrate_file]
> 0-tier2-dht: Create dst failed on - tier2-disperse-9 for file -
> /CSP/sp1/CESM/archive/sps_200508_003/atm/hist/postproc/s
> ps_200508_003.cam.h0.2005-09_grid.nc
> [2018-09-12 13:15:50.759536] E [MSGID: 109023] 
> [dht-rebalance.c:2733:gf_defrag_migrate_single_file]
> 0-tier2-dht: migrate-data failed for /CSP/sp1/CESM/archive/sps_
> 200508_003/atm/hist/postproc/sps_200508_003.cam.h0.2005-09_grid.nc
> [2018-09-12 13:15:50.777219] E [MSGID: 109023] 
> [dht-rebalance.c:844:__dht_rebalance_create_dst_file]
> 0-tier2-dht: fallocate failed for /CSP/sp1/CESM/archive/sps_
> 200508_003/atm/hist/postproc/sps_200508_003.cam.h0.2006-01_grid.nc on
> tier2-disperse-10 (Operation not supported)
> [2018-09-12 13:15:50.777241] E [MSGID: 0] 
> [dht-rebalance.c:1696:dht_migrate_file]
> 0-tier2-dht: Create dst failed on - tier2-disperse-10 for file -
> /CSP/sp1/CESM/archive/sps_200508_003/atm/hist/postproc/s
> ps_200508_003.cam.h0.2006-01_grid.nc
> [2018-09-12 13:15:50.777676] E [MSGID: 109023] 
> [dht-rebalance.c:2733:gf_defrag_migrate_single_file]
> 0-tier2-dht: migrate-data failed for /CSP/sp1/CESM/archive/sps_
> 200508_003/atm/hist/postproc/sps_200508_003.cam.h0.2006-01_grid.nc
>
> Could you please help me to understand what is happening and how to solve
> it?
>
> Our Gluster implementation is based on Gluster v.3.10.5
>
> Thank you in advance,
> Mauro
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] mv lost some files ?

[Nithya Balachandran]

On 5 September 2018 at 14:10, Nithya Balachandran 
wrote:

>
>
> On 5 September 2018 at 14:02, Nithya Balachandran 
> wrote:
>
>> Hi,
>>
>> Please try turning off cluster.readdir-optimize and see it if helps.
>>
>
> You can also try turning off parallel-readdir.
>


Please ignore this - it is not enabled on your volume.

>
>
>> If not, please send us the client mount logs and a tcpdump of when the
>> *ls* is performed from the client.  Use the following to capture the
>> dump:
>>
>> tcpdump -i any -s 0 -w /var/tmp/dirls.pcap tcp and not port 22
>>
>>
>>
>> Thanks,
>> Nithya
>>
>>
>>>
>>> Raghavendra Gowdappa  于2018年9月5日周三 下午12:40写道：
>>>
>>>>
>>>>
>>>>
>>>>>
>>>>>>
>>>>>> On Tue, Sep 4, 2018 at 5:28 PM, yu sun  wrote:
>>>>>>
>>>>>>> Hi all:
>>>>>>>
>>>>>>> I have a replicated volume project2 with info:
>>>>>>> Volume Name: project2 Type: Distributed-Replicate Volume ID:
>>>>>>> 60175b8e-de0e-4409-81ae-7bb5eb5cacbf Status: Started Snapshot
>>>>>>> Count: 0 Number of Bricks: 84 x 2 = 168 Transport-type: tcp Bricks: 
>>>>>>> Brick1:
>>>>>>> node20:/data2/bricks/project2 Brick2: node21:/data2/bricks/project2 
>>>>>>> Brick3:
>>>>>>> node22:/data2/bricks/project2 Brick4: node23:/data2/bricks/project2 
>>>>>>> Brick5:
>>>>>>> node24:/data2/bricks/project2 Brick6: node25:/data2/bricks/project2 
>>>>>>> Brick7:
>>>>>>> node26:/data2/bricks/project2 Brick8: node27:/data2/bricks/project2 
>>>>>>> Brick9:
>>>>>>> node28:/data2/bricks/project2 Brick10: node29:/data2/bricks/project2
>>>>>>> Brick11: node30:/data2/bricks/project2 Brick12:
>>>>>>> node31:/data2/bricks/project2 Brick13: node32:/data2/bricks/project2
>>>>>>> Brick14: node33:/data2/bricks/project2 Brick15:
>>>>>>> node20:/data3/bricks/project2 Brick16: node21:/data3/bricks/project2
>>>>>>> Brick17: node22:/data3/bricks/project2 Brick18:
>>>>>>> node23:/data3/bricks/project2 Brick19: node24:/data3/bricks/project2
>>>>>>> Brick20: node25:/data3/bricks/project2 Brick21:
>>>>>>> node26:/data3/bricks/project2 Brick22: node27:/data3/bricks/project2
>>>>>>> Brick23: node28:/data3/bricks/project2 Brick24:
>>>>>>> node29:/data3/bricks/project2 Brick25: node30:/data3/bricks/project2
>>>>>>> Brick26: node31:/data3/bricks/project2 Brick27:
>>>>>>> node32:/data3/bricks/project2 Brick28: node33:/data3/bricks/project2
>>>>>>> Brick29: node20:/data4/bricks/project2 Brick30:
>>>>>>> node21:/data4/bricks/project2 Brick31: node22:/data4/bricks/project2
>>>>>>> Brick32: node23:/data4/bricks/project2 Brick33:
>>>>>>> node24:/data4/bricks/project2 Brick34: node25:/data4/bricks/project2
>>>>>>> Brick35: node26:/data4/bricks/project2 Brick36:
>>>>>>> node27:/data4/bricks/project2 Brick37: node28:/data4/bricks/project2
>>>>>>> Brick38: node29:/data4/bricks/project2 Brick39:
>>>>>>> node30:/data4/bricks/project2 Brick40: node31:/data4/bricks/project2
>>>>>>> Brick41: node32:/data4/bricks/project2 Brick42:
>>>>>>> node33:/data4/bricks/project2 Brick43: node20:/data5/bricks/project2
>>>>>>> Brick44: node21:/data5/bricks/project2 Brick45:
>>>>>>> node22:/data5/bricks/project2 Brick46: node23:/data5/bricks/project2
>>>>>>> Brick47: node24:/data5/bricks/project2 Brick48:
>>>>>>> node25:/data5/bricks/project2 Brick49: node26:/data5/bricks/project2
>>>>>>> Brick50: node27:/data5/bricks/project2 Brick51:
>>>>>>> node28:/data5/bricks/project2 Brick52: node29:/data5/bricks/project2
>>>>>>> Brick53: node30:/data5/bricks/project2 Brick54:
>>>>>>> node31:/data5/bricks/project2 Brick55: node32:/data5/bricks/project2
>>>>>>> Brick56: node33:/data5/bricks/project2 Brick57:
>>>>>>> node20:/data6/bricks/project2 Brick58: node21:/data6/bricks/project2
>>>>>>> Brick59: node22:/data6/bricks/project2 Brick60:
>>>>>>> node

Re: [Gluster-users] mv lost some files ?

[Nithya Balachandran]

On 5 September 2018 at 14:02, Nithya Balachandran 
wrote:

> Hi,
>
> Please try turning off cluster.readdir-optimize and see it if helps.
>

You can also try turning off parallel-readdir.


> If not, please send us the client mount logs and a tcpdump of when the
> *ls* is performed from the client.  Use the following to capture the dump:
>
> tcpdump -i any -s 0 -w /var/tmp/dirls.pcap tcp and not port 22
>
>
>
> Thanks,
> Nithya
>
>
>>
>> Raghavendra Gowdappa  于2018年9月5日周三 下午12:40写道：
>>
>>>
>>>
>>>
>>>>
>>>>>
>>>>> On Tue, Sep 4, 2018 at 5:28 PM, yu sun  wrote:
>>>>>
>>>>>> Hi all:
>>>>>>
>>>>>> I have a replicated volume project2 with info:
>>>>>> Volume Name: project2 Type: Distributed-Replicate Volume ID:
>>>>>> 60175b8e-de0e-4409-81ae-7bb5eb5cacbf Status: Started Snapshot Count:
>>>>>> 0 Number of Bricks: 84 x 2 = 168 Transport-type: tcp Bricks: Brick1:
>>>>>> node20:/data2/bricks/project2 Brick2: node21:/data2/bricks/project2 
>>>>>> Brick3:
>>>>>> node22:/data2/bricks/project2 Brick4: node23:/data2/bricks/project2 
>>>>>> Brick5:
>>>>>> node24:/data2/bricks/project2 Brick6: node25:/data2/bricks/project2 
>>>>>> Brick7:
>>>>>> node26:/data2/bricks/project2 Brick8: node27:/data2/bricks/project2 
>>>>>> Brick9:
>>>>>> node28:/data2/bricks/project2 Brick10: node29:/data2/bricks/project2
>>>>>> Brick11: node30:/data2/bricks/project2 Brick12:
>>>>>> node31:/data2/bricks/project2 Brick13: node32:/data2/bricks/project2
>>>>>> Brick14: node33:/data2/bricks/project2 Brick15:
>>>>>> node20:/data3/bricks/project2 Brick16: node21:/data3/bricks/project2
>>>>>> Brick17: node22:/data3/bricks/project2 Brick18:
>>>>>> node23:/data3/bricks/project2 Brick19: node24:/data3/bricks/project2
>>>>>> Brick20: node25:/data3/bricks/project2 Brick21:
>>>>>> node26:/data3/bricks/project2 Brick22: node27:/data3/bricks/project2
>>>>>> Brick23: node28:/data3/bricks/project2 Brick24:
>>>>>> node29:/data3/bricks/project2 Brick25: node30:/data3/bricks/project2
>>>>>> Brick26: node31:/data3/bricks/project2 Brick27:
>>>>>> node32:/data3/bricks/project2 Brick28: node33:/data3/bricks/project2
>>>>>> Brick29: node20:/data4/bricks/project2 Brick30:
>>>>>> node21:/data4/bricks/project2 Brick31: node22:/data4/bricks/project2
>>>>>> Brick32: node23:/data4/bricks/project2 Brick33:
>>>>>> node24:/data4/bricks/project2 Brick34: node25:/data4/bricks/project2
>>>>>> Brick35: node26:/data4/bricks/project2 Brick36:
>>>>>> node27:/data4/bricks/project2 Brick37: node28:/data4/bricks/project2
>>>>>> Brick38: node29:/data4/bricks/project2 Brick39:
>>>>>> node30:/data4/bricks/project2 Brick40: node31:/data4/bricks/project2
>>>>>> Brick41: node32:/data4/bricks/project2 Brick42:
>>>>>> node33:/data4/bricks/project2 Brick43: node20:/data5/bricks/project2
>>>>>> Brick44: node21:/data5/bricks/project2 Brick45:
>>>>>> node22:/data5/bricks/project2 Brick46: node23:/data5/bricks/project2
>>>>>> Brick47: node24:/data5/bricks/project2 Brick48:
>>>>>> node25:/data5/bricks/project2 Brick49: node26:/data5/bricks/project2
>>>>>> Brick50: node27:/data5/bricks/project2 Brick51:
>>>>>> node28:/data5/bricks/project2 Brick52: node29:/data5/bricks/project2
>>>>>> Brick53: node30:/data5/bricks/project2 Brick54:
>>>>>> node31:/data5/bricks/project2 Brick55: node32:/data5/bricks/project2
>>>>>> Brick56: node33:/data5/bricks/project2 Brick57:
>>>>>> node20:/data6/bricks/project2 Brick58: node21:/data6/bricks/project2
>>>>>> Brick59: node22:/data6/bricks/project2 Brick60:
>>>>>> node23:/data6/bricks/project2 Brick61: node24:/data6/bricks/project2
>>>>>> Brick62: node25:/data6/bricks/project2 Brick63:
>>>>>> node26:/data6/bricks/project2 Brick64: node27:/data6/bricks/project2
>>>>>> Brick65: node28:/data6/bricks/project2 Brick66:
>>>>>> node29:/data6/bricks/project2 Brick67: node30:/data6/bricks/project2
>>>>>> Brick68: node31:/data6/bricks/project2 Brick69:
>>>>>

Re: [Gluster-users] mv lost some files ?

[Nithya Balachandran]

Hi,

Please try turning off cluster.readdir-optimize and see it if helps.
If not, please send us the client mount logs and a tcpdump of when the *ls*
is performed from the client.  Use the following to capture the dump:

tcpdump -i any -s 0 -w /var/tmp/dirls.pcap tcp and not port 22



Thanks,
Nithya

On 5 September 2018 at 12:46, yu sun  wrote:

>
> [root@ml-ctl-ser01 sunyuyusun]$ salt "ml-storage-ser2*.nmg01" cmd.run
> "getfattr -d -m. -e hex /data2/bricks/project2/371_37829/face_landmarks"
> ml-storage-ser24.nmg01:
> getfattr: Removing leading '/' from absolute path names
> # file: data2/bricks/project2/371_37829/face_landmarks
> trusted.gfid=0x3764e261741d423d8c8eb6ad5b716429
> trusted.glusterfs.dht=0xdbde0bb1b0c30c2eb3cf3cf0
> trusted.glusterfs.quota.dirty=0x3000
> trusted.glusterfs.quota.f8c0ee37-c980-4162-b7ed-15f911d0843a.contri.1=
> 0x000d76010003
> trusted.glusterfs.quota.size.1=0x000d7600
> 00010003
> ml-storage-ser26.nmg01:
> getfattr: Removing leading '/' from absolute path names
> # file: data2/bricks/project2/371_37829/face_landmarks
> trusted.gfid=0x3764e261741d423d8c8eb6ad5b716429
> trusted.glusterfs.dht=0xdbde0bb1d249248fd551
> trusted.glusterfs.quota.dirty=0x3000
> trusted.glusterfs.quota.f8c0ee37-c980-4162-b7ed-15f911d0843a.contri.1=
> 0x0003
> trusted.glusterfs.quota.size.1=0x
> 0003
> ml-storage-ser23.nmg01:
> getfattr: Removing leading '/' from absolute path names
> # file: data2/bricks/project2/371_37829/face_landmarks
> trusted.gfid=0x3764e261741d423d8c8eb6ad5b716429
> trusted.glusterfs.dht=0xdbde0bb18f3cf3cd9249248f
> trusted.glusterfs.quota.dirty=0x3000
> trusted.glusterfs.quota.f8c0ee37-c980-4162-b7ed-15f911d0843a.contri.1=
> 0x0003
> trusted.glusterfs.quota.size.1=0x
> 0003
> ml-storage-ser27.nmg01:
> getfattr: Removing leading '/' from absolute path names
> # file: data2/bricks/project2/371_37829/face_landmarks
> trusted.gfid=0x3764e261741d423d8c8eb6ad5b716429
> trusted.glusterfs.dht=0xdbde0bb1d249248fd551
> trusted.glusterfs.quota.dirty=0x3000
> trusted.glusterfs.quota.f8c0ee37-c980-4162-b7ed-15f911d0843a.contri.1=
> 0x0003
> trusted.glusterfs.quota.size.1=0x
> 0003
> ml-storage-ser25.nmg01:
> getfattr: Removing leading '/' from absolute path names
> # file: data2/bricks/project2/371_37829/face_landmarks
> trusted.gfid=0x3764e261741d423d8c8eb6ad5b716429
> trusted.glusterfs.dht=0xdbde0bb1b0c30c2eb3cf3cf0
> trusted.glusterfs.quota.dirty=0x3000
> trusted.glusterfs.quota.f8c0ee37-c980-4162-b7ed-15f911d0843a.contri.1=
> 0x000d76010003
> trusted.glusterfs.quota.size.1=0x000d7600
> 00010003
> ml-storage-ser28.nmg01:
> getfattr: Removing leading '/' from absolute path names
> # file: data2/bricks/project2/371_37829/face_landmarks
> trusted.gfid=0x3764e261741d423d8c8eb6ad5b716429
> trusted.glusterfs.dht=0xdbde0bb1f3cf3cf0f6db6db2
> trusted.glusterfs.quota.dirty=0x3000
> trusted.glusterfs.quota.f8c0ee37-c980-4162-b7ed-15f911d0843a.contri.1=
> 0x0003
> trusted.glusterfs.quota.size.1=0x
> 0003
> ml-storage-ser29.nmg01:
> getfattr: Removing leading '/' from absolute path names
> # file: data2/bricks/project2/371_37829/face_landmarks
> trusted.gfid=0x3764e261741d423d8c8eb6ad5b716429
> trusted.glusterfs.dht=0xdbde0bb1f3cf3cf0f6db6db2
> trusted.glusterfs.quota.dirty=0x3000
> trusted.glusterfs.quota.f8c0ee37-c980-4162-b7ed-15f911d0843a.contri.1=
> 0x0003
> trusted.glusterfs.quota.size.1=0x
> 0003
> ml-storage-ser20.nmg01:
> getfattr: Removing leading '/' from absolute path names
> # file: data2/bricks/project2/371_37829/face_landmarks
> trusted.gfid=0x3764e261741d423d8c8eb6ad5b716429
> trusted.glusterfs.dht=0xdbde0bb18c30c30a8f3cf3cc
> trusted.glusterfs.quota.dirty=0x3000
> trusted.glusterfs.quota.f8c0ee37-c980-4162-b7ed-15f911d0843a.contri.1=
> 0x0003
> trusted.glusterfs.quota.size.1=0x
> 0003
> ml-storage-ser22.nmg01:
> getfattr: Removing leading '/' from absolute path names
> # file: data2/bricks/project2/371_37829/face_landmarks
> trusted.gfid=0x3764e261741d423d8c8eb6ad5b716429
> 

Re: [Gluster-users] gluster 3.12.8 fuse consume huge memory

[Nithya Balachandran]

Hi,

Please take statedumps of the 3.12.13 client process at intervals when the
memory is increasing and send those across.
We will also need the gluster volume info for the volume int question.

Thanks,
Nithya


On 31 August 2018 at 08:32, huting3  wrote:

> Thanks for your reply, I also test gluster 3.12.13 and found the client
> also consumes huge memory:
>
> PID USER  PR  NIVIRTRESSHR S  %CPU %MEM TIME+ COMMAND
> 180095 root  20   0 4752256 4.091g   4084 S  43.5  1.6  17:54.70
> glusterfs
>
> I read and write some files on the gluster fuse client, the client consume
> 4g memory and it keeps arising.
>
> Does it really fixed in 3.12.13?
>
> huting3
> huti...@corp.netease.com
>
> 
> 签名由 网易邮箱大师  定制
>
> On 08/30/2018 22:37，Darrell Budic
>  wrote：
>
> It’s probably https://bugzilla.redhat.com/show_bug.cgi?id=1593826,
> although I did not encounter it in 3.12.8, only 3.12.9 - 12.
>
> It’s fixed in 3.12.13.
>
> --
> *From:* huting3 
> *Subject:* [Gluster-users] gluster 3.12.8 fuse consume huge memory
> *Date:* August 30, 2018 at 2:02:01 AM CDT
> *To:* gluster-users@gluster.org
>
> The version of glusterfs I installed is 3.12.8, and I find it`s client
> also consume huge memory.
>
>
> I dumped the statedump file, and I found the size of a variable is
> extreamly huge like below:
>
>
> [mount/fuse.fuse - usage-type gf_fuse_mt_iov_base memusage]
>
>   49805 size=4250821416
>
>   49806 num_allocs=1
>
>   49807 max_size=4294960048
>
>   49808 max_num_allocs=3
>
>   49809 total_allocs=12330719
>
>
> Is it means a memory leak exist in glusterfs client?
>
> huting3
> huti...@corp.netease.com
>
> 
> 签名由 网易邮箱大师  定制
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] gluster connection interrupted during transfer

[Nithya Balachandran]

Hi Richard,



On 29 August 2018 at 18:11, Richard Neuboeck  wrote:

> Hi Gluster Community,
>
> I have problems with a glusterfs 'Transport endpoint not connected'
> connection abort during file transfers that I can replicate (all the
> time now) but not pinpoint as to why this is happening.
>
> The volume is set up in replica 3 mode and accessed with the fuse
> gluster client. Both client and server are running CentOS and the
> supplied 3.12.11 version of gluster.
>
> The connection abort happens at different times during rsync but
> occurs every time I try to sync all our files (1.1TB) to the empty
> volume.
>
> Client and server side I don't find errors in the gluster log files.
> rsync logs the obvious transfer problem. The only log that shows
> anything related is the server brick log which states that the
> connection is shutting down:
>
> [2018-08-18 22:40:35.502510] I [MSGID: 115036]
> [server.c:527:server_rpc_notify] 0-home-server: disconnecting
> connection from brax-110405-2018/08/16-08:36:28:575972-home-client-0-0-0
> [2018-08-18 22:40:35.502620] W
> [inodelk.c:499:pl_inodelk_log_cleanup] 0-home-server: releasing lock
> on eaeb0398-fefd-486d-84a7-f13744d1cf10 held by
> {client=0x7f83ec0b3ce0, pid=110423 lk-owner=d0fd5ffb427f}
> [2018-08-18 22:40:35.502692] W
> [entrylk.c:864:pl_entrylk_log_cleanup] 0-home-server: releasing lock
> on faa93f7b-6c46-4251-b2b2-abcd2f2613e1 held by
> {client=0x7f83ec0b3ce0, pid=110423 lk-owner=703dd4cc407f}
> [2018-08-18 22:40:35.502719] W
> [entrylk.c:864:pl_entrylk_log_cleanup] 0-home-server: releasing lock
> on faa93f7b-6c46-4251-b2b2-abcd2f2613e1 held by
> {client=0x7f83ec0b3ce0, pid=110423 lk-owner=703dd4cc407f}
> [2018-08-18 22:40:35.505950] I [MSGID: 101055]
> [client_t.c:443:gf_client_unref] 0-home-server: Shutting down
> connection brax-110405-2018/08/16-08:36:28:575972-home-client-0-0-0


> Since I'm running another replica 3 setup for oVirt for a long time
>

Is this setup running with the same gluster version and on the same nodes
or is it a different cluster?



> now which is completely stable I thought I made a mistake setting
> different options at first. However even when I reset those options
> I'm able to reproduce the connection problem.
>
> The unoptimized volume setup looks like this:


> Volume Name: home
> Type: Replicate
> Volume ID: c92fa4cc-4a26-41ff-8c70-1dd07f733ac8
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x 3 = 3
> Transport-type: tcp
> Bricks:
> Brick1: sphere-four:/srv/gluster_home/brick
> Brick2: sphere-five:/srv/gluster_home/brick
> Brick3: sphere-six:/srv/gluster_home/brick
> Options Reconfigured:
> nfs.disable: on
> transport.address-family: inet
> cluster.quorum-type: auto
> cluster.server-quorum-type: server
> cluster.server-quorum-ratio: 50%
>
>
> The following additional options were used before:
>
> performance.cache-size: 5GB
> client.event-threads: 4
> server.event-threads: 4
> cluster.lookup-optimize: on
> features.cache-invalidation: on
> performance.stat-prefetch: on
> performance.cache-invalidation: on
> network.inode-lru-limit: 5
> features.cache-invalidation-timeout: 600
> performance.md-cache-timeout: 600
> performance.parallel-readdir: on
>
>
> In this case the gluster servers and also the client is using a
> bonded network device running in adaptive load balancing mode.
>
> I've tried using the debug option for the client mount. But except
> for a ~0.5TB log file I didn't get information that seems helpful to me.
>
> Transferring just a couple of GB works without problems.
>
> It may very well be that I'm already blind to the obvious but after
> many long running tests I can't find the crux in the setup.
>
> Does anyone have an idea as how to approach this problem in a way
> that sheds some useful information?
>
> Any help is highly appreciated!
> Cheers
> Richard
>
> --
> /dev/null
>
>
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster release 3.12.13 (Long Term Maintenance) Canceled for 10th of August, 2018

[Nithya Balachandran]

I agree as well. This is a bug that is impacting users.

On 14 August 2018 at 16:30, Ravishankar N  wrote:

> +1
>
> Considering that master is no longer locked, it would be nice if a release
> can be made sooner.  Amar sent a missing back port [1] which also fixes a
> mem leak issue on the client side. This needs to go in too.
> Regards,
> Ravi
>
> [1] https://review.gluster.org/#/c/glusterfs/+/20723/
>
>
> On 08/14/2018 04:20 PM, lemonni...@ulrar.net wrote:
>
>> Hi,
>>
>> That's actually pretty bad, we've all been waiting for the memory leak
>> patch for a while now, an extra month is a bit of a nightmare for us.
>>
>> Is there no way to get 3.12.12 with that patch sooner, at least ? I'm
>> getting a bit tired of rebooting virtual machines by hand everyday to
>> avoid the OOM killer ..
>>
>> On Tue, Aug 14, 2018 at 04:12:28PM +0530, Jiffin Tony Thottan wrote:
>>
>>> Hi,
>>>
>>> Currently master branch is lock for fixing failures in the regression
>>> test suite [1].
>>>
>>> As a result we are not releasing the next minor update for the 3.12
>>> branch,
>>>
>>> which falls on the 10th of every month.
>>>
>>> The next 3.12 update would be around the 10th of September, 2018.
>>>
>>> Apologies for the delay to inform above details.
>>>
>>> [1]
>>> https://lists.gluster.org/pipermail/gluster-devel/2018-Augus
>>> t/055160.html
>>>
>>> Regards,
>>>
>>> Jiffin
>>>
>>> ___
>>> Gluster-users mailing list
>>> Gluster-users@gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] blocking process on FUSE mount in directory which is using quota

[Nithya Balachandran]

Thanks for letting us know. Sanoj, can you take a look at this?

Thanks.
Nithya

On 14 August 2018 at 13:58, mabi  wrote:

> As you mentioned after creating the /var/run/gluster directory I got a
> statedump file in there.
>
> As a workaround I have now removed the quota for this specific directory
> and as it is a production server I can currently not "play" with it by
> adding the quota back and having the same problem as it requires me to
> reboot the server with downtime...
>
> But I can confirm that by removing the quota from that directory, the
> problem is gone (no more blocking processes such as "ls") so there must be
> an issue or bug with the quota part of gluster.
>
>
>
> ‐‐‐ Original Message ‐‐‐
> On August 10, 2018 4:19 PM, Nithya Balachandran 
> wrote:
>
>
>
> On 9 August 2018 at 19:54, mabi  wrote:
>
>> Thanks for the documentation. On my client using FUSE mount I found the
>> PID by using ps (output below):
>>
>> root   456 1  4 14:17 ?00:05:15 /usr/sbin/glusterfs
>> --volfile-server=gfs1a --volfile-id=myvol-private /mnt/myvol-private
>>
>> Then I ran the following command
>>
>> sudo kill -USR1 456
>>
>> but now I can't find where the files are stored. Are these supposed to be
>> stored on the client directly? I checked /var/run/gluster and
>> /var/log/gluster but could not see anything and /var/log/gluster does not
>> even exist on the client.
>>
>
> They are usually created in /var/run/gluster. You will need to create the
> directory on the client if it does not exist.
>
>
>
> ‐‐‐ Original Message ‐‐‐
>> On August 9, 2018 3:59 PM, Raghavendra Gowdappa 
>> wrote:
>>
>>
>>
>> On Thu, Aug 9, 2018 at 6:47 PM, mabi  wrote:
>>
>>> Hi Nithya,
>>>
>>> Thanks for the fast answer. Here the additional info:
>>>
>>> 1. gluster volume info
>>>
>>> Volume Name: myvol-private
>>> Type: Replicate
>>> Volume ID: e7a40a1b-45c9-4d3c-bb19-0c59b4eceec5
>>> Status: Started
>>> Snapshot Count: 0
>>> Number of Bricks: 1 x (2 + 1) = 3
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: gfs1a:/data/myvol-private/brick
>>> Brick2: gfs1b:/data/myvol-private/brick
>>> Brick3: gfs1c:/srv/glusterfs/myvol-private/brick (arbiter)
>>> Options Reconfigured:
>>> features.default-soft-limit: 95%
>>> transport.address-family: inet
>>> features.quota-deem-statfs: on
>>> features.inode-quota: on
>>> features.quota: on
>>> nfs.disable: on
>>> performance.readdir-ahead: on
>>> client.event-threads: 4
>>> server.event-threads: 4
>>> auth.allow: 192.168.100.92
>>>
>>>
>>>
>>> 2. Sorry I have no clue how to take a "statedump" of a process on Linux.
>>> Which command should I use for that? and which process would you like, the
>>> blocked process (for example "ls")?
>>>
>>
>> Statedumps are gluster specific. Please refer to
>> https://docs.gluster.org/en/v3/Troubleshooting/statedump/ for
>> instructions.
>>
>>
>>>
>>> Regards,
>>> M.
>>>
>>> ‐‐‐ Original Message ‐‐‐
>>> On August 9, 2018 3:10 PM, Nithya Balachandran 
>>> wrote:
>>>
>>> Hi,
>>>
>>> Please provide the following:
>>>
>>>1. gluster volume info
>>>2. statedump of the fuse process when it hangs
>>>
>>>
>>> Thanks,
>>> Nithya
>>>
>>> On 9 August 2018 at 18:24, mabi  wrote:
>>>
>>>> Hello,
>>>>
>>>> I recently upgraded my GlusterFS replica 2+1 (aribter) to version
>>>> 3.12.12 and now I see a weird behaviour on my client (using FUSE mount)
>>>> where I have processes (PHP 5.6 FPM) trying to access a specific directory
>>>> and then the process blocks. I can't kill the process either, not even with
>>>> kill -9. I need to reboot the machine in order to get rid of these blocked
>>>> processes.
>>>>
>>>> This directory has one particularity compared to the other directories
>>>> it is that it has reached it's quota soft-limit as you can see here in the
>>>> output of gluster volume quota list:
>>>>
>>>>   Path   Hard-limit  Soft-limit
>>>> Used  Available  Soft-limit exceeded? Hard-limit exceeded?
>>>> 

Re: [Gluster-users] blocking process on FUSE mount in directory which is using quota

[Nithya Balachandran]

On 9 August 2018 at 19:54, mabi  wrote:

> Thanks for the documentation. On my client using FUSE mount I found the
> PID by using ps (output below):
>
> root   456 1  4 14:17 ?00:05:15 /usr/sbin/glusterfs
> --volfile-server=gfs1a --volfile-id=myvol-private /mnt/myvol-private
>
> Then I ran the following command
>
> sudo kill -USR1 456
>
> but now I can't find where the files are stored. Are these supposed to be
> stored on the client directly? I checked /var/run/gluster and
> /var/log/gluster but could not see anything and /var/log/gluster does not
> even exist on the client.
>

They are usually created in /var/run/gluster. You will need to create the
directory on the client if it does not exist.



‐‐‐ Original Message ‐‐‐
> On August 9, 2018 3:59 PM, Raghavendra Gowdappa 
> wrote:
>
>
>
> On Thu, Aug 9, 2018 at 6:47 PM, mabi  wrote:
>
>> Hi Nithya,
>>
>> Thanks for the fast answer. Here the additional info:
>>
>> 1. gluster volume info
>>
>> Volume Name: myvol-private
>> Type: Replicate
>> Volume ID: e7a40a1b-45c9-4d3c-bb19-0c59b4eceec5
>> Status: Started
>> Snapshot Count: 0
>> Number of Bricks: 1 x (2 + 1) = 3
>> Transport-type: tcp
>> Bricks:
>> Brick1: gfs1a:/data/myvol-private/brick
>> Brick2: gfs1b:/data/myvol-private/brick
>> Brick3: gfs1c:/srv/glusterfs/myvol-private/brick (arbiter)
>> Options Reconfigured:
>> features.default-soft-limit: 95%
>> transport.address-family: inet
>> features.quota-deem-statfs: on
>> features.inode-quota: on
>> features.quota: on
>> nfs.disable: on
>> performance.readdir-ahead: on
>> client.event-threads: 4
>> server.event-threads: 4
>> auth.allow: 192.168.100.92
>>
>>
>>
>> 2. Sorry I have no clue how to take a "statedump" of a process on Linux.
>> Which command should I use for that? and which process would you like, the
>> blocked process (for example "ls")?
>>
>
> Statedumps are gluster specific. Please refer to
> https://docs.gluster.org/en/v3/Troubleshooting/statedump/ for
> instructions.
>
>
>>
>> Regards,
>> M.
>>
>> ‐‐‐ Original Message ‐‐‐
>> On August 9, 2018 3:10 PM, Nithya Balachandran 
>> wrote:
>>
>> Hi,
>>
>> Please provide the following:
>>
>>1. gluster volume info
>>2. statedump of the fuse process when it hangs
>>
>>
>> Thanks,
>> Nithya
>>
>> On 9 August 2018 at 18:24, mabi  wrote:
>>
>>> Hello,
>>>
>>> I recently upgraded my GlusterFS replica 2+1 (aribter) to version
>>> 3.12.12 and now I see a weird behaviour on my client (using FUSE mount)
>>> where I have processes (PHP 5.6 FPM) trying to access a specific directory
>>> and then the process blocks. I can't kill the process either, not even with
>>> kill -9. I need to reboot the machine in order to get rid of these blocked
>>> processes.
>>>
>>> This directory has one particularity compared to the other directories
>>> it is that it has reached it's quota soft-limit as you can see here in the
>>> output of gluster volume quota list:
>>>
>>>   Path   Hard-limit  Soft-limit
>>> Used  Available  Soft-limit exceeded? Hard-limit exceeded?
>>> 
>>> ---
>>> /directory  100.0GB 80%(80.0GB)   90.5GB
>>>  9.5GB Yes   No
>>>
>>> That does not mean that it is the quota's fault but it might be a hint
>>> where to start looking for... And by the way can someone explain me what
>>> the soft-limit does? or does it not do anything special?
>>>
>>> Here is an the linux stack of a blocking process on that directory which
>>> happened with a simple "ls -la":
>>>
>>> [Thu Aug  9 14:21:07 2018] INFO: task ls:2272 blocked for more than 120
>>> seconds.
>>> [Thu Aug  9 14:21:07 2018]   Not tainted 3.16.0-4-amd64 #1
>>> [Thu Aug  9 14:21:07 2018] "echo 0 > 
>>> /proc/sys/kernel/hung_task_timeout_secs"
>>> disables this message.
>>> [Thu Aug  9 14:21:07 2018] ls  D 88017ef93200 0
>>> 2272   2268 0x0004
>>> [Thu Aug  9 14:21:07 2018]  88017653f490 0286
>>> 00013200 880174d7bfd8
>>> [Thu Aug  9 14:21:07 2018]  00013200 ff

Re: [Gluster-users] blocking process on FUSE mount in directory which is using quota

[Nithya Balachandran]

Hi,

Please provide the following:

   1. gluster volume info
   2. statedump of the fuse process when it hangs


Thanks,
Nithya

On 9 August 2018 at 18:24, mabi  wrote:

> Hello,
>
> I recently upgraded my GlusterFS replica 2+1 (aribter) to version 3.12.12
> and now I see a weird behaviour on my client (using FUSE mount) where I
> have processes (PHP 5.6 FPM) trying to access a specific directory and then
> the process blocks. I can't kill the process either, not even with kill -9.
> I need to reboot the machine in order to get rid of these blocked processes.
>
> This directory has one particularity compared to the other directories it
> is that it has reached it's quota soft-limit as you can see here in the
> output of gluster volume quota list:
>
>   Path   Hard-limit  Soft-limit  Used
> Available  Soft-limit exceeded? Hard-limit exceeded?
> 
> ---
> /directory  100.0GB 80%(80.0GB)   90.5GB
>  9.5GB Yes   No
>
> That does not mean that it is the quota's fault but it might be a hint
> where to start looking for... And by the way can someone explain me what
> the soft-limit does? or does it not do anything special?
>
> Here is an the linux stack of a blocking process on that directory which
> happened with a simple "ls -la":
>
> [Thu Aug  9 14:21:07 2018] INFO: task ls:2272 blocked for more than 120
> seconds.
> [Thu Aug  9 14:21:07 2018]   Not tainted 3.16.0-4-amd64 #1
> [Thu Aug  9 14:21:07 2018] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [Thu Aug  9 14:21:07 2018] ls  D 88017ef93200 0  2272
>  2268 0x0004
> [Thu Aug  9 14:21:07 2018]  88017653f490 0286
> 00013200 880174d7bfd8
> [Thu Aug  9 14:21:07 2018]  00013200 88017653f490
> 8800eeb3d5f0 8800fefac800
> [Thu Aug  9 14:21:07 2018]  880174d7bbe0 8800eeb3d6d0
> 8800fefac800 8800ffe1e1c0
> [Thu Aug  9 14:21:07 2018] Call Trace:
> [Thu Aug  9 14:21:07 2018]  [] ?
> __fuse_request_send+0xbd/0x270 [fuse]
> [Thu Aug  9 14:21:07 2018]  [] ?
> prepare_to_wait_event+0xf0/0xf0
> [Thu Aug  9 14:21:07 2018]  [] ?
> fuse_dentry_revalidate+0x181/0x300 [fuse]
> [Thu Aug  9 14:21:07 2018]  [] ? lookup_fast+0x25e/0x2b0
> [Thu Aug  9 14:21:07 2018]  [] ?
> path_lookupat+0x155/0x780
> [Thu Aug  9 14:21:07 2018]  [] ?
> kmem_cache_alloc+0x75/0x480
> [Thu Aug  9 14:21:07 2018]  [] ?
> fuse_getxattr+0xe9/0x150 [fuse]
> [Thu Aug  9 14:21:07 2018]  [] ?
> filename_lookup+0x26/0xc0
> [Thu Aug  9 14:21:07 2018]  [] ?
> user_path_at_empty+0x54/0x90
> [Thu Aug  9 14:21:07 2018]  [] ?
> kmem_cache_free+0xd8/0x210
> [Thu Aug  9 14:21:07 2018]  [] ?
> user_path_at_empty+0x5f/0x90
> [Thu Aug  9 14:21:07 2018]  [] ? vfs_fstatat+0x46/0x90
> [Thu Aug  9 14:21:07 2018]  [] ? SYSC_newlstat+0x1d/0x40
> [Thu Aug  9 14:21:07 2018]  [] ? SyS_lgetxattr+0x58/0x80
> [Thu Aug  9 14:21:07 2018]  [] ?
> system_call_fast_compare_end+0x10/0x15
>
>
> My 3 gluster nodes are all Debian 9 and my client Debian 8.
>
> Let me know if you need more information.
>
> Best regards,
> Mabi
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] gluster 3.12 memory leak

[Nithya Balachandran]

Hi,

What version of gluster were you using before you  upgraded?

Regards,
Nithya

On 3 August 2018 at 16:56, Alex K  wrote:

> Hi all,
>
> I am using gluster 3.12.9-1 on ovirt 4.1.9 and I have observed consistent
> high memory use which at some point renders the hosts unresponsive. This
> behavior is observed also while using 3.12.11-1 with ovirt 4.2.5. I did not
> have this issue prior to upgrading gluster.
>
> I have seen a relevant bug reporting memory leaks of gluster and it seems
> that this is the case for my trouble. To temporarily resolve the high
> memory issue, I put hosts in maintenance then activate them back again.
> This indicates that the memory leak is caused from the gluster client.
> Ovirt is using fuse mounts.
>
> Is there any bug fx available for this?
> This issue is hitting us hard with several production installations.
>
> Thanx,
> Alex
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Rebalance taking > 2 months

[Nithya Balachandran]

On 31 July 2018 at 22:17, Rusty Bower  wrote:

> Is it possible to pause the rebalance to get those number? I'm hesitant to
> stop the rebalance and have to redo the entire thing again.
>
> I'm afraid not. Rebalance will start from the beginning if you do so.




> On Tue, Jul 31, 2018 at 11:40 AM, Nithya Balachandran  > wrote:
>
>>
>>
>> On 31 July 2018 at 19:44, Rusty Bower  wrote:
>>
>>> I'll figure out what hasn't been rebalanced yet and run the script.
>>>
>>> There's only a single client accessing this gluster volume, and while
>>> the rebalance is taking place, the I am only able to read/write to the
>>> volume at around 3MB/s. If I log onto one of the bricks, I can read/write
>>> to the physical volumes at speed greater than 100MB/s (which is what I
>>> would expect).
>>>
>>
>> What are the numbers when accessing the volume when rebalance is not
>> running?
>> Regards,
>> Nithya
>>
>>>
>>> Thanks!
>>> Rusty
>>>
>>> On Tue, Jul 31, 2018 at 3:28 AM, Nithya Balachandran <
>>> nbala...@redhat.com> wrote:
>>>
>>>> Hi Rusty,
>>>>
>>>> A rebalance involves 2 steps:
>>>>
>>>>1. Setting a new layout on a directory
>>>>2. Migrating any files inside that directory that hash to a
>>>>different subvol based on the new layout set in step 1.
>>>>
>>>>
>>>> A few things to keep in mind :
>>>>
>>>>- Any new content created on this volume will currently go to the
>>>>newly added brick.
>>>>- Having a more equitable file distribution is beneficial but you
>>>>might not need to do a complete rebalance to do this. You can run the
>>>>script on  just enough directories to free up space on your older 
>>>> bricks.
>>>>This should be done on bricks which contains large files to speed this 
>>>> up.
>>>>
>>>> Do the following on one of the server nodes:
>>>>
>>>>- Create a tmp mount point and mount the volume using the rebalance
>>>>volfile
>>>>- mkdir /mnt/rebal
>>>>   - glusterfs -s localhost --volfile-id rebalance/data /mnt/rebal
>>>>- Select a directory in the volume which contains a lot of large
>>>>files and which has not been processed by the rebalance yet - the lower
>>>>down in the tree the better. Check the rebalance logs to figure out 
>>>> which
>>>>dirs have not been processed yet.
>>>>   - cd /mnt/rebal/
>>>>   - for dir in `find . -type d`; do echo $dir |xargs -0 -n1 -P10
>>>>   bash process_dir.sh;done
>>>>- You can run this for different values of  and on
>>>>multiple server nodes in parallel as long as the directory trees for the
>>>>different  don't overlap.
>>>>- Do this for multiple directories until the disk space used
>>>>reduces on the older bricks.
>>>>
>>>> This is a very simple script. Let me know how it works - we can always
>>>> tweak it for your particular data set.
>>>>
>>>>
>>>> >and performance is basically garbage while it rebalances
>>>> Can you provide more detail on this? What kind of effects are you
>>>> seeing?
>>>> How many clients access this volume?
>>>>
>>>>
>>>> Regards,
>>>> Nithya
>>>>
>>>> On 30 July 2018 at 22:18, Nithya Balachandran 
>>>> wrote:
>>>>
>>>>> I have not documented this yet - I will send you the steps tomorrow.
>>>>>
>>>>> Regards,
>>>>> Nithya
>>>>>
>>>>> On 30 July 2018 at 20:23, Rusty Bower  wrote:
>>>>>
>>>>>> That would be awesome. Where can I find these?
>>>>>>
>>>>>> Rusty
>>>>>>
>>>>>> Sent from my iPhone
>>>>>>
>>>>>> On Jul 30, 2018, at 03:40, Nithya Balachandran 
>>>>>> wrote:
>>>>>>
>>>>>> Hi Rusty,
>>>>>>
>>>>>> Sorry for the delay getting back to you. I had a quick look at the
>>>>>> rebalance logs - it looks like the estimates are based on the time taken 
>>>>>> to
>>>>>> rebalance the smaller fi

Re: [Gluster-users] Rebalance taking > 2 months

[Nithya Balachandran]

On 31 July 2018 at 19:44, Rusty Bower  wrote:

> I'll figure out what hasn't been rebalanced yet and run the script.
>
> There's only a single client accessing this gluster volume, and while the
> rebalance is taking place, the I am only able to read/write to the volume
> at around 3MB/s. If I log onto one of the bricks, I can read/write to the
> physical volumes at speed greater than 100MB/s (which is what I would
> expect).
>

What are the numbers when accessing the volume when rebalance is not
running?
Regards,
Nithya

>
> Thanks!
> Rusty
>
> On Tue, Jul 31, 2018 at 3:28 AM, Nithya Balachandran 
> wrote:
>
>> Hi Rusty,
>>
>> A rebalance involves 2 steps:
>>
>>1. Setting a new layout on a directory
>>2. Migrating any files inside that directory that hash to a different
>>subvol based on the new layout set in step 1.
>>
>>
>> A few things to keep in mind :
>>
>>- Any new content created on this volume will currently go to the
>>newly added brick.
>>- Having a more equitable file distribution is beneficial but you
>>might not need to do a complete rebalance to do this. You can run the
>>script on  just enough directories to free up space on your older bricks.
>>This should be done on bricks which contains large files to speed this up.
>>
>> Do the following on one of the server nodes:
>>
>>- Create a tmp mount point and mount the volume using the rebalance
>>volfile
>>- mkdir /mnt/rebal
>>   - glusterfs -s localhost --volfile-id rebalance/data /mnt/rebal
>>- Select a directory in the volume which contains a lot of large
>>files and which has not been processed by the rebalance yet - the lower
>>down in the tree the better. Check the rebalance logs to figure out which
>>dirs have not been processed yet.
>>   - cd /mnt/rebal/
>>   - for dir in `find . -type d`; do echo $dir |xargs -0 -n1 -P10
>>   bash process_dir.sh;done
>>- You can run this for different values of  and on
>>multiple server nodes in parallel as long as the directory trees for the
>>different  don't overlap.
>>- Do this for multiple directories until the disk space used reduces
>>on the older bricks.
>>
>> This is a very simple script. Let me know how it works - we can always
>> tweak it for your particular data set.
>>
>>
>> >and performance is basically garbage while it rebalances
>> Can you provide more detail on this? What kind of effects are you seeing?
>> How many clients access this volume?
>>
>>
>> Regards,
>> Nithya
>>
>> On 30 July 2018 at 22:18, Nithya Balachandran 
>> wrote:
>>
>>> I have not documented this yet - I will send you the steps tomorrow.
>>>
>>> Regards,
>>> Nithya
>>>
>>> On 30 July 2018 at 20:23, Rusty Bower  wrote:
>>>
>>>> That would be awesome. Where can I find these?
>>>>
>>>> Rusty
>>>>
>>>> Sent from my iPhone
>>>>
>>>> On Jul 30, 2018, at 03:40, Nithya Balachandran 
>>>> wrote:
>>>>
>>>> Hi Rusty,
>>>>
>>>> Sorry for the delay getting back to you. I had a quick look at the
>>>> rebalance logs - it looks like the estimates are based on the time taken to
>>>> rebalance the smaller files.
>>>>
>>>> We do have a scripting option where we can use virtual xattrs to
>>>> trigger file migration from a mount point. That would speed things up.
>>>>
>>>>
>>>> Regards,
>>>> Nithya
>>>>
>>>> On 28 July 2018 at 07:11, Rusty Bower  wrote:
>>>>
>>>>> Just wanted to ping this to see if you guys had any thoughts, or other
>>>>> scripts I can run for this stuff. It's still predicting another 90 days to
>>>>> rebalance this, and performance is basically garbage while it rebalances.
>>>>>
>>>>> Rusty
>>>>>
>>>>> On Mon, Jul 23, 2018 at 10:19 AM, Rusty Bower 
>>>>> wrote:
>>>>>
>>>>>> datanode03 is the newest brick
>>>>>>
>>>>>> the bricks had gotten pretty full, which I think might be part of the
>>>>>> issue:
>>>>>> - datanode01 /dev/sda1 51T   48T  3.3T  94% /mnt/data
>>>>>> - datanode02 /dev/sda1 51T   48T  3.4T  94% /mnt/data
>>>&

Re: [Gluster-users] Rebalance taking > 2 months

[Nithya Balachandran]

Hi Rusty,

A rebalance involves 2 steps:

   1. Setting a new layout on a directory
   2. Migrating any files inside that directory that hash to a different
   subvol based on the new layout set in step 1.


A few things to keep in mind :

   - Any new content created on this volume will currently go to the newly
   added brick.
   - Having a more equitable file distribution is beneficial but you might
   not need to do a complete rebalance to do this. You can run the script on
   just enough directories to free up space on your older bricks. This should
   be done on bricks which contains large files to speed this up.

Do the following on one of the server nodes:

   - Create a tmp mount point and mount the volume using the rebalance
   volfile
   - mkdir /mnt/rebal
  - glusterfs -s localhost --volfile-id rebalance/data /mnt/rebal
   - Select a directory in the volume which contains a lot of large files
   and which has not been processed by the rebalance yet - the lower down in
   the tree the better. Check the rebalance logs to figure out which dirs have
   not been processed yet.
  - cd /mnt/rebal/
  - for dir in `find . -type d`; do echo $dir |xargs -0 -n1 -P10 bash
  process_dir.sh;done
   - You can run this for different values of  and on multiple
   server nodes in parallel as long as the directory trees for the different
don't overlap.
   - Do this for multiple directories until the disk space used reduces on
   the older bricks.

This is a very simple script. Let me know how it works - we can always
tweak it for your particular data set.


>and performance is basically garbage while it rebalances
Can you provide more detail on this? What kind of effects are you seeing?
How many clients access this volume?


Regards,
Nithya

On 30 July 2018 at 22:18, Nithya Balachandran  wrote:

> I have not documented this yet - I will send you the steps tomorrow.
>
> Regards,
> Nithya
>
> On 30 July 2018 at 20:23, Rusty Bower  wrote:
>
>> That would be awesome. Where can I find these?
>>
>> Rusty
>>
>> Sent from my iPhone
>>
>> On Jul 30, 2018, at 03:40, Nithya Balachandran 
>> wrote:
>>
>> Hi Rusty,
>>
>> Sorry for the delay getting back to you. I had a quick look at the
>> rebalance logs - it looks like the estimates are based on the time taken to
>> rebalance the smaller files.
>>
>> We do have a scripting option where we can use virtual xattrs to trigger
>> file migration from a mount point. That would speed things up.
>>
>>
>> Regards,
>> Nithya
>>
>> On 28 July 2018 at 07:11, Rusty Bower  wrote:
>>
>>> Just wanted to ping this to see if you guys had any thoughts, or other
>>> scripts I can run for this stuff. It's still predicting another 90 days to
>>> rebalance this, and performance is basically garbage while it rebalances.
>>>
>>> Rusty
>>>
>>> On Mon, Jul 23, 2018 at 10:19 AM, Rusty Bower 
>>> wrote:
>>>
>>>> datanode03 is the newest brick
>>>>
>>>> the bricks had gotten pretty full, which I think might be part of the
>>>> issue:
>>>> - datanode01 /dev/sda1 51T   48T  3.3T  94% /mnt/data
>>>> - datanode02 /dev/sda1     51T   48T  3.4T  94% /mnt/data
>>>> - datanode03 /dev/md0 128T  4.6T  123T   4% /mnt/data
>>>>
>>>> each of the bricks are on a completely separate disk from the OS
>>>>
>>>> I'll shoot you the log files offline :)
>>>>
>>>> Thanks!
>>>> Rusty
>>>>
>>>> On Mon, Jul 23, 2018 at 3:12 AM, Nithya Balachandran <
>>>> nbala...@redhat.com> wrote:
>>>>
>>>>> Hi Rusty,
>>>>>
>>>>> Sorry I took so long to get back to you.
>>>>>
>>>>> Which is the newly added brick? I see datanode02 has not picked up
>>>>> any files for migration which is odd.
>>>>> How full are the individual bricks (df -h ) output.
>>>>> Is each of your bricks in a separate partition?
>>>>> Can you send me the rebalance logs from all 3 nodes (offline if you
>>>>> prefer)?
>>>>>
>>>>> We can try using scripts to speed up the rebalance if you prefer.
>>>>>
>>>>> Regards,
>>>>> Nithya
>>>>>
>>>>>
>>>>>
>>>>> On 16 July 2018 at 22:06, Rusty Bower  wrote:
>>>>>
>>>>>> Thanks for the reply Nithya.
>>>>>>
>>>>>> 1. glusterfs 4.1.1
>>>>>>
>&

Re: [Gluster-users] Rebalance taking > 2 months

[Nithya Balachandran]

I have not documented this yet - I will send you the steps tomorrow.

Regards,
Nithya

On 30 July 2018 at 20:23, Rusty Bower  wrote:

> That would be awesome. Where can I find these?
>
> Rusty
>
> Sent from my iPhone
>
> On Jul 30, 2018, at 03:40, Nithya Balachandran 
> wrote:
>
> Hi Rusty,
>
> Sorry for the delay getting back to you. I had a quick look at the
> rebalance logs - it looks like the estimates are based on the time taken to
> rebalance the smaller files.
>
> We do have a scripting option where we can use virtual xattrs to trigger
> file migration from a mount point. That would speed things up.
>
>
> Regards,
> Nithya
>
> On 28 July 2018 at 07:11, Rusty Bower  wrote:
>
>> Just wanted to ping this to see if you guys had any thoughts, or other
>> scripts I can run for this stuff. It's still predicting another 90 days to
>> rebalance this, and performance is basically garbage while it rebalances.
>>
>> Rusty
>>
>> On Mon, Jul 23, 2018 at 10:19 AM, Rusty Bower 
>> wrote:
>>
>>> datanode03 is the newest brick
>>>
>>> the bricks had gotten pretty full, which I think might be part of the
>>> issue:
>>> - datanode01 /dev/sda1 51T   48T  3.3T  94% /mnt/data
>>> - datanode02 /dev/sda1 51T   48T  3.4T  94% /mnt/data
>>> - datanode03 /dev/md0 128T  4.6T  123T   4% /mnt/data
>>>
>>> each of the bricks are on a completely separate disk from the OS
>>>
>>> I'll shoot you the log files offline :)
>>>
>>> Thanks!
>>> Rusty
>>>
>>> On Mon, Jul 23, 2018 at 3:12 AM, Nithya Balachandran <
>>> nbala...@redhat.com> wrote:
>>>
>>>> Hi Rusty,
>>>>
>>>> Sorry I took so long to get back to you.
>>>>
>>>> Which is the newly added brick? I see datanode02 has not picked up any
>>>> files for migration which is odd.
>>>> How full are the individual bricks (df -h ) output.
>>>> Is each of your bricks in a separate partition?
>>>> Can you send me the rebalance logs from all 3 nodes (offline if you
>>>> prefer)?
>>>>
>>>> We can try using scripts to speed up the rebalance if you prefer.
>>>>
>>>> Regards,
>>>> Nithya
>>>>
>>>>
>>>>
>>>> On 16 July 2018 at 22:06, Rusty Bower  wrote:
>>>>
>>>>> Thanks for the reply Nithya.
>>>>>
>>>>> 1. glusterfs 4.1.1
>>>>>
>>>>> 2. Volume Name: data
>>>>> Type: Distribute
>>>>> Volume ID: 294d95ce-0ff3-4df9-bd8c-a52fc50442ba
>>>>> Status: Started
>>>>> Snapshot Count: 0
>>>>> Number of Bricks: 3
>>>>> Transport-type: tcp
>>>>> Bricks:
>>>>> Brick1: datanode01:/mnt/data/bricks/data
>>>>> Brick2: datanode02:/mnt/data/bricks/data
>>>>> Brick3: datanode03:/mnt/data/bricks/data
>>>>> Options Reconfigured:
>>>>> performance.readdir-ahead: on
>>>>>
>>>>> 3.
>>>>> Node Rebalanced-files
>>>>> size   scanned  failures   skipped   status  run
>>>>> time in h:m:s
>>>>>-  ---
>>>>>  ---   ---   ---   ---
>>>>>   --
>>>>>localhost36822
>>>>> 11.3GB 50715 0 0  in progress
>>>>>  26:46:17
>>>>>   datanode020
>>>>> 0Bytes  2852 0 0  in progress
>>>>>  26:46:16
>>>>>   datanode03 3128
>>>>>  513.7MB 11442 0  3128  in progress
>>>>>26:46:17
>>>>> Estimated time left for rebalance to complete : > 2 months. Please try
>>>>> again later.
>>>>> volume rebalance: data: success
>>>>>
>>>>> 4. Directory structure is basically an rsync backup of some old
>>>>> systems as well as all of my personal media. I can elaborate more, but 
>>>>> it's
>>>>> a pretty standard filesystem.
>>>>>
>>>>

Re: [Gluster-users] Questions on adding brick to gluster

[Nithya Balachandran]

Hi,


On 30 July 2018 at 00:45, Pat Haley  wrote:

>
> Hi All,
>
> We are adding a new brick (91TB) to our existing gluster volume (328 TB).
> The new brick is on a new physical server and we want to make sure that we
> are doing this correctly (the existing volume had 2 bricks on a single
> physical server).  Both are running glusterfs 3.7.11. The steps we've
> followed are
>
>- Setup glusterfs and created brick on new server mseas-data3
>- Peer probe commands in order to make sure that meas-data2 and
>mseas-data3 are in a Trusted Storage Pool
>- Added brick3 (lives on mseas-data3, is 91TB) to data-volume on
>mseas-data2 (328TB)
>- Ran *gluster volume rebalance data-volume fix-layout start *cmd
>(fix-layout to NOT migrate data during process, we did not want to yet)
>Still running as of this Email
>
> The first question we have is at what point should the additional 91 TB be
> visible to the client servers? Currently when we run "df -h" on any client
> we still only see the original 328 TB. Does the additional space only
> appear to the clients after the rebalance using fix-layout finishes?
>
> No, this should be available immediately. Do you see any error messages in
the client log files when running df? Does running the command from a fresh
mount succeed?


> The follow-up question is how long should the rebalance using fix-layout
> take?
>

This is dependent on the number of directories on the volume.

Regards,
Nithya

>
> Some additional information
>
> # gluster volume info
>
> Volume Name: data-volume
> Type: Distribute
> Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18
> Status: Started
> Number of Bricks: 3
> Transport-type: tcp
> Bricks:
> Brick1: mseas-data2:/mnt/brick1
> Brick2: mseas-data2:/mnt/brick2
> Brick3: mseas-data3:/export/sda/brick3
> Options Reconfigured:
> diagnostics.client-log-level: ERROR
> network.inode-lru-limit: 5
> performance.md-cache-timeout: 60
> performance.open-behind: off
> disperse.eager-lock: off
> auth.allow: *
> server.allow-insecure: on
> nfs.exports-auth-enable: on
> diagnostics.brick-sys-log-level: WARNING
> performance.readdir-ahead: on
> nfs.disable: on
> nfs.export-volumes: off
>
> Thanks
>
> --
>
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> Pat Haley  Email:  pha...@mit.edu
> Center for Ocean Engineering   Phone:  (617) 253-6824
> Dept. of Mechanical EngineeringFax:(617) 253-8125
> MIT, Room 5-213http://web.mit.edu/phaley/www/
> 77 Massachusetts Avenue
> Cambridge, MA  02139-4301
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Rebalance taking > 2 months

[Nithya Balachandran]

Hi Rusty,

Sorry for the delay getting back to you. I had a quick look at the
rebalance logs - it looks like the estimates are based on the time taken to
rebalance the smaller files.

We do have a scripting option where we can use virtual xattrs to trigger
file migration from a mount point. That would speed things up.


Regards,
Nithya

On 28 July 2018 at 07:11, Rusty Bower  wrote:

> Just wanted to ping this to see if you guys had any thoughts, or other
> scripts I can run for this stuff. It's still predicting another 90 days to
> rebalance this, and performance is basically garbage while it rebalances.
>
> Rusty
>
> On Mon, Jul 23, 2018 at 10:19 AM, Rusty Bower 
> wrote:
>
>> datanode03 is the newest brick
>>
>> the bricks had gotten pretty full, which I think might be part of the
>> issue:
>> - datanode01 /dev/sda1 51T   48T  3.3T  94% /mnt/data
>> - datanode02 /dev/sda1 51T   48T  3.4T  94% /mnt/data
>> - datanode03 /dev/md0 128T  4.6T  123T   4% /mnt/data
>>
>> each of the bricks are on a completely separate disk from the OS
>>
>> I'll shoot you the log files offline :)
>>
>> Thanks!
>> Rusty
>>
>> On Mon, Jul 23, 2018 at 3:12 AM, Nithya Balachandran > > wrote:
>>
>>> Hi Rusty,
>>>
>>> Sorry I took so long to get back to you.
>>>
>>> Which is the newly added brick? I see datanode02 has not picked up any
>>> files for migration which is odd.
>>> How full are the individual bricks (df -h ) output.
>>> Is each of your bricks in a separate partition?
>>> Can you send me the rebalance logs from all 3 nodes (offline if you
>>> prefer)?
>>>
>>> We can try using scripts to speed up the rebalance if you prefer.
>>>
>>> Regards,
>>> Nithya
>>>
>>>
>>>
>>> On 16 July 2018 at 22:06, Rusty Bower  wrote:
>>>
>>>> Thanks for the reply Nithya.
>>>>
>>>> 1. glusterfs 4.1.1
>>>>
>>>> 2. Volume Name: data
>>>> Type: Distribute
>>>> Volume ID: 294d95ce-0ff3-4df9-bd8c-a52fc50442ba
>>>> Status: Started
>>>> Snapshot Count: 0
>>>> Number of Bricks: 3
>>>> Transport-type: tcp
>>>> Bricks:
>>>> Brick1: datanode01:/mnt/data/bricks/data
>>>> Brick2: datanode02:/mnt/data/bricks/data
>>>> Brick3: datanode03:/mnt/data/bricks/data
>>>> Options Reconfigured:
>>>> performance.readdir-ahead: on
>>>>
>>>> 3.
>>>> Node Rebalanced-files
>>>> size   scanned  failures   skipped   status  run
>>>> time in h:m:s
>>>>-  ---
>>>>  ---   ---   ---   ---
>>>>   --
>>>>localhost36822
>>>> 11.3GB 50715 0 0  in progress
>>>>  26:46:17
>>>>   datanode020
>>>> 0Bytes  2852 0 0  in progress
>>>>  26:46:16
>>>>   datanode03 3128
>>>>  513.7MB 11442 0  3128  in progress
>>>>26:46:17
>>>> Estimated time left for rebalance to complete : > 2 months. Please try
>>>> again later.
>>>> volume rebalance: data: success
>>>>
>>>> 4. Directory structure is basically an rsync backup of some old systems
>>>> as well as all of my personal media. I can elaborate more, but it's a
>>>> pretty standard filesystem.
>>>>
>>>> 5. In some folders there might be up to like 12-15 levels of
>>>> directories (especially the backups)
>>>>
>>>> 6. I'm honestly not sure, I can try to scrounge this number up
>>>>
>>>> 7. My guess would be > 100k
>>>>
>>>> 8. Most files are pretty large (media files), but there's a lot of
>>>> small files (metadata and configuration files) as well
>>>>
>>>> I've also appended a (moderately sanitized) snippet of the rebalance
>>>> log (let me know if you need more)
>>>>
>>>> [2018-07-16 17:37:59.979003] I [MSGID: 0] 
>>>> [dht-rebalance.c:1799:dht_migrate_file]
>>>> 0-data-dht: destination fo

Re: [Gluster-users] Rebalance taking > 2 months

[Nithya Balachandran]

t-rebalance.c:2274:dht_migrate_file]
> 0-data-dht: completed migration of /this/is/a/file/path/that/
> exists/wz/wz/Npc.wz/2040036.img.xml from subvolume data-client-0 to
> data-client-2
> [2018-07-16 17:38:04.738885] I [dht-rebalance.c:1645:dht_migrate_file]
> 0-data-dht: /this/is/a/file/path/that/exists/wz/wz/Npc.wz/9110012.img.xml:
> attempting to move from data-client-0 to data-client-2
> [2018-07-16 17:38:04.745722] I [MSGID: 109022] 
> [dht-rebalance.c:2274:dht_migrate_file]
> 0-data-dht: completed migration of /this/is/a/file/path/that/
> exists/wz/wz/Npc.wz/9201002.img.xml from subvolume data-client-0 to
> data-client-2
> [2018-07-16 17:38:04.812368] I [dht-rebalance.c:4982:gf_
> defrag_get_estimates_based_on_size] 0-glusterfs: TIME: (size)
> total_processed=43108308134 tmp_cnt = 
> 55419279917056,rate_processed=446579.386035,
> elapsed = 96530.00
> [2018-07-16 17:38:04.812417] I [dht-rebalance.c:5130:gf_defrag_status_get]
> 0-glusterfs: TIME: Estimated total time to complete (size)= 124097263
> seconds, seconds left = 124000733
> [2018-07-16 17:38:04.812465] I [MSGID: 109028] 
> [dht-rebalance.c:5210:gf_defrag_status_get]
> 0-glusterfs: Rebalance is in progress. Time taken is 96530.00 secs
> [2018-07-16 17:38:04.812489] I [MSGID: 109028] 
> [dht-rebalance.c:5214:gf_defrag_status_get]
> 0-glusterfs: Files migrated: 36877, size: 12270261443, lookups: 50715,
> failures: 0, skipped: 0
> [2018-07-16 17:38:04.992413] I [dht-rebalance.c:1645:dht_migrate_file]
> 0-data-dht: /this/is/a/file/path/that/exists/wz/wz/Npc.wz/205.img.xml:
> attempting to move from data-client-0 to data-client-2
> [2018-07-16 17:38:04.994122] I [MSGID: 109022] 
> [dht-rebalance.c:2274:dht_migrate_file]
> 0-data-dht: completed migration of /this/is/a/file/path/that/
> exists/wz/wz/Npc.wz/9110012.img.xml from subvolume data-client-0 to
> data-client-2
> [2018-07-16 17:38:06.855618] I [dht-rebalance.c:4982:gf_
> defrag_get_estimates_based_on_size] 0-glusterfs: TIME: (size)
> total_processed=43108318798 tmp_cnt = 
> 55419279917056,rate_processed=446570.244043,
> elapsed = 96532.00
> [2018-07-16 17:38:06.855719] I [dht-rebalance.c:5130:gf_defrag_status_get]
> 0-glusterfs: TIME: Estimated total time to complete (size)= 124099804
> seconds, seconds left = 124003272
> [2018-07-16 17:38:06.855770] I [MSGID: 109028] 
> [dht-rebalance.c:5210:gf_defrag_status_get]
> 0-glusterfs: Rebalance is in progress. Time taken is 96532.00 secs
> [2018-07-16 17:38:06.855793] I [MSGID: 109028] 
> [dht-rebalance.c:5214:gf_defrag_status_get]
> 0-glusterfs: Files migrated: 36879, size: 12270266602, lookups: 50715,
> failures: 0, skipped: 0
> [2018-07-16 17:38:08.511064] I [dht-rebalance.c:1645:dht_migrate_file]
> 0-data-dht: /this/is/a/file/path/that/exists/wz/wz/Npc.wz/9201055.img.xml:
> attempting to move from data-client-0 to data-client-2
> [2018-07-16 17:38:08.533029] I [MSGID: 109022] 
> [dht-rebalance.c:2274:dht_migrate_file]
> 0-data-dht: completed migration of /this/is/a/file/path/that/
> exists/wz/wz/Npc.wz/205.img.xml from subvolume data-client-0 to
> data-client-2
> [2018-07-16 17:38:08.899708] I [dht-rebalance.c:4982:gf_
> defrag_get_estimates_based_on_size] 0-glusterfs: TIME: (size)
> total_processed=43108318798 tmp_cnt = 
> 55419279917056,rate_processed=446560.991961,
> elapsed = 96534.00
> [2018-07-16 17:38:08.899791] I [dht-rebalance.c:5130:gf_defrag_status_get]
> 0-glusterfs: TIME: Estimated total time to complete (size)= 124102375
> seconds, seconds left = 124005841
> [2018-07-16 17:38:08.899842] I [MSGID: 109028] 
> [dht-rebalance.c:5210:gf_defrag_status_get]
> 0-glusterfs: Rebalance is in progress. Time taken is 96534.00 secs
> [2018-07-16 17:38:08.899865] I [MSGID: 109028] 
> [dht-rebalance.c:5214:gf_defrag_status_get]
> 0-glusterfs: Files migrated: 36879, size: 12270266602, lookups: 50715,
> failures: 0, skipped: 0
>
>
> On Mon, Jul 16, 2018 at 7:37 AM, Nithya Balachandran 
> wrote:
>
>> If possible, please send the rebalance logs as well.
>>
>>
>> On 16 July 2018 at 10:14, Nithya Balachandran 
>> wrote:
>>
>>> Hi Rusty,
>>>
>>> We need the following information:
>>>
>>>1. The exact gluster version you are running
>>>2. gluster volume info 
>>>3. gluster rebalance status
>>>4. Information on the directory structure and file locations on your
>>>volume.
>>>5. How many levels of directories
>>>6. How many files and directories in each level
>>>7. How many directories and files in total (a rough estimate)
>>>8. Average file size
>>>
>>> Please note that having a rebalance running in the backgr

Re: [Gluster-users] Rebalance taking > 2 months

[Nithya Balachandran]

If possible, please send the rebalance logs as well.

On 16 July 2018 at 10:14, Nithya Balachandran  wrote:

> Hi Rusty,
>
> We need the following information:
>
>1. The exact gluster version you are running
>2. gluster volume info 
>3. gluster rebalance status
>4. Information on the directory structure and file locations on your
>volume.
>5. How many levels of directories
>6. How many files and directories in each level
>7. How many directories and files in total (a rough estimate)
>8. Average file size
>
> Please note that having a rebalance running in the background should not
> affect your volume access in any way. However I would like to know why only
> 6000 files have been scanned in 6 hours.
>
> Regards,
> Nithya
>
>
> On 16 July 2018 at 06:13, Rusty Bower  wrote:
>
>> Hey folks,
>>
>> I just added a new brick to my existing gluster volume, but *gluster
>> volume rebalance data status* is telling me the following: Estimated
>> time left for rebalance to complete : > 2 months. Please try again later.
>>
>> I already did a fix-mapping, but this thing is absolutely crawling trying
>> to rebalance everything (last estimate was ~40 years)
>>
>> Any thoughts on if this is a bug, or ways to speed this up? It's taking
>> ~6 hours to scan 6000 files, which seems unreasonably slow.
>>
>> Thanks
>> Rusty
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users