Re: [Gluster-users] [FOSDEM'18] Optimizing Software Defined Storage for the Age of Flash

[Raghavendra Gowdappa]

Note that live-streaming is available at:
https://fosdem.org/2018/schedule/streaming/

The talks will be archived too.

- Original Message -
> From: "Raghavendra Gowdappa" 
> To: "Gluster Devel" , "gluster-users" 
> 
> Sent: Monday, January 29, 2018 1:32:53 PM
> Subject: [FOSDEM'18] Optimizing Software Defined Storage for the Age of Flash
> 
> All,
> 
> Krutika, Manoj and me are presenting a talk during FOSDEM'18 [1]. Please plan
> to attend.
> 
> While we are at the event (present on 3rd and 4th), we are happy to chat with
> you anything related to Glusterfs.
> 
> The efforts leading to this talk are captured in [2].
> 
> [1] https://fosdem.org/2018/schedule/event/optimizing_sds/
> [2] https://bugzilla.redhat.com/show_bug.cgi?id=1467614
> 
> regards,
> Raghavendra
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] df does not show full volume capacity after update to 3.12.4

[Freer, Eva B.]

After OS update to CentOS 7.4 or RedHat 6.9 and update to Gluster 3.12.4, the 
‘df’ command shows only part of the available space on the mount point for 
multi-brick volumes. All nodes are at 3.12.4. This occurs on both servers and 
clients.

We have 2 different server configurations.

Configuration 1: A distributed volume of 8 bricks with 4 on each server. The 
initial configuration had 4 bricks of 59TB each with 2 on each server. Prior to 
the update to CentOS 7.4 and gluster 3.12.4, ‘df’ correctly showed the size for 
the volume as 233TB. After the update, we added 2 bricks with 1 on each server, 
but the output of ‘df’ still only listed 233TB for the volume. We added 2 more 
bricks, again with 1 on each server. The output of ‘df’ now shows 350TB, but 
the aggregate of 8 – 59TB bricks should be ~466TB.

Configuration 2: A distributed, replicated volume with 9 bricks on each server 
for a total of ~350TB of storage. After the server update to RHEL 6.9 and 
gluster 3.12.4, the volume now shows as having 50TB with ‘df’. No changes were 
made to this volume after the update.

In both cases, examining the bricks shows that the space and files are still 
there, just not reported correctly with ‘df’. All machines have been rebooted 
and the problem persists.

Any help/advice you can give on this would be greatly appreciated.

Thanks in advance.
Eva Freer

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] df does not show full volume capacity after update to 3.12.4

[Sam McLeod]

We noticed something similar.

Out of interest, does du -sh . show the same size?

--
Sam McLeod (protoporpoise on IRC)
https://smcleod.net
https://twitter.com/s_mcleod

Words are my own opinions and do not necessarily represent those of my employer 
or partners.

> On 31 Jan 2018, at 12:47 pm, Freer, Eva B.  wrote:
> 
> After OS update to CentOS 7.4 or RedHat 6.9 and update to Gluster 3.12.4, the 
> ‘df’ command shows only part of the available space on the mount point for 
> multi-brick volumes. All nodes are at 3.12.4. This occurs on both servers and 
> clients. 
>  
> We have 2 different server configurations.
>  
> Configuration 1: A distributed volume of 8 bricks with 4 on each server. The 
> initial configuration had 4 bricks of 59TB each with 2 on each server. Prior 
> to the update to CentOS 7.4 and gluster 3.12.4, ‘df’ correctly showed the 
> size for the volume as 233TB. After the update, we added 2 bricks with 1 on 
> each server, but the output of ‘df’ still only listed 233TB for the volume. 
> We added 2 more bricks, again with 1 on each server. The output of ‘df’ now 
> shows 350TB, but the aggregate of 8 – 59TB bricks should be ~466TB.
>  
> Configuration 2: A distributed, replicated volume with 9 bricks on each 
> server for a total of ~350TB of storage. After the server update to RHEL 6.9 
> and gluster 3.12.4, the volume now shows as having 50TB with ‘df’. No changes 
> were made to this volume after the update.
>  
> In both cases, examining the bricks shows that the space and files are still 
> there, just not reported correctly with ‘df’. All machines have been rebooted 
> and the problem persists.
>  
> Any help/advice you can give on this would be greatly appreciated.
>  
> Thanks in advance.
> Eva Freer
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org 
> http://lists.gluster.org/mailman/listinfo/gluster-users 
> 
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] geo-replication initial setup with existing data

[Kotresh Hiremath Ravishankar]

Hi,

Geo-replication expects the gfids (unique identifier similar to inode
number in backend file systems) to be same
for a file both on master and slave gluster volume. If the data is directly
copied by other means other than geo-replication,
gfid will be different. The crashes you are seeing is because of that.

If the data is not huge, I would recommend to sync the data from master
volume using geo-replication. The other way
is to copy directly to slave and set the gfid at the backend for each file
to be same as in master volume and then setup
geo-replication. To do the latter, follow below steps. (Note that it's not
tested extensively)


   1. Run the following commands on any one of the master nodes:

   # cd /usr/share/glusterfs/scripts/
   # sh generate-gfid-file.sh localhost:${master-vol} $PWD/get-gfid.sh
/tmp/tmp.*atyEmKyCjo*/upgrade-gfid-values.txt
   # scp /tmp/tmp.*atyEmKyCjo*/upgrade-gfid-values.txt root@${slavehost}:/tmp/

   2. Run the following commands on a slave node:

   # cd /usr/share/glusterfs/scripts/
   # sh slave-upgrade.sh localhost:${slave-vol}
/tmp/tmp.*atyEmKyCjo*/upgrade-gfid-values.txt $PWD/gsync-sync-gfid

   3. Setup geo-rep and start


Thanks,
Kotresh HR

On Mon, Jan 22, 2018 at 4:55 PM,  wrote:

> Hello everyone,
>
> i was searching for a replacement of my rsync dublication of data to
> second server and ended up with gluster geo-replicaiton. But after reading
> the documentation and lots of websites I'm still unsure how to setup a
> geo-replication without retransfering all data.
> I succeeded in converting my existing data folder to a gluster volume by
> creating a volume and using the "find /media/brick -noleaf -print0 | xargs
> --null stat" inside the mounted gluster volume folder on the master.
> But how to I have to prepare the slave? I try to do it in the same way as
> with the master, but this will only result in error messages like
> [2018-01-22 11:17:05.209027] E [repce(/media/brick):209:__call__]
> RepceClient: call failed on peer call=27401:140641732232960:1516619825.12
>method=entry_opserror=OSError
> [2018-01-22 11:17:05.209497] E 
> [syncdutils(/media/brick):331:log_raise_exception]
> : FAIL:
> Traceback (most recent call last):
>   File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/gsyncd.py",
> line 210, in main
> main_i()
>   File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/gsyncd.py",
> line 801, in main_i
> local.service_loop(*[r for r in [remote] if r])
>   File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/resource.py",
> line 1670, in service_loop
> g1.crawlwrap(oneshot=True, register_time=register_time)
>   File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py",
> line 597, in crawlwrap
> self.crawl()
>   File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py",
> line 1555, in crawl
> self.process([item[1]], 0)
>   File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py",
> line 1204, in process
> self.process_change(change, done, retry)
>   File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py",
> line 1114, in process_change
> failures = self.slave.server.entry_ops(entries)
>   File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/repce.py",
> line 228, in __call__
> return self.ins(self.meth, *a)
>   File "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/repce.py",
> line 210, in __call__
> raise res
> OSError: [Errno 42] .gfid/----0001
> I removed now all extended attributes on the slave and deleted the
> .glusterfs folder in the brick, so the system is hopefully in the inital
> state again.
> Is there any way to setup a geo-replication session without resyncing all
> data by gluster? Because this will take month with my poor connection over
> here. I'm using gluster 3.13.1 on two Ubuntu 16.04.3 LTS hosts.
>
> I hope someone can help me with some hints, thanks and best regards
> Tino
>
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>



-- 
Thanks and Regards,
Kotresh H R
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] df does not show full volume capacity after update to 3.12.4

[Freer, Eva B.]

Sam,

For du –sh on my newer volume, the result is 161T. The sum of the Used space in 
the df –h output for all the bricks is ~163T. Close enough for me to believe 
everything is there. The total for used space in the df –h of the mountpoint it 
83T, roughly half what is used.

Relevant lines from df –h on server-A:
Filesystem  Size  Used Avail Use% Mounted on
/dev/sda159T   42T   17T  72% /bricks/data_A1
/dev/sdb159T   45T   14T  77% /bricks/data_A2
/dev/sdd159T   39M   59T   1% /bricks/data_A4
/dev/sdc159T  1.9T   57T   4% /bricks/data_A3
server-A:/dataeng   350T   83T  268T  24% /dataeng

And on server-B:
Filesystem Size  Used Avail Use% Mounted on
/dev/sdb1   59T   34T   25T  58% /bricks/data_B2
/dev/sdc1   59T  2.0T   57T   4% /bricks/data_B3
/dev/sdd1   59T   39M   59T   1% /bricks/data_B4
/dev/sda1   59T   38T   22T  64% /bricks/data_B1
server-B:/dataeng  350T   83T  268T  24% /dataeng

Eva Freer

From: Sam McLeod 
Date: Tuesday, January 30, 2018 at 9:43 PM
To: Eva Freer 
Cc: "gluster-users@gluster.org" , "Greene, Tami 
McFarlin" 
Subject: Re: [Gluster-users] df does not show full volume capacity after update 
to 3.12.4

We noticed something similar.

Out of interest, does du -sh . show the same size?

--
Sam McLeod (protoporpoise on IRC)
https://smcleod.net
https://twitter.com/s_mcleod

Words are my own opinions and do not necessarily represent those of my employer 
or partners.


On 31 Jan 2018, at 12:47 pm, Freer, Eva B. 
> wrote:

After OS update to CentOS 7.4 or RedHat 6.9 and update to Gluster 3.12.4, the 
‘df’ command shows only part of the available space on the mount point for 
multi-brick volumes. All nodes are at 3.12.4. This occurs on both servers and 
clients.

We have 2 different server configurations.

Configuration 1: A distributed volume of 8 bricks with 4 on each server. The 
initial configuration had 4 bricks of 59TB each with 2 on each server. Prior to 
the update to CentOS 7.4 and gluster 3.12.4, ‘df’ correctly showed the size for 
the volume as 233TB. After the update, we added 2 bricks with 1 on each server, 
but the output of ‘df’ still only listed 233TB for the volume. We added 2 more 
bricks, again with 1 on each server. The output of ‘df’ now shows 350TB, but 
the aggregate of 8 – 59TB bricks should be ~466TB.

Configuration 2: A distributed, replicated volume with 9 bricks on each server 
for a total of ~350TB of storage. After the server update to RHEL 6.9 and 
gluster 3.12.4, the volume now shows as having 50TB with ‘df’. No changes were 
made to this volume after the update.

In both cases, examining the bricks shows that the space and files are still 
there, just not reported correctly with ‘df’. All machines have been rebooted 
and the problem persists.

Any help/advice you can give on this would be greatly appreciated.

Thanks in advance.
Eva Freer


___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] df does not show full volume capacity after update to 3.12.4

[Sam McLeod]

Very similar to what we noticed too.

I suspect it's something to do with the metadata or xattrs stored on the 
filesystem that gives quite different results from the actual file sizes.

--
Sam McLeod (protoporpoise on IRC)
https://smcleod.net
https://twitter.com/s_mcleod

Words are my own opinions and do not necessarily represent those of my employer 
or partners.

> On 31 Jan 2018, at 2:14 pm, Freer, Eva B.  wrote:
> 
> Sam,
>  
> For du –sh on my newer volume, the result is 161T. The sum of the Used space 
> in the df –h output for all the bricks is ~163T. Close enough for me to 
> believe everything is there. The total for used space in the df –h of the 
> mountpoint it 83T, roughly half what is used.
>  
> Relevant lines from df –h on server-A:
> Filesystem  Size  Used Avail Use% Mounted on
> /dev/sda159T   42T   17T  72% /bricks/data_A1
> /dev/sdb159T   45T   14T  77% /bricks/data_A2
> /dev/sdd159T   39M   59T   1% /bricks/data_A4
> /dev/sdc159T  1.9T   57T   4% /bricks/data_A3
> server-A:/dataeng   350T   83T  268T  24% /dataeng
>  
> And on server-B:
> Filesystem Size  Used Avail Use% Mounted on
> /dev/sdb1   59T   34T   25T  58% /bricks/data_B2
> /dev/sdc1   59T  2.0T   57T   4% /bricks/data_B3
> /dev/sdd1   59T   39M   59T   1% /bricks/data_B4
> /dev/sda1   59T   38T   22T  64% /bricks/data_B1
> server-B:/dataeng  350T   83T  268T  24% /dataeng
>  
> Eva Freer
>  
> From: Sam McLeod 
> Date: Tuesday, January 30, 2018 at 9:43 PM
> To: Eva Freer 
> Cc: "gluster-users@gluster.org" , "Greene, Tami 
> McFarlin" 
> Subject: Re: [Gluster-users] df does not show full volume capacity after 
> update to 3.12.4
>  
> We noticed something similar. 
>  
> Out of interest, does du -sh . show the same size?
>  
> --
> Sam McLeod (protoporpoise on IRC)
> https://smcleod.net 
> https://twitter.com/s_mcleod
> 
> Words are my own opinions and do not necessarily represent those of my 
> employer or partners.
> 
> 
>> On 31 Jan 2018, at 12:47 pm, Freer, Eva B. > > wrote:
>>  
>> After OS update to CentOS 7.4 or RedHat 6.9 and update to Gluster 3.12.4, 
>> the ‘df’ command shows only part of the available space on the mount point 
>> for multi-brick volumes. All nodes are at 3.12.4. This occurs on both 
>> servers and clients. 
>>  
>> We have 2 different server configurations.
>>  
>> Configuration 1: A distributed volume of 8 bricks with 4 on each server. The 
>> initial configuration had 4 bricks of 59TB each with 2 on each server. Prior 
>> to the update to CentOS 7.4 and gluster 3.12.4, ‘df’ correctly showed the 
>> size for the volume as 233TB. After the update, we added 2 bricks with 1 on 
>> each server, but the output of ‘df’ still only listed 233TB for the volume. 
>> We added 2 more bricks, again with 1 on each server. The output of ‘df’ now 
>> shows 350TB, but the aggregate of 8 – 59TB bricks should be ~466TB.
>>  
>> Configuration 2: A distributed, replicated volume with 9 bricks on each 
>> server for a total of ~350TB of storage. After the server update to RHEL 6.9 
>> and gluster 3.12.4, the volume now shows as having 50TB with ‘df’. No 
>> changes were made to this volume after the update.
>>  
>> In both cases, examining the bricks shows that the space and files are still 
>> there, just not reported correctly with ‘df’. All machines have been 
>> rebooted and the problem persists.
>>  
>> Any help/advice you can give on this would be greatly appreciated.
>>  
>> Thanks in advance.
>> Eva Freer
>> 
>> 
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org 
>> http://lists.gluster.org/mailman/listinfo/gluster-users 
>> 
>  

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] df does not show full volume capacity after update to 3.12.4

[Nithya Balachandran]

Hi Eva,

Can you send us the following:

gluster volume info
gluster volume status

The log files and tcpdump for df on a fresh mount point for that volume.

Thanks,
Nithya


On 31 January 2018 at 07:17, Freer, Eva B.  wrote:

> After OS update to CentOS 7.4 or RedHat 6.9 and update to Gluster 3.12.4,
> the ‘df’ command shows only part of the available space on the mount point
> for multi-brick volumes. All nodes are at 3.12.4. This occurs on both
> servers and clients.
>
>
>
> We have 2 different server configurations.
>
>
>
> Configuration 1: A distributed volume of 8 bricks with 4 on each server.
> The initial configuration had 4 bricks of 59TB each with 2 on each server.
> Prior to the update to CentOS 7.4 and gluster 3.12.4, ‘df’ correctly showed
> the size for the volume as 233TB. After the update, we added 2 bricks with
> 1 on each server, but the output of ‘df’ still only listed 233TB for the
> volume. We added 2 more bricks, again with 1 on each server. The output of
> ‘df’ now shows 350TB, but the aggregate of 8 – 59TB bricks should be ~466TB.
>
>
>
> Configuration 2: A distributed, replicated volume with 9 bricks on each
> server for a total of ~350TB of storage. After the server update to RHEL
> 6.9 and gluster 3.12.4, the volume now shows as having 50TB with ‘df’. No
> changes were made to this volume after the update.
>
>
>
> In both cases, examining the bricks shows that the space and files are
> still there, just not reported correctly with ‘df’. All machines have been
> rebooted and the problem persists.
>
>
>
> Any help/advice you can give on this would be greatly appreciated.
>
>
>
> Thanks in advance.
>
> Eva Freer
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Tiered volume performance degrades badly after a volume stop/start or system restart.

[Jeff Byers]

I am fighting this issue:

  Bug 1540376 – Tiered volume performance degrades badly after a
volume stop/start or system restart.
  https://bugzilla.redhat.com/show_bug.cgi?id=1540376

Does anyone have any ideas on what might be causing this, and
what a fix or work-around might be?

Thanks!

~ Jeff Byers ~

Tiered volume performance degrades badly after a volume
stop/start or system restart.

The degradation is very significant, making the performance of
an SSD hot tiered volume a fraction of what it was with the
HDD before tiering.

Stopping and starting the tiered volume causes the problem to
exhibit. Stopping and starting the Gluster services also does.

Nothing in the tier is being promoted or demoted, the volume
starts empty, a file is written, then read, then deleted. The
file(s) only ever exist on the hot tier.

This affects GlusterFS FUSE mounts, and also NFSv3 NFS mounts.
The problem has been reproduced in two test lab environments.
The issue was first seen using GlusterFS 3.7.18, and retested
with the same result using GlusterFS 3.12.3.

I'm using the default tiering settings, no adjustments.

Nothing of any significance appears to be being reported in
the GlusterFS logs.

Summary:

Before SSD tiering, HDD performance on a FUSE mount was 130.87
MB/sec writes, 128.53 MB/sec reads.

After SSD tiering, performance on a FUSE mount was 199.99
MB/sec writes, 257.28 MB/sec reads.

After GlusterFS volume stop/start, SSD tiering performance on
FUSE mount was 35.81 MB/sec writes, 37.33 MB/sec reads. A very
significant reduction in performance.

Detaching and reattaching the SSD tier restores the good
tiered performance.

~ Jeff Byers ~
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] df does not show full volume capacity after update to 3.12.4

[Nithya Balachandran]

I found this on the mailing list:








*I found the issue.The CentOS 7 RPMs, upon upgrade, modifies the .vol
files. Among other things, it adds "option shared-brick-count \d", using
the number of bricks in the volume.This gives you an average free space per
brick, instead of total free space in the volume.When I create a new
volume, the value of "shared-brick-count" is "1".find
/var/lib/glusterd/vols -type f|xargs sed -i -e 's/option shared-brick-count
[0-9]*/option shared-brick-count 1/g'*



Eva, can you send me the contents of the /var/lib/glusterd/ folder
from any one node so I can confirm if this is the problem?

Regards,
Nithya


On 31 January 2018 at 10:47, Nithya Balachandran 
wrote:

> Hi Eva,
>
> One more question. What version of gluster were you running before the
> upgrade?
>
> Thanks,
> Nithya
>
> On 31 January 2018 at 09:52, Nithya Balachandran 
> wrote:
>
>> Hi Eva,
>>
>> Can you send us the following:
>>
>> gluster volume info
>> gluster volume status
>>
>> The log files and tcpdump for df on a fresh mount point for that volume.
>>
>> Thanks,
>> Nithya
>>
>>
>> On 31 January 2018 at 07:17, Freer, Eva B.  wrote:
>>
>>> After OS update to CentOS 7.4 or RedHat 6.9 and update to Gluster
>>> 3.12.4, the ‘df’ command shows only part of the available space on the
>>> mount point for multi-brick volumes. All nodes are at 3.12.4. This occurs
>>> on both servers and clients.
>>>
>>>
>>>
>>> We have 2 different server configurations.
>>>
>>>
>>>
>>> Configuration 1: A distributed volume of 8 bricks with 4 on each server.
>>> The initial configuration had 4 bricks of 59TB each with 2 on each server.
>>> Prior to the update to CentOS 7.4 and gluster 3.12.4, ‘df’ correctly showed
>>> the size for the volume as 233TB. After the update, we added 2 bricks with
>>> 1 on each server, but the output of ‘df’ still only listed 233TB for the
>>> volume. We added 2 more bricks, again with 1 on each server. The output of
>>> ‘df’ now shows 350TB, but the aggregate of 8 – 59TB bricks should be ~466TB.
>>>
>>>
>>>
>>> Configuration 2: A distributed, replicated volume with 9 bricks on each
>>> server for a total of ~350TB of storage. After the server update to RHEL
>>> 6.9 and gluster 3.12.4, the volume now shows as having 50TB with ‘df’. No
>>> changes were made to this volume after the update.
>>>
>>>
>>>
>>> In both cases, examining the bricks shows that the space and files are
>>> still there, just not reported correctly with ‘df’. All machines have been
>>> rebooted and the problem persists.
>>>
>>>
>>>
>>> Any help/advice you can give on this would be greatly appreciated.
>>>
>>>
>>>
>>> Thanks in advance.
>>>
>>> Eva Freer
>>>
>>>
>>> ___
>>> Gluster-users mailing list
>>> Gluster-users@gluster.org
>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>
>>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Tiered volume performance degrades badly after a volume stop/start or system restart.

[Vlad Kopylov]

Tested it in two different environments lately with exactly same results.
Was trying to get better read performance from local mounts with
hundreds of thousands maildir email files by using SSD,
   hoping that .gluster file stat read will improve which does migrate
to hot tire.
After seeing what you described for 24 hours and confirming all move
around on the tires is done - killed it.
Here are my volume settings - maybe will be useful to spot conflicting ones.

cluster.shd-max-threads: 12
performance.rda-cache-limit: 128MB
cluster.readdir-optimize: on
cluster.read-hash-mode: 0
performance.strict-o-direct: on
cluster.lookup-unhashed: auto
performance.nl-cache: on
performance.nl-cache-timeout: 600
cluster.lookup-optimize: on
client.event-threads: 8
performance.client-io-threads: on
performance.md-cache-timeout: 600
server.event-threads: 8
features.cache-invalidation: on
features.cache-invalidation-timeout: 600
performance.stat-prefetch: on
performance.cache-invalidation: on
network.inode-lru-limit: 9
performance.cache-refresh-timeout: 10
performance.enable-least-priority: off
performance.cache-size: 2GB
cluster.nufa: on
cluster.choose-local: on
server.outstanding-rpc-limit: 128

fuse mounting 
defaults,_netdev,negative-timeout=10,attribute-timeout=30,fopen-keep-cache,direct-io-mode=enable,fetch-attempts=5

On Tue, Jan 30, 2018 at 6:29 PM, Jeff Byers  wrote:
> I am fighting this issue:
>
>   Bug 1540376 – Tiered volume performance degrades badly after a
> volume stop/start or system restart.
>   https://bugzilla.redhat.com/show_bug.cgi?id=1540376
>
> Does anyone have any ideas on what might be causing this, and
> what a fix or work-around might be?
>
> Thanks!
>
> ~ Jeff Byers ~
>
> Tiered volume performance degrades badly after a volume
> stop/start or system restart.
>
> The degradation is very significant, making the performance of
> an SSD hot tiered volume a fraction of what it was with the
> HDD before tiering.
>
> Stopping and starting the tiered volume causes the problem to
> exhibit. Stopping and starting the Gluster services also does.
>
> Nothing in the tier is being promoted or demoted, the volume
> starts empty, a file is written, then read, then deleted. The
> file(s) only ever exist on the hot tier.
>
> This affects GlusterFS FUSE mounts, and also NFSv3 NFS mounts.
> The problem has been reproduced in two test lab environments.
> The issue was first seen using GlusterFS 3.7.18, and retested
> with the same result using GlusterFS 3.12.3.
>
> I'm using the default tiering settings, no adjustments.
>
> Nothing of any significance appears to be being reported in
> the GlusterFS logs.
>
> Summary:
>
> Before SSD tiering, HDD performance on a FUSE mount was 130.87
> MB/sec writes, 128.53 MB/sec reads.
>
> After SSD tiering, performance on a FUSE mount was 199.99
> MB/sec writes, 257.28 MB/sec reads.
>
> After GlusterFS volume stop/start, SSD tiering performance on
> FUSE mount was 35.81 MB/sec writes, 37.33 MB/sec reads. A very
> significant reduction in performance.
>
> Detaching and reattaching the SSD tier restores the good
> tiered performance.
>
> ~ Jeff Byers ~
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] parallel-readdir is not recognized in GlusterFS 3.12.4

[Alan Orth]

Thank you, Raghavendra. I guess this cosmetic fix will be in 3.12.6?

I'm also looking forward to seeing stability fixes to parallel-readdir and
or readdir-ahead in 3.12.x. :)

Cheers,

On Mon, Jan 29, 2018 at 9:26 AM Raghavendra Gowdappa 
wrote:

>
>
> - Original Message -
> > From: "Pranith Kumar Karampuri" 
> > To: "Alan Orth" 
> > Cc: "gluster-users" 
> > Sent: Saturday, January 27, 2018 7:31:30 AM
> > Subject: Re: [Gluster-users] parallel-readdir is not recognized in
> GlusterFS  3.12.4
> >
> > Adding devs who work on it
> >
> > On 23 Jan 2018 10:40 pm, "Alan Orth" < alan.o...@gmail.com > wrote:
> >
> >
> >
> > Hello,
> >
> > I saw that parallel-readdir was an experimental feature in GlusterFS
> version
> > 3.10.0, became stable in version 3.11.0, and is now recommended for small
> > file workloads in the Red Hat Gluster Storage Server documentation[2].
> I've
> > successfully enabled this on one of my volumes but I notice the
> following in
> > the client mount log:
> >
> > [2018-01-23 10:24:24.048055] W [MSGID: 101174]
> > [graph.c:363:_log_if_unknown_option] 0-homes-readdir-ahead-1: option
> > 'parallel-readdir' is not recognized
> > [2018-01-23 10:24:24.048072] W [MSGID: 101174]
> > [graph.c:363:_log_if_unknown_option] 0-homes-readdir-ahead-0: option
> > 'parallel-readdir' is not recognized
> >
>
> This is fixed by patch:
> https://review.gluster.org/#/c/18572/
>
> Please note that the log message is harmless and parallel-readdir is
> enabled even with this log. You can verify that by looking into gluster
> volume configuration logged in glusterfs logs. If parallel-readdir is
> enabled, you should see a readdir-ahead translator loaded as a parent of
> each DHT subvolume.
>
> > The GlusterFS version on the client and server is 3.12.4. What is going
> on?
> >
> > [0]
> >
> https://github.com/gluster/glusterfs/blob/release-3.10/doc/release-notes/3.10.0.md
> > [1]
> >
> https://github.com/gluster/glusterfs/blob/release-3.11/doc/release-notes/3.11.0.md
> > [2]
> >
> https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.3/html/administration_guide/small_file_performance_enhancements
> >
> > Thank you,
> >
> >
> > --
> >
> >
> > Alan Orth
> > alan.o...@gmail.com
> > https://picturingjordan.com
> > https://englishbulgaria.net
> > https://mjanja.ch
> >
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > http://lists.gluster.org/mailman/listinfo/gluster-users
> >
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > http://lists.gluster.org/mailman/listinfo/gluster-users
>
-- 

Alan Orth
alan.o...@gmail.com
https://picturingjordan.com
https://englishbulgaria.net
https://mjanja.ch
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Run away memory with gluster mount

[Raghavendra Gowdappa]

- Original Message -
> From: "Dan Ragle" 
> To: "Raghavendra Gowdappa" , "Ravishankar N" 
> 
> Cc: gluster-users@gluster.org, "Csaba Henk" , "Niels de 
> Vos" , "Nithya
> Balachandran" 
> Sent: Monday, January 29, 2018 9:02:21 PM
> Subject: Re: [Gluster-users] Run away memory with gluster mount
> 
> 
> 
> On 1/29/2018 2:36 AM, Raghavendra Gowdappa wrote:
> > 
> > 
> > - Original Message -
> >> From: "Ravishankar N" 
> >> To: "Dan Ragle" , gluster-users@gluster.org
> >> Cc: "Csaba Henk" , "Niels de Vos" ,
> >> "Nithya Balachandran" ,
> >> "Raghavendra Gowdappa" 
> >> Sent: Saturday, January 27, 2018 10:23:38 AM
> >> Subject: Re: [Gluster-users] Run away memory with gluster mount
> >>
> >>
> >>
> >> On 01/27/2018 02:29 AM, Dan Ragle wrote:
> >>>
> >>> On 1/25/2018 8:21 PM, Ravishankar N wrote:
> 
> 
>  On 01/25/2018 11:04 PM, Dan Ragle wrote:
> > *sigh* trying again to correct formatting ... apologize for the
> > earlier mess.
> >
> > Having a memory issue with Gluster 3.12.4 and not sure how to
> > troubleshoot. I don't *think* this is expected behavior.
> >
> > This is on an updated CentOS 7 box. The setup is a simple two node
> > replicated layout where the two nodes act as both server and
> > client.
> >
> > The volume in question:
> >
> > Volume Name: GlusterWWW
> > Type: Replicate
> > Volume ID: 8e9b0e79-f309-4d9b-a5bb-45d065f3
> > Status: Started
> > Snapshot Count: 0
> > Number of Bricks: 1 x 2 = 2
> > Transport-type: tcp
> > Bricks:
> > Brick1: vs1dlan.mydomain.com:/glusterfs_bricks/brick1/www
> > Brick2: vs2dlan.mydomain.com:/glusterfs_bricks/brick1/www
> > Options Reconfigured:
> > nfs.disable: on
> > cluster.favorite-child-policy: mtime
> > transport.address-family: inet
> >
> > I had some other performance options in there, (increased
> > cache-size, md invalidation, etc) but stripped them out in an
> > attempt to
> > isolate the issue. Still got the problem without them.
> >
> > The volume currently contains over 1M files.
> >
> > When mounting the volume, I get (among other things) a process as such:
> >
> > /usr/sbin/glusterfs --volfile-server=localhost
> > --volfile-id=/GlusterWWW /var/www
> >
> > This process begins with little memory, but then as files are
> > accessed in the volume the memory increases. I setup a script that
> > simply reads the files in the volume one at a time (no writes). It's
> > been running on and off about 12 hours now and the resident
> > memory of the above process is already at 7.5G and continues to grow
> > slowly. If I stop the test script the memory stops growing,
> > but does not reduce. Restart the test script and the memory begins
> > slowly growing again.
> >
> > This is obviously a contrived app environment. With my intended
> > application load it takes about a week or so for the memory to get
> > high enough to invoke the oom killer.
> 
>  Can you try debugging with the statedump
>  (https://gluster.readthedocs.io/en/latest/Troubleshooting/statedump/#read-a-statedump)
>  of
>  the fuse mount process and see what member is leaking? Take the
>  statedumps in succession, maybe once initially during the I/O and
>  once the memory gets high enough to hit the OOM mark.
>  Share the dumps here.
> 
>  Regards,
>  Ravi
> >>>
> >>> Thanks for the reply. I noticed yesterday that an update (3.12.5) had
> >>> been posted so I went ahead and updated and repeated the test
> >>> overnight. The memory usage does not appear to be growing as quickly
> >>> as is was with 3.12.4, but does still appear to be growing.
> >>>
> >>> I should also mention that there is another process beyond my test app
> >>> that is reading the files from the volume. Specifically, there is an
> >>> rsync that runs from the second node 2-4 times an hour that reads from
> >>> the GlusterWWW volume mounted on node 1. Since none of the files in
> >>> that mount are changing it doesn't actually rsync anything, but
> >>> nonetheless it is running and reading the files in addition to my test
> >>> script. (It's a part of my intended production setup that I forgot was
> >>> still running.)
> >>>
> >>> The mount process appears to be gaining memory at a rate of about 1GB
> >>> every 4 hours or so. At that rate it'll take several days before it
> >>> runs the box out of memory. But I took your suggestion and made some
> >>> statedumps today anyway, about 2 hours apart, 4 total so far. It looks
> >>> like there may already be some actionable information. These are the
> >>> only 

Re: [Gluster-users] parallel-readdir is not recognized in GlusterFS 3.12.4

[Raghavendra Gowdappa]

- Original Message -
> From: "Alan Orth" 
> To: "Raghavendra Gowdappa" 
> Cc: "gluster-users" 
> Sent: Tuesday, January 30, 2018 1:37:40 PM
> Subject: Re: [Gluster-users] parallel-readdir is not recognized in GlusterFS  
> 3.12.4
> 
> Thank you, Raghavendra. I guess this cosmetic fix will be in 3.12.6?

Yes.

> 
> I'm also looking forward to seeing stability fixes to parallel-readdir and or
> readdir-ahead in 3.12.x. :)

I am not sure what caused disconnects. readdir-ahead shouldn't result in 
purging of connections. We've not seen this behavior in our internal tests (We 
have done quite a significant amount of testing on this to ascertain 
performance impact). We need to investigate this. Can you file a bug on this 
providing following information:
* log files of client and brick process
* test-case to reproduce the issue.
* Test environment (OS, volume profile info etc).

regards,
Raghavendra
> 
> Cheers,
> 
> On Mon, Jan 29, 2018 at 9:26 AM Raghavendra Gowdappa < rgowd...@redhat.com >
> wrote:
> 
> 
> 
> 
> - Original Message -
> > From: "Pranith Kumar Karampuri" < pkara...@redhat.com >
> > To: "Alan Orth" < alan.o...@gmail.com >
> > Cc: "gluster-users" < gluster-users@gluster.org >
> > Sent: Saturday, January 27, 2018 7:31:30 AM
> > Subject: Re: [Gluster-users] parallel-readdir is not recognized in
> > GlusterFS 3.12.4
> > 
> > Adding devs who work on it
> > 
> > On 23 Jan 2018 10:40 pm, "Alan Orth" < alan.o...@gmail.com > wrote:
> > 
> > 
> > 
> > Hello,
> > 
> > I saw that parallel-readdir was an experimental feature in GlusterFS
> > version
> > 3.10.0, became stable in version 3.11.0, and is now recommended for small
> > file workloads in the Red Hat Gluster Storage Server documentation[2]. I've
> > successfully enabled this on one of my volumes but I notice the following
> > in
> > the client mount log:
> > 
> > [2018-01-23 10:24:24.048055] W [MSGID: 101174]
> > [graph.c:363:_log_if_unknown_option] 0-homes-readdir-ahead-1: option
> > 'parallel-readdir' is not recognized
> > [2018-01-23 10:24:24.048072] W [MSGID: 101174]
> > [graph.c:363:_log_if_unknown_option] 0-homes-readdir-ahead-0: option
> > 'parallel-readdir' is not recognized
> > 
> 
> This is fixed by patch:
> https://review.gluster.org/#/c/18572/
> 
> Please note that the log message is harmless and parallel-readdir is enabled
> even with this log. You can verify that by looking into gluster volume
> configuration logged in glusterfs logs. If parallel-readdir is enabled, you
> should see a readdir-ahead translator loaded as a parent of each DHT
> subvolume.
> 
> > The GlusterFS version on the client and server is 3.12.4. What is going on?
> > 
> > [0]
> > https://github.com/gluster/glusterfs/blob/release-3.10/doc/release-notes/3.10.0.md
> > [1]
> > https://github.com/gluster/glusterfs/blob/release-3.11/doc/release-notes/3.11.0.md
> > [2]
> > https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.3/html/administration_guide/small_file_performance_enhancements
> > 
> > Thank you,
> > 
> > 
> > --
> > 
> > 
> > Alan Orth
> > alan.o...@gmail.com
> > https://picturingjordan.com
> > https://englishbulgaria.net
> > https://mjanja.ch
> > 
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > http://lists.gluster.org/mailman/listinfo/gluster-users
> > 
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > http://lists.gluster.org/mailman/listinfo/gluster-users
> --
> 
> 
> Alan Orth
> alan.o...@gmail.com
> https://picturingjordan.com
> https://englishbulgaria.net
> https://mjanja.ch
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users