Re: [Gluster-users] Performance: lots of small files, hdd, nvme etc.

[Hu Bert]

Hi there,
well... i know that you can read from the bricks themselves, but when
there are 7 bricks with each 1/7 of the data - which one do you
choose? ;-) Maybe ONE raid1 or raid10 and a replicate 3 performs
better than a "Number of Bricks: 5 x 3 = 15" Distributed-Replicate...

systems are under heavy load. Did some reading regarding tuning, most
of the stuff for small-file-scenario already done, did some more (just
by guessing, as documentation is a bit... poor):

performance.io-cache on
performance.io-cache-size 6GB
performance.quick-read-cache-size 6GB
group nl-cache
network.inode-lru-limit 40

Well, doesn't really help. Here are some images from one server (doing
some reads, 100K for each server per day) and one client (doing
frequent operations on the volume):

server:
cpu: https://abload.de/img/server-cpub3cf8.png
diskstats: https://abload.de/img/server-diskstats_iopsbleui.png
throughput: https://abload.de/img/server-diskstats_thro0kfma.png
disk util: https://abload.de/img/server-diskstats_utili2isl.png
network: https://abload.de/img/server-if_ens5-dayeac51.png
interrupts: https://abload.de/img/server-interruptsxycwd.png
load: https://abload.de/img/server-loadp7cid.png

client:
cpu: https://abload.de/img/client-cpuiuc0h.png
load: https://abload.de/img/client-loadsadod.png

Well, i hope you people can see why i keep asking such stuff
frequently. 4 options:

1) find some good tuning
2) check if a raid10 (with 10-14 huge hdds) performs better
3) migrate to nvme (JBOD or raid10)
4) or, if none of the above is feasible or reasonable, migrate to a
different solution (like ceph, minio, ...)


Thx for reading && best regards,

Hubert

Am Mo., 3. Apr. 2023 um 19:10 Uhr schrieb :
>
> hello
> you can read files from underlying filesystem first (ext4,xfs...), for
> ex: /srv/glusterfs//brick.
>
> as fall back you can check mounted glusterfs path, to heal missing local
> node entries. ex: /mnt/shared/www/...
>
> you need only to write to mount.glusterfs mount point.
>
>
>
>
>
> On 3/30/2023 11:26 AM, Hu Bert wrote:
> > - workload: the (un)famous "lots of small files" setting
> > - currently 70% of the of the volume is used: ~32TB
> > - file size: few KB up to 1MB
> > - so there are hundreds of millions of files (and millions of directories)
> > - each image has an ID
> > - under the base dir the IDs are split into 3 digits
> > - dir structure: /basedir/(000-999)/(000-999)/ID/[lotsoffileshere]
> > - example for ID 123456789: /basedir/123/456/123456789/default.jpg
> > - maybe this structure isn't good and e.g. this would be better:
> > /basedir/IDs/[here the files] - so millions of ID-dirs directly under
> > /basedir/
> > - frequent access to the files by webservers (nginx, tomcat): lookup
> > if file exists, read/write images etc.
> > - Strahil mentioned: "Keep in mind that negative searches (searches of
> > non-existing/deleted objects) has highest penalty." <--- that happens
> > very often...
> > - server load on high traffic days: > 100 (mostly iowait)
> > - bad are server reboots (read filesystem info etc.)
> > - really bad is a sw raid rebuild/resync
>
>
> --
> S pozdravom / Yours sincerely
> Ing. Jan Hudoba
>
> http://www.jahu.sk
> 
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://meet.google.com/cpu-eiue-hvk
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Performance: lots of small files, hdd, nvme etc.

[gluster-users]

hello
you can read files from underlying filesystem first (ext4,xfs...), for 
ex: /srv/glusterfs//brick.


as fall back you can check mounted glusterfs path, to heal missing local 
node entries. ex: /mnt/shared/www/...


you need only to write to mount.glusterfs mount point.





On 3/30/2023 11:26 AM, Hu Bert wrote:

- workload: the (un)famous "lots of small files" setting
- currently 70% of the of the volume is used: ~32TB
- file size: few KB up to 1MB
- so there are hundreds of millions of files (and millions of directories)
- each image has an ID
- under the base dir the IDs are split into 3 digits
- dir structure: /basedir/(000-999)/(000-999)/ID/[lotsoffileshere]
- example for ID 123456789: /basedir/123/456/123456789/default.jpg
- maybe this structure isn't good and e.g. this would be better:
/basedir/IDs/[here the files] - so millions of ID-dirs directly under
/basedir/
- frequent access to the files by webservers (nginx, tomcat): lookup
if file exists, read/write images etc.
- Strahil mentioned: "Keep in mind that negative searches (searches of
non-existing/deleted objects) has highest penalty." <--- that happens
very often...
- server load on high traffic days: > 100 (mostly iowait)
- bad are server reboots (read filesystem info etc.)
- really bad is a sw raid rebuild/resync



--
S pozdravom / Yours sincerely
Ing. Jan Hudoba

http://www.jahu.sk




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Performance: lots of small files, hdd, nvme etc.

[Hu Bert]

Hi Diego,

> > Just an observation: is there a performance difference between a sw
> > raid10 (10 disks -> one brick) or 5x raid1 (each raid1 a brick)
> Err... RAID10 is not 10 disks unless you stripe 5 mirrors of 2 disks.

Maybe i was unprecise?

md3 : active raid10 sdh1[7] sde1[4] sda1[0] sdg1[6] sdc1[2] sdd1[3]
sdf1[5] sdb1[1] sdi1[8] sdj1[9]
 48831518720 blocks super 1.2 512K chunks 2 near-copies [10/10] [UU]

mdadm --detail /dev/md3
/dev/md3:
  Version : 1.2
Creation Time : Fri Jan 18 08:59:51 2019
   Raid Level : raid10
[...]
   Number   Major   Minor   RaidDevice State
  0   810  active sync set-A   /dev/sda1
  1   8   171  active sync set-B   /dev/sdb1
  2   8   332  active sync set-A   /dev/sdc1
  3   8   493  active sync set-B   /dev/sdd1
  4   8   654  active sync set-A   /dev/sde1
  5   8   815  active sync set-B   /dev/sdf1
  9   8  1456  active sync set-A   /dev/sdj1
  8   8  1297  active sync set-B   /dev/sdi1
  7   8  1138  active sync set-A   /dev/sdh1
  6   8   979  active sync set-B   /dev/sdg1


> > with
> > the same disks (10TB hdd)? The heal processes on the 5xraid1-scenario
> > seems faster. Just out of curiosity...
> It should be, since the bricks are smaller. But given you're using a
> replica 3 I don't understand why you're also using RAID1: for each 10T
> of user-facing capacity you're keeping 60TB of data on disks.
> I'd ditch local RAIDs to double the space available. Unless you
> desperately need the extra read performance.

Well, long time ago we used 10TB disks as bricks (JBOD). replicate
3 setup. Then one of the bricks failed: the volume was ok (since 2
bricks were left), but after the hdd change the reset-brick produced a
very high load/iowait. So a raid1 or raid10 is the attempt to avoid
the reset-brick in favor of a sw raid rebuild - iirc this can run with
a lower priority -> less problems in the running system.


Best regards,
Hubert




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Performance: lots of small files, hdd, nvme etc.

[Diego Zuccato]

Well, you have *way* more files than we do... :)

Il 30/03/2023 11:26, Hu Bert ha scritto:


Just an observation: is there a performance difference between a sw
raid10 (10 disks -> one brick) or 5x raid1 (each raid1 a brick)

Err... RAID10 is not 10 disks unless you stripe 5 mirrors of 2 disks.


with
the same disks (10TB hdd)? The heal processes on the 5xraid1-scenario
seems faster. Just out of curiosity...
It should be, since the bricks are smaller. But given you're using a 
replica 3 I don't understand why you're also using RAID1: for each 10T 
of user-facing capacity you're keeping 60TB of data on disks.
I'd ditch local RAIDs to double the space available. Unless you 
desperately need the extra read performance.


Options Reconfigured:I'll have a look at the options you use. Maybe something can be useful 

in our case. Tks :)

--
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Performance: lots of small files, hdd, nvme etc.

[Hu Bert]

Hello there,

as Strahil suggested a separate thread might be better.

current state:
- servers with 10TB hdds
- 2 hdds build up a sw raid1
- each raid1 is a brick
- so 5 bricks per server
- Volume info (complete below):
Volume Name: workdata
Type: Distributed-Replicate
Number of Bricks: 5 x 3 = 15
Bricks:
Brick1: gls1:/gluster/md3/workdata
Brick2: gls2:/gluster/md3/workdata
Brick3: gls3:/gluster/md3/workdata
Brick4: gls1:/gluster/md4/workdata
Brick5: gls2:/gluster/md4/workdata
Brick6: gls3:/gluster/md4/workdata
etc.

- workload: the (un)famous "lots of small files" setting
- currently 70% of the of the volume is used: ~32TB
- file size: few KB up to 1MB
- so there are hundreds of millions of files (and millions of directories)
- each image has an ID
- under the base dir the IDs are split into 3 digits
- dir structure: /basedir/(000-999)/(000-999)/ID/[lotsoffileshere]
- example for ID 123456789: /basedir/123/456/123456789/default.jpg
- maybe this structure isn't good and e.g. this would be better:
/basedir/IDs/[here the files] - so millions of ID-dirs directly under
/basedir/
- frequent access to the files by webservers (nginx, tomcat): lookup
if file exists, read/write images etc.
- Strahil mentioned: "Keep in mind that negative searches (searches of
non-existing/deleted objects) has highest penalty." <--- that happens
very often...
- server load on high traffic days: > 100 (mostly iowait)
- bad are server reboots (read filesystem info etc.)
- really bad is a sw raid rebuild/resync

Some images:
https://abload.de/img/gls-diskutilfti5d.png
https://abload.de/img/gls-io6cfgp.png
https://abload.de/img/gls-throughput3oicf.png

Our conclusion: the hardware is too slow, the disks are too big. For a
future setup we need to improve the performance (or switch to a
different solution). HW-Raid-controller might be an option, but SAS
disks are not available.

Options:
- scale broader: more servers with smaller disks
- faster disks: nvme

Both are costly. Any suggestions, recommendations, ideas?

Just an observation: is there a performance difference between a sw
raid10 (10 disks -> one brick) or 5x raid1 (each raid1 a brick) with
the same disks (10TB hdd)? The heal processes on the 5xraid1-scenario
seems faster. Just out of curiosity...

whoa, lofs of text - thx for reading if you reached ths point :-)


Best regards

Hubert

Volume Name: workdata
Type: Distributed-Replicate
Volume ID: 7d1e23e5-0308-4443-a832-d36f85ff7959
Status: Started
Snapshot Count: 0
Number of Bricks: 5 x 3 = 15
Transport-type: tcp
Bricks:
Brick1: glusterpub1:/gluster/md3/workdata
Brick2: glusterpub2:/gluster/md3/workdata
Brick3: glusterpub3:/gluster/md3/workdata
Brick4: glusterpub1:/gluster/md4/workdata
Brick5: glusterpub2:/gluster/md4/workdata
Brick6: glusterpub3:/gluster/md4/workdata
Brick7: glusterpub1:/gluster/md5/workdata
Brick8: glusterpub2:/gluster/md5/workdata
Brick9: glusterpub3:/gluster/md5/workdata
Brick10: glusterpub1:/gluster/md6/workdata
Brick11: glusterpub2:/gluster/md6/workdata
Brick12: glusterpub3:/gluster/md6/workdata
Brick13: glusterpub1:/gluster/md7/workdata
Brick14: glusterpub2:/gluster/md7/workdata
Brick15: glusterpub3:/gluster/md7/workdata
Options Reconfigured:
performance.client-io-threads: off
nfs.disable: on
transport.address-family: inet
performance.cache-invalidation: on
performance.stat-prefetch: on
features.cache-invalidation-timeout: 600
features.cache-invalidation: on
performance.read-ahead: off
performance.io-cache: off
performance.quick-read: on
cluster.self-heal-window-size: 16
cluster.heal-wait-queue-length: 1
cluster.data-self-heal-algorithm: full
cluster.background-self-heal-count: 256
network.inode-lru-limit: 20
cluster.shd-max-threads: 8
server.outstanding-rpc-limit: 128
transport.listen-backlog: 100
performance.least-prio-threads: 8
performance.cache-size: 6GB
cluster.min-free-disk: 1%
performance.io-thread-count: 32
performance.write-behind-window-size: 16MB
performance.cache-max-file-size: 128MB
client.event-threads: 8
server.event-threads: 8
performance.parallel-readdir: on
performance.cache-refresh-timeout: 4
cluster.readdir-optimize: off
performance.md-cache-timeout: 600
performance.nl-cache: off
cluster.lookup-unhashed: on
cluster.shd-wait-qlength: 1
performance.readdir-ahead: on
storage.build-pgfid: off




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Performance Questions - not only small files

[Schlick Rupert]

Dear Felix,

as requested, volume info, xfs_info, fstab.

Volume Name: gfs_scratch
Type: Replicate
Volume ID: d99b6154-bf34-49d6-a06b-0e29bfc2a0fb
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: server3:/data/glusterfs_scratch/gfs_scratch_brick1
Brick2: server2:/data/glusterfs_scratch/gfs_scratch_brick1
Brick3: server1:/data/glusterfs_scratch/gfs_scratch_brick1
Options Reconfigured:
performance.parallel-readdir: on
cluster.self-heal-daemon: enable
features.uss: disable
cluster.server-quorum-type: none
performance.cache-size: 256MB
cluster.granular-entry-heal: on
storage.fips-mode-rchecksum: on
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
features.cache-invalidation: on
features.cache-invalidation-timeout: 600
performance.stat-prefetch: on
performance.cache-invalidation: on
performance.md-cache-timeout: 600
network.inode-lru-limit: 20
performance.write-behind-window-size: 128MB
diagnostics.latency-measurement: on
diagnostics.count-fop-hits: on
snap-activate-on-create: enable
auto-delete: enable

$ xfs_info /data/glusterfs_scratch
meta-data=/dev/mapper/GVG-GLV_scratch isize=512agcount=16, agsize=3276768 
blks
 =   sectsz=4096  attr=2, projid32bit=1
 =   crc=1finobt=1, sparse=1, rmapbt=0
 =   reflink=1
data =   bsize=4096   blocks=52428288, imaxpct=25
 =   sunit=32 swidth=128 blks
naming   =version 2  bsize=8192   ascii-ci=0, ftype=1
log  =internal log   bsize=4096   blocks=25599, version=2
 =   sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none   extsz=4096   blocks=0, rtextents=0

# /etc/fstab: static file system information.
#
/dev/disk/by-id/dm-uuid-LVM-IHnHtvssE4vRhHeAkSbMX7DDF8uwnlsZsaQvW2bySOYnc17QGlT7FiESTD9GloaL
 / ext4 defaults 0 0
/dev/disk/by-uuid/7aa78d34-0c19-47dc-85e2-70b54cfb9868 /boot ext4 defaults 0 0
/dev/disk/by-uuid/37A6-0D34 /boot/efi vfat defaults 0 0
/swap.img   noneswapsw  0   0
UUID=f070d8a9-ade5-451f-9bf6-53de6c1a3789 /data/glusterfs_home xfs 
inode64,noatime,nodiratime 0 0
UUID=187654ce-99f7-4aea-b2f6-701cea801b01 /data/glusterfs_sw xfs 
inode64,noatime,nodiratime 0 0
UUID=1176552b-7233-4354-be32-a6dc0e899d64 /data/glusterfs_scratch xfs 
inode64,noatime,nodiratime 0 0
server1:/gfs_home /home/gfs glusterfs defaults,_netdev 0 0
server1:/gfs_sw /sw glusterfs defaults,_netdev 0 0
server1:/gfs_scratch /scratch glusterfs defaults,_netdev 0 0
/dev/mapper/GVG-gitlab--builds /mnt/gitlab-builds ext4 
defaults,noatime,nodiratime 0 0

Cheers
Rupert




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Performance Questions - not only small files

[Felix Kölzow]

Dear Rupert,

can you provide

gluster volume info volumeName


and in addition the


xfs_info  of your brick-mountpoints

and furthermore


cat /etc/fstab


Regards,

Felix





Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Performance Questions - not only small files

[Schlick Rupert]

Dear list,

we have replicated gluster volumes much slower than the brick disks and I 
wonder if this is a configuration issue, a conceptual issue of our setup or 
really how slow gluster just is.

The setup:

  *   Three servers in a ring connected via IP over Infiniband, 100Gb/s on each 
link
  *   2 x U.2 2TB-SSDs as RAID1, on a Megaraid controller, connected via NVMe 
links on each node
  *   A 3x replicated volume on a thin-pool LVM on the SSD-RAIDs.
  *   2 x 26 cores, lots of RAM
  *   The volumes are locally mounted and used as shared disks for computation 
jobs (CPU and GPU) on the nodes.
  *   The LVM thinpool is shared with other volumes and with a cache pool for 
the hard disks.

The measurement:

  *   Iozone in automated mode, on the gluster volume (Set1) and on the mounted 
brick disk (Baseline)
  *   Compared with iozone_results_comparator.py

The issue:

  *   Smaller files are around factor 2 slower than larger files on the SSD, 
but factor 6-10 slower than larger files on the gluster volume (somewhat 
expected)
  *   Larger files are, except for certain read accesses, still factor 3-10 
slower on the gluster volume than on the SSD RAID directly, depending on the 
operation.
  *   Operations with many, admittedly smaller files (checkouts, copying, rsync 
and unpacking) can extend into hours, where they take tens of seconds to few 
minutes on disk.
  *   atop sometimes shows 9x% busy on some of the LVM block devices - and 
sometimes the avio value increases from 4-26ms to 3000-5000ms. Otherwise, there 
is nothing mentionable observed regarding system load.

This does not seem to be network bound. Accessing from a remote VM via 
glusterfs-mount is not much slower despite connecting via 10GbE instead of 
100GB. Nethogs shows sent traffic on the Infiniband interface going up to 
3kB/s for large records - 240Mb/s over a 100Gb/s Interface ...
The machine itself is mostly idle - no iowaits, a load average of 6 on 104 
logical cpus. Glusterfsd is sometimes pushing up to 10% CPU. I just do not see 
what the bottleneck should be.

Below a summary out of the compared iozone reports.

Any hints if it is worth trying to optimize this at all and where to start 
looking? I believe I have checked all the "standard" hints, but might have 
missed something.

The alternative would be a single storage node connected via nfs, sacrificing 
the probably not needed redundancy/high availability of the replicated 
filesystem.

Thanks a lot for any comments in advance,

Best
Rupert
[summary]


Operation

Write

Re-write

Read

Re-read

Random read

Random write

Backwards read

Record rewrite

Strided Read

Fwrite

Frewrite

Fread

Freread

ALL

baseline

first quartile

1260.16

1894.11

3171.44

3002.55

3679.12

2214.07

2987.15

2701.4

3361.62

1783.45

1739.6

3231.97

3405.8

2152.28

median

1474.31

2145.7

3637.6

3383.24

4167.16

2570.24

3435.44

3751.41

3967.55

2059.25

1992.64

3611.78

3936.96

3125.35

third quartile

1595.17

2318.0

3992.78

3840.0

4803.77

3013.13

3864.87

4420.33

4673.13

2354.6

2258.15

4036.73

4570.02

3915.54

minimum

194.64

960.92

1785.65

2152.15

1840.86

947.03

1491.1

1135.78

1732.88

999.01

933.54

1745.15

2083.72

194.64

maximum

2041.55

2819.84

5628.61

7342.68

7142.61

4776.16

6091.17

15908.02

6922.3

3540.22

3787.35

5933.0

7510.19

15908.02

mean val.

1408.31

2070.62

3663.03

3505.74

4231.31

2633.07

3444.43

4102.42

4029.73

2087.25

2027.23

3641.8

4113.02

3150.61

standard dev.

306.93

366.43

652.73

816.6

1048.38

781.84

796.13

2277.33

1041.12

511.69

464.33

687.27

1012.13

1335.36

ci. min. 90%

1363.0

2016.52

3566.67

3385.19

4076.54

2517.65

3326.9

3766.21

3876.03

2011.71

1958.68

3540.34

3963.6

3096.31

ci. max. 90%

1453.62

2124.71

3759.39

3626.3

4386.08

2748.5

3561.96

4438.62

4183.44

2162.79

2095.78

3743.27

4262.44

3204.91

geom. mean

1362.58

2034.2

3604.92

3422.21

4094.96

2511.93

3346.74

3637.05

3892.76

2024.11

1975.46

3576.11

3996.51

2883.44

set1

first quartile

173.4

174.43

208.58

2813.71

3498.77

139.63

51.51

91.49

124.34

201.55

199.05

187.72

2509.21

189.36

median

312.05

293.47

267.53

3134.22

4005.72

332.09

110.38

302.02

216.41

359.67

355.33

234.42

2742.35

357.31

third quartile

414.07

410.15

401.42

3446.69

4429.1

436.45

178.94

429.6

397.15

463.62

480.29

317.98

3092.73

686.35

minimum

44.11

45.6

59.07

2089.07

2004.07

9.39

8.67

2.77

32.58

38.63

41.09

26.0

1359.63

2.77

maximum

606.95

610.49

4457.8

6248.75

6153.85

666.55

374.16

696.91

2593.43

647.26

700.02

451.39

4353.18

6248.75

mean val.

308.37

309.17

544.27

3181.18

3964.96

305.76

123.49

282.89

402.01

337.11

342.11

248.76

2780.37

1010.03

standard dev.

154.66

157.37

892.64

635.03

825.98

183.62

87.03

191.43

493.51

160.15

173.1

100.31

484.33

1358.34

ci. min. 90%

285.54

285.93

412.49

3087.43

3843.02

278.65

110.65

Re: [Gluster-users] performance

[Gionatan Danti]

Il 2020-09-11 01:03 Computerisms Corporation ha scritto:

Hi Danti,

the notes are not very verbose, but looks like the following lines
were removed from their virtualization config:

  
  
  

They also enabled hyperthreading, so having 12 "cores" instead of 6
now.  Guessing that had a lot to do with it...


Ok, so I can imagine the issue was timer related...
Thanks for reporting back!

--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.da...@assyoma.it - i...@assyoma.it
GPG public key ID: FF5F32A8




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] performance

[Computerisms Corporation]

Hi Danti,

the notes are not very verbose, but looks like the following lines were 
removed from their virtualization config:


  
  
  

They also enabled hyperthreading, so having 12 "cores" instead of 6 now. 
 Guessing that had a lot to do with it...


On 2020-09-04 8:20 a.m., Gionatan Danti wrote:

Il 2020-09-04 01:00 Computerisms Corporation ha scritto:

For the sake of completeness I am reporting back that your suspicions
seem to have been validated.  I talked to the data center, they made
some changes.  we talked again some days later, and they made some
more changes, and for several days now load average on both machines
is staying consistently below 5 on both servers.  I still have some
issues to deal with, but the performance of the machines are now no
longer a problem.


Great! Any chances to know *what* they did change?
Thanks.






Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] performance

[Strahil Nikolov]

Seems that this time your email went to yahoo's spam , I am still too lazy to 
get my own domain ...

It is good to know that they fixed your issues.

Best Regards,
Strahil Nikolov






В петък, 4 септември 2020 г., 02:00:32 Гринуич+3, Computerisms Corporation 
 написа: 





Hi Strahil,

For the sake of completeness I am reporting back that your suspicions 
seem to have been validated.  I talked to the data center, they made 
some changes.  we talked again some days later, and they made some more 
changes, and for several days now load average on both machines is 
staying consistently below 5 on both servers.  I still have some issues 
to deal with, but the performance of the machines are now no longer a 
problem.

I owe you a steak and a beer, at the least; I sincerely hope chance 
allows me to pay up on that at some point in the future.

On 2020-08-21 12:02 a.m., Computerisms Corporation wrote:
> Hi Strahil,
> 
> 
>> You can use 'virt-what' binary to find if and what type of 
>> Virtualization is used.
> 
> cool, did not know about that.  trouble server:
> 
> root@moogle:/# virt-what
> hyperv
> kvm
> 
> good server:
> root@mooglian:/# virt-what
> kvm
> 
> 
> 
>> I have a suspicion you are ontop of Openstack (which uses CEPH), so I 
>> guess you can try to get more  info.
>> For example, an Openstack instance can have '0x1af4' in 
>> '/sys/block/vdX/device/vendor' (replace X with actual device letter).
>> Another check could be:
>> /usr/lib/udev/scsi_id -g -u -d /dev/vda
> 
> This command returns no output on the bad server.  Good server returns:
> 
> root@mooglian:/# /usr/lib/udev/scsi_id -g -u -d /dev/vda
> -bash: /usr/lib/udev/scsi_id: No such file or directory
> 
>> And also, you can try to take a look with smartctl from smartmontools 
>> package:
>> smartctl -a /dev/vdX
> 
> Both servers return:
> /dev/vda: Unable to detect device type
> 
> When I asked them about this earlier this week I was told the two 
> servers are identical, but I guess there is something different about 
> the server giving me trouble.  I will go back to them and see what they 
> have to say.  Thanks for pointing me at this...

> 
> 
> 
> 
> 
> Community Meeting Calendar:
> 
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
> 
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users





Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] performance

[Gionatan Danti]

Il 2020-09-04 01:00 Computerisms Corporation ha scritto:

For the sake of completeness I am reporting back that your suspicions
seem to have been validated.  I talked to the data center, they made
some changes.  we talked again some days later, and they made some
more changes, and for several days now load average on both machines
is staying consistently below 5 on both servers.  I still have some
issues to deal with, but the performance of the machines are now no
longer a problem.


Great! Any chances to know *what* they did change?
Thanks.

--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.da...@assyoma.it - i...@assyoma.it
GPG public key ID: FF5F32A8




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] performance

[Computerisms Corporation]

Hi Strahil,

For the sake of completeness I am reporting back that your suspicions 
seem to have been validated.  I talked to the data center, they made 
some changes.  we talked again some days later, and they made some more 
changes, and for several days now load average on both machines is 
staying consistently below 5 on both servers.  I still have some issues 
to deal with, but the performance of the machines are now no longer a 
problem.


I owe you a steak and a beer, at the least; I sincerely hope chance 
allows me to pay up on that at some point in the future.


On 2020-08-21 12:02 a.m., Computerisms Corporation wrote:

Hi Strahil,


You can use 'virt-what' binary to find if and what type of 
Virtualization is used.


cool, did not know about that.  trouble server:

root@moogle:/# virt-what
hyperv
kvm

good server:
root@mooglian:/# virt-what
kvm



I have a suspicion you are ontop of Openstack (which uses CEPH), so I 
guess you can try to get more  info.
For example, an Openstack instance can have '0x1af4' in 
'/sys/block/vdX/device/vendor' (replace X with actual device letter).

Another check could be:
/usr/lib/udev/scsi_id -g -u -d /dev/vda


This command returns no output on the bad server.  Good server returns:

root@mooglian:/# /usr/lib/udev/scsi_id -g -u -d /dev/vda
-bash: /usr/lib/udev/scsi_id: No such file or directory

And also, you can try to take a look with smartctl from smartmontools 
package:

smartctl -a /dev/vdX


Both servers return:
/dev/vda: Unable to detect device type

When I asked them about this earlier this week I was told the two 
servers are identical, but I guess there is something different about 
the server giving me trouble.  I will go back to them and see what they 
have to say.  Thanks for pointing me at this...






Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users





Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] performance

[Strahil Nikolov]

На 20 август 2020 г. 3:46:41 GMT+03:00, Computerisms Corporation 
 написа:
>Hi Strahil,
>
>so over the last two weeks, the system has been relatively stable.  I 
>have powered off both servers at least once, for about 5 minutes each 
>time.  server came up, auto-healed what it needed to, so all of that 
>part is working as expected.
>
>will answer things inline and follow with more questions:
>
 Hm...  OK. I guess you can try 7.7 whenever it's possible.
>>>
>>> Acknowledged.
>
>Still on my list.
>> It could be a bad firmware also. If you get the opportunity,  flash
>the firmware and bump the OS to the max.
>
>Datacenter says everything was up to date as of installation, not
>really 
>wanting them to take the servers offline for long enough to redo all
>the 
>hardware.
>
> more number of CPU cycles than needed, increasing the event thread
> count
> would enhance the performance of the Red Hat Storage Server." 
>which
>>> is
> why I had it at 8.
 Yeah, but you got only 6 cores  and they are not dedicated for
>>> gluster only. I think that you need to test with lower values.
>
>figured out my magic number for client/server threads, it should be 5. 
>I set it to 5, observed no change I could attribute to it, so tried 4, 
>and got the same thing; no visible effect.
>
> right now the only suggested parameter I haven't played with is
>the
> performance.io-thread-count, which I currently have at 64.
>>> not really sure what would be a reasonable value for my system.
>> I guess you can try to increase it a little bit and check how is it
>going.
>
>turns out if you try to set this higher than 64, you get an error
>saying 
>64 is the max.
>
 What I/O scheduler are you using for the SSDs (you can check via
>'cat
>>> /sys/block/sdX/queue/scheduler)?
>>>
>>> # cat /sys/block/vda/queue/scheduler
>>> [mq-deadline] none
>> 
>> Deadline prioritizes  reads in a 2:1 ratio /default tunings/ . You
>can consider testing 'none' if your SSDs are good.
>
>I did this.  I would say it did have a positive effect, but it was a 
>minimal one.
>
>> I see vda , please share details on the infra as this is very
>important. Virtual disks have their limitations and if you are on a VM,
> then there might be chance to increase the CPU count.
>> If you are on a VM, I would recommend you to use more (in numbers) 
>and smaller disks in stripe sets (either raid0  via mdadm,  or pure
>striped LV).
>> Also, if you are  on a VM -> there is no reason to reorder  your I/O
>requests  in the VM, just to do  it again on the Hypervisour. In such
>case 'none' can bring better performance,  but this varies on the
>workload.
>
>hm, this is a good question, one I have been asking the datacenter for
>a 
>while, but they are a little bit slippery on what exactly it is they 
>have going on there.  They advertise the servers as metal with a
>virtual 
>layer.  The virtual layer is so you can log into a site and power the 
>server down or up, mount an ISO to boot from, access a console, and
>some 
>other nifty things.  can't any more, but when they first introduced the
>
>system, you could even access the BIOS of the server.  But apparently, 
>and they swear up and down by this, it is a physical server, with real 
>dedicated SSDs and real sticks of RAM.  I have found virtio and qemu as
>
>loaded kernel modules, so certainly there is something virtual
>involved, 
>but other than that and their nifty little tools, it has always acted 
>and worked like a metal server to me.

You can use 'virt-what' binary to find if and what type of Virtualization is 
used.
I have a suspicion you are ontop of Openstack (which uses CEPH), so I guess you 
can try to get more  info.
For example, an Openstack instance can have '0x1af4' in 
'/sys/block/vdX/device/vendor' (replace X with actual device letter).
Another check could be:
/usr/lib/udev/scsi_id -g -u -d /dev/vda

And also, you can try to take a look with smartctl from smartmontools package:
smartctl -a /dev/vdX

>> All necessary data is in the file attributes on the brick. I doubt
>you will need to have access times on the brick itself. Another
>possibility is to use 'relatime'.
>
>remounted all bricks with noatime, no significant difference.
>
>>> cache unless flush-behind is on.  So seems that is a way to throw
>ram
>>> to
>>> it?  I put performance.write-behind-window-size: 512MB and
>>> performance.flush-behind: on and the whole system calmed down pretty
>>> much immediately.  could be just timing, though, will have to see
>>> tomorrow during business hours whether the system stays at a
>reasonable
>
>Tried increasing this to its max of 1GB, no noticeable change from
>512MB.
>
>The 2nd server is not acting inline with the first server.  glusterfsd 
>processes are running at 50-80% of a core each, with one brick often 
>going over 200%, where as they usually stick to 30-45% on the first 
>server.  apache processes consume as much as 90% of a core where as
>they 
>rarely go over 15% on the 

Re: [Gluster-users] performance

[Computerisms Corporation]

Hi Strahil,



You can use 'virt-what' binary to find if and what type of Virtualization is 
used.


cool, did not know about that.  trouble server:

root@moogle:/# virt-what
hyperv
kvm

good server:
root@mooglian:/# virt-what
kvm




I have a suspicion you are ontop of Openstack (which uses CEPH), so I guess you 
can try to get more  info.
For example, an Openstack instance can have '0x1af4' in 
'/sys/block/vdX/device/vendor' (replace X with actual device letter).
Another check could be:
/usr/lib/udev/scsi_id -g -u -d /dev/vda


This command returns no output on the bad server.  Good server returns:

root@mooglian:/# /usr/lib/udev/scsi_id -g -u -d /dev/vda
-bash: /usr/lib/udev/scsi_id: No such file or directory


And also, you can try to take a look with smartctl from smartmontools package:
smartctl -a /dev/vdX


Both servers return:
/dev/vda: Unable to detect device type

When I asked them about this earlier this week I was told the two 
servers are identical, but I guess there is something different about 
the server giving me trouble.  I will go back to them and see what they 
have to say.  Thanks for pointing me at this...






Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] performance

[Computerisms Corporation]

Hi Strahil,

so over the last two weeks, the system has been relatively stable.  I 
have powered off both servers at least once, for about 5 minutes each 
time.  server came up, auto-healed what it needed to, so all of that 
part is working as expected.


will answer things inline and follow with more questions:


Hm...  OK. I guess you can try 7.7 whenever it's possible.


Acknowledged.


Still on my list.

It could be a bad firmware also. If you get the opportunity,  flash the 
firmware and bump the OS to the max.


Datacenter says everything was up to date as of installation, not really 
wanting them to take the servers offline for long enough to redo all the 
hardware.



more number of CPU cycles than needed, increasing the event thread
count
would enhance the performance of the Red Hat Storage Server."  which

is

why I had it at 8.

Yeah, but you got only 6 cores  and they are not dedicated for

gluster only. I think that you need to test with lower values.


figured out my magic number for client/server threads, it should be 5. 
I set it to 5, observed no change I could attribute to it, so tried 4, 
and got the same thing; no visible effect.



right now the only suggested parameter I haven't played with is the
performance.io-thread-count, which I currently have at 64.

not really sure what would be a reasonable value for my system.

I guess you can try to increase it a little bit and check how is it going.


turns out if you try to set this higher than 64, you get an error saying 
64 is the max.



What I/O scheduler are you using for the SSDs (you can check via 'cat

/sys/block/sdX/queue/scheduler)?

# cat /sys/block/vda/queue/scheduler
[mq-deadline] none


Deadline prioritizes  reads in a 2:1 ratio /default tunings/ . You can consider 
testing 'none' if your SSDs are good.


I did this.  I would say it did have a positive effect, but it was a 
minimal one.



I see vda , please share details on the infra as this is very important. 
Virtual disks have their limitations and if you are on a VM,  then there might 
be chance to increase the CPU count.
If you are on a VM, I would recommend you to use more (in numbers)  and smaller 
disks in stripe sets (either raid0  via mdadm,  or pure striped LV).
Also, if you are  on a VM -> there is no reason to reorder  your I/O requests  
in the VM, just to do  it again on the Hypervisour. In such case 'none' can bring 
better performance,  but this varies on the workload.


hm, this is a good question, one I have been asking the datacenter for a 
while, but they are a little bit slippery on what exactly it is they 
have going on there.  They advertise the servers as metal with a virtual 
layer.  The virtual layer is so you can log into a site and power the 
server down or up, mount an ISO to boot from, access a console, and some 
other nifty things.  can't any more, but when they first introduced the 
system, you could even access the BIOS of the server.  But apparently, 
and they swear up and down by this, it is a physical server, with real 
dedicated SSDs and real sticks of RAM.  I have found virtio and qemu as 
loaded kernel modules, so certainly there is something virtual involved, 
but other than that and their nifty little tools, it has always acted 
and worked like a metal server to me.



All necessary data is in the file attributes on the brick. I doubt you will 
need to have access times on the brick itself. Another possibility is to use 
'relatime'.


remounted all bricks with noatime, no significant difference.


cache unless flush-behind is on.  So seems that is a way to throw ram
to
it?  I put performance.write-behind-window-size: 512MB and
performance.flush-behind: on and the whole system calmed down pretty
much immediately.  could be just timing, though, will have to see
tomorrow during business hours whether the system stays at a reasonable


Tried increasing this to its max of 1GB, no noticeable change from 512MB.

The 2nd server is not acting inline with the first server.  glusterfsd 
processes are running at 50-80% of a core each, with one brick often 
going over 200%, where as they usually stick to 30-45% on the first 
server.  apache processes consume as much as 90% of a core where as they 
rarely go over 15% on the first server, and they frequently stack up to 
having more than 100 running at once, which drives load average up to 
40-60.  It's very much like the first server was before I found the 
flush-behind setting, but not as bad; at least it isn't going completely 
non-responsive.


Additionally, it is still taking an excessive time to load the first 
page of most sites.  I am guessing I need to increase read speeds to fix 
this, so I have played with 
performance.io-cache/cache-max-file-size(slight positive change), 
read-ahead/read-ahead-page-count(negative change till page count set to 
max of 16, then no noticeable difference), and 
rda-cache-limit/rda-request-size(minimal positive effect).  I still have 
RAM to spare, so 

Re: [Gluster-users] performance

[Artem Russakovskii]

Hmm, in our case of running gluster across Linode block storage (which
itself runs inside Ceph, as I found out), the only thing that helped with
the hangs so far was defragmenting xfs.

I tried changing many things, including the scheduler to "none" and
this performance.write-behind-window-size setting, and nothing seemed to
help or provide any meaningful difference.

Sincerely,
Artem

--
Founder, Android Police , APK Mirror
, Illogical Robot LLC
beerpla.net | @ArtemR 


On Fri, Aug 7, 2020 at 11:28 AM Computerisms Corporation <
b...@computerisms.ca> wrote:

> Hi Artem and others,
>
> Happy to report the system has been relatively stable for the remainder
> of the week.  I have one wordpress site that seems to get hung processes
> when someone logs in with an incorrect password.  Since it is only one,
> and reliably reproduceable, I am not sure if the issue is to do with
> Gluster or Wordpress itself, but afaik it was not doing it some months
> back before the system was using Gluster so I am guessing some combo of
> both.
>
> Regardless, that is the one and only time apache processes stacked up to
> over 150, and that still only brought the load average up to just under
> 25; the system did go a bit sluggish, but remained fairly responsive
> throughout until I restarted apache.  Otherwise 15 minute load average
> consistently runs between 8 and 11 during peak hours and between 4 and 7
> during off hours, and other than the one time I have not seen the
> one-minute load average go over 15.  all resources still spike to full
> capacity from time to time, but it never remains that way for long like
> it did before.
>
> For site responsiveness, first visit to any given site is quite slow,
> like 3-5 seconds on straight html pages, 10-15 seconds for some of the
> more bloated WP themes, but clicking links within the site after the
> first page is loaded is relatively quick, like 1 second on straight html
> pages, and ~5-6 seconds on the bloated themes.  Again, not sure if that
> is a Gluster related thing or something else.
>
> So, still holding my breath a bit, but seems this solution is working,
> at least for me.  I haven't played with any of the other settings yet to
> see if I can improve it further, probably will next week.  thinking to
> increase the write behind window size further to see what happens, as
> well as play with the settings suggested by Strahil.
>
> On 2020-08-05 5:28 p.m., Artem Russakovskii wrote:
> > I'm very curious whether these improvements hold up over the next few
> > days. Please report back.
> >
> > Sincerely,
> > Artem
> >
> > --
> > Founder, Android Police , APK Mirror
> > , Illogical Robot LLC
> > beerpla.net  | @ArtemR 
> >
> >
> > On Wed, Aug 5, 2020 at 9:44 AM Computerisms Corporation
> > mailto:b...@computerisms.ca>> wrote:
> >
> > Hi List,
> >
> >  > So, we just moved into a quieter time of the day, but maybe I just
> >  > stumbled onto something.  I was trying to figure out if/how I
> could
> >  > throw more RAM at the problem.  gluster docs says write behind is
> > not a
> >  > cache unless flush-behind is on.  So seems that is a way to throw
> > ram to
> >  > it?  I put performance.write-behind-window-size: 512MB and
> >  > performance.flush-behind: on and the whole system calmed down
> pretty
> >  > much immediately.  could be just timing, though, will have to see
> >  > tomorrow during business hours whether the system stays at a
> > reasonable
> >  > load.
> >
> > so reporting back that this seems to have definitely had a
> significant
> > positive effect.
> >
> > So far today I have not seen the load average climb over 13 with the
> > 15minute average hovering around 7.  cpus are still spiking from
> > time to
> > time, but they are not staying maxed out all the time, and
> frequently I
> > am seeing brief periods of up to 80% idle.  glusterfs process still
> > spiking up to 180% or so, but consistently running around 70%, and
> the
> > brick processes still spiking up to 70-80%, but consistently running
> > around 20%.  Disk has only been above 50% in atop once so far today
> > when
> > it spiked up to 92%, and still lots of RAM left over.  So far nload
> > even
> > seems indicates I could get away with a 100Mbit network connection.
> > Websites are snappy relative to what they were, still a bit sluggish
> on
> > the first page of any given site, but tolerable or close to.  Apache
> > processes are opening and closing right away, instead of stacking up.
> >
> > Overall, system is performing pretty much like I would expect it to
> > without gluster.  I haven't played with any of the other settings
> yet,
> > just going to leave it like this for a 

Re: [Gluster-users] performance

[Computerisms Corporation]

Hi Artem and others,

Happy to report the system has been relatively stable for the remainder 
of the week.  I have one wordpress site that seems to get hung processes 
when someone logs in with an incorrect password.  Since it is only one, 
and reliably reproduceable, I am not sure if the issue is to do with 
Gluster or Wordpress itself, but afaik it was not doing it some months 
back before the system was using Gluster so I am guessing some combo of 
both.


Regardless, that is the one and only time apache processes stacked up to 
over 150, and that still only brought the load average up to just under 
25; the system did go a bit sluggish, but remained fairly responsive 
throughout until I restarted apache.  Otherwise 15 minute load average 
consistently runs between 8 and 11 during peak hours and between 4 and 7 
during off hours, and other than the one time I have not seen the 
one-minute load average go over 15.  all resources still spike to full 
capacity from time to time, but it never remains that way for long like 
it did before.


For site responsiveness, first visit to any given site is quite slow, 
like 3-5 seconds on straight html pages, 10-15 seconds for some of the 
more bloated WP themes, but clicking links within the site after the 
first page is loaded is relatively quick, like 1 second on straight html 
pages, and ~5-6 seconds on the bloated themes.  Again, not sure if that 
is a Gluster related thing or something else.


So, still holding my breath a bit, but seems this solution is working, 
at least for me.  I haven't played with any of the other settings yet to 
see if I can improve it further, probably will next week.  thinking to 
increase the write behind window size further to see what happens, as 
well as play with the settings suggested by Strahil.


On 2020-08-05 5:28 p.m., Artem Russakovskii wrote:
I'm very curious whether these improvements hold up over the next few 
days. Please report back.


Sincerely,
Artem

--
Founder, Android Police , APK Mirror 
, Illogical Robot LLC

beerpla.net  | @ArtemR 


On Wed, Aug 5, 2020 at 9:44 AM Computerisms Corporation 
mailto:b...@computerisms.ca>> wrote:


Hi List,

 > So, we just moved into a quieter time of the day, but maybe I just
 > stumbled onto something.  I was trying to figure out if/how I could
 > throw more RAM at the problem.  gluster docs says write behind is
not a
 > cache unless flush-behind is on.  So seems that is a way to throw
ram to
 > it?  I put performance.write-behind-window-size: 512MB and
 > performance.flush-behind: on and the whole system calmed down pretty
 > much immediately.  could be just timing, though, will have to see
 > tomorrow during business hours whether the system stays at a
reasonable
 > load.

so reporting back that this seems to have definitely had a significant
positive effect.

So far today I have not seen the load average climb over 13 with the
15minute average hovering around 7.  cpus are still spiking from
time to
time, but they are not staying maxed out all the time, and frequently I
am seeing brief periods of up to 80% idle.  glusterfs process still
spiking up to 180% or so, but consistently running around 70%, and the
brick processes still spiking up to 70-80%, but consistently running
around 20%.  Disk has only been above 50% in atop once so far today
when
it spiked up to 92%, and still lots of RAM left over.  So far nload
even
seems indicates I could get away with a 100Mbit network connection.
Websites are snappy relative to what they were, still a bit sluggish on
the first page of any given site, but tolerable or close to.  Apache
processes are opening and closing right away, instead of stacking up.

Overall, system is performing pretty much like I would expect it to
without gluster.  I haven't played with any of the other settings yet,
just going to leave it like this for a day.

I have to admit I am a little bit suspicious.  I have been arguing with
Gluster for a very long time, and I have never known it to play this
nice.  kind feels like when your girl tells you she is "fine";
conversation has stopped, but you aren't really sure if it's done...

 >
 > I will still test the other options you suggested tonight,
though, this
 > is probably too good to be true.
 >
 > Can't thank you enough for your input, Strahil, your help is truly
 > appreciated!
 >
 >
 >
 >
 >
 >
 >>
 
 
  Best Regards,
  Strahil Nikolov
 
 >>> 
 >>>
 >>>
 >>>
 >>> Community Meeting Calendar:
 >>>
 >>> Schedule -
 >>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
 >>> Bridge: https://bluejeans.com/441850968
 >>>
 

Re: [Gluster-users] performance

[Strahil Nikolov]

На 5 август 2020 г. 4:53:34 GMT+03:00, Computerisms Corporation 
 написа:
>Hi Strahil,
>
>thanks again for sticking with me on this.
>> Hm...  OK. I guess you can try 7.7 whenever it's possible.
>
>Acknowledged.
>
>>> Perhaps I am not understanding it correctly.  I tried these
>suggestions
>>>
>>> before and it got worse, not better.  so I have been operating under
>>> the
>>> assumption that maybe these guidelines are not appropriate for newer
>>> versions.
>> 
>> Actually, the  settings are not changed  much,  so they should work
>for you.
>
>Okay, then maybe I am doing something incorrectly, or not understanding
>
>some fundamental piece of things that I should be.

To be honest, the documentation seems pretty useless to me.

> Interestingly, mostly because it is not something I have ever
> experienced before, software interrupts sit between 1 and 5 on
>each
> core, but the last core is usually sitting around 20.  Have never
> encountered a high load average where the si number was ever
> significant.  I have googled the crap out of that (as well as
>>> gluster
> performance in general), there are nearly limitless posts about
>what
>>> it
>
> is, but have yet to see one thing to explain what to do about it.
>> 
>> This is happening on all nodes ?
>> I got a similar situation caused by bad NIC  (si in top was way
>high), but the chance for bad NIC on all servers is very low.
>> You can still patch OS + Firmware on your next maintenance.
>
>Yes, but it's not to the same extreme.  The other node is currently not
>
>actually serving anything to the internet, so right now it's only 
>function is replicated gluster and databases.  On the 2nd node there is
>
>also one core, the first one in this case as opposed to the last one on
>
>the main node, but it sits between 10 and 15 instead of 20 and 25, and 
>the remaining cores will be between 0 and 2 instead of 1 and 5.


>I have no evidence of any bad hardware, and these servers were both 
>commissioned only within the last couple of months.  But will still
>poke 
>around on this path.

It could be a bad firmware also. If you get the opportunity,  flash the 
firmware and bump the OS to the max.


>>> more number of CPU cycles than needed, increasing the event thread
>>> count
>>> would enhance the performance of the Red Hat Storage Server."  which
>is
>>>
>>> why I had it at 8.
>> 
>> Yeah, but you got only 6 cores  and they are not dedicated for
>gluster only. I think that you need to test with lower values.
>
>Okay, I will change these values a few times over the next couple of 
>hours and see what happens.
>
>>> right now the only suggested parameter I haven't played with is the
>>> performance.io-thread-count, which I currently have at 64.
>> 
>> I think that as you have SSDs only,  you might have some results by
>changing this one.
>
>Okay, will also modify this incrementally.  do you think it can go 
>higher?  I think I got this number from a thread on this list, but I am
>
>not really sure what would be a reasonable value for my system.

I guess you can try to increase it a little bit and check how is it going.

>>>
>>> For what it's worth, I am running ext4 as my underlying fs and I
>have
>>> read a few times that XFS might have been a better choice.  But that
>is
>>>
>>> not a trivial experiment to make at this time with the system in
>>> production.  It's one thing (and still a bad thing to be sure) to
>>> semi-bork the system for an hour or two while I play with
>>> configurations, but would take a day or so offline to reformat and
>>> restore the data.
>> 
>> XFS  should bring better performance, but if the issue is not in FS
>->  it won't make  a change...
>> What I/O scheduler are you using for the SSDs (you can check via 'cat
>/sys/block/sdX/queue/scheduler)?
>
># cat /sys/block/vda/queue/scheduler
>[mq-deadline] none

Deadline prioritizes  reads in a 2:1 ratio /default tunings/ . You can consider 
testing 'none' if your SSDs are good.

I see vda , please share details on the infra as this is very important. 
Virtual disks have their limitations and if you are on a VM,  then there might 
be chance to increase the CPU count.
If you are on a VM, I would recommend you to use more (in numbers)  and smaller 
disks in stripe sets (either raid0  via mdadm,  or pure striped LV).
Also, if you are  on a VM -> there is no reason to reorder  your I/O requests  
in the VM, just to do  it again on the Hypervisour. In such case 'none' can 
bring better performance,  but this varies on the workload.



>>> in the past I have tried 2, 4, 8, 16, and 32.  Playing with just
>those
>>> I
>>> never noticed that any of them made any difference.  Though I might
>>> have
>>> some different options now than I did then, so might try these again
>>> throughout the day...
>> 
>> Are you talking about server or client event threads (or both)?
>
>It never occurred to me to set them to different values.  so far when I
>
>set one I set the other 

Re: [Gluster-users] performance

[Artem Russakovskii]

I'm very curious whether these improvements hold up over the next few days.
Please report back.

Sincerely,
Artem

--
Founder, Android Police , APK Mirror
, Illogical Robot LLC
beerpla.net | @ArtemR 


On Wed, Aug 5, 2020 at 9:44 AM Computerisms Corporation 
wrote:

> Hi List,
>
> > So, we just moved into a quieter time of the day, but maybe I just
> > stumbled onto something.  I was trying to figure out if/how I could
> > throw more RAM at the problem.  gluster docs says write behind is not a
> > cache unless flush-behind is on.  So seems that is a way to throw ram to
> > it?  I put performance.write-behind-window-size: 512MB and
> > performance.flush-behind: on and the whole system calmed down pretty
> > much immediately.  could be just timing, though, will have to see
> > tomorrow during business hours whether the system stays at a reasonable
> > load.
>
> so reporting back that this seems to have definitely had a significant
> positive effect.
>
> So far today I have not seen the load average climb over 13 with the
> 15minute average hovering around 7.  cpus are still spiking from time to
> time, but they are not staying maxed out all the time, and frequently I
> am seeing brief periods of up to 80% idle.  glusterfs process still
> spiking up to 180% or so, but consistently running around 70%, and the
> brick processes still spiking up to 70-80%, but consistently running
> around 20%.  Disk has only been above 50% in atop once so far today when
> it spiked up to 92%, and still lots of RAM left over.  So far nload even
> seems indicates I could get away with a 100Mbit network connection.
> Websites are snappy relative to what they were, still a bit sluggish on
> the first page of any given site, but tolerable or close to.  Apache
> processes are opening and closing right away, instead of stacking up.
>
> Overall, system is performing pretty much like I would expect it to
> without gluster.  I haven't played with any of the other settings yet,
> just going to leave it like this for a day.
>
> I have to admit I am a little bit suspicious.  I have been arguing with
> Gluster for a very long time, and I have never known it to play this
> nice.  kind feels like when your girl tells you she is "fine";
> conversation has stopped, but you aren't really sure if it's done...
>
> >
> > I will still test the other options you suggested tonight, though, this
> > is probably too good to be true.
> >
> > Can't thank you enough for your input, Strahil, your help is truly
> > appreciated!
> >
> >
> >
> >
> >
> >
> >>
> 
> 
>  Best Regards,
>  Strahil Nikolov
> 
> >>> 
> >>>
> >>>
> >>>
> >>> Community Meeting Calendar:
> >>>
> >>> Schedule -
> >>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> >>> Bridge: https://bluejeans.com/441850968
> >>>
> >>> Gluster-users mailing list
> >>> Gluster-users@gluster.org
> >>> https://lists.gluster.org/mailman/listinfo/gluster-users
> > 
> >
> >
> >
> > Community Meeting Calendar:
> >
> > Schedule -
> > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> > Bridge: https://bluejeans.com/441850968
> >
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > https://lists.gluster.org/mailman/listinfo/gluster-users
> 
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] performance

[Computerisms Corporation]

Hi List,

So, we just moved into a quieter time of the day, but maybe I just 
stumbled onto something.  I was trying to figure out if/how I could 
throw more RAM at the problem.  gluster docs says write behind is not a 
cache unless flush-behind is on.  So seems that is a way to throw ram to 
it?  I put performance.write-behind-window-size: 512MB and 
performance.flush-behind: on and the whole system calmed down pretty 
much immediately.  could be just timing, though, will have to see 
tomorrow during business hours whether the system stays at a reasonable 
load.


so reporting back that this seems to have definitely had a significant 
positive effect.


So far today I have not seen the load average climb over 13 with the 
15minute average hovering around 7.  cpus are still spiking from time to 
time, but they are not staying maxed out all the time, and frequently I 
am seeing brief periods of up to 80% idle.  glusterfs process still 
spiking up to 180% or so, but consistently running around 70%, and the 
brick processes still spiking up to 70-80%, but consistently running 
around 20%.  Disk has only been above 50% in atop once so far today when 
it spiked up to 92%, and still lots of RAM left over.  So far nload even 
seems indicates I could get away with a 100Mbit network connection. 
Websites are snappy relative to what they were, still a bit sluggish on 
the first page of any given site, but tolerable or close to.  Apache 
processes are opening and closing right away, instead of stacking up.


Overall, system is performing pretty much like I would expect it to 
without gluster.  I haven't played with any of the other settings yet, 
just going to leave it like this for a day.


I have to admit I am a little bit suspicious.  I have been arguing with 
Gluster for a very long time, and I have never known it to play this 
nice.  kind feels like when your girl tells you she is "fine"; 
conversation has stopped, but you aren't really sure if it's done...




I will still test the other options you suggested tonight, though, this 
is probably too good to be true.


Can't thank you enough for your input, Strahil, your help is truly 
appreciated!












Best Regards,
Strahil Nikolov






Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users





Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users





Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] performance

[Strahil Nikolov]

На 4 август 2020 г. 22:47:44 GMT+03:00, Computerisms Corporation 
 написа:
>Hi Strahil, thanks for your response.
>
>>>
>>> I have compiled gluster 7.6 from sources on both servers.
>> 
>> There  is a 7.7 version which is fixing somw stuff. Why do you have
>to compile it from source ?
>
>Because I have often found with other stuff in the past compiling from 
>source makes a bunch of problems go away.  software generally works the
>
>way the developers expect it to if you use the sources, so they are 
>better able to help if required.  so now I generally compile most of my
>
>center-piece softwares and use packages for all the supporting stuff.

Hm...  OK. I guess you can try 7.7 whenever it's possible.

>> 
>>> Servers are 6core/3.4Ghz with 32 GB RAM, no swap, and SSD and
>gigabit
>>> network connections.  They are running debian, and are being used as
>>> redundant web servers.  There is some 3Million files on the Gluster
>>> Storage averaging 130KB/file.
>> 
>> This type of workload is called 'metadata-intensive'.
>
>does this mean the metadata-cache group file would be a good one to 
>enable?  will try.
>
>waited 10 minutes, no change that I can see.
>
>> There are some recommendations for this type of workload:
>>
>https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3/html/administration_guide/small_file_performance_enhancements
>> 
>> Keep an eye on the section that mentions dirty-ratio = 5
> = 2.
>
>I have actually read that whole manual, and specifically that page 
>several times.  And also this one:
>
>https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.1/html/administration_guide/small_file_performance_enhancements
>
>Perhaps I am not understanding it correctly.  I tried these suggestions
>
>before and it got worse, not better.  so I have been operating under
>the 
>assumption that maybe these guidelines are not appropriate for newer 
>versions.

Actually, the  settings are not changed  much,  so they should work for you.

>But will try again.  adjusting the dirty ratios.
>
>Load average went from around 15 to 35 in about 2-3 minutes, but 20 
>minutes later, it is back down to 20.  It may be having a minimal 
>positive impact on cpu, though, I haven't see the main glusterfs go
>over 
>200% since I changed this, an the brick processes are hovering just 
>below 50%  where they were consistently above 50% before.  Might just
>be 
>time of day with the system not as busy.
>
>after watching for 30 minutes, load average is fluctuating between 10 
>and 30, but cpu idle appears marginally better on average than it was.
>
>>> Interestingly, mostly because it is not something I have ever
>>> experienced before, software interrupts sit between 1 and 5 on each
>>> core, but the last core is usually sitting around 20.  Have never
>>> encountered a high load average where the si number was ever
>>> significant.  I have googled the crap out of that (as well as
>gluster
>>> performance in general), there are nearly limitless posts about what
>it
>>>
>>> is, but have yet to see one thing to explain what to do about it.

This is happening on all nodes ?
I got a similar situation caused by bad NIC  (si in top was way high), but the 
chance for bad NIC on all servers is very low.
You can still patch OS + Firmware on your next maintenance.

>> There is an explanation  about that in the link I provided above:
>> 
>> Configuring a higher event threads value than the available
>processing units could again cause context switches on these threads.
>As a result reducing the number deduced from the previous step to a
>number that is less that the available processing units is recommended.
>
>Okay, again, have played with these numbers before and it did not pan 
>out as expected.  if I understand it correctly, I have 3 brick
>processes 
>(glusterfsd), so the "deduced" number should be 3, and I should set it 
>lower than that, so 2.  but it also says "If a specific thread consumes
>
>more number of CPU cycles than needed, increasing the event thread
>count 
>would enhance the performance of the Red Hat Storage Server."  which is
>
>why I had it at 8.

Yeah, but you got only 6 cores  and they are not dedicated for gluster only. I 
think that you need to test with lower values.

>but will set it to 2 now.  load average is at 17 to start, waiting a 
>while to see what happens.
>
>so 15 minutes later, load average is currently 12, but is fluctuating 
>between 10 and 20, have seen no significant change in cpu usage or 
>anything else in top.
>
>now try also changing server.outstanding-rpc-limit to 256 and wait.
>
>15 minutes later; load has been above 30 but is currently back down to 
>12.  no significant change in cpu.  try increasing to 512 and wait.
>
>15 minutes later, load average is 50.  no signficant difference in cpu.
>
>Software interrupts remain around where they were.  wa from top remains
>
>about where it was.  not sure why load average is climbing so high. 
>changing rpc-limit 

Re: [Gluster-users] performance

[Computerisms Corporation]

Hi Strahil,

thanks again for sticking with me on this.

Hm...  OK. I guess you can try 7.7 whenever it's possible.


Acknowledged.


Perhaps I am not understanding it correctly.  I tried these suggestions

before and it got worse, not better.  so I have been operating under
the
assumption that maybe these guidelines are not appropriate for newer
versions.


Actually, the  settings are not changed  much,  so they should work for you.


Okay, then maybe I am doing something incorrectly, or not understanding 
some fundamental piece of things that I should be.



Interestingly, mostly because it is not something I have ever
experienced before, software interrupts sit between 1 and 5 on each
core, but the last core is usually sitting around 20.  Have never
encountered a high load average where the si number was ever
significant.  I have googled the crap out of that (as well as

gluster

performance in general), there are nearly limitless posts about what

it


is, but have yet to see one thing to explain what to do about it.


This is happening on all nodes ?
I got a similar situation caused by bad NIC  (si in top was way high), but the 
chance for bad NIC on all servers is very low.
You can still patch OS + Firmware on your next maintenance.


Yes, but it's not to the same extreme.  The other node is currently not 
actually serving anything to the internet, so right now it's only 
function is replicated gluster and databases.  On the 2nd node there is 
also one core, the first one in this case as opposed to the last one on 
the main node, but it sits between 10 and 15 instead of 20 and 25, and 
the remaining cores will be between 0 and 2 instead of 1 and 5.


I have no evidence of any bad hardware, and these servers were both 
commissioned only within the last couple of months.  But will still poke 
around on this path.



more number of CPU cycles than needed, increasing the event thread
count
would enhance the performance of the Red Hat Storage Server."  which is

why I had it at 8.


Yeah, but you got only 6 cores  and they are not dedicated for gluster only. I 
think that you need to test with lower values.


Okay, I will change these values a few times over the next couple of 
hours and see what happens.



right now the only suggested parameter I haven't played with is the
performance.io-thread-count, which I currently have at 64.


I think that as you have SSDs only,  you might have some results by changing 
this one.


Okay, will also modify this incrementally.  do you think it can go 
higher?  I think I got this number from a thread on this list, but I am 
not really sure what would be a reasonable value for my system.




For what it's worth, I am running ext4 as my underlying fs and I have
read a few times that XFS might have been a better choice.  But that is

not a trivial experiment to make at this time with the system in
production.  It's one thing (and still a bad thing to be sure) to
semi-bork the system for an hour or two while I play with
configurations, but would take a day or so offline to reformat and
restore the data.


XFS  should bring better performance, but if the issue is not in FS ->  it 
won't make  a change...
What I/O scheduler are you using for the SSDs (you can check via 'cat 
/sys/block/sdX/queue/scheduler)?


# cat /sys/block/vda/queue/scheduler
[mq-deadline] none


in the past I have tried 2, 4, 8, 16, and 32.  Playing with just those
I
never noticed that any of them made any difference.  Though I might
have
some different options now than I did then, so might try these again
throughout the day...


Are you talking about server or client event threads (or both)?


It never occurred to me to set them to different values.  so far when I 
set one I set the other to the same value.





Thanks again for your time Strahil, if you have any more thoughts would

love to hear them.


Can you check if you use 'noatime' for the bricks ? It won't bring any effect 
on the CPU side, but it might help with the I/O.


I checked into this, and I have nodiratime set, but not noatime.  from 
what I can gather, it should provide nearly the same benefit performance 
wise while leaving the atime attribute on the files.  Never know, I may 
decide I want those at some point in the future.



I see that your indicator for high load  is loadavg,  but have you actually 
checked how many processes are in 'R' or 'D' state ?
Some  monitoring checks can raise loadavg artificially.


occasionally a batch of processes will be in R state, and I see the D 
state show up from time to time, but mostly everything is S.



Also,  are you using software mirroring (either mdadm or striped/mirrored LVs )?


No, single disk.  And I opted to not put the gluster on a thinLVM, as I 
don't see myself using the lvm snapshots in this scenario.


So, we just moved into a quieter time of the day, but maybe I just 
stumbled onto something.  I was trying to figure out if/how I could 
throw more RAM at the problem.  gluster 

Re: [Gluster-users] performance

[Computerisms Corporation]

Hi Artem,

would also like this recipe.  If you have any comments on my answer to 
Strahil, would love to hear them...


On 2020-08-03 9:42 p.m., Artem Russakovskii wrote:
I tried putting all web files (specifically WordPress php and static 
files as well as various cache files) on gluster before, and the results 
were miserable on a busy site - our usual ~8-10 load quickly turned into 
100+ and killed everything.


I had to go back to running just the user uploads (which are static 
files in the Wordpress uploads/ dir) on gluster and using rsync (via 
lsyncd) for the frequently executed php / cache.


I'd love to figure this out as well and tune gluster for heavy reads and 
moderate writes, but I haven't cracked that recipe yet.


On Mon, Aug 3, 2020, 8:08 PM Computerisms Corporation 
mailto:b...@computerisms.ca>> wrote:


Hi Gurus,

I have been trying to wrap my head around performance improvements
on my
gluster setup, and I don't seem to be making any progress.  I mean
forward progress.  making it worse takes practically no effort at all.

My gluster is distributed-replicated across 6 bricks and 2 servers,
with
an arbiter on each server.  I designed it like this so I have an
expansion path to more servers in the future (like the staggered
arbiter
diagram in the red hat documentation).  gluster v info output is below.
I have compiled gluster 7.6 from sources on both servers.

Servers are 6core/3.4Ghz with 32 GB RAM, no swap, and SSD and gigabit
network connections.  They are running debian, and are being used as
redundant web servers.  There is some 3Million files on the Gluster
Storage averaging 130KB/file.  Currently only one of the two servers is
serving web services.  There are well over 100 sites, and apache
server-status claims around 5 hits per second, depending on time of
day,
so a fair bit of logging going on.  The gluster is only holding website
data and config files that will be common between the two servers, no
databases or anything like that on the Gluster.

When the serving server is under load load average is consistently
12-20.  glusterfs is always at the top with 150%-250% cpu, and each
of 3
bricks at roughly 50-70%, so consistently pegging 4 of the 6 cores.
apache processes will easily eat up all the rest of the cpus after
that.
   And web page response time is underwhelming at best.

Interestingly, mostly because it is not something I have ever
experienced before, software interrupts sit between 1 and 5 on each
core, but the last core is usually sitting around 20.  Have never
encountered a high load average where the si number was ever
significant.  I have googled the crap out of that (as well as gluster
performance in general), there are nearly limitless posts about what it
is, but have yet to see one thing to explain what to do about it. 
Sadly

I can't really shut down the gluster process to confirm if that is the
cause, but it's a pretty good bet, I think.

When the system is not under load, glusterfs will be running at around
100% with each of the 3 bricks around 35%, so using 2 cores when doing
not much of anything.

nload shows the network cards rarely climb above 300 Mbps unless I am
doing a direct file transfer between the servers, in which case it gets
right up to the 1Gbps limit.  RAM is never above 15GB unless I am
causing it to happen.  atop show a disk busy percentage, it is often
above 50% and sometimes will hit 100%, and is no where near as
consistently showing excessive usage like the cpu cores are.  The cpu
definitely seems to be the bottleneck.

When I found out about the groups directory, I figured one of those
must
be useful to me, but as best as I can tell they are not.  But I am
really hoping that someone has configured a system like mine and has a
good group file they might share for this situation, or a peak at their
volume info output?

or maybe this is really just about as good as I should expect?  Maybe
the fix is that I need more/faster cores?  I hope not, as that isn't
really an option.

Anyway, here is my volume info as promised.

root@mooglian:/Computerisms/sites/computerisms.ca/log#
 gluster v info

Volume Name: webisms
Type: Distributed-Replicate
Volume ID: 261901e7-60b4-4760-897d-0163beed356e
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x (2 + 1) = 6
Transport-type: tcp
Bricks:
Brick1: mooglian:/var/GlusterBrick/replset-0/webisms-replset-0
Brick2: moogle:/var/GlusterBrick/replset-0/webisms-replset-0
Brick3: moogle:/var/GlusterBrick/replset-0-arb/webisms-replset-0-arb
(arbiter)
Brick4: moogle:/var/GlusterBrick/replset-1/webisms-replset-1
Brick5: mooglian:/var/GlusterBrick/replset-1/webisms-replset-1
Brick6: 

Re: [Gluster-users] performance

[Computerisms Corporation]

Hi Strahil, thanks for your response.



I have compiled gluster 7.6 from sources on both servers.


There  is a 7.7 version which is fixing somw stuff. Why do you have to compile 
it from source ?


Because I have often found with other stuff in the past compiling from 
source makes a bunch of problems go away.  software generally works the 
way the developers expect it to if you use the sources, so they are 
better able to help if required.  so now I generally compile most of my 
center-piece softwares and use packages for all the supporting stuff.





Servers are 6core/3.4Ghz with 32 GB RAM, no swap, and SSD and gigabit
network connections.  They are running debian, and are being used as
redundant web servers.  There is some 3Million files on the Gluster
Storage averaging 130KB/file.


This type of workload is called 'metadata-intensive'.


does this mean the metadata-cache group file would be a good one to 
enable?  will try.


waited 10 minutes, no change that I can see.


There are some recommendations for this type of workload:
https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3/html/administration_guide/small_file_performance_enhancements

Keep an eye on the section that mentions dirty-ratio = 5 
 = 2.


I have actually read that whole manual, and specifically that page 
several times.  And also this one:


https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.1/html/administration_guide/small_file_performance_enhancements

Perhaps I am not understanding it correctly.  I tried these suggestions 
before and it got worse, not better.  so I have been operating under the 
assumption that maybe these guidelines are not appropriate for newer 
versions.


But will try again.  adjusting the dirty ratios.

Load average went from around 15 to 35 in about 2-3 minutes, but 20 
minutes later, it is back down to 20.  It may be having a minimal 
positive impact on cpu, though, I haven't see the main glusterfs go over 
200% since I changed this, an the brick processes are hovering just 
below 50%  where they were consistently above 50% before.  Might just be 
time of day with the system not as busy.


after watching for 30 minutes, load average is fluctuating between 10 
and 30, but cpu idle appears marginally better on average than it was.



Interestingly, mostly because it is not something I have ever
experienced before, software interrupts sit between 1 and 5 on each
core, but the last core is usually sitting around 20.  Have never
encountered a high load average where the si number was ever
significant.  I have googled the crap out of that (as well as gluster
performance in general), there are nearly limitless posts about what it

is, but have yet to see one thing to explain what to do about it.


There is an explanation  about that in the link I provided above:

Configuring a higher event threads value than the available processing units 
could again cause context switches on these threads. As a result reducing the 
number deduced from the previous step to a number that is less that the 
available processing units is recommended.


Okay, again, have played with these numbers before and it did not pan 
out as expected.  if I understand it correctly, I have 3 brick processes 
(glusterfsd), so the "deduced" number should be 3, and I should set it 
lower than that, so 2.  but it also says "If a specific thread consumes 
more number of CPU cycles than needed, increasing the event thread count 
would enhance the performance of the Red Hat Storage Server."  which is 
why I had it at 8.


but will set it to 2 now.  load average is at 17 to start, waiting a 
while to see what happens.


so 15 minutes later, load average is currently 12, but is fluctuating 
between 10 and 20, have seen no significant change in cpu usage or 
anything else in top.


now try also changing server.outstanding-rpc-limit to 256 and wait.

15 minutes later; load has been above 30 but is currently back down to 
12.  no significant change in cpu.  try increasing to 512 and wait.


15 minutes later, load average is 50.  no signficant difference in cpu. 
Software interrupts remain around where they were.  wa from top remains 
about where it was.  not sure why load average is climbing so high. 
changing rpc-limit to 128.


ugh.  10 minutes later, load average just popped over 100.  resetting 
rpc-limit.


now trying cluster.lookup-optimize on, lazy rebalancing (probably a bad 
idea on the live system, but how much worse can it get?)  Ya, bad idea, 
80 hours estimated to complete, load is over 50 and server is crawling. 
disabling rebalance and turning lookup-optimize off, for now.


right now the only suggested parameter I haven't played with is the 
performance.io-thread-count, which I currently have at 64.


sigh.  an hour later load average is 80 and climbing.  apache processes 
are numbering in the hundreds and I am constantly having to restart it. 
this brings load average down to 5, but as apache 

Re: [Gluster-users] performance

[Strahil Nikolov]

На 4 август 2020 г. 6:01:17 GMT+03:00, Computerisms Corporation 
 написа:
>Hi Gurus,
>
>I have been trying to wrap my head around performance improvements on
>my 
>gluster setup, and I don't seem to be making any progress.  I mean 
>forward progress.  making it worse takes practically no effort at all.
>
>My gluster is distributed-replicated across 6 bricks and 2 servers,
>with 
>an arbiter on each server.  I designed it like this so I have an 
>expansion path to more servers in the future (like the staggered
>arbiter 
>diagram in the red hat documentation).  gluster v info output is below.
>
>I have compiled gluster 7.6 from sources on both servers.

There  is a 7.7 version which is fixing somw stuff. Why do you have to compile 
it from source ?

>Servers are 6core/3.4Ghz with 32 GB RAM, no swap, and SSD and gigabit 
>network connections.  They are running debian, and are being used as 
>redundant web servers.  There is some 3Million files on the Gluster 
>Storage averaging 130KB/file.  

This type of workload is called 'metadata-intensive'.
There are some recommendations for this type of workload:
https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3/html/administration_guide/small_file_performance_enhancements

Keep an eye on the section that mentions dirty-ratio = 5 
 = 2. 


>Currently only one of the two servers is
>
>serving web services.  There are well over 100 sites, and apache 
>server-status claims around 5 hits per second, depending on time of
>day, 
>so a fair bit of logging going on.  The gluster is only holding website
>
>data and config files that will be common between the two servers, no 
>databases or anything like that on the Gluster.
>
>When the serving server is under load load average is consistently 
>12-20.  glusterfs is always at the top with 150%-250% cpu, and each of
>3 
>bricks at roughly 50-70%, so consistently pegging 4 of the 6 cores. 
>apache processes will easily eat up all the rest of the cpus after
>that. 
>  And web page response time is underwhelming at best.
>
>Interestingly, mostly because it is not something I have ever 
>experienced before, software interrupts sit between 1 and 5 on each 
>core, but the last core is usually sitting around 20.  Have never 
>encountered a high load average where the si number was ever 
>significant.  I have googled the crap out of that (as well as gluster 
>performance in general), there are nearly limitless posts about what it
>
>is, but have yet to see one thing to explain what to do about it. 

There is an explanation  about that in the link I provided above:

Configuring a higher event threads value than the available processing units 
could again cause context switches on these threads. As a result reducing the 
number deduced from the previous step to a number that is less that the 
available processing units is recommended.


>Sadly 
>I can't really shut down the gluster process to confirm if that is the 
>cause, but it's a pretty good bet, I think.
>
>When the system is not under load, glusterfs will be running at around 
>100% with each of the 3 bricks around 35%, so using 2 cores when doing 
>not much of anything.
>
>nload shows the network cards rarely climb above 300 Mbps unless I am 
>doing a direct file transfer between the servers, in which case it gets
>
>right up to the 1Gbps limit.  RAM is never above 15GB unless I am 
>causing it to happen.  atop show a disk busy percentage, it is often 
>above 50% and sometimes will hit 100%, and is no where near as 
>consistently showing excessive usage like the cpu cores are.  The cpu 
>definitely seems to be the bottleneck.
>When I found out about the groups directory, I figured one of those
>must 
>be useful to me, but as best as I can tell they are not.  But I am 
>really hoping that someone has configured a system like mine and has a 
>good group file they might share for this situation, or a peak at their
>
>volume info output?
>
>or maybe this is really just about as good as I should expect?  Maybe 
>the fix is that I need more/faster cores?  I hope not, as that isn't 
>really an option.
>
>Anyway, here is my volume info as promised.
>
>root@mooglian:/Computerisms/sites/computerisms.ca/log# gluster v info
>
>Volume Name: webisms
>Type: Distributed-Replicate
>Volume ID: 261901e7-60b4-4760-897d-0163beed356e
>Status: Started
>Snapshot Count: 0
>Number of Bricks: 2 x (2 + 1) = 6
>Transport-type: tcp
>Bricks:
>Brick1: mooglian:/var/GlusterBrick/replset-0/webisms-replset-0
>Brick2: moogle:/var/GlusterBrick/replset-0/webisms-replset-0
>Brick3: moogle:/var/GlusterBrick/replset-0-arb/webisms-replset-0-arb 
>(arbiter)
>Brick4: moogle:/var/GlusterBrick/replset-1/webisms-replset-1
>Brick5: mooglian:/var/GlusterBrick/replset-1/webisms-replset-1
>Brick6: mooglian:/var/GlusterBrick/replset-1-arb/webisms-replset-1-arb 
>(arbiter)
>Options Reconfigured:
>auth.allow: 
>performance.client-io-threads: off
>nfs.disable: on
>storage.fips-mode-rchecksum: on

Re: [Gluster-users] performance

[Artem Russakovskii]

I tried putting all web files (specifically WordPress php and static files
as well as various cache files) on gluster before, and the results were
miserable on a busy site - our usual ~8-10 load quickly turned into 100+
and killed everything.

I had to go back to running just the user uploads (which are static files
in the Wordpress uploads/ dir) on gluster and using rsync (via lsyncd) for
the frequently executed php / cache.

I'd love to figure this out as well and tune gluster for heavy reads and
moderate writes, but I haven't cracked that recipe yet.

On Mon, Aug 3, 2020, 8:08 PM Computerisms Corporation 
wrote:

> Hi Gurus,
>
> I have been trying to wrap my head around performance improvements on my
> gluster setup, and I don't seem to be making any progress.  I mean
> forward progress.  making it worse takes practically no effort at all.
>
> My gluster is distributed-replicated across 6 bricks and 2 servers, with
> an arbiter on each server.  I designed it like this so I have an
> expansion path to more servers in the future (like the staggered arbiter
> diagram in the red hat documentation).  gluster v info output is below.
> I have compiled gluster 7.6 from sources on both servers.
>
> Servers are 6core/3.4Ghz with 32 GB RAM, no swap, and SSD and gigabit
> network connections.  They are running debian, and are being used as
> redundant web servers.  There is some 3Million files on the Gluster
> Storage averaging 130KB/file.  Currently only one of the two servers is
> serving web services.  There are well over 100 sites, and apache
> server-status claims around 5 hits per second, depending on time of day,
> so a fair bit of logging going on.  The gluster is only holding website
> data and config files that will be common between the two servers, no
> databases or anything like that on the Gluster.
>
> When the serving server is under load load average is consistently
> 12-20.  glusterfs is always at the top with 150%-250% cpu, and each of 3
> bricks at roughly 50-70%, so consistently pegging 4 of the 6 cores.
> apache processes will easily eat up all the rest of the cpus after that.
>   And web page response time is underwhelming at best.
>
> Interestingly, mostly because it is not something I have ever
> experienced before, software interrupts sit between 1 and 5 on each
> core, but the last core is usually sitting around 20.  Have never
> encountered a high load average where the si number was ever
> significant.  I have googled the crap out of that (as well as gluster
> performance in general), there are nearly limitless posts about what it
> is, but have yet to see one thing to explain what to do about it.  Sadly
> I can't really shut down the gluster process to confirm if that is the
> cause, but it's a pretty good bet, I think.
>
> When the system is not under load, glusterfs will be running at around
> 100% with each of the 3 bricks around 35%, so using 2 cores when doing
> not much of anything.
>
> nload shows the network cards rarely climb above 300 Mbps unless I am
> doing a direct file transfer between the servers, in which case it gets
> right up to the 1Gbps limit.  RAM is never above 15GB unless I am
> causing it to happen.  atop show a disk busy percentage, it is often
> above 50% and sometimes will hit 100%, and is no where near as
> consistently showing excessive usage like the cpu cores are.  The cpu
> definitely seems to be the bottleneck.
>
> When I found out about the groups directory, I figured one of those must
> be useful to me, but as best as I can tell they are not.  But I am
> really hoping that someone has configured a system like mine and has a
> good group file they might share for this situation, or a peak at their
> volume info output?
>
> or maybe this is really just about as good as I should expect?  Maybe
> the fix is that I need more/faster cores?  I hope not, as that isn't
> really an option.
>
> Anyway, here is my volume info as promised.
>
> root@mooglian:/Computerisms/sites/computerisms.ca/log# gluster v info
>
> Volume Name: webisms
> Type: Distributed-Replicate
> Volume ID: 261901e7-60b4-4760-897d-0163beed356e
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 2 x (2 + 1) = 6
> Transport-type: tcp
> Bricks:
> Brick1: mooglian:/var/GlusterBrick/replset-0/webisms-replset-0
> Brick2: moogle:/var/GlusterBrick/replset-0/webisms-replset-0
> Brick3: moogle:/var/GlusterBrick/replset-0-arb/webisms-replset-0-arb
> (arbiter)
> Brick4: moogle:/var/GlusterBrick/replset-1/webisms-replset-1
> Brick5: mooglian:/var/GlusterBrick/replset-1/webisms-replset-1
> Brick6: mooglian:/var/GlusterBrick/replset-1-arb/webisms-replset-1-arb
> (arbiter)
> Options Reconfigured:
> auth.allow: 
> performance.client-io-threads: off
> nfs.disable: on
> storage.fips-mode-rchecksum: on
> transport.address-family: inet
> performance.stat-prefetch: on
> network.inode-lru-limit: 20
> performance.write-behind-window-size: 4MB
> performance.readdir-ahead: on
> 

[Gluster-users] performance

[Computerisms Corporation]

Hi Gurus,

I have been trying to wrap my head around performance improvements on my 
gluster setup, and I don't seem to be making any progress.  I mean 
forward progress.  making it worse takes practically no effort at all.


My gluster is distributed-replicated across 6 bricks and 2 servers, with 
an arbiter on each server.  I designed it like this so I have an 
expansion path to more servers in the future (like the staggered arbiter 
diagram in the red hat documentation).  gluster v info output is below. 
I have compiled gluster 7.6 from sources on both servers.


Servers are 6core/3.4Ghz with 32 GB RAM, no swap, and SSD and gigabit 
network connections.  They are running debian, and are being used as 
redundant web servers.  There is some 3Million files on the Gluster 
Storage averaging 130KB/file.  Currently only one of the two servers is 
serving web services.  There are well over 100 sites, and apache 
server-status claims around 5 hits per second, depending on time of day, 
so a fair bit of logging going on.  The gluster is only holding website 
data and config files that will be common between the two servers, no 
databases or anything like that on the Gluster.


When the serving server is under load load average is consistently 
12-20.  glusterfs is always at the top with 150%-250% cpu, and each of 3 
bricks at roughly 50-70%, so consistently pegging 4 of the 6 cores. 
apache processes will easily eat up all the rest of the cpus after that. 
 And web page response time is underwhelming at best.


Interestingly, mostly because it is not something I have ever 
experienced before, software interrupts sit between 1 and 5 on each 
core, but the last core is usually sitting around 20.  Have never 
encountered a high load average where the si number was ever 
significant.  I have googled the crap out of that (as well as gluster 
performance in general), there are nearly limitless posts about what it 
is, but have yet to see one thing to explain what to do about it.  Sadly 
I can't really shut down the gluster process to confirm if that is the 
cause, but it's a pretty good bet, I think.


When the system is not under load, glusterfs will be running at around 
100% with each of the 3 bricks around 35%, so using 2 cores when doing 
not much of anything.


nload shows the network cards rarely climb above 300 Mbps unless I am 
doing a direct file transfer between the servers, in which case it gets 
right up to the 1Gbps limit.  RAM is never above 15GB unless I am 
causing it to happen.  atop show a disk busy percentage, it is often 
above 50% and sometimes will hit 100%, and is no where near as 
consistently showing excessive usage like the cpu cores are.  The cpu 
definitely seems to be the bottleneck.


When I found out about the groups directory, I figured one of those must 
be useful to me, but as best as I can tell they are not.  But I am 
really hoping that someone has configured a system like mine and has a 
good group file they might share for this situation, or a peak at their 
volume info output?


or maybe this is really just about as good as I should expect?  Maybe 
the fix is that I need more/faster cores?  I hope not, as that isn't 
really an option.


Anyway, here is my volume info as promised.

root@mooglian:/Computerisms/sites/computerisms.ca/log# gluster v info

Volume Name: webisms
Type: Distributed-Replicate
Volume ID: 261901e7-60b4-4760-897d-0163beed356e
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x (2 + 1) = 6
Transport-type: tcp
Bricks:
Brick1: mooglian:/var/GlusterBrick/replset-0/webisms-replset-0
Brick2: moogle:/var/GlusterBrick/replset-0/webisms-replset-0
Brick3: moogle:/var/GlusterBrick/replset-0-arb/webisms-replset-0-arb 
(arbiter)

Brick4: moogle:/var/GlusterBrick/replset-1/webisms-replset-1
Brick5: mooglian:/var/GlusterBrick/replset-1/webisms-replset-1
Brick6: mooglian:/var/GlusterBrick/replset-1-arb/webisms-replset-1-arb 
(arbiter)

Options Reconfigured:
auth.allow: 
performance.client-io-threads: off
nfs.disable: on
storage.fips-mode-rchecksum: on
transport.address-family: inet
performance.stat-prefetch: on
network.inode-lru-limit: 20
performance.write-behind-window-size: 4MB
performance.readdir-ahead: on
performance.io-thread-count: 64
performance.cache-size: 8GB
server.event-threads: 8
client.event-threads: 8
performance.nl-cache-timeout: 600


--
Bob Miller
Cell: 867-334-7117
Office: 867-633-3760
Office: 867-322-0362
www.computerisms.ca




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Performance tuning suggestions for nvme on aws (Strahil)

[Mohit Agrawal]

Hi,

 Thanks for sharing the output. May be in the case of NVME there is no
performance difference in SSL vs Non-SSL.
 Can you please share the output of the command "gluster v info" and what
client you are using to mount the volume?
 If you are using fuse have u used any specific argument at the time of
mounting the volume?
 Please use default event-threads if you are not getting any performance
improvement and share the command output
 "top -bH -d 10" at the time of writing IO.

Thanks,
Mohit Agrawal

On Mon, Jan 6, 2020 at 4:01 AM Michael Richardson <
he...@mikerichardson.com.au> wrote:

> Hi Mohit and Strahil,
>
> Thank you for taking the time to share your advice. Strahil, unfortunately
> your message didn't come to my inbox, so I'm combining my reply to both
> yourself and Mohit.
>
> Mohit,
>
> I mentioned there was no performance difference between using SSL and not
> using SSL. I did try setting it to AES128 anyway, but that seemed to have
> no effect as well.
>
> Strahil,
>
> I've had a go at various different settings this morning, including all of
> those you've supplied. I couldn't see any improvement in throughput when
> messing with these, but I could make it worse by adjusting things like
> event-threads.
>
> I've also tried adjusting the fio test types from one large file to many
> small files and a mix in between without any real change in peak
> performance.
>
> In answer to your other question, I'm running Gluster 7.1-1 on Debian 10.
> I have confirmed that blk-mq is on and my scheduler is 'none'. I don't seem
> to have an option to change it to 'noop' but I'm not sure why.
>
> Here's some profile output as well, in case that's helpful. These' values
> don't look so bad (to me, at least), but again this is much less than the
> raw nvme output that I was hoping to get closer to.
>
> Brick: node1:/data/nvme/testvol1/brick
> -
> Cumulative Stats:
>Block Size:  16384b+
>  No. of Reads:   492397
> No. of Writes:  2849474
>  %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
>   Fop
>  -   ---   ---   ---   
>  
>   0.00   0.00 us   0.00 us   0.00 us 57
>  FORGET
>   0.00   0.00 us   0.00 us   0.00 us   4133
> RELEASE
>   0.00   0.00 us   0.00 us   0.00 us 22
>  RELEASEDIR
>   0.00  33.67 us  30.56 us  43.31 us  5
>  GETXATTR
>   0.01  26.05 us  14.36 us 128.33 us870
> ENTRYLK
>   0.02  53.48 us  43.23 us  87.31 us435
> FTRUNCATE
>   0.02  58.54 us  46.75 us  94.94 us435
> FALLOCATE
>   0.04  24.21 us   9.58 us 679.17 us   2436
> FLUSH
>   0.04 145.95 us 116.32 us 211.72 us435
>  CREATE
>   0.28 211.24 us  32.92 us   17442.92 us   2000
>  OPEN
>   0.38  82.16 us  34.00 us   15537.54 us   6929
>  LOOKUP
>   1.926683.77 us 437.30 us   28616.62 us436
> FSYNC
>   2.03  48.87 us   7.50 us   24840.37 us  63019
>  FINODELK
>  18.97 455.94 us  43.78 us   43701.58 us  63019
>  FXATTROP
>  28.06 134.43 us  41.87 us   36537.65 us 316217
> WRITE
>  48.22 611.65 us 100.31 us   43842.80 us 119420
>  READ
>
> Duration: 3505 seconds
>Data Read: 8067432448 bytes
> Data Written: 46685782016 bytes
>
> Interval 0 Stats:
>Block Size:  16384b+
>  No. of Reads:   492397
> No. of Writes:  2849474
>  %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
>   Fop
>  -   ---   ---   ---   
>  
>   0.00   0.00 us   0.00 us   0.00 us 57
>  FORGET
>   0.00   0.00 us   0.00 us   0.00 us   4133
> RELEASE
>   0.00   0.00 us   0.00 us   0.00 us 22
>  RELEASEDIR
>   0.00  33.67 us  30.56 us  43.31 us  5
>  GETXATTR
>   0.01  26.05 us  14.36 us 128.33 us870
> ENTRYLK
>   0.02  53.48 us  43.23 us  87.31 us435
> FTRUNCATE
>   0.02  58.54 us  46.75 us  94.94 us435
> FALLOCATE
>   0.04  24.21 us   9.58 us 679.17 us   2436
> FLUSH
>   0.04 145.95 us 116.32 us 211.72 us435
>  CREATE
>   0.28 211.24 us  32.92 us   17442.92 us   2000
>  OPEN
>   0.38  82.16 us  34.00 us   15537.54 us   6929
>  LOOKUP
>   1.926683.77 us 437.30 us   28616.62 us436
> FSYNC
>   2.03  48.87 us   7.50 us   24840.37 us  63019
>  FINODELK
>  18.97 455.94 us  43.78 us   43701.58 

Re: [Gluster-users] Performance tuning suggestions for nvme on aws (Strahil)

[Michael Richardson]

Hi Mohit and Strahil,

Thank you for taking the time to share your advice. Strahil, unfortunately
your message didn't come to my inbox, so I'm combining my reply to both
yourself and Mohit.

Mohit,

I mentioned there was no performance difference between using SSL and not
using SSL. I did try setting it to AES128 anyway, but that seemed to have
no effect as well.

Strahil,

I've had a go at various different settings this morning, including all of
those you've supplied. I couldn't see any improvement in throughput when
messing with these, but I could make it worse by adjusting things like
event-threads.

I've also tried adjusting the fio test types from one large file to many
small files and a mix in between without any real change in peak
performance.

In answer to your other question, I'm running Gluster 7.1-1 on Debian 10. I
have confirmed that blk-mq is on and my scheduler is 'none'. I don't seem
to have an option to change it to 'noop' but I'm not sure why.

Here's some profile output as well, in case that's helpful. These' values
don't look so bad (to me, at least), but again this is much less than the
raw nvme output that I was hoping to get closer to.

Brick: node1:/data/nvme/testvol1/brick
-
Cumulative Stats:
   Block Size:  16384b+
 No. of Reads:   492397
No. of Writes:  2849474
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 -   ---   ---   ---   
 
  0.00   0.00 us   0.00 us   0.00 us 57
 FORGET
  0.00   0.00 us   0.00 us   0.00 us   4133
RELEASE
  0.00   0.00 us   0.00 us   0.00 us 22
 RELEASEDIR
  0.00  33.67 us  30.56 us  43.31 us  5
 GETXATTR
  0.01  26.05 us  14.36 us 128.33 us870
ENTRYLK
  0.02  53.48 us  43.23 us  87.31 us435
FTRUNCATE
  0.02  58.54 us  46.75 us  94.94 us435
FALLOCATE
  0.04  24.21 us   9.58 us 679.17 us   2436
FLUSH
  0.04 145.95 us 116.32 us 211.72 us435
 CREATE
  0.28 211.24 us  32.92 us   17442.92 us   2000
 OPEN
  0.38  82.16 us  34.00 us   15537.54 us   6929
 LOOKUP
  1.926683.77 us 437.30 us   28616.62 us436
FSYNC
  2.03  48.87 us   7.50 us   24840.37 us  63019
 FINODELK
 18.97 455.94 us  43.78 us   43701.58 us  63019
 FXATTROP
 28.06 134.43 us  41.87 us   36537.65 us 316217
WRITE
 48.22 611.65 us 100.31 us   43842.80 us 119420
 READ

Duration: 3505 seconds
   Data Read: 8067432448 bytes
Data Written: 46685782016 bytes

Interval 0 Stats:
   Block Size:  16384b+
 No. of Reads:   492397
No. of Writes:  2849474
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 -   ---   ---   ---   
 
  0.00   0.00 us   0.00 us   0.00 us 57
 FORGET
  0.00   0.00 us   0.00 us   0.00 us   4133
RELEASE
  0.00   0.00 us   0.00 us   0.00 us 22
 RELEASEDIR
  0.00  33.67 us  30.56 us  43.31 us  5
 GETXATTR
  0.01  26.05 us  14.36 us 128.33 us870
ENTRYLK
  0.02  53.48 us  43.23 us  87.31 us435
FTRUNCATE
  0.02  58.54 us  46.75 us  94.94 us435
FALLOCATE
  0.04  24.21 us   9.58 us 679.17 us   2436
FLUSH
  0.04 145.95 us 116.32 us 211.72 us435
 CREATE
  0.28 211.24 us  32.92 us   17442.92 us   2000
 OPEN
  0.38  82.16 us  34.00 us   15537.54 us   6929
 LOOKUP
  1.926683.77 us 437.30 us   28616.62 us436
FSYNC
  2.03  48.87 us   7.50 us   24840.37 us  63019
 FINODELK
 18.97 455.94 us  43.78 us   43701.58 us  63019
 FXATTROP
 28.06 134.43 us  41.87 us   36537.65 us 316217
WRITE
 48.23 611.65 us 100.31 us   43842.80 us 119420
 READ

Duration: 3505 seconds
   Data Read: 8067432448 bytes
Data Written: 46685782016 bytes

Brick: node2:/data/nvme/testvol1/brick
-
Cumulative Stats:
   Block Size:  16384b+
 No. of Reads:0
No. of Writes:  2849474
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 -   ---   ---   ---   
 
  0.00   0.00 us   0.00 us   0.00 us 57
 FORGET
  0.00   0.00 us   0.00 us   0.00 us   4133
RELEASE
  0.00   0.00 us  

Re: [Gluster-users] Performance tuning suggestions for nvme on aws (Strahil)

[Mohit Agrawal]

Hi,

Along with previous tunning suggested by Strahil please configure "
ssl.cipher-list"
to AES128 for specific volume to improve the performance. As you mentioned
you
have configured SSL on a volume and the performance is drop in case of SSL.
To improve the same please configure the AES128 cipher list, I hope you
should get sufficient performance improvement.

Please share the result of a performance improvement after configuring the
cipher
AES128 if it is possible for you.

Thanks,
Mohit Agrawal


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Performance tuning suggestions for nvme on aws

[Michael Richardson]

Hi all!

I'm experimenting with GFS for the first time have built a simple
three-node cluster using AWS 'i3en' type instances. These instances provide
raw nvme devices that are incredibly fast.

What I'm finding in these tests is that gluster is offering only a fraction
of the raw nvme performance in a 3 replica set (ie, 3 nodes with 1 brick
each). I'm wondering if there is anything I can do to squeeze more
performance out.

For testing, I'm running fio using a 16GB test file with a 75/25 read/write
split. Basically I'm trying to replicate a MySQL database which is what I'd
ideally like to host here (which I realise is probably not practical).

My fio test command is:
$ fio --name=fio-test2 --filename=fio-test \
--randrepeat=1 \
--ioengine=libaio \
--direct=1 \
--runtime=300 \
--bs=16k \
--iodepth=64 \
--size=16G \
--readwrite=randrw \
--rwmixread=75 \
--group_reporting \
--numjobs=4

When I test this command directly on the nvme disk, I get:

   READ: bw=313MiB/s (328MB/s), 313MiB/s-313MiB/s (328MB/s-328MB/s),
io=47.0GiB (51.5GB), run=156806-156806msec
  WRITE: bw=105MiB/s (110MB/s), 105MiB/s-105MiB/s (110MB/s-110MB/s),
io=16.0GiB (17.2GB), run=156806-156806msec

When I install the disk into a gluster 3-replica volume, I get:

   READ: bw=86.3MiB/s (90.5MB/s), 86.3MiB/s-86.3MiB/s
(90.5MB/s-90.5MB/s), io=25.3GiB (27.2GB), run=32-32msec
  WRITE: bw=28.9MiB/s (30.3MB/s), 28.9MiB/s-28.9MiB/s
(30.3MB/s-30.3MB/s), io=8676MiB (9098MB), run=32-32msec

If I do the same but with only 2 replicas, I get the same performance
results. I also get the same rough values when doing 'read',
'randread', 'write', and 'randwrite' tests.

I'm testing directly on one of the storage nodes, so there's no
variables line client/server network performance in the mix.

I ran the same test with EBS volumes and I saw similar performance
drops when offering up the volume using gluster. A "Provisioned IOPS"
EBS volume that could offer 10,000 IOPS directly, was getting only
about 3500 IOPS when running as part of a gluster volume.

We're using TLS on the management and volume connections, but I'm not
seeing any CPU or memory constraint when using these volumes, so I
don't believe that is the bottleneck. Similarly, when I try with SSL
turned off, I see no change in performance.

Does anyone have any suggestions on things I might try to increase
performance when using these very fast disks as part of a gluster
volume, or is this is to be expected when factoring in all the extra
work that gluster needs to do when replicating data around volumes?

Thanks very much for your time!! I'll put the two full fio outputs
below if anyone wants more details.

Mike


- First full fio test, nvme device without gluster

fio-test: (groupid=0, jobs=4): err= 0: pid=5636: Sat Jan  4 23:09:18 2020
  read: IOPS=20.0k, BW=313MiB/s (328MB/s)(47.0GiB/156806msec)
slat (usec): min=3, max=6476, avg=88.44, stdev=326.96
clat (usec): min=218, max=89292, avg=11141.58, stdev=1871.14
 lat (usec): min=226, max=89311, avg=11230.16, stdev=1883.88
clat percentiles (usec):
 |  1.00th=[ 3654],  5.00th=[ 8455], 10.00th=[ 9372], 20.00th=[10159],
 | 30.00th=[10552], 40.00th=[10814], 50.00th=[11076], 60.00th=[11338],
 | 70.00th=[11731], 80.00th=[12256], 90.00th=[13042], 95.00th=[13960],
 | 99.00th=[15795], 99.50th=[16581], 99.90th=[19268], 99.95th=[23200],
 | 99.99th=[36439]
   bw (  KiB/s): min=75904, max=257120, per=25.00%, avg=80178.59,
stdev=9421.58, samples=1252
   iops: min= 4744, max=16070, avg=5011.15, stdev=588.85, samples=1252
  write: IOPS=6702, BW=105MiB/s (110MB/s)(16.0GiB/156806msec); 0 zone resets
slat (usec): min=4, max=5587, avg=88.52, stdev=325.86
clat (usec): min=54, max=29847, avg=4491.18, stdev=1481.06
 lat (usec): min=63, max=29859, avg=4579.83, stdev=1508.50
clat percentiles (usec):
 |  1.00th=[  947],  5.00th=[ 1975], 10.00th=[ 2737], 20.00th=[ 3458],
 | 30.00th=[ 3916], 40.00th=[ 4178], 50.00th=[ 4424], 60.00th=[ 4686],
 | 70.00th=[ 5014], 80.00th=[ 5473], 90.00th=[ 6259], 95.00th=[ 6980],
 | 99.00th=[ 8717], 99.50th=[ 9503], 99.90th=[10945], 99.95th=[11600],
 | 99.99th=[13698]
   bw (  KiB/s): min=23296, max=86432, per=25.00%, avg=26812.24,
stdev=3375.69, samples=1252
   iops: min= 1456, max= 5402, avg=1675.75, stdev=210.98, samples=1252
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.06%, 750=0.11%, 1000=0.10%
  lat (msec)   : 2=1.12%, 4=7.69%, 10=28.88%, 20=61.95%, 50=0.06%
  lat (msec)   : 100=0.01%
  cpu  : usr=1.56%, sys=7.85%, ctx=1905114, majf=0, minf=56
  IO depths: 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
 submit: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
 issued rwts: total=3143262,1051042,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 

Re: [Gluster-users] Performance is falling rapidly when updating from v5.5 to v7.0

[David Spisla]

I did another test with inode_size on xfs bricks=1024Bytes, but it had also
no effect. Here is the measurement:

(All values in MiB/s)
64KiB1MiB 10MiB
0,16   2,52   76,58

Beside of that I was not able to set the xattr trusted.io-stats-dump. I am
wondering myself why it is not working

Regards
David Spisla

Am Mi., 6. Nov. 2019 um 11:16 Uhr schrieb RAFI KC :

>
> On 11/6/19 3:42 PM, David Spisla wrote:
>
> Hello Rafi,
>
> I tried to set the xattr via
>
> setfattr -n trusted.io-stats-dump -v '/tmp/iostat.log'
> /gluster/repositories/repo1/
>
> but it had no effect. There is no such a xattr via getfattr and no
> logfile. The command setxattr is not available. What I am doing wrong?
>
>
> I will check it out and get back to you.
>
>
> By the way, you mean to increase the inode size of xfs layer from 512
> Bytes to 1024KB(!)? I think it should be 1024 Bytes because 2048 Bytes is
> the maximum
>
> It was a type, I meant to set up 1024 bytes, sorry for that.
>
>
> Regards
> David
>
> Am Mi., 6. Nov. 2019 um 04:10 Uhr schrieb RAFI KC :
>
>> I will take a look at the profile info shared. Since there is a huge
>> difference in the performance numbers between fuse and samba, it would be
>> great if we can get the profile info of fuse (on v7). This will help to
>> compare the number of calls for each fops. There should be some fops that
>> samba repeat, and we can find out it by comparing with fuse.
>>
>> Also if possible, can you please get client profile info from fuse mount
>> using the command `setxattr -n trusted.io-stats-dump -v > /tmp/iostat.log> `.
>>
>>
>> Regards
>>
>> Rafi KC
>>
>> On 11/5/19 11:05 PM, David Spisla wrote:
>>
>> I did the test with Gluster 7.0 ctime disabled. But it had no effect:
>> (All values in MiB/s)
>> 64KiB1MiB 10MiB
>> 0,16   2,60   54,74
>>
>> Attached there is now the complete profile file also with the results
>> from the last test. I will not repeat it with an higher inode size because
>> I don't think this will have an effect.
>> There must be another cause for the low performance
>>
>>
>> Yes. No need to try with higher inode size
>>
>>
>>
>> Regards
>> David Spisla
>>
>> Am Di., 5. Nov. 2019 um 16:25 Uhr schrieb David Spisla <
>> spisl...@gmail.com>:
>>
>>>
>>>
>>> Am Di., 5. Nov. 2019 um 12:06 Uhr schrieb RAFI KC :
>>>

 On 11/4/19 8:46 PM, David Spisla wrote:

 Dear Gluster Community,

 I also have a issue concerning performance. The last days I updated our
 test cluster from GlusterFS v5.5 to v7.0 . The setup in general:

 2 HP DL380 Servers with 10Gbit NICs, 1 Distribute-Replica 2 Volume with
 2 Replica Pairs. Client is SMB Samba (access via vfs_glusterfs) . I did
 several tests to ensure that Samba don't causes the fall.
 The setup ist completely the same except the Gluster Version
 Here are my results:
 64KiB   1MiB 10MiB(Filesize)
 3,49 47,41300,50  (Values in MiB/s with
 GlusterFS v5.5)
 0,16  2,61 76,63(Values in MiB/s
 with GlusterFS v7.0)


 Can you please share the profile information [1] for both versions?
 Also it would be really helpful if you can mention the io patterns that
 used for this tests.

 [1] :
 https://docs.gluster.org/en/latest/Administrator%20Guide/Monitoring%20Workload/

>>> Hello Rafi,
>>> thank you for your help.
>>>
>>> * First more information about the io patterns: As a client we use a
>>> DL360 Windws Server 2017 machine with 10Gbit NIC connected to the storage
>>> machines. The share will be mounted via SMB and the tests writes with fio.
>>> We use this job files (see attachment). Each job file will be executed
>>> separetely and there is a sleep about 60s between each test run to calm
>>> down the system before starting a new test.
>>>
>>> * Attached below you find the profile output from the tests with v5.5
>>> (ctime enabled), v7.0 (ctime enabled).
>>>
>>> * Beside of the tests with Samba I did also some fio tests directly on
>>> the FUSE Mounts (locally on one of the storage nodes). The results show
>>> that there is only a small decrease of performance between v5.5 and v7.0
>>> (All values in MiB/s)
>>> 64KiB1MiB 10MiB
>>> 50,09 679,96   1023,02 (v5.5)
>>> 47,00 656,46977,60 (v7.0)
>>>
>>> It seems to be that the combination of samba + gluster7.0 has a lot of
>>> problems, or not?
>>>
>>>

 We use this volume options (GlusterFS 7.0):

 Volume Name: archive1
 Type: Distributed-Replicate
 Volume ID: 44c17844-0bd4-4ca2-98d8-a1474add790c
 Status: Started
 Snapshot Count: 0
 Number of Bricks: 2 x 2 = 4
 Transport-type: tcp
 Bricks:
 Brick1: fs-dl380-c1-n1:/gluster/brick1/glusterbrick
 Brick2: fs-dl380-c1-n2:/gluster/brick1/glusterbrick
 Brick3: fs-dl380-c1-n1:/gluster/brick2/glusterbrick
 Brick4: 

Re: [Gluster-users] Performance is falling rapidly when updating from v5.5 to v7.0

[RAFI KC]

On 11/6/19 3:42 PM, David Spisla wrote:

Hello Rafi,

I tried to set the xattr via

setfattr -n trusted.io-stats-dump -v '/tmp/iostat.log' 
/gluster/repositories/repo1/


but it had no effect. There is no such a xattr via getfattr and no 
logfile. The command setxattr is not available. What I am doing wrong?



I will check it out and get back to you.


By the way, you mean to increase the inode size of xfs layer from 512 
Bytes to 1024KB(!)? I think it should be 1024 Bytes because 2048 Bytes 
is the maximum

It was a type, I meant to set up 1024 bytes, sorry for that.


Regards
David

Am Mi., 6. Nov. 2019 um 04:10 Uhr schrieb RAFI KC >:


I will take a look at the profile info shared. Since there is a
huge difference in the performance numbers between fuse and samba,
it would be great if we can get the profile info of fuse (on v7).
This will help to compare the number of calls for each fops. There
should be some fops that samba repeat, and we can find out it by
comparing with fuse.

Also if possible, can you please get client profile info from fuse
mount using the command `setxattr -n trusted.io-stats-dump -v
 `.


Regards

Rafi KC


On 11/5/19 11:05 PM, David Spisla wrote:

I did the test with Gluster 7.0 ctime disabled. But it had no effect:
(All values in MiB/s)
64KiB    1MiB     10MiB
0,16   2,60   54,74

Attached there is now the complete profile file also with the
results from the last test. I will not repeat it with an higher
inode size because I don't think this will have an effect.
There must be another cause for the low performance



Yes. No need to try with higher inode size




Regards
David Spisla

Am Di., 5. Nov. 2019 um 16:25 Uhr schrieb David Spisla
mailto:spisl...@gmail.com>>:



Am Di., 5. Nov. 2019 um 12:06 Uhr schrieb RAFI KC
mailto:rkavu...@redhat.com>>:


On 11/4/19 8:46 PM, David Spisla wrote:

Dear Gluster Community,

I also have a issue concerning performance. The last
days I updated our test cluster from GlusterFS v5.5 to
v7.0 . The setup in general:

2 HP DL380 Servers with 10Gbit NICs, 1
Distribute-Replica 2 Volume with 2 Replica Pairs. Client
is SMB Samba (access via vfs_glusterfs) . I did several
tests to ensure that Samba don't causes the fall.
The setup ist completely the same except the Gluster Version
Here are my results:
64KiB       1MiB 10MiB            (Filesize)
3,49             47,41     300,50  (Values in
MiB/s with GlusterFS v5.5)
0,16              2,61    76,63 (Values in MiB/s
with GlusterFS v7.0)



Can you please share the profile information [1] for both
versions?  Also it would be really helpful if you can
mention the io patterns that used for this tests.

[1] :

https://docs.gluster.org/en/latest/Administrator%20Guide/Monitoring%20Workload/

Hello Rafi,
thank you for your help.

* First more information about the io patterns: As a client
we use a DL360 Windws Server 2017 machine with 10Gbit NIC
connected to the storage machines. The share will be mounted
via SMB and the tests writes with fio. We use this job files
(see attachment). Each job file will be executed separetely
and there is a sleep about 60s between each test run to calm
down the system before starting a new test.

* Attached below you find the profile output from the tests
with v5.5 (ctime enabled), v7.0 (ctime enabled).

* Beside of the tests with Samba I did also some fio tests
directly on the FUSE Mounts (locally on one of the storage
nodes). The results show that there is only a small decrease
of performance between v5.5 and v7.0
(All values in MiB/s)
64KiB    1MiB     10MiB
50,09 679,96   1023,02 (v5.5)
47,00 656,46    977,60 (v7.0)

It seems to be that the combination of samba + gluster7.0 has
a lot of problems, or not?




We use this volume options (GlusterFS 7.0):

Volume Name: archive1
Type: Distributed-Replicate
Volume ID: 44c17844-0bd4-4ca2-98d8-a1474add790c
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: fs-dl380-c1-n1:/gluster/brick1/glusterbrick
Brick2: fs-dl380-c1-n2:/gluster/brick1/glusterbrick
Brick3: fs-dl380-c1-n1:/gluster/brick2/glusterbrick
Brick4: fs-dl380-c1-n2:/gluster/brick2/glusterbrick
Options Reconfigured:
performance.client-io-threads: off
 

Re: [Gluster-users] Performance is falling rapidly when updating from v5.5 to v7.0

[David Spisla]

Hello Rafi,

I tried to set the xattr via

setfattr -n trusted.io-stats-dump -v '/tmp/iostat.log'
/gluster/repositories/repo1/

but it had no effect. There is no such a xattr via getfattr and no logfile.
The command setxattr is not available. What I am doing wrong?
By the way, you mean to increase the inode size of xfs layer from 512 Bytes
to 1024KB(!)? I think it should be 1024 Bytes because 2048 Bytes is the
maximum

Regards
David

Am Mi., 6. Nov. 2019 um 04:10 Uhr schrieb RAFI KC :

> I will take a look at the profile info shared. Since there is a huge
> difference in the performance numbers between fuse and samba, it would be
> great if we can get the profile info of fuse (on v7). This will help to
> compare the number of calls for each fops. There should be some fops that
> samba repeat, and we can find out it by comparing with fuse.
>
> Also if possible, can you please get client profile info from fuse mount
> using the command `setxattr -n trusted.io-stats-dump -v  /tmp/iostat.log> `.
>
>
> Regards
>
> Rafi KC
>
> On 11/5/19 11:05 PM, David Spisla wrote:
>
> I did the test with Gluster 7.0 ctime disabled. But it had no effect:
> (All values in MiB/s)
> 64KiB1MiB 10MiB
> 0,16   2,60   54,74
>
> Attached there is now the complete profile file also with the results from
> the last test. I will not repeat it with an higher inode size because I
> don't think this will have an effect.
> There must be another cause for the low performance
>
>
> Yes. No need to try with higher inode size
>
>
>
> Regards
> David Spisla
>
> Am Di., 5. Nov. 2019 um 16:25 Uhr schrieb David Spisla  >:
>
>>
>>
>> Am Di., 5. Nov. 2019 um 12:06 Uhr schrieb RAFI KC :
>>
>>>
>>> On 11/4/19 8:46 PM, David Spisla wrote:
>>>
>>> Dear Gluster Community,
>>>
>>> I also have a issue concerning performance. The last days I updated our
>>> test cluster from GlusterFS v5.5 to v7.0 . The setup in general:
>>>
>>> 2 HP DL380 Servers with 10Gbit NICs, 1 Distribute-Replica 2 Volume with
>>> 2 Replica Pairs. Client is SMB Samba (access via vfs_glusterfs) . I did
>>> several tests to ensure that Samba don't causes the fall.
>>> The setup ist completely the same except the Gluster Version
>>> Here are my results:
>>> 64KiB   1MiB 10MiB(Filesize)
>>> 3,49 47,41300,50  (Values in MiB/s with
>>> GlusterFS v5.5)
>>> 0,16  2,61 76,63(Values in MiB/s
>>> with GlusterFS v7.0)
>>>
>>>
>>> Can you please share the profile information [1] for both versions?
>>> Also it would be really helpful if you can mention the io patterns that
>>> used for this tests.
>>>
>>> [1] :
>>> https://docs.gluster.org/en/latest/Administrator%20Guide/Monitoring%20Workload/
>>>
>> Hello Rafi,
>> thank you for your help.
>>
>> * First more information about the io patterns: As a client we use a
>> DL360 Windws Server 2017 machine with 10Gbit NIC connected to the storage
>> machines. The share will be mounted via SMB and the tests writes with fio.
>> We use this job files (see attachment). Each job file will be executed
>> separetely and there is a sleep about 60s between each test run to calm
>> down the system before starting a new test.
>>
>> * Attached below you find the profile output from the tests with v5.5
>> (ctime enabled), v7.0 (ctime enabled).
>>
>> * Beside of the tests with Samba I did also some fio tests directly on
>> the FUSE Mounts (locally on one of the storage nodes). The results show
>> that there is only a small decrease of performance between v5.5 and v7.0
>> (All values in MiB/s)
>> 64KiB1MiB 10MiB
>> 50,09 679,96   1023,02 (v5.5)
>> 47,00 656,46977,60 (v7.0)
>>
>> It seems to be that the combination of samba + gluster7.0 has a lot of
>> problems, or not?
>>
>>
>>>
>>> We use this volume options (GlusterFS 7.0):
>>>
>>> Volume Name: archive1
>>> Type: Distributed-Replicate
>>> Volume ID: 44c17844-0bd4-4ca2-98d8-a1474add790c
>>> Status: Started
>>> Snapshot Count: 0
>>> Number of Bricks: 2 x 2 = 4
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: fs-dl380-c1-n1:/gluster/brick1/glusterbrick
>>> Brick2: fs-dl380-c1-n2:/gluster/brick1/glusterbrick
>>> Brick3: fs-dl380-c1-n1:/gluster/brick2/glusterbrick
>>> Brick4: fs-dl380-c1-n2:/gluster/brick2/glusterbrick
>>> Options Reconfigured:
>>> performance.client-io-threads: off
>>> nfs.disable: on
>>> storage.fips-mode-rchecksum: on
>>> transport.address-family: inet
>>> user.smb: disable
>>> features.read-only: off
>>> features.worm: off
>>> features.worm-file-level: on
>>> features.retention-mode: enterprise
>>> features.default-retention-period: 120
>>> network.ping-timeout: 10
>>> features.cache-invalidation: on
>>> features.cache-invalidation-timeout: 600
>>> performance.nl-cache: on
>>> performance.nl-cache-timeout: 600
>>> client.event-threads: 32
>>> server.event-threads: 32
>>> cluster.lookup-optimize: on
>>> performance.stat-prefetch: on
>>> performance.cache-invalidation: on

Re: [Gluster-users] Performance drop when upgrading from 3.8 to 6.5

[Riccardo Murri]

Dear Rafi, all,

please find attached two profile files; both are profiling the same command:
```
time rsync -a $SRC root@172.23.187.207:/glusterfs
```

In both cases, the target is a Ubuntu 16.04 VM mounting a pure
distributed GlusterFS 7 filesystem on `/glusterfs`.  The GlusterFS 7
cluster is comprised of 3 identical Ubuntu 16.04 VMs, each with one
brick of 200GB.  I have turned ctime off, as suggested in a previous
email.

In the case of file `profile1.txt` (larger file), $SRC is a directory
tree containing ~94'500 files, collectively weighing ~376GB.  The
transfer takes between 250 and 300 minutes (I've made several attempts
now), for an average bandwidth of ~21MB/s.

In the case of `profile3.txt`, $SRC is a single tar file weighing
72GB.  It takes between 16 and 30 minutes to write it into the
GlusterFS 7 filesystem; average bandwidth is ~60MB/s.

To me, this seems to indicate that, while write performance on data is
good, metadata ops on GlusterFS 7 are rather slow, and much slower
than in the 3.x series.  Is there any other tweak that I may try to
apply?

Thanks,
Riccardo
Brick: server001:/srv/glusterfs
---
Cumulative Stats:
   Block Size:  2b+   4b+   8b+ 
 No. of Reads:0 0 0 
No. of Writes:4 2 9 
 
   Block Size: 16b+  32b+  64b+ 
 No. of Reads:0 0 0 
No. of Writes:   322837 
 
   Block Size:128b+ 256b+ 512b+ 
 No. of Reads:0 0 0 
No. of Writes:  108   297   575 
 
   Block Size:   1024b+2048b+4096b+ 
 No. of Reads:0 0 0 
No. of Writes:  984  1994  4334 
 
   Block Size:   8192b+   16384b+   32768b+ 
 No. of Reads:0 0 0 
No. of Writes: 8304 16146 33163 
 
   Block Size:  65536b+  131072b+  262144b+ 
 No. of Reads:0 0 0 
No. of Writes:64431   4173687   207 
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls Fop
 -   ---   ---   ---   
  0.00   0.00 us   0.00 us   0.00 us 114999  FORGET
  0.00   0.00 us   0.00 us   0.00 us 130690 RELEASE
  0.00   0.00 us   0.00 us   0.00 us   1886  RELEASEDIR
  0.00 449.84 us 222.38 us2924.41 us 31   MKNOD
  0.01 177.17 us 104.16 us 436.34 us281LINK
  0.01 128.41 us  44.77 us2512.16 us414SETXATTR
  0.01  82.37 us  29.26 us4792.82 us   1319   FSTAT
  0.02 414.20 us 181.30 us   10818.54 us414   MKDIR
  0.02 741.77 us 103.38 us  158364.81 us281  UNLINK
  0.29  77.67 us  17.57 us4887.93 us  34118  STATFS
  0.37  50.29 us  10.83 us7341.33 us  67134   FLUSH
  0.73  46.83 us   9.41 us7990.78 us 140764 ENTRYLK
  0.85 219.73 us  75.32 us  125748.37 us  35106  RENAME
  1.21  51.71 us  10.32 us   10816.56 us 211798 INODELK
  1.33 377.95 us  38.65 us  150160.77 us  31778   FTRUNCATE
  1.64  97.82 us  23.85 us6386.67 us 151947STAT
  3.25 123.40 us  35.48 us   43775.28 us 238449 SETATTR
  4.31 581.91 us 123.91 us  261522.99 us  67134  CREATE
  7.30 137.63 us  23.45 us  153825.73 us 480472  LOOKUP
 78.65 321.99 us  32.95 us  169393.74 us2211891   WRITE
 
Duration: 156535 seconds
   Data Read: 0 bytes
Data Written: 555615327151 bytes
 
Interval 2 Stats:
   Block Size:  2b+   8b+  16b+ 
 No. of Reads:0 0 0 
No. of Writes:3 624 
 
   Block Size: 32b+  64b+ 128b+ 
 No. of Reads:0 0 0 
No. of Writes:   2127

Re: [Gluster-users] Performance drop when upgrading from 3.8 to 6.5

[RAFI KC]

On 11/5/19 4:53 PM, Riccardo Murri wrote:

Is it possible for you to repeat the test by disabling ctime or increasing the 
inode size to a higher value say 1024?

Sure!  How do I disable ctime or increase the inode size?

Would this suffice to disable `ctime`?

 sudo gluster volume set glusterfs ctime off

Can it be done on a running cluster?  Do I need to unmount and remount
for clients to see the effect?



This is good enough.  There is no need to unmount.



Thanks,
R




Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Performance is falling rapidly when updating from v5.5 to v7.0

[RAFI KC]

I will take a look at the profile info shared. Since there is a huge 
difference in the performance numbers between fuse and samba, it would 
be great if we can get the profile info of fuse (on v7). This will help 
to compare the number of calls for each fops. There should be some fops 
that samba repeat, and we can find out it by comparing with fuse.


Also if possible, can you please get client profile info from fuse mount 
using the command `setxattr -n trusted.io-stats-dump -v /tmp/iostat.log> `.



Regards

Rafi KC


On 11/5/19 11:05 PM, David Spisla wrote:

I did the test with Gluster 7.0 ctime disabled. But it had no effect:
(All values in MiB/s)
64KiB    1MiB     10MiB
0,16   2,60   54,74

Attached there is now the complete profile file also with the results 
from the last test. I will not repeat it with an higher inode size 
because I don't think this will have an effect.

There must be another cause for the low performance



Yes. No need to try with higher inode size




Regards
David Spisla

Am Di., 5. Nov. 2019 um 16:25 Uhr schrieb David Spisla 
mailto:spisl...@gmail.com>>:




Am Di., 5. Nov. 2019 um 12:06 Uhr schrieb RAFI KC
mailto:rkavu...@redhat.com>>:


On 11/4/19 8:46 PM, David Spisla wrote:

Dear Gluster Community,

I also have a issue concerning performance. The last days I
updated our test cluster from GlusterFS v5.5 to v7.0 . The
setup in general:

2 HP DL380 Servers with 10Gbit NICs, 1 Distribute-Replica 2
Volume with 2 Replica Pairs. Client is SMB Samba (access via
vfs_glusterfs) . I did several tests to ensure that Samba
don't causes the fall.
The setup ist completely the same except the Gluster Version
Here are my results:
64KiB       1MiB             10MiB          (Filesize)
3,49             47,41 300,50  (Values in MiB/s with
GlusterFS v5.5)
0,16              2,61 76,63    (Values in MiB/s with
GlusterFS v7.0)



Can you please share the profile information [1] for both
versions?  Also it would be really helpful if you can mention
the io patterns that used for this tests.

[1] :

https://docs.gluster.org/en/latest/Administrator%20Guide/Monitoring%20Workload/

Hello Rafi,
thank you for your help.

* First more information about the io patterns: As a client we use
a DL360 Windws Server 2017 machine with 10Gbit NIC connected to
the storage machines. The share will be mounted via SMB and the
tests writes with fio. We use this job files (see attachment).
Each job file will be executed separetely and there is a sleep
about 60s between each test run to calm down the system before
starting a new test.

* Attached below you find the profile output from the tests with
v5.5 (ctime enabled), v7.0 (ctime enabled).

* Beside of the tests with Samba I did also some fio tests
directly on the FUSE Mounts (locally on one of the storage nodes).
The results show that there is only a small decrease of
performance between v5.5 and v7.0
(All values in MiB/s)
64KiB    1MiB     10MiB
50,09 679,96   1023,02 (v5.5)
47,00 656,46    977,60 (v7.0)

It seems to be that the combination of samba + gluster7.0 has a
lot of problems, or not?




We use this volume options (GlusterFS 7.0):

Volume Name: archive1
Type: Distributed-Replicate
Volume ID: 44c17844-0bd4-4ca2-98d8-a1474add790c
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: fs-dl380-c1-n1:/gluster/brick1/glusterbrick
Brick2: fs-dl380-c1-n2:/gluster/brick1/glusterbrick
Brick3: fs-dl380-c1-n1:/gluster/brick2/glusterbrick
Brick4: fs-dl380-c1-n2:/gluster/brick2/glusterbrick
Options Reconfigured:
performance.client-io-threads: off
nfs.disable: on
storage.fips-mode-rchecksum: on
transport.address-family: inet
user.smb: disable
features.read-only: off
features.worm: off
features.worm-file-level: on
features.retention-mode: enterprise
features.default-retention-period: 120
network.ping-timeout: 10
features.cache-invalidation: on
features.cache-invalidation-timeout: 600
performance.nl-cache: on
performance.nl-cache-timeout: 600
client.event-threads: 32
server.event-threads: 32
cluster.lookup-optimize: on
performance.stat-prefetch: on
performance.cache-invalidation: on
performance.md-cache-timeout: 600
performance.cache-samba-metadata: on
performance.cache-ima-xattrs: on
performance.io-thread-count: 64
cluster.use-compound-fops: on
performance.cache-size: 512MB
performance.cache-refresh-timeout: 10
   

Re: [Gluster-users] Performance is falling rapidly when updating from v5.5 to v7.0

[David Spisla]

I did the test with Gluster 7.0 ctime disabled. But it had no effect:
(All values in MiB/s)
64KiB1MiB 10MiB
0,16   2,60   54,74

Attached there is now the complete profile file also with the results from
the last test. I will not repeat it with an higher inode size because I
don't think this will have an effect.
There must be another cause for the low performance

Regards
David Spisla

Am Di., 5. Nov. 2019 um 16:25 Uhr schrieb David Spisla :

>
>
> Am Di., 5. Nov. 2019 um 12:06 Uhr schrieb RAFI KC :
>
>>
>> On 11/4/19 8:46 PM, David Spisla wrote:
>>
>> Dear Gluster Community,
>>
>> I also have a issue concerning performance. The last days I updated our
>> test cluster from GlusterFS v5.5 to v7.0 . The setup in general:
>>
>> 2 HP DL380 Servers with 10Gbit NICs, 1 Distribute-Replica 2 Volume with 2
>> Replica Pairs. Client is SMB Samba (access via vfs_glusterfs) . I did
>> several tests to ensure that Samba don't causes the fall.
>> The setup ist completely the same except the Gluster Version
>> Here are my results:
>> 64KiB   1MiB 10MiB(Filesize)
>> 3,49 47,41300,50  (Values in MiB/s with
>> GlusterFS v5.5)
>> 0,16  2,61 76,63(Values in MiB/s with
>> GlusterFS v7.0)
>>
>>
>> Can you please share the profile information [1] for both versions?  Also
>> it would be really helpful if you can mention the io patterns that used for
>> this tests.
>>
>> [1] :
>> https://docs.gluster.org/en/latest/Administrator%20Guide/Monitoring%20Workload/
>>
> Hello Rafi,
> thank you for your help.
>
> * First more information about the io patterns: As a client we use a DL360
> Windws Server 2017 machine with 10Gbit NIC connected to the storage
> machines. The share will be mounted via SMB and the tests writes with fio.
> We use this job files (see attachment). Each job file will be executed
> separetely and there is a sleep about 60s between each test run to calm
> down the system before starting a new test.
>
> * Attached below you find the profile output from the tests with v5.5
> (ctime enabled), v7.0 (ctime enabled).
>
> * Beside of the tests with Samba I did also some fio tests directly on the
> FUSE Mounts (locally on one of the storage nodes). The results show that
> there is only a small decrease of performance between v5.5 and v7.0
> (All values in MiB/s)
> 64KiB1MiB 10MiB
> 50,09 679,96   1023,02 (v5.5)
> 47,00 656,46977,60 (v7.0)
>
> It seems to be that the combination of samba + gluster7.0 has a lot of
> problems, or not?
>
>
>>
>> We use this volume options (GlusterFS 7.0):
>>
>> Volume Name: archive1
>> Type: Distributed-Replicate
>> Volume ID: 44c17844-0bd4-4ca2-98d8-a1474add790c
>> Status: Started
>> Snapshot Count: 0
>> Number of Bricks: 2 x 2 = 4
>> Transport-type: tcp
>> Bricks:
>> Brick1: fs-dl380-c1-n1:/gluster/brick1/glusterbrick
>> Brick2: fs-dl380-c1-n2:/gluster/brick1/glusterbrick
>> Brick3: fs-dl380-c1-n1:/gluster/brick2/glusterbrick
>> Brick4: fs-dl380-c1-n2:/gluster/brick2/glusterbrick
>> Options Reconfigured:
>> performance.client-io-threads: off
>> nfs.disable: on
>> storage.fips-mode-rchecksum: on
>> transport.address-family: inet
>> user.smb: disable
>> features.read-only: off
>> features.worm: off
>> features.worm-file-level: on
>> features.retention-mode: enterprise
>> features.default-retention-period: 120
>> network.ping-timeout: 10
>> features.cache-invalidation: on
>> features.cache-invalidation-timeout: 600
>> performance.nl-cache: on
>> performance.nl-cache-timeout: 600
>> client.event-threads: 32
>> server.event-threads: 32
>> cluster.lookup-optimize: on
>> performance.stat-prefetch: on
>> performance.cache-invalidation: on
>> performance.md-cache-timeout: 600
>> performance.cache-samba-metadata: on
>> performance.cache-ima-xattrs: on
>> performance.io-thread-count: 64
>> cluster.use-compound-fops: on
>> performance.cache-size: 512MB
>> performance.cache-refresh-timeout: 10
>> performance.read-ahead: off
>> performance.write-behind-window-size: 4MB
>> performance.write-behind: on
>> storage.build-pgfid: on
>> features.ctime: on
>> cluster.quorum-type: fixed
>> cluster.quorum-count: 1
>> features.bitrot: on
>> features.scrub: Active
>> features.scrub-freq: daily
>>
>> For GlusterFS 5.5 its nearly the same except the fact that there were 2
>> options to enable ctime feature.
>>
>>
>>
>> Ctime stores additional metadata information as an extended attributes
>> which sometimes exceeds the default inode size. In such scenarios the
>> additional xattrs won't fit into the default size. This will result in
>> additional blocks to be used to store xattrs in the inide, which will
>> effect the latency. This is purely based on the i/o operations and the
>> total xattrs size stored in the inode.
>>
>> Is it possible for you to repeat the test by disabling ctime or
>> increasing the inode size to a higher value say 1024KB?
>>
> I will do so but for today I could not 

Re: [Gluster-users] Performance is falling rapidly when updating from v5.5 to v7.0

[David Spisla]

Am Di., 5. Nov. 2019 um 12:06 Uhr schrieb RAFI KC :

>
> On 11/4/19 8:46 PM, David Spisla wrote:
>
> Dear Gluster Community,
>
> I also have a issue concerning performance. The last days I updated our
> test cluster from GlusterFS v5.5 to v7.0 . The setup in general:
>
> 2 HP DL380 Servers with 10Gbit NICs, 1 Distribute-Replica 2 Volume with 2
> Replica Pairs. Client is SMB Samba (access via vfs_glusterfs) . I did
> several tests to ensure that Samba don't causes the fall.
> The setup ist completely the same except the Gluster Version
> Here are my results:
> 64KiB   1MiB 10MiB(Filesize)
> 3,49 47,41300,50  (Values in MiB/s with
> GlusterFS v5.5)
> 0,16  2,61 76,63(Values in MiB/s with
> GlusterFS v7.0)
>
>
> Can you please share the profile information [1] for both versions?  Also
> it would be really helpful if you can mention the io patterns that used for
> this tests.
>
> [1] :
> https://docs.gluster.org/en/latest/Administrator%20Guide/Monitoring%20Workload/
>
Hello Rafi,
thank you for your help.

* First more information about the io patterns: As a client we use a DL360
Windws Server 2017 machine with 10Gbit NIC connected to the storage
machines. The share will be mounted via SMB and the tests writes with fio.
We use this job files (see attachment). Each job file will be executed
separetely and there is a sleep about 60s between each test run to calm
down the system before starting a new test.

* Attached below you find the profile output from the tests with v5.5
(ctime enabled), v7.0 (ctime enabled).

* Beside of the tests with Samba I did also some fio tests directly on the
FUSE Mounts (locally on one of the storage nodes). The results show that
there is only a small decrease of performance between v5.5 and v7.0
(All values in MiB/s)
64KiB1MiB 10MiB
50,09 679,96   1023,02 (v5.5)
47,00 656,46977,60 (v7.0)

It seems to be that the combination of samba + gluster7.0 has a lot of
problems, or not?


>
> We use this volume options (GlusterFS 7.0):
>
> Volume Name: archive1
> Type: Distributed-Replicate
> Volume ID: 44c17844-0bd4-4ca2-98d8-a1474add790c
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 2 x 2 = 4
> Transport-type: tcp
> Bricks:
> Brick1: fs-dl380-c1-n1:/gluster/brick1/glusterbrick
> Brick2: fs-dl380-c1-n2:/gluster/brick1/glusterbrick
> Brick3: fs-dl380-c1-n1:/gluster/brick2/glusterbrick
> Brick4: fs-dl380-c1-n2:/gluster/brick2/glusterbrick
> Options Reconfigured:
> performance.client-io-threads: off
> nfs.disable: on
> storage.fips-mode-rchecksum: on
> transport.address-family: inet
> user.smb: disable
> features.read-only: off
> features.worm: off
> features.worm-file-level: on
> features.retention-mode: enterprise
> features.default-retention-period: 120
> network.ping-timeout: 10
> features.cache-invalidation: on
> features.cache-invalidation-timeout: 600
> performance.nl-cache: on
> performance.nl-cache-timeout: 600
> client.event-threads: 32
> server.event-threads: 32
> cluster.lookup-optimize: on
> performance.stat-prefetch: on
> performance.cache-invalidation: on
> performance.md-cache-timeout: 600
> performance.cache-samba-metadata: on
> performance.cache-ima-xattrs: on
> performance.io-thread-count: 64
> cluster.use-compound-fops: on
> performance.cache-size: 512MB
> performance.cache-refresh-timeout: 10
> performance.read-ahead: off
> performance.write-behind-window-size: 4MB
> performance.write-behind: on
> storage.build-pgfid: on
> features.ctime: on
> cluster.quorum-type: fixed
> cluster.quorum-count: 1
> features.bitrot: on
> features.scrub: Active
> features.scrub-freq: daily
>
> For GlusterFS 5.5 its nearly the same except the fact that there were 2
> options to enable ctime feature.
>
>
>
> Ctime stores additional metadata information as an extended attributes
> which sometimes exceeds the default inode size. In such scenarios the
> additional xattrs won't fit into the default size. This will result in
> additional blocks to be used to store xattrs in the inide, which will
> effect the latency. This is purely based on the i/o operations and the
> total xattrs size stored in the inode.
>
> Is it possible for you to repeat the test by disabling ctime or increasing
> the inode size to a higher value say 1024KB?
>
I will do so but for today I could not finish tests with ctime disabled (or
higher inode value) because it takes a lot of time with v7.0 due to the low
performance and I will perform it tomorrow. As soon as possible I give you
the results.
By the way: You really mean inode size on xfs layer 1024KB? Or do you mean
1024Bytes? We use per default 512Bytes, because this is the recommended
size until now . But it seems to be that there is a need for a new
recommendation when using ctime feature as a default. I can not image that
this is the real cause for the low performance because in v5.5 we also use
ctime feature with inode size 512Bytes.

Re: [Gluster-users] Performance drop when upgrading from 3.8 to 6.5

[Riccardo Murri]

> > Is it possible for you to repeat the test by disabling ctime or increasing 
> > the inode size to a higher value say 1024?
>
> Sure!  How do I disable ctime or increase the inode size?

Would this suffice to disable `ctime`?

sudo gluster volume set glusterfs ctime off

Can it be done on a running cluster?  Do I need to unmount and remount
for clients to see the effect?

Thanks,
R


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Performance drop when upgrading from 3.8 to 6.5

[Riccardo Murri]

Hello Rafi,

many thanks for looking into this!

> Is it possible for you to repeat the test by disabling ctime or increasing 
> the inode size to a higher value say 1024?

Sure!  How do I disable ctime or increase the inode size?

Ciao,
R


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Performance is falling rapidly when updating from v5.5 to v7.0

[RAFI KC]

On 11/4/19 8:46 PM, David Spisla wrote:

Dear Gluster Community,

I also have a issue concerning performance. The last days I updated 
our test cluster from GlusterFS v5.5 to v7.0 . The setup in general:


2 HP DL380 Servers with 10Gbit NICs, 1 Distribute-Replica 2 Volume 
with 2 Replica Pairs. Client is SMB Samba (access via vfs_glusterfs) . 
I did several tests to ensure that Samba don't causes the fall.

The setup ist completely the same except the Gluster Version
Here are my results:
64KiB       1MiB             10MiB (Filesize)
3,49            47,41            300,50  (Values in MiB/s with 
GlusterFS v5.5)
0,16             2,61     76,63    (Values in MiB/s 
with GlusterFS v7.0)



Can you please share the profile information [1] for both versions?  
Also it would be really helpful if you can mention the io patterns that 
used for this tests.


[1] : 
https://docs.gluster.org/en/latest/Administrator%20Guide/Monitoring%20Workload/




We use this volume options (GlusterFS 7.0):

Volume Name: archive1
Type: Distributed-Replicate
Volume ID: 44c17844-0bd4-4ca2-98d8-a1474add790c
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: fs-dl380-c1-n1:/gluster/brick1/glusterbrick
Brick2: fs-dl380-c1-n2:/gluster/brick1/glusterbrick
Brick3: fs-dl380-c1-n1:/gluster/brick2/glusterbrick
Brick4: fs-dl380-c1-n2:/gluster/brick2/glusterbrick
Options Reconfigured:
performance.client-io-threads: off
nfs.disable: on
storage.fips-mode-rchecksum: on
transport.address-family: inet
user.smb: disable
features.read-only: off
features.worm: off
features.worm-file-level: on
features.retention-mode: enterprise
features.default-retention-period: 120
network.ping-timeout: 10
features.cache-invalidation: on
features.cache-invalidation-timeout: 600
performance.nl-cache: on
performance.nl-cache-timeout: 600
client.event-threads: 32
server.event-threads: 32
cluster.lookup-optimize: on
performance.stat-prefetch: on
performance.cache-invalidation: on
performance.md-cache-timeout: 600
performance.cache-samba-metadata: on
performance.cache-ima-xattrs: on
performance.io-thread-count: 64
cluster.use-compound-fops: on
performance.cache-size: 512MB
performance.cache-refresh-timeout: 10
performance.read-ahead: off
performance.write-behind-window-size: 4MB
performance.write-behind: on
storage.build-pgfid: on
features.ctime: on
cluster.quorum-type: fixed
cluster.quorum-count: 1
features.bitrot: on
features.scrub: Active
features.scrub-freq: daily

For GlusterFS 5.5 its nearly the same except the fact that there were 
2 options to enable ctime feature.




Ctime stores additional metadata information as an extended attributes 
which sometimes exceeds the default inode size. In such scenarios the 
additional xattrs won't fit into the default size. This will result in 
additional blocks to be used to store xattrs in the inide, which will 
effect the latency. This is purely based on the i/o operations and the 
total xattrs size stored in the inode.


Is it possible for you to repeat the test by disabling ctime or 
increasing the inode size to a higher value say 1024KB?




Our optimization for Samba looks like this (for every version):

[global]
workgroup = SAMBA
netbios name = CLUSTER
kernel share modes = no
aio read size = 1
aio write size = 1
kernel oplocks = no
max open files = 10
nt acl support = no
security = user
server min protocol = SMB2
store dos attributes = no
strict locking = no
full_audit:failure = pwrite_send pwrite_recv pwrite offload_write_send 
offload_write_recv create_file open unlink connect disconnect rename 
chown fchown lchown chmod fchmod mkdir rmdir ntimes ftruncate fallocate
full_audit:success = pwrite_send pwrite_recv pwrite offload_write_send 
offload_write_recv create_file open unlink connect disconnect rename 
chown fchown lchown chmod fchmod mkdir rmdir ntimes ftruncate fallocate

full_audit:facility = local5
durable handles = yes
posix locking = no
log level = 2
max log size = 10
debug pid = yes

What can be the cause for this rapid falling of the performance for 
small files? Are some of our vol options not recommended anymore?
There were some patches concerning performance for small files in v6.0 
und v7.0 :


#1670031 : performance regression 
seen with smallfile workload tests


#1659327 : 43% regression in 
small-file sequential read performance


And one patch for the io-cache:

#1659869 : improvements to io-cache

Regards

David Spisla




Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users


Community Meeting Calendar:

Re: [Gluster-users] Performance drop when upgrading from 3.8 to 6.5

[RAFI KC]

On 11/4/19 2:41 PM, Riccardo Murri wrote:

Hello Amar,


Can you please check the profile info [1]  ? That may give some hints.

I am attaching the output of `sudo gluster volume profile info` as a text file
to preserve formatting.  This covers the time from Friday night to
Monday morning;
during this time the cluster has been the target of a an `rsync` command copying
a large directory tree (14TB; it's taking more than 2 weeks now...).

My take from cursorily reading the profile:

- metadata operations (opendir, create, rename) seem to have very high
latency (up to 25 seconds for a rename!)



We have identified a performance drop with ctime and rename operations. 
This is because ctime stores some metadata information as an extended 
attributes which sometimes exceeds the default inode size. In such 
scenarios the additional xattrs won't fit into the default size. This 
will result in additional blocks to be used which will effect the latency.


Is it possible for you to repeat the test by disabling ctime or 
increasing the inode size to a higher value say 1024?




- large block size (>128kB) are rare; most blocks seems to be 8kB,
16kB, or 128kB.

I personally do not see any obvious culprit or optimization
opportunities; is there something I'm missing out that catches the eye
of someone more experienced?

Thanks,
R



Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Performance is falling rapidly when updating from v5.5 to v7.0

[David Spisla]

Dear Gluster Community,

I also have a issue concerning performance. The last days I updated our
test cluster from GlusterFS v5.5 to v7.0 . The setup in general:

2 HP DL380 Servers with 10Gbit NICs, 1 Distribute-Replica 2 Volume with 2
Replica Pairs. Client is SMB Samba (access via vfs_glusterfs) . I did
several tests to ensure that Samba don't causes the fall.
The setup ist completely the same except the Gluster Version
Here are my results:
64KiB   1MiB 10MiB(Filesize)
3,49 47,41300,50  (Values in MiB/s with
GlusterFS v5.5)
0,16  2,61 76,63(Values in MiB/s with
GlusterFS v7.0)

We use this volume options (GlusterFS 7.0):

Volume Name: archive1
Type: Distributed-Replicate
Volume ID: 44c17844-0bd4-4ca2-98d8-a1474add790c
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: fs-dl380-c1-n1:/gluster/brick1/glusterbrick
Brick2: fs-dl380-c1-n2:/gluster/brick1/glusterbrick
Brick3: fs-dl380-c1-n1:/gluster/brick2/glusterbrick
Brick4: fs-dl380-c1-n2:/gluster/brick2/glusterbrick
Options Reconfigured:
performance.client-io-threads: off
nfs.disable: on
storage.fips-mode-rchecksum: on
transport.address-family: inet
user.smb: disable
features.read-only: off
features.worm: off
features.worm-file-level: on
features.retention-mode: enterprise
features.default-retention-period: 120
network.ping-timeout: 10
features.cache-invalidation: on
features.cache-invalidation-timeout: 600
performance.nl-cache: on
performance.nl-cache-timeout: 600
client.event-threads: 32
server.event-threads: 32
cluster.lookup-optimize: on
performance.stat-prefetch: on
performance.cache-invalidation: on
performance.md-cache-timeout: 600
performance.cache-samba-metadata: on
performance.cache-ima-xattrs: on
performance.io-thread-count: 64
cluster.use-compound-fops: on
performance.cache-size: 512MB
performance.cache-refresh-timeout: 10
performance.read-ahead: off
performance.write-behind-window-size: 4MB
performance.write-behind: on
storage.build-pgfid: on
features.ctime: on
cluster.quorum-type: fixed
cluster.quorum-count: 1
features.bitrot: on
features.scrub: Active
features.scrub-freq: daily

For GlusterFS 5.5 its nearly the same except the fact that there were 2
options to enable ctime feature.
Our optimization for Samba looks like this (for every version):

[global]
workgroup = SAMBA
netbios name = CLUSTER
kernel share modes = no
aio read size = 1
aio write size = 1
kernel oplocks = no
max open files = 10
nt acl support = no
security = user
server min protocol = SMB2
store dos attributes = no
strict locking = no
full_audit:failure = pwrite_send pwrite_recv pwrite offload_write_send
offload_write_recv create_file open unlink connect disconnect rename chown
fchown lchown chmod fchmod mkdir rmdir ntimes ftruncate fallocate
full_audit:success = pwrite_send pwrite_recv pwrite offload_write_send
offload_write_recv create_file open unlink connect disconnect rename chown
fchown lchown chmod fchmod mkdir rmdir ntimes ftruncate fallocate
full_audit:facility = local5
durable handles = yes
posix locking = no
log level = 2
max log size = 10
debug pid = yes

What can be the cause for this rapid falling of the performance for small
files? Are some of our vol options not recommended anymore?
There were some patches concerning performance for small files in v6.0 und
v7.0 :

#1670031 : performance regression seen
with smallfile workload tests

#1659327 : 43% regression in
small-file sequential read performance

And one patch for the io-cache:

#1659869 : improvements to io-cache

Regards

David Spisla


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Performance drop when upgrading from 3.8 to 6.5

[Riccardo Murri]

Hello Strahil,

> You can set your mounts with 'noatime,nodiratime' options for better 
> performance.

Thanks for the suggestion!  I'll try that eventually, but I don't
think `noatime` will make much difference on write-mostly workload.

Thanks,
R


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Performance drop when upgrading from 3.8 to 6.5

[Riccardo Murri]

Hello Amar,

> Can you please check the profile info [1]  ? That may give some hints.

I am attaching the output of `sudo gluster volume profile info` as a text file
to preserve formatting.  This covers the time from Friday night to
Monday morning;
during this time the cluster has been the target of a an `rsync` command copying
a large directory tree (14TB; it's taking more than 2 weeks now...).

My take from cursorily reading the profile:

- metadata operations (opendir, create, rename) seem to have very high
latency (up to 25 seconds for a rename!)
- large block size (>128kB) are rare; most blocks seems to be 8kB,
16kB, or 128kB.

I personally do not see any obvious culprit or optimization
opportunities; is there something I'm missing out that catches the eye
of someone more experienced?

Thanks,
R
$ sudo gluster volume profile glusterfs info
Brick: glusterfs-server-001:/srv/glusterfs
--
Cumulative Stats:
   Block Size:  2b+   4b+   8b+
 No. of Reads:21812
No. of Writes:   471522

   Block Size: 16b+  32b+  64b+
 No. of Reads:   3177   336
No. of Writes:   65  7869 47979

   Block Size:128b+ 256b+ 512b+
 No. of Reads:  291   648  1125
No. of Writes: 8909125293 62952

   Block Size:   1024b+2048b+4096b+
 No. of Reads: 2054  3868  7903
No. of Writes:11728293386 87850

   Block Size:   8192b+   16384b+   32768b+
 No. of Reads:16084 35981 66646
No. of Writes: 10758668   6852707289622

   Block Size:  65536b+  131072b+  262144b+
 No. of Reads:   128659  15220392 0
No. of Writes:   239551  12723330   316

 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls Fop
 -   ---   ---   ---   
  0.00   0.00 us   0.00 us   0.00 us 760721  FORGET
  0.00   0.00 us   0.00 us   0.00 us 877203 RELEASE
  0.00   0.00 us   0.00 us   0.00 us  16092  RELEASEDIR
  0.00   16174.60 us 554.21 us   31794.98 us  2READDIRP
  0.00 185.11 us  82.75 us 660.68 us377OPEN
  0.00   95974.52 us   95974.52 us   95974.52 us  1 OPENDIR
  0.01   59611.16 us 444.39 us  160460.53 us  3 SYMLINK
  0.01 217.87 us  72.47 us   15655.59 us   1337SETXATTR
  0.30  79.48 us  16.15 us6242.48 us 123275   FLUSH
  0.31 120.49 us  30.68 us   10641.08 us  83225   FSTAT
  0.50   12161.43 us 273.52 us  298385.31 us   1337   MKDIR
  0.86 129.11 us  32.64 us9029.93 us 216678  STATFS
  1.35  89.34 us  12.29 us   16695.73 us 491862 ENTRYLK
  2.09  91.83 us  11.85 us   20615.78 us 737946 INODELK
  2.38 628.28 us 115.01 us 25374396.01 us 122889  RENAME
  2.422080.96 us  36.39 us  403164.19 us  37810READ
  2.74 179.20 us  49.63 us 1041137.91 us 497183 SETATTR
  6.34 154.95 us  33.60 us   17451.12 us1329171STAT
 11.35 201.91 us  31.14 us  181613.31 us1824910  LOOKUP
 28.657568.96 us 169.02 us 1888964.31 us 122898  CREATE
 40.68 517.99 us  58.77 us 1076940.31 us2549418   WRITE

Duration: 2936614 seconds
   Data Read: 2012268179193 bytes
Data Written: 2005120204522 bytes

Interval 0 Stats:
   Block Size:  2b+   4b+   8b+
 No. of Reads:21812
No. of Writes:   471522

   Block Size: 16b+  32b+  64b+
 No. of Reads:   3177   336
No. of Writes:   65  7869 47979

   Block Size:128b+ 256b+ 512b+
 No. of Reads:  291   648  

Re: [Gluster-users] Performance drop when upgrading from 3.8 to 6.5

[Strahil]

Hm... This seems to be cluster-wide effect than a single brick.

In order to make things faster, can you remount (mount -o 
remount,noatime,nodiratime /gluster_brick/) on all bricks in the same 
volume and take the test again ?

I think I saw your gluster bricks are mounted without these options.


Also, are you using XFS  as brick FS?

Best Regards,
Strahil NikolovOn Nov 1, 2019 21:21, Riccardo Murri  
wrote:
>
> Dear Strahil, 
>
> > Have you noticed  if slowness  is  only when accessing the files  from 
> > specific node  ? 
>
> I am copying a largest of image files into the GlusterFS volume -- 
> slowness is on the aggregated performance (e.g., it takes ~300 minutes 
> to copy 376GB worth of files).  Given the high number of files 
> (O(100'000)), I guess they're +/- equally distributed across nodes. 
> Report of `df -h` across server nodes shows no imbalance. 
>
> Thanks, 
> R 


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Performance drop when upgrading from 3.8 to 6.5

[Riccardo Murri]

Dear Strahil,

> Have you noticed  if slowness  is  only when accessing the files  from 
> specific node  ?

I am copying a largest of image files into the GlusterFS volume --
slowness is on the aggregated performance (e.g., it takes ~300 minutes
to copy 376GB worth of files).  Given the high number of files
(O(100'000)), I guess they're +/- equally distributed across nodes.
Report of `df -h` across server nodes shows no imbalance.

Thanks,
R


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Performance drop when upgrading from 3.8 to 6.5

[Riccardo Murri]

Dear Amar,

> Can you please check the profile info [1]  ? That may give some hints.

I have started profiling, will check what info has been collected on Monday.

Many thanks for the suggestion!

Riccardo


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Performance drop when upgrading from 3.8 to 6.5

[Amar Tumballi]

Hi Riccardo,

Can you please check the profile info [1]  ? That may give some hints.

[1] -
https://docs.gluster.org/en/latest/Administrator%20Guide/Monitoring%20Workload/
 ?



On Fri, 1 Nov, 2019, 9:55 AM Riccardo Murri, 
wrote:

> Hello all,
>
> I have done some further testing and found out that I get the bad
> performance with a freshly-installed cluster running 6.6. Also the
> performance drop is there with plain `rsync` into the GlusterFS
> mountpoint, so SAMBA plays no role in it.  In other words, for my
> installations, performance of 6.5 and 6.6 is *half* of what 3.8 used
> to deliver.
>
> Was any default option changed (e.g., in the FUSE client) or in the
> xlator stack that I can start looking at as a potential culprit?  Or
> any direction where to look at for debugging?
>
> Thanks for any help!
>
> Riccardo
> 
>
> Community Meeting Calendar:
>
> APAC Schedule -
> Every 2nd and 4th Tuesday at 11:30 AM IST
> Bridge: https://bluejeans.com/118564314
>
> NA/EMEA Schedule -
> Every 1st and 3rd Tuesday at 01:00 PM EDT
> Bridge: https://bluejeans.com/118564314
>
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Performance drop when upgrading from 3.8 to 6.5

[Strahil]

I'm using replicated volumes.
In your case, you got a distribited volume.
Have you noticed  if slowness  is  only when accessing the files  from specific 
node  ?

Best Regards,
Strahil NikolovOn Nov 1, 2019 17:28, Riccardo Murri  
wrote:
>
> Hello Strahil 
>
> > What options do you use i  your cluster? 
>
> I'm not sure what exact info you would like to see? 
>
> Here's how clients mount the GlusterFS volume: 
> ``` 
> $ fgrep gluster /proc/mounts 
> tp-glusterfs5:/glusterfs /net/glusterfs fuse.glusterfs 
> rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072
>  
> 0 0 
> ``` 
>
> Here's some server-side info: 
>
> ``` 
> $ sudo gluster volume info glusterfs 
>
> Volume Name: glusterfs 
> Type: Distribute 
> Volume ID: a3358ff6-5cec-4a65-9ecf-a63bbe56dfd9 
> Status: Started 
> Snapshot Count: 0 
> Number of Bricks: 5 
> Transport-type: tcp 
> Bricks: 
> Brick1: pelkmanslab-tp-glusterfs5:/srv/glusterfs 
> Brick2: pelkmanslab-tp-glusterfs4:/srv/glusterfs 
> Brick3: pelkmanslab-tp-glusterfs3:/srv/glusterfs 
> Brick4: pelkmanslab-tp-glusterfs1:/srv/glusterfs 
> Brick5: pelkmanslab-tp-glusterfs2:/srv/glusterfs 
> Options Reconfigured: 
> diagnostics.client-log-level: WARNING 
> diagnostics.brick-log-level: INFO 
> features.uss: disable 
> features.barrier: disable 
> performance.client-io-threads: on 
> transport.address-family: inet 
> nfs.disable: on 
> snap-activate-on-create: enable 
>
> $ sudo gluster volume get all all 
> Option  Value 
> --  - 
> cluster.server-quorum-ratio 51 
> cluster.enable-shared-storage   disable 
> cluster.op-version  6 
> cluster.max-op-version  6 
> cluster.brick-multiplex disable 
> cluster.max-bricks-per-process  250 
> cluster.localtime-logging   disable 
> cluster.daemon-log-level    INFO 
>
> $ cat /etc/glusterfs/glusterd.vol 
> volume management 
>     type mgmt/glusterd 
>     option working-directory /var/lib/glusterd 
>     option transport-type socket,rdma 
>     option transport.socket.keepalive-time 10 
>     option transport.socket.keepalive-interval 2 
>     option transport.socket.read-fail-log off 
>     option transport.socket.listen-port 24007 
>     option transport.rdma.listen-port 24008 
>     option ping-timeout 0 
>     option event-threads 1 
> #   option lock-timer 180 
> #   option transport.address-family inet6 
> #   option base-port 49152 
>     option max-port  60999 
> end-volume 
> ``` 
>
> Both server and clients are running v6.5 and op-version is 6 everywhere. 
>
> Thanks, 
> Riccardo 


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Performance drop when upgrading from 3.8 to 6.5

[Riccardo Murri]

Hello Strahil

> What options do you use i  your cluster?

I'm not sure what exact info you would like to see?

Here's how clients mount the GlusterFS volume:
```
$ fgrep gluster /proc/mounts
tp-glusterfs5:/glusterfs /net/glusterfs fuse.glusterfs
rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072
0 0
```

Here's some server-side info:

```
$ sudo gluster volume info glusterfs

Volume Name: glusterfs
Type: Distribute
Volume ID: a3358ff6-5cec-4a65-9ecf-a63bbe56dfd9
Status: Started
Snapshot Count: 0
Number of Bricks: 5
Transport-type: tcp
Bricks:
Brick1: pelkmanslab-tp-glusterfs5:/srv/glusterfs
Brick2: pelkmanslab-tp-glusterfs4:/srv/glusterfs
Brick3: pelkmanslab-tp-glusterfs3:/srv/glusterfs
Brick4: pelkmanslab-tp-glusterfs1:/srv/glusterfs
Brick5: pelkmanslab-tp-glusterfs2:/srv/glusterfs
Options Reconfigured:
diagnostics.client-log-level: WARNING
diagnostics.brick-log-level: INFO
features.uss: disable
features.barrier: disable
performance.client-io-threads: on
transport.address-family: inet
nfs.disable: on
snap-activate-on-create: enable

$ sudo gluster volume get all all
Option  Value
--  -
cluster.server-quorum-ratio 51
cluster.enable-shared-storage   disable
cluster.op-version  6
cluster.max-op-version  6
cluster.brick-multiplex disable
cluster.max-bricks-per-process  250
cluster.localtime-logging   disable
cluster.daemon-log-levelINFO

$ cat /etc/glusterfs/glusterd.vol
volume management
type mgmt/glusterd
option working-directory /var/lib/glusterd
option transport-type socket,rdma
option transport.socket.keepalive-time 10
option transport.socket.keepalive-interval 2
option transport.socket.read-fail-log off
option transport.socket.listen-port 24007
option transport.rdma.listen-port 24008
option ping-timeout 0
option event-threads 1
#   option lock-timer 180
#   option transport.address-family inet6
#   option base-port 49152
option max-port  60999
end-volume
```

Both server and clients are running v6.5 and op-version is 6 everywhere.

Thanks,
Riccardo


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Performance drop when upgrading from 3.8 to 6.5

[Strahil]

What options do you use i  your cluster?

Best Regards,
Strahil NikolovOn Nov 1, 2019 06:24, Riccardo Murri  
wrote:
>
> Hello all, 
>
> I have done some further testing and found out that I get the bad 
> performance with a freshly-installed cluster running 6.6. Also the 
> performance drop is there with plain `rsync` into the GlusterFS 
> mountpoint, so SAMBA plays no role in it.  In other words, for my 
> installations, performance of 6.5 and 6.6 is *half* of what 3.8 used 
> to deliver. 
>
> Was any default option changed (e.g., in the FUSE client) or in the 
> xlator stack that I can start looking at as a potential culprit?  Or 
> any direction where to look at for debugging? 
>
> Thanks for any help! 
>
> Riccardo 
>  
>
> Community Meeting Calendar: 
>
> APAC Schedule - 
> Every 2nd and 4th Tuesday at 11:30 AM IST 
> Bridge: https://bluejeans.com/118564314 
>
> NA/EMEA Schedule - 
> Every 1st and 3rd Tuesday at 01:00 PM EDT 
> Bridge: https://bluejeans.com/118564314 
>
> Gluster-users mailing list 
> Gluster-users@gluster.org 
> https://lists.gluster.org/mailman/listinfo/gluster-users 


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Performance drop when upgrading from 3.8 to 6.5

[Riccardo Murri]

Hello all,

I have done some further testing and found out that I get the bad
performance with a freshly-installed cluster running 6.6. Also the
performance drop is there with plain `rsync` into the GlusterFS
mountpoint, so SAMBA plays no role in it.  In other words, for my
installations, performance of 6.5 and 6.6 is *half* of what 3.8 used
to deliver.

Was any default option changed (e.g., in the FUSE client) or in the
xlator stack that I can start looking at as a potential culprit?  Or
any direction where to look at for debugging?

Thanks for any help!

Riccardo


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Performance drop when upgrading from 3.8 to 6.5

[Riccardo Murri]

> In previous discussions it was confirmed by others that v5.5 is a little bit 
> slower than v3.12 , but I think that most of those issues were fixed in v6 .
> What was the exact version you have?

6.5 according to the package version; op-version is 6.

Thanks,
Riccardo


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Performance drop when upgrading from 3.8 to 6.5

[Strahil]

In previous discussions it was confirmed by others that v5.5 is a little bit 
slower than v3.12 , but I think that most of those issues were fixed in v6 .
What was the exact version you have?

Best Regards,
Strahil NikolovOn Oct 29, 2019 12:50, Riccardo Murri  
wrote:
>
> Hello Anoop, 
>
> many thanks for your fast reply!  My comments inline below: 
>
>
> > > [1]: I have tried both the config where SAMBA 4.8 is using the 
> > > vfs_glusterfs.so backend, and the one where `smbd` is just writing to 
> > > a locally-mounted directory.  Doesn't seem to make a difference. 
> > 
> > Samba v4.8 is an EOL ed version. Please consider updating Samba to at 
> > least v4.9(rather v4.10) or higher. 
>
> This is going to be tricky: I could find no backport package of recent 
> SAMBA to Ubuntu 16.04; I am using this one which has SAMBA 4.8 
> https://launchpad.net/~mumblepins 
>
> More recent packages from either the Ubuntu or Debian repositories do 
> not build on Ubuntu 16.04 because of changes in the packaging 
> infrastructure. 
>
> Anyway, I was running SAMBA 4.8 before the upgrade and still getting 
> 40MB/s, so I don't think SAMBA is the core of the issue... 
>
> > Can you paste the output of `testparm -s` along with the output of 
> > `gluster volume info ` ? 
>
> Here's `testparm -s` on the server using `vfs_glusterfs` (the "active" 
> share is the one with the perf problems):: 
>
> ``` 
> $ testparm -s 
> Load smb config files from /etc/samba/smb.conf 
> rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384) 
> WARNING: The "syslog only" option is deprecated 
> Processing section "[homes]" 
> Processing section "[active]" 
> Loaded services file OK. 
> WARNING: some services use vfs_fruit, others don't. Mounting them in 
> conjunction on OS X clients results in undefined behaviour. 
>
> Server role: ROLE_STANDALONE 
>
> # Global parameters 
> [global] 
>     dns proxy = No 
>     load printers = No 
>     map to guest = Bad User 
>     name resolve order = lmhosts 
>     netbios name = REDACTED1 
>     obey pam restrictions = Yes 
>     pam password change = Yes 
>     passwd chat = *Enter\snew\s*\spassword:* %n\n 
> *Retype\snew\s*\spassword:* %n\n *password\supdated\ssuccessfully* . 
>     passwd program = /usr/bin/passwd %u 
>     printcap cache time = 0 
>     printcap name = /dev/null 
>     security = USER 
>     server role = standalone server 
>     server string = SAMBA Server %v 
>     syslog only = Yes 
>     unix password sync = Yes 
>     workgroup = REDACTED 
>     idmap config * : backend = tdb 
>
>
> [homes] 
>     browseable = No 
>     comment = Work Directories 
>     create mask = 0700 
>     directory mask = 0700 
>     read only = No 
>     valid users = %S 
>     vfs objects = fruit streams_xattr 
>
>
> [active] 
>     create mask = 0775 
>     directory mask = 0775 
>     kernel share modes = No 
>     path = /active 
>     read only = No 
>     vfs objects = glusterfs 
>     glusterfs:volume = glusterfs 
>     glusterfs:volfile_server = glusterfs5 glusterfs4 glusterfs3 
> glusterfs2 glusterfs1 
>     glusterfs:logfile = /var/log/samba/glusterfs-vol-active.log 
>     glusterfs:loglevel = 1 
> ``` 
>
>
> Here's `testparm -s` on the server writing directly to the GlusterFS 
> mount point:: 
>
> ``` 
> $ testparm -s 
> Load smb config files from /etc/samba/smb.conf 
> rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384) 
> WARNING: The "syslog only" option is deprecated 
> Processing section "[homes]" 
> Processing section "[active]" 
> Loaded services file OK. 
> WARNING: some services use vfs_fruit, others don't. Mounting them in 
> conjunction on OS X clients results in undefined behaviour. 
>
> Server role: ROLE_STANDALONE 
>
> # Global parameters 
> [global] 
>     allow insecure wide links = Yes 
>     dns proxy = No 
>     load printers = No 
>     map to guest = Bad User 
>     name resolve order = lmhosts 
>     netbios name = REDACTED2 
>     obey pam restrictions = Yes 
>     pam password change = Yes 
>     passwd chat = *Enter\snew\s*\spassword:* %n\n 
> *Retype\snew\s*\spassword:* %n\n *password\supdated\ssuccessfully* . 
>     passwd program = /usr/bin/passwd %u 
>     printcap cache time = 0 
>     printcap name = /dev/null 
>     security = USER 
>     server role = standalone server 
>     server string = SAMBA Server %v 
>     syslog only = Yes 
>     unix password sync = Yes 
>     workgroup = REDACTED 
>     idmap config * : backend = tdb 
>
>
> [homes] 
>     browseable = No 
>     comment = Work Directories 
>     create mask = 0700 
>     directory mask = 0700 
>     read only = No 
>     valid users = %S 
>     vfs objects = fruit streams_xattr 
>
>
> [active] 
>     create mask = 0775 
>     directory mask = 0775 
>     path = /data/active 
>     read only = No 
>     wide links = Yes 
> ``` 
>
> Here's the volume info: 
> ``` 
> $ sudo gluster volume info glusterfs 
>
> Volume Name: glusterfs 
> Type: Distribute 
> Volume 

Re: [Gluster-users] Performance drop when upgrading from 3.8 to 6.5

[Strahil]

Hi Riccardo,

You can set your mounts with 'noatime,nodiratime' options for better 
performance.

Best  Regards,
Strahil NikolovOn Oct 29, 2019 12:50, Riccardo Murri  
wrote:
>
> Hello Anoop, 
>
> many thanks for your fast reply!  My comments inline below: 
>
>
> > > [1]: I have tried both the config where SAMBA 4.8 is using the 
> > > vfs_glusterfs.so backend, and the one where `smbd` is just writing to 
> > > a locally-mounted directory.  Doesn't seem to make a difference. 
> > 
> > Samba v4.8 is an EOL ed version. Please consider updating Samba to at 
> > least v4.9(rather v4.10) or higher. 
>
> This is going to be tricky: I could find no backport package of recent 
> SAMBA to Ubuntu 16.04; I am using this one which has SAMBA 4.8 
> https://launchpad.net/~mumblepins 
>
> More recent packages from either the Ubuntu or Debian repositories do 
> not build on Ubuntu 16.04 because of changes in the packaging 
> infrastructure. 
>
> Anyway, I was running SAMBA 4.8 before the upgrade and still getting 
> 40MB/s, so I don't think SAMBA is the core of the issue... 
>
> > Can you paste the output of `testparm -s` along with the output of 
> > `gluster volume info ` ? 
>
> Here's `testparm -s` on the server using `vfs_glusterfs` (the "active" 
> share is the one with the perf problems):: 
>
> ``` 
> $ testparm -s 
> Load smb config files from /etc/samba/smb.conf 
> rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384) 
> WARNING: The "syslog only" option is deprecated 
> Processing section "[homes]" 
> Processing section "[active]" 
> Loaded services file OK. 
> WARNING: some services use vfs_fruit, others don't. Mounting them in 
> conjunction on OS X clients results in undefined behaviour. 
>
> Server role: ROLE_STANDALONE 
>
> # Global parameters 
> [global] 
>     dns proxy = No 
>     load printers = No 
>     map to guest = Bad User 
>     name resolve order = lmhosts 
>     netbios name = REDACTED1 
>     obey pam restrictions = Yes 
>     pam password change = Yes 
>     passwd chat = *Enter\snew\s*\spassword:* %n\n 
> *Retype\snew\s*\spassword:* %n\n *password\supdated\ssuccessfully* . 
>     passwd program = /usr/bin/passwd %u 
>     printcap cache time = 0 
>     printcap name = /dev/null 
>     security = USER 
>     server role = standalone server 
>     server string = SAMBA Server %v 
>     syslog only = Yes 
>     unix password sync = Yes 
>     workgroup = REDACTED 
>     idmap config * : backend = tdb 
>
>
> [homes] 
>     browseable = No 
>     comment = Work Directories 
>     create mask = 0700 
>     directory mask = 0700 
>     read only = No 
>     valid users = %S 
>     vfs objects = fruit streams_xattr 
>
>
> [active] 
>     create mask = 0775 
>     directory mask = 0775 
>     kernel share modes = No 
>     path = /active 
>     read only = No 
>     vfs objects = glusterfs 
>     glusterfs:volume = glusterfs 
>     glusterfs:volfile_server = glusterfs5 glusterfs4 glusterfs3 
> glusterfs2 glusterfs1 
>     glusterfs:logfile = /var/log/samba/glusterfs-vol-active.log 
>     glusterfs:loglevel = 1 
> ``` 
>
>
> Here's `testparm -s` on the server writing directly to the GlusterFS 
> mount point:: 
>
> ``` 
> $ testparm -s 
> Load smb config files from /etc/samba/smb.conf 
> rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384) 
> WARNING: The "syslog only" option is deprecated 
> Processing section "[homes]" 
> Processing section "[active]" 
> Loaded services file OK. 
> WARNING: some services use vfs_fruit, others don't. Mounting them in 
> conjunction on OS X clients results in undefined behaviour. 
>
> Server role: ROLE_STANDALONE 
>
> # Global parameters 
> [global] 
>     allow insecure wide links = Yes 
>     dns proxy = No 
>     load printers = No 
>     map to guest = Bad User 
>     name resolve order = lmhosts 
>     netbios name = REDACTED2 
>     obey pam restrictions = Yes 
>     pam password change = Yes 
>     passwd chat = *Enter\snew\s*\spassword:* %n\n 
> *Retype\snew\s*\spassword:* %n\n *password\supdated\ssuccessfully* . 
>     passwd program = /usr/bin/passwd %u 
>     printcap cache time = 0 
>     printcap name = /dev/null 
>     security = USER 
>     server role = standalone server 
>     server string = SAMBA Server %v 
>     syslog only = Yes 
>     unix password sync = Yes 
>     workgroup = REDACTED 
>     idmap config * : backend = tdb 
>
>
> [homes] 
>     browseable = No 
>     comment = Work Directories 
>     create mask = 0700 
>     directory mask = 0700 
>     read only = No 
>     valid users = %S 
>     vfs objects = fruit streams_xattr 
>
>
> [active] 
>     create mask = 0775 
>     directory mask = 0775 
>     path = /data/active 
>     read only = No 
>     wide links = Yes 
> ``` 
>
> Here's the volume info: 
> ``` 
> $ sudo gluster volume info glusterfs 
>
> Volume Name: glusterfs 
> Type: Distribute 
> Volume ID: a3358ff6-5cec-4a65-9ecf-a63bbe56dfd9 
> Status: Started 
> Snapshot Count: 0 
> Number of 

Re: [Gluster-users] Performance drop when upgrading from 3.8 to 6.5

[Riccardo Murri]

Hello Anoop,

many thanks for your fast reply!  My comments inline below:


> > [1]: I have tried both the config where SAMBA 4.8 is using the
> > vfs_glusterfs.so backend, and the one where `smbd` is just writing to
> > a locally-mounted directory.  Doesn't seem to make a difference.
>
> Samba v4.8 is an EOL ed version. Please consider updating Samba to at
> least v4.9(rather v4.10) or higher.

This is going to be tricky: I could find no backport package of recent
SAMBA to Ubuntu 16.04; I am using this one which has SAMBA 4.8
https://launchpad.net/~mumblepins

More recent packages from either the Ubuntu or Debian repositories do
not build on Ubuntu 16.04 because of changes in the packaging
infrastructure.

Anyway, I was running SAMBA 4.8 before the upgrade and still getting
40MB/s, so I don't think SAMBA is the core of the issue...

> Can you paste the output of `testparm -s` along with the output of
> `gluster volume info ` ?

Here's `testparm -s` on the server using `vfs_glusterfs` (the "active"
share is the one with the perf problems)::

```
$ testparm -s
Load smb config files from /etc/samba/smb.conf
rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384)
WARNING: The "syslog only" option is deprecated
Processing section "[homes]"
Processing section "[active]"
Loaded services file OK.
WARNING: some services use vfs_fruit, others don't. Mounting them in
conjunction on OS X clients results in undefined behaviour.

Server role: ROLE_STANDALONE

# Global parameters
[global]
dns proxy = No
load printers = No
map to guest = Bad User
name resolve order = lmhosts
netbios name = REDACTED1
obey pam restrictions = Yes
pam password change = Yes
passwd chat = *Enter\snew\s*\spassword:* %n\n
*Retype\snew\s*\spassword:* %n\n *password\supdated\ssuccessfully* .
passwd program = /usr/bin/passwd %u
printcap cache time = 0
printcap name = /dev/null
security = USER
server role = standalone server
server string = SAMBA Server %v
syslog only = Yes
unix password sync = Yes
workgroup = REDACTED
idmap config * : backend = tdb


[homes]
browseable = No
comment = Work Directories
create mask = 0700
directory mask = 0700
read only = No
valid users = %S
vfs objects = fruit streams_xattr


[active]
create mask = 0775
directory mask = 0775
kernel share modes = No
path = /active
read only = No
vfs objects = glusterfs
glusterfs:volume = glusterfs
glusterfs:volfile_server = glusterfs5 glusterfs4 glusterfs3
glusterfs2 glusterfs1
glusterfs:logfile = /var/log/samba/glusterfs-vol-active.log
glusterfs:loglevel = 1
```


Here's `testparm -s` on the server writing directly to the GlusterFS
mount point::

```
$ testparm -s
Load smb config files from /etc/samba/smb.conf
rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384)
WARNING: The "syslog only" option is deprecated
Processing section "[homes]"
Processing section "[active]"
Loaded services file OK.
WARNING: some services use vfs_fruit, others don't. Mounting them in
conjunction on OS X clients results in undefined behaviour.

Server role: ROLE_STANDALONE

# Global parameters
[global]
allow insecure wide links = Yes
dns proxy = No
load printers = No
map to guest = Bad User
name resolve order = lmhosts
netbios name = REDACTED2
obey pam restrictions = Yes
pam password change = Yes
passwd chat = *Enter\snew\s*\spassword:* %n\n
*Retype\snew\s*\spassword:* %n\n *password\supdated\ssuccessfully* .
passwd program = /usr/bin/passwd %u
printcap cache time = 0
printcap name = /dev/null
security = USER
server role = standalone server
server string = SAMBA Server %v
syslog only = Yes
unix password sync = Yes
workgroup = REDACTED
idmap config * : backend = tdb


[homes]
browseable = No
comment = Work Directories
create mask = 0700
directory mask = 0700
read only = No
valid users = %S
vfs objects = fruit streams_xattr


[active]
create mask = 0775
directory mask = 0775
path = /data/active
read only = No
wide links = Yes
```

Here's the volume info:
```
$ sudo gluster volume info glusterfs

Volume Name: glusterfs
Type: Distribute
Volume ID: a3358ff6-5cec-4a65-9ecf-a63bbe56dfd9
Status: Started
Snapshot Count: 0
Number of Bricks: 5
Transport-type: tcp
Bricks:
Brick1: glusterfs5:/srv/glusterfs
Brick2: glusterfs4:/srv/glusterfs
Brick3: glusterfs3:/srv/glusterfs
Brick4: glusterfs1:/srv/glusterfs
Brick5: glusterfs2:/srv/glusterfs
Options Reconfigured:
diagnostics.client-log-level: WARNING
diagnostics.brick-log-level: INFO
features.uss: disable
features.barrier: disable
performance.client-io-threads: on
transport.address-family: inet
nfs.disable: on
snap-activate-on-create: enable
```


> > [2]: Actually, since the servers are VMs on an OpenStack cloud, I
> > created new virtual machines, installed GlusterFS 6 

Re: [Gluster-users] Performance drop when upgrading from 3.8 to 6.5

[Anoop C S]

On Tue, 2019-10-29 at 10:59 +0100, Riccardo Murri wrote:
> Hello,
> 
> I recently upgraded[2] our servers from GlusterFS 3.8 (old GlusterFS
> repo for Ubuntu 16.04) to 6.0 (gotten from the GlusterFS PPA for
> Ubuntu 16.04 "xenial").
> 
> The sustained write performance nearly dropped to half it was before.
> We copy a large (a few 10'000s) number of image files (each 2 to 10
> MB
> size) from the microscope where they were acquired to a SAMBA server
> which mounts[1] the GlusterFS volume; before the upgrade, we could
> write at about 40MB/s, after the upgrade, this dropped to 20MB/s.
> 
> This is the version of server and client software installed:
> ```
> $ dpkg -l '*gluster*'
> Desired=Unknown/Install/Remove/Purge/Hold
> > Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-
> > aWait/Trig-pend
> > / Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
> > > /
> > > Name   Version  Architecture
>   Description
> +++-==--
> -
> ==
> ii  glusterfs-client   6.5-ubuntu1~xenial1  amd64
>   clustered file-system (client package)
> ii  glusterfs-common   6.5-ubuntu1~xenial1  amd64
>   GlusterFS common libraries and translator modules
> ii  glusterfs-server   6.5-ubuntu1~xenial1  amd64
>   clustered file-system (server package)
> ```
> Op version has been upped to 6:
> ```
> $ sudo gluster volume get all cluster.op-version
> Option  Value
> --  -
> cluster.op-version  6
> 
> $ sudo gluster volume get all cluster.max-op-version
> Option  Value
> --  -
> cluster.max-op-version  6
> ```
> 
> Running `sudo gluster volume status all clients` reports that all
> clients are on op-version 6, too.
> 
> Any suggestions on what to look for or changes to try out?
> 
> Thanks,
> Riccardo
> 
> [1]: I have tried both the config where SAMBA 4.8 is using the
> vfs_glusterfs.so backend, and the one where `smbd` is just writing to
> a locally-mounted directory.  Doesn't seem to make a difference.

Samba v4.8 is an EOL ed version. Please consider updating Samba to at
least v4.9(rather v4.10) or higher.

Can you paste the output of `testparm -s` along with the output of
`gluster volume info ` ?

> [2]: Actually, since the servers are VMs on an OpenStack cloud, I
> created new virtual machines, installed GlusterFS 6 fresh, mounted
> the
> old bricks in the same brick locations,

How did you mount old bricks in the new location?

> and restarted the cluster.  I
> had to fiddle a bit with the files in `/var/lib/glusterfs` because
> the
> hostnames and IPs changed but did not do anything else than `sed -e
> s/old_hostname/new_hostname/` or similarly renaming files. In
> particular, I did not touch the extended attributes in the brick
> directory.
> 
> 
> Community Meeting Calendar:
> 
> APAC Schedule -
> Every 2nd and 4th Tuesday at 11:30 AM IST
> Bridge: https://bluejeans.com/118564314
> 
> NA/EMEA Schedule -
> Every 1st and 3rd Tuesday at 01:00 PM EDT
> Bridge: https://bluejeans.com/118564314
> 
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users



Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Performance drop when upgrading from 3.8 to 6.5

[Riccardo Murri]

Hello,

I recently upgraded[2] our servers from GlusterFS 3.8 (old GlusterFS
repo for Ubuntu 16.04) to 6.0 (gotten from the GlusterFS PPA for
Ubuntu 16.04 "xenial").

The sustained write performance nearly dropped to half it was before.
We copy a large (a few 10'000s) number of image files (each 2 to 10 MB
size) from the microscope where they were acquired to a SAMBA server
which mounts[1] the GlusterFS volume; before the upgrade, we could
write at about 40MB/s, after the upgrade, this dropped to 20MB/s.

This is the version of server and client software installed:
```
$ dpkg -l '*gluster*'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name   Version  Architecture
  Description
+++-==---==
ii  glusterfs-client   6.5-ubuntu1~xenial1  amd64
  clustered file-system (client package)
ii  glusterfs-common   6.5-ubuntu1~xenial1  amd64
  GlusterFS common libraries and translator modules
ii  glusterfs-server   6.5-ubuntu1~xenial1  amd64
  clustered file-system (server package)
```
Op version has been upped to 6:
```
$ sudo gluster volume get all cluster.op-version
Option  Value
--  -
cluster.op-version  6

$ sudo gluster volume get all cluster.max-op-version
Option  Value
--  -
cluster.max-op-version  6
```

Running `sudo gluster volume status all clients` reports that all
clients are on op-version 6, too.

Any suggestions on what to look for or changes to try out?

Thanks,
Riccardo

[1]: I have tried both the config where SAMBA 4.8 is using the
vfs_glusterfs.so backend, and the one where `smbd` is just writing to
a locally-mounted directory.  Doesn't seem to make a difference.

[2]: Actually, since the servers are VMs on an OpenStack cloud, I
created new virtual machines, installed GlusterFS 6 fresh, mounted the
old bricks in the same brick locations, and restarted the cluster.  I
had to fiddle a bit with the files in `/var/lib/glusterfs` because the
hostnames and IPs changed but did not do anything else than `sed -e
s/old_hostname/new_hostname/` or similarly renaming files. In
particular, I did not touch the extended attributes in the brick
directory.


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] performance - what can I expect

[Amar Tumballi Suryanarayan]

On Thu, May 2, 2019 at 1:21 PM Pascal Suter  wrote:

> Hi Amar
>
> thanks for rolling this back up. Actually i have done some more
> benchmarking and fiddled with the config to finally reach a performance
> figure i could live with. I now can squeeze about 3GB/s out of that server
> which seems to be close to what i can get out of its network uplink (using
> IP over Omni-Path). The system is now set up and in production so i can't
> run any benchmarks on it anymore but i will get back at benchmarking in the
> near future to test some storage related hardware, and i will try it with
> gluster on top again.
>
> embarassingly the biggest performance issue was that the default
> installation of the server was running the "performance" profile of tuned.
> once i switched it to "throughput-performance" performance increased
> dramatically.
>
> the volume info now looks pretty unspectacular:
>
> Volume Name: storage
> Type: Distribute
> Volume ID: c81c7e46-add5-4d88-9945-24cf7947ef8c
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 3
> Transport-type: tcp
> Bricks:
> Brick1: themis01:/data/brick1/brick
> Brick2: themis01:/data/brick2/brick
> Brick3: themis01:/data/brick3/brick
> Options Reconfigured:
> transport.address-family: inet
> nfs.disable: on
>
> thanks for pointing out gluster volume profile, i'll have a go with it
> during my next benchmarking session. so far i was using iostat to track
> brick-level io performance during my benchmarks.
>
> the main question i wanted to ask was, if there is a general rule of
> thumb, how much throughput of the original bare brick throughput would be
> expected to be left over once gluster is added on top of it. to give you an
> example: when I use a parallel filesystem like Lustre or BeeGFS i usually
> expect to get at least about 85% of the raw storage target throughput as
> aggregated bandwidth over a multi-node test out of my Lustre or BeeGFS
> setup. I consider any numbers below that to be too low and therefore will
> have to dig into performance tuning to find the bottle neck.
>
> i was hoping someone could give me a rule-of-thumb number for a simple
> distributed gluster setup, like that 85% number i've established for a
> parallel file system.
>
> so at the moment my takeaway is, in a simple distributed volume across 3
> bricks with an aggregated bandwidth of 6GB/s i can expect to get about
> 3GB/s aggregated bandwith out of the gluster mount, given there are no
> bottle necks in the network. the 3GB/s is a number conducted under ideal
> circumstances, meaning, i primed the storage to make sure i could run a
> benchmark run using three nodes, with each node running a single thread
> writing to a single file and each file was located on another bricke. this
> yielded the maximum perfomance as this was pure streaming IO without any
> overlapping file writing to the bricks other than the overhead created by
> gluster's own internal mechanisms.
>
> Interestingly, the performance didn't drop much when i added nodes and
> threads and introduced more random-ish io by having several processes write
> to the same brick. So I assume, what "eats" up the 50% performance in the
> end is probably Gluster writing all these additional hidden files which I
> assume is some sort of Metadata. This causes additional IO on the disk that
> i'm streaming my one file to and therefore turns my streaming IO into a
> random io load for the raid controller and underlying harddisks which on
> spinning disks would have about the performance impact i was seing in my
> benchmarks.
>

Thanks for all these details.

I have yet to try gluster on a Flash based brick and test its performance
> there.. i would expect to see a better "efficiency" than the 50% i've
> measured on this system here as random io vs. streaming io should not make
> such a difference (or acutally almost no difference at all) on a flash
> based storage. but that's  me guessing now.
>
> so for the moment i'm fine but i would still be interested in hearing
> ball-park figure "efficiency" numbers from others using gluster in a
> similar setup.
>

We couldn't get a single number on this yet. Mainly because of multiple
reasons.
* Gluster's volume type has different behavior (performance wise)
* Network plays more significant role than that of disk performance. Mostly
latency involved in n/w than the throughput.
* Different work loads (like create heavy Vs read/write, sequential
read/write Vs random read/write) needs different options (currently they
are not auto-tuned).
* If one has good n/w and disk speed, even back end filesystem
configuration (because of the layout we have with gfid etc) too matter a
bit.

Best thing is to understand the workload first, and then tuning for it (at
present).

cheers
>
> Pascal
> On 01.05.19 14:55, Amar Tumballi Suryanarayan wrote:
>
> Hi Pascal,
>
> Sorry for complete delay in this one. And thanks for testing out in
> different scenarios.  Few questions before others can have a look and

Re: [Gluster-users] performance - what can I expect

[Pascal Suter]

Hi Amar

thanks for rolling this back up. Actually i have done some more 
benchmarking and fiddled with the config to finally reach a performance 
figure i could live with. I now can squeeze about 3GB/s out of that 
server which seems to be close to what i can get out of its network 
uplink (using IP over Omni-Path). The system is now set up and in 
production so i can't run any benchmarks on it anymore but i will get 
back at benchmarking in the near future to test some storage related 
hardware, and i will try it with gluster on top again.


embarassingly the biggest performance issue was that the default 
installation of the server was running the "performance" profile of 
tuned. once i switched it to "throughput-performance" performance 
increased dramatically.


the volume info now looks pretty unspectacular:

Volume Name: storage
Type: Distribute
Volume ID: c81c7e46-add5-4d88-9945-24cf7947ef8c
Status: Started
Snapshot Count: 0
Number of Bricks: 3
Transport-type: tcp
Bricks:
Brick1: themis01:/data/brick1/brick
Brick2: themis01:/data/brick2/brick
Brick3: themis01:/data/brick3/brick
Options Reconfigured:
transport.address-family: inet
nfs.disable: on

thanks for pointing out gluster volume profile, i'll have a go with it 
during my next benchmarking session. so far i was using iostat to track 
brick-level io performance during my benchmarks.


the main question i wanted to ask was, if there is a general rule of 
thumb, how much throughput of the original bare brick throughput would 
be expected to be left over once gluster is added on top of it. to give 
you an example: when I use a parallel filesystem like Lustre or BeeGFS i 
usually expect to get at least about 85% of the raw storage target 
throughput as aggregated bandwidth over a multi-node test out of my 
Lustre or BeeGFS setup. I consider any numbers below that to be too low 
and therefore will have to dig into performance tuning to find the 
bottle neck.


i was hoping someone could give me a rule-of-thumb number for a simple 
distributed gluster setup, like that 85% number i've established for a 
parallel file system.


so at the moment my takeaway is, in a simple distributed volume across 3 
bricks with an aggregated bandwidth of 6GB/s i can expect to get about 
3GB/s aggregated bandwith out of the gluster mount, given there are no 
bottle necks in the network. the 3GB/s is a number conducted under ideal 
circumstances, meaning, i primed the storage to make sure i could run a 
benchmark run using three nodes, with each node running a single thread 
writing to a single file and each file was located on another bricke. 
this yielded the maximum perfomance as this was pure streaming IO 
without any overlapping file writing to the bricks other than the 
overhead created by gluster's own internal mechanisms.


Interestingly, the performance didn't drop much when i added nodes and 
threads and introduced more random-ish io by having several processes 
write to the same brick. So I assume, what "eats" up the 50% performance 
in the end is probably Gluster writing all these additional hidden files 
which I assume is some sort of Metadata. This causes additional IO on 
the disk that i'm streaming my one file to and therefore turns my 
streaming IO into a random io load for the raid controller and 
underlying harddisks which on spinning disks would have about the 
performance impact i was seing in my benchmarks.


I have yet to try gluster on a Flash based brick and test its 
performance there.. i would expect to see a better "efficiency" than the 
50% i've measured on this system here as random io vs. streaming io 
should not make such a difference (or acutally almost no difference at 
all) on a flash based storage. but that's  me guessing now.


so for the moment i'm fine but i would still be interested in hearing 
ball-park figure "efficiency" numbers from others using gluster in a 
similar setup.


cheers

Pascal

On 01.05.19 14:55, Amar Tumballi Suryanarayan wrote:

Hi Pascal,

Sorry for complete delay in this one. And thanks for testing out in 
different scenarios.  Few questions before others can have a look and 
advice you.


1. What is the volume info output ?

2. Do you see any concerning logs in glusterfs log files?

3. Please use `gluster volume profile` while running the tests, and 
that gives a lot of information.


4. Considering you are using glusterfs-6.0, please take statedump of 
client process (on any node) before and after the test, so we can 
analyze the latency information of each translators.


With these information, I hope we will be in a better state to answer 
the questions.



On Wed, Apr 10, 2019 at 3:45 PM Pascal Suter > wrote:


i continued my testing with 5 clients, all attached over 100Gbit/s
omni-path via IP over IB. when i run the same iozone benchmark across
all 5 clients where gluster is mounted using the glusterfs client,
i get
an aggretated write throughput of 

Re: [Gluster-users] performance - what can I expect

[Amar Tumballi Suryanarayan]

Hi Pascal,

Sorry for complete delay in this one. And thanks for testing out in
different scenarios.  Few questions before others can have a look and
advice you.

1. What is the volume info output ?

2. Do you see any concerning logs in glusterfs log files?

3. Please use `gluster volume profile` while running the tests, and that
gives a lot of information.

4. Considering you are using glusterfs-6.0, please take statedump of client
process (on any node) before and after the test, so we can analyze the
latency information of each translators.

With these information, I hope we will be in a better state to answer the
questions.


On Wed, Apr 10, 2019 at 3:45 PM Pascal Suter  wrote:

> i continued my testing with 5 clients, all attached over 100Gbit/s
> omni-path via IP over IB. when i run the same iozone benchmark across
> all 5 clients where gluster is mounted using the glusterfs client, i get
> an aggretated write throughput of only about 400GB/s and an aggregated
> read throughput of 1.5GB/s. Each node was writing a single 200Gb file in
> 16MB chunks and the files where distributed across all three bricks on
> the server.
>
> the connection was established over Omnipath for sure, as there is no
> other link between the nodes and server.
>
> i have no clue what i'm doing wrong here. i can't believe that this is a
> normal performance people would expect to see from gluster. i guess
> nobody would be using it if it was this slow.
>
> again, when written dreictly to the xfs filesystem on the bricks, i get
> over 6GB/s read and write throughput using the same benchmark.
>
> any advise is appreciated
>
> cheers
>
> Pascal
>
> On 04.04.19 12:03, Pascal Suter wrote:
> > I just noticed i left the most important parameters out :)
> >
> > here's the write command with filesize and recordsize in it as well :)
> >
> > ./iozone -i 0 -t 1 -F /mnt/gluster/storage/thread1 -+n -c -C -e -I -w
> > -+S 0 -s 200G -r 16384k
> >
> > also i ran the benchmark without direct_io which resulted in an even
> > worse performance.
> >
> > i also tried to mount the gluster volume via nfs-ganesha which further
> > reduced throughput down to about 450MB/s
> >
> > if i run the iozone benchmark with 3 threads writing to all three
> > bricks directly (from the xfs filesystem) i get throughputs of around
> > 6GB/s .. if I run the same benchmark through gluster mounted locally
> > using the fuse client and with enough threads so that each brick gets
> > at least one file written to it, i end up seing throughputs around
> > 1.5GB/s .. that's a 4x decrease in performance. at it actually is the
> > same if i run the benchmark with less threads and files only get
> > written to two out of three bricks.
> >
> > cpu load on the server is around 25% by the way, nicely distributed
> > across all available cores.
> >
> > i can't believe that gluster should really be so slow and everybody is
> > just happily using it. any hints on what i'm doing wrong are very
> > welcome.
> >
> > i'm using gluster 6.0 by the way.
> >
> > regards
> >
> > Pascal
> >
> > On 03.04.19 12:28, Pascal Suter wrote:
> >> Hi all
> >>
> >> I am currently testing gluster on a single server. I have three
> >> bricks, each a hardware RAID6 volume with thin provisioned LVM that
> >> was aligned to the RAID and then formatted with xfs.
> >>
> >> i've created a distributed volume so that entire files get
> >> distributed across my three bricks.
> >>
> >> first I ran a iozone benchmark across each brick testing the read and
> >> write perofrmance of a single large file per brick
> >>
> >> i then mounted my gluster volume locally and ran another iozone run
> >> with the same parameters writing a single file. the file went to
> >> brick 1 which, when used driectly, would write with 2.3GB/s and read
> >> with 1.5GB/s. however, through gluster i got only 800MB/s read and
> >> 750MB/s write throughput
> >>
> >> another run with two processes each writing a file, where one file
> >> went to the first brick and the other file to the second brick (which
> >> by itself when directly accessed wrote at 2.8GB/s and read at
> >> 2.7GB/s) resulted in 1.2GB/s of aggregated write and also aggregated
> >> read throughput.
> >>
> >> Is this a normal performance i can expect out of a glusterfs or is it
> >> worth tuning in order to really get closer to the actual brick
> >> filesystem performance?
> >>
> >> here are the iozone commands i use for writing and reading.. note
> >> that i am using directIO in order to make sure i don't get fooled by
> >> cache :)
> >>
> >> ./iozone -i 0 -t 1 -F /mnt/brick${b}/thread1 -+n -c -C -e -I -w -+S 0
> >> -s $filesize -r $recordsize > iozone-brick${b}-write.txt
> >>
> >> ./iozone -i 1 -t 1 -F /mnt/brick${b}/thread1 -+n -c -C -e -I -w -+S 0
> >> -s $filesize -r $recordsize > iozone-brick${b}-read.txt
> >>
> >> cheers
> >>
> >> Pascal
> >>
> >> ___
> >> Gluster-users mailing list
> >> Gluster-users@gluster.org
> >> 

Re: [Gluster-users] performance - what can I expect

[Pascal Suter]

i continued my testing with 5 clients, all attached over 100Gbit/s 
omni-path via IP over IB. when i run the same iozone benchmark across 
all 5 clients where gluster is mounted using the glusterfs client, i get 
an aggretated write throughput of only about 400GB/s and an aggregated 
read throughput of 1.5GB/s. Each node was writing a single 200Gb file in 
16MB chunks and the files where distributed across all three bricks on 
the server.


the connection was established over Omnipath for sure, as there is no 
other link between the nodes and server.


i have no clue what i'm doing wrong here. i can't believe that this is a 
normal performance people would expect to see from gluster. i guess 
nobody would be using it if it was this slow.


again, when written dreictly to the xfs filesystem on the bricks, i get 
over 6GB/s read and write throughput using the same benchmark.


any advise is appreciated

cheers

Pascal

On 04.04.19 12:03, Pascal Suter wrote:

I just noticed i left the most important parameters out :)

here's the write command with filesize and recordsize in it as well :)

./iozone -i 0 -t 1 -F /mnt/gluster/storage/thread1 -+n -c -C -e -I -w 
-+S 0 -s 200G -r 16384k


also i ran the benchmark without direct_io which resulted in an even 
worse performance.


i also tried to mount the gluster volume via nfs-ganesha which further 
reduced throughput down to about 450MB/s


if i run the iozone benchmark with 3 threads writing to all three 
bricks directly (from the xfs filesystem) i get throughputs of around 
6GB/s .. if I run the same benchmark through gluster mounted locally 
using the fuse client and with enough threads so that each brick gets 
at least one file written to it, i end up seing throughputs around 
1.5GB/s .. that's a 4x decrease in performance. at it actually is the 
same if i run the benchmark with less threads and files only get 
written to two out of three bricks.


cpu load on the server is around 25% by the way, nicely distributed 
across all available cores.


i can't believe that gluster should really be so slow and everybody is 
just happily using it. any hints on what i'm doing wrong are very 
welcome.


i'm using gluster 6.0 by the way.

regards

Pascal

On 03.04.19 12:28, Pascal Suter wrote:

Hi all

I am currently testing gluster on a single server. I have three 
bricks, each a hardware RAID6 volume with thin provisioned LVM that 
was aligned to the RAID and then formatted with xfs.


i've created a distributed volume so that entire files get 
distributed across my three bricks.


first I ran a iozone benchmark across each brick testing the read and 
write perofrmance of a single large file per brick


i then mounted my gluster volume locally and ran another iozone run 
with the same parameters writing a single file. the file went to 
brick 1 which, when used driectly, would write with 2.3GB/s and read 
with 1.5GB/s. however, through gluster i got only 800MB/s read and 
750MB/s write throughput


another run with two processes each writing a file, where one file 
went to the first brick and the other file to the second brick (which 
by itself when directly accessed wrote at 2.8GB/s and read at 
2.7GB/s) resulted in 1.2GB/s of aggregated write and also aggregated 
read throughput.


Is this a normal performance i can expect out of a glusterfs or is it 
worth tuning in order to really get closer to the actual brick 
filesystem performance?


here are the iozone commands i use for writing and reading.. note 
that i am using directIO in order to make sure i don't get fooled by 
cache :)


./iozone -i 0 -t 1 -F /mnt/brick${b}/thread1 -+n -c -C -e -I -w -+S 0 
-s $filesize -r $recordsize > iozone-brick${b}-write.txt


./iozone -i 1 -t 1 -F /mnt/brick${b}/thread1 -+n -c -C -e -I -w -+S 0 
-s $filesize -r $recordsize > iozone-brick${b}-read.txt


cheers

Pascal

___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] performance - what can I expect

[Pascal Suter]

I just noticed i left the most important parameters out :)

here's the write command with filesize and recordsize in it as well :)

./iozone -i 0 -t 1 -F /mnt/gluster/storage/thread1 -+n -c -C -e -I -w 
-+S 0 -s 200G -r 16384k


also i ran the benchmark without direct_io which resulted in an even 
worse performance.


i also tried to mount the gluster volume via nfs-ganesha which further 
reduced throughput down to about 450MB/s


if i run the iozone benchmark with 3 threads writing to all three bricks 
directly (from the xfs filesystem) i get throughputs of around 6GB/s .. 
if I run the same benchmark through gluster mounted locally using the 
fuse client and with enough threads so that each brick gets at least one 
file written to it, i end up seing throughputs around 1.5GB/s .. that's 
a 4x decrease in performance. at it actually is the same if i run the 
benchmark with less threads and files only get written to two out of 
three bricks.


cpu load on the server is around 25% by the way, nicely distributed 
across all available cores.


i can't believe that gluster should really be so slow and everybody is 
just happily using it. any hints on what i'm doing wrong are very welcome.


i'm using gluster 6.0 by the way.

regards

Pascal

On 03.04.19 12:28, Pascal Suter wrote:

Hi all

I am currently testing gluster on a single server. I have three 
bricks, each a hardware RAID6 volume with thin provisioned LVM that 
was aligned to the RAID and then formatted with xfs.


i've created a distributed volume so that entire files get distributed 
across my three bricks.


first I ran a iozone benchmark across each brick testing the read and 
write perofrmance of a single large file per brick


i then mounted my gluster volume locally and ran another iozone run 
with the same parameters writing a single file. the file went to brick 
1 which, when used driectly, would write with 2.3GB/s and read with 
1.5GB/s. however, through gluster i got only 800MB/s read and 750MB/s 
write throughput


another run with two processes each writing a file, where one file 
went to the first brick and the other file to the second brick (which 
by itself when directly accessed wrote at 2.8GB/s and read at 2.7GB/s) 
resulted in 1.2GB/s of aggregated write and also aggregated read 
throughput.


Is this a normal performance i can expect out of a glusterfs or is it 
worth tuning in order to really get closer to the actual brick 
filesystem performance?


here are the iozone commands i use for writing and reading.. note that 
i am using directIO in order to make sure i don't get fooled by cache :)


./iozone -i 0 -t 1 -F /mnt/brick${b}/thread1 -+n -c -C -e -I -w -+S 0 
-s $filesize -r $recordsize > iozone-brick${b}-write.txt


./iozone -i 1 -t 1 -F /mnt/brick${b}/thread1 -+n -c -C -e -I -w -+S 0 
-s $filesize -r $recordsize > iozone-brick${b}-read.txt


cheers

Pascal

___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] performance - what can I expect

[Pascal Suter]

Hi all

I am currently testing gluster on a single server. I have three bricks, 
each a hardware RAID6 volume with thin provisioned LVM that was aligned 
to the RAID and then formatted with xfs.


i've created a distributed volume so that entire files get distributed 
across my three bricks.


first I ran a iozone benchmark across each brick testing the read and 
write perofrmance of a single large file per brick


i then mounted my gluster volume locally and ran another iozone run with 
the same parameters writing a single file. the file went to brick 1 
which, when used driectly, would write with 2.3GB/s and read with 
1.5GB/s. however, through gluster i got only 800MB/s read and 750MB/s 
write throughput


another run with two processes each writing a file, where one file went 
to the first brick and the other file to the second brick (which by 
itself when directly accessed wrote at 2.8GB/s and read at 2.7GB/s) 
resulted in 1.2GB/s of aggregated write and also aggregated read 
throughput.


Is this a normal performance i can expect out of a glusterfs or is it 
worth tuning in order to really get closer to the actual brick 
filesystem performance?


here are the iozone commands i use for writing and reading.. note that i 
am using directIO in order to make sure i don't get fooled by cache :)


./iozone -i 0 -t 1 -F /mnt/brick${b}/thread1 -+n -c -C -e -I -w -+S 0 -s 
$filesize -r $recordsize > iozone-brick${b}-write.txt


./iozone -i 1 -t 1 -F /mnt/brick${b}/thread1 -+n -c -C -e -I -w -+S 0 -s 
$filesize -r $recordsize > iozone-brick${b}-read.txt


cheers

Pascal

___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Performance issue, need guidance

[Strahil]

I have just checked the archive and it seems that the diagram is missing, so I'm adding an URL link to it:https://drive.google.com/file/d/1SiW21ASPXHRAEuE_jZ50R3FoO-NcnFqT/view?usp=sharingMy version is 3.12.15Best Regards,Strahil Nikolov___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Performance question: Replicated with RAID0, or Distributed with RAID5?

[Ernie Dunbar]

Hi everyone. I have a question about performance, hoping that perhaps 
someone has already tested these scenarios so that I don't have to.


In order to maximize a Gluster array's performance, which is faster: 
Gluster servers with 6 SAS disks each set up in a RAID0 configuration, 
letting Gluster handle the data duplication, or letting the servers 
handle data integrity with some kind of striped RAID configuration, and 
setting Gluster up in a Distributed configuration?


My hypothesis is that the latter is actually faster because data 
integrity is handled in hardware, especially if the load is distributed 
over several Gluster servers. Half of our use involves large VM files, 
and the other half is in e-mail, with lots of random writes to small 
files. So far, we've got 4 servers to dedicate to this task, although 
under the right circumstances, that can be changed.


___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Performance drop from 3.8 to 3.10

[WK]

Maybe next week we can all explore this.

I'm on 3.10.5 and I don't have any complaints. Actually we are quite 
happy with the new clusters,  but these were green field installations 
that were built and then replaced our old 3.4 stuff.


So we are still really enjoying the sharding and arbiter improvements.

I therefore don't have any baseline stats to compare any performance diffs.

I'm curious as to what changed in 3.10 that would account for any change 
in performance from 3.8 and in a similar vein what changes to expect in 
3.12.x as we are thinking about making that jump soon.


-wk



On 9/22/2017 8:07 AM, Lindsay Mathieson wrote:


On 22/09/2017 1:21 PM, Krutika Dhananjay wrote:

Could you disable cluster.eager-lock and try again?



Thanks, but didn't seem to make any difference.


Can't test anymore at the moment as down a server that hung on reboot :(




___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Performance drop from 3.8 to 3.10

[Lindsay Mathieson]

On 22/09/2017 1:21 PM, Krutika Dhananjay wrote:

Could you disable cluster.eager-lock and try again?



Thanks, but didn't seem to make any difference.


Can't test anymore at the moment as down a server that hung on reboot :(


--
Lindsay Mathieson

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Performance drop from 3.8 to 3.10

[Krutika Dhananjay]

Could you disable cluster.eager-lock and try again?

-Krutika

On Thu, Sep 21, 2017 at 6:31 PM, Lindsay Mathieson <
lindsay.mathie...@gmail.com> wrote:

> Upgraded recently from 3.8.15 to 3.10.5 and have seen a fairly substantial
> drop in read/write perfomance
>
> env:
>
> - 3 node, replica 3 cluster
>
> - Private dedicated Network: 1Gx3, bond: balance-alb
>
> - was able to down the volume for the upgrade and reboot each node
>
> - Usage: VM Hosting (qemu)
>
> - Sharded Volume
>
> - sequential read performance in VM's has dropped from 700Mbps to 300mbs
>
> - Seq Write has dropped from 115MB/s (approx) to 110
>
> - Write IOPS have dropped from 12MB/s to 8MB/s
>
> Apart from increasing the op version I made no changes to the volume
> settings.
>
> op.version is 31004
>
> gluster v info
>
> Volume Name: datastore4
> Type: Replicate
> Volume ID: 0ba131ef-311d-4bb1-be46-596e83b2f6ce
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x 3 = 3
> Transport-type: tcp
> Bricks:
> Brick1: vnb.proxmox.softlog:/tank/vmdata/datastore4
> Brick2: vng.proxmox.softlog:/tank/vmdata/datastore4
> Brick3: vnh.proxmox.softlog:/tank/vmdata/datastore4
> Options Reconfigured:
> transport.address-family: inet
> cluster.locking-scheme: granular
> cluster.granular-entry-heal: yes
> features.shard-block-size: 64MB
> network.remote-dio: enable
> cluster.eager-lock: enable
> performance.io-cache: off
> performance.read-ahead: off
> performance.quick-read: off
> performance.stat-prefetch: on
> performance.strict-write-ordering: off
> nfs.enable-ino32: off
> nfs.addr-namelookup: off
> nfs.disable: on
> cluster.server-quorum-type: server
> cluster.quorum-type: auto
> features.shard: on
> cluster.data-self-heal: on
> performance.readdir-ahead: on
> performance.low-prio-threads: 32
> user.cifs: off
> performance.flush-behind: on
> server.event-threads: 4
> client.event-threads: 4
> server.allow-insecure: on
>
>
> --
> Lindsay Mathieson
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Performance drop from 3.8 to 3.10

[Lindsay Mathieson]

Upgraded recently from 3.8.15 to 3.10.5 and have seen a fairly 
substantial drop in read/write perfomance


env:

- 3 node, replica 3 cluster

- Private dedicated Network: 1Gx3, bond: balance-alb

- was able to down the volume for the upgrade and reboot each node

- Usage: VM Hosting (qemu)

- Sharded Volume

- sequential read performance in VM's has dropped from 700Mbps to 300mbs

- Seq Write has dropped from 115MB/s (approx) to 110

- Write IOPS have dropped from 12MB/s to 8MB/s

Apart from increasing the op version I made no changes to the volume 
settings.


op.version is 31004

gluster v info

Volume Name: datastore4
Type: Replicate
Volume ID: 0ba131ef-311d-4bb1-be46-596e83b2f6ce
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: vnb.proxmox.softlog:/tank/vmdata/datastore4
Brick2: vng.proxmox.softlog:/tank/vmdata/datastore4
Brick3: vnh.proxmox.softlog:/tank/vmdata/datastore4
Options Reconfigured:
transport.address-family: inet
cluster.locking-scheme: granular
cluster.granular-entry-heal: yes
features.shard-block-size: 64MB
network.remote-dio: enable
cluster.eager-lock: enable
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
performance.stat-prefetch: on
performance.strict-write-ordering: off
nfs.enable-ino32: off
nfs.addr-namelookup: off
nfs.disable: on
cluster.server-quorum-type: server
cluster.quorum-type: auto
features.shard: on
cluster.data-self-heal: on
performance.readdir-ahead: on
performance.low-prio-threads: 32
user.cifs: off
performance.flush-behind: on
server.event-threads: 4
client.event-threads: 4
server.allow-insecure: on


--
Lindsay Mathieson

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Performance testing with sysbench...

[Krist van Besien]

Hi all,

I'm doing some performance test...

If I test a simple sequential write using dd I get a thorughput of about
550 Mb/s. When I do a sequential write test using sysbench this drops to
about 200. Is this due to the way sysbench tests? Or has in this case the
performance of sysbench itself become the bottleneck?

Krist


-- 
Vriendelijke Groet |  Best Regards | Freundliche Grüße | Cordialement
--

Krist van Besien

senior architect, RHCE, RHCSA Open Stack

Red Hat Red Hat Switzerland S.A. 

kr...@redhat.comM: +41-79-5936260

TRIED. TESTED. TRUSTED. 
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Performance Translators Documentation

[Christopher Schmidt]

Hi,

I had some issues (org.apache.lucene.index.CorruptIndexException) with
Lucene (resp. ElasticSearch) working on a GlusterFS Volume and Kubernetes.

For testing I switched off all performance translators...

And I wonder if there is somewhere documentation, who they are and what
they are doing?

best Chris
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Performance testing

[Krist van Besien]

Hi All,

I build a Gluster 3.8.4 (RHGS 3.2) cluster for a customer, and I am having
some issue demonstrating that it performs well.

The customer compares it with his old NFS based NAS, and runs FIO to test
workloads.

What I notice is that FIO throughtput is only +-20Mb/s, which is not a lot.
When I do a simple test with dd I easily get 600Mb/s throughput.
In the fio job file the option "direct=1" is used, which bypasses caching.
If we run a fio job with direct=0 the performance goes up a lot, and is
near 600Mb/s as well.

The customer insists that on his old system (that Gluster should replace)
he could get 600Mb/s throughput with fio, with the setting direct=1. and
that he was rather underwhelmed by the performance of Gluster here.

What I need is answers to either:
- Have I overlooked something? I have not really done much tuning yet. Is
there some obvious paremeter I overlooked that could change the results of
a fio performance test?

or:

- Is testing with "direct=1" not really a way to test Gluster, as the cache
is a rather important part of what is needed to make gluster perform?

-- 
Vriendelijke Groet |  Best Regards | Freundliche Grüße | Cordialement
--
Krist van Besien | Senior Architect | Red Hat EMEA Cloud Practice | RHCE |
RHCSA Open Stack
@: kr...@redhat.com | M: +41-79-5936260
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Performance optimization

[Gandalf Corvotempesta]

Workload: VM hosting with sharding enabled, replica 3 (with or without
distribution, see below)

Which configuration will perform better:

a) 1 ZFS disk per brick, 1 brick per server. 1 disk for each server.
b) 1 ZFS mirror per brick, 1 brick per server. 1 disk for each server.
c) 1 ZFS disk per brick, 2 bricks per server (with distribution). 4
disks on each server.
d) 1 ZFS mirror per brick, 2 bricks per server (with distribution) 4
disks on each server.
e) any other combination of "c" or "d" by increasing the number of
disks per server.

I think that "distribution" ('c' or 'd') would give me better
performance, as in both reads and writes, multiple disks are involved,
as long Gluster tries to always use the maximum number of bricks,
during distribution.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Performance testing striped 4 volume

[Cedric Lemarchand]

It could be some extended attributes that still exists on folders brick{1.4}, 
you could either remove them with attr or simply remove/recreate them.

Cheers,


> On 5 Jan 2017, at 01:23, Zack Boll  wrote:
> 
> In performance testing a striped 4 volume, I appeared to have crashed 
> glusterfs using version 3.8.7 on Ubuntu 16.04.  I then stopped the volume and 
> deleted it.  I am now having trouble creating a new volume, below is output
> 
> sudo gluster volume create gluster1 transport tcp cyan:/gluster/ssd1/brick1 
> green:/gluster/ssd1/brick2 red:/gluster/ssd1/brick3 pink:/gluster/ssd1/brick4
> 
> volume create: gluster1: failed: Staging failed on green. Error: 
> /gluster/ssd1/brick2 is already part of a volume
> Staging failed on pink. Error: /gluster/ssd1/brick4 is already part of a 
> volume
> Staging failed on cyan. Error: /gluster/ssd1/brick1 is already part of a 
> volume
> Staging failed on red. Error: /gluster/ssd1/brick3 is already part of a volume
> 
> sudo gluster volume info
> No volumes present
> 
> Any ideas on how to fix this?
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Performance testing striped 4 volume

[Karan Sandha]

Hi Zack,

As the bricks had already been used before, gluster doesn't allow to 
create volume with same brick path until you use "force" at the end of 
the command. As you are doing performance testing i would recommend to 
clean the bricks and  issue the same command.


. sudo gluster volume create gluster1 transport tcp 
cyan:/gluster/ssd1/brick1*new* green:/gluster/ssd1/brick2*new* 
red:/gluster/ssd1/brick3*new* pink:/gluster/ssd1/brick4*new

*

for time being this will solve the your problem.


Thanks & Regards

Karan Sandha


On 01/05/2017 05:53 AM, Zack Boll wrote:
In performance testing a striped 4 volume, I appeared to have crashed 
glusterfs using version 3.8.7 on Ubuntu 16.04.  I then stopped the 
volume and deleted it.  I am now having trouble creating a new volume, 
below is output


sudo gluster volume create gluster1 transport tcp 
cyan:/gluster/ssd1/brick1 green:/gluster/ssd1/brick2 
red:/gluster/ssd1/brick3 pink:/gluster/ssd1/brick4


volume create: gluster1: failed: Staging failed on green. Error: 
/gluster/ssd1/brick2 is already part of a volume
Staging failed on pink. Error: /gluster/ssd1/brick4 is already part of 
a volume
Staging failed on cyan. Error: /gluster/ssd1/brick1 is already part of 
a volume
Staging failed on red. Error: /gluster/ssd1/brick3 is already part of 
a volume


sudo gluster volume info
No volumes present

Any ideas on how to fix this?


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Performance testing striped 4 volume

[Zack Boll]

In performance testing a striped 4 volume, I appeared to have crashed
glusterfs using version 3.8.7 on Ubuntu 16.04.  I then stopped the volume
and deleted it.  I am now having trouble creating a new volume, below is
output

sudo gluster volume create gluster1 transport tcp cyan:/gluster/ssd1/brick1
green:/gluster/ssd1/brick2 red:/gluster/ssd1/brick3
pink:/gluster/ssd1/brick4

volume create: gluster1: failed: Staging failed on green. Error:
/gluster/ssd1/brick2 is already part of a volume
Staging failed on pink. Error: /gluster/ssd1/brick4 is already part of a
volume
Staging failed on cyan. Error: /gluster/ssd1/brick1 is already part of a
volume
Staging failed on red. Error: /gluster/ssd1/brick3 is already part of a
volume

sudo gluster volume info
No volumes present

Any ideas on how to fix this?
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Performance

[Lindsay Mathieson]

On 26/10/2016 2:50 AM, Service Mail wrote:
3x zfs raidz2 servers with a single gluster 3.8 replicated volume 
across a 10G network




SSD slog? they make a big difference for sync writes


If you have no slog try with zfs "sync=disabled" on all three pools. Not 
recommended for  production, but would like to see the results.


--
Lindsay Mathieson

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Performance

[Alex Crow]

I last used GlusterFS around early 3.6. I could get great results for streaming 
large files. I was seeing up to 700MB/s with a DD test. Small file/metadata 
access wasn't right for our use, but for VMs it should be fine with a bit of 
tuning.

On 31 October 2016 15:33:06 GMT+00:00, Joe Julian  wrote:
>On 10/31/2016 08:29 AM, Alastair Neil wrote:
>> What version of Gluster?  Are you using glusterfs or nfs mount?  Any 
>> other traffic on the network, is the cluster quiescent apart from
>your 
>> dd test?
>>
>
>What type of volume?
>
>> It does seem slow.  I have a three server cluster, using straight xfs
>
>> over 10G with Gluster 3.8 and glusterfs mounts and I see:
>>
>> [root@sb-c 192.168.10.49:VM]# sync; dd if=/dev/zero of=nfsp2 bs=1M 
>> count=1024; sync
>> 1024+0 records in
>> 1024+0 records out
>> 1073741824 bytes (1.1 GB) copied, 11.3322 s, 94.8 MB/s
>> [root@sb-c 192.168.10.49:VM]# sync; dd if=/dev/zero of=nfsp2 bs=1M 
>> count=10240; sync
>> 10240+0 records in
>> 10240+0 records out
>> 10737418240 bytes (11 GB) copied, 117.854 s, 91.1 MB/s
>>
>> this is on a cluster serving 5 ovirt nodes and about 60 running VMs.
>>
>>
>>
>> On 25 October 2016 at 12:50, Service Mail > > wrote:
>>
>> Hello,
>>
>> I have the following setup:
>>
>> 3x zfs raidz2 servers with a single gluster 3.8 replicated volume
>> across a 10G network
>>
>> Everything is working fine however performance looks very poor to
>me:
>>
>>
>> root@Client:/test_mount# sync; dd if=/dev/zero of=nfsp2 bs=1M
>> count=1024; sync
>> 1024+0 records in
>> 1024+0 records out
>> 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 32.1786 s, 33.4 MB/s
>>
>> root@Client:/test_mount# sync; dd if=/dev/zero of=nfsp2 bs=1M
>> count=10240; sync
>> 10240+0 records in
>> 10240+0 records out
>> 10737418240 bytes (11 GB, 10 GiB) copied, 301.563 s, 35.6 MB/s
>>
>> Are those reading normal? Where should I look to increase
>performance?
>>
>> Thanks,
>>
>> Ciclope
>>
>>
>>
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org 
>> http://www.gluster.org/mailman/listinfo/gluster-users
>> 
>>
>>
>>
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
>
>
>
>
>
>___
>Gluster-users mailing list
>Gluster-users@gluster.org
>http://www.gluster.org/mailman/listinfo/gluster-users

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.
--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300.
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Performance

[Joe Julian]

On 10/31/2016 08:29 AM, Alastair Neil wrote:
What version of Gluster?  Are you using glusterfs or nfs mount?  Any 
other traffic on the network, is the cluster quiescent apart from your 
dd test?




What type of volume?

It does seem slow.  I have a three server cluster, using straight xfs 
over 10G with Gluster 3.8 and glusterfs mounts and I see:


[root@sb-c 192.168.10.49:VM]# sync; dd if=/dev/zero of=nfsp2 bs=1M 
count=1024; sync

1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 11.3322 s, 94.8 MB/s
[root@sb-c 192.168.10.49:VM]# sync; dd if=/dev/zero of=nfsp2 bs=1M 
count=10240; sync

10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 117.854 s, 91.1 MB/s

this is on a cluster serving 5 ovirt nodes and about 60 running VMs.



On 25 October 2016 at 12:50, Service Mail > wrote:


Hello,

I have the following setup:

3x zfs raidz2 servers with a single gluster 3.8 replicated volume
across a 10G network

Everything is working fine however performance looks very poor to me:


root@Client:/test_mount# sync; dd if=/dev/zero of=nfsp2 bs=1M
count=1024; sync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 32.1786 s, 33.4 MB/s

root@Client:/test_mount# sync; dd if=/dev/zero of=nfsp2 bs=1M
count=10240; sync
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 301.563 s, 35.6 MB/s

Are those reading normal? Where should I look to increase performance?

Thanks,

Ciclope




___
Gluster-users mailing list
Gluster-users@gluster.org 
http://www.gluster.org/mailman/listinfo/gluster-users





___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Performance

[Alastair Neil]

What version of Gluster?  Are you using glusterfs or nfs mount?  Any other
traffic on the network, is the cluster quiescent apart from your dd test?

It does seem slow.  I have a three server cluster, using straight xfs over
10G with Gluster 3.8 and glusterfs mounts and I see:

[root@sb-c 192.168.10.49:VM]# sync; dd if=/dev/zero of=nfsp2 bs=1M
count=1024; sync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 11.3322 s, 94.8 MB/s
[root@sb-c 192.168.10.49:VM]# sync; dd if=/dev/zero of=nfsp2 bs=1M
count=10240; sync
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 117.854 s, 91.1 MB/s

this is on a cluster serving 5 ovirt nodes and about 60 running VMs.



On 25 October 2016 at 12:50, Service Mail  wrote:

> Hello,
>
> I have the following setup:
>
> 3x zfs raidz2 servers with a single gluster 3.8 replicated volume across a
> 10G network
>
> Everything is working fine however performance looks very poor to me:
>
>
> root@Client:/test_mount# sync; dd if=/dev/zero of=nfsp2 bs=1M count=1024;
> sync
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 32.1786 s, 33.4 MB/s
>
> root@Client:/test_mount# sync; dd if=/dev/zero of=nfsp2 bs=1M
> count=10240; sync
> 10240+0 records in
> 10240+0 records out
> 10737418240 bytes (11 GB, 10 GiB) copied, 301.563 s, 35.6 MB/s
>
> Are those reading normal? Where should I look to increase performance?
>
> Thanks,
>
> Ciclope
>
>
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Performance

[Service Mail]

Hello,

I have the following setup:

3x zfs raidz2 servers with a single gluster 3.8 replicated volume across a
10G network

Everything is working fine however performance looks very poor to me:


root@Client:/test_mount# sync; dd if=/dev/zero of=nfsp2 bs=1M count=1024;
sync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 32.1786 s, 33.4 MB/s

root@Client:/test_mount# sync; dd if=/dev/zero of=nfsp2 bs=1M count=10240;
sync
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 301.563 s, 35.6 MB/s

Are those reading normal? Where should I look to increase performance?

Thanks,

Ciclope
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Performance gap between clients

[Pavel Szalbot]

Hello everybody,

I am experiencing peculiar performance difference on my client nodes.

One node is blank Ubuntu (Xenial), second is also Xenial with a web server
(nginx) that serves media files stored on disk image that is on Gluster
volume.
Both clients are 3.8.5, 10Gbe NICs used for Gluster network, 32 GB RAM on
both.
Gluster servers are 64GB RAM, 6 SSDs on each.
Switching is done on Juniper EX4550, load is very low, MTU 9000 (almost no
difference to 1500).

I get about 300MB/s on the node with nginx and only 160MB/s on the second
one.

[global]
filename=/mnt/gluster_vol/fio
ioengine=libaio
direct=1
bs=256k
rw=read
iodepth=1
numjobs=1
size=8192m

I did check packet drops, if 10Gbe is actually used, vmstat for iowait,
traffic distribution on server nodes, sysctl -a diff, iperf between
clients, servers, client-server, and probably a dozen of other things. I
tried to install nginx on the "blank" client with but it did not make any
difference. Slower node actually has more RAM available.

Do you have any ideas what could cause this?

Volume options:
cluster.self-heal-daemon: enable
nfs.disable: on
performance.readdir-ahead: on
performance.cache-size: 1GB
performance.client-io-threads: on
performance.io-thread-count: 64
performance.read-ahead: off

-ps
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] performance issue in gluster volume

[Anuradha Talur]

- Original Message -
> From: "Ramavtar" <ramavtar.rath...@everdata.com>
> To: gluster-users@gluster.org
> Sent: Friday, May 20, 2016 11:12:43 PM
> Subject: [Gluster-users] performance issue in gluster volume
> 
> Hi Ravi,
> 
> I am using gluster volume and it's having 2.7 TB data ( mp4 and jpeg
> files)  with nginx webserver
> 
> I am facing performance related issue with gluster volume please help me
Hi,

Could you elaborate what kind of performance issue you are talking about?

> please find  the gluster details :
> 
> 
> [root@webnode3 ~]# gluster --version
> glusterfs 3.7.11 built on Apr 27 2016 14:09:22
> Repository revision: git://git.gluster.com/glusterfs.git
> Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
> GlusterFS comes with ABSOLUTELY NO WARRANTY.
> You may redistribute copies of GlusterFS under the terms of the GNU
> General Publ
> ic License.
> 
> 
> [root@webnode3 ~]# gluster volume info
> 
> Volume Name: DATA-STORE
> Type: Distributed-Replicate
> Volume ID: b64c1fea-1500-4014-b36a-0487818fa893
> Status: Started
> Number of Bricks: 2 x 2 = 4
> Transport-type: tcp
> Bricks:
> Brick1: webhost75:/home/DATABRINK
> Brick2: webhost90:/home/DATABRINK
> Brick3: mysqlhost1:/home/DATABRINK
> Brick4: mysqlhost2:/home/DATABRINK
> Options Reconfigured:
> performance.readdir-ahead: on
> 
> I mounted this volume on same server with fuse.
> 
> please help me.
> 
> Thanks,
> Ram
> 
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
> 

-- 
Thanks,
Anuradha.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] performance issue in gluster volume

[Ramavtar]

Hi Ravi,

I am using gluster volume and it's having 2.7 TB data ( mp4 and jpeg 
files)  with nginx webserver


I am facing performance related issue with gluster volume please help me 
please find  the gluster details :



[root@webnode3 ~]# gluster --version
glusterfs 3.7.11 built on Apr 27 2016 14:09:22
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. 
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU 
General Publ

ic License.


[root@webnode3 ~]# gluster volume info

Volume Name: DATA-STORE
Type: Distributed-Replicate
Volume ID: b64c1fea-1500-4014-b36a-0487818fa893
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: webhost75:/home/DATABRINK
Brick2: webhost90:/home/DATABRINK
Brick3: mysqlhost1:/home/DATABRINK
Brick4: mysqlhost2:/home/DATABRINK
Options Reconfigured:
performance.readdir-ahead: on

I mounted this volume on same server with fuse.

please help me.

Thanks,
Ram


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Performance tuning: How do I measure the performance of IMAP-on-Gluster?

[Ernie Dunbar]

Hi everyone.

My Gluster cluster is finally behaving fairly well, CPU, disk and 
network performance has returned to a stable state, and I'd like to 
start doing some performance tuning. To do that though, we need to have 
some metrics to see if the changes we make, are making any difference at 
all.


The basic problem I'm trying to solve is how slow IMAP and Webmail are 
on our high-availability cluster. All of our e-mail is stored in Maildir 
format on the Gluster drive, and as a result, using Webmail to "read the 
next message" is something that takes a couple of seconds, even if we go 
back to a file that's been read recently and should be cached. I'm not 
entirely sure how to measure that with the `time` command on the Unix 
command line in any consistent manner, and other commands (like `ls` or 
`cat` or `du`) do work a lot faster if repeated within a short period of 
time. It appears to be the case that producing consistent results 
(especially the results that matter the most to the customers) basically 
requires an IMAP connection to read a message.


I need some other ideas about how I can perform this test. Your help 
will be much appreciated.

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Performance with Gluster+Fuse is 60x slower then Gluster+NFS ?

[Van Renterghem Stijn]

Hi,

I have setup a server with a new installation of Gluster.
The volume type is 'Replicate'.

1)
I mounted the volume with Fuse
IP1:/app   /srv/data   glusterfs   
defaults,_netdev,backupvolfile-server=IP2,fetch-attempts=2  0 0

When I start my application, it takes 2h until the application is started
Below you can see the stats after the application is started. I can see a very 
high LOOKUP value.
Can you explain this high value ? The volume type is replicate, so I should 
think, I shouldn't have LOOKUPs ?

Interval2
Block Size:  1b+  16b+  32b+
No. of Reads:0 0 0
No. of Writes:  34225   575

   Block Size: 64b+ 128b+ 256b+
No. of Reads:0 0 0
No. of Writes:  143   898   118

   Block Size:512b+1024b+2048b+
No. of Reads:1 411
No. of Writes:   82 0 0

   Block Size:   4096b+8192b+   16384b+
No. of Reads:   113139
No. of Writes:0 0 0

   Block Size:  32768b+   65536b+  131072b+
No. of Reads:   59   148   555
No. of Writes:0 0 0

%-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls Fop
-   ---   ---   ---   
  0.00   0.00 us   0.00 us   0.00 us  1  FORGET
  0.00   0.00 us   0.00 us   0.00 us201 RELEASE
  0.00   0.00 us   0.00 us   0.00 us  54549  RELEASEDIR
  0.00  47.00 us  47.00 us  47.00 us  1 REMOVEXATTR
  0.00  94.00 us  74.00 us 114.00 us  2 XATTROP
  0.00 191.00 us 191.00 us 191.00 us  1TRUNCATE
  0.00  53.50 us  35.00 us  74.00 us  4  STATFS
  0.00  79.67 us  70.00 us  91.00 us  3  RENAME
  0.00  37.33 us  27.00 us  68.00 us 15 INODELK
  0.00 190.67 us 116.00 us 252.00 us  3  UNLINK
  0.00  28.83 us   8.00 us  99.00 us 30 ENTRYLK
  0.00 146.33 us 117.00 us 188.00 us  6  CREATE
  0.00  37.63 us  12.00 us  73.00 us 84 READDIR
  0.00  23.75 us   8.00 us  75.00 us198   FLUSH
  0.00  65.33 us  42.00 us 141.00 us204OPEN
  0.01  45.78 us  11.00 us 191.00 us944FINODELK
  0.01  80.34 us  31.00 us 211.00 us859READ
  0.02  96.74 us  50.00 us 188.00 us944FXATTROP
  0.02  55.84 us  24.00 us 140.00 us   1707   FSTAT
  0.02  52.89 us  21.00 us 175.00 us   2183   WRITE
  0.02  59.69 us  11.00 us 235.00 us   2312GETXATTR
  0.03  51.18 us   8.00 us 142.00 us   3091STAT
  0.46  48.66 us   1.00 us 179.00 us  54549 OPENDIR
  1.13 135.93 us  18.00 us   16362.00 us  48124READDIRP
 98.29  70.46 us  16.00 us2903.00 us8104385  LOOKUP

Duration: 7560 seconds
   Data Read: 91208567 bytes = 91MB
Data Written: 292007 bytes = 0,292MB



2)
I have tried some tuning options, but that didn't changed anything:

#gluster volume info app

Volume Name: app
Type: Replicate
Volume ID: f1b59aec-adf8-41f8-ad95-839ace247041
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: IP1:/exports/app/app
Brick2: IP2:/exports/app/app
Options Reconfigured:
cluster.readdir-optimize: on
server.event-threads: 8
client.event-threads: 8
cluster.lookup-optimize: on
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
auth.allow: client1,client2
nfs.rpc-auth-allow: client1,client2
nfs.export-volumes: on
nfs.addr-namelookup: off
nfs.disable: off
performance.readdir-ahead: on
performance.io-thread-count: 64




3)
I then have enabled NFS support.
I stopped the application and unmounted the volume. I then mounted it again 
with nfs:
IP1:/app/srv/data   nfs 
rsize=4096,wsize=4096,hard,intr  0 0

I started the application again and it was running within 3minutes.
The 

Re: [Gluster-users] Performance issues with one node

[Mathieu Chateau]

Hello,

sorry operating-version is a variable like others, just need to find the
good name: op-version:

gluster volume get all cluster.op-version

then to set version (global to all volumes):
gluster volume set all cluster.op-version 30702


Cordialement,
Mathieu CHATEAU
http://www.lotp.fr

2015-07-28 8:03 GMT+02:00 Mathieu Chateau mathieu.chat...@lotp.fr:

 Hello,

 thanks for this guidance, I wasn't aware of!

 Any doc that describe all settings values ?
 For example, I can't find documentation for cluster.lookup-optimize


 Cordialement,
 Mathieu CHATEAU
 http://www.lotp.fr

 2015-07-27 14:58 GMT+02:00 André Bauer aba...@magix.net:

 Some more infos:

 http://gluster.readthedocs.org/en/latest/Feature%20Planning/GlusterFS%203.7/Small%20File%20Performance/index.html?highlight=small%20file%20performance

 Am 24.07.2015 um 20:15 schrieb Mathieu Chateau:
  Hello,
 
  gluster performance are not good with large number of small files.
  Recent version do a better job with them, but not yet what I would
 enjoy.
 
  As you are starting at gluster having an existing architecture, you
  should first setup a lab to learn about it Else you will learn the hard
 way.
  Don't play with turning off nodes, as you may create more issues than
 solve.
 
  just my 2cents
 
  Cordialement,
  Mathieu CHATEAU
  http://www.lotp.fr
 
  2015-07-24 19:34 GMT+02:00 John Kennedy skeb...@gmail.com
  mailto:skeb...@gmail.com:
 
  I am new to Gluster and have not found anything useful from my
  friend Google. I have not dealt with physical hardware in a few
  years (my last few jobs have been VM's and AWS based)
 
  I inherited a 4 node gluster configuration. There are 2 bricks, one
  is 9TB the other 11TB.
 
  The 11TB brick has a HUGE number of small files taking up only 1.3TB
  of the brick. For some reason, even a simple ls command can take
  hours to even start listing files. I removed a node by shutting down
  gluster on that node. The change in performance is dramatic. If I
  try and do ls on the 11TB brick on the downed node, I am still
  getting the slow response. I have narrowed the issue down to this
  one node as a result.
 
  When I start gluster on the bad node, glusterfsd hits over 1000%CPU
  use (The server has dual 8 core CPU's) and the load will jump to
  25-30 within 5 minutes. As such, I think this is a gluster issue and
  not a hardware issue. I am trying to not reinstall gluster yet.
 
  Is there something I am missing in my checks or will I need to
  reinstall gluster on that node?
 
  Thanks,
  John
 
  John Kennedy  (_8(|)
  Sometimes it happens, sometimes it doesn't - Pedro Catacora
 
  Just because nobody disagrees with you doesn't mean you are correct.
 
  Anatidaephobia is the fear that somehow, somewhere a duck is
  watching you - urbandictionary.com http://urbandictionary.com
 
  The Dunning-Kruger effect occurs when incompetent people not only
  fail to realize their incompetence, but consider themselves much
  more competent than everyone else. Basically - they're too stupid to
  know that they're stupid.
 
  ___
  Gluster-users mailing list
  Gluster-users@gluster.org mailto:Gluster-users@gluster.org
  http://www.gluster.org/mailman/listinfo/gluster-users
 
 
 
 
  ___
  Gluster-users mailing list
  Gluster-users@gluster.org
  http://www.gluster.org/mailman/listinfo/gluster-users
 


 --
 Mit freundlichen Grüßen
 André Bauer

 MAGIX Software GmbH
 André Bauer
 Administrator
 August-Bebel-Straße 48
 01219 Dresden
 GERMANY

 tel.: 0351 41884875
 e-mail: aba...@magix.net
 aba...@magix.net mailto:Email
 www.magix.com http://www.magix.com/


 Geschäftsführer | Managing Directors: Dr. Arnd Schröder, Michael Keith
 Amtsgericht | Commercial Register: Berlin Charlottenburg, HRB 127205

 Find us on:

 http://www.facebook.com/MAGIX http://www.twitter.com/magix_de
 http://www.youtube.com/wwwmagixcom http://www.magixmagazin.de
 --
 The information in this email is intended only for the addressee named
 above. Access to this email by anyone else is unauthorized. If you are
 not the intended recipient of this message any disclosure, copying,
 distribution or any action taken in reliance on it is prohibited and
 may be unlawful. MAGIX does not warrant that any attachments are free
 from viruses or other defects and accepts no liability for any losses
 resulting from infected email transmissions. Please note that any
 views expressed in this email may be those of the originator and do
 not necessarily represent the agenda of the company.
 --
 ___
 Gluster-users mailing list
 Gluster-users@gluster.org
 

Re: [Gluster-users] Performance issues with one node

[Mathieu Chateau]

Hello,

thanks for this guidance, I wasn't aware of!

Any doc that describe all settings values ?
For example, I can't find documentation for cluster.lookup-optimize


Cordialement,
Mathieu CHATEAU
http://www.lotp.fr

2015-07-27 14:58 GMT+02:00 André Bauer aba...@magix.net:

 Some more infos:

 http://gluster.readthedocs.org/en/latest/Feature%20Planning/GlusterFS%203.7/Small%20File%20Performance/index.html?highlight=small%20file%20performance

 Am 24.07.2015 um 20:15 schrieb Mathieu Chateau:
  Hello,
 
  gluster performance are not good with large number of small files.
  Recent version do a better job with them, but not yet what I would enjoy.
 
  As you are starting at gluster having an existing architecture, you
  should first setup a lab to learn about it Else you will learn the hard
 way.
  Don't play with turning off nodes, as you may create more issues than
 solve.
 
  just my 2cents
 
  Cordialement,
  Mathieu CHATEAU
  http://www.lotp.fr
 
  2015-07-24 19:34 GMT+02:00 John Kennedy skeb...@gmail.com
  mailto:skeb...@gmail.com:
 
  I am new to Gluster and have not found anything useful from my
  friend Google. I have not dealt with physical hardware in a few
  years (my last few jobs have been VM's and AWS based)
 
  I inherited a 4 node gluster configuration. There are 2 bricks, one
  is 9TB the other 11TB.
 
  The 11TB brick has a HUGE number of small files taking up only 1.3TB
  of the brick. For some reason, even a simple ls command can take
  hours to even start listing files. I removed a node by shutting down
  gluster on that node. The change in performance is dramatic. If I
  try and do ls on the 11TB brick on the downed node, I am still
  getting the slow response. I have narrowed the issue down to this
  one node as a result.
 
  When I start gluster on the bad node, glusterfsd hits over 1000%CPU
  use (The server has dual 8 core CPU's) and the load will jump to
  25-30 within 5 minutes. As such, I think this is a gluster issue and
  not a hardware issue. I am trying to not reinstall gluster yet.
 
  Is there something I am missing in my checks or will I need to
  reinstall gluster on that node?
 
  Thanks,
  John
 
  John Kennedy  (_8(|)
  Sometimes it happens, sometimes it doesn't - Pedro Catacora
 
  Just because nobody disagrees with you doesn't mean you are correct.
 
  Anatidaephobia is the fear that somehow, somewhere a duck is
  watching you - urbandictionary.com http://urbandictionary.com
 
  The Dunning-Kruger effect occurs when incompetent people not only
  fail to realize their incompetence, but consider themselves much
  more competent than everyone else. Basically - they're too stupid to
  know that they're stupid.
 
  ___
  Gluster-users mailing list
  Gluster-users@gluster.org mailto:Gluster-users@gluster.org
  http://www.gluster.org/mailman/listinfo/gluster-users
 
 
 
 
  ___
  Gluster-users mailing list
  Gluster-users@gluster.org
  http://www.gluster.org/mailman/listinfo/gluster-users
 


 --
 Mit freundlichen Grüßen
 André Bauer

 MAGIX Software GmbH
 André Bauer
 Administrator
 August-Bebel-Straße 48
 01219 Dresden
 GERMANY

 tel.: 0351 41884875
 e-mail: aba...@magix.net
 aba...@magix.net mailto:Email
 www.magix.com http://www.magix.com/


 Geschäftsführer | Managing Directors: Dr. Arnd Schröder, Michael Keith
 Amtsgericht | Commercial Register: Berlin Charlottenburg, HRB 127205

 Find us on:

 http://www.facebook.com/MAGIX http://www.twitter.com/magix_de
 http://www.youtube.com/wwwmagixcom http://www.magixmagazin.de
 --
 The information in this email is intended only for the addressee named
 above. Access to this email by anyone else is unauthorized. If you are
 not the intended recipient of this message any disclosure, copying,
 distribution or any action taken in reliance on it is prohibited and
 may be unlawful. MAGIX does not warrant that any attachments are free
 from viruses or other defects and accepts no liability for any losses
 resulting from infected email transmissions. Please note that any
 views expressed in this email may be those of the originator and do
 not necessarily represent the agenda of the company.
 --
 ___
 Gluster-users mailing list
 Gluster-users@gluster.org
 http://www.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Performance issues with one node

[Mathieu Chateau]

Hello,

gluster performance are not good with large number of small files. Recent
version do a better job with them, but not yet what I would enjoy.

As you are starting at gluster having an existing architecture, you should
first setup a lab to learn about it Else you will learn the hard way.
Don't play with turning off nodes, as you may create more issues than solve.

just my 2cents

Cordialement,
Mathieu CHATEAU
http://www.lotp.fr

2015-07-24 19:34 GMT+02:00 John Kennedy skeb...@gmail.com:

 I am new to Gluster and have not found anything useful from my friend
 Google. I have not dealt with physical hardware in a few years (my last few
 jobs have been VM's and AWS based)

 I inherited a 4 node gluster configuration. There are 2 bricks, one is 9TB
 the other 11TB.

 The 11TB brick has a HUGE number of small files taking up only 1.3TB of
 the brick. For some reason, even a simple ls command can take hours to even
 start listing files. I removed a node by shutting down gluster on that
 node. The change in performance is dramatic. If I try and do ls on the 11TB
 brick on the downed node, I am still getting the slow response. I have
 narrowed the issue down to this one node as a result.

 When I start gluster on the bad node, glusterfsd hits over 1000%CPU use
 (The server has dual 8 core CPU's) and the load will jump to 25-30 within 5
 minutes. As such, I think this is a gluster issue and not a hardware issue.
 I am trying to not reinstall gluster yet.

 Is there something I am missing in my checks or will I need to reinstall
 gluster on that node?

 Thanks,
 John

 John Kennedy  (_8(|)
 Sometimes it happens, sometimes it doesn't - Pedro Catacora

 Just because nobody disagrees with you doesn't mean you are correct.

 Anatidaephobia is the fear that somehow, somewhere a duck is watching you
 - urbandictionary.com

 The Dunning-Kruger effect occurs when incompetent people not only fail to
 realize their incompetence, but consider themselves much more competent
 than everyone else. Basically - they're too stupid to know that they're
 stupid.

 ___
 Gluster-users mailing list
 Gluster-users@gluster.org
 http://www.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Performance issues with one node

[John Kennedy]

I am new to Gluster and have not found anything useful from my friend
Google. I have not dealt with physical hardware in a few years (my last few
jobs have been VM's and AWS based)

I inherited a 4 node gluster configuration. There are 2 bricks, one is 9TB
the other 11TB.

The 11TB brick has a HUGE number of small files taking up only 1.3TB of the
brick. For some reason, even a simple ls command can take hours to even
start listing files. I removed a node by shutting down gluster on that
node. The change in performance is dramatic. If I try and do ls on the 11TB
brick on the downed node, I am still getting the slow response. I have
narrowed the issue down to this one node as a result.

When I start gluster on the bad node, glusterfsd hits over 1000%CPU use
(The server has dual 8 core CPU's) and the load will jump to 25-30 within 5
minutes. As such, I think this is a gluster issue and not a hardware issue.
I am trying to not reinstall gluster yet.

Is there something I am missing in my checks or will I need to reinstall
gluster on that node?

Thanks,
John

John Kennedy  (_8(|)
Sometimes it happens, sometimes it doesn't - Pedro Catacora

Just because nobody disagrees with you doesn't mean you are correct.

Anatidaephobia is the fear that somehow, somewhere a duck is watching you -
urbandictionary.com

The Dunning-Kruger effect occurs when incompetent people not only fail to
realize their incompetence, but consider themselves much more competent
than everyone else. Basically - they're too stupid to know that they're
stupid.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] performance tuning - list of available options?

[Vijay Bellur]

On 05/05/2015 04:34 PM, Kingsley wrote:

On Tue, 2015-05-05 at 15:08 +0530, Vijay Bellur wrote:
[snip]

I have seen this before and it primarily seems to be related to the
readdir calls done by git clone.

Turning on these options might help to some extent:

gluster volume set volname performance.readdir-ahead on

gluster volume set volname cluster.readdir-optimize on


Is there a list of all performance tuning options, and a description of
what they do? Even better if it details pros/cons of using a particular
option, or circumstances in which you might want to use it.

If such a list exists, it would be very useful. I've tried googling but
can only find pages for very old versions (eg gluster 3.2) and only
detailing /some/ options.

Knowing what the value defaults to would also be handy. In a separate
thread, someone advised me to do gluster volume set VOLNAME
performance.flush-behind off, but when I found that didn't help (the
issue turned out to be something I was doing, not a fault in gluster) I
couldn't figure out what the default value was, so I didn't know whether
to leave it off, or whether it had previously been on.

Is there anything available?



Thanks for this feedback. This is something that I would like to see 
captured too.


Raghavendra, Rafi - would you be able to help create a document on this one?

-Vijay
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] performance tuning - list of available options?

[Vijay Bellur]

On 05/05/2015 06:39 PM, Mohammed Rafi K C wrote:



On 05/05/2015 05:24 PM, Vijay Bellur wrote:

On 05/05/2015 04:34 PM, Kingsley wrote:

On Tue, 2015-05-05 at 15:08 +0530, Vijay Bellur wrote:
[snip]

I have seen this before and it primarily seems to be related to the
readdir calls done by git clone.

Turning on these options might help to some extent:

gluster volume set volname performance.readdir-ahead on

gluster volume set volname cluster.readdir-optimize on


Is there a list of all performance tuning options, and a description of
what they do? Even better if it details pros/cons of using a particular
option, or circumstances in which you might want to use it.

If such a list exists, it would be very useful. I've tried googling but
can only find pages for very old versions (eg gluster 3.2) and only
detailing /some/ options.

Knowing what the value defaults to would also be handy. In a separate
thread, someone advised me to do gluster volume set VOLNAME
performance.flush-behind off, but when I found that didn't help (the
issue turned out to be something I was doing, not a fault in gluster) I
couldn't figure out what the default value was, so I didn't know whether
to leave it off, or whether it had previously been on.

Is there anything available?



Thanks for this feedback. This is something that I would like to see
captured too.

Raghavendra, Rafi - would you be able to help create a document on
this one?

I would be happy to contribute for this, but I can only spend time after
3.7 release :-) .



Thanks!


Along with the performance turning guide, we can also enhance our
development guide.



Yes, there is an ongoing effort to improve our developer guide. We 
expect to see some results soon.


-Vijay

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] performance tuning - list of available options?

[Mohammed Rafi K C]

On 05/05/2015 05:24 PM, Vijay Bellur wrote:
 On 05/05/2015 04:34 PM, Kingsley wrote:
 On Tue, 2015-05-05 at 15:08 +0530, Vijay Bellur wrote:
 [snip]
 I have seen this before and it primarily seems to be related to the
 readdir calls done by git clone.

 Turning on these options might help to some extent:

 gluster volume set volname performance.readdir-ahead on

 gluster volume set volname cluster.readdir-optimize on

 Is there a list of all performance tuning options, and a description of
 what they do? Even better if it details pros/cons of using a particular
 option, or circumstances in which you might want to use it.

 If such a list exists, it would be very useful. I've tried googling but
 can only find pages for very old versions (eg gluster 3.2) and only
 detailing /some/ options.

 Knowing what the value defaults to would also be handy. In a separate
 thread, someone advised me to do gluster volume set VOLNAME
 performance.flush-behind off, but when I found that didn't help (the
 issue turned out to be something I was doing, not a fault in gluster) I
 couldn't figure out what the default value was, so I didn't know whether
 to leave it off, or whether it had previously been on.

 Is there anything available?


 Thanks for this feedback. This is something that I would like to see
 captured too.

 Raghavendra, Rafi - would you be able to help create a document on
 this one?
I would be happy to contribute for this, but I can only spend time after
3.7 release :-) .

Along with the performance turning guide, we can also enhance our
development guide.

Rafi KC

 -Vijay

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] performance tuning - list of available options?

[Ben Turner]

- Original Message -
 From: Mohammed Rafi K C rkavu...@redhat.com
 To: Vijay Bellur vbel...@redhat.com, Kingsley 
 glus...@gluster.dogwind.com, gluster-users@gluster.org,
 Raghavendra Gowdappa rgowd...@redhat.com
 Sent: Tuesday, May 5, 2015 9:09:31 AM
 Subject: Re: [Gluster-users] performance tuning - list of available options?
 
 
 
 On 05/05/2015 05:24 PM, Vijay Bellur wrote:
  On 05/05/2015 04:34 PM, Kingsley wrote:
  On Tue, 2015-05-05 at 15:08 +0530, Vijay Bellur wrote:
  [snip]
  I have seen this before and it primarily seems to be related to the
  readdir calls done by git clone.
 
  Turning on these options might help to some extent:
 
  gluster volume set volname performance.readdir-ahead on
 
  gluster volume set volname cluster.readdir-optimize on
 
  Is there a list of all performance tuning options, and a description of
  what they do? Even better if it details pros/cons of using a particular
  option, or circumstances in which you might want to use it.
 
  If such a list exists, it would be very useful. I've tried googling but
  can only find pages for very old versions (eg gluster 3.2) and only
  detailing /some/ options.
 
  Knowing what the value defaults to would also be handy. In a separate
  thread, someone advised me to do gluster volume set VOLNAME
  performance.flush-behind off, but when I found that didn't help (the
  issue turned out to be something I was doing, not a fault in gluster) I
  couldn't figure out what the default value was, so I didn't know whether
  to leave it off, or whether it had previously been on.
 
  Is there anything available?
 
 
  Thanks for this feedback. This is something that I would like to see
  captured too.
 
  Raghavendra, Rafi - would you be able to help create a document on
  this one?
 I would be happy to contribute for this, but I can only spend time after
 3.7 release :-) .
 
 Along with the performance turning guide, we can also enhance our
 development guide.

I have some ideas here, happy to help.

-b

 
 Rafi KC
 
  -Vijay
 
 ___
 Gluster-users mailing list
 Gluster-users@gluster.org
 http://www.gluster.org/mailman/listinfo/gluster-users
 
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] performance tuning - list of available options?

[Kingsley]

On Tue, 2015-05-05 at 15:08 +0530, Vijay Bellur wrote:
[snip] 
 I have seen this before and it primarily seems to be related to the 
 readdir calls done by git clone.
 
 Turning on these options might help to some extent:
 
 gluster volume set volname performance.readdir-ahead on
 
 gluster volume set volname cluster.readdir-optimize on

Is there a list of all performance tuning options, and a description of
what they do? Even better if it details pros/cons of using a particular
option, or circumstances in which you might want to use it.

If such a list exists, it would be very useful. I've tried googling but
can only find pages for very old versions (eg gluster 3.2) and only
detailing /some/ options.

Knowing what the value defaults to would also be handy. In a separate
thread, someone advised me to do gluster volume set VOLNAME
performance.flush-behind off, but when I found that didn't help (the
issue turned out to be something I was doing, not a fault in gluster) I
couldn't figure out what the default value was, so I didn't know whether
to leave it off, or whether it had previously been on.

Is there anything available?

-- 
Cheers,
Kingsley.

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Performance and blocksize

[Gregor Burck]

Hi,

I test diffrent blocksize on a local mount.

with dd and a bs=1MB the performance is much better than bs=1KB, I read here  
that a to small blocksize is poisen for performance,...

I plan to create a glustercluster for virtual image files, what should the best 
blocksize for the virtual machines? Like the nodes bs? And how calculate the 
bs?

Bye

Gregor
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users