date:20190703

Re: [Gluster-users] Extremely low performance - am I doing somethingwrong?

2019-07-03 Thread Strahil

I think , it'related to the sync type of oflag.
Do you have a raid controller on each brick , to immediate take the data into 
the cache ?

Best Regards,
Strahil NikolovOn Jul 3, 2019 23:15, Vladimir Melnik  wrote:
>
> Indeed, I wouldn't be surprised if I had around 80-100 MB/s, but 10-15 
> MB/s is really few. :-( 
>
> Even when I mount a filesystem on the same GlusterFS node, I have the 
> following result: 
> 10485760 bytes (10 MB) copied, 0.409856 s, 25.6 MB/s 
> 10485760 bytes (10 MB) copied, 0.38967 s, 26.9 MB/s 
> 10485760 bytes (10 MB) copied, 0.466758 s, 22.5 MB/s 
> 10485760 bytes (10 MB) copied, 0.412075 s, 25.4 MB/s 
> 10485760 bytes (10 MB) copied, 0.381626 s, 27.5 MB/s 
>
> At the same time on the same node when I'm writing directly to the disk: 
> 10485760 bytes (10 MB) copied, 0.0326612 s, 321 MB/s 
> 10485760 bytes (10 MB) copied, 0.0302878 s, 346 MB/s 
> 10485760 bytes (10 MB) copied, 0.0352449 s, 298 MB/s 
> 10485760 bytes (10 MB) copied, 0.0316872 s, 331 MB/s 
> 10485760 bytes (10 MB) copied, 0.0333189 s, 315 MB/s 
>
> Can't explain it to myself. Are replica-3 volumes really so slow? 
>
> On Wed, Jul 03, 2019 at 03:16:45PM -0400, Dmitry Filonov wrote: 
> > Well, if your network is limited to 100MB/s then it doesn't matter if 
> > storage is capable of doing 300+MB/s. 
> > But 15 MB/s is still way less than 100 MB/s 
> > 
> > P.S. just tried on my gluster and found out that am getting ~15MB/s on 
> > replica 3 volume on SSDs and... 2MB/s on replica 3 volume on HDDs. 
> > Something to look at next week. 
> > 
> > 
> > 
> > -- 
> > Dmitry Filonov 
> > Linux Administrator 
> > SBGrid Core | Harvard Medical School 
> > 250 Longwood Ave, SGM-114 
> > Boston, MA 02115 
> > 
> > 
> > On Wed, Jul 3, 2019 at 12:18 PM Vladimir Melnik  wrote: 
> > 
> > > Thank you, it helped a little: 
> > > 
> > > $ for i in {1..5}; do { dd if=/dev/zero of=/mnt/glusterfs1/test.tmp bs=1M 
> > > count=10 oflag=sync; rm -f /mnt/glusterfs1/test.tmp; } done 2>&1 | grep 
> > > copied 
> > > 10485760 bytes (10 MB) copied, 0.738968 s, 14.2 MB/s 
> > > 10485760 bytes (10 MB) copied, 0.725296 s, 14.5 MB/s 
> > > 10485760 bytes (10 MB) copied, 0.681508 s, 15.4 MB/s 
> > > 10485760 bytes (10 MB) copied, 0.85566 s, 12.3 MB/s 
> > > 10485760 bytes (10 MB) copied, 0.661457 s, 15.9 MB/s 
> > > 
> > > But 14-15 MB/s is still quite far from the actual storage's performance 
> > > (200-3000 MB/s). :-( 
> > > 
> > > Here's full configuration dump (just in case): 
> > > 
> > > Option  Value 
> > > --  - 
> > > cluster.lookup-unhashed on 
> > > cluster.lookup-optimize on 
> > > cluster.min-free-disk   10% 
> > > cluster.min-free-inodes 5% 
> > > cluster.rebalance-stats off 
> > > cluster.subvols-per-directory   (null) 
> > > cluster.readdir-optimize    off 
> > > cluster.rsync-hash-regex    (null) 
> > > cluster.extra-hash-regex    (null) 
> > > cluster.dht-xattr-name  trusted.glusterfs.dht 
> > > cluster.randomize-hash-range-by-gfid    off 
> > > cluster.rebal-throttle  normal 
> > > cluster.lock-migration  off 
> > > cluster.force-migration off 
> > > cluster.local-volume-name   (null) 
> > > cluster.weighted-rebalance  on 
> > > cluster.switch-pattern  (null) 
> > > cluster.entry-change-log    on 
> > > cluster.read-subvolume  (null) 
> > > cluster.read-subvolume-index    -1 
> > > cluster.read-hash-mode  1 
> > > cluster.background-self-heal-count  8 
> > > cluster.metadata-self-heal  off 
> > > cluster.data-self-heal  off 
> > > cluster.entry-self-heal off 
> > > cluster.self-heal-daemon    on 
> > > cluster.heal-timeout    600 
> > > cluster.self-heal-window-size   1 
> > > cluster.data-change-log on 
> > > cluster.metadata-change-log on 
> > > cluster.data-self-heal-algorithm    full 
> > > cluster.eager-lock  enable 
> > > disperse.eager-lock on 
> > > disperse.other-eager-lock   on 
> > > disperse.eager-lock-timeout 1 
> > > disperse.other-eager-lock-timeout   1 
> > > cluster.quorum-type auto 
> > > cluster.quorum-count    (null) 
> > > cluster.choose-local    off 
> > > cluster.self-heal-readdir-size  1KB 
> > > cluster.post-op-delay-secs  1 
> > > cluster.ensure-durability   on 
> > > cluster.consistent-metadata no 
> > > cluster.heal-wait-queue-length  128 
> > > cluster.favorite-child-policy   none 
> > > cluster.full-lock   yes 
> > >

Re: [Gluster-users] Extremely low performance - am I doing somethingwrong?

2019-07-03 Thread Vladimir Melnik

OK, I tweaked the virtualization parameters and now I have ~10 Gbit/s
between all the nodes.

$ iperf3 -c 10.13.1.16
Connecting to host 10.13.1.16, port 5201
[  4] local 10.13.1.17 port 47242 connected to 10.13.1.16 port 5201
[ ID] Interval   Transfer Bandwidth   Retr  Cwnd
[  4]   0.00-1.00   sec  1.42 GBytes  12.2 Gbits/sec0   1.86 MBytes
[  4]   1.00-2.00   sec  1.54 GBytes  13.3 Gbits/sec0   2.53 MBytes
[  4]   2.00-3.00   sec  1.37 GBytes  11.8 Gbits/sec0   2.60 MBytes
[  4]   3.00-4.00   sec  1.25 GBytes  10.7 Gbits/sec0   2.70 MBytes
[  4]   4.00-5.00   sec  1.30 GBytes  11.1 Gbits/sec0   2.81 MBytes
[  4]   5.00-6.00   sec  1.55 GBytes  13.3 Gbits/sec0   2.86 MBytes
[  4]   6.00-7.00   sec  1.46 GBytes  12.6 Gbits/sec0   2.92 MBytes
[  4]   7.00-8.00   sec  1.41 GBytes  12.1 Gbits/sec0   2.97 MBytes
[  4]   8.00-9.00   sec  1.39 GBytes  12.0 Gbits/sec0   2.98 MBytes
[  4]   9.00-10.00  sec  1.46 GBytes  12.5 Gbits/sec0   3.00 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval   Transfer Bandwidth   Retr
[  4]   0.00-10.00  sec  14.2 GBytes  12.2 Gbits/sec0 sender
[  4]   0.00-10.00  sec  14.1 GBytes  12.2 Gbits/sec  receiver

iperf Done.

$ iperf3 -c 10.13.1.16 -R
Connecting to host 10.13.1.16, port 5201
Reverse mode, remote host 10.13.1.16 is sending
[  4] local 10.13.1.17 port 47246 connected to 10.13.1.16 port 5201
[ ID] Interval   Transfer Bandwidth
[  4]   0.00-1.00   sec  1.63 GBytes  14.0 Gbits/sec
[  4]   1.00-2.00   sec  1.63 GBytes  14.0 Gbits/sec
[  4]   2.00-3.00   sec  1.56 GBytes  13.4 Gbits/sec
[  4]   3.00-4.00   sec  1.24 GBytes  10.7 Gbits/sec
[  4]   4.00-5.00   sec  1.51 GBytes  13.0 Gbits/sec
[  4]   5.00-6.00   sec  1.40 GBytes  12.0 Gbits/sec
[  4]   6.00-7.00   sec  1.49 GBytes  12.8 Gbits/sec
[  4]   7.00-8.00   sec  1.58 GBytes  13.6 Gbits/sec
[  4]   8.00-9.00   sec  1.45 GBytes  12.4 Gbits/sec
[  4]   9.00-10.00  sec  1.47 GBytes  12.6 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval   Transfer Bandwidth   Retr
[  4]   0.00-10.00  sec  15.0 GBytes  12.9 Gbits/sec0 sender
[  4]   0.00-10.00  sec  15.0 GBytes  12.9 Gbits/sec  receiver

iperf Done.

Looks good, right? Let's see the what it has given to us...

$ for i in {1..5}; do { dd if=/dev/zero of=/mnt/tmp/test.tmp bs=1M count=10 
oflag=sync; rm -f /mnt/tmp/test.tmp; } done 2>&1 | grep copied
10485760 bytes (10 MB) copied, 0.403512 s, 26.0 MB/s
10485760 bytes (10 MB) copied, 0.354702 s, 29.6 MB/s
10485760 bytes (10 MB) copied, 0.386806 s, 27.1 MB/s
10485760 bytes (10 MB) copied, 0.405671 s, 25.8 MB/s
10485760 bytes (10 MB) copied, 0.426986 s, 24.6 MB/s

So, the network can do ~10 Gbit/s, the disk can do ~2 Gbit/s, the
GlusterFS can do ~0.2 Gbit/s.

Am I the only one so lucky? :-)

Does anyone else observe the samephenomenon?

On Wed, Jul 03, 2019 at 05:01:37PM -0400, Guy Boisvert wrote:
> Yeah, 10 Gbps is affordable these days, even 25 Gbps!  Wouldn't go lower than 
> 10 Gbps.
> 
> ⁣Get BlueMail for Android 
> 
> On Jul 3, 2019, 16:59, at 16:59, Marcus Schopen  wrote:
> >Hi,
> >
> >Am Mittwoch, den 03.07.2019, 15:16 -0400 schrieb Dmitry Filonov:
> >> Well, if your network is limited to 100MB/s then it doesn't matter if
> >> storage is capable of doing 300+MB/s.
> >> But 15 MB/s is still way less than 100 MB/s
> >
> >What network is recommended in the backend, 10 Gigabit or better more?
> >
> >Ciao!
> >Marcus
> >
> >
> >___
> >Gluster-users mailing list
> >Gluster-users@gluster.org
> >https://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Extremely low performance - am I doing somethingwrong?

2019-07-03 Thread Guy Boisvert

Yeah, 10 Gbps is affordable these days, even 25 Gbps!  Wouldn't go lower than 
10 Gbps.

⁣Get BlueMail for Android 

On Jul 3, 2019, 16:59, at 16:59, Marcus Schopen  wrote:
>Hi,
>
>Am Mittwoch, den 03.07.2019, 15:16 -0400 schrieb Dmitry Filonov:
>> Well, if your network is limited to 100MB/s then it doesn't matter if
>> storage is capable of doing 300+MB/s.
>> But 15 MB/s is still way less than 100 MB/s
>
>What network is recommended in the backend, 10 Gigabit or better more?
>
>Ciao!
>Marcus
>
>
>___
>Gluster-users mailing list
>Gluster-users@gluster.org
>https://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Extremely low performance - am I doing somethingwrong?

2019-07-03 Thread Marcus Schopen

Hi,

Am Mittwoch, den 03.07.2019, 15:16 -0400 schrieb Dmitry Filonov:
> Well, if your network is limited to 100MB/s then it doesn't matter if
> storage is capable of doing 300+MB/s. 
> But 15 MB/s is still way less than 100 MB/s

What network is recommended in the backend, 10 Gigabit or better more?

Ciao!
Marcus


___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Extremely low performance - am I doing somethingwrong?

2019-07-03 Thread Vladimir Melnik

Indeed, I wouldn't be surprised if I had around 80-100 MB/s, but 10-15
MB/s is really few. :-(

Even when I mount a filesystem on the same GlusterFS node, I have the
following result:
10485760 bytes (10 MB) copied, 0.409856 s, 25.6 MB/s
10485760 bytes (10 MB) copied, 0.38967 s, 26.9 MB/s
10485760 bytes (10 MB) copied, 0.466758 s, 22.5 MB/s
10485760 bytes (10 MB) copied, 0.412075 s, 25.4 MB/s
10485760 bytes (10 MB) copied, 0.381626 s, 27.5 MB/s

At the same time on the same node when I'm writing directly to the disk:
10485760 bytes (10 MB) copied, 0.0326612 s, 321 MB/s
10485760 bytes (10 MB) copied, 0.0302878 s, 346 MB/s
10485760 bytes (10 MB) copied, 0.0352449 s, 298 MB/s
10485760 bytes (10 MB) copied, 0.0316872 s, 331 MB/s
10485760 bytes (10 MB) copied, 0.0333189 s, 315 MB/s

Can't explain it to myself. Are replica-3 volumes really so slow?

On Wed, Jul 03, 2019 at 03:16:45PM -0400, Dmitry Filonov wrote:
> Well, if your network is limited to 100MB/s then it doesn't matter if
> storage is capable of doing 300+MB/s.
> But 15 MB/s is still way less than 100 MB/s
> 
> P.S. just tried on my gluster and found out that am getting ~15MB/s on
> replica 3 volume on SSDs and... 2MB/s on replica 3 volume on HDDs.
> Something to look at next week.
> 
> 
> 
> --
> Dmitry Filonov
> Linux Administrator
> SBGrid Core | Harvard Medical School
> 250 Longwood Ave, SGM-114
> Boston, MA 02115
> 
> 
> On Wed, Jul 3, 2019 at 12:18 PM Vladimir Melnik  wrote:
> 
> > Thank you, it helped a little:
> >
> > $ for i in {1..5}; do { dd if=/dev/zero of=/mnt/glusterfs1/test.tmp bs=1M
> > count=10 oflag=sync; rm -f /mnt/glusterfs1/test.tmp; } done 2>&1 | grep
> > copied
> > 10485760 bytes (10 MB) copied, 0.738968 s, 14.2 MB/s
> > 10485760 bytes (10 MB) copied, 0.725296 s, 14.5 MB/s
> > 10485760 bytes (10 MB) copied, 0.681508 s, 15.4 MB/s
> > 10485760 bytes (10 MB) copied, 0.85566 s, 12.3 MB/s
> > 10485760 bytes (10 MB) copied, 0.661457 s, 15.9 MB/s
> >
> > But 14-15 MB/s is still quite far from the actual storage's performance
> > (200-3000 MB/s). :-(
> >
> > Here's full configuration dump (just in case):
> >
> > Option  Value
> > --  -
> > cluster.lookup-unhashed on
> > cluster.lookup-optimize on
> > cluster.min-free-disk   10%
> > cluster.min-free-inodes 5%
> > cluster.rebalance-stats off
> > cluster.subvols-per-directory   (null)
> > cluster.readdir-optimizeoff
> > cluster.rsync-hash-regex(null)
> > cluster.extra-hash-regex(null)
> > cluster.dht-xattr-name  trusted.glusterfs.dht
> > cluster.randomize-hash-range-by-gfidoff
> > cluster.rebal-throttle  normal
> > cluster.lock-migration  off
> > cluster.force-migration off
> > cluster.local-volume-name   (null)
> > cluster.weighted-rebalance  on
> > cluster.switch-pattern  (null)
> > cluster.entry-change-logon
> > cluster.read-subvolume  (null)
> > cluster.read-subvolume-index-1
> > cluster.read-hash-mode  1
> > cluster.background-self-heal-count  8
> > cluster.metadata-self-heal  off
> > cluster.data-self-heal  off
> > cluster.entry-self-heal off
> > cluster.self-heal-daemonon
> > cluster.heal-timeout600
> > cluster.self-heal-window-size   1
> > cluster.data-change-log on
> > cluster.metadata-change-log on
> > cluster.data-self-heal-algorithmfull
> > cluster.eager-lock  enable
> > disperse.eager-lock on
> > disperse.other-eager-lock   on
> > disperse.eager-lock-timeout 1
> > disperse.other-eager-lock-timeout   1
> > cluster.quorum-type auto
> > cluster.quorum-count(null)
> > cluster.choose-localoff
> > cluster.self-heal-readdir-size  1KB
> > cluster.post-op-delay-secs  1
> > cluster.ensure-durability   on
> > cluster.consistent-metadata no
> > cluster.heal-wait-queue-length  128
> > cluster.favorite-child-policy   none
> > cluster.full-lock   yes
> > diagnostics.latency-measurement off
> > diagnostics.dump-fd-stats   off
> > diagnostics.count-fop-hits  off
> > diagnostics.brick-log-level INFO
> > diagnostics.client-log-levelINFO
> > diagnostics.brick-sys-log-level CRITICAL
> > diagnostics.client-sys-log-levelCRITICAL
> > diagnostics.brick-logger(null)
> > diagnostics.client-logger   (null)
> > diagnostics.brick-log-format(null)
> > diagnostics.client-log-format   (null)
>

Re: [Gluster-users] What is the right way to bring down a Glusterfs server for maintenance?

2019-07-03 Thread Carl Sirotic


Very good, I will give this a try.

Thank you.


Carl

On 2019-07-03 3:56 p.m., John Strunk wrote:

Nope. Just:
* Ensure all volumes are fully healed so you don't run into split brain
* Go ahead and shutdown the server needing maintenance
** If you just want gluster down on that sever node: stop glusterd and 
kill the glusterfs bricks, then do what you need to do
** If you just want to power off: then shutdown -h as usual (don't 
worry about stopping gluster)


When you bring the server back up, glusterd and the bricks should 
start, and the bricks should heal from the 2 replicas that remained up 
during maintenance.


-John


On Wed, Jul 3, 2019 at 3:48 PM Carl Sirotic 
mailto:csiro...@evoqarchitecture.com>> 
wrote:


I have a replica 3 cluster, 3 nodes with bricks and 2 "client" nodes,
that run the VMs through a mount of the data on the bricks.

Now, one of the bricks need maintenance and I will need to shut it
down
for about 15 minutes.

I didn't find any information on what I am suposed to do.

If I get this right, I am suposed to remove the brick completely from
the cluster and add them again when the maintenance is finished ?


Carl

___
Gluster-users mailing list
Gluster-users@gluster.org 
https://lists.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] What is the right way to bring down a Glusterfs server for maintenance?

2019-07-03 Thread John Strunk

Nope. Just:
* Ensure all volumes are fully healed so you don't run into split brain
* Go ahead and shutdown the server needing maintenance
** If you just want gluster down on that sever node: stop glusterd and kill
the glusterfs bricks, then do what you need to do
** If you just want to power off: then shutdown -h as usual (don't worry
about stopping gluster)

When you bring the server back up, glusterd and the bricks should start,
and the bricks should heal from the 2 replicas that remained up during
maintenance.

-John

On Wed, Jul 3, 2019 at 3:48 PM Carl Sirotic 
wrote:

> I have a replica 3 cluster, 3 nodes with bricks and 2 "client" nodes,
> that run the VMs through a mount of the data on the bricks.
>
> Now, one of the bricks need maintenance and I will need to shut it down
> for about 15 minutes.
>
> I didn't find any information on what I am suposed to do.
>
> If I get this right, I am suposed to remove the brick completely from
> the cluster and add them again when the maintenance is finished ?
>
>
> Carl
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] GlusterFS FUSE client on BSD

2019-07-03 Thread mabi

Hello,

Is there a way to mount a GlusterFS volume using FUSE on an BSD machine such as 
OpenBSD?

If not, what is the alternative, I guess NFS?

Regards,
M.





___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] What is the right way to bring down a Glusterfs server for maintenance?

2019-07-03 Thread Carl Sirotic

I have a replica 3 cluster, 3 nodes with bricks and 2 "client" nodes, 
that run the VMs through a mount of the data on the bricks.


Now, one of the bricks need maintenance and I will need to shut it down 
for about 15 minutes.


I didn't find any information on what I am suposed to do.

If I get this right, I am suposed to remove the brick completely from 
the cluster and add them again when the maintenance is finished ?



Carl

___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Extremely low performance - am I doing somethingwrong?

2019-07-03 Thread Dmitry Filonov

Well, if your network is limited to 100MB/s then it doesn't matter if
storage is capable of doing 300+MB/s.
But 15 MB/s is still way less than 100 MB/s

P.S. just tried on my gluster and found out that am getting ~15MB/s on
replica 3 volume on SSDs and... 2MB/s on replica 3 volume on HDDs.
Something to look at next week.



--
Dmitry Filonov
Linux Administrator
SBGrid Core | Harvard Medical School
250 Longwood Ave, SGM-114
Boston, MA 02115


On Wed, Jul 3, 2019 at 12:18 PM Vladimir Melnik  wrote:

> Thank you, it helped a little:
>
> $ for i in {1..5}; do { dd if=/dev/zero of=/mnt/glusterfs1/test.tmp bs=1M
> count=10 oflag=sync; rm -f /mnt/glusterfs1/test.tmp; } done 2>&1 | grep
> copied
> 10485760 bytes (10 MB) copied, 0.738968 s, 14.2 MB/s
> 10485760 bytes (10 MB) copied, 0.725296 s, 14.5 MB/s
> 10485760 bytes (10 MB) copied, 0.681508 s, 15.4 MB/s
> 10485760 bytes (10 MB) copied, 0.85566 s, 12.3 MB/s
> 10485760 bytes (10 MB) copied, 0.661457 s, 15.9 MB/s
>
> But 14-15 MB/s is still quite far from the actual storage's performance
> (200-3000 MB/s). :-(
>
> Here's full configuration dump (just in case):
>
> Option  Value
> --  -
> cluster.lookup-unhashed on
> cluster.lookup-optimize on
> cluster.min-free-disk   10%
> cluster.min-free-inodes 5%
> cluster.rebalance-stats off
> cluster.subvols-per-directory   (null)
> cluster.readdir-optimizeoff
> cluster.rsync-hash-regex(null)
> cluster.extra-hash-regex(null)
> cluster.dht-xattr-name  trusted.glusterfs.dht
> cluster.randomize-hash-range-by-gfidoff
> cluster.rebal-throttle  normal
> cluster.lock-migration  off
> cluster.force-migration off
> cluster.local-volume-name   (null)
> cluster.weighted-rebalance  on
> cluster.switch-pattern  (null)
> cluster.entry-change-logon
> cluster.read-subvolume  (null)
> cluster.read-subvolume-index-1
> cluster.read-hash-mode  1
> cluster.background-self-heal-count  8
> cluster.metadata-self-heal  off
> cluster.data-self-heal  off
> cluster.entry-self-heal off
> cluster.self-heal-daemonon
> cluster.heal-timeout600
> cluster.self-heal-window-size   1
> cluster.data-change-log on
> cluster.metadata-change-log on
> cluster.data-self-heal-algorithmfull
> cluster.eager-lock  enable
> disperse.eager-lock on
> disperse.other-eager-lock   on
> disperse.eager-lock-timeout 1
> disperse.other-eager-lock-timeout   1
> cluster.quorum-type auto
> cluster.quorum-count(null)
> cluster.choose-localoff
> cluster.self-heal-readdir-size  1KB
> cluster.post-op-delay-secs  1
> cluster.ensure-durability   on
> cluster.consistent-metadata no
> cluster.heal-wait-queue-length  128
> cluster.favorite-child-policy   none
> cluster.full-lock   yes
> diagnostics.latency-measurement off
> diagnostics.dump-fd-stats   off
> diagnostics.count-fop-hits  off
> diagnostics.brick-log-level INFO
> diagnostics.client-log-levelINFO
> diagnostics.brick-sys-log-level CRITICAL
> diagnostics.client-sys-log-levelCRITICAL
> diagnostics.brick-logger(null)
> diagnostics.client-logger   (null)
> diagnostics.brick-log-format(null)
> diagnostics.client-log-format   (null)
> diagnostics.brick-log-buf-size  5
> diagnostics.client-log-buf-size 5
> diagnostics.brick-log-flush-timeout 120
> diagnostics.client-log-flush-timeout120
> diagnostics.stats-dump-interval 0
> diagnostics.fop-sample-interval 0
> diagnostics.stats-dump-format   json
> diagnostics.fop-sample-buf-size 65535
> diagnostics.stats-dnscache-ttl-sec  86400
> performance.cache-max-file-size 0
> performance.cache-min-file-size 0
> performance.cache-refresh-timeout   1
> performance.cache-priority
> performance.cache-size  32MB
> performance.io-thread-count 16
> performance.high-prio-threads   16
> performance.normal-prio-threads 16
> performance.low-prio-threads32
> performance.least-prio-threads  1
> performance.enable-least-priority   on
> performance.iot-watchdog-secs   (null)
> performance.iot-cleanup-disconnected-reqsoff
> performance.iot-pass-throughfalse
> performance.io-cache-pass-through   false
> performance.cache-size

Re: [Gluster-users] Extremely low performance - am I doing somethingwrong?

2019-07-03 Thread Vladimir Melnik

Thank you, I tried to do that.

Created a new volume:
$ gluster volume create storage2 \
replica 3 \
arbiter 1 \
transport tcp \
gluster1.k8s.maitre-d.tucha.ua:/mnt/storage2/brick1 \
gluster2.k8s.maitre-d.tucha.ua:/mnt/storage2/brick2 \
gluster3.k8s.maitre-d.tucha.ua:/mnt/storage2/brick_arbiter \
gluster3.k8s.maitre-d.tucha.ua:/mnt/storage2/brick3 \
gluster4.k8s.maitre-d.tucha.ua:/mnt/storage2/brick4 \
gluster4.k8s.maitre-d.tucha.ua:/mnt/storage2/brick_arbiter

Changed the volume's settings:
$ gluster volume set storage2 group virt

Started the volume:
$ gluster volume start storage2

And the same thing:
$ for i in {1..5}; do { dd if=/dev/zero of=/mnt/glusterfs2/test.tmp bs=1M 
count=10 oflag=sync; rm -f /mnt/glusterfs2/test.tmp; } done 2>&1 | grep copied
10485760 bytes (10 MB) copied, 0.988662 s, 10.6 MB/s
10485760 bytes (10 MB) copied, 0.768863 s, 13.6 MB/s
10485760 bytes (10 MB) copied, 0.828568 s, 12.7 MB/s
10485760 bytes (10 MB) copied, 0.84322 s, 12.4 MB/s
10485760 bytes (10 MB) copied, 0.812504 s, 12.9 MB/s

On Wed, Jul 03, 2019 at 05:59:24PM +, Strahil Nikolov wrote:
>  Can you try with a fresh replica volume with 'virt' group applied ?
> Best Regards,Strahil Nikolov
> В сряда, 3 юли 2019 г., 19:18:18 ч. Гринуич+3, Vladimir Melnik 
>  написа:  
>  
>  Thank you, it helped a little:
> 
> $ for i in {1..5}; do { dd if=/dev/zero of=/mnt/glusterfs1/test.tmp bs=1M 
> count=10 oflag=sync; rm -f /mnt/glusterfs1/test.tmp; } done 2>&1 | grep copied
> 10485760 bytes (10 MB) copied, 0.738968 s, 14.2 MB/s
> 10485760 bytes (10 MB) copied, 0.725296 s, 14.5 MB/s
> 10485760 bytes (10 MB) copied, 0.681508 s, 15.4 MB/s
> 10485760 bytes (10 MB) copied, 0.85566 s, 12.3 MB/s
> 10485760 bytes (10 MB) copied, 0.661457 s, 15.9 MB/s
> 
> But 14-15 MB/s is still quite far from the actual storage's performance 
> (200-3000 MB/s). :-(
> 
> Here's full configuration dump (just in case):
> 
> Option                                  Value
> --                                  -
> cluster.lookup-unhashed                on
> cluster.lookup-optimize                on
> cluster.min-free-disk                  10%
> cluster.min-free-inodes                5%
> cluster.rebalance-stats                off
> cluster.subvols-per-directory          (null)
> cluster.readdir-optimize                off
> cluster.rsync-hash-regex                (null)
> cluster.extra-hash-regex                (null)
> cluster.dht-xattr-name                  trusted.glusterfs.dht
> cluster.randomize-hash-range-by-gfid    off
> cluster.rebal-throttle                  normal
> cluster.lock-migration                  off
> cluster.force-migration                off
> cluster.local-volume-name              (null)
> cluster.weighted-rebalance              on
> cluster.switch-pattern                  (null)
> cluster.entry-change-log                on
> cluster.read-subvolume                  (null)
> cluster.read-subvolume-index            -1
> cluster.read-hash-mode                  1
> cluster.background-self-heal-count      8
> cluster.metadata-self-heal              off
> cluster.data-self-heal                  off
> cluster.entry-self-heal                off
> cluster.self-heal-daemon                on
> cluster.heal-timeout                    600
> cluster.self-heal-window-size          1
> cluster.data-change-log                on
> cluster.metadata-change-log            on
> cluster.data-self-heal-algorithm        full
> cluster.eager-lock                      enable
> disperse.eager-lock                    on
> disperse.other-eager-lock              on
> disperse.eager-lock-timeout            1
> disperse.other-eager-lock-timeout      1
> cluster.quorum-type                    auto
> cluster.quorum-count                    (null)
> cluster.choose-local                    off
> cluster.self-heal-readdir-size          1KB
> cluster.post-op-delay-secs              1
> cluster.ensure-durability              on
> cluster.consistent-metadata            no
> cluster.heal-wait-queue-length          128
> cluster.favorite-child-policy          none
> cluster.full-lock                      yes
> diagnostics.latency-measurement        off
> diagnostics.dump-fd-stats              off
> diagnostics.count-fop-hits              off
> diagnostics.brick-log-level            INFO
> diagnostics.client-log-level            INFO
> diagnostics.brick-sys-log-level        CRITICAL
> diagnostics.client-sys-log-level        CRITICAL
> diagnostics.brick-logger                (null)
> diagnostics.client-logger              (null)
> diagnostics.brick-log-format            (null)
> diagnostics.client-log-format          (null)
> diagnostics.brick-log-buf-size          5
> diagnostics.client-log-buf-size        5
> diagnostics.brick-log-flush-timeout    120
> diagnostics.client-log-flush-timeout    120
> diagnostics.stats-dump-interval        0
> diagnostics.fop-sample-interval        0
>

Re: [Gluster-users] What is TCP/4007 for ?

2019-07-03 Thread nico

Thanks for reply, I can show this usage with lsof on some 3.12.15 clients :

# ps -ef | awk '/[g]lusterfs/ {print $2}' | xargs -n 1 lsof -p | grep -w 4007
glusterfs 28011 root7u  IPv4  205739452  0t0  TCP 
traitNetVM1:49148->glusterVMa:4007 (SYN_SENT)
glusterfs 49535 root7u  IPv4  205739406  0t0  TCP 
traitNetVM1:49150->glusterVMa:4007 (SYN_SENT)
glusterfs 92903 root6u  IPv4  205738745  0t0  TCP 
traitNetVM1:49151->glusterVMa:4007 (SYN_SENT)
glusterfs 96465 root6u  IPv4  205739411  0t0  TCP 
traitNetVM1:49149->glusterVMa:4007 (SYN_SENT)

- Mail original -
De: "Amar Tumballi Suryanarayan" 
À: n...@furyweb.fr
Cc: "gluster-users" 
Envoyé: Mercredi 3 Juillet 2019 19:11:07
Objet: Re: [Gluster-users] What is TCP/4007 for ?

I am not aware of port 4007 usage in entire Gluster. Not aware of any dependent 
projects too. 
-Amar 

On Wed, Jul 3, 2019 at 9:44 PM < [ mailto:n...@furyweb.fr | n...@furyweb.fr ] > 
wrote: 


Does anybody have info about this port, is it mandatory, is there any way to 
disable it if so ? 

- Mail original - 
De: [ mailto:n...@furyweb.fr | n...@furyweb.fr ] 
À: "gluster-users" < [ mailto:gluster-users@gluster.org | 
gluster-users@gluster.org ] > 
Envoyé: Jeudi 20 Juin 2019 19:01:40 
Objet: [Gluster-users] What is TCP/4007 for ? 

I have several Gluster clients behind firewalls and opened 
24007:24008,49152:49241 to gluster servers. 

I recently found that TCP/4007 is blocked from clients to servers but found 
this port reference nowhere on google search. 

On server side there's no process listening on TCP/4007 so I would like to 
disable it on client side, maybe a volume setting should be initialized. 

Gluster servers are 5.1 
Volumes are replica 3 with arbiter 
Clients are 3.12.15 (Wheezy) or 4.1.5 (Jessie/Stretch) 

I thank anyone able to give me some information. 

___ 
Gluster-users mailing list 
[ mailto:Gluster-users@gluster.org | Gluster-users@gluster.org ] 
[ https://lists.gluster.org/mailman/listinfo/gluster-users | 
https://lists.gluster.org/mailman/listinfo/gluster-users ] 

___ 
Gluster-users mailing list 
[ mailto:Gluster-users@gluster.org | Gluster-users@gluster.org ] 
[ https://lists.gluster.org/mailman/listinfo/gluster-users | 
https://lists.gluster.org/mailman/listinfo/gluster-users ] 





-- 
Amar Tumballi (amarts) 

___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] What is TCP/4007 for ?

2019-07-03 Thread Amar Tumballi Suryanarayan

I am not aware of port 4007 usage in entire Gluster. Not aware of any
dependent projects too.

-Amar

On Wed, Jul 3, 2019 at 9:44 PM  wrote:

> Does anybody have info about this port, is it mandatory, is there any way
> to disable it if so ?
>
> - Mail original -
> De: n...@furyweb.fr
> À: "gluster-users" 
> Envoyé: Jeudi 20 Juin 2019 19:01:40
> Objet: [Gluster-users] What is TCP/4007 for ?
>
> I have several Gluster clients behind firewalls and opened
> 24007:24008,49152:49241 to gluster servers.
>
> I recently found that TCP/4007 is blocked from clients to servers but
> found this port reference nowhere on google search.
>
> On server side there's no process listening on TCP/4007 so I would like to
> disable it on client side, maybe a volume setting should be initialized.
>
> Gluster servers are 5.1
> Volumes are replica 3 with arbiter
> Clients are 3.12.15 (Wheezy) or 4.1.5 (Jessie/Stretch)
>
> I thank anyone able to give me some information.
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users



-- 
Amar Tumballi (amarts)
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Extremely low performance - am I doing somethingwrong?

2019-07-03 Thread Vladimir Melnik

Thank you, it helped a little:

$ for i in {1..5}; do { dd if=/dev/zero of=/mnt/glusterfs1/test.tmp bs=1M 
count=10 oflag=sync; rm -f /mnt/glusterfs1/test.tmp; } done 2>&1 | grep copied
10485760 bytes (10 MB) copied, 0.738968 s, 14.2 MB/s
10485760 bytes (10 MB) copied, 0.725296 s, 14.5 MB/s
10485760 bytes (10 MB) copied, 0.681508 s, 15.4 MB/s
10485760 bytes (10 MB) copied, 0.85566 s, 12.3 MB/s
10485760 bytes (10 MB) copied, 0.661457 s, 15.9 MB/s

But 14-15 MB/s is still quite far from the actual storage's performance 
(200-3000 MB/s). :-(

Here's full configuration dump (just in case):

Option  Value
--  -
cluster.lookup-unhashed on
cluster.lookup-optimize on
cluster.min-free-disk   10%
cluster.min-free-inodes 5%
cluster.rebalance-stats off
cluster.subvols-per-directory   (null)
cluster.readdir-optimizeoff
cluster.rsync-hash-regex(null)
cluster.extra-hash-regex(null)
cluster.dht-xattr-name  trusted.glusterfs.dht
cluster.randomize-hash-range-by-gfidoff
cluster.rebal-throttle  normal
cluster.lock-migration  off
cluster.force-migration off
cluster.local-volume-name   (null)
cluster.weighted-rebalance  on
cluster.switch-pattern  (null)
cluster.entry-change-logon
cluster.read-subvolume  (null)
cluster.read-subvolume-index-1
cluster.read-hash-mode  1
cluster.background-self-heal-count  8
cluster.metadata-self-heal  off
cluster.data-self-heal  off
cluster.entry-self-heal off
cluster.self-heal-daemonon
cluster.heal-timeout600
cluster.self-heal-window-size   1
cluster.data-change-log on
cluster.metadata-change-log on
cluster.data-self-heal-algorithmfull
cluster.eager-lock  enable
disperse.eager-lock on
disperse.other-eager-lock   on
disperse.eager-lock-timeout 1
disperse.other-eager-lock-timeout   1
cluster.quorum-type auto
cluster.quorum-count(null)
cluster.choose-localoff
cluster.self-heal-readdir-size  1KB
cluster.post-op-delay-secs  1
cluster.ensure-durability   on
cluster.consistent-metadata no
cluster.heal-wait-queue-length  128
cluster.favorite-child-policy   none
cluster.full-lock   yes
diagnostics.latency-measurement off
diagnostics.dump-fd-stats   off
diagnostics.count-fop-hits  off
diagnostics.brick-log-level INFO
diagnostics.client-log-levelINFO
diagnostics.brick-sys-log-level CRITICAL
diagnostics.client-sys-log-levelCRITICAL
diagnostics.brick-logger(null)
diagnostics.client-logger   (null)
diagnostics.brick-log-format(null)
diagnostics.client-log-format   (null)
diagnostics.brick-log-buf-size  5
diagnostics.client-log-buf-size 5
diagnostics.brick-log-flush-timeout 120
diagnostics.client-log-flush-timeout120
diagnostics.stats-dump-interval 0
diagnostics.fop-sample-interval 0
diagnostics.stats-dump-format   json
diagnostics.fop-sample-buf-size 65535
diagnostics.stats-dnscache-ttl-sec  86400
performance.cache-max-file-size 0
performance.cache-min-file-size 0
performance.cache-refresh-timeout   1
performance.cache-priority
performance.cache-size  32MB
performance.io-thread-count 16
performance.high-prio-threads   16
performance.normal-prio-threads 16
performance.low-prio-threads32
performance.least-prio-threads  1
performance.enable-least-priority   on
performance.iot-watchdog-secs   (null)
performance.iot-cleanup-disconnected-reqsoff
performance.iot-pass-throughfalse
performance.io-cache-pass-through   false
performance.cache-size  128MB
performance.qr-cache-timeout1
performance.cache-invalidation  false
performance.ctime-invalidation  false
performance.flush-behindon
performance.nfs.flush-behindon
performance.write-behind-window-size1MB
performance.resync-failed-syncs-after-fsyncoff
performance.nfs.write-behind-window-size1MB
performance.strict-o-direct off
performance.nfs.strict-o-direct off
performance.strict-write-ordering   off
performance.nfs.strict-write-ordering   off
performance.write-behind-trickling-writeson
performance.aggregate-size  128KB
performance.nfs.write-behind-trickling-writeson

Re: [Gluster-users] What is TCP/4007 for ?

2019-07-03 Thread nico

Does anybody have info about this port, is it mandatory, is there any way to 
disable it if so ?

- Mail original -
De: n...@furyweb.fr
À: "gluster-users" 
Envoyé: Jeudi 20 Juin 2019 19:01:40
Objet: [Gluster-users] What is TCP/4007 for ?

I have several Gluster clients behind firewalls and opened 
24007:24008,49152:49241 to gluster servers. 

I recently found that TCP/4007 is blocked from clients to servers but found 
this port reference nowhere on google search. 

On server side there's no process listening on TCP/4007 so I would like to 
disable it on client side, maybe a volume setting should be initialized. 

Gluster servers are 5.1 
Volumes are replica 3 with arbiter 
Clients are 3.12.15 (Wheezy) or 4.1.5 (Jessie/Stretch) 

I thank anyone able to give me some information. 

___ 
Gluster-users mailing list 
Gluster-users@gluster.org 
https://lists.gluster.org/mailman/listinfo/gluster-users 

___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Parallel process hang on gluster volume

2019-07-03 Thread nico

Am I alone having this problem ?

- Mail original -
De: n...@furyweb.fr
À: "gluster-users" 
Envoyé: Vendredi 21 Juin 2019 09:48:47
Objet: [Gluster-users] Parallel process hang on gluster volume

I encounterd an issue on production servers using GlusterFS servers 5.1 and 
clients 4.1.5 when several process write at the same time on a gluster volume. 

With more than 48 process writes on the volume at the same time, they are 
blocked in D state (uninterruptible sleep), I guess some volume settings have 
to be tuned but can't figure out which.

The client is using op-version 40100 on this volume
Below are volume info, volume settings and ps output on blocked processes.

___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Extremely low performance - am I doing something wrong?

2019-07-03 Thread Vladimir Melnik

Just to be exact - here are the results of iperf3 measurements between the 
client and one of servers:

$ iperf3 -c gluster1
Connecting to host gluster1, port 5201
[  4] local 10.13.16.1 port 33156 connected to 10.13.1.16 port 5201
[ ID] Interval   Transfer Bandwidth   Retr  Cwnd
[  4]   0.00-1.00   sec  77.8 MBytes   652 Mbits/sec3337 KBytes
[  4]   1.00-2.00   sec  89.7 MBytes   752 Mbits/sec0505 KBytes
[  4]   2.00-3.00   sec   103 MBytes   862 Mbits/sec2631 KBytes
[  4]   3.00-4.00   sec   104 MBytes   870 Mbits/sec1741 KBytes
[  4]   4.00-5.00   sec  98.8 MBytes   828 Mbits/sec1834 KBytes
[  4]   5.00-6.00   sec   101 MBytes   849 Mbits/sec0923 KBytes
[  4]   6.00-7.00   sec   102 MBytes   860 Mbits/sec0   1005 KBytes
[  4]   7.00-8.00   sec   106 MBytes   890 Mbits/sec0   1.06 MBytes
[  4]   8.00-9.00   sec   109 MBytes   913 Mbits/sec0   1.13 MBytes
[  4]   9.00-10.00  sec   109 MBytes   912 Mbits/sec0   1.20 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval   Transfer Bandwidth   Retr
[  4]   0.00-10.00  sec  1000 MBytes   839 Mbits/sec7 sender
[  4]   0.00-10.00  sec   998 MBytes   837 Mbits/sec  receiver

iperf Done.

$ iperf3 -c gluster1 -R
Connecting to host gluster1, port 5201
Reverse mode, remote host gluster1 is sending
[  4] local 10.13.16.1 port 33160 connected to 10.13.1.16 port 5201
[ ID] Interval   Transfer Bandwidth
[  4]   0.00-1.00   sec  58.8 MBytes   492 Mbits/sec
[  4]   1.00-2.00   sec  80.1 MBytes   673 Mbits/sec
[  4]   2.00-3.00   sec  83.8 MBytes   703 Mbits/sec
[  4]   3.00-4.00   sec  95.6 MBytes   800 Mbits/sec
[  4]   4.00-5.00   sec   102 MBytes   858 Mbits/sec
[  4]   5.00-6.00   sec   101 MBytes   850 Mbits/sec
[  4]   6.00-7.00   sec   102 MBytes   860 Mbits/sec
[  4]   7.00-8.00   sec   107 MBytes   898 Mbits/sec
[  4]   8.00-9.00   sec   106 MBytes   893 Mbits/sec
[  4]   9.00-10.00  sec   108 MBytes   904 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval   Transfer Bandwidth   Retr
[  4]   0.00-10.00  sec   949 MBytes   796 Mbits/sec6 sender
[  4]   0.00-10.00  sec   946 MBytes   794 Mbits/sec  receiver

iperf Done.

So, the bandwidth seems to be OK too.

On Wed, Jul 03, 2019 at 11:39:41AM +0300, Vladimir Melnik wrote:
> Dear colleagues,
> 
> I have a lab with a bunch of virtual machines (the virtualization is
> provided by KVM) running on the same physical host. 4 of these VMs are
> working as a GlusterFS cluster and there's one more VM that works as a
> client. I'll specify all the packages' versions in the ending of this
> message.
> 
> I created 2 volumes - one is having type "Distributed-Replicate" and
> another one is "Distribute". The problem is that both of volumes are
> showing really poor performance.
> 
> Here's what I see on the client:
> $ mount | grep gluster
> 10.13.1.16:storage1 on /mnt/glusterfs1 type 
> fuse.glusterfs(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
> 10.13.1.16:storage2 on /mnt/glusterfs2 type 
> fuse.glusterfs(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
> 
> $ for i in {1..5}; do { dd if=/dev/zero of=/mnt/glusterfs1/test.tmp bs=1M 
> count=10 oflag=sync; rm -f /mnt/glusterfs1/test.tmp; } done
> 10+0 records in
> 10+0 records out
> 10485760 bytes (10 MB) copied, 1.47936 s, 7.1 MB/s
> 10+0 records in
> 10+0 records out
> 10485760 bytes (10 MB) copied, 1.62546 s, 6.5 MB/s
> 10+0 records in
> 10+0 records out
> 10485760 bytes (10 MB) copied, 1.71229 s, 6.1 MB/s
> 10+0 records in
> 10+0 records out
> 10485760 bytes (10 MB) copied, 1.68607 s, 6.2 MB/s
> 10+0 records in
> 10+0 records out
> 10485760 bytes (10 MB) copied, 1.82204 s, 5.8 MB/s
> 
> $ for i in {1..5}; do { dd if=/dev/zero of=/mnt/glusterfs2/test.tmp bs=1M 
> count=10 oflag=sync; rm -f /mnt/glusterfs2/test.tmp; } done
> 10+0 records in
> 10+0 records out
> 10485760 bytes (10 MB) copied, 1.15739 s, 9.1 MB/s
> 10+0 records in
> 10+0 records out
> 10485760 bytes (10 MB) copied, 0.978528 s, 10.7 MB/s
> 10+0 records in
> 10+0 records out
> 10485760 bytes (10 MB) copied, 0.910642 s, 11.5 MB/s
> 10+0 records in
> 10+0 records out
> 10485760 bytes (10 MB) copied, 0.998249 s, 10.5 MB/s
> 10+0 records in
> 10+0 records out
> 10485760 bytes (10 MB) copied, 1.03377 s, 10.1 MB/s
> 
> The distributed one shows a bit better performance than the
> distributed-replicated one, but it's still poor. :-(
> 
> The disk storage itself is OK, here's what I see on each of 4 GlusterFS
> servers:
> for i in {1..5}; do { dd if=/dev/zero of=/mnt/storage1/test.tmp bs=1M 
> count=10 oflag=sync; rm -f /mnt/storage1/test.tmp; } done
> 10+0 records in
> 10+0 records out
> 10485760 bytes (10 MB) copied, 0.0656698 s, 160 MB/s
> 10+0 records in
> 10+0 records out
> 10485760 bytes (10 MB) copied, 0.0476927 s,

[Gluster-users] Extremely low performance - am I doing something wrong?

2019-07-03 Thread Vladimir Melnik

Dear colleagues,

I have a lab with a bunch of virtual machines (the virtualization is
provided by KVM) running on the same physical host. 4 of these VMs are
working as a GlusterFS cluster and there's one more VM that works as a
client. I'll specify all the packages' versions in the ending of this
message.

I created 2 volumes - one is having type "Distributed-Replicate" and
another one is "Distribute". The problem is that both of volumes are
showing really poor performance.

Here's what I see on the client:
$ mount | grep gluster
10.13.1.16:storage1 on /mnt/glusterfs1 type 
fuse.glusterfs(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
10.13.1.16:storage2 on /mnt/glusterfs2 type 
fuse.glusterfs(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)

$ for i in {1..5}; do { dd if=/dev/zero of=/mnt/glusterfs1/test.tmp bs=1M 
count=10 oflag=sync; rm -f /mnt/glusterfs1/test.tmp; } done
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 1.47936 s, 7.1 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 1.62546 s, 6.5 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 1.71229 s, 6.1 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 1.68607 s, 6.2 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 1.82204 s, 5.8 MB/s

$ for i in {1..5}; do { dd if=/dev/zero of=/mnt/glusterfs2/test.tmp bs=1M 
count=10 oflag=sync; rm -f /mnt/glusterfs2/test.tmp; } done
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 1.15739 s, 9.1 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.978528 s, 10.7 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.910642 s, 11.5 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.998249 s, 10.5 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 1.03377 s, 10.1 MB/s

The distributed one shows a bit better performance than the
distributed-replicated one, but it's still poor. :-(

The disk storage itself is OK, here's what I see on each of 4 GlusterFS
servers:
for i in {1..5}; do { dd if=/dev/zero of=/mnt/storage1/test.tmp bs=1M count=10 
oflag=sync; rm -f /mnt/storage1/test.tmp; } done
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.0656698 s, 160 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.0476927 s, 220 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.036526 s, 287 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.0329145 s, 319 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.0403988 s, 260 MB/s

The network between all 5 VMs is OK, they all are working on the same
physical host.

Can't understand, what am I doing wrong. :-(

Here's the detailed info about the volumes:
Volume Name: storage1
Type: Distributed-Replicate
Volume ID: a42e2554-99e5-4331-bcc4-0900d002ae32
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x (2 + 1) = 6
Transport-type: tcp
Bricks:
Brick1: gluster1.k8s.maitre-d.tucha.ua:/mnt/storage1/brick1
Brick2: gluster2.k8s.maitre-d.tucha.ua:/mnt/storage1/brick2
Brick3: gluster3.k8s.maitre-d.tucha.ua:/mnt/storage1/brick_arbiter (arbiter)
Brick4: gluster3.k8s.maitre-d.tucha.ua:/mnt/storage1/brick3
Brick5: gluster4.k8s.maitre-d.tucha.ua:/mnt/storage1/brick4
Brick6: gluster1.k8s.maitre-d.tucha.ua:/mnt/storage1/brick_arbiter (arbiter)
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off

Volume Name: storage2
Type: Distribute
Volume ID: df4d8096-ad03-493e-9e0e-586ce21fb067
Status: Started
Snapshot Count: 0
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: gluster1.k8s.maitre-d.tucha.ua:/mnt/storage2
Brick2: gluster2.k8s.maitre-d.tucha.ua:/mnt/storage2
Brick3: gluster3.k8s.maitre-d.tucha.ua:/mnt/storage2
Brick4: gluster4.k8s.maitre-d.tucha.ua:/mnt/storage2
Options Reconfigured:
transport.address-family: inet
nfs.disable: on

The OS is CentOS Linux release 7.6.1810. The packages I'm using are:
glusterfs-6.3-1.el7.x86_64
glusterfs-api-6.3-1.el7.x86_64
glusterfs-cli-6.3-1.el7.x86_64
glusterfs-client-xlators-6.3-1.el7.x86_64
glusterfs-fuse-6.3-1.el7.x86_64
glusterfs-libs-6.3-1.el7.x86_64
glusterfs-server-6.3-1.el7.x86_64
kernel-3.10.0-327.el7.x86_64
kernel-3.10.0-514.2.2.el7.x86_64
kernel-3.10.0-957.12.1.el7.x86_64
kernel-3.10.0-957.12.2.el7.x86_64
kernel-3.10.0-957.21.3.el7.x86_64
kernel-tools-3.10.0-957.21.3.el7.x86_64
kernel-tools-libs-3.10.0-957.21.3.el7.x86_6

Please, be so kind as to help me to understand, did I do it wrong or
that's quite normal performance of GlusterFS?

Thanks in advance!
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] ls slow on a GlusterFS mounted filesystem

2019-07-03 Thread Milos Cuculovic

Any idea?

- Kindest regards,

Milos Cuculovic
IT Manager

---
MDPI AG
Postfach, CH-4020 Basel, Switzerland
Office: St. Alban-Anlage 66, 4052 Basel, Switzerland
Tel. +41 61 683 77 35
Fax +41 61 302 89 18
Email: cuculo...@mdpi.com
Skype: milos.cuculovic.mdpi

Disclaimer: The information and files contained in this message are 
confidential and intended solely for the use of the individual or entity to 
whom they are addressed. If you have received this message in error, please 
notify me and delete this message from your system. You may not copy this 
message in its entirety or in part, or disclose its contents to anyone.

> On 1 Jul 2019, at 13:52, Milos Cuculovic  wrote:
> 
> We have two replicated GlusterFS servers. When trying to run ls on any of the 
> GlusterFS mounted directories, it takes long time to process.
> For example, on a irectory with 10 files and 10 subdirectories, it takes 5sec 
> to do the ls.
> 
> Here is the volume config info:
> 
> Volume Name: storage2
> Type: Replicate
> Volume ID: 8af504b7-8a95-463b-8578-9a969ae060ff
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x 2 = 2
> Transport-type: tcp
> Bricks:
> Brick1: storage3:/data/data-cluster
> Brick2: storage4:/data/data-cluster
> Options Reconfigured:
> storage.build-pgfid: on
> cluster.readdir-optimize: off
> storage.fips-mode-rchecksum: on
> client.event-threads: 12
> server.event-threads: 12
> nfs.disable: on
> transport.address-family: inet
> auth.allow: 10.10.0.*
> features.cache-invalidation: on
> features.cache-invalidation-timeout: 600
> performance.stat-prefetch: on
> performance.readdir-ahead: on
> performance.cache-refresh-timeout: 1
> network.compression.compression-level: -1
> network.compression: off
> cluster.min-free-disk: 2%
> performance.cache-size: 1GB
> features.bitrot: on
> features.scrub: Active
> performance.client-io-threads: on
> features.scrub-freq: monthly
> features.scrub-throttle: normal
> 
> Any idea on how to improve this?
> 
> Looking forward to hearing from you.
> 
> Milos

___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Extremely low performance - am I doing somethingwrong?

Re: [Gluster-users] Extremely low performance - am I doing somethingwrong?

Re: [Gluster-users] Extremely low performance - am I doing somethingwrong?

Re: [Gluster-users] Extremely low performance - am I doing somethingwrong?

Re: [Gluster-users] Extremely low performance - am I doing somethingwrong?

Re: [Gluster-users] What is the right way to bring down a Glusterfs server for maintenance?

Re: [Gluster-users] What is the right way to bring down a Glusterfs server for maintenance?

[Gluster-users] GlusterFS FUSE client on BSD

[Gluster-users] What is the right way to bring down a Glusterfs server for maintenance?

Re: [Gluster-users] Extremely low performance - am I doing somethingwrong?

Re: [Gluster-users] Extremely low performance - am I doing somethingwrong?

Re: [Gluster-users] What is TCP/4007 for ?

Re: [Gluster-users] What is TCP/4007 for ?

Re: [Gluster-users] Extremely low performance - am I doing somethingwrong?

Re: [Gluster-users] What is TCP/4007 for ?

Re: [Gluster-users] Parallel process hang on gluster volume

Re: [Gluster-users] Extremely low performance - am I doing something wrong?

[Gluster-users] Extremely low performance - am I doing something wrong?

Re: [Gluster-users] ls slow on a GlusterFS mounted filesystem

19 matches

Site Navigation

Mail list logo

Footer information