Re: [Gluster-users] New cluster - first experience

[Pranith Kumar Karampuri]

On Mon, Jul 18, 2016 at 11:58 PM, Gandalf Corvotempesta <
gandalf.corvotempe...@gmail.com> wrote:

> Il 18/07/2016 20:13, Alastair Neil ha scritto:
>
>> It does not seem to me that this is a gluster issue.  I just quickly
>> reviewed the thread and you said that you saw 60 MB/s with plain nfs to the
>> bricks and with gluster and no sharding you got 59 MB/s
>>
>
> That's true but I have to use sharding that kills my transfer rate.
>

What is the shard size you are looking to set?


> Additionally, I would like to optimize as much as i can the current
> network and I'm looking for
> some suggestions by gluster users, as this network is totally dedicated to
> a gluster cluster.
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>



-- 
Pranith
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Gandalf Corvotempesta]

Il 18/07/2016 20:13, Alastair Neil ha scritto:
It does not seem to me that this is a gluster issue.  I just quickly  
reviewed the thread and you said that you saw 60 MB/s with plain nfs 
to the bricks and with gluster and no sharding you got 59 MB/s


That's true but I have to use sharding that kills my transfer rate.
Additionally, I would like to optimize as much as i can the current 
network and I'm looking for
some suggestions by gluster users, as this network is totally dedicated 
to a gluster cluster.

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Alastair Neil]

It does not seem to me that this is a gluster issue.  I just quickly
reviewed the thread and you said that you saw 60 MB/s with plain nfs to the
bricks and with gluster and no sharding you got 59 MB/s

With plain NFS (no gluster involved) i'm getting almost the same
> speed: about 60MB/s
>
> Without sharding:
> # echo 3 > /proc/sys/vm/drop_caches; dd if=/dev/zero of=test bs=1M
> count=1000 conv=fsync
> 1000+0 record dentro
> 1000+0 record fuori
> 1048576000 byte (1,0 GB) copiati, 17,759 s, 59,0 MB/s




On 18 July 2016 at 06:35, Gandalf Corvotempesta <
gandalf.corvotempe...@gmail.com> wrote:

> 2016-07-16 15:07 GMT+02:00 Gandalf Corvotempesta
> :
> > 2016-07-16 15:04 GMT+02:00 Gandalf Corvotempesta
> > :
> >> [ ID] Interval   Transfer Bandwidth
> >> [  3]  0.0-10.0 sec  2.31 GBytes  1.98 Gbits/sec
> >
> > Obviously i did the same test with all gluster server. Speed is always
> > near 2gbit, so, the network is not an issue here.
>
> Any help? I would like to start real-test with virtual machines and
> proxmox before the August holiday.
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Gandalf Corvotempesta]

2016-07-16 15:07 GMT+02:00 Gandalf Corvotempesta
:
> 2016-07-16 15:04 GMT+02:00 Gandalf Corvotempesta
> :
>> [ ID] Interval   Transfer Bandwidth
>> [  3]  0.0-10.0 sec  2.31 GBytes  1.98 Gbits/sec
>
> Obviously i did the same test with all gluster server. Speed is always
> near 2gbit, so, the network is not an issue here.

Any help? I would like to start real-test with virtual machines and
proxmox before the August holiday.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Gandalf Corvotempesta]

2016-07-16 15:04 GMT+02:00 Gandalf Corvotempesta
:
> [ ID] Interval   Transfer Bandwidth
> [  3]  0.0-10.0 sec  2.31 GBytes  1.98 Gbits/sec

Obviously i did the same test with all gluster server. Speed is always
near 2gbit, so, the network is not an issue here.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Gandalf Corvotempesta]

2016-07-15 11:07 GMT+02:00 Gandalf Corvotempesta
:
> I also did some test with balance-alb/balance-tld but i had some issues.
> Please don't think about the bonding, i would like to get maximum
> speed even with a single connection.

This is an iperf test made from the client to one of gluster server,
with bonding "balance-rr" and a single connection

$ iperf -c x

Client connecting to x, TCP port 5001
TCP window size:  325 KByte (default)

[  3] local 77.95.175.121 port 37324 connected with 1.2.3.4 port 5001
[ ID] Interval   Transfer Bandwidth
[  3]  0.0-10.0 sec  2.31 GBytes  1.98 Gbits/sec

as you can see, with a single iper connection and balance-rr i'm able
to reach about 2gbit (i'm using 2 gigabit NIC bonded together)
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Gandalf Corvotempesta]

2016-07-14 17:36 GMT+02:00 Alastair Neil :
> I am not sure if your nics support it but you could try balance-alb (bonding
> mode 6), this does not require special switch support and I have had good
> results with it.  As Lindsey said the switch configuration could be limiting
> the bandwidth between nodes in balance-rr.

I also did some test with balance-alb/balance-tld but i had some issues.
Please don't think about the bonding, i would like to get maximum
speed even with a single connection.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Gandalf Corvotempesta]

Eventi with limited bandwidth should i reach at least 1gbit

In my case I'm not going over 480mbit
Il 14 lug 2016 5:36 PM, "Alastair Neil"  ha scritto:

> I am not sure if your nics support it but you could try balance-alb
> (bonding mode 6), this does not require special switch support and I have
> had good results with it.  As Lindsey said the switch configuration could
> be limiting the bandwidth between nodes in balance-rr.
>
> On 14 July 2016 at 05:21, Gandalf Corvotempesta <
> gandalf.corvotempe...@gmail.com> wrote:
>
>> 2016-07-14 11:19 GMT+02:00, Gandalf Corvotempesta
>> :
>> > Yes, but my iperf test was made with a wrong bonding configuration.
>>
>> Anyway, even with direct NFS mount (not involving gluster) i'm stuck
>> as 60MB/s (480mbit/s)
>> about 50% of available bandwidth with a single nic/connection.
>>
>> Any change to get this cluster faster ?
>> Which speed are you seeing with gluster or nfs ? I would like to
>> archieve the best possible speed before buying more powerful hardware
>> (10Gb switches)
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
>>
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Alastair Neil]

I am not sure if your nics support it but you could try balance-alb
(bonding mode 6), this does not require special switch support and I have
had good results with it.  As Lindsey said the switch configuration could
be limiting the bandwidth between nodes in balance-rr.

On 14 July 2016 at 05:21, Gandalf Corvotempesta <
gandalf.corvotempe...@gmail.com> wrote:

> 2016-07-14 11:19 GMT+02:00, Gandalf Corvotempesta
> :
> > Yes, but my iperf test was made with a wrong bonding configuration.
>
> Anyway, even with direct NFS mount (not involving gluster) i'm stuck
> as 60MB/s (480mbit/s)
> about 50% of available bandwidth with a single nic/connection.
>
> Any change to get this cluster faster ?
> Which speed are you seeing with gluster or nfs ? I would like to
> archieve the best possible speed before buying more powerful hardware
> (10Gb switches)
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Gandalf Corvotempesta]

2016-07-14 11:19 GMT+02:00, Gandalf Corvotempesta
:
> Yes, but my iperf test was made with a wrong bonding configuration.

Anyway, even with direct NFS mount (not involving gluster) i'm stuck
as 60MB/s (480mbit/s)
about 50% of available bandwidth with a single nic/connection.

Any change to get this cluster faster ?
Which speed are you seeing with gluster or nfs ? I would like to
archieve the best possible speed before buying more powerful hardware
(10Gb switches)
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Gandalf Corvotempesta]

2016-07-14 9:44 GMT+02:00, Lindsay Mathieson :
> Depends a lot on the switch and the modes chose (something about mode
> 0 and striping:
>
> http://serverfault.com/questions/26720/how-to-achieve-2-gigabit-total-throughput-on-linux-using-the-bonding-driver?rq=1

Ports are grouped together on the switch.

> Did you say you got 1.7Mbps with iperf between two machines? was that
> going through a switch?

Yes, but my iperf test was made with a wrong bonding configuration.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Lindsay Mathieson]

On 14 July 2016 at 00:40, Gandalf Corvotempesta
 wrote:
> That's not true.
> from the kernel docs:
>
> "balance-rr: This mode is the only mode that will permit a single TCP/IP
> connection to stripe traffic across multiple interfaces. It is therefore the
> only mode that will allow a single TCP/IP stream to utilize more than one
> interface's worth of throughput. "


Depends a lot on the switch and the modes chose (something about mode
0 and striping:

  
http://serverfault.com/questions/26720/how-to-achieve-2-gigabit-total-throughput-on-linux-using-the-bonding-driver?rq=1

Did you say you got 1.7Mbps with iperf between two machines? was that
going through a switch?



-- 
Lindsay
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Gandalf Corvotempesta]

2016-07-13 14:16 GMT+02:00, Pranith Kumar Karampuri :
> If you do a plain nfs mount of the brick without any gluster in picture
> what is the speed you get?

With plain NFS (no gluster involved) i'm getting almost the same
speed: about 60MB/s
This is strange, is less than half of a single gigabit connection (and
I'm using a bonded nic)
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Gandalf Corvotempesta]

Il 13 lug 2016 2:16 PM, "Pranith Kumar Karampuri"  ha
scritto:
> If you do a plain nfs mount of the brick without any gluster in picture
what is the speed you get?
>

I'm out of office now
I've asked to my colleagues to spin up an nfs server and I'll post here the
results
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Pranith Kumar Karampuri]

On Wed, Jul 13, 2016 at 4:47 PM, Gandalf Corvotempesta <
gandalf.corvotempe...@gmail.com> wrote:

> 2016-07-13 10:24 GMT+02:00 Pranith Kumar Karampuri :
> > We are in the process of making shard available for general purpose
> > usecases.
>
> Any ETA?
>

Design hasn't finalized so hard to tell.


>
> Anyway, i did another test. This time the "client" has 4 bonded
> gigabit nic (balance-rr) and each server has 2 bonded gigabit nic
> (balance-rr). (previously i had an issue with bonding on our switch)
>
> With sharding (64MB):
> # echo 3 > /proc/sys/vm/drop_caches; dd if=/dev/zero of=test bs=1M
> count=1000 conv=fsync
> 1000+0 record dentro
> 1000+0 record fuori
> 1048576000 byte (1,0 GB) copiati, 89,2066 s, 11,8 MB/s
>
>
> Without sharding:
> # echo 3 > /proc/sys/vm/drop_caches; dd if=/dev/zero of=test bs=1M
> count=1000 conv=fsync
> 1000+0 record dentro
> 1000+0 record fuori
> 1048576000 byte (1,0 GB) copiati, 17,759 s, 59,0 MB/s
>
> WOW! But still far away from HDD speed (about 120MB/s)
>
>
> With 4 bonded nic, I have 4000gbit of bandwidth.
> 4000/8 = 500MB/s
> With this configuration gluster should be able to write at full speed
> on 3 nodes at once.
> Each node has 2000/8 = 250MB/s of bandwidth, twice the hdd speed.
>
If you do a plain nfs mount of the brick without any gluster in picture
what is the speed you get?


-- 
Pranith
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Gandalf Corvotempesta]

2016-07-13 10:24 GMT+02:00 Pranith Kumar Karampuri :
> We are in the process of making shard available for general purpose
> usecases.

Any ETA?

Anyway, i did another test. This time the "client" has 4 bonded
gigabit nic (balance-rr) and each server has 2 bonded gigabit nic
(balance-rr). (previously i had an issue with bonding on our switch)

With sharding (64MB):
# echo 3 > /proc/sys/vm/drop_caches; dd if=/dev/zero of=test bs=1M
count=1000 conv=fsync
1000+0 record dentro
1000+0 record fuori
1048576000 byte (1,0 GB) copiati, 89,2066 s, 11,8 MB/s


Without sharding:
# echo 3 > /proc/sys/vm/drop_caches; dd if=/dev/zero of=test bs=1M
count=1000 conv=fsync
1000+0 record dentro
1000+0 record fuori
1048576000 byte (1,0 GB) copiati, 17,759 s, 59,0 MB/s

WOW! But still far away from HDD speed (about 120MB/s)


With 4 bonded nic, I have 4000gbit of bandwidth.
4000/8 = 500MB/s
With this configuration gluster should be able to write at full speed
on 3 nodes at once.
Each node has 2000/8 = 250MB/s of bandwidth, twice the hdd speed.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Pranith Kumar Karampuri]

We are in the process of making shard available for general purpose
usecases.

On Wed, Jul 13, 2016 at 12:14 PM, Gandalf Corvotempesta <
gandalf.corvotempe...@gmail.com> wrote:

> Il 13 lug 2016 05:39, "Pranith Kumar Karampuri" :
> > May be you gave very small shard size(default is 4MB) which makes the
> workload small-file workload? Can you try with 512MB or something and see?
> >
>
> volume configuration was posted previously
> I'm trying with 64MB but i didn't see any appreciable speed gain moving
> from 4mb to 64mb
>



-- 
Pranith
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Gandalf Corvotempesta]

Il 13 lug 2016 05:39, "Pranith Kumar Karampuri" :
> May be you gave very small shard size(default is 4MB) which makes the
workload small-file workload? Can you try with 512MB or something and see?
>

volume configuration was posted previously
I'm trying with 64MB but i didn't see any appreciable speed gain moving
from 4mb to 64mb
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Pranith Kumar Karampuri]

On Tue, Jul 12, 2016 at 10:29 PM, Gandalf Corvotempesta <
gandalf.corvotempe...@gmail.com> wrote:

> 2016-07-12 18:15 GMT+02:00, Pranith Kumar Karampuri :
> > Will it be possible to get this profile output for a volume without
> > sharding enabled? If it still doesn't look like the one I gave in the
> mail
> > before then we have some debugging to do to find why there are extra
> > operations we are seeing.
>
> Very strange. With shard disabled speed boost to 33MB/s
>
> I think there are some issues with sharding.
> Do you still need te profile output with shard disabled?
> I think now we have to troubleshoot why sharding is very very slow,
> almost unusable
>

May be you gave very small shard size(default is 4MB) which makes the
workload small-file workload? Can you try with 512MB or something and see?

-- 
Pranith
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Gandalf Corvotempesta]

2016-07-12 18:15 GMT+02:00, Pranith Kumar Karampuri :
> Will it be possible to get this profile output for a volume without
> sharding enabled? If it still doesn't look like the one I gave in the mail
> before then we have some debugging to do to find why there are extra
> operations we are seeing.

Very strange. With shard disabled speed boost to 33MB/s

I think there are some issues with sharding.
Do you still need te profile output with shard disabled?
I think now we have to troubleshoot why sharding is very very slow,
almost unusable
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Pranith Kumar Karampuri]

Will it be possible to get this profile output for a volume without
sharding enabled? If it still doesn't look like the one I gave in the mail
before then we have some debugging to do to find why there are extra
operations we are seeing.

On Tue, Jul 12, 2016 at 9:27 PM, Gandalf Corvotempesta <
gandalf.corvotempe...@gmail.com> wrote:

> Shard was enabled during this test that previously i did other tests with
> shard disabled and speed doesn't change at all.
> Il 12 lug 2016 5:17 PM, "Pranith Kumar Karampuri" 
> ha scritto:
>
>> You got this for single dd workload? Ideally single file dd workload
>> should be dominated by 'WRITE' operation, but seems like it is dominated by
>> too many FINODELK, I see quite a few mknods too. What is puzzling is the
>> number of ENTRYLKs which is of the order of 10k. I see some discussion
>> about enabling sharding, Did you enable sharding by anychance on this
>> volume? Sharding is not yet ready for general purpose workloads. As long as
>> you have single writer workload it is fine. It is very well tested for VM
>> workload.
>>
>>
>> dd workload generally looks like this (dd if=/dev/zero of=a.txt bs=1M
>> count=1000):
>>
>>
>> Brick: localhost.localdomain:/home/gfs/r2_0
>> ---
>> Cumulative Stats:
>>Block Size: 131072b+  262144b+
>>  No. of Reads:0 0
>> No. of Writes: 7996 2
>>  %-latency   Avg-latency   Min-Latency   Max-Latency   No. of
>> calls Fop
>>  -   ---   ---   ---
>> 
>>   0.00   0.00 us   0.00 us   0.00 us  1
>> RELEASE
>>   0.00   0.00 us   0.00 us   0.00 us  3
>> RELEASEDIR
>>   0.00  24.00 us  24.00 us  24.00 us  1
>> STATFS
>>   0.00  22.50 us  22.00 us  23.00 us  2
>> ENTRYLK
>>   0.00  28.00 us  27.00 us  29.00 us  2
>> FINODELK
>>   0.00  67.00 us  67.00 us  67.00 us  1
>> GETXATTR
>>   0.00  35.50 us  28.00 us  43.00 us  2
>> FLUSH
>>   0.01 342.00 us 342.00 us 342.00 us  1
>> CREATE
>>   0.10 134.61 us  54.00 us 379.00 us 18
>> FXATTROP
>>   0.23  67.71 us  41.00 us 156.00 us 83
>> LOOKUP
>>  99.65 307.53 us  61.00 us   50633.00 us   7998
>> WRITE
>>
>>
>> On Tue, Jul 12, 2016 at 8:16 PM, Gandalf Corvotempesta <
>> gandalf.corvotempe...@gmail.com> wrote:
>>
>>> 2016-07-12 15:55 GMT+02:00 Pranith Kumar Karampuri >> >:
>>> > Could you do the following?
>>> >
>>> > # gluster volume profile  start
>>> > # run dd command
>>> > # gluster volume profile  info >
>>> > /path/to/file/that/you/need/to/send/us.txt
>>>
>>> http://pastebin.com/raw/wcA0i335
>>>
>>
>>
>>
>> --
>> Pranith
>>
>


-- 
Pranith
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Gandalf Corvotempesta]

Shard was enabled during this test that previously i did other tests with
shard disabled and speed doesn't change at all.
Il 12 lug 2016 5:17 PM, "Pranith Kumar Karampuri"  ha
scritto:

> You got this for single dd workload? Ideally single file dd workload
> should be dominated by 'WRITE' operation, but seems like it is dominated by
> too many FINODELK, I see quite a few mknods too. What is puzzling is the
> number of ENTRYLKs which is of the order of 10k. I see some discussion
> about enabling sharding, Did you enable sharding by anychance on this
> volume? Sharding is not yet ready for general purpose workloads. As long as
> you have single writer workload it is fine. It is very well tested for VM
> workload.
>
>
> dd workload generally looks like this (dd if=/dev/zero of=a.txt bs=1M
> count=1000):
>
>
> Brick: localhost.localdomain:/home/gfs/r2_0
> ---
> Cumulative Stats:
>Block Size: 131072b+  262144b+
>  No. of Reads:0 0
> No. of Writes: 7996 2
>  %-latency   Avg-latency   Min-Latency   Max-Latency   No. of
> calls Fop
>  -   ---   ---   ---   
> 
>   0.00   0.00 us   0.00 us   0.00 us  1
> RELEASE
>   0.00   0.00 us   0.00 us   0.00 us  3
> RELEASEDIR
>   0.00  24.00 us  24.00 us  24.00 us  1
> STATFS
>   0.00  22.50 us  22.00 us  23.00 us  2
> ENTRYLK
>   0.00  28.00 us  27.00 us  29.00 us  2
> FINODELK
>   0.00  67.00 us  67.00 us  67.00 us  1
> GETXATTR
>   0.00  35.50 us  28.00 us  43.00 us  2
> FLUSH
>   0.01 342.00 us 342.00 us 342.00 us  1
> CREATE
>   0.10 134.61 us  54.00 us 379.00 us 18
> FXATTROP
>   0.23  67.71 us  41.00 us 156.00 us 83
> LOOKUP
>  99.65 307.53 us  61.00 us   50633.00 us   7998
> WRITE
>
>
> On Tue, Jul 12, 2016 at 8:16 PM, Gandalf Corvotempesta <
> gandalf.corvotempe...@gmail.com> wrote:
>
>> 2016-07-12 15:55 GMT+02:00 Pranith Kumar Karampuri :
>> > Could you do the following?
>> >
>> > # gluster volume profile  start
>> > # run dd command
>> > # gluster volume profile  info >
>> > /path/to/file/that/you/need/to/send/us.txt
>>
>> http://pastebin.com/raw/wcA0i335
>>
>
>
>
> --
> Pranith
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Pranith Kumar Karampuri]

You got this for single dd workload? Ideally single file dd workload should
be dominated by 'WRITE' operation, but seems like it is dominated by too
many FINODELK, I see quite a few mknods too. What is puzzling is the number
of ENTRYLKs which is of the order of 10k. I see some discussion about
enabling sharding, Did you enable sharding by anychance on this volume?
Sharding is not yet ready for general purpose workloads. As long as you
have single writer workload it is fine. It is very well tested for VM
workload.


dd workload generally looks like this (dd if=/dev/zero of=a.txt bs=1M
count=1000):


Brick: localhost.localdomain:/home/gfs/r2_0
---
Cumulative Stats:
   Block Size: 131072b+  262144b+
 No. of Reads:0 0
No. of Writes: 7996 2
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 -   ---   ---   ---   

  0.00   0.00 us   0.00 us   0.00 us  1
RELEASE
  0.00   0.00 us   0.00 us   0.00 us  3
RELEASEDIR
  0.00  24.00 us  24.00 us  24.00 us  1
STATFS
  0.00  22.50 us  22.00 us  23.00 us  2
ENTRYLK
  0.00  28.00 us  27.00 us  29.00 us  2
FINODELK
  0.00  67.00 us  67.00 us  67.00 us  1
GETXATTR
  0.00  35.50 us  28.00 us  43.00 us  2
FLUSH
  0.01 342.00 us 342.00 us 342.00 us  1
CREATE
  0.10 134.61 us  54.00 us 379.00 us 18
FXATTROP
  0.23  67.71 us  41.00 us 156.00 us 83
LOOKUP
 99.65 307.53 us  61.00 us   50633.00 us   7998
WRITE


On Tue, Jul 12, 2016 at 8:16 PM, Gandalf Corvotempesta <
gandalf.corvotempe...@gmail.com> wrote:

> 2016-07-12 15:55 GMT+02:00 Pranith Kumar Karampuri :
> > Could you do the following?
> >
> > # gluster volume profile  start
> > # run dd command
> > # gluster volume profile  info >
> > /path/to/file/that/you/need/to/send/us.txt
>
> http://pastebin.com/raw/wcA0i335
>



-- 
Pranith
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Gandalf Corvotempesta]

2016-07-12 15:55 GMT+02:00 Pranith Kumar Karampuri :
> Could you do the following?
>
> # gluster volume profile  start
> # run dd command
> # gluster volume profile  info >
> /path/to/file/that/you/need/to/send/us.txt

http://pastebin.com/raw/wcA0i335
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Pranith Kumar Karampuri]

Could you do the following?

# gluster volume profile  start
# run dd command
# gluster volume profile  info >
/path/to/file/that/you/need/to/send/us.txt

On Tue, Jul 12, 2016 at 5:33 PM, Gandalf Corvotempesta <
gandalf.corvotempe...@gmail.com> wrote:

> 2016-07-12 14:02 GMT+02:00 Gandalf Corvotempesta
> :
> > As wrote, currently i'm not using bonding but a single gigabit
> > connection and i'm still
> > unable to go over 1/4 of the theoretical speed.
> >
> > 1000/8/3 = 41MB/s
> >
> > I'm stuck at 10MB/s
>
> Just to clarify: any other protocol, like rsync, is able to saturate
> the network.
> *only* gluster can't go over 10MB/s
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>



-- 
Pranith
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Gandalf Corvotempesta]

2016-07-12 13:36 GMT+02:00 David Gossage :
> Did you try by chance running 3 transfers at once from one server to all 3
> nodes outside of gluster?  Maybe bonding isn't picking alternate routes like
> it would be expected.

As wrote, currently i'm not using bonding but a single gigabit
connection and i'm still
unable to go over 1/4 of the theoretical speed.

1000/8/3 = 41MB/s

I'm stuck at 10MB/s
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Gandalf Corvotempesta]

2016-07-12 14:02 GMT+02:00 Gandalf Corvotempesta
:
> As wrote, currently i'm not using bonding but a single gigabit
> connection and i'm still
> unable to go over 1/4 of the theoretical speed.
>
> 1000/8/3 = 41MB/s
>
> I'm stuck at 10MB/s

Just to clarify: any other protocol, like rsync, is able to saturate
the network.
*only* gluster can't go over 10MB/s
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[David Gossage]

On Tue, Jul 12, 2016 at 5:28 AM, Gandalf Corvotempesta <
gandalf.corvotempe...@gmail.com> wrote:

> 2016-07-11 19:31 GMT+02:00 Gandalf Corvotempesta
> :
> > Each disks on each node is able to saturate the network, so I would
> > expect about 950mbit when writing in parallel to 3 nodes. I'm reaching
> > 1/4 of available speed.
>
> I did more tests, even by transfering files with rsync i'm able to
> saturate the network.
> The only slow thing is gluster.
>

Did you try by chance running 3 transfers at once from one server to all 3
nodes outside of gluster?  Maybe bonding isn't picking alternate routes
like it would be expected.

> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Gandalf Corvotempesta]

2016-07-11 19:31 GMT+02:00 Gandalf Corvotempesta
:
> Each disks on each node is able to saturate the network, so I would
> expect about 950mbit when writing in parallel to 3 nodes. I'm reaching
> 1/4 of available speed.

I did more tests, even by transfering files with rsync i'm able to
saturate the network.
The only slow thing is gluster.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Gandalf Corvotempesta]

2016-07-11 19:23 GMT+02:00, Gandalf Corvotempesta
:
> 2016-07-11 18:52 GMT+02:00, Alastair Neil :
>> what performance do you see with the dd directly to the bricks?
>
> # echo 3 > /proc/sys/vm/drop_caches; dd if=/dev/zero of=test bs=1M
> count=1000 conv=fsync
> 1000+0 records in
> 1000+0 records out
> 1048576000 bytes (1.0 GB) copied, 8.20727 s, 128 MB/s
>
> # echo 3 > /proc/sys/vm/drop_caches; dd if=/dev/zero of=test bs=1M
> count=1000 conv=fsync
> 1000+0 records in
> 1000+0 records out
> 1048576000 bytes (1.0 GB) copied, 8.24893 s, 127 MB/s
>
> # echo 3 > /proc/sys/vm/drop_caches; dd if=/dev/zero of=test bs=1M
> count=1000 conv=fsync
> 1000+0 records in
> 1000+0 records out
> 1048576000 bytes (1.0 GB) copied, 8.64035 s, 121 MB/s
>
> # echo 3 > /proc/sys/vm/drop_caches; dd if=/dev/zero of=test bs=1M
> count=1000 conv=fsync
> 1000+0 records in
> 1000+0 records out
> 1048576000 bytes (1.0 GB) copied, 8.29078 s, 126 MB/s
>
> # echo 3 > /proc/sys/vm/drop_caches; dd if=/dev/zero of=test bs=1M
> count=1000 conv=fsync
> 1000+0 records in
> 1000+0 records out
> 1048576000 bytes (1.0 GB) copied, 8.28962 s, 126 MB/s
>


This is the same but from the client, using the gluster volume.

# echo 3 > /proc/sys/vm/drop_caches; dd if=/dev/zero of=test bs=1M
count=1000 conv=fsync
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 111.786 s, 9.4 MB/s

9.4MB/s = 75mbit

As I'm using replica 3, 75mbit*3 = 225mbit of aggregated bandwidth.

I think it's too low on a gigabit network.
Each disks on each node is able to saturate the network, so I would
expect about 950mbit when writing in parallel to 3 nodes. I'm reaching
1/4 of available speed.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Gandalf Corvotempesta]

2016-07-11 18:52 GMT+02:00, Alastair Neil :
> what performance do you see with the dd directly to the bricks?

# echo 3 > /proc/sys/vm/drop_caches; dd if=/dev/zero of=test bs=1M
count=1000 conv=fsync
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 8.20727 s, 128 MB/s

# echo 3 > /proc/sys/vm/drop_caches; dd if=/dev/zero of=test bs=1M
count=1000 conv=fsync
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 8.24893 s, 127 MB/s

# echo 3 > /proc/sys/vm/drop_caches; dd if=/dev/zero of=test bs=1M
count=1000 conv=fsync
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 8.64035 s, 121 MB/s

# echo 3 > /proc/sys/vm/drop_caches; dd if=/dev/zero of=test bs=1M
count=1000 conv=fsync
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 8.29078 s, 126 MB/s

# echo 3 > /proc/sys/vm/drop_caches; dd if=/dev/zero of=test bs=1M
count=1000 conv=fsync
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 8.28962 s, 126 MB/s
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Gandalf Corvotempesta]

Il 11 lug 2016 3:43 PM, "Lindsay Mathieson" 
ha scritto:
> With Replica 3, your client will be writing to 3 servers simultaneously
over a single 1Gb connection, resulting in a maximum write bandwidth of
33Mbs.
>
>
> In your earlier testing, if bonding wasn't enabled on the switch - which
I believe only supports LACP, then I think you would again be limited to a
single 1Gb connection shared over 3 servers. But I'm not an expert on
bonding interfaces.

Ok but I'm still unable to reach more than 15/18mbytes

Even with bonding speed doesn't change
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Lindsay Mathieson]

On 11/07/2016 10:33 PM, Gandalf Corvotempesta wrote:

Just to keep things simple, i've removed the bonding and I'm using a
single gigabit connection on each server.


With Replica 3, your client will be writing to 3 servers simultaneously 
over a single 1Gb connection, resulting in a maximum write bandwidth of 
33Mbs.



In your earlier testing, if bonding wasn't enabled on the switch - which 
I believe only supports LACP, then I think you would again be limited to 
a single 1Gb connection shared over 3 servers. But I'm not an expert on 
bonding interfaces.


--
Lindsay Mathieson

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Gandalf Corvotempesta]

2016-07-09 5:03 GMT+02:00 Lindsay Mathieson :
> Whats your network topology? servers are connected via a switch? what brand
> /model is the switch? does it have bonding setup?

4 servers: 3 gluster storage nodes and 1 gluster client
All of them connected to a single SMC gigabit TigerSwitch

> Also whats the underlying disk setup - models, raid etc.

No RAID, raw disks:

Model Family: Western Digital Se
Device Model: WDC WD2000F9YZ-09N20L0
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Gandalf Corvotempesta]

2016-07-09 7:05 GMT+02:00 Alastair Neil :
> did you test both nics in the bond separately on all the servers and client?

Just to keep things simple, i've removed the bonding and I'm using a
single gigabit connection on each server.

These are some iperf results, made from the only client connecting to
this gluster cluster (via fuse/gluster):

# for i in 1.2.3.112 1.2.3.113 1.2.3.114; do iperf -c $i; done

Client connecting to 1.2.3.112, TCP port 5001
TCP window size: 85.0 KByte (default)

[  3] local 1.2.3.121 port 34405 connected with 1.2.3.112 port 5001
[ ID] Interval   Transfer Bandwidth
[  3]  0.0-10.0 sec  1.10 GBytes   943 Mbits/sec

Client connecting to 1.2.3.113, TCP port 5001
TCP window size: 85.0 KByte (default)

[  3] local 1.2.3.121 port 51302 connected with 1.2.3.113 port 5001
[ ID] Interval   Transfer Bandwidth
[  3]  0.0-10.0 sec  1.10 GBytes   943 Mbits/sec

Client connecting to 1.2.3.114, TCP port 5001
TCP window size: 85.0 KByte (default)

[  3] local 1.2.3.121 port 35503 connected with 1.2.3.114 port 5001
[ ID] Interval   Transfer Bandwidth
[  3]  0.0-10.0 sec  1.10 GBytes   942 Mbits/sec


Network seems to be fine but i'm still having low performances when
trying to extract the kernel sources or "dd" a huge file.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Lindsay Mathieson]

On 9/07/2016 6:07 AM, Gandalf Corvotempesta wrote:

With balance-rr you should reach 2000gbit as client is writing to 3
server simultaneously, thus, is using different
destinations/connections that are balanced.

You wont see the aggregated speed trying to communicate directly
between 2 hosts, but in this case, there are 4 hosts involved (1
"client", 3 servers) and thus 4 ips


Nope, communications with each server will be limited to 1Gbps max. Its 
just you will be able to write to each server *simultaneously* at 1Gbps.



Whats your network topology? servers are connected via a switch? what 
brand /model is the switch? does it have bonding setup?



Also whats the underlying disk setup - models, raid etc.

--
Lindsay Mathieson

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Gandalf Corvotempesta]

2016-07-08 21:53 GMT+02:00 Alastair Neil :
> Also remember with a single transfer you will not see 2000 gb/s only 1000
> gb/s

With balance-rr you should reach 2000gbit as client is writing to 3
server simultaneously, thus, is using different
destinations/connections that are balanced.

You wont see the aggregated speed trying to communicate directly
between 2 hosts, but in this case, there are 4 hosts involved (1
"client", 3 servers) and thus 4 ips
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Alastair Neil]

Also remember with a single transfer you will not see 2000 gb/s only 1000
gb/s

On 8 July 2016 at 15:14, Gandalf Corvotempesta <
gandalf.corvotempe...@gmail.com> wrote:

> 2016-07-08 20:43 GMT+02:00  :
> > Gluster, and in particular the fuse mounter, do not operate on small
> file workloads anywhere near wire speed in their current arch.
>
> I know that i'll unable to reach wire speed, but with 2000gbit
> available, reaching only 88mbit with 1GB file is really low.
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Gandalf Corvotempesta]

2016-07-08 20:43 GMT+02:00  :
> Gluster, and in particular the fuse mounter, do not operate on small file 
> workloads anywhere near wire speed in their current arch.

I know that i'll unable to reach wire speed, but with 2000gbit
available, reaching only 88mbit with 1GB file is really low.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Gandalf Corvotempesta]

2016-07-08 20:35 GMT+02:00 Joe Julian :
> Assuming this dd was run from one of the servers to a replica 3 volume, you
> have one localhost write and two network writes for 88 Mbit/s which looks
> like the maxing out of a 100Mbit connection. That is so coincidental it
> would lead me to look at the network.

Exactly but with iperf i'm able to reach about 1.70gbit on a dual gigabit bond.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Joe Julian]

Assuming this dd was run from one of the servers to a replica 3 volume, 
you have one localhost write and two network writes for 88 Mbit/s which 
looks like the maxing out of a 100Mbit connection. That is so 
coincidental it would lead me to look at the network.



On 07/08/2016 11:23 AM, Gandalf Corvotempesta wrote:

2016-07-08 10:55 GMT+02:00 Gandalf Corvotempesta
:

Now i'm using a bonded gigabit (2x1GB) on every server but i'm still
stuck at about 15-30mbit/s when extracting the Linux kernel. Total
extraction is still about 10minutes.
Something strange is going on, on a dual gigabit connection (2000mbit)
I'm expecting to reach hundreds of megabit (200-300), not tens (15-20)

# dd if=/dev/zero of=/mnt/glusterfs/zero4 bs=1M count=1000
1000+0 record dentro
1000+0 record fuori
1048576000 byte (1,0 GB) copiati, 88,2981 s, 11,9 MB/s

No suggestions?
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Gandalf Corvotempesta]

2016-07-08 10:55 GMT+02:00 Gandalf Corvotempesta
:
> Now i'm using a bonded gigabit (2x1GB) on every server but i'm still
> stuck at about 15-30mbit/s when extracting the Linux kernel. Total
> extraction is still about 10minutes.
> Something strange is going on, on a dual gigabit connection (2000mbit)
> I'm expecting to reach hundreds of megabit (200-300), not tens (15-20)
>
> # dd if=/dev/zero of=/mnt/glusterfs/zero4 bs=1M count=1000
> 1000+0 record dentro
> 1000+0 record fuori
> 1048576000 byte (1,0 GB) copiati, 88,2981 s, 11,9 MB/s

No suggestions?
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Gandalf Corvotempesta]

2016-07-08 10:55 GMT+02:00 Gandalf Corvotempesta
:
> # gluster volume info gv0
>
> Volume Name: gv0
> Type: Replicate
> Volume ID: 2a36dc0f-1d9b-469c-82de-9d8d98321b83
> Status: Started
> Number of Bricks: 1 x 3 = 3
> Transport-type: tcp
> Bricks:
> Brick1: 77.95.175.112:/export/sdb1/brick
> Brick2: 77.95.175.113:/export/sdb1/brick
> Brick3: 77.95.175.114:/export/sdb1/brick
> Options Reconfigured:
> performance.cache-size: 2GB
> performance.write-behind-window-size: 2GB
> features.shard-block-size: 64MB
> features.shard: on
> transport.address-family: inet
> performance.readdir-ahead: on
> nfs.disable: on


Bond configuration on each server:

# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: load balancing (round-robin)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 200
Down Delay (ms): 200

Slave Interface: eth0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:25:90:c8:a0:6c
Slave queue ID: 0

Slave Interface: eth1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:25:90:c8:a0:6d
Slave queue ID: 0
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Gandalf Corvotempesta]

2016-07-07 14:04 GMT+02:00 Lindsay Mathieson :
> The default shard size is 4MB, I'd tend towards a larger one which improves
> write speed. For my VM cluster I use shardsize = 64MB
>
> nb. To change the shardsize you should recreate the volume.

I did more test.
Is not a shard issue, with shard enabled:

# dd if=/dev/zero of=/mnt/glusterfs/zero1 bs=1M count=1000
1000+0 record dentro
1000+0 record fuori
1048576000 byte (1,0 GB) copiati, 93,8803 s, 11,2 MB/s

With shard disabled:

# dd if=/dev/zero of=/mnt/glusterfs/zero2 bs=1M count=1000
1000+0 record dentro
1000+0 record fuori
1048576000 byte (1,0 GB) copiati, 100,051 s, 10,5 MB/s

mostly the same.



# iperf -c 1.2.3.4

Client connecting to 1.2.3.4, TCP port 5001
TCP window size: 85.0 KByte (default)

[  3] local 1.2.3.5 port 34357 connected with 1.2.3.4 port 5001
[ ID] Interval   Transfer Bandwidth
[  3]  0.0-10.0 sec  1.10 GBytes   943 Mbits/sec
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Gandalf Corvotempesta]

2016-07-07 15:19 GMT+02:00 Lindsay Mathieson :
> What about existing files?

Untouched. Still at the previous shard size
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Lindsay Mathieson]

On 7/07/2016 10:20 PM, Gandalf Corvotempesta wrote:

There is no need to recreate the volume.
Shard size is an option to set, not a parameter during the volume creation.

I did the following:
# gluster volume set dis-rep features.shard-block-size 64MB

and shard size changed properly. Then I've created a 1GB file and it
was sharded in 15 chunks


What about existing files?

--
Lindsay Mathieson

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Gandalf Corvotempesta]

2016-07-07 14:04 GMT+02:00 Lindsay Mathieson :
> The default shard size is 4MB, I'd tend towards a larger one which improves
> write speed. For my VM cluster I use shardsize = 64MB

Whis the following:

# gluster volume info gv0

Volume Name: gv0
Type: Replicate
Volume ID: 2a36dc0f-1d9b-469c-82de-9d8d98321b83
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 77.95.175.112:/export/sdb1/brick
Brick2: 77.95.175.113:/export/sdb1/brick
Brick3: 77.95.175.114:/export/sdb1/brick
Options Reconfigured:
features.shard-block-size: 64MB
features.shard: on
transport.address-family: inet
performance.readdir-ahead: on
nfs.disable: on

i'm still stuck at 10m35.507s

I only have 1GB connection, thus 1GB/3 should be the maximum speed.
1000/3/8 = about 40MB/s

# du -sh linux-4.7-rc6
632M linux-4.7-rc6

632M / 40 = 15.8 seconds.

It should take 15.8 seconds to extract.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Kevin Lemonnier]

On Thu, Jul 07, 2016 at 02:20:03PM +0200, Gandalf Corvotempesta wrote:
> 2016-07-07 14:04 GMT+02:00 Lindsay Mathieson :
> > nb. To change the shardsize you should recreate the volume.
> 
> There is no need to recreate the volume.
> Shard size is an option to set, not a parameter during the volume creation.


It is, but existing shards won't be touched : You'll need to move the file out 
and back
in to apply the new shard size.

-- 
Kevin Lemonnier
PGP Fingerprint : 89A5 2283 04A0 E6E9 0111


signature.asc
Description: Digital signature
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Gandalf Corvotempesta]

2016-07-07 14:04 GMT+02:00 Lindsay Mathieson :
> nb. To change the shardsize you should recreate the volume.

There is no need to recreate the volume.
Shard size is an option to set, not a parameter during the volume creation.

I did the following:
# gluster volume set dis-rep features.shard-block-size 64MB

and shard size changed properly. Then I've created a 1GB file and it
was sharded in 15 chunks
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Lindsay Mathieson]

On 7/07/2016 9:52 PM, Gandalf Corvotempesta wrote:

Volume Name: gv0
Type: Replicate
Volume ID: 2a36dc0f-1d9b-469c-82de-9d8d98321b83
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 77.95.175.112:/export/sdb1/brick
Brick2: 77.95.175.113:/export/sdb1/brick
Brick3: 77.95.175.114:/export/sdb1/brick
Options Reconfigured:
features.shard: on
transport.address-family: inet
performance.readdir-ahead: on
nfs.disable: on


The default shard size is 4MB, I'd tend towards a larger one which 
improves write speed. For my VM cluster I use shardsize = 64MB



nb. To change the shardsize you should recreate the volume.

--
Lindsay Mathieson

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Gandalf Corvotempesta]

2016-07-07 13:22 GMT+02:00 Lindsay Mathieson :
> Yes. However maildir involves many tens of thousands of small & large files,
> I *think* that glusters performances isn't the best with very large numbers
> of files in a dir, but hopefully someone else with more experience can chime
> in on that.

Performance for maildir shoul not be much important, I think.
I can also create a VM with the mail server, in this case, on gluster
i'll put the VM image and not the plain maildir, but I prefere the
first solution.

> That does sound slow - how big was the tar file? what is your network speed
> and setup?

I've used the 4.7-rc6 from here: https://www.kernel.org/
Gigabit network

# gluster volume info

Volume Name: gv0
Type: Replicate
Volume ID: 2a36dc0f-1d9b-469c-82de-9d8d98321b83
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 77.95.175.112:/export/sdb1/brick
Brick2: 77.95.175.113:/export/sdb1/brick
Brick3: 77.95.175.114:/export/sdb1/brick
Options Reconfigured:
features.shard: on
transport.address-family: inet
performance.readdir-ahead: on
nfs.disable: on

> Not sure. Disperse Replicated vol maybe?

Is disperse based on erasure code ?
I've read that erasure code store the encoded file, I don't like
to store encoded file, in case of issue, the encoding could lead
to a mess.

> If you're not using a dual or better bonded connection on replica 3 then
> your write speeds will be limited to 1Gb/3 max.

Ok.

> Are your clients on the storage nodes or are they dedicated?

I'm using a dedicated client.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] New cluster - first experience

[Lindsay Mathieson]

On 7/07/2016 8:30 PM, Gandalf Corvotempesta wrote:

1) Can I use the same cluster for both, maildir hosting and VM images hosting?


Yes. However maildir involves many tens of thousands of small & large 
files, I *think* that glusters performances isn't the best with very 
large numbers of files in a dir, but hopefully someone else with more 
experience can chime in on that.




2) a simple kernel extract (with replica 3) is
slw. Direct extract in a brick took 12 seconds.
Extract through gluster (replica 3) took 10m2.214s. In a perfect
world, it should be 3 times the direct extraction (due to the replica)
plus a little bit overhead for the network. Can I optimize this ?


That does sound slow - how big was the tar file? what is your network 
speed and setup?




3) I'm thinking creating a volume for Maildir, with no sharding, and 1
volume for VM hosting in the future (with sharding enabled). Is this
OK?


Yes, sounds like a good idea.



4) I would like to get replica 3 for all, but also the ability to add
1 brick/server per time and not in multiple (in our case, 3 server
every time), like Ceph does. Is this a supported configuration?


Not sure. Disperse Replicated vol maybe?



5) currently i'm testing gluster on a gigabit network (not bonded,
it's just a test). When running in production, i'll move to 10GbE but
currently I would like to get the best performance from the gigabit
network.


If you're not using a dual or better bonded connection on replica 3 then 
your write speeds will be limited to 1Gb/3 max.




some hardware info: currently 2x E5-2603, 16GB RAM on each storage node.


Are your clients on the storage nodes or are they dedicated?


--
Lindsay Mathieson

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] New cluster - first experience

[Gandalf Corvotempesta]

Ok, i'm built my first test cluster with 3.8

Here some questions:

1) Can I use the same cluster for both, maildir hosting and VM images hosting?

2) a simple kernel extract (with replica 3) is
slw. Direct extract in a brick took 12 seconds.
Extract through gluster (replica 3) took 10m2.214s. In a perfect
world, it should be 3 times the direct extraction (due to the replica)
plus a little bit overhead for the network. Can I optimize this ?

3) I'm thinking creating a volume for Maildir, with no sharding, and 1
volume for VM hosting in the future (with sharding enabled). Is this
OK?

4) I would like to get replica 3 for all, but also the ability to add
1 brick/server per time and not in multiple (in our case, 3 server
every time), like Ceph does. Is this a supported configuration?

5) currently i'm testing gluster on a gigabit network (not bonded,
it's just a test). When running in production, i'll move to 10GbE but
currently I would like to get the best performance from the gigabit
network.

some hardware info: currently 2x E5-2603, 16GB RAM on each storage node.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users