Re: [Gluster-users] [External] Re: file metadata operations performance - gluster 4.1

[Raghavendra Gowdappa]

On Thu, Aug 30, 2018 at 8:38 PM, Davide Obbi 
wrote:

> yes "performance.parallel-readdir on and 1x3 replica
>

That's surprising. I thought performance.parallel-readdir will help only
when distribute count is fairly high. This is something worth investigating
further.


> On Thu, Aug 30, 2018 at 5:00 PM Raghavendra Gowdappa 
> wrote:
>
>>
>>
>> On Thu, Aug 30, 2018 at 8:08 PM, Davide Obbi 
>> wrote:
>>
>>> Thanks Amar,
>>>
>>> i have enabled the negative lookups cache on the volume:
>>>
>>
I think enabling nl-cache-positive-entry might help for untarring or git
clone into glusterfs. Its disabled by default. can you let us know the
results?

Option: performance.nl-cache-positive-entry
Default Value: (null)
Description: enable/disable storing of entries that were lookedup and found
to be present in the volume, thus lookup on non existent file is served
from the cache


>>> To deflate a tar archive (not compressed) of 1.3GB it takes aprox 9mins
>>> which can be considered a slight improvement from the previous 12-15
>>> however still not fast enough compared to local disk. The tar is present on
>>> the gluster share/volume and deflated inside the same folder structure.
>>>
>>
>> I am assuming this is with parallel-readdir enabled, right?
>>
>>
>>> Running the operation twice (without removing the already deflated
>>> files) also did not reduce the time spent.
>>>
>>> Running the operation with the tar archive on local disk made no
>>> difference
>>>
>>> What really made a huge difference while git cloning was setting
>>> "performance.parallel-readdir on". During the phase "Receiving objects" ,
>>> as i enabled the xlator it bumped up from 3/4MBs to 27MBs
>>>
>>
>> What is the distribute count? Is it 1x3 replica?
>>
>>
>>> So in conclusion i'm trying to make the untar operation working at an
>>> acceptable level, not expecting local disks speed but at least being within
>>> the 4mins
>>>
>>> I have attached the profiles collected at the end of the untar
>>> operations with the archive on the mount and outside
>>>
>>> thanks
>>> Davide
>>>
>>>
>>> On Tue, Aug 28, 2018 at 8:41 AM Amar Tumballi 
>>> wrote:
>>>
 One of the observation we had with git clone like work load was,
 nl-cache (negative-lookup cache), helps here.

 Try 'gluster volume set $volume-name nl-cache enable'.

 Also sharing the 'profile info' during this performance observations
 also helps us to narrow down the situation.

 More on how to capture profile info @ https://hackmd.io/
 PhhT5jPdQIKxzfeLQmnjJQ?view

 -Amar


 On Thu, Aug 23, 2018 at 7:11 PM, Davide Obbi 
 wrote:

> Hello,
>
> did anyone ever managed to achieve reasonable waiting time while
> performing metadata intensive operations such as git clone, untar etc...?
> Is this possible workload or will never be in scope for glusterfs?
>
> I'd like to know, if possible, what would be the options that affect
> such volume performances.
> Albeit i managed to achieve decent git status/git grep operations, 3
> and 30 secs, the git clone and untarring a file from/to the same share 
> take
> ages. for a git repo of aprox 6GB.
>
> I'm running a test environment with 3 way replica 128GB RAM and 24
> cores are  2.40GHz, one internal SSD dedicated to the volume brick and 
> 10Gb
> network
>
> The options set so far that affects volume performances are:
>  48 performance.readdir-ahead: on
>  49 features.cache-invalidation-timeout: 600
>  50 features.cache-invalidation: on
>  51 performance.md-cache-timeout: 600
>  52 performance.stat-prefetch: on
>  53 performance.cache-invalidation: on
>  54 performance.parallel-readdir: on
>  55 network.inode-lru-limit: 90
>  56 performance.io-thread-count: 32
>  57 performance.cache-size: 10GB
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>



 --
 Amar Tumballi (amarts)

>>>
>>>
>>> --
>>> Davide Obbi
>>> System Administrator
>>>
>>> Booking.com B.V.
>>> Vijzelstraat 66
>>> -80
>>> Amsterdam 1017HL Netherlands
>>> Direct +31207031558
>>> [image: Booking.com] 
>>> The world's #1 accommodation site
>>> 43 languages, 198+ offices worldwide, 120,000+ global destinations,
>>> 1,550,000+ room nights booked every day
>>> No booking fees, best price always guaranteed
>>> Subsidiary of Booking Holdings Inc. (NASDAQ: BKNG)
>>>
>>> ___
>>> Gluster-users mailing list
>>> Gluster-users@gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>
>>
>
> --
> Davide Obbi
> System Administrator
>
> Booking.com B.V.
> Vijzelstraat 66
> 

Re: [Gluster-users] [External] Re: file metadata operations performance - gluster 4.1

[Davide Obbi]

yes "performance.parallel-readdir on and 1x3 replica

On Thu, Aug 30, 2018 at 5:00 PM Raghavendra Gowdappa 
wrote:

>
>
> On Thu, Aug 30, 2018 at 8:08 PM, Davide Obbi 
> wrote:
>
>> Thanks Amar,
>>
>> i have enabled the negative lookups cache on the volume:
>>
>> To deflate a tar archive (not compressed) of 1.3GB it takes aprox 9mins
>> which can be considered a slight improvement from the previous 12-15
>> however still not fast enough compared to local disk. The tar is present on
>> the gluster share/volume and deflated inside the same folder structure.
>>
>
> I am assuming this is with parallel-readdir enabled, right?
>
>
>> Running the operation twice (without removing the already deflated files)
>> also did not reduce the time spent.
>>
>> Running the operation with the tar archive on local disk made no
>> difference
>>
>> What really made a huge difference while git cloning was setting
>> "performance.parallel-readdir on". During the phase "Receiving objects" ,
>> as i enabled the xlator it bumped up from 3/4MBs to 27MBs
>>
>
> What is the distribute count? Is it 1x3 replica?
>
>
>> So in conclusion i'm trying to make the untar operation working at an
>> acceptable level, not expecting local disks speed but at least being within
>> the 4mins
>>
>> I have attached the profiles collected at the end of the untar operations
>> with the archive on the mount and outside
>>
>> thanks
>> Davide
>>
>>
>> On Tue, Aug 28, 2018 at 8:41 AM Amar Tumballi 
>> wrote:
>>
>>> One of the observation we had with git clone like work load was,
>>> nl-cache (negative-lookup cache), helps here.
>>>
>>> Try 'gluster volume set $volume-name nl-cache enable'.
>>>
>>> Also sharing the 'profile info' during this performance observations
>>> also helps us to narrow down the situation.
>>>
>>> More on how to capture profile info @
>>> https://hackmd.io/PhhT5jPdQIKxzfeLQmnjJQ?view
>>>
>>> -Amar
>>>
>>>
>>> On Thu, Aug 23, 2018 at 7:11 PM, Davide Obbi 
>>> wrote:
>>>
 Hello,

 did anyone ever managed to achieve reasonable waiting time while
 performing metadata intensive operations such as git clone, untar etc...?
 Is this possible workload or will never be in scope for glusterfs?

 I'd like to know, if possible, what would be the options that affect
 such volume performances.
 Albeit i managed to achieve decent git status/git grep operations, 3
 and 30 secs, the git clone and untarring a file from/to the same share take
 ages. for a git repo of aprox 6GB.

 I'm running a test environment with 3 way replica 128GB RAM and 24
 cores are  2.40GHz, one internal SSD dedicated to the volume brick and 10Gb
 network

 The options set so far that affects volume performances are:
  48 performance.readdir-ahead: on
  49 features.cache-invalidation-timeout: 600
  50 features.cache-invalidation: on
  51 performance.md-cache-timeout: 600
  52 performance.stat-prefetch: on
  53 performance.cache-invalidation: on
  54 performance.parallel-readdir: on
  55 network.inode-lru-limit: 90
  56 performance.io-thread-count: 32
  57 performance.cache-size: 10GB

 ___
 Gluster-users mailing list
 Gluster-users@gluster.org
 https://lists.gluster.org/mailman/listinfo/gluster-users

>>>
>>>
>>>
>>> --
>>> Amar Tumballi (amarts)
>>>
>>
>>
>> --
>> Davide Obbi
>> System Administrator
>>
>> Booking.com B.V.
>> Vijzelstraat 66-80 Amsterdam 1017HL Netherlands
>> Direct +31207031558
>> [image: Booking.com] 
>> The world's #1 accommodation site
>> 43 languages, 198+ offices worldwide, 120,000+ global destinations,
>> 1,550,000+ room nights booked every day
>> No booking fees, best price always guaranteed
>> Subsidiary of Booking Holdings Inc. (NASDAQ: BKNG)
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
>

-- 
Davide Obbi
System Administrator

Booking.com B.V.
Vijzelstraat 66-80 Amsterdam 1017HL Netherlands
Direct +31207031558
[image: Booking.com] 
The world's #1 accommodation site
43 languages, 198+ offices worldwide, 120,000+ global destinations,
1,550,000+ room nights booked every day
No booking fees, best price always guaranteed
Subsidiary of Booking Holdings Inc. (NASDAQ: BKNG)
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [External] Re: file metadata operations performance - gluster 4.1

[Poornima Gurusiddaiah]

To enable nl-cache please use group option instead of single volume set:

#gluster vol set VOLNAME group nl-cache

This sets few other things including time out, invalidation etc.

For enabling the option Raghavendra mentioned, you ll have to execute it
explicitly, as it's not part of group option yet:

#gluster vol set VOLNAME performance.nl-cache-positive-entry on

Also from the past experience, setting the below option has helped in
performance:

# gluster vol set VOLNAME network.inode-lru-limit 20

Regards,
Poornima


On Thu, Aug 30, 2018, 8:49 PM Raghavendra Gowdappa 
wrote:

>
>
> On Thu, Aug 30, 2018 at 8:38 PM, Davide Obbi 
> wrote:
>
>> yes "performance.parallel-readdir on and 1x3 replica
>>
>
> That's surprising. I thought performance.parallel-readdir will help only
> when distribute count is fairly high. This is something worth investigating
> further.
>
>
>> On Thu, Aug 30, 2018 at 5:00 PM Raghavendra Gowdappa 
>> wrote:
>>
>>>
>>>
>>> On Thu, Aug 30, 2018 at 8:08 PM, Davide Obbi 
>>> wrote:
>>>
 Thanks Amar,

 i have enabled the negative lookups cache on the volume:

>>>
> I think enabling nl-cache-positive-entry might help for untarring or git
> clone into glusterfs. Its disabled by default. can you let us know the
> results?
>
> Option: performance.nl-cache-positive-entry
> Default Value: (null)
> Description: enable/disable storing of entries that were lookedup and
> found to be present in the volume, thus lookup on non existent file is
> served from the cache
>
>
 To deflate a tar archive (not compressed) of 1.3GB it takes aprox 9mins
 which can be considered a slight improvement from the previous 12-15
 however still not fast enough compared to local disk. The tar is present on
 the gluster share/volume and deflated inside the same folder structure.

>>>
>>> I am assuming this is with parallel-readdir enabled, right?
>>>
>>>
 Running the operation twice (without removing the already deflated
 files) also did not reduce the time spent.

 Running the operation with the tar archive on local disk made no
 difference

 What really made a huge difference while git cloning was setting
 "performance.parallel-readdir on". During the phase "Receiving objects" ,
 as i enabled the xlator it bumped up from 3/4MBs to 27MBs

>>>
>>> What is the distribute count? Is it 1x3 replica?
>>>
>>>
 So in conclusion i'm trying to make the untar operation working at an
 acceptable level, not expecting local disks speed but at least being within
 the 4mins

 I have attached the profiles collected at the end of the untar
 operations with the archive on the mount and outside

 thanks
 Davide


 On Tue, Aug 28, 2018 at 8:41 AM Amar Tumballi 
 wrote:

> One of the observation we had with git clone like work load was,
> nl-cache (negative-lookup cache), helps here.
>
> Try 'gluster volume set $volume-name nl-cache enable'.
>
> Also sharing the 'profile info' during this performance observations
> also helps us to narrow down the situation.
>
> More on how to capture profile info @
> https://hackmd.io/PhhT5jPdQIKxzfeLQmnjJQ?view
>
> -Amar
>
>
> On Thu, Aug 23, 2018 at 7:11 PM, Davide Obbi 
> wrote:
>
>> Hello,
>>
>> did anyone ever managed to achieve reasonable waiting time while
>> performing metadata intensive operations such as git clone, untar etc...?
>> Is this possible workload or will never be in scope for glusterfs?
>>
>> I'd like to know, if possible, what would be the options that affect
>> such volume performances.
>> Albeit i managed to achieve decent git status/git grep operations, 3
>> and 30 secs, the git clone and untarring a file from/to the same share 
>> take
>> ages. for a git repo of aprox 6GB.
>>
>> I'm running a test environment with 3 way replica 128GB RAM and 24
>> cores are  2.40GHz, one internal SSD dedicated to the volume brick and 
>> 10Gb
>> network
>>
>> The options set so far that affects volume performances are:
>>  48 performance.readdir-ahead: on
>>  49 features.cache-invalidation-timeout: 600
>>  50 features.cache-invalidation: on
>>  51 performance.md-cache-timeout: 600
>>  52 performance.stat-prefetch: on
>>  53 performance.cache-invalidation: on
>>  54 performance.parallel-readdir: on
>>  55 network.inode-lru-limit: 90
>>  56 performance.io-thread-count: 32
>>  57 performance.cache-size: 10GB
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
>
>
> --
> Amar Tumballi (amarts)
>


 --
 Davide Obbi
 System Administrator

 Booking.com B.V.
 Vijzelstraat 66
 

Re: [Gluster-users] gluster 3.12.8 fuse consume huge memory

[Nithya Balachandran]

Hi,

Please take statedumps of the 3.12.13 client process at intervals when the
memory is increasing and send those across.
We will also need the gluster volume info for the volume int question.

Thanks,
Nithya


On 31 August 2018 at 08:32, huting3  wrote:

> Thanks for your reply, I also test gluster 3.12.13 and found the client
> also consumes huge memory:
>
> PID USER  PR  NIVIRTRESSHR S  %CPU %MEM TIME+ COMMAND
> 180095 root  20   0 4752256 4.091g   4084 S  43.5  1.6  17:54.70
> glusterfs
>
> I read and write some files on the gluster fuse client, the client consume
> 4g memory and it keeps arising.
>
> Does it really fixed in 3.12.13?
>
> huting3
> huti...@corp.netease.com
>
> 
> 签名由 网易邮箱大师  定制
>
> On 08/30/2018 22:37，Darrell Budic
>  wrote：
>
> It’s probably https://bugzilla.redhat.com/show_bug.cgi?id=1593826,
> although I did not encounter it in 3.12.8, only 3.12.9 - 12.
>
> It’s fixed in 3.12.13.
>
> --
> *From:* huting3 
> *Subject:* [Gluster-users] gluster 3.12.8 fuse consume huge memory
> *Date:* August 30, 2018 at 2:02:01 AM CDT
> *To:* gluster-users@gluster.org
>
> The version of glusterfs I installed is 3.12.8, and I find it`s client
> also consume huge memory.
>
>
> I dumped the statedump file, and I found the size of a variable is
> extreamly huge like below:
>
>
> [mount/fuse.fuse - usage-type gf_fuse_mt_iov_base memusage]
>
>   49805 size=4250821416
>
>   49806 num_allocs=1
>
>   49807 max_size=4294960048
>
>   49808 max_num_allocs=3
>
>   49809 total_allocs=12330719
>
>
> Is it means a memory leak exist in glusterfs client?
>
> huting3
> huti...@corp.netease.com
>
> 
> 签名由 网易邮箱大师  定制
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not work

[Kotresh Hiremath Ravishankar]

On Thu, Aug 30, 2018 at 3:51 PM, Krishna Verma  wrote:

> Hi Kotresh,
>
>
>
> Yes, this include the time take  to write 1GB file to master. geo-rep was
> not stopped while the data was copying to master.
>

This way, you can't really measure how much time geo-rep took.


>
> But now I am trouble, My putty session was timed out while copying data to
> master and geo replication was active. After I restart putty session My
> Master data is not syncing with slave. Its Last_synced time is  1hrs behind
> the current time.
>
>
>
> I restart the geo rep and also delete and again create the session but its
>  “LAST_SYNCED” time is same.
>

Unless, geo-rep is Faulty, it would be processing/syncing. You should check
logs for any errors.


>
> Please help in this.
>
>
>
> …. It's better if gluster volume has more distribute count like  3*3 or
> 4*3 :- Are you refereeing to create a distributed volume with 3 master
> node and 3 slave node?
>

Yes,  that's correct. Please do the test with this. I recommend you to run
the actual workload for which you are planning to use gluster instead of
copying 1GB file and testing.


>
>
>
> /krishna
>
>
>
> *From:* Kotresh Hiremath Ravishankar 
> *Sent:* Thursday, August 30, 2018 3:20 PM
>
> *To:* Krishna Verma 
> *Cc:* Sunny Kumar ; Gluster Users <
> gluster-users@gluster.org>
> *Subject:* Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not
> work
>
>
>
> EXTERNAL MAIL
>
>
>
>
>
> On Thu, Aug 30, 2018 at 1:52 PM, Krishna Verma  wrote:
>
> Hi Kotresh,
>
>
>
> After fix the library link on node "noi-poc-gluster ", the status of one
> mater node is “Active” and another is “Passive”. Can I setup both the
> master as “Active” ?
>
>
>
> Nope, since it's replica, it's redundant to sync same files from two
> nodes. Both replicas can't be Active.
>
>
>
>
>
> Also, when I copy a 1GB size of file from gluster client to master gluster
> volume which is replicated with the slave volume, it tooks 35 minutes and
> 49 seconds. Is there any way to reduce its time taken to rsync data.
>
>
>
> How did you measure this time? Does this include the time take for you to
> write 1GB file to master?
>
> There are two aspects to consider while measuring this.
>
>
>
> 1. Time to write 1GB to master
>
> 2. Time for geo-rep to transfer 1GB to slave.
>
>
>
> In your case, since the setup is 1*2 and only one geo-rep worker is
> Active, Step2 above equals to time for step1 + network transfer time.
>
>
>
> You can measure time in two scenarios
>
> 1. If geo-rep is started while the data is still being written to master.
> It's one way.
>
> 2. Or stop geo-rep until the 1GB file is written to master and then start
> geo-rep to get actual geo-rep time.
>
>
>
> To improve replicating speed,
>
> 1. You can play around with rsync options depending on the kind of I/O
>
> and configure the same for geo-rep as it also uses rsync internally.
>
> 2. It's better if gluster volume has more distribute count like  3*3 or 4*3
>
> It will help in two ways.
>
>1. The files gets distributed on master to multiple bricks
>
>2. So above will help geo-rep as files on multiple bricks are
> synced in parallel (multiple Actives)
>
>
>
> NOTE: Gluster master server and one client is in Noida, India Location.
>
>  Gluster Slave server and one client is in USA.
>
>
>
> Our approach is to transfer data from Noida gluster client will reach to
> the USA gluster client in a minimum time. Please suggest the best approach
> to achieve it.
>
>
>
> [root@noi-dcops ~]# date ; rsync -avh --progress /tmp/gentoo_root.img
> /glusterfs/ ; date
>
> Thu Aug 30 12:26:26 IST 2018
>
> sending incremental file list
>
> gentoo_root.img
>
>   1.07G 100%  490.70kB/s0:35:36 (xfr#1, to-chk=0/1)
>
>
>
> Is this I/O time to write to master volume?
>
>
>
> sent 1.07G bytes  received 35 bytes  499.65K bytes/sec
>
> total size is 1.07G  speedup is 1.00
>
> Thu Aug 30 13:02:15 IST 2018
>
> [root@noi-dcops ~]#
>
>
>
>
>
>
>
> [root@gluster-poc-noida gluster]#  gluster volume geo-replication status
>
>
>
> MASTER NODE  MASTER VOLMASTER BRICK SLAVE USER
> SLAVE  SLAVE NODESTATUS CRAWL
> STATUS   LAST_SYNCED
>
> 
> 
> ---
>
> gluster-poc-noidaglusterep /data/gluster/gv0root
> ssh://gluster-poc-sj::glusterepgluster-poc-sjActive Changelog
> Crawl2018-08-30 13:42:18
>
> noi-poc-gluster  glusterep /data/gluster/gv0root
> ssh://gluster-poc-sj::glusterepgluster-poc-sjPassive
> N/AN/A
>
> [root@gluster-poc-noida gluster]#
>
>
>
> Thanks in advance for your all time support.
>
>
>
> /Krishna
>
>
>
> *From:* Kotresh Hiremath Ravishankar 
> *Sent:* Thursday, August 30, 2018 10:51 AM
>
>
> *To:* Krishna Verma 
> *Cc:* Sunny Kumar ; 

Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not work

[Kotresh Hiremath Ravishankar]

On Thu, Aug 30, 2018 at 4:55 PM, Krishna Verma  wrote:

> Hi Kotresh,
>
>
>
> 1. Time to write 1GB to master:   27 minutes and 29 seconds
>
> 2. Time for geo-rep to transfer 1GB to slave.   8 minutes
>
This is hard to believe, considering there is no
distribution and there is only one brick participating in syncing.
Could you retest and confirm.

>
>
> /Krishna
>
>
>
>
>
> *From:* Kotresh Hiremath Ravishankar 
> *Sent:* Thursday, August 30, 2018 3:20 PM
>
> *To:* Krishna Verma 
> *Cc:* Sunny Kumar ; Gluster Users <
> gluster-users@gluster.org>
> *Subject:* Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not
> work
>
>
>
> EXTERNAL MAIL
>
>
>
>
>
> On Thu, Aug 30, 2018 at 1:52 PM, Krishna Verma  wrote:
>
> Hi Kotresh,
>
>
>
> After fix the library link on node "noi-poc-gluster ", the status of one
> mater node is “Active” and another is “Passive”. Can I setup both the
> master as “Active” ?
>
>
>
> Nope, since it's replica, it's redundant to sync same files from two
> nodes. Both replicas can't be Active.
>
>
>
>
>
> Also, when I copy a 1GB size of file from gluster client to master gluster
> volume which is replicated with the slave volume, it tooks 35 minutes and
> 49 seconds. Is there any way to reduce its time taken to rsync data.
>
>
>
> How did you measure this time? Does this include the time take for you to
> write 1GB file to master?
>
> There are two aspects to consider while measuring this.
>
>
>
> 1. Time to write 1GB to master
>
> 2. Time for geo-rep to transfer 1GB to slave.
>
>
>
> In your case, since the setup is 1*2 and only one geo-rep worker is
> Active, Step2 above equals to time for step1 + network transfer time.
>
>
>
> You can measure time in two scenarios
>
> 1. If geo-rep is started while the data is still being written to master.
> It's one way.
>
> 2. Or stop geo-rep until the 1GB file is written to master and then start
> geo-rep to get actual geo-rep time.
>
>
>
> To improve replicating speed,
>
> 1. You can play around with rsync options depending on the kind of I/O
>
> and configure the same for geo-rep as it also uses rsync internally.
>
> 2. It's better if gluster volume has more distribute count like  3*3 or 4*3
>
> It will help in two ways.
>
>1. The files gets distributed on master to multiple bricks
>
>2. So above will help geo-rep as files on multiple bricks are
> synced in parallel (multiple Actives)
>
>
>
> NOTE: Gluster master server and one client is in Noida, India Location.
>
>  Gluster Slave server and one client is in USA.
>
>
>
> Our approach is to transfer data from Noida gluster client will reach to
> the USA gluster client in a minimum time. Please suggest the best approach
> to achieve it.
>
>
>
> [root@noi-dcops ~]# date ; rsync -avh --progress /tmp/gentoo_root.img
> /glusterfs/ ; date
>
> Thu Aug 30 12:26:26 IST 2018
>
> sending incremental file list
>
> gentoo_root.img
>
>   1.07G 100%  490.70kB/s0:35:36 (xfr#1, to-chk=0/1)
>
>
>
> Is this I/O time to write to master volume?
>
>
>
> sent 1.07G bytes  received 35 bytes  499.65K bytes/sec
>
> total size is 1.07G  speedup is 1.00
>
> Thu Aug 30 13:02:15 IST 2018
>
> [root@noi-dcops ~]#
>
>
>
>
>
>
>
> [root@gluster-poc-noida gluster]#  gluster volume geo-replication status
>
>
>
> MASTER NODE  MASTER VOLMASTER BRICK SLAVE USER
> SLAVE  SLAVE NODESTATUS CRAWL
> STATUS   LAST_SYNCED
>
> 
> 
> ---
>
> gluster-poc-noidaglusterep /data/gluster/gv0root
> ssh://gluster-poc-sj::glusterepgluster-poc-sjActive Changelog
> Crawl2018-08-30 13:42:18
>
> noi-poc-gluster  glusterep /data/gluster/gv0root
> ssh://gluster-poc-sj::glusterepgluster-poc-sjPassive
> N/AN/A
>
> [root@gluster-poc-noida gluster]#
>
>
>
> Thanks in advance for your all time support.
>
>
>
> /Krishna
>
>
>
> *From:* Kotresh Hiremath Ravishankar 
> *Sent:* Thursday, August 30, 2018 10:51 AM
>
>
> *To:* Krishna Verma 
> *Cc:* Sunny Kumar ; Gluster Users <
> gluster-users@gluster.org>
> *Subject:* Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not
> work
>
>
>
> EXTERNAL MAIL
>
> Did you fix the library link on node "noi-poc-gluster " as well?
>
> If not please fix it. Please share the geo-rep log this node if it's
>
> as different issue.
>
> -Kotresh HR
>
>
>
> On Thu, Aug 30, 2018 at 12:17 AM, Krishna Verma 
> wrote:
>
> Hi Kotresh,
>
>
>
> Thank you so much for you input. Geo-replication is now showing “Active”
> atleast for 1 master node. But its still at faulty state for the 2nd
> master server.
>
>
>
> Below is the detail.
>
>
>
> [root@gluster-poc-noida glusterfs]# gluster volume geo-replication
> glusterep gluster-poc-sj::glusterep status
>
>
>
> MASTER NODE   

Re: [Gluster-users] gluster connection interrupted during transfer

[Raghavendra Gowdappa]

+Mohit. +Milind

@Mohit/Milind,

Can you check logs and see whether you can find anything relevant?

On Thu, Aug 30, 2018 at 7:04 PM, Richard Neuboeck 
wrote:

> Hi,
>
> I'm attaching a shortened version since the whole is about 5.8GB of
> the client mount log. It includes the initial mount messages and the
> last two minutes of log entries.
>
> It ends very anticlimactic without an obvious error. Is there
> anything specific I should be looking for?
>

Normally I look logs around disconnect msgs to find out the reason. But as
you said, sometimes one can see just disconnect msgs without any reason.
That normally points to reason for disconnect in the network rather than a
Glusterfs initiated disconnect.


> Cheers
> Richard
>
> On 08/30/2018 02:40 PM, Raghavendra Gowdappa wrote:
> > Normally client logs will give a clue on why the disconnections are
> > happening (ping-timeout, wrong port etc). Can you look into client
> > logs to figure out what's happening? If you can't find anything, can
> > you send across client logs?
> >
> > On Wed, Aug 29, 2018 at 6:11 PM, Richard Neuboeck
> > mailto:h...@tbi.univie.ac.at>> wrote:
> >
> > Hi Gluster Community,
> >
> > I have problems with a glusterfs 'Transport endpoint not connected'
> > connection abort during file transfers that I can replicate (all the
> > time now) but not pinpoint as to why this is happening.
> >
> > The volume is set up in replica 3 mode and accessed with the fuse
> > gluster client. Both client and server are running CentOS and the
> > supplied 3.12.11 version of gluster.
> >
> > The connection abort happens at different times during rsync but
> > occurs every time I try to sync all our files (1.1TB) to the empty
> > volume.
> >
> > Client and server side I don't find errors in the gluster log files.
> > rsync logs the obvious transfer problem. The only log that shows
> > anything related is the server brick log which states that the
> > connection is shutting down:
> >
> > [2018-08-18 22:40:35.502510] I [MSGID: 115036]
> > [server.c:527:server_rpc_notify] 0-home-server: disconnecting
> > connection from
> > brax-110405-2018/08/16-08:36:28:575972-home-client-0-0-0
> > [2018-08-18 22:40:35.502620] W
> > [inodelk.c:499:pl_inodelk_log_cleanup] 0-home-server: releasing lock
> > on eaeb0398-fefd-486d-84a7-f13744d1cf10 held by
> > {client=0x7f83ec0b3ce0, pid=110423 lk-owner=d0fd5ffb427f}
> > [2018-08-18 22:40:35.502692] W
> > [entrylk.c:864:pl_entrylk_log_cleanup] 0-home-server: releasing lock
> > on faa93f7b-6c46-4251-b2b2-abcd2f2613e1 held by
> > {client=0x7f83ec0b3ce0, pid=110423 lk-owner=703dd4cc407f}
> > [2018-08-18 22:40:35.502719] W
> > [entrylk.c:864:pl_entrylk_log_cleanup] 0-home-server: releasing lock
> > on faa93f7b-6c46-4251-b2b2-abcd2f2613e1 held by
> > {client=0x7f83ec0b3ce0, pid=110423 lk-owner=703dd4cc407f}
> > [2018-08-18 22:40:35.505950] I [MSGID: 101055]
> > [client_t.c:443:gf_client_unref] 0-home-server: Shutting down
> > connection brax-110405-2018/08/16-08:36:28:575972-home-client-0-0-0
> >
> > Since I'm running another replica 3 setup for oVirt for a long time
> > now which is completely stable I thought I made a mistake setting
> > different options at first. However even when I reset those options
> > I'm able to reproduce the connection problem.
> >
> > The unoptimized volume setup looks like this:
> >
> > Volume Name: home
> > Type: Replicate
> > Volume ID: c92fa4cc-4a26-41ff-8c70-1dd07f733ac8
> > Status: Started
> > Snapshot Count: 0
> > Number of Bricks: 1 x 3 = 3
> > Transport-type: tcp
> > Bricks:
> > Brick1: sphere-four:/srv/gluster_home/brick
> > Brick2: sphere-five:/srv/gluster_home/brick
> > Brick3: sphere-six:/srv/gluster_home/brick
> > Options Reconfigured:
> > nfs.disable: on
> > transport.address-family: inet
> > cluster.quorum-type: auto
> > cluster.server-quorum-type: server
> > cluster.server-quorum-ratio: 50%
> >
> >
> > The following additional options were used before:
> >
> > performance.cache-size: 5GB
> > client.event-threads: 4
> > server.event-threads: 4
> > cluster.lookup-optimize: on
> > features.cache-invalidation: on
> > performance.stat-prefetch: on
> > performance.cache-invalidation: on
> > network.inode-lru-limit: 5
> > features.cache-invalidation-timeout: 600
> > performance.md-cache-timeout: 600
> > performance.parallel-readdir: on
> >
> >
> > In this case the gluster servers and also the client is using a
> > bonded network device running in adaptive load balancing mode.
> >
> > I've tried using the debug option for the client mount. But except
> > for a ~0.5TB log file I didn't get information that seems
> > helpful to me.
> >
> > Transferring just a couple of GB works without 

Re: [Gluster-users] gluster connection interrupted during transfer

[Richard Neuboeck]

On 08/31/2018 03:50 AM, Raghavendra Gowdappa wrote:
> +Mohit. +Milind
> 
> @Mohit/Milind,
> 
> Can you check logs and see whether you can find anything relevant?

From glances at the system logs nothing out of the ordinary
occurred. However I'll start another rsync and take a closer look.
It will take a few days.

> 
> On Thu, Aug 30, 2018 at 7:04 PM, Richard Neuboeck
> mailto:h...@tbi.univie.ac.at>> wrote:
> 
> Hi,
> 
> I'm attaching a shortened version since the whole is about 5.8GB of
> the client mount log. It includes the initial mount messages and the
> last two minutes of log entries.
> 
> It ends very anticlimactic without an obvious error. Is there
> anything specific I should be looking for?
> 
> 
> Normally I look logs around disconnect msgs to find out the reason.
> But as you said, sometimes one can see just disconnect msgs without
> any reason. That normally points to reason for disconnect in the
> network rather than a Glusterfs initiated disconnect.

The rsync source is serving our homes currently so there are NFS
connections 24/7. There don't seem to be any network related
interruptions - a co-worker would be here faster than I could check
the logs if the connection to home would be broken ;-)
The three gluster machines are due to this problem reduced to only
testing so there is nothing else running.


> 
> Cheers
> Richard
> 
> On 08/30/2018 02:40 PM, Raghavendra Gowdappa wrote:
> > Normally client logs will give a clue on why the disconnections are
> > happening (ping-timeout, wrong port etc). Can you look into client
> > logs to figure out what's happening? If you can't find anything, can
> > you send across client logs?
> > 
> > On Wed, Aug 29, 2018 at 6:11 PM, Richard Neuboeck
> > mailto:h...@tbi.univie.ac.at>
> >>
> wrote:
> >
> >     Hi Gluster Community,
> >
> >     I have problems with a glusterfs 'Transport endpoint not
> connected'
> >     connection abort during file transfers that I can
> replicate (all the
> >     time now) but not pinpoint as to why this is happening.
> >
> >     The volume is set up in replica 3 mode and accessed with
> the fuse
> >     gluster client. Both client and server are running CentOS
> and the
> >     supplied 3.12.11 version of gluster.
> >
> >     The connection abort happens at different times during
> rsync but
> >     occurs every time I try to sync all our files (1.1TB) to
> the empty
> >     volume.
> >
> >     Client and server side I don't find errors in the gluster
> log files.
> >     rsync logs the obvious transfer problem. The only log that
> shows
> >     anything related is the server brick log which states that the
> >     connection is shutting down:
> >
> >     [2018-08-18 22:40:35.502510] I [MSGID: 115036]
> >     [server.c:527:server_rpc_notify] 0-home-server: disconnecting
> >     connection from
> >     brax-110405-2018/08/16-08:36:28:575972-home-client-0-0-0
> >     [2018-08-18 22:40:35.502620] W
> >     [inodelk.c:499:pl_inodelk_log_cleanup] 0-home-server:
> releasing lock
> >     on eaeb0398-fefd-486d-84a7-f13744d1cf10 held by
> >     {client=0x7f83ec0b3ce0, pid=110423 lk-owner=d0fd5ffb427f}
> >     [2018-08-18 22:40:35.502692] W
> >     [entrylk.c:864:pl_entrylk_log_cleanup] 0-home-server:
> releasing lock
> >     on faa93f7b-6c46-4251-b2b2-abcd2f2613e1 held by
> >     {client=0x7f83ec0b3ce0, pid=110423 lk-owner=703dd4cc407f}
> >     [2018-08-18 22:40:35.502719] W
> >     [entrylk.c:864:pl_entrylk_log_cleanup] 0-home-server:
> releasing lock
> >     on faa93f7b-6c46-4251-b2b2-abcd2f2613e1 held by
> >     {client=0x7f83ec0b3ce0, pid=110423 lk-owner=703dd4cc407f}
> >     [2018-08-18 22:40:35.505950] I [MSGID: 101055]
> >     [client_t.c:443:gf_client_unref] 0-home-server: Shutting down
> >     connection
> brax-110405-2018/08/16-08:36:28:575972-home-client-0-0-0
> >
> >     Since I'm running another replica 3 setup for oVirt for a
> long time
> >     now which is completely stable I thought I made a mistake
> setting
> >     different options at first. However even when I reset
> those options
> >     I'm able to reproduce the connection problem.
> >
> >     The unoptimized volume setup looks like this:
> >
> >     Volume Name: home
> >     Type: Replicate
> >     Volume ID: c92fa4cc-4a26-41ff-8c70-1dd07f733ac8
> >     Status: Started
> >     Snapshot Count: 0
> >     Number of Bricks: 1 x 3 = 3
> >     Transport-type: tcp
> >     Bricks:
> >     Brick1: sphere-four:/srv/gluster_home/brick
> >     Brick2: sphere-five:/srv/gluster_home/brick
> >     Brick3: sphere-six:/srv/gluster_home/brick
>

Re: [Gluster-users] gluster 3.12.8 fuse consume huge memory

[huting3]

Thanks for your reply, I also test gluster 3.12.13 and found the client also consumes huge memory:PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND180095 root      20   0 4752256 4.091g   4084 S  43.5  1.6  17:54.70 glusterfsI read and write some files on the gluster fuse client, the client consume 4g memory and it keeps arising.Does it really fixed in 3.12.13?


  










huting3







huti...@corp.netease.com








签名由
网易邮箱大师
定制

 


On 08/30/2018 22:37，Darrell Budic wrote： 


It’s probably https://bugzilla.redhat.com/show_bug.cgi?id=1593826, although I did not encounter it in 3.12.8, only 3.12.9 - 12.It’s fixed in 3.12.13.From: huting3 
Subject: [Gluster-users] gluster 3.12.8 fuse consume huge memory
Date: August 30, 2018 at 2:02:01 AM CDT
To: gluster-users@gluster.org
The version of glusterfs I installed is 3.12.8, and I find it`s client also consume huge memory.I dumped the statedump file, and I found the size of a variable is extreamly huge like below:[mount/fuse.fuse - usage-type gf_fuse_mt_iov_base memusage]  49805 size=4250821416  49806 num_allocs=1  49807 max_size=4294960048  49808 max_num_allocs=3  49809 total_allocs=12330719Is it means a memory leak exist in glusterfs client?huting3huti...@corp.netease.com签名由 网易邮箱大师 定制___Gluster-users mailing listGluster-users@gluster.orghttps://lists.gluster.org/mailman/listinfo/gluster-users



___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] gluster 3.12.8 fuse consume huge memory

[huting3]

The version of glusterfs I installed is 3.12.8, and I find it`s client also consume huge memory.I dumped the statedump file, and I found the size of a variable is extreamly huge like below:[mount/fuse.fuse - usage-type gf_fuse_mt_iov_base memusage]
  49805 size=4250821416
  49806 num_allocs=1
  49807 max_size=4294960048
  49808 max_num_allocs=3
  49809 total_allocs=12330719Is it means a memory leak exist in glusterfs client?


  










huting3







huti...@corp.netease.com








签名由
网易邮箱大师
定制

 




___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not work

[Kotresh Hiremath Ravishankar]

On Thu, Aug 30, 2018 at 1:52 PM, Krishna Verma  wrote:

> Hi Kotresh,
>
>
>
> After fix the library link on node "noi-poc-gluster ", the status of one
> mater node is “Active” and another is “Passive”. Can I setup both the
> master as “Active” ?
>

Nope, since it's replica, it's redundant to sync same files from two nodes.
Both replicas can't be Active.


>
> Also, when I copy a 1GB size of file from gluster client to master gluster
> volume which is replicated with the slave volume, it tooks 35 minutes and
> 49 seconds. Is there any way to reduce its time taken to rsync data.
>

How did you measure this time? Does this include the time take for you to
write 1GB file to master?
There are two aspects to consider while measuring this.

1. Time to write 1GB to master
2. Time for geo-rep to transfer 1GB to slave.

In your case, since the setup is 1*2 and only one geo-rep worker is Active,
Step2 above equals to time for step1 + network transfer time.

You can measure time in two scenarios
1. If geo-rep is started while the data is still being written to master.
It's one way.
2. Or stop geo-rep until the 1GB file is written to master and then start
geo-rep to get actual geo-rep time.

To improve replicating speed,
1. You can play around with rsync options depending on the kind of I/O
and configure the same for geo-rep as it also uses rsync internally.
2. It's better if gluster volume has more distribute count like  3*3 or 4*3
It will help in two ways.
   1. The files gets distributed on master to multiple bricks
   2. So above will help geo-rep as files on multiple bricks are synced
in parallel (multiple Actives)

>
>
> NOTE: Gluster master server and one client is in Noida, India Location.
>
>  Gluster Slave server and one client is in USA.
>
>
>
> Our approach is to transfer data from Noida gluster client will reach to
> the USA gluster client in a minimum time. Please suggest the best approach
> to achieve it.
>
>
>
> [root@noi-dcops ~]# date ; rsync -avh --progress /tmp/gentoo_root.img
> /glusterfs/ ; date
>
> Thu Aug 30 12:26:26 IST 2018
>
> sending incremental file list
>
> gentoo_root.img
>
>   1.07G 100%  490.70kB/s0:35:36 (xfr#1, to-chk=0/1)
>

Is this I/O time to write to master volume?

>
>
> sent 1.07G bytes  received 35 bytes  499.65K bytes/sec
>
> total size is 1.07G  speedup is 1.00
>
> Thu Aug 30 13:02:15 IST 2018
>
> [root@noi-dcops ~]#
>


>
>
>
>
> [root@gluster-poc-noida gluster]#  gluster volume geo-replication status
>
>
>
> MASTER NODE  MASTER VOLMASTER BRICK SLAVE USER
> SLAVE  SLAVE NODESTATUS CRAWL
> STATUS   LAST_SYNCED
>
> 
> 
> ---
>
> gluster-poc-noidaglusterep /data/gluster/gv0root
> ssh://gluster-poc-sj::glusterepgluster-poc-sjActive Changelog
> Crawl2018-08-30 13:42:18
>
> noi-poc-gluster  glusterep /data/gluster/gv0root
> ssh://gluster-poc-sj::glusterepgluster-poc-sjPassive
> N/AN/A
>
> [root@gluster-poc-noida gluster]#
>
>
>
> Thanks in advance for your all time support.
>
>
>
> /Krishna
>
>
>
> *From:* Kotresh Hiremath Ravishankar 
> *Sent:* Thursday, August 30, 2018 10:51 AM
>
> *To:* Krishna Verma 
> *Cc:* Sunny Kumar ; Gluster Users <
> gluster-users@gluster.org>
> *Subject:* Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not
> work
>
>
>
> EXTERNAL MAIL
>
> Did you fix the library link on node "noi-poc-gluster " as well?
>
> If not please fix it. Please share the geo-rep log this node if it's
>
> as different issue.
>
> -Kotresh HR
>
>
>
> On Thu, Aug 30, 2018 at 12:17 AM, Krishna Verma 
> wrote:
>
> Hi Kotresh,
>
>
>
> Thank you so much for you input. Geo-replication is now showing “Active”
> atleast for 1 master node. But its still at faulty state for the 2nd
> master server.
>
>
>
> Below is the detail.
>
>
>
> [root@gluster-poc-noida glusterfs]# gluster volume geo-replication
> glusterep gluster-poc-sj::glusterep status
>
>
>
> MASTER NODE  MASTER VOLMASTER BRICK SLAVE USER
> SLAVESLAVE NODESTATUSCRAWL STATUS
> LAST_SYNCED
>
> 
> 
> 
>
> gluster-poc-noidaglusterep /data/gluster/gv0root
> gluster-poc-sj::glusterepgluster-poc-sjActiveChangelog Crawl
> 2018-08-29 23:56:06
>
> noi-poc-gluster  glusterep /data/gluster/gv0root
> gluster-poc-sj::glusterepN/A   FaultyN/A
> N/A
>
>
>
>
>
> [root@gluster-poc-noida glusterfs]# gluster volume status
>
> Status of volume: glusterep
>
> Gluster process TCP Port  RDMA Port  Online
> Pid
>
> 

Re: [Gluster-users] gluster connection interrupted during transfer

[Nithya Balachandran]

Hi Richard,



On 29 August 2018 at 18:11, Richard Neuboeck  wrote:

> Hi Gluster Community,
>
> I have problems with a glusterfs 'Transport endpoint not connected'
> connection abort during file transfers that I can replicate (all the
> time now) but not pinpoint as to why this is happening.
>
> The volume is set up in replica 3 mode and accessed with the fuse
> gluster client. Both client and server are running CentOS and the
> supplied 3.12.11 version of gluster.
>
> The connection abort happens at different times during rsync but
> occurs every time I try to sync all our files (1.1TB) to the empty
> volume.
>
> Client and server side I don't find errors in the gluster log files.
> rsync logs the obvious transfer problem. The only log that shows
> anything related is the server brick log which states that the
> connection is shutting down:
>
> [2018-08-18 22:40:35.502510] I [MSGID: 115036]
> [server.c:527:server_rpc_notify] 0-home-server: disconnecting
> connection from brax-110405-2018/08/16-08:36:28:575972-home-client-0-0-0
> [2018-08-18 22:40:35.502620] W
> [inodelk.c:499:pl_inodelk_log_cleanup] 0-home-server: releasing lock
> on eaeb0398-fefd-486d-84a7-f13744d1cf10 held by
> {client=0x7f83ec0b3ce0, pid=110423 lk-owner=d0fd5ffb427f}
> [2018-08-18 22:40:35.502692] W
> [entrylk.c:864:pl_entrylk_log_cleanup] 0-home-server: releasing lock
> on faa93f7b-6c46-4251-b2b2-abcd2f2613e1 held by
> {client=0x7f83ec0b3ce0, pid=110423 lk-owner=703dd4cc407f}
> [2018-08-18 22:40:35.502719] W
> [entrylk.c:864:pl_entrylk_log_cleanup] 0-home-server: releasing lock
> on faa93f7b-6c46-4251-b2b2-abcd2f2613e1 held by
> {client=0x7f83ec0b3ce0, pid=110423 lk-owner=703dd4cc407f}
> [2018-08-18 22:40:35.505950] I [MSGID: 101055]
> [client_t.c:443:gf_client_unref] 0-home-server: Shutting down
> connection brax-110405-2018/08/16-08:36:28:575972-home-client-0-0-0


> Since I'm running another replica 3 setup for oVirt for a long time
>

Is this setup running with the same gluster version and on the same nodes
or is it a different cluster?



> now which is completely stable I thought I made a mistake setting
> different options at first. However even when I reset those options
> I'm able to reproduce the connection problem.
>
> The unoptimized volume setup looks like this:


> Volume Name: home
> Type: Replicate
> Volume ID: c92fa4cc-4a26-41ff-8c70-1dd07f733ac8
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x 3 = 3
> Transport-type: tcp
> Bricks:
> Brick1: sphere-four:/srv/gluster_home/brick
> Brick2: sphere-five:/srv/gluster_home/brick
> Brick3: sphere-six:/srv/gluster_home/brick
> Options Reconfigured:
> nfs.disable: on
> transport.address-family: inet
> cluster.quorum-type: auto
> cluster.server-quorum-type: server
> cluster.server-quorum-ratio: 50%
>
>
> The following additional options were used before:
>
> performance.cache-size: 5GB
> client.event-threads: 4
> server.event-threads: 4
> cluster.lookup-optimize: on
> features.cache-invalidation: on
> performance.stat-prefetch: on
> performance.cache-invalidation: on
> network.inode-lru-limit: 5
> features.cache-invalidation-timeout: 600
> performance.md-cache-timeout: 600
> performance.parallel-readdir: on
>
>
> In this case the gluster servers and also the client is using a
> bonded network device running in adaptive load balancing mode.
>
> I've tried using the debug option for the client mount. But except
> for a ~0.5TB log file I didn't get information that seems helpful to me.
>
> Transferring just a couple of GB works without problems.
>
> It may very well be that I'm already blind to the obvious but after
> many long running tests I can't find the crux in the setup.
>
> Does anyone have an idea as how to approach this problem in a way
> that sheds some useful information?
>
> Any help is highly appreciated!
> Cheers
> Richard
>
> --
> /dev/null
>
>
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not work

[Krishna Verma]

Hi Kotresh,

Yes, this include the time take  to write 1GB file to master. geo-rep was not 
stopped while the data was copying to master.

But now I am trouble, My putty session was timed out while copying data to 
master and geo replication was active. After I restart putty session My Master 
data is not syncing with slave. Its Last_synced time is  1hrs behind the 
current time.

I restart the geo rep and also delete and again create the session but its  
“LAST_SYNCED” time is same.

Please help in this.

…. It's better if gluster volume has more distribute count like  3*3 or 4*3 :- 
Are you refereeing to create a distributed volume with 3 master node and 3 
slave node?


/krishna

From: Kotresh Hiremath Ravishankar 
Sent: Thursday, August 30, 2018 3:20 PM
To: Krishna Verma 
Cc: Sunny Kumar ; Gluster Users 
Subject: Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not work

EXTERNAL MAIL


On Thu, Aug 30, 2018 at 1:52 PM, Krishna Verma 
mailto:kve...@cadence.com>> wrote:
Hi Kotresh,

After fix the library link on node "noi-poc-gluster ", the status of one mater 
node is “Active” and another is “Passive”. Can I setup both the master as 
“Active” ?

Nope, since it's replica, it's redundant to sync same files from two nodes. 
Both replicas can't be Active.


Also, when I copy a 1GB size of file from gluster client to master gluster 
volume which is replicated with the slave volume, it tooks 35 minutes and 49 
seconds. Is there any way to reduce its time taken to rsync data.

How did you measure this time? Does this include the time take for you to write 
1GB file to master?
There are two aspects to consider while measuring this.

1. Time to write 1GB to master
2. Time for geo-rep to transfer 1GB to slave.

In your case, since the setup is 1*2 and only one geo-rep worker is Active, 
Step2 above equals to time for step1 + network transfer time.

You can measure time in two scenarios
1. If geo-rep is started while the data is still being written to master. It's 
one way.
2. Or stop geo-rep until the 1GB file is written to master and then start 
geo-rep to get actual geo-rep time.

To improve replicating speed,
1. You can play around with rsync options depending on the kind of I/O
and configure the same for geo-rep as it also uses rsync internally.
2. It's better if gluster volume has more distribute count like  3*3 or 4*3
It will help in two ways.
   1. The files gets distributed on master to multiple bricks
   2. So above will help geo-rep as files on multiple bricks are synced in 
parallel (multiple Actives)

NOTE: Gluster master server and one client is in Noida, India Location.
 Gluster Slave server and one client is in USA.

Our approach is to transfer data from Noida gluster client will reach to the 
USA gluster client in a minimum time. Please suggest the best approach to 
achieve it.

[root@noi-dcops ~]# date ; rsync -avh --progress /tmp/gentoo_root.img 
/glusterfs/ ; date
Thu Aug 30 12:26:26 IST 2018
sending incremental file list
gentoo_root.img
  1.07G 100%  490.70kB/s0:35:36 (xfr#1, to-chk=0/1)

Is this I/O time to write to master volume?

sent 1.07G bytes  received 35 bytes  499.65K bytes/sec
total size is 1.07G  speedup is 1.00
Thu Aug 30 13:02:15 IST 2018
[root@noi-dcops ~]#



[root@gluster-poc-noida gluster]#  gluster volume geo-replication status

MASTER NODE  MASTER VOLMASTER BRICK SLAVE USERSLAVE 
 SLAVE NODESTATUS CRAWL STATUS   
LAST_SYNCED
---
gluster-poc-noidaglusterep /data/gluster/gv0root  
ssh://gluster-poc-sj::glusterepgluster-poc-sjActive Changelog Crawl 
   2018-08-30 13:42:18
noi-poc-gluster  glusterep /data/gluster/gv0root  
ssh://gluster-poc-sj::glusterepgluster-poc-sjPassiveN/A 
   N/A
[root@gluster-poc-noida gluster]#

Thanks in advance for your all time support.

/Krishna

From: Kotresh Hiremath Ravishankar 
mailto:khire...@redhat.com>>
Sent: Thursday, August 30, 2018 10:51 AM

To: Krishna Verma mailto:kve...@cadence.com>>
Cc: Sunny Kumar mailto:sunku...@redhat.com>>; Gluster 
Users mailto:gluster-users@gluster.org>>
Subject: Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not work

EXTERNAL MAIL
Did you fix the library link on node "noi-poc-gluster " as well?
If not please fix it. Please share the geo-rep log this node if it's
as different issue.
-Kotresh HR

On Thu, Aug 30, 2018 at 12:17 AM, Krishna Verma 
mailto:kve...@cadence.com>> wrote:
Hi Kotresh,

Thank you so much for you input. Geo-replication is now showing “Active” 
atleast for 1 master node. But its still at faulty state for the 2nd master 
server.

Below is the detail.

[root@gluster-poc-noida glusterfs]# gluster volume geo-replication glusterep 

Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not work

[Krishna Verma]

Hi Kotresh,

Just to update for sync issue.

I erased the indexing by doing gluster volume set  indexing off,
after stopping the geo-rep session and then start the session. Now its syncing.

For your query:

Yes, this include the time take  to write 1GB file to master. geo-rep was not 
stopped while the data was copying to master.

…. It's better if gluster volume has more distribute count like  3*3 or 4*3 :- 
Are you refereeing to create a distributed volume with 3 master node and 3 
slave node?


/Krishna


From: Krishna Verma
Sent: Thursday, August 30, 2018 3:52 PM
To: 'Kotresh Hiremath Ravishankar' 
Cc: Sunny Kumar ; Gluster Users 
Subject: RE: [Gluster-users] Upgrade to 4.1.2 geo-replication does not work

Hi Kotresh,

Yes, this include the time take  to write 1GB file to master. geo-rep was not 
stopped while the data was copying to master.

But now I am trouble, My putty session was timed out while copying data to 
master and geo replication was active. After I restart putty session My Master 
data is not syncing with slave. Its Last_synced time is  1hrs behind the 
current time.

I restart the geo rep and also delete and again create the session but its  
“LAST_SYNCED” time is same.

Please help in this.

…. It's better if gluster volume has more distribute count like  3*3 or 4*3 :- 
Are you refereeing to create a distributed volume with 3 master node and 3 
slave node?


/krishna

From: Kotresh Hiremath Ravishankar 
mailto:khire...@redhat.com>>
Sent: Thursday, August 30, 2018 3:20 PM
To: Krishna Verma mailto:kve...@cadence.com>>
Cc: Sunny Kumar mailto:sunku...@redhat.com>>; Gluster 
Users mailto:gluster-users@gluster.org>>
Subject: Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not work

EXTERNAL MAIL


On Thu, Aug 30, 2018 at 1:52 PM, Krishna Verma 
mailto:kve...@cadence.com>> wrote:
Hi Kotresh,

After fix the library link on node "noi-poc-gluster ", the status of one mater 
node is “Active” and another is “Passive”. Can I setup both the master as 
“Active” ?

Nope, since it's replica, it's redundant to sync same files from two nodes. 
Both replicas can't be Active.


Also, when I copy a 1GB size of file from gluster client to master gluster 
volume which is replicated with the slave volume, it tooks 35 minutes and 49 
seconds. Is there any way to reduce its time taken to rsync data.

How did you measure this time? Does this include the time take for you to write 
1GB file to master?
There are two aspects to consider while measuring this.

1. Time to write 1GB to master
2. Time for geo-rep to transfer 1GB to slave.

In your case, since the setup is 1*2 and only one geo-rep worker is Active, 
Step2 above equals to time for step1 + network transfer time.

You can measure time in two scenarios
1. If geo-rep is started while the data is still being written to master. It's 
one way.
2. Or stop geo-rep until the 1GB file is written to master and then start 
geo-rep to get actual geo-rep time.

To improve replicating speed,
1. You can play around with rsync options depending on the kind of I/O
and configure the same for geo-rep as it also uses rsync internally.
2. It's better if gluster volume has more distribute count like  3*3 or 4*3
It will help in two ways.
   1. The files gets distributed on master to multiple bricks
   2. So above will help geo-rep as files on multiple bricks are synced in 
parallel (multiple Actives)

NOTE: Gluster master server and one client is in Noida, India Location.
 Gluster Slave server and one client is in USA.

Our approach is to transfer data from Noida gluster client will reach to the 
USA gluster client in a minimum time. Please suggest the best approach to 
achieve it.

[root@noi-dcops ~]# date ; rsync -avh --progress /tmp/gentoo_root.img 
/glusterfs/ ; date
Thu Aug 30 12:26:26 IST 2018
sending incremental file list
gentoo_root.img
  1.07G 100%  490.70kB/s0:35:36 (xfr#1, to-chk=0/1)

Is this I/O time to write to master volume?

sent 1.07G bytes  received 35 bytes  499.65K bytes/sec
total size is 1.07G  speedup is 1.00
Thu Aug 30 13:02:15 IST 2018
[root@noi-dcops ~]#



[root@gluster-poc-noida gluster]#  gluster volume geo-replication status

MASTER NODE  MASTER VOLMASTER BRICK SLAVE USERSLAVE 
 SLAVE NODESTATUS CRAWL STATUS   
LAST_SYNCED
---
gluster-poc-noidaglusterep /data/gluster/gv0root  
ssh://gluster-poc-sj::glusterepgluster-poc-sjActive Changelog Crawl 
   2018-08-30 13:42:18
noi-poc-gluster  glusterep /data/gluster/gv0root  
ssh://gluster-poc-sj::glusterepgluster-poc-sjPassiveN/A 
   N/A
[root@gluster-poc-noida gluster]#

Thanks in advance for your all time support.

/Krishna

From: Kotresh Hiremath 

Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not work

[Krishna Verma]

Hi Kotresh,

1. Time to write 1GB to master:   27 minutes and 29 seconds
2. Time for geo-rep to transfer 1GB to slave.   8 minutes

/Krishna


From: Kotresh Hiremath Ravishankar 
Sent: Thursday, August 30, 2018 3:20 PM
To: Krishna Verma 
Cc: Sunny Kumar ; Gluster Users 
Subject: Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not work

EXTERNAL MAIL


On Thu, Aug 30, 2018 at 1:52 PM, Krishna Verma 
mailto:kve...@cadence.com>> wrote:
Hi Kotresh,

After fix the library link on node "noi-poc-gluster ", the status of one mater 
node is “Active” and another is “Passive”. Can I setup both the master as 
“Active” ?

Nope, since it's replica, it's redundant to sync same files from two nodes. 
Both replicas can't be Active.


Also, when I copy a 1GB size of file from gluster client to master gluster 
volume which is replicated with the slave volume, it tooks 35 minutes and 49 
seconds. Is there any way to reduce its time taken to rsync data.

How did you measure this time? Does this include the time take for you to write 
1GB file to master?
There are two aspects to consider while measuring this.

1. Time to write 1GB to master
2. Time for geo-rep to transfer 1GB to slave.

In your case, since the setup is 1*2 and only one geo-rep worker is Active, 
Step2 above equals to time for step1 + network transfer time.

You can measure time in two scenarios
1. If geo-rep is started while the data is still being written to master. It's 
one way.
2. Or stop geo-rep until the 1GB file is written to master and then start 
geo-rep to get actual geo-rep time.

To improve replicating speed,
1. You can play around with rsync options depending on the kind of I/O
and configure the same for geo-rep as it also uses rsync internally.
2. It's better if gluster volume has more distribute count like  3*3 or 4*3
It will help in two ways.
   1. The files gets distributed on master to multiple bricks
   2. So above will help geo-rep as files on multiple bricks are synced in 
parallel (multiple Actives)

NOTE: Gluster master server and one client is in Noida, India Location.
 Gluster Slave server and one client is in USA.

Our approach is to transfer data from Noida gluster client will reach to the 
USA gluster client in a minimum time. Please suggest the best approach to 
achieve it.

[root@noi-dcops ~]# date ; rsync -avh --progress /tmp/gentoo_root.img 
/glusterfs/ ; date
Thu Aug 30 12:26:26 IST 2018
sending incremental file list
gentoo_root.img
  1.07G 100%  490.70kB/s0:35:36 (xfr#1, to-chk=0/1)

Is this I/O time to write to master volume?

sent 1.07G bytes  received 35 bytes  499.65K bytes/sec
total size is 1.07G  speedup is 1.00
Thu Aug 30 13:02:15 IST 2018
[root@noi-dcops ~]#



[root@gluster-poc-noida gluster]#  gluster volume geo-replication status

MASTER NODE  MASTER VOLMASTER BRICK SLAVE USERSLAVE 
 SLAVE NODESTATUS CRAWL STATUS   
LAST_SYNCED
---
gluster-poc-noidaglusterep /data/gluster/gv0root  
ssh://gluster-poc-sj::glusterepgluster-poc-sjActive Changelog Crawl 
   2018-08-30 13:42:18
noi-poc-gluster  glusterep /data/gluster/gv0root  
ssh://gluster-poc-sj::glusterepgluster-poc-sjPassiveN/A 
   N/A
[root@gluster-poc-noida gluster]#

Thanks in advance for your all time support.

/Krishna

From: Kotresh Hiremath Ravishankar 
mailto:khire...@redhat.com>>
Sent: Thursday, August 30, 2018 10:51 AM

To: Krishna Verma mailto:kve...@cadence.com>>
Cc: Sunny Kumar mailto:sunku...@redhat.com>>; Gluster 
Users mailto:gluster-users@gluster.org>>
Subject: Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not work

EXTERNAL MAIL
Did you fix the library link on node "noi-poc-gluster " as well?
If not please fix it. Please share the geo-rep log this node if it's
as different issue.
-Kotresh HR

On Thu, Aug 30, 2018 at 12:17 AM, Krishna Verma 
mailto:kve...@cadence.com>> wrote:
Hi Kotresh,

Thank you so much for you input. Geo-replication is now showing “Active” 
atleast for 1 master node. But its still at faulty state for the 2nd master 
server.

Below is the detail.

[root@gluster-poc-noida glusterfs]# gluster volume geo-replication glusterep 
gluster-poc-sj::glusterep status

MASTER NODE  MASTER VOLMASTER BRICK SLAVE USERSLAVE 
   SLAVE NODESTATUSCRAWL STATUS   LAST_SYNCED

gluster-poc-noidaglusterep /data/gluster/gv0root  
gluster-poc-sj::glusterepgluster-poc-sjActiveChangelog Crawl
2018-08-29 23:56:06
noi-poc-gluster  

Re: [Gluster-users] gluster connection interrupted during transfer

[Richard Neuboeck]

Hi Nithya,


On 08/30/2018 09:45 AM, Nithya Balachandran wrote:
> Hi Richard,
> 
> 
> 
> On 29 August 2018 at 18:11, Richard Neuboeck  > wrote:
> 
> Hi Gluster Community,
> 
> I have problems with a glusterfs 'Transport endpoint not connected'
> connection abort during file transfers that I can replicate (all the
> time now) but not pinpoint as to why this is happening.
> 
> The volume is set up in replica 3 mode and accessed with the fuse
> gluster client. Both client and server are running CentOS and the
> supplied 3.12.11 version of gluster.
> 
> The connection abort happens at different times during rsync but
> occurs every time I try to sync all our files (1.1TB) to the empty
> volume.
> 
> Client and server side I don't find errors in the gluster log files.
> rsync logs the obvious transfer problem. The only log that shows
> anything related is the server brick log which states that the
> connection is shutting down:
> 
> [2018-08-18 22:40:35.502510] I [MSGID: 115036]
> [server.c:527:server_rpc_notify] 0-home-server: disconnecting
> connection from
> brax-110405-2018/08/16-08:36:28:575972-home-client-0-0-0
> [2018-08-18 22:40:35.502620] W
> [inodelk.c:499:pl_inodelk_log_cleanup] 0-home-server: releasing lock
> on eaeb0398-fefd-486d-84a7-f13744d1cf10 held by
> {client=0x7f83ec0b3ce0, pid=110423 lk-owner=d0fd5ffb427f}
> [2018-08-18 22:40:35.502692] W
> [entrylk.c:864:pl_entrylk_log_cleanup] 0-home-server: releasing lock
> on faa93f7b-6c46-4251-b2b2-abcd2f2613e1 held by
> {client=0x7f83ec0b3ce0, pid=110423 lk-owner=703dd4cc407f}
> [2018-08-18 22:40:35.502719] W
> [entrylk.c:864:pl_entrylk_log_cleanup] 0-home-server: releasing lock
> on faa93f7b-6c46-4251-b2b2-abcd2f2613e1 held by
> {client=0x7f83ec0b3ce0, pid=110423 lk-owner=703dd4cc407f}
> [2018-08-18 22:40:35.505950] I [MSGID: 101055]
> [client_t.c:443:gf_client_unref] 0-home-server: Shutting down
> connection brax-110405-2018/08/16-08:36:28:575972-home-client-0-0-0 
> 
> 
> Since I'm running another replica 3 setup for oVirt for a long time
> 
> 
> Is this setup running with the same gluster version and on the same
> nodes or is it a different cluster?


It's a different cluster (sphere-one, sphere-two and sphere-three)
but the same gluster version and basically the same hardware.

Cheers
Richard

> 
>  
> 
> now which is completely stable I thought I made a mistake setting
> different options at first. However even when I reset those options
> I'm able to reproduce the connection problem.
> 
> The unoptimized volume setup looks like this: 
> 
> 
> Volume Name: home
> Type: Replicate
> Volume ID: c92fa4cc-4a26-41ff-8c70-1dd07f733ac8
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x 3 = 3
> Transport-type: tcp
> Bricks:
> Brick1: sphere-four:/srv/gluster_home/brick
> Brick2: sphere-five:/srv/gluster_home/brick
> Brick3: sphere-six:/srv/gluster_home/brick
> Options Reconfigured:
> nfs.disable: on
> transport.address-family: inet
> cluster.quorum-type: auto
> cluster.server-quorum-type: server
> cluster.server-quorum-ratio: 50%
> 
> 
> The following additional options were used before:
> 
> performance.cache-size: 5GB
> client.event-threads: 4
> server.event-threads: 4
> cluster.lookup-optimize: on
> features.cache-invalidation: on
> performance.stat-prefetch: on
> performance.cache-invalidation: on
> network.inode-lru-limit: 5
> features.cache-invalidation-timeout: 600
> performance.md-cache-timeout: 600
> performance.parallel-readdir: on
> 
> 
> In this case the gluster servers and also the client is using a
> bonded network device running in adaptive load balancing mode.
> 
> I've tried using the debug option for the client mount. But except
> for a ~0.5TB log file I didn't get information that seems
> helpful to me.
> 
> Transferring just a couple of GB works without problems.
> 
> It may very well be that I'm already blind to the obvious but after
> many long running tests I can't find the crux in the setup.
> 
> Does anyone have an idea as how to approach this problem in a way
> that sheds some useful information?
> 
> Any help is highly appreciated!
> Cheers
> Richard
> 
> -- 
> /dev/null
> 
> 
> 
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org 
> https://lists.gluster.org/mailman/listinfo/gluster-users
> 
> 
> 


-- 
/dev/null



signature.asc
Description: OpenPGP digital signature
___
Gluster-users mailing list
Gluster-users@gluster.org

Re: [Gluster-users] gluster 3.12.8 fuse consume huge memory

[Darrell Budic]

It’s probably https://bugzilla.redhat.com/show_bug.cgi?id=1593826 
, although I did not 
encounter it in 3.12.8, only 3.12.9 - 12.

It’s fixed in 3.12.13.

> From: huting3 
> Subject: [Gluster-users] gluster 3.12.8 fuse consume huge memory
> Date: August 30, 2018 at 2:02:01 AM CDT
> To: gluster-users@gluster.org
> 
> The version of glusterfs I installed is 3.12.8, and I find it`s client also 
> consume huge memory.
> 
> 
> 
> I dumped the statedump file, and I found the size of a variable is extreamly 
> huge like below:
> 
> 
> 
> [mount/fuse.fuse - usage-type gf_fuse_mt_iov_base memusage]
> 
>   49805 size=4250821416
> 
>   49806 num_allocs=1
> 
>   49807 max_size=4294960048
> 
>   49808 max_num_allocs=3
> 
>   49809 total_allocs=12330719
> 
> 
> 
> Is it means a memory leak exist in glusterfs client?
> 
> 
>   
> huting3
> 
> huti...@corp.netease.com
>  
> 
> 签名由 网易邮箱大师  定制
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org 
> https://lists.gluster.org/mailman/listinfo/gluster-users 
> 
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] gluster connection interrupted during transfer

[Raghavendra Gowdappa]

Normally client logs will give a clue on why the disconnections are
happening (ping-timeout, wrong port etc). Can you look into client logs to
figure out what's happening? If you can't find anything, can you send
across client logs?

On Wed, Aug 29, 2018 at 6:11 PM, Richard Neuboeck 
wrote:

> Hi Gluster Community,
>
> I have problems with a glusterfs 'Transport endpoint not connected'
> connection abort during file transfers that I can replicate (all the
> time now) but not pinpoint as to why this is happening.
>
> The volume is set up in replica 3 mode and accessed with the fuse
> gluster client. Both client and server are running CentOS and the
> supplied 3.12.11 version of gluster.
>
> The connection abort happens at different times during rsync but
> occurs every time I try to sync all our files (1.1TB) to the empty
> volume.
>
> Client and server side I don't find errors in the gluster log files.
> rsync logs the obvious transfer problem. The only log that shows
> anything related is the server brick log which states that the
> connection is shutting down:
>
> [2018-08-18 22:40:35.502510] I [MSGID: 115036]
> [server.c:527:server_rpc_notify] 0-home-server: disconnecting
> connection from brax-110405-2018/08/16-08:36:28:575972-home-client-0-0-0
> [2018-08-18 22:40:35.502620] W
> [inodelk.c:499:pl_inodelk_log_cleanup] 0-home-server: releasing lock
> on eaeb0398-fefd-486d-84a7-f13744d1cf10 held by
> {client=0x7f83ec0b3ce0, pid=110423 lk-owner=d0fd5ffb427f}
> [2018-08-18 22:40:35.502692] W
> [entrylk.c:864:pl_entrylk_log_cleanup] 0-home-server: releasing lock
> on faa93f7b-6c46-4251-b2b2-abcd2f2613e1 held by
> {client=0x7f83ec0b3ce0, pid=110423 lk-owner=703dd4cc407f}
> [2018-08-18 22:40:35.502719] W
> [entrylk.c:864:pl_entrylk_log_cleanup] 0-home-server: releasing lock
> on faa93f7b-6c46-4251-b2b2-abcd2f2613e1 held by
> {client=0x7f83ec0b3ce0, pid=110423 lk-owner=703dd4cc407f}
> [2018-08-18 22:40:35.505950] I [MSGID: 101055]
> [client_t.c:443:gf_client_unref] 0-home-server: Shutting down
> connection brax-110405-2018/08/16-08:36:28:575972-home-client-0-0-0
>
> Since I'm running another replica 3 setup for oVirt for a long time
> now which is completely stable I thought I made a mistake setting
> different options at first. However even when I reset those options
> I'm able to reproduce the connection problem.
>
> The unoptimized volume setup looks like this:
>
> Volume Name: home
> Type: Replicate
> Volume ID: c92fa4cc-4a26-41ff-8c70-1dd07f733ac8
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x 3 = 3
> Transport-type: tcp
> Bricks:
> Brick1: sphere-four:/srv/gluster_home/brick
> Brick2: sphere-five:/srv/gluster_home/brick
> Brick3: sphere-six:/srv/gluster_home/brick
> Options Reconfigured:
> nfs.disable: on
> transport.address-family: inet
> cluster.quorum-type: auto
> cluster.server-quorum-type: server
> cluster.server-quorum-ratio: 50%
>
>
> The following additional options were used before:
>
> performance.cache-size: 5GB
> client.event-threads: 4
> server.event-threads: 4
> cluster.lookup-optimize: on
> features.cache-invalidation: on
> performance.stat-prefetch: on
> performance.cache-invalidation: on
> network.inode-lru-limit: 5
> features.cache-invalidation-timeout: 600
> performance.md-cache-timeout: 600
> performance.parallel-readdir: on
>
>
> In this case the gluster servers and also the client is using a
> bonded network device running in adaptive load balancing mode.
>
> I've tried using the debug option for the client mount. But except
> for a ~0.5TB log file I didn't get information that seems helpful to me.
>
> Transferring just a couple of GB works without problems.
>
> It may very well be that I'm already blind to the obvious but after
> many long running tests I can't find the crux in the setup.
>
> Does anyone have an idea as how to approach this problem in a way
> that sheds some useful information?
>
> Any help is highly appreciated!
> Cheers
> Richard
>
> --
> /dev/null
>
>
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not work

[Krishna Verma]

Hi Kotresh,

After fix the library link on node "noi-poc-gluster ", the status of one mater 
node is “Active” and another is “Passive”. Can I setup both the master as 
“Active” ?

Also, when I copy a 1GB size of file from gluster client to master gluster 
volume which is replicated with the slave volume, it tooks 35 minutes and 49 
seconds. Is there any way to reduce its time taken to rsync data.

NOTE: Gluster master server and one client is in Noida, India Location.
 Gluster Slave server and one client is in USA.

Our approach is to transfer data from Noida gluster client will reach to the 
USA gluster client in a minimum time. Please suggest the best approach to 
achieve it.

[root@noi-dcops ~]# date ; rsync -avh --progress /tmp/gentoo_root.img 
/glusterfs/ ; date
Thu Aug 30 12:26:26 IST 2018
sending incremental file list
gentoo_root.img
  1.07G 100%  490.70kB/s0:35:36 (xfr#1, to-chk=0/1)

sent 1.07G bytes  received 35 bytes  499.65K bytes/sec
total size is 1.07G  speedup is 1.00
Thu Aug 30 13:02:15 IST 2018
[root@noi-dcops ~]#


[root@gluster-poc-noida gluster]#  gluster volume geo-replication status

MASTER NODE  MASTER VOLMASTER BRICK SLAVE USERSLAVE 
 SLAVE NODESTATUS CRAWL STATUS   
LAST_SYNCED
---
gluster-poc-noidaglusterep /data/gluster/gv0root  
ssh://gluster-poc-sj::glusterepgluster-poc-sjActive Changelog Crawl 
   2018-08-30 13:42:18
noi-poc-gluster  glusterep /data/gluster/gv0root  
ssh://gluster-poc-sj::glusterepgluster-poc-sjPassiveN/A 
   N/A
[root@gluster-poc-noida gluster]#

Thanks in advance for your all time support.

/Krishna

From: Kotresh Hiremath Ravishankar 
Sent: Thursday, August 30, 2018 10:51 AM
To: Krishna Verma 
Cc: Sunny Kumar ; Gluster Users 
Subject: Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not work

EXTERNAL MAIL
Did you fix the library link on node "noi-poc-gluster " as well?
If not please fix it. Please share the geo-rep log this node if it's
as different issue.
-Kotresh HR

On Thu, Aug 30, 2018 at 12:17 AM, Krishna Verma 
mailto:kve...@cadence.com>> wrote:
Hi Kotresh,

Thank you so much for you input. Geo-replication is now showing “Active” 
atleast for 1 master node. But its still at faulty state for the 2nd master 
server.

Below is the detail.

[root@gluster-poc-noida glusterfs]# gluster volume geo-replication glusterep 
gluster-poc-sj::glusterep status

MASTER NODE  MASTER VOLMASTER BRICK SLAVE USERSLAVE 
   SLAVE NODESTATUSCRAWL STATUS   LAST_SYNCED

gluster-poc-noidaglusterep /data/gluster/gv0root  
gluster-poc-sj::glusterepgluster-poc-sjActiveChangelog Crawl
2018-08-29 23:56:06
noi-poc-gluster  glusterep /data/gluster/gv0root  
gluster-poc-sj::glusterepN/A   FaultyN/AN/A


[root@gluster-poc-noida glusterfs]# gluster volume status
Status of volume: glusterep
Gluster process TCP Port  RDMA Port  Online  Pid
--
Brick gluster-poc-noida:/data/gluster/gv0   49152 0  Y   22463
Brick noi-poc-gluster:/data/gluster/gv0 49152 0  Y   19471
Self-heal Daemon on localhost   N/A   N/AY   32087
Self-heal Daemon on noi-poc-gluster N/A   N/AY   6272

Task Status of Volume glusterep
--
There are no active volume tasks



[root@gluster-poc-noida glusterfs]# gluster volume info

Volume Name: glusterep
Type: Replicate
Volume ID: 4a71bc94-14ce-4b2c-abc4-e6a9a9765161
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: gluster-poc-noida:/data/gluster/gv0
Brick2: noi-poc-gluster:/data/gluster/gv0
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
geo-replication.indexing: on
geo-replication.ignore-pid-check: on
changelog.changelog: on
[root@gluster-poc-noida glusterfs]#

Could you please help me in that also please?

It would be really a great help from your side.

/Krishna
From: Kotresh Hiremath Ravishankar 
mailto:khire...@redhat.com>>
Sent: Wednesday, August 29, 2018 10:47 AM

To: Krishna Verma mailto:kve...@cadence.com>>
Cc: Sunny Kumar mailto:sunku...@redhat.com>>; Gluster 
Users mailto:gluster-users@gluster.org>>
Subject: Re: [Gluster-users] Upgrade to 4.1.2 

Re: [Gluster-users] [External] Re: file metadata operations performance - gluster 4.1

[Raghavendra Gowdappa]

On Thu, Aug 30, 2018 at 8:08 PM, Davide Obbi 
wrote:

> Thanks Amar,
>
> i have enabled the negative lookups cache on the volume:
>
> To deflate a tar archive (not compressed) of 1.3GB it takes aprox 9mins
> which can be considered a slight improvement from the previous 12-15
> however still not fast enough compared to local disk. The tar is present on
> the gluster share/volume and deflated inside the same folder structure.
>

I am assuming this is with parallel-readdir enabled, right?


> Running the operation twice (without removing the already deflated files)
> also did not reduce the time spent.
>
> Running the operation with the tar archive on local disk made no difference
>
> What really made a huge difference while git cloning was setting
> "performance.parallel-readdir on". During the phase "Receiving objects" ,
> as i enabled the xlator it bumped up from 3/4MBs to 27MBs
>

What is the distribute count? Is it 1x3 replica?


> So in conclusion i'm trying to make the untar operation working at an
> acceptable level, not expecting local disks speed but at least being within
> the 4mins
>
> I have attached the profiles collected at the end of the untar operations
> with the archive on the mount and outside
>
> thanks
> Davide
>
>
> On Tue, Aug 28, 2018 at 8:41 AM Amar Tumballi  wrote:
>
>> One of the observation we had with git clone like work load was, nl-cache
>> (negative-lookup cache), helps here.
>>
>> Try 'gluster volume set $volume-name nl-cache enable'.
>>
>> Also sharing the 'profile info' during this performance observations also
>> helps us to narrow down the situation.
>>
>> More on how to capture profile info @ https://hackmd.io/
>> PhhT5jPdQIKxzfeLQmnjJQ?view
>>
>> -Amar
>>
>>
>> On Thu, Aug 23, 2018 at 7:11 PM, Davide Obbi 
>> wrote:
>>
>>> Hello,
>>>
>>> did anyone ever managed to achieve reasonable waiting time while
>>> performing metadata intensive operations such as git clone, untar etc...?
>>> Is this possible workload or will never be in scope for glusterfs?
>>>
>>> I'd like to know, if possible, what would be the options that affect
>>> such volume performances.
>>> Albeit i managed to achieve decent git status/git grep operations, 3 and
>>> 30 secs, the git clone and untarring a file from/to the same share take
>>> ages. for a git repo of aprox 6GB.
>>>
>>> I'm running a test environment with 3 way replica 128GB RAM and 24 cores
>>> are  2.40GHz, one internal SSD dedicated to the volume brick and 10Gb
>>> network
>>>
>>> The options set so far that affects volume performances are:
>>>  48 performance.readdir-ahead: on
>>>  49 features.cache-invalidation-timeout: 600
>>>  50 features.cache-invalidation: on
>>>  51 performance.md-cache-timeout: 600
>>>  52 performance.stat-prefetch: on
>>>  53 performance.cache-invalidation: on
>>>  54 performance.parallel-readdir: on
>>>  55 network.inode-lru-limit: 90
>>>  56 performance.io-thread-count: 32
>>>  57 performance.cache-size: 10GB
>>>
>>> ___
>>> Gluster-users mailing list
>>> Gluster-users@gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>
>>
>>
>> --
>> Amar Tumballi (amarts)
>>
>
>
> --
> Davide Obbi
> System Administrator
>
> Booking.com B.V.
> Vijzelstraat 66-80 Amsterdam 1017HL Netherlands
> Direct +31207031558
> [image: Booking.com] 
> The world's #1 accommodation site
> 43 languages, 198+ offices worldwide, 120,000+ global destinations,
> 1,550,000+ room nights booked every day
> No booking fees, best price always guaranteed
> Subsidiary of Booking Holdings Inc. (NASDAQ: BKNG)
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users