Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Hu Bert]

Hi Pranith,

i recently upgraded to version 3.12.14, still no change in
load/performance. Have you received any feedback?

At the moment i have 3 options:
- problem can be fixed within version 3.12
- upgrade to 4.1 and magically/hopefully "fix" the problem (might not
help when problem is within brick)
- replace glusterfs with $whatever (defeat... :-( )

thx
Hubert


2018-09-03 7:55 GMT+02:00 Pranith Kumar Karampuri :
>
>
> On Fri, Aug 31, 2018 at 1:18 PM Hu Bert  wrote:
>>
>> Hi Pranith,
>>
>> i just wanted to ask if you were able to get any feedback from your
>> colleagues :-)
>
>
> Sorry, I didn't get a chance to. I am working on a customer issue which is
> taking away cycles from any other work. Let me get back to you once I get
> time this week.
>
>>
>>
>> btw.: we migrated some stuff (static resources, small files) to a nfs
>> server that we actually wanted to replace by glusterfs. Load and cpu
>> usage has gone down a bit, but still is asymmetric on the 3 gluster
>> servers.
>>
>>
>> 2018-08-28 9:24 GMT+02:00 Hu Bert :
>> > Hm, i noticed that in the shared.log (volume log file) on gluster11
>> > and gluster12 (but not on gluster13) i now see these warnings:
>> >
>> > [2018-08-28 07:18:57.224367] W [MSGID: 109011]
>> > [dht-layout.c:186:dht_layout_search] 0-shared-dht: no subvolume for
>> > hash (value) = 3054593291
>> > [2018-08-28 07:19:17.733625] W [MSGID: 109011]
>> > [dht-layout.c:186:dht_layout_search] 0-shared-dht: no subvolume for
>> > hash (value) = 2595205890
>> > [2018-08-28 07:19:27.950355] W [MSGID: 109011]
>> > [dht-layout.c:186:dht_layout_search] 0-shared-dht: no subvolume for
>> > hash (value) = 3105728076
>> > [2018-08-28 07:19:42.519010] W [MSGID: 109011]
>> > [dht-layout.c:186:dht_layout_search] 0-shared-dht: no subvolume for
>> > hash (value) = 3740415196
>> > [2018-08-28 07:19:48.194774] W [MSGID: 109011]
>> > [dht-layout.c:186:dht_layout_search] 0-shared-dht: no subvolume for
>> > hash (value) = 2922795043
>> > [2018-08-28 07:19:52.506135] W [MSGID: 109011]
>> > [dht-layout.c:186:dht_layout_search] 0-shared-dht: no subvolume for
>> > hash (value) = 2841655539
>> > [2018-08-28 07:19:55.466352] W [MSGID: 109011]
>> > [dht-layout.c:186:dht_layout_search] 0-shared-dht: no subvolume for
>> > hash (value) = 3049465001
>> >
>> > Don't know if that could be related.
>> >
>> >
>> > 2018-08-28 8:54 GMT+02:00 Hu Bert :
>> >> a little update after about 2 hours of uptime: still/again high cpu
>> >> usage by one brick processes. server load >30.
>> >>
>> >> gluster11: high cpu; brick /gluster/bricksdd1/; no hdd exchange so far
>> >> gluster12: normal cpu; brick /gluster/bricksdd1_new/; hdd change
>> >> /dev/sdd
>> >> gluster13: high cpu; brick /gluster/bricksdd1_new/; hdd change /dev/sdd
>> >>
>> >> The process for brick bricksdd1 consumes almost all 12 cores.
>> >> Interestingly there are more threads for the bricksdd1 process than
>> >> for the other bricks. Counted with "ps huH p  | wc
>> >> -l"
>> >>
>> >> gluster11:
>> >> bricksda1 59 threads, bricksdb1 65 threads, bricksdc1 68 threads,
>> >> bricksdd1 85 threads
>> >> gluster12:
>> >> bricksda1 65 threads, bricksdb1 60 threads, bricksdc1 61 threads,
>> >> bricksdd1_new 58 threads
>> >> gluster13:
>> >> bricksda1 61 threads, bricksdb1 60 threads, bricksdc1 61 threads,
>> >> bricksdd1_new 82 threads
>> >>
>> >> Don't know if that could be relevant.
>> >>
>> >> 2018-08-28 7:04 GMT+02:00 Hu Bert :
>> >>> Good Morning,
>> >>>
>> >>> today i update + rebooted all gluster servers, kernel update to
>> >>> 4.9.0-8 and gluster to 3.12.13. Reboots went fine, but on one of the
>> >>> gluster servers (gluster13) one of the bricks did come up at the
>> >>> beginning but then lost connection.
>> >>>
>> >>> OK:
>> >>>
>> >>> Status of volume: shared
>> >>> Gluster process TCP Port  RDMA Port
>> >>> Online  Pid
>> >>>
>> >>> --
>> >>> [...]
>> >>> Brick gluster11:/gluster/bricksdd1/shared 49155 0
>> >>> Y   2506
>> >>> Brick gluster12:/gluster/bricksdd1_new/shared49155 0
>> >>> Y   2097
>> >>> Brick gluster13:/gluster/bricksdd1_new/shared49155 0
>> >>> Y   2136
>> >>>
>> >>> Lost connection:
>> >>>
>> >>> Brick gluster11:/gluster/bricksdd1/shared  49155 0
>> >>>  Y   2506
>> >>> Brick gluster12:/gluster/bricksdd1_new/shared 49155 0
>> >>> Y   2097
>> >>> Brick gluster13:/gluster/bricksdd1_new/shared N/A   N/A
>> >>> N   N/A
>> >>>
>> >>> gluster volume heal shared info:
>> >>> Brick gluster13:/gluster/bricksdd1_new/shared
>> >>> Status: Transport endpoint is not connected
>> >>> Number of entries: -
>> >>>
>> >>> reboot was at 06:15:39; brick then worked for a short period, but then
>> >>> somehow disconnected.
>> >>>
>> >>> from gluster13:/var/log/glusterfs/glusterd.log:
>> >>>
>> >>> [2018-08-28 04:27:36.944608] I [MSGID: 106005]
>> >>> 

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Pranith Kumar Karampuri]

On Fri, Aug 31, 2018 at 1:18 PM Hu Bert  wrote:

> Hi Pranith,
>
> i just wanted to ask if you were able to get any feedback from your
> colleagues :-)
>

Sorry, I didn't get a chance to. I am working on a customer issue which is
taking away cycles from any other work. Let me get back to you once I get
time this week.


>
> btw.: we migrated some stuff (static resources, small files) to a nfs
> server that we actually wanted to replace by glusterfs. Load and cpu
> usage has gone down a bit, but still is asymmetric on the 3 gluster
> servers.
>
>
> 2018-08-28 9:24 GMT+02:00 Hu Bert :
> > Hm, i noticed that in the shared.log (volume log file) on gluster11
> > and gluster12 (but not on gluster13) i now see these warnings:
> >
> > [2018-08-28 07:18:57.224367] W [MSGID: 109011]
> > [dht-layout.c:186:dht_layout_search] 0-shared-dht: no subvolume for
> > hash (value) = 3054593291
> > [2018-08-28 07:19:17.733625] W [MSGID: 109011]
> > [dht-layout.c:186:dht_layout_search] 0-shared-dht: no subvolume for
> > hash (value) = 2595205890
> > [2018-08-28 07:19:27.950355] W [MSGID: 109011]
> > [dht-layout.c:186:dht_layout_search] 0-shared-dht: no subvolume for
> > hash (value) = 3105728076
> > [2018-08-28 07:19:42.519010] W [MSGID: 109011]
> > [dht-layout.c:186:dht_layout_search] 0-shared-dht: no subvolume for
> > hash (value) = 3740415196
> > [2018-08-28 07:19:48.194774] W [MSGID: 109011]
> > [dht-layout.c:186:dht_layout_search] 0-shared-dht: no subvolume for
> > hash (value) = 2922795043
> > [2018-08-28 07:19:52.506135] W [MSGID: 109011]
> > [dht-layout.c:186:dht_layout_search] 0-shared-dht: no subvolume for
> > hash (value) = 2841655539
> > [2018-08-28 07:19:55.466352] W [MSGID: 109011]
> > [dht-layout.c:186:dht_layout_search] 0-shared-dht: no subvolume for
> > hash (value) = 3049465001
> >
> > Don't know if that could be related.
> >
> >
> > 2018-08-28 8:54 GMT+02:00 Hu Bert :
> >> a little update after about 2 hours of uptime: still/again high cpu
> >> usage by one brick processes. server load >30.
> >>
> >> gluster11: high cpu; brick /gluster/bricksdd1/; no hdd exchange so far
> >> gluster12: normal cpu; brick /gluster/bricksdd1_new/; hdd change
> /dev/sdd
> >> gluster13: high cpu; brick /gluster/bricksdd1_new/; hdd change /dev/sdd
> >>
> >> The process for brick bricksdd1 consumes almost all 12 cores.
> >> Interestingly there are more threads for the bricksdd1 process than
> >> for the other bricks. Counted with "ps huH p  | wc
> >> -l"
> >>
> >> gluster11:
> >> bricksda1 59 threads, bricksdb1 65 threads, bricksdc1 68 threads,
> >> bricksdd1 85 threads
> >> gluster12:
> >> bricksda1 65 threads, bricksdb1 60 threads, bricksdc1 61 threads,
> >> bricksdd1_new 58 threads
> >> gluster13:
> >> bricksda1 61 threads, bricksdb1 60 threads, bricksdc1 61 threads,
> >> bricksdd1_new 82 threads
> >>
> >> Don't know if that could be relevant.
> >>
> >> 2018-08-28 7:04 GMT+02:00 Hu Bert :
> >>> Good Morning,
> >>>
> >>> today i update + rebooted all gluster servers, kernel update to
> >>> 4.9.0-8 and gluster to 3.12.13. Reboots went fine, but on one of the
> >>> gluster servers (gluster13) one of the bricks did come up at the
> >>> beginning but then lost connection.
> >>>
> >>> OK:
> >>>
> >>> Status of volume: shared
> >>> Gluster process TCP Port  RDMA Port
> Online  Pid
> >>>
> --
> >>> [...]
> >>> Brick gluster11:/gluster/bricksdd1/shared 49155 0
> >>> Y   2506
> >>> Brick gluster12:/gluster/bricksdd1_new/shared49155 0
> >>> Y   2097
> >>> Brick gluster13:/gluster/bricksdd1_new/shared49155 0
> >>> Y   2136
> >>>
> >>> Lost connection:
> >>>
> >>> Brick gluster11:/gluster/bricksdd1/shared  49155 0
> >>>  Y   2506
> >>> Brick gluster12:/gluster/bricksdd1_new/shared 49155 0
> >>> Y   2097
> >>> Brick gluster13:/gluster/bricksdd1_new/shared N/A   N/A
> >>> N   N/A
> >>>
> >>> gluster volume heal shared info:
> >>> Brick gluster13:/gluster/bricksdd1_new/shared
> >>> Status: Transport endpoint is not connected
> >>> Number of entries: -
> >>>
> >>> reboot was at 06:15:39; brick then worked for a short period, but then
> >>> somehow disconnected.
> >>>
> >>> from gluster13:/var/log/glusterfs/glusterd.log:
> >>>
> >>> [2018-08-28 04:27:36.944608] I [MSGID: 106005]
> >>> [glusterd-handler.c:6071:__glusterd_brick_rpc_notify] 0-management:
> >>> Brick gluster13:/gluster/bricksdd1_new/shared has disconnected from
> >>> glusterd.
> >>> [2018-08-28 04:28:57.869666] I
> >>> [glusterd-utils.c:6056:glusterd_brick_start] 0-management: starting a
> >>> fresh brick process for brick /gluster/bricksdd1_new/shared
> >>> [2018-08-28 04:35:20.732666] I [MSGID: 106143]
> >>> [glusterd-pmap.c:295:pmap_registry_bind] 0-pmap: adding brick
> >>> /gluster/bricksdd1_new/shared on port 49157
> >>>
> >>> After 'gluster volume start shared force' (then with new 

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Hu Bert]

Hi Pranith,

i just wanted to ask if you were able to get any feedback from your
colleagues :-)

btw.: we migrated some stuff (static resources, small files) to a nfs
server that we actually wanted to replace by glusterfs. Load and cpu
usage has gone down a bit, but still is asymmetric on the 3 gluster
servers.


2018-08-28 9:24 GMT+02:00 Hu Bert :
> Hm, i noticed that in the shared.log (volume log file) on gluster11
> and gluster12 (but not on gluster13) i now see these warnings:
>
> [2018-08-28 07:18:57.224367] W [MSGID: 109011]
> [dht-layout.c:186:dht_layout_search] 0-shared-dht: no subvolume for
> hash (value) = 3054593291
> [2018-08-28 07:19:17.733625] W [MSGID: 109011]
> [dht-layout.c:186:dht_layout_search] 0-shared-dht: no subvolume for
> hash (value) = 2595205890
> [2018-08-28 07:19:27.950355] W [MSGID: 109011]
> [dht-layout.c:186:dht_layout_search] 0-shared-dht: no subvolume for
> hash (value) = 3105728076
> [2018-08-28 07:19:42.519010] W [MSGID: 109011]
> [dht-layout.c:186:dht_layout_search] 0-shared-dht: no subvolume for
> hash (value) = 3740415196
> [2018-08-28 07:19:48.194774] W [MSGID: 109011]
> [dht-layout.c:186:dht_layout_search] 0-shared-dht: no subvolume for
> hash (value) = 2922795043
> [2018-08-28 07:19:52.506135] W [MSGID: 109011]
> [dht-layout.c:186:dht_layout_search] 0-shared-dht: no subvolume for
> hash (value) = 2841655539
> [2018-08-28 07:19:55.466352] W [MSGID: 109011]
> [dht-layout.c:186:dht_layout_search] 0-shared-dht: no subvolume for
> hash (value) = 3049465001
>
> Don't know if that could be related.
>
>
> 2018-08-28 8:54 GMT+02:00 Hu Bert :
>> a little update after about 2 hours of uptime: still/again high cpu
>> usage by one brick processes. server load >30.
>>
>> gluster11: high cpu; brick /gluster/bricksdd1/; no hdd exchange so far
>> gluster12: normal cpu; brick /gluster/bricksdd1_new/; hdd change /dev/sdd
>> gluster13: high cpu; brick /gluster/bricksdd1_new/; hdd change /dev/sdd
>>
>> The process for brick bricksdd1 consumes almost all 12 cores.
>> Interestingly there are more threads for the bricksdd1 process than
>> for the other bricks. Counted with "ps huH p  | wc
>> -l"
>>
>> gluster11:
>> bricksda1 59 threads, bricksdb1 65 threads, bricksdc1 68 threads,
>> bricksdd1 85 threads
>> gluster12:
>> bricksda1 65 threads, bricksdb1 60 threads, bricksdc1 61 threads,
>> bricksdd1_new 58 threads
>> gluster13:
>> bricksda1 61 threads, bricksdb1 60 threads, bricksdc1 61 threads,
>> bricksdd1_new 82 threads
>>
>> Don't know if that could be relevant.
>>
>> 2018-08-28 7:04 GMT+02:00 Hu Bert :
>>> Good Morning,
>>>
>>> today i update + rebooted all gluster servers, kernel update to
>>> 4.9.0-8 and gluster to 3.12.13. Reboots went fine, but on one of the
>>> gluster servers (gluster13) one of the bricks did come up at the
>>> beginning but then lost connection.
>>>
>>> OK:
>>>
>>> Status of volume: shared
>>> Gluster process TCP Port  RDMA Port  Online  Pid
>>> --
>>> [...]
>>> Brick gluster11:/gluster/bricksdd1/shared 49155 0
>>> Y   2506
>>> Brick gluster12:/gluster/bricksdd1_new/shared49155 0
>>> Y   2097
>>> Brick gluster13:/gluster/bricksdd1_new/shared49155 0
>>> Y   2136
>>>
>>> Lost connection:
>>>
>>> Brick gluster11:/gluster/bricksdd1/shared  49155 0
>>>  Y   2506
>>> Brick gluster12:/gluster/bricksdd1_new/shared 49155 0
>>> Y   2097
>>> Brick gluster13:/gluster/bricksdd1_new/shared N/A   N/A
>>> N   N/A
>>>
>>> gluster volume heal shared info:
>>> Brick gluster13:/gluster/bricksdd1_new/shared
>>> Status: Transport endpoint is not connected
>>> Number of entries: -
>>>
>>> reboot was at 06:15:39; brick then worked for a short period, but then
>>> somehow disconnected.
>>>
>>> from gluster13:/var/log/glusterfs/glusterd.log:
>>>
>>> [2018-08-28 04:27:36.944608] I [MSGID: 106005]
>>> [glusterd-handler.c:6071:__glusterd_brick_rpc_notify] 0-management:
>>> Brick gluster13:/gluster/bricksdd1_new/shared has disconnected from
>>> glusterd.
>>> [2018-08-28 04:28:57.869666] I
>>> [glusterd-utils.c:6056:glusterd_brick_start] 0-management: starting a
>>> fresh brick process for brick /gluster/bricksdd1_new/shared
>>> [2018-08-28 04:35:20.732666] I [MSGID: 106143]
>>> [glusterd-pmap.c:295:pmap_registry_bind] 0-pmap: adding brick
>>> /gluster/bricksdd1_new/shared on port 49157
>>>
>>> After 'gluster volume start shared force' (then with new port 49157):
>>>
>>> Brick gluster11:/gluster/bricksdd1/shared   49155 0
>>>   Y   2506
>>> Brick gluster12:/gluster/bricksdd1_new/shared  49155 0
>>>  Y   2097
>>> Brick gluster13:/gluster/bricksdd1_new/shared  49157 0
>>>  Y   3994
>>>
>>> from /var/log/syslog:
>>>
>>> Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]: pending 
>>> frames:
>>> Aug 28 06:27:36 gluster13 

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Hu Bert]

Hm, i noticed that in the shared.log (volume log file) on gluster11
and gluster12 (but not on gluster13) i now see these warnings:

[2018-08-28 07:18:57.224367] W [MSGID: 109011]
[dht-layout.c:186:dht_layout_search] 0-shared-dht: no subvolume for
hash (value) = 3054593291
[2018-08-28 07:19:17.733625] W [MSGID: 109011]
[dht-layout.c:186:dht_layout_search] 0-shared-dht: no subvolume for
hash (value) = 2595205890
[2018-08-28 07:19:27.950355] W [MSGID: 109011]
[dht-layout.c:186:dht_layout_search] 0-shared-dht: no subvolume for
hash (value) = 3105728076
[2018-08-28 07:19:42.519010] W [MSGID: 109011]
[dht-layout.c:186:dht_layout_search] 0-shared-dht: no subvolume for
hash (value) = 3740415196
[2018-08-28 07:19:48.194774] W [MSGID: 109011]
[dht-layout.c:186:dht_layout_search] 0-shared-dht: no subvolume for
hash (value) = 2922795043
[2018-08-28 07:19:52.506135] W [MSGID: 109011]
[dht-layout.c:186:dht_layout_search] 0-shared-dht: no subvolume for
hash (value) = 2841655539
[2018-08-28 07:19:55.466352] W [MSGID: 109011]
[dht-layout.c:186:dht_layout_search] 0-shared-dht: no subvolume for
hash (value) = 3049465001

Don't know if that could be related.


2018-08-28 8:54 GMT+02:00 Hu Bert :
> a little update after about 2 hours of uptime: still/again high cpu
> usage by one brick processes. server load >30.
>
> gluster11: high cpu; brick /gluster/bricksdd1/; no hdd exchange so far
> gluster12: normal cpu; brick /gluster/bricksdd1_new/; hdd change /dev/sdd
> gluster13: high cpu; brick /gluster/bricksdd1_new/; hdd change /dev/sdd
>
> The process for brick bricksdd1 consumes almost all 12 cores.
> Interestingly there are more threads for the bricksdd1 process than
> for the other bricks. Counted with "ps huH p  | wc
> -l"
>
> gluster11:
> bricksda1 59 threads, bricksdb1 65 threads, bricksdc1 68 threads,
> bricksdd1 85 threads
> gluster12:
> bricksda1 65 threads, bricksdb1 60 threads, bricksdc1 61 threads,
> bricksdd1_new 58 threads
> gluster13:
> bricksda1 61 threads, bricksdb1 60 threads, bricksdc1 61 threads,
> bricksdd1_new 82 threads
>
> Don't know if that could be relevant.
>
> 2018-08-28 7:04 GMT+02:00 Hu Bert :
>> Good Morning,
>>
>> today i update + rebooted all gluster servers, kernel update to
>> 4.9.0-8 and gluster to 3.12.13. Reboots went fine, but on one of the
>> gluster servers (gluster13) one of the bricks did come up at the
>> beginning but then lost connection.
>>
>> OK:
>>
>> Status of volume: shared
>> Gluster process TCP Port  RDMA Port  Online  Pid
>> --
>> [...]
>> Brick gluster11:/gluster/bricksdd1/shared 49155 0
>> Y   2506
>> Brick gluster12:/gluster/bricksdd1_new/shared49155 0
>> Y   2097
>> Brick gluster13:/gluster/bricksdd1_new/shared49155 0
>> Y   2136
>>
>> Lost connection:
>>
>> Brick gluster11:/gluster/bricksdd1/shared  49155 0
>>  Y   2506
>> Brick gluster12:/gluster/bricksdd1_new/shared 49155 0
>> Y   2097
>> Brick gluster13:/gluster/bricksdd1_new/shared N/A   N/A
>> N   N/A
>>
>> gluster volume heal shared info:
>> Brick gluster13:/gluster/bricksdd1_new/shared
>> Status: Transport endpoint is not connected
>> Number of entries: -
>>
>> reboot was at 06:15:39; brick then worked for a short period, but then
>> somehow disconnected.
>>
>> from gluster13:/var/log/glusterfs/glusterd.log:
>>
>> [2018-08-28 04:27:36.944608] I [MSGID: 106005]
>> [glusterd-handler.c:6071:__glusterd_brick_rpc_notify] 0-management:
>> Brick gluster13:/gluster/bricksdd1_new/shared has disconnected from
>> glusterd.
>> [2018-08-28 04:28:57.869666] I
>> [glusterd-utils.c:6056:glusterd_brick_start] 0-management: starting a
>> fresh brick process for brick /gluster/bricksdd1_new/shared
>> [2018-08-28 04:35:20.732666] I [MSGID: 106143]
>> [glusterd-pmap.c:295:pmap_registry_bind] 0-pmap: adding brick
>> /gluster/bricksdd1_new/shared on port 49157
>>
>> After 'gluster volume start shared force' (then with new port 49157):
>>
>> Brick gluster11:/gluster/bricksdd1/shared   49155 0
>>   Y   2506
>> Brick gluster12:/gluster/bricksdd1_new/shared  49155 0
>>  Y   2097
>> Brick gluster13:/gluster/bricksdd1_new/shared  49157 0
>>  Y   3994
>>
>> from /var/log/syslog:
>>
>> Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]: pending frames:
>> Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]: frame :
>> type(0) op(0)
>> Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]: frame :
>> type(0) op(0)
>> Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]:
>> patchset: git://git.gluster.org/glusterfs.git
>> Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]: signal
>> received: 11
>> Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]: time of crash:
>> Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]:
>> 2018-08-28 

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Hu Bert]

a little update after about 2 hours of uptime: still/again high cpu
usage by one brick processes. server load >30.

gluster11: high cpu; brick /gluster/bricksdd1/; no hdd exchange so far
gluster12: normal cpu; brick /gluster/bricksdd1_new/; hdd change /dev/sdd
gluster13: high cpu; brick /gluster/bricksdd1_new/; hdd change /dev/sdd

The process for brick bricksdd1 consumes almost all 12 cores.
Interestingly there are more threads for the bricksdd1 process than
for the other bricks. Counted with "ps huH p  | wc
-l"

gluster11:
bricksda1 59 threads, bricksdb1 65 threads, bricksdc1 68 threads,
bricksdd1 85 threads
gluster12:
bricksda1 65 threads, bricksdb1 60 threads, bricksdc1 61 threads,
bricksdd1_new 58 threads
gluster13:
bricksda1 61 threads, bricksdb1 60 threads, bricksdc1 61 threads,
bricksdd1_new 82 threads

Don't know if that could be relevant.

2018-08-28 7:04 GMT+02:00 Hu Bert :
> Good Morning,
>
> today i update + rebooted all gluster servers, kernel update to
> 4.9.0-8 and gluster to 3.12.13. Reboots went fine, but on one of the
> gluster servers (gluster13) one of the bricks did come up at the
> beginning but then lost connection.
>
> OK:
>
> Status of volume: shared
> Gluster process TCP Port  RDMA Port  Online  Pid
> --
> [...]
> Brick gluster11:/gluster/bricksdd1/shared 49155 0
> Y   2506
> Brick gluster12:/gluster/bricksdd1_new/shared49155 0
> Y   2097
> Brick gluster13:/gluster/bricksdd1_new/shared49155 0
> Y   2136
>
> Lost connection:
>
> Brick gluster11:/gluster/bricksdd1/shared  49155 0
>  Y   2506
> Brick gluster12:/gluster/bricksdd1_new/shared 49155 0
> Y   2097
> Brick gluster13:/gluster/bricksdd1_new/shared N/A   N/A
> N   N/A
>
> gluster volume heal shared info:
> Brick gluster13:/gluster/bricksdd1_new/shared
> Status: Transport endpoint is not connected
> Number of entries: -
>
> reboot was at 06:15:39; brick then worked for a short period, but then
> somehow disconnected.
>
> from gluster13:/var/log/glusterfs/glusterd.log:
>
> [2018-08-28 04:27:36.944608] I [MSGID: 106005]
> [glusterd-handler.c:6071:__glusterd_brick_rpc_notify] 0-management:
> Brick gluster13:/gluster/bricksdd1_new/shared has disconnected from
> glusterd.
> [2018-08-28 04:28:57.869666] I
> [glusterd-utils.c:6056:glusterd_brick_start] 0-management: starting a
> fresh brick process for brick /gluster/bricksdd1_new/shared
> [2018-08-28 04:35:20.732666] I [MSGID: 106143]
> [glusterd-pmap.c:295:pmap_registry_bind] 0-pmap: adding brick
> /gluster/bricksdd1_new/shared on port 49157
>
> After 'gluster volume start shared force' (then with new port 49157):
>
> Brick gluster11:/gluster/bricksdd1/shared   49155 0
>   Y   2506
> Brick gluster12:/gluster/bricksdd1_new/shared  49155 0
>  Y   2097
> Brick gluster13:/gluster/bricksdd1_new/shared  49157 0
>  Y   3994
>
> from /var/log/syslog:
>
> Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]: pending frames:
> Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]: frame :
> type(0) op(0)
> Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]: frame :
> type(0) op(0)
> Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]:
> patchset: git://git.gluster.org/glusterfs.git
> Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]: signal
> received: 11
> Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]: time of crash:
> Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]:
> 2018-08-28 04:27:36
> Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]:
> configuration details:
> Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]: argp 1
> Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]: backtrace 1
> Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]: dlfcn 1
> Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]: libpthread 1
> Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]: llistxattr 1
> Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]: setfsid 1
> Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]: spinlock 1
> Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]: epoll.h 1
> Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]: xattr.h 1
> Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]: st_atim.tv_nsec 
> 1
> Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]:
> package-string: glusterfs 3.12.13
> Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]: -
>
> There are some errors+warnings in the shared.log (volume logfile), but
> no error message telling me why
> gluster13:/gluster/bricksdd1_new/shared has disconnected.
>
> Well... at the moment load is ok, all 3 servers at about 15 (but i
> think it will go up when more users will cause more 

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Hu Bert]

Good Morning,

today i update + rebooted all gluster servers, kernel update to
4.9.0-8 and gluster to 3.12.13. Reboots went fine, but on one of the
gluster servers (gluster13) one of the bricks did come up at the
beginning but then lost connection.

OK:

Status of volume: shared
Gluster process TCP Port  RDMA Port  Online  Pid
--
[...]
Brick gluster11:/gluster/bricksdd1/shared 49155 0
Y   2506
Brick gluster12:/gluster/bricksdd1_new/shared49155 0
Y   2097
Brick gluster13:/gluster/bricksdd1_new/shared49155 0
Y   2136

Lost connection:

Brick gluster11:/gluster/bricksdd1/shared  49155 0
 Y   2506
Brick gluster12:/gluster/bricksdd1_new/shared 49155 0
Y   2097
Brick gluster13:/gluster/bricksdd1_new/shared N/A   N/A
N   N/A

gluster volume heal shared info:
Brick gluster13:/gluster/bricksdd1_new/shared
Status: Transport endpoint is not connected
Number of entries: -

reboot was at 06:15:39; brick then worked for a short period, but then
somehow disconnected.

from gluster13:/var/log/glusterfs/glusterd.log:

[2018-08-28 04:27:36.944608] I [MSGID: 106005]
[glusterd-handler.c:6071:__glusterd_brick_rpc_notify] 0-management:
Brick gluster13:/gluster/bricksdd1_new/shared has disconnected from
glusterd.
[2018-08-28 04:28:57.869666] I
[glusterd-utils.c:6056:glusterd_brick_start] 0-management: starting a
fresh brick process for brick /gluster/bricksdd1_new/shared
[2018-08-28 04:35:20.732666] I [MSGID: 106143]
[glusterd-pmap.c:295:pmap_registry_bind] 0-pmap: adding brick
/gluster/bricksdd1_new/shared on port 49157

After 'gluster volume start shared force' (then with new port 49157):

Brick gluster11:/gluster/bricksdd1/shared   49155 0
  Y   2506
Brick gluster12:/gluster/bricksdd1_new/shared  49155 0
 Y   2097
Brick gluster13:/gluster/bricksdd1_new/shared  49157 0
 Y   3994

from /var/log/syslog:

Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]: pending frames:
Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]: frame :
type(0) op(0)
Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]: frame :
type(0) op(0)
Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]:
patchset: git://git.gluster.org/glusterfs.git
Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]: signal
received: 11
Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]: time of crash:
Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]:
2018-08-28 04:27:36
Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]:
configuration details:
Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]: argp 1
Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]: backtrace 1
Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]: dlfcn 1
Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]: libpthread 1
Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]: llistxattr 1
Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]: setfsid 1
Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]: spinlock 1
Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]: epoll.h 1
Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]: xattr.h 1
Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]: st_atim.tv_nsec 1
Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]:
package-string: glusterfs 3.12.13
Aug 28 06:27:36 gluster13 gluster-bricksdd1_new-shared[2136]: -

There are some errors+warnings in the shared.log (volume logfile), but
no error message telling me why
gluster13:/gluster/bricksdd1_new/shared has disconnected.

Well... at the moment load is ok, all 3 servers at about 15 (but i
think it will go up when more users will cause more traffic -> more
work on servers), 'gluster volume heal shared info' shows no entries,
status:

Status of volume: shared
Gluster process TCP Port  RDMA Port  Online  Pid
--
Brick gluster11:/gluster/bricksda1/shared   49152 0  Y   2482
Brick gluster12:/gluster/bricksda1/shared   49152 0  Y   2088
Brick gluster13:/gluster/bricksda1/shared   49152 0  Y   2115
Brick gluster11:/gluster/bricksdb1/shared   49153 0  Y   2489
Brick gluster12:/gluster/bricksdb1/shared   49153 0  Y   2094
Brick gluster13:/gluster/bricksdb1/shared   49153 0  Y   2116
Brick gluster11:/gluster/bricksdc1/shared   49154 0  Y   2497
Brick gluster12:/gluster/bricksdc1/shared   49154 0  Y   2095
Brick gluster13:/gluster/bricksdc1/shared   49154 0  Y   2127
Brick gluster11:/gluster/bricksdd1/shared   49155 0  Y   2506
Brick 

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Hu Bert]

yeah, on debian xyz.log.1 is always the former logfile which has been
rotated by logrotate. Just checked the 3 servers: now it looks good, i
will check it again tomorrow. very strange, maybe logrotate hasn't
worked properly.

the performance problems remain :-)

2018-08-27 15:41 GMT+02:00 Milind Changire :
> On Thu, Aug 23, 2018 at 5:28 PM, Pranith Kumar Karampuri
>  wrote:
>>
>> On Wed, Aug 22, 2018 at 12:01 PM Hu Bert  wrote:
>>>
>>> Just an addition: in general there are no log messages in
>>> /var/log/glusterfs/ (if you don't all 'gluster volume ...'), but on
>>> the node with the lowest load i see in cli.log.1:
>>>
>>> [2018-08-22 06:20:43.291055] I [socket.c:2474:socket_event_handler]
>>> 0-transport: EPOLLERR - disconnecting now
>>> [2018-08-22 06:20:46.291327] I [socket.c:2474:socket_event_handler]
>>> 0-transport: EPOLLERR - disconnecting now
>>> [2018-08-22 06:20:49.291575] I [socket.c:2474:socket_event_handler]
>>> 0-transport: EPOLLERR - disconnecting now
>>>
>>> every 3 seconds. Looks like this bug:
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1484885 - but that shoud
>>> have been fixed in the 3.12.x release, and network is fine.
>>
>>
>> +Milind Changire
>
> That's odd. Presuming cli.log.1 is the logrotated file, it should be showing
> older log entries than cli.log. But its not the case here.
> Or maybe, there's something running on the command-line on the node with the
> lowest load.
>
>>
>>>
>>> In cli.log there are only these entries:
>>>
>>> [2018-08-22 06:19:23.428520] I [cli.c:765:main] 0-cli: Started running
>>> gluster with version 3.12.12
>>> [2018-08-22 06:19:23.800895] I [MSGID: 101190]
>>> [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started
>>> thread with index 1
>>> [2018-08-22 06:19:23.800978] I [socket.c:2474:socket_event_handler]
>>> 0-transport: EPOLLERR - disconnecting now
>>> [2018-08-22 06:19:23.809366] I [input.c:31:cli_batch] 0-: Exiting with: 0
>>>
>>> Just wondered if this could related anyhow.
>>>
>>> 2018-08-21 8:17 GMT+02:00 Pranith Kumar Karampuri :
>>> >
>>> >
>>> > On Tue, Aug 21, 2018 at 11:40 AM Hu Bert 
>>> > wrote:
>>> >>
>>> >> Good morning :-)
>>> >>
>>> >> gluster11:
>>> >> ls -l /gluster/bricksdd1/shared/.glusterfs/indices/xattrop/
>>> >> total 0
>>> >> -- 1 root root 0 Aug 14 06:14
>>> >> xattrop-006b65d8-9e81-4886-b380-89168ea079bd
>>> >>
>>> >> gluster12:
>>> >> ls -l /gluster/bricksdd1_new/shared/.glusterfs/indices/xattrop/
>>> >> total 0
>>> >> -- 1 root root 0 Jul 17 11:24
>>> >> xattrop-c7c6f765-ce17-4361-95fb-2fd7f31c7b82
>>> >>
>>> >> gluster13:
>>> >> ls -l /gluster/bricksdd1_new/shared/.glusterfs/indices/xattrop/
>>> >> total 0
>>> >> -- 1 root root 0 Aug 16 07:54
>>> >> xattrop-16b696a0-4214-4999-b277-0917c76c983e
>>> >>
>>> >>
>>> >> And here's the output of 'perf ...' which ran almost a minute - file
>>> >> grew pretty fast to a size of 17 GB and system load went up heavily.
>>> >> Had to wait a while until load dropped :-)
>>> >>
>>> >> fyi - load at the moment:
>>> >> load gluster11: ~90
>>> >> load gluster12: ~10
>>> >> load gluster13: ~50
>>> >>
>>> >> perf record --call-graph=dwarf -p 7897 -o
>>> >> /tmp/perf.gluster11.bricksdd1.out
>>> >> [ perf record: Woken up 9837 times to write data ]
>>> >> Warning:
>>> >> Processed 2137218 events and lost 33446 chunks!
>>> >>
>>> >> Check IO/CPU overload!
>>> >>
>>> >> [ perf record: Captured and wrote 16576.374 MB
>>> >> /tmp/perf.gluster11.bricksdd1.out (2047760 samples) ]
>>> >>
>>> >> Here's an excerpt.
>>> >>
>>> >> +1.93% 0.00%  glusteriotwr0[unknown]  [k]
>>> >> 0x
>>> >> +1.89% 0.00%  glusteriotwr28   [unknown]  [k]
>>> >> 0x
>>> >> +1.86% 0.00%  glusteriotwr15   [unknown]  [k]
>>> >> 0x
>>> >> +1.85% 0.00%  glusteriotwr63   [unknown]  [k]
>>> >> 0x
>>> >> +1.83% 0.01%  glusteriotwr0[kernel.kallsyms]  [k]
>>> >> entry_SYSCALL_64_after_swapgs
>>> >> +1.82% 0.00%  glusteriotwr38   [unknown]  [k]
>>> >> 0x
>>> >> +1.82% 0.01%  glusteriotwr28   [kernel.kallsyms]  [k]
>>> >> entry_SYSCALL_64_after_swapgs
>>> >> +1.82% 0.00%  glusteriotwr0[kernel.kallsyms]  [k]
>>> >> do_syscall_64
>>> >> +1.81% 0.00%  glusteriotwr28   [kernel.kallsyms]  [k]
>>> >> do_syscall_64
>>> >> +1.81% 0.00%  glusteriotwr15   [kernel.kallsyms]  [k]
>>> >> entry_SYSCALL_64_after_swapgs
>>> >> +1.81% 0.00%  glusteriotwr36   [unknown]  [k]
>>> >> 0x
>>> >> +1.80% 0.00%  glusteriotwr15   [kernel.kallsyms]  [k]
>>> >> do_syscall_64
>>> >> +1.78% 0.01%  glusteriotwr63   [kernel.kallsyms]  [k]
>>> >> entry_SYSCALL_64_after_swapgs
>>> >> +1.77% 0.00%  glusteriotwr63   [kernel.kallsyms]  [k]
>>> >> do_syscall_64
>>> >> +1.75% 0.01%  glusteriotwr38   

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Milind Changire]

On Thu, Aug 23, 2018 at 5:28 PM, Pranith Kumar Karampuri <
pkara...@redhat.com> wrote:

> On Wed, Aug 22, 2018 at 12:01 PM Hu Bert  wrote:
>
>> Just an addition: in general there are no log messages in
>> /var/log/glusterfs/ (if you don't all 'gluster volume ...'), but on
>> the node with the lowest load i see in cli.log.1:
>>
>> [2018-08-22 06:20:43.291055] I [socket.c:2474:socket_event_handler]
>> 0-transport: EPOLLERR - disconnecting now
>> [2018-08-22 06:20:46.291327] I [socket.c:2474:socket_event_handler]
>> 0-transport: EPOLLERR - disconnecting now
>> [2018-08-22 06:20:49.291575] I [socket.c:2474:socket_event_handler]
>> 0-transport: EPOLLERR - disconnecting now
>>
>> every 3 seconds. Looks like this bug:
>> https://bugzilla.redhat.com/show_bug.cgi?id=1484885 - but that shoud
>> have been fixed in the 3.12.x release, and network is fine.
>>
>
> +Milind Changire 
>
That's odd. Presuming cli.log.1 is the logrotated file, it should be
showing older log entries than cli.log. But its not the case here.
Or maybe, there's something running on the command-line on the node with
the lowest load.


>
>> In cli.log there are only these entries:
>>
>> [2018-08-22 06:19:23.428520] I [cli.c:765:main] 0-cli: Started running
>> gluster with version 3.12.12
>> [2018-08-22 06:19:23.800895] I [MSGID: 101190]
>> [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started
>> thread with index 1
>> [2018-08-22 06:19:23.800978] I [socket.c:2474:socket_event_handler]
>> 0-transport: EPOLLERR - disconnecting now
>> [2018-08-22 06:19:23.809366] I [input.c:31:cli_batch] 0-: Exiting with: 0
>>
>> Just wondered if this could related anyhow.
>>
>> 2018-08-21 8:17 GMT+02:00 Pranith Kumar Karampuri :
>> >
>> >
>> > On Tue, Aug 21, 2018 at 11:40 AM Hu Bert 
>> wrote:
>> >>
>> >> Good morning :-)
>> >>
>> >> gluster11:
>> >> ls -l /gluster/bricksdd1/shared/.glusterfs/indices/xattrop/
>> >> total 0
>> >> -- 1 root root 0 Aug 14 06:14
>> >> xattrop-006b65d8-9e81-4886-b380-89168ea079bd
>> >>
>> >> gluster12:
>> >> ls -l /gluster/bricksdd1_new/shared/.glusterfs/indices/xattrop/
>> >> total 0
>> >> -- 1 root root 0 Jul 17 11:24
>> >> xattrop-c7c6f765-ce17-4361-95fb-2fd7f31c7b82
>> >>
>> >> gluster13:
>> >> ls -l /gluster/bricksdd1_new/shared/.glusterfs/indices/xattrop/
>> >> total 0
>> >> -- 1 root root 0 Aug 16 07:54
>> >> xattrop-16b696a0-4214-4999-b277-0917c76c983e
>> >>
>> >>
>> >> And here's the output of 'perf ...' which ran almost a minute - file
>> >> grew pretty fast to a size of 17 GB and system load went up heavily.
>> >> Had to wait a while until load dropped :-)
>> >>
>> >> fyi - load at the moment:
>> >> load gluster11: ~90
>> >> load gluster12: ~10
>> >> load gluster13: ~50
>> >>
>> >> perf record --call-graph=dwarf -p 7897 -o
>> >> /tmp/perf.gluster11.bricksdd1.out
>> >> [ perf record: Woken up 9837 times to write data ]
>> >> Warning:
>> >> Processed 2137218 events and lost 33446 chunks!
>> >>
>> >> Check IO/CPU overload!
>> >>
>> >> [ perf record: Captured and wrote 16576.374 MB
>> >> /tmp/perf.gluster11.bricksdd1.out (2047760 samples) ]
>> >>
>> >> Here's an excerpt.
>> >>
>> >> +1.93% 0.00%  glusteriotwr0[unknown]  [k]
>> >> 0x
>> >> +1.89% 0.00%  glusteriotwr28   [unknown]  [k]
>> >> 0x
>> >> +1.86% 0.00%  glusteriotwr15   [unknown]  [k]
>> >> 0x
>> >> +1.85% 0.00%  glusteriotwr63   [unknown]  [k]
>> >> 0x
>> >> +1.83% 0.01%  glusteriotwr0[kernel.kallsyms]  [k]
>> >> entry_SYSCALL_64_after_swapgs
>> >> +1.82% 0.00%  glusteriotwr38   [unknown]  [k]
>> >> 0x
>> >> +1.82% 0.01%  glusteriotwr28   [kernel.kallsyms]  [k]
>> >> entry_SYSCALL_64_after_swapgs
>> >> +1.82% 0.00%  glusteriotwr0[kernel.kallsyms]  [k]
>> >> do_syscall_64
>> >> +1.81% 0.00%  glusteriotwr28   [kernel.kallsyms]  [k]
>> >> do_syscall_64
>> >> +1.81% 0.00%  glusteriotwr15   [kernel.kallsyms]  [k]
>> >> entry_SYSCALL_64_after_swapgs
>> >> +1.81% 0.00%  glusteriotwr36   [unknown]  [k]
>> >> 0x
>> >> +1.80% 0.00%  glusteriotwr15   [kernel.kallsyms]  [k]
>> >> do_syscall_64
>> >> +1.78% 0.01%  glusteriotwr63   [kernel.kallsyms]  [k]
>> >> entry_SYSCALL_64_after_swapgs
>> >> +1.77% 0.00%  glusteriotwr63   [kernel.kallsyms]  [k]
>> >> do_syscall_64
>> >> +1.75% 0.01%  glusteriotwr38   [kernel.kallsyms]  [k]
>> >> entry_SYSCALL_64_after_swapgs
>> >> +1.75% 0.00%  glusteriotwr38   [kernel.kallsyms]  [k]
>> >> do_syscall_64
>> >> +1.74% 0.00%  glusteriotwr17   [unknown]  [k]
>> >> 0x
>> >> +1.74% 0.00%  glusteriotwr44   [unknown]  [k]
>> >> 0x
>> >> +1.73% 0.00%  glusteriotwr6[unknown]  [k]
>> >> 

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Pranith Kumar Karampuri]

On Wed, Aug 22, 2018 at 12:01 PM Hu Bert  wrote:

> Just an addition: in general there are no log messages in
> /var/log/glusterfs/ (if you don't all 'gluster volume ...'), but on
> the node with the lowest load i see in cli.log.1:
>
> [2018-08-22 06:20:43.291055] I [socket.c:2474:socket_event_handler]
> 0-transport: EPOLLERR - disconnecting now
> [2018-08-22 06:20:46.291327] I [socket.c:2474:socket_event_handler]
> 0-transport: EPOLLERR - disconnecting now
> [2018-08-22 06:20:49.291575] I [socket.c:2474:socket_event_handler]
> 0-transport: EPOLLERR - disconnecting now
>
> every 3 seconds. Looks like this bug:
> https://bugzilla.redhat.com/show_bug.cgi?id=1484885 - but that shoud
> have been fixed in the 3.12.x release, and network is fine.
>

+Milind Changire 


> In cli.log there are only these entries:
>
> [2018-08-22 06:19:23.428520] I [cli.c:765:main] 0-cli: Started running
> gluster with version 3.12.12
> [2018-08-22 06:19:23.800895] I [MSGID: 101190]
> [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started
> thread with index 1
> [2018-08-22 06:19:23.800978] I [socket.c:2474:socket_event_handler]
> 0-transport: EPOLLERR - disconnecting now
> [2018-08-22 06:19:23.809366] I [input.c:31:cli_batch] 0-: Exiting with: 0
>
> Just wondered if this could related anyhow.
>
> 2018-08-21 8:17 GMT+02:00 Pranith Kumar Karampuri :
> >
> >
> > On Tue, Aug 21, 2018 at 11:40 AM Hu Bert  wrote:
> >>
> >> Good morning :-)
> >>
> >> gluster11:
> >> ls -l /gluster/bricksdd1/shared/.glusterfs/indices/xattrop/
> >> total 0
> >> -- 1 root root 0 Aug 14 06:14
> >> xattrop-006b65d8-9e81-4886-b380-89168ea079bd
> >>
> >> gluster12:
> >> ls -l /gluster/bricksdd1_new/shared/.glusterfs/indices/xattrop/
> >> total 0
> >> -- 1 root root 0 Jul 17 11:24
> >> xattrop-c7c6f765-ce17-4361-95fb-2fd7f31c7b82
> >>
> >> gluster13:
> >> ls -l /gluster/bricksdd1_new/shared/.glusterfs/indices/xattrop/
> >> total 0
> >> -- 1 root root 0 Aug 16 07:54
> >> xattrop-16b696a0-4214-4999-b277-0917c76c983e
> >>
> >>
> >> And here's the output of 'perf ...' which ran almost a minute - file
> >> grew pretty fast to a size of 17 GB and system load went up heavily.
> >> Had to wait a while until load dropped :-)
> >>
> >> fyi - load at the moment:
> >> load gluster11: ~90
> >> load gluster12: ~10
> >> load gluster13: ~50
> >>
> >> perf record --call-graph=dwarf -p 7897 -o
> >> /tmp/perf.gluster11.bricksdd1.out
> >> [ perf record: Woken up 9837 times to write data ]
> >> Warning:
> >> Processed 2137218 events and lost 33446 chunks!
> >>
> >> Check IO/CPU overload!
> >>
> >> [ perf record: Captured and wrote 16576.374 MB
> >> /tmp/perf.gluster11.bricksdd1.out (2047760 samples) ]
> >>
> >> Here's an excerpt.
> >>
> >> +1.93% 0.00%  glusteriotwr0[unknown]  [k]
> >> 0x
> >> +1.89% 0.00%  glusteriotwr28   [unknown]  [k]
> >> 0x
> >> +1.86% 0.00%  glusteriotwr15   [unknown]  [k]
> >> 0x
> >> +1.85% 0.00%  glusteriotwr63   [unknown]  [k]
> >> 0x
> >> +1.83% 0.01%  glusteriotwr0[kernel.kallsyms]  [k]
> >> entry_SYSCALL_64_after_swapgs
> >> +1.82% 0.00%  glusteriotwr38   [unknown]  [k]
> >> 0x
> >> +1.82% 0.01%  glusteriotwr28   [kernel.kallsyms]  [k]
> >> entry_SYSCALL_64_after_swapgs
> >> +1.82% 0.00%  glusteriotwr0[kernel.kallsyms]  [k]
> >> do_syscall_64
> >> +1.81% 0.00%  glusteriotwr28   [kernel.kallsyms]  [k]
> >> do_syscall_64
> >> +1.81% 0.00%  glusteriotwr15   [kernel.kallsyms]  [k]
> >> entry_SYSCALL_64_after_swapgs
> >> +1.81% 0.00%  glusteriotwr36   [unknown]  [k]
> >> 0x
> >> +1.80% 0.00%  glusteriotwr15   [kernel.kallsyms]  [k]
> >> do_syscall_64
> >> +1.78% 0.01%  glusteriotwr63   [kernel.kallsyms]  [k]
> >> entry_SYSCALL_64_after_swapgs
> >> +1.77% 0.00%  glusteriotwr63   [kernel.kallsyms]  [k]
> >> do_syscall_64
> >> +1.75% 0.01%  glusteriotwr38   [kernel.kallsyms]  [k]
> >> entry_SYSCALL_64_after_swapgs
> >> +1.75% 0.00%  glusteriotwr38   [kernel.kallsyms]  [k]
> >> do_syscall_64
> >> +1.74% 0.00%  glusteriotwr17   [unknown]  [k]
> >> 0x
> >> +1.74% 0.00%  glusteriotwr44   [unknown]  [k]
> >> 0x
> >> +1.73% 0.00%  glusteriotwr6[unknown]  [k]
> >> 0x
> >> +1.73% 0.00%  glusteriotwr37   [unknown]  [k]
> >> 0x
> >> +1.73% 0.01%  glusteriotwr36   [kernel.kallsyms]  [k]
> >> entry_SYSCALL_64_after_swapgs
> >> +1.72% 0.00%  glusteriotwr34   [unknown]  [k]
> >> 0x
> >> +1.72% 0.00%  glusteriotwr36   [kernel.kallsyms]  [k]
> >> do_syscall_64
> >> +1.71% 0.00%  

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Hu Bert]

Just an addition: in general there are no log messages in
/var/log/glusterfs/ (if you don't all 'gluster volume ...'), but on
the node with the lowest load i see in cli.log.1:

[2018-08-22 06:20:43.291055] I [socket.c:2474:socket_event_handler]
0-transport: EPOLLERR - disconnecting now
[2018-08-22 06:20:46.291327] I [socket.c:2474:socket_event_handler]
0-transport: EPOLLERR - disconnecting now
[2018-08-22 06:20:49.291575] I [socket.c:2474:socket_event_handler]
0-transport: EPOLLERR - disconnecting now

every 3 seconds. Looks like this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1484885 - but that shoud
have been fixed in the 3.12.x release, and network is fine.

In cli.log there are only these entries:

[2018-08-22 06:19:23.428520] I [cli.c:765:main] 0-cli: Started running
gluster with version 3.12.12
[2018-08-22 06:19:23.800895] I [MSGID: 101190]
[event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started
thread with index 1
[2018-08-22 06:19:23.800978] I [socket.c:2474:socket_event_handler]
0-transport: EPOLLERR - disconnecting now
[2018-08-22 06:19:23.809366] I [input.c:31:cli_batch] 0-: Exiting with: 0

Just wondered if this could related anyhow.

2018-08-21 8:17 GMT+02:00 Pranith Kumar Karampuri :
>
>
> On Tue, Aug 21, 2018 at 11:40 AM Hu Bert  wrote:
>>
>> Good morning :-)
>>
>> gluster11:
>> ls -l /gluster/bricksdd1/shared/.glusterfs/indices/xattrop/
>> total 0
>> -- 1 root root 0 Aug 14 06:14
>> xattrop-006b65d8-9e81-4886-b380-89168ea079bd
>>
>> gluster12:
>> ls -l /gluster/bricksdd1_new/shared/.glusterfs/indices/xattrop/
>> total 0
>> -- 1 root root 0 Jul 17 11:24
>> xattrop-c7c6f765-ce17-4361-95fb-2fd7f31c7b82
>>
>> gluster13:
>> ls -l /gluster/bricksdd1_new/shared/.glusterfs/indices/xattrop/
>> total 0
>> -- 1 root root 0 Aug 16 07:54
>> xattrop-16b696a0-4214-4999-b277-0917c76c983e
>>
>>
>> And here's the output of 'perf ...' which ran almost a minute - file
>> grew pretty fast to a size of 17 GB and system load went up heavily.
>> Had to wait a while until load dropped :-)
>>
>> fyi - load at the moment:
>> load gluster11: ~90
>> load gluster12: ~10
>> load gluster13: ~50
>>
>> perf record --call-graph=dwarf -p 7897 -o
>> /tmp/perf.gluster11.bricksdd1.out
>> [ perf record: Woken up 9837 times to write data ]
>> Warning:
>> Processed 2137218 events and lost 33446 chunks!
>>
>> Check IO/CPU overload!
>>
>> [ perf record: Captured and wrote 16576.374 MB
>> /tmp/perf.gluster11.bricksdd1.out (2047760 samples) ]
>>
>> Here's an excerpt.
>>
>> +1.93% 0.00%  glusteriotwr0[unknown]  [k]
>> 0x
>> +1.89% 0.00%  glusteriotwr28   [unknown]  [k]
>> 0x
>> +1.86% 0.00%  glusteriotwr15   [unknown]  [k]
>> 0x
>> +1.85% 0.00%  glusteriotwr63   [unknown]  [k]
>> 0x
>> +1.83% 0.01%  glusteriotwr0[kernel.kallsyms]  [k]
>> entry_SYSCALL_64_after_swapgs
>> +1.82% 0.00%  glusteriotwr38   [unknown]  [k]
>> 0x
>> +1.82% 0.01%  glusteriotwr28   [kernel.kallsyms]  [k]
>> entry_SYSCALL_64_after_swapgs
>> +1.82% 0.00%  glusteriotwr0[kernel.kallsyms]  [k]
>> do_syscall_64
>> +1.81% 0.00%  glusteriotwr28   [kernel.kallsyms]  [k]
>> do_syscall_64
>> +1.81% 0.00%  glusteriotwr15   [kernel.kallsyms]  [k]
>> entry_SYSCALL_64_after_swapgs
>> +1.81% 0.00%  glusteriotwr36   [unknown]  [k]
>> 0x
>> +1.80% 0.00%  glusteriotwr15   [kernel.kallsyms]  [k]
>> do_syscall_64
>> +1.78% 0.01%  glusteriotwr63   [kernel.kallsyms]  [k]
>> entry_SYSCALL_64_after_swapgs
>> +1.77% 0.00%  glusteriotwr63   [kernel.kallsyms]  [k]
>> do_syscall_64
>> +1.75% 0.01%  glusteriotwr38   [kernel.kallsyms]  [k]
>> entry_SYSCALL_64_after_swapgs
>> +1.75% 0.00%  glusteriotwr38   [kernel.kallsyms]  [k]
>> do_syscall_64
>> +1.74% 0.00%  glusteriotwr17   [unknown]  [k]
>> 0x
>> +1.74% 0.00%  glusteriotwr44   [unknown]  [k]
>> 0x
>> +1.73% 0.00%  glusteriotwr6[unknown]  [k]
>> 0x
>> +1.73% 0.00%  glusteriotwr37   [unknown]  [k]
>> 0x
>> +1.73% 0.01%  glusteriotwr36   [kernel.kallsyms]  [k]
>> entry_SYSCALL_64_after_swapgs
>> +1.72% 0.00%  glusteriotwr34   [unknown]  [k]
>> 0x
>> +1.72% 0.00%  glusteriotwr36   [kernel.kallsyms]  [k]
>> do_syscall_64
>> +1.71% 0.00%  glusteriotwr45   [unknown]  [k]
>> 0x
>> +1.70% 0.00%  glusteriotwr7[unknown]  [k]
>> 0x
>> +1.68% 0.00%  glusteriotwr15   [kernel.kallsyms]  [k]
>> sys_getdents
>> +1.68% 0.00%  glusteriotwr15   [kernel.kallsyms]  [k] filldir

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Pranith Kumar Karampuri]

On Tue, Aug 21, 2018 at 11:40 AM Hu Bert  wrote:

> Good morning :-)
>
> gluster11:
> ls -l /gluster/bricksdd1/shared/.glusterfs/indices/xattrop/
> total 0
> -- 1 root root 0 Aug 14 06:14
> xattrop-006b65d8-9e81-4886-b380-89168ea079bd
>
> gluster12:
> ls -l /gluster/bricksdd1_new/shared/.glusterfs/indices/xattrop/
> total 0
> -- 1 root root 0 Jul 17 11:24
> xattrop-c7c6f765-ce17-4361-95fb-2fd7f31c7b82
>
> gluster13:
> ls -l /gluster/bricksdd1_new/shared/.glusterfs/indices/xattrop/
> total 0
> -- 1 root root 0 Aug 16 07:54
> xattrop-16b696a0-4214-4999-b277-0917c76c983e
>
>
> And here's the output of 'perf ...' which ran almost a minute - file
> grew pretty fast to a size of 17 GB and system load went up heavily.
> Had to wait a while until load dropped :-)
>
> fyi - load at the moment:
> load gluster11: ~90
> load gluster12: ~10
> load gluster13: ~50
>
> perf record --call-graph=dwarf -p 7897 -o /tmp/perf.gluster11.bricksdd1.out
> [ perf record: Woken up 9837 times to write data ]
> Warning:
> Processed 2137218 events and lost 33446 chunks!
>
> Check IO/CPU overload!
>
> [ perf record: Captured and wrote 16576.374 MB
> /tmp/perf.gluster11.bricksdd1.out (2047760 samples) ]
>
> Here's an excerpt.
>
> +1.93% 0.00%  glusteriotwr0[unknown]  [k]
> 0x
> +1.89% 0.00%  glusteriotwr28   [unknown]  [k]
> 0x
> +1.86% 0.00%  glusteriotwr15   [unknown]  [k]
> 0x
> +1.85% 0.00%  glusteriotwr63   [unknown]  [k]
> 0x
> +1.83% 0.01%  glusteriotwr0[kernel.kallsyms]  [k]
> entry_SYSCALL_64_after_swapgs
> +1.82% 0.00%  glusteriotwr38   [unknown]  [k]
> 0x
> +1.82% 0.01%  glusteriotwr28   [kernel.kallsyms]  [k]
> entry_SYSCALL_64_after_swapgs
> +1.82% 0.00%  glusteriotwr0[kernel.kallsyms]  [k]
> do_syscall_64
> +1.81% 0.00%  glusteriotwr28   [kernel.kallsyms]  [k]
> do_syscall_64
> +1.81% 0.00%  glusteriotwr15   [kernel.kallsyms]  [k]
> entry_SYSCALL_64_after_swapgs
> +1.81% 0.00%  glusteriotwr36   [unknown]  [k]
> 0x
> +1.80% 0.00%  glusteriotwr15   [kernel.kallsyms]  [k]
> do_syscall_64
> +1.78% 0.01%  glusteriotwr63   [kernel.kallsyms]  [k]
> entry_SYSCALL_64_after_swapgs
> +1.77% 0.00%  glusteriotwr63   [kernel.kallsyms]  [k]
> do_syscall_64
> +1.75% 0.01%  glusteriotwr38   [kernel.kallsyms]  [k]
> entry_SYSCALL_64_after_swapgs
> +1.75% 0.00%  glusteriotwr38   [kernel.kallsyms]  [k]
> do_syscall_64
> +1.74% 0.00%  glusteriotwr17   [unknown]  [k]
> 0x
> +1.74% 0.00%  glusteriotwr44   [unknown]  [k]
> 0x
> +1.73% 0.00%  glusteriotwr6[unknown]  [k]
> 0x
> +1.73% 0.00%  glusteriotwr37   [unknown]  [k]
> 0x
> +1.73% 0.01%  glusteriotwr36   [kernel.kallsyms]  [k]
> entry_SYSCALL_64_after_swapgs
> +1.72% 0.00%  glusteriotwr34   [unknown]  [k]
> 0x
> +1.72% 0.00%  glusteriotwr36   [kernel.kallsyms]  [k]
> do_syscall_64
> +1.71% 0.00%  glusteriotwr45   [unknown]  [k]
> 0x
> +1.70% 0.00%  glusteriotwr7[unknown]  [k]
> 0x
> +1.68% 0.00%  glusteriotwr15   [kernel.kallsyms]  [k]
> sys_getdents
> +1.68% 0.00%  glusteriotwr15   [kernel.kallsyms]  [k] filldir
> +1.68% 0.00%  glusteriotwr15   libc-2.24.so   [.]
> 0x80c60db8ef2b
> +1.68% 0.00%  glusteriotwr15   libc-2.24.so   [.]
> readdir64
> +1.68% 0.00%  glusteriotwr15   index.so   [.]
> 0x80c6192a1888
> +1.68% 0.00%  glusteriotwr15   [kernel.kallsyms]  [k]
> iterate_dir
> +1.68% 0.00%  glusteriotwr15   [kernel.kallsyms]  [k]
> ext4_htree_fill_tree
> +1.68% 0.00%  glusteriotwr15   [kernel.kallsyms]  [k]
> ext4_readdir
>
> Or do you want to download the file /tmp/perf.gluster11.bricksdd1.out
> and examine it yourself? If so i could send you a link.
>

Thank you! yes a link would be great. I am not as good with kernel side of
things. So I will have to show this information to someone else who knows
these things so expect delay in response.


>
>
> 2018-08-21 7:13 GMT+02:00 Pranith Kumar Karampuri :
> >
> >
> > On Tue, Aug 21, 2018 at 10:13 AM Pranith Kumar Karampuri
> >  wrote:
> >>
> >>
> >>
> >> On Mon, Aug 20, 2018 at 3:20 PM Hu Bert  wrote:
> >>>
> >>> Regarding hardware the machines are identical. Intel Xeon E5-1650 v3
> >>> Hexa-Core; 64 GB DDR4 ECC; Dell PERC H330 8 Port SAS/SATA 12 GBit/s
> >>> RAID Controller; operating system running on a raid1, then 4 disks
> >>> (JBOD) as bricks.
> >>>
> >>> Ok, i ran perf for a few 

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Hu Bert]

Good morning :-)

gluster11:
ls -l /gluster/bricksdd1/shared/.glusterfs/indices/xattrop/
total 0
-- 1 root root 0 Aug 14 06:14
xattrop-006b65d8-9e81-4886-b380-89168ea079bd

gluster12:
ls -l /gluster/bricksdd1_new/shared/.glusterfs/indices/xattrop/
total 0
-- 1 root root 0 Jul 17 11:24
xattrop-c7c6f765-ce17-4361-95fb-2fd7f31c7b82

gluster13:
ls -l /gluster/bricksdd1_new/shared/.glusterfs/indices/xattrop/
total 0
-- 1 root root 0 Aug 16 07:54
xattrop-16b696a0-4214-4999-b277-0917c76c983e


And here's the output of 'perf ...' which ran almost a minute - file
grew pretty fast to a size of 17 GB and system load went up heavily.
Had to wait a while until load dropped :-)

fyi - load at the moment:
load gluster11: ~90
load gluster12: ~10
load gluster13: ~50

perf record --call-graph=dwarf -p 7897 -o /tmp/perf.gluster11.bricksdd1.out
[ perf record: Woken up 9837 times to write data ]
Warning:
Processed 2137218 events and lost 33446 chunks!

Check IO/CPU overload!

[ perf record: Captured and wrote 16576.374 MB
/tmp/perf.gluster11.bricksdd1.out (2047760 samples) ]

Here's an excerpt.

+1.93% 0.00%  glusteriotwr0[unknown]  [k]
0x
+1.89% 0.00%  glusteriotwr28   [unknown]  [k]
0x
+1.86% 0.00%  glusteriotwr15   [unknown]  [k]
0x
+1.85% 0.00%  glusteriotwr63   [unknown]  [k]
0x
+1.83% 0.01%  glusteriotwr0[kernel.kallsyms]  [k]
entry_SYSCALL_64_after_swapgs
+1.82% 0.00%  glusteriotwr38   [unknown]  [k]
0x
+1.82% 0.01%  glusteriotwr28   [kernel.kallsyms]  [k]
entry_SYSCALL_64_after_swapgs
+1.82% 0.00%  glusteriotwr0[kernel.kallsyms]  [k] do_syscall_64
+1.81% 0.00%  glusteriotwr28   [kernel.kallsyms]  [k] do_syscall_64
+1.81% 0.00%  glusteriotwr15   [kernel.kallsyms]  [k]
entry_SYSCALL_64_after_swapgs
+1.81% 0.00%  glusteriotwr36   [unknown]  [k]
0x
+1.80% 0.00%  glusteriotwr15   [kernel.kallsyms]  [k] do_syscall_64
+1.78% 0.01%  glusteriotwr63   [kernel.kallsyms]  [k]
entry_SYSCALL_64_after_swapgs
+1.77% 0.00%  glusteriotwr63   [kernel.kallsyms]  [k] do_syscall_64
+1.75% 0.01%  glusteriotwr38   [kernel.kallsyms]  [k]
entry_SYSCALL_64_after_swapgs
+1.75% 0.00%  glusteriotwr38   [kernel.kallsyms]  [k] do_syscall_64
+1.74% 0.00%  glusteriotwr17   [unknown]  [k]
0x
+1.74% 0.00%  glusteriotwr44   [unknown]  [k]
0x
+1.73% 0.00%  glusteriotwr6[unknown]  [k]
0x
+1.73% 0.00%  glusteriotwr37   [unknown]  [k]
0x
+1.73% 0.01%  glusteriotwr36   [kernel.kallsyms]  [k]
entry_SYSCALL_64_after_swapgs
+1.72% 0.00%  glusteriotwr34   [unknown]  [k]
0x
+1.72% 0.00%  glusteriotwr36   [kernel.kallsyms]  [k] do_syscall_64
+1.71% 0.00%  glusteriotwr45   [unknown]  [k]
0x
+1.70% 0.00%  glusteriotwr7[unknown]  [k]
0x
+1.68% 0.00%  glusteriotwr15   [kernel.kallsyms]  [k] sys_getdents
+1.68% 0.00%  glusteriotwr15   [kernel.kallsyms]  [k] filldir
+1.68% 0.00%  glusteriotwr15   libc-2.24.so   [.]
0x80c60db8ef2b
+1.68% 0.00%  glusteriotwr15   libc-2.24.so   [.] readdir64
+1.68% 0.00%  glusteriotwr15   index.so   [.]
0x80c6192a1888
+1.68% 0.00%  glusteriotwr15   [kernel.kallsyms]  [k] iterate_dir
+1.68% 0.00%  glusteriotwr15   [kernel.kallsyms]  [k]
ext4_htree_fill_tree
+1.68% 0.00%  glusteriotwr15   [kernel.kallsyms]  [k] ext4_readdir

Or do you want to download the file /tmp/perf.gluster11.bricksdd1.out
and examine it yourself? If so i could send you a link.


2018-08-21 7:13 GMT+02:00 Pranith Kumar Karampuri :
>
>
> On Tue, Aug 21, 2018 at 10:13 AM Pranith Kumar Karampuri
>  wrote:
>>
>>
>>
>> On Mon, Aug 20, 2018 at 3:20 PM Hu Bert  wrote:
>>>
>>> Regarding hardware the machines are identical. Intel Xeon E5-1650 v3
>>> Hexa-Core; 64 GB DDR4 ECC; Dell PERC H330 8 Port SAS/SATA 12 GBit/s
>>> RAID Controller; operating system running on a raid1, then 4 disks
>>> (JBOD) as bricks.
>>>
>>> Ok, i ran perf for a few seconds.
>>>
>>> 
>>> perf record --call-graph=dwarf -p 7897 -o
>>> /tmp/perf.gluster11.bricksdd1.out
>>> ^C[ perf record: Woken up 378 times to write data ]
>>> Warning:
>>> Processed 83690 events and lost 96 chunks!
>>>
>>> Check IO/CPU overload!
>>>
>>> [ perf record: Captured and wrote 423.087 MB
>>> /tmp/perf.gluster11.bricksdd1.out (51744 samples) ]
>>> 
>>>
>>> I copied a couple of lines:
>>>
>>> +8.10% 0.00%  

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Pranith Kumar Karampuri]

On Tue, Aug 21, 2018 at 10:13 AM Pranith Kumar Karampuri <
pkara...@redhat.com> wrote:

>
>
> On Mon, Aug 20, 2018 at 3:20 PM Hu Bert  wrote:
>
>> Regarding hardware the machines are identical. Intel Xeon E5-1650 v3
>> Hexa-Core; 64 GB DDR4 ECC; Dell PERC H330 8 Port SAS/SATA 12 GBit/s
>> RAID Controller; operating system running on a raid1, then 4 disks
>> (JBOD) as bricks.
>>
>> Ok, i ran perf for a few seconds.
>>
>> 
>> perf record --call-graph=dwarf -p 7897 -o
>> /tmp/perf.gluster11.bricksdd1.out
>> ^C[ perf record: Woken up 378 times to write data ]
>> Warning:
>> Processed 83690 events and lost 96 chunks!
>>
>> Check IO/CPU overload!
>>
>> [ perf record: Captured and wrote 423.087 MB
>> /tmp/perf.gluster11.bricksdd1.out (51744 samples) ]
>> 
>>
>> I copied a couple of lines:
>>
>> +8.10% 0.00%  glusteriotwr22   [unknown]  [k]
>> 0x
>> +8.10% 0.00%  glusteriotwr22   [kernel.kallsyms]  [k]
>> iterate_dir
>> +8.10% 0.00%  glusteriotwr22   [kernel.kallsyms]  [k]
>> sys_getdents
>> +8.10% 0.00%  glusteriotwr22   [kernel.kallsyms]  [k] filldir
>> +8.10% 0.00%  glusteriotwr22   [kernel.kallsyms]  [k]
>> do_syscall_64
>> +8.10% 0.00%  glusteriotwr22   [kernel.kallsyms]  [k]
>> entry_SYSCALL_64_after_swapgs
>> +8.10% 0.00%  glusteriotwr22   libc-2.24.so   [.]
>> 0x80c60db8ef2b
>> +8.10% 0.00%  glusteriotwr22   libc-2.24.so   [.]
>> readdir64
>> +8.10% 0.00%  glusteriotwr22   index.so   [.]
>> 0x80c6192a1888
>> +8.10% 0.04%  glusteriotwr22   [kernel.kallsyms]  [k]
>> ext4_htree_fill_tree
>> +8.10% 0.00%  glusteriotwr22   [kernel.kallsyms]  [k]
>> ext4_readdir
>> +7.95% 0.12%  glusteriotwr22   [kernel.kallsyms]  [k]
>> htree_dirblock_to_tree
>> +5.78% 0.96%  glusteriotwr22   [kernel.kallsyms]  [k]
>> __ext4_read_dirblock
>> +4.80% 0.02%  glusteriotwr22   [kernel.kallsyms]  [k]
>> ext4_bread
>> +4.78% 0.04%  glusteriotwr22   [kernel.kallsyms]  [k]
>> ext4_getblk
>> +4.72% 0.02%  glusteriotwr22   [kernel.kallsyms]  [k]
>> __getblk_gfp
>> +4.57% 0.00%  glusteriotwr3[unknown]  [k]
>> 0x
>> +4.55% 0.00%  glusteriotwr3[kernel.kallsyms]  [k]
>> do_syscall_64
>>
>> Do you need different or additional information?
>>
>
> This looks like there are lot of readdirs going on which is different from
> what we observed earlier, how many seconds did you do perf record for? Will
> it be possible for you to do this for some more time? may be a minute? Just
> want to be sure that the data actually represents what we are observing.
>

I found one code path which on lookup does readdirs. Could you give me the
output of ls -l /.glusterfs/indices/xattrop on all the three
bricks? It can probably give a correlation to see if it is indeed the same
issue or not.


>
>
>>
>>
>> 2018-08-20 11:20 GMT+02:00 Pranith Kumar Karampuri :
>> > Even the brick which doesn't have high CPU seems to have same number of
>> > lookups, so that's not it.
>> > Is there any difference at all between the machines which have high CPU
>> vs
>> > low CPU?
>> > I think the only other thing I would do is to install perf tools and
>> try to
>> > figure out the call-graph which is leading to so much CPU
>> >
>> > This affects performance of the brick I think, so you may have to do it
>> > quickly and for less time.
>> >
>> > perf record --call-graph=dwarf -p-o 
>> > then
>> > perf report -i 
>> >
>> >
>> > On Mon, Aug 20, 2018 at 2:40 PM Hu Bert  wrote:
>> >>
>> >> gluster volume heal shared info | grep -i number
>> >> Number of entries: 0
>> >> Number of entries: 0
>> >> Number of entries: 0
>> >> Number of entries: 0
>> >> Number of entries: 0
>> >> Number of entries: 0
>> >> Number of entries: 0
>> >> Number of entries: 0
>> >> Number of entries: 0
>> >> Number of entries: 0
>> >> Number of entries: 0
>> >> Number of entries: 0
>> >>
>> >> Looks good to me.
>> >>
>> >>
>> >> 2018-08-20 10:51 GMT+02:00 Pranith Kumar Karampuri <
>> pkara...@redhat.com>:
>> >> > There are a lot of Lookup operations in the system. But I am not
>> able to
>> >> > find why. Could you check the output of
>> >> >
>> >> > # gluster volume heal  info | grep -i number
>> >> >
>> >> > it should print all zeros.
>> >> >
>> >> > On Fri, Aug 17, 2018 at 1:49 PM Hu Bert 
>> wrote:
>> >> >>
>> >> >> I don't know what you exactly mean with workload, but the main
>> >> >> function of the volume is storing (incl. writing, reading) images
>> >> >> (from hundreds of bytes up to 30 MBs, overall ~7TB). The work is
>> done
>> >> >> by apache tomcat servers writing to / reading from the volume.
>> Besides
>> >> >> images there are some text files and binaries that are stored on the
>> >> >> volume and get updated regularly (every x hours); we'll try to
>> 

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Pranith Kumar Karampuri]

On Mon, Aug 20, 2018 at 3:20 PM Hu Bert  wrote:

> Regarding hardware the machines are identical. Intel Xeon E5-1650 v3
> Hexa-Core; 64 GB DDR4 ECC; Dell PERC H330 8 Port SAS/SATA 12 GBit/s
> RAID Controller; operating system running on a raid1, then 4 disks
> (JBOD) as bricks.
>
> Ok, i ran perf for a few seconds.
>
> 
> perf record --call-graph=dwarf -p 7897 -o
> /tmp/perf.gluster11.bricksdd1.out
> ^C[ perf record: Woken up 378 times to write data ]
> Warning:
> Processed 83690 events and lost 96 chunks!
>
> Check IO/CPU overload!
>
> [ perf record: Captured and wrote 423.087 MB
> /tmp/perf.gluster11.bricksdd1.out (51744 samples) ]
> 
>
> I copied a couple of lines:
>
> +8.10% 0.00%  glusteriotwr22   [unknown]  [k]
> 0x
> +8.10% 0.00%  glusteriotwr22   [kernel.kallsyms]  [k]
> iterate_dir
> +8.10% 0.00%  glusteriotwr22   [kernel.kallsyms]  [k]
> sys_getdents
> +8.10% 0.00%  glusteriotwr22   [kernel.kallsyms]  [k] filldir
> +8.10% 0.00%  glusteriotwr22   [kernel.kallsyms]  [k]
> do_syscall_64
> +8.10% 0.00%  glusteriotwr22   [kernel.kallsyms]  [k]
> entry_SYSCALL_64_after_swapgs
> +8.10% 0.00%  glusteriotwr22   libc-2.24.so   [.]
> 0x80c60db8ef2b
> +8.10% 0.00%  glusteriotwr22   libc-2.24.so   [.]
> readdir64
> +8.10% 0.00%  glusteriotwr22   index.so   [.]
> 0x80c6192a1888
> +8.10% 0.04%  glusteriotwr22   [kernel.kallsyms]  [k]
> ext4_htree_fill_tree
> +8.10% 0.00%  glusteriotwr22   [kernel.kallsyms]  [k]
> ext4_readdir
> +7.95% 0.12%  glusteriotwr22   [kernel.kallsyms]  [k]
> htree_dirblock_to_tree
> +5.78% 0.96%  glusteriotwr22   [kernel.kallsyms]  [k]
> __ext4_read_dirblock
> +4.80% 0.02%  glusteriotwr22   [kernel.kallsyms]  [k]
> ext4_bread
> +4.78% 0.04%  glusteriotwr22   [kernel.kallsyms]  [k]
> ext4_getblk
> +4.72% 0.02%  glusteriotwr22   [kernel.kallsyms]  [k]
> __getblk_gfp
> +4.57% 0.00%  glusteriotwr3[unknown]  [k]
> 0x
> +4.55% 0.00%  glusteriotwr3[kernel.kallsyms]  [k]
> do_syscall_64
>
> Do you need different or additional information?
>

This looks like there are lot of readdirs going on which is different from
what we observed earlier, how many seconds did you do perf record for? Will
it be possible for you to do this for some more time? may be a minute? Just
want to be sure that the data actually represents what we are observing.


>
>
> 2018-08-20 11:20 GMT+02:00 Pranith Kumar Karampuri :
> > Even the brick which doesn't have high CPU seems to have same number of
> > lookups, so that's not it.
> > Is there any difference at all between the machines which have high CPU
> vs
> > low CPU?
> > I think the only other thing I would do is to install perf tools and try
> to
> > figure out the call-graph which is leading to so much CPU
> >
> > This affects performance of the brick I think, so you may have to do it
> > quickly and for less time.
> >
> > perf record --call-graph=dwarf -p-o 
> > then
> > perf report -i 
> >
> >
> > On Mon, Aug 20, 2018 at 2:40 PM Hu Bert  wrote:
> >>
> >> gluster volume heal shared info | grep -i number
> >> Number of entries: 0
> >> Number of entries: 0
> >> Number of entries: 0
> >> Number of entries: 0
> >> Number of entries: 0
> >> Number of entries: 0
> >> Number of entries: 0
> >> Number of entries: 0
> >> Number of entries: 0
> >> Number of entries: 0
> >> Number of entries: 0
> >> Number of entries: 0
> >>
> >> Looks good to me.
> >>
> >>
> >> 2018-08-20 10:51 GMT+02:00 Pranith Kumar Karampuri  >:
> >> > There are a lot of Lookup operations in the system. But I am not able
> to
> >> > find why. Could you check the output of
> >> >
> >> > # gluster volume heal  info | grep -i number
> >> >
> >> > it should print all zeros.
> >> >
> >> > On Fri, Aug 17, 2018 at 1:49 PM Hu Bert 
> wrote:
> >> >>
> >> >> I don't know what you exactly mean with workload, but the main
> >> >> function of the volume is storing (incl. writing, reading) images
> >> >> (from hundreds of bytes up to 30 MBs, overall ~7TB). The work is done
> >> >> by apache tomcat servers writing to / reading from the volume.
> Besides
> >> >> images there are some text files and binaries that are stored on the
> >> >> volume and get updated regularly (every x hours); we'll try to
> migrate
> >> >> the latter ones to local storage asap.
> >> >>
> >> >> Interestingly it's only one process (and its threads) of the same
> >> >> brick on 2 of the gluster servers that consumes the CPU.
> >> >>
> >> >> gluster11: bricksdd1; not healed; full CPU
> >> >> gluster12: bricksdd1; got healed; normal CPU
> >> >> gluster13: bricksdd1; got healed; full CPU
> >> >>
> >> >> Besides: performance during heal (e.g. gluster12, bricksdd1) was way
> >> >> better than it is now. I've 

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Hu Bert]

Regarding hardware the machines are identical. Intel Xeon E5-1650 v3
Hexa-Core; 64 GB DDR4 ECC; Dell PERC H330 8 Port SAS/SATA 12 GBit/s
RAID Controller; operating system running on a raid1, then 4 disks
(JBOD) as bricks.

Ok, i ran perf for a few seconds.


perf record --call-graph=dwarf -p 7897 -o
/tmp/perf.gluster11.bricksdd1.out
^C[ perf record: Woken up 378 times to write data ]
Warning:
Processed 83690 events and lost 96 chunks!

Check IO/CPU overload!

[ perf record: Captured and wrote 423.087 MB
/tmp/perf.gluster11.bricksdd1.out (51744 samples) ]


I copied a couple of lines:

+8.10% 0.00%  glusteriotwr22   [unknown]  [k]
0x
+8.10% 0.00%  glusteriotwr22   [kernel.kallsyms]  [k] iterate_dir
+8.10% 0.00%  glusteriotwr22   [kernel.kallsyms]  [k] sys_getdents
+8.10% 0.00%  glusteriotwr22   [kernel.kallsyms]  [k] filldir
+8.10% 0.00%  glusteriotwr22   [kernel.kallsyms]  [k] do_syscall_64
+8.10% 0.00%  glusteriotwr22   [kernel.kallsyms]  [k]
entry_SYSCALL_64_after_swapgs
+8.10% 0.00%  glusteriotwr22   libc-2.24.so   [.]
0x80c60db8ef2b
+8.10% 0.00%  glusteriotwr22   libc-2.24.so   [.] readdir64
+8.10% 0.00%  glusteriotwr22   index.so   [.]
0x80c6192a1888
+8.10% 0.04%  glusteriotwr22   [kernel.kallsyms]  [k]
ext4_htree_fill_tree
+8.10% 0.00%  glusteriotwr22   [kernel.kallsyms]  [k] ext4_readdir
+7.95% 0.12%  glusteriotwr22   [kernel.kallsyms]  [k]
htree_dirblock_to_tree
+5.78% 0.96%  glusteriotwr22   [kernel.kallsyms]  [k]
__ext4_read_dirblock
+4.80% 0.02%  glusteriotwr22   [kernel.kallsyms]  [k] ext4_bread
+4.78% 0.04%  glusteriotwr22   [kernel.kallsyms]  [k] ext4_getblk
+4.72% 0.02%  glusteriotwr22   [kernel.kallsyms]  [k] __getblk_gfp
+4.57% 0.00%  glusteriotwr3[unknown]  [k]
0x
+4.55% 0.00%  glusteriotwr3[kernel.kallsyms]  [k] do_syscall_64

Do you need different or additional information?


2018-08-20 11:20 GMT+02:00 Pranith Kumar Karampuri :
> Even the brick which doesn't have high CPU seems to have same number of
> lookups, so that's not it.
> Is there any difference at all between the machines which have high CPU vs
> low CPU?
> I think the only other thing I would do is to install perf tools and try to
> figure out the call-graph which is leading to so much CPU
>
> This affects performance of the brick I think, so you may have to do it
> quickly and for less time.
>
> perf record --call-graph=dwarf -p-o 
> then
> perf report -i 
>
>
> On Mon, Aug 20, 2018 at 2:40 PM Hu Bert  wrote:
>>
>> gluster volume heal shared info | grep -i number
>> Number of entries: 0
>> Number of entries: 0
>> Number of entries: 0
>> Number of entries: 0
>> Number of entries: 0
>> Number of entries: 0
>> Number of entries: 0
>> Number of entries: 0
>> Number of entries: 0
>> Number of entries: 0
>> Number of entries: 0
>> Number of entries: 0
>>
>> Looks good to me.
>>
>>
>> 2018-08-20 10:51 GMT+02:00 Pranith Kumar Karampuri :
>> > There are a lot of Lookup operations in the system. But I am not able to
>> > find why. Could you check the output of
>> >
>> > # gluster volume heal  info | grep -i number
>> >
>> > it should print all zeros.
>> >
>> > On Fri, Aug 17, 2018 at 1:49 PM Hu Bert  wrote:
>> >>
>> >> I don't know what you exactly mean with workload, but the main
>> >> function of the volume is storing (incl. writing, reading) images
>> >> (from hundreds of bytes up to 30 MBs, overall ~7TB). The work is done
>> >> by apache tomcat servers writing to / reading from the volume. Besides
>> >> images there are some text files and binaries that are stored on the
>> >> volume and get updated regularly (every x hours); we'll try to migrate
>> >> the latter ones to local storage asap.
>> >>
>> >> Interestingly it's only one process (and its threads) of the same
>> >> brick on 2 of the gluster servers that consumes the CPU.
>> >>
>> >> gluster11: bricksdd1; not healed; full CPU
>> >> gluster12: bricksdd1; got healed; normal CPU
>> >> gluster13: bricksdd1; got healed; full CPU
>> >>
>> >> Besides: performance during heal (e.g. gluster12, bricksdd1) was way
>> >> better than it is now. I've attached 2 pngs showing the differing cpu
>> >> usage of last week before/after heal.
>> >>
>> >> 2018-08-17 9:30 GMT+02:00 Pranith Kumar Karampuri
>> >> :
>> >> > There seems to be too many lookup operations compared to any other
>> >> > operations. What is the workload on the volume?
>> >> >
>> >> > On Fri, Aug 17, 2018 at 12:47 PM Hu Bert 
>> >> > wrote:
>> >> >>
>> >> >> i hope i did get it right.
>> >> >>
>> >> >> gluster volume profile shared start
>> >> >> wait 10 minutes
>> >> >> gluster volume profile shared info
>> >> >> gluster volume profile shared stop
>> >> >>
>> >> >> If that's ok, 

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Hu Bert]

gluster volume heal shared info | grep -i number
Number of entries: 0
Number of entries: 0
Number of entries: 0
Number of entries: 0
Number of entries: 0
Number of entries: 0
Number of entries: 0
Number of entries: 0
Number of entries: 0
Number of entries: 0
Number of entries: 0
Number of entries: 0

Looks good to me.


2018-08-20 10:51 GMT+02:00 Pranith Kumar Karampuri :
> There are a lot of Lookup operations in the system. But I am not able to
> find why. Could you check the output of
>
> # gluster volume heal  info | grep -i number
>
> it should print all zeros.
>
> On Fri, Aug 17, 2018 at 1:49 PM Hu Bert  wrote:
>>
>> I don't know what you exactly mean with workload, but the main
>> function of the volume is storing (incl. writing, reading) images
>> (from hundreds of bytes up to 30 MBs, overall ~7TB). The work is done
>> by apache tomcat servers writing to / reading from the volume. Besides
>> images there are some text files and binaries that are stored on the
>> volume and get updated regularly (every x hours); we'll try to migrate
>> the latter ones to local storage asap.
>>
>> Interestingly it's only one process (and its threads) of the same
>> brick on 2 of the gluster servers that consumes the CPU.
>>
>> gluster11: bricksdd1; not healed; full CPU
>> gluster12: bricksdd1; got healed; normal CPU
>> gluster13: bricksdd1; got healed; full CPU
>>
>> Besides: performance during heal (e.g. gluster12, bricksdd1) was way
>> better than it is now. I've attached 2 pngs showing the differing cpu
>> usage of last week before/after heal.
>>
>> 2018-08-17 9:30 GMT+02:00 Pranith Kumar Karampuri :
>> > There seems to be too many lookup operations compared to any other
>> > operations. What is the workload on the volume?
>> >
>> > On Fri, Aug 17, 2018 at 12:47 PM Hu Bert  wrote:
>> >>
>> >> i hope i did get it right.
>> >>
>> >> gluster volume profile shared start
>> >> wait 10 minutes
>> >> gluster volume profile shared info
>> >> gluster volume profile shared stop
>> >>
>> >> If that's ok, i've attached the output of the info command.
>> >>
>> >>
>> >> 2018-08-17 8:31 GMT+02:00 Pranith Kumar Karampuri
>> >> :
>> >> > Please do volume profile also for around 10 minutes when CPU% is
>> >> > high.
>> >> >
>> >> > On Fri, Aug 17, 2018 at 11:56 AM Pranith Kumar Karampuri
>> >> >  wrote:
>> >> >>
>> >> >> As per the output, all io-threads are using a lot of CPU. It is
>> >> >> better
>> >> >> to
>> >> >> check what the volume profile is to see what is leading to so much
>> >> >> work
>> >> >> for
>> >> >> io-threads. Please follow the documentation at
>> >> >>
>> >> >>
>> >> >> https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Monitoring%20Workload/
>> >> >> section: "
>> >> >>
>> >> >> Running GlusterFS Volume Profile Command"
>> >> >>
>> >> >> and attach output of  "gluster volume profile info",
>> >> >>
>> >> >> On Fri, Aug 17, 2018 at 11:24 AM Hu Bert 
>> >> >> wrote:
>> >> >>>
>> >> >>> Good morning,
>> >> >>>
>> >> >>> i ran the command during 100% CPU usage and attached the file.
>> >> >>> Hopefully it helps.
>> >> >>>
>> >> >>> 2018-08-17 7:33 GMT+02:00 Pranith Kumar Karampuri
>> >> >>> :
>> >> >>> > Could you do the following on one of the nodes where you are
>> >> >>> > observing
>> >> >>> > high
>> >> >>> > CPU usage and attach that file to this thread? We can find what
>> >> >>> > threads/processes are leading to high usage. Do this for say 10
>> >> >>> > minutes
>> >> >>> > when
>> >> >>> > you see the ~100% CPU.
>> >> >>> >
>> >> >>> > top -bHd 5 > /tmp/top.${HOSTNAME}.txt
>> >> >>> >
>> >> >>> > On Wed, Aug 15, 2018 at 2:37 PM Hu Bert 
>> >> >>> > wrote:
>> >> >>> >>
>> >> >>> >> Hello again :-)
>> >> >>> >>
>> >> >>> >> The self heal must have finished as there are no log entries in
>> >> >>> >> glustershd.log files anymore. According to munin disk latency
>> >> >>> >> (average
>> >> >>> >> io wait) has gone down to 100 ms, and disk utilization has gone
>> >> >>> >> down
>> >> >>> >> to ~60% - both on all servers and hard disks.
>> >> >>> >>
>> >> >>> >> But now system load on 2 servers (which were in the good state)
>> >> >>> >> fluctuates between 60 and 100; the server with the formerly
>> >> >>> >> failed
>> >> >>> >> disk has a load of 20-30.I've uploaded some munin graphics of
>> >> >>> >> the
>> >> >>> >> cpu
>> >> >>> >> usage:
>> >> >>> >>
>> >> >>> >> https://abload.de/img/gluster11_cpu31d3a.png
>> >> >>> >> https://abload.de/img/gluster12_cpu8sem7.png
>> >> >>> >> https://abload.de/img/gluster13_cpud7eni.png
>> >> >>> >>
>> >> >>> >> This can't be normal. 2 of the servers under heavy load and one
>> >> >>> >> not
>> >> >>> >> that much. Does anyone have an explanation of this strange
>> >> >>> >> behaviour?
>> >> >>> >>
>> >> >>> >>
>> >> >>> >> Thx :-)
>> >> >>> >>
>> >> >>> >> 2018-08-14 9:37 GMT+02:00 Hu Bert :
>> >> >>> >> > Hi there,
>> >> >>> >> >
>> >> >>> >> > well, it seems the heal has finally finished. Couldn't
>> >> >>> >> > see/find
>> >> >>> >> > any
>> >> >>> >> 

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Pranith Kumar Karampuri]

There are a lot of Lookup operations in the system. But I am not able to
find why. Could you check the output of

# gluster volume heal  info | grep -i number

it should print all zeros.

On Fri, Aug 17, 2018 at 1:49 PM Hu Bert  wrote:

> I don't know what you exactly mean with workload, but the main
> function of the volume is storing (incl. writing, reading) images
> (from hundreds of bytes up to 30 MBs, overall ~7TB). The work is done
> by apache tomcat servers writing to / reading from the volume. Besides
> images there are some text files and binaries that are stored on the
> volume and get updated regularly (every x hours); we'll try to migrate
> the latter ones to local storage asap.
>
> Interestingly it's only one process (and its threads) of the same
> brick on 2 of the gluster servers that consumes the CPU.
>
> gluster11: bricksdd1; not healed; full CPU
> gluster12: bricksdd1; got healed; normal CPU
> gluster13: bricksdd1; got healed; full CPU
>
> Besides: performance during heal (e.g. gluster12, bricksdd1) was way
> better than it is now. I've attached 2 pngs showing the differing cpu
> usage of last week before/after heal.
>
> 2018-08-17 9:30 GMT+02:00 Pranith Kumar Karampuri :
> > There seems to be too many lookup operations compared to any other
> > operations. What is the workload on the volume?
> >
> > On Fri, Aug 17, 2018 at 12:47 PM Hu Bert  wrote:
> >>
> >> i hope i did get it right.
> >>
> >> gluster volume profile shared start
> >> wait 10 minutes
> >> gluster volume profile shared info
> >> gluster volume profile shared stop
> >>
> >> If that's ok, i've attached the output of the info command.
> >>
> >>
> >> 2018-08-17 8:31 GMT+02:00 Pranith Kumar Karampuri  >:
> >> > Please do volume profile also for around 10 minutes when CPU% is high.
> >> >
> >> > On Fri, Aug 17, 2018 at 11:56 AM Pranith Kumar Karampuri
> >> >  wrote:
> >> >>
> >> >> As per the output, all io-threads are using a lot of CPU. It is
> better
> >> >> to
> >> >> check what the volume profile is to see what is leading to so much
> work
> >> >> for
> >> >> io-threads. Please follow the documentation at
> >> >>
> >> >>
> https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Monitoring%20Workload/
> >> >> section: "
> >> >>
> >> >> Running GlusterFS Volume Profile Command"
> >> >>
> >> >> and attach output of  "gluster volume profile info",
> >> >>
> >> >> On Fri, Aug 17, 2018 at 11:24 AM Hu Bert 
> >> >> wrote:
> >> >>>
> >> >>> Good morning,
> >> >>>
> >> >>> i ran the command during 100% CPU usage and attached the file.
> >> >>> Hopefully it helps.
> >> >>>
> >> >>> 2018-08-17 7:33 GMT+02:00 Pranith Kumar Karampuri
> >> >>> :
> >> >>> > Could you do the following on one of the nodes where you are
> >> >>> > observing
> >> >>> > high
> >> >>> > CPU usage and attach that file to this thread? We can find what
> >> >>> > threads/processes are leading to high usage. Do this for say 10
> >> >>> > minutes
> >> >>> > when
> >> >>> > you see the ~100% CPU.
> >> >>> >
> >> >>> > top -bHd 5 > /tmp/top.${HOSTNAME}.txt
> >> >>> >
> >> >>> > On Wed, Aug 15, 2018 at 2:37 PM Hu Bert 
> >> >>> > wrote:
> >> >>> >>
> >> >>> >> Hello again :-)
> >> >>> >>
> >> >>> >> The self heal must have finished as there are no log entries in
> >> >>> >> glustershd.log files anymore. According to munin disk latency
> >> >>> >> (average
> >> >>> >> io wait) has gone down to 100 ms, and disk utilization has gone
> >> >>> >> down
> >> >>> >> to ~60% - both on all servers and hard disks.
> >> >>> >>
> >> >>> >> But now system load on 2 servers (which were in the good state)
> >> >>> >> fluctuates between 60 and 100; the server with the formerly
> failed
> >> >>> >> disk has a load of 20-30.I've uploaded some munin graphics of the
> >> >>> >> cpu
> >> >>> >> usage:
> >> >>> >>
> >> >>> >> https://abload.de/img/gluster11_cpu31d3a.png
> >> >>> >> https://abload.de/img/gluster12_cpu8sem7.png
> >> >>> >> https://abload.de/img/gluster13_cpud7eni.png
> >> >>> >>
> >> >>> >> This can't be normal. 2 of the servers under heavy load and one
> not
> >> >>> >> that much. Does anyone have an explanation of this strange
> >> >>> >> behaviour?
> >> >>> >>
> >> >>> >>
> >> >>> >> Thx :-)
> >> >>> >>
> >> >>> >> 2018-08-14 9:37 GMT+02:00 Hu Bert :
> >> >>> >> > Hi there,
> >> >>> >> >
> >> >>> >> > well, it seems the heal has finally finished. Couldn't see/find
> >> >>> >> > any
> >> >>> >> > related log message; is there such a message in a specific log
> >> >>> >> > file?
> >> >>> >> >
> >> >>> >> > But i see the same behaviour when the last heal finished: all
> CPU
> >> >>> >> > cores are consumed by brick processes; not only by the formerly
> >> >>> >> > failed
> >> >>> >> > bricksdd1, but by all 4 brick processes (and their threads).
> Load
> >> >>> >> > goes
> >> >>> >> > up to > 100 on the 2 servers with the not-failed brick, and
> >> >>> >> > glustershd.log gets filled with a lot of entries. Load on the
> >> >>> >> > server
> >> >>> >> > with the then 

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Hu Bert]

I don't know what you exactly mean with workload, but the main
function of the volume is storing (incl. writing, reading) images
(from hundreds of bytes up to 30 MBs, overall ~7TB). The work is done
by apache tomcat servers writing to / reading from the volume. Besides
images there are some text files and binaries that are stored on the
volume and get updated regularly (every x hours); we'll try to migrate
the latter ones to local storage asap.

Interestingly it's only one process (and its threads) of the same
brick on 2 of the gluster servers that consumes the CPU.

gluster11: bricksdd1; not healed; full CPU
gluster12: bricksdd1; got healed; normal CPU
gluster13: bricksdd1; got healed; full CPU

Besides: performance during heal (e.g. gluster12, bricksdd1) was way
better than it is now. I've attached 2 pngs showing the differing cpu
usage of last week before/after heal.

2018-08-17 9:30 GMT+02:00 Pranith Kumar Karampuri :
> There seems to be too many lookup operations compared to any other
> operations. What is the workload on the volume?
>
> On Fri, Aug 17, 2018 at 12:47 PM Hu Bert  wrote:
>>
>> i hope i did get it right.
>>
>> gluster volume profile shared start
>> wait 10 minutes
>> gluster volume profile shared info
>> gluster volume profile shared stop
>>
>> If that's ok, i've attached the output of the info command.
>>
>>
>> 2018-08-17 8:31 GMT+02:00 Pranith Kumar Karampuri :
>> > Please do volume profile also for around 10 minutes when CPU% is high.
>> >
>> > On Fri, Aug 17, 2018 at 11:56 AM Pranith Kumar Karampuri
>> >  wrote:
>> >>
>> >> As per the output, all io-threads are using a lot of CPU. It is better
>> >> to
>> >> check what the volume profile is to see what is leading to so much work
>> >> for
>> >> io-threads. Please follow the documentation at
>> >>
>> >> https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Monitoring%20Workload/
>> >> section: "
>> >>
>> >> Running GlusterFS Volume Profile Command"
>> >>
>> >> and attach output of  "gluster volume profile info",
>> >>
>> >> On Fri, Aug 17, 2018 at 11:24 AM Hu Bert 
>> >> wrote:
>> >>>
>> >>> Good morning,
>> >>>
>> >>> i ran the command during 100% CPU usage and attached the file.
>> >>> Hopefully it helps.
>> >>>
>> >>> 2018-08-17 7:33 GMT+02:00 Pranith Kumar Karampuri
>> >>> :
>> >>> > Could you do the following on one of the nodes where you are
>> >>> > observing
>> >>> > high
>> >>> > CPU usage and attach that file to this thread? We can find what
>> >>> > threads/processes are leading to high usage. Do this for say 10
>> >>> > minutes
>> >>> > when
>> >>> > you see the ~100% CPU.
>> >>> >
>> >>> > top -bHd 5 > /tmp/top.${HOSTNAME}.txt
>> >>> >
>> >>> > On Wed, Aug 15, 2018 at 2:37 PM Hu Bert 
>> >>> > wrote:
>> >>> >>
>> >>> >> Hello again :-)
>> >>> >>
>> >>> >> The self heal must have finished as there are no log entries in
>> >>> >> glustershd.log files anymore. According to munin disk latency
>> >>> >> (average
>> >>> >> io wait) has gone down to 100 ms, and disk utilization has gone
>> >>> >> down
>> >>> >> to ~60% - both on all servers and hard disks.
>> >>> >>
>> >>> >> But now system load on 2 servers (which were in the good state)
>> >>> >> fluctuates between 60 and 100; the server with the formerly failed
>> >>> >> disk has a load of 20-30.I've uploaded some munin graphics of the
>> >>> >> cpu
>> >>> >> usage:
>> >>> >>
>> >>> >> https://abload.de/img/gluster11_cpu31d3a.png
>> >>> >> https://abload.de/img/gluster12_cpu8sem7.png
>> >>> >> https://abload.de/img/gluster13_cpud7eni.png
>> >>> >>
>> >>> >> This can't be normal. 2 of the servers under heavy load and one not
>> >>> >> that much. Does anyone have an explanation of this strange
>> >>> >> behaviour?
>> >>> >>
>> >>> >>
>> >>> >> Thx :-)
>> >>> >>
>> >>> >> 2018-08-14 9:37 GMT+02:00 Hu Bert :
>> >>> >> > Hi there,
>> >>> >> >
>> >>> >> > well, it seems the heal has finally finished. Couldn't see/find
>> >>> >> > any
>> >>> >> > related log message; is there such a message in a specific log
>> >>> >> > file?
>> >>> >> >
>> >>> >> > But i see the same behaviour when the last heal finished: all CPU
>> >>> >> > cores are consumed by brick processes; not only by the formerly
>> >>> >> > failed
>> >>> >> > bricksdd1, but by all 4 brick processes (and their threads). Load
>> >>> >> > goes
>> >>> >> > up to > 100 on the 2 servers with the not-failed brick, and
>> >>> >> > glustershd.log gets filled with a lot of entries. Load on the
>> >>> >> > server
>> >>> >> > with the then failed brick not that high, but still ~60.
>> >>> >> >
>> >>> >> > Is this behaviour normal? Is there some post-heal after a heal
>> >>> >> > has
>> >>> >> > finished?
>> >>> >> >
>> >>> >> > thx in advance :-)
>> >>> >
>> >>> >
>> >>> >
>> >>> > --
>> >>> > Pranith
>> >>
>> >>
>> >>
>> >> --
>> >> Pranith
>> >
>> >
>> >
>> > --
>> > Pranith
>
>
>
> --
> Pranith
___
Gluster-users mailing list
Gluster-users@gluster.org

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Pranith Kumar Karampuri]

There seems to be too many lookup operations compared to any other
operations. What is the workload on the volume?

On Fri, Aug 17, 2018 at 12:47 PM Hu Bert  wrote:

> i hope i did get it right.
>
> gluster volume profile shared start
> wait 10 minutes
> gluster volume profile shared info
> gluster volume profile shared stop
>
> If that's ok, i've attached the output of the info command.
>
>
> 2018-08-17 8:31 GMT+02:00 Pranith Kumar Karampuri :
> > Please do volume profile also for around 10 minutes when CPU% is high.
> >
> > On Fri, Aug 17, 2018 at 11:56 AM Pranith Kumar Karampuri
> >  wrote:
> >>
> >> As per the output, all io-threads are using a lot of CPU. It is better
> to
> >> check what the volume profile is to see what is leading to so much work
> for
> >> io-threads. Please follow the documentation at
> >>
> https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Monitoring%20Workload/
> >> section: "
> >>
> >> Running GlusterFS Volume Profile Command"
> >>
> >> and attach output of  "gluster volume profile info",
> >>
> >> On Fri, Aug 17, 2018 at 11:24 AM Hu Bert 
> wrote:
> >>>
> >>> Good morning,
> >>>
> >>> i ran the command during 100% CPU usage and attached the file.
> >>> Hopefully it helps.
> >>>
> >>> 2018-08-17 7:33 GMT+02:00 Pranith Kumar Karampuri  >:
> >>> > Could you do the following on one of the nodes where you are
> observing
> >>> > high
> >>> > CPU usage and attach that file to this thread? We can find what
> >>> > threads/processes are leading to high usage. Do this for say 10
> minutes
> >>> > when
> >>> > you see the ~100% CPU.
> >>> >
> >>> > top -bHd 5 > /tmp/top.${HOSTNAME}.txt
> >>> >
> >>> > On Wed, Aug 15, 2018 at 2:37 PM Hu Bert 
> wrote:
> >>> >>
> >>> >> Hello again :-)
> >>> >>
> >>> >> The self heal must have finished as there are no log entries in
> >>> >> glustershd.log files anymore. According to munin disk latency
> (average
> >>> >> io wait) has gone down to 100 ms, and disk utilization has gone down
> >>> >> to ~60% - both on all servers and hard disks.
> >>> >>
> >>> >> But now system load on 2 servers (which were in the good state)
> >>> >> fluctuates between 60 and 100; the server with the formerly failed
> >>> >> disk has a load of 20-30.I've uploaded some munin graphics of the
> cpu
> >>> >> usage:
> >>> >>
> >>> >> https://abload.de/img/gluster11_cpu31d3a.png
> >>> >> https://abload.de/img/gluster12_cpu8sem7.png
> >>> >> https://abload.de/img/gluster13_cpud7eni.png
> >>> >>
> >>> >> This can't be normal. 2 of the servers under heavy load and one not
> >>> >> that much. Does anyone have an explanation of this strange
> behaviour?
> >>> >>
> >>> >>
> >>> >> Thx :-)
> >>> >>
> >>> >> 2018-08-14 9:37 GMT+02:00 Hu Bert :
> >>> >> > Hi there,
> >>> >> >
> >>> >> > well, it seems the heal has finally finished. Couldn't see/find
> any
> >>> >> > related log message; is there such a message in a specific log
> file?
> >>> >> >
> >>> >> > But i see the same behaviour when the last heal finished: all CPU
> >>> >> > cores are consumed by brick processes; not only by the formerly
> >>> >> > failed
> >>> >> > bricksdd1, but by all 4 brick processes (and their threads). Load
> >>> >> > goes
> >>> >> > up to > 100 on the 2 servers with the not-failed brick, and
> >>> >> > glustershd.log gets filled with a lot of entries. Load on the
> server
> >>> >> > with the then failed brick not that high, but still ~60.
> >>> >> >
> >>> >> > Is this behaviour normal? Is there some post-heal after a heal has
> >>> >> > finished?
> >>> >> >
> >>> >> > thx in advance :-)
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> > Pranith
> >>
> >>
> >>
> >> --
> >> Pranith
> >
> >
> >
> > --
> > Pranith
>


-- 
Pranith
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Hu Bert]

i hope i did get it right.

gluster volume profile shared start
wait 10 minutes
gluster volume profile shared info
gluster volume profile shared stop

If that's ok, i've attached the output of the info command.


2018-08-17 8:31 GMT+02:00 Pranith Kumar Karampuri :
> Please do volume profile also for around 10 minutes when CPU% is high.
>
> On Fri, Aug 17, 2018 at 11:56 AM Pranith Kumar Karampuri
>  wrote:
>>
>> As per the output, all io-threads are using a lot of CPU. It is better to
>> check what the volume profile is to see what is leading to so much work for
>> io-threads. Please follow the documentation at
>> https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Monitoring%20Workload/
>> section: "
>>
>> Running GlusterFS Volume Profile Command"
>>
>> and attach output of  "gluster volume profile info",
>>
>> On Fri, Aug 17, 2018 at 11:24 AM Hu Bert  wrote:
>>>
>>> Good morning,
>>>
>>> i ran the command during 100% CPU usage and attached the file.
>>> Hopefully it helps.
>>>
>>> 2018-08-17 7:33 GMT+02:00 Pranith Kumar Karampuri :
>>> > Could you do the following on one of the nodes where you are observing
>>> > high
>>> > CPU usage and attach that file to this thread? We can find what
>>> > threads/processes are leading to high usage. Do this for say 10 minutes
>>> > when
>>> > you see the ~100% CPU.
>>> >
>>> > top -bHd 5 > /tmp/top.${HOSTNAME}.txt
>>> >
>>> > On Wed, Aug 15, 2018 at 2:37 PM Hu Bert  wrote:
>>> >>
>>> >> Hello again :-)
>>> >>
>>> >> The self heal must have finished as there are no log entries in
>>> >> glustershd.log files anymore. According to munin disk latency (average
>>> >> io wait) has gone down to 100 ms, and disk utilization has gone down
>>> >> to ~60% - both on all servers and hard disks.
>>> >>
>>> >> But now system load on 2 servers (which were in the good state)
>>> >> fluctuates between 60 and 100; the server with the formerly failed
>>> >> disk has a load of 20-30.I've uploaded some munin graphics of the cpu
>>> >> usage:
>>> >>
>>> >> https://abload.de/img/gluster11_cpu31d3a.png
>>> >> https://abload.de/img/gluster12_cpu8sem7.png
>>> >> https://abload.de/img/gluster13_cpud7eni.png
>>> >>
>>> >> This can't be normal. 2 of the servers under heavy load and one not
>>> >> that much. Does anyone have an explanation of this strange behaviour?
>>> >>
>>> >>
>>> >> Thx :-)
>>> >>
>>> >> 2018-08-14 9:37 GMT+02:00 Hu Bert :
>>> >> > Hi there,
>>> >> >
>>> >> > well, it seems the heal has finally finished. Couldn't see/find any
>>> >> > related log message; is there such a message in a specific log file?
>>> >> >
>>> >> > But i see the same behaviour when the last heal finished: all CPU
>>> >> > cores are consumed by brick processes; not only by the formerly
>>> >> > failed
>>> >> > bricksdd1, but by all 4 brick processes (and their threads). Load
>>> >> > goes
>>> >> > up to > 100 on the 2 servers with the not-failed brick, and
>>> >> > glustershd.log gets filled with a lot of entries. Load on the server
>>> >> > with the then failed brick not that high, but still ~60.
>>> >> >
>>> >> > Is this behaviour normal? Is there some post-heal after a heal has
>>> >> > finished?
>>> >> >
>>> >> > thx in advance :-)
>>> >
>>> >
>>> >
>>> > --
>>> > Pranith
>>
>>
>>
>> --
>> Pranith
>
>
>
> --
> Pranith
Brick: gluster11:/gluster/bricksda1/shared
--
Cumulative Stats:
   Block Size:  1b+   2b+   4b+ 
 No. of Reads:42346 
No. of Writes:   2768   130 
 
   Block Size:  8b+  16b+  32b+ 
 No. of Reads:   87   162   386 
No. of Writes:  252   496  1524 
 
   Block Size: 64b+ 128b+ 256b+ 
 No. of Reads:  600  1236  2849 
No. of Writes: 5862 18485  8234 
 
   Block Size:512b+1024b+2048b+ 
 No. of Reads: 8971 12202 22124 
No. of Writes:17856 89024 94131 
 
   Block Size:   4096b+8192b+   16384b+ 
 No. of Reads:52532109761340599 
No. of Writes:   231908   2099445401203 
 
   Block Size:  32768b+   65536b+  131072b+ 
 No. of Reads:   241543   1160765   3331181 
No. of Writes:   843531   1576773767822 
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls Fop
 -   ---   ---   --- 

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Pranith Kumar Karampuri]

Please do volume profile also for around 10 minutes when CPU% is high.

On Fri, Aug 17, 2018 at 11:56 AM Pranith Kumar Karampuri <
pkara...@redhat.com> wrote:

> As per the output, all io-threads are using a lot of CPU. It is better to
> check what the volume profile is to see what is leading to so much work for
> io-threads. Please follow the documentation at
> https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Monitoring%20Workload/
> section: "Running GlusterFS Volume Profile Command"and attach output of  
> "gluster
> volume profile info",
>
> On Fri, Aug 17, 2018 at 11:24 AM Hu Bert  wrote:
>
>> Good morning,
>>
>> i ran the command during 100% CPU usage and attached the file.
>> Hopefully it helps.
>>
>> 2018-08-17 7:33 GMT+02:00 Pranith Kumar Karampuri :
>> > Could you do the following on one of the nodes where you are observing
>> high
>> > CPU usage and attach that file to this thread? We can find what
>> > threads/processes are leading to high usage. Do this for say 10 minutes
>> when
>> > you see the ~100% CPU.
>> >
>> > top -bHd 5 > /tmp/top.${HOSTNAME}.txt
>> >
>> > On Wed, Aug 15, 2018 at 2:37 PM Hu Bert  wrote:
>> >>
>> >> Hello again :-)
>> >>
>> >> The self heal must have finished as there are no log entries in
>> >> glustershd.log files anymore. According to munin disk latency (average
>> >> io wait) has gone down to 100 ms, and disk utilization has gone down
>> >> to ~60% - both on all servers and hard disks.
>> >>
>> >> But now system load on 2 servers (which were in the good state)
>> >> fluctuates between 60 and 100; the server with the formerly failed
>> >> disk has a load of 20-30.I've uploaded some munin graphics of the cpu
>> >> usage:
>> >>
>> >> https://abload.de/img/gluster11_cpu31d3a.png
>> >> https://abload.de/img/gluster12_cpu8sem7.png
>> >> https://abload.de/img/gluster13_cpud7eni.png
>> >>
>> >> This can't be normal. 2 of the servers under heavy load and one not
>> >> that much. Does anyone have an explanation of this strange behaviour?
>> >>
>> >>
>> >> Thx :-)
>> >>
>> >> 2018-08-14 9:37 GMT+02:00 Hu Bert :
>> >> > Hi there,
>> >> >
>> >> > well, it seems the heal has finally finished. Couldn't see/find any
>> >> > related log message; is there such a message in a specific log file?
>> >> >
>> >> > But i see the same behaviour when the last heal finished: all CPU
>> >> > cores are consumed by brick processes; not only by the formerly
>> failed
>> >> > bricksdd1, but by all 4 brick processes (and their threads). Load
>> goes
>> >> > up to > 100 on the 2 servers with the not-failed brick, and
>> >> > glustershd.log gets filled with a lot of entries. Load on the server
>> >> > with the then failed brick not that high, but still ~60.
>> >> >
>> >> > Is this behaviour normal? Is there some post-heal after a heal has
>> >> > finished?
>> >> >
>> >> > thx in advance :-)
>> >
>> >
>> >
>> > --
>> > Pranith
>>
>
>
> --
> Pranith
>


-- 
Pranith
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Pranith Kumar Karampuri]

As per the output, all io-threads are using a lot of CPU. It is better to
check what the volume profile is to see what is leading to so much work for
io-threads. Please follow the documentation at
https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Monitoring%20Workload/
section: "Running GlusterFS Volume Profile Command"and attach output
of  "gluster
volume profile info",

On Fri, Aug 17, 2018 at 11:24 AM Hu Bert  wrote:

> Good morning,
>
> i ran the command during 100% CPU usage and attached the file.
> Hopefully it helps.
>
> 2018-08-17 7:33 GMT+02:00 Pranith Kumar Karampuri :
> > Could you do the following on one of the nodes where you are observing
> high
> > CPU usage and attach that file to this thread? We can find what
> > threads/processes are leading to high usage. Do this for say 10 minutes
> when
> > you see the ~100% CPU.
> >
> > top -bHd 5 > /tmp/top.${HOSTNAME}.txt
> >
> > On Wed, Aug 15, 2018 at 2:37 PM Hu Bert  wrote:
> >>
> >> Hello again :-)
> >>
> >> The self heal must have finished as there are no log entries in
> >> glustershd.log files anymore. According to munin disk latency (average
> >> io wait) has gone down to 100 ms, and disk utilization has gone down
> >> to ~60% - both on all servers and hard disks.
> >>
> >> But now system load on 2 servers (which were in the good state)
> >> fluctuates between 60 and 100; the server with the formerly failed
> >> disk has a load of 20-30.I've uploaded some munin graphics of the cpu
> >> usage:
> >>
> >> https://abload.de/img/gluster11_cpu31d3a.png
> >> https://abload.de/img/gluster12_cpu8sem7.png
> >> https://abload.de/img/gluster13_cpud7eni.png
> >>
> >> This can't be normal. 2 of the servers under heavy load and one not
> >> that much. Does anyone have an explanation of this strange behaviour?
> >>
> >>
> >> Thx :-)
> >>
> >> 2018-08-14 9:37 GMT+02:00 Hu Bert :
> >> > Hi there,
> >> >
> >> > well, it seems the heal has finally finished. Couldn't see/find any
> >> > related log message; is there such a message in a specific log file?
> >> >
> >> > But i see the same behaviour when the last heal finished: all CPU
> >> > cores are consumed by brick processes; not only by the formerly failed
> >> > bricksdd1, but by all 4 brick processes (and their threads). Load goes
> >> > up to > 100 on the 2 servers with the not-failed brick, and
> >> > glustershd.log gets filled with a lot of entries. Load on the server
> >> > with the then failed brick not that high, but still ~60.
> >> >
> >> > Is this behaviour normal? Is there some post-heal after a heal has
> >> > finished?
> >> >
> >> > thx in advance :-)
> >
> >
> >
> > --
> > Pranith
>


-- 
Pranith
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Pranith Kumar Karampuri]

Could you do the following on one of the nodes where you are observing high
CPU usage and attach that file to this thread? We can find what
threads/processes are leading to high usage. Do this for say 10 minutes
when you see the ~100% CPU.

top -bHd 5 > /tmp/top.${HOSTNAME}.txt

On Wed, Aug 15, 2018 at 2:37 PM Hu Bert  wrote:

> Hello again :-)
>
> The self heal must have finished as there are no log entries in
> glustershd.log files anymore. According to munin disk latency (average
> io wait) has gone down to 100 ms, and disk utilization has gone down
> to ~60% - both on all servers and hard disks.
>
> But now system load on 2 servers (which were in the good state)
> fluctuates between 60 and 100; the server with the formerly failed
> disk has a load of 20-30.I've uploaded some munin graphics of the cpu
> usage:
>
> https://abload.de/img/gluster11_cpu31d3a.png
> https://abload.de/img/gluster12_cpu8sem7.png
> https://abload.de/img/gluster13_cpud7eni.png
>
> This can't be normal. 2 of the servers under heavy load and one not
> that much. Does anyone have an explanation of this strange behaviour?
>
>
> Thx :-)
>
> 2018-08-14 9:37 GMT+02:00 Hu Bert :
> > Hi there,
> >
> > well, it seems the heal has finally finished. Couldn't see/find any
> > related log message; is there such a message in a specific log file?
> >
> > But i see the same behaviour when the last heal finished: all CPU
> > cores are consumed by brick processes; not only by the formerly failed
> > bricksdd1, but by all 4 brick processes (and their threads). Load goes
> > up to > 100 on the 2 servers with the not-failed brick, and
> > glustershd.log gets filled with a lot of entries. Load on the server
> > with the then failed brick not that high, but still ~60.
> >
> > Is this behaviour normal? Is there some post-heal after a heal has
> finished?
> >
> > thx in advance :-)
>


-- 
Pranith
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Hu Bert]

Hi,

well, as the situation doesn't get better, we're quite helpless and
mostly in the dark, so we're thinking about hiring some professional
support. Any hint? :-)


2018-08-15 11:07 GMT+02:00 Hu Bert :
> Hello again :-)
>
> The self heal must have finished as there are no log entries in
> glustershd.log files anymore. According to munin disk latency (average
> io wait) has gone down to 100 ms, and disk utilization has gone down
> to ~60% - both on all servers and hard disks.
>
> But now system load on 2 servers (which were in the good state)
> fluctuates between 60 and 100; the server with the formerly failed
> disk has a load of 20-30.I've uploaded some munin graphics of the cpu
> usage:
>
> https://abload.de/img/gluster11_cpu31d3a.png
> https://abload.de/img/gluster12_cpu8sem7.png
> https://abload.de/img/gluster13_cpud7eni.png
>
> This can't be normal. 2 of the servers under heavy load and one not
> that much. Does anyone have an explanation of this strange behaviour?
>
>
> Thx :-)
>
> 2018-08-14 9:37 GMT+02:00 Hu Bert :
>> Hi there,
>>
>> well, it seems the heal has finally finished. Couldn't see/find any
>> related log message; is there such a message in a specific log file?
>>
>> But i see the same behaviour when the last heal finished: all CPU
>> cores are consumed by brick processes; not only by the formerly failed
>> bricksdd1, but by all 4 brick processes (and their threads). Load goes
>> up to > 100 on the 2 servers with the not-failed brick, and
>> glustershd.log gets filled with a lot of entries. Load on the server
>> with the then failed brick not that high, but still ~60.
>>
>> Is this behaviour normal? Is there some post-heal after a heal has finished?
>>
>> thx in advance :-)
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Hu Bert]

Hello again :-)

The self heal must have finished as there are no log entries in
glustershd.log files anymore. According to munin disk latency (average
io wait) has gone down to 100 ms, and disk utilization has gone down
to ~60% - both on all servers and hard disks.

But now system load on 2 servers (which were in the good state)
fluctuates between 60 and 100; the server with the formerly failed
disk has a load of 20-30.I've uploaded some munin graphics of the cpu
usage:

https://abload.de/img/gluster11_cpu31d3a.png
https://abload.de/img/gluster12_cpu8sem7.png
https://abload.de/img/gluster13_cpud7eni.png

This can't be normal. 2 of the servers under heavy load and one not
that much. Does anyone have an explanation of this strange behaviour?


Thx :-)

2018-08-14 9:37 GMT+02:00 Hu Bert :
> Hi there,
>
> well, it seems the heal has finally finished. Couldn't see/find any
> related log message; is there such a message in a specific log file?
>
> But i see the same behaviour when the last heal finished: all CPU
> cores are consumed by brick processes; not only by the formerly failed
> bricksdd1, but by all 4 brick processes (and their threads). Load goes
> up to > 100 on the 2 servers with the not-failed brick, and
> glustershd.log gets filled with a lot of entries. Load on the server
> with the then failed brick not that high, but still ~60.
>
> Is this behaviour normal? Is there some post-heal after a heal has finished?
>
> thx in advance :-)
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Hu Bert]

Hi there,

well, it seems the heal has finally finished. Couldn't see/find any
related log message; is there such a message in a specific log file?

But i see the same behaviour when the last heal finished: all CPU
cores are consumed by brick processes; not only by the formerly failed
bricksdd1, but by all 4 brick processes (and their threads). Load goes
up to > 100 on the 2 servers with the not-failed brick, and
glustershd.log gets filled with a lot of entries. Load on the server
with the then failed brick not that high, but still ~60.

Is this behaviour normal? Is there some post-heal after a heal has finished?

thx in advance :-)
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Hu Bert]

Hello :-) Just wanted to give a short report...

>> It could be saturating in the day. But if enough self-heals are going on,
>> even in the night it should have been close to 100%.
>
> Lowest utilization was 70% over night, but i'll check this
> evening/weekend. Also that 'stat...' is running.

At the moment 1.1TB of 2.0TB got healed, disk utilization still
between 100% (day) and 70% (night). So this will take another 10-14
days.

> What, in your opinion, would be better for permormance?
>
> - Having 3 servers and RAID10 (with conventional disks)
> - Having 3 additional servers with 4 hdds (JBOD) each (distribute
> replicate, replica 3)
> - SSDs? (would be quite expensive to reach the storage amount we have
> at the moment)
>
> Just curious. It seems we'll have to adjust our setup during winter anyway :-)

Well, we'll definitely rethink our setup this autumn :-)
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Hu Bert]

>> Btw.: i've seen in the munin stats that the disk utilization for
>> bricksdd1 on the healthy gluster servers is between 70% (night) and
>> almost 99% (daytime). So it looks like that the basic problem is the
>> disk which seems not to be able to work faster? If so (heal)
>> performance won't improve with this setup, i assume.
>
>
> It could be saturating in the day. But if enough self-heals are going on,
> even in the night it should have been close to 100%.

Lowest utilization was 70% over night, but i'll check this
evening/weekend. Also that 'stat...' is running.

>> Maybe switching
>> to RAID10 (conventional hard disks), SSDs or even add 3 additional
>> gluster servers (distributed replicated) could help?
>
>
> It definitely will give better protection against hardware failure. Failure
> domain will be lesser.

What, in your opinion, would be better for permormance?

- Having 3 servers and RAID10 (with conventional disks)
- Having 3 additional servers with 4 hdds (JBOD) each (distribute
replicate, replica 3)
- SSDs? (would be quite expensive to reach the storage amount we have
at the moment)

Just curious. It seems we'll have to adjust our setup during winter anyway :-)

Thanx again :-)
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Pranith Kumar Karampuri]

On Fri, Jul 27, 2018 at 1:32 PM, Hu Bert  wrote:

> 2018-07-27 9:22 GMT+02:00 Pranith Kumar Karampuri :
> >
> >
> > On Fri, Jul 27, 2018 at 12:36 PM, Hu Bert 
> wrote:
> >>
> >> 2018-07-27 8:52 GMT+02:00 Pranith Kumar Karampuri  >:
> >> >
> >> >
> >> > On Fri, Jul 27, 2018 at 11:53 AM, Hu Bert 
> >> > wrote:
> >> >>
> >> >> > Do you already have all the 19 directories already created? If
> >> >> > not
> >> >> > could you find out which of the paths need it and do a stat
> directly
> >> >> > instead
> >> >> > of find?
> >> >>
> >> >> Quite probable not all of them have been created (but counting how
> >> >> much would take very long...). Hm, maybe running stat in a double
> loop
> >> >> (thx to our directory structure) would help. Something like this (may
> >> >> be not 100% correct):
> >> >>
> >> >> for a in ${100..999}; do
> >> >> for b in ${100..999}; do
> >> >> stat /$a/$b/
> >> >> done
> >> >> done
> >> >>
> >> >> Should run stat on all directories. I think i'll give this a try.
> >> >
> >> >
> >> > Just to prevent these served from a cache, it is probably better to do
> >> > this
> >> > from a fresh mount?
> >> >
> >> > --
> >> > Pranith
> >>
> >> Good idea. I'll install glusterfs client on a little used machine, so
> >> there should be no caching. Thx! Have a good weekend when the time
> >> comes :-)
> >
> >
> > If this proves effective, what you need to also do is unmount and mount
> > again, something like:
> >
> > mount
> > for a in ${100..999}; do
> >  for b in ${100..999}; do
> >  stat /$a/$b/
> >  done
> >   done
> > unmount
>
> I'll see what is possible over the weekend.
>
> Btw.: i've seen in the munin stats that the disk utilization for
> bricksdd1 on the healthy gluster servers is between 70% (night) and
> almost 99% (daytime). So it looks like that the basic problem is the
> disk which seems not to be able to work faster? If so (heal)
> performance won't improve with this setup, i assume.


It could be saturating in the day. But if enough self-heals are going on,
even in the night
it should have been close to 100%.



> Maybe switching
> to RAID10 (conventional hard disks), SSDs or even add 3 additional
> gluster servers (distributed replicated) could help?
>

It definitely will give better protection against hardware failure. Failure
domain will be lesser.
-- 
Pranith
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Hu Bert]

2018-07-27 9:22 GMT+02:00 Pranith Kumar Karampuri :
>
>
> On Fri, Jul 27, 2018 at 12:36 PM, Hu Bert  wrote:
>>
>> 2018-07-27 8:52 GMT+02:00 Pranith Kumar Karampuri :
>> >
>> >
>> > On Fri, Jul 27, 2018 at 11:53 AM, Hu Bert 
>> > wrote:
>> >>
>> >> > Do you already have all the 19 directories already created? If
>> >> > not
>> >> > could you find out which of the paths need it and do a stat directly
>> >> > instead
>> >> > of find?
>> >>
>> >> Quite probable not all of them have been created (but counting how
>> >> much would take very long...). Hm, maybe running stat in a double loop
>> >> (thx to our directory structure) would help. Something like this (may
>> >> be not 100% correct):
>> >>
>> >> for a in ${100..999}; do
>> >> for b in ${100..999}; do
>> >> stat /$a/$b/
>> >> done
>> >> done
>> >>
>> >> Should run stat on all directories. I think i'll give this a try.
>> >
>> >
>> > Just to prevent these served from a cache, it is probably better to do
>> > this
>> > from a fresh mount?
>> >
>> > --
>> > Pranith
>>
>> Good idea. I'll install glusterfs client on a little used machine, so
>> there should be no caching. Thx! Have a good weekend when the time
>> comes :-)
>
>
> If this proves effective, what you need to also do is unmount and mount
> again, something like:
>
> mount
> for a in ${100..999}; do
>  for b in ${100..999}; do
>  stat /$a/$b/
>  done
>   done
> unmount

I'll see what is possible over the weekend.

Btw.: i've seen in the munin stats that the disk utilization for
bricksdd1 on the healthy gluster servers is between 70% (night) and
almost 99% (daytime). So it looks like that the basic problem is the
disk which seems not to be able to work faster? If so (heal)
performance won't improve with this setup, i assume. Maybe switching
to RAID10 (conventional hard disks), SSDs or even add 3 additional
gluster servers (distributed replicated) could help?
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Pranith Kumar Karampuri]

On Fri, Jul 27, 2018 at 12:36 PM, Hu Bert  wrote:

> 2018-07-27 8:52 GMT+02:00 Pranith Kumar Karampuri :
> >
> >
> > On Fri, Jul 27, 2018 at 11:53 AM, Hu Bert 
> wrote:
> >>
> >> > Do you already have all the 19 directories already created? If not
> >> > could you find out which of the paths need it and do a stat directly
> instead
> >> > of find?
> >>
> >> Quite probable not all of them have been created (but counting how
> >> much would take very long...). Hm, maybe running stat in a double loop
> >> (thx to our directory structure) would help. Something like this (may
> >> be not 100% correct):
> >>
> >> for a in ${100..999}; do
> >> for b in ${100..999}; do
> >> stat /$a/$b/
> >> done
> >> done
> >>
> >> Should run stat on all directories. I think i'll give this a try.
> >
> >
> > Just to prevent these served from a cache, it is probably better to do
> this
> > from a fresh mount?
> >
> > --
> > Pranith
>
> Good idea. I'll install glusterfs client on a little used machine, so
> there should be no caching. Thx! Have a good weekend when the time
> comes :-)
>

If this proves effective, what you need to also do is unmount and mount
again, something like:

mount
for a in ${100..999}; do
 for b in ${100..999}; do
 stat /$a/$b/
 done
  done
unmount

-- 
Pranith
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Hu Bert]

2018-07-27 8:52 GMT+02:00 Pranith Kumar Karampuri :
>
>
> On Fri, Jul 27, 2018 at 11:53 AM, Hu Bert  wrote:
>>
>> > Do you already have all the 19 directories already created? If not
>> > could you find out which of the paths need it and do a stat directly 
>> > instead
>> > of find?
>>
>> Quite probable not all of them have been created (but counting how
>> much would take very long...). Hm, maybe running stat in a double loop
>> (thx to our directory structure) would help. Something like this (may
>> be not 100% correct):
>>
>> for a in ${100..999}; do
>> for b in ${100..999}; do
>> stat /$a/$b/
>> done
>> done
>>
>> Should run stat on all directories. I think i'll give this a try.
>
>
> Just to prevent these served from a cache, it is probably better to do this
> from a fresh mount?
>
> --
> Pranith

Good idea. I'll install glusterfs client on a little used machine, so
there should be no caching. Thx! Have a good weekend when the time
comes :-)
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Pranith Kumar Karampuri]

On Fri, Jul 27, 2018 at 11:53 AM, Hu Bert  wrote:

> > Do you already have all the 19 directories already created? If not
> could you find out which of the paths need it and do a stat directly
> instead of find?
>
> Quite probable not all of them have been created (but counting how
> much would take very long...). Hm, maybe running stat in a double loop
> (thx to our directory structure) would help. Something like this (may
> be not 100% correct):
>
> for a in ${100..999}; do
> for b in ${100..999}; do
> stat /$a/$b/
> done
> done
>
> Should run stat on all directories. I think i'll give this a try.
>

Just to prevent these served from a cache, it is probably better to do this
from a fresh mount?

-- 
Pranith
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Hu Bert]

> Do you already have all the 19 directories already created? If not could 
> you find out which of the paths need it and do a stat directly instead of 
> find?

Quite probable not all of them have been created (but counting how
much would take very long...). Hm, maybe running stat in a double loop
(thx to our directory structure) would help. Something like this (may
be not 100% correct):

for a in ${100..999}; do
for b in ${100..999}; do
stat /$a/$b/
done
done

Should run stat on all directories. I think i'll give this a try.
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Pranith Kumar Karampuri]

On Fri, Jul 27, 2018 at 11:11 AM, Hu Bert  wrote:

> Good Morning :-)
>
> on server gluster11 about 1.25 million and on gluster13 about 1.35
> million log entries in glustershd.log file. About 70 GB got healed,
> overall ~700GB of 2.0TB. Doesn't seem to run faster. I'm calling
> 'find...' whenever i notice that it has finished. Hmm... is it
> possible and reasonable to run 2 finds in parallel, maybe on different
> subdirectories? E.g. running one one $volume/public/ and on one
> $volume/private/ ?
>

Do you already have all the 19 directories already created? If not
could you find out which of the paths need it and do a stat directly
instead of find?


>
> 2018-07-26 11:29 GMT+02:00 Pranith Kumar Karampuri :
> >
> >
> > On Thu, Jul 26, 2018 at 2:41 PM, Hu Bert  wrote:
> >>
> >> > Sorry, bad copy/paste :-(.
> >>
> >> np :-)
> >>
> >> The question regarding version 4.1 was meant more generally: does
> >> gluster v4.0 etc. have a better performance than version 3.12 etc.?
> >> Just curious :-) Sooner or later we have to upgrade anyway.
> >
> >
> > You can check what changed @
> > https://github.com/gluster/glusterfs/blob/release-4.0/
> doc/release-notes/4.0.0.md#performance
> > https://github.com/gluster/glusterfs/blob/release-4.1/
> doc/release-notes/4.1.0.md#performance
> >
> >>
> >>
> >> btw.: gluster12 was the node with the failed brick, and i started the
> >> full heal on this node (has the biggest uuid as well). Is it normal
> >> that the glustershd.log on this node is rather empty (some hundred
> >> entries), but the glustershd.log files on the 2 other nodes have
> >> hundreds of thousands of entries?
> >
> >
> > heals happen on the good bricks, so this is expected.
> >
> >>
> >> (sry, mail twice, didn't go to the list, but maybe others are
> >> interested... :-) )
> >>
> >> 2018-07-26 10:17 GMT+02:00 Pranith Kumar Karampuri  >:
> >> >
> >> >
> >> > On Thu, Jul 26, 2018 at 12:59 PM, Hu Bert 
> >> > wrote:
> >> >>
> >> >> Hi Pranith,
> >> >>
> >> >> thanks a lot for your efforts and for tracking "my" problem with an
> >> >> issue.
> >> >> :-)
> >> >>
> >> >>
> >> >> I've set this params on the gluster volume and will start the
> >> >> 'find...' command within a short time. I'll probably add another
> >> >> answer to the list to document the progress.
> >> >>
> >> >> btw. - you had some typos:
> >> >> gluster volume set  cluster.cluster.heal-wait-queue-length
> >> >> 1 => cluster is doubled
> >> >> gluster volume set  cluster.data-self-heal-window-size 16
> =>
> >> >> it's actually cluster.self-heal-window-size
> >> >>
> >> >> but actually no problem :-)
> >> >
> >> >
> >> > Sorry, bad copy/paste :-(.
> >> >
> >> >>
> >> >>
> >> >> Just curious: would gluster 4.1 improve the performance for healing
> >> >> and in general for "my" scenario?
> >> >
> >> >
> >> > No, this issue is present in all the existing releases. But it is
> >> > solvable.
> >> > You can follow that issue to see progress and when it is fixed etc.
> >> >
> >> >>
> >> >>
> >> >> 2018-07-26 8:56 GMT+02:00 Pranith Kumar Karampuri
> >> >> :
> >> >> > Thanks a lot for detailed write-up, this helps find the bottlenecks
> >> >> > easily.
> >> >> > On a high level, to handle this directory hierarchy i.e. lots of
> >> >> > directories
> >> >> > with files, we need to improve healing
> >> >> > algorithms. Based on the data you provided, we need to make the
> >> >> > following
> >> >> > enhancements:
> >> >> >
> >> >> > 1) At the moment directories are healed one at a time, but files
> can
> >> >> > be
> >> >> > healed upto 64 in parallel per replica subvolume.
> >> >> > So if you have nX2 or nX3 distributed subvolumes, it can heal 64n
> >> >> > number
> >> >> > of
> >> >> > files in parallel.
> >> >> >
> >> >> > I raised https://github.com/gluster/glusterfs/issues/477 to track
> >> >> > this.
> >> >> > In
> >> >> > the mean-while you can use the following work-around:
> >> >> > a) Increase background heals on the mount:
> >> >> > gluster volume set  cluster.background-self-heal-count
> 256
> >> >> > gluster volume set  cluster.cluster.heal-wait-
> queue-length
> >> >> > 1
> >> >> > find  -type d | xargs stat
> >> >> >
> >> >> > one 'find' will trigger 10256 directories. So you may have to do
> this
> >> >> > periodically until all directories are healed.
> >> >> >
> >> >> > 2) Self-heal heals a file 128KB at a
> >> >> > time(data-self-heal-window-size). I
> >> >> > think for your environment bumping upto MBs is better. Say 2MB i.e.
> >> >> > 16*128KB?
> >> >> >
> >> >> > Command to do that is:
> >> >> > gluster volume set  cluster.data-self-heal-window-size 16
> >> >> >
> >> >> >
> >> >> > On Thu, Jul 26, 2018 at 10:40 AM, Hu Bert 
> >> >> > wrote:
> >> >> >>
> >> >> >> Hi Pranith,
> >> >> >>
> >> >> >> Sry, it took a while to count the directories. I'll try to answer
> >> >> >> your
> >> >> >> questions as good as possible.
> >> >> >>
> >> >> >> > What kind of data do you have?
> >> >> >> > How many directories in the filesystem?
> 

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Hu Bert]

Good Morning :-)

on server gluster11 about 1.25 million and on gluster13 about 1.35
million log entries in glustershd.log file. About 70 GB got healed,
overall ~700GB of 2.0TB. Doesn't seem to run faster. I'm calling
'find...' whenever i notice that it has finished. Hmm... is it
possible and reasonable to run 2 finds in parallel, maybe on different
subdirectories? E.g. running one one $volume/public/ and on one
$volume/private/ ?

2018-07-26 11:29 GMT+02:00 Pranith Kumar Karampuri :
>
>
> On Thu, Jul 26, 2018 at 2:41 PM, Hu Bert  wrote:
>>
>> > Sorry, bad copy/paste :-(.
>>
>> np :-)
>>
>> The question regarding version 4.1 was meant more generally: does
>> gluster v4.0 etc. have a better performance than version 3.12 etc.?
>> Just curious :-) Sooner or later we have to upgrade anyway.
>
>
> You can check what changed @
> https://github.com/gluster/glusterfs/blob/release-4.0/doc/release-notes/4.0.0.md#performance
> https://github.com/gluster/glusterfs/blob/release-4.1/doc/release-notes/4.1.0.md#performance
>
>>
>>
>> btw.: gluster12 was the node with the failed brick, and i started the
>> full heal on this node (has the biggest uuid as well). Is it normal
>> that the glustershd.log on this node is rather empty (some hundred
>> entries), but the glustershd.log files on the 2 other nodes have
>> hundreds of thousands of entries?
>
>
> heals happen on the good bricks, so this is expected.
>
>>
>> (sry, mail twice, didn't go to the list, but maybe others are
>> interested... :-) )
>>
>> 2018-07-26 10:17 GMT+02:00 Pranith Kumar Karampuri :
>> >
>> >
>> > On Thu, Jul 26, 2018 at 12:59 PM, Hu Bert 
>> > wrote:
>> >>
>> >> Hi Pranith,
>> >>
>> >> thanks a lot for your efforts and for tracking "my" problem with an
>> >> issue.
>> >> :-)
>> >>
>> >>
>> >> I've set this params on the gluster volume and will start the
>> >> 'find...' command within a short time. I'll probably add another
>> >> answer to the list to document the progress.
>> >>
>> >> btw. - you had some typos:
>> >> gluster volume set  cluster.cluster.heal-wait-queue-length
>> >> 1 => cluster is doubled
>> >> gluster volume set  cluster.data-self-heal-window-size 16 =>
>> >> it's actually cluster.self-heal-window-size
>> >>
>> >> but actually no problem :-)
>> >
>> >
>> > Sorry, bad copy/paste :-(.
>> >
>> >>
>> >>
>> >> Just curious: would gluster 4.1 improve the performance for healing
>> >> and in general for "my" scenario?
>> >
>> >
>> > No, this issue is present in all the existing releases. But it is
>> > solvable.
>> > You can follow that issue to see progress and when it is fixed etc.
>> >
>> >>
>> >>
>> >> 2018-07-26 8:56 GMT+02:00 Pranith Kumar Karampuri
>> >> :
>> >> > Thanks a lot for detailed write-up, this helps find the bottlenecks
>> >> > easily.
>> >> > On a high level, to handle this directory hierarchy i.e. lots of
>> >> > directories
>> >> > with files, we need to improve healing
>> >> > algorithms. Based on the data you provided, we need to make the
>> >> > following
>> >> > enhancements:
>> >> >
>> >> > 1) At the moment directories are healed one at a time, but files can
>> >> > be
>> >> > healed upto 64 in parallel per replica subvolume.
>> >> > So if you have nX2 or nX3 distributed subvolumes, it can heal 64n
>> >> > number
>> >> > of
>> >> > files in parallel.
>> >> >
>> >> > I raised https://github.com/gluster/glusterfs/issues/477 to track
>> >> > this.
>> >> > In
>> >> > the mean-while you can use the following work-around:
>> >> > a) Increase background heals on the mount:
>> >> > gluster volume set  cluster.background-self-heal-count 256
>> >> > gluster volume set  cluster.cluster.heal-wait-queue-length
>> >> > 1
>> >> > find  -type d | xargs stat
>> >> >
>> >> > one 'find' will trigger 10256 directories. So you may have to do this
>> >> > periodically until all directories are healed.
>> >> >
>> >> > 2) Self-heal heals a file 128KB at a
>> >> > time(data-self-heal-window-size). I
>> >> > think for your environment bumping upto MBs is better. Say 2MB i.e.
>> >> > 16*128KB?
>> >> >
>> >> > Command to do that is:
>> >> > gluster volume set  cluster.data-self-heal-window-size 16
>> >> >
>> >> >
>> >> > On Thu, Jul 26, 2018 at 10:40 AM, Hu Bert 
>> >> > wrote:
>> >> >>
>> >> >> Hi Pranith,
>> >> >>
>> >> >> Sry, it took a while to count the directories. I'll try to answer
>> >> >> your
>> >> >> questions as good as possible.
>> >> >>
>> >> >> > What kind of data do you have?
>> >> >> > How many directories in the filesystem?
>> >> >> > On average how many files per directory?
>> >> >> > What is the depth of your directory hierarchy on average?
>> >> >> > What is average filesize?
>> >> >>
>> >> >> We have mostly images (more than 95% of disk usage, 90% of file
>> >> >> count), some text files (like css, jsp, gpx etc.) and some binaries.
>> >> >>
>> >> >> There are about 190.000 directories in the file system; maybe there
>> >> >> are some more because we're hit by bug 1512371 (parallel-readdir =
>> >> >> TRUE prevents 

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Pranith Kumar Karampuri]

On Thu, Jul 26, 2018 at 2:41 PM, Hu Bert  wrote:

> > Sorry, bad copy/paste :-(.
>
> np :-)
>
> The question regarding version 4.1 was meant more generally: does
> gluster v4.0 etc. have a better performance than version 3.12 etc.?
> Just curious :-) Sooner or later we have to upgrade anyway.


You can check what changed @
https://github.com/gluster/glusterfs/blob/release-4.0/doc/release-notes/4.0.0.md#performance
https://github.com/gluster/glusterfs/blob/release-4.1/doc/release-notes/4.1.0.md#performance

>
>
> btw.: gluster12 was the node with the failed brick, and i started the
> full heal on this node (has the biggest uuid as well). Is it normal
> that the glustershd.log on this node is rather empty (some hundred
> entries), but the glustershd.log files on the 2 other nodes have
> hundreds of thousands of entries?
>

heals happen on the good bricks, so this is expected.


> (sry, mail twice, didn't go to the list, but maybe others are
> interested... :-) )
>
> 2018-07-26 10:17 GMT+02:00 Pranith Kumar Karampuri :
> >
> >
> > On Thu, Jul 26, 2018 at 12:59 PM, Hu Bert 
> wrote:
> >>
> >> Hi Pranith,
> >>
> >> thanks a lot for your efforts and for tracking "my" problem with an
> issue.
> >> :-)
> >>
> >>
> >> I've set this params on the gluster volume and will start the
> >> 'find...' command within a short time. I'll probably add another
> >> answer to the list to document the progress.
> >>
> >> btw. - you had some typos:
> >> gluster volume set  cluster.cluster.heal-wait-queue-length
> >> 1 => cluster is doubled
> >> gluster volume set  cluster.data-self-heal-window-size 16 =>
> >> it's actually cluster.self-heal-window-size
> >>
> >> but actually no problem :-)
> >
> >
> > Sorry, bad copy/paste :-(.
> >
> >>
> >>
> >> Just curious: would gluster 4.1 improve the performance for healing
> >> and in general for "my" scenario?
> >
> >
> > No, this issue is present in all the existing releases. But it is
> solvable.
> > You can follow that issue to see progress and when it is fixed etc.
> >
> >>
> >>
> >> 2018-07-26 8:56 GMT+02:00 Pranith Kumar Karampuri  >:
> >> > Thanks a lot for detailed write-up, this helps find the bottlenecks
> >> > easily.
> >> > On a high level, to handle this directory hierarchy i.e. lots of
> >> > directories
> >> > with files, we need to improve healing
> >> > algorithms. Based on the data you provided, we need to make the
> >> > following
> >> > enhancements:
> >> >
> >> > 1) At the moment directories are healed one at a time, but files can
> be
> >> > healed upto 64 in parallel per replica subvolume.
> >> > So if you have nX2 or nX3 distributed subvolumes, it can heal 64n
> number
> >> > of
> >> > files in parallel.
> >> >
> >> > I raised https://github.com/gluster/glusterfs/issues/477 to track
> this.
> >> > In
> >> > the mean-while you can use the following work-around:
> >> > a) Increase background heals on the mount:
> >> > gluster volume set  cluster.background-self-heal-count 256
> >> > gluster volume set  cluster.cluster.heal-wait-queue-length
> >> > 1
> >> > find  -type d | xargs stat
> >> >
> >> > one 'find' will trigger 10256 directories. So you may have to do this
> >> > periodically until all directories are healed.
> >> >
> >> > 2) Self-heal heals a file 128KB at a time(data-self-heal-window-size).
> I
> >> > think for your environment bumping upto MBs is better. Say 2MB i.e.
> >> > 16*128KB?
> >> >
> >> > Command to do that is:
> >> > gluster volume set  cluster.data-self-heal-window-size 16
> >> >
> >> >
> >> > On Thu, Jul 26, 2018 at 10:40 AM, Hu Bert 
> >> > wrote:
> >> >>
> >> >> Hi Pranith,
> >> >>
> >> >> Sry, it took a while to count the directories. I'll try to answer
> your
> >> >> questions as good as possible.
> >> >>
> >> >> > What kind of data do you have?
> >> >> > How many directories in the filesystem?
> >> >> > On average how many files per directory?
> >> >> > What is the depth of your directory hierarchy on average?
> >> >> > What is average filesize?
> >> >>
> >> >> We have mostly images (more than 95% of disk usage, 90% of file
> >> >> count), some text files (like css, jsp, gpx etc.) and some binaries.
> >> >>
> >> >> There are about 190.000 directories in the file system; maybe there
> >> >> are some more because we're hit by bug 1512371 (parallel-readdir =
> >> >> TRUE prevents directories listing). But the number of directories
> >> >> could/will rise in the future (maybe millions).
> >> >>
> >> >> files per directory: ranges from 0 to 100, on average it should be 20
> >> >> files per directory (well, at least in the deepest dirs, see
> >> >> explanation below).
> >> >>
> >> >> Average filesize: ranges from a few hundred bytes up to 30 MB, on
> >> >> average it should be 2-3 MB.
> >> >>
> >> >> Directory hierarchy: maximum depth as seen from within the volume is
> >> >> 6, the average should be 3.
> >> >>
> >> >> volume name: shared
> >> >> mount point on clients: /data/repository/shared/
> >> >> below /shared/ there are 2 

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Hu Bert]

> Sorry, bad copy/paste :-(.

np :-)

The question regarding version 4.1 was meant more generally: does
gluster v4.0 etc. have a better performance than version 3.12 etc.?
Just curious :-) Sooner or later we have to upgrade anyway.

btw.: gluster12 was the node with the failed brick, and i started the
full heal on this node (has the biggest uuid as well). Is it normal
that the glustershd.log on this node is rather empty (some hundred
entries), but the glustershd.log files on the 2 other nodes have
hundreds of thousands of entries?

(sry, mail twice, didn't go to the list, but maybe others are
interested... :-) )

2018-07-26 10:17 GMT+02:00 Pranith Kumar Karampuri :
>
>
> On Thu, Jul 26, 2018 at 12:59 PM, Hu Bert  wrote:
>>
>> Hi Pranith,
>>
>> thanks a lot for your efforts and for tracking "my" problem with an issue.
>> :-)
>>
>>
>> I've set this params on the gluster volume and will start the
>> 'find...' command within a short time. I'll probably add another
>> answer to the list to document the progress.
>>
>> btw. - you had some typos:
>> gluster volume set  cluster.cluster.heal-wait-queue-length
>> 1 => cluster is doubled
>> gluster volume set  cluster.data-self-heal-window-size 16 =>
>> it's actually cluster.self-heal-window-size
>>
>> but actually no problem :-)
>
>
> Sorry, bad copy/paste :-(.
>
>>
>>
>> Just curious: would gluster 4.1 improve the performance for healing
>> and in general for "my" scenario?
>
>
> No, this issue is present in all the existing releases. But it is solvable.
> You can follow that issue to see progress and when it is fixed etc.
>
>>
>>
>> 2018-07-26 8:56 GMT+02:00 Pranith Kumar Karampuri :
>> > Thanks a lot for detailed write-up, this helps find the bottlenecks
>> > easily.
>> > On a high level, to handle this directory hierarchy i.e. lots of
>> > directories
>> > with files, we need to improve healing
>> > algorithms. Based on the data you provided, we need to make the
>> > following
>> > enhancements:
>> >
>> > 1) At the moment directories are healed one at a time, but files can be
>> > healed upto 64 in parallel per replica subvolume.
>> > So if you have nX2 or nX3 distributed subvolumes, it can heal 64n number
>> > of
>> > files in parallel.
>> >
>> > I raised https://github.com/gluster/glusterfs/issues/477 to track this.
>> > In
>> > the mean-while you can use the following work-around:
>> > a) Increase background heals on the mount:
>> > gluster volume set  cluster.background-self-heal-count 256
>> > gluster volume set  cluster.cluster.heal-wait-queue-length
>> > 1
>> > find  -type d | xargs stat
>> >
>> > one 'find' will trigger 10256 directories. So you may have to do this
>> > periodically until all directories are healed.
>> >
>> > 2) Self-heal heals a file 128KB at a time(data-self-heal-window-size). I
>> > think for your environment bumping upto MBs is better. Say 2MB i.e.
>> > 16*128KB?
>> >
>> > Command to do that is:
>> > gluster volume set  cluster.data-self-heal-window-size 16
>> >
>> >
>> > On Thu, Jul 26, 2018 at 10:40 AM, Hu Bert 
>> > wrote:
>> >>
>> >> Hi Pranith,
>> >>
>> >> Sry, it took a while to count the directories. I'll try to answer your
>> >> questions as good as possible.
>> >>
>> >> > What kind of data do you have?
>> >> > How many directories in the filesystem?
>> >> > On average how many files per directory?
>> >> > What is the depth of your directory hierarchy on average?
>> >> > What is average filesize?
>> >>
>> >> We have mostly images (more than 95% of disk usage, 90% of file
>> >> count), some text files (like css, jsp, gpx etc.) and some binaries.
>> >>
>> >> There are about 190.000 directories in the file system; maybe there
>> >> are some more because we're hit by bug 1512371 (parallel-readdir =
>> >> TRUE prevents directories listing). But the number of directories
>> >> could/will rise in the future (maybe millions).
>> >>
>> >> files per directory: ranges from 0 to 100, on average it should be 20
>> >> files per directory (well, at least in the deepest dirs, see
>> >> explanation below).
>> >>
>> >> Average filesize: ranges from a few hundred bytes up to 30 MB, on
>> >> average it should be 2-3 MB.
>> >>
>> >> Directory hierarchy: maximum depth as seen from within the volume is
>> >> 6, the average should be 3.
>> >>
>> >> volume name: shared
>> >> mount point on clients: /data/repository/shared/
>> >> below /shared/ there are 2 directories:
>> >> - public/: mainly calculated images (file sizes from a few KB up to
>> >> max 1 MB) and some resouces (small PNGs with a size of a few hundred
>> >> bytes).
>> >> - private/: mainly source images; file sizes from 50 KB up to 30MB
>> >>
>> >> We migrated from a NFS server (SPOF) to glusterfs and simply copied
>> >> our files. The images (which have an ID) are stored in the deepest
>> >> directories of the dir tree. I'll better explain it :-)
>> >>
>> >> directory structure for the images (i'll omit some other miscellaneous
>> >> stuff, but it looks quite similar):
>> >> - 

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Pranith Kumar Karampuri]

On Thu, Jul 26, 2018 at 12:59 PM, Hu Bert  wrote:

> Hi Pranith,
>
> thanks a lot for your efforts and for tracking "my" problem with an issue.
> :-)
>

> I've set this params on the gluster volume and will start the
> 'find...' command within a short time. I'll probably add another
> answer to the list to document the progress.
>
> btw. - you had some typos:
> gluster volume set  cluster.cluster.heal-wait-queue-length
> 1 => cluster is doubled
> gluster volume set  cluster.data-self-heal-window-size 16 =>
> it's actually cluster.self-heal-window-size
>
> but actually no problem :-)
>

Sorry, bad copy/paste :-(.


>
> Just curious: would gluster 4.1 improve the performance for healing
> and in general for "my" scenario?
>

No, this issue is present in all the existing releases. But it is solvable.
You can follow that issue to see progress and when it is fixed etc.


>
> 2018-07-26 8:56 GMT+02:00 Pranith Kumar Karampuri :
> > Thanks a lot for detailed write-up, this helps find the bottlenecks
> easily.
> > On a high level, to handle this directory hierarchy i.e. lots of
> directories
> > with files, we need to improve healing
> > algorithms. Based on the data you provided, we need to make the following
> > enhancements:
> >
> > 1) At the moment directories are healed one at a time, but files can be
> > healed upto 64 in parallel per replica subvolume.
> > So if you have nX2 or nX3 distributed subvolumes, it can heal 64n number
> of
> > files in parallel.
> >
> > I raised https://github.com/gluster/glusterfs/issues/477 to track this.
> In
> > the mean-while you can use the following work-around:
> > a) Increase background heals on the mount:
> > gluster volume set  cluster.background-self-heal-count 256
> > gluster volume set  cluster.cluster.heal-wait-queue-length
> 1
> > find  -type d | xargs stat
> >
> > one 'find' will trigger 10256 directories. So you may have to do this
> > periodically until all directories are healed.
> >
> > 2) Self-heal heals a file 128KB at a time(data-self-heal-window-size). I
> > think for your environment bumping upto MBs is better. Say 2MB i.e.
> > 16*128KB?
> >
> > Command to do that is:
> > gluster volume set  cluster.data-self-heal-window-size 16
> >
> >
> > On Thu, Jul 26, 2018 at 10:40 AM, Hu Bert 
> wrote:
> >>
> >> Hi Pranith,
> >>
> >> Sry, it took a while to count the directories. I'll try to answer your
> >> questions as good as possible.
> >>
> >> > What kind of data do you have?
> >> > How many directories in the filesystem?
> >> > On average how many files per directory?
> >> > What is the depth of your directory hierarchy on average?
> >> > What is average filesize?
> >>
> >> We have mostly images (more than 95% of disk usage, 90% of file
> >> count), some text files (like css, jsp, gpx etc.) and some binaries.
> >>
> >> There are about 190.000 directories in the file system; maybe there
> >> are some more because we're hit by bug 1512371 (parallel-readdir =
> >> TRUE prevents directories listing). But the number of directories
> >> could/will rise in the future (maybe millions).
> >>
> >> files per directory: ranges from 0 to 100, on average it should be 20
> >> files per directory (well, at least in the deepest dirs, see
> >> explanation below).
> >>
> >> Average filesize: ranges from a few hundred bytes up to 30 MB, on
> >> average it should be 2-3 MB.
> >>
> >> Directory hierarchy: maximum depth as seen from within the volume is
> >> 6, the average should be 3.
> >>
> >> volume name: shared
> >> mount point on clients: /data/repository/shared/
> >> below /shared/ there are 2 directories:
> >> - public/: mainly calculated images (file sizes from a few KB up to
> >> max 1 MB) and some resouces (small PNGs with a size of a few hundred
> >> bytes).
> >> - private/: mainly source images; file sizes from 50 KB up to 30MB
> >>
> >> We migrated from a NFS server (SPOF) to glusterfs and simply copied
> >> our files. The images (which have an ID) are stored in the deepest
> >> directories of the dir tree. I'll better explain it :-)
> >>
> >> directory structure for the images (i'll omit some other miscellaneous
> >> stuff, but it looks quite similar):
> >> - ID of an image has 7 or 8 digits
> >> - /shared/private/: /(first 3 digits of ID)/(next 3 digits of
> ID)/$ID.jpg
> >> - /shared/public/: /(first 3 digits of ID)/(next 3 digits of
> >> ID)/$ID/$misc_formats.jpg
> >>
> >> That's why we have that many (sub-)directories. Files are only stored
> >> in the lowest directory hierarchy. I hope i could make our structure
> >> at least a bit more transparent.
> >>
> >> i hope there's something we can do to raise performance a bit. thx in
> >> advance :-)
> >>
> >>
> >> 2018-07-24 10:40 GMT+02:00 Pranith Kumar Karampuri  >:
> >> >
> >> >
> >> > On Mon, Jul 23, 2018 at 4:16 PM, Hu Bert 
> wrote:
> >> >>
> >> >> Well, over the weekend about 200GB were copied, so now there are
> >> >> ~400GB copied to the brick. That's far beyond a speed of 10GB per
> >> >> hour. If I 

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Hu Bert]

Hi Pranith,

thanks a lot for your efforts and for tracking "my" problem with an issue. :-)

I've set this params on the gluster volume and will start the
'find...' command within a short time. I'll probably add another
answer to the list to document the progress.

btw. - you had some typos:
gluster volume set  cluster.cluster.heal-wait-queue-length
1 => cluster is doubled
gluster volume set  cluster.data-self-heal-window-size 16 =>
it's actually cluster.self-heal-window-size

but actually no problem :-)

Just curious: would gluster 4.1 improve the performance for healing
and in general for "my" scenario?

2018-07-26 8:56 GMT+02:00 Pranith Kumar Karampuri :
> Thanks a lot for detailed write-up, this helps find the bottlenecks easily.
> On a high level, to handle this directory hierarchy i.e. lots of directories
> with files, we need to improve healing
> algorithms. Based on the data you provided, we need to make the following
> enhancements:
>
> 1) At the moment directories are healed one at a time, but files can be
> healed upto 64 in parallel per replica subvolume.
> So if you have nX2 or nX3 distributed subvolumes, it can heal 64n number of
> files in parallel.
>
> I raised https://github.com/gluster/glusterfs/issues/477 to track this. In
> the mean-while you can use the following work-around:
> a) Increase background heals on the mount:
> gluster volume set  cluster.background-self-heal-count 256
> gluster volume set  cluster.cluster.heal-wait-queue-length 1
> find  -type d | xargs stat
>
> one 'find' will trigger 10256 directories. So you may have to do this
> periodically until all directories are healed.
>
> 2) Self-heal heals a file 128KB at a time(data-self-heal-window-size). I
> think for your environment bumping upto MBs is better. Say 2MB i.e.
> 16*128KB?
>
> Command to do that is:
> gluster volume set  cluster.data-self-heal-window-size 16
>
>
> On Thu, Jul 26, 2018 at 10:40 AM, Hu Bert  wrote:
>>
>> Hi Pranith,
>>
>> Sry, it took a while to count the directories. I'll try to answer your
>> questions as good as possible.
>>
>> > What kind of data do you have?
>> > How many directories in the filesystem?
>> > On average how many files per directory?
>> > What is the depth of your directory hierarchy on average?
>> > What is average filesize?
>>
>> We have mostly images (more than 95% of disk usage, 90% of file
>> count), some text files (like css, jsp, gpx etc.) and some binaries.
>>
>> There are about 190.000 directories in the file system; maybe there
>> are some more because we're hit by bug 1512371 (parallel-readdir =
>> TRUE prevents directories listing). But the number of directories
>> could/will rise in the future (maybe millions).
>>
>> files per directory: ranges from 0 to 100, on average it should be 20
>> files per directory (well, at least in the deepest dirs, see
>> explanation below).
>>
>> Average filesize: ranges from a few hundred bytes up to 30 MB, on
>> average it should be 2-3 MB.
>>
>> Directory hierarchy: maximum depth as seen from within the volume is
>> 6, the average should be 3.
>>
>> volume name: shared
>> mount point on clients: /data/repository/shared/
>> below /shared/ there are 2 directories:
>> - public/: mainly calculated images (file sizes from a few KB up to
>> max 1 MB) and some resouces (small PNGs with a size of a few hundred
>> bytes).
>> - private/: mainly source images; file sizes from 50 KB up to 30MB
>>
>> We migrated from a NFS server (SPOF) to glusterfs and simply copied
>> our files. The images (which have an ID) are stored in the deepest
>> directories of the dir tree. I'll better explain it :-)
>>
>> directory structure for the images (i'll omit some other miscellaneous
>> stuff, but it looks quite similar):
>> - ID of an image has 7 or 8 digits
>> - /shared/private/: /(first 3 digits of ID)/(next 3 digits of ID)/$ID.jpg
>> - /shared/public/: /(first 3 digits of ID)/(next 3 digits of
>> ID)/$ID/$misc_formats.jpg
>>
>> That's why we have that many (sub-)directories. Files are only stored
>> in the lowest directory hierarchy. I hope i could make our structure
>> at least a bit more transparent.
>>
>> i hope there's something we can do to raise performance a bit. thx in
>> advance :-)
>>
>>
>> 2018-07-24 10:40 GMT+02:00 Pranith Kumar Karampuri :
>> >
>> >
>> > On Mon, Jul 23, 2018 at 4:16 PM, Hu Bert  wrote:
>> >>
>> >> Well, over the weekend about 200GB were copied, so now there are
>> >> ~400GB copied to the brick. That's far beyond a speed of 10GB per
>> >> hour. If I copied the 1.6 TB directly, that would be done within max 2
>> >> days. But with the self heal this will take at least 20 days minimum.
>> >>
>> >> Why is the performance that bad? No chance of speeding this up?
>> >
>> >
>> > What kind of data do you have?
>> > How many directories in the filesystem?
>> > On average how many files per directory?
>> > What is the depth of your directory hierarchy on average?
>> > What is average filesize?
>> >
>> > Based on this data we can 

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Pranith Kumar Karampuri]

Thanks a lot for detailed write-up, this helps find the bottlenecks easily.
On a high level, to handle this directory hierarchy i.e. lots of
directories with files, we need to improve healing
algorithms. Based on the data you provided, we need to make the following
enhancements:

1) At the moment directories are healed one at a time, but files can be
healed upto 64 in parallel per replica subvolume.
So if you have nX2 or nX3 distributed subvolumes, it can heal 64n number of
files in parallel.

I raised https://github.com/gluster/glusterfs/issues/477 to track this. In
the mean-while you can use the following work-around:
a) Increase background heals on the mount:
gluster volume set  cluster.background-self-heal-count 256
gluster volume set  cluster.cluster.heal-wait-queue-length 1
find  -type d | xargs stat

one 'find' will trigger 10256 directories. So you may have to do this
periodically until all directories are healed.

2) Self-heal heals a file 128KB at a time(data-self-heal-window-size). I
think for your environment bumping upto MBs is better. Say 2MB i.e.
16*128KB?

Command to do that is:
gluster volume set  cluster.data-self-heal-window-size 16


On Thu, Jul 26, 2018 at 10:40 AM, Hu Bert  wrote:

> Hi Pranith,
>
> Sry, it took a while to count the directories. I'll try to answer your
> questions as good as possible.
>
> > What kind of data do you have?
> > How many directories in the filesystem?
> > On average how many files per directory?
> > What is the depth of your directory hierarchy on average?
> > What is average filesize?
>
> We have mostly images (more than 95% of disk usage, 90% of file
> count), some text files (like css, jsp, gpx etc.) and some binaries.
>
> There are about 190.000 directories in the file system; maybe there
> are some more because we're hit by bug 1512371 (parallel-readdir =
> TRUE prevents directories listing). But the number of directories
> could/will rise in the future (maybe millions).
>
> files per directory: ranges from 0 to 100, on average it should be 20
> files per directory (well, at least in the deepest dirs, see
> explanation below).
>
> Average filesize: ranges from a few hundred bytes up to 30 MB, on
> average it should be 2-3 MB.
>
> Directory hierarchy: maximum depth as seen from within the volume is
> 6, the average should be 3.
>
> volume name: shared
> mount point on clients: /data/repository/shared/
> below /shared/ there are 2 directories:
> - public/: mainly calculated images (file sizes from a few KB up to
> max 1 MB) and some resouces (small PNGs with a size of a few hundred
> bytes).
> - private/: mainly source images; file sizes from 50 KB up to 30MB
>
> We migrated from a NFS server (SPOF) to glusterfs and simply copied
> our files. The images (which have an ID) are stored in the deepest
> directories of the dir tree. I'll better explain it :-)
>
> directory structure for the images (i'll omit some other miscellaneous
> stuff, but it looks quite similar):
> - ID of an image has 7 or 8 digits
> - /shared/private/: /(first 3 digits of ID)/(next 3 digits of ID)/$ID.jpg
> - /shared/public/: /(first 3 digits of ID)/(next 3 digits of
> ID)/$ID/$misc_formats.jpg
>
> That's why we have that many (sub-)directories. Files are only stored
> in the lowest directory hierarchy. I hope i could make our structure
> at least a bit more transparent.
>
> i hope there's something we can do to raise performance a bit. thx in
> advance :-)
>
>
> 2018-07-24 10:40 GMT+02:00 Pranith Kumar Karampuri :
> >
> >
> > On Mon, Jul 23, 2018 at 4:16 PM, Hu Bert  wrote:
> >>
> >> Well, over the weekend about 200GB were copied, so now there are
> >> ~400GB copied to the brick. That's far beyond a speed of 10GB per
> >> hour. If I copied the 1.6 TB directly, that would be done within max 2
> >> days. But with the self heal this will take at least 20 days minimum.
> >>
> >> Why is the performance that bad? No chance of speeding this up?
> >
> >
> > What kind of data do you have?
> > How many directories in the filesystem?
> > On average how many files per directory?
> > What is the depth of your directory hierarchy on average?
> > What is average filesize?
> >
> > Based on this data we can see if anything can be improved. Or if there
> are
> > some
> > enhancements that need to be implemented in gluster to address this kind
> of
> > data layout
> >>
> >>
> >> 2018-07-20 9:41 GMT+02:00 Hu Bert :
> >> > hmm... no one any idea?
> >> >
> >> > Additional question: the hdd on server gluster12 was changed, so far
> >> > ~220 GB were copied. On the other 2 servers i see a lot of entries in
> >> > glustershd.log, about 312.000 respectively 336.000 entries there
> >> > yesterday, most of them (current log output) looking like this:
> >> >
> >> > [2018-07-20 07:30:49.757595] I [MSGID: 108026]
> >> > [afr-self-heal-common.c:1724:afr_log_selfheal] 0-shared-replicate-3:
> >> > Completed data selfheal on 0d863a62-0dd8-401c-b699-2b642d9fd2b6.
> >> > sources=0 [2]  sinks=1
> >> > [2018-07-20 

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Hu Bert]

Hi Pranith,

Sry, it took a while to count the directories. I'll try to answer your
questions as good as possible.

> What kind of data do you have?
> How many directories in the filesystem?
> On average how many files per directory?
> What is the depth of your directory hierarchy on average?
> What is average filesize?

We have mostly images (more than 95% of disk usage, 90% of file
count), some text files (like css, jsp, gpx etc.) and some binaries.

There are about 190.000 directories in the file system; maybe there
are some more because we're hit by bug 1512371 (parallel-readdir =
TRUE prevents directories listing). But the number of directories
could/will rise in the future (maybe millions).

files per directory: ranges from 0 to 100, on average it should be 20
files per directory (well, at least in the deepest dirs, see
explanation below).

Average filesize: ranges from a few hundred bytes up to 30 MB, on
average it should be 2-3 MB.

Directory hierarchy: maximum depth as seen from within the volume is
6, the average should be 3.

volume name: shared
mount point on clients: /data/repository/shared/
below /shared/ there are 2 directories:
- public/: mainly calculated images (file sizes from a few KB up to
max 1 MB) and some resouces (small PNGs with a size of a few hundred
bytes).
- private/: mainly source images; file sizes from 50 KB up to 30MB

We migrated from a NFS server (SPOF) to glusterfs and simply copied
our files. The images (which have an ID) are stored in the deepest
directories of the dir tree. I'll better explain it :-)

directory structure for the images (i'll omit some other miscellaneous
stuff, but it looks quite similar):
- ID of an image has 7 or 8 digits
- /shared/private/: /(first 3 digits of ID)/(next 3 digits of ID)/$ID.jpg
- /shared/public/: /(first 3 digits of ID)/(next 3 digits of
ID)/$ID/$misc_formats.jpg

That's why we have that many (sub-)directories. Files are only stored
in the lowest directory hierarchy. I hope i could make our structure
at least a bit more transparent.

i hope there's something we can do to raise performance a bit. thx in
advance :-)


2018-07-24 10:40 GMT+02:00 Pranith Kumar Karampuri :
>
>
> On Mon, Jul 23, 2018 at 4:16 PM, Hu Bert  wrote:
>>
>> Well, over the weekend about 200GB were copied, so now there are
>> ~400GB copied to the brick. That's far beyond a speed of 10GB per
>> hour. If I copied the 1.6 TB directly, that would be done within max 2
>> days. But with the self heal this will take at least 20 days minimum.
>>
>> Why is the performance that bad? No chance of speeding this up?
>
>
> What kind of data do you have?
> How many directories in the filesystem?
> On average how many files per directory?
> What is the depth of your directory hierarchy on average?
> What is average filesize?
>
> Based on this data we can see if anything can be improved. Or if there are
> some
> enhancements that need to be implemented in gluster to address this kind of
> data layout
>>
>>
>> 2018-07-20 9:41 GMT+02:00 Hu Bert :
>> > hmm... no one any idea?
>> >
>> > Additional question: the hdd on server gluster12 was changed, so far
>> > ~220 GB were copied. On the other 2 servers i see a lot of entries in
>> > glustershd.log, about 312.000 respectively 336.000 entries there
>> > yesterday, most of them (current log output) looking like this:
>> >
>> > [2018-07-20 07:30:49.757595] I [MSGID: 108026]
>> > [afr-self-heal-common.c:1724:afr_log_selfheal] 0-shared-replicate-3:
>> > Completed data selfheal on 0d863a62-0dd8-401c-b699-2b642d9fd2b6.
>> > sources=0 [2]  sinks=1
>> > [2018-07-20 07:30:49.992398] I [MSGID: 108026]
>> > [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do]
>> > 0-shared-replicate-3: performing metadata selfheal on
>> > 0d863a62-0dd8-401c-b699-2b642d9fd2b6
>> > [2018-07-20 07:30:50.243551] I [MSGID: 108026]
>> > [afr-self-heal-common.c:1724:afr_log_selfheal] 0-shared-replicate-3:
>> > Completed metadata selfheal on 0d863a62-0dd8-401c-b699-2b642d9fd2b6.
>> > sources=0 [2]  sinks=1
>> >
>> > or like this:
>> >
>> > [2018-07-20 07:38:41.726943] I [MSGID: 108026]
>> > [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do]
>> > 0-shared-replicate-3: performing metadata selfheal on
>> > 9276097a-cdac-4d12-9dc6-04b1ea4458ba
>> > [2018-07-20 07:38:41.855737] I [MSGID: 108026]
>> > [afr-self-heal-common.c:1724:afr_log_selfheal] 0-shared-replicate-3:
>> > Completed metadata selfheal on 9276097a-cdac-4d12-9dc6-04b1ea4458ba.
>> > sources=[0] 2  sinks=1
>> > [2018-07-20 07:38:44.755800] I [MSGID: 108026]
>> > [afr-self-heal-entry.c:887:afr_selfheal_entry_do]
>> > 0-shared-replicate-3: performing entry selfheal on
>> > 9276097a-cdac-4d12-9dc6-04b1ea4458ba
>> >
>> > is this behaviour normal? I'd expect these messages on the server with
>> > the failed brick, not on the other ones.
>> >
>> > 2018-07-19 8:31 GMT+02:00 Hu Bert :
>> >> Hi there,
>> >>
>> >> sent this mail yesterday, but somehow it didn't work? Wasn't archived,
>> >> so please be indulgent 

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Pranith Kumar Karampuri]

On Mon, Jul 23, 2018 at 4:16 PM, Hu Bert  wrote:

> Well, over the weekend about 200GB were copied, so now there are
> ~400GB copied to the brick. That's far beyond a speed of 10GB per
> hour. If I copied the 1.6 TB directly, that would be done within max 2
> days. But with the self heal this will take at least 20 days minimum.
>
> Why is the performance that bad? No chance of speeding this up?
>

What kind of data do you have?
How many directories in the filesystem?
On average how many files per directory?
What is the depth of your directory hierarchy on average?
What is average filesize?

Based on this data we can see if anything can be improved. Or if there are
some
enhancements that need to be implemented in gluster to address this kind of
data layout

>
> 2018-07-20 9:41 GMT+02:00 Hu Bert :
> > hmm... no one any idea?
> >
> > Additional question: the hdd on server gluster12 was changed, so far
> > ~220 GB were copied. On the other 2 servers i see a lot of entries in
> > glustershd.log, about 312.000 respectively 336.000 entries there
> > yesterday, most of them (current log output) looking like this:
> >
> > [2018-07-20 07:30:49.757595] I [MSGID: 108026]
> > [afr-self-heal-common.c:1724:afr_log_selfheal] 0-shared-replicate-3:
> > Completed data selfheal on 0d863a62-0dd8-401c-b699-2b642d9fd2b6.
> > sources=0 [2]  sinks=1
> > [2018-07-20 07:30:49.992398] I [MSGID: 108026]
> > [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do]
> > 0-shared-replicate-3: performing metadata selfheal on
> > 0d863a62-0dd8-401c-b699-2b642d9fd2b6
> > [2018-07-20 07:30:50.243551] I [MSGID: 108026]
> > [afr-self-heal-common.c:1724:afr_log_selfheal] 0-shared-replicate-3:
> > Completed metadata selfheal on 0d863a62-0dd8-401c-b699-2b642d9fd2b6.
> > sources=0 [2]  sinks=1
> >
> > or like this:
> >
> > [2018-07-20 07:38:41.726943] I [MSGID: 108026]
> > [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do]
> > 0-shared-replicate-3: performing metadata selfheal on
> > 9276097a-cdac-4d12-9dc6-04b1ea4458ba
> > [2018-07-20 07:38:41.855737] I [MSGID: 108026]
> > [afr-self-heal-common.c:1724:afr_log_selfheal] 0-shared-replicate-3:
> > Completed metadata selfheal on 9276097a-cdac-4d12-9dc6-04b1ea4458ba.
> > sources=[0] 2  sinks=1
> > [2018-07-20 07:38:44.755800] I [MSGID: 108026]
> > [afr-self-heal-entry.c:887:afr_selfheal_entry_do]
> > 0-shared-replicate-3: performing entry selfheal on
> > 9276097a-cdac-4d12-9dc6-04b1ea4458ba
> >
> > is this behaviour normal? I'd expect these messages on the server with
> > the failed brick, not on the other ones.
> >
> > 2018-07-19 8:31 GMT+02:00 Hu Bert :
> >> Hi there,
> >>
> >> sent this mail yesterday, but somehow it didn't work? Wasn't archived,
> >> so please be indulgent it you receive this mail again :-)
> >>
> >> We are currently running a replicate setup and are experiencing a
> >> quite poor performance. It got even worse when within a couple of
> >> weeks 2 bricks (disks) crashed. Maybe some general information of our
> >> setup:
> >>
> >> 3 Dell PowerEdge R530 (Xeon E5-1650 v3 Hexa-Core, 64 GB DDR4, OS on
> >> separate disks); each server has 4 10TB disks -> each is a brick;
> >> replica 3 setup (see gluster volume status below). Debian stretch,
> >> kernel 4.9.0, gluster version 3.12.12. Servers and clients are
> >> connected via 10 GBit ethernet.
> >>
> >> About a month ago and 2 days ago a disk died (on different servers);
> >> disk were replaced, were brought back into the volume and full self
> >> heal started. But the speed for this is quite... disappointing. Each
> >> brick has ~1.6TB of data on it (mostly the infamous small files). The
> >> full heal i started yesterday copied only ~50GB within 24 hours (48
> >> hours: about 100GB) - with
> >> this rate it would take weeks until the self heal finishes.
> >>
> >> After the first heal (started on gluster13 about a month ago, took
> >> about 3 weeks) finished we had a terrible performance; CPU on one or
> >> two of the nodes (gluster11, gluster12) was up to 1200%, consumed by
> >> the brick process of the former crashed brick (bricksdd1),
> >> interestingly not on the server with the failed this, but on the other
> >> 2 ones...
> >>
> >> Well... am i doing something wrong? Some options wrongly configured?
> >> Terrible setup? Anyone got an idea? Any additional information needed?
> >>
> >>
> >> Thx in advance :-)
> >>
> >> gluster volume status
> >>
> >> Volume Name: shared
> >> Type: Distributed-Replicate
> >> Volume ID: e879d208-1d8c-4089-85f3-ef1b3aa45d36
> >> Status: Started
> >> Snapshot Count: 0
> >> Number of Bricks: 4 x 3 = 12
> >> Transport-type: tcp
> >> Bricks:
> >> Brick1: gluster11:/gluster/bricksda1/shared
> >> Brick2: gluster12:/gluster/bricksda1/shared
> >> Brick3: gluster13:/gluster/bricksda1/shared
> >> Brick4: gluster11:/gluster/bricksdb1/shared
> >> Brick5: gluster12:/gluster/bricksdb1/shared
> >> Brick6: gluster13:/gluster/bricksdb1/shared
> >> Brick7: gluster11:/gluster/bricksdc1/shared
> >> Brick8: 

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Hu Bert]

Well, over the weekend about 200GB were copied, so now there are
~400GB copied to the brick. That's far beyond a speed of 10GB per
hour. If I copied the 1.6 TB directly, that would be done within max 2
days. But with the self heal this will take at least 20 days minimum.

Why is the performance that bad? No chance of speeding this up?

2018-07-20 9:41 GMT+02:00 Hu Bert :
> hmm... no one any idea?
>
> Additional question: the hdd on server gluster12 was changed, so far
> ~220 GB were copied. On the other 2 servers i see a lot of entries in
> glustershd.log, about 312.000 respectively 336.000 entries there
> yesterday, most of them (current log output) looking like this:
>
> [2018-07-20 07:30:49.757595] I [MSGID: 108026]
> [afr-self-heal-common.c:1724:afr_log_selfheal] 0-shared-replicate-3:
> Completed data selfheal on 0d863a62-0dd8-401c-b699-2b642d9fd2b6.
> sources=0 [2]  sinks=1
> [2018-07-20 07:30:49.992398] I [MSGID: 108026]
> [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do]
> 0-shared-replicate-3: performing metadata selfheal on
> 0d863a62-0dd8-401c-b699-2b642d9fd2b6
> [2018-07-20 07:30:50.243551] I [MSGID: 108026]
> [afr-self-heal-common.c:1724:afr_log_selfheal] 0-shared-replicate-3:
> Completed metadata selfheal on 0d863a62-0dd8-401c-b699-2b642d9fd2b6.
> sources=0 [2]  sinks=1
>
> or like this:
>
> [2018-07-20 07:38:41.726943] I [MSGID: 108026]
> [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do]
> 0-shared-replicate-3: performing metadata selfheal on
> 9276097a-cdac-4d12-9dc6-04b1ea4458ba
> [2018-07-20 07:38:41.855737] I [MSGID: 108026]
> [afr-self-heal-common.c:1724:afr_log_selfheal] 0-shared-replicate-3:
> Completed metadata selfheal on 9276097a-cdac-4d12-9dc6-04b1ea4458ba.
> sources=[0] 2  sinks=1
> [2018-07-20 07:38:44.755800] I [MSGID: 108026]
> [afr-self-heal-entry.c:887:afr_selfheal_entry_do]
> 0-shared-replicate-3: performing entry selfheal on
> 9276097a-cdac-4d12-9dc6-04b1ea4458ba
>
> is this behaviour normal? I'd expect these messages on the server with
> the failed brick, not on the other ones.
>
> 2018-07-19 8:31 GMT+02:00 Hu Bert :
>> Hi there,
>>
>> sent this mail yesterday, but somehow it didn't work? Wasn't archived,
>> so please be indulgent it you receive this mail again :-)
>>
>> We are currently running a replicate setup and are experiencing a
>> quite poor performance. It got even worse when within a couple of
>> weeks 2 bricks (disks) crashed. Maybe some general information of our
>> setup:
>>
>> 3 Dell PowerEdge R530 (Xeon E5-1650 v3 Hexa-Core, 64 GB DDR4, OS on
>> separate disks); each server has 4 10TB disks -> each is a brick;
>> replica 3 setup (see gluster volume status below). Debian stretch,
>> kernel 4.9.0, gluster version 3.12.12. Servers and clients are
>> connected via 10 GBit ethernet.
>>
>> About a month ago and 2 days ago a disk died (on different servers);
>> disk were replaced, were brought back into the volume and full self
>> heal started. But the speed for this is quite... disappointing. Each
>> brick has ~1.6TB of data on it (mostly the infamous small files). The
>> full heal i started yesterday copied only ~50GB within 24 hours (48
>> hours: about 100GB) - with
>> this rate it would take weeks until the self heal finishes.
>>
>> After the first heal (started on gluster13 about a month ago, took
>> about 3 weeks) finished we had a terrible performance; CPU on one or
>> two of the nodes (gluster11, gluster12) was up to 1200%, consumed by
>> the brick process of the former crashed brick (bricksdd1),
>> interestingly not on the server with the failed this, but on the other
>> 2 ones...
>>
>> Well... am i doing something wrong? Some options wrongly configured?
>> Terrible setup? Anyone got an idea? Any additional information needed?
>>
>>
>> Thx in advance :-)
>>
>> gluster volume status
>>
>> Volume Name: shared
>> Type: Distributed-Replicate
>> Volume ID: e879d208-1d8c-4089-85f3-ef1b3aa45d36
>> Status: Started
>> Snapshot Count: 0
>> Number of Bricks: 4 x 3 = 12
>> Transport-type: tcp
>> Bricks:
>> Brick1: gluster11:/gluster/bricksda1/shared
>> Brick2: gluster12:/gluster/bricksda1/shared
>> Brick3: gluster13:/gluster/bricksda1/shared
>> Brick4: gluster11:/gluster/bricksdb1/shared
>> Brick5: gluster12:/gluster/bricksdb1/shared
>> Brick6: gluster13:/gluster/bricksdb1/shared
>> Brick7: gluster11:/gluster/bricksdc1/shared
>> Brick8: gluster12:/gluster/bricksdc1/shared
>> Brick9: gluster13:/gluster/bricksdc1/shared
>> Brick10: gluster11:/gluster/bricksdd1/shared
>> Brick11: gluster12:/gluster/bricksdd1_new/shared
>> Brick12: gluster13:/gluster/bricksdd1_new/shared
>> Options Reconfigured:
>> cluster.shd-max-threads: 4
>> performance.md-cache-timeout: 60
>> cluster.lookup-optimize: on
>> cluster.readdir-optimize: on
>> performance.cache-refresh-timeout: 4
>> performance.parallel-readdir: on
>> server.event-threads: 8
>> client.event-threads: 8
>> performance.cache-max-file-size: 128MB
>> performance.write-behind-window-size: 16MB
>> 

Re: [Gluster-users] Gluter 3.12.12: performance during heal and in general

[Hu Bert]

hmm... no one any idea?

Additional question: the hdd on server gluster12 was changed, so far
~220 GB were copied. On the other 2 servers i see a lot of entries in
glustershd.log, about 312.000 respectively 336.000 entries there
yesterday, most of them (current log output) looking like this:

[2018-07-20 07:30:49.757595] I [MSGID: 108026]
[afr-self-heal-common.c:1724:afr_log_selfheal] 0-shared-replicate-3:
Completed data selfheal on 0d863a62-0dd8-401c-b699-2b642d9fd2b6.
sources=0 [2]  sinks=1
[2018-07-20 07:30:49.992398] I [MSGID: 108026]
[afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do]
0-shared-replicate-3: performing metadata selfheal on
0d863a62-0dd8-401c-b699-2b642d9fd2b6
[2018-07-20 07:30:50.243551] I [MSGID: 108026]
[afr-self-heal-common.c:1724:afr_log_selfheal] 0-shared-replicate-3:
Completed metadata selfheal on 0d863a62-0dd8-401c-b699-2b642d9fd2b6.
sources=0 [2]  sinks=1

or like this:

[2018-07-20 07:38:41.726943] I [MSGID: 108026]
[afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do]
0-shared-replicate-3: performing metadata selfheal on
9276097a-cdac-4d12-9dc6-04b1ea4458ba
[2018-07-20 07:38:41.855737] I [MSGID: 108026]
[afr-self-heal-common.c:1724:afr_log_selfheal] 0-shared-replicate-3:
Completed metadata selfheal on 9276097a-cdac-4d12-9dc6-04b1ea4458ba.
sources=[0] 2  sinks=1
[2018-07-20 07:38:44.755800] I [MSGID: 108026]
[afr-self-heal-entry.c:887:afr_selfheal_entry_do]
0-shared-replicate-3: performing entry selfheal on
9276097a-cdac-4d12-9dc6-04b1ea4458ba

is this behaviour normal? I'd expect these messages on the server with
the failed brick, not on the other ones.

2018-07-19 8:31 GMT+02:00 Hu Bert :
> Hi there,
>
> sent this mail yesterday, but somehow it didn't work? Wasn't archived,
> so please be indulgent it you receive this mail again :-)
>
> We are currently running a replicate setup and are experiencing a
> quite poor performance. It got even worse when within a couple of
> weeks 2 bricks (disks) crashed. Maybe some general information of our
> setup:
>
> 3 Dell PowerEdge R530 (Xeon E5-1650 v3 Hexa-Core, 64 GB DDR4, OS on
> separate disks); each server has 4 10TB disks -> each is a brick;
> replica 3 setup (see gluster volume status below). Debian stretch,
> kernel 4.9.0, gluster version 3.12.12. Servers and clients are
> connected via 10 GBit ethernet.
>
> About a month ago and 2 days ago a disk died (on different servers);
> disk were replaced, were brought back into the volume and full self
> heal started. But the speed for this is quite... disappointing. Each
> brick has ~1.6TB of data on it (mostly the infamous small files). The
> full heal i started yesterday copied only ~50GB within 24 hours (48
> hours: about 100GB) - with
> this rate it would take weeks until the self heal finishes.
>
> After the first heal (started on gluster13 about a month ago, took
> about 3 weeks) finished we had a terrible performance; CPU on one or
> two of the nodes (gluster11, gluster12) was up to 1200%, consumed by
> the brick process of the former crashed brick (bricksdd1),
> interestingly not on the server with the failed this, but on the other
> 2 ones...
>
> Well... am i doing something wrong? Some options wrongly configured?
> Terrible setup? Anyone got an idea? Any additional information needed?
>
>
> Thx in advance :-)
>
> gluster volume status
>
> Volume Name: shared
> Type: Distributed-Replicate
> Volume ID: e879d208-1d8c-4089-85f3-ef1b3aa45d36
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 4 x 3 = 12
> Transport-type: tcp
> Bricks:
> Brick1: gluster11:/gluster/bricksda1/shared
> Brick2: gluster12:/gluster/bricksda1/shared
> Brick3: gluster13:/gluster/bricksda1/shared
> Brick4: gluster11:/gluster/bricksdb1/shared
> Brick5: gluster12:/gluster/bricksdb1/shared
> Brick6: gluster13:/gluster/bricksdb1/shared
> Brick7: gluster11:/gluster/bricksdc1/shared
> Brick8: gluster12:/gluster/bricksdc1/shared
> Brick9: gluster13:/gluster/bricksdc1/shared
> Brick10: gluster11:/gluster/bricksdd1/shared
> Brick11: gluster12:/gluster/bricksdd1_new/shared
> Brick12: gluster13:/gluster/bricksdd1_new/shared
> Options Reconfigured:
> cluster.shd-max-threads: 4
> performance.md-cache-timeout: 60
> cluster.lookup-optimize: on
> cluster.readdir-optimize: on
> performance.cache-refresh-timeout: 4
> performance.parallel-readdir: on
> server.event-threads: 8
> client.event-threads: 8
> performance.cache-max-file-size: 128MB
> performance.write-behind-window-size: 16MB
> performance.io-thread-count: 64
> cluster.min-free-disk: 1%
> performance.cache-size: 24GB
> nfs.disable: on
> transport.address-family: inet
> performance.high-prio-threads: 32
> performance.normal-prio-threads: 32
> performance.low-prio-threads: 32
> performance.least-prio-threads: 8
> performance.io-cache: on
> server.allow-insecure: on
> performance.strict-o-direct: off
> transport.listen-backlog: 100
> server.outstanding-rpc-limit: 128
___
Gluster-users mailing list