Re: [Gluster-users] Reliability issues with Gluster 3.10 and shard

2017-05-15 Thread Nithya Balachandran 
On 15 May 2017 at 11:01, Benjamin Kingston  wrote:

> I resolved this with the following settings, particularly disabling
> features.ctr-enabled
>

That's odd. CTR should be enabled for tiered volumes. Was it enabled by
default?



>
> olume Name: storage2
> Type: Distributed-Replicate
> Volume ID: adaabca5-25ed-4e7f-ae86-2f20fc0143a8
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 3 x (2 + 1) = 9
> Transport-type: tcp
> Bricks:
> Brick1: fd00:0:0:3::6:/mnt/gluster/storage/brick0/glusterfs2
> Brick2: fd00:0:0:3::8:/mnt/gluster/storage/brick0/glusterfs2
> Brick3: fd00:0:0:3::10:/mnt/gluster/storage/brick0/glusterfs (arbiter)
> Brick4: fd00:0:0:3::6:/mnt/gluster/storage/brick1/glusterfs2
> Brick5: fd00:0:0:3::8:/mnt/gluster/storage/brick1/glusterfs2
> Brick6: fd00:0:0:3::10:/mnt/gluster/storage/brick1/glusterfs (arbiter)
> Brick7: fd00:0:0:3::6:/mnt/gluster/storage/brick2/glusterfs2
> Brick8: fd00:0:0:3::8:/mnt/gluster/storage/brick2/glusterfs2
> Brick9: fd00:0:0:3::10:/mnt/gluster/storage/brick2/glusterfs (arbiter)
> Options Reconfigured:
> performance.write-behind-window-size: 4MB
> performance.cache-invalidation: on
> transport.keepalive: on
> performance.write-behind: on
> performance.read-ahead: on
> performance.io-cache: on
> performance.stat-prefetch: on
> performance.open-behind: on
> cluster.use-compound-fops: on
> performance.cache-ima-xattrs: on
> features.cache-invalidation: on
> client.event-threads: 4
> cluster.data-self-heal-algorithm: full
> performance.client-io-threads: on
> server.event-threads: 4
> performance.quick-read: on
> features.scrub: Active
> features.bitrot: on
> features.shard: on
> transport.address-family: inet6
> nfs.disable: on
> server.allow-insecure: on
> user.cifs: off
> cluster.quorum-type: auto
> cluster.server-quorum-type: server
> cluster.tier-compact: on
> diagnostics.brick-log-level: WARNING
> diagnostics.client-log-level: WARNING
> cluster.self-heal-daemon: enable
> performance.cache-samba-metadata: on
> cluster.brick-multiplex: off
> cluster.enable-shared-storage: enable
> nfs-ganesha: enable
>
>
>
> -ben
>
> On Sat, May 13, 2017 at 12:20 PM, Benjamin Kingston 
> wrote:
>
>> Hers's some log entries from nfs-ganesha gfapi
>>
>> [2017-05-13 19:02:54.105936] E [MSGID: 133010]
>> [shard.c:1706:shard_common_lookup_shards_cbk] 0-storage2-shard: Lookup
>> on shard 11 failed. Base file gfid = 1494c083-a618-4eba-80a0-147e656dd9d0
>> [Input/output error]
>> [2017-05-13 19:02:54.106176] E [MSGID: 133010]
>> [shard.c:1706:shard_common_lookup_shards_cbk] 0-storage2-shard: Lookup
>> on shard 2 failed. Base file gfid = 1494c083-a618-4eba-80a0-147e656dd9d0
>> [Input/output error]
>> [2017-05-13 19:02:54.106288] E [MSGID: 133010]
>> [shard.c:1706:shard_common_lookup_shards_cbk] 0-storage2-shard: Lookup
>> on shard 1 failed. Base file gfid = 1494c083-a618-4eba-80a0-147e656dd9d0
>> [Input/output error]
>> [2017-05-13 19:02:54.384922] I [MSGID: 108026]
>> [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do]
>> 0-storage2-replicate-2: performing metadata selfheal on
>> fe651475-226e-42a3-be2d-751d4f58e383
>> [2017-05-13 19:02:54.385894] W [MSGID: 114031]
>> [client-rpc-fops.c:2258:client3_3_setattr_cbk] 0-storage2-client-8:
>> remote operation failed [Operation not permitted]
>> [2017-05-13 19:02:54.401187] I [MSGID: 108026]
>> [afr-self-heal-common.c:1255:afr_log_selfheal] 0-storage2-replicate-2:
>> Completed metadata selfheal on fe651475-226e-42a3-be2d-751d4f58e383.
>> sources=[0] 1  sinks=
>> [2017-05-13 19:02:57.830019] I [MSGID: 109066]
>> [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.par2.tmp
>> (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) =>
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.par2
>> (hash=storage2-readdir-ahead-0/cache=)
>>
>> [2017-05-13 19:08:22.014899] I [MSGID: 109066]
>> [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.srr.tmp
>> (hash=storage2-readdir-ahead-1/cache=storage2-readdir-ahead-1) =>
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.srr
>> (hash=storage2-readdir-ahead-1/cache=)
>> [2017-05-13 19:08:22.463840] I [MSGID: 109066]
>> [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r04.tmp
>> (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) =>
>> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r04
>> (hash=storage2-readdir-ahead-0/cache=)
>> [2017-05-13 19:08:22.769542] I [MSGID:

Re: [Gluster-users] Reliability issues with Gluster 3.10 and shard

2017-05-15 Thread Benjamin Kingston 
I resolved this with the following settings, particularly disabling
features.ctr-enabled

olume Name: storage2
Type: Distributed-Replicate
Volume ID: adaabca5-25ed-4e7f-ae86-2f20fc0143a8
Status: Started
Snapshot Count: 0
Number of Bricks: 3 x (2 + 1) = 9
Transport-type: tcp
Bricks:
Brick1: fd00:0:0:3::6:/mnt/gluster/storage/brick0/glusterfs2
Brick2: fd00:0:0:3::8:/mnt/gluster/storage/brick0/glusterfs2
Brick3: fd00:0:0:3::10:/mnt/gluster/storage/brick0/glusterfs (arbiter)
Brick4: fd00:0:0:3::6:/mnt/gluster/storage/brick1/glusterfs2
Brick5: fd00:0:0:3::8:/mnt/gluster/storage/brick1/glusterfs2
Brick6: fd00:0:0:3::10:/mnt/gluster/storage/brick1/glusterfs (arbiter)
Brick7: fd00:0:0:3::6:/mnt/gluster/storage/brick2/glusterfs2
Brick8: fd00:0:0:3::8:/mnt/gluster/storage/brick2/glusterfs2
Brick9: fd00:0:0:3::10:/mnt/gluster/storage/brick2/glusterfs (arbiter)
Options Reconfigured:
performance.write-behind-window-size: 4MB
performance.cache-invalidation: on
transport.keepalive: on
performance.write-behind: on
performance.read-ahead: on
performance.io-cache: on
performance.stat-prefetch: on
performance.open-behind: on
cluster.use-compound-fops: on
performance.cache-ima-xattrs: on
features.cache-invalidation: on
client.event-threads: 4
cluster.data-self-heal-algorithm: full
performance.client-io-threads: on
server.event-threads: 4
performance.quick-read: on
features.scrub: Active
features.bitrot: on
features.shard: on
transport.address-family: inet6
nfs.disable: on
server.allow-insecure: on
user.cifs: off
cluster.quorum-type: auto
cluster.server-quorum-type: server
cluster.tier-compact: on
diagnostics.brick-log-level: WARNING
diagnostics.client-log-level: WARNING
cluster.self-heal-daemon: enable
performance.cache-samba-metadata: on
cluster.brick-multiplex: off
cluster.enable-shared-storage: enable
nfs-ganesha: enable



-ben

On Sat, May 13, 2017 at 12:20 PM, Benjamin Kingston 
wrote:

> Hers's some log entries from nfs-ganesha gfapi
>
> [2017-05-13 19:02:54.105936] E [MSGID: 133010] 
> [shard.c:1706:shard_common_lookup_shards_cbk]
> 0-storage2-shard: Lookup on shard 11 failed. Base file gfid =
> 1494c083-a618-4eba-80a0-147e656dd9d0 [Input/output error]
> [2017-05-13 19:02:54.106176] E [MSGID: 133010] 
> [shard.c:1706:shard_common_lookup_shards_cbk]
> 0-storage2-shard: Lookup on shard 2 failed. Base file gfid =
> 1494c083-a618-4eba-80a0-147e656dd9d0 [Input/output error]
> [2017-05-13 19:02:54.106288] E [MSGID: 133010] 
> [shard.c:1706:shard_common_lookup_shards_cbk]
> 0-storage2-shard: Lookup on shard 1 failed. Base file gfid =
> 1494c083-a618-4eba-80a0-147e656dd9d0 [Input/output error]
> [2017-05-13 19:02:54.384922] I [MSGID: 108026]
> [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do]
> 0-storage2-replicate-2: performing metadata selfheal on
> fe651475-226e-42a3-be2d-751d4f58e383
> [2017-05-13 19:02:54.385894] W [MSGID: 114031] 
> [client-rpc-fops.c:2258:client3_3_setattr_cbk]
> 0-storage2-client-8: remote operation failed [Operation not permitted]
> [2017-05-13 19:02:54.401187] I [MSGID: 108026]
> [afr-self-heal-common.c:1255:afr_log_selfheal] 0-storage2-replicate-2:
> Completed metadata selfheal on fe651475-226e-42a3-be2d-751d4f58e383.
> sources=[0] 1  sinks=
> [2017-05-13 19:02:57.830019] I [MSGID: 109066]
> [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.par2.tmp
> (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) =>
> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.par2
> (hash=storage2-readdir-ahead-0/cache=)
>
> [2017-05-13 19:08:22.014899] I [MSGID: 109066]
> [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.srr.tmp
> (hash=storage2-readdir-ahead-1/cache=storage2-readdir-ahead-1) =>
> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.srr
> (hash=storage2-readdir-ahead-1/cache=)
> [2017-05-13 19:08:22.463840] I [MSGID: 109066]
> [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r04.tmp
> (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) =>
> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r04
> (hash=storage2-readdir-ahead-0/cache=)
> [2017-05-13 19:08:22.769542] I [MSGID: 109066]
> [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r01.tmp
> (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) =>
> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
>

Re: [Gluster-users] Reliability issues with Gluster 3.10 and shard

2017-05-15 Thread Krutika Dhananjay 
Shard translator is currently supported only for VM image store workload.

-Krutika

On Sun, May 14, 2017 at 12:50 AM, Benjamin Kingston 
wrote:

> Hers's some log entries from nfs-ganesha gfapi
>
> [2017-05-13 19:02:54.105936] E [MSGID: 133010] 
> [shard.c:1706:shard_common_lookup_shards_cbk]
> 0-storage2-shard: Lookup on shard 11 failed. Base file gfid =
> 1494c083-a618-4eba-80a0-147e656dd9d0 [Input/output error]
> [2017-05-13 19:02:54.106176] E [MSGID: 133010] 
> [shard.c:1706:shard_common_lookup_shards_cbk]
> 0-storage2-shard: Lookup on shard 2 failed. Base file gfid =
> 1494c083-a618-4eba-80a0-147e656dd9d0 [Input/output error]
> [2017-05-13 19:02:54.106288] E [MSGID: 133010] 
> [shard.c:1706:shard_common_lookup_shards_cbk]
> 0-storage2-shard: Lookup on shard 1 failed. Base file gfid =
> 1494c083-a618-4eba-80a0-147e656dd9d0 [Input/output error]
> [2017-05-13 19:02:54.384922] I [MSGID: 108026]
> [afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do]
> 0-storage2-replicate-2: performing metadata selfheal on
> fe651475-226e-42a3-be2d-751d4f58e383
> [2017-05-13 19:02:54.385894] W [MSGID: 114031] 
> [client-rpc-fops.c:2258:client3_3_setattr_cbk]
> 0-storage2-client-8: remote operation failed [Operation not permitted]
> [2017-05-13 19:02:54.401187] I [MSGID: 108026]
> [afr-self-heal-common.c:1255:afr_log_selfheal] 0-storage2-replicate-2:
> Completed metadata selfheal on fe651475-226e-42a3-be2d-751d4f58e383.
> sources=[0] 1  sinks=
> [2017-05-13 19:02:57.830019] I [MSGID: 109066]
> [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.par2.tmp
> (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) =>
> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.par2
> (hash=storage2-readdir-ahead-0/cache=)
>
> [2017-05-13 19:08:22.014899] I [MSGID: 109066]
> [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.srr.tmp
> (hash=storage2-readdir-ahead-1/cache=storage2-readdir-ahead-1) =>
> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.srr
> (hash=storage2-readdir-ahead-1/cache=)
> [2017-05-13 19:08:22.463840] I [MSGID: 109066]
> [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r04.tmp
> (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) =>
> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r04
> (hash=storage2-readdir-ahead-0/cache=)
> [2017-05-13 19:08:22.769542] I [MSGID: 109066]
> [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r01.tmp
> (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) =>
> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r01
> (hash=storage2-readdir-ahead-0/cache=)
> [2017-05-13 19:08:23.141069] I [MSGID: 109066]
> [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.nfo.tmp
> (hash=storage2-readdir-ahead-1/cache=storage2-readdir-ahead-1) =>
> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.nfo
> (hash=storage2-readdir-ahead-0/cache=)
> [2017-05-13 19:08:23.468554] I [MSGID: 109066]
> [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r00.tmp
> (hash=storage2-readdir-ahead-0/cache=storage2-readdir-ahead-0) =>
> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r00
> (hash=storage2-readdir-ahead-2/cache=)
> [2017-05-13 19:08:23.671753] I [MSGID: 109066]
> [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.sfv.tmp
> (hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) =>
> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.sfv
> (hash=storage2-readdir-ahead-2/cache=)
> [2017-05-13 19:08:23.812152] I [MSGID: 109066]
> [dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
> /content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.
> 720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r11.tmp
>

Re: [Gluster-users] Reliability issues with Gluster 3.10 and shard

2017-05-14 Thread Benjamin Kingston 
Hers's some log entries from nfs-ganesha gfapi

[2017-05-13 19:02:54.105936] E [MSGID: 133010]
[shard.c:1706:shard_common_lookup_shards_cbk] 0-storage2-shard: Lookup on
shard 11 failed. Base file gfid = 1494c083-a618-4eba-80a0-147e656dd9d0
[Input/output error]
[2017-05-13 19:02:54.106176] E [MSGID: 133010]
[shard.c:1706:shard_common_lookup_shards_cbk] 0-storage2-shard: Lookup on
shard 2 failed. Base file gfid = 1494c083-a618-4eba-80a0-147e656dd9d0
[Input/output error]
[2017-05-13 19:02:54.106288] E [MSGID: 133010]
[shard.c:1706:shard_common_lookup_shards_cbk] 0-storage2-shard: Lookup on
shard 1 failed. Base file gfid = 1494c083-a618-4eba-80a0-147e656dd9d0
[Input/output error]
[2017-05-13 19:02:54.384922] I [MSGID: 108026]
[afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do]
0-storage2-replicate-2: performing metadata selfheal on
fe651475-226e-42a3-be2d-751d4f58e383
[2017-05-13 19:02:54.385894] W [MSGID: 114031]
[client-rpc-fops.c:2258:client3_3_setattr_cbk] 0-storage2-client-8: remote
operation failed [Operation not permitted]
[2017-05-13 19:02:54.401187] I [MSGID: 108026]
[afr-self-heal-common.c:1255:afr_log_selfheal] 0-storage2-replicate-2:
Completed metadata selfheal on fe651475-226e-42a3-be2d-751d4f58e383.
sources=[0] 1  sinks=
[2017-05-13 19:02:57.830019] I [MSGID: 109066]
[dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
/content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.par2.tmp
(hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) =>
/content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.par2
(hash=storage2-readdir-ahead-0/cache=)

[2017-05-13 19:08:22.014899] I [MSGID: 109066]
[dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
/content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.srr.tmp
(hash=storage2-readdir-ahead-1/cache=storage2-readdir-ahead-1) =>
/content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.srr
(hash=storage2-readdir-ahead-1/cache=)
[2017-05-13 19:08:22.463840] I [MSGID: 109066]
[dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
/content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r04.tmp
(hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) =>
/content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r04
(hash=storage2-readdir-ahead-0/cache=)
[2017-05-13 19:08:22.769542] I [MSGID: 109066]
[dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
/content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r01.tmp
(hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) =>
/content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r01
(hash=storage2-readdir-ahead-0/cache=)
[2017-05-13 19:08:23.141069] I [MSGID: 109066]
[dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
/content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.nfo.tmp
(hash=storage2-readdir-ahead-1/cache=storage2-readdir-ahead-1) =>
/content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.nfo
(hash=storage2-readdir-ahead-0/cache=)
[2017-05-13 19:08:23.468554] I [MSGID: 109066]
[dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
/content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r00.tmp
(hash=storage2-readdir-ahead-0/cache=storage2-readdir-ahead-0) =>
/content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r00
(hash=storage2-readdir-ahead-2/cache=)
[2017-05-13 19:08:23.671753] I [MSGID: 109066]
[dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
/content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.sfv.tmp
(hash=storage2-readdir-ahead-2/cache=storage2-readdir-ahead-2) =>
/content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.sfv
(hash=storage2-readdir-ahead-2/cache=)
[2017-05-13 19:08:23.812152] I [MSGID: 109066]
[dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming
/content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r11.tmp
(hash=storage2-readdir-ahead-0/cache=storage2-readdir-ahead-0) =>
/content/Downloads/incomplete/usenet/Attack.on.Titan.S02E05.720p.WEB.x264-ANiURL.#27/aniurl-aot.s02e05.720p.web.r11
(hash=storage2-readdir-ahead-1/cache=)
[2017-05-13 19:08:24.244584] I [MSGID: 109066]
[dht-rename.c:1608:dht_rename] 0-storage2-dht: renaming

[Gluster-users] Reliability issues with Gluster 3.10 and shard

2017-05-13 Thread Benjamin Kingston 
Hello all,

I'm trying to take advantage of the shard xlator, however I've found it
causes a lot of issues that I hope is easily resolvable

1) large file operations work well (copy file from folder a to folder b
2) seek operations and list operations frequently fail (ls directory, read
bytes xyz at offset 235567)

Turning off the shard feature resolves this issue for new files created in
the volume. mounted using the gluster fuse mount

here's my volume settings, please let me know if there's some changes I can
make.

Option
Value
--
-
cluster.lookup-unhashed
on
cluster.lookup-optimize
on
cluster.min-free-disk
10%
cluster.min-free-inodes
5%
cluster.rebalance-stats
off
cluster.subvols-per-directory
(null)
cluster.readdir-optimize
off
cluster.rsync-hash-regex
(null)
cluster.extra-hash-regex
(null)
cluster.dht-xattr-name
trusted.glusterfs.dht
cluster.randomize-hash-range-by-gfid
off
cluster.rebal-throttle
normal
cluster.lock-migration
off
cluster.local-volume-name
(null)
cluster.weighted-rebalance
on
cluster.switch-pattern
(null)
cluster.entry-change-log
on
cluster.read-subvolume
(null)
cluster.read-subvolume-index
-1
cluster.read-hash-mode
1
cluster.background-self-heal-count
8
cluster.metadata-self-heal
on
cluster.data-self-heal
on
cluster.entry-self-heal
on
cluster.self-heal-daemon
on
cluster.heal-timeout
600
cluster.self-heal-window-size
1
cluster.data-change-log
on
cluster.metadata-change-log
on
cluster.data-self-heal-algorithm
diff
cluster.eager-lock
enable
disperse.eager-lock
on
cluster.quorum-type
auto
cluster.quorum-count
(null)
cluster.choose-local
on
cluster.self-heal-readdir-size
1KB
cluster.post-op-delay-secs
1
cluster.ensure-durability
on
cluster.consistent-metadata
no
cluster.heal-wait-queue-length
128
cluster.favorite-child-policy
none
cluster.stripe-block-size
128KB
cluster.stripe-coalesce
true
diagnostics.latency-measurement
off
diagnostics.dump-fd-stats
off
diagnostics.count-fop-hits
off
diagnostics.brick-log-level
INFO
diagnostics.client-log-level
INFO
diagnostics.brick-sys-log-level
CRITICAL
diagnostics.client-sys-log-level
CRITICAL
diagnostics.brick-logger
(null)
diagnostics.client-logger
(null)
diagnostics.brick-log-format
(null)
diagnostics.client-log-format
(null)
diagnostics.brick-log-buf-size
5
diagnostics.client-log-buf-size
5
diagnostics.brick-log-flush-timeout
120
diagnostics.client-log-flush-timeout
120
diagnostics.stats-dump-interval
0
diagnostics.fop-sample-interval
0
diagnostics.fop-sample-buf-size
65535
diagnostics.stats-dnscache-ttl-sec
86400
performance.cache-max-file-size
0
performance.cache-min-file-size
0
performance.cache-refresh-timeout
1
performance.cache-priority

performance.cache-size
1GB
performance.io-thread-count
64
performance.high-prio-threads
16
performance.normal-prio-threads
16
performance.low-prio-threads
32
performance.least-prio-threads
1
performance.enable-least-priority
on
performance.cache-size
1GB
performance.flush-behind
on
performance.nfs.flush-behind
on
performance.write-behind-window-size
2GB
performance.resync-failed-syncs-after-fsyncoff

performance.nfs.write-behind-window-size1MB

performance.strict-o-direct
off
performance.nfs.strict-o-direct
off
performance.strict-write-ordering
off
performance.nfs.strict-write-ordering
off
performance.lazy-open
yes
performance.read-after-open
no
performance.read-ahead-page-count
4
performance.md-cache-timeout
1
performance.cache-swift-metadata
true
performance.cache-samba-metadata
false
performance.cache-capability-xattrs
true
performance.cache-ima-xattrs
on
features.encryption
off
encryption.master-key
(null)
encryption.data-key-size
256
encryption.block-size
4096
network.frame-timeout
1800
network.ping-timeout
42
network.tcp-window-size
(null)
features.lock-heal
off
features.grace-timeout
10
network.remote-dio
disable
client.event-threads
3
network.ping-timeout
42
network.tcp-window-size
(null)
network.inode-lru-limit
9
auth.allow
*
auth.reject
(null)
transport.keepalive
on
server.allow-insecure
on
server.root-squash
off
server.anonuid
65534
server.anongid
65534
server.statedump-path
/var/run/gluster
server.outstanding-rpc-limit
64
features.lock-heal
off
features.grace-timeout
10
server.ssl
(null)
auth.ssl-allow
*
server.manage-gids
off
server.dynamic-auth
on
client.send-gids
on
server.gid-timeout
300
server.own-thread
(null)
server.event-threads
3
ssl.own-cert
(null)
ssl.private-key
(null)
ssl.ca-list
(null)
ssl.crl-path
(null)
ssl.certificate-depth
(null)
ssl.cipher-list
(null)
ssl.dh-param
(null)
ssl.ec-curve
(null)
transport.address-family
inet6
performance.write-behind
on
performance.read-ahead
off
performance.readdir-ahead
on
performance.io-cache
on
performance.quick-read
off
performance.open-behind
on
performance.stat-prefetch
on
performance.client-io-threads
on
performance.nfs.write-behind
on
performance.nfs.read-ahead
off
performance.nfs.io-cache
off
performance.nfs.quick-read
off
performance.nfs.stat-prefetch
off
performance.nfs.io-threads
off
performance.force-readdirp
true