[Gluster-users] Can't stop volume using gluster volume stop

[mabi]

Hello,

On one of my GlusterFS 3.12.7 3-way replica volume I can't stop it using the 
standard gluster volume stop command as you can see below:

$ sudo gluster volume stop myvolume
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) 
y
volume stop: myvolume: failed: geo-replication Unable to get the status of 
active geo-replication session for the volume 'myvolume'.
 Please check the log file for more info.

In the past I had geo-replication running on that volume but because it did not 
perform well with millions of files I decided to delete it. Somehow it looks 
like it has not been totally deleted or correctly deleted else the volume stop 
command above should have worked. Nevertheless I can't find any traces of the 
geo-replication still configured as you can see below withe a geo-replication 
status command:

$ sudo gluster volume geo-replication myvolume geo.domain.tld::myvolume-geo 
status detail
No active geo-replication sessions between myvolume and 
geo.domain.tld::myvolume-geo
​​
Any ideas how I can fix that?

Best regards,
Mabi
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] ETA for 3.10.12 (was "Planned for the 30th of Mar, 2018")

[Shyam Ranganathan]

Hi,

We postponed this and I did not announce this to the lists. The number
of bugs fixed against 3.10.12 is low, and I decided to move this to the
30th of Apr instead.

Is there a specific fix that you are looking for in the release?

Thanks,
Shyam

On 04/06/2018 11:47 AM, Marco Lorenzo Crociani wrote:
> Hi,
> are there any news for 3.10.12 release?
> 
> Regards,
> 
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] ETA for 3.10.12 (was "Planned for the 30th of Mar, 2018")

[Marco Lorenzo Crociani]

Hi,
are there any news for 3.10.12 release?

Regards,

--
Marco Crociani
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Sharding problem - multiple shard copies with mismatching gfids

[Ian Halliday]

Raghavendra,

Thanks! I'll get you this info within the next few days and will file a 
bug report at the same time.


For what its worth, we were able to reproduce the issue on a completely 
new cluster running 3.13. The IO pattern that most easily causes it to 
fail is a VM image format with XFS. Formatting VMS with Ext4 will create 
the additional shard files, but the GFIDs will usually match. I'm not 
sure if there are supposed to be 2 identical shard filenames, with one 
being empty, but they don't seem to cause VMs to pause or fail when the 
GFID matches.


Both of these clusters are pure SSD (one replica 3 arbiter 1, the other 
replica 3). I haven't seen any issues with our non-SSD clusters yet, but 
they aren't pushed as hard.


Ian

-- Original Message --
From: "Raghavendra Gowdappa" 
To: "Ian Halliday" 
Cc: "Krutika Dhananjay" ; "gluster-user" 
; "Nithya Balachandran" 

Sent: 4/5/2018 10:39:47 PM
Subject: Re: Re[2]: [Gluster-users] Sharding problem - multiple shard 
copies with mismatching gfids



Sorry for the delay, Ian :).

This looks to be a genuine issue which requires some effort in fixing 
it. Can you file a bug? I need following information attached to bug:


* Client and bricks logs. If you can reproduce the issue, please set 
diagnostics.client-log-level and diagnostics.brick-log-level to TRACE. 
If you cannot reproduce the issue or if you cannot accommodate such big 
logs, please set the log-level to DEBUG.
* If possible a simple reproducer. A simple script or steps are 
appreciated.
* strace of VM (to find out I/O pattern). If possible, dump of traffic 
between kernel and glusterfs. This can be captured by mounting 
glusterfs using --dump-fuse option.


Note that the logs you've posted here captures the scenario _after_ the 
shard file has gone into bad state. But I need information on what led 
to that situation. So, please start collecting this diagnostic 
information as early as you can.


regards,
Raghavendra

On Tue, Apr 3, 2018 at 7:52 AM, Ian Halliday  
wrote:

Raghavendra,

Sorry for the late follow up. I have some more data on the issue.

The issue tends to happen when the shards are created. The easiest 
time to reproduce this is during an initial VM disk format. This is a 
log from a test VM that was launched, and then partitioned and 
formatted with LVM / XFS:


[2018-04-03 02:05:00.838440] W [MSGID: 109048] 
[dht-common.c:9732:dht_rmdir_cached_lookup_cbk] 0-ovirt-350-zone1-dht: 
/489c6fb7-fe61-4407-8160-35c0aac40c85/images/_remove_me_9a0660e1-bd86-47ea-8e09-865c14f11f26/e2645bd1-a7f3-4cbd-9036-3d3cbc7204cd.meta 
found on cached subvol ovirt-350-zone1-replicate-5
[2018-04-03 02:07:57.967489] I [MSGID: 109070] 
[dht-common.c:2796:dht_lookup_linkfile_cbk] 0-ovirt-350-zone1-dht: 
Lookup of /.shard/927c6620-848b-4064-8c88-68a332b645c2.7 on 
ovirt-350-zone1-replicate-3 (following linkfile) failed ,gfid = 
---- [No such file or directory]
[2018-04-03 02:07:57.974815] I [MSGID: 109069] 
[dht-common.c:2095:dht_lookup_unlink_stale_linkto_cbk] 
0-ovirt-350-zone1-dht: Returned with op_ret 0 and op_errno 0 for 
/.shard/927c6620-848b-4064-8c88-68a332b645c2.3
[2018-04-03 02:07:57.979851] W [MSGID: 109009] 
[dht-common.c:2831:dht_lookup_linkfile_cbk] 0-ovirt-350-zone1-dht: 
/.shard/927c6620-848b-4064-8c88-68a332b645c2.3: gfid different on data 
file on ovirt-350-zone1-replicate-3, gfid local = 
----, gfid node = 
55f86aa0-e7a0-4075-b46b-a11f8bdbbceb
[2018-04-03 02:07:57.980716] W [MSGID: 109009] 
[dht-common.c:2570:dht_lookup_everywhere_cbk] 0-ovirt-350-zone1-dht: 
/.shard/927c6620-848b-4064-8c88-68a332b645c2.3: gfid differs on 
subvolume ovirt-350-zone1-replicate-3, gfid local = 
b1e3f299-32ff-497e-918b-090e957090f6, gfid node = 
55f86aa0-e7a0-4075-b46b-a11f8bdbbceb
[2018-04-03 02:07:57.980763] E [MSGID: 133010] 
[shard.c:1724:shard_common_lookup_shards_cbk] 0-ovirt-350-zone1-shard: 
Lookup on shard 3 failed. Base file gfid = 
927c6620-848b-4064-8c88-68a332b645c2 [Stale file handle]
[2018-04-03 02:07:57.983016] I [MSGID: 109069] 
[dht-common.c:2095:dht_lookup_unlink_stale_linkto_cbk] 
0-ovirt-350-zone1-dht: Returned with op_ret 0 and op_errno 0 for 
/.shard/927c6620-848b-4064-8c88-68a332b645c2.7
[2018-04-03 02:07:57.988761] W [MSGID: 109009] 
[dht-common.c:2570:dht_lookup_everywhere_cbk] 0-ovirt-350-zone1-dht: 
/.shard/927c6620-848b-4064-8c88-68a332b645c2.3: gfid differs on 
subvolume ovirt-350-zone1-replicate-3, gfid local = 
b1e3f299-32ff-497e-918b-090e957090f6, gfid node = 
55f86aa0-e7a0-4075-b46b-a11f8bdbbceb
[2018-04-03 02:07:57.988844] W [MSGID: 109009] 
[dht-common.c:2831:dht_lookup_linkfile_cbk] 0-ovirt-350-zone1-dht: 
/.shard/927c6620-848b-4064-8c88-68a332b645c2.7: gfid different on data 
file on ovirt-350-zone1-replicate-3, gfid local = 

Re: [Gluster-users] glusterd2 problem

[Dmitry Melekhov]

06.04.2018 14:46, Prashanth Pai пишет:

Hi Dmitry,

How many nodes does the cluster have ?

If the quorum is lost (majority of nodes are down), additional 
recovery steps are necessary to bring it back up:

https://github.com/gluster/glusterd2/wiki/Recovery



Hello!

just 2 nodes and looks like we had both down.

We tried to create test cluster with 3 nodes, to learn how to use 
glusterd2, but had this issue when we had only 2 nodes, i.e. after we 
added second node.

just because we found that
gluster
cli uses localhost by default, but we configured glusterd2 to ethernet 
interface address, this is another problem we hit, so we restarted both 
nodes I guess :-)


Glusterd2 state looks as pre-alpha for me, so we'll wait for at least beta.

Thank you!






On Wed, Apr 4, 2018 at 11:08 AM, Dmitry Melekhov > wrote:


Hello!

Installed packages from SIG on centos7 ,

at first start it works, but after restart- not:


 glusterd2 --config /etc/glusterd2/glusterd2.toml
DEBU[2018-04-04 09:28:16.945267] Starting GlusterD   
pid=221581 source="[main.go:55:main.main]" version=v4.0.0-0
INFO[2018-04-04 09:28:16.945824] loaded configuration from
file    file=/etc/glusterd2/glusterd2.toml
source="[config.go:163:main.in itConfig]"
DEBU[2018-04-04 09:28:16.946343] running with configuration   
cert-file= clientaddress="192.168.222.24:24007
"
config=/etc/glusterd2/glusterd2.toml defaultpeerport=24008
etcdcurls="http://192.168.222.24:2379
" etcdendpoints="[]"
etcdpurls="http://192.168.222.24:2380
" key-file=
localstatedir=/var/lib/glusterd2 logdir=/var/log/glusterd2
logfile=glusterd2.log loglevel=debug noembed=false
peeraddress="192.168.222.24:24008 "
rundir=/usr/var/run/glusterd2
source="[config.go:129:main.dumpConfigToLog]" statedump=true
version=false workdir=/etc/glusterd2
DEBU[2018-04-04 09:28:17.067244] loading templates
source="[template.go:64:volgen.LoadTemplates]" templatesdir=
DEBU[2018-04-04 09:28:17.068207] generated default templates
source="[template.go:82:volgen.LoadTemplates]"
templates="[brick.graph distreplicate.graph fuse.graph
distribute.graph replicate.graph disperse.graph]"
DEBU[2018-04-04 09:28:17.068361] loaded template
source="[template.go:102:volgen.LoadTemplate]" template=brick.graph
DEBU[2018-04-04 09:28:17.068450] loaded template
source="[template.go:102:volgen.LoadTemplate]"
template=distreplicate.graph
DEBU[2018-04-04 09:28:17.068539] loaded template
source="[template.go:102:volgen.LoadTemplate]" template=fuse.graph
DEBU[2018-04-04 09:28:17.068621] loaded template
source="[template.go:102:volgen.LoadTemplate]"
template=distribute.graph
DEBU[2018-04-04 09:28:17.068798] loaded template
source="[template.go:102:volgen.LoadTemplate]"
template=replicate.graph
DEBU[2018-04-04 09:28:17.068878] loaded template
source="[template.go:102:volgen.LoadTemplate]" template=disperse.graph
DEBU[2018-04-04 09:28:17.069316] saving updated store
config   source="[config.go:166:store.GetConfig]"
DEBU[2018-04-04 09:28:17.069745] starting embedded
store   cafile= certfile=
curls="http://192.168.222.24:2379 "
datadir=/var/lib/glusterd2/store
endpoints="http://192.168.222.24:2379
" keyfile=
logdir=/var/log/glusterd2/store
name=10641b8a-7135-4694-9b90-955827c8eba9
purls="http://192.168.222.24:2380 "
source="[embed.go:32:store.newEmbedStore]" trustedcafile=
2018-04-04 09:28:17.071331 I | etcdserver/api/v3rpc: grpc:
addrConn.resetTransport failed to create client transport:
connection error: desc = "transport: dial tcp 192.168.222.24:2379
: getsockopt: connection refused";
Reconnecting to {192.168.222.24:2379 
}
2018-04-04 09:28:18.071206 I | etcdserver/api/v3rpc: grpc:
addrConn.resetTransport failed to create client transport:
connection error: desc = "transport: dial tcp 192.168.222.24:2379
: getsockopt: connection refused";
Reconnecting to {192.168.222.24:2379 
}
2018-04-04 09:28:19.681081 I | etcdserver/api/v3rpc: grpc:
addrConn.resetTransport failed to create client transport:
connection error: desc = "transport: dial tcp 192.168.222.24:2379
: getsockopt: connection refused";
Reconnecting to {192.168.222.24:2379 
}
2018-04-04 09:28:22.070811 I | etcdserver/api/v3rpc: Failed to
dial 192.168.222.24:2379 : context

Re: [Gluster-users] glusterd2 problem

[Prashanth Pai]

Hi Dmitry,

How many nodes does the cluster have ?

If the quorum is lost (majority of nodes are down), additional recovery
steps are necessary to bring it back up:
https://github.com/gluster/glusterd2/wiki/Recovery


On Wed, Apr 4, 2018 at 11:08 AM, Dmitry Melekhov  wrote:

> Hello!
>
> Installed packages from SIG on centos7 ,
>
> at first start it works, but after restart- not:
>
>
>  glusterd2 --config /etc/glusterd2/glusterd2.toml
> DEBU[2018-04-04 09:28:16.945267] Starting GlusterD
> pid=221581 source="[main.go:55:main.main]" version=v4.0.0-0
> INFO[2018-04-04 09:28:16.945824] loaded configuration from
> filefile=/etc/glusterd2/glusterd2.toml
> source="[config.go:163:main.initConfig]"
> DEBU[2018-04-04 09:28:16.946343] running with
> configurationcert-file= clientaddress="192.168.222.24:
> 24007" config=/etc/glusterd2/glusterd2.toml defaultpeerport=24008
> etcdcurls="http://192.168.222.24:2379; etcdendpoints="[]" etcdpurls="
> http://192.168.222.24:2380; key-file= localstatedir=/var/lib/glusterd2
> logdir=/var/log/glusterd2 logfile=glusterd2.log loglevel=debug
> noembed=false peeraddress="192.168.222.24:24008"
> rundir=/usr/var/run/glusterd2 source="[config.go:129:main.dumpConfigToLog]"
> statedump=true version=false workdir=/etc/glusterd2
> DEBU[2018-04-04 09:28:17.067244] loading templates
> source="[template.go:64:volgen.LoadTemplates]" templatesdir=
> DEBU[2018-04-04 09:28:17.068207] generated default templates
> source="[template.go:82:volgen.LoadTemplates]" templates="[brick.graph
> distreplicate.graph fuse.graph distribute.graph replicate.graph
> disperse.graph]"
> DEBU[2018-04-04 09:28:17.068361] loaded template
> source="[template.go:102:volgen.LoadTemplate]" template=brick.graph
> DEBU[2018-04-04 09:28:17.068450] loaded template
> source="[template.go:102:volgen.LoadTemplate]"
> template=distreplicate.graph
> DEBU[2018-04-04 09:28:17.068539] loaded template
> source="[template.go:102:volgen.LoadTemplate]" template=fuse.graph
> DEBU[2018-04-04 09:28:17.068621] loaded template
> source="[template.go:102:volgen.LoadTemplate]" template=distribute.graph
> DEBU[2018-04-04 09:28:17.068798] loaded template
> source="[template.go:102:volgen.LoadTemplate]" template=replicate.graph
> DEBU[2018-04-04 09:28:17.068878] loaded template
> source="[template.go:102:volgen.LoadTemplate]" template=disperse.graph
> DEBU[2018-04-04 09:28:17.069316] saving updated store
> config   source="[config.go:166:store.GetConfig]"
> DEBU[2018-04-04 09:28:17.069745] starting embedded
> store   cafile= certfile= curls="
> http://192.168.222.24:2379; datadir=/var/lib/glusterd2/store endpoints="
> http://192.168.222.24:2379; keyfile= logdir=/var/log/glusterd2/store
> name=10641b8a-7135-4694-9b90-955827c8eba9 purls="http://192.168.222.24:2
> 380" source="[embed.go:32:store.newEmbedStore]" trustedcafile=
> 2018-04-04 09:28:17.071331 I | etcdserver/api/v3rpc: grpc:
> addrConn.resetTransport failed to create client transport: connection
> error: desc = "transport: dial tcp 192.168.222.24:2379: getsockopt:
> connection refused"; Reconnecting to {192.168.222.24:2379 }
> 2018-04-04 09:28:18.071206 I | etcdserver/api/v3rpc: grpc:
> addrConn.resetTransport failed to create client transport: connection
> error: desc = "transport: dial tcp 192.168.222.24:2379: getsockopt:
> connection refused"; Reconnecting to {192.168.222.24:2379 }
> 2018-04-04 09:28:19.681081 I | etcdserver/api/v3rpc: grpc:
> addrConn.resetTransport failed to create client transport: connection
> error: desc = "transport: dial tcp 192.168.222.24:2379: getsockopt:
> connection refused"; Reconnecting to {192.168.222.24:2379 }
> 2018-04-04 09:28:22.070811 I | etcdserver/api/v3rpc: Failed to dial
> 192.168.222.24:2379: context canceled; please retry.
> ERRO[2018-04-04 09:28:22.070891] failed to start embedded
> storeerror="dial tcp 192.168.222.24:2379: getsockopt:
> connection refused" source="[embed.go:36:store.newEmbedStore]"
> FATA[2018-04-04 09:28:22.071038] Failed to initialize store (etcd
> client)  error="dial tcp 192.168.222.24:2379: getsockopt: connection
> refused" source="[main.go:89:main.main]"
>
>
> cat glusterd2.toml
> localstatedir = "/var/lib/glusterd2"
> logdir = "/var/log/glusterd2"
> logfile = "glusterd2.log"
> rundir = "/usr/var/run/glusterd2"
> peeraddress = "192.168.222.24:24008"
> clientaddress = "192.168.222.24:24007"
> etcdcurls = "http://192.168.222.24:2379;
> etcdpurls = "http://192.168.222.24:2380;
>
>
> Any ideas?
>
>
> Thank you!
>
>
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Turn off replication

[Karthik Subrahmanya]

Hi Jose,

By switching into pure distribute volume you will lose availability if
something goes bad.

I am guessing you have a nX2 volume.
If you want to preserve one copy of the data in all the distributes, you
can do that by decreasing the replica count in the remove-brick operation.
If you have any inconsistency, heal them first using the "gluster volume
heal " command and wait till the
"gluster volume heal  info" output becomes zero, before removing
the bricks, so that you will have the correct data.
If you do not want to preserve the data then you can directly remove the
bricks.
Even after removing the bricks the data will be present in the backend of
the removed bricks. You have to manually erase them (both data and
.glusterfs folder).
See [1] for more details on remove-brick.

[1].
https://docs.gluster.org/en/latest/Administrator%20Guide/Managing%20Volumes/#shrinking-volumes

HTH,
Karthik


On Thu, Apr 5, 2018 at 8:17 PM, Jose Sanchez  wrote:

>
> We have a Gluster setup with 2 nodes (distributed replication) and we
> would like to switch it to the distributed mode. I know the data is
> duplicated between those nodes, what is the proper way of switching it to a
> distributed, we would like to double or gain the storage space on our
> gluster storage node. what happens with the data, do i need to erase one of
> the nodes?
>
> Jose
>
>
> -
> Jose Sanchez
> Systems/Network Analyst
> Center of Advanced Research Computing
> 1601 Central Ave.
> MSC 01 1190
> Albuquerque, NM 87131-0001
> carc.unm.edu
> 575.636.4232
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] performance.cache-size for high-RAM clients/servers, other tweaks for performance, and improvements to Gluster docs

[Artem Russakovskii]

I restarted rsync, and this has been sitting there for almost a minute,
barely moved several bytes in that time:

2014/11/545b06baa3d98/com.google.android.apps.inputmethod.zhuyin-2.1.0.79226761-armeabi-v7a-175-minAPI14.apk
  6,389,760  45%   18.76kB/s0:06:50

I straced each of the 3 processes rsync created and saw this (note: every
time there were several seconds of no output, I ctrl-C'ed and detached from
strace):

citadel:/home/archon810 # strace -p 16776
Process 16776 attached
select(6, [5], [4], [5], {45, 293510})  = 1 (out [4], left {44, 71342})
write(4,
"\4\200\0\7\3513>\2755\360[\372\317\337DZ\36\324\300o\235\377\247\367\177%\37\226\352\377\256\351"...,
32776) = 32776
ioctl(1, TIOCGPGRP, [16776])= 0
write(1, "\r  4,292,608  30%   27.07kB/"..., 46) = 46
select(6, [5], [4], [5], {60, 0})   = 1 (out [4], left {59, 98})
write(4,
"\4\200\0\7\270\224\277\24\31\247f\32\233x\t\276l\f-\254\r\246\324\360\30\235\350\6\34\304\230\242"...,
32776) = 32776
select(6, [5], [4], [5], {60, 0})   = 1 (out [4], left {59, 98})
write(4,
"\4\200\0\7\346_\363\36\33\320}Dd\5_\327\250\237i\242?B\276e\245\202Z\213\301[\25S"...,
32776) = 32776
select(6, [5], [4], [5], {60, 0})   = 1 (out [4], left {59, 98})
write(4,
"\4\200\0\7\330\303\221\357\225\37h\373\366X\306L\f>\234\\%n\253\266\5\372c\257>V\366\255"...,
32776) = 32776
select(6, [5], [4], [5], {60, 0})   = 1 (out [4], left {59, 98})
write(4,
"\4\200\0\7i\301\17u\224{/O\213\330\33\317\272\246\221\22\261|w\244\5\307|\21\373\v\356k"...,
32776) = 32776
select(6, [5], [4], [5], {60, 0})   = 1 (out [4], left {59, 98})
write(4,
"\4\200\0\7\270\277\233\206n\304:\362_\213~\356bm\5\350\337\26\203\225\332\277\372\275\247<\307\22"...,
32776) = 32776
read(3, "\316\214\260\341:\263P\214\373n\313\10\333
}\323\364Q\353\r\232d\204\257\\Q\306/\277\253/\356"..., 262144) = 262144
select(6, [5], [4], [5], {60, 0})   = 1 (out [4], left {59, 98})
write(4,
"\4\200\0\7\314\233\274S08\330\276\226\267\233\360rp\210x)\320\0314\223\323\3335Y\312\313\307"...,
32776) = 32776
select(6, [5], [4], [5], {60, 0})   = 1 (out [4], left {59, 98})
write(4, "\4\200\0\7\316\214\260\341:\263P\214\373n\313\10\333
}\323\364Q\353\r\232d\204\257\\Q\306/"..., 32776) = 32776
select(6, [5], [4], [5], {60, 0}^CProcess 16776 detached
 
citadel:/home/archon810 # strace -p 16777
Process 16777 attached
select(4, [3], [], [3], {38, 210908}^CProcess 16777 detached
 
citadel:/home/archon810 # strace -p 16776
Process 16776 attached
select(6, [5], [4], [5], {48, 295874}^CProcess 16776 detached
 
citadel:/home/archon810 # strace -p 16778
Process 16778 attached
select(1, [0], [], [0], {60, 0})= 1 (in [0], left {59, 96})
read(0,
"\0\200\0\0\4\200\0\7\3508\343\204\207\255\4\212y\230&&\372\30*\322\f\325v\335\230
\16v"..., 32768) = 32768
select(1, [0], [], [0], {60, 0})= 1 (in [0], left {59, 98})
read(0, "\373\30\2\2667\371\207)", 8)   = 8
select(1, [0], [], [0], {60, 0})= 1 (in [0], left {59, 99})
read(0,
"\0\200\0\0\4\200\0\7\6\213\2223\233\36-\350,\303\0\234\7`\317\276H\353u\217\275\316\333@"...,
32768) = 32768
select(1, [0], [], [0], {60, 0})= 1 (in [0], left {59, 99})
read(0, "\375\33\367_\357\330\362\222", 8) = 8
select(1, [0], [], [0], {60, 0})= 1 (in [0], left {59, 99})
read(0,
"\0\200\0\0\4\200\0\7`Nv\355\275\336wzQ\365\264\364\20AX\365DG\372\311\216\212\375\276"...,
32768) = 32768
select(1, [0], [], [0], {60, 0})= 1 (in [0], left {59, 99})
read(0, "\374)\300\264}\21\226s", 8)= 8
select(1, [0], [], [0], {60, 0})= 1 (in [0], left {59, 99})
read(0, "\0\200\0\0\4\200\0\7\10:\v\342O\305\374\5:Y+
\250\315\24\202J-@\256WC\320\371"...,
32768) = 32768
select(1, [0], [], [0], {60, 0})= 1 (in [0], left {59, 99})
read(0, "\3023\24O\343y\312\204", 8)= 8
select(1, [0], [], [0], {60, 0})= 1 (in [0], left {59, 99})
read(0,
"\0\200\0\0\4\200\0\7\27\22^\n/S.\215\362T\f\257Q\207z\241~B\3\32\32344\17"...,
32768) = 32768
select(1, [0], [], [0], {60, 0})= 1 (in [0], left {59, 98})
read(0, "\367P\222\262\224\17\25\250", 8) = 8
select(1, [0], [], [0], {60, 0})= 1 (in [0], left {59, 99})
read(0,
"\0\200\0\0\4\200\0\7FujR\213\372\341E\232\360\n\257\323\233>\364\245\37\3\31\314\20\206\362"...,
32768) = 32768
select(1, [0], [], [0], {60, 0})= 1 (in [0], left {59, 99})
read(0, "\203o\300\341\37\340(8", 8)= 8
select(1, [0], [], [0], {60, 0})= 1 (in [0], left {59, 98})
read(0,
"\0\200\0\0\4\200\0\7n\211\357\301\217\210\23\341$\342d8\25N\2035[\260\1\206B\206!\2"...,
32768) = 32768
select(1, [0], [], [0], {60, 0})= 1 (in [0], left {59, 99})
read(0, "|\222\223\336\201w\325\356", 8) = 8
select(1, [0], [], [0], {60, 0})= 1 (in [0], left {59, 99})
read(0,
"\0\200\0\0\4\200\0\7\220\216Y\343\362\366\231\372?\334N^\303\35\374cC;\vtx\231

Re: [Gluster-users] performance.cache-size for high-RAM clients/servers, other tweaks for performance, and improvements to Gluster docs

[Artem Russakovskii]

Hi again,

I'd like to expand on the performance issues and plead for help. Here's one
case which shows these odd hiccups: https://i.imgur.com/CXBPjTK.gifv.

In this GIF where I switch back and forth between copy operations on 2
servers, I'm copying a 10GB dir full of .apk and image files.

On server "hive" I'm copying straight from the main disk to an attached
volume block (xfs). As you can see, the transfers are relatively speedy and
don't hiccup.
On server "citadel" I'm copying the same set of data to a 4-replicate
gluster which uses block storage as a brick. As you can see, performance is
much worse, and there are frequent pauses for many seconds where nothing
seems to be happening - just freezes.

All 4 servers have the same specs, and all of them have performance issues
with gluster and no such issues when raw xfs block storage is used.

hive has long finished copying the data, while citadel is barely chugging
along and is expected to take probably half an hour to an hour. I have over
1TB of data to migrate, at which point if we went live, I'm not even sure
gluster would be able to keep up instead of bringing the machines and
services down.



Here's the cluster config, though it didn't seem to make any difference
performance-wise before I applied the customizations vs after.

Volume Name: apkmirror_data1
Type: Replicate
Volume ID: 11ecee7e-d4f8-497a-9994-ceb144d6841e
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 4 = 4
Transport-type: tcp
Bricks:
Brick1: nexus2:/mnt/nexus2_block1/apkmirror_data1
Brick2: forge:/mnt/forge_block1/apkmirror_data1
Brick3: hive:/mnt/hive_block1/apkmirror_data1
Brick4: citadel:/mnt/citadel_block1/apkmirror_data1
Options Reconfigured:
cluster.quorum-count: 1
cluster.quorum-type: fixed
network.ping-timeout: 5
network.remote-dio: enable
performance.rda-cache-limit: 256MB
performance.readdir-ahead: on
performance.parallel-readdir: on
network.inode-lru-limit: 50
performance.md-cache-timeout: 600
performance.cache-invalidation: on
performance.stat-prefetch: on
features.cache-invalidation-timeout: 600
features.cache-invalidation: on
cluster.readdir-optimize: on
performance.io-thread-count: 32
server.event-threads: 4
client.event-threads: 4
performance.read-ahead: off
cluster.lookup-optimize: on
performance.cache-size: 1GB
cluster.self-heal-daemon: enable
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: on


The mounts are done as follows in /etc/fstab:
/dev/disk/by-id/scsi-0Linode_Volume_citadel_block1 /mnt/citadel_block1 xfs
defaults 0 2
localhost:/apkmirror_data1 /mnt/apkmirror_data1 glusterfs defaults,_netdev
0 0

I'm really not sure if direct-io-mode mount tweaks would do anything here,
what the value should be set to, and what it is by default.

The OS is OpenSUSE 42.3, 64-bit. 80GB of RAM, 20 CPUs, hosted by Linode.

I'd really appreciate any help in the matter.

Thank you.


Sincerely,
Artem

--
Founder, Android Police , APK Mirror
, Illogical Robot LLC
beerpla.net | +ArtemRussakovskii
 | @ArtemR


On Thu, Apr 5, 2018 at 11:13 PM, Artem Russakovskii 
wrote:

> Hi,
>
> I'm trying to squeeze performance out of gluster on 4 80GB RAM 20-CPU
> machines where Gluster runs on attached block storage (Linode) in (4
> replicate bricks), and so far everything I tried results in sub-optimal
> performance.
>
> There are many files - mostly images, several million - and many
> operations take minutes, copying multiple files (even if they're small)
> suddenly freezes up for seconds at a time, then continues, iostat
> frequently shows large r_await and w_awaits with 100% utilization for the
> attached block device, etc.
>
> But anyway, there are many guides out there for small-file performance
> improvements, but more explanation is needed, and I think more tweaks
> should be possible.
>
> My question today is about performance.cache-size. Is this a size of cache
> in RAM? If so, how do I view the current cache size to see if it gets full
> and I should increase its size? Is it advisable to bump it up if I have
> many tens of gigs of RAM free?
>
>
>
> More generally, in the last 2 months since I first started working with
> gluster and set a production system live, I've been feeling frustrated
> because Gluster has a lot of poorly-documented and confusing options. I
> really wish documentation could be improved with examples and better
> explanations.
>
> Specifically, it'd be absolutely amazing if the docs offered a strategy
> for setting each value and ways of determining more optimal values. For
> example, for performance.cache-size, if it said something like "run command
> abc to see your current cache size, and if it's hurting, up it, but be
> aware that it's limited by RAM," it'd be already a huge improvement to the
> docs. And so on with other options.
>
>
>
> The gluster team is quite helpful 

Re: [Gluster-users] Any feature allow to add lock on a file between different apps?

[Pranith Kumar Karampuri]

You can use posix-locks i.e. fnctl based advisory locks on glusterfs just
like any other fs.

On Wed, Apr 4, 2018 at 8:30 AM, Lei Gong  wrote:

> Hello there，
>
>
>
> I want to know if there is a feature allow user to add lock on a file when
> their app is modifying that file, so that other apps could use the file
> when its unlocked.
>
>
>
> Thanks
>
>
>
> Lei
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>



-- 
Pranith
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] performance.cache-size for high-RAM clients/servers, other tweaks for performance, and improvements to Gluster docs

[Artem Russakovskii]

Hi,

I'm trying to squeeze performance out of gluster on 4 80GB RAM 20-CPU
machines where Gluster runs on attached block storage (Linode) in (4
replicate bricks), and so far everything I tried results in sub-optimal
performance.

There are many files - mostly images, several million - and many operations
take minutes, copying multiple files (even if they're small) suddenly
freezes up for seconds at a time, then continues, iostat frequently shows
large r_await and w_awaits with 100% utilization for the attached block
device, etc.

But anyway, there are many guides out there for small-file performance
improvements, but more explanation is needed, and I think more tweaks
should be possible.

My question today is about performance.cache-size. Is this a size of cache
in RAM? If so, how do I view the current cache size to see if it gets full
and I should increase its size? Is it advisable to bump it up if I have
many tens of gigs of RAM free?



More generally, in the last 2 months since I first started working with
gluster and set a production system live, I've been feeling frustrated
because Gluster has a lot of poorly-documented and confusing options. I
really wish documentation could be improved with examples and better
explanations.

Specifically, it'd be absolutely amazing if the docs offered a strategy for
setting each value and ways of determining more optimal values. For
example, for performance.cache-size, if it said something like "run command
abc to see your current cache size, and if it's hurting, up it, but be
aware that it's limited by RAM," it'd be already a huge improvement to the
docs. And so on with other options.



The gluster team is quite helpful on this mailing list, but in a reactive
rather than proactive way. Perhaps it's tunnel vision once you've worked on
a project for so long where less technical explanations and even proper
documentation of options takes a back seat, but I encourage you to be more
proactive about helping us understand and optimize Gluster.

Thank you.

Sincerely,
Artem

--
Founder, Android Police , APK Mirror
, Illogical Robot LLC
beerpla.net | +ArtemRussakovskii
 | @ArtemR

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users