[Gluster-users] recover from failed RAID

[Jim Kinney]

All,

I had a "bad timing" event where I lost 3 drives in a RAID6 array and
the structure of all of the LVM pools and nodes was lost. All total,
nearly 100TB of storage was scrambled.

This array was 1/2 of a redundant (replica 2) gluster config (will be
adding additional 3rd soon for split brain/redundancy with failure
issues) so the data was not lost but just running in a degraded mode.

The failed drives were replaced, the array rebuilt, all the thin_pools
and thin_volumes recreated, LUKS recreated and now the data from the
other half has been bulk copied to the rebuilt mirror locations - along
with all the .glusterfs space - not sure if that should be copied over
or not.

It's time to bring gluster back online. The original gluster data was
not part of the raid failure and all the names are the same (except one
LV, thin_lv, and it's thin_pool - working to change those now). 

s... Now what? At this time if I try and activate gluster it runs
(and I shut it down awaiting better wisdom from here than I have). Do I
need to just sit back and "let the magic happen"? What should I be
looking for to head off issues.

(getting prices now for a third node).___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] returning from a failed RAID

[Jim Kinney]

All,

I had a "bad timing" event where I lost 3 drives in a RAID6 array and
the structure of all of the LVM pools and nodes was lost.

This array was 1/2 of a redundant (replica 2) gluster config (will be
adding additional 3rd soon for split brain/redundancy with failure
issues).

The failed drives were replaced, the array rebuilt, all the thin_pools
and thin_volumes recreated, LUKS recreated and now the data from the
other half has been bulk copied to the rebuilt mirror locations.

It's time to bring gluster back online. The original gluster data was
not part of the raid failure and all the names are the same (except one
LV, thin_lv, and it's thin_pool). 

s... Now what? At this time if I try and activate gluster it runs.
Do I need to just sit back and "let the magic happen"? What should I be
looking for to head off issues.
-- 
James P. Kinney III

Every time you stop a school, you will have to build a jail. What you
gain at one end you lose at the other. It's like feeding a dog on his
own tail. It won't fatten the dog.
- Speech 11/23/1900 Mark Twain

http://heretothereideas.blogspot.com/
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] high (sys) cpu util and high load

[Ingard Mevåg]

2017-09-05 19:58 GMT+02:00 Vijay Bellur :

> On 09/04/2017 10:00 AM, Ingard Mevåg wrote:
>
>> Hi
>>
>> I'm seeing quite high cpu sys utilisation and an increased system load
>> the past few days on my servers. It appears it doesn't start at exactly the
>> same time for the different servers, but I've not (yet) been able to pin
>> the cpu usage to a specific task or entries in the logs.
>>
>> The cluster is running distribute with 8nodes and 4 bricks each version
>> 3.10
>> The nodes have 32 (HT) cores and 64 gigs of ram.
>>
>> Does anyone know what I can attribute this behaviour to? Or how I can
>> figure out what is causing it?
>>
>>
> Looking for logs of servers and clients around the spike interval is a
> good place to start. Operations like self-heal in gluster can cause higher
> resource utilization than normal.
>

I did think that it might be related to self-heal or rebalance or something
similar as it seemingly started at different times for the different
members of the cluster. But I couldn't find anything in the logs that
suggested that some background operation was running. I did notice the call
log saying that a lot of the bricks had Pending tasks. Could it be
connected to that? Or some other resource being starved?
Would gluster volume status show self heal or distribute running?


>
> HTH,
> Vijay
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [Gluster-devel] docs.gluster.org

[Amye Scavarda]

On Mon, Sep 4, 2017 at 1:28 AM, Niels de Vos  wrote:
> On Fri, Sep 01, 2017 at 06:21:38PM -0400, Amye Scavarda wrote:
>> On Fri, Sep 1, 2017 at 9:42 AM, Michael Scherer  wrote:
>> > Le vendredi 01 septembre 2017 à 14:02 +0100, Michael Scherer a écrit :
>> >> Le mercredi 30 août 2017 à 12:11 +0530, Nigel Babu a écrit :
>> >> > Hello,
>> >> >
>> >> > To reduce confusion, we've setup docs.gluster.org pointing to
>> >> > gluster.readthedocs.org. Both URLs will continue to work for the
>> >> > forseeable
>> >> > future.
>> >> >
>> >> > Please update any references that you control to point to
>> >> > docs.gluster.org. At
>> >> > some point in the distant future, we will switch to hosting
>> >> > docs.gluster.org on
>> >> > our own servers.
>> >> >
>> >> > RTD will set up a canonical link to docs.gluster.org[1]. Over time,
>> >> > this will
>> >> > change update the results on search engines to docs.gluster.org.
>> >> > This
>> >> > change
>> >> > will reduce confusion we've had with copies of our docs hosted on
>> >> > RTD.
>> >> >
>> >> > [1]: https://docs.readthedocs.io/en/latest/canonical.html
>> >>
>> >> So , seems TLS certificate is wrong, should we correct the link to be
>> >> http for now ?
>> >
>> > So I opened a few PR/review:
>> > https://github.com/gluster/glusterdocs/pull/259
>> >
>> > https://review.gluster.org/#/c/18182/
>> >
>> > https://github.com/gluster/glusterweb/pull/148
>> >
>> >
>> > --
>> > Michael Scherer
>> > Sysadmin, Community Infrastructure and Platform, OSAS
>> >
>> >
>> > ___
>> > Gluster-devel mailing list
>> > gluster-de...@gluster.org
>> > http://lists.gluster.org/mailman/listinfo/gluster-devel
>>
>> Fair warning, glusterweb is now deprecated as discussed in Community
>> Meeting on 30 Aug. However, github.com/gluster/glusterweb will be used
>> as a bug tracker moving forward.
>
> Could you please replace the contents of the current glusterweb
> repository with a README explaining this? The old files can stay in the
> history so that we can reference them in case something on the new
> website needs to be re-added.
>
> Being able to send pull requests and get contributions from users that
> way was one of the main reasons to move to GitHub. Is that still
> possible with the new site, in a different repository? I guess WordPress
> has some import/export features, but I don't know if those can get
> nicely combined with a repository on GitHub.
>
> Thanks!
> Niels

So we're actually going to move forward in a different direction with
the glusterweb repository once we get everything cleaned up.

We'll still use the GitHub issue queue as a method of tracking
improvements/bugs/feature requests , but I'd like to get us to a place
where we have a WordPress theme that people can contribute to and code
for custom plugins that we've developed. I'll be looking for
volunteers that can edit content directly as well.

I'll create a branch outlining what that could look like and we'll
discuss at the next community meeting.
- amye


-- 
Amye Scavarda | a...@redhat.com | Gluster Community Lead
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] high (sys) cpu util and high load

[Vijay Bellur]

On 09/04/2017 10:00 AM, Ingard Mevåg wrote:

Hi

I'm seeing quite high cpu sys utilisation and an increased system load 
the past few days on my servers. It appears it doesn't start at exactly 
the same time for the different servers, but I've not (yet) been able to 
pin the cpu usage to a specific task or entries in the logs.


The cluster is running distribute with 8nodes and 4 bricks each version 3.10
The nodes have 32 (HT) cores and 64 gigs of ram.

Does anyone know what I can attribute this behaviour to? Or how I can 
figure out what is causing it?




Looking for logs of servers and clients around the spike interval is a 
good place to start. Operations like self-heal in gluster can cause 
higher resource utilization than normal.


HTH,
Vijay

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Announcing GlusterFS release 3.12.0 (Long Term Maintenance)

[Shyam Ranganathan]

This is a major Gluster release that includes, features and bug fixes. 
Notable feature highlights are,


* Ability to mount sub-directories using the Gluster native 
protocol (FUSE)


* Brick multiplexing enhancements that help scale to larger brick 
counts per node


* Enhancements to gluster get-state CLI enabling better 
understanding of various bricks and nodes participation/roles in the cluster


* Ability to resolve GFID split-brain using existing CLI

* Easier GFID to real path mapping thus enabling diagnostics and 
correction for reported GFID issues (healing among other uses where GFID 
is the only available source for identifying a file)


The features and changes are documented in the release notes [1]. A full 
list of bugs that have been addressed is included in the release notes 
as well [1].


The packages can be downloaded from [2] and are signed with [3].

Further, as 3.11 release is a short term maintenance release, features 
included in 3.11 are available with 3.12 as well, and could be of 
interest to users upgrading to 3.12 from older than 3.11 releases. The 
3.11 release notes [2] captures the list of features that were 
introduced with 3.11.


Upgrade guide for release-3.12 can be found here [4].

Releases that will no longer receive updates (or are reaching EOL) with 
this release are: 3.11, 3.8 (see [5])


Releases that will continue to be updated in the future as of this 
release are: 3.12, 3.10 (see [5])


[1] 3.12.0 release notes: 
https://github.com/gluster/glusterfs/blob/release-3.12/doc/release-notes/3.12.0.md


[2] Packages: https://download.gluster.org/pub/gluster/glusterfs/3.12/

[3] Packages signed with: 
https://download.gluster.org/pub/gluster/glusterfs/3.12/rsa.pub


[4] Upgrade guide to release 3.12: 
https://gluster.readthedocs.io/en/latest/Upgrade-Guide/upgrade_to_3.12/


[5] Release schedule: https://www.gluster.org/release-schedule/
___
Announce mailing list
annou...@gluster.org
http://lists.gluster.org/mailman/listinfo/announce
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] GlusterFS as virtual machine storage

[Gionatan Danti]

Il 04-09-2017 19:27 Ivan Rossi ha scritto:

The latter one is the one I have been referring to. And it is pretty
dangerous Imho


Sorry, I can not find the bug report/entry by myself.
Can you link to some more information, or explain which bug are your 
referring and how to trigger it?


Thanks.

--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.da...@assyoma.it - i...@assyoma.it
GPG public key ID: FF5F32A8
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Poor performance with shard

[Diego Remolina]

>From my understanding of gluster, that is to be expected since instead
of having to stat a single file without sharding, now you have to stat
multiple files when you shard. Remember that gluster is not so great
at dealing with "lots" of files, so if you have a single 100GB
file/image stored in gluster, and it gets sharded into 512MB pieces,
you are going to now have to stat ~195 files instead of a single file.
The more files you have to stat, the slower gluster is, specially in
replica situations since my understanding is that you have to stat
each file in each brick.

On the other hand, if you had a single file and one of your nodes is
rebooted, the i/o to the whole 100GB is stopped while healing if you
are *not* using sharding, so you have to wait for the whole 100GB to
be synced without sharding. It is a trade off that you will have to
evaluate, single file vs sharded and performance when healing.

If you are storing VM images, you may want to look into applying the
gluster settings for VMs:

https://github.com/gluster/glusterfs/blob/master/extras/group-virt.example

This may help improve performance, but I think you will still have
higher throughput with a single file vs sharded, whereas you will have
faster healing with sharding, as you only heal the modified shards.
Also make sure to be careful because once sharding is enabled and you
have sharded files, if you disable it, it will corrupt your sharded
vms.

Diego

On Sun, Sep 3, 2017 at 12:22 PM, Roei G  wrote:
> Hey everyone!
> I have deployed gluster on 3 nodes with 4 SSDs each and 10Gb Ethernet
> connection.
>
> The storage is configured with 3 gluster volumes, every volume has 12 bricks
> (4 bricks on every server, 1 per ssd in the server).
>
> With the 'features.shard' off option my writing speed (using the 'dd'
> command) is approximately 250 Mbs and when the feature is on the writing
> speed is around 130mbs.
>
> - gluster version 3.8.13 
>
> Volume name: data
> Number of bricks : 4 * 3 = 12
> Bricks:
> Brick1: server1:/brick/data1
> Brick2: server1:/brick/data2
> Brick3: server1:/brick/data3
> Brick4: server1:/brick/data4
> Brick5: server2:/brick/data1
> .
> .
> .
> Options reconfigure:
> Performance.strict-o-direct: off
> Cluster.nufa: off
> Features.shard-block-size: 512MB
> Features.shard: on
> Cluster.server-quorum-type: server
> Cluster.quorum-type: auto
> Cluster.eager-lock: enable
> Network.remote-dio: on
> Performance.readdir-ahead: on
>
> Any idea on how to improve my performance?
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Poor performance with shard

[Roei G]

Hey everyone!
I have deployed gluster on 3 nodes with 4 SSDs each and 10Gb Ethernet
connection.

The storage is configured with 3 gluster volumes, every volume has 12
bricks (4 bricks on every server, 1 per ssd in the server).

With the 'features.shard' off option my writing speed (using the 'dd'
command) is approximately 250 Mbs and when the feature is on the writing
speed is around 130mbs.

- gluster version 3.8.13 

Volume name: data
Number of bricks : 4 * 3 = 12
Bricks:
Brick1: server1:/brick/data1
Brick2: server1:/brick/data2
Brick3: server1:/brick/data3
Brick4: server1:/brick/data4
Brick5: server2:/brick/data1
.
.
.
Options reconfigure:
Performance.strict-o-direct: off
Cluster.nufa: off
Features.shard-block-size: 512MB
Features.shard: on
Cluster.server-quorum-type: server
Cluster.quorum-type: auto
Cluster.eager-lock: enable
Network.remote-dio: on
Performance.readdir-ahead: on

Any idea on how to improve my performance?
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Glusterd proccess hangs on reboot

[Serkan Çoban]

Ok, I am going for 2x40 server clusters then, thanks for help.

On Tue, Sep 5, 2017 at 4:57 PM, Atin Mukherjee  wrote:
>
>
> On Tue, Sep 5, 2017 at 6:13 PM, Serkan Çoban  wrote:
>>
>> Some corrections about the previous mails. Problem does not happen
>> when no volumes created.
>> Problem happens volumes created but in stopped state. Problem also
>> happens when volumes started state.
>> Below is the 5 stack traces taken by 10 min intervals and volumes stopped
>> state.
>
>
> As I mentioned earlier, this is technically not a *hang* . Due to the costly
> handshaking operations for too many bricks from too many nodes, the glusterd
> takes a quite long amount of time to finish the handshake.
>
>>
>>
>> --1--
>> Thread 8 (Thread 0x7f413f3a7700 (LWP 104249)):
>> #0  0x003d99c0f00d in nanosleep () from /lib64/libpthread.so.0
>> #1  0x7f4146312d57 in gf_timer_proc () from
>> /usr/lib64/libglusterfs.so.0
>> #2  0x003d99c07aa1 in start_thread () from /lib64/libpthread.so.0
>> #3  0x003d998e8bbd in clone () from /lib64/libc.so.6
>> Thread 7 (Thread 0x7f413e9a6700 (LWP 104250)):
>> #0  0x003d99c0f585 in sigwait () from /lib64/libpthread.so.0
>> #1  0x0040643b in glusterfs_sigwaiter ()
>> #2  0x003d99c07aa1 in start_thread () from /lib64/libpthread.so.0
>> #3  0x003d998e8bbd in clone () from /lib64/libc.so.6
>> Thread 6 (Thread 0x7f413dfa5700 (LWP 104251)):
>> #0  0x003d998acc4d in nanosleep () from /lib64/libc.so.6
>> #1  0x003d998acac0 in sleep () from /lib64/libc.so.6
>> #2  0x7f414632d8fb in pool_sweeper () from
>> /usr/lib64/libglusterfs.so.0
>> #3  0x003d99c07aa1 in start_thread () from /lib64/libpthread.so.0
>> #4  0x003d998e8bbd in clone () from /lib64/libc.so.6
>> Thread 5 (Thread 0x7f413d5a4700 (LWP 104252)):
>> #0  0x003d99c0ba5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1  0x7f414633fafc in syncenv_task () from
>> /usr/lib64/libglusterfs.so.0
>> #2  0x7f414634d9f0 in syncenv_processor () from
>> /usr/lib64/libglusterfs.so.0
>> #3  0x003d99c07aa1 in start_thread () from /lib64/libpthread.so.0
>> #4  0x003d998e8bbd in clone () from /lib64/libc.so.6
>> Thread 4 (Thread 0x7f413cba3700 (LWP 104253)):
>> #0  0x003d99c0ba5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1  0x7f414633fafc in syncenv_task () from
>> /usr/lib64/libglusterfs.so.0
>> #2  0x7f414634d9f0 in syncenv_processor () from
>> /usr/lib64/libglusterfs.so.0
>> #3  0x003d99c07aa1 in start_thread () from /lib64/libpthread.so.0
>> #4  0x003d998e8bbd in clone () from /lib64/libc.so.6
>> Thread 3 (Thread 0x7f413aa48700 (LWP 104255)):
>> #0  0x003d99c0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1  0x7f413befb99b in hooks_worker () from
>> /usr/lib64/glusterfs/3.10.5/xlator/mgmt/glusterd.so
>> #2  0x003d99c07aa1 in start_thread () from /lib64/libpthread.so.0
>> #3  0x003d998e8bbd in clone () from /lib64/libc.so.6
>> Thread 2 (Thread 0x7f413a047700 (LWP 104256)):
>> #0  0x7f41462fd43d in dict_lookup_common () from
>> /usr/lib64/libglusterfs.so.0
>> #1  0x7f41462ff33d in dict_set_lk () from /usr/lib64/libglusterfs.so.0
>> #2  0x7f41462ff5f5 in dict_set () from /usr/lib64/libglusterfs.so.0
>> #3  0x7f414630024c in dict_set_str () from
>> /usr/lib64/libglusterfs.so.0
>> #4  0x7f413be75f29 in glusterd_add_volume_to_dict () from
>> /usr/lib64/glusterfs/3.10.5/xlator/mgmt/glusterd.so
>> #5  0x7f413be7647c in glusterd_add_volumes_to_export_dict () from
>> /usr/lib64/glusterfs/3.10.5/xlator/mgmt/glusterd.so
>> #6  0x7f413be8cedf in glusterd_rpc_friend_add () from
>> /usr/lib64/glusterfs/3.10.5/xlator/mgmt/glusterd.so
>> #7  0x7f413be4d8f7 in glusterd_ac_friend_add () from
>> /usr/lib64/glusterfs/3.10.5/xlator/mgmt/glusterd.so
>> #8  0x7f413be4bbb9 in glusterd_friend_sm () from
>> /usr/lib64/glusterfs/3.10.5/xlator/mgmt/glusterd.so
>> #9  0x7f413bea789a in __glusterd_mgmt_hndsk_version_ack_cbk ()
>> from /usr/lib64/glusterfs/3.10.5/xlator/mgmt/glusterd.so
>> #10 0x7f413be8d3ee in glusterd_big_locked_cbk () from
>> /usr/lib64/glusterfs/3.10.5/xlator/mgmt/glusterd.so
>> #11 0x7f41460cfad5 in rpc_clnt_handle_reply () from
>> /usr/lib64/libgfrpc.so.0
>> #12 0x7f41460d0c85 in rpc_clnt_notify () from /usr/lib64/libgfrpc.so.0
>> #13 0x7f41460cbd68 in rpc_transport_notify () from
>> /usr/lib64/libgfrpc.so.0
>> #14 0x7f413ae8dccd in socket_event_poll_in () from
>> /usr/lib64/glusterfs/3.10.5/rpc-transport/socket.so
>> #15 0x7f413ae8effe in socket_event_handler () from
>> /usr/lib64/glusterfs/3.10.5/rpc-transport/socket.so
>> #16 0x7f4146362806 in event_dispatch_epoll_worker () from
>> /usr/lib64/libglusterfs.so.0
>> #17 0x003d99c07aa1 in start_thread () from /lib64/libpthread.so.0
>> #18 0x003d998e8bbd in clone () from /lib64/libc.so.6
>> Thread 1 

Re: [Gluster-users] Glusterd proccess hangs on reboot

[Atin Mukherjee]

On Tue, Sep 5, 2017 at 6:13 PM, Serkan Çoban  wrote:

> Some corrections about the previous mails. Problem does not happen
> when no volumes created.
> Problem happens volumes created but in stopped state. Problem also
> happens when volumes started state.
> Below is the 5 stack traces taken by 10 min intervals and volumes stopped
> state.
>

As I mentioned earlier, this is technically not a *hang* . Due to the
costly handshaking operations for too many bricks from too many nodes, the
glusterd takes a quite long amount of time to finish the handshake.


>
> --1--
> Thread 8 (Thread 0x7f413f3a7700 (LWP 104249)):
> #0  0x003d99c0f00d in nanosleep () from /lib64/libpthread.so.0
> #1  0x7f4146312d57 in gf_timer_proc () from
> /usr/lib64/libglusterfs.so.0
> #2  0x003d99c07aa1 in start_thread () from /lib64/libpthread.so.0
> #3  0x003d998e8bbd in clone () from /lib64/libc.so.6
> Thread 7 (Thread 0x7f413e9a6700 (LWP 104250)):
> #0  0x003d99c0f585 in sigwait () from /lib64/libpthread.so.0
> #1  0x0040643b in glusterfs_sigwaiter ()
> #2  0x003d99c07aa1 in start_thread () from /lib64/libpthread.so.0
> #3  0x003d998e8bbd in clone () from /lib64/libc.so.6
> Thread 6 (Thread 0x7f413dfa5700 (LWP 104251)):
> #0  0x003d998acc4d in nanosleep () from /lib64/libc.so.6
> #1  0x003d998acac0 in sleep () from /lib64/libc.so.6
> #2  0x7f414632d8fb in pool_sweeper () from /usr/lib64/libglusterfs.so.0
> #3  0x003d99c07aa1 in start_thread () from /lib64/libpthread.so.0
> #4  0x003d998e8bbd in clone () from /lib64/libc.so.6
> Thread 5 (Thread 0x7f413d5a4700 (LWP 104252)):
> #0  0x003d99c0ba5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1  0x7f414633fafc in syncenv_task () from /usr/lib64/libglusterfs.so.0
> #2  0x7f414634d9f0 in syncenv_processor () from
> /usr/lib64/libglusterfs.so.0
> #3  0x003d99c07aa1 in start_thread () from /lib64/libpthread.so.0
> #4  0x003d998e8bbd in clone () from /lib64/libc.so.6
> Thread 4 (Thread 0x7f413cba3700 (LWP 104253)):
> #0  0x003d99c0ba5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1  0x7f414633fafc in syncenv_task () from /usr/lib64/libglusterfs.so.0
> #2  0x7f414634d9f0 in syncenv_processor () from
> /usr/lib64/libglusterfs.so.0
> #3  0x003d99c07aa1 in start_thread () from /lib64/libpthread.so.0
> #4  0x003d998e8bbd in clone () from /lib64/libc.so.6
> Thread 3 (Thread 0x7f413aa48700 (LWP 104255)):
> #0  0x003d99c0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1  0x7f413befb99b in hooks_worker () from
> /usr/lib64/glusterfs/3.10.5/xlator/mgmt/glusterd.so
> #2  0x003d99c07aa1 in start_thread () from /lib64/libpthread.so.0
> #3  0x003d998e8bbd in clone () from /lib64/libc.so.6
> Thread 2 (Thread 0x7f413a047700 (LWP 104256)):
> #0  0x7f41462fd43d in dict_lookup_common () from
> /usr/lib64/libglusterfs.so.0
> #1  0x7f41462ff33d in dict_set_lk () from /usr/lib64/libglusterfs.so.0
> #2  0x7f41462ff5f5 in dict_set () from /usr/lib64/libglusterfs.so.0
> #3  0x7f414630024c in dict_set_str () from /usr/lib64/libglusterfs.so.0
> #4  0x7f413be75f29 in glusterd_add_volume_to_dict () from
> /usr/lib64/glusterfs/3.10.5/xlator/mgmt/glusterd.so
> #5  0x7f413be7647c in glusterd_add_volumes_to_export_dict () from
> /usr/lib64/glusterfs/3.10.5/xlator/mgmt/glusterd.so
> #6  0x7f413be8cedf in glusterd_rpc_friend_add () from
> /usr/lib64/glusterfs/3.10.5/xlator/mgmt/glusterd.so
> #7  0x7f413be4d8f7 in glusterd_ac_friend_add () from
> /usr/lib64/glusterfs/3.10.5/xlator/mgmt/glusterd.so
> #8  0x7f413be4bbb9 in glusterd_friend_sm () from
> /usr/lib64/glusterfs/3.10.5/xlator/mgmt/glusterd.so
> #9  0x7f413bea789a in __glusterd_mgmt_hndsk_version_ack_cbk ()
> from /usr/lib64/glusterfs/3.10.5/xlator/mgmt/glusterd.so
> #10 0x7f413be8d3ee in glusterd_big_locked_cbk () from
> /usr/lib64/glusterfs/3.10.5/xlator/mgmt/glusterd.so
> #11 0x7f41460cfad5 in rpc_clnt_handle_reply () from
> /usr/lib64/libgfrpc.so.0
> #12 0x7f41460d0c85 in rpc_clnt_notify () from /usr/lib64/libgfrpc.so.0
> #13 0x7f41460cbd68 in rpc_transport_notify () from
> /usr/lib64/libgfrpc.so.0
> #14 0x7f413ae8dccd in socket_event_poll_in () from
> /usr/lib64/glusterfs/3.10.5/rpc-transport/socket.so
> #15 0x7f413ae8effe in socket_event_handler () from
> /usr/lib64/glusterfs/3.10.5/rpc-transport/socket.so
> #16 0x7f4146362806 in event_dispatch_epoll_worker () from
> /usr/lib64/libglusterfs.so.0
> #17 0x003d99c07aa1 in start_thread () from /lib64/libpthread.so.0
> #18 0x003d998e8bbd in clone () from /lib64/libc.so.6
> Thread 1 (Thread 0x7f4145e9e740 (LWP 104248)):
> #0  0x003d99c082fd in pthread_join () from /lib64/libpthread.so.0
> #1  0x7f41463622d5 in event_dispatch_epoll () from
> /usr/lib64/libglusterfs.so.0
> #2  0x00409020 in main ()
>
> --2--
> 

Re: [Gluster-users] ganesha error ?

[Renaud Fortier]

Hi,
This was my fstab line : 
gluster-node3v:/dev /data nfs4 noatime,_netdev,auto 0 0

I tought it was the way to mount nfs v4.0 but now, I don't think so ! So I 
changed it to :
gluster-node3v:/dev /data nfs nfsvers=4,noatime,_netdev,auto 0 0

If I look at the ganesha (niv_debug) logs, now I see it is mount in nfs v4.0 
instead of v4.2. 

I will first try with this and ask for help if it fails again.

Thank you
Renaud

-Message d'origine-
De : Soumya Koduri [mailto:skod...@redhat.com] 
Envoyé : 2 septembre 2017 03:10
À : Renaud Fortier ; gluster-users@gluster.org
Cc : nfs-ganesha-supp...@lists.sourceforge.net
Objet : Re: [Gluster-users] ganesha error ?


On 09/02/2017 02:09 AM, Renaud Fortier wrote:
> Hi,
> 
> I got these errors 3 times since I'm testing gluster with nfs-ganesha. 
> The clients are php apps and when this happen, clients got strange php 
> session error. Below, the first error only happen once but other 
> errors happen every time a clients try to create a new session file. 
> To make php apps work again, I had to restart the client. Do you have 
> an idea of what's happening here ?
> 
> 31/08/2017 17:02:55 : epoch 59a450e6 : GLUSTER-NODE3 : 
> ganesha.nfsd-2067[work-161] nfs4_State_Del :STATE :CRIT :hashtable get 
> latch failed: 2
> 
> 31/08/2017 17:04:00 : epoch 59a450e6 : GLUSTER-NODE3 : 
> ganesha.nfsd-2067[work-31] create_nfs4_owner :NFS4 LOCK :CRIT :Related
> {STATE_OPEN_OWNER_NFSV4 0x7f0274475840: clientid={0x7f036836b0f0
> ClientID={Epoch=0x59a450e6 Counter=0x0008} CONFIRMED
> Client={0x7f0278001940 name=(24:Linux NFSv4.2 devel-web3) refcount=4}
> t_delta=0 reservations=1 refcount=466}
> owner=(24:0x6f70656e2069643a0027a95056901336)
> confirmed=0 seqid=0 refcount=2} doesn't match for
> {STATE_LOCK_OWNER_NFSV4 0x7f0328479b10: clientid={0x7f036836b0f0
> ClientID={Epoch=0x59a450e6 Counter=0x0008} CONFIRMED
> Client={0x7f0278001940 name=(24:Linux NFSv4.2 devel-web3) refcount=4}
> t_delta=0 reservations=1 refcount=466}
> owner=(20:0x6c6f636b2069643a0027) confirmed=0
> seqid=0 related_owner={STATE_OPEN_OWNER_NFSV4 0x7f03204cb2d0: 
> clientid={0x7f036836b0f0 ClientID={Epoch=0x59a450e6 
> Counter=0x0008} CONFIRMED Client={0x7f0278001940 name=(24:Linux 
> NFSv4.2 devel-web3) refcount=4} t_delta=0 reservations=1 refcount=466}
> owner=(24:0x6f70656e2069643a0027a90f9b20feca)
> confirmed=0 seqid=0 refcount=3} refcount=5}

The open owners doesn't seem to be matching. Pkt trace may help in further 
debugging.

Also we haven't tested v4.1 and v4.2 protocols extensively with glusterfs. 
Could you please check the behavior with v4.0 protocol mount.

Thanks,
Soumya

> 
> 31/08/2017 17:04:00 : epoch 59a450e6 : GLUSTER-NODE3 : 
> ganesha.nfsd-2067[work-31] nfs4_op_lock :NFS4 LOCK :EVENT :LOCK failed 
> to create new lock owner Lock: obj=0x7f036c3a8268, 
> fileid=11757051714723246668, type=READ , start=0x0, 
> end=0x, owner={STATE_OPEN_OWNER_NFSV4 0x7f0274475840:
> clientid={0x7f036836b0f0 ClientID={Epoch=0x59a450e6 
> Counter=0x0008} CONFIRMED Client={0x7f0278001940 name=(24:Linux 
> NFSv4.2 devel-web3) refcount=4} t_delta=0 reservations=1 refcount=466}
> owner=(24:0x6f70656e2069643a0027a95056901336)
> confirmed=0 seqid=0 refcount=2}
> 
> 31/08/2017 17:04:00 : epoch 59a450e6 : GLUSTER-NODE3 : 
> ganesha.nfsd-2067[work-71] create_nfs4_owner :NFS4 LOCK :CRIT :Related
> {STATE_OPEN_OWNER_NFSV4 0x7f0274475840: clientid={0x7f036836b0f0
> ClientID={Epoch=0x59a450e6 Counter=0x0008} CONFIRMED
> Client={0x7f0278001940 name=(24:Linux NFSv4.2 devel-web3) refcount=4}
> t_delta=0 reservations=1 refcount=466}
> owner=(24:0x6f70656e2069643a0027a95056901336)
> confirmed=0 seqid=0 refcount=2} doesn't match for
> {STATE_LOCK_OWNER_NFSV4 0x7f0328479b10: clientid={0x7f036836b0f0
> ClientID={Epoch=0x59a450e6 Counter=0x0008} CONFIRMED
> Client={0x7f0278001940 name=(24:Linux NFSv4.2 devel-web3) refcount=4}
> t_delta=0 reservations=1 refcount=466}
> owner=(20:0x6c6f636b2069643a0027) confirmed=0
> seqid=0 related_owner={STATE_OPEN_OWNER_NFSV4 0x7f03204cb2d0: 
> clientid={0x7f036836b0f0 ClientID={Epoch=0x59a450e6 
> Counter=0x0008} CONFIRMED Client={0x7f0278001940 name=(24:Linux 
> NFSv4.2 devel-web3) refcount=4} t_delta=0 reservations=1 refcount=466}
> owner=(24:0x6f70656e2069643a0027a90f9b20feca)
> confirmed=0 seqid=0 refcount=3} refcount=5}
> 
> 31/08/2017 17:04:00 : epoch 59a450e6 : GLUSTER-NODE3 : 
> ganesha.nfsd-2067[work-71] nfs4_op_lock :NFS4 LOCK :EVENT :LOCK failed 
> to create new lock owner Lock: obj=0x7f036c3a8268, 
> fileid=11757051714723246668, type=WRITE, start=0x0, 
> end=0x, owner={STATE_OPEN_OWNER_NFSV4 0x7f0274475840:
> clientid={0x7f036836b0f0 ClientID={Epoch=0x59a450e6 
> Counter=0x0008} CONFIRMED Client={0x7f0278001940 name=(24:Linux 
> NFSv4.2 devel-web3) refcount=4} t_delta=0 

[Gluster-users] samba gluster vfs & symlinks/submounts... ?

[lejeczek]

.. or just a way to make samba, when samba shares via 
glusterfs api, to show in a share/vol a submount(like fs 
bind) - how, if possible at all?
I guess it would have to be some sort of crossing from one 
vol to another vol's dir or something, hmm...


many thanks, L.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Glusterd proccess hangs on reboot

[Serkan Çoban]

Some corrections about the previous mails. Problem does not happen
when no volumes created.
Problem happens volumes created but in stopped state. Problem also
happens when volumes started state.
Below is the 5 stack traces taken by 10 min intervals and volumes stopped state.


--1--
Thread 8 (Thread 0x7f413f3a7700 (LWP 104249)):
#0  0x003d99c0f00d in nanosleep () from /lib64/libpthread.so.0
#1  0x7f4146312d57 in gf_timer_proc () from /usr/lib64/libglusterfs.so.0
#2  0x003d99c07aa1 in start_thread () from /lib64/libpthread.so.0
#3  0x003d998e8bbd in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f413e9a6700 (LWP 104250)):
#0  0x003d99c0f585 in sigwait () from /lib64/libpthread.so.0
#1  0x0040643b in glusterfs_sigwaiter ()
#2  0x003d99c07aa1 in start_thread () from /lib64/libpthread.so.0
#3  0x003d998e8bbd in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f413dfa5700 (LWP 104251)):
#0  0x003d998acc4d in nanosleep () from /lib64/libc.so.6
#1  0x003d998acac0 in sleep () from /lib64/libc.so.6
#2  0x7f414632d8fb in pool_sweeper () from /usr/lib64/libglusterfs.so.0
#3  0x003d99c07aa1 in start_thread () from /lib64/libpthread.so.0
#4  0x003d998e8bbd in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f413d5a4700 (LWP 104252)):
#0  0x003d99c0ba5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1  0x7f414633fafc in syncenv_task () from /usr/lib64/libglusterfs.so.0
#2  0x7f414634d9f0 in syncenv_processor () from /usr/lib64/libglusterfs.so.0
#3  0x003d99c07aa1 in start_thread () from /lib64/libpthread.so.0
#4  0x003d998e8bbd in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f413cba3700 (LWP 104253)):
#0  0x003d99c0ba5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1  0x7f414633fafc in syncenv_task () from /usr/lib64/libglusterfs.so.0
#2  0x7f414634d9f0 in syncenv_processor () from /usr/lib64/libglusterfs.so.0
#3  0x003d99c07aa1 in start_thread () from /lib64/libpthread.so.0
#4  0x003d998e8bbd in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f413aa48700 (LWP 104255)):
#0  0x003d99c0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1  0x7f413befb99b in hooks_worker () from
/usr/lib64/glusterfs/3.10.5/xlator/mgmt/glusterd.so
#2  0x003d99c07aa1 in start_thread () from /lib64/libpthread.so.0
#3  0x003d998e8bbd in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f413a047700 (LWP 104256)):
#0  0x7f41462fd43d in dict_lookup_common () from
/usr/lib64/libglusterfs.so.0
#1  0x7f41462ff33d in dict_set_lk () from /usr/lib64/libglusterfs.so.0
#2  0x7f41462ff5f5 in dict_set () from /usr/lib64/libglusterfs.so.0
#3  0x7f414630024c in dict_set_str () from /usr/lib64/libglusterfs.so.0
#4  0x7f413be75f29 in glusterd_add_volume_to_dict () from
/usr/lib64/glusterfs/3.10.5/xlator/mgmt/glusterd.so
#5  0x7f413be7647c in glusterd_add_volumes_to_export_dict () from
/usr/lib64/glusterfs/3.10.5/xlator/mgmt/glusterd.so
#6  0x7f413be8cedf in glusterd_rpc_friend_add () from
/usr/lib64/glusterfs/3.10.5/xlator/mgmt/glusterd.so
#7  0x7f413be4d8f7 in glusterd_ac_friend_add () from
/usr/lib64/glusterfs/3.10.5/xlator/mgmt/glusterd.so
#8  0x7f413be4bbb9 in glusterd_friend_sm () from
/usr/lib64/glusterfs/3.10.5/xlator/mgmt/glusterd.so
#9  0x7f413bea789a in __glusterd_mgmt_hndsk_version_ack_cbk ()
from /usr/lib64/glusterfs/3.10.5/xlator/mgmt/glusterd.so
#10 0x7f413be8d3ee in glusterd_big_locked_cbk () from
/usr/lib64/glusterfs/3.10.5/xlator/mgmt/glusterd.so
#11 0x7f41460cfad5 in rpc_clnt_handle_reply () from /usr/lib64/libgfrpc.so.0
#12 0x7f41460d0c85 in rpc_clnt_notify () from /usr/lib64/libgfrpc.so.0
#13 0x7f41460cbd68 in rpc_transport_notify () from /usr/lib64/libgfrpc.so.0
#14 0x7f413ae8dccd in socket_event_poll_in () from
/usr/lib64/glusterfs/3.10.5/rpc-transport/socket.so
#15 0x7f413ae8effe in socket_event_handler () from
/usr/lib64/glusterfs/3.10.5/rpc-transport/socket.so
#16 0x7f4146362806 in event_dispatch_epoll_worker () from
/usr/lib64/libglusterfs.so.0
#17 0x003d99c07aa1 in start_thread () from /lib64/libpthread.so.0
#18 0x003d998e8bbd in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f4145e9e740 (LWP 104248)):
#0  0x003d99c082fd in pthread_join () from /lib64/libpthread.so.0
#1  0x7f41463622d5 in event_dispatch_epoll () from
/usr/lib64/libglusterfs.so.0
#2  0x00409020 in main ()

--2--
Thread 8 (Thread 0x7f413f3a7700 (LWP 104249)):
#0  0x003d99c0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1  0x7f41463405cb in __synclock_lock () from /usr/lib64/libglusterfs.so.0
#2  0x7f41463407ae in synclock_lock () from /usr/lib64/libglusterfs.so.0
#3  0x7f413be8d3df in glusterd_big_locked_cbk () from
/usr/lib64/glusterfs/3.10.5/xlator/mgmt/glusterd.so
#4  0x7f41460d04c4 in call_bail () from /usr/lib64/libgfrpc.so.0
#5  

Re: [Gluster-users] Slow performance of gluster volume

[Abi Askushi]

Hi Krutika,

Attached the profile stats. I enabled profiling then ran some dd tests.
Also 3 Windows VMs are running on top this volume but did not do any stress
testing on the VMs. I have left the profiling enabled in case more time is
needed for useful stats.

Thanx

On Tue, Sep 5, 2017 at 12:48 PM, Krutika Dhananjay 
wrote:

> OK my understanding is that with preallocated disks the performance with
> and without shard will be the same.
>
> In any case, please attach the volume profile[1], so we can see what else
> is slowing things down.
>
> -Krutika
>
> [1] - https://gluster.readthedocs.io/en/latest/Administrator%
> 20Guide/Monitoring%20Workload/#running-glusterfs-volume-profile-command
>
> On Tue, Sep 5, 2017 at 2:32 PM, Abi Askushi 
> wrote:
>
>> Hi Krutika,
>>
>> I already have a preallocated disk on VM.
>> Now I am checking performance with dd on the hypervisors which have the
>> gluster volume configured.
>>
>> I tried also several values of shard-block-size and I keep getting the
>> same low values on write performance.
>> Enabling client-io-threads also did not have any affect.
>>
>> The version of gluster I am using is glusterfs 3.8.12 built on May 11
>> 2017 18:46:20.
>> The setup is a set of 3 Centos 7.3 servers and ovirt 4.1, using gluster
>> as storage.
>>
>> Below are the current settings:
>>
>>
>> Volume Name: vms
>> Type: Replicate
>> Volume ID: 4513340d-7919-498b-bfe0-d836b5cea40b
>> Status: Started
>> Snapshot Count: 0
>> Number of Bricks: 1 x (2 + 1) = 3
>> Transport-type: tcp
>> Bricks:
>> Brick1: gluster0:/gluster/vms/brick
>> Brick2: gluster1:/gluster/vms/brick
>> Brick3: gluster2:/gluster/vms/brick (arbiter)
>> Options Reconfigured:
>> server.event-threads: 4
>> client.event-threads: 4
>> performance.client-io-threads: on
>> features.shard-block-size: 512MB
>> cluster.granular-entry-heal: enable
>> performance.strict-o-direct: on
>> network.ping-timeout: 30
>> storage.owner-gid: 36
>> storage.owner-uid: 36
>> user.cifs: off
>> features.shard: on
>> cluster.shd-wait-qlength: 1
>> cluster.shd-max-threads: 8
>> cluster.locking-scheme: granular
>> cluster.data-self-heal-algorithm: full
>> cluster.server-quorum-type: server
>> cluster.quorum-type: auto
>> cluster.eager-lock: enable
>> network.remote-dio: off
>> performance.low-prio-threads: 32
>> performance.stat-prefetch: on
>> performance.io-cache: off
>> performance.read-ahead: off
>> performance.quick-read: off
>> transport.address-family: inet
>> performance.readdir-ahead: on
>> nfs.disable: on
>> nfs.export-volumes: on
>>
>>
>> I observed that when testing with dd if=/dev/zero of=testfile bs=1G
>> count=1 I get 65MB/s on the vms gluster volume (and the network traffic
>> between the servers reaches ~ 500Mbps), while when testing with dd
>> if=/dev/zero of=testfile bs=1G count=1 *oflag=direct *I get a consistent
>> 10MB/s and the network traffic hardly reaching 100Mbps.
>>
>> Any other things one can do?
>>
>> On Tue, Sep 5, 2017 at 5:57 AM, Krutika Dhananjay 
>> wrote:
>>
>>> I'm assuming you are using this volume to store vm images, because I see
>>> shard in the options list.
>>>
>>> Speaking from shard translator's POV, one thing you can do to improve
>>> performance is to use preallocated images.
>>> This will at least eliminate the need for shard to perform multiple
>>> steps as part of the writes - such as creating the shard and then writing
>>> to it and then updating the aggregated file size - all of which require one
>>> network call each, which further get blown up once they reach AFR
>>> (replicate) into many more network calls.
>>>
>>> Second, I'm assuming you're using the default shard block size of 4MB
>>> (you can confirm this using `gluster volume get  shard-block-size`).
>>> In our tests, we've found that larger shard sizes perform better. So maybe
>>> change the shard-block-size to 64MB (`gluster volume set 
>>> shard-block-size 64MB`).
>>>
>>> Third, keep stat-prefetch enabled. We've found that qemu sends quite a
>>> lot of [f]stats which can be served from the (md)cache to improve
>>> performance. So enable that.
>>>
>>> Also, could you also enable client-io-threads and see if that improves
>>> performance?
>>>
>>> Which version of gluster are you using BTW?
>>>
>>> -Krutika
>>>
>>>
>>> On Tue, Sep 5, 2017 at 4:32 AM, Abi Askushi 
>>> wrote:
>>>
 Hi all,

 I have a gluster volume used to host several VMs (managed through
 oVirt).
 The volume is a replica 3 with arbiter and the 3 servers use 1 Gbit
 network for the storage.

 When testing with dd (dd if=/dev/zero of=testfile bs=1G count=1
 oflag=direct) out of the volume (e.g. writing at /root/) the performance of
 the dd is reported to be ~ 700MB/s, which is quite decent. When testing the
 dd on the gluster volume I get ~ 43 MB/s which way lower from the previous.
 When testing with dd the gluster volume, the network 

Re: [Gluster-users] Slow performance of gluster volume

[Krutika Dhananjay]

OK my understanding is that with preallocated disks the performance with
and without shard will be the same.

In any case, please attach the volume profile[1], so we can see what else
is slowing things down.

-Krutika

[1] -
https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Monitoring%20Workload/#running-glusterfs-volume-profile-command

On Tue, Sep 5, 2017 at 2:32 PM, Abi Askushi  wrote:

> Hi Krutika,
>
> I already have a preallocated disk on VM.
> Now I am checking performance with dd on the hypervisors which have the
> gluster volume configured.
>
> I tried also several values of shard-block-size and I keep getting the
> same low values on write performance.
> Enabling client-io-threads also did not have any affect.
>
> The version of gluster I am using is glusterfs 3.8.12 built on May 11 2017
> 18:46:20.
> The setup is a set of 3 Centos 7.3 servers and ovirt 4.1, using gluster as
> storage.
>
> Below are the current settings:
>
>
> Volume Name: vms
> Type: Replicate
> Volume ID: 4513340d-7919-498b-bfe0-d836b5cea40b
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x (2 + 1) = 3
> Transport-type: tcp
> Bricks:
> Brick1: gluster0:/gluster/vms/brick
> Brick2: gluster1:/gluster/vms/brick
> Brick3: gluster2:/gluster/vms/brick (arbiter)
> Options Reconfigured:
> server.event-threads: 4
> client.event-threads: 4
> performance.client-io-threads: on
> features.shard-block-size: 512MB
> cluster.granular-entry-heal: enable
> performance.strict-o-direct: on
> network.ping-timeout: 30
> storage.owner-gid: 36
> storage.owner-uid: 36
> user.cifs: off
> features.shard: on
> cluster.shd-wait-qlength: 1
> cluster.shd-max-threads: 8
> cluster.locking-scheme: granular
> cluster.data-self-heal-algorithm: full
> cluster.server-quorum-type: server
> cluster.quorum-type: auto
> cluster.eager-lock: enable
> network.remote-dio: off
> performance.low-prio-threads: 32
> performance.stat-prefetch: on
> performance.io-cache: off
> performance.read-ahead: off
> performance.quick-read: off
> transport.address-family: inet
> performance.readdir-ahead: on
> nfs.disable: on
> nfs.export-volumes: on
>
>
> I observed that when testing with dd if=/dev/zero of=testfile bs=1G
> count=1 I get 65MB/s on the vms gluster volume (and the network traffic
> between the servers reaches ~ 500Mbps), while when testing with dd
> if=/dev/zero of=testfile bs=1G count=1 *oflag=direct *I get a consistent
> 10MB/s and the network traffic hardly reaching 100Mbps.
>
> Any other things one can do?
>
> On Tue, Sep 5, 2017 at 5:57 AM, Krutika Dhananjay 
> wrote:
>
>> I'm assuming you are using this volume to store vm images, because I see
>> shard in the options list.
>>
>> Speaking from shard translator's POV, one thing you can do to improve
>> performance is to use preallocated images.
>> This will at least eliminate the need for shard to perform multiple steps
>> as part of the writes - such as creating the shard and then writing to it
>> and then updating the aggregated file size - all of which require one
>> network call each, which further get blown up once they reach AFR
>> (replicate) into many more network calls.
>>
>> Second, I'm assuming you're using the default shard block size of 4MB
>> (you can confirm this using `gluster volume get  shard-block-size`).
>> In our tests, we've found that larger shard sizes perform better. So maybe
>> change the shard-block-size to 64MB (`gluster volume set 
>> shard-block-size 64MB`).
>>
>> Third, keep stat-prefetch enabled. We've found that qemu sends quite a
>> lot of [f]stats which can be served from the (md)cache to improve
>> performance. So enable that.
>>
>> Also, could you also enable client-io-threads and see if that improves
>> performance?
>>
>> Which version of gluster are you using BTW?
>>
>> -Krutika
>>
>>
>> On Tue, Sep 5, 2017 at 4:32 AM, Abi Askushi 
>> wrote:
>>
>>> Hi all,
>>>
>>> I have a gluster volume used to host several VMs (managed through
>>> oVirt).
>>> The volume is a replica 3 with arbiter and the 3 servers use 1 Gbit
>>> network for the storage.
>>>
>>> When testing with dd (dd if=/dev/zero of=testfile bs=1G count=1
>>> oflag=direct) out of the volume (e.g. writing at /root/) the performance of
>>> the dd is reported to be ~ 700MB/s, which is quite decent. When testing the
>>> dd on the gluster volume I get ~ 43 MB/s which way lower from the previous.
>>> When testing with dd the gluster volume, the network traffic was not
>>> exceeding 450 Mbps on the network interface. I would expect to reach near
>>> 900 Mbps considering that there is 1 Gbit of bandwidth available. This
>>> results having VMs with very slow performance (especially on their write
>>> operations).
>>>
>>> The full details of the volume are below. Any advise on what can be
>>> tweaked will be highly appreciated.
>>>
>>> Volume Name: vms
>>> Type: Replicate
>>> Volume ID: 4513340d-7919-498b-bfe0-d836b5cea40b
>>> Status: 

Re: [Gluster-users] Slow performance of gluster volume

[Abi Askushi]

Hi Krutika,

I already have a preallocated disk on VM.
Now I am checking performance with dd on the hypervisors which have the
gluster volume configured.

I tried also several values of shard-block-size and I keep getting the same
low values on write performance.
Enabling client-io-threads also did not have any affect.

The version of gluster I am using is glusterfs 3.8.12 built on May 11 2017
18:46:20.
The setup is a set of 3 Centos 7.3 servers and ovirt 4.1, using gluster as
storage.

Below are the current settings:


Volume Name: vms
Type: Replicate
Volume ID: 4513340d-7919-498b-bfe0-d836b5cea40b
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (2 + 1) = 3
Transport-type: tcp
Bricks:
Brick1: gluster0:/gluster/vms/brick
Brick2: gluster1:/gluster/vms/brick
Brick3: gluster2:/gluster/vms/brick (arbiter)
Options Reconfigured:
server.event-threads: 4
client.event-threads: 4
performance.client-io-threads: on
features.shard-block-size: 512MB
cluster.granular-entry-heal: enable
performance.strict-o-direct: on
network.ping-timeout: 30
storage.owner-gid: 36
storage.owner-uid: 36
user.cifs: off
features.shard: on
cluster.shd-wait-qlength: 1
cluster.shd-max-threads: 8
cluster.locking-scheme: granular
cluster.data-self-heal-algorithm: full
cluster.server-quorum-type: server
cluster.quorum-type: auto
cluster.eager-lock: enable
network.remote-dio: off
performance.low-prio-threads: 32
performance.stat-prefetch: on
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
transport.address-family: inet
performance.readdir-ahead: on
nfs.disable: on
nfs.export-volumes: on


I observed that when testing with dd if=/dev/zero of=testfile bs=1G count=1
I get 65MB/s on the vms gluster volume (and the network traffic between the
servers reaches ~ 500Mbps), while when testing with dd if=/dev/zero
of=testfile bs=1G count=1 *oflag=direct *I get a consistent 10MB/s and the
network traffic hardly reaching 100Mbps.

Any other things one can do?

On Tue, Sep 5, 2017 at 5:57 AM, Krutika Dhananjay 
wrote:

> I'm assuming you are using this volume to store vm images, because I see
> shard in the options list.
>
> Speaking from shard translator's POV, one thing you can do to improve
> performance is to use preallocated images.
> This will at least eliminate the need for shard to perform multiple steps
> as part of the writes - such as creating the shard and then writing to it
> and then updating the aggregated file size - all of which require one
> network call each, which further get blown up once they reach AFR
> (replicate) into many more network calls.
>
> Second, I'm assuming you're using the default shard block size of 4MB (you
> can confirm this using `gluster volume get  shard-block-size`). In our
> tests, we've found that larger shard sizes perform better. So maybe change
> the shard-block-size to 64MB (`gluster volume set  shard-block-size
> 64MB`).
>
> Third, keep stat-prefetch enabled. We've found that qemu sends quite a lot
> of [f]stats which can be served from the (md)cache to improve performance.
> So enable that.
>
> Also, could you also enable client-io-threads and see if that improves
> performance?
>
> Which version of gluster are you using BTW?
>
> -Krutika
>
>
> On Tue, Sep 5, 2017 at 4:32 AM, Abi Askushi 
> wrote:
>
>> Hi all,
>>
>> I have a gluster volume used to host several VMs (managed through oVirt).
>> The volume is a replica 3 with arbiter and the 3 servers use 1 Gbit
>> network for the storage.
>>
>> When testing with dd (dd if=/dev/zero of=testfile bs=1G count=1
>> oflag=direct) out of the volume (e.g. writing at /root/) the performance of
>> the dd is reported to be ~ 700MB/s, which is quite decent. When testing the
>> dd on the gluster volume I get ~ 43 MB/s which way lower from the previous.
>> When testing with dd the gluster volume, the network traffic was not
>> exceeding 450 Mbps on the network interface. I would expect to reach near
>> 900 Mbps considering that there is 1 Gbit of bandwidth available. This
>> results having VMs with very slow performance (especially on their write
>> operations).
>>
>> The full details of the volume are below. Any advise on what can be
>> tweaked will be highly appreciated.
>>
>> Volume Name: vms
>> Type: Replicate
>> Volume ID: 4513340d-7919-498b-bfe0-d836b5cea40b
>> Status: Started
>> Snapshot Count: 0
>> Number of Bricks: 1 x (2 + 1) = 3
>> Transport-type: tcp
>> Bricks:
>> Brick1: gluster0:/gluster/vms/brick
>> Brick2: gluster1:/gluster/vms/brick
>> Brick3: gluster2:/gluster/vms/brick (arbiter)
>> Options Reconfigured:
>> cluster.granular-entry-heal: enable
>> performance.strict-o-direct: on
>> network.ping-timeout: 30
>> storage.owner-gid: 36
>> storage.owner-uid: 36
>> user.cifs: off
>> features.shard: on
>> cluster.shd-wait-qlength: 1
>> cluster.shd-max-threads: 8
>> cluster.locking-scheme: granular
>> cluster.data-self-heal-algorithm: full
>> cluster.server-quorum-type: 

Re: [Gluster-users] heal info OK but statistics not working

[Ravishankar N]

On 09/04/2017 07:35 PM, Atin Mukherjee wrote:

Ravi/Karthick,

If one of the self heal process is down, will the statstics heal-count 
command work?


No it doesn't seem to: glusterd stage-op phase fails because shd was 
down on that node and we error out.
FWIW, the error message "Gathering crawl statistics on volume GROUP-WORK 
has been unsuccessful on bricks that are down. Please check if all brick 
processes are running." is incorrect and once 
https://review.gluster.org/#/c/15724/ gets merged, you will get the 
correct error message like so:

/
/root@vm2 glusterfs]# gluster v heal testvol statistics
Gathering crawl statistics on volume testvol has been unsuccessful:
 Staging failed on vm1. Error: Self-heal daemon is not running. Check 
self-heal daemon log file./

/

-Ravi


On Mon, Sep 4, 2017 at 7:24 PM, lejeczek > wrote:


1) one peer, out of four, got separated from the network, from the
rest of the cluster.
2) that unavailable(while it was unavailable) peer got detached
with "gluster peer detach" command which succeeded, so now cluster
comprise of three peers
3) Self-heal daemon (for some reason) does not start(with an
attempt to restart glusted) on the peer which probed that fourth peer.
4) fourth unavailable peer is still up & running but is
inaccessible to other peers for network is disconnected,
segmented. That peer's gluster status show peer is still in the
cluster.
5) So, fourth peer's gluster(nor other processes) stack did not
fail nor crushed, just network got, is disconnected.
6) peer status show ok & connected for current three peers.

This is third time when it happens to me, very same way: each time
net-disjointed peer was brought back online then statistics &
details worked again.

can you not reproduce it?

$ gluster vol info QEMU-VMs

Volume Name: QEMU-VMs
Type: Replicate
Volume ID: 8709782a-daa5-4434-a816-c4e0aef8fef2
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.5.6.32:/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER-QEMU-VMs
Brick2: 10.5.6.49:/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER-QEMU-VMs
Brick3: 10.5.6.100:/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER-QEMU-VMs
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
storage.owner-gid: 107
storage.owner-uid: 107
performance.readdir-ahead: on
geo-replication.indexing: on
geo-replication.ignore-pid-check: on
changelog.changelog: on

$ gluster vol status QEMU-VMs
Status of volume: QEMU-VMs
Gluster process   TCP Port RDMA Port Online  Pid

--
Brick 10.5.6.32:/__.aLocalStorages/0/0-GLUS
TERs/0GLUSTER-QEMU-VMs   49156 0 Y   9302
Brick 10.5.6.49:/__.aLocalStorages/0/0-GLUS
TERs/0GLUSTER-QEMU-VMs   49156 0 Y   7610
Brick 10.5.6.100:/__.aLocalStorages/0/0-GLU
STERs/0GLUSTER-QEMU-VMs   49156 0 Y   11013
Self-heal Daemon on localhost   N/A   N/A Y  
3069276
Self-heal Daemon on 10.5.6.32   N/A   N/A Y  
3315870
Self-heal Daemon on 10.5.6.49   N/A   N/A N  
N/A  <--- HERE
Self-heal Daemon on 10.5.6.17   N/A   N/A Y   5163

Task Status of Volume QEMU-VMs

--
There are no active volume tasks

$ gluster vol heal QEMU-VMs statistics heal-count
Gathering count of entries to be healed on volume QEMU-VMs has
been unsuccessful on bricks that are down. Please check if all
brick processes are running.



On 04/09/17 11:47, Atin Mukherjee wrote:

Please provide the output of gluster volume info, gluster
volume status and gluster peer status.

On Mon, Sep 4, 2017 at 4:07 PM, lejeczek  >> wrote:

    hi all

    this:
    $ vol heal $_vol info
    outputs ok and exit code is 0
    But if I want to see statistics:
    $ gluster vol heal $_vol statistics
    Gathering crawl statistics on volume GROUP-WORK has
    been unsuccessful on bricks that are down. Please
    check if all brick processes are running.

    I suspect - gluster inability to cope with a situation
    where one peer(which is not even a brick for a single
    vol on the cluster!) is inaccessible to the rest of
    cluster.
    I have not played with any other variations of this
    case, eg. more than one peer goes down, etc.
    But I hope someone could try to replicate