Re: [Gluster-users] Unreasonably poor performance of replicated volumes

[Vlad Kopylov]

Guess you went through user lists and tried something like this already
http://lists.gluster.org/pipermail/gluster-users/2018-April/033811.html
I have a same exact setup and below is as far as it went after months of
trail and error.
We all have somewhat same setup and same issue with this - you can find
same post as yours on the daily basis.

On Wed, Apr 11, 2018 at 3:03 PM, Anastasia Belyaeva  wrote:

> Hello everybody!
>
> I have 3 gluster servers (*gluster 3.12.6, Centos 7.2*; those are
> actually virtual machines located on 3 separate physical XenServer7.1
> servers)
>
> They are all connected via infiniband network. Iperf3 shows around *23
> Gbit/s network bandwidth *between each 2 of them.
>
> Each server has 3 HDD put into a *stripe*3 thin pool (LVM2) *with logical
> volume created on top of it, formatted with *xfs*. Gluster top reports
> the following throughput:
>
> root@fsnode2 ~ $ gluster volume top r3vol write-perf bs 4096 count 524288
>> list-cnt 0
>> Brick: fsnode2.ibnet:/data/glusterfs/r3vol/brick1/brick
>> Throughput *631.82 MBps *time 3.3989 secs
>> Brick: fsnode6.ibnet:/data/glusterfs/r3vol/brick1/brick
>> Throughput *566.96 MBps *time 3.7877 secs
>> Brick: fsnode4.ibnet:/data/glusterfs/r3vol/brick1/brick
>> Throughput *546.65 MBps *time 3.9285 secs
>
>
> root@fsnode2 ~ $ gluster volume top r2vol write-perf bs 4096 count 524288
>> list-cnt 0
>> Brick: fsnode2.ibnet:/data/glusterfs/r2vol/brick1/brick
>> Throughput *539.60 MBps *time 3.9798 secs
>> Brick: fsnode4.ibnet:/data/glusterfs/r2vol/brick1/brick
>> Throughput *580.07 MBps *time 3.7021 secs
>
>
> And two *pure replicated ('replica 2' and 'replica 3')* volumes. *The
> 'replica 2' volume is for testing purpose only.
>
>> Volume Name: r2vol
>> Type: Replicate
>> Volume ID: 4748d0c0-6bef-40d5-b1ec-d30e10cfddd9
>> Status: Started
>> Snapshot Count: 0
>> Number of Bricks: 1 x 2 = 2
>> Transport-type: tcp
>> Bricks:
>> Brick1: fsnode2.ibnet:/data/glusterfs/r2vol/brick1/brick
>> Brick2: fsnode4.ibnet:/data/glusterfs/r2vol/brick1/brick
>> Options Reconfigured:
>> nfs.disable: on
>>
>
>
>> Volume Name: r3vol
>> Type: Replicate
>> Volume ID: b0f64c28-57e1-4b9d-946b-26ed6b499f29
>> Status: Started
>> Snapshot Count: 0
>> Number of Bricks: 1 x 3 = 3
>> Transport-type: tcp
>> Bricks:
>> Brick1: fsnode2.ibnet:/data/glusterfs/r3vol/brick1/brick
>> Brick2: fsnode4.ibnet:/data/glusterfs/r3vol/brick1/brick
>> Brick3: fsnode6.ibnet:/data/glusterfs/r3vol/brick1/brick
>> Options Reconfigured:
>> nfs.disable: on
>
>
>
> *Client *is also gluster 3.12.6, Centos 7.3 virtual machine, *FUSE mount*
>
>> root@centos7u3-nogdesktop2 ~ $ mount |grep gluster
>> gluster-host.ibnet:/r2vol on /mnt/gluster/r2 type fuse.glusterfs
>> (rw,relatime,user_id=0,group_id=0,default_permissions,
>> allow_other,max_read=131072)
>> gluster-host.ibnet:/r3vol on /mnt/gluster/r3 type fuse.glusterfs
>> (rw,relatime,user_id=0,group_id=0,default_permissions,
>> allow_other,max_read=131072)
>
>
>
> *The problem *is that there is a significant performance loss with
> smaller block sizes. For example:
>
> *4K block size*
> [replica 3 volume]
> root@centos7u3-nogdesktop2 ~ $ dd if=/dev/zero
> of=/mnt/gluster/r3/file$RANDOM bs=4096 count=262144
> 262144+0 records in
> 262144+0 records out
> 1073741824 bytes (1.1 GB) copied, 11.2207 s, *95.7 MB/s*
>
> [replica 2 volume]
> root@centos7u3-nogdesktop2 ~ $ dd if=/dev/zero
> of=/mnt/gluster/r2/file$RANDOM bs=4096 count=262144
> 262144+0 records in
> 262144+0 records out
> 1073741824 bytes (1.1 GB) copied, 12.0149 s, *89.4 MB/s*
>
> *512K block size*
> [replica 3 volume]
> root@centos7u3-nogdesktop2 ~ $ dd if=/dev/zero
> of=/mnt/gluster/r3/file$RANDOM bs=512K count=2048
> 2048+0 records in
> 2048+0 records out
> 1073741824 bytes (1.1 GB) copied, 5.27207 s, *204 MB/s*
>
> [replica 2 volume]
> root@centos7u3-nogdesktop2 ~ $ dd if=/dev/zero
> of=/mnt/gluster/r2/file$RANDOM bs=512K count=2048
> 2048+0 records in
> 2048+0 records out
> 1073741824 bytes (1.1 GB) copied, 4.22321 s, *254 MB/s*
>
> With bigger block size It's still not where I expect it to be, but at
> least it starts to make some sense.
>
> I've been trying to solve this for a very long time with no luck.
> I've already tried both kernel tuning (different 'tuned' profiles and the
> ones recommended in the "Linux Kernel Tuning" section) and tweaking gluster
> volume options, including write-behind/flush-behind/
> write-behind-window-size.
> The latter, to my surprise, didn't make any difference. 'Cause at first I
> thought it was the buffering issue but it turns out it does buffer writes,
> just not very efficient (well at least what it looks like in the *gluster
> profile output*)
>
> root@fsnode2 ~ $ gluster volume profile r3vol info clear
>> ...
>> Cleared stats.
>
>
> root@centos7u3-nogdesktop2 ~ $ dd if=/dev/zero
>> of=/mnt/gluster/r3/file$RANDOM bs=4096 count=262144
>> 262144+0 records in
>> 262144+0 records out
>> 1073741824 bytes (1.1 GB) copied, 10.9743 s, 

Re: [Gluster-users] JBOD / ZFS / Flash backed

[Alex Crow]

On 09/04/18 22:15, Vincent Royer wrote:

Thanks,

The 3 servers are new Lenovo units with redundant PS backed by two 
huge UPS units (on for each bank of power supplies).  I think the 
chances of losing two nodes is incredibly slim, and in that case a 
Disaster Recovery from offsite backups would be reasonable.


My requirements are about 2TB, highly available (so that I can reboot 
one of the 3 servers without taking down services).


Beyond that my focus is high performance for small I/O.
This can be a difficult case for GlusterFS, if you mean "small files", 
as the metadata lookups are relatively costly (no separate MDS with 
in-memory or memory cached database). It's ideally placed for large 
files, and small I/O within those files should be OK. just speaking from 
experience - should be fine for VMs with such loads, especially if you 
shard.


So I could do a single 2TB SSD per server, or two, or many more if 
that is "what is required".  But I don't want to waste money...


Resilience is never a waste. Skimping may well prove to be a waste of 
*your time* when you get woken up at 3am and have to fix a downed 
system. Your call entirely. I'm too old for that kind of thing, so I 
tend to push for both per-server and per-cluster redundancy. It usually 
gets approved after something "unexpected" happens the first time.


Gluster and ZFS will be fine with onboard controllers. If you have 
enough ports you'll be just fine. If you need more buy HBA's to stick in 
your PCIe slots, M1015s and M1115s on ebay perform very well and are 
still dirt cheap.


So are you using ZFS to get compression and checksumming down to the 
disk platter level? ZFS will give some gains in performance with 
compressible data and corruption protection, but, don't bother with 
dedup, I've tried it on 3 distributed filesystems and it bought less 
than 3% capacity and slammed performance. If you don't need either 
feature just stick with XFS for single-disk or software-RAIDed mirrors 
per brick. My personal opinion would be do a ZFS mirror of two SSDs per 
server, per brick, ie in your initial case, 2x2TB SSD per box in ZFS 
mirror. You can add more mirror sets later to add additional bricks.




I like the idea of forgoing the RAID cards as they are quite 
expensive, especially the capacitor backed ones.  The onboard 
controller can handle JBOD just fine, if Gluster is OK with it!


As I also said, if said expensive card dies, and you don't have another 
one in stock, you will effectively have lost everything on that server 
until you can source a new one (or even /if/ you can).


Use the power of the software to get where you need to be, the tools are 
there...


Alex

--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300.
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] glusterfs disperse volume input output error

[Alexey Shcherbakov]

Hi,


Could you help me?


i have a problem with file on disperse volume. When i try to read this from 
mount point i recieve error,

# md5sum /mnt/glfs/vmfs/slake-test-bck-m1-d1.qcow2
md5sum: /mnt/glfs/vmfs/slake-test-bck-m1-d1.qcow2: Input/output error


Configuration and status of volume is:


# gluster volume info vol1

Volume Name: vol1
Type: Disperse
Volume ID: a7d52933-fccc-4b07-9c3b-5b92f398aa79
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (13 + 2) = 15
Transport-type: tcp
Bricks:
Brick1: glfs-node11.local:/data1/bricks/brick1
Brick2: glfs-node12.local:/data1/bricks/brick1
Brick3: glfs-node13.local:/data1/bricks/brick1
Brick4: glfs-node14.local:/data1/bricks/brick1
Brick5: glfs-node15.local:/data1/bricks/brick1
Brick6: glfs-node16.local:/data1/bricks/brick1
Brick7: glfs-node17.local:/data1/bricks/brick1
Brick8: glfs-node18.local:/data1/bricks/brick1
Brick9: glfs-node19.local:/data1/bricks/brick1
Brick10: glfs-node20.local:/data1/bricks/brick1
Brick11: glfs-node21.local:/data1/bricks/brick1
Brick12: glfs-node22.local:/data1/bricks/brick1
Brick13: glfs-node23.local:/data1/bricks/brick1
Brick14: glfs-node24.local:/data1/bricks/brick1
Brick15: glfs-node25.local:/data1/bricks/brick1
Options Reconfigured:
transport.address-family: inet
performance.readdir-ahead: on
nfs.disable: on


# gluster volume status vol1
Status of volume: vol1
Gluster process TCP Port  RDMA Port  Online  Pid
--
Brick glfs-node11.local:/data1/bricks/brick
1   49152 0  Y   1781
Brick glfs-node12.local:/data1/bricks/brick
1   49152 0  Y   3026
Brick glfs-node13.local:/data1/bricks/brick
1   49152 0  Y   1991
Brick glfs-node14.local:/data1/bricks/brick
1   49152 0  Y   2029
Brick glfs-node15.local:/data1/bricks/brick
1   49152 0  Y   1745
Brick glfs-node16.local:/data1/bricks/brick
1   49152 0  Y   1841
Brick glfs-node17.local:/data1/bricks/brick
1   49152 0  Y   3597
Brick glfs-node18.local:/data1/bricks/brick
1   49152 0  Y   2035
Brick glfs-node19.local:/data1/bricks/brick
1   49152 0  Y   1785
Brick glfs-node20.local:/data1/bricks/brick
1   49152 0  Y   1755
Brick glfs-node21.local:/data1/bricks/brick
1   49152 0  Y   1772
Brick glfs-node22.local:/data1/bricks/brick
1   49152 0  Y   1757
Brick glfs-node23.local:/data1/bricks/brick
1   49152 0  Y   1825
Brick glfs-node24.local:/data1/bricks/brick
1   49152 0  Y   1963
Brick glfs-node25.local:/data1/bricks/brick
1   49152 0  Y   2376
Self-heal Daemon on localhost   N/A   N/AY   2018
Self-heal Daemon on glfs-node15.local   N/A   N/AY   38261
Self-heal Daemon on glfs-node16.local   N/A   N/AY   36005
Self-heal Daemon on glfs-node12.local   N/A   N/AY   25785
Self-heal Daemon on glfs-node27.local   N/A   N/AY   13248
Self-heal Daemon on glfs-node19.local   N/A   N/AY   38535
Self-heal Daemon on glfs-node18.local   N/A   N/AY   21067
Self-heal Daemon on glfs-node21.local   N/A   N/AY   5926
Self-heal Daemon on glfs-node22.local   N/A   N/AY   12980
Self-heal Daemon on glfs-node23.local   N/A   N/AY   8368
Self-heal Daemon on glfs-node26.local   N/A   N/AY   8268
Self-heal Daemon on glfs-node25.local   N/A   N/AY   7872
Self-heal Daemon on glfs-node17.local   N/A   N/AY   15884
Self-heal Daemon on glfs-node11.local   N/A   N/AY   36075
Self-heal Daemon on glfs-node24.local   N/A   N/AY   37905
Self-heal Daemon on glfs-node30.local   N/A   N/AY   31820
Self-heal Daemon on glfs-node14.local   N/A   N/AY   3236
Self-heal Daemon on glfs-node13.local   N/A   N/AY   25817
Self-heal Daemon on glfs-node29.local   N/A   N/AY   21261
Self-heal Daemon on glfs-node28.local   N/A   N/AY   32641

Task Status of Volume vol1

[Gluster-users] Question concerning TLS encryption of network traffic

[David Spisla]

Hello Gluster Community,

according to that set steps I have configured network encryption for
management and I/O traffic:
https://www.cyberciti.biz/faq/how-to-enable-tlsssl-encryption-with-glusterfs-storage-cluster-on-linux/

I have chose the option for self-signed certificates, so each of the nodes
has its own certificate and all of them are stored in the file glusterfs.ca.
Each node in my cluster has a copy of that file.

Everything is working fine.

I set the volume option "auth.ssl-allow" with "*", but I am not sure what
does this exactly means?

1. Does it mean, that only all clients which are listed in glusterfs.ca has
access to the volume?
or
2. Does it mean, that any TLS authenticated client can access the volume
(maybe a client which is not in the glusterfs.ca list)?

Regards
David Spisla
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] wrong size displayed with df after upgrade to 3.12.6

[Paolo Margara]

Dear all,

I encountered the same issue, I saw that this is fixed in 3.12.7 but I
cannot find this release in the main repo (centos storage SIG), only in
the test one.

What is the expectation to see this release available into the main repo?


Greetings,

    Paolo


Il 09/03/2018 10:41, Stefan Solbrig ha scritto:
> Dear Nithya,
>
> Thank you so much.  This fixed the problem immediately. 
>
> best wishes,
> Stefan
>
> -- 
> Dr. Stefan Solbrig
> Universität Regensburg, Fakultät für Physik,
> 93040 Regensburg, Germany
> Tel +49-941-943-2097
>
>
>> Am 09.03.2018 um 09:47 schrieb Nithya Balachandran
>> >:
>>
>> Hi Stefan,
>>
>> There is a known issue with gluster 3.12.x builds (see [1]) so you
>> may be running into this. Please take a look at [1] and try out the
>> workaround provided in the comments.
>>
>>
>> Regards,
>> Nithya
>>
>>
>>
>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1517260
>> 
>>
>> On 9 March 2018 at 13:37, Stefan Solbrig > > wrote:
>>
>> Dear all,
>>
>> I have a problem with df  after I upgraded from 3.12.4 to 3.12.6
>> All four bricks are shown als online, and all bricks are being used.
>> gluster v stats shows the correct sizes for all devices.
>> However,  df  does not show the correct glusterfs volume size.
>> It seems to me that it "forgets" one brick.  Although all bricks
>> are used when I'm writing files.
>>
>> best wishes,
>> Stefan
>>
>>
>> details:
>>
>> [root@pcph00131 glusterfs]# gluster v status all detail
>> Status of volume: glurch
>> 
>> --
>> Brick                : Brick qloginx:/gl/lv1osb06/glurchbrick
>> Total Disk Space     : 236.4TB
>> 
>> --
>> Brick                : Brick qloginx:/gl/lv2osb06/glurchbrick
>> Total Disk Space     : 177.3TB
>> 
>> --
>> Brick                : Brick gluster2x:/gl/lv3osb06/glurchbrick
>> Total Disk Space     : 177.3TB
>> 
>> --
>> Brick                : Brick gluster2x:/gl/lv4osb06/glurchbrick
>> Total Disk Space     : 231.0TB
>>
>> [root@pcph00131 glusterfs]# mount | grep gluster
>> qloginx:/glurch.rdma on /glurch type fuse.glusterfs
>> (rw,relatime,user_id=0,group_id=0,allow_other,max_read=131072)
>>
>> [root@pcph00131 glusterfs]# df -h /glurch
>> Filesystem            Size  Used Avail Use% Mounted on
>> qloginx:/glurch.rdma  618T  363T  256T  59% /glurch
>>
>> ## df -h should show ~ 800T but only shows ~600T.
>>
>> --
>> Dr. Stefan Solbrig
>> Universität Regensburg, Fakultät für Physik,
>> 93040 Regensburg, Germany
>> Tel +49-941-943-2097
>>
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org 
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>> 
>>
>>
>
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] ETA for 3.10.12 (was "Planned for the 30th of Mar, 2018")

[Marco Lorenzo Crociani]

On 09/04/2018 21:36, Shyam Ranganathan wrote:

On 04/09/2018 04:48 AM, Marco Lorenzo Crociani wrote:

On 06/04/2018 19:33, Shyam Ranganathan wrote:

Hi,

We postponed this and I did not announce this to the lists. The number
of bugs fixed against 3.10.12 is low, and I decided to move this to the
30th of Apr instead.

Is there a specific fix that you are looking for in the release?



Hi,
yes, it's this: https://review.gluster.org/19730
https://bugzilla.redhat.com/show_bug.cgi?id=1442983


We will roll out 3.10.12 including this fix in a few days, we have a
3.12 build and release tomorrow, hence looking to get 3.10 done by this
weekend.

Thanks for your patience!



Hi,
ok thanks, stand by for the release!

--
Marco Crociani
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] issues with replicating data to a new brick

[Bernhard Dübi]

Hello everybody,

I have some kind of a situation here

I want to move some volumes to new hosts. the idea is to add the new
bricks to the volume, sync and then drop the old bricks.

starting point is:


Volume Name: Server_Monthly_02
Type: Replicate
Volume ID: 0ada8e12-15f7-42e9-9da3-2734b04e04e9
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: chastcvtprd04:/data/glusterfs/Server_Monthly/2I-1-40/brick
Brick2: chglbcvtprd04:/data/glusterfs/Server_Monthly/2I-1-40/brick
Options Reconfigured:
features.scrub: Inactive
features.bitrot: off
nfs.disable: on
auth.allow: 
127.0.0.1,10.30.28.43,10.30.28.44,10.30.28.17,10.30.28.18,10.8.13.132,10.30.28.30,10.30.28.31
performance.readdir-ahead: on
diagnostics.latency-measurement: on
diagnostics.count-fop-hits: on


root@chastcvtprd04:~# cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.3 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.3 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/;
SUPPORT_URL="http://help.ubuntu.com/;
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/;
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
root@chastcvtprd04:~# uname -a
Linux chastcvtprd04 4.4.0-109-generic #132-Ubuntu SMP Tue Jan 9
19:52:39 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
root@chastcvtprd04:~# dpkg -l | grep gluster
ii  glusterfs-client 3.8.15-ubuntu1~xenial1
   amd64clustered file-system (client package)
ii  glusterfs-common 3.8.15-ubuntu1~xenial1
   amd64GlusterFS common libraries and translator
modules
ii  glusterfs-server 3.8.15-ubuntu1~xenial1
   amd64clustered file-system (server package)



root@chastcvtprd04:~# df -h  /data/glusterfs/Server_Monthly/2I-1-40/brick
Filesystem  Size  Used Avail Use% Mounted on
/dev/bcache47   7.3T  7.3T   45G 100% /data/glusterfs/Server_Monthly/2I-1-40


then I add the new brick


Volume Name: Server_Monthly_02
Type: Replicate
Volume ID: 0ada8e12-15f7-42e9-9da3-2734b04e04e9
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: chastcvtprd04:/data/glusterfs/Server_Monthly/2I-1-40/brick
Brick2: chglbcvtprd04:/data/glusterfs/Server_Monthly/2I-1-40/brick
Brick3: chglbglsprd02:/data/glusterfs/Server_Monthly/1I-1-51/brick
Options Reconfigured:
features.scrub: Inactive
features.bitrot: off
nfs.disable: on
auth.allow: 
127.0.0.1,10.30.28.43,10.30.28.44,10.30.28.17,10.30.28.18,10.8.13.132,10.30.28.30,10.30.28.31
performance.readdir-ahead: on
diagnostics.latency-measurement: on
diagnostics.count-fop-hits: on


root@chglbglsprd02:~# cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.4 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.4 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/;
SUPPORT_URL="http://help.ubuntu.com/;
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/;
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
root@chglbglsprd02:~# uname -a
Linux chglbglsprd02 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12
21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
root@chglbglsprd02:~# dpkg -l | grep gluster
ii  glusterfs-client 3.8.15-ubuntu1~xenial1
 amd64clustered file-system (client package)
ii  glusterfs-common 3.8.15-ubuntu1~xenial1
 amd64GlusterFS common libraries and translator
modules
ii  glusterfs-server 3.8.15-ubuntu1~xenial1
 amd64clustered file-system (server package)


then healing kicks in and the cluster starts copying data to the new brick
unfortunately after a while it starts complaining

[2018-04-10 14:39:32.057443] E [MSGID: 113072]
[posix.c:3457:posix_writev] 0-Server_Monthly_02-posix: write failed:
offset 0, [No space left on device]
[2018-04-10 14:39:32.057538] E [MSGID: 115067]
[server-rpc-fops.c:1346:server_writev_cbk] 0-Server_Monthly_02-server:
22835126: WRITEV 0 (48949669-ba1c-4735-b83c-71340f1bb64f) ==> (No
space left on device) [No space left on device]


root@chglbglsprd02:~# df -h /data/glusterfs/Server_Monthly/1I-1-51/brick
Filesystem  Size  Used Avail Use% Mounted on
/dev/sdaq   7.3T  7.3T   20K 100% /data/glusterfs/Server_Monthly/1I-1-51



there's no other I/O going on on this volume, so the copy process
should be straight forward
BUT I noticed that there are a lot of sparse files on this volume



Any ideas on how to make it work?
If you need more details, please let me known and I'll try to make
them available


Kind Regards
Bernhard
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Turn off replication

[Karthik Subrahmanya]

On Wed, Apr 11, 2018 at 7:38 PM, Jose Sanchez  wrote:

> Hi Karthik
>
> Looking at the information you have provided me, I would like to make sure
> that I’m running the right commands.
>
> 1.   gluster volume heal scratch info
>
If the count is non zero, trigger the heal and wait for heal info count to
become zero.

> 2. gluster volume remove-brick scratch *replica 1 *
> gluster02ib:/gdata/brick1/scratch gluster02ib:/gdata/brick2/scratch force
>
3.  gluster volume add-brick* “#"* scratch gluster02ib:/gdata/brick1/
> scratch gluster02ib:/gdata/brick2/scratch
>
>
> Based on the configuration I have, Brick 1 from Node A and B are tide
> together and Brick 2 from Node A and B are also tide together. Looking at
> your remove command (step #2), it seems that you want me to remove Brick 1
> and 2 from Node B (gluster02ib). is that correct? I thought the data was
> distributed in bricks 1 between nodes A and B) and duplicated on Bricks 2
> (node A and B).
>
Data is duplicated between bricks 1 of nodes A & B and bricks 2 of nodes A
& B and data is distributed between these two pairs.
You need not always remove the bricks 1 & 2 from node B itself. The idea
here is to keep one copy from both the replica pairs.

>
> Also when I add the bricks back to gluster, do I need to specify if it is
> distributed or replicated?? and Do i need a configuration #?? for example
> on your command (Step #2) you have “replica 1” when remove bricks, do I
> need to do the same when adding the nodes back ?
>
No. You just need to erase the data on those bricks and add those bricks
back to the volume. The previous remove-brick command will make the volume
plain distribute. Then simply adding the bricks without specifying any "#"
will expand the volume as a plain distribute volue.

>
> Im planning on moving with this changes in few days. At this point each
> brick has 14tb and adding bricks 1 from node A and B, i have a total of
> 28tb, After doing all the process, (removing and adding bricks) I should be
> able to see a total of 56Tb right ?
>
Yes after all these you will have 56TB in total.
After adding the bricks, do volume rebalance, so that the data which were
present previously, will be moved to the correct bricks.

HTH,
Karthik

>
> Thanks
>
> Jose
>
>
>
>
> -
> Jose Sanchez
> Systems/Network Analyst 1
> Center of Advanced Research Computing
> 1601 Central Ave
> .
> MSC 01 1190
> Albuquerque, NM 87131-0001
> carc.unm.edu
> 575.636.4232
>
> On Apr 7, 2018, at 8:29 AM, Karthik Subrahmanya 
> wrote:
>
> Hi Jose,
>
> Thanks for providing the volume info. You have 2 subvolumes. Data is
> replicated within the bricks of that subvolumes.
> First one consisting of Node A's brick1 & Node B's brick1 and the second
> one consisting of Node A's brick2 and Node B's brick2.
> You don't have the same data on all the 4 bricks. Data are distributed
> between these two subvolumes.
> To remove the replica you can use the command
> gluster volume remove-brick scratch replica 1 gluster02ib:/gdata/brick1/
> scratch gluster02ib:/gdata/brick2/scratch force
> So you will have one copy of data present from both the distributes.
> Before doing this make sure "gluster volume heal scratch info" value is
> zero. So copies you retain will have the correct data.
> After the remove-brick erase the data from the backend.
> Then you can expand the volume by following the steps at [1].
>
> [1] https://docs.gluster.org/en/latest/Administrator%
> 20Guide/Managing%20Volumes/#expanding-volumes
>
> Regards,
> Karthik
>
> On Fri, Apr 6, 2018 at 11:39 PM, Jose Sanchez 
> wrote:
>
>> Hi Karthik
>>
>> this is our configuration,  is 2x2 =4 , they are all replicated , each
>> brick has 14tb. we have 2 nodes A and B, each one with brick 1 and 2.
>>
>> Node A  (replicated A1 (14tb) and B1 (14tb) ) same with node B
>> (Replicated A2 (14tb) and B2 (14tb)).
>>
>> Do you think we need to degrade the node first before removing it. i
>> believe the same copy of data is on all 4 bricks, we would like to keep one
>> of them, and add the other bricks as extra space
>>
>> Thanks for your help on this
>>
>> Jose
>>
>>
>>
>>
>>
>> [root@gluster01 ~]# gluster volume info scratch
>>
>> Volume Name: scratch
>> Type: Distributed-Replicate
>> Volume ID: 23f1e4b1-b8e0-46c3-874a-58b4728ea106
>> Status: Started
>> Snapshot Count: 0
>> Number of Bricks: 2 x 2 = 4
>> Transport-type: tcp,rdma
>> Bricks:
>> Brick1: gluster01ib:/gdata/brick1/scratch
>> Brick2: gluster02ib:/gdata/brick1/scratch
>> Brick3: gluster01ib:/gdata/brick2/scratch
>> Brick4: gluster02ib:/gdata/brick2/scratch
>> Options Reconfigured:
>> performance.readdir-ahead: on
>> nfs.disable: on
>>
>> [root@gluster01 ~]# gluster volume status all
>> Status of volume: scratch
>> Gluster process TCP Port  RDMA Port  Online
>> Pid
>>