Re: [Gluster-users] Finding performance bottlenecks

[Thing]

Hi,

So is the KVM or Vmware as the host(s)?  I basically have the same setup ie
3 x 1TB "raid1" nodes and VMs, but 1gb networking.  I do notice with vmware
using NFS disk was pretty slow (40% of a single disk) but this was over 1gb
networking which was clearly saturating.  Hence I am moving to KVM to use
glusterfs hoping for better performance and bonding, it will be interesting
to see which host type runs faster.

Which operating system is gluster on?

Did you do iperf between all nodes?





On 1 May 2018 at 00:14, Tony Hoyle  wrote:

> Hi
>
> I'm trying to setup a 3 node gluster, and am hitting huge performance
> bottlenecks.
>
> The 3 servers are connected over 10GB and glusterfs is set to create a 3
> node replica.
>
> With a single VM performance was poor, but I could have lived with it.
>
> I tried to stress it by putting copies of a bunch of VMs on the servers
> and seeing what happened with parallel nodes..  network load never broke
> 13Mbps and disk load peaked at under 1Mbps.  VMs were so slow that
> services timed out during boot causing failures.
>
> Checked the network with iperf and it reached 9.7Gb so the hardware is
> fine.. it just seems that for some reason glusterfs just isn't using it.
>
> gluster volume top gv0 read-perf shows 0Mbps for all files, although I'm
> not sure whether the command is working.
>
> There's probably a magic setting somewhere, but I've been a couple of
> days trying to find it now..
>
> Tony
>
> stats:
>Block Size:512b+1024b+
> 2048b+
>  No. of Reads:0 2
>  0
> No. of Writes:   40   141
>399
>
>Block Size:   4096b+8192b+
> 16384b+
>  No. of Reads:  17324
>  4
> No. of Writes:18351  5049
>   2478
>
>Block Size:  32768b+   65536b+
> 131072b+
>  No. of Reads:   12   113
>  0
> No. of Writes: 1640   648
>200
>
>Block Size: 262144b+  524288b+
> 1048576b+
>  No. of Reads:0 0
>  0
> No. of Writes:  32955
>139
>
>Block Size:2097152b+
>  No. of Reads:0
> No. of Writes:1
>  %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
>Fop
>  -   ---   ---   ---   
>   
>   0.00   0.00 us   0.00 us   0.00 us 41
> RELEASE
>   0.00   0.00 us   0.00 us   0.00 us  6
> RELEASEDIR
>   0.00   3.43 us   2.65 us   4.10 us  6
> OPENDIR
>   0.00 217.85 us 217.85 us 217.85 us  1
> SETATTR
>   0.00  66.38 us  49.47 us  80.57 us  4
>   SEEK
>   0.00 394.18 us 394.18 us 394.18 us  1
> FTRUNCATE
>   0.00 116.68 us  29.88 us 186.25 us 16
> GETXATTR
>   0.00 397.32 us 267.18 us 540.38 us 10
> XATTROP
>   0.00 553.09 us 244.97 us1242.98 us 12
> READDIR
>   0.00 201.60 us  69.61 us 744.71 us 41
>   OPEN
>   0.00 734.96 us  75.05 us   37399.38 us328
>   READ
>   0.011750.65 us  33.99 us  750562.48 us591
> LOOKUP
>   0.022972.84 us  30.72 us  788018.47 us496
> STATFS
>   0.03   10951.33 us  35.36 us  695155.13 us166
>   STAT
>   0.422574.98 us 208.73 us 1710282.73 us  11877
> FXATTROP
>   2.80 609.20 us 468.51 us  321422.91 us 333946
> RCHECKSUM
>   5.04 548.76 us  14.83 us 76288179.46 us 668188
> INODELK
>  18.46  149940.70 us  13.59 us 79966278.04 us   8949
> FINODELK
>  20.04  395073.91 us  84.99 us 3835355.67 us   3688
>  FSYNC
>  53.17  131171.66 us  85.76 us 3838020.34 us  29470
>  WRITE
>   0.00   0.00 us   0.00 us   0.00 us   7238
> UPCALL
>   0.00   0.00 us   0.00 us   0.00 us   7238
> CI_IATT
>
> Duration: 1655 seconds
>Data Read: 8804864 bytes
> Data Written: 612756480 bytes
>
> config:
> Volume Name: gv0
> Type: Replicate
> Volume ID: a0b6635a-ae48-491b-834a-08e849e87642
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x 3 = 3
> Transport-type: tcp
> Bricks:
> Brick1: barbelith10:/tank/vmdata/gv0
> Brick2: rommel10:/tank/vmdata/gv0
> Brick3: panzer10:/tank/vmdata/gv0
> Options Reconfigured:
> diagnostics.count-fop-hits: on
> diagnostics.latency-measurement: on
> features.cache-invalidation: on
> nfs.disable: on
> cluster.server-quorum-type: server
> cluster.quorum-type: auto
> 

Re: [Gluster-users] Turn off replication

[Jose Sanchez]

Hi All

We were able to get all 4 bricks are distributed , we can see the right amount 
of space. but we have been rebalancing since 4 days ago for 16Tb. and still 
only 8tb. is there a way to speed up. there is also data we can remove from it 
to speed it up, but what is the best procedures removing data , is it from the 
Gluster main export point or going on each brick and remove it . We would like 
to stop rebalancing , delete the data and rebalancing again. 

is there a down side, doing this, What happens with Gluster missing data when 
rebalancing?

Thanks

Jose






-
Jose Sanchez
Systems/Network Analyst 1
Center of Advanced Research Computing
1601 Central Ave.
MSC 01 1190
Albuquerque, NM 87131-0001
carc.unm.edu 
575.636.4232

> On Apr 27, 2018, at 4:16 AM, Hari Gowtham  wrote:
> 
> Hi Jose,
> 
> Why are all the bricks visible in volume info if the pre-validation
> for add-brick failed? I suspect that the remove brick wasn't done
> properly.
> 
> You can provide the cmd_history.log to verify this. Better to get the
> other log messages.
> 
> Also I need to know what are the bricks that were actually removed,
> the command used and its output.
> 
> On Thu, Apr 26, 2018 at 3:47 AM, Jose Sanchez  wrote:
>> Looking at the logs , it seems that it is trying to add using the same port
>> was assigned for gluster01ib:
>> 
>> 
>> Any Ideas??
>> 
>> Jose
>> 
>> 
>> 
>> [2018-04-25 22:08:55.169302] I [MSGID: 106482]
>> [glusterd-brick-ops.c:447:__glusterd_handle_add_brick] 0-management:
>> Received add brick req
>> [2018-04-25 22:08:55.186037] I [run.c:191:runner_log]
>> (-->/usr/lib64/glusterfs/3.8.15/xlator/mgmt/glusterd.so(+0x33045)
>> [0x7f5464b9b045]
>> -->/usr/lib64/glusterfs/3.8.15/xlator/mgmt/glusterd.so(+0xcbd85)
>> [0x7f5464c33d85] -->/lib64/libglusterfs.so.0(runner_log+0x115)
>> [0x7f54704cf1e5] ) 0-management: Ran script:
>> /var/lib/glusterd/hooks/1/add-brick/pre/S28Quota-enable-root-xattr-heal.sh
>> --volname=scratch --version=1 --volume-op=add-brick
>> --gd-workdir=/var/lib/glusterd
>> [2018-04-25 22:08:55.309534] I [MSGID: 106143]
>> [glusterd-pmap.c:250:pmap_registry_bind] 0-pmap: adding brick
>> /gdata/brick1/scratch on port 49152
>> [2018-04-25 22:08:55.309659] I [MSGID: 106143]
>> [glusterd-pmap.c:250:pmap_registry_bind] 0-pmap: adding brick
>> /gdata/brick1/scratch.rdma on port 49153
>> [2018-04-25 22:08:55.310231] E [MSGID: 106005]
>> [glusterd-utils.c:4877:glusterd_brick_start] 0-management: Unable to start
>> brick gluster02ib:/gdata/brick1/scratch
>> [2018-04-25 22:08:55.310275] E [MSGID: 106074]
>> [glusterd-brick-ops.c:2493:glusterd_op_add_brick] 0-glusterd: Unable to add
>> bricks
>> [2018-04-25 22:08:55.310304] E [MSGID: 106123]
>> [glusterd-mgmt.c:294:gd_mgmt_v3_commit_fn] 0-management: Add-brick commit
>> failed.
>> [2018-04-25 22:08:55.310316] E [MSGID: 106123]
>> [glusterd-mgmt.c:1427:glusterd_mgmt_v3_commit] 0-management: Commit failed
>> for operation Add brick on local node
>> [2018-04-25 22:08:55.310330] E [MSGID: 106123]
>> [glusterd-mgmt.c:2018:glusterd_mgmt_v3_initiate_all_phases] 0-management:
>> Commit Op Failed
>> [2018-04-25 22:09:11.678141] E [MSGID: 106452]
>> [glusterd-utils.c:6064:glusterd_new_brick_validate] 0-management: Brick:
>> gluster02ib:/gdata/brick1/scratch not available. Brick may be containing or
>> be contained by an existing brick
>> [2018-04-25 22:09:11.678184] W [MSGID: 106122]
>> [glusterd-mgmt.c:188:gd_mgmt_v3_pre_validate_fn] 0-management: ADD-brick
>> prevalidation failed.
>> [2018-04-25 22:09:11.678200] E [MSGID: 106122]
>> [glusterd-mgmt-handler.c:337:glusterd_handle_pre_validate_fn] 0-management:
>> Pre Validation failed on operation Add brick
>> [root@gluster02 glusterfs]# gluster volume status scratch
>> Status of volume: scratch
>> Gluster process TCP Port  RDMA Port  Online  Pid
>> --
>> Brick gluster01ib:/gdata/brick1/scratch 49152 49153  Y
>> 1819
>> Brick gluster01ib:/gdata/brick2/scratch 49154 49155  Y
>> 1827
>> Brick gluster02ib:/gdata/brick1/scratch N/A   N/AN   N/A
>> 
>> 
>> 
>> Task Status of Volume scratch
>> --
>> There are no active volume tasks
>> 
>> 
>> 
>> [root@gluster02 glusterfs]#
>> 
>> 
>> 
>> On Apr 25, 2018, at 3:23 PM, Jose Sanchez  wrote:
>> 
>> Hello Karthik
>> 
>> 
>> Im having trouble adding the two bricks back online.  Any help is
>> appreciated
>> 
>> thanks
>> 
>> 
>> when i try to add-brick command this is what i get
>> 
>> [root@gluster01 ~]# gluster volume add-brick scratch
>> gluster02ib:/gdata/brick2/scratch/
>> volume add-brick: failed: Pre Validation failed on gluster02ib. Brick:
>> gluster02ib:/gdata/brick2/scratch not available. Brick may be containing or
>> 

[Gluster-users] Finding performance bottlenecks

[Tony Hoyle]

Hi

I'm trying to setup a 3 node gluster, and am hitting huge performance
bottlenecks.

The 3 servers are connected over 10GB and glusterfs is set to create a 3
node replica.

With a single VM performance was poor, but I could have lived with it.

I tried to stress it by putting copies of a bunch of VMs on the servers
and seeing what happened with parallel nodes..  network load never broke
13Mbps and disk load peaked at under 1Mbps.  VMs were so slow that
services timed out during boot causing failures.

Checked the network with iperf and it reached 9.7Gb so the hardware is
fine.. it just seems that for some reason glusterfs just isn't using it.

gluster volume top gv0 read-perf shows 0Mbps for all files, although I'm
not sure whether the command is working.

There's probably a magic setting somewhere, but I've been a couple of
days trying to find it now..

Tony

stats:
   Block Size:512b+1024b+
2048b+
 No. of Reads:0 2
 0
No. of Writes:   40   141
   399

   Block Size:   4096b+8192b+
16384b+
 No. of Reads:  17324
 4
No. of Writes:18351  5049
  2478

   Block Size:  32768b+   65536b+
131072b+
 No. of Reads:   12   113
 0
No. of Writes: 1640   648
   200

   Block Size: 262144b+  524288b+
1048576b+
 No. of Reads:0 0
 0
No. of Writes:  32955
   139

   Block Size:2097152b+
 No. of Reads:0
No. of Writes:1
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
   Fop
 -   ---   ---   ---   
  
  0.00   0.00 us   0.00 us   0.00 us 41
RELEASE
  0.00   0.00 us   0.00 us   0.00 us  6
RELEASEDIR
  0.00   3.43 us   2.65 us   4.10 us  6
OPENDIR
  0.00 217.85 us 217.85 us 217.85 us  1
SETATTR
  0.00  66.38 us  49.47 us  80.57 us  4
  SEEK
  0.00 394.18 us 394.18 us 394.18 us  1
FTRUNCATE
  0.00 116.68 us  29.88 us 186.25 us 16
GETXATTR
  0.00 397.32 us 267.18 us 540.38 us 10
XATTROP
  0.00 553.09 us 244.97 us1242.98 us 12
READDIR
  0.00 201.60 us  69.61 us 744.71 us 41
  OPEN
  0.00 734.96 us  75.05 us   37399.38 us328
  READ
  0.011750.65 us  33.99 us  750562.48 us591
LOOKUP
  0.022972.84 us  30.72 us  788018.47 us496
STATFS
  0.03   10951.33 us  35.36 us  695155.13 us166
  STAT
  0.422574.98 us 208.73 us 1710282.73 us  11877
FXATTROP
  2.80 609.20 us 468.51 us  321422.91 us 333946
RCHECKSUM
  5.04 548.76 us  14.83 us 76288179.46 us 668188
INODELK
 18.46  149940.70 us  13.59 us 79966278.04 us   8949
FINODELK
 20.04  395073.91 us  84.99 us 3835355.67 us   3688
 FSYNC
 53.17  131171.66 us  85.76 us 3838020.34 us  29470
 WRITE
  0.00   0.00 us   0.00 us   0.00 us   7238
UPCALL
  0.00   0.00 us   0.00 us   0.00 us   7238
CI_IATT

Duration: 1655 seconds
   Data Read: 8804864 bytes
Data Written: 612756480 bytes

config:
Volume Name: gv0
Type: Replicate
Volume ID: a0b6635a-ae48-491b-834a-08e849e87642
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: barbelith10:/tank/vmdata/gv0
Brick2: rommel10:/tank/vmdata/gv0
Brick3: panzer10:/tank/vmdata/gv0
Options Reconfigured:
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
features.cache-invalidation: on
nfs.disable: on
cluster.server-quorum-type: server
cluster.quorum-type: auto
network.remote-dio: enable
cluster.eager-lock: enable
performance.stat-prefetch: off
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] New Style Replication in Version 4

[John Hearns]

I am landing here again following a spell on the list when I worked at XMA
in the UK. Hello again.


I have a use case of having a remote office, which should be able to have a
common storage area with a main office.

At the last company I worked with, we used GPFS with AFM to achieve this
(not really relevant to this list).


I at first though Geo Replication would be ideal for this use case, however
further reading says that it is suitable for making a slave only copy for
disaster recovery. Ie. there is no 'two way' syncing of files.

To make it a bit clearer, the concept is that the remote site might be in
the USA, and the home site in Europe.

Scientists at the remote site copy files for analysis to a local storage
server.

At the home site there should be a replica of the analysis files. At the
home site there are HPC resources to run simulations or process the data
somehow. Results files are written to a storage server and then should be
replicated back to the remote site.



Would anyone be able to comment on the New Style Geo Replication which has
come along in Gluster version 4?

What is the status please, and is it a suitable method for enabling a share
at a remote office which syncs either way?


thankyou

John Hearns
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Updated Gluster Releases

[Amye Scavarda]

The Gluster community has released an out-of-normal-cadence release for
Gluster 3.10, 3.12, and 4.0 that resolves a CVE[1] that has been classified
as Important. A privilege escalation flaw was found in the gluster snapshot
scheduler.

Any gluster client allowed to mount gluster volumes could also mount shared
gluster storage volumes and escalate privileges by scheduling malicious
cronjobs via symlink. Beyond installing the new release, additional
mitigation would include limiting exposure of gluster server nodes by these
practices:

Gluster server should be on LAN and not reachable from public networks.
Use gluster auth.allow and auth.reject.
Use TLS certificates between gluster server nodes and clients.

Please note: these practices would only mitigate attacks from unauthorized
malicious clients. Gluster clients allowed by auth.allow or having signed
TLS client certificates would still be able to trigger this attack.

Further information can be found about CVE-2018-1088 from the MITRE CVE
database.[2]

Our recommendation is to upgrade to these new releases:
https://download.gluster.org/pub/gluster/glusterfs/3.10/3.10.12/
https://download.gluster.org/pub/gluster/glusterfs/3.12/3.12.9/
https://download.gluster.org/pub/gluster/glusterfs/4.0/4.0.2/

[1] https://access.redhat.com/security/cve/cve-2018-1088
[2] http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-1088


-- 
Amye Scavarda | a...@redhat.com | Gluster Community Lead
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Announcing Gluster release 4.0.2 (Short Term Maintenance)

[Shyam Ranganathan]

The Gluster community is pleased to announce the release of Gluster
4.0.2 (packages available at [1]).

Release notes for the release can be found at [2].

This release contains fixes for CVE-2018-1088 and CVE-2018-1112, among
other fixes. Please use the release notes to check on the fix list.

Thanks,
Gluster community

[1] Packages:
https://download.gluster.org/pub/gluster/glusterfs/4.0/4.0.2/

[2] Release notes:
http://docs.gluster.org/en/latest/release-notes/4.0.2/ (link may not be
active yet!)
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Announcing Gluster release 3.12.9 (Long Term Maintenance)

[Shyam Ranganathan]

The Gluster community is pleased to announce the release of Gluster
3.12.9 (packages available at [1]).

Release notes for the release can be found at [2].

This release contains fixes for CVE-2018-1088 and CVE-2018-1112, among
other fixes. Please use the release notes to check on the fix list.

Thanks,
Gluster community

[1] Packages:
https://download.gluster.org/pub/gluster/glusterfs/3.12/3.12.9/

[2] Release notes:
http://docs.gluster.org/en/latest/release-notes/3.12.9/ (link may not be
active yet!)
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Announcing Gluster release 3.10.12 (Long Term Maintenance)

[Shyam Ranganathan]

The Gluster community is pleased to announce the release of Gluster
3.10.12 (packages available at [1]).

Release notes for the release can be found at [2].

This release contains fixes for CVE-2018-1088 and CVE-2018-1112, among
other fixes. Please use the release notes to check on the fix list.

Thanks,
Gluster community

[1] Packages:
https://download.gluster.org/pub/gluster/glusterfs/3.10/3.10.12/

[2] Release notes:
http://docs.gluster.org/en/latest/release-notes/3.10.12/ (link may not
be active yet!)
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster rebalance taking many years

[shadowsocks飞飞]

 I cannot calculate the number of files normally

Through df -i I got the approximate number of files is  63694442

[root@CentOS-73-64-minimal ~]# df -i
Filesystem  InodesIUsed  IFree IUse%
Mounted on
/dev/md2 131981312 30901030  101080282   24% /
devtmpfs   8192893  43581924581%
/dev
tmpfs  8199799 802981917701%
/dev/shm
tmpfs  8199799 141581983841%
/run
tmpfs  8199799   1681997831%
/sys/fs/cgroup
/dev/md3 110067712 29199861   80867851   27%
/home
/dev/md1131072  363 1307091%
/boot
gluster1:/web   2559860992 63694442 24961665503%
/web
tmpfs  8199799181997981%
/run/user/0


The rebalance log is in the attachment

the cluster information

gluster volume status web detail
Status of volume: web

--
Brick: Brick gluster1:/home/export/md3/brick
TCP Port : 49154
RDMA Port: 0
Online   : Y
Pid  : 16730
File System  : ext4
Device   : /dev/md3
Mount Options: rw,noatime,nodiratime,nobarrier,data=ordered
Inode Size   : 256
Disk Space Free  : 239.4GB
Total Disk Space : 1.6TB
Inode Count  : 110067712
Free Inodes  : 80867992

--
Brick: Brick gluster1:/export/md2/brick
TCP Port : 49155
RDMA Port: 0
Online   : Y
Pid  : 16758
File System  : ext4
Device   : /dev/md2
Mount Options: rw,noatime,nodiratime,nobarrier,data=ordered
Inode Size   : 256
Disk Space Free  : 589.4GB
Total Disk Space : 1.9TB
Inode Count  : 131981312
Free Inodes  : 101080484

--
Brick: Brick gluster2:/home/export/md3/brick
TCP Port : 49152
RDMA Port: 0
Online   : Y
Pid  : 12556
File System  : xfs
Device   : /dev/md3
Mount Options: rw,noatime,nodiratime,attr2,
inode64,sunit=1024,swidth=3072,noquota
Inode Size   : 256
Disk Space Free  : 10.7TB
Total Disk Space : 10.8TB
Inode Count  : 2317811968
Free Inodes  : 2314218207

Most of the files in the cluster are pictures smaller than 1M


2018-04-30 15:16 GMT+08:00 Nithya Balachandran :

> Hi,
>
>
> This value is an ongoing rough estimate based on the amount of data
> rebalance has migrated since it started. The values will cange as the
> rebalance progresses.
> A few questions:
>
>1. How many files/dirs do you have on this volume?
>2. What is the average size of the files?
>3. What is the total size of the data on the volume?
>
>
> Can you send us the rebalance log?
>
>
> Thanks,
> Nithya
>
> On 30 April 2018 at 10:33, kiwizhang618  wrote:
>
>>  I met a big problem,the cluster rebalance takes a long time after adding
>> a new node
>>
>> gluster volume rebalance web status
>> Node Rebalanced-files  size
>> scanned  failures   skipped   status  run time in
>> h:m:s
>>-  ---   ---
>> ---   ---   --- 
>> --
>>localhost  90043.5MB
>>2232 069  in progress0:36:49
>> gluster2 105239.3MB
>>4393 0  1052  in progress0:36:49
>> Estimated time left for rebalance to complete : 9919:44:34
>> volume rebalance: web: success
>>
>> the rebalance log
>> [glusterfsd.c:2511:main] 0-/usr/sbin/glusterfs: Started running
>> /usr/sbin/glusterfs version 3.12.8 (args: /usr/sbin/glusterfs -s localhost
>> --volfile-id rebalance/web --xlator-option *dht.use-readdirp=yes
>> --xlator-option *dht.lookup-unhashed=yes --xlator-option
>> *dht.assert-no-child-down=yes --xlator-option
>> *replicate*.data-self-heal=off --xlator-option
>> *replicate*.metadata-self-heal=off --xlator-option
>> *replicate*.entry-self-heal=off --xlator-option *dht.readdir-optimize=on
>> --xlator-option *dht.rebalance-cmd=1 --xlator-option
>> *dht.node-uuid=d47ad89d-7979-4ede-9aba-e04f020bb4f0 --xlator-option
>> *dht.commit-hash=3610561770 --socket-file /var/run/gluster/gluster-rebal
>> ance-bdef10eb-1c83-410c-8ad3-fe286450004b.sock --pid-file
>> 

Re: [Gluster-users] Gluster rebalance taking many years

[Nithya Balachandran]

Hi,


This value is an ongoing rough estimate based on the amount of data
rebalance has migrated since it started. The values will cange as the
rebalance progresses.
A few questions:

   1. How many files/dirs do you have on this volume?
   2. What is the average size of the files?
   3. What is the total size of the data on the volume?


Can you send us the rebalance log?


Thanks,
Nithya

On 30 April 2018 at 10:33, kiwizhang618  wrote:

>  I met a big problem,the cluster rebalance takes a long time after adding
> a new node
>
> gluster volume rebalance web status
> Node Rebalanced-files  size
> scanned  failures   skipped   status  run time in
> h:m:s
>-  ---   ---
> ---   ---   --- 
> --
>localhost  90043.5MB
>2232 069  in progress0:36:49
> gluster2 105239.3MB
>4393 0  1052  in progress0:36:49
> Estimated time left for rebalance to complete : 9919:44:34
> volume rebalance: web: success
>
> the rebalance log
> [glusterfsd.c:2511:main] 0-/usr/sbin/glusterfs: Started running
> /usr/sbin/glusterfs version 3.12.8 (args: /usr/sbin/glusterfs -s localhost
> --volfile-id rebalance/web --xlator-option *dht.use-readdirp=yes
> --xlator-option *dht.lookup-unhashed=yes --xlator-option
> *dht.assert-no-child-down=yes --xlator-option
> *replicate*.data-self-heal=off --xlator-option 
> *replicate*.metadata-self-heal=off
> --xlator-option *replicate*.entry-self-heal=off --xlator-option
> *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=1
> --xlator-option *dht.node-uuid=d47ad89d-7979-4ede-9aba-e04f020bb4f0
> --xlator-option *dht.commit-hash=3610561770 --socket-file
> /var/run/gluster/gluster-rebalance-bdef10eb-1c83-410c-8ad3-fe286450004b.sock
> --pid-file 
> /var/lib/glusterd/vols/web/rebalance/d47ad89d-7979-4ede-9aba-e04f020bb4f0.pid
> -l /var/log/glusterfs/web-rebalance.log)
> [2018-04-30 04:20:45.100902] W [MSGID: 101002] [options.c:995:xl_opt_validate]
> 0-glusterfs: option 'address-family' is deprecated, preferred is
> 'transport.address-family', continuing with correction
> [2018-04-30 04:20:45.103927] I [MSGID: 101190] 
> [event-epoll.c:613:event_dispatch_epoll_worker]
> 0-epoll: Started thread with index 1
> [2018-04-30 04:20:55.191261] E [MSGID: 109039] 
> [dht-common.c:3113:dht_find_local_subvol_cbk]
> 0-web-dht: getxattr err for dir [No data available]
> [2018-04-30 04:21:19.783469] E [MSGID: 109023] 
> [dht-rebalance.c:2669:gf_defrag_migrate_single_file]
> 0-web-dht: Migrate file failed: 
> /2018/02/x187f6596-36ac-45e6-bd7a-019804dfe427.jpg,
> lookup failed [Stale file handle]
> The message "E [MSGID: 109039] [dht-common.c:3113:dht_find_local_subvol_cbk]
> 0-web-dht: getxattr err for dir [No data available]" repeated 2 times
> between [2018-04-30 04:20:55.191261] and [2018-04-30 04:20:55.193615]
>
> the gluster info
> Volume Name: web
> Type: Distribute
> Volume ID: bdef10eb-1c83-410c-8ad3-fe286450004b
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 3
> Transport-type: tcp
> Bricks:
> Brick1: gluster1:/home/export/md3/brick
> Brick2: gluster1:/export/md2/brick
> Brick3: gluster2:/home/export/md3/brick
> Options Reconfigured:
> nfs.trusted-sync: on
> nfs.trusted-write: on
> cluster.rebal-throttle: aggressive
> features.inode-quota: off
> features.quota: off
> cluster.shd-wait-qlength: 1024
> transport.address-family: inet
> cluster.lookup-unhashed: auto
> performance.cache-size: 1GB
> performance.client-io-threads: on
> performance.write-behind-window-size: 4MB
> performance.io-thread-count: 8
> performance.force-readdirp: on
> performance.readdir-ahead: on
> cluster.readdir-optimize: on
> performance.high-prio-threads: 8
> performance.flush-behind: on
> performance.write-behind: on
> performance.quick-read: off
> performance.io-cache: on
> performance.read-ahead: off
> server.event-threads: 8
> cluster.lookup-optimize: on
> features.cache-invalidation: on
> features.cache-invalidation-timeout: 600
> performance.stat-prefetch: off
> performance.md-cache-timeout: 60
> network.inode-lru-limit: 9
> diagnostics.brick-log-level: ERROR
> diagnostics.brick-sys-log-level: ERROR
> diagnostics.client-log-level: ERROR
> diagnostics.client-sys-log-level: ERROR
> cluster.min-free-disk: 20%
> cluster.self-heal-window-size: 16
> cluster.self-heal-readdir-size: 1024
> cluster.background-self-heal-count: 4
> cluster.heal-wait-queue-length: 128
> client.event-threads: 8
> performance.cache-invalidation: on
> nfs.disable: off
> nfs.acl: off
> cluster.brick-multiplex: disable
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> 

[Gluster-users] Gluster rebalance taking many years

[shadowsocks飞飞]

 I met a big problem,the cluster rebalance takes a long time after adding a
new node

gluster volume rebalance web status
Node Rebalanced-files  size
  scanned  failures   skipped   status  run time in
h:m:s
   -  ---   ---
---   ---   --- 
--
   localhost  90043.5MB
 2232 069  in progress0:36:49
gluster2 105239.3MB
 4393 0  1052  in progress0:36:49
Estimated time left for rebalance to complete : 9919:44:34
volume rebalance: web: success

the rebalance log
[glusterfsd.c:2511:main] 0-/usr/sbin/glusterfs: Started running
/usr/sbin/glusterfs version 3.12.8 (args: /usr/sbin/glusterfs -s localhost
--volfile-id rebalance/web --xlator-option *dht.use-readdirp=yes
--xlator-option *dht.lookup-unhashed=yes --xlator-option
*dht.assert-no-child-down=yes --xlator-option
*replicate*.data-self-heal=off --xlator-option
*replicate*.metadata-self-heal=off --xlator-option
*replicate*.entry-self-heal=off --xlator-option *dht.readdir-optimize=on
--xlator-option *dht.rebalance-cmd=1 --xlator-option
*dht.node-uuid=d47ad89d-7979-4ede-9aba-e04f020bb4f0 --xlator-option
*dht.commit-hash=3610561770 --socket-file
/var/run/gluster/gluster-rebalance-bdef10eb-1c83-410c-8ad3-fe286450004b.sock
--pid-file
/var/lib/glusterd/vols/web/rebalance/d47ad89d-7979-4ede-9aba-e04f020bb4f0.pid
-l /var/log/glusterfs/web-rebalance.log)
[2018-04-30 04:20:45.100902] W [MSGID: 101002]
[options.c:995:xl_opt_validate] 0-glusterfs: option 'address-family' is
deprecated, preferred is 'transport.address-family', continuing with
correction
[2018-04-30 04:20:45.103927] I [MSGID: 101190]
[event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 1
[2018-04-30 04:20:55.191261] E [MSGID: 109039]
[dht-common.c:3113:dht_find_local_subvol_cbk] 0-web-dht: getxattr err for
dir [No data available]
[2018-04-30 04:21:19.783469] E [MSGID: 109023]
[dht-rebalance.c:2669:gf_defrag_migrate_single_file] 0-web-dht: Migrate
file failed: /2018/02/x187f6596-36ac-45e6-bd7a-019804dfe427.jpg, lookup
failed [Stale file handle]
The message "E [MSGID: 109039]
[dht-common.c:3113:dht_find_local_subvol_cbk] 0-web-dht: getxattr err for
dir [No data available]" repeated 2 times between [2018-04-30
04:20:55.191261] and [2018-04-30 04:20:55.193615]

the gluster info
Volume Name: web
Type: Distribute
Volume ID: bdef10eb-1c83-410c-8ad3-fe286450004b
Status: Started
Snapshot Count: 0
Number of Bricks: 3
Transport-type: tcp
Bricks:
Brick1: gluster1:/home/export/md3/brick
Brick2: gluster1:/export/md2/brick
Brick3: gluster2:/home/export/md3/brick
Options Reconfigured:
nfs.trusted-sync: on
nfs.trusted-write: on
cluster.rebal-throttle: aggressive
features.inode-quota: off
features.quota: off
cluster.shd-wait-qlength: 1024
transport.address-family: inet
cluster.lookup-unhashed: auto
performance.cache-size: 1GB
performance.client-io-threads: on
performance.write-behind-window-size: 4MB
performance.io-thread-count: 8
performance.force-readdirp: on
performance.readdir-ahead: on
cluster.readdir-optimize: on
performance.high-prio-threads: 8
performance.flush-behind: on
performance.write-behind: on
performance.quick-read: off
performance.io-cache: on
performance.read-ahead: off
server.event-threads: 8
cluster.lookup-optimize: on
features.cache-invalidation: on
features.cache-invalidation-timeout: 600
performance.stat-prefetch: off
performance.md-cache-timeout: 60
network.inode-lru-limit: 9
diagnostics.brick-log-level: ERROR
diagnostics.brick-sys-log-level: ERROR
diagnostics.client-log-level: ERROR
diagnostics.client-sys-log-level: ERROR
cluster.min-free-disk: 20%
cluster.self-heal-window-size: 16
cluster.self-heal-readdir-size: 1024
cluster.background-self-heal-count: 4
cluster.heal-wait-queue-length: 128
client.event-threads: 8
performance.cache-invalidation: on
nfs.disable: off
nfs.acl: off
cluster.brick-multiplex: disable
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users