[Gluster-users] Evergrowing distributed volume question

[nux]

Hello,

A while ago I attempted and failed to maintain an "evergrowing" storage 
solution based on GlusterFS.
I was relying on a distributed non-replicated volume to host backups and 
so on, in the idea that when it was close to full I would just add 
another brick (server) and keep it going like that.
In reality what happened was that many of the writes were distributed to 
the brick that was (in time) full, ending up with "out of space" errors, 
despite having one or more bricks with plenty of space.


Can anyone advise whether current Glusterfs behaviour has improved in 
this regard, ie does it check if a brick is full and redirect the 
"write" to one that is not?


Regards,
Lucian




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster usage scenarios in HPC cluster management

[Ewen Chan]

Erik:

Thank you for sharing your insights in terms of how Gluster is used, in a 
professional, production environment (or at least how HPE is using it, either 
internally, and/or for your clients).

I really appreciated reading this.

I am just a home lab user and I have a very tiny 4-node micro cluster for 
HPC/CAE applications, and where I run into issues with storage is that NAND 
flash based SSDs have finite write endurance limits, which, as a home lab user, 
gets expensive.

But I've also tested using tmpfs (allocating half of the RAM per compute node) 
and exporting that as a distributed stripped GlusterFS volume over NFS over 
RDMA to the 100 Gbps IB network so that the "ramdrives" can be used as a high 
speed "scratch disk space" that doesn't have the write endurance limits that 
NAND based flash memory SSDs have.

Yes, it isn't as reliable or certainly not high availability (power goes down, 
and the battery backup is exhausted, then the data is lost because it sat in 
RAM), but it's to solve the problems of mechanically rotating hard drives are 
too slow, NAND flash based SSDs has finite write endurance limits, and RAM 
drives, whilst in theory, faster, is also the most expensive in a $/GB basis 
compared to the other storage solutions.

It's rather unfortunately that you have these different "tiers" of storage, and 
there's really nothing else in between that can help address all of these 
issues simultaneously.

Thank you for sharing your thoughts.

Sincerely,

Ewen Chan


From: gluster-users-boun...@gluster.org  on 
behalf of Erik Jacobson 
Sent: March 19, 2021 11:03 AM
To: gluster-users@gluster.org 
Subject: [Gluster-users] Gluster usage scenarios in HPC cluster management

A while back I was asked to make a blog or something similar to discuss
the use cases the team I work on (HPCM cluster management) at HPE.

If you are not interested in reading about what I'm up to, just delete
this and move on.

I really don't have a public blogging mechanism so I'll just describe
what we're up to here. Some of this was posted in some form in the past.
Since this contains the raw materials, I could make a wiki-ized version
if there were a public place to put it.



We currently use gluster in two parts of cluster management.

In fact, gluster in our management node infrastructure is helping us to
provide scaling and consistency to some of the largest clusters in the
world, clusters in the TOP100 list. While I can get in to trouble by
sharing too much, I will just say that trends are continuing and the
future may have some exciting announcements on where on TOP100 certain
new giant systems may end up in the coming 1-2 years.

At HPE, HPCM is the "traditional cluster manager." There is another team
that develops a more cloud-like solution and I am not discussing that
solution here.


Use Case #1: Leader Nodes and Scale Out
--
- Why?
  * Scale out
  * Redundancy (combined with CTDB, any leader can fail)
  * Consistency (All servers and compute agree on what the content is)

- Cluster manager has an admin or head node and zero or more leader nodes

- Leader nodes are provisioned in groups of 3 to use distributed
  replica-3 volumes (although at least one customer has interest
  in replica-5)

- We configure a few different volumes for different use cases

- We use Gluster NFS still because, over a year ago, Ganesha was not
  working with our workload and we haven't had time to re-test and
  engage with the community. No blame - we would also owe making sure
  our settings are right.

- We use CTDB for a measure of HA and IP alias management. We use this
  instead of pacemaker to reduce complexity.

- The volume use cases are:
  * Image sharing for diskless compute nodes (sometimes 6,000 nodes)
-> Normally squashFS image files for speed/efficiency exported NFS
-> Expanded ("chrootable") traditional NFS trees for people who
   prefer that, but they don't scale as well and are slower to boot
-> Squashfs images sit on a sharded volume while traditional gluster
   used for expanded tree.
  * TFTP/HTTP for network boot/PXE including miniroot
-> Spread across leaders too due so one node is not saturated with
   PXE/DHCP requests
-> Miniroot is a "fatter initrd" that has our CM toolchain
  * Logs/consoles
-> For traditional logs and consoles (HCPM also uses
   elasticsearch/kafka/friends but we don't put that in gluster)
-> Separate volume to have more non-cached friendly settings
  * 4 total volumes used (one sharded, one heavily optimized for
caching, one for ctdb lock, and one traditional for logging/etc)

- Leader Setup
  * Admin node installs the leaders like any other compute nodes
  * A setup tool operates that configures gluster volumes and CTDB
  * When ready, an admin/head node can be engaged with the leaders
  * At that point, certain paths on the 

Re: [Gluster-users] Gluster usage scenarios in HPC cluster management

[Erik Jacobson]

> - Gluster sizing
>   * We typically state compute nodes per leader but this is not for
> gluster per-se. Squashfs image objects are very efficient and
> probably would be fine for 2k nodes per leader. Leader nodes provide
> other services including console logs, system logs, and monitoring
> services.

I tried to avoid typos and mistakes but I missed something above. Argues
for wiki right? :)  I missed "512" :)

  * We typically state 512 compute nodes per leader but this is not for
gluster per-se. Squashfs image objects are very efficient and
probably would be fine for 2k nodes per leader. Leader nodes provide
other services including console logs, system logs, and monitoring
services.





Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Volume not healing

[Diego Zuccato]

Il 19/03/21 13:17, Strahil Nikolov ha scritto:

> find /FUSE/mountpoint -exec stat {} \;
Running it now (redirecting stdout to /dev/null).
It's finding quite a lot of "no such file or directory" errors.

-- 
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Gluster usage scenarios in HPC cluster management

[Erik Jacobson]

A while back I was asked to make a blog or something similar to discuss
the use cases the team I work on (HPCM cluster management) at HPE.

If you are not interested in reading about what I'm up to, just delete
this and move on.

I really don't have a public blogging mechanism so I'll just describe
what we're up to here. Some of this was posted in some form in the past.
Since this contains the raw materials, I could make a wiki-ized version
if there were a public place to put it.



We currently use gluster in two parts of cluster management.

In fact, gluster in our management node infrastructure is helping us to
provide scaling and consistency to some of the largest clusters in the
world, clusters in the TOP100 list. While I can get in to trouble by
sharing too much, I will just say that trends are continuing and the
future may have some exciting announcements on where on TOP100 certain
new giant systems may end up in the coming 1-2 years.

At HPE, HPCM is the "traditional cluster manager." There is another team
that develops a more cloud-like solution and I am not discussing that
solution here.


Use Case #1: Leader Nodes and Scale Out
--
- Why?
  * Scale out
  * Redundancy (combined with CTDB, any leader can fail)
  * Consistency (All servers and compute agree on what the content is)

- Cluster manager has an admin or head node and zero or more leader nodes

- Leader nodes are provisioned in groups of 3 to use distributed
  replica-3 volumes (although at least one customer has interest
  in replica-5)

- We configure a few different volumes for different use cases

- We use Gluster NFS still because, over a year ago, Ganesha was not
  working with our workload and we haven't had time to re-test and
  engage with the community. No blame - we would also owe making sure
  our settings are right.

- We use CTDB for a measure of HA and IP alias management. We use this
  instead of pacemaker to reduce complexity.

- The volume use cases are:
  * Image sharing for diskless compute nodes (sometimes 6,000 nodes)
-> Normally squashFS image files for speed/efficiency exported NFS
-> Expanded ("chrootable") traditional NFS trees for people who
   prefer that, but they don't scale as well and are slower to boot
-> Squashfs images sit on a sharded volume while traditional gluster
   used for expanded tree.
  * TFTP/HTTP for network boot/PXE including miniroot
-> Spread across leaders too due so one node is not saturated with
   PXE/DHCP requests
-> Miniroot is a "fatter initrd" that has our CM toolchain
  * Logs/consoles
-> For traditional logs and consoles (HCPM also uses
   elasticsearch/kafka/friends but we don't put that in gluster)
-> Separate volume to have more non-cached friendly settings
  * 4 total volumes used (one sharded, one heavily optimized for
caching, one for ctdb lock, and one traditional for logging/etc)

- Leader Setup
  * Admin node installs the leaders like any other compute nodes
  * A setup tool operates that configures gluster volumes and CTDB
  * When ready, an admin/head node can be engaged with the leaders
  * At that point, certain paths on the admin become gluster fuse mounts
and bind mounts to gluster fuse mounts.

- How images are deployed (squashfs mode)
  * User creates an image using image creation tools that make a
chrootable tree style image on the admin/head node
  * mksquashfs will generate a squashfs image file on to a shared
storage gluster mount
  * Nodes will mount the filesystem with the squashfs images and then
loop mount the squashfs as part of the boot process.

- How are compute nodes tied to leaders
  * We simply have a variable in our database where human or automated
discovery tools can assign a given node to a given IP alias. This
works better for us than trying to play routing tricks or load
balance tricks
  * When leaders PXE, the DHCP response includes next-server and the
compute node uses the leader IP alias for the tftp/http for
getting the boot loader DHCP config files are on shared storage
to facilitate future scaling of DHCP services.
  * ipxe or grub2 network config files then fetch the kernel, initrd
  * initrd has a small update to load a miniroot (install environment)
 which has more tooling
  * Node is installed (for nodes with root disks) or does a network boot
cycle.

- Gluster sizing
  * We typically state compute nodes per leader but this is not for
gluster per-se. Squashfs image objects are very efficient and
probably would be fine for 2k nodes per leader. Leader nodes provide
other services including console logs, system logs, and monitoring
services.
  * Our biggest deployment at a customer site right now has 24 leader
nodes. Bigger systems are coming.

- Startup scripts - Getting all the gluster mounts and many bind mounts
  used in the solution, as well 

Re: [Gluster-users] Gluster usage scenarios in HPC cluster management

[Erik Jacobson]

> But I've also tested using tmpfs (allocating half of the RAM per compute node)
> and exporting that as a distributed stripped GlusterFS volume over NFS over
> RDMA to the 100 Gbps IB network so that the "ramdrives" can be used as a high
> speed "scratch disk space" that doesn't have the write endurance limits that
> NAND based flash memory SSDs have.

In my world, we leave the high speed networks to jobs so I don't have
much to offer. In our test SU Leader setup where we may not have disks,
we do carve gluster bricks out of TMPS mounts. However, in that test
case, designed to test the tooling and not the workload, I use iscsi to
emulate disks to test the true solution.

I will just mention that the cluster manager use of squashfs image
objects sitting on NFS mounts is very fast even on top of 20G (2x10G)
mgmt infrastructure. If you combine it with a TMPFS overlay, which is
our default, you will have a writable area in to TMPFS that doesn't
persist. You will have low memory usage.

For a 4-node cluster, you probably don't need to bother with squashfs
even and just mount the directory tree for the image at the right time.

By using tmpfs overlay and some post-boot configuration, you can perhaps
avoid the memory usage of what you are doing. As long as you don't need
to beat the crap out of root, an NFS root is fine and using gluster
backed disks is fine. Note that if you use exported trees with gnfs
instead of image objects, there are lots of volume tweaks you can make
to push efficiency up. For squashfs, I used a sharded volume.

It's easy for me to write this since we have the install environment.
While nothing is "Hard" in there, it's a bunch of code developed over
time. That said, if you wanted to experiment, I can share some pieces of
what we do. I just fear it's too complicated.

I will note that some customers advocate for a tiny root - say 1.5G --
that could fit in TMPFS easily and then attach in workloads (other
filesystems with development environments over the network, or container
environments, etc). That would be another way to keep memory use low for
a diskless cluster.

(we use gnfs because we're not ready to switch to ganesha yet. It's on
our list to move if we can get it working for our load).

> Yes, it isn't as reliable or certainly not high availability (power goes down,
> and the battery backup is exhausted, then the data is lost because it sat in
> RAM), but it's to solve the problems of mechanically rotating hard drives are
> too slow, NAND flash based SSDs has finite write endurance limits, and RAM
> drives, whilst in theory, faster, is also the most expensive in a $/GB basis
> compared to the other storage solutions.
> 
> It's rather unfortunately that you have these different "tiers" of storage, 
> and
> there's really nothing else in between that can help address all of these
> issues simultaneously.
> 
> Thank you for sharing your thoughts.
> 
> Sincerely,
> 
> Ewen Chan
> 
> ━━━
> From: gluster-users-boun...@gluster.org  on
> behalf of Erik Jacobson 
> Sent: March 19, 2021 11:03 AM
> To: gluster-users@gluster.org 
> Subject: [Gluster-users] Gluster usage scenarios in HPC cluster management
>  
> A while back I was asked to make a blog or something similar to discuss
> the use cases the team I work on (HPCM cluster management) at HPE.
> 
> If you are not interested in reading about what I'm up to, just delete
> this and move on.
> 
> I really don't have a public blogging mechanism so I'll just describe
> what we're up to here. Some of this was posted in some form in the past.
> Since this contains the raw materials, I could make a wiki-ized version
> if there were a public place to put it.
> 
> 
> 
> We currently use gluster in two parts of cluster management.
> 
> In fact, gluster in our management node infrastructure is helping us to
> provide scaling and consistency to some of the largest clusters in the
> world, clusters in the TOP100 list. While I can get in to trouble by
> sharing too much, I will just say that trends are continuing and the
> future may have some exciting announcements on where on TOP100 certain
> new giant systems may end up in the coming 1-2 years.
> 
> At HPE, HPCM is the "traditional cluster manager." There is another team
> that develops a more cloud-like solution and I am not discussing that
> solution here.
> 
> 
> Use Case #1: Leader Nodes and Scale Out
> --
> - Why?
>   * Scale out
>   * Redundancy (combined with CTDB, any leader can fail)
>   * Consistency (All servers and compute agree on what the content is)
> 
> - Cluster manager has an admin or head node and zero or more leader nodes
> 
> - Leader nodes are provisioned in groups of 3 to use distributed
>   replica-3 volumes (although at least one customer has interest
>   in replica-5)
> 
> - We configure a few different volumes for different 

Re: [Gluster-users] Gluster usage scenarios in HPC cluster management

[Ewen Chan]

Erik:

My apologies for not being more clear originally.

What I meant to say was that I was using GlusterFS for HPC jobs because my 
understanding is that most HPC environments often or tend to use, for example, 
NVMe SSDs for their high speed storage tier, but even those have a finite write 
endurance limit as well.

And whilst normally, for a large corporation, the consumption of the write 
endurance limit of the NVMe SSDs and their replacement would just be a cost of 
"normal part of doing business", but for a home lab, I can't afford to spend 
that kind of money whenever the drives wear out like that.

And this is what drove me to testing GlusterFS distributed stripped volume 
exported to NFS over RDMA so that the RAM was used both in the execution of the 
jobs as well as for the high speed scratch disk space during job execution such 
that it wouldn't be subject to the write endurance limits of NAND flash SSDs 
(NVMe or otherwise), nor the significantly slower performance of mechanically 
rotating hard disk drives.

So, I was talking about using GlusterFS for HPC as well, but in the context of 
job execution rather than more the "management" tasks/operations that you 
described in your message below.

Thank you.

Sincerely,
Ewen


From: Erik Jacobson 
Sent: March 19, 2021 12:24 PM
To: Ewen Chan 
Cc: Erik Jacobson ; gluster-users@gluster.org 

Subject: Re: [Gluster-users] Gluster usage scenarios in HPC cluster management

> But I've also tested using tmpfs (allocating half of the RAM per compute node)
> and exporting that as a distributed stripped GlusterFS volume over NFS over
> RDMA to the 100 Gbps IB network so that the "ramdrives" can be used as a high
> speed "scratch disk space" that doesn't have the write endurance limits that
> NAND based flash memory SSDs have.

In my world, we leave the high speed networks to jobs so I don't have
much to offer. In our test SU Leader setup where we may not have disks,
we do carve gluster bricks out of TMPS mounts. However, in that test
case, designed to test the tooling and not the workload, I use iscsi to
emulate disks to test the true solution.

I will just mention that the cluster manager use of squashfs image
objects sitting on NFS mounts is very fast even on top of 20G (2x10G)
mgmt infrastructure. If you combine it with a TMPFS overlay, which is
our default, you will have a writable area in to TMPFS that doesn't
persist. You will have low memory usage.

For a 4-node cluster, you probably don't need to bother with squashfs
even and just mount the directory tree for the image at the right time.

By using tmpfs overlay and some post-boot configuration, you can perhaps
avoid the memory usage of what you are doing. As long as you don't need
to beat the crap out of root, an NFS root is fine and using gluster
backed disks is fine. Note that if you use exported trees with gnfs
instead of image objects, there are lots of volume tweaks you can make
to push efficiency up. For squashfs, I used a sharded volume.

It's easy for me to write this since we have the install environment.
While nothing is "Hard" in there, it's a bunch of code developed over
time. That said, if you wanted to experiment, I can share some pieces of
what we do. I just fear it's too complicated.

I will note that some customers advocate for a tiny root - say 1.5G --
that could fit in TMPFS easily and then attach in workloads (other
filesystems with development environments over the network, or container
environments, etc). That would be another way to keep memory use low for
a diskless cluster.

(we use gnfs because we're not ready to switch to ganesha yet. It's on
our list to move if we can get it working for our load).

> Yes, it isn't as reliable or certainly not high availability (power goes down,
> and the battery backup is exhausted, then the data is lost because it sat in
> RAM), but it's to solve the problems of mechanically rotating hard drives are
> too slow, NAND flash based SSDs has finite write endurance limits, and RAM
> drives, whilst in theory, faster, is also the most expensive in a $/GB basis
> compared to the other storage solutions.
>
> It's rather unfortunately that you have these different "tiers" of storage, 
> and
> there's really nothing else in between that can help address all of these
> issues simultaneously.
>
> Thank you for sharing your thoughts.
>
> Sincerely,
>
> Ewen Chan
>
> ━━━
> From: gluster-users-boun...@gluster.org  on
> behalf of Erik Jacobson 
> Sent: March 19, 2021 11:03 AM
> To: gluster-users@gluster.org 
> Subject: [Gluster-users] Gluster usage scenarios in HPC cluster management
>
> A while back I was asked to make a blog or something similar to discuss
> the use cases the team I work on (HPCM cluster management) at HPE.
>
> If you are not interested in reading about what I'm up to, just delete
> this and move on.
>
> I really don't have a 

Re: [Gluster-users] Brick offline after upgrade

[David Cunningham]

Hi Strahil,

It's as follows. Do you see anything unusual? Thanks.

root@caes8:~# ls -al /var/lib/glusterd/vols/gvol0/
total 52
drwxr-xr-x 3 root root 4096 Mar 18 17:06 .
drwxr-xr-x 3 root root 4096 Jul 17  2018 ..
drwxr-xr-x 2 root root 4096 Mar 18 17:06 bricks
-rw--- 1 root root   16 Mar 18 17:06 cksum
-rw--- 1 root root 3848 Mar 18 16:52
gvol0.caes8.nodirectwritedata-gluster-gvol0.vol
-rw--- 1 root root 2270 Feb 14  2020 gvol0.gfproxyd.vol
-rw--- 1 root root 1715 Mar 18 16:52 gvol0.tcp-fuse.vol
-rw--- 1 root root  729 Mar 18 17:06 info
-rw--- 1 root root0 Feb 14  2020 marker.tstamp
-rw--- 1 root root  168 Mar 18 17:06 node_state.info
-rw--- 1 root root   18 Mar 18 17:06 quota.cksum
-rw--- 1 root root0 Jul 17  2018 quota.conf
-rw--- 1 root root   13 Mar 18 17:06 snapd.info
-rw--- 1 root root 1829 Mar 18 16:52 trusted-gvol0.tcp-fuse.vol
-rw--- 1 root root  896 Feb 14  2020 trusted-gvol0.tcp-gfproxy-fuse.vol


On Fri, 19 Mar 2021 at 17:51, Strahil Nikolov  wrote:

> [2021-03-18 23:52:52.084754] E [MSGID: 101019] [xlator.c:715:xlator_init] 
> 0-gvol0-server: Initialization of volume 'gvol0-server' failed, review your 
> volfile again
>
> What is the content of :
>
> /var/lib/glusterd/vols/gvol0 ?
>
>
> Best Regards,
>
> Strahil Nikolov
>
> On Fri, Mar 19, 2021 at 3:02, David Cunningham
>  wrote:
> Hello,
>
> We have a single node/brick GlusterFS test system which unfortunately had
> GlusterFS upgraded from version 5 to 6 while the GlusterFS processes were
> still running. I know this is not what the "Generic Upgrade procedure"
> recommends.
>
> Following a restart the brick is not online, and we can't see any error
> message explaining exactly why. Would anyone have an idea of where to look?
>
> Since the logs from the time of the upgrade and reboot are a bit lengthy
> I've attached them in a text file.
>
> Thank you in advance for any advice!
>
> --
> David Cunningham, Voisonics Limited
> http://voisonics.com/
> USA: +1 213 221 1092
> New Zealand: +64 (0)28 2558 3782
> 
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://meet.google.com/cpu-eiue-hvk
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
>

-- 
David Cunningham, Voisonics Limited
http://voisonics.com/
USA: +1 213 221 1092
New Zealand: +64 (0)28 2558 3782




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Volume not healing

[Diego Zuccato]

Hello all.

I have a "problematic" volume. It was Rep3a1 with a dedicated VM for the
arbiters.

Too bad I understimated RAM needs and the arbiters VM crashed frequently
for OOM (had just 8GB allocated). Even the other two nodes sometimes
crashed, too, during a remove-brick operation (other thread).

So I've had to stop & re-run the remove-brick multiple times, even
rebooting the nodes, but it never completed.

Now, I decided to move all the files to a temporary storage to rebuild
the volume from scratch, but I find directories with duplicated files
(two identical files, same name, size and contents), probably the two
replicas.

I tried to run "gluster v heal BigVol info summary" and got quite a high
count of entries to be healed on some bricks:
# gluster v heal BigVol info summary|grep pending|grep -v ' 0$'
Number of entries in heal pending: 41
Number of entries in heal pending: 2971
Number of entries in heal pending: 20
Number of entries in heal pending: 2393

Too bad that those numbers aren't decreasing with time.

Seems no entries are considered in split-brain condition (all counts for
"gluster v heal BigVol info split-brain" are 0).

Is there something I can do to convince Gluster to heal those entries
w/o going entry-by-entry manually?

Thanks.

-- 
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Geo-rep: Version upgrade to version 8 and above

[Shwetha Acharya]

Hi all,

With version 8, we have made certain changes
 to the
directory structure of changelog files in gluster geo-replication.

Thus, *before the upgrade, we need to execute the upgrade script

with the brick path as argument, *as described below:

1. Stop the geo-rep session
2. Run the upgrade script with the brick path as the argument. Script can
be  used in loop for multiple bricks.
3. Start the upgradation process.

This script will update the existing changelog directory structure and the
paths inside the htime files to a new format introduced in version 8.

If the above mentioned script is not executed, the search algorithm, used
during the history crawl will fail with the wrong result for upgradation
from version 7 and below to version 8 and above.

Regards,
Shwetha




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Volume not healing

[Diego Zuccato]

Il 19/03/21 11:06, Diego Zuccato ha scritto:

> I tried to run "gluster v heal BigVol info summary" and got quite a high
> count of entries to be healed on some bricks:
> # gluster v heal BigVol info summary|grep pending|grep -v ' 0$'
> Number of entries in heal pending: 41
> Number of entries in heal pending: 2971
> Number of entries in heal pending: 20
> Number of entries in heal pending: 2393
> 
> Too bad that those numbers aren't decreasing with time.
Slight correction. Seems the numbers are *slowly* decreasing. After one
hour I see:
# gluster v heal BigVol info summary|grep pending|grep -v ' 0$'
Number of entries in heal pending: 41
Number of entries in heal pending: 2955
Number of entries in heal pending: 20
Number of entries in heal pending: 2384

Is it possible to speed it up? Nodes are nearly idle...

-- 
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users