Re: [Gluster-users] GlusterFS storage driver deprecation in Kubernetes.
Thanks for the heads up Humble. This would help many of the gluster community users, who may not be following k8s threads actively to be planning their migration plans actively. For all the users who are currently running heketi + glusterfs, starting from k8s v1.26, you CANNOT use heketi + glusterfs based storage in k8s. Below are my personal suggestions for the users. Please treat these options as my personal opinion, and not an official stand of the gluster community. 0. Use an older (< 1.25) version of k8s, and keep using the setup :-) 1. Use current storage nodes as part of storage, but managed separately, and expose NFS and use NFS CSI to get the data in the pods. (Note the change over to new PV through CSI based PVC, which means applications need a migration). - I haven't tested this setup, hence can't vouch for this. 2. Use kadalu [6] operator to manage currently deployed glusterfs nodes as 'External' storage class, and use kadalu CSI (which uses native glusterfs mount as part of CSI node plugin) to get PV for your application pods. NOTE: here too, there is an application migration needed to use kadalu CSI based PVC. Suggested for those users with bigger PVs used in k8s setup already. There is a team to help with this migration if you wish to. 3. Use kadalu (or any 'other' CSI providers), provision a new storage, and copy over the data set to new storage: Would be an option if the storage is smaller in size. In this case, there would be extra time to do a copy of data through starting a pod with both existing PV and new PV added as mounts in the same pod, so you can copy off the data quickly. In any case, considering you do not have a lot of time before kubernetes v1.26 comes out, please do start your migration plans soon. For the developers of the glusterfs community, what are the thoughts you have on this? I know there is some effort started on keeping glusterfs-containers repo relevant, and I see PRs coming out. Happy to open up a discussion on the same. Regards, Amar (@amarts) [6] - https://github.com/kadalu/kadalu On Thu, Aug 11, 2022 at 5:47 PM Humble Chirammal wrote: > Hey Gluster Community, > > As you might be aware, there is an effort in the kubernetes community to > remove in-tree storage plugins to reduce external dependencies and security > concerns in the core Kubernetes. Thus, we are in a process to gradually > deprecate all the in-tree external storage plugins and eventually remove > them from the core Kubernetes codebase. GlusterFS is one of the very first > dynamic provisioners which was made into kubernetes v1.4 ( 2016 ) release > via https://github.com/kubernetes/kubernetes/pull/30888 . From then on > many deployments were/are making use of this driver to consume GlusterFS > volumes in Kubernetes/Openshift clusters. > > As part of this effort, we are planning to deprecate GlusterFS intree > plugin in 1.25 release and planning to take out Heketi code from > Kubernetes Code base in subsequent release. This code removal will not be > following kubernetes' normal deprecation policy [1] and will be treated as > an exception [2]. The main reason for this exception is that, Heketi is in > "Deep Maintenance" [3], also please see [4] for the latest push back from > the Heketi team on changes we would need to keep vendoring heketi into > kubernetes/kubernetes. We cannot keep heketi in the kubernetes code base as > heketi itself is literally going away. The current plan is to start > declaring the deprecation in kubernetes v1.25 and code removal in v1.26. > > If you are using GlusterFS driver in your cluster setup, please reply > with below info before 16-Aug-2022 to d...@kubernetes.io ML on thread ( > Deprecation > of intree GlusterFS driver in 1.25) or to this thread which can help us > to make a decision on when to completely remove this code base from the > repo. > > - what version of kubernetes are you running in your setup ? > > - how often do you upgrade your cluster? > > - what vendor or distro you are using ? Is it any (downstream) product > offering or upstream GlusterFS driver directly used in your setup? > > Awaiting your feedback. > > Thanks, > > Humble > > [1] https://kubernetes.io/docs/reference/using-api/deprecation-policy/ > > [2] > https://kubernetes.io/docs/reference/using-api/deprecation-policy/#exceptions > > [3] https://github.com/heketi/heketi#maintenance-status > > [4] https://github.com/heketi/heketi/pull/1904#issuecomment-1197100513 > [5] https://github.com/kubernetes/kubernetes/issues/100897 > > -- > > > > Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] GlusterFS storage driver deprecation in Kubernetes.
Hey Gluster Community, As you might be aware, there is an effort in the kubernetes community to remove in-tree storage plugins to reduce external dependencies and security concerns in the core Kubernetes. Thus, we are in a process to gradually deprecate all the in-tree external storage plugins and eventually remove them from the core Kubernetes codebase. GlusterFS is one of the very first dynamic provisioners which was made into kubernetes v1.4 ( 2016 ) release via https://github.com/kubernetes/kubernetes/pull/30888 . From then on many deployments were/are making use of this driver to consume GlusterFS volumes in Kubernetes/Openshift clusters. As part of this effort, we are planning to deprecate GlusterFS intree plugin in 1.25 release and planning to take out Heketi code from Kubernetes Code base in subsequent release. This code removal will not be following kubernetes' normal deprecation policy [1] and will be treated as an exception [2]. The main reason for this exception is that, Heketi is in "Deep Maintenance" [3], also please see [4] for the latest push back from the Heketi team on changes we would need to keep vendoring heketi into kubernetes/kubernetes. We cannot keep heketi in the kubernetes code base as heketi itself is literally going away. The current plan is to start declaring the deprecation in kubernetes v1.25 and code removal in v1.26. If you are using GlusterFS driver in your cluster setup, please reply with below info before 16-Aug-2022 to d...@kubernetes.io ML on thread ( Deprecation of intree GlusterFS driver in 1.25) or to this thread which can help us to make a decision on when to completely remove this code base from the repo. - what version of kubernetes are you running in your setup ? - how often do you upgrade your cluster? - what vendor or distro you are using ? Is it any (downstream) product offering or upstream GlusterFS driver directly used in your setup? Awaiting your feedback. Thanks, Humble [1] https://kubernetes.io/docs/reference/using-api/deprecation-policy/ [2] https://kubernetes.io/docs/reference/using-api/deprecation-policy/#exceptions [3] https://github.com/heketi/heketi#maintenance-status [4] https://github.com/heketi/heketi/pull/1904#issuecomment-1197100513 [5] https://github.com/kubernetes/kubernetes/issues/100897 -- Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] bitd.log and quotad.log flooding /var
Yup.
Seems the /etc/sysconfig/glusterd setting got finally applied and I now
have a process like this:
root 4107315 0.0 0.0 529244 40124 ?Ssl ago08 2:44
/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level ERROR
but bitd still spits out (some) 'I' lines
[2022-08-11 07:02:21.072943 +] I [MSGID: 118016]
[bit-rot.c:1052:bitd_oneshot_crawl] 0-cluster_data-bit-rot-0: Triggering
signing [{path=/extra/some/other/dirs/file.dat},
{gfid=3e35b158-35a6-4e63-adbd-41075a11022e}, {Brick-path=/srv/bricks/00/d}]
Moreover I've had to disable quota, since quota processes were eating
more than *75GB* RAM on each storage node! :(
Il 11/08/2022 07:12, Strahil Nikolov ha scritto:
Have you decreased glusterd log level via:
glusterd --log-level WARNING|ERROR
It seems that bitrot doesn't have it's own log level.
As a workaround, you can configure syslog to send the logs only remotely
and thus preventing the overfill of the /var .
Best Regards,
Strahil Nikolov
On Wed, Aug 10, 2022 at 7:52, Diego Zuccato
wrote:
Hi Strahil.
Sure. Luckily I didn't delete 'em all :)
From bitd.log:
-8<--
[2022-08-09 05:58:12.075999 +] I [MSGID: 118016]
[bit-rot.c:1052:bitd_oneshot_crawl] 0-cluster_data-bit-rot-0:
Triggering
signing [{path=/astro/...omisis.../file.dat},
{gfid=5956af24-5efc-496c-8d7e-ea6656f298de},
{Brick-path=/srv/bricks/10/d}]
[2022-08-09 05:58:12.082264 +] I [MSGID: 118016]
[bit-rot.c:1052:bitd_oneshot_crawl] 0-cluster_data-bit-rot-0:
Triggering
signing [{path=/astro/...omisis.../file.txt},
{gfid=afb75c03-0d29-414e-917a-ff718982c849},
{Brick-path=/srv/bricks/13/d}]
[2022-08-09 05:58:12.082267 +] I [MSGID: 118016]
[bit-rot.c:1052:bitd_oneshot_crawl] 0-cluster_data-bit-rot-0:
Triggering
signing [{path=/astro/...omisis.../file.dat},
{gfid=982bc7a8-d4ba-45d7-9104-044e5d446802},
{Brick-path=/srv/bricks/06/d}]
[2022-08-09 05:58:12.084960 +] I [MSGID: 118016]
[bit-rot.c:1052:bitd_oneshot_crawl] 0-cluster_data-bit-rot-0:
Triggering
signing [{path=/atmos/...omisis.../file},
{gfid=17e4dfb0-1f64-47a3-9aa8-b3fa05b7cd4e},
{Brick-path=/srv/bricks/15/d}]
[2022-08-09 05:58:12.089357 +] I [MSGID: 118016]
[bit-rot.c:1052:bitd_oneshot_crawl] 0-cluster_data-bit-rot-0:
Triggering
signing [{path=/astro/...omisis.../file.txt},
{gfid=e70bf289-5aeb-43c2-aadd-d18979cf62b5},
{Brick-path=/srv/bricks/00/d}]
[2022-08-09 05:58:12.094440 +] I [MSGID: 100011]
[glusterfsd.c:1511:reincarnate] 0-glusterfsd: Fetching the volume file
from server... []
[2022-08-09 05:58:12.096299 +] I
[glusterfsd-mgmt.c:2170:mgmt_getspec_cbk] 0-glusterfs: Received list of
available volfile servers: clustor00:24007 clustor02:24007
[2022-08-09 05:58:12.096653 +] I [MSGID: 101221]
[common-utils.c:3851:gf_set_volfile_server_common] 0-gluster: duplicate
entry for volfile-server [{errno=17}, {error=File già esistente}]
[2022-08-09 05:58:12.096853 +] I
[glusterfsd-mgmt.c:2203:mgmt_getspec_cbk] 0-glusterfs: No change in
volfile,continuing
[2022-08-09 05:58:12.096702 +] I [MSGID: 101221]
[common-utils.c:3851:gf_set_volfile_server_common] 0-gluster: duplicate
entry for volfile-server [{errno=17}, {error=File già esistente}]
[2022-08-09 05:58:12.102176 +] I [MSGID: 118016]
[bit-rot.c:1052:bitd_oneshot_crawl] 0-cluster_data-bit-rot-0:
Triggering
signing [{path=/astro/...omisis.../file.dat},
{gfid=45f59e3f-eef4-4ccf-baac-bc8bf10c5ced},
{Brick-path=/srv/bricks/09/d}]
[2022-08-09 05:58:12.106120 +] I [MSGID: 118016]
[bit-rot.c:1052:bitd_oneshot_crawl] 0-cluster_data-bit-rot-0:
Triggering
signing [{path=/astro/...omisis.../file.txt},
{gfid=216832dd-0a1c-4593-8a9e-f54d70efc637},
{Brick-path=/srv/bricks/13/d}]
-8<--
And from quotad.log:
-<--
[2022-08-09 05:58:12.291030 +] I
[glusterfsd-mgmt.c:2170:mgmt_getspec_cbk] 0-glusterfs: Received list of
available volfile servers: clustor00:24007 clustor02:24007
[2022-08-09 05:58:12.291143 +] I [MSGID: 101221]
[common-utils.c:3851:gf_set_volfile_server_common] 0-gluster: duplicate
entry for volfile-server [{errno=17}, {error=File già esistente}]
[2022-08-09 05:58:12.291653 +] I
[glusterfsd-mgmt.c:2203:mgmt_getspec_cbk] 0-glusterfs: No change in
volfile,continuing
[2022-08-09 05:58:12.292990 +] I
[glusterfsd-mgmt.c:2170:mgmt_getspec_cbk] 0-glusterfs: Received list of
available volfile servers: clustor00:24007 clustor02:24007
[2022-08-09 05:58:12.293204 +] I
[glusterfsd-mgmt.c:2170:mgmt_getspec_cbk] 0-glusterfs: Received list of
available volfile servers: clustor00:24007 clustor02:24007
[2022-08-09 05:58:12.293500 +] I
[glusterfsd-mgmt.c:2203:mgmt_getspec_cbk] 0-glusterfs: No change in
volfile,continuing
[2022-08-09
