Re: [Gluster-users] bitd.log and quotad.log flooding /var
Seems it's accumulating again. ATM it's like this:
root 2134553 2.1 11.2 23071940 22091644 ? Ssl set23 1059:58
/usr/sbin/glusterfs -s localhost --volfile-id gluster/quotad -p
/var/run/gluster/quotad/quotad.pid -l /var/log/glusterfs/quotad.log -S
/var/run/gluster/321cad6822171c64.socket --process-name quotad
Uptime is 77d.
The other 2 nodes are in the same situation.
Gluster is 9.5-1 amd64. Is it latest enough or should I plan a migration
to 10?
Hints?
Diego
Il 12/08/2022 22:18, Strahil Nikolov ha scritto:
75GB -> that's definately a memory leak.
What version do you use ?
If latest - open a github issue.
Best Regards,
Strahil Nikolov
On Thu, Aug 11, 2022 at 10:06, Diego Zuccato
wrote:
Yup.
Seems the /etc/sysconfig/glusterd setting got finally applied and I now
have a process like this:
root 4107315 0.0 0.0 529244 40124 ? Ssl ago08 2:44
/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level ERROR
but bitd still spits out (some) 'I' lines
[2022-08-11 07:02:21.072943 +] I [MSGID: 118016]
[bit-rot.c:1052:bitd_oneshot_crawl] 0-cluster_data-bit-rot-0:
Triggering
signing [{path=/extra/some/other/dirs/file.dat},
{gfid=3e35b158-35a6-4e63-adbd-41075a11022e},
{Brick-path=/srv/bricks/00/d}]
Moreover I've had to disable quota, since quota processes were eating
more than *75GB* RAM on each storage node! :(
Il 11/08/2022 07:12, Strahil Nikolov ha scritto:
> Have you decreased glusterd log level via:
> glusterd --log-level WARNING|ERROR
>
> It seems that bitrot doesn't have it's own log level.
>
> As a workaround, you can configure syslog to send the logs only
remotely
> and thus preventing the overfill of the /var .
>
>
> Best Regards,
> Strahil Nikolov
>
> On Wed, Aug 10, 2022 at 7:52, Diego Zuccato
> mailto:diego.zucc...@unibo.it>> wrote:
> Hi Strahil.
>
> Sure. Luckily I didn't delete 'em all :)
>
> From bitd.log:
> -8<--
> [2022-08-09 05:58:12.075999 +] I [MSGID: 118016]
> [bit-rot.c:1052:bitd_oneshot_crawl] 0-cluster_data-bit-rot-0:
> Triggering
> signing [{path=/astro/...omisis.../file.dat},
> {gfid=5956af24-5efc-496c-8d7e-ea6656f298de},
> {Brick-path=/srv/bricks/10/d}]
> [2022-08-09 05:58:12.082264 +] I [MSGID: 118016]
> [bit-rot.c:1052:bitd_oneshot_crawl] 0-cluster_data-bit-rot-0:
> Triggering
> signing [{path=/astro/...omisis.../file.txt},
> {gfid=afb75c03-0d29-414e-917a-ff718982c849},
> {Brick-path=/srv/bricks/13/d}]
> [2022-08-09 05:58:12.082267 +] I [MSGID: 118016]
> [bit-rot.c:1052:bitd_oneshot_crawl] 0-cluster_data-bit-rot-0:
> Triggering
> signing [{path=/astro/...omisis.../file.dat},
> {gfid=982bc7a8-d4ba-45d7-9104-044e5d446802},
> {Brick-path=/srv/bricks/06/d}]
> [2022-08-09 05:58:12.084960 +] I [MSGID: 118016]
> [bit-rot.c:1052:bitd_oneshot_crawl] 0-cluster_data-bit-rot-0:
> Triggering
> signing [{path=/atmos/...omisis.../file},
> {gfid=17e4dfb0-1f64-47a3-9aa8-b3fa05b7cd4e},
> {Brick-path=/srv/bricks/15/d}]
> [2022-08-09 05:58:12.089357 +] I [MSGID: 118016]
> [bit-rot.c:1052:bitd_oneshot_crawl] 0-cluster_data-bit-rot-0:
> Triggering
> signing [{path=/astro/...omisis.../file.txt},
> {gfid=e70bf289-5aeb-43c2-aadd-d18979cf62b5},
> {Brick-path=/srv/bricks/00/d}]
> [2022-08-09 05:58:12.094440 +] I [MSGID: 100011]
> [glusterfsd.c:1511:reincarnate] 0-glusterfsd: Fetching the
volume file
> from server... []
> [2022-08-09 05:58:12.096299 +] I
> [glusterfsd-mgmt.c:2170:mgmt_getspec_cbk] 0-glusterfs:
Received list of
> available volfile servers: clustor00:24007 clustor02:24007
> [2022-08-09 05:58:12.096653 +] I [MSGID: 101221]
> [common-utils.c:3851:gf_set_volfile_server_common] 0-gluster:
duplicate
> entry for volfile-server [{errno=17}, {error=File già esistente}]
> [2022-08-09 05:58:12.096853 +] I
> [glusterfsd-mgmt.c:2203:mgmt_getspec_cbk] 0-glusterfs: No
change in
> volfile,continuing
> [2022-08-09 05:58:12.096702 +] I [MSGID: 101221]
> [common-utils.c:3851:gf_set_volfile_server_common] 0-gluster:
duplicate
> entry for volfile-server [{errno=17}, {error=File già esistente}]
> [2022-08-09 05:58:12.102176 +] I [MSGID: 118016]
> [bit-rot.c:1052:bitd_oneshot_crawl] 0-cluster_data-bit-rot-0:
> Triggering
> signing [{path=/astro/...omisis.../file.dat},
> {gfid=45f59e3f-eef4-4ccf-baac-bc8bf10c5ced},
> {Brick-path=/srv/bricks/09/d}]
> [2022-08-09 05:58:12.106120 +] I [MSGID: 118016]
>
Re: [Gluster-users] bitd.log and quotad.log flooding /var
Yup.
Seems the /etc/sysconfig/glusterd setting got finally applied and I now
have a process like this:
root 4107315 0.0 0.0 529244 40124 ?Ssl ago08 2:44
/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level ERROR
but bitd still spits out (some) 'I' lines
[2022-08-11 07:02:21.072943 +] I [MSGID: 118016]
[bit-rot.c:1052:bitd_oneshot_crawl] 0-cluster_data-bit-rot-0: Triggering
signing [{path=/extra/some/other/dirs/file.dat},
{gfid=3e35b158-35a6-4e63-adbd-41075a11022e}, {Brick-path=/srv/bricks/00/d}]
Moreover I've had to disable quota, since quota processes were eating
more than *75GB* RAM on each storage node! :(
Il 11/08/2022 07:12, Strahil Nikolov ha scritto:
Have you decreased glusterd log level via:
glusterd --log-level WARNING|ERROR
It seems that bitrot doesn't have it's own log level.
As a workaround, you can configure syslog to send the logs only remotely
and thus preventing the overfill of the /var .
Best Regards,
Strahil Nikolov
On Wed, Aug 10, 2022 at 7:52, Diego Zuccato
wrote:
Hi Strahil.
Sure. Luckily I didn't delete 'em all :)
From bitd.log:
-8<--
[2022-08-09 05:58:12.075999 +] I [MSGID: 118016]
[bit-rot.c:1052:bitd_oneshot_crawl] 0-cluster_data-bit-rot-0:
Triggering
signing [{path=/astro/...omisis.../file.dat},
{gfid=5956af24-5efc-496c-8d7e-ea6656f298de},
{Brick-path=/srv/bricks/10/d}]
[2022-08-09 05:58:12.082264 +] I [MSGID: 118016]
[bit-rot.c:1052:bitd_oneshot_crawl] 0-cluster_data-bit-rot-0:
Triggering
signing [{path=/astro/...omisis.../file.txt},
{gfid=afb75c03-0d29-414e-917a-ff718982c849},
{Brick-path=/srv/bricks/13/d}]
[2022-08-09 05:58:12.082267 +] I [MSGID: 118016]
[bit-rot.c:1052:bitd_oneshot_crawl] 0-cluster_data-bit-rot-0:
Triggering
signing [{path=/astro/...omisis.../file.dat},
{gfid=982bc7a8-d4ba-45d7-9104-044e5d446802},
{Brick-path=/srv/bricks/06/d}]
[2022-08-09 05:58:12.084960 +] I [MSGID: 118016]
[bit-rot.c:1052:bitd_oneshot_crawl] 0-cluster_data-bit-rot-0:
Triggering
signing [{path=/atmos/...omisis.../file},
{gfid=17e4dfb0-1f64-47a3-9aa8-b3fa05b7cd4e},
{Brick-path=/srv/bricks/15/d}]
[2022-08-09 05:58:12.089357 +] I [MSGID: 118016]
[bit-rot.c:1052:bitd_oneshot_crawl] 0-cluster_data-bit-rot-0:
Triggering
signing [{path=/astro/...omisis.../file.txt},
{gfid=e70bf289-5aeb-43c2-aadd-d18979cf62b5},
{Brick-path=/srv/bricks/00/d}]
[2022-08-09 05:58:12.094440 +] I [MSGID: 100011]
[glusterfsd.c:1511:reincarnate] 0-glusterfsd: Fetching the volume file
from server... []
[2022-08-09 05:58:12.096299 +] I
[glusterfsd-mgmt.c:2170:mgmt_getspec_cbk] 0-glusterfs: Received list of
available volfile servers: clustor00:24007 clustor02:24007
[2022-08-09 05:58:12.096653 +] I [MSGID: 101221]
[common-utils.c:3851:gf_set_volfile_server_common] 0-gluster: duplicate
entry for volfile-server [{errno=17}, {error=File già esistente}]
[2022-08-09 05:58:12.096853 +] I
[glusterfsd-mgmt.c:2203:mgmt_getspec_cbk] 0-glusterfs: No change in
volfile,continuing
[2022-08-09 05:58:12.096702 +] I [MSGID: 101221]
[common-utils.c:3851:gf_set_volfile_server_common] 0-gluster: duplicate
entry for volfile-server [{errno=17}, {error=File già esistente}]
[2022-08-09 05:58:12.102176 +] I [MSGID: 118016]
[bit-rot.c:1052:bitd_oneshot_crawl] 0-cluster_data-bit-rot-0:
Triggering
signing [{path=/astro/...omisis.../file.dat},
{gfid=45f59e3f-eef4-4ccf-baac-bc8bf10c5ced},
{Brick-path=/srv/bricks/09/d}]
[2022-08-09 05:58:12.106120 +] I [MSGID: 118016]
[bit-rot.c:1052:bitd_oneshot_crawl] 0-cluster_data-bit-rot-0:
Triggering
signing [{path=/astro/...omisis.../file.txt},
{gfid=216832dd-0a1c-4593-8a9e-f54d70efc637},
{Brick-path=/srv/bricks/13/d}]
-8<--
And from quotad.log:
-<--
[2022-08-09 05:58:12.291030 +] I
[glusterfsd-mgmt.c:2170:mgmt_getspec_cbk] 0-glusterfs: Received list of
available volfile servers: clustor00:24007 clustor02:24007
[2022-08-09 05:58:12.291143 +] I [MSGID: 101221]
[common-utils.c:3851:gf_set_volfile_server_common] 0-gluster: duplicate
entry for volfile-server [{errno=17}, {error=File già esistente}]
[2022-08-09 05:58:12.291653 +] I
[glusterfsd-mgmt.c:2203:mgmt_getspec_cbk] 0-glusterfs: No change in
volfile,continuing
[2022-08-09 05:58:12.292990 +] I
[glusterfsd-mgmt.c:2170:mgmt_getspec_cbk] 0-glusterfs: Received list of
available volfile servers: clustor00:24007 clustor02:24007
[2022-08-09 05:58:12.293204 +] I
[glusterfsd-mgmt.c:2170:mgmt_getspec_cbk] 0-glusterfs: Received list of
available volfile servers: clustor00:24007 clustor02:24007
[2022-08-09 05:58:12.293500 +] I
[glusterfsd-mgmt.c:2203:mgmt_getspec_cbk] 0-glusterfs: No change in
volfile,continuing
[2022-08-09
Re: [Gluster-users] bitd.log and quotad.log flooding /var
Hi Strahil.
Sure. Luckily I didn't delete 'em all :)
From bitd.log:
-8<--
[2022-08-09 05:58:12.075999 +] I [MSGID: 118016]
[bit-rot.c:1052:bitd_oneshot_crawl] 0-cluster_data-bit-rot-0: Triggering
signing [{path=/astro/...omisis.../file.dat},
{gfid=5956af24-5efc-496c-8d7e-ea6656f298de}, {Brick-path=/srv/bricks/10/d}]
[2022-08-09 05:58:12.082264 +] I [MSGID: 118016]
[bit-rot.c:1052:bitd_oneshot_crawl] 0-cluster_data-bit-rot-0: Triggering
signing [{path=/astro/...omisis.../file.txt},
{gfid=afb75c03-0d29-414e-917a-ff718982c849}, {Brick-path=/srv/bricks/13/d}]
[2022-08-09 05:58:12.082267 +] I [MSGID: 118016]
[bit-rot.c:1052:bitd_oneshot_crawl] 0-cluster_data-bit-rot-0: Triggering
signing [{path=/astro/...omisis.../file.dat},
{gfid=982bc7a8-d4ba-45d7-9104-044e5d446802}, {Brick-path=/srv/bricks/06/d}]
[2022-08-09 05:58:12.084960 +] I [MSGID: 118016]
[bit-rot.c:1052:bitd_oneshot_crawl] 0-cluster_data-bit-rot-0: Triggering
signing [{path=/atmos/...omisis.../file},
{gfid=17e4dfb0-1f64-47a3-9aa8-b3fa05b7cd4e}, {Brick-path=/srv/bricks/15/d}]
[2022-08-09 05:58:12.089357 +] I [MSGID: 118016]
[bit-rot.c:1052:bitd_oneshot_crawl] 0-cluster_data-bit-rot-0: Triggering
signing [{path=/astro/...omisis.../file.txt},
{gfid=e70bf289-5aeb-43c2-aadd-d18979cf62b5}, {Brick-path=/srv/bricks/00/d}]
[2022-08-09 05:58:12.094440 +] I [MSGID: 100011]
[glusterfsd.c:1511:reincarnate] 0-glusterfsd: Fetching the volume file
from server... []
[2022-08-09 05:58:12.096299 +] I
[glusterfsd-mgmt.c:2170:mgmt_getspec_cbk] 0-glusterfs: Received list of
available volfile servers: clustor00:24007 clustor02:24007
[2022-08-09 05:58:12.096653 +] I [MSGID: 101221]
[common-utils.c:3851:gf_set_volfile_server_common] 0-gluster: duplicate
entry for volfile-server [{errno=17}, {error=File già esistente}]
[2022-08-09 05:58:12.096853 +] I
[glusterfsd-mgmt.c:2203:mgmt_getspec_cbk] 0-glusterfs: No change in
volfile,continuing
[2022-08-09 05:58:12.096702 +] I [MSGID: 101221]
[common-utils.c:3851:gf_set_volfile_server_common] 0-gluster: duplicate
entry for volfile-server [{errno=17}, {error=File già esistente}]
[2022-08-09 05:58:12.102176 +] I [MSGID: 118016]
[bit-rot.c:1052:bitd_oneshot_crawl] 0-cluster_data-bit-rot-0: Triggering
signing [{path=/astro/...omisis.../file.dat},
{gfid=45f59e3f-eef4-4ccf-baac-bc8bf10c5ced}, {Brick-path=/srv/bricks/09/d}]
[2022-08-09 05:58:12.106120 +] I [MSGID: 118016]
[bit-rot.c:1052:bitd_oneshot_crawl] 0-cluster_data-bit-rot-0: Triggering
signing [{path=/astro/...omisis.../file.txt},
{gfid=216832dd-0a1c-4593-8a9e-f54d70efc637}, {Brick-path=/srv/bricks/13/d}]
-8<--
And from quotad.log:
-<--
[2022-08-09 05:58:12.291030 +] I
[glusterfsd-mgmt.c:2170:mgmt_getspec_cbk] 0-glusterfs: Received list of
available volfile servers: clustor00:24007 clustor02:24007
[2022-08-09 05:58:12.291143 +] I [MSGID: 101221]
[common-utils.c:3851:gf_set_volfile_server_common] 0-gluster: duplicate
entry for volfile-server [{errno=17}, {error=File già esistente}]
[2022-08-09 05:58:12.291653 +] I
[glusterfsd-mgmt.c:2203:mgmt_getspec_cbk] 0-glusterfs: No change in
volfile,continuing
[2022-08-09 05:58:12.292990 +] I
[glusterfsd-mgmt.c:2170:mgmt_getspec_cbk] 0-glusterfs: Received list of
available volfile servers: clustor00:24007 clustor02:24007
[2022-08-09 05:58:12.293204 +] I
[glusterfsd-mgmt.c:2170:mgmt_getspec_cbk] 0-glusterfs: Received list of
available volfile servers: clustor00:24007 clustor02:24007
[2022-08-09 05:58:12.293500 +] I
[glusterfsd-mgmt.c:2203:mgmt_getspec_cbk] 0-glusterfs: No change in
volfile,continuing
[2022-08-09 05:58:12.293663 +] I
[glusterfsd-mgmt.c:2203:mgmt_getspec_cbk] 0-glusterfs: No change in
volfile,continuing
The message "I [MSGID: 100011] [glusterfsd.c:1511:reincarnate]
0-glusterfsd: Fetching the volume file from server... []" repeated 2
times between [2022-08-09 05:58:12.094470 +] and [2022-08-09
05:58:12.291149 +]
The message "I [MSGID: 101221]
[common-utils.c:3851:gf_set_volfile_server_common] 0-gluster: duplicate
entry for volfile-server [{errno=17}, {error=File già esistente}]"
repeated 5 times between [2022-08-09 05:58:12.291143 +] and
[2022-08-09 05:58:12.293234 +]
[2022-08-09 06:00:23.180856 +] I
[glusterfsd-mgmt.c:77:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2022-08-09 06:00:23.324981 +] I
[glusterfsd-mgmt.c:2170:mgmt_getspec_cbk] 0-glusterfs: Received list of
available volfile servers: clustor00:24007 clustor02:24007
[2022-08-09 06:00:23.325025 +] I [MSGID: 101221]
[common-utils.c:3851:gf_set_volfile_server_common] 0-gluster: duplicate
entry for volfile-server [{errno=17}, {error=File già esistente}]
[2022-08-09 06:00:23.325498 +] I
[glusterfsd-mgmt.c:2203:mgmt_getspec_cbk] 0-glusterfs: No change in
volfile,continuing
[2022-08-09 06:00:23.325046 +] I [MSGID: 101221]
[common-utils.c:3851:gf_set_volfile_server_common] 0-gluster: duplicate
entry for
