Hi Yes I know I have monstrous queues, Im still probing with several configurations and loads.
So before rsyslogd starts I have : On Kafka Consumer group 'nms-pit' has no active members. nms-pit syslog-global 0 158036144 158425372 389228 - - - nms-pit syslog-global 1 158017045 158406807 389762 - - - nms-pit syslog-global 2 158023011 158410967 387956 - - - On the filesystems: Device r/s w/s rkB/s wkB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdd1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdj 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Device r/s w/s rkB/s wkB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdd1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdj 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Device r/s w/s rkB/s wkB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util sdd 0.00 0.20 0.00 0.40 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 2.00 2.00 0.04 sdd1 0.00 0.20 0.00 0.40 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 2.00 2.00 0.04 sdj 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Device r/s w/s rkB/s wkB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdd1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdj 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdd / sdd1 = spool fs, where the rsyslog stores queue on shutdown sdj = Write fs, where the rsyslogd stores the logs it receives from kafka. When rsyslogd starts: On kafka we can see the consumers: nms-pit syslog-global 0 158220675 158974807 754132 rdkafka-1bf57873-da82-4a1e-9cc7-4a98e17e0340 /x.x.x.103 rdkafka nms-pit syslog-global 1 158566106 158956792 390686 rdkafka-1bf57873-da82-4a1e-9cc7-4a98e17e0340 /x.x.x.103 rdkafka nms-pit syslog-global 2 158206768 158961218 754450 rdkafka-1bf57873-da82-4a1e-9cc7-4a98e17e0340 /x.x.x.103 rdkafka On the filesystems we can see activity also: Device r/s w/s rkB/s wkB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdd1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdj 0.00 30.60 0.00 9673.60 0.00 31.00 0.00 50.32 0.00 13.41 0.39 0.00 316.13 0.74 2.26 Device r/s w/s rkB/s wkB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util sdd 0.00 0.60 0.00 0.50 0.00 0.00 0.00 0.00 0.00 0.67 0.00 0.00 0.83 1.33 0.08 sdd1 0.00 0.60 0.00 0.50 0.00 0.00 0.00 0.00 0.00 0.67 0.00 0.00 0.83 1.33 0.08 sdj 0.00 35.80 0.00 524.00 0.00 57.60 0.00 61.67 0.00 1.02 0.02 0.00 14.64 0.11 0.40 Device r/s w/s rkB/s wkB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdd1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdj 0.00 4.40 0.00 174.40 0.00 9.60 0.00 68.57 0.00 1.41 0.00 0.00 39.64 0.36 0.16 Device r/s w/s rkB/s wkB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util sdd 0.20 0.00 0.80 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 4.00 0.00 2.00 0.04 sdd1 0.20 0.00 0.80 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 4.00 0.00 2.00 0.04 sdj 0.00 2.20 0.00 27.20 0.00 4.60 0.00 67.65 0.00 0.91 0.00 0.00 12.36 0.45 0.10 Device r/s w/s rkB/s wkB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdd1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdj 0.00 1.20 0.00 16.00 0.00 2.80 0.00 70.00 0.00 0.83 0.00 0.00 13.33 0.50 0.06 rsyslogd was running for about 20m After sending the systemctl stop rsyslog, after about 10-15 secs: On kafka I see no consumers now. Consumer group 'nms-pit' has no active members. nms-pit syslog-global 0 163396912 163680765 283853 - - - nms-pit syslog-global 1 163371712 163654818 283106 - - - nms-pit syslog-global 2 163382362 163666497 284135 - - - On the filesystems no activity: Device r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util sdd 0.00 0.20 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 16.00 2.00 0.04 sdd1 0.00 0.20 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 16.00 2.00 0.04 sdj 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Device r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdd1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdj 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Device r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdd1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdj 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Device r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdd1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdj 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Device r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdd1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdj 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Device r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util sdd 0.00 0.20 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 1.00 1.00 0.02 sdd1 0.00 0.20 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 1.00 1.00 0.02 sdj 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Device r/s w/s rMB/s wMB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util sdd 0.00 0.40 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.50 0.00 0.00 0.75 1.00 0.04 sdd1 0.00 0.40 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.50 0.00 0.00 0.75 1.00 0.04 sdj 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 And the rsyslog is still running. It gets killed by systemd after the timout. Im going to upgrade to the most recent version and see how it behaves. -- Sent from: http://rsyslog-users.1305293.n2.nabble.com/ _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
