Re: [pmacct-discussion] pmbgpd -> Kafka Local Queue Full
Hi, Thanks for replying so quickly. The Kafka server has plenty of headroom left, and there is no barrier to producing and consuming to the topic (though I used kafkacat as I find it a lot easier than kafka-console-*.sh) However, as usual you nailed it with a suggestion - upgrading librdkafka has brought the feed to life. Debian Stable had these packages: ii pmacct 1.7.2-3 amd64 promiscuous mode traffic accountant ii librdkafka1:amd64 0.11.6-1.1 amd64 library implementing the Apache Kafka protocol Debian testing has these: ii pmacct 1.7.5-1 amd64 promiscuous mode traffic accountant ii librdkafka1:amd64 1.4.2-1 amd64 library implementing the Apache Kafka protocol Although most of our estate runs stable, it's all containerised nicely so I can run this service in a testing container simply. It's worth noting that I had to stop and start my pmbgpd system service to make this work, not just 'restart'. Thanks Andy On 02/09/2020, 13:47, "Paolo Lucente" wrote: Hi Andy, I may suggest to check Kafka logs and perhaps see if anything useful comes out of librdkafka stats (ie. set "global, statistics.interval.ms, 6" in your librdkafka.conf). Check also that, if you are adding load to existing load, the Kafka broker is not pegging 100% CPU or maxing out some threads count (or perhaps, if this is a testing environment, remove the existing load and test with only the pmbgpd export .. that may proof something too). My first suggestion would have been to tune buffers in librdkafka but you did that already. In any case this is, yes, an interaction between librdkafka and the Kafka broker; i go a bit errands here: make sure you have recent versions of both the library and the broker. Especially if the topic is newly provisioned i may also suggest to try to produce / consume some data "by hand", like using the kafka-console-producer.sh and kafka-console-consumer.sh scripts shipped with Kafka to proof data passing through no problem. Paolo On 02/09/2020 09:09, Andy Davidson wrote: > Hello! > > I am feeding some BMP feeds via pmbmpd into Kafka and it’s working well. I now want to feed some BGP feeds into a Kafka topic using pmbgpd but similar configuration is causing a different behaviour. > > Sep 1 22:48:00 bump pmbgpd[10992]: INFO ( default/core ): Reading configuration file '/etc/pmacct/pmbgpd.conf'. > Sep 1 22:48:00 bump pmbgpd[10992]: INFO ( default/core ): maximum BGP peers allowed: 100 > Sep 1 22:48:00 bump pmbgpd[10992]: INFO ( default/core ): waiting for BGP data on 185.1.94.6:179 > Sep 1 22:48:03 bump pmbgpd[10992]: INFO ( default/core ): [185.1.94.1] BGP peers usage: 1/100 > Sep 1 22:48:03 bump pmbgpd[10992]: INFO ( default/core ): [185.1.94.1] Capability: MultiProtocol [1] AFI [1] SAFI [1] > Sep 1 22:48:03 bump pmbgpd[10992]: INFO ( default/core ): [185.1.94.1] Capability: 4-bytes AS [41] ASN [59964] > Sep 1 22:48:03 bump pmbgpd[10992]: INFO ( default/core ): [185.1.94.1] BGP_OPEN: Local AS: 43470 Remote AS: 59964 HoldTime: 240 > Sep 1 22:48:05 bump pmbgpd[10992]: ERROR ( default/core ): Failed to produce to topic bgptest partition -1: Local: Queue full > Sep 1 22:48:05 bump pmbgpd[10992]: ERROR ( default/core ): Connection failed to Kafka: p_kafka_close() > Sep 1 22:48:05 bump systemd[1]: pmbgpd.service: Main process exited, code=killed, status=11/SEGV > Sep 1 22:48:05 bump systemd[1]: pmbgpd.service: Failed with result 'signal'. > > I have verified that it's not connectivity - the topic is created at the Kafka end of the link, and I can open a tcp socket with telnet from the computer running pmbgpd and the Kafka server's port 9092 > > I have of course read some Github issues and list archives about the Local: Queue full fault and it suggests some librdkafka buffer and timer tweaking, I have played with various values (some of them insane) and I don't see any different behaviour logged by pmbgpd: > > root@bump:/home/andy# cat /etc/pmacct/pmbgpd.conf > bgp_daemon_ip: 185.1.94.6 > bgp_daemon_max_peers: 100 > bgp_daemon_as: 43470 > ! > syslog: user > daemonize: true > ! > kafka_config_file: /etc/pmacct/librdkafka.conf > ! > bgp_daemon_msglog_kafka_output: json > bgp_daemon_msglog_kafka_broker_host: .hostname > bgp_daemon_msglog_kafka_broker_port: 9092 > bgp_daemon_msglog_kafka_topic: bgptest > > root@bump:/home/andy# cat /etc/pmacct/librdkafka.conf > global, queue.buffering.max.messages, 800 > global, batch.num.messages, 10 > global, queue.buffering.max.messages, 2 > global, queue.buffering.max.ms, 100 > global, queue.buffer
Re: [pmacct-discussion] pmbgpd -> Kafka Local Queue Full
Hi Andy, I may suggest to check Kafka logs and perhaps see if anything useful comes out of librdkafka stats (ie. set "global, statistics.interval.ms, 6" in your librdkafka.conf). Check also that, if you are adding load to existing load, the Kafka broker is not pegging 100% CPU or maxing out some threads count (or perhaps, if this is a testing environment, remove the existing load and test with only the pmbgpd export .. that may proof something too). My first suggestion would have been to tune buffers in librdkafka but you did that already. In any case this is, yes, an interaction between librdkafka and the Kafka broker; i go a bit errands here: make sure you have recent versions of both the library and the broker. Especially if the topic is newly provisioned i may also suggest to try to produce / consume some data "by hand", like using the kafka-console-producer.sh and kafka-console-consumer.sh scripts shipped with Kafka to proof data passing through no problem. Paolo On 02/09/2020 09:09, Andy Davidson wrote: Hello! I am feeding some BMP feeds via pmbmpd into Kafka and it’s working well. I now want to feed some BGP feeds into a Kafka topic using pmbgpd but similar configuration is causing a different behaviour. Sep 1 22:48:00 bump pmbgpd[10992]: INFO ( default/core ): Reading configuration file '/etc/pmacct/pmbgpd.conf'. Sep 1 22:48:00 bump pmbgpd[10992]: INFO ( default/core ): maximum BGP peers allowed: 100 Sep 1 22:48:00 bump pmbgpd[10992]: INFO ( default/core ): waiting for BGP data on 185.1.94.6:179 Sep 1 22:48:03 bump pmbgpd[10992]: INFO ( default/core ): [185.1.94.1] BGP peers usage: 1/100 Sep 1 22:48:03 bump pmbgpd[10992]: INFO ( default/core ): [185.1.94.1] Capability: MultiProtocol [1] AFI [1] SAFI [1] Sep 1 22:48:03 bump pmbgpd[10992]: INFO ( default/core ): [185.1.94.1] Capability: 4-bytes AS [41] ASN [59964] Sep 1 22:48:03 bump pmbgpd[10992]: INFO ( default/core ): [185.1.94.1] BGP_OPEN: Local AS: 43470 Remote AS: 59964 HoldTime: 240 Sep 1 22:48:05 bump pmbgpd[10992]: ERROR ( default/core ): Failed to produce to topic bgptest partition -1: Local: Queue full Sep 1 22:48:05 bump pmbgpd[10992]: ERROR ( default/core ): Connection failed to Kafka: p_kafka_close() Sep 1 22:48:05 bump systemd[1]: pmbgpd.service: Main process exited, code=killed, status=11/SEGV Sep 1 22:48:05 bump systemd[1]: pmbgpd.service: Failed with result 'signal'. I have verified that it's not connectivity - the topic is created at the Kafka end of the link, and I can open a tcp socket with telnet from the computer running pmbgpd and the Kafka server's port 9092 I have of course read some Github issues and list archives about the Local: Queue full fault and it suggests some librdkafka buffer and timer tweaking, I have played with various values (some of them insane) and I don't see any different behaviour logged by pmbgpd: root@bump:/home/andy# cat /etc/pmacct/pmbgpd.conf bgp_daemon_ip: 185.1.94.6 bgp_daemon_max_peers: 100 bgp_daemon_as: 43470 ! syslog: user daemonize: true ! kafka_config_file: /etc/pmacct/librdkafka.conf ! bgp_daemon_msglog_kafka_output: json bgp_daemon_msglog_kafka_broker_host: .hostname bgp_daemon_msglog_kafka_broker_port: 9092 bgp_daemon_msglog_kafka_topic: bgptest root@bump:/home/andy# cat /etc/pmacct/librdkafka.conf global, queue.buffering.max.messages, 800 global, batch.num.messages, 10 global, queue.buffering.max.messages, 2 global, queue.buffering.max.ms, 100 global, queue.buffering.max.kbytes, 900 global, linger.ms, 100 global, socket.request.max.bytes, 104857600 global, socket.receive.buffer.bytes, 10485760 global, socket.send.buffer.bytes, 10485760 global, queued.max.requests, 1000 Any advice on where to troubleshoot next? Thanks Andy ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
[pmacct-discussion] pmbgpd -> Kafka Local Queue Full
Hello! I am feeding some BMP feeds via pmbmpd into Kafka and it’s working well. I now want to feed some BGP feeds into a Kafka topic using pmbgpd but similar configuration is causing a different behaviour. Sep 1 22:48:00 bump pmbgpd[10992]: INFO ( default/core ): Reading configuration file '/etc/pmacct/pmbgpd.conf'. Sep 1 22:48:00 bump pmbgpd[10992]: INFO ( default/core ): maximum BGP peers allowed: 100 Sep 1 22:48:00 bump pmbgpd[10992]: INFO ( default/core ): waiting for BGP data on 185.1.94.6:179 Sep 1 22:48:03 bump pmbgpd[10992]: INFO ( default/core ): [185.1.94.1] BGP peers usage: 1/100 Sep 1 22:48:03 bump pmbgpd[10992]: INFO ( default/core ): [185.1.94.1] Capability: MultiProtocol [1] AFI [1] SAFI [1] Sep 1 22:48:03 bump pmbgpd[10992]: INFO ( default/core ): [185.1.94.1] Capability: 4-bytes AS [41] ASN [59964] Sep 1 22:48:03 bump pmbgpd[10992]: INFO ( default/core ): [185.1.94.1] BGP_OPEN: Local AS: 43470 Remote AS: 59964 HoldTime: 240 Sep 1 22:48:05 bump pmbgpd[10992]: ERROR ( default/core ): Failed to produce to topic bgptest partition -1: Local: Queue full Sep 1 22:48:05 bump pmbgpd[10992]: ERROR ( default/core ): Connection failed to Kafka: p_kafka_close() Sep 1 22:48:05 bump systemd[1]: pmbgpd.service: Main process exited, code=killed, status=11/SEGV Sep 1 22:48:05 bump systemd[1]: pmbgpd.service: Failed with result 'signal'. I have verified that it's not connectivity - the topic is created at the Kafka end of the link, and I can open a tcp socket with telnet from the computer running pmbgpd and the Kafka server's port 9092 I have of course read some Github issues and list archives about the Local: Queue full fault and it suggests some librdkafka buffer and timer tweaking, I have played with various values (some of them insane) and I don't see any different behaviour logged by pmbgpd: root@bump:/home/andy# cat /etc/pmacct/pmbgpd.conf bgp_daemon_ip: 185.1.94.6 bgp_daemon_max_peers: 100 bgp_daemon_as: 43470 ! syslog: user daemonize: true ! kafka_config_file: /etc/pmacct/librdkafka.conf ! bgp_daemon_msglog_kafka_output: json bgp_daemon_msglog_kafka_broker_host: .hostname bgp_daemon_msglog_kafka_broker_port: 9092 bgp_daemon_msglog_kafka_topic: bgptest root@bump:/home/andy# cat /etc/pmacct/librdkafka.conf global, queue.buffering.max.messages, 800 global, batch.num.messages, 10 global, queue.buffering.max.messages, 2 global, queue.buffering.max.ms, 100 global, queue.buffering.max.kbytes, 900 global, linger.ms, 100 global, socket.request.max.bytes, 104857600 global, socket.receive.buffer.bytes, 10485760 global, socket.send.buffer.bytes, 10485760 global, queued.max.requests, 1000 Any advice on where to troubleshoot next? Thanks Andy ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
