Re: [pmacct-discussion] Added support for durable messages to amqp plugin
Hi, On 22-02-14 15:07, Paolo Lucente wrote: Thanks for the elaboration, very useful to know. Your patch has been incorporated in current CVS code. The feature is available via the amqp_persistent_msg configuration directive. Thanks. This version is currently running on our test setup, and has run stable over the weekend. Kind regards, Nick Douma signature.asc Description: OpenPGP digital signature ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Added support for durable messages to amqp plugin
Hi Nick, Thanks for the elaboration, very useful to know. Your patch has been incorporated in current CVS code. The feature is available via the amqp_persistent_msg configuration directive. Cheers, Paolo On Fri, Feb 21, 2014 at 09:54:40AM +0100, Nick Douma wrote: Hi, On 20-02-14 16:55, Paolo Lucente wrote: Thanks for your patches. Most of the minor issues were already fixed in the CVS code; i've fixed the couple remaining. The memory leak is not a real one: some structures are not free'd up in amqp_cache_purge() but such function is called in a writer, ephemeral processes spawned each amqp_refresh_time intervals: once these processes are terminated the OS does take back all resources anyway. About the new feature you implemented: i'm reviewing the code (you did a very good job) and i'm happy to commit it to the CVS code. Just one question: what is your use-case for durable message support? Something like be able to recover un-processed JSON objects in case of reboot/ restart of RabbitMQ? We currently have a setup that uses sfacct to read sFlow data, and uses rabbitmq with logstash to finally insert it into elasticsearch after some filtering by logstash. We use sFlow/sfacct to monitor and store data usage for all IPs traveling through our network. This data is then used to bill our customers. In this case, any loss of traffic information is unwanted, because that would either be unfair to our customers, or cut our revenue. By default, the sfacct / rabbitmq / logstash setup will drop any messages and clear all queues during reboot/restart. Logstash can create a rabbitmq queue as durable, but sfacct could not mark the messages durable as well. Both settings are needed to keep messages from being dropped in case logstash can't consume them, or any of the services are restarted. Kind regards, Nick Douma ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Added support for durable messages to amqp plugin
Hi, On 20-02-14 16:55, Paolo Lucente wrote: Thanks for your patches. Most of the minor issues were already fixed in the CVS code; i've fixed the couple remaining. The memory leak is not a real one: some structures are not free'd up in amqp_cache_purge() but such function is called in a writer, ephemeral processes spawned each amqp_refresh_time intervals: once these processes are terminated the OS does take back all resources anyway. About the new feature you implemented: i'm reviewing the code (you did a very good job) and i'm happy to commit it to the CVS code. Just one question: what is your use-case for durable message support? Something like be able to recover un-processed JSON objects in case of reboot/ restart of RabbitMQ? We currently have a setup that uses sfacct to read sFlow data, and uses rabbitmq with logstash to finally insert it into elasticsearch after some filtering by logstash. We use sFlow/sfacct to monitor and store data usage for all IPs traveling through our network. This data is then used to bill our customers. In this case, any loss of traffic information is unwanted, because that would either be unfair to our customers, or cut our revenue. By default, the sfacct / rabbitmq / logstash setup will drop any messages and clear all queues during reboot/restart. Logstash can create a rabbitmq queue as durable, but sfacct could not mark the messages durable as well. Both settings are needed to keep messages from being dropped in case logstash can't consume them, or any of the services are restarted. Kind regards, Nick Douma signature.asc Description: OpenPGP digital signature ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Added support for durable messages to amqp plugin
Hi Nick, Thanks for your patches. Most of the minor issues were already fixed in the CVS code; i've fixed the couple remaining. The memory leak is not a real one: some structures are not free'd up in amqp_cache_purge() but such function is called in a writer, ephemeral processes spawned each amqp_refresh_time intervals: once these processes are terminated the OS does take back all resources anyway. About the new feature you implemented: i'm reviewing the code (you did a very good job) and i'm happy to commit it to the CVS code. Just one question: what is your use-case for durable message support? Something like be able to recover un-processed JSON objects in case of reboot/ restart of RabbitMQ? Cheers, Paolo On Thu, Feb 20, 2014 at 10:56:09AM +0100, Nick Douma wrote: Hi, For our own use case of pmacct, I have added durable message support to the amqp plugin. I have formatted this change into a set of Quilt patches, so I can easily compile my changes into a proper Debian package. The patches introduce a new config variable for the amqp plugin called 'amqp_persistent_message'. If set to true, the delivery_mode of the message is set to 2, which will cause the message to be saved to disk by RabbitMQ. The patches are available on github at: https://github.com/LordGaav/pmacct/tree/master/debian/patches Prebuilt Ubuntu Trusty packages can be found on my Launchpad PPA: https://launchpad.net/~lordgaav/+archive/pmacct The patches are based against the 1.5.0rc2 tarball on pmacct.net. The proper order off applying can be found in the series file. The repository is fully Debian/Ubuntu compatible, and can be autobuilt using Debian standard tools. I can provide extra instructions if someone is interested, but I think the patches themselves are more appropriate for pmacct developers. I also tried to fix the few implicit function declaration warning that prevent the package from being built on Launchpad, and I tried to fix a few memory leaks in the amqp cache flush function. I give these patches back to you to use as you please, as required by GPL2. Small disclaimer: I am not a C programmer, so I tried my best to hack the wanted changes into pmacct. I tried to adhere to the style of the rest of the code. I'm interested in your feedback on my changes, as this has also partly been a learning experience for me. Kind regards, Nick Douma ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
