[pmacct-discussion] Added support for durable messages to amqp plugin

2014-02-20 Thread Nick Douma
Hi,

For our own use case of pmacct, I have added durable message support to
the amqp plugin. I have formatted this change into a set of Quilt
patches, so I can easily compile my changes into a proper Debian package.

The patches introduce a new config variable for the amqp plugin called
'amqp_persistent_message'. If set to true, the delivery_mode of the
message is set to 2, which will cause the message to be saved to disk by
RabbitMQ.

The patches are available on github at:

https://github.com/LordGaav/pmacct/tree/master/debian/patches

Prebuilt Ubuntu Trusty packages can be found on my Launchpad PPA:

https://launchpad.net/~lordgaav/+archive/pmacct

The patches are based against the 1.5.0rc2 tarball on pmacct.net. The
proper order off applying can be found in the series file. The
repository is fully Debian/Ubuntu compatible, and can be autobuilt using
Debian standard tools. I can provide extra instructions if someone is
interested, but I think the patches themselves are more appropriate for
pmacct developers.

I also tried to fix the few implicit function declaration warning that
prevent the package from being built on Launchpad, and I tried to fix a
few memory leaks in the amqp cache flush function.

I give these patches back to you to use as you please, as required by GPL2.

Small disclaimer: I am not a C programmer, so I tried my best to hack
the wanted changes into pmacct. I tried to adhere to the style of the
rest of the code. I'm interested in your feedback on my changes, as this
has also partly been a learning experience for me.

Kind regards,

Nick Douma



signature.asc
Description: OpenPGP digital signature
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Added support for durable messages to amqp plugin

2014-02-21 Thread Nick Douma
Hi,

On 20-02-14 16:55, Paolo Lucente wrote:
 Thanks for your patches. Most of the minor issues were already fixed in
 the CVS code; i've fixed the couple remaining. The memory leak is not a
 real one: some structures are not free'd up in amqp_cache_purge() but
 such function is called in a writer, ephemeral processes spawned each
 amqp_refresh_time intervals: once these processes are terminated the OS
 does take back all resources anyway.
 
 About the new feature you implemented: i'm reviewing the code (you did
 a very good job) and i'm happy to commit it to the CVS code. Just one
 question: what is your use-case for durable message support? Something
 like be able to recover un-processed JSON objects in case of reboot/
 restart of RabbitMQ?

We currently have a setup that uses sfacct to read sFlow data, and uses
rabbitmq with logstash to finally insert it into elasticsearch after
some filtering by logstash.

We use sFlow/sfacct to monitor and store data usage for all IPs
traveling through our network. This data is then used to bill our
customers. In this case, any loss of traffic information is unwanted,
because that would either be unfair to our customers, or cut our revenue.

By default, the sfacct / rabbitmq / logstash setup will drop any
messages and clear all queues during reboot/restart. Logstash can create
a rabbitmq queue as durable, but sfacct could not mark the messages
durable as well. Both settings are needed to keep messages from being
dropped in case logstash can't consume them, or any of the services are
restarted.

Kind regards,

Nick Douma



signature.asc
Description: OpenPGP digital signature
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Added support for durable messages to amqp plugin

2014-02-24 Thread Nick Douma
Hi,

On 22-02-14 15:07, Paolo Lucente wrote:
 Thanks for the elaboration, very useful to know. Your patch has
 been incorporated in current CVS code. The feature is available
 via the amqp_persistent_msg configuration directive.

Thanks. This version is currently running on our test setup, and has run
stable over the weekend.

Kind regards,

Nick Douma




signature.asc
Description: OpenPGP digital signature
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] pmacct -1.5.0rc3 rabbitmq integration issue.

2014-07-18 Thread Nick Douma
Hi,

On 17-07-14 20:08, Dhanasekaran Anbalagan wrote:
 I am trying to integrate pmacct with logstash. I already tired print
 plugin to write to flat-files it's working. Now I am trying to integrate
 with  rabbitmq. It's not working..
 
 I went to rabbitmq admin. I don't see any message queues. Also tired
 Debug mode 
 
 I seen some Json data printing. But actually It's not sending to
 rabbitmq exchange. But Exchange name created in rabbitmq end,
 
 console log:
 http://paste.ubuntu.com/7809820/

In your log I see:

... publishing [E=pmacct RK=5m_ipip DM=0] ...

I read this as publishing to Exchange pmacct with routing key
5m_ipip. Are you sure that your config is properly loaded by pmacct?

King regards,

Nick Douma



signature.asc
Description: OpenPGP digital signature
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] [PATCH] Minor AMQP improvements

2015-04-01 Thread Nick Douma
Hi,

On 01-04-15 05:18, Will Dowling wrote:
 I forgot to address the other (more important) bug I encountered:
 
 * Content type is only specified for messages published when the
   amqp_persistent_msg configuration option is specified. This information
   should always be applied to describe the payload of the message.
 
 * Modified the content type declaration to application/json in line with
   RFC4627.
 
 The second change may cause issues in clients relying on the content type flag
 being output already - so I can understand why this might not be adopted.
 
 That being said, making the change should improve compatability with tools
 that correctly interpret known MIME types.
 
 This patch is designed to be applied on top of my previous one.

Looks like I made an error indeed in my original amqp_persistent_msg
patch. For my use case, application/json as mime-type should not matter.

Kind regards,

Nick Douma



signature.asc
Description: OpenPGP digital signature
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] effort to relicense pmacct from GPL to a BSD-style license

2020-02-20 Thread Nick Douma
Hi,

On 08-01-2020 14:52, Job Snijders wrote:
> We need explicit approval from all contributors, and carefully keep
> track of those agreements. If a contributor doesn't agree or answer,
> we'll have to re-implement the contributed functionality or remove the
> contribution from the code base.
>
> REQUEST TO THE PMACCT CONTRIBUTOR COMMUNITY
> ---
>
> If you have contributed to the pmacct project (your name may be listed
> below), please consider a reply-all to this email expressing your
> explicit consent (or disapproval) to change the license governing your
> contributions to the pmacct project, to the following license:
>
> """
> Permission to use, copy, modify, and distribute this software for
> any purpose with or without fee is hereby granted, provided that the
> above copyright notice and this permission notice appear in all
> copies.
>
> THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL
> WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
> WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE
> AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL
> DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA
> OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER
> TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
> PERFORMANCE OF THIS SOFTWARE.
> """
>
> ---

I made a (small) contribution years ago, which was merged to the CVS/SVN
repo by Paolo (if I remember correctly). I agree to the relicensing.

1.5.0rc3 -- 18-04-2014
  + AMQP plugin: implemented persistent messaging via
amqp_persistent_msg configuration directive so to protect against
RabbitMQ restarts. Feature is courtesy by Nick Douma.

Kind regards,

Nick Douma

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists