You might want to take a look at the omczmq and imczmq (the new ZeroMQ input and output plugins I've been working on). See - https://github.com/rsyslog/rsyslog/tree/master/contrib/omczmq and https://github.com/rsyslog/rsyslog/tree/master/contrib/imczmq
"Out of the box" they currently support fan out / fan in and pub-sub ( note: pub sub does not apply backpressure - it's meant to be lossy in zeromq). topologies. I recently started adding support for zbeacon ( http://czmq.zeromq.org/manual:zbeacon ), a UDP based discovery service. While UDP multicast is not the best solution everywhere, it works for some cases. My short to medium term plans going forward with these plugins include: * Support for non encrypted connections (curvezmq encryption was my first priority, the plugins don't actually have options for non encrypted connections yet). * credit based flow control * malamute (an embedded broker - see https://github.com/zeromq/malamute ) I've been pondering other discovery support. So far, I've been wary of introducing additional dependencies. There's so many discovery services that are en vogue right now (zookeeper, etcd, consul...) and every additional protocol supported is more overhead from a support and maintenance standpoint. So at the moment I've been focused solely on ZMTP ( the protocol libzmq implements - http://rfc.zeromq.org/spec:23 ) Brian On Thu, Jun 4, 2015 at 4:46 AM, singh.janmejay <[email protected]> wrote: > It won't be a very large change really if we develop it in an external > library. > > In rsyslog codebase, its a fairly small change, limited to input and > output modules that we pick. It'll be small parts of plugin code > (where new connection is established that will call this library > function conditionally, thats about it). > > > > On Thu, Jun 4, 2015 at 2:09 PM, Rainer Gerhards > <[email protected]> wrote: > > Sorry if this sounds discouraging: I currently have such a large > > backlog that I can't engage in that effort and I think I am also > > unable to merge any change of this magnitude any time before the > > backlog has become shorter (Q4+ 2015 I guess). > > > > Sorry I have no better answer, but you see yourself what all is going > > on and I really need to make sure I can follow at least the bare > > essentials. > > > > Rainer > > > > 2015-06-04 5:53 GMT+02:00 singh.janmejay <[email protected]>: > >> Yes L4 load-balancing will work to significant scale. L7 > >> load-balancing will do even better in terms of even load, but not sure > >> if syslog protocol is widely supported in load-balancers. > >> > >> DNS scaling and propagation delay are sometimes not acceptable, but > >> BGP anycast is something that'd work at data-center scale with very > >> large PODs. > >> > >> This is an alternative to that. It has fewer moving parts (just > >> producer and consumer), no LB and it doesn't require the complexity of > >> anycast. > >> > >> It trades-off engineering complexity of load-balancer and anycast with > >> smarter-clients and servers (increasing the complexity of clients and > >> servers a little, but also simplifying the deployment topology > >> significantly). > >> > >> I think all three are valid approaches and choice of one over the > >> other(best fit) will vary across deployments. > >> > >> > >> On Thu, Jun 4, 2015 at 8:45 AM, David Lang <[email protected]> wrote: > >>> I don't see the advantage of adding all this complexity as opposed to > using > >>> existing load balancing approaches. With existing tools we can deliver > the > >>> log stream to a cluster of systems, and deal with them failing. Yes, > the > >>> easy approaches to doing this are limited to the throughput of a single > >>> wire, but since that single wire is commonly 10Gb/sec (and easily > 40Gb/sec) > >>> with off-the-shelf technology, and the fact that the log stream can be > >>> compressed, this isn't likely to be an issue for much of anyone below > Google > >>> scale. > >>> > >>> There is a lot of advantages to keeping the failover logic and config > >>> contained to as small an area of the network and as few devices as > possible. > >>> The systems accepting the ogs _must- participate in the process > (responding > >>> to health checks if nothing else), it only takes a couple other boxes > (if > >>> any) to perform TCP load balancing. And having everything local > increases > >>> the accuracy of the detection and speed of recovery. > >>> > >>> If you want to deal with larger failures (datacenter scale), then > existing > >>> DNS/BGP failover tools can come into play. > >>> > >>> What advantage do we gain by pushing the configuration and failover > logic to > >>> the senders? > >>> > >>> David Lang > >>> > >>> > >>> On Thu, 4 Jun 2015, singh.janmejay wrote: > >>> > >>>> Hi, > >>>> > >>>> This is proposal towards first-class support for notion of a 'cluster' > >>>> as a log-forwarding destination. It talks about a > >>>> technology-independent service-discovery-support implementation. > >>>> > >>>> Scenario / Context: > >>>> ------------------------ > >>>> Say an environment is supposed to relay all logs to a logical > >>>> destination for aggregation/archival purpose. Such a setup at large > >>>> scale would have a several log-producers and several log-receivers (or > >>>> listeners). > >>>> > >>>> Because of several machines being involved, probability of some > >>>> listener nodes failing is significant, and log-producers should > >>>> ideally be somehow aware of "current" log-receiving-cluster definition > >>>> (that is, nodes that are healthy and receiving logs). > >>>> > >>>> This proposal talks about first-class definition and usage of > >>>> log-receiver-cluster. > >>>> > >>>> Configuration concepts: > >>>> ----------------------------- > >>>> > >>>> action(...): gets an additional parameter called cluster, which > >>>> follows URL like conversion (scheme is used to indicate technology > >>>> used for service-discovery). > >>>> > >>>> input(...): gets an additional parameter called cluster, same as > action > >>>> > >>>> The URL must resolve to currently-available host+port pairs, and may > >>>> support notifications, polling or both (depends on underlying > >>>> service-discovery technology). > >>>> > >>>> The resolution of URLs and representation of host+port pairs in the > >>>> service-endpoint is left to service-discovery technology. > >>>> > >>>> Configuration examples: > >>>> ------------------------------ > >>>> > >>>> action(type="omfwd" cluster="zk://10.20.30.40,10.20.30.41/foo_cluster > " > >>>> template="foo-fwd" protocol="tcp" tcp_framing="octet-counted" > >>>> ResendLastMSGOnReconnect="on") > >>>> > >>>> input(type="imptcp" port="10514" name="foo" ruleset="foo.recv" > >>>> cluster="zk://10.20.30.40,10.20.30.41/foo_cluster") > >>>> > >>>> The parameter "cluster" in both cases points to a url > >>>> "zk://10.20.30.40,10.20.30.41/foo_cluster" which does: > >>>> > >>>> [behaviour common to both input and action] > >>>> - scheme 'zk' resolves to use of zookeeper based > >>>> service-discovery (picks one of the multiple supported > >>>> service-discovery technologies) > >>>> - reach the zookeeper at IP address: 10.20.30.40 or 10.20.30.41 > >>>> - look for znode (zookeeper specific): /foo_cluster > >>>> > >>>> > >>>> [action specific behaviour] > >>>> - enumerate ephemeral nodes under the said znode > >>>> - pick one of the child-nodes at random > >>>> - expect it to be follow naming convension host:port > >>>> - break host and port apart and use it as destination for > forwarding > >>>> logs > >>>> > >>>> > >>>> [input specific behaviour] > >>>> - enumerate IP addresses that this input is listening to > >>>> - create ephemeral child-znodes under the foo_cluster znode with > >>>> name IP:port (follow this convention, enforced by service discovery > >>>> technology specific implementation) > >>>> - keep the session live as long as input is live (which means, > >>>> kill the session if queues are filled up, or the heartbeat-mechanism > >>>> will automatically kill it if rsyslog process dies, or host freezes up > >>>> or fails, or connecting switch fails, etc). > >>>> > >>>> Service-discovery technology independence: > >>>> -------------------------------------------------------- > >>>> > >>>> URL-like naming convention allows for scheme to indicate > >>>> service-discovery technology. Cluster-url for simple polling based > >>>> discovery technology implemented over http would look like this: > >>>> > >>>> http://10.20.30.40/foo_cluster > >>>> > >>>> A log-cabin based impl would have something similar to: > >>>> > >>>> logcabin://10.20.30.40,10.20.30.40/foo_cluster > >>>> > >>>> Implementation sketch: > >>>> ---------------------------- > >>>> > >>>> Service-discovery support can be implemented within rsyslog(runtime), > >>>> or as an independent library under rsyslog umbrella (I think I prefer > >>>> independent library). > >>>> > >>>> Any input or output modules that choose to support it basically link > >>>> with the library and do one of the following: > >>>> > >>>> - as an action: > >>>> when doAction fails, it picks a random-endpoint from a collection > >>>> of currently-available-endpoints provided by service discovery (random > >>>> picking from the collection is implemented inside the library, because > >>>> then it allows us later to make use of other parameters (such as > >>>> current load, which will be better for load-balancing)). > >>>> > >>>> - as an input: > >>>> if listen is successful and input is ready to take traffic, it > >>>> enumerates all IP addresses it is listening on, and registers with > >>>> service discovery technology those end-points (which can then be > >>>> discovered by senders). > >>>> > >>>> This can also be used at source or sink where producer or consumer is > >>>> aware of the service-discovery technology specific protocol and > >>>> rsyslog is only at one end of the pipe (as long as the same protocol > >>>> is followed, which will be easy considering the implementation is > >>>> available as a library). > >>>> > >>>> Host and port as required by to-be-supported input/output module will > >>>> become optional. Either host+port or cluster parameter must be > >>>> specified, specifying both will generate a warning, and it'll discard > >>>> cluster in favour of host+port (based on specificity). > >>>> > >>>> The support for this can be built incrementally for each input/output > >>>> module as desired, I have omfwd and imptcp in mind at this time. > >>>> > >>>> Thoughts? > >>>> > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com/professional-services/ > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad of > >>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T > >>> LIKE THAT. > >> > >> > >> > >> -- > >> Regards, > >> Janmejay > >> http://codehunk.wordpress.com > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you DON'T LIKE THAT. > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > > > > -- > Regards, > Janmejay > http://codehunk.wordpress.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
