On Wed, 23 Nov 2016, [email protected] wrote:
However, if you really want to go this way, one thing you can do is to make
use of the multicast mac feature in ethernet to distribute the same logs to
multiple systems/containers and have each container throw away all logs
except what it's configured to handle.
This lets you add/remove log processing at any time and even have multiple
systems processing the same logs in different ways
https://www.usenix.org/conference/lisa12/technical-sessions/presentation/lang_david
Network traffic x2
Actually, we are using a similar environment for other things, but I don't
think that's the way to go.
This doesn't need to double the network traffic in the way you are thinking. The
IP address that the senders deliver to is shared across all your processing
boxes. The switch replicates the traffic on it's backbone and delivers it to
each machine.
with your current approach you do
sender -> rsyslog -> redis -> logstash -> ES
so there are 3-4 copies of the logs (depending on if sender and rsyslog are the
same box)
if instead you did
sender -> multicast mac to rsyslog -> ES
there would only be two copies of the logs on the wire at any point (although N
copies total going into the rsyslog box, but that's only on the interface to
those boxes)
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
