We have this overall architecture:
- all hosts run rsyslog and send logs to load balancer over TCP using
RELP
- load balancer has one or more log aggregator hosts that run rsyslog
that accept logs (also RELP protocol of course)
- on log aggregator: write logs to S3 for storage
- on log aggregator: write logs to e.g. RedShift (amazon distributed DB)
What is the best way to get logs to S3? Looked around and it seems
that the best way is to write them to files, on rotation transfer them
to S3 using custom scripts. Seems fairly reasonable, is there anything
better?
Looks like omhdfs might be able to handle S3 (Flume also uses hdfs
module to write to S3) but not sure if it makes much sense. How does it
write logs to different hdfs files? Looking at does at
http://www.rsyslog.com/doc/omhdfs.html it seems that it would just write
one file (forever?).
Any thoughts on DB output? Redshift can use Postgresql client, seems
like there is ompgsql (see it mentioned at
http://www.rsyslog.com/doc/rsyslog_conf_modules.html) but don't see any
links to docs.
Thanks!
erik
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
