The key thing to realize is that S3 stores objects. You can update, replace, or
delete an object, but you cannot append to it.
So you have to have the complete object available before you start the upload to
S3. This means that you _must_ buffer the logs somewhere before you send them
into S3
now, for hdfs and similar, you have a case where updates are written to a
temporary location, and then periodically those updates get merged into the
other files, at which point those other files get completely re-written. As long
as these temporary copies are not in S3, having the main files in S3 is just
fine.
Unfortunantly, I don't think this really helps you much.
David Lang
On Tue, 13 Aug 2013, Erik Steffl wrote:
We have this overall architecture:
- all hosts run rsyslog and send logs to load balancer over TCP using RELP
- load balancer has one or more log aggregator hosts that run rsyslog that
accept logs (also RELP protocol of course)
- on log aggregator: write logs to S3 for storage
- on log aggregator: write logs to e.g. RedShift (amazon distributed DB)
What is the best way to get logs to S3? Looked around and it seems that the
best way is to write them to files, on rotation transfer them to S3 using
custom scripts. Seems fairly reasonable, is there anything better?
Looks like omhdfs might be able to handle S3 (Flume also uses hdfs module
to write to S3) but not sure if it makes much sense. How does it write logs
to different hdfs files? Looking at does at
http://www.rsyslog.com/doc/omhdfs.html it seems that it would just write one
file (forever?).
Any thoughts on DB output? Redshift can use Postgresql client, seems like
there is ompgsql (see it mentioned at
http://www.rsyslog.com/doc/rsyslog_conf_modules.html) but don't see any links
to docs.
Thanks!
erik
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
