by the way, writing such a script is on my to-do list, so if you get a chance to
write it first, could you pass it along?
David Lang
On Wed, 14 Aug 2013, David Lang wrote:
I believe so.
David Lang
On Wed, 14 Aug 2013, Erik Steffl wrote:
ok, so it seems that the best option is to write files to local filesystem
then write a custom script to move them to S3 (which is fairly simple)?
erik
On 08/13/2013 10:35 PM, David Lang wrote:
The key thing to realize is that S3 stores objects. You can update,
replace, or delete an object, but you cannot append to it.
So you have to have the complete object available before you start the
upload to S3. This means that you _must_ buffer the logs somewhere
before you send them into S3
now, for hdfs and similar, you have a case where updates are written to
a temporary location, and then periodically those updates get merged
into the other files, at which point those other files get completely
re-written. As long as these temporary copies are not in S3, having the
main files in S3 is just fine.
Unfortunantly, I don't think this really helps you much.
David Lang
On Tue, 13 Aug 2013, Erik Steffl wrote:
We have this overall architecture:
- all hosts run rsyslog and send logs to load balancer over TCP using
RELP
- load balancer has one or more log aggregator hosts that run rsyslog
that accept logs (also RELP protocol of course)
- on log aggregator: write logs to S3 for storage
- on log aggregator: write logs to e.g. RedShift (amazon distributed DB)
What is the best way to get logs to S3? Looked around and it seems
that the best way is to write them to files, on rotation transfer them
to S3 using custom scripts. Seems fairly reasonable, is there anything
better?
Looks like omhdfs might be able to handle S3 (Flume also uses hdfs
module to write to S3) but not sure if it makes much sense. How does
it write logs to different hdfs files? Looking at does at
http://www.rsyslog.com/doc/omhdfs.html it seems that it would just
write one file (forever?).
Any thoughts on DB output? Redshift can use Postgresql client, seems
like there is ompgsql (see it mentioned at
http://www.rsyslog.com/doc/rsyslog_conf_modules.html) but don't see
any links to docs.
Thanks!
erik
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
