Thanks for the reply, we're starting with just focusing on output to Carbon but will look at something like this when we get there now we know what's possible.
Another related question we've found is the json format in the FileOutput plugin. I think it exports json that resembles the internal data structures but I think it would be possible to output cleaner json. Have you thought about having output encoders at all? I've seen code in the Elasticsearch output that exports simple json I think might work so it could be good to share a json decoder between that, the FileOutput and a possible S3 output plugin. Are you using Heka to archive log data at mozilla? what format are you storing that in if so? Thanks, Dan On 4 March 2014 17:44, Rob Miller <[email protected]> wrote: > On Tue 04 Mar 2014 06:26:16 AM PST, Dan wrote: > >> Hi, >> > > Hi back! > > > We are just evaluating Heka for use as our log and metrics aggregation >> system. >> > > Great! Hope you like what you find. > > > We would like to archive our logs in S3 so it would be good if >> Heka could also store batches directly into a bucket. >> >> Is anyone working on a S3 output plugin for Heka? If not we might look >> at starting to write one. >> > > I'm not aware of anyone actively working on an S3 output at the moment, > no. We have, however, built Cloudwatch plugins, both an input and an output: > > https://github.com/mozilla-services/heka-mozsvc-plugins/ > blob/dev/cloudwatch.go > > Those use the crowdmob fork of Canonical's goamz package to handle the > details of interfacing w/ Amazon's API authentication framework. You should > be able to use that code as a model to get something bootstrapped pretty > easily. > > Our Cloudwatch plugins aren't in the Heka core, they're in a separate repo > we set up for plugins that we think would be less widely used. Ultimately > we'll probably create a separate repo specifically for AWS related plugins, > so the Cloudwatch, S3, and any other Amazon-related plugins that get > developed could have a nice cozy home together. > > Hope this helps! > > -r > >
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
