Kafka - done (heavily used in our production setup and has been issue free compared to the Go version) https://mozilla-services.github.io/lua_sandbox_extensions/kafka/index.html HttpInput - easy (just use the lua socket HTTP client) HttpListenInput - pita (no HTTP server is currently available to the sandbox so one would have to be integrated/built). We actually wrote an nginx module for our specific use case https://github.com/mozilla-services/nginx_moz_ingest. ProcessInput - easy (not even worth a dedicated generic plugin) FilePollingInput - easy (again probably not even worth a dedicated plugin since it is just a few lines of code) StatsdInput - easy/medium (lua socket plus a simple parser and a bit more if you want to include the accumulator functionality)
As for the LogStreamerInput; yes there is a lot complexity there but it could be greatly simplified and tuned to the OS it is running on. The addition of the lua filesystem module will make this easier also pulling in the inotify module on Linux could eliminate a lot of these polling interfaces but will require some up front work. FYI: Everything in the Mozilla production environments will be on the 1.0 sandbox and Hindsight by the end of the year. Trink On Wed, Sep 7, 2016 at 9:59 AM, Mike Gibson <[email protected]> wrote: > Yup, would be happy to move to Hingsight someday. > LogstreamerInput is a biggie. I imagine that will be difficult to > re-implement in lua due to the complexity mentioned here: > https://mail.mozilla.org/pipermail/heka/2015-November/000882.html > > KafkaInput, HttpInput, HttpListenInput, ProcessInput, FilePollingInput and > StatsdInput are all required for our different use cases. > Not sure how difficult it would be to emulate all those in Hindsight/Lua > sandbox. > My Lua/C skills are even worse than my Go ones. :P > I know that progress is being made and I see quite a few new contributions > here: https://github.com/mozilla-services/lua_sandbox_extensions > > Will definitely continue to monitor/evaluate Hindsight but for now at > least, Heka is still seemingly the only thing that meets our requirements. > > Updated: > https://github.com/mozilla-services/heka/wiki/Community-Plugins > With some stuff. > > On Tue, Sep 6, 2016 at 11:52 PM, Mathieu Parent <[email protected]> > wrote: > >> 2016-09-06 22:03 GMT+02:00 Mike Gibson <[email protected]>: >> > Hello existing Heka users. >> >> Hello, >> >> >> > I'm aware of the "DEPRECATED" status of the project but I'm still not >> > prepared to abandon using Heka just yet. >> >> I'm still using it too, but plan to migrate to hindsight. >> >> What features are missing for you? (for me: LogStreamerInput) >> >> > I just can't seem to find another tool out there with the equivalent >> number >> > of I/O plugins, performance(hasn't been an issue for my scale) and >> > flexibility. >> > In the absence of something comparable to take its place, I'll continue >> > using Heka. >> > >> > So for those of us who have trouble letting go, I've made a few new >> external >> > plugins that may or may not be useful if anyone is interested: >> > >> > CMFilter - A Filter Plugin that processes messages containing TOML >> > configurations in their payload, executes an "Action" based on the set >> field >> > value which adds/removes/returns configurations for Input Plugins >> within the >> > currently running Heka agent. >> > >> > Currently supports managing Inputs for the following *DirectoryInputs: >> > ProcessDirectoryInput >> > LogstreamerDirectoryInput - New >> > HttpDirectoryInput - New >> > FilePollingDirectoryInput - New >> > >> > Others: >> > StreamAggregatorFilter/StreamSplitterDecoder - Provides bulk/batching >> of >> > Payloads and splitting of previously batched Payloads. >> > ZlibDecoder/ZlibEncoder - Provides compression/decompression of message >> > payloads >> >> >> Can you add those to >> https://github.com/mozilla-services/heka/wiki/Community-Plugins? >> >> Thanks >> >> -- >> Mathieu >> > > > _______________________________________________ > Heka mailing list > [email protected] > https://mail.mozilla.org/listinfo/heka > >
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
