On 16 September 2015 at 00:19, Michael Trinkala <[email protected]> wrote:
> Generally I only use the prune for reporting in which case I don't want it > to clean up the results but it would be pretty easy to include the analysis > directory as a config option. > Here is a pull request to support analysis plugins too: https://github.com/trink/hindsight/pull/8 > Yes, there is up to a second of latency on detecting/reading the rolled > logs. It is there to prevent rapid polling when the data flow is pretty > low. When using a reasonably sized roll configuration the potential of up > to a second of delay every few minutes hasn't been an issue for us. In > fact most of the Hindsight backoffs/retries don't use anything smaller than > a one second resolution as Hindsight is designed to be near real time > (within a few seconds). > > Cool, the numbers are still looking good on this end too and it is pretty > close to feature complete (with the recent addition of the dynamic loading). > > This issue is on my radar and will be addressed soon, as well as the > documentation. > Cool, thanks. > Trink > > On Tue, Sep 15, 2015 at 8:04 AM, bruno binet <[email protected]> > wrote: > >> Hi Michael, >> >> Thanks again for your answer. >> I've also seen that you have checked in your "prune_input.lua plugin" on >> github: >> https://github.com/trink/hindsight/blob/master/sandboxes/input/prune_input.lua >> This is very useful, and I'm wondering if you also have a similar pruning >> plugin that could prune the output files produced by the analysis plugins? >> >> Also you said that we must be aware the roll transition can introduce up >> to a second of latency in the analysis and output plugins: why does the >> roll transition do introduce such a delay? Does this latency delay will be >> decreased if we use your "prune_input.lua" plugin to remove the files that >> would have been rolled? >> >> FYI, I've migrated my plugins and configuration from Heka to Hindsight, >> and the performance is actually largely improved (CPU usage decreased by >> ~10x, RAM usage decreased by ~80x !) >> So thanks a lot for that great piece of software! >> And that means that I can now use my Hindsight pipeline seamlessly on a >> Raspberry PI 1 device without facing any performance issues anymore. :-) >> >> The last thing that I would like to fix before pushing Hindsight in >> production is to be able to compile the master branch on Raspbery PI ARM >> platform (for now I monkey patch the code for the compilation to succeed), >> see github issue #7: https://github.com/trink/hindsight/pull/7 >> >> Cheers, >> Bruno >> >> On 18 August 2015 at 06:18, Michael Trinkala <[email protected]> >> wrote: >> >>> Yes, I will get the plugins checked in shortly I just need to add a >>> config option or two. >>> >>> run once - set the ticker_interval to 0 and return from process_message >>> when you are done >>> polling - set the ticker_interval > 0 and return from process_message, >>> it will be called again when the ticker interval expires. >>> continuous - don't return from process_message >>> >>> No, this feature does not need to be implemented in the infrastructure >>> the plugin extends the functionality nicely and can be easily customized as >>> needed. >>> >>> It totally depends on your throughput and message size, I don't expect >>> any problems even if you roll frequently (be aware the roll transition can >>> introduce up to a second of latency in the analysis and output plugins). >>> >>> Not at the moment since it is just two executables. >>> >>> Trink >>> >>> >>> On Thu, Aug 13, 2015 at 5:36 AM, bruno binet <[email protected]> >>> wrote: >>> >>>> >>>> >>>> On 12 August 2015 at 18:16, Michael Trinkala <[email protected]> >>>> wrote: >>>> >>>>> Most looping requirements can be flatten out: i.e., alerting can be >>>>> handled in the output plugins in your example and >>>>> aggregation/sessionization etc can be handled in the inputs. As for >>>>> sharing: things fall into place when you start thinking about it from a >>>>> module level instead of an individual plugin level. >>>>> >>>>> The locations are configurable 'output_path' so you can put the output >>>>> files anywhere you want. >>>>> >>>> >>>> Good, I will set it somewhere on the /tmp/ partition, which is already >>>> a stored in RAM. >>>> >>>> >>>>> I have some plugins (like a stdin, simple file, TCP. and a pruning >>>>> (cleans up the output files when everyone in done with them) inputs; >>>>> >>>> >>>> I'm interested by this pruning plugin: could you share it? Or don't you >>>> think it would make sense for Hindsight to provide an option to >>>> automaticallly clean up output files? >>>> >>>> >>>>> heka protobuf and a payload outputs etc. but they haven't commited >>>>> yet). As for ProcessInput the Input sandbox has access to os.execute so >>>>> there won't be a generic version you can just call what you want and >>>>> handle >>>>> its output directly (Hindsight already supports run once, polling, and >>>>> continuous input plugins) >>>>> >>>> >>>> Can you explain briefly how to configure input plugins for these 3 >>>> options (run once, polling, or continuous)? >>>> >>>> >>>>> The output files will grow until 'output_size' (defaults to 64MiB) >>>>> before they are rolled (they are not deleted by default). >>>>> >>>> >>>> Ok. Would it make sense to introduce an option to delete them by >>>> default? >>>> >>>> >>>>> I would not make it too small unless you need to prune really quickly >>>>> generally I run with a config of 1GIB (on some of our systems that rolls >>>>> several times a minute) space permitting I would just roll them after >>>>> several minutes of what would be average data flow on your system) >>>>> >>>> >>>> I plan to use a ram disk (tmpfs) to store these files (to avoid writing >>>> too much on the sd card), so I would like them to stay small in order to >>>> prevent eating too much RAM: does 5MB to 10MB seems reasonable to you? >>>> >>>> Also, do you plan to provide a debian package for Hindsight to ease >>>> installation? >>>> >>>> Thanks for the great help! >>>> Bruno >>>> >>>> >>>>> >>>>> Trink >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Wed, Aug 12, 2015 at 1:55 AM, bruno binet <[email protected]> >>>>> wrote: >>>>> >>>>>> >>>>>> >>>>>> On 12 August 2015 at 10:19, bruno binet <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Thanks for all this valuable information. >>>>>>> >>>>>>> On 11 August 2015 at 17:17, Michael Trinkala <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> There are a few intentional changes between Heka and Hindsight. >>>>>>>> Looping messages in Heka has always been a bad idea so it was removed. >>>>>>>> >>>>>>> >>>>>>> Personally I like the looping messages feature in Heka as it is very >>>>>>> flexible and could be useful to share ready-to-use plugins. Also it >>>>>>> supports processing messages through multiple ticker_interval which can >>>>>>> be >>>>>>> useful (alerting, aggregations). >>>>>>> >>>>>>> >>>>>>>> There are a few API enhancements such as a protobuf stream reader >>>>>>>> and writer. Checkpoint are all managed by the Hindsight >>>>>>>> infrastructure (so >>>>>>>> much of the burden is removed from the plugin writer, this also alters >>>>>>>> the >>>>>>>> plugin API slightly). The write_message hack for Go has been removed >>>>>>>> since >>>>>>>> messages are immutable. read_config now has access to all related >>>>>>>> sandbox >>>>>>>> config options (standard and user defined). read_next_field is not >>>>>>>> supported (this will also be removed from Heka in 0.11). >>>>>>>> >>>>>>>> In most cases you will find the Hindsight IOPS lower than Heka due >>>>>>>> to the much more efficient check pointing (btw Heka 0.11 is moving to >>>>>>>> a >>>>>>>> disk buffer everywhere). >>>>>>>> >>>>>>> >>>>>>> Great, that is good to know. >>>>>>> >>>>>>> >>>>>>>> output_hi/input/* - contains the output from all of the input >>>>>>>> plugins >>>>>>>> output_hi/analysis/* - contains the output from all of the analysis >>>>>>>> plugins >>>>>>>> >>>>>>> >>>>>> Are the above files always growing? >>>>>> I suppose the output_limit configuration allow us limit their size: >>>>>> what are the implications if I limit their size to a few KB? Will it >>>>>> reduce >>>>>> Hindsight performance? >>>>>> >>>>>> >>>>>>> hindsight.cp - in the checkpoint file for all I/O (inputs, analysis, >>>>>>>> and output plugins) >>>>>>>> hindsight.tsv - in the self monitoring performance stats >>>>>>>> >>>>>>>> They files are all mandatory. They are the reason Hindsight has an >>>>>>>> at least once delivery guarantee and they provide valuable insight on >>>>>>>> system operation and performance. >>>>>>>> >>>>>>> >>>>>>> If I don't need delivery guarantee, do you think it could make sense >>>>>>> to move these files to a ramdisk (tmpfs) partition in order to preserve >>>>>>> the >>>>>>> flash sd/usb card? >>>>>>> >>>>>>> >>>>>>>> decode_message needs to be turned on for analysis and output >>>>>>>> plugins, I will enable it. >>>>>>>> >>>>>>> >>>>>>> Ok, thank you: this is now working as expected. >>>>>>> >>>>>>> Also, do you plan to implement some additional lua modules to help >>>>>>> build input sandboxes similar to Heka input plugins (like the >>>>>>> FilePollingInput, the ProcessInput, or the LogstreamerInput)? >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Trink >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Aug 11, 2015 at 1:54 AM, bruno binet <[email protected] >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> I see, so I need to investigate how I can merge my multiple lua >>>>>>>>> sandbox filters into a single one. >>>>>>>>> >>>>>>>>> This make me wondering if there is some other differences between >>>>>>>>> Hindsight and Heka? >>>>>>>>> The fact that only one analysis plugin cannot consume the output >>>>>>>>> of another analysis plugin is the only difference beween Hindsight >>>>>>>>> analysis >>>>>>>>> plugins and Heka filter sandbox plugins? >>>>>>>>> >>>>>>>>> Also I saw in another thread that Hindsight uses disk buffers at >>>>>>>>> every stage, so there's only ever one >>>>>>>>> message in memory at every step of the pipeline: does it mean >>>>>>>>> Hindsight will write much more frequently to the disk than Heka? This >>>>>>>>> may >>>>>>>>> be an issue for me since we use a raspberry pi which disk is a sdcard >>>>>>>>> or >>>>>>>>> usb flash key. >>>>>>>>> >>>>>>>>> I see that some data is written to the output_path (output_hl/ >>>>>>>>> directory in my case): can you explain what are all these files: >>>>>>>>> $ tree output_hl/ >>>>>>>>> output_hl/ >>>>>>>>> |-- analysis >>>>>>>>> | `-- 0.log >>>>>>>>> |-- hindsight.cp >>>>>>>>> |-- hindsight.tsv >>>>>>>>> `-- input >>>>>>>>> `-- 0.log >>>>>>>>> >>>>>>>>> Can we avoid generating all these files? >>>>>>>>> >>>>>>>>> Last question: I don't manage to use the "read_next_field" or >>>>>>>>> "decode_message" api function from the output plugin, are they >>>>>>>>> available? >>>>>>>>> >>>>>>>>> The following error is returned: >>>>>>>>> 1439280624615780495 [error] output_plugins terminated: >>>>>>>>> output/encode_metric.cfg msg: process_message() >>>>>>>>> _hl/output/encode_metric.lua:16: attempt to call global >>>>>>>>> 'read_next_field' >>>>>>>>> (a nil value) >>>>>>>>> >>>>>>>>> or when I change my output plugin to use the decode_message api >>>>>>>>> function: >>>>>>>>> 1439282566555139226 [error] output_plugins terminated: >>>>>>>>> output/encode_metric.cfg msg: process_message() >>>>>>>>> _hl/output/encode_metric.lua:15: attempt to call global >>>>>>>>> 'decode_message' (a >>>>>>>>> nil value) >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Bruno >>>>>>>>> >>>>>>>>> >>>>>>>>> On 10 August 2015 at 19:19, Michael Trinkala < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> There is no message looping in Hindsight (one analysis plugin >>>>>>>>>> cannot consume the output of another analysis plugin). In your >>>>>>>>>> example the >>>>>>>>>> decoding should happen in the input. Heka has Inputs, splitters, and >>>>>>>>>> decoder (in Hindsight it is just an Input and common functionality >>>>>>>>>> can be >>>>>>>>>> split into modules for code reuse). This in general simplifies the >>>>>>>>>> configuration, is easier to follow (since everything is in one >>>>>>>>>> place) and >>>>>>>>>> has performance benefits as well. >>>>>>>>>> >>>>>>>>>> Trink >>>>>>>>>> >>>>>>>>>> On Mon, Aug 10, 2015 at 9:23 AM, bruno binet < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Back from vacations, I'm now playing again with Hindsight on a >>>>>>>>>>> raspberry pi. >>>>>>>>>>> As reported on github >>>>>>>>>>> https://github.com/trink/hindsight/issues/1#issuecomment-119593775 >>>>>>>>>>> the compilation now succeeds. >>>>>>>>>>> >>>>>>>>>>> So getting inspiration from the examples in the benchmarks >>>>>>>>>>> directory, I tried to create a Hindsight configuration to use my >>>>>>>>>>> own lua >>>>>>>>>>> sandboxes: I can successfully read data from udp and use a filter >>>>>>>>>>> to decode >>>>>>>>>>> data, then I would like to use another filter to handle generated >>>>>>>>>>> messages, >>>>>>>>>>> but I can't get any message in the second filter. Does Hindsight >>>>>>>>>>> support >>>>>>>>>>> more than one filter like Heka? >>>>>>>>>>> >>>>>>>>>>> Here is the Hindsight configuration, Lua sandboxes and output >>>>>>>>>>> directory generated by Hindsight: >>>>>>>>>>> https://github.com/bbinet/hindsight_hl_test >>>>>>>>>>> >>>>>>>>>>> Do you see anything wrong? Do I use hindsight correctly? >>>>>>>>>>> >>>>>>>>>>> Cheers, >>>>>>>>>>> Bruno >>>>>>>>>>> >>>>>>>>>>> On 8 July 2015 at 09:44, bruno binet <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Sure, I will try your branch and report possible new >>>>>>>>>>>> compilation issues in github. >>>>>>>>>>>> >>>>>>>>>>>> Cheers, >>>>>>>>>>>> Bruno >>>>>>>>>>>> >>>>>>>>>>>> On 7 July 2015 at 18:26, Michael Trinkala < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> I changed the checkpoint id to an unsigned long long. Can you >>>>>>>>>>>>> test out the branch and add any other compilation errors to the >>>>>>>>>>>>> issue >>>>>>>>>>>>> (closing out this email thread). I am also taking >>>>>>>>>>>>> suggestions/recommendations for a CI build system that supports >>>>>>>>>>>>> multiple >>>>>>>>>>>>> platforms. TravisCI adds almost no value since I am already >>>>>>>>>>>>> building on a >>>>>>>>>>>>> Debian based box. >>>>>>>>>>>>> >>>>>>>>>>>>> https://github.com/trink/hindsight/tree/issue_1 >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Trink >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Jul 7, 2015 at 8:21 AM, bruno binet < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Ok, thanks. >>>>>>>>>>>>>> And sorry, but I don't have a patch (don't know how to fix >>>>>>>>>>>>>> this kind of compilation issue). >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 7 July 2015 at 16:17, Michael Trinkala < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Yeah, I have only been building on Ubuntu and haven't done >>>>>>>>>>>>>>> any cross platform clean-up. Thanks for the build output I >>>>>>>>>>>>>>> will fix those >>>>>>>>>>>>>>> errors (unless you already have a patch). >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Trink >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Tue, Jul 7, 2015 at 5:57 AM, bruno binet < >>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I now have some time to do a few tests with Hindsight, so I >>>>>>>>>>>>>>>> tried to compile it on our targeted arm platform (raspberry >>>>>>>>>>>>>>>> pi), but I get >>>>>>>>>>>>>>>> the following error: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> root@hl-mc-9999-dev:~/hindsight/release# cmake >>>>>>>>>>>>>>>> -DCMAKE_BUILD_TYPE=release .. >>>>>>>>>>>>>>>> -- The C compiler identification is GNU 4.7.2 >>>>>>>>>>>>>>>> -- The CXX compiler identification is GNU 4.7.2 >>>>>>>>>>>>>>>> -- Check for working C compiler: /usr/bin/gcc >>>>>>>>>>>>>>>> -- Check for working C compiler: /usr/bin/gcc -- works >>>>>>>>>>>>>>>> -- Detecting C compiler ABI info >>>>>>>>>>>>>>>> -- Detecting C compiler ABI info - done >>>>>>>>>>>>>>>> -- Detecting C compile features >>>>>>>>>>>>>>>> -- Detecting C compile features - done >>>>>>>>>>>>>>>> -- Check for working CXX compiler: /usr/bin/g++ >>>>>>>>>>>>>>>> -- Check for working CXX compiler: /usr/bin/g++ -- works >>>>>>>>>>>>>>>> -- Detecting CXX compiler ABI info >>>>>>>>>>>>>>>> -- Detecting CXX compiler ABI info - done >>>>>>>>>>>>>>>> -- Detecting CXX compile features >>>>>>>>>>>>>>>> -- Detecting CXX compile features - done >>>>>>>>>>>>>>>> -- Found LUASANDBOX: /usr/local/lib/libluasandbox.so >>>>>>>>>>>>>>>> -- Configuring done >>>>>>>>>>>>>>>> -- Generating done >>>>>>>>>>>>>>>> -- Build files have been written to: /root/hindsight/release >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> root@hl-mc-9999-dev:~/hindsight/release# make >>>>>>>>>>>>>>>> Scanning dependencies of target hindsight >>>>>>>>>>>>>>>> [ 2%] Building C object src/CMakeFiles/hindsight.dir/ >>>>>>>>>>>>>>>> hindsight.c.o >>>>>>>>>>>>>>>> [ 4%] Building C object src/CMakeFiles/hindsight.dir/ >>>>>>>>>>>>>>>> hs_analysis_plugins.c.o >>>>>>>>>>>>>>>> [ 6%] Building C object src/CMakeFiles/hindsight.dir/ >>>>>>>>>>>>>>>> hs_checkpoint_reader.c.o >>>>>>>>>>>>>>>> /root/hindsight/src/hs_checkpoint_reader.c: In function >>>>>>>>>>>>>>>> 'find_first_id': >>>>>>>>>>>>>>>> /root/hindsight/src/hs_checkpoint_reader.c:46:3: error: >>>>>>>>>>>>>>>> large integer implicitly truncated to unsigned type >>>>>>>>>>>>>>>> [-Werror=overflow] >>>>>>>>>>>>>>>> /root/hindsight/src/hs_checkpoint_reader.c:55:3: error: >>>>>>>>>>>>>>>> comparison is always false due to limited range of data type >>>>>>>>>>>>>>>> [-Werror=type-limits] >>>>>>>>>>>>>>>> cc1: all warnings being treated as errors >>>>>>>>>>>>>>>> src/CMakeFiles/hindsight.dir/build.make:100: recipe for >>>>>>>>>>>>>>>> target 'src/CMakeFiles/hindsight.dir/hs_checkpoint_reader.c.o' >>>>>>>>>>>>>>>> failed >>>>>>>>>>>>>>>> make[2]: *** >>>>>>>>>>>>>>>> [src/CMakeFiles/hindsight.dir/hs_checkpoint_reader.c.o] >>>>>>>>>>>>>>>> Error 1 >>>>>>>>>>>>>>>> CMakeFiles/Makefile2:947: recipe for target >>>>>>>>>>>>>>>> 'src/CMakeFiles/hindsight.dir/all' failed >>>>>>>>>>>>>>>> make[1]: *** [src/CMakeFiles/hindsight.dir/all] Error 2 >>>>>>>>>>>>>>>> Makefile:146: recipe for target 'all' failed >>>>>>>>>>>>>>>> make: *** [all] Error 2 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Do you know what is going on here? I guess this is an issue >>>>>>>>>>>>>>>> with the arm platform only? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>>> Bruno >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 10 June 2015 at 18:41, bruno binet < >>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks a lot for your answers. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> And yes, I'm very interested in bootstrapping a first >>>>>>>>>>>>>>>>> prototype of my own data pipeline based on Hindsight so that >>>>>>>>>>>>>>>>> I can compare >>>>>>>>>>>>>>>>> the performance on a raspberry pi. >>>>>>>>>>>>>>>>> (here is the current state of our Heka-based data >>>>>>>>>>>>>>>>> pipeline: >>>>>>>>>>>>>>>>> https://bitbucket.org/helioslite/heka-hl-sandboxes) >>>>>>>>>>>>>>>>> So it would be great if you can give me the first >>>>>>>>>>>>>>>>> instructions on how to build and setup Hindsight. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>>>>> Bruno >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 10 June 2015 at 18:18, Michael Trinkala < >>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> - It is usable and being actively developed with the >>>>>>>>>>>>>>>>>> intent to move it into production later this year. >>>>>>>>>>>>>>>>>> - We are currently running production data through it for >>>>>>>>>>>>>>>>>> testing but it is not deployed in an official capacity. It >>>>>>>>>>>>>>>>>> has been very >>>>>>>>>>>>>>>>>> stable but until a more robust set of tests have been build >>>>>>>>>>>>>>>>>> out I will not >>>>>>>>>>>>>>>>>> consider it production ready. >>>>>>>>>>>>>>>>>> - Yes, it can decode/encode Heka protobuf format >>>>>>>>>>>>>>>>>> - Yes, the router/message matcher is complete. The only >>>>>>>>>>>>>>>>>> difference is that it supports Lua string pattern matching >>>>>>>>>>>>>>>>>> instead of re2 >>>>>>>>>>>>>>>>>> regexp (Heka 'Hostname =~ /^foo/' vs Hindsight 'Hostname =~ >>>>>>>>>>>>>>>>>> "^foo"') >>>>>>>>>>>>>>>>>> - Yes, but you would need a lua-socket input and output >>>>>>>>>>>>>>>>>> sandbox (see benchmarks/hsr_run for related examples) >>>>>>>>>>>>>>>>>> - No documentation yet, only examples in the benchmarks >>>>>>>>>>>>>>>>>> directory. I could have you bootstrapped in about 30 >>>>>>>>>>>>>>>>>> minutes (and >>>>>>>>>>>>>>>>>> hopefully turn that into a getting started guide) if you are >>>>>>>>>>>>>>>>>> interested. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Implementation wise the only missing piece is support for >>>>>>>>>>>>>>>>>> dynamically loading plugins. The actual code to accomplish >>>>>>>>>>>>>>>>>> it is very >>>>>>>>>>>>>>>>>> small (just detecting files in the load directory and moving >>>>>>>>>>>>>>>>>> them to the >>>>>>>>>>>>>>>>>> run directory) but ideally it would be fronted by a web >>>>>>>>>>>>>>>>>> server and a GUI >>>>>>>>>>>>>>>>>> with access control and validation (a much larger effort and >>>>>>>>>>>>>>>>>> actually a >>>>>>>>>>>>>>>>>> separate project). >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Trink >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Wed, Jun 10, 2015 at 8:15 AM, bruno binet < >>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I recently discovered the work pushed into the Hindsight >>>>>>>>>>>>>>>>>>> repository (https://github.com/trink/hindsight) which >>>>>>>>>>>>>>>>>>> seems to be a lightweight alternative to Heka, based on the >>>>>>>>>>>>>>>>>>> lua sandbox. >>>>>>>>>>>>>>>>>>> The Hindsight vs Heka benchmarks are quite impressive. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I'm currently running Heka on the raspberry pi (not so >>>>>>>>>>>>>>>>>>> powerful) device and the load average quickly increases and >>>>>>>>>>>>>>>>>>> exceeds 1 when >>>>>>>>>>>>>>>>>>> Heka is ingesting data, so Hindsight could be a good fit >>>>>>>>>>>>>>>>>>> for us if it can >>>>>>>>>>>>>>>>>>> perform better than Heka in terms of CPU cycles. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> What is the current status of Hindsight? Is it just an >>>>>>>>>>>>>>>>>>> temporary experiment or will it be maintained and actually >>>>>>>>>>>>>>>>>>> used in >>>>>>>>>>>>>>>>>>> production? >>>>>>>>>>>>>>>>>>> Is it currently usable and stable? >>>>>>>>>>>>>>>>>>> Is Hindsight able to decode and encode Heka protobuf >>>>>>>>>>>>>>>>>>> format? >>>>>>>>>>>>>>>>>>> Does Hindsight have a complete router implementation to >>>>>>>>>>>>>>>>>>> dispatch messages to sandboxes like in Heka? >>>>>>>>>>>>>>>>>>> My use case is basically to read raw text data from UDP >>>>>>>>>>>>>>>>>>> socket, parse text data with lua patterns or lpeg, process >>>>>>>>>>>>>>>>>>> data through a >>>>>>>>>>>>>>>>>>> few lua sandbox filters, then write output messages both to >>>>>>>>>>>>>>>>>>> a file >>>>>>>>>>>>>>>>>>> (protobuf heka format) and a HTTP server (json format): can >>>>>>>>>>>>>>>>>>> this be easily >>>>>>>>>>>>>>>>>>> accomplished with Hindsight? >>>>>>>>>>>>>>>>>>> Is there any documentation somewhere to get started with >>>>>>>>>>>>>>>>>>> Hindsight? >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>> Bruno >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>> Heka mailing list >>>>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>>>> https://mail.mozilla.org/listinfo/heka >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
