Back from vacations, I'm now playing again with Hindsight on a raspberry pi. As reported on github https://github.com/trink/hindsight/issues/1#issuecomment-119593775 the compilation now succeeds.
So getting inspiration from the examples in the benchmarks directory, I tried to create a Hindsight configuration to use my own lua sandboxes: I can successfully read data from udp and use a filter to decode data, then I would like to use another filter to handle generated messages, but I can't get any message in the second filter. Does Hindsight support more than one filter like Heka? Here is the Hindsight configuration, Lua sandboxes and output directory generated by Hindsight: https://github.com/bbinet/hindsight_hl_test Do you see anything wrong? Do I use hindsight correctly? Cheers, Bruno On 8 July 2015 at 09:44, bruno binet <[email protected]> wrote: > Sure, I will try your branch and report possible new compilation issues in > github. > > Cheers, > Bruno > > On 7 July 2015 at 18:26, Michael Trinkala <[email protected]> wrote: > >> I changed the checkpoint id to an unsigned long long. Can you test out >> the branch and add any other compilation errors to the issue (closing out >> this email thread). I am also taking suggestions/recommendations for a CI >> build system that supports multiple platforms. TravisCI adds almost no >> value since I am already building on a Debian based box. >> >> https://github.com/trink/hindsight/tree/issue_1 >> >> Thanks, >> Trink >> >> On Tue, Jul 7, 2015 at 8:21 AM, bruno binet <[email protected]> >> wrote: >> >>> Ok, thanks. >>> And sorry, but I don't have a patch (don't know how to fix this kind of >>> compilation issue). >>> >>> On 7 July 2015 at 16:17, Michael Trinkala <[email protected]> wrote: >>> >>>> Yeah, I have only been building on Ubuntu and haven't done any cross >>>> platform clean-up. Thanks for the build output I will fix those errors >>>> (unless you already have a patch). >>>> >>>> Trink >>>> >>>> On Tue, Jul 7, 2015 at 5:57 AM, bruno binet <[email protected]> >>>> wrote: >>>> >>>>> I now have some time to do a few tests with Hindsight, so I tried to >>>>> compile it on our targeted arm platform (raspberry pi), but I get the >>>>> following error: >>>>> >>>>> root@hl-mc-9999-dev:~/hindsight/release# cmake >>>>> -DCMAKE_BUILD_TYPE=release .. >>>>> -- The C compiler identification is GNU 4.7.2 >>>>> -- The CXX compiler identification is GNU 4.7.2 >>>>> -- Check for working C compiler: /usr/bin/gcc >>>>> -- Check for working C compiler: /usr/bin/gcc -- works >>>>> -- Detecting C compiler ABI info >>>>> -- Detecting C compiler ABI info - done >>>>> -- Detecting C compile features >>>>> -- Detecting C compile features - done >>>>> -- Check for working CXX compiler: /usr/bin/g++ >>>>> -- Check for working CXX compiler: /usr/bin/g++ -- works >>>>> -- Detecting CXX compiler ABI info >>>>> -- Detecting CXX compiler ABI info - done >>>>> -- Detecting CXX compile features >>>>> -- Detecting CXX compile features - done >>>>> -- Found LUASANDBOX: /usr/local/lib/libluasandbox.so >>>>> -- Configuring done >>>>> -- Generating done >>>>> -- Build files have been written to: /root/hindsight/release >>>>> >>>>> root@hl-mc-9999-dev:~/hindsight/release# make >>>>> Scanning dependencies of target hindsight >>>>> [ 2%] Building C object src/CMakeFiles/hindsight.dir/hindsight.c.o >>>>> [ 4%] Building C object src/CMakeFiles/hindsight.dir/ >>>>> hs_analysis_plugins.c.o >>>>> [ 6%] Building C object src/CMakeFiles/hindsight.dir/ >>>>> hs_checkpoint_reader.c.o >>>>> /root/hindsight/src/hs_checkpoint_reader.c: In function >>>>> 'find_first_id': >>>>> /root/hindsight/src/hs_checkpoint_reader.c:46:3: error: large integer >>>>> implicitly truncated to unsigned type [-Werror=overflow] >>>>> /root/hindsight/src/hs_checkpoint_reader.c:55:3: error: comparison is >>>>> always false due to limited range of data type [-Werror=type-limits] >>>>> cc1: all warnings being treated as errors >>>>> src/CMakeFiles/hindsight.dir/build.make:100: recipe for target >>>>> 'src/CMakeFiles/hindsight.dir/hs_checkpoint_reader.c.o' failed >>>>> make[2]: *** [src/CMakeFiles/hindsight.dir/hs_checkpoint_reader.c.o] >>>>> Error 1 >>>>> CMakeFiles/Makefile2:947: recipe for target >>>>> 'src/CMakeFiles/hindsight.dir/all' failed >>>>> make[1]: *** [src/CMakeFiles/hindsight.dir/all] Error 2 >>>>> Makefile:146: recipe for target 'all' failed >>>>> make: *** [all] Error 2 >>>>> >>>>> Do you know what is going on here? I guess this is an issue with the >>>>> arm platform only? >>>>> >>>>> Cheers, >>>>> Bruno >>>>> >>>>> >>>>> On 10 June 2015 at 18:41, bruno binet <[email protected]> wrote: >>>>> >>>>>> Thanks a lot for your answers. >>>>>> >>>>>> And yes, I'm very interested in bootstrapping a first prototype of my >>>>>> own data pipeline based on Hindsight so that I can compare the >>>>>> performance >>>>>> on a raspberry pi. >>>>>> (here is the current state of our Heka-based data pipeline: >>>>>> https://bitbucket.org/helioslite/heka-hl-sandboxes) >>>>>> So it would be great if you can give me the first instructions on how >>>>>> to build and setup Hindsight. >>>>>> >>>>>> Thanks. >>>>>> Bruno >>>>>> >>>>>> On 10 June 2015 at 18:18, Michael Trinkala <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> - It is usable and being actively developed with the intent to move >>>>>>> it into production later this year. >>>>>>> - We are currently running production data through it for testing >>>>>>> but it is not deployed in an official capacity. It has been very stable >>>>>>> but until a more robust set of tests have been build out I will not >>>>>>> consider it production ready. >>>>>>> - Yes, it can decode/encode Heka protobuf format >>>>>>> - Yes, the router/message matcher is complete. The only difference >>>>>>> is that it supports Lua string pattern matching instead of re2 regexp >>>>>>> (Heka 'Hostname =~ /^foo/' vs Hindsight 'Hostname =~ "^foo"') >>>>>>> - Yes, but you would need a lua-socket input and output sandbox (see >>>>>>> benchmarks/hsr_run for related examples) >>>>>>> - No documentation yet, only examples in the benchmarks directory. >>>>>>> I could have you bootstrapped in about 30 minutes (and hopefully turn >>>>>>> that >>>>>>> into a getting started guide) if you are interested. >>>>>>> >>>>>>> Implementation wise the only missing piece is support for >>>>>>> dynamically loading plugins. The actual code to accomplish it is very >>>>>>> small (just detecting files in the load directory and moving them to the >>>>>>> run directory) but ideally it would be fronted by a web server and a GUI >>>>>>> with access control and validation (a much larger effort and actually a >>>>>>> separate project). >>>>>>> >>>>>>> Trink >>>>>>> >>>>>>> On Wed, Jun 10, 2015 at 8:15 AM, bruno binet <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I recently discovered the work pushed into the Hindsight repository >>>>>>>> (https://github.com/trink/hindsight) which seems to be a >>>>>>>> lightweight alternative to Heka, based on the lua sandbox. >>>>>>>> The Hindsight vs Heka benchmarks are quite impressive. >>>>>>>> >>>>>>>> I'm currently running Heka on the raspberry pi (not so powerful) >>>>>>>> device and the load average quickly increases and exceeds 1 when Heka >>>>>>>> is >>>>>>>> ingesting data, so Hindsight could be a good fit for us if it can >>>>>>>> perform >>>>>>>> better than Heka in terms of CPU cycles. >>>>>>>> >>>>>>>> What is the current status of Hindsight? Is it just an temporary >>>>>>>> experiment or will it be maintained and actually used in production? >>>>>>>> Is it currently usable and stable? >>>>>>>> Is Hindsight able to decode and encode Heka protobuf format? >>>>>>>> Does Hindsight have a complete router implementation to dispatch >>>>>>>> messages to sandboxes like in Heka? >>>>>>>> My use case is basically to read raw text data from UDP socket, >>>>>>>> parse text data with lua patterns or lpeg, process data through a few >>>>>>>> lua >>>>>>>> sandbox filters, then write output messages both to a file (protobuf >>>>>>>> heka >>>>>>>> format) and a HTTP server (json format): can this be easily >>>>>>>> accomplished >>>>>>>> with Hindsight? >>>>>>>> Is there any documentation somewhere to get started with Hindsight? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Bruno >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Heka mailing list >>>>>>>> [email protected] >>>>>>>> https://mail.mozilla.org/listinfo/heka >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
