> >The performance of this configuration was unsuitable for production > >taking over 12 full hours to complete the processing of a single log > >file. In comparison to using a LogOutput which completed in > >approximately 3 minutes it was clear I needed to batch write records to > >InfluxDB. > Yup, clearly unacceptable. > >My initial attempt to batch records via Lua was a weak effort and > >ultimately unsuccessful. Attempting to queue records into a Lua table > >(likely the incorrect approach) > If I were to do this, I'd encode the lines right in the filter, so the filter > periodically emits a message payload where each line is an InfluxDB line. As > you can see from looking at the encoder > (https://github.com/mozilla-services/heka/blob/dev/sandbox/lua/encoders/schema_influx_line.lua#L142), > all of the hard work is done in a reusable module. You'd just call > `add_to_payload` in the process_message function and then `inject_payload` in > the timer_event function. Then you'd just use a PayloadEncoder w/ the > HttpOutput.
It is kind of funny how things work out. I had started working on this very problem the day before this thread started, so thanks for the pointers :) I tested by letting it take 2.5 million lines from a log file all at once, and it took a minute or two to get through it (sorry didn't time it exactly), but it was plenty good enough. We would have to worry about the influx server long before any of this. Cheers! Chris _______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
