Rob, > I'd probably send the protobuf encoded data over the transport, and then use > Heka to feed ES, holding off on encoding to the JSON that ElasticSearch > wants until the last moment.
That would make sense if I planned to have heka at the other side of the (Kafka|AMQP) pipeline but I am still not locked into that path. It is certainly something I continue to consider. > Ah, right, I forgot that since the TcpInput defaults to using a > ProtobufDecoder, leaving out the decoder setting will give you the wrong > results. You can work around this by explicitly setting the decoder value to > an empty string: > > [TcpInput] > address = "127.0.0.1:5565" > splitter = "TokenSplitter" > decoder = "" Worked like a charm $ /tmp/loggen -r 50000 -i -S -s 256 localhost 5565 average rate = 49520.12 msg/sec, count=495227, time=10.005, (last) msg size=256, bandwidth=12380.03 kB/sec $ wc -l /tmp/message-output.log 495227 /tmp/message-output.log nearly 50K EPS, minimal memory consumption. Amazing. > This is correct, but FYI a TokenSplitter with this configuration is > automatically defined for you and is available using the name > "TokenSplitter". Aware of it. I just tend to keep config verbose in order to increase supportability by people. :-) > ----START---- > function process_message() > return 0 > end > ----END---- > > But, again, even this is moot, you should explicitly specify no decoder. Slower but still quite good... $ /tmp/loggen -r 50000 -i -S -s 256 localhost 5565 average rate = 44263.30 msg/sec, count=442633, time=10.000, (last) msg size=256, bandwidth=11065.82 kB/sec > Pool size and channel size aren't likely to change much here. Tweaking them > is useful in certain specific situations, but this is usually when you have > a bunch of filter and output plugins and you're bumping up against all of > the available packs being used at the same time. > > Hope this helps, You did help lots. Thank you very much Rob. I will also have another read over the documentation and see if we maybe need to update it with references to the decoder = "" trick. If needed I will submit the PRs. Once again, thank you very much, your prompt support is much appreciated. Cheers _______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
