I mostly agree with Jonathan's post. I agree completely with his descriptions of Heka and Kafka, and how they differ. I wouldn't write Heka off as a choice for ETL, though. It can be nice when your ETL tool provides the option for doing crunching, graphing, and alerting on the gathered data while it's in transit.
Anyway, my responses to your questions: 1. Yes, Heka can be used. It's primary use case is stream processing, but collecting and shipping data is an important part of stream processing, so you can pu that functionality to other use. 2. Right, you can't find good examples for that, because there aren't any. AFAIK nobody has written either a postgres or mongo output plugin, those would have to exist before you could push in. You could probably use the HttpOutput to push data into at least mongo, though. 3. Kafka is heavier, and different. Jonathan covered this nicely. 4. If you're still interested in trying Heka for your needs, and are serious about wanting to contribute, you might start by trying to write a postgres output. Hope this info is useful, sorry we don't have a better solution for you. -r On 01/20/2015 09:22 PM, Monosij Dutta-Roy wrote:
Hello Rob and Heka users - I was exploring Heka (on Ubuntu VM) and had a few questions. i will try to keep them brief. ... 1. Can Heka be used, being used as an ETL tool to load data from various files into DBMS? or is it mainly for log processing and/or stream processing? 2. To me it seems that it can be, but I cannot seem to find good examples / tutorials to go through in figuring out config files to load into Postgres and MongoDB simultaneously - my use case. ... 3. I was comparing it as an alternative to the Mesos / Kafka / Aurora architecture - which seems much heavier than what Heka accomplishes much simpler. But again in going through the tutorials on your site I am not able to configure for DBMS from the flat files I am loading. The tutorial seems to want to write a log file in my /var/heka directory - whuch I had to remove my root permissions from (in my VM). Perhaps some of this baseline files, should be configurable of course, but perhaps start off in ~/heka? ... 4. Again, I am interested in using this towards ETL - if there are some examples you may be able to point me to would be great. I hope to learn to use it and then contribute towards its development as well. Thank you much for your help. And thank you for thinking through the details, in identifying, collating the shortcomings of other tools, in coming up with this great toolset. Mono
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
