Yes, currently a single KafkaInput can only pull from a single Kafka partition. You can think of Heka's KafkaInput as analogous to a SimpleConsumer (see https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example).
If you want to manage inter-partition coordination, along the lines of what is described as a "High Level Consumer" (https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example), you'd handle that at the filter layer. For instance, you might set up a filter plugin with a message_matcher constructed such that it catches all of the messages from a single topic, regardless of partition, and perform any necessary correlations therein. The delivery semantics to this filter would match that described on the consumer group example page linked above, i.e. all of the messages from a single partition will be received in the correct order, but the messages from across partitions would be non-deterministically interleaved. If there are so many partitions carrying so much data that a single Heka instance can't handle them all, then you might have to have one box handling one subset of partitions, another box processing a different subset, and each of *those* in turn feeding into a third box that performs the next level of correlation. In other words, the building blocks are there, but you have to actually use them to put together a more sophisticated system. We're unfortunately not yet at the point where there are higher level constructs that will automatically distribute load for you. Hope this helps! -r On 04/10/2015 02:21 PM, Gil Fliker wrote:
Hi, We are about to start a poc using Heka. The plan is to pipe messages via Kafka transport and Heka being the endpoints speaking http with various producers and consumers. I saw in the documentation that you have to specify a partition number and only one partition number ? Our Kafka topic setup will be made of around 1000 partitions. What is the best way to approach this ? Thx Gil Fliker Outbrain Operations Manager The above terms reflect a potential business arrangement, are provided solely as a basis for further discussion, and are not intended to be and do not constitute a legally binding obligation. No legally binding obligations will be created, implied, or inferred until an agreement in final form is executed in writing by all parties involved. This email and any attachments hereto may be confidential or privileged. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person. Thanks. _______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
