Hello, Gwen. I have tried flafka, and used it as a sink for producing logs to kafka cluster. It produced 1M logs in few seconds which is pretty fast. However, I have a problem here. It produced all logs into one partition, even if there are three partitions. What I want is that 1M logs are well distributed through the three partitions, so I can have the benefit of using three consumers to improve the consume speed. Is there any way to get what I want? I am going to try 'topic.metadata.refresh.interval.ms' option, but I believe it will not work as I expect.
Thanks. On Sat, Oct 25, 2014 at 2:50 AM, Gwen Shapira <[email protected]> wrote: > Its not in the user guide because the user guide refers to releases > and Flafka is only on the unreleased trunk right now. > > On Fri, Oct 24, 2014 at 1:08 AM, Mongeol Heo <[email protected]> > wrote: > > I wonder why flume 1.5.0.1 user guide, which shows below, does not > include > > it although the link you give is for 1.5.0, and I think this is the one > > which is mentioned by Gwen Shapira above. > > Thank you. > > > > http://flume.apache.org/FlumeUserGuide.html > > > > On Fri, Oct 24, 2014 at 4:55 PM, Hari Shreedharan > > <[email protected]> wrote: > >> > >> > >> > http://archive.cloudera.com/cdh5/cdh/5/flume-ng/FlumeUserGuide.html#kafka-source > >> > >> This will come in the next Apache release. > >> > >> > >> On Fri, Oct 24, 2014 at 12:28 AM, Mongeol Heo <[email protected]> > >> wrote: > >>> > >>> Hi, Hari. > >>> Could you point out one of the kafka source which supports version > 0.8+? > >>> since all I found are for 0.7+. > >>> Thanks, > >>> > >>> Mungeol > >>> > >>> On Fri, Oct 24, 2014 at 4:15 PM, Hari Shreedharan > >>> <[email protected]> wrote: > >>>> > >>>> There is a Kafka Source too, but both are not yet released in a Flume > >>>> release, though one will hopefully be. > >>>> > >>>> Thanks, > >>>> Hari > >>>> > >>>> > >>>> On Thu, Oct 23, 2014 at 11:53 PM, Mangtani, Kushal > >>>> <[email protected]> wrote: > >>>>> > >>>>> Yes, Flume has support for Kafka 0.8.1. Refer the below link for > Flume > >>>>> Kafka sink,source. I have been using this in my prod and it is > relatively > >>>>> stable > >>>>> https://github.com/thilinamb/flume-ng-kafka-sink > >>>>> > >>>>> -Kushal Mangtani > >>>>> ________________________________ > >>>>> From: Mongeol Heo [[email protected]] > >>>>> Sent: Thursday, October 23, 2014 11:38 PM > >>>>> To: [email protected] > >>>>> Subject: Kafka version > >>>>> > >>>>> Hi, > >>>>> > >>>>> My question is that will flume sink and source for apache kafka > support > >>>>> latest version of it? > >>>>> As I know, even sink/source for kafka 0.7.2 is not released yet. > >>>>> It will be really great if it does. > >>>>> Thanks. > >>>>> > >>>>> Best regards, > >>>>> > >>>>> - Mungeol > >>>> > >>>> > >>> > >> > > >
