> I am totally new to Kafka and we are evaluating Kafka like Felix. I'm > looking for a utility to pipe data from a file (or multiple files) to a > producer (like command tail -f in linux). This is an our typical use-case. I > saw the patch (KAFKA-130) you provided but not sure if this will work well > without loosing data if the file (or multiple files) will be rotated.
Would "tail -F --retry" work for you? Also, if these are log files and you are using log4j, then kafka has a log4j appender that you can use to produce messages. > > I would care about plugin that will such as filter message at producer; > modify message body in order to mark some critical kind of information, etc. > It would be good if you provide documentation about how to write a plugin > for Kafka. You can use async producer callback handlers to do this sort of filtering. I don't see an example in the distribution, but it is pretty straightforward: see https://svn.apache.org/repos/asf/incubator/kafka/trunk/core/src/main/scala/kafka/javaapi/producer/async/CallbackHandler.java (The kafka log4j producer currently uses the syncproducer, so you won't be able to use callback handlers with it). Thanks, Joel > > On Sun, Oct 2, 2011 at 9:51 PM, Jay Kreps <jay.kr...@gmail.com> wrote: > >> Hi, >> >> Currently all the patch does is take input from stdin. Can you help me >> understand what you are looking for? >> >> What kind of plugin are you trying to write? >> >> -Jay >> >> >> On Thu, Sep 29, 2011 at 10:41 PM, Bao Thai Ngo <baothai...@gmail.com>wrote: >> >>> Jay, >>> >>> Does the patch already support log rotation (file rolling)? >>> >>> Also, I wonder if there is any documentation about how to write a plug-in >>> for Kafka. >>> >>> Thanks, >>> ~Thai >>> >>> On Mon, Sep 12, 2011 at 10:17 AM, Jay Kreps <jay.kr...@gmail.com> wrote: >>> >>> > I added a patch for this: >>> > https://issues.apache.org/jira/browse/KAFKA-130 >>> > >>> > Felix, if you are still interested in this, let me know if it works for >>> > you. >>> > >>> > -Jay >>> > >>> > On Thu, Sep 1, 2011 at 1:22 PM, Felix Giguere Villegas < >>> > felix.gigu...@mate1inc.com> wrote: >>> > >>> > > Hi :) >>> > > >>> > > Thanks for the reply :) . I have created >>> > > KAFKA-130<https://issues.apache.org/jira/browse/KAFKA-130> >>> > > . >>> > > >>> > > I'm not sure I have respected the proper standard for JIRA issues. >>> Please >>> > > let me know if I haven't. >>> > > >>> > > -- >>> > > Felix >>> > > >>> > > >>> > > >>> > > On Thu, Sep 1, 2011 at 4:04 PM, Jun Rao <jun...@gmail.com> wrote: >>> > > >>> > > > Hi, Felix, >>> > > > >>> > > > Currently, we don't have a util to pipe data from a file to a >>> producer. >>> > I >>> > > > agree that it's a very convenient tool to have. Could you open a >>> jira >>> > for >>> > > > that? >>> > > > >>> > > > Thanks, >>> > > > >>> > > > Jun >>> >> >
