On Tue, May 3, 2016 at 1:00 PM Banias H <banias4sp...@gmail.com> wrote:
> I should add Flume is not an option for various reasons. > > On Tue, May 3, 2016 at 2:49 PM, Banias H <banias4sp...@gmail.com> wrote: > > > We use Kafka (0.9.x) internally in our pipeline and now we would like to > > ingest application logs sitting in local file system of servers external > to > > the Kafka cluster. > > > > We could write a producer program running on the application servers to > > push files to Kafka. However we wonder if we can leverage Kafka Connect > to > > pull files into Kafka. It requires a connector to access local file > systems > > and I am not sure if I can make use of existing connectors or I need to > > write one. > > > > Any thoughts or reference would be helpful. > heka - great at reading arbitrary inputs, files being a first-class input, can write to kafka. next release should see use of librdkafka which should handle leader changeovers more smoothly than the existing go library. > > > > Thanks, > > B > > >
