There is discussion on the dev list about a Flume 1.7.0 release, but no committed date yet. We're making some progress towards that goal, though.
Mike On Fri, Jun 10, 2016 at 10:06 PM, Jason Williams <[email protected]> wrote: > Hey Joe, > > I may try 1.7 if you think it's pretty stable in master. Flume as a relay > actually solves our problem pretty well, in that if the Kafka DC becomes > unavailable for any reason, the Flume file channel will queue up the > entries reliably and replay when the link comes back. > > -J > > Sent via iPhone > > On Jun 10, 2016, at 21:25, Joe Lawson <[email protected]> > wrote: > > So 1.7 would support Kafka 0.9. The issues are pretty minimal around the > sink in master. Have you tried it? As for other means flume feels the > purest in not manipulating the data. If you are shipping logs, Logstash > (probably a bad word here :) is an option as it supports Kafka 0.9. > > You could do something more hacky like piping into the Kafka console > producer too. > > I know (elastic's) beats supports Kafka 0.9 in master but don't think it > is released either. Remember Kafka 0.9 came out in Oct or Dec so it's > pretty durn new compared to 0.8 which had over a year to get momentum. > Let's not even talk about 0.10 ;) > > maybe this helps? > > -Joe > On Jun 10, 2016 10:34 PM, "Jason J. W. Williams" < > [email protected]> wrote: > > Yeah. > > Flume (relay)----WAN (Avro SSL)---->Flume (concentrator)--->Kafka. > > The remote Flume is talking to a "concentrator" flume in the Kafka > datacenter, so that the data is transferred via SSL over the WAN, since the > Kafka sink doesn't support SSL in 1.6.0. Otherwise, we could push from the > remote Flume directly into Kafka over the WAN. > > -J > > > On Fri, Jun 10, 2016 at 8:29 PM, Joe Lawson < > [email protected]> wrote: > >> Do you mean that you are replicating topics over WAN using Flume as a >> shipping agent? Sorry if I misunderstood the scenario. >> On Jun 10, 2016 9:58 PM, "Jason J. W. Williams" < >> [email protected]> wrote: >> >>> Thanks Joe that helps. What do you see folks using instead of Flume? >>> We're using largely a WAN relay for Kafka for remote sites that don't >>> warrant a local cluster. >>> >>> -J >>> >>> On Fri, Jun 10, 2016 at 6:17 PM, Joe Lawson < >>> [email protected]> wrote: >>> >>>> Flume seems to have lost some stream. It's improving but it feels like >>>> the priorities have become somewhat introspective so I'd say build from >>>> master is you want the latest. >>>> >>>> Don't hold your breath for 1.7. just an opinion. No bearing on reality. >>>> >>>> -Joe >>>> On Jun 10, 2016 12:45 PM, "Jason J. W. Williams" < >>>> [email protected]> wrote: >>>> >>>>> Is there an estimated release date for Flume 1.7.0? Having TLS support >>>>> in the Kafka sink would allow us to simplify our topology a lot.=20 >>>>> >>>>> -J >>>> >>>> >>> >
