So 1.7 would support Kafka 0.9. The issues are pretty minimal around the sink in master. Have you tried it? As for other means flume feels the purest in not manipulating the data. If you are shipping logs, Logstash (probably a bad word here :) is an option as it supports Kafka 0.9.
You could do something more hacky like piping into the Kafka console producer too. I know (elastic's) beats supports Kafka 0.9 in master but don't think it is released either. Remember Kafka 0.9 came out in Oct or Dec so it's pretty durn new compared to 0.8 which had over a year to get momentum. Let's not even talk about 0.10 ;) maybe this helps? -Joe On Jun 10, 2016 10:34 PM, "Jason J. W. Williams" <[email protected]> wrote: Yeah. Flume (relay)----WAN (Avro SSL)---->Flume (concentrator)--->Kafka. The remote Flume is talking to a "concentrator" flume in the Kafka datacenter, so that the data is transferred via SSL over the WAN, since the Kafka sink doesn't support SSL in 1.6.0. Otherwise, we could push from the remote Flume directly into Kafka over the WAN. -J On Fri, Jun 10, 2016 at 8:29 PM, Joe Lawson < [email protected]> wrote: > Do you mean that you are replicating topics over WAN using Flume as a > shipping agent? Sorry if I misunderstood the scenario. > On Jun 10, 2016 9:58 PM, "Jason J. W. Williams" <[email protected]> > wrote: > >> Thanks Joe that helps. What do you see folks using instead of Flume? >> We're using largely a WAN relay for Kafka for remote sites that don't >> warrant a local cluster. >> >> -J >> >> On Fri, Jun 10, 2016 at 6:17 PM, Joe Lawson < >> [email protected]> wrote: >> >>> Flume seems to have lost some stream. It's improving but it feels like >>> the priorities have become somewhat introspective so I'd say build from >>> master is you want the latest. >>> >>> Don't hold your breath for 1.7. just an opinion. No bearing on reality. >>> >>> -Joe >>> On Jun 10, 2016 12:45 PM, "Jason J. W. Williams" < >>> [email protected]> wrote: >>> >>>> Is there an estimated release date for Flume 1.7.0? Having TLS support >>>> in the Kafka sink would allow us to simplify our topology a lot.=20 >>>> >>>> -J >>> >>> >>
