Re: Mirror Maker - Message Format Issue?

Hello, Just to close this issue out. The 8 producer going to the 10 cluster was the root issue. The mirror maker by default was unable to produce the message to the destination cluster. The work around was to include a MirrorMakerMessageHandler that did nothing but repackage the message again. In

Mirror Maker - Message Format Issue?

Hello, I think we're misunderstanding the docs on some level and I need a little clarification. We have the following setup: 1) 0.8.2 producer -> writing to Kafka 0.10.0.1 cluster w/ version 10 message format (source cluster). 2) 0.10.0.1 mirror using the 'new consumer' reading from the source

Re: Kafka 10 Consumer Reading from Kafka 8 Cluster?

; On Thu, Oct 6, 2016 at 12:00 PM Craig Swift > <craig.sw...@returnpath.com.invalid> wrote: > > > We're doing some fairly intensive data transformations in the current > > workers so it's not as straight forward as just reading/producing to > > another topic.

Re: Kafka 10 Consumer Reading from Kafka 8 Cluster?

luence/pages/viewpage.action?pageId=27846330 > > -David > > On 10/6/16, 1:32 PM, "Craig Swift" <craig.sw...@returnpath.com.INVALID> > wrote: > > Hello, > > We're in the process of upgrading several of our clusters to Kafka 10. > I > was wo

Kafka 10 Consumer Reading from Kafka 8 Cluster?

Hello, We're in the process of upgrading several of our clusters to Kafka 10. I was wondering if it's possible to use the Kafka 10 client code (old or new) to read from a source Kafka 8 cluster and then use the new 10 producer to write to a destination Kafka 10 cluster? I know there's a

Re: Running Zookeeper on same instance as kafka brokers in multi node cluster

Hello, Initially we had our setup with both ZK and Kafka on the same nodes as well. Over time though this proved problematic as we both increased the usage of Kafka and had to scale out the cluster. ZK specifically can be very touchy if it gets IO bound - so when the Kafka cluster would get under

JMXTrans Monitoring for Individual Topics

Hello, I was hoping someone might know this off the top of their head. I haven't been able to find any documentation on it so I was curious if it was still available. We're upgrading to Kafka 8.2 from 8.1 (ya, we're a bit behind) and we used to have the following in jmxtrans that would give us

Re: [DISCUSS] Java 8 as a minimum requirement

+1 Craig J. Swift Principal Software Engineer - Data Pipeline ReturnPath Inc. Work: 303-999-3220 Cell: 720-560-7038 On Thu, Jun 16, 2016 at 2:50 PM, Henry Cai wrote: > +1 for Lambda expression. > > On Thu, Jun 16, 2016 at 1:48 PM, Rajiv Kurian

Re: Will Mirror Maker only support 1 to 1?

Hello, We've done a variety of different mirror configurations in the past (we mirror from AWS into several different data centers) including the first one you describe. In fact for some high volume / large message topics we actually found splitting it up and placing a dedicated mirror for the

Re: MirrorMaker - Not consuming from all partitions

to be doing a capture all the > time, and then figure out when MirrorMaker stops fetching from that broker, > and then stare at a ton of data to find whatever protocol thing has > happened -- all of which is ugly enough that I'd been reluctant to mention > it until now). > >

Re: MirrorMaker - Not consuming from all partitions

sure how the MirrorMaker cycles through it's topics/partitions and why it would read very slowly or bypass reading from certain partitions. Craig J. Swift Software Engineer - Data Pipeline ReturnPath Inc. Work: 303-999-3220 Cell: 720-560-7038 On Wed, Sep 9, 2015 at 9:29 AM, Craig Swift <craig

MirrorMaker - Not consuming from all partitions

Hello, Hope everyone is doing well. I was hoping to get some assistance with a strange issue we're experiencing while using the MirrorMaker to pull data down from an 8 node Kafka cluster in AWS into our data center. Both Kafka clusters and the mirror are using version 0.8.1.1 with dedicated