lynda, another possibility is to simply rsync the segment files periodically to another machine, and fire up a "standby" kafka there if the main one fails.
we use this setup on 0.6 in production, and it works quite well (but you can lose a few messages, of course). cheers tim On 2011-12-13, at 18:33 , Neha Narkhede wrote: >>> However, when I try to submit three properties files, I get a usage > error (see below). > > The wiki doesn't clearly state this, but it is written for Kafka v0.7 (to > be released soon). The setup instructions do not work with 0.6, since in > that version, the mirror didn't use a producer to write the data, it merely > used internal Kafka APIs to write to the local broker. > > We will update the wiki to reflect that, for now, would it work for you to > try this out on trunk ? Another option is using RC7 from branch > kafka/branches/0.7. > > Thanks, > Neha > > On Tue, Dec 13, 2011 at 9:07 AM, Carey, Lynda > <lynda.ca...@six3systems.com>wrote: > >> So, I've read all the documents that were suggested and now I'm trying to >> follow the instructions for "How to set up a Mirror" on this site: >> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+mirroring. The >> instructions state that you provide the broker with the 3 properties files. >> However, when I try to submit three properties files, I get a usage error >> (see below). >> >> [kafka %] JMX_PORT=8888 bin/kafka-server-start.sh >> config/mirror-server.properties config/mirror-consumer.properties >> config/mirror-producer.properties >> >> USAGE: java [options] KafkaServer server.properties [consumer.properties >> >> >> I am using kafka-0.6, downloaded from >> http://sna-projects.com/kafka/downloads.php >> >> >> _______________________________ >> Lynda Carey, Software Engineer >> Six3 Systems, Enterprise Systems Division >> 301-206-6000 (Office) >> 410-262-4942(Cell) >> >> This e-mail message is for the sole use of the intended recipient(s) and >> may contain Six3 Systems Private or Six3 Systems Proprietary information. >> Any unauthorized review, use, disclosure, or distribution is prohibited. If >> you are not an intended recipient, please contact the sender by reply >> e-mail and destroy all copies of the original message. >> >> >> ________________________________________ >> From: Jay Kreps [jay.kr...@gmail.com] >> Sent: Monday, December 12, 2011 11:49 PM >> To: kafka-users@incubator.apache.org >> Subject: Re: Kafak Mirroring Questions >> >> Hi Olivier, >> >> No the mirrored cluster is essentially a completely different cluster. It >> may have a different number of partitions or servers and there is no >> correspondence between offsets. >> >> -Jay >> >> On Mon, Dec 12, 2011 at 8:35 PM, Olivier Pomel <o...@datadoghq.com> wrote: >> >>> One more question: will the offsets for individual messages in the >>> master and mirror always be the same? In other words, would a failover >>> be completely transparent to consumers that may persist state linked >>> to specific offsets? >>> Thanks, >>> Olivier. >>> >>> On Mon, Dec 12, 2011 at 8:18 PM, Jun Rao <jun...@gmail.com> wrote: >>>> You can find more information in the patch of this jira: >>>> https://issues.apache.org/jira/browse/KAFKA-199 >>>> >>>> Thanks, >>>> >>>> Jun >>>> >>>> On Mon, Dec 12, 2011 at 11:57 AM, Carey, Lynda >>>> <lynda.ca...@six3systems.com>wrote: >>>> >>>>> Hi! I've been tasked with setting up a kafka cluster and mirroring. >>> I've >>>>> set up a small cluster of just two machines. I am running zookeeper, >>> two >>>>> brokers, 1 producer and 3 consumers. Each consumer is in it's own >> group >>>>> and all are reading from the same topic. So, with this setup, I'm >> using >>>>> simple console producer that reads from the command line and writes to >>>>> kafka. The consumer read the messages fine. >>>>> >>>>> Now, I need to set up a mirroring cluster. I've read the wiki >> document >>>>> about kafka mirroring and it leaves me a little confused. I >> understand >>>>> that the mirror cluster uses an embedded consumer to read from the >>> source >>>>> cluster and writes the messages to kafka on the mirrored cluster. >>> What's >>>>> not clear to me is what else needs to be established on the mirrored >>>>> cluster? Do I only need one embedded consumer and producer on the >>> mirror >>>>> to get all the messages (regardless of topic/broker/etc)? of do I >> need >>> a >>>>> pair of embedded consumer and producer for each topic? What is >> reading >>> the >>>>> messages on the mirrored cluster? -- do I need to deploy the same >>> consumers >>>>> there as on the source cluster? Is there any other documentation >>> regarding? >>>>> >>>>> Any information you can give me would be awesome. I'm just not >> getting >>> it >>>>> from the documentation alone. >>>>> >>>>> Thanks >>>>> _______________________________ >>>>> Lynda Carey, Software Engineer >>>>> Six3 Systems, Enterprise Systems Division >>>>> >>>>> This e-mail message is for the sole use of the intended recipient(s) >> and >>>>> may contain Six3 Systems Private or Six3 Systems Proprietary >>> information. >>>>> Any unauthorized review, use, disclosure, or distribution is >>> prohibited. If >>>>> you are not an intended recipient, please contact the sender by reply >>>>> e-mail and destroy all copies of the original message. >>>>> >>>>> >>> >> -- http://tim.lossen.de
