>> i mean, the log segments will always be in a consistent state on disk, right?
Yes. You can just copy over the topic directories and start the Kafka cluster Thanks, Neha On Fri, Nov 4, 2011 at 9:35 AM, Tim Lossen <t...@lossen.de> wrote: > ok, but apart from the possibility of data loss, rsync > in principle should work fine? i mean, the log segments > will always be in a consistent state on disk, right? > > > On 2011-11-04, at 17:18 , Neha Narkhede wrote: > >>>> for redundancy, we were planning to simply rsync the kafka >> message logs to a second machine periodically. or do you >> see any obvious problems with that? >> >> Rsync approach has several problems and could be a lossy solution. We >> moved away from that by replacing a legacy system with Kafka. >> We recommend you setup your redundant cluster using the mirroring >> approach, which is much more reliable and real-time than rsync. >> >> I think 0.6 has a stripped down version of mirroring, where you cannot >> control the mirroring for specific topics. >> >> Thanks, >> Neha >> >> On Fri, Nov 4, 2011 at 9:12 AM, Tim Lossen <t...@lossen.de> wrote: >>> >>> interesting. is this already available in 0.6? >>> >>> for redundancy, we were planning to simply rsync the kafka >>> message logs to a second machine periodically. or do you >>> see any obvious problems with that? >>> >>> cheers >>> tim >>> >>> On 2011-11-04, at 17:07 , Neha Narkhede wrote: >>>> We have a mirroring feature where you can setup 2 clusters, one to be a >>>> mirror of the other. At LinkedIn, we have a production Kafka cluster, and >>>> an analytics Kafka cluster that mirrors the production one in real time. We >>>> still haven't updated the documentation to describe this in detail. >>> >>> -- >>> http://tim.lossen.de >>> >>> >>> > > -- > http://tim.lossen.de > > > >
