ok, but apart from the possibility of data loss, rsync in principle should work fine? i mean, the log segments will always be in a consistent state on disk, right?
On 2011-11-04, at 17:18 , Neha Narkhede wrote: >>> for redundancy, we were planning to simply rsync the kafka > message logs to a second machine periodically. or do you > see any obvious problems with that? > > Rsync approach has several problems and could be a lossy solution. We > moved away from that by replacing a legacy system with Kafka. > We recommend you setup your redundant cluster using the mirroring > approach, which is much more reliable and real-time than rsync. > > I think 0.6 has a stripped down version of mirroring, where you cannot > control the mirroring for specific topics. > > Thanks, > Neha > > On Fri, Nov 4, 2011 at 9:12 AM, Tim Lossen <t...@lossen.de> wrote: >> >> interesting. is this already available in 0.6? >> >> for redundancy, we were planning to simply rsync the kafka >> message logs to a second machine periodically. or do you >> see any obvious problems with that? >> >> cheers >> tim >> >> On 2011-11-04, at 17:07 , Neha Narkhede wrote: >>> We have a mirroring feature where you can setup 2 clusters, one to be a >>> mirror of the other. At LinkedIn, we have a production Kafka cluster, and >>> an analytics Kafka cluster that mirrors the production one in real time. We >>> still haven't updated the documentation to describe this in detail. >> >> -- >> http://tim.lossen.de >> >> >> -- http://tim.lossen.de
