We would love to have help! There is a tree of JIRAs now under KAFKA-50 that give some break down of the work but they are very very sparse and don't capture the dependencies between items. Some work has started on (KAFKA-48), but for the last month or so we linkedin folks have been focusing on compression and the 0.7 release as well as our internal rollout of 0.7 across our clusters. Jun could chime in with more details, our roadmap looks a little like this:
- This week we are working on 0.7 open source release - The next two weeks (I think) we had planned work on a few operational issues and bugs we are hitting as well as go to ApacheCon - Then we will start the replication work in earnest full time as our primary focus We would *love* to get some help on this--it is a fairly big feature and could definitely benefit from more brains and hands as well as just people thinking through different trade-offs against their own use cases. So maybe a good plan in the neartime (say next week) is try to expand out the work in more detail. I think it would be useful to get the JIRAs to have a lot more details and work out some of the dependencies between projects. Once the dependencies are worked out I think any of those JIRAs could be picked up by any interested person and we can start to do the design and implementation discussions on that aspect. Jun, does that sound reasonable? Did I correctly summarize your plan? -Jay On Fri, Nov 4, 2011 at 11:53 AM, Joe Stein <crypt...@gmail.com> wrote: > Thanks for quick responses. > > Jun: KAFKA-50 looks pretty interesting. I am going to go through it in > more detail again tonight. This feature not being in yet is not going to > block me from getting going but will for wide spread use I think. Is there > any opportunity to help contribute to this? Maybe start with something > else (smaller) might be better to get my feet wet? I am going to start to > cozy up to the code this weekend.... right now "Test Starting: > testProduceAndMultiFetch(kafka.javaapi.integration.PrimitiveApiTest)" keeps > hanging but maybe it resource issue on my machine (will try it on another > machine and if still and issue will send to dev) > > Neha: I think as you said "just copy over the topic directories and start > the Kafka cluster" for now will be a sufficient approach if/when a server > dies since it sounds like another broker (Y) would get the requests when > broker (X) dies (this is an assumption and easy enough for me to test). > For the "at least" delivery guarantee I am assuming you are asking what we > would do if we get another event that maybe we already got because of a > failure? We actually deal with this type of thing a lot already (mostly > from our mobile devices). It depends on the data that is flowing (in this > paradigm I guess it would be the topic) and if N errors occurred within the > last T seconds each processor has logic what todo (even some a > system.exit())... with Kafka I would hope we would know when this happens > meaning we know an error has occurred and we might for this one row get it > again (in which case we could keep it somewhere resident to validate for Y > time period). > > /* > Joe Stein > http://www.linkedin.com/in/charmalloc > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> > */ > > On Fri, Nov 4, 2011 at 12:41 PM, Neha Narkhede <neha.narkh...@gmail.com > >wrote: > > > >> i mean, the log segments > > will always be in a consistent state on disk, right? > > > > Yes. You can just copy over the topic directories and start the Kafka > > cluster > > > > Thanks, > > Neha > > > > On Fri, Nov 4, 2011 at 9:35 AM, Tim Lossen <t...@lossen.de> wrote: > > > ok, but apart from the possibility of data loss, rsync > > > in principle should work fine? i mean, the log segments > > > will always be in a consistent state on disk, right? > > > > > > > > > On 2011-11-04, at 17:18 , Neha Narkhede wrote: > > > > > >>>> for redundancy, we were planning to simply rsync the kafka > > >> message logs to a second machine periodically. or do you > > >> see any obvious problems with that? > > >> > > >> Rsync approach has several problems and could be a lossy solution. We > > >> moved away from that by replacing a legacy system with Kafka. > > >> We recommend you setup your redundant cluster using the mirroring > > >> approach, which is much more reliable and real-time than rsync. > > >> > > >> I think 0.6 has a stripped down version of mirroring, where you cannot > > >> control the mirroring for specific topics. > > >> > > >> Thanks, > > >> Neha > > >> > > >> On Fri, Nov 4, 2011 at 9:12 AM, Tim Lossen <t...@lossen.de> wrote: > > >>> > > >>> interesting. is this already available in 0.6? > > >>> > > >>> for redundancy, we were planning to simply rsync the kafka > > >>> message logs to a second machine periodically. or do you > > >>> see any obvious problems with that? > > >>> > > >>> cheers > > >>> tim > > >>> > > >>> On 2011-11-04, at 17:07 , Neha Narkhede wrote: > > >>>> We have a mirroring feature where you can setup 2 clusters, one to > be > > a > > >>>> mirror of the other. At LinkedIn, we have a production Kafka > cluster, > > and > > >>>> an analytics Kafka cluster that mirrors the production one in real > > time. We > > >>>> still haven't updated the documentation to describe this in detail. > > >>> > > >>> -- > > >>> http://tim.lossen.de > > >>> > > >>> > > >>> > > > > > > -- > > > http://tim.lossen.de > > > > > > > > > > > > > > >
