I just created https://cwiki.apache.org/confluence/display/HELIX/API+Redesign+-+Progress so that we have a common place where we can track work and update whenever decisions are made.
Kanak ---------------------------------------- > From: [email protected] > To: [email protected] > Subject: RE: Summary of IRC Meeting in #apachehelix > Date: Wed, 5 Mar 2014 18:24:50 -0800 > > > Yes, that sounds like a reasonable plan for 0.7.1. > > ---------------------------------------- >> Date: Wed, 5 Mar 2014 18:12:43 -0800 >> Subject: Re: Summary of IRC Meeting in #apachehelix >> From: [email protected] >> To: [email protected] >> >> Looks like you guys had lot of fun. >> >> Agree with Kanak, the main goal of API works is to get the external >> interface right and if we want to get there as soon as possible. >> >> I would like to finalize the API for all roles not just Administrator. So >> here is the order I would suggest >> >> >> 1. * Admin API >> 2. * Admin Impl >> 3. * Participant/Spectator/Controller api. >> 4. * P/S/C implemenation. >> >> I think we need do atleast 3 to get a new release. what do you think >> >> thanks, >> Kishore G >> >> >> On Wed, Mar 5, 2014 at 11:02 AM, ASF IRC Bot >> <[email protected]>wrote: >> >>> Summary of IRC Meeting in #apachehelix at Wed Mar 5 18:02:59 2014: >>> >>> Attendees: hsaputra, kanakb, osgigeek >>> >>> - Preface >>> - 3/5/14 morning meeting >>> >>> >>> IRC log follows: >>> >>> ## Preface ## >>> ## 3/5/14 morning meeting ## >>> [Wed Mar 5 18:05:23 2014] <kanakb>: so for those unaware, we decided to >>> do biweekly morning meetings at 10am PT (including this week) and biweekly >>> evening meetings at 10pm PT (starting last week, and the next is next >>> Wednesday) >>> [Wed Mar 5 18:05:48 2014] <osgigeek>: I have a few questions which >>> probably will help set a broader scoped agenda or item >>> [Wed Mar 5 18:06:11 2014] <osgigeek>: (1) What are our timelines for the >>> API rework >>> [Wed Mar 5 18:06:30 2014] <osgigeek>: (2) How do we merge back bug fixes >>> which are made on the other branches into the reworked branch? >>> [Wed Mar 5 18:07:11 2014] <osgigeek>: (3) How do we keep the rework >>> aligned to make sure it functions as desired and does not deviate too far >>> away from existing capability? >>> [Wed Mar 5 18:07:30 2014] <osgigeek>: lets start there and then see if >>> more questions arise >>> [Wed Mar 5 18:08:05 2014] <kanakb>: so kishore probably has some thoughts >>> on #1, but my view is that the api rework is very much a "when it's ready" >>> type of thing >>> [Wed Mar 5 18:08:18 2014] <kanakb>: we can push 0.6.x releases whenever >>> we want to support existing users >>> [Wed Mar 5 18:08:31 2014] <kanakb>: and 0.7 is something that we really >>> want to just get right >>> [Wed Mar 5 18:08:49 2014] <kanakb>: at the same time, i hope that we'll >>> have something functional within the next month or so >>> [Wed Mar 5 18:08:51 2014] <osgigeek>: ok good, just wanting to make sure >>> so I know how hard to push it >>> [Wed Mar 5 18:09:22 2014] <osgigeek>: on the note of something >>> functional, can we set smaller goals as we go? >>> [Wed Mar 5 18:09:35 2014] <osgigeek>: like we need 'x' functionality >>> working then 'y' and so on >>> [Wed Mar 5 18:09:38 2014] <kanakb>: yeah that's why i suggested breaking >>> it down by role >>> [Wed Mar 5 18:10:06 2014] <kanakb>: also, it probably makes sense to >>> start with interfaces and skeletons >>> [Wed Mar 5 18:10:14 2014] <kanakb>: just so that we know what it would be >>> like to use these things >>> [Wed Mar 5 18:10:26 2014] <kanakb>: so i'd say first milestone is just >>> getting connections and admin correct >>> [Wed Mar 5 18:10:44 2014] <kanakb>: and usable >>> [Wed Mar 5 18:10:44 2014] <osgigeek>: yes bingo that is what I was >>> looking for >>> [Wed Mar 5 18:10:46 2014] <osgigeek>: milestones >>> [Wed Mar 5 18:11:10 2014] <kanakb>: then next milestone is participant >>> lifecycle >>> [Wed Mar 5 18:11:18 2014] <kanakb>: so setting transition callbacks, etc >>> [Wed Mar 5 18:11:31 2014] <kanakb>: then spectator >>> [Wed Mar 5 18:11:31 2014] <kanakb>: then controller >>> [Wed Mar 5 18:11:45 2014] <osgigeek>: nice kanakb: exactly the guideline >>> I was seeking, this is good >>> [Wed Mar 5 18:12:12 2014] <osgigeek>: so then we can focus on test cases >>> assigned to those guidelines and make sure they are working before we move >>> to the next set >>> [Wed Mar 5 18:12:12 2014] <kanakb>: i would say admin+connection will >>> take longer than anything else >>> [Wed Mar 5 18:12:19 2014] <kanakb>: right >>> [Wed Mar 5 18:12:30 2014] <osgigeek>: ok sounds like a plan >>> [Wed Mar 5 18:12:42 2014] <kanakb>: ok so re:merging >>> [Wed Mar 5 18:12:47 2014] <kanakb>: git makes this pretty easy >>> [Wed Mar 5 18:12:59 2014] <kanakb>: depending on the change, we can >>> either do a cherry pick or a straight merge >>> [Wed Mar 5 18:13:20 2014] <kanakb>: it's really important that our >>> existing tests don't significantly break >>> [Wed Mar 5 18:13:22 2014] <osgigeek>: we probably need to track from the >>> date the api rework started so we know what set of changes we are looking at >>> [Wed Mar 5 18:13:30 2014] <kanakb>: because that serves as regression >>> testing >>> [Wed Mar 5 18:13:41 2014] <kanakb>: github has a tree view >>> [Wed Mar 5 18:13:45 2014] <kanakb>: this is pretty easy to reconstruct >>> [Wed Mar 5 18:14:39 2014] <osgigeek>: yes I am hoping the regression >>> tests need simply be realigned to the new APIs >>> [Wed Mar 5 18:14:51 2014] <osgigeek>: but the essence of the test should >>> remain >>> [Wed Mar 5 18:15:01 2014] <kanakb>: we need to be a little careful here >>> [Wed Mar 5 18:15:11 2014] <kanakb>: if we're maintaining any level of >>> backward compatibility >>> [Wed Mar 5 18:15:21 2014] <kanakb>: we have to add a new test rather than >>> changing the existing one >>> [Wed Mar 5 18:16:06 2014] <osgigeek>: sure >>> [Wed Mar 5 18:16:28 2014] <kanakb>: so re:keeping things aligned >>> [Wed Mar 5 18:16:36 2014] <kanakb>: this is why we need to break it down >>> by role >>> [Wed Mar 5 18:16:48 2014] <kanakb>: if something is too hard to use as a >>> user or too hard to implement as a helix developer >>> [Wed Mar 5 18:16:53 2014] <kanakb>: then we need to iterate >>> [Wed Mar 5 18:17:12 2014] <osgigeek>: which is fine, we can do so we >>> still need to get to the HelixAdmin API which is our first API >>> [Wed Mar 5 18:17:21 2014] <kanakb>: right >>> [Wed Mar 5 18:18:03 2014] <osgigeek>: ok so I have input on most of the >>> questions I had >>> [Wed Mar 5 18:18:20 2014] <kanakb>: ok so questions from me >>> [Wed Mar 5 18:18:24 2014] <osgigeek>: sure >>> [Wed Mar 5 18:18:48 2014] <kanakb>: what do you envision your role being >>> in this api rework, and where do you need the most assistance from kishore, >>> jason, and me? >>> [Wed Mar 5 18:19:40 2014] <osgigeek>: Good question, and you guys can >>> help shape that role for sure but here is how I see it >>> [Wed Mar 5 18:20:22 2014] <osgigeek>: I am hoping to help with the API >>> definition, I would like for one of you guys to be there along the way >>> because I am new to Helix and mistakes avoided is time saved >>> [Wed Mar 5 18:20:32 2014] <osgigeek>: so active participation from one of >>> you guys would surely help >>> [Wed Mar 5 18:20:52 2014] <kanakb>: maybe as a code reviewer? >>> [Wed Mar 5 18:21:18 2014] <osgigeek>: not just a code reviewer, but also >>> as another individual contributing to the api rework >>> [Wed Mar 5 18:21:24 2014] <kanakb>: i see >>> [Wed Mar 5 18:21:43 2014] <osgigeek>: I dont expect myself to be the sole >>> contributor to this rework >>> [Wed Mar 5 18:21:52 2014] <kanakb>: i agree with this >>> [Wed Mar 5 18:21:59 2014] <osgigeek>: I am hoping its a shared ownership >>> between myself and one or more people whoever has time >>> [Wed Mar 5 18:22:18 2014] <osgigeek>: In the process I imagine I will >>> learn more on the internals and would like to contribute to the core >>> subsequently as I draw more experience >>> [Wed Mar 5 18:22:23 2014] <osgigeek>: so be it bug-fixes etc >>> [Wed Mar 5 18:22:27 2014] <kanakb>: i've been a little swamped lately, >>> but i should be able to be more available >>> [Wed Mar 5 18:22:54 2014] <kanakb>: perhaps we can set up sync-up every >>> other day on this irc channel to discuss the plan for the next two days >>> [Wed Mar 5 18:22:57 2014] <osgigeek>: no worries I understand you guys >>> are tied down so no complaints on that front >>> [Wed Mar 5 18:23:25 2014] <osgigeek>: also a note I am contributing on my >>> off hrs so progress will be slow, so please bear with me >>> [Wed Mar 5 18:23:32 2014] <kanakb>: yeah totally understand >>> [Wed Mar 5 18:24:02 2014] <osgigeek>: in the end I want to adopt Helix in >>> my work >>> [Wed Mar 5 18:24:08 2014] <osgigeek>: I do think its a good fit >>> [Wed Mar 5 18:24:24 2014] <kanakb>: yeah definitely appreciate you taking >>> the time to help make it better >>> [Wed Mar 5 18:24:46 2014] <osgigeek>: there are also other aspects to it >>> like zookeeper >>> [Wed Mar 5 18:25:18 2014] <osgigeek>: there are opinions and I use the >>> word opinions because it is just that, that zookeeper is hard to >>> manage/maintain >>> [Wed Mar 5 18:25:31 2014] <hsaputra>: Hi Guys, I am newbie in Helix but >>> would like to help if possible. Need some rampup with it to know the >>> internal nuts and bolts but I will help as needed. >>> [Wed Mar 5 18:25:53 2014] <kanakb>: definitely >>> [Wed Mar 5 18:26:17 2014] <kanakb>: please let me know if there are gaps >>> in our current onboarding, so that i can help resolve them >>> [Wed Mar 5 18:26:37 2014] <osgigeek>: hello hsaputra: welcome, I think >>> your help will be very appreciated. I do send out summaries on the dev >>> maillist so see if you can catch up on those mails >>> [Wed Mar 5 18:27:08 2014] <hsaputra>: yeah, it has been long discussions >>> =) Been trying to catch up with prev discussion about the new admin API >>> [Wed Mar 5 18:27:40 2014] <osgigeek>: so sorry kanakb: going back to the >>> topic of zookeeper, there are preferences over zookeeper like hazelcast >>> [Wed Mar 5 18:28:14 2014] <osgigeek>: so when we put the spi in place >>> having hazelcast as a provider might soothe out those concerned with >>> zookeeper >>> [Wed Mar 5 18:28:23 2014] <kanakb>: sure >>> [Wed Mar 5 18:28:27 2014] <osgigeek>: i.e. if we think that fits >>> [Wed Mar 5 18:28:31 2014] <kanakb>: that will be a good test actually >>> [Wed Mar 5 18:28:38 2014] <osgigeek>: yes I agree >>> [Wed Mar 5 18:29:31 2014] <kanakb>: so looks like hazelcast is AP >>> [Wed Mar 5 18:29:40 2014] <kanakb>: we've wanted to have an AP >>> alternative to zookeeper for some time >>> [Wed Mar 5 18:29:58 2014] <hsaputra>: AP? >>> [Wed Mar 5 18:30:00 2014] <osgigeek>: what is AP? >>> [Wed Mar 5 18:30:26 2014] <kanakb>: CAP theorem is consistency, >>> availability, partition tolerance >>> [Wed Mar 5 18:30:41 2014] <kanakb>: its conjecture is that any >>> distributed system has to give up at least one >>> [Wed Mar 5 18:31:05 2014] <kanakb>: when zookeeper is exposed to network >>> partition, it prefers consistency at the expense of availability >>> [Wed Mar 5 18:31:08 2014] <kanakb>: so it is CP >>> [Wed Mar 5 18:31:11 2014] <osgigeek>: ah I follow >>> [Wed Mar 5 18:31:17 2014] <hsaputra>: ah ok =) >>> [Wed Mar 5 18:31:49 2014] <osgigeek>: so I think going back to onboarding >>> I think we should put together some document on just the API rework >>> [Wed Mar 5 18:32:05 2014] <kanakb>: sure, i think the wiki is a good >>> place for this >>> [Wed Mar 5 18:32:07 2014] <osgigeek>: we can keep it an active document >>> which keeps summary of models, APIs which have been finalized >>> [Wed Mar 5 18:32:26 2014] <osgigeek>: that way people know what is on the >>> table for discussion and what is off >>> [Wed Mar 5 18:32:35 2014] <kanakb>: agree >>> [Wed Mar 5 18:32:41 2014] <kanakb>: i can do this tonight >>> [Wed Mar 5 18:33:04 2014] <kanakb>: will need your input though in case i >>> miss something >>> [Wed Mar 5 18:33:11 2014] <osgigeek>: and like kanakb: you mentioned when >>> we meet regularly outside of this weekly on IRC we can keep the doc updated >>> [Wed Mar 5 18:33:17 2014] <osgigeek>: yeah definitenly >>> [Wed Mar 5 18:33:42 2014] <osgigeek>: I will help in keeping it updated, >>> its for our benefit that we keep that up-dated >>> [Wed Mar 5 18:33:49 2014] <kanakb>: ok >>> [Wed Mar 5 18:34:07 2014] <kanakb>: sounds good >>> [Wed Mar 5 18:35:06 2014] <osgigeek>: ok I dont have anything else to >>> talk about and will need to get back to other tasks so if you dont mind I >>> will drop off, unless you guys need me around >>> [Wed Mar 5 18:35:19 2014] <kanakb>: no problem >>> [Wed Mar 5 18:35:23 2014] <kanakb>: thanks for joining in >>> [Wed Mar 5 18:35:45 2014] <osgigeek>: sure, np >>> [Wed Mar 5 18:35:56 2014] <kanakb>: hsaputra: if you don't mind sharing, >>> what is the use case you're planning to use helix for? >>> [Wed Mar 5 18:36:33 2014] <hsaputra>: planning to use Helix as cluster >>> management for Apache Tajo >>> [Wed Mar 5 18:37:02 2014] <hsaputra>: that actually given that Helix >>> could work with external provisioning framework like YARN or Mesos >>> [Wed Mar 5 18:37:16 2014] <hsaputra>: I am planning to work with Kishore >>> about that portion =) >>> [Wed Mar 5 18:37:32 2014] <kanakb>: we've made some significant progress >>> in that direction >>> [Wed Mar 5 18:37:38 2014] <hsaputra>: yay =) >>> [Wed Mar 5 18:37:39 2014] <kanakb>: at least for yarn >>> [Wed Mar 5 18:38:00 2014] <hsaputra>: I reviewed the blog from Kishore >>> [Wed Mar 5 18:38:09 2014] <kanakb>: cool >>> [Wed Mar 5 18:38:44 2014] <kanakb>: so right now i'm working on finer >>> grained assignment of tasks to run on the cluster >>> [Wed Mar 5 18:38:58 2014] <kanakb>: i'll be sure to keep you posted on >>> that if it's of interest >>> [Wed Mar 5 18:39:13 2014] <hsaputra>: would love to know more >>> [Wed Mar 5 18:39:42 2014] <kanakb>: in general, the motivation is that >>> people don't know a priori how many resources their short to medium lived >>> tasks will take up >>> [Wed Mar 5 18:39:46 2014] <hsaputra>: to be honest Helix is a bit though >>> for ramping up =) >>> [Wed Mar 5 18:39:54 2014] <kanakb>: haha >>> [Wed Mar 5 18:40:01 2014] <kanakb>: yeah for sure >>> [Wed Mar 5 18:40:10 2014] <hsaputra>: the new API rewrite hopefully will >>> reduce the friction to get more usage >>> [Wed Mar 5 18:40:16 2014] <kanakb>: yeah >>> [Wed Mar 5 18:40:32 2014] <kanakb>: the other problem we need to solve is >>> how to get people through the design phase more painlessly >>> [Wed Mar 5 18:40:48 2014] <hsaputra>: agree >>> [Wed Mar 5 18:40:49 2014] <kanakb>: because one of the major things to do >>> with helix is actually plan how your system would fit in the helix model >>> [Wed Mar 5 18:41:09 2014] <hsaputra>: the premise of Helix is super >>> useful if you designing from scratch is pretty hard to sell >>> [Wed Mar 5 18:41:23 2014] <kanakb>: right >>> [Wed Mar 5 18:41:50 2014] <hsaputra>: especially most devs already have >>> somewhat system implemented that need to be migrated to scalable cluster >>> management like Helix or others >>> [Wed Mar 5 18:42:06 2014] <kanakb>: yeah so hopefully the python api and >>> other things will also help with that >>> [Wed Mar 5 18:42:19 2014] <kanakb>: since that can more easily manage >>> cross-process than the jvm stuff >>> [Wed Mar 5 18:42:30 2014] <hsaputra>: how would you propose change to >>> improve the "design" portion to ingest Helix >>> [Wed Mar 5 18:42:48 2014] <kanakb>: one thing is that we can keep writing >>> recipes >>> [Wed Mar 5 18:42:56 2014] <hsaputra>: agree >>> [Wed Mar 5 18:43:06 2014] <kanakb>: showing how to move a single-node or >>> hard-coded system can be adapted to work with helix >>> [Wed Mar 5 18:44:15 2014] <kanakb>: the other thing we need to do is >>> really "sell" helix as totally production-ready >>> [Wed Mar 5 18:44:36 2014] <kanakb>: we've been using it reliably at >>> linkedin for years, but from an ASF perspective, it still looks like a >>> relatively young project >>> [Wed Mar 5 18:44:51 2014] <kanakb>: especially since helix is used to >>> manage online systems, people really have to trust it >>> [Wed Mar 5 18:45:37 2014] <hsaputra>: I think a lot of similarities to >>> Mesos as generic resource manager also add some "competition" >>> [Wed Mar 5 18:45:50 2014] <kanakb>: yeah that's another communication >>> thing >>> [Wed Mar 5 18:46:07 2014] <kanakb>: since they solve mostly different >>> problems >>> [Wed Mar 5 18:46:09 2014] <kanakb>: and work better together >>> [Wed Mar 5 18:47:33 2014] <hsaputra>: are you working on some branch for >>> the YARN support? >>> [Wed Mar 5 18:47:43 2014] <kanakb>: yeah helix-provisioning >>> [Wed Mar 5 18:47:51 2014] <hsaputra>: I would like to test as early >>> adopter and help if possible >>> [Wed Mar 5 18:47:52 2014] <hsaputra>: ah cool >>> [Wed Mar 5 18:48:04 2014] <kanakb>: i believe kishore sent some >>> instructions a while back to get the helloworld example running >>> [Wed Mar 5 18:48:13 2014] <hsaputra>: uff yeah >>> [Wed Mar 5 18:48:22 2014] <hsaputra>: I forgot about that email >>> [Wed Mar 5 18:48:43 2014] <kanakb>: the tough part is getting hadoop/yarn >>> running, but you might not have that issue since you already work on tajo >>> [Wed Mar 5 18:48:50 2014] <hsaputra>: and you and Kishore can take pull >>> request via Github mirror for contributions I suppose? >>> [Wed Mar 5 18:48:59 2014] <kanakb>: yeah for sure >>> [Wed Mar 5 18:49:37 2014] <hsaputra>: hehehe sometimes Hadoop YARN is >>> still hard to bring up even though been use it for a while =P >>> [Wed Mar 5 18:49:49 2014] <hsaputra>: especially when new version >>> available >>> [Wed Mar 5 18:59:43 2014] <kanakb>: okay folks, anything else that we >>> should get on the record? >>> [Wed Mar 5 19:01:49 2014] <hsaputra>: I am good for now =) >>> [Wed Mar 5 19:02:19 2014] <kanakb>: okay thanks everyone... we'll still >>> be on IRC for general questions >>> [Wed Mar 5 19:02:24 2014] <kanakb>: ASFBot: meeting end >>> >>> >>> Meeting ended at Wed Mar 5 19:02:24 2014 >>> >
