Yes, that sounds like a reasonable plan for 0.7.1.
---------------------------------------- > Date: Wed, 5 Mar 2014 18:12:43 -0800 > Subject: Re: Summary of IRC Meeting in #apachehelix > From: [email protected] > To: [email protected] > > Looks like you guys had lot of fun. > > Agree with Kanak, the main goal of API works is to get the external > interface right and if we want to get there as soon as possible. > > I would like to finalize the API for all roles not just Administrator. So > here is the order I would suggest > > > 1. * Admin API > 2. * Admin Impl > 3. * Participant/Spectator/Controller api. > 4. * P/S/C implemenation. > > I think we need do atleast 3 to get a new release. what do you think > > thanks, > Kishore G > > > On Wed, Mar 5, 2014 at 11:02 AM, ASF IRC Bot > <[email protected]>wrote: > >> Summary of IRC Meeting in #apachehelix at Wed Mar 5 18:02:59 2014: >> >> Attendees: hsaputra, kanakb, osgigeek >> >> - Preface >> - 3/5/14 morning meeting >> >> >> IRC log follows: >> >> ## Preface ## >> ## 3/5/14 morning meeting ## >> [Wed Mar 5 18:05:23 2014] <kanakb>: so for those unaware, we decided to >> do biweekly morning meetings at 10am PT (including this week) and biweekly >> evening meetings at 10pm PT (starting last week, and the next is next >> Wednesday) >> [Wed Mar 5 18:05:48 2014] <osgigeek>: I have a few questions which >> probably will help set a broader scoped agenda or item >> [Wed Mar 5 18:06:11 2014] <osgigeek>: (1) What are our timelines for the >> API rework >> [Wed Mar 5 18:06:30 2014] <osgigeek>: (2) How do we merge back bug fixes >> which are made on the other branches into the reworked branch? >> [Wed Mar 5 18:07:11 2014] <osgigeek>: (3) How do we keep the rework >> aligned to make sure it functions as desired and does not deviate too far >> away from existing capability? >> [Wed Mar 5 18:07:30 2014] <osgigeek>: lets start there and then see if >> more questions arise >> [Wed Mar 5 18:08:05 2014] <kanakb>: so kishore probably has some thoughts >> on #1, but my view is that the api rework is very much a "when it's ready" >> type of thing >> [Wed Mar 5 18:08:18 2014] <kanakb>: we can push 0.6.x releases whenever >> we want to support existing users >> [Wed Mar 5 18:08:31 2014] <kanakb>: and 0.7 is something that we really >> want to just get right >> [Wed Mar 5 18:08:49 2014] <kanakb>: at the same time, i hope that we'll >> have something functional within the next month or so >> [Wed Mar 5 18:08:51 2014] <osgigeek>: ok good, just wanting to make sure >> so I know how hard to push it >> [Wed Mar 5 18:09:22 2014] <osgigeek>: on the note of something >> functional, can we set smaller goals as we go? >> [Wed Mar 5 18:09:35 2014] <osgigeek>: like we need 'x' functionality >> working then 'y' and so on >> [Wed Mar 5 18:09:38 2014] <kanakb>: yeah that's why i suggested breaking >> it down by role >> [Wed Mar 5 18:10:06 2014] <kanakb>: also, it probably makes sense to >> start with interfaces and skeletons >> [Wed Mar 5 18:10:14 2014] <kanakb>: just so that we know what it would be >> like to use these things >> [Wed Mar 5 18:10:26 2014] <kanakb>: so i'd say first milestone is just >> getting connections and admin correct >> [Wed Mar 5 18:10:44 2014] <kanakb>: and usable >> [Wed Mar 5 18:10:44 2014] <osgigeek>: yes bingo that is what I was >> looking for >> [Wed Mar 5 18:10:46 2014] <osgigeek>: milestones >> [Wed Mar 5 18:11:10 2014] <kanakb>: then next milestone is participant >> lifecycle >> [Wed Mar 5 18:11:18 2014] <kanakb>: so setting transition callbacks, etc >> [Wed Mar 5 18:11:31 2014] <kanakb>: then spectator >> [Wed Mar 5 18:11:31 2014] <kanakb>: then controller >> [Wed Mar 5 18:11:45 2014] <osgigeek>: nice kanakb: exactly the guideline >> I was seeking, this is good >> [Wed Mar 5 18:12:12 2014] <osgigeek>: so then we can focus on test cases >> assigned to those guidelines and make sure they are working before we move >> to the next set >> [Wed Mar 5 18:12:12 2014] <kanakb>: i would say admin+connection will >> take longer than anything else >> [Wed Mar 5 18:12:19 2014] <kanakb>: right >> [Wed Mar 5 18:12:30 2014] <osgigeek>: ok sounds like a plan >> [Wed Mar 5 18:12:42 2014] <kanakb>: ok so re:merging >> [Wed Mar 5 18:12:47 2014] <kanakb>: git makes this pretty easy >> [Wed Mar 5 18:12:59 2014] <kanakb>: depending on the change, we can >> either do a cherry pick or a straight merge >> [Wed Mar 5 18:13:20 2014] <kanakb>: it's really important that our >> existing tests don't significantly break >> [Wed Mar 5 18:13:22 2014] <osgigeek>: we probably need to track from the >> date the api rework started so we know what set of changes we are looking at >> [Wed Mar 5 18:13:30 2014] <kanakb>: because that serves as regression >> testing >> [Wed Mar 5 18:13:41 2014] <kanakb>: github has a tree view >> [Wed Mar 5 18:13:45 2014] <kanakb>: this is pretty easy to reconstruct >> [Wed Mar 5 18:14:39 2014] <osgigeek>: yes I am hoping the regression >> tests need simply be realigned to the new APIs >> [Wed Mar 5 18:14:51 2014] <osgigeek>: but the essence of the test should >> remain >> [Wed Mar 5 18:15:01 2014] <kanakb>: we need to be a little careful here >> [Wed Mar 5 18:15:11 2014] <kanakb>: if we're maintaining any level of >> backward compatibility >> [Wed Mar 5 18:15:21 2014] <kanakb>: we have to add a new test rather than >> changing the existing one >> [Wed Mar 5 18:16:06 2014] <osgigeek>: sure >> [Wed Mar 5 18:16:28 2014] <kanakb>: so re:keeping things aligned >> [Wed Mar 5 18:16:36 2014] <kanakb>: this is why we need to break it down >> by role >> [Wed Mar 5 18:16:48 2014] <kanakb>: if something is too hard to use as a >> user or too hard to implement as a helix developer >> [Wed Mar 5 18:16:53 2014] <kanakb>: then we need to iterate >> [Wed Mar 5 18:17:12 2014] <osgigeek>: which is fine, we can do so we >> still need to get to the HelixAdmin API which is our first API >> [Wed Mar 5 18:17:21 2014] <kanakb>: right >> [Wed Mar 5 18:18:03 2014] <osgigeek>: ok so I have input on most of the >> questions I had >> [Wed Mar 5 18:18:20 2014] <kanakb>: ok so questions from me >> [Wed Mar 5 18:18:24 2014] <osgigeek>: sure >> [Wed Mar 5 18:18:48 2014] <kanakb>: what do you envision your role being >> in this api rework, and where do you need the most assistance from kishore, >> jason, and me? >> [Wed Mar 5 18:19:40 2014] <osgigeek>: Good question, and you guys can >> help shape that role for sure but here is how I see it >> [Wed Mar 5 18:20:22 2014] <osgigeek>: I am hoping to help with the API >> definition, I would like for one of you guys to be there along the way >> because I am new to Helix and mistakes avoided is time saved >> [Wed Mar 5 18:20:32 2014] <osgigeek>: so active participation from one of >> you guys would surely help >> [Wed Mar 5 18:20:52 2014] <kanakb>: maybe as a code reviewer? >> [Wed Mar 5 18:21:18 2014] <osgigeek>: not just a code reviewer, but also >> as another individual contributing to the api rework >> [Wed Mar 5 18:21:24 2014] <kanakb>: i see >> [Wed Mar 5 18:21:43 2014] <osgigeek>: I dont expect myself to be the sole >> contributor to this rework >> [Wed Mar 5 18:21:52 2014] <kanakb>: i agree with this >> [Wed Mar 5 18:21:59 2014] <osgigeek>: I am hoping its a shared ownership >> between myself and one or more people whoever has time >> [Wed Mar 5 18:22:18 2014] <osgigeek>: In the process I imagine I will >> learn more on the internals and would like to contribute to the core >> subsequently as I draw more experience >> [Wed Mar 5 18:22:23 2014] <osgigeek>: so be it bug-fixes etc >> [Wed Mar 5 18:22:27 2014] <kanakb>: i've been a little swamped lately, >> but i should be able to be more available >> [Wed Mar 5 18:22:54 2014] <kanakb>: perhaps we can set up sync-up every >> other day on this irc channel to discuss the plan for the next two days >> [Wed Mar 5 18:22:57 2014] <osgigeek>: no worries I understand you guys >> are tied down so no complaints on that front >> [Wed Mar 5 18:23:25 2014] <osgigeek>: also a note I am contributing on my >> off hrs so progress will be slow, so please bear with me >> [Wed Mar 5 18:23:32 2014] <kanakb>: yeah totally understand >> [Wed Mar 5 18:24:02 2014] <osgigeek>: in the end I want to adopt Helix in >> my work >> [Wed Mar 5 18:24:08 2014] <osgigeek>: I do think its a good fit >> [Wed Mar 5 18:24:24 2014] <kanakb>: yeah definitely appreciate you taking >> the time to help make it better >> [Wed Mar 5 18:24:46 2014] <osgigeek>: there are also other aspects to it >> like zookeeper >> [Wed Mar 5 18:25:18 2014] <osgigeek>: there are opinions and I use the >> word opinions because it is just that, that zookeeper is hard to >> manage/maintain >> [Wed Mar 5 18:25:31 2014] <hsaputra>: Hi Guys, I am newbie in Helix but >> would like to help if possible. Need some rampup with it to know the >> internal nuts and bolts but I will help as needed. >> [Wed Mar 5 18:25:53 2014] <kanakb>: definitely >> [Wed Mar 5 18:26:17 2014] <kanakb>: please let me know if there are gaps >> in our current onboarding, so that i can help resolve them >> [Wed Mar 5 18:26:37 2014] <osgigeek>: hello hsaputra: welcome, I think >> your help will be very appreciated. I do send out summaries on the dev >> maillist so see if you can catch up on those mails >> [Wed Mar 5 18:27:08 2014] <hsaputra>: yeah, it has been long discussions >> =) Been trying to catch up with prev discussion about the new admin API >> [Wed Mar 5 18:27:40 2014] <osgigeek>: so sorry kanakb: going back to the >> topic of zookeeper, there are preferences over zookeeper like hazelcast >> [Wed Mar 5 18:28:14 2014] <osgigeek>: so when we put the spi in place >> having hazelcast as a provider might soothe out those concerned with >> zookeeper >> [Wed Mar 5 18:28:23 2014] <kanakb>: sure >> [Wed Mar 5 18:28:27 2014] <osgigeek>: i.e. if we think that fits >> [Wed Mar 5 18:28:31 2014] <kanakb>: that will be a good test actually >> [Wed Mar 5 18:28:38 2014] <osgigeek>: yes I agree >> [Wed Mar 5 18:29:31 2014] <kanakb>: so looks like hazelcast is AP >> [Wed Mar 5 18:29:40 2014] <kanakb>: we've wanted to have an AP >> alternative to zookeeper for some time >> [Wed Mar 5 18:29:58 2014] <hsaputra>: AP? >> [Wed Mar 5 18:30:00 2014] <osgigeek>: what is AP? >> [Wed Mar 5 18:30:26 2014] <kanakb>: CAP theorem is consistency, >> availability, partition tolerance >> [Wed Mar 5 18:30:41 2014] <kanakb>: its conjecture is that any >> distributed system has to give up at least one >> [Wed Mar 5 18:31:05 2014] <kanakb>: when zookeeper is exposed to network >> partition, it prefers consistency at the expense of availability >> [Wed Mar 5 18:31:08 2014] <kanakb>: so it is CP >> [Wed Mar 5 18:31:11 2014] <osgigeek>: ah I follow >> [Wed Mar 5 18:31:17 2014] <hsaputra>: ah ok =) >> [Wed Mar 5 18:31:49 2014] <osgigeek>: so I think going back to onboarding >> I think we should put together some document on just the API rework >> [Wed Mar 5 18:32:05 2014] <kanakb>: sure, i think the wiki is a good >> place for this >> [Wed Mar 5 18:32:07 2014] <osgigeek>: we can keep it an active document >> which keeps summary of models, APIs which have been finalized >> [Wed Mar 5 18:32:26 2014] <osgigeek>: that way people know what is on the >> table for discussion and what is off >> [Wed Mar 5 18:32:35 2014] <kanakb>: agree >> [Wed Mar 5 18:32:41 2014] <kanakb>: i can do this tonight >> [Wed Mar 5 18:33:04 2014] <kanakb>: will need your input though in case i >> miss something >> [Wed Mar 5 18:33:11 2014] <osgigeek>: and like kanakb: you mentioned when >> we meet regularly outside of this weekly on IRC we can keep the doc updated >> [Wed Mar 5 18:33:17 2014] <osgigeek>: yeah definitenly >> [Wed Mar 5 18:33:42 2014] <osgigeek>: I will help in keeping it updated, >> its for our benefit that we keep that up-dated >> [Wed Mar 5 18:33:49 2014] <kanakb>: ok >> [Wed Mar 5 18:34:07 2014] <kanakb>: sounds good >> [Wed Mar 5 18:35:06 2014] <osgigeek>: ok I dont have anything else to >> talk about and will need to get back to other tasks so if you dont mind I >> will drop off, unless you guys need me around >> [Wed Mar 5 18:35:19 2014] <kanakb>: no problem >> [Wed Mar 5 18:35:23 2014] <kanakb>: thanks for joining in >> [Wed Mar 5 18:35:45 2014] <osgigeek>: sure, np >> [Wed Mar 5 18:35:56 2014] <kanakb>: hsaputra: if you don't mind sharing, >> what is the use case you're planning to use helix for? >> [Wed Mar 5 18:36:33 2014] <hsaputra>: planning to use Helix as cluster >> management for Apache Tajo >> [Wed Mar 5 18:37:02 2014] <hsaputra>: that actually given that Helix >> could work with external provisioning framework like YARN or Mesos >> [Wed Mar 5 18:37:16 2014] <hsaputra>: I am planning to work with Kishore >> about that portion =) >> [Wed Mar 5 18:37:32 2014] <kanakb>: we've made some significant progress >> in that direction >> [Wed Mar 5 18:37:38 2014] <hsaputra>: yay =) >> [Wed Mar 5 18:37:39 2014] <kanakb>: at least for yarn >> [Wed Mar 5 18:38:00 2014] <hsaputra>: I reviewed the blog from Kishore >> [Wed Mar 5 18:38:09 2014] <kanakb>: cool >> [Wed Mar 5 18:38:44 2014] <kanakb>: so right now i'm working on finer >> grained assignment of tasks to run on the cluster >> [Wed Mar 5 18:38:58 2014] <kanakb>: i'll be sure to keep you posted on >> that if it's of interest >> [Wed Mar 5 18:39:13 2014] <hsaputra>: would love to know more >> [Wed Mar 5 18:39:42 2014] <kanakb>: in general, the motivation is that >> people don't know a priori how many resources their short to medium lived >> tasks will take up >> [Wed Mar 5 18:39:46 2014] <hsaputra>: to be honest Helix is a bit though >> for ramping up =) >> [Wed Mar 5 18:39:54 2014] <kanakb>: haha >> [Wed Mar 5 18:40:01 2014] <kanakb>: yeah for sure >> [Wed Mar 5 18:40:10 2014] <hsaputra>: the new API rewrite hopefully will >> reduce the friction to get more usage >> [Wed Mar 5 18:40:16 2014] <kanakb>: yeah >> [Wed Mar 5 18:40:32 2014] <kanakb>: the other problem we need to solve is >> how to get people through the design phase more painlessly >> [Wed Mar 5 18:40:48 2014] <hsaputra>: agree >> [Wed Mar 5 18:40:49 2014] <kanakb>: because one of the major things to do >> with helix is actually plan how your system would fit in the helix model >> [Wed Mar 5 18:41:09 2014] <hsaputra>: the premise of Helix is super >> useful if you designing from scratch is pretty hard to sell >> [Wed Mar 5 18:41:23 2014] <kanakb>: right >> [Wed Mar 5 18:41:50 2014] <hsaputra>: especially most devs already have >> somewhat system implemented that need to be migrated to scalable cluster >> management like Helix or others >> [Wed Mar 5 18:42:06 2014] <kanakb>: yeah so hopefully the python api and >> other things will also help with that >> [Wed Mar 5 18:42:19 2014] <kanakb>: since that can more easily manage >> cross-process than the jvm stuff >> [Wed Mar 5 18:42:30 2014] <hsaputra>: how would you propose change to >> improve the "design" portion to ingest Helix >> [Wed Mar 5 18:42:48 2014] <kanakb>: one thing is that we can keep writing >> recipes >> [Wed Mar 5 18:42:56 2014] <hsaputra>: agree >> [Wed Mar 5 18:43:06 2014] <kanakb>: showing how to move a single-node or >> hard-coded system can be adapted to work with helix >> [Wed Mar 5 18:44:15 2014] <kanakb>: the other thing we need to do is >> really "sell" helix as totally production-ready >> [Wed Mar 5 18:44:36 2014] <kanakb>: we've been using it reliably at >> linkedin for years, but from an ASF perspective, it still looks like a >> relatively young project >> [Wed Mar 5 18:44:51 2014] <kanakb>: especially since helix is used to >> manage online systems, people really have to trust it >> [Wed Mar 5 18:45:37 2014] <hsaputra>: I think a lot of similarities to >> Mesos as generic resource manager also add some "competition" >> [Wed Mar 5 18:45:50 2014] <kanakb>: yeah that's another communication >> thing >> [Wed Mar 5 18:46:07 2014] <kanakb>: since they solve mostly different >> problems >> [Wed Mar 5 18:46:09 2014] <kanakb>: and work better together >> [Wed Mar 5 18:47:33 2014] <hsaputra>: are you working on some branch for >> the YARN support? >> [Wed Mar 5 18:47:43 2014] <kanakb>: yeah helix-provisioning >> [Wed Mar 5 18:47:51 2014] <hsaputra>: I would like to test as early >> adopter and help if possible >> [Wed Mar 5 18:47:52 2014] <hsaputra>: ah cool >> [Wed Mar 5 18:48:04 2014] <kanakb>: i believe kishore sent some >> instructions a while back to get the helloworld example running >> [Wed Mar 5 18:48:13 2014] <hsaputra>: uff yeah >> [Wed Mar 5 18:48:22 2014] <hsaputra>: I forgot about that email >> [Wed Mar 5 18:48:43 2014] <kanakb>: the tough part is getting hadoop/yarn >> running, but you might not have that issue since you already work on tajo >> [Wed Mar 5 18:48:50 2014] <hsaputra>: and you and Kishore can take pull >> request via Github mirror for contributions I suppose? >> [Wed Mar 5 18:48:59 2014] <kanakb>: yeah for sure >> [Wed Mar 5 18:49:37 2014] <hsaputra>: hehehe sometimes Hadoop YARN is >> still hard to bring up even though been use it for a while =P >> [Wed Mar 5 18:49:49 2014] <hsaputra>: especially when new version >> available >> [Wed Mar 5 18:59:43 2014] <kanakb>: okay folks, anything else that we >> should get on the record? >> [Wed Mar 5 19:01:49 2014] <hsaputra>: I am good for now =) >> [Wed Mar 5 19:02:19 2014] <kanakb>: okay thanks everyone... we'll still >> be on IRC for general questions >> [Wed Mar 5 19:02:24 2014] <kanakb>: ASFBot: meeting end >> >> >> Meeting ended at Wed Mar 5 19:02:24 2014 >>
