Summary of IRC Meeting in #apachehelix at Wed Mar 5 18:02:59 2014: Attendees: hsaputra, kanakb, osgigeek
- Preface - 3/5/14 morning meeting IRC log follows: ## Preface ## ## 3/5/14 morning meeting ## [Wed Mar 5 18:05:23 2014] <kanakb>: so for those unaware, we decided to do biweekly morning meetings at 10am PT (including this week) and biweekly evening meetings at 10pm PT (starting last week, and the next is next Wednesday) [Wed Mar 5 18:05:48 2014] <osgigeek>: I have a few questions which probably will help set a broader scoped agenda or item [Wed Mar 5 18:06:11 2014] <osgigeek>: (1) What are our timelines for the API rework [Wed Mar 5 18:06:30 2014] <osgigeek>: (2) How do we merge back bug fixes which are made on the other branches into the reworked branch? [Wed Mar 5 18:07:11 2014] <osgigeek>: (3) How do we keep the rework aligned to make sure it functions as desired and does not deviate too far away from existing capability? [Wed Mar 5 18:07:30 2014] <osgigeek>: lets start there and then see if more questions arise [Wed Mar 5 18:08:05 2014] <kanakb>: so kishore probably has some thoughts on #1, but my view is that the api rework is very much a "when it's ready" type of thing [Wed Mar 5 18:08:18 2014] <kanakb>: we can push 0.6.x releases whenever we want to support existing users [Wed Mar 5 18:08:31 2014] <kanakb>: and 0.7 is something that we really want to just get right [Wed Mar 5 18:08:49 2014] <kanakb>: at the same time, i hope that we'll have something functional within the next month or so [Wed Mar 5 18:08:51 2014] <osgigeek>: ok good, just wanting to make sure so I know how hard to push it [Wed Mar 5 18:09:22 2014] <osgigeek>: on the note of something functional, can we set smaller goals as we go? [Wed Mar 5 18:09:35 2014] <osgigeek>: like we need 'x' functionality working then 'y' and so on [Wed Mar 5 18:09:38 2014] <kanakb>: yeah that's why i suggested breaking it down by role [Wed Mar 5 18:10:06 2014] <kanakb>: also, it probably makes sense to start with interfaces and skeletons [Wed Mar 5 18:10:14 2014] <kanakb>: just so that we know what it would be like to use these things [Wed Mar 5 18:10:26 2014] <kanakb>: so i'd say first milestone is just getting connections and admin correct [Wed Mar 5 18:10:44 2014] <kanakb>: and usable [Wed Mar 5 18:10:44 2014] <osgigeek>: yes bingo that is what I was looking for [Wed Mar 5 18:10:46 2014] <osgigeek>: milestones [Wed Mar 5 18:11:10 2014] <kanakb>: then next milestone is participant lifecycle [Wed Mar 5 18:11:18 2014] <kanakb>: so setting transition callbacks, etc [Wed Mar 5 18:11:31 2014] <kanakb>: then spectator [Wed Mar 5 18:11:31 2014] <kanakb>: then controller [Wed Mar 5 18:11:45 2014] <osgigeek>: nice kanakb: exactly the guideline I was seeking, this is good [Wed Mar 5 18:12:12 2014] <osgigeek>: so then we can focus on test cases assigned to those guidelines and make sure they are working before we move to the next set [Wed Mar 5 18:12:12 2014] <kanakb>: i would say admin+connection will take longer than anything else [Wed Mar 5 18:12:19 2014] <kanakb>: right [Wed Mar 5 18:12:30 2014] <osgigeek>: ok sounds like a plan [Wed Mar 5 18:12:42 2014] <kanakb>: ok so re:merging [Wed Mar 5 18:12:47 2014] <kanakb>: git makes this pretty easy [Wed Mar 5 18:12:59 2014] <kanakb>: depending on the change, we can either do a cherry pick or a straight merge [Wed Mar 5 18:13:20 2014] <kanakb>: it's really important that our existing tests don't significantly break [Wed Mar 5 18:13:22 2014] <osgigeek>: we probably need to track from the date the api rework started so we know what set of changes we are looking at [Wed Mar 5 18:13:30 2014] <kanakb>: because that serves as regression testing [Wed Mar 5 18:13:41 2014] <kanakb>: github has a tree view [Wed Mar 5 18:13:45 2014] <kanakb>: this is pretty easy to reconstruct [Wed Mar 5 18:14:39 2014] <osgigeek>: yes I am hoping the regression tests need simply be realigned to the new APIs [Wed Mar 5 18:14:51 2014] <osgigeek>: but the essence of the test should remain [Wed Mar 5 18:15:01 2014] <kanakb>: we need to be a little careful here [Wed Mar 5 18:15:11 2014] <kanakb>: if we're maintaining any level of backward compatibility [Wed Mar 5 18:15:21 2014] <kanakb>: we have to add a new test rather than changing the existing one [Wed Mar 5 18:16:06 2014] <osgigeek>: sure [Wed Mar 5 18:16:28 2014] <kanakb>: so re:keeping things aligned [Wed Mar 5 18:16:36 2014] <kanakb>: this is why we need to break it down by role [Wed Mar 5 18:16:48 2014] <kanakb>: if something is too hard to use as a user or too hard to implement as a helix developer [Wed Mar 5 18:16:53 2014] <kanakb>: then we need to iterate [Wed Mar 5 18:17:12 2014] <osgigeek>: which is fine, we can do so we still need to get to the HelixAdmin API which is our first API [Wed Mar 5 18:17:21 2014] <kanakb>: right [Wed Mar 5 18:18:03 2014] <osgigeek>: ok so I have input on most of the questions I had [Wed Mar 5 18:18:20 2014] <kanakb>: ok so questions from me [Wed Mar 5 18:18:24 2014] <osgigeek>: sure [Wed Mar 5 18:18:48 2014] <kanakb>: what do you envision your role being in this api rework, and where do you need the most assistance from kishore, jason, and me? [Wed Mar 5 18:19:40 2014] <osgigeek>: Good question, and you guys can help shape that role for sure but here is how I see it [Wed Mar 5 18:20:22 2014] <osgigeek>: I am hoping to help with the API definition, I would like for one of you guys to be there along the way because I am new to Helix and mistakes avoided is time saved [Wed Mar 5 18:20:32 2014] <osgigeek>: so active participation from one of you guys would surely help [Wed Mar 5 18:20:52 2014] <kanakb>: maybe as a code reviewer? [Wed Mar 5 18:21:18 2014] <osgigeek>: not just a code reviewer, but also as another individual contributing to the api rework [Wed Mar 5 18:21:24 2014] <kanakb>: i see [Wed Mar 5 18:21:43 2014] <osgigeek>: I dont expect myself to be the sole contributor to this rework [Wed Mar 5 18:21:52 2014] <kanakb>: i agree with this [Wed Mar 5 18:21:59 2014] <osgigeek>: I am hoping its a shared ownership between myself and one or more people whoever has time [Wed Mar 5 18:22:18 2014] <osgigeek>: In the process I imagine I will learn more on the internals and would like to contribute to the core subsequently as I draw more experience [Wed Mar 5 18:22:23 2014] <osgigeek>: so be it bug-fixes etc [Wed Mar 5 18:22:27 2014] <kanakb>: i've been a little swamped lately, but i should be able to be more available [Wed Mar 5 18:22:54 2014] <kanakb>: perhaps we can set up sync-up every other day on this irc channel to discuss the plan for the next two days [Wed Mar 5 18:22:57 2014] <osgigeek>: no worries I understand you guys are tied down so no complaints on that front [Wed Mar 5 18:23:25 2014] <osgigeek>: also a note I am contributing on my off hrs so progress will be slow, so please bear with me [Wed Mar 5 18:23:32 2014] <kanakb>: yeah totally understand [Wed Mar 5 18:24:02 2014] <osgigeek>: in the end I want to adopt Helix in my work [Wed Mar 5 18:24:08 2014] <osgigeek>: I do think its a good fit [Wed Mar 5 18:24:24 2014] <kanakb>: yeah definitely appreciate you taking the time to help make it better [Wed Mar 5 18:24:46 2014] <osgigeek>: there are also other aspects to it like zookeeper [Wed Mar 5 18:25:18 2014] <osgigeek>: there are opinions and I use the word opinions because it is just that, that zookeeper is hard to manage/maintain [Wed Mar 5 18:25:31 2014] <hsaputra>: Hi Guys, I am newbie in Helix but would like to help if possible. Need some rampup with it to know the internal nuts and bolts but I will help as needed. [Wed Mar 5 18:25:53 2014] <kanakb>: definitely [Wed Mar 5 18:26:17 2014] <kanakb>: please let me know if there are gaps in our current onboarding, so that i can help resolve them [Wed Mar 5 18:26:37 2014] <osgigeek>: hello hsaputra: welcome, I think your help will be very appreciated. I do send out summaries on the dev maillist so see if you can catch up on those mails [Wed Mar 5 18:27:08 2014] <hsaputra>: yeah, it has been long discussions =) Been trying to catch up with prev discussion about the new admin API [Wed Mar 5 18:27:40 2014] <osgigeek>: so sorry kanakb: going back to the topic of zookeeper, there are preferences over zookeeper like hazelcast [Wed Mar 5 18:28:14 2014] <osgigeek>: so when we put the spi in place having hazelcast as a provider might soothe out those concerned with zookeeper [Wed Mar 5 18:28:23 2014] <kanakb>: sure [Wed Mar 5 18:28:27 2014] <osgigeek>: i.e. if we think that fits [Wed Mar 5 18:28:31 2014] <kanakb>: that will be a good test actually [Wed Mar 5 18:28:38 2014] <osgigeek>: yes I agree [Wed Mar 5 18:29:31 2014] <kanakb>: so looks like hazelcast is AP [Wed Mar 5 18:29:40 2014] <kanakb>: we've wanted to have an AP alternative to zookeeper for some time [Wed Mar 5 18:29:58 2014] <hsaputra>: AP? [Wed Mar 5 18:30:00 2014] <osgigeek>: what is AP? [Wed Mar 5 18:30:26 2014] <kanakb>: CAP theorem is consistency, availability, partition tolerance [Wed Mar 5 18:30:41 2014] <kanakb>: its conjecture is that any distributed system has to give up at least one [Wed Mar 5 18:31:05 2014] <kanakb>: when zookeeper is exposed to network partition, it prefers consistency at the expense of availability [Wed Mar 5 18:31:08 2014] <kanakb>: so it is CP [Wed Mar 5 18:31:11 2014] <osgigeek>: ah I follow [Wed Mar 5 18:31:17 2014] <hsaputra>: ah ok =) [Wed Mar 5 18:31:49 2014] <osgigeek>: so I think going back to onboarding I think we should put together some document on just the API rework [Wed Mar 5 18:32:05 2014] <kanakb>: sure, i think the wiki is a good place for this [Wed Mar 5 18:32:07 2014] <osgigeek>: we can keep it an active document which keeps summary of models, APIs which have been finalized [Wed Mar 5 18:32:26 2014] <osgigeek>: that way people know what is on the table for discussion and what is off [Wed Mar 5 18:32:35 2014] <kanakb>: agree [Wed Mar 5 18:32:41 2014] <kanakb>: i can do this tonight [Wed Mar 5 18:33:04 2014] <kanakb>: will need your input though in case i miss something [Wed Mar 5 18:33:11 2014] <osgigeek>: and like kanakb: you mentioned when we meet regularly outside of this weekly on IRC we can keep the doc updated [Wed Mar 5 18:33:17 2014] <osgigeek>: yeah definitenly [Wed Mar 5 18:33:42 2014] <osgigeek>: I will help in keeping it updated, its for our benefit that we keep that up-dated [Wed Mar 5 18:33:49 2014] <kanakb>: ok [Wed Mar 5 18:34:07 2014] <kanakb>: sounds good [Wed Mar 5 18:35:06 2014] <osgigeek>: ok I dont have anything else to talk about and will need to get back to other tasks so if you dont mind I will drop off, unless you guys need me around [Wed Mar 5 18:35:19 2014] <kanakb>: no problem [Wed Mar 5 18:35:23 2014] <kanakb>: thanks for joining in [Wed Mar 5 18:35:45 2014] <osgigeek>: sure, np [Wed Mar 5 18:35:56 2014] <kanakb>: hsaputra: if you don't mind sharing, what is the use case you're planning to use helix for? [Wed Mar 5 18:36:33 2014] <hsaputra>: planning to use Helix as cluster management for Apache Tajo [Wed Mar 5 18:37:02 2014] <hsaputra>: that actually given that Helix could work with external provisioning framework like YARN or Mesos [Wed Mar 5 18:37:16 2014] <hsaputra>: I am planning to work with Kishore about that portion =) [Wed Mar 5 18:37:32 2014] <kanakb>: we've made some significant progress in that direction [Wed Mar 5 18:37:38 2014] <hsaputra>: yay =) [Wed Mar 5 18:37:39 2014] <kanakb>: at least for yarn [Wed Mar 5 18:38:00 2014] <hsaputra>: I reviewed the blog from Kishore [Wed Mar 5 18:38:09 2014] <kanakb>: cool [Wed Mar 5 18:38:44 2014] <kanakb>: so right now i'm working on finer grained assignment of tasks to run on the cluster [Wed Mar 5 18:38:58 2014] <kanakb>: i'll be sure to keep you posted on that if it's of interest [Wed Mar 5 18:39:13 2014] <hsaputra>: would love to know more [Wed Mar 5 18:39:42 2014] <kanakb>: in general, the motivation is that people don't know a priori how many resources their short to medium lived tasks will take up [Wed Mar 5 18:39:46 2014] <hsaputra>: to be honest Helix is a bit though for ramping up =) [Wed Mar 5 18:39:54 2014] <kanakb>: haha [Wed Mar 5 18:40:01 2014] <kanakb>: yeah for sure [Wed Mar 5 18:40:10 2014] <hsaputra>: the new API rewrite hopefully will reduce the friction to get more usage [Wed Mar 5 18:40:16 2014] <kanakb>: yeah [Wed Mar 5 18:40:32 2014] <kanakb>: the other problem we need to solve is how to get people through the design phase more painlessly [Wed Mar 5 18:40:48 2014] <hsaputra>: agree [Wed Mar 5 18:40:49 2014] <kanakb>: because one of the major things to do with helix is actually plan how your system would fit in the helix model [Wed Mar 5 18:41:09 2014] <hsaputra>: the premise of Helix is super useful if you designing from scratch is pretty hard to sell [Wed Mar 5 18:41:23 2014] <kanakb>: right [Wed Mar 5 18:41:50 2014] <hsaputra>: especially most devs already have somewhat system implemented that need to be migrated to scalable cluster management like Helix or others [Wed Mar 5 18:42:06 2014] <kanakb>: yeah so hopefully the python api and other things will also help with that [Wed Mar 5 18:42:19 2014] <kanakb>: since that can more easily manage cross-process than the jvm stuff [Wed Mar 5 18:42:30 2014] <hsaputra>: how would you propose change to improve the "design" portion to ingest Helix [Wed Mar 5 18:42:48 2014] <kanakb>: one thing is that we can keep writing recipes [Wed Mar 5 18:42:56 2014] <hsaputra>: agree [Wed Mar 5 18:43:06 2014] <kanakb>: showing how to move a single-node or hard-coded system can be adapted to work with helix [Wed Mar 5 18:44:15 2014] <kanakb>: the other thing we need to do is really "sell" helix as totally production-ready [Wed Mar 5 18:44:36 2014] <kanakb>: we've been using it reliably at linkedin for years, but from an ASF perspective, it still looks like a relatively young project [Wed Mar 5 18:44:51 2014] <kanakb>: especially since helix is used to manage online systems, people really have to trust it [Wed Mar 5 18:45:37 2014] <hsaputra>: I think a lot of similarities to Mesos as generic resource manager also add some "competition" [Wed Mar 5 18:45:50 2014] <kanakb>: yeah that's another communication thing [Wed Mar 5 18:46:07 2014] <kanakb>: since they solve mostly different problems [Wed Mar 5 18:46:09 2014] <kanakb>: and work better together [Wed Mar 5 18:47:33 2014] <hsaputra>: are you working on some branch for the YARN support? [Wed Mar 5 18:47:43 2014] <kanakb>: yeah helix-provisioning [Wed Mar 5 18:47:51 2014] <hsaputra>: I would like to test as early adopter and help if possible [Wed Mar 5 18:47:52 2014] <hsaputra>: ah cool [Wed Mar 5 18:48:04 2014] <kanakb>: i believe kishore sent some instructions a while back to get the helloworld example running [Wed Mar 5 18:48:13 2014] <hsaputra>: uff yeah [Wed Mar 5 18:48:22 2014] <hsaputra>: I forgot about that email [Wed Mar 5 18:48:43 2014] <kanakb>: the tough part is getting hadoop/yarn running, but you might not have that issue since you already work on tajo [Wed Mar 5 18:48:50 2014] <hsaputra>: and you and Kishore can take pull request via Github mirror for contributions I suppose? [Wed Mar 5 18:48:59 2014] <kanakb>: yeah for sure [Wed Mar 5 18:49:37 2014] <hsaputra>: hehehe sometimes Hadoop YARN is still hard to bring up even though been use it for a while =P [Wed Mar 5 18:49:49 2014] <hsaputra>: especially when new version available [Wed Mar 5 18:59:43 2014] <kanakb>: okay folks, anything else that we should get on the record? [Wed Mar 5 19:01:49 2014] <hsaputra>: I am good for now =) [Wed Mar 5 19:02:19 2014] <kanakb>: okay thanks everyone... we'll still be on IRC for general questions [Wed Mar 5 19:02:24 2014] <kanakb>: ASFBot: meeting end Meeting ended at Wed Mar 5 19:02:24 2014
