Summary of IRC Meeting in #apachehelix at Wed Mar 19 17:01:10 2014: Attendees: hsaputra, kanakb, kishoreg
- Preface - task scheduling - administrator, connection, listeners - provisioners, tasks IRC log follows: ## Preface ## [Wed Mar 19 17:01:32 2014] <kanakb>: hi, this is the biweekly morning meeting [Wed Mar 19 17:01:36 2014] <kanakb>: anyone have topics? [Wed Mar 19 17:03:28 2014] <kishoreg>: shud we discuss about schedulers? [Wed Mar 19 17:03:33 2014] <kishoreg>: task schedulers [Wed Mar 19 17:04:00 2014] <kishoreg>: it wud be nice to have agenda for the meeting upfront [Wed Mar 19 17:04:16 2014] <kanakb>: yeah [Wed Mar 19 17:04:21 2014] <kanakb>: ok, ## task scheduling ## [Wed Mar 19 17:05:06 2014] <kanakb>: so as i said in the email, there's really two parts to this [Wed Mar 19 17:05:12 2014] <kanakb>: a framework that can run tasks [Wed Mar 19 17:05:28 2014] <kanakb>: and support for triggering tasks on this framework according to a schedule [Wed Mar 19 17:05:35 2014] <kanakb>: this could mean things like [Wed Mar 19 17:06:04 2014] <kanakb>: "run one in 1 hour", "run every hour starting now", "run every day at [time]" [Wed Mar 19 17:06:34 2014] <kishoreg>: so we will need booking of when was the last time it was run [Wed Mar 19 17:06:50 2014] <kanakb>: or the next time it should run [Wed Mar 19 17:07:06 2014] <kishoreg>: yeah [Wed Mar 19 17:07:28 2014] <kanakb>: i would lean towards the latter since then one-time scheduling and recurring are handled similarly [Wed Mar 19 17:08:27 2014] <kanakb>: so if we have a "next time" semantic, the controller needs to be able to read this [Wed Mar 19 17:08:36 2014] <kanakb>: and then schedule a wakeup at exactly that time [Wed Mar 19 17:08:53 2014] <kanakb>: i.e. timer-based rebalance [Wed Mar 19 17:09:37 2014] <kishoreg>: makes sense [Wed Mar 19 17:09:54 2014] <kanakb>: as for the task framework [Wed Mar 19 17:10:15 2014] <kanakb>: i think we can use what abe/chris did as-is, plus the independent assignment stuff i added to the provisioning branch [Wed Mar 19 17:10:38 2014] <kishoreg>: so scheduling will be a superset of task framework [Wed Mar 19 17:10:59 2014] <kishoreg>: lets define the attributes for a task [Wed Mar 19 17:11:12 2014] <kishoreg>: - one time v/s recurring [Wed Mar 19 17:11:41 2014] <kishoreg>: - nextTime to run [Wed Mar 19 17:12:45 2014] <kanakb>: well from a user perspective, it should be the schedule, and internally we should always compute next time [Wed Mar 19 17:14:22 2014] <kanakb>: or the user could specify the first next time [Wed Mar 19 17:14:27 2014] <kanakb>: which could be "now" [Wed Mar 19 17:14:28 2014] <kishoreg>: rt [Wed Mar 19 17:14:36 2014] <kishoreg>: so at a high level [Wed Mar 19 17:14:45 2014] <kishoreg>: runTask when howoften where [Wed Mar 19 17:14:50 2014] <kanakb>: yeah [Wed Mar 19 17:15:06 2014] <kishoreg>: when is now or some time in future or none [Wed Mar 19 17:15:11 2014] <kishoreg>: howoften can be once or a cron expression [Wed Mar 19 17:15:15 2014] <kanakb>: so if none, we expect a specific time of day? [Wed Mar 19 17:17:34 2014] <kishoreg>: its actually better to split into adhoc and recurring [Wed Mar 19 17:17:37 2014] <kishoreg>: -adhoc when where [Wed Mar 19 17:17:40 2014] <kishoreg>: -schedule howoften where [Wed Mar 19 17:17:43 2014] <kanakb>: makes sense [Wed Mar 19 17:18:17 2014] <kishoreg>: we need an expression for where [Wed Mar 19 17:19:18 2014] <kanakb>: this is already somewhat supported by the task framework [Wed Mar 19 17:19:19 2014] <kishoreg>: something like node resource state count [Wed Mar 19 17:19:21 2014] <kanakb>: yeah [Wed Mar 19 17:19:24 2014] <kanakb>: so we have resource, partitions, state right now [Wed Mar 19 17:19:26 2014] <kanakb>: we can add count and node (or banned nodes) [Wed Mar 19 17:20:03 2014] <kishoreg>: ok [Wed Mar 19 17:23:42 2014] <kanakb>: ok cool [Wed Mar 19 17:23:46 2014] <kanakb>: so ## administrator, connection, listeners ## [Wed Mar 19 17:24:07 2014] <kanakb>: last night we talked about implementing listeners [Wed Mar 19 17:24:39 2014] <kanakb>: i think now we have some idea of the model, most of the attributes, and how to listen on them [Wed Mar 19 17:24:55 2014] <kanakb>: probably this weekend i will try to sketch out an administrator and connection interface based on this [Wed Mar 19 17:25:11 2014] <kanakb>: (for those unaware of the discussion, the log is at wilderness.apache.org) [Wed Mar 19 17:26:17 2014] <kishoreg>: i dint know about the log [Wed Mar 19 17:27:30 2014] <kanakb>: i meant http://wilderness.apache.org/archives/ [Wed Mar 19 17:27:47 2014] <kanakb>: but yes all past meeting logs are available there [Wed Mar 19 17:28:07 2014] <kishoreg>: thats cool. [Wed Mar 19 17:28:34 2014] <kishoreg>: regarding listeners, is hud have the initial set of API for listener and events today [Wed Mar 19 17:28:48 2014] <kanakb>: alright [Wed Mar 19 17:28:55 2014] <kishoreg>: thats all i have ## provisioners, tasks ## [Wed Mar 19 17:29:45 2014] <kanakb>: i don't have much to add here, just that tasks inside containers work a little differently from how the task framework uses them [Wed Mar 19 17:29:54 2014] <hsaputra>: is this related integration with YARN? [Wed Mar 19 17:29:58 2014] <kanakb>: yep [Wed Mar 19 17:30:15 2014] <kanakb>: so the idea is that you can specify tasks to run on the cluster [Wed Mar 19 17:30:22 2014] <kanakb>: where a queue of tasks is a resource [Wed Mar 19 17:30:27 2014] <hsaputra>: I remember Kishore is updating his blog to have different of flow [Wed Mar 19 17:30:27 2014] <kanakb>: and a single task is a partition [Wed Mar 19 17:30:56 2014] <kanakb>: yeah i believe kishoreg was in the process of changing it to make tasks the key problem this can really get right [Wed Mar 19 17:31:10 2014] <hsaputra>: single task is partition? hmm could you elaborate more? [Wed Mar 19 17:31:20 2014] <kanakb>: that's really an implementation detail [Wed Mar 19 17:31:27 2014] <hsaputra>: currently Helix maintain the set of partitions of the resources [Wed Mar 19 17:31:34 2014] <kanakb>: right [Wed Mar 19 17:31:46 2014] <kanakb>: so because helix sends callbacks for every partition state change [Wed Mar 19 17:31:52 2014] <kanakb>: we can leverage this to say when to run a task [Wed Mar 19 17:32:01 2014] <hsaputra>: but with different provision engine (YARN, Mesos) it could be "split" into different set of partitions [Wed Mar 19 17:32:08 2014] <hsaputra>: or maybe I am way off? [Wed Mar 19 17:32:24 2014] <kanakb>: so there's containers (or VMs) and then there's tasks [Wed Mar 19 17:32:30 2014] <kanakb>: we use yarn/mesos to spawn containers [Wed Mar 19 17:32:34 2014] <kanakb>: and then we schedule tasks on those containers [Wed Mar 19 17:32:47 2014] <kanakb>: the advantage is that people don't need to estimate cpu/memory for each task [Wed Mar 19 17:32:56 2014] <kanakb>: because people are typically pretty bad at this [Wed Mar 19 17:33:36 2014] <kishoreg>: hsaputra: i dint get your question [Wed Mar 19 17:33:59 2014] <hsaputra>: Helix maintain set of partitions for the resources ... [Wed Mar 19 17:34:35 2014] <kishoreg>: yes [Wed Mar 19 17:35:19 2014] <hsaputra>: ⦠so Helix provision "master" in each of the resource manager (app master in YARN) need to be able to provision the resource manually to the worker (node manager in YARN) right? [Wed Mar 19 17:36:02 2014] <hsaputra>: ahhh sorry I think I am thinking in mapreduce 1 mode in YARN where input split is being used [Wed Mar 19 17:36:14 2014] <hsaputra>: never mind, sorry guys =( [Wed Mar 19 17:36:44 2014] <kanakb>: no worries [Wed Mar 19 17:36:50 2014] <kanakb>: is the general idea clear though? [Wed Mar 19 17:36:54 2014] <kanakb>: tasks inside containers? [Wed Mar 19 17:36:59 2014] <hsaputra>: getting confuse =( [Wed Mar 19 17:37:02 2014] <hsaputra>: yes +1 [Wed Mar 19 17:37:30 2014] <kanakb>: the other big thing with this is that when containers go down [Wed Mar 19 17:37:34 2014] <kanakb>: we can spawn new ones [Wed Mar 19 17:37:45 2014] <kanakb>: in addition to what we can already do, which is reassign stuff to other containers [Wed Mar 19 17:38:12 2014] <kishoreg>: i need to sign off, will sign in again after 30 minutes [Wed Mar 19 17:38:56 2014] <hsaputra>: there has been some interest in Helix when we are able to provision nodes [Wed Mar 19 17:39:04 2014] <hsaputra>: especially with Mesos and YARN [Wed Mar 19 17:39:21 2014] <kanakb>: which one do would you be more interested in? mesos or yarn? [Wed Mar 19 17:39:32 2014] <hsaputra>: Me will be YARN [Wed Mar 19 17:39:42 2014] <hsaputra>: I am willing to help on it for sure [Wed Mar 19 17:40:32 2014] <hsaputra>: but I know there is another org interested if Helix do it right so it play well with Mesos [Wed Mar 19 17:41:11 2014] <kanakb>: yeah for us, yarn is probably higher priority [Wed Mar 19 17:41:20 2014] <kanakb>: i'm the semi-owner for the mesos stuff though [Wed Mar 19 17:41:32 2014] <kanakb>: i just need to find time to get something going with it [Wed Mar 19 17:45:22 2014] <hsaputra>: =) [Wed Mar 19 17:51:57 2014] <hsaputra>: kanakb is there JIRA filed to track dev for provisioner support? [Wed Mar 19 17:52:21 2014] <kanakb>: not really... it's basically just a bunch of commits on the helix-provisioning branch [Wed Mar 19 17:52:35 2014] <hsaputra>: ok cool [Wed Mar 19 17:52:49 2014] <hsaputra>: I will follow up in that dev branch then [Wed Mar 19 17:53:27 2014] <kanakb>: definitely take a look at the helloworld yaml recipe [Wed Mar 19 17:53:32 2014] <hsaputra>: will do [Wed Mar 19 17:53:42 2014] <kanakb>: kishore sent an email a while back describe how to use it [Wed Mar 19 17:53:53 2014] <hsaputra>: yes [Wed Mar 19 17:55:06 2014] <hsaputra>: just need to fork the time =( [Wed Mar 19 17:55:55 2014] <hsaputra>: that is all from me. Thanks for the time to hangout in the chat guys [Wed Mar 19 17:56:02 2014] <kanakb>: alright thanks for coming by [Wed Mar 19 17:56:22 2014] <kanakb>: anyone else have any discussion topics? [Wed Mar 19 17:58:25 2014] <kanakb>: okay thanks everyone! [Wed Mar 19 17:58:31 2014] <kanakb>: ASFBot: meeting end Meeting ended at Wed Mar 19 17:58:31 2014
