Hi Jose, Your minions are very similar to the Helix Agent. But there is slight difference, Helix agents were used to launch non java processes but in your case you plan to launch jvm applications on demand.
Your description makes sense. Applications can also modeled as resources. Here is how you can model it. There are two types of Participants : Minions and Application Processes. STARTING MINION At first, you start one Minion on each machine. There will be one resource created to manage the minion. Lets call it MINION_RESOURCE. Each minion that is started and joins the cluster and tags itself as MINION. We will see why we tag it as MINION in the next section. LAUNCHING APPLICATIONS There are two ways to model each Application Process, create one uber resource for all Processes or one per Application Group. For now, I will go with uber resource. You create a new uber resource called JVM_APPS and tag it as MINION. The idea is all sub parts of this resource will be assigned to minions. When the minions get a state transition per application, they simply launch that application. Lets say you want to start 5 application processes/jvms name them as MyApp_0, .. MyApp_1. You can add them to the JVM_APPS resource. The assignment mapping of MyApp to a specific MINION can be automatic or custom by using the appropriate helix rebalancer (See AUTO, SEMI AUTO, CUSTOM idealstate modes). MODELING EACH APPLICATION When each minion launches a JVM, the JVM process registers itself in the cluster and tags it self as MyApp. After this you can create a MyApp resource (tag it has MyApp) and configure number of partitions, replicas etc. This partitions/replicas of this resource will only be assigned to these nodes. Hope that make sense. This will be a great recipe :-) thanks, Kishore G On Fri, Apr 24, 2015 at 3:14 AM, Half Warp <[email protected]> wrote: > Hi! > > I'm trying to use Helix as the core of a project I'm starting. I'll start > by describing the project > in simple terms. > > The main goal of this project is to have a way to launch jobs (which will > actually be new JVM processes) > in a cluster of nodes (physical machines). The idea is to have dummy a > dummy Java process (which I call Minion) in each of > the nodes. From an external client, I wanted to them launch applications > to be run on nodes. The way I was thinking > about it was to first decide in which nodes the Application and its > replicas would run, then tell the Minion processes running > on those nodes to start the job as a separate JVM process (using a message > queue approach or something to be decided). > > How could this map to Helix? At first sight I thought that Minions could > be Helix Participants and Applications could be Helix Resources. > Upon manually starting Minions on each physical machine, each would > register itself as a Helix Participant. In order to start an > Application in the cluster, a client would create a new Helix Resource, > configure its replication and partitioning schema, and add it to the > cluster. Helix would then decide in which Minions (Helix Participants) the > Application would run. I'd then lookup the list of Minion (Helix > Participants) the resource was assigned to (using Helix's External View > maybe), and send a message to those Minions to tell them > to start a new JVM process. > > So far I think this more or less matches Helix's model, right? > > However, I'm not sure how to model Application processes. I still want > them to be part of the cluster, but they don't seem to fit neither the > Participant nor the Resource concept... I would like to be aware when they > fail. > > > Sorry for the long description, > > > Regards, > > José >
