Hi! I'm trying to use Helix as the core of a project I'm starting. I'll start by describing the project in simple terms.
The main goal of this project is to have a way to launch jobs (which will actually be new JVM processes) in a cluster of nodes (physical machines). The idea is to have dummy a dummy Java process (which I call Minion) in each of the nodes. From an external client, I wanted to them launch applications to be run on nodes. The way I was thinking about it was to first decide in which nodes the Application and its replicas would run, then tell the Minion processes running on those nodes to start the job as a separate JVM process (using a message queue approach or something to be decided). How could this map to Helix? At first sight I thought that Minions could be Helix Participants and Applications could be Helix Resources. Upon manually starting Minions on each physical machine, each would register itself as a Helix Participant. In order to start an Application in the cluster, a client would create a new Helix Resource, configure its replication and partitioning schema, and add it to the cluster. Helix would then decide in which Minions (Helix Participants) the Application would run. I'd then lookup the list of Minion (Helix Participants) the resource was assigned to (using Helix's External View maybe), and send a message to those Minions to tell them to start a new JVM process. So far I think this more or less matches Helix's model, right? However, I'm not sure how to model Application processes. I still want them to be part of the cluster, but they don't seem to fit neither the Participant nor the Resource concept... I would like to be aware when they fail. Sorry for the long description, Regards, José
