S4 clusters are currently statically partitioned. We have been looking at how to enable some elasticity by providing means to (efficiently) repartition the cluster while it's running.
This is the objective of S4-110 (https://issues.apache.org/jira/browse/S4-110) where we leverage Apache Helix for that purpose. It's not that simple though and we don't have a working implementation yet. Any help would be welcome! Regards, Matthieu On Jun 9, 2013, at 08:32 , baojian Zhou wrote: > when we start a newCluster, we must define the number of Tasks, and i notice > that in TaskSetup.class: > > public void setup(String cluster, int tasks, int initialPort) { > try { > zkclient.createPersistent("/s4/streams", true); > } catch (ZkException ignored) { > // ignore if exists > } > > zkclient.createPersistent("/s4/clusters/" + cluster + "/tasks", > true); > zkclient.createPersistent("/s4/clusters/" + cluster + > "/process", true); > zkclient.createPersistent("/s4/clusters/" + cluster + "/app", > true); > for (int i = 0; i < tasks; i++) { > String taskId = "Task-" + i; > ZNRecord record = new ZNRecord(taskId); > record.putSimpleField("taskId", taskId); > record.putSimpleField("port", > String.valueOf(initialPort + i)); > record.putSimpleField("partition", String.valueOf(i)); > record.putSimpleField("cluster", cluster); > zkclient.createPersistent("/s4/clusters/" + cluster + > "/tasks/" > + taskId, record); > } > } > > > so my question is: > if i want to change the nbTasks dynamically and then add the corresponding > S4Nodes , do i just need to write some codes in the TaskSetUp class？ or i > have to change some other codes
