I will take a stab at making S4-110-new branch working. Repartitioning is dependent on the application and is hard to implement. Helix architecture does not restrict one from implementing repartitioning. In general, partitioning splitting is hard and a common work around is to over partition and simply move partition around when new machines are added.
thanks, Kishore G On Thu, Jan 9, 2014 at 8:22 AM, Sergio Vavassori <svavass...@conwet.com>wrote: > On 12 December 2013 18:54, kishore g <g.kish...@gmail.com> wrote: > > > > > > https://git-wip-us.apache.org/repos/asf?p=incubator-s4.git;a=blob;f=subprojects/s4-tools/src/main/java/org/apache/s4/tools/helix/CreateTask.java;h=36a45bc5bef5e4c4f56bfb4c740abb315197ca21;hb=59bcb94cc727cb1600074e6da81330953aeaa566 > > > > When you add the task, you can use pe type s the name of the task and > > specify the streams it is interested in. > > > > Give it a try :-) > > > > Hello, > > First of all I'm sorry for the delay in reply but Christmas vacation didn't > help. > > I tried to use the CreateTask but it raised a ZK Exception about S4 cluster > not present (even when it was created manually). > > Is it correct that CreateTask creates all resources under > S4HelixConstants.HELIX_CLUSTER_NAME and a property "GROUP" with the > clusterName instead of using the clusterName as reference for > ConfigScopeBuilder? > > Then I found this repository: https://github.com/kishoreg/incubator-s4 and > even if it's older and needed some patches to make it work, I was able to > run it. Nevertheless the application didn't respond as supposed. (echo > "Bob" | nc localhost 16000 didn't return). > In details I did the following: > > ./gradlew install > ./gradlew s4-tools:installApp > > ./s4 zkServer & > ./s4 newCluster -c=cluster1 -nbNodes=2 -flp=12000 > ./s4 newCluster -c=cluster2 -nbNodes=1 -flp=16000 > ./s4 createTask -zk localhost:2181 -c cluster1 -id names -t consumer > -p 4 -r 1 -s names > ./s4 createTask -zk localhost:2181 -c cluster2 -id names -t producer > -p 1 -r 0 -s names > > ./s4 newApp myApp -parentDir=/tmp > cd /tmp/myApp > ./s4 s4r -a=hello.HelloApp -b=/tmp/myApp/build.gradle myApp > > ./s4 deployApp -c=cluster1 -s4r=/tmp/myApp/build/libs/myApp.s4r > -appName=myApp -zk=localhost:2181 & > ./s4 adapter -c=cluster2 -zk=localhost:2181 > -p=s4.adapter.output.stream=names,s4.instance.name=localhost_16000 & > > (I had to add the "-p" parameters because guava was requiring a value for > that) > > and these are their statuses > > > ./s4 status -c=cluster1 > ----------------------- > > calling referenced s4 script : /home/sergio/upm/repos/s4-helix/s4 > 1 [main] INFO org.apache.helix.manager.zk.ZKHelixManager - Create a > zk-based cluster manager. clusterName:cluster1, instanceName:ADMIN, > type:ADMINISTRATOR, zkSvr:localhost:2181 > > 57 [main] INFO org.apache.helix.messaging.handling.HelixTaskExecutor > - Adding msg factory for type TASK_REPLY threadpool size 40 > 59 [main] INFO org.apache.helix.manager.zk.ZKHelixManager - > ClusterManager.connect() > > 135 [main] INFO org.apache.helix.manager.zk.ZkStateChangeListener - > KeeperState: SyncConnected, zookeeper:State:CONNECTED Timeout:30000 > sessionid:0x14376f2de0a002b local:/127.0.0.1:41285 > remoteserver:localhost/127.0.0.1:2181 lastZxid:0 xid:1 sent:1 recv:1 > queuedpkts:0 pendingresp:0 queuedevents:0 > > 136 [main] INFO org.apache.helix.manager.zk.ZKHelixManager - Handling > new session, session id:14376f2de0a002b, instance:ADMIN, instanceTye: > ADMINISTRATOR, cluster: cluster1 > 136 [main] INFO org.apache.helix.manager.zk.ZKHelixManager - > State:CONNECTED Timeout:30000 sessionid:0x14376f2de0a002b > local:/127.0.0.1:41285 remoteserver:localhost/127.0.0.1:2181 > lastZxid:0 xid:1 sent:1 recv:1 queuedpkts:0 pendingresp:0 > queuedevents:0 > > Cluster Status > > ---------------------------------------------------------------------------------------------------------------------------------- > > Nodes > > Cluster Name Nodes Active > > -------------------------------------------------------------------------------- > Node id > Host Port Active > > > ---------------------------------------------------------------------------------------------------------------------------------- > cluster1 2 2 localhost_12000 > localhost 12000 Y > > localhost_12001 > localhost 12001 Y > App Status > > ---------------------------------------------------------------------------------------------------------------------------------- > > Name Cluster > URI > > ---------------------------------------------------------------------------------------------------------------------------------- > > myApp cluster1 > file:/tmp/myApp/build/libs/myApp.s4r > > ---------------------------------------------------------------------------------------------------------------------------------- > > Node id DEPLOYED > > ---------------------------------------------------------------------------------------------------------------------------------- > localhost_12000 Y > > localhost_12001 Y > > ---------------------------------------------------------------------------------------------------------------------------------- > App Status > > ---------------------------------------------------------------------------------------------------------------------------------- > > Name Cluster > URI > > ---------------------------------------------------------------------------------------------------------------------------------- > > adapter cluster1 > file:/tmp/myApp/build/libs/myApp.s4r > > ---------------------------------------------------------------------------------------------------------------------------------- > > Node id DEPLOYED > > ---------------------------------------------------------------------------------------------------------------------------------- > localhost_12000 Y > > localhost_12001 Y > > ---------------------------------------------------------------------------------------------------------------------------------- > Task Status > > ---------------------------------------------------------------------------------------------------------------------------------- > > Task Id Cluster > Description > > ---------------------------------------------------------------------------------------------------------------------------------- > > names cluster1 > names producer > > ---------------------------------------------------------------------------------------------------------------------------------- > > Partition State > > ---------------------------------------------------------------------------------------------------------------------------------- > > names_0 > localhost_12000:STANDBY, localhost_12001:LEADER > names_1 > localhost_12000:LEADER, localhost_12001:STANDBY > > names_2 > localhost_12000:STANDBY, localhost_12001:LEADER > names_3 > localhost_12000:LEADER, localhost_12001:STANDBY > > > ---------------------------------------------------------------------------------------------------------------------------------- > > > > ./s4 status -c=cluster2 > ----------------------- > > calling referenced s4 script : /home/sergio/upm/repos/s4-helix/s4 > 0 [main] INFO org.apache.helix.manager.zk.ZKHelixManager - Create a > zk-based cluster manager. clusterName:cluster2, instanceName:ADMIN, > type:ADMINISTRATOR, zkSvr:localhost:2181 > > 57 [main] INFO org.apache.helix.messaging.handling.HelixTaskExecutor > - Adding msg factory for type TASK_REPLY threadpool size 40 > 60 [main] INFO org.apache.helix.manager.zk.ZKHelixManager - > ClusterManager.connect() > > 133 [main] INFO org.apache.helix.manager.zk.ZkStateChangeListener - > KeeperState: SyncConnected, zookeeper:State:CONNECTED Timeout:30000 > sessionid:0x14376f2de0a0030 local:/127.0.0.1:41291 > remoteserver:localhost/127.0.0.1:2181 lastZxid:0 xid:1 sent:1 recv:1 > queuedpkts:0 pendingresp:0 queuedevents:0 > > 133 [main] INFO org.apache.helix.manager.zk.ZKHelixManager - Handling > new session, session id:14376f2de0a0030, instance:ADMIN, instanceTye: > ADMINISTRATOR, cluster: cluster2 > 133 [main] INFO org.apache.helix.manager.zk.ZKHelixManager - > State:CONNECTED Timeout:30000 sessionid:0x14376f2de0a0030 > local:/127.0.0.1:41291 remoteserver:localhost/127.0.0.1:2181 > lastZxid:0 xid:1 sent:1 recv:1 queuedpkts:0 pendingresp:0 > queuedevents:0 > > Cluster Status > > ---------------------------------------------------------------------------------------------------------------------------------- > > Nodes > > Cluster Name Nodes Active > > -------------------------------------------------------------------------------- > Node id > Host Port Active > > > ---------------------------------------------------------------------------------------------------------------------------------- > cluster2 1 1 localhost_16000 > localhost 16000 Y > > App Status > > ---------------------------------------------------------------------------------------------------------------------------------- > Name Cluster > URI > > > ---------------------------------------------------------------------------------------------------------------------------------- > adapter cluster2 > file:/tmp/myApp/build/libs/myApp.s4r > > > ---------------------------------------------------------------------------------------------------------------------------------- > Node id DEPLOYED > > ---------------------------------------------------------------------------------------------------------------------------------- > > localhost_16000 Y > > ---------------------------------------------------------------------------------------------------------------------------------- > Task Status > > ---------------------------------------------------------------------------------------------------------------------------------- > > Task Id Cluster > Description > > ---------------------------------------------------------------------------------------------------------------------------------- > > names cluster2 > names producer > > ---------------------------------------------------------------------------------------------------------------------------------- > > Partition State > > ---------------------------------------------------------------------------------------------------------------------------------- > > names_0 > localhost_16000:LEADER > > ------------------------------------------------------------------------------------------------------------------------ > > > Am I missing something? > > Is S4-110-new branch (commit 59bcb94cc727cb1600074e6da81330953aeaa566 ) the > correct one to use or is best to use the other git repo? > In the case it's the S4-110-new branch, would you mind to provide a working > command-list for the same basic example? > > I saw there is a request to support repartitioning in Helix, but nobody > replied ( > > http://mail-archives.apache.org/mod_mbox/helix-dev/201304.mbox/%3CJIRA.12640790.1365088831943.107882.1365088877867@arcas%3E > ). > > Is it too much application-dependent or are there any problems with Helix > architecture? > > Best Regards, > Sergio Vavassori >
