On 12 December 2013 18:54, kishore g <g.kish...@gmail.com> wrote: > > https://git-wip-us.apache.org/repos/asf?p=incubator-s4.git;a=blob;f=subprojects/s4-tools/src/main/java/org/apache/s4/tools/helix/CreateTask.java;h=36a45bc5bef5e4c4f56bfb4c740abb315197ca21;hb=59bcb94cc727cb1600074e6da81330953aeaa566 > > When you add the task, you can use pe type s the name of the task and > specify the streams it is interested in. > > Give it a try :-) >
Hello, First of all I'm sorry for the delay in reply but Christmas vacation didn't help. I tried to use the CreateTask but it raised a ZK Exception about S4 cluster not present (even when it was created manually). Is it correct that CreateTask creates all resources under S4HelixConstants.HELIX_CLUSTER_NAME and a property "GROUP" with the clusterName instead of using the clusterName as reference for ConfigScopeBuilder? Then I found this repository: https://github.com/kishoreg/incubator-s4 and even if it's older and needed some patches to make it work, I was able to run it. Nevertheless the application didn't respond as supposed. (echo "Bob" | nc localhost 16000 didn't return). In details I did the following: ./gradlew install ./gradlew s4-tools:installApp ./s4 zkServer & ./s4 newCluster -c=cluster1 -nbNodes=2 -flp=12000 ./s4 newCluster -c=cluster2 -nbNodes=1 -flp=16000 ./s4 createTask -zk localhost:2181 -c cluster1 -id names -t consumer -p 4 -r 1 -s names ./s4 createTask -zk localhost:2181 -c cluster2 -id names -t producer -p 1 -r 0 -s names ./s4 newApp myApp -parentDir=/tmp cd /tmp/myApp ./s4 s4r -a=hello.HelloApp -b=/tmp/myApp/build.gradle myApp ./s4 deployApp -c=cluster1 -s4r=/tmp/myApp/build/libs/myApp.s4r -appName=myApp -zk=localhost:2181 & ./s4 adapter -c=cluster2 -zk=localhost:2181 -p=s4.adapter.output.stream=names,s4.instance.name=localhost_16000 & (I had to add the "-p" parameters because guava was requiring a value for that) and these are their statuses ./s4 status -c=cluster1 ----------------------- calling referenced s4 script : /home/sergio/upm/repos/s4-helix/s4 1 [main] INFO org.apache.helix.manager.zk.ZKHelixManager - Create a zk-based cluster manager. clusterName:cluster1, instanceName:ADMIN, type:ADMINISTRATOR, zkSvr:localhost:2181 57 [main] INFO org.apache.helix.messaging.handling.HelixTaskExecutor - Adding msg factory for type TASK_REPLY threadpool size 40 59 [main] INFO org.apache.helix.manager.zk.ZKHelixManager - ClusterManager.connect() 135 [main] INFO org.apache.helix.manager.zk.ZkStateChangeListener - KeeperState: SyncConnected, zookeeper:State:CONNECTED Timeout:30000 sessionid:0x14376f2de0a002b local:/127.0.0.1:41285 remoteserver:localhost/127.0.0.1:2181 lastZxid:0 xid:1 sent:1 recv:1 queuedpkts:0 pendingresp:0 queuedevents:0 136 [main] INFO org.apache.helix.manager.zk.ZKHelixManager - Handling new session, session id:14376f2de0a002b, instance:ADMIN, instanceTye: ADMINISTRATOR, cluster: cluster1 136 [main] INFO org.apache.helix.manager.zk.ZKHelixManager - State:CONNECTED Timeout:30000 sessionid:0x14376f2de0a002b local:/127.0.0.1:41285 remoteserver:localhost/127.0.0.1:2181 lastZxid:0 xid:1 sent:1 recv:1 queuedpkts:0 pendingresp:0 queuedevents:0 Cluster Status ---------------------------------------------------------------------------------------------------------------------------------- Nodes Cluster Name Nodes Active -------------------------------------------------------------------------------- Node id Host Port Active ---------------------------------------------------------------------------------------------------------------------------------- cluster1 2 2 localhost_12000 localhost 12000 Y localhost_12001 localhost 12001 Y App Status ---------------------------------------------------------------------------------------------------------------------------------- Name Cluster URI ---------------------------------------------------------------------------------------------------------------------------------- myApp cluster1 file:/tmp/myApp/build/libs/myApp.s4r ---------------------------------------------------------------------------------------------------------------------------------- Node id DEPLOYED ---------------------------------------------------------------------------------------------------------------------------------- localhost_12000 Y localhost_12001 Y ---------------------------------------------------------------------------------------------------------------------------------- App Status ---------------------------------------------------------------------------------------------------------------------------------- Name Cluster URI ---------------------------------------------------------------------------------------------------------------------------------- adapter cluster1 file:/tmp/myApp/build/libs/myApp.s4r ---------------------------------------------------------------------------------------------------------------------------------- Node id DEPLOYED ---------------------------------------------------------------------------------------------------------------------------------- localhost_12000 Y localhost_12001 Y ---------------------------------------------------------------------------------------------------------------------------------- Task Status ---------------------------------------------------------------------------------------------------------------------------------- Task Id Cluster Description ---------------------------------------------------------------------------------------------------------------------------------- names cluster1 names producer ---------------------------------------------------------------------------------------------------------------------------------- Partition State ---------------------------------------------------------------------------------------------------------------------------------- names_0 localhost_12000:STANDBY, localhost_12001:LEADER names_1 localhost_12000:LEADER, localhost_12001:STANDBY names_2 localhost_12000:STANDBY, localhost_12001:LEADER names_3 localhost_12000:LEADER, localhost_12001:STANDBY ---------------------------------------------------------------------------------------------------------------------------------- ./s4 status -c=cluster2 ----------------------- calling referenced s4 script : /home/sergio/upm/repos/s4-helix/s4 0 [main] INFO org.apache.helix.manager.zk.ZKHelixManager - Create a zk-based cluster manager. clusterName:cluster2, instanceName:ADMIN, type:ADMINISTRATOR, zkSvr:localhost:2181 57 [main] INFO org.apache.helix.messaging.handling.HelixTaskExecutor - Adding msg factory for type TASK_REPLY threadpool size 40 60 [main] INFO org.apache.helix.manager.zk.ZKHelixManager - ClusterManager.connect() 133 [main] INFO org.apache.helix.manager.zk.ZkStateChangeListener - KeeperState: SyncConnected, zookeeper:State:CONNECTED Timeout:30000 sessionid:0x14376f2de0a0030 local:/127.0.0.1:41291 remoteserver:localhost/127.0.0.1:2181 lastZxid:0 xid:1 sent:1 recv:1 queuedpkts:0 pendingresp:0 queuedevents:0 133 [main] INFO org.apache.helix.manager.zk.ZKHelixManager - Handling new session, session id:14376f2de0a0030, instance:ADMIN, instanceTye: ADMINISTRATOR, cluster: cluster2 133 [main] INFO org.apache.helix.manager.zk.ZKHelixManager - State:CONNECTED Timeout:30000 sessionid:0x14376f2de0a0030 local:/127.0.0.1:41291 remoteserver:localhost/127.0.0.1:2181 lastZxid:0 xid:1 sent:1 recv:1 queuedpkts:0 pendingresp:0 queuedevents:0 Cluster Status ---------------------------------------------------------------------------------------------------------------------------------- Nodes Cluster Name Nodes Active -------------------------------------------------------------------------------- Node id Host Port Active ---------------------------------------------------------------------------------------------------------------------------------- cluster2 1 1 localhost_16000 localhost 16000 Y App Status ---------------------------------------------------------------------------------------------------------------------------------- Name Cluster URI ---------------------------------------------------------------------------------------------------------------------------------- adapter cluster2 file:/tmp/myApp/build/libs/myApp.s4r ---------------------------------------------------------------------------------------------------------------------------------- Node id DEPLOYED ---------------------------------------------------------------------------------------------------------------------------------- localhost_16000 Y ---------------------------------------------------------------------------------------------------------------------------------- Task Status ---------------------------------------------------------------------------------------------------------------------------------- Task Id Cluster Description ---------------------------------------------------------------------------------------------------------------------------------- names cluster2 names producer ---------------------------------------------------------------------------------------------------------------------------------- Partition State ---------------------------------------------------------------------------------------------------------------------------------- names_0 localhost_16000:LEADER ------------------------------------------------------------------------------------------------------------------------ Am I missing something? Is S4-110-new branch (commit 59bcb94cc727cb1600074e6da81330953aeaa566 ) the correct one to use or is best to use the other git repo? In the case it's the S4-110-new branch, would you mind to provide a working command-list for the same basic example? I saw there is a request to support repartitioning in Helix, but nobody replied ( http://mail-archives.apache.org/mod_mbox/helix-dev/201304.mbox/%3CJIRA.12640790.1365088831943.107882.1365088877867@arcas%3E). Is it too much application-dependent or are there any problems with Helix architecture? Best Regards, Sergio Vavassori
