Re: Information about Apache S4 and Helix
On 23 January 2014 23:54, kishore g wrote: > Hi Sergio, > > Did this work ? > > thanks, > Kishore G > Yes, it works! Best Regards, Sergio
Re: Information about Apache S4 and Helix
Hi Sergio, Did this work ? thanks, Kishore G On Wed, Jan 15, 2014 at 9:49 AM, kishore g wrote: > Hi Sergio, > > The first two commands must use newCluster instead of addNodes. > > Change > ./s4 addNodes -helix -c=cluster1 -nbNodes=2 -flp=12000 to > ./s4 newCluster -helix -c=cluster1 -nbNodes=2 -flp=12000 > > > > > > On Wed, Jan 15, 2014 at 12:06 AM, Sergio Vavassori > wrote: > >> On 14 January 2014 17:28, kishore g wrote: >> >> > It did work for me, there were some changes we did to keep it backward >> > compatible with existing S4. >> > >> > Here are the steps >> > BUILD >> > ./gradlew eclipse >> > ./gradlew clean install -DskipTests >> > ./gradlew s4-tools:installApp >> > >> > SETUP CLUSTER >> > >> > ./s4 zkServer -clean >> > & >> > //start zookeeper >> > ./s4 addNodes -helix -c=cluster1 -nbNodes=2 -flp=12000 >> > >> >> Hello, >> >> I tried but I still got the same exception: >> --- >> $ ./s4 addNodes -helix -c=cluster1 -nbNodes=2 -flp=12000 >> 08:55:16.765 [main] INFO org.apache.s4.tools.helix.AddNodes - Adding new >> nodes [2] to cluster [cluster1] node(s) >> 0 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster >> setup, >> missing znode path: /S4/IDEALSTATES >> 2 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster >> setup, >> missing znode path: /S4/CONFIGS/CLUSTER/S4 >> 3 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster >> setup, >> missing znode path: /S4/CONFIGS/PARTICIPANT >> 5 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster >> setup, >> missing znode path: /S4/CONFIGS/RESOURCE >> 6 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster >> setup, >> missing znode path: /S4/PROPERTYSTORE >> 7 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster >> setup, >> missing znode path: /S4/LIVEINSTANCES >> 9 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster >> setup, >> missing znode path: /S4/INSTANCES >> 10 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster >> setup, missing znode path: /S4/EXTERNALVIEW >> 11 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster >> setup, missing znode path: /S4/CONTROLLER >> 12 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster >> setup, missing znode path: /S4/STATEMODELDEFS >> 13 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster >> setup, missing znode path: /S4/CONTROLLER/MESSAGES >> 15 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster >> setup, missing znode path: /S4/CONTROLLER/ERRORS >> 15 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster >> setup, missing znode path: /S4/CONTROLLER/STATUSUPDATES >> 16 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster >> setup, missing znode path: /S4/CONTROLLER/HISTORY >> 08:55:16.884 [main] ERROR org.apache.s4.tools.helix.AddNodes - Cannot >> initialize zookeeper with specified configuration >> org.apache.helix.HelixException: cluster S4 is not setup yet >> at >> >> org.apache.helix.manager.zk.ZKHelixAdmin.addInstance(ZKHelixAdmin.java:105) >> ~[helix-core-0.6.0-incubating.jar:0.6.0-incubating] >> at org.apache.s4.tools.helix.AddNodes.main(AddNodes.java:59) >> ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating] >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> ~[na:1.7.0_21] >> at >> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> ~[na:1.7.0_21] >> at >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> ~[na:1.7.0_21] >> at java.lang.reflect.Method.invoke(Method.java:601) ~[na:1.7.0_21] >> at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:95) >> [s4-tools-0.5.0-incubating.jar:0.5.0-incubating] >> at org.apache.s4.tools.Tools.main(Tools.java:136) >> [s4-tools-0.5.0-incubating.jar:0.5.0-incubating] >> --- >> >> Actually I don't see anything new into S4-100-new log: >> >> $ git log -n 1 --date=iso >> commit 59bcb94cc727cb1600074e6da81330953aeaa566 >> Author: Daniel Gómez Ferro >> Date: 2013-02-22 18:55:20 +0100 >> >> S4 nodes will pickup the first available instance name, no need to >> specify it >> >> Am I using the correct repo? >> >> $ git remote -v >> originhttps://git-wip-us.apache.org/repos/asf/incubator-s4.git(fetch) >> originhttps://git-wip-us.apache.org/repos/asf/incubator-s4.git (push) >> >> >> Best regards, >> Sergio >> > >
Re: Information about Apache S4 and Helix
Hi Sergio, The first two commands must use newCluster instead of addNodes. Change ./s4 addNodes -helix -c=cluster1 -nbNodes=2 -flp=12000 to ./s4 newCluster -helix -c=cluster1 -nbNodes=2 -flp=12000 On Wed, Jan 15, 2014 at 12:06 AM, Sergio Vavassori wrote: > On 14 January 2014 17:28, kishore g wrote: > > > It did work for me, there were some changes we did to keep it backward > > compatible with existing S4. > > > > Here are the steps > > BUILD > > ./gradlew eclipse > > ./gradlew clean install -DskipTests > > ./gradlew s4-tools:installApp > > > > SETUP CLUSTER > > > > ./s4 zkServer -clean > > & > > //start zookeeper > > ./s4 addNodes -helix -c=cluster1 -nbNodes=2 -flp=12000 > > > > Hello, > > I tried but I still got the same exception: > --- > $ ./s4 addNodes -helix -c=cluster1 -nbNodes=2 -flp=12000 > 08:55:16.765 [main] INFO org.apache.s4.tools.helix.AddNodes - Adding new > nodes [2] to cluster [cluster1] node(s) > 0 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster setup, > missing znode path: /S4/IDEALSTATES > 2 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster setup, > missing znode path: /S4/CONFIGS/CLUSTER/S4 > 3 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster setup, > missing znode path: /S4/CONFIGS/PARTICIPANT > 5 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster setup, > missing znode path: /S4/CONFIGS/RESOURCE > 6 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster setup, > missing znode path: /S4/PROPERTYSTORE > 7 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster setup, > missing znode path: /S4/LIVEINSTANCES > 9 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster setup, > missing znode path: /S4/INSTANCES > 10 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster > setup, missing znode path: /S4/EXTERNALVIEW > 11 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster > setup, missing znode path: /S4/CONTROLLER > 12 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster > setup, missing znode path: /S4/STATEMODELDEFS > 13 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster > setup, missing znode path: /S4/CONTROLLER/MESSAGES > 15 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster > setup, missing znode path: /S4/CONTROLLER/ERRORS > 15 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster > setup, missing znode path: /S4/CONTROLLER/STATUSUPDATES > 16 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster > setup, missing znode path: /S4/CONTROLLER/HISTORY > 08:55:16.884 [main] ERROR org.apache.s4.tools.helix.AddNodes - Cannot > initialize zookeeper with specified configuration > org.apache.helix.HelixException: cluster S4 is not setup yet > at > org.apache.helix.manager.zk.ZKHelixAdmin.addInstance(ZKHelixAdmin.java:105) > ~[helix-core-0.6.0-incubating.jar:0.6.0-incubating] > at org.apache.s4.tools.helix.AddNodes.main(AddNodes.java:59) > ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > ~[na:1.7.0_21] > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > ~[na:1.7.0_21] > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[na:1.7.0_21] > at java.lang.reflect.Method.invoke(Method.java:601) ~[na:1.7.0_21] > at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:95) > [s4-tools-0.5.0-incubating.jar:0.5.0-incubating] > at org.apache.s4.tools.Tools.main(Tools.java:136) > [s4-tools-0.5.0-incubating.jar:0.5.0-incubating] > --- > > Actually I don't see anything new into S4-100-new log: > > $ git log -n 1 --date=iso > commit 59bcb94cc727cb1600074e6da81330953aeaa566 > Author: Daniel Gómez Ferro > Date: 2013-02-22 18:55:20 +0100 > > S4 nodes will pickup the first available instance name, no need to > specify it > > Am I using the correct repo? > > $ git remote -v > originhttps://git-wip-us.apache.org/repos/asf/incubator-s4.git (fetch) > originhttps://git-wip-us.apache.org/repos/asf/incubator-s4.git (push) > > > Best regards, > Sergio >
Re: Information about Apache S4 and Helix
On 14 January 2014 17:28, kishore g wrote: > It did work for me, there were some changes we did to keep it backward > compatible with existing S4. > > Here are the steps > BUILD > ./gradlew eclipse > ./gradlew clean install -DskipTests > ./gradlew s4-tools:installApp > > SETUP CLUSTER > > ./s4 zkServer -clean > & > //start zookeeper > ./s4 addNodes -helix -c=cluster1 -nbNodes=2 -flp=12000 > Hello, I tried but I still got the same exception: --- $ ./s4 addNodes -helix -c=cluster1 -nbNodes=2 -flp=12000 08:55:16.765 [main] INFO org.apache.s4.tools.helix.AddNodes - Adding new nodes [2] to cluster [cluster1] node(s) 0 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster setup, missing znode path: /S4/IDEALSTATES 2 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster setup, missing znode path: /S4/CONFIGS/CLUSTER/S4 3 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster setup, missing znode path: /S4/CONFIGS/PARTICIPANT 5 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster setup, missing znode path: /S4/CONFIGS/RESOURCE 6 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster setup, missing znode path: /S4/PROPERTYSTORE 7 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster setup, missing znode path: /S4/LIVEINSTANCES 9 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster setup, missing znode path: /S4/INSTANCES 10 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster setup, missing znode path: /S4/EXTERNALVIEW 11 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster setup, missing znode path: /S4/CONTROLLER 12 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster setup, missing znode path: /S4/STATEMODELDEFS 13 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster setup, missing znode path: /S4/CONTROLLER/MESSAGES 15 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster setup, missing znode path: /S4/CONTROLLER/ERRORS 15 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster setup, missing znode path: /S4/CONTROLLER/STATUSUPDATES 16 [main] ERROR org.apache.helix.manager.zk.ZKUtil - Invalid cluster setup, missing znode path: /S4/CONTROLLER/HISTORY 08:55:16.884 [main] ERROR org.apache.s4.tools.helix.AddNodes - Cannot initialize zookeeper with specified configuration org.apache.helix.HelixException: cluster S4 is not setup yet at org.apache.helix.manager.zk.ZKHelixAdmin.addInstance(ZKHelixAdmin.java:105) ~[helix-core-0.6.0-incubating.jar:0.6.0-incubating] at org.apache.s4.tools.helix.AddNodes.main(AddNodes.java:59) ~[s4-tools-0.5.0-incubating.jar:0.5.0-incubating] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_21] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_21] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_21] at java.lang.reflect.Method.invoke(Method.java:601) ~[na:1.7.0_21] at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:95) [s4-tools-0.5.0-incubating.jar:0.5.0-incubating] at org.apache.s4.tools.Tools.main(Tools.java:136) [s4-tools-0.5.0-incubating.jar:0.5.0-incubating] --- Actually I don't see anything new into S4-100-new log: $ git log -n 1 --date=iso commit 59bcb94cc727cb1600074e6da81330953aeaa566 Author: Daniel Gómez Ferro Date: 2013-02-22 18:55:20 +0100 S4 nodes will pickup the first available instance name, no need to specify it Am I using the correct repo? $ git remote -v originhttps://git-wip-us.apache.org/repos/asf/incubator-s4.git (fetch) originhttps://git-wip-us.apache.org/repos/asf/incubator-s4.git (push) Best regards, Sergio
Re: Information about Apache S4 and Helix
Hi Sergio, It did work for me, there were some changes we did to keep it backward compatible with existing S4. Here are the steps BUILD ./gradlew eclipse ./gradlew clean install -DskipTests ./gradlew s4-tools:installApp SETUP CLUSTER ./s4 zkServer -clean & //start zookeeper ./s4 addNodes -helix -c=cluster1 -nbNodes=2 -flp=12000 //create cluster for app with 2 nodes ./s4 addNodes -helix -c=cluster2 -nbNodes=1 -flp=13000 //create cluster for adapter with 1 node ./s4 createTask -helix -c=cluster1 -id=AggregatedTopicSeen -p=6 -r=1 -s=AggregatedTopicSeen //create the task, this will partition the stream and not PE ./s4 createTask -helix -c=cluster1 -id=RawStatus -p=6 -r=1 -s=RawStatus //create the task, this will partition the stream and not PE ./s4 createTask -helix -c=cluster1 -id=TopicSeen -p=6 -r=1 -s=TopicSeen //create the task, this will partition the stream and not PE BUILD APP & DEPLOY ./s4 s4r -a=org.apache.s4.example.twitter.TwitterCounterApp -b=`pwd`/test-apps/twitter-counter/build.gradle twitter-counter //build twitter-counter app ./s4 deploy -helix -c=cluster1 -s4r=file:`pwd`/test-apps/twitter-counter/build/libs/twitter-counter.s4r -appName=twitter-counter //deploy twitter-counter app ./s4 deploy -appName=twitter-adapter -c=cluster2 -b=`pwd`/test-apps/twitter-adapter/build.gradle -p=s4.adapter.output.stream=RawStatus //build twitter-adapter, deploy does not work here, use next command to deploy. ./s4 deploy -helix -c=cluster2 -s4r=file:`pwd`/test-apps/twitter-adapter/build/libs/twitter-adapter.s4r -appName=twitter-adapter -p=s4.adapter.output.stream=RawStatus START S4 Nodes ./s4 node -helix -cluster=cluster1 -zk=localhost:2181 & ./s4 node -helix -cluster=cluster1 -zk=localhost:2181 & START ADAPTER Node ./s4 node -helix -cluster=cluster2 -zk=localhost:2181 & tail -f TopNTopics.txt //shows the top trending topics You need to setup twitter4j.properties in your home directory to access twitter feed. thanks, Kishore G On Mon, Jan 13, 2014 at 7:13 AM, Sergio Vavassori wrote: > On 9 January 2014 20:29, kishore g wrote: > > > I will take a stab at making S4-110-new branch working. > > > > OK, thanks. > > Let me know if you progress on that. > > Best Regards, > Sergio Vavassori >
Re: Information about Apache S4 and Helix
On 9 January 2014 20:29, kishore g wrote: > I will take a stab at making S4-110-new branch working. > OK, thanks. Let me know if you progress on that. Best Regards, Sergio Vavassori
Re: Information about Apache S4 and Helix
I will take a stab at making S4-110-new branch working. Repartitioning is dependent on the application and is hard to implement. Helix architecture does not restrict one from implementing repartitioning. In general, partitioning splitting is hard and a common work around is to over partition and simply move partition around when new machines are added. thanks, Kishore G On Thu, Jan 9, 2014 at 8:22 AM, Sergio Vavassori wrote: > On 12 December 2013 18:54, kishore g wrote: > > > > > > https://git-wip-us.apache.org/repos/asf?p=incubator-s4.git;a=blob;f=subprojects/s4-tools/src/main/java/org/apache/s4/tools/helix/CreateTask.java;h=36a45bc5bef5e4c4f56bfb4c740abb315197ca21;hb=59bcb94cc727cb1600074e6da81330953aeaa566 > > > > When you add the task, you can use pe type s the name of the task and > > specify the streams it is interested in. > > > > Give it a try :-) > > > > Hello, > > First of all I'm sorry for the delay in reply but Christmas vacation didn't > help. > > I tried to use the CreateTask but it raised a ZK Exception about S4 cluster > not present (even when it was created manually). > > Is it correct that CreateTask creates all resources under > S4HelixConstants.HELIX_CLUSTER_NAME and a property "GROUP" with the > clusterName instead of using the clusterName as reference for > ConfigScopeBuilder? > > Then I found this repository: https://github.com/kishoreg/incubator-s4 and > even if it's older and needed some patches to make it work, I was able to > run it. Nevertheless the application didn't respond as supposed. (echo > "Bob" | nc localhost 16000 didn't return). > In details I did the following: > > ./gradlew install > ./gradlew s4-tools:installApp > > ./s4 zkServer & > ./s4 newCluster -c=cluster1 -nbNodes=2 -flp=12000 > ./s4 newCluster -c=cluster2 -nbNodes=1 -flp=16000 > ./s4 createTask -zk localhost:2181 -c cluster1 -id names -t consumer > -p 4 -r 1 -s names > ./s4 createTask -zk localhost:2181 -c cluster2 -id names -t producer > -p 1 -r 0 -s names > > ./s4 newApp myApp -parentDir=/tmp > cd /tmp/myApp > ./s4 s4r -a=hello.HelloApp -b=/tmp/myApp/build.gradle myApp > > ./s4 deployApp -c=cluster1 -s4r=/tmp/myApp/build/libs/myApp.s4r > -appName=myApp -zk=localhost:2181 & > ./s4 adapter -c=cluster2 -zk=localhost:2181 > -p=s4.adapter.output.stream=names,s4.instance.name=localhost_16000 & > > (I had to add the "-p" parameters because guava was requiring a value for > that) > > and these are their statuses > > > ./s4 status -c=cluster1 > --- > > calling referenced s4 script : /home/sergio/upm/repos/s4-helix/s4 > 1 [main] INFO org.apache.helix.manager.zk.ZKHelixManager - Create a > zk-based cluster manager. clusterName:cluster1, instanceName:ADMIN, > type:ADMINISTRATOR, zkSvr:localhost:2181 > > 57 [main] INFO org.apache.helix.messaging.handling.HelixTaskExecutor > - Adding msg factory for type TASK_REPLY threadpool size 40 > 59 [main] INFO org.apache.helix.manager.zk.ZKHelixManager - > ClusterManager.connect() > > 135 [main] INFO org.apache.helix.manager.zk.ZkStateChangeListener - > KeeperState: SyncConnected, zookeeper:State:CONNECTED Timeout:3 > sessionid:0x14376f2de0a002b local:/127.0.0.1:41285 > remoteserver:localhost/127.0.0.1:2181 lastZxid:0 xid:1 sent:1 recv:1 > queuedpkts:0 pendingresp:0 queuedevents:0 > > 136 [main] INFO org.apache.helix.manager.zk.ZKHelixManager - Handling > new session, session id:14376f2de0a002b, instance:ADMIN, instanceTye: > ADMINISTRATOR, cluster: cluster1 > 136 [main] INFO org.apache.helix.manager.zk.ZKHelixManager - > State:CONNECTED Timeout:3 sessionid:0x14376f2de0a002b > local:/127.0.0.1:41285 remoteserver:localhost/127.0.0.1:2181 > lastZxid:0 xid:1 sent:1 recv:1 queuedpkts:0 pendingresp:0 > queuedevents:0 > > Cluster Status > > -- > > Nodes > > Cluster Name Nodes Active > > >Node id > Host PortActive > > > -- > cluster1 2 2localhost_12000 > localhost 12000 Y > > localhost_12001 > localhost 12001 Y > App Status > > -- > > Name Cluster > URI > > -- > >myApp cluster1 > file:/tmp/myApp/build/libs/m
Re: Information about Apache S4 and Helix
On 12 December 2013 18:54, kishore g wrote: > > https://git-wip-us.apache.org/repos/asf?p=incubator-s4.git;a=blob;f=subprojects/s4-tools/src/main/java/org/apache/s4/tools/helix/CreateTask.java;h=36a45bc5bef5e4c4f56bfb4c740abb315197ca21;hb=59bcb94cc727cb1600074e6da81330953aeaa566 > > When you add the task, you can use pe type s the name of the task and > specify the streams it is interested in. > > Give it a try :-) > Hello, First of all I'm sorry for the delay in reply but Christmas vacation didn't help. I tried to use the CreateTask but it raised a ZK Exception about S4 cluster not present (even when it was created manually). Is it correct that CreateTask creates all resources under S4HelixConstants.HELIX_CLUSTER_NAME and a property "GROUP" with the clusterName instead of using the clusterName as reference for ConfigScopeBuilder? Then I found this repository: https://github.com/kishoreg/incubator-s4 and even if it's older and needed some patches to make it work, I was able to run it. Nevertheless the application didn't respond as supposed. (echo "Bob" | nc localhost 16000 didn't return). In details I did the following: ./gradlew install ./gradlew s4-tools:installApp ./s4 zkServer & ./s4 newCluster -c=cluster1 -nbNodes=2 -flp=12000 ./s4 newCluster -c=cluster2 -nbNodes=1 -flp=16000 ./s4 createTask -zk localhost:2181 -c cluster1 -id names -t consumer -p 4 -r 1 -s names ./s4 createTask -zk localhost:2181 -c cluster2 -id names -t producer -p 1 -r 0 -s names ./s4 newApp myApp -parentDir=/tmp cd /tmp/myApp ./s4 s4r -a=hello.HelloApp -b=/tmp/myApp/build.gradle myApp ./s4 deployApp -c=cluster1 -s4r=/tmp/myApp/build/libs/myApp.s4r -appName=myApp -zk=localhost:2181 & ./s4 adapter -c=cluster2 -zk=localhost:2181 -p=s4.adapter.output.stream=names,s4.instance.name=localhost_16000 & (I had to add the "-p" parameters because guava was requiring a value for that) and these are their statuses ./s4 status -c=cluster1 --- calling referenced s4 script : /home/sergio/upm/repos/s4-helix/s4 1 [main] INFO org.apache.helix.manager.zk.ZKHelixManager - Create a zk-based cluster manager. clusterName:cluster1, instanceName:ADMIN, type:ADMINISTRATOR, zkSvr:localhost:2181 57 [main] INFO org.apache.helix.messaging.handling.HelixTaskExecutor - Adding msg factory for type TASK_REPLY threadpool size 40 59 [main] INFO org.apache.helix.manager.zk.ZKHelixManager - ClusterManager.connect() 135 [main] INFO org.apache.helix.manager.zk.ZkStateChangeListener - KeeperState: SyncConnected, zookeeper:State:CONNECTED Timeout:3 sessionid:0x14376f2de0a002b local:/127.0.0.1:41285 remoteserver:localhost/127.0.0.1:2181 lastZxid:0 xid:1 sent:1 recv:1 queuedpkts:0 pendingresp:0 queuedevents:0 136 [main] INFO org.apache.helix.manager.zk.ZKHelixManager - Handling new session, session id:14376f2de0a002b, instance:ADMIN, instanceTye: ADMINISTRATOR, cluster: cluster1 136 [main] INFO org.apache.helix.manager.zk.ZKHelixManager - State:CONNECTED Timeout:3 sessionid:0x14376f2de0a002b local:/127.0.0.1:41285 remoteserver:localhost/127.0.0.1:2181 lastZxid:0 xid:1 sent:1 recv:1 queuedpkts:0 pendingresp:0 queuedevents:0 Cluster Status -- Nodes Cluster Name Nodes Active Node id Host PortActive -- cluster1 2 2localhost_12000 localhost 12000 Y localhost_12001 localhost 12001 Y App Status -- Name Cluster URI -- myApp cluster1 file:/tmp/myApp/build/libs/myApp.s4r -- Node idDEPLOYED -- localhost_12000 Y localhost_12001 Y -- App Status -
Re: Information about Apache S4 and Helix
There is a discussion about stream partitioning v/s PE partitioning in that JIRA. I had some implementation that allows both types of partitioning schemes. https://git-wip-us.apache.org/repos/asf?p=incubator-s4.git;a=blob;f=subprojects/s4-tools/src/main/java/org/apache/s4/tools/helix/CreateTask.java;h=36a45bc5bef5e4c4f56bfb4c740abb315197ca21;hb=59bcb94cc727cb1600074e6da81330953aeaa566 When you add the task, you can use pe type s the name of the task and specify the streams it is interested in. Give it a try :-) On Wed, Dec 11, 2013 at 8:50 AM, Sergio Vavassori wrote: > > > > It's my understanding that a cluster is a group of nodes and each node > has > >> the same application-code copy; this means that if I want to partition > the > >> ProcessingElements between nodes I need to group them in different > >> clusters. > >> So, mapping S4 elements into a "classical" Stream Processing naming > >> (Nodes, > >> Operators, Slides...), would be having one application (Operator) per > >> cluster and configure the ProcessingElements as singleton (1 Slide per > >> Node). > >> > > > > you should use a key to partition your stream. Use the KeyFinder to > > identify keys in events. > > > but this is Stream Partitioning, not Processing Element Partitioning. I > could use this system, but for my needs I should then multiply each message > for the number of partitions and I still wouldn’t be able to control PE > allocation between types in the same nodes. > > E.g. if I have: > > * 2 PE types: PE1 and PE2, configured as singletons. > * 3 nodes > > in each node both PE types can be allocated, so 6 instead of 3. Is there a > way to allocate only one PE per node? > > Regards, > Sergio Vavassori >
Re: Information about Apache S4 and Helix
Hi, please find some answers inline. Hope this helps On Dec 2, 2013, at 17:47 , Sergio Vavassori wrote: > Good morning, > > I have started using Apache S4 for a university project and I wanted to ask > you some question about its architecture, mainly to be sure to do the > modifications I need in the right way and to see if there is a cleaner and > simpler one. > > It's my understanding that a cluster is a group of nodes and each node has > the same application-code copy; this means that if I want to partition the > ProcessingElements between nodes I need to group them in different clusters. > So, mapping S4 elements into a "classical" Stream Processing naming (Nodes, > Operators, Slides...), would be having one application (Operator) per > cluster and configure the ProcessingElements as singleton (1 Slide per > Node). you should use a key to partition your stream. Use the KeyFinder to identify keys in events. > > About inter-cluster streaming: > Is it possible to have broadcast stream between one cluster and all nodes > of another cluster? Or should I re-implement RemoteSenders to do that? In > this last case, is there a way to unbind Module mappring between interface > and class used to resolve @inject? Normally when key is null, dispatch mode is broadcast. Except for inter-cluster communication, where events are sent in round-robin mode by default. To change that, you'd need to define your own senders, a configuration module, and override or replace the existing related modules. > Is there a way to have that feature as per-stream configuration rather than > all-stream cluster-wide? That would be possible. Maybe even just by injecting your own implementations. > Is there any functional difference (or limitation) between "RemoteStreams" > and "Streams" beyond the naming to recognize inter-cluster vs intra-cluster > streaming? Differentiating inter and intra cluster communications is the reason. It's useful for publishing and performing bindings. And usually inter-cluster events are generic types, whereas you may have application-specific typed events for intra cluster communications. > > I saw there is an ongoing integration with helix project, which has a > slightly different concept for partition since it can host more than one on > the same node, but I couldn't find any example. Is there any work on it? > > > Regards, > Sergio Vavassori
Re: Information about Apache S4 and Helix
Work on S4 and Helix integration is here https://git-wip-us.apache.org/repos/asf?p=incubator-s4.git;a=shortlog;h=refs/heads/S4-110-new Here is the JIRA https://issues.apache.org/jira/browse/S4-110 thanks, Kishore G On Mon, Dec 2, 2013 at 8:47 AM, Sergio Vavassori wrote: > Good morning, > > I have started using Apache S4 for a university project and I wanted to ask > you some question about its architecture, mainly to be sure to do the > modifications I need in the right way and to see if there is a cleaner and > simpler one. > > It's my understanding that a cluster is a group of nodes and each node has > the same application-code copy; this means that if I want to partition the > ProcessingElements between nodes I need to group them in different > clusters. > So, mapping S4 elements into a "classical" Stream Processing naming (Nodes, > Operators, Slides...), would be having one application (Operator) per > cluster and configure the ProcessingElements as singleton (1 Slide per > Node). > > About inter-cluster streaming: > Is it possible to have broadcast stream between one cluster and all nodes > of another cluster? Or should I re-implement RemoteSenders to do that? In > this last case, is there a way to unbind Module mappring between interface > and class used to resolve @inject? > Is there a way to have that feature as per-stream configuration rather than > all-stream cluster-wide? > Is there any functional difference (or limitation) between "RemoteStreams" > and "Streams" beyond the naming to recognize inter-cluster vs intra-cluster > streaming? > > I saw there is an ongoing integration with helix project, which has a > slightly different concept for partition since it can host more than one on > the same node, but I couldn't find any example. Is there any work on it? > > > Regards, > Sergio Vavassori >
