hi Matthieu
i am still confused about zookeeper. the first step to deploy our
applications is using the zkServer -clean; then i use the zookeeper command:
[s4@node4 ~]$echo srvr | nc localhost 2181
Zookeeper version: 3.3.3-1073969, built on 02/23/2011 22:27 GMT
Latency min/avg/max: 0/0/45
Received: 1647
Sent: 1662
Outstanding: 0
Zxid: 0x49
Mode: standalone
Node count: 33
that means zookeeper run in the standalone model.
and now i want to run the HelloApp in a distributed model as following:
1. start a zkServer at node4:
[s4@node4 ~]$./s4 zkServer -clean
2. start a new cluster1 with 2 nbTasks, and a new cluster2 with 1 nbTasks:
[s4@node5 myApp]$./s4 newCluster -c=cluster1 -flp=12000 -nbTasks=2
-zk=node4:2181
[s4@node5 myApp]$./s4 newCluster -c=cluster2 -flp=13000 -nbTasks=1
-zk=node4:2181
3. get the s4r:
[s4@node5 myApp]$ ./s4 s4r -a=hello.HelloApp -b=`pwd`/build.gradle myApp
4. started a 2 nodes with 1 nbTasks in node5, i give the adapter each of
them started in node5 and node6;
[s4@node5 myApp]$./s4 node -c=cluster1 -zk=node4:2181
[s4@node6 myApp]$./s4 node -c=cluster1 -zk=node4:2181
5.deploy the helloApp
[s4@node5 myApp]$./s4 deploy -s4r=`pwd`/build/libs/myApp.s4r -c=cluster1
-appName=myApp -zk=node4:2181
6.get the input and run:
[s4@node5 myApp]$./s4 deploy -appClass=hello.HelloInputAdapter
-p=s4.adapter.output.stream=names -c=cluster2 -appName=adapter
-zk=node4:2181
[s4@node5 myApp]$./s4 adapter -c=cluster2 -zk=node4:2181
[s4@node5 ~]$ echo "bob" | nc localhost 15000
[s4@node5 ~]$ echo "bob" | nc localhost 15000
[s4@node5 ~]$ echo "bob" | nc localhost 15000
[s4@node5 ~]$ echo "bob" | nc localhost 15000
and at the node5 console i got：
14:59:57.545 [S4 platform loader] INFO o.a.s4.comm.topology.ClusterFromZK
- Changing cluster topology to {
nbNodes=2,name=cluster1,mode=unicast,type=,nodes=[{partition=0,port=12000,machineName=node6,taskId=Task-0},
{partition=1,port=12001,machineName=node5,taskId=Task-1}]} from null
14:59:57.548 [S4 platform loader] INFO org.apache.s4.core.S4Bootstrap -
Loaded application from file /tmp/tmp7714456429861539962s4r
14:59:57.599 [S4 platform loader] DEBUG o.a.s4.comm.topology.ClustersFromZK
- Adding input stream [names] in cluster [cluster1]
14:59:57.638 [S4 platform loader] INFO org.apache.s4.core.App - Init
prototype [hello.HelloPE].
Hello bob!
Hello again bob!
but at the node6 console i got:
content: bob
StreamID: names
getTime: 1369897516243
14:59:47.724 [names] ERROR org.apache.s4.core.S4Node - Uncaught exception
in thread names
java.lang.NullPointerException: null
at org.apache.s4.base.Event$Data.access$100(Event.java:162)
~[s4-base-0.6.0-incubating.jar:0.6.0-incubating]
at org.apache.s4.base.Event.get(Event.java:137)
~[s4-base-0.6.0-incubating.jar:0.6.0-incubating]
at hello.HelloPE.onEvent(HelloPE.java:43) ~[myApp.jar:unspecified]
at OverloadDispatcher145.dispatchEvent(Unknown Source) ~[na:na]
at
org.apache.s4.core.ProcessingElement.handleInputEvent(ProcessingElement.java:461)
~[s4-core-0.6.0-incubating.jar:0.6.0-incubating]
at
org.apache.s4.core.Stream$StreamEventProcessingTask.run(Stream.java:333)
~[s4-core-0.6.0-incubating.jar:0.6.0-incubating]
at
org.apache.s4.comm.staging.BlockingThreadPoolExecutorService$RunnableWithPermitRelease.run(BlockingThreadPoolExecutorService.java:178)
~[s4-comm-0.6.0-incubating.jar:0.6.0-incubating]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
~[na:1.6.0_17]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
~[na:1.6.0_17]
at java.lang.Thread.run(Thread.java:636) ~[na:1.6.0_17]
content: bob
StreamID: names
getTime: 1369897516243
14:59:47.724 [names] ERROR org.apache.s4.core.S4Node - Uncaught exception
in thread names
java.lang.NullPointerException: null
at org.apache.s4.base.Event$Data.access$100(Event.java:162)
~[s4-base-0.6.0-incubating.jar:0.6.0-incubating]
at org.apache.s4.base.Event.get(Event.java:137)
~[s4-base-0.6.0-incubating.jar:0.6.0-incubating]
at hello.HelloPE.onEvent(HelloPE.java:43) ~[myApp.jar:unspecified]
at OverloadDispatcher145.dispatchEvent(Unknown Source) ~[na:na]
at
org.apache.s4.core.ProcessingElement.handleInputEvent(ProcessingElement.java:461)
~[s4-core-0.6.0-incubating.jar:0.6.0-incubating]
at
org.apache.s4.core.Stream$StreamEventProcessingTask.run(Stream.java:333)
~[s4-core-0.6.0-incubating.jar:0.6.0-incubating]
at
org.apache.s4.comm.staging.BlockingThreadPoolExecutorService$RunnableWithPermitRelease.run(BlockingThreadPoolExecutorService.java:178)
~[s4-comm-0.6.0-incubating.jar:0.6.0-incubating]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
~[na:1.6.0_17]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
~[na:1.6.0_17]
at java.lang.Thread.run(Thread.java:636) ~[na:1.6.0_17]
what is the problem ? did i lose configuration before deploy a distributed
application?
thanks a lot.
2013/5/29 Matthieu Morel <mmo...@apache.org>
>
> On May 29, 2013, at 03:50 , baojian Zhou wrote:
>
> > hi all, i want to run the benchmark in 4 physical machines in a cluster.
> These 4 physical machines had installed zookeeper. But i have a problem
> that confuse me a lot. As we all know, before we run the s4 application we
> have to start the zk Server first. this is my confuse:
> > Do i need to start all zk Server in these 4 machines,or just to start a
> zk Server in one of them ? just like this:
> > ./s4 zkServer -clean
> -clusters=c=testCluster1:flp=12000:nbTasks=$NB_INJECTORS,c=testCluster2:flp=13000:nbTasks=$NB_NODES
> &)
> >
>
> ZooKeeper is used to configure/coordinate S4 nodes, in particular it is
> used for the assignment of partitions. Therefore you have to share a
> ZooKeeper ensemble (or a single instance for testing) across nodes, by
> referring to the ZooKeeper server(s) when starting S4 nodes.
>
> Matthieu
>
>
