[
https://issues.apache.org/jira/browse/GEARPUMP-147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karol Brejna updated GEARPUMP-147:
----------------------------------
Description:
Doing GEARPUMP-146 I hit a problem: I was able to run a master on pc and a
worker on raspi. After deploying wordcountjava-2.11-0.8.0-assembly.jar the
example didn't start.
I reproduced this on "pure pc" cluster.
I've prepared dedicated config files:
* gear.pc.conf - normal (default) configuration for master
and workers ran on a PC
* gear.pc.slimworker.conf - intends to have lower requirements for memory, cpu
- to be used on a device (raspi, edison)
*Obtaining binary and creating config files*
{code}
wget
https://github.com/gearpump/gearpump/releases/download/0.8.0/gearpump-2.11-0.8.0.zip
unzip gearpump-2.11-0.8.0.zip
cd gearpump-2.11-0.8.0
cp conf/gear.conf conf/gear.original.conf
cp conf/gear.conf conf/gear.pc.conf
cp conf/gear.conf conf/gear.pc.slimworker.conf
{code}
*Preparing "PC" conf*
The only real change here is putting the IP of the machine that will run the
master:
{code}
cluster {
masters = ["192.168.1.115:3000"]
}
hostname = "192.168.1.115"
{code}
*Preparing device conf*
Compared to previous config, this config should set up less worker slots,
smaller -Xmx jvm switch for worker, and executor-share-same-jvm-as-worker set
to true:
{code}
worker.slots = 100
worker.executor-share-same-jvm-as-worker = true
executor {
vmargs = "-server -Xms512M -Xmx512M -Xss1M -XX:+HeapDumpOnOutOfMemoryError
-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=80 -XX:+UseParNewGC
-XX:NewRatio=3 -Djava.rmi.server.hostname=localhost"
extraClasspath = ""
}
{code}
*Running the cluster:*
{code}
echo "Starting the master"
cp conf/gear.pc.conf conf/gear.conf
bin/master -ip 192.168.1.115 -port 3000 &
echo "Starting services"
bin/services &
echo "Starting a worker (configured as on the device)"
cp conf/gear.pc.slimworker.conf conf/gear.conf
bin/worker &
{code}
*Deploing the DAG*
I've deployed wordcountjava-2.11-0.8.0-assembly.jar.
The app deploys, but doesn't run.
(After changing worker.executor-share-same-jvm-as-worker to false and
restarting the cluster, example dag works.)
See the attached logs:
{code}
2016-05-18 08:47:31,453 ERROR OneForOneStrategy:
io.gearpump.streaming.examples.wordcountjava.Split
java.lang.ClassNotFoundException:
io.gearpump.streaming.examples.wordcountjava.Split
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at io.gearpump.streaming.task.TaskUtil$.loadClass(TaskUtil.scala:30)
at
io.gearpump.streaming.executor.TaskLauncher.launch(TaskLauncher.scala:55)
at
io.gearpump.streaming.executor.Executor$$anonfun$dynamicDagPhase1$1.applyOrElse(Executor.scala:147)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
at akka.actor.Actor$class.aroundReceive(Actor.scala:480)
at
io.gearpump.streaming.executor.Executor.aroundReceive(Executor.scala:55)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
at akka.actor.ActorCell.invoke(ActorCell.scala:495)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
{code}
was:
Doing GEARPUMP-146 I hit a problem: I was able to run a master on pc and a
worker on raspi. After deploying wordcountjava-2.11-0.8.0-assembly.jar the
example didn't start.
I reproduced this on "pure pc" cluster.
I've prepared dedicated config files:
* gear.pc.conf - normal (default) configuration for master
and workers ran on a PC
* gear.pc.slimworker.conf - intends to have lower requirements for memory, cpu
- to be used on a device (raspi, edison)
*Obtaining binary and creating config files*
{code}
wget
https://github.com/gearpump/gearpump/releases/download/0.8.0/gearpump-2.11-0.8.0.zip
unzip gearpump-2.11-0.8.0.zip
cd gearpump-2.11-0.8.0
cp conf/gear.conf conf/gear.original.conf
cp conf/gear.conf conf/gear.pc.conf
cp conf/gear.conf conf/gear.pc.slimworker.conf
{code}
*Preparing "PC" conf*
The only real change here is putting the IP of the machine that will run the
master:
{code}
cluster {
masters = ["192.168.1.115:3000"]
}
hostname = "192.168.1.115"
{code}
*Preparing device conf*
Compared to previous config, this config should set up less worker slots,
smaller -Xmx jvm switch for worker, and executor-share-same-jvm-as-worker set
to true:
{code}
worker.slots = 100
worker.executor-share-same-jvm-as-worker = true
executor {
vmargs = "-server -Xms512M -Xmx512M -Xss1M -XX:+HeapDumpOnOutOfMemoryError
-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=80 -XX:+UseParNewGC
-XX:NewRatio=3 -Djava.rmi.server.hostname=localhost"
extraClasspath = ""
}
{code}
*Running the cluster:*
{code}
echo "Starting the master"
cp conf/gear.pc.conf conf/gear.conf
bin/master -ip 192.168.1.115 -port 3000 &
echo "Starting services"
bin/services &
echo "Starting a worker (configured as on the device)"
cp conf/gear.pc.slimworker.conf conf/gear.conf
bin/worker &
{code}
*Deploing the DAG*
I've deployed wordcountjava-2.11-0.8.0-assembly.jar.
The app deploys, but doesn't run.
After changing worker.executor-share-same-jvm-as-worker to false and
restarting the cluster, example dag works.
> java.lang.ClassNotFoundException when running on worker with
> worker.executor-share-same-jvm-as-worker = true
> ------------------------------------------------------------------------------------------------------------
>
> Key: GEARPUMP-147
> URL: https://issues.apache.org/jira/browse/GEARPUMP-147
> Project: Apache Gearpump
> Issue Type: Task
> Reporter: Karol Brejna
> Attachments: logs.slimworker.zip
>
>
> Doing GEARPUMP-146 I hit a problem: I was able to run a master on pc and a
> worker on raspi. After deploying wordcountjava-2.11-0.8.0-assembly.jar the
> example didn't start.
> I reproduced this on "pure pc" cluster.
> I've prepared dedicated config files:
> * gear.pc.conf - normal (default) configuration for
> master and workers ran on a PC
> * gear.pc.slimworker.conf - intends to have lower requirements for memory,
> cpu - to be used on a device (raspi, edison)
> *Obtaining binary and creating config files*
> {code}
> wget
> https://github.com/gearpump/gearpump/releases/download/0.8.0/gearpump-2.11-0.8.0.zip
> unzip gearpump-2.11-0.8.0.zip
> cd gearpump-2.11-0.8.0
> cp conf/gear.conf conf/gear.original.conf
> cp conf/gear.conf conf/gear.pc.conf
> cp conf/gear.conf conf/gear.pc.slimworker.conf
> {code}
> *Preparing "PC" conf*
> The only real change here is putting the IP of the machine that will run the
> master:
> {code}
> cluster {
> masters = ["192.168.1.115:3000"]
> }
>
> hostname = "192.168.1.115"
> {code}
>
> *Preparing device conf*
> Compared to previous config, this config should set up less worker slots,
> smaller -Xmx jvm switch for worker, and executor-share-same-jvm-as-worker set
> to true:
> {code}
> worker.slots = 100
> worker.executor-share-same-jvm-as-worker = true
> executor {
> vmargs = "-server -Xms512M -Xmx512M -Xss1M
> -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC
> -XX:CMSInitiatingOccupancyFraction=80 -XX:+UseParNewGC -XX:NewRatio=3
> -Djava.rmi.server.hostname=localhost"
> extraClasspath = ""
> }
> {code}
> *Running the cluster:*
> {code}
> echo "Starting the master"
> cp conf/gear.pc.conf conf/gear.conf
> bin/master -ip 192.168.1.115 -port 3000 &
> echo "Starting services"
> bin/services &
> echo "Starting a worker (configured as on the device)"
> cp conf/gear.pc.slimworker.conf conf/gear.conf
> bin/worker &
> {code}
> *Deploing the DAG*
> I've deployed wordcountjava-2.11-0.8.0-assembly.jar.
> The app deploys, but doesn't run.
> (After changing worker.executor-share-same-jvm-as-worker to false and
> restarting the cluster, example dag works.)
> See the attached logs:
> {code}
> 2016-05-18 08:47:31,453 ERROR OneForOneStrategy:
> io.gearpump.streaming.examples.wordcountjava.Split
> java.lang.ClassNotFoundException:
> io.gearpump.streaming.examples.wordcountjava.Split
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at io.gearpump.streaming.task.TaskUtil$.loadClass(TaskUtil.scala:30)
> at
> io.gearpump.streaming.executor.TaskLauncher.launch(TaskLauncher.scala:55)
> at
> io.gearpump.streaming.executor.Executor$$anonfun$dynamicDagPhase1$1.applyOrElse(Executor.scala:147)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> at akka.actor.Actor$class.aroundReceive(Actor.scala:480)
> at
> io.gearpump.streaming.executor.Executor.aroundReceive(Executor.scala:55)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
> at akka.actor.ActorCell.invoke(ActorCell.scala:495)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
> at akka.dispatch.Mailbox.run(Mailbox.scala:224)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
