Re: Deploying samza job to a remote yarn resource manager

2015-06-23 Thread Navina Ramesh
Hi George, JobRunner is meant to run on the localhost only. You have write a wrapper script (perhaps in python or bash) to ssh to the RM and then run the JobRunner. RM machine should already have HADOOP_YARN_HOME and HADOOP_YARN_CONF_DIR configured, if the RM is running. So, probably not necessary

Re: 3 processed message per incoming message

2015-06-23 Thread Yan Fang
Hi Shekar, Cool. Let me know if it works. Thanks, Fang, Yan yanfang...@gmail.com On Tue, Jun 23, 2015 at 12:40 PM, Shekar Tippur wrote: > Yan, > > I have restarted Yarn but I still see the same issue. I will post this > question on Yarn mailing list. > yarn-...@hadoop.apache.org > > - Shekar

Re: How to configure log4j separately for the AM versus containers?

2015-06-23 Thread Roger Hoover
Ah, this seems to work. I saw the YarnJob.scala was referencing __package to launch to AM itself. yarn.am.opts=-Xmx768m -XX:+UseSerialGC -Dlog4j.configuration=file://$(pwd)/__package/lib/log4j-am.xml On Tue, Jun 23, 2015 at 12:40 PM, Roger Hoover wrote: > Hi, > > I want the App Master to log a

Re: 3 processed message per incoming message

2015-06-23 Thread Shekar Tippur
Yan, I have restarted Yarn but I still see the same issue. I will post this question on Yarn mailing list. yarn-...@hadoop.apache.org - Shekar

How to configure log4j separately for the AM versus containers?

2015-06-23 Thread Roger Hoover
Hi, I want the App Master to log at INFO level and the container to log at ERROR. Is there a way to configure the AM to use a different log4j config file? I'm trying to setting yarn.am.opts but ran couldn't get it to work with system properties. yarn.am.opts=-Xmx768m -XX:+UseSerialGC -Dlog4j.co

Re: 3 processed message per incoming message

2015-06-23 Thread Yan Fang
Hi Shekar, My guess is that, in order to get this property to take effect, you may need to restart the yarn. Otherwise, send this question to the Yarn mailing list, you may get more valuable answers. Thanks, Fang, Yan yanfang...@gmail.com On Mon, Jun 22, 2015 at 2:43 PM, Shekar Tippur wrote:

Re: Deploying samza job to a remote yarn resource manager

2015-06-23 Thread George Li
Hi Navina and Milinda, Thanks for your reply. I get it working by setting HADOOP_YARN_HOME and HADOOP_CONF_DIR export HADOOP_YARN_HOME=$HOME/hadoop-2.6.0/ export HADOOP_CONF_DIR=$HOME/hadoop-2.6.0/etc/hadoop ,i.e., similar to what run-class.sh does I am also trying to figure out how to deploy