You can add the key/values you are setting with -D in hadoop jar to your giraph-site.xml and load those values in your job?
Unrelated, but it doesn't look like you need to set the ZK java opts in your -D args since you also specify a ZK server list which tells Giraph to assume the quorum is already up and running and to use those addresses to connect to it. On Wed, Oct 2, 2013 at 11:22 AM, Jyotirmoy Sundi <[email protected]> wrote: > Hi, > Giraph is running awesomely. But it cant start the tasks when run from > oozie. > > I am able to run from the terminal using fat jar without any issues. I > placed the code and examples jar in the libpath. Now I am trying to run a > job through oozie. The GiraphRunner starts the oozie job, but fails to > start any tasks. When i see the TT machines, there are no logs, the > counters remain at 0 and all get killed after few seconds., its displays > like this: > task_201309260044_1170_m_000000<http://had24.rsk.admobius.com:50030/taskdetails.jsp?tipid=task_201309260044_1170_m_000000> > 0.00% > > 2-Oct-2013 17:28:23 > > > 0<http://had24.rsk.admobius.com:50030/taskstats.jsp?tipid=task_201309260044_1170_m_000000> > task_201309260044_1170_m_000001<http://had24.rsk.admobius.com:50030/taskdetails.jsp?tipid=task_201309260044_1170_m_000001> > 0.00% > > 2-Oct-2013 17:28:23 > > > 0<http://had24.rsk.admobius.com:50030/taskstats.jsp?tipid=task_201309260044_1170_m_000001> > task_201309260044_1170_m_000002<http://had24.rsk.admobius.com:50030/taskdetails.jsp?tipid=task_201309260044_1170_m_000002> > 0.00% > > 2-Oct-2013 17:28:23 > > > 0<http://had24.rsk.admobius.com:50030/taskstats.jsp?tipid=task_201309260044_1170_m_000002> > <http://had24.rsk.admobius.com:50030/taskdetails.jsp?tipid=task_201309260044_1170_m_000003> > > *The logs are here:* > > 2013-10-02 18:06:00,744 INFO org.apache.hadoop.mapred.MapTask: > numReduceTasks: 0 > 2013-10-02 18:06:00,929 INFO org.apache.giraph.utils.ConfigurationUtils: No > edge input format specified. Ensure your InputFormat does not require one. > 2013-10-02 18:06:00,990 INFO org.apache.giraph.job.GiraphJob: run: Since > checkpointing is disabled (default), do not allow any task retries (setting > mapred.map.max.attempts = 0, old value = 4) > 2013-10-02 18:06:01,040 WARN org.apache.hadoop.mapred.JobClient: Use > GenericOptionsParser for parsing the arguments. Applications should implement > Tool for the same. > 2013-10-02 18:06:01,232 INFO org.apache.hadoop.mapred.JobClient: Running job: > job_201309260044_1176 > 2013-10-02 18:06:02,238 INFO org.apache.hadoop.mapred.JobClient: map 0% > reduce 0% > 2013-10-02 18:06:11,276 INFO org.apache.hadoop.mapred.JobClient: Job > complete: job_201309260044_1176 > 2013-10-02 18:06:11,279 INFO org.apache.hadoop.mapred.JobClient: Counters: 4 > 2013-10-02 18:06:11,282 INFO org.apache.hadoop.mapred.JobClient: Job > Counters > 2013-10-02 18:06:11,284 INFO org.apache.hadoop.mapred.JobClient: Total > time spent by all maps in occupied slots (ms)=9390 > 2013-10-02 18:06:11,284 INFO org.apache.hadoop.mapred.JobClient: Total > time spent by all reduces in occupied slots (ms)=0 > 2013-10-02 18:06:11,284 INFO org.apache.hadoop.mapred.JobClient: Total > time spent by all maps waiting after reserving slots (ms)=0 > 2013-10-02 18:06:11,284 INFO org.apache.hadoop.mapred.JobClient: Total > time spent by all reduces waiting after reserving slots (ms)=0 > 2013-10-02 18:06:11,305 INFO org.apache.hadoop.mapred.Task: > Task:attempt_201309260044_1175_m_000000_0 is done. And is in the process of > commiting > 2013-10-02 18:06:12,399 INFO org.apache.hadoop.mapred.Task: Task > attempt_201309260044_1175_m_000000_0 is allowed to commit now > 2013-10-02 18:06:12,427 INFO org.apache.hadoop.mapred.FileOutputCommitter: > Saved output of task 'attempt_201309260044_1175_m_000000_0' to hdfs: > /user/sundi133/oozie-oozi/0000243-130928181649085-oozie-oozi-W/fingerprint-05-giraph--java/output > 2013-10-02 18:06:12,431 INFO org.apache.hadoop.mapred.Task: Task > 'attempt_201309260044_1175_m_000000_0' done. > 2013 > > > > *Terminal command : success* > hadoop jar > /home/sundi133/giraph/giraph/giraph-examples/target/giraph-examples-1.0.0-for-hadoop-2.0.3-alpha-jar-with-dependencies.jar > org.apache.giraph.GiraphRunner -Dgiraph.zKForceSync=true > -Dgiraph.zkJavaOpts="-Xmx2g -XX:ParallelGCThreads=4 -XX:+UseConcMarkSweepGC > -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxGCPauseMillis=100" > -Dmapred.child.java.opts="-Xmx64g" -Dgiraph.zkSessionMsecTimeout=600000 > -Dgiraph.useInputSplitLocality=false -Dgiraph.zkList= > mymac23.rsk.admobs.com:2181,mymac16.rsk.admobs.com:2181,mymac > 22.rsk.admobs.com:2181 -Dgiraph.maxVerticesPerTransfer=100000 > -Dgiraph.maxEdgesPerTransfer=200000 -Dgiraph.isStaticGraph=true > org.apache.giraph.examples.ConnectedComponentsVertexAdmobius -vif > org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormatAdmobius > -vip /data/production/batched/fingerprint/fp_match_matrix/batch_id= > 2013090614/pa* -op > /data/production/batched/fingerprint/hashed_cc/batch_id=2013090639 -of > org.apache.giraph.examples.VertexWithDoubleValueDoubleEdgeTextOutputFormatAdmobius > -c org.apache.giraph.combiner.MinimumLongCombiner -w 115 > > > *Oozie xml:* > > > > > * <action name="fingerprint-05-giraph"> <java> > <job-tracker>${ createCluster == true ? > wf:actionData('start-new-cluster')['jobTracker'] : jobTracker } > </job-tracker> <name-node>${ > createCluster == true ? wf:actionData('start-new-cluster')['nameNode'] : > nameNode } </name-node> > <job-xml>${nameNode}/user/${wf:user()}/hive/${env}/job.xml</job-xml> > <configuration> <property> > <name>mapred.job.queue.name > <http://mapred.job.queue.name></name> > <value>${queueName}</value> </property> > </configuration> > <main-class>org.apache.giraph.GiraphRunner</main-class> > <arg>org.apache.giraph.examples.ConnectedComponentsVertexAdmobius</arg> > <arg>-vif</arg> > <arg>org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormatAdmobius</arg> > <arg>-vip</arg> > <arg>${nameNode}/data/production/batched/fingerprint/fp_match_matrix/batch_id=${batchId}/pa*</arg> > <arg>-op</arg> > <arg>${nameNode}/data/production/batched/fingerprint/hashed_cc/batch_id=${batchId}</arg> > <arg>-of</arg> > <arg>org.apache.giraph.examples.VertexWithDoubleValueDoubleEdgeTextOutputFormatAdmobius</arg> > <arg>-c</arg> > <arg>org.apache.giraph.combiner.MinimumLongCombiner</arg> > <arg>-w</arg> <arg>118</arg> > <file>giraph-site.xml</file> <capture-output > /> </java> <ok > to="fingerprint-update-connnectedcomponents" /> <error > to="fail" /> </action> Any pointers over it would be appreciated.* > -- > Best Regards, > Jyotirmoy Sundi > Data Engineer, > Admobius > > San Francisco, CA 94158 >
