Hi Steve, Currently the counters/stats information is only limited to Pig and Map-reduce actions. (http://incubator.apache.org/oozie/docs/3.2.0-incubating/docs/WorkflowFuncti onalSpec.html#a4.2.5_Hadoop_EL_Functions)
Thanks, Virag On 7/11/12 4:01 PM, "[email protected]" <[email protected]> wrote: > I'm finally getting back to this. Trying to get counter information looked > promising. I tried it, but my workflow job fails with exception below. I'll > keep looking over my syntax, but does anyone know if maybe this feature is > limited to map/reduce actions? The action I'm using is: > <sqoop xmlns="uri:oozie:sqoop-action:0.2"> > > Any advice on what to check or try is appreciated. > Thanks. > > > ACTION[0035106-120705224209703-oozie-oozi-W@test_counters] Exception in > ActionStartCommand > java.lang.IllegalArgumentException: Hadoop counters not available for action > [sqoop_task] > at > org.apache.oozie.action.hadoop.HadoopELFunctions.getCounters(HadoopELFunctions > .java:57) > at > org.apache.oozie.action.hadoop.HadoopELFunctions.hadoop_counters(HadoopELFunct > ions.java:47) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.j > ava:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.commons.el.FunctionInvocation.evaluate(FunctionInvocation.java:172) > at org.apache.commons.el.ComplexValue.evaluate(ComplexValue.java:140) > at org.apache.commons.el.ExpressionString.evaluate(ExpressionString.java:114) > at > org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl > .java:274) > at > org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl > .java:190) > at org.apache.oozie.util.ELEvaluator.evaluate(ELEvaluator.java:199) > at > org.apache.oozie.command.wf.ActionStartCommand.call(ActionStartCommand.java:94> ) > at > org.apache.oozie.command.wf.ActionStartCommand.execute(ActionStartCommand.java > :249) > at > org.apache.oozie.command.wf.ActionStartCommand.execute(ActionStartCommand.java > :47) > at org.apache.oozie.command.Command.call(Command.java:202) > at > org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueu > eService.java:132) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java > :886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908> ) > at java.lang.Thread.run(Thread.java:662) > > > Steve Charles > > Sogeti USA > 3636 N. Central Ave. > #810 > Phoenix, AZ 85012 > www.us.sogeti.com <http://www.us.sogeti.com/> > > > ________________________________ > > From: Jarek Jarcec Cecho [mailto:[email protected]] > Sent: Wed 7/4/2012 12:19 AM > To: [email protected] > Cc: [email protected] > Subject: Re: question on oozie workflow and log output > > > > Hi Steve, > I'm not sure how to process log output, but oozie do have the ability to work > with counters. There is EL function hadoop:counters(action name) that will > return all counters for given action. You can find examples in the > documentation: > > http://incubator.apache.org/oozie/docs/3.2.0-incubating/docs/WorkflowFunctiona > lSpec.html > > Those counters are available even for the sqoop action if I remember > correctly. > > Jarcec > > On Tue, Jul 03, 2012 at 04:29:38PM -0400, [email protected] wrote: >> Does anyone know how to make use of log output from one workflow action in a >> following action? >> Specifically, I have a sqoop action to import data into the HDFS. Following >> that action (if ok) I want to make use of information like rows/bytes >> imported in the sqoop action. I can, via the oozie console, look at the log >> file for the sqoop action and scrolling down, I see something like: >> >> Job complete: job_201205212157_421694 >> Counters: 12 >> Job Counters >> SLOTS_MILLIS_MAPS=4794 >> Total time spent by all reduces waiting after reserving slots (ms)=0 >> Total time spent by all maps waiting after reserving slots (ms)=0 >> Launched map tasks=1 >> SLOTS_MILLIS_REDUCES=0 >> FileSystemCounters >> HDFS_BYTES_READ=87 >> FILE_BYTES_WRITTEN=79795 >> HDFS_BYTES_WRITTEN=95560 >> Map-Reduce Framework >> Map input records=1000 >> Spilled Records=0 >> Map output records=1000 >> SPLIT_RAW_BYTES=87 >> >> So my thought is, to have a follow up java action, pass it the path to the >> log file, and have the java class extract the data I want, and update a >> results table. >> I also considered whether I could direct the sqoop action to output >> information to a properties file. Oozie does NOT like the <capture-output> >> tag in the sqoop action. >> Any help on this is appreciated. >> Thanks. >> >> >> Steve Charles >> Senior Consultant >> >> Sogeti USA >> 3636 N. Central Ave. >> #810 >> Phoenix, AZ 85012 >> www.us.sogeti.com <http://www.us.sogeti.com/> >> > >
