Illustrate code is not as stable as the main code. Please open a Jira ticket for that.
Thanks, Daniel On Thu, Jan 12, 2012 at 9:25 PM, Dan Young <[email protected]> wrote: > Hello All, > > New to Pig and I'm playing around with using a Javascript UDF to parse some > Google Adwords reports and am having some issues with Illustrate. I always > get a java.lang.NullPointerException when I issue an Illustrate, although > when I do a dump everything seems to work properly. Below is an example > run and I've gist the udf.js and log here: > > https://gist.github.com/f472b92b3e048599aa8c > > > Any ideas on why I'm getting the NPE and what I might be doing wrong? > > dans-MacBook-Pro:pig danoyoung$ pig -x local > 2012-01-12 22:16:57,131 [main] INFO org.apache.pig.Main - Logging error > messages to: /Users/danoyoung/projects/pig/pig_1326431817128.log > 2012-01-12 22:16:57,386 [main] INFO > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - > Connecting to hadoop file system at: file:/// > 2012-01-12 22:16:57.483 java[27809:1903] Unable to load realm info from > SCDynamicStore > grunt> set io.sort.mb 500; > grunt> register '/Users/danoyoung/projects/pig/udf/udf.js' using > org.apache.pig.scripting.js.JsScriptEngine as myfuncs; > 2012-01-12 22:17:24,803 [main] INFO > org.apache.pig.scripting.js.JsScriptEngine - Register scripting UDF: > get_date_marker > 2012-01-12 22:17:24,804 [main] INFO > org.apache.pig.scripting.js.JsScriptEngine - Register scripting UDF: > get_record > grunt> register '/usr/local/pig/piggybank.jar'; > grunt> > grunt> a = LOAD > '/Users/danoyoung/Downloads/adwords_KeywordDailyReport_1-1-2012_daily' > USING org.apache.pig.piggybank.storage.XMLLoader('row') AS (doc:chararray); > grunt> > grunt> b = FOREACH a GENERATE FLATTEN(myfuncs.get_record(doc)) AS > > (dw_date_marker:int,ad_network_ad_group_key:long,ad_network_keyword_key:long,firstpagecpc:int,qualityscore:int,cost:float,position:float); > grunt> describe b; > b: {dw_date_marker: int,ad_network_ad_group_key: > long,ad_network_keyword_key: long,firstpagecpc: int,qualityscore: int,cost: > float,position: float} > grunt> > grunt> illustrate b; > 2012-01-12 22:17:25,642 [main] INFO > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - > Connecting to hadoop file system at: file:/// > 2012-01-12 22:17:25,846 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - > File concatenation threshold: 100 optimistic? false > 2012-01-12 22:17:25,860 [main] INFO > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer > - MR plan size before optimization: 1 > 2012-01-12 22:17:25,860 [main] INFO > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer > - MR plan size after optimization: 1 > 2012-01-12 22:17:25,870 [main] INFO > org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added > to the job > 2012-01-12 22:17:25,887 [main] INFO > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler > - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3 > 2012-01-12 22:17:26,161 [main] INFO > org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths > to process : 1 > 2012-01-12 22:17:26,721 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - > File concatenation threshold: 100 optimistic? false > 2012-01-12 22:17:26,723 [main] INFO > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer > - MR plan size before optimization: 1 > 2012-01-12 22:17:26,723 [main] INFO > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer > - MR plan size after optimization: 1 > 2012-01-12 22:17:26,724 [main] INFO > org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added > to the job > 2012-01-12 22:17:26,725 [main] INFO > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler > - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3 > java.lang.NullPointerException > at org.apache.pig.scripting.js.JsFunction.exec(JsFunction.java:200) > at > > org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:225) > at > > org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:262) > at > > org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:334) > at > > org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:332) > at > > org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:284) > at > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:271) > at > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:266) > at > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) > at > > org.apache.pig.pen.LocalMapReduceSimulator.launchPig(LocalMapReduceSimulator.java:194) > at org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:257) > at org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:238) > at > > org.apache.pig.pen.LineageTrimmingVisitor.init(LineageTrimmingVisitor.java:103) > at > > org.apache.pig.pen.LineageTrimmingVisitor.<init>(LineageTrimmingVisitor.java:98) > at > org.apache.pig.pen.ExampleGenerator.getExamples(ExampleGenerator.java:166) > at org.apache.pig.PigServer.getExamples(PigServer.java:1202) > at > > org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:698) > at > > org.apache.pig.tools.pigscript.parser.PigScriptParser.Illustrate(PigScriptParser.java:591) > at > > org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:306) > at > > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188) > at > > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164) > at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69) > at org.apache.pig.Main.run(Main.java:523) > at org.apache.pig.Main.main(Main.java:148) > 2012-01-12 22:17:26,767 [main] ERROR org.apache.pig.tools.grunt.Grunt - > ERROR 2997: Encountered IOException. Exception : null > Details at logfile: /Users/danoyoung/projects/pig/pig_1326431817128.log > grunt> > > > > Regards, > > Dan >
