Hi I'm trying to run pig script with builtin macros like count/sum/avg ... (org.apache.pig.builtin) But on every method I trued I get the same ERROR
ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1200: <file /tmp/test.pig, line 3> Cannot expand macro 'SUM'. Reason: Macro must be defined before expansion I understood that macro cant be used from grunt ( http://www.mail-archive.com/[email protected]/msg04926.html) So how can I use the org.apache.pig.builtin macro's I tried to run script using following methods - Command line : pig [script file path] - PigRunner : java MyPigRunner.java [script file path] - PigServer : java MyPigServer.java [script file path] But for all I get the expension error, what is the way to run pig script with builtin macros I'm using followng pig pacakge : pig-0.10.0+61-1.cdh4.1.3.p0.23.el6.noarch *Pig script:* *log = LOAD '/tmp/data' AS (value:long) ;* *total = SUM(log);* *DUMP log;* * * PigRunner.java: *public class MyPigRunner {* * * * * *static public void main(String [] args) {* * * * org.apache.pig.PigRunner.run(args, null);* * * *}* * * *}* PigServer.java: *public class MyPigServer {* * * * * *static public void main (String[] args) throws IOException{* * * * String filename = args[0];* * PigServer pig = new PigServer(ExecType.LOCAL); * * pig.registerScript(filename);* * pig.store("Result", "/tmp/pig.result");* * * *}* * * *}* *Details at logfile: /usr/lib/pig/bin/pig_1365300225744.log 0_31/lib/tools.jar:/usr/lib/hbase/bin/..:/usr/lib/hbase/ bin/../hbase-0.92.1-cdh4.1.3-security.jar:/usr/lib/hbase/ bin/../hbase-0.92.1-cdh4.1.3-security-tests.jar:/usr/lib/ hbase/bin/../hbase.jar:/usr/lib/hbase/bin/../lib/ activation-1.1.jar:/usr/lib/hbase/bin/../lib/aopalliance- 1.0.jar:/usr/lib/hbase/bin/../lib/asm-3.2.jar:/usr/lib/ hbase/bin/../lib/avro-1.7.1.cloudera.2.jar:/usr/lib/hbase/ bin/../lib/commons-beanutils-1.7.0.jar:/usr/lib/hbase/bin/. ./lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hbase/bin/. ./lib/commons-cli-1.2.jar:/usr/lib/hbase/bin/../lib/ commons-codec-1.4.jar:/usr/lib/hbase/bin/../lib/commons- collections-3.2.1.jar:/usr/lib/hbase/bin/../lib/commons- configuration-1.6.jar:/usr/lib/hbase/bin/../lib/commons- daemon-1.0.3.jar:/usr/lib/hbase/bin/../lib/commons- digester-1.8.jar:/usr/lib/hbase/bin/../lib/commons-el-1. 0.jar:/usr/lib/hbase/bin/../lib/commons-httpclient-3.1. jar:/usr/lib/hbase/bin/../lib/commons-io-2.1.jar:/usr/lib/ hbase/bin/../lib/commons-lang-2.5.jar:/usr/lib/hbase/bin/../ lib/commons-logging-1.1.1.jar:/usr/lib/hbase/bin/../lib/ commons-net-3.1.jar:/usr/lib/hbase/bin/../lib/core-3.1.1. jar:/usr/lib/hbase/bin/../lib/gmbal-api-only-3.0.0-b023.jar: /usr/lib/hbase/bin/../lib/grizzly-framework-2.1.1.jar:/ usr/lib/hbase/bin/../lib/grizzly-framework-2.1.1-tests. jar:/usr/lib/hbase/bin/../lib/grizzly-http-2.1.1.jar:/usr/ lib/hbase/bin/../lib/grizzly-http-server-2.1.1.jar:/usr/ lib/hbase/bin/../lib/grizzly-http-servlet-2.1.1.jar:/usr/ lib/hbase/bin/../lib/grizzly-rcm-2.1.1.jar:/usr/lib/hbase/ bin/../lib/guava-11.0.2.jar:/usr/lib/hbase/bin/../lib/ guice-3.0.jar:/usr/lib/hbase/bin/../lib/guice-servlet-3.0. jar:/usr/lib/hbase/bin/../lib/high-scale-lib-1.1.1.jar:/usr/ lib/hbase/bin/../lib/httpclient-4.0.1.jar:/usr/lib/ hbase/bin/../lib/httpcore-4.0.1.jar:/usr/lib/hbase/bin/../ lib/jackson-core-asl-1.8.8.jar:/usr/lib/hbase/bin/../lib/ jackson-jaxrs-1.8.8.jar:/usr/lib/hbase/bin/../lib/jackson- mapper-asl-1.8.8.jar:/usr/lib/hbase/bin/../lib/jackson-xc-1. 8.8.jar:/usr/lib/hbase/bin/../lib/jamon-runtime-2.3.1.jar:/ usr/lib/hbase/bin/../lib/jasper-compiler-5.5.23.jar:/ usr/lib/hbase/bin/../lib/jasper-runtime-5.5.23.jar:/ usr/lib/hbase/bin/../lib/javax.inject-1.jar:/usr/lib/hbase/bin/../lib/javax. servlet-3.0.jar:/usr/lib/hbase/bin/../lib/jaxb-api-2.1. jar:/usr/lib/hbase/bin/../lib/jaxb-impl-2.2.3-1.jar:/usr/ lib/hbase/bin/../lib/jersey-client-1.8.jar:/usr/lib/hbase/ bin/../lib/jersey-core-1.8.jar:/usr/lib/hbase/bin/../lib/ jersey-grizzly2-1.8.jar:/usr/lib/hbase/bin/../lib/jersey- guice-1.8.jar:/usr/lib/hbase/bin/../lib/jersey-json-1.8. jar:/usr/lib/hbase/bin/../lib/jersey-server-1.8.jar:/usr/ lib/hbase/bin/../lib/jersey-test-framework-core-1.8.jar:/ usr/lib/hbase/bin/../lib/jersey-test-framework-grizzly2-1.8.jar:/usr/lib/ hbase/bin/../lib/jets3t-0.6.1.jar:/usr/lib/hbase/bin/../lib/ jettison-1.1.jar:/usr/lib/hbase/bin/../lib/jetty-6.1.26. cloudera.2.jar:/usr/lib/hbase/bin/../lib/jetty-util-6.1.26. cloudera.2.jar:/usr/lib/hbase/bin/../lib/jruby-complete-1.6. 5.jar:/usr/lib/hbase/bin/../lib/jsch-0.1.42.jar:/usr/lib/ hbase/bin/../lib/jsp-2.1-6.1.14.jar:/usr/lib/hbase/bin/../ lib/jsp-api-2.1-6.1.14.jar:/usr/lib/hbase/bin/../lib/jsp- api-2.1.jar:/usr/lib/hbase/bin/../lib/jsr305-1.3.9.jar:/ usr/lib/hbase/bin/../lib/kfs-0.3.jar:/usr/lib/hbase/bin/../ lib/libthrift-0.7.0.jar:/usr/lib/hbase/bin/../lib/log4j-1. 2.17.jar:/usr/lib/hbase/bin/../lib/management-api-3.0.0- b012.jar:/usr/lib/hbase/bin/../lib/metrics-core-2.1.2.jar:/ usr/lib/hbase/bin/../lib/netty-3.2.4.Final.jar:/usr/lib/hbase/bin/../lib/ paranamer-2.3.jar:/usr/lib/hbase/bin/../lib/protobuf- java-2.4.0a.jar:/usr/lib/hbase/bin/../lib/servlet-api- 2.5-6.1.14.jar:/usr/lib/hbase/bin/../lib/servlet-api-2.5. jar:/usr/lib/hbase/bin/../lib/slf4j-api-1.6.1.jar:/usr/lib/ hbase/bin/../lib/snappy-java-1.0.4.1.jar:/usr/lib/hbase/ bin/../lib/stax-api-1.0.1.jar:/usr/lib/hbase/bin/../lib/ xmlenc-0.52.jar:/usr/lib/hbase/bin/../lib/zookeeper. jar:/etc/hadoop/conf:/*:/lib/*:/usr/lib/zookeeper/*:/usr/ lib/zookeeper/lib/*::/etc/hadoop/conf:/usr/lib/hadoop/ lib/*:/usr/lib/hadoop/.//*:/usr/lib/hadoop-hdfs/./:/usr/ lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-hdfs/.//*:/usr/lib/ hadoop-yarn/lib/*:/usr/lib/hadoop-yarn/.//*:/usr/lib/ hadoop-mapreduce/lib/*:/usr/lib/hadoop-mapreduce/.//*:/ usr/lib/pig/bin/../lib/automaton.jar:/usr/lib/pig/bin/../lib/avro-1.7.1. cloudera.2.jar:/usr/lib/pig/bin/../lib/jackson-core-asl-1. 8.8.jar:/usr/lib/pig/bin/../lib/jackson-mapper-asl-1.8.8. jar:/usr/lib/pig/bin/../lib/json-simple-1.1.jar:/usr/lib/ pig/bin/../lib/jython-2.5.0.jar:/usr/lib/pig/bin/../lib/ snappy-java-1.0.4.1.jar:/usr/lib/pig/bin/../pig.jar:/usr/lib/pig/bin/MyPigServer.jar' org.apache.pig.Main /tmp/test.pig -x mapreduce* *2013-04-06 22:03:46,009 [main] INFO org.apache.pig.Main - Apache Pig version 0.10.0-cdh4.1.3 (rexported) compiled Jan 26 2013, 17:36:42 2013-04-06 22:03:46,013 [main] INFO org.apache.pig.Main - Logging error messages to: /usr/lib/pig/bin/pig_1365300225744.log 2013-04-06 22:03:49,344 [main] INFO org.apache.pig.backend.hadoop. executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://127.0.0.1:8020 2013-04-06 22:03:53,110 [main] DEBUG org.apache.pig.PigServer - Create a new graph. 2013-04-06 22:03:54,036 [main] DEBUG org.apache.pig.parser.QueryParserDriver - Original macro AST: (QUERY (STATEMENT log (LOAD '/tmp/data' (AS (FIELD_DEF value))))) 2013-04-06 22:03:54,036 [main] DEBUG org.apache.pig.parser.QueryParserDriver - macro AST after import: (QUERY (STATEMENT log (LOAD '/tmp/data' (AS (FIELD_DEF value))))) 2013-04-06 22:03:54,036 [main] DEBUG org.apache.pig.parser.QueryParserDriver - Resulting macro AST: (QUERY (STATEMENT log (LOAD '/tmp/data' (AS (FIELD_DEF value))))) 2013-04-06 22:03:54,867 [main] DEBUG org.apache.pig.impl. logicalLayer.schema.Schema$FieldSchema - t: 50 Bag: 120 tuple: 110 2013-04-06 22:03:54,882 [main] DEBUG org.apache.pig.parser.QueryParserDriver - Original macro AST: (QUERY (STATEMENT log (LOAD '/tmp/data' (AS (FIELD_DEF value)))) (MACRO_INLINE SUM (RETURN_VAL total) (PARAMS log))) 2013-04-06 22:03:54,882 [main] DEBUG org.apache.pig.parser.QueryParserDriver - macro AST after import: (QUERY (STATEMENT log (LOAD '/tmp/data' (AS (FIELD_DEF value)))) (MACRO_INLINE SUM (RETURN_VAL total) (PARAMS log))) 2013-04-06 22:03:54,889 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1200: <file /tmp/test.pig, line 3> Cannot expand macro 'SUM'. Reason: Macro must be defined before expansion. Details at logfile: /usr/lib/pig/bin/pig_1365300225744.log*
