Hi, It worked! My outputschema was wrong. Also, I realized that any other error in the Python code will also result in this error.
Learnings: 1. Include path to jython.jar in classpath in bashrc, or put jython.jar is in your classpath. 2. Give exact path to UDF in register statement 3. Put Make sure UDF is in your classpath. If everything is all right, this should work: register '/absolute/path/udf.py' using jython as thisisaudf; Thanks -----Original Message----- From: Jonathan Coveney [mailto:jcove...@gmail.com] Sent: Friday, January 07, 2011 10:48 PM To: user@pig.apache.org Subject: Re: Python UDF Internal Error when Registering Given that I'm not really sure what is going wrong, here is what I would stay to begin with (I don't know if you copy pasted directly or what so forgive me if I'm nitpicking something that isn't an actual issue): 1) you forgot a ":" at the end of your function, and you need #!/usr/bin/python (or the env version, whatever applies to you 2) your outputSchema looks wrong, it should just be @outputSchema("t:(field1:chararray, field2:chararray)"). You are not doing an operation that yields a bag, just a touple. 3) udf might be a reserved word, I'd name it something like what they do in their examples, myudf or whatever. I had a lot of problems getting my first python udf to work, but now that it does it has been fine. Hopefully we can help you get ot the bottom of it for reference, what works for me in the script is: register '/absolute/path/udf.py' using jython as thisisaudf; I have my classpath pointing to udf.py and jython.jar. So I think the issue might just be in how it's all linking together. 2011/1/7 <deepak....@wipro.com> > Hi, > > My bashrc file contains $PIG_CLASSPATH that points to the absolute > path of jython.jar Global classpath $CLASSPATH in bashrc also does > similarly. > > I have also put the UDFs in the classpath. > > I have tried these queries > > Register 'udf.py' USING jython AS udf > Register 'exact/path/to/udf.py' USING jython AS udf Register > 'exact/path/to/udf.py' USING jython.jar AS udf Register > 'exact/path/to/udf.py' USING 'exact/path/to/jython.jar' AS udf > Register 'exact/path/to/udf.py' USING exact/path/to/jython.jar AS udf > REGISTER 'udf.py' USING jython AS udf REGISTER 'exact/path/to/udf.py' > USING jython AS udf REGISTER 'exact/path/to/udf.py' USING jython.jar > AS udf REGISTER 'exact/path/to/udf.py' USING > 'exact/path/to/jython.jar' AS udf REGISTER 'exact/path/to/udf.py' > USING exact/path/to/jython.jar AS udf > > All of them are giving the same errors. > > Here's a very basic script that I'm trying to run: > > ------- > > REGISTER 'exact/path/to/udf.py' USING jython AS udf; > > records = LOAD 'path/to/data' AS (input_line:chararray); > > schema_records = FOREACH records GENERATE > udf.split_into_words(input_line); > > DUMP schema_records; > > ----------- > > Here's the python udf: > > @outputSchema("x:{t:(field1:chararray, field1:chararray)}") def > split_into_words(input_line) > line = input_line.strip() > words = line.split() > return (words[0], words[1]) > > ----------- > > Am I missing something here? > > -----Original Message----- > From: Jonathan Coveney [mailto:jcove...@gmail.com] > Sent: Friday, January 07, 2011 10:03 PM > To: user@pig.apache.org > Subject: Re: Python UDF Internal Error when Registering > > You need to point exactly to jython.jar. so /path/to/jython.jar has to > be there. The same is true of the UDF (you have to point to the exact file). > > 2011/1/7 <deepak....@wipro.com> > > > Oh Okay. I thought Jython comes bundled with PIG 0.8. > > > > Anyway, I downloaded Jython_Installer-x.xx.jar and unpacked it in > > the PIG Classpath. Also kept my UDF in classpath as suggested. But I > > am still getting the exact same error. > > > > Any more ideas? > > > > -----Original Message----- > > From: Jonathan Coveney [mailto:jcove...@gmail.com] > > Sent: Friday, January 07, 2011 8:04 PM > > To: user@pig.apache.org > > Subject: Re: Python UDF Internal Error when Registering > > > > Make sure that in the register, you use the absolute path. Also make > > sure that both the jython.jar and udf asre in your classpath. A good > > test is to do that, then try and register in local mode. If that > > works, then try on your cluster. Always making sure that your > > classpath has those details > > > > Sent via BlackBerry > > > > -----Original Message----- > > From: <deepak....@wipro.com> > > Date: Fri, 7 Jan 2011 19:57:16 > > To: <user@pig.apache.org> > > Reply-To: user@pig.apache.org > > Subject: Python UDF Internal Error when Registering > > > > Hi, > > > > I get an error when I try to register my python udf. Why is this > happening? > > > > grunt> Register 'udf.py' USING jython AS udf > > 2011-01-07 19:39:31,818 [main] ERROR > > org.apache.pig.tools.grunt.Grunt > > - ERROR 2998: Unhandled internal error. > > org/python/util/PythonInterpreter > > > > The Logs say the following: > > > > Pig Stack Trace > > --------------- > > ERROR 2998: Unhandled internal error. > > org/python/util/PythonInterpreter > > > > java.lang.NoClassDefFoundError: org/python/util/PythonInterpreter > > at > > > org.apache.pig.scripting.jython.JythonScriptEngine$Interpreter.<clinit > >(JythonScriptEngine.java:54) > > at > > > org.apache.pig.scripting.jython.JythonScriptEngine.registerFunctions(J > ythonScriptEngine.java:105) > > at org.apache.pig.PigServer.registerCode(PigServer.java:507) > > at > > > org.apache.pig.tools.grunt.GruntParser.processRegister(GruntParser.jav > a:381) > > at > > > org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptP > arser.java:421) > > at > > > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.ja > va:165) > > at > > > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.ja > va:141) > > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90) > > at org.apache.pig.Main.run(Main.java:510) > > at org.apache.pig.Main.main(Main.java:107) > > Caused by: java.lang.ClassNotFoundException: > > org.python.util.PythonInterpreter > > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > > at java.security.AccessController.doPrivileged(Native Method) > > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:307) > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:248) > > ... 10 more > > > > ==================================================================== > > == > > ========== > > Any ideas? > > > > Please do not print this email unless it is absolutely necessary. > > > > The information contained in this electronic message and any > > attachments to this message are intended for the exclusive use of > > the > > addressee(s) and may contain proprietary, confidential or privileged > > information. If you are not the intended recipient, you should not > > disseminate, distribute or copy this e-mail. Please notify the > > sender immediately and destroy all copies of this message and any > > attachments. > > > > WARNING: Computer viruses can be transmitted via email. The > > recipient should check this email and any attachments for the > > presence of viruses. The company accepts no liability for any damage > > caused by any virus transmitted by this email. > > > > www.wipro.com > > > > Please do not print this email unless it is absolutely necessary. > > The information contained in this electronic message and any > attachments to this message are intended for the exclusive use of the > addressee(s) and may contain proprietary, confidential or privileged > information. If you are not the intended recipient, you should not > disseminate, distribute or copy this e-mail. Please notify the sender > immediately and destroy all copies of this message and any attachments. > > WARNING: Computer viruses can be transmitted via email. The recipient > should check this email and any attachments for the presence of > viruses. The company accepts no liability for any damage caused by any > virus transmitted by this email. > > www.wipro.com > Please do not print this email unless it is absolutely necessary. The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com
