[
https://issues.apache.org/jira/browse/PIG-2745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13296047#comment-13296047
]
Rohini Palaniswamy commented on PIG-2745:
-----------------------------------------
+1. Tested this patch with relative path and absolute path for 20.205 and 23.
Works fine.
Daniel,
Can you include this in 0.10 also. Without this scripting udfs do not work
in 23 for both relative and absolute path. e2e tests currently marked ignored
for MAPREDUCE-3700 also will have to be enabled again.
> Pig e2e test RubyUDFs fails in MR mode when running from tarball
> ----------------------------------------------------------------
>
> Key: PIG-2745
> URL: https://issues.apache.org/jira/browse/PIG-2745
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.10.1
> Reporter: Cheolsoo Park
> Attachments: PIG-2745-2.patch, PIG-2745.patch
>
>
> To reproduce the issue, please run the e2e test "RubyUDFs_1" in MR mode from
> the tarball (not from installed Pig - please see why below). Either
> pseudo-distributed-mode or full-mode Hadoop can be used.
> {code}
> ant -Dhadoopversion=23 -Dharness.old.pig=`pwd`
> -Dharness.cluster.conf=/etc/hadoop/conf/
> -Dharness.cluster.bin=/usr/lib/hadoop/bin/hadoop test-e2e -Dtests.to.run="-t
> RubyUDFs_1"
> {code}
> The test fails with the following error:
> {code}
> java.lang.IllegalStateException: Could not initialize interpreter (from file
> system or classpath) with
> /home/cheolsoo/pig-0.10/test/e2e/pig/testdist/libexec/ruby/scriptingudfs.rb
> {code}
> Looking at the job jar generated by Pig, "scriptingudfs.rb" can be found as
> follows:
> {code}
> [cheolsoo@c1405 pig-cheolsoo]$ jar tvf bad.jar | grep scriptingudfs.rb
> 2491 Fri Jun 08 15:52:08 PDT 2012
> /home/cheolsoo/pig-0.10/test/e2e/pig/testdist/libexec/ruby/scriptingudfs.rb
> {code}
> Looking at getScriptAsStream() method in ScriptEngine.java,
> "scriptingudfs.rb" is supposed to be read from the job jar, but it is not.
> The reason is because getResourceAsStream("/x") looks for "x" (without the
> leading "/") not "/x". Since "scriptingudfs.rb" is stored with it absolute
> path, it ends up being not found by getResourceAsStream(scriptPath).
> {code}
> File file = new File(scriptPath);
> if (file.exists()) {
> try {
> is = new FileInputStream(file);
> } catch (FileNotFoundException e) {
> throw new IllegalStateException("could not find existing file
> "+scriptPath, e);
> }
> } else {
> if (file.isAbsolute()) {
> is = ScriptEngine.class.getResourceAsStream(scriptPath);
> } else {
> is = ScriptEngine.class.getResourceAsStream("/" + scriptPath);
> }
> }
> {code}
> In fact, the test passes if you run in local mode or from installed Pig. The
> reason is because "scriptingudfs.rb" is found in local file system (e.g
> /usr/share/pig/test/e2e/pig/udfs/ruby/scriptingudfs.rb).
> The fix seems straightforward. Attached is the patch that removes the leading
> "/" when registering UDF scripts so that they are stored without the leading
> "/" in the job jar as follows:
> {code}
> [cheolsoo@c1405 pig-cheolsoo]$ jar tvf good.jar | grep scriptingudfs.rb
> 2491 Fri Jun 08 15:52:08 PDT 2012
> home/cheolsoo/pig-0.10/test/e2e/pig/testdist/libexec/ruby/scriptingudfs.rb
> {code}
> Thanks!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
