[jira] Commented: (PIG-1505) support jars and scripts in dfs
[
https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12900811#action_12900811
]
Richard Ding commented on PIG-1505:
---
The results of test-patch:
{code}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 3 new or
modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning
messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number
of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs
warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the
total number of release audit warnings.
{code}
I'll commit the patch after running unit tests.
support jars and scripts in dfs
---
Key: PIG-1505
URL: https://issues.apache.org/jira/browse/PIG-1505
Project: Pig
Issue Type: Improvement
Reporter: Andrew Hitchcock
Assignee: Andrew Hitchcock
Attachments: PIG-1505-4.patch, pig-jars-and-scripts-from-dfs-3.patch,
pig-jars-and-scripts-from-dfs-trunk-1.patch,
pig-jars-and-scripts-from-dfs-trunk-2.patch,
pig-jars-and-scripts-from-dfs-trunk.patch
Pig can't operate on files stored in Amazon S3.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1505) support jars and scripts in dfs
[
https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12891685#action_12891685
]
Richard Ding commented on PIG-1505:
---
You can take a look at the test cases in TestPigRunner where local Pig scripts
are passed to the PigRunner.run method.
You can first copy a local Pig script to the mini-cluster using
{code}
Util.copyFromLocalToCluster(cluster, localScriptFileName,
scriptFileNameOnCluster);
{code}
and then invoke run method with argument
{code}
String[] args = { -f, hdfs://scriptFileNameOnCluste };
PigRunner.run(args, null);
{code}
support jars and scripts in dfs
---
Key: PIG-1505
URL: https://issues.apache.org/jira/browse/PIG-1505
Project: Pig
Issue Type: Improvement
Reporter: Andrew Hitchcock
Assignee: Andrew Hitchcock
Attachments: pig-jars-and-scripts-from-dfs-3.patch,
pig-jars-and-scripts-from-dfs-trunk-1.patch,
pig-jars-and-scripts-from-dfs-trunk-2.patch,
pig-jars-and-scripts-from-dfs-trunk.patch
Pig can't operate on files stored in Amazon S3.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1505) support jars and scripts in dfs
[ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12891258#action_12891258 ] Alan Gates commented on PIG-1505: - I ran core and contrib tests manually and they both pass. Richard will be reviewing the patch. support jars and scripts in dfs --- Key: PIG-1505 URL: https://issues.apache.org/jira/browse/PIG-1505 Project: Pig Issue Type: Improvement Reporter: Andrew Hitchcock Assignee: Andrew Hitchcock Attachments: pig-jars-and-scripts-from-dfs-3.patch, pig-jars-and-scripts-from-dfs-trunk-1.patch, pig-jars-and-scripts-from-dfs-trunk-2.patch, pig-jars-and-scripts-from-dfs-trunk.patch Pig can't operate on files stored in Amazon S3. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1505) support jars and scripts in dfs
[
https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12891347#action_12891347
]
Richard Ding commented on PIG-1505:
---
Thank you for the update. A few more comments:
* According to Pig Latin manual, user can also register additional files (to
use with user's Pig script) via the command line using the
-Dpig.additional.jars option (in addition to the REGISTER statement inside a
Pig script). I suggest you call FileLocalizer.fetchFile from the shared method
PigServer.registerJar so both cases will be covered.
* Can you change the method signature to
{code}
public static FetchFileRet fetchFile(Properties properties, String filePath)
throws IOException
{code}
The reason is that we have deprecated all other public methods on FileLocalizer
which has DataStorage as a parameter (so we can deprecate DataStorage in the
future). I think this is safe since the condition in the method
{code}
((fileUri.getScheme() == null) (dfs == null))
{code}
is not used in the patch.
* You need to add a unit test in the patch (by first copying a Pig script to
the mini-cluster).
* Finally, since this is a new feature, can you add a release note (On jira,
there is a Release Note field) so that it will be incorporated in the next Pig
release notes.
support jars and scripts in dfs
---
Key: PIG-1505
URL: https://issues.apache.org/jira/browse/PIG-1505
Project: Pig
Issue Type: Improvement
Reporter: Andrew Hitchcock
Assignee: Andrew Hitchcock
Attachments: pig-jars-and-scripts-from-dfs-3.patch,
pig-jars-and-scripts-from-dfs-trunk-1.patch,
pig-jars-and-scripts-from-dfs-trunk-2.patch,
pig-jars-and-scripts-from-dfs-trunk.patch
Pig can't operate on files stored in Amazon S3.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1505) support jars and scripts in dfs
[ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12891436#action_12891436 ] Andrew Hitchcock commented on PIG-1505: --- Thanks Richard. Is there a unit test you recommend that I can model mine after? Something that uses the mini-cluster. support jars and scripts in dfs --- Key: PIG-1505 URL: https://issues.apache.org/jira/browse/PIG-1505 Project: Pig Issue Type: Improvement Reporter: Andrew Hitchcock Assignee: Andrew Hitchcock Attachments: pig-jars-and-scripts-from-dfs-3.patch, pig-jars-and-scripts-from-dfs-trunk-1.patch, pig-jars-and-scripts-from-dfs-trunk-2.patch, pig-jars-and-scripts-from-dfs-trunk.patch Pig can't operate on files stored in Amazon S3. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1505) support jars and scripts in dfs
[ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12890913#action_12890913 ] Andrew Hitchcock commented on PIG-1505: --- I think the test failures were unrelated. Can you confirm Alan? support jars and scripts in dfs --- Key: PIG-1505 URL: https://issues.apache.org/jira/browse/PIG-1505 Project: Pig Issue Type: Improvement Reporter: Andrew Hitchcock Assignee: Andrew Hitchcock Attachments: pig-jars-and-scripts-from-dfs-trunk-1.patch, pig-jars-and-scripts-from-dfs-trunk-2.patch, pig-jars-and-scripts-from-dfs-trunk.patch Pig can't operate on files stored in Amazon S3. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1505) support jars and scripts in dfs
[ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12890954#action_12890954 ] Richard Ding commented on PIG-1505: --- The patch looks good. A couple of comments: * The property pig.initial.fs.name is set, but I don't see where it's used. * Pig is prepared to work in the multi-name-node environment (HDFS-1052). It seems that FileLocalizer.fetchFile method doesn't need to use the default dfs. I also looked into the failed tests and they were unrelated to the patch. support jars and scripts in dfs --- Key: PIG-1505 URL: https://issues.apache.org/jira/browse/PIG-1505 Project: Pig Issue Type: Improvement Reporter: Andrew Hitchcock Assignee: Andrew Hitchcock Attachments: pig-jars-and-scripts-from-dfs-trunk-1.patch, pig-jars-and-scripts-from-dfs-trunk-2.patch, pig-jars-and-scripts-from-dfs-trunk.patch Pig can't operate on files stored in Amazon S3. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1505) support jars and scripts in dfs
[ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12890964#action_12890964 ] Andrew Hitchcock commented on PIG-1505: --- Thanks Richard. Regarding your points. * You are correct, that snuck in from another patch I am working on. * I'm not sure I understand what you mean on this one. What should I change? support jars and scripts in dfs --- Key: PIG-1505 URL: https://issues.apache.org/jira/browse/PIG-1505 Project: Pig Issue Type: Improvement Reporter: Andrew Hitchcock Assignee: Andrew Hitchcock Attachments: pig-jars-and-scripts-from-dfs-trunk-1.patch, pig-jars-and-scripts-from-dfs-trunk-2.patch, pig-jars-and-scripts-from-dfs-trunk.patch Pig can't operate on files stored in Amazon S3. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1505) support jars and scripts in dfs
[
https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12890977#action_12890977
]
Richard Ding commented on PIG-1505:
---
The line
{code}
if (dfs != null){
src = new Path(dfs.asElement(filePath).toString());
}
{code}
seems unnecessary.
Do you allow user to register a jar with relative path?
support jars and scripts in dfs
---
Key: PIG-1505
URL: https://issues.apache.org/jira/browse/PIG-1505
Project: Pig
Issue Type: Improvement
Reporter: Andrew Hitchcock
Assignee: Andrew Hitchcock
Attachments: pig-jars-and-scripts-from-dfs-trunk-1.patch,
pig-jars-and-scripts-from-dfs-trunk-2.patch,
pig-jars-and-scripts-from-dfs-trunk.patch
Pig can't operate on files stored in Amazon S3.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1505) support jars and scripts in dfs
[ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12890985#action_12890985 ] Andrew Hitchcock commented on PIG-1505: --- By default, the relative path is relative to your local file system directory. If you set pig.jars.relative.to.dfs to true then you can register jars that are relative to your current grunt working directory, even within HDFS. I'll post a patch in a couple of minutes that address your issues. support jars and scripts in dfs --- Key: PIG-1505 URL: https://issues.apache.org/jira/browse/PIG-1505 Project: Pig Issue Type: Improvement Reporter: Andrew Hitchcock Assignee: Andrew Hitchcock Attachments: pig-jars-and-scripts-from-dfs-trunk-1.patch, pig-jars-and-scripts-from-dfs-trunk-2.patch, pig-jars-and-scripts-from-dfs-trunk.patch Pig can't operate on files stored in Amazon S3. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1505) support jars and scripts in dfs
[ https://issues.apache.org/jira/browse/PIG-1505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12889421#action_12889421 ] Hadoop QA commented on PIG-1505: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449736/pig-jars-and-scripts-from-dfs-trunk.patch against trunk revision 964182. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/345/console This message is automatically generated. support jars and scripts in dfs --- Key: PIG-1505 URL: https://issues.apache.org/jira/browse/PIG-1505 Project: Pig Issue Type: Improvement Reporter: Andrew Hitchcock Attachments: pig-jars-and-scripts-from-dfs-trunk.patch Pig can't operate on files stored in Amazon S3. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
