[jira] Commented: (PIG-909) Allow Pig executable to use hadoop jars not bundled with pig
[
https://issues.apache.org/jira/browse/PIG-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12887476#action_12887476
]
Olga Natkovich commented on PIG-909:
Did this actually get checked in? Should this be resurrected for Pig 0.8.0 or
closed?
Allow Pig executable to use hadoop jars not bundled with pig
Key: PIG-909
URL: https://issues.apache.org/jira/browse/PIG-909
Project: Pig
Issue Type: Improvement
Reporter: Dmitriy V. Ryaboy
Assignee: Dmitriy V. Ryaboy
Priority: Minor
Attachments: pig_909.patch
The current pig executable (bin/pig) looks for a file named
hadoop${PIG_HADOOP_VERSION}.jar that comes bundled with Pig.
The proposed change will allow Pig to look in $HADOOP_HOME for the hadoop
jars, if that variable is set.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-909) Allow Pig executable to use hadoop jars not bundled with pig
[
https://issues.apache.org/jira/browse/PIG-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12739351#action_12739351
]
Daniel Dai commented on PIG-909:
Hi, Dmitriy
Sorry if my previous comments confuse you. What I mean is pig script is
generating the following command line:
java -Xmx1000m -Dpig.log.dir=$PIG_HOME/bin/../logs -Dpig.log.file=pig.log
-Dpig.home.dir=$PIG_HOME/bin/.. -Dpig.root.logger=INFO,console,DRFA -classpath
$PIG_HOME/bin/../conf:/usr/java/default//lib/tools.jar:$PIG_HOME/bin/../build/classes:$PIG_HOME/bin/../build/test/classes:$PIG_HOME/bin/../pig-*-core.jar:$PIG_HOME/bin/../build/pig-*-core.jar:$PIG_HOME/bin/../lib/hadoop18.jar:$PIG_HOME/bin/../lib/hbase-0.18.1.jar:$PIG_HOME/bin/../lib/hbase-0.18.1-test.jar
org.apache.pig.Main
-classpath does not include all libraries required by Pig, so just run the
scripts will generate error. It does not related to your patch.
Allow Pig executable to use hadoop jars not bundled with pig
Key: PIG-909
URL: https://issues.apache.org/jira/browse/PIG-909
Project: Pig
Issue Type: Improvement
Reporter: Dmitriy V. Ryaboy
Priority: Minor
Attachments: pig_909.patch
The current pig executable (bin/pig) looks for a file named
hadoop${PIG_HADOOP_VERSION}.jar that comes bundled with Pig.
The proposed change will allow Pig to look in $HADOOP_HOME for the hadoop
jars, if that variable is set.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-909) Allow Pig executable to use hadoop jars not bundled with pig
[
https://issues.apache.org/jira/browse/PIG-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12739570#action_12739570
]
Dmitriy V. Ryaboy commented on PIG-909:
---
Sorry I am being slow -- which libraries are missing from the classpath you
posted?
Allow Pig executable to use hadoop jars not bundled with pig
Key: PIG-909
URL: https://issues.apache.org/jira/browse/PIG-909
Project: Pig
Issue Type: Improvement
Reporter: Dmitriy V. Ryaboy
Priority: Minor
Attachments: pig_909.patch
The current pig executable (bin/pig) looks for a file named
hadoop${PIG_HADOOP_VERSION}.jar that comes bundled with Pig.
The proposed change will allow Pig to look in $HADOOP_HOME for the hadoop
jars, if that variable is set.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-909) Allow Pig executable to use hadoop jars not bundled with pig
[
https://issues.apache.org/jira/browse/PIG-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12739643#action_12739643
]
Dmitriy V. Ryaboy commented on PIG-909:
---
Oh I see.
I have this in my bashrc:
export PIG_CLASSPATH=$PIGDIR/pig.jar
I thought this was included in a README somewhere. I guess we can modify
bin/pig to use this as a default value (so a user can still override by setting
PIG_CLASSPATH to something else).
Allow Pig executable to use hadoop jars not bundled with pig
Key: PIG-909
URL: https://issues.apache.org/jira/browse/PIG-909
Project: Pig
Issue Type: Improvement
Reporter: Dmitriy V. Ryaboy
Priority: Minor
Attachments: pig_909.patch
The current pig executable (bin/pig) looks for a file named
hadoop${PIG_HADOOP_VERSION}.jar that comes bundled with Pig.
The proposed change will allow Pig to look in $HADOOP_HOME for the hadoop
jars, if that variable is set.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-909) Allow Pig executable to use hadoop jars not bundled with pig
[
https://issues.apache.org/jira/browse/PIG-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12739244#action_12739244
]
Daniel Dai commented on PIG-909:
Seems like bin/pig is broken for a while. Some libraries have been moved to
build/ivy/lib/Pig, and pig script does not take care of it correctly.
Allow Pig executable to use hadoop jars not bundled with pig
Key: PIG-909
URL: https://issues.apache.org/jira/browse/PIG-909
Project: Pig
Issue Type: Improvement
Reporter: Dmitriy V. Ryaboy
Priority: Minor
Attachments: pig_909.patch
The current pig executable (bin/pig) looks for a file named
hadoop${PIG_HADOOP_VERSION}.jar that comes bundled with Pig.
The proposed change will allow Pig to look in $HADOOP_HOME for the hadoop
jars, if that variable is set.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-909) Allow Pig executable to use hadoop jars not bundled with pig
[
https://issues.apache.org/jira/browse/PIG-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12739269#action_12739269
]
Daniel Dai commented on PIG-909:
Hi, Dmitriy,
One problem is that hadoop.jar comes with pig actually bundles lots of external
libraries needed by hadoop such as log4j, common-logging. If we skip hadoop.jar
and use external one, we miss all those libraries. Can we try this? If we have
external hadoop.jar, put it in front of pig.jar in classpath. So java will pick
classes in external hadoop.jar first.
Allow Pig executable to use hadoop jars not bundled with pig
Key: PIG-909
URL: https://issues.apache.org/jira/browse/PIG-909
Project: Pig
Issue Type: Improvement
Reporter: Dmitriy V. Ryaboy
Priority: Minor
Attachments: pig_909.2.patch, pig_909.patch
The current pig executable (bin/pig) looks for a file named
hadoop${PIG_HADOOP_VERSION}.jar that comes bundled with Pig.
The proposed change will allow Pig to look in $HADOOP_HOME for the hadoop
jars, if that variable is set.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-909) Allow Pig executable to use hadoop jars not bundled with pig
[
https://issues.apache.org/jira/browse/PIG-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12739282#action_12739282
]
Daniel Dai commented on PIG-909:
Yes, Dmitriy, you said it. However, if we do not have external hadoop, pig
script do not currently work. We need to fix it.
Allow Pig executable to use hadoop jars not bundled with pig
Key: PIG-909
URL: https://issues.apache.org/jira/browse/PIG-909
Project: Pig
Issue Type: Improvement
Reporter: Dmitriy V. Ryaboy
Priority: Minor
Attachments: pig_909.2.patch, pig_909.patch
The current pig executable (bin/pig) looks for a file named
hadoop${PIG_HADOOP_VERSION}.jar that comes bundled with Pig.
The proposed change will allow Pig to look in $HADOOP_HOME for the hadoop
jars, if that variable is set.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-909) Allow Pig executable to use hadoop jars not bundled with pig
[
https://issues.apache.org/jira/browse/PIG-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12739287#action_12739287
]
Dmitriy V. Ryaboy commented on PIG-909:
---
Daniel, not sure what you mean.
Do you mean that the patch makes it necessary to have an external version of
hadoop to build/run pig?
That's not the case, as I wrapped the whole thing in an if -- external hadoop
jars will only be used instead of the bundled hadoop.jar if HADOOP_HOME is
defined (and valid).
Allow Pig executable to use hadoop jars not bundled with pig
Key: PIG-909
URL: https://issues.apache.org/jira/browse/PIG-909
Project: Pig
Issue Type: Improvement
Reporter: Dmitriy V. Ryaboy
Priority: Minor
Attachments: pig_909.2.patch, pig_909.patch
The current pig executable (bin/pig) looks for a file named
hadoop${PIG_HADOOP_VERSION}.jar that comes bundled with Pig.
The proposed change will allow Pig to look in $HADOOP_HOME for the hadoop
jars, if that variable is set.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-909) Allow Pig executable to use hadoop jars not bundled with pig
[
https://issues.apache.org/jira/browse/PIG-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12739292#action_12739292
]
Daniel Dai commented on PIG-909:
Hi, Dmitriy,
It does not related to the patch. What I mean is pig script in trunk is not
working correctly even before patch.
Allow Pig executable to use hadoop jars not bundled with pig
Key: PIG-909
URL: https://issues.apache.org/jira/browse/PIG-909
Project: Pig
Issue Type: Improvement
Reporter: Dmitriy V. Ryaboy
Priority: Minor
Attachments: pig_909.2.patch, pig_909.patch
The current pig executable (bin/pig) looks for a file named
hadoop${PIG_HADOOP_VERSION}.jar that comes bundled with Pig.
The proposed change will allow Pig to look in $HADOOP_HOME for the hadoop
jars, if that variable is set.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-909) Allow Pig executable to use hadoop jars not bundled with pig
[
https://issues.apache.org/jira/browse/PIG-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12739297#action_12739297
]
Dmitriy V. Ryaboy commented on PIG-909:
---
Actually I looked at build.xml for pig, and it includes the Ivy dependencies in
pig.jar
Which explains why this stuff has been working for me.
I'll delete the second patch -- that change is unnecessary.
Allow Pig executable to use hadoop jars not bundled with pig
Key: PIG-909
URL: https://issues.apache.org/jira/browse/PIG-909
Project: Pig
Issue Type: Improvement
Reporter: Dmitriy V. Ryaboy
Priority: Minor
Attachments: pig_909.patch
The current pig executable (bin/pig) looks for a file named
hadoop${PIG_HADOOP_VERSION}.jar that comes bundled with Pig.
The proposed change will allow Pig to look in $HADOOP_HOME for the hadoop
jars, if that variable is set.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
