[jira] Commented: (PIG-602) Pass global configurations to UDF
[ https://issues.apache.org/jira/browse/PIG-602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891800#action_12891800 ] Olga Natkovich commented on PIG-602: This work is already done. The user can propagate the properties via "-propertyfile from the command line and the retrieve the properties via call to UDFContext.getJobConf. Just need to document this for Pig 0.8.0 release > Pass global configurations to UDF > - > > Key: PIG-602 > URL: https://issues.apache.org/jira/browse/PIG-602 > Project: Pig > Issue Type: New Feature > Components: impl >Reporter: Yiping Han > Fix For: 0.8.0 > > > We are seeking an easy way to pass a large number of global configurations to > UDFs. > Since our application contains many pig jobs, and has a large number of > configurations. Passing configurations through command line is not an ideal > way (i.e. modifying single parameter needs to change multiple command lines). > And to put everything into the hadoop conf is not an ideal way either. > We would like to see if Pig can provide such a facility that allows us to > pass a configuration file in some format(XML?) and then make it available > through out all the UDFs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-602) Pass global configurations to UDF
[ https://issues.apache.org/jira/browse/PIG-602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12705812#action_12705812 ] David Ciemiewicz commented on PIG-602: -- JIRA PIG-477 is related to this, I think. > Pass global configurations to UDF > - > > Key: PIG-602 > URL: https://issues.apache.org/jira/browse/PIG-602 > Project: Pig > Issue Type: New Feature > Components: impl >Reporter: Yiping Han >Assignee: Alan Gates > > We are seeking an easy way to pass a large number of global configurations to > UDFs. > Since our application contains many pig jobs, and has a large number of > configurations. Passing configurations through command line is not an ideal > way (i.e. modifying single parameter needs to change multiple command lines). > And to put everything into the hadoop conf is not an ideal way either. > We would like to see if Pig can provide such a facility that allows us to > pass a configuration file in some format(XML?) and then make it available > through out all the UDFs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-602) Pass global configurations to UDF
[ https://issues.apache.org/jira/browse/PIG-602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12702932#action_12702932 ] David Ciemiewicz commented on PIG-602: -- I think PIG-768 is related and would benefit from a consistent set of properties and configuration values that are populated by Pig run-time and user set values. > Pass global configurations to UDF > - > > Key: PIG-602 > URL: https://issues.apache.org/jira/browse/PIG-602 > Project: Pig > Issue Type: New Feature > Components: impl >Reporter: Yiping Han >Assignee: Alan Gates > > We are seeking an easy way to pass a large number of global configurations to > UDFs. > Since our application contains many pig jobs, and has a large number of > configurations. Passing configurations through command line is not an ideal > way (i.e. modifying single parameter needs to change multiple command lines). > And to put everything into the hadoop conf is not an ideal way either. > We would like to see if Pig can provide such a facility that allows us to > pass a configuration file in some format(XML?) and then make it available > through out all the UDFs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-602) Pass global configurations to UDF
[ https://issues.apache.org/jira/browse/PIG-602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12702707#action_12702707 ] David Ciemiewicz commented on PIG-602: -- This sounds a lot like shell script environment variables. As such maybe it should follow the same rich level of operations and semantics that you get with environment variables. How is PigConf different from "set" properties in Pig? Why can't both use the same mechanism? Should they use the same mechanism? Can / should this same mechanism let my UDFs "know" when Pig is in local mode versus hdfs mode? [JIRA PIG-756] (or should something different be used? When in grunt, how can I inspect what the current PigConf values are? (Useful for logging and debugging) By what mechanism can I set or override these values from within my Pig script? Can I set the values to be one thing at one point in the Pig script and change it later to a new value in the Pig script? > Pass global configurations to UDF > - > > Key: PIG-602 > URL: https://issues.apache.org/jira/browse/PIG-602 > Project: Pig > Issue Type: New Feature > Components: impl >Reporter: Yiping Han >Assignee: Alan Gates > > We are seeking an easy way to pass a large number of global configurations to > UDFs. > Since our application contains many pig jobs, and has a large number of > configurations. Passing configurations through command line is not an ideal > way (i.e. modifying single parameter needs to change multiple command lines). > And to put everything into the hadoop conf is not an ideal way either. > We would like to see if Pig can provide such a facility that allows us to > pass a configuration file in some format(XML?) and then make it available > through out all the UDFs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-602) Pass global configurations to UDF
[ https://issues.apache.org/jira/browse/PIG-602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12672377#action_12672377 ] Olga Natkovich commented on PIG-602: Alan, looks good. The only thing I would propose to change is to handle information on the client side as Java properties. This means (1) That individual properties can be passed via -D switch from the command file (2) .properties files passed by users will get picked up. (Not sure exactly what Java's mechanism for this is.) > Pass global configurations to UDF > - > > Key: PIG-602 > URL: https://issues.apache.org/jira/browse/PIG-602 > Project: Pig > Issue Type: New Feature > Components: impl >Reporter: Yiping Han >Assignee: Alan Gates > > We are seeking an easy way to pass a large number of global configurations to > UDFs. > Since our application contains many pig jobs, and has a large number of > configurations. Passing configurations through command line is not an ideal > way (i.e. modifying single parameter needs to change multiple command lines). > And to put everything into the hadoop conf is not an ideal way either. > We would like to see if Pig can provide such a facility that allows us to > pass a configuration file in some format(XML?) and then make it available > through out all the UDFs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-602) Pass global configurations to UDF
[ https://issues.apache.org/jira/browse/PIG-602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12672376#action_12672376 ] Yiping Han commented on PIG-602: Alan, this plan looks good for our requirements. > Pass global configurations to UDF > - > > Key: PIG-602 > URL: https://issues.apache.org/jira/browse/PIG-602 > Project: Pig > Issue Type: New Feature > Components: impl >Reporter: Yiping Han >Assignee: Alan Gates > > We are seeking an easy way to pass a large number of global configurations to > UDFs. > Since our application contains many pig jobs, and has a large number of > configurations. Passing configurations through command line is not an ideal > way (i.e. modifying single parameter needs to change multiple command lines). > And to put everything into the hadoop conf is not an ideal way either. > We would like to see if Pig can provide such a facility that allows us to > pass a configuration file in some format(XML?) and then make it available > through out all the UDFs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-602) Pass global configurations to UDF
[
https://issues.apache.org/jira/browse/PIG-602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12672360#action_12672360
]
Alan Gates commented on PIG-602:
I propose the following solution.
First, a singleton class will added to pig.
{code}
public class PigConf implements Serializable {
private static PigConf self;
private Map userConf;
private PigConf() { ... }
public static getPigConf() { return self; }
public Map getUserConf();
}
{code}
Pig would take care of serializing this class between the front end and
backend. So users UDFs could stash keys and values away in this on the front
end and then be guaranteed to pick them back up on the back end. Pig's map,
reduce, and combiner frameworks would need to change to explicitly desieralize
this and populate it. The front end would need to change to serialize this as
part of submitting the job to hadoop.
Furthermore, users could populate this from a configuration file by providing
a file on the command line. We would add a command line argument (such as
-u/-userconf). Contents of this file would be read using
Properties.loadFromXml and then loaded to PigConf.userConf.
The reason a Properties object is not used for this is that Properties is a
Map which is too generic. We would like to constrain the keys
to be Strings, and the values must be Serializable so that we can guarantee
that we can transmit them from front end to back.
Thoughts?
> Pass global configurations to UDF
> -
>
> Key: PIG-602
> URL: https://issues.apache.org/jira/browse/PIG-602
> Project: Pig
> Issue Type: New Feature
> Components: impl
>Reporter: Yiping Han
>Assignee: Alan Gates
>
> We are seeking an easy way to pass a large number of global configurations to
> UDFs.
> Since our application contains many pig jobs, and has a large number of
> configurations. Passing configurations through command line is not an ideal
> way (i.e. modifying single parameter needs to change multiple command lines).
> And to put everything into the hadoop conf is not an ideal way either.
> We would like to see if Pig can provide such a facility that allows us to
> pass a configuration file in some format(XML?) and then make it available
> through out all the UDFs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
