RE: Is it possible to access Configuration in UDF ?
At the moment we can't make UDFs dependant on Hadoop as people also use them for testing in local mode which is currently not based on Hadoop local mode due to performance constrains. I agree that we need to provide a way to get UDF a configuration/property object. Olga -Original Message- From: Daniel Dai [mailto:dai...@gmail.com] Sent: Monday, August 03, 2009 9:20 PM To: pig-dev@hadoop.apache.org; pig-u...@hadoop.apache.org Subject: Re: Is it possible to access Configuration in UDF ? Hi, Jeff, This is not API at all, this is a hack to make things work. We do lack couples of features for UDF: 1. reporter and counter (PIG-889) 2. access global properties 3. ability to maintain states across different UDF invocations 4. input schema 5. variable length arguments (PIG-902) Your suggestion sounds resonable. We need to provide a well designed interface for these features. - Original Message - From: "zhang jianfeng" To: ; Sent: Monday, August 03, 2009 8:03 PM Subject: Re: Is it possible to access Configuration in UDF ? > Dmitriy, > > Thank you for your help. > > I find this way of using API is not so intuitive , I recommend the base > class of UDF to implements the Configurable interface. > Then each UDF can use the getConf() to get the Configuration object. > Because UDF is part of MapReduce , it makes sense to make it Configurable. > > The following is what I recommend to change the EvalFunc > > public abstract class EvalFunc implements Configurable{ > .. > protected Configuration conf; > .. > public EvalFunc(){ > conf=PigMapReduce.sJobConf; > } > .. > @Override >public void setConf(Configuration conf) { >this.conf=conf; >} > >@Override >public Configuration getConf() { >return this.conf; >} > > > > > Jeff Zhang > > > > > > On Mon, Aug 3, 2009 at 8:52 PM, Dmitriy Ryaboy > wrote: > >> You can access the JobConf with the following call: >> >> ConfigurationUtil.toProperties(PigMapReduce.sJobConf) >> >> On Mon, Aug 3, 2009 at 12:40 AM, zhang jianfeng wrote: >> > Hi all, >> > >> > I'd like to set property in Configuration to customize my UDF. But it >> looks >> > like I can not access the Configuration object in UDF. >> > >> > Does pig have a plan to support this feature ? >> > >> > >> > Thank you. >> > >> > Jeff Zhang >> > >> >
Re: Is it possible to access Configuration in UDF ?
Hi, Jeff, This is not API at all, this is a hack to make things work. We do lack couples of features for UDF: 1. reporter and counter (PIG-889) 2. access global properties 3. ability to maintain states across different UDF invocations 4. input schema 5. variable length arguments (PIG-902) Your suggestion sounds resonable. We need to provide a well designed interface for these features. - Original Message - From: "zhang jianfeng" To: ; Sent: Monday, August 03, 2009 8:03 PM Subject: Re: Is it possible to access Configuration in UDF ? Dmitriy, Thank you for your help. I find this way of using API is not so intuitive , I recommend the base class of UDF to implements the Configurable interface. Then each UDF can use the getConf() to get the Configuration object. Because UDF is part of MapReduce , it makes sense to make it Configurable. The following is what I recommend to change the EvalFunc public abstract class EvalFunc implements Configurable{ .. protected Configuration conf; .. public EvalFunc(){ conf=PigMapReduce.sJobConf; } .. @Override public void setConf(Configuration conf) { this.conf=conf; } @Override public Configuration getConf() { return this.conf; } Jeff Zhang On Mon, Aug 3, 2009 at 8:52 PM, Dmitriy Ryaboy wrote: You can access the JobConf with the following call: ConfigurationUtil.toProperties(PigMapReduce.sJobConf) On Mon, Aug 3, 2009 at 12:40 AM, zhang jianfeng wrote: > Hi all, > > I'd like to set property in Configuration to customize my UDF. But it looks > like I can not access the Configuration object in UDF. > > Does pig have a plan to support this feature ? > > > Thank you. > > Jeff Zhang >
Re: Is it possible to access Configuration in UDF ?
Dmitriy, Thank you for your help. I find this way of using API is not so intuitive , I recommend the base class of UDF to implements the Configurable interface. Then each UDF can use the getConf() to get the Configuration object. Because UDF is part of MapReduce , it makes sense to make it Configurable. The following is what I recommend to change the EvalFunc public abstract class EvalFunc implements Configurable{ .. protected Configuration conf; .. public EvalFunc(){ conf=PigMapReduce.sJobConf; } .. @Override public void setConf(Configuration conf) { this.conf=conf; } @Override public Configuration getConf() { return this.conf; } Jeff Zhang On Mon, Aug 3, 2009 at 8:52 PM, Dmitriy Ryaboy wrote: > You can access the JobConf with the following call: > > ConfigurationUtil.toProperties(PigMapReduce.sJobConf) > > On Mon, Aug 3, 2009 at 12:40 AM, zhang jianfeng wrote: > > Hi all, > > > > I'd like to set property in Configuration to customize my UDF. But it > looks > > like I can not access the Configuration object in UDF. > > > > Does pig have a plan to support this feature ? > > > > > > Thank you. > > > > Jeff Zhang > > >