Ok thank you guys. I will check with your suggestions. On 13 August 2013 15:45, Serega Sheypak <[email protected]> wrote:
> If you run your script using oozie, you can control it using > <property> > <name>mapreduce.task.classpath.user.precedence</name> > <value>true</value> > </property> > > Also try to add to add to the top of your script: > set mapreduce.task.classpath.user.precedence=true > > > > > 2013/8/13 Niels Basjes <[email protected]> > > > I've had the exact same problem with my own udf and joda. > > It turn out that an old version of joda is embedded inside the pig > > application. > > My workaround was limitying myself to the old joda api. > > > > Niels > > On Aug 13, 2013 10:31 AM, "Darpan R" <[email protected]> wrote: > > > > > Thanks Serega and Ankit. > > > > > > According to my understanding it looks that the actual joda-time jar is > > in > > > the classpath (otherwise it would have failed saying no classDef > found). > > > But it is not able to find the "now" static method which is part of the > > > DateTime class. > > > Not sure if I am missing something. > > > > > > The transformation I am doing on the date field using joda date time > API. > > > > > > Let me paste the complete stack trace : > > > > > > > > > 2013-08-11 13:01:06,911 [Thread-9] WARN > > > org.apache.hadoop.mapred.LocalJobRunner - job_local_0001 > > > java.lang.NoSuchMethodError: > > > > > > > > > org.joda.time.DateTime.now(Lorg/joda/time/DateTimeZone;)Lorg/joda/time/DateTime; > > > at > > > > > > com.myproject.pig.udf.ExtractDataByDates.exec(ExtractDataByDates.java:178) > > > at > > > > com.myproject.pig.udf.ExtractDataByDates.exec(ExtractDataByDates.java:12) > > > at > > > > > > > > > org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:337) > > > at > > > > > > > > > org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:381) > > > at > > > > > > > > > org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:334) > > > at > > > > > > > > > org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:372) > > > at > > > > > > > > > org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:297) > > > at > > > > > > > > > org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:308) > > > at > > > > > > > > > org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:241) > > > at > > > > > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.runPipeline(PigGenericMapReduce.java:465) > > > at > > > > > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.processOnePackageOutput(PigGenericMapReduce.java:433) > > > at > > > > > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:413) > > > at > > > > > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:257) > > > at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176) > > > at > > > org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:650) > > > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418) > > > at > > > > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:262) > > > 2013-08-11 13:01:10,417 [main] WARN > > > > > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > > > - Ooops! Some job has failed! Specify -stop_on_fail > > > > > > > > > > > > Thanks again. > > > DR > > > > > > On 13 August 2013 11:28, Serega Sheypak <[email protected]> > > wrote: > > > > > > > Pig uses reflection. The top exception says that there is no such > > method > > > > signature. The problem is in the way you are trying to call method. > > > > And its better to paste the whole stavktrace > > > > 13.08.2013 7:39 пользователь "Darpan R" <[email protected]> > написал: > > > > > > > > > I've a UDF which I use to do custom processing on the records. In > the > > > > eval > > > > > function I am using a third party jar for processing. I saw the job > > jar > > > > > file, but it does not include this dependency. Is there any way to > > > > include > > > > > dependent jar in the job jar ? (For testing I am running the > cluster > > in > > > > the > > > > > local mode). > > > > > > > > > > Or can I use distributed cache to make the dependent jar available > to > > > the > > > > > UDF ? > > > > > > > > > > I've tried registering the dependent jars in the pig. For the first > > > > > registered jar (all udfs are bundled in this jar) I do not face the > > > > issues. > > > > > But for the second jar, I am facing issues when UDF tries to access > > the > > > > > class from it. > > > > > REGISTER '/home/user/pig/udfrepository/projectUDF.jar' > > > > > REGISTER '/home/user/thridpartyjars/xyz.jar'; > > > > > The logs I get on the console are like this : > > > > > 2013-08-11 10:35:02,485 [Thread-14] WARN > > > > > org.apache.hadoop.mapred.LocalJobRunner - job_local_0001 > > > > > java.lang.NoSuchMethodError: > > > > > org.xyz.abc.convertToOtherFormat(Lorg/DateTimeZone;)Lorg/DateTime; > > > > > at com.myproject.MyUDF.exec(MyUDF.java:70) > > > > > Any help on this is highly appreciated. > > > > > Thanks in advance. > > > > > > > > > > -Darpan > > > > > > > > > > > > > > >
