The classpath will be composed of the following: 1) The TaskTracker classpath 2) The job jar 3) Jars in lib/ folder of the job jar 4) Anything specified in the libjars parameter, which will be dropped into distcache and stuck onto the classpath before starting your task
On Thu, Dec 15, 2011 at 5:38 AM, Gayatri Rao <[email protected]> wrote: > Thank you. But where does pig copy the jar on the slave machines? > Incase some of my older jars with conflicting classes already exists on the > machines, is there a way for me to specify what classpath to use for the > child jvms? > > On Wed, Dec 14, 2011 at 6:59 PM, Dmitriy Ryaboy <[email protected]> > wrote: > > > Yes, you do this by using register.... :) > > Just figure out what jars are conflicting and get rid of the old one. > > > > On Wed, Dec 14, 2011 at 6:01 PM, Gayatri Rao <[email protected]> > wrote: > > > > > Thanks so much for your input Dmitriy. It looks like thats what is > > > happening. Is there a way I can set classpath on child jvms in pig? > > > > > > Thanks > > > Gayatri > > > > > > On Wed, Dec 14, 2011 at 5:57 PM, Dmitriy Ryaboy <[email protected]> > > > wrote: > > > > > > > Can you send the stack trace? > > > > If it's not finding a *method* then it's likely you have a jar > conflict > > > > (several jars have different versions of the class and you are > > > accidentally > > > > using the wrong one) > > > > > > > > D > > > > > > > > On Wed, Dec 14, 2011 at 4:41 PM, Gayatri Rao <[email protected]> > > > wrote: > > > > > > > > > It picksup the loader method but not the helper method. They are in > > > > > spearate jars and I have registered both. > > > > > I also checked my job.jar that was created in > > > > > /user/gayatri/.staging/job2011xxxx/job.jar and all the classes both > > the > > > > > loader as well as the helper classes are present. > > > > > > > > > > On Wed, Dec 14, 2011 at 4:27 PM, Prashant Kommireddi < > > > > [email protected] > > > > > >wrote: > > > > > > > > > > > Does your script not find the loader or helper methods? Are they > > > > packaged > > > > > > together? If they are separate do you register both jars? > > > > > > > > > > > > Take a look at > > http://pig.apache.org/docs/r0.9.1/basic.html#register > > > . > > > > > > > > > > > > > > > > > > On Wed, Dec 14, 2011 at 4:15 PM, Gayatri Rao < > [email protected]> > > > > > wrote: > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > Can someone please point me to any documentation on how exactly > > > > > REGISTER > > > > > > > copies over the job.jar to the slave machines? > > > > > > > I have some loader UDFs and some helper methods which are > > > utilized > > > > by > > > > > > the > > > > > > > loader UDFs. My job.jar has all the classes but my pig job > fails > > > > saying > > > > > > > that its not able to find the helper methods used by the loader > > > UDFs. > > > > > > > I cant seem to understand why. Any insights into what might be > > the > > > > > > problem? > > > > > > > > > > > > > > Thanks > > > > > > > Gayatri > > > > > > > > > > > > > > > > > > > > > > > > > > > >
