Hi Mark, I also met your problem,I found my way finally. Firstly,your basic idea is right,we need to move these jars in to HDFS,because files in HDFS are shared by all the node automatically. So,There seem to be two solutions here.
solution: a)After you export your project as a jar,you add a directory named lib in the root of your project jar manually,after that you can add ALL external jars which your project needs in this directory made by you. By this way,when you submit your job to the jobtracker, you also submit the lib jars into the HDFS together,and Hadoop will add your small lib into its lib for your project.I tried,it works.This may reduce your work,I think. solution: b)There seems to be a Hadoop command with parameter as '-libjars jar1,jar2' ,I read this from *Hadoop.The.Definitive.Guide.* * *the description of this command is*"*Copies the specified JAR files from the local filesystem (or any filesystem if a scheme is specified) to the shared filesystem used by the jobtracker (usually HDFS), and adds them to the MapReduce task’s classpath. This option is a useful way of shipping JAR files that *a job is dependent on." *howerver,I still did not figure it out now,It did not works for me. The above are what I have experienced,I hope it can help you. If you still have some problems,please feel free to let me.Good Luck. Regards. Walker Gu. 2011/4/9 Mark <static.void....@gmail.com> > If I have some jars that I would like including into ALL of my jobs is > there some shared lib directory on HDFS that will be available to all of my > nodes? Something similar to how Oozie uses a shared lib directory when > submitting workflows. > > As of right now I've been cheating and copying these jars into the hadoop > lib directory on each node. I figured there had to be a better way. > > Thanks >
