Scott, I should have read your original post in more detail. I was assuming you were just trying to get the root directory of the plugin, not loading resources during a MR job. I would have to agree with Andrzej approach if this were to be used during a MR job. Sorry for the confusion.
Dennis Kubes Scott Green wrote: > Thanks Andrzej and Doug! > > I will try both in my later work and evaluate them. > > On 1/17/07, Doug Cutting <[EMAIL PROTECTED]> wrote: >> Andrzej Bialecki wrote: >> > The reason is that if you pack this file into your job JAR, the job jar >> > would become very large (presumably this 40MB is already compressed?). >> > Job jar needs to be copied to each tasktracker for each task, so you >> > will experience performance hit just because of the size of the job jar >> > ... whereas if this file sits on DFS and is highly replicated, its >> > content will always be available locally. >> >> Note that the job jar is copied into HDFS with a highish replication >> (10?), and that it is only copied to each tasktracker node once per >> *job*, not per task. So it's only faster to manage this yourself if you >> have a sequence of jobs that share this data, and if the time to >> re-replicate it per job is significant. >> >> Doug >> ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ Nutch-developers mailing list Nutch-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nutch-developers
