Hi, 192MB is not an issue if you are gonig to process gigabytes of data. >adding my dependencies to the classpath of my tasktrackers
you should be prepared to start to resolve weird jar-hell problems. Probably, you can save seconds putting your jars into tasktracker classpath 2015-07-20 15:47 GMT+02:00 Vincent Russell <[email protected]>: > Hello, > > I am using Oozie 4.1.0 with CDH4.6. I have been adding some custom > action types to fit our use cases and in doing that my sharelib has gotten > pretty big (192MB). I am now concerned that this makes my distributed > cache way too big and could make my MapReduce job startup times a lot > longer than they have to be. (And I'm potentially having unnecessary > network traffic). > > Am I at the point where I should be adding my dependencies to the classpath > of my tasktrackers directly so that the jars aren't downloaded to the > tasktrackers every time a MapReduce job is run? > > If so, what's the best way to achieve this? > > Thanks in advance for your guidance and assistance, > > Vincent >
