Apache Wiki
Tue, 06 May 2008 15:31:14 -0700
Dear Wiki user, You have subscribed to a wiki page or wiki category on "Pig Wiki" for change notification. The following page has been changed by Shravan Narayanamurthy: http://wiki.apache.org/pig/LocalJobRunner ------------------------------------------------------------------------------ Instead of this, we can reuse Hadoop's LocalJobRunner to execute the same Map Reduce Physical plans locally. So we compile the logical plan into a map reduce physical plan and create the jobcontrol object corresponding to the mapred plan. We just need to write a separate launcher which will submit the job to the LocalJobRunner instead of submitting to an external Job Tracker. == Pros == - * Code Reuse + * Code Reuse - * No need to write and maintain + * No need to write and maintain * Different operators * Different logical to physical tranlators * Different launchers - * The current framework does not have any progress reporting. With this approach we will have it at no extra cost. + * The current framework does not have any progress reporting. With this approach we will have it at no extra cost. == Cons == - * Not sure how stable LocalJobRunner is. + * Not sure how stable LocalJobRunner is. - * Found some bugs in hadoop-15 on it which makes it practically useless for us right now. + * Found some bugs in hadoop-15 on it which makes it practically useless for us right now. - * These have been fixed however in hadoop-16 + * These have been fixed however in hadoop-16 - * Not sure how this will affect Example generator + * Not sure how this will affect Example generator