Dear Wiki user,
You have subscribed to a wiki page or wiki category on "Pig Wiki" for change
The following page has been changed by Shravan Narayanamurthy:
= Local Execution Mode using LocalJobRunner from Hadoop =
Currently we have an additional Local execution engine which handles the local
execution mode. The physical plan for this has to be built separately and its
execution should be done differently. This also introduces different operators
for local and map reduce.
Instead of this, we can reuse Hadoop's LocalJobRunner to execute the same Map
Reduce Physical plans locally. So we compile the logical plan into a map reduce
physical plan and create the jobcontrol object corresponding to the mapred
plan. We just need to write a separate launcher which will submit the job to
the LocalJobRunner instead of submitting to an external Job Tracker.
== Pros ==
* Code Reuse
* No need to write and maintain
* Different operators
* Different logical to physical tranlators
* Different launchers
* The current framework does not have any progress reporting. With this
approach we will have it at no extra cost.
== Cons ==
* Not sure how stable LocalJobRunner is.
* Found some bugs in hadoop-15 on it which makes it practically useless for us
* These have been fixed however in hadoop-16
* Not sure how this will affect Example generator