Thanks Alan for your nice explanation. I am not quite familiar with YARN, but it seems to me M/R architecture is not fully supportive of pipeline data flow paradigm in its core. Unless there be a strong Resource Navigator that could navigate between M/R jobs and control the flow, it's impossible to have a pipeline style execution of pig script.
I think a multi-channel parallel pipeline paradigm can greatly improve the performance of Pig on ETL tasks. And hope that can be realized with YARN. 2012/10/22 Alan Gates <[email protected]> > At this point, no. In the current MapReduce infrastructure it would take > a lot of hackery that breaks the MR abstraction to make this work[1]. This > is one thing we'd like to do as we move Pig to work on Hadoop 2.0 (aka > YARN) where it is easier for applications to build these types of features. > > [1] Details on why this is so: Assume you want to pipeline two jobs. > When job 1 gets to it's reduces, it has to pause until job 2 starts, > because it can't know where job 2's map tasks will run a priori. Job 1's > reducer has to be able to handle the case where job 2's map task fails and > it needs to restart the streaming, which means it has to spool to HDFS > anyway. In the same way job 2's map tasks need to be able to handle > failure and restart of job 1's reducer (which is easier, they could just > die). Plus you need to handle the possibility of dead locks (ie, so much > of your cluster or your user's quota may be taken up by job 1 that job 2 > will never start or get enough map tasks until job 1 ends). Current > MapReduce strongly discourages intertask communication for exactly these > reasons. > > Alan. > > On Oct 22, 2012, at 3:34 AM, W W wrote: > > > Hello, > > > > I wonder if M/R jobs compiled from pig script support pipeline between > jobs. > > > > For example, let's assume there are 5 independent consecutive M/R jobs > > doing some joining and aggregating task. > > My question is can one job be started before it's previous job finished > so > > that the previous job doesn't need to write all the output data from > reduce > > to HDFS , I just can't find any material talking about this. > > > > I think Abinitio is a good example for the full pipeline architecture. > > > > Thanks & Regards > > Stephen > >
