Gunther Hagleitner commented on PIG-777:


Per PIG-627 the first example you gave will result in a single map reduce job 
that is going to process both store operations. No duplication of steps A thru 
D. So, yes you shouldn't need to introduce "D = load". Also PIG-627 introduced 
an optimization that will throw the "D = load" out - basically transforming 
your second example into the first.

This bug is mostly about the way the optimization is written. Some code should 
be moved around to align it with the optimization framework.

Adding a log message when this happens is a good idea though. Let me add that. 

> Code refactoring: Create optimization out of store/load post processing code
> ----------------------------------------------------------------------------
>                 Key: PIG-777
>                 URL: https://issues.apache.org/jira/browse/PIG-777
>             Project: Pig
>          Issue Type: Improvement
>            Reporter: Gunther Hagleitner
> The postProcessing method in the pig server checks whether a logical graph 
> contains stores to and loads from the same location. If so, it will either 
> connect the store and load, or optimize by throwing out the load and 
> connecting the store predecessor with the successor of the load.
> Ideally the introduction of the store and load connection should happen in 
> the query compiler, while the optimization should then happen in an separate 
> optimizer step as part of the optimizer framework.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

Reply via email to