Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Pig Wiki" for change 
notification.

The "LoadStoreRedesignProposal" page has been changed by PradeepKamath.
http://wiki.apache.org/pig/LoadStoreRedesignProposal?action=diff&rev1=24&rev2=25

--------------------------------------------------

   * invoke !LoadFunc.setLocation()
   * Call getInputFormat() on the !LoadFunc and then createRecordReader() on 
the !InputFormat returned. Note that the above setLocation call needs to happen 
*before* the createRecordReader() call and the createRecordReader() call needs 
to be given a !TaskAttemptContext built out of the "updated (with location)" 
Configuration.
   * Wrap the !RecordReader returned above in !PigRecordReader class which is 
returned to Hadoop as the !RecordReader. !PigRecordReader has Text as key type 
(which is always sent with a null value to Hadoop since in pig, we really do 
not extract a key from input records) and a Tuple as a the value type (which is 
a tuple constructed from the input record). 
+ 
+ '''Open Question''': - We are hoping that !LoadFunc actually sets up the 
input location on the conf in the setLocation() call - and then using that conf 
in createRecordReader() call - what if it does this in getInputFormat()?
  
  ==== Changes to work with Hadoop OutputFormat model ====
  Hadoop has the notion of a single !OutputFormat per job. !PigOutputFormat is 
the class indicated by Pig as the !OutputFormat for map reduce jobs compiled 
from pig scripts. 

Reply via email to