Pradeep Kamath commented on PIG-934:

To get an idea of how this seeking in case of regular loads in map tasks, I 
looked at PigSlice.java, the seek happens in the init() code before bindTo():
public void init(DataStorage base) throws IOException {

        fsis = base.asElement(base.getActiveContainer(), file).sopen();

        fsis.seek(start, FLAGS.SEEK_CUR);

        end = start + getLength();

        if (file.endsWith(".bz") || file.endsWith(".bz2")) {

            is = new CBZip2InputStream(fsis, 9);

        } else if (file.endsWith(".gz")) {

            is = new GZIPInputStream(fsis);

            // We can't tell how much of the underlying stream GZIPInputStream

            // has actually consumed

            end = Long.MAX_VALUE;

        } else {

            is = fsis;


        loader.bindTo(file.toString(), new BufferedPositionedInputStream(is,

                start), start, end);

I think we need a FileLocalizer.sOpenSingleFile() method which can return a 
SeekableInputStream and we can use that in setup() in POLoad.
Something along the lines of :
static public InputStream open(String fileSpec, PigContext pigContext) throws 
IOException {
        fileSpec = checkDefaultPrefix(pigContext.getExecType(), fileSpec);
        if (!fileSpec.startsWith(LOCAL_PREFIX)) {
            ElementDescriptor elem = 
pigContext.getDfs().asElement(fullPath(fileSpec, pigContext));
            return elem.sopen();
        else {
            fileSpec = fileSpec.substring(LOCAL_PREFIX.length());
            //buffering because we only want buffered streams to be passed to 
load functions.
            /*return new BufferedInputStream(new FileInputStream(fileSpec));*/
            ElementDescriptor elem = 
pigContext.getLfs().asElement(fullPath(fileSpec, pigContext));
            return elem.sopen;

 The above code would only work with single files and not dirs which should be 
ok for merge join. We should probably set this up with a new constructor in 
POLoad which also indicates that a single file is being processed.

> Merge join implementation currently does not seek to right point on the right 
> side input based on the offset provided by the index
> ----------------------------------------------------------------------------------------------------------------------------------
>                 Key: PIG-934
>                 URL: https://issues.apache.org/jira/browse/PIG-934
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.3.1
>            Reporter: Pradeep Kamath
> We use POLoad to seek into right file which has the following code: 
> {noformat}
>    public void setUp() throws IOException{
>         String filename = lFile.getFileName();
>         loader = 
> (LoadFunc)PigContext.instantiateFuncFromSpec(lFile.getFuncSpec());        
>         is = FileLocalizer.open(filename, pc);
>         loader.bindTo(filename , new BufferedPositionedInputStream(is), 
> this.offset, Long.MAX_VALUE);
>     }
> {noformat}
> Between opening the stream and bindTo we do not seek to the right offset. 
> bindTo itself does not perform any seek.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

Reply via email to