[jira] Updated: (PIG-1231) DataBagIterator.hasNext() should be idempotent

[Daniel Dai (JIRA)]

     [ 
https://issues.apache.org/jira/browse/PIG-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated PIG-1231:
----------------------------

    Description: 
DataBagIterator.hasNext() is not repeatable in some situations. This is not 
acceptable cuz the name hasNext() implies that it is idempotent. While 
hasNext() returns true, it is repeatable, but if hasNext() returns false, it is 
not. In BagFormat, we do misuse DataBagIterator.hasNext() because of the 
assumption that hasNext() is always idempotent, which leads to some mysterious 
errors. Here is one error we saw:

Caused by: java.io.IOException: Stream closed
        at 
java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:145)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:189)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
        at java.io.DataInputStream.readByte(DataInputStream.java:248)
        at org.apache.pig.data.DefaultTuple.readFields(DefaultTuple.java:278)
        at 
org.apache.pig.data.DefaultDataBag$DefaultDataBagIterator.readFromFile(DefaultDataBag.java:237)
        ... 20 more

This happens because: we call hasNext(), which reach EOF and we close the file. 
Then we call hasNext() again in the assumption that it is idempotent. However, 
the stream is closed so we get this error message.

This fix will go to DefaultDataBagIterator, DistinctDataBagIterator, 
CachedBagIterator, SortedDataBagIterator. 

  was:
Current implementation of DataBagIterator.hasNext() will actually fetch the 
next tuple every time. So if we call hasNext() consecutively, more than 1 
tuples will be fetched. This is confusing cuz the name hasNext() implies that 
it is idempotent. In BagFormat, we do misuse DataBagIterator.hasNext() because 
of this, which leads to some mysterious errors. Here is one error we saw:

Caused by: java.io.IOException: Stream closed
        at 
java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:145)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:189)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
        at java.io.DataInputStream.readByte(DataInputStream.java:248)
        at org.apache.pig.data.DefaultTuple.readFields(DefaultTuple.java:278)
        at 
org.apache.pig.data.DefaultDataBag$DefaultDataBagIterator.readFromFile(DefaultDataBag.java:237)
        ... 20 more

This happens because: we call hasNext(), which reach EOF and we close the file. 
Then we call hasNext() again in the assumption that it is idempotent. However, 
the stream is closed so we get this error message.

This fix will go to DefaultDataBagIterator, DistinctDataBagIterator, 
CachedBagIterator, SortedDataBagIterator. 


> DataBagIterator.hasNext() should be idempotent
> ----------------------------------------------
>
>                 Key: PIG-1231
>                 URL: https://issues.apache.org/jira/browse/PIG-1231
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>    Affects Versions: 0.6.0
>            Reporter: Daniel Dai
>            Assignee: Daniel Dai
>             Fix For: 0.6.0
>
>         Attachments: PIG-1231-1.patch
>
>
> DataBagIterator.hasNext() is not repeatable in some situations. This is not 
> acceptable cuz the name hasNext() implies that it is idempotent. While 
> hasNext() returns true, it is repeatable, but if hasNext() returns false, it 
> is not. In BagFormat, we do misuse DataBagIterator.hasNext() because of the 
> assumption that hasNext() is always idempotent, which leads to some 
> mysterious errors. Here is one error we saw:
> Caused by: java.io.IOException: Stream closed
>         at 
> java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:145)
>         at java.io.BufferedInputStream.fill(BufferedInputStream.java:189)
>         at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
>         at java.io.DataInputStream.readByte(DataInputStream.java:248)
>         at org.apache.pig.data.DefaultTuple.readFields(DefaultTuple.java:278)
>         at 
> org.apache.pig.data.DefaultDataBag$DefaultDataBagIterator.readFromFile(DefaultDataBag.java:237)
>         ... 20 more
> This happens because: we call hasNext(), which reach EOF and we close the 
> file. Then we call hasNext() again in the assumption that it is idempotent. 
> However, the stream is closed so we get this error message.
> This fix will go to DefaultDataBagIterator, DistinctDataBagIterator, 
> CachedBagIterator, SortedDataBagIterator. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.