Support for Hadoop Speculative Execution
----------------------------------------
Key: PIG-71
URL: https://issues.apache.org/jira/browse/PIG-71
Project: Pig
Issue Type: New Feature
Environment: Hadoop
Reporter: Amir Youssefi
Priority: Minor
If Speculative Execution is used in Hadoop while creating a data-set then Pig
scripts loading this data-set may fail. Reason is temp directories generated in
the process.
Pig can filter out these temp directories and problem gets solved. Here is
sample error:
[main] ERROR org.apache.pig - Error message from task (map)
tip_..._0001_m_002735 java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:180)
at
org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:56)
at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:90)
at
org.apache.hadoop.io.SequenceFile$Reader.readBuffer(SequenceFile.java:1524)
at
org.apache.hadoop.io.SequenceFile$Reader.seekToCurrentValue(SequenceFile.java:1590)
at
org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1626)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1712)
at
org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:79)
...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
