[
https://issues.apache.org/jira/browse/MAPREDUCE-5948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14597824#comment-14597824
]
Hudson commented on MAPREDUCE-5948:
-----------------------------------
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2183 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2183/])
MAPREDUCE-5948. org.apache.hadoop.mapred.LineRecordReader does not handle
multibyte record delimiters well. Contributed by Vinayakumar B, Rushabh Shah,
and Akira AJISAKA (jlowe: rev 077250d8d7b4b757543a39a6ce8bb6e3be356c6f)
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/LineRecordReader.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/UncompressedSplitLineReader.java
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestLineRecordReader.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestLineRecordReader.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/LineRecordReader.java
> org.apache.hadoop.mapred.LineRecordReader does not handle multibyte record
> delimiters well
> ------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-5948
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5948
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.2, 0.23.9, 2.2.0
> Environment: CDH3U2 Redhat linux 5.7
> Reporter: Kris Geusebroek
> Assignee: Akira AJISAKA
> Priority: Critical
> Fix For: 2.8.0
>
> Attachments: HADOOP-9867.patch, HADOOP-9867.patch, HADOOP-9867.patch,
> HADOOP-9867.patch, MAPREDUCE-5948.002.patch, MAPREDUCE-5948.003.patch
>
>
> Having defined a recorddelimiter of multiple bytes in a new InputFileFormat
> sometimes has the effect of skipping records from the input.
> This happens when the input splits are split off just after a
> recordseparator. Starting point for the next split would be non zero and
> skipFirstLine would be true. A seek into the file is done to start - 1 and
> the text until the first recorddelimiter is ignored (due to the presumption
> that this record is already handled by the previous maptask). Since the re
> ord delimiter is multibyte the seek only got the last byte of the delimiter
> into scope and its not recognized as a full delimiter. So the text is skipped
> until the next delimiter (ignoring a full record!!)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
