[
https://issues.apache.org/jira/browse/HADOOP-13725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590692#comment-15590692
]
VITALIY SAVCHENKO commented on HADOOP-13725:
{code}
1.
MapFile.Writer w = null;
try {
w = new MapFile.Writer(
new Configuration(),
new Path("hdfs://192.168.56.101:9000/20161020/data"),
MapFile.Writer.keyClass(IntWritable.class),
MapFile.Writer.valueClass(IntWritable.class)
);
w.append(new IntWritable(0), new IntWritable(100));
w.append(new IntWritable(10), new IntWritable(200));
w.append(new IntWritable(5), new IntWritable(400));
//java.io.IOException: key out of order: 5 after 10
} catch (Exception ex) {
ex.printStackTrace();
} finally {
w.close();
}
MapFile.Reader reader = new MapFile.Reader(
new Path("hdfs://192.168.56.101:9000/20161020/data"),
new Configuration()
);
System.out.println(reader.get(new IntWritable(0), new IntWritable()));
//print 100
System.out.println(reader.get(new IntWritable(10), new IntWritable()));
// print 200 MapFile corrent
reader.close();
2. Open MapFile for apped
MapFile.Writer w = null;
try {
w = new MapFile.Writer(
new Configuration(),
new Path("hdfs://192.168.56.101:9000/20161020/data"),
MapFile.Writer.keyClass(IntWritable.class),
MapFile.Writer.valueClass(IntWritable.class)
);
w.append(new IntWritable(0), new IntWritable(100));
w.append(new IntWritable(10), new IntWritable(200));
w.close();
w = new MapFile.Writer(
new Configuration(),
new Path("hdfs://192.168.56.101:9000/20161020/data"),
SequenceFile.Writer.appendIfExists(true), // append to exist
MapFile
SequenceFile.Writer.replication((short)2),
MapFile.Writer.keyClass(IntWritable.class),
MapFile.Writer.valueClass(IntWritable.class)
);
w.append(new IntWritable(20), new IntWritable(300));
w.append(new IntWritable(30), new IntWritable(400));
} catch (Exception ex) {
ex.printStackTrace();
} finally {
w.close();
}
MapFile.Reader reader = new MapFile.Reader(
new Path("hdfs://192.168.56.101:9000/20161020/data"),
new Configuration()
);
System.out.println(reader.get(new IntWritable(10), new IntWritable()));
//print 200
System.out.println(reader.get(new IntWritable(20), new IntWritable()));
//print 300 MapFile correct
reader.close();
3. Append to exist MapFile, but incorrect range
MapFile.Writer w = null;
try {
w = new MapFile.Writer(
new Configuration(),
new Path("hdfs://192.168.56.101:9000/20161020/data"),
MapFile.Writer.keyClass(IntWritable.class),
MapFile.Writer.valueClass(IntWritable.class)
);
w.append(new IntWritable(10), new IntWritable(100));
w.append(new IntWritable(20), new IntWritable(200));
w.close();
w = new MapFile.Writer(
new Configuration(),
new Path("hdfs://192.168.56.101:9000/20161020/data"),
SequenceFile.Writer.appendIfExists(true), //append to MapFile
MapFile.Writer.keyClass(IntWritable.class),
MapFile.Writer.valueClass(IntWritable.class)
);
w.append(new IntWritable(5), new IntWritable(300)); //No exception
here
w.append(new IntWritable(10), new IntWritable(400));
} catch (Exception ex) {
ex.printStackTrace();
} finally {
w.close();
}
MapFile.Reader reader = new MapFile.Reader(
new Path("hdfs://192.168.56.101:9000/20161020/data"),
new Configuration()
);
System.out.println(reader.get(new IntWritable(5), new IntWritable()));
// java.io.IOException: key out of order: 5 after 10 - MapFile corrupted
System.out.println(reader.get(new IntWritable(10), new IntWritable()));
System.out.println(reader.get(new IntWritable(20), new IntWritable()));
reader.close();
{code}
Reason: When open MapFile with option SequenceFile.Writer.appendIfExists(true)
- writer not read last key from exist MapFile.
> Open MapFile for append
> ---
>
> Key: HADOOP-13725
> URL: https://issues.apache.org/jira/browse/HADOOP-13725
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: VITALIY SAVCHENKO
>
> I think it possible
