[jira] [Commented] (HADOOP-13725) Open MapFile for append

2016-10-19 Thread VITALIY SAVCHENKO (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-13725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590692#comment-15590692
 ] 

VITALIY SAVCHENKO commented on HADOOP-13725:


{code}
1.
MapFile.Writer w = null;
try {
w = new MapFile.Writer(
new Configuration(),
new Path("hdfs://192.168.56.101:9000/20161020/data"),
MapFile.Writer.keyClass(IntWritable.class),
MapFile.Writer.valueClass(IntWritable.class)
);
w.append(new IntWritable(0), new IntWritable(100));
w.append(new IntWritable(10), new IntWritable(200));
w.append(new IntWritable(5), new IntWritable(400)); 
//java.io.IOException: key out of order: 5 after 10
} catch (Exception ex) {
ex.printStackTrace();
} finally {
w.close();
}
MapFile.Reader reader = new MapFile.Reader(
new Path("hdfs://192.168.56.101:9000/20161020/data"),
new Configuration()
);
System.out.println(reader.get(new IntWritable(0), new IntWritable())); 
//print 100
System.out.println(reader.get(new IntWritable(10), new IntWritable())); 
// print 200 MapFile corrent
reader.close();
2. Open MapFile for apped
MapFile.Writer w = null;
try {
w = new MapFile.Writer(
new Configuration(),
new Path("hdfs://192.168.56.101:9000/20161020/data"),
MapFile.Writer.keyClass(IntWritable.class),
MapFile.Writer.valueClass(IntWritable.class)
);
w.append(new IntWritable(0), new IntWritable(100));
w.append(new IntWritable(10), new IntWritable(200));
w.close();

w = new MapFile.Writer(
new Configuration(),
new Path("hdfs://192.168.56.101:9000/20161020/data"),
SequenceFile.Writer.appendIfExists(true), // append to exist 
MapFile
SequenceFile.Writer.replication((short)2),
MapFile.Writer.keyClass(IntWritable.class),
MapFile.Writer.valueClass(IntWritable.class)
);
w.append(new IntWritable(20), new IntWritable(300));
w.append(new IntWritable(30), new IntWritable(400));
} catch (Exception ex) {
ex.printStackTrace();
} finally {
w.close();
}
MapFile.Reader reader = new MapFile.Reader(
new Path("hdfs://192.168.56.101:9000/20161020/data"),
new Configuration()
);
System.out.println(reader.get(new IntWritable(10), new IntWritable())); 
//print 200
System.out.println(reader.get(new IntWritable(20), new IntWritable())); 
//print 300 MapFile correct
reader.close();

3. Append to exist MapFile, but incorrect range
MapFile.Writer w = null;
try {
w = new MapFile.Writer(
new Configuration(),
new Path("hdfs://192.168.56.101:9000/20161020/data"),
MapFile.Writer.keyClass(IntWritable.class),
MapFile.Writer.valueClass(IntWritable.class)
);
w.append(new IntWritable(10), new IntWritable(100));
w.append(new IntWritable(20), new IntWritable(200));
w.close();

w = new MapFile.Writer(
new Configuration(),
new Path("hdfs://192.168.56.101:9000/20161020/data"),
SequenceFile.Writer.appendIfExists(true), //append to MapFile
MapFile.Writer.keyClass(IntWritable.class),
MapFile.Writer.valueClass(IntWritable.class)
);
w.append(new IntWritable(5), new IntWritable(300)); //No exception 
here
w.append(new IntWritable(10), new IntWritable(400));
} catch (Exception ex) {
ex.printStackTrace();
} finally {
w.close();
}

MapFile.Reader reader = new MapFile.Reader(
new Path("hdfs://192.168.56.101:9000/20161020/data"),
new Configuration()
);
System.out.println(reader.get(new IntWritable(5), new IntWritable())); 
// java.io.IOException: key out of order: 5 after 10 - MapFile corrupted
System.out.println(reader.get(new IntWritable(10), new IntWritable()));
System.out.println(reader.get(new IntWritable(20), new IntWritable()));
reader.close();
{code}

Reason: When open MapFile with option SequenceFile.Writer.appendIfExists(true) 
- writer not read last key from exist MapFile.


> Open MapFile for append
> ---
>
> Key: HADOOP-13725
> URL: https://issues.apache.org/jira/browse/HADOOP-13725
> Project: Hadoop Common
>  Issue Type: New Feature
>Reporter: VITALIY SAVCHENKO
>
> I think it possible 

[jira] [Commented] (HADOOP-13725) Open MapFile for append

2016-10-19 Thread Yulei Li (JIRA)

[ 
https://issues.apache.org/jira/browse/HADOOP-13725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15587885#comment-15587885
 ] 

Yulei Li commented on HADOOP-13725:
---

Could you describe the problem more detailed? In the MapFile source code, I see 
it have append method and before appending the method actually checks the new 
key. So how to reproduce your problem? Thanks

> Open MapFile for append
> ---
>
> Key: HADOOP-13725
> URL: https://issues.apache.org/jira/browse/HADOOP-13725
> Project: Hadoop Common
>  Issue Type: New Feature
>Reporter: VITALIY SAVCHENKO
>
> I think it possible to open MapFile for appending.
> SequenceFile support it (Option SequenceFile.Writer.appendIfExists(true) 
> HADOOP-7139)
> Now it almost working. But if use SequenceFile.Writer.appendIfExists(true) 
> MapFile.Writer - it not read last key and does not check new keys. That's why 
> MapFile can be corrupted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org