[
https://issues.apache.org/jira/browse/HDFS-4849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HDFS-4849:
--------------------------------------
Comment: was deleted
(was: > As regards creating blocks simultaneously, it is prevented precisely
because multiple creates are not allowed today and hence the application does
not even get to allocating blocks.
My example is different. You create file once. THEN spawn 10 threads to add
blocks in the same file.
To demonstrate my point I wrote a snippet below. Just past© it into your
favourite test and run. You will be amazed what you will get, I was. Also try
to vary the {{bufLen}}. Some interesting things happen when you get to the
second block.
The system can be abused and this has nothing to do with the retries.
{code}
static volatile int nrDone;
public static void main(String[] args) throws Exception {
Configuration conf = new HdfsConfiguration();
MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf).
numDataNodes(3).build();
FileSystem fs = cluster.getFileSystem();
Path path = new Path("testFile.txt");
final FSDataOutputStream out = fs.create(path);
nrDone = 0;
for(int i = 0; i < 10; i++) {
new Thread() {
@Override
public void run() {
int bufLen = 128*1024*1024 + 7;
byte[] toWrite = new byte[bufLen];
try {
out.write(toWrite, 0, toWrite.length);
} catch (IOException e) {
LOG.error("Failed to write a string: ", e);
}
nrDone++;
}
}.start();
}
while(nrDone < 10) ;
out.close();
LOG.info("Done. " + fs.getFileStatus(path));
for(BlockLocation bl : fs.getFileBlockLocations(path, 0, Long.MAX_VALUE))
LOG.info(bl);
}
{code})
> Idempotent create and append operations.
> ----------------------------------------
>
> Key: HDFS-4849
> URL: https://issues.apache.org/jira/browse/HDFS-4849
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Affects Versions: 2.0.4-alpha
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Attachments: idempotentCreate.patch, idempotentCreate.patch,
> idempotentCreate.patch
>
>
> create, append and delete operations can be made idempotent. This will reduce
> chances for a job or other app failures when NN fails over.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
