Github user selvaganesang commented on a diff in the pull request:
https://github.com/apache/trafodion/pull/1612#discussion_r197957847
--- Diff: core/sql/src/main/java/org/trafodion/sql/HDFSClient.java ---
@@ -359,36 +392,54 @@ int hdfsWrite(byte[] buff) throws IOException
logger_.debug("HDFSClient.hdfsWrite() - output stream created"
);
}
outStream_.write(buff);
+ if (outStream_ instanceof FSDataOutputStream)
+ ((FSDataOutputStream)outStream_).hsync();
if (logger_.isDebugEnabled())
logger_.debug("HDFSClient.hdfsWrite() - bytes written " +
buff.length);
return buff.length;
}
--- End diff --
hdfsWrite doesn't close the open and it is assumed to single writer so that
we can do streaming write and close it when you are done. hdfsWriteImmediate
closes the stream for every write.
hdfsWriteImmediate uses two APIs. One API to get the position where it
would be written and another API to do the write. So, it is potentially
possible for a different thread and a process to sneak in. However we could do
some optimistic concurrency control concepts and avoid locking.
The locking mechanism is not changed. These discussions are just different
concepts that could be deployed to improve the locking mechanism and
hdfsWriteImmediate method would help to implement the improvements at a later
date.
---
