[
https://issues.apache.org/jira/browse/FLUME-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13425977#comment-13425977
]
Mike Percy commented on FLUME-1391:
-----------------------------------
So, does sync() in the latest versions call hsync() ? Have we verified that
this change is actually durable?
Note that Hadoop 0.20.2 does not support hsync and so it's not durable, and
therefore Flume NG cannot really support that version.
> Use sync() instead of syncFs() in HDFS Sink to be compatible with hadoop
> 0.20.2
> -------------------------------------------------------------------------------
>
> Key: FLUME-1391
> URL: https://issues.apache.org/jira/browse/FLUME-1391
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
> Affects Versions: v1.1.0
> Reporter: Yongkun Wang
> Assignee: Yongkun Wang
> Labels: hadoop
> Fix For: v1.3.0
>
> Attachments: HDFSSink-for-hadoop-0.20.2.patch
>
>
> For HDFS sink, the syncFs() is called in HDFSSequenceFile. But syncFs() is
> not available in legacy hadoop 0.20.2, which may be a widely used version.
> sync() method is available for all hadoop versions. And syncFs() is also
> implemented by sync() in hadoop (SequenceFile):
> {code}
> /** create a sync point */
> public void sync() throws IOException {
> if (sync != null && lastSyncPos != out.getPos()) {
> out.writeInt(SYNC_ESCAPE); // mark the start of the
> sync
> out.write(sync); // write sync
> lastSyncPos = out.getPos(); // update lastSyncPos
> }
> }
> /** flush all currently written data to the file system */
> public void syncFs() throws IOException {
> if (out != null) {
> out.sync(); // flush contents to file
> system
> }
> }
> {code}
> Therefore, using sync() in HDFSSequenceFile may be better.
> {code}
> @Override
> public void sync() throws IOException {
> //writer.syncFs(); //for hadoop 0.20.205.0+
> writer.sync(); //support hadoop 0.20.2+
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
