[
https://issues.apache.org/jira/browse/FLUME-2922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323107#comment-15323107
]
Kevin Conaway commented on FLUME-2922:
--------------------------------------
{quote}
As for the hflush vs hsync: I don't know of any real-world HDFS-based system
that uses hsync, because the hsync implementation is known to be not-optimized
and hits performance pretty drastically. hflush is what most systems use - as
it flushes to datanode memory. There is data loss only if all 3 datanodes go
down before namenode detects under-replication and replicates the block - which
is really unlikely. I believe the performance cost may not be worth it.
{quote}
OK, thanks. My understanding of this is still limited. I had referenced the
HDFS storm integration as well as the Kafka HDFS Connector which both use hsync
but I don't know how "real world" those integration are (yet)
> HDFSSequenceFile Should Sync Writer
> -----------------------------------
>
> Key: FLUME-2922
> URL: https://issues.apache.org/jira/browse/FLUME-2922
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.6.0
> Reporter: Kevin Conaway
> Priority: Critical
>
> There is a possibility of losing data with the current HDFS sequence file
> writer.
> Internally, the `SequenceFile.Writer` buffers data and periodically syncs it
> to the underlying output stream. The mechanism for doing this is dependent
> on whether you are using compression or not but in both scenarios, the
> key/values are appended to an internal buffer and only flushed to disk after
> the buffer reaches a certain size.
> Thus it is quite possible for Flume to lose messages if the agent crashes, or
> is stopped, before the internal buffer is flushed to disk.
> The correct action is to force the writer to sync its internal buffers to the
> underlying `FSDataOutputStream` first before calling hflush/sync.
> Additionally, I believe we should be calling hsync instead of hflush. Its my
> understanding writes with hsync should be more durable which I believe are
> the semantics we want here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
