[
https://issues.apache.org/jira/browse/FLUME-2250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136408#comment-14136408
]
Gwen Shapira commented on FLUME-2250:
-------------------------------------
Kafka's AutoCommit:
If autocommit is set to true, Kafka will commit offset for messages in a
background thread every 10 seconds (IIRC the default).
This means that if Flume is unable to write to channel - messages that we
consumed may be lost when autocommit happens.
It also means that if Flume agent crashes, we don't know if we lost messages in
the buffer that were not written to channel yet, or if we will read the same
messages twice because autocommit did not happen.
If autocommit is disabled, the Kafka Source will commit on every batch. This
can slow down ingest rates if the batches are small, but is far safer option.
We recommend autocommit=false. I'll make sure we document this.
> Add support for Kafka Source
> ----------------------------
>
> Key: FLUME-2250
> URL: https://issues.apache.org/jira/browse/FLUME-2250
> Project: Flume
> Issue Type: Sub-task
> Components: Sinks+Sources
> Affects Versions: v1.5.0
> Reporter: Ashish Paliwal
> Priority: Minor
> Attachments: FLUME-2250-0.patch, FLUME-2250-1.patch,
> FLUME-2250-2.patch, FLUME-2250.patch
>
>
> Add support for Kafka Source
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
