[
https://issues.apache.org/jira/browse/FLUME-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162702#comment-16162702
]
Bessenyei Balázs Donát commented on FLUME-3149:
-----------------------------------------------
Hi [~zyfo2],
I have added you to the contributor group so now you should be able to assign
tickets to yourself.
For the commits: I don't think each commit shows up here, only opening and
closing PRs.
I'll take a look at the patch ASAP and hope that other contributors can also
provide their comments in the meantime.
Thanks,
Donat
> reduce cpu cost for file source transfer while still maintaining reliability
> ----------------------------------------------------------------------------
>
> Key: FLUME-3149
> URL: https://issues.apache.org/jira/browse/FLUME-3149
> Project: Flume
> Issue Type: Improvement
> Components: File Channel
> Reporter: Will Zhang
>
> File channel tracks transferred events and use transnational mechanism to
> make transfer recoverable. However, it increases CPU cost due to frequent
> system calls like write, read, etc. The Cpu cost could be very high if the
> transfer rate is high. In contrast, Memory channel has no such issue which
> requires only about 10% of CPU cost in the same environment but it's not
> recovered if the system is down accidentally.
> For sources like taildir/spooldir, I propose we could track offsets of file
> and store them locally to achieve reliability while still using memory
> channel to reduce CPU cost. Actually, I have already implemented this feature
> by storing the offsets in event headers and passing it to my own
> "offsetMemoryChannel" and store theses offsets in local disk in our
> production which reduces CPU cost by about 90 percent.
> Please let me know if it's worthwhile to have this feature in community
> version. Thank you.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
