[
https://issues.apache.org/jira/browse/FLUME-1507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13440066#comment-13440066
]
Jarek Jarcec Cecho commented on FLUME-1507:
-------------------------------------------
I would also recommend to mention some concrete layer designs and their
benefits.
At top of my head, I would mention that there might be just single layer -
flume agents running exactly on source machines and sending data directly to
target destination (HDFS). Advantage is that you do not need any extra
machines, disadvantage is limited ability for "caching" data in case that
target destination is down or creating a high number of files on HDFS. Another
layering design would be to have two layers, where second layer would be very
close to "collectors" in flume OG...
Jarcec
> Have "Topology Design Considerations" in User Guide
> ---------------------------------------------------
>
> Key: FLUME-1507
> URL: https://issues.apache.org/jira/browse/FLUME-1507
> Project: Flume
> Issue Type: Improvement
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
>
> It would be nice if the User Guide had a section which lists considerations
> for designing an end-to-end flume topology. I think a lot of people get lost
> in the long list of sources/sinks/etc and need a higher level overview of
> what to think about when designing a flow. Examples would be:
> - When to use Flume? Types of data that flume is good at handling (e.g.
> regularly generated, event-based, etc).
> - Reliability (explaining that flow reliability is a function of channels
> used, redundancy in the flow, and other factors)
> - Flume sizing (some basic ideas about how to size the nodes or network you
> are running on)
> The design space is too large here to give precise recommendations, but just
> orienting users that these are the main things they need to be thinking about
> would be really helpful. Some issues like reliability are much harder to
> explain for Flume-ng than OG, and I think we need documentation making this
> explicit. Down the road, a "cookbook" with specific examples would be even
> better.
> Thoughts?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
