[
https://issues.apache.org/jira/browse/HTRACE-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14244993#comment-14244993
]
Long Zhou commented on HTRACE-18:
---------------------------------
Hi Colin,
Thanks for taking the time to review my request.
To answer your question, Flume aggregates data from multiple servers in remote
data center, forward to local Flume agent, which then writes to storage sinks.
Flume supports various types of sinks, HDFS, various databases, etc. We usually
store trace to MongoDB, and sometimes to HDFS when data become too large. You
are right that we can't easily analyze HDFS files, and we only use it when have
to. Flume supports HBase sink; I am not sure about LevelDB though.
Flume allows us to change configurations dynamically, e.g. switch sinks or
update filters. Because there are usually much fewer Flume servers than Hadoop
servers, it is easier to change. We usually only need to change the local Flume
agents. Also Flume allows us to filter on headers (like span ProcessId or
Description), and redirect or drop events. This feature is quite handy.
Flume is similar to Scribe. We already use flume for other types of data, so it
is easier to add trace data to the existing infrastructure than to switch to
Zipkin + Scribe.
I can remove Avro dependency and refactor the existing Json converter. I
originally implemented Avro in hope to conserve storage size and bandwidth, but
did not really use it in practice. I can also look into removing Guava
dependency.
Thanks,
Long
> Support flume receiver
> ----------------------
>
> Key: HTRACE-18
> URL: https://issues.apache.org/jira/browse/HTRACE-18
> Project: HTrace
> Issue Type: Improvement
> Reporter: Long Zhou
> Attachments: htrace-flume01.patch
>
>
> Hi htrace devs,
> I have been using htrace for a while and find it very useful.
> I needed a way to collect traces from remote servers via flume, so I
> implemented the flume receiver (patch attached). If this code is useful to
> other users, I would like to contribute it to the project.
> Please kindly review the patch, and let me know if anything I should
> fix/improve.
> Thanks,
> Long Zhou
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
