[
https://issues.apache.org/jira/browse/FLUME-2010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom White updated FLUME-2010:
-----------------------------
Attachment: FLUME-2010.patch
I had a closer look at AVRO-1124 and wrote a patch that allows the
Log4jAppender or the HDFS sink to use a Repository to lookup schemas (and
register them if they are not already there). With this approach there is no
need for the user to use the MDC to register the schema, so I think it will be
an improvement.
Since AVRO-1124 isn't available yet I changed my patch to use a
manually-generated properties file to provide a mapping between ID and schema.
The ID is the 64-bit fingerprint. The idea is that you generate this file for
you schemas and then distribute it to every flume node. I updated the example
at https://github.com/tomwhite/flume-log4j-example/tree/avro. (Once AVRO-1124
is in a release it will be straightforward to update Log4jAppender to use it.)
This version of the patch adds unit tests too.
> Support Avro records in Log4jAppender and the HDFS Sink
> -------------------------------------------------------
>
> Key: FLUME-2010
> URL: https://issues.apache.org/jira/browse/FLUME-2010
> Project: Flume
> Issue Type: New Feature
> Components: Client SDK, Sinks+Sources
> Reporter: Tom White
> Assignee: Tom White
> Fix For: v1.4.0
>
> Attachments: FLUME-2010.patch, FLUME-2010.patch, FLUME-2010.patch
>
>
> It would be nice to support logging arbitrary Avro records via the Log4j
> Flume logger, and have them written to HDFS in Avro data files (using an
> appropriately configured HDFS sink).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
