[
https://issues.apache.org/jira/browse/FLUME-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258766#comment-13258766
]
[email protected] commented on FLUME-1097:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4826/#review7102
-----------------------------------------------------------
Thanks for the patch, Inder! This looks to be very useful. Below are a few
suggestions.
flume-ng-core/src/main/java/org/apache/flume/event/EventHeaderDecorator.java
<https://reviews.apache.org/r/4826/#comment15714>
This abstract class should probably be an interface (it has no
implementation).
flume-ng-core/src/main/java/org/apache/flume/event/EventHeaderDecoratorFactory.java
<https://reviews.apache.org/r/4826/#comment15716>
Please don't use static variables. Just create a new instance every time
from this factory.
flume-ng-core/src/main/java/org/apache/flume/event/SinkEventHeaderDecorator.java
<https://reviews.apache.org/r/4826/#comment15717>
Does this class belong in the core at all?
In my view, this class isn't general enough to be checked into the Flume
core. This should be kept as an application-specific plugin which is added to
the runtime classpath and referenced in the config. That's sort of the point of
adding this EventHeaderDecorator plugin interface, right?
- Mike
On 2012-04-20 10:33:14, Inder Singh wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4826/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-04-20 10:33:14)
bq.
bq.
bq. Review request for Flume, Arvind Prabhakar, Mike Percy, and Hari
Shreedharan.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Overview of Changes
bq.
bq. 1.EventHeaderDecorator -> generic class having processHeader()
bq. 2.SinkEventHeaderDecorator is a EventHeaderDecorator which overrides
timestamp header at this hop.
bq. 3.EventHeaderDecoratorFactory provides EventHeaderDecorator's based on
<sink>.headerDecorator config
bq. 4.AbstractSink -> has a eventHeaderDecortaor
bq. 5.HDFSEventSink -> checks for decorator through config and calls
processHeader() for each event which modifies/add headers based on event
decorator defined through config.
bq.
bq. Sample Configuration for testing
bq. ----------------------------------
bq. agent1.sinks.log-sink1.hdfs.headerDecorator = sinkDecorator
bq.
bq.
bq. This addresses bug FLUME-1097.
bq. https://issues.apache.org/jira/browse/FLUME-1097
bq.
bq.
bq. Diffs
bq. -----
bq.
bq.
flume-ng-core/src/main/java/org/apache/flume/event/EventHeaderDecorator.java
PRE-CREATION
bq.
flume-ng-core/src/main/java/org/apache/flume/event/EventHeaderDecoratorFactory.java
PRE-CREATION
bq.
flume-ng-core/src/main/java/org/apache/flume/event/SinkEventHeaderDecorator.java
PRE-CREATION
bq. flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java
2334059
bq.
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
1b61cad
bq.
bq. Diff: https://reviews.apache.org/r/4826/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Testing Done
bq. -------------
bq. 1. Unit tests
bq. 2. manual tests done - agent1.sinks.log-sink1.hdfs.filePrefix =
flume-data/%D results in publishing data with timestamp at agent's hop running
HDFSEventSink
bq.
bq.
bq. Thanks,
bq.
bq. Inder
bq.
bq.
> Support modification of event Headers through configuration
> -----------------------------------------------------------
>
> Key: FLUME-1097
> URL: https://issues.apache.org/jira/browse/FLUME-1097
> Project: Flume
> Issue Type: New Feature
> Reporter: Inder SIngh
> Assignee: Inder SIngh
> Priority: Minor
> Attachments: FLUME-1097-1.patch
>
>
> As discussed on the flume threads we want to support adding/modifying headers
> at flume agents through configuration.
> Primary use-case as of now -
> Use HDFSEventSink to publish data in paths like /flume-data/YYYY/mm/DD/HH/MM
> wherein timestamp is of agent running HDFSEventSink.
> What's not there today
> -------------------------
> 1. headers are put by clientLIB and can't be added/modified in flight by
> agents.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
