-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47098/
-----------------------------------------------------------
(Updated May 12, 2016, 1:03 a.m.)
Review request for Flume.
Changes
-------
Implemented the following change: add an abstract class that inherits this one,
and have an instance of check in the Source itself, and call this method only
if the handler is an instance of the abstract class
Repository: flume-git
Description
-------
The issue is when the header value is null it throws null pointer exception and
flume stops processing further events.
For example:
[{
"headers" : {
"timestamp" : "434324343",
"host" : null
},
"body" : "random_body"
}]
The solution to fix this is:
1. If the header has a null value in the json, flume will replace it with a
replacement string.
2. The default value for a replacement string is an empty string.
3. To overwrite default string, set "handler.nullReplacementHeader" property
in flume config.
Diffs (updated)
-----
flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSource.java
b520b03
flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceConfigurationConstants.java
86caf7d
flume-ng-core/src/main/java/org/apache/flume/source/http/JSONHandler.java
197f66a
flume-ng-core/src/main/java/org/apache/flume/source/http/NullHeaderReplacement.java
PRE-CREATION
flume-ng-core/src/test/java/org/apache/flume/source/http/TestJSONHandler.java
455781c
Diff: https://reviews.apache.org/r/47098/diff/
Testing
-------
The following are the test cases:
1. Header has null value in json and handler.nullReplacementHeader is not set
in flume config. The default value will be used to replace null.
[{
"headers" : {
"timestamp" : "434324343",
"host" : null
},
"body" : "random_body"
}]
Output in hdfs : {timestamp=434324343, host=} random_body
2. Header is not null in json and handler.nullReplacementHeader is not set in
flume config. The replacement implementation doesnt come in to consideration.
[{
"headers" : {
"timestamp" : "434324343",
"host" : 1
},
"body" : "random_body"
}]
Output in hdfs : {timestamp=434324343, host=1} random_body
3. Header has null value in json and handler.nullReplacementHeader=abc is set
in flume config. The null value in header will be replaced by abc.
[{
"headers" : {
"timestamp" : "434324343",
"host" : null
},
"body" : "random_body"
}]
Output in hdfs {timestamp=434324343, host=abc} random_body
4. Header has null value in json and handler.nullReplacementHeader=1 is set
in flume config. The null value in header will be replaced by 1 as a string .
[{
"headers" : {
"timestamp" : "434324343",
"host" : null
},
"body" : "random_body"
}]
Output in hdfs: {timestamp=434324343, host=1} random_body
5. Header is not null in json and handler.nullReplacementHeader is also set
in flume config. The replacement implementation doesnt come in to consideration.
[{
"headers" : {
"timestamp" : "434324343",
"host" : 1
},
"body" : "random_body"
}]
Output in hdfs : {timestamp=434324343, host=1} random_body
File Attachments
----------------
flume-2620
https://reviews.apache.org/media/uploaded/files/2016/05/09/0eff1d56-caf3-4d36-bb45-6b9e7fd6a1ff__FLUME-2620-1.patch
Thanks,
neerja khattar
