----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47098/#review140181 -----------------------------------------------------------
This doesn't look like it changes the file channel at all. Isn't that the underlying issue? Couldn't the FileChannel allow a null value in a header, or treat it as an empty string? Maybe that would be a cleaner solution. - Mike Percy On May 24, 2016, 4:11 p.m., neerja khattar wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/47098/ > ----------------------------------------------------------- > > (Updated May 24, 2016, 4:11 p.m.) > > > Review request for Flume. > > > Repository: flume-git > > > Description > ------- > > The issue is when the header value is null it throws null pointer exception > and flume stops processing further events. > For example: > [{ > "headers" : { > "timestamp" : "434324343", > "host" : null > }, > "body" : "random_body" > }] > > The solution to fix this is: > > 1. If the header has a null value in the json, flume will replace it with a > replacement string. > 2. The default value for a replacement string is an empty string. > 3. To overwrite default string, set "handler.nullReplacementHeader" > property in flume config. > > > Diffs > ----- > > flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSource.java > b520b03 > > flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceConfigurationConstants.java > 86caf7d > flume-ng-core/src/main/java/org/apache/flume/source/http/JSONHandler.java > 197f66a > > flume-ng-core/src/main/java/org/apache/flume/source/http/NullHeaderReplacement.java > PRE-CREATION > > flume-ng-core/src/test/java/org/apache/flume/source/http/TestJSONHandler.java > 455781c > flume-ng-doc/sphinx/FlumeUserGuide.rst e04fd50 > > Diff: https://reviews.apache.org/r/47098/diff/ > > > Testing > ------- > > The following are the test cases: > > 1. Header has null value in json and handler.nullReplacementHeader is not set > in flume config. The default value will be used to replace null. > > [{ > "headers" : { > "timestamp" : "434324343", > "host" : null > }, > "body" : "random_body" > }] > > Output in hdfs : {timestamp=434324343, host=} random_body > > 2. Header is not null in json and handler.nullReplacementHeader is not set > in flume config. The replacement implementation doesnt come in to > consideration. > > [{ > "headers" : { > "timestamp" : "434324343", > "host" : 1 > }, > "body" : "random_body" > }] > Output in hdfs : {timestamp=434324343, host=1} random_body > > 3. Header has null value in json and handler.nullReplacementHeader=abc is > set in flume config. The null value in header will be replaced by abc. > > > [{ > "headers" : { > "timestamp" : "434324343", > "host" : null > }, > "body" : "random_body" > }] > > > Output in hdfs {timestamp=434324343, host=abc} random_body > > 4. Header has null value in json and handler.nullReplacementHeader=1 is set > in flume config. The null value in header will be replaced by 1 as a string . > > [{ > "headers" : { > "timestamp" : "434324343", > "host" : null > }, > "body" : "random_body" > }] > > > Output in hdfs: {timestamp=434324343, host=1} random_body > > 5. Header is not null in json and handler.nullReplacementHeader is also set > in flume config. The replacement implementation doesnt come in to > consideration. > > [{ > "headers" : { > "timestamp" : "434324343", > "host" : 1 > }, > "body" : "random_body" > }] > Output in hdfs : {timestamp=434324343, host=1} random_body > > > File Attachments > ---------------- > > flume-2620 > > https://reviews.apache.org/media/uploaded/files/2016/05/09/0eff1d56-caf3-4d36-bb45-6b9e7fd6a1ff__FLUME-2620-1.patch > > > Thanks, > > neerja khattar > >
