> On May 8, 2016, 9:16 p.m., Jarek Cecho wrote: > > flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceHandler.java, > > line 44 > > <https://reviews.apache.org/r/47098/diff/1/?file=1375930#file1375930line44> > > > > This will be a backward in-compatible change. Can we perhaps convert > > this interface to abstract class, move handling of the nullReplacement > > here, so that we will break the contract only once? > > > > E.g. that we won't have to break it somewhere in the future when we > > will need to add yet another argument. > > Hari Shreedharan wrote: > Unfortunately, changing this to an abstract class does not solve the > issue here - since that is a bytecode change and breaks compat (code too - > implements vs extends). One way of fixing this is to add an abstract class > that inherits this one, and have an instance of check in the Source itself, > and call this method only if the handler is an instance of the abstract class. > > neerja khattar wrote: > ahh i was just compiling flume-ng-core didnt realize had to compile whole > project for dependency check:) working on it now.. sorry about that.
Thanks hari. I am making the change - neerja ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47098/#review132183 ----------------------------------------------------------- On May 9, 2016, 4:57 a.m., neerja khattar wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/47098/ > ----------------------------------------------------------- > > (Updated May 9, 2016, 4:57 a.m.) > > > Review request for Flume. > > > Repository: flume-git > > > Description > ------- > > The issue is when the header value is null it throws null pointer exception > and flume stops processing further events. > For example: > [{ > "headers" : { > "timestamp" : "434324343", > "host" : null > }, > "body" : "random_body" > }] > > The solution to fix this is: > > 1. If the header has a null value in the json, flume will replace it with a > replacement string. > 2. The default value for a replacement string is an empty string. > 3. To overwrite default string, set "handler.nullReplacementHeader" > property in flume config. > > > Diffs > ----- > > flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSource.java > b520b03 > > flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceConfigurationConstants.java > 86caf7d > > flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceHandler.java > 726bf0c > flume-ng-core/src/main/java/org/apache/flume/source/http/JSONHandler.java > 197f66a > > flume-ng-core/src/test/java/org/apache/flume/source/http/TestJSONHandler.java > 455781c > > flume-ng-sinks/flume-ng-morphline-solr-sink/src/main/java/org/apache/flume/sink/solr/morphline/BlobHandler.java > e84dec1 > > Diff: https://reviews.apache.org/r/47098/diff/ > > > Testing > ------- > > The following are the test cases: > > 1. Header has null value in json and handler.nullReplacementHeader is not set > in flume config. The default value will be used to replace null. > > [{ > "headers" : { > "timestamp" : "434324343", > "host" : null > }, > "body" : "random_body" > }] > > Output in hdfs : {timestamp=434324343, host=} random_body > > 2. Header is not null in json and handler.nullReplacementHeader is not set > in flume config. The replacement implementation doesnt come in to > consideration. > > [{ > "headers" : { > "timestamp" : "434324343", > "host" : 1 > }, > "body" : "random_body" > }] > Output in hdfs : {timestamp=434324343, host=1} random_body > > 3. Header has null value in json and handler.nullReplacementHeader=abc is > set in flume config. The null value in header will be replaced by abc. > > > [{ > "headers" : { > "timestamp" : "434324343", > "host" : null > }, > "body" : "random_body" > }] > > > Output in hdfs {timestamp=434324343, host=abc} random_body > > 4. Header has null value in json and handler.nullReplacementHeader=1 is set > in flume config. The null value in header will be replaced by 1 as a string . > > [{ > "headers" : { > "timestamp" : "434324343", > "host" : null > }, > "body" : "random_body" > }] > > > Output in hdfs: {timestamp=434324343, host=1} random_body > > 5. Header is not null in json and handler.nullReplacementHeader is also set > in flume config. The replacement implementation doesnt come in to > consideration. > > [{ > "headers" : { > "timestamp" : "434324343", > "host" : 1 > }, > "body" : "random_body" > }] > Output in hdfs : {timestamp=434324343, host=1} random_body > > > File Attachments > ---------------- > > flume-2620 > > https://reviews.apache.org/media/uploaded/files/2016/05/09/0eff1d56-caf3-4d36-bb45-6b9e7fd6a1ff__FLUME-2620-1.patch > > > Thanks, > > neerja khattar > >
