[jira] [Updated] (FLUME-2907) Correct Flume NetCat Source agent Example

[Neerja Khattar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neerja Khattar updated FLUME-2907:
--
Attachment: FLUME-2907.patch

Patch is attached

> Correct Flume NetCat Source agent Example
> -
>
> Key: FLUME-2907
> URL: https://issues.apache.org/jira/browse/FLUME-2907
> Project: Flume
>  Issue Type: Bug
>Reporter: Abbass Marouni
>Assignee: Neerja Khattar
>Priority: Minor
> Attachments: FLUME-2907.patch
>
>
> In NetCat Source agent Example 
> (https://flume.apache.org/FlumeUserGuide.html#netcat-source) :
> {code}a1.sources.r1.bind{code} is  defined 2 times :
> {code}
> a1.sources = r1
> a1.channels = c1
> a1.sources.r1.type = netcat
> a1.sources.r1.bind = 0.0.0.0
> a1.sources.r1.bind = 
> a1.sources.r1.channels = c1
> {code}
> This should be corrected to :
> {code}
> a1.sources = r1
> a1.channels = c1
> a1.sources.r1.type = netcat
> a1.sources.r1.bind = 0.0.0.0
> a1.sources.r1.port = 
> a1.sources.r1.channels = c1
> {code}
> People might not pay attention when copying the configurations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (FLUME-2907) Correct Flume NetCat Source agent Example

[Neerja Khattar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neerja Khattar reassigned FLUME-2907:
-

Assignee: Neerja Khattar

> Correct Flume NetCat Source agent Example
> -
>
> Key: FLUME-2907
> URL: https://issues.apache.org/jira/browse/FLUME-2907
> Project: Flume
>  Issue Type: Bug
>Reporter: Abbass Marouni
>Assignee: Neerja Khattar
>Priority: Minor
>
> In NetCat Source agent Example 
> (https://flume.apache.org/FlumeUserGuide.html#netcat-source) :
> {code}a1.sources.r1.bind{code} is  defined 2 times :
> {code}
> a1.sources = r1
> a1.channels = c1
> a1.sources.r1.type = netcat
> a1.sources.r1.bind = 0.0.0.0
> a1.sources.r1.bind = 
> a1.sources.r1.channels = c1
> {code}
> This should be corrected to :
> {code}
> a1.sources = r1
> a1.channels = c1
> a1.sources.r1.type = netcat
> a1.sources.r1.bind = 0.0.0.0
> a1.sources.r1.port = 
> a1.sources.r1.channels = c1
> {code}
> People might not pay attention when copying the configurations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 47098: FLUME-2620 File channel throws NullPointerException if a header value is null

[neerja khattar]

> On May 8, 2016, 9:16 p.m., Jarek Cecho wrote:
> > I'm getting the following when I tried to compile flume after applying the 
> > patch:
> > 
> > [ERROR] Failed to execute goal 
> > org.apache.maven.plugins:maven-compiler-plugin:2.3.2:compile 
> > (default-compile) on project flume-ng-core: Compilation failure
> > [ERROR] 
> > /Users/jarcec/apache/flume/flume-ng-core/src/main/java/org/apache/flume/source/http/BLOBHandler.java:[43,7]
> >  error: BLOBHandler is not abstract and does not override abstract method 
> > setNullHeaderReplacement(String) in HTTPSourceHandler
> > [ERROR] -> [Help 1]
> > 
> > Which points to:

Jarec, I have made the following change and attached the patch.
we perhaps convert this interface to abstract class, move handling of the 
nullReplacement here, so that we will break the contract only once

Let me know whats the next step.


> On May 8, 2016, 9:16 p.m., Jarek Cecho wrote:
> > flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceHandler.java,
> >  line 44
> > 
> >
> > This will be a backward in-compatible change. Can we perhaps convert 
> > this interface to abstract class, move handling of the nullReplacement 
> > here, so that we will break the contract only once?
> > 
> > E.g. that we won't have to break it somewhere in the future when we 
> > will need to add yet another argument.
> 
> Hari Shreedharan wrote:
> Unfortunately, changing this to an abstract class does not solve the 
> issue here - since that is a bytecode change and breaks compat (code too - 
> implements vs extends). One way of fixing this is to add an abstract class 
> that inherits this one, and have an instance of check in the Source itself, 
> and call this method only if the handler is an instance of the abstract class.

ahh i was just compiling flume-ng-core didnt realize had to compile whole 
project for dependency check:) working on it now.. sorry about that.


- neerja


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47098/#review132183
---


On May 9, 2016, 4:57 a.m., neerja khattar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47098/
> ---
> 
> (Updated May 9, 2016, 4:57 a.m.)
> 
> 
> Review request for Flume.
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> The issue is when the header value is null it throws null pointer exception 
> and flume stops processing further events.
> For example:
> [{
>   "headers" : {
>  "timestamp" : "434324343",
>  "host" : null
>  },
>   "body" : "random_body"
>   }]
>   
>   The solution to fix this is:
>   
>   1. If the header has a null value in the json, flume will replace it with a 
> replacement string.
>   2. The default value for a replacement string is an empty string.
>   3. To overwrite default string, set "handler.nullReplacementHeader" 
> property in flume config.
> 
> 
> Diffs
> -
> 
>   flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSource.java 
> b520b03 
>   
> flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceConfigurationConstants.java
>  86caf7d 
>   
> flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceHandler.java
>  726bf0c 
>   flume-ng-core/src/main/java/org/apache/flume/source/http/JSONHandler.java 
> 197f66a 
>   
> flume-ng-core/src/test/java/org/apache/flume/source/http/TestJSONHandler.java 
> 455781c 
>   
> flume-ng-sinks/flume-ng-morphline-solr-sink/src/main/java/org/apache/flume/sink/solr/morphline/BlobHandler.java
>  e84dec1 
> 
> Diff: https://reviews.apache.org/r/47098/diff/
> 
> 
> Testing
> ---
> 
> The following are the test cases:
> 
> 1. Header has null value in json and handler.nullReplacementHeader is not set 
> in flume config. The default value will be used to replace null.  
> 
> [{
>   "headers" : {
>  "timestamp" : "434324343",
>  "host" : null
>  },
>   "body" : "random_body"
>   }]
>   
>   Output in hdfs : {timestamp=434324343, host=} random_body  
>   
>   2. Header is not null in json and handler.nullReplacementHeader is not set 
> in flume config. The replacement implementation doesnt come in to 
> consideration.
>
>   [{
>   "headers" : {
>  "timestamp" : "434324343",
>  "host" : 1
>  },
>   "body" : "random_body"
>   }]
>   Output in hdfs : {timestamp=434324343, host=1} random_body 
>   
>   3. Header has null value in json and handler.nullReplacementHeader=abc is 
> set in flume config. The null value in header will be replaced by abc.
> 
>   
>   [{
>   "headers" : {
>  "timestamp"