[jira] [Created] (FLUME-2390) Flume-ElasticSearch Data gets posted multiple times when one of the event fail validation at elastic search sink for JSON Data

[Deepak Subhramanian (JIRA)]

Deepak Subhramanian created FLUME-2390:
------------------------------------------

             Summary: Flume-ElasticSearch Data gets posted multiple times when 
one of the event fail validation at elastic search sink for JSON Data
                 Key: FLUME-2390
                 URL: https://issues.apache.org/jira/browse/FLUME-2390
             Project: Flume
          Issue Type: Bug
          Components: Sinks+Sources
    Affects Versions: v1.4.0
         Environment: CDH4.5
            Reporter: Deepak Subhramanian


Hi,

I am using Elastic Search Sink to post JSON data. I used the temporary fix 
mentioned in https://issues.apache.org/jira/browse/FLUME-2126 to get JSON data 
posted to elastic search. When one of the message fail validation at 
ElasticSearch mapping for JSON data ( For example - getting empty message) , 
Flume seems to post the entire batch again and again until I restart Flume.  
Because of that no of events went from an avg of 100 to avg of 2000 per 10 
minutes. As a temporary fix I set a header in my FlumeHTTP Source for non valid 
JSON and used a interceptor to send data to multiple ESSINKS which has 
different index names. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)