Thanks, Matt. will do. regarding "ValidateRecord" I'll look into that, I just had assumed that there was validation when you go from the RecordReader to RecordWritter since it would look at the schema, but even then I'm not sure avcroschema will let you validate timestamp, I'll have to look further.
On Thu, Apr 26, 2018 at 3:37 PM Matt Burgess <[email protected]> wrote: > Juan, > > Yes the current behavior is to move the entire input flowfile to > failure if any errors occur. Some other record-aware processors create > separate flow files for failed and successful records, but > PutElasticsearchHttpRecord does not (yet) do that. Please feel free to > write a Jira for this improvement. > > Since your failure is based on bad input data, try a ValidateRecord > before the ES processor, and send the "valid" relationship to the ES > processor. Then you can handle the invalid records separately, before > trying to put them into ES (which in your case you know will fail). > > Regards, > Matt > > > On Thu, Apr 26, 2018 at 1:28 PM, Juan Sequeiros <[email protected]> > wrote: > > Hello all, > > > > I am using PutElasticSearchRecord. > > My Reader is JsonTreeReader that validates against an avro schema I've > > created. > > > > It passes probably since I don't have a good Avro schema for it since my > > timestamp key is declared as a string. > > > > My observation and not sure if working as expected but when I send my > output > > from MergeRecord ( set to 10000 ) max number of records and one of those > > records has an invalid timestamp "bogusdata" value ES rejects it, > rightly so > > since on ES we have a schema template more granular and is expecting > > timestamp as type "date". > > > > However what I see happening is that my whole "mergedRecord" is rejected > and > > not just that one record. Is that expected? > > > > thanks > > > > > > > > > > > > >
