Hello James, Thank you for your reply!
The problem was that the JSON rows weren't new line delimited and hence it was only considering the last matched valid record and discarding everything else! Now that we are passing new line delimited records, it is able to convert the correct ones to Avro format and discarding the incompatible ones. Why should we split the JSON, won't it be an overkill of processing each record separately? Thanks you for the reply again. ______________________ *Kind Regards,* *Anshuman Ghosh* *Contact - +49 179 9090964* On Tue, Mar 14, 2017 at 9:15 PM, James Wing <[email protected]> wrote: > Anshuman, > > What kind of error messages are you seeing? If the input is > newline-delimited JSON records, I believe ConvertJSONToAvro should transfer > all valid records to the "success" relationship as a single Avro, and the > entire input flowfile to the "incompatible" output if any are invalid. > Does your "success" Avro contain more than one record? > > One thing you could do is split the JSON records before conversion, so they > will succeed or fail individually. > > Thanks, > > James > > On Tue, Mar 14, 2017 at 8:59 AM, Anshuman Ghosh < > [email protected]> wrote: > > > Hello Group/ Bryan, > > > > Anshu here. > > While trying the "*ConvertJSONToAvro*" processor, I came across the > > following issue. I am not sure whether it is correct or not though. > > So I have this input JSON file which might contain one or more valid/ > > invalid JSON records. Need to use this processor to convert them into > Avro > > format before I write and process them onto HDFS. > > However strangely in the output FlowFile, I am getting only 1 record > > converted (last valid JSON record is only getting converted). > > > > Following "*Record Schem*a" I am using for this purpose. > > > > *{"type": "record",* > > * "name": "ClickData",* > > * "fields": [* > > * {"name": "SuperShopId", "type": ["string", "null"]},* > > * {"name": "LinkId", "type": ["string", "null"]},* > > * {"name": "ClickId", "type": ["string", "null"]}, * > > * {"name": "VisitorId", "type": ["string", "null"]},* > > * {"name": "RawHTTPRequest", "type": ["string", "null"]},* > > * {"name": "RequestTimestamp", "type": "long"},* > > * {"name": "RedirectURL", "type": ["string", "null"]},* > > * {"name": "ResponseTimestamp", "type": "long"}* > > * ]* > > *}* > > > > Please let me know if I am doing something wrong! > > Thanking you in advance! > > > > > > ______________________ > > > > *Kind Regards,* > > *Anshuman Ghosh* > > *Contact - +49 179 9090964* > > >
