Hi, to me, it seems from the first picture, that you are routing the "split" connection out of SpliJson back to SplitJson again. And the failure on splitting $.Records happens when already splitted files - which are routed to split connection from SplitJson are tried to be splitted again on $.Records, which, of course, is not present at the moment.
On Fri, Oct 28, 2016 at 12:15 PM, Arsalan Siddiqi <[email protected]> wrote: > Hi > I want to get data from the REST server and make it available in NiFi. It > makes sense to have it in avro format so once read i want to convert the > data to avro. The server delivers data in JSON format. I have tested the > JSON and it is in the correct format. Have a look at the image to check the > flow. > I use Apache NiFi version 1.0.0. Firstly the split json processor is > correctly configured to break the incoming data into one record per file. I > do not know why but the correctly split jsons are routed to the failure > relationship. > <http://apache-nifi-developer-list.39713.n7.nabble.com/file/n13769/JSON_AVRO.png> > The data fetched from the get http (1) request is attached below.there are a > total of 10 records in the file. > json_adabas_input.txt > <http://apache-nifi-developer-list.39713.n7.nabble.com/file/n13769/json_adabas_input.txt> > > The format is JSON and the file is without any errors. i do a split on > $.Records which correctly splits the input with one record per file and I do > not know why routes it to failure. > The flow file routed to failure has the following content. (2) > {"ISN":200,"AA":"50005900","AB":{"AC":"GASTON","AD":"","AE":"BODIN"},"AF":"S","AG":"M","AH":713043,"AW":[{"AX":19990801,"AY":19990831}]} > > which is correct and valid JSON. > At 3 I try to convert the above JSON to avro: > Using the avro schema: > > { "type" : "record", > "name" : "adabas", > "fields" : [ > { "name" : "ISN", "type" : "int"}, > { "name" : "AA", "type" : "string"}, > { "name" : "AB", "type" : { "type" : "record", "name" : "AB", "fields" : > [ { "name" : "AC", "type" : "string"}, > { "name" : "AD", "type" : "string"}, > { "name" : "AE", "type" : "string" } ] } }, > { "name" : "AF", "type" : "string"}, > { "name" : "AG", "type" : "string"}, > { "name" : "AH", "type" : "int" }, > { "name" : "AW", "type" : { "type" : "array", "items" : { "type" : > "record", "name" : "AW", "fields" : > [ { "name" : "AX", "type" : "int" }, > { "name" : "AY", "type" : "int" }]}}} ] } > Here is the resulting avro file which is routed to success: > ConvertJSONtoAvro.ConvertJSONtoAvro > <http://apache-nifi-developer-list.39713.n7.nabble.com/file/n13769/ConvertJSONtoAvro.ConvertJSONtoAvro> > > Now i expected if the file was routed to success, it would have been > correctly converted so I can simple get it back in JSON. > In the convert avro to json (4) i configured the avro schema field to the > same as the schema mentioned above and i get the follwoing error: > > 2016-10-28 12:08:41,439 ERROR [Timer-Driven Process Thread-7] > o.a.n.processors.avro.ConvertAvroToJSON > ConvertAvroToJSON[id=fb761b66-1010-1157-4673-c5198a522367] Failed to convert > StandardFlowFileRecord[uuid=3f234e05-6fa9-4569-9660-c2513f483016,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1477643564106-1, container=default, > section=1], offset=9094, length=602],offset=0,name=adabas,size=602] from > Avro to JSON due to org.apache.nifi.processor.exception.ProcessException: > IOException thrown from > ConvertAvroToJSON[id=fb761b66-1010-1157-4673-c5198a522367]: > java.io.EOFException; transferring to failure: > org.apache.nifi.processor.exception.ProcessException: IOException thrown > from ConvertAvroToJSON[id=fb761b66-1010-1157-4673-c5198a522367]: > java.io.EOFException > 2016-10-28 12:08:41,440 ERROR [Timer-Driven Process Thread-7] > o.a.n.processors.avro.ConvertAvroToJSON > org.apache.nifi.processor.exception.ProcessException: IOException thrown > from ConvertAvroToJSON[id=fb761b66-1010-1157-4673-c5198a522367]: > java.io.EOFException > at > org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2388) > ~[na:na] > at > org.apache.nifi.processors.avro.ConvertAvroToJSON.onTrigger(ConvertAvroToJSON.java:148) > ~[nifi-avro-processors-1.0.0.jar:1.0.0] > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > [nifi-api-1.0.0.jar:1.0.0] > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1064) > [nifi-framework-core-1.0.0.jar:1.0.0] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) > [nifi-framework-core-1.0.0.jar:1.0.0] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) > [nifi-framework-core-1.0.0.jar:1.0.0] > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) > [nifi-framework-core-1.0.0.jar:1.0.0] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_101] > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > [na:1.8.0_101] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > [na:1.8.0_101] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > [na:1.8.0_101] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_101] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_101] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101] > Caused by: java.io.EOFException: null > at org.apache.avro.io.BinaryDecoder.readInt(BinaryDecoder.java:153) > ~[avro-1.7.7.jar:1.7.7] > at > org.apache.avro.io.ValidatingDecoder.readInt(ValidatingDecoder.java:83) > ~[avro-1.7.7.jar:1.7.7] > at > org.apache.avro.generic.GenericDatumReader.readInt(GenericDatumReader.java:444) > ~[avro-1.7.7.jar:1.7.7] > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:159) > ~[avro-1.7.7.jar:1.7.7] > at > org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193) > ~[avro-1.7.7.jar:1.7.7] > at > org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183) > ~[avro-1.7.7.jar:1.7.7] > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151) > ~[avro-1.7.7.jar:1.7.7] > at > org.apache.avro.generic.GenericDatumReader.readArray(GenericDatumReader.java:219) > ~[avro-1.7.7.jar:1.7.7] > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153) > ~[avro-1.7.7.jar:1.7.7] > at > org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193) > ~[avro-1.7.7.jar:1.7.7] > at > org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183) > ~[avro-1.7.7.jar:1.7.7] > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151) > ~[avro-1.7.7.jar:1.7.7] > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142) > ~[avro-1.7.7.jar:1.7.7] > at > org.apache.nifi.processors.avro.ConvertAvroToJSON$1.process(ConvertAvroToJSON.java:161) > ~[nifi-avro-processors-1.0.0.jar:1.0.0] > at > org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2370) > ~[na:na] > ... 13 common frames omitted > > > As Nifi has made the conversion to avro I expected it should easily convert > it back without any error.] > In this case the file is transferred to failure and nothing happens to the > file. > > > > If I remove the configured value set at (4) for the field avro schema and > leave it empty i get the following error: > > 2016-10-28 13:25:42,968 ERROR [Timer-Driven Process Thread-4] > o.a.n.processors.avro.ConvertAvroToJSON > ConvertAvroToJSON[id=fb761b66-1010-1157-4673-c5198a522367] Failed to convert > StandardFlowFileRecord[uuid=3f234e05-6fa9-4569-9660-c2513f483016,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1477643564106-1, container=default, > section=1], offset=17978, length=157],offset=0,name=adabas,size=157] from > Avro to JSON due to org.apache.nifi.processor.exception.ProcessException: > IOException thrown from > ConvertAvroToJSON[id=fb761b66-1010-1157-4673-c5198a522367]: > java.io.IOException: Not a data file.; transferring to failure: > org.apache.nifi.processor.exception.ProcessException: IOException thrown > from ConvertAvroToJSON[id=fb761b66-1010-1157-4673-c5198a522367]: > java.io.IOException: Not a data file. > 2016-10-28 13:25:42,969 ERROR [Timer-Driven Process Thread-4] > o.a.n.processors.avro.ConvertAvroToJSON > org.apache.nifi.processor.exception.ProcessException: IOException thrown > from ConvertAvroToJSON[id=fb761b66-1010-1157-4673-c5198a522367]: > java.io.IOException: Not a data file. > at > org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2388) > ~[na:na] > at > org.apache.nifi.processors.avro.ConvertAvroToJSON.onTrigger(ConvertAvroToJSON.java:148) > ~[nifi-avro-processors-1.0.0.jar:1.0.0] > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > [nifi-api-1.0.0.jar:1.0.0] > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1064) > [nifi-framework-core-1.0.0.jar:1.0.0] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) > [nifi-framework-core-1.0.0.jar:1.0.0] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) > [nifi-framework-core-1.0.0.jar:1.0.0] > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) > [nifi-framework-core-1.0.0.jar:1.0.0] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_101] > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > [na:1.8.0_101] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > [na:1.8.0_101] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > [na:1.8.0_101] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_101] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_101] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101] > Caused by: java.io.IOException: Not a data file. > at > org.apache.avro.file.DataFileStream.initialize(DataFileStream.java:105) > ~[avro-1.7.7.jar:1.7.7] > at org.apache.avro.file.DataFileStream.<init>(DataFileStream.java:84) > ~[avro-1.7.7.jar:1.7.7] > at > org.apache.nifi.processors.avro.ConvertAvroToJSON$1.process(ConvertAvroToJSON.java:179) > ~[nifi-avro-processors-1.0.0.jar:1.0.0] > at > org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2370) > ~[na:na] > ... 13 common frames omitted > > If i now check the file in the failure relationship it is as follows: > {"ISN": 200, "AA": "50005900", "AB": {"AC": "GASTON", "AD": "", "AE": > "BODIN"}, "AF": "S", "AG": "M", "AH": 713043, "AW": [{"AX": 19990801, "AY": > 19990831}]} > > which again is correct but why the error and why routed to failure?? this > first happened with the Split json processor and now with convert avro to > json. both work fine but sent the files to failure. > > > Moving to the other branch where i do not specify the schema but instead use > the infer schema processor, it successfully infers the schema and routes it > to success (5). Then i use (6) to convert JSON to avro and it also transfers > the flow file to success. At (7) if I use the ${inferred.avro.schema} I get > the follwoing error: > > 2016-10-28 13:34:21,577 INFO [StandardProcessScheduler Thread-2] > o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled > ConvertAvroToJSON[id=000aac50-0158-1000-d9b3-dd8343ed57a7] to run with 1 > threads > 2016-10-28 13:34:21,579 ERROR [Timer-Driven Process Thread-6] > o.a.n.processors.avro.ConvertAvroToJSON > ConvertAvroToJSON[id=000aac50-0158-1000-d9b3-dd8343ed57a7] > ConvertAvroToJSON[id=000aac50-0158-1000-d9b3-dd8343ed57a7] failed to process > due to org.apache.avro.AvroRuntimeException: Malformed data. Length is > negative: -62; rolling back session: org.apache.avro.AvroRuntimeException: > Malformed data. Length is negative: -62 > 2016-10-28 13:34:21,581 ERROR [Timer-Driven Process Thread-6] > o.a.n.processors.avro.ConvertAvroToJSON > org.apache.avro.AvroRuntimeException: Malformed data. Length is negative: > -62 > at > org.apache.avro.io.BinaryDecoder.doReadBytes(BinaryDecoder.java:336) > ~[avro-1.7.7.jar:1.7.7] > at org.apache.avro.io.BinaryDecoder.readString(BinaryDecoder.java:263) > ~[avro-1.7.7.jar:1.7.7] > at > org.apache.avro.io.ResolvingDecoder.readString(ResolvingDecoder.java:201) > ~[avro-1.7.7.jar:1.7.7] > at > org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:363) > ~[avro-1.7.7.jar:1.7.7] > at > org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:355) > ~[avro-1.7.7.jar:1.7.7] > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:157) > ~[avro-1.7.7.jar:1.7.7] > at > org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193) > ~[avro-1.7.7.jar:1.7.7] > at > org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183) > ~[avro-1.7.7.jar:1.7.7] > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151) > ~[avro-1.7.7.jar:1.7.7] > at > org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193) > ~[avro-1.7.7.jar:1.7.7] > at > org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183) > ~[avro-1.7.7.jar:1.7.7] > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151) > ~[avro-1.7.7.jar:1.7.7] > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142) > ~[avro-1.7.7.jar:1.7.7] > at > org.apache.nifi.processors.avro.ConvertAvroToJSON$1.process(ConvertAvroToJSON.java:161) > ~[nifi-avro-processors-1.0.0.jar:1.0.0] > at > org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2370) > ~[nifi-framework-core-1.0.0.jar:1.0.0] > at > org.apache.nifi.processors.avro.ConvertAvroToJSON.onTrigger(ConvertAvroToJSON.java:148) > ~[nifi-avro-processors-1.0.0.jar:1.0.0] > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > ~[nifi-api-1.0.0.jar:1.0.0] > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1064) > [nifi-framework-core-1.0.0.jar:1.0.0] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) > [nifi-framework-core-1.0.0.jar:1.0.0] > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) > [nifi-framework-core-1.0.0.jar:1.0.0] > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) > [nifi-framework-core-1.0.0.jar:1.0.0] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_101] > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > [na:1.8.0_101] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > [na:1.8.0_101] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > [na:1.8.0_101] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_101] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_101] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101] > > > In this case the file stays in the incoming connection. It is the same > situation if i enter my schema definition. > > Lasty if i remove the value for the schema field i get the same result as in > (4). The record is correctly converted to JSON but routed to failure. > > > I am very confused. Can anyone clarify what is going on?????? > > Refards > Arsalan Siddiqi > > > > > -- > View this message in context: > http://apache-nifi-developer-list.39713.n7.nabble.com/JSON-to-AVRO-AVRO-to-JSON-tp13769.html > Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
