Re: Re: Nifi 1.4: problem with QueryRecord Precessor
t; write Schema Name As Attribute because the Schema Name is not known >> > at >> > >> > org.apache.nifi.schema.access.SchemaNameAsAttribute.validateSchema(SchemaNameAsAttribute.java:53) >> > at >> > >> > org.apache.nifi.serialization.SchemaRegistryRecordSetWriter.getSchemaAccessWriter(SchemaRegistryRecordSetWriter.java:137) >> > at >> > >> > org.apache.nifi.csv.CSVRecordSetWriter.createWriter(CSVRecordSetWriter.java:72) >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> > >> > >> > >> > I am using an AvroReader using the embedded schema. Maybe this helps. >> > >> > Rgds, >> > >> > Uwe >> > >> > >> > Gesendet: Mittwoch, 04. Oktober 2017 um 16:55 Uhr >> > Von: "Mark Payne" <marka...@hotmail.com> >> > An: "users@nifi.apache.org" <users@nifi.apache.org> >> > >> > Betreff: Re: Nifi 1.4: problem with QueryRecord Precessor >> > Dan, Uwe, >> > >> > This is something that I will be looking into shortly. It is a known >> > issue >> > that exists in both 1.3 >> > and 1.4. This occurs whenever there is some other sort of failure, when >> > the >> > processor attempts >> > to route the FlowFile to 'failure'. This occurs most often when there is >> > a >> > problem obtaining the >> > schema for the Record. Can you check your logs and see what other error >> > may >> > be present >> > in the logs? >> > >> > Thanks >> > -Mark >> > >> > >> > >> > On Oct 4, 2017, at 10:39 AM, dan young <danoyo...@gmail.com> wrote: >> > >> > >> > It might be...maybe others can share experience with 1.4... >> > >> > >> > On Wed, Oct 4, 2017, 8:37 AM Uwe Geercken <uwe.geerc...@web.de> wrote: >> >> >> >> thanks Dano. >> >> >> >> Any other comments? Is this a bug in 1.4 then? >> >> >> >> Rgds, >> >> >> >> Uwe >> >> >> >> Gesendet: Mittwoch, 04. Oktober 2017 um 16:33 Uhr >> >> Von: "dan young" <danoyo...@gmail.com> >> >> An: nifi <users@nifi.apache.org> >> >> Betreff: Re: Nifi 1.4: problem with QueryRecord Precessor >> >> >> >> We're seeing the same thing. Works fine in 1.3 >> >> >> >> Dano >> >> >> >> >> >> On Wed, Oct 4, 2017, 8:13 AM Uwe Geercken <uwe.geerc...@web.de> wrote: >> >>> >> >>> Hello, >> >>> >> >>> I have created a flow: GetFile >> QueryRecord >> Putfile. GetFile >> >>> reads >> >>> an avro file. QueryRecord has one property/sql and the result is >> >>> routed to >> >>> PutFile. >> >>> >> >>> When I run the processor, I get following error: >> >>> >> >>> failed to process session due to java.lang.IllegalStateException >> >>> >> >>> already in use or an active callback or an inputstream >> >>> created by ProcessSession.read(FlowFile) has not been closed. >> >>> >> >>> >> >>> Can somebody help? >> >>> >> >>> Rgds, >> >>> >> >>> Uwe
Re: Re: Nifi 1.4: problem with QueryRecord Precessor
Arrghh..sorry folks for the noiseit was a schema issue with a conversion/cast exception. ;( Regards, Dano On Wed, Oct 4, 2017 at 9:33 AM Matt Burgess <mattyb...@apache.org> wrote: > All, > > The known issue Mark is referring to is NIFI-4349 [1], however it is > not causing the problem; rather it is hiding the problem. If an error > occurs (due to misonfiguration, schema errors, etc.), certain cleanup > activities aren't being performed so the processor ends up with the > error about processing flow files, InputStreams, etc. instead. > Regarding it working in 1.3.0 and not in 1.4.0, I think some of the > logic changes in NIFI-3921 [2] may have changed the handling of > inherited schemas and now generates an exception where it may have > been ignored before. > > Regards, > Matt > > [1] https://issues.apache.org/jira/browse/NIFI-4349 > [2] https://issues.apache.org/jira/browse/NIFI-3921 > > On Wed, Oct 4, 2017 at 11:22 AM, Uwe Geercken <uwe.geerc...@web.de> wrote: > > > > Mark, > > > > I stopped 1.4 and started 1.3 and created the same flow. And in 1.3 it > works > > without a problem. > > > > And I looked into the 1.4 log (I should have looked before, but only > looked > > at the processor's tooltip). You are right, it's a problem with the > schema. > > Here is the real error: > > > > 2017-10-04 16:01:16,640 ERROR [Timer-Driven Process Thread-10] > > o.a.nifi.processors.standard.QueryRecord > > QueryRecord[id=e798adc7-015e-1000-a43d-68a02b5d425a] Unable to qu$ > > org.apache.nifi.processor.exception.ProcessException: IOException thrown > > from QueryRecord[id=e798adc7-015e-1000-a43d-68a02b5d425a]: > > java.io.IOException: org.apache.nifi.$ > > at > > > org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2589) > > at > > > org.apache.nifi.processors.standard.QueryRecord.onTrigger(QueryRecord.java:284) > > at > > > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > > at > > > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1119) > > at > > > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147) > > at > > > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) > > at > > > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128) > > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > at > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > > at > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > > at > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > > at > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > > Caused by: java.io.IOException: > > org.apache.nifi.schema.access.SchemaNotFoundException: Cannot write > Schema > > Name As Attribute because the Schema Name is not known > > at > > > org.apache.nifi.processors.standard.QueryRecord$1.process(QueryRecord.java:302) > > at > > > org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2570) > > ... 13 common frames omitted > > Caused by: org.apache.nifi.schema.access.SchemaNotFoundException: Cannot > > write Schema Name As Attribute because the Schema Name is not known > > at > > > org.apache.nifi.schema.access.SchemaNameAsAttribute.validateSchema(SchemaNameAsAttribute.java:53) > > at > > > org.apache.nifi.serialization.SchemaRegistryRecordSetWriter.getSchemaAccessWriter(SchemaRegistryRecordSetWriter.java:137) > > at > > > org.apache.nifi.csv.CSVRecordSetWriter.createWriter(CSVRecordSetWriter.java:72) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > > > > > I am using an AvroReader using the embedded schema. Maybe this helps. > > > > Rgds, > > > > Uwe > > > > > > Gesendet: Mittwoch, 04. Oktober 2017 um 16:55 Uhr > > Von: "Mark Payne" <marka...
Re: Re: Nifi 1.4: problem with QueryRecord Precessor
Hey Matt, Thank you. I think my specific issue is a misconfig with the schema types.looking into it an will report back On Wed, Oct 4, 2017 at 9:33 AM Matt Burgess <mattyb...@apache.org> wrote: > All, > > The known issue Mark is referring to is NIFI-4349 [1], however it is > not causing the problem; rather it is hiding the problem. If an error > occurs (due to misonfiguration, schema errors, etc.), certain cleanup > activities aren't being performed so the processor ends up with the > error about processing flow files, InputStreams, etc. instead. > Regarding it working in 1.3.0 and not in 1.4.0, I think some of the > logic changes in NIFI-3921 [2] may have changed the handling of > inherited schemas and now generates an exception where it may have > been ignored before. > > Regards, > Matt > > [1] https://issues.apache.org/jira/browse/NIFI-4349 > [2] https://issues.apache.org/jira/browse/NIFI-3921 > > On Wed, Oct 4, 2017 at 11:22 AM, Uwe Geercken <uwe.geerc...@web.de> wrote: > > > > Mark, > > > > I stopped 1.4 and started 1.3 and created the same flow. And in 1.3 it > works > > without a problem. > > > > And I looked into the 1.4 log (I should have looked before, but only > looked > > at the processor's tooltip). You are right, it's a problem with the > schema. > > Here is the real error: > > > > 2017-10-04 16:01:16,640 ERROR [Timer-Driven Process Thread-10] > > o.a.nifi.processors.standard.QueryRecord > > QueryRecord[id=e798adc7-015e-1000-a43d-68a02b5d425a] Unable to qu$ > > org.apache.nifi.processor.exception.ProcessException: IOException thrown > > from QueryRecord[id=e798adc7-015e-1000-a43d-68a02b5d425a]: > > java.io.IOException: org.apache.nifi.$ > > at > > > org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2589) > > at > > > org.apache.nifi.processors.standard.QueryRecord.onTrigger(QueryRecord.java:284) > > at > > > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > > at > > > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1119) > > at > > > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147) > > at > > > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) > > at > > > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128) > > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > at > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > > at > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > > at > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > > at > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > > Caused by: java.io.IOException: > > org.apache.nifi.schema.access.SchemaNotFoundException: Cannot write > Schema > > Name As Attribute because the Schema Name is not known > > at > > > org.apache.nifi.processors.standard.QueryRecord$1.process(QueryRecord.java:302) > > at > > > org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2570) > > ... 13 common frames omitted > > Caused by: org.apache.nifi.schema.access.SchemaNotFoundException: Cannot > > write Schema Name As Attribute because the Schema Name is not known > > at > > > org.apache.nifi.schema.access.SchemaNameAsAttribute.validateSchema(SchemaNameAsAttribute.java:53) > > at > > > org.apache.nifi.serialization.SchemaRegistryRecordSetWriter.getSchemaAccessWriter(SchemaRegistryRecordSetWriter.java:137) > > at > > > org.apache.nifi.csv.CSVRecordSetWriter.createWriter(CSVRecordSetWriter.java:72) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > > > > > I am using an AvroReader using the embedded schema. Maybe this helps. > > > > Rgds, > > > > Uwe > > > > > > Gesendet: Mittwoch, 04. Oktober 2017 um 16:55 Uhr > > Von: "Mark Payne&quo
Aw: Re: Re: Nifi 1.4: problem with QueryRecord Precessor
thanks Matt. Uwe Gesendet: Mittwoch, 04. Oktober 2017 um 17:32 Uhr Von: "Matt Burgess" <mattyb...@apache.org> An: users@nifi.apache.org Betreff: Re: Re: Nifi 1.4: problem with QueryRecord Precessor All, The known issue Mark is referring to is NIFI-4349 [1], however it is not causing the problem; rather it is hiding the problem. If an error occurs (due to misonfiguration, schema errors, etc.), certain cleanup activities aren't being performed so the processor ends up with the error about processing flow files, InputStreams, etc. instead. Regarding it working in 1.3.0 and not in 1.4.0, I think some of the logic changes in NIFI-3921 [2] may have changed the handling of inherited schemas and now generates an exception where it may have been ignored before. Regards, Matt [1] https://issues.apache.org/jira/browse/NIFI-4349 [2] https://issues.apache.org/jira/browse/NIFI-3921 On Wed, Oct 4, 2017 at 11:22 AM, Uwe Geercken <uwe.geerc...@web.de> wrote: > > Mark, > > I stopped 1.4 and started 1.3 and created the same flow. And in 1.3 it works > without a problem. > > And I looked into the 1.4 log (I should have looked before, but only looked > at the processor's tooltip). You are right, it's a problem with the schema. > Here is the real error: > > 2017-10-04 16:01:16,640 ERROR [Timer-Driven Process Thread-10] > o.a.nifi.processors.standard.QueryRecord > QueryRecord[id=e798adc7-015e-1000-a43d-68a02b5d425a] Unable to qu$ > org.apache.nifi.processor.exception.ProcessException: IOException thrown > from QueryRecord[id=e798adc7-015e-1000-a43d-68a02b5d425a]: > java.io.IOException: org.apache.nifi.$ > at > org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2589) > at > org.apache.nifi.processors.standard.QueryRecord.onTrigger(QueryRecord.java:284) > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1119) > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147) > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.io.IOException: > org.apache.nifi.schema.access.SchemaNotFoundException: Cannot write Schema > Name As Attribute because the Schema Name is not known > at > org.apache.nifi.processors.standard.QueryRecord$1.process(QueryRecord.java:302) > at > org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2570) > ... 13 common frames omitted > Caused by: org.apache.nifi.schema.access.SchemaNotFoundException: Cannot > write Schema Name As Attribute because the Schema Name is not known > at > org.apache.nifi.schema.access.SchemaNameAsAttribute.validateSchema(SchemaNameAsAttribute.java:53) > at > org.apache.nifi.serialization.SchemaRegistryRecordSetWriter.getSchemaAccessWriter(SchemaRegistryRecordSetWriter.java:137) > at > org.apache.nifi.csv.CSVRecordSetWriter.createWriter(CSVRecordSetWriter.java:72) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > I am using an AvroReader using the embedded schema. Maybe this helps. > > Rgds, > > Uwe > > > Gesendet: Mittwoch, 04. Oktober 2017 um 16:55 Uhr > Von: "Mark Payne" <marka...@hotmail.com> > An: "users@nifi.apache.org" <users@nifi.apache.org> > > Betreff: Re: Nifi 1.4: problem with QueryRecord Precessor > Dan, Uwe, > > This is something that I will be looking into shortly. It is a known issue > that exists in both 1.3 > and 1.4. This occurs whenever there is some other sort of failure, when the > processor attempts > to route the FlowFile to 'failure'. This occurs most often when there is a > problem obtaining the > schema for the Record. Can you check your logs and see what other error may > be present > in the logs? > > Thanks > -Mark > > > > On Oct 4, 2017, at 10:39 AM, d
Re: Re: Nifi 1.4: problem with QueryRecord Precessor
All, The known issue Mark is referring to is NIFI-4349 [1], however it is not causing the problem; rather it is hiding the problem. If an error occurs (due to misonfiguration, schema errors, etc.), certain cleanup activities aren't being performed so the processor ends up with the error about processing flow files, InputStreams, etc. instead. Regarding it working in 1.3.0 and not in 1.4.0, I think some of the logic changes in NIFI-3921 [2] may have changed the handling of inherited schemas and now generates an exception where it may have been ignored before. Regards, Matt [1] https://issues.apache.org/jira/browse/NIFI-4349 [2] https://issues.apache.org/jira/browse/NIFI-3921 On Wed, Oct 4, 2017 at 11:22 AM, Uwe Geercken <uwe.geerc...@web.de> wrote: > > Mark, > > I stopped 1.4 and started 1.3 and created the same flow. And in 1.3 it works > without a problem. > > And I looked into the 1.4 log (I should have looked before, but only looked > at the processor's tooltip). You are right, it's a problem with the schema. > Here is the real error: > > 2017-10-04 16:01:16,640 ERROR [Timer-Driven Process Thread-10] > o.a.nifi.processors.standard.QueryRecord > QueryRecord[id=e798adc7-015e-1000-a43d-68a02b5d425a] Unable to qu$ > org.apache.nifi.processor.exception.ProcessException: IOException thrown > from QueryRecord[id=e798adc7-015e-1000-a43d-68a02b5d425a]: > java.io.IOException: org.apache.nifi.$ > at > org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2589) > at > org.apache.nifi.processors.standard.QueryRecord.onTrigger(QueryRecord.java:284) > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1119) > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147) > at > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.io.IOException: > org.apache.nifi.schema.access.SchemaNotFoundException: Cannot write Schema > Name As Attribute because the Schema Name is not known > at > org.apache.nifi.processors.standard.QueryRecord$1.process(QueryRecord.java:302) > at > org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2570) > ... 13 common frames omitted > Caused by: org.apache.nifi.schema.access.SchemaNotFoundException: Cannot > write Schema Name As Attribute because the Schema Name is not known > at > org.apache.nifi.schema.access.SchemaNameAsAttribute.validateSchema(SchemaNameAsAttribute.java:53) > at > org.apache.nifi.serialization.SchemaRegistryRecordSetWriter.getSchemaAccessWriter(SchemaRegistryRecordSetWriter.java:137) > at > org.apache.nifi.csv.CSVRecordSetWriter.createWriter(CSVRecordSetWriter.java:72) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > I am using an AvroReader using the embedded schema. Maybe this helps. > > Rgds, > > Uwe > > > Gesendet: Mittwoch, 04. Oktober 2017 um 16:55 Uhr > Von: "Mark Payne" <marka...@hotmail.com> > An: "users@nifi.apache.org" <users@nifi.apache.org> > > Betreff: Re: Nifi 1.4: problem with QueryRecord Precessor > Dan, Uwe, > > This is something that I will be looking into shortly. It is a known issue > that exists in both 1.3 > and 1.4. This occurs whenever there is some other sort of failure, when the > processor attempts > to route the FlowFile to 'failure'. This occurs most often when there is a > problem obtaining the > schema for the Record. Can you check your logs and see what other error may > be present > in the logs? > > Thanks > -Mark > > > > On Oct 4, 2017, at 10:39 AM, dan young <danoyo...@gmail.com> wrote: > > >
