Hi everyone,
I am currently doing a small PoC, if we can use Apache NiFi as
integration component for our application.
Right now, I just read files from local disk and write them to a Kafka
topic (0.10.0.1):
GetFile -> PublishKafka_0_10 (terminates success and failure)
After a couple of files and a few million messages the PublishKafka_0_10
processors run into:
2017-06-16 12:05:07,972 ERROR [Timer-Driven Process Thread-10]
o.a.n.p.kafka.pubsub.PublishKafka_0_10
PublishKafka_0_10[id=b0578b3d-015c-100
0-e685-07f301c90901]
PublishKafka_0_10[id=b0578b3d-015c-1000-e685-07f301c90901] failed to
process due to org.apache.nifi.processor.exception
.FlowFileHandlingException:
StandardFlowFileRecord[uuid=c150df70-8f89-4f6b-9a2d-5be4df726df0,claim=StandardContentClaim
[resourceClaim=Stand
ardResourceClaim[id=1497607276559-29, container=default, section=29],
offset=0, length=0],offset=0,name=file5.csv,size=0] transfer relations
hip not specified; rolling back session: {}
You can find more logs attached. The processor never recovers. The Kafka
Cluster is fine.
Any pointer to what is going on or what I might be doing wrong is highly
appreciated.
Cheers and thank you,
Konstantin
--
Konstantin Knauf * [email protected] * +49-174-3413182
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
Sitz: Unterföhring * Amtsgericht München * HRB 135082
2017-06-16 12:05:06,971 ERROR [Timer-Driven Process Thread-8]
o.a.n.p.kafka.pubsub.PublishKafka_0_10
PublishKafka_0_10[id=b0578b3d-015c-1000-e685-07f301c90901]
PublishKafka_0_10[id=b0578b3d-015c-1000-e685-07f301c90901] failed to process
due to org.apache.nifi.processor.exception.FlowFileHandlingException:
StandardFlowFileRecord[uuid=c150df70-8f89-4f6b-9a2d-5be4df726df0,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1497607276559-29, container=default,
section=29], offset=0, length=0],offset=0,name=file5.csv,size=0] transfer
relationship not specified; rolling back session: {}
org.apache.nifi.processor.exception.FlowFileHandlingException:
StandardFlowFileRecord[uuid=c150df70-8f89-4f6b-9a2d-5be4df726df0,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1497607276559-29, container=default,
section=29], offset=0, length=0],offset=0,name=file5.csv,size=0] transfer
relationship not specified
at
org.apache.nifi.controller.repository.StandardProcessSession.checkpoint(StandardProcessSession.java:248)
at
org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:318)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2017-06-16 12:05:06,971 ERROR [Timer-Driven Process Thread-8]
o.a.n.p.kafka.pubsub.PublishKafka_0_10
PublishKafka_0_10[id=b0578b3d-015c-1000-e685-07f301c90901]
PublishKafka_0_10[id=b0578b3d-015c-1000-e685-07f301c90901] failed to process
session due to org.apache.nifi.processor.exception.FlowFileHandlingException:
StandardFlowFileRecord[uuid=c150df70-8f89-4f6b-9a2d-5be4df726df0,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1497607276559-29, container=default,
section=29], offset=0, length=0],offset=0,name=file5.csv,size=0] transfer
relationship not specified: {}
org.apache.nifi.processor.exception.FlowFileHandlingException:
StandardFlowFileRecord[uuid=c150df70-8f89-4f6b-9a2d-5be4df726df0,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1497607276559-29, container=default,
section=29], offset=0, length=0],offset=0,name=file5.csv,size=0] transfer
relationship not specified
at
org.apache.nifi.controller.repository.StandardProcessSession.checkpoint(StandardProcessSession.java:248)
at
org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:318)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2017-06-16 12:05:06,971 WARN [Timer-Driven Process Thread-8]
o.a.n.p.kafka.pubsub.PublishKafka_0_10
PublishKafka_0_10[id=b0578b3d-015c-1000-e685-07f301c90901] Processor
Administratively Yielded for 1 sec due to processing failure
2017-06-16 12:05:06,971 WARN [Timer-Driven Process Thread-8]
o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding
PublishKafka_0_10[id=b0578b3d-015c-1000-e685-07f301c90901] due to uncaught
Exception: org.apache.nifi.processor.exception.FlowFileHandlingException:
StandardFlowFileRecord[uuid=c150df70-8f89-4f6b-9a2d-5be4df726df0,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1497607276559-29, container=default,
section=29], offset=0, length=0],offset=0,name=file5.csv,size=0] transfer
relationship not specified
2017-06-16 12:05:06,971 WARN [Timer-Driven Process Thread-8]
o.a.n.c.t.ContinuallyRunProcessorTask
org.apache.nifi.processor.exception.FlowFileHandlingException:
StandardFlowFileRecord[uuid=c150df70-8f89-4f6b-9a2d-5be4df726df0,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1497607276559-29, container=default,
section=29], offset=0, length=0],offset=0,name=file5.csv,size=0] transfer
relationship not specified
at
org.apache.nifi.controller.repository.StandardProcessSession.checkpoint(StandardProcessSession.java:248)
at
org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:318)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2017-06-16 12:05:07,972 ERROR [Timer-Driven Process Thread-10]
o.a.n.p.kafka.pubsub.PublishKafka_0_10
PublishKafka_0_10[id=b0578b3d-015c-1000-e685-07f301c90901]
PublishKafka_0_10[id=b0578b3d-015c-1000-e685-07f301c90901] failed to process
due to org.apache.nifi.processor.exception.FlowFileHandlingException:
StandardFlowFileRecord[uuid=c150df70-8f89-4f6b-9a2d-5be4df726df0,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1497607276559-29, container=default,
section=29], offset=0, length=0],offset=0,name=file5.csv,size=0] transfer
relationship not specified; rolling back session: {}
org.apache.nifi.processor.exception.FlowFileHandlingException:
StandardFlowFileRecord[uuid=c150df70-8f89-4f6b-9a2d-5be4df726df0,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1497607276559-29, container=default,
section=29], offset=0, length=0],offset=0,name=file5.csv,size=0] transfer
relationship not specified
at
org.apache.nifi.controller.repository.StandardProcessSession.checkpoint(StandardProcessSession.java:248)
at
org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:318)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2017-06-16 12:05:07,972 ERROR [Timer-Driven Process Thread-10]
o.a.n.p.kafka.pubsub.PublishKafka_0_10
PublishKafka_0_10[id=b0578b3d-015c-1000-e685-07f301c90901]
PublishKafka_0_10[id=b0578b3d-015c-1000-e685-07f301c90901] failed to process
session due to org.apache.nifi.processor.exception.FlowFileHandlingException:
StandardFlowFileRecord[uuid=c150df70-8f89-4f6b-9a2d-5be4df726df0,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1497607276559-29, container=default,
section=29], offset=0, length=0],offset=0,name=file5.csv,size=0] transfer
relationship not specified: {}
org.apache.nifi.processor.exception.FlowFileHandlingException:
StandardFlowFileRecord[uuid=c150df70-8f89-4f6b-9a2d-5be4df726df0,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1497607276559-29, container=default,
section=29], offset=0, length=0],offset=0,name=file5.csv,size=0] transfer
relationship not specified
at
org.apache.nifi.controller.repository.StandardProcessSession.checkpoint(StandardProcessSession.java:248)
at
org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:318)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2017-06-16 12:05:07,972 WARN [Timer-Driven Process Thread-10]
o.a.n.p.kafka.pubsub.PublishKafka_0_10
PublishKafka_0_10[id=b0578b3d-015c-1000-e685-07f301c90901] Processor
Administratively Yielded for 1 sec due to processing failure
2017-06-16 12:05:07,972 WARN [Timer-Driven Process Thread-10]
o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding
PublishKafka_0_10[id=b0578b3d-015c-1000-e685-07f301c90901] due to uncaught
Exception: org.apache.nifi.processor.exception.FlowFileHandlingException:
StandardFlowFileRecord[uuid=c150df70-8f89-4f6b-9a2d-5be4df726df0,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1497607276559-29, container=default,
section=29], offset=0, length=0],offset=0,name=file5.csv,size=0] transfer
relationship not specified
2017-06-16 12:05:07,972 WARN [Timer-Driven Process Thread-10]
o.a.n.c.t.ContinuallyRunProcessorTask
org.apache.nifi.processor.exception.FlowFileHandlingException:
StandardFlowFileRecord[uuid=c150df70-8f89-4f6b-9a2d-5be4df726df0,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1497607276559-29, container=default,
section=29], offset=0, length=0],offset=0,name=file5.csv,size=0] transfer
relationship not specified
at
org.apache.nifi.controller.repository.StandardProcessSession.checkpoint(StandardProcessSession.java:248)
at
org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:318)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
