Hi Purushotham， Since you are using cluster mode， just delete the flow.xml.gz file and restart the node, the flow file will be synced from the other two nodes.
Regards, Lei wangl...@geekplus.com.cn From: Purushotham Pushpavanthar Date: 2019-12-19 17:05 To: dev Subject: Unable to access flowfile content Hi, We've have 3 node production cluster running seamlessly for almost 8 month with manageable ups and downs. However, yesterday we ran into an issue in one of the processors due to which CPU shot up and node went down. On restart, the contents of few enqueued flowfiles went missing all of sudden (I was unable to view content from the content viewer in UI). This also resulted in below exception, when was blocking downstream processor from processing any flowfile. We are using version 1.9.2. It would be very helpful if you can help me debug this issue. 2019-12-19 07:05:03,653 ERROR [Timer-Driven Process Thread-4] o.apache.nifi.processors.hive.PutHiveQL PutHiveQL[id=c820350d-d6fd-183d-a3d5-006a2b14d10a] PutHiveQL[id=c820350d-d6fd-183d-a3d5-006a2b14d10a] failed to process session due to java.lang.RuntimeException: Failed to execute due to org.apache.nifi.processor.exception.FlowFileAccessException: Could not read from StandardFlowFileRecord[uuid=253e1652-6e3f-49c3-b190-3788fcbc1480,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1576648697457-40, container=default, section=40], offset=10977, length=83],offset=0,name=hid_1004.ejuserstruct2.2019121100.sql,size=83]; Processor Administratively Yielded for 1 sec: java.lang.RuntimeException: Failed to execute due to org.apache.nifi.processor.exception.FlowFileAccessException: Could not read from StandardFlowFileRecord[uuid=253e1652-6e3f-49c3-b190-3788fcbc1480,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1576648697457-40, container=default, section=40], offset=10977, length=83],offset=0,name=hid_1004.ejuserstruct2.2019121100.sql,size=83] java.lang.RuntimeException: Failed to execute due to org.apache.nifi.processor.exception.FlowFileAccessException: Could not read from StandardFlowFileRecord[uuid=253e1652-6e3f-49c3-b190-3788fcbc1480,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1576648697457-40, container=default, section=40], offset=10977, length=83],offset=0,name=hid_1004.ejuserstruct2.2019121100.sql,size=83] at org.apache.nifi.processor.util.pattern.Put.onTrigger(Put.java:145) at org.apache.nifi.processors.hive.PutHiveQL.lambda$onTrigger$6(PutHiveQL.java:295) at org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114) at org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184) at org.apache.nifi.processors.hive.PutHiveQL.onTrigger(PutHiveQL.java:295) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.nifi.processor.exception.FlowFileAccessException: Could not read from StandardFlowFileRecord[uuid=253e1652-6e3f-49c3-b190-3788fcbc1480,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1576648697457-40, container=default, section=40], offset=10977, length=83],offset=0,name=hid_1004.ejuserstruct2.2019121100.sql,size=83] at org.apache.nifi.controller.repository.io.FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:93) at org.apache.nifi.controller.repository.io.TaskTerminationInputStream.read(TaskTerminationInputStream.java:68) at org.apache.nifi.stream.io.StreamUtils.fillBuffer(StreamUtils.java:89) at org.apache.nifi.stream.io.StreamUtils.fillBuffer(StreamUtils.java:72) at org.apache.nifi.processors.hive.AbstractHiveQLProcessor$1.process(AbstractHiveQLProcessor.java:92) at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2212) at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2180) at org.apache.nifi.processors.hive.AbstractHiveQLProcessor.getHiveQL(AbstractHiveQLProcessor.java:89) at org.apache.nifi.processors.hive.PutHiveQL.lambda$new$4(PutHiveQL.java:220) at org.apache.nifi.processor.util.pattern.Put.putFlowFiles(Put.java:59) at org.apache.nifi.processor.util.pattern.Put.onTrigger(Put.java:102) ... 15 common frames omitted Caused by: java.io.EOFException: null at org.apache.nifi.stream.io.StreamUtils.skip(StreamUtils.java:270) at org.apache.nifi.controller.repository.FileSystemRepository.read(FileSystemRepository.java:859) at org.apache.nifi.controller.repository.io.ContentClaimInputStream.formDelegate(ContentClaimInputStream.java:154) at org.apache.nifi.controller.repository.io.ContentClaimInputStream.getDelegate(ContentClaimInputStream.java:51) at org.apache.nifi.controller.repository.io.ContentClaimInputStream.read(ContentClaimInputStream.java:89) at org.apache.nifi.controller.repository.io.DisableOnCloseInputStream.read(DisableOnCloseInputStream.java:49) at org.apache.nifi.controller.repository.io.LimitedInputStream.read(LimitedInputStream.java:86) at org.apache.nifi.controller.repository.io.DisableOnCloseInputStream.read(DisableOnCloseInputStream.java:49) at org.apache.nifi.stream.io.ByteCountingInputStream.read(ByteCountingInputStream.java:51) at java.io.FilterInputStream.read(FilterInputStream.java:133) at org.apache.nifi.controller.repository.io.FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:82) ... 25 common frames omitted 2019-12-19 07:05:03,654 WARN [Timer-Driven Process Thread-4] o.a.n.controller.tasks.ConnectableTask Administratively Yielding PutHiveQL[id=c820350d-d6fd-183d-a3d5-006a2b14d10a] due to uncaught Exception: java.lang.RuntimeException: Failed to execute due to org.apache.nifi.processor.exception.FlowFileAccessException: Could not read from StandardFlowFileRecord[uuid=253e1652-6e3f-49c3-b190-3788fcbc1480,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1576648697457-40, container=default, section=40], offset=10977, length=83],offset=0,name=hid_1004.ejuserstruct2.2019121100.sql,size=83] java.lang.RuntimeException: Failed to execute due to org.apache.nifi.processor.exception.FlowFileAccessException: Could not read from StandardFlowFileRecord[uuid=253e1652-6e3f-49c3-b190-3788fcbc1480,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1576648697457-40, container=default, section=40], offset=10977, length=83],offset=0,name=hid_1004.ejuserstruct2.2019121100.sql,size=83] at org.apache.nifi.processor.util.pattern.Put.onTrigger(Put.java:145) at org.apache.nifi.processors.hive.PutHiveQL.lambda$onTrigger$6(PutHiveQL.java:295) at org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114) at org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184) at org.apache.nifi.processors.hive.PutHiveQL.onTrigger(PutHiveQL.java:295) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.nifi.processor.exception.FlowFileAccessException: Could not read from StandardFlowFileRecord[uuid=253e1652-6e3f-49c3-b190-3788fcbc1480,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1576648697457-40, container=default, section=40], offset=10977, length=83],offset=0,name=hid_1004.ejuserstruct2.2019121100.sql,size=83] at org.apache.nifi.controller.repository.io.FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:93) at org.apache.nifi.controller.repository.io.TaskTerminationInputStream.read(TaskTerminationInputStream.java:68) at org.apache.nifi.stream.io.StreamUtils.fillBuffer(StreamUtils.java:89) at org.apache.nifi.stream.io.StreamUtils.fillBuffer(StreamUtils.java:72) at org.apache.nifi.processors.hive.AbstractHiveQLProcessor$1.process(AbstractHiveQLProcessor.java:92) at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2212) at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2180) at org.apache.nifi.processors.hive.AbstractHiveQLProcessor.getHiveQL(AbstractHiveQLProcessor.java:89) at org.apache.nifi.processors.hive.PutHiveQL.lambda$new$4(PutHiveQL.java:220) at org.apache.nifi.processor.util.pattern.Put.putFlowFiles(Put.java:59) at org.apache.nifi.processor.util.pattern.Put.onTrigger(Put.java:102) ... 15 common frames omitted Caused by: java.io.EOFException: null at org.apache.nifi.stream.io.StreamUtils.skip(StreamUtils.java:270) at org.apache.nifi.controller.repository.FileSystemRepository.read(FileSystemRepository.java:859) at org.apache.nifi.controller.repository.io.ContentClaimInputStream.formDelegate(ContentClaimInputStream.java:154) at org.apache.nifi.controller.repository.io.ContentClaimInputStream.getDelegate(ContentClaimInputStream.java:51) at org.apache.nifi.controller.repository.io.ContentClaimInputStream.read(ContentClaimInputStream.java:89) at org.apache.nifi.controller.repository.io.DisableOnCloseInputStream.read(DisableOnCloseInputStream.java:49) at org.apache.nifi.controller.repository.io.LimitedInputStream.read(LimitedInputStream.java:86) at org.apache.nifi.controller.repository.io.DisableOnCloseInputStream.read(DisableOnCloseInputStream.java:49) at org.apache.nifi.stream.io.ByteCountingInputStream.read(ByteCountingInputStream.java:51) at java.io.FilterInputStream.read(FilterInputStream.java:133) at org.apache.nifi.controller.repository.io.FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:82) ... 25 common frames omitted Regards, Purushotham Pushpavanth
