[
https://issues.apache.org/jira/browse/NIFI-2524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421207#comment-15421207
]
ASF GitHub Bot commented on NIFI-2524:
--------------------------------------
Github user olegz commented on a diff in the pull request:
https://github.com/apache/nifi/pull/840#discussion_r74787227
--- Diff:
nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java
---
@@ -1302,27 +1301,35 @@ public void run() {
}
logger.info("Successfully Rolled over
Provenance Event file containing {} records", recordsWritten);
+ }
+
+ //if files were rolled over or if out of retries
stop the future
+ if(fileRolledOver != null ||
retryAttempts.decrementAndGet() == 0) {
+
+ if(fileRolledOver== null &&
retryAttempts.get() == 0){
+ logger.error("Failed to merge Journal
Files {} after {} attempts. ",journalsToMerge, MAX_JOURNAL_ROLLOVER_RETRIES);
+ }
+
rolloverCompletions.getAndIncrement();
- // We have finished successfully. Cancel the
future so that we don't run anymore
+ // Cancel the future so that we don't run
anymore
Future<?> future;
while ((future = futureReference.get()) ==
null) {
--- End diff --
Also, catching Throwable is very dangerous. We are essentially including
Errors in the mix such as OOM etc, and by doing so proclaiming that we can
actually do something about it when in reality we simply can't. Those are
generally unrecoverable
> Unable to Merge Journal Files after OutOfMemory Error
> -----------------------------------------------------
>
> Key: NIFI-2524
> URL: https://issues.apache.org/jira/browse/NIFI-2524
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Reporter: Yolanda M. Davis
> Assignee: Yolanda M. Davis
> Fix For: 1.0.0
>
> Attachments: nifi-app-1.log, nifi-app-2.log
>
>
> While running a flow that attempted to SplitText by eol an OutOfMemory Error
> occurred (which wasn't surprising since this was a standalone instance with
> no tuning for memory, backpressure, etc).
> {noformat}
> ERROR [Timer-Driven Process Thread-9] o.a.nifi.processors.standard.SplitText
> SplitText[id=6bb0fe36-0156-1000-795b-55cf4237a389]
> SplitText[id=6bb0fe36-0156-1000-795b-55cf4237a389] failed to process due to
> java.lang.OutOfMemoryError: Java heap space; rolling back session:
> java.lang.OutOfMemoryError: Java heap space
> 2016-08-08 20:12:36,068 ERROR
> [LeaseRenewer:[email protected]:8020]
> org.apache.nifi.NiFi An Unknown Error Occurred in Thread
> Thread[LeaseRenewer:[email protected]:8020,5,main]:
> java.lang.OutOfMemoryError: Java heap space
> 2016-08-08 20:12:36,072 ERROR [Timer-Driven Process Thread-8]
> o.a.n.p.PersistentProvenanceRepository Failed to persist Provenance Event due
> to java.io.IOException: Stream Closed.
> 2016-08-08 20:12:55,111 ERROR
> [LeaseRenewer:[email protected]:8020]
> org.apache.nifi.NiFi
> java.lang.OutOfMemoryError: Java heap space
> at java.util.ArrayList.iterator(ArrayList.java:834) ~[na:1.8.0_101]
> at
> org.apache.hadoop.hdfs.LeaseRenewer.clientsRunning(LeaseRenewer.java:241)
> ~[na:na]
> at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:491)
> ~[na:na]
> at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
> ~[na:na]
> at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304)
> ~[na:na]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_101]
> 2016-08-08 20:12:58,022 ERROR [Timer-Driven Process Thread-9]
> o.a.nifi.processors.standard.SplitText
> java.lang.OutOfMemoryError: Java heap space
> at java.util.HashMap.newNode(HashMap.java:1742) ~[na:1.8.0_101]
> at java.util.HashMap.putVal(HashMap.java:630) ~[na:1.8.0_101]
> at java.util.HashMap.putMapEntries(HashMap.java:514) ~[na:1.8.0_101]
> at java.util.HashMap.putAll(HashMap.java:784) ~[na:1.8.0_101]
> at
> org.apache.nifi.controller.repository.StandardFlowFileRecord$Builder.fromFlowFile(StandardFlowFileRecord.java:307)
> ~[nifi-framework-core-1.0.0-BETA.jar:1.0.0-BETA]
> at
> org.apache.nifi.controller.repository.StandardProcessSession.putAttribute(StandardProcessSession.java:1462)
> ~[nifi-framework-core-1.0.0-BETA.jar:1.0.0-BETA]
> at
> org.apache.nifi.processors.standard.SplitText$1.process(SplitText.java:499)
> ~[na:na]
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1880)
> ~[nifi-framework-core-1.0.0-BETA.jar:1.0.0-BETA]
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1851)
> ~[nifi-framework-core-1.0.0-BETA.jar:1.0.0-BETA]
> at
> org.apache.nifi.processors.standard.SplitText.onTrigger(SplitText.java:420)
> ~[na:na]
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> ~[nifi-api-1.0.0-BETA.jar:1.0.0-BETA]
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1060)
> [nifi-framework-core-1.0.0-BETA.jar:1.0.0-BETA]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
> [nifi-framework-core-1.0.0-BETA.jar:1.0.0-BETA]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> [nifi-framework-core-1.0.0-BETA.jar:1.0.0-BETA]
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
> [nifi-framework-core-1.0.0-BETA.jar:1.0.0-BETA]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_101]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [na:1.8.0_101]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_101]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [na:1.8.0_101]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_101]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_101]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
> 2016-08-08 20:13:04,246 ERROR [Timer-Driven Process Thread-8]
> o.a.n.p.PersistentProvenanceRepository
> java.io.IOException: Stream Closed
> at java.io.FileOutputStream.writeBytes(Native Method) ~[na:1.8.0_101]
> at java.io.FileOutputStream.write(FileOutputStream.java:326)
> ~[na:1.8.0_101]
> at
> org.apache.nifi.stream.io.ByteCountingOutputStream.write(ByteCountingOutputStream.java:49)
> ~[nifi-utils-1.0.0-BETA.jar:1.0.0-BETA]
> at
> org.apache.nifi.stream.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:69)
> ~[nifi-utils-1.0.0-BETA.jar:1.0.0-BETA]
> at
> org.apache.nifi.stream.io.BufferedOutputStream.flush(BufferedOutputStream.java:126)
> ~[nifi-utils-1.0.0-BETA.jar:1.0.0-BETA]
> at
> org.apache.nifi.stream.io.ByteCountingOutputStream.flush(ByteCountingOutputStream.java:59)
> ~[nifi-utils-1.0.0-BETA.jar:1.0.0-BETA]
> at
> org.apache.nifi.stream.io.DataOutputStream.flush(DataOutputStream.java:104)
> ~[nifi-utils-1.0.0-BETA.jar:1.0.0-BETA]
> at
> org.apache.nifi.provenance.StandardRecordWriter.writeRecord(StandardRecordWriter.java:247)
> ~[nifi-persistent-provenance-repository-1.0.0-BETA.jar:1.0.0-BETA]
> at
> org.apache.nifi.provenance.PersistentProvenanceRepository.persistRecord(PersistentProvenanceRepository.java:744)
> [nifi-persistent-provenance-repository-1.0.0-BETA.jar:1.0.0-BETA]
> at
> org.apache.nifi.provenance.PersistentProvenanceRepository.registerEvent(PersistentProvenanceRepository.java:405)
> [nifi-persistent-provenance-repository-1.0.0-BETA.jar:1.0.0-BETA]
> at
> org.apache.nifi.controller.repository.StandardProvenanceReporter.send(StandardProvenanceReporter.java:203)
> [nifi-framework-core-1.0.0-BETA.jar:1.0.0-BETA]
> at
> org.apache.nifi.controller.repository.StandardProvenanceReporter.send(StandardProvenanceReporter.java:173)
> [nifi-framework-core-1.0.0-BETA.jar:1.0.0-BETA]
> at
> org.apache.nifi.processors.hadoop.PutHDFS.onTrigger(PutHDFS.java:345)
> [nifi-hdfs-processors-1.0.0-BETA.jar:1.0.0-BETA]
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> [nifi-api-1.0.0-BETA.jar:1.0.0-BETA]
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1060)
> [nifi-framework-core-1.0.0-BETA.jar:1.0.0-BETA]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
> [nifi-framework-core-1.0.0-BETA.jar:1.0.0-BETA]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> [nifi-framework-core-1.0.0-BETA.jar:1.0.0-BETA]
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
> [nifi-framework-core-1.0.0-BETA.jar:1.0.0-BETA]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_101]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [na:1.8.0_101]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_101]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [na:1.8.0_101]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_101]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_101]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
> 2016-08-08 20:13:34,028 ERROR [Timer-Driven Process Thread-9]
> o.a.nifi.processors.standard.SplitText
> SplitText[id=6bb0fe36-0156-1000-795b-55cf4237a389]
> SplitText[id=6bb0fe36-0156-1000-795b-55cf4237a389] failed to process session
> due to java.lang.OutOfMemoryError: Java heap space:
> java.lang.OutOfMemoryError: Java heap space
> 2016-08-08 20:13:34,028 ERROR [Timer-Driven Process Thread-9]
> o.a.nifi.processors.standard.SplitText
> java.lang.OutOfMemoryError: Java heap space
> {noformat}
> However after stopping the offending processor yet allowing processors down
> the line to continue processing (including putting into hdfs) NiFi reported
> ProvenanceRepository warnings rooted in a FileNotFoundException:
> {noformat}
> 2016-08-08 20:28:31,358 WARN [Provenance Repository Rollover Thread-2]
> o.a.n.p.PersistentProvenanceRepository Unable to merge
> ./provenance_repository/journals/91137.journal.0 with other Journal Files due
> to java.io.FileNotFoundException: Unable to locate file
> ./provenance_repository/journals/91137.journal.0
> 2016-08-08 20:28:31,358 WARN [Provenance Repository Rollover Thread-2]
> o.a.n.p.PersistentProvenanceRepository Unable to merge
> ./provenance_repository/journals/91137.journal.1 with other Journal Files due
> to java.io.FileNotFoundException: Unable to locate file
> ./provenance_repository/journals/91137.journal.1
> 2016-08-08 20:28:31,358 WARN [Provenance Repository Rollover Thread-2]
> o.a.n.p.PersistentProvenanceRepository Unable to merge
> ./provenance_repository/journals/91137.journal.2 with other Journal Files due
> to java.io.FileNotFoundException: Unable to locate file
> ./provenance_repository/journals/91137.journal.2
> 2016-08-08 20:28:31,358 WARN [Provenance Repository Rollover Thread-2]
> o.a.n.p.PersistentProvenanceRepository Unable to merge
> ./provenance_repository/journals/91137.journal.3 with other Journal Files due
> to java.io.FileNotFoundException: Unable to locate file
> ./provenance_repository/journals/91137.journal.3
> 2016-08-08 20:28:31,358 WARN [Provenance Repository Rollover Thread-2]
> o.a.n.p.PersistentProvenanceRepository Unable to merge
> ./provenance_repository/journals/91137.journal.4 with other Journal Files due
> to java.io.FileNotFoundException: Unable to locate file
> ./provenance_repository/journals/91137.journal.4
> {noformat}
> These warnings persisted until NiFi was restarted completely. Upon restart
> the system started successfully yet appear to purge unknown files from the
> ContentRepository (via archiving)
> {noformat}
> 2016-08-08 20:37:48,146 INFO [main] o.a.n.c.repository.FileSystemRepository
> Found unknown file
> /usr/nifi/nifi-1.0.0-BETA/content_repository/210/1470687051782-210 (1049769
> bytes) in File System Repository; archiving file
> 2016-08-08 20:37:48,148 INFO [main] o.a.n.c.repository.FileSystemRepository
> Found unknown file
> /usr/nifi/nifi-1.0.0-BETA/content_repository/99/1470687045979-99 (1049310
> bytes) in File System Repository; archiving file
> 2016-08-08 20:37:48,148 INFO [main] o.a.n.c.repository.FileSystemRepository
> Found unknown file
> /usr/nifi/nifi-1.0.0-BETA/content_repository/481/1470687065579-481 (1048861
> bytes) in File System Repository; archiving file
> 2016-08-08 20:37:48,149 INFO [main] o.a.n.c.repository.FileSystemRepository
> Found unknown file
> /usr/nifi/nifi-1.0.0-BETA/content_repository/281/1470687054990-281 (1049854
> bytes) in File System Repository; archiving file
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
