[
https://issues.apache.org/jira/browse/NIFI-2934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15606160#comment-15606160
]
Joseph Gresock commented on NIFI-2934:
--------------------------------------
Attaching the lsof output, when I encountered this error:
2016-10-25 19:00:00,000 ERROR [Timer-Driven Process Thread-46]
o.a.n.processors.standard.MergeContent
org.apache.nifi.processor.exception.FlowFileAccessException: Failed to read
content of
StandardFlowFileRecord[uuid=8ae7babc-0e22-4b5e-aa2b-46b2b4f05f1b,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1477400327827-10807,
container=default, section=567], offset=41511,
length=33613],offset=0,name=84bff10f9855a2295a8bb958b5a91262a34ec5c5.json,size=33613]
at
org.apache.nifi.controller.repository.StandardProcessSession.getInputStream(StandardProcessSession.java:1816)
~[na:na]
at
org.apache.nifi.controller.repository.StandardProcessSession.exportTo(StandardProcessSession.java:2395)
~[na:na]
at
org.apache.nifi.processors.standard.MergeContent$ZipMerge$1.process(MergeContent.java:854)
~[na:na]
at
org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:1998)
~[na:na]
at
org.apache.nifi.processors.standard.MergeContent$ZipMerge.merge(MergeContent.java:839)
~[na:na]
at
org.apache.nifi.processors.standard.MergeContent.processBin(MergeContent.java:459)
~[na:na]
at
org.apache.nifi.processor.util.bin.BinFiles.processBins(BinFiles.java:233)
[nifi-processor-utils-0.7.1.jar:0.7.1]
at
org.apache.nifi.processor.util.bin.BinFiles.onTrigger(BinFiles.java:190)
[nifi-processor-utils-0.7.1.jar:0.7.1]
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054)
[nifi-framework-core-0.7.1.jar:0.7.1]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
[nifi-framework-core-0.7.1.jar:0.7.1]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
[nifi-framework-core-0.7.1.jar:0.7.1]
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
[nifi-framework-core-0.7.1.jar:0.7.1]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_101]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[na:1.8.0_101]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_101]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_101]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_101]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
Caused by: java.io.FileNotFoundException:
/data/nifi/content_repository/567/1477400327827-10807 (Too many open files)
at java.io.FileInputStream.open0(Native Method) ~[na:1.8.0_101]
at java.io.FileInputStream.open(FileInputStream.java:195)
~[na:1.8.0_101]
at java.io.FileInputStream.<init>(FileInputStream.java:138)
~[na:1.8.0_101]
at
org.apache.nifi.controller.repository.FileSystemRepository.read(FileSystemRepository.java:801)
~[na:na]
at
org.apache.nifi.controller.repository.StandardProcessSession.getInputStream(StandardProcessSession.java:1788)
~[na:na]
... 18 common frames omitted
>From the command line as the nifi user, I can indeed open that file.
> Archiver still not respecting
> nifi.content.repository.archive.max.usage.percentage
> ----------------------------------------------------------------------------------
>
> Key: NIFI-2934
> URL: https://issues.apache.org/jira/browse/NIFI-2934
> Project: Apache NiFi
> Issue Type: Bug
> Affects Versions: 0.7.0, 0.7.1
> Reporter: Joseph Gresock
> Attachments: Disk-Usage-Increasing.png, NiFi-80-percent-disk.png,
> Queued.png, System-Diagnostics.png, content_repository usage.png
>
>
> This seems related to NIFI-1726: we've noticed that the content repository
> takes up increasingly more space over time, even beyond the configured max
> usage percentage (see images). After restarting the NiFi cluster we get an
> immediate drop in disk usage with lots of log statements indicating that
> expired content is being removed.
> Not sure if this is related, but we also often get "Too many open files"
> during this expiration process after NiFi restart, despite lsof indicating a
> count far lower than our configured nofile and fs-max.
> In the environment indicated by the pictures,
> nifi.content.repository.archive.max.usage.percentage = 50%. Note that the
> flow itself only has ~240GB queued across the entire cluster, but each
> content_repository directory has over 360GB on each worker. Also note the
> disk usage graph increasing above 50% on each worker, until we finally
> restart and then the usage drops below 50%.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
