[
https://issues.apache.org/jira/browse/NIFI-9572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17492641#comment-17492641
]
Joe Witt commented on NIFI-9572:
--------------------------------
Hello [~mayki]
In requesting the 'java -version' output you instead appear to have sent the
output of other bash commands. Further the link implies you're on JDK 1.8
update 261. However, the beginning of the JIRA you suggest you're on update
311.
In requesting the lsof output periodically as the number of open files grow you
are instead showing that now nifi fails to start with an open file handle limit
hit during the startup process. I again suspect you do not have the proper
file limit set for the user you're actually running nifi as (which the above
info implies might be the user 'nifi'). I dont know that the ulimit results
you've shown are for the user nifi or whether they're for the root user or some
other user.
It is very difficult to be helpful when the various details don't line up
because we're not making progress. I appreciate there is a chance that there
is some bug in NIFi causing this but at this stage I dont suspect that - I
suspect environment/config setup not matching what you're intending.
We really need to see full commands/full output at this point to help progress.
I think for me to be helpful via a JIRA here I still need to see:
1. 'whoami'
2. 'java -version' including command entry and results
3. 'ulimit -a' including command entry and results
4. 'lsof -p <pidofnifi>' run every 30 minutes while nifi is running
Thanks
Joe
> Failed to index Provenance Events and (Too many Files)
> ------------------------------------------------------
>
> Key: NIFI-9572
> URL: https://issues.apache.org/jira/browse/NIFI-9572
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI
> Affects Versions: 1.15.2
> Reporter: mayki
> Priority: Major
> Attachments: nifi-app.log
>
>
> Hello
> I have upgraded NIFI 1.15.2 since 2022/01/05
> No issue until this night 2022/01/13
> * nifi version 1.15.2
> * jdk-1.8.0_311
> And the limit is high
> {code:java}
> Last login: Fri Jan 14 09:57:06 CET 2022 on pts/2
> -bash-4.2@nifi$ ulimit -a
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) rg
> pending signals (-i) 63278
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files (-n) 50000
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 10000
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
> {code}
>
> We got a lot error about provenance_repository, it fill our filesystem logs ..
>
> {code:java}
> 2022-01-14 10:19:00,963 ERROR [Index Provenance Events-2]
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
> at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877)
> at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891)
> at
> org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468)
> at
> org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444)
> at
> org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70)
> at
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202)
> at
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.nio.file.FileSystemException:
> /data/nifi/provenance_repository/lucene-8-index-1642145908399/_4_Lucene80_0.dvd:
> Too many open files
> {code}
>
>
> We expect upgrade all nifi instances to 1.15.2 to avoid log4j vulnerability.
> But it is impossible to do that if we got this error.
>
> Thanks for you help.
>
> Regards
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
