I think with systemd services can have their own handler limits. Check in the NiFi service config for that. IIRC, we had to set configuration flags in the NiFi config specifically to force systemd to give that service unlimited handles.
On Mon, Jul 3, 2023 at 10:31 AM Greene (US), Geoffrey N < [email protected]> wrote: > > > So, I came back from two weeks vacation… > > > > My nifi (1.17.0). is misbehaving. > > > > The logs say > > 2023-07-03 07:04:07,668 ERROR [Index Provenance Events-1] > o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events > > org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed > > … > > Caused by: java.nio.file.FileSystemException: > /data/nifi-1.17.0/provenance_repository/lucene-8-index-1687383779103/_2y6.kdi: > Too many open files > > at > java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100) > > … > > > > I’m out of file descriptors? > > > > After determining that nifi is running as process id 19655, I see: > > > > lsof -p 19655 | wc -l > > 85 > > > > 85? That’s really low. > > > > /etc/sysctl.conf contains > > Fs.file-max = 9815744 > > > > > > $ ulimit -a > > address space limit (Kibytes) (-M) unlimited > > core file size (blocks) (-c) 0 > > cpu time (seconds) (-t) unlimited > > data size (Kibytes) (-d) unlimited > > file size (blocks) (-f) unlimited > > locks (-x) unlimited > > locked address space (Kibytes) (-l) 64 > > message queue size (Kibytes) (-q) 800 > > nice (-e) 0 > > nofile (-n) 1048576 > > nproc (-u) 4096 > > pipe buffer size (bytes) (-p) 4096 > > max memory size (Kibytes) (-m) unlimited > > rtprio (-r) 0 > > socket buffer size (bytes) (-b) 4096 > > sigpend (-i) 192389 > > stack size (Kibytes) (-s) 8192 > > swap size (Kibytes) (-w) not supported > > threads (-T) not supported > > process size (Kibytes) (-v) unlimited > > > > I DO notice one more odd thing: > > > > I have 0 flow files (though a number of flows running) > > > > Despite that, I see that thre repositories aredefinitely taking up some > space. > > > > 350G content_repository > > 712K database_repository > > 28K flowfile_repository > > 10G provenance_repository > > > > > > > > Any thoughts? I was thinking about doing an rm –rf on > provenance_repository…..or maybe decreasing > > nifi.provenance.repository.max.storage.time > > or increasing > > nifi.provenance.repository.max.storage.size (currently set to 10 G) > > in nifi.properties. > > > > > > > > Geoffrey Greene > > >
