So, I came back from two weeks vacation...
My nifi (1.17.0). is misbehaving.
The logs say
2023-07-03 07:04:07,668 ERROR [Index Provenance Events-1]
o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
...
Caused by: java.nio.file.FileSystemException:
/data/nifi-1.17.0/provenance_repository/lucene-8-index-1687383779103/_2y6.kdi:
Too many open files
at
java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
...
I'm out of file descriptors?
After determining that nifi is running as process id 19655, I see:
lsof -p 19655 | wc -l
85
85? That's really low.
/etc/sysctl.conf contains
Fs.file-max = 9815744
$ ulimit -a
address space limit (Kibytes) (-M) unlimited
core file size (blocks) (-c) 0
cpu time (seconds) (-t) unlimited
data size (Kibytes) (-d) unlimited
file size (blocks) (-f) unlimited
locks (-x) unlimited
locked address space (Kibytes) (-l) 64
message queue size (Kibytes) (-q) 800
nice (-e) 0
nofile (-n) 1048576
nproc (-u) 4096
pipe buffer size (bytes) (-p) 4096
max memory size (Kibytes) (-m) unlimited
rtprio (-r) 0
socket buffer size (bytes) (-b) 4096
sigpend (-i) 192389
stack size (Kibytes) (-s) 8192
swap size (Kibytes) (-w) not supported
threads (-T) not supported
process size (Kibytes) (-v) unlimited
I DO notice one more odd thing:
I have 0 flow files (though a number of flows running)
Despite that, I see that thre repositories aredefinitely taking up some space.
350G content_repository
712K database_repository
28K flowfile_repository
10G provenance_repository
Any thoughts? I was thinking about doing an rm -rf on
provenance_repository.....or maybe decreasing
nifi.provenance.repository.max.storage.time
or increasing
nifi.provenance.repository.max.storage.size (currently set to 10 G)
in nifi.properties.
Geoffrey Greene
