Wiktor Kubicki created NIFI-9464:
------------------------------------
Summary: Provenance Events files corrupted
Key: NIFI-9464
URL: https://issues.apache.org/jira/browse/NIFI-9464
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 1.15.0, 1.11.0
Environment: java 11, centos 7, nifi standalone
Reporter: Wiktor Kubicki
In my logs i found:
{code:java}
SiteToSiteProvenanceReportingTask[id=b209c0ae-016e-1000-ae39-301c9dcfc544]
Failed to retrieve Provenance Events from repository due to: Attempted to skip
to byte offset 9149491 for 1125432890.prov.gz but file does not have that many
bytes (TOC
Reader=StandardTocReader[file=/..../provenance_repository/toc/1125432890.toc,
compressed=false]): java.io.EOFException: Attempted to skip to byte offset
9149491 for 1125432890.prov.gz but file does not have that many bytes (TOC
Reader=StandardTocReader[file=/.../provenance_repository/toc/1125432890.toc,
compressed=false])
{code}
It happened about 100 times in last 1 year. It is criticaly important for me to
have 100% sure of my logs.
My config for provenance (BTW if you see posibbility for tune it, tell me):
{code:java}
nifi.provenance.repository.directory.default=/....../provenance_repository
nifi.provenance.repository.max.storage.time=730 days
nifi.provenance.repository.max.storage.size=512 GB
nifi.provenance.repository.rollover.time=10 mins
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.query.threads=2
nifi.provenance.repository.index.threads=1
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false
nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename,
ProcessorID
nifi.provenance.repository.indexed.attributes=
nifi.provenance.repository.index.shard.size=1 GB
nifi.provenance.repository.max.attribute.length=65536
nifi.provenance.repository.concurrent.merge.threads=1
nifi.provenance.repository.buffer.size=100000
{code}
Now my provenance repo has 140GB of data.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
