[
https://issues.apache.org/jira/browse/NIFI-7893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne resolved NIFI-7893.
------------------------------
Resolution: Duplicate
> Provenance is intermittently blank when
> `nifi.provenance.repository.compress.on.rollover` is set to `true`
> ----------------------------------------------------------------------------------------------------------
>
> Key: NIFI-7893
> URL: https://issues.apache.org/jira/browse/NIFI-7893
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 1.11.4, 1.12.1
> Environment: Docker 19.03.13
> Image: apache/nifi:1.12.1
> Python: nipyapi: 0.14.1
> Reporter: Daniel Khodabakhsh
> Priority: Major
>
> On Nifi 1.11.4 and 1.12.1, I intermittently am unable to view the provenance
> in the UI and through the `nipyapi` python library.
> Checking the logs, I get the following message:
> {code:java}
> 2020-10-06 22:12:39,789 INFO [Timer-Driven Process Thread-10]
> o.a.n.p.store.WriteAheadStorePartition Successfully rolled over Event Writer
> for Provenance Event Store Partition[directory=./provenance_repository] due
> to MAX_TIME_REACHED
> 2020-10-06 22:12:39,789 ERROR [Compress Provenance Logs-1-thread-2]
> o.a.n.p.s.EventFileCompressor Failed to compress
> ./provenance_repository/0.prov on rollover
> java.io.FileNotFoundException: ./provenance_repository/0.prov (No such file
> or directory)
> at java.io.FileInputStream.open0(Native Method)
> at java.io.FileInputStream.open(FileInputStream.java:195)
> at java.io.FileInputStream.<init>(FileInputStream.java:138)
> at
> org.apache.nifi.provenance.serialization.EventFileCompressor.compress(EventFileCompressor.java:164)
> at
> org.apache.nifi.provenance.serialization.EventFileCompressor.run(EventFileCompressor.java:115)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748){code}
> Setting `nifi.provenance.repository.compress.on.rollover=false` fixes this.
> I'd also like to note that my `/opt/nifi/nifi-current/provenance_repository`
> directory had a `0.prov.gz` which would be around 10-20 kB and never grow. It
> also contained a few other files but it never contained a `0.prov` file.
> Here's a copy of all my `nifi.provenance.repository` settings from
> `nifi.properties`:
> {code:java}
> # Provenance Repository Properties
>
> nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository
> nifi.provenance.repository.debug.frequency=1_000_000
> nifi.provenance.repository.encryption.key.provider.implementation=
> nifi.provenance.repository.encryption.key.provider.location=
> nifi.provenance.repository.encryption.key.id=
> nifi.provenance.repository.encryption.key=
> # Persistent Provenance Repository Properties
> nifi.provenance.repository.directory.default=./provenance_repository
> nifi.provenance.repository.max.storage.time=24 hours
> nifi.provenance.repository.max.storage.size=1 GB
> nifi.provenance.repository.rollover.time=30 secs
> nifi.provenance.repository.rollover.size=100 MB
> nifi.provenance.repository.query.threads=2
> nifi.provenance.repository.index.threads=2
> nifi.provenance.repository.compress.on.rollover=false
> nifi.provenance.repository.always.sync=false
> # Comma-separated list of fields. Fields that are not indexed will not be
> searchable. Valid fields are:
> # EventType, FlowFileUUID, Filename, TransitURI, ProcessorID,
> AlternateIdentifierURI, Relationship, Details
> nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename,
> ProcessorID, Relationship
> # FlowFile Attributes that should be indexed and made searchable. Some
> examples to consider are filename, uuid, mime.type
> nifi.provenance.repository.indexed.attributes=
> # Large values for the shard size will result in more Java heap usage when
> searching the Provenance Repository
> # but should provide better performance
> nifi.provenance.repository.index.shard.size=500 MB
> # Indicates the maximum length that a FlowFile attribute can be when
> retrieving a Provenance Event from
> # the repository. If the length of any attribute exceeds this value, it will
> be truncated when the event is retrieved.
> nifi.provenance.repository.max.attribute.length=65536
> nifi.provenance.repository.concurrent.merge.threads=2
> # Volatile Provenance Respository Properties
> nifi.provenance.repository.buffer.size=100000
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
