[
https://issues.apache.org/jira/browse/NIFI-7893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Khodabakhsh updated NIFI-7893:
-------------------------------------
Description:
On Nifi 1.11.4 and 1.12.1, I intermittently am unable to view the provenance in
the UI and through the `nipyapi` python library.
Checking the logs, I get the following message:
```
{color:#ff0000}2020-10-06 22:12:39,789 INFO [Timer-Driven Process Thread-10]
o.a.n.p.store.WriteAheadStorePartition Successfully rolled over Event Writer
for Provenance Event Store Partition[directory=./provenance_repository] due to
MAX_TIME_REACHED
2020-10-06 22:12:39,789 ERROR [Compress Provenance Logs-1-thread-2]
o.a.n.p.s.EventFileCompressor Failed to compress ./provenance_repository/0.prov
on rollover
java.io.FileNotFoundException: ./provenance_repository/0.prov (No such file or
directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at
org.apache.nifi.provenance.serialization.EventFileCompressor.compress(EventFileCompressor.java:164)
at
org.apache.nifi.provenance.serialization.EventFileCompressor.run(EventFileCompressor.java:115)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748){color}
```
Setting `nifi.provenance.repository.compress.on.rollover=false` fixes this.
Here's a copy of all my `nifi.provenance.repository` settings from
`nifi.properties`:
```
{color:#6a9955}# Provenance Repository Properties{color}
{color:#569cd6}nifi.provenance.repository.implementation{color}{color:#d4d4d4}=org.apache.nifi.provenance.WriteAheadProvenanceRepository{color}
{color:#569cd6}nifi.provenance.repository.debug.frequency{color}{color:#d4d4d4}=1_000_000{color}
{color:#569cd6}nifi.provenance.repository.encryption.key.provider.implementation{color}{color:#d4d4d4}={color}
{color:#569cd6}nifi.provenance.repository.encryption.key.provider.location{color}{color:#d4d4d4}={color}
{color:#569cd6}nifi.provenance.repository.encryption.key.id{color}{color:#d4d4d4}={color}
{color:#569cd6}nifi.provenance.repository.encryption.key{color}{color:#d4d4d4}={color}
{color:#6a9955}# Persistent Provenance Repository Properties{color}
{color:#569cd6}nifi.provenance.repository.directory.default{color}{color:#d4d4d4}=./provenance_repository{color}
{color:#569cd6}nifi.provenance.repository.max.storage.time{color}{color:#d4d4d4}=24
hours{color}
{color:#569cd6}nifi.provenance.repository.max.storage.size{color}{color:#d4d4d4}=1
GB{color}
{color:#569cd6}nifi.provenance.repository.rollover.time{color}{color:#d4d4d4}=30
secs{color}
{color:#569cd6}nifi.provenance.repository.rollover.size{color}{color:#d4d4d4}=100
MB{color}
{color:#569cd6}nifi.provenance.repository.query.threads{color}{color:#d4d4d4}=2{color}
{color:#569cd6}nifi.provenance.repository.index.threads{color}{color:#d4d4d4}=2{color}
{color:#569cd6}nifi.provenance.repository.compress.on.rollover{color}{color:#d4d4d4}=false{color}
{color:#569cd6}nifi.provenance.repository.always.sync{color}{color:#d4d4d4}=false{color}
{color:#6a9955}# Comma-separated list of fields. Fields that are not indexed
will not be searchable. Valid fields are:{color}
{color:#6a9955}# EventType, FlowFileUUID, Filename, TransitURI, ProcessorID,
AlternateIdentifierURI, Relationship, Details{color}
{color:#569cd6}nifi.provenance.repository.indexed.fields{color}{color:#d4d4d4}=EventType,
FlowFileUUID, Filename, ProcessorID, Relationship{color}
{color:#6a9955}# FlowFile Attributes that should be indexed and made
searchable. Some examples to consider are filename, uuid, mime.type{color}
{color:#569cd6}nifi.provenance.repository.indexed.attributes{color}{color:#d4d4d4}={color}
{color:#6a9955}# Large values for the shard size will result in more Java heap
usage when searching the Provenance Repository{color}
{color:#6a9955}# but should provide better performance{color}
{color:#569cd6}nifi.provenance.repository.index.shard.size{color}{color:#d4d4d4}=500
MB{color}
{color:#6a9955}# Indicates the maximum length that a FlowFile attribute can be
when retrieving a Provenance Event from{color}
{color:#6a9955}# the repository. If the length of any attribute exceeds this
value, it will be truncated when the event is retrieved.{color}
{color:#569cd6}nifi.provenance.repository.max.attribute.length{color}{color:#d4d4d4}=65536{color}
{color:#569cd6}nifi.provenance.repository.concurrent.merge.threads{color}{color:#d4d4d4}=2{color}
{color:#6a9955}# Volatile Provenance Respository Properties{color}
{color:#569cd6}nifi.provenance.repository.buffer.size{color}{color:#d4d4d4}=100000{color}
{color:#d4d4d4}```{color}
was:
On Nifi 1.11.4 and 1.12.1, I intermittently am unable to view the provenance in
the UI and through the `nipyapi` python library.
Checking the logs, I get the following message:
{color:#FF0000}2020-10-06 22:12:39,789 INFO [Timer-Driven Process Thread-10]
o.a.n.p.store.WriteAheadStorePartition Successfully rolled over Event Writer
for Provenance Event Store Partition[directory=./provenance_repository] due to
MAX_TIME_REACHED
2020-10-06 22:12:39,789 ERROR [Compress Provenance Logs-1-thread-2]
o.a.n.p.s.EventFileCompressor Failed to compress ./provenance_repository/0.prov
on rollover
java.io.FileNotFoundException: ./provenance_repository/0.prov (No such file or
directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at
org.apache.nifi.provenance.serialization.EventFileCompressor.compress(EventFileCompressor.java:164)
at
org.apache.nifi.provenance.serialization.EventFileCompressor.run(EventFileCompressor.java:115)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748){color}
Setting `nifi.provenance.repository.compress.on.rollover=false` fixes this.
Here's a copy of all my `nifi.provenance.repository` settings from
`nifi.properties`:
{color:#6a9955}# Provenance Repository Properties{color}
{color:#569cd6}nifi.provenance.repository.implementation{color}{color:#d4d4d4}=org.apache.nifi.provenance.WriteAheadProvenanceRepository{color}
{color:#569cd6}nifi.provenance.repository.debug.frequency{color}{color:#d4d4d4}=1_000_000{color}
{color:#569cd6}nifi.provenance.repository.encryption.key.provider.implementation{color}{color:#d4d4d4}={color}
{color:#569cd6}nifi.provenance.repository.encryption.key.provider.location{color}{color:#d4d4d4}={color}
{color:#569cd6}nifi.provenance.repository.encryption.key.id{color}{color:#d4d4d4}={color}
{color:#569cd6}nifi.provenance.repository.encryption.key{color}{color:#d4d4d4}={color}
{color:#6a9955}# Persistent Provenance Repository Properties{color}
{color:#569cd6}nifi.provenance.repository.directory.default{color}{color:#d4d4d4}=./provenance_repository{color}
{color:#569cd6}nifi.provenance.repository.max.storage.time{color}{color:#d4d4d4}=24
hours{color}
{color:#569cd6}nifi.provenance.repository.max.storage.size{color}{color:#d4d4d4}=1
GB{color}
{color:#569cd6}nifi.provenance.repository.rollover.time{color}{color:#d4d4d4}=30
secs{color}
{color:#569cd6}nifi.provenance.repository.rollover.size{color}{color:#d4d4d4}=100
MB{color}
{color:#569cd6}nifi.provenance.repository.query.threads{color}{color:#d4d4d4}=2{color}
{color:#569cd6}nifi.provenance.repository.index.threads{color}{color:#d4d4d4}=2{color}
{color:#569cd6}nifi.provenance.repository.compress.on.rollover{color}{color:#d4d4d4}=false{color}
{color:#569cd6}nifi.provenance.repository.always.sync{color}{color:#d4d4d4}=false{color}
{color:#6a9955}# Comma-separated list of fields. Fields that are not indexed
will not be searchable. Valid fields are:{color}
{color:#6a9955}# EventType, FlowFileUUID, Filename, TransitURI, ProcessorID,
AlternateIdentifierURI, Relationship, Details{color}
{color:#569cd6}nifi.provenance.repository.indexed.fields{color}{color:#d4d4d4}=EventType,
FlowFileUUID, Filename, ProcessorID, Relationship{color}
{color:#6a9955}# FlowFile Attributes that should be indexed and made
searchable. Some examples to consider are filename, uuid, mime.type{color}
{color:#569cd6}nifi.provenance.repository.indexed.attributes{color}{color:#d4d4d4}={color}
{color:#6a9955}# Large values for the shard size will result in more Java heap
usage when searching the Provenance Repository{color}
{color:#6a9955}# but should provide better performance{color}
{color:#569cd6}nifi.provenance.repository.index.shard.size{color}{color:#d4d4d4}=500
MB{color}
{color:#6a9955}# Indicates the maximum length that a FlowFile attribute can be
when retrieving a Provenance Event from{color}
{color:#6a9955}# the repository. If the length of any attribute exceeds this
value, it will be truncated when the event is retrieved.{color}
{color:#569cd6}nifi.provenance.repository.max.attribute.length{color}{color:#d4d4d4}=65536{color}
{color:#569cd6}nifi.provenance.repository.concurrent.merge.threads{color}{color:#d4d4d4}=2{color}
{color:#6a9955}# Volatile Provenance Respository Properties{color}
{color:#569cd6}nifi.provenance.repository.buffer.size{color}{color:#d4d4d4}=100000{color}
> Provenance is blank is `nifi.provenance.repository.compress.on.rollover` is
> set to `true`
> -----------------------------------------------------------------------------------------
>
> Key: NIFI-7893
> URL: https://issues.apache.org/jira/browse/NIFI-7893
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 1.11.4, 1.12.1
> Environment: Docker 19.03.13
> Image: apache/nifi:1.12.1
> Python: nipyapi: 0.14.1
> Reporter: Daniel Khodabakhsh
> Priority: Major
>
> On Nifi 1.11.4 and 1.12.1, I intermittently am unable to view the provenance
> in the UI and through the `nipyapi` python library.
> Checking the logs, I get the following message:
> ```
> {color:#ff0000}2020-10-06 22:12:39,789 INFO [Timer-Driven Process Thread-10]
> o.a.n.p.store.WriteAheadStorePartition Successfully rolled over Event Writer
> for Provenance Event Store Partition[directory=./provenance_repository] due
> to MAX_TIME_REACHED
> 2020-10-06 22:12:39,789 ERROR [Compress Provenance Logs-1-thread-2]
> o.a.n.p.s.EventFileCompressor Failed to compress
> ./provenance_repository/0.prov on rollover
> java.io.FileNotFoundException: ./provenance_repository/0.prov (No such file
> or directory)
> at java.io.FileInputStream.open0(Native Method)
> at java.io.FileInputStream.open(FileInputStream.java:195)
> at java.io.FileInputStream.<init>(FileInputStream.java:138)
> at
> org.apache.nifi.provenance.serialization.EventFileCompressor.compress(EventFileCompressor.java:164)
> at
> org.apache.nifi.provenance.serialization.EventFileCompressor.run(EventFileCompressor.java:115)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748){color}
> ```
> Setting `nifi.provenance.repository.compress.on.rollover=false` fixes this.
> Here's a copy of all my `nifi.provenance.repository` settings from
> `nifi.properties`:
> ```
> {color:#6a9955}# Provenance Repository Properties{color}
>
> {color:#569cd6}nifi.provenance.repository.implementation{color}{color:#d4d4d4}=org.apache.nifi.provenance.WriteAheadProvenanceRepository{color}
>
> {color:#569cd6}nifi.provenance.repository.debug.frequency{color}{color:#d4d4d4}=1_000_000{color}
>
> {color:#569cd6}nifi.provenance.repository.encryption.key.provider.implementation{color}{color:#d4d4d4}={color}
>
> {color:#569cd6}nifi.provenance.repository.encryption.key.provider.location{color}{color:#d4d4d4}={color}
>
> {color:#569cd6}nifi.provenance.repository.encryption.key.id{color}{color:#d4d4d4}={color}
>
> {color:#569cd6}nifi.provenance.repository.encryption.key{color}{color:#d4d4d4}={color}
> {color:#6a9955}# Persistent Provenance Repository Properties{color}
>
> {color:#569cd6}nifi.provenance.repository.directory.default{color}{color:#d4d4d4}=./provenance_repository{color}
>
> {color:#569cd6}nifi.provenance.repository.max.storage.time{color}{color:#d4d4d4}=24
> hours{color}
>
> {color:#569cd6}nifi.provenance.repository.max.storage.size{color}{color:#d4d4d4}=1
> GB{color}
>
> {color:#569cd6}nifi.provenance.repository.rollover.time{color}{color:#d4d4d4}=30
> secs{color}
>
> {color:#569cd6}nifi.provenance.repository.rollover.size{color}{color:#d4d4d4}=100
> MB{color}
>
> {color:#569cd6}nifi.provenance.repository.query.threads{color}{color:#d4d4d4}=2{color}
>
> {color:#569cd6}nifi.provenance.repository.index.threads{color}{color:#d4d4d4}=2{color}
>
> {color:#569cd6}nifi.provenance.repository.compress.on.rollover{color}{color:#d4d4d4}=false{color}
>
> {color:#569cd6}nifi.provenance.repository.always.sync{color}{color:#d4d4d4}=false{color}
> {color:#6a9955}# Comma-separated list of fields. Fields that are not indexed
> will not be searchable. Valid fields are:{color}
> {color:#6a9955}# EventType, FlowFileUUID, Filename, TransitURI, ProcessorID,
> AlternateIdentifierURI, Relationship, Details{color}
>
> {color:#569cd6}nifi.provenance.repository.indexed.fields{color}{color:#d4d4d4}=EventType,
> FlowFileUUID, Filename, ProcessorID, Relationship{color}
> {color:#6a9955}# FlowFile Attributes that should be indexed and made
> searchable. Some examples to consider are filename, uuid, mime.type{color}
>
> {color:#569cd6}nifi.provenance.repository.indexed.attributes{color}{color:#d4d4d4}={color}
> {color:#6a9955}# Large values for the shard size will result in more Java
> heap usage when searching the Provenance Repository{color}
> {color:#6a9955}# but should provide better performance{color}
>
> {color:#569cd6}nifi.provenance.repository.index.shard.size{color}{color:#d4d4d4}=500
> MB{color}
> {color:#6a9955}# Indicates the maximum length that a FlowFile attribute can
> be when retrieving a Provenance Event from{color}
> {color:#6a9955}# the repository. If the length of any attribute exceeds this
> value, it will be truncated when the event is retrieved.{color}
>
> {color:#569cd6}nifi.provenance.repository.max.attribute.length{color}{color:#d4d4d4}=65536{color}
>
> {color:#569cd6}nifi.provenance.repository.concurrent.merge.threads{color}{color:#d4d4d4}=2{color}
> {color:#6a9955}# Volatile Provenance Respository Properties{color}
>
> {color:#569cd6}nifi.provenance.repository.buffer.size{color}{color:#d4d4d4}=100000{color}
> {color:#d4d4d4}```{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
