[
https://issues.apache.org/jira/browse/NIFI-9464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-9464:
-------------------------------
Fix Version/s: 1.25.0
2.0.0
Status: Patch Available (was: Open)
> Provenance Events files corrupted
> ---------------------------------
>
> Key: NIFI-9464
> URL: https://issues.apache.org/jira/browse/NIFI-9464
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 1.15.0, 1.11.0
> Environment: java 11, centos 7, nifi standalone
> Reporter: Wiktor Kubicki
> Assignee: Tamas Palfy
> Priority: Minor
> Fix For: 1.25.0, 2.0.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In my logs i found:
> {code:java}
> SiteToSiteProvenanceReportingTask[id=b209c0ae-016e-1000-ae39-301c9dcfc544]
> Failed to retrieve Provenance Events from repository due to: Attempted to
> skip to byte offset 9149491 for 1125432890.prov.gz but file does not have
> that many bytes (TOC
> Reader=StandardTocReader[file=/..../provenance_repository/toc/1125432890.toc,
> compressed=false]): java.io.EOFException: Attempted to skip to byte offset
> 9149491 for 1125432890.prov.gz but file does not have that many bytes (TOC
> Reader=StandardTocReader[file=/.../provenance_repository/toc/1125432890.toc,
> compressed=false])
> {code}
> It is criticaly important for me to have 100% sure of my logs. It happened
> about 100 times in last 1 year for 15 *.prov.gz files:
> {code:java}
> -rw-rw-rw-. 1 user user 1013923 Oct 17 21:17 1075441276.prov.gz
> -rw-rw-rw-. 1 user user 1345431 Oct 24 13:06 1083362251.prov.gz
> -rw-rw-rw-. 1 user user 1359282 Oct 25 13:07 1084546392.prov.gz
> -rw-rw-rw-. 1 user user 1155791 Nov 2 17:08 1094516954.prov.gz
> -rw-rw-r--. 1 user user 974136 Nov 18 22:07 1113402183.prov.gz
> -rw-rw-r--. 1 user user 1125608 Nov 28 22:00 1125097576.prov.gz
> -rw-rw-r--. 1 user user 1248319 Nov 29 04:30 1125432890.prov.gz
> -rw-rw-r--. 1 user user 832120 Feb 2 2021 661957813.prov.gz
> -rw-rw-r--. 1 user user 1110978 Mar 17 2021 734807613.prov.gz
> -rw-rw-r--. 1 user user 1506819 Apr 16 2021 786154249.prov.gz
> -rw-rw-r--. 1 user user 1763198 May 25 2021 852626782.prov.gz
> -rw-rw-r--. 1 user user 1580598 Jun 15 08:32 891934274.prov.gz
> -rw-rw-r--. 1 user user 2960296 Jun 28 17:07 917991812.prov.gz
> -rw-rw-r--. 1 user user 1808037 Jun 28 17:37 918051650.prov.gz
> -rw-rw-rw-. 1 user user 765924 Aug 14 13:09 991505484.prov.gz
> {code}
> BTW it's interesting why thera ere different chmods
> My config for provenance (BTW if you see posibbility for tune it, please tell
> me):
> {code:java}
> nifi.provenance.repository.directory.default=/....../provenance_repository
> nifi.provenance.repository.max.storage.time=730 days
> nifi.provenance.repository.max.storage.size=512 GB
> nifi.provenance.repository.rollover.time=10 mins
> nifi.provenance.repository.rollover.size=100 MB
> nifi.provenance.repository.query.threads=2
> nifi.provenance.repository.index.threads=1
> nifi.provenance.repository.compress.on.rollover=true
> nifi.provenance.repository.always.sync=false
> nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename,
> ProcessorID
> nifi.provenance.repository.indexed.attributes=
> nifi.provenance.repository.index.shard.size=1 GB
> nifi.provenance.repository.max.attribute.length=65536
> nifi.provenance.repository.concurrent.merge.threads=1
> nifi.provenance.repository.buffer.size=100000
> {code}
> Now my provenance repo has 140GB of data.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
