[jira] [Commented] (NIFI-2890) Provenance Repository corruption

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869863#comment-15869863
 ] 

ASF GitHub Bot commented on NIFI-2890:
--

Github user jskora closed the pull request at:

https://github.com/apache/nifi/pull/1484


> Provenance Repository corruption
> 
>
> Key: NIFI-2890
> URL: https://issues.apache.org/jira/browse/NIFI-2890
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Brandon DeVries
>Assignee: Joe Skora
> Fix For: 0.8.0, 1.2.0
>
>
> It looks as thought there was a ticket to address this\[1\], but it appears 
> to not have completely addressed the issue. On an 0.6.1 instance (that 
> crashed hard), I'm getting a stack trace on startup indicating a corrupt prov 
> repo. The relevant part of the stack trace (unfortunately typed by hand, so 
> forgive the inevitable typos) is:
> {quote}
> ...
> Caused By java.lang.IllegalArgumentException: No enum constant 
> org.apahe.nifi.provenance.ProvenenceEventType.
> at java.lang/.Enum.valueOf...
> at 
> org.apache.nifi.provenance.ProvenanaceEventType.valueOf(ProvenanaceEventType.java:19)
> at
> org.apache.nifi.provenance.StandardRecordReader.nextRecord(StandardRecordReader.java:284)
> at 
> org.apache.nifi.provenance.PersistentProvenanceRepository.mergeJournals(PersistentProvenanceRepository.java:1678)
>  \[2\]
> ...
> {quote}
> In 0.x HEAD, this appears to have moved a bit\[3\]. Same is true for 
> 1.x\[4\]. In both 0.x\[5\] and 1.x\[6\], code was added to check for "bad" 
> records coming from the reader, and skip them. However, this appears to only 
> address the issue if the corrupt / bad record is the *first* record coming 
> out of a reader. If it is a subsequent record, the corruption remains.
> \[1\] https://issues.apache.org/jira/browse/NIFI-803
> \[2\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[3\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1711
> \[4\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[5\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1564-L1578
> \[6\] 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1658-L1672



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-2890) Provenance Repository corruption

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869862#comment-15869862
 ] 

ASF GitHub Bot commented on NIFI-2890:
--

Github user jskora commented on the issue:

https://github.com/apache/nifi/pull/1484
  
Committed on 
[7d23bd7](https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=7d23bd7ac6af228a3e723d18253063231c919cf9).


> Provenance Repository corruption
> 
>
> Key: NIFI-2890
> URL: https://issues.apache.org/jira/browse/NIFI-2890
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Brandon DeVries
>Assignee: Joe Skora
> Fix For: 0.8.0, 1.2.0
>
>
> It looks as thought there was a ticket to address this\[1\], but it appears 
> to not have completely addressed the issue. On an 0.6.1 instance (that 
> crashed hard), I'm getting a stack trace on startup indicating a corrupt prov 
> repo. The relevant part of the stack trace (unfortunately typed by hand, so 
> forgive the inevitable typos) is:
> {quote}
> ...
> Caused By java.lang.IllegalArgumentException: No enum constant 
> org.apahe.nifi.provenance.ProvenenceEventType.
> at java.lang/.Enum.valueOf...
> at 
> org.apache.nifi.provenance.ProvenanaceEventType.valueOf(ProvenanaceEventType.java:19)
> at
> org.apache.nifi.provenance.StandardRecordReader.nextRecord(StandardRecordReader.java:284)
> at 
> org.apache.nifi.provenance.PersistentProvenanceRepository.mergeJournals(PersistentProvenanceRepository.java:1678)
>  \[2\]
> ...
> {quote}
> In 0.x HEAD, this appears to have moved a bit\[3\]. Same is true for 
> 1.x\[4\]. In both 0.x\[5\] and 1.x\[6\], code was added to check for "bad" 
> records coming from the reader, and skip them. However, this appears to only 
> address the issue if the corrupt / bad record is the *first* record coming 
> out of a reader. If it is a subsequent record, the corruption remains.
> \[1\] https://issues.apache.org/jira/browse/NIFI-803
> \[2\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[3\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1711
> \[4\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[5\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1564-L1578
> \[6\] 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1658-L1672



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-2890) Provenance Repository corruption

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869526#comment-15869526
 ] 

ASF GitHub Bot commented on NIFI-2890:
--

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/1484
  
Hey @jskora, could you manually close this PR?


> Provenance Repository corruption
> 
>
> Key: NIFI-2890
> URL: https://issues.apache.org/jira/browse/NIFI-2890
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Brandon DeVries
>Assignee: Joe Skora
> Fix For: 0.8.0, 1.2.0
>
>
> It looks as thought there was a ticket to address this\[1\], but it appears 
> to not have completely addressed the issue. On an 0.6.1 instance (that 
> crashed hard), I'm getting a stack trace on startup indicating a corrupt prov 
> repo. The relevant part of the stack trace (unfortunately typed by hand, so 
> forgive the inevitable typos) is:
> {quote}
> ...
> Caused By java.lang.IllegalArgumentException: No enum constant 
> org.apahe.nifi.provenance.ProvenenceEventType.
> at java.lang/.Enum.valueOf...
> at 
> org.apache.nifi.provenance.ProvenanaceEventType.valueOf(ProvenanaceEventType.java:19)
> at
> org.apache.nifi.provenance.StandardRecordReader.nextRecord(StandardRecordReader.java:284)
> at 
> org.apache.nifi.provenance.PersistentProvenanceRepository.mergeJournals(PersistentProvenanceRepository.java:1678)
>  \[2\]
> ...
> {quote}
> In 0.x HEAD, this appears to have moved a bit\[3\]. Same is true for 
> 1.x\[4\]. In both 0.x\[5\] and 1.x\[6\], code was added to check for "bad" 
> records coming from the reader, and skip them. However, this appears to only 
> address the issue if the corrupt / bad record is the *first* record coming 
> out of a reader. If it is a subsequent record, the corruption remains.
> \[1\] https://issues.apache.org/jira/browse/NIFI-803
> \[2\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[3\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1711
> \[4\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[5\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1564-L1578
> \[6\] 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1658-L1672



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-2890) Provenance Repository corruption

[Michael Moser (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868564#comment-15868564
 ] 

Michael Moser commented on NIFI-2890:
-

+1 to PR #1484 as well, I merged it to 0.x.

> Provenance Repository corruption
> 
>
> Key: NIFI-2890
> URL: https://issues.apache.org/jira/browse/NIFI-2890
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Brandon DeVries
>Assignee: Joe Skora
> Fix For: 0.8.0, 1.2.0
>
>
> It looks as thought there was a ticket to address this\[1\], but it appears 
> to not have completely addressed the issue. On an 0.6.1 instance (that 
> crashed hard), I'm getting a stack trace on startup indicating a corrupt prov 
> repo. The relevant part of the stack trace (unfortunately typed by hand, so 
> forgive the inevitable typos) is:
> {quote}
> ...
> Caused By java.lang.IllegalArgumentException: No enum constant 
> org.apahe.nifi.provenance.ProvenenceEventType.
> at java.lang/.Enum.valueOf...
> at 
> org.apache.nifi.provenance.ProvenanaceEventType.valueOf(ProvenanaceEventType.java:19)
> at
> org.apache.nifi.provenance.StandardRecordReader.nextRecord(StandardRecordReader.java:284)
> at 
> org.apache.nifi.provenance.PersistentProvenanceRepository.mergeJournals(PersistentProvenanceRepository.java:1678)
>  \[2\]
> ...
> {quote}
> In 0.x HEAD, this appears to have moved a bit\[3\]. Same is true for 
> 1.x\[4\]. In both 0.x\[5\] and 1.x\[6\], code was added to check for "bad" 
> records coming from the reader, and skip them. However, this appears to only 
> address the issue if the corrupt / bad record is the *first* record coming 
> out of a reader. If it is a subsequent record, the corruption remains.
> \[1\] https://issues.apache.org/jira/browse/NIFI-803
> \[2\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[3\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1711
> \[4\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[5\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1564-L1578
> \[6\] 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1658-L1672



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-2890) Provenance Repository corruption

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868563#comment-15868563
 ] 

ASF subversion and git services commented on NIFI-2890:
---

Commit 7d23bd7ac6af228a3e723d18253063231c919cf9 in nifi's branch refs/heads/0.x 
from [~jskora]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=7d23bd7 ]

NIFI-2890 Provenance Repository Corruption (0.x)
* Corrected handling of corrupt journal file records that prevents instance 
startup and loss of records from corrupt files.  Specifically, exception 
handling was expanded to cover failures on records after the first the same as 
failures on the first record.
* Adjusted log messages  to reflect that the remainder or all of the journal 
will be skipped, not just the current record.


> Provenance Repository corruption
> 
>
> Key: NIFI-2890
> URL: https://issues.apache.org/jira/browse/NIFI-2890
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Brandon DeVries
>Assignee: Joe Skora
> Fix For: 0.8.0, 1.2.0
>
>
> It looks as thought there was a ticket to address this\[1\], but it appears 
> to not have completely addressed the issue. On an 0.6.1 instance (that 
> crashed hard), I'm getting a stack trace on startup indicating a corrupt prov 
> repo. The relevant part of the stack trace (unfortunately typed by hand, so 
> forgive the inevitable typos) is:
> {quote}
> ...
> Caused By java.lang.IllegalArgumentException: No enum constant 
> org.apahe.nifi.provenance.ProvenenceEventType.
> at java.lang/.Enum.valueOf...
> at 
> org.apache.nifi.provenance.ProvenanaceEventType.valueOf(ProvenanaceEventType.java:19)
> at
> org.apache.nifi.provenance.StandardRecordReader.nextRecord(StandardRecordReader.java:284)
> at 
> org.apache.nifi.provenance.PersistentProvenanceRepository.mergeJournals(PersistentProvenanceRepository.java:1678)
>  \[2\]
> ...
> {quote}
> In 0.x HEAD, this appears to have moved a bit\[3\]. Same is true for 
> 1.x\[4\]. In both 0.x\[5\] and 1.x\[6\], code was added to check for "bad" 
> records coming from the reader, and skip them. However, this appears to only 
> address the issue if the corrupt / bad record is the *first* record coming 
> out of a reader. If it is a subsequent record, the corruption remains.
> \[1\] https://issues.apache.org/jira/browse/NIFI-803
> \[2\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[3\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1711
> \[4\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[5\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1564-L1578
> \[6\] 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1658-L1672



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-2890) Provenance Repository corruption

[Joe Skora (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866319#comment-15866319
 ] 

Joe Skora commented on NIFI-2890:
-

[~markap14], I think they are all set, except I don't a commit for 
[#1484|https://github.com/apache/nifi/pull/1484] into the [ASF repo 0.x 
branch|https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=shortlog;h=refs/heads/0.x].
  Were there any errors on the push back to ASF?

> Provenance Repository corruption
> 
>
> Key: NIFI-2890
> URL: https://issues.apache.org/jira/browse/NIFI-2890
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Brandon DeVries
>Assignee: Joe Skora
> Fix For: 0.8.0, 1.2.0
>
>
> It looks as thought there was a ticket to address this\[1\], but it appears 
> to not have completely addressed the issue. On an 0.6.1 instance (that 
> crashed hard), I'm getting a stack trace on startup indicating a corrupt prov 
> repo. The relevant part of the stack trace (unfortunately typed by hand, so 
> forgive the inevitable typos) is:
> {quote}
> ...
> Caused By java.lang.IllegalArgumentException: No enum constant 
> org.apahe.nifi.provenance.ProvenenceEventType.
> at java.lang/.Enum.valueOf...
> at 
> org.apache.nifi.provenance.ProvenanaceEventType.valueOf(ProvenanaceEventType.java:19)
> at
> org.apache.nifi.provenance.StandardRecordReader.nextRecord(StandardRecordReader.java:284)
> at 
> org.apache.nifi.provenance.PersistentProvenanceRepository.mergeJournals(PersistentProvenanceRepository.java:1678)
>  \[2\]
> ...
> {quote}
> In 0.x HEAD, this appears to have moved a bit\[3\]. Same is true for 
> 1.x\[4\]. In both 0.x\[5\] and 1.x\[6\], code was added to check for "bad" 
> records coming from the reader, and skip them. However, this appears to only 
> address the issue if the corrupt / bad record is the *first* record coming 
> out of a reader. If it is a subsequent record, the corruption remains.
> \[1\] https://issues.apache.org/jira/browse/NIFI-803
> \[2\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[3\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1711
> \[4\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[5\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1564-L1578
> \[6\] 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1658-L1672



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-2890) Provenance Repository corruption

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866291#comment-15866291
 ] 

ASF GitHub Bot commented on NIFI-2890:
--

Github user jskora commented on the issue:

https://github.com/apache/nifi/pull/1203
  
Replaced by #1485 


> Provenance Repository corruption
> 
>
> Key: NIFI-2890
> URL: https://issues.apache.org/jira/browse/NIFI-2890
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Brandon DeVries
>Assignee: Joe Skora
> Fix For: 0.8.0, 1.2.0
>
>
> It looks as thought there was a ticket to address this\[1\], but it appears 
> to not have completely addressed the issue. On an 0.6.1 instance (that 
> crashed hard), I'm getting a stack trace on startup indicating a corrupt prov 
> repo. The relevant part of the stack trace (unfortunately typed by hand, so 
> forgive the inevitable typos) is:
> {quote}
> ...
> Caused By java.lang.IllegalArgumentException: No enum constant 
> org.apahe.nifi.provenance.ProvenenceEventType.
> at java.lang/.Enum.valueOf...
> at 
> org.apache.nifi.provenance.ProvenanaceEventType.valueOf(ProvenanaceEventType.java:19)
> at
> org.apache.nifi.provenance.StandardRecordReader.nextRecord(StandardRecordReader.java:284)
> at 
> org.apache.nifi.provenance.PersistentProvenanceRepository.mergeJournals(PersistentProvenanceRepository.java:1678)
>  \[2\]
> ...
> {quote}
> In 0.x HEAD, this appears to have moved a bit\[3\]. Same is true for 
> 1.x\[4\]. In both 0.x\[5\] and 1.x\[6\], code was added to check for "bad" 
> records coming from the reader, and skip them. However, this appears to only 
> address the issue if the corrupt / bad record is the *first* record coming 
> out of a reader. If it is a subsequent record, the corruption remains.
> \[1\] https://issues.apache.org/jira/browse/NIFI-803
> \[2\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[3\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1711
> \[4\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[5\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1564-L1578
> \[6\] 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1658-L1672



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-2890) Provenance Repository corruption

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866292#comment-15866292
 ] 

ASF GitHub Bot commented on NIFI-2890:
--

Github user jskora closed the pull request at:

https://github.com/apache/nifi/pull/1203


> Provenance Repository corruption
> 
>
> Key: NIFI-2890
> URL: https://issues.apache.org/jira/browse/NIFI-2890
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Brandon DeVries
>Assignee: Joe Skora
> Fix For: 0.8.0, 1.2.0
>
>
> It looks as thought there was a ticket to address this\[1\], but it appears 
> to not have completely addressed the issue. On an 0.6.1 instance (that 
> crashed hard), I'm getting a stack trace on startup indicating a corrupt prov 
> repo. The relevant part of the stack trace (unfortunately typed by hand, so 
> forgive the inevitable typos) is:
> {quote}
> ...
> Caused By java.lang.IllegalArgumentException: No enum constant 
> org.apahe.nifi.provenance.ProvenenceEventType.
> at java.lang/.Enum.valueOf...
> at 
> org.apache.nifi.provenance.ProvenanaceEventType.valueOf(ProvenanaceEventType.java:19)
> at
> org.apache.nifi.provenance.StandardRecordReader.nextRecord(StandardRecordReader.java:284)
> at 
> org.apache.nifi.provenance.PersistentProvenanceRepository.mergeJournals(PersistentProvenanceRepository.java:1678)
>  \[2\]
> ...
> {quote}
> In 0.x HEAD, this appears to have moved a bit\[3\]. Same is true for 
> 1.x\[4\]. In both 0.x\[5\] and 1.x\[6\], code was added to check for "bad" 
> records coming from the reader, and skip them. However, this appears to only 
> address the issue if the corrupt / bad record is the *first* record coming 
> out of a reader. If it is a subsequent record, the corruption remains.
> \[1\] https://issues.apache.org/jira/browse/NIFI-803
> \[2\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[3\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1711
> \[4\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[5\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1564-L1578
> \[6\] 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1658-L1672



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-2890) Provenance Repository corruption

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866284#comment-15866284
 ] 

ASF GitHub Bot commented on NIFI-2890:
--

Github user jskora commented on the issue:

https://github.com/apache/nifi/pull/1201
  
Replaced by #1484.


> Provenance Repository corruption
> 
>
> Key: NIFI-2890
> URL: https://issues.apache.org/jira/browse/NIFI-2890
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Brandon DeVries
>Assignee: Joe Skora
> Fix For: 0.8.0, 1.2.0
>
>
> It looks as thought there was a ticket to address this\[1\], but it appears 
> to not have completely addressed the issue. On an 0.6.1 instance (that 
> crashed hard), I'm getting a stack trace on startup indicating a corrupt prov 
> repo. The relevant part of the stack trace (unfortunately typed by hand, so 
> forgive the inevitable typos) is:
> {quote}
> ...
> Caused By java.lang.IllegalArgumentException: No enum constant 
> org.apahe.nifi.provenance.ProvenenceEventType.
> at java.lang/.Enum.valueOf...
> at 
> org.apache.nifi.provenance.ProvenanaceEventType.valueOf(ProvenanaceEventType.java:19)
> at
> org.apache.nifi.provenance.StandardRecordReader.nextRecord(StandardRecordReader.java:284)
> at 
> org.apache.nifi.provenance.PersistentProvenanceRepository.mergeJournals(PersistentProvenanceRepository.java:1678)
>  \[2\]
> ...
> {quote}
> In 0.x HEAD, this appears to have moved a bit\[3\]. Same is true for 
> 1.x\[4\]. In both 0.x\[5\] and 1.x\[6\], code was added to check for "bad" 
> records coming from the reader, and skip them. However, this appears to only 
> address the issue if the corrupt / bad record is the *first* record coming 
> out of a reader. If it is a subsequent record, the corruption remains.
> \[1\] https://issues.apache.org/jira/browse/NIFI-803
> \[2\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[3\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1711
> \[4\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[5\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1564-L1578
> \[6\] 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1658-L1672



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-2890) Provenance Repository corruption

[Mark Payne (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865934#comment-15865934
 ] 

Mark Payne commented on NIFI-2890:
--

@jskora I have merged and closed PR #1484 and PR #1485. I think #1201 and #1203 
are OBE? Can you please close those PR's, or correct if I am wrong? Thanks!

> Provenance Repository corruption
> 
>
> Key: NIFI-2890
> URL: https://issues.apache.org/jira/browse/NIFI-2890
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Brandon DeVries
>Assignee: Joe Skora
> Fix For: 0.8.0, 1.2.0
>
>
> It looks as thought there was a ticket to address this\[1\], but it appears 
> to not have completely addressed the issue. On an 0.6.1 instance (that 
> crashed hard), I'm getting a stack trace on startup indicating a corrupt prov 
> repo. The relevant part of the stack trace (unfortunately typed by hand, so 
> forgive the inevitable typos) is:
> {quote}
> ...
> Caused By java.lang.IllegalArgumentException: No enum constant 
> org.apahe.nifi.provenance.ProvenenceEventType.
> at java.lang/.Enum.valueOf...
> at 
> org.apache.nifi.provenance.ProvenanaceEventType.valueOf(ProvenanaceEventType.java:19)
> at
> org.apache.nifi.provenance.StandardRecordReader.nextRecord(StandardRecordReader.java:284)
> at 
> org.apache.nifi.provenance.PersistentProvenanceRepository.mergeJournals(PersistentProvenanceRepository.java:1678)
>  \[2\]
> ...
> {quote}
> In 0.x HEAD, this appears to have moved a bit\[3\]. Same is true for 
> 1.x\[4\]. In both 0.x\[5\] and 1.x\[6\], code was added to check for "bad" 
> records coming from the reader, and skip them. However, this appears to only 
> address the issue if the corrupt / bad record is the *first* record coming 
> out of a reader. If it is a subsequent record, the corruption remains.
> \[1\] https://issues.apache.org/jira/browse/NIFI-803
> \[2\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[3\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1711
> \[4\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[5\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1564-L1578
> \[6\] 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1658-L1672



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-2890) Provenance Repository corruption

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865922#comment-15865922
 ] 

ASF GitHub Bot commented on NIFI-2890:
--

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/1484
  
Looks good. Merged to 0.x brach. Thanks!


> Provenance Repository corruption
> 
>
> Key: NIFI-2890
> URL: https://issues.apache.org/jira/browse/NIFI-2890
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Brandon DeVries
>Assignee: Joe Skora
> Fix For: 0.8.0, 1.2.0
>
>
> It looks as thought there was a ticket to address this\[1\], but it appears 
> to not have completely addressed the issue. On an 0.6.1 instance (that 
> crashed hard), I'm getting a stack trace on startup indicating a corrupt prov 
> repo. The relevant part of the stack trace (unfortunately typed by hand, so 
> forgive the inevitable typos) is:
> {quote}
> ...
> Caused By java.lang.IllegalArgumentException: No enum constant 
> org.apahe.nifi.provenance.ProvenenceEventType.
> at java.lang/.Enum.valueOf...
> at 
> org.apache.nifi.provenance.ProvenanaceEventType.valueOf(ProvenanaceEventType.java:19)
> at
> org.apache.nifi.provenance.StandardRecordReader.nextRecord(StandardRecordReader.java:284)
> at 
> org.apache.nifi.provenance.PersistentProvenanceRepository.mergeJournals(PersistentProvenanceRepository.java:1678)
>  \[2\]
> ...
> {quote}
> In 0.x HEAD, this appears to have moved a bit\[3\]. Same is true for 
> 1.x\[4\]. In both 0.x\[5\] and 1.x\[6\], code was added to check for "bad" 
> records coming from the reader, and skip them. However, this appears to only 
> address the issue if the corrupt / bad record is the *first* record coming 
> out of a reader. If it is a subsequent record, the corruption remains.
> \[1\] https://issues.apache.org/jira/browse/NIFI-803
> \[2\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[3\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1711
> \[4\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[5\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1564-L1578
> \[6\] 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1658-L1672



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-2890) Provenance Repository corruption

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865910#comment-15865910
 ] 

ASF GitHub Bot commented on NIFI-2890:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1485


> Provenance Repository corruption
> 
>
> Key: NIFI-2890
> URL: https://issues.apache.org/jira/browse/NIFI-2890
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Brandon DeVries
>Assignee: Joe Skora
>
> It looks as thought there was a ticket to address this\[1\], but it appears 
> to not have completely addressed the issue. On an 0.6.1 instance (that 
> crashed hard), I'm getting a stack trace on startup indicating a corrupt prov 
> repo. The relevant part of the stack trace (unfortunately typed by hand, so 
> forgive the inevitable typos) is:
> {quote}
> ...
> Caused By java.lang.IllegalArgumentException: No enum constant 
> org.apahe.nifi.provenance.ProvenenceEventType.
> at java.lang/.Enum.valueOf...
> at 
> org.apache.nifi.provenance.ProvenanaceEventType.valueOf(ProvenanaceEventType.java:19)
> at
> org.apache.nifi.provenance.StandardRecordReader.nextRecord(StandardRecordReader.java:284)
> at 
> org.apache.nifi.provenance.PersistentProvenanceRepository.mergeJournals(PersistentProvenanceRepository.java:1678)
>  \[2\]
> ...
> {quote}
> In 0.x HEAD, this appears to have moved a bit\[3\]. Same is true for 
> 1.x\[4\]. In both 0.x\[5\] and 1.x\[6\], code was added to check for "bad" 
> records coming from the reader, and skip them. However, this appears to only 
> address the issue if the corrupt / bad record is the *first* record coming 
> out of a reader. If it is a subsequent record, the corruption remains.
> \[1\] https://issues.apache.org/jira/browse/NIFI-803
> \[2\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[3\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1711
> \[4\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[5\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1564-L1578
> \[6\] 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1658-L1672



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-2890) Provenance Repository corruption

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865909#comment-15865909
 ] 

ASF GitHub Bot commented on NIFI-2890:
--

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/1485
  
@jskora this is a good catch. Code looks good. +1 merged to master.


> Provenance Repository corruption
> 
>
> Key: NIFI-2890
> URL: https://issues.apache.org/jira/browse/NIFI-2890
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Brandon DeVries
>Assignee: Joe Skora
>
> It looks as thought there was a ticket to address this\[1\], but it appears 
> to not have completely addressed the issue. On an 0.6.1 instance (that 
> crashed hard), I'm getting a stack trace on startup indicating a corrupt prov 
> repo. The relevant part of the stack trace (unfortunately typed by hand, so 
> forgive the inevitable typos) is:
> {quote}
> ...
> Caused By java.lang.IllegalArgumentException: No enum constant 
> org.apahe.nifi.provenance.ProvenenceEventType.
> at java.lang/.Enum.valueOf...
> at 
> org.apache.nifi.provenance.ProvenanaceEventType.valueOf(ProvenanaceEventType.java:19)
> at
> org.apache.nifi.provenance.StandardRecordReader.nextRecord(StandardRecordReader.java:284)
> at 
> org.apache.nifi.provenance.PersistentProvenanceRepository.mergeJournals(PersistentProvenanceRepository.java:1678)
>  \[2\]
> ...
> {quote}
> In 0.x HEAD, this appears to have moved a bit\[3\]. Same is true for 
> 1.x\[4\]. In both 0.x\[5\] and 1.x\[6\], code was added to check for "bad" 
> records coming from the reader, and skip them. However, this appears to only 
> address the issue if the corrupt / bad record is the *first* record coming 
> out of a reader. If it is a subsequent record, the corruption remains.
> \[1\] https://issues.apache.org/jira/browse/NIFI-803
> \[2\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[3\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1711
> \[4\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[5\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1564-L1578
> \[6\] 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1658-L1672



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-2890) Provenance Repository corruption

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865908#comment-15865908
 ] 

ASF subversion and git services commented on NIFI-2890:
---

Commit bd3e0438a32a7107d91d2e98e47dcf64af2021b4 in nifi's branch 
refs/heads/master from [~jskora]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=bd3e043 ]

NIFI-2890 Provenance Repository Corruption (1.x)
* Corrected handling of corrupt journal file records that prevents instance 
startup and loss of records from corrupt files.  Specifically, exception 
handling was expanded to cover failures on records after the first the same as 
failures on the first record.
* Adjusted log messages  to reflect that the remainder or all of the journal 
will be skipped, not just the current record.

This closes #1485.


> Provenance Repository corruption
> 
>
> Key: NIFI-2890
> URL: https://issues.apache.org/jira/browse/NIFI-2890
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Brandon DeVries
>Assignee: Joe Skora
>
> It looks as thought there was a ticket to address this\[1\], but it appears 
> to not have completely addressed the issue. On an 0.6.1 instance (that 
> crashed hard), I'm getting a stack trace on startup indicating a corrupt prov 
> repo. The relevant part of the stack trace (unfortunately typed by hand, so 
> forgive the inevitable typos) is:
> {quote}
> ...
> Caused By java.lang.IllegalArgumentException: No enum constant 
> org.apahe.nifi.provenance.ProvenenceEventType.
> at java.lang/.Enum.valueOf...
> at 
> org.apache.nifi.provenance.ProvenanaceEventType.valueOf(ProvenanaceEventType.java:19)
> at
> org.apache.nifi.provenance.StandardRecordReader.nextRecord(StandardRecordReader.java:284)
> at 
> org.apache.nifi.provenance.PersistentProvenanceRepository.mergeJournals(PersistentProvenanceRepository.java:1678)
>  \[2\]
> ...
> {quote}
> In 0.x HEAD, this appears to have moved a bit\[3\]. Same is true for 
> 1.x\[4\]. In both 0.x\[5\] and 1.x\[6\], code was added to check for "bad" 
> records coming from the reader, and skip them. However, this appears to only 
> address the issue if the corrupt / bad record is the *first* record coming 
> out of a reader. If it is a subsequent record, the corruption remains.
> \[1\] https://issues.apache.org/jira/browse/NIFI-803
> \[2\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[3\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1711
> \[4\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[5\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1564-L1578
> \[6\] 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1658-L1672



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-2890) Provenance Repository corruption

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15858105#comment-15858105
 ] 

ASF GitHub Bot commented on NIFI-2890:
--

GitHub user jskora opened a pull request:

https://github.com/apache/nifi/pull/1485

NIFI-2890 Provenance Repository Corruption (1.x)

* Corrected handling of corrupt journal file records that prevents instance 
startup and loss of records from corrupt files.  Specifically, exception 
handling was expanded to cover failures on records after the first the same as 
failures on the first record.
* Adjusted log messages  to reflect that the remainder or all of the 
journal will be skipped, not just the current record.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jskora/nifi NIFI-2890-1.x-v2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1485.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1485


commit d844c21683cd177e7ae3122762bbe4c737a23934
Author: Joe Skora 
Date:   2017-02-07T20:43:08Z

NIFI-2890 Provenance Repository Corruption (1.x)
* Corrected handling of corrupt journal file records that prevents instance 
startup and loss of records from corrupt files.  Specifically, exception 
handling was expanded to cover failures on records after the first the same as 
failures on the first record.
* Adjusted log messages  to reflect that the remainder or all of the 
journal will be skipped, not just the current record.




> Provenance Repository corruption
> 
>
> Key: NIFI-2890
> URL: https://issues.apache.org/jira/browse/NIFI-2890
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Brandon DeVries
>Assignee: Joe Skora
>
> It looks as thought there was a ticket to address this\[1\], but it appears 
> to not have completely addressed the issue. On an 0.6.1 instance (that 
> crashed hard), I'm getting a stack trace on startup indicating a corrupt prov 
> repo. The relevant part of the stack trace (unfortunately typed by hand, so 
> forgive the inevitable typos) is:
> {quote}
> ...
> Caused By java.lang.IllegalArgumentException: No enum constant 
> org.apahe.nifi.provenance.ProvenenceEventType.
> at java.lang/.Enum.valueOf...
> at 
> org.apache.nifi.provenance.ProvenanaceEventType.valueOf(ProvenanaceEventType.java:19)
> at
> org.apache.nifi.provenance.StandardRecordReader.nextRecord(StandardRecordReader.java:284)
> at 
> org.apache.nifi.provenance.PersistentProvenanceRepository.mergeJournals(PersistentProvenanceRepository.java:1678)
>  \[2\]
> ...
> {quote}
> In 0.x HEAD, this appears to have moved a bit\[3\]. Same is true for 
> 1.x\[4\]. In both 0.x\[5\] and 1.x\[6\], code was added to check for "bad" 
> records coming from the reader, and skip them. However, this appears to only 
> address the issue if the 

[jira] [Commented] (NIFI-2890) Provenance Repository corruption

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15858104#comment-15858104
 ] 

ASF GitHub Bot commented on NIFI-2890:
--

GitHub user jskora opened a pull request:

https://github.com/apache/nifi/pull/1484

NIFI-2890 Provenance Repository Corruption (0.x)

* Corrected handling of corrupt journal file records that prevents instance 
startup and loss of records from corrupt files.  Specifically, exception 
handling was expanded to cover failures on records after the first the same as 
failures on the first record.
* Adjusted log messages  to reflect that the remainder or all of the 
journal will be skipped, not just the current record.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jskora/nifi NIFI-2890-0.x-v2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1484.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1484


commit cedbbb5aef4c457f0a44435d4201ae70c600f2db
Author: Joe Skora 
Date:   2017-02-07T21:02:19Z

NIFI-2890 Provenance Repository Corruption (0.x)
* Corrected handling of corrupt journal file records that prevents instance 
startup and loss of records from corrupt files.  Specifically, exception 
handling was expanded to cover failures on records after the first the same as 
failures on the first record.
* Adjusted log messages  to reflect that the remainder or all of the 
journal will be skipped, not just the current record.




> Provenance Repository corruption
> 
>
> Key: NIFI-2890
> URL: https://issues.apache.org/jira/browse/NIFI-2890
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Brandon DeVries
>Assignee: Joe Skora
>
> It looks as thought there was a ticket to address this\[1\], but it appears 
> to not have completely addressed the issue. On an 0.6.1 instance (that 
> crashed hard), I'm getting a stack trace on startup indicating a corrupt prov 
> repo. The relevant part of the stack trace (unfortunately typed by hand, so 
> forgive the inevitable typos) is:
> {quote}
> ...
> Caused By java.lang.IllegalArgumentException: No enum constant 
> org.apahe.nifi.provenance.ProvenenceEventType.
> at java.lang/.Enum.valueOf...
> at 
> org.apache.nifi.provenance.ProvenanaceEventType.valueOf(ProvenanaceEventType.java:19)
> at
> org.apache.nifi.provenance.StandardRecordReader.nextRecord(StandardRecordReader.java:284)
> at 
> org.apache.nifi.provenance.PersistentProvenanceRepository.mergeJournals(PersistentProvenanceRepository.java:1678)
>  \[2\]
> ...
> {quote}
> In 0.x HEAD, this appears to have moved a bit\[3\]. Same is true for 
> 1.x\[4\]. In both 0.x\[5\] and 1.x\[6\], code was added to check for "bad" 
> records coming from the reader, and skip them. However, this appears to only 
> address the issue if the 

[jira] [Commented] (NIFI-2890) Provenance Repository corruption

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15805509#comment-15805509
 ] 

ASF GitHub Bot commented on NIFI-2890:
--

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/1201
  
Hey @jskora, it appears you have check style issues. 

I'm not entirely sure how you pulled in the commits but are you able to 
rebase instead? This throws off how I normally squash/sign-off PRs.


> Provenance Repository corruption
> 
>
> Key: NIFI-2890
> URL: https://issues.apache.org/jira/browse/NIFI-2890
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Brandon DeVries
>Assignee: Joe Skora
>
> It looks as thought there was a ticket to address this\[1\], but it appears 
> to not have completely addressed the issue. On an 0.6.1 instance (that 
> crashed hard), I'm getting a stack trace on startup indicating a corrupt prov 
> repo. The relevant part of the stack trace (unfortunately typed by hand, so 
> forgive the inevitable typos) is:
> {quote}
> ...
> Caused By java.lang.IllegalArgumentException: No enum constant 
> org.apahe.nifi.provenance.ProvenenceEventType.
> at java.lang/.Enum.valueOf...
> at 
> org.apache.nifi.provenance.ProvenanaceEventType.valueOf(ProvenanaceEventType.java:19)
> at
> org.apache.nifi.provenance.StandardRecordReader.nextRecord(StandardRecordReader.java:284)
> at 
> org.apache.nifi.provenance.PersistentProvenanceRepository.mergeJournals(PersistentProvenanceRepository.java:1678)
>  \[2\]
> ...
> {quote}
> In 0.x HEAD, this appears to have moved a bit\[3\]. Same is true for 
> 1.x\[4\]. In both 0.x\[5\] and 1.x\[6\], code was added to check for "bad" 
> records coming from the reader, and skip them. However, this appears to only 
> address the issue if the corrupt / bad record is the *first* record coming 
> out of a reader. If it is a subsequent record, the corruption remains.
> \[1\] https://issues.apache.org/jira/browse/NIFI-803
> \[2\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[3\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1711
> \[4\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[5\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1564-L1578
> \[6\] 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1658-L1672



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2890) Provenance Repository corruption

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15804782#comment-15804782
 ] 

ASF GitHub Bot commented on NIFI-2890:
--

Github user jskora commented on the issue:

https://github.com/apache/nifi/pull/1203
  
@JPercivall This has @been updated to align with the changes in Apache/0.x. 
 I couldn't squash without creating a new branch, but a diff with Apache/0.x 
only shows the PersistentProvenanceRepository and 
TestPersistentProvenanceRepository classes.


> Provenance Repository corruption
> 
>
> Key: NIFI-2890
> URL: https://issues.apache.org/jira/browse/NIFI-2890
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Brandon DeVries
>Assignee: Joe Skora
>
> It looks as thought there was a ticket to address this\[1\], but it appears 
> to not have completely addressed the issue. On an 0.6.1 instance (that 
> crashed hard), I'm getting a stack trace on startup indicating a corrupt prov 
> repo. The relevant part of the stack trace (unfortunately typed by hand, so 
> forgive the inevitable typos) is:
> {quote}
> ...
> Caused By java.lang.IllegalArgumentException: No enum constant 
> org.apahe.nifi.provenance.ProvenenceEventType.
> at java.lang/.Enum.valueOf...
> at 
> org.apache.nifi.provenance.ProvenanaceEventType.valueOf(ProvenanaceEventType.java:19)
> at
> org.apache.nifi.provenance.StandardRecordReader.nextRecord(StandardRecordReader.java:284)
> at 
> org.apache.nifi.provenance.PersistentProvenanceRepository.mergeJournals(PersistentProvenanceRepository.java:1678)
>  \[2\]
> ...
> {quote}
> In 0.x HEAD, this appears to have moved a bit\[3\]. Same is true for 
> 1.x\[4\]. In both 0.x\[5\] and 1.x\[6\], code was added to check for "bad" 
> records coming from the reader, and skip them. However, this appears to only 
> address the issue if the corrupt / bad record is the *first* record coming 
> out of a reader. If it is a subsequent record, the corruption remains.
> \[1\] https://issues.apache.org/jira/browse/NIFI-803
> \[2\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[3\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1711
> \[4\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[5\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1564-L1578
> \[6\] 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1658-L1672



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2890) Provenance Repository corruption

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15804780#comment-15804780
 ] 

ASF GitHub Bot commented on NIFI-2890:
--

Github user jskora commented on the issue:

https://github.com/apache/nifi/pull/1201
  
@JPercivall this has been updated to align with other changes in Master.  I 
couldn't squash without creating a new branch, but a diff with Master only 
shows the PersistentProvenanceRepository and TestPersistentProvenanceRepository 
classes.


> Provenance Repository corruption
> 
>
> Key: NIFI-2890
> URL: https://issues.apache.org/jira/browse/NIFI-2890
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Brandon DeVries
>Assignee: Joe Skora
>
> It looks as thought there was a ticket to address this\[1\], but it appears 
> to not have completely addressed the issue. On an 0.6.1 instance (that 
> crashed hard), I'm getting a stack trace on startup indicating a corrupt prov 
> repo. The relevant part of the stack trace (unfortunately typed by hand, so 
> forgive the inevitable typos) is:
> {quote}
> ...
> Caused By java.lang.IllegalArgumentException: No enum constant 
> org.apahe.nifi.provenance.ProvenenceEventType.
> at java.lang/.Enum.valueOf...
> at 
> org.apache.nifi.provenance.ProvenanaceEventType.valueOf(ProvenanaceEventType.java:19)
> at
> org.apache.nifi.provenance.StandardRecordReader.nextRecord(StandardRecordReader.java:284)
> at 
> org.apache.nifi.provenance.PersistentProvenanceRepository.mergeJournals(PersistentProvenanceRepository.java:1678)
>  \[2\]
> ...
> {quote}
> In 0.x HEAD, this appears to have moved a bit\[3\]. Same is true for 
> 1.x\[4\]. In both 0.x\[5\] and 1.x\[6\], code was added to check for "bad" 
> records coming from the reader, and skip them. However, this appears to only 
> address the issue if the corrupt / bad record is the *first* record coming 
> out of a reader. If it is a subsequent record, the corruption remains.
> \[1\] https://issues.apache.org/jira/browse/NIFI-803
> \[2\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[3\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1711
> \[4\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[5\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1564-L1578
> \[6\] 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1658-L1672



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2890) Provenance Repository corruption

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796074#comment-15796074
 ] 

ASF GitHub Bot commented on NIFI-2890:
--

Github user jskora commented on the issue:

https://github.com/apache/nifi/pull/1201
  
@JPercivall, no problem.  I'll take a look at this and 
[https://github.com/apache/nifi/pull/1203](1203)


> Provenance Repository corruption
> 
>
> Key: NIFI-2890
> URL: https://issues.apache.org/jira/browse/NIFI-2890
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Brandon DeVries
>Assignee: Joe Skora
>
> It looks as thought there was a ticket to address this\[1\], but it appears 
> to not have completely addressed the issue. On an 0.6.1 instance (that 
> crashed hard), I'm getting a stack trace on startup indicating a corrupt prov 
> repo. The relevant part of the stack trace (unfortunately typed by hand, so 
> forgive the inevitable typos) is:
> {quote}
> ...
> Caused By java.lang.IllegalArgumentException: No enum constant 
> org.apahe.nifi.provenance.ProvenenceEventType.
> at java.lang/.Enum.valueOf...
> at 
> org.apache.nifi.provenance.ProvenanaceEventType.valueOf(ProvenanaceEventType.java:19)
> at
> org.apache.nifi.provenance.StandardRecordReader.nextRecord(StandardRecordReader.java:284)
> at 
> org.apache.nifi.provenance.PersistentProvenanceRepository.mergeJournals(PersistentProvenanceRepository.java:1678)
>  \[2\]
> ...
> {quote}
> In 0.x HEAD, this appears to have moved a bit\[3\]. Same is true for 
> 1.x\[4\]. In both 0.x\[5\] and 1.x\[6\], code was added to check for "bad" 
> records coming from the reader, and skip them. However, this appears to only 
> address the issue if the corrupt / bad record is the *first* record coming 
> out of a reader. If it is a subsequent record, the corruption remains.
> \[1\] https://issues.apache.org/jira/browse/NIFI-803
> \[2\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[3\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1711
> \[4\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[5\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1564-L1578
> \[6\] 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1658-L1672



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2890) Provenance Repository corruption

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796026#comment-15796026
 ] 

ASF GitHub Bot commented on NIFI-2890:
--

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/1201
  
Hey @jskora, sorry this has sat for so long, this PR review should have 
been prioritized. That said, there appear to be merge conflicts and checkstyle 
issues. I could resolve them for you if you wanted but, if able, I'd prefer you 
fix them.


> Provenance Repository corruption
> 
>
> Key: NIFI-2890
> URL: https://issues.apache.org/jira/browse/NIFI-2890
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Brandon DeVries
>Assignee: Joe Skora
>
> It looks as thought there was a ticket to address this\[1\], but it appears 
> to not have completely addressed the issue. On an 0.6.1 instance (that 
> crashed hard), I'm getting a stack trace on startup indicating a corrupt prov 
> repo. The relevant part of the stack trace (unfortunately typed by hand, so 
> forgive the inevitable typos) is:
> {quote}
> ...
> Caused By java.lang.IllegalArgumentException: No enum constant 
> org.apahe.nifi.provenance.ProvenenceEventType.
> at java.lang/.Enum.valueOf...
> at 
> org.apache.nifi.provenance.ProvenanaceEventType.valueOf(ProvenanaceEventType.java:19)
> at
> org.apache.nifi.provenance.StandardRecordReader.nextRecord(StandardRecordReader.java:284)
> at 
> org.apache.nifi.provenance.PersistentProvenanceRepository.mergeJournals(PersistentProvenanceRepository.java:1678)
>  \[2\]
> ...
> {quote}
> In 0.x HEAD, this appears to have moved a bit\[3\]. Same is true for 
> 1.x\[4\]. In both 0.x\[5\] and 1.x\[6\], code was added to check for "bad" 
> records coming from the reader, and skip them. However, this appears to only 
> address the issue if the corrupt / bad record is the *first* record coming 
> out of a reader. If it is a subsequent record, the corruption remains.
> \[1\] https://issues.apache.org/jira/browse/NIFI-803
> \[2\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[3\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1711
> \[4\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[5\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1564-L1578
> \[6\] 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1658-L1672



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2890) Provenance Repository corruption

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15795877#comment-15795877
 ] 

ASF GitHub Bot commented on NIFI-2890:
--

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/1201
  
Reviewing


> Provenance Repository corruption
> 
>
> Key: NIFI-2890
> URL: https://issues.apache.org/jira/browse/NIFI-2890
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Brandon DeVries
>Assignee: Joe Skora
>
> It looks as thought there was a ticket to address this\[1\], but it appears 
> to not have completely addressed the issue. On an 0.6.1 instance (that 
> crashed hard), I'm getting a stack trace on startup indicating a corrupt prov 
> repo. The relevant part of the stack trace (unfortunately typed by hand, so 
> forgive the inevitable typos) is:
> {quote}
> ...
> Caused By java.lang.IllegalArgumentException: No enum constant 
> org.apahe.nifi.provenance.ProvenenceEventType.
> at java.lang/.Enum.valueOf...
> at 
> org.apache.nifi.provenance.ProvenanaceEventType.valueOf(ProvenanaceEventType.java:19)
> at
> org.apache.nifi.provenance.StandardRecordReader.nextRecord(StandardRecordReader.java:284)
> at 
> org.apache.nifi.provenance.PersistentProvenanceRepository.mergeJournals(PersistentProvenanceRepository.java:1678)
>  \[2\]
> ...
> {quote}
> In 0.x HEAD, this appears to have moved a bit\[3\]. Same is true for 
> 1.x\[4\]. In both 0.x\[5\] and 1.x\[6\], code was added to check for "bad" 
> records coming from the reader, and skip them. However, this appears to only 
> address the issue if the corrupt / bad record is the *first* record coming 
> out of a reader. If it is a subsequent record, the corruption remains.
> \[1\] https://issues.apache.org/jira/browse/NIFI-803
> \[2\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[3\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1711
> \[4\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[5\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1564-L1578
> \[6\] 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1658-L1672



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2890) Provenance Repository corruption

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15654975#comment-15654975
 ] 

ASF GitHub Bot commented on NIFI-2890:
--

GitHub user jskora opened a pull request:

https://github.com/apache/nifi/pull/1203

NIFI-2890 Provenance Repository Corruption

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [X ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [X ] Does your PR title start with NIFI- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.

- [X ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [X ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [X ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.

Correct handling of corrupt journal file that was preventing instance 
startup and causing loss of records from uncorrupt files.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jskora/nifi NIFI-2890-0.x

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1203.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1203


commit bdb00ecd78d16dd33903c3cfd69271444ccf0f37
Author: Joe Skora 
Date:   2016-11-10T19:52:18Z

NIFI-2890 Provenance Repository Corruption - Correct handling of corrupt 
journal file that was preventing instance startup and causing loss of records 
from uncorrupt files. (0.x)




> Provenance Repository corruption
> 
>
> Key: NIFI-2890
> URL: https://issues.apache.org/jira/browse/NIFI-2890
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Brandon DeVries
>Assignee: Joe Skora
>
> It looks as thought there was a ticket to address this\[1\], but it appears 
> to not have completely addressed the issue. On an 0.6.1 instance (that 
> crashed hard), I'm getting a stack trace on startup indicating a corrupt prov 
> repo. The relevant part of the stack trace (unfortunately typed by hand, so 
> forgive the inevitable typos) is:
> {quote}
> ...
> Caused By java.lang.IllegalArgumentException: No enum constant 
> org.apahe.nifi.provenance.ProvenenceEventType.
> at java.lang/.Enum.valueOf...
> at 
> org.apache.nifi.provenance.ProvenanaceEventType.valueOf(ProvenanaceEventType.java:19)
> at
> org.apache.nifi.provenance.StandardRecordReader.nextRecord(StandardRecordReader.java:284)
> at 
> org.apache.nifi.provenance.PersistentProvenanceRepository.mergeJournals(PersistentProvenanceRepository.java:1678)
>  \[2\]
> ...
> {quote}
> In 0.x HEAD, this appears to have moved a bit\[3\]. Same is true for 
> 1.x\[4\]. In both 0.x\[5\] and 1.x\[6\], code was added to check for "bad" 
> records coming from the reader, and skip them. However, this appears to only 
> address the issue if the corrupt / bad record is the *first* record coming 
> out of a reader. If it is a subsequent record, the corruption remains.
> \[1\] https://issues.apache.org/jira/browse/NIFI-803
> \[2\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[3\] 
> 

[jira] [Commented] (NIFI-2890) Provenance Repository corruption

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15654762#comment-15654762
 ] 

ASF GitHub Bot commented on NIFI-2890:
--

GitHub user jskora opened a pull request:

https://github.com/apache/nifi/pull/1201

NIFI-2890 Provenance Repository Corruption

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [X ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [X ] Does your PR title start with NIFI- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.

- [X ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [X ] Is your initial contribution a single, squashed commit?

### For code changes:
- [X ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [X ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.

Correct handling of corrupt journal file that was preventing instance 
startup and causing loss of records from uncorrupt files.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jskora/nifi NIFI-2890-1.x

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1201.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1201


commit d89d5f684e1ea19b74167a03c6b987e161fceab1
Author: Joe Skora 
Date:   2016-11-10T18:06:41Z

NIFI-2890 Provenance Repository Corruption - Correct handling of corrupt 
journal file that was preventing instance startup and causing loss of records 
from uncorrupt files.




> Provenance Repository corruption
> 
>
> Key: NIFI-2890
> URL: https://issues.apache.org/jira/browse/NIFI-2890
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Brandon DeVries
>Assignee: Joe Skora
>
> It looks as thought there was a ticket to address this\[1\], but it appears 
> to not have completely addressed the issue. On an 0.6.1 instance (that 
> crashed hard), I'm getting a stack trace on startup indicating a corrupt prov 
> repo. The relevant part of the stack trace (unfortunately typed by hand, so 
> forgive the inevitable typos) is:
> {quote}
> ...
> Caused By java.lang.IllegalArgumentException: No enum constant 
> org.apahe.nifi.provenance.ProvenenceEventType.
> at java.lang/.Enum.valueOf...
> at 
> org.apache.nifi.provenance.ProvenanaceEventType.valueOf(ProvenanaceEventType.java:19)
> at
> org.apache.nifi.provenance.StandardRecordReader.nextRecord(StandardRecordReader.java:284)
> at 
> org.apache.nifi.provenance.PersistentProvenanceRepository.mergeJournals(PersistentProvenanceRepository.java:1678)
>  \[2\]
> ...
> {quote}
> In 0.x HEAD, this appears to have moved a bit\[3\]. Same is true for 
> 1.x\[4\]. In both 0.x\[5\] and 1.x\[6\], code was added to check for "bad" 
> records coming from the reader, and skip them. However, this appears to only 
> address the issue if the corrupt / bad record is the *first* record coming 
> out of a reader. If it is a subsequent record, the corruption remains.
> \[1\] https://issues.apache.org/jira/browse/NIFI-803
> \[2\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[3\] 
> 

[jira] [Commented] (NIFI-2890) Provenance Repository corruption

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15624098#comment-15624098
 ] 

Joseph Witt commented on NIFI-2890:
---

[~devriesb] on Oct 11th Writes:
 I just opened a ticket to address an issue we've encountered with
Provenance repo corruption[1].  This would address (as is currently
partially being done) how to recover from a corrupt provenance repo.
However, the question is whether we can avoid this sort of corruption in
the first place.  The immediate thought that jumped to mind was wrapping
the writes to lucene with a write ahead log.  Obviously, this would
increase the overhead on something that is already fairly expensive.
However, in cases where provenance is *really* important, it might be worth
considering.  This could potentially be another flavor of
ProvenanceEventRepository, e.g.  WriteAheadPersistentProvenanceRepository
or FaultTolerantProvenanceRepository.  Does anyone have any thoughts /
opinions?


--- 

This looks like an important issue.  [~markap14] Can you take a look at this?

> Provenance Repository corruption
> 
>
> Key: NIFI-2890
> URL: https://issues.apache.org/jira/browse/NIFI-2890
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Brandon DeVries
>
> It looks as thought there was a ticket to address this\[1\], but it appears 
> to not have completely addressed the issue. On an 0.6.1 instance (that 
> crashed hard), I'm getting a stack trace on startup indicating a corrupt prov 
> repo. The relevant part of the stack trace (unfortunately typed by hand, so 
> forgive the inevitable typos) is:
> {quote}
> ...
> Caused By java.lang.IllegalArgumentException: No enum constant 
> org.apahe.nifi.provenance.ProvenenceEventType.
> at java.lang/.Enum.valueOf...
> at 
> org.apache.nifi.provenance.ProvenanaceEventType.valueOf(ProvenanaceEventType.java:19)
> at
> org.apache.nifi.provenance.StandardRecordReader.nextRecord(StandardRecordReader.java:284)
> at 
> org.apache.nifi.provenance.PersistentProvenanceRepository.mergeJournals(PersistentProvenanceRepository.java:1678)
>  \[2\]
> ...
> {quote}
> In 0.x HEAD, this appears to have moved a bit\[3\]. Same is true for 
> 1.x\[4\]. In both 0.x\[5\] and 1.x\[6\], code was added to check for "bad" 
> records coming from the reader, and skip them. However, this appears to only 
> address the issue if the corrupt / bad record is the *first* record coming 
> out of a reader. If it is a subsequent record, the corruption remains.
> \[1\] https://issues.apache.org/jira/browse/NIFI-803
> \[2\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[3\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1711
> \[4\] 
> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1678
> \[5\] 
> https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1564-L1578
> \[6\] 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1658-L1672



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)