[jira] [Commented] (NIFI-7856) Provenance failed to be compressed after nifi upgrade to 1.12
[
https://issues.apache.org/jira/browse/NIFI-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17219396#comment-17219396
]
Mengze Li commented on NIFI-7856:
-
Hi Mark,
I see your PR hasn't been merged yet however the status of this ticket is
"PATCH AVAILABLE".
Does that mean there will be a patch version including the fix?
> Provenance failed to be compressed after nifi upgrade to 1.12
> -
>
> Key: NIFI-7856
> URL: https://issues.apache.org/jira/browse/NIFI-7856
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Mengze Li
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.13.0
>
> Attachments: 1683472.prov, NIFI-7856.xml, ls.png, screenshot-1.png,
> screenshot-2.png, screenshot-3.png
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> We upgraded our nifi cluster from 1.11.3 to 1.12.0.
> The nodes come up and everything looks to be functional. I can see 1.12.0 is
> running.
> Later on, we discovered that the data provenance is missing. From checking
> our logs, we see tons of errors compressing the logs.
> {code}
> 2020-09-28 03:38:35,205 ERROR [Compress Provenance Logs-1-thread-1]
> o.a.n.p.s.EventFileCompressor Failed to compress
> ./provenance_repository/2752821.prov on rollover
> {code}
> This didn't happen in 1.11.3.
> Is this a known issue? We are considering reverting back if there is no
> solution for this since we can't go prod with no/broken data provenance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7856) Provenance failed to be compressed after nifi upgrade to 1.12
[
https://issues.apache.org/jira/browse/NIFI-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17209735#comment-17209735
]
Mengze Li commented on NIFI-7856:
-
woohoo! Keep me posted please and appreciate your time and effort looking into
this.
I assume the fix would be in 1.13.0 or another patch version of 1.12. Let me
know.
> Provenance failed to be compressed after nifi upgrade to 1.12
> -
>
> Key: NIFI-7856
> URL: https://issues.apache.org/jira/browse/NIFI-7856
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Mengze Li
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.13.0
>
> Attachments: 1683472.prov, NIFI-7856.xml, ls.png, screenshot-1.png,
> screenshot-2.png, screenshot-3.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We upgraded our nifi cluster from 1.11.3 to 1.12.0.
> The nodes come up and everything looks to be functional. I can see 1.12.0 is
> running.
> Later on, we discovered that the data provenance is missing. From checking
> our logs, we see tons of errors compressing the logs.
> {code}
> 2020-09-28 03:38:35,205 ERROR [Compress Provenance Logs-1-thread-1]
> o.a.n.p.s.EventFileCompressor Failed to compress
> ./provenance_repository/2752821.prov on rollover
> {code}
> This didn't happen in 1.11.3.
> Is this a known issue? We are considering reverting back if there is no
> solution for this since we can't go prod with no/broken data provenance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7856) Provenance failed to be compressed after nifi upgrade to 1.12
[
https://issues.apache.org/jira/browse/NIFI-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17206300#comment-17206300
]
Mengze Li commented on NIFI-7856:
-
Another observations that we had from flipping the log levels to debug was that:
the provenance files are zipped up *10 mins before* the scheduled run, and then
it tried to look for a .prov file then it couldn't find it which resulted the
error since it is already compressed.
> Provenance failed to be compressed after nifi upgrade to 1.12
> -
>
> Key: NIFI-7856
> URL: https://issues.apache.org/jira/browse/NIFI-7856
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Mengze Li
>Priority: Major
> Attachments: 1683472.prov, ls.png, screenshot-1.png,
> screenshot-2.png, screenshot-3.png
>
>
> We upgraded our nifi cluster from 1.11.3 to 1.12.0.
> The nodes come up and everything looks to be functional. I can see 1.12.0 is
> running.
> Later on, we discovered that the data provenance is missing. From checking
> our logs, we see tons of errors compressing the logs.
> {code}
> 2020-09-28 03:38:35,205 ERROR [Compress Provenance Logs-1-thread-1]
> o.a.n.p.s.EventFileCompressor Failed to compress
> ./provenance_repository/2752821.prov on rollover
> {code}
> This didn't happen in 1.11.3.
> Is this a known issue? We are considering reverting back if there is no
> solution for this since we can't go prod with no/broken data provenance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (NIFI-7856) Provenance failed to be compressed after nifi upgrade to 1.12
[
https://issues.apache.org/jira/browse/NIFI-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17206244#comment-17206244
]
Mengze Li edited comment on NIFI-7856 at 10/2/20, 3:06 PM:
---
Hi [~markap14], that's a great question since that's one major reason that we
upgraded to 1.12.0 since sometimes the data provenance is missing in the
processor view but visible in the global view in 1.11.3 that was fixed in
1.12.0.
So to answer your question, we do both and in 1.12.0, the missing and delaying
data provenance records are consistent from either view.
In global view, we usually get component id and search that way since it is
unique.
We are speculating that some processes is compressing the files before the
scheduled time and those ones didn't make it to the lucene index to be searched.
Will try to dig around the logs more to provide information that I can could
help you debug further.
was (Author: leeyoda):
Hi [~markap14], that's a great question since that's one major reason that we
upgraded to 1.12.0 since sometimes the data provenance is missing in the
processor view but visible in the global view.
So to answer your question, we do both and in 1.12.0, the missing and delaying
data provenance records are consistent from either view.
In global view, we usually get component id and search that way since it is
unique.
We are speculating that some processes is compressing the files before the
scheduled time and those ones didn't make it to the lucene index to be searched.
Will try to dig around the logs more to provide information that I can could
help you debug further.
> Provenance failed to be compressed after nifi upgrade to 1.12
> -
>
> Key: NIFI-7856
> URL: https://issues.apache.org/jira/browse/NIFI-7856
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Mengze Li
>Priority: Major
> Attachments: 1683472.prov, ls.png, screenshot-1.png,
> screenshot-2.png, screenshot-3.png
>
>
> We upgraded our nifi cluster from 1.11.3 to 1.12.0.
> The nodes come up and everything looks to be functional. I can see 1.12.0 is
> running.
> Later on, we discovered that the data provenance is missing. From checking
> our logs, we see tons of errors compressing the logs.
> {code}
> 2020-09-28 03:38:35,205 ERROR [Compress Provenance Logs-1-thread-1]
> o.a.n.p.s.EventFileCompressor Failed to compress
> ./provenance_repository/2752821.prov on rollover
> {code}
> This didn't happen in 1.11.3.
> Is this a known issue? We are considering reverting back if there is no
> solution for this since we can't go prod with no/broken data provenance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7856) Provenance failed to be compressed after nifi upgrade to 1.12
[
https://issues.apache.org/jira/browse/NIFI-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17206244#comment-17206244
]
Mengze Li commented on NIFI-7856:
-
Hi [~markap14], that's a great question since that's one major reason that we
upgraded to 1.12.0 since sometimes the data provenance is missing in the
processor view but visible in the global view.
So to answer your question, we do both and in 1.12.0, the missing and delaying
data provenance records are consistent from either view.
In global view, we usually get component id and search that way since it is
unique.
We are speculating that some processes is compressing the files before the
scheduled time and those ones didn't make it to the lucene index to be searched.
Will try to dig around the logs more to provide information that I can could
help you debug further.
> Provenance failed to be compressed after nifi upgrade to 1.12
> -
>
> Key: NIFI-7856
> URL: https://issues.apache.org/jira/browse/NIFI-7856
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Mengze Li
>Priority: Major
> Attachments: 1683472.prov, ls.png, screenshot-1.png,
> screenshot-2.png, screenshot-3.png
>
>
> We upgraded our nifi cluster from 1.11.3 to 1.12.0.
> The nodes come up and everything looks to be functional. I can see 1.12.0 is
> running.
> Later on, we discovered that the data provenance is missing. From checking
> our logs, we see tons of errors compressing the logs.
> {code}
> 2020-09-28 03:38:35,205 ERROR [Compress Provenance Logs-1-thread-1]
> o.a.n.p.s.EventFileCompressor Failed to compress
> ./provenance_repository/2752821.prov on rollover
> {code}
> This didn't happen in 1.11.3.
> Is this a known issue? We are considering reverting back if there is no
> solution for this since we can't go prod with no/broken data provenance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7856) Provenance failed to be compressed after nifi upgrade to 1.12
[
https://issues.apache.org/jira/browse/NIFI-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205531#comment-17205531
]
Mengze Li commented on NIFI-7856:
-
Thanks for the reply.
Do you mind trying ReplaceText 1.12.0 which for us, doesn't show any data
provenance since the upgrade.
A few records in the 28th pop up randomly.
> Provenance failed to be compressed after nifi upgrade to 1.12
> -
>
> Key: NIFI-7856
> URL: https://issues.apache.org/jira/browse/NIFI-7856
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Mengze Li
>Priority: Major
> Attachments: 1683472.prov, ls.png, screenshot-1.png,
> screenshot-2.png, screenshot-3.png
>
>
> We upgraded our nifi cluster from 1.11.3 to 1.12.0.
> The nodes come up and everything looks to be functional. I can see 1.12.0 is
> running.
> Later on, we discovered that the data provenance is missing. From checking
> our logs, we see tons of errors compressing the logs.
> {code}
> 2020-09-28 03:38:35,205 ERROR [Compress Provenance Logs-1-thread-1]
> o.a.n.p.s.EventFileCompressor Failed to compress
> ./provenance_repository/2752821.prov on rollover
> {code}
> This didn't happen in 1.11.3.
> Is this a known issue? We are considering reverting back if there is no
> solution for this since we can't go prod with no/broken data provenance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-7856) Provenance failed to be compressed after nifi upgrade to 1.12
[
https://issues.apache.org/jira/browse/NIFI-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mengze Li updated NIFI-7856:
Attachment: screenshot-3.png
> Provenance failed to be compressed after nifi upgrade to 1.12
> -
>
> Key: NIFI-7856
> URL: https://issues.apache.org/jira/browse/NIFI-7856
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Mengze Li
>Priority: Major
> Attachments: 1683472.prov, ls.png, screenshot-1.png,
> screenshot-2.png, screenshot-3.png
>
>
> We upgraded our nifi cluster from 1.11.3 to 1.12.0.
> The nodes come up and everything looks to be functional. I can see 1.12.0 is
> running.
> Later on, we discovered that the data provenance is missing. From checking
> our logs, we see tons of errors compressing the logs.
> {code}
> 2020-09-28 03:38:35,205 ERROR [Compress Provenance Logs-1-thread-1]
> o.a.n.p.s.EventFileCompressor Failed to compress
> ./provenance_repository/2752821.prov on rollover
> {code}
> This didn't happen in 1.11.3.
> Is this a known issue? We are considering reverting back if there is no
> solution for this since we can't go prod with no/broken data provenance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7856) Provenance failed to be compressed after nifi upgrade to 1.12
[
https://issues.apache.org/jira/browse/NIFI-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205501#comment-17205501
]
Mengze Li commented on NIFI-7856:
-
[~markap14] any chance that you have looked at this issue? Thanks
> Provenance failed to be compressed after nifi upgrade to 1.12
> -
>
> Key: NIFI-7856
> URL: https://issues.apache.org/jira/browse/NIFI-7856
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Mengze Li
>Priority: Major
> Attachments: 1683472.prov, ls.png, screenshot-1.png, screenshot-2.png
>
>
> We upgraded our nifi cluster from 1.11.3 to 1.12.0.
> The nodes come up and everything looks to be functional. I can see 1.12.0 is
> running.
> Later on, we discovered that the data provenance is missing. From checking
> our logs, we see tons of errors compressing the logs.
> {code}
> 2020-09-28 03:38:35,205 ERROR [Compress Provenance Logs-1-thread-1]
> o.a.n.p.s.EventFileCompressor Failed to compress
> ./provenance_repository/2752821.prov on rollover
> {code}
> This didn't happen in 1.11.3.
> Is this a known issue? We are considering reverting back if there is no
> solution for this since we can't go prod with no/broken data provenance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7856) Provenance failed to be compressed after nifi upgrade to 1.12
[
https://issues.apache.org/jira/browse/NIFI-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203949#comment-17203949
]
Mengze Li commented on NIFI-7856:
-
sure, this is our setting around provenance
{code}
# Provenance Repository Properties
nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository
nifi.provenance.repository.encryption.key.provider.implementation=
nifi.provenance.repository.encryption.key.provider.location=
nifi.provenance.repository.encryption.key.id=
nifi.provenance.repository.encryption.key=
# Persistent Provenance Repository Properties
nifi.provenance.repository.directory.default=./provenance_repository
nifi.provenance.repository.max.storage.time=30 days
nifi.provenance.repository.max.storage.size=10 GB
nifi.provenance.repository.rollover.time=10 mins
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.query.threads=2
nifi.provenance.repository.index.threads=2
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false
# Comma-separated list of fields. Fields that are not indexed will not be
searchable. Valid fields are:
# EventType, FlowFileUUID, Filename, TransitURI, ProcessorID,
AlternateIdentifierURI, Relationship, Details
nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename,
ProcessorID, Relationship
# FlowFile Attributes that should be indexed and made searchable. Some
examples to consider are filename, uuid, mime.type
nifi.provenance.repository.indexed.attributes=
# Large values for the shard size will result in more Java heap usage when
searching the Provenance Repository
# but should provide better performance
nifi.provenance.repository.index.shard.size=500 MB
# Indicates the maximum length that a FlowFile attribute can be when retrieving
a Provenance Event from
# the repository. If the length of any attribute exceeds this value, it will be
truncated when the event is retrieved.
nifi.provenance.repository.max.attribute.length=65536
nifi.provenance.repository.concurrent.merge.threads=2
# Volatile Provenance Respository Properties
nifi.provenance.repository.buffer.size=10
{code}
> Provenance failed to be compressed after nifi upgrade to 1.12
> -
>
> Key: NIFI-7856
> URL: https://issues.apache.org/jira/browse/NIFI-7856
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Mengze Li
>Priority: Major
> Attachments: 1683472.prov, ls.png, screenshot-1.png, screenshot-2.png
>
>
> We upgraded our nifi cluster from 1.11.3 to 1.12.0.
> The nodes come up and everything looks to be functional. I can see 1.12.0 is
> running.
> Later on, we discovered that the data provenance is missing. From checking
> our logs, we see tons of errors compressing the logs.
> {code}
> 2020-09-28 03:38:35,205 ERROR [Compress Provenance Logs-1-thread-1]
> o.a.n.p.s.EventFileCompressor Failed to compress
> ./provenance_repository/2752821.prov on rollover
> {code}
> This didn't happen in 1.11.3.
> Is this a known issue? We are considering reverting back if there is no
> solution for this since we can't go prod with no/broken data provenance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (NIFI-7856) Provenance failed to be compressed after nifi upgrade to 1.12
[
https://issues.apache.org/jira/browse/NIFI-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203925#comment-17203925
]
Mengze Li edited comment on NIFI-7856 at 9/29/20, 1:40 PM:
---
>From our logs, it happens every hour (seems that rollover MAX_TIME_REACHED is
>met, not sure the exact schedule is), see screenshot.
It happens consistently after the restart, the cluster has been running for 4+
days.
The issue for us is that the data provenance is missing for some processors
(never show up after the upgrade so latest record was 25th) and data provenance
is displaying either incomplete or delayed records.
This can be a huge issue for our prod troubleshooting if we move this to our
prod env.
Attached one prov file as well.
!screenshot-2.png!
was (Author: leeyoda):
>From our logs, it happens every hour (seems that rollover MAX_TIME_REACHED is
>set to be an hour), see screenshot.
It happens consistently after the restart, the cluster has been running for 4+
days.
The issue for us is that the data provenance is missing for some processors
(never show up after the upgrade so latest record was 25th) and data provenance
is displaying either incomplete or delayed records.
This can be a huge issue for our prod troubleshooting if we move this to our
prod env.
Attached one prov file as well.
!screenshot-2.png!
> Provenance failed to be compressed after nifi upgrade to 1.12
> -
>
> Key: NIFI-7856
> URL: https://issues.apache.org/jira/browse/NIFI-7856
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Mengze Li
>Priority: Major
> Attachments: 1683472.prov, ls.png, screenshot-1.png, screenshot-2.png
>
>
> We upgraded our nifi cluster from 1.11.3 to 1.12.0.
> The nodes come up and everything looks to be functional. I can see 1.12.0 is
> running.
> Later on, we discovered that the data provenance is missing. From checking
> our logs, we see tons of errors compressing the logs.
> {code}
> 2020-09-28 03:38:35,205 ERROR [Compress Provenance Logs-1-thread-1]
> o.a.n.p.s.EventFileCompressor Failed to compress
> ./provenance_repository/2752821.prov on rollover
> {code}
> This didn't happen in 1.11.3.
> Is this a known issue? We are considering reverting back if there is no
> solution for this since we can't go prod with no/broken data provenance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7856) Provenance failed to be compressed after nifi upgrade to 1.12
[
https://issues.apache.org/jira/browse/NIFI-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203925#comment-17203925
]
Mengze Li commented on NIFI-7856:
-
>From our logs, it happens every hour (seems that rollover MAX_TIME_REACHED is
>set to be an hour), see screenshot.
It happens consistently after the restart, the cluster has been running for 4+
days.
The issue for us is that the data provenance is missing for some processors
(never show up after the upgrade so latest record was 25th) and data provenance
is displaying either incomplete or delayed records.
This can be a huge issue for our prod troubleshooting if we move this to our
prod env.
Attached one prov file as well.
!screenshot-2.png!
> Provenance failed to be compressed after nifi upgrade to 1.12
> -
>
> Key: NIFI-7856
> URL: https://issues.apache.org/jira/browse/NIFI-7856
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Mengze Li
>Priority: Major
> Attachments: 1683472.prov, ls.png, screenshot-1.png, screenshot-2.png
>
>
> We upgraded our nifi cluster from 1.11.3 to 1.12.0.
> The nodes come up and everything looks to be functional. I can see 1.12.0 is
> running.
> Later on, we discovered that the data provenance is missing. From checking
> our logs, we see tons of errors compressing the logs.
> {code}
> 2020-09-28 03:38:35,205 ERROR [Compress Provenance Logs-1-thread-1]
> o.a.n.p.s.EventFileCompressor Failed to compress
> ./provenance_repository/2752821.prov on rollover
> {code}
> This didn't happen in 1.11.3.
> Is this a known issue? We are considering reverting back if there is no
> solution for this since we can't go prod with no/broken data provenance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-7856) Provenance failed to be compressed after nifi upgrade to 1.12
[
https://issues.apache.org/jira/browse/NIFI-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mengze Li updated NIFI-7856:
Attachment: 1683472.prov
> Provenance failed to be compressed after nifi upgrade to 1.12
> -
>
> Key: NIFI-7856
> URL: https://issues.apache.org/jira/browse/NIFI-7856
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Mengze Li
>Priority: Major
> Attachments: 1683472.prov, ls.png, screenshot-1.png, screenshot-2.png
>
>
> We upgraded our nifi cluster from 1.11.3 to 1.12.0.
> The nodes come up and everything looks to be functional. I can see 1.12.0 is
> running.
> Later on, we discovered that the data provenance is missing. From checking
> our logs, we see tons of errors compressing the logs.
> {code}
> 2020-09-28 03:38:35,205 ERROR [Compress Provenance Logs-1-thread-1]
> o.a.n.p.s.EventFileCompressor Failed to compress
> ./provenance_repository/2752821.prov on rollover
> {code}
> This didn't happen in 1.11.3.
> Is this a known issue? We are considering reverting back if there is no
> solution for this since we can't go prod with no/broken data provenance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-7856) Provenance failed to be compressed after nifi upgrade to 1.12
[
https://issues.apache.org/jira/browse/NIFI-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mengze Li updated NIFI-7856:
Attachment: screenshot-2.png
> Provenance failed to be compressed after nifi upgrade to 1.12
> -
>
> Key: NIFI-7856
> URL: https://issues.apache.org/jira/browse/NIFI-7856
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Mengze Li
>Priority: Major
> Attachments: ls.png, screenshot-1.png, screenshot-2.png
>
>
> We upgraded our nifi cluster from 1.11.3 to 1.12.0.
> The nodes come up and everything looks to be functional. I can see 1.12.0 is
> running.
> Later on, we discovered that the data provenance is missing. From checking
> our logs, we see tons of errors compressing the logs.
> {code}
> 2020-09-28 03:38:35,205 ERROR [Compress Provenance Logs-1-thread-1]
> o.a.n.p.s.EventFileCompressor Failed to compress
> ./provenance_repository/2752821.prov on rollover
> {code}
> This didn't happen in 1.11.3.
> Is this a known issue? We are considering reverting back if there is no
> solution for this since we can't go prod with no/broken data provenance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7856) Provenance failed to be compressed after nifi upgrade to 1.12
[
https://issues.apache.org/jira/browse/NIFI-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203535#comment-17203535
]
Mengze Li commented on NIFI-7856:
-
Here is the stack trace of one incident, hopefully it is helpful.
Also attached the ls results, it seems that these files are all compressed fine
but the logs seem to show that it doesn't exist.
A race condition?
{code}
2020-09-27 21:37:34,747 INFO [Clustering Tasks Thread-3]
o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2020-09-27
21:37:34,616 and sent to 10.51.8.18: at 2020-09-27 21:37:34,747; send took
131 millis
2020-09-27 21:37:39,660 INFO [pool-15-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile
Repository
2020-09-27 21:37:39,660 INFO [pool-15-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile
Repository with 15079 records in 0 milliseconds
2020-09-27 21:37:49,109 INFO [pool-61-thread-1]
c.a.s.k.clientlibrary.lib.worker.Worker Current stream shard assignments:
shardId-
2020-09-27 21:37:49,110 INFO [pool-61-thread-1]
c.a.s.k.clientlibrary.lib.worker.Worker Sleeping ...
2020-09-27 21:37:59,660 INFO [pool-15-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile
Repository
2020-09-27 21:37:59,660 INFO [pool-15-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile
Repository with 15079 records in 0 milliseconds
2020-09-27 21:38:02,196 INFO [pool-43-thread-1]
c.a.s.k.clientlibrary.lib.worker.Worker Current stream shard assignments:
shardId-0012
2020-09-27 21:38:02,196 INFO [pool-43-thread-1]
c.a.s.k.clientlibrary.lib.worker.Worker Sleeping ...
2020-09-27 21:38:19,660 INFO [pool-15-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile
Repository
2020-09-27 21:38:19,660 INFO [pool-15-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile
Repository with 15079 records in 0 milliseconds
2020-09-27 21:38:20,688 INFO [Timer-Driven Process Thread-6]
o.a.nifi.groups.StandardProcessGroup
StandardProcessGroup[identifier=9e102d08-0174-1000--db703545,name=ContactLookup]
is not the most recent version of the flow that is under Version Control;
current version is 3; most recent version is 7
2020-09-27 21:38:20,691 INFO [Timer-Driven Process Thread-6]
o.a.nifi.groups.StandardProcessGroup
StandardProcessGroup[identifier=4b226950-0174-1000--64a82b74,name=EcomdashOrderProcessingMain]
is not the most recent version of the flow that is under Version Control;
current version is 8; most recent version is 10
2020-09-27 21:38:20,694 INFO [Timer-Driven Process Thread-6]
o.a.nifi.groups.StandardProcessGroup
StandardProcessGroup[identifier=e366c899-0173-1000--26d80b41,name=ContactLookup]
is not the most recent version of the flow that is under Version Control;
current version is 5; most recent version is 7
2020-09-27 21:38:20,697 INFO [Timer-Driven Process Thread-6]
o.a.nifi.groups.StandardProcessGroup
StandardProcessGroup[identifier=a17c8629-0173-1000--055a79e8,name=HandleFailedMessages]
is not the most recent version of the flow that is under Version Control;
current version is 2; most recent version is 3
2020-09-27 21:38:34,799 INFO [Framework Task Thread Thread-3]
o.a.n.p.store.WriteAheadStorePartition Successfully rolled over Event Writer
for Provenance Event Store Partition[directory=./provenance_repository] due to
MAX_TIME_REACHED
2020-09-27 21:38:34,799 ERROR [Compress Provenance Logs-1-thread-2]
o.a.n.p.s.EventFileCompressor Failed to compress
./provenance_repository/1693519.prov on rollover
java.io.FileNotFoundException: ./provenance_repository/1693519.prov (No such
file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.(FileInputStream.java:138)
at
org.apache.nifi.provenance.serialization.EventFileCompressor.compress(EventFileCompressor.java:164)
at
org.apache.nifi.provenance.serialization.EventFileCompressor.run(EventFileCompressor.java:115)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2020-09-27 21:38:34,799 WARN [Compress Provenance Logs-1-thread-2]
o.a.n.p.s.EventFileCompressor Failed to delete
./provenance_repository/1693519.prov; this file should be cleaned up manually
2020-09-27 21:38:34,887 INFO [Clustering Tasks Thread-3]
o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2020-09-27
21:38:34,748 and sent to
[jira] [Updated] (NIFI-7856) Provenance failed to be compressed after nifi upgrade to 1.12
[
https://issues.apache.org/jira/browse/NIFI-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mengze Li updated NIFI-7856:
Attachment: ls.png
> Provenance failed to be compressed after nifi upgrade to 1.12
> -
>
> Key: NIFI-7856
> URL: https://issues.apache.org/jira/browse/NIFI-7856
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Mengze Li
>Priority: Major
> Attachments: ls.png, screenshot-1.png
>
>
> We upgraded our nifi cluster from 1.11.3 to 1.12.0.
> The nodes come up and everything looks to be functional. I can see 1.12.0 is
> running.
> Later on, we discovered that the data provenance is missing. From checking
> our logs, we see tons of errors compressing the logs.
> {code}
> 2020-09-28 03:38:35,205 ERROR [Compress Provenance Logs-1-thread-1]
> o.a.n.p.s.EventFileCompressor Failed to compress
> ./provenance_repository/2752821.prov on rollover
> {code}
> This didn't happen in 1.11.3.
> Is this a known issue? We are considering reverting back if there is no
> solution for this since we can't go prod with no/broken data provenance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-7856) Provenance failed to be compressed after nifi upgrade to 1.12
[
https://issues.apache.org/jira/browse/NIFI-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mengze Li updated NIFI-7856:
Attachment: screenshot-1.png
> Provenance failed to be compressed after nifi upgrade to 1.12
> -
>
> Key: NIFI-7856
> URL: https://issues.apache.org/jira/browse/NIFI-7856
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.12.1
>Reporter: Mengze Li
>Priority: Major
> Attachments: screenshot-1.png
>
>
> We upgraded our nifi cluster from 1.11.3 to 1.12.0.
> The nodes come up and everything looks to be functional. I can see 1.12.0 is
> running.
> Later on, we discovered that the data provenance is missing. From checking
> our logs, we see tons of errors compressing the logs.
> {code}
> 2020-09-28 03:38:35,205 ERROR [Compress Provenance Logs-1-thread-1]
> o.a.n.p.s.EventFileCompressor Failed to compress
> ./provenance_repository/2752821.prov on rollover
> {code}
> This didn't happen in 1.11.3.
> Is this a known issue? We are considering reverting back if there is no
> solution for this since we can't go prod with no/broken data provenance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7856) Provenance failed to be compressed after nifi upgrade to 1.12
[
https://issues.apache.org/jira/browse/NIFI-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203488#comment-17203488
]
Mengze Li commented on NIFI-7856:
-
Do we have any pointers how to address/debug this? appreciate it.
> Provenance failed to be compressed after nifi upgrade to 1.12
> -
>
> Key: NIFI-7856
> URL: https://issues.apache.org/jira/browse/NIFI-7856
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.12.1
>Reporter: Mengze Li
>Priority: Major
> Attachments: screenshot-1.png
>
>
> We upgraded our nifi cluster from 1.11.3 to 1.12.0.
> The nodes come up and everything looks to be functional. I can see 1.12.0 is
> running.
> Later on, we discovered that the data provenance is missing. From checking
> our logs, we see tons of errors compressing the logs.
> {code}
> 2020-09-28 03:38:35,205 ERROR [Compress Provenance Logs-1-thread-1]
> o.a.n.p.s.EventFileCompressor Failed to compress
> ./provenance_repository/2752821.prov on rollover
> {code}
> This didn't happen in 1.11.3.
> Is this a known issue? We are considering reverting back if there is no
> solution for this since we can't go prod with no/broken data provenance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (NIFI-7856) Provenance failed to be compressed after nifi upgrade to 1.12
Mengze Li created NIFI-7856:
---
Summary: Provenance failed to be compressed after nifi upgrade to
1.12
Key: NIFI-7856
URL: https://issues.apache.org/jira/browse/NIFI-7856
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 1.12.1
Reporter: Mengze Li
We upgraded our nifi cluster from 1.11.3 to 1.12.0.
The nodes come up and everything looks to be functional. I can see 1.12.0 is
running.
Later on, we discovered that the data provenance is missing. From checking our
logs, we see tons of errors compressing the logs.
{code}
2020-09-28 03:38:35,205 ERROR [Compress Provenance Logs-1-thread-1]
o.a.n.p.s.EventFileCompressor Failed to compress
./provenance_repository/2752821.prov on rollover
{code}
This didn't happen in 1.11.3.
Is this a known issue? We are considering reverting back if there is no
solution for this since we can't go prod with no/broken data provenance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
