[
https://issues.apache.org/jira/browse/NIFI-9012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17393993#comment-17393993
]
Michal Šunka edited comment on NIFI-9012 at 8/5/21, 3:15 PM:
-------------------------------------------------------------
I found those somewhat close-sounding lines:
{code:java}
2021-08-05 09:54:08,150 INFO [pool-10-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile
Repository
2021-08-05 09:54:08,151 INFO [pool-10-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile
Repository with 46 records in 0 milliseconds
2021-08-05 09:54:11,271 INFO [Cleanup Archive for default]
o.a.n.c.repository.FileSystemRepository Successfully deleted 0 files (0 bytes)
from archive
2021-08-05 09:54:11,271 INFO [Timer-Driven Process Thread-1]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,368 INFO [Timer-Driven Process Thread-2]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,374 INFO [Timer-Driven Process Thread-6]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,375 INFO [Timer-Driven Process Thread-4]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,375 INFO [Timer-Driven Process Thread-9 <Terminated Task>]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,375 INFO [Timer-Driven Process Thread-3 <Terminated Task>]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,375 INFO [Timer-Driven Process Thread-7 <Terminated Task>]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,375 INFO [Timer-Driven Process Thread-8 <Terminated Task>]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,375 INFO [Timer-Driven Process Thread-11 <Terminated Task>]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,375 INFO [Timer-Driven Process Thread-13 <Terminated Task>]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,375 INFO [Timer-Driven Process Thread-17]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,375 INFO [Timer-Driven Process Thread-12 <Terminated Task>]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,375 INFO [Timer-Driven Process Thread-5 <Terminated Task>]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,375 INFO [Timer-Driven Process Thread-18 <Terminated Task>]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,375 INFO [Timer-Driven Process Thread-15 <Terminated Task>]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,376 INFO [Timer-Driven Process Thread-19]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,376 INFO [Timer-Driven Process Thread-16 <Terminated Task>]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,376 INFO [Timer-Driven Process Thread-20 <Terminated Task>]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,376 INFO [Timer-Driven Process Thread-14 <Terminated Task>]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,376 INFO [Timer-Driven Process Thread-10 <Terminated Task>]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,376 INFO [Timer-Driven Process Thread-23]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,376 INFO [Timer-Driven Process Thread-24]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,376 INFO [Timer-Driven Process Thread-21]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,376 INFO [Timer-Driven Process Thread-22]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:19,183 INFO [Framework Task Thread Thread-4]
o.a.n.p.store.WriteAheadStorePartition Successfully rolled over Event Writer
for Provenance Event Store Partition[directory=./provenance_repository] due to
MAX_TIME_REACHED. Event File was 63.73 KB and contained 52 events.
2021-08-05 09:54:23,393 INFO [pool-2-thread-2]
org.apache.nifi.BootstrapListener Received SHUTDOWN request from Bootstrap
2021-08-05 09:54:23,394 INFO [pool-2-thread-2] org.apache.nifi.NiFi Initiating
shutdown of Jetty web server...{code}
Right after this is a restart of container with NiFi. In startup is those lines
about size (poped up after searching for "FileSystemRepository"):
{code:java}
2021-08-05 09:55:26,413 INFO [main] o.a.n.p.store.WriteAheadStorePartition
After recovering ./provenance_repository, next Event ID to be generated will be
113923
2021-08-05 09:55:26,751 INFO [main] o.a.n.p.index.lucene.LuceneEventIndex Will
avoid re-indexing Provenance Events because the newest index is defunct, so it
will be re-indexed in the background
2021-08-05 09:55:26,809 INFO [pool-17-thread-1]
o.a.n.p.index.lucene.LuceneEventIndex Determined that Max Event ID indexed for
Partition default is approximately 113922 based on index
./provenance_repository/lucene-8-index-1628081907193
2021-08-05 09:55:26,813 INFO [pool-17-thread-1]
o.a.n.p.store.WriteAheadStorePartition The last Provenance Event indexed for
partition default is 103922, but the last event written to partition has ID
113922. Re-indexing up to the last 10000 events for ./provenance_repository to
ensure that the Event Index is accurate and up-to-date
2021-08-05 09:55:27,875 INFO [pool-17-thread-1]
o.a.n.p.store.WriteAheadStorePartition Finished re-indexing 10001 events across
16 files for ./provenance_repository in 1.61 seconds
2021-08-05 09:55:27,890 INFO [main] o.a.n.c.repository.FileSystemRepository
Maximum Threshold for Container default set to 53786475724 bytes; if volume
exceeds this size, archived data will be deleted until it no longer exceeds
this size
2021-08-05 09:55:27,891 INFO [main] o.a.n.c.repository.FileSystemRepository
Initializing FileSystemRepository with 'Always Sync' set to false
2021-08-05 09:55:28,648 INFO [main] org.wali.MinimalLockingWriteAheadLog
org.wali.MinimalLockingWriteAheadLog@3b64f131 finished recovering records.
Performing Checkpoint to ensure proper state of Partitions before updates
2021-08-05 09:55:28,648 INFO [main] org.wali.MinimalLockingWriteAheadLog
Successfully recovered 17 records in 8 milliseconds
2021-08-05 09:55:28,678 INFO [main] org.wali.MinimalLockingWriteAheadLog
org.wali.MinimalLockingWriteAheadLog@3b64f131 checkpointed with 17 Records and
0 Swap Files in 29 milliseconds (Stop-the-world time = 2 milliseconds, Clear
Edit Logs time = 8 millis), max Transaction ID 16923
2021-08-05 09:55:28,844 INFO [main] o.a.n.w.c.ApplicationStartupContextListener
Starting Flow Controller...{code}
Anything more I can provide you with?
was (Author: raocz):
I found those somewhat close-sounding lines:
{code:java}
2021-08-05 09:54:08,150 INFO [pool-10-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile
Repository
2021-08-05 09:54:08,151 INFO [pool-10-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile
Repository with 46 records in 0 milliseconds
2021-08-05 09:54:11,271 INFO [Cleanup Archive for default]
o.a.n.c.repository.FileSystemRepository Successfully deleted 0 files (0 bytes)
from archive
2021-08-05 09:54:11,271 INFO [Timer-Driven Process Thread-1]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,368 INFO [Timer-Driven Process Thread-2]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,374 INFO [Timer-Driven Process Thread-6]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,375 INFO [Timer-Driven Process Thread-4]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,375 INFO [Timer-Driven Process Thread-9 <Terminated Task>]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,375 INFO [Timer-Driven Process Thread-3 <Terminated Task>]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,375 INFO [Timer-Driven Process Thread-7 <Terminated Task>]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,375 INFO [Timer-Driven Process Thread-8 <Terminated Task>]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,375 INFO [Timer-Driven Process Thread-11 <Terminated Task>]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,375 INFO [Timer-Driven Process Thread-13 <Terminated Task>]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,375 INFO [Timer-Driven Process Thread-17]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,375 INFO [Timer-Driven Process Thread-12 <Terminated Task>]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,375 INFO [Timer-Driven Process Thread-5 <Terminated Task>]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,375 INFO [Timer-Driven Process Thread-18 <Terminated Task>]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,375 INFO [Timer-Driven Process Thread-15 <Terminated Task>]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,376 INFO [Timer-Driven Process Thread-19]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,376 INFO [Timer-Driven Process Thread-16 <Terminated Task>]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,376 INFO [Timer-Driven Process Thread-20 <Terminated Task>]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,376 INFO [Timer-Driven Process Thread-14 <Terminated Task>]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,376 INFO [Timer-Driven Process Thread-10 <Terminated Task>]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,376 INFO [Timer-Driven Process Thread-23]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,376 INFO [Timer-Driven Process Thread-24]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,376 INFO [Timer-Driven Process Thread-21]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:11,376 INFO [Timer-Driven Process Thread-22]
o.a.n.c.repository.FileSystemRepository Unable to write to container default
due to archive file size constraints; waiting for archive cleanup
2021-08-05 09:54:19,183 INFO [Framework Task Thread Thread-4]
o.a.n.p.store.WriteAheadStorePartition Successfully rolled over Event Writer
for Provenance Event Store Partition[directory=./provenance_repository] due to
MAX_TIME_REACHED. Event File was 63.73 KB and contained 52 events.
2021-08-05 09:54:23,393 INFO [pool-2-thread-2]
org.apache.nifi.BootstrapListener Received SHUTDOWN request from Bootstrap
2021-08-05 09:54:23,394 INFO [pool-2-thread-2] org.apache.nifi.NiFi Initiating
shutdown of Jetty web server...{code}
Right after this is a restart of container with NiFi. In startup is thise lines
about size (poped up after searching for "FileSystemRepository"):
{code:java}
2021-08-05 09:55:26,413 INFO [main] o.a.n.p.store.WriteAheadStorePartition
After recovering ./provenance_repository, next Event ID to be generated will be
113923
2021-08-05 09:55:26,751 INFO [main] o.a.n.p.index.lucene.LuceneEventIndex Will
avoid re-indexing Provenance Events because the newest index is defunct, so it
will be re-indexed in the background
2021-08-05 09:55:26,809 INFO [pool-17-thread-1]
o.a.n.p.index.lucene.LuceneEventIndex Determined that Max Event ID indexed for
Partition default is approximately 113922 based on index
./provenance_repository/lucene-8-index-1628081907193
2021-08-05 09:55:26,813 INFO [pool-17-thread-1]
o.a.n.p.store.WriteAheadStorePartition The last Provenance Event indexed for
partition default is 103922, but the last event written to partition has ID
113922. Re-indexing up to the last 10000 events for ./provenance_repository to
ensure that the Event Index is accurate and up-to-date
2021-08-05 09:55:27,875 INFO [pool-17-thread-1]
o.a.n.p.store.WriteAheadStorePartition Finished re-indexing 10001 events across
16 files for ./provenance_repository in 1.61 seconds
2021-08-05 09:55:27,890 INFO [main] o.a.n.c.repository.FileSystemRepository
Maximum Threshold for Container default set to 53786475724 bytes; if volume
exceeds this size, archived data will be deleted until it no longer exceeds
this size
2021-08-05 09:55:27,891 INFO [main] o.a.n.c.repository.FileSystemRepository
Initializing FileSystemRepository with 'Always Sync' set to false
2021-08-05 09:55:28,648 INFO [main] org.wali.MinimalLockingWriteAheadLog
org.wali.MinimalLockingWriteAheadLog@3b64f131 finished recovering records.
Performing Checkpoint to ensure proper state of Partitions before updates
2021-08-05 09:55:28,648 INFO [main] org.wali.MinimalLockingWriteAheadLog
Successfully recovered 17 records in 8 milliseconds
2021-08-05 09:55:28,678 INFO [main] org.wali.MinimalLockingWriteAheadLog
org.wali.MinimalLockingWriteAheadLog@3b64f131 checkpointed with 17 Records and
0 Swap Files in 29 milliseconds (Stop-the-world time = 2 milliseconds, Clear
Edit Logs time = 8 millis), max Transaction ID 16923
2021-08-05 09:55:28,844 INFO [main] o.a.n.w.c.ApplicationStartupContextListener
Starting Flow Controller...{code}
Anything more I can provide you with?
> InvokeHttps
> -----------
>
> Key: NIFI-9012
> URL: https://issues.apache.org/jira/browse/NIFI-9012
> Project: Apache NiFi
> Issue Type: Bug
> Affects Versions: 1.14.0
> Environment: Deckerized NiFi with container from Docker Hub, no
> custom processors
> Reporter: Michal Šunka
> Priority: Major
> Attachments: docker_inspect_excerpt.txt, nifi.properties,
> thread-dump.txt, thread-dump2.txt
>
>
> I was running a NiFi version 1.12.1 without any problems with this regard.
> Yesterday I updated to version 1.14.0 (really wanted to get fix
> https://issues.apache.org/jira/browse/NIFI-7849). But now I am in a way worse
> state.
> I use InvokeHTTP processor to self-connect to NiFi API to get count of
> flowfiles being processed by given processor group.
> However this InvokeHTTP started getting stuck. Processor would just spin up a
> new thread and this will sit there indefinitely (six hours at-least),
> blocking incoming flowfile. After forceful termination of processor (simple
> stop is not enough) the flowfile would get released and next start will
> consume the flowfile just fine. (Timeouts are: connection 5 seconds, read 10
> seconds and idle 1 minute.) And of course log is clear of any errors which
> seems to be connected to this...
> I had noticed similar issue also with other processors, namely MergeContent
> processor or ExecuteSQLProcessor - e.g. flowfile gets stuck for 15 minutes
> (merge bin max age is 5 seconds). After forced termination all starts working
> (for some time - a few dozens of minutes).
> Oh and in all cases the thread hangs in terminating state (red thread icon
> with a digit in parenthesis; the digit sits there all the time and wont
> disappear). Only restart of whole NiFi clears the issue. For some time.
>
> Maybe some disk I/O issue?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
