Joseph Gresock created NIFI-3582:
------------------------------------
Summary: DistributedMapCacheServer does not evict persistent
records
Key: NIFI-3582
URL: https://issues.apache.org/jira/browse/NIFI-3582
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework
Affects Versions: 1.1.1
Reporter: Joseph Gresock
Assignee: Mark Payne
The PersistentMapCache does not appear to be evicting records when used by the
DistributedMapCacheServer, at least in the FIFO eviction mode.
To replicate this behavior in NiFi 1.1.1, add the following to your flow:
*Services*
*DistributedMapCacheServer, port 4557, maximum cache entries = 10,000, FIFO
eviction, persistence directory specified
*DistributedMapCacheClientService, point to the same host and port
*Flow*
GenerateFlowFile (randomize 1K binary files in batches of 10, schedule 10
threads) -> HashContent (md5) into hash.value -> DetectDuplicate with
identifier = $\{hash.value\}, description = ., no age off, select your cache
client, cache identifier = true
This should cause the snapshot file in your cache's persistence directory to
exceed 100,000 keys pretty quickly, and as far as I can tell, it never goes
back down.
On our production system, we have the cache server configured for 100,000 max
entries with FIFO eviction, and we recently saw this log statement showing the
writeahead log with over 4 million entries:
{code}
nifi-app.log:2017-03-09 15:03:00,670 INFO [Distributed Cache Server
Communications Thread: ac907dec-49a4-439e-99f5-1558f2358d87]
org.wali.MinimalLockingWriteAheadLog
org.wali.MinimalLockingWriteAheadLog@40569408
checkpointed with 4262902 Records and 0 Swap Files in 256302 milliseconds
(Stop-the-world time = 1378 milliseconds, Clear Edit Logs time = 19 millis),
max Transaction ID 4263237
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
