[
https://issues.apache.org/jira/browse/NIFI-16098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18098488#comment-18098488
]
ASF subversion and git services commented on NIFI-16098:
--------------------------------------------------------
Commit a62c856388f32984a4a7795324d81bd0861f7276 in nifi's branch
refs/heads/main from Pierre Villard
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=a62c856388f ]
NIFI-16098 Prevent content-claim truncation from dropping content referenced by
a live sibling FlowFile (#11414)
* NIFI-16098 Prevent content-claim truncation from dropping content referenced
by a live sibling FlowFile
A FlowFile created and removed within a single session (e.g. ExecuteGroovyScript
with Failure Strategy 'transfer to failure', which clones on get and removes the
clone at commit) yields one repository record whose type transitions
CREATE->DELETE.
The truncation reference count is incremented only for a committed CREATE, so
that
increment never ran, but the DELETE still decremented it. For a content claim
shared
with a live sibling FlowFile this drove the truncation reference count to zero,
allowing the claim to be truncated and the sibling's content to be lost.
Only decrement the truncation reference count on delete of a persisted FlowFile
(one that has an original claim).
* NIFI-16098 Add system test reproducing content-claim truncation loss with an
in-session cloned FlowFile
Adds a CloneAndTerminate test processor that, for each incoming FlowFile,
clones it and removes both the
clone and the input within the same session (the record set produced by
ExecuteGroovyScript with Failure
Strategy 'transfer to failure'). A new ContentClaimTruncationIT case forks a
generator to this processor
and to a leg that holds the FlowFiles, and verifies the shared content claims
are not truncated while the
holding leg still references them. The test fails without the accompanying
framework fix (content is
truncated away) and passes with it.
> Content can be lost due to content claim truncation
> ---------------------------------------------------
>
> Key: NIFI-16098
> URL: https://issues.apache.org/jira/browse/NIFI-16098
> Project: Apache NiFi
> Issue Type: Bug
> Affects Versions: 2.10.0
> Reporter: Paul Kelly
> Assignee: Pierre Villard
> Priority: Major
> Attachments: Content_Loss_Sample-1.json, flowscreenshot-1.png
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> We recently upgraded some of our servers to NiFi 2.10.0 and started seeing
> MissingFlowFileException and ContentNotFoundException occasionally in some of
> our flows. Reviewing the app logs, it appears to be related to NIFI-15570
> because we see the content claim being truncated while the FlowFile is still
> active in a different part of the flow.
> Attached is a sample flow definition that reproduces the problem. Generating
> two FlowFiles of different sizes (one under the
> {{nifi.content.claim.max.appendable.size}} and one big enough to trigger
> claim truncation), forking a copy of the FlowFiles both to a leg containing
> an ExecuteGroovyScript that simply moves the FlowFile to the REL_SUCCESS
> relationship which is set to terminate and to a leg that delays the FlowFile
> for more than one minute causes the content in the live leg to get truncated
> and lost.
> !flowscreenshot.png!
> This test was conducted on a brand new deployment of NiFi 2.10.0, where the
> only change made to nifi.properties was to disable the content repository
> archive ({{{}nifi.content.repository.archive.enabled=false{}}}) in order to
> trigger truncation to occur earlier.
> App logs show that the content is truncated and could no longer be found.
>
> {code:java}
> 2026-07-10 00:32:35,854 INFO [NiFi Web Server-114]
> o.a.n.c.s.StandardProcessScheduler Starting
> RetryFlowFile[id=a2374f4c-f914-320a-ee04-3f3287d7d3e7]
> 2026-07-10 00:32:35,855 INFO [NiFi Web Server-114]
> o.a.n.controller.StandardProcessorNode Desired State for
> RetryFlowFile[id=a2374f4c-f914-320a-ee04-3f3287d7d3e7] now set to RUNNING
> 2026-07-10 00:32:35,856 INFO [NiFi Web Server-114]
> o.a.n.c.s.StandardProcessScheduler Starting
> GenerateFlowFile[id=41672154-bb8f-3b17-8ecb-023a557f7b68]
> 2026-07-10 00:32:35,856 INFO [NiFi Web Server-114]
> o.a.n.controller.StandardProcessorNode Desired State for
> GenerateFlowFile[id=41672154-bb8f-3b17-8ecb-023a557f7b68] now set to RUNNING
> 2026-07-10 00:32:35,856 INFO [NiFi Web Server-114]
> o.a.n.c.s.StandardProcessScheduler Starting
> CryptographicHashContent[id=8af51d88-adee-3b8f-851e-b72ab8d752fa]
> 2026-07-10 00:32:35,857 INFO [NiFi Web Server-114]
> o.a.n.controller.StandardProcessorNode Desired State for
> CryptographicHashContent[id=8af51d88-adee-3b8f-851e-b72ab8d752fa] now set to
> RUNNING
> 2026-07-10 00:32:35,857 INFO [NiFi Web Server-114]
> o.a.n.c.s.StandardProcessScheduler Starting
> ExecuteGroovyScript[id=26dab1f8-6aad-3b6a-2d54-741aaecd26f0]
> 2026-07-10 00:32:35,857 INFO [NiFi Web Server-114]
> o.a.n.controller.StandardProcessorNode Desired State for
> ExecuteGroovyScript[id=26dab1f8-6aad-3b6a-2d54-741aaecd26f0] now set to
> RUNNING
> 2026-07-10 00:32:35,857 INFO [NiFi Web Server-114]
> o.a.n.c.s.StandardProcessScheduler Starting
> GenerateFlowFile[id=3f3a0a68-e377-32c0-31b6-4f914025e627]
> 2026-07-10 00:32:35,857 INFO [Timer-Driven Process Thread-4]
> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
> RetryFlowFile[id=a2374f4c-f914-320a-ee04-3f3287d7d3e7] to run with 1 threads
> 2026-07-10 00:32:35,857 INFO [Timer-Driven Process Thread-8]
> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
> CryptographicHashContent[id=8af51d88-adee-3b8f-851e-b72ab8d752fa] to run with
> 1 threads
> 2026-07-10 00:32:35,857 INFO [NiFi Web Server-114]
> o.a.n.controller.StandardProcessorNode Desired State for
> GenerateFlowFile[id=3f3a0a68-e377-32c0-31b6-4f914025e627] now set to RUNNING
> 2026-07-10 00:32:35,858 INFO [Timer-Driven Process Thread-10]
> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
> ExecuteGroovyScript[id=26dab1f8-6aad-3b6a-2d54-741aaecd26f0] to run with 1
> threads
> 2026-07-10 00:32:35,862 INFO [Timer-Driven Process Thread-5]
> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
> GenerateFlowFile[id=3f3a0a68-e377-32c0-31b6-4f914025e627] to run with 1
> threads
> 2026-07-10 00:32:35,904 INFO [Timer-Driven Process Thread-3]
> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
> GenerateFlowFile[id=41672154-bb8f-3b17-8ecb-023a557f7b68] to run with 1
> threads
> 2026-07-10 00:32:36,298 INFO [Flow Service Tasks Thread-1]
> o.a.nifi.controller.StandardFlowService Saved flow controller
> org.apache.nifi.controller.FlowController@74bb4bb5 // Another save pending =
> false
> 2026-07-10 00:32:47,386 INFO [Checkpoint FlowFile Repository]
> o.a.n.wali.SequentialAccessWriteAheadLog Checkpointed Write-Ahead Log with 46
> Records and 0 Swap Files in 28 milliseconds (Stop-the-world time = 19
> milliseconds), max Transaction ID 184
> 2026-07-10 00:32:47,386 INFO [Checkpoint FlowFile Repository]
> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile
> Repository with 46 records in 28 milliseconds
> 2026-07-10 00:33:07,417 INFO [Checkpoint FlowFile Repository]
> o.a.n.wali.SequentialAccessWriteAheadLog Checkpointed Write-Ahead Log with
> 126 Records and 0 Swap Files in 30 milliseconds (Stop-the-world time = 22
> milliseconds), max Transaction ID 504
> 2026-07-10 00:33:07,417 INFO [Checkpoint FlowFile Repository]
> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile
> Repository with 126 records in 30 milliseconds
> 2026-07-10 00:33:23,751 INFO [Write-Ahead Local State Provider Maintenance]
> o.a.n.wali.SequentialAccessWriteAheadLog Checkpointed Write-Ahead Log with 2
> Records and 0 Swap Files in 17 milliseconds (Stop-the-world time = 13
> milliseconds), max Transaction ID 4
> 2026-07-10 00:33:25,165 INFO [FileSystemRepository Workers Thread-4]
> o.a.n.c.repository.FileSystemRepository Truncating
> StandardResourceClaim[id=1783657966517-22, container=default, section=22] to
> 20480 bytes because the FlowFile occupying the last 3145728 bytes has been
> removed
> ...
> 2026-07-10 00:44:01,738 WARN [Timer-Driven Process Thread-7]
> o.a.n.c.r.StandardProcessSession Unable to find content for
> StandardFlowFileRecord[uuid=88662526-ef54-47d4-9e91-d24c49a47fc7,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1783657966517-22, container=default,
> section=22], offset=20480,
> length=3145728],offset=0,name=88662526-ef54-47d4-9e91-d24c49a47fc7,size=3145728];
> dropping FlowFile
> org.apache.nifi.controller.repository.ContentNotFoundException: Could not
> find content for StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1783657966517-22, container=default,
> section=22], offset=20480, length=3145728]: Stream contained only 0 bytes but
> should have contained 3145728
> at
> org.apache.nifi.controller.repository.io.FlowFileAccessInputStream.ensureAllContentRead(FlowFileAccessInputStream.java:49)
> at
> org.apache.nifi.controller.repository.io.FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:84)
> at
> org.apache.nifi.controller.repository.io.FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:99)
> at
> org.apache.nifi.controller.repository.io.TaskTerminationInputStream.read(TaskTerminationInputStream.java:62)
> at
> org.apache.nifi.processors.standard.hash.HashService.traditionalHashStreaming(HashService.java:201)
> at
> org.apache.nifi.processors.standard.hash.HashService.hashValueStreaming(HashService.java:113)
> at
> org.apache.nifi.processors.standard.CryptographicHashContent.lambda$onTrigger$0(CryptographicHashContent.java:132)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2738)
> at
> org.apache.nifi.controller.repository.WeakHashMapProcessSessionFactory$FactoryRetainingProcessSession.read(WeakHashMapProcessSessionFactory.java:260)
> at
> org.apache.nifi.processors.standard.CryptographicHashContent.onTrigger(CryptographicHashContent.java:132)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1292)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:229)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102)
> at
> org.apache.nifi.engine.FlowEngine.lambda$wrap$1(FlowEngine.java:105)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
> at
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> 2026-07-10 00:44:01,739 ERROR [Timer-Driven Process Thread-7]
> o.a.n.p.s.CryptographicHashContent
> CryptographicHashContent[id=8af51d88-adee-3b8f-851e-b72ab8d752fa] Processing
> halted: yielding [1 sec]
> org.apache.nifi.processor.exception.MissingFlowFileException: Unable to find
> content for
> StandardFlowFileRecord[uuid=88662526-ef54-47d4-9e91-d24c49a47fc7,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1783657966517-22, container=default,
> section=22], offset=20480,
> length=3145728],offset=0,name=88662526-ef54-47d4-9e91-d24c49a47fc7,size=3145728];
> dropping FlowFile
> at
> org.apache.nifi.controller.repository.StandardProcessSession.handleContentNotFound(StandardProcessSession.java:3756)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2753)
> at
> org.apache.nifi.controller.repository.WeakHashMapProcessSessionFactory$FactoryRetainingProcessSession.read(WeakHashMapProcessSessionFactory.java:260)
> at
> org.apache.nifi.processors.standard.CryptographicHashContent.onTrigger(CryptographicHashContent.java:132)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1292)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:229)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102)
> at
> org.apache.nifi.engine.FlowEngine.lambda$wrap$1(FlowEngine.java:105)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
> at
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> Caused by: org.apache.nifi.controller.repository.ContentNotFoundException:
> Could not find content for StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1783657966517-22, container=default,
> section=22], offset=20480, length=3145728]: Stream contained only 0 bytes but
> should have contained 3145728
> at
> org.apache.nifi.controller.repository.io.FlowFileAccessInputStream.ensureAllContentRead(FlowFileAccessInputStream.java:49)
> at
> org.apache.nifi.controller.repository.io.FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:84)
> at
> org.apache.nifi.controller.repository.io.FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:99)
> at
> org.apache.nifi.controller.repository.io.TaskTerminationInputStream.read(TaskTerminationInputStream.java:62)
> at
> org.apache.nifi.processors.standard.hash.HashService.traditionalHashStreaming(HashService.java:201)
> at
> org.apache.nifi.processors.standard.hash.HashService.hashValueStreaming(HashService.java:113)
> at
> org.apache.nifi.processors.standard.CryptographicHashContent.lambda$onTrigger$0(CryptographicHashContent.java:132)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2738)
> ... 13 common frames omitted
> 2026-07-10 00:44:01,739 WARN [Timer-Driven Process Thread-7]
> o.a.n.controller.tasks.ConnectableTask Processing halted: uncaught exception
> in Component
> [CryptographicHashContent[id=8af51d88-adee-3b8f-851e-b72ab8d752fa]]
> org.apache.nifi.processor.exception.MissingFlowFileException: Unable to find
> content for
> StandardFlowFileRecord[uuid=88662526-ef54-47d4-9e91-d24c49a47fc7,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1783657966517-22, container=default,
> section=22], offset=20480,
> length=3145728],offset=0,name=88662526-ef54-47d4-9e91-d24c49a47fc7,size=3145728];
> dropping FlowFile{code}
>
> Interestingly if you replace the nearly empty ExecuteGroovyScript processor
> with an empty UpdateAttribute processor that terminates the success
> relationship, the problem does not occur. I have not done much testing to
> see which processors can cause it, or examined what might be different about
> these two processors that could cause the problem.
> Disabling content claim truncation by setting
> {{nifi.content.claim.truncation.enabled=false}} in nifi.properties prevents
> this problem from occurring.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
