[
https://issues.apache.org/jira/browse/NIFI-14724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18013445#comment-18013445
]
David Handermann commented on NIFI-14724:
-----------------------------------------
You're welcome [~deragon].
Adjusting the error message to indicate Maximum Read Size instead of Maximum
Length sounds like a good idea, and should be straightforward based on the
current implementation.
The default 1 MB limit should cover a number of standard cases, as a cache
service is not intended to store large values.
> "java.lang.IllegalArgumentException: Maximum Length [1048576] exceeded" in
> CacheRequestDecoder
> ----------------------------------------------------------------------------------------------
>
> Key: NIFI-14724
> URL: https://issues.apache.org/jira/browse/NIFI-14724
> Project: Apache NiFi
> Issue Type: Bug
> Affects Versions: 2.0.0-M4
> Reporter: Hans Deragon
> Priority: Blocker
>
> In production (cannot reproduce the problem in dev yet), get the error below.
> All of this happens within a single NiFi node. The
> DistributedMapCacheClientService (calling localhost) reside on the same node
> as the DistributedMapCacheServer. Both are running from the same JVM running
> the common NiFi application.
> The question is, how is this possible?
> The consequence of this bug is that whenever the instance is restarted, the
> *tracking of
> [ListFile|https://nifi.apache.org/components/org.apache.nifi.processors.standard.ListFile/]
> is lost* and it reloads all the files, causing massive duplicates being
> generated.
> {code:java}
> 2025-07-07 08:01:00,717 WARN
> [StandardMapCacheServer[92bca411-c87f-3b3e-b168-21d22cfbf4f2]-6-4]
> o.a.n.d.c.s.m.DistributedMapCacheServer
> DistributedMapCacheServer[id=92bca411-c87f-3b3e-b168-21d22cfbf4f2] Request
> Decoding Failed: Closing Connection [/127.0.0.1:33614]
> io.netty.handler.codec.DecoderException: java.lang.IllegalArgumentException:
> Maximum Length [1048576] exceeded [1396023]
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:500)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1407)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918)
> at
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
> at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.IllegalArgumentException: Maximum Length [1048576]
> exceeded [1396023]
> at
> org.apache.nifi.distributed.cache.server.codec.CacheRequestDecoder.readInt(CacheRequestDecoder.java:206)
> at
> org.apache.nifi.distributed.cache.server.codec.CacheRequestDecoder.readBytes(CacheRequestDecoder.java:151)
> at
> org.apache.nifi.distributed.cache.server.codec.MapCacheRequestDecoder.readKeyValueRequest(MapCacheRequestDecoder.java:87)
> at
> org.apache.nifi.distributed.cache.server.codec.MapCacheRequestDecoder.readRequest(MapCacheRequestDecoder.java:67)
> at
> org.apache.nifi.distributed.cache.server.codec.CacheRequestDecoder.decode(CacheRequestDecoder.java:99)
> at
> io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
> ... 17 common frames omitted
> 2025-07-07 08:01:00,718 ERROR [Timer-Driven Process Thread-10]
> o.a.n.c.r.StandardProcessSession Successfully committed session
> StandardProcessSession[id=2820] for
> ListFile[id=f10e2306-5b2d-3d27-49ad-79959ea84d3a] but failed to trigger
> success callback
> org.apache.nifi.processor.exception.ProcessException: Failed to persist
> already-listed entities due to java.io.IOException: Request invocation failed
> at
> org.apache.nifi.processor.util.list.ListedEntityTracker.lambda$trackEntities$6(ListedEntityTracker.java:342)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.commitAsync(StandardProcessSession.java:553)
> at
> org.apache.nifi.processor.ProcessSession.commitAsync(ProcessSession.java:160)
> at
> org.apache.nifi.processor.util.list.ListedEntityTracker.trackEntities(ListedEntityTracker.java:335)
> at
> org.apache.nifi.processor.util.list.AbstractListProcessor.listByTrackingEntities(AbstractListProcessor.java:1119)
> at
> org.apache.nifi.processor.util.list.AbstractListProcessor.onTrigger(AbstractListProcessor.java:525)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1274)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:244)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> Source)
> at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
> Source)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> Source)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
> at java.base/java.lang.Thread.run(Unknown Source)
> Caused by: java.io.IOException: Request invocation failed
> at
> org.apache.nifi.distributed.cache.client.CacheClientRequestHandler.invoke(CacheClientRequestHandler.java:117)
> at
> org.apache.nifi.distributed.cache.client.DistributedCacheClient.invoke(DistributedCacheClient.java:80)
> at
> org.apache.nifi.distributed.cache.client.NettyDistributedMapCacheClient.put(NettyDistributedMapCacheClient.java:104)
> at
> org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.put(DistributedMapCacheClientService.java:141)
> at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown
> Source)
> at java.base/java.lang.reflect.Method.invoke(Unknown Source)
> at
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254)
> at
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:105)
> at jdk.proxy12/jdk.proxy12.$Proxy149.put(Unknown Source)
> at
> org.apache.nifi.processor.util.list.ListedEntityTracker.persistListedEntities(ListedEntityTracker.java:215)
> at
> org.apache.nifi.processor.util.list.ListedEntityTracker.lambda$trackEntities$6(ListedEntityTracker.java:340)
> ... 16 common frames omitted
> Caused by: java.io.IOException: Channel unregistered before processing
> completed: [id: 0x301f3f8b, L:/127.0.0.1:33614 ! R:localhost/127.0.0.1:4558]
> at
> org.apache.nifi.distributed.cache.client.CacheClientRequestHandler.channelUnregistered(CacheClientRequestHandler.java:81)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:217)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:195)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:188)
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelUnregistered(DefaultChannelPipeline.java:1385)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:215)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:195)
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:820)
> at
> io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:819)
> at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
> at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106)
> at
> io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
> at
> io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
> at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> ... 1 common frames omitted {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
