[jira] [Created] (NIFI-13106) Loadbalance does not handle all cases of ContentNotFound correctly

[saarbs (Jira)]

saarbs created NIFI-13106:
-

 Summary: Loadbalance does not handle all cases of ContentNotFound 
correctly
 Key: NIFI-13106
 URL: https://issues.apache.org/jira/browse/NIFI-13106
 Project: Apache NiFi
  Issue Type: Bug
  Components: Core Framework
Affects Versions: 1.22.0
Reporter: saarbs


In cases where the ContentRepositoryFlowFileAccess is able to skip to the 
content but the content is cut in the middle then an EOF exception is thrown 
and not handled accordingly

 

Log Sample:
[ERROR] [Load-Balanced Client Thread-5] 
[o.a.n.c.q.c.c.a.n.NioAsyncLoadBalanceClient] Failed to communicate with Peer 
localhost:8080
java.io.EOFException: Expected 
StandardFlowFileRecord[uuid=848d2364-e7db-4071-a345-b9b3f66d3afc,claim=StandardContentClaim
 [resourceClaim=StandardResourceClaim[id=1714171534814-132572, 
container=default, section=476], offset=0, 
length=1741],offest-0,name=848d2364-e7db-4071-a345-b9b3f66d3afc,size=1741] to 
contain 1741 bytes but the content repository only had 0 bytes for it

at 
org.apache.nifi.controller.queue.clustered.ContentRepositoryFlowFileAccess$1.ensureNotTruncated(ContentRepositoryFlowFileAccess.java:83)

at 
org.apache.nifi.controller.queue.clustered.ContentRepositoryFlowFileAccess$1.read(ContentRepositoryFlowFileAccess.java:63)

at org.apache.nifi.stream.io.StreamUtils.fillBuffer(StreamUtils.java:89)

at 
org.apache.nifi.controller.queue.clustered.client.async.nio.LoadBalanceSession.getFlowFileContent(LoadBalanceSession.java:298)

at 
org.apache.nifi.controller.queue.clustered.client.async.nio.LoadBalanceSession.getDataFrame(LoadBalanceSession.java:253)

at 
org.apache.nifi.controller.queue.clustered.client.async.nio.LoadBalanceSession.communicate(LoadBalanceSession.java:155)

at 
org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.communicate(NioAsyncLoadBalanceClient.java:265)

at 
org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.run(NioAsyncLoadBalanceClient.java:81)

at org.apache.nifi.engine.FlowEngine$2.run(FlowFileEngine.java:110)

at java.base/java.util.concurrent.Executer$RunnableAdapter.call(Unknown Source)

at java.base/java.util.concurrent.FututeTask.run(Unknown Source)

at 
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
 Source)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.base/java.lang.Thread.run(Unknown Source)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (NIFI-12882) Allow control over unexpected failure behavior in processors

[saarbs (Jira)]

saarbs created NIFI-12882:
-

 Summary: Allow control over unexpected failure behavior in 
processors
 Key: NIFI-12882
 URL: https://issues.apache.org/jira/browse/NIFI-12882
 Project: Apache NiFi
  Issue Type: Bug
  Components: Core Framework
Reporter: saarbs


Recently we had a problem with a flow using UpdateAttribute that does 
base64decode, and one of our flow sources began sending invalid base64. This 
resulted in exceptions in the processor causing rollbacks to the flowfiles and 
backpressure in the flow.



Since there is no failure relationship to the processor there was no way to 
resolve this issue, And we realized there are a lot of processors facing this 
issue where an unexpected failure could cause an infinite loop where some flows 
may desire to send them to a failure relationship.

I suggest a setting or an unexpected failure relationship, which would allow 
keeping the existing behavior of a rollback but would also allow terminating or 
processing the failure in different ways.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (NIFI-12830) Memory leak outside of heap

[saarbs (Jira)]

saarbs created NIFI-12830:
-

 Summary: Memory leak outside of heap
 Key: NIFI-12830
 URL: https://issues.apache.org/jira/browse/NIFI-12830
 Project: Apache NiFi
  Issue Type: Bug
Affects Versions: 1.22.0
 Environment: Openshift
Reporter: saarbs


We run NiFi on Openshift and we are experiencing problems regarding memory 
leaks in some of our clusters. we set the max heap size to 50% (6g) of the pod 
request and limit (12Gi), and we experience frequent OOM kills almost 4-5 times 
a day per pod in our 5 pod cluster.



Using {{top}} we see the process memory usage increases over time until it 
reaches twice the heap size and is killed by the Openshift OOM Killer.

Using 
{code:java}
jcmd VM.native_memory{code}
 we determined that the leak is not in the heap and is not in memory tracked 
off-heap.



Then we used {{pmap}} and partial memory dumps of the parts we suspect are part 
of the leak.

Inspecting the memory using {{strings}} the notable content is JDI type 
signatures, flowfile attributes, and flowfile contents.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)