Lei, I don't have MiNiFi setup or a MySQL instance to test against either. I attempted to replicate by creating a 2-node nifi cluster and using a GenerateFlowFile to generate the data and send over HTTP-based site-to-site. Stopped the processor after the input port to see if i could duplicate the behavior. Unfortunately, though, I wasn't able to cause any sort of problems.
In order to understand what's happening, you'll likely have to grab a heap dump and analyze that. On Jul 28, 2019, at 7:18 AM, [email protected]<mailto:[email protected]> wrote: Hi Mark, Let me describe my scenario with details. * My sending site is a MiNiFi instance, an CaptureChangeMySQL local to a remote process group. The jvm option: java.arg.2=-Xms512m java.arg.3=-Xmx512m version: minifi-0.5.0 <Catch.jpg> * The remote instance is a cluster with two nodes. The remote process group is configured with HTTP protocol. The cluster side: <Catch8314.jpg> I stop the JoltTransfromJSON processor deliberately, after a while, the MiNiFi log shows: o.a.nifi.remote.client.http.HttpClient Penalizing a peer Peer[url=http://10.44.51.33:8080/nifi-api] due to java.io.IOException: Unexpected response code: 503 errCode:Port's Destination is Full errMessage:Received port identifier 18c0bff2-016c-1000-ffff-ffffe3eb065c, but its destination is full A few minutes later： 2019-07-28 19:03:22,273 WARN [Timer-Driven Process Thread-7] o.apache.nifi.controller.FlowController Unable to update Remote Process Groups due to java.lang.OutOfMemoryError: GC overhead limit exceeded 2019-07-28 19:03:28,508 ERROR [Remote Process Group 9b03993a-f870-3258-0000-000000000000 Thread-1] org.apache.nifi.engine.FlowEngine A flow controller task execution stopped abnormally java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC overhead limit exceeded at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.nifi.engine.FlowEngine.afterExecute(FlowEngine.java:100) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1157) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded Thanks， Lei ________________________________ [email protected]<mailto:[email protected]> From: Mark Payne<mailto:[email protected]> Date: 2019-07-27 21:24 To: [email protected]<mailto:[email protected]> Subject: Re: OutOfMemoryError if the remote process group down Hello, This is not something that should be occurring, so it's difficult to say how to avoid the problem. I have a few questions to help understand the scenario better. What version of NiFi is the sending side running? Is the remote instance a single NiFi instance or a cluster? When you start seeing OutOfMemoryError, how many FlowFiles are queued up waiting to be sent? Do the FlowFiles to be sent have lots of attributes or any really large attributes? Is the Remote Process Group configured to send data via "Raw Socket" or HTTP protocol? (this can be determined by right clicking on the Remote Process Group and going to Configure) How large is the NiFi heap? (set in conf/bootstrap.conf, as the -Xms and -Xmx parameters) Thanks -Mark On Jul 27, 2019, at 9:09 AM, [email protected]<mailto:[email protected]> wrote: A processor send message to a remote process group。 If the remote process group crash or blocked， after some time the local nifi instance will throw OutOfMemoryError and hanged to die. How can i avoid the Dying of the local Nifi instance? ________________________________ [email protected]<mailto:[email protected]>
