Hi Bryan, Thanks for the ideas so far -- It is certainly possible these flow files were swapped out, since I have 150,000 queued up on each node, but so far when I just switched back to "do not load balance" on that connection, no flow files appear to have been lost.
I've attached a thread dump on the server where I noticed the flow files were sitting in this queue while it was still "round robin". On Tue, Jun 4, 2019 at 3:13 PM Bryan Bende <[email protected]> wrote: > Joe, > > There are two known issues that possibly seem related... > > The first was already addressed in 1.9.0, but the reason I mention it > is because it was specific to a connection between two ports: > > https://issues.apache.org/jira/browse/NIFI-5919 > > The second is not in a release yet, but is addressed in master, and > has to do with swapping: > > https://issues.apache.org/jira/browse/NIFI-6285 > > Seems like you wouldn't hit the first one since you are on 1.9.2, but > does seem odd that is the same scenario. > > Mark P probably knows best about debugging, but I'm guessing possibly > a thread dump while in this state would be helpful. > > -Bryan > > On Tue, Jun 4, 2019 at 10:56 AM Joe Gresock <[email protected]> wrote: > > > > I have round robin load balanced connections working on one cluster, but > on another, this type of connection seems to be stuck. > > > > What would be the best way to debug this problem? The connection is > from one processor group to another, so it's from an Output Port to an > Input Port. > > > > My configuration is as follows: > > nifi.cluster.load.balance.host= > > nifi.cluster.load.balance.port=6342 > > nifi.cluster.load.balance.connections.per.node=4 > > nifi.cluster.load.balance.max.thread.count=8 > > nifi.cluster.load.balance.comms.timeout=30 sec > > > > And I ensured port 6342 is open from one node to another using the > cluster node addresses. > > > > Is there some error that should appear in the logs if flow files get > stuck here? > > > > I suspect they are actually stuck, not just missing, because the > remainder of the flow is back-pressured up until this point in the flow. > > > > Thanks! > > Joe > > > -- I know what it is to be in need, and I know what it is to have plenty. I have learned the secret of being content in any and every situation, whether well fed or hungry, whether living in plenty or in want. I can do all this through him who gives me strength. *-Philippians 4:12-13*
nifi-bootstrap.log.gz
Description: application/gzip
