Joe Can you tell us more about the processors using cluster scoped state and what the rates are through them?
I could envision us putting too much strain on zk in some cases. Thanks Joe On Mon, Feb 13, 2017 at 10:51 AM, Joe Gresock <[email protected]> wrote: > I was able to externalize my zookeeper quorum, which is now running on 3 > separate VMs. I am able to bring up the nifi cluster when my data flow is > stopped, and I can tell the zk migration worked because I have some > processors with cluster-scoped state. > > However, I am still having a hard time getting the console to stay up, with > the same error messages from my original post. > > I also noticed the following error that I was wondering about: > > ThreadPoolRequestReplicator: Cannot replicate request GET > /nifi-api/site-to-site because there are 100 outstanding HTTP Requests > already. Request Counts per URI = {/nifi-api/site-to-site=100}. > > I'm wondering if this is the underlying problem, though I don't know why it > would happen only during a high data volume, because I am currently not > using site-to-site when I let the data run. I have several self-RPG > connections in the flow, but they are not being actively used when I > process the data at the moment. > > Interestingly, I am able to run a custom processor that stores records in > MongoDB without issue, but as soon as I run a RouteOnAttribute processor as > well, the console goes down again. > > Any other thoughts? > > On Fri, Feb 10, 2017 at 1:29 PM, Andrew Grande <[email protected]> wrote: > >> Joe, >> >> External ZK quorum would be my first move. And make sure those boxes have >> fast disks and no heavy load from other processes. >> >> Andrew >> >> On Fri, Feb 10, 2017, 7:23 AM Joe Gresock <[email protected]> wrote: >> >> > I should add that the flows on the individual nodes appear to be >> processing >> > the data just fine, and the solution I've found so far is to just wait >> for >> > the data to subside, after which point the console comes up successfully. >> > So, no complaint on the durability of the underlying data flows. It's >> just >> > problematic that I can't reliably make changes to the flow during high >> > traffic periods. >> > >> > On Fri, Feb 10, 2017 at 12:00 PM, Joe Gresock <[email protected]> >> wrote: >> > >> > > We have a 7-node cluster and we currently use the embedded zookeepers >> on >> > 3 >> > > of the nodes. I've noticed that when we have a high volume in our flow >> > > (which is causing the CPU to be hit pretty hard), I have a really hard >> > time >> > > getting the console page to come up, as it cycles through the following >> > > error messages when I relolad the page: >> > > >> > > >> > > - An unexpected error has occurred. Please check the logs. (there >> is >> > > never any error in the logs for this one) >> > > - Could not replicate request to <hostname> because the node is not >> > > connected (this is never the current host I'm trying to hit, which >> > makes >> > > the error text feel a bit irrelevant to the user. i.e., "I wasn't >> > trying >> > > to replicate a request to that node, I just want to load the console >> > on >> > > this node") >> > > - An error occurred communicating with the application core. Please >> > > check the logs and fix any configuration issues before restarting. >> > (Again, >> > > can't find any errors in nifi-app.log or nifi-user.log) >> > > >> > > I can go about a half-hour reloading the page before it comes up once, >> > and >> > > then I can only get maybe one action in before it auto-refreshes and >> > shows >> > > me one of the above error messages again. >> > > >> > > My first thought was that using some external zookeeper servers would >> > > improve this, but that's just a hunch. Has anyone encountered this >> > > behavior with high data volume? >> > > Joe >> > > >> > > -- >> > > I know what it is to be in need, and I know what it is to have >> plenty. I >> > > have learned the secret of being content in any and every situation, >> > > whether well fed or hungry, whether living in plenty or in want. I can >> > > do all this through him who gives me strength. *-Philippians >> 4:12-13* >> > > >> > >> > >> > >> > -- >> > I know what it is to be in need, and I know what it is to have plenty. I >> > have learned the secret of being content in any and every situation, >> > whether well fed or hungry, whether living in plenty or in want. I can >> do >> > all this through him who gives me strength. *-Philippians 4:12-13* >> > >> > > > > -- > I know what it is to be in need, and I know what it is to have plenty. I > have learned the secret of being content in any and every situation, > whether well fed or hungry, whether living in plenty or in want. I can do > all this through him who gives me strength. *-Philippians 4:12-13*
