Btw, I was wrong about one statement in my last email.. this particular RPG (705cd4bd-1b98-3c4b-b467-c455ad52fd90) IS in the state where some nodes are STOPPED and some are RUNNING. However, even on the nodes that show RUNNING in the flow.xml.gz, the queue is still blocked.
On Thu, Jun 15, 2017 at 2:07 PM, Joe Gresock <[email protected]> wrote: > Great questions, Koji: > > 1. RAW > 2. Push > 3. No, compression is not enabled > 4. Several, but it has the same effect to "Enable transmission" as it does > to specifically disable and then enable a specific port. The key being > that all the ports are marked as enabled to start, even when not sending. > 5. This is anecdotal, but it feels like 1+ times per day. > 6. CentOS 6.9, Java 1.8.0_131. This is a 4-node NiFi cluster. There are > multiple self-RPGs, including this one, at various points on the graph. > 7. Attached. The relevant queue that is inexplicably blocked shows this > in the log: > "Remote Process Group 705cd4bd-1b98-3c4b-b467-c455ad52fd90: > https://ip-172-31-55-35.ec2.internal:8443/nifi Thread-1" Id=258 > TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ > ConditionObject@2edb0217 > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.parkNanos( > LockSupport.java:215) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ > ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > at java.util.concurrent.ScheduledThreadPoolExecutor$ > DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) > at java.util.concurrent.ScheduledThreadPoolExecutor$ > DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > at java.util.concurrent.ThreadPoolExecutor.getTask( > ThreadPoolExecutor.java:1067) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1127) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:748) > > 8. No proxy > > One strange thing I noticed was that the <scheduledState> is not > consistent in the flow.xml.gz between some of the nodes for other instances > of this self-RPG (but not for the one that is currently backing up): > 1. On disk, some of the nodes have this input port set to STOPPED, and > some have it set to RUNNING. > 2. In the UI, the input port is displayed as RUNNING. > > > > On Thu, Jun 15, 2017 at 12:21 AM, Koji Kawamura <[email protected]> > wrote: > >> Hi Joe, >> >> I haven't encountered that issue. Would you tell us more details? So >> that we can investigate it. >> >> 1. Which transport protocol are you using? RAW or HTTP >> 2. Is this a pull or push transfer? Pull = client NiFi pulls from >> remote OutputPort, Push = sends to remote InputPort >> 3. Is compression enabled? >> 4. How many ports are configured? If there are more than one, RPG >> indicates that it's 'Transmitting' as long as at least one port is >> transmitting, while others can be disabled. Right click a RPG then >> select 'Remote Ports' to see each port status. >> 5. How often does it happen? Stops once per hour/day/week?? >> 6. Please share your environment, OS, Java version ... etc >> 7. If possible, please take stack-trace while you are facing that >> issue, and share it with us, by running: "$NIFI_HOME/bin/nifh.sh dump" >> stack-trace will be written to nifi-bootstrap.log >> 8. Is proxy used? >> >> Thanks, >> Koji >> >> >> On Thu, Jun 15, 2017 at 12:41 AM, Joe Gresock <[email protected]> wrote: >> > Before I investigate further, has anyone encountered NiFi RPGs >> "disabling" >> > after a while in 1.3.0? Their status is reported as >> enabled/transmitting, >> > but the flow files just sit there until I "enable transmission" again. >> > This happens for nearly all of the RPGs I have in my flow. I just have >> to >> > keep "kick starting" them periodically to keep things flowing. >> > >> > This behavior did not happen in 1.2.0. >> > -- >> > I know what it is to be in need, and I know what it is to have plenty. >> I >> > have learned the secret of being content in any and every situation, >> > whether well fed or hungry, whether living in plenty or in want. I can >> do >> > all this through him who gives me strength. *-Philippians 4:12-13* >> > > > > -- > I know what it is to be in need, and I know what it is to have plenty. I > have learned the secret of being content in any and every situation, > whether well fed or hungry, whether living in plenty or in want. I can > do all this through him who gives me strength. *-Philippians 4:12-13* > -- I know what it is to be in need, and I know what it is to have plenty. I have learned the secret of being content in any and every situation, whether well fed or hungry, whether living in plenty or in want. I can do all this through him who gives me strength. *-Philippians 4:12-13*
