Joe, Thanks for the info.
I think this might be similar to https://issues.apache.org/jira/browse/NIFI-3900, but the fix didn't account for RPGs. I created this JIRA - https://issues.apache.org/jira/browse/NIFI-4075 -Bryan On Thu, Jun 15, 2017 at 10:14 AM, Joe Gresock <jgres...@gmail.com> wrote: > Btw, I was wrong about one statement in my last email.. this particular RPG > (705cd4bd-1b98-3c4b-b467-c455ad52fd90) IS in the state where some nodes are > STOPPED and some are RUNNING. However, even on the nodes that show RUNNING > in the flow.xml.gz, the queue is still blocked. > > > > On Thu, Jun 15, 2017 at 2:07 PM, Joe Gresock <jgres...@gmail.com> wrote: > >> Great questions, Koji: >> >> 1. RAW >> 2. Push >> 3. No, compression is not enabled >> 4. Several, but it has the same effect to "Enable transmission" as it does >> to specifically disable and then enable a specific port. The key being >> that all the ports are marked as enabled to start, even when not sending. >> 5. This is anecdotal, but it feels like 1+ times per day. >> 6. CentOS 6.9, Java 1.8.0_131. This is a 4-node NiFi cluster. There are >> multiple self-RPGs, including this one, at various points on the graph. >> 7. Attached. The relevant queue that is inexplicably blocked shows this >> in the log: >> "Remote Process Group 705cd4bd-1b98-3c4b-b467-c455ad52fd90: >> https://ip-172-31-55-35.ec2.internal:8443/nifi Thread-1" Id=258 >> TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ >> ConditionObject@2edb0217 >> at sun.misc.Unsafe.park(Native Method) >> at java.util.concurrent.locks.LockSupport.parkNanos( >> LockSupport.java:215) >> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ >> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) >> at java.util.concurrent.ScheduledThreadPoolExecutor$ >> DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) >> at java.util.concurrent.ScheduledThreadPoolExecutor$ >> DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) >> at java.util.concurrent.ThreadPoolExecutor.getTask( >> ThreadPoolExecutor.java:1067) >> at java.util.concurrent.ThreadPoolExecutor.runWorker( >> ThreadPoolExecutor.java:1127) >> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >> ThreadPoolExecutor.java:617) >> at java.lang.Thread.run(Thread.java:748) >> >> 8. No proxy >> >> One strange thing I noticed was that the <scheduledState> is not >> consistent in the flow.xml.gz between some of the nodes for other instances >> of this self-RPG (but not for the one that is currently backing up): >> 1. On disk, some of the nodes have this input port set to STOPPED, and >> some have it set to RUNNING. >> 2. In the UI, the input port is displayed as RUNNING. >> >> >> >> On Thu, Jun 15, 2017 at 12:21 AM, Koji Kawamura <ijokaruma...@gmail.com> >> wrote: >> >>> Hi Joe, >>> >>> I haven't encountered that issue. Would you tell us more details? So >>> that we can investigate it. >>> >>> 1. Which transport protocol are you using? RAW or HTTP >>> 2. Is this a pull or push transfer? Pull = client NiFi pulls from >>> remote OutputPort, Push = sends to remote InputPort >>> 3. Is compression enabled? >>> 4. How many ports are configured? If there are more than one, RPG >>> indicates that it's 'Transmitting' as long as at least one port is >>> transmitting, while others can be disabled. Right click a RPG then >>> select 'Remote Ports' to see each port status. >>> 5. How often does it happen? Stops once per hour/day/week?? >>> 6. Please share your environment, OS, Java version ... etc >>> 7. If possible, please take stack-trace while you are facing that >>> issue, and share it with us, by running: "$NIFI_HOME/bin/nifh.sh dump" >>> stack-trace will be written to nifi-bootstrap.log >>> 8. Is proxy used? >>> >>> Thanks, >>> Koji >>> >>> >>> On Thu, Jun 15, 2017 at 12:41 AM, Joe Gresock <jgres...@gmail.com> wrote: >>> > Before I investigate further, has anyone encountered NiFi RPGs >>> "disabling" >>> > after a while in 1.3.0? Their status is reported as >>> enabled/transmitting, >>> > but the flow files just sit there until I "enable transmission" again. >>> > This happens for nearly all of the RPGs I have in my flow. I just have >>> to >>> > keep "kick starting" them periodically to keep things flowing. >>> > >>> > This behavior did not happen in 1.2.0. >>> > -- >>> > I know what it is to be in need, and I know what it is to have plenty. >>> I >>> > have learned the secret of being content in any and every situation, >>> > whether well fed or hungry, whether living in plenty or in want. I can >>> do >>> > all this through him who gives me strength. *-Philippians 4:12-13* >>> >> >> >> >> -- >> I know what it is to be in need, and I know what it is to have plenty. I >> have learned the secret of being content in any and every situation, >> whether well fed or hungry, whether living in plenty or in want. I can >> do all this through him who gives me strength. *-Philippians 4:12-13* >> > > > > -- > I know what it is to be in need, and I know what it is to have plenty. I > have learned the secret of being content in any and every situation, > whether well fed or hungry, whether living in plenty or in want. I can do > all this through him who gives me strength. *-Philippians 4:12-13*
