Great, thanks Bryan. On Thu, Jun 15, 2017 at 2:38 PM, Bryan Bende <[email protected]> wrote:
> Joe, > > Thanks for the info. > > I think this might be similar to > https://issues.apache.org/jira/browse/NIFI-3900, but the fix didn't > account for RPGs. > > I created this JIRA - https://issues.apache.org/jira/browse/NIFI-4075 > > -Bryan > > On Thu, Jun 15, 2017 at 10:14 AM, Joe Gresock <[email protected]> wrote: > > Btw, I was wrong about one statement in my last email.. this particular > RPG > > (705cd4bd-1b98-3c4b-b467-c455ad52fd90) IS in the state where some nodes > are > > STOPPED and some are RUNNING. However, even on the nodes that show > RUNNING > > in the flow.xml.gz, the queue is still blocked. > > > > > > > > On Thu, Jun 15, 2017 at 2:07 PM, Joe Gresock <[email protected]> wrote: > > > >> Great questions, Koji: > >> > >> 1. RAW > >> 2. Push > >> 3. No, compression is not enabled > >> 4. Several, but it has the same effect to "Enable transmission" as it > does > >> to specifically disable and then enable a specific port. The key being > >> that all the ports are marked as enabled to start, even when not > sending. > >> 5. This is anecdotal, but it feels like 1+ times per day. > >> 6. CentOS 6.9, Java 1.8.0_131. This is a 4-node NiFi cluster. There > are > >> multiple self-RPGs, including this one, at various points on the graph. > >> 7. Attached. The relevant queue that is inexplicably blocked shows this > >> in the log: > >> "Remote Process Group 705cd4bd-1b98-3c4b-b467-c455ad52fd90: > >> https://ip-172-31-55-35.ec2.internal:8443/nifi Thread-1" Id=258 > >> TIMED_WAITING on java.util.concurrent.locks. > AbstractQueuedSynchronizer$ > >> ConditionObject@2edb0217 > >> at sun.misc.Unsafe.park(Native Method) > >> at java.util.concurrent.locks.LockSupport.parkNanos( > >> LockSupport.java:215) > >> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ > >> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > >> at java.util.concurrent.ScheduledThreadPoolExecutor$ > >> DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) > >> at java.util.concurrent.ScheduledThreadPoolExecutor$ > >> DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) > >> at java.util.concurrent.ThreadPoolExecutor.getTask( > >> ThreadPoolExecutor.java:1067) > >> at java.util.concurrent.ThreadPoolExecutor.runWorker( > >> ThreadPoolExecutor.java:1127) > >> at java.util.concurrent.ThreadPoolExecutor$Worker.run( > >> ThreadPoolExecutor.java:617) > >> at java.lang.Thread.run(Thread.java:748) > >> > >> 8. No proxy > >> > >> One strange thing I noticed was that the <scheduledState> is not > >> consistent in the flow.xml.gz between some of the nodes for other > instances > >> of this self-RPG (but not for the one that is currently backing up): > >> 1. On disk, some of the nodes have this input port set to STOPPED, and > >> some have it set to RUNNING. > >> 2. In the UI, the input port is displayed as RUNNING. > >> > >> > >> > >> On Thu, Jun 15, 2017 at 12:21 AM, Koji Kawamura <[email protected] > > > >> wrote: > >> > >>> Hi Joe, > >>> > >>> I haven't encountered that issue. Would you tell us more details? So > >>> that we can investigate it. > >>> > >>> 1. Which transport protocol are you using? RAW or HTTP > >>> 2. Is this a pull or push transfer? Pull = client NiFi pulls from > >>> remote OutputPort, Push = sends to remote InputPort > >>> 3. Is compression enabled? > >>> 4. How many ports are configured? If there are more than one, RPG > >>> indicates that it's 'Transmitting' as long as at least one port is > >>> transmitting, while others can be disabled. Right click a RPG then > >>> select 'Remote Ports' to see each port status. > >>> 5. How often does it happen? Stops once per hour/day/week?? > >>> 6. Please share your environment, OS, Java version ... etc > >>> 7. If possible, please take stack-trace while you are facing that > >>> issue, and share it with us, by running: "$NIFI_HOME/bin/nifh.sh dump" > >>> stack-trace will be written to nifi-bootstrap.log > >>> 8. Is proxy used? > >>> > >>> Thanks, > >>> Koji > >>> > >>> > >>> On Thu, Jun 15, 2017 at 12:41 AM, Joe Gresock <[email protected]> > wrote: > >>> > Before I investigate further, has anyone encountered NiFi RPGs > >>> "disabling" > >>> > after a while in 1.3.0? Their status is reported as > >>> enabled/transmitting, > >>> > but the flow files just sit there until I "enable transmission" > again. > >>> > This happens for nearly all of the RPGs I have in my flow. I just > have > >>> to > >>> > keep "kick starting" them periodically to keep things flowing. > >>> > > >>> > This behavior did not happen in 1.2.0. > >>> > -- > >>> > I know what it is to be in need, and I know what it is to have > plenty. > >>> I > >>> > have learned the secret of being content in any and every situation, > >>> > whether well fed or hungry, whether living in plenty or in want. I > can > >>> do > >>> > all this through him who gives me strength. *-Philippians 4:12-13* > >>> > >> > >> > >> > >> -- > >> I know what it is to be in need, and I know what it is to have plenty. > I > >> have learned the secret of being content in any and every situation, > >> whether well fed or hungry, whether living in plenty or in want. I can > >> do all this through him who gives me strength. *-Philippians 4:12-13* > >> > > > > > > > > -- > > I know what it is to be in need, and I know what it is to have plenty. I > > have learned the secret of being content in any and every situation, > > whether well fed or hungry, whether living in plenty or in want. I can > do > > all this through him who gives me strength. *-Philippians 4:12-13* > -- I know what it is to be in need, and I know what it is to have plenty. I have learned the secret of being content in any and every situation, whether well fed or hungry, whether living in plenty or in want. I can do all this through him who gives me strength. *-Philippians 4:12-13*
