Peter, Would you be able to share what you've configured for the batch size of PutSQL (assuming that 'fragmented transactions' is disabled) and what your swap threshold is configured to (nifi.queue.swap.threshold in nifi.properties)?
Also, what is following the PutSQL? Had any of those connections exceeded their configured back pressure threshold? Thanks again. Matt On Fri, Sep 9, 2016 at 11:18 AM, Peter Wicks (pwicks) <[email protected]> wrote: > PutSQL. The 100k FlowFiles are all SQL Insert queries with associated > attributys, generated by a JSONToSQL processor. > > > > *From:* Matt Gilman [mailto:[email protected]] > *Sent:* Friday, September 09, 2016 8:51 AM > > *To:* [email protected] > *Subject:* Re: Erroneous Queue has No FlowFiles message > > > > Peter, > > > > What is the processor downstream of the connection in question? Thanks. > > > > Matt > > > > On Fri, Sep 9, 2016 at 10:39 AM, Matt Gilman <[email protected]> > wrote: > > Peter, > > > > Thanks for the answers. Still not quite sure what's causing this and am > trying to narrow down the possible cause. Are you still able to replicate > the issue? If so, can you enable debug level logging for > > > > org.apache.nifi.controller.StandardFlowFileQueue > > > > and see if there are any meaningful messages in the nifi-app.log? > > > > Thanks! > > > > Matt > > > > > > On Fri, Sep 9, 2016 at 9:52 AM, Peter Wicks (pwicks) <[email protected]> > wrote: > > Matt, > > > > This is not a cluster. > > Yes, it’s secured. Kerberos. > > > > The thing that gets me is I can list another queue on the same graph/same > processor group. > > > > --Peter > > > > > > *From:* Matt Gilman [mailto:[email protected]] > *Sent:* Friday, September 09, 2016 5:25 AM > > > *To:* [email protected] > *Subject:* Re: Erroneous Queue has No FlowFiles message > > > > Peter, > > > > Thanks for the details! These will be very helpful investigating what's > happening here. A couple follow-up questions... > > > > - Is this a cluster? > > - Is this instance secured? > > > > Thanks > > > > Matt > > > > On Fri, Sep 9, 2016 at 12:13 AM, Peter Wicks (pwicks) <[email protected]> > wrote: > > Gunjan, > > > > Thanks for the response. I included those messages to emphasize the > difference between a normal Queue List and mine. In a normal queue list > the GET step includes a non-empty “flowFileSummaries” array, assuming there > are FlowFiles to show. > > When I list my other queue, the one with 23 FlowFiles in it, I get back an > array with 23 entries. Based on the JSON I’m assuming that my queue with > 100,000 files in it should return 100, but instead I get 0. > > > > Thanks, > > Peter > > > > *From:* Gunjan Dave [mailto:[email protected]] > *Sent:* Thursday, September 08, 2016 9:26 PM > *To:* [email protected] > *Subject:* Re: Erroneous Queue has No FlowFiles message > > > > Hi Peter, once you post the request, your first step, you get a listing > request reference handle UUID as part of response. > This UUID is used to perform the all the operations on the queue. > This UUID is active until a DELETE request is sent. Once you delete the > active request, you get the message you mentioned in the logs, this is not > an issue. > If you check the developer panel in chrome, you will see all 3 operations, > post-get-delete in succession. > > > > On Fri, Sep 9, 2016, 8:48 AM Peter Wicks (pwicks) <[email protected]> > wrote: > > Running NiFI 1.0.0, I’m listing a queue that has 100k files queued. I’ve > stopped both the incoming and outgoing processors, so the files are just > hanging out in the queue, no possible motion. > > > > I get, “The queue has no FlowFiles” message. Here are the actual > responses from the REST calls: > > > > *POST - Listing-requests* > > {"listingRequest":{"id":"0cee44de-0157-1000-5668-6e93a465e227","uri":" > https://localhost:8443/nifi-api/flowfile-queues/0bacce2d-0157- > 1000-1a6d-6e0fd84a6bd6/listing-requests/0cee44de- > 0157-1000-5668-6e93a465e227","submissionTime":"09/09/2016 03:12:04.318 > GMT+00:00","lastUpdated":"03:12:04 GMT+00:00","percentCompleted": > 0,"finished":false,"maxResults":100,"state":"Waiting for other queue > requests to complete","queueSize":{"byteCount":25400000," > objectCount":100000},"sourceRunning":false,"destinationRunning":false}} > > > > *GET* > > {"listingRequest":{"id":"0cee44de-0157-1000-5668- > 6e93a465e227","uri":"https:// localhost:8443/nifi-api/ > flowfile-queues/0bacce2d-0157-1000-1a6d-6e0fd84a6bd6/ > listing-requests/0cee44de-0157-1000-5668-6e93a465e227","submissionTime":"09/09/2016 > 03:12:04.318 GMT+00:00","lastUpdated":"03:12:04 > GMT+00:00","percentCompleted":100,"finished":true," > maxResults":100,"state":"Completed successfully","queueSize":{" > byteCount":25400000,"objectCount":100000},"flowFileSummaries":[]," > sourceRunning":false,"destinationRunning":false}} > > > > *DELETE* > > {"listingRequest":{"id":"0cee44de-0157-1000-5668- > 6e93a465e227","uri":"https:// localhost:8443/nifi-api/ > flowfile-queues/0bacce2d-0157-1000-1a6d-6e0fd84a6bd6/ > listing-requests/0cee44de-0157-1000-5668-6e93a465e227","submissionTime":"09/09/2016 > 03:12:04.318 GMT+00:00","lastUpdated":"03:12:04 > GMT+00:00","percentCompleted":100,"finished":true," > maxResults":100,"state":"Completed successfully","queueSize":{" > byteCount":25400000,"objectCount":100000},"sourceRunning":false," > destinationRunning":false}} > > > > On a subsequent test (thus the difference in ID’s) I checked the > nifi-app.log file and found this single message: > > > > 2016-09-09 03:15:50,043 INFO [NiFi Web Server-828] > o.a.n.controller.StandardFlowFileQueue > Canceling ListFlowFile Request with ID 0cf1b178-0157-1000-9111- > 9b889415bcdc > > > > Not clear why it was canceled. > > > > I went up one step in the process, and that queue has 23 items in it. I > was able to list it without issue. > > > > Any ideas why I can’t list the queue? > > > > Thanks, > > Peter Wicks > > > > > > >
