RE: Erroneous Queue has No FlowFiles message

Matt, I’ve identified the source of the issue, created a patch/unit test, and PR. In StandardFlowFileQueue: writeSwapFilesIfNecessary. When it calculates the `numSwapFiles`, if the number of FlowFiles in the queue is perfectly splitable (in my case 10/2 = 5) and the Active Queue is emp

Re: Erroneous Queue has No FlowFiles message

Peter, Thanks for the confirmation. I think there is some case were hitting here where some flowfiles are being swapped instead of added back to the active queue. The queue listing only returns the top 100 entries in the active queue. Haven't identified the case that's causing it yet but defini

RE: Upgrade 0.7.0 to 1.0.0

These are two specific questions that we have. 1) In Nifi Summary/Remove Process Group, what value need to be used for the Target URI. We were able to use the specific URI in Nifi 0.7.0 because we had to specified the master node; however, Nifi 1.0.0 uses zero master clustering and autom

RE: Erroneous Queue has No FlowFiles message

Matt, I followed the swapping train of thought and debugged the code. When I debug the code where it gets the files the `size` variable looks like this: FlowFile Queue Size[ ActiveQueue=[0, 0 Bytes], Swap Queue=[10, 2660 Bytes], Swap Files=[10], Unacknowledged=[0, 0 Bytes] ] But the Li

RE: Erroneous Queue has No FlowFiles message

Matt, You also asked in an earlier email if I could still reproduce it, and if so to try enabling DEBUG level logging. I am able to reproduce, so I enabled it: 2016-09-09 21:27:28,352 DEBUG [List FlowFiles for Connection 0f620e2d-0157-1000-4a1d-fd988c59e290] o.a.n.controller.StandardFlowFileQu

RE: Erroneous Queue has No FlowFiles message

Matt, PutSQL is the end of the line, no downstream processors. Batch size is 1000, yes I have fragmented transactions set to false. nifi.queue.swap.threshold=2 --Peter From: Matt Gilman [mailto:matt.c.gil...@gmail.com] Sent: Friday, September 09, 2016 2:23 PM To: users@nifi.apache.org Subj

Re: Erroneous Queue has No FlowFiles message

Peter, Would you be able to share what you've configured for the batch size of PutSQL (assuming that 'fragmented transactions' is disabled) and what your swap threshold is configured to (nifi.queue.swap.threshold in nifi.properties)? Also, what is following the PutSQL? Had any of those connection

Upgrade 0.7.0 to 1.0.0

Hello, We upgraded from 0.7.0 to 1.0.0, and we encountered some issues. We think that this upgrade brought a change to how the site-to-site calls are made for the remote process groups and is causing us to hit a max requests limit that was also introduced with v1.0. Is this a known issue? The

RE: Appending files in Hadoop with PutHDFS ...

Thanks for your help Matt and Andre. We will try out your proposed solutions. Regards, Suyog Kulkarni Email: suyog_kulka...@csx.com From: Andre [mailto:andre-li...@fucs.org] Sent: Wednesday, September 07, 2016 8:54 PM To: users@nifi.apache.org Subject: Re: Appending

Re: PermissionBasedStatusMergerSpec is failing

Tijo, I just ran this test on master and it's passing for me. Can you provide some details about the branch you're on when running the tests? I see that tasksDuration is 00:30:00.000 when it's expecting 00:00:00.000, and that's why the JSON isn't matching. On Thu, Sep 8, 2016 at 4:58 PM Tijo Th

RE: Erroneous Queue has No FlowFiles message

PutSQL. The 100k FlowFiles are all SQL Insert queries with associated attributys, generated by a JSONToSQL processor. From: Matt Gilman [mailto:matt.c.gil...@gmail.com] Sent: Friday, September 09, 2016 8:51 AM To: users@nifi.apache.org Subject: Re: Erroneous Queue has No FlowFiles message Peter

Re: Erroneous Queue has No FlowFiles message

Peter, What is the processor downstream of the connection in question? Thanks. Matt On Fri, Sep 9, 2016 at 10:39 AM, Matt Gilman wrote: > Peter, > > Thanks for the answers. Still not quite sure what's causing this and am > trying to narrow down the possible cause. Are you still able to replica

Re: Provenance expiration error

--bump-- Any ideas on the below issue? Thanks, --Adam On Wed, Aug 31, 2016 at 4:46 PM, Adam J. Shook wrote: > Hello, > > I continue to receive the below error regarding deleting entries from the > provenance repository. The Googles aren't returning anything too helpful. > > NiFi v0.7.0 on RHE

Re: Erroneous Queue has No FlowFiles message

Peter, Thanks for the answers. Still not quite sure what's causing this and am trying to narrow down the possible cause. Are you still able to replicate the issue? If so, can you enable debug level logging for org.apache.nifi.controller.StandardFlowFileQueue and see if there are any meaningful m

RE: Erroneous Queue has No FlowFiles message

Matt, This is not a cluster. Yes, it’s secured. Kerberos. The thing that gets me is I can list another queue on the same graph/same processor group. --Peter From: Matt Gilman [mailto:matt.c.gil...@gmail.com] Sent: Friday, September 09, 2016 5:25 AM To: users@nifi.apache.org Subject: Re: Erron

Re: Nifi 1.0.0 compatibility with Hive 1.1.0

I suspected the same, and that's what I've been trying to accomplish so far (with little luck). Thanks for confirming, Matt. I'll try to see if we can upgrade our Hive version to Hive 1.2.1 in the mean time. thanks, Yari On 9 September 2016 at 15:10, Matt Burgess wrote: > Yari, > > NiFi was c

Re: Nifi 1.0.0 compatibility with Hive 1.1.0

Yari, NiFi was coded and built against Apache Hive 1.2.1. Some of the API, files, and folders had changed between Hive 1.1.0 and Hive 1.2.1, such as the org.apache.hadoop.hive.ql.io.filters package being added in 1.2.0. Bringing in Apache ORC won't work for the same reason, it was split from Hive

Re: Nifi 1.0.0 compatibility with Hive 1.1.0

Thanks, Andre. This is the command I'm using: mvn install -DskipTests -Pcloudera -Dhadoop.version=2.6.0-cdh5.5.2 -Dhive.version=1.1.0-cdh5.5.2 Adding as dependency: org.apache.orc orc-core 1.1.0 Yari On 9 September 2016 at 13:23, Andre wrote: > Yari, > > Can you please confir

Re: Dynamic property in QueryDatabaseTable

Hi Matt, Sorry for this inconvenience. The dynamic property support from version 1.0.0. Since, i am using 0.6.0 version. Regards, Ravisankar On Fri, Sep 9, 2016 at 12:56 AM, Ravisankar Mani wrote: > Hi Matt, > > Thanks for your reply. > I can add the new property like initial.maxvalue.id and

Re: Erroneous Queue has No FlowFiles message

Peter, Thanks for the details! These will be very helpful investigating what's happening here. A couple follow-up questions... - Is this a cluster? - Is this instance secured? Thanks Matt On Fri, Sep 9, 2016 at 12:13 AM, Peter Wicks (pwicks) wrote: > Gunjan, > > > > Thanks for the response.

Re: Nifi 1.0.0 compatibility with Hive 1.1.0

Yari, Can you please confirm what command line are you using to compile? Happy yo try to reproduce. Cheers On Fri, Sep 9, 2016 at 8:49 PM, Yari Marchetti < yari.marche...@buongiorno.com> wrote: > Hi Peter/Andre, > I cherrypicked the commit and added the dependency on: > > > org.apache.orc

Re: Nifi 1.0.0 compatibility with Hive 1.1.0

Hi Peter/Andre, I cherrypicked the commit and added the dependency on: org.apache.orc orc-core 1.1.0 (tried also with version 1.2.0) but stil no luck, as I keep on getting compilation errors: [ERROR] /nifi/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apa