Re: Deadlocks after upgrade from 0.6.1 to 1.1.1

Mike, Can you capture a series of thread dumps as the gradual decay occurs and signal at what point they were generated specifically calling out the "now the system is doing nothing" point. Can you check for space available on the system during these times as well. Also, please advise on the

Re: NiFi Users: Powered by NiFi page

wow! This is looker of LookML fame. Nice one! I remember watching this video https://youtu.be/8LqnC9IjPYY about their platform a long time ago... On Fri, Feb 17, 2017 at 8:10 AM, Andy LoPresto wrote: > Documentation that this was ok: > > Begin forwarded message: > >

Re: Deadlocks after upgrade from 0.6.1 to 1.1.1

when I said one more thing i definitely lied. Can you see anything in the UI indicating provenance backpressure is being applied and if you look in the app log is there anything interesting that isn't too sensitive to share? On Thu, Feb 16, 2017 at 11:56 PM, Joe Witt wrote:

Re: Deadlocks after upgrade from 0.6.1 to 1.1.1

Joe, Many thanks for the pointer on the Volatile provenance. It is, indeed, more critical for us that the data moves. Before receiving this message, I changed the config and restarted. The data started moving which is awesome! I'm happy to help you debug this issue. Do you need these collections

Re: Deadlocks after upgrade from 0.6.1 to 1.1.1

Cool thanks Mike. Mount question/concern resolved. On Fri, Feb 17, 2017 at 12:33 AM, Mikhail Sosonkin wrote: > I'm really happy that you guys responded so well, it's quite lonely googling > for this stuff :) > > Right now the volume is high because nifi is catching up at

Re: Deadlocks after upgrade from 0.6.1 to 1.1.1

Hi Joe, Thank you for your quick response. The system is currently in the deadlock state with 10 worker threads spinning. So, I'll gather the info you requested. - The available space on the partition is 223G free of 500G (same as was available for 0.6.1) - java.arg.3=-Xmx4096m in bootstrap.conf

Re: Deadlocks after upgrade from 0.6.1 to 1.1.1

Let me look through the log. I've not seen anything too weird there before, but I'll check again. In the UI, I quite normally see flows getting slowed because provenance can't keep up. But it hasn't been too slow for us, so I didn't pay much attention. On Fri, Feb 17, 2017 at 12:03 AM, Joe Witt

Re: Deadlocks after upgrade from 0.6.1 to 1.1.1

k - very happy to mention there is a PR out that is under review which will offer an alternative provenance implementation. Sustained high rate testing has shown out of the box 2.5x improvement with immediate indexing/results so all kinds of fun there. On Fri, Feb 17, 2017 at 12:13 AM, Mikhail

Re: Deadlocks after upgrade from 0.6.1 to 1.1.1

Mike One more thing...can you please grab a couple more thread dumps for us with 5 to 10 mins between? I don't see a deadlock but do suspect either just crazy slow IO going on or a possible livelock. The thread dump will help narrow that down a bit. Can you run 'iostat -xmh 20' for a bit (or

Re: Deadlocks after upgrade from 0.6.1 to 1.1.1

I'm really happy that you guys responded so well, it's quite lonely googling for this stuff :) Right now the volume is high because nifi is catching up at about 2.5G/5m and 500FlowFile/5m, but normally we're at about 100Mb/5m with a few spikes here and there, nothing too intense. We are using an

Re: How to avoid this splitting of single line as multi lines in SplitText?

This isn’t working because of known issue NIFI-3255. Oleg has submitted a PR with a patch and Koji has been reviewing. There are some outstanding questions about provenance chain decisions with original vs. split, but the code fixes the exception which was raised and I was able to make a

Re: Exporting and importing workflows, and restoring from backup snapshots

Thank you very much for your patience with my questions Andy. I will do that for pages I choose to watch. On Wed, Feb 15, 2017 at 5:05 AM, Andy LoPresto wrote: > Jim, > > If you have or create an account with Confluence, you can receive > notifications. There is a “Watch”

Re: Returning Responses to Http Requests

Thank you Mark. I saw that config parm and have been running with "Request Expiration" set to 10 min. Based on the behavior I've observed, it buys me time plus a large margin before the request times out with a "500: Service Unavailable". It does not seem to restrict me to 10 minutes if I have a

Wait/Notify State in a Cluster

Hi Guys, I've noticed the upcoming version has wait/notify set of processors (great write ups by Koji, avain). Does it support clustered state, though? Planned maybe? For high throughput scenarios, is it possible to switch it out to algernative implementations if e.g. default relies on a ZK?

Means to access StandardHttpContextMap for monitoring purposes

Hello. I have a suite of applications that use Http POSTs to push content to NiFi "on [their] demand". My HandleHttpRequest processor works for the most part, employing an SSL Context Service and a StandardHttpContextMap. Yesterday apps were unable to post. We are in development and test, and so

Re: Wait/Notify State in a Cluster

Andrew, With state management, only the processor that stored the state can read it. So Wait wouldn't be able to see what Notify wrote. For that reason they use distributed map cache to communicate. -Bryan On Thu, Feb 16, 2017 at 8:09 AM Andrew Grande wrote: > Hi Guys, >

Re: Need to migrate the NiFi 1.0.0 nar file to NiFi 1.1.1

Manojkumar The short answer is “yes" it is possible since NAR by definition is a self-sustaining (resources, class path etc) package. However, I am a bit confused about the UI changes you are mentioning and can’t seem to find a correlation to NAR in general. Are you saying you have changed

Need to migrate the NiFi 1.0.0 nar file to NiFi 1.1.1

Hi Team, I have done some UI changes in nifi 1.0.0 framework nar file and working on it.Now I want to migrate my changes to nifi version 1.1.1. Is it possible to reflect the changes on NiFi 1.1.1 package by just replacing the older nar file in to it or else I need to do all my changes again in

Re: Need to migrate the NiFi 1.0.0 nar file to NiFi 1.1.1

Hi Oleg, Yes,I have just changed UI code for our convenient in framework nar file. So from your response it seems that I can just replace *nifi-framework-nar-1.0.0.nar *in to *nifi 1.1.1* package library and it will reflect the changes in NiFi 1.1.1 UI Regards, Manojkumar R On Thu, Feb 16, 2017

To create csv file without double quotes.

Hi, I am creating custom processor, in which it uses hive database connection pooling service to perform file joining operation using csv file. I stored processed values in *resultSet *and then setted that *resultSet* value in AtomicLong variable named as *sample*. Then updated the flowfile

Restoring from archived flow.xml.gz files

Good morning. Our team has been discussing backup, versioning, and restore strategies to manage the backups we create of our NiFi flow.xml.gz files. I have a few questions about this. Currently I manually execute a daily backup of my NiFi workflow from the Controller Settings UI, General tab,

ConvertJSONToSQL in combination with PutCassandraQL

Hi All, For one of my requirements, I will get data in JSON and need to upload it to Cassandra. So, I have planned to use ConvertJSONToSQL processor to convert JSON to CQL and PutCassandraQL processor will execute those CQL statements for inserting the data. For this purpose, I have modified

Re: Restoring from archived flow.xml.gz files

Thank you very much for these insights Bryan. I will dig into these resources you recommended and learn more about these efforts. -Jim On Thu, Feb 16, 2017 at 10:28 AM, Bryan Bende wrote: > Hey Jim, > > Great questions! The area of deployment & version-control of flows >

No Response from NiFi to Http POST Request

I am unable to get a response from my NiFi server HandleHttpRequest processor when I issue a POST from Python code on a remote box. I suspect my configuration of certs is somehow missing a vital piece of the puzzle, though that seems contrary to my configuration of HandleHttpRequest (No

Re: NiFi Users: Powered by NiFi page

Documentation that this was ok: > Begin forwarded message: > > From: dan young > Subject: Re: Apache NiFi + Looker > Date: February 16, 2017 at 11:59:16 AM PST > To: Andy LoPresto > > Sure, i think that's fine. > > On Thu, Feb 16, 2017 at 12:12 PM

Re: Means to access StandardHttpContextMap for monitoring purposes

Jim, This is a great question and good point. For controller services it would be really valuable to have a monitoring function exposed in some standard manner via their REST API. I don't believe we have any generic support for this yet nor do I think the StandardHttpContentMap does this. It

Re: Deadlocks after upgrade from 0.6.1 to 1.1.1

Mike, No need for more info. Heap/GC looks beautiful. The thread dump however, shows some problems. The provenance repository is locked up. Numerous threads are sitting here at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727) at

Re: Deadlocks after upgrade from 0.6.1 to 1.1.1

Been a while since I've used a profiler, but I'll give it a shot when I get to a place with faster internet link :) On Fri, Feb 17, 2017 at 12:08 AM, Tony Kurc wrote: > Mike, also if what Joe asked with the backpressure is "not being applied", > if you're good with a profiler,