Re: Nifi taking forever to start

I'm not sure piggy-backing on the host entropy will work reliably. I have seen this issue in ec2, openstack boxes, etc. A newly spun up box will exhibit this issue often. Andrew On Wed, Feb 15, 2017, 10:09 AM Bryan Rosander wrote: > Hey Arnaud, > > Andy's solution is

Re: Outputting flowfiles to disk

Or, 3? -> MergeContent +-> PutFile AttributesToJson > MergeContent Or, 4? Join the ranks of custom processor writers and write one to do exactly what you want--good idea if this a pretty permanent part of your

Re: Integration between Apache NiFi and Parquet or Workaround?

Thank you, both Bryan and Giovanni for giving me so much insight on this matter. I see why you would strongly prefer Kite over this, now that I landed on one tutorial on kite-dataset and their documentation

Re: Outputting flowfiles to disk

Hello Brian A good way to do this pattern is to use 'MergeContent' and set the Merge Format to Flow File Stream, v3. This way the errors are bundled together nicely/efficiently. Ensure it gets a unique filename whenever it is dumped on disk too. When you read that bundle/file back off disk you

Outputting flowfiles to disk

Hello, Within my NiFi flows for the error scenarios I would really like the option of outputting the flow file to an error directory (the outputted file contains the flow file contents and well as the attributes). This way once the error has been resolved I can replay the FlowFile by reading it

Re[2]: MergeContent across a NiFi cluster

Thanks for the reply Joe. I'm glad I wasn't missing something obvious. I'm afraid I'm stuck with file size limitation but I'll have a word with the guys who configure the load balancer to see what affinity options they have. Thanks Brian -- Original Message -- From: "Joe Witt"

Re: MergeContent across a NiFi cluster

Brian, Great use case and you're right we don't have an easy way of handling this now. If you do indeed have a load balancer in front of the receiving nifi cluster and it can support affinity of some kind then it is possible you can set a header in HTTP Post I believe which would come from a

MergeContent across a NiFi cluster

Hello, I need to send data from one organisation to another but there are data size limits between them (this isn't my choice and has been enforced on me). I've got a 4 node NiFi cluster in each organisation. The sending NiFi cluster has the following data flow: Ingest the data by various means

RE: Integration between Apache NiFi and Parquet or Workaround?

Hi Carlos, I’m just chiming in, but if I wouldn’t use Kite (disclaimer: I would in this case) the workflow would look like this: - do stuff with NiFi - convert flowfiles to Avro - (optional: merge Avro files) - PutHDFS into a temp folder - periodically run Spark on that temp folder to convert

Re: NiFi Users: Powered by NiFi page

Hi Giovanni, GoDataDriven has been included. Thanks! On Wed, Feb 15, 2017 at 11:48 AM, Giovanni Lanzani < giovannilanz...@godatadriven.com> wrote: > Hi Joe, > > You can put GoDataDriven (https://godatadriven.com) > Summary: GoDataDriven, a Dutch service company in the data science and >

RE: NiFi Users: Powered by NiFi page

Hi Joe, You can put GoDataDriven (https://godatadriven.com) Summary: GoDataDriven, a Dutch service company in the data science and engineering space, helps customers ingest and process data in real time from the most disparate devices (including but not limited to trains!). Cheers, Giovanni

Re: Nifi taking forever to start

Hey Arnaud, Andy's solution is definitely the right answer for Java applications in general (on docker or in vm or anywhere with more limited entropy). A more general way to take care of entropy issues in docker containers (applicable beyond NiFi) is to mount the host's /dev/random or

Re: Returning Responses to Http Requests

Jim, When you configure your HandleHttpRequest processor, there is a property for the HttpContextMap to use. Within the Standard Http Context Map you can configure a property named "Request Expiration". By default, it is set to 1 minute. If any request is not handled within that time limit, it

Re: Returning Responses to Http Requests

During this time when some of the steps are stopped, could just connect your HandleHttpRequest to a different path through the flow that returns an unavailable, and then when everything is back to normal connect it back to the regular path? On Mon, Feb 13, 2017 at 8:15 AM, James McMahon

Re: Sentry & NIFI

Hello, Are you talking about sentry.io? >From Googling, it looks like they have logback support [1], and NiFi uses logback for logging, so theoretically it could work. You would have to add the raven-logback JAR and all of its transitive dependencies to the lib directory of NiFi, and then

Sentry & NIFI

Hello all, is there a simple way to connect nifi to sentry using the log appender facility?

Re: How to avoid this splitting of single line as multi lines in SplitText?

Andy, I have used following properties in ReplaceText processor. Search Value:"(.*?)(\n)(.*?)" Replacement Value:"$1\\n$3" Character Set:UTF-8 MaximumBuffer Size:1MB Replacement Strategy:Regex Replace Evaluation Mode:Entire Text Result of this processor same as like input.It could n't

Re: Nifi taking forever to start

Hi Andy, Thank you very much, and indeed it seems that you pointed the right problem. The docker is running in a VM and it seems that I had a lack of entroy. I changed the entropy source to /dev/urandom and Nifi was able to start immediately. Thank you very much for your help Arnaud On Wed,

Re: Exporting and importing workflows, and restoring from backup snapshots

Jim, If you have or create an account with Confluence, you can receive notifications. There is a “Watch” button at the top right of each page which will subscribe you to email notifications when the page changes. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint:

Re: Nifi taking forever to start

If this is not the issue, can you try starting NiFi and then run the following command to generate a thread dump and provide that to the lists? It will greatly help us determine the issue you are encountering. Thanks. $ jcmd Thread.print or $ ./bin/nifi.sh dump filewithdump.txt Andy LoPresto

Re: Nifi taking forever to start

Hi Arnaud, I’m sorry you are having trouble getting NiFi going. We want to minimize any inconvenience and get you up and running quickly. Are you by any chance running on a VM that does not have access to any physical inputs to generate entropy for secure random seeding? There is a known issue

Nifi taking forever to start

Hi guys! I'm trying to play with nifi (1.1.1) in a docker image. I tried different configuration (cluster, single node, secured, etc.), however whatever I try, Nifi takes forever to start (like 30-45 minutes). This not related to data as I observe this behavior even when I instantiate the docker