Data Ingestion using NiFi

Hi NiFi users, We had been using the kylo data ingest template to read the data from our Oracle and DB2 databases and move it into HDFS and Hive. The kylo data ingest template also provided some features to validate, profile and split the data based on validation rules. We also built some custom

Re: Using Nifi OIDC authentication through a proxy?

Hi Pat, Koji Kawamura has a set of examples on how to configure NiFi behind reverse proxy that I have successfully used for my setup: https://github.com/ijokarumawak/nifi-reverseproxy Also, check the following thread:

Re: data metrics / data monitoring

Hi Peter, I think this depends on where this data is stored. If this data is avaiable as metrics record by Nifi, then a reporting task would be the best way forward. However if this is data that is recorded in your FlowFiles as part of your flow then I think you're looking at either collecting in

data metrics / data monitoring

Hello, does anyone have a tip for me on how I can provide metrics about data processed in nifi in a web UI? I process XML files with nifi. for each file I calculate how many new, modified, unmodified, and deleted records are contained. for each record checks are still made. For example,

Using Nifi OIDC authentication through a proxy?

Hi Folks, Wondered if anyone has been able to configure Nifi to use oidc identity provider through a reverse proxy? I've been able to configure oidc and successfully work with an authentication provider directly, however i need to do this through a proxy, and not able to get the callback

Re: Nifi capabilities

Our recommendation to our clients is usually the same. We've saved them a lot of money by focusing on a more "slow and steady wins the race" pace for data ingestion since there is no critical business need to get the data ingested anything like real time. On Mon, Aug 12, 2019 at 8:58 AM Pierre

Re: Abstract Processor > Static final properties

Also, static fields are scoped to the classloader, they are not global in a sense you described. NiFi uses classloader isolation for its processors. Andrew On Mon, Aug 12, 2019, 12:24 AM Craig Knell wrote: > Thanks > > Best regards > > Craig Knell > > Mobile 61 402128615 > Skype craigknell > >

Re: Nifi capabilities

It mainly depends of your workloads. NiFi is not memory consuming unless you're doing specific operations on the data / use memory intensive processors. For high performance you'd likely go for CPU-optimized VMs with attached SSDs for repositories. But my recommendation is to start small and adapt

Re: Nifi capabilities

Thanks, We are pretty much on the AWS cloud and Hardware/OS failures are very unlikely. Can you please suggest a machine type on AWS, I am considering M5.xLarge. Need to choose a machine type based on prioritizing. 1) High Disk I/O 2) Memory 3) CPU -Dweep On Mon, Aug 5, 2019 at 5:32 PM

Re: Abstract Processor > Static final properties

Thanks Best regards Craig Knell Mobile 61 402128615 Skype craigknell > On 12 Aug 2019, at 13:57, Bryan Bende wrote: > > Hello, > > The final static variables are usually the descriptors which are just the > definition of the properties. The actual values of the properties are storied > in