Re: DB Dynamic Connection
Eduardo, It doesn't sound like DBCPConnectionPoolLookup will work for you because of all the different connection strings. I don't know if there's a good reason why we couldn't create the BasicDataSource when getConnection() is called, passing in a Map of FlowFile attributes (that's how the Lookup version works). One issue I do see is with "churn" if we're recreating the data source each time. At that point it's not pooling connections. I suppose you could have an internal cache of data sources but it would have to be bounded and/or configurable and have a least-recently-used (LRU) eviction strategy. DBCPService is the name of the controller service interface that the database processors use, but that's a misnomer since the API doesn't mention pooling specifically. Instead you could have an implementation that uses a cache vs a pooling approach. But Apache DBCP does handle a lot of the management (validation, eviction, idle timeouts, etc.) so unless there's no way to avoid the potential memory/performance issues (like having 50+ controller services in a PG) you could try to wrangle smaller pools per data source and cache those if that's ok for your use case. My two cents, Matt On Tue, Mar 5, 2024 at 7:25 PM Eduardo Fontes wrote: > Hi Everybody! > > I'm thinking about make a generic ingestor with Apache NiFi but I found > some difficulties because of the DataBase Connection Pool controller. It > doesn't accept flowfiles parameters for its properties, specially > connection string, username and password (for security reasons, some > sensitive parameter name instead password itself). > > This is important because, as a generic ingestor, I might have hundreds of > different connection strings, and I had a lot of problems when I tried to > put 50 DBCP controllers in a Process Group. > > I wouldn't like to create a flow for each ingestion, but one flow for each > database vendor. > > Does anyone have any suggestions on how I can achieve this? Would it be > easy to create a parameterized DBCP controller? (That I could do it myself) > > Best regards. > > Eduardo Fontes > Data Eng / System Analyst Sr. >
DB Dynamic Connection
Hi Everybody! I'm thinking about make a generic ingestor with Apache NiFi but I found some difficulties because of the DataBase Connection Pool controller. It doesn't accept flowfiles parameters for its properties, specially connection string, username and password (for security reasons, some sensitive parameter name instead password itself). This is important because, as a generic ingestor, I might have hundreds of different connection strings, and I had a lot of problems when I tried to put 50 DBCP controllers in a Process Group. I wouldn't like to create a flow for each ingestion, but one flow for each database vendor. Does anyone have any suggestions on how I can achieve this? Would it be easy to create a parameterized DBCP controller? (That I could do it myself) Best regards. Eduardo Fontes Data Eng / System Analyst Sr.
Re: NiFi 2.0.0-M2
CentOS 7 is a very old distribution, so you may run into issues, but in theory, if you can install Java 21 on it, and start NiFi using that, then it should work. I'd be surprised if Java 21 was included in the CentOS 7 package repositories. You'll most likely need to install it manually, and set JAVA_HOME appropriately in the environment you're using to start NiFi. Marton On 3/5/24 14:10, Pierre Villard wrote: Hi, NiFi is OS-agnostic for linux based distributions. You can download the tar.gz from the download page of the website and follow the instructions to get started on CentOS 7. HTH, Pierre Le mar. 5 mars 2024 à 14:07, Gleb Efimov a écrit : Good afternoon I'm trying to deploy NiFi 2.0.0-M2 on Centos 7, but, unfortunately, I can't find a tar archive with the distribution I need. Could you tell me where I can get it? And is it even possible to deploy NiFi 2.0.0-M2 on Centos 7? Thank you very much. Sincerely, Efimov Gleb.
Re: NiFi 2.0.0-M2
Hi, NiFi is OS-agnostic for linux based distributions. You can download the tar.gz from the download page of the website and follow the instructions to get started on CentOS 7. HTH, Pierre Le mar. 5 mars 2024 à 14:07, Gleb Efimov a écrit : > Good afternoon I'm trying to deploy NiFi 2.0.0-M2 on Centos 7, but, > unfortunately, I can't find a tar archive with the distribution I need. > Could you tell me where I can get it? > And is it even possible to deploy NiFi 2.0.0-M2 on Centos 7? > Thank you very much. > > Sincerely, Efimov Gleb. >
NiFi 2.0.0-M2
Good afternoon I'm trying to deploy NiFi 2.0.0-M2 on Centos 7, but, unfortunately, I can't find a tar archive with the distribution I need. Could you tell me where I can get it? And is it even possible to deploy NiFi 2.0.0-M2 on Centos 7? Thank you very much. Sincerely, Efimov Gleb.
