Yes, this was just one idea based on Charlie's solution. I'm not saying that approach solves the original request in this email, I was just saying its another nice idea that could be easily implemented once we make the changes in the JIRA.
There can be as many "dynamic" DBCPService implementations as we want, and anyone can implement their own, the key is making the API changes to allow for it. On Wed, Apr 25, 2018 at 12:05 PM, Sivaprasanna <sivaprasanna...@gmail.com> wrote: > Okay.. but two questions: > > > 1. We are passing the attribute 'db.id' that means, we'll be using > 'UpdateAttribute' to do add that attribute to flowfile? > 2. If we are to use 'UpdateAttribute' to set the value for 'db.id', we > need to know before hand, right? > > - > > Sivaprasanna > > On Wed, Apr 25, 2018 at 8:38 PM, Bryan Bende <bbe...@gmail.com> wrote: > >> Charlie, >> >> That is a really nice solution, thanks for sharing. >> >> If we make the API changes in that JIRA I just sent, I could see >> having a new implementation of DBCPService that does something very >> similar. >> >> Basically there could be "DelegatingDBCPService" which still >> implemented the same DBCPService interface but followed a convention >> where it always looked in the attribute map for an attribute called >> "db.id", and then it itself has dynamic properties to define many >> DBCPServices where the property name was the db.id, and it would just >> return a Connection from the one with the given id. >> >> There are definitely other options for how to implement the dynamic >> connection service, but this would be a good one to have. >> >> -Bryan >> >> >> On Wed, Apr 25, 2018 at 10:58 AM, Charlie Meyer >> <charlie.me...@civitaslearning.com> wrote: >> > Chiming in a bit late on this, but we faced this same issue and got >> around >> > it by implementing a custom controller service which acts as a "router" >> to >> > different dbcp services. It exposes a method which given a uuid, returns >> > back the DBCPservice that corresponds with that uuid if it exists using >> > >> > DBCPService dbcpService = >> > (DBCPService) >> > getControllerServiceLookup().getControllerService(uuid); >> > >> > From there, we created processors we needed based on the stock ones which >> > relied on our "router" service rather than a single DBCP. Our custom >> > processors read an attribute from incoming flow files, then send that to >> > the router which returns back the connection pool. >> > >> > On Wed, Apr 25, 2018 at 9:48 AM, Bryan Bende <bbe...@gmail.com> wrote: >> > >> >> Here is a proposal for how to modify the existing API to support both >> >> scenarios: >> >> >> >> https://issues.apache.org/jira/browse/NIFI-5121 >> >> >> >> The scope of that ticket would be to make the interface change, and >> >> then update all of NiFi's DB processors to pass in the attribute map. >> >> >> >> Then a separate effort to provide a new service implementation that >> >> used the attribute map to somehow manage multiple connection pools, or >> >> create connections on the fly, or whatever the desired behavior is. >> >> >> >> On Wed, Apr 25, 2018 at 9:34 AM, Bryan Bende <bbe...@gmail.com> wrote: >> >> > To Otto's question... >> >> > >> >> > For simplicity sake, there is a new implementation of >> >> > DBCPConnectionPool that behind the scenes has two connection pools, >> >> > one for DB A and one for DB B, it doesn't matter how these are >> >> > configured. >> >> > >> >> > Now a flow file comes into the ExecuteSQL and it goes to >> >> > connectionPoolService.getConnection()... >> >> > >> >> > How does it know which DB to return a connection for? >> >> > >> >> > >> >> > On Wed, Apr 25, 2018 at 9:01 AM, Sivaprasanna < >> sivaprasanna...@gmail.com> >> >> wrote: >> >> >> Option 2 and 3 seem to be a probable approach. However creating >> varying >> >> >> number of connections based on *each* flowfile still sounds to be >> >> >> suboptimal. If the requirement still demands to take that road, then >> >> it’s >> >> >> better to do some prep-work.. as in the list of probable connections >> >> that >> >> >> are required are taken and connection pools are created for them and >> >> then >> >> >> based on the flowfiles (which connection it needs), we use the >> relevant >> >> >> one. >> >> >> >> >> >> Thanks, >> >> >> Sivaprasanna >> >> >> >> >> >> On Wed, 25 Apr 2018 at 6:07 PM, Bryan Bende <bbe...@gmail.com> >> wrote: >> >> >> >> >> >>> The issue here is more about the service API and not the >> >> implementations. >> >> >>> >> >> >>> The current API has no way to pass information between the processor >> >> and >> >> >>> service. >> >> >>> >> >> >>> The options boil down to... >> >> >>> >> >> >>> - Make a new API, but then you need all new processors that use the >> >> new API >> >> >>> >> >> >>> - Modify the current API to have a new method, but then we are >> combing >> >> two >> >> >>> concepts into one API and some impls may not implement both >> >> >>> >> >> >>> - Modify the processors to use two different service APIs, but >> enforce >> >> that >> >> >>> only one can be used at a time, so it can have either the original >> >> >>> connection pool service or can have some new dynamic connection >> >> factory, >> >> >>> but not both, and then modify all DB processors to have logic to >> >> determine >> >> >>> which service to use. >> >> >>> >> >> >>> On Wed, Apr 25, 2018 at 8:28 AM Otto Fowler < >> ottobackwa...@gmail.com> >> >> >>> wrote: >> >> >>> >> >> >>> > Or you could just call every time you needed properties more >> likely. >> >> >>> > This would still be custom unless integrated…. >> >> >>> > >> >> >>> > >> >> >>> > On April 25, 2018 at 08:26:57, Otto Fowler ( >> ottobackwa...@gmail.com) >> >> >>> > wrote: >> >> >>> > >> >> >>> > Can services work with other controller services? >> >> >>> > Maybe a PropertiesControllerService, >> FilePropertiesControllerService >> >> >>> could >> >> >>> > work with your service? >> >> >>> > the PCS could fire events on property changes etc. >> >> >>> > >> >> >>> > >> >> >>> > >> >> >>> > On April 25, 2018 at 08:05:27, Mike Thomsen ( >> mikerthom...@gmail.com) >> >> >>> > wrote: >> >> >>> > >> >> >>> > Shot in the dark here, but what you try to do is create a custom >> >> >>> connection >> >> >>> > pool service that uses dynamic properties to build a "pool of >> >> connection >> >> >>> > pools." You could then use the property names as hints for where >> to >> >> send >> >> >>> > the queries. >> >> >>> > >> >> >>> > On Wed, Apr 25, 2018 at 6:19 AM Rishab Prasad < >> >> rishabprasad...@gmail.com >> >> >>> > >> >> >>> > wrote: >> >> >>> > >> >> >>> > > Hi, >> >> >>> > > >> >> >>> > > Basically, there are 'n' number of databases that we are dealing >> >> with. >> >> >>> We >> >> >>> > > need to fetch the data from the source database into HDFS. Now >> >> since we >> >> >>> > are >> >> >>> > > dealing with many databases, the source database is not static >> and >> >> >>> > changes >> >> >>> > > every now and then. And every time the source database changes >> we >> >> >>> > manually >> >> >>> > > need to change the value for the connection parameters in >> >> >>> > > DBCPConnectionPool. Now, people suggest that for 'n' databases >> >> create >> >> >>> 'n' >> >> >>> > > connections for each database, but that is not possible because >> >> 'n' is >> >> >>> a >> >> >>> > > big number and creating that many connections in >> >> DBCPConnectionPool is >> >> >>> > not >> >> >>> > > possible. So we were looking for a way where we can specify all >> the >> >> >>> > > connection parameters in a file present in our local system and >> >> then >> >> >>> make >> >> >>> > > the DBCPConnectionPool controller service to read the values >> from >> >> the >> >> >>> > file. >> >> >>> > > In that way we can simply change the value in the file present >> in >> >> the >> >> >>> > local >> >> >>> > > system. No need to alter anything in the dataflow. But it turns >> out >> >> >>> that >> >> >>> > > FlowFile attributes are not available to the controller services >> >> as the >> >> >>> > > expression language is evaluated at the time of service enable. >> >> >>> > > >> >> >>> > > So can you suggest a way where I can achieve my requirement >> (except >> >> >>> > > 'variable.registry' ) ? I am looking to develop a custom >> controller >> >> >>> > service >> >> >>> > > that can serve the requirement but how do I make the flowfile >> >> >>> attributes >> >> >>> > > available to the service? >> >> >>> > > >> >> >>> > >> >> >>> -- >> >> >>> Sent from Gmail Mobile >> >> >>> >> >> >>
