To Otto's question... For simplicity sake, there is a new implementation of DBCPConnectionPool that behind the scenes has two connection pools, one for DB A and one for DB B, it doesn't matter how these are configured.
Now a flow file comes into the ExecuteSQL and it goes to connectionPoolService.getConnection()... How does it know which DB to return a connection for? On Wed, Apr 25, 2018 at 9:01 AM, Sivaprasanna <sivaprasanna...@gmail.com> wrote: > Option 2 and 3 seem to be a probable approach. However creating varying > number of connections based on *each* flowfile still sounds to be > suboptimal. If the requirement still demands to take that road, then it’s > better to do some prep-work.. as in the list of probable connections that > are required are taken and connection pools are created for them and then > based on the flowfiles (which connection it needs), we use the relevant > one. > > Thanks, > Sivaprasanna > > On Wed, 25 Apr 2018 at 6:07 PM, Bryan Bende <bbe...@gmail.com> wrote: > >> The issue here is more about the service API and not the implementations. >> >> The current API has no way to pass information between the processor and >> service. >> >> The options boil down to... >> >> - Make a new API, but then you need all new processors that use the new API >> >> - Modify the current API to have a new method, but then we are combing two >> concepts into one API and some impls may not implement both >> >> - Modify the processors to use two different service APIs, but enforce that >> only one can be used at a time, so it can have either the original >> connection pool service or can have some new dynamic connection factory, >> but not both, and then modify all DB processors to have logic to determine >> which service to use. >> >> On Wed, Apr 25, 2018 at 8:28 AM Otto Fowler <ottobackwa...@gmail.com> >> wrote: >> >> > Or you could just call every time you needed properties more likely. >> > This would still be custom unless integrated…. >> > >> > >> > On April 25, 2018 at 08:26:57, Otto Fowler (ottobackwa...@gmail.com) >> > wrote: >> > >> > Can services work with other controller services? >> > Maybe a PropertiesControllerService, FilePropertiesControllerService >> could >> > work with your service? >> > the PCS could fire events on property changes etc. >> > >> > >> > >> > On April 25, 2018 at 08:05:27, Mike Thomsen (mikerthom...@gmail.com) >> > wrote: >> > >> > Shot in the dark here, but what you try to do is create a custom >> connection >> > pool service that uses dynamic properties to build a "pool of connection >> > pools." You could then use the property names as hints for where to send >> > the queries. >> > >> > On Wed, Apr 25, 2018 at 6:19 AM Rishab Prasad <rishabprasad...@gmail.com >> > >> > wrote: >> > >> > > Hi, >> > > >> > > Basically, there are 'n' number of databases that we are dealing with. >> We >> > > need to fetch the data from the source database into HDFS. Now since we >> > are >> > > dealing with many databases, the source database is not static and >> > changes >> > > every now and then. And every time the source database changes we >> > manually >> > > need to change the value for the connection parameters in >> > > DBCPConnectionPool. Now, people suggest that for 'n' databases create >> 'n' >> > > connections for each database, but that is not possible because 'n' is >> a >> > > big number and creating that many connections in DBCPConnectionPool is >> > not >> > > possible. So we were looking for a way where we can specify all the >> > > connection parameters in a file present in our local system and then >> make >> > > the DBCPConnectionPool controller service to read the values from the >> > file. >> > > In that way we can simply change the value in the file present in the >> > local >> > > system. No need to alter anything in the dataflow. But it turns out >> that >> > > FlowFile attributes are not available to the controller services as the >> > > expression language is evaluated at the time of service enable. >> > > >> > > So can you suggest a way where I can achieve my requirement (except >> > > 'variable.registry' ) ? I am looking to develop a custom controller >> > service >> > > that can serve the requirement but how do I make the flowfile >> attributes >> > > available to the service? >> > > >> > >> -- >> Sent from Gmail Mobile >>
