Option 2 and 3 seem to be a probable approach. However creating varying number of connections based on *each* flowfile still sounds to be suboptimal. If the requirement still demands to take that road, then it’s better to do some prep-work.. as in the list of probable connections that are required are taken and connection pools are created for them and then based on the flowfiles (which connection it needs), we use the relevant one.
Thanks, Sivaprasanna On Wed, 25 Apr 2018 at 6:07 PM, Bryan Bende <bbe...@gmail.com> wrote: > The issue here is more about the service API and not the implementations. > > The current API has no way to pass information between the processor and > service. > > The options boil down to... > > - Make a new API, but then you need all new processors that use the new API > > - Modify the current API to have a new method, but then we are combing two > concepts into one API and some impls may not implement both > > - Modify the processors to use two different service APIs, but enforce that > only one can be used at a time, so it can have either the original > connection pool service or can have some new dynamic connection factory, > but not both, and then modify all DB processors to have logic to determine > which service to use. > > On Wed, Apr 25, 2018 at 8:28 AM Otto Fowler <ottobackwa...@gmail.com> > wrote: > > > Or you could just call every time you needed properties more likely. > > This would still be custom unless integrated…. > > > > > > On April 25, 2018 at 08:26:57, Otto Fowler (ottobackwa...@gmail.com) > > wrote: > > > > Can services work with other controller services? > > Maybe a PropertiesControllerService, FilePropertiesControllerService > could > > work with your service? > > the PCS could fire events on property changes etc. > > > > > > > > On April 25, 2018 at 08:05:27, Mike Thomsen (mikerthom...@gmail.com) > > wrote: > > > > Shot in the dark here, but what you try to do is create a custom > connection > > pool service that uses dynamic properties to build a "pool of connection > > pools." You could then use the property names as hints for where to send > > the queries. > > > > On Wed, Apr 25, 2018 at 6:19 AM Rishab Prasad <rishabprasad...@gmail.com > > > > wrote: > > > > > Hi, > > > > > > Basically, there are 'n' number of databases that we are dealing with. > We > > > need to fetch the data from the source database into HDFS. Now since we > > are > > > dealing with many databases, the source database is not static and > > changes > > > every now and then. And every time the source database changes we > > manually > > > need to change the value for the connection parameters in > > > DBCPConnectionPool. Now, people suggest that for 'n' databases create > 'n' > > > connections for each database, but that is not possible because 'n' is > a > > > big number and creating that many connections in DBCPConnectionPool is > > not > > > possible. So we were looking for a way where we can specify all the > > > connection parameters in a file present in our local system and then > make > > > the DBCPConnectionPool controller service to read the values from the > > file. > > > In that way we can simply change the value in the file present in the > > local > > > system. No need to alter anything in the dataflow. But it turns out > that > > > FlowFile attributes are not available to the controller services as the > > > expression language is evaluated at the time of service enable. > > > > > > So can you suggest a way where I can achieve my requirement (except > > > 'variable.registry' ) ? I am looking to develop a custom controller > > service > > > that can serve the requirement but how do I make the flowfile > attributes > > > available to the service? > > > > > > -- > Sent from Gmail Mobile >
