Chiming in a bit late on this, but we faced this same issue and got around
it by implementing a custom controller service which acts as a "router" to
different dbcp services. It exposes a method which given a uuid, returns
back the DBCPservice that corresponds with that uuid if it exists using
DBCPService dbcpService =
(DBCPService)
getControllerServiceLookup().getControllerService(uuid);
>From there, we created processors we needed based on the stock ones which
relied on our "router" service rather than a single DBCP. Our custom
processors read an attribute from incoming flow files, then send that to
the router which returns back the connection pool.
On Wed, Apr 25, 2018 at 9:48 AM, Bryan Bende <bbe...@gmail.com> wrote:
> Here is a proposal for how to modify the existing API to support both
> scenarios:
>
> https://issues.apache.org/jira/browse/NIFI-5121
>
> The scope of that ticket would be to make the interface change, and
> then update all of NiFi's DB processors to pass in the attribute map.
>
> Then a separate effort to provide a new service implementation that
> used the attribute map to somehow manage multiple connection pools, or
> create connections on the fly, or whatever the desired behavior is.
>
> On Wed, Apr 25, 2018 at 9:34 AM, Bryan Bende <bbe...@gmail.com> wrote:
> > To Otto's question...
> >
> > For simplicity sake, there is a new implementation of
> > DBCPConnectionPool that behind the scenes has two connection pools,
> > one for DB A and one for DB B, it doesn't matter how these are
> > configured.
> >
> > Now a flow file comes into the ExecuteSQL and it goes to
> > connectionPoolService.getConnection()...
> >
> > How does it know which DB to return a connection for?
> >
> >
> > On Wed, Apr 25, 2018 at 9:01 AM, Sivaprasanna <sivaprasanna...@gmail.com>
> wrote:
> >> Option 2 and 3 seem to be a probable approach. However creating varying
> >> number of connections based on *each* flowfile still sounds to be
> >> suboptimal. If the requirement still demands to take that road, then
> it’s
> >> better to do some prep-work.. as in the list of probable connections
> that
> >> are required are taken and connection pools are created for them and
> then
> >> based on the flowfiles (which connection it needs), we use the relevant
> >> one.
> >>
> >> Thanks,
> >> Sivaprasanna
> >>
> >> On Wed, 25 Apr 2018 at 6:07 PM, Bryan Bende <bbe...@gmail.com> wrote:
> >>
> >>> The issue here is more about the service API and not the
> implementations.
> >>>
> >>> The current API has no way to pass information between the processor
> and
> >>> service.
> >>>
> >>> The options boil down to...
> >>>
> >>> - Make a new API, but then you need all new processors that use the
> new API
> >>>
> >>> - Modify the current API to have a new method, but then we are combing
> two
> >>> concepts into one API and some impls may not implement both
> >>>
> >>> - Modify the processors to use two different service APIs, but enforce
> that
> >>> only one can be used at a time, so it can have either the original
> >>> connection pool service or can have some new dynamic connection
> factory,
> >>> but not both, and then modify all DB processors to have logic to
> determine
> >>> which service to use.
> >>>
> >>> On Wed, Apr 25, 2018 at 8:28 AM Otto Fowler <ottobackwa...@gmail.com>
> >>> wrote:
> >>>
> >>> > Or you could just call every time you needed properties more likely.
> >>> > This would still be custom unless integrated….
> >>> >
> >>> >
> >>> > On April 25, 2018 at 08:26:57, Otto Fowler (ottobackwa...@gmail.com)
> >>> > wrote:
> >>> >
> >>> > Can services work with other controller services?
> >>> > Maybe a PropertiesControllerService, FilePropertiesControllerService
> >>> could
> >>> > work with your service?
> >>> > the PCS could fire events on property changes etc.
> >>> >
> >>> >
> >>> >
> >>> > On April 25, 2018 at 08:05:27, Mike Thomsen (mikerthom...@gmail.com)
> >>> > wrote:
> >>> >
> >>> > Shot in the dark here, but what you try to do is create a custom
> >>> connection
> >>> > pool service that uses dynamic properties to build a "pool of
> connection
> >>> > pools." You could then use the property names as hints for where to
> send
> >>> > the queries.
> >>> >
> >>> > On Wed, Apr 25, 2018 at 6:19 AM Rishab Prasad <
> rishabprasad...@gmail.com
> >>> >
> >>> > wrote:
> >>> >
> >>> > > Hi,
> >>> > >
> >>> > > Basically, there are 'n' number of databases that we are dealing
> with.
> >>> We
> >>> > > need to fetch the data from the source database into HDFS. Now
> since we
> >>> > are
> >>> > > dealing with many databases, the source database is not static and
> >>> > changes
> >>> > > every now and then. And every time the source database changes we
> >>> > manually
> >>> > > need to change the value for the connection parameters in
> >>> > > DBCPConnectionPool. Now, people suggest that for 'n' databases
> create
> >>> 'n'
> >>> > > connections for each database, but that is not possible because
> 'n' is
> >>> a
> >>> > > big number and creating that many connections in
> DBCPConnectionPool is
> >>> > not
> >>> > > possible. So we were looking for a way where we can specify all the
> >>> > > connection parameters in a file present in our local system and
> then
> >>> make
> >>> > > the DBCPConnectionPool controller service to read the values from
> the
> >>> > file.
> >>> > > In that way we can simply change the value in the file present in
> the
> >>> > local
> >>> > > system. No need to alter anything in the dataflow. But it turns out
> >>> that
> >>> > > FlowFile attributes are not available to the controller services
> as the
> >>> > > expression language is evaluated at the time of service enable.
> >>> > >
> >>> > > So can you suggest a way where I can achieve my requirement (except
> >>> > > 'variable.registry' ) ? I am looking to develop a custom controller
> >>> > service
> >>> > > that can serve the requirement but how do I make the flowfile
> >>> attributes
> >>> > > available to the service?
> >>> > >
> >>> >
> >>> --
> >>> Sent from Gmail Mobile
> >>>
>
