Clarification Regarding custom controller service & AWS-CREDENTIAL-CONTROLER-SERVICE
Hi Team, I have a requirement of building a custom controller service(AWS-CREDENTIAL-CONTROLER-SERVICE) and custom processor service(NIFI-AWS-SERVICE).for puts3Object. Already code changes are done for both of it. and i am building 2 separate nars . The custom controller service(AWS-CREDENTIAL-CONTROLER-SERVICE) nar contain dependency of custom processor service(NIFI-AWS-SERVICE). So for puts3Object processor, it should use my custom controller service. My question is, Should i build 2 separate nars , 1 for custom processor service(NIFI-AWS-SERVICE) and 1 for custom controller service(AWS-CREDENTIAL-CONTROLER-SERVICE). and putting both the nars , in lib folder to make nifi work. Or should i build only 1 nar file for custom controller service(AWS-CREDENTIAL-CONTROLER-SERVICE) as it has a pom dependency of custom processor service(NIFI-AWS-SERVICE). Please clarify. Regards, Sanjeet -- Sanjeet Kumar Rath, mob- +91 8777577470
RE: [EXT] Re: Auto cleanup custom Controller Service properties?
Bryan, That wasn't the behavior I was seeing, but then when I went and tested it again, that was the behavior I saw so never mind. --Peter -Original Message- From: Bryan Bende Sent: Friday, July 19, 2019 9:08 AM To: dev@nifi.apache.org Subject: [EXT] Re: Auto cleanup custom Controller Service properties? I believe you should still be able to delete the property with a delete icon next to it. This should be the same behavior for processors when a property is removed, regardless of dynamic properties. On Fri, Jul 19, 2019 at 11:01 AM Peter Wicks (pwicks) wrote: > > I ran into a fairly rare situation. While working on a custom Controller > Service I removed a previously present property from the code, and this > specific Controller Service does not support Dynamic Properties. > > Like a Processor, the property now shows up as a custom property at the > bottom, but since the Controller Service is not flagged for Dynamic > Properties, there is no way to remove it. > > Is this a situation we should care about, in case we do remove a property > from a controller service in the future and otherwise leave users with > re-creating all affected controller services to resolve the issue? > > Thanks, > Peter
Re: Auto cleanup custom Controller Service properties?
I believe you should still be able to delete the property with a delete icon next to it. This should be the same behavior for processors when a property is removed, regardless of dynamic properties. On Fri, Jul 19, 2019 at 11:01 AM Peter Wicks (pwicks) wrote: > > I ran into a fairly rare situation. While working on a custom Controller > Service I removed a previously present property from the code, and this > specific Controller Service does not support Dynamic Properties. > > Like a Processor, the property now shows up as a custom property at the > bottom, but since the Controller Service is not flagged for Dynamic > Properties, there is no way to remove it. > > Is this a situation we should care about, in case we do remove a property > from a controller service in the future and otherwise leave users with > re-creating all affected controller services to resolve the issue? > > Thanks, > Peter
Re: Custom Controller Service
In case it helps, Matt B was nice enough to quickly put up a PR for
the API change discussed yesterday:
https://github.com/apache/nifi/pull/2658
Once this is tested and merged it make it would make it easier to
implement Charlie's approach without needing a new interface.
On Thu, Apr 26, 2018 at 12:36 PM, Charlie Meyer
<charlie.me...@civitaslearning.com> wrote:
> Hello,
>
> I'm working to see what we can share, but for a start, here is how we
> handle the delegating controller service:
>
> package com.civitaslearning.nifi.service.impl;
>
>
> import org.apache.nifi.components.PropertyDescriptor;
> import org.apache.nifi.controller.AbstractControllerService;
> import org.apache.nifi.dbcp.DBCPService;
>
> import java.util.Collections;
> import java.util.List;
> import java.util.UUID;
>
> public class DelegatingDBCPServiceImpl extends
> AbstractControllerService implements DelegatingDBCPService {
>
> private static final List properties =
> Collections.emptyList();
>
> @Override
> public DBCPService getDBCPService(UUID uuid) {
> DBCPService dbcpService =
> (DBCPService)
> getControllerServiceLookup().getControllerService(uuid.toString());
>
> if (dbcpService == null) {
> throw new NullPointerException("Couldn't find DBCP service with
> ID " + uuid);
> }
>
> return dbcpService;
> }
>
> @Override
> public List getSupportedPropertyDescriptors() {
> return properties;
> }
> }
>
>
> From there, we took existing nifi processor code such as ExecuteSQL and
> modified it to require a reference to a delegating service rather than a
> DBCP service and added another property to specify which flow file
> attribute has the id of the dbcp service that should be used, similar to:
>
> final DelegatingDBCPService delegatingDBCPService =
> context.getProperty(DELEGATING_DBCP_SERVICE).asControllerService(DelegatingDBCPService.class);
> final UUID dbcpServiceId =
> UUID.fromString(context.getProperty(DBCP_SERVICE_ATTRIBUTE).evaluateAttributeExpressions(fileToProcess).getValue());
> final DBCPService dbcpService =
> delegatingDBCPService.getDBCPService(dbcpServiceId);
>
> thanks
>
> On Thu, Apr 26, 2018 at 4:07 AM, RP <rishabprasad...@gmail.com> wrote:
>
>> Hi Charlie,
>>
>> Thanks for the reply. It seems that you already have a solution to my
>> problem. If possible, can you please share you code for the custom
>> controller service and the custom processor?
>>
>>
>>
>> -
>> RP
>> --
>> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
>>
Re: Custom Controller Service
Hello,
I'm working to see what we can share, but for a start, here is how we
handle the delegating controller service:
package com.civitaslearning.nifi.service.impl;
import org.apache.nifi.components.PropertyDescriptor;
import org.apache.nifi.controller.AbstractControllerService;
import org.apache.nifi.dbcp.DBCPService;
import java.util.Collections;
import java.util.List;
import java.util.UUID;
public class DelegatingDBCPServiceImpl extends
AbstractControllerService implements DelegatingDBCPService {
private static final List properties =
Collections.emptyList();
@Override
public DBCPService getDBCPService(UUID uuid) {
DBCPService dbcpService =
(DBCPService)
getControllerServiceLookup().getControllerService(uuid.toString());
if (dbcpService == null) {
throw new NullPointerException("Couldn't find DBCP service with
ID " + uuid);
}
return dbcpService;
}
@Override
public List getSupportedPropertyDescriptors() {
return properties;
}
}
>From there, we took existing nifi processor code such as ExecuteSQL and
modified it to require a reference to a delegating service rather than a
DBCP service and added another property to specify which flow file
attribute has the id of the dbcp service that should be used, similar to:
final DelegatingDBCPService delegatingDBCPService =
context.getProperty(DELEGATING_DBCP_SERVICE).asControllerService(DelegatingDBCPService.class);
final UUID dbcpServiceId =
UUID.fromString(context.getProperty(DBCP_SERVICE_ATTRIBUTE).evaluateAttributeExpressions(fileToProcess).getValue());
final DBCPService dbcpService =
delegatingDBCPService.getDBCPService(dbcpServiceId);
thanks
On Thu, Apr 26, 2018 at 4:07 AM, RP <rishabprasad...@gmail.com> wrote:
> Hi Charlie,
>
> Thanks for the reply. It seems that you already have a solution to my
> problem. If possible, can you please share you code for the custom
> controller service and the custom processor?
>
>
>
> -
> RP
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
>
Re: Custom Controller Service
Hi Charlie, Thanks for the reply. It seems that you already have a solution to my problem. If possible, can you please share you code for the custom controller service and the custom processor? - RP -- Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: Custom Controller Service
Yes, this was just one idea based on Charlie's solution. I'm not saying that approach solves the original request in this email, I was just saying its another nice idea that could be easily implemented once we make the changes in the JIRA. There can be as many "dynamic" DBCPService implementations as we want, and anyone can implement their own, the key is making the API changes to allow for it. On Wed, Apr 25, 2018 at 12:05 PM, Sivaprasanna <sivaprasanna...@gmail.com> wrote: > Okay.. but two questions: > > >1. We are passing the attribute 'db.id' that means, we'll be using >'UpdateAttribute' to do add that attribute to flowfile? >2. If we are to use 'UpdateAttribute' to set the value for 'db.id', we >need to know before hand, right? > > - > > Sivaprasanna > > On Wed, Apr 25, 2018 at 8:38 PM, Bryan Bende <bbe...@gmail.com> wrote: > >> Charlie, >> >> That is a really nice solution, thanks for sharing. >> >> If we make the API changes in that JIRA I just sent, I could see >> having a new implementation of DBCPService that does something very >> similar. >> >> Basically there could be "DelegatingDBCPService" which still >> implemented the same DBCPService interface but followed a convention >> where it always looked in the attribute map for an attribute called >> "db.id", and then it itself has dynamic properties to define many >> DBCPServices where the property name was the db.id, and it would just >> return a Connection from the one with the given id. >> >> There are definitely other options for how to implement the dynamic >> connection service, but this would be a good one to have. >> >> -Bryan >> >> >> On Wed, Apr 25, 2018 at 10:58 AM, Charlie Meyer >> <charlie.me...@civitaslearning.com> wrote: >> > Chiming in a bit late on this, but we faced this same issue and got >> around >> > it by implementing a custom controller service which acts as a "router" >> to >> > different dbcp services. It exposes a method which given a uuid, returns >> > back the DBCPservice that corresponds with that uuid if it exists using >> > >> > DBCPService dbcpService = >> > (DBCPService) >> > getControllerServiceLookup().getControllerService(uuid); >> > >> > From there, we created processors we needed based on the stock ones which >> > relied on our "router" service rather than a single DBCP. Our custom >> > processors read an attribute from incoming flow files, then send that to >> > the router which returns back the connection pool. >> > >> > On Wed, Apr 25, 2018 at 9:48 AM, Bryan Bende <bbe...@gmail.com> wrote: >> > >> >> Here is a proposal for how to modify the existing API to support both >> >> scenarios: >> >> >> >> https://issues.apache.org/jira/browse/NIFI-5121 >> >> >> >> The scope of that ticket would be to make the interface change, and >> >> then update all of NiFi's DB processors to pass in the attribute map. >> >> >> >> Then a separate effort to provide a new service implementation that >> >> used the attribute map to somehow manage multiple connection pools, or >> >> create connections on the fly, or whatever the desired behavior is. >> >> >> >> On Wed, Apr 25, 2018 at 9:34 AM, Bryan Bende <bbe...@gmail.com> wrote: >> >> > To Otto's question... >> >> > >> >> > For simplicity sake, there is a new implementation of >> >> > DBCPConnectionPool that behind the scenes has two connection pools, >> >> > one for DB A and one for DB B, it doesn't matter how these are >> >> > configured. >> >> > >> >> > Now a flow file comes into the ExecuteSQL and it goes to >> >> > connectionPoolService.getConnection()... >> >> > >> >> > How does it know which DB to return a connection for? >> >> > >> >> > >> >> > On Wed, Apr 25, 2018 at 9:01 AM, Sivaprasanna < >> sivaprasanna...@gmail.com> >> >> wrote: >> >> >> Option 2 and 3 seem to be a probable approach. However creating >> varying >> >> >> number of connections based on *each* flowfile still sounds to be >> >> >> suboptimal. If the requirement still demands to take that road, then >> >> it’s >> >> >> better to do some prep-work.. as in the list of pr
Re: Custom Controller Service
Okay.. but two questions: 1. We are passing the attribute 'db.id' that means, we'll be using 'UpdateAttribute' to do add that attribute to flowfile? 2. If we are to use 'UpdateAttribute' to set the value for 'db.id', we need to know before hand, right? - Sivaprasanna On Wed, Apr 25, 2018 at 8:38 PM, Bryan Bende <bbe...@gmail.com> wrote: > Charlie, > > That is a really nice solution, thanks for sharing. > > If we make the API changes in that JIRA I just sent, I could see > having a new implementation of DBCPService that does something very > similar. > > Basically there could be "DelegatingDBCPService" which still > implemented the same DBCPService interface but followed a convention > where it always looked in the attribute map for an attribute called > "db.id", and then it itself has dynamic properties to define many > DBCPServices where the property name was the db.id, and it would just > return a Connection from the one with the given id. > > There are definitely other options for how to implement the dynamic > connection service, but this would be a good one to have. > > -Bryan > > > On Wed, Apr 25, 2018 at 10:58 AM, Charlie Meyer > <charlie.me...@civitaslearning.com> wrote: > > Chiming in a bit late on this, but we faced this same issue and got > around > > it by implementing a custom controller service which acts as a "router" > to > > different dbcp services. It exposes a method which given a uuid, returns > > back the DBCPservice that corresponds with that uuid if it exists using > > > > DBCPService dbcpService = > > (DBCPService) > > getControllerServiceLookup().getControllerService(uuid); > > > > From there, we created processors we needed based on the stock ones which > > relied on our "router" service rather than a single DBCP. Our custom > > processors read an attribute from incoming flow files, then send that to > > the router which returns back the connection pool. > > > > On Wed, Apr 25, 2018 at 9:48 AM, Bryan Bende <bbe...@gmail.com> wrote: > > > >> Here is a proposal for how to modify the existing API to support both > >> scenarios: > >> > >> https://issues.apache.org/jira/browse/NIFI-5121 > >> > >> The scope of that ticket would be to make the interface change, and > >> then update all of NiFi's DB processors to pass in the attribute map. > >> > >> Then a separate effort to provide a new service implementation that > >> used the attribute map to somehow manage multiple connection pools, or > >> create connections on the fly, or whatever the desired behavior is. > >> > >> On Wed, Apr 25, 2018 at 9:34 AM, Bryan Bende <bbe...@gmail.com> wrote: > >> > To Otto's question... > >> > > >> > For simplicity sake, there is a new implementation of > >> > DBCPConnectionPool that behind the scenes has two connection pools, > >> > one for DB A and one for DB B, it doesn't matter how these are > >> > configured. > >> > > >> > Now a flow file comes into the ExecuteSQL and it goes to > >> > connectionPoolService.getConnection()... > >> > > >> > How does it know which DB to return a connection for? > >> > > >> > > >> > On Wed, Apr 25, 2018 at 9:01 AM, Sivaprasanna < > sivaprasanna...@gmail.com> > >> wrote: > >> >> Option 2 and 3 seem to be a probable approach. However creating > varying > >> >> number of connections based on *each* flowfile still sounds to be > >> >> suboptimal. If the requirement still demands to take that road, then > >> it’s > >> >> better to do some prep-work.. as in the list of probable connections > >> that > >> >> are required are taken and connection pools are created for them and > >> then > >> >> based on the flowfiles (which connection it needs), we use the > relevant > >> >> one. > >> >> > >> >> Thanks, > >> >> Sivaprasanna > >> >> > >> >> On Wed, 25 Apr 2018 at 6:07 PM, Bryan Bende <bbe...@gmail.com> > wrote: > >> >> > >> >>> The issue here is more about the service API and not the > >> implementations. > >> >>> > >> >>> The current API has no way to pass information between the processor > >> and > >> >>> service. > >> >>> > >> >>> The options boil down to... >
Re: Custom Controller Service
Charlie, That is a really nice solution, thanks for sharing. If we make the API changes in that JIRA I just sent, I could see having a new implementation of DBCPService that does something very similar. Basically there could be "DelegatingDBCPService" which still implemented the same DBCPService interface but followed a convention where it always looked in the attribute map for an attribute called "db.id", and then it itself has dynamic properties to define many DBCPServices where the property name was the db.id, and it would just return a Connection from the one with the given id. There are definitely other options for how to implement the dynamic connection service, but this would be a good one to have. -Bryan On Wed, Apr 25, 2018 at 10:58 AM, Charlie Meyer <charlie.me...@civitaslearning.com> wrote: > Chiming in a bit late on this, but we faced this same issue and got around > it by implementing a custom controller service which acts as a "router" to > different dbcp services. It exposes a method which given a uuid, returns > back the DBCPservice that corresponds with that uuid if it exists using > > DBCPService dbcpService = > (DBCPService) > getControllerServiceLookup().getControllerService(uuid); > > From there, we created processors we needed based on the stock ones which > relied on our "router" service rather than a single DBCP. Our custom > processors read an attribute from incoming flow files, then send that to > the router which returns back the connection pool. > > On Wed, Apr 25, 2018 at 9:48 AM, Bryan Bende <bbe...@gmail.com> wrote: > >> Here is a proposal for how to modify the existing API to support both >> scenarios: >> >> https://issues.apache.org/jira/browse/NIFI-5121 >> >> The scope of that ticket would be to make the interface change, and >> then update all of NiFi's DB processors to pass in the attribute map. >> >> Then a separate effort to provide a new service implementation that >> used the attribute map to somehow manage multiple connection pools, or >> create connections on the fly, or whatever the desired behavior is. >> >> On Wed, Apr 25, 2018 at 9:34 AM, Bryan Bende <bbe...@gmail.com> wrote: >> > To Otto's question... >> > >> > For simplicity sake, there is a new implementation of >> > DBCPConnectionPool that behind the scenes has two connection pools, >> > one for DB A and one for DB B, it doesn't matter how these are >> > configured. >> > >> > Now a flow file comes into the ExecuteSQL and it goes to >> > connectionPoolService.getConnection()... >> > >> > How does it know which DB to return a connection for? >> > >> > >> > On Wed, Apr 25, 2018 at 9:01 AM, Sivaprasanna <sivaprasanna...@gmail.com> >> wrote: >> >> Option 2 and 3 seem to be a probable approach. However creating varying >> >> number of connections based on *each* flowfile still sounds to be >> >> suboptimal. If the requirement still demands to take that road, then >> it’s >> >> better to do some prep-work.. as in the list of probable connections >> that >> >> are required are taken and connection pools are created for them and >> then >> >> based on the flowfiles (which connection it needs), we use the relevant >> >> one. >> >> >> >> Thanks, >> >> Sivaprasanna >> >> >> >> On Wed, 25 Apr 2018 at 6:07 PM, Bryan Bende <bbe...@gmail.com> wrote: >> >> >> >>> The issue here is more about the service API and not the >> implementations. >> >>> >> >>> The current API has no way to pass information between the processor >> and >> >>> service. >> >>> >> >>> The options boil down to... >> >>> >> >>> - Make a new API, but then you need all new processors that use the >> new API >> >>> >> >>> - Modify the current API to have a new method, but then we are combing >> two >> >>> concepts into one API and some impls may not implement both >> >>> >> >>> - Modify the processors to use two different service APIs, but enforce >> that >> >>> only one can be used at a time, so it can have either the original >> >>> connection pool service or can have some new dynamic connection >> factory, >> >>> but not both, and then modify all DB processors to have logic to >> determine >> >>> which service to use. >> >>> >
Re: Custom Controller Service
Chiming in a bit late on this, but we faced this same issue and got around it by implementing a custom controller service which acts as a "router" to different dbcp services. It exposes a method which given a uuid, returns back the DBCPservice that corresponds with that uuid if it exists using DBCPService dbcpService = (DBCPService) getControllerServiceLookup().getControllerService(uuid); >From there, we created processors we needed based on the stock ones which relied on our "router" service rather than a single DBCP. Our custom processors read an attribute from incoming flow files, then send that to the router which returns back the connection pool. On Wed, Apr 25, 2018 at 9:48 AM, Bryan Bende <bbe...@gmail.com> wrote: > Here is a proposal for how to modify the existing API to support both > scenarios: > > https://issues.apache.org/jira/browse/NIFI-5121 > > The scope of that ticket would be to make the interface change, and > then update all of NiFi's DB processors to pass in the attribute map. > > Then a separate effort to provide a new service implementation that > used the attribute map to somehow manage multiple connection pools, or > create connections on the fly, or whatever the desired behavior is. > > On Wed, Apr 25, 2018 at 9:34 AM, Bryan Bende <bbe...@gmail.com> wrote: > > To Otto's question... > > > > For simplicity sake, there is a new implementation of > > DBCPConnectionPool that behind the scenes has two connection pools, > > one for DB A and one for DB B, it doesn't matter how these are > > configured. > > > > Now a flow file comes into the ExecuteSQL and it goes to > > connectionPoolService.getConnection()... > > > > How does it know which DB to return a connection for? > > > > > > On Wed, Apr 25, 2018 at 9:01 AM, Sivaprasanna <sivaprasanna...@gmail.com> > wrote: > >> Option 2 and 3 seem to be a probable approach. However creating varying > >> number of connections based on *each* flowfile still sounds to be > >> suboptimal. If the requirement still demands to take that road, then > it’s > >> better to do some prep-work.. as in the list of probable connections > that > >> are required are taken and connection pools are created for them and > then > >> based on the flowfiles (which connection it needs), we use the relevant > >> one. > >> > >> Thanks, > >> Sivaprasanna > >> > >> On Wed, 25 Apr 2018 at 6:07 PM, Bryan Bende <bbe...@gmail.com> wrote: > >> > >>> The issue here is more about the service API and not the > implementations. > >>> > >>> The current API has no way to pass information between the processor > and > >>> service. > >>> > >>> The options boil down to... > >>> > >>> - Make a new API, but then you need all new processors that use the > new API > >>> > >>> - Modify the current API to have a new method, but then we are combing > two > >>> concepts into one API and some impls may not implement both > >>> > >>> - Modify the processors to use two different service APIs, but enforce > that > >>> only one can be used at a time, so it can have either the original > >>> connection pool service or can have some new dynamic connection > factory, > >>> but not both, and then modify all DB processors to have logic to > determine > >>> which service to use. > >>> > >>> On Wed, Apr 25, 2018 at 8:28 AM Otto Fowler <ottobackwa...@gmail.com> > >>> wrote: > >>> > >>> > Or you could just call every time you needed properties more likely. > >>> > This would still be custom unless integrated…. > >>> > > >>> > > >>> > On April 25, 2018 at 08:26:57, Otto Fowler (ottobackwa...@gmail.com) > >>> > wrote: > >>> > > >>> > Can services work with other controller services? > >>> > Maybe a PropertiesControllerService, FilePropertiesControllerService > >>> could > >>> > work with your service? > >>> > the PCS could fire events on property changes etc. > >>> > > >>> > > >>> > > >>> > On April 25, 2018 at 08:05:27, Mike Thomsen (mikerthom...@gmail.com) > >>> > wrote: > >>> > > >>> > Shot in the dark here, but what you try to do is create a custom > >>> connection > >>> > pool service that uses dynamic properties to build
Re: Custom Controller Service
Here is a proposal for how to modify the existing API to support both scenarios: https://issues.apache.org/jira/browse/NIFI-5121 The scope of that ticket would be to make the interface change, and then update all of NiFi's DB processors to pass in the attribute map. Then a separate effort to provide a new service implementation that used the attribute map to somehow manage multiple connection pools, or create connections on the fly, or whatever the desired behavior is. On Wed, Apr 25, 2018 at 9:34 AM, Bryan Bende <bbe...@gmail.com> wrote: > To Otto's question... > > For simplicity sake, there is a new implementation of > DBCPConnectionPool that behind the scenes has two connection pools, > one for DB A and one for DB B, it doesn't matter how these are > configured. > > Now a flow file comes into the ExecuteSQL and it goes to > connectionPoolService.getConnection()... > > How does it know which DB to return a connection for? > > > On Wed, Apr 25, 2018 at 9:01 AM, Sivaprasanna <sivaprasanna...@gmail.com> > wrote: >> Option 2 and 3 seem to be a probable approach. However creating varying >> number of connections based on *each* flowfile still sounds to be >> suboptimal. If the requirement still demands to take that road, then it’s >> better to do some prep-work.. as in the list of probable connections that >> are required are taken and connection pools are created for them and then >> based on the flowfiles (which connection it needs), we use the relevant >> one. >> >> Thanks, >> Sivaprasanna >> >> On Wed, 25 Apr 2018 at 6:07 PM, Bryan Bende <bbe...@gmail.com> wrote: >> >>> The issue here is more about the service API and not the implementations. >>> >>> The current API has no way to pass information between the processor and >>> service. >>> >>> The options boil down to... >>> >>> - Make a new API, but then you need all new processors that use the new API >>> >>> - Modify the current API to have a new method, but then we are combing two >>> concepts into one API and some impls may not implement both >>> >>> - Modify the processors to use two different service APIs, but enforce that >>> only one can be used at a time, so it can have either the original >>> connection pool service or can have some new dynamic connection factory, >>> but not both, and then modify all DB processors to have logic to determine >>> which service to use. >>> >>> On Wed, Apr 25, 2018 at 8:28 AM Otto Fowler <ottobackwa...@gmail.com> >>> wrote: >>> >>> > Or you could just call every time you needed properties more likely. >>> > This would still be custom unless integrated…. >>> > >>> > >>> > On April 25, 2018 at 08:26:57, Otto Fowler (ottobackwa...@gmail.com) >>> > wrote: >>> > >>> > Can services work with other controller services? >>> > Maybe a PropertiesControllerService, FilePropertiesControllerService >>> could >>> > work with your service? >>> > the PCS could fire events on property changes etc. >>> > >>> > >>> > >>> > On April 25, 2018 at 08:05:27, Mike Thomsen (mikerthom...@gmail.com) >>> > wrote: >>> > >>> > Shot in the dark here, but what you try to do is create a custom >>> connection >>> > pool service that uses dynamic properties to build a "pool of connection >>> > pools." You could then use the property names as hints for where to send >>> > the queries. >>> > >>> > On Wed, Apr 25, 2018 at 6:19 AM Rishab Prasad <rishabprasad...@gmail.com >>> > >>> > wrote: >>> > >>> > > Hi, >>> > > >>> > > Basically, there are 'n' number of databases that we are dealing with. >>> We >>> > > need to fetch the data from the source database into HDFS. Now since we >>> > are >>> > > dealing with many databases, the source database is not static and >>> > changes >>> > > every now and then. And every time the source database changes we >>> > manually >>> > > need to change the value for the connection parameters in >>> > > DBCPConnectionPool. Now, people suggest that for 'n' databases create >>> 'n' >>> > > connections for each database, but that is not possible because 'n' is >>> a >>> > > big number and creating that many connections in DBCPConnec
Re: Custom Controller Service
To Otto's question... For simplicity sake, there is a new implementation of DBCPConnectionPool that behind the scenes has two connection pools, one for DB A and one for DB B, it doesn't matter how these are configured. Now a flow file comes into the ExecuteSQL and it goes to connectionPoolService.getConnection()... How does it know which DB to return a connection for? On Wed, Apr 25, 2018 at 9:01 AM, Sivaprasanna <sivaprasanna...@gmail.com> wrote: > Option 2 and 3 seem to be a probable approach. However creating varying > number of connections based on *each* flowfile still sounds to be > suboptimal. If the requirement still demands to take that road, then it’s > better to do some prep-work.. as in the list of probable connections that > are required are taken and connection pools are created for them and then > based on the flowfiles (which connection it needs), we use the relevant > one. > > Thanks, > Sivaprasanna > > On Wed, 25 Apr 2018 at 6:07 PM, Bryan Bende <bbe...@gmail.com> wrote: > >> The issue here is more about the service API and not the implementations. >> >> The current API has no way to pass information between the processor and >> service. >> >> The options boil down to... >> >> - Make a new API, but then you need all new processors that use the new API >> >> - Modify the current API to have a new method, but then we are combing two >> concepts into one API and some impls may not implement both >> >> - Modify the processors to use two different service APIs, but enforce that >> only one can be used at a time, so it can have either the original >> connection pool service or can have some new dynamic connection factory, >> but not both, and then modify all DB processors to have logic to determine >> which service to use. >> >> On Wed, Apr 25, 2018 at 8:28 AM Otto Fowler <ottobackwa...@gmail.com> >> wrote: >> >> > Or you could just call every time you needed properties more likely. >> > This would still be custom unless integrated…. >> > >> > >> > On April 25, 2018 at 08:26:57, Otto Fowler (ottobackwa...@gmail.com) >> > wrote: >> > >> > Can services work with other controller services? >> > Maybe a PropertiesControllerService, FilePropertiesControllerService >> could >> > work with your service? >> > the PCS could fire events on property changes etc. >> > >> > >> > >> > On April 25, 2018 at 08:05:27, Mike Thomsen (mikerthom...@gmail.com) >> > wrote: >> > >> > Shot in the dark here, but what you try to do is create a custom >> connection >> > pool service that uses dynamic properties to build a "pool of connection >> > pools." You could then use the property names as hints for where to send >> > the queries. >> > >> > On Wed, Apr 25, 2018 at 6:19 AM Rishab Prasad <rishabprasad...@gmail.com >> > >> > wrote: >> > >> > > Hi, >> > > >> > > Basically, there are 'n' number of databases that we are dealing with. >> We >> > > need to fetch the data from the source database into HDFS. Now since we >> > are >> > > dealing with many databases, the source database is not static and >> > changes >> > > every now and then. And every time the source database changes we >> > manually >> > > need to change the value for the connection parameters in >> > > DBCPConnectionPool. Now, people suggest that for 'n' databases create >> 'n' >> > > connections for each database, but that is not possible because 'n' is >> a >> > > big number and creating that many connections in DBCPConnectionPool is >> > not >> > > possible. So we were looking for a way where we can specify all the >> > > connection parameters in a file present in our local system and then >> make >> > > the DBCPConnectionPool controller service to read the values from the >> > file. >> > > In that way we can simply change the value in the file present in the >> > local >> > > system. No need to alter anything in the dataflow. But it turns out >> that >> > > FlowFile attributes are not available to the controller services as the >> > > expression language is evaluated at the time of service enable. >> > > >> > > So can you suggest a way where I can achieve my requirement (except >> > > 'variable.registry' ) ? I am looking to develop a custom controller >> > service >> > > that can serve the requirement but how do I make the flowfile >> attributes >> > > available to the service? >> > > >> > >> -- >> Sent from Gmail Mobile >>
Re: Custom Controller Service
Option 2 and 3 seem to be a probable approach. However creating varying number of connections based on *each* flowfile still sounds to be suboptimal. If the requirement still demands to take that road, then it’s better to do some prep-work.. as in the list of probable connections that are required are taken and connection pools are created for them and then based on the flowfiles (which connection it needs), we use the relevant one. Thanks, Sivaprasanna On Wed, 25 Apr 2018 at 6:07 PM, Bryan Bende <bbe...@gmail.com> wrote: > The issue here is more about the service API and not the implementations. > > The current API has no way to pass information between the processor and > service. > > The options boil down to... > > - Make a new API, but then you need all new processors that use the new API > > - Modify the current API to have a new method, but then we are combing two > concepts into one API and some impls may not implement both > > - Modify the processors to use two different service APIs, but enforce that > only one can be used at a time, so it can have either the original > connection pool service or can have some new dynamic connection factory, > but not both, and then modify all DB processors to have logic to determine > which service to use. > > On Wed, Apr 25, 2018 at 8:28 AM Otto Fowler <ottobackwa...@gmail.com> > wrote: > > > Or you could just call every time you needed properties more likely. > > This would still be custom unless integrated…. > > > > > > On April 25, 2018 at 08:26:57, Otto Fowler (ottobackwa...@gmail.com) > > wrote: > > > > Can services work with other controller services? > > Maybe a PropertiesControllerService, FilePropertiesControllerService > could > > work with your service? > > the PCS could fire events on property changes etc. > > > > > > > > On April 25, 2018 at 08:05:27, Mike Thomsen (mikerthom...@gmail.com) > > wrote: > > > > Shot in the dark here, but what you try to do is create a custom > connection > > pool service that uses dynamic properties to build a "pool of connection > > pools." You could then use the property names as hints for where to send > > the queries. > > > > On Wed, Apr 25, 2018 at 6:19 AM Rishab Prasad <rishabprasad...@gmail.com > > > > wrote: > > > > > Hi, > > > > > > Basically, there are 'n' number of databases that we are dealing with. > We > > > need to fetch the data from the source database into HDFS. Now since we > > are > > > dealing with many databases, the source database is not static and > > changes > > > every now and then. And every time the source database changes we > > manually > > > need to change the value for the connection parameters in > > > DBCPConnectionPool. Now, people suggest that for 'n' databases create > 'n' > > > connections for each database, but that is not possible because 'n' is > a > > > big number and creating that many connections in DBCPConnectionPool is > > not > > > possible. So we were looking for a way where we can specify all the > > > connection parameters in a file present in our local system and then > make > > > the DBCPConnectionPool controller service to read the values from the > > file. > > > In that way we can simply change the value in the file present in the > > local > > > system. No need to alter anything in the dataflow. But it turns out > that > > > FlowFile attributes are not available to the controller services as the > > > expression language is evaluated at the time of service enable. > > > > > > So can you suggest a way where I can achieve my requirement (except > > > 'variable.registry' ) ? I am looking to develop a custom controller > > service > > > that can serve the requirement but how do I make the flowfile > attributes > > > available to the service? > > > > > > -- > Sent from Gmail Mobile >
Re: Custom Controller Service
If any controller service optionally supported this external service ( like the AWS processors optional support the credentials service ) then there is no need for the processor to change though right? On April 25, 2018 at 08:37:50, Bryan Bende (bbe...@gmail.com) wrote: The issue here is more about the service API and not the implementations. The current API has no way to pass information between the processor and service. The options boil down to... - Make a new API, but then you need all new processors that use the new API - Modify the current API to have a new method, but then we are combing two concepts into one API and some impls may not implement both - Modify the processors to use two different service APIs, but enforce that only one can be used at a time, so it can have either the original connection pool service or can have some new dynamic connection factory, but not both, and then modify all DB processors to have logic to determine which service to use. On Wed, Apr 25, 2018 at 8:28 AM Otto Fowler <ottobackwa...@gmail.com> wrote: > Or you could just call every time you needed properties more likely. > This would still be custom unless integrated…. > > > On April 25, 2018 at 08:26:57, Otto Fowler (ottobackwa...@gmail.com) > wrote: > > Can services work with other controller services? > Maybe a PropertiesControllerService, FilePropertiesControllerService could > work with your service? > the PCS could fire events on property changes etc. > > > > On April 25, 2018 at 08:05:27, Mike Thomsen (mikerthom...@gmail.com) > wrote: > > Shot in the dark here, but what you try to do is create a custom connection > pool service that uses dynamic properties to build a "pool of connection > pools." You could then use the property names as hints for where to send > the queries. > > On Wed, Apr 25, 2018 at 6:19 AM Rishab Prasad <rishabprasad...@gmail.com> > wrote: > > > Hi, > > > > Basically, there are 'n' number of databases that we are dealing with. We > > need to fetch the data from the source database into HDFS. Now since we > are > > dealing with many databases, the source database is not static and > changes > > every now and then. And every time the source database changes we > manually > > need to change the value for the connection parameters in > > DBCPConnectionPool. Now, people suggest that for 'n' databases create 'n' > > connections for each database, but that is not possible because 'n' is a > > big number and creating that many connections in DBCPConnectionPool is > not > > possible. So we were looking for a way where we can specify all the > > connection parameters in a file present in our local system and then make > > the DBCPConnectionPool controller service to read the values from the > file. > > In that way we can simply change the value in the file present in the > local > > system. No need to alter anything in the dataflow. But it turns out that > > FlowFile attributes are not available to the controller services as the > > expression language is evaluated at the time of service enable. > > > > So can you suggest a way where I can achieve my requirement (except > > 'variable.registry' ) ? I am looking to develop a custom controller > service > > that can serve the requirement but how do I make the flowfile attributes > > available to the service? > > > -- Sent from Gmail Mobile
Re: Custom Controller Service
Or you could just call every time you needed properties more likely. This would still be custom unless integrated…. On April 25, 2018 at 08:26:57, Otto Fowler (ottobackwa...@gmail.com) wrote: Can services work with other controller services? Maybe a PropertiesControllerService, FilePropertiesControllerService could work with your service? the PCS could fire events on property changes etc. On April 25, 2018 at 08:05:27, Mike Thomsen (mikerthom...@gmail.com) wrote: Shot in the dark here, but what you try to do is create a custom connection pool service that uses dynamic properties to build a "pool of connection pools." You could then use the property names as hints for where to send the queries. On Wed, Apr 25, 2018 at 6:19 AM Rishab Prasad <rishabprasad...@gmail.com> wrote: > Hi, > > Basically, there are 'n' number of databases that we are dealing with. We > need to fetch the data from the source database into HDFS. Now since we are > dealing with many databases, the source database is not static and changes > every now and then. And every time the source database changes we manually > need to change the value for the connection parameters in > DBCPConnectionPool. Now, people suggest that for 'n' databases create 'n' > connections for each database, but that is not possible because 'n' is a > big number and creating that many connections in DBCPConnectionPool is not > possible. So we were looking for a way where we can specify all the > connection parameters in a file present in our local system and then make > the DBCPConnectionPool controller service to read the values from the file. > In that way we can simply change the value in the file present in the local > system. No need to alter anything in the dataflow. But it turns out that > FlowFile attributes are not available to the controller services as the > expression language is evaluated at the time of service enable. > > So can you suggest a way where I can achieve my requirement (except > 'variable.registry' ) ? I am looking to develop a custom controller service > that can serve the requirement but how do I make the flowfile attributes > available to the service? >
Re: Custom Controller Service
Can services work with other controller services? Maybe a PropertiesControllerService, FilePropertiesControllerService could work with your service? the PCS could fire events on property changes etc. On April 25, 2018 at 08:05:27, Mike Thomsen (mikerthom...@gmail.com) wrote: Shot in the dark here, but what you try to do is create a custom connection pool service that uses dynamic properties to build a "pool of connection pools." You could then use the property names as hints for where to send the queries. On Wed, Apr 25, 2018 at 6:19 AM Rishab Prasad <rishabprasad...@gmail.com> wrote: > Hi, > > Basically, there are 'n' number of databases that we are dealing with. We > need to fetch the data from the source database into HDFS. Now since we are > dealing with many databases, the source database is not static and changes > every now and then. And every time the source database changes we manually > need to change the value for the connection parameters in > DBCPConnectionPool. Now, people suggest that for 'n' databases create 'n' > connections for each database, but that is not possible because 'n' is a > big number and creating that many connections in DBCPConnectionPool is not > possible. So we were looking for a way where we can specify all the > connection parameters in a file present in our local system and then make > the DBCPConnectionPool controller service to read the values from the file. > In that way we can simply change the value in the file present in the local > system. No need to alter anything in the dataflow. But it turns out that > FlowFile attributes are not available to the controller services as the > expression language is evaluated at the time of service enable. > > So can you suggest a way where I can achieve my requirement (except > 'variable.registry' ) ? I am looking to develop a custom controller service > that can serve the requirement but how do I make the flowfile attributes > available to the service? >
Re: Custom Controller Service
Shot in the dark here, but what you try to do is create a custom connection pool service that uses dynamic properties to build a "pool of connection pools." You could then use the property names as hints for where to send the queries. On Wed, Apr 25, 2018 at 6:19 AM Rishab Prasad <rishabprasad...@gmail.com> wrote: > Hi, > > Basically, there are 'n' number of databases that we are dealing with. We > need to fetch the data from the source database into HDFS. Now since we are > dealing with many databases, the source database is not static and changes > every now and then. And every time the source database changes we manually > need to change the value for the connection parameters in > DBCPConnectionPool. Now, people suggest that for 'n' databases create 'n' > connections for each database, but that is not possible because 'n' is a > big number and creating that many connections in DBCPConnectionPool is not > possible. So we were looking for a way where we can specify all the > connection parameters in a file present in our local system and then make > the DBCPConnectionPool controller service to read the values from the file. > In that way we can simply change the value in the file present in the local > system. No need to alter anything in the dataflow. But it turns out that > FlowFile attributes are not available to the controller services as the > expression language is evaluated at the time of service enable. > > So can you suggest a way where I can achieve my requirement (except > 'variable.registry' ) ? I am looking to develop a custom controller service > that can serve the requirement but how do I make the flowfile attributes > available to the service? >
Re: Custom Controller Service
Hello, Others who have worked on the DB related services and processors can correct me if I'm wrong here, but... In general the idea of a connection pool is that creating connections is somewhat expensive, and for a high-volume of operations you don't want to create a connection for each DB operation, so the connection pool creates some number of connections ahead of time and makes them re-usable across operations, thus being more efficient. In your case you want dynamically created connections based on flow file attributes, which means potentially the connection information could be different for each flow file. At that point it starts to feel like it isn't really a connection pool and is just a factory to obtain a one-time use connection because otherwise would end-up needing to obtain multiple connection pools behind the scenes. A controller service has an API and then implementations of the API, and the API just a Java interface. The Java interface (API) is the contract with processors... a processor gets a reference to an object that implements the interface, and the processor can call methods on the interface. So if you want to pass information from flow files to a controller service, then the methods in the interface need to somehow accept that information. The DBCPConnectionPool interface [1] is just a single method "getConnection()" and it is designed to be a re-usable pool of connections against a single DB (as I described in the first paragraph). You could define your own "DynamicDBCPConnectionPool" API with a method like "getConnection(String dbIdentifier)" and have an implementation that loaded connection information from properties file and kept a lookup from dbIdentifier to connection info, but then you also need your own set DB processor because none of the existing DB processors work against your new API. Hope this helps. -Bryan [1] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-dbcp-service-api/src/main/java/org/apache/nifi/dbcp/DBCPService.java#L33 On Wed, Apr 25, 2018 at 3:24 AM, Rishab Prasad <rishabprasad...@gmail.com> wrote: > Hi, > > Basically, there are 'n' number of databases that we are dealing with. We > need to fetch the data from the source database into HDFS. Now since we are > dealing with many databases, the source database is not static and changes > every now and then. And every time the source database changes we manually > need to change the value for the connection parameters in > DBCPConnectionPool. Now, people suggest that for 'n' databases create 'n' > connections for each database, but that is not possible because 'n' is a > big number and creating that many connections in DBCPConnectionPool is not > possible. So we were looking for a way where we can specify all the > connection parameters in a file present in our local system and then make > the DBCPConnectionPool controller service to read the values from the file. > In that way we can simply change the value in the file present in the local > system. No need to alter anything in the dataflow. But it turns out that > FlowFile attributes are not available to the controller services as the > expression language is evaluated at the time of service enable. > > So can you suggest a way where I can achieve my requirement (except > 'variable.registry' ) ? I am looking to develop a custom controller service > that can serve the requirement but how do I make the flowfile attributes > available to the service?
Custom Controller Service
Hi, Basically, there are 'n' number of databases that we are dealing with. We need to fetch the data from the source database into HDFS. Now since we are dealing with many databases, the source database is not static and changes every now and then. And every time the source database changes we manually need to change the value for the connection parameters in DBCPConnectionPool. Now, people suggest that for 'n' databases create 'n' connections for each database, but that is not possible because 'n' is a big number and creating that many connections in DBCPConnectionPool is not possible. So we were looking for a way where we can specify all the connection parameters in a file present in our local system and then make the DBCPConnectionPool controller service to read the values from the file. In that way we can simply change the value in the file present in the local system. No need to alter anything in the dataflow. But it turns out that FlowFile attributes are not available to the controller services as the expression language is evaluated at the time of service enable. So can you suggest a way where I can achieve my requirement (except 'variable.registry' ) ? I am looking to develop a custom controller service that can serve the requirement but how do I make the flowfile attributes available to the service?
Re: Custom controller service
Varun, You didn't say what type of database is it. Can you share that info? On Thu, Mar 8, 2018 at 1:15 AM, varun yadav <varunyadav1...@gmail.com> wrote: > Hii, > > I am trying to create a custom controller service for connection to a local > database. > I am getting so many issues in that. If you can help with a few documents > or videos, it will be very helpful for me to proceed. > > Thanks Varun >
Re: Custom controller service
Varun, Bryan Bende has a github repo with examples[1] as well as nifi.rocks[2] has an example. Thanks, Chad 1. https://github.com/bbende/nifi-dependency-example 2. http://nifi.rocks/developing-a-custom-apache-nifi-controller-service/ On Thu, Mar 8, 2018 at 1:47 AM Sivaprasanna <sivaprasanna...@gmail.com> wrote: > Hey Varun, > > It would br helpful if you could share the error stacktrace. By the way, > you can use DBCP Connection Pool for connecting to databases. > > On Thu, 8 Mar 2018 at 12:14 PM, varun yadav <varunyadav1...@gmail.com> > wrote: > > > Hii, > > > > I am trying to create a custom controller service for connection to a > local > > database. > > I am getting so many issues in that. If you can help with a few documents > > or videos, it will be very helpful for me to proceed. > > > > Thanks Varun > > >
Re: Custom controller service
Hey Varun, It would br helpful if you could share the error stacktrace. By the way, you can use DBCP Connection Pool for connecting to databases. On Thu, 8 Mar 2018 at 12:14 PM, varun yadav <varunyadav1...@gmail.com> wrote: > Hii, > > I am trying to create a custom controller service for connection to a local > database. > I am getting so many issues in that. If you can help with a few documents > or videos, it will be very helpful for me to proceed. > > Thanks Varun >
Custom controller service
Hii, I am trying to create a custom controller service for connection to a local database. I am getting so many issues in that. If you can help with a few documents or videos, it will be very helpful for me to proceed. Thanks Varun
Re: Custom Controller Service Debugging
Matt and Bryan, Thanks for the detailed explanation. The class loading aspect makes perfect sense, and yes, importing as disabled is understood as well. I do think I've come across a small bug with automatically creating Controller Services on template imports though. NiFi is not automatically creating Controller Services for Processors inside of Process Groups on template imports. Can one of you check me on this before we write up a Jira ticket? Steps to reproduce: 1. Start fresh (clean canvas, no controller services) 2. Create Process Group and go into it 3. Add DetectDuplicate processor to Process Group 4. Configure DetectDuplicate processor with a new DistributedMapCacheClientService and enable that service so that the DetectDuplicate processor is valid and ready to run 5. Navigate to root canvas (see the Process Group) 6. Create template from root canvas (name: Controller Service Test) 7. Delete everything from the canvas and disable + delete the controller service so that we're essentially back to the start 8. Drag and drop the Controller Service Test template onto the canvas (expecting that everything will be auto-created) 9. Navigate into the Process Group and notice that the DetectDuplicate processor is invalid because the DistributedMapCacheClientService ID cannot be found As an additional test, I created a template with a processor at the top-level and one inside the Process Group that are using the same Controller Service, and that works just fine. My guess is that the top-level processor is used to create the Controller Service and the inner-processor picks it up by that same ID. I can provide a sample template in a Jira ticket if that would help. Thanks, Brian On Fri, Aug 14, 2015 at 8:24 AM, Bryan Bende bbe...@gmail.com wrote: Brian, Glad it worked! I think it was just a combination of things... I had previously spent a lot of time studying the existing dependency setup in nifi-standard-bundle, as well as the dev guide section on NARs [1]. Also, someone else had ran into a similar problem recently and it turned out to be a dependency problem as well. There should be a Maven archetype for Controller Services [2] coming soon which might help nail down the correct dependency setup, also after seeing your email last night I put together a bare-bones project [3] that has a custom processor using a custom controller service (both created using the archetypes) to serve as an example. -Bryan [1] https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#nars [2] https://issues.apache.org/jira/browse/NIFI-736 [3] https://github.com/bbende/nifi-dependency-example On Fri, Aug 14, 2015 at 8:17 AM, Matt Gilman matt.c.gil...@gmail.com wrote: Brian, The NAR dependency stuff all breaks down to providing classpath isolation and not polluting the classpath of downstream NARs with dependencies required for Controller Service implementation. We try to keep Controller Service interfaces lean by not depending on external libs in the interface (generally a good practice anyways). Both the implementation of the Controller Service and the any Processor NAR that want to consume the Controller Service needs to depend on the Controller Service API NAR (where the interface lives). This established a common parent ClassLoader which allows the service to be access across NARs. If your building Processor NARs that depend on custom Controller Services and standard Controller Services, you'll want to have your custom Controller Services API NAR depend on the standard Controller Services API NAR. The NAR dependency tree (if you will) maps to a ClassLoader hierarchy within NiFi. Controller Services being added in a disabled state is remaining consistent with Processors (and other components) in template. Also, because sensitive properties are not included in templates, they may require the user to complete the configuration prior to enabling. Matt On Fri, Aug 14, 2015 at 8:01 AM, Brian Ghigiarelli briang...@gmail.com wrote: Bryan, Thank you!! Adding a NAR dependency to my standard-processors NAR did the trick to get NiFi to recognize the service for these processors and auto-create a new instance of the Controller Service on template import. I owe ya a beer or six the next time our paths cross. How did you figure out the required NAR dependency? Any great debugging tricks to share, or was it your existing depth of NiFi knowledge that led you to that solution? As a follow-on to that, I'm seeing that Controller Services created via Template instantiation are created in Disabled state. Is that expected? Is there any way to drag-and-drop a template with controller services onto a canvas and have the newly created controller services enabled? Thanks again, Brian On Thu, Aug 13, 2015 at 5:57 PM, Bryan Bende bbe...@gmail.com wrote: Brian, I just ran
Re: Custom Controller Service Debugging
Brian, I see the same behavior regarding Controller Service creation when referenced by Processor's in sub Groups. Please file the JIRA ticket and it will be addressed in our release. Thanks! Matt On Fri, Aug 14, 2015 at 11:30 AM, Brian Ghigiarelli briang...@gmail.com wrote: Matt and Bryan, Thanks for the detailed explanation. The class loading aspect makes perfect sense, and yes, importing as disabled is understood as well. I do think I've come across a small bug with automatically creating Controller Services on template imports though. NiFi is not automatically creating Controller Services for Processors inside of Process Groups on template imports. Can one of you check me on this before we write up a Jira ticket? Steps to reproduce: 1. Start fresh (clean canvas, no controller services) 2. Create Process Group and go into it 3. Add DetectDuplicate processor to Process Group 4. Configure DetectDuplicate processor with a new DistributedMapCacheClientService and enable that service so that the DetectDuplicate processor is valid and ready to run 5. Navigate to root canvas (see the Process Group) 6. Create template from root canvas (name: Controller Service Test) 7. Delete everything from the canvas and disable + delete the controller service so that we're essentially back to the start 8. Drag and drop the Controller Service Test template onto the canvas (expecting that everything will be auto-created) 9. Navigate into the Process Group and notice that the DetectDuplicate processor is invalid because the DistributedMapCacheClientService ID cannot be found As an additional test, I created a template with a processor at the top-level and one inside the Process Group that are using the same Controller Service, and that works just fine. My guess is that the top-level processor is used to create the Controller Service and the inner-processor picks it up by that same ID. I can provide a sample template in a Jira ticket if that would help. Thanks, Brian On Fri, Aug 14, 2015 at 8:24 AM, Bryan Bende bbe...@gmail.com wrote: Brian, Glad it worked! I think it was just a combination of things... I had previously spent a lot of time studying the existing dependency setup in nifi-standard-bundle, as well as the dev guide section on NARs [1]. Also, someone else had ran into a similar problem recently and it turned out to be a dependency problem as well. There should be a Maven archetype for Controller Services [2] coming soon which might help nail down the correct dependency setup, also after seeing your email last night I put together a bare-bones project [3] that has a custom processor using a custom controller service (both created using the archetypes) to serve as an example. -Bryan [1] https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#nars [2] https://issues.apache.org/jira/browse/NIFI-736 [3] https://github.com/bbende/nifi-dependency-example On Fri, Aug 14, 2015 at 8:17 AM, Matt Gilman matt.c.gil...@gmail.com wrote: Brian, The NAR dependency stuff all breaks down to providing classpath isolation and not polluting the classpath of downstream NARs with dependencies required for Controller Service implementation. We try to keep Controller Service interfaces lean by not depending on external libs in the interface (generally a good practice anyways). Both the implementation of the Controller Service and the any Processor NAR that want to consume the Controller Service needs to depend on the Controller Service API NAR (where the interface lives). This established a common parent ClassLoader which allows the service to be access across NARs. If your building Processor NARs that depend on custom Controller Services and standard Controller Services, you'll want to have your custom Controller Services API NAR depend on the standard Controller Services API NAR. The NAR dependency tree (if you will) maps to a ClassLoader hierarchy within NiFi. Controller Services being added in a disabled state is remaining consistent with Processors (and other components) in template. Also, because sensitive properties are not included in templates, they may require the user to complete the configuration prior to enabling. Matt On Fri, Aug 14, 2015 at 8:01 AM, Brian Ghigiarelli briang...@gmail.com wrote: Bryan, Thank you!! Adding a NAR dependency to my standard-processors NAR did the trick to get NiFi to recognize the service for these processors and auto-create a new instance of the Controller Service on template import. I owe ya a beer or six the next time our paths cross. How did you figure out the required NAR dependency? Any great debugging tricks to share, or was it your existing depth of NiFi knowledge that led you to that solution? As a follow-on to that, I'm seeing
Custom Controller Service Debugging
NiFi Gurus, Regarding Controller Services, I'm seeing some strange behavior regarding a custom Controller Service only showing its ID in a Processor and was wondering if anyone's encountered similar issues. Seeing this on the latest develop branch, 0.3.0-SNAPSHOT. Steps taken: 1. Create instance of Controller Service in the Controller Settings Controller Services tab 1a. Enable that instance of the Controller Service 2. Add new Processor on canvas that will use Controller Service 3. Configure Processor 3a. Existing Controller Service does not show in property options. Only No value and Create new service... 3b. Clicking Create new service... opens modal with the custom Controller Service as the only item in the dropdown 3c. Clicking Create button succeeds, but puts the UUID of the Controller Service in the Value of the Processor Property. Observed state: - Processor is still invalid - The ID in the Processor Property matches the ID of the second (disabled) instance of the Controller Service in the Controller Settings modal. There are no errors in nifi-app.log - only contains log messages from the StandardControllerServiceProvider that it Created Controller Service of type [my-implementation-class] with identifier [UUID-that-shows-in-the-GUI] Adding a DetectDuplicate processor with the DistributedMapCacheClientService works just fine, so it'd hard to point fingers at the framework. Any thoughts on where to begin debugging and/or loggers to turn up would be greatly appreciated! Thanks, Brian
Re: Help required with my custom controller service
All,
Was this ever solved / explained? I'm having a similar issue. If there's
a simple answer that I'm missing that would be great... otherwise I might
post an example set of NARs somewhere tomorrow. Probably the simplest
question first is, if I want to create a custom processor that uses a
SSLContextService, what dependencies do I need to add where (assuming we're
using the processor NAR archetype)? I've created an example that compiles,
loads in NiFi, and can be dropped on the graph. However, it can not be
configured to use any currently existing SSLContextService, and any that
are created have the same UUID issue described by David. Thoughts?
Brandon
On Tue, Jul 28, 2015 at 10:50 AM Mark Payne marka...@hotmail.com wrote:
Dave,
In addition to the points that Aldrin brought up, are your CacheService
interface
and StandardCacheService classes in the same NAR?
If they are not in the same NAR, does your NAR for StandardCacheService
have a Maven dependency
on the NAR that houses CacheService?
Thanks
-Mark
From: aldrinp...@gmail.com
Date: Mon, 27 Jul 2015 23:26:58 -0400
Subject: Re: Help required with my custom controller service
To: dev@nifi.apache.org; davidrsm...@btinternet.com
Dave,
Some quick notes as I work through your setup.
Did you create an entry in your
NAR's org.apache.nifi.controller.ControllerService file? This would be in
src/main/resources/META-INF/services directory of your project. It seems
like this isn't likely the case, but a point that can sometimes get
overlooked.
Your accessing of the service property in your processor looks good.
Did you accidentally override AbstractControllerService#getIdentifier()?
It seems like this could lead to the error you are seeing, but it is hard
to be sure.
If you are able to share more code with us, we can provide some more
pointed assistance. Have you managed to get your processor and controller
service running within the test framework [1]? If not, I would suggest
doing so to cut down on iterations in debugging your problems as well as
providing a means for you to easily step through the code.
[1]
http://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#testing
On Sun, Jul 26, 2015 at 4:45 PM, DAVID SMITH davidrsm...@btinternet.com
wrote:
Hi
I have been trying to write a simple Cache Controller Service that uses
a
Map to hold String, String key/value pairs. I am using
nifi-0.1.0-incubating. I have created an Interface called CacheService
which extends ControllerService.public interface CacheService extends
ControllerService {
I have created an implementation called StandardCacheService which is as
follows:
public class StandardCacheService extends AbstractControllerService
implements CacheService {
I have compiled the StandardCacheService and created a nar which I have
then put into my NiFi lib directory, my StandardCacheService Controller
Service appears in the Controller services tab of the NiFi Flow Settings
panel, I can edit it and enable it. Also there are no abnormal log
messages.
However, when I try to configure the CacheService in my test Processor,
again I get no errors but when I cannot see my StandardCacheService. My
processor has the following snippets to implement the Cache
Service/Controller:
public static final PropertyDescriptor CACHE_SERVICE = new
PropertyDescriptor.Builder()
.name(Cache Service)
.description(The Controller Service to use in order to obtain a
Cache Service)
.required(false)
.identifiesControllerService(CacheService.class)
.build();
final CacheService cache =
context.getProperty(CACHE_SERVICE).asControllerService(CacheService.class);
When configuring the processor, the 'Cache Service' drop down doesn't
show
a populated list showing my StandardCacheService, but it does allow me
to
create a new service, the resulting panel allows me to then select
StandardCacheService. When I select 'create' I get a UUID value in the
Value property of the Configure Processor panel. When I then apply this
the processor shows a yellow triangle, when I hover over it I get the
following message:
'Cache Service validated against UUID is invalid because Invalid
Controller service UUID is not a valid Controller service Identifier
or
does not reference the correct type of Controller service.'
Can you help please
Many thanksDave
RE: Help required with my custom controller service
Brandon,
I've seen this occur when I forgot to add a dependency on the
ssl-context-service-api-nar in my NAR's pom.
You should have something like:
dependency
groupIdorg.apache.nifi/groupId
artifactIdssl-context-service-api/artifactId
version0.3.0-SNAPSHOT/version
typenar/type
/dependency
Is that by chance the issue, or is there something else going on?
Thanks
-Mark
From: b...@jhu.edu
Date: Mon, 10 Aug 2015 19:13:50 +
Subject: Re: Help required with my custom controller service
To: dev@nifi.apache.org
All,
Was this ever solved / explained? I'm having a similar issue. If there's
a simple answer that I'm missing that would be great... otherwise I might
post an example set of NARs somewhere tomorrow. Probably the simplest
question first is, if I want to create a custom processor that uses a
SSLContextService, what dependencies do I need to add where (assuming we're
using the processor NAR archetype)? I've created an example that compiles,
loads in NiFi, and can be dropped on the graph. However, it can not be
configured to use any currently existing SSLContextService, and any that
are created have the same UUID issue described by David. Thoughts?
Brandon
On Tue, Jul 28, 2015 at 10:50 AM Mark Payne marka...@hotmail.com wrote:
Dave,
In addition to the points that Aldrin brought up, are your CacheService
interface
and StandardCacheService classes in the same NAR?
If they are not in the same NAR, does your NAR for StandardCacheService
have a Maven dependency
on the NAR that houses CacheService?
Thanks
-Mark
From: aldrinp...@gmail.com
Date: Mon, 27 Jul 2015 23:26:58 -0400
Subject: Re: Help required with my custom controller service
To: dev@nifi.apache.org; davidrsm...@btinternet.com
Dave,
Some quick notes as I work through your setup.
Did you create an entry in your
NAR's org.apache.nifi.controller.ControllerService file? This would be in
src/main/resources/META-INF/services directory of your project. It seems
like this isn't likely the case, but a point that can sometimes get
overlooked.
Your accessing of the service property in your processor looks good.
Did you accidentally override AbstractControllerService#getIdentifier()?
It seems like this could lead to the error you are seeing, but it is hard
to be sure.
If you are able to share more code with us, we can provide some more
pointed assistance. Have you managed to get your processor and controller
service running within the test framework [1]? If not, I would suggest
doing so to cut down on iterations in debugging your problems as well as
providing a means for you to easily step through the code.
[1]
http://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#testing
On Sun, Jul 26, 2015 at 4:45 PM, DAVID SMITH davidrsm...@btinternet.com
wrote:
Hi
I have been trying to write a simple Cache Controller Service that uses
a
Map to hold String, String key/value pairs. I am using
nifi-0.1.0-incubating. I have created an Interface called CacheService
which extends ControllerService.public interface CacheService extends
ControllerService {
I have created an implementation called StandardCacheService which is as
follows:
public class StandardCacheService extends AbstractControllerService
implements CacheService {
I have compiled the StandardCacheService and created a nar which I have
then put into my NiFi lib directory, my StandardCacheService Controller
Service appears in the Controller services tab of the NiFi Flow Settings
panel, I can edit it and enable it. Also there are no abnormal log
messages.
However, when I try to configure the CacheService in my test Processor,
again I get no errors but when I cannot see my StandardCacheService. My
processor has the following snippets to implement the Cache
Service/Controller:
public static final PropertyDescriptor CACHE_SERVICE = new
PropertyDescriptor.Builder()
.name(Cache Service)
.description(The Controller Service to use in order to obtain a
Cache Service)
.required(false)
.identifiesControllerService(CacheService.class)
.build();
final CacheService cache =
context.getProperty(CACHE_SERVICE).asControllerService(CacheService.class);
When configuring the processor, the 'Cache Service' drop down doesn't
show
a populated list showing my StandardCacheService, but it does allow me
to
create a new service, the resulting panel allows me to then select
StandardCacheService. When I select 'create' I get a UUID value in the
Value property of the Configure Processor panel. When I then apply this
the processor shows a yellow triangle, when I hover over it I get the
following message:
'Cache Service validated against UUID is invalid because Invalid
Controller service UUID is not a valid Controller service Identifier
or
does not reference the correct type of Controller service.'
Can you help please
Many
