Mans, Thanks for sticking with this and continuing to see things through, the community certainly appreciates it as these are very popular processors and this functionality will help a wide base of users.
I am poking around a bit more and thinking we might be able to work something out with a class that adapts an AWSCredentialsProvider to AWSCredentials. The AWSProvider interface is just composition of AWSCredentials with a refresh method. Need to mull things over a bit, and dig through the associated libraries to understand how these are typically used, but this feels like it could be another avenue to consider and where I am directing my attention at the moment. --aldrin On Fri, Jan 8, 2016 at 9:58 AM, M Singh <[email protected]> wrote: > Just one more note Joe (as mentioned in the Jira Ticket) > From what I can see, we cannot just deprecate createClient method in the > AbstractAWSProcessor which uses the AWSCredentials argument, since the > subclasses AbstractS3/SNS/SQSProcessor call that to create the respective > clients. We will have to change the argument to AWSCredentialProvider. > If I can assist with any other investigation, please let me know. > Thanks again. > > On Friday, January 8, 2016 5:31 AM, M Singh > <[email protected]> wrote: > > > Thanks Joe. > If you think that we can accept the change to creds provider, I will work > on making all the components in nifi aws processors to be consistent. I > think using the creds provider interface is the way to go since it is more > flexible and at this moment we just have 3 aws processors to migrate. > Looking forward to hearing from you/anyone else for advice/feedback. > Mans > > On Friday, January 8, 2016 5:18 AM, Joe Witt <[email protected]> > wrote: > > > Mans, > > I am working to put out a proposed roadmap and then probably won't be > very responsive until later tonight. Will try to help then if no one > else has had a chance to. > > That said I see what you mean in terms of a breaking change in the > processor implementation as far as anyone else that has extended it. > There have been some discussions recently about this and I think the > plan is to start annotating everything with the audience and stability > of a given bit of code. Processors are not meant to be locked down > APIs. So, for now, given that it has been ambiguous to the community > the best course is to probably just deprecate a given method if it > cannot be safely repurposed and then use a new one which does meet the > need in the event the controller service is supplied. This last > statement though is not based on me having looked at the code in any > detail yet. > > Thanks > Joe > > On Fri, Jan 8, 2016 at 8:08 AM, M Singh <[email protected]> > wrote: > > Hi Joe: > > I have not worked with the controller interface and aws processors so > perhaps you can help me understand it . > > From what I can see (as mentioned in > https://issues.apache.org/jira/browse/NIFI-1325): currently, the Nifi > AbstractAWSProcessor has a method > > protected abstract ClientType createClient(final ProcessContext context, > final AWSCredentials credentials, final ClientConfiguration config); > > This method is overridden in the AbstractS3/SNS/SQSProcesors to provide > the respective amazon s3/sns/sqs client using AWSCredentials argument. > > Here is a snippet from AmazonS3Client: > > public AmazonS3Client(AWSCredentials awsCredentials, > ClientConfiguration clientConfiguration) { > super(clientConfiguration); this.awsCredentialsProvider = new > StaticCredentialsProvider(awsCredentials); init(); } > > So, AmazonS3/SNS/SQSClient created with AWSCredentials use > StaticCredentialsProvider in their implementation. > > All the AWSCredentials impls are static creds (Anonymous/Properties > Credentials) except for the STSSessionCredentials which has a refresh > method but is deprecated in favor of the STSSessionCredentialsProvider > interface. AWSCredentials is the interface being used in nifi aws > processors. > > The AWSCredentialsProvider interface has a fresh method which all it's > subclasses implement appropriately - the static ones (like > PropertyFileCredentialsProvider/StaticCredentialsProvider have a no op for > refresh method) as follows: > > public void refresh() {} > > From what I can see, there is no common interface available for > AWSCredentials and AWSCredentialsProvider that Nifi's > AbstractAWSProcessor.createClient can support. So if we need to use the > controller interface with creds providers, will will have to change > AbstractAWSProcessor.createClient to the following > > > > protected abstract ClientType createClient(final ProcessContext context, > final AWSCredentialsProvider credentialsProvider, > > final ClientConfiguration config); > > This appears to be a breaking change for the clients who have extended > the AbstractAWSProcessor.createClient with the AWSCredentials argument > rather that the AWSCredentialsProvider. > > So, can you please elaborate on how the AbstractAWSProcessor will be > able to support both the current impl (ie, invoking aws components with > creds) and the proposed credentials provider interface ? > > Thanks > > Mans > > > > On Thursday, January 7, 2016 9:00 PM, Joe Witt <[email protected]> > wrote: > > > > > > Mans, > > > > It appears to me that there is a path for this not to be a breaking > > change for the flow. By creating a controller service to handle the > > credential provider piece you should be able to just update the > > processor to support that controller service interface. If the user > > sets that controller service then you use that and if they don't then > > you revert to using the older properties. We can mark those > > properties as no longer the preferred model and deprecate them in the > > codebase then when we reach a 1.0 milestone we can remove them. > > > > Thanks > > Joe > > > > On Thu, Jan 7, 2016 at 9:07 PM, M Singh <[email protected]> > wrote: > >> Hi: > >> Just wanted to mention that if we go with the creds provider interface > it will be breaking change for nifi aws components as mentioned in > https://issues.apache.org/jira/browse/NIFI-1325. > >> Also, I am considering creating one aws creds provider controller which > will provide creds provider based on property file, basic, anonymous or > assume role session params. > >> Please let me know if there is any additional feedback for me. > >> Thanks again. > >> Mans > >> > >> On Thursday, January 7, 2016 2:56 PM, M Singh > <[email protected]> wrote: > >> > >> > >> Hi Joe: > >> Based on your feedback I will try to explore the controller interface > for aws creds provider. > >> Thanks for your advice. > >> Mans > >> > >> On Thursday, January 7, 2016 4:15 AM, Joe Witt <[email protected]> > wrote: > >> > >> > >> Mans > >> > >> Appreciate you pushing this forward. There is a related idea to better > >> handle aws credentials for all the aws procs. Will look more and > respond. > >> > >> Thanks > >> Joe > >> On Jan 7, 2016 6:52 AM, "M Singh" <[email protected]> wrote: > >> > >>> Hi: > >>> Just wanted to follow-up and see if anyone has any feedback on . > >>> https://issues.apache.org/jira/browse/NIFI-1325. > >>> Thanks > >>> Mans > >> > >> > >> > >> > >> > > > > > > > > > > >
