I agree, the Controller Service needs to be enhanced to support rotating SAS tokens. I know of a few users that have run into this exact problem.
Technically, it should possible to write a custom implementation of the ADLSCredentialsService API that did this, but it would be ideal if NiFi provided some capability out of the box. On Mar 13, 2023 at 21:28:33, Adam Taft <[email protected]> wrote: > Hi Mike, > > You're right, I don't think you're going to get anywhere with the > flowfile attribute providing the value, as like you said, > ListAzureBlobStorage is a source processor and doesn't accept any input > flowfiles. > > So currently, you're really looking at two choices, which you've already > identified: > 1. Manually rotate the SAS token through the user interface. > 2. Make an HTTP call to the NiFI API causing an update to the controller > service SAS token property. > > Now, #2 is not quite as bad as you think it is. Luckily, NiFi will deal > very nicely with the temporary restart of the backing controller service. > NiFi handles shutting down all the processors that are connected to the > controller service and then bringing them back up (from multiple API > calls). It will queue up the flowfiles for the short time this operation > occurs (but yes, it does need to stop those processors before any changes > can be made). In theory, you shouldn't have any corruption issues going > this path. The problem side of it will be in writing your script to connect > with the API and issuing the request to make this change. A bit painful, > but possible. Your script would need to locate or know how to find the > correct controller service (typically via its uuid). > > All this really speaks to a feature request, to be honest. It would be > ideal that the controller service can reconfigure itself by grabbing the > new SAS token for your storage account from Azure. Presumably the best > practice would be to pick up the new SAS token from Key Vault (since Key > Vault can be configured to own SAS Token generation for the storage > account)? I guess Azure AD / OAuth is also a possible (alternative) > solution? > > Anyway, short term, consider the API approach. Long term, a contribution > to the project along these lines would probably be gratefully accepted. At > minimum, a Jira ticket to suggest the new feature would be a good starting > place. > > /Adam > > > > On Wed, Mar 8, 2023 at 9:53 AM Mike Rutlin <[email protected]> wrote: > >> >> I have a general question on how to dynamically update and sync SAS >> tokens being utilized by the AzureBlogStorage processors ( >> List,Fetch,Delete, etc) . >> >> From our NIFI we are accessing a storage container in a different cloud >> and thus cannot use AZURE MI as standard for storage containers in the same >> cloud. Thus we need to use the SAS Tokens attribute / parameter supported >> by the controller service or by the processors. >> >> We pull the SAS token on a periodic basis, before the old SAS token >> expires. >> >> In manual testing of this we set a sensitive Parameter to the SAS token >> and set up the Controller service SAS Token attribute / parameter to use >> the Parameter we created. Everything worked as it should. >> >> The problem/issue occurs when the SAS token expires and we pull the new >> one. How do I modify the Parameter ? Using the NIFI API seems messy and >> potential to corrupt the flow configuration. along with the fact that it >> stops and starts service / processors with the Parameter we are updating. >> >> I would like to use a flowfile attribute ( which the documentation >> suggests is possible ). However the ListAzureBlobStorage processor is a >> start flow processor ( no inputs ). Thus I can't run a script to get the >> new key and set an attribute that could then be used by the SAS Token >> attribute / parameter in the processor. >> >> Any ideas on how I can do this? It seems this would be a common practice >> to roll keys and then inject the new key into the data flow for either AWS >> or Azure. >> >> If the ListAzureBlobStoreage had an input I coud update with new keys OR >> if the list processor had a failure output I could update the key upon >> expiration. >> >> Thanks for any and all guidance. >> >> Mike R >> >>
