Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
AnTu2702 commented on PR #6900: URL: https://github.com/apache/nifi/pull/6900#issuecomment-2106808373 @tpalfy Hey. How are you? First of all: Thanks for the fix/workaround in NIFI 1.26. Although things are basically working there, we still don't seem to be through. The ConsumeIMAP Processor is continuously logging this ERROR... (although the mailbox can and will be processed successfully using a refreshed token...)  We still have serious doubts to take this into production because of the log being spammed and of course the fact, that we have no clue, that causes this error. Could you help and double check your fix/workaround? Thank you in advance... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
tpalfy commented on PR #6900: URL: https://github.com/apache/nifi/pull/6900#issuecomment-1994553665 > Hm... Sorry, but I have doubts about this... > > @tpalfy Our access tokens e.g. are valid for 60 minutes only. Due to my understanding, the StandardOauth2AccessTokenProvider.java is intentionally written such that an authentication exception shall never occur at all! It will always automatically refresh any token, BEFORE it will expire (using the refresh window) by just asking it for "the access token" from outside calling `getAccessDetail()`. It will always be valid this way... > > So an authentication exception should IMHO never be intended to occur at all... > > ... > > But simply to have a quickfix für NIFI 1.26 still better than nothing. ;-) Okay, that's fair. I have a PR up with my proposed changes here: https://github.com/apache/nifi/pull/8494 In it's first state it only recreates the mail receiver if an error is encountered. I'm going to update it by keeping the first commit, adding a revert commit and adding a new one that checks if the token has been refreshed. So both approach can be checked. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
AnTu2702 commented on PR #6900: URL: https://github.com/apache/nifi/pull/6900#issuecomment-1993901612 Hm... Not sure about this... Our access tokens e.g. are valid for 60 minutes only. Due to my understanding, the StandardOauth2AccessTokenProvider.java is intentionally written such that an Authentication Exception shall never occur at all! It will automatically refresh any token, BEFORE it will expire (using the refresh window) by just asking it for "the access token". It will always be valid this way... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
AnTu2702 commented on PR #6900: URL: https://github.com/apache/nifi/pull/6900#issuecomment-1993902221 > @AnTu2702 I think I can add a not too complicated change that only recreates the `messageReciever` when an authentication exception is encountered in a couple of days, hopefully today. > > Since there are connections involved and given the fact that onTriggers can be called very frequently, it's better to have it like this right away. Hm... Not sure about this... @tpalfy Our access tokens e.g. are valid for 60 minutes only. Due to my understanding, the StandardOauth2AccessTokenProvider.java is intentionally written such that an Authentication Exception shall never occur at all! It will automatically refresh any token, BEFORE it will expire (using the refresh window) by just asking it for "the access token". It will always be valid this way... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
tpalfy commented on PR #6900: URL: https://github.com/apache/nifi/pull/6900#issuecomment-1988565123 @AnTu2702 I think I can add a not too complicated change that only recreates the `messageReciever` when an authentication exception is encountered in a couple of days, hopefully today. Since there are connections involved and given the fact that onTriggers can be called very frequently, it's better to have it like this right away. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
AnTu2702 commented on PR #6900: URL: https://github.com/apache/nifi/pull/6900#issuecomment-1981577552 How about a quick bugfix using this new approacj for NIFI 1.26 and then refactoring it afterwards? We really really depend on having that fixed in the next couple of weeks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
tpalfy commented on PR #6900: URL: https://github.com/apache/nifi/pull/6900#issuecomment-1981377674 @AnTu2702 Not sure about this new approach. I'm not completely against it but I feel like rebuilding only when we get an authentication error would be more porper. Not trivial as it would require some refactoring but I think it could be worth it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
AnTu2702 commented on PR #6900: URL: https://github.com/apache/nifi/pull/6900#issuecomment-1981009647 @tpalfy @RomanOttGmx I tried a different solution that does seem to do the job:  It is a bit ugly, because the `initializeIfNecessary method` will this way reinitialize the `messageRecevier` for OAuth2 based access on every `trigger()` call. What do you think? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
AnTu2702 commented on PR #6900: URL: https://github.com/apache/nifi/pull/6900#issuecomment-1978792542 @tpalfy You may be right. I just tested the refresh mechanism (not did not wait the 3600 s until the first access token really expired) with my intended fix that morning. But the refreshed token nonetheless needs to be considered by the processor accessing the IMAP server. And this happens in your `buildurl method`. So the real fix needs to be a bit different. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
tpalfy commented on PR #6900: URL: https://github.com/apache/nifi/pull/6900#issuecomment-1978692871 Thanks @AnTu2702 and @RomanOttGmx for the explanaion. As for your fix @AnTu2702 I'm a bit confused though. If we only refresh the token but don't rebuild the url I don't see how that would work. Yes, we have a fresh token but are we actually using it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
AnTu2702 commented on PR #6900: URL: https://github.com/apache/nifi/pull/6900#issuecomment-1978453332 I was able to build and deploy a local nar file...  ...working fine with the patch! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
AnTu2702 commented on PR #6900: URL: https://github.com/apache/nifi/pull/6900#issuecomment-1978198291 > @AnTu2702 The `oauth2AccessTokenProviderOptional` field is used in the `buildUrl` method where we call `oauth2AccessTokenProvider.getAccessDetails().getAccessToken()`. The `getAccessDetail()` should take care of refreshing/reacquiring tokens (this is the same call you suggest doing at the beginning of `onTrigger`). > > So according to this, token expiration is handled already. Do we have proof that this doesn't work for some reason? Hey. The `this.messageReceiver = this.buildMessageReceiver(context)` and thus `buildUrl` method in line 331 is only called once. Reason is line 329: `if (this.messageReceiver == null)`  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
AnTu2702 commented on PR #6900: URL: https://github.com/apache/nifi/pull/6900#issuecomment-1978169106 Overview:  ConsumeImap:   StandardOAuth2TokenProvider:  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
RomanOttGmx commented on PR #6900:
URL: https://github.com/apache/nifi/pull/6900#issuecomment-1977185841
In IMAP Processor buildUrl will be triggered on creation of new receiver.
```
`protected ImapMailReceiver buildMessageReceiver(ProcessContext
processContext) {
ImapMailReceiver receiver = new
ImapMailReceiver(this.buildUrl(processContext));`
```
And the buildMessageReceiver is requested here in AbstractEmailProcessor
```
private synchronized void initializeIfNecessary(ProcessContext context,
ProcessSession processSession) {
if (this.messageReceiver == null) {
this.processSession = processSession;
this.messageReceiver = this.buildMessageReceiver(context);
```
This is called in onTrigger method of AbstractEmailProcessor, but the
receiver still exists since start up, so buildUrl will never be requested again
after enabling the ConsumeImapProcessor.
```
public void onTrigger(ProcessContext context, ProcessSession processSession)
throws ProcessException {
this.initializeIfNecessary(context, processSession);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
RomanOttGmx commented on PR #6900: URL: https://github.com/apache/nifi/pull/6900#issuecomment-1977172784 Yes, implement ConsumeImapProcessor with OAUTH2 and an associated oauth2AccessTokenProviderService. This Provider is only triggered on the onScheduled Method of the AbstractEmailProcessor and oauth2AccessTokenProvider.getAccessDetails() to refresh the token is only requested also on onScheduledMethod. But in onTriggered Method this will not trigger the oauth2AccessTokenProvider.getAccessDetails() We startet a local debug with break points. Configured the ConsumeImapProcessor with Schedule "Cron-Based" every 5 minutes. onScheduled => triggered on enabling the Processor Every 5 minutes there is the onTriggered Method triggered and not the onScheduled. That is general for all Processors which use the AbstractEmailProcessor. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
tpalfy commented on PR #6900: URL: https://github.com/apache/nifi/pull/6900#issuecomment-1976510039 @AnTu2702 The `oauth2AccessTokenProviderOptional` field is used exclusively in the `buildUrl` method where we call `oauth2AccessTokenProvider.getAccessDetails().getAccessToken()`. The `getAccessDetail()` should take care of refreshing/reacquiring tokens (this is the same call you suggest doing at the beginning of `onTrigger`). So according to this token expiration is handled already. Do we have proof that this doesn't work for some reason? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
AnTu2702 commented on PR #6900: URL: https://github.com/apache/nifi/pull/6900#issuecomment-1976176344 Here would be my suggestion for a fix:  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
AnTu2702 commented on PR #6900: URL: https://github.com/apache/nifi/pull/6900#issuecomment-1975971033 I debugged the AbstractEmailProcessor.java and it seems, that AbstractEmailProcessor.java does not handle token expiration at all. The StandardOAuth2AccessTokenProvider.Java does, but it would need the AbstractEmailProcessor.java to call the: oauth2AccessTokenProvider.getAccessDetails() in the onTrigger() method instead the onSchedule() method to do so.  The AbstractEmailProcessor.java should be fixed to handle expired OAuth2 access tokens! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-11107 In ConsumeIMAP and ConsumePOP3 added support for OAUTH based authorization. [nifi]
araujf commented on PR #6900: URL: https://github.com/apache/nifi/pull/6900#issuecomment-1753670203 The refresh authentication token is working? When my token is expired, a authenticate error is returned. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
