On Tue, Jul 24, 2018 at 3:17 PM Ishara Cooray <isha...@wso2.com> wrote:
> Our objective is to run this clean up process without anyone having to > configure anything. Even today we have instructions and scripts to clean up > the tables. But in practice, nobody even notices this and never does these > kind of stuff until they hit an issue and consult us through support. So if > we design this in such a way that someone has to turn on something or > configure something, I am positive our efforts will be in waste. > > So by default are we going to enable the token cleanup feature and storing > old access tokens in audit table? > Yes, otherwise there's no point of this feature IMO because we currently have a solution that cleans data but in reality nobody uses it unless they hit a issue and request our assistance to find and run this. > > *@Nalaka,* > Have we done a load test to verify whether this feature does not affect to > the functionalities when there is a high load in a high concurrency > situation? > If so appreciate if you can share that details as well. > Yes, we did. @Nalaka Senarathna <nala...@wso2.com> please point to the relevant thread. > > > > Thanks & Regards, > Ishara Cooray > Senior Software Engineer > Mobile : +9477 262 9512 > WSO2, Inc. | http://wso2.com/ > Lean . Enterprise . Middleware > > On Fri, Jul 20, 2018 at 10:04 AM, Harshan Liyanage <hars...@wso2.com> > wrote: > >> Hi Nuwan, >> >> Thanks for your detailed clarifications. Both explanations are perfectly >> valid. >> >> Regards, >> >> Harshan Liyanage >> Mobile: *+94765672894* >> Email: hars...@wso2.com >> Blog: http://harshanliyanage.blogspot.com/ >> Medium: https://medium.com/@harshan.dll >> *WSO2, Inc.:** wso2.com <http://wso2.com/>* >> lean.enterprise.middleware. >> >> >> On Thu, Jul 19, 2018 at 8:35 PM Nuwan Dias <nuw...@wso2.com> wrote: >> >>> >>> >>> On Wed, Jul 18, 2018 at 9:09 PM Harshan Liyanage <hars...@wso2.com> >>> wrote: >>> >>>> Hi all, >>>> >>>> @Nuwan: That's why I suggested having a configurable cron expression so >>>> that users can configure the task to run on an optimal schedule instead of >>>> blocking vital functionalities. Also in that way, we could do a batch >>>> deletes and insertions instead of single rows. For example, they could let >>>> it run every mid-night so then the access token tables won't grow for >>>> millions of records and not affecting the user traffic. >>>> >>> >>> Our objective is to run this clean up process without anyone having to >>> configure anything. Even today we have instructions and scripts to clean up >>> the tables. But in practice, nobody even notices this and never does these >>> kind of stuff until they hit an issue and consult us through support. So if >>> we design this in such a way that someone has to turn on something or >>> configure something, I am positive our efforts will be in waste. >>> >>>> >>>> @Nalaka: You could let the task to run only on a manager node in a >>>> distributed setup using a configuration. >>>> >>> >>> There is no such thing as a "manager" node in our architecture. All >>> nodes are equal. Plus, we cannot introduce such changes as well due to >>> numerous complexities such as how to HA the manager node, too many varying >>> configurations, etc. >>> >>>> >>>> BTW that was just a suggestion. It doesn't mean I'm -1 on this proposed >>>> design. :) >>>> >>>> Thanks, >>>> >>>> Harshan Liyanage >>>> Mobile: *+94765672894* >>>> Email: hars...@wso2.com >>>> Blog: http://harshanliyanage.blogspot.com/ >>>> Medium: https://medium.com/@harshan.dll >>>> *WSO2, Inc.:** wso2.com <http://wso2.com/>* >>>> lean.enterprise.middleware. >>>> >>>> >>>> On Wed, Jul 18, 2018 at 8:51 PM Nalaka Senarathna <nala...@wso2.com> >>>> wrote: >>>> >>>>> hi harshan, >>>>> >>>>> Also if there are multiple nodes then those nodes also may attempt to >>>>> clean up the same access token at the same time. >>>>> >>>>> related mail thread:[1] >>>>> [1]Access token Table cleaning and keeping the access token data for >>>>> future purposes >>>>> >>>>> regards. >>>>> >>>>> >>>>> On Wed, Jul 18, 2018 at 8:28 PM, Nuwan Dias <nuw...@wso2.com> wrote: >>>>> >>>>>> A periodic task won't work for this because when the system runs on >>>>>> tables with millions of records, the task will lock the table for the >>>>>> clean >>>>>> up process. This will impact other queries being executed on the table >>>>>> and >>>>>> hence block user flows. >>>>>> >>>>>> On Wed, Jul 18, 2018 at 6:17 AM Harshan Liyanage <hars...@wso2.com> >>>>>> wrote: >>>>>> >>>>>>> Hi Nalaka, >>>>>>> >>>>>>> You could do the same with a configurable periodic task instead of >>>>>>> modifying existing token request flows. What you have to do is to >>>>>>> implement >>>>>>> the token cleanup feature as a periodic task which scans the token >>>>>>> related >>>>>>> tables and move EXPIRED, INACTIVE and REVOKED tokens to the audit >>>>>>> table. There will be a configuration to configure the CRON expression >>>>>>> for >>>>>>> that task. >>>>>>> >>>>>>> WDYT? >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Harshan Liyanage >>>>>>> Mobile: *+94765672894* >>>>>>> Email: hars...@wso2.com >>>>>>> Blog: http://harshanliyanage.blogspot.com/ >>>>>>> Medium: https://medium.com/@harshan.dll >>>>>>> *WSO2, Inc.:** wso2.com <http://wso2.com/>* >>>>>>> lean.enterprise.middleware. >>>>>>> >>>>>>> >>>>>>> On Wed, Jul 18, 2018 at 2:45 PM Nalaka Senarathna <nala...@wso2.com> >>>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>>> A solution for the access token table filled up with EXPIRED, >>>>>>>> INACTIVE and REVOKED tokens in the Access token table, old access >>>>>>>> token can >>>>>>>> move to the Audit table when the new token is generating, renewing or >>>>>>>> token >>>>>>>> revoking. >>>>>>>> >>>>>>>> >>>>>>>> Old Access tokens will be stored in newly created Audit table for >>>>>>>> audit purposes. >>>>>>>> >>>>>>>> >>>>>>>> *Token cleanup feature can be turn enable / disable. >>>>>>>> >>>>>>>> *When the token cleanup feature is enabled old access token can be >>>>>>>> stored in the audit table >>>>>>>> >>>>>>>> For audit purposes or can disable that feature too, not to store >>>>>>>> old access tokens(this also can enable/disable). >>>>>>>> >>>>>>>> >>>>>>>> This project avoids the access token table growing and improve the >>>>>>>> token lookup. For existing users also can get advantage from this. >>>>>>>> >>>>>>>> >>>>>>>> Growing Audit table is no problem if the token cleanup feature and >>>>>>>> retain old access token feature is enabled .because writing operation >>>>>>>> to >>>>>>>> Audit table is not increased when the token table is growing(Audit >>>>>>>> table >>>>>>>> has no constraint).This property was tested until 14 million token >>>>>>>> generation. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> regards. >>>>>>>> >>>>>>>> -- >>>>>>>> *Nalaka Senarathna* >>>>>>>> *Associate Software Engineer | WSO2* >>>>>>>> >>>>>>>> *Email : nala...@wso2.com <nala...@wso2.com>* >>>>>>>> *Mobile : +94714118474* >>>>>>>> *web : https://wso2.com <https://wso2.com>* >>>>>>>> <https://wso2.com/signature> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Architecture mailing list >>>>>>>> Architecture@wso2.org >>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> Nuwan Dias >>>>>> >>>>>> Director - WSO2, Inc. http://wso2.com >>>>>> email : nuw...@wso2.com >>>>>> Phone : +94 777 775 729 >>>>>> >>>>>> _______________________________________________ >>>>>> Architecture mailing list >>>>>> Architecture@wso2.org >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> *Nalaka Senarathna* >>>>> *Associate Software Engineer | WSO2* >>>>> >>>>> *Email : nala...@wso2.com <nala...@wso2.com>* >>>>> *Mobile : +94714118474* >>>>> *web : https://wso2.com <https://wso2.com>* >>>>> <https://wso2.com/signature> >>>>> >>>>> _______________________________________________ >>>> Architecture mailing list >>>> Architecture@wso2.org >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>> >>> >>> -- >>> Nuwan Dias >>> >>> Director - WSO2, Inc. http://wso2.com >>> email : nuw...@wso2.com >>> Phone : +94 777 775 729 >>> _______________________________________________ >>> Architecture mailing list >>> Architecture@wso2.org >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >> >> _______________________________________________ >> Architecture mailing list >> Architecture@wso2.org >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > -- Nuwan Dias Director - WSO2, Inc. http://wso2.com email : nuw...@wso2.com Phone : +94 777 775 729
_______________________________________________ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
