Hi Chathura, On Wed, Jul 5, 2017 at 7:14 PM, Chathura Kulasinghe <[email protected]> wrote:
> > > Chathura Asanga Kulasinghe | + > > 4 > 4 > 77 2920 2448 | Senior Solutions Engineer | WSO₂ > > On 5 July 2017 at 05:35, Lakmal Warusawithana <[email protected]> wrote: > >> >> >> On Wed, Jul 5, 2017 at 3:53 AM, Chathura Kulasinghe <[email protected]> >> wrote: >> >>> >>> >>> Chathura Asanga Kulasinghe | + >>> >>> 4 >>> 4 >>> 77 2920 2448 | Senior Solutions Engineer | WSO₂ >>> >>> On 4 July 2017 at 18:02, Lakmal Warusawithana <[email protected]> wrote: >>> >>>> >>>> >>>> On Tue, Jul 4, 2017 at 10:07 PM, Chathura Kulasinghe < >>>> [email protected]> wrote: >>>> >>>>> >>>>> >>>>> Chathura Asanga Kulasinghe | + >>>>> >>>>> 4 >>>>> 4 >>>>> 77 2920 2448 | Senior Solutions Engineer | WSO₂ >>>>> >>>>> On 30 June 2017 at 10:17, Lakmal Warusawithana <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Maninda, >>>>>> >>>>>> On Fri, Jun 30, 2017 at 12:02 PM, Maninda Edirisooriya < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi Dinusha, >>>>>>> >>>>>>> Why do we think it is the responsibility of the CEP to pull poilcies >>>>>>> from the APIM? Just forgetting the APIM case, if we think this as a >>>>>>> general >>>>>>> feature of CEP, here the CEP is periodically pulling artifacts from a >>>>>>> given >>>>>>> set of APIs. There are 2 drawbacks I see in this approach. >>>>>>> >>>>>>> 1. Updating the CEP with latest artifacts is restricted by the given >>>>>>> polling time interval. (realtime CEP artifact updates are not possible >>>>>>> from >>>>>>> an external party which is important as CEP is dealing with realtime >>>>>>> requirements) >>>>>>> >>>>>>> 2. Dynamically adding unplanned artifacts to a running CEP server is >>>>>>> not possible as the polling APIs has to be pre configured. For example >>>>>>> this >>>>>>> can happen when the existing APIM server is stopped and a different APIM >>>>>>> server is started with a different host address. In that case the CEP >>>>>>> also >>>>>>> has to be restarted to update the policy API location, which would lose >>>>>>> some data in the running CEP. Correct me if I am wrong. >>>>>>> >>>>>>> Therefore I think the current policy push based approach is correct. >>>>>>> We should be able to correct the syncing policies across the CEP nodes >>>>>>> with >>>>>>> a different approach like deploment synchronizing, registry based >>>>>>> approach, >>>>>>> with a shared database based approach or with a clustering solution like >>>>>>> Hazelcast. >>>>>>> >>>>>> >>>>>> This is exactly what we want to avoid. :) >>>>>> >>>>> >>>>> Would it be practical to think of spinning-up the Traffic-Manager >>>>> instances with updated policies as an immutable-node-group? (...more a >>>>> dev-ops oriented approach). Because, a poll-and-pull type of mechanism >>>>> anyway will have a delay in applying the policy updates, which could have >>>>> become a concern for some industries (even if that may be accepted in >>>>> general case). >>>>> >>>>> >>>> If someone want immutability, yes they can bern policy to image and >>>> spin-up. Having pull method not harm for them as well. VM scenario and some >>>> container cases, they don't want to build docker image or VM with every >>>> policy update. So default it pull, but they can avoid that as well. >>>> >>> >>> I am sorry for narrowing my comment down to containers based >>> deployment. However, what I wanted to mention is was a set of reasons for >>> thinking of sticking to the push model, based on a few concerns that may be >>> raised by customers. In order to describe this, let me take a sample >>> use-case. >>> >>> If we consider Telco business for example, what is allowed & >>> restricted/controlled under a certain policy is critical as every >>> millisecond counts from the finance departments' point-of-view. Therefore, >>> those customers like that may expect the policy updates to become available >>> in the runtime immediately after modifying/publishing them. In order to >>> comply with this requirement at the expected level, with the poll-and-pull >>> method, we need to configure a polling interval of a few milliseconds. >>> However, such policy updates are not that frequent in a typical business >>> scenario (even if it is crucial to apply it immediately once a change is >>> made). >>> >>> For example, a policy update may happen once a month, but the >>> organisation expect to apply changes immediately as they are made (without >>> waiting a few minutes until the Traffic Manager fetches the updates with a >>> noticeable delay). >>> >>> In such a scenario, if we ask the customer to configure a polling >>> interval of a few milliseconds, many EAs, infra/network staff may consider >>> that as an unnecessary overhead as the system is supposed to make frequent >>> network polls in order to fetch any updates that possibly may happen once a >>> month. >>> >>> Therefore, wouldn't it be advisable to perform this following a >>> 'recipient-list' style push mechanism? Typically, the traffic manager >>> wouldn't scale into a number of instances that exceeds 4-6. And if we >>> introduce a 'recipient-list' style policy updates distribution mechanism, >>> that wouldn't have any dependency over clustering and hazelcast as well. >>> >>> I understand that you guys may have many other reasons/concerns to >>> consider while selecting the best possible option. But, I just thought of >>> adding thoughts to be considered in further brainstorming and to make sure >>> that no doubts/concerns were left unspoken. :) >>> >> >> I am seen some contradiction within your usecase. IMO, if customer >> updating policies in months and they want to update runtime in >> milliseconds? Sorry, to me this is not valid use case. >> >> Please apply 80/20 rule. Most of the case these policies not going to be >> change after deploying. So in these cases no need to have pull model as >> well. Few cases, they will change these policies and that totally fine with >> pull model or manually handle artifact syncing. (may be puppet etc). >> >> Pull model is one reference implementation. What we providing is an API >> in the core to retrieve updated siddhi artifacts. They can run this in TM >> to get the new artifacts. It can be manual, shell script, puppet with >> m-collective, cron etc. If someone wan't realtime updates, after creating >> policies in TM, they can should trigger any update mechanism IMO. >> > > Thanks Lakmal. With the manual/script-based approach that you described, > I believe that we could think of a solution for the scenario which I > described. (Because, in that case we make sure that no action related to > SiddhiApp.zip is triggered unless there is an actual update of policies). > > However, regarding the normal pull approach mentioned in your comment, I > have the following question. > > [image: Inline images 1] > Let's forget about any specific use-case (similar to what I have given > earlier) for now and think of the general scenario. In that case, > > 1. The initial API invocation *[indicated with (1) in the diagram]* is > initiated by the CEP/TM at a given interval/frequency, isn't it? > 2. If YES, what would be the ideal interval that we could propose > practically? (10 seconds, 5 mins, 30 mins, 2 Hrs etc). > - The reason for this question is a situation that may come in to > the picture from the point-of-view of 2 different types of parties. > - If we assume that any policy update happens very rarely, hence > decided to keep a much longer interval between API-M core API > invocation > made by the CEP/TM; the business/admin users who updates policies > may have > to wonder/wait for a long until the changes become available in the > run-time (less control and visibility). > - If we introduce a shorter interval instead, the EAs may > question the number of frequent API invocations (network calls) as > the > policy changes happen very rarely (hence, seeing these frequent > calls as an > unnecessary cost). In scenarios where organizations outsource > infrastructure administration work etc, these sort of details could > be > questioned. > > > IMO 10 min cron job is more than enough. > >> >>> >>>> >>>>> >>>>>> #1 is not an valid concern. We are not going to solve generic case >>>>>> here. In the APIM, throttling policy update not a frequent thing. >>>>>> Sometimes it only doing one time for a deployment. >>>>>> >>>>>> I can't understand #2 concern. No need CEP to restart anytime. >>>>>> >>>>>> >>>>>>> >>>>>>> Thanks. >>>>>>> >>>>>>> >>>>>>> *Maninda Edirisooriya* >>>>>>> Senior Software Engineer >>>>>>> >>>>>>> *WSO2, Inc.*lean.enterprise.middleware. >>>>>>> >>>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>>> *E-mail* : [email protected] >>>>>>> *Skype* : @manindae >>>>>>> *Twitter* : @maninda >>>>>>> >>>>>>> On Thu, Jun 29, 2017 at 5:10 PM, Dinusha Dissanayake < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> +Adding architecture >>>>>>>> >>>>>>>> On Thu, Jun 29, 2017 at 5:05 PM, Nuwan Dias <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Let's discuss publicly please, this thread is internal only. >>>>>>>>> >>>>>>>>> On Thu, Jun 29, 2017 at 5:01 PM, Dinusha Dissanayake < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi All, >>>>>>>>>> >>>>>>>>>> In earlier versions of APIM, throttle policy deployment for CEP >>>>>>>>>> was handled using push mechanism meaning APIM itself had to deploy >>>>>>>>>> throttle >>>>>>>>>> policies in CEP. If there are multiple CEP nodes, then the Siddhi >>>>>>>>>> apps/execution plans syncing would be an issue here. >>>>>>>>>> >>>>>>>>>> We were thinking of pull based mechanism from CEP side to >>>>>>>>>> overcome this. To do that, from APIM side we need to add an API to >>>>>>>>>> the core >>>>>>>>>> which allows CEP to get all Siddhi apps for existing throttle >>>>>>>>>> policies. >>>>>>>>>> Then every CEP node will call APIM (once or periodically) and then >>>>>>>>>> it will >>>>>>>>>> check the DB and generate a zip containing all the Siddhi Apps for >>>>>>>>>> existing >>>>>>>>>> throttle policies. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> This API will only need GET resource. The resource path for this >>>>>>>>>> API would be *"export/policies/throttle"*. As mentioned above, >>>>>>>>>> this will return a zip. Mutual SSL would be used to secure the API >>>>>>>>>> since if >>>>>>>>>> we use OAuth2, then the token may expire in a while. >>>>>>>>>> >>>>>>>>>> Exported zip can be manually deployed in the CEP or deployment >>>>>>>>>> can be done using e a curl command. If this needs to be happened >>>>>>>>>> periodically, then a cron job can be written. >>>>>>>>>> >>>>>>>>>> Please provide suggestions for improvements. >>>>>>>>>> >>>>>>>>>> Thanks & Regards. >>>>>>>>>> -- >>>>>>>>>> Dinusha Dissanayake >>>>>>>>>> Software Engineer >>>>>>>>>> WSO2 Inc >>>>>>>>>> Mobile: +94712939439 <+94%2071%20293%209439> >>>>>>>>>> <https://wso2.com/signature> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Nuwan Dias >>>>>>>>> >>>>>>>>> Software Architect - WSO2, Inc. http://wso2.com >>>>>>>>> email : [email protected] >>>>>>>>> Phone : +94 777 775 729 <+94%2077%20777%205729> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Dinusha Dissanayake >>>>>>>> Software Engineer >>>>>>>> WSO2 Inc >>>>>>>> Mobile: +94712939439 <+94%2071%20293%209439> >>>>>>>> <https://wso2.com/signature> >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Lakmal Warusawithana >>>>>> Director - Cloud Architecture; WSO2 Inc. >>>>>> Mobile : +94714289692 <+94%2071%20428%209692> >>>>>> Blogs : https://medium.com/@lakwarus/ >>>>>> http://lakmalsview.blogspot.com/ >>>>>> >>>>>> >>>>> >>>> >>>> >>>> -- >>>> Lakmal Warusawithana >>>> Director - Cloud Architecture; WSO2 Inc. >>>> Mobile : +94714289692 <+94%2071%20428%209692> >>>> Blogs : https://medium.com/@lakwarus/ >>>> http://lakmalsview.blogspot.com/ >>>> >>>> >>> >> >> >> -- >> Lakmal Warusawithana >> Director - Cloud Architecture; WSO2 Inc. >> Mobile : +94714289692 <+94%2071%20428%209692> >> Blogs : https://medium.com/@lakwarus/ >> http://lakmalsview.blogspot.com/ >> >> > -- Lakmal Warusawithana Director - Cloud Architecture; WSO2 Inc. Mobile : +94714289692 Blogs : https://medium.com/@lakwarus/ http://lakmalsview.blogspot.com/
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
