Hi folks, I apologize again. My inexperience with the mailing list is quite obvious :( I will send again - as pure text
Regards, -- Erez Erez Hadad, PhD Cloud System Technologies IBM Research - Haifa email: [email protected] phone: +972-4-829-6509 From: "Erez Hadad" <[email protected]> To: [email protected] Date: 06/12/2018 18:10 Subject: Re: Shared Context in OpenWhisk Hi Olivier, Thank you. The presentation is available at the link posted by Priti. Let me try to explain what I mean by "orthogonal". A shared context is propagated along a flow as it happens. It does not prescribe the flow itself, it only evolves with it, over invocations and rules. On the other hand, a conductor action implements a certain flow of execution, as defined in its code. Therefore, adding a feature that relies on shared context (tracing, shared console, stack trace, ... - see presentation) using conductor actions requires instrumenting all conductor actions with this context - not likely feasible. We cannot foresee all the possible applications of shared context beyond the 7-8 options I listed in the presentation, so we cannot instrument all the conductor actions to support all of them. On the other hand, I'm not sure generic shared context can be implemented by conductor actions due to limitations already mentioned - not covering rules, not sharing information inside derived actions, only conductor can inject additional context. So, each mechanism (conductor actions and shared context) has independent value, and should stand on its own, independently of the other. I agree that some features of shared context can be implemented to some degree in conductor actions. However, the implementation would probably be unique to conductor actions (as an external orchestrator) and the results are not quite the same. For example, a stack trace can be managed by the conductor action, but I'm not sure what good it is, because the stack is always at depth of 2 (conductor action + current derived action). If a derived action invokes another action internally, sync/async/rule, I don't see how the conductor stack trace can capture this, because it cannot inject anything to the derived action to capture the internal call. Agreed ;) Implementation cost - not sure it's that great, but I need to study the issue more. I put one slide in the backup section of the presentation listing the main components involved, and this could be further discussed. For example, I think that environment variables are provided as a package to the runtimes, so adding one more env. var. should make no code difference in the runtimes. Alternatively, if using external storage (e.g., Redis) for all the context, the key can simply be the activation id, which is already there, so no code change at either invoker or runtimes. A separate library or service can implement the access to the context. The only suggestion I made for the structure of the shared context is using key/value pairs, allow different features to be piggybacked independently in the same shared context - for example, enable both tracing, unified console and QoS scheduling, each using its own keys. Beyond that, implementers of each feature can choose how design their data. For example, stack trace can be a flat identifier (later combined with time stamps to infer order), or contain the activation id of the caller to actually form the stack using pointers, or many other options. Regards, -- Erez From: "Olivier Tardieu" <[email protected]> To: [email protected] Date: 05/12/2018 19:39 Subject: Re: Shared Context in OpenWhisk Erez, I did not see an attachment or a link in your email. --- Some thoughts about the relationship between contexts and conductor actions. 1. Contexts are very useful. Love it! 2. Conductor actions depend on managing contexts. So contexts and conductors are not orthogonal concerns. 3. Conductor actions make it possible to hide unnecessary context from an invoked action but preserve the context so it is still available downstream. So one can leverage conductor action for _some_ context management including managing a context stack in an invocation tree. 4. Conductor actions piggy back on the parameter object to manage contexts and rely on conventions about reserved parameter names. Having a separate context table (key/value pair) would have clear advantages. No disagreement here. :) 5. For conductor actions, we considered implementing a context separate from the parameter object but chickened out basically because it touches a lot of OpenWhisk components. This potentially affects all the runtimes, the messaging, the activation records, the payload and record sizes, the controller memory footprint, the quotas... 6. Conductor actions give the flexibility to customize the context management because we can write arbitrary action code to manage context propagation. Should we manage the context as a stack? Should it be flat? Should an asynchronous invocation preserve or drop the current context? It all depends on use cases. If the context becomes a pure runtime facility however, implicitly managed and propagated, we loose some of that flexibility. So we need to make sure that we identify and support all the important patterns beforehand (including obviously conductor action contexts). Cheers, Olivier From: "Erez Hadad" <[email protected]> To: [email protected] Date: 12/05/2018 11:58 AM Subject: Shared Context in OpenWhisk Hi folks, Following today's call, here is the presentation. Note the additional implementation details in the "Backup" section. Please comment! Regards, -- Erez Erez Hadad, PhD Cloud System Technologies IBM Research - Haifa email: [email protected] phone: +972-4-829-6509
