chetanmeh opened a new pull request #4632: Activation Persister Service
   This PR introduces a new micro service - `Activation Persister Service`. 
This is based on discussion done [here][1] and [here][2]. 
   ## Description
   This service fullfills following objective
   1. Move out the load of persisting Activation results from Invoker to a new 
   2. Enables a more controlled write to backend db in case rate of production 
of activation is more than the rate at which db can persist them within given 
   With this service the activation results would now instead written to a 
Kafka topic and then picked by this service and then saved in db. The approach 
taken here varies with following parameters
   1. Is the activation for a blocking or non blocking call
   2. Is the activation result being persisted from Controller (for 
sequence/composition) or from Invoker
   3. Does the setup stores logs as part of activation or store them seprately
   The implementation consist of following high level components
   1. Activation Persister Service - A new service impl in `core/persister` 
module. It is based on [Alpakka Kafka][3] and reads the activation record from 
Kafka and saves them to db
   2. `KafkaActivationStore` - An extension of `ArtifactActivationStore` which 
also routes to Kafka via `MessageProducer` connector
   3. InvokerReactive - The `send` logic would be modfied (details below)
   4. New Kafka topic `completed-non-blocking`
   Some key aspects which impact the implementation are
   1. Blocking/Non Blocking - Depending on invocation type the nature of 
activation result sent via Kafka `completed` topic varies
       a. Blocking - The `ResultMessage` contains the activation result without 
       b. Non Blocking - Only `CompletionMessage` is sent
   ### Activation Persister Service
   This service implements a streaming flow based on [Alpakka Kafka][3]. It 
runs in 2 modes
   #### ActiationResult without Logs* 
   If the setup does not store the activation with logs then persister service 
would listen for `ResultMessage` on all `completed.*` topics. This would also 
pickup records from `completed-non-blocking` topic.
   Further `KafkaActivationStore` behaviour would change depending on where its 
being used
   1. Within `Controller` - Store all activations to `completed-non-blocking` 
   2. Within `Invoker` - Noop. It would just drop all `store` calls 
   InvokerReactive send flow would send all WhiskActivation result for blocking 
calls to existing `completed<instanceId>` topic. In addition non blocking 
results would also get routed to Kafka to `completed-non-blocking`
   #### ActiationResult with Logs
   ## Related issue and scope
   <!--- Please include a link to a related issue if there is one. -->
   - [ ] I opened an issue to propose and discuss this change (#????)
   ## My changes affect the following components
   <!--- Select below all system components are affected by your change. -->
   <!--- Enter an `x` in all applicable boxes. -->
   - [ ] API
   - [ ] Controller
   - [ ] Message Bus (e.g., Kafka)
   - [ ] Loadbalancer
   - [ ] Invoker
   - [ ] Intrinsic actions (e.g., sequences, conductors)
   - [ ] Data stores (e.g., CouchDB)
   - [ ] Tests
   - [ ] Deployment
   - [ ] CLI
   - [ ] General tooling
   - [ ] Documentation
   ## Types of changes
   <!--- What types of changes does your code introduce? Use `x` in all the 
boxes that apply: -->
   - [ ] Bug fix (generally a non-breaking change which closes an issue).
   - [ ] Enhancement or new feature (adds new functionality).
   - [ ] Breaking change (a bug fix or enhancement which changes existing 
   ## Checklist:
   <!--- Please review the points below which help you make sure you've covered 
all aspects of the change you're making. -->
   - [ ] I signed an [Apache 
   - [ ] I reviewed the [style 
 and followed the recommendations (Travis CI will check :).
   - [ ] I added tests to cover my changes.
   - [ ] My changes require further changes to the documentation.
   - [ ] I updated the documentation where necessary.

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:

With regards,
Apache Git Services

Reply via email to