style95 commented on issue #4626: Allow limiting DB bloat by excluding response 
from Activation record in some cases
URL: https://github.com/apache/openwhisk/issues/4626#issuecomment-532503097
 
 
   @sven-lange-last All makes sense to me.
   
   We also introduced a flag(`volatile`) to let users decide whether to store 
their activations or not when we were using CouchDB as an activation store.
   At that time, it was an optional flag so if a user does not explicitly set 
the flag, all activations were stored. Since it was disabled by default, no one 
tried to use it. We had to urge heavy users to use the flag.
   Since the main problem is the scalability of the system, users are supposed 
not to consider the system issue.
   They always wanted to see their results no matter of the 
environment(dev/production).
   Even though we enable the feature by default, we can not stop users 
disabling the flag(store activations) all the time.
   
   We stored failed activations even though the flag is enabled for better 
debugging, but anyway users wanted to see their successful activation results 
sometimes. Not only the activation results but they also wanted to see metadata 
such as `initTime`, `waitTime`, etc. (I think if we want to reduce the size of 
activaitons, it could be one option to store metadata and just skip results.)
   
   So we dropped `volatile` flag, stopped using `CouchDbActivationStore` and 
introduced `ElasticSearchActivationStore`.
   
   Regarding reducing the number of activations, I think there are also two 
aspects.
   1. Reducing the number of activations/second
   2. Reducing the number of stored activations.
   
   With CouchDB, we observed issues in both cases.
   (Even with 10GB data, CouchDB sometimes started dragging.)
   
   Regarding number 1, I think even though we reduce the amount, there would be 
a limitation as the size of cluster grows. We anyway need to handle(store) some 
portion of activations and if the cluster scales, anyway this portion would be 
beyond the CouchDB can handle. I still agree that this would be a good option 
no matter which datastores are used, but anyway we need to secure some level of 
scalability at some point.
   
   Regarding number 2, most of the users query relatively recent activations.
   They tend to query activations within 1 month ~ 3 months.
   It would not be cost-effective to store all activation data during 1 year ~ 
2 years.
   So we decided to keep relatively "recent", but "all" activations.
   (I think many other OW operators already took a similar approach.)
   
   So I think our datastore should have enough scalability to handle some level 
of requests/s and it should have the ability to take care of "cold" data which 
is rarely accessed.
   ElasticSearch is a great option for this.
   
   
   With regard to log collection, I have been curious whether there "is" a case 
where one function generates 10MB logs. I am not sure it is realistic. As the 
granularity level of invocation is small in the serverless world, I think the 
size of the logs should be small as well. 
   
   Currently, logs are collected asynchronously aside from activation storing 
even though logs are also included in activation data. I think this is because 
the log size can be up to 10MB. 
   If we can limit the size of the maximum logs to relatively small one such as 
1 MB or 512KB,(I think this is also quite big enough) we can just store logs 
with activations with one request.
   
   Storing logs with ELK is great but if we store them along with activations 
in ElasticSearch, it would give another value to users. Users can query data 
using logs in conjunction with metadata.
   (e.g: They might want to see logs whose `waitTime` is bigger than 1s.)
   
   
   
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Reply via email to