Re: Relieve CouchDB on high load
Thanks for the update - will review the PR again. On Mon, Nov 26, 2018 at 9:26 AM Christian Bickel wrote: > Hi Rodric, > > thanks a lot for your feedback. I changed my PR to use the quota instead of > switching it off completely. > > Now the controller checks, how many activations have been written in the > current minute. The remaining quota is passed on. On trying to save an > activation (of a trigger, a sequence or an activation) this quota is > checked and only saved, if it is above 0. > > Greetings > Christian > > Am Mi., 24. Okt. 2018 um 17:38 Uhr schrieb Rodric Rabbah >: > > > Thanks for the additional information Christian - I took a look at the PR > > and added a few comments. I think you can approach this as another quota > > (number of activations stored per minute or per hour) and setting the > quota > > to 0 would disable all stores. > > > > I also don't like the way this was done in the artifact store - it's a > fast > > hack, but bleeds through the abstractions. We should not do this in my > > opinion. At the least, the invoker (and controller) wasted effort and > held > > up resources gathering logs etc only to be thrown away later. > > > > -r > > >
Re: Relieve CouchDB on high load
Hi Rodric, thanks a lot for your feedback. I changed my PR to use the quota instead of switching it off completely. Now the controller checks, how many activations have been written in the current minute. The remaining quota is passed on. On trying to save an activation (of a trigger, a sequence or an activation) this quota is checked and only saved, if it is above 0. Greetings Christian Am Mi., 24. Okt. 2018 um 17:38 Uhr schrieb Rodric Rabbah : > Thanks for the additional information Christian - I took a look at the PR > and added a few comments. I think you can approach this as another quota > (number of activations stored per minute or per hour) and setting the quota > to 0 would disable all stores. > > I also don't like the way this was done in the artifact store - it's a fast > hack, but bleeds through the abstractions. We should not do this in my > opinion. At the least, the invoker (and controller) wasted effort and held > up resources gathering logs etc only to be thrown away later. > > -r >
Re: Relieve CouchDB on high load
Thanks for the additional information Christian - I took a look at the PR and added a few comments. I think you can approach this as another quota (number of activations stored per minute or per hour) and setting the quota to 0 would disable all stores. I also don't like the way this was done in the artifact store - it's a fast hack, but bleeds through the abstractions. We should not do this in my opinion. At the least, the invoker (and controller) wasted effort and held up resources gathering logs etc only to be thrown away later. -r
Re: Relieve CouchDB on high load
Markus, Christian - it's not clear to me from the first note if Christian is proposing excluding all activation data (including the metadata, not just the logs) from couchdb, for a namespace. I think this is a very crude patch and wonder why we don't address the underlying issue instead. I haven't looked at the PR which could answer my question. -r On Tue, Oct 23, 2018 at 10:23 AM Markus Thömmes wrote: > Hi Christian, > > given the recent work towards getting logs to a separate store to relieve > CouchDB and to make it possible to move it into a store that's more > appropriate for logging load and some other bits that are already > implemented (removing the DB based polling, being able to disable log > collection), I think it makes sense to be able to disable activation > writing as well! In that sense, yes I do agree on going forward with the > proposal + implementation. > > One thing that has me worried a bit is the way we're handling global, > namespaced and per-action limits today. I'm wondering if at some point > we'll need to consolidate those and make them cascade reliably. For this > specific knob, for example, I think it would make sense to have one at the > system level and on the action/trigger level accordingly. For some of our > limits that might be true today already, I feel like it might not be for > all though (although I admittedly haven't double-checked). > > Cheers, > Markus > > Am Di., 23. Okt. 2018 um 14:22 Uhr schrieb Christian Bickel < > cbic...@apache.org>: > > > Hi developpers, > > > > in some performance tests in the past we've seen, that there are the > > following issues with Cloudant/CouchDB: > > - not all activations can be stored (there are error logs in the invoker, > > that storing was not possible) > > - during bursts, CouchDB needs to process each document to update all > views > > in the activations-DB. If CouchDB is not able to process them > immediately, > > because of a queue, calling these views returns an error or the result > will > > be outdated (on calling them with stale). This has the impact, that there > > is a delay for all users of the system until their activation appear > after > > calling `activation list`. > > > > To not have negative impact of some high-load users on the system the > > proposal is, to not store activations in activations-store for some > > specified namespaces. > > My proposal of an implementation is to put this flag into the > > limits-document in subjects database. > > This means, it can only be set by the administrator with wskadmin. > > > > I already opened a PR with this proposal: > > https://github.com/apache/incubator-openwhisk/pull/4078 > > > > Do you agree on going forward with this proposal and implementation? > > > > Thanks a lot in advance for your feedback. > > > > Greetings > > Christian > > >
Re: Relieve CouchDB on high load
Hi Markus, thanks a lot for your response. I definitely agree, that all limits should be reviewed and reworked. The reason why I've chosen this limit as per-namespace-limit and not as per-action-limit is, to give the operator of Openwhisk the ability to protect the own database. If the limit would be implemented on a per-action basis, the user could always change it back and flood the database during very high load intervals. but adding this as per-action-limit, like ningyougan proposed in the PR, makes definetly sense. But I would see it as separate PR. Greetings Christian Am Di., 23. Okt. 2018 um 16:23 Uhr schrieb Markus Thömmes < markusthoem...@apache.org>: > Hi Christian, > > given the recent work towards getting logs to a separate store to relieve > CouchDB and to make it possible to move it into a store that's more > appropriate for logging load and some other bits that are already > implemented (removing the DB based polling, being able to disable log > collection), I think it makes sense to be able to disable activation > writing as well! In that sense, yes I do agree on going forward with the > proposal + implementation. > > One thing that has me worried a bit is the way we're handling global, > namespaced and per-action limits today. I'm wondering if at some point > we'll need to consolidate those and make them cascade reliably. For this > specific knob, for example, I think it would make sense to have one at the > system level and on the action/trigger level accordingly. For some of our > limits that might be true today already, I feel like it might not be for > all though (although I admittedly haven't double-checked). > > Cheers, > Markus > > Am Di., 23. Okt. 2018 um 14:22 Uhr schrieb Christian Bickel < > cbic...@apache.org>: > > > Hi developpers, > > > > in some performance tests in the past we've seen, that there are the > > following issues with Cloudant/CouchDB: > > - not all activations can be stored (there are error logs in the invoker, > > that storing was not possible) > > - during bursts, CouchDB needs to process each document to update all > views > > in the activations-DB. If CouchDB is not able to process them > immediately, > > because of a queue, calling these views returns an error or the result > will > > be outdated (on calling them with stale). This has the impact, that there > > is a delay for all users of the system until their activation appear > after > > calling `activation list`. > > > > To not have negative impact of some high-load users on the system the > > proposal is, to not store activations in activations-store for some > > specified namespaces. > > My proposal of an implementation is to put this flag into the > > limits-document in subjects database. > > This means, it can only be set by the administrator with wskadmin. > > > > I already opened a PR with this proposal: > > https://github.com/apache/incubator-openwhisk/pull/4078 > > > > Do you agree on going forward with this proposal and implementation? > > > > Thanks a lot in advance for your feedback. > > > > Greetings > > Christian > > >
Re: Relieve CouchDB on high load
Hi Christian, given the recent work towards getting logs to a separate store to relieve CouchDB and to make it possible to move it into a store that's more appropriate for logging load and some other bits that are already implemented (removing the DB based polling, being able to disable log collection), I think it makes sense to be able to disable activation writing as well! In that sense, yes I do agree on going forward with the proposal + implementation. One thing that has me worried a bit is the way we're handling global, namespaced and per-action limits today. I'm wondering if at some point we'll need to consolidate those and make them cascade reliably. For this specific knob, for example, I think it would make sense to have one at the system level and on the action/trigger level accordingly. For some of our limits that might be true today already, I feel like it might not be for all though (although I admittedly haven't double-checked). Cheers, Markus Am Di., 23. Okt. 2018 um 14:22 Uhr schrieb Christian Bickel < cbic...@apache.org>: > Hi developpers, > > in some performance tests in the past we've seen, that there are the > following issues with Cloudant/CouchDB: > - not all activations can be stored (there are error logs in the invoker, > that storing was not possible) > - during bursts, CouchDB needs to process each document to update all views > in the activations-DB. If CouchDB is not able to process them immediately, > because of a queue, calling these views returns an error or the result will > be outdated (on calling them with stale). This has the impact, that there > is a delay for all users of the system until their activation appear after > calling `activation list`. > > To not have negative impact of some high-load users on the system the > proposal is, to not store activations in activations-store for some > specified namespaces. > My proposal of an implementation is to put this flag into the > limits-document in subjects database. > This means, it can only be set by the administrator with wskadmin. > > I already opened a PR with this proposal: > https://github.com/apache/incubator-openwhisk/pull/4078 > > Do you agree on going forward with this proposal and implementation? > > Thanks a lot in advance for your feedback. > > Greetings > Christian >
