Hi Anthony, Thanks again for the information.
I have played a bit with the the client timeouts and retries and have seen operations being rejected when load is high due to get or put operations. Nevertheless, I have not seen that happen when the load in the server is high due to functions invoked. Is there a reason for not seeing errors with functions or is it just that my test was not good to hit the limits? What if queries are sent with OQL? Do the timeout and retries apply? Is there a similar protection on the native C++ API? I'd be willing to contribute to the improvements you mention. Do you already have ideas? Anything written down? /Alberto On 14/5/19 17:01, Anthony Baker wrote: The primary load limiter between the client tier and the Geode servers is via the max connections limit as noted in this writeup: https://cwiki.apache.org/confluence/display/GEODE/Resource+Management+in+Geode <https://cwiki.apache.org/confluence/display/GEODE/Resource+Management+in+Geode> When the load is sufficiently high, operations may timeout and a geode client will failover to less loaded servers. You can limit the number of retries the client will attempt (each gated by a read timeout) and thus slow down incoming operations. We’re looking into some improvements in the client connection pool to improve both performance and behaviors at the ragged edge when resources are saturated. Contributions welcome! Anthony On May 13, 2019, at 9:02 AM, Alberto Gomez <alberto.go...@est.tech> wrote: Hi Anthony! Thanks a lot for your prompt answer. I think it is great that Geode can preserve the availability and predictable low latency of the cluster when some members are unresponsive by means of the GMS. My question was more targeted to situations in which the load received by the cluster is so high that all members struggle to offer low latency. Under such circumstances, does Geode take any action to back-off some of the incoming load? Thanks in advance, Alberto On 10/5/19 17:52, Anthony Baker wrote: Hi Alberto! Great questions. One of the fundamental characteristics of Geode is its Group Membership System (GMS). You can read more about it here [1]. The membership system ensures that failures due to unresponsive members and/or network partitions are detected quickly. Given that we use synchronous replication for consistent updates, the GMS algorithms fence off unresponsive members to preserve the availability (and predictable low latency) of the cluster as a whole. Another factor of resilience is memory load. Regions can be configured to automatically evict data to disk based on heap usage. In addition, when a Region exceeds a critical memory usage thresholds further updates are blocked until the overload is resolved. Geode clients route operations to cluster members based on connection load. This helps balance cpu load across the entire cluster. Cluster members can set connection maximums to prevent overrunning the available capacity of an individual server. I hope this helps and feel free to keep asking questions :-) Anthony [1] https://cwiki.apache.org/confluence/display/GEODE/Core+Distributed+System+Concepts <https://cwiki.apache.org/confluence/display/GEODE/Core+Distributed+System+Concepts><https://cwiki.apache.org/confluence/display/GEODE/Core+Distributed+System+Concepts <https://cwiki.apache.org/confluence/display/GEODE/Core+Distributed+System+Concepts>> On May 10, 2019, at 3:22 AM, Alberto Gomez <alberto.go...@est.tech> wrote: Hi Geode community! I'd like to know if Geode implements any kind of self-protection against overload. What I mean by this is some mechanism that allows Geode servers (and possibly locators) to reject incoming operations before processing them when it detects that it is not able to handle the amount of operations received in a reasonable way (with reasonable latency and without experiencing processes crashing). The goal would be to make sure that Geode (or some parts of it) do not crash under too heavy load and also that the latency level is always under control at least for the amount of traffic the Geode cluster is supposed to support. If Geode does not offer such mechanism, I would also like to get your opinion about this possible feature, (if you find it interesting) and also on how it could be implemented. One possible approach could be having some measure of the current CPU consumption that allows to decide if a given operation must be processed or not, taking into account the CPU consumption value with respect to an overload threshold. Thanks in advance for your answers, -Alberto
