[jira] [Commented] (CASSANDRA-8969) Add indication in cassandra.yaml that rpc timeouts going too high will cause memory build up
[ https://issues.apache.org/jira/browse/CASSANDRA-8969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17422060#comment-17422060 ] Andres de la Peña commented on CASSANDRA-8969: -- [~bereng] last changes look good to me, +1 > Add indication in cassandra.yaml that rpc timeouts going too high will cause > memory build up > > > Key: CASSANDRA-8969 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8969 > Project: Cassandra > Issue Type: Improvement > Components: Local/Config >Reporter: Jeremy Hanna >Assignee: Berenguer Blasi >Priority: Low > Labels: lhf > Fix For: 3.11.x, 4.0.x, 4.x > > Attachments: 8969.txt > > > It would be helpful to communicate that setting the rpc timeouts too high may > cause memory problems on the server as it can become overloaded and has to > retain the in flight requests in memory. I'll get this done but just adding > the ticket as a placeholder for memory. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-8969) Add indication in cassandra.yaml that rpc timeouts going too high will cause memory build up
[ https://issues.apache.org/jira/browse/CASSANDRA-8969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17419022#comment-17419022 ] Berenguer Blasi commented on CASSANDRA-8969: Anybody up for driving this across the finish line? Otherwise I can take over. > Add indication in cassandra.yaml that rpc timeouts going too high will cause > memory build up > > > Key: CASSANDRA-8969 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8969 > Project: Cassandra > Issue Type: Improvement > Components: Local/Config >Reporter: Jeremy Hanna >Assignee: Jeremy Hanna >Priority: Low > Labels: lhf > Fix For: 3.11.x > > Attachments: 8969.txt > > > It would be helpful to communicate that setting the rpc timeouts too high may > cause memory problems on the server as it can become overloaded and has to > retain the in flight requests in memory. I'll get this done but just adding > the ticket as a placeholder for memory. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-8969) Add indication in cassandra.yaml that rpc timeouts going too high will cause memory build up
[ https://issues.apache.org/jira/browse/CASSANDRA-8969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582668#comment-16582668 ] Jeremy Hanna commented on CASSANDRA-8969: - Any more thoughts on this change to make sure people don't unwittingly cause heap problems by setting the timeout extremely high? > Add indication in cassandra.yaml that rpc timeouts going too high will cause > memory build up > > > Key: CASSANDRA-8969 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8969 > Project: Cassandra > Issue Type: Improvement > Components: Configuration >Reporter: Jeremy Hanna >Assignee: Jeremy Hanna >Priority: Minor > Labels: lhf > Fix For: 3.11.x > > Attachments: 8969.txt > > > It would be helpful to communicate that setting the rpc timeouts too high may > cause memory problems on the server as it can become overloaded and has to > retain the in flight requests in memory. I'll get this done but just adding > the ticket as a placeholder for memory. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-8969) Add indication in cassandra.yaml that rpc timeouts going too high will cause memory build up
[
https://issues.apache.org/jira/browse/CASSANDRA-8969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15208747#comment-15208747
]
Jeremy Hanna commented on CASSANDRA-8969:
-
Not to bikeshed too much but what do you think of making it a little more
explicit?
{quote}
+# For this reason, you should avoid putting these settings too high. Of course
+# putting them too low is equally ill-advised since clients could get timeouts
even
+# for successful operations just because the timeout setting is too tight.
+# In other words, if you are timing out requests because of underlying
resource constraints
+# then increasing the timeout will just cause more problems.
{quote}
> Add indication in cassandra.yaml that rpc timeouts going too high will cause
> memory build up
>
>
> Key: CASSANDRA-8969
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8969
> Project: Cassandra
> Issue Type: Improvement
> Components: Configuration
>Reporter: Jeremy Hanna
>Assignee: Jeremy Hanna
>Priority: Minor
> Labels: lhf
> Fix For: 3.x
>
> Attachments: 8969.txt
>
>
> It would be helpful to communicate that setting the rpc timeouts too high may
> cause memory problems on the server as it can become overloaded and has to
> retain the in flight requests in memory. I'll get this done but just adding
> the ticket as a placeholder for memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8969) Add indication in cassandra.yaml that rpc timeouts going too high will cause memory build up
[ https://issues.apache.org/jira/browse/CASSANDRA-8969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15208691#comment-15208691 ] J.B. Langston commented on CASSANDRA-8969: -- I agree this could be a good warning to have. I've seen a lot of customers naively increase the timeout. Usually it's caused by I/O not keeping up with requests, but a lot of users won't take the time to figure that out. They just see their application timing out and they see something in cassandra.yaml called timeout so they increase it without thinking of the cost. Now they have GC death spiral and OOM to contend with in addition to the original problem. > Add indication in cassandra.yaml that rpc timeouts going too high will cause > memory build up > > > Key: CASSANDRA-8969 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8969 > Project: Cassandra > Issue Type: Improvement > Components: Configuration >Reporter: Jeremy Hanna >Assignee: Jeremy Hanna >Priority: Minor > Labels: lhf > Fix For: 3.x > > Attachments: 8969.txt > > > It would be helpful to communicate that setting the rpc timeouts too high may > cause memory problems on the server as it can become overloaded and has to > retain the in flight requests in memory. I'll get this done but just adding > the ticket as a placeholder for memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8969) Add indication in cassandra.yaml that rpc timeouts going too high will cause memory build up
[ https://issues.apache.org/jira/browse/CASSANDRA-8969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15208680#comment-15208680 ] Jeremiah Jordan commented on CASSANDRA-8969: [~slebresne] the issue he is talking about is when a node gets overloaded such that requests start backing up. If I am issuing 100 req/s and something happens to stall things for 10 seconds (say some other node that my requests go to stalls), with a 1 second timeout I only ever have 100 requests in flight. With a 10 second timeout I could have 1000 requests in flight on the coordinator. > Add indication in cassandra.yaml that rpc timeouts going too high will cause > memory build up > > > Key: CASSANDRA-8969 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8969 > Project: Cassandra > Issue Type: Improvement > Components: Configuration >Reporter: Jeremy Hanna >Assignee: Jeremy Hanna >Priority: Minor > Labels: lhf > Fix For: 3.x > > Attachments: 8969.txt > > > It would be helpful to communicate that setting the rpc timeouts too high may > cause memory problems on the server as it can become overloaded and has to > retain the in flight requests in memory. I'll get this done but just adding > the ticket as a placeholder for memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8969) Add indication in cassandra.yaml that rpc timeouts going too high will cause memory build up
[ https://issues.apache.org/jira/browse/CASSANDRA-8969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15208605#comment-15208605 ] Sylvain Lebresne commented on CASSANDRA-8969: - bq. you could potentially have 2x in-flight requests being kept in memory Not totally sure I follow. Are we talking about the request objects? Because those are really tiny and I don't that being that relevant. Besides, the number of max in-flight queries is currently really limited by the number of native transport threads, and I'm not sure to see in which way a bigger rpc timeouts changes much here. Anyway, feel free to check the attached patch to see if we're talking of the same thing with different words. But if we aren't, I'm not sure to understand the problem you're mentioning. > Add indication in cassandra.yaml that rpc timeouts going too high will cause > memory build up > > > Key: CASSANDRA-8969 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8969 > Project: Cassandra > Issue Type: Improvement > Components: Configuration >Reporter: Jeremy Hanna >Assignee: Jeremy Hanna >Priority: Minor > Labels: lhf > Fix For: 3.x > > Attachments: 8969.txt > > > It would be helpful to communicate that setting the rpc timeouts too high may > cause memory problems on the server as it can become overloaded and has to > retain the in flight requests in memory. I'll get this done but just adding > the ticket as a placeholder for memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8969) Add indication in cassandra.yaml that rpc timeouts going too high will cause memory build up
[ https://issues.apache.org/jira/browse/CASSANDRA-8969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15208544#comment-15208544 ] Jeremy Hanna commented on CASSANDRA-8969: - Sorry about the lag in responsiveness on my part. The idea was that if you set it to 2x, then you could potentially have 2x in-flight requests being kept in memory on the server in cases where requests get queued up. I think that's not immediately obvious to people who increase it to just avoid client-side timeouts. > Add indication in cassandra.yaml that rpc timeouts going too high will cause > memory build up > > > Key: CASSANDRA-8969 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8969 > Project: Cassandra > Issue Type: Improvement > Components: Configuration >Reporter: Jeremy Hanna >Assignee: Jeremy Hanna >Priority: Minor > Labels: lhf > Fix For: 3.x > > Attachments: 8969.txt > > > It would be helpful to communicate that setting the rpc timeouts too high may > cause memory problems on the server as it can become overloaded and has to > retain the in flight requests in memory. I'll get this done but just adding > the ticket as a placeholder for memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8969) Add indication in cassandra.yaml that rpc timeouts going too high will cause memory build up
[ https://issues.apache.org/jira/browse/CASSANDRA-8969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15154387#comment-15154387 ] Aleksey Yeschenko commented on CASSANDRA-8969: -- Can you cook up a patch for it? Should be trivial enough. Thanks. > Add indication in cassandra.yaml that rpc timeouts going too high will cause > memory build up > > > Key: CASSANDRA-8969 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8969 > Project: Cassandra > Issue Type: Improvement > Components: Configuration >Reporter: Jeremy Hanna >Assignee: Jeremy Hanna >Priority: Minor > Labels: lhf > > It would be helpful to communicate that setting the rpc timeouts too high may > cause memory problems on the server as it can become overloaded and has to > retain the in flight requests in memory. I'll get this done but just adding > the ticket as a placeholder for memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
