[jira] [Commented] (CASSANDRA-8116) HSHA fails with default rpc_max_threads setting
[ https://issues.apache.org/jira/browse/CASSANDRA-8116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14188625#comment-14188625 ] Peter Haggerty commented on CASSANDRA-8116: --- The latest 2.0.x release of Cassandra using hsha with default settings either stalls after a few minutes of operation or crashes. This does not seem like it should have a priority of Minor. This is a major problem. The longer that 2.0.11 is the latest version the bigger the problem becomes for new users and existing users that have automation and high levels of trust in minor version upgrades. HSHA fails with default rpc_max_threads setting --- Key: CASSANDRA-8116 URL: https://issues.apache.org/jira/browse/CASSANDRA-8116 Project: Cassandra Issue Type: Bug Components: Core Reporter: Mike Adamson Assignee: Tyler Hobbs Priority: Minor Fix For: 2.0.12, 2.1.2 Attachments: 8116-throw-exc-2.0.txt, 8116.txt The HSHA server fails with 'Out of heap space' error if the rpc_max_threads is left at its default setting (unlimited) in cassandra.yaml. I'm not proposing any code change for this but have submitted a patch for a comment change in cassandra.yaml to indicate that rpc_max_threads needs to be changed if you use HSHA. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8116) HSHA fails with default rpc_max_threads setting
[ https://issues.apache.org/jira/browse/CASSANDRA-8116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14182490#comment-14182490 ] Jeremy Hanna commented on CASSANDRA-8116: - is there any reason to have a default of unlimited? it just seems like it would be a simple change to have a reasonable default. HSHA fails with default rpc_max_threads setting --- Key: CASSANDRA-8116 URL: https://issues.apache.org/jira/browse/CASSANDRA-8116 Project: Cassandra Issue Type: Bug Components: Core Reporter: Mike Adamson Assignee: Mike Adamson Priority: Minor Fix For: 2.0.11, 2.1.1 Attachments: 8116-throw-exc-2.0.txt, 8116.txt The HSHA server fails with 'Out of heap space' error if the rpc_max_threads is left at its default setting (unlimited) in cassandra.yaml. I'm not proposing any code change for this but have submitted a patch for a comment change in cassandra.yaml to indicate that rpc_max_threads needs to be changed if you use HSHA. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8116) HSHA fails with default rpc_max_threads setting
[ https://issues.apache.org/jira/browse/CASSANDRA-8116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14182811#comment-14182811 ] Mike Adamson commented on CASSANDRA-8116: - +1 on the latest patch because that will be far clearer to the user than the OOM they get at the moment. HSHA fails with default rpc_max_threads setting --- Key: CASSANDRA-8116 URL: https://issues.apache.org/jira/browse/CASSANDRA-8116 Project: Cassandra Issue Type: Bug Components: Core Reporter: Mike Adamson Assignee: Mike Adamson Priority: Minor Fix For: 2.0.11, 2.1.1 Attachments: 8116-throw-exc-2.0.txt, 8116.txt The HSHA server fails with 'Out of heap space' error if the rpc_max_threads is left at its default setting (unlimited) in cassandra.yaml. I'm not proposing any code change for this but have submitted a patch for a comment change in cassandra.yaml to indicate that rpc_max_threads needs to be changed if you use HSHA. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8116) HSHA fails with default rpc_max_threads setting
[ https://issues.apache.org/jira/browse/CASSANDRA-8116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14183080#comment-14183080 ] Tyler Hobbs commented on CASSANDRA-8116: bq. is there any reason to have a default of unlimited? it just seems like it would be a simple change to have a reasonable default. I'm not sure about the rationale for that default. Would you mind opening a new ticket to discuss a better default? I'm not sure if it's something we would want to change in 2.0 or 2.1. HSHA fails with default rpc_max_threads setting --- Key: CASSANDRA-8116 URL: https://issues.apache.org/jira/browse/CASSANDRA-8116 Project: Cassandra Issue Type: Bug Components: Core Reporter: Mike Adamson Assignee: Tyler Hobbs Priority: Minor Fix For: 2.0.12, 2.1.2 Attachments: 8116-throw-exc-2.0.txt, 8116.txt The HSHA server fails with 'Out of heap space' error if the rpc_max_threads is left at its default setting (unlimited) in cassandra.yaml. I'm not proposing any code change for this but have submitted a patch for a comment change in cassandra.yaml to indicate that rpc_max_threads needs to be changed if you use HSHA. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8116) HSHA fails with default rpc_max_threads setting
[ https://issues.apache.org/jira/browse/CASSANDRA-8116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14178689#comment-14178689 ] Tyler Hobbs commented on CASSANDRA-8116: bq. I suppose you could check and throw if the value is Integer.MAX_VALUE but you aren't going to be able to check every value. unlimited is the default value, so I think there's still quite a bit of value in checking for just that. I'll put together a patch. HSHA fails with default rpc_max_threads setting --- Key: CASSANDRA-8116 URL: https://issues.apache.org/jira/browse/CASSANDRA-8116 Project: Cassandra Issue Type: Bug Components: Core Reporter: Mike Adamson Assignee: Mike Adamson Priority: Minor Fix For: 2.0.11, 2.1.1 Attachments: 8116.txt The HSHA server fails with 'Out of heap space' error if the rpc_max_threads is left at its default setting (unlimited) in cassandra.yaml. I'm not proposing any code change for this but have submitted a patch for a comment change in cassandra.yaml to indicate that rpc_max_threads needs to be changed if you use HSHA. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8116) HSHA fails with default rpc_max_threads setting
[ https://issues.apache.org/jira/browse/CASSANDRA-8116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14177193#comment-14177193 ] Michael Shuler commented on CASSANDRA-8116: --- Just as an FYI - I bisected dtest thrift_hsha_test.ThriftHSHATest.test_6285 failures to the CASSANDRA-7594 commit, and had to increase CCM_MAX_HEAP_SIZE to 4G before the ccm nodes would start successfully without a heap OOM. HSHA fails with default rpc_max_threads setting --- Key: CASSANDRA-8116 URL: https://issues.apache.org/jira/browse/CASSANDRA-8116 Project: Cassandra Issue Type: Bug Components: Core Reporter: Mike Adamson Assignee: Mike Adamson Priority: Minor Fix For: 2.0.11, 2.1.1 Attachments: 8116.txt The HSHA server fails with 'Out of heap space' error if the rpc_max_threads is left at its default setting (unlimited) in cassandra.yaml. I'm not proposing any code change for this but have submitted a patch for a comment change in cassandra.yaml to indicate that rpc_max_threads needs to be changed if you use HSHA. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8116) HSHA fails with default rpc_max_threads setting
[ https://issues.apache.org/jira/browse/CASSANDRA-8116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14177206#comment-14177206 ] Michael Shuler commented on CASSANDRA-8116: --- Setting rpc_max_threads=20 in the thrift_hsha_test test_6285 does appear to keep dtest running with the default ccm heap. HSHA fails with default rpc_max_threads setting --- Key: CASSANDRA-8116 URL: https://issues.apache.org/jira/browse/CASSANDRA-8116 Project: Cassandra Issue Type: Bug Components: Core Reporter: Mike Adamson Assignee: Mike Adamson Priority: Minor Fix For: 2.0.11, 2.1.1 Attachments: 8116.txt The HSHA server fails with 'Out of heap space' error if the rpc_max_threads is left at its default setting (unlimited) in cassandra.yaml. I'm not proposing any code change for this but have submitted a patch for a comment change in cassandra.yaml to indicate that rpc_max_threads needs to be changed if you use HSHA. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8116) HSHA fails with default rpc_max_threads setting
[ https://issues.apache.org/jira/browse/CASSANDRA-8116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14176336#comment-14176336 ] Mike Adamson commented on CASSANDRA-8116: - It's pretty much guaranteed to OOM because the number of handlers per SelectorThread is based on the the max pool size which for the default is Integer.MAX_VALUE. This change happened as part of CASSANDRA-7594. I suppose you could check and throw if the value is Integer.MAX_VALUE but you aren't going to be able to check every value. It also happens when the node is started so is pretty immediate which is why I suggested a doc change rather than a code change. HSHA fails with default rpc_max_threads setting --- Key: CASSANDRA-8116 URL: https://issues.apache.org/jira/browse/CASSANDRA-8116 Project: Cassandra Issue Type: Bug Components: Core Reporter: Mike Adamson Assignee: Mike Adamson Priority: Minor Fix For: 2.0.11, 2.1.1 Attachments: 8116.txt The HSHA server fails with 'Out of heap space' error if the rpc_max_threads is left at its default setting (unlimited) in cassandra.yaml. I'm not proposing any code change for this but have submitted a patch for a comment change in cassandra.yaml to indicate that rpc_max_threads needs to be changed if you use HSHA. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8116) HSHA fails with default rpc_max_threads setting
[ https://issues.apache.org/jira/browse/CASSANDRA-8116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14175498#comment-14175498 ] Tyler Hobbs commented on CASSANDRA-8116: Is it guaranteed to OOM? Can't we just check for that combination and provide a sensible error instead of OOMing and letting the user figure it out? HSHA fails with default rpc_max_threads setting --- Key: CASSANDRA-8116 URL: https://issues.apache.org/jira/browse/CASSANDRA-8116 Project: Cassandra Issue Type: Bug Components: Core Reporter: Mike Adamson Assignee: Mike Adamson Priority: Minor Fix For: 2.0.11, 2.1.1 Attachments: 8116.txt The HSHA server fails with 'Out of heap space' error if the rpc_max_threads is left at its default setting (unlimited) in cassandra.yaml. I'm not proposing any code change for this but have submitted a patch for a comment change in cassandra.yaml to indicate that rpc_max_threads needs to be changed if you use HSHA. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
