[jira] [Commented] (CASSANDRA-19534) unbounded queues in native transport requests lead to node instability

[ https://issues.apache.org/jira/browse/CASSANDRA-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17849812#comment-17849812 ] Alex Petrov commented on CASSANDRA-19534: - [~e.dimitrova] I believe it does. I was just

[jira] [Commented] (CASSANDRA-19534) unbounded queues in native transport requests lead to node instability

[ https://issues.apache.org/jira/browse/CASSANDRA-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848777#comment-17848777 ] Ekaterina Dimitrova commented on CASSANDRA-19534: - {quote}+1 LGTM (dropped a couple

[jira] [Commented] (CASSANDRA-19534) unbounded queues in native transport requests lead to node instability

[ https://issues.apache.org/jira/browse/CASSANDRA-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846031#comment-17846031 ] Caleb Rackliffe commented on CASSANDRA-19534: - +1 LGTM (dropped a couple more little

[jira] [Commented] (CASSANDRA-19534) unbounded queues in native transport requests lead to node instability

[ https://issues.apache.org/jira/browse/CASSANDRA-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845930#comment-17845930 ] Alex Petrov commented on CASSANDRA-19534: - Pushed a new commit that should address your

[jira] [Commented] (CASSANDRA-19534) unbounded queues in native transport requests lead to node instability

[ https://issues.apache.org/jira/browse/CASSANDRA-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1788#comment-1788 ] Caleb Rackliffe commented on CASSANDRA-19534: - Finished my first pass and left a bunch

[jira] [Commented] (CASSANDRA-19534) unbounded queues in native transport requests lead to node instability

[ https://issues.apache.org/jira/browse/CASSANDRA-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843389#comment-17843389 ] Alex Petrov commented on CASSANDRA-19534: - These tests look really good! I haven’t expected

[jira] [Commented] (CASSANDRA-19534) unbounded queues in native transport requests lead to node instability

[ https://issues.apache.org/jira/browse/CASSANDRA-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843382#comment-17843382 ] Jon Haddad commented on CASSANDRA-19534: Last one.  I set the timeout for 2 seconds and

[jira] [Commented] (CASSANDRA-19534) unbounded queues in native transport requests lead to node instability

[ https://issues.apache.org/jira/browse/CASSANDRA-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843379#comment-17843379 ] Jon Haddad commented on CASSANDRA-19534: For one last test, I've updated cassandra2 to use

[jira] [Commented] (CASSANDRA-19534) unbounded queues in native transport requests lead to node instability

[ https://issues.apache.org/jira/browse/CASSANDRA-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843377#comment-17843377 ] Jon Haddad commented on CASSANDRA-19534: Switched the entire cluster to use the patch,

[jira] [Commented] (CASSANDRA-19534) unbounded queues in native transport requests lead to node instability

[ https://issues.apache.org/jira/browse/CASSANDRA-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843376#comment-17843376 ] Jon Haddad commented on CASSANDRA-19534: I'm running these two workloads concurrently:

[jira] [Commented] (CASSANDRA-19534) unbounded queues in native transport requests lead to node instability

[ https://issues.apache.org/jira/browse/CASSANDRA-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843375#comment-17843375 ] Alex Petrov commented on CASSANDRA-19534: - Thank you for looking into this. Just to make

[jira] [Commented] (CASSANDRA-19534) unbounded queues in native transport requests lead to node instability

[ https://issues.apache.org/jira/browse/CASSANDRA-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843366#comment-17843366 ] Jon Haddad commented on CASSANDRA-19534: Gotcha. Updated here and it's fired up. Currently

[jira] [Commented] (CASSANDRA-19534) unbounded queues in native transport requests lead to node instability

[ https://issues.apache.org/jira/browse/CASSANDRA-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843358#comment-17843358 ] Brandon Williams commented on CASSANDRA-19534: -- [~rustyrazorblade] the parameter is

[jira] [Commented] (CASSANDRA-19534) unbounded queues in native transport requests lead to node instability

[ https://issues.apache.org/jira/browse/CASSANDRA-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843356#comment-17843356 ] Jon Haddad commented on CASSANDRA-19534: I'm juggling a couple things right now, so I

[jira] [Commented] (CASSANDRA-19534) unbounded queues in native transport requests lead to node instability

[ https://issues.apache.org/jira/browse/CASSANDRA-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842171#comment-17842171 ] Alex Petrov commented on CASSANDRA-19534: - This is great, thank you for testing! My 100s

[jira] [Commented] (CASSANDRA-19534) unbounded queues in native transport requests lead to node instability

[ https://issues.apache.org/jira/browse/CASSANDRA-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842164#comment-17842164 ] Brandon Williams commented on CASSANDRA-19534: -- bq. I'd suggest setting cql_start_time

[jira] [Commented] (CASSANDRA-19534) unbounded queues in native transport requests lead to node instability

[ https://issues.apache.org/jira/browse/CASSANDRA-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842114#comment-17842114 ] Jon Haddad commented on CASSANDRA-19534: I can fire it up this week. > unbounded queues

[jira] [Commented] (CASSANDRA-19534) unbounded queues in native transport requests lead to node instability

[ https://issues.apache.org/jira/browse/CASSANDRA-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842112#comment-17842112 ] Alex Petrov commented on CASSANDRA-19534: - [~brandon.williams] [~rustyrazorblade] would you

[jira] [Commented] (CASSANDRA-19534) unbounded queues in native transport requests lead to node instability

[ https://issues.apache.org/jira/browse/CASSANDRA-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840173#comment-17840173 ] Alex Petrov commented on CASSANDRA-19534: - Sorry for the lack of clarity; today there’s no

[jira] [Commented] (CASSANDRA-19534) unbounded queues in native transport requests lead to node instability

[ https://issues.apache.org/jira/browse/CASSANDRA-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840171#comment-17840171 ] Brandon Williams commented on CASSANDRA-19534: -- I think this all sounds good, though

[jira] [Commented] (CASSANDRA-19534) unbounded queues in native transport requests lead to node instability

[ https://issues.apache.org/jira/browse/CASSANDRA-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840058#comment-17840058 ] Alex Petrov commented on CASSANDRA-19534: - The main change is the introduction of (currently

[jira] [Commented] (CASSANDRA-19534) unbounded queues in native transport requests lead to node instability

[ https://issues.apache.org/jira/browse/CASSANDRA-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17838960#comment-17838960 ] Alex Petrov commented on CASSANDRA-19534: - I guess this can explain it. We have 32 read, 32

[jira] [Commented] (CASSANDRA-19534) unbounded queues in native transport requests lead to node instability

[ https://issues.apache.org/jira/browse/CASSANDRA-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17838957#comment-17838957 ] Brandon Williams commented on CASSANDRA-19534: -- FWIW, that was a single node. >

[jira] [Commented] (CASSANDRA-19534) unbounded queues in native transport requests lead to node instability

[ https://issues.apache.org/jira/browse/CASSANDRA-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1783#comment-1783 ] Alex Petrov commented on CASSANDRA-19534: - Talked to [~brandon.williams] and checked the

[jira] [Commented] (CASSANDRA-19534) unbounded queues in native transport requests lead to node instability

[ https://issues.apache.org/jira/browse/CASSANDRA-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17838360#comment-17838360 ] Alex Petrov commented on CASSANDRA-19534: - Do you have observability data from the cluster

[jira] [Commented] (CASSANDRA-19534) unbounded queues in native transport requests lead to node instability

[ https://issues.apache.org/jira/browse/CASSANDRA-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17838359#comment-17838359 ] Brandon Williams commented on CASSANDRA-19534: -- The p99 from easy-cass-stress does

[jira] [Commented] (CASSANDRA-19534) unbounded queues in native transport requests lead to node instability

[ https://issues.apache.org/jira/browse/CASSANDRA-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17838355#comment-17838355 ] Alex Petrov commented on CASSANDRA-19534: - I am a bit surprised to see that on 4.1 we seem

[jira] [Commented] (CASSANDRA-19534) unbounded queues in native transport requests lead to node instability

[ https://issues.apache.org/jira/browse/CASSANDRA-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17837458#comment-17837458 ] Brandon Williams commented on CASSANDRA-19534: -- I do have to use a higher rate, but

[jira] [Commented] (CASSANDRA-19534) unbounded queues in native transport requests lead to node instability

[ https://issues.apache.org/jira/browse/CASSANDRA-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17837445#comment-17837445 ] Jon Haddad commented on CASSANDRA-19534: OK.  The issue is how C* recovers from being

[jira] [Commented] (CASSANDRA-19534) unbounded queues in native transport requests lead to node instability

[ https://issues.apache.org/jira/browse/CASSANDRA-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17837443#comment-17837443 ] Brandon Williams commented on CASSANDRA-19534: -- I did run the first workload to

[jira] [Commented] (CASSANDRA-19534) unbounded queues in native transport requests lead to node instability

[ https://issues.apache.org/jira/browse/CASSANDRA-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17837437#comment-17837437 ] Jon Haddad commented on CASSANDRA-19534: [~brandon.williams] not offhand.  Are you running

[jira] [Commented] (CASSANDRA-19534) unbounded queues in native transport requests lead to node instability

[ https://issues.apache.org/jira/browse/CASSANDRA-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17837427#comment-17837427 ] Brandon Williams commented on CASSANDRA-19534: -- [~rustyrazorblade] do you have more

[jira] [Commented] (CASSANDRA-19534) unbounded queues in native transport requests lead to node instability

[ https://issues.apache.org/jira/browse/CASSANDRA-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17837419#comment-17837419 ] Alex Petrov commented on CASSANDRA-19534: - Sounds good, I'll tag you as soon as I have it

[jira] [Commented] (CASSANDRA-19534) unbounded queues in native transport requests lead to node instability

[ https://issues.apache.org/jira/browse/CASSANDRA-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17837417#comment-17837417 ] Jon Haddad commented on CASSANDRA-19534: I don't have a patch for this, will wait for yours. 

[jira] [Commented] (CASSANDRA-19534) unbounded queues in native transport requests lead to node instability

[ https://issues.apache.org/jira/browse/CASSANDRA-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17837416#comment-17837416 ] Alex Petrov commented on CASSANDRA-19534: - [~rustyrazorblade] oh yes, that would exist in a

[jira] [Commented] (CASSANDRA-19534) unbounded queues in native transport requests lead to node instability

[ https://issues.apache.org/jira/browse/CASSANDRA-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17837406#comment-17837406 ] Jon Haddad commented on CASSANDRA-19534: In the example I listed, this was a single node, so

[jira] [Commented] (CASSANDRA-19534) unbounded queues in native transport requests lead to node instability

[ https://issues.apache.org/jira/browse/CASSANDRA-19534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17834560#comment-17834560 ] Alex Petrov commented on CASSANDRA-19534: - I believe I know the root cause of this. I will