[ 
https://issues.apache.org/jira/browse/CASSANDRA-13265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15986786#comment-15986786
 ] 

Ariel Weisberg edited comment on CASSANDRA-13265 at 4/27/17 3:33 PM:
---------------------------------------------------------------------

You don't need to get all the tests passing. Only the ones broken specifically 
by your changes. So you have to run the tests on the base branch 
(cassandra-2.2, cassandra-3.0, cassandra-3.11, cassandra-trunk) or compare the 
results to 
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-2.2-test-all/ 
or the one specific to the version of your branch.

You also need to be aware that a test might just be unreliable in CircleCI. The 
unit tests seem to be reliable to me when I was running them with my 
circle.yml, but I am not sure about the one on trunk.

Certainly it runs additional tests which weren't running all the time before 
Circle and I don't think they were reliable when we started running them.


was (Author: aweisberg):
You don't need to get all the tests passing. Only the ones broken specifically 
by your changes. So you have to run the tests on the base branch 
(cassandra-2.2, cassandra-3.0, cassandra-3.11, cassandra-trunk). Compare the 
results to 
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-2.2-test-all/ 
or the one specific to the version of your branch.

You also need to be aware that a test might just be unreliable in CircleCI. The 
unit tests seem to be reliable to me when I was running them with my 
circle.yml, but I am not sure about the one on trunk.

Certainly it runs additional tests which weren't running all the time before 
Circle and I don't think they were reliable when we started running them.

> Expiration in OutboundTcpConnection can block the reader Thread
> ---------------------------------------------------------------
>
>                 Key: CASSANDRA-13265
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13265
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: Cassandra 3.0.9
> Java HotSpot(TM) 64-Bit Server VM version 25.112-b15 (Java version 
> 1.8.0_112-b15)
> Linux 3.16
>            Reporter: Christian Esken
>            Assignee: Christian Esken
>             Fix For: 3.0.x
>
>         Attachments: cassandra.pb-cache4-dus.2017-02-17-19-36-26.chist.xz, 
> cassandra.pb-cache4-dus.2017-02-17-19-36-26.td.xz
>
>
> I observed that sometimes a single node in a Cassandra cluster fails to 
> communicate to the other nodes. This can happen at any time, during peak load 
> or low load. Restarting that single node from the cluster fixes the issue.
> Before going in to details, I want to state that I have analyzed the 
> situation and am already developing a possible fix. Here is the analysis so 
> far:
> - A Threaddump in this situation showed  324 Threads in the 
> OutboundTcpConnection class that want to lock the backlog queue for doing 
> expiration.
> - A class histogram shows 262508 instances of 
> OutboundTcpConnection$QueuedMessage.
> What is the effect of it? As soon as the Cassandra node has reached a certain 
> amount of queued messages, it starts thrashing itself to death. Each of the 
> Thread fully locks the Queue for reading and writing by calling 
> iterator.next(), making the situation worse and worse.
> - Writing: Only after 262508 locking operation it can progress with actually 
> writing to the Queue.
> - Reading: Is also blocked, as 324 Threads try to do iterator.next(), and 
> fully lock the Queue
> This means: Writing blocks the Queue for reading, and readers might even be 
> starved which makes the situation even worse.
> -----
> The setup is:
>  - 3-node cluster
>  - replication factor 2
>  - Consistency LOCAL_ONE
>  - No remote DC's
>  - high write throughput (100000 INSERT statements per second and more during 
> peak times).
>  



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

Reply via email to