[
https://issues.apache.org/jira/browse/CASSANDRA-12920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15675205#comment-15675205
]
Ariel Weisberg commented on CASSANDRA-12920:
--------------------------------------------
The best way to confirm is to correlate GCs from the GC log with timestamps
from traces. I am assuming those are time UUIDs.
> Possible messaging latency caused by not flushing in time
> ---------------------------------------------------------
>
> Key: CASSANDRA-12920
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12920
> Project: Cassandra
> Issue Type: Bug
> Reporter: Simon Zhou
> Assignee: Simon Zhou
> Priority: Minor
> Fix For: 2.2.5
>
>
> Not sure if this can be improved. We're using Cassandra 2.2.5. Cassandra
> considers messages whose payload size <= 64KB as small messages. For response
> for such as digest request, it's very likely a small message. Even quite a
> few such responses cannot fill in the output buffer and thus trigger a flush.
> One possible issue is that, we use conditions count == 1 && backlog.isEmpty()
> to decide whether to flush the output stream. That means if the backlog is
> empty, we won't flush the output stream even we drained, say, 10 messages
> whose sizes sum < 64K. This may cause delay for small messages. Shouldn't we
> flush after writing the last drained message if backlog is empty? Of course
> if backlog is not empty, we can continue draining more messages and very
> likely trigger a flush very soon.
> Here are some tracing events that show the latency. Be noted that the
> involved nodes are in the same data center whose round trip network latency
> is well below 1ms. Alao we have disabled message coalescing due to
> CASSANDRA-12676.
> {noformat}
> 126571b0-aaa6-11e6-ab2b-77eb6f529c59 | 127c7c28-aaa6-11e6-941f-f3a2e2759bd2
> | Enqueuing response to /*******32.28 |
> *******29.3 | 1199 | SharedPool-Worker-4
> 126571b0-aaa6-11e6-ab2b-77eb6f529c59 | 127cca45-aaa6-11e6-941f-f3a2e2759bd2
> | Sending REQUEST_RESPONSE message to /*******32.28 |
> *******29.3 | 2667 | MessagingService-Outgoing-/*******32.28
> 126571b0-aaa6-11e6-ab2b-77eb6f529c59 | 127cca40-aaa6-11e6-ab2b-77eb6f529c59
> | REQUEST_RESPONSE message received from /*******29.3 |
> *******32.28 | 152635 | MessagingService-Incoming-/*******29.3
> 126571b0-aaa6-11e6-ab2b-77eb6f529c59 | 127cca41-aaa6-11e6-ab2b-77eb6f529c59
> | Processing response from /*******29.3 |
> *******32.28 | 152665 | SharedPool-Worker-45
> 1af31490-aaa6-11e6-ab2b-77eb6f529c59 | 1b08e68a-aaa6-11e6-8dce-951d39203586
> | Enqueuing response to /*******32.28 |
> *******29.5 | 532 | SharedPool-Worker-178
> 1af31490-aaa6-11e6-ab2b-77eb6f529c59 | 1b090d90-aaa6-11e6-8dce-951d39203586
> | Sending REQUEST_RESPONSE message to /*******32.28 |
> *******29.5 | 579 | MessagingService-Outgoing-/*******32.28
> 1af31490-aaa6-11e6-ab2b-77eb6f529c59 | 1b090d90-aaa6-11e6-ab2b-77eb6f529c59
> | REQUEST_RESPONSE message received from /*******29.5 |
> *******32.28 | 143865 | MessagingService-Incoming-/*******29.5
> 1af31490-aaa6-11e6-ab2b-77eb6f529c59 | 1b090d91-aaa6-11e6-ab2b-77eb6f529c59
> | Processing response from /*******29.5 |
> *******32.28 | 143908 |
> SharedPool-Worker-41
> 1af58590-aaa6-11e6-ab2b-77eb6f529c59 | 1b0934a8-aaa6-11e6-8dce-951d39203586
> | Enqueuing response to /*******32.28 |
> *******29.5 | 1632 |
> SharedPool-Worker-159 1af58590-aaa6-11e6-ab2b-77eb6f529c59 |
> 1b0934a9-aaa6-11e6-8dce-951d39203586 | Sending
> REQUEST_RESPONSE message to /*******32.28 | *******29.5 | 1660 |
> MessagingService
> -Outgoing-/*******32.28 1af58590-aaa6-11e6-ab2b-77eb6f529c59 |
> 1b095bb0-aaa6-11e6-ab2b-77eb6f529c59 | REQUEST_RESPONSE
> message received from /*******29.5 | *******32.28 | 129813 |
> MessagingServic
> e-Incoming-/*******29.5 1af58590-aaa6-11e6-ab2b-77eb6f529c59 |
> 1b095bb1-aaa6-11e6-ab2b-77eb6f529c59 |
> Processing response from /*******29.5 | *******32.28 | 129835 |
>
> SharedPool-Worker-3
> 3010cfc0-aaa6-11e6-ab2b-77eb6f529c59 | 3029fd17-aaa6-11e6-a5cc-8b24438aa927
> | Enqueuing response to /*******32.28 |
> *******30.8 | 1686 | SharedPool-Worker-271
> 3010cfc0-aaa6-11e6-ab2b-77eb6f529c59 | 302ae770-aaa6-11e6-a5cc-8b24438aa927
> | Sending REQUEST_RESPONSE message to /*******32.28 |
> *******30.8 | 6938 | MessagingService-Outgoing-/*******32.28
> 3010cfc0-aaa6-11e6-ab2b-77eb6f529c59 | 302ae770-aaa6-11e6-ab2b-77eb6f529c59
> | REQUEST_RESPONSE message received from /*******30.8 |
> *******32.28 | 171380 | MessagingService-Incoming-/*******30.8
> 3010cfc0-aaa6-11e6-ab2b-77eb6f529c59 | 302ae771-aaa6-11e6-ab2b-77eb6f529c59
> | Processing response from /*******30.8 |
> *******32.28 | 171472 | SharedPool-Worker-55
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
