[
https://issues.apache.org/jira/browse/CASSANDRA-9558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14575808#comment-14575808
]
Andy Tolbert edited comment on CASSANDRA-9558 at 6/6/15 4:50 PM:
-----------------------------------------------------------------
{quote}This logic makes sense on the server-side, since there is likely more
useful work for the server to do in the meantime, but on the client we're just
delaying the server from getting started on processing our requests. We should
be batching, but not delaying. i.e. flushing as soon as we have written all
pending messages.{quote}
One motivation for coalescing/batching on the driver side is that in Netty 4.0
[channel.write no longer implicitly
flushes|http://netty.io/wiki/new-and-noteworthy-in-4.0.html#wiki-h4-20] and it
now must be explicit. We found during testing that if you explicitly flush
outside of the netty event loop there is sizable performance cost. I added a
row to my table in the previous comment showing the op rate degrading with
coalescing disabled via -Dcom.datastax.driver.DISABLE_COALESCING=true (also see
attached [^atolber-trunk-driver-coalescing-disabled.txt]).
That being said, if there is a more optimal way of doing this I think we should
look into it. I think it would be interesting to see how things perform if the
driver simply submits a flush on the event loop and see what netty does for us
(I'll try this out) compared to coalescing or flushing outside the event loop.
If there is no perceived benefit of coalescing vs. submitting a flush in the
event loop we could consider changing the default behavior or removing the
functionality. (EDIT: Evidently this is exactly what writeAndFlush does which
is what the driver is using when coalescing is disabled, but i'll keep
exploring alternatives)
I think with my findings in my [previous
comment|https://issues.apache.org/jira/browse/CASSANDRA-9558?focusedCommentId=14575616&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14575616],
this issue should be resolved by downgrading the protocol version to v2 (until
the driver is updated to support connection sizing with v3+) and setting core
connections per host to 8.
was (Author: andrew.tolbert):
{quote}This logic makes sense on the server-side, since there is likely more
useful work for the server to do in the meantime, but on the client we're just
delaying the server from getting started on processing our requests. We should
be batching, but not delaying. i.e. flushing as soon as we have written all
pending messages.{quote}
One motivation for coalescing/batching on the driver side is that in Netty 4.0
[channel.write no longer implicitly
flushes|http://netty.io/wiki/new-and-noteworthy-in-4.0.html#wiki-h4-20] and it
now must be explicit. We found during testing that if you explicitly flush
outside of the netty event loop there is sizable performance cost. I added a
row to my table in the previous comment showing the op rate degrading with
coalescing disabled via -Dcom.datastax.driver.DISABLE_COALESCING=true (also see
attached [^atolber-trunk-driver-coalescing-disabled.txt]).
That being said, if there is a more optimal way of doing this I think we should
look into it. I think it would be interesting to see how things perform if the
driver simply submits a flush on the event loop and see what netty does for us
(I'll try this out) compared to coalescing or flushing outside the event loop.
If there is no perceived benefit of coalescing vs. submitting a flush in the
event loop we could consider changing the default behavior or removing the
functionality.
I think with my findings in my [previous
comment|https://issues.apache.org/jira/browse/CASSANDRA-9558?focusedCommentId=14575616&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14575616],
this issue should be resolved by downgrading the protocol version to v2 (until
the driver is updated to support connection sizing with v3+) and setting core
connections per host to 8.
> Cassandra-stress regression in 2.2
> ----------------------------------
>
> Key: CASSANDRA-9558
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9558
> Project: Cassandra
> Issue Type: Bug
> Reporter: Alan Boudreault
> Priority: Blocker
> Attachments: 2.1.log, 2.2.log, CASSANDRA-9558-2.patch,
> CASSANDRA-9558-ProtocolV2.patch, atolber-CASSANDRA-9558-stress.tgz,
> atolber-trunk-driver-coalescing-disabled.txt,
> stress-2.1-java-driver-2.0.9.2.log, stress-2.1-java-driver-2.2+PATCH.log,
> stress-2.1-java-driver-2.2.log, stress-2.2-java-driver-2.2+PATCH.log,
> stress-2.2-java-driver-2.2.log
>
>
> We are seeing some regression in performance when using cassandra-stress 2.2.
> You can see the difference at this url:
> http://riptano.github.io/cassandra_performance/graph_v5/graph.html?stats=stress_regression.json&metric=op_rate&operation=1_write&smoothing=1&show_aggregates=true&xmin=0&xmax=108.57&ymin=0&ymax=168147.1
> The cassandra version of the cluster doesn't seem to have any impact.
> //cc [~tjake] [~benedict]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
