[
https://issues.apache.org/jira/browse/CASSANDRA-14318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paulo Motta updated CASSANDRA-14318:
------------------------------------
Resolution: Fixed
Reviewer: Paulo Motta
Fix Version/s: (was: 3.11.x)
(was: 4.x)
(was: 3.0.x)
(was: 2.2.x)
2.2.13
Status: Resolved (was: Patch Available)
{quote}I think anyone performing benchmarks for Cassandra changes should be
aware that the predefined mode isn't relevant and that a user defined test
should be used (maybe we should create one that would be used as standard
benchmark).
{quote}
Good find! Can you check if this is the case in trunk, and if so maybe open a
lhf ticket to change that?
{quote}For the record, the same tests on 3.11.2 didn't show any notable
performance difference between debug on and off
{quote}
Nice to know we managed to handle all debug/verbose log leaks there. It will be
easier to maintain this after CASSANDRA-14326.
{quote}here's the patch if you're willing to review/commit it, and the unit
test results in CircleCI.
{quote}
Thanks for the patch, experiments and analysis! Even though 2.2 is on critical
fixes only mode, 50% is a significant performance hit on throughput for this
workload, and since the patch is pretty simple I don't see a reason not to
commit it.
CI looks good. I added a CHANGES.txt not and committed as
{{ac77e5e7742548f7c7c25da3923841f59d4b2713}} to cassandra-2.2 branch.
> Debug logging can create massive performance issues
> ---------------------------------------------------
>
> Key: CASSANDRA-14318
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14318
> Project: Cassandra
> Issue Type: Bug
> Reporter: Alexander Dejanovski
> Assignee: Alexander Dejanovski
> Priority: Major
> Labels: lhf, performance
> Fix For: 2.2.13
>
> Attachments: cassandra-2.2-debug.yaml, debuglogging.png, flame22
> nodebug sjk svg.png, flame22-nodebug-sjk.svg, flame22-sjk.svg,
> flame_graph_snapshot.png
>
>
> Debug logging can involve in many cases (especially very low latency ones) a
> very important overhead on the read path in 2.2 as we've seen when upgrading
> clusters from 2.0 to 2.2.
> The performance impact was especially noticeable on the client side metrics,
> where p99 could go up to 10 times higher, while ClientRequest metrics
> recorded by Cassandra didn't show any overhead.
> Below shows latencies recorded on the client side with debug logging on
> first, and then without it :
> !debuglogging.png!
> We generated a flame graph before turning off debug logging that shows the
> read call stack is dominated by debug logging :
> !flame_graph_snapshot.png!
> I've attached the original flame graph for exploration.
> Once disabled, the new flame graph shows that the read call stack gets
> extremely thin, which is further confirmed by client recorded metrics :
> !flame22 nodebug sjk svg.png!
> The query pager code has been reworked since 3.0 and it looks like
> log.debug() calls are gone there, but for 2.2 users and to prevent such
> issues to appear with default settings, I really think debug logging should
> be disabled by default.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
