[
https://issues.apache.org/jira/browse/CASSANDRA-14318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paulo Motta updated CASSANDRA-14318:
------------------------------------
Summary: Fix query pager DEBUG log leak causing hit in paged reads
throughput (was: Debug logging can create massive performance issues)
> Fix query pager DEBUG log leak causing hit in paged reads throughput
> --------------------------------------------------------------------
>
> Key: CASSANDRA-14318
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14318
> Project: Cassandra
> Issue Type: Bug
> Reporter: Alexander Dejanovski
> Assignee: Alexander Dejanovski
> Priority: Major
> Labels: lhf, performance
> Fix For: 2.2.13
>
> Attachments: cassandra-2.2-debug.yaml, debuglogging.png, flame22
> nodebug sjk svg.png, flame22-nodebug-sjk.svg, flame22-sjk.svg,
> flame_graph_snapshot.png
>
>
> Debug logging can involve in many cases (especially very low latency ones) a
> very important overhead on the read path in 2.2 as we've seen when upgrading
> clusters from 2.0 to 2.2.
> The performance impact was especially noticeable on the client side metrics,
> where p99 could go up to 10 times higher, while ClientRequest metrics
> recorded by Cassandra didn't show any overhead.
> Below shows latencies recorded on the client side with debug logging on
> first, and then without it :
> !debuglogging.png!
> We generated a flame graph before turning off debug logging that shows the
> read call stack is dominated by debug logging :
> !flame_graph_snapshot.png!
> I've attached the original flame graph for exploration.
> Once disabled, the new flame graph shows that the read call stack gets
> extremely thin, which is further confirmed by client recorded metrics :
> !flame22 nodebug sjk svg.png!
> The query pager code has been reworked since 3.0 and it looks like
> log.debug() calls are gone there, but for 2.2 users and to prevent such
> issues to appear with default settings, I really think debug logging should
> be disabled by default.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
