Alexander Dejanovski created CASSANDRA-14318:
------------------------------------------------
Summary: Debug logging can create massive performance issues
Key: CASSANDRA-14318
URL: https://issues.apache.org/jira/browse/CASSANDRA-14318
Project: Cassandra
Issue Type: Bug
Reporter: Alexander Dejanovski
Fix For: 2.2.x, 3.0.x, 3.11.x, 4.x
Attachments: debuglogging.png, flame22 nodebug sjk svg.png,
flame22-nodebug-sjk.svg, flame22-sjk.svg, flame_graph_snapshot.png
Debug logging can involve in many cases (especially very low latency ones) a
very important overhead on the read path in 2.2 as we've seen when upgrading
clusters from 2.0 to 2.2.
The performance impact was especially noticeable on the client side metrics,
where p99 could go up to 10 times higher, while ClientRequest metrics recorded
by Cassandra didn't show any overhead.
Below shows latencies recorded on the client side with debug logging on first,
and then without it :
!debuglogging.png!
We generated a flame graph before turning off debug logging that shows the read
call stack is dominated by debug logging :
!flame_graph_snapshot.png!
I've attached the original flame graph for exploration.
Once disabled, the new flame graph shows that the read call stack gets
extremely thin, which is further confirmed by client recorded metrics :
!flame22 nodebug sjk svg.png!
The query pager code has been reworked since 3.0 and it looks like log.debug()
calls are gone there, but for 2.2 users and to prevent such issues to appear
with default settings, I really think debug logging should be disabled by
default.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
