[
https://issues.apache.org/jira/browse/CASSANDRA-10241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739122#comment-14739122
]
Jeff Jirsa commented on CASSANDRA-10241:
----------------------------------------
Appreciate the ping, [~jbellis]
Pretty torn on this on the operator side. I have to imagine most production
operators (larger than RF=N=3) probably:
- Rely on JMX for understanding most day to day state (tracking exceptions,
compaction, gc, etc), and only go to logs when things break, which supports the
idea that having more verbose logs probably would help us when we're debugging
something unusual.
- Rely at least partly on {{tail -f}} during those times, so binary log would
hurt immediate access to the logs when they're actually needed (probably more
often with the "dozens" of nodes guys than the "hundreds" / "thousands")
- Would probably disable the extra verbose logging if it hit us with more than
a few % performance impact
If things go bad and the log is there and it helps us find some weird behavior,
then it's great. But if it kills performance we'd turn it off, and if it has to
be binary to avoid killing performance, we wouldn't be able to {{tail -f}} it,
which would interrupt at least some of the normal operator behavior.
I'm not sure there's a happy answer for everyone here on the operations side,
so as long as it can be turned off if it's really a performance hit, and
doesn't remove otherwise useful information from the existing log, I'm sure
everyone will adapt over time.
Personally, I might be tempted to keep a ring buffer of selective debug logged
statements somewhere and allow those to be flushed to disk on command
("nodetool debug"), and then that can be uploaded to support / jira as
requested, or flushed on jvm shutdown for historical records if desired - no
disk hit unless requested, just a small memory penalty? Of course, that makes
it far less useful for looking far into the past, so I'm not sure if that's a
win, either.
> Keep a separate production debug log for troubleshooting
> --------------------------------------------------------
>
> Key: CASSANDRA-10241
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10241
> Project: Cassandra
> Issue Type: New Feature
> Components: Config
> Reporter: Jonathan Ellis
> Assignee: Paulo Motta
> Fix For: 2.1.x, 2.2.x, 3.0.x
>
>
> [~aweisberg] had the suggestion to keep a separate debug log for aid in
> troubleshooting, not intended for regular human consumption but where we can
> log things that might help if something goes wrong.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
