[jira] [Updated] (CASSANDRA-3670) provide "red flags" JMX instrumentation
[ https://issues.apache.org/jira/browse/CASSANDRA-3670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] C. Scott Andreas updated CASSANDRA-3670: Component/s: Observability > provide "red flags" JMX instrumentation > --- > > Key: CASSANDRA-3670 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3670 > Project: Cassandra > Issue Type: Improvement > Components: Observability >Reporter: Peter Schuller >Priority: Minor > > As discussed in CASSANDRA-3641, it would be nice to expose through JMX > certain information which is almost without exception indicative of something > being wrong with the node or cluster. > In the CASSANDRA-3641 case, it was the detection of corrupt counter shards. > Other examples include: > * Number of times the selection of files to compact was adjusted due to disk > space heuristics > * Number of times compaction has failed > * Any I/O error reading from or writing to disk (the work here is collecting, > not exposing, so maybe not in an initial version) > * Any data skipped due to checksum mismatches (when checksumming is being > used); e.g., "number of skips". > * Any arbitrary exception at least in certain code paths (compaction, scrub, > cleanup for starters) > Probably other things. > The motivation is that if we have clear and obvious indications that > something truly is wrong, it seems suboptimal to just leave that information > in the log somewhere, for someone to discover later when something else broke > as a result and a human investigates. You might argue that one should use > non-trivial log analysis to detect these things, but I highly doubt a lot of > people do this and it seems very wasteful to require that in comparison to > just providing the MBean. > It is important to note that the *lack* of a certain problem being advertised > in this MBean is not supposed to be indicative of a *lack* of a problem. > Rather, the point is that to the extent we can easily do so, it is nice to > have a clear method of communicating to monitoring systems where there *is* a > clear indication of something being wrong. > The main part of this ticket is not to cover everything under the sun, but > rather to reach agreement on adding an MBean where these types of indicators > can be collected. Individual counters can then be added over time as one > thinks of them. > I propose: > * Create an org.apache.cassandra.db.RedFlags MBean > * Populate with a few things to begin with. > I'll submit the patch if there is agreement. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-3670) provide red flags JMX instrumentation
[ https://issues.apache.org/jira/browse/CASSANDRA-3670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-3670: -- Reviewer: (was: Brandon Williams) Assignee: (was: Tyler Hobbs) provide red flags JMX instrumentation --- Key: CASSANDRA-3670 URL: https://issues.apache.org/jira/browse/CASSANDRA-3670 Project: Cassandra Issue Type: Improvement Reporter: Peter Schuller Priority: Minor As discussed in CASSANDRA-3641, it would be nice to expose through JMX certain information which is almost without exception indicative of something being wrong with the node or cluster. In the CASSANDRA-3641 case, it was the detection of corrupt counter shards. Other examples include: * Number of times the selection of files to compact was adjusted due to disk space heuristics * Number of times compaction has failed * Any I/O error reading from or writing to disk (the work here is collecting, not exposing, so maybe not in an initial version) * Any data skipped due to checksum mismatches (when checksumming is being used); e.g., number of skips. * Any arbitrary exception at least in certain code paths (compaction, scrub, cleanup for starters) Probably other things. The motivation is that if we have clear and obvious indications that something truly is wrong, it seems suboptimal to just leave that information in the log somewhere, for someone to discover later when something else broke as a result and a human investigates. You might argue that one should use non-trivial log analysis to detect these things, but I highly doubt a lot of people do this and it seems very wasteful to require that in comparison to just providing the MBean. It is important to note that the *lack* of a certain problem being advertised in this MBean is not supposed to be indicative of a *lack* of a problem. Rather, the point is that to the extent we can easily do so, it is nice to have a clear method of communicating to monitoring systems where there *is* a clear indication of something being wrong. The main part of this ticket is not to cover everything under the sun, but rather to reach agreement on adding an MBean where these types of indicators can be collected. Individual counters can then be added over time as one thinks of them. I propose: * Create an org.apache.cassandra.db.RedFlags MBean * Populate with a few things to begin with. I'll submit the patch if there is agreement. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (CASSANDRA-3670) provide red flags JMX instrumentation
[ https://issues.apache.org/jira/browse/CASSANDRA-3670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-3670: -- Reviewer: brandon.williams (was: slebresne) Assignee: Tyler Hobbs (was: Peter Schuller) WDYT [~thobbs]? provide red flags JMX instrumentation --- Key: CASSANDRA-3670 URL: https://issues.apache.org/jira/browse/CASSANDRA-3670 Project: Cassandra Issue Type: Improvement Reporter: Peter Schuller Assignee: Tyler Hobbs Priority: Minor As discussed in CASSANDRA-3641, it would be nice to expose through JMX certain information which is almost without exception indicative of something being wrong with the node or cluster. In the CASSANDRA-3641 case, it was the detection of corrupt counter shards. Other examples include: * Number of times the selection of files to compact was adjusted due to disk space heuristics * Number of times compaction has failed * Any I/O error reading from or writing to disk (the work here is collecting, not exposing, so maybe not in an initial version) * Any data skipped due to checksum mismatches (when checksumming is being used); e.g., number of skips. * Any arbitrary exception at least in certain code paths (compaction, scrub, cleanup for starters) Probably other things. The motivation is that if we have clear and obvious indications that something truly is wrong, it seems suboptimal to just leave that information in the log somewhere, for someone to discover later when something else broke as a result and a human investigates. You might argue that one should use non-trivial log analysis to detect these things, but I highly doubt a lot of people do this and it seems very wasteful to require that in comparison to just providing the MBean. It is important to note that the *lack* of a certain problem being advertised in this MBean is not supposed to be indicative of a *lack* of a problem. Rather, the point is that to the extent we can easily do so, it is nice to have a clear method of communicating to monitoring systems where there *is* a clear indication of something being wrong. The main part of this ticket is not to cover everything under the sun, but rather to reach agreement on adding an MBean where these types of indicators can be collected. Individual counters can then be added over time as one thinks of them. I propose: * Create an org.apache.cassandra.db.RedFlags MBean * Populate with a few things to begin with. I'll submit the patch if there is agreement. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3670) provide red flags JMX instrumentation
[ https://issues.apache.org/jira/browse/CASSANDRA-3670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Schuller updated CASSANDRA-3670: -- Reviewer: slebresne provide red flags JMX instrumentation --- Key: CASSANDRA-3670 URL: https://issues.apache.org/jira/browse/CASSANDRA-3670 Project: Cassandra Issue Type: Improvement Reporter: Peter Schuller Assignee: Peter Schuller Priority: Minor As discussed in CASSANDRA-3641, it would be nice to expose through JMX certain information which is almost without exception indicative of something being wrong with the node or cluster. In the CASSANDRA-3641 case, it was the detection of corrupt counter shards. Other examples include: * Number of times the selection of files to compact was adjusted due to disk space heuristics * Number of times compaction has failed * Any I/O error reading from or writing to disk (the work here is collecting, not exposing, so maybe not in an initial version) * Any data skipped due to checksum mismatches (when checksumming is being used); e.g., number of skips. * Any arbitrary exception at least in certain code paths (compaction, scrub, cleanup for starters) Probably other things. The motivation is that if we have clear and obvious indications that something truly is wrong, it seems suboptimal to just leave that information in the log somewhere, for someone to discover later when something else broke as a result and a human investigates. You might argue that one should use non-trivial log analysis to detect these things, but I highly doubt a lot of people do this and it seems very wasteful to require that in comparison to just providing the MBean. It is important to note that the *lack* of a certain problem being advertised in this MBean is not supposed to be indicative of a *lack* of a problem. Rather, the point is that to the extent we can easily do so, it is nice to have a clear method of communicating to monitoring systems where there *is* a clear indication of something being wrong. The main part of this ticket is not to cover everything under the sun, but rather to reach agreement on adding an MBean where these types of indicators can be collected. Individual counters can then be added over time as one thinks of them. I propose: * Create an org.apache.cassandra.db.RedFlags MBean * Populate with a few things to begin with. I'll submit the patch if there is agreement. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
