[jira] [Commented] (CASSANDRA-15582) 4.0 quality testing: metrics
[
https://issues.apache.org/jira/browse/CASSANDRA-15582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17304809#comment-17304809
]
Benjamin Lerer commented on CASSANDRA-15582:
I kept CASSANDRA-16192 and CASSANDRA-16190 for 4.0-RC because we discovered
minor bugs introduced in 4.0.
> 4.0 quality testing: metrics
>
>
> Key: CASSANDRA-15582
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15582
> Project: Cassandra
> Issue Type: Task
> Components: Test/dtest/python
>Reporter: Josh McKenzie
>Assignee: Benjamin Lerer
>Priority: Normal
> Fix For: 4.0-rc
>
> Attachments: Screen Shot 2020-04-07 at 5.47.17 PM.png
>
>
> The goal of this ticket is to have a proper testing of the different metrics
> exposed via JMX, and to ensure that metrics that are not in used in 4.0 have
> been properly deprecated.
> The following table show the current status of the metric tests and can be
> used to track the progress of that ticket:
> || Metrics || Status || test types || JIRA tickets ||
> | Batch | {color:#00875A}*COVERED*{color} | unit tests | CASSANDRA-15718 |
> | BufferPool | {color:#00875A}*COVERED*{color} | unit tests | CASSANDRA-15773
> |
> | Cache | {color:#00875A}*COVERED*{color} | unit tests | CASSANDRA-15788 |
> | Client | {color:#DE350B}*TESTS MISSING*{color}| unit tests |
> CASSANDRA-16216 |
> | ClientRequest | {color:#00875A}*COVERED*{color} | in-jvm tests |
> CASSANDRA-16183 |
> | ClientRequestSize | {color:#00875A}*COVERED*{color} | unit tests |
> CASSANDRA-16184 |
> | Cache | {color:#00875A}*COVERED*{color} | unit tests | CASSANDRA-15788 |
> | CommitLog | {color:#DE350B}*TESTS MISSING*{color} | unit tests |
> CASSANDRA-16185 |
> | Compaction | {color:#DE350B}*TESTS MISSING*{color} | unit tests |
> CASSANDRA-16192 |
> | CQL | {color:#00875A}*COVERED*{color}| unit tests | |
> | HintService | {color:#DE350B}*NO TESTS*{color} | dtests | CASSANDRA-16189 |
> | Messaging/Internode| {color:#DE350B}*NO TESTS*{color} | in-jvm dtests |
> CASSANDRA-16193 |
> | ReadRepair| {color:#DE350B}*TESTS MISSING*{color} | dtests,in-jvm dtests |
> CASSANDRA-16187 |
> | Repair | {color:#00875A}*COVERED*{color} | in-jvm dtests | CASSANDRA-16191 |
> | Storage | {color:#00875A}*COVERED*{color}| unit tests | |
> | Streaming | {color:#DE350B}*NO TESTS*{color} | dtests | CASSANDRA-16190 |
> | Keyspace | {color:#DE350B}*TESTS MISSING*{color} | unit tests/in-jvm dtests
> | CASSANDRA-16188 |
> | Table | {color:#DE350B}*TESTS MISSING*{color} | unit tests/in-jvm dtests |
> CASSANDRA-16188 |
> | ThreadPoolMetrics |{color:#00875A}*COVERED*{color} | unit tests |
> CASSANDRA-16186 |
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15582) 4.0 quality testing: metrics
[
https://issues.apache.org/jira/browse/CASSANDRA-15582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17304786#comment-17304786
]
Benjamin Lerer commented on CASSANDRA-15582:
[~cscotta] It makes sense to me. I will go through the different tickets to
check how close they are from completion to see if it makes sense to defer them
or not.
> 4.0 quality testing: metrics
>
>
> Key: CASSANDRA-15582
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15582
> Project: Cassandra
> Issue Type: Task
> Components: Test/dtest/python
>Reporter: Josh McKenzie
>Assignee: Benjamin Lerer
>Priority: Normal
> Fix For: 4.0-rc
>
> Attachments: Screen Shot 2020-04-07 at 5.47.17 PM.png
>
>
> The goal of this ticket is to have a proper testing of the different metrics
> exposed via JMX, and to ensure that metrics that are not in used in 4.0 have
> been properly deprecated.
> The following table show the current status of the metric tests and can be
> used to track the progress of that ticket:
> || Metrics || Status || test types || JIRA tickets ||
> | Batch | {color:#00875A}*COVERED*{color} | unit tests | CASSANDRA-15718 |
> | BufferPool | {color:#00875A}*COVERED*{color} | unit tests | CASSANDRA-15773
> |
> | Cache | {color:#00875A}*COVERED*{color} | unit tests | CASSANDRA-15788 |
> | Client | {color:#DE350B}*TESTS MISSING*{color}| unit tests |
> CASSANDRA-16216 |
> | ClientRequest | {color:#00875A}*COVERED*{color} | in-jvm tests |
> CASSANDRA-16183 |
> | ClientRequestSize | {color:#00875A}*COVERED*{color} | unit tests |
> CASSANDRA-16184 |
> | Cache | {color:#00875A}*COVERED*{color} | unit tests | CASSANDRA-15788 |
> | CommitLog | {color:#DE350B}*TESTS MISSING*{color} | unit tests |
> CASSANDRA-16185 |
> | Compaction | {color:#DE350B}*TESTS MISSING*{color} | unit tests |
> CASSANDRA-16192 |
> | CQL | {color:#00875A}*COVERED*{color}| unit tests | |
> | HintService | {color:#DE350B}*NO TESTS*{color} | dtests | CASSANDRA-16189 |
> | Messaging/Internode| {color:#DE350B}*NO TESTS*{color} | in-jvm dtests |
> CASSANDRA-16193 |
> | ReadRepair| {color:#DE350B}*TESTS MISSING*{color} | dtests,in-jvm dtests |
> CASSANDRA-16187 |
> | Repair | {color:#00875A}*COVERED*{color} | in-jvm dtests | CASSANDRA-16191 |
> | Storage | {color:#00875A}*COVERED*{color}| unit tests | |
> | Streaming | {color:#DE350B}*NO TESTS*{color} | dtests | CASSANDRA-16190 |
> | Keyspace | {color:#DE350B}*TESTS MISSING*{color} | unit tests/in-jvm dtests
> | CASSANDRA-16188 |
> | Table | {color:#DE350B}*TESTS MISSING*{color} | unit tests/in-jvm dtests |
> CASSANDRA-16188 |
> | ThreadPoolMetrics |{color:#00875A}*COVERED*{color} | unit tests |
> CASSANDRA-16186 |
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15582) 4.0 quality testing: metrics
[
https://issues.apache.org/jira/browse/CASSANDRA-15582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17304522#comment-17304522
]
Ekaterina Dimitrova commented on CASSANDRA-15582:
-
+1 on the suggestion
> 4.0 quality testing: metrics
>
>
> Key: CASSANDRA-15582
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15582
> Project: Cassandra
> Issue Type: Task
> Components: Test/dtest/python
>Reporter: Josh McKenzie
>Assignee: Benjamin Lerer
>Priority: Normal
> Fix For: 4.0-rc
>
> Attachments: Screen Shot 2020-04-07 at 5.47.17 PM.png
>
>
> The goal of this ticket is to have a proper testing of the different metrics
> exposed via JMX, and to ensure that metrics that are not in used in 4.0 have
> been properly deprecated.
> The following table show the current status of the metric tests and can be
> used to track the progress of that ticket:
> || Metrics || Status || test types || JIRA tickets ||
> | Batch | {color:#00875A}*COVERED*{color} | unit tests | CASSANDRA-15718 |
> | BufferPool | {color:#00875A}*COVERED*{color} | unit tests | CASSANDRA-15773
> |
> | Cache | {color:#00875A}*COVERED*{color} | unit tests | CASSANDRA-15788 |
> | Client | {color:#DE350B}*TESTS MISSING*{color}| unit tests |
> CASSANDRA-16216 |
> | ClientRequest | {color:#00875A}*COVERED*{color} | in-jvm tests |
> CASSANDRA-16183 |
> | ClientRequestSize | {color:#00875A}*COVERED*{color} | unit tests |
> CASSANDRA-16184 |
> | Cache | {color:#00875A}*COVERED*{color} | unit tests | CASSANDRA-15788 |
> | CommitLog | {color:#DE350B}*TESTS MISSING*{color} | unit tests |
> CASSANDRA-16185 |
> | Compaction | {color:#DE350B}*TESTS MISSING*{color} | unit tests |
> CASSANDRA-16192 |
> | CQL | {color:#00875A}*COVERED*{color}| unit tests | |
> | HintService | {color:#DE350B}*NO TESTS*{color} | dtests | CASSANDRA-16189 |
> | Messaging/Internode| {color:#DE350B}*NO TESTS*{color} | in-jvm dtests |
> CASSANDRA-16193 |
> | ReadRepair| {color:#DE350B}*TESTS MISSING*{color} | dtests,in-jvm dtests |
> CASSANDRA-16187 |
> | Repair | {color:#00875A}*COVERED*{color} | in-jvm dtests | CASSANDRA-16191 |
> | Storage | {color:#00875A}*COVERED*{color}| unit tests | |
> | Streaming | {color:#DE350B}*NO TESTS*{color} | dtests | CASSANDRA-16190 |
> | Keyspace | {color:#DE350B}*TESTS MISSING*{color} | unit tests/in-jvm dtests
> | CASSANDRA-16188 |
> | Table | {color:#DE350B}*TESTS MISSING*{color} | unit tests/in-jvm dtests |
> CASSANDRA-16188 |
> | ThreadPoolMetrics |{color:#00875A}*COVERED*{color} | unit tests |
> CASSANDRA-16186 |
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15582) 4.0 quality testing: metrics
[
https://issues.apache.org/jira/browse/CASSANDRA-15582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17304497#comment-17304497
]
C. Scott Andreas commented on CASSANDRA-15582:
--
How do we feel about the possibility of deferring adding further tests for
metrics to 4.0.x, given that we've dramatically improved coverage relative to
3.x?
> 4.0 quality testing: metrics
>
>
> Key: CASSANDRA-15582
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15582
> Project: Cassandra
> Issue Type: Task
> Components: Test/dtest/python
>Reporter: Josh McKenzie
>Assignee: Benjamin Lerer
>Priority: Normal
> Fix For: 4.0-rc
>
> Attachments: Screen Shot 2020-04-07 at 5.47.17 PM.png
>
>
> The goal of this ticket is to have a proper testing of the different metrics
> exposed via JMX, and to ensure that metrics that are not in used in 4.0 have
> been properly deprecated.
> The following table show the current status of the metric tests and can be
> used to track the progress of that ticket:
> || Metrics || Status || test types || JIRA tickets ||
> | Batch | {color:#00875A}*COVERED*{color} | unit tests | CASSANDRA-15718 |
> | BufferPool | {color:#00875A}*COVERED*{color} | unit tests | CASSANDRA-15773
> |
> | Cache | {color:#00875A}*COVERED*{color} | unit tests | CASSANDRA-15788 |
> | Client | {color:#DE350B}*TESTS MISSING*{color}| unit tests |
> CASSANDRA-16216 |
> | ClientRequest | {color:#00875A}*COVERED*{color} | in-jvm tests |
> CASSANDRA-16183 |
> | ClientRequestSize | {color:#00875A}*COVERED*{color} | unit tests |
> CASSANDRA-16184 |
> | Cache | {color:#00875A}*COVERED*{color} | unit tests | CASSANDRA-15788 |
> | CommitLog | {color:#DE350B}*TESTS MISSING*{color} | unit tests |
> CASSANDRA-16185 |
> | Compaction | {color:#DE350B}*TESTS MISSING*{color} | unit tests |
> CASSANDRA-16192 |
> | CQL | {color:#00875A}*COVERED*{color}| unit tests | |
> | HintService | {color:#DE350B}*NO TESTS*{color} | dtests | CASSANDRA-16189 |
> | Messaging/Internode| {color:#DE350B}*NO TESTS*{color} | in-jvm dtests |
> CASSANDRA-16193 |
> | ReadRepair| {color:#DE350B}*TESTS MISSING*{color} | dtests,in-jvm dtests |
> CASSANDRA-16187 |
> | Repair | {color:#00875A}*COVERED*{color} | in-jvm dtests | CASSANDRA-16191 |
> | Storage | {color:#00875A}*COVERED*{color}| unit tests | |
> | Streaming | {color:#DE350B}*NO TESTS*{color} | dtests | CASSANDRA-16190 |
> | Keyspace | {color:#DE350B}*TESTS MISSING*{color} | unit tests/in-jvm dtests
> | CASSANDRA-16188 |
> | Table | {color:#DE350B}*TESTS MISSING*{color} | unit tests/in-jvm dtests |
> CASSANDRA-16188 |
> | ThreadPoolMetrics |{color:#00875A}*COVERED*{color} | unit tests |
> CASSANDRA-16186 |
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15582) 4.0 quality testing: metrics
[
https://issues.apache.org/jira/browse/CASSANDRA-15582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17236458#comment-17236458
]
David Capwell commented on CASSANDRA-15582:
---
Linking CASSANDRA-16095 as it impacts metrics when schema changes happens.
> 4.0 quality testing: metrics
>
>
> Key: CASSANDRA-15582
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15582
> Project: Cassandra
> Issue Type: Task
> Components: Test/dtest/python
>Reporter: Josh McKenzie
>Assignee: Benjamin Lerer
>Priority: Normal
> Fix For: 4.0-beta
>
> Attachments: Screen Shot 2020-04-07 at 5.47.17 PM.png
>
>
> The goal of this ticket is to have a proper testing of the different metrics
> exposed via JMX, and to ensure that metrics that are not in used in 4.0 have
> been properly deprecated.
> The following table show the current status of the metric tests and can be
> used to track the progress of that ticket:
> || Metrics || Status || test types || JIRA tickets ||
> | Batch | {color:#00875A}*COVERED*{color} | unit tests | CASSANDRA-15718 |
> | BufferPool | {color:#00875A}*COVERED*{color} | unit tests | CASSANDRA-15773
> |
> | Cache | {color:#00875A}*COVERED*{color} | unit tests | CASSANDRA-15788 |
> | Client | {color:#DE350B}*TESTS MISSING*{color}| unit tests |
> CASSANDRA-16216 |
> | ClientRequest | {color:#00875A}*COVERED*{color} | in-jvm tests |
> CASSANDRA-16183 |
> | ClientRequestSize | {color:#00875A}*COVERED*{color} | unit tests |
> CASSANDRA-16184 |
> | Cache | {color:#00875A}*COVERED*{color} | unit tests | CASSANDRA-15788 |
> | CommitLog | {color:#DE350B}*TESTS MISSING*{color} | unit tests |
> CASSANDRA-16185 |
> | Compaction | {color:#DE350B}*TESTS MISSING*{color} | unit tests |
> CASSANDRA-16192 |
> | CQL | {color:#00875A}*COVERED*{color}| unit tests | |
> | HintService | {color:#DE350B}*NO TESTS*{color} | dtests | CASSANDRA-16189 |
> | Messaging/Internode| {color:#DE350B}*NO TESTS*{color} | in-jvm dtests |
> CASSANDRA-16193 |
> | ReadRepair| {color:#DE350B}*TESTS MISSING*{color} | dtests,in-jvm dtests |
> CASSANDRA-16187 |
> | Repair | {color:#DE350B}*NO TESTS*{color} | in-jvm dtests | CASSANDRA-16191
> |
> | Storage | {color:#00875A}*COVERED*{color}| unit tests | |
> | Streaming | {color:#DE350B}*NO TESTS*{color} | dtests | CASSANDRA-16190 |
> | Keyspace | {color:#DE350B}*TESTS MISSING*{color} | unit tests/in-jvm dtests
> | CASSANDRA-16188 |
> | Table | {color:#DE350B}*TESTS MISSING*{color} | unit tests/in-jvm dtests |
> CASSANDRA-16188 |
> | ThreadPoolMetrics |{color:#00875A}*COVERED*{color} | unit tests |
> CASSANDRA-16186 |
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15582) 4.0 quality testing: metrics
[ https://issues.apache.org/jira/browse/CASSANDRA-15582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17206488#comment-17206488 ] Josh McKenzie commented on CASSANDRA-15582: --- Sounds good. Also sounds like it's be pretty solid LHF for new contributors to get involved in if we enumerate it clearly. > 4.0 quality testing: metrics > > > Key: CASSANDRA-15582 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15582 > Project: Cassandra > Issue Type: Task > Components: Test/dtest/python >Reporter: Josh McKenzie >Assignee: Benjamin Lerer >Priority: Normal > Fix For: 4.0-beta, 4.0-triage > > Attachments: Screen Shot 2020-04-07 at 5.47.17 PM.png > > > In past releases we've unknowingly broken metrics integrations and introduced > performance regressions in metrics collection and reporting. We strive in 4.0 > to not do that. Metrics should work well! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15582) 4.0 quality testing: metrics
[ https://issues.apache.org/jira/browse/CASSANDRA-15582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17206223#comment-17206223 ] Benjamin Lerer commented on CASSANDRA-15582: I want to go throught the list of metrics that we have and which tests/coverage we have for each of them next week. Once we know what is not cover we should have a clear idea of the amount of work to be done. > 4.0 quality testing: metrics > > > Key: CASSANDRA-15582 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15582 > Project: Cassandra > Issue Type: Task > Components: Test/dtest/python >Reporter: Josh McKenzie >Assignee: Benjamin Lerer >Priority: Normal > Fix For: 4.0-beta, 4.0-triage > > Attachments: Screen Shot 2020-04-07 at 5.47.17 PM.png > > > In past releases we've unknowingly broken metrics integrations and introduced > performance regressions in metrics collection and reporting. We strive in 4.0 > to not do that. Metrics should work well! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15582) 4.0 quality testing: metrics
[ https://issues.apache.org/jira/browse/CASSANDRA-15582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17206171#comment-17206171 ] Josh McKenzie commented on CASSANDRA-15582: --- [~blerer] - do we have a point of view on what work is remaining before we have confidence in our metrics? > 4.0 quality testing: metrics > > > Key: CASSANDRA-15582 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15582 > Project: Cassandra > Issue Type: Task > Components: Test/dtest/python >Reporter: Josh McKenzie >Assignee: Benjamin Lerer >Priority: Normal > Fix For: 4.0-beta, 4.0-triage > > Attachments: Screen Shot 2020-04-07 at 5.47.17 PM.png > > > In past releases we've unknowingly broken metrics integrations and introduced > performance regressions in metrics collection and reporting. We strive in 4.0 > to not do that. Metrics should work well! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15582) 4.0 quality testing: metrics
[ https://issues.apache.org/jira/browse/CASSANDRA-15582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111405#comment-17111405 ] Stephen Mallette commented on CASSANDRA-15582: -- Just added CASSANDRA-15821 that presents some analysis on how published metrics match up to documentation. > 4.0 quality testing: metrics > > > Key: CASSANDRA-15582 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15582 > Project: Cassandra > Issue Type: Task > Components: Test/dtest >Reporter: Josh McKenzie >Assignee: Romain Hardouin >Priority: Normal > Fix For: 4.0-beta > > Attachments: Screen Shot 2020-04-07 at 5.47.17 PM.png > > > In past releases we've unknowingly broken metrics integrations and introduced > performance regressions in metrics collection and reporting. We strive in 4.0 > to not do that. Metrics should work well! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15582) 4.0 quality testing: metrics
[ https://issues.apache.org/jira/browse/CASSANDRA-15582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17106324#comment-17106324 ] Stephen Mallette commented on CASSANDRA-15582: -- > How do we ensure that all code paths that generate the metrics are exercised? >> I think this was a good point as well. I've yet to come across a metric that >> wasn't initialized as start of cassandra After a bit of digging it seems that there are a number of metrics that don't simply initialize as a result of cassandra starting up successfully. * Streaming Metrics * HintedHandoff Metrics * HintsService Metrics * Batch Metrics It's also worth noting that certain metric scopes don't seem to initialize on startup, specifically: ThreadPools. As a result there a number of missing metrics to evaluate. It seems that to cover everything we'd need to fire some sort of "metric initialization" script prior to gathering metrics for analysis/testing. I guess I will start researching how best to do that. Of course, my knowledge of what's missing comes from the documentation being compared to the results of my jmx scripts so if there is a metric not in the docs AND also not initialized at cassandra start, I don't know how to uncover that except by way of digging through code. > 4.0 quality testing: metrics > > > Key: CASSANDRA-15582 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15582 > Project: Cassandra > Issue Type: Task > Components: Test/dtest >Reporter: Josh McKenzie >Assignee: Romain Hardouin >Priority: Normal > Fix For: 4.0-beta > > Attachments: Screen Shot 2020-04-07 at 5.47.17 PM.png > > > In past releases we've unknowingly broken metrics integrations and introduced > performance regressions in metrics collection and reporting. We strive in 4.0 > to not do that. Metrics should work well! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15582) 4.0 quality testing: metrics
[
https://issues.apache.org/jira/browse/CASSANDRA-15582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099928#comment-17099928
]
Stephen Mallette commented on CASSANDRA-15582:
--
I just added CASSANDRA-15788 to add tests for {{CacheMetrics}} and
{{ChunkCacheMetrics}}
> 4.0 quality testing: metrics
>
>
> Key: CASSANDRA-15582
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15582
> Project: Cassandra
> Issue Type: Task
> Components: Test/dtest
>Reporter: Josh McKenzie
>Assignee: Romain Hardouin
>Priority: Normal
> Fix For: 4.0-rc
>
> Attachments: Screen Shot 2020-04-07 at 5.47.17 PM.png
>
>
> In past releases we've unknowingly broken metrics integrations and introduced
> performance regressions in metrics collection and reporting. We strive in 4.0
> to not do that. Metrics should work well!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15582) 4.0 quality testing: metrics
[
https://issues.apache.org/jira/browse/CASSANDRA-15582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17095565#comment-17095565
]
Stephen Mallette commented on CASSANDRA-15582:
--
I just added CASSANDRA-15773 to add tests for {{BufferPoolMetrics}} - I didn't
see any tests related to those metrics.
> 4.0 quality testing: metrics
>
>
> Key: CASSANDRA-15582
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15582
> Project: Cassandra
> Issue Type: Task
> Components: Test/dtest
>Reporter: Josh McKenzie
>Assignee: Romain Hardouin
>Priority: Normal
> Fix For: 4.0-rc
>
> Attachments: Screen Shot 2020-04-07 at 5.47.17 PM.png
>
>
> In past releases we've unknowingly broken metrics integrations and introduced
> performance regressions in metrics collection and reporting. We strive in 4.0
> to not do that. Metrics should work well!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15582) 4.0 quality testing: metrics
[ https://issues.apache.org/jira/browse/CASSANDRA-15582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17088060#comment-17088060 ] Stephen Mallette commented on CASSANDRA-15582: -- Getting back to the discussion on comparing metrics on 3 and 4, I think it makes sense to work on the manual process of this effort first to see if it can at least have a reasonable documented approach. I spent a fair bit of time trying multiple ways to get two separate cassandra clusters running locally (one for 3 and one for 4). I'd found it easy enough to do by installing cassandra locally and editing some of the config a bit. I had less success with nicer approaches like ccm and docker. The latter was pretty annoying as that seems like such a simple way to get things working but I clearly was confounded by something in the step of remotely connecting to cassandra/JMX in a docker container. For now, I will continue my analysis with this simple rig I have currently working. > Possible to print out what was added and what was removed? I was able to get a list of newly added metric names - so metric names added in 4 that were not in 3: https://gist.github.com/spmallette/c443716e1c0de40b4a5bb0ef5422aeee There are a fair number of them and many do not appear to exist in the documentation, so perhaps this is another area that needs some attention in relation to this ticket. > How do we ensure that all code paths that generate the metrics are exercised? I think this was a good point as well. I've yet to come across a metric that wasn't initialized as start of cassandra, but I've only determined that by random diggings in code and so I can't say with confidence that this is always the case. If anyone is familiar with any metrics that are fired up as a result of specific code paths having to be exercised I'd be interested to know about them as it would mean some new surface area to consider in all this. I think I will venture to try to better understand the nature of the property keys shifting between 3 and 4 to see if I can firm up my understanding of what is happening there. And, hopefully, I can figure out a nicer way for anyone to easily setup an environment to run some of this analysis if they wanted to. > 4.0 quality testing: metrics > > > Key: CASSANDRA-15582 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15582 > Project: Cassandra > Issue Type: Task > Components: Test/dtest >Reporter: Josh McKenzie >Assignee: Romain Hardouin >Priority: Normal > Fix For: 4.0-rc > > Attachments: Screen Shot 2020-04-07 at 5.47.17 PM.png > > > In past releases we've unknowingly broken metrics integrations and introduced > performance regressions in metrics collection and reporting. We strive in 4.0 > to not do that. Metrics should work well! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15582) 4.0 quality testing: metrics
[
https://issues.apache.org/jira/browse/CASSANDRA-15582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17082367#comment-17082367
]
Stephen Mallette commented on CASSANDRA-15582:
--
I've added CASSANDRA-15718 to continue the review of the {{BatchMetricsTest}}
> 4.0 quality testing: metrics
>
>
> Key: CASSANDRA-15582
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15582
> Project: Cassandra
> Issue Type: Task
> Components: Test/dtest
>Reporter: Josh McKenzie
>Assignee: Romain Hardouin
>Priority: Normal
> Fix For: 4.0-rc
>
> Attachments: Screen Shot 2020-04-07 at 5.47.17 PM.png
>
>
> In past releases we've unknowingly broken metrics integrations and introduced
> performance regressions in metrics collection and reporting. We strive in 4.0
> to not do that. Metrics should work well!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15582) 4.0 quality testing: metrics
[
https://issues.apache.org/jira/browse/CASSANDRA-15582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077730#comment-17077730
]
David Capwell commented on CASSANDRA-15582:
---
For
https://github.com/apache/cassandra/compare/trunk...spmallette:CASSANDRA-15582-trunk-batchmetrics
* I like the usage of BatchStatement.Type for the tests
* honestly feel quick theories is better than random, but glad you added the
seed to all asserts =). Would still be better as a quick theories test since
you basically wrote a property anyways!
*
https://github.com/apache/cassandra/compare/trunk...spmallette:CASSANDRA-15582-trunk-batchmetrics#diff-8948cec1f9d33f10b15c38de80141548R131
feel you should rename to
expectedPartitionsPerLoggedBatch{Count,Logged,Unlogged}. pre is what the value
is, post is what the value is expected to be (rather than what it is).
*
https://github.com/apache/cassandra/compare/trunk...spmallette:CASSANDRA-15582-trunk-batchmetrics#diff-8948cec1f9d33f10b15c38de80141548R150
this doesn't look correct. the batch has distinctPartitions mutations, so
shouldn't max reflect that? I ran the current test in a debugger and see that
that is the case (aka current test is wrong).
most of the comments are nit picks, but the last one looks like a test bug to me
> 4.0 quality testing: metrics
>
>
> Key: CASSANDRA-15582
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15582
> Project: Cassandra
> Issue Type: Task
> Components: Test/dtest
>Reporter: Josh McKenzie
>Assignee: Romain Hardouin
>Priority: Normal
> Fix For: 4.0-rc
>
> Attachments: Screen Shot 2020-04-07 at 5.47.17 PM.png
>
>
> In past releases we've unknowingly broken metrics integrations and introduced
> performance regressions in metrics collection and reporting. We strive in 4.0
> to not do that. Metrics should work well!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15582) 4.0 quality testing: metrics
[ https://issues.apache.org/jira/browse/CASSANDRA-15582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077717#comment-17077717 ] David Capwell commented on CASSANDRA-15582: --- [~spmallette] possible to make a sub task for review of the patch? Ill start reviewing now, but would be easier as its own jira IMO. > 4.0 quality testing: metrics > > > Key: CASSANDRA-15582 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15582 > Project: Cassandra > Issue Type: Task > Components: Test/dtest >Reporter: Josh McKenzie >Assignee: Romain Hardouin >Priority: Normal > Fix For: 4.0-rc > > Attachments: Screen Shot 2020-04-07 at 5.47.17 PM.png > > > In past releases we've unknowingly broken metrics integrations and introduced > performance regressions in metrics collection and reporting. We strive in 4.0 > to not do that. Metrics should work well! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15582) 4.0 quality testing: metrics
[ https://issues.apache.org/jira/browse/CASSANDRA-15582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077714#comment-17077714 ] David Capwell commented on CASSANDRA-15582: --- bq. Analysis of that fashion through my experimental script is more of a one-time test however and I wasn't sure if there was some intent based on these comments to more fully automate this sort of testing going forward to help avoid breaking metric changes as part of the standard and ongoing testing process. I'd be curious what the direction is in this area and I'm happy to try to help. In my opinion automation is great, but I am also fine with incremental improvements. If we had a documented way to collect all metrics, we can eventually improve this to be an automated thing which generates a report of what metrics exist, and with that we could diff with previous results. If the very first deliverable is not automated (circle ci or Jenkins running) I am totally ok with this. bq. Assuming I haven't missed something important, this initial body of analysis seems to indicate that there isn't widespread breaking change in JMX for 4.x. I'm still staring at all this to determine what else might be tested or if I've missed something important in this initial work. Possible to print out what was added and what was removed? bq. I'm happy to hear if any of this is interesting or helpful (or if I'm on the completely wrong track to what is desired here). If not, I'll assume I should just continue on this track of analysis and continue to post findings. This does look promising. A good way to test this would be to cause a regression and see if you catch it! But overall the idea makes sense. I am curious the results for all prefixes, I spun up a cluster and see the following !Screen Shot 2020-04-07 at 5.47.17 PM.png! > 4.0 quality testing: metrics > > > Key: CASSANDRA-15582 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15582 > Project: Cassandra > Issue Type: Task > Components: Test/dtest >Reporter: Josh McKenzie >Assignee: Romain Hardouin >Priority: Normal > Fix For: 4.0-rc > > Attachments: Screen Shot 2020-04-07 at 5.47.17 PM.png > > > In past releases we've unknowingly broken metrics integrations and introduced > performance regressions in metrics collection and reporting. We strive in 4.0 > to not do that. Metrics should work well! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15582) 4.0 quality testing: metrics
[ https://issues.apache.org/jira/browse/CASSANDRA-15582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077588#comment-17077588 ] Dinesh Joshi commented on CASSANDRA-15582: -- [~spmallette] thanks for your contribution. I think improving unit test coverage is always welcome. Your groovy script approach to enumerate JMX metrics is also a good one. I think we need to ensure that all possible metrics are generated. How do we ensure that all code paths that generate the metrics are exercised? I would also perhaps add a file with the expected baseline paths that should exist. It would also be helpful to check that the values are the same (within some delta) for those metrics that are comparable. From a maintainability perspective, we should consider using Java as most of us are familiar with it. > 4.0 quality testing: metrics > > > Key: CASSANDRA-15582 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15582 > Project: Cassandra > Issue Type: Task > Components: Test/dtest >Reporter: Josh McKenzie >Assignee: Romain Hardouin >Priority: Normal > Fix For: 4.0-rc > > > In past releases we've unknowingly broken metrics integrations and introduced > performance regressions in metrics collection and reporting. We strive in 4.0 > to not do that. Metrics should work well! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15582) 4.0 quality testing: metrics
[
https://issues.apache.org/jira/browse/CASSANDRA-15582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077438#comment-17077438
]
Stephen Mallette commented on CASSANDRA-15582:
--
In an effort to continue with what I referred to as a one-time test in my
previous comment, I've continued tinkering with the groovy script I'd written.
I adjusted it to connect to two separate running instances of Cassandra so as
to compare the JMX metrics of each. Here's the core substance of the script to
see the specifics of what it's doing - I've added comments which contain some
analysis of my findings so far:
{code:java}
import javax.management.ObjectName
import javax.management.remote.JMXConnectorFactory as JmxFactory
import javax.management.remote.JMXServiceURL as JmxUrl
server4Url = 'service:jmx:rmi:///jndi/rmi://localhost:7199/jmxrmi'
server4 = JmxFactory.connect(new JmxUrl(server4Url)).MBeanServerConnection
query = new ObjectName('org.apache.cassandra.metrics:*')
allNames4 = server4.queryNames(query, null)
server3Url = 'service:jmx:rmi:///jndi/rmi://localhost:7200/jmxrmi'
server3 = JmxFactory.connect(new JmxUrl(server3Url)).MBeanServerConnection
query3 = new ObjectName('org.apache.cassandra.metrics:*')
allNames3 = server3.queryNames(query3, null)
// While there are definitely metrics that are registered in 3.11.6 that are
not in 4.0
// the difference seems bound not to the "name" of the metrics but to other
property keys
// associated outside of that one. For example, in 3.x there is a "scope" key
of:
//
// scope=views_builds_in_progress
//
// which in 4.0 is:
//
// scope=view_builds_in_progress
//
// and seems related to CASSANDRA-12245. So, if the concern here is mostly with
just
// metric names and not other property keys shifting then we can at least
confirm that
// all the metric names in 3.x are at least still available in 4.x. If there
are concerns
// about documenting changes like the "scope" example from above then some more
work would
// need to be done to identify those.
metricNamesIn3NotIn4 = allNames3.collect{it.getKeyProperty('name')}.unique().
findAll{it3 -> !allNames4.collect{it4 ->
it4.getKeyProperty('name')}.any{it4 -> it4 == it3}}
assert metricNamesIn3NotIn4.size() == 0
// For those metrics that exist in 3.x and are registered in 4.x, I wanted to
see if any
// of those metrics changed in an incompatible way which meant that pure
equality checks
// on the MBeans themselves wouldn't work so well because 4.x could have added
something
// new which would not create a breaking condition. We would only have such
trouble if
// all the attributes, operations, notifications, and constructors defined in
3.x were
// somehow not represented in 4.x in the same way and thus required individual
comparison
// on equality.
//
// While there were definitely new attributes, operations, notifications, and
constructors
// added in 4.x it does not appear as though any breaking changes occurred in
that way.
metricIn3And4DiffAtr = allNames3.findAll{ server4.isRegistered(it) }.findAll{
m3 = new GroovyMBean(server3, it)
m4 = new GroovyMBean(server4, it)
attrs3 = m3.info().attributes
attrs4 = m4.info().attributes
ops3 = m3.info().operations
ops4 = m4.info().operations
nots3 = m3.info().notifications
nots4 = m4.info().notifications
cons3 = m3.info().constructors
cons4 = m4.info().constructors
!(attrs3.size() <= attrs4.size() && attrs3.every{it3 -> attrs4.any{it4 ->
it4.equals(it3) }} &&
ops3.size() <= ops4.size() && ops3.every{it3 -> ops4.any{it4 ->
it4.equals(it3) }} &&
nots3.size() <= nots4.size() && nots3.every{it3 -> nots4.any{it4 ->
it4.equals(it3) }} &&
cons3.size() <= cons4.size() && cons3.every{it3 -> cons4.any{it4 ->
it4.equals(it3) }})
}
assert metricIn3And4DiffAtr.size() == 0
{code}
Assuming I haven't missed something important, this initial body of analysis
seems to indicate that there isn't widespread breaking change in JMX for 4.x.
I'm still staring at all this to determine what else might be tested or if I've
missed something important in this initial work. I'm happy to hear if any of
this is interesting or helpful (or if I'm on the completely wrong track to what
is desired here). If not, I'll assume I should just continue on this track of
analysis and continue to post findings.
Also, I'm not sure if anyone has had a moment to look but I still do have an
improvement to unit tests for metrics that could use some review if someone has
the chance to take a look:
[https://github.com/apache/cassandra/compare/trunk...spmallette:CASSANDRA-15582-trunk-batchmetrics]
> 4.0 quality testing: metrics
>
>
> Key: CASSANDRA-15582
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15582
> Project: Cassandra
> Issue Type: Task
>
[jira] [Commented] (CASSANDRA-15582) 4.0 quality testing: metrics
[ https://issues.apache.org/jira/browse/CASSANDRA-15582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074619#comment-17074619 ] Stephen Mallette commented on CASSANDRA-15582: -- Going up further in the comments here there was mention of a desire to do: > 1) diff metrics names (did we remove something? Change name is a remove) > 2) diff metric signature (Used to return long but now string, used to take no > arguments but now takes string, etc) I've experimented a bit with a Groovy script (because it has nice syntax to interact with jmx) to connect to cassandra's jmx port and pull all the MBeans. I could further this script to do some automated analysis to determine items 1 and 2 above as mentioned in an earlier comment: > we should have some automated way of pulling all the metrics out so we can > detect changes in the list of published metrics in an automated way. Since we > publish all metrics via JMX, I think it is doable. Analysis of that fashion through my experimental script is more of a one-time test however and I wasn't sure if there was some intent based on these comments to more fully automate this sort of testing going forward to help avoid breaking metric changes as part of the standard and ongoing testing process. I'd be curious what the direction is in this area and I'm happy to try to help. > 4.0 quality testing: metrics > > > Key: CASSANDRA-15582 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15582 > Project: Cassandra > Issue Type: Task > Components: Test/dtest >Reporter: Josh McKenzie >Assignee: Romain Hardouin >Priority: Normal > Fix For: 4.0-rc > > > In past releases we've unknowingly broken metrics integrations and introduced > performance regressions in metrics collection and reporting. We strive in 4.0 > to not do that. Metrics should work well! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15582) 4.0 quality testing: metrics
[
https://issues.apache.org/jira/browse/CASSANDRA-15582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072999#comment-17072999
]
Stephen Mallette commented on CASSANDRA-15582:
--
Here is a first contribution on this issue:
https://github.com/apache/cassandra/compare/trunk...spmallette:CASSANDRA-15582-trunk-batchmetrics
It improves the {{BatchMetricsTest}} as I alluded to above. All three metrics
are now covered instead of just two and I've modified the test to validate
random ranges of batch sizes, partitions, and statement counts. On failure it
will print the seed for the random as part of the failure method to ensure that
the failing condition can be recreated. Happy to hear any feedback reviewers
might have.
> 4.0 quality testing: metrics
>
>
> Key: CASSANDRA-15582
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15582
> Project: Cassandra
> Issue Type: Task
> Components: Test/dtest
>Reporter: Josh McKenzie
>Assignee: Romain Hardouin
>Priority: Normal
> Fix For: 4.0-rc
>
>
> In past releases we've unknowingly broken metrics integrations and introduced
> performance regressions in metrics collection and reporting. We strive in 4.0
> to not do that. Metrics should work well!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15582) 4.0 quality testing: metrics
[ https://issues.apache.org/jira/browse/CASSANDRA-15582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072226#comment-17072226 ] David Capwell commented on CASSANDRA-15582: --- agree. I recently sent a patch for CASSANDRA-15674, this is a regression in 3.0. Metrics are under tested but super important for operating a cluster, so we need to improve our testing of metrics. bq. ideally ones that tested edge cases Yeah thats where they are easier to regress. CASSANDRA-15674 only happens during compaction interruption (or disk failure)... > 4.0 quality testing: metrics > > > Key: CASSANDRA-15582 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15582 > Project: Cassandra > Issue Type: Task > Components: Test/dtest >Reporter: Josh McKenzie >Assignee: Romain Hardouin >Priority: Normal > Fix For: 4.0-rc > > > In past releases we've unknowingly broken metrics integrations and introduced > performance regressions in metrics collection and reporting. We strive in 4.0 > to not do that. Metrics should work well! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15582) 4.0 quality testing: metrics
[ https://issues.apache.org/jira/browse/CASSANDRA-15582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071283#comment-17071283 ] Jordan West commented on CASSANDRA-15582: - [~spmallette] adding robust unit test coverage would be included – ideally ones that tested edge cases or perhaps in-jvm dtests as well. > 4.0 quality testing: metrics > > > Key: CASSANDRA-15582 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15582 > Project: Cassandra > Issue Type: Task > Components: Test/dtest >Reporter: Josh McKenzie >Assignee: Romain Hardouin >Priority: Normal > Fix For: 4.0-rc > > > In past releases we've unknowingly broken metrics integrations and introduced > performance regressions in metrics collection and reporting. We strive in 4.0 > to not do that. Metrics should work well! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15582) 4.0 quality testing: metrics
[ https://issues.apache.org/jira/browse/CASSANDRA-15582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070901#comment-17070901 ] Stephen Mallette commented on CASSANDRA-15582: -- I was just checking in on this one and wondering if anyone had any thoughts on my previous comment. Thanks > 4.0 quality testing: metrics > > > Key: CASSANDRA-15582 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15582 > Project: Cassandra > Issue Type: Task > Components: Test/dtest >Reporter: Josh McKenzie >Assignee: Romain Hardouin >Priority: Normal > Fix For: 4.0-rc > > > In past releases we've unknowingly broken metrics integrations and introduced > performance regressions in metrics collection and reporting. We strive in 4.0 > to not do that. Metrics should work well! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15582) 4.0 quality testing: metrics
[
https://issues.apache.org/jira/browse/CASSANDRA-15582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066592#comment-17066592
]
Stephen Mallette commented on CASSANDRA-15582:
--
Would the spirit of this issue carry over into generally improving unit testing
coverage over the metrics? For instance I noticed that {{BatchMetricsTest}}
covers only two of the three {{BatchMetrics}} - logged and unlogged but not
counter. If it would be deemed helpful, I could make an effort to try to make
improvements in areas like this. Thoughts?
> 4.0 quality testing: metrics
>
>
> Key: CASSANDRA-15582
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15582
> Project: Cassandra
> Issue Type: Task
> Components: Test/dtest
>Reporter: Josh McKenzie
>Assignee: Romain Hardouin
>Priority: Normal
> Fix For: 4.0-rc
>
>
> In past releases we've unknowingly broken metrics integrations and introduced
> performance regressions in metrics collection and reporting. We strive in 4.0
> to not do that. Metrics should work well!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15582) 4.0 quality testing: metrics
[ https://issues.apache.org/jira/browse/CASSANDRA-15582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056346#comment-17056346 ] Jordan West commented on CASSANDRA-15582: - Thanks [~djoshi] [~dcapwell]. I agree with [~dcapwell] that 1 & 2 should be requirements (and seem relatively low hanging in terms of work). Regarding #3, I think a prioritized list (coming from 1 & 2) would be a good start. Should we create JIRAs for at least the first two? > 4.0 quality testing: metrics > > > Key: CASSANDRA-15582 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15582 > Project: Cassandra > Issue Type: Task > Components: Test/dtest >Reporter: Josh McKenzie >Assignee: Romain Hardouin >Priority: Normal > Fix For: 4.0-rc > > > In past releases we've unknowingly broken metrics integrations and introduced > performance regressions in metrics collection and reporting. We strive in 4.0 > to not do that. Metrics should work well! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15582) 4.0 quality testing: metrics
[ https://issues.apache.org/jira/browse/CASSANDRA-15582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056254#comment-17056254 ] David Capwell commented on CASSANDRA-15582: --- In short diff metric names? Not ambitious enough for me ;) I’m finding that 3.x disk metrics are inaccurate (Marcus is looking at parts, when I get time I’ll fix other issues), Jordan pointed out the need to verify a performance changes impact on metric quality; for me there are a few things 1) diff metrics names (did we remove something? Change name is a remove) 2) diff metric signature (Used to return long but now string, used to take no arguments but now takes string, etc) 3) metric value accuracy #1 and #2 are a api compatibility test where as #3 is a metrics quality test. For #3 I care less about new metrics than metrics people use in prod (new metrics should be tested as well but don’t break users if wrong), so #3 may need feedback on what metrics people monitor > 4.0 quality testing: metrics > > > Key: CASSANDRA-15582 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15582 > Project: Cassandra > Issue Type: Task > Components: Test/dtest >Reporter: Josh McKenzie >Assignee: Romain Hardouin >Priority: Normal > Fix For: 4.0-rc > > > In past releases we've unknowingly broken metrics integrations and introduced > performance regressions in metrics collection and reporting. We strive in 4.0 > to not do that. Metrics should work well! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15582) 4.0 quality testing: metrics
[ https://issues.apache.org/jira/browse/CASSANDRA-15582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056213#comment-17056213 ] Dinesh Joshi commented on CASSANDRA-15582: -- Here's the outline of my plan. We should enumerate all metrics being published in 3.0 and 3.11. We should verify if those metrics are present in 4.0 and if we deviate from that list (and we will), we should document the difference. Ideally, we should have some automated way of pulling all the metrics out so we can detect changes in the list of published metrics in an automated way. Since we publish all metrics via JMX, I think it is doable. Does this sound too ambitious? > 4.0 quality testing: metrics > > > Key: CASSANDRA-15582 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15582 > Project: Cassandra > Issue Type: Task > Components: Test/dtest >Reporter: Josh McKenzie >Assignee: Romain Hardouin >Priority: Normal > Fix For: 4.0-rc > > > In past releases we've unknowingly broken metrics integrations and introduced > performance regressions in metrics collection and reporting. We strive in 4.0 > to not do that. Metrics should work well! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15582) 4.0 quality testing: metrics
[ https://issues.apache.org/jira/browse/CASSANDRA-15582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17050493#comment-17050493 ] Ekaterina Dimitrova commented on CASSANDRA-15582: - [~rha] [~djoshi] Any particular tickets or work that I can help with here? > 4.0 quality testing: metrics > > > Key: CASSANDRA-15582 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15582 > Project: Cassandra > Issue Type: Task > Components: Test/dtest >Reporter: Josh McKenzie >Assignee: Romain Hardouin >Priority: Normal > Fix For: 4.0-rc > > > In past releases we've unknowingly broken metrics integrations and introduced > performance regressions in metrics collection and reporting. We strive in 4.0 > to not do that. Metrics should work well! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
