[
https://issues.apache.org/jira/browse/CASSANDRA-19630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845056#comment-17845056
]
Bret McGuire commented on CASSANDRA-19630:
------------------------------------------
Haven't dug into this in great detail but this looks to be another
timing-related issue. The [test method in
question|https://github.com/apache/cassandra-java-driver/blob/4.x/integration-tests/src/test/java/com/datastax/oss/driver/core/metrics/MetricsITBase.java#L172-L183]
sends a "node down" event over the EventBus, waits for the specified duration
time and then immediately checks to see if the metric has been removal.
There's no accounting here for possible delays in delivery (or any other kind
of delay).
> Intermittent test failures for
> com.datastax.oss.driver.metrics.microprofile.MicroProfileMetricsIT.should_evict_down_node_metrics_when_timeout_fires
> ---------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-19630
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19630
> Project: Cassandra
> Issue Type: Bug
> Components: Client/java-driver
> Reporter: Bret McGuire
> Priority: Normal
>
> Observed on a couple different Jenkins runs. Failure on first run was
> against Java8/Cassandra 4.1 (other combinations were fine), failure on second
> run was against Java11/Cassandra 4.1 (again other combinations were fine).
>
> Representative stack trace:
> {code:java}
> Assertion condition defined as a lambda expression in
> com.datastax.oss.driver.core.metrics.MetricsITBase
> Expecting:
> {MetricID{name='s289.bytes-received',
> tags=[]}=io.smallrye.metrics.app.MeterImpl@5a070350,
> MetricID{name='s289.bytes-sent',
> tags=[]}=io.smallrye.metrics.app.MeterImpl@74ab40b6,
> MetricID{name='s289.connected-nodes',
> tags=[]}=com.datastax.oss.driver.internal.metrics.microprofile.MicroProfileSessionMetricUpdater$$Lambda$1092/0x00000008007f2440@4310e1a6,
> MetricID{name='s289.cql-client-timeouts',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@973bf98,
> MetricID{name='s289.cql-prepared-cache-size',
> tags=[]}=com.datastax.oss.driver.internal.metrics.microprofile.MicroProfileSessionMetricUpdater$$Lambda$1096/0x00000008007f7840@6b0c2065,
> MetricID{name='s289.cql-requests',
> tags=[]}=io.smallrye.metrics.app.TimerImpl@458c7342,
> MetricID{name='s289.nodes.127_0_0_130:9043.bytes-received',
> tags=[]}=io.smallrye.metrics.app.MeterImpl@155b784,
> MetricID{name='s289.nodes.127_0_0_130:9043.bytes-sent',
> tags=[]}=io.smallrye.metrics.app.MeterImpl@154e5749,
> MetricID{name='s289.nodes.127_0_0_130:9043.cql-messages',
> tags=[]}=io.smallrye.metrics.app.TimerImpl@1f4b528d,
> MetricID{name='s289.nodes.127_0_0_130:9043.errors.connection.auth',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@397b2445,
> MetricID{name='s289.nodes.127_0_0_130:9043.errors.connection.init',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@15be5d79,
> MetricID{name='s289.nodes.127_0_0_130:9043.errors.request.aborted',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@722db498,
> MetricID{name='s289.nodes.127_0_0_130:9043.errors.request.others',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@37b75f6a,
> MetricID{name='s289.nodes.127_0_0_130:9043.errors.request.read-timeouts',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@570e0cd5,
> MetricID{name='s289.nodes.127_0_0_130:9043.errors.request.unavailables',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@30bcd6d8,
> MetricID{name='s289.nodes.127_0_0_130:9043.errors.request.unsent',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@2f2a2d23,
> MetricID{name='s289.nodes.127_0_0_130:9043.errors.request.write-timeouts',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@22eb6ba3,
> MetricID{name='s289.nodes.127_0_0_130:9043.ignores.aborted',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@524b38f,
> MetricID{name='s289.nodes.127_0_0_130:9043.ignores.other',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@53f8afe5,
> MetricID{name='s289.nodes.127_0_0_130:9043.ignores.read-timeout',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@2dc5c38c,
> MetricID{name='s289.nodes.127_0_0_130:9043.ignores.total',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@548a13e4,
> MetricID{name='s289.nodes.127_0_0_130:9043.ignores.unavailable',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@74126a31,
> MetricID{name='s289.nodes.127_0_0_130:9043.ignores.write-timeout',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@628364d1,
> MetricID{name='s289.nodes.127_0_0_130:9043.pool.available-streams',
> tags=[]}=com.datastax.oss.driver.internal.metrics.microprofile.MicroProfileNodeMetricUpdater$$Lambda$1117/0x0000000800809040@61d912ae,
> MetricID{name='s289.nodes.127_0_0_130:9043.pool.in-flight',
> tags=[]}=com.datastax.oss.driver.internal.metrics.microprofile.MicroProfileNodeMetricUpdater$$Lambda$1118/0x0000000800809440@65c82842,
> MetricID{name='s289.nodes.127_0_0_130:9043.pool.open-connections',
> tags=[]}=com.datastax.oss.driver.internal.metrics.microprofile.MicroProfileNodeMetricUpdater$$Lambda$1114/0x0000000800808040@26c9528,
> MetricID{name='s289.nodes.127_0_0_130:9043.pool.orphaned-streams',
> tags=[]}=com.datastax.oss.driver.internal.metrics.microprofile.MicroProfileNodeMetricUpdater$$Lambda$1119/0x0000000800809840@7bb0dc58,
> MetricID{name='s289.nodes.127_0_0_130:9043.retries.aborted',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@52d5fde2,
> MetricID{name='s289.nodes.127_0_0_130:9043.retries.other',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@76fb45c2,
> MetricID{name='s289.nodes.127_0_0_130:9043.retries.read-timeout',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@10584be0,
> MetricID{name='s289.nodes.127_0_0_130:9043.retries.total',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@6df717af,
> MetricID{name='s289.nodes.127_0_0_130:9043.retries.unavailable',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@1030cd1e,
> MetricID{name='s289.nodes.127_0_0_130:9043.retries.write-timeout',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@3cc95f8f,
> MetricID{name='s289.nodes.127_0_0_130:9043.speculative-executions',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@6e9c04bd,
> MetricID{name='s289.nodes.127_0_0_131:9043.bytes-received',
> tags=[]}=io.smallrye.metrics.app.MeterImpl@3b033cae,
> MetricID{name='s289.nodes.127_0_0_131:9043.bytes-sent',
> tags=[]}=io.smallrye.metrics.app.MeterImpl@2d4d5eed,
> MetricID{name='s289.nodes.127_0_0_131:9043.cql-messages',
> tags=[]}=io.smallrye.metrics.app.TimerImpl@5b8e49ec,
> MetricID{name='s289.nodes.127_0_0_131:9043.errors.connection.auth',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@553907cc,
> MetricID{name='s289.nodes.127_0_0_131:9043.errors.connection.init',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@77fdb789,
> MetricID{name='s289.nodes.127_0_0_131:9043.errors.request.aborted',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@300025ad,
> MetricID{name='s289.nodes.127_0_0_131:9043.errors.request.others',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@1968db68,
> MetricID{name='s289.nodes.127_0_0_131:9043.errors.request.read-timeouts',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@472b8ee,
> MetricID{name='s289.nodes.127_0_0_131:9043.errors.request.unavailables',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@1e290616,
> MetricID{name='s289.nodes.127_0_0_131:9043.errors.request.unsent',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@2fb56533,
> MetricID{name='s289.nodes.127_0_0_131:9043.errors.request.write-timeouts',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@70595836,
> MetricID{name='s289.nodes.127_0_0_131:9043.ignores.aborted',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@617d3b1b,
> MetricID{name='s289.nodes.127_0_0_131:9043.ignores.other',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@37329894,
> MetricID{name='s289.nodes.127_0_0_131:9043.ignores.read-timeout',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@33aaffbc,
> MetricID{name='s289.nodes.127_0_0_131:9043.ignores.total',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@9088917,
> MetricID{name='s289.nodes.127_0_0_131:9043.ignores.unavailable',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@125cece,
> MetricID{name='s289.nodes.127_0_0_131:9043.ignores.write-timeout',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@22701046,
> MetricID{name='s289.nodes.127_0_0_131:9043.pool.available-streams',
> tags=[]}=com.datastax.oss.driver.internal.metrics.microprofile.MicroProfileNodeMetricUpdater$$Lambda$1117/0x0000000800809040@46435cab,
> MetricID{name='s289.nodes.127_0_0_131:9043.pool.in-flight',
> tags=[]}=com.datastax.oss.driver.internal.metrics.microprofile.MicroProfileNodeMetricUpdater$$Lambda$1118/0x0000000800809440@b4e2545,
> MetricID{name='s289.nodes.127_0_0_131:9043.pool.open-connections',
> tags=[]}=com.datastax.oss.driver.internal.metrics.microprofile.MicroProfileNodeMetricUpdater$$Lambda$1114/0x0000000800808040@24b2c95c,
> MetricID{name='s289.nodes.127_0_0_131:9043.pool.orphaned-streams',
> tags=[]}=com.datastax.oss.driver.internal.metrics.microprofile.MicroProfileNodeMetricUpdater$$Lambda$1119/0x0000000800809840@6e20fac,
> MetricID{name='s289.nodes.127_0_0_131:9043.retries.aborted',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@5a18addf,
> MetricID{name='s289.nodes.127_0_0_131:9043.retries.other',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@2006cfd4,
> MetricID{name='s289.nodes.127_0_0_131:9043.retries.read-timeout',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@12cc1aa1,
> MetricID{name='s289.nodes.127_0_0_131:9043.retries.total',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@445e83b,
> MetricID{name='s289.nodes.127_0_0_131:9043.retries.unavailable',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@1baccb39,
> MetricID{name='s289.nodes.127_0_0_131:9043.retries.write-timeout',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@dc2cff7,
> MetricID{name='s289.nodes.127_0_0_131:9043.speculative-executions',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@7fa9d5fc,
> MetricID{name='s289.nodes.127_0_0_132:9043.bytes-received',
> tags=[]}=io.smallrye.metrics.app.MeterImpl@5486fe02,
> MetricID{name='s289.nodes.127_0_0_132:9043.bytes-sent',
> tags=[]}=io.smallrye.metrics.app.MeterImpl@298e12a8,
> MetricID{name='s289.nodes.127_0_0_132:9043.cql-messages',
> tags=[]}=io.smallrye.metrics.app.TimerImpl@32a8dd19,
> MetricID{name='s289.nodes.127_0_0_132:9043.errors.connection.auth',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@31ff885,
> MetricID{name='s289.nodes.127_0_0_132:9043.errors.connection.init',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@608a600a,
> MetricID{name='s289.nodes.127_0_0_132:9043.errors.request.aborted',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@39c835e1,
> MetricID{name='s289.nodes.127_0_0_132:9043.errors.request.others',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@4d7c77d4,
> MetricID{name='s289.nodes.127_0_0_132:9043.errors.request.read-timeouts',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@315b752e,
> MetricID{name='s289.nodes.127_0_0_132:9043.errors.request.unavailables',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@262d93f,
> MetricID{name='s289.nodes.127_0_0_132:9043.errors.request.unsent',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@7afd83af,
> MetricID{name='s289.nodes.127_0_0_132:9043.errors.request.write-timeouts',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@549189f3,
> MetricID{name='s289.nodes.127_0_0_132:9043.ignores.aborted',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@3e09744a,
> MetricID{name='s289.nodes.127_0_0_132:9043.ignores.other',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@2ab6f995,
> MetricID{name='s289.nodes.127_0_0_132:9043.ignores.read-timeout',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@3c40e797,
> MetricID{name='s289.nodes.127_0_0_132:9043.ignores.total',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@64462ffd,
> MetricID{name='s289.nodes.127_0_0_132:9043.ignores.unavailable',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@1198bc1b,
> MetricID{name='s289.nodes.127_0_0_132:9043.ignores.write-timeout',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@c7f85ec,
> MetricID{name='s289.nodes.127_0_0_132:9043.pool.available-streams',
> tags=[]}=com.datastax.oss.driver.internal.metrics.microprofile.MicroProfileNodeMetricUpdater$$Lambda$1117/0x0000000800809040@3738a7ab,
> MetricID{name='s289.nodes.127_0_0_132:9043.pool.in-flight',
> tags=[]}=com.datastax.oss.driver.internal.metrics.microprofile.MicroProfileNodeMetricUpdater$$Lambda$1118/0x0000000800809440@62d45f76,
> MetricID{name='s289.nodes.127_0_0_132:9043.pool.open-connections',
> tags=[]}=com.datastax.oss.driver.internal.metrics.microprofile.MicroProfileNodeMetricUpdater$$Lambda$1114/0x0000000800808040@36f85f53,
> MetricID{name='s289.nodes.127_0_0_132:9043.pool.orphaned-streams',
> tags=[]}=com.datastax.oss.driver.internal.metrics.microprofile.MicroProfileNodeMetricUpdater$$Lambda$1119/0x0000000800809840@4658fc85,
> MetricID{name='s289.nodes.127_0_0_132:9043.retries.aborted',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@34af0e1a,
> MetricID{name='s289.nodes.127_0_0_132:9043.retries.other',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@74c0d816,
> MetricID{name='s289.nodes.127_0_0_132:9043.retries.read-timeout',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@b6031a,
> MetricID{name='s289.nodes.127_0_0_132:9043.retries.total',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@18b7b5d,
> MetricID{name='s289.nodes.127_0_0_132:9043.retries.unavailable',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@4d987aa8,
> MetricID{name='s289.nodes.127_0_0_132:9043.retries.write-timeout',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@3f6a0b65,
> MetricID{name='s289.nodes.127_0_0_132:9043.speculative-executions',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@ff47141,
> MetricID{name='s289.throttling.delay',
> tags=[]}=io.smallrye.metrics.app.TimerImpl@55ffb271,
> MetricID{name='s289.throttling.errors',
> tags=[]}=io.smallrye.metrics.app.CounterImpl@5bbccf5c,
> MetricID{name='s289.throttling.queue-size',
> tags=[]}=com.datastax.oss.driver.internal.metrics.microprofile.MicroProfileSessionMetricUpdater$$Lambda$1095/0x00000008007f3440@34e2d66d}
> not to contain key:
> MetricID{name='s289.nodes.127_0_0_130:9043.pool.open-connections', tags=[]}
> within 10 seconds. {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
