[
https://issues.apache.org/jira/browse/KAFKA-13672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17509374#comment-17509374
]
Liam Clarke-Hutchinson edited comment on KAFKA-13672 at 3/20/22, 4:53 AM:
--------------------------------------------------------------------------
I've renamed this issue (was Flaky test
`kafka.server.DynamicBrokerReconfigurationTest.testThreadPoolResize()`) because
the test was actually exposing a race condition in `DynamicBrokerConfig` that
occurred when the broker under test was running sufficiently slowly. Many
thanks are due to [~showuon] for all of his guidance and insight, and for
identifying the root cause.
`DynamicBrokerConfig` stores a mutable buffer of `Reconfigurable`s which are
protected by a read lock and write lock. However, these locks can't prevent the
issue this test was exposing in the CI environment, which is - if a
`Reconfigurable` is added by one thread while another thread is updating
dynamic configs (which involves iterating over the buffer of `Reconfigurable`s)
then this causes a `ConcurrentModificationException` for the thread that is
iterating, thus preventing all dynamic configs from being updated.
I'll be submitting a PR shortly to guard against this race condition.
was (Author: JIRAUSER279886):
I've renamed this issue (was Flaky test
`kafka.server.DynamicBrokerReconfigurationTest.testThreadPoolResize()`) because
the test was actually exposing a race condition in `DynamicBrokerConfig` that
occurred when the broker under test was running sufficiently slowly. Many
thanks are due to [~showuon] for all of his guidance and insight in identifying
this root cause.
`DynamicBrokerConfig` stores a mutable buffer of `Reconfigurable`s which are
protected by a read lock and write lock. However, these locks can't prevent the
issue this test was exposing in the CI environment, which is - if a
`Reconfigurable` is added by one thread while another thread is updating
dynamic configs (which involves iterating over the buffer of `Reconfigurable`s)
then this causes a `ConcurrentModificationException` for the thread that is
iterating, thus preventing all dynamic configs from being updated.
I'll be submitting a PR shortly to guard against this race condition.
> Race condition in DynamicBrokerConfig
> -------------------------------------
>
> Key: KAFKA-13672
> URL: https://issues.apache.org/jira/browse/KAFKA-13672
> Project: Kafka
> Issue Type: Bug
> Reporter: Bruno Cadonna
> Assignee: Liam Clarke-Hutchinson
> Priority: Blocker
> Fix For: 3.2.0
>
>
> Stacktrace:
> {code:java}
> org.opentest4j.AssertionFailedError: expected: <2> but was: <1>
> at
> app//org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55)
> at
> app//org.junit.jupiter.api.AssertionUtils.failNotEqual(AssertionUtils.java:62)
> at
> app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
> at
> app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)
> at
> app//org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1141)
> at
> app//kafka.server.DynamicBrokerReconfigurationTest.$anonfun$waitForConfigOnServer$1(DynamicBrokerReconfigurationTest.scala:1500)
> at
> app//kafka.server.DynamicBrokerReconfigurationTest.waitForConfigOnServer(DynamicBrokerReconfigurationTest.scala:1500)
> at
> app//kafka.server.DynamicBrokerReconfigurationTest.$anonfun$waitForConfig$1(DynamicBrokerReconfigurationTest.scala:1495)
> at
> app//kafka.server.DynamicBrokerReconfigurationTest.$anonfun$waitForConfig$1$adapted(DynamicBrokerReconfigurationTest.scala:1495)
> at app//scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
> at
> app//scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
> at app//scala.collection.AbstractIterable.foreach(Iterable.scala:926)
> at
> app//kafka.server.DynamicBrokerReconfigurationTest.waitForConfig(DynamicBrokerReconfigurationTest.scala:1495)
> at
> app//kafka.server.DynamicBrokerReconfigurationTest.reconfigureServers(DynamicBrokerReconfigurationTest.scala:1440)
> at
> app//kafka.server.DynamicBrokerReconfigurationTest.resizeThreadPool$1(DynamicBrokerReconfigurationTest.scala:775)
> at
> app//kafka.server.DynamicBrokerReconfigurationTest.$anonfun$testThreadPoolResize$3(DynamicBrokerReconfigurationTest.scala:768)
> at app//scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:190)
> at
> app//kafka.server.DynamicBrokerReconfigurationTest.testThreadPoolResize(DynamicBrokerReconfigurationTest.scala:784)
> {code}
> Job:
> https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-11751/5/testReport/
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
