[jira] [Comment Edited] (KAFKA-8011) Flaky Test RegexSourceIntegrationTest#testRegexMatchesTopicsAWhenCreated
[
https://issues.apache.org/jira/browse/KAFKA-8011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118222#comment-17118222
]
Sophie Blee-Goldman edited comment on KAFKA-8011 at 6/4/20, 12:58 AM:
--
[~mjsax] should we create a separate ticket for that test? AFAICT the test this
ticket is for is only failing because that test is failing and the cleanup
isn't done properly. But the test you pointed out seems to be a real bug:
{code:java}
[2020-05-27 17:56:00,413] ERROR Exception caught during Deserialization,
taskId: 0_0, topic: TEST-TOPIC-1, partition: 0, offset: 0
(org.apache.kafka.streams.errors.LogAndFailExceptionHandler:39)
java.lang.NullPointerException
at
org.apache.kafka.streams.processor.internals.SourceNode.deserializeKey(SourceNode.java:58)
at
org.apache.kafka.streams.processor.internals.RecordDeserializer.deserialize(RecordDeserializer.java:65)
at
org.apache.kafka.streams.processor.internals.RecordQueue.updateHead(RecordQueue.java:176)
at
org.apache.kafka.streams.processor.internals.RecordQueue.addRawRecords(RecordQueue.java:112)
at
org.apache.kafka.streams.processor.internals.PartitionGroup.addRawRecords(PartitionGroup.java:185)
at
org.apache.kafka.streams.processor.internals.StreamTask.addRecords(StreamTask.java:844)
at
org.apache.kafka.streams.processor.internals.StreamThread.addRecordsToTasks(StreamThread.java:836)
at
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:646)
at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:550)
at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:509)
{code}
was (Author: ableegoldman):
[~mjsax] should we create a separate ticket for that test? AFAICT the test this
ticket is for is only failing because that test is failing and the cleanup
isn't done properly. But the test you pointed out seems to be a real bug:
{code:java}
[2020-05-27 17:56:00,413] ERROR Exception caught during Deserialization,
taskId: 0_0, topic: TEST-TOPIC-1, partition: 0, offset: 0
(org.apache.kafka.streams.errors.LogAndFailExceptionHandler:39)
java.lang.NullPointerException
at
org.apache.kafka.streams.processor.internals.SourceNode.deserializeKey(SourceNode.java:58)
at
org.apache.kafka.streams.processor.internals.RecordDeserializer.deserialize(RecordDeserializer.java:65)
at
org.apache.kafka.streams.processor.internals.RecordQueue.updateHead(RecordQueue.java:176)
at
org.apache.kafka.streams.processor.internals.RecordQueue.addRawRecords(RecordQueue.java:112)
at
org.apache.kafka.streams.processor.internals.PartitionGroup.addRawRecords(PartitionGroup.java:185)
at
org.apache.kafka.streams.processor.internals.StreamTask.addRecords(StreamTask.java:844)
at
org.apache.kafka.streams.processor.internals.StreamThread.addRecordsToTasks(StreamThread.java:836)
at
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:646)
at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:550)
at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:509)
{code}
I also later see
{code:java}
[2020-05-27 17:57:23,145] ERROR stream-thread
[regex-source-integration-test-9e3e325f-3fe3-4741-9862-eff246a8fb2a-StreamThread-1]
Encountered the following exception during processing and the thread is going
to shut down: (org.apache.kafka.streams.processor.internals.StreamThread:529)
org.apache.kafka.streams.errors.TopologyException: Invalid topology: Topic foo
is already matched for another regex pattern foo.* and hence cannot be matched
to this regex pattern f.* any more. at
org.apache.kafka.streams.processor.internals.InternalTopologyBuilder$SourceNodeFactory.getTopics(InternalTopologyBuilder.java:261)
at
org.apache.kafka.streams.processor.internals.InternalTopologyBuilder.setRegexMatchedTopicsToSourceNodes(InternalTopologyBuilder.java:1122)
at
org.apache.kafka.streams.processor.internals.InternalTopologyBuilder.updateSubscribedTopics(InternalTopologyBuilder.java:1983)
at
org.apache.kafka.streams.processor.internals.InternalTopologyBuilder.addSubscribedTopicsFromMetadata(InternalTopologyBuilder.java:1973)
at
org.apache.kafka.streams.processor.internals.TaskManager.handleRebalanceStart(TaskManager.java:136)
at
org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor.handleRebalanceStart(StreamsPartitionAssignor.java:1585)
at
org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor.subscriptionUserData(StreamsPartitionAssignor.java:232)
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.metadata(ConsumerCoordinator.java:222)
at
[jira] [Comment Edited] (KAFKA-8011) Flaky Test RegexSourceIntegrationTest#testRegexMatchesTopicsAWhenCreated
[
https://issues.apache.org/jira/browse/KAFKA-8011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16781755#comment-16781755
]
Bill Bejeck edited comment on KAFKA-8011 at 3/1/19 3:10 PM:
I've been able to verify that both 1.0 and 1.1 tests pass locally. If there is
a failure in a subsequent build should I revert the cherry-pick?
EDIT: Additionally the error message in the 1.1 failure
{noformat}
Exception in thread
"regex-source-integration-test-e94954f0-2bda-4967-89ee-3de8a64522ea-StreamThread-6"
org.apache.kafka.streams.errors.TopologyException: Invalid topology: Topic foo
is already matched for another regex pattern foo.* and hence cannot be matched
to this regex pattern f.* any more.{noformat}
includes a topic name that is not even in the test.
The commit only changed the type of the List used in two tests; I made no logic
changes were at all.
was (Author: bbejeck):
I've been able to verify that both 1.0 and 1.1 tests pass locally. If there is
a failure in a subsequent build should I revert the cherry-pick?
EDIT: Additionally the error message
{noformat}
Exception in thread
"regex-source-integration-test-e94954f0-2bda-4967-89ee-3de8a64522ea-StreamThread-6"
org.apache.kafka.streams.errors.TopologyException: Invalid topology: Topic foo
is already matched for another regex pattern foo.* and hence cannot be matched
to this regex pattern f.* any more.{noformat}
includes a topic name that is not even in the test.
The commit only changed the type of the List used in two tests; I made no logic
changes were at all.
> Flaky Test RegexSourceIntegrationTest#testRegexMatchesTopicsAWhenCreated
>
>
> Key: KAFKA-8011
> URL: https://issues.apache.org/jira/browse/KAFKA-8011
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Reporter: Bill Bejeck
>Assignee: Bill Bejeck
>Priority: Critical
> Labels: flaky-test, newbie
> Attachments: streams_1_0_test_results.png, streams_1_1_tests.png
>
>
> The RegexSourceIntegrationTest#testRegexMatchesTopicsAWhenCreated
> and RegexSourceIntegrationTest#testRegexMatchesTopicsAWhenDeleted tests use
> an ArrayList to assert the topics assigned to the Streams application.
> The ConsumerRebalanceListener used in the test operates on this list as does
> the TestUtils.waitForCondition() to verify the expected topic assignments.
> Using the same list in both places can cause a ConcurrentModficationException
> if the rebalance listener modifies the assignment at the same time
> TestUtils.waitForCondition() is using the list to verify the expected topics.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-8011) Flaky Test RegexSourceIntegrationTest#testRegexMatchesTopicsAWhenCreated
[
https://issues.apache.org/jira/browse/KAFKA-8011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16781755#comment-16781755
]
Bill Bejeck edited comment on KAFKA-8011 at 3/1/19 3:09 PM:
I've been able to verify that both 1.0 and 1.1 tests pass locally. If there is
a failure in a subsequent build should I revert the cherry-pick?
EDIT: Additionally the error message
{noformat}
Exception in thread
"regex-source-integration-test-e94954f0-2bda-4967-89ee-3de8a64522ea-StreamThread-6"
org.apache.kafka.streams.errors.TopologyException: Invalid topology: Topic foo
is already matched for another regex pattern foo.* and hence cannot be matched
to this regex pattern f.* any more.{noformat}
includes a topic name that is not even in the test.
The commit only changed the type of the List used in two tests; I made no logic
changes were at all.
was (Author: bbejeck):
I've been able to verify that both 1.0 and 1.1 tests pass locally. If there is
a failure in a subsequent build should I revert the cherry-pick?
> Flaky Test RegexSourceIntegrationTest#testRegexMatchesTopicsAWhenCreated
>
>
> Key: KAFKA-8011
> URL: https://issues.apache.org/jira/browse/KAFKA-8011
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Reporter: Bill Bejeck
>Assignee: Bill Bejeck
>Priority: Critical
> Labels: flaky-test, newbie
> Attachments: streams_1_0_test_results.png, streams_1_1_tests.png
>
>
> The RegexSourceIntegrationTest#testRegexMatchesTopicsAWhenCreated
> and RegexSourceIntegrationTest#testRegexMatchesTopicsAWhenDeleted tests use
> an ArrayList to assert the topics assigned to the Streams application.
> The ConsumerRebalanceListener used in the test operates on this list as does
> the TestUtils.waitForCondition() to verify the expected topic assignments.
> Using the same list in both places can cause a ConcurrentModficationException
> if the rebalance listener modifies the assignment at the same time
> TestUtils.waitForCondition() is using the list to verify the expected topics.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-8011) Flaky Test RegexSourceIntegrationTest#testRegexMatchesTopicsAWhenCreated
[
https://issues.apache.org/jira/browse/KAFKA-8011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16779472#comment-16779472
]
Bill Bejeck edited comment on KAFKA-8011 at 2/27/19 4:17 PM:
-
Here's the link to the build
[https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/2730/]
{noformat}
java.util.ConcurrentModificationException
at
java.base/java.util.ArrayList.checkForComodification(ArrayList.java:604)
at java.base/java.util.ArrayList.equals(ArrayList.java:563)
at
org.apache.kafka.streams.integration.RegexSourceIntegrationTest.lambda$testRegexMatchesTopicsAWhenDeleted$3(RegexSourceIntegrationTest.java:213)
at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:355)
at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:325)
at
org.apache.kafka.streams.integration.RegexSourceIntegrationTest.testRegexMatchesTopicsAWhenDeleted(RegexSourceIntegrationTest.java:213)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
at
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:330)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:78)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:328)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:65)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
at org.junit.runners.ParentRunner.run(ParentRunner.java:412)
at
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
at
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
at
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
at
org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
at
org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at
org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
at
org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
at
org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
[jira] [Comment Edited] (KAFKA-8011) Flaky Test RegexSourceIntegrationTest#testRegexMatchesTopicsAWhenCreated
[ https://issues.apache.org/jira/browse/KAFKA-8011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16779448#comment-16779448 ] Matthias J. Sax edited comment on KAFKA-8011 at 2/27/19 3:51 PM: - Did this fail on `trunk` ? Please provide link to the build and also copy the stack trace into the ticket (the build is deleted eventually). was (Author: mjsax): Did this fail on `trunk` ? > Flaky Test RegexSourceIntegrationTest#testRegexMatchesTopicsAWhenCreated > > > Key: KAFKA-8011 > URL: https://issues.apache.org/jira/browse/KAFKA-8011 > Project: Kafka > Issue Type: Bug > Components: streams >Reporter: Bill Bejeck >Priority: Critical > Labels: flaky-test, newbie > > The RegexSourceIntegrationTest#testRegexMatchesTopicsAWhenCreated > and RegexSourceIntegrationTest#testRegexMatchesTopicsAWhenDeleted tests use > an ArrayList to assert the topics assigned to the Streams application. > The ConsumerRebalanceListener used in the test operates on this list as does > the TestUtils.waitForCondition() to verify the expected topic assignments. > Using the same list in both places can cause a ConcurrentModficationException > if the rebalance listener modifies the assignment at the same time > TestUtils.waitForCondition() is using the list to verify the expected topics. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
