Re: TimeoutException: Timed out waiting for a node assignment. when AdminClient attempts to create topics
) at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:97) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:190) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'kafkaEmbedded': Invocation of init method failed; nested exception is java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1702) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:414) at org.springframework.kafka.test.context.EmbeddedKafkaContextCustomizer.customizeContext(EmbeddedKafkaContextCustomizer.java:76) at org.springframework.test.context.support.AbstractContextLoader.customizeContext(AbstractContextLoader.java:187) at org.springframework.test.context.support.AbstractGenericContextLoader.loadContext(AbstractGenericContextLoader.java:127) at org.springframework.test.context.support.AbstractGenericContextLoader.loadContext(AbstractGenericContextLoader.java:60) at org.springframework.test.context.support.AbstractDelegatingSmartContextLoader.delegateLoading(AbstractDelegatingSmartContextLoader.java:109) at org.springframework.test.context.support.AbstractDelegatingSmartContextLoader.loadContext(AbstractDelegatingSmartContextLoader.java:246) at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContextInternal(DefaultCacheAwareContextLoaderDelegate.java:99) at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContext(DefaultCacheAwareContextLoaderDelegate.java:117) ... 29 common frames omitted Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45) at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32) at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89) at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:213) at org.springframework.kafka.test.rule.KafkaEmbedded.before(KafkaEmbedded.java:293) at org.springframework.kafka.test.rule.KafkaEmbedded.afterPropertiesSet(KafkaEmbedded.java:218) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1761) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1698) ... 38 common frames omitted Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. On Tue, Oct 3, 2017 at 3:35 PM, Elliot Metsger <emets...@gmail.com> wrote: > Hi everyone, > > New user to Kafka, and am psyched to get it working in our environment! > I'm on 0.11.0.0, and using spring-kafka 2.0.0.rc1. I'm struggling to get > things working in our unit/integration tests. > > I don't want to overloa
TimeoutException: Timed out waiting for a node assignment. when AdminClient attempts to create topics
Hi everyone,
New user to Kafka, and am psyched to get it working in our environment!
I'm on 0.11.0.0, and using spring-kafka 2.0.0.rc1. I'm struggling to get
things working in our unit/integration tests.
I don't want to overload everyone with the context of the issues, but this
is the stacktrace I'm seeing (from the tail end of the Maven build):
https://gist.github.com/emetsger/05ef47ce15ca54b2735a65627992a2d0
The full build output is here: https://goo.gl/pwM8Lw
It appears that the AdminClient times out trying to create the topic for
the unit test:
13:54:40.964 [-client-thread | adminclient-1] DEBUG [
KafkaAdminClient$Call] - Call(callName=createTopics,
deadlineMs=1507053280963) timed out at 1507053280964 after 1 attempt(s)
java.lang.Exception: TimeoutException: Timed out waiting for a node
assignment.
at
org.apache.kafka.clients.admin.KafkaAdminClient$Call.fail(KafkaAdminClient.java:475)
at
org.apache.kafka.clients.admin.KafkaAdminClient$TimeoutProcessor.handleTimeouts(KafkaAdminClient.java:591)
at
org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.timeoutNewCalls(KafkaAdminClient.java:663)
at
org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:908)
at java.lang.Thread.run(Thread.java:748)
The unit test uses the maven-buildhelper-plugin to reserve a port for the
Kakfa broker, in this case, port 55383. If you look at the gist, I've
added the properties used by the AdminClient and the Broker, and it seems
that they are configured properly (that is, the AC is configured to talk to
127.0.0.1:55383, which is where the broker is listening).
Another relevant note is that I'm using spring-kafka, and Spring is
managing all the beans related to Kafka: the producer, consumer, and
broker. The broker is being started using the @EmbeddedKafka annotation,
which will create an embedded instance of Zookeeper, and an EmbeddedKafka
instance:
@TestPropertySource(locations = { "classpath:/rmapcore.properties",
"classpath:/kafka-broker.properties" })
@EmbeddedKafka(topics = { "rmap-event-topic" }, brokerProperties = {
"log.dir=${kafka.broker.logs-dir}", "port=${kafka.broker.port}" })
Kafka should be automatically creating the topic (`rmap-event-topic` in
this case), and it looks like the AdminClient tries, but ultimately fails,
timing out trying to connect to the broker:
15:21:34.977 [ main] DEBUG
[KafkaAdminClient$AdminClientRu] - adminclient-1: queueing
Call(callName=createTopics, deadlineMs=1507058614976) with a timeout 12
ms from now.
...
15:21:35.047 [-client-thread | adminclient-1] TRACE [
NetworkClient] - Found least loaded node 127.0.0.1:58230 (id: -1 rack:
null)
15:21:35.047 [-client-thread | adminclient-1] DEBUG
[NetworkClient$DefaultMetadataU] - Initialize connection to node -1 for
sending metadata request
15:21:35.047 [-client-thread | adminclient-1] DEBUG [
NetworkClient] - Initiating connection to node -1 at 127.0.0.1:58230.
15:21:35.048 [-client-thread | adminclient-1] DEBUG [
Metrics] - Added sensor with name node--1.bytes-sent
15:21:35.048 [-client-thread | adminclient-1] DEBUG [
Metrics] - Added sensor with name node--1.bytes-received
15:21:35.049 [-client-thread | adminclient-1] DEBUG [
Metrics] - Added sensor with name node--1.latency
15:21:35.055 [-client-thread | adminclient-1] DEBUG [
Selector] - Connection with /127.0.0.1 disconnected
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at
org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:50)
at
org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:95)
at
org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:359)
at org.apache.kafka.common.network.Selector.poll(Selector.java:326)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:432)
at
org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:927)
at java.lang.Thread.run(Thread.java:748)
Polling continues, and ultimately times out:
15:23:34.981 [-client-thread | adminclient-1] DEBUG [
KafkaAdminClient$Call] - Call(callName=createTopics,
deadlineMs=1507058614976) timed out at 1507058614981 after 1 attempt(s)
java.lang.Exception: TimeoutException: Timed out waiting for a node
assignment.
at
org.apache.kafka.clients.admin.KafkaAdminClient$Call.fail(KafkaAdminClient.java:475)
at
org.apache.kafka.clients.admin.KafkaAdminClient$TimeoutProcessor.handleTimeouts(KafkaAdminClient.java:591)
at
org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.timeoutNewCalls(KafkaAdminClient.java:663)
at
org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:908)
at java.lang.Thread.run(Thread.java:748)
Any help or advice is appreciated!
Thanks,
Elliot
Feel free to follow the steps to reproduce
