[
https://issues.apache.org/jira/browse/KAFKA-10902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17371873#comment-17371873
]
Aleksandr commented on KAFKA-10902:
-----------------------------------
We have observed this bug on kafka client 2.5.0, 2.6.0 with kafka 2.0 and kafka
2.6.0 (openjdk-13, openjdk-14). Still cannot reproduce this bug on test.
The bug is very annoying because all we can do is restart application. We tried
to work around the bug by creating a new producer - this didn't work.
{code:java}
java.lang.IllegalMonitorStateException: current thread is not owner
at java.base/java.lang.Object.wait(Native Method)
at
org.apache.kafka.common.utils.SystemTime.waitObject(SystemTime.java:55)
at
org.apache.kafka.clients.producer.internals.ProducerMetadata.awaitUpdate(ProducerMetadata.java:119)
at
org.apache.kafka.clients.producer.KafkaProducer.waitOnMetadata(KafkaProducer.java:1032)
at
org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:891)
at
org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:870)
at
org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:758)
< application specific>
at
org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at
org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88)
< application specific>
at jdk.internal.reflect.GeneratedMethodAccessor435.invoke(Unknown
Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633)
at
org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at
org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691)
< application specific>
at jdk.internal.reflect.GeneratedMethodAccessor495.invoke(Unknown
Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at
org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84)
at
org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at
java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at java.base/java.lang.Thread.run(Thread.java:832)
{code}
This bug reproduces several time per year.
> IllegalMonitorStateException in KafkaProducer.waitOnMetadata
> ------------------------------------------------------------
>
> Key: KAFKA-10902
> URL: https://issues.apache.org/jira/browse/KAFKA-10902
> Project: Kafka
> Issue Type: Bug
> Components: producer
> Affects Versions: 2.5.1
> Reporter: M.P. Korstanje
> Priority: Major
>
> We observe the following exception while using
> {{org.apache.kafka:kafka-clients:jar:2.5.1}} as part of a Spring Boot
> application running in a docker container on {{openjdk:13-jdk-alpine3.10}}
> (so {{openjdk 13-ea+32}}).
>
> {code:java}
> j.l.IllegalMonitorStateException: null
> at java.lang.Object.wait(Object.java)
> at o.a.k.common.utils.SystemTime.waitObject(SystemTime.java:55)
> at o.a.k.c.p.i.ProducerMetadata.awaitUpdate(ProducerMetadata.java:119)
> at
> o.a.k.c.producer.KafkaProducer.waitOnMetadata(KafkaProducer.java:1029)
> at o.a.k.c.producer.KafkaProducer.doSend(KafkaProducer.java:883)
> at o.a.k.c.producer.KafkaProducer.send(KafkaProducer.java:862)
> at
> o.s.k.c.DefaultKafkaProducerFactory$CloseSafeProducer.send(DefaultKafkaProducerFactory.java:816)
> at b.k.clients.TracingProducer.send(TracingProducer.java:129)
> at o.s.kafka.core.KafkaTemplate.doSend(KafkaTemplate.java:562)
> at o.s.kafka.core.KafkaTemplate.send(KafkaTemplate.java:401)
> < application specific>
> at j.i.r.GeneratedMethodAccessor167.invoke(Unknown Source)
> at
> j.i.r.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:567)
> at
> o.s.w.m.s.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
> at
> o.s.w.m.s.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
> at
> o.s.w.s.m.m.a.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105)
> at
> o.s.w.s.m.m.a.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:878)
> ... 87 frames truncated
> {code}
>
> The exception occurs when using a {{KafkaTemplate}} ultimately invoking a
> {{KafkaProducer}} to send a message. E..g:
>
> {code:java}
> @Service
> public class Service {
> private final KafkaTemplate<String, UserChangedPinEvent> kafkaTemplate;
> // Constructor ommited
> public void publishEvent(final UUID userId) {
> final Event event = new Event(userId);
> final Message<Event> message = MessageBuilder
> .withPayload(event)
> .setHeader(KafkaHeaders.TOPIC, "some-topic")
> .build();
> kafkaTemplate.send(message);
> }
> }
> {code}
>
> I've not been able to reproduce this in isolation, we have observed this
> exception twice in the last six months. But once the exception occurs, it
> occurs frequently. The system was not under any significant amount of load at
> the time.
>
>
> Looking at the code this exception is unexpected because the
> {{SystemTime.waitObject}} correctly aquires a monitor before calling
> {{Object.wait}}.
> {code:java}
> @Override
> public void waitObject(Object obj, Supplier<Boolean> condition, long
> deadlineMs) throws InterruptedException {
> synchronized (obj) {
> while (true) {
> if (condition.get())
> return;
> long currentTimeMs = milliseconds();
> if (currentTimeMs >= deadlineMs)
> throw new TimeoutException("Condition not satisfied before
> deadline");
> obj.wait(deadlineMs - currentTimeMs);
> }
> }
> }
> {code}
> And in the caller, {{ProducerMetadata.awaitUpdate,}} a the monitor was also
> already acquired.
> {code:java}
> public synchronized void awaitUpdate(final int lastVersion, final long
> timeoutMs) throws InterruptedException {
> long currentTimeMs = time.milliseconds();
> long deadlineMs = currentTimeMs + timeoutMs < 0 ? Long.MAX_VALUE :
> currentTimeMs + timeoutMs;
> time.waitObject(this, () -> {
> // Throw fatal exceptions, if there are any. Recoverable topic errors
> will be handled by the caller.
> maybeThrowFatalException();
> return updateVersion() > lastVersion || isClosed();
> }, deadlineMs);
> if (isClosed())
> throw new KafkaException("Requested metadata update after close");
> }
> {code}
> So it is not clear to me how this exception can occur barring a JDK bug. You
> may want to consider this issue informative.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
