[
https://issues.apache.org/jira/browse/NIFI-6341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-6341:
-----------------------------
Description:
We can occasionally encounter a deadlock if a component is being validated at
the same time that it is being added to a Process Group. Relevant parts of a
thread dump attached.
was:
We can occasionally encounter a deadlock if a component is being validated at
the same time that it is being added to a Process Group. Relevant parts of a
thread dump:
Found one Java-level deadlock: ============================= "Reconnect
ctr-e139-1542663976389-127583-01-000004.hwx.site:8443": waiting for ownable
synchronizer 0x00000000e68fddc8, (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync), which is held
by "NiFi Web Server-294" "NiFi Web Server-294": waiting for ownable
synchronizer 0x00000000e694a2e0, (a
java.util.concurrent.locks.ReentrantLock$NonfairSync), which is held by
"Validate Components Thread-2" "Validate Components Thread-2": waiting for
ownable synchronizer 0x00000000e693c938, (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync), which is held
by "NiFi Web Server-294" Java stack information for the threads listed above:
=================================================== "Reconnect
ctr-e139-1542663976389-127583-01-000004.hwx.site:8443": at
sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000e68fddc8>
(a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
at
org.apache.nifi.groups.StandardProcessGroup.getProcessors(StandardProcessGroup.java:963)
at
org.apache.nifi.controller.serialization.StandardFlowSerializer.addProcessGroup(StandardFlowSerializer.java:211)
at
org.apache.nifi.controller.serialization.StandardFlowSerializer.addProcessGroup(StandardFlowSerializer.java:242)
at
org.apache.nifi.controller.serialization.StandardFlowSerializer.transform(StandardFlowSerializer.java:109)
at
org.apache.nifi.controller.serialization.StandardFlowSerializer.transform(StandardFlowSerializer.java:77)
at
org.apache.nifi.controller.FlowController.serialize(FlowController.java:1268) -
locked <0x00000000e4200c70> (a org.apache.nifi.controller.FlowController) at
org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.save(StandardXMLFlowConfigurationDAO.java:145)
- locked <0x00000000e4777020> (a
org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO) at
org.apache.nifi.controller.StandardFlowService.createDataFlowFromController(StandardFlowService.java:631)
at
org.apache.nifi.controller.StandardFlowService.createDataFlowFromController(StandardFlowService.java:109)
at
org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator$3.run(NodeClusterCoordinator.java:979)
at java.lang.Thread.run(Thread.java:748) "NiFi Web Server-294": at
sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000e694a2e0>
(a java.util.concurrent.locks.ReentrantLock$NonfairSync) at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at
org.apache.nifi.controller.AbstractComponentNode.resetValidationState(AbstractComponentNode.java:632)
at
org.apache.nifi.controller.service.StandardControllerServiceNode.setProcessGroup(StandardControllerServiceNode.java:225)
at
org.apache.nifi.groups.StandardProcessGroup.addControllerService(StandardProcessGroup.java:2055)
at
org.apache.nifi.web.dao.impl.StandardControllerServiceDAO.createControllerService(StandardControllerServiceDAO.java:108)
at
org.apache.nifi.web.dao.impl.StandardControllerServiceDAO$$FastClassBySpringCGLIB$$6b77954.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:736)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at
org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:84)
at
org.apache.nifi.audit.ControllerServiceAuditor.createControllerServiceAdvice(ControllerServiceAuditor.java:79)
at sun.reflect.GeneratedMethodAccessor851.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:627)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:616)
at
org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at
org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:671)
at
org.apache.nifi.web.dao.impl.StandardControllerServiceDAO$$EnhancerBySpringCGLIB$$9dd13ad6.createControllerService(<generated>)
at
org.apache.nifi.web.StandardNiFiServiceFacade.lambda$createControllerService$137(StandardNiFiServiceFacade.java:2208)
at
org.apache.nifi.web.StandardNiFiServiceFacade$$Lambda$675/1037808689.update(Unknown
Source) at
org.apache.nifi.web.revision.NaiveRevisionManager.updateRevision(NaiveRevisionManager.java:117)
at
org.apache.nifi.web.StandardNiFiServiceFacade.createControllerService(StandardNiFiServiceFacade.java:2207)
at
org.apache.nifi.web.StandardNiFiServiceFacade$$FastClassBySpringCGLIB$$358780e0.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:736)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at
org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:84)
at
org.apache.nifi.web.NiFiServiceFacadeLock.proceedWithWriteLock(NiFiServiceFacadeLock.java:173)
at
org.apache.nifi.web.NiFiServiceFacadeLock.createLock(NiFiServiceFacadeLock.java:42)
at sun.reflect.GeneratedMethodAccessor195.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:627)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:616)
at
org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at
org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:671)
at
org.apache.nifi.web.StandardNiFiServiceFacade$$EnhancerBySpringCGLIB$$63530562.createControllerService(<generated>)
at
org.apache.nifi.web.api.ProcessGroupResource.lambda$createControllerService$89(ProcessGroupResource.java:3699)
at
org.apache.nifi.web.api.ProcessGroupResource$$Lambda$674/1852375052.apply(Unknown
Source) at
org.apache.nifi.web.api.ApplicationResource.withWriteLock(ApplicationResource.java:740)
at
org.apache.nifi.web.api.ProcessGroupResource.createControllerService(ProcessGroupResource.java:3664)
at sun.reflect.GeneratedMethodAccessor644.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
at
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$$Lambda$453/1048464106.invoke(Unknown
Source) at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
at
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200)
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) at
org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at
org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at
org.glassfish.jersey.internal.Errors.process(Errors.java:316) at
org.glassfish.jersey.internal.Errors.process(Errors.java:298) at
org.glassfish.jersey.internal.Errors.process(Errors.java:268) at
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
at
org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
at
org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) at
org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865) at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655)
at org.apache.nifi.web.filter.RequestLogger.doFilter(RequestLogger.java:66) at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
at
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
at
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:60)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:60)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:60)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.apache.nifi.web.security.NiFiAuthenticationFilter.authenticate(NiFiAuthenticationFilter.java:84)
at
org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:58)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:96)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
at
org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
at
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)
at
org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at org.apache.nifi.web.filter.TimerFilter.doFilter(TimerFilter.java:51) at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at
org.apache.nifi.web.filter.ExceptionFilter.doFilter(ExceptionFilter.java:46) at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
at
org.apache.nifi.web.security.headers.StrictTransportSecurityFilter.doFilter(StrictTransportSecurityFilter.java:48)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at
org.apache.nifi.web.security.headers.XSSProtectionFilter.doFilter(XSSProtectionFilter.java:48)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at
org.apache.nifi.web.security.headers.ContentSecurityPolicyFilter.doFilter(ContentSecurityPolicyFilter.java:47)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at
org.apache.nifi.web.security.headers.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:48)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
at
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:724)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:61) at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:531) at
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352) at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) at
org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:291) at
org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:151) at
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) at
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748) "Validate Components Thread-2": at
sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000e693c938>
(a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
at
org.apache.nifi.controller.service.StandardControllerServiceNode.getProcessGroup(StandardControllerServiceNode.java:211)
at
org.apache.nifi.controller.service.StandardControllerServiceNode.getProcessGroupIdentifier(StandardControllerServiceNode.java:548)
at
org.apache.nifi.controller.AbstractComponentNode.getValidationContext(AbstractComponentNode.java:732)
at
org.apache.nifi.controller.AbstractComponentNode.performValidation(AbstractComponentNode.java:395)
at
org.apache.nifi.components.validation.StandardValidationTrigger.trigger(StandardValidationTrigger.java:52)
at
org.apache.nifi.components.validation.StandardValidationTrigger.lambda$triggerAsync$0(StandardValidationTrigger.java:45)
at
org.apache.nifi.components.validation.StandardValidationTrigger$$Lambda$633/533962649.run(Unknown
Source) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at
java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) Found 1 deadlock.
> Deadlock can occur during asynchronous validation
> -------------------------------------------------
>
> Key: NIFI-6341
> URL: https://issues.apache.org/jira/browse/NIFI-6341
> Project: Apache NiFi
> Issue Type: Bug
> Reporter: Mark Payne
> Assignee: Mark Payne
> Priority: Major
>
> We can occasionally encounter a deadlock if a component is being validated at
> the same time that it is being added to a Process Group. Relevant parts of a
> thread dump attached.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
