Joe, Looks like there was no other process running but the actual culprit was this property " nifi.remote.input.secure=false". I set the flag back to true and everything started working again and I am really confused by this vague error, right now I am facing a different issue when I'm trying to send data via site to site for the same instance , I keep getting the below trace.
ERROR [NiFi Web Server-646] o.a.nifi.web.api.ApplicationResource Unexpected exception occurred. portId=401a3083-015c-1000-695b-fa269fc7432f 2017-05-25 16:28:47,349 ERROR [NiFi Web Server-646] o.a.nifi.web.api.ApplicationResource Exception detail: org.apache.nifi.processor.exception.ProcessException: org.apache.nifi.processor.exception.FlowFileAccessException: Failed to import data from org.apache.nifi.stream.io.MinimumLengthInputStream@1a1c7b29 for StandardFlowFileRecord[uuid=aebf1e86-bb15-445e-b4fa-4c46e752a761,claim=,offset=0,name=134240548094011,size=0] due to org.apache.nifi.processor.exception.FlowFileAccessException: Unable to create ContentClaim due to org.eclipse.jetty.io.EofException: Early EOF at org.apache.nifi.remote.StandardRootGroupPort.receiveFlowFiles(StandardRootGroupPort.java:543) ~[nifi-site-to-site-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at org.apache.nifi.web.api.DataTransferResource.receiveFlowFiles(DataTransferResource.java:277) ~[classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_101] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_101] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_101] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_101] at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) [jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) [jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) [jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) [jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) [jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) [jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) [jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) [jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) [jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) [jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) [jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) [jersey-server-1.19.jar:1.19] at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) [jersey-servlet-1.19.jar:1.19] at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) [jersey-servlet-1.19.jar:1.19] at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) [jersey-servlet-1.19.jar:1.19] at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [javax.servlet-api-3.1.0.jar:3.1.0] at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845) [jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1689) [jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] at org.apache.nifi.web.filter.RequestLogger.doFilter(RequestLogger.java:66) [classes/:na] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676) [jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:316) [spring-security-web-4.0.3.RELEASE.jar:4.0.3.RELEASE] at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:126) [spring-security-web-4.0.3.RELEASE.jar:4.0.3.RELEASE] at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:90) [spring-security-web-4.0.3.RELEASE.jar:4.0.3.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) [spring-security-web-4.0.3.RELEASE.jar:4.0.3.RELEASE] at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:122) [spring-security-web-4.0.3.RELEASE.jar:4.0.3.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) [spring-security-web-4.0.3.RELEASE.jar:4.0.3.RELEASE] at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) [spring-security-web-4.0.3.RELEASE.jar:4.0.3.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) [spring-security-web-4.0.3.RELEASE.jar:4.0.3.RELEASE] at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:59) [nifi-web-security-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) [spring-security-web-4.0.3.RELEASE.jar:4.0.3.RELEASE] at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:59) [nifi-web-security-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) [spring-security-web-4.0.3.RELEASE.jar:4.0.3.RELEASE] at org.apache.nifi.web.security.NiFiAuthenticationFilter.authenticate(NiFiAuthenticationFilter.java:83) [nifi-web-security-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:57) [nifi-web-security-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) [spring-security-web-4.0.3.RELEASE.jar:4.0.3.RELEASE] at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:213) [spring-security-web-4.0.3.RELEASE.jar:4.0.3.RELEASE] at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:176) [spring-security-web-4.0.3.RELEASE.jar:4.0.3.RELEASE] at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346) [spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262) [spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676) [jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] at org.apache.nifi.web.filter.TimerFilter.doFilter(TimerFilter.java:51) [classes/:na] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668) [jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581) [jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) [jetty-security-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1174) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511) [jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1106) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:459) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.Server.handle(Server.java:524) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:319) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:253) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) [jetty-io-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) [jetty-io-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186) [jetty-io-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) [jetty-io-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) [jetty-io-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) [jetty-io-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101] Caused by: org.apache.nifi.processor.exception.FlowFileAccessException: Failed to import data from org.apache.nifi.stream.io.MinimumLengthInputStream@1a1c7b29 for StandardFlowFileRecord[uuid=aebf1e86-bb15-445e-b4fa-4c46e752a761,claim=,offset=0,name=134240548094011,size=0] due to org.apache.nifi.processor.exception.FlowFileAccessException: Unable to create ContentClaim due to org.eclipse.jetty.io.EofException: Early EOF at org.apache.nifi.controller.repository.StandardProcessSession.importFrom(StandardProcessSession.java:2690) ~[nifi-framework-core-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at org.apache.nifi.remote.protocol.AbstractFlowFileServerProtocol.receiveFlowFiles(AbstractFlowFileServerProtocol.java:448) ~[nifi-site-to-site-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at org.apache.nifi.remote.StandardRootGroupPort.receiveFlowFiles(StandardRootGroupPort.java:262) ~[nifi-site-to-site-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at org.apache.nifi.remote.StandardRootGroupPort.onTrigger(StandardRootGroupPort.java:236) ~[nifi-site-to-site-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at org.apache.nifi.remote.StandardRootGroupPort.onTrigger(StandardRootGroupPort.java:171) ~[nifi-site-to-site-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at org.apache.nifi.controller.tasks.ContinuallyRunConnectableTask.call(ContinuallyRunConnectableTask.java:80) ~[nifi-framework-core-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at org.apache.nifi.controller.tasks.ContinuallyRunConnectableTask.call(ContinuallyRunConnectableTask.java:40) ~[nifi-framework-core-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) ~[nifi-framework-core-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_101] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[na:1.8.0_101] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_101] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[na:1.8.0_101] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_101] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_101] ... 1 common frames omitted Caused by: org.apache.nifi.processor.exception.FlowFileAccessException: Unable to create ContentClaim due to org.eclipse.jetty.io.EofException: Early EOF at org.apache.nifi.controller.repository.StandardProcessSession.importFrom(StandardProcessSession.java:2683) ~[nifi-framework-core-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] ... 14 common frames omitted Caused by: org.eclipse.jetty.io.EofException: Early EOF at org.eclipse.jetty.server.HttpInput$3.noContent(HttpInput.java:821) ~[na:na] at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:160) ~[na:na] at org.apache.nifi.remote.io.InterruptableInputStream.read(InterruptableInputStream.java:57) ~[nifi-utils-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at org.apache.nifi.stream.io.ByteCountingInputStream.read(ByteCountingInputStream.java:51) ~[nifi-utils-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at java.io.DataInputStream.read(DataInputStream.java:149) ~[na:1.8.0_101] at java.util.zip.CheckedInputStream.read(CheckedInputStream.java:82) ~[na:1.8.0_101] at org.apache.nifi.stream.io.LimitingInputStream.read(LimitingInputStream.java:88) ~[nifi-utils-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at java.io.FilterInputStream.read(FilterInputStream.java:133) ~[na:1.8.0_101] at org.apache.nifi.stream.io.MinimumLengthInputStream.read(MinimumLengthInputStream.java:57) ~[nifi-utils-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at org.apache.nifi.stream.io.MinimumLengthInputStream.read(MinimumLengthInputStream.java:53) ~[nifi-utils-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at org.apache.nifi.stream.io.StreamUtils.copy(StreamUtils.java:35) ~[nifi-utils-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at org.apache.nifi.controller.repository.FileSystemRepository.importFrom(FileSystemRepository.java:700) ~[nifi-framework-core-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at org.apache.nifi.controller.repository.StandardProcessSession.importFrom(StandardProcessSession.java:2680) ~[nifi-framework-core-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] ... 14 common frames omitted 2017-05-25 16:28:47,350 WARN [NiFi Web Server-646] org.eclipse.jetty.servlet.ServletHandler /nifi-api/data-transfer/input-ports/401a3083-015c-1000-695b-fa269fc7432f/transactions/ef1aa6a8-4ee4-4628-826b-ece2c9af2441/flow-files java.nio.channels.ClosedChannelException: null at org.eclipse.jetty.util.IteratingCallback.close(IteratingCallback.java:427) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.HttpConnection.onClose(HttpConnection.java:491) ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.io.ssl.SslConnection.onClose(SslConnection.java:152) ~[jetty-io-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.io.SelectorManager.connectionClosed(SelectorManager.java:345) ~[jetty-io-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.io.ManagedSelector$2.run(ManagedSelector.java:442) ~[jetty-io-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101] 2017-05-25 16:28:47,351 WARN [NiFi Web Server-646] org.eclipse.jetty.server.HttpChannel https://domain:8443/nifi-api/data-transfer/input-ports/401a3083-015c-1000-695b-fa269fc7432f/transactions/ef1aa6a8-4ee4-4628-826b-ece2c9af2441/flow-files java.nio.channels.ClosedChannelException: null at org.eclipse.jetty.util.IteratingCallback.close(IteratingCallback.java:427) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.HttpConnection.onClose(HttpConnection.java:491) ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.io.ssl.SslConnection.onClose(SslConnection.java:152) ~[jetty-io-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.io.SelectorManager.connectionClosed(SelectorManager.java:345) ~[jetty-io-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.io.ManagedSelector$2.run(ManagedSelector.java:442) ~[jetty-io-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101] 2017-05-25 16:28:47,922 INFO [Timer-Driven Process Thread-3] o.apache.nifi.remote.client.PeerSelector New Weighted Distribution of Nodes: PeerStatus[hostname=bolhdppbuffer01,port=8443,secure=true,flowFileCount=0] will receive 100.0% of data 2017-05-25 16:28:51,155 WARN [Http Site-to-Site PeerSelector] o.a.n.r.util.SiteToSiteRestApiClient Failed to get controller from https://tslhdppbuffer01:8443/nifi-api due to org.apache.nifi.remote.util.SiteToSiteRestApiClient$HttpGetFailedException: response code 403:Forbidden with explanation: null 2017-05-25 16:28:51,155 WARN [Http Site-to-Site PeerSelector] o.apache.nifi.remote.client.PeerSelector org.apache.nifi.remote.client.PeerSelector@72b85c29 Unable to refresh Remote Group's peers due to response code 403:Forbidden with explanation: null 2017-05-25 16:28:51,384 ERROR [I/O dispatcher 462] o.a.n.r.util.SiteToSiteRestApiClient Failed to send data to https://domain:8443/nifi-api/data-transfer/input-ports/401a3083-015c-1000-695b-fa269fc7432f/transactions/07c2125d-ac66-4b19-94e5-bc72a0a31632/flow-files due to java.net.SocketTimeoutException java.net.SocketTimeoutException: null at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:375) [httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92) [httpasyncclient-4.1.2.jar:4.1.2] at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39) [httpasyncclient-4.1.2.jar:4.1.2] at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175) [httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:263) [httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:492) [httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:213) [httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) [httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) [httpcore-nio-4.4.5.jar:4.4.5] at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588) [httpcore-nio-4.4.5.jar:4.4.5] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101] 2017-05-25 16:28:51,386 ERROR [Timer-Driven Process Thread-8] o.a.nifi.remote.StandardRemoteGroupPort RemoteGroupPort[name=test,targets=https://domain:8443/nifi/] failed to communicate with remote NiFi instance due to org.apache.nifi.processor.exception.ProcessException: IOException thrown from RemoteGroupPort[name=test,targets=https://domain:8443/nifi/]: java.io.IOException: Failed to send data to Peer[url=https://domain:8443/nifi-api] due to java.io.IOException: Pipe closed 2017-05-25 16:28:51,920 INFO [Timer-Driven Process Thread-3] o.a.nifi.remote.StandardRemoteGroupPort RemoteGroupPort[name=Receive_Sigma,targets=https://bolhdppbuffer01:8443/nifi/] Successfully sent [StandardFlowFileRecord[uuid=4ebf85ae-b2c6-46ae-9c67-e292da308416,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1495725985075-657, container=default, section=657], offset=0, length=821579],offset=0,name=sigma_measurement.dat.2017_05_25_00_40_02.9736_65.gz,size=821579]] (802.32 KB) to https://bolhdppbuffer01:8443/nifi-api in 2452 milliseconds at a rate of 327.12 KB/sec 2017-05-25 16:28:52,643 ERROR [NiFi Web Server-17] o.a.nifi.web.api.ApplicationResource Unexpected exception occurred. portId=401a3083-015c-1000-695b-fa269fc7432f 2017-05-25 16:28:52,649 ERROR [NiFi Web Server-17] o.a.nifi.web.api.ApplicationResource Exception detail: org.apache.nifi.processor.exception.ProcessException: org.apache.nifi.processor.exception.FlowFileAccessException: Failed to import data from org.apache.nifi.stream.io.MinimumLengthInputStream@6573eb8e for StandardFlowFileRecord[uuid=a9a884f1-4c18-44b6-b549-344ee9d1efa2,claim=,offset=0,name=134245802899530,size=0] due to org.apache.nifi.processor.exception.FlowFileAccessException: Unable to create ContentClaim due to org.eclipse.jetty.io.EofException: Early EOF at org.apache.nifi.remote.StandardRootGroupPort.receiveFlowFiles(StandardRootGroupPort.java:543) ~[nifi-site-to-site-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at org.apache.nifi.web.api.DataTransferResource.receiveFlowFiles(DataTransferResource.java:277) ~[classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_101] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_101] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_101] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_101] at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) [jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) [jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) [jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) [jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) [jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) [jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) [jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) [jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) [jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) [jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) [jersey-server-1.19.jar:1.19] at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) [jersey-server-1.19.jar:1.19] at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) [jersey-servlet-1.19.jar:1.19] at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) [jersey-servlet-1.19.jar:1.19] at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) [jersey-servlet-1.19.jar:1.19] at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [javax.servlet-api-3.1.0.jar:3.1.0] at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845) [jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1689) [jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] at org.apache.nifi.web.filter.RequestLogger.doFilter(RequestLogger.java:66) [classes/:na] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676) [jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:316) [spring-security-web-4.0.3.RELEASE.jar:4.0.3.RELEASE] at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:126) [spring-security-web-4.0.3.RELEASE.jar:4.0.3.RELEASE] at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:90) [spring-security-web-4.0.3.RELEASE.jar:4.0.3.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) [spring-security-web-4.0.3.RELEASE.jar:4.0.3.RELEASE] at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:122) [spring-security-web-4.0.3.RELEASE.jar:4.0.3.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) [spring-security-web-4.0.3.RELEASE.jar:4.0.3.RELEASE] at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) [spring-security-web-4.0.3.RELEASE.jar:4.0.3.RELEASE] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) [spring-security-web-4.0.3.RELEASE.jar:4.0.3.RELEASE] at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:59) [nifi-web-security-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) [spring-security-web-4.0.3.RELEASE.jar:4.0.3.RELEASE] at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:59) [nifi-web-security-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) [spring-security-web-4.0.3.RELEASE.jar:4.0.3.RELEASE] at org.apache.nifi.web.security.NiFiAuthenticationFilter.authenticate(NiFiAuthenticationFilter.java:83) [nifi-web-security-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:57) [nifi-web-security-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) [spring-security-web-4.0.3.RELEASE.jar:4.0.3.RELEASE] at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:213) [spring-security-web-4.0.3.RELEASE.jar:4.0.3.RELEASE] at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:176) [spring-security-web-4.0.3.RELEASE.jar:4.0.3.RELEASE] at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346) [spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262) [spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676) [jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] at org.apache.nifi.web.filter.TimerFilter.doFilter(TimerFilter.java:51) [classes/:na] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668) [jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581) [jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) [jetty-security-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1174) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511) [jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1106) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:459) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.Server.handle(Server.java:524) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:319) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:253) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) [jetty-io-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) [jetty-io-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186) [jetty-io-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) [jetty-io-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) [jetty-io-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) [jetty-io-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101] Caused by: org.apache.nifi.processor.exception.FlowFileAccessException: Failed to import data from org.apache.nifi.stream.io.MinimumLengthInputStream@6573eb8e for StandardFlowFileRecord[uuid=a9a884f1-4c18-44b6-b549-344ee9d1efa2,claim=,offset=0,name=134245802899530,size=0] due to org.apache.nifi.processor.exception.FlowFileAccessException: Unable to create ContentClaim due to org.eclipse.jetty.io.EofException: Early EOF at org.apache.nifi.controller.repository.StandardProcessSession.importFrom(StandardProcessSession.java:2690) ~[nifi-framework-core-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at org.apache.nifi.remote.protocol.AbstractFlowFileServerProtocol.receiveFlowFiles(AbstractFlowFileServerProtocol.java:448) ~[nifi-site-to-site-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at org.apache.nifi.remote.StandardRootGroupPort.receiveFlowFiles(StandardRootGroupPort.java:262) ~[nifi-site-to-site-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at org.apache.nifi.remote.StandardRootGroupPort.onTrigger(StandardRootGroupPort.java:236) ~[nifi-site-to-site-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at org.apache.nifi.remote.StandardRootGroupPort.onTrigger(StandardRootGroupPort.java:171) ~[nifi-site-to-site-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at org.apache.nifi.controller.tasks.ContinuallyRunConnectableTask.call(ContinuallyRunConnectableTask.java:80) ~[nifi-framework-core-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at org.apache.nifi.controller.tasks.ContinuallyRunConnectableTask.call(ContinuallyRunConnectableTask.java:40) ~[nifi-framework-core-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) ~[nifi-framework-core-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_101] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[na:1.8.0_101] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_101] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[na:1.8.0_101] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_101] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_101] ... 1 common frames omitted Caused by: org.apache.nifi.processor.exception.FlowFileAccessException: Unable to create ContentClaim due to org.eclipse.jetty.io.EofException: Early EOF at org.apache.nifi.controller.repository.StandardProcessSession.importFrom(StandardProcessSession.java:2683) ~[nifi-framework-core-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] ... 14 common frames omitted Caused by: org.eclipse.jetty.io.EofException: Early EOF at org.eclipse.jetty.server.HttpInput$3.noContent(HttpInput.java:821) ~[na:na] at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:160) ~[na:na] at org.apache.nifi.remote.io.InterruptableInputStream.read(InterruptableInputStream.java:57) ~[nifi-utils-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at org.apache.nifi.stream.io.ByteCountingInputStream.read(ByteCountingInputStream.java:51) ~[nifi-utils-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at java.io.DataInputStream.read(DataInputStream.java:149) ~[na:1.8.0_101] at java.util.zip.CheckedInputStream.read(CheckedInputStream.java:82) ~[na:1.8.0_101] at org.apache.nifi.stream.io.LimitingInputStream.read(LimitingInputStream.java:88) ~[nifi-utils-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at java.io.FilterInputStream.read(FilterInputStream.java:133) ~[na:1.8.0_101] at org.apache.nifi.stream.io.MinimumLengthInputStream.read(MinimumLengthInputStream.java:57) ~[nifi-utils-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at org.apache.nifi.stream.io.MinimumLengthInputStream.read(MinimumLengthInputStream.java:53) ~[nifi-utils-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at org.apache.nifi.stream.io.StreamUtils.copy(StreamUtils.java:35) ~[nifi-utils-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at org.apache.nifi.controller.repository.FileSystemRepository.importFrom(FileSystemRepository.java:700) ~[nifi-framework-core-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] at org.apache.nifi.controller.repository.StandardProcessSession.importFrom(StandardProcessSession.java:2680) ~[nifi-framework-core-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] ... 14 common frames omitted 2017-05-25 16:28:52,651 WARN [NiFi Web Server-17] org.eclipse.jetty.servlet.ServletHandler /nifi-api/data-transfer/input-ports/401a3083-015c-1000-695b-fa269fc7432f/transactions/68e7b4ac-6b81-4dca-a530-6cbd2b314365/flow-files java.nio.channels.ClosedChannelException: null at org.eclipse.jetty.util.IteratingCallback.close(IteratingCallback.java:427) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.HttpConnection.onClose(HttpConnection.java:491) ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.io.ssl.SslConnection.onClose(SslConnection.java:152) ~[jetty-io-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.io.SelectorManager.connectionClosed(SelectorManager.java:345) ~[jetty-io-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.io.ManagedSelector$2.run(ManagedSelector.java:442) ~[jetty-io-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101] 2017-05-25 16:28:52,651 WARN [NiFi Web Server-17] org.eclipse.jetty.server.HttpChannel https://domain:8443/nifi-api/data-transfer/input-ports/401a3083-015c-1000-695b-fa269fc7432f/transactions/68e7b4ac-6b81-4dca-a530-6cbd2b314365/flow-files java.nio.channels.ClosedChannelException: null at org.eclipse.jetty.util.IteratingCallback.close(IteratingCallback.java:427) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.server.HttpConnection.onClose(HttpConnection.java:491) ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.io.ssl.SslConnection.onClose(SslConnection.java:152) ~[jetty-io-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.io.SelectorManager.connectionClosed(SelectorManager.java:345) ~[jetty-io-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.io.ManagedSelector$2.run(ManagedSelector.java:442) ~[jetty-io-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101] Below are the settings I've been using # Site to Site properties nifi.remote.input.host=domain nifi.remote.input.secure=true nifi.remote.input.socket.port=10443 nifi.remote.input.http.enabled=true nifi.remote.input.http.transaction.ttl=30 sec # web properties # nifi.web.war.directory=./lib nifi.web.http.host= nifi.web.http.port= nifi.web.https.host=domain nifi.web.https.port=8443 nifi.web.jetty.working.directory=./work/jetty nifi.web.jetty.threads=200 Thanks Karthik -----Original Message----- From: Joe Witt [mailto:joe.w...@gmail.com] Sent: Thursday, May 25, 2017 10:14 AM To: dev@nifi.apache.org Subject: [EXT] Re: OverlappingFileLockException while restarting most likely there is another instance of nifi still running. Check ps -ef | grep nifi. Kill that and try again. Thanks On Thu, May 25, 2017 at 12:10 PM, Karthik Kothareddy (karthikk) [CONT - Type 2] <karth...@micron.com> wrote: > Hello, > > I am running 1.2.0 Snapshot on a Linux instance with some custom > processors. I was trying to update a certificate this morning and > restart the instance and I'm stuck with the > "OverlappingFileLockException". The instance would not start and soon > after I start it ,it will shutdown with the following logs in > bootstrap.log > > 2017-05-25 15:37:12,935 ERROR [NiFi logging handler] org.apache.nifi.StdErr > Failed to start web server: Unable to start Flow Controller. > 2017-05-25 15:37:12,936 ERROR [NiFi logging handler] org.apache.nifi.StdErr > Shutting down... > 2017-05-25 15:37:13,884 INFO [main] org.apache.nifi.bootstrap.RunNiFi > NiFi never started. Will not restart NiFi > > However, I was going through the app-log to make sure everything is > okay and I found the below trace > > > WARN [Thread-1] org.apache.nifi.web.server.JettyServer Failed to stop > web server > org.springframework.beans.factory.BeanCreationException: Error creating bean > with name 'flowService': FactoryBean threw exception on object creation; > nested exception is org.springframework.beans.factory.BeanCreationException: > Error creating bean with name 'flowController': FactoryBean threw exception > on object creation; nested exception is java.lang.RuntimeException: > java.nio.channels.OverlappingFileLockException > at > org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175) > ~[na:na] > at > org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103) > ~[na:na] > at > org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1585) > ~[na:na] > at > org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:254) > ~[na:na] > at > org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) > ~[na:na] > at > org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1060) > ~[na:na] > at > org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextDestroyed(ApplicationStartupContextListener.java:103) > ~[na:na] > at > org.eclipse.jetty.server.handler.ContextHandler.callContextDestroyed(ContextHandler.java:845) > ~[na:na] > at > org.eclipse.jetty.servlet.ServletContextHandler.callContextDestroyed(ServletContextHandler.java:546) > ~[na:na] > at > org.eclipse.jetty.server.handler.ContextHandler.stopContext(ContextHandler.java:826) > ~[na:na] > at > org.eclipse.jetty.servlet.ServletContextHandler.stopContext(ServletContextHandler.java:356) > ~[na:na] > at > org.eclipse.jetty.webapp.WebAppContext.stopWebapp(WebAppContext.java:1410) > ~[na:na] > at > org.eclipse.jetty.webapp.WebAppContext.stopContext(WebAppContext.java:1374) > ~[na:na] > at > org.eclipse.jetty.server.handler.ContextHandler.doStop(ContextHandler.java:874) > ~[na:na] > at > org.eclipse.jetty.servlet.ServletContextHandler.doStop(ServletContextHandler.java:272) > ~[na:na] > at > org.eclipse.jetty.webapp.WebAppContext.doStop(WebAppContext.java:544) ~[na:na] > at > org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) > ~[na:na] > at > org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:143) > ~[na:na] > at > org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:161) > ~[na:na] > at > org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:73) > ~[na:na] > at > org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) > ~[na:na] > at > org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:143) > ~[na:na] > at > org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:161) > ~[na:na] > at > org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:73) > ~[na:na] > at > org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) > ~[na:na] > at > org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:143) > ~[na:na] > at > org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:161) > ~[na:na] > at > org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:73) > ~[na:na] > at org.eclipse.jetty.server.Server.doStop(Server.java:482) ~[na:na] > at > org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) > ~[na:na] > at org.apache.nifi.web.server.JettyServer.stop(JettyServer.java:854) > ~[na:na] > at org.apache.nifi.NiFi.shutdownHook(NiFi.java:188) > [nifi-runtime-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at org.apache.nifi.NiFi$2.run(NiFi.java:89) > [nifi-runtime-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101] Caused > by: org.springframework.beans.factory.BeanCreationException: Error creating > bean with name 'flowController': FactoryBean threw exception on object > creation; nested exception is java.lang.RuntimeException: > java.nio.channels.OverlappingFileLockException > at > org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175) > ~[na:na] > at > org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103) > ~[na:na] > at > org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1585) > ~[na:na] > at > org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:254) > ~[na:na] > at > org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) > ~[na:na] > at > org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1060) > ~[na:na] > at > org.apache.nifi.spring.StandardFlowServiceFactoryBean.getObject(StandardFlowServiceFactoryBean.java:48) > ~[na:na] > at > org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168) > ~[na:na] > ... 33 common frames omitted > Caused by: java.lang.RuntimeException: > java.nio.channels.OverlappingFileLockException > at > org.apache.nifi.controller.FlowController.createFlowFileRepository(FlowController.java:641) > ~[na:na] > at > org.apache.nifi.controller.FlowController.<init>(FlowController.java:447) > ~[na:na] > at > org.apache.nifi.controller.FlowController.createStandaloneInstance(FlowController.java:375) > ~[na:na] > at > org.apache.nifi.spring.FlowControllerFactoryBean.getObject(FlowControllerFactoryBean.java:74) > ~[na:na] > at > org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168) > ~[na:na] > ... 40 common frames omitted > Caused by: java.nio.channels.OverlappingFileLockException: null > at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255) > ~[na:1.8.0_101] > at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152) > ~[na:1.8.0_101] > at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1063) > ~[na:1.8.0_101] > at java.nio.channels.FileChannel.lock(FileChannel.java:1053) > ~[na:1.8.0_101] > at > org.wali.MinimalLockingWriteAheadLog.<init>(MinimalLockingWriteAheadLog.java:187) > ~[na:na] > at > org.wali.MinimalLockingWriteAheadLog.<init>(MinimalLockingWriteAheadLog.java:112) > ~[na:na] > at > org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.initialize(WriteAheadFlowFileRepository.java:148) > ~[na:na] > at > org.apache.nifi.controller.FlowController.createFlowFileRepository(FlowController.java:637) > ~[na:na] > ... 44 common frames omitted > 2017-05-25 15:41:28,117 INFO [Thread-1] org.apache.nifi.NiFi Jetty web server > shutdown completed (nicely or otherwise). > > > Can anyone suggest anything at this point, I double checked all the server > configurations like jetty properties and web ports, everything seems to be > normal. I still cannot think of why this would be happening all of a sudden. > Any help on this would be appreciated. > > > Thanks > Karthik > > > > >