Chakri, If you restore the flow.tar then that means that the request that was being made will not have been successful. As a result, it's possible to have 1 or more nodes (those that did complete the request) complaining that the flow is different from the Cluster's flow upon restart. When this happens, we usually just remove the flow.xml.gz from the node and restart -- this will cause the node to pull the flow from the Cluster Manager and use that flow.
Thanks -Mark > On Nov 5, 2015, at 3:07 PM, Chakrader Dewaragatla > <[email protected]> wrote: > > Question – After I moved the stale file and recovered the cluster, how does > it perform the pending requests that were in pipeline ? > I am trying to understand if a instance crash in uneventful situation, how do > I recover the data or the flow state. > > Thanks, > -Chakri > > From: Chakrader Dewaragatla <[email protected] > <mailto:[email protected]>> > Reply-To: "[email protected] <mailto:[email protected]>" > <[email protected] <mailto:[email protected]>> > Date: Thursday, November 5, 2015 at 11:53 AM > To: "[email protected] <mailto:[email protected]>" > <[email protected] <mailto:[email protected]>> > Subject: Re: nifi stale flow file > > Mark Thanks, I moved to another directory to keep it as a backup. > I moved it back to flow.tar and cluster is back online. Make sense, I will > make sure workflows are stopped prior restart. > > Thanks, > -Chakri > > From: Mark Payne <[email protected] <mailto:[email protected]>> > Reply-To: "[email protected] <mailto:[email protected]>" > <[email protected] <mailto:[email protected]>> > Date: Thursday, November 5, 2015 at 11:45 AM > To: "[email protected] <mailto:[email protected]>" > <[email protected] <mailto:[email protected]>> > Subject: Re: nifi stale flow file > > Chakri, > > When the Cluster Manager sends a request to nodes and that request will > change the flow, it marks its flow as "stale" > until it hears back from the nodes whether or not the request was successful. > So it appears that you restarted while > a request was being replicated to the nodes. > > Typically, the answer to this is to rename flow.tar.stale to flow.tar and > restart the Cluster Manager. If you simply remove > the flow.tar you are effectively deleting your entire flow, so be careful > with that! > > When you say you "moved the stale file" what do you mean? You renamed it, > removed it, moved to another directory to keep > it as a backup? > > > > >> On Nov 5, 2015, at 1:53 PM, Chakrader Dewaragatla >> <[email protected] >> <mailto:[email protected]>> wrote: >> >> >> I moved the stale file and it started clean. >> I take it back, server came up clean. Slave nodes fails to connect server >> with following error. >> >> >> ==> nifi-bootstrap.log <== >> 2015-11-05 18:52:37,987 ERROR [NiFi logging handler] org.apache.nifi.StdErr >> Failed to start web server: Unable to load flow due to: java.io.IOException: >> org.apache.nifi.cluster.ConnectionException: Failed to connect node to >> cluster because local flow is different than cluster flow. >> 2015-11-05 18:52:37,988 ERROR [NiFi logging handler] org.apache.nifi.StdErr >> Shutting down... >> >> ==> nifi-app.log <== >> 2015-11-05 18:52:37,988 WARN [main] org.apache.nifi.web.server.JettyServer >> Failed to start web server... shutting down. >> java.lang.Exception: Unable to load flow due to: java.io.IOException: >> org.apache.nifi.cluster.ConnectionException: Failed to connect node to >> cluster because local flow is different than cluster flow. >> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:779) >> ~[nifi-jetty-0.3.0.jar:0.3.0] >> at org.apache.nifi.NiFi.<init>(NiFi.java:137) [nifi-runtime-0.3.0.jar:0.3.0] >> at org.apache.nifi.NiFi.main(NiFi.java:227) [nifi-runtime-0.3.0.jar:0.3.0] >> Caused by: java.io.IOException: org.apache.nifi.cluster.ConnectionException: >> Failed to connect node to cluster because local flow is different than >> cluster flow. >> at >> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:453) >> ~[nifi-framework-core-0.3.0.jar:0.3.0] >> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:770) >> ~[nifi-jetty-0.3.0.jar:0.3.0] >> ... 2 common frames omitted >> Caused by: org.apache.nifi.cluster.ConnectionException: Failed to connect >> node to cluster because local flow is different than cluster flow. >> at >> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:732) >> ~[nifi-framework-core-0.3.0.jar:0.3.0] >> at >> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:433) >> ~[nifi-framework-core-0.3.0.jar:0.3.0] >> ... 3 common frames omitted >> Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed >> configuration is not inheritable by the flow controller because of flow >> differences: Found difference in Flows: >> Local Fingerprint: >> cf058baf-83bf-46f2-893c-df4efb928d9c1477b29c-0ca2-4bfa-a94e-984a6d9e516forg.apache.nifi.processors.a >> Cluster Fingerprint: 6b5d80a1-d85b-4e3c-8c29-fd525296e266 >> at >> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:216) >> ~[nifi-framework-core-0.3.0.jar:0.3.0] >> at >> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1160) >> ~[nifi-framework-core-0.3.0.jar:0.3.0] >> at >> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:72) >> ~[nifi-framework-core-0.3.0.jar:0.3.0] >> at >> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:608) >> ~[nifi-framework-core-0.3.0.jar:0.3.0] >> at >> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:709) >> ~[nifi-framework-core-0.3.0.jar:0.3.0] >> ... 4 common frames omitted >> >> >> From: Chakrader Dewaragatla <[email protected] >> <mailto:[email protected]>> >> Reply-To: "[email protected] <mailto:[email protected]>" >> <[email protected] <mailto:[email protected]>> >> Date: Thursday, November 5, 2015 at 10:50 AM >> To: "[email protected] <mailto:[email protected]>" >> <[email protected] <mailto:[email protected]>> >> Subject: nifi stale flow file >> >> Hi – I have a clustered setup with NCM and two slaves. After few days of >> stable work, I tried to restart the server and it failed to come up with >> error: >> >> " Failed to instantiate [org.apache.nifi.cluster.flow.impl.DataFlowDaoImpl]: >> Constructor threw exception; nested exception is >> org.apache.nifi.cluster.flow.DaoException: Dataflow state file >> '/usr/local/nifi/./conf/flow.tar.stale' must be current.” >> >> I moved the stale file and it started clean. Any idea why I ran into stale >> file ? >> >> Also, why was it landed in conf dir ? >> >> Thanks, >> -Chakri >> >> >> >> Stack trace : >> >> 2015-11-05 18:27:33,616 ERROR [main] o.s.web.context.ContextLoader Context >> initialization failed >> org.springframework.beans.factory.BeanCreationException: Error creating bean >> with name 'org.springframework.security.filterChains': Cannot resolve >> reference to bean >> 'org.springframework.security.web.DefaultSecurityFilterChain#0' while >> setting bean property 'sourceList' with key [0]; nested exception is >> org.springframework.beans.factory.BeanCreationException: Error creating bean >> with name 'org.springframework.security.web.DefaultSecurityFilterChain#0': >> Cannot resolve reference to bean 'x509Filter' while setting constructor >> argument with key [3]; nested exception is >> org.springframework.beans.factory.BeanCreationException: Error creating bean >> with name 'x509Filter' defined in class path resource >> [nifi-web-security-context.xml]: Cannot resolve reference to bean >> 'authenticationManager' while setting bean property 'authenticationManager'; >> nested exception is org.springframework.beans.factory.BeanCreationException: >> Error creating bean with name >> 'org.springframework.security.authenticationManager': Cannot resolve >> reference to bean 'preauthAuthProvider' while setting constructor argument >> with key [0]; nested exception is >> org.springframework.beans.factory.BeanCreationException: Error creating bean >> with name 'preauthAuthProvider' defined in class path resource >> [nifi-web-security-context.xml]: Cannot create inner bean >> 'org.springframework.security.core.userdetails.UserDetailsByNameServiceWrapper#45987154' >> of type >> [org.springframework.security.core.userdetails.UserDetailsByNameServiceWrapper] >> while setting bean property 'preAuthenticatedUserDetailsService'; nested >> exception is org.springframework.beans.factory.BeanCreationException: Error >> creating bean with name >> 'org.springframework.security.core.userdetails.UserDetailsByNameServiceWrapper#45987154' >> defined in class path resource [nifi-web-security-context.xml]: Cannot >> resolve reference to bean 'userAuthorizationService' while setting bean >> property 'userDetailsService'; nested exception is >> org.springframework.beans.factory.BeanCreationException: Error creating bean >> with name 'userAuthorizationService' defined in class path resource >> [nifi-web-security-context.xml]: Cannot resolve reference to bean >> 'userService' while setting bean property 'userService'; nested exception is >> org.springframework.beans.factory.BeanCreationException: Error creating bean >> with name 'userService' defined in class path resource >> [nifi-administration-context.xml]: Cannot resolve reference to bean >> 'userTransactionBuilder' while setting bean property 'transactionBuilder'; >> nested exception is org.springframework.beans.factory.BeanCreationException: >> Error creating bean with name 'userTransactionBuilder' defined in class path >> resource [nifi-administration-context.xml]: Cannot resolve reference to bean >> 'authorityProvider' while setting bean property 'authorityProvider'; nested >> exception is org.springframework.beans.factory.BeanCreationException: Error >> creating bean with name 'clusterManager': FactoryBean threw exception on >> object creation; nested exception is >> org.springframework.beans.factory.BeanCreationException: Error creating bean >> with name 'dataFlowManagementService' defined in class path resource >> [nifi-cluster-manager-context.xml]: Cannot resolve reference to bean >> 'dataFlowDao' while setting constructor argument; nested exception is >> org.springframework.beans.factory.BeanCreationException: Error creating bean >> with name 'dataFlowDao' defined in class path resource >> [nifi-cluster-manager-context.xml]: Bean instantiation via constructor >> failed; nested exception is >> org.springframework.beans.BeanInstantiationException: Failed to instantiate >> [org.apache.nifi.cluster.flow.impl.DataFlowDaoImpl]: Constructor threw >> exception; nested exception is org.apache.nifi.cluster.flow.DaoException: >> Dataflow state file '/usr/local/nifi/./conf/flow.tar.stale' must be current. >> at >> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359) >> ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE] >> at >> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108) >> ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE] >> at >> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveManagedList(BeanDefinitionValueResolver.java:382) >> ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE] >> at >> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:157) >> ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE] >> at >> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1477) >> ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE] >> at >> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1222) >> ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE] >> at >> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:537) >> ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE] >> at >> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476) >> ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE] >> at >> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303) >> ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE] >> at >> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) >> ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE] >> at >> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299) >> ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE] >> at >> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194) >> ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE] >> at >> org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:736) >> ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE] >> at >> org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:757) >> ~[spring-context-4.1.6.RELEASE.jar:4.1.6.RELEASE] >> at >> org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480) >> ~[spring-context-4.1.6.RELEASE.jar:4.1.6.RELEASE] >> at >> org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:403) >> ~[spring-web-4.1.6.RELEASE.jar:4.1.6.RELEASE] >> at >> org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306) >> ~[spring-web-4.1.6.RELEASE.jar:4.1.6.RELEASE] >> at >> org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:106) >> [spring-web-4.1.6.RELEASE.jar:4.1.6.RELEASE] >> at >> org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:800) >> [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529] >> at >> org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:444) >> [jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529] >> at >> org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:791) >> [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529] >> at >> org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:294) >> [jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529] >> at >> org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1349) >> [jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529] >> at >> org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342) >> [jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529] >> at >> org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741) >> [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529] >> at >> org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:505) >> [jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529] >> at >> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) >> [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529] >> at >> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) >> [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529] >> at >> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) >> [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529] >> at >> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) >> [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529] >> at >> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) >> [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529] >> at >> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) >> [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529] >> at org.eclipse.jetty.server.Server.start(Server.java:387) >> [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529] >> at >> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) >> [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529] >> at >> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) >> [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529] >> at org.eclipse.jetty.server.Server.doStart(Server.java:354) >> [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529] >> at >> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) >> [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529] >> at >> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:658) >> [nifi-jetty-0.3.0.jar:0.3.0] >> at org.apache.nifi.NiFi.<init>(NiFi.java:137) >> [nifi-runtime-0.3.0.jar:0.3.0] >> at org.apache.nifi.NiFi.main(NiFi.java:227) >> [nifi-runtime-0.3.0.jar:0.3.0] >> >> The information contained in this transmission may contain privileged and >> confidential information. It is intended only for the use of the person(s) >> named above. If you are not the intended recipient, you are hereby notified >> that any review, dissemination, distribution or duplication of this >> communication is strictly prohibited. If you are not the intended recipient, >> please contact the sender by reply email and destroy all copies of the >> original message. >> The information contained in this transmission may contain privileged and >> confidential information. It is intended only for the use of the person(s) >> named above. If you are not the intended recipient, you are hereby notified >> that any review, dissemination, distribution or duplication of this >> communication is strictly prohibited. If you are not the intended recipient, >> please contact the sender by reply email and destroy all copies of the >> original message. > > The information contained in this transmission may contain privileged and > confidential information. It is intended only for the use of the person(s) > named above. If you are not the intended recipient, you are hereby notified > that any review, dissemination, distribution or duplication of this > communication is strictly prohibited. If you are not the intended recipient, > please contact the sender by reply email and destroy all copies of the > original message. > The information contained in this transmission may contain privileged and > confidential information. It is intended only for the use of the person(s) > named above. If you are not the intended recipient, you are hereby notified > that any review, dissemination, distribution or duplication of this > communication is strictly prohibited. If you are not the intended recipient, > please contact the sender by reply email and destroy all copies of the > original message.
