Chakri, Great, glad you're back up & running. There's no need to ensure that the workflows are stopped before restarting -- just need to ensure that nobody is changing the workflow. This is something that we will improve over time, as well.
Thanks -Mark > On Nov 5, 2015, at 2:53 PM, Chakrader Dewaragatla > <[email protected]> wrote: > > Mark Thanks, I moved to another directory to keep it as a backup. > I moved it back to flow.tar and cluster is back online. Make sense, I will > make sure workflows are stopped prior restart. > > Thanks, > -Chakri > > From: Mark Payne <[email protected] <mailto:[email protected]>> > Reply-To: "[email protected] <mailto:[email protected]>" > <[email protected] <mailto:[email protected]>> > Date: Thursday, November 5, 2015 at 11:45 AM > To: "[email protected] <mailto:[email protected]>" > <[email protected] <mailto:[email protected]>> > Subject: Re: nifi stale flow file > > Chakri, > > When the Cluster Manager sends a request to nodes and that request will > change the flow, it marks its flow as "stale" > until it hears back from the nodes whether or not the request was successful. > So it appears that you restarted while > a request was being replicated to the nodes. > > Typically, the answer to this is to rename flow.tar.stale to flow.tar and > restart the Cluster Manager. If you simply remove > the flow.tar you are effectively deleting your entire flow, so be careful > with that! > > When you say you "moved the stale file" what do you mean? You renamed it, > removed it, moved to another directory to keep > it as a backup? > > > > >> On Nov 5, 2015, at 1:53 PM, Chakrader Dewaragatla >> <[email protected] >> <mailto:[email protected]>> wrote: >> >> >> I moved the stale file and it started clean. >> I take it back, server came up clean. Slave nodes fails to connect server >> with following error. >> >> >> ==> nifi-bootstrap.log <== >> 2015-11-05 18:52:37,987 ERROR [NiFi logging handler] org.apache.nifi.StdErr >> Failed to start web server: Unable to load flow due to: java.io.IOException: >> org.apache.nifi.cluster.ConnectionException: Failed to connect node to >> cluster because local flow is different than cluster flow. >> 2015-11-05 18:52:37,988 ERROR [NiFi logging handler] org.apache.nifi.StdErr >> Shutting down... >> >> ==> nifi-app.log <== >> 2015-11-05 18:52:37,988 WARN [main] org.apache.nifi.web.server.JettyServer >> Failed to start web server... shutting down. >> java.lang.Exception: Unable to load flow due to: java.io.IOException: >> org.apache.nifi.cluster.ConnectionException: Failed to connect node to >> cluster because local flow is different than cluster flow. >> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:779) >> ~[nifi-jetty-0.3.0.jar:0.3.0] >> at org.apache.nifi.NiFi.<init>(NiFi.java:137) [nifi-runtime-0.3.0.jar:0.3.0] >> at org.apache.nifi.NiFi.main(NiFi.java:227) [nifi-runtime-0.3.0.jar:0.3.0] >> Caused by: java.io.IOException: org.apache.nifi.cluster.ConnectionException: >> Failed to connect node to cluster because local flow is different than >> cluster flow. >> at >> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:453) >> ~[nifi-framework-core-0.3.0.jar:0.3.0] >> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:770) >> ~[nifi-jetty-0.3.0.jar:0.3.0] >> ... 2 common frames omitted >> Caused by: org.apache.nifi.cluster.ConnectionException: Failed to connect >> node to cluster because local flow is different than cluster flow. >> at >> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:732) >> ~[nifi-framework-core-0.3.0.jar:0.3.0] >> at >> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:433) >> ~[nifi-framework-core-0.3.0.jar:0.3.0] >> ... 3 common frames omitted >> Caused by: org.apache.nifi.controller.UninheritableFlowException: Proposed >> configuration is not inheritable by the flow controller because of flow >> differences: Found difference in Flows: >> Local Fingerprint: >> cf058baf-83bf-46f2-893c-df4efb928d9c1477b29c-0ca2-4bfa-a94e-984a6d9e516forg.apache.nifi.processors.a >> Cluster Fingerprint: 6b5d80a1-d85b-4e3c-8c29-fd525296e266 >> at >> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:216) >> ~[nifi-framework-core-0.3.0.jar:0.3.0] >> at >> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1160) >> ~[nifi-framework-core-0.3.0.jar:0.3.0] >> at >> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:72) >> ~[nifi-framework-core-0.3.0.jar:0.3.0] >> at >> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:608) >> ~[nifi-framework-core-0.3.0.jar:0.3.0] >> at >> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:709) >> ~[nifi-framework-core-0.3.0.jar:0.3.0] >> ... 4 common frames omitted >> >> >> From: Chakrader Dewaragatla <[email protected] >> <mailto:[email protected]>> >> Reply-To: "[email protected] <mailto:[email protected]>" >> <[email protected] <mailto:[email protected]>> >> Date: Thursday, November 5, 2015 at 10:50 AM >> To: "[email protected] <mailto:[email protected]>" >> <[email protected] <mailto:[email protected]>> >> Subject: nifi stale flow file >> >> Hi – I have a clustered setup with NCM and two slaves. After few days of >> stable work, I tried to restart the server and it failed to come up with >> error: >> >> " Failed to instantiate [org.apache.nifi.cluster.flow.impl.DataFlowDaoImpl]: >> Constructor threw exception; nested exception is >> org.apache.nifi.cluster.flow.DaoException: Dataflow state file >> '/usr/local/nifi/./conf/flow.tar.stale' must be current.” >> >> I moved the stale file and it started clean. Any idea why I ran into stale >> file ? >> >> Also, why was it landed in conf dir ? >> >> Thanks, >> -Chakri >> >> >> >> Stack trace : >> >> 2015-11-05 18:27:33,616 ERROR [main] o.s.web.context.ContextLoader Context >> initialization failed >> org.springframework.beans.factory.BeanCreationException: Error creating bean >> with name 'org.springframework.security.filterChains': Cannot resolve >> reference to bean >> 'org.springframework.security.web.DefaultSecurityFilterChain#0' while >> setting bean property 'sourceList' with key [0]; nested exception is >> org.springframework.beans.factory.BeanCreationException: Error creating bean >> with name 'org.springframework.security.web.DefaultSecurityFilterChain#0': >> Cannot resolve reference to bean 'x509Filter' while setting constructor >> argument with key [3]; nested exception is >> org.springframework.beans.factory.BeanCreationException: Error creating bean >> with name 'x509Filter' defined in class path resource >> [nifi-web-security-context.xml]: Cannot resolve reference to bean >> 'authenticationManager' while setting bean property 'authenticationManager'; >> nested exception is org.springframework.beans.factory.BeanCreationException: >> Error creating bean with name >> 'org.springframework.security.authenticationManager': Cannot resolve >> reference to bean 'preauthAuthProvider' while setting constructor argument >> with key [0]; nested exception is >> org.springframework.beans.factory.BeanCreationException: Error creating bean >> with name 'preauthAuthProvider' defined in class path resource >> [nifi-web-security-context.xml]: Cannot create inner bean >> 'org.springframework.security.core.userdetails.UserDetailsByNameServiceWrapper#45987154' >> of type >> [org.springframework.security.core.userdetails.UserDetailsByNameServiceWrapper] >> while setting bean property 'preAuthenticatedUserDetailsService'; nested >> exception is org.springframework.beans.factory.BeanCreationException: Error >> creating bean with name >> 'org.springframework.security.core.userdetails.UserDetailsByNameServiceWrapper#45987154' >> defined in class path resource [nifi-web-security-context.xml]: Cannot >> resolve reference to bean 'userAuthorizationService' while setting bean >> property 'userDetailsService'; nested exception is >> org.springframework.beans.factory.BeanCreationException: Error creating bean >> with name 'userAuthorizationService' defined in class path resource >> [nifi-web-security-context.xml]: Cannot resolve reference to bean >> 'userService' while setting bean property 'userService'; nested exception is >> org.springframework.beans.factory.BeanCreationException: Error creating bean >> with name 'userService' defined in class path resource >> [nifi-administration-context.xml]: Cannot resolve reference to bean >> 'userTransactionBuilder' while setting bean property 'transactionBuilder'; >> nested exception is org.springframework.beans.factory.BeanCreationException: >> Error creating bean with name 'userTransactionBuilder' defined in class path >> resource [nifi-administration-context.xml]: Cannot resolve reference to bean >> 'authorityProvider' while setting bean property 'authorityProvider'; nested >> exception is org.springframework.beans.factory.BeanCreationException: Error >> creating bean with name 'clusterManager': FactoryBean threw exception on >> object creation; nested exception is >> org.springframework.beans.factory.BeanCreationException: Error creating bean >> with name 'dataFlowManagementService' defined in class path resource >> [nifi-cluster-manager-context.xml]: Cannot resolve reference to bean >> 'dataFlowDao' while setting constructor argument; nested exception is >> org.springframework.beans.factory.BeanCreationException: Error creating bean >> with name 'dataFlowDao' defined in class path resource >> [nifi-cluster-manager-context.xml]: Bean instantiation via constructor >> failed; nested exception is >> org.springframework.beans.BeanInstantiationException: Failed to instantiate >> [org.apache.nifi.cluster.flow.impl.DataFlowDaoImpl]: Constructor threw >> exception; nested exception is org.apache.nifi.cluster.flow.DaoException: >> Dataflow state file '/usr/local/nifi/./conf/flow.tar.stale' must be current. >> at >> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359) >> ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE] >> at >> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108) >> ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE] >> at >> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveManagedList(BeanDefinitionValueResolver.java:382) >> ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE] >> at >> org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:157) >> ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE] >> at >> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1477) >> ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE] >> at >> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1222) >> ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE] >> at >> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:537) >> ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE] >> at >> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476) >> ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE] >> at >> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303) >> ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE] >> at >> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) >> ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE] >> at >> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299) >> ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE] >> at >> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194) >> ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE] >> at >> org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:736) >> ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE] >> at >> org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:757) >> ~[spring-context-4.1.6.RELEASE.jar:4.1.6.RELEASE] >> at >> org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480) >> ~[spring-context-4.1.6.RELEASE.jar:4.1.6.RELEASE] >> at >> org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:403) >> ~[spring-web-4.1.6.RELEASE.jar:4.1.6.RELEASE] >> at >> org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306) >> ~[spring-web-4.1.6.RELEASE.jar:4.1.6.RELEASE] >> at >> org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:106) >> [spring-web-4.1.6.RELEASE.jar:4.1.6.RELEASE] >> at >> org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:800) >> [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529] >> at >> org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:444) >> [jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529] >> at >> org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:791) >> [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529] >> at >> org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:294) >> [jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529] >> at >> org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1349) >> [jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529] >> at >> org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342) >> [jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529] >> at >> org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741) >> [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529] >> at >> org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:505) >> [jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529] >> at >> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) >> [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529] >> at >> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) >> [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529] >> at >> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) >> [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529] >> at >> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) >> [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529] >> at >> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) >> [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529] >> at >> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) >> [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529] >> at org.eclipse.jetty.server.Server.start(Server.java:387) >> [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529] >> at >> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) >> [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529] >> at >> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) >> [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529] >> at org.eclipse.jetty.server.Server.doStart(Server.java:354) >> [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529] >> at >> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) >> [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529] >> at >> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:658) >> [nifi-jetty-0.3.0.jar:0.3.0] >> at org.apache.nifi.NiFi.<init>(NiFi.java:137) >> [nifi-runtime-0.3.0.jar:0.3.0] >> at org.apache.nifi.NiFi.main(NiFi.java:227) >> [nifi-runtime-0.3.0.jar:0.3.0] >> >> The information contained in this transmission may contain privileged and >> confidential information. It is intended only for the use of the person(s) >> named above. If you are not the intended recipient, you are hereby notified >> that any review, dissemination, distribution or duplication of this >> communication is strictly prohibited. If you are not the intended recipient, >> please contact the sender by reply email and destroy all copies of the >> original message. >> The information contained in this transmission may contain privileged and >> confidential information. It is intended only for the use of the person(s) >> named above. If you are not the intended recipient, you are hereby notified >> that any review, dissemination, distribution or duplication of this >> communication is strictly prohibited. If you are not the intended recipient, >> please contact the sender by reply email and destroy all copies of the >> original message. > > The information contained in this transmission may contain privileged and > confidential information. It is intended only for the use of the person(s) > named above. If you are not the intended recipient, you are hereby notified > that any review, dissemination, distribution or duplication of this > communication is strictly prohibited. If you are not the intended recipient, > please contact the sender by reply email and destroy all copies of the > original message.
