Benji, I would recommend you try to remove (or rename) the flow.json.gz - but not the flow.xml.gz. See if that makes any difference.
Thanks -Mark Sent from my iPhone On Jul 31, 2022, at 11:35 PM, BeNJ <[email protected]> wrote: Also please see the attached nifi.properties Thanks, Benji On Sun, Jul 31, 2022 at 4:28 PM BeNJ <[email protected]<mailto:[email protected]>> wrote: Hi Joe, Stack with a couple of info logs from before and after, and the final exit shortly after. ------------------------------------------------------------------------------------------ 2022-07-31 16:20:35,311 INFO [main] o.a.n.g.StandardProcessGroupSynchronizer Added Connection[ID=cfee198f-3d2b-1513-f741-e71ad122aaaa, Source ID=cfee198e-3d2b-1513-7a9c-f0c2a8cf0d43, Dest ID=cfee19cf-3d2b-1513-4a87-5f50a90fdabf] to StandardProcessGroup[identifier=cfee1961-3d2b-1513-c8a6-fdf1a8fe4ff5,name=Add Customer User] 2022-07-31 16:20:35,311 INFO [main] o.a.n.g.StandardProcessGroupSynchronizer Added Connection[ID=cfee1974-3d2b-1513-e4df-9dbba1241682, Source ID=cfee1971-3d2b-1513-555c-1aedf0f0801f, Dest ID=cfee1970-3d2b-1513-de1d-f5bee9ad679e] to StandardProcessGroup[identifier=cfee1961-3d2b-1513-c8a6-fdf1a8fe4ff5,name=Add Customer User] 2022-07-31 16:20:35,317 INFO [Timer-Driven Process Thread-9] o.a.n.c.s.StandardControllerServiceNode Successfully enabled StandardControllerServiceNode[service=StandardOauth2AccessTokenProvider[id=cfee1b5b-3d2b-1513-7124-85b028901ac8], name=customer user management idp, active=true] 2022-07-31 16:20:35,325 INFO [main] o.a.n.c.s.AffectedComponentSet Starting the following components: AffectedComponentSet[inputPorts=[], outputPorts=[], remoteInputPorts=[], remoteOutputPorts=[], processors=[], controllerServices=[], reportingTasks=[]] 2022-07-31 16:20:35,328 WARN [main] org.eclipse.jetty.webapp.WebAppContext Failed startup of context o.e.j.w.WebAppContext@ffaaaf0{nifi-api,/nifi-api,file:///opt/nifi/nifi-current/work/jetty/nifi-web-api-1.16.0.war/webapp/,UNAVAILABLE}{./work/nar/extensions/nifi-server-nar-1.16.0.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-api-1.16.0.war} org.apache.nifi.controller.serialization.FlowSynchronizationException: java.lang.IllegalStateException: Cannot add Connection to Process Group because source is an Output Port that does not belong to a child Process Group at org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:362) at org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:185) at org.apache.nifi.controller.serialization.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:43) at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1479) at org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:104) at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:815) at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:538) at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:67) at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1073) at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572) at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:1002) at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:746) at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379) at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449) at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414) at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916) at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288) at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:426) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.server.Server.start(Server.java:423) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) at org.eclipse.jetty.server.Server.doStart(Server.java:387) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1008) at org.apache.nifi.NiFi.<init>(NiFi.java:170) at org.apache.nifi.NiFi.<init>(NiFi.java:82) at org.apache.nifi.NiFi.main(NiFi.java:330) Caused by: java.lang.IllegalStateException: Cannot add Connection to Process Group because source is an Output Port that does not belong to a child Process Group at org.apache.nifi.groups.StandardProcessGroup.addConnection(StandardProcessGroup.java:1154) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.addConnection(StandardProcessGroupSynchronizer.java:1853) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronizeConnections(StandardProcessGroupSynchronizer.java:531) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:388) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.addProcessGroup(StandardProcessGroupSynchronizer.java:1019) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronizeChildGroups(StandardProcessGroupSynchronizer.java:423) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:373) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.addProcessGroup(StandardProcessGroupSynchronizer.java:1019) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronizeChildGroups(StandardProcessGroupSynchronizer.java:423) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:373) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.addProcessGroup(StandardProcessGroupSynchronizer.java:1019) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronizeChildGroups(StandardProcessGroupSynchronizer.java:423) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:373) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.addProcessGroup(StandardProcessGroupSynchronizer.java:1019) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronizeChildGroups(StandardProcessGroupSynchronizer.java:423) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:373) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.addProcessGroup(StandardProcessGroupSynchronizer.java:1019) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronizeChildGroups(StandardProcessGroupSynchronizer.java:423) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:373) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.lambda$synchronize$0(StandardProcessGroupSynchronizer.java:221) at org.apache.nifi.controller.flow.AbstractFlowManager.withParameterContextResolution(AbstractFlowManager.java:462) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:219) at org.apache.nifi.groups.StandardProcessGroup.synchronizeFlow(StandardProcessGroup.java:3830) at org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:353) ... 45 common frames omitted 2022-07-31 16:20:35,552 INFO [Timer-Driven Process Thread-3] o.a.n.c.s.StandardControllerServiceNode Successfully enabled StandardControllerServiceNode[service=MongoDBControllerService[id=19573a5c-3f9e-3cdc-b80c-1990fdd33c69], name=MongoDB CR - Main, active=true] ... ... 2022-07-31 16:20:35,733 INFO [main] org.eclipse.jetty.server.Server Started @26964ms 2022-07-31 16:20:35,734 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down. org.apache.nifi.controller.serialization.FlowSynchronizationException: java.lang.IllegalStateException: Cannot add Connection to Process Group because source is an Output Port that does not belong to a child Process Group at org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:362) at org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:185) at org.apache.nifi.controller.serialization.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:43) at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1479) at org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:104) at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:815) at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:538) at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:67) at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1073) at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572) at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:1002) at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:746) at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379) at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449) at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414) at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916) at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288) at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:426) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.server.Server.start(Server.java:423) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) at org.eclipse.jetty.server.Server.doStart(Server.java:387) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1008) at org.apache.nifi.NiFi.<init>(NiFi.java:170) at org.apache.nifi.NiFi.<init>(NiFi.java:82) at org.apache.nifi.NiFi.main(NiFi.java:330) Caused by: java.lang.IllegalStateException: Cannot add Connection to Process Group because source is an Output Port that does not belong to a child Process Group at org.apache.nifi.groups.StandardProcessGroup.addConnection(StandardProcessGroup.java:1154) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.addConnection(StandardProcessGroupSynchronizer.java:1853) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronizeConnections(StandardProcessGroupSynchronizer.java:531) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:388) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.addProcessGroup(StandardProcessGroupSynchronizer.java:1019) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronizeChildGroups(StandardProcessGroupSynchronizer.java:423) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:373) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.addProcessGroup(StandardProcessGroupSynchronizer.java:1019) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronizeChildGroups(StandardProcessGroupSynchronizer.java:423) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:373) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.addProcessGroup(StandardProcessGroupSynchronizer.java:1019) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronizeChildGroups(StandardProcessGroupSynchronizer.java:423) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:373) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.addProcessGroup(StandardProcessGroupSynchronizer.java:1019) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronizeChildGroups(StandardProcessGroupSynchronizer.java:423) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:373) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.addProcessGroup(StandardProcessGroupSynchronizer.java:1019) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronizeChildGroups(StandardProcessGroupSynchronizer.java:423) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:373) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.lambda$synchronize$0(StandardProcessGroupSynchronizer.java:221) at org.apache.nifi.controller.flow.AbstractFlowManager.withParameterContextResolution(AbstractFlowManager.java:462) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:219) at org.apache.nifi.groups.StandardProcessGroup.synchronizeFlow(StandardProcessGroup.java:3830) at org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:353) ... 45 common frames omitted 2022-07-31 16:20:35,738 INFO [Thread-1] org.apache.nifi.NiFi Application Server shutdown started ------------------------------------------------------------------------------------------ David: df -h for this fs: /dev/mapper/ubuntu--vg-ubuntu--lv 97G 41G 52G 44% / I'll be able to share the nifi.properties in a few hours when I'm back, please let me know if there's anything else I can share or enable. Thanks, Benji On Sun, Jul 31, 2022 at 3:37 PM Joe Witt <[email protected]<mailto:[email protected]>> wrote: Benji What would be helpful to see is the full stacktrace like shown in this stackoverflow post https://stackoverflow.com/questions/72459434/apache-nifi-1-16-0-windows-server-closes-immediately This class changes often so I want to make sure we check the right one for your version. Thanks On Sun, Jul 31, 2022 at 1:25 PM David Early via users <[email protected]<mailto:[email protected]>> wrote: Benji, Can you share your nifi.properties file? This problem can show up if the disk fills up but depends on the configuration. Also, what does df -h show for your disks (assuming you are running on Linux)? Dave On Sun, Jul 31, 2022, 2:21 PM Benji Benning <[email protected]<mailto:[email protected]>> wrote: Not really around the below error. Anything I can move to debug or trace in logback.xml to add some color around this? Thanks, Benji Sent from my iPhone On Jul 31, 2022, at 15:12, Joe Witt <[email protected]<mailto:[email protected]>> wrote: Benji Is there more with the logs like component uuids and such? Thanks On Sun, Jul 31, 2022 at 1:08 PM Benji Benning <[email protected]<mailto:[email protected]>> wrote: Hi Joe, Unfortunately it does indeed contain sensitive info. Is there any debug info I can provide? Thanks, Benji Sent from my iPhone On Jul 31, 2022, at 12:25, Joe Witt <[email protected]<mailto:[email protected]>> wrote: Can you share the flow or does it contain sensitve info? On Sun, Jul 31, 2022 at 10:01 AM BeNJ <[email protected]<mailto:[email protected]>> wrote: Hi, I'm a little at a loss so I figured maybe someone here would have an idea of how to get out of this situation. My NiFI is crashing on startup with the following: -------------------------------------- 2022-07-31 03:10:05,448 INFO [main] o.a.n.c.s.AffectedComponentSet Starting the following components: AffectedComponentSet[inputPorts=[], outputPorts=[], remoteInputPorts=[], remoteOutputPorts=[], processors=[], controllerServices=[], reportingTasks=[]] 2022-07-31 03:10:05,452 WARN [main] org.eclipse.jetty.webapp.WebAppContext Failed startup of context o.e.j.w.WebAppContext@3ed03652{nifi-api,/nifi-api,file:///opt/nifi/nifi-current/work/jetty/nifi-web-api-1.16.0.war/webapp/,UNAVAILABLE}{./work/nar/extensions/nifi-server-nar-1.16.0.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-api-1.16.0.war} org.apache.nifi.controller.serialization.FlowSynchronizationException: java.lang.IllegalStateException: Cannot add Connection to Process Group because source is an Output Port that does not belong to a child Process Group -------------------------------------- I only restarted because out of nowhere i started seeing the following errors repeating: Processors cannot write to Content Repo: Unable to write flowfile content to content repository container default due to archive file size constraints; waiting for archive cleanup To resolve this issue, someone said online to set nifi.content.repository.archive.enabled to false. I restarted and nifi wouldn't come up again. I've tried reverting the flow.xml.gz to the oldest one in the archive folder (was from 7pm yesterday), no luck. it's a large config (1000+ active processors). I would try to disable / delete the offending output port, but I don't see how to find it from the logs. Any direction would be greatly appreciated. Thanks, Benji
<<attachment: nifi_pros.zip>>
