Benji, Fantastic, thanks for following up! We’ve clearly got something to address but I’m very happy that we’ve found a way to get you unblocked for now.
Thanks Mark Sent from my iPhone On Aug 1, 2022, at 12:46 PM, BeNJ <[email protected]> wrote: Updates: Mark: Deleting the json allows nifi to start up(!!) - note that restarting (with the newly generated json in place) causes the same issue on startup. Joe: I'm happy to run custom debug nars to help figure out how to resolve this. Thank you, I really appreciate everyone's help with this! Benji On Mon, Aug 1, 2022 at 8:00 AM Joe Witt <[email protected]<mailto:[email protected]>> wrote: Ben I didn't see any log levels that would help narrow this down. If Mark's suggestion does not address it it will likely require updating the flow configuration manually or a patched framework nar to do better logging. I'm also wondering if the json form of the flow could be imported on a clean nifi (you'd have to re-enter sensitive values) or imported to a registry then imported to nifi. Given you have hit this and someone else did (on stackoverflow) we clearly have an important case to figure out here as it is obviously disruptive. Thanks On Mon, Aug 1, 2022 at 5:41 AM Mark Payne <[email protected]<mailto:[email protected]>> wrote: Benji, I would recommend you try to remove (or rename) the flow.json.gz - but not the flow.xml.gz. See if that makes any difference. Thanks -Mark Sent from my iPhone On Jul 31, 2022, at 11:35 PM, BeNJ <[email protected]<mailto:[email protected]>> wrote: Also please see the attached nifi.properties Thanks, Benji On Sun, Jul 31, 2022 at 4:28 PM BeNJ <[email protected]<mailto:[email protected]>> wrote: Hi Joe, Stack with a couple of info logs from before and after, and the final exit shortly after. ------------------------------------------------------------------------------------------ 2022-07-31 16:20:35,311 INFO [main] o.a.n.g.StandardProcessGroupSynchronizer Added Connection[ID=cfee198f-3d2b-1513-f741-e71ad122aaaa, Source ID=cfee198e-3d2b-1513-7a9c-f0c2a8cf0d43, Dest ID=cfee19cf-3d2b-1513-4a87-5f50a90fdabf] to StandardProcessGroup[identifier=cfee1961-3d2b-1513-c8a6-fdf1a8fe4ff5,name=Add Customer User] 2022-07-31 16:20:35,311 INFO [main] o.a.n.g.StandardProcessGroupSynchronizer Added Connection[ID=cfee1974-3d2b-1513-e4df-9dbba1241682, Source ID=cfee1971-3d2b-1513-555c-1aedf0f0801f, Dest ID=cfee1970-3d2b-1513-de1d-f5bee9ad679e] to StandardProcessGroup[identifier=cfee1961-3d2b-1513-c8a6-fdf1a8fe4ff5,name=Add Customer User] 2022-07-31 16:20:35,317 INFO [Timer-Driven Process Thread-9] o.a.n.c.s.StandardControllerServiceNode Successfully enabled StandardControllerServiceNode[service=StandardOauth2AccessTokenProvider[id=cfee1b5b-3d2b-1513-7124-85b028901ac8], name=customer user management idp, active=true] 2022-07-31 16:20:35,325 INFO [main] o.a.n.c.s.AffectedComponentSet Starting the following components: AffectedComponentSet[inputPorts=[], outputPorts=[], remoteInputPorts=[], remoteOutputPorts=[], processors=[], controllerServices=[], reportingTasks=[]] 2022-07-31 16:20:35,328 WARN [main] org.eclipse.jetty.webapp.WebAppContext Failed startup of context o.e.j.w.WebAppContext@ffaaaf0{nifi-api,/nifi-api,file:///opt/nifi/nifi-current/work/jetty/nifi-web-api-1.16.0.war/webapp/,UNAVAILABLE}{./work/nar/extensions/nifi-server-nar-1.16.0.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-api-1.16.0.war} org.apache.nifi.controller.serialization.FlowSynchronizationException: java.lang.IllegalStateException: Cannot add Connection to Process Group because source is an Output Port that does not belong to a child Process Group at org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:362) at org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:185) at org.apache.nifi.controller.serialization.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:43) at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1479) at org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:104) at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:815) at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:538) at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:67) at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1073) at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572) at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:1002) at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:746) at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379) at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449) at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414) at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916) at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288) at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:426) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.server.Server.start(Server.java:423) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) at org.eclipse.jetty.server.Server.doStart(Server.java:387) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1008) at org.apache.nifi.NiFi.<init>(NiFi.java:170) at org.apache.nifi.NiFi.<init>(NiFi.java:82) at org.apache.nifi.NiFi.main(NiFi.java:330) Caused by: java.lang.IllegalStateException: Cannot add Connection to Process Group because source is an Output Port that does not belong to a child Process Group at org.apache.nifi.groups.StandardProcessGroup.addConnection(StandardProcessGroup.java:1154) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.addConnection(StandardProcessGroupSynchronizer.java:1853) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronizeConnections(StandardProcessGroupSynchronizer.java:531) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:388) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.addProcessGroup(StandardProcessGroupSynchronizer.java:1019) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronizeChildGroups(StandardProcessGroupSynchronizer.java:423) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:373) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.addProcessGroup(StandardProcessGroupSynchronizer.java:1019) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronizeChildGroups(StandardProcessGroupSynchronizer.java:423) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:373) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.addProcessGroup(StandardProcessGroupSynchronizer.java:1019) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronizeChildGroups(StandardProcessGroupSynchronizer.java:423) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:373) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.addProcessGroup(StandardProcessGroupSynchronizer.java:1019) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronizeChildGroups(StandardProcessGroupSynchronizer.java:423) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:373) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.addProcessGroup(StandardProcessGroupSynchronizer.java:1019) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronizeChildGroups(StandardProcessGroupSynchronizer.java:423) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:373) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.lambda$synchronize$0(StandardProcessGroupSynchronizer.java:221) at org.apache.nifi.controller.flow.AbstractFlowManager.withParameterContextResolution(AbstractFlowManager.java:462) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:219) at org.apache.nifi.groups.StandardProcessGroup.synchronizeFlow(StandardProcessGroup.java:3830) at org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:353) ... 45 common frames omitted 2022-07-31 16:20:35,552 INFO [Timer-Driven Process Thread-3] o.a.n.c.s.StandardControllerServiceNode Successfully enabled StandardControllerServiceNode[service=MongoDBControllerService[id=19573a5c-3f9e-3cdc-b80c-1990fdd33c69], name=MongoDB CR - Main, active=true] ... ... 2022-07-31 16:20:35,733 INFO [main] org.eclipse.jetty.server.Server Started @26964ms 2022-07-31 16:20:35,734 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down. org.apache.nifi.controller.serialization.FlowSynchronizationException: java.lang.IllegalStateException: Cannot add Connection to Process Group because source is an Output Port that does not belong to a child Process Group at org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:362) at org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:185) at org.apache.nifi.controller.serialization.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:43) at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1479) at org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:104) at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:815) at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:538) at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:67) at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1073) at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572) at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:1002) at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:746) at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379) at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449) at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414) at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916) at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288) at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:426) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.server.Server.start(Server.java:423) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) at org.eclipse.jetty.server.Server.doStart(Server.java:387) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1008) at org.apache.nifi.NiFi.<init>(NiFi.java:170) at org.apache.nifi.NiFi.<init>(NiFi.java:82) at org.apache.nifi.NiFi.main(NiFi.java:330) Caused by: java.lang.IllegalStateException: Cannot add Connection to Process Group because source is an Output Port that does not belong to a child Process Group at org.apache.nifi.groups.StandardProcessGroup.addConnection(StandardProcessGroup.java:1154) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.addConnection(StandardProcessGroupSynchronizer.java:1853) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronizeConnections(StandardProcessGroupSynchronizer.java:531) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:388) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.addProcessGroup(StandardProcessGroupSynchronizer.java:1019) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronizeChildGroups(StandardProcessGroupSynchronizer.java:423) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:373) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.addProcessGroup(StandardProcessGroupSynchronizer.java:1019) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronizeChildGroups(StandardProcessGroupSynchronizer.java:423) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:373) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.addProcessGroup(StandardProcessGroupSynchronizer.java:1019) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronizeChildGroups(StandardProcessGroupSynchronizer.java:423) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:373) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.addProcessGroup(StandardProcessGroupSynchronizer.java:1019) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronizeChildGroups(StandardProcessGroupSynchronizer.java:423) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:373) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.addProcessGroup(StandardProcessGroupSynchronizer.java:1019) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronizeChildGroups(StandardProcessGroupSynchronizer.java:423) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:373) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.lambda$synchronize$0(StandardProcessGroupSynchronizer.java:221) at org.apache.nifi.controller.flow.AbstractFlowManager.withParameterContextResolution(AbstractFlowManager.java:462) at org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:219) at org.apache.nifi.groups.StandardProcessGroup.synchronizeFlow(StandardProcessGroup.java:3830) at org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:353) ... 45 common frames omitted 2022-07-31 16:20:35,738 INFO [Thread-1] org.apache.nifi.NiFi Application Server shutdown started ------------------------------------------------------------------------------------------ David: df -h for this fs: /dev/mapper/ubuntu--vg-ubuntu--lv 97G 41G 52G 44% / I'll be able to share the nifi.properties in a few hours when I'm back, please let me know if there's anything else I can share or enable. Thanks, Benji On Sun, Jul 31, 2022 at 3:37 PM Joe Witt <[email protected]<mailto:[email protected]>> wrote: Benji What would be helpful to see is the full stacktrace like shown in this stackoverflow post https://stackoverflow.com/questions/72459434/apache-nifi-1-16-0-windows-server-closes-immediately This class changes often so I want to make sure we check the right one for your version. Thanks On Sun, Jul 31, 2022 at 1:25 PM David Early via users <[email protected]<mailto:[email protected]>> wrote: Benji, Can you share your nifi.properties file? This problem can show up if the disk fills up but depends on the configuration. Also, what does df -h show for your disks (assuming you are running on Linux)? Dave On Sun, Jul 31, 2022, 2:21 PM Benji Benning <[email protected]<mailto:[email protected]>> wrote: Not really around the below error. Anything I can move to debug or trace in logback.xml to add some color around this? Thanks, Benji Sent from my iPhone On Jul 31, 2022, at 15:12, Joe Witt <[email protected]<mailto:[email protected]>> wrote: Benji Is there more with the logs like component uuids and such? Thanks On Sun, Jul 31, 2022 at 1:08 PM Benji Benning <[email protected]<mailto:[email protected]>> wrote: Hi Joe, Unfortunately it does indeed contain sensitive info. Is there any debug info I can provide? Thanks, Benji Sent from my iPhone On Jul 31, 2022, at 12:25, Joe Witt <[email protected]<mailto:[email protected]>> wrote: Can you share the flow or does it contain sensitve info? On Sun, Jul 31, 2022 at 10:01 AM BeNJ <[email protected]<mailto:[email protected]>> wrote: Hi, I'm a little at a loss so I figured maybe someone here would have an idea of how to get out of this situation. My NiFI is crashing on startup with the following: -------------------------------------- 2022-07-31 03:10:05,448 INFO [main] o.a.n.c.s.AffectedComponentSet Starting the following components: AffectedComponentSet[inputPorts=[], outputPorts=[], remoteInputPorts=[], remoteOutputPorts=[], processors=[], controllerServices=[], reportingTasks=[]] 2022-07-31 03:10:05,452 WARN [main] org.eclipse.jetty.webapp.WebAppContext Failed startup of context o.e.j.w.WebAppContext@3ed03652{nifi-api,/nifi-api,file:///opt/nifi/nifi-current/work/jetty/nifi-web-api-1.16.0.war/webapp/,UNAVAILABLE}{./work/nar/extensions/nifi-server-nar-1.16.0.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-api-1.16.0.war} org.apache.nifi.controller.serialization.FlowSynchronizationException: java.lang.IllegalStateException: Cannot add Connection to Process Group because source is an Output Port that does not belong to a child Process Group -------------------------------------- I only restarted because out of nowhere i started seeing the following errors repeating: Processors cannot write to Content Repo: Unable to write flowfile content to content repository container default due to archive file size constraints; waiting for archive cleanup To resolve this issue, someone said online to set nifi.content.repository.archive.enabled to false. I restarted and nifi wouldn't come up again. I've tried reverting the flow.xml.gz to the oldest one in the archive folder (was from 7pm yesterday), no luck. it's a large config (1000+ active processors). I would try to disable / delete the offending output port, but I don't see how to find it from the logs. Any direction would be greatly appreciated. Thanks, Benji
