Hi Joe,
Stack with a couple of info logs from before and after, and the final exit
shortly after.
------------------------------------------------------------------------------------------
2022-07-31 16:20:35,311 INFO [main]
o.a.n.g.StandardProcessGroupSynchronizer Added
Connection[ID=cfee198f-3d2b-1513-f741-e71ad122aaaa, Source
ID=cfee198e-3d2b-1513-7a9c-f0c2a8cf0d43, Dest
ID=cfee19cf-3d2b-1513-4a87-5f50a90fdabf] to
StandardProcessGroup[identifier=cfee1961-3d2b-1513-c8a6-fdf1a8fe4ff5,name=Add
Customer User]
2022-07-31 16:20:35,311 INFO [main]
o.a.n.g.StandardProcessGroupSynchronizer Added
Connection[ID=cfee1974-3d2b-1513-e4df-9dbba1241682, Source
ID=cfee1971-3d2b-1513-555c-1aedf0f0801f, Dest
ID=cfee1970-3d2b-1513-de1d-f5bee9ad679e] to
StandardProcessGroup[identifier=cfee1961-3d2b-1513-c8a6-fdf1a8fe4ff5,name=Add
Customer User]
2022-07-31 16:20:35,317 INFO [Timer-Driven Process Thread-9]
o.a.n.c.s.StandardControllerServiceNode Successfully enabled
StandardControllerServiceNode[service=StandardOauth2AccessTokenProvider[id=cfee1b5b-3d2b-1513-7124-85b028901ac8],
name=customer user management idp, active=true]
2022-07-31 16:20:35,325 INFO [main] o.a.n.c.s.AffectedComponentSet Starting
the following components: AffectedComponentSet[inputPorts=[],
outputPorts=[], remoteInputPorts=[], remoteOutputPorts=[], processors=[],
controllerServices=[], reportingTasks=[]]
2022-07-31 16:20:35,328 WARN [main] org.eclipse.jetty.webapp.WebAppContext
Failed startup of context o.e.j.w.WebAppContext@ffaaaf0
{nifi-api,/nifi-api,file:///opt/nifi/nifi-current/work/jetty/nifi-web-api-1.16.0.war/webapp/,UNAVAILABLE}{./work/nar/extensions/nifi-server-nar-1.16.0.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-api-1.16.0.war}
org.apache.nifi.controller.serialization.FlowSynchronizationException:
java.lang.IllegalStateException: Cannot add Connection to Process Group
because source is an Output Port that does not belong to a child Process
Group
at
org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:362)
at
org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:185)
at
org.apache.nifi.controller.serialization.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:43)
at
org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1479)
at
org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:104)
at
org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:815)
at
org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:538)
at
org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:67)
at
org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1073)
at
org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
at
org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:1002)
at
org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:746)
at
org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
at
org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
at
org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
at
org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916)
at
org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at
org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:426)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.server.Server.start(Server.java:423)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.server.Server.doStart(Server.java:387)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1008)
at org.apache.nifi.NiFi.<init>(NiFi.java:170)
at org.apache.nifi.NiFi.<init>(NiFi.java:82)
at org.apache.nifi.NiFi.main(NiFi.java:330)
Caused by: java.lang.IllegalStateException: Cannot add Connection to
Process Group because source is an Output Port that does not belong to a
child Process Group
at
org.apache.nifi.groups.StandardProcessGroup.addConnection(StandardProcessGroup.java:1154)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.addConnection(StandardProcessGroupSynchronizer.java:1853)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronizeConnections(StandardProcessGroupSynchronizer.java:531)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:388)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.addProcessGroup(StandardProcessGroupSynchronizer.java:1019)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronizeChildGroups(StandardProcessGroupSynchronizer.java:423)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:373)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.addProcessGroup(StandardProcessGroupSynchronizer.java:1019)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronizeChildGroups(StandardProcessGroupSynchronizer.java:423)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:373)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.addProcessGroup(StandardProcessGroupSynchronizer.java:1019)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronizeChildGroups(StandardProcessGroupSynchronizer.java:423)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:373)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.addProcessGroup(StandardProcessGroupSynchronizer.java:1019)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronizeChildGroups(StandardProcessGroupSynchronizer.java:423)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:373)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.addProcessGroup(StandardProcessGroupSynchronizer.java:1019)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronizeChildGroups(StandardProcessGroupSynchronizer.java:423)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:373)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.lambda$synchronize$0(StandardProcessGroupSynchronizer.java:221)
at
org.apache.nifi.controller.flow.AbstractFlowManager.withParameterContextResolution(AbstractFlowManager.java:462)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:219)
at
org.apache.nifi.groups.StandardProcessGroup.synchronizeFlow(StandardProcessGroup.java:3830)
at
org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:353)
... 45 common frames omitted
2022-07-31 16:20:35,552 INFO [Timer-Driven Process Thread-3]
o.a.n.c.s.StandardControllerServiceNode Successfully enabled
StandardControllerServiceNode[service=MongoDBControllerService[id=19573a5c-3f9e-3cdc-b80c-1990fdd33c69],
name=MongoDB CR - Main, active=true]
...
...
2022-07-31 16:20:35,733 INFO [main] org.eclipse.jetty.server.Server Started
@26964ms
2022-07-31 16:20:35,734 WARN [main] org.apache.nifi.web.server.JettyServer
Failed to start web server... shutting down.
org.apache.nifi.controller.serialization.FlowSynchronizationException:
java.lang.IllegalStateException: Cannot add Connection to Process Group
because source is an Output Port that does not belong to a child Process
Group
at
org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:362)
at
org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:185)
at
org.apache.nifi.controller.serialization.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:43)
at
org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1479)
at
org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:104)
at
org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:815)
at
org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:538)
at
org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:67)
at
org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1073)
at
org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
at
org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:1002)
at
org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:746)
at
org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
at
org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
at
org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
at
org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916)
at
org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at
org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:426)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.server.Server.start(Server.java:423)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.server.Server.doStart(Server.java:387)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1008)
at org.apache.nifi.NiFi.<init>(NiFi.java:170)
at org.apache.nifi.NiFi.<init>(NiFi.java:82)
at org.apache.nifi.NiFi.main(NiFi.java:330)
Caused by: java.lang.IllegalStateException: Cannot add Connection to
Process Group because source is an Output Port that does not belong to a
child Process Group
at
org.apache.nifi.groups.StandardProcessGroup.addConnection(StandardProcessGroup.java:1154)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.addConnection(StandardProcessGroupSynchronizer.java:1853)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronizeConnections(StandardProcessGroupSynchronizer.java:531)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:388)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.addProcessGroup(StandardProcessGroupSynchronizer.java:1019)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronizeChildGroups(StandardProcessGroupSynchronizer.java:423)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:373)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.addProcessGroup(StandardProcessGroupSynchronizer.java:1019)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronizeChildGroups(StandardProcessGroupSynchronizer.java:423)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:373)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.addProcessGroup(StandardProcessGroupSynchronizer.java:1019)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronizeChildGroups(StandardProcessGroupSynchronizer.java:423)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:373)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.addProcessGroup(StandardProcessGroupSynchronizer.java:1019)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronizeChildGroups(StandardProcessGroupSynchronizer.java:423)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:373)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.addProcessGroup(StandardProcessGroupSynchronizer.java:1019)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronizeChildGroups(StandardProcessGroupSynchronizer.java:423)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:373)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.lambda$synchronize$0(StandardProcessGroupSynchronizer.java:221)
at
org.apache.nifi.controller.flow.AbstractFlowManager.withParameterContextResolution(AbstractFlowManager.java:462)
at
org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:219)
at
org.apache.nifi.groups.StandardProcessGroup.synchronizeFlow(StandardProcessGroup.java:3830)
at
org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:353)
... 45 common frames omitted
2022-07-31 16:20:35,738 INFO [Thread-1] org.apache.nifi.NiFi Application
Server shutdown started
------------------------------------------------------------------------------------------David: df -h for this fs: /dev/mapper/ubuntu--vg-ubuntu--lv 97G 41G 52G 44% / I'll be able to share the nifi.properties in a few hours when I'm back, please let me know if there's anything else I can share or enable. Thanks, Benji On Sun, Jul 31, 2022 at 3:37 PM Joe Witt <[email protected]> wrote: > Benji > > What would be helpful to see is the full stacktrace like shown in this > stackoverflow post > https://stackoverflow.com/questions/72459434/apache-nifi-1-16-0-windows-server-closes-immediately > > This class changes often so I want to make sure we check the right one for > your version. > > Thanks > > On Sun, Jul 31, 2022 at 1:25 PM David Early via users < > [email protected]> wrote: > >> Benji, >> >> Can you share your nifi.properties file? >> >> This problem can show up if the disk fills up but depends on the >> configuration. >> >> Also, what does df -h show for your disks (assuming you are running on >> Linux)? >> >> Dave >> >> >> On Sun, Jul 31, 2022, 2:21 PM Benji Benning <[email protected]> >> wrote: >> >>> Not really around the below error. >>> Anything I can move to debug or trace in logback.xml to add some color >>> around this? >>> >>> Thanks, >>> Benji >>> >>> Sent from my iPhone >>> >>> On Jul 31, 2022, at 15:12, Joe Witt <[email protected]> wrote: >>> >>> >>> Benji >>> >>> Is there more with the logs like component uuids and such? >>> >>> Thanks >>> >>> On Sun, Jul 31, 2022 at 1:08 PM Benji Benning <[email protected]> >>> wrote: >>> >>>> Hi Joe, >>>> Unfortunately it does indeed contain sensitive info. >>>> Is there any debug info I can provide? >>>> >>>> Thanks, >>>> Benji >>>> >>>> Sent from my iPhone >>>> >>>> On Jul 31, 2022, at 12:25, Joe Witt <[email protected]> wrote: >>>> >>>> >>>> >>>> Can you share the flow or does it contain sensitve info? >>>> >>>> On Sun, Jul 31, 2022 at 10:01 AM BeNJ <[email protected]> wrote: >>>> >>>>> Hi, >>>>> I'm a little at a loss so I figured maybe someone here would have an >>>>> idea of how to get out of this situation. >>>>> My NiFI is crashing on startup with the following: >>>>> -------------------------------------- >>>>> 2022-07-31 03:10:05,448 INFO [main] o.a.n.c.s.AffectedComponentSet >>>>> Starting the following components: AffectedComponentSet[inputPorts=[], >>>>> outputPorts=[], remoteInputPorts=[], remoteOutputPorts=[], processors=[], >>>>> controllerServices=[], reportingTasks=[]] >>>>> 2022-07-31 03:10:05,452 WARN [main] >>>>> org.eclipse.jetty.webapp.WebAppContext Failed startup of context >>>>> o.e.j.w.WebAppContext@3ed03652 >>>>> {nifi-api,/nifi-api,file:///opt/nifi/nifi-current/work/jetty/nifi-web-api-1.16.0.war/webapp/,UNAVAILABLE}{./work/nar/extensions/nifi-server-nar-1.16.0.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-api-1.16.0.war} >>>>> org.apache.nifi.controller.serialization.FlowSynchronizationException: >>>>> java.lang.IllegalStateException: Cannot add Connection to Process Group >>>>> because source is an Output Port that does not belong to a child Process >>>>> Group >>>>> -------------------------------------- >>>>> I only restarted because out of nowhere i started seeing the following >>>>> errors repeating: >>>>> Processors cannot write to Content Repo: Unable to write flowfile >>>>> content to content repository container default due to archive file size >>>>> constraints; waiting for archive cleanup >>>>> >>>>> To resolve this issue, someone said online to set >>>>> nifi.content.repository.archive.enabled to false. I restarted and nifi >>>>> wouldn't come up again. >>>>> >>>>> I've tried reverting the flow.xml.gz to the oldest one in the archive >>>>> folder (was from 7pm yesterday), no luck. >>>>> it's a large config (1000+ active processors). >>>>> I would try to disable / delete the offending output port, but I don't >>>>> see how to find it from the logs. >>>>> >>>>> Any direction would be greatly appreciated. >>>>> >>>>> Thanks, >>>>> Benji >>>>> >>>>
