I'm not an expert in this arena but there a couple of JIRAs that may be relevant which are targeted for the NiFi 1.3.0 release.
https://issues.apache.org/jira/browse/NIFI-3112 https://issues.apache.org/jira/browse/NIFI-2199 On Tue, May 30, 2017 at 10:17 PM, Jeremy Taylor <[email protected]> wrote: > James and all, > > If you’ll please read item #4 all the way through, you’ll see that I’m using > “nifi.sh stop” and more for our scenario. I actually use this technique and > the full Java command to kill nifi due to “nifi.sh stop” no longer working > for both of our nifi running scenarios as of 1.2.0. Prior to 1.2.0, > “nifi.sh stop” worked for all scenarios for us. > > > > Bootrap log: I’ve not paid attention to this before. But, as I’m killing > nifi at the moment w/ both “nifi.sh stop” and the full Java command to do > so, I don’t feel this an extremely helpful exercise at the moment. If we > had one technique or the other only, then great. Nevertheless, I do see > when nifi stops and starts from these logs. If you’ll read #4 all the way > through you’ll see some other details I pointed out. > > > > Regards, > > Jeremy > > > > > > From: James Wing <[email protected]> > Reply-To: "[email protected]" <[email protected]> > Date: Tuesday, May 30, 2017 at 8:14 PM > > > To: "[email protected]" <[email protected]> > Subject: Re: difficulty killing NiFi w/ nifi.sh stop under 1.2.0 now vs. > 1.1.2 when run as systemd service > > > > Jeremy, > > Have you tried "nifi.sh run" instead of start/stop? I'm not an expert on > supervisord, but it seems like it might be a better fit. Also, was anything > logged in the bootstrap log? > > Thanks, > > James > > > > On Tue, May 30, 2017 at 8:28 AM, Jeremy Taylor <[email protected]> > wrote: > > > > > > Andre and all, > > So you’ve mis-read this basically, but I should also provide an update. We > cannot use sudo rights when deploying in bitbucket pipelines. That is one > of the restrictions to consider. Let me go through a few more things again > and add to them this time. However, please remember that we are now w/ this > newer version of nifi having difficulty killing nifi (will explain details), > which is the big reason for a desire for help. > > > > 1) We are configuring bootstrap.conf to start up nifi as a non-root > user. (The user does happen to have sudo privileges, but we’re trying not > to use them.) This nifi.sh will start up in this way w/ the user configured > there. > > 2) If nifi was installed as an init.d service and bootstrap.conf had a > non-root user filled in the “run.as” area, then on reboot nifi would not > start up – BAD. This is the kicker of why I went the systemd service route > as we can have run-as configured, have the service start-up on reboot, and > have things work. W/ the init.d problem we however could start up the > service as that user, but it was a dealbreaker to not have the service > automatically come up when the VM was starting back up. > > 3) We use a nifi.service file to configure nifi as a systemd service. > This is an example below of how we are doing that via an example > nifi.service snippet. > > [Unit] > > Description=nifi > > After=network.target > > > > [Service] > > User=nonroot-user > > Group=nonroot-user > > WorkingDirectory=/tools/nifi-1.2.0/lib/bootstrap > > ExecStart=/usr/bin/java -cp > /tools/nifi-1.2.0/lib/bootstrap/activation-1.1.jar:/tools/nifi-1.2.0/lib/bootstrap/json-smart-2.1.1.jar:/tools/nifi-1.2.0/lib/bootstrap/antlr-runtime-3.5.2.jar:/tools/nifi-1.2.0/lib/bootstrap/logback-classic-1.2.3.jar:/tools/nifi-1.2.0/lib/bootstrap/asm-1.0.2.jar:/tools/nifi-1.2.0/lib/bootstrap/logback-core-1.2.3.jar:/tools/nifi-1.2.0/lib/bootstrap/asm-3.3.1.jar:/tools/nifi-1.2.0/lib/bootstrap/mail-1.4.7.jar:/tools/nifi-1.2.0/lib/bootstrap/bcpkix-jdk15on-1.55.jar:/tools/nifi-1.2.0/lib/bootstrap/nifi-api-1.2.0.jar:/tools/nifi-1.2.0/lib/bootstrap/bcprov-jdk15on-1.55.jar:/tools/nifi-1.2.0/lib/bootstrap/nifi-bootstrap-1.2.0.jar:/tools/nifi-1.2.0/lib/bootstrap/commons-codec-1.10.jar:/tools/nifi-1.2.0/lib/bootstrap/nifi-expression-language-1.2.0.jar:/tools/nifi-1.2.0/lib/bootstrap/commons-lang3-3.5.jar:/tools/nifi-1.2.0/lib/bootstrap/nifi-properties-loader-1.2.0.jar:/tools/nifi-1.2.0/lib/bootstrap/jackson-annotations-2.6.0.jar:/tools/nifi-1.2.0/lib/bootstrap/nifi-security-utils-1.2.0.jar:/tools/nifi-1.2.0/lib/bootstrap/jackson-core-2.6.1.jar:/tools/nifi-1.2.0/lib/bootstrap/nifi-utils-1.2.0.jar:/tools/nifi-1.2.0/lib/bootstrap/jackson-databind-2.6.1.jar:/tools/nifi-1.2.0/lib/bootstrap/okhttp-3.6.0.jar:/tools/nifi-1.2.0/lib/bootstrap/jna-4.4.0.jar:/tools/nifi-1.2.0/lib/bootstrap/okio-1.11.0.jar:/tools/nifi-1.2.0/lib/bootstrap/jna-platform-4.4.0.jar:/tools/nifi-1.2.0/lib/bootstrap/slf4j-api-1.7.25.jar:/tools/nifi-1.2.0/lib/bootstrap/json-path-2.0.0.jar > -Xms12m -Xmx24m > -Dorg.apache.nifi.bootstrap.config.file=/tools/nifi-1.2.0/conf/bootstrap.conf > org.apache.nifi.bootstrap.RunNiFi start > > Type=simple > > SuccessExitStatus=143 > > > > [Install] > > WantedBy=multi-user.target > > 4) In nifi versions prior to 1.2.0, we’ve been able to kill nifi over > bitbucket piplelines w/ the simple use of “nifi.sh stop” when nifi has > previously started either as a systemd service or a deployment has occurred > and it has started up via “nifi.sh start” on bitbucket pipelines. If an > auto-deployment has not occurred yet on a particular day, then nifi is still > running as a systemd service until the first deployment. > > a. As of 1.2.0, if nifi is still running as a systemd service, we seem > to be able to only kill nifi via explicit execution of > “org.apache.nifi.bootstrap.RunNiFi stop”. We’ll then observe a “PING” > occurring, which will help cause nifi to shutdown. > > b. As of 1.2.0, if nifi is still running as a systemd service, we no > longer seem to be able to kill its execution any longer with simple use of > “nifi.sh stop” as we have for all previous versions. We were surprised > about this and wanted to reach out for help for especially this reason and > concern. > > c. Once nifi is running, but not as a service, the use of “nifi.sh > stop” works, but using “org.apache.nifi.bootstrap.RunNiFi stop” does not. > This is not horrible, but this means that we have to use both ways of > killing nifi, the “org.apache.nifi.bootstrap.RunNiFi stop” technique and the > “nifi.sh stop” technique. And, this seems silly as you’d think both ways > would work the same or at least “nifi.sh stop” would always work as it > always did before, but they do not work equivalently in our case. And, > thus, this sloppy need for both is yet another reason for us to reach out > along w/ 4b. > > > > Further thoughts? Further thoughts on #4? > > > > Regards, > > Jeremy > > From: Andre <[email protected]> > Reply-To: "[email protected]" <[email protected]> > Date: Friday, May 26, 2017 at 8:48 PM > To: "[email protected]" <[email protected]> > Subject: Re: difficulty killing NiFi w/ nifi.sh stop under 1.2.0 now vs. > 1.1.2 when run as systemd service > > > > Jeremy, > > I am not familiar with Bitbucket pipelines but would you mind explaining why > are you wrapping the deployment with sudo? > > > > Reason I ask is because nifi.sh already has logic to drop the process > privileges from root to an uid of choice. > > > > Cheers > > > > On Sat, May 27, 2017 at 1:33 AM, Jeremy Taylor <[email protected]> > wrote: > > Greetings, > > I’m hoping to upgrade the my software team’s baseline to use NiFi 1.2.0. > I’m having trouble w/ a particular situation of trying to take nifi down. > Prior to this version, I’ve been able to use the nifi.sh script w/o sudo > rights to kill nifi when it has been brought up by a service (systemd, not > init.d). > > Main points: > > 1) We have had auto-deploy working via bitbucket pipelines to a staging > system that has auto deployed all the nifi required files from our baseline > (not nifi internal configs). > > 2) The AWS VM that runs nifi autmatically starts up nifi in a service state > via systemd services. We masquerade running it as a non-root user w/ sudo > rights, but do not use those sudo rights as scripts in bitbucket pipelines > won’t support sudo rights. > > 3) The deploy script for our staging server that runs nifi attempts to take > down nifi with a `nifi.sh stop`, which has worked prior to this version. > The newer nifi flow being deployed is manipulated via XSL and then copied in > where it needs to go before restarting nifi w/ `nifi.sh start` ; (we stop > using systemd upon the first deploy to the staging system) > > 4) I’ve done a diff between the latest nifi.sh and the previous nifi.sh from > 1.1.2 and only see a tiny difference. > > 5) Our staging server starts up every morning and brings up nifi and related > services via systemd services. > > 6) I realize using nifi as an init.d service is more supported than using it > as a systemd service. However, we’ve not been able to masquerade as a > different user properly very well when using init.d. Having to be root for > bitbucket pipelines would also be a dealbreaker for us. Thus, having the > service being run as a user other than root is important to us. > > 7) For reference, our systemd service file only deals with calling the > RunNiFi class and does not bother with the nifi shell script. > > > > Questions: > 1) Would anyone know why a simple “nifi.sh stop” would no longer kill a > pre-existing nifi process being run as a systemd service in nifi 1.2.0? > > 2) We are thinking of attempting a brute force kill that would kill the > “RunNiFi start” Java process. We are concerned that not exiting gracefully > would be really bad for nifi and related nifi repositories. Would this > route be recommended anyway in our circumstance? > 3) Any further recommendations? > > > > --Jeremy > > > >
