I used the command docker exec -it nifi /bin/bash to review what I understand to be the nifi directories and config files in the container.
I notice nifi in the container is in many ways not configured to Apache NiFi recommendations for optimal performance. For example, the nifi.properties repo params all refer to repos placed on one common disk device (the one where the container lives, presumably). I've configured external ebs volumes that I've mounted on my instance. One for content_repository, one for flowfile_repository, and likewise for database and provenance repositories. I'd like to have the containerized nifi write to and read from those so that I don't bottleneck performance reading and writing to the same device for repos. I need to persist my changes to nifi config files. How does one avoid making changes in nifi.properties and the like that are lost when the docker container is stopped, deleted, and a new one instantiated? I need to leverage external resources when nifi runs within my container. How do we direct nifi in the container to use those external resources outside of the container to host content_repository, etc etc? Thank you in advance for any help. Jim On Tue, Nov 8, 2022 at 10:28 PM David Handermann < [email protected]> wrote: > Jim, > > You're welcome! Thanks for following up and confirming the solution, great > collaborative effort! > > Regard, > David Handermann > > > > > On Tue, Nov 8, 2022, 7:25 PM James McMahon <[email protected]> wrote: > >> That was it. Adding the port to the docker run command proxy got me to >> the promised land. I was then able to use the userid and password from the >> docker log to access nifi on my ec2 instance. >> >> David, Dmitry - thank you so much. This was a huge help to me, and I hope >> it will help others trying the same approach in the future. >> Jim >> >> On Tue, Nov 8, 2022 at 8:13 PM David Handermann < >> [email protected]> wrote: >> >>> It may also be necessary to include the port in the host variable: >>> >>> docker run --name nifi -p 8443:8443 -e NIFI_WEB_PROXY_HOST= >>> ec2-3-238-27-220.compute-1.amazonaws.com:8443 -d apache/nifi:latest >>> >>> It is possible to access the configuration and logs files using an >>> interactive shell with the following Docker command: >>> >>> docker exec -it nifi /bin/bash >>> >>> Regards, >>> David Handermann >>> >>> On Tue, Nov 8, 2022 at 7:09 PM Dmitry Stepanov <[email protected]> >>> wrote: >>> >>>> Make sure you use your full domain name >>>> ec2-3-238-27-220.compute-1.amazonaws.com >>>> David shorten it in his code >>>> >>>> On November 8, 2022 5:57:26 p.m. James McMahon <[email protected]> >>>> wrote: >>>> >>>>> Thank you, David. I’ve made that change, adding the proxy host >>>>> specification on the docker command line. I continue to get the same error >>>>> message. Is it possible I need to indicate my key on the docker command >>>>> line too? >>>>> >>>>> Related, how can one access nifi.properties and the usual nifi config >>>>> files, as well as the family of nifi-app.log files and bootstrap.conf, >>>>> when >>>>> nifi is running inside a docker container? >>>>> >>>>> Thanks again for sticking with this. I feel like we’re getting closer. >>>>> Jim >>>>> >>>>> On Tue, Nov 8, 2022 at 7:31 PM David Handermann < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi Jim, >>>>>> >>>>>> Good adjustment on the security group inbound rules. >>>>>> >>>>>> The error page is the result of NiFi receiving an unexpected HTTP >>>>>> Host header, not matching one of the expected values. >>>>>> >>>>>> For this to work, it is possible to pass the external DNS name as the >>>>>> value of the NIFI_WEB_PROXY_HOST environment variable. This can be >>>>>> specified in the docker run command as follows: >>>>>> >>>>>> docker run --name nifi -p 8443:8443 -e NIFI_WEB_PROXY_HOST=ec2... >>>>>> amazonaws.com -d apache/nifi:latest >>>>>> >>>>>> That will allow NiFi to accept the Host header from the browser, and >>>>>> then present the login screen. >>>>>> >>>>>> Regards, >>>>>> David Handermann >>>>>> >>>>>> On Tue, Nov 8, 2022 at 6:06 PM James McMahon <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi David. This is very helpful, thank you. I feel like I am close, >>>>>>> but I get an error. My Inbound Rules for my security group now include: >>>>>>> 8443 TCP (MyIP)/32 >>>>>>> 443 TCP (MyIP)/32 >>>>>>> 22 TCP (MyIP)/32 >>>>>>> >>>>>>> In my browser - I tried both Edge and Chrome - I use this >>>>>>> URL: >>>>>>> https://ec2-3-238-27-230.compute-1.amazonaws.com:8443 >>>>>>> I have also tried with /nifi at the tail end. >>>>>>> >>>>>>> I get this error: >>>>>>> >>>>>>> *System Error* >>>>>>> >>>>>>> *The request contained an invalid host header >>>>>>> [ec2-3-238-27-220.compute-1.amazonaws.com:8443 >>>>>>> <http://ec2-3-238-27-220.compute-1.amazonaws.com:8443/>] in the request >>>>>>> [/]. Check for request manipulation or third-party intercept.* >>>>>>> >>>>>>> *Valid host headers are [empty] or:* >>>>>>> >>>>>>> - *127.0.0.1* >>>>>>> - *127.0.0.1:8443 <http://127.0.0.1:8443/>* >>>>>>> - *localhost* >>>>>>> - *localhost:8443* >>>>>>> - *[::1]* >>>>>>> - *[::1]:8443* >>>>>>> - *7f661ae687d7* >>>>>>> - *7f661ae687d7:8443* >>>>>>> - *172.17.0.2* >>>>>>> - *172.17.0.2:8443 <http://172.17.0.2:8443/>* >>>>>>> >>>>>>> >>>>>>> Does this mean I have formed the URL incorrectly? >>>>>>> >>>>>>> I also see that I had to add an exception to permit https. When I >>>>>>> created the instance, I created my own pem key pair. It is not signed by >>>>>>> any CA. For a self-signed key pair like this, do I need to install a >>>>>>> key in >>>>>>> my browser security store to avoid adding that exception? >>>>>>> >>>>>>> Thank you for helping me get that much closer. >>>>>>> Jim >>>>>>> >>>>>>> On Tue, Nov 8, 2022 at 5:13 PM David Handermann < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi Jim, >>>>>>>> >>>>>>>> Thanks for the reply and additional background. >>>>>>>> >>>>>>>> The instructions are dated March 2021, which is prior to the >>>>>>>> release of NiFi 1.14.0. In particular, the run command is no longer >>>>>>>> accurate with the default NiFi container image. >>>>>>>> >>>>>>>> The current Docker Hub instructions [1] show the basic command >>>>>>>> needed >>>>>>>> >>>>>>>> docker run --name nifi -p 8443:8443 -d apache/nifi:latest >>>>>>>> >>>>>>>> In addition, any references to port 8080 in the AWS Security Group >>>>>>>> rules should be changed to 8443. The security group rules for port 80 >>>>>>>> and >>>>>>>> 18080 should be removed. >>>>>>>> >>>>>>>> The instructions that allow plain HTTP access to NiFi on port 8080 >>>>>>>> should NEVER be followed, as this exposes unfiltered and >>>>>>>> unauthenticated >>>>>>>> access. >>>>>>>> >>>>>>>> Following those changes, it should be possible to access the NiFi >>>>>>>> UI using the AWS URL: >>>>>>>> >>>>>>>> https://ec2...amazonaws.com:8443 >>>>>>>> >>>>>>>> The default installation will generate a username and password, >>>>>>>> which can be found in the container logs: >>>>>>>> >>>>>>>> docker logs nifi | grep Generated >>>>>>>> >>>>>>>> Regards, >>>>>>>> David Handermann >>>>>>>> >>>>>>>> [1] https://hub.docker.com/r/apache/nifi >>>>>>>> >>>>>>>> On Tue, Nov 8, 2022 at 4:00 PM James McMahon <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi and thank you, David and Dmitry. In my case I was following >>>>>>>>> this example, >>>>>>>>> >>>>>>>>> https://joeygoksu.com/software/apache-nifi-on-aws/ >>>>>>>>> >>>>>>>>> which results in NiFi installed within a container. So to answer >>>>>>>>> one of your questions, I don’t yet know how or where to find >>>>>>>>> nifi.properties in the container framework. I don’t seem to have the >>>>>>>>> usual >>>>>>>>> /opt/nifi/….. directories on my ec2 instance. Any idea where I need >>>>>>>>> to look >>>>>>>>> for that? >>>>>>>>> >>>>>>>>> These ports are open by my security group Inbound Rules: 22 to >>>>>>>>> MyIP, 80, 8080, and 18080 (per the link) to 0.0.0.0/0, 443 to >>>>>>>>> MyIP. >>>>>>>>> >>>>>>>>> I am able to Putty into my instance as ec2-user with my ppk file, >>>>>>>>> which I created using putty tools from the original pem key pair. >>>>>>>>> When I do >>>>>>>>> putty in, under /opt I find three subdirectories: aws, containerd, >>>>>>>>> and rh. >>>>>>>>> Nothing nifi under any of the three that I can see so far. >>>>>>>>> >>>>>>>>> I start my docker instance with this command: >>>>>>>>> docker run —name nifi -p 18080:8080 -d apache/nifi:latest >>>>>>>>> >>>>>>>>> I can do a ps -ef and see running nifi processes. But I don’t yet >>>>>>>>> know how to get to the nifi logs or properties file. >>>>>>>>> >>>>>>>>> You mentioned using using localhost to get to the canvas UI. This >>>>>>>>> confuses me. Nifi is running on my EC2 instance - a linux host >>>>>>>>> without a >>>>>>>>> browser. I’m in a browser on my laptop. How would localhost in my >>>>>>>>> browser >>>>>>>>> get me to my EC2 instance running nifi? >>>>>>>>> >>>>>>>>> This is the URL I’m using in my browser: >>>>>>>>> http://ec2-3-238-27-220.compute-1.amazonaws.com >>>>>>>>> (that url changes with each Stop/Start of my instance. I’ve yet to >>>>>>>>> investigate how to get AWS to stop changing that IP, but I know it >>>>>>>>> can be >>>>>>>>> done). >>>>>>>>> >>>>>>>>> The browser replies with: ec2…….amazonaws refused to connect. >>>>>>>>> >>>>>>>>> I can ping my laptop IP address from the putty terminal where I am >>>>>>>>> logged in to my instance. I cannot ping the Public DNS of my instance >>>>>>>>> from >>>>>>>>> Powershell on my laptop. Again, that Public DNS is >>>>>>>>> ec2-3-238-27-220.compute-1.amazonaws.com >>>>>>>>> >>>>>>>>> Any help is much appreciated. >>>>>>>>> Jim >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Nov 8, 2022 at 3:03 PM David Handermann < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi Jim, >>>>>>>>>> >>>>>>>>>> NiFi 1.14.0 and following default to HTTPS on port 8443, >>>>>>>>>> listening on the localhost address. The nifi.web.https.host can be >>>>>>>>>> changed >>>>>>>>>> to blank in order to listen on all interfaces, but the default HTTPS >>>>>>>>>> setting with authenticated required should be retained. >>>>>>>>>> >>>>>>>>>> Can you provide the version of NiFi and some additional details >>>>>>>>>> on the nifi.web values from nifi.properties? >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> David Handermann >>>>>>>>>> >>>>>>>>>> On Tue, Nov 8, 2022 at 1:54 PM James McMahon < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Has anyone successfully configured NiFi on AWS, and accessed it >>>>>>>>>>> from a browser on a Windows desktop? I’ve tried following a few >>>>>>>>>>> links to do >>>>>>>>>>> this. I’ve verified that my instance security group allows access >>>>>>>>>>> to 8080 >>>>>>>>>>> via its inbound rules. I’ve putty’ed into the instance via ssh port >>>>>>>>>>> 22 to >>>>>>>>>>> verify that there are no firewall restrictions. But still I get a >>>>>>>>>>> message >>>>>>>>>>> to the effect that the server rejected the connection request. Can >>>>>>>>>>> anyone >>>>>>>>>>> recommend a link that describes a success path for this? >>>>>>>>>>> Thanks in advance for your help. >>>>>>>>>>> Jim >>>>>>>>>>> >>>>>>>>>> >>>>
