No matter how secure I make the cluster, what's to prevent a user from: * Using GetFile to copy the NiFi configuration files and keystores into flows. * Creating flows which contain python scripts that utilize the keys/secrets in the nifi configuration files to decrypt the contents of the keystores. * Executing those flows to gain access to the private keys used to encrypt data in NiFi * Using that data to gain unauthorized access to the cluster to further compromise it, possibly by generating their own admin-level user certificates.
I'm genuinely stumped here as to how to create a solution that is production ready and that prevents a complete system failure by a user accidentally supplying the wrong directory to a processor component or generally creating python scripts that execute as the nifi user, being able to basically use a NiFi server as their own script-running playground. Tom From: Mike Sofen <[email protected]> Sent: Friday, June 4, 2021 2:02 PM To: [email protected] Subject: RE: Problem with the GetFile processor deleting my entire installation Setting aside the user auth/permissions issue (which sounds like, from your initial email, that you nailed perfectly with a secured cluster), I'll address this accidental deletion issue using what I consider a general design pattern for Nifi (or any process that write-touches files): * I don't allow physical file deletes (which happen with the default GetFiles settings) * I create an archive folder for files I've processed if I want/need to move them out of the processing folder * I always set the "leave files" flag * I use either the timestamp or filename tracker to eliminate repeated processing if I'm not moving the files * OR - I use Move File flag to the archive folder destination. * And I advise new users to always spec a small test folder to start, and to double-check that the remove files is turned off. In some cases, like when I'm processing text files in place in secure repo, I can't move the files so I use the timestamp eval method. In others, like log files where I definitely want to archive them out of that processing folder, I use the move model. Don't get discouraged about this twist in the road - you'll find Nifi to be a truly exceptional product for pipeline automation, wildly more powerful/flexible/stable than any other product out there (I've used most). I've used it IoT to DB to ML processing, document processing, metrology data processing, you name it. In particular, the built data provenance and auditability will be valuable for your situation at Optum. All the best, Mike From: Ruth, Thomas <[email protected]<mailto:[email protected]>> Sent: Friday, June 04, 2021 12:26 PM To: [email protected]<mailto:[email protected]> Subject: RE: Problem with the GetFile processor deleting my entire installation Is it recommended that after installing NiFi, that I then proceed to remove read permissions from all installation files and directories in order to protect them from removal by users? Will this present a problem running the software if it's unable to read any files? So for fun, I loaded up a quick nifi container: docker run -name nifi -p 8080:8080 -d apache/nifi:latest Connected to localhost:8080/nifi Created a GetFile processor, with the directory /opt/nifi. Connected the success to a LogAttributes. Hit Run.... Now I get this: HTTP ERROR 404 /nifi/canvas.jsp URI: /nifi/ STATUS: 404 MESSAGE: /nifi/canvas.jsp SERVLET: jsp This behavior shouldn't be possible, in my opinion. It's putting security in the hands of my developers. I really am looking for a solution to this issue. The help text says this: Creates FlowFiles from files in a directory. NiFi will ignore files it doesn't have at least read permissions for. So as you suggested, I removed the read permission recursively from all files in /opt/nifi. After doing this, nifi no longer starts. find /opt/nifi -print > /tmp/files.txt for I in `cat /tmp/files.txt`; do chmod a-r $i; done I also went to https://nvd.nist.gov/vuln-metrics/cvss/v3-calculator and attempted to calculate a CVSS score for this vulnerability. I ended up calculating a score of 8.0. Tom From: Russell Bateman <[email protected]<mailto:[email protected]>> Sent: Friday, June 4, 2021 11:20 AM To: [email protected]<mailto:[email protected]> Subject: Re: Problem with the GetFile processor deleting my entire installation Oh, sorry, to finish the answer, yes, you do need to be very careful how you specify the Input Directory and File Filter properties; the last one is a regular expression. It's true that the documentation is less than flag-waving or hair-lighting-on-fire as it presents its help on filling those in. Russ On 6/4/21 11:16 AM, Russell Bateman wrote: Sorry for this behavior of GetFile which is purposeful. If you configure to keep the files instead of removing them, you'll keep getting the same files ingested over and over again as flow files. It's just how it is. The secret was to read the help blurb when configuring this processor. Hope this helps, Russ On 6/4/21 10:44 AM, Ruth, Thomas wrote: Hello, I recently built a 3-node NiFi cluster in my organization as a proof-of-concept for some work we are doing. I used version 1.13.2 and installed it onto 3 CentOS 7.9 systems. In my organization, I don't have root access to the system, so I used a different user called "nfadm" to install and run the product. I don't remember seeing anything in the documentation that stated that this would be an issue. I am also new to NiFi, and was relying heavily on the Admin documentation on the web site for instructions to set up the OS and NiFi installations. I configured certificate-based security and distributed them to my users. I also configured policies for groups that I thought were OK for them from a development standpoint. I had an incident occur yesterday in which a user, who is also new to NiFi, ran a component called "GetFile" for the filesystem "/" with the default settings (Recurse=true, KeepSourceFile=false). Well, this essentially ran "rm -rf /" as the user that owns all the installation files and files in the various repositories, nfadm, the same user running the NiFi processes. This deleted all the installation and configuration files for the entire cluster, making it completely useless now. I am surprised to find out that NiFi allowed a user to basically wipe out all the files the user running the NiFi server had access to. I would expect much higher security to be present in a default system. I have some questions that hopefully you can help me with: Is this a known issue in NiFi? Am I doing something wrong when configuring or installing NiFi? Is there a section in the documentation that warns me of this type of scenario? Thanks in advance for your help with this, Tom Ruth Sr. Data Engineer Optum, Inc. E: [email protected]<mailto:[email protected]> This e-mail, including attachments, may include confidential and/or proprietary information, and may be used only by the person or entity to which it is addressed. If the reader of this e-mail is not the intended recipient or his or her authorized agent, the reader is hereby notified that any dissemination, distribution or copying of this e-mail is prohibited. If you have received this e-mail in error, please notify the sender by replying to this message and delete this e-mail immediately. This e-mail, including attachments, may include confidential and/or proprietary information, and may be used only by the person or entity to which it is addressed. If the reader of this e-mail is not the intended recipient or his or her authorized agent, the reader is hereby notified that any dissemination, distribution or copying of this e-mail is prohibited. If you have received this e-mail in error, please notify the sender by replying to this message and delete this e-mail immediately. This e-mail, including attachments, may include confidential and/or proprietary information, and may be used only by the person or entity to which it is addressed. If the reader of this e-mail is not the intended recipient or his or her authorized agent, the reader is hereby notified that any dissemination, distribution or copying of this e-mail is prohibited. If you have received this e-mail in error, please notify the sender by replying to this message and delete this e-mail immediately.
