Great advice! I just might wind up using that. Is there any way to have some processors use one repository and other processors use the default repository? If not, then I may need to have two separate nifi instances running for that.
On 12/23/16, 11:42 AM, "Pierre Villard" <[email protected]> wrote: Kind of. WriteAheadFlowFileRepository is for FlowFile repository, it only stores attributes and flow file states [1]. For flow files content, you would need to extend FileSystemRepository. And if you need to also encrypt provenance data, that's an additional repository to extend. Just a quick remark: I didn't say it in my previous mail, but a short term solution for you could be to use the volatile repositories to keep everything in memory instead of disks, but be aware of the limitations (in case of a NiFi restart for example). [1] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#flowfile-repository 2016-12-23 17:26 GMT+01:00 Knapp, Michael <[email protected]>: > Pierre, > > OK so let me see if I am interpreting your advice correctly: > 1. Extend WriteAheadFlowFileRepository. > 2. Override methods as necessary. > 3. Make it encrypt everything that goes to disk. I could have it use > AWS’s KMS service to do that. I might even have custom encryption logic > here. > > Is that right? > > On 12/23/16, 10:23 AM, "Pierre Villard" <[email protected]> > wrote: > > Hi Michael, > > The feature you describe is on the roadmap and you may find more > details > here [1] and/or participate into the discussion. Regarding the > implementation you suggest, I think that implementing a custom > repository > instead of a controller service would be an easier approach (no need to > change the processors). In any case, this would be a great feature and > it > is possible that Andy already started some developments on this > subject. > > [1] > https://cwiki.apache.org/confluence/display/NIFI/ > Security+Feature+Roadmap > > -Pierre > > > 2016-12-23 15:43 GMT+01:00 Knapp, Michael < > [email protected]>: > > > Nifi Developers, > > > > So I have a somewhat interesting task. I want to run Nifi on AWS, > but at > > the same time there is a lot of red tape involved with putting data > on > > AWS. Some data may not be placed on an AWS disk unless it is > encrypted. > > Running Nifi on top of an encrypted EBS is not considered good > enough in my > > case. The ListenHTTP processor does not let people encrypt content > before > > it is written to disk. You can encrypt the content downstream, but > at that > > point it has already been written to disk so that is too late. > People can > > encrypt content before it is sent to Nifi, but in some situations > that may > > be very challenging for the developer, as they may have limited > access to > > the source of the data. > > > > I was thinking of modifying the ListenHTTP processor and other > similar > > processors. I want to create a ControllerService interface that > merely > > returns a StreamCallback implementation. The ListenHTTP processor > would > > take this as an optional property. If that property is set, then the > > processor will use that to modify/encrypt content before it is even > written > > to disk. If the property is not set, then it will operate the same > way it > > does now. > > > > I looked for a good project to place this controller service > interface in, > > I feel like this service is so basic that it should really be part > of the > > framework’s core, but I don’t see any other controller services > there. So > > my best guess for now is to put this in the > nifi-ssl-context-service-(api|bundle) > > projects. I feel like this is not really related to SSL, but that > is the > > only project I found that has controller services listed and is a > > dependency of both the nifi-kafka projects and the > nifi-standard-processors > > projects. I think it would be a waste to set up a new api/bundle > pair just > > for one interface. > > > > So my questions are: > > > > 1. Do you think this is a good idea? > > > > 2. Where should I put this code if I write it? > > > > Michael Knapp > > Capital One > > ________________________________________________________ > > > > The information contained in this e-mail is confidential and/or > > proprietary to Capital One and/or its affiliates and may only be used > > solely in performance of work or services for Capital One. The > information > > transmitted herewith is intended only for use by the individual or > entity > > to which it is addressed. If the reader of this message is not the > intended > > recipient, you are hereby notified that any review, retransmission, > > dissemination, distribution, copying or other use of, or taking of > any > > action in reliance upon this information is strictly prohibited. If > you > > have received this communication in error, please contact the sender > and > > delete the material from your computer. > > > > > ________________________________________________________ > > The information contained in this e-mail is confidential and/or > proprietary to Capital One and/or its affiliates and may only be used > solely in performance of work or services for Capital One. The information > transmitted herewith is intended only for use by the individual or entity > to which it is addressed. If the reader of this message is not the intended > recipient, you are hereby notified that any review, retransmission, > dissemination, distribution, copying or other use of, or taking of any > action in reliance upon this information is strictly prohibited. If you > have received this communication in error, please contact the sender and > delete the material from your computer. > ________________________________________________________ The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
