Kind of. WriteAheadFlowFileRepository is for FlowFile repository, it only stores attributes and flow file states [1]. For flow files content, you would need to extend FileSystemRepository. And if you need to also encrypt provenance data, that's an additional repository to extend.
Just a quick remark: I didn't say it in my previous mail, but a short term solution for you could be to use the volatile repositories to keep everything in memory instead of disks, but be aware of the limitations (in case of a NiFi restart for example). [1] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#flowfile-repository 2016-12-23 17:26 GMT+01:00 Knapp, Michael <[email protected]>: > Pierre, > > OK so let me see if I am interpreting your advice correctly: > 1. Extend WriteAheadFlowFileRepository. > 2. Override methods as necessary. > 3. Make it encrypt everything that goes to disk. I could have it use > AWS’s KMS service to do that. I might even have custom encryption logic > here. > > Is that right? > > On 12/23/16, 10:23 AM, "Pierre Villard" <[email protected]> > wrote: > > Hi Michael, > > The feature you describe is on the roadmap and you may find more > details > here [1] and/or participate into the discussion. Regarding the > implementation you suggest, I think that implementing a custom > repository > instead of a controller service would be an easier approach (no need to > change the processors). In any case, this would be a great feature and > it > is possible that Andy already started some developments on this > subject. > > [1] > https://cwiki.apache.org/confluence/display/NIFI/ > Security+Feature+Roadmap > > -Pierre > > > 2016-12-23 15:43 GMT+01:00 Knapp, Michael < > [email protected]>: > > > Nifi Developers, > > > > So I have a somewhat interesting task. I want to run Nifi on AWS, > but at > > the same time there is a lot of red tape involved with putting data > on > > AWS. Some data may not be placed on an AWS disk unless it is > encrypted. > > Running Nifi on top of an encrypted EBS is not considered good > enough in my > > case. The ListenHTTP processor does not let people encrypt content > before > > it is written to disk. You can encrypt the content downstream, but > at that > > point it has already been written to disk so that is too late. > People can > > encrypt content before it is sent to Nifi, but in some situations > that may > > be very challenging for the developer, as they may have limited > access to > > the source of the data. > > > > I was thinking of modifying the ListenHTTP processor and other > similar > > processors. I want to create a ControllerService interface that > merely > > returns a StreamCallback implementation. The ListenHTTP processor > would > > take this as an optional property. If that property is set, then the > > processor will use that to modify/encrypt content before it is even > written > > to disk. If the property is not set, then it will operate the same > way it > > does now. > > > > I looked for a good project to place this controller service > interface in, > > I feel like this service is so basic that it should really be part > of the > > framework’s core, but I don’t see any other controller services > there. So > > my best guess for now is to put this in the > nifi-ssl-context-service-(api|bundle) > > projects. I feel like this is not really related to SSL, but that > is the > > only project I found that has controller services listed and is a > > dependency of both the nifi-kafka projects and the > nifi-standard-processors > > projects. I think it would be a waste to set up a new api/bundle > pair just > > for one interface. > > > > So my questions are: > > > > 1. Do you think this is a good idea? > > > > 2. Where should I put this code if I write it? > > > > Michael Knapp > > Capital One > > ________________________________________________________ > > > > The information contained in this e-mail is confidential and/or > > proprietary to Capital One and/or its affiliates and may only be used > > solely in performance of work or services for Capital One. The > information > > transmitted herewith is intended only for use by the individual or > entity > > to which it is addressed. If the reader of this message is not the > intended > > recipient, you are hereby notified that any review, retransmission, > > dissemination, distribution, copying or other use of, or taking of > any > > action in reliance upon this information is strictly prohibited. If > you > > have received this communication in error, please contact the sender > and > > delete the material from your computer. > > > > > ________________________________________________________ > > The information contained in this e-mail is confidential and/or > proprietary to Capital One and/or its affiliates and may only be used > solely in performance of work or services for Capital One. The information > transmitted herewith is intended only for use by the individual or entity > to which it is addressed. If the reader of this message is not the intended > recipient, you are hereby notified that any review, retransmission, > dissemination, distribution, copying or other use of, or taking of any > action in reliance upon this information is strictly prohibited. If you > have received this communication in error, please contact the sender and > delete the material from your computer. >
