No this is not something you can do. Two separate NiFi instances sounds good to me with remote process groups if you need to move data from one to another.
Pierre 2016-12-23 17:46 GMT+01:00 Knapp, Michael <[email protected]>: > Great advice! I just might wind up using that. > > Is there any way to have some processors use one repository and other > processors use the default repository? If not, then I may need to have two > separate nifi instances running for that. > > On 12/23/16, 11:42 AM, "Pierre Villard" <[email protected]> > wrote: > > Kind of. > > WriteAheadFlowFileRepository is for FlowFile repository, it only stores > attributes and flow file states [1]. > For flow files content, you would need to extend FileSystemRepository. > And if you need to also encrypt provenance data, that's an additional > repository to extend. > > Just a quick remark: I didn't say it in my previous mail, but a short > term > solution for you could be to use the volatile repositories to keep > everything in memory instead of disks, but be aware of the limitations > (in > case of a NiFi restart for example). > > [1] > https://nifi.apache.org/docs/nifi-docs/html/administration- > guide.html#flowfile-repository > > 2016-12-23 17:26 GMT+01:00 Knapp, Michael < > [email protected]>: > > > Pierre, > > > > OK so let me see if I am interpreting your advice correctly: > > 1. Extend WriteAheadFlowFileRepository. > > 2. Override methods as necessary. > > 3. Make it encrypt everything that goes to disk. I could have it use > > AWS’s KMS service to do that. I might even have custom encryption > logic > > here. > > > > Is that right? > > > > On 12/23/16, 10:23 AM, "Pierre Villard" <[email protected] > > > > wrote: > > > > Hi Michael, > > > > The feature you describe is on the roadmap and you may find more > > details > > here [1] and/or participate into the discussion. Regarding the > > implementation you suggest, I think that implementing a custom > > repository > > instead of a controller service would be an easier approach (no > need to > > change the processors). In any case, this would be a great > feature and > > it > > is possible that Andy already started some developments on this > > subject. > > > > [1] > > https://cwiki.apache.org/confluence/display/NIFI/ > > Security+Feature+Roadmap > > > > -Pierre > > > > > > 2016-12-23 15:43 GMT+01:00 Knapp, Michael < > > [email protected]>: > > > > > Nifi Developers, > > > > > > So I have a somewhat interesting task. I want to run Nifi on > AWS, > > but at > > > the same time there is a lot of red tape involved with putting > data > > on > > > AWS. Some data may not be placed on an AWS disk unless it is > > encrypted. > > > Running Nifi on top of an encrypted EBS is not considered good > > enough in my > > > case. The ListenHTTP processor does not let people encrypt > content > > before > > > it is written to disk. You can encrypt the content > downstream, but > > at that > > > point it has already been written to disk so that is too late. > > People can > > > encrypt content before it is sent to Nifi, but in some > situations > > that may > > > be very challenging for the developer, as they may have limited > > access to > > > the source of the data. > > > > > > I was thinking of modifying the ListenHTTP processor and other > > similar > > > processors. I want to create a ControllerService interface > that > > merely > > > returns a StreamCallback implementation. The ListenHTTP > processor > > would > > > take this as an optional property. If that property is set, > then the > > > processor will use that to modify/encrypt content before it is > even > > written > > > to disk. If the property is not set, then it will operate the > same > > way it > > > does now. > > > > > > I looked for a good project to place this controller service > > interface in, > > > I feel like this service is so basic that it should really be > part > > of the > > > framework’s core, but I don’t see any other controller services > > there. So > > > my best guess for now is to put this in the > > nifi-ssl-context-service-(api|bundle) > > > projects. I feel like this is not really related to SSL, but > that > > is the > > > only project I found that has controller services listed and > is a > > > dependency of both the nifi-kafka projects and the > > nifi-standard-processors > > > projects. I think it would be a waste to set up a new > api/bundle > > pair just > > > for one interface. > > > > > > So my questions are: > > > > > > 1. Do you think this is a good idea? > > > > > > 2. Where should I put this code if I write it? > > > > > > Michael Knapp > > > Capital One > > > ________________________________________________________ > > > > > > The information contained in this e-mail is confidential and/or > > > proprietary to Capital One and/or its affiliates and may only > be used > > > solely in performance of work or services for Capital One. The > > information > > > transmitted herewith is intended only for use by the > individual or > > entity > > > to which it is addressed. If the reader of this message is not > the > > intended > > > recipient, you are hereby notified that any review, > retransmission, > > > dissemination, distribution, copying or other use of, or > taking of > > any > > > action in reliance upon this information is strictly > prohibited. If > > you > > > have received this communication in error, please contact the > sender > > and > > > delete the material from your computer. > > > > > > > > > ________________________________________________________ > > > > The information contained in this e-mail is confidential and/or > > proprietary to Capital One and/or its affiliates and may only be used > > solely in performance of work or services for Capital One. The > information > > transmitted herewith is intended only for use by the individual or > entity > > to which it is addressed. If the reader of this message is not the > intended > > recipient, you are hereby notified that any review, retransmission, > > dissemination, distribution, copying or other use of, or taking of > any > > action in reliance upon this information is strictly prohibited. If > you > > have received this communication in error, please contact the sender > and > > delete the material from your computer. > > > > > ________________________________________________________ > > The information contained in this e-mail is confidential and/or > proprietary to Capital One and/or its affiliates and may only be used > solely in performance of work or services for Capital One. The information > transmitted herewith is intended only for use by the individual or entity > to which it is addressed. If the reader of this message is not the intended > recipient, you are hereby notified that any review, retransmission, > dissemination, distribution, copying or other use of, or taking of any > action in reliance upon this information is strictly prohibited. If you > have received this communication in error, please contact the sender and > delete the material from your computer. >
