I didn't realize ContentRepository was found using the Service Loader, I will try that out soon.
BTW I never got around to responding the last time you helped, but I remember that your advice helped me get NIFI working again. Thanks again Bryan. ________________________________ From: Bryan Bende <[email protected]> Sent: Tuesday, January 24, 2017 12:54:53 PM To: [email protected] Subject: Re: [DISCUSS] Encrypted repositories (content, flowfile, provenance) Michael, While processors, controller services, and reporting tasks are definitely the most common extension points, you should still be able to deploy a NAR with a custom repository implementation. The provenance repository is a good example to look at since it is deployed as it's own NAR [1]. All extension points use the Java Service Loader so your JAR that would packaged in your NAR needs to have the appropriate src/main/resources/META-INF/services file as can be seen here for provenance [2]. Without that file NiFi would not recognize the extension point. Thanks, Bryan [1] https://github.com/apache/nifi/tree/master/nifi-nar-bundles/nifi-provenance-repository-bundle [2] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/resources/META-INF/services/org.apache.nifi.provenance.ProvenanceRepository On Tue, Jan 24, 2017 at 12:11 PM, Knapp, Michael < [email protected]> wrote: > Andy, > > > I and several of my co-workers are very interested in seeing this feature > added to NIFI. I have been trying to write an > EncryptedFileSystemRepository myself but unfortunately did not get very far > before running into issues. > > > So far I just tried a simple solution: I essentially extended > FileSystemRepository, and overrode the "read" and "write" methods to wrap > them with java's CipherInputStream or CipherOutputStream. I also added > ways to pass in a secret key. If that had worked, I would have updated it > to use AWS's KMS service to provide secret keys. I do think we would need > to roll keys periodically, but still haven't figured out how that would > work. > > > I ran into a lot of classpath issues. First I tried making a thin jar and > putting it on NIFI's lib, but unfortunately NIFI's classloader ensured that > my jar did not see any of nifi's framework classes. I tried making an uber > jar but encountered a lot of other classpath issues. I tried making a nar, > but NIFI still could not find my EncryptedRepository when spring was wiring > up the application. It seems you can only extend repositories from within > the source code. So I tried putting my code into the source and > re-building. Unfortunately my work's proxy prevented me from getting some > essential apache artifacts so I am unable to build NIFI from source. I > might try this again from a personal account. > > > So it seems that with NIFI it is not easy to implement things that are not > processors or controller services, since you have to add that to the source > and re-build it. Perhaps NIFI needs an easier way to add these > implementations without re-building from source. > > > Since I don't want to put the entire AWS core artifact and its transitive > dependencies inside NIFI, I was thinking of using java's SPI framework to > provide encryption services to the EncryptedRepository. > > > I'm not sure I really answered any of your questions, but hopefully I gave > you an idea of how we might use this. I might be available to help develop > and/or test the solution. > > > Michael Knapp > > Capital One > > ________________________________ > From: Andy LoPresto <[email protected]> > Sent: Tuesday, January 24, 2017 12:13:07 AM > To: [email protected] > Subject: [DISCUSS] Encrypted repositories (content, flowfile, provenance) > > I am working on building drop-in encrypted repositories for NiFi [1]. I am > currently in the planning stages and have written up a fairly extensive > Jira ticket documenting my plans (high-level) and some concerns (much > bigger section). I would welcome community feedback in order to capture > expectations, concerns (for security, performance, and usability), and any > other valuable contributions. All of us are smarter than one of us, and I > am not that smart. Thanks. > > [1] https://issues.apache.org/jira/browse/NIFI-3388 > > Andy LoPresto > [email protected]<mailto:[email protected]> > [email protected]<mailto:[email protected]> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > ________________________________________________________ > > The information contained in this e-mail is confidential and/or > proprietary to Capital One and/or its affiliates and may only be used > solely in performance of work or services for Capital One. The information > transmitted herewith is intended only for use by the individual or entity > to which it is addressed. If the reader of this message is not the intended > recipient, you are hereby notified that any review, retransmission, > dissemination, distribution, copying or other use of, or taking of any > action in reliance upon this information is strictly prohibited. If you > have received this communication in error, please contact the sender and > delete the material from your computer. > ________________________________________________________ The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
