Michael, While processors, controller services, and reporting tasks are definitely the most common extension points, you should still be able to deploy a NAR with a custom repository implementation.
The provenance repository is a good example to look at since it is deployed as it's own NAR [1]. All extension points use the Java Service Loader so your JAR that would packaged in your NAR needs to have the appropriate src/main/resources/META-INF/services file as can be seen here for provenance [2]. Without that file NiFi would not recognize the extension point. Thanks, Bryan [1] https://github.com/apache/nifi/tree/master/nifi-nar-bundles/nifi-provenance-repository-bundle [2] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/resources/META-INF/services/org.apache.nifi.provenance.ProvenanceRepository On Tue, Jan 24, 2017 at 12:11 PM, Knapp, Michael < [email protected]> wrote: > Andy, > > > I and several of my co-workers are very interested in seeing this feature > added to NIFI. I have been trying to write an > EncryptedFileSystemRepository myself but unfortunately did not get very far > before running into issues. > > > So far I just tried a simple solution: I essentially extended > FileSystemRepository, and overrode the "read" and "write" methods to wrap > them with java's CipherInputStream or CipherOutputStream. I also added > ways to pass in a secret key. If that had worked, I would have updated it > to use AWS's KMS service to provide secret keys. I do think we would need > to roll keys periodically, but still haven't figured out how that would > work. > > > I ran into a lot of classpath issues. First I tried making a thin jar and > putting it on NIFI's lib, but unfortunately NIFI's classloader ensured that > my jar did not see any of nifi's framework classes. I tried making an uber > jar but encountered a lot of other classpath issues. I tried making a nar, > but NIFI still could not find my EncryptedRepository when spring was wiring > up the application. It seems you can only extend repositories from within > the source code. So I tried putting my code into the source and > re-building. Unfortunately my work's proxy prevented me from getting some > essential apache artifacts so I am unable to build NIFI from source. I > might try this again from a personal account. > > > So it seems that with NIFI it is not easy to implement things that are not > processors or controller services, since you have to add that to the source > and re-build it. Perhaps NIFI needs an easier way to add these > implementations without re-building from source. > > > Since I don't want to put the entire AWS core artifact and its transitive > dependencies inside NIFI, I was thinking of using java's SPI framework to > provide encryption services to the EncryptedRepository. > > > I'm not sure I really answered any of your questions, but hopefully I gave > you an idea of how we might use this. I might be available to help develop > and/or test the solution. > > > Michael Knapp > > Capital One > > ________________________________ > From: Andy LoPresto <[email protected]> > Sent: Tuesday, January 24, 2017 12:13:07 AM > To: [email protected] > Subject: [DISCUSS] Encrypted repositories (content, flowfile, provenance) > > I am working on building drop-in encrypted repositories for NiFi [1]. I am > currently in the planning stages and have written up a fairly extensive > Jira ticket documenting my plans (high-level) and some concerns (much > bigger section). I would welcome community feedback in order to capture > expectations, concerns (for security, performance, and usability), and any > other valuable contributions. All of us are smarter than one of us, and I > am not that smart. Thanks. > > [1] https://issues.apache.org/jira/browse/NIFI-3388 > > Andy LoPresto > [email protected]<mailto:[email protected]> > [email protected]<mailto:[email protected]> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > ________________________________________________________ > > The information contained in this e-mail is confidential and/or > proprietary to Capital One and/or its affiliates and may only be used > solely in performance of work or services for Capital One. The information > transmitted herewith is intended only for use by the individual or entity > to which it is addressed. If the reader of this message is not the intended > recipient, you are hereby notified that any review, retransmission, > dissemination, distribution, copying or other use of, or taking of any > action in reliance upon this information is strictly prohibited. If you > have received this communication in error, please contact the sender and > delete the material from your computer. >
