Hi Bryan, That all makes sense...except this is my own NiFi processor, not one of the built-in Hadoop ones:
https://github.com/jrs53/geomesa-nifi So I'm still confused as to why core-site.xml is picked up from NiFi's conf directory, but very happy that it is! On 5 July 2017 at 21:55, Bryan Bende <[email protected]> wrote: > James, > > Just to clarify... the conf directory of NiFi is not on the classpath > of processors. > > The behavior with core-site.xml is because the Hadoop client used by > NiFi provides this behavior, and not NiFi itself. Basically if you > create a new Hadoop Configuration object and you don't provide it with > a list of resources, it will find any that are on the classpath, which > in this case would be any that are bundled in JARs inside the given > NAR, or anything in NiFI's lib directory. Putting stuff in NiFi's lib > directory makes it visible to every processor and is generally > dangerous, which is why the existing Hadoop processors have a property > for the path to the core-site.xml file and then load it like so: > > String resource = ... // get the value of the property where the path is > Configuration config = new Configuration(); > config.addResource(new Path(resource.trim())); > > Hope that helps. > > -Bryan > > > > On Wed, Jul 5, 2017 at 4:15 PM, James Srinivasan > <[email protected]> wrote: >> Thanks, I ended up stracing the NiFi process to see where it was >> actually trying to load my config file from. Helpfully, the list of >> locations included the NiFi conf directory, which is just perfect for >> me. >> >> I don't want to keep the config file (core-site.xml for Hadoop) in my >> NAR itself since it will obviously change between different >> deployments. I was inspired by: >> >> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-extension-utils/nifi-hadoop-utils/src/main/java/org/apache/nifi/processors/hadoop/AbstractHadoopProcessor.java#L69-L70 >> >> Which says if that property isn't set, then NiFi searches the >> classpath for the necessary file. >> >> Thanks again, >> >> James >> >> On 5 July 2017 at 00:07, Jeremy Dyer <[email protected]> wrote: >>> Hi James, >>> >>> NiFi doesn't do anything too crazy from what Java itself does. The "funky" >>> part is the isolated classpath loaders. Really since your building a custom >>> processor I would also assume your building a custom NAR. If that is the >>> case you can read that resource from the "nar" just like you would any "jar" >>> in java. The HDFS processor your referenced loads an external file that >>> isn't bundled in the actual classpath at all. If that is the functionality >>> you want I suggest following the template from the HDFS processors as you >>> stated. Otherwise just carryon as usual and pretend its a regular java >>> application. >>> >>> On Tue, Jul 4, 2017 at 4:31 PM, James Srinivasan >>> <[email protected]> wrote: >>>> >>>> Hi, >>>> >>>> I'm developing a processor which needs to read some of its config from >>>> the classpath. Reading the docs etc., NiFi's classpath is a little >>>> funky - where's the best (least worst?) location for such files? I >>>> note that the HDFS processors can read their config (core-site.xml >>>> etc) from the classpath, but I can't find where that actually is. >>>> >>>> Thanks in advance, >>>> >>>> James >>> >>>
