No worries, while "it's working but I don't entirely know why"
satisfies me immediately, it does keep niggling at the back of my mind
(not least in case it randomly stops working!).
Thanks for all the help,
James
On 7 July 2017 at 17:44, Bryan Bende <bbe...@gmail.com> wrote:
> Actually after looking at the code, I stand corrected...
>
> The conf directory is actually on the class path of the system class
> loader when NiFi is launched, so this makes sense.
>
> The classpath hierarchy is something like...
>
> - System Class Loader (conf dir and jar files in lib)
> - Jetty NAR Class Loader with a parent of System Class Loader
> - Geomesa NAR Class Loader with a parent of Jetty NAR Class Loader
>
> So when the code in your Gemoesa NAR looks for it, it will traverse up
> the chain to the system class loader and find it there.
>
> The classpath of the system class loader should be what you see in
> nifi-bootstrap.log for the command that started NiFi.
>
> Sorry for the confusion.
>
>
> On Fri, Jul 7, 2017 at 12:03 PM, James Srinivasan
> <james.sriniva...@gmail.com> wrote:
>> Here's where strace tells me my processor is looking for core-site.xml:
>>
>> [nifi@nifi nifi-1.3.0]$ strace -p 22189 -f 2>&1 | grep core-site.xml
>> [pid 22284] stat("/opt/nifi/nifi-1.3.0/conf/core-site.xml", <unfinished ...>
>> [pid 22284]
>> stat("/opt/nifi/nifi-1.3.0/work/nar/extensions/nifi-jetty-bundle-1.3.0.nar-unpacked/core-site.xml",
>> 0x7fa0831f1df0) = -1 ENOENT (No such file or directory)
>> [pid 22284]
>> stat("/opt/nifi/nifi-1.3.0/work/nar/extensions/nifi-jetty-bundle-1.3.0.nar-unpacked/META-INF/bundled-dependencies/core-site.xml",
>> <unfinished ...>
>> [pid 22284]
>> stat("/opt/nifi/nifi-1.3.0/work/nar/extensions/geomesa-nifi-nar-0.9.0-SNAPSHOT.nar-unpacked/core-site.xml",
>> <unfinished ...>
>> [pid 22284]
>> stat("/opt/nifi/nifi-1.3.0/work/nar/extensions/geomesa-nifi-nar-0.9.0-SNAPSHOT.nar-unpacked/META-INF/bundled-dependencies/core-site.xml",
>> <unfinished ...>
>> [pid 22284] stat("/opt/nifi/nifi-1.3.0/conf/core-site.xml", <unfinished ...>
>> [pid 22284]
>> stat("/opt/nifi/nifi-1.3.0/work/nar/extensions/nifi-jetty-bundle-1.3.0.nar-unpacked/core-site.xml",
>> <unfinished ...>
>> [pid 22284]
>> stat("/opt/nifi/nifi-1.3.0/work/nar/extensions/nifi-jetty-bundle-1.3.0.nar-unpacked/META-INF/bundled-dependencies/core-site.xml",
>> 0x7fa0831f2600) = -1 ENOENT (No such file or directory)
>> [pid 22284]
>> stat("/opt/nifi/nifi-1.3.0/work/nar/extensions/geomesa-nifi-nar-0.9.0-SNAPSHOT.nar-unpacked/core-site.xml",
>> 0x7fa0831f2680) = -1 ENOENT (No such file or directory)
>> [pid 22284]
>> stat("/opt/nifi/nifi-1.3.0/work/nar/extensions/geomesa-nifi-nar-0.9.0-SNAPSHOT.nar-unpacked/META-INF/bundled-dependencies/core-site.xml",
>> 0x7fa0831f2680) = -1 ENOENT (No such file or directory)
>> [pid 22284] stat("/opt/nifi/nifi-1.3.0/conf/core-site.xml",
>> 0x7fa0831f23c0) = -1 ENOENT (No such file or directory)
>> [pid 22284]
>> stat("/opt/nifi/nifi-1.3.0/work/nar/extensions/nifi-jetty-bundle-1.3.0.nar-unpacked/core-site.xml",
>> 0x7fa0831f2440) = -1 ENOENT (No such file or directory)
>> [pid 22284]
>> stat("/opt/nifi/nifi-1.3.0/work/nar/extensions/nifi-jetty-bundle-1.3.0.nar-unpacked/META-INF/bundled-dependencies/core-site.xml",
>> 0x7fa0831f2440) = -1 ENOENT (No such file or directory)
>> [pid 22284]
>> stat("/opt/nifi/nifi-1.3.0/work/nar/extensions/geomesa-nifi-nar-0.9.0-SNAPSHOT.nar-unpacked/core-site.xml",
>> 0x7fa0831f24c0) = -1 ENOENT (No such file or directory)
>> [pid 22284]
>> stat("/opt/nifi/nifi-1.3.0/work/nar/extensions/geomesa-nifi-nar-0.9.0-SNAPSHOT.nar-unpacked/META-INF/bundled-dependencies/core-site.xml",
>> 0x7fa0831f24c0) = -1 ENOENT (No such file or directory)
>>
>> I've also separately dumped the classpath and confirmed that the NiFi
>> conf directory isn't on it. The Hadoop docs tell me it searches the
>> classpath for core-site.xml [1], so I'm confused as to why it works,
>> but happy that it does work!
>>
>> [1]
>> http://hadoop.apache.org/docs/r2.6.4/api/org/apache/hadoop/conf/Configuration.html
>> as one example
>>
>> On 7 July 2017 at 16:17, Bryan Bende <bbe...@gmail.com> wrote:
>>> Hi James,
>>>
>>> I'm saying that it should not find a core-site.xml in NiFi's conf
>>> directory because the conf directory is not on the classpath of
>>> processors.
>>>
>>> Are you saying you have tested putting it there and believe it is finding
>>> it?
>>>
>>> -Bryan
>>>
>>> On Fri, Jul 7, 2017 at 11:08 AM, James Srinivasan
>>> <james.sriniva...@gmail.com> wrote:
>>>> Hi Bryan,
>>>>
>>>> That all makes sense...except this is my own NiFi processor, not one
>>>> of the built-in Hadoop ones:
>>>>
>>>> https://github.com/jrs53/geomesa-nifi
>>>>
>>>> So I'm still confused as to why core-site.xml is picked up from NiFi's
>>>> conf directory, but very happy that it is!
>>>>
>>>> On 5 July 2017 at 21:55, Bryan Bende <bbe...@gmail.com> wrote:
>>>>> James,
>>>>>
>>>>> Just to clarify... the conf directory of NiFi is not on the classpath
>>>>> of processors.
>>>>>
>>>>> The behavior with core-site.xml is because the Hadoop client used by
>>>>> NiFi provides this behavior, and not NiFi itself. Basically if you
>>>>> create a new Hadoop Configuration object and you don't provide it with
>>>>> a list of resources, it will find any that are on the classpath, which
>>>>> in this case would be any that are bundled in JARs inside the given
>>>>> NAR, or anything in NiFI's lib directory. Putting stuff in NiFi's lib
>>>>> directory makes it visible to every processor and is generally
>>>>> dangerous, which is why the existing Hadoop processors have a property
>>>>> for the path to the core-site.xml file and then load it like so:
>>>>>
>>>>> String resource = ... // get the value of the property where the path is
>>>>> Configuration config = new Configuration();
>>>>> config.addResource(new Path(resource.trim()));
>>>>>
>>>>> Hope that helps.
>>>>>
>>>>> -Bryan
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Jul 5, 2017 at 4:15 PM, James Srinivasan
>>>>> <james.sriniva...@gmail.com> wrote:
>>>>>> Thanks, I ended up stracing the NiFi process to see where it was
>>>>>> actually trying to load my config file from. Helpfully, the list of
>>>>>> locations included the NiFi conf directory, which is just perfect for
>>>>>> me.
>>>>>>
>>>>>> I don't want to keep the config file (core-site.xml for Hadoop) in my
>>>>>> NAR itself since it will obviously change between different
>>>>>> deployments. I was inspired by:
>>>>>>
>>>>>> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-extension-utils/nifi-hadoop-utils/src/main/java/org/apache/nifi/processors/hadoop/AbstractHadoopProcessor.java#L69-L70
>>>>>>
>>>>>> Which says if that property isn't set, then NiFi searches the
>>>>>> classpath for the necessary file.
>>>>>>
>>>>>> Thanks again,
>>>>>>
>>>>>> James
>>>>>>
>>>>>> On 5 July 2017 at 00:07, Jeremy Dyer <jdy...@gmail.com> wrote:
>>>>>>> Hi James,
>>>>>>>
>>>>>>> NiFi doesn't do anything too crazy from what Java itself does. The
>>>>>>> "funky"
>>>>>>> part is the isolated classpath loaders. Really since your building a
>>>>>>> custom
>>>>>>> processor I would also assume your building a custom NAR. If that is the
>>>>>>> case you can read that resource from the "nar" just like you would any
>>>>>>> "jar"
>>>>>>> in java. The HDFS processor your referenced loads an external file that
>>>>>>> isn't bundled in the actual classpath at all. If that is the
>>>>>>> functionality
>>>>>>> you want I suggest following the template from the HDFS processors as
>>>>>>> you
>>>>>>> stated. Otherwise just carryon as usual and pretend its a regular java
>>>>>>> application.
>>>>>>>
>>>>>>> On Tue, Jul 4, 2017 at 4:31 PM, James Srinivasan
>>>>>>> <james.sriniva...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I'm developing a processor which needs to read some of its config from
>>>>>>>> the classpath. Reading the docs etc., NiFi's classpath is a little
>>>>>>>> funky - where's the best (least worst?) location for such files? I
>>>>>>>> note that the HDFS processors can read their config (core-site.xml
>>>>>>>> etc) from the classpath, but I can't find where that actually is.
>>>>>>>>
>>>>>>>> Thanks in advance,
>>>>>>>>
>>>>>>>> James
>>>>>>>
>>>>>>>
