Russell, This sounds like a great idea. I'm no clojure expert but it seems like it would be quite reasonable to build processors in Clojure. We've also discussed doing something similar in Scala. Basically languages that run on the JVM we should have a good chance of providing a nice developer experience for.
It is quite common for folks in the earlier stages of learning nifi to utilize ExecuteStreamCommand, ExecuteProcess, etc.. type of processors. This is a nice and gradual transition model. But we do like to help avoid having to make those external calls over time. Would you be interested in collaborating on putting together some nice examples or talking about how we could support clojure nicely? Thanks Joe On Tue, Jun 9, 2015 at 3:27 PM, Russell Whitaker <[email protected]> wrote: > At work, I've found the sweet spot for Clojure programming in our Hadoop > data processing stack: writing Hive UDFs (user-defined functions) which get > distributed to HDFS, registered ("add jar ..."), and invoked as needed by > users. > It's been a real treat to avoid having to write Java (which I can do > and have done > much of in a past life) but still interoperate in the JVM. > > Now we're adding Nifi as a generalized data ingestion system into our Hadoop > processing clusters, with various sources and (mostly) PutHDFS targets (hoping > to do PutS3 in future), and are wondering how we might consider our team's > emerging development pattern of Clojure coding to the JVM and plugging into an > otherwise "pure Java" framework; i.e. we'd like to explore doing under > Nifi what > we've done under Hadoop. > > So far, the only option that really comes to mind is invoking "java > -cp <my_clojure_lib>" > in the context of an ExecuteStream processor, which is fine as things > go - and how > we're likely to quickly prototype new core libs for business-specific > ingestion > logic in Nifi - but I'm wondering if there's been some > as-yet-undiscussed thinking on > this matter in the Nifi community. > > Thanks, R > > -- > Russell Whitaker > http://twitter.com/OrthoNormalRuss > http://github.com/russellwhitaker
