At work, I've found the sweet spot for Clojure programming in our Hadoop
data processing stack: writing Hive UDFs (user-defined functions) which get
distributed to HDFS, registered ("add jar ..."), and invoked as needed by users.
It's been a real treat to avoid having to write Java (which I can do
and have done
much of in a past life) but still interoperate in the JVM.Now we're adding Nifi as a generalized data ingestion system into our Hadoop processing clusters, with various sources and (mostly) PutHDFS targets (hoping to do PutS3 in future), and are wondering how we might consider our team's emerging development pattern of Clojure coding to the JVM and plugging into an otherwise "pure Java" framework; i.e. we'd like to explore doing under Nifi what we've done under Hadoop. So far, the only option that really comes to mind is invoking "java -cp <my_clojure_lib>" in the context of an ExecuteStream processor, which is fine as things go - and how we're likely to quickly prototype new core libs for business-specific ingestion logic in Nifi - but I'm wondering if there's been some as-yet-undiscussed thinking on this matter in the Nifi community. Thanks, R -- Russell Whitaker http://twitter.com/OrthoNormalRuss http://github.com/russellwhitaker
