Andrew Awesome and thanks for sharing that!
Ricky, Definitely with you seeing these as valuable tools for the toolbox. The rub on supporting these languages is that they require folks with solid understanding of the fundamentals of those languages and with knowledge of NiFi. So it is just something that will take time but i am confident we'll get there. The other thing that is a bit of a bummer with these languages is they tend to come with very large dependency trails or sizes. The nifi build is already too big in my opinion (150+ MB). I really would like to see us move to a registry construct whereby folks can indicate which processors/extensions they want activated on their system. There are lots of challenges with that so I'm guessing it won't be soon. Russell What you're proposing just simply makes sense and is a good idea so it is easy to be welcoming - but thanks. Surely there are going to be some gotchas but let's work through them. Ultimately this is about the power of the JVM - not Java. You know Clojure and we know nifi. Do you know if in the case of Clojure one could do something like Andrew did with his Scala example? The dynamic loading aspect these could provide, as Ricky points out, could be hugely cool when combined with NiFi's existing capabilities. I'll start a Jira for these in a bit if they're not already there (and Ricky's Python and Groovy and Andrew's Scala). Thanks Joe On Jun 10, 2015 10:44 AM, "Andrew Hulbert" <[email protected]> wrote: > Russell, > > I've had good success building 3-4 processors in Scala so far such as this > one...but haven't used too many advanced scala features in it at the > moment. Mostly I'm just mimicking Java style but it does work. > > > https://github.com/jahhulbert-ccri/geomesa-nifi/blob/master/nifi-geomesa-nifi-processors/src/main/scala/org/locationtech/geomesa/nifi/GeoMesaIngestProcessor.scala > > -Andrew > > On 06/10/2015 12:14 AM, Joe Witt wrote: > >> Russell, >> >> This sounds like a great idea. I'm no clojure expert but it seems >> like it would be quite reasonable to build processors in Clojure. >> We've also discussed doing something similar in Scala. Basically >> languages that run on the JVM we should have a good chance of >> providing a nice developer experience for. >> >> It is quite common for folks in the earlier stages of learning nifi to >> utilize ExecuteStreamCommand, ExecuteProcess, etc.. type of >> processors. This is a nice and gradual transition model. But we do >> like to help avoid having to make those external calls over time. >> >> Would you be interested in collaborating on putting together some nice >> examples or talking about how we could support clojure nicely? >> >> Thanks >> Joe >> >> On Tue, Jun 9, 2015 at 3:27 PM, Russell Whitaker >> <[email protected]> wrote: >> >>> At work, I've found the sweet spot for Clojure programming in our Hadoop >>> data processing stack: writing Hive UDFs (user-defined functions) which >>> get >>> distributed to HDFS, registered ("add jar ..."), and invoked as needed >>> by users. >>> It's been a real treat to avoid having to write Java (which I can do >>> and have done >>> much of in a past life) but still interoperate in the JVM. >>> >>> Now we're adding Nifi as a generalized data ingestion system into our >>> Hadoop >>> processing clusters, with various sources and (mostly) PutHDFS targets >>> (hoping >>> to do PutS3 in future), and are wondering how we might consider our >>> team's >>> emerging development pattern of Clojure coding to the JVM and plugging >>> into an >>> otherwise "pure Java" framework; i.e. we'd like to explore doing under >>> Nifi what >>> we've done under Hadoop. >>> >>> So far, the only option that really comes to mind is invoking "java >>> -cp <my_clojure_lib>" >>> in the context of an ExecuteStream processor, which is fine as things >>> go - and how >>> we're likely to quickly prototype new core libs for business-specific >>> ingestion >>> logic in Nifi - but I'm wondering if there's been some >>> as-yet-undiscussed thinking on >>> this matter in the Nifi community. >>> >>> Thanks, R >>> >>> -- >>> Russell Whitaker >>> http://twitter.com/OrthoNormalRuss >>> http://github.com/russellwhitaker >>> >> >
