I would personally like to see better support for dynamically loaded processors. I'm thinking along the lines of a custom processor (e.g. ExecuteGroovy or ExecuteJython) which would dynamically load a script passed by a user, and then expose things like log, session, and context to the executing script. This would eliminate the requirement of building, and deploying of a new NAR for each processor you create.
On Wed, Jun 10, 2015 at 12:44 PM, Andrew Hulbert <[email protected]> wrote: > Russell, > > I've had good success building 3-4 processors in Scala so far such as this > one...but haven't used too many advanced scala features in it at the > moment. Mostly I'm just mimicking Java style but it does work. > > > https://github.com/jahhulbert-ccri/geomesa-nifi/blob/master/nifi-geomesa-nifi-processors/src/main/scala/org/locationtech/geomesa/nifi/GeoMesaIngestProcessor.scala > > -Andrew > > > On 06/10/2015 12:14 AM, Joe Witt wrote: > >> Russell, >> >> This sounds like a great idea. I'm no clojure expert but it seems >> like it would be quite reasonable to build processors in Clojure. >> We've also discussed doing something similar in Scala. Basically >> languages that run on the JVM we should have a good chance of >> providing a nice developer experience for. >> >> It is quite common for folks in the earlier stages of learning nifi to >> utilize ExecuteStreamCommand, ExecuteProcess, etc.. type of >> processors. This is a nice and gradual transition model. But we do >> like to help avoid having to make those external calls over time. >> >> Would you be interested in collaborating on putting together some nice >> examples or talking about how we could support clojure nicely? >> >> Thanks >> Joe >> >> On Tue, Jun 9, 2015 at 3:27 PM, Russell Whitaker >> <[email protected]> wrote: >> >>> At work, I've found the sweet spot for Clojure programming in our Hadoop >>> data processing stack: writing Hive UDFs (user-defined functions) which >>> get >>> distributed to HDFS, registered ("add jar ..."), and invoked as needed >>> by users. >>> It's been a real treat to avoid having to write Java (which I can do >>> and have done >>> much of in a past life) but still interoperate in the JVM. >>> >>> Now we're adding Nifi as a generalized data ingestion system into our >>> Hadoop >>> processing clusters, with various sources and (mostly) PutHDFS targets >>> (hoping >>> to do PutS3 in future), and are wondering how we might consider our >>> team's >>> emerging development pattern of Clojure coding to the JVM and plugging >>> into an >>> otherwise "pure Java" framework; i.e. we'd like to explore doing under >>> Nifi what >>> we've done under Hadoop. >>> >>> So far, the only option that really comes to mind is invoking "java >>> -cp <my_clojure_lib>" >>> in the context of an ExecuteStream processor, which is fine as things >>> go - and how >>> we're likely to quickly prototype new core libs for business-specific >>> ingestion >>> logic in Nifi - but I'm wondering if there's been some >>> as-yet-undiscussed thinking on >>> this matter in the Nifi community. >>> >>> Thanks, R >>> >>> -- >>> Russell Whitaker >>> http://twitter.com/OrthoNormalRuss >>> http://github.com/russellwhitaker >>> >> > -- Ricky Saltzer http://www.cloudera.com
