Russell,
I've had good success building 3-4 processors in Scala so far such as
this one...but haven't used too many advanced scala features in it at
the moment. Mostly I'm just mimicking Java style but it does work.
https://github.com/jahhulbert-ccri/geomesa-nifi/blob/master/nifi-geomesa-nifi-processors/src/main/scala/org/locationtech/geomesa/nifi/GeoMesaIngestProcessor.scala
-Andrew
On 06/10/2015 12:14 AM, Joe Witt wrote:
Russell,
This sounds like a great idea. I'm no clojure expert but it seems
like it would be quite reasonable to build processors in Clojure.
We've also discussed doing something similar in Scala. Basically
languages that run on the JVM we should have a good chance of
providing a nice developer experience for.
It is quite common for folks in the earlier stages of learning nifi to
utilize ExecuteStreamCommand, ExecuteProcess, etc.. type of
processors. This is a nice and gradual transition model. But we do
like to help avoid having to make those external calls over time.
Would you be interested in collaborating on putting together some nice
examples or talking about how we could support clojure nicely?
Thanks
Joe
On Tue, Jun 9, 2015 at 3:27 PM, Russell Whitaker
<[email protected]> wrote:
At work, I've found the sweet spot for Clojure programming in our Hadoop
data processing stack: writing Hive UDFs (user-defined functions) which get
distributed to HDFS, registered ("add jar ..."), and invoked as needed by users.
It's been a real treat to avoid having to write Java (which I can do
and have done
much of in a past life) but still interoperate in the JVM.
Now we're adding Nifi as a generalized data ingestion system into our Hadoop
processing clusters, with various sources and (mostly) PutHDFS targets (hoping
to do PutS3 in future), and are wondering how we might consider our team's
emerging development pattern of Clojure coding to the JVM and plugging into an
otherwise "pure Java" framework; i.e. we'd like to explore doing under
Nifi what
we've done under Hadoop.
So far, the only option that really comes to mind is invoking "java
-cp <my_clojure_lib>"
in the context of an ExecuteStream processor, which is fine as things
go - and how
we're likely to quickly prototype new core libs for business-specific ingestion
logic in Nifi - but I'm wondering if there's been some
as-yet-undiscussed thinking on
this matter in the Nifi community.
Thanks, R
--
Russell Whitaker
http://twitter.com/OrthoNormalRuss
http://github.com/russellwhitaker
