Hi Alexander,

I had a similar task, but it my case flowfiles were tiny. It might still
work well, because you can stream stdout / sdterr in real-time. Check my
blog post.

I also wanted to use Jython, but Groovy is a really fun language, that you
can grasp basics in hours. It also does not have performance issues like
with Jython

http://boristyukin.com/how-to-run-sqoop-from-nifi/

Boris

On Wed, Mar 7, 2018 at 12:34 PM, Oleksandr Lobunets <
alexander.lobun...@gmail.com> wrote:

> Hello everyone,
>
> I have a case  of running the 3rd party CLI (linux) with the following
> behaviour:
>  - Should be executed upon a FlowFile with attributes/content containing
> parameters to CLI
>  - Accepts params via flags or environment variables
>  - Writes output to stdout as a stream of JSON objects
>  - The output might be huge (millions and millions of objects), which
> means caching stdout is not an option - each line/object should be sent as
> a separate FlowFile
>  - The errors/log is written to stderr (might be very chatty)
>
> Using ExecuteProcessor is not an option (cannot be trigger by incoming
> FlowFile), but the way it treats stdout is what is desired.
> Using ExecuteStreamCommand is not an option as it buffers the output until
> the binary exists with a status code 0.
>
> Does anybody know if there’s a hybrid component somewhere out there? ;-)
>
> Thank you in advance!
>
> P.S. I’ve tried to write a wrapping script in Python using ExecuteScript
> processor, but:
>  - it looks rather an overkill (JVM -> Jython -> Python -> System process
> -> …)
>  - scripting for NiFi is not providing a pleasant debugging experience
>  - I get weird random errors when moving flow from machine to machine -
> exact copies of VMs (like the example below).
>
> > Caused by: javax.script.ScriptException: AttributeError: type object
> 'java.lang.Thread' has no attribute 'State' in <script> at line number 1
> >         at org.python.jsr223.PyScriptEngine.scriptException(
> PyScriptEngine.java:222)
> >         at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:59)
> >         at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:31)
> >         at javax.script.AbstractScriptEngine.eval(
> AbstractScriptEngine.java:264)
> >         at org.apache.nifi.script.impl.JythonScriptEngineConfigurator
> .eval(JythonScriptEngineConfigurator.java:59)
> >         at org.apache.nifi.processors.script.ExecuteScript.
> onTrigger(ExecuteScript.java:220)
>
>
> Kind regards,
> Alexander

Reply via email to