Thanks, Matt, great info.
On Fri, Mar 25, 2016 at 10:35 AM, Matt Gilman <[email protected]>
wrote:
> Dmitry,
>
> /controller/templates
>
> Will return all the templates that have been created/imported in your NiFi
> instance. Created/Imported in this case does not mean instantiated in your
> data flow on the canvas. It's just the definition of the template that has
> been defined. Each template will have an identifier, name, description, etc.
>
> /controller/process-groups/{process-group-id}/template-instance
>
> Will instantiate a given template (that's previously been
> created/imported) into the process group specified as the path parameter.
> The process group id can be the identifier for the desired group or the
> alias 'root' if the target is the root level process group. Additionally,
> this endpoint will require an x and y coordinate. This is the origin (upper
> left) of the bounding box of all components in the template. This will
> drive the placement in the graph on your canvas.
>
> You can start and stop individual components or perform bulk actions using
> the encapsulating process group. For instance, you can start all components
> in a given process group.
>
> I don't know what 'done', 'in progress', or 'failed with error' mean in
> the context of your data flow. You'll need to monitor the status of the
> various processor that drive those states in your workflow.
>
> Everything that you see in the UI is happening through the REST API. I
> would suggest using NiFi with your browser's Developer Tools open and
> seeing the requests that are being made. You should be able to extract your
> desired curl command from there. I'm pretty sure that Chrome even supports
> a 'Copy as cUrl' option.
>
> Matt
>
> On Fri, Mar 25, 2016 at 9:27 AM, Dmitry Goldenberg <
> [email protected]> wrote:
>
>> Matt,
>>
>> >> The "official" API is the REST API (see link above), so you can use
>> "curl" or something like that to interact with the NiFi instance.
>>
>> I'm looking at the REST API documentation
>> <https://nifi.apache.org/docs/nifi-docs/rest-api/>. What I'd like to be
>> able to do is: look up a template, instantiate it as a dataflow, kick off
>> the dataflow, and get its status as "done", "failed with error", or "in
>> progress". How much of this can be done via the REST API?
>>
>> It looks like template instantiation can be done as
>> /controller/process-groups/{process-group-id}/template-instance. I've got
>> questions about this as, firstly I don't have any process groups defined in
>> my template, so must I organize the template as a process group? The
>> request parameters ask for a templateId; is that just the name of the
>> template?
>>
>> On the question of kicking off the dataflow. Basically I see that
>> Get/ingress processors (at least typically) already take care of their
>> scheduling strategies. However, for a DevOps case, devops engineers will
>> likely want to be in control of scheduling of dataflows and monitoring
>> their status, not via the GUI but through a command-line interface (which,
>> as we've discussed, could be done with something like curl via the REST
>> API).
>>
>> Would these abilities to schedule (or kick off once, immediately) a given
>> dataflow outside of the GUI and monitor its status be enhancement requests
>> then for NiFi? I'd imagine this would affect the existing Ingress
>> processors, as they'd need an alternative to the timer-driven and
>> CRON-driven scheduling strategies, to allow for ad hoc invocations and for
>> invocations made through the REST API.
>>
>> Thanks,
>> - Dmitry
>>
>>
>>
>>
>> ---------- Forwarded message ----------
>> From: Matt Burgess <[email protected]>
>> Date: Sun, Mar 20, 2016 at 3:23 PM
>> Subject: Re: NiFi: command-line interface ?
>> To: [email protected]
>>
>>
>> Dmitry,
>>
>> With regards to nifi-client (I am the author), that exception occurs when
>> the flow has been changed externally and the shell has not recognized it.
>> What the result of the following command?
>>
>> nifi.currentVersion
>>
>> If it is -1, then I recommend restarting the shell. It should be a
>> non-negative integer, a -1 usually indicates there was an error
>> communicating with the NiFi instance, or it couldn't parse the response.
>> The response containing the current revision of the NiFi workflow can be
>> obtained from your browser or "curl" using the following REST endpoint:
>>
>> /nifi-api/controller/revision
>>
>> It should be a JSON document, but there isn't any code in nifi-client
>> currently to force a JSON response. I will add that shortly for version 0.4
>> (which will allow you to add processors, interact with process groups,
>> etc.) What version of nifi-client and NiFi are you using?
>>
>> nifi-client is a Groovy object model for NiFi concepts (processor, e.g.),
>> and performing operations on the objects results in calls to the NiFi REST
>> API (https://nifi.apache.org/docs/nifi-docs/rest-api/index.html) To
>> answer the questions:
>>
>> 1) Your nifi-client example code should work as-is (but see above)
>>
>> 2) There is at least one other solution for command-line deployment:
>> https://github.com/aperepel/nifi-api-deploy. It uses Groovy scripts and
>> YAML configuration files to interact with the REST API. These are
>> contributions from the community and not "official" command-line tools for
>> Apache NiFi. Having said that, I'm quite sure we'd both appreciate any
>> suggestions (and especially contributions!) to make these better.
>>
>> The "official" API is the REST API (see link above), so you can use
>> "curl" or something like that to interact with the NiFi instance. However
>> Andrew and I have both used Groovy to abstract the complexities of the API
>> away from the user.
>>
>> Regards,
>> Matt
>>
>> On Sun, Mar 20, 2016 at 2:37 PM, Dmitry Goldenberg <
>> [email protected]> wrote:
>>
>>> I see some posts on the Web regarding NiFi's command-line interface.
>>> One of them lead me to a client library
>>> <https://github.com/mattyb149/nifi-client> that one can use via a
>>> Groovy shell.
>>>
>>> I seem to have been able to instantiate my sample dataflow through this
>>> interface but when I tried kicking off the dataflow by starting its first
>>> processor, I got an exception
>>> "processor:[id:6c71c1c9-1220-44a2-a020-8bd3c82d1692,
>>> state:RUNNING]] groovyx.net.http.HttpResponseException: Conflict.
>>>
>>> In the canvas, I've made sure nothing is running but I'm getting this
>>> exception (full stack trace below).
>>>
>>> My questions are:
>>>
>>> a) is this the right way to kick off a given existing dataflow;
>>> b) are there any other command-line options for performing operations on
>>> dataflows (and templates)?
>>>
>>> The CLI options are often critical from the DevOps perspective, mostly
>>> for being able to script things up.
>>>
>>> Thanks.
>>> - Dmitry
>>>
>>> ---------------------------------------
>>>
>>>
>>> *nifi-shell:*000*>* nifi.processors.'GetXmlFiles'.start()
>>>
>>> [revision:[version:713],
>>> processor:[id:6c71c1c9-1220-44a2-a020-8bd3c82d1692, state:RUNNING]]
>>>
>>> groovyx.net.http.HttpResponseException: Conflict
>>>
>>> at
>>> groovyx.net.http.HTTPBuilder.defaultFailureHandler(HTTPBuilder.java:651)
>>>
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>
>>> at java.lang.reflect.Method.invoke(Method.java:497)
>>>
>>> at
>>> org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
>>>
>>> at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
>>>
>>> at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1210)
>>>
>>> at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1077)
>>>
>>> at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1019)
>>>
>>> at groovy.lang.Closure.call(Closure.java:426)
>>>
>>> at groovyx.net.http.HTTPBuilder$1.handleResponse(HTTPBuilder.java:503)
>>>
>>> at
>>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:1070)
>>>
>>> at
>>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:1044)
>>>
>>> at groovyx.net.http.HTTPBuilder.doRequest(HTTPBuilder.java:515)
>>>
>>> at groovyx.net.http.HTTPBuilder.doRequest(HTTPBuilder.java:434)
>>>
>>> at groovyx.net.http.HTTPBuilder.request(HTTPBuilder.java:366)
>>>
>>> at groovyx.net.http.HTTPBuilder$request.call(Unknown Source)
>>>
>>> at nifi.client.Processor.updateState(Processor.groovy:149)
>>>
>>> at nifi.client.Processor$updateState$0.callCurrent(Unknown Source)
>>>
>>> at nifi.client.Processor.start(Processor.groovy:127)
>>>
>>> at nifi.client.Processor$start.call(Unknown Source)
>>>
>>> at
>>> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
>>>
>>> at
>>> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
>>>
>>> at
>>> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:117)
>>>
>>> at groovysh_evaluate.run(groovysh_evaluate:3)
>>>
>>> at groovysh_evaluate$run.call(Unknown Source)
>>>
>>> at
>>> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
>>>
>>> at groovysh_evaluate$run.call(Unknown Source)
>>>
>>> at
>>> org.codehaus.groovy.tools.shell.Interpreter.evaluate(Interpreter.groovy:70)
>>>
>>> at org.codehaus.groovy.tools.shell.Evaluator$evaluate.call(Unknown
>>> Source)
>>>
>>> at org.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:187)
>>>
>>> at org.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:122)
>>>
>>> at org.codehaus.groovy.tools.shell.Shell$leftShift$0.call(Unknown Source)
>>>
>>> at
>>> org.codehaus.groovy.tools.shell.ShellRunner.work(ShellRunner.groovy:95)
>>>
>>> at
>>> org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$work(InteractiveShellRunner.groovy)
>>>
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>
>>> at java.lang.reflect.Method.invoke(Method.java:497)
>>>
>>> at
>>> org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
>>>
>>> at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
>>>
>>> at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1210)
>>>
>>> at
>>> org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
>>>
>>> at
>>> org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)
>>>
>>> at
>>> org.codehaus.groovy.tools.shell.InteractiveShellRunner.work(InteractiveShellRunner.groovy:124)
>>>
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>
>>> at java.lang.reflect.Method.invoke(Method.java:497)
>>>
>>> at
>>> org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:210)
>>>
>>> at
>>> org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:59)
>>>
>>> at
>>> org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:158)
>>>
>>> at org.codehaus.groovy.tools.shell.ShellRunner.run(ShellRunner.groovy:59)
>>>
>>> at
>>> org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$run(InteractiveShellRunner.groovy)
>>>
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>
>>> at java.lang.reflect.Method.invoke(Method.java:497)
>>>
>>> at
>>> org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
>>>
>>> at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
>>>
>>> at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1210)
>>>
>>> at
>>> org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:132)
>>>
>>> at
>>> org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:152)
>>>
>>> at
>>> org.codehaus.groovy.tools.shell.InteractiveShellRunner.run(InteractiveShellRunner.groovy:83)
>>>
>>> at java_lang_Runnable$run.call(Unknown Source)
>>>
>>> at
>>> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
>>>
>>> at
>>> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
>>>
>>> at
>>> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:117)
>>>
>>> at org.codehaus.groovy.tools.shell.Groovysh.run(Groovysh.groovy:587)
>>>
>>> at org.codehaus.groovy.tools.shell.Groovysh$run$0.callCurrent(Unknown
>>> Source)
>>>
>>> at
>>> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:52)
>>>
>>> at
>>> org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:154)
>>>
>>> at
>>> org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:166)
>>>
>>> at org.codehaus.groovy.tools.shell.Groovysh.run(Groovysh.groovy:552)
>>>
>>> at org.codehaus.groovy.tools.shell.Groovysh$run.call(Unknown Source)
>>>
>>> at
>>> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
>>>
>>> at
>>> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
>>>
>>> at
>>> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133)
>>>
>>> at org.codehaus.groovy.tools.shell.Main.startGroovysh(Main.groovy:186)
>>>
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>
>>> at java.lang.reflect.Method.invoke(Method.java:497)
>>>
>>> at
>>> org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:210)
>>>
>>> at
>>> org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:71)
>>>
>>> at
>>> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
>>>
>>> at
>>> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
>>>
>>> at
>>> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133)
>>>
>>> at org.codehaus.groovy.tools.shell.Main.main(Main.groovy:152)
>>>
>>>
>>>
>>
>>
>>
>
