[
https://issues.apache.org/jira/browse/NIFI-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14380196#comment-14380196
]
Toivo Adams commented on NIFI-421:
----------------------------------
I created test and did some experiments.
1. Under Windows COMMAND and COMMAND_ARGUMENTS must be described little
bit differently.
Because echo, type etc. are not separate runnable but shell commands under
Windows.
So I ended up:
runner.setProperty(ExecuteProcess.COMMAND, "cmd");
runner.setProperty(ExecuteProcess.COMMAND_ARGUMENTS, " /c type " +
testFile);
this will send testFile content to STDOUT
2. When using batching current implementation will create separate
FlowFile’s
C:\var\test\eclipse-java-luna-SR2-win32.zip
[pool-2-thread-1] INFO org.apache.nifi.processors.standard.ExecuteProcess -
ExecuteProcess[id=b3409684-f547-4b86-b024-b7530ad47830] Created
FlowFile[0,190679803977701.mockFlowFile,8744508B] and routed to success
[pool-2-thread-1] INFO org.apache.nifi.processors.standard.ExecuteProcess -
ExecuteProcess[id=b3409684-f547-4b86-b024-b7530ad47830] Created
FlowFile[1,190680309319120.mockFlowFile,13564457B] and routed to success
[pool-2-thread-1] INFO org.apache.nifi.processors.standard.ExecuteProcess -
ExecuteProcess[id=b3409684-f547-4b86-b024-b7530ad47830] Created
FlowFile[2,190680809224379.mockFlowFile,13023526B] and routed to success
[pool-2-thread-1] INFO org.apache.nifi.processors.standard.ExecuteProcess -
ExecuteProcess[id=b3409684-f547-4b86-b024-b7530ad47830] Created
FlowFile[3,190681308388289.mockFlowFile,16149174B] and routed to success
[pool-2-thread-1] INFO org.apache.nifi.processors.standard.ExecuteProcess -
ExecuteProcess[id=b3409684-f547-4b86-b024-b7530ad47830] Created
FlowFile[4,190681866955600.mockFlowFile,16738580B] and routed to success
[pool-2-thread-1] INFO org.apache.nifi.processors.standard.ExecuteProcess -
ExecuteProcess[id=b3409684-f547-4b86-b024-b7530ad47830] Created
FlowFile[5,190682394811035.mockFlowFile,16847781B] and routed to success
[pool-2-thread-1] INFO org.apache.nifi.processors.standard.ExecuteProcess -
ExecuteProcess[id=b3409684-f547-4b86-b024-b7530ad47830] Created
FlowFile[6,190682900217983.mockFlowFile,15156431B] and routed to success
[pool-2-thread-1] INFO org.apache.nifi.processors.standard.ExecuteProcess -
ExecuteProcess[id=b3409684-f547-4b86-b024-b7530ad47830] Created
FlowFile[7,190683405733488.mockFlowFile,16615835B] and routed to success
[pool-2-thread-1] INFO org.apache.nifi.processors.standard.ExecuteProcess -
ExecuteProcess[id=b3409684-f547-4b86-b024-b7530ad47830] Created
FlowFile[8,190683923060434.mockFlowFile,17215366B] and routed to success
[pool-2-thread-1] INFO org.apache.nifi.processors.standard.ExecuteProcess -
ExecuteProcess[id=b3409684-f547-4b86-b024-b7530ad47830] Created
FlowFile[9,190684429768887.mockFlowFile,16142399B] and routed to success
[pool-2-thread-1] INFO org.apache.nifi.processors.standard.ExecuteProcess -
ExecuteProcess[id=b3409684-f547-4b86-b024-b7530ad47830] Created
FlowFile[10,190684987105749.mockFlowFile,16714082B] and routed to success
[pool-2-thread-1] INFO org.apache.nifi.processors.standard.ExecuteProcess -
ExecuteProcess[id=b3409684-f547-4b86-b024-b7530ad47830] Created
FlowFile[11,190685493712750.mockFlowFile,17301576B] and routed to success
[pool-2-thread-1] INFO org.apache.nifi.processors.standard.ExecuteProcess -
ExecuteProcess[id=b3409684-f547-4b86-b024-b7530ad47830] Created
FlowFile[12,190686008815255.mockFlowFile,17081644B] and routed to success
[pool-2-thread-1] INFO org.apache.nifi.processors.standard.ExecuteProcess -
ExecuteProcess[id=b3409684-f547-4b86-b024-b7530ad47830] Created
FlowFile[13,190686509522260.mockFlowFile,17044027B] and routed to success
[pool-2-thread-1] INFO org.apache.nifi.processors.standard.ExecuteProcess -
ExecuteProcess[id=b3409684-f547-4b86-b024-b7530ad47830] Created
FlowFile[14,190687010689548.mockFlowFile,13969489B] and routed to success
[pool-2-thread-1] INFO org.apache.nifi.processors.standard.ExecuteProcess -
ExecuteProcess[id=b3409684-f547-4b86-b024-b7530ad47830] Created
FlowFile[15,190687511876179.mockFlowFile,17165472B] and routed to success
[pool-2-thread-1] INFO org.apache.nifi.processors.standard.ExecuteProcess -
ExecuteProcess[id=b3409684-f547-4b86-b024-b7530ad47830] Created
FlowFile[16,190688012807799.mockFlowFile,16494533B] and routed to success
[pool-2-thread-1] INFO org.apache.nifi.processors.standard.ExecuteProcess -
ExecuteProcess[id=b3409684-f547-4b86-b024-b7530ad47830] Created
FlowFile[17,190688518508444.mockFlowFile,16645927B] and routed to success
[pool-2-thread-1] INFO org.apache.nifi.processors.standard.ExecuteProcess -
ExecuteProcess[id=b3409684-f547-4b86-b024-b7530ad47830] Created
FlowFile[18,190689029899077.mockFlowFile,6990808B] and routed to success
[pool-2-thread-1] INFO org.apache.nifi.processors.standard.ExecuteProcess -
ExecuteProcess[id=b3409684-f547-4b86-b024-b7530ad47830] Process finished with
exit code 0 after creating 19 FlowFiles in 9750 millis
FlowFile[0,190679803977701.mockFlowFile,8744508B]
FlowFile[1,190680309319120.mockFlowFile,13564457B]
FlowFile[2,190680809224379.mockFlowFile,13023526B]
FlowFile[3,190681308388289.mockFlowFile,16149174B]
FlowFile[4,190681866955600.mockFlowFile,16738580B]
FlowFile[5,190682394811035.mockFlowFile,16847781B]
FlowFile[6,190682900217983.mockFlowFile,15156431B]
FlowFile[7,190683405733488.mockFlowFile,16615835B]
FlowFile[8,190683923060434.mockFlowFile,17215366B]
FlowFile[9,190684429768887.mockFlowFile,16142399B]
FlowFile[10,190684987105749.mockFlowFile,16714082B]
FlowFile[11,190685493712750.mockFlowFile,17301576B]
FlowFile[12,190686008815255.mockFlowFile,17081644B]
FlowFile[13,190686509522260.mockFlowFile,17044027B]
FlowFile[14,190687010689548.mockFlowFile,13969489B]
FlowFile[15,190687511876179.mockFlowFile,17165472B]
FlowFile[16,190688012807799.mockFlowFile,16494533B]
FlowFile[17,190688518508444.mockFlowFile,16645927B]
FlowFile[18,190689029899077.mockFlowFile,6990808B]
3. But when batching is not used, only single big FlowFile will be created
// runner.setProperty(ExecuteProcess.BATCH_DURATION, "500 millis");
C:\var\test\eclipse-java-luna-SR2-win32.zip
[pool-2-thread-1] INFO org.apache.nifi.processors.standard.ExecuteProcess -
ExecuteProcess[id=f3faa98a-a18b-46ed-ab29-398834ba6a68] Created
FlowFile[0,190811209417098.mockFlowFile,162114318B] and routed to success
[pool-2-thread-1] INFO org.apache.nifi.processors.standard.ExecuteProcess -
ExecuteProcess[id=f3faa98a-a18b-46ed-ab29-398834ba6a68] Process finished with
exit code 0 after creating 1 FlowFiles in 4768 millis
FlowFile[0,190811209417098.mockFlowFile,162114318B]
So how this can be splitted to several FlowFiles?
I used eclipse-java-luna-SR2-win32.zip as test file just to have enough bytes –
its size is 158 MB
Also when we return after first FlowFile is created, future will keep running?
> ExecuteProcess does not honor back pressure setting
> ---------------------------------------------------
>
> Key: NIFI-421
> URL: https://issues.apache.org/jira/browse/NIFI-421
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Reporter: Joseph Witt
> Assignee: Toivo Adams
> Labels: beginner
> Fix For: 0.1.0
>
>
> ExecuteProcess does not pay attention to back pressure settings when
> operating on a continuous stream of output. It needs to be able to do so.
> It should also document or make optional its behavior in the event of a back
> pressure condition. One it can kill the process it is executing or two it
> can simply stop reading from the output stream. There may be valid cases for
> both options.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
