[
https://issues.apache.org/jira/browse/NIFI-421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384001#comment-14384001
]
Toivo Adams commented on NIFI-421:
----------------------------------
http://docs.oracle.com/javase/7/docs/api/java/lang/Process.html
“Because some native platforms only provide limited buffer size for standard
input and output streams, failure to promptly write the input stream or read
the output stream of the subprocess may cause the subprocess to block, or even
deadlock.”
So then processor which consumes ExecuteProcess output is very slow and
back-pressure is applied – subprocess output stream maybe unread for long
period and may block or deadlock.
One possible solution is instead of ProxyOutputStream to use java standard
LinkedBlockingQueue.
In this case volatile member LinkedBlockingQueue queue will be created.
We can separate code to two parts.
Producer part is inside current Future and will read from process InputStream
and write to queue continuously until queue is full.
Consumer part is which will read from queue and write to FlowFile.
After writing one FlowFile we just return and next time will continue reading
from queue.
BlockingQueue will help to smooth processing, we can make queue size
configurable.
> ExecuteProcess does not honor back pressure setting
> ---------------------------------------------------
>
> Key: NIFI-421
> URL: https://issues.apache.org/jira/browse/NIFI-421
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Reporter: Joseph Witt
> Assignee: Toivo Adams
> Labels: beginner
> Fix For: 0.1.0
>
> Attachments: TestExecuteProcess.java
>
>
> ExecuteProcess does not pay attention to back pressure settings when
> operating on a continuous stream of output. It needs to be able to do so.
> It should also document or make optional its behavior in the event of a back
> pressure condition. One it can kill the process it is executing or two it
> can simply stop reading from the output stream. There may be valid cases for
> both options.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
