[
https://issues.apache.org/jira/browse/NIFI-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15868186#comment-15868186
]
ASF GitHub Bot commented on NIFI-1856:
--------------------------------------
Github user brosander commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1364#discussion_r101328875
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteStreamCommand.java
---
@@ -342,16 +393,45 @@ public void onTrigger(ProcessContext context, final
ProcessSession session) thro
final InputStream pis = process.getInputStream();
final InputStream pes = process.getErrorStream();
final BufferedInputStream bis = new
BufferedInputStream(pis);
+ final BufferedInputStream bes = new
BufferedInputStream(pes);
final BufferedReader bufferedReader = new
BufferedReader(new InputStreamReader(pes))) {
int exitCode = -1;
final BufferedOutputStream bos = new BufferedOutputStream(pos);
+ final StringBuilder strBldr = new StringBuilder();
FlowFile outputFlowFile = putToAttribute ? inputFlowFile :
session.create(inputFlowFile);
+ FlowFile errorFlowFile=null;
+ FlowWriterThread flowWriter = null;
+ if (REDIRECT_ERROR_ERROR_STREAM.equals(redirectErrorStream)) {
+ errorFlowFile = session.create(inputFlowFile);
+ flowWriter = new FlowWriterThread(errorFlowFile, session,
bes);
+ this.executor.submit(flowWriter);
+ }
+ //if redirect error stream is set to log, then write the error
to nifi component log.
+ if (REDIRECT_ERROR_LOG.equals(redirectErrorStream)) {
+ this.executor.submit(new Runnable() {
+ @Override
+ public void run() {
+ try (final BufferedReader reader = new
BufferedReader(new InputStreamReader(pes))) {
+ String line;
+ while ((line = bufferedReader.readLine()) !=
null) {
+ logger.warn("ExecuteStreamCommand
:"+line);
+ strBldr.append(line).append("\n");
--- End diff --
I think this probably warrants its own jira as it was existing behavior,
I'm going to write it up separately
> ExecuteStreamCommand Needs to Consume Standard Error
> ----------------------------------------------------
>
> Key: NIFI-1856
> URL: https://issues.apache.org/jira/browse/NIFI-1856
> Project: Apache NiFi
> Issue Type: Bug
> Reporter: Alan Jackoway
> Assignee: Karthik Narayanan
>
> I was using ExecuteStreamProcess to run certain hdfs commands that are tricky
> to write in nifi but easy in bash (e.g. {{hadoop fs -rm -r
> /data/*/2014/05/05}})
> However, my larger commands kept hanging even though when I run them from the
> command line they finish quickly.
> Based on
> http://www.javaworld.com/article/2071275/core-java/when-runtime-exec---won-t.html
> I believe that ExecuteStreamCommand and possibly other processors need to
> consume the standard error stream to prevent the processes from blocking when
> standard error gets filled.
> To reproduce. Create this as ~/write.py
> {code:python}
> import sys
> count = int(sys.argv[1])
> for x in range(count):
> sys.stderr.write("ERROR %d\n" % x)
> sys.stdout.write("OUTPUT %d\n" % x)
> {code}
> Create a flow that goes
> # GenerateFlowFile - 5 minutes schedule 0 bytes size
> # ExecuteStreamCommand - Command arguments /Users/alanj/write.py;100 Command
> Path python
> # PutFile - /tmp/write/
> routing output stream of ExecuteStreamCommand to PutFile
> When you turn everything on, you get 100 lines (not 200) of just the standard
> output in /tmp/write.
> Next, change the command arguments to /Users/alanj/write.py;100000 and turn
> everything on again. The command will hang.
> I believe that whenever you execute a process the way ExecuteStreamCommand is
> doing, you need to consume the standard error stream to keep it from
> blocking. This may also affect things like ExecuteProcess and ExecuteScript
> as well.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
