[
https://issues.apache.org/jira/browse/NIFI-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15866834#comment-15866834
]
ASF GitHub Bot commented on NIFI-1856:
--------------------------------------
Github user brosander commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1364#discussion_r101160112
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteStreamCommand.java
---
@@ -342,16 +393,45 @@ public void onTrigger(ProcessContext context, final
ProcessSession session) thro
final InputStream pis = process.getInputStream();
final InputStream pes = process.getErrorStream();
final BufferedInputStream bis = new
BufferedInputStream(pis);
+ final BufferedInputStream bes = new
BufferedInputStream(pes);
final BufferedReader bufferedReader = new
BufferedReader(new InputStreamReader(pes))) {
int exitCode = -1;
final BufferedOutputStream bos = new BufferedOutputStream(pos);
+ final StringBuilder strBldr = new StringBuilder();
FlowFile outputFlowFile = putToAttribute ? inputFlowFile :
session.create(inputFlowFile);
+ FlowFile errorFlowFile=null;
+ FlowWriterThread flowWriter = null;
+ if (REDIRECT_ERROR_ERROR_STREAM.equals(redirectErrorStream)) {
+ errorFlowFile = session.create(inputFlowFile);
+ flowWriter = new FlowWriterThread(errorFlowFile, session,
bes);
+ this.executor.submit(flowWriter);
+ }
+ //if redirect error stream is set to log, then write the error
to nifi component log.
+ if (REDIRECT_ERROR_LOG.equals(redirectErrorStream)) {
+ this.executor.submit(new Runnable() {
+ @Override
+ public void run() {
+ try (final BufferedReader reader = new
BufferedReader(new InputStreamReader(pes))) {
+ String line;
+ while ((line = bufferedReader.readLine()) !=
null) {
+ logger.warn("ExecuteStreamCommand
:"+line);
+ strBldr.append(line).append("\n");
--- End diff --
It looks like we're only going to use the first 4000 characters of strBldr,
we might want to preemptively throw away anything before that so we don't have
to store all the output in memory
> ExecuteStreamCommand Needs to Consume Standard Error
> ----------------------------------------------------
>
> Key: NIFI-1856
> URL: https://issues.apache.org/jira/browse/NIFI-1856
> Project: Apache NiFi
> Issue Type: Bug
> Reporter: Alan Jackoway
> Assignee: Karthik Narayanan
>
> I was using ExecuteStreamProcess to run certain hdfs commands that are tricky
> to write in nifi but easy in bash (e.g. {{hadoop fs -rm -r
> /data/*/2014/05/05}})
> However, my larger commands kept hanging even though when I run them from the
> command line they finish quickly.
> Based on
> http://www.javaworld.com/article/2071275/core-java/when-runtime-exec---won-t.html
> I believe that ExecuteStreamCommand and possibly other processors need to
> consume the standard error stream to prevent the processes from blocking when
> standard error gets filled.
> To reproduce. Create this as ~/write.py
> {code:python}
> import sys
> count = int(sys.argv[1])
> for x in range(count):
> sys.stderr.write("ERROR %d\n" % x)
> sys.stdout.write("OUTPUT %d\n" % x)
> {code}
> Create a flow that goes
> # GenerateFlowFile - 5 minutes schedule 0 bytes size
> # ExecuteStreamCommand - Command arguments /Users/alanj/write.py;100 Command
> Path python
> # PutFile - /tmp/write/
> routing output stream of ExecuteStreamCommand to PutFile
> When you turn everything on, you get 100 lines (not 200) of just the standard
> output in /tmp/write.
> Next, change the command arguments to /Users/alanj/write.py;100000 and turn
> everything on again. The command will hang.
> I believe that whenever you execute a process the way ExecuteStreamCommand is
> doing, you need to consume the standard error stream to keep it from
> blocking. This may also affect things like ExecuteProcess and ExecuteScript
> as well.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
