Hi Huagen, It looks to me like the ExecuteStreamCommand is designed to run a single command. A potential work around could be to make a shell script that executes the pipeline you’d like and then use the ExecuteStreamCommand to invoke that.
I’d use your shell command (e.g. /bin/bash) for the command and the script name for the first argument. (http://stackoverflow.com/questions/18240944/cannot-launch-shell-script-with-arguments-using-java-processbuilder#answer-18241604) If any of this info is bad, hopefully someone else will chime in. I’m still new around here ☺ Thanks, Bryan Rosander From: Huagen peng <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Friday, June 3, 2016 at 12:38 PM To: "[email protected]" <[email protected]> Subject: Piping commands in ExecuteStreamCommand/ExecuteProcess Hi, I was trying to get the ExecuteStreamCommand processor to execute one command and then pipe the result right to another Linux command in the same processor. This is how I try to configure the processor: Property Value Command Arguments -r;${log_dir};|;bzip2;-;>;/tmp/test.bz2 Command Path zcat The error I got is this: execution.command zcat execution.command.args -r;/data/similarweb/staging/170_16-05-22_01/log;|;bzip2;-;>;/tmp/test.bz2 execution.error zcat: can't stat: | (|.gz): No such file or directory zcat: can't stat: bzip2 (bzip2.gz): No such file or directory gzcat: unknown compression format gzcat: can't stat: > (>.gz): No such file or directory gzcat: can't stat: /tmp/test.bz2 (/tmp/test.bz2.gz): No such file or directory previous Empty string set I understand that I can execute the bzip2 command in a second ExecuteStreamCommand processor or simply use the CompressContent processor, and I event got them to work. The issue I was trying to avoid is to have a lot of data in the data flow clogging up the memory. What I wanted to achieve in my naive approach is to leverage the series of Linux commands do that job for me without bringing the huge amount of data into the multiple steps of flow. Can piping commands in the processor be done in NiFi? Thanks, Huagen
