Re: Trouble configuring logging

[Kevin Doran]

 Yeah, if it helps at all, this is how we work around this in the Apache
Dockerfile (I knew realize why this method was used rather than a standard
console appender in logback):

# Continuously provide logs so that 'docker logs' can produce them
"${NIFI_HOME}/bin/nifi.sh" run &
nifi_pid="$!"
tail -F --pid=${nifi_pid} "${NIFI_HOME}/logs/nifi-app.log" &

https://github.com/apache/nifi/blob/main/nifi-docker/dockerhub/sh/start.sh#L126-L129


Something to look into. As more NiFi deployments become containerized, this
may be an area for improvement over time.

Thanks,
Kevin

On Sep 28, 2022 at 19:40:35, Dylan Klomparens 
wrote:

> Mark and Kevin, thank you for your insightful comments. That information
> allowed me to piece together the puzzle. Indeed, anything that is sent to
> standard out subsequently *causes* the log message to come from the the
> org.apache.nifi.StdOut logger in the RunNiFi class. Your description
> allowed me to find the exact line of code that does this
> 
> .
>
> From my perspective this is an unfortunate design choice for NiFi because
> it does not allow standard out to be redirected easily. (For example,
> flowing onward to Docker, and forwarded to AWS CloudWatch using Docker's
> built-in log driver, in my case). I suppose I'll have to find an
> alternative logback appender class to output the logs to, then find a way
> to forward them on from there.
>
> Thanks again for the additional information that put the picture into view.
> --
> *From:* Mark Payne 
> *Sent:* Wednesday, September 28, 2022 10:14 AM
> *To:* users 
> *Subject:* Re: Trouble configuring logging
>
>
> [EXTERNAL]
> This is because of how NiFi is run. When you startup nifi (bin/nifi.sh
> start) it doesn’t directly start the NiFi process.
> Instead, it starts a different processor, which is called RunNiFi. This
> RunNiFi process is responsible for doing a lot of things, including
> monitoring the nifi process and if it dies restarting it.
> Anything written to NiFi’s Standard Out goes to this processor, which then
> logs it.
> So you’d probably need to update the logging for the appender writing to
> the bootstrap file:
>  class="ch.qos.logback.core.rolling.RollingFileAppender”>
>
> And redirect that to standard out
>
> Thanks
> -Mark
>
>
> On Sep 28, 2022, at 9:48 AM, Kevin Doran  wrote:
>
> Dylan - I looked into this and am yet unable to offer an explaination.
> Perhaps others that are familiar with how org.apache.nifi.StdOut can shed
> some light, or else I will keep digging when I have a block of time. To
> help in my understanding: Which Docker image are you using? Is it the
> apace/nifi image or a custom one, and if custom, can you share the
> Dockerfile?
>
> Thanks,
> Kevin
>
> On Sep 27, 2022 at 10:21:12, Dylan Klomparens 
> wrote:
>
> I am attempting to configure logging for NiFi. I have NiFi running in a
> Docker container, which sends all console logs to AWS CloudWatch.
> Therefore, I am configuring NiFi to send all logs to the console.
>
> The problem is, for some reason *all log messages are coming from the
> org.apache.nifi.StdOut logger*. I cannot figure out why, since I would
> like messages to be printed directly from the logger that is receiving them.
>
> It seems like messages are "passing through" loggers, which are ultimately
> printed out from the org.apache.nifi.StdOut logger. Here is an example of
> *one* log message:
> *2022-09-27 10:08:01,849 INFO [NiFi logging handler]
> org.apache.nifi.StdOut* *2022-09-27 10:08:01,848 INFO [pool-6-thread-1]
> o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile
> Repository*
>
> *Why would every single log message come from the StdOut logger? And how
> can I have logs delivered from the logger they're supposedly originally
> coming from?*
>
> My logback.xml configuration is below for reference.
>
> 
> 
> 
>
> 
> true
> 
>
> 
> 
> %date %level [%thread] %logger{40} %msg%n
> 
> 
>
> 
> 
>  level="INFO"/>
>  level="INFO"/>
>  name="org.apache.nifi.controller.repository.StandardProcessSession"
> level="WARN" />
>
> 
> 
>  level="ERROR" />
>  level="ERROR" />
> 
> 
>  level="ERROR" />
>  level="ERROR" />
>
> 
>
>  level="OFF" />
> 
>
> 
> 
>
> 
> 
>
> 
> 
>
> 
> 
>
> 
> 
>
> 
>  level="ERROR"/>
>
> 
> 
> 
>
> 
> 
>
> 
> 
>
> 
> 
>
> 
>
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>
> 
> 
>
> 
> 
> 
>
> 
>
>
>

Re: Trouble configuring logging

[Dylan Klomparens]

Mark and Kevin, thank you for your insightful comments. That information 
allowed me to piece together the puzzle. Indeed, anything that is sent to 
standard out subsequently causes the log message to come from the the 
org.apache.nifi.StdOut logger in the RunNiFi class. Your description allowed me 
to find the exact line of code that does 
this.

From my perspective this is an unfortunate design choice for NiFi because it 
does not allow standard out to be redirected easily. (For example, flowing 
onward to Docker, and forwarded to AWS CloudWatch using Docker's built-in log 
driver, in my case). I suppose I'll have to find an alternative logback 
appender class to output the logs to, then find a way to forward them on from 
there.

Thanks again for the additional information that put the picture into view.

From: Mark Payne 
Sent: Wednesday, September 28, 2022 10:14 AM
To: users 
Subject: Re: Trouble configuring logging


[EXTERNAL]

This is because of how NiFi is run. When you startup nifi (bin/nifi.sh start) 
it doesn’t directly start the NiFi process.
Instead, it starts a different processor, which is called RunNiFi. This RunNiFi 
process is responsible for doing a lot of things, including monitoring the nifi 
process and if it dies restarting it.
Anything written to NiFi’s Standard Out goes to this processor, which then logs 
it.
So you’d probably need to update the logging for the appender writing to the 
bootstrap file:
mailto:kdo...@apache.org>> wrote:

Dylan - I looked into this and am yet unable to offer an explaination. Perhaps 
others that are familiar with how org.apache.nifi.StdOut can shed some light, 
or else I will keep digging when I have a block of time. To help in my 
understanding: Which Docker image are you using? Is it the apace/nifi image or 
a custom one, and if custom, can you share the Dockerfile?

Thanks,
Kevin

On Sep 27, 2022 at 10:21:12, Dylan Klomparens 
mailto:dklompar...@foodallergy.org>> wrote:
I am attempting to configure logging for NiFi. I have NiFi running in a Docker 
container, which sends all console logs to AWS CloudWatch. Therefore, I am 
configuring NiFi to send all logs to the console.

The problem is, for some reason all log messages are coming from the 
org.apache.nifi.StdOut logger. I cannot figure out why, since I would like 
messages to be printed directly from the logger that is receiving them.

It seems like messages are "passing through" loggers, which are ultimately 
printed out from the org.apache.nifi.StdOut logger. Here is an example of one 
log message:
2022-09-27 10:08:01,849 INFO [NiFi logging handler] org.apache.nifi.StdOut 
2022-09-27 10:08:01,848 INFO [pool-6-thread-1] 
o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile 
Repository

Why would every single log message come from the StdOut logger? And how can I 
have logs delivered from the logger they're supposedly originally coming from?

My logback.xml configuration is below for reference.



  

  
true
  

  

  %date %level [%thread] %logger{40} %msg%n

  

  
  
  
  
  

  
  
  
  
  
  
  
  

  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  
  

  
  

  
  

  
  

  

  
  
  
  
  
  
  
  
  
  
  

  
  

  

  

Re: Trouble configuring logging

[Lars Winderling]

Hi Dylan,

although it's not related to your question I would like to point out that your 
docker image (unless squashed to a single layer) will include even layers for 
deleted files afaik. If you put all your commands into a single RUN directive, 
you might save some space.

Best, Lars

On 28 September 2022 16:13:48 CEST, Dylan Klomparens 
 wrote:
>​Kevin, thank you for taking a look. I am using a custom Dockerfile (and 
>docker-compose.yaml) I wrote, pasted at the end of this email.
>
>I also ran NiFi natively, without Docker, on my desktop computer and observed 
>identical results. I initially suspected Docker could somehow be affecting 
>stdout somehow, but in testing I've not found any evidence of that.
>
>
>Dockerfile
>FROM openjdk:11
>
># These are environment variables that are not meant to be
># configurable, they are essential to the operation of the container.
>ENV NIFI_HOME /opt/nifi/
>ENV NIFI_LOG_DIR /persistent-storage/logs
>ENV NIFI_OVERRIDE_NIFIENV true
>
>WORKDIR /opt
>
># Unpack NiFi and Toolkit
>COPY nifi.zip /opt
>COPY nifi-toolkit.zip /opt
>RUN unzip nifi.zip
>RUN unzip nifi-toolkit.zip
>RUN mv nifi-1.17.0 nifi
>RUN mv nifi-toolkit-1.17.0 nifi-toolkit
>
># Clean out unused files
>RUN rm --force nifi.tar.gz nifi-toolkit.tar.gz
>RUN rm --force /opt/nifi/bin/*.bat
>RUN rm --force /opt/nifi/conf/*
>RUN rm --force /opt/nifi-toolkit/*.bat
>
>RUN mkdir /persistent-storage
>
>WORKDIR /opt/nifi
>
># Set configuration
>COPY bootstrap.conf bootstrap-notification-services.xml authorizers.xml 
>state-management.xml logback.xml /opt/nifi/conf/
>COPY initialize.sh /opt/nifi/bin/
>
>COPY postgres-jdbc-driver.jar /opt
>COPY snowflake-jdbc-driver.jar /opt
>COPY ZIP_codes_to_states.csv /opt
>
>CMD ["/opt/nifi/bin/nifi.sh", "run"]
>EXPOSE 8000/tcp
>
>
>docker-compose.yaml
>services:
>  nifi:
>container_name: nifi
>build: .
>image: nifi
>volumes: [/persistent-storage/:/persistent-storage/]
>ports: [8000:8000]
>environment: [INITIAL_ADMIN_IDENTITY:'*redacted*']
>logging:
>driver: "awslogs"
>options:
>awslogs-region: "*redacted*"
>awslogs-group: "*redacted*"
>awslogs-stream: "All logs"
>
>
>
>
>
>From: Kevin Doran 
>Sent: Wednesday, September 28, 2022 9:48 AM
>To: users@nifi.apache.org 
>Subject: Re: Trouble configuring logging
>
>
>[EXTERNAL]
>
>Dylan - I looked into this and am yet unable to offer an explaination. Perhaps 
>others that are familiar with how org.apache.nifi.StdOut can shed some light, 
>or else I will keep digging when I have a block of time. To help in my 
>understanding: Which Docker image are you using? Is it the apace/nifi image or 
>a custom one, and if custom, can you share the Dockerfile?
>
>Thanks,
>Kevin
>
>On Sep 27, 2022 at 10:21:12, Dylan Klomparens 
>mailto:dklompar...@foodallergy.org>> wrote:
>I am attempting to configure logging for NiFi. I have NiFi running in a Docker 
>container, which sends all console logs to AWS CloudWatch. Therefore, I am 
>configuring NiFi to send all logs to the console.
>
>The problem is, for some reason all log messages are coming from the 
>org.apache.nifi.StdOut logger. I cannot figure out why, since I would like 
>messages to be printed directly from the logger that is receiving them.
>
>It seems like messages are "passing through" loggers, which are ultimately 
>printed out from the org.apache.nifi.StdOut logger. Here is an example of one 
>log message:
>2022-09-27 10:08:01,849 INFO [NiFi logging handler] org.apache.nifi.StdOut 
>2022-09-27 10:08:01,848 INFO [pool-6-thread-1] 
>o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile 
>Repository
>
>Why would every single log message come from the StdOut logger? And how can I 
>have logs delivered from the logger they're supposedly originally coming from?
>
>My logback.xml configuration is below for reference.
>
>
>
>  
>
>  class="ch.qos.logback.classic.jul.LevelChangePropagator">
>true
>  
>
>  
>class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
>  %date %level [%thread] %logger{40} %msg%n
>
>  
>
>  
>  
>  level="INFO"/>
>  level="INFO"/>
>  name="org.apache.nifi.controller.repository.StandardProcessSession" 
>level="WARN" />
>
>  
>  
>  level="ERROR" />
>  level="ERROR" />
>  
>  
>  level="ERROR" />
>  level="ERROR" />
>
>  
>
>  name="org.apache.curator.framework.recipes.leader.LeaderSelector" level="OFF" 
>/>
>  
>
>  
>  
>
>  
>  
>
>  
>  
>
>  
>  
>
>  
>  
>
>  
>  level="ERROR"/>
>
>  
>  
>  
>
>  
>  
>
>  
>  
>
>  
>  
>
>  
>
>  
>  
>  
>  
>  
>  
>  
>  
>  
>  
>  
>
>  
>  
>
>  
>
>  
>
>

Re: Trouble configuring logging

[Mark Payne]

This is because of how NiFi is run. When you startup nifi (bin/nifi.sh start) 
it doesn’t directly start the NiFi process.
Instead, it starts a different processor, which is called RunNiFi. This RunNiFi 
process is responsible for doing a lot of things, including monitoring the nifi 
process and if it dies restarting it.
Anything written to NiFi’s Standard Out goes to this processor, which then logs 
it.
So you’d probably need to update the logging for the appender writing to the 
bootstrap file:
mailto:kdo...@apache.org>> wrote:

Dylan - I looked into this and am yet unable to offer an explaination. Perhaps 
others that are familiar with how org.apache.nifi.StdOut can shed some light, 
or else I will keep digging when I have a block of time. To help in my 
understanding: Which Docker image are you using? Is it the apace/nifi image or 
a custom one, and if custom, can you share the Dockerfile?

Thanks,
Kevin

On Sep 27, 2022 at 10:21:12, Dylan Klomparens 
mailto:dklompar...@foodallergy.org>> wrote:
I am attempting to configure logging for NiFi. I have NiFi running in a Docker 
container, which sends all console logs to AWS CloudWatch. Therefore, I am 
configuring NiFi to send all logs to the console.

The problem is, for some reason all log messages are coming from the 
org.apache.nifi.StdOut logger. I cannot figure out why, since I would like 
messages to be printed directly from the logger that is receiving them.

It seems like messages are "passing through" loggers, which are ultimately 
printed out from the org.apache.nifi.StdOut logger. Here is an example of one 
log message:
2022-09-27 10:08:01,849 INFO [NiFi logging handler] org.apache.nifi.StdOut 
2022-09-27 10:08:01,848 INFO [pool-6-thread-1] 
o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile 
Repository

Why would every single log message come from the StdOut logger? And how can I 
have logs delivered from the logger they're supposedly originally coming from?

My logback.xml configuration is below for reference.



  

  
true
  

  

  %date %level [%thread] %logger{40} %msg%n

  

  
  
  
  
  

  
  
  
  
  
  
  
  

  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  
  

  
  

  
  

  
  

  

  
  
  
  
  
  
  
  
  
  
  

  
  

  

  

Re: Trouble configuring logging

[Dylan Klomparens]

​Kevin, thank you for taking a look. I am using a custom Dockerfile (and 
docker-compose.yaml) I wrote, pasted at the end of this email.

I also ran NiFi natively, without Docker, on my desktop computer and observed 
identical results. I initially suspected Docker could somehow be affecting 
stdout somehow, but in testing I've not found any evidence of that.


Dockerfile
FROM openjdk:11

# These are environment variables that are not meant to be
# configurable, they are essential to the operation of the container.
ENV NIFI_HOME /opt/nifi/
ENV NIFI_LOG_DIR /persistent-storage/logs
ENV NIFI_OVERRIDE_NIFIENV true

WORKDIR /opt

# Unpack NiFi and Toolkit
COPY nifi.zip /opt
COPY nifi-toolkit.zip /opt
RUN unzip nifi.zip
RUN unzip nifi-toolkit.zip
RUN mv nifi-1.17.0 nifi
RUN mv nifi-toolkit-1.17.0 nifi-toolkit

# Clean out unused files
RUN rm --force nifi.tar.gz nifi-toolkit.tar.gz
RUN rm --force /opt/nifi/bin/*.bat
RUN rm --force /opt/nifi/conf/*
RUN rm --force /opt/nifi-toolkit/*.bat

RUN mkdir /persistent-storage

WORKDIR /opt/nifi

# Set configuration
COPY bootstrap.conf bootstrap-notification-services.xml authorizers.xml 
state-management.xml logback.xml /opt/nifi/conf/
COPY initialize.sh /opt/nifi/bin/

COPY postgres-jdbc-driver.jar /opt
COPY snowflake-jdbc-driver.jar /opt
COPY ZIP_codes_to_states.csv /opt

CMD ["/opt/nifi/bin/nifi.sh", "run"]
EXPOSE 8000/tcp


docker-compose.yaml
services:
  nifi:
container_name: nifi
build: .
image: nifi
volumes: [/persistent-storage/:/persistent-storage/]
ports: [8000:8000]
environment: [INITIAL_ADMIN_IDENTITY:'*redacted*']
logging:
driver: "awslogs"
options:
awslogs-region: "*redacted*"
awslogs-group: "*redacted*"
awslogs-stream: "All logs"





From: Kevin Doran 
Sent: Wednesday, September 28, 2022 9:48 AM
To: users@nifi.apache.org 
Subject: Re: Trouble configuring logging


[EXTERNAL]

Dylan - I looked into this and am yet unable to offer an explaination. Perhaps 
others that are familiar with how org.apache.nifi.StdOut can shed some light, 
or else I will keep digging when I have a block of time. To help in my 
understanding: Which Docker image are you using? Is it the apace/nifi image or 
a custom one, and if custom, can you share the Dockerfile?

Thanks,
Kevin

On Sep 27, 2022 at 10:21:12, Dylan Klomparens 
mailto:dklompar...@foodallergy.org>> wrote:
I am attempting to configure logging for NiFi. I have NiFi running in a Docker 
container, which sends all console logs to AWS CloudWatch. Therefore, I am 
configuring NiFi to send all logs to the console.

The problem is, for some reason all log messages are coming from the 
org.apache.nifi.StdOut logger. I cannot figure out why, since I would like 
messages to be printed directly from the logger that is receiving them.

It seems like messages are "passing through" loggers, which are ultimately 
printed out from the org.apache.nifi.StdOut logger. Here is an example of one 
log message:
2022-09-27 10:08:01,849 INFO [NiFi logging handler] org.apache.nifi.StdOut 
2022-09-27 10:08:01,848 INFO [pool-6-thread-1] 
o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile 
Repository

Why would every single log message come from the StdOut logger? And how can I 
have logs delivered from the logger they're supposedly originally coming from?

My logback.xml configuration is below for reference.



  

  
true
  

  

  %date %level [%thread] %logger{40} %msg%n

  

  
  
  
  
  

  
  
  
  
  
  
  
  

  

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  
  

  
  

  
  

  
  

  

  
  
  
  
  
  
  
  
  
  
  

  
  

  

  

Re: Trouble configuring logging

[Kevin Doran]

 Dylan - I looked into this and am yet unable to offer an explaination.
Perhaps others that are familiar with how org.apache.nifi.StdOut can shed
some light, or else I will keep digging when I have a block of time. To
help in my understanding: Which Docker image are you using? Is it the
apace/nifi image or a custom one, and if custom, can you share the
Dockerfile?

Thanks,
Kevin

On Sep 27, 2022 at 10:21:12, Dylan Klomparens 
wrote:

> I am attempting to configure logging for NiFi. I have NiFi running in a
> Docker container, which sends all console logs to AWS CloudWatch.
> Therefore, I am configuring NiFi to send all logs to the console.
>
> The problem is, for some reason *all log messages are coming from the
> org.apache.nifi.StdOut logger*. I cannot figure out why, since I would
> like messages to be printed directly from the logger that is receiving them.
>
> It seems like messages are "passing through" loggers, which are ultimately
> printed out from the org.apache.nifi.StdOut logger. Here is an example of
> *one* log message:
> *2022-09-27 10:08:01,849 INFO [NiFi logging handler]
> org.apache.nifi.StdOut* *2022-09-27 10:08:01,848 INFO [pool-6-thread-1]
> o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile
> Repository*
>
> *Why would every single log message come from the StdOut logger? And how
> can I have logs delivered from the logger they're supposedly originally
> coming from?*
>
> My logback.xml configuration is below for reference.
>
> 
> 
> 
>
> 
> true
> 
>
> 
> 
> %date %level [%thread] %logger{40} %msg%n
> 
> 
>
> 
> 
>  level="INFO"/>
>  level="INFO"/>
>  name="org.apache.nifi.controller.repository.StandardProcessSession"
> level="WARN" />
>
> 
> 
>  level="ERROR" />
>  level="ERROR" />
> 
> 
>  level="ERROR" />
>  level="ERROR" />
>
> 
>
>  level="OFF" />
> 
>
> 
> 
>
> 
> 
>
> 
> 
>
> 
> 
>
> 
> 
>
> 
>  level="ERROR"/>
>
> 
> 
> 
>
> 
> 
>
> 
> 
>
> 
> 
>
> 
>
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>
> 
> 
>
> 
> 
> 
>
> 
>

Re: Can ExecuteStreamCommand do this?

[James McMahon]

Thank you Steve. I 've employed a ListFile/FetchFile to load the 7z files
into the flow . When I have my ESC configured like this following, I get my
unpacked files results to the #{unpacked.destination} directory on disk:
Command Arguments
x;${filename};-spf;-o#{unpacked.destination};-aou
Command Path/bin/7a
Ignore STDIN   true
Working Directory#{unpacked.destination}
Argument Delimiter   ;
Output Destination Attribute  No value set
I get twelve files in my output destination folder.

When I try this one, get an error and no output:
Command Argumentsx;${filename};-si;-so;-spf;-aou
Command Path/bin/7a
Ignore STDIN   false
Working Directory#{unpacked.destination}
Argument Delimiter   ;
Output Destination Attribute  No value set

This yields this error...
Executable command /bin/7za ended in an error: ERROR: Can not open the file
as archive
E_NOTIMPL
...and it yields only one flowfile result in Output Stream, and that is a
brief text/plain report of the results of the 7za extraction like this:

This indicates it did indeed find my 7z file and it did indeed identify the
12 files in it, yet still I get no output to my outgoing flow path:
Extracting archive: /parent/subparent/testArchive.7z
- -
Path = /parentdir/subdir/testArchive.7z
Type = 7z
Physical Size = 7204
Headers Size = 298
Method = LZMA2:96k
Solid = +
Blocks = 1

Everything is Ok

Folders: 1
Files: 12
Size: 90238
Compressed: 7204

${filename} in both cases is a fully qualified name to the file, like this:
/dir/subdir/myTestFile.7z.

I can't seem to get the ESC output stream to be the extracted files.
Anything jump out at you?

On Wed, Sep 28, 2022 at 8:06 AM stephen.hindmarch.bt.com via users <
users@nifi.apache.org> wrote:

> Hi James,
>
>
>
> I am not in a position to test this right now, but you have to think of
> the flowfile content as STDIN and STDOUT. So with 7zip you need to use the
> “-si” and “-so” flags to ensure there are no files involved. Then if you
> can load the content of a file into a flowfile, eg with GetFile, then you
> should be able to unpack it with ExecuteStreamCommand. Set “Ignore STDIN” =
> “false”.
>
>
>
> I have written up my own use case on github. This involves having a Redis
> script as the input, and results of the script as the output.
>
>
>
> my-nifi-cluster/experiment-redis_direct.md at main ·
> hindmasj/my-nifi-cluster · GitHub
> 
>
>
>
> The first part of the post shows how to do it with the input commands on
> the command line, so a bit like you running “7za ${filename} -so”. The
> second part has the script inside the flowfile and is treated as STDIN, a
> bit like you doing “unzip -si -so”.
>
>
>
> See if that helps. Fundamentally, if you do “7za -si -so < myfile.7z” on
> the command line and see the output on the console, ExecuteStreamCommand
> will behave the same.
>
>
>
> *Steve Hindmarch*
>
> *From:* James McMahon 
> *Sent:* 28 September 2022 12:02
> *To:* users@nifi.apache.org
> *Subject:* Can ExecuteStreamCommand do this?
>
>
>
> I continue to struggle with ExecuteStreamCommand, and am hoping one of you
> from our user community can help me with the following:
>
> 1. Can ExecuteStreamCommand be used as I am trying to use it?
>
> 2. Can you direct me to an example where ExecuteStreamCommand is
> configured to do something similar to my use case?
>
>
>
> My use case:
>
> The incoming flowfiles in my flow path are 7z zips. Based on what I've
> researched so far, NiFi's native processors don't handle unpacking of 7z
> files.
>
>
>
> I want to read the 7z files as STDIN to ExecuteStreamCommand.
>
> I'd like the processor to call out to a 7za app, which will unpack the 7z.
>
> One incoming flowfile will yield multiple output files. Let's say twelve
> in this case.
>
> My goal is to output those twelve as new flowfiles out of
> ExecuteStreamCommand, to its output stream path.
>
>
>
> I can't yet get this to work. Best I've been able to do is configure
> ExecuteStreamCommand to unpack ${filename} to a temporary output directory
> on disk. Then I have another path in my flow polling that directory every
> few minutes looking for new data. Am hoping to eliminate that intermediate
> write/read to/from disk by keeping this all within the flow and JVM memory.
>
>
>
> Thanks very much in advance for any assistance.
>

RE: Can ExecuteStreamCommand do this?

[stephen.hindmarch.bt.com via users]

Hi James,

I am not in a position to test this right now, but you have to think of the 
flowfile content as STDIN and STDOUT. So with 7zip you need to use the “-si” 
and “-so” flags to ensure there are no files involved. Then if you can load the 
content of a file into a flowfile, eg with GetFile, then you should be able to 
unpack it with ExecuteStreamCommand. Set “Ignore STDIN” = “false”.

I have written up my own use case on github. This involves having a Redis 
script as the input, and results of the script as the output.

my-nifi-cluster/experiment-redis_direct.md at main · hindmasj/my-nifi-cluster · 
GitHub

The first part of the post shows how to do it with the input commands on the 
command line, so a bit like you running “7za ${filename} -so”. The second part 
has the script inside the flowfile and is treated as STDIN, a bit like you 
doing “unzip -si -so”.

See if that helps. Fundamentally, if you do “7za -si -so < myfile.7z” on the 
command line and see the output on the console, ExecuteStreamCommand will 
behave the same.

Steve Hindmarch
From: James McMahon 
Sent: 28 September 2022 12:02
To: users@nifi.apache.org
Subject: Can ExecuteStreamCommand do this?

I continue to struggle with ExecuteStreamCommand, and am hoping one of you from 
our user community can help me with the following:
1. Can ExecuteStreamCommand be used as I am trying to use it?
2. Can you direct me to an example where ExecuteStreamCommand is configured to 
do something similar to my use case?

My use case:
The incoming flowfiles in my flow path are 7z zips. Based on what I've 
researched so far, NiFi's native processors don't handle unpacking of 7z files.

I want to read the 7z files as STDIN to ExecuteStreamCommand.
I'd like the processor to call out to a 7za app, which will unpack the 7z.
One incoming flowfile will yield multiple output files. Let's say twelve in 
this case.
My goal is to output those twelve as new flowfiles out of ExecuteStreamCommand, 
to its output stream path.

I can't yet get this to work. Best I've been able to do is configure 
ExecuteStreamCommand to unpack ${filename} to a temporary output directory on 
disk. Then I have another path in my flow polling that directory every few 
minutes looking for new data. Am hoping to eliminate that intermediate 
write/read to/from disk by keeping this all within the flow and JVM memory.

Thanks very much in advance for any assistance.

Can ExecuteStreamCommand do this?

[James McMahon]

I continue to struggle with ExecuteStreamCommand, and am hoping one of you
from our user community can help me with the following:
1. Can ExecuteStreamCommand be used as I am trying to use it?
2. Can you direct me to an example where ExecuteStreamCommand is configured
to do something similar to my use case?

My use case:
The incoming flowfiles in my flow path are 7z zips. Based on what I've
researched so far, NiFi's native processors don't handle unpacking of 7z
files.

I want to read the 7z files as STDIN to ExecuteStreamCommand.
I'd like the processor to call out to a 7za app, which will unpack the 7z.
One incoming flowfile will yield multiple output files. Let's say twelve in
this case.
My goal is to output those twelve as new flowfiles out of
ExecuteStreamCommand, to its output stream path.

I can't yet get this to work. Best I've been able to do is configure
ExecuteStreamCommand to unpack ${filename} to a temporary output directory
on disk. Then I have another path in my flow polling that directory every
few minutes looking for new data. Am hoping to eliminate that intermediate
write/read to/from disk by keeping this all within the flow and JVM memory.

Thanks very much in advance for any assistance.