Hi Dylan, although it's not related to your question I would like to point out that your docker image (unless squashed to a single layer) will include even layers for deleted files afaik. If you put all your commands into a single RUN directive, you might save some space.
Best, Lars On 28 September 2022 16:13:48 CEST, Dylan Klomparens <[email protected]> wrote: >Kevin, thank you for taking a look. I am using a custom Dockerfile (and >docker-compose.yaml) I wrote, pasted at the end of this email. > >I also ran NiFi natively, without Docker, on my desktop computer and observed >identical results. I initially suspected Docker could somehow be affecting >stdout somehow, but in testing I've not found any evidence of that. > > >Dockerfile >FROM openjdk:11 > ># These are environment variables that are not meant to be ># configurable, they are essential to the operation of the container. >ENV NIFI_HOME /opt/nifi/ >ENV NIFI_LOG_DIR /persistent-storage/logs >ENV NIFI_OVERRIDE_NIFIENV true > >WORKDIR /opt > ># Unpack NiFi and Toolkit >COPY nifi.zip /opt >COPY nifi-toolkit.zip /opt >RUN unzip nifi.zip >RUN unzip nifi-toolkit.zip >RUN mv nifi-1.17.0 nifi >RUN mv nifi-toolkit-1.17.0 nifi-toolkit > ># Clean out unused files >RUN rm --force nifi.tar.gz nifi-toolkit.tar.gz >RUN rm --force /opt/nifi/bin/*.bat >RUN rm --force /opt/nifi/conf/* >RUN rm --force /opt/nifi-toolkit/*.bat > >RUN mkdir /persistent-storage > >WORKDIR /opt/nifi > ># Set configuration >COPY bootstrap.conf bootstrap-notification-services.xml authorizers.xml >state-management.xml logback.xml /opt/nifi/conf/ >COPY initialize.sh /opt/nifi/bin/ > >COPY postgres-jdbc-driver.jar /opt >COPY snowflake-jdbc-driver.jar /opt >COPY ZIP_codes_to_states.csv /opt > >CMD ["/opt/nifi/bin/nifi.sh", "run"] >EXPOSE 8000/tcp > > >docker-compose.yaml >services: > nifi: > container_name: nifi > build: . > image: nifi > volumes: [/persistent-storage/:/persistent-storage/] > ports: [8000:8000] > environment: [INITIAL_ADMIN_IDENTITY:'*redacted*'] > logging: > driver: "awslogs" > options: > awslogs-region: "*redacted*" > awslogs-group: "*redacted*" > awslogs-stream: "All logs" > > > > >________________________________ >From: Kevin Doran <[email protected]> >Sent: Wednesday, September 28, 2022 9:48 AM >To: [email protected] <[email protected]> >Subject: Re: Trouble configuring logging > > >[EXTERNAL] > >Dylan - I looked into this and am yet unable to offer an explaination. Perhaps >others that are familiar with how org.apache.nifi.StdOut can shed some light, >or else I will keep digging when I have a block of time. To help in my >understanding: Which Docker image are you using? Is it the apace/nifi image or >a custom one, and if custom, can you share the Dockerfile? > >Thanks, >Kevin > >On Sep 27, 2022 at 10:21:12, Dylan Klomparens ><[email protected]<mailto:[email protected]>> wrote: >I am attempting to configure logging for NiFi. I have NiFi running in a Docker >container, which sends all console logs to AWS CloudWatch. Therefore, I am >configuring NiFi to send all logs to the console. > >The problem is, for some reason all log messages are coming from the >org.apache.nifi.StdOut logger. I cannot figure out why, since I would like >messages to be printed directly from the logger that is receiving them. > >It seems like messages are "passing through" loggers, which are ultimately >printed out from the org.apache.nifi.StdOut logger. Here is an example of one >log message: >2022-09-27 10:08:01,849 INFO [NiFi logging handler] org.apache.nifi.StdOut >2022-09-27 10:08:01,848 INFO [pool-6-thread-1] >o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile >Repository > >Why would every single log message come from the StdOut logger? And how can I >have logs delivered from the logger they're supposedly originally coming from? > >My logback.xml configuration is below for reference. > ><?xml version="1.0" encoding="UTF-8"?> ><configuration debug="true"> > <shutdownHook class="ch.qos.logback.core.hook.DelayingShutdownHook" /> > > <contextListener >class="ch.qos.logback.classic.jul.LevelChangePropagator"> > <resetJUL>true</resetJUL> > </contextListener> > > <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender"> > <encoder >class="ch.qos.logback.classic.encoder.PatternLayoutEncoder"> > <pattern>%date %level [%thread] %logger{40} %msg%n</pattern> > </encoder> > </appender> > > <logger name="org.apache.nifi" level="INFO"/> > <logger name="org.apache.nifi.processors" level="WARN"/> > <logger name="org.apache.nifi.processors.standard.LogAttribute" >level="INFO"/> > <logger name="org.apache.nifi.processors.standard.LogMessage" >level="INFO"/> > <logger >name="org.apache.nifi.controller.repository.StandardProcessSession" >level="WARN" /> > > <logger name="org.apache.zookeeper.ClientCnxn" level="ERROR" /> > <logger name="org.apache.zookeeper.server.NIOServerCnxn" level="ERROR" /> > <logger name="org.apache.zookeeper.server.NIOServerCnxnFactory" >level="ERROR" /> > <logger name="org.apache.zookeeper.server.NettyServerCnxnFactory" >level="ERROR" /> > <logger name="org.apache.zookeeper.server.quorum" level="ERROR" /> > <logger name="org.apache.zookeeper.ZooKeeper" level="ERROR" /> > <logger name="org.apache.zookeeper.server.PrepRequestProcessor" >level="ERROR" /> > <logger name="org.apache.nifi.controller.reporting.LogComponentStatuses" >level="ERROR" /> > > <logger name="org.apache.calcite.runtime.CalciteException" level="OFF" /> > > <logger >name="org.apache.curator.framework.recipes.leader.LeaderSelector" level="OFF" >/> > <logger name="org.apache.curator.ConnectionState" level="OFF" /> > > <!-- Logger for managing logging statements for nifi clusters. --> > <logger name="org.apache.nifi.cluster" level="INFO"/> > > <!-- Logger for logging HTTP requests received by the web server. --> > <logger name="org.apache.nifi.server.JettyServer" level="INFO"/> > > <!-- Logger for managing logging statements for jetty --> > <logger name="org.eclipse.jetty" level="INFO"/> > > <!-- Suppress non-error messages due to excessive logging by class or >library --> > <logger name="org.springframework" level="ERROR"/> > > <!-- Suppress non-error messages due to known warning about redundant >path annotation (NIFI-574) --> > <logger name="org.glassfish.jersey.internal.Errors" level="ERROR"/> > > <!-- Suppress non-error messages due to Jetty AnnotationParser emitting >a large amount of WARNS. Issue described in NIFI-5479. --> > <logger name="org.eclipse.jetty.annotations.AnnotationParser" >level="ERROR"/> > > <!-- Suppress non-error messages from SSHJ which was emitting large >amounts of INFO logs by default --> > <logger name="net.schmizz.sshj" level="WARN" /> > <logger name="com.hierynomus.sshj" level="WARN" /> > > <!-- Suppress non-error messages from SMBJ which was emitting large >amounts of INFO logs by default --> > <logger name="com.hierynomus.smbj" level="WARN" /> > > <!-- Suppress non-error messages from AWS KCL which was emitting large >amounts of INFO logs by default --> > <logger name="com.amazonaws.services.kinesis" level="WARN" /> > > <!-- Suppress non-error messages from Apache Atlas which was emitting >large amounts of INFO logs by default --> > <logger name="org.apache.atlas" level="WARN" /> > > <logger name="org.apache.nifi.web.security.requests" level="INFO" /> > > <!-- > Logger for capturing user events. We do not want to propagate these > log events to the root logger. These messages are only sent to the > user-log appender. > --> > <logger name="org.apache.nifi.web.security" level="INFO" /> > <logger name="org.apache.nifi.web.api.config" level="INFO" /> > <logger name="org.apache.nifi.authorization" level="INFO" /> > <logger name="org.apache.nifi.cluster.authorization" level="INFO" /> > <logger name="org.apache.nifi.web.api.AccessResource" level="INFO" /> > <logger name="org.springframework.security.saml.log" level="WARN" /> > <logger name="org.opensaml" level="WARN" /> > <logger name="org.apache.nifi.web.server.RequestLog" level="INFO" /> > <logger name="org.apache.nifi.bootstrap" level="INFO" /> > <logger name="org.apache.nifi.bootstrap.Command" level="INFO" /> > > <logger name="org.apache.nifi.StdOut" level="INFO" /> > <logger name="org.apache.nifi.StdErr" level="INFO" /> > > <root level="INFO"> > <appender-ref ref="CONSOLE" /> > </root> > ></configuration>
