Ronald, On 2/13/17 2:29 PM, DiFrango, Ronald wrote: > Yeah, I’m aware and agreed, but this is done in the Apache HTTP Client > Component’s Code, not ours and existed as-is with log4j2 in ALL the versions > we’ve been using. > > That said, I think this is a red herring and it’s still true that at least > for our application any version of log4j2 beyond 2.5 consumes more CPU that > 2.5 this is a REST based APi using Tomcat+Jersey+Spring [again nothing has > changed] and processing around 50 transactions per second with a guarantee of > a minimum of 1 log record to request. > > We just haven’t identified WHERE its causing the issue. > > One additional data point, we are using the bridging JAR’s for the following > as not all 3rd party libraries follow any standard on what they use: > > Classis log4j > SLF4J > commons-logging
You mentioned that lots of threads were being created and that they were
short-lived. What code is creating those threads? You never said...
-chris
> On 2/13/17, 2:11 PM, "Apache" <ralph.go...@dslextreme.com> wrote:
>
> The log.debug call will exit quickly, but if it is constructing a large
> message that will happen anyway unless one of the standard techniques (like
> wrapping in a logger.isDebugEnabled call) is used to avoid it. Constructing
> debug Strings can be expensive.
>
> Ralph
>
> > On Feb 13, 2017, at 10:58 AM, DiFrango, Ronald
> <ronald.difra...@capitalone.com> wrote:
> >
> > No we are not and since it’s a log.debug() it should just get dropped
> on the floor before any sort of copying occurs, right?
> >
> > Ron DiFrango
> >
> > On 2/13/17, 12:50 PM, "Matt Sicker" <boa...@gmail.com> wrote:
> >
> > Are you logging that info?
> >
> > On 13 February 2017 at 11:46, DiFrango, Ronald <
> > ronald.difra...@capitalone.com> wrote:
> >
> >> All,
> >>
> >> One thing we just noticed is that we are using Apache’s HTTP connection
> >> pooling for our downstream calls and in our latest performance run is
> that
> >> the READ operation on org.apache.http.impl.conn.LoggingInputStream
> seems
> >> to be taking a bulk of the time.
> >>
> >> If I look at the code, once the HTTP client reads the byte stream it
> >> issues a log.debug which could be a large payload, I wonder if that’s
> what
> >> is causing the issue?
> >>
> >> Please note aware, the version of httpclient we’ve been using has also
> >> been constant:
> >>
> >> <dependency>
> >> <groupId>org.apache.httpcomponents</groupId>
> >> <artifactId>httpclient</artifactId>
> >> <version>4.5.2</version>
> >> </dependency>
> >>
> >> Ron DiFrango
> >>
> >>
> >> On 2/13/17, 11:44 AM, "DiFrango, Ronald"
> <ronald.difra...@capitalone.com>
> >> wrote:
> >>
> >> This is running in Tomcat 8.0.33 in a Docker Container via AWS’s ECS
> >> which is identical to before with log4j2 2.5.
> >>
> >> We’ve run the application with Visual VM and the one thing for sure
> >> that we’ve seen is that in 2.6.2 it created tons of threads, something
> like
> >> 50+ but on 2.6 or 2.7 it was only 2. Now the threads were short
> lived, but
> >> they got created.
> >>
> >> We’re running another performance test today with Async loggers to
> see
> >> if that helps or exhibits the same thing though previous testing with
> Async
> >> had some of the same char
> >>
> >> Here’s our layout pattern:
> >>
> >> [%t] %d{DATE} %-5p %-15c{1} [%X]: %cm%n
> >>
> >> Please not the %cm is a custom message handler that we use to use to
> >> handle security filtering of the message payload aka we extend from
> >> LogEventPatternConverter.
> >>
> >> Thanks,
> >>
> >> Ron DiFrango
> >>
> >> On 2/13/17, 11:22 AM, "Matt Sicker" <boa...@gmail.com> wrote:
> >>
> >> What server environment are you running this in?
> >>
> >> On 13 February 2017 at 09:19, Remko Popma
> <remko.po...@gmail.com>
> >> wrote:
> >>
> >>> Ron,
> >>>
> >>> We haven't heard of any issues like you describe.
> >>> Have you tried running your application with Java Flight Recorder
> >>> <https://docs.oracle.com/javacomponents/jmc-5-4/jfr-
> >>> runtime-guide/run.htm#JFRUH176>?
> >>> This should help diagnose what is going on.
> >>>
> >>> Remko
> >>>
> >>>
> >>>
> >>> On Mon, Feb 13, 2017 at 9:59 PM, DiFrango, Ronald <
> >>> ronald.difra...@capitalone.com> wrote:
> >>>
> >>>> All,
> >>>>
> >>>> We recently upgrade to 2.6 and noticed a dramatic increase in
> >> CPU and
> >>>> Thread utilization that seems to be tied to the new “garbage
> >> free” mode
> >>> of
> >>>> log4j 2.6. Here’s some of the baseline numbers:
> >>>>
> >>>>
> >>>> · Log4j 2.5: CPU typically ran around 25%
> >>>>
> >>>> · Log4j 2.6: CPU typically ran around 75%
> >>>>
> >>>> · Log4j 2.6.2+: CPU typically ran around 100%
> >>>>
> >>>> · We’ve also tried turning off garbage free mode and
> >> that made
> >>>> things worse as the CPU was around 120% and caused us to not
> >> meet our
> >>> SLA’s
> >>>>
> >>>> It important to note that this is a REST Api using Jersey and
> >> typically
> >>>> responds in about in under 50ms on a per request so its high
> >> volume, but
> >>>> the logging level is WARN or higher except for our single
> >> performance log
> >>>> record which is written once per request using the lambda base
> >> approach.
> >>>>
> >>>> Our next test is going to be switching to all ASYNC loggers
> >> and see what
> >>>> effect that has, but I guess the general question is, has
> >> anyone else
> >>> seen
> >>>> this? Any thoughts?
> >>>>
> >>>> Ron DiFrango
> >>>> ________________________________________________________
> >>>>
> >>>> The information contained in this e-mail is confidential and/or
> >>>> proprietary to Capital One and/or its affiliates and may only
> >> be used
> >>>> solely in performance of work or services for Capital One. The
> >>> information
> >>>> transmitted herewith is intended only for use by the
> >> individual or entity
> >>>> to which it is addressed. If the reader of this message is not
> >> the
> >>> intended
> >>>> recipient, you are hereby notified that any review,
> >> retransmission,
> >>>> dissemination, distribution, copying or other use of, or
> >> taking of any
> >>>> action in reliance upon this information is strictly
> >> prohibited. If you
> >>>> have received this communication in error, please contact the
> >> sender and
> >>>> delete the material from your computer.
> >>>>
> >>>
> >>
> >>
> >>
> >> --
> >> Matt Sicker <boa...@gmail.com>
> >>
> >>
> >> ________________________________________________________
> >>
> >> The information contained in this e-mail is confidential and/or
> >> proprietary to Capital One and/or its affiliates and may only be used
> >> solely in performance of work or services for Capital One. The
> information
> >> transmitted herewith is intended only for use by the individual or
> entity
> >> to which it is addressed. If the reader of this message is not the
> intended
> >> recipient, you are hereby notified that any review, retransmission,
> >> dissemination, distribution, copying or other use of, or taking of any
> >> action in reliance upon this information is strictly prohibited. If you
> >> have received this communication in error, please contact the sender
> and
> >> delete the material from your computer.
> >>
> >>
> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: log4j-user-unsubscr...@logging.apache.org
> >> For additional commands, e-mail: log4j-user-h...@logging.apache.org
> >>
> >>
> >>
> >> ________________________________________________________
> >>
> >> The information contained in this e-mail is confidential and/or
> >> proprietary to Capital One and/or its affiliates and may only be used
> >> solely in performance of work or services for Capital One. The
> information
> >> transmitted herewith is intended only for use by the individual or
> entity
> >> to which it is addressed. If the reader of this message is not the
> intended
> >> recipient, you are hereby notified that any review, retransmission,
> >> dissemination, distribution, copying or other use of, or taking of any
> >> action in reliance upon this information is strictly prohibited. If you
> >> have received this communication in error, please contact the sender
> and
> >> delete the material from your computer.
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: log4j-user-unsubscr...@logging.apache.org
> >> For additional commands, e-mail: log4j-user-h...@logging.apache.org
> >>
> >>
> >
> >
> > --
> > Matt Sicker <boa...@gmail.com>
> >
> >
> > ________________________________________________________
> >
> > The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used solely
> in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity to
> which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any action
> in reliance upon this information is strictly prohibited. If you have
> received this communication in error, please contact the sender and delete
> the material from your computer.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: log4j-user-unsubscr...@logging.apache.org
> > For additional commands, e-mail: log4j-user-h...@logging.apache.org
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: log4j-user-unsubscr...@logging.apache.org
> For additional commands, e-mail: log4j-user-h...@logging.apache.org
>
>
>
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or proprietary
> to Capital One and/or its affiliates and may only be used solely in
> performance of work or services for Capital One. The information transmitted
> herewith is intended only for use by the individual or entity to which it is
> addressed. If the reader of this message is not the intended recipient, you
> are hereby notified that any review, retransmission, dissemination,
> distribution, copying or other use of, or taking of any action in reliance
> upon this information is strictly prohibited. If you have received this
> communication in error, please contact the sender and delete the material
> from your computer.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: log4j-user-unsubscr...@logging.apache.org
> For additional commands, e-mail: log4j-user-h...@logging.apache.org
>
signature.asc
Description: OpenPGP digital signature
