Liang Xie created LOGBACK-1512:
----------------------------------
Summary: let SCHEDULED_EXECUTOR_POOL_SIZE be configurable
Key: LOGBACK-1512
URL: https://jira.qos.ch/browse/LOGBACK-1512
Project: logback
Issue Type: Improvement
Components: logback-classic, logback-core
Affects Versions: 1.3.0-alpha5, 1.2.3, 1.1.11
Environment: AWS m5.xlarge ec2 instance, vCPU num: 4
Reporter: Liang Xie
Assignee: Logback dev list
Attachments: cpu_spike.png, latency.png
Our log servers deployed logback component, the logback configuration contains
tens of appenders, this is one of the appender definition:
{quote}<appender name="rawdata"
class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>/data/xxx/rawdata/rawdata.log</file>
<encoder>
<pattern>%m%n</pattern>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>/data/xxx/rawdata/rawdata.log.%d\{yyyy-MM-dd-HH}.gz</fileNamePattern>
</rollingPolicy>
</appender>
{quote}
please note the "fileNamePattern" value, we use gzip compression, it means
in each hour beginning, all the tens of files are renamed to tmp files, then
be compressed in the thread pool. Currently, this thread pool core size is
immutable, a fixed value: 8, which came from
[https://github.com/qos-ch/logback/commit/b946551439134]
In an small ec2 env with tens of big log files(a few of our rolling raw log
files size are 10 Gb), say 4 vcore, the compression will be last several
minutes with all vcores running in full speed.
It will disturb the normal user request latency espencially under a high
throughput env.
Our proposed solution are as following:
1. let SCHEDULED_EXECUTOR_POOL_SIZE be configurable, it is meaningful since
the logback users have different cpu core configs, different
requirements(latency vs throughput)
2. let the compression level be configurable, currently, the logback uses
JDK's GZIPOutputStream class, which deflate algorithm is DEFAULT_COMPRESSION
always. It would be great if we could make it be configurable.
We forked a branch, after updating SCHEDULED_EXECUTOR_POOL_SIZE to 1 and
deflate algorithm to BEST_SPEED, we have observed the each hour beginning's
cpu spikes were relieved , and the user request latency is more stable than
before
Any comments are welcome
--
This message was sent by Atlassian JIRA
(v7.3.1#73012)
_______________________________________________
logback-dev mailing list
logback-dev@qos.ch
http://mailman.qos.ch/mailman/listinfo/logback-dev
