[
https://issues.apache.org/jira/browse/SOLR-7887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16414870#comment-16414870
]
Shawn Heisey commented on SOLR-7887:
------------------------------------
Here's some data for the discussion about what file size to use to rotation:
I used "tail" to give me the last 4194304 bytes of a log, and then the last
33554432 bytes of the same log. It is the primary online server for our
installation, serving as a shard aggregator for an index that spans two
servers, with seven shards.
The query rate for the last fifteen minutes is 0.73 per second on one handler,
and 0.06 per second on another handler. The load balancer sends pings pretty
frequently. I did not check the query rate on the handler that the ping
handler uses for its queries. It is a separate handler so that those queries
do not pollute the statistics for the "real" handlers.
The 4MB log covered times from 18:51:26.552 to 18:57:58.139, about six and a
half minutes. The 32MB log covered times from 18:01:23.375 to 18:58:13.886,
which is almost an hour.
Keep in mind that the query rate on this system is VERY VERY low. Users that
are handling 20-100 queries per second on each server are going to have a LOT
more data logged in a given time period.
My queries tend to be REALLY large. One of them that I just looked at had
nearly 4000 bytes of URL parameters, and I have seen some approaching 20000
bytes. When a query like this happens, it typically gets logged four times by
each server. The online primary system with the aggregating core has three
shards, plus has to log once at the end for the aggregating core, and the other
system has four shards.
In my logging config, I set log4j.appender.file.MaxFileSize to 2GB. Which is a
lot larger than I think Solr should use by default, but not outrageous for a
really busy server. I make it that big so that I don't have to look through a
lot of logs to find info for a query that somebody had a problem with yesterday.
It would be really nice if the size could be configured in solr.in.sh.
On the subject of gzipping the rotated logs: I like log rotation systems that
take care of compressing old logs for me. How about we do that change in
master? A new major version is a great time for a change like that.
> Upgrade Solr to use log4j2 -- log4j 1 now officially end of life
> ----------------------------------------------------------------
>
> Key: SOLR-7887
> URL: https://issues.apache.org/jira/browse/SOLR-7887
> Project: Solr
> Issue Type: Task
> Reporter: Shawn Heisey
> Assignee: Erick Erickson
> Priority: Major
> Fix For: 7.4
>
> Attachments: SOLR-7887-WIP.patch, SOLR-7887-eoe-review.patch,
> SOLR-7887-eoe-review.patch, SOLR-7887-followup_1.patch, SOLR-7887.patch,
> SOLR-7887.patch, SOLR-7887.patch, SOLR-7887.patch, SOLR-7887.patch,
> SOLR-7887.patch, SOLR-7887.patch, SOLR-7887.patch, SOLR-7887.patch,
> SOLR-7887.patch, SOLR-7887.patch, SOLR-7887.patch, SOLR-7887.patch,
> SOLR-7887_followup_2.patch, SOLR-7887_followup_2.patch
>
>
> The logging services project has officially announced the EOL of log4j 1:
> https://blogs.apache.org/foundation/entry/apache_logging_services_project_announces
> In the official binary jetty deployment, we use use log4j 1.2 as our final
> logging destination, so the admin UI has a log watcher that actually uses
> log4j and java.util.logging classes. That will need to be extended to add
> log4j2. I think that might be the largest pain point to this upgrade.
> There is some crossover between log4j2 and slf4j. Figuring out exactly which
> jars need to be in the lib/ext directory will take some research.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
