Re: SOLR: Unnecessary logging

[Gus Heck]

> There's already LOG4J_PROPS so if you have a favorite log4j config you
> want to use, just set that env variable and forget about it, does that
> work?
>
>
That sounds great, now if only it were in the docs :)
https://lucene.apache.org/solr/guide/7_5/configuring-logging.html

I'll add it later if nobody beats me to it.

-Gus

-- 
http://www.the111shift.com

Re: SOLR: Unnecessary logging

[Shawn Heisey]

On 11/27/2018 7:29 AM, Gus Heck wrote:
Startup logging that happens just on startup should err on the side of 
verbosity.


I think this is a good policy.

As I said before, I really do applaud the efforts to shrink the logs and 
eliminate cruft.  But I don't think that the motivation and goal should 
just be "short logs".  It should be about ensuring that the only thing 
being logged by default is information that most users will find to be 
relevant, and moving other logs to less severe logging levels.


The default logging should provide information that helps out in 
something I like to think of as "typical troubleshooting". Startup and 
reload logging can be a fairly verbose.  Unless the user is doing 
something very odd, that information is not likely to be logged frequently.


In some cases, achieving the goal I've outlined might mean *ADDING* some 
logging to the default level, to provide information about unexpected 
situations.


The logging that typically happens during normal operation (queries and 
updates, mostly) only really gets verbose if the server is busy.  I 
don't know that there's much cruft in that logging, but we can review it 
just to make sure.  Giving the user the option to push request logging 
into a separate logfile would do a lot to reduce the size of the main 
log.  Troubleshooting is sometimes more difficult when logs are 
separated, but if that becomes an issue the user can always reconfigure 
to put it all back in solr.log.


Thanks,
Shawn


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: SOLR: Unnecessary logging

[Walter Underwood]

I’m not a big fan of console/file magic. If that is done, there needs to be a 
big WARN at the beginning that says some log messages are suppressed because a 
console has been detected.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Nov 27, 2018, at 6:00 PM, David Smiley  wrote:
> 
> Maybe we can have it both ways: might the distinction be done between console 
> & files?  So for example keep the console more brief, but have the log files 
> contain more logs?  Just an idea.
> 
> On Tue, Nov 27, 2018 at 8:07 AM Erick Erickson  > wrote:
> bq: Maybe the real thing to do since everyone's preferences vary is to
> have a --log-config start script option that points solr at one's
> favorite dev/testing/demo oriented logging config
> 
> There's already LOG4J_PROPS so if you have a favorite log4j config you
> want to use, just set that env variable and forget about it, does that
> work?
> 
> I spent a fair amount of time untangling the multiple log config files
> down to two, I would be loathe to add any more config files to the
> release. We could also expand the one we _do_ release for with
> comments for various options...
> 
> Best,
> Erick
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org 
> 
> For additional commands, e-mail: dev-h...@lucene.apache.org 
> 
> 
> -- 
> Lucene/Solr Search Committer (PMC), Developer, Author, Speaker
> LinkedIn: http://linkedin.com/in/davidwsmiley 
>  | Book: 
> http://www.solrenterprisesearchserver.com 
> 

Re: SOLR: Unnecessary logging

[David Smiley]

Maybe we can have it both ways: might the distinction be done between
console & files?  So for example keep the console more brief, but have the
log files contain more logs?  Just an idea.

On Tue, Nov 27, 2018 at 8:07 AM Erick Erickson 
wrote:

> bq: Maybe the real thing to do since everyone's preferences vary is to
> have a --log-config start script option that points solr at one's
> favorite dev/testing/demo oriented logging config
>
> There's already LOG4J_PROPS so if you have a favorite log4j config you
> want to use, just set that env variable and forget about it, does that
> work?
>
> I spent a fair amount of time untangling the multiple log config files
> down to two, I would be loathe to add any more config files to the
> release. We could also expand the one we _do_ release for with
> comments for various options...
>
> Best,
> Erick
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
> --
Lucene/Solr Search Committer (PMC), Developer, Author, Speaker
LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
http://www.solrenterprisesearchserver.com

Re: SOLR: Unnecessary logging

[Erick Erickson]

bq: Maybe the real thing to do since everyone's preferences vary is to
have a --log-config start script option that points solr at one's
favorite dev/testing/demo oriented logging config

There's already LOG4J_PROPS so if you have a favorite log4j config you
want to use, just set that env variable and forget about it, does that
work?

I spent a fair amount of time untangling the multiple log config files
down to two, I would be loathe to add any more config files to the
release. We could also expand the one we _do_ release for with
comments for various options...

Best,
Erick

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: SOLR: Unnecessary logging

[Gus Heck]

Startup logging that happens just on startup should err on the side of
verbosity. I like the ulimit checks even if they are sometimes irrelevant
on my dev systems, the lib listing also sounds useful. Of course truly
useful things like the jetty jsp message should be eliminated. Default
logging generally should be verbose, and tuning it down when needed is just
part of tuning the system imho. Once an event has passed un-logged, it's
gone and the clue to what happened is missing and the developer has to
spend time to even know what log to turn on, if it's even reproducible. If
we do anything, it ought to be some sort of logging profiles (beyond -q and
-v I suppose), but it seems to me we have lots of configurability there
already https://lucene.apache.org/solr/guide/7_5/configuring-logging.html.

Maybe the real thing to do since everyone's preferences vary is to have a
--log-config start script option that points solr at one's favorite
dev/testing/demo oriented logging config. This would also be nice for cases
where you want the logging config to live outside of the install where
someone can easily play with it.

-Gus

On Wed, Nov 21, 2018 at 5:50 PM Uwe Schindler  wrote:

> No it won't deduplicate. It is actually worse: Every class would be a
> different class instance per core, just with same name (they won't even
> compare equals to each other). This why you have classloader, so you can
> have same class name in different parts of app. OSGI... The expressions
> module does the same. Every compiled Lucene expression has the same class
> name, just in a different classloader.
>
> So every core has its own copy of all classes. That's a common problem for
> users with many cores each of those having Tika configured per core only...
>
> Uwe
>
> Am November 21, 2018 10:23:29 PM UTC schrieb Shawn Heisey <
> apa...@elyograg.org>:
>>
>> On 11/21/2018 1:40 AM, Jan Høydahl wrote:
>>
>>> It is super easy to change the log level of one particular class or
>>> package
>>> selectively, that's why we have log4j2.xml and also the Admin's
>>> Logging levels menu.
>>> The occational need to see all of the 150 jar files loaded on startup
>>> should not be
>>> solved by spewing out that for first-time users of Solr who really
>>> just care for whether
>>> the server started OK or not.
>>>
>>
>> If somebody's got a config that loads 150 jars, I really do think that
>> having that info in the log would be useful information.  That's a LOT
>> of jars, and it could be vital for troubleshooting a classloader problem
>> with one jar -- need to be able to tell whether it has been loaded or not.
>>
>> It definitely shouldn't be default to have Jetty log its jar loading.  I
>> just want to know how to make it do so, for situations that require very
>> in-depth study.
>>
>> A user who's got 20 cores and each of them is loading the same 10 jars
>> with  config ... IMHO that would ALSO be useful information to be
>> able to see in the default log.  Without it, the user will probably have
>> absolutely no idea that they have 200 jar loads.  (when that happens,
>> does Java realize that the classloader is loading the same jars and
>> eliminate duplicates in memory?)
>>
>> Thanks,
>> Shawn
>> --
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>>
> --
> Uwe Schindler
> Achterdiek 19, 28357 Bremen
> https://www.thetaphi.de
>


-- 
http://www.the111shift.com

Re: SOLR: Unnecessary logging

[Uwe Schindler]

No it won't deduplicate. It is actually worse: Every class would be a different 
class instance per core, just with same name (they won't even compare equals to 
each other). This why you have classloader, so you can have same class name in 
different parts of app. OSGI... The expressions module does the same. Every 
compiled Lucene expression has the same class name, just in a different 
classloader.

So every core has its own copy of all classes. That's a common problem for 
users with many cores each of those having Tika configured per core only...

Uwe

Am November 21, 2018 10:23:29 PM UTC schrieb Shawn Heisey :
>On 11/21/2018 1:40 AM, Jan Høydahl wrote:
>> It is super easy to change the log level of one particular class or 
>> package
>> selectively, that's why we have log4j2.xml and also the Admin's 
>> Logging levels menu.
>> The occational need to see all of the 150 jar files loaded on startup
>
>> should not be
>> solved by spewing out that for first-time users of Solr who really 
>> just care for whether
>> the server started OK or not.
>
>If somebody's got a config that loads 150 jars, I really do think that 
>having that info in the log would be useful information.  That's a LOT 
>of jars, and it could be vital for troubleshooting a classloader
>problem 
>with one jar -- need to be able to tell whether it has been loaded or
>not.
>
>It definitely shouldn't be default to have Jetty log its jar loading. 
>I 
>just want to know how to make it do so, for situations that require
>very 
>in-depth study.
>
>A user who's got 20 cores and each of them is loading the same 10 jars 
>with  config ... IMHO that would ALSO be useful information to be 
>able to see in the default log.  Without it, the user will probably
>have 
>absolutely no idea that they have 200 jar loads.  (when that happens, 
>does Java realize that the classloader is loading the same jars and 
>eliminate duplicates in memory?)
>
>Thanks,
>Shawn
>
>
>-
>To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>For additional commands, e-mail: dev-h...@lucene.apache.org

--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de

Re: SOLR: Unnecessary logging

[Shawn Heisey]

On 11/21/2018 1:40 AM, Jan Høydahl wrote:
It is super easy to change the log level of one particular class or 
package
selectively, that's why we have log4j2.xml and also the Admin's 
Logging levels menu.
The occational need to see all of the 150 jar files loaded on startup 
should not be
solved by spewing out that for first-time users of Solr who really 
just care for whether

the server started OK or not.


If somebody's got a config that loads 150 jars, I really do think that 
having that info in the log would be useful information.  That's a LOT 
of jars, and it could be vital for troubleshooting a classloader problem 
with one jar -- need to be able to tell whether it has been loaded or not.


It definitely shouldn't be default to have Jetty log its jar loading.  I 
just want to know how to make it do so, for situations that require very 
in-depth study.


A user who's got 20 cores and each of them is loading the same 10 jars 
with  config ... IMHO that would ALSO be useful information to be 
able to see in the default log.  Without it, the user will probably have 
absolutely no idea that they have 200 jar loads.  (when that happens, 
does Java realize that the classloader is loading the same jars and 
eliminate duplicates in memory?)


Thanks,
Shawn


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: SOLR: Unnecessary logging

[Erick Erickson]

bq. t is super easy to change the log level of one particular class or package

If anyone has prod logs laying around collected at INFO level and could
provide any counts of what packages the most frequent log messages come
from, please attach the data to SOLR-11934. I've seen some cases where
just changing the log levels for a single class from INFO to  WARN reduced
the log volume tremendously (90% or  so). That was an app that had  a
high update volume of a single doc at a time so doesn't reflect other
scenarios.

I'll be glad to analyze any shareable log files, but understand that sharing
log files often violates corporate policy.

My suspicion is that there are just a few classes that could be configured
at WARN level that would reduce the volume of log messages greatly.

That's really a separate question from what level we log messages at,
much of the logging in the code is specified at levels suitable for
troubleshooting rather than an ops perspective. But that's a discussion
for whenever I/we get to SOLR-11934. I think configuring  some
classes to WARN by default or just providing them commented-out
in the log4j2.xml files would be an easy win.

**

bq. We can find a way to distinguish between dev/prod
usage and only warn about file descriptors, suspiciously low heap,
lack of SSL etc if
Solr is bound to a public IP address.

Don't want to bikeshed this too much, so take this for what it's worth.

Seeing it for the first time when you deploy to prod is too late IMO. I
suppose we're looking through two different lenses. We bounce around
from client to client, so my lens is how often people new to
Solr and/or operations folks who get the finished app from development
miss things like this. And then spend time in panic mode because their
prod system broke. And how long we've spent on the phone with them
before remembering to check this (it's part of a standard checklist
now of course, but still)... And how it tarnishes their opinion of Solr.

I'd be less worried about this if we could throw errors that pointed to
this issue, but the errors that come out when ulimits are exceeded
are obscure and varied.

Sure, the same problem can occur if the devs set the variable
in solr.in.sh then forget about it and don't tell ops to bump these
limits, but at least some of the time the problem will  be avoided with
the message front and center.

Given the cost to devs is to set a single system var I think it's a price
worth paying. All IMO and reflecting my experience of course.

Best,
Erick
On Wed, Nov 21, 2018 at 12:40 AM Jan Høydahl  wrote:
>
> bq. I don't want to see file limit warnings on a dev laptop
>
> "If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to
> false in your profile or solr.in.sh"
>
> Not having these limits set properly has caused weird errors for a lot
> of people, but I agree they are annoying at times so there's a way to
> shut them off.
>
>
> Note my wording - "on a dev laptop". We can find a way to distinguish between 
> dev/prod
> usage and only warn about file descriptors, suspiciously low heap, lack of 
> SSL etc if
> Solr is bound to a public IP address.
>
> The only logging config files you should have to worry about are in
> ./solr/server/resources.
>
> But yeah, there are several Solr JIRAs lying around that I'll break
> some time loose _sometime_ to work on  to try to make logging more
> rational. Unfortunately as Shawn says, there's always tension. When
> running "normally", I pretty much only want to see WARN messages. Then
> every time I want to diagnose something, I want to see _everything_.
> And telling the user to change their logging and call me again next
> time the problem happens is not very satisfying for anyone.
>
>
> It is super easy to change the log level of one particular class or package
> selectively, that's why we have log4j2.xml and also the Admin's Logging 
> levels menu.
> The occational need to see all of the 150 jar files loaded on startup should 
> not be
> solved by spewing out that for first-time users of Solr who really just care 
> for whether
> the server started OK or not.
>
> Agree that adding commented-out snippets in the logging config could be a
> good way to quickly let people debug "ClassNotFound issues", "Indexing 
> issues",
> "Faceting issues", "Security issues" etc. We could have pre-canned configs 
> that
> would upper the logging of classes of particular interest for each case.
>
> Jan

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: SOLR: Unnecessary logging

[Jan Høydahl]

> bq. I don't want to see file limit warnings on a dev laptop
> 
> "If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to
> false in your profile or solr.in.sh"
> 
> Not having these limits set properly has caused weird errors for a lot
> of people, but I agree they are annoying at times so there's a way to
> shut them off.

Note my wording - "on a dev laptop". We can find a way to distinguish between 
dev/prod
usage and only warn about file descriptors, suspiciously low heap, lack of SSL 
etc if
Solr is bound to a public IP address.

> The only logging config files you should have to worry about are in
> ./solr/server/resources.
> 
> But yeah, there are several Solr JIRAs lying around that I'll break
> some time loose _sometime_ to work on  to try to make logging more
> rational. Unfortunately as Shawn says, there's always tension. When
> running "normally", I pretty much only want to see WARN messages. Then
> every time I want to diagnose something, I want to see _everything_.
> And telling the user to change their logging and call me again next
> time the problem happens is not very satisfying for anyone.

It is super easy to change the log level of one particular class or package
selectively, that's why we have log4j2.xml and also the Admin's Logging levels 
menu.
The occational need to see all of the 150 jar files loaded on startup should 
not be
solved by spewing out that for first-time users of Solr who really just care 
for whether
the server started OK or not.

Agree that adding commented-out snippets in the logging config could be a
good way to quickly let people debug "ClassNotFound issues", "Indexing issues",
"Faceting issues", "Security issues" etc. We could have pre-canned configs that
would upper the logging of classes of particular interest for each case.

Jan

Re: SOLR: Unnecessary logging

[Erick Erickson]

bq. I don't want to see file limit warnings on a dev laptop

"If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to
false in your profile or solr.in.sh"

Not having these limits set properly has caused weird errors for a lot
of people, but I agree they are annoying at times so there's a way to
shut them off.

The only logging config files you should have to worry about are in
./solr/server/resources.

But yeah, there are several Solr JIRAs lying around that I'll break
some time loose _sometime_ to work on  to try to make logging more
rational. Unfortunately as Shawn says, there's always tension. When
running "normally", I pretty much only want to see WARN messages. Then
every time I want to diagnose something, I want to see _everything_.
And telling the user to change their logging and call me again next
time the problem happens is not very satisfying for anyone.

Best,
Erick
On Tue, Nov 20, 2018 at 6:40 PM Shawn Heisey  wrote:
>
> On 11/20/2018 5:34 PM, Alexandre Rafalovitch wrote:
> > How much of this controlled by log4j property file and we can include
> > usecases as commented out things to let people choose?
> >
> > Because we can set the logging very precisely without recompiling.
>
> If we can control it with log4j config, that would be very nice.
>
> A quick check of the SolrResourceLoader code suggests that if the
> logging level for that class were changed to DEBUG, it would nicely
> provide the info that I'm after, wouldn't log a huge amount of useless
> cruft, and wouldn't even require a code change.  Maybe we could put this
> config in log4j2.xml, commented out, as an option that the end user
> could enable if they wish.
>
> Another solution would be to create a special logger, along the lines of
> the ones in SolrCore that log requests and slow requests, and have log4j
> config for that special logger.  And that's not necessarily a bad idea,
> but I think the idea mentioned above is good enough, and a lot easier.
>
> Thanks,
> Shawn
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: SOLR: Unnecessary logging

[Shawn Heisey]

On 11/20/2018 5:34 PM, Alexandre Rafalovitch wrote:
How much of this controlled by log4j property file and we can include 
usecases as commented out things to let people choose?


Because we can set the logging very precisely without recompiling.


If we can control it with log4j config, that would be very nice.

A quick check of the SolrResourceLoader code suggests that if the 
logging level for that class were changed to DEBUG, it would nicely 
provide the info that I'm after, wouldn't log a huge amount of useless 
cruft, and wouldn't even require a code change.  Maybe we could put this 
config in log4j2.xml, commented out, as an option that the end user 
could enable if they wish.


Another solution would be to create a special logger, along the lines of 
the ones in SolrCore that log requests and slow requests, and have log4j 
config for that special logger.  And that's not necessarily a bad idea, 
but I think the idea mentioned above is good enough, and a lot easier.


Thanks,
Shawn


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: SOLR: Unnecessary logging

[Alexandre Rafalovitch]

How much of this controlled by log4j property file and we can include
usecases as commented out things to let people choose?

Because we can set the logging very precisely without recompiling.

Regards,
Alex

On Tue, Nov 20, 2018, 7:21 PM Shawn Heisey  On 11/20/2018 4:09 PM, Jan Høydahl wrote:
> > In Solr 6.3 (SOLR-6677) we reduced a lot of noisy logging in the default
> start/stop scenario. We ended up with a very compact set of log lines. I
> just realized that this is getting fatter for every release, and now a solr
> start -f looks like this on my laptop:
>
> 
>
> > Am I alone valuing a much more minimalistic output and hiding stuff that
> won't add value? After pruning away what I see no value in, here's what I
> end up with:
>
> 
>
> > That's from 31 to 12 lines. I don't want to see file limit warnings on a
> dev laptop, and Jetty's lack of JSP support is totally irrelevant :)
>
> I wonder what we need to do with Jetty to make that log line go away.
> It's thankfully only one line!  I worry about changing the log level for
> o.e.j.w.StandardDescriptorProcessor to WARN or ERROR ... I wonder if
> that might get rid of messages that we would actually want to see.  Have
> to ask the jetty mailing list.  I can do that, unless you'd really like
> to do it.
>
> > The line "SSLCredentialProviderFactory Processing SSL Credential
> Provider chain: env;sysprop" is also popping up even in non-SSL and for
> things like bin/solr status. Why?
>
> That does look like a good candidate for changing to DEBUG.
>
> > If you all agree I'll open a JIRA and solve some of this with
> INFO->DEBUG changes, and others in log4j2.xml (e.g. o.e.j.s.session -> WARN)
>
> I agree with the general direction of making the logs leaner.  But, if I
> may drag out the soapbox for a moment, there are some things that have
> been removed from default logs that I would like to be restored.  The
> one on my mind right now:  A list of every file (usually jars) loaded by
>  config elements or a file's presence in ${solr.solr.home}/lib or
> ${instanceDir}/lib.  I think that information adds a LOT of value to the
> logs.  I know that it can be a lot of lines, though.
>
> If consensus is to not log that by default, which I completely
> understand, then can we put a config element in solrconfig.xml that
> would instruct Solr to log files loaded after startup, at the INFO level
> instead of DEBUG?  I do not like the option of starting Solr with the -v
> flag -- this does get the info I'm after, but the logfile has way too
> much OTHER cruft that is not helpful.  Some of the tougher problems to
> diagnose on the user list include ClassNotFoundException, and that would
> be a lot easier if the logfile included a list of every jar loaded after
> the moment that Jetty starts Solr.  Having Jetty log every jar loaded
> *before* it starts Solr might also be useful, but I'll need to ask about
> that on the Jetty list.
>
> Having Solr log Java's classpath would be a welcome addition for the
> same reason, and that would only be one long line.
>
> Thanks,
> Shawn
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: SOLR: Unnecessary logging

[Shawn Heisey]

On 11/20/2018 4:09 PM, Jan Høydahl wrote:

In Solr 6.3 (SOLR-6677) we reduced a lot of noisy logging in the default 
start/stop scenario. We ended up with a very compact set of log lines. I just 
realized that this is getting fatter for every release, and now a solr start -f 
looks like this on my laptop:





Am I alone valuing a much more minimalistic output and hiding stuff that won't 
add value? After pruning away what I see no value in, here's what I end up with:





That's from 31 to 12 lines. I don't want to see file limit warnings on a dev 
laptop, and Jetty's lack of JSP support is totally irrelevant :)


I wonder what we need to do with Jetty to make that log line go away.  
It's thankfully only one line!  I worry about changing the log level for 
o.e.j.w.StandardDescriptorProcessor to WARN or ERROR ... I wonder if 
that might get rid of messages that we would actually want to see.  Have 
to ask the jetty mailing list.  I can do that, unless you'd really like 
to do it.



The line "SSLCredentialProviderFactory Processing SSL Credential Provider chain: 
env;sysprop" is also popping up even in non-SSL and for things like bin/solr status. 
Why?


That does look like a good candidate for changing to DEBUG.


If you all agree I'll open a JIRA and solve some of this with INFO->DEBUG changes, 
and others in log4j2.xml (e.g. o.e.j.s.session -> WARN)


I agree with the general direction of making the logs leaner.  But, if I 
may drag out the soapbox for a moment, there are some things that have 
been removed from default logs that I would like to be restored.  The 
one on my mind right now:  A list of every file (usually jars) loaded by 
 config elements or a file's presence in ${solr.solr.home}/lib or 
${instanceDir}/lib.  I think that information adds a LOT of value to the 
logs.  I know that it can be a lot of lines, though.


If consensus is to not log that by default, which I completely 
understand, then can we put a config element in solrconfig.xml that 
would instruct Solr to log files loaded after startup, at the INFO level 
instead of DEBUG?  I do not like the option of starting Solr with the -v 
flag -- this does get the info I'm after, but the logfile has way too 
much OTHER cruft that is not helpful.  Some of the tougher problems to 
diagnose on the user list include ClassNotFoundException, and that would 
be a lot easier if the logfile included a list of every jar loaded after 
the moment that Jetty starts Solr.  Having Jetty log every jar loaded 
*before* it starts Solr might also be useful, but I'll need to ask about 
that on the Jetty list.


Having Solr log Java's classpath would be a welcome addition for the 
same reason, and that would only be one long line.


Thanks,
Shawn


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

SOLR: Unnecessary logging

[Jan Høydahl]

In Solr 6.3 (SOLR-6677) we reduced a lot of noisy logging in the default 
start/stop scenario. We ended up with a very compact set of log lines. I just 
realized that this is getting fatter for every release, and now a solr start -f 
looks like this on my laptop:

$ solr start -f
*** [WARN] *** Your open file limit is currently 256.  
 It should be set to 65000 to avoid operational disruption. 
 If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in 
your profile or solr.in.sh
*** [WARN] ***  Your Max Processes Limit is currently 1418. 
 It should be set to 65000 to avoid operational disruption. 
 If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in 
your profile or solr.in.sh
2018-11-20 22:58:38.281 INFO  (main) [   ] o.e.j.u.log Logging initialized 
@817ms to org.eclipse.jetty.util.log.Slf4jLog
2018-11-20 22:58:38.443 INFO  (main) [   ] o.e.j.s.Server 
jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: 
d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_102-b14
2018-11-20 22:58:38.472 INFO  (main) [   ] o.e.j.d.p.ScanningAppProvider 
Deployment monitor 
[file:///Users/janhoy/Applications/solr-7.5.0/server/contexts/] at interval 0
2018-11-20 22:58:38.729 INFO  (main) [   ] o.e.j.w.StandardDescriptorProcessor 
NO JSP Support for /solr, did not find org.apache.jasper.servlet.JspServlet
2018-11-20 22:58:38.739 INFO  (main) [   ] o.e.j.s.session 
DefaultSessionIdManager workerName=node0
2018-11-20 22:58:38.740 INFO  (main) [   ] o.e.j.s.session No SessionScavenger 
set, using defaults
2018-11-20 22:58:38.741 INFO  (main) [   ] o.e.j.s.session node0 Scavenging 
every 66ms
2018-11-20 22:58:38.784 INFO  (main) [   ] 
o.a.s.u.c.SSLCredentialProviderFactory Processing SSL Credential Provider 
chain: env;sysprop
2018-11-20 22:58:38.829 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter Using 
logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2018-11-20 22:58:38.834 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter  ___  
_   Welcome to Apache Solr™ version 7.5.0
2018-11-20 22:58:38.834 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter / __| 
___| |_ _   Starting in standalone mode on port 8983
2018-11-20 22:58:38.834 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter \__ \/ _ 
\ | '_|  Install dir: /Users/janhoy/Applications/solr
2018-11-20 22:58:38.835 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter 
|___/\___/_|_|Start time: 2018-11-20T22:58:38.835Z
2018-11-20 22:58:38.860 INFO  (main) [   ] o.a.s.c.SolrResourceLoader Using 
system property solr.solr.home: /Users/janhoy/Applications/solr/server/solr
2018-11-20 22:58:38.869 INFO  (main) [   ] o.a.s.c.SolrXmlConfig Loading 
container configuration from 
/Users/janhoy/Applications/solr/server/solr/solr.xml
2018-11-20 22:58:38.929 INFO  (main) [   ] o.a.s.c.SolrXmlConfig MBean server 
found: com.sun.jmx.mbeanserver.JmxMBeanServer@58fdd99, but no JMX reporters 
were configured - adding default JMX reporter.
2018-11-20 22:58:40.069 INFO  (main) [   ] 
o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 
transient cores
2018-11-20 22:58:40.072 INFO  (main) [   ] o.a.s.h.a.MetricsHistoryHandler No 
.system collection, keeping metrics history in memory.
2018-11-20 22:58:40.138 INFO  (main) [   ] o.a.s.m.r.SolrJmxReporter JMX 
monitoring for 'solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@58fdd99
2018-11-20 22:58:40.138 INFO  (main) [   ] o.a.s.m.r.SolrJmxReporter JMX 
monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@58fdd99
2018-11-20 22:58:40.146 INFO  (main) [   ] o.a.s.m.r.SolrJmxReporter JMX 
monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@58fdd99
2018-11-20 22:58:40.186 INFO  (main) [   ] o.a.s.c.CorePropertiesLocator Found 
0 core definitions underneath /Users/janhoy/Applications/solr/server/solr
2018-11-20 22:58:40.241 INFO  (main) [   ] o.e.j.s.h.ContextHandler Started 
o.e.j.w.WebAppContext@291b4bf5{/solr,file:///Users/janhoy/Applications/solr-7.5.0/server/solr-webapp/webapp/,AVAILABLE}{/Users/janhoy/Applications/solr-7.5.0/server/solr-webapp/webapp}
2018-11-20 22:58:40.261 INFO  (main) [   ] o.e.j.s.AbstractConnector Started 
ServerConnector@24b6b8f6{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}
2018-11-20 22:58:40.262 INFO  (main) [   ] o.e.j.s.Server Started @2800ms

Am I alone valuing a much more minimalistic output and hiding stuff that won't 
add value? After pruning away what I see no value in, here's what I end up with:

2018-11-20 22:58:38.443 INFO  (main) [   ] o.e.j.s.Server 
jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: 
d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_102-b14
2018-11-20 22:58:38.829 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter Using 
logger factory org.apache.logging.slf4j.Log4jLoggerFactory
2018-11-20 22:58:38.834 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter  ___