[jira] [Commented] (SOLR-7215) non reproducible Suite failures due to excessive sysout due to HDFS lease renewal WARN logs due to connection refused -- even if test doesn't use HDFS (ie: threads leaki
[
https://issues.apache.org/jira/browse/SOLR-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16759194#comment-16759194
]
Kevin Risden commented on SOLR-7215:
Not sure what the status is here. I would guess this is either
a) not an issue any more (lots has changed with HDFS thread cleanup)
b) fixed later due to HDFS thread cleaup
c) still any issue but isn't clear that this has happened recently.
Planning to resolve this since I haven't seen this and last comment was 3+
years ago.
SOLR-9515 with Hadoop 3 upgrade was recent so trying to cleanup old HDFS
related JIRAs if it isn't clear they still happen.
> non reproducible Suite failures due to excessive sysout due to HDFS lease
> renewal WARN logs due to connection refused -- even if test doesn't use HDFS
> (ie: threads leaking between tests)
> --
>
> Key: SOLR-7215
> URL: https://issues.apache.org/jira/browse/SOLR-7215
> Project: Solr
> Issue Type: Bug
>Reporter: Hoss Man
>Priority: Major
> Attachments: tests-report.txt_suite-failure-due-to-sysout.txt.zip
>
>
> On my local machine, i've noticed lately a lot of sporadic, non reproducible,
> failures like these...
> {noformat}
> 2> NOTE: reproduce with: ant test -Dtestcase=ScriptEngineTest
> -Dtests.seed=E254A7E69EC7212A -Dtests.slow=true -Dtests.locale=sv
> -Dtests.timezone=SystemV/CST6 -Dtests.asserts=true -Dtests.file.encoding=UTF-8
> [14:34:23.749] ERROR 0.00s J1 | ScriptEngineTest (suite) <<<
>> Throwable #1: java.lang.AssertionError: The test or suite printed 10984
> bytes to stdout and stderr, even though the limit was set to 8192 bytes.
> Increase the limit with @Limit, ignore it completely with
> @SuppressSysoutChecks or run with -Dtests.verbose=true
>> at __randomizedtesting.SeedInfo.seed([E254A7E69EC7212A]:0)
>> at
> org.apache.lucene.util.TestRuleLimitSysouts.afterIfSuccessful(TestRuleLimitSysouts.java:212)
> {noformat}
> Invariably, looking at the logs of test that fail for this reason, i see
> multiple instances of these WARN msgs...
> {noformat}
> 2> 601361 T3064 oahh.LeaseRenewer.run WARN Failed to renew lease for
> [DFSClient_NONMAPREDUCE_-253604438_2947] for 92 seconds. Will retry shortly
> ... java.net.ConnectException: Call From frisbee/127.0.1.1 to localhost:40618
> failed on connection exception: java.net.ConnectException: Connection
> refused; For more details see:
> http://wiki.apache.org/hadoop/ConnectionRefused
> 2> at sun.reflect.GeneratedConstructorAccessor268.newInstance(Unknown
> Source)
> 2> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> ...
> {noformat}
> ...the full stack traces of these exceptions typically being 36 lines long
> (not counting the supressed "... 17 more" at the end)
> doing some basic crunching of the "tests-report.txt" file from a recent run
> of all "solr-core" tests (that caused the above failure) leads to some pretty
> damn disconcerting numbers...
> {noformat}
> hossman@frisbee:~/tmp$ wc -l tests-report.txt_suite-failure-due-to-sysout.txt
> 1049177 tests-report.txt_suite-failure-due-to-sysout.txt
> hossman@frisbee:~/tmp$ grep "Suite: org.apache.solr"
> tests-report.txt_suite-failure-due-to-sysout.txt | wc -l
> 465
> hossman@frisbee:~/tmp$ grep "LeaseRenewer.run WARN Failed to renew lease"
> tests-report.txt_suite-failure-due-to-sysout.txt | grep
> http://wiki.apache.org/hadoop/ConnectionRefused | wc -l
> 1988
> hossman@frisbee:~/tmp$ calc
> 1988 * 36
> 71568
> {noformat}
> So running 465 Solr test suites, we got ~2 thousand of these "Failed to renew
> lease" WARNings. Of the ~1 million total lines of log messages from all
> tests, ~70 thousand (~7%) are coming from these WARNing mesages -- which can
> evidently be safetly ignored?
> Something seems broken here.
> Someone who understands this area of the code should either:
> * investigate & fix the code/test not to have these lease renewal problems
> * tweak our test logging configs to supress these WARN messages
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7215) non reproducible Suite failures due to excessive sysout due to HDFS lease renewal WARN logs due to connection refused -- even if test doesn't use HDFS (ie: threads leaki
[
https://issues.apache.org/jira/browse/SOLR-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359227#comment-14359227
]
Dawid Weiss commented on SOLR-7215:
---
Uncomment the ThreadLeakFilters, Hoss. Nothing should get through.
SolrIgnoredThreadsFilter has way too many exclusions -- these have to be shut
down and cleaned properly, not ignored (leading to errors like this one):
{code}
/*
* IMPORTANT! IMPORTANT!
*
* Any threads added here should have ABSOLUTELY NO SIDE EFFECTS
* (should be stateless). This includes no references to cores or other
* test-dependent information.
*/
String threadName = t.getName();
if (threadName.equals(TimerThread.THREAD_NAME)) {
return true;
}
if (threadName.startsWith(facetExecutor-) ||
threadName.startsWith(cmdDistribExecutor-) ||
threadName.startsWith(httpShardExecutor-)) {
return true;
}
// This is a bug in ZooKeeper where they call System.exit(11) when
// this thread receives an interrupt signal.
if (threadName.startsWith(SyncThread)) {
return true;
}
// THESE ARE LIKELY BUGS - these threads should be closed!
if (threadName.startsWith(Overseer-) ||
threadName.startsWith(aliveCheckExecutor-) ||
threadName.startsWith(concurrentUpdateScheduler-)) {
return true;
}
return false;
{code}
non reproducible Suite failures due to excessive sysout due to HDFS lease
renewal WARN logs due to connection refused -- even if test doesn't use HDFS
(ie: threads leaking between tests)
--
Key: SOLR-7215
URL: https://issues.apache.org/jira/browse/SOLR-7215
Project: Solr
Issue Type: Bug
Reporter: Hoss Man
Attachments: tests-report.txt_suite-failure-due-to-sysout.txt.zip
On my local machine, i've noticed lately a lot of sporadic, non reproducible,
failures like these...
{noformat}
2 NOTE: reproduce with: ant test -Dtestcase=ScriptEngineTest
-Dtests.seed=E254A7E69EC7212A -Dtests.slow=true -Dtests.locale=sv
-Dtests.timezone=SystemV/CST6 -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[14:34:23.749] ERROR 0.00s J1 | ScriptEngineTest (suite)
Throwable #1: java.lang.AssertionError: The test or suite printed 10984
bytes to stdout and stderr, even though the limit was set to 8192 bytes.
Increase the limit with @Limit, ignore it completely with
@SuppressSysoutChecks or run with -Dtests.verbose=true
at __randomizedtesting.SeedInfo.seed([E254A7E69EC7212A]:0)
at
org.apache.lucene.util.TestRuleLimitSysouts.afterIfSuccessful(TestRuleLimitSysouts.java:212)
{noformat}
Invariably, looking at the logs of test that fail for this reason, i see
multiple instances of these WARN msgs...
{noformat}
2 601361 T3064 oahh.LeaseRenewer.run WARN Failed to renew lease for
[DFSClient_NONMAPREDUCE_-253604438_2947] for 92 seconds. Will retry shortly
... java.net.ConnectException: Call From frisbee/127.0.1.1 to localhost:40618
failed on connection exception: java.net.ConnectException: Connection
refused; For more details see:
http://wiki.apache.org/hadoop/ConnectionRefused
2 at sun.reflect.GeneratedConstructorAccessor268.newInstance(Unknown
Source)
2 at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
...
{noformat}
...the full stack traces of these exceptions typically being 36 lines long
(not counting the supressed ... 17 more at the end)
doing some basic crunching of the tests-report.txt file from a recent run
of all solr-core tests (that caused the above failure) leads to some pretty
damn disconcerting numbers...
{noformat}
hossman@frisbee:~/tmp$ wc -l tests-report.txt_suite-failure-due-to-sysout.txt
1049177 tests-report.txt_suite-failure-due-to-sysout.txt
hossman@frisbee:~/tmp$ grep Suite: org.apache.solr
tests-report.txt_suite-failure-due-to-sysout.txt | wc -l
465
hossman@frisbee:~/tmp$ grep LeaseRenewer.run WARN Failed to renew lease
tests-report.txt_suite-failure-due-to-sysout.txt | grep
http://wiki.apache.org/hadoop/ConnectionRefused | wc -l
1988
hossman@frisbee:~/tmp$ calc
1988 * 36
71568
{noformat}
So running 465 Solr test suites, we got ~2 thousand of these Failed to renew
lease WARNings. Of the ~1 million total lines of log messages from all
tests, ~70 thousand (~7%) are coming from these WARNing mesages -- which can
evidently be safetly ignored?
Something seems broken here.
Someone who understands this area of the code should either:
* investigate fix the code/test not to have these lease renewal problems
* tweak our test logging configs to supress
