[Wikidata-bugs] [Maniphest] [Commented On] T178721: WDQS tests unstable with some thread leak errors
Smalyshev added a comment. I think the leaks come from combination of RandomizedTesting, which insists on policing threads (not sure why randomization should come with thread policing?), some classes we use in tests using thread pools (Metrics has its own pools, Http clients too, also Zookeeper) and even if we meticulously shut them down, some of them may be doing it asynchronously which means by the time RandomizedTesting gets to check threads, sometimes shutdown has not been completed yet. I think ensuring we do not have thread leaks is important, but putting it into RandomizedTesting is probably a wrong place to check it. @Gehel, do you know any better way? Absent that, I would just turn off the thread police in RandomizedTesting (or migrate to some other solution that doesn't mix concerns) and ensure threads not leaking in some other way that works better with tests.TASK DETAILhttps://phabricator.wikimedia.org/T178721EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Gehel, SmalyshevCc: Gehel, Aklapper, Smalyshev, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Avner, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T178721: WDQS tests unstable with some thread leak errors
gerritbot added a comment. Change 401705 merged by jenkins-bot: [wikidata/query/rdf@master] Properly close HC client in WikibaseRepository https://gerrit.wikimedia.org/r/401705TASK DETAILhttps://phabricator.wikimedia.org/T178721EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Gehel, gerritbotCc: Gehel, Aklapper, Smalyshev, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, Avner, Lewizho99, Maathavan, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T178721: WDQS tests unstable with some thread leak errors
gerritbot added a comment. Change 401705 had a related patch set uploaded (by Gehel; owner: Guillaume Lederrey): [wikidata/query/rdf@master] Properly close HC client in WikibaseRepository https://gerrit.wikimedia.org/r/401705TASK DETAILhttps://phabricator.wikimedia.org/T178721EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Gehel, gerritbotCc: Gehel, Aklapper, Smalyshev, Lahi, Gq86, Darkminds3113, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Avner, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T178721: WDQS tests unstable with some thread leak errors
gerritbot added a comment. Change 396380 merged by jenkins-bot: [wikidata/query/rdf@master] Extract RdfRepositoryForTesting to a JUnit @Rule https://gerrit.wikimedia.org/r/396380TASK DETAILhttps://phabricator.wikimedia.org/T178721EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gerritbotCc: gerritbot, Gehel, Aklapper, Smalyshev, Lahi, Gq86, Baloch007, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, merbst, Avner, Lewizho99, Maathavan, Jonas, FloNight, Xmlizer, NavinoEvans, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T178721: WDQS tests unstable with some thread leak errors
gerritbot added a comment. Change 396380 had a related patch set uploaded (by Gehel; owner: Gehel): [wikidata/query/rdf@master] Extract RdfRepositoryForTesting to a JUnit @Rule https://gerrit.wikimedia.org/r/396380TASK DETAILhttps://phabricator.wikimedia.org/T178721EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gerritbotCc: gerritbot, Gehel, Aklapper, Smalyshev, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, Avner, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T178721: WDQS tests unstable with some thread leak errors
Smalyshev added a comment. Also, the problem unfortunately is not very well reproducible, it comes and goes, though it shows up often enough to be annoying. But of course, once I start debugging it locally, it vanishes.TASK DETAILhttps://phabricator.wikimedia.org/T178721EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Gehel, Aklapper, Smalyshev, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, Avner, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T178721: WDQS tests unstable with some thread leak errors
Smalyshev added a comment. Blazegraph uses Jetty and I think I tried to use Apache with it and there were some problems. So I use Jetty to communicate to Blazegraph.TASK DETAILhttps://phabricator.wikimedia.org/T178721EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Gehel, Aklapper, Smalyshev, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, Avner, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T178721: WDQS tests unstable with some thread leak errors
Gehel added a comment. No idea what's going on here. I have seen the same error a few times as well. It looks like the leaked thread is an HttpClient, so my guess is that in some cases RdfRepository.httpClient isn't closed. But I can't understand in which code path this could happen. Looking deeper into HttpClient itself, it looks like HttpClient.executor is shutdown correctly (as a LifeCycle bean). The executor itself is a QueuedThreadPool, which has some non trivial logic to poison the threads and wait for completion. Increasing log level on org.eclipse.jetty.util.thread might give us some insight on what is happening there. Side note, I'm not a big fan of Jetty's HTTP client. I think that Apache HC has a simpler interface (maybe it's just me being more used to it). Side note 2: The test classes hierarchy is a bit more complex than I'd like it to be. Implementing RdfRepositoryForTesting as a Rule instead of a parent-defined property would IMHO make the code easier to read, and maybe easier to debug.TASK DETAILhttps://phabricator.wikimedia.org/T178721EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: GehelCc: Gehel, Aklapper, Smalyshev, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, Avner, debt, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T178721: WDQS tests unstable with some thread leak errors
Smalyshev added a comment. @Gehel, do you have any thoughts on this?TASK DETAILhttps://phabricator.wikimedia.org/T178721EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: Gehel, Aklapper, Smalyshev, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, Avner, debt, Jonas, FloNight, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
