[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

Gehel closed this task as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T293862 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, Gehel Cc: RKemper, bking, Aklapper, dcausse, Fernandobacasegua34, Astuthiodit_1, 786, Suran38,

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

gerritbot added a comment. Change 779831 **merged** by jenkins-bot: [operations/alerts@master] team-search-platform: remove BlazegraphJvmQuakeWarnGC https://gerrit.wikimedia.org/r/779831 TASK DETAIL https://phabricator.wikimedia.org/T293862 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

gerritbot added a comment. Change 779440 **merged** by Bking: [operations/puppet@production] wdqs: activate jvmquake at 300:5 https://gerrit.wikimedia.org/r/779440 TASK DETAIL https://phabricator.wikimedia.org/T293862 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

gerritbot added a comment. Change 779831 had a related patch set uploaded (by DCausse; author: DCausse): [operations/alerts@master] team-search-platform: remove BlazegraphJvmQuakeWarnGC https://gerrit.wikimedia.org/r/779831 TASK DETAIL https://phabricator.wikimedia.org/T293862

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T293862 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, gerritbot Cc: RKemper, bking, Aklapper, dcausse, Fernandobacasegua34, Astuthiodit_1, 786,

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

gerritbot added a comment. Change 779440 had a related patch set uploaded (by DCausse; author: DCausse): [operations/puppet@production] wdqs: activate jvmquake at 300:5 https://gerrit.wikimedia.org/r/779440 TASK DETAIL https://phabricator.wikimedia.org/T293862 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

RKemper removed a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T293862 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, RKemper Cc: RKemper, bking, Aklapper, dcausse, Astuthiodit_1, karapayneWMDE, Invadibot,

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

RKemper reopened this task as "In Progress". RKemper added a comment. Oops, I just meant to move on workboard, not sure how I closed it as well TASK DETAIL https://phabricator.wikimedia.org/T293862 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To:

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

RKemper closed this task as "Resolved". RKemper moved this task from Needs review to Waiting on the Discovery-Search (Current work) board. RKemper added a comment. Moving to `Waiting` while we see how the newest settings do TASK DETAIL https://phabricator.wikimedia.org/T293862 WORKBOARD

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2022-04-07T17:50:24Z] T293862 Removed touched files so that it'll be easier to see when the new jvmquake threshold is crossed: `ryankemper@cumin1001:~$ sudo -E cumin

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

RKemper added a comment. To check for presence of touched file: ryankemper@cumin1001:~$ sudo -E cumin 'A:wdqs-public' '[ -f "/tmp/wdqs_blazegraph_jvmquake_warn_gc" ] && echo yes || echo no' 11 hosts will be targeted:

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2022-04-07T17:44:11Z] T293862 Rolling restart of wdqs public is complete; new jvmquake settings have been uptaken on wdqs public hosts:

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2022-04-07T17:31:13Z] [WDQS] T293862 Need to do a rolling restart of wdqs public; going to just roll a full deploy since it's equal work TASK DETAIL

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

gerritbot added a comment. Change 776857 **merged** by Bking: [operations/puppet@production] wdqs: tune jvmquake settings (take 2) https://gerrit.wikimedia.org/r/776857 TASK DETAIL https://phabricator.wikimedia.org/T293862 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

gerritbot added a comment. Change 776857 had a related patch set uploaded (by DCausse; author: DCausse): [operations/puppet@production] wdqs: tune jvmquake settings (take 2) https://gerrit.wikimedia.org/r/776857 TASK DETAIL https://phabricator.wikimedia.org/T293862 EMAIL

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

dcausse added a comment. Actually wdqs2007, wdqs2004 and wdqs2003 also triggered jvmquake, GC activity increased and wdqs2007 & wdqs2003 were unresponsive for a couple minutes. For wdqs2004 there are no visible blips in the various graph. I guess we should relax the settings a bit more.

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

dcausse added a comment. With the settings we properly detected wdqs1006 going down for 30minutes at `2022-04-22T12:30:00` (this 2minutes after the first blip in the graph). Unfortunately there was a false positive wdqs1012 at `2022-04-22T10:00:00` as this machine was unavailable from 2

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

gerritbot added a comment. Change 775254 **merged** by Ryan Kemper: [operations/puppet@production] wdqs: tune jvmquake settings https://gerrit.wikimedia.org/r/775254 TASK DETAIL https://phabricator.wikimedia.org/T293862 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

gerritbot added a comment. Change 775254 had a related patch set uploaded (by Ryan Kemper; author: DCausse): [operations/puppet@production] wdqs: tune jvmquake settings https://gerrit.wikimedia.org/r/775254 TASK DETAIL https://phabricator.wikimedia.org/T293862 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

gerritbot added a comment. Change 773758 **merged** by jenkins-bot: [operations/alerts@master] team-search-platform: add jvmquake alerting https://gerrit.wikimedia.org/r/773758 TASK DETAIL https://phabricator.wikimedia.org/T293862 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T293862 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, gerritbot Cc: bking, Aklapper, dcausse, Fernandobacasegua34, Astuthiodit_1, 786, Suran38,

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

gerritbot added a comment. Change 773758 had a related patch set uploaded (by DCausse; author: DCausse): [operations/alerts@master] team-search-platform: add jvmquake alerting https://gerrit.wikimedia.org/r/773758 TASK DETAIL https://phabricator.wikimedia.org/T293862 EMAIL

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2022-03-24T21:11:20Z] bking@cumin1001 restarting blazegraph on wdqs[1003-1013].eqiad.wmnet for T293862 TASK DETAIL https://phabricator.wikimedia.org/T293862 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

Maintenance_bot removed a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T293862 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, Maintenance_bot Cc: bking, Aklapper, dcausse, Astuthiodit_1, karapayneWMDE, Invadibot,

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

gerritbot added a comment. Change 770978 **merged** by Bking: [operations/puppet@production] [wdqs] test jvmquake options on the public cluster https://gerrit.wikimedia.org/r/770978 TASK DETAIL https://phabricator.wikimedia.org/T293862 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2022-03-16T09:36:05Z] T293862 : manually restarted blazegraph on wdqs1010 with "-agentpath:/usr/lib/libjvmquake.so=1000,1,0,warn=30,touch=/tmp/jvmquake" TASK DETAIL

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T293862 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, gerritbot Cc: bking, Aklapper, dcausse, Fernandobacasegua34, Astuthiodit_1, 786, Suran38,

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

gerritbot added a comment. Change 770978 had a related patch set uploaded (by DCausse; author: DCausse): [operations/puppet@production] [wdqs] add jvmquake options to wdqs1010 for testing https://gerrit.wikimedia.org/r/770978 TASK DETAIL https://phabricator.wikimedia.org/T293862

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

bking added a comment. Manually installed on wdqs1010 TASK DETAIL https://phabricator.wikimedia.org/T293862 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, bking Cc: bking, Aklapper, dcausse, Astuthiodit_1, karapayneWMDE, Invadibot,

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2022-03-08T16:53:57Z] bking@deneb manually installed tox for T293862 . moritzm will add puppet patch for this TASK DETAIL https://phabricator.wikimedia.org/T293862 EMAIL

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2022-03-08T16:02:10Z] bking@deneb manually installed openjdk-11-jdk for T293862 . moritzm will add puppet patch for this TASK DETAIL https://phabricator.wikimedia.org/T293862

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

dcausse added a comment. Pushed https://gitlab.wikimedia.org/repos/search-platform/jvmquake/-/merge_requests/1 (up for review) to have a debian package that we could install on production machines. TASK DETAIL https://phabricator.wikimedia.org/T293862 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

dcausse claimed this task. dcausse moved this task from Ready for Development to In Progress on the Discovery-Search (Current work) board. TASK DETAIL https://phabricator.wikimedia.org/T293862 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

EBernhardson updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T293862 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana,

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

TJones removed the point value for this task. TASK DETAIL https://phabricator.wikimedia.org/T293862 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: TJones Cc: Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana,

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

TJones set the point value for this task to "5". TASK DETAIL https://phabricator.wikimedia.org/T293862 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: TJones Cc: Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana,

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

Gehel updated the task description. Gehel removed the point value for this task. TASK DETAIL https://phabricator.wikimedia.org/T293862 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja,

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

MPhamWMF set the point value for this task to "5". TASK DETAIL https://phabricator.wikimedia.org/T293862 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: MPhamWMF Cc: Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana,

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

Gehel updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T293862 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi,

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

MPhamWMF moved this task from All WDQS-related tasks to Current work on the Wikidata-Query-Service board. MPhamWMF added a project: Discovery-Search (Current work). TASK DETAIL https://phabricator.wikimedia.org/T293862 WORKBOARD https://phabricator.wikimedia.org/project/board/891/ EMAIL

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

Maintenance_bot added a project: Wikidata. TASK DETAIL https://phabricator.wikimedia.org/T293862 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Maintenance_bot Cc: Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana,

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

dcausse updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T293862 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: Aklapper, dcausse, MPhamWMF, CBogen, Namenlos314, Gq86, Lucas_Werkmeister_WMDE, EBjune, merbst,

[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead

dcausse created this task. dcausse added a project: Wikidata-Query-Service. Restricted Application added a subscriber: Aklapper. TASK DESCRIPTION As as a maintainer of a service running on top of the JVM I want the JVM to rapidly quit if it enters a gc death spiral so that the service increase