[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

dcausse closed this task as "Resolved". dcausse added a comment. Thanks for the report, blazegraph died I restarted it, should be available again now. TASK DETAIL https://phabricator.wikimedia.org/T297454 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

Fuzheado added a comment. Agree, here's the specific error coming back: 502 Bad Gateway 502 Bad Gateway nginx/1.13.9 TASK DETAIL https://phabricator.wikimedia.org/T297454 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To:

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

Dominicbm added a comment. I reopened this, as WCQS is again giving 502 Bad Gateway error, though slightly different behavior for me. wcqs-beta.wmflabs.org resolves, but gives the error as the response when clicking the "Run" button. TASK DETAIL https://phabricator.wikimedia.org/T297454

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

Dominicbm reopened this task as "Open". TASK DETAIL https://phabricator.wikimedia.org/T297454 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcaro, Dominicbm Cc: Theklan, Marsupium, Vojtech.dostal, Base, RhinosF1, Majavah, aborrero,

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

dcaro added a comment. If you create any new VMs, or re-image the existing ones, use that new flavor and it will avoid getting the host out of space (it has a slightly smaller disk). TASK DETAIL https://phabricator.wikimedia.org/T297454 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

Stashbot added a comment. Mentioned in SAL (#wikimedia-cloud) [2022-01-10T13:56:24Z] Replace too big flavor t206636 for a one with a smaller disk, t206636v2 (T297454 ) TASK DETAIL https://phabricator.wikimedia.org/T297454 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

Gehel closed this task as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T297454 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcaro, Gehel Cc: Theklan, Marsupium, Vojtech.dostal, Base, RhinosF1, Majavah, aborrero, GFontenelle_WMF, Sj,

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

EBernhardson added a comment. The beta service looks to have unintentionally picked up some of the configuration of the production cluster. I've put the configuration back and disabled the beta instance (puppet) from updating itself which should keep things in the current state as we make

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

RhinosF1 added a comment. Assuming it is released on time, yes. The search team aren't going to put something unstable in production. The production service will use its own dedicated and seperate hardware to beta though. TASK DETAIL https://phabricator.wikimedia.org/T297454

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

Sj added a comment. I get a 500 error now... will this be back up before the general release on Feb 1? TASK DETAIL https://phabricator.wikimedia.org/T297454 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcaro, Sj Cc: Theklan, Marsupium,

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

Gehel moved this task from Incoming to Current work on the Wikidata-Query-Service board. Gehel added a project: Discovery-Search (Current work). TASK DETAIL https://phabricator.wikimedia.org/T297454 WORKBOARD https://phabricator.wikimedia.org/project/board/891/ EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

EBernhardson added a comment. In T297454#7572089 , @dcaro wrote: > I had to stop (shutoff) and start the VM again (to clear the error state), the VM is up and running, and what's using all the space is: > >

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

Stashbot added a comment. Mentioned in SAL (#wikimedia-cloud) [2021-12-15T12:44:25Z] Downtiming cloudvirt-wdqs1001 as it has no VMs running until disk space is fixed (T297454 ) TASK DETAIL https://phabricator.wikimedia.org/T297454 EMAIL

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-12-15T12:43:27Z] END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on cloudvirt-wdqs1001.eqiad.wmnet with reason: T297454 TASK DETAIL

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-12-15T12:43:23Z] START - Cookbook sre.hosts.downtime for 30 days, 0:00:00 on cloudvirt-wdqs1001.eqiad.wmnet with reason: T297454 TASK DETAIL

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-12-15T12:43:14Z] END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt-wdqs1001.eqiad.wmnet with reason: T297454 TASK DETAIL

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-12-15T12:43:09Z] START - Cookbook sre.hosts.downtime for 4:00:00 on cloudvirt-wdqs1001.eqiad.wmnet with reason: T297454 TASK DETAIL

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

dcaro added a comment. I had to stop (shutoff) and start the VM again (to clear the error state), the VM is up and running, and what's using all the space is: root@wcqs-beta-01:/srv/wdqs-data# du -hs * 4.0Kaliases.map 4.0Kdumps 28G latest-mediainfo.ttl.gz

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

EBernhardson added a comment. In T297454#7568856 , @dcaro wrote: > I've moved the other VM around, so there's a little bit of space free now, should be enough to start the VM but you'll have to make sure to cleanup right when it comes

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

Stashbot added a comment. Mentioned in SAL (#wikimedia-cloud) [2021-12-14T10:26:14Z] Moved the nova cache (/var/lib/nova/instances/_base) and the canary image local data (/var/lib/nova/instance/) to the root disk on cloudvirt-wdqs1001 to temporary free some space (T297454

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

dcaro added a comment. I've moved the other VM around, so there's a little bit of space free now, should be enough to start the VM but you'll have to make sure to cleanup right when it comes back up, if it's not possible to get it to less than half it's size, we will not be able to shrink

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

dcaro reopened this task as "Open". dcaro added a comment. Sorry about that. TASK DETAIL https://phabricator.wikimedia.org/T297454 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcaro Cc: Base, RhinosF1, Majavah, aborrero, GFontenelle_WMF, Sj,

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

dcaro closed this task as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T297454 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcaro Cc: Base, RhinosF1, Majavah, aborrero, GFontenelle_WMF, Sj, FRomeo_WMF, Fuzheado, Dominicbm, HenkvD,

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

dcaro added a comment. The underlying filesystem is full, the disk size was set too big (3.4T), there's a cloudcanary instance taking 20G that can be removed to try to start the VM and cleanup stuff, but we'll need to resize the disk later once the sapce is freed. Not sure if that's

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

dcaro changed the task status from "In Progress" to "Open". TASK DETAIL https://phabricator.wikimedia.org/T297454 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcaro Cc: RhinosF1, Majavah, aborrero, GFontenelle_WMF, Sj, FRomeo_WMF, Fuzheado,

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

Majavah added a comment. The hypervisor box it's running on has apparently filled its disk: taavi@cloudvirt-wdqs1001 ~ $ df -h /var/lib/nova/instances/ Filesystem Size Used Avail Use% Mounted on /dev/mapper/tank-data 3.3T 3.3T 20K 100% /var/lib/nova/instances

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

EBernhardson added a comment. In T297454#7567004 , @Sj wrote: > Is this monitored by any of the status tools? Does it just need to be restarted? As a beta service running outside of production this does not have access to much of

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

aborrero added a comment. the VM is in error state: This is what I get from the openstack API: {'code': 500, 'created': '2021-12-10T19:19:35Z', 'message': 'OSError', 'details': 'Traceback (most recent call last):\n File "/usr/lib/python3/dist-packages/nova/compute/manager.py",

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

dcaro added a project: Cloud-Services-Origin-User. TASK DETAIL https://phabricator.wikimedia.org/T297454 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcaro Cc: GFontenelle_WMF, Sj, FRomeo_WMF, Fuzheado, Dominicbm, HenkvD, Alicia_Fagerving_WMSE,

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

dcaro added a project: Cloud-Services-Worktype-Unplanned. TASK DETAIL https://phabricator.wikimedia.org/T297454 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcaro Cc: GFontenelle_WMF, Sj, FRomeo_WMF, Fuzheado, Dominicbm, HenkvD,

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

dcaro claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T297454 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcaro Cc: GFontenelle_WMF, Sj, FRomeo_WMF, Fuzheado, Dominicbm, HenkvD, Alicia_Fagerving_WMSE, EBernhardson, Aklapper,

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

dcaro changed the task status from "Open" to "In Progress". TASK DETAIL https://phabricator.wikimedia.org/T297454 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcaro Cc: GFontenelle_WMF, Sj, FRomeo_WMF, Fuzheado, Dominicbm, HenkvD,

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

dcaro added a project: User-dcaro. TASK DETAIL https://phabricator.wikimedia.org/T297454 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcaro Cc: GFontenelle_WMF, Sj, FRomeo_WMF, Fuzheado, Dominicbm, HenkvD, Alicia_Fagerving_WMSE, EBernhardson,

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

Sj added a comment. Is this monitored by any of the status tools? Does it just need to be restarted? (and is it normal for projectadmin's to not have the ability to restart, or a feature of this being complex service?) TASK DETAIL https://phabricator.wikimedia.org/T297454 EMAIL

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

EBernhardson added a comment. This is still the correct URL. I don't have exact logs, but it looks like around 2021-12-09T17:44 the instance stopped running and upon inspection reports a status of error and power state of paused. Around 20 minutes after I started poking it the power state

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

Maintenance_bot added a project: Wikidata. TASK DETAIL https://phabricator.wikimedia.org/T297454 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Maintenance_bot Cc: Aklapper, Jarekt, Invadibot, GFontenelle_WMF, MPhamWMF, maantietaja, FRomeo_WMF,

[Wikidata-bugs] [Maniphest] T297454: WCQS gives "502 Bad Gateway Error"

Jarekt created this task. Jarekt added projects: Wikidata-Query-Service, SDC General. Restricted Application added a subscriber: Aklapper. TASK DESCRIPTION Wikimedia Commons Query Service seems to be down, as https://wcqs-beta.wmflabs.org/ is giving me "502 Bad Gateway Error". I have not used