[Wikidata-bugs] [Maniphest] T281504: Degraded RAID on wdqs2007

2021-05-03 Thread RKemper
RKemper added a comment.


  Re-image of `wdqs2007` was completed successfully.

TASK DETAIL
  https://phabricator.wikimedia.org/T281504

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, Aklapper, ops-monitoring-bot, Ramtin0071, MPhamWMF, Devnull, 
lmata, Muchiri124, CBogen, RhinosF1, Legado_Shulgin, ReaperDawn, Namenlos314, 
Davinaclare77, Qtn1293, Techguru.pc, Gq86, Lucas_Werkmeister_WMDE, Th3d3v1ls, 
Hfbn0, EBjune, merbst, Zppix, Jonas, Xmlizer, Wong128hk, Cmjohnson, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, 
Jay8g, fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T281504: Degraded RAID on wdqs2007

2021-05-03 Thread RKemper
RKemper closed this task as "Resolved".

TASK DETAIL
  https://phabricator.wikimedia.org/T281504

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, Aklapper, ops-monitoring-bot, Ramtin0071, MPhamWMF, Devnull, 
lmata, Muchiri124, CBogen, RhinosF1, Legado_Shulgin, ReaperDawn, Namenlos314, 
Davinaclare77, Qtn1293, Techguru.pc, Gq86, Lucas_Werkmeister_WMDE, Th3d3v1ls, 
Hfbn0, EBjune, merbst, Zppix, Jonas, Xmlizer, Wong128hk, Cmjohnson, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, 
Jay8g, fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T281504: Degraded RAID on wdqs2007

2021-05-03 Thread RKemper
RKemper claimed this task.

TASK DETAIL
  https://phabricator.wikimedia.org/T281504

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, Aklapper, ops-monitoring-bot, Ramtin0071, MPhamWMF, Devnull, 
lmata, Muchiri124, CBogen, RhinosF1, Legado_Shulgin, ReaperDawn, Namenlos314, 
Davinaclare77, Qtn1293, Techguru.pc, Gq86, Lucas_Werkmeister_WMDE, Th3d3v1ls, 
Hfbn0, EBjune, merbst, Zppix, Jonas, Xmlizer, Wong128hk, Cmjohnson, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Southparkfan, faidon, 
Jay8g, fgiunchedi
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org


[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps

2021-04-16 Thread RKemper
RKemper updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T267927

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: jbond, Zbyszko, Gehel, Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, 
CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps

2021-04-16 Thread RKemper
RKemper added a comment.


  We can't proceed any further on this ticket until we address 
https://phabricator.wikimedia.org/T280382, because the short-term fix for disk 
space will require re-imaging the affected `wdqs` hosts which will require 
starting the `data-transfer` process over (not the `data-reload`, fortunately).

TASK DETAIL
  https://phabricator.wikimedia.org/T267927

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: jbond, Zbyszko, Gehel, Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, 
CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps

2021-04-16 Thread RKemper
RKemper updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T267927

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: jbond, Zbyszko, Gehel, Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, 
CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps

2021-04-15 Thread RKemper
RKemper updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T267927

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: jbond, Zbyszko, Gehel, Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, 
CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T280108: WDQS data-transfer cookbook needs to wait for updater to catchup on lag

2021-04-15 Thread RKemper
RKemper added a comment.


100.0% (2/2) success ratio (>= 100.0% threshold) for command: 'systemctl 
start ...art wdqs-updater'.
100.0% (2/2) success ratio (>= 100.0% threshold) of nodes successfully 
executed all commands.
[1/1000, retrying in 600.00s] Attempt to run 
'cookbooks.sre.wdqs.wait_for_updater' raised: invalid literal for int() with 
base 10: 'NaN'
[2/1000, retrying in 600.00s] Attempt to run 
'cookbooks.sre.wdqs.wait_for_updater' raised: invalid literal for int() with 
base 10: '4458.16434332'
[3/1000, retrying in 600.00s] Attempt to run 
'cookbooks.sre.wdqs.wait_for_updater' raised: invalid literal for int() with 
base 10: '4230.269809265'
[4/1000, retrying in 600.00s] Attempt to run 
'cookbooks.sre.wdqs.wait_for_updater' raised: invalid literal for int() with 
base 10: '3850.3779346'
[5/1000, retrying in 600.00s] Attempt to run 
'cookbooks.sre.wdqs.wait_for_updater' raised: invalid literal for int() with 
base 10: '3175.483400543'
[6/1000, retrying in 600.00s] Attempt to run 
'cookbooks.sre.wdqs.wait_for_updater' raised: invalid literal for int() with 
base 10: '888.591801086'
[7/1000, retrying in 600.00s] Attempt to run 
'cookbooks.sre.wdqs.wait_for_updater' raised: invalid literal for int() with 
base 10: '39.694332428'
[8/1000, retrying in 600.00s] Attempt to run 
'cookbooks.sre.wdqs.wait_for_updater' raised: invalid literal for int() with 
base 10: '42.79952316284'
  
  Another small issue. Looks like we need to tell it to truncate (or just use 
doubles instead of ints)

TASK DETAIL
  https://phabricator.wikimedia.org/T280108

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T280108: WDQS data-transfer cookbook needs to wait for updater to catchup on lag

2021-04-15 Thread RKemper
RKemper added a comment.


  (For future reference)
  
import logging
logging.basicConfig(level=logging.DEBUG)
from spicerack import Spicerack
spicerack = Spicerack(verbose=True, dry_run=True)


def site(host):
"""Hacky way to get the site in which a host is located."""
if 'eqiad' in str(host):
return 'eqiad'
if 'codfw' in str(host):
return 'codfw'
raise ValueError('Site is unknown for {host}.'.format(host=host))

remote = spicerack.remote()
prometheus = spicerack.prometheus()

remote_host = remote.query('wdqs2008.codfw.wmnet')

host = remote_host.hosts[0].split(".")[0]
query = "scalar(time() - blazegraph_lastupdated{instance='%s:9193'})" % host
result = prometheus.query(query, site(remote_host))
last_updated = int(result[1])
if last_updated > 1200:
raise ValueError("Let's wait for updater to catch up, current 
last_updated of {} is too high.".format(last_updated))
  
  Snippet used to diagnose the above issue without having to actually run the 
cookbook

TASK DETAIL
  https://phabricator.wikimedia.org/T280108

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, Alter-paule, 
Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, 
Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, 
Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps

2021-04-15 Thread RKemper
RKemper added a comment.


  Since we've seeded to `wdqs1004.eqiad.wmnet` we can start using that to 
transfer to other eqiad public nodes. Still need to seed to the other 3 (codfw 
public, codfw internal, eqiad internal) before we'll be at the peak "velocity".

TASK DETAIL
  https://phabricator.wikimedia.org/T267927

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: jbond, Zbyszko, Gehel, Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, 
CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps

2021-04-15 Thread RKemper
RKemper updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T267927

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: jbond, Zbyszko, Gehel, Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, 
CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T280108: WDQS data-transfer cookbook needs to wait for updater to catchup on lag

2021-04-15 Thread RKemper
RKemper added a comment.


  Tested out the new cookbook, there's at least one small issue:
  
Exception raised while executing cookbook sre.wdqs.data-transfer:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/spicerack/_menu.py", line 234, in run
raw_ret = runner.run()
  File "/usr/lib/python3/dist-packages/spicerack/_module_api.py", line 18, 
in run
return self._run(self.args, self.spicerack)
  File "/srv/deployment/spicerack/cookbooks/sre/wdqs/data-transfer.py", 
line 179, in run
wait_for_updater(prometheus, site(source), source)
  File "/usr/lib/python3/dist-packages/spicerack/decorators.py", line 104, 
in wrapper
return func(*args, **kwargs)  # type: ignore
  File "/srv/deployment/spicerack/cookbooks/sre/wdqs/__init__.py", line 25, 
in wait_for_updater
last_updated = int(result['value'][1])
TypeError: list indices must be integers or slices, not str
END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99)
  
  Error seems quite straightforward - we'll want to patch this.

TASK DETAIL
  https://phabricator.wikimedia.org/T280108

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps

2021-04-14 Thread RKemper
RKemper removed a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T267927

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: jbond, Zbyszko, Gehel, Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, 
CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331, Alter-paule, Beast1978, Un1tY, Hook696, Kent7301, 
joker88john, CucyNoiD, Gaboe420, Giuliamocci, Cpaulf30, Af420, Bsandipan, 
Lewizho99, Maathavan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps

2021-04-14 Thread RKemper
RKemper updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T267927

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: jbond, Zbyszko, Gehel, Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, 
Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, 
joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, 
Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps

2021-04-14 Thread RKemper
RKemper added a comment.


  I was having trouble understanding what the restart status of blazegraph has 
to do with the transfer of `/srv/wdqs/wikidata.jnl`. (As an aside, the point 
that running multiple simultaneous transfers from the same source is 
bottlenecked by outgoing bandwidth anyway makes perfect sense to me).
  
  It just clicked for me though - the cookbook transfers the file while 
blazegraph is disabled, so if another cookbook instance re-enables blazegraph 
while another cookbook instance is still receiving `/srv/wdqs/wikidata.jnl`, 
the `wikidata.jnl` will be getting mutated during the transfer.
  
  Note: I updated my comment above to indicate that `wdqs100[3,4]` are not 
properly done due to the data corruption.

TASK DETAIL
  https://phabricator.wikimedia.org/T267927

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: jbond, Zbyszko, Gehel, Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, 
Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, 
joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, 
Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps

2021-04-14 Thread RKemper
RKemper added a comment.


  [EQIAD PUBLIC]
  DONE sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet 
--dest wdqs1004.eqiad.wmnet --reason "transferring wikidata journal following 
reload from dumps" --blazegraph_instance blazegraph --task-id T267927 
<https://phabricator.wikimedia.org/T267927>
  sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest 
wdqs1005.eqiad.wmnet --reason "transferring wikidata journal following reload 
from dumps" --blazegraph_instance blazegraph --task-id T267927 
<https://phabricator.wikimedia.org/T267927>
  sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest 
wdqs1006.eqiad.wmnet --reason "transferring wikidata journal following reload 
from dumps" --blazegraph_instance blazegraph --task-id T267927 
<https://phabricator.wikimedia.org/T267927>
  sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest 
wdqs1007.eqiad.wmnet --reason "transferring wikidata journal following reload 
from dumps" --blazegraph_instance blazegraph --task-id T267927 
<https://phabricator.wikimedia.org/T267927>
  
  sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest 
wdqs1012.eqiad.wmnet --reason "transferring wikidata journal following reload 
from dumps" --blazegraph_instance blazegraph --task-id T267927 
<https://phabricator.wikimedia.org/T267927>
  sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest 
wdqs1013.eqiad.wmnet --reason "transferring wikidata journal following reload 
from dumps" --blazegraph_instance blazegraph --task-id T267927 
<https://phabricator.wikimedia.org/T267927>
  
  [CODFW PUBLIC]
  DONE sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet 
--dest wdqs2001.codfw.wmnet --reason "transferring wikidata journal following 
reload from dumps" --blazegraph_instance blazegraph --task-id T267927 
<https://phabricator.wikimedia.org/T267927>
  sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest 
wdqs2002.codfw.wmnet --reason "transferring wikidata journal following reload 
from dumps" --blazegraph_instance blazegraph --task-id T267927 
<https://phabricator.wikimedia.org/T267927>
  sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest 
wdqs2003.codfw.wmnet --reason "transferring wikidata journal following reload 
from dumps" --blazegraph_instance blazegraph --task-id T267927 
<https://phabricator.wikimedia.org/T267927>
  sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest 
wdqs2007.codfw.wmnet --reason "transferring wikidata journal following reload 
from dumps" --blazegraph_instance blazegraph --task-id T267927 
<https://phabricator.wikimedia.org/T267927>
  
  [EQIAD INTERNAL]
  DONE sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet 
--dest wdqs1003.eqiad.wmnet --reason "transferring wikidata journal following 
reload from dumps" --blazegraph_instance blazegraph --task-id T267927 
<https://phabricator.wikimedia.org/T267927>
  sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest 
wdqs1008.eqiad.wmnet --reason "transferring wikidata journal following reload 
from dumps" --blazegraph_instance blazegraph --task-id T267927 
<https://phabricator.wikimedia.org/T267927>
  sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest 
wdqs1011.eqiad.wmnet --reason "transferring wikidata journal following reload 
from dumps" --blazegraph_instance blazegraph --task-id T267927 
<https://phabricator.wikimedia.org/T267927>
  
  [CODFW INTERNAL]
  DONE sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet 
--dest wdqs2004.codfw.wmnet --reason "transferring wikidata journal following 
reload from dumps" --blazegraph_instance blazegraph --task-id T267927 
<https://phabricator.wikimedia.org/T267927>
  sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest 
wdqs2005.codfw.wmnet --reason "transferring wikidata journal following reload 
from dumps" --blazegraph_instance blazegraph --task-id T267927 
<https://phabricator.wikimedia.org/T267927>
  sudo -i cookbook sre.wdqs.data-transfer --source wdqs2008.codfw.wmnet --dest 
wdqs2006.codfw.wmnet --reason "transferring wikidata journal following reload 
from dumps" --blazegraph_instance blazegraph --task-id T267927 
<https://phabricator.wikimedia.org/T267927>

TASK DETAIL
  https://phabricator.wikimedia.org/T267927

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: jbond, Zbyszko, Gehel, Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, 
Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, 
joker88john, CucyNoiD, Nanda

[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps

2021-03-24 Thread RKemper
RKemper added a comment.


  `wdqs1009`, `wdqs1010`, and `wdqs2008` are done, so we need to 
`data-transfer` to the remaining instances.
  
  It looks like `wdqs1010` is warning (not critical) for disk space on `/srv`. 
Glancing at the box it's still got 44G left but is at 96% utilization:
  
  `/dev/mapper/vg0-srv   1.1T  999G   44G  96% /srv`
  
  This is essentially entirely attributable to `wikidata.jnl` which is **1 TB** 
in size.

TASK DETAIL
  https://phabricator.wikimedia.org/T267927

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: ayounsi, jbond, Zbyszko, Gehel, Aklapper, dcausse, Invadibot, MPhamWMF, 
maantietaja, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, 
Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, 
Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, 
_jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-03-12 Thread RKemper
RKemper added a comment.


  **DNS change logs**
  
ryankemper@authdns1001:~$ sudo authdns-update
Updating authdns1001.wikimedia.org (self)...
Pulling the current revision from 
https://gerrit.wikimedia.org/r/operations/dns.git
Reviewing 85d9b49dc2ff0f8e3657f6f2cd91ce3df79bd1cf...

 templates/wikidata.org | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git templates/wikidata.org templates/wikidata.org
index 9f1a5ed3..cea393b1 100644
--- templates/wikidata.org
+++ templates/wikidata.org
@@ -32,7 +32,8 @@
 ; Servers (alphabetic order)

 ; Service aliases
-query   1D IN CNAME dyna.wikimedia.org.
+query 1D IN CNAME dyna.wikimedia.org.
+query-preview 1D IN CNAME dyna.wikimedia.org.

 ; Wikis (alphabetic order)


Merge these changes? (yes/no)? yes
Updating 3a84bd1c..85d9b49d
Fast-forward
 templates/wikidata.org | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
Deploying via utils/deploy-check.py...
Assembling and testing data in /tmp/dns-check.voy682v4
 -- Generating zonefiles from zone templates
 -- Processed 205 zones into directory /tmp/dns-check.voy682v4/zones
OK: No tabs
Summary of violations:
W001|MISSING_IP_FOR_NAME_AND_PTR: 35
W002|MISSING_PTR_FOR_NAME_AND_IP: 27
W103|MISSING_MGMT_FOR_NAME: 59
W105|TOO_MANY_PUBLIC_NAMES: 3
RESULT: 0 Errors, 124 Warnings, 0 Ignored violations, 0 Ignored lines
 -- Copying automatically generated zone files under target tree
 -- Copying repo-driven real config files and admin_state
 -- Copying puppetized config and GeoIP from /etc/gdnsd
 -- Checking for illegal tabs in zonefiles
 -- Running zone_validator to check WMF rules
 -- Running /usr/sbin/gdnsd checkconf on /tmp/dns-check.voy682v4
 -- Preflight checkconf is OK
Deploying from /tmp/dns-check.voy682v4 to system dirs
 -- Descending to subdirectory: netbox
 -- Done with subdir: netbox
 -- Zone changed: wikidata.org
Reloading gdnsd zonefiles
info: Zone data reloaded
OK
---

authdns2001.wikimedia.org,dns[1001-1002,2001-2002,3001-3002,4001-4002,5001-5002].wikimedia.org
 (11)
---
OK - authdns updated successfully

OK - authdns-update successful on all nodes!
  
  Validation looks good:
  
user@computer ~/wmf/gui-deploy [production]% for i in 0 1 2 ; do dig 
@ns${i}.wikimedia.org -t any query-preview.wikidata.org ; done

; <<>> DiG 9.10.6 <<>> @ns0.wikimedia.org -t any query-preview.wikidata.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58419
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1024
;; QUESTION SECTION:
;query-preview.wikidata.org.IN  ANY

;; ANSWER SECTION:
query-preview.wikidata.org. 86400 INCNAME   dyna.wikimedia.org.

;; Query time: 82 msec
;; SERVER: 208.80.154.238#53(208.80.154.238)
;; WHEN: Fri Mar 12 17:06:00 PST 2021
;; MSG SIZE  rcvd: 84


; <<>> DiG 9.10.6 <<>> @ns1.wikimedia.org -t any query-preview.wikidata.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3886
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1024
;; QUESTION SECTION:
;query-preview.wikidata.org.IN  ANY

;; ANSWER SECTION:
query-preview.wikidata.org. 86400 INCNAME   dyna.wikimedia.org.

;; Query time: 61 msec
;; SERVER: 208.80.153.231#53(208.80.153.231)
;; WHEN: Fri Mar 12 17:06:00 PST 2021
;; MSG SIZE  rcvd: 84


; <<>> DiG 9.10.6 <<>> @ns2.wikimedia.org -t any query-preview.wikidata.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53512
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1024
;; QUESTION SECTION:
;query-preview.wikidata.org.IN  ANY

;; ANSWER SECTION:
query-preview.wikidata.org. 86400 INCNAME   dyna.wikimedia.org.

;; Query time: 345 msec
;; SERVER: 91.198.174.239#53(91.198.174.239)
;; WHEN: Fri Mar 12 17:06:01 PST 2021
;; MSG SIZE  rcvd: 84

TASK DETAIL
  https://phabricator.wikimedia.org/T266470

EMAIL PREFERENCES
  https://phabricator.wikimedi

[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-03-12 Thread RKemper
RKemper added a comment.


  The issues with envoy were resolved by running `sudo 
/usr/local/sbin/build-envoy-config -c /etc/envoy` to properly build 
`/etc/envoy/envoy.yaml`. That should have been done by puppet already, 
triggered upon a `sudo systemctl restart envoyproxy.service`, but it didn't - 
perhaps a race condition. See 
https://gerrit.wikimedia.org/g/operations/puppet/+/b7dacbca9fae42b32bb91fd485a3f2c70ff903b3/modules/envoyproxy/manifests/init.pp#81
 and 
https://gerrit.wikimedia.org/g/operations/puppet/+/b7dacbca9fae42b32bb91fd485a3f2c70ff903b3/modules/envoyproxy/manifests/conf.pp#30
 for the puppet code that normally does it automatically.
  
  **Envoy is working properly now and wdqs1009 is now reachable from the 
caching layer**

TASK DETAIL
  https://phabricator.wikimedia.org/T266470

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Gehel, Lea_Lacroix_WMDE, dcausse, Aklapper, MPhamWMF, maantietaja, 
Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, 
joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, 
Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, abian, jkroll, Wikidata-bugs, Jdouglas, 
aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-03-12 Thread RKemper
RKemper added a comment.


  **DNS change logs**
  
ryankemper@authdns1001:~$ sudo authdns-update
Updating authdns1001.wikimedia.org (self)...
Pulling the current revision from 
https://gerrit.wikimedia.org/r/operations/dns.git
Reviewing 85d9b49dc2ff0f8e3657f6f2cd91ce3df79bd1cf...

 templates/wikidata.org | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git templates/wikidata.org templates/wikidata.org
index 9f1a5ed3..cea393b1 100644
--- templates/wikidata.org
+++ templates/wikidata.org
@@ -32,7 +32,8 @@
 ; Servers (alphabetic order)

 ; Service aliases
-query   1D IN CNAME dyna.wikimedia.org.
+query 1D IN CNAME dyna.wikimedia.org.
+query-preview 1D IN CNAME dyna.wikimedia.org.

 ; Wikis (alphabetic order)


Merge these changes? (yes/no)? yes
Updating 3a84bd1c..85d9b49d
Fast-forward
 templates/wikidata.org | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
Deploying via utils/deploy-check.py...
Assembling and testing data in /tmp/dns-check.voy682v4
 -- Generating zonefiles from zone templates
 -- Processed 205 zones into directory /tmp/dns-check.voy682v4/zones
OK: No tabs
Summary of violations:
W001|MISSING_IP_FOR_NAME_AND_PTR: 35
W002|MISSING_PTR_FOR_NAME_AND_IP: 27
W103|MISSING_MGMT_FOR_NAME: 59
W105|TOO_MANY_PUBLIC_NAMES: 3
RESULT: 0 Errors, 124 Warnings, 0 Ignored violations, 0 Ignored lines
 -- Copying automatically generated zone files under target tree
 -- Copying repo-driven real config files and admin_state
 -- Copying puppetized config and GeoIP from /etc/gdnsd
 -- Checking for illegal tabs in zonefiles
 -- Running zone_validator to check WMF rules
 -- Running /usr/sbin/gdnsd checkconf on /tmp/dns-check.voy682v4
 -- Preflight checkconf is OK
Deploying from /tmp/dns-check.voy682v4 to system dirs
 -- Descending to subdirectory: netbox
 -- Done with subdir: netbox
 -- Zone changed: wikidata.org
Reloading gdnsd zonefiles
info: Zone data reloaded
OK
---

authdns2001.wikimedia.org,dns[1001-1002,2001-2002,3001-3002,4001-4002,5001-5002].wikimedia.org
 (11)
---
OK - authdns updated successfully

OK - authdns-update successful on all nodes!
  
  Validation looks good:
  
user@computer ~/wmf/gui-deploy [production]% for i in 0 1 2 ; do dig 
@ns${i}.wikimedia.org -t any query-preview.wikidata.org ; done

; <<>> DiG 9.10.6 <<>> @ns0.wikimedia.org -t any query-preview.wikidata.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58419
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1024
;; QUESTION SECTION:
;query-preview.wikidata.org.IN  ANY

;; ANSWER SECTION:
query-preview.wikidata.org. 86400 INCNAME   dyna.wikimedia.org.

;; Query time: 82 msec
;; SERVER: 208.80.154.238#53(208.80.154.238)
;; WHEN: Fri Mar 12 17:06:00 PST 2021
;; MSG SIZE  rcvd: 84


; <<>> DiG 9.10.6 <<>> @ns1.wikimedia.org -t any query-preview.wikidata.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3886
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1024
;; QUESTION SECTION:
;query-preview.wikidata.org.IN  ANY

;; ANSWER SECTION:
query-preview.wikidata.org. 86400 INCNAME   dyna.wikimedia.org.

;; Query time: 61 msec
;; SERVER: 208.80.153.231#53(208.80.153.231)
;; WHEN: Fri Mar 12 17:06:00 PST 2021
;; MSG SIZE  rcvd: 84


; <<>> DiG 9.10.6 <<>> @ns2.wikimedia.org -t any query-preview.wikidata.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53512
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1024
;; QUESTION SECTION:
;query-preview.wikidata.org.IN  ANY

;; ANSWER SECTION:
query-preview.wikidata.org. 86400 INCNAME   dyna.wikimedia.org.

;; Query time: 345 msec
;; SERVER: 91.198.174.239#53(91.198.174.239)
;; WHEN: Fri Mar 12 17:06:01 PST 2021
;; MSG SIZE  rcvd: 84

TASK DETAIL
  https://phabricator.wikimedia.org/T266470

EMAIL PREFERENCES
  https://phabricator.wikimedi

[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-03-12 Thread RKemper
RKemper added a comment.


  I missed a step yesterday: I'd updated `/srv/private` as well as the public 
`labs/private` repo but missed the step for updating `operations/puppet` with 
the new pubkey: https://gerrit.wikimedia.org/r/c/operations/puppet/+/671267

TASK DETAIL
  https://phabricator.wikimedia.org/T266470

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Gehel, Lea_Lacroix_WMDE, dcausse, Aklapper, MPhamWMF, maantietaja, 
Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, 
joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, 
Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, abian, jkroll, Wikidata-bugs, Jdouglas, 
aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-03-12 Thread RKemper
RKemper added a comment.


  Current status for when I pick this back up:
  
  - `query-preview.wikidata.org` (with `wdqs1009.eqiad.wmnet` as an `alt_name`) 
is generated
  
  - reverted https://gerrit.wikimedia.org/r/c/operations/puppet/+/670339/ for 
now, envoy wasn't happy:
  
Error: /Stage[main]/Envoyproxy/Exec[verify-envoy-config]: Failed to call 
refresh: '/usr/local/sbin/build-envoy-config -c '/etc/envoy'' returned 1 
instead of one of [0]
Error: /Stage[main]/Envoyproxy/Exec[verify-envoy-config]: 
'/usr/local/sbin/build-envoy-config -c '/etc/envoy'' returned 1 instead of one 
of [0]

Error: 
/Stage[main]/Profile::Tlsproxy::Envoy/Sslcert::Certificate[query-preview.wikidata.org]/File[/etc/ssl/localcerts/query-preview.wikidata.org.crt]:
 Could not evaluate: Could not retrieve information from environment production 
source(s) puppet:///files/ssl/query-preview.wikidata.org.crt

TASK DETAIL
  https://phabricator.wikimedia.org/T266470

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Gehel, Lea_Lacroix_WMDE, dcausse, Aklapper, MPhamWMF, maantietaja, 
Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, 
joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, 
Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, abian, jkroll, Wikidata-bugs, Jdouglas, 
aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-03-12 Thread RKemper
RKemper added a comment.


  Ah, so poking around the `certificate.manifests.d` repo I see certs that 
don't necessarily follow the `discovery.wmnet` pattern. To me that implies 
**Option 2** should be working, so I might be missing something. Here's an 
example that doesn't use discovery:
  
ryankemper@puppetmaster1001:/srv/private$ cat 
modules/secret/secrets/certificates/certificate.manifests.d/analytics_http_ui.certs.yaml
yarn.wikimedia.org:
  authority: puppet_ca
  expiry: null
  alt_names: ["yarn.wikimedia.org", "hue.wikimedia.org", 
"hue-next.wikimedia.org", "superset.wikimedia.org", "pivot.wikimedia.org", 
"turnilo.wikimedia.org", "stats.wikimedia.org", "analytics.wikimedia.org", 
"piwik.wikimedia.org", "datasets.wikimedia.org"]
  key:
password: REDACTED
algorithm: ec

TASK DETAIL
  https://phabricator.wikimedia.org/T266470

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Gehel, Lea_Lacroix_WMDE, dcausse, Aklapper, MPhamWMF, maantietaja, 
Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, 
joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, 
Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, abian, jkroll, Wikidata-bugs, Jdouglas, 
aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-03-12 Thread RKemper
RKemper added a comment.


  **Option 2 ** fails to even generate the cert. All the cergen documentation 
is written for a certificate like `query-preview.discovery.wmnet` and not 
`wdqs1009.eqiad.wmnet` or `query-preview.wikidata.org`. So I do think this just 
isn't what cergen is built to do.
  
ryankemper@puppetmaster1001:/srv/private$ git status
On branch master
Untracked files:
  (use "git add ..." to include in what will be committed)


modules/secret/secrets/certificates/certificate.manifests.d/wdqs1009.certs.yaml

nothing added to commit but untracked files present (use "git add" to track)
ryankemper@puppetmaster1001:/srv/private$ sudo cergen -c 'wdqs1009.*' 
--generate --base-path /srv/private/modules/secret/secrets/certificates 
/srv/private/modules/secret/secrets/certificates/certificate.manifests.d
2021-03-12 08:06:03,210 INFO cergen   
Generating certificates [] with force=False

Status of certificates []

TASK DETAIL
  https://phabricator.wikimedia.org/T266470

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Gehel, Lea_Lacroix_WMDE, dcausse, Aklapper, MPhamWMF, maantietaja, 
Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, 
joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, 
Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, abian, jkroll, Wikidata-bugs, Jdouglas, 
aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-03-12 Thread RKemper
RKemper added a comment.


  Hit a big blocker with the current proposed approach of using 
`wdqs1009.eqiad.wmnet` as the cert name:
  
ryankemper@puppetmaster1001:/srv/private$ sudo cergen -c 'wdqs1009.*' 
--generate --base-path /srv/private/modules/secret/secrets/certificates 
/srv/private/modules/secret/secrets/certificates/certificate.manifests.d
2021-03-12 07:55:03,774 INFO cergen   
Generating certificates ['wdqs1009.eqiad.wmnet'] with force=False
2021-03-12 07:55:03,774 INFO Certificate(wdqs1009.eqiad.wmnet)
Generating all files, force=False...
2021-03-12 07:55:03,776 INFO Certificate(wdqs1009.eqiad.wmnet)
Generating certificate file
/usr/lib/python3/dist-packages/urllib3/connection.py:362: 
SubjectAltNameWarning: Certificate for puppetmaster1001.eqiad.wmnet has no 
`subjectAltName`, falling back to check for a `commonName` for now. This 
feature is being removed by major browsers and deprecated by RFC 2818. (See 
https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
Traceback (most recent call last):
  File "/usr/bin/cergen", line 11, in 
load_entry_point('cergen==0.2.5', 'console_scripts', 'cergen')()
  File "/usr/lib/python3/dist-packages/cergen/main.py", line 93, in main
certificate.generate(force=args['--force'])
  File "/usr/lib/python3/dist-packages/cergen/certificate.py", line 304, in 
generate
self.generate_crt(force=force)
  File "/usr/lib/python3/dist-packages/cergen/certificate.py", line 344, in 
generate_crt
self.x509_cert = self.authority.sign(csr, self.expiry)
  File "/usr/lib/python3/dist-packages/cergen/puppet.py", line 193, in sign
self.name, common_name
RuntimeError: puppetmaster1001.eqiad.wmnet_8140 already has a signed 
certificate for wdqs1009.eqiad.wmnet. If you are trying to regenerate a Puppet 
CA signed certificate, you need to first remove the certificate from the Puppet 
CA. On the puppetmaster, `puppet cert clean wdqs1009.eqiad.wmnet` should do it.
  
  Corresponding to this cergen manifest file 
`modules/secret/secrets/certificates/certificate.manifests.d/wdqs1009.certs.yaml`:
  
wdqs1009.eqiad.wmnet:
  authority: puppet_ca
  expiry: null
  alt_names: ["wdqs1009.eqiad.wmnet","query-preview.wikidata.org"]
  key:
password: REDACTED
algorithm: ec
  
  I'm a bit lost on the right way to proceed. Thinking out loud here:
  
  **Option 1** Just actually re-generate the cert with `puppet cert clean 
wdqs1009.eqiad.wmnet` before running cergen. This feels wrong though, and I 
think ultimately it comes down to using cergen in a way that wasn't intended: 
name collision with the existing cert puppet has.
  
  **Option 2** Do this instead:
  
query-preview.wikidata.org:
  authority: puppet_ca
  expiry: null
  alt_names: ["query-preview.wikidata.org", "wdqs1009.eqiad.wmnet"]
  key:
password: REDACTED
algorithm: ec
  
  Might want to talk to traffic again. But in the interest of iteration speed I 
think trying out Option **2** is not a bad idea; there's not really any risk 
since this is pertaining to `wdqs1009.eqiad.wmnet` which is a test host.

TASK DETAIL
  https://phabricator.wikimedia.org/T266470

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Gehel, Lea_Lacroix_WMDE, dcausse, Aklapper, MPhamWMF, maantietaja, 
Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, 
joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, 
Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, abian, jkroll, Wikidata-bugs, Jdouglas, 
aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-03-10 Thread RKemper
RKemper added a comment.


  Finished rolling back to the previous iteration of `wdqs.discovery.wmnet` 
cert since we're now going to create a net-new cert `wdqs1009.eqiad.wmnet` for 
`wdqs-test`

TASK DETAIL
  https://phabricator.wikimedia.org/T266470

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Gehel, Lea_Lacroix_WMDE, dcausse, Aklapper, MPhamWMF, maantietaja, 
Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, 
joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, 
Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, abian, jkroll, Wikidata-bugs, Jdouglas, 
aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-03-10 Thread RKemper
RKemper added a comment.


Certificate(wdqs.discovery.wmnet, 
authorities=[PuppetCA(puppetmaster1001.eqiad.wmnet_8140)]):

/srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.key.private.pem:
 PRESENT (mtime: 2019-10-21T08:11:52.741452)

/srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.key.public.pem:
 PRESENT (mtime: 2019-10-21T08:11:52.745452)

/srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.crt.pem:
 PRESENT (mtime: 2021-03-10T19:17:36.206506)

/srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/ca.crt.pem:
 PRESENT (mtime: 2019-12-10T14:26:42.496697)

/srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.keystore.p12:
 PRESENT (mtime: 2021-03-10T19:17:36.226506)

/srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.keystore.jks:
 PRESENT (mtime: 2021-03-10T19:17:38.030503)

/srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/truststore.jks:
 PRESENT (mtime: 2021-03-10T19:17:38.938502)


ryankemper@puppetmaster1001:/srv/private$ git status
On branch master
Changes not staged for commit:
  (use "git add ..." to update what will be committed)
  (use "git checkout -- ..." to discard changes in working directory)

modified:   
modules/secret/secrets/certificates/certificate.manifests.d/wdqs.certs.yaml
modified:   
modules/secret/secrets/certificates/wdqs.discovery.wmnet/truststore.jks
modified:   
modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.crt.pem
modified:   
modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.csr.pem
modified:   
modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.keystore.jks
modified:   
modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.keystore.p12

TASK DETAIL
  https://phabricator.wikimedia.org/T266470

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Gehel, Lea_Lacroix_WMDE, dcausse, Aklapper, MPhamWMF, maantietaja, 
Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, 
joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, 
Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, abian, jkroll, Wikidata-bugs, Jdouglas, 
aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-03-09 Thread RKemper
RKemper added a comment.


Certificate(wdqs.discovery.wmnet, 
authorities=[PuppetCA(puppetmaster1001.eqiad.wmnet_8140)]):

/srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.key.private.pem:
 PRESENT (mtime: 2019-10-21T08:11:52.741452)

/srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.key.public.pem:
 PRESENT (mtime: 2019-10-21T08:11:52.745452)

/srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.crt.pem:
 PRESENT (mtime: 2021-03-10T04:58:24.029629)

/srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/ca.crt.pem:
 PRESENT (mtime: 2019-12-10T14:26:42.496697)

/srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.keystore.p12:
 PRESENT (mtime: 2021-03-10T04:58:24.049629)

/srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.keystore.jks:
 PRESENT (mtime: 2021-03-10T04:58:25.893621)

/srv/private/modules/secret/secrets/certificates/wdqs.discovery.wmnet/truststore.jks:
 PRESENT (mtime: 2021-03-10T04:58:26.865617)


ryankemper@puppetmaster1001:/srv/private$ git status
On branch master
Changes not staged for commit:
  (use "git add ..." to update what will be committed)
  (use "git checkout -- ..." to discard changes in working directory)

modified:   
modules/secret/secrets/certificates/certificate.manifests.d/wdqs.certs.yaml
modified:   
modules/secret/secrets/certificates/wdqs.discovery.wmnet/truststore.jks
modified:   
modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.crt.pem
modified:   
modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.csr.pem
modified:   
modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.keystore.jks
modified:   
modules/secret/secrets/certificates/wdqs.discovery.wmnet/wdqs.discovery.wmnet.keystore.p12

no changes added to commit (use "git add" and/or "git commit -a")
  
  Relevant status after performing the cergen. (Git commit not created yet as 
seen by the dirty working directory)

TASK DETAIL
  https://phabricator.wikimedia.org/T266470

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Gehel, Lea_Lacroix_WMDE, dcausse, Aklapper, MPhamWMF, maantietaja, 
Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, 
joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, 
Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, abian, jkroll, Wikidata-bugs, Jdouglas, 
aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T252504: Automate the smoke test of the canary deployment of WDQS

2021-03-08 Thread RKemper
RKemper renamed this task from "Smoke test the canary deployment of WDQS" to 
"Automate the smoke test of the canary deployment of WDQS".
RKemper updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T252504

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Aklapper, Zbyszko, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, 
Namenlos314, jijiki, Klaas_Z4us_V, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, elukey, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, abian, jkroll, Wikidata-bugs, Jdouglas, 
aude, Tobias1984, Manybubbles, Addshore, Mbch331, Jay8g
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-03-04 Thread RKemper
RKemper added a comment.


  Here's how the ats mapping looks afterdeploy of the `backend.yaml` changes:
  
ryankemper@cp1075:~$ sudo cat /etc/trafficserver/remap.config | grep query
map http://query-preview.wikidata.org/bigdata 
https://wdqs1009.eqiad.wmnet/bigdata
map http://query-preview.wikidata.org/sparql 
https://wdqs1009.eqiad.wmnet/sparql
map http://query-preview.wikidata.org 
https://webserver-misc-apps.discovery.wmnet
map http://query.wikidata.org/bigdata/ldf 
https://wdqs1005.eqiad.wmnet/bigdata/ldf
map http://query.wikidata.org/sparql https://wdqs.discovery.wmnet/sparql
map http://query.wikidata.org/bigdata https://wdqs.discovery.wmnet/bigdata
map http://query.wikidata.org https://webserver-misc-apps.discovery.wmnet

TASK DETAIL
  https://phabricator.wikimedia.org/T266470

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Gehel, Lea_Lacroix_WMDE, dcausse, Aklapper, MPhamWMF, maantietaja, 
Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, 
joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, 
Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, abian, jkroll, Wikidata-bugs, Jdouglas, 
aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-03-03 Thread RKemper
RKemper added a comment.


  Posting logs of our IRC convo from ~1 month ago for context when I tag people 
for review:
  
[2021-02-04 11:31:36]  bblack: around ?
[2021-02-04 11:31:58]  I'm here with ryankemper to talk about 
T266470, unless you want to jump in a meet
[2021-02-04 11:31:59]  T266470: Expose wdqs1009 to wdqs users and 
gather feedback - https://phabricator.wikimedia.org/T266470
[2021-02-04 11:32:51]  taking notes in 
https://etherpad.wikimedia.org/p/expose-wdqs1009
[2021-02-04 11:33:08]  gehel: hi :)
[2021-02-04 11:33:15]  o/
[2021-02-04 11:33:15]  ryankemper: hi too :)
[2021-02-04 11:33:23]  \o
[2021-02-04 11:33:40]  so, I guess my best understanding of the 
present stuff is what I see in:
[2021-02-04 11:33:47]  context: we want to expose wdqs1009 as a test 
server so that our users can make sure we're not breaking anything with the new 
WDQS updater
[2021-02-04 11:33:53]  
https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/hieradata/common/profile/trafficserver/backend.yaml#157
[2021-02-04 11:34:12]  there's several related definitions there 
for https://query.wikidata.org/
[2021-02-04 11:34:24]  for different subspaces of the URI routing 
to different backends
[2021-02-04 11:34:40]   /bigdata/ldf goes explicitly to wds1005
[2021-02-04 11:35:07]   /bigdata and /sparql both go to the 
discovery service to a set of wdqs
[2021-02-04 11:35:25]  but the root url hits our misc webapps 
setup, for the JS powering the query UI
[2021-02-04 11:35:29]  yep, somewhat similar, but in this case, we 
want to have a different FQDN (probably query-preview.wikidata.org or something 
similar) and route to a different server that isn't part of the current WDQS 
pools
[2021-02-04 11:36:45]  so in this case, I would guess, you'd want 
no special exception for /bigdata/ldf, and you'd want all of /bigdata and 
/sparql -> wdqs1009
[2021-02-04 11:37:06]  which is something we can just define in 
that file I linked, and adding a name at the DNS layer, and done
[2021-02-04 11:37:16]  but the root URL for the UI part, might be a 
little trickier
[2021-02-04 11:37:39]  I need to see where that's defined in git, 
and it might need a new site definition, and to template in a new place to send 
the UI queries...
[2021-02-04 11:37:57]  but there's no other special requirements I 
think/expect?
[2021-02-04 11:37:57]  we'll probably want to create another 
microsite for the JS part, that's fairly straightforward
[2021-02-04 11:38:15]  no other special requirements IIRC
[2021-02-04 11:38:17]  yep, that seems fairly simple to me
[2021-02-04 11:38:22]  so to resume:
[2021-02-04 11:39:08]  create a DNS entry (CNAME to dyna.wm.o), 
another set of entries in backend.yaml map, create another minisite (with the 
appropriate configuration)
[2021-02-04 11:39:22]  do we need to do anything for SSL certs? Or 
is that fully automated?
[2021-02-04 11:39:48]  that part's all shared between everything 
that runs through the cache, so long as you stick to our canonical domains and 
just change the leading hostname part
[2021-02-04 11:40:04]  to e.g. query-preview.wikidata.org
[2021-02-04 11:40:05]  "canonical" includes wikidata.org?
[2021-02-04 11:40:11]  ok, good!
[2021-02-04 11:40:37]  yeah for reference, the canonical set is: 
https://wikitech.wikimedia.org/wiki/HTTPS#For_the_Foundation's_canonical_domain_names
[2021-02-04 11:41:13]  ok, one last stupid and off topic question: 
what does "dyna" stands for in dyna.wm.o?
[2021-02-04 11:42:14]  it's not a "real" DNS RR-type you would see 
in the wild.  It's a custom thing our authserver implements which provides 
dynamic A and  records depending on geodns.  that's how we route to the 
closest of our geographic edges for users
[2021-02-04 11:42:36]  
https://github.com/gdnsd/gdnsd/wiki/GdnsdPluginGeoip
[2021-02-04 11:42:41]  ^ that, basically
[2021-02-04 11:42:57]  thanks!
[2021-02-04 11:43:23]  templates/wikimedia.org:dyna600 
IN DYNA geoip!text-addrs
[2021-02-04 11:43:36]  ^ is what all the other production CNAMEs 
are pointing towards
[2021-02-04 11:43:54]  gehel: also I don't know if you've created 
new service hostnames since the move to netbox
[2021-02-04 11:44:03]  but we don't do commits on the ops/dns repo 
for that kind of thing anymore
[2021-02-04 11:44:25]  Oh no, I havent
[2021-02-04 11:44:33]  ryankemper: is the one who's goign to do it
[2021-02-04 11:45:00]  I assume it's documented
[2021-02-04 11:45:01]  oh wait, I'm getting ahead of ourselves
[2021-02-04 11:45:03] — gehel is searching
[2021-02-04 11:45:13]  that part still is done in ops/dns ! :)
[2021-02-04 11:45:25]  now I'm curious anyway!
[2021-02-04 11:45:43]  
https://gerrit.wikimedia.org/r/plugins/gitiles/operations/dns/+/refs/heads

[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps

2021-02-25 Thread RKemper
RKemper added a comment.


  Note: Puppet is still disabled on `wdqs2008` while the reload runs. It 
occurred to me that I'm not sure if puppet actually **needs** to be disabled 
during data reloads or if that's just a precaution we've historically taken - 
any insight here @Gehel?

TASK DETAIL
  https://phabricator.wikimedia.org/T267927

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Zbyszko, Gehel, Aklapper, dcausse, MPhamWMF, CBogen, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T274751: Upgrade firmware on wdqs1009

2021-02-25 Thread RKemper
RKemper added a comment.


  @Cmjohnson The data reload is complete on `wdqs1009`, so the host can now 
have its firmware upgraded and be rebooted at its convenience. Note this is an 
internal wdqs test host, so there is no public-facing service for us to worry 
about.
  
  Feel free to proceed when convenient for you - just shoot me a ping or drop a 
line in #wikimedia-discovery, but no need to schedule a formal window.

TASK DETAIL
  https://phabricator.wikimedia.org/T274751

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Cmjohnson, RKemper
Cc: Jclark-ctr, RKemper, Cmjohnson, Gehel, MoritzMuehlenhoff, Ramtin0071, 
MPhamWMF, Devnull, lmata, Muchiri124, CBogen, Akuckartz, Legado_Shulgin, 
Nandana, Namenlos314, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, 
merbst, LawExplorer, Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, 
Southparkfan, faidon, Addshore, Mbch331, Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps

2021-02-25 Thread RKemper
RKemper added a comment.


  Downtimed `wdqs2008` until `2021-03-04 21:56:59`

TASK DETAIL
  https://phabricator.wikimedia.org/T267927

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Zbyszko, Gehel, Aklapper, dcausse, MPhamWMF, CBogen, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps

2021-02-08 Thread RKemper
RKemper added a comment.


  Still waiting for the latest dumps to be downloaded (few more hours), then 
need to reboot WDQS hosts as part of https://phabricator.wikimedia.org/T274213, 
then can do the actual data-reload

TASK DETAIL
  https://phabricator.wikimedia.org/T267927

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Aklapper, dcausse, MPhamWMF, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, 
Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T266495: Create Debian Package for Flink

2021-02-08 Thread RKemper
RKemper added a comment.


  (See https://phabricator.wikimedia.org/T273097#6805355 for why this ticket 
has been closed)

TASK DETAIL
  https://phabricator.wikimedia.org/T266495

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, dcausse, elukey, EBernhardson, Aklapper, Gehel, Mstyles, Ottomata, 
MPhamWMF, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T273636: Blazegraph journal for wcqs is too big

2021-02-08 Thread RKemper
RKemper added a comment.


  Updated:
  
  [X] https://commons.wikimedia.org/wiki/Commons:SPARQL_query_service#Updates
  [X] 
https://commons.wikimedia.org/wiki/Commons:Village_pump#Unscheduled_maintenance%3A_Wikimedia_Commons_Query_Service
  [X] WikiData ML mailing list (note: A community member pointed out that 
sending an e-mail to a Commons mailing list would have made more sense, which 
is a great point. We'll do that in the future)

TASK DETAIL
  https://phabricator.wikimedia.org/T273636

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: MPhamWMF, RKemper, Aklapper, dcausse, CBogen, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T273636: Blazegraph journal for wcqs is too big

2021-02-08 Thread RKemper
RKemper moved this task from In Progress to Needs Reporting on the 
Discovery-Search (Current work) board.
RKemper added a comment.


  WCQS is back in service; updating the notification channels right now and 
will comment back here after

TASK DETAIL
  https://phabricator.wikimedia.org/T273636

WORKBOARD
  https://phabricator.wikimedia.org/project/board/1227/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: MPhamWMF, RKemper, Aklapper, dcausse, CBogen, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T273636: Blazegraph journal for wcqs is too big

2021-02-08 Thread RKemper
RKemper claimed this task.

TASK DETAIL
  https://phabricator.wikimedia.org/T273636

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: MPhamWMF, RKemper, Aklapper, dcausse, CBogen, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps

2021-02-05 Thread RKemper
RKemper added a comment.


  `sudo cookbook sre.wdqs.data-reload wdqs1009.eqiad.wmnet 
--reuse-downloaded-dump --reload-data wikidata --skolemize --reason 'T267927: 
Reload wikidata jnl from fresh dumps' --task-id T267927 ` is failing with:
  
- OUTPUT of 'test -f /srv/wdq...test-all.ttl.bz2' -

- OUTPUT of 'test -f /srv/wdq...-lexemes.ttl.bz2' -

- OUTPUT of 'dump_size=`du /s...avail+$db_size))' -

- OUTPUT of 'rm -rf /srv/wdqs...d -- --skolemize' -
#logback.classic pattern: %d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - 
%msg%n
22:46:13.025 [main] ERROR org.wikidata.query.rdf.tool.Munge - Fatal error 
munging RDF
org.openrdf.rio.RDFParseException: Expected an RDF value here, found '{' 
[line 2]
at 
org.openrdf.rio.helpers.RDFParserHelper.reportFatalError(RDFParserHelper.java:440)
at 
org.openrdf.rio.helpers.RDFParserBase.reportFatalError(RDFParserBase.java:685)
at 
org.openrdf.rio.turtle.TurtleParser.reportFatalError(TurtleParser.java:1405)
at org.openrdf.rio.turtle.TurtleParser.parseValue(TurtleParser.java:662)
at 
org.openrdf.rio.turtle.TurtleParser.parsePredicate(TurtleParser.java:505)
at 
org.openrdf.rio.turtle.TurtleParser.parsePredicateObjectList(TurtleParser.java:420)
at 
org.openrdf.rio.turtle.TurtleParser.parseImplicitBlank(TurtleParser.java:613)
at 
org.openrdf.rio.turtle.TurtleParser.parseTriples(TurtleParser.java:394)
at 
org.openrdf.rio.turtle.TurtleParser.parseStatement(TurtleParser.java:259)
at org.openrdf.rio.turtle.TurtleParser.parse(TurtleParser.java:214)
at org.wikidata.query.rdf.tool.Munge.run(Munge.java:105)
at org.wikidata.query.rdf.tool.Munge.main(Munge.java:59)
22:46:13.024 
[org.wikidata.query.rdf.tool.rdf.AsyncRDFHandler$RDFActionsReplayer] INFO  
org.wikidata.query.rdf.tool.Munge - Switching to 
/srv/wdqs/munged/wikidump-1.ttl.gz

  
  `sudo cookbook sre.wdqs.data-reload wdqs1010.eqiad.wmnet 
--reuse-downloaded-dump --reload-data wikidata --reason 'T267927: Reload 
wikidata jnl from fresh dumps' --task-id T267927`is failing with:
  
ryankemper@cumin1001:~$ sudo cookbook sre.wdqs.data-reload 
wdqs1010.eqiad.wmnet --reuse-downloaded-dump --reload-data wikidata --reason 
'T267927: Reload wikidata jnl from fresh dumps' --task-id T267927
START - Cookbook sre.wdqs.data-reload
- OUTPUT of 'test -f /srv/wdq...test-all.ttl.bz2' -

PASS 
||
 100% (1/1) [00:00<00:00,  1.92hosts/s]
FAIL |  
  |   0% (0/1) 
[00:00= 100.0% threshold) for command: 'test -f 
/srv/wdq...test-all.ttl.bz2'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully 
executed all commands.
Detected dump (/srv/wdqs/latest-all.ttl.bz2). Skipping download
- OUTPUT of 'test -f /srv/wdq...-lexemes.ttl.bz2' -

PASS 
||
 100% (1/1) [00:00<00:00,  2.44hosts/s]
FAIL |  
  |   0% (0/1) 
[00:00= 100.0% threshold) for command: 'test -f 
/srv/wdq...-lexemes.ttl.bz2'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully 
executed all commands.
Detected dump (/srv/wdqs/latest-lexemes.ttl.bz2). Skipping download
checking available disk space
- OUTPUT of 'dump_size=`du /s...avail+$db_size))' -

PASS 
||
 100% (1/1) [00:00<00:00,  2.45hosts/s]
FAIL |  
  |   0% (0/1) 
[00:00= 100.0% threshold) for command: 
'dump_size=`du /s...avail+$db_size))'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully 
executed all commands.
Running munger for main database and then lexeme
munging /srv/wdqs/munged (skolemizaton: False)
- OUTPUT of 'rm -rf /srv/wdqs.../wdqs/munged -- ' -
#logback.classic pattern: %d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - 
%msg%n
22:50:03.137 [main] ERROR org.wikidata.query.rdf.tool.Munge - Fatal error 
munging RDF
org.openrdf.rio.RDFParseException: Expected an RDF value here, found '{' 
[line 2]
at 
org.openrdf.rio.helpers.RDFParserHelper.reportFatalError(RDFPar

[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-02-04 Thread RKemper
RKemper added a comment.


  TODO from IRC meeting with bblack/gehel: create a DNS entry (CNAME to 
dyna.wm.o), another set of entries in backend.yaml map, create another minisite 
(with the appropriate configuration)

TASK DETAIL
  https://phabricator.wikimedia.org/T266470

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Gehel, Lea_Lacroix_WMDE, dcausse, Aklapper, MPhamWMF, CBogen, Akuckartz, 
Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T273636: Blazegraph journal for wcqs is too big

2021-02-03 Thread RKemper
RKemper added a comment.


  Notified WikiData mailing list and also posted here: 
https://commons.wikimedia.org/wiki/Commons:SPARQL_query_service#Updates

TASK DETAIL
  https://phabricator.wikimedia.org/T273636

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, Aklapper, dcausse, MPhamWMF, CBogen, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T273636: Blazegraph journal for wcqs is too big

2021-02-03 Thread RKemper
RKemper added a comment.


  `wcqs-beta-01.eqiad.wmflabs` is running low on disk space due to its 
blazegraph journal dataset size. In order to free up space we will need to take 
the service down, delete the journal and re-import from the latest dump. 
Service interruption will begin at Feb 4 18:30 UTC and continue until the data 
reload is complete.

TASK DETAIL
  https://phabricator.wikimedia.org/T273636

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, Aklapper, dcausse, MPhamWMF, CBogen, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T273097: Create Flink Base Image

2021-02-03 Thread RKemper
RKemper added a comment.


  @akosiaris Is your concern with the idea of using a`flink` base image 
solution mainly just centered around the inefficiency/inconvenience of needing 
SRE to merge any flink version upgrades? Since we have an embedded SRE on 
search (me) and to a lesser extent Guillaume, I think it wouldn't be too much 
of a problem. In general having our dependencies managed by a docker image will 
make it easier for us to be explicit about what version we're using, and it 
seems like the default docker-y way of doing things. Is there a technical 
reason why a base image might not be a good idea?

TASK DETAIL
  https://phabricator.wikimedia.org/T273097

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, dcausse, Aklapper, Gehel, akosiaris, Mstyles, MPhamWMF, CBogen, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps

2021-02-03 Thread RKemper
RKemper added a comment.


  We'll want to reload these this Friday, because the latest dumps should be 
available thursday evening.

TASK DETAIL
  https://phabricator.wikimedia.org/T267927

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Aklapper, dcausse, MPhamWMF, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, 
Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T267927: Reload wikidata journal from fresh dumps

2021-02-01 Thread RKemper
RKemper claimed this task.

TASK DETAIL
  https://phabricator.wikimedia.org/T267927

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Aklapper, dcausse, MPhamWMF, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, 
Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T266470: Expose wdqs1009 to wdqs users and gather feedback

2021-02-01 Thread RKemper
RKemper triaged this task as "Medium" priority.
RKemper claimed this task.

TASK DETAIL
  https://phabricator.wikimedia.org/T266470

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Gehel, Lea_Lacroix_WMDE, dcausse, Aklapper, MPhamWMF, CBogen, Akuckartz, 
Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T273097: Create Flink Base Image

2021-02-01 Thread RKemper
RKemper updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T273097

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: dcausse, Aklapper, Gehel, akosiaris, Mstyles, MPhamWMF, CBogen, Akuckartz, 
Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T271851: Clean up gui from the wdqs deploy repo and puppet

2021-01-21 Thread RKemper
RKemper added a comment.


  In T271851#6765207 <https://phabricator.wikimedia.org/T271851#6765207>, 
@Ladsgroup wrote:
  
  > @RKemper I'm so sorry it was buggy, the cleanest way forward here is to 
deploy wcqs using microsites and we can easily drop everything from puppet 
then. Does that make sense?
  
  Makes sense! I'm working on getting the wcqs microsite working and then we 
can revert my earlier revert and nuke the remaining gui file at the same time. 
I'll tag you on the patch(es) when ready.

TASK DETAIL
  https://phabricator.wikimedia.org/T271851

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Ladsgroup, RKemper
Cc: Addshore, dcausse, RKemper, Gehel, Aklapper, Ladsgroup, MPhamWMF, 
Hazizibinmahdi, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, Mahir256, QZanden, EBjune, merbst, 
LawExplorer, Salgo60, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Lydia_Pintscher, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

2021-01-20 Thread RKemper
RKemper added a comment.


  Post deploy check:
  
  `wdqs-updater.service` is now running without the `--import-async` flag as 
expected:
  
  `ExecStart=/bin/bash /srv/deployment/wdqs/wdqs/runUpdate.sh -n wdq -- --kafka 
kafka-main1001.eqiad.wmnet:9092,kafka-main1002.eqiad.wmnet:9092,kafka-main1003.eqiad.wmnet:9092
 --consumer wdqs1003 -b 700 --clusters eqiad,codfw --constraints --oldRevision 
3 --entityNamespaces 0,120,146`
  
  I think now we just wait and see if the issue in the ticket description pops 
up again.

TASK DETAIL
  https://phabricator.wikimedia.org/T267175

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Skim, Strepon, Multichill, Zbyszko, RKemper, Epidosis, dcausse, 
Tagishsimon, Lydia_Pintscher, CBogen, Z_thomas, agray, Gehel, 
Lucas_Werkmeister_WMDE, Aklapper, M2k_dewiki, MPhamWMF, Alter-paule, Beast1978, 
Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, 
_jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

2021-01-19 Thread RKemper
RKemper added a comment.


  (Puppet run successful on wdqs nodes following deploy)

TASK DETAIL
  https://phabricator.wikimedia.org/T267175

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Skim, Strepon, Multichill, Zbyszko, RKemper, Epidosis, dcausse, 
Tagishsimon, Lydia_Pintscher, CBogen, Z_thomas, agray, Gehel, 
Lucas_Werkmeister_WMDE, Aklapper, M2k_dewiki, MPhamWMF, Alter-paule, Beast1978, 
Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, 
_jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T267175: SPARQL-Query shows entries, which should be filter out; number of entries in result set might change when executed repeatedly (possible caching/indexing problem)

2021-01-19 Thread RKemper
RKemper moved this task from In Progress to To Be Deployed on the 
Discovery-Search (Current work) board.
RKemper added a comment.


  Deploying https://gerrit.wikimedia.org/r/656833 now

TASK DETAIL
  https://phabricator.wikimedia.org/T267175

WORKBOARD
  https://phabricator.wikimedia.org/project/board/1227/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Skim, Strepon, Multichill, Zbyszko, RKemper, Epidosis, dcausse, 
Tagishsimon, Lydia_Pintscher, CBogen, Z_thomas, agray, Gehel, 
Lucas_Werkmeister_WMDE, Aklapper, M2k_dewiki, MPhamWMF, Alter-paule, Beast1978, 
Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, 
_jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T270476: Linked Data Fragments endpoint returns IllegalStateException

2021-01-12 Thread RKemper
RKemper added a comment.


  I responded to Maciej (the one who reported the problem originally) on the 
mailing list. Looks like we accidentally gave them the wrong ticket number 
earlier: https://lists.wikimedia.org/pipermail/wikidata/2021-January/014451.html
  
  Looks like this ticket hasn't been triaged/assigned yet

TASK DETAIL
  https://phabricator.wikimedia.org/T270476

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, Gehel, Aklapper, MPhamWMF, CBogen, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T271412: Investigate an alert on statements volume in Blazegraph instances

2021-01-11 Thread RKemper
RKemper updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T271412

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Aklapper, Zbyszko, MPhamWMF, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, 
Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T244753: Migrate WDQS to Debian Buster

2020-12-17 Thread RKemper
RKemper moved this task from In Progress to Needs Reporting on the 
Discovery-Search (Current work) board.
RKemper added a comment.


  This is all done. No issues currently.

TASK DETAIL
  https://phabricator.wikimedia.org/T244753

WORKBOARD
  https://phabricator.wikimedia.org/project/board/1227/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Aklapper, MoritzMuehlenhoff, Gehel, Ramtin0071, MPhamWMF, Devnull, lmata, 
Muchiri124, CBogen, Akuckartz, Legado_Shulgin, Nandana, Namenlos314, 
Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, 
Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Rxy, 
Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T270236: wdqs1005: disk space critical on /srv/

2020-12-16 Thread RKemper
RKemper triaged this task as "High" priority.

TASK DETAIL
  https://phabricator.wikimedia.org/T270236

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Aklapper, RKemper, MPhamWMF, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, 
Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T270236: wdqs1005: disk space critical on /srv/

2020-12-16 Thread RKemper
RKemper added a project: Wikidata-Query-Service.
Restricted Application added a project: Wikidata.

TASK DETAIL
  https://phabricator.wikimedia.org/T270236

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Aklapper, RKemper, MPhamWMF, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, 
Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T269872: wdqs-categories prometheus exporter failing on select wdqs instances

2020-12-14 Thread RKemper
RKemper moved this task from In Progress to Needs Reporting on the 
Discovery-Search (Current work) board.
RKemper added a comment.


  The above two patches (really one patch with a followup patch to fix a typo) 
seem to have fixed the problem. I'll want to circle back to verify we're 
getting all the metrics we expect, but at a minimum the prometheus blazegraph 
exporter for wdqs-categories is working again.

TASK DETAIL
  https://phabricator.wikimedia.org/T269872

WORKBOARD
  https://phabricator.wikimedia.org/project/board/1227/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Gehel, RKemper, Aklapper, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, 
Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, 
Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, 
_jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T269204: Some wdqs metrics changed when switching to python3

2020-12-14 Thread RKemper
RKemper added a comment.


  Restarting `prometheus-blazegraph-exporter-wdqs-blazegraph.service` after 
switching from Counter to Gauge now shows the correct `blazegraph_lastupdated` 
metric when running `curl localhost:9193` (9193 is the port for 
wdqs-blazegraph).
  
  Still waiting to see if the downstream consumers of this metric are working 
properly now, but that *should* have fixed it.

TASK DETAIL
  https://phabricator.wikimedia.org/T269204

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Gehel, RKemper, dcausse, Aklapper, lmata, Alter-paule, Beast1978, CBogen, 
Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, 
Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
herron, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Chicocvenancio, QZanden, 
EBjune, merbst, LawExplorer, Lewizho99, Volans, Maathavan, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T269872: wdqs-categories prometheus exporter failing on select wdqs instances

2020-12-10 Thread RKemper
RKemper renamed this task from "wdqs-categories prometheus exporter failing on 
wdqs1011" to "wdqs-categories prometheus exporter failing on select wdqs 
instances".
RKemper updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T269872

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, Aklapper, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T269872: wdqs-categories prometheus exporter failing on wdqs1011

2020-12-10 Thread RKemper
RKemper created this task.
RKemper added projects: Wikidata-Query-Service, Discovery-Search (Current work).
Restricted Application added a subscriber: Aklapper.
Restricted Application added a project: Wikidata.

TASK DESCRIPTION
ryankemper@wdqs1011:~$ sudo systemctl status 
prometheus-blazegraph-exporter-wdqs-categories.service
● prometheus-blazegraph-exporter-wdqs-categories.service - Prometheus 
Blazegraph Exporter (wdqs-categories)
   Loaded: loaded 
(/lib/systemd/system/prometheus-blazegraph-exporter-wdqs-categories.service; 
enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2020-12-10 17:10:22 UTC; 
12min ago
  Process: 21672 ExecStart=/usr/local/bin/prometheus-blazegraph-exporter 
--listen :9194 --port 9990 (code=exited, status=1/FAILURE)
 Main PID: 21672 (code=exited, status=1/FAILURE)

Dec 10 17:10:22 wdqs1011 systemd[1]: 
prometheus-blazegraph-exporter-wdqs-categories.service: Service 
RestartSec=100ms expired, scheduling restart.
Dec 10 17:10:22 wdqs1011 systemd[1]: 
prometheus-blazegraph-exporter-wdqs-categories.service: Scheduled restart job, 
restart counter is at 5.
Dec 10 17:10:22 wdqs1011 systemd[1]: Stopped Prometheus Blazegraph Exporter 
(wdqs-categories).
Dec 10 17:10:22 wdqs1011 systemd[1]: 
prometheus-blazegraph-exporter-wdqs-categories.service: Start request repeated 
too quickly.
Dec 10 17:10:22 wdqs1011 systemd[1]: 
prometheus-blazegraph-exporter-wdqs-categories.service: Failed with result 
'exit-code'.
Dec 10 17:10:22 wdqs1011 systemd[1]: Failed to start Prometheus Blazegraph 
Exporter (wdqs-categories).
  
  This exporter relies on `localhost:9194`, which is refusing connections 
currently. It seems like the actual categories service on the "other side" is 
failing:
  
ryankemper@wdqs1011:~$ sudo systemctl status wdqs-categories.service
● wdqs-categories.service - Query Service - Blazegraph - wdqs-categories
   Loaded: loaded (/lib/systemd/system/wdqs-categories.service; static; 
vendor preset: enabled)
   Active: active (running) since Thu 2020-12-10 17:04:48 UTC; 19min ago
 Main PID: 18590 (java)
Tasks: 114 (limit: 1)
   Memory: 2.3G
   CGroup: /system.slice/wdqs-categories.service
   └─18590 java -server -XX:+UseG1GC -Xmx8g 
-Xloggc:/var/log/wdqs/wdqs-categories_jvm_gc.%p-%t.log -XX:+PrintGCDetails 
-XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintAdaptiveSizePolicy -X

Dec 10 17:10:21 wdqs1011 wdqs-categories[18590]:   
SELECT ( COUNT( * ) AS ?count ) { ?s ?p ?o }
Dec 10 17:10:21 wdqs1011 wdqs-categories[18590]: } 
UNION {
Dec 10 17:10:21 wdqs1011 wdqs-categories[18590]:   
SELECT * WHERE { <http://www.wikidata.org> schema:dateModified ?y }
Dec 10 17:10:21 wdqs1011 wdqs-categories[18590]: } 
} req.requestURI=/bigdata/namespace/wdq/sparql, req.xForwardedFor=null, 
req.queryString=format=json=+prefix+schema%3A+%3Cht
Dec 10 17:10:21 wdqs1011 wdqs-categories[18590]: 
com.bigdata.rdf.sail.webapp.DatasetNotFoundException: namespace=wdq
Dec 10 17:10:21 wdqs1011 wdqs-categories[18590]: at 
com.bigdata.rdf.sail.BigdataSail$BigdataSailConnection.(BigdataSail.java:2071)
Dec 10 17:10:21 wdqs1011 wdqs-categories[18590]: Wrapped by: 
java.lang.RuntimeException: 
com.bigdata.rdf.sail.webapp.DatasetNotFoundException: namespace=wdq
Dec 10 17:10:21 wdqs1011 wdqs-categories[18590]: at 
com.bigdata.rdf.sail.BigdataSail.getReadOnlyConnection(BigdataSail.java:1506)
Dec 10 17:10:21 wdqs1011 wdqs-categories[18590]: Wrapped by: 
java.util.concurrent.ExecutionException: java.lang.RuntimeException: 
com.bigdata.rdf.sail.webapp.DatasetNotFoundException: namespace=wdq
Dec 10 17:10:21 wdqs1011 wdqs-categories[18590]: at 
java.util.concurrent.FutureTask.report(FutureTask.java:122)

TASK DETAIL
  https://phabricator.wikimedia.org/T269872

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, Aklapper, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T269693: mediawiki_job_wikidata-updateQueryServiceLag failing

2020-12-09 Thread RKemper
RKemper added a comment.


  Job lives here: 
https://github.com/wikimedia/mediawiki-extensions-Wikidata.org/blob/60c5f96ebf424b792077bb7c6b533a68702e7aea/maintenance/updateQueryServiceLag.php#L70
  
  I have a patch open here: 
https://gerrit.wikimedia.org/r/c/operations/puppet/+/646888 that addresses a 
different manifestation of this same issue with `blazegraph_lastupdated`. 
Currently it looks like we'll want to switch `blazegraph_lastupdated` from a 
`Counter` to a `Gauge`, but I need to figure out if changing the metric type 
will break the behavior in `updateQueryServiceLag.php` or elsewhere.

TASK DETAIL
  https://phabricator.wikimedia.org/T269693

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Dzahn, RLazarus, RKemper, Aklapper, Ramtin0071, Devnull, lmata, Muchiri124, 
CBogen, Akuckartz, Legado_Shulgin, Nandana, Namenlos314, Davinaclare77, 
Qtn1293, Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T269693: mediawiki_job_wikidata-updateQueryServiceLag failing

2020-12-09 Thread RKemper
RKemper added a comment.


  There's some context in the description of 
https://phabricator.wikimedia.org/T269204 that mentions that the counter metric 
`blazegraph_lastupdated` is now `blazegraph_lastupdated_total`, so if 
the`mediawiki_job_wikidata-updateQueryServiceLag`  job has to do with that 
metric then the re-image is likely the source of the problem

TASK DETAIL
  https://phabricator.wikimedia.org/T269693

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Dzahn, RLazarus, RKemper, Aklapper, Ramtin0071, Devnull, lmata, Muchiri124, 
CBogen, Akuckartz, Legado_Shulgin, Nandana, Namenlos314, Davinaclare77, 
Qtn1293, Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, Zppix, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Rxy, Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T259588: Reload categories once 1.36.0-wmf.3 is running on all groups

2020-12-08 Thread RKemper
RKemper moved this task from In Progress to Needs Reporting on the 
Discovery-Search (Current work) board.
RKemper added a comment.


  All categories reloads/transfers are complete

TASK DETAIL
  https://phabricator.wikimedia.org/T259588

WORKBOARD
  https://phabricator.wikimedia.org/project/board/1227/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, dcausse, Aklapper, Wilmanbeno, CBogen, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, jayvdb, 
Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T269302: Many files on Commons cannot be found in WCQS

2020-12-07 Thread RKemper
RKemper updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T269302

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Lydia_Pintscher, Vojtech.dostal, Dipsacus_fullonum, Aklapper, Muchiri124, 
CBogen, Nintendofan885, Akuckartz, Nandana, JKSTNK, Namenlos314, Lahi, 
PDrouin-WMF, Gq86, E1presidente, Ramsey-WMF, Cparle, Anooprao, SandraF_WMF, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, Tramullas, Acer, 
merbst, LawExplorer, Salgo60, Silverfish, Poyekhali, _jensen, rosalieper, 
Taiwania_Justo, Scott_WUaS, Jonas, Xmlizer, Susannaanas, Ixocactus, Wong128hk, 
Jane023, jkroll, Wikidata-bugs, Jdouglas, Base, matthiasmullie, aude, 
Tobias1984, El_Grafo, Dinoguy1000, Manybubbles, Ricordisamoa, Wesalius, 
Fabrice_Florin, Raymond, Steinsplitter, Mbch331, Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T269331: wdqs.data-reload cookbook fails when switching

2020-12-03 Thread RKemper
RKemper added a comment.


  Here's something interesting: On a server where I've recently run the 
`data-reload` for `categories` (`wdqs1006` in this case), the categories 
journal looks like so:
  
  `-rw-rw-r-- 1 blazegraph blazegraph  21G Dec  3 07:19 categories.jnl`
  
  Yet on a wdqs instance where I have not yet run the reload (`wdqs2007` for 
example), it looks like this:
  
  `-rw-rw-r-- 1 blazegraph blazegraph  23G Dec  3 07:32 categories.jnl`
  
  So the freshly reloaded instances have only a 21G `categories.jnl`, whereas 
the not-freshly-reloaded instances have a 23G `categories.jnl` - a full 2GB 
bigger. I'm not sure if that indicates a problem somewhere, but naively I would 
not expect the journal to be smaller after reloading.

TASK DETAIL
  https://phabricator.wikimedia.org/T269331

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, dcausse, Aklapper, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, 
Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T259588: Reload categories once 1.36.0-wmf.3 is running on all groups

2020-12-03 Thread RKemper
RKemper added a comment.


  Note: https://phabricator.wikimedia.org/T269331 was created due to an 
exception encountered every time we run the data reload cookbook. The exception 
only occurs at the very end, so it might not indicate an actual problem

TASK DETAIL
  https://phabricator.wikimedia.org/T259588

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, dcausse, Aklapper, Wilmanbeno, CBogen, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, jayvdb, 
Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T269204: Some wdqs metrics changed when switching to python3

2020-12-03 Thread RKemper
RKemper claimed this task.
RKemper updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T269204

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Gehel, RKemper, dcausse, Aklapper, lmata, CBogen, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, herron, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
Chicocvenancio, QZanden, EBjune, merbst, LawExplorer, Volans, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, Manybubbles, Mbch331, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T269204: Some wdqs metrics changed when switching to python3

2020-12-03 Thread RKemper
RKemper added a comment.


  `sudo -i wmf-auto-reimage-host --conftool -p T269204 wdqs2004.codfw.wmnet` is 
an example of how to reimage hosts (run from cumin)

TASK DETAIL
  https://phabricator.wikimedia.org/T269204

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Gehel, RKemper, dcausse, Aklapper, lmata, CBogen, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, herron, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
Chicocvenancio, QZanden, EBjune, merbst, LawExplorer, Volans, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, Manybubbles, Mbch331, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T259588: Reload categories once 1.36.0-wmf.3 is running on all groups

2020-11-30 Thread RKemper
RKemper moved this task from Ready for Development to In Progress on the 
Discovery-Search (Current work) board.
RKemper added a comment.


  Will start on this today

TASK DETAIL
  https://phabricator.wikimedia.org/T259588

WORKBOARD
  https://phabricator.wikimedia.org/project/board/1227/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, dcausse, Aklapper, Wilmanbeno, CBogen, Akuckartz, Nandana, 
Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, jayvdb, 
Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T255399: Prepare wdqs1009 to run the streaming updater

2020-11-16 Thread RKemper
RKemper moved this task from In Progress to Needs Reporting on the 
Discovery-Search (Current work) board.
RKemper added a comment.


  We need to do another reload but not under this ticket

TASK DETAIL
  https://phabricator.wikimedia.org/T255399

WORKBOARD
  https://phabricator.wikimedia.org/project/board/1227/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Gehel, Ottomata, Aklapper, dcausse, Alter-paule, Beast1978, CBogen, Un1tY, 
Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, 
Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T265868: wdqs2002 out of disk space

2020-10-19 Thread RKemper
RKemper moved this task from In Progress to Needs Reporting on the 
Discovery-Search (Current work) board.
RKemper added a comment.


  This should be resolved now; the transfer is complete and the node matches 
others in the fleet.
  
ryankemper@wdqs2002:~$ ls -lh /srv/wdqs/wikidata.jnl
-rw-rw-r-- 1 blazegraph blazegraph 806G Oct 19 22:57 /srv/wdqs/wikidata.jnl
  
  vs
  
ryankemper@wdqs2004:~$ ls -lh /srv/wdqs/wikidata.jnl
-rw-rw-r-- 1 blazegraph blazegraph 806G Oct 19 22:57 /srv/wdqs/wikidata.jnl

TASK DETAIL
  https://phabricator.wikimedia.org/T265868

WORKBOARD
  https://phabricator.wikimedia.org/project/board/1227/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Gehel, RKemper
Cc: RKemper, dcausse, Gehel, Aklapper, CBogen, Akuckartz, darthmon_wmde, 
Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T262009: Add an entry in the WDQS Runbook on killer queries

2020-10-14 Thread RKemper
RKemper updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T262009

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Aklapper, RKemper, Gehel, CBogen, Akuckartz, darthmon_wmde, Nandana, 
Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, 
EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T258835: Update Commons query service favicon

2020-09-24 Thread RKemper
RKemper moved this task from Needs review to Needs Reporting on the 
Discovery-Search (Current work) board.
RKemper added a comment.


  Applied this today

TASK DETAIL
  https://phabricator.wikimedia.org/T258835

WORKBOARD
  https://phabricator.wikimedia.org/project/board/1227/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse, RKemper
Cc: RKemper, Jarekt, CBogen, Gehel, Aklapper, Multichill, Akuckartz, 
darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86, Ramsey-WMF, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Poyekhali, _jensen, rosalieper, Taiwania_Justo, Scott_WUaS, Jonas, 
Xmlizer, Ixocactus, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, El_Grafo, Dinoguy1000, Manybubbles, Isarra, Steinsplitter, Mbch331, 
Keegan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T261204: Wikidata lexeme ttl dumps should be in a "predictable" folder

2020-09-12 Thread RKemper
RKemper moved this task from Needs review to Needs Reporting on the 
Discovery-Search (Current work) board.
RKemper added a comment.


  We'll want to monitor across the next week to verify all looks good.

TASK DETAIL
  https://phabricator.wikimedia.org/T261204

WORKBOARD
  https://phabricator.wikimedia.org/project/board/1227/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, ArielGlenn, dcausse, Alter-paule, jannee_e, Beast1978, CBogen, 
Un1tY, Akuckartz, Hook696, darthmon_wmde, Kent7301, joker88john, CucyNoiD, 
Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, 
Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Lunewa, QZanden, EBjune, 
merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, 
Jonas, Xmlizer, gnosygnu, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T261204: Wikidata lexeme ttl dumps should be in a "predictable" folder

2020-09-12 Thread RKemper
RKemper added a comment.


dumpsgen  58563  0.0  0.0   4276   700 ?Ss   20:50   0:00 /bin/sh 
-c python3 /srv/deployment/dumps/dumps/xmldumps-backup/generatemiscdumps.py 
--configfile /etc/dumps/confs/addschanges.conf --dumptype incrdumps --quiet
dumpsgen  58565  0.0  0.0  57976 18128 ?S20:50   0:03 python3 
/srv/deployment/dumps/dumps/xmldumps-backup/generatemiscdumps.py --configfile 
/etc/dumps/confs/addschanges.conf --dumptype incrdumps --quiet
dumpsgen  63246  3.9  0.0 456952 64728 ?S21:08   2:51 
/usr/bin/php7.2 /srv/mediawiki/multiversion/MWScript.php dumpTextPass.php 
--wiki=commonswiki 
--stub=gzip:/mnt/dumpsdata/otherdumps/incr/commonswiki/20200912/commonswiki-20200912-stubs-meta-hist-incr.xml.gz
 --quiet --spawn=/usr/bin/php7.2 
--output=bzip2:/mnt/dumpsdata/otherdumps/incr/commonswiki/20200912/commonswiki-20200912-pages-meta-hist-incr.xml.bz2
dumpsgen  63248  0.0  0.0   4276   740 ?S21:08   0:00 sh -c 
bzip2 > 
'/mnt/dumpsdata/otherdumps/incr/commonswiki/20200912/commonswiki-20200912-pages-meta-hist-incr.xml.bz2'
dumpsgen  63249  2.9  0.0  13604  8492 ?S21:08   2:08 bzip2
dumpsgen  63250  0.0  0.0   4276   736 ?S21:08   0:00 sh -c 
'/usr/bin/php7.2' 
'/srv/mediawiki/php-1.36.0-wmf.8/../multiversion/MWScript.php' 'fetchText.php' 
'--wiki' 'commonswiki'
dumpsgen  63251 18.5  0.0 453812 63484 ?S21:08  13:24 
/usr/bin/php7.2 /srv/mediawiki/php-1.36.0-wmf.8/../multiversion/MWScript.php 
fetchText.php --wiki commonswiki
  
  Dumps before merging. Nothing specific to `wikidata` is running so I just 
merged it now.

TASK DETAIL
  https://phabricator.wikimedia.org/T261204

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, ArielGlenn, dcausse, Alter-paule, jannee_e, Beast1978, CBogen, 
Un1tY, Akuckartz, Hook696, darthmon_wmde, Kent7301, joker88john, CucyNoiD, 
Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, 
Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Lunewa, QZanden, EBjune, 
merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, 
Jonas, Xmlizer, gnosygnu, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T258739: wdqs admins should have access to nginx logs, jstack on wdqs machines

2020-07-27 Thread RKemper
RKemper added a comment.


  We discussed this during this week's SRE meeting and resolved to enable full 
root access for wdqs admins, rather than granularly expanding access one file 
at a time. This also lines up better with how we currently manage 
Elasticsearch, where admins have root access.

TASK DETAIL
  https://phabricator.wikimedia.org/T258739

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Joe, RKemper
Cc: herron, RKemper, CDanis, dcausse, Aklapper, Dzahn, lmata, Alter-paule, 
Beast1978, CBogen, Un1tY, Akuckartz, Hook696, darthmon_wmde, Kent7301, 
joker88john, Legado_Shulgin, CucyNoiD, Nandana, Namenlos314, jijiki, 
Klaas_Z4us_V, Gaboe420, Giuliamocci, Davinaclare77, Qtn1293, Cpaulf30, 
Techguru.pc, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, 
Lewizho99, Zppix, Maathavan, elukey, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, mark, JanZerebecki, faidon, Mbch331, Rxy, Jay8g, Krenair, 
fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T258739: wdqs admins should have access to nginx logs, jstack on wdqs machines

2020-07-23 Thread RKemper
RKemper added a project: Sustainability (Incident Followup).

TASK DETAIL
  https://phabricator.wikimedia.org/T258739

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, CDanis, dcausse, Aklapper, Dzahn, lmata, Alter-paule, Beast1978, 
CBogen, Un1tY, Akuckartz, Hook696, darthmon_wmde, Kent7301, joker88john, 
Legado_Shulgin, CucyNoiD, Nandana, Namenlos314, jijiki, Klaas_Z4us_V, Gaboe420, 
Giuliamocci, Davinaclare77, Qtn1293, Cpaulf30, Techguru.pc, Lahi, Gq86, Af420, 
Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, 
QZanden, EBjune, merbst, LawExplorer, Lewizho99, Zppix, Maathavan, elukey, 
_jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, mark, JanZerebecki, 
faidon, Mbch331, Rxy, Jay8g, Krenair, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T258739: wdqs admins should have access to nginx logs, jstack on wdqs machines

2020-07-23 Thread RKemper
RKemper renamed this task from "wdqs admins should have access to nginx logs on 
wdqs machines" to "wdqs admins should have access to nginx logs, jstack on wdqs 
machines".

TASK DETAIL
  https://phabricator.wikimedia.org/T258739

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, CDanis, dcausse, Aklapper, Dzahn, lmata, Alter-paule, Beast1978, 
CBogen, Un1tY, Akuckartz, Hook696, darthmon_wmde, Kent7301, joker88john, 
Legado_Shulgin, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, 
Davinaclare77, Qtn1293, Cpaulf30, Techguru.pc, Lahi, Gq86, Af420, Bsandipan, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, 
merbst, LawExplorer, Lewizho99, Zppix, Maathavan, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, Manybubbles, mark, JanZerebecki, faidon, Mbch331, Rxy, Jay8g, 
Krenair, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] [Unblock] T230588: Wikidata Query Service is swapping items and properties

2020-05-19 Thread RKemper
RKemper closed subtask T252068: WQDS Data Reload as Resolved.

TASK DETAIL
  https://phabricator.wikimedia.org/T230588

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Zbyszko, RKemper
Cc: Ghuron, Nikki, Toni_001, Dipsacus_fullonum, Larske, Mathew.onipe, Esc3300, 
agray, Jakub.klimek, Gehel, bennofs, Magnus, Robby, Mmarx, jeremyb, 
Lea_Lacroix_WMDE, Manu1400, Lucas_Werkmeister_WMDE, Smalyshev, Moebeus, 
Aklapper, Fnielsen, CBogen, darthmon_wmde, Nandana, Lahi, Gq86, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] [Unblock] T240831: Sometimes Q77104211 shows up in sparql query as having a wrong value for P2397, however item is deleted since December 3rd 2019

2020-05-19 Thread RKemper
RKemper closed subtask T252068: WQDS Data Reload as Resolved.

TASK DETAIL
  https://phabricator.wikimedia.org/T240831

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Gehel, Aklapper, Mbch331, CBogen, darthmon_wmde, Nandana, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] [Closed] T252068: WQDS Data Reload

2020-05-19 Thread RKemper
RKemper closed this task as "Resolved".
RKemper added a comment.


  Still need to circle back to dependent tickets and verify that those problems 
are solved

TASK DETAIL
  https://phabricator.wikimedia.org/T252068

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Gehel, RKemper, Aklapper, CBogen, darthmon_wmde, Legado_Shulgin, Nandana, 
Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, 
Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Rxy, 
Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] [Unblock] T245135: Too many values for wikibase:geoLatitude and wikibase:geoLongitude in WDQS

2020-05-19 Thread RKemper
RKemper closed subtask T252068: WQDS Data Reload as Resolved.

TASK DETAIL
  https://phabricator.wikimedia.org/T245135

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Lucas_Werkmeister_WMDE, Aklapper, Dipsacus_fullonum, CBogen, darthmon_wmde, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, 
_jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, 
Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] [Commented On] T252068: WQDS Data Reload

2020-05-19 Thread RKemper
RKemper added a comment.


  Data transfer is done across all instances as of last friday, along with a 
wdqs-categories reload that we tacked on.
  
  Circling back to mark this ticket as done.

TASK DETAIL
  https://phabricator.wikimedia.org/T252068

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: Gehel, RKemper, Aklapper, CBogen, darthmon_wmde, Legado_Shulgin, Nandana, 
Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, 
Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Rxy, 
Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] [Commented On] T252068: WQDS Data Reload

2020-05-06 Thread RKemper
RKemper added a comment.


  should also circle back and fix 
https://github.com/wikimedia/puppet/blob/production/manifests/site.pp#L2207-L2242,`
 wdqs200[78]` are declared twice

TASK DETAIL
  https://phabricator.wikimedia.org/T252068

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, Aklapper, CBogen, darthmon_wmde, Legado_Shulgin, Nandana, 
Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, 
Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Rxy, 
Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] [Commented On] T252068: WQDS Data Reload

2020-05-06 Thread RKemper
RKemper added a comment.


  `wdqs1010` is one of our test servers
  
  Therefore can screw up `wdqs1010` as much as we want, but not the others
  
  ---
  
  we need to transfer that data to each server
  which implies we need to open a port for data transfer
  
  we have 2 primary DC: eqiad and codfw
  due to the magic of geo DNS, the servers in codfw see much less traffic than 
eqiad
  in each DC, we have 2 wdqs clusters, one public, one internal
  one place to find which server is where: 
https://github.com/wikimedia/puppet/blob/production/manifests/site.pp#L2207-L2242
  
  
https://github.com/wikimedia/puppet/blob/production/manifests/site.pp#L2207-L2242
 to find where each server is

TASK DETAIL
  https://phabricator.wikimedia.org/T252068

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, Aklapper, CBogen, darthmon_wmde, Legado_Shulgin, Nandana, 
Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, merbst, LawExplorer, 
Zppix, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Wong128hk, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon, Mbch331, Rxy, 
Jay8g, fgiunchedi
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] [Created] T252068: WQDS Data Reload

2020-05-06 Thread RKemper
RKemper created this task.
RKemper added a project: Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.
Restricted Application added a project: Wikidata.

TASK DESCRIPTION
  We have already done the reload on `wdqs1010`, so now we need to xfer to the 
various other relevant servers

TASK DETAIL
  https://phabricator.wikimedia.org/T252068

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, Aklapper, CBogen, darthmon_wmde, Nandana, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, 
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs