[Wikidata-bugs] [Maniphest] T280108: WDQS data-transfer cookbook needs to wait for updater to catchup on lag

[Gehel]

Gehel closed this task as "Resolved".
Gehel claimed this task.

TASK DETAIL
  https://phabricator.wikimedia.org/T280108

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Gehel
Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T280108: WDQS data-transfer cookbook needs to wait for updater to catchup on lag

[Maintenance_bot]

Maintenance_bot removed a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T280108

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Maintenance_bot
Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331, Alter-paule, Beast1978, Un1tY, Hook696, Kent7301, 
joker88john, CucyNoiD, Gaboe420, Giuliamocci, Cpaulf30, Af420, Bsandipan, 
Lewizho99, Maathavan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T280108: WDQS data-transfer cookbook needs to wait for updater to catchup on lag

[gerritbot]

gerritbot added a comment.


  Change 680095 **merged** by Ryan Kemper:
  
  [operations/cookbooks@master] wdqs: int can't take in float as string
  
  https://gerrit.wikimedia.org/r/680095

TASK DETAIL
  https://phabricator.wikimedia.org/T280108

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, Alter-paule, 
Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, 
Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, 
Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T280108: WDQS data-transfer cookbook needs to wait for updater to catchup on lag

[MPhamWMF]

MPhamWMF moved this task from All WDQS-related tasks to Current work on the 
Wikidata-Query-Service board.
MPhamWMF added a project: Discovery-Search (Current work).

TASK DETAIL
  https://phabricator.wikimedia.org/T280108

WORKBOARD
  https://phabricator.wikimedia.org/project/board/891/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: MPhamWMF
Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, Alter-paule, 
Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, 
Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, 
Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T280108: WDQS data-transfer cookbook needs to wait for updater to catchup on lag

[gerritbot]

gerritbot added a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T280108

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, Alter-paule, 
Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, 
Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, 
Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T280108: WDQS data-transfer cookbook needs to wait for updater to catchup on lag

[gerritbot]

gerritbot added a comment.


  Change 680095 had a related patch set uploaded (by Ryan Kemper; author: Ryan 
Kemper):
  
  [operations/cookbooks@master] wdqs: int can't take in float as string
  
  https://gerrit.wikimedia.org/r/680095

TASK DETAIL
  https://phabricator.wikimedia.org/T280108

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T280108: WDQS data-transfer cookbook needs to wait for updater to catchup on lag

[Stashbot]

Stashbot added a comment.


  Mentioned in SAL (#wikimedia-operations) [2021-04-15T22:46:40Z]  
T280108  T267927 
 Manually re-enabled and ran puppet 
on `wdqs1005` (had closed the tmux pane which terminated the cookbook without 
letting it do its final cleanup)

TASK DETAIL
  https://phabricator.wikimedia.org/T280108

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Stashbot
Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T280108: WDQS data-transfer cookbook needs to wait for updater to catchup on lag

[Stashbot]

Stashbot added a comment.


  Mentioned in SAL (#wikimedia-operations) [2021-04-15T22:33:28Z]  
T280108  T267927 
 Data transfers completed 
successfully; small issue with new `wait_for_updater` logic is preventing 
termination so I ctrl+c'd manually

TASK DETAIL
  https://phabricator.wikimedia.org/T280108

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Stashbot
Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T280108: WDQS data-transfer cookbook needs to wait for updater to catchup on lag

[RKemper]

RKemper added a comment.


100.0% (2/2) success ratio (>= 100.0% threshold) for command: 'systemctl 
start ...art wdqs-updater'.
100.0% (2/2) success ratio (>= 100.0% threshold) of nodes successfully 
executed all commands.
[1/1000, retrying in 600.00s] Attempt to run 
'cookbooks.sre.wdqs.wait_for_updater' raised: invalid literal for int() with 
base 10: 'NaN'
[2/1000, retrying in 600.00s] Attempt to run 
'cookbooks.sre.wdqs.wait_for_updater' raised: invalid literal for int() with 
base 10: '4458.16434332'
[3/1000, retrying in 600.00s] Attempt to run 
'cookbooks.sre.wdqs.wait_for_updater' raised: invalid literal for int() with 
base 10: '4230.269809265'
[4/1000, retrying in 600.00s] Attempt to run 
'cookbooks.sre.wdqs.wait_for_updater' raised: invalid literal for int() with 
base 10: '3850.3779346'
[5/1000, retrying in 600.00s] Attempt to run 
'cookbooks.sre.wdqs.wait_for_updater' raised: invalid literal for int() with 
base 10: '3175.483400543'
[6/1000, retrying in 600.00s] Attempt to run 
'cookbooks.sre.wdqs.wait_for_updater' raised: invalid literal for int() with 
base 10: '888.591801086'
[7/1000, retrying in 600.00s] Attempt to run 
'cookbooks.sre.wdqs.wait_for_updater' raised: invalid literal for int() with 
base 10: '39.694332428'
[8/1000, retrying in 600.00s] Attempt to run 
'cookbooks.sre.wdqs.wait_for_updater' raised: invalid literal for int() with 
base 10: '42.79952316284'
  
  Another small issue. Looks like we need to tell it to truncate (or just use 
doubles instead of ints)

TASK DETAIL
  https://phabricator.wikimedia.org/T280108

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T280108: WDQS data-transfer cookbook needs to wait for updater to catchup on lag

[Maintenance_bot]

Maintenance_bot removed a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T280108

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Maintenance_bot
Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331, Alter-paule, Beast1978, Un1tY, Hook696, Kent7301, 
joker88john, CucyNoiD, Gaboe420, Giuliamocci, Cpaulf30, Af420, Bsandipan, 
Lewizho99, Maathavan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T280108: WDQS data-transfer cookbook needs to wait for updater to catchup on lag

[Stashbot]

Stashbot added a comment.


  Mentioned in SAL (#wikimedia-operations) [2021-04-15T16:21:44Z]  
T280108  T267927 
 Current wdqs transfers in progress: 
`wqds1004`->`wdqs1005`, `wdqs2008`->`wdqs2001`

TASK DETAIL
  https://phabricator.wikimedia.org/T280108

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Stashbot
Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, Alter-paule, 
Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, 
Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, 
Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T280108: WDQS data-transfer cookbook needs to wait for updater to catchup on lag

[Stashbot]

Stashbot added a comment.


  Mentioned in SAL (#wikimedia-operations) [2021-04-15T16:17:13Z]  
T280108  T267927 
 Merged 
https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/679702 and ran 
puppet-agent on `cumin2001` before next round of wdqs `data-transfer`s

TASK DETAIL
  https://phabricator.wikimedia.org/T280108

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Stashbot
Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, Alter-paule, 
Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, 
Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, 
Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T280108: WDQS data-transfer cookbook needs to wait for updater to catchup on lag

[gerritbot]

gerritbot added a comment.


  Change 679702 **merged** by Ryan Kemper:
  
  [operations/cookbooks@master] wdqs: result is list not dictionary
  
  https://gerrit.wikimedia.org/r/679702

TASK DETAIL
  https://phabricator.wikimedia.org/T280108

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, Alter-paule, 
Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, 
Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, 
Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T280108: WDQS data-transfer cookbook needs to wait for updater to catchup on lag

[RKemper]

RKemper added a comment.


  (For future reference)
  
import logging
logging.basicConfig(level=logging.DEBUG)
from spicerack import Spicerack
spicerack = Spicerack(verbose=True, dry_run=True)


def site(host):
"""Hacky way to get the site in which a host is located."""
if 'eqiad' in str(host):
return 'eqiad'
if 'codfw' in str(host):
return 'codfw'
raise ValueError('Site is unknown for {host}.'.format(host=host))

remote = spicerack.remote()
prometheus = spicerack.prometheus()

remote_host = remote.query('wdqs2008.codfw.wmnet')

host = remote_host.hosts[0].split(".")[0]
query = "scalar(time() - blazegraph_lastupdated{instance='%s:9193'})" % host
result = prometheus.query(query, site(remote_host))
last_updated = int(result[1])
if last_updated > 1200:
raise ValueError("Let's wait for updater to catch up, current 
last_updated of {} is too high.".format(last_updated))
  
  Snippet used to diagnose the above issue without having to actually run the 
cookbook

TASK DETAIL
  https://phabricator.wikimedia.org/T280108

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, Alter-paule, 
Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, 
Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, 
Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T280108: WDQS data-transfer cookbook needs to wait for updater to catchup on lag

[gerritbot]

gerritbot added a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T280108

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, Alter-paule, 
Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, 
Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, 
Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T280108: WDQS data-transfer cookbook needs to wait for updater to catchup on lag

[gerritbot]

gerritbot added a comment.


  Change 679702 had a related patch set uploaded (by Ryan Kemper; author: Ryan 
Kemper):
  
  [operations/cookbooks@master] wdqs: result is list not dictionary
  
  https://gerrit.wikimedia.org/r/679702

TASK DETAIL
  https://phabricator.wikimedia.org/T280108

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T280108: WDQS data-transfer cookbook needs to wait for updater to catchup on lag

[RKemper]

RKemper added a comment.


  Tested out the new cookbook, there's at least one small issue:
  
Exception raised while executing cookbook sre.wdqs.data-transfer:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/spicerack/_menu.py", line 234, in run
raw_ret = runner.run()
  File "/usr/lib/python3/dist-packages/spicerack/_module_api.py", line 18, 
in run
return self._run(self.args, self.spicerack)
  File "/srv/deployment/spicerack/cookbooks/sre/wdqs/data-transfer.py", 
line 179, in run
wait_for_updater(prometheus, site(source), source)
  File "/usr/lib/python3/dist-packages/spicerack/decorators.py", line 104, 
in wrapper
return func(*args, **kwargs)  # type: ignore
  File "/srv/deployment/spicerack/cookbooks/sre/wdqs/__init__.py", line 25, 
in wait_for_updater
last_updated = int(result['value'][1])
TypeError: list indices must be integers or slices, not str
END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99)
  
  Error seems quite straightforward - we'll want to patch this.

TASK DETAIL
  https://phabricator.wikimedia.org/T280108

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: RKemper
Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T280108: WDQS data-transfer cookbook needs to wait for updater to catchup on lag

[Stashbot]

Stashbot added a comment.


  Mentioned in SAL (#wikimedia-operations) [2021-04-15T06:33:53Z]  
!log T280108  T267927 
 `data-transfer` to `wdqs1004` was 
successful; cookbook failed due to a newly introduced minor type error that 
didn't effect the transfer itself

TASK DETAIL
  https://phabricator.wikimedia.org/T280108

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Stashbot
Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T280108: WDQS data-transfer cookbook needs to wait for updater to catchup on lag

[Stashbot]

Stashbot added a comment.


  Mentioned in SAL (#wikimedia-operations) [2021-04-15T04:14:14Z]  
T280108  T267927 
  `wdqs2008` (source) caught up on 
lag, xfering to `wdqs1004`: `sudo -i cookbook sre.wdqs.data-transfer --source 
wdqs2008.codfw.wmnet --dest wdqs1004.eqiad.wmnet --reason "transferring 
wikidata journal following reload from dumps" --blazegraph_instance blazegraph 
--task-id T267927`

TASK DETAIL
  https://phabricator.wikimedia.org/T280108

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Stashbot
Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T280108: WDQS data-transfer cookbook needs to wait for updater to catchup on lag

[Maintenance_bot]

Maintenance_bot removed a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T280108

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Maintenance_bot
Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331, Alter-paule, Beast1978, Un1tY, Hook696, Kent7301, 
joker88john, CucyNoiD, Gaboe420, Giuliamocci, Cpaulf30, Af420, Bsandipan, 
Lewizho99, Maathavan
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T280108: WDQS data-transfer cookbook needs to wait for updater to catchup on lag

[Stashbot]

Stashbot added a comment.


  Mentioned in SAL (#wikimedia-operations) [2021-04-15T04:06:00Z]  
T280108  T267927 
 Merged 
https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/679320, will verify 
correct behavior of `data-transfer` cookbook

TASK DETAIL
  https://phabricator.wikimedia.org/T280108

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Stashbot
Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, Alter-paule, 
Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, 
Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, 
Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T280108: WDQS data-transfer cookbook needs to wait for updater to catchup on lag

[gerritbot]

gerritbot added a comment.


  Change 679320 **merged** by Ryan Kemper:
  
  [operations/cookbooks@master] WDQS: Wait for updater to catchup during data 
transfer.
  
  https://gerrit.wikimedia.org/r/679320

TASK DETAIL
  https://phabricator.wikimedia.org/T280108

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, Alter-paule, 
Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, 
Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, 
Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T280108: WDQS data-transfer cookbook needs to wait for updater to catchup on lag

[gerritbot]

gerritbot added a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T280108

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, Alter-paule, 
Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, 
Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, 
Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T280108: WDQS data-transfer cookbook needs to wait for updater to catchup on lag

[gerritbot]

gerritbot added a comment.


  Change 679320 had a related patch set uploaded (by Gehel; author: Gehel):
  
  [operations/cookbooks@master] WDQS: Wait for updater to catchup during data 
transfer.
  
  https://gerrit.wikimedia.org/r/679320

TASK DETAIL
  https://phabricator.wikimedia.org/T280108

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T280108: WDQS data-transfer cookbook needs to wait for updater to catchup on lag

[Gehel]

Gehel added a parent task: T267927: Reload wikidata journal from fresh dumps.

TASK DETAIL
  https://phabricator.wikimedia.org/T280108

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Gehel
Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T280108: WDQS data-transfer cookbook needs to wait for updater to catchup on lag

[Gehel]

Gehel updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T280108

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Gehel
Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T280108: WDQS data-transfer cookbook needs to wait for updater to catchup on lag

[Gehel]

Gehel created this task.
Gehel added a project: Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.
Restricted Application added a project: Wikidata.

TASK DESCRIPTION
  As a user of WDQS, I only want non-lagged servers to be pooled.
  As an operator of WDQS I don't want to be alerted transient issues that are 
part of normal operations during data reload.
  
  WDQS data-transfer cookbook needs to wait for updater to catchup on lag 
before removing icinga downtime and repooling servers.
  
  We already have code to wait for updater to catch up on lag in the 
data-reload cookbook. This code should be moved to Spicerack (so that it is in 
a central location) and should be used for both the data-reload and 
data-transfer cookbooks.
  
  AC:
  
  - no alert are raised during normal operations of the data-transfer cookbook
  - wdqs servers are not pooled until they are fully catched up on lag

TASK DETAIL
  https://phabricator.wikimedia.org/T280108

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Gehel
Cc: RKemper, Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs