ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/306436
Change subject: handle prereq properly when a previous dump task has failed
rather than not run
..
handle prereq properly when a previous
ArielGlenn has submitted this change and it was merged.
Change subject: scheduler: move command retry updating into its own method
..
scheduler: move command retry updating into its own method
Change-Id:
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/306291
Change subject: scheduler: move command retry updating into its own method
..
scheduler: move command retry updating into its own method
ArielGlenn has submitted this change and it was merged.
Change subject: scheduler: move population of dict for opts out to module
function
..
scheduler: move population of dict for opts out to module function
also toss unused
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/306233
Change subject: scheduler: move population of dict for opts out to module
function
..
scheduler: move population of dict for opts out to
ArielGlenn has submitted this change and it was merged.
Change subject: scheduler: move setup of stdin for commands out to a module
function
..
scheduler: move setup of stdin for commands out to a module function
Change-Id:
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/306231
Change subject: scheduler: move setup of stdin for commands out to a module
function
..
scheduler: move setup of stdin for commands out
ArielGlenn has submitted this change and it was merged.
Change subject: scheduler: move log level setup out to a module function
..
scheduler: move log level setup out to a module function
Change-Id:
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/306230
Change subject: scheduler: move log level setup out to a module function
..
scheduler: move log level setup out to a module function
ArielGlenn has submitted this change and it was merged.
Change subject: scheduler: move plugables setup out into a module function
..
scheduler: move plugables setup out into a module function
Change-Id:
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/306229
Change subject: scheduler: move plugables setup out into a module function
..
scheduler: move plugables setup out into a module function
ArielGlenn has submitted this change and it was merged.
Change subject: convert options and flags into dict for easier handling
..
convert options and flags into dict for easier handling
Change-Id:
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/306228
Change subject: convert options and flags into dict for easier handling
..
convert options and flags into dict for easier handling
ArielGlenn has submitted this change and it was merged.
Change subject: move id string setup into a module-level function
..
move id string setup into a module-level function
it's intended to be called once before anything
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/306227
Change subject: move id string setup into a module-level function
..
move id string setup into a module-level function
it's intended to
ArielGlenn has submitted this change and it was merged.
Change subject: scheduler: change scheduler class to take dict of plugables
..
scheduler: change scheduler class to take dict of plugables
We can add to this dict as
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/306226
Change subject: scheduler: change scheduler class to take dict of plugables
..
scheduler: change scheduler class to take dict of plugables
ArielGlenn has submitted this change and it was merged.
Change subject: scheduler: move process result checking out to a class
..
scheduler: move process result checking out to a class
Change-Id:
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/306224
Change subject: scheduler: move process result checking out to a class
..
scheduler: move process result checking out to a class
ArielGlenn has submitted this change and it was merged.
Change subject: scheduler: split up the check for running processses
..
scheduler: split up the check for running processses
We have two types of checks now, one by Popen
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/306215
Change subject: scheduler: split up the check for running processses
..
scheduler: split up the check for running processses
We have two
ArielGlenn has submitted this change and it was merged.
Change subject: scheduler: move generation of unique id string out to the module
..
scheduler: move generation of unique id string out to the module
We need this to get
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/306212
Change subject: scheduler: move generation of unique id string out to the module
..
scheduler: move generation of unique id string out to
ArielGlenn has submitted this change and it was merged.
Change subject: scheduler: move slot related stuff out into a class for
resource allocation
..
scheduler: move slot related stuff out into a class for resource allocation
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/306211
Change subject: scheduler: move slot related stuff out into a class for
resource allocation
..
scheduler: move slot related stuff out
ArielGlenn has submitted this change and it was merged.
Change subject: scheduler: make pylint not whine about unused args to signal
handler
..
scheduler: make pylint not whine about unused args to signal handler
also fix up
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/306207
Change subject: scheduler: make pylint not whine about unused args to signal
handler
..
scheduler: make pylint not whine about unused
ArielGlenn has submitted this change and it was merged.
Change subject: scheduler: move email related methods into a class
..
scheduler: move email related methods into a class
also, break up the email sender function into
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/306203
Change subject: scheduler: move email related methods into a class
..
scheduler: move email related methods into a class
also, break up
ArielGlenn has submitted this change and it was merged.
Change subject: scheduler: move cache related methods into a class
..
scheduler: move cache related methods into a class
Change-Id:
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/306201
Change subject: scheduler: move cache related methods into a class
..
scheduler: move cache related methods into a class
Change-Id:
ArielGlenn has submitted this change and it was merged.
Change subject: scheduler: move methods out into module that don't need 'self'
..
scheduler: move methods out into module that don't need 'self'
Change-Id:
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/306200
Change subject: scheduler: move methods out into module that don't need 'self'
..
scheduler: move methods out into module that don't need
ArielGlenn has submitted this change and it was merged.
Change subject: scheduler: make all except clauses catch specific exceptions
..
scheduler: make all except clauses catch specific exceptions
Change-Id:
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/306198
Change subject: scheduler: make all except clauses catch specific exceptions
..
scheduler: make all except clauses catch specific
ArielGlenn has submitted this change and it was merged.
Change subject: scheduler: make json default object handler do something a bit
more useful
..
scheduler: make json default object handler do something a bit more useful
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/306197
Change subject: scheduler: make json default object handler do something a bit
more useful
..
scheduler: make json default object handler
ArielGlenn has submitted this change and it was merged.
Change subject: scheduler: on HUP, clean up zombies after killing child
processes
..
scheduler: on HUP, clean up zombies after killing child processes
Change-Id:
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/305982
Change subject: scheduler: on HUP, clean up zombies after killing child
processes
..
scheduler: on HUP, clean up zombies after killing
ArielGlenn has submitted this change and it was merged.
Change subject: fix up prereq option in bash wrapper script for dumps
..
fix up prereq option in bash wrapper script for dumps
Change-Id:
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/305893
Change subject: fix up prereq option in bash wrapper script for dumps
..
fix up prereq option in bash wrapper script for dumps
Change-Id:
ArielGlenn has submitted this change and it was merged.
Change subject: enable runs of missing prereq steps for dumps at later step
..
enable runs of missing prereq steps for dumps at later step
Change-Id:
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/305872
Change subject: enable runs of missing prereq steps for dumps at later step
..
enable runs of missing prereq steps for dumps at later step
ArielGlenn has submitted this change and it was merged.
Change subject: reduce cronspam, eliminate dataset rsync whines about vanished
files
..
reduce cronspam, eliminate dataset rsync whines about vanished files
followup to
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/305621
Change subject: reduce cronspam, eliminate dataset rsync whines about vanished
files
..
reduce cronspam, eliminate dataset rsync whines
ArielGlenn has submitted this change and it was merged.
Change subject: reduce cronspam: filter out dataset rsync error messages about
vanishing files
..
reduce cronspam: filter out dataset rsync error messages about vanishing
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/304988
Change subject: reduce cronspam: filter out dataset rsync error messages about
vanishing files
..
reduce cronspam: filter out dataset
ArielGlenn has submitted this change and it was merged.
Change subject: more pylint: no more filter, one more staticmethod decorator
..
more pylint: no more filter, one more staticmethod decorator
Change-Id:
ArielGlenn has submitted this change and it was merged.
Change subject: check for empty output files, not just truncated ones
..
check for empty output files, not just truncated ones
Change-Id:
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/304281
Change subject: check for empty output files, not just truncated ones
..
check for empty output files, not just truncated ones
Change-Id:
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/304229
Change subject: more pylint: no more filter, one more staticmethod decorator
..
more pylint: no more filter, one more staticmethod
ArielGlenn has submitted this change and it was merged.
Change subject: remove wikiqueries script, replaced by onallwikis which does
more
..
remove wikiqueries script, replaced by onallwikis which does more
* also remove docs
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/304187
Change subject: remove wikiqueries script, replaced by onallwikis which does
more
..
remove wikiqueries script, replaced by onallwikis
ArielGlenn has submitted this change and it was merged.
Change subject: updated pylint means new pylint cleanup
..
updated pylint means new pylint cleanup
put imports in proper order
replace some ifs with boolean tests
use
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/304179
Change subject: updated pylint means new pylint cleanup
..
updated pylint means new pylint cleanup
put imports in proper order
replace
ArielGlenn has submitted this change and it was merged.
Change subject: Make scheduler hupable.
..
Make scheduler hupable.
HUP will have scheduler shoot all children and restart itself. All threads
set to daemon so they will
ArielGlenn has submitted this change and it was merged.
Change subject: make the 'force' (lock stealing) option for dumps useful
..
make the 'force' (lock stealing) option for dumps useful
* for regular runs, only remove the
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/304013
Change subject: make the 'force' (lock stealing) option for dumps useful
..
make the 'force' (lock stealing) option for dumps useful
*
ArielGlenn has submitted this change and it was merged.
Change subject: turn on checkpointing for dumps of jawiki, commonswiki
..
turn on checkpointing for dumps of jawiki, commonswiki
These now take a couple of days for the
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/303813
Change subject: turn on checkpointing for dumps of jawiki, commonswiki
..
turn on checkpointing for dumps of jawiki, commonswiki
These
ArielGlenn has submitted this change and it was merged.
Change subject: If a prereq job is missing, optionally run it instead of giving
up
..
If a prereq job is missing, optionally run it instead of giving up
Bug: T141981
ArielGlenn has submitted this change and it was merged.
Change subject: fix up pgrep error check for rsyncs between dataset hosts
..
fix up pgrep error check for rsyncs between dataset hosts
Change-Id:
ArielGlenn has submitted this change and it was merged.
Change subject: make dump run locks stale and therefore removable after 15
minutes
..
make dump run locks stale and therefore removable after 15 minutes
an hour just
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/303707
Change subject: generate and email a few fun dump-related stats each month
..
generate and email a few fun dump-related stats each month
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/302910
Change subject: fix up pgrep error check for rsyncs between dataset hosts
..
fix up pgrep error check for rsyncs between dataset hosts
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/302831
Change subject: Make scheduler hupable.
..
Make scheduler hupable.
[WIP] broken. here as a draft only.
HUP will have it shoot all
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/302827
Change subject: capture dumps cron job output in log and add log rotation
..
capture dumps cron job output in log and add log rotation
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/302729
Change subject: make dump run locks stale and therefore removable after 15
minutes
..
make dump run locks stale and therefore removable
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/302706
Change subject: If a prereq job is missing, run it instead of giving up
..
If a prereq job is missing, run it instead of giving up
[WIP
ArielGlenn has submitted this change and it was merged.
Change subject: move management of pagetitles and mediatitles dirs to datasets
manifest
..
move management of pagetitles and mediatitles dirs to datasets manifest
these
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/302603
Change subject: move management of pagetitles and mediatitles dirs to datasets
manifest
..
move management of pagetitles and mediatitles
ArielGlenn has submitted this change and it was merged.
Change subject: add some missing directories to the datasets dir manifest
..
add some missing directories to the datasets dir manifest
use the opportunity to base the
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/302547
Change subject: add some missing directories to the datasets dir manifest
..
add some missing directories to the datasets dir manifest
ArielGlenn has submitted this change and it was merged.
Change subject: run the content translation dump job once a week
..
run the content translation dump job once a week
instead of every day, heh
Change-Id:
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/302503
Change subject: run the content translation dump job once a week
..
run the content translation dump job once a week
instead of every
ArielGlenn has submitted this change and it was merged.
Change subject: add cron job for Content Translation dumps
..
add cron job for Content Translation dumps
Bug: T127793
Change-Id: I77ee28e9e08e365d2044a54eadfcc9d7f30a26d3
ArielGlenn has submitted this change and it was merged.
Change subject: don't use -q arg for php, we run a modern version which doesn't
need it
..
don't use -q arg for php, we run a modern version which doesn't need it
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/302451
Change subject: don't use -q arg for php, we run a modern version which doesn't
need it
..
don't use -q arg for php, we run a modern
ArielGlenn has submitted this change and it was merged.
Change subject: tiny script that retrieves config values from dump config files
..
tiny script that retrieves config values from dump config files
for use by bash
ArielGlenn has submitted this change and it was merged.
Change subject: move ferm rules for dataset roles into common manifest
..
move ferm rules for dataset roles into common manifest
also include standard and base::firewall
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/302281
Change subject: move ferm rules for dataset roles into common manifest
..
move ferm rules for dataset roles into common manifest
also
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/302279
Change subject: include standard and base::firewall in manifest common to
snapshots
..
include standard and base::firewall in manifest
ArielGlenn has submitted this change and it was merged.
Change subject: include standard and base::firewall in manifest common to
snapshots
..
include standard and base::firewall in manifest common to snapshots
this gets it
ArielGlenn has submitted this change and it was merged.
Change subject: remove all traces of snapshot1002 1003 1004
..
remove all traces of snapshot1002 1003 1004
these hosts are being decommissioned
Change-Id:
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/302241
Change subject: remove all traces of snapshot1002 1003 1004
..
remove all traces of snapshot1002 1003 1004
these hosts are being
ArielGlenn has submitted this change and it was merged.
Change subject: combine steps in dump stages for enwiki so that dependent steps
are together
..
combine steps in dump stages for enwiki so that dependent steps are
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/302230
Change subject: combine steps in dump stages for enwiki so that dependent steps
are together
..
combine steps in dump stages for enwiki
ArielGlenn has submitted this change and it was merged.
Change subject: reduce cronspam from dataset pagecounts rsync
..
reduce cronspam from dataset pagecounts rsync
we don't care if files have vanished so filter out those
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/302225
Change subject: reduce cronspam from dataset pagecounts rsync
..
reduce cronspam from dataset pagecounts rsync
we don't care if files
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/301773
Change subject: add cron job for Content Translation dumps
..
add cron job for Content Translation dumps
[WIP] prolly various errors,
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/301712
Change subject: tiny script that retrieves config values from dump config files
..
tiny script that retrieves config values from dump
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/301670
Change subject: provide the equivalent of getSlaveServer.php for centralauth
..
provide the equivalent of getSlaveServer.php for
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/301182
Change subject: add notes about manual additions needed for new snapshot nodes
..
add notes about manual additions needed for new snapshot
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/301181
Change subject: use fixed repodir setting for dumps jobs
..
use fixed repodir setting for dumps jobs
old hosts are gone, new hosts all
ArielGlenn has submitted this change and it was merged.
Change subject: include the dumps packages in the dumps manifest in snapshot
module
..
include the dumps packages in the dumps manifest in snapshot module
treat it just
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/301169
Change subject: include the dumps packages in the dumps manifest in snapshot
module
..
include the dumps packages in the dumps manifest
ArielGlenn has submitted this change and it was merged.
Change subject: get rid of the useless snapshot cron role wrapper
..
get rid of the useless snapshot cron role wrapper
just include the cron class in the module directly
ArielGlenn has submitted this change and it was merged.
Change subject: move cronjobs class from role to snapshot module and add user
param
..
move cronjobs class from role to snapshot module and add user param
this leaves us
ArielGlenn has submitted this change and it was merged.
Change subject: clean up dumps dirs manifest for snapshots
..
clean up dumps dirs manifest for snapshots
remove user/group params, never used
get rid of scriptsdir, no
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/301150
Change subject: clean up dumps dirs manifest for snapshots
..
clean up dumps dirs manifest for snapshots
remove user/group params, never
901 - 1000 of 3487 matches
Mail list logo