ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/231993
Change subject: dumps: do abstract dumps stream in smaller pieces
..
dumps: do abstract dumps stream in smaller pieces
Change-Id: I49e8ca1
ArielGlenn has submitted this change and it was merged.
Change subject: dumps: do abstract dumps stream in smaller pieces
..
dumps: do abstract dumps stream in smaller pieces
Change-Id: I49e8ca1c2e1a2caa35f782c6c7dae6f616319a05
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233417
Change subject: dumps: redo handling of jobs with unrun prereqs
..
dumps: redo handling of jobs with unrun prereqs
for a wiki not currentl
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233418
Change subject: dumps: tweak stages a bit
..
dumps: tweak stages a bit
we want to avoid having a stage claim it's complete when there
it's
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233449
Change subject: turn retention modules into a package
..
turn retention modules into a package
update all the import statements, add __ini
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233445
Change subject: update code for collection of minions with given grain
..
update code for collection of minions with given grain
Change-Id
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233446
Change subject: check mysqlconf for log retention and skip explicit logfiles
check
..
check mysqlconf for log retention and skip explicit
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233451
Change subject: write config files to subdir in the salt file_root
..
write config files to subdir in the salt file_root
this will allow a
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233452
Change subject: use cp.get_file to retrieve configs
..
use cp.get_file to retrieve configs
next step will be to get rid of the executor cr
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233453
Change subject: turn files auditor into salt module
..
turn files auditor into salt module
remotefileauditor invokes the salt module on th
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233455
Change subject: homes audit done via salt module
..
homes audit done via salt module
remote homes audit file and class, local homes audit
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233447
Change subject: allow for more than one mysql conf file, audit all of /a
..
allow for more than one mysql conf file, audit all of /a
Chang
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233450
Change subject: remove dup copy of Runner class; read confs from disk, not stdin
..
remove dup copy of Runner class; read confs from disk,
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233454
Change subject: logs audit via salt module
..
logs audit via salt module
split into two files, one for the remote audit of all host and
on
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233459
Change subject: file examiner: use salt module for remote cmd
..
file examiner: use salt module for remote cmd
next up: do the same for di
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233457
Change subject: move rule static methods out to a seperate utils file
..
move rule static methods out to a seperate utils file
Change-Id:
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233458
Change subject: get rid of executor and related vars for most files
..
get rid of executor and related vars for most files
cli and examine
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233460
Change subject: convert dir examiner to use salt module
..
convert dir examiner to use salt module
next up: cli must be converted
Change-
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233461
Change subject: last 'executor' code removed, all salt module now
..
last 'executor' code removed, all salt module now
Change-Id: I5700057
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233456
Change subject: remove now unused auditor.py, move static methods into module
..
remove now unused auditor.py, move static methods into mod
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233464
Change subject: cleanup unused function, unneeded special class instantiations
..
cleanup unused function, unneeded special class instantia
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233463
Change subject: move readline completion and ignored entry lists out to
separate files
..
move readline completion and ignored entry lists
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233466
Change subject: use yaml instead of python for global config, ignore list
..
use yaml instead of python for global config, ignore list
Cha
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233462
Change subject: move cli static methods out to separate file
..
move cli static methods out to separate file
Change-Id: I3db85c76ea268ff57
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233471
Change subject: remove old audit_files.py, no longer needed, remove unused Rule
class remove unused timeout from local audit methods remove some dead attrs and
args, clean up handling of ignore dicts
..
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233467
Change subject: don't modify sys.path, convert remaining execs into local and
remote
..
don't modify sys.path, convert remaining execs int
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233465
Change subject: clean up ignore list code
..
clean up ignore list code
move duplicate ignoreslist methods into module
move ignore list met
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233470
Change subject: some pylint fixes
..
some pylint fixes
Change-Id: I0f577afbdf75773cd0ec288998ecdb595f586ffe
---
M dataretention/retention/
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233469
Change subject: enable directory prompt in cli in a few places in the menu
..
enable directory prompt in cli in a few places in the menu
C
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233468
Change subject: clean up names and paths, remove last runpy calls, use more yaml
..
clean up names and paths, remove last runpy calls, use
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233473
Change subject: more pylint: move dup code for file ignore check into its own
method take care of unused imports in borrowed code for 'magic' more pylint and
misc cleanups
..
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233472
Change subject: more pylint/pep8 cleanup
..
more pylint/pep8 cleanup
remove some blank lines
clean up some duplicated code
move some cli c
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233475
Change subject: fix up for install via setup.py, bug fixes
..
fix up for install via setup.py, bug fixes
rename and organize files so that
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233478
Change subject: retention: split setup.py into two files, for clients and master
..
retention: split setup.py into two files, for clients a
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233477
Change subject: retention emails fixes:
..
retention emails fixes:
send emails to a default email if no user->email entry
fix typo in opti
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233476
Change subject: bit more pylint
..
bit more pylint
Change-Id: I231df12881a08c75fe80408b155cfd5a47929955
---
M clouseau/clouseau/retention/
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233474
Change subject: pep8 fixup: remove extra spaces pylint cleanup: work around no
method overrides in python
..
pep8 fixup: remove extra spac
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/230507
Change subject: dumps: fix link cleanup for stubs/content parallel runs
..
dumps: fix link cleanup for stubs/content parallel runs
in this
ArielGlenn has submitted this change and it was merged.
Change subject: dumps: fix link cleanup for stubs/content parallel runs
..
dumps: fix link cleanup for stubs/content parallel runs
in this case there will be a recombine p
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/230524
Change subject: remove now obselete snapshot hosts sudoers file
..
remove now obselete snapshot hosts sudoers file
Change-Id: Ib44864755dc
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/230734
Change subject: add iridum to dumps rsync clients
..
add iridum to dumps rsync clients
otherwise copy of phab dumps to dataset hosts is br
ArielGlenn has submitted this change and it was merged.
Change subject: add iridum to dumps rsync clients
..
add iridum to dumps rsync clients
otherwise copy of phab dumps to dataset hosts is broken
Change-Id: I3bb35b49f1459de
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/230738
Change subject: dataset: add redirect for fundraising data and link on web page
..
dataset: add redirect for fundraising data and link on w
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/230739
Change subject: dumps mirrors rsync conf: remove/update useless comments
..
dumps mirrors rsync conf: remove/update useless comments
Chang
ArielGlenn has submitted this change and it was merged.
Change subject: dumps mirrors rsync conf: remove/update useless comments
..
dumps mirrors rsync conf: remove/update useless comments
Change-Id: I8e40842f80fcc726d937e9c609
ArielGlenn has submitted this change and it was merged.
Change subject: remove now obselete snapshot hosts sudoers file
..
remove now obselete snapshot hosts sudoers file
Change-Id: Ib44864755dc33d3b851945e683622ceed917f015
---
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/230767
Change subject: dumps: get rid of one more eval.php call, correct usage message
..
dumps: get rid of one more eval.php call, correct usage
ArielGlenn has submitted this change and it was merged.
Change subject: dumps: get rid of one more eval.php call, correct usage message
..
dumps: get rid of one more eval.php call, correct usage message
Change-Id: Ic27b1169eb76
ArielGlenn has submitted this change and it was merged.
Change subject: Add ssh key for new notebook
..
Add ssh key for new notebook
Also uploaded to terbium (to confirm it's me for real):
https://people.wikimedia.org/~hoo/tmp/
ArielGlenn has submitted this change and it was merged.
Change subject: Add Chris Steipp to analytics-privatedata-users
..
Add Chris Steipp to analytics-privatedata-users
Bug: T108227
Change-Id: Ib65380cf5aa2878c666e3784abce80e
ArielGlenn has submitted this change and it was merged.
Change subject: Create Wikidata ttl dumps on Mondays 23:00:00 (UTC)
..
Create Wikidata ttl dumps on Mondays 23:00:00 (UTC)
Change-Id: I66da5b2d374d6496f1f3f424d3ba4ba0ad5f
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/204257
Change subject: html dumps will be served from host where they are produced,
via proxy
..
html dumps will be served from host where they a
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/215657
Change subject: dumps: add 'skipdone' to worker script and its wrapper
..
dumps: add 'skipdone' to worker script and its wrapper
allows us
ArielGlenn has submitted this change and it was merged.
Change subject: dumps: add 'skipdone' to worker script and its wrapper
..
dumps: add 'skipdone' to worker script and its wrapper
allows us to run some or all jobs across o
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/215661
Change subject: dumps: allow 'last' to be specified as date of run
..
dumps: allow 'last' to be specified as date of run
this will let us
ArielGlenn has submitted this change and it was merged.
Change subject: dumps: allow 'last' to be specified as date of run
..
dumps: allow 'last' to be specified as date of run
this will let us (re)run jobs for one or all wikis
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/215662
Change subject: dumps: add onepass to worker script and wrapper, fix cutoff
option
..
dumps: add onepass to worker script and wrapper, fix
ArielGlenn has submitted this change and it was merged.
Change subject: dumps: add onepass to worker script and wrapper, fix cutoff
option
..
dumps: add onepass to worker script and wrapper, fix cutoff option
onepass lets us r
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/215666
Change subject: dumps: do xml stubs via streaming
..
dumps: do xml stubs via streaming
this works around php memory leaks, running dumps f
ArielGlenn has submitted this change and it was merged.
Change subject: dumps: do xml stubs via streaming
..
dumps: do xml stubs via streaming
this works around php memory leaks, running dumps for small ranges
and feeding the o
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/215671
Change subject: dumps: do xml page logs via streaming
..
dumps: do xml page logs via streaming
as with the xml stubs streaming, this works
ArielGlenn has submitted this change and it was merged.
Change subject: dumps: do xml page logs via streaming
..
dumps: do xml page logs via streaming
as with the xml stubs streaming, this works around php memory
leaks by dumpi
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/215674
Change subject: dumps: only show one list of dumps running
..
dumps: only show one list of dumps running
we generate multiple index html p
ArielGlenn has submitted this change and it was merged.
Change subject: dumps: only show one list of dumps running
..
dumps: only show one list of dumps running
we generate multiple index html pages but only one of those
needs
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/215685
Change subject: dumps: fix up job name of wb property info dump
..
dumps: fix up job name of wb property info dump
all dumps of tables sho
ArielGlenn has submitted this change and it was merged.
Change subject: dumps: fix up job name of wb property info dump
..
dumps: fix up job name of wb property info dump
all dumps of tables should have a job name ending in 'ta
ArielGlenn has submitted this change and it was merged.
Change subject: Add wb_changes_subscription table to xml dumps
..
Add wb_changes_subscription table to xml dumps
this table is on Wikidata only. It has approximately
46,18
ArielGlenn has submitted this change and it was merged.
Change subject: Add wbc_entity_usage table to xml dumps
..
Add wbc_entity_usage table to xml dumps
Bug: T98743
Change-Id: I940c44811a8b9e4b85e7807a478872ebd63a7d07
---
M x
ArielGlenn has submitted this change and it was merged.
Change subject: Per bug #48012. Compressed possible errors into for loop;
program exits if there is one or more errors and writes all of them.
..
Per bug #48012. Compresse
ArielGlenn has submitted this change and it was merged.
Change subject: Dump GeoData information
..
Dump GeoData information
Would need geo_tags table to be created everywhere before deploying
Bug: 51225
Change-Id: I6c7d120b2
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/216058
Change subject: dumps: tweak 7z compression arg so compression runs much faster
..
dumps: tweak 7z compression arg so compression runs much
ArielGlenn has submitted this change and it was merged.
Change subject: dumps: tweak 7z compression arg so compression runs much faster
..
dumps: tweak 7z compression arg so compression runs much faster
thanks to Nemo_bis for t
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/216096
Change subject: dumps: fix exception raised in stub phase on failure
..
dumps: fix exception raised in stub phase on failure
Change-Id: I4
ArielGlenn has submitted this change and it was merged.
Change subject: dumps: fix exception raised in stub phase on failure
..
dumps: fix exception raised in stub phase on failure
Change-Id: I48d26f122ef664cb4210190e9d8a083458
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/216404
Change subject: dumps: clean up error handling in xml streamed dumps
..
dumps: clean up error handling in xml streamed dumps
leave temp fi
ArielGlenn has submitted this change and it was merged.
Change subject: dumps: clean up error handling in xml streamed dumps
..
dumps: clean up error handling in xml streamed dumps
leave temp files around for investigation late
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/216416
Change subject: dumps: for small wikis that are dumped in one run, fix item
range
..
dumps: for small wikis that are dumped in one run, fi
ArielGlenn has submitted this change and it was merged.
Change subject: dumps: for small wikis that are dumped in one run, fix item
range
..
dumps: for small wikis that are dumped in one run, fix item range
previously the last
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/216539
Change subject: dumps: on dry run for streaming dumps, no compressors write to
files
..
dumps: on dry run for streaming dumps, no compress
ArielGlenn has submitted this change and it was merged.
Change subject: dumps: on dry run for streaming dumps, no compressors write to
files
..
dumps: on dry run for streaming dumps, no compressors write to files
Change-Id: I9
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/216617
Change subject: xml dumps: remove logs batchsize from config, obsolete
..
xml dumps: remove logs batchsize from config, obsolete
Change-Id
ArielGlenn has submitted this change and it was merged.
Change subject: xml dumps: remove logs batchsize from config, obsolete
..
xml dumps: remove logs batchsize from config, obsolete
Change-Id: I9a3d056ca8e4c1f25fd4c3210d13d3
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/216714
Change subject: dumps: clean up old private dumps and files, not just public
ones
..
dumps: clean up old private dumps and files, not just
ArielGlenn has submitted this change and it was merged.
Change subject: dumps: clean up old private dumps and files, not just public
ones
..
dumps: clean up old private dumps and files, not just public ones
Change-Id: I8ff5e15
ArielGlenn has submitted this change and it was merged.
Change subject: deployment server init should configure repo every time
..
deployment server init should configure repo every time
formerly it would refuse to configure a
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/235704
Change subject: dumps: be able to specify number of chunks for abstracts
..
dumps: be able to specify number of chunks for abstracts
inste
ArielGlenn has submitted this change and it was merged.
Change subject: dumps: be able to specify number of chunks for abstracts
..
dumps: be able to specify number of chunks for abstracts
instead of a number of pages or a list
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/236487
Change subject: dumps: big wikis do abstracts in 4 chunks, no need for page
ranges
..
dumps: big wikis do abstracts in 4 chunks, no need f
ArielGlenn has submitted this change and it was merged.
Change subject: dumps: big wikis do abstracts in 4 chunks, no need for page
ranges
..
dumps: big wikis do abstracts in 4 chunks, no need for page ranges
Change-Id: Ibb5e1
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/236798
Change subject: crap salt cleanup scripts primarily for labs use
..
crap salt cleanup scripts primarily for labs use
Change-Id: I30a549252
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233683
Change subject: dumps: job 'createdir' which just creates new dir for dump run
..
dumps: job 'createdir' which just creates new dir for dum
ArielGlenn has submitted this change and it was merged.
Change subject: dumps: job 'createdir' which just creates new dir for dump run
..
dumps: job 'createdir' which just creates new dir for dump run
this will also create the
ArielGlenn has submitted this change and it was merged.
Change subject: dumps: redo handling of jobs with unrun prereqs
..
dumps: redo handling of jobs with unrun prereqs
for a wiki not currently locked by another worker, if a
ArielGlenn has submitted this change and it was merged.
Change subject: dumps: tweak stages a bit
..
dumps: tweak stages a bit
we want to avoid having a stage claim it's complete when there
it's dependent on a previous stage, b
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233954
Change subject: dumps: for "createdirs" job don't touch latest symlinks or rss
feeds
..
dumps: for "createdirs" job don't touch latest sym
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233953
Change subject: dumps: allow user to specify how long dumper sleeps between
wikis
..
dumps: allow user to specify how long dumper sleeps b
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233961
Change subject: staged dumps: sleep only 5 seconds between wikis for starting
job
..
staged dumps: sleep only 5 seconds between wikis for
ArielGlenn has uploaded a new change for review.
https://gerrit.wikimedia.org/r/233999
Change subject: dumps: move scheduler args out of dump arg list
..
dumps: move scheduler args out of dump arg list
so we can have multiple
ArielGlenn has submitted this change and it was merged.
Change subject: dumps: move scheduler args out of dump arg list
..
dumps: move scheduler args out of dump arg list
so we can have multiple dump run commands on the same li
ArielGlenn has submitted this change and it was merged.
Change subject: dumps: allow user to specify how long dumper sleeps between
wikis
..
dumps: allow user to specify how long dumper sleeps between wikis
there is now the 'c
1 - 100 of 3536 matches
Mail list logo
