[MediaWiki-commits] [Gerrit] mediawiki/core[wmf/1.30.0-wmf.18]: Speed up populateIpChanges maintenance script.

2017-09-18 Thread jenkins-bot (Code Review) 
jenkins-bot has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/378736 )

Change subject: Speed up populateIpChanges maintenance script.
..


Speed up populateIpChanges maintenance script.

Use BETWEEN in populateIpChanges maintenance script, which will make it
more efficient when copying revisions with a high rev_id. Also adding a
'max-rev-id' option to prevent the script from looping through IP
changes that have already been copied since the core patch was deployed.

Bug: T175962
Change-Id: I1df10c9b7237ad5002f76f9d354c36ce879d9d9f
(cherry picked from commit 2eac9d7ef4953fdfe5f710fb3dc98c360c80c5ae)
---
M maintenance/populateIpChanges.php
1 file changed, 15 insertions(+), 4 deletions(-)

Approvals:
  Reedy: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/maintenance/populateIpChanges.php 
b/maintenance/populateIpChanges.php
index ffb8c43..c173270 100644
--- a/maintenance/populateIpChanges.php
+++ b/maintenance/populateIpChanges.php
@@ -47,6 +47,12 @@
);
$this->addOption( 'rev-id', 'The rev_id to start copying from. 
Default: 0', false, true );
$this->addOption(
+   'max-rev-id',
+   'The rev_id to stop at. Default: result of MAX(rev_id)',
+   false,
+   true
+   );
+   $this->addOption(
'throttle',
'Wait this many milliseconds after copying each batch 
of revisions. Default: 0',
false,
@@ -57,20 +63,25 @@
 
public function doDBUpdates() {
$lbFactory = 
MediaWikiServices::getInstance()->getDBLoadBalancerFactory();
+   $dbr = $this->getDB( DB_REPLICA, [ 'vslow' ] );
$dbw = $this->getDB( DB_MASTER );
$throttle = intval( $this->getOption( 'throttle', 0 ) );
+   $maxRevId = intval( $this->getOption( 'max-rev-id', 0 ) );
$start = $this->getOption( 'rev-id', 0 );
-   $end = $dbw->selectField( 'revision', 'MAX(rev_id)', false, 
__METHOD__ );
+   $end = $maxRevId > 0
+   ? $maxRevId
+   : $dbw->selectField( 'revision', 'MAX(rev_id)', false, 
__METHOD__ );
$blockStart = $start;
$revCount = 0;
 
$this->output( "Copying IP revisions to ip_changes, from rev_id 
$start to rev_id $end\n" );
 
while ( $blockStart <= $end ) {
-   $rows = $dbw->select(
+   $blockEnd = min( $blockStart + 200, $end );
+   $rows = $dbr->select(
'revision',
[ 'rev_id', 'rev_timestamp', 'rev_user_text' ],
-   [ "rev_id >= $blockStart", 'rev_user' => 0 ],
+   [ "rev_id BETWEEN $blockStart AND $blockEnd", 
'rev_user' => 0 ],
__METHOD__,
[ 'ORDER BY' => 'rev_id ASC', 'LIMIT' => 
$this->mBatchSize ]
);
@@ -80,7 +91,7 @@
}
 
$this->output( "...checking $this->mBatchSize revisions 
for IP edits that need copying, " .
-   "starting with rev_id $blockStart\n" );
+   "between rev_ids $blockStart and $blockEnd\n" );
 
$insertRows = [];
foreach ( $rows as $row ) {

-- 
To view, visit https://gerrit.wikimedia.org/r/378736
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I1df10c9b7237ad5002f76f9d354c36ce879d9d9f
Gerrit-PatchSet: 1
Gerrit-Project: mediawiki/core
Gerrit-Branch: wmf/1.30.0-wmf.18
Gerrit-Owner: Reedy 
Gerrit-Reviewer: MusikAnimal 
Gerrit-Reviewer: Parent5446 
Gerrit-Reviewer: Reedy 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] mediawiki/core[wmf/1.30.0-wmf.18]: Speed up populateIpChanges maintenance script.

2017-09-18 Thread Reedy (Code Review) 
Reedy has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/378736 )

Change subject: Speed up populateIpChanges maintenance script.
..

Speed up populateIpChanges maintenance script.

Use BETWEEN in populateIpChanges maintenance script, which will make it
more efficient when copying revisions with a high rev_id. Also adding a
'max-rev-id' option to prevent the script from looping through IP
changes that have already been copied since the core patch was deployed.

Bug: T175962
Change-Id: I1df10c9b7237ad5002f76f9d354c36ce879d9d9f
(cherry picked from commit 2eac9d7ef4953fdfe5f710fb3dc98c360c80c5ae)
---
M maintenance/populateIpChanges.php
1 file changed, 15 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/mediawiki/core 
refs/changes/36/378736/1

diff --git a/maintenance/populateIpChanges.php 
b/maintenance/populateIpChanges.php
index ffb8c43..c173270 100644
--- a/maintenance/populateIpChanges.php
+++ b/maintenance/populateIpChanges.php
@@ -47,6 +47,12 @@
);
$this->addOption( 'rev-id', 'The rev_id to start copying from. 
Default: 0', false, true );
$this->addOption(
+   'max-rev-id',
+   'The rev_id to stop at. Default: result of MAX(rev_id)',
+   false,
+   true
+   );
+   $this->addOption(
'throttle',
'Wait this many milliseconds after copying each batch 
of revisions. Default: 0',
false,
@@ -57,20 +63,25 @@
 
public function doDBUpdates() {
$lbFactory = 
MediaWikiServices::getInstance()->getDBLoadBalancerFactory();
+   $dbr = $this->getDB( DB_REPLICA, [ 'vslow' ] );
$dbw = $this->getDB( DB_MASTER );
$throttle = intval( $this->getOption( 'throttle', 0 ) );
+   $maxRevId = intval( $this->getOption( 'max-rev-id', 0 ) );
$start = $this->getOption( 'rev-id', 0 );
-   $end = $dbw->selectField( 'revision', 'MAX(rev_id)', false, 
__METHOD__ );
+   $end = $maxRevId > 0
+   ? $maxRevId
+   : $dbw->selectField( 'revision', 'MAX(rev_id)', false, 
__METHOD__ );
$blockStart = $start;
$revCount = 0;
 
$this->output( "Copying IP revisions to ip_changes, from rev_id 
$start to rev_id $end\n" );
 
while ( $blockStart <= $end ) {
-   $rows = $dbw->select(
+   $blockEnd = min( $blockStart + 200, $end );
+   $rows = $dbr->select(
'revision',
[ 'rev_id', 'rev_timestamp', 'rev_user_text' ],
-   [ "rev_id >= $blockStart", 'rev_user' => 0 ],
+   [ "rev_id BETWEEN $blockStart AND $blockEnd", 
'rev_user' => 0 ],
__METHOD__,
[ 'ORDER BY' => 'rev_id ASC', 'LIMIT' => 
$this->mBatchSize ]
);
@@ -80,7 +91,7 @@
}
 
$this->output( "...checking $this->mBatchSize revisions 
for IP edits that need copying, " .
-   "starting with rev_id $blockStart\n" );
+   "between rev_ids $blockStart and $blockEnd\n" );
 
$insertRows = [];
foreach ( $rows as $row ) {

-- 
To view, visit https://gerrit.wikimedia.org/r/378736
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I1df10c9b7237ad5002f76f9d354c36ce879d9d9f
Gerrit-PatchSet: 1
Gerrit-Project: mediawiki/core
Gerrit-Branch: wmf/1.30.0-wmf.18
Gerrit-Owner: Reedy 
Gerrit-Reviewer: MusikAnimal 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits