[MediaWiki-commits] [Gerrit] [s23.org wikistats] Throttle updates for big farms, keep upd... - change (operations...wikistats)

2014-01-24 Thread Dzahn (Code Review) 
Dzahn has submitted this change and it was merged.

Change subject: [s23.org wikistats] Throttle updates for big farms, keep 
updating big wikis' stats
..


[s23.org wikistats] Throttle updates for big farms, keep updating big wikis' 
stats

Patch quickly put together after my last comment on the bug.
If it looks reasonably ok, mutante will have to deploy it, change whatever 
switch
kept the wikia wikis stalled and then let the updates run to test.

Bug: 59943

PS2: sneak in a small live hack to keep it in sync with the labs instance

Change-Id: Ieb3c1d49b6c6b7208172f7e0bac810ce72fe3bc2
---
M usr/lib/wikistats/update.php
1 file changed, 22 insertions(+), 1 deletion(-)

Approvals:
  Dzahn: Verified; Looks good to me, approved



diff --git a/usr/lib/wikistats/update.php b/usr/lib/wikistats/update.php
index c4aa74f..bce1a07 100644
--- a/usr/lib/wikistats/update.php
+++ b/usr/lib/wikistats/update.php
@@ -256,9 +256,24 @@
 
 #want to know number of wikis and progress in logs
 $mycount=0;
+$mybigcount=0;
 $totalcount=mysql_num_rows($myresult);
+$bigfarm=false;
+if ($totalcount  1000  $table != mediawikis) {
+   $bigfarm=true;
+}
 
 while($row = mysql_fetch_array( $myresult )) {
+
+   // If it's a big lazy wikifarm, give precedence to bigger wikis,
+   // don't update too many of the others.
+   if ($bigfarm) {
+   if ($row['good'] = 100) {
+   $mybigcount++;
+   } elseif ($mycount = 1000) {
+   continue;
+   }
+   }
$mycount++;
 
if ($row['method']==8) {
@@ -478,7 +493,7 @@
}
} elseif (isset($myextinfo['statuscode'])) {
$statuscode=$myextinfo['statuscode'];
-   $extquery=update mediawikis set http='$statuscode' 
where id=.$row['id'].;;
+   $extquery=update ${table} set http='$statuscode' where 
id=.$row['id'].;;
$extresult = mysql_query($extquery) or 
die(mysql_error());
$extcount++;
echo $extquery\n;
@@ -487,6 +502,12 @@
}
}
 
+   // Let the big but lazy wikifarms rest a second, or till next run if 
we've done enough
+   if ($bigfarm  $mycount = 1000  $mybigcount = 500) {
+   break;
+   }
+   sleep(1);
+
 }
 
 if (isset($convert)  $convert) {

-- 
To view, visit https://gerrit.wikimedia.org/r/108670
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ieb3c1d49b6c6b7208172f7e0bac810ce72fe3bc2
Gerrit-PatchSet: 2
Gerrit-Project: operations/debs/wikistats
Gerrit-Branch: master
Gerrit-Owner: Nemo bis federicol...@tiscali.it
Gerrit-Reviewer: ArielGlenn ar...@wikimedia.org
Gerrit-Reviewer: Dzahn dz...@wikimedia.org
Gerrit-Reviewer: jenkins-bot 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] [s23.org wikistats] Throttle updates for big farms, keep upd... - change (operations...wikistats)

2014-01-21 Thread Nemo bis (Code Review) 
Nemo bis has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/108670


Change subject: [s23.org wikistats] Throttle updates for big farms, keep 
updating big wikis' stats
..

[s23.org wikistats] Throttle updates for big farms, keep updating big wikis' 
stats

Patch quickly put together after my last comment on the bug.
If it looks reasonably ok, mutante will have to deploy it, change whatever 
switch
kept the wikia wikis stalled and then let the updates run to test.

Bug: 59943
Change-Id: Ieb3c1d49b6c6b7208172f7e0bac810ce72fe3bc2
---
M usr/lib/wikistats/update.php
1 file changed, 21 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/wikistats 
refs/changes/70/108670/1

diff --git a/usr/lib/wikistats/update.php b/usr/lib/wikistats/update.php
index c4aa74f..cfcebd0 100644
--- a/usr/lib/wikistats/update.php
+++ b/usr/lib/wikistats/update.php
@@ -256,9 +256,24 @@
 
 #want to know number of wikis and progress in logs
 $mycount=0;
+$mybigcount=0;
 $totalcount=mysql_num_rows($myresult);
+$bigfarm=false;
+if ($totalcount  1000  $table != mediawikis) {
+   $bigfarm=true;
+}
 
 while($row = mysql_fetch_array( $myresult )) {
+
+   // If it's a big lazy wikifarm, give precedence to bigger wikis,
+   // don't update too many of the others.
+   if ($bigfarm) {
+   if ($row['good'] = 100) {
+   $mybigcount++;
+   } elseif ($mycount = 1000) {
+   continue;
+   }
+   }
$mycount++;
 
if ($row['method']==8) {
@@ -487,6 +502,12 @@
}
}
 
+   // Let the big but lazy wikifarms rest a second, or till next run if 
we've done enough
+   if ($bigfarm  $mycount = 1000  $mybigcount = 500) {
+   break;
+   }
+   sleep(1);
+
 }
 
 if (isset($convert)  $convert) {

-- 
To view, visit https://gerrit.wikimedia.org/r/108670
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ieb3c1d49b6c6b7208172f7e0bac810ce72fe3bc2
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/wikistats
Gerrit-Branch: master
Gerrit-Owner: Nemo bis federicol...@tiscali.it
Gerrit-Reviewer: jenkins-bot

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits