Bearloga has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/349232 )
Change subject: Nice/ionice-ize Reportupdater runs
......................................................................
Nice/ionice-ize Reportupdater runs
When the scripts are run (especially the forecasting scripts),
they put a load on stat1002 that might make it worse for other
users. This adds runs the commands through nice & ionice.
Change-Id: I4c900149c0ea54aabf644cea9e0b366d5927050d
---
M main.sh
M modules/forecasts/forecast.R
M modules/forecasts/search/api_cirrus_prophet
M modules/forecasts/search/zrr_overall_prophet
M modules/forecasts/wdqs/homepage_traffic_prophet
M modules/forecasts/wdqs/sparql_usage_prophet
6 files changed, 8 insertions(+), 8 deletions(-)
Approvals:
Bearloga: Verified; Looks good to me, approved
diff --git a/main.sh b/main.sh
index c89d3fa..ef2a221 100644
--- a/main.sh
+++ b/main.sh
@@ -13,7 +13,7 @@
for module in "external_traffic" "wdqs" "maps" "search" "portal"
do
echo "Running Reportupdater on ${module} metrics..."
- reportupdater/update_reports.py "modules/metrics/${module}"
"/a/aggregate-datasets/discovery/${module}"
+ nice ionice reportupdater/update_reports.py "modules/metrics/${module}"
"/a/aggregate-datasets/discovery/${module}"
done
# Forecasts (dependent on latest metrics)
@@ -23,5 +23,5 @@
for module in "search" "wdqs"
do
echo "Running Reportupdater on ${module} forecasts..."
- reportupdater/update_reports.py "modules/forecasts/${module}"
"/a/aggregate-datasets/discovery-forecasts/${module}"
+ nice -n 17 ionice -c 2 -n 6 reportupdater/update_reports.py
"modules/forecasts/${module}"
"/a/aggregate-datasets/discovery-forecasts/${module}"
done
diff --git a/modules/forecasts/forecast.R b/modules/forecasts/forecast.R
index 051eefc..285d330 100755
--- a/modules/forecasts/forecast.R
+++ b/modules/forecasts/forecast.R
@@ -12,11 +12,11 @@
* wdqs_sparql"),
make_option("--model", default = NA, action = "store", type = "character",
help = "Available: ARIMA, BSTS, Prophet"),
- make_option("--iters", default = 5000, action = "store", type = "numeric",
+ make_option("--iters", default = 1000, action = "store", type = "numeric",
help = "Number of MCMC iterations to keep in BSTS models
[default %default]
If using a Prophet model, then setting this to 0 will perform
MAP estimation,
otherwise a full Bayesian inference is performed using Stan."),
- make_option("--burnin", default = 1000, action = "store", type = "numeric",
+ make_option("--burnin", default = 500, action = "store", type = "numeric",
help = "Number of iterations to use as burn-in in BSTS models
[default %default]")
)
diff --git a/modules/forecasts/search/api_cirrus_prophet
b/modules/forecasts/search/api_cirrus_prophet
index 3afc098..26a4a06 100755
--- a/modules/forecasts/search/api_cirrus_prophet
+++ b/modules/forecasts/search/api_cirrus_prophet
@@ -1,3 +1,3 @@
#!/bin/bash
-Rscript modules/forecasts/forecast.R --date=$1 --metric=search_api_cirrus
--model=Prophet --iters=0
+Rscript modules/forecasts/forecast.R --date=$1 --metric=search_api_cirrus
--model=Prophet
diff --git a/modules/forecasts/search/zrr_overall_prophet
b/modules/forecasts/search/zrr_overall_prophet
index 4eb5bca..9bba5c0 100755
--- a/modules/forecasts/search/zrr_overall_prophet
+++ b/modules/forecasts/search/zrr_overall_prophet
@@ -1,3 +1,3 @@
#!/bin/bash
-Rscript modules/forecasts/forecast.R --date=$1 --metric=search_zrr_overall
--model=Prophet --iters=0
+Rscript modules/forecasts/forecast.R --date=$1 --metric=search_zrr_overall
--model=Prophet
diff --git a/modules/forecasts/wdqs/homepage_traffic_prophet
b/modules/forecasts/wdqs/homepage_traffic_prophet
index 3763c5c..a595ed1 100755
--- a/modules/forecasts/wdqs/homepage_traffic_prophet
+++ b/modules/forecasts/wdqs/homepage_traffic_prophet
@@ -1,3 +1,3 @@
#!/bin/bash
-Rscript modules/forecasts/forecast.R --date=$1 --metric=wdqs_homepage
--model=Prophet --iters=0
+Rscript modules/forecasts/forecast.R --date=$1 --metric=wdqs_homepage
--model=Prophet
diff --git a/modules/forecasts/wdqs/sparql_usage_prophet
b/modules/forecasts/wdqs/sparql_usage_prophet
index 3e0e136..f62866f 100755
--- a/modules/forecasts/wdqs/sparql_usage_prophet
+++ b/modules/forecasts/wdqs/sparql_usage_prophet
@@ -1,3 +1,3 @@
#!/bin/bash
-Rscript modules/forecasts/forecast.R --date=$1 --metric=wdqs_sparql
--model=Prophet --iters=0
+Rscript modules/forecasts/forecast.R --date=$1 --metric=wdqs_sparql
--model=Prophet
--
To view, visit https://gerrit.wikimedia.org/r/349232
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I4c900149c0ea54aabf644cea9e0b366d5927050d
Gerrit-PatchSet: 1
Gerrit-Project: wikimedia/discovery/golden
Gerrit-Branch: master
Gerrit-Owner: Bearloga <[email protected]>
Gerrit-Reviewer: Bearloga <[email protected]>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
