Bearloga has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/367931 )
Change subject: Compatibility with Puppetized runs
......................................................................
Compatibility with Puppetized runs
Bug: T170494
Depends-On: I6c5996c7ea0c616560ae77dc797f9435828a2c5c
Change-Id: I25573e2d552ef7388c83fbbefca6ceab94adacc8
---
M README.md
M main.sh
M modules/metrics/maps/config.yaml
M modules/metrics/portal/config.yaml
M modules/metrics/search/config.yaml
M test.R
6 files changed, 18 insertions(+), 68 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/wikimedia/discovery/golden
refs/changes/31/367931/1
diff --git a/README.md b/README.md
index 89ad0b0..5aeabd4 100644
--- a/README.md
+++ b/README.md
@@ -11,11 +11,8 @@
## Table of Contents
-- [Setup](#setup)
- - [Dependencies](#dependencies)
-- [Usage](#usage)
- - [Production](#production)
- - [Testing](#testing)
+- [Setup](#setup-and-usage)
+- [Dependencies](#dependencies)
- [Modules](#modules)
- [Adding New Metrics Modules](#adding-new-metrics-modules)
- [MySQL](#mysql)
@@ -26,23 +23,11 @@
- [Adding New Forecasting Modules](#adding-new-forecasting-modules)
- [Additional Information](#additional-information)
-## Setup
+## Setup and Usage
-On [stat1002](https://wikitech.wikimedia.org/wiki/Stat1002):
+As of [T170494](https://phabricator.wikimedia.org/T170494), the setup and
daily runs are Puppetized on
[stat1005](https://wikitech.wikimedia.org/wiki/Stat1005) via the
[statistics::discovery](https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/statistics/manifests/discovery.pp)
module (also mirrored on
[GitHub](https://github.com/wikimedia/operations-puppet/blob/production/modules/statistics/manifests/discovery.pp)).
-```bash
-cd /a/discovery/
-git clone --recursive https://gerrit.wikimedia.org/r/wikimedia/discovery/golden
-cd golden
-
-# If already cloned without --recursive:
-git submodule update --init --recursive
-
-# Add execution permission to scripts:
-chmod -R +x modules/
-```
-
-### Dependencies
+## Dependencies
```bash
pip install -r reportupdater/requirements.txt
@@ -56,7 +41,7 @@
Sys.setenv("https_proxy" = "http://webproxy.eqiad.wmnet:8080";)
# Set path for packages:
-.libPaths("/a/discovery/r-library")
+.libPaths("/srv/discovery/r-library")
# Essentials:
install.packages(
@@ -93,37 +78,9 @@
Don't forget to add packages to [test.R](test.R) because that script checks
that all packages are installed before performing a test run of the reports.
-**Note**: we have had problems installing R package
[bsts](https://cran.r-project.org/package=bsts) and its dependencies
[Boom](https://cran.r-project.org/package=Boom) and
[BoomSpikeSlab](https://cran.r-project.org/package=BoomSpikeSlab) on stat1002
(but not stat1003). Fortunately, [Andrew
Otto](https://meta.wikimedia.org/wiki/User:Ottomata) has figured out what to
put in
[~/.R/Makevars](https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Using-Makevars)
to make those packages compile. From
[T147682#2837271](https://phabricator.wikimedia.org/T147682#2837271):
-
-```
-CXX=g++-4.8
-CXX1X=g++-4.8
-CXX1XFLAGS=-std=c++11 -g -O2 -fstack-protector --param=ssp-buffer-size=4
-Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -g
-CXX1XPICFLAGS=-fPIC
-SHLIB_CXX1XLD=g++-4.8
-SHLIB_CXX1XLDFLAGS=-std=c++11 -shared
-LDFLAGS=-L/usr/lib/R/lib -Wl,-Bsymbolic-functions -Wl,-z,relro
-```
-
To **update packages**, run `Rscript test.R --update_packages` which will
update all the dependencies listed in **test.R**
-## Usage
-
-**Note**: You don't need to use the `--config-path` argument if your config
file is inside the query folder and is named **config.yaml**, that is the
default.
-
-### Production
-
-To use in production, add **main.sh** to `crontab`:
-
-```
-$ crontab -e
-
-12 20 * * * cd /a/discovery/golden/ && sh main.sh
-```
-
-**main.sh** executes **reportupdater/update_reports.py** on each module and
writes data to the respective files in **/a/aggregate-datasets/discovery/**
-
-### Testing
+## Testing
If you wish to run all the modules without writing data to files or checking
for missingness, use:
diff --git a/main.sh b/main.sh
index ca5d4e8..c350a55 100644
--- a/main.sh
+++ b/main.sh
@@ -1,24 +1,18 @@
#!/bin/bash
-# Check if Reportupdater git submodule is set up
-if [ ! -f reportupdater/update_reports.py ]; then
- echo "Warning: Reportupdater needs to be initialized and updated..."
- git submodule init && git submodule update
-fi
-
# Sync README
-rsync -c docs/README.md /a/published-datasets/discovery/README.md
+rsync -c docs/README.md /srv/published-datasets/discovery/README.md
# Metrics
for module in "external_traffic" "wdqs" "maps" "search" "portal"
do
echo "Running Reportupdater on ${module} metrics..."
- nice ionice reportupdater/update_reports.py "modules/metrics/${module}"
"/a/published-datasets/discovery/metrics/${module}"
+ nice ionice reportupdater/update_reports.py "modules/metrics/${module}"
"/srv/published-datasets/discovery/metrics/${module}"
done
# Forecasts (dependent on latest metrics)
for module in "search" "wdqs"
do
echo "Running Reportupdater on ${module} forecasts..."
- nice -n 17 ionice -c 2 -n 6 reportupdater/update_reports.py
"modules/forecasts/${module}"
"/a/published-datasets/discovery/forecasts/${module}"
+ nice -n 17 ionice -c 2 -n 6 reportupdater/update_reports.py
"modules/forecasts/${module}"
"/srv/published-datasets/discovery/forecasts/${module}"
done
diff --git a/modules/metrics/maps/config.yaml b/modules/metrics/maps/config.yaml
index a7c4618..d0dae6e 100644
--- a/modules/metrics/maps/config.yaml
+++ b/modules/metrics/maps/config.yaml
@@ -2,7 +2,7 @@
el:
host: analytics-store.eqiad.wmnet
port: 3306
- creds_file: /etc/mysql/conf.d/statistics-private-client.cnf
+ creds_file: /etc/mysql/conf.d/discovery-stats-client.cnf
db: log
defaults:
db: el
diff --git a/modules/metrics/portal/config.yaml
b/modules/metrics/portal/config.yaml
index ae3817c..b980a64 100644
--- a/modules/metrics/portal/config.yaml
+++ b/modules/metrics/portal/config.yaml
@@ -2,7 +2,7 @@
el:
host: analytics-store.eqiad.wmnet
port: 3306
- creds_file: /etc/mysql/conf.d/statistics-private-client.cnf
+ creds_file: /etc/mysql/conf.d/discovery-stats-client.cnf
db: log
defaults:
db: el
diff --git a/modules/metrics/search/config.yaml
b/modules/metrics/search/config.yaml
index 46d9768..82f1c3f 100644
--- a/modules/metrics/search/config.yaml
+++ b/modules/metrics/search/config.yaml
@@ -2,7 +2,7 @@
el:
host: analytics-store.eqiad.wmnet
port: 3306
- creds_file: /etc/mysql/conf.d/statistics-private-client.cnf
+ creds_file: /etc/mysql/conf.d/discovery-stats-client.cnf
db: log
defaults:
db: el
diff --git a/test.R b/test.R
index c689fb6..b8f7066 100644
--- a/test.R
+++ b/test.R
@@ -1,6 +1,7 @@
#!/usr/bin/env Rscript
-.libPaths("/a/discovery/r-library")
+lib_path <- "/srv/discovery/r-library"
+.libPaths(lib_path)
# Check dependencies:
dependencies <- c(
@@ -9,10 +10,8 @@
"tidyverse", "data.table", "plyr",
"optparse", "yaml", "data.tree",
"knitr",
-
# For forecasting modules:
"bsts", "forecast", "prophet",
-
# For querying, etc.:
"ISOcodes", "uaparser", "ortiz", "wmf", "polloi"
)
@@ -28,7 +27,7 @@
})
option_list <- list(
- make_option("--start_date", default = as.character(Sys.Date()-1,
"%Y-%m-%d"), action = "store", type = "character"),
+ make_option("--start_date", default = as.character(Sys.Date() - 1,
"%Y-%m-%d"), action = "store", type = "character"),
make_option("--end_date", default = as.character(Sys.Date(), "%Y-%m-%d"),
action = "store", type = "character",
help = "This is required for proper Reportupdater emulation;
should be 'start_date' + 1"),
make_option("--omit_times", default = FALSE, action = "store_true",
@@ -52,8 +51,8 @@
opt <- parse_args(OptionParser(option_list = option_list))
if (opt$update_packages) {
- withr::with_libpaths("/a/discovery/r-library",
devtools::update_packages(dependencies))
- library("uaparser", lib.loc = "/a/discovery/r-library")
+ withr::with_libpaths(lib_path, devtools::update_packages(dependencies))
+ library("uaparser", lib.loc = lib_path)
update_regexes()
q(save = "no")
}
--
To view, visit https://gerrit.wikimedia.org/r/367931
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I25573e2d552ef7388c83fbbefca6ceab94adacc8
Gerrit-PatchSet: 1
Gerrit-Project: wikimedia/discovery/golden
Gerrit-Branch: master
Gerrit-Owner: Bearloga <[email protected]>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
