Yuvipanda has submitted this change and it was merged. Change subject: Add scripts for local testing ......................................................................
Add scripts for local testing And describe how to use them in the readme. Change-Id: I01dbecd5c0d4563cfa136005bb833499c8bd6fe6 --- M .gitignore M README.md A scripts/config.yaml A scripts/localurl A scripts/remoteurl A scripts/ssh 6 files changed, 66 insertions(+), 8 deletions(-) Approvals: Yuvipanda: Verified; Looks good to me, approved diff --git a/.gitignore b/.gitignore index c07d99c..163d7bf 100644 --- a/.gitignore +++ b/.gitignore @@ -3,3 +3,5 @@ .output *.swp *.pyc +my.cnf.research + diff --git a/README.md b/README.md index 37b4f39..0511260 100644 --- a/README.md +++ b/README.md @@ -28,14 +28,6 @@ Then you should be able to start Limn by running `npm start` and see it in action at `http://localhost:8081`. - -## Testing using local data - -By default the instance you run will show graphs using production data. - -TODO: ssh tunnel, sed scripts - - ### Mac OS X You will need to install some header files for some of the Python dependencies @@ -63,3 +55,43 @@ `generate.py -h` for details) - Deploy to limn! (Ask analytics to get you access) + +## Testing using local data + +By default the instance you run will show graphs using production data. +To generate the data for your local instance you need to make the analytics +databases available to your local machine. If you have access to stat1, you +can do that by running: + + $ scripts/ssh + +in a separate terminal window and leaving it open. + +Then, you should create a file called `scripts/my.cnf.research` with the +following content: + + [client] + user=[analytics DB user] + password=[analytics DB password] + +Now, you should be able to run `generate.py` with config overrides: + + $ python generate.py -c scripts/config.yaml mobile + +When all the data is generated you still need to do one more thing to let +Limn know that it should use the local data. The hacky solution is to replace +all the data URLs temporarily. You can do it by running: + + $ scripts/localurl + +Now, you should be able to do: + + $ cd ~/limn + $ npm start + +and see your local instance data at `http://localhost:8081`. You have to +remember to replace all the local URLs to remote URLs before pushing your +changes though by running: + + $ scripts/remoteurl + diff --git a/scripts/config.yaml b/scripts/config.yaml new file mode 100644 index 0000000..c46e270 --- /dev/null +++ b/scripts/config.yaml @@ -0,0 +1,15 @@ +# config overrides for local testing +databases: + el: + host: "127.0.0.1" + port: 3307 + creds_file: scripts/my.cnf.research + db: log + commons: + host: "127.0.0.1" + port: 3308 + creds_file: scripts/my.cnf.research + db: commonswiki +output: + path: datafiles + url: /data/datafiles/mobile diff --git a/scripts/localurl b/scripts/localurl new file mode 100755 index 0000000..df498da --- /dev/null +++ b/scripts/localurl @@ -0,0 +1,3 @@ +# make all data URLs local (for testing) +find datasources/ -name "*.json" -print | xargs sed -i 's/http:\/\/stat1001\.wikimedia\.org\/limn-public-data\/mobile\/datafiles/\/data\/datafiles\/mobile/g' + diff --git a/scripts/remoteurl b/scripts/remoteurl new file mode 100755 index 0000000..ebb0b65 --- /dev/null +++ b/scripts/remoteurl @@ -0,0 +1,3 @@ +# make all data URLs remote (use before pushing to repo if you used localurl) +find datasources/ -name "*.json" -print | xargs sed -i 's/\/data\/datafiles\/mobile/http:\/\/stat1001\.wikimedia\.org\/limn-public-data\/mobile\/datafiles/g' + diff --git a/scripts/ssh b/scripts/ssh new file mode 100755 index 0000000..9d85aa3 --- /dev/null +++ b/scripts/ssh @@ -0,0 +1,3 @@ +# create SSH tunnel to EventLogging databases through stat1 +ssh -L 3307:s1-analytics-slave.eqiad.wmnet:3306 -L 3308:s4-analytics-slave.eqiad.wmnet:3306 stat1.wikimedia.org + -- To view, visit https://gerrit.wikimedia.org/r/76003 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I01dbecd5c0d4563cfa136005bb833499c8bd6fe6 Gerrit-PatchSet: 1 Gerrit-Project: analytics/limn-mobile-data Gerrit-Branch: master Gerrit-Owner: JGonera <[email protected]> Gerrit-Reviewer: Yuvipanda <[email protected]> _______________________________________________ MediaWiki-commits mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
