Stefan.petrea has submitted this change and it was merged.
Change subject: Added creation of output directory in cron script
......................................................................
Added creation of output directory in cron script
* Added creation of output directory in cron script
* Cleaned up cron script
Change-Id: Iaaedebfc7cf119f62fcf9f689c7bab62800c0228
---
M pageviews_reports/bin/stat1-cron-script.sh
A pageviews_reports/conf/json-to-wikireport.json
2 files changed, 32 insertions(+), 3 deletions(-)
Approvals:
Stefan.petrea: Verified; Looks good to me, approved
jenkins-bot: Verified
diff --git a/pageviews_reports/bin/stat1-cron-script.sh
b/pageviews_reports/bin/stat1-cron-script.sh
index 8bb6e9d..6988575 100755
--- a/pageviews_reports/bin/stat1-cron-script.sh
+++ b/pageviews_reports/bin/stat1-cron-script.sh
@@ -1,9 +1,8 @@
#!/bin/bash
#env
#set -x
-USER=`whoami`;
#echo $USER >> /tmp/cronlog_1
-stat1002_new_mobile=stat1002.wikimedia.org::a/wikistats_git/dumps/csv/csv_sp
+# stat1002_new_mobile=stat1002.wikimedia.org::a/wikistats_git/dumps/csv/csv_sp
eval $(perl -I$HOME/perl5/lib/perl5 -Mlocal::lib);
export PATH=$PATH:$HOME/local/bin;
#unpigz --help 2>> /tmp/cronlog_1
@@ -11,15 +10,28 @@
WIKISTATS_DIR=/a/wikistats_git
OUTPUT_DIR=/tmp/pageviews-full-cron
MOBILE_PAGEVIEWS_DIR=$WIKISTATS_DIR/pageviews_reports
+
+mkdir $OUTPUT_DIR;
/usr/bin/env perl -V > /tmp/cperlver;
/bin/date >> /tmp/cperlver;
+###########################################
# Clean up mappers output from previous run
+###########################################
rm -f $OUTPUT_DIR/map/*;
+###########################################
+# Process the data
+###########################################
/usr/bin/env perl -I$MOBILE_PAGEVIEWS_DIR/lib \
$MOBILE_PAGEVIEWS_DIR/bin/pageviews.pl \
$MOBILE_PAGEVIEWS_DIR/conf/stat1-full-cron.json 2>&1
>/tmp/cperlerr;
+
+###########################################
+# Copy the CSV so it can be picked up by
+# other cron jobs
+###########################################
cp $OUTPUT_DIR/PageViewsPerMonthAll.csv \
$WIKISTATS_DIR/dumps/csv/csv_sp/ ;
-rsync -av $OUTPUT_DIR/PageViewsPerMonthAll.csv $stat1002_new_mobile/ ;
+#rsync -av $OUTPUT_DIR/PageViewsPerMonthAll.csv $stat1002_new_mobile/ ;
+# test
diff --git a/pageviews_reports/conf/json-to-wikireport.json
b/pageviews_reports/conf/json-to-wikireport.json
new file mode 100644
index 0000000..b8ce9cc
--- /dev/null
+++ b/pageviews_reports/conf/json-to-wikireport.json
@@ -0,0 +1,17 @@
+{
+ "model" : "json",
+ "max-children" : 8,
+ "input-path" : "/home/user/new_mobile_pageviews_data",
+ "children-output-path" : "/home/user/new_mobile_pageviews_data",
+ "output-path" : "/home/user/new_mobile_pageviews_data",
+ "output-formats" : ["web"],
+ "logs-prefix" : "sampled",
+ "start" : {
+ "year" : 2012,
+ "month" : 8
+ },
+ "end" : {
+ "year" : 2013,
+ "month" : 2
+ }
+}
--
To view, visit https://gerrit.wikimedia.org/r/72556
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Iaaedebfc7cf119f62fcf9f689c7bab62800c0228
Gerrit-PatchSet: 2
Gerrit-Project: analytics/wikistats
Gerrit-Branch: master
Gerrit-Owner: Stefan.petrea <[email protected]>
Gerrit-Reviewer: Stefan.petrea <[email protected]>
Gerrit-Reviewer: jenkins-bot
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
