Hello Ottomata,
I'd like you to do a code review. Please visit
https://gerrit.wikimedia.org/r/189926
to review the following change.
Change subject: Mark udp2log jobs that are duplicated already on Hive
......................................................................
Mark udp2log jobs that are duplicated already on Hive
Change-Id: I35f8a7086c6ad86263704724af1da04171a84b23
---
M templates/udp2log/filters.erbium.erb
M templates/udp2log/filters.oxygen.erb
2 files changed, 18 insertions(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/26/189926/1
diff --git a/templates/udp2log/filters.erbium.erb
b/templates/udp2log/filters.erbium.erb
index 44f9785..025b4a3 100644
--- a/templates/udp2log/filters.erbium.erb
+++ b/templates/udp2log/filters.erbium.erb
@@ -14,11 +14,18 @@
pipe 100 /usr/bin/udp-filter -F '\t' -p Special:BannerRandom >> <%=
@template_variables['fundraising_log_directory']
%>/logs/bannerRequests-sampled100.tsv.log
### API Usage
+# This filter has been migrated to Hive. It's output is available on
+# stat1002 at /a/log/webrequest/archive/api
pipe 100 /usr/bin/udp-filter -F '\t' -p /w/api.php >> <%=
scope.lookupvar('role::logging::udp2log::erbium::webrequest_log_directory')
%>/api-usage.tsv.log
### GLAM NARA / National Archives - RT 2212
+# This filter has been migrated to Hive. It's output is available on
+# stat1002 at /a/log/webrequest/archive/glam_nara
pipe 10 /usr/bin/udp-filter -F '\t' -p _NARA_ -g -b country >>
<%=log_directory %>/glam_nara.tsv.log
### 0.0001 of all udp2log messages
## This log file is also on gadolinium for redundancy
+#
+# This filter has been migrated to Hive. It's output is available on
+# stat1002 at /a/log/webrequest/archive/sampled
file 1000 <%= log_directory %>/sampled-1000.tsv.log
diff --git a/templates/udp2log/filters.oxygen.erb
b/templates/udp2log/filters.oxygen.erb
index a0be6f1..7d06de6 100644
--- a/templates/udp2log/filters.oxygen.erb
+++ b/templates/udp2log/filters.oxygen.erb
@@ -7,9 +7,14 @@
### Capture all logs with 'zero=' set. The X-Analytics header is set with this
# by mobile varnish frontends upon getting a Wikipedia Zero request.
+#
+# This filter has been migrated to Hive. It's output is available on
+# stat1002 at /a/log/webrequest/archive/zero
pipe 1 /bin/grep -P 'zero=\d' >> <%= log_directory %>/zero.tsv.log
### All edits
+# This filter has been migrated to Hive. It's output is available on
+# stat1002 at /a/log/webrequest/archive/edits
pipe 1 /usr/bin/udp-filter -F '\t' -p action=submit,action=edit >> <%=
log_directory %>/edits.tsv.log
<%
@@ -21,8 +26,14 @@
### Mobile traffic filter
# All mobile traffic goes through CC-cp1044 varnish hosts.
+#
+# This filter has been migrated to Hive. It's output is available on
+# stat1002 at /a/log/webrequest/archive/mobile
pipe 100 /bin/grep -P '<%= mobile_hosts_regex %>' >> <%= log_directory
%>/mobile-sampled-100.tsv.log
### All 5xx error responses -- domas (now using udp-filter instead of
5xx-filter).
# pipe 1 <%= @template_variables['webrequest_filter_directory'] %>/5xx-filter
| awk -W interactive '$9 !~ "upload.wikimedia.org"' >> <%= log_directory
%>/5xx.tsv.log
+#
+# This filter has been migrated to Hive. It's output is available on
+# stat1002 at /a/log/webrequest/archive/5xx
pipe 1 /usr/bin/udp-filter -F '\t' -r -s '^5' | awk -W interactive '$9 !~
"upload.wikimedia.org"' >> <%= log_directory %>/5xx.tsv.log
--
To view, visit https://gerrit.wikimedia.org/r/189926
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I35f8a7086c6ad86263704724af1da04171a84b23
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: QChris <[email protected]>
Gerrit-Reviewer: Ottomata <[email protected]>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
