[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_upload: upgrade esams to varnish 5

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/405007 )

Change subject: cache_upload: upgrade esams to varnish 5
..


cache_upload: upgrade esams to varnish 5

Bug: T180433
Change-Id: Ia4746701606dd31da26fb49b13eaaa3e085d0f98
---
D hieradata/hosts/cp3034.yaml
D hieradata/hosts/cp3035.yaml
D hieradata/hosts/cp3037.yaml
D hieradata/hosts/cp3038.yaml
M hieradata/role/esams/cache/upload.yaml
5 files changed, 2 insertions(+), 8 deletions(-)

Approvals:
  Ema: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/hosts/cp3034.yaml b/hieradata/hosts/cp3034.yaml
deleted file mode 100644
index ca4f448..000
--- a/hieradata/hosts/cp3034.yaml
+++ /dev/null
@@ -1,2 +0,0 @@
-profile::cache::base::varnish_version: 5
-apt::use_experimental: true
diff --git a/hieradata/hosts/cp3035.yaml b/hieradata/hosts/cp3035.yaml
deleted file mode 100644
index ca4f448..000
--- a/hieradata/hosts/cp3035.yaml
+++ /dev/null
@@ -1,2 +0,0 @@
-profile::cache::base::varnish_version: 5
-apt::use_experimental: true
diff --git a/hieradata/hosts/cp3037.yaml b/hieradata/hosts/cp3037.yaml
deleted file mode 100644
index ca4f448..000
--- a/hieradata/hosts/cp3037.yaml
+++ /dev/null
@@ -1,2 +0,0 @@
-profile::cache::base::varnish_version: 5
-apt::use_experimental: true
diff --git a/hieradata/hosts/cp3038.yaml b/hieradata/hosts/cp3038.yaml
deleted file mode 100644
index ca4f448..000
--- a/hieradata/hosts/cp3038.yaml
+++ /dev/null
@@ -1,2 +0,0 @@
-profile::cache::base::varnish_version: 5
-apt::use_experimental: true
diff --git a/hieradata/role/esams/cache/upload.yaml 
b/hieradata/role/esams/cache/upload.yaml
index f6e702a..fbc1db5 100644
--- a/hieradata/role/esams/cache/upload.yaml
+++ b/hieradata/role/esams/cache/upload.yaml
@@ -1 +1,3 @@
 profile::cache::base::purge_varnishes: ['127.0.0.1:3128,1.0', 
'127.0.0.1:3127,1.0']
+profile::cache::base::varnish_version: 5
+apt::use_experimental: true

-- 
To view, visit https://gerrit.wikimedia.org/r/405007
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ia4746701606dd31da26fb49b13eaaa3e085d0f98
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_upload: upgrade esams to varnish 5

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/405007 )

Change subject: cache_upload: upgrade esams to varnish 5
..

cache_upload: upgrade esams to varnish 5

Bug: T180433
Change-Id: Ia4746701606dd31da26fb49b13eaaa3e085d0f98
---
D hieradata/hosts/cp3034.yaml
D hieradata/hosts/cp3035.yaml
D hieradata/hosts/cp3037.yaml
D hieradata/hosts/cp3038.yaml
M hieradata/role/esams/cache/upload.yaml
5 files changed, 2 insertions(+), 8 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/07/405007/1

diff --git a/hieradata/hosts/cp3034.yaml b/hieradata/hosts/cp3034.yaml
deleted file mode 100644
index ca4f448..000
--- a/hieradata/hosts/cp3034.yaml
+++ /dev/null
@@ -1,2 +0,0 @@
-profile::cache::base::varnish_version: 5
-apt::use_experimental: true
diff --git a/hieradata/hosts/cp3035.yaml b/hieradata/hosts/cp3035.yaml
deleted file mode 100644
index ca4f448..000
--- a/hieradata/hosts/cp3035.yaml
+++ /dev/null
@@ -1,2 +0,0 @@
-profile::cache::base::varnish_version: 5
-apt::use_experimental: true
diff --git a/hieradata/hosts/cp3037.yaml b/hieradata/hosts/cp3037.yaml
deleted file mode 100644
index ca4f448..000
--- a/hieradata/hosts/cp3037.yaml
+++ /dev/null
@@ -1,2 +0,0 @@
-profile::cache::base::varnish_version: 5
-apt::use_experimental: true
diff --git a/hieradata/hosts/cp3038.yaml b/hieradata/hosts/cp3038.yaml
deleted file mode 100644
index ca4f448..000
--- a/hieradata/hosts/cp3038.yaml
+++ /dev/null
@@ -1,2 +0,0 @@
-profile::cache::base::varnish_version: 5
-apt::use_experimental: true
diff --git a/hieradata/role/esams/cache/upload.yaml 
b/hieradata/role/esams/cache/upload.yaml
index f6e702a..fbc1db5 100644
--- a/hieradata/role/esams/cache/upload.yaml
+++ b/hieradata/role/esams/cache/upload.yaml
@@ -1 +1,3 @@
 profile::cache::base::purge_varnishes: ['127.0.0.1:3128,1.0', 
'127.0.0.1:3127,1.0']
+profile::cache::base::varnish_version: 5
+apt::use_experimental: true

-- 
To view, visit https://gerrit.wikimedia.org/r/405007
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia4746701606dd31da26fb49b13eaaa3e085d0f98
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_upload: upgrade cp3038 to varnish 5

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/404993 )

Change subject: cache_upload: upgrade cp3038 to varnish 5
..


cache_upload: upgrade cp3038 to varnish 5

Bug: T180433
Change-Id: I805f6e0a6eacf9374210d88925ad93c63b69a2dc
---
A hieradata/hosts/cp3038.yaml
1 file changed, 2 insertions(+), 0 deletions(-)

Approvals:
  Ema: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/hosts/cp3038.yaml b/hieradata/hosts/cp3038.yaml
new file mode 100644
index 000..ca4f448
--- /dev/null
+++ b/hieradata/hosts/cp3038.yaml
@@ -0,0 +1,2 @@
+profile::cache::base::varnish_version: 5
+apt::use_experimental: true

-- 
To view, visit https://gerrit.wikimedia.org/r/404993
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I805f6e0a6eacf9374210d88925ad93c63b69a2dc
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_upload: upgrade cp3038 to varnish 5

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/404993 )

Change subject: cache_upload: upgrade cp3038 to varnish 5
..

cache_upload: upgrade cp3038 to varnish 5

Bug: T180433
Change-Id: I805f6e0a6eacf9374210d88925ad93c63b69a2dc
---
A hieradata/hosts/cp3038.yaml
1 file changed, 2 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/93/404993/1

diff --git a/hieradata/hosts/cp3038.yaml b/hieradata/hosts/cp3038.yaml
new file mode 100644
index 000..ca4f448
--- /dev/null
+++ b/hieradata/hosts/cp3038.yaml
@@ -0,0 +1,2 @@
+profile::cache::base::varnish_version: 5
+apt::use_experimental: true

-- 
To view, visit https://gerrit.wikimedia.org/r/404993
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I805f6e0a6eacf9374210d88925ad93c63b69a2dc
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_upload: upgrade cp3035 to varnish 5

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/404990 )

Change subject: cache_upload: upgrade cp3035 to varnish 5
..


cache_upload: upgrade cp3035 to varnish 5

Bug: T180433
Change-Id: Icb967224e8fa179d94f2207328a0a3141d0cdc89
---
A hieradata/hosts/cp3035.yaml
1 file changed, 2 insertions(+), 0 deletions(-)

Approvals:
  Ema: Verified; Looks good to me, approved



diff --git a/hieradata/hosts/cp3035.yaml b/hieradata/hosts/cp3035.yaml
new file mode 100644
index 000..ca4f448
--- /dev/null
+++ b/hieradata/hosts/cp3035.yaml
@@ -0,0 +1,2 @@
+profile::cache::base::varnish_version: 5
+apt::use_experimental: true

-- 
To view, visit https://gerrit.wikimedia.org/r/404990
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Icb967224e8fa179d94f2207328a0a3141d0cdc89
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_upload: upgrade cp3035 to varnish 5

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/404990 )

Change subject: cache_upload: upgrade cp3035 to varnish 5
..

cache_upload: upgrade cp3035 to varnish 5

Bug: T180433
Change-Id: Icb967224e8fa179d94f2207328a0a3141d0cdc89
---
A hieradata/hosts/cp3035.yaml
1 file changed, 2 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/90/404990/1

diff --git a/hieradata/hosts/cp3035.yaml b/hieradata/hosts/cp3035.yaml
new file mode 100644
index 000..ca4f448
--- /dev/null
+++ b/hieradata/hosts/cp3035.yaml
@@ -0,0 +1,2 @@
+profile::cache::base::varnish_version: 5
+apt::use_experimental: true

-- 
To view, visit https://gerrit.wikimedia.org/r/404990
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Icb967224e8fa179d94f2207328a0a3141d0cdc89
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_upload: upgrade cp3037 to varnish 5

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/404979 )

Change subject: cache_upload: upgrade cp3037 to varnish 5
..


cache_upload: upgrade cp3037 to varnish 5

Bug: T180433
Change-Id: Ia33b5295d9a713ea152e6e9ad9208d00fc1905cc
---
A hieradata/hosts/cp3037.yaml
1 file changed, 2 insertions(+), 0 deletions(-)

Approvals:
  Ema: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/hosts/cp3037.yaml b/hieradata/hosts/cp3037.yaml
new file mode 100644
index 000..ca4f448
--- /dev/null
+++ b/hieradata/hosts/cp3037.yaml
@@ -0,0 +1,2 @@
+profile::cache::base::varnish_version: 5
+apt::use_experimental: true

-- 
To view, visit https://gerrit.wikimedia.org/r/404979
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ia33b5295d9a713ea152e6e9ad9208d00fc1905cc
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_upload: upgrade cp3037 to varnish 5

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/404979 )

Change subject: cache_upload: upgrade cp3037 to varnish 5
..

cache_upload: upgrade cp3037 to varnish 5

Bug: T180433
Change-Id: Ia33b5295d9a713ea152e6e9ad9208d00fc1905cc
---
A hieradata/hosts/cp3037.yaml
1 file changed, 2 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/79/404979/1

diff --git a/hieradata/hosts/cp3037.yaml b/hieradata/hosts/cp3037.yaml
new file mode 100644
index 000..ca4f448
--- /dev/null
+++ b/hieradata/hosts/cp3037.yaml
@@ -0,0 +1,2 @@
+profile::cache::base::varnish_version: 5
+apt::use_experimental: true

-- 
To view, visit https://gerrit.wikimedia.org/r/404979
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia33b5295d9a713ea152e6e9ad9208d00fc1905cc
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_upload: upgrade cp3034 to varnish 5

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/404944 )

Change subject: cache_upload: upgrade cp3034 to varnish 5
..


cache_upload: upgrade cp3034 to varnish 5

Bug: T180433
Change-Id: Iac1fbfe76c6d61d433604e35b9d9288793b5a011
---
A hieradata/hosts/cp3034.yaml
1 file changed, 2 insertions(+), 0 deletions(-)

Approvals:
  Ema: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/hosts/cp3034.yaml b/hieradata/hosts/cp3034.yaml
new file mode 100644
index 000..ca4f448
--- /dev/null
+++ b/hieradata/hosts/cp3034.yaml
@@ -0,0 +1,2 @@
+profile::cache::base::varnish_version: 5
+apt::use_experimental: true

-- 
To view, visit https://gerrit.wikimedia.org/r/404944
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Iac1fbfe76c6d61d433604e35b9d9288793b5a011
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_upload: upgrade cp3034 to varnish 5

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/404944 )

Change subject: cache_upload: upgrade cp3034 to varnish 5
..

cache_upload: upgrade cp3034 to varnish 5

Bug: T180433
Change-Id: Iac1fbfe76c6d61d433604e35b9d9288793b5a011
---
A hieradata/hosts/cp3034.yaml
1 file changed, 2 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/44/404944/1

diff --git a/hieradata/hosts/cp3034.yaml b/hieradata/hosts/cp3034.yaml
new file mode 100644
index 000..ca4f448
--- /dev/null
+++ b/hieradata/hosts/cp3034.yaml
@@ -0,0 +1,2 @@
+profile::cache::base::varnish_version: 5
+apt::use_experimental: true

-- 
To view, visit https://gerrit.wikimedia.org/r/404944
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Iac1fbfe76c6d61d433604e35b9d9288793b5a011
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_upload: use resp.reason in vtc test cases

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/404940 )

Change subject: cache_upload: use resp.reason in vtc test cases
..


cache_upload: use resp.reason in vtc test cases

resp.msg has been renamed into resp.reason in varnish 5.

Bug: T180433
Change-Id: Ib38a470dabc2a6ac5e48e240560804b1621b36a3
---
M modules/varnish/files/tests/upload/03-backend-if-cached.vtc
M modules/varnish/files/tests/upload/09-frontend-homepage-commons-redirect.vtc
2 files changed, 3 insertions(+), 3 deletions(-)

Approvals:
  Ema: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/varnish/files/tests/upload/03-backend-if-cached.vtc 
b/modules/varnish/files/tests/upload/03-backend-if-cached.vtc
index 34c85c7..b63e0c9 100644
--- a/modules/varnish/files/tests/upload/03-backend-if-cached.vtc
+++ b/modules/varnish/files/tests/upload/03-backend-if-cached.vtc
@@ -18,7 +18,7 @@
 txreq -url "/test" -hdr "Host: upload.wikimedia.org" -hdr "If-Cached: 
15f0fff99ed5aae4edffdd6496d7131f"
 rxresp
 expect resp.status == 412
-expect resp.msg == "Entity not in cache"
+expect resp.reason == "Entity not in cache"
 } -run
 
 client c2 {
@@ -35,5 +35,5 @@
 txreq -url "/test" -hdr "Host: upload.wikimedia.org" -hdr "If-Cached: 
mismatched-etag"
 rxresp
 expect resp.status == 412
-expect resp.msg == "Etag mismatch"
+expect resp.reason == "Etag mismatch"
 } -run
diff --git 
a/modules/varnish/files/tests/upload/09-frontend-homepage-commons-redirect.vtc 
b/modules/varnish/files/tests/upload/09-frontend-homepage-commons-redirect.vtc
index 7c75527..bcfa975 100644
--- 
a/modules/varnish/files/tests/upload/09-frontend-homepage-commons-redirect.vtc
+++ 
b/modules/varnish/files/tests/upload/09-frontend-homepage-commons-redirect.vtc
@@ -18,7 +18,7 @@
 rxresp
 
 expect resp.status == 301
-expect resp.msg == "Moved Permanently"
+expect resp.reason == "Moved Permanently"
 expect resp.http.Location == "https://commons.wikimedia.org/;
 expect resp.http.Connection == "keep-alive"
 expect resp.http.Content-Length == 0

-- 
To view, visit https://gerrit.wikimedia.org/r/404940
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ib38a470dabc2a6ac5e48e240560804b1621b36a3
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_upload: use resp.reason in vtc test cases

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/404940 )

Change subject: cache_upload: use resp.reason in vtc test cases
..

cache_upload: use resp.reason in vtc test cases

resp.msg has been renamed into resp.reason in varnish 5.

Bug: T180433
Change-Id: Ib38a470dabc2a6ac5e48e240560804b1621b36a3
---
M modules/varnish/files/tests/upload/03-backend-if-cached.vtc
M modules/varnish/files/tests/upload/09-frontend-homepage-commons-redirect.vtc
2 files changed, 3 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/40/404940/1

diff --git a/modules/varnish/files/tests/upload/03-backend-if-cached.vtc 
b/modules/varnish/files/tests/upload/03-backend-if-cached.vtc
index 34c85c7..b63e0c9 100644
--- a/modules/varnish/files/tests/upload/03-backend-if-cached.vtc
+++ b/modules/varnish/files/tests/upload/03-backend-if-cached.vtc
@@ -18,7 +18,7 @@
 txreq -url "/test" -hdr "Host: upload.wikimedia.org" -hdr "If-Cached: 
15f0fff99ed5aae4edffdd6496d7131f"
 rxresp
 expect resp.status == 412
-expect resp.msg == "Entity not in cache"
+expect resp.reason == "Entity not in cache"
 } -run
 
 client c2 {
@@ -35,5 +35,5 @@
 txreq -url "/test" -hdr "Host: upload.wikimedia.org" -hdr "If-Cached: 
mismatched-etag"
 rxresp
 expect resp.status == 412
-expect resp.msg == "Etag mismatch"
+expect resp.reason == "Etag mismatch"
 } -run
diff --git 
a/modules/varnish/files/tests/upload/09-frontend-homepage-commons-redirect.vtc 
b/modules/varnish/files/tests/upload/09-frontend-homepage-commons-redirect.vtc
index 7c75527..bcfa975 100644
--- 
a/modules/varnish/files/tests/upload/09-frontend-homepage-commons-redirect.vtc
+++ 
b/modules/varnish/files/tests/upload/09-frontend-homepage-commons-redirect.vtc
@@ -18,7 +18,7 @@
 rxresp
 
 expect resp.status == 301
-expect resp.msg == "Moved Permanently"
+expect resp.reason == "Moved Permanently"
 expect resp.http.Location == "https://commons.wikimedia.org/;
 expect resp.http.Connection == "keep-alive"
 expect resp.http.Content-Length == 0

-- 
To view, visit https://gerrit.wikimedia.org/r/404940
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib38a470dabc2a6ac5e48e240560804b1621b36a3
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Expand test coverage of server.py

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/404762 )

Change subject: Expand test coverage of server.py
..


Expand test coverage of server.py

Change-Id: I56c521a1a4f24c73839f76bcf52afb11efa739e1
---
M pybal/test/test_server.py
1 file changed, 52 insertions(+), 4 deletions(-)

Approvals:
  Ema: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/pybal/test/test_server.py b/pybal/test/test_server.py
index 01da0d9..e6d6bc1 100644
--- a/pybal/test/test_server.py
+++ b/pybal/test/test_server.py
@@ -9,12 +9,12 @@
 
 import mock
 
-import pybal.coordinator
+import pybal.server
 
 from twisted.python import failure
 from twisted.internet.reactor import getDelayedCalls
 
-from .fixtures import PyBalTestCase
+from .fixtures import PyBalTestCase, StubLVSService
 
 class ServerTestCase(PyBalTestCase):
 """Test case for `pybal.server.Server`."""
@@ -23,9 +23,10 @@
 super(ServerTestCase, self).setUp()
 
 self.server = pybal.server.Server(
-'example.com', mock.MagicMock())
+'example.com', self.lvsservice)
 
 self.mockMonitor = mock.MagicMock()
+self.mockCoordinator = mock.MagicMock()
 self.server.addMonitor(self.mockMonitor)
 
 self.exampleConfigDict = {
@@ -40,6 +41,26 @@
 for call in getDelayedCalls():
 if call.func.func_name == 'maybeParseConfig':
 call.cancel()
+
+def testEq(self):
+self.assertEquals(self.server, self.server)
+
+# Create a Server instance with different hostname
+otherServer = pybal.server.Server('other.example.com', self.lvsservice)
+self.assertNotEqual(self.server, otherServer)
+
+# Create a Server instance with equal hostname but different LVSService
+otherLVSService = StubLVSService(
+'otherservice',
+(self.protocol, self.ip, self.port, self.scheduler),
+self.config)
+otherServer = pybal.server.Server('example.com', otherLVSService)
+self.assertNotEqual(self.server, otherServer)
+
+def testHash(self):
+# Create a Server instance with different hostname
+otherServer = pybal.server.Server('other.example.com', self.lvsservice)
+self.assertNotEqual(hash(self.server), hash(otherServer))
 
 def testAddMonitor(self):
 self.assertIn(self.mockMonitor, self.server.monitors)
@@ -68,9 +89,31 @@
 self.assertEquals(self.server.ready, result)
 
 self.server.createMonitoringInstances = mock.MagicMock()
-deferred = self.server.initialize(coordinator=mock.MagicMock())
+deferred = self.server.initialize(self.mockCoordinator)
 deferred.addCallback(callback)
 return deferred
+
+@mock.patch('pybal.server.Server.createMonitoringInstances')
+def testReady(self, mock_createMonitoringInstances):
+r = self.server._ready(True, self.mockCoordinator)
+self.assertTrue(r)
+self.assertTrue(self.server.ready)
+mock_createMonitoringInstances.assert_called()
+
+def testInitFailed(self):
+r = self.server._initFailed(failure.Failure(Exception("Fake failure")))
+self.assertFalse(r)
+self.assertFalse(self.server.ready)
+
+def testCreateMonitoringInstances(self):
+assert 'monitors' not in self.config
+self.assertRaises(KeyError,
+self.server.createMonitoringInstances, self.mockCoordinator)
+
+self.config['monitors'] = "[ \"NonexistentMonitor\" ]"
+self.server.createMonitoringInstances(self.mockCoordinator)
+
+# TODO: test creation of a (mock) monitor
 
 def testCalcStatus(self):
 self.mockMonitor.up = True
@@ -95,6 +138,11 @@
 self.assertFalse(self.server.calcStatus())
 self.assertTrue(self.server.calcPartialStatus())
 
+def testTextStatus(self):
+textStatus = self.server.textStatus()
+self.assertTrue(isinstance(textStatus, str))
+self.assertEquals(len(textStatus.split('/')), 3)
+
 def testMaintainState(self):
 self.server.pooled = True
 self.server.enabled = False

-- 
To view, visit https://gerrit.wikimedia.org/r/404762
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I56c521a1a4f24c73839f76bcf52afb11efa739e1
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Mark Bergsma 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Volans 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Separate out coordinator.Server into its own module

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/404713 )

Change subject: Separate out coordinator.Server into its own module
..


Separate out coordinator.Server into its own module

Change-Id: I925f0abe553cd82603014cd8a0760be8bc8ad880
---
M pybal/coordinator.py
A pybal/server.py
M pybal/test/test_coordinator.py
M pybal/test/test_server.py
4 files changed, 277 insertions(+), 264 deletions(-)

Approvals:
  Ema: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/pybal/coordinator.py b/pybal/coordinator.py
index a4b4e16..b7b3591 100755
--- a/pybal/coordinator.py
+++ b/pybal/coordinator.py
@@ -6,270 +6,14 @@
 
 LVS Squid balancer/monitor for managing the Wikimedia Squid servers using LVS
 """
-import importlib
-import random
-import socket
 
-from twisted.internet import defer, reactor
-from twisted.names import client, dns
-from twisted.python import failure
+from twisted.internet import defer
 
 from pybal import config, util
 from pybal.metrics import Counter, Gauge
+import pybal.server
 
 log = util.log
-
-
-class Server:
-"""
-Class that maintains configuration and state of a single (real)server
-"""
-
-# Defaults
-DEF_STATE = True
-DEF_WEIGHT = 10
-
-# Set of attributes allowed to be overridden in a server list
-allowedConfigKeys = [ ('host', str), ('weight', int), ('enabled', bool) ]
-
-def __init__(self, host, lvsservice, addressFamily=None):
-"""Constructor"""
-
-self.host = host
-self.lvsservice = lvsservice
-if addressFamily:
-self.addressFamily = addressFamily
-else:
-self.addressFamily = (':' in self.lvsservice.ip) and 
socket.AF_INET6 or socket.AF_INET
-self.ip = None
-self.port = 80
-self.ip4_addresses = set()
-self.ip6_addresses = set()
-self.monitors = set()
-
-# A few invariants that SHOULD be maintained (but currently may not 
be):
-# P0: pooled => enabled /\ ready
-# P1: up => pooled \/ !enabled \/ !ready
-# P2: pooled => up \/ !canDepool
-
-self.weight = self.DEF_WEIGHT
-self.up = False
-self.pooled = False
-self.enabled = True
-self.ready = False
-self.modified = None
-
-def __eq__(self, other):
-return isinstance(other, Server) and self.host == other.host and 
self.lvsservice == other.lvsservice
-
-def __hash__(self):
-return hash(self.host)
-
-def addMonitor(self, monitor):
-"""Adds a monitor instance to the set"""
-
-self.monitors.add(monitor)
-
-def removeMonitors(self):
-"""Removes all monitors"""
-
-for monitor in self.monitors:
-monitor.stop()
-
-self.monitors.clear()
-
-def resolveHostname(self):
-"""Attempts to resolve the server's hostname to an IP address for 
better reliability."""
-
-timeout = [1, 2, 5]
-lookups = []
-
-query = dns.Query(self.host, dns.A)
-lookups.append(client.lookupAddress(self.host, timeout
-).addCallback(self._lookupFinished, socket.AF_INET, query))
-
-query = dns.Query(self.host, dns.)
-lookups.append(client.lookupIPV6Address(self.host, timeout
-).addCallback(self._lookupFinished, socket.AF_INET6, query))
-
-return defer.DeferredList(lookups).addBoth(self._hostnameResolved)
-
-def _lookupFinished(self, (answers, authority, additional), addressFamily, 
query):
-ips = set([socket.inet_ntop(addressFamily, r.payload.address)
-   for r in answers
-   if r.name == query.name and r.type == query.type])
-
-if query.type == dns.A:
-self.ip4_addresses = ips
-elif query.type == dns.:
-self.ip6_addresses = ips
-
-# TODO: expire TTL
-#if self.ip:
-#minTTL = min([r.ttl for r in answers
-#  if r.name == query.name and r.type == query.type])
-
-return ips
-
-def _hostnameResolved(self, result):
-# Pick *1* main ip address to use. Prefer any existing one
-# if still available.
-
-addr = " ".join(
-list(self.ip4_addresses) + list(self.ip6_addresses))
-msg = "Resolved {} to addresses {}".format(self.host, addr)
-log.debug(msg)
-
-ip_addresses = {
-socket.AF_INET:
-self.ip4_addresses,
-socket.AF_INET6:
-self.ip6_addresses
-}[self.addressFamily]
-
-try:
-if not self.ip or self.ip not in ip_addresses:
-self.ip = random.choice(list(ip_addresses))
-# TODO: (re)pool
-except IndexError:
-return failure.Failure() # TODO: be more specific?
-else:
-return True
-
-def destroy(self):
-self.enabled = False

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Add unit test cases for Server

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/404704 )

Change subject: Add unit test cases for Server
..


Add unit test cases for Server

Basic test cases for most of the Server class, with the notable
exception of method createMonitoringInstances.

Change-Id: Icdc4f9753e95859623b512634c314649fcfda56d
---
A pybal/test/test_server.py
1 file changed, 128 insertions(+), 0 deletions(-)

Approvals:
  Ema: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/pybal/test/test_server.py b/pybal/test/test_server.py
new file mode 100644
index 000..3eefca4
--- /dev/null
+++ b/pybal/test/test_server.py
@@ -0,0 +1,128 @@
+# -*- coding: utf-8 -*-
+"""
+  PyBal unit tests
+  
+
+  This module contains tests for `pybal.coordinator.Server`.
+
+"""
+
+import mock
+
+import pybal.coordinator
+
+from twisted.python import failure
+from twisted.internet.reactor import getDelayedCalls
+
+from .fixtures import PyBalTestCase
+
+class ServerTestCase(PyBalTestCase):
+"""Test case for `pybal.coordinator.Server`."""
+
+def setUp(self):
+super(ServerTestCase, self).setUp()
+
+self.server = pybal.coordinator.Server(
+'example.com', mock.MagicMock())
+
+self.mockMonitor = mock.MagicMock()
+self.server.addMonitor(self.mockMonitor)
+
+self.exampleConfigDict = {
+'host': "example1.example.com",
+'weight': 66,
+'enabled': True,
+# FIXME: bug in Server.merge
+#'rogue': "this attribute should not be merged"
+}
+
+def tearDown(self):
+for call in getDelayedCalls():
+if call.func.func_name == 'maybeParseConfig':
+call.cancel()
+
+def testAddMonitor(self):
+self.assertIn(self.mockMonitor, self.server.monitors)
+
+def testRemoveMonitors(self):
+self.server.removeMonitors()
+self.assertEqual(len(self.server.monitors), 0)
+self.mockMonitor.stop.assert_called()
+
+def testResolveHostname(self):
+def callback(result):
+self.assertTrue((result == True or isinstance(result, 
failure.Failure)))
+
+deferred = self.server.resolveHostname()
+deferred.addCallback(callback)
+return deferred
+
+def testDestroy(self):
+self.server.destroy()
+self.assertFalse(self.server.enabled)
+self.assertEqual(len(self.server.monitors), 0)
+
+def testInitialize(self):
+def callback(result):
+self.assertTrue(isinstance(result, bool))
+self.assertEquals(self.server.ready, result)
+
+self.server.createMonitoringInstances = mock.MagicMock()
+deferred = self.server.initialize(coordinator=mock.MagicMock())
+deferred.addCallback(callback)
+return deferred
+
+def testCalcStatus(self):
+self.mockMonitor.up = True
+self.assertTrue(self.server.calcStatus())
+self.assertTrue(self.server.calcPartialStatus())
+
+m = mock.MagicMock()
+m.up = True
+self.server.addMonitor(m)
+self.assertTrue(self.server.calcStatus())
+self.assertTrue(self.server.calcPartialStatus())
+
+m.up = False
+self.assertFalse(self.server.calcStatus())
+self.assertTrue(self.server.calcPartialStatus())
+
+self.mockMonitor.up = False
+self.assertFalse(self.server.calcPartialStatus())
+
+# Currently, no monitors implies False Status
+self.server.removeMonitors()
+self.assertFalse(self.server.calcStatus())
+self.assertTrue(self.server.calcPartialStatus())
+
+def testMaintainState(self):
+self.server.pooled = True
+self.server.enabled = False
+self.server.maintainState()
+self.assertFalse(self.server.pooled)
+
+self.server.pooled = False
+self.server.enabled = True
+self.server.maintainState()
+self.assertFalse(self.server.up)
+
+def testMerge(self):
+self.server.merge(self.exampleConfigDict)
+self.assertEquals(self.server.host, self.exampleConfigDict['host'])
+self.assertEquals(self.server.weight, self.exampleConfigDict['weight'])
+self.assertEquals(self.server.enabled, 
self.exampleConfigDict['enabled'])
+self.assertDictContainsSubset(self.exampleConfigDict, 
self.server.__dict__)
+
+def testDumpState(self):
+state = self.server.dumpState()
+self.assertLessEqual(
+{'pooled', 'weight', 'up', 'enabled'},
+set(state.keys()))
+
+def testBuildServer(self):
+server = self.server.buildServer(
+hostName=self.exampleConfigDict['host'],
+configuration=self.exampleConfigDict,
+lvsservice=mock.MagicMock())
+self.assertTrue(isinstance(server, pybal.coordinator.Server))
+

[MediaWiki-commits] [Gerrit] operations...pybal[1.14]: 1.14.3: canDepool and alert instrumentation bugfixes

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/404695 )

Change subject: 1.14.3: canDepool and alert instrumentation bugfixes
..


1.14.3: canDepool and alert instrumentation bugfixes

- Use up-and-enabled servers in can-depool logic (Bug: T184715)
  1264a784bbcbedee37466102246775e2ee26367a

- Alerts instrumentation: return instance of bytes (Bug: T184721)
  3505626c9d1590e84525560edbc2f5bf01a7be99

Change-Id: Ia7675b06eaa56c742b71827b68ba12f637c65744
(cherry picked from commit b9c561d38c32e4624fa3fe98afc384daa6369ac9)
---
M debian/changelog
1 file changed, 10 insertions(+), 0 deletions(-)

Approvals:
  Ema: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/debian/changelog b/debian/changelog
index 130..43c39cf 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,13 @@
+pybal (1.14.3) jessie-wikimedia; urgency=medium
+
+  * Use up-and-enabled servers in can-depool logic (Bug: T184715)
+1264a784bbcbedee37466102246775e2ee26367a
+
+  * Alerts instrumentation: return instance of bytes (Bug: T184721)
+3505626c9d1590e84525560edbc2f5bf01a7be99
+
+ -- Emanuele Rocca   Wed, 17 Jan 2018 16:49:48 +0100
+
 pybal (1.14.2) jessie-wikimedia; urgency=medium
 
   * runcommand: do not crash on empty runcommand.arguments (Bug: T178149)

-- 
To view, visit https://gerrit.wikimedia.org/r/404695
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ia7675b06eaa56c742b71827b68ba12f637c65744
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: 1.14
Gerrit-Owner: Ema 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[1.14]: 1.14.3: canDepool and alert instrumentation bugfixes

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/404695 )

Change subject: 1.14.3: canDepool and alert instrumentation bugfixes
..

1.14.3: canDepool and alert instrumentation bugfixes

- Use up-and-enabled servers in can-depool logic (Bug: T184715)
  1264a784bbcbedee37466102246775e2ee26367a

- Alerts instrumentation: return instance of bytes (Bug: T184721)
  3505626c9d1590e84525560edbc2f5bf01a7be99

Change-Id: Ia7675b06eaa56c742b71827b68ba12f637c65744
(cherry picked from commit b9c561d38c32e4624fa3fe98afc384daa6369ac9)
---
M debian/changelog
1 file changed, 10 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/95/404695/1

diff --git a/debian/changelog b/debian/changelog
index 130..43c39cf 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,13 @@
+pybal (1.14.3) jessie-wikimedia; urgency=medium
+
+  * Use up-and-enabled servers in can-depool logic (Bug: T184715)
+1264a784bbcbedee37466102246775e2ee26367a
+
+  * Alerts instrumentation: return instance of bytes (Bug: T184721)
+3505626c9d1590e84525560edbc2f5bf01a7be99
+
+ -- Emanuele Rocca   Wed, 17 Jan 2018 16:49:48 +0100
+
 pybal (1.14.2) jessie-wikimedia; urgency=medium
 
   * runcommand: do not crash on empty runcommand.arguments (Bug: T178149)

-- 
To view, visit https://gerrit.wikimedia.org/r/404695
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia7675b06eaa56c742b71827b68ba12f637c65744
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: 1.14
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[master]: 1.14.3: canDepool and alert instrumentation bugfixes

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/404694 )

Change subject: 1.14.3: canDepool and alert instrumentation bugfixes
..


1.14.3: canDepool and alert instrumentation bugfixes

- Use up-and-enabled servers in can-depool logic (Bug: T184715)
  1264a784bbcbedee37466102246775e2ee26367a

- Alerts instrumentation: return instance of bytes (Bug: T184721)
  3505626c9d1590e84525560edbc2f5bf01a7be99

Change-Id: Ia7675b06eaa56c742b71827b68ba12f637c65744
---
M debian/changelog
1 file changed, 10 insertions(+), 0 deletions(-)

Approvals:
  Ema: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/debian/changelog b/debian/changelog
index 130..43c39cf 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,13 @@
+pybal (1.14.3) jessie-wikimedia; urgency=medium
+
+  * Use up-and-enabled servers in can-depool logic (Bug: T184715)
+1264a784bbcbedee37466102246775e2ee26367a
+
+  * Alerts instrumentation: return instance of bytes (Bug: T184721)
+3505626c9d1590e84525560edbc2f5bf01a7be99
+
+ -- Emanuele Rocca   Wed, 17 Jan 2018 16:49:48 +0100
+
 pybal (1.14.2) jessie-wikimedia; urgency=medium
 
   * runcommand: do not crash on empty runcommand.arguments (Bug: T178149)

-- 
To view, visit https://gerrit.wikimedia.org/r/404694
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ia7675b06eaa56c742b71827b68ba12f637c65744
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Ema 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[master]: 1.14.3: canDepool and alert instrumentation bugfixes

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/404694 )

Change subject: 1.14.3: canDepool and alert instrumentation bugfixes
..

1.14.3: canDepool and alert instrumentation bugfixes

- Use up-and-enabled servers in can-depool logic (Bug: T184715)
  1264a784bbcbedee37466102246775e2ee26367a

- Alerts instrumentation: return instance of bytes (Bug: T184721)
  3505626c9d1590e84525560edbc2f5bf01a7be99

Change-Id: Ia7675b06eaa56c742b71827b68ba12f637c65744
---
M debian/changelog
1 file changed, 10 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/94/404694/1

diff --git a/debian/changelog b/debian/changelog
index 130..43c39cf 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,13 @@
+pybal (1.14.3) jessie-wikimedia; urgency=medium
+
+  * Use up-and-enabled servers in can-depool logic (Bug: T184715)
+1264a784bbcbedee37466102246775e2ee26367a
+
+  * Alerts instrumentation: return instance of bytes (Bug: T184721)
+3505626c9d1590e84525560edbc2f5bf01a7be99
+
+ -- Emanuele Rocca   Wed, 17 Jan 2018 16:49:48 +0100
+
 pybal (1.14.2) jessie-wikimedia; urgency=medium
 
   * runcommand: do not crash on empty runcommand.arguments (Bug: T178149)

-- 
To view, visit https://gerrit.wikimedia.org/r/404694
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia7675b06eaa56c742b71827b68ba12f637c65744
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[1.14]: Use up-and-enabled servers in can-depool logic

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/404680 )

Change subject: Use up-and-enabled servers in can-depool logic
..


Use up-and-enabled servers in can-depool logic

The number of pooled servers should not be computed as `total-servers -
servers-down`. We need to consider hosts which are serving traffic, that
is: servers administratively enabled and up according to monitoring.
Change can-depool logic accordingly.

Bug: T184715
Change-Id: Ic83382938810331dd4292f45cbb85f6aaa7b7707
(cherry picked from commit 1264a784bbcbedee37466102246775e2ee26367a)
---
M pybal/coordinator.py
A pybal/test/test_coordinator.py
2 files changed, 113 insertions(+), 4 deletions(-)

Approvals:
  Ema: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/pybal/coordinator.py b/pybal/coordinator.py
index a526b5b..a4b4e16 100755
--- a/pybal/coordinator.py
+++ b/pybal/coordinator.py
@@ -442,11 +442,21 @@
 def canDepool(self):
 """Returns a boolean denoting whether another server can be depooled"""
 
-# Construct a list of servers that have status 'down'
-downServers = [server for server in self.servers.itervalues() if not 
server.up]
+# Total number of servers
+totalServerCount = len(self.servers)
 
-# The total amount of pooled servers may never drop below a configured 
threshold
-return len(self.servers) - len(downServers) >= len(self.servers) * 
self.lvsservice.getDepoolThreshold()
+# Number of hosts considered to be up by PyBal's monitoring and
+# administratively enabled. Under normal circumstances, they would be
+# the hosts serving traffic.
+# However, a host can go down after PyBal has reached the depool
+# threshold for the service the host belongs to. In that case, the
+# misbehaving server is kept pooled. This count does not include such
+# hosts.
+upServerCount = len([server for server in self.servers.itervalues() if 
server.up and server.enabled])
+
+# The total amount of hosts serving traffic may never drop below a
+# configured threshold
+return upServerCount >= totalServerCount * 
self.lvsservice.getDepoolThreshold()
 
 def onConfigUpdate(self, config):
 """Parses the server list and changes the state accordingly."""
diff --git a/pybal/test/test_coordinator.py b/pybal/test/test_coordinator.py
new file mode 100644
index 000..35cede6
--- /dev/null
+++ b/pybal/test/test_coordinator.py
@@ -0,0 +1,99 @@
+# -*- coding: utf-8 -*-
+"""
+  PyBal unit tests
+  
+
+  This module contains tests for `pybal.coordinator`.
+
+"""
+import mock
+
+import pybal.coordinator
+import pybal.util
+
+from twisted.internet.reactor import getDelayedCalls
+
+from .fixtures import PyBalTestCase
+
+
+class CoordinatorTestCase(PyBalTestCase):
+"""Test case for `pybal.coordinator.Coordinator`."""
+
+def setUp(self):
+super(CoordinatorTestCase, self).setUp()
+
+configUrl = "file:///dev/null"
+
+self.coordinator = pybal.coordinator.Coordinator(
+mock.MagicMock(), configUrl)
+
+self.coordinator.lvsservice.getDepoolThreshold = mock.MagicMock(
+return_value=0.5)
+
+pybal.coordinator.Server.initialize = mock.MagicMock()
+
+def tearDown(self):
+self.coordinator.configObserver.reloadTask.stop()
+
+for call in getDelayedCalls():
+if call.func.func_name == 'maybeParseConfig':
+call.cancel()
+
+def setServers(self, servers):
+self.coordinator.onConfigUpdate(config=servers)
+
+for server in self.coordinator.servers.itervalues():
+server.up = True
+server.enabled = True
+
+def test2serversCanDepool(self):
+servers = {
+'cp1045.eqiad.wmnet': {},
+'cp1046.eqiad.wmnet': {},
+}
+self.setServers(servers)
+
+# 2/2 hosts serving traffic. We can depool.
+self.assertTrue(self.coordinator.canDepool())
+
+# 1 host goes down
+self.coordinator.servers['cp1045.eqiad.wmnet'].up = False
+
+# By depooling, we would end up with 1/2 hosts serving traffic. We can
+# depool.
+self.assertTrue(self.coordinator.canDepool())
+
+# The other host goes down too
+self.coordinator.servers['cp1046.eqiad.wmnet'].up = False
+
+# By depooling, we would end up with 0/2 hosts serving traffic. We
+# cannot depool.
+self.assertFalse(self.coordinator.canDepool())
+
+def test4serversCanDepool(self):
+servers = {
+'cp1045.eqiad.wmnet': {},
+'cp1046.eqiad.wmnet': {},
+'cp1047.eqiad.wmnet': {},
+'cp1048.eqiad.wmnet': {},
+}
+self.setServers(servers)
+
+self.coordinator.servers['cp1045.eqiad.wmnet'].enabled = 

[MediaWiki-commits] [Gerrit] operations...pybal[1.14]: Use up-and-enabled servers in can-depool logic

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/404680 )

Change subject: Use up-and-enabled servers in can-depool logic
..

Use up-and-enabled servers in can-depool logic

The number of pooled servers should not be computed as `total-servers -
servers-down`. We need to consider hosts which are serving traffic, that
is: servers administratively enabled and up according to monitoring.
Change can-depool logic accordingly.

Bug: T184715
Change-Id: Ic83382938810331dd4292f45cbb85f6aaa7b7707
(cherry picked from commit 1264a784bbcbedee37466102246775e2ee26367a)
---
M pybal/coordinator.py
A pybal/test/test_coordinator.py
2 files changed, 113 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/80/404680/1

diff --git a/pybal/coordinator.py b/pybal/coordinator.py
index a526b5b..a4b4e16 100755
--- a/pybal/coordinator.py
+++ b/pybal/coordinator.py
@@ -442,11 +442,21 @@
 def canDepool(self):
 """Returns a boolean denoting whether another server can be depooled"""
 
-# Construct a list of servers that have status 'down'
-downServers = [server for server in self.servers.itervalues() if not 
server.up]
+# Total number of servers
+totalServerCount = len(self.servers)
 
-# The total amount of pooled servers may never drop below a configured 
threshold
-return len(self.servers) - len(downServers) >= len(self.servers) * 
self.lvsservice.getDepoolThreshold()
+# Number of hosts considered to be up by PyBal's monitoring and
+# administratively enabled. Under normal circumstances, they would be
+# the hosts serving traffic.
+# However, a host can go down after PyBal has reached the depool
+# threshold for the service the host belongs to. In that case, the
+# misbehaving server is kept pooled. This count does not include such
+# hosts.
+upServerCount = len([server for server in self.servers.itervalues() if 
server.up and server.enabled])
+
+# The total amount of hosts serving traffic may never drop below a
+# configured threshold
+return upServerCount >= totalServerCount * 
self.lvsservice.getDepoolThreshold()
 
 def onConfigUpdate(self, config):
 """Parses the server list and changes the state accordingly."""
diff --git a/pybal/test/test_coordinator.py b/pybal/test/test_coordinator.py
new file mode 100644
index 000..35cede6
--- /dev/null
+++ b/pybal/test/test_coordinator.py
@@ -0,0 +1,99 @@
+# -*- coding: utf-8 -*-
+"""
+  PyBal unit tests
+  
+
+  This module contains tests for `pybal.coordinator`.
+
+"""
+import mock
+
+import pybal.coordinator
+import pybal.util
+
+from twisted.internet.reactor import getDelayedCalls
+
+from .fixtures import PyBalTestCase
+
+
+class CoordinatorTestCase(PyBalTestCase):
+"""Test case for `pybal.coordinator.Coordinator`."""
+
+def setUp(self):
+super(CoordinatorTestCase, self).setUp()
+
+configUrl = "file:///dev/null"
+
+self.coordinator = pybal.coordinator.Coordinator(
+mock.MagicMock(), configUrl)
+
+self.coordinator.lvsservice.getDepoolThreshold = mock.MagicMock(
+return_value=0.5)
+
+pybal.coordinator.Server.initialize = mock.MagicMock()
+
+def tearDown(self):
+self.coordinator.configObserver.reloadTask.stop()
+
+for call in getDelayedCalls():
+if call.func.func_name == 'maybeParseConfig':
+call.cancel()
+
+def setServers(self, servers):
+self.coordinator.onConfigUpdate(config=servers)
+
+for server in self.coordinator.servers.itervalues():
+server.up = True
+server.enabled = True
+
+def test2serversCanDepool(self):
+servers = {
+'cp1045.eqiad.wmnet': {},
+'cp1046.eqiad.wmnet': {},
+}
+self.setServers(servers)
+
+# 2/2 hosts serving traffic. We can depool.
+self.assertTrue(self.coordinator.canDepool())
+
+# 1 host goes down
+self.coordinator.servers['cp1045.eqiad.wmnet'].up = False
+
+# By depooling, we would end up with 1/2 hosts serving traffic. We can
+# depool.
+self.assertTrue(self.coordinator.canDepool())
+
+# The other host goes down too
+self.coordinator.servers['cp1046.eqiad.wmnet'].up = False
+
+# By depooling, we would end up with 0/2 hosts serving traffic. We
+# cannot depool.
+self.assertFalse(self.coordinator.canDepool())
+
+def test4serversCanDepool(self):
+servers = {
+'cp1045.eqiad.wmnet': {},
+'cp1046.eqiad.wmnet': {},
+'cp1047.eqiad.wmnet': {},
+'cp1048.eqiad.wmnet': {},
+}
+self.setServers(servers)
+
+

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Use up-and-enabled servers in can-depool logic

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/403677 )

Change subject: Use up-and-enabled servers in can-depool logic
..


Use up-and-enabled servers in can-depool logic

The number of pooled servers should not be computed as `total-servers -
servers-down`. We need to consider hosts which are serving traffic, that
is: servers administratively enabled and up according to monitoring.
Change can-depool logic accordingly.

Bug: T184715
Change-Id: Ic83382938810331dd4292f45cbb85f6aaa7b7707
---
M pybal/coordinator.py
A pybal/test/test_coordinator.py
2 files changed, 113 insertions(+), 4 deletions(-)

Approvals:
  Ema: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/pybal/coordinator.py b/pybal/coordinator.py
index a526b5b..a4b4e16 100755
--- a/pybal/coordinator.py
+++ b/pybal/coordinator.py
@@ -442,11 +442,21 @@
 def canDepool(self):
 """Returns a boolean denoting whether another server can be depooled"""
 
-# Construct a list of servers that have status 'down'
-downServers = [server for server in self.servers.itervalues() if not 
server.up]
+# Total number of servers
+totalServerCount = len(self.servers)
 
-# The total amount of pooled servers may never drop below a configured 
threshold
-return len(self.servers) - len(downServers) >= len(self.servers) * 
self.lvsservice.getDepoolThreshold()
+# Number of hosts considered to be up by PyBal's monitoring and
+# administratively enabled. Under normal circumstances, they would be
+# the hosts serving traffic.
+# However, a host can go down after PyBal has reached the depool
+# threshold for the service the host belongs to. In that case, the
+# misbehaving server is kept pooled. This count does not include such
+# hosts.
+upServerCount = len([server for server in self.servers.itervalues() if 
server.up and server.enabled])
+
+# The total amount of hosts serving traffic may never drop below a
+# configured threshold
+return upServerCount >= totalServerCount * 
self.lvsservice.getDepoolThreshold()
 
 def onConfigUpdate(self, config):
 """Parses the server list and changes the state accordingly."""
diff --git a/pybal/test/test_coordinator.py b/pybal/test/test_coordinator.py
new file mode 100644
index 000..35cede6
--- /dev/null
+++ b/pybal/test/test_coordinator.py
@@ -0,0 +1,99 @@
+# -*- coding: utf-8 -*-
+"""
+  PyBal unit tests
+  
+
+  This module contains tests for `pybal.coordinator`.
+
+"""
+import mock
+
+import pybal.coordinator
+import pybal.util
+
+from twisted.internet.reactor import getDelayedCalls
+
+from .fixtures import PyBalTestCase
+
+
+class CoordinatorTestCase(PyBalTestCase):
+"""Test case for `pybal.coordinator.Coordinator`."""
+
+def setUp(self):
+super(CoordinatorTestCase, self).setUp()
+
+configUrl = "file:///dev/null"
+
+self.coordinator = pybal.coordinator.Coordinator(
+mock.MagicMock(), configUrl)
+
+self.coordinator.lvsservice.getDepoolThreshold = mock.MagicMock(
+return_value=0.5)
+
+pybal.coordinator.Server.initialize = mock.MagicMock()
+
+def tearDown(self):
+self.coordinator.configObserver.reloadTask.stop()
+
+for call in getDelayedCalls():
+if call.func.func_name == 'maybeParseConfig':
+call.cancel()
+
+def setServers(self, servers):
+self.coordinator.onConfigUpdate(config=servers)
+
+for server in self.coordinator.servers.itervalues():
+server.up = True
+server.enabled = True
+
+def test2serversCanDepool(self):
+servers = {
+'cp1045.eqiad.wmnet': {},
+'cp1046.eqiad.wmnet': {},
+}
+self.setServers(servers)
+
+# 2/2 hosts serving traffic. We can depool.
+self.assertTrue(self.coordinator.canDepool())
+
+# 1 host goes down
+self.coordinator.servers['cp1045.eqiad.wmnet'].up = False
+
+# By depooling, we would end up with 1/2 hosts serving traffic. We can
+# depool.
+self.assertTrue(self.coordinator.canDepool())
+
+# The other host goes down too
+self.coordinator.servers['cp1046.eqiad.wmnet'].up = False
+
+# By depooling, we would end up with 0/2 hosts serving traffic. We
+# cannot depool.
+self.assertFalse(self.coordinator.canDepool())
+
+def test4serversCanDepool(self):
+servers = {
+'cp1045.eqiad.wmnet': {},
+'cp1046.eqiad.wmnet': {},
+'cp1047.eqiad.wmnet': {},
+'cp1048.eqiad.wmnet': {},
+}
+self.setServers(servers)
+
+self.coordinator.servers['cp1045.eqiad.wmnet'].enabled = False
+
+# 3/4 hosts serving traffic. We can depool.
+

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Revert "eqiad: temporarily remove chromium from LVS nameserv...

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/404674 )

Change subject: Revert "eqiad: temporarily remove chromium from LVS nameservers"
..


Revert "eqiad: temporarily remove chromium from LVS nameservers"

The host is back online.

This reverts commit 676bde44b8ffc5337bb1f57dd46de57a742da779.

Change-Id: Id585628b9f37e591a1d013ba0391f8853ec44f0f
---
M hieradata/role/eqiad/lvs/balancer.yaml
1 file changed, 1 insertion(+), 0 deletions(-)

Approvals:
  Ema: Verified; Looks good to me, approved



diff --git a/hieradata/role/eqiad/lvs/balancer.yaml 
b/hieradata/role/eqiad/lvs/balancer.yaml
index 888e756..b14b9ba 100644
--- a/hieradata/role/eqiad/lvs/balancer.yaml
+++ b/hieradata/role/eqiad/lvs/balancer.yaml
@@ -14,4 +14,5 @@
 # (doing this for all lvs for now, see T103921)
 profile::base::nameservers:
   - '208.80.154.50' # hydrogen
+  - '208.80.154.157' # chromium
   - '208.80.153.254' # codfw lvs

-- 
To view, visit https://gerrit.wikimedia.org/r/404674
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Id585628b9f37e591a1d013ba0391f8853ec44f0f
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Muehlenhoff 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Revert "eqiad: temporarily remove chromium from LVS nameserv...

[Ema (Code Review)]

Hello Muehlenhoff, jenkins-bot,

I'd like you to do a code review.  Please visit

https://gerrit.wikimedia.org/r/404674

to review the following change.


Change subject: Revert "eqiad: temporarily remove chromium from LVS nameservers"
..

Revert "eqiad: temporarily remove chromium from LVS nameservers"

The host is back online.

This reverts commit 676bde44b8ffc5337bb1f57dd46de57a742da779.

Change-Id: Id585628b9f37e591a1d013ba0391f8853ec44f0f
---
M hieradata/role/eqiad/lvs/balancer.yaml
1 file changed, 1 insertion(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/74/404674/1

diff --git a/hieradata/role/eqiad/lvs/balancer.yaml 
b/hieradata/role/eqiad/lvs/balancer.yaml
index 888e756..b14b9ba 100644
--- a/hieradata/role/eqiad/lvs/balancer.yaml
+++ b/hieradata/role/eqiad/lvs/balancer.yaml
@@ -14,4 +14,5 @@
 # (doing this for all lvs for now, see T103921)
 profile::base::nameservers:
   - '208.80.154.50' # hydrogen
+  - '208.80.154.157' # chromium
   - '208.80.153.254' # codfw lvs

-- 
To view, visit https://gerrit.wikimedia.org/r/404674
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Id585628b9f37e591a1d013ba0391f8853ec44f0f
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: Muehlenhoff 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: eqiad: temporarily remove chromium from LVS nameservers

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/404672 )

Change subject: eqiad: temporarily remove chromium from LVS nameservers
..


eqiad: temporarily remove chromium from LVS nameservers

In preparation to the reboot of chromium, remove it from the list of
recursive DNS servers used directly by eqiad LVSs.

Change-Id: I7bb975ee350587637c377e9f1b04170e3700fec1
---
M hieradata/role/eqiad/lvs/balancer.yaml
1 file changed, 0 insertions(+), 1 deletion(-)

Approvals:
  Muehlenhoff: Looks good to me, but someone else must approve
  Ema: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/role/eqiad/lvs/balancer.yaml 
b/hieradata/role/eqiad/lvs/balancer.yaml
index b14b9ba..888e756 100644
--- a/hieradata/role/eqiad/lvs/balancer.yaml
+++ b/hieradata/role/eqiad/lvs/balancer.yaml
@@ -14,5 +14,4 @@
 # (doing this for all lvs for now, see T103921)
 profile::base::nameservers:
   - '208.80.154.50' # hydrogen
-  - '208.80.154.157' # chromium
   - '208.80.153.254' # codfw lvs

-- 
To view, visit https://gerrit.wikimedia.org/r/404672
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I7bb975ee350587637c377e9f1b04170e3700fec1
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Muehlenhoff 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: eqiad: temporarily remove chromium from LVS nameservers

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/404672 )

Change subject: eqiad: temporarily remove chromium from LVS nameservers
..

eqiad: temporarily remove chromium from LVS nameservers

In preparation to the reboot of chromium, remove it from the list of
recursive DNS servers used directly by eqiad LVSs.

Change-Id: I7bb975ee350587637c377e9f1b04170e3700fec1
---
M hieradata/role/eqiad/lvs/balancer.yaml
1 file changed, 0 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/72/404672/1

diff --git a/hieradata/role/eqiad/lvs/balancer.yaml 
b/hieradata/role/eqiad/lvs/balancer.yaml
index b14b9ba..888e756 100644
--- a/hieradata/role/eqiad/lvs/balancer.yaml
+++ b/hieradata/role/eqiad/lvs/balancer.yaml
@@ -14,5 +14,4 @@
 # (doing this for all lvs for now, see T103921)
 profile::base::nameservers:
   - '208.80.154.50' # hydrogen
-  - '208.80.154.157' # chromium
   - '208.80.153.254' # codfw lvs

-- 
To view, visit https://gerrit.wikimedia.org/r/404672
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I7bb975ee350587637c377e9f1b04170e3700fec1
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Revert "Remove hydrogen from LVS name server config"

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/404649 )

Change subject: Revert "Remove hydrogen from LVS name server config"
..


Revert "Remove hydrogen from LVS name server config"

Hydrogen rebooted properly.

This reverts commit 90981077386e713b33b5f1bc1f1614fd6079900f.

Change-Id: I15c1c248b5dd28f7691ac368f1258c04e5dea1cb
---
M hieradata/role/eqiad/lvs/balancer.yaml
1 file changed, 1 insertion(+), 0 deletions(-)

Approvals:
  Muehlenhoff: Looks good to me, but someone else must approve
  Ema: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/role/eqiad/lvs/balancer.yaml 
b/hieradata/role/eqiad/lvs/balancer.yaml
index 1b0bcb2..b14b9ba 100644
--- a/hieradata/role/eqiad/lvs/balancer.yaml
+++ b/hieradata/role/eqiad/lvs/balancer.yaml
@@ -13,5 +13,6 @@
 #   (chromium and hydrogen) with fallback to codfw
 # (doing this for all lvs for now, see T103921)
 profile::base::nameservers:
+  - '208.80.154.50' # hydrogen
   - '208.80.154.157' # chromium
   - '208.80.153.254' # codfw lvs

-- 
To view, visit https://gerrit.wikimedia.org/r/404649
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I15c1c248b5dd28f7691ac368f1258c04e5dea1cb
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Muehlenhoff 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Revert "Remove hydrogen from LVS name server config"

[Ema (Code Review)]

Hello Muehlenhoff, jenkins-bot,

I'd like you to do a code review.  Please visit

https://gerrit.wikimedia.org/r/404649

to review the following change.


Change subject: Revert "Remove hydrogen from LVS name server config"
..

Revert "Remove hydrogen from LVS name server config"

Hydrogen rebooted properly.

This reverts commit 90981077386e713b33b5f1bc1f1614fd6079900f.

Change-Id: I15c1c248b5dd28f7691ac368f1258c04e5dea1cb
---
M hieradata/role/eqiad/lvs/balancer.yaml
1 file changed, 1 insertion(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/49/404649/1

diff --git a/hieradata/role/eqiad/lvs/balancer.yaml 
b/hieradata/role/eqiad/lvs/balancer.yaml
index 1b0bcb2..b14b9ba 100644
--- a/hieradata/role/eqiad/lvs/balancer.yaml
+++ b/hieradata/role/eqiad/lvs/balancer.yaml
@@ -13,5 +13,6 @@
 #   (chromium and hydrogen) with fallback to codfw
 # (doing this for all lvs for now, see T103921)
 profile::base::nameservers:
+  - '208.80.154.50' # hydrogen
   - '208.80.154.157' # chromium
   - '208.80.153.254' # codfw lvs

-- 
To view, visit https://gerrit.wikimedia.org/r/404649
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I15c1c248b5dd28f7691ac368f1258c04e5dea1cb
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: Muehlenhoff 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Revert "vcl: remove X-CP-Full-Cipher"

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/404426 )

Change subject: Revert "vcl: remove X-CP-Full-Cipher"
..


Revert "vcl: remove X-CP-Full-Cipher"

The commit messed up stats.

This reverts commit 75a508b3197d4bf8da047b661ee25d3151a1e6ac.

Change-Id: I121990115fafafe875a21358f781ea7abcdd3353
---
M modules/varnish/files/tests/upload/16-x-connection-properties.vtc
M modules/varnish/files/varnishmtail
M modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
3 files changed, 11 insertions(+), 2 deletions(-)

Approvals:
  Ema: Verified; Looks good to me, approved



diff --git a/modules/varnish/files/tests/upload/16-x-connection-properties.vtc 
b/modules/varnish/files/tests/upload/16-x-connection-properties.vtc
index 0dfc2eb..e100dec 100644
--- a/modules/varnish/files/tests/upload/16-x-connection-properties.vtc
+++ b/modules/varnish/files/tests/upload/16-x-connection-properties.vtc
@@ -10,6 +10,7 @@
 expect req.http.X-CP-Key-Exchange == "prime256v1"
 expect req.http.X-CP-Auth == "ECDSA"
 expect req.http.X-CP-Cipher == "AES256-GCM-SHA384"
+expect req.http.X-CP-Full-Cipher == "ECDHE-ECDSA-AES256-GCM-SHA384"
 
 txresp
 
@@ -22,6 +23,7 @@
 expect req.http.X-CP-Key-Exchange == "prime256v1"
 expect req.http.X-CP-Auth == "ECDSA"
 expect req.http.X-CP-Cipher == "AES128-SHA"
+expect req.http.X-CP-Full-Cipher == "ECDHE-ECDSA-AES128-SHA"
 
 txresp
 
@@ -34,6 +36,7 @@
 expect req.http.X-CP-Key-Exchange == "X25519"
 expect req.http.X-CP-Auth == "ECDSA"
 expect req.http.X-CP-Cipher == "AES256-GCM-SHA384"
+expect req.http.X-CP-Full-Cipher == "ECDHE-ECDSA-AES256-GCM-SHA384"
 
 txresp
 
@@ -46,6 +49,7 @@
 expect req.http.X-CP-Key-Exchange == "RSA"
 expect req.http.X-CP-Auth == "RSA"
 expect req.http.X-CP-Cipher == "AES128-SHA"
+expect req.http.X-CP-Full-Cipher == "AES128-SHA"
 
 txresp
 
diff --git a/modules/varnish/files/varnishmtail 
b/modules/varnish/files/varnishmtail
index 6e08dac..5ffe9cd 100644
--- a/modules/varnish/files/varnishmtail
+++ b/modules/varnish/files/varnishmtail
@@ -16,7 +16,8 @@
 fmt_key_exchange='key_exchange %{VCL_Log:CP-Key-Exchange}x'
 fmt_auth='auth %{VCL_Log:CP-Auth}x'
 fmt_cipher='cipher %{VCL_Log:CP-Cipher}x'
+fmt_full_cipher='full_cipher %{VCL_Log:CP-Full-Cipher}x'
 
-FMT="${fmt_url}\t${fmt_cache_status}\t${fmt_http_status}\t${fmt_http_method}\t${fmt_cache_control}\t${fmt_inm}\t${fmt_h2}\t${fmt_tls_version}\t${fmt_session_reused}\t${fmt_key_exchange}\t${fmt_auth}\t${fmt_cipher}\t"
+FMT="${fmt_url}\t${fmt_cache_status}\t${fmt_http_status}\t${fmt_http_method}\t${fmt_cache_control}\t${fmt_inm}\t${fmt_h2}\t${fmt_tls_version}\t${fmt_session_reused}\t${fmt_key_exchange}\t${fmt_auth}\t${fmt_cipher}\t${fmt_full_cipher}\t"
 
 /usr/bin/varnishncsa -n frontend -F "${FMT}" | mtail -progs "${PROGS}" -logfds 0
diff --git a/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb 
b/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
index 008c9b0..65950d7 100644
--- a/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
+++ b/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
@@ -290,7 +290,9 @@
 
set req.http.X-CP-Key-Exchange = 
regsub(req.http.X-Connection-Properties, ".* EC=([A-Za-z0-9]+);.*", "\1");
 
-   set req.http.X-CP-Auth = regsub(req.http.X-Connection-Properties, ".* 
C=([A-Z0-9-]+);.*", "\1");
+   set req.http.X-CP-Full-Cipher = 
regsub(req.http.X-Connection-Properties, ".* C=([A-Z0-9-]+);.*", "\1");
+
+   set req.http.X-CP-Auth = req.http.X-CP-Full-Cipher;
 
if (req.http.X-CP-Auth ~ "^(ECDHE-(ECDSA|RSA)|DHE-RSA|TLS13)-") {
set req.http.X-CP-Cipher = regsub(req.http.X-CP-Auth, 
"^(ECDHE-(ECDSA|RSA)|DHE-RSA|TLS13)-", "");
@@ -325,6 +327,7 @@
std.log("CP-Key-Exchange: " + req.http.X-CP-Key-Exchange);
std.log("CP-Auth: " + req.http.X-CP-Auth);
std.log("CP-Cipher: " + req.http.X-CP-Cipher);
+   std.log("CP-Full-Cipher: " + req.http.X-CP-Full-Cipher);
 
<%- if !@varnish_testing -%>
// Keep these in the test VCL version to ease testing
@@ -334,6 +337,7 @@
unset req.http.X-CP-Key-Exchange;
unset req.http.X-CP-Auth;
unset req.http.X-CP-Cipher;
+   unset req.http.X-CP-Full-Cipher;
<%- end -%>
 
// The idea here is to block our worst clients (in TLS terms: those

-- 
To view, visit https://gerrit.wikimedia.org/r/404426
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I121990115fafafe875a21358f781ea7abcdd3353
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Revert "vcl: remove X-CP-Full-Cipher"

[Ema (Code Review)]

Hello BBlack, jenkins-bot,

I'd like you to do a code review.  Please visit

https://gerrit.wikimedia.org/r/404426

to review the following change.


Change subject: Revert "vcl: remove X-CP-Full-Cipher"
..

Revert "vcl: remove X-CP-Full-Cipher"

The commit messed up stats.

This reverts commit 75a508b3197d4bf8da047b661ee25d3151a1e6ac.

Change-Id: I121990115fafafe875a21358f781ea7abcdd3353
---
M modules/varnish/files/tests/upload/16-x-connection-properties.vtc
M modules/varnish/files/varnishmtail
M modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
3 files changed, 11 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/26/404426/1

diff --git a/modules/varnish/files/tests/upload/16-x-connection-properties.vtc 
b/modules/varnish/files/tests/upload/16-x-connection-properties.vtc
index 0dfc2eb..e100dec 100644
--- a/modules/varnish/files/tests/upload/16-x-connection-properties.vtc
+++ b/modules/varnish/files/tests/upload/16-x-connection-properties.vtc
@@ -10,6 +10,7 @@
 expect req.http.X-CP-Key-Exchange == "prime256v1"
 expect req.http.X-CP-Auth == "ECDSA"
 expect req.http.X-CP-Cipher == "AES256-GCM-SHA384"
+expect req.http.X-CP-Full-Cipher == "ECDHE-ECDSA-AES256-GCM-SHA384"
 
 txresp
 
@@ -22,6 +23,7 @@
 expect req.http.X-CP-Key-Exchange == "prime256v1"
 expect req.http.X-CP-Auth == "ECDSA"
 expect req.http.X-CP-Cipher == "AES128-SHA"
+expect req.http.X-CP-Full-Cipher == "ECDHE-ECDSA-AES128-SHA"
 
 txresp
 
@@ -34,6 +36,7 @@
 expect req.http.X-CP-Key-Exchange == "X25519"
 expect req.http.X-CP-Auth == "ECDSA"
 expect req.http.X-CP-Cipher == "AES256-GCM-SHA384"
+expect req.http.X-CP-Full-Cipher == "ECDHE-ECDSA-AES256-GCM-SHA384"
 
 txresp
 
@@ -46,6 +49,7 @@
 expect req.http.X-CP-Key-Exchange == "RSA"
 expect req.http.X-CP-Auth == "RSA"
 expect req.http.X-CP-Cipher == "AES128-SHA"
+expect req.http.X-CP-Full-Cipher == "AES128-SHA"
 
 txresp
 
diff --git a/modules/varnish/files/varnishmtail 
b/modules/varnish/files/varnishmtail
index 6e08dac..5ffe9cd 100644
--- a/modules/varnish/files/varnishmtail
+++ b/modules/varnish/files/varnishmtail
@@ -16,7 +16,8 @@
 fmt_key_exchange='key_exchange %{VCL_Log:CP-Key-Exchange}x'
 fmt_auth='auth %{VCL_Log:CP-Auth}x'
 fmt_cipher='cipher %{VCL_Log:CP-Cipher}x'
+fmt_full_cipher='full_cipher %{VCL_Log:CP-Full-Cipher}x'
 
-FMT="${fmt_url}\t${fmt_cache_status}\t${fmt_http_status}\t${fmt_http_method}\t${fmt_cache_control}\t${fmt_inm}\t${fmt_h2}\t${fmt_tls_version}\t${fmt_session_reused}\t${fmt_key_exchange}\t${fmt_auth}\t${fmt_cipher}\t"
+FMT="${fmt_url}\t${fmt_cache_status}\t${fmt_http_status}\t${fmt_http_method}\t${fmt_cache_control}\t${fmt_inm}\t${fmt_h2}\t${fmt_tls_version}\t${fmt_session_reused}\t${fmt_key_exchange}\t${fmt_auth}\t${fmt_cipher}\t${fmt_full_cipher}\t"
 
 /usr/bin/varnishncsa -n frontend -F "${FMT}" | mtail -progs "${PROGS}" -logfds 0
diff --git a/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb 
b/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
index 008c9b0..65950d7 100644
--- a/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
+++ b/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
@@ -290,7 +290,9 @@
 
set req.http.X-CP-Key-Exchange = 
regsub(req.http.X-Connection-Properties, ".* EC=([A-Za-z0-9]+);.*", "\1");
 
-   set req.http.X-CP-Auth = regsub(req.http.X-Connection-Properties, ".* 
C=([A-Z0-9-]+);.*", "\1");
+   set req.http.X-CP-Full-Cipher = 
regsub(req.http.X-Connection-Properties, ".* C=([A-Z0-9-]+);.*", "\1");
+
+   set req.http.X-CP-Auth = req.http.X-CP-Full-Cipher;
 
if (req.http.X-CP-Auth ~ "^(ECDHE-(ECDSA|RSA)|DHE-RSA|TLS13)-") {
set req.http.X-CP-Cipher = regsub(req.http.X-CP-Auth, 
"^(ECDHE-(ECDSA|RSA)|DHE-RSA|TLS13)-", "");
@@ -325,6 +327,7 @@
std.log("CP-Key-Exchange: " + req.http.X-CP-Key-Exchange);
std.log("CP-Auth: " + req.http.X-CP-Auth);
std.log("CP-Cipher: " + req.http.X-CP-Cipher);
+   std.log("CP-Full-Cipher: " + req.http.X-CP-Full-Cipher);
 
<%- if !@varnish_testing -%>
// Keep these in the test VCL version to ease testing
@@ -334,6 +337,7 @@
unset req.http.X-CP-Key-Exchange;
unset req.http.X-CP-Auth;
unset req.http.X-CP-Cipher;
+   unset req.http.X-CP-Full-Cipher;
<%- end -%>
 
// The idea here is to block our worst clients (in TLS terms: those

-- 
To view, visit https://gerrit.wikimedia.org/r/404426
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I121990115fafafe875a21358f781ea7abcdd3353
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: jenkins-bot <>

[MediaWiki-commits] [Gerrit] operations/puppet[production]: vcl: remove X-CP-Full-Cipher

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398314 )

Change subject: vcl: remove X-CP-Full-Cipher
..


vcl: remove X-CP-Full-Cipher

Change-Id: I056fb1a07dfbe9dea43c832dae795937e480c3dd
---
M modules/varnish/files/tests/upload/16-x-connection-properties.vtc
M modules/varnish/files/varnishmtail
M modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
3 files changed, 2 insertions(+), 11 deletions(-)

Approvals:
  Ema: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/varnish/files/tests/upload/16-x-connection-properties.vtc 
b/modules/varnish/files/tests/upload/16-x-connection-properties.vtc
index e100dec..0dfc2eb 100644
--- a/modules/varnish/files/tests/upload/16-x-connection-properties.vtc
+++ b/modules/varnish/files/tests/upload/16-x-connection-properties.vtc
@@ -10,7 +10,6 @@
 expect req.http.X-CP-Key-Exchange == "prime256v1"
 expect req.http.X-CP-Auth == "ECDSA"
 expect req.http.X-CP-Cipher == "AES256-GCM-SHA384"
-expect req.http.X-CP-Full-Cipher == "ECDHE-ECDSA-AES256-GCM-SHA384"
 
 txresp
 
@@ -23,7 +22,6 @@
 expect req.http.X-CP-Key-Exchange == "prime256v1"
 expect req.http.X-CP-Auth == "ECDSA"
 expect req.http.X-CP-Cipher == "AES128-SHA"
-expect req.http.X-CP-Full-Cipher == "ECDHE-ECDSA-AES128-SHA"
 
 txresp
 
@@ -36,7 +34,6 @@
 expect req.http.X-CP-Key-Exchange == "X25519"
 expect req.http.X-CP-Auth == "ECDSA"
 expect req.http.X-CP-Cipher == "AES256-GCM-SHA384"
-expect req.http.X-CP-Full-Cipher == "ECDHE-ECDSA-AES256-GCM-SHA384"
 
 txresp
 
@@ -49,7 +46,6 @@
 expect req.http.X-CP-Key-Exchange == "RSA"
 expect req.http.X-CP-Auth == "RSA"
 expect req.http.X-CP-Cipher == "AES128-SHA"
-expect req.http.X-CP-Full-Cipher == "AES128-SHA"
 
 txresp
 
diff --git a/modules/varnish/files/varnishmtail 
b/modules/varnish/files/varnishmtail
index 5ffe9cd..6e08dac 100644
--- a/modules/varnish/files/varnishmtail
+++ b/modules/varnish/files/varnishmtail
@@ -16,8 +16,7 @@
 fmt_key_exchange='key_exchange %{VCL_Log:CP-Key-Exchange}x'
 fmt_auth='auth %{VCL_Log:CP-Auth}x'
 fmt_cipher='cipher %{VCL_Log:CP-Cipher}x'
-fmt_full_cipher='full_cipher %{VCL_Log:CP-Full-Cipher}x'
 
-FMT="${fmt_url}\t${fmt_cache_status}\t${fmt_http_status}\t${fmt_http_method}\t${fmt_cache_control}\t${fmt_inm}\t${fmt_h2}\t${fmt_tls_version}\t${fmt_session_reused}\t${fmt_key_exchange}\t${fmt_auth}\t${fmt_cipher}\t${fmt_full_cipher}\t"
+FMT="${fmt_url}\t${fmt_cache_status}\t${fmt_http_status}\t${fmt_http_method}\t${fmt_cache_control}\t${fmt_inm}\t${fmt_h2}\t${fmt_tls_version}\t${fmt_session_reused}\t${fmt_key_exchange}\t${fmt_auth}\t${fmt_cipher}\t"
 
 /usr/bin/varnishncsa -n frontend -F "${FMT}" | mtail -progs "${PROGS}" -logfds 0
diff --git a/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb 
b/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
index 65950d7..008c9b0 100644
--- a/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
+++ b/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
@@ -290,9 +290,7 @@
 
set req.http.X-CP-Key-Exchange = 
regsub(req.http.X-Connection-Properties, ".* EC=([A-Za-z0-9]+);.*", "\1");
 
-   set req.http.X-CP-Full-Cipher = 
regsub(req.http.X-Connection-Properties, ".* C=([A-Z0-9-]+);.*", "\1");
-
-   set req.http.X-CP-Auth = req.http.X-CP-Full-Cipher;
+   set req.http.X-CP-Auth = regsub(req.http.X-Connection-Properties, ".* 
C=([A-Z0-9-]+);.*", "\1");
 
if (req.http.X-CP-Auth ~ "^(ECDHE-(ECDSA|RSA)|DHE-RSA|TLS13)-") {
set req.http.X-CP-Cipher = regsub(req.http.X-CP-Auth, 
"^(ECDHE-(ECDSA|RSA)|DHE-RSA|TLS13)-", "");
@@ -327,7 +325,6 @@
std.log("CP-Key-Exchange: " + req.http.X-CP-Key-Exchange);
std.log("CP-Auth: " + req.http.X-CP-Auth);
std.log("CP-Cipher: " + req.http.X-CP-Cipher);
-   std.log("CP-Full-Cipher: " + req.http.X-CP-Full-Cipher);
 
<%- if !@varnish_testing -%>
// Keep these in the test VCL version to ease testing
@@ -337,7 +334,6 @@
unset req.http.X-CP-Key-Exchange;
unset req.http.X-CP-Auth;
unset req.http.X-CP-Cipher;
-   unset req.http.X-CP-Full-Cipher;
<%- end -%>
 
// The idea here is to block our worst clients (in TLS terms: those

-- 
To view, visit https://gerrit.wikimedia.org/r/398314
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I056fb1a07dfbe9dea43c832dae795937e480c3dd
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Fix varnishslowlog logstash configuration

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/404288 )

Change subject: Fix varnishslowlog logstash configuration
..


Fix varnishslowlog logstash configuration

Change-Id: Ie1cefcbe8f1e544702bb44fdd9bbe535ce39003d
---
M modules/role/files/logstash/filter-logback.conf
M modules/varnish/files/varnishslowlog.py
2 files changed, 3 insertions(+), 2 deletions(-)

Approvals:
  Ema: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/role/files/logstash/filter-logback.conf 
b/modules/role/files/logstash/filter-logback.conf
index 2f38919..6821adc 100644
--- a/modules/role/files/logstash/filter-logback.conf
+++ b/modules/role/files/logstash/filter-logback.conf
@@ -3,7 +3,7 @@
 filter {
   if [type] == "logback" {
 # General message cleanup
-if "thumbor" not in [tags] {
+if "thumbor" not in [tags] and "varnishslowlog" not in [tags] {
   mutate {
 replace => [ "host", "%{HOSTNAME}" ]
   }
diff --git a/modules/varnish/files/varnishslowlog.py 
b/modules/varnish/files/varnishslowlog.py
index 0883267..242dbae 100644
--- a/modules/varnish/files/varnishslowlog.py
+++ b/modules/varnish/files/varnishslowlog.py
@@ -87,7 +87,8 @@
 self.args.logstash_server[0],
 port=self.args.logstash_server[1],
 version=1,
-message_type='varnishslowlog'
+message_type='logback',
+tags=['varnishslowlog']
 )
 else:
 handler = logging.StreamHandler(sys.stdout)

-- 
To view, visit https://gerrit.wikimedia.org/r/404288
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ie1cefcbe8f1e544702bb44fdd9bbe535ce39003d
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gilles 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Volans 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnishslowlog: do not crash on empty respheader values

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/404282 )

Change subject: varnishslowlog: do not crash on empty respheader values
..


varnishslowlog: do not crash on empty respheader values

Similarly to 7a5c50f, RespHeader can also be empty. Do not crash if that
is the case.

Change-Id: I46a991b60ac6f4910a5fa85efeec362c1a4df06b
---
M modules/varnish/files/varnishslowlog.py
1 file changed, 8 insertions(+), 1 deletion(-)

Approvals:
  Ema: Looks good to me, approved
  jenkins-bot: Verified
  Gilles: Looks good to me, but someone else must approve



diff --git a/modules/varnish/files/varnishslowlog.py 
b/modules/varnish/files/varnishslowlog.py
index 4e5ad1d..0883267 100644
--- a/modules/varnish/files/varnishslowlog.py
+++ b/modules/varnish/files/varnishslowlog.py
@@ -151,7 +151,14 @@
 elif tag == 'RespHeader':
 splitagain = value.split(None, 1)
 header_name = splitagain[0][:-1]
-header_value = splitagain[1]
+
+if len(splitagain) == 2:
+header_value = splitagain[1]
+else:
+# Similarly to ReqHeader above, RespHeader can also occasionaly
+# have no associated value.
+header_value = ''
+
 self.tx['response-' + header_name] = header_value
 
 def main(self):

-- 
To view, visit https://gerrit.wikimedia.org/r/404282
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I46a991b60ac6f4910a5fa85efeec362c1a4df06b
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Gilles 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnishslowlog: do not crash on empty respheader values

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/404282 )

Change subject: varnishslowlog: do not crash on empty respheader values
..

varnishslowlog: do not crash on empty respheader values

Similarly to 7a5c50f, RespHeader can also be empty. Do not crash if that
is the case.

Change-Id: I46a991b60ac6f4910a5fa85efeec362c1a4df06b
---
M modules/varnish/files/varnishslowlog.py
1 file changed, 8 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/82/404282/1

diff --git a/modules/varnish/files/varnishslowlog.py 
b/modules/varnish/files/varnishslowlog.py
index 4e5ad1d..0883267 100644
--- a/modules/varnish/files/varnishslowlog.py
+++ b/modules/varnish/files/varnishslowlog.py
@@ -151,7 +151,14 @@
 elif tag == 'RespHeader':
 splitagain = value.split(None, 1)
 header_name = splitagain[0][:-1]
-header_value = splitagain[1]
+
+if len(splitagain) == 2:
+header_value = splitagain[1]
+else:
+# Similarly to ReqHeader above, RespHeader can also occasionaly
+# have no associated value.
+header_value = ''
+
 self.tx['response-' + header_name] = header_value
 
 def main(self):

-- 
To view, visit https://gerrit.wikimedia.org/r/404282
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I46a991b60ac6f4910a5fa85efeec362c1a4df06b
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnishslowlog: do not crash on empty reqheader values

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/404279 )

Change subject: varnishslowlog: do not crash on empty reqheader values
..


varnishslowlog: do not crash on empty reqheader values

ReqHeader can occasionaly have no associated value. Avoid crashing and
set value to empty string if that is the case.

Change-Id: Iae09345373ff7ad6119c9c87fe7cc4a95e2896db
---
M modules/varnish/files/varnishslowlog.py
1 file changed, 8 insertions(+), 1 deletion(-)

Approvals:
  Ema: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/varnish/files/varnishslowlog.py 
b/modules/varnish/files/varnishslowlog.py
index 349e347..4e5ad1d 100644
--- a/modules/varnish/files/varnishslowlog.py
+++ b/modules/varnish/files/varnishslowlog.py
@@ -139,7 +139,14 @@
 elif tag == 'ReqHeader':
 splitagain = value.split(None, 1)
 header_name = splitagain[0][:-1]
-header_value = splitagain[1]
+
+if len(splitagain) == 2:
+header_value = splitagain[1]
+else:
+# ReqHeader can occasionaly have no associated value. Set value
+# to empty string if that is the case.
+header_value = ''
+
 self.tx['request-' + header_name] = header_value
 elif tag == 'RespHeader':
 splitagain = value.split(None, 1)

-- 
To view, visit https://gerrit.wikimedia.org/r/404279
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Iae09345373ff7ad6119c9c87fe7cc4a95e2896db
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnishslowlog: do not crash on empty reqheader values

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/404279 )

Change subject: varnishslowlog: do not crash on empty reqheader values
..

varnishslowlog: do not crash on empty reqheader values

ReqHeader can occasionaly have no associated value. Avoid crashing and
set value to empty string if that is the case.

Change-Id: Iae09345373ff7ad6119c9c87fe7cc4a95e2896db
---
M modules/varnish/files/varnishslowlog.py
1 file changed, 8 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/79/404279/1

diff --git a/modules/varnish/files/varnishslowlog.py 
b/modules/varnish/files/varnishslowlog.py
index 349e347..4e5ad1d 100644
--- a/modules/varnish/files/varnishslowlog.py
+++ b/modules/varnish/files/varnishslowlog.py
@@ -139,7 +139,14 @@
 elif tag == 'ReqHeader':
 splitagain = value.split(None, 1)
 header_name = splitagain[0][:-1]
-header_value = splitagain[1]
+
+if len(splitagain) == 2:
+header_value = splitagain[1]
+else:
+# ReqHeader can occasionaly have no associated value. Set value
+# to empty string if that is the case.
+header_value = ''
+
 self.tx['request-' + header_name] = header_value
 elif tag == 'RespHeader':
 splitagain = value.split(None, 1)

-- 
To view, visit https://gerrit.wikimedia.org/r/404279
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Iae09345373ff7ad6119c9c87fe7cc4a95e2896db
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Smarter Varnish slow log

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/399176 )

Change subject: Smarter Varnish slow log
..


Smarter Varnish slow log

Using python-logstash allows us to send structured data
to ELK. Additionally, we can leverage varnishlog, which
has more options than varnishncsa.

Bug: T181315
Change-Id: I08851a84857783cfacc75768a3c0216633aa9242
---
M modules/profile/manifests/cache/base.pp
A modules/varnish/files/varnishslowlog.py
M modules/varnish/manifests/common.pp
M modules/varnish/manifests/instance.pp
M modules/varnish/manifests/logging.pp
D modules/varnish/templates/initscripts/varnish-slowreqs.systemd.erb
A modules/varnish/templates/initscripts/varnishslowlog.systemd.erb
7 files changed, 193 insertions(+), 25 deletions(-)

Approvals:
  Ema: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/profile/manifests/cache/base.pp 
b/modules/profile/manifests/cache/base.pp
index 6fc84a6..179a534 100644
--- a/modules/profile/manifests/cache/base.pp
+++ b/modules/profile/manifests/cache/base.pp
@@ -22,6 +22,7 @@
 $be_runtime_params = hiera('profile::cache::base::be_runtime_params', []),
 $logstash_host = hiera('logstash_host', undef),
 $logstash_syslog_port = hiera('logstash_syslog_port', undef),
+$logstash_json_lines_port = hiera('logstash_json_lines_port', undef),
 $log_slow_request_threshold = 
hiera('profile::cache::base::log_slow_request_threshold', '60.0'),
 $allow_iptables = hiera('profile::cache::base::allow_iptables', false),
 ) {
@@ -78,6 +79,8 @@
 fe_runtime_params  => $fe_runtime_params,
 be_runtime_params  => $be_runtime_params,
 log_slow_request_threshold => $log_slow_request_threshold,
+logstash_host  => $logstash_host,
+logstash_json_lines_port   => $logstash_json_lines_port,
 }
 
 class { [
diff --git a/modules/varnish/files/varnishslowlog.py 
b/modules/varnish/files/varnishslowlog.py
new file mode 100644
index 000..349e347
--- /dev/null
+++ b/modules/varnish/files/varnishslowlog.py
@@ -0,0 +1,164 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""
+  VarnishSlowLog
+  ~~
+  VarnishSlowLog is responsible for gathering slow requests from varnishlog
+  and sending them to logstash.
+
+  Copyright 2016-2017 Emanuele Rocca 
+  Copyright 2017 Gilles Dubuc 
+
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+
+"""
+
+import argparse
+import logging
+import logstash
+import os
+import sys
+import urlparse
+
+from subprocess import PIPE, Popen
+
+
+def parse_logstash_server_string(server_string):
+"""Convert logstash server string into (hostname, port) tuple."""
+parsed = urlparse.urlparse('//' + server_string)
+return parsed.hostname, parsed.port or 12202
+
+
+class VarnishSlowLog(object):
+description = 'Varnish slow log logstash logger'
+
+def __init__(self, argument_list):
+"""Parse CLI arguments.
+
+argument_list is a list such as ['--foo', 'FOO', '--bar', 'BAR']"""
+ap = argparse.ArgumentParser(
+description=self.description,
+epilog='If no logstash server is specified, '
+   'prints log entries to stdout instead.')
+
+ap.add_argument('--logstash-server', help='logstash server',
+type=parse_logstash_server_string, default=None)
+
+ap.add_argument('--slow-threshold', help='varnish fetch duration 
threshold',
+type=float, default=10.0)
+
+ap.add_argument('--transaction-timeout', help='varnish transaction 
timeout',
+type=int, default=600)
+
+ap.add_argument('--varnishd-instance-name', help='varnishd instance 
name',
+default=None)
+
+ap.add_argument('--varnishlog-path', help='varnishlog full path',
+default='/usr/bin/varnishlog')
+
+self.args = ap.parse_args(argument_list)
+
+self.cmd = [
+self.args.varnishlog_path,
+# VSL query matching anything but purges
+'-q', 'ReqMethod ne "PURGE" and Timestamp:Fetch[3] > %f' % 
self.args.slow_threshold,
+# Set maximum Varnish transaction duration to track
+'-T', '%d' % self.args.transaction_timeout
+]
+
+self.layer = 'backend'
+
+if 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: vcl: add hash function name to CHACHA20-POLY1305 cipher

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398311 )

Change subject: vcl: add hash function name to CHACHA20-POLY1305 cipher
..


vcl: add hash function name to CHACHA20-POLY1305 cipher

The hash function used by all ciphersuites described in rfc7905 is
SHA-256. Starting with TLSv1.3, CHACHA20-POLY1305 will be renamed into
CHACHA20-POLY1305-SHA256. Do the renaming now in our VCL to avoid stats
getting skewed later on.

Ref: https://tools.ietf.org/html/rfc7905#section-2

Change-Id: I9dec5f879c1b53be2232da83bbbf76170b49a18c
---
M modules/mtail/files/test/logs/varnish.test
M modules/mtail/files/test/varnish_test.py
M modules/varnish/files/tests/upload/16-x-connection-properties.vtc
M modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
4 files changed, 22 insertions(+), 2 deletions(-)

Approvals:
  Ema: Verified; Looks good to me, approved



diff --git a/modules/mtail/files/test/logs/varnish.test 
b/modules/mtail/files/test/logs/varnish.test
index e5155da..205e671 100644
--- a/modules/mtail/files/test/logs/varnish.test
+++ b/modules/mtail/files/test/logs/varnish.test
@@ -1,5 +1,5 @@
 url /  cache_status int-front  http_status 301 http_method GET cache_control - 
inm -   h2 0tls_version session_reused 0key_exchangeauth
cipher  full_cipher 
-url /w/index.php   cache_status hit-front  http_status 304 http_method GET 
cache_control private, s-maxage=0, max-age=0, must-revalidate   inm -   h2 1
tls_version TLSv1.2 session_reused 1key_exchange X25519 auth 
ECDSA  cipher CHACHA20-POLY1305full_cipher 
ECDHE-ECDSA-CHACHA20-POLY1305   
+url /w/index.php   cache_status hit-front  http_status 304 http_method GET 
cache_control private, s-maxage=0, max-age=0, must-revalidate   inm -   h2 1
tls_version TLSv1.2 session_reused 1key_exchange X25519 auth 
ECDSA  cipher CHACHA20-POLY1305-SHA256 full_cipher 
ECDHE-ECDSA-CHACHA20-POLY1305-SHA256
 url 
/api/rest_v1/metrics/pageviews/per-article/fr.wikipedia.org/all-access/user/S%C3%A9lection_sexuelle/daily/20171126/20171201
cache_status hit-front  http_status 200 http_method HEAD
cache_control s-maxage=86400, max-age=86400 inm -
 url /w/load.phpcache_status hit-front  http_status 200 http_method GET 
cache_control public, max-age=2592000, s-maxage=2592000 inm -
 url /w/load.phpcache_status hit-front  http_status 200 http_method 
HEADcache_control public, max-age=2592000, s-maxage=2592000 inm -
diff --git a/modules/mtail/files/test/varnish_test.py 
b/modules/mtail/files/test/varnish_test.py
index 0abe36e..bc584aa 100644
--- a/modules/mtail/files/test/varnish_test.py
+++ b/modules/mtail/files/test/varnish_test.py
@@ -60,7 +60,7 @@
 'version=TLSv1.2',
 'key_exchange=X25519',
 'auth=ECDSA',
-'cipher=CHACHA20-POLY1305',
+'cipher=CHACHA20-POLY1305-SHA256',
 ]
 for value in expected:
 self.assertIn(value, labels)
diff --git a/modules/varnish/files/tests/upload/16-x-connection-properties.vtc 
b/modules/varnish/files/tests/upload/16-x-connection-properties.vtc
index eaa4037..e100dec 100644
--- a/modules/varnish/files/tests/upload/16-x-connection-properties.vtc
+++ b/modules/varnish/files/tests/upload/16-x-connection-properties.vtc
@@ -52,6 +52,18 @@
 expect req.http.X-CP-Full-Cipher == "AES128-SHA"
 
 txresp
+
+rxreq
+expect req.url == "/5"
+
+expect req.http.X-CP-HTTP2 == "1"
+expect req.http.X-CP-TLS-Version == "TLSv1.2"
+expect req.http.X-CP-TLS-Session-Reused == 0
+expect req.http.X-CP-Key-Exchange == "X25519"
+expect req.http.X-CP-Auth == "ECDSA"
+expect req.http.X-CP-Cipher == "CHACHA20-POLY1305-SHA256"
+
+txresp
 } -start
 
 varnish v1 -arg "-p vcc_err_unref=false -p vcc_allow_inline_c=true" 
-vcl+backend {
@@ -74,4 +86,7 @@
 
 txreq -url "/4" -hdr "X-Forwarded-Proto: https" -hdr "Host: 
upload.wikimedia.org" -hdr "X-Connection-Properties: H2=0; SSR=1; SSL=TLSv1; 
C=AES128-SHA; EC=UNDEF;"
 rxresp
+
+txreq -url "/5" -hdr "X-Forwarded-Proto: https" -hdr "Host: 
upload.wikimedia.org" -hdr "X-Connection-Properties: H2=1; SSR=0; SSL=TLSv1.2; 
C=ECDHE-ECDSA-CHACHA20-POLY1305; EC=X25519;"
+rxresp
 } -run
diff --git a/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb 
b/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
index e8c0153..65950d7 100644
--- a/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
+++ b/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
@@ -314,6 +314,11 @@
set req.http.X-CP-Key-Exchange = "RSA";
}
 
+   // Starting with TLSv1.3, CHACHA20-POLY1305 will be renamed into
+   // CHACHA20-POLY1305-SHA256. Do the renaming now in VCL to avoid stats 
skew
+   // later on.
+   set req.http.X-CP-Cipher = 

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Use pooled-and-up servers in can-depool logic

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/403677 )

Change subject: Use pooled-and-up servers in can-depool logic
..

Use pooled-and-up servers in can-depool logic

The number of pooled servers should not be computed as `total-servers -
servers-down`. Pooled servers are hosts which are both enabled
administratively and up according to pybal. Change can-depool logic
accordingly.

Bug: T184715
Change-Id: Ic83382938810331dd4292f45cbb85f6aaa7b7707
---
M pybal/coordinator.py
1 file changed, 3 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/77/403677/1

diff --git a/pybal/coordinator.py b/pybal/coordinator.py
index a526b5b..2fb64e7 100755
--- a/pybal/coordinator.py
+++ b/pybal/coordinator.py
@@ -442,11 +442,11 @@
 def canDepool(self):
 """Returns a boolean denoting whether another server can be depooled"""
 
-# Construct a list of servers that have status 'down'
-downServers = [server for server in self.servers.itervalues() if not 
server.up]
+# Construct a list of pooled servers
+pooledServers = [server for server in self.servers.itervalues() if 
server.up and server.pooled]
 
 # The total amount of pooled servers may never drop below a configured 
threshold
-return len(self.servers) - len(downServers) >= len(self.servers) * 
self.lvsservice.getDepoolThreshold()
+return len(pooledServers) >= len(self.servers) * 
self.lvsservice.getDepoolThreshold()
 
 def onConfigUpdate(self, config):
 """Parses the server list and changes the state accordingly."""

-- 
To view, visit https://gerrit.wikimedia.org/r/403677
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic83382938810331dd4292f45cbb85f6aaa7b7707
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[1.14]: Alerts instrumentation: return instance of bytes

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/403671 )

Change subject: Alerts instrumentation: return instance of bytes
..


Alerts instrumentation: return instance of bytes

Twisted returns a 500 if the returned body is not an instance of
'bytes'. 'unicode' values are not. Make sure we return a value of type
'str' instead.

Bug: T184721
Ref: 
https://github.com/twisted/twisted/blob/8857cbf4ec6a88b6cfb758ccaa9161d6b2f48009/src/twisted/web/server.py#L314
Change-Id: I23aa6f24ed729487621fe61625d86600d8c25b03
(cherry picked from commit 3505626c9d1590e84525560edbc2f5bf01a7be99)
---
M pybal/instrumentation.py
1 file changed, 4 insertions(+), 1 deletion(-)

Approvals:
  Ema: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/pybal/instrumentation.py b/pybal/instrumentation.py
index 386b174..64437e8 100644
--- a/pybal/instrumentation.py
+++ b/pybal/instrumentation.py
@@ -90,7 +90,10 @@
 if wantJson(request):
 return json.dumps(resp)
 else:
-return "%s - %s" % (resp['status'].upper(), resp['msg'])
+# Twisted returns a 500 if the returned body is not an instance of
+# 'bytes'. 'unicode' values are not. Make sure we return a 'str'
+# instead. See https://phabricator.wikimedia.org/T184721
+return str("%s - %s" % (resp['status'].upper(), resp['msg']))
 
 class PoolsRoot(Resource):
 """Pools base resource.

-- 
To view, visit https://gerrit.wikimedia.org/r/403671
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I23aa6f24ed729487621fe61625d86600d8c25b03
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: 1.14
Gerrit-Owner: Ema 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[1.14]: Alerts instrumentation: return instance of bytes

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/403671 )

Change subject: Alerts instrumentation: return instance of bytes
..

Alerts instrumentation: return instance of bytes

Twisted returns a 500 if the returned body is not an instance of
'bytes'. 'unicode' values are not. Make sure we return a value of type
'str' instead.

Bug: T184721
Ref: 
https://github.com/twisted/twisted/blob/8857cbf4ec6a88b6cfb758ccaa9161d6b2f48009/src/twisted/web/server.py#L314
Change-Id: I23aa6f24ed729487621fe61625d86600d8c25b03
(cherry picked from commit 3505626c9d1590e84525560edbc2f5bf01a7be99)
---
M pybal/instrumentation.py
1 file changed, 4 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/71/403671/1

diff --git a/pybal/instrumentation.py b/pybal/instrumentation.py
index 386b174..64437e8 100644
--- a/pybal/instrumentation.py
+++ b/pybal/instrumentation.py
@@ -90,7 +90,10 @@
 if wantJson(request):
 return json.dumps(resp)
 else:
-return "%s - %s" % (resp['status'].upper(), resp['msg'])
+# Twisted returns a 500 if the returned body is not an instance of
+# 'bytes'. 'unicode' values are not. Make sure we return a 'str'
+# instead. See https://phabricator.wikimedia.org/T184721
+return str("%s - %s" % (resp['status'].upper(), resp['msg']))
 
 class PoolsRoot(Resource):
 """Pools base resource.

-- 
To view, visit https://gerrit.wikimedia.org/r/403671
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I23aa6f24ed729487621fe61625d86600d8c25b03
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: 1.14
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Alerts instrumentation: return instance of bytes

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/403664 )

Change subject: Alerts instrumentation: return instance of bytes
..


Alerts instrumentation: return instance of bytes

Twisted returns a 500 if the returned body is not an instance of
'bytes'. 'unicode' values are not. Make sure we return a value of type
'str' instead.

Bug: T184721
Ref: 
https://github.com/twisted/twisted/blob/8857cbf4ec6a88b6cfb758ccaa9161d6b2f48009/src/twisted/web/server.py#L314
Change-Id: I23aa6f24ed729487621fe61625d86600d8c25b03
---
M pybal/instrumentation.py
1 file changed, 4 insertions(+), 1 deletion(-)

Approvals:
  Giuseppe Lavagetto: Looks good to me, but someone else must approve
  Ema: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/pybal/instrumentation.py b/pybal/instrumentation.py
index 386b174..64437e8 100644
--- a/pybal/instrumentation.py
+++ b/pybal/instrumentation.py
@@ -90,7 +90,10 @@
 if wantJson(request):
 return json.dumps(resp)
 else:
-return "%s - %s" % (resp['status'].upper(), resp['msg'])
+# Twisted returns a 500 if the returned body is not an instance of
+# 'bytes'. 'unicode' values are not. Make sure we return a 'str'
+# instead. See https://phabricator.wikimedia.org/T184721
+return str("%s - %s" % (resp['status'].upper(), resp['msg']))
 
 class PoolsRoot(Resource):
 """Pools base resource.

-- 
To view, visit https://gerrit.wikimedia.org/r/403664
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I23aa6f24ed729487621fe61625d86600d8c25b03
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Ema 
Gerrit-Reviewer: Alexandros Kosiaris 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Volans 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...pybal[master]: Alerts instrumentation: return instance of bytes

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/403664 )

Change subject: Alerts instrumentation: return instance of bytes
..

Alerts instrumentation: return instance of bytes

Twisted returns a 500 if the returned body is not an instance of
'bytes'. 'unicode' values are not. Make sure we return a value of type
'str' instead.

Bug: T184721
Ref: 
https://github.com/twisted/twisted/blob/8857cbf4ec6a88b6cfb758ccaa9161d6b2f48009/src/twisted/web/server.py#L314
Change-Id: I23aa6f24ed729487621fe61625d86600d8c25b03
---
M pybal/instrumentation.py
1 file changed, 4 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/pybal 
refs/changes/64/403664/1

diff --git a/pybal/instrumentation.py b/pybal/instrumentation.py
index 386b174..64437e8 100644
--- a/pybal/instrumentation.py
+++ b/pybal/instrumentation.py
@@ -90,7 +90,10 @@
 if wantJson(request):
 return json.dumps(resp)
 else:
-return "%s - %s" % (resp['status'].upper(), resp['msg'])
+# Twisted returns a 500 if the returned body is not an instance of
+# 'bytes'. 'unicode' values are not. Make sure we return a 'str'
+# instead. See https://phabricator.wikimedia.org/T184721
+return str("%s - %s" % (resp['status'].upper(), resp['msg']))
 
 class PoolsRoot(Resource):
 """Pools base resource.

-- 
To view, visit https://gerrit.wikimedia.org/r/403664
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I23aa6f24ed729487621fe61625d86600d8c25b03
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/pybal
Gerrit-Branch: master
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: pybaltest: accept RAs even if forwarding is enabled

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/403380 )

Change subject: pybaltest: accept RAs even if forwarding is enabled
..


pybaltest: accept RAs even if forwarding is enabled

With IP forwarding enabled, the Linux kernel ignores all Router
Advertisements, breaking IPv6 SLAAC. Accept Router Advertisements even
if forwarding is enabled.

Change-Id: I98821528bbb123d11f736206758b05dcdf508628
---
M modules/role/manifests/pybaltest.pp
1 file changed, 9 insertions(+), 0 deletions(-)

Approvals:
  Ema: Looks good to me, approved
  Alexandros Kosiaris: Looks good to me, but someone else must approve
  jenkins-bot: Verified



diff --git a/modules/role/manifests/pybaltest.pp 
b/modules/role/manifests/pybaltest.pp
index ea0ae26..f33220e 100644
--- a/modules/role/manifests/pybaltest.pp
+++ b/modules/role/manifests/pybaltest.pp
@@ -18,6 +18,15 @@
 srange => "(@resolve((${pybaltest_hosts_ferm})) 
@resolve((${pybaltest_hosts_ferm}), ))",
 }
 
+# If the host considers itself as a router (IP forwarding enabled), it will
+# ignore all router advertisements, breaking IPv6 SLAAC. Accept Router
+# Advertisements even if forwarding is enabled.
+sysctl::parameters { 'accept-ra':
+values => {
+"net.ipv6.conf.${facts['interface_primary']}.accept_ra" => 2,
+},
+}
+
 # Install conftool-master for conftool testing
 class  { '::puppetmaster::base_repo':
 gitdir   => '/var/lib/git',

-- 
To view, visit https://gerrit.wikimedia.org/r/403380
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I98821528bbb123d11f736206758b05dcdf508628
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: Alexandros Kosiaris 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Muehlenhoff 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: pybaltest: accept RAs even if forwarding is enabled

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/403380 )

Change subject: pybaltest: accept RAs even if forwarding is enabled
..

pybaltest: accept RAs even if forwarding is enabled

With IP forwarding enabled, the Linux kernel ignores all Router
Advertisements, breaking IPv6 SLAAC. Accept Router Advertisements even
if forwarding is enabled.

Change-Id: I98821528bbb123d11f736206758b05dcdf508628
---
M modules/role/manifests/pybaltest.pp
1 file changed, 9 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/80/403380/1

diff --git a/modules/role/manifests/pybaltest.pp 
b/modules/role/manifests/pybaltest.pp
index ea0ae26..f33220e 100644
--- a/modules/role/manifests/pybaltest.pp
+++ b/modules/role/manifests/pybaltest.pp
@@ -18,6 +18,15 @@
 srange => "(@resolve((${pybaltest_hosts_ferm})) 
@resolve((${pybaltest_hosts_ferm}), ))",
 }
 
+# If the host considers itself as a router (IP forwarding enabled), it will
+# ignore all router advertisements, breaking IPv6 SLAAC. Accept Router
+# Advertisements even if forwarding is enabled.
+sysctl::parameters { 'accept-ra':
+values => {
+"net.ipv6.conf.${facts['interface_primary']}.accept_ra" => 2,
+},
+}
+
 # Install conftool-master for conftool testing
 class  { '::puppetmaster::base_repo':
 gitdir   => '/var/lib/git',

-- 
To view, visit https://gerrit.wikimedia.org/r/403380
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I98821528bbb123d11f736206758b05dcdf508628
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_upload vtc: allow_inline_c for backend tests

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/403213 )

Change subject: cache_upload vtc: allow_inline_c for backend tests
..


cache_upload vtc: allow_inline_c for backend tests

Since 8f5337c, cache_upload requires inline-c on both frontend and
backend VCL. Update VTC test cases accordingly.

Change-Id: I7e3eb590c3857cf1b970fb7e71c6c514617fa37a
---
M modules/varnish/files/tests/upload/01-basic-caching.vtc
M modules/varnish/files/tests/upload/02-unset-xrange.vtc
M modules/varnish/files/tests/upload/03-backend-if-cached.vtc
M modules/varnish/files/tests/upload/05-range-requests.vtc
M modules/varnish/files/tests/upload/06-range-requests-convert-into-pass.vtc
M modules/varnish/files/tests/upload/07-x-mediawiki-original.vtc
M modules/varnish/files/tests/upload/13-upload-hfp-small-objects.vtc
7 files changed, 7 insertions(+), 7 deletions(-)

Approvals:
  Ema: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/varnish/files/tests/upload/01-basic-caching.vtc 
b/modules/varnish/files/tests/upload/01-basic-caching.vtc
index 222e2ae..e2927f4 100644
--- a/modules/varnish/files/tests/upload/01-basic-caching.vtc
+++ b/modules/varnish/files/tests/upload/01-basic-caching.vtc
@@ -5,7 +5,7 @@
 txresp -bodylen 1024
 } -start
 
-varnish v1 -arg "-p vcc_err_unref=false" -vcl+backend {
+varnish v1 -arg "-p vcc_err_unref=false -p vcc_allow_inline_c=true" 
-vcl+backend {
 backend vtc_backend {
 .host = "${s1_addr}"; .port = "${s1_port}";
 }
diff --git a/modules/varnish/files/tests/upload/02-unset-xrange.vtc 
b/modules/varnish/files/tests/upload/02-unset-xrange.vtc
index bc5e198..5952d78 100644
--- a/modules/varnish/files/tests/upload/02-unset-xrange.vtc
+++ b/modules/varnish/files/tests/upload/02-unset-xrange.vtc
@@ -7,7 +7,7 @@
 txresp
 } -start
 
-varnish v1 -arg "-p vcc_err_unref=false" -vcl+backend {
+varnish v1 -arg "-p vcc_err_unref=false -p vcc_allow_inline_c=true" 
-vcl+backend {
 backend vtc_backend {
 .host = "${s1_addr}"; .port = "${s1_port}";
 }
diff --git a/modules/varnish/files/tests/upload/03-backend-if-cached.vtc 
b/modules/varnish/files/tests/upload/03-backend-if-cached.vtc
index 1c67fbc..34c85c7 100644
--- a/modules/varnish/files/tests/upload/03-backend-if-cached.vtc
+++ b/modules/varnish/files/tests/upload/03-backend-if-cached.vtc
@@ -5,7 +5,7 @@
 txresp -hdr "ETag: 15f0fff99ed5aae4edffdd6496d7131f" -bodylen 1024
 } -start
 
-varnish v1 -arg "-p vcc_err_unref=false" -vcl+backend {
+varnish v1 -arg "-p vcc_err_unref=false -p vcc_allow_inline_c=true" 
-vcl+backend {
 backend vtc_backend {
 .host = "${s1_addr}"; .port = "${s1_port}";
 }
diff --git a/modules/varnish/files/tests/upload/05-range-requests.vtc 
b/modules/varnish/files/tests/upload/05-range-requests.vtc
index 53f3a9b..5239a57 100644
--- a/modules/varnish/files/tests/upload/05-range-requests.vtc
+++ b/modules/varnish/files/tests/upload/05-range-requests.vtc
@@ -11,7 +11,7 @@
 txresp -bodylen 20
 } -start
 
-varnish v1 -arg "-p vcc_err_unref=false" -vcl+backend {
+varnish v1 -arg "-p vcc_err_unref=false -p vcc_allow_inline_c=true" 
-vcl+backend {
 backend vtc_backend {
 .host = "${s1_addr}"; .port = "${s1_port}";
 }
diff --git 
a/modules/varnish/files/tests/upload/06-range-requests-convert-into-pass.vtc 
b/modules/varnish/files/tests/upload/06-range-requests-convert-into-pass.vtc
index 3e4ea8e..d6d21f2 100644
--- a/modules/varnish/files/tests/upload/06-range-requests-convert-into-pass.vtc
+++ b/modules/varnish/files/tests/upload/06-range-requests-convert-into-pass.vtc
@@ -7,7 +7,7 @@
 txresp -bodylen 20
 } -start
 
-varnish v1 -arg "-p vcc_err_unref=false" -vcl+backend {
+varnish v1 -arg "-p vcc_err_unref=false -p vcc_allow_inline_c=true" 
-vcl+backend {
 backend vtc_backend {
 .host = "${s1_addr}"; .port = "${s1_port}";
 }
diff --git a/modules/varnish/files/tests/upload/07-x-mediawiki-original.vtc 
b/modules/varnish/files/tests/upload/07-x-mediawiki-original.vtc
index 6e1545d..8dec18a 100644
--- a/modules/varnish/files/tests/upload/07-x-mediawiki-original.vtc
+++ b/modules/varnish/files/tests/upload/07-x-mediawiki-original.vtc
@@ -5,7 +5,7 @@
 txresp -bodylen 1024
 } -start
 
-varnish v1 -arg "-p vcc_err_unref=false" -vcl+backend {
+varnish v1 -arg "-p vcc_err_unref=false -p vcc_allow_inline_c=true" 
-vcl+backend {
 backend vtc_backend {
 .host = "${s1_addr}"; .port = "${s1_port}";
 }
diff --git a/modules/varnish/files/tests/upload/13-upload-hfp-small-objects.vtc 
b/modules/varnish/files/tests/upload/13-upload-hfp-small-objects.vtc
index 80e162c..2a88974 100644
--- a/modules/varnish/files/tests/upload/13-upload-hfp-small-objects.vtc
+++ b/modules/varnish/files/tests/upload/13-upload-hfp-small-objects.vtc
@@ -14,7 +14,7 @@
 txresp -bodylen 1025
 } -start
 
-varnish v1 -arg "-p vcc_err_unref=false" 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_upload vtc: allow_inline_c for backend tests

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/403213 )

Change subject: cache_upload vtc: allow_inline_c for backend tests
..

cache_upload vtc: allow_inline_c for backend tests

Since 8f5337c, cache_upload requires inline-c on both frontend and
backend VCL. Update VTC test cases accordingly.

Change-Id: I7e3eb590c3857cf1b970fb7e71c6c514617fa37a
---
M modules/varnish/files/tests/upload/01-basic-caching.vtc
M modules/varnish/files/tests/upload/02-unset-xrange.vtc
M modules/varnish/files/tests/upload/03-backend-if-cached.vtc
M modules/varnish/files/tests/upload/05-range-requests.vtc
M modules/varnish/files/tests/upload/06-range-requests-convert-into-pass.vtc
M modules/varnish/files/tests/upload/07-x-mediawiki-original.vtc
M modules/varnish/files/tests/upload/13-upload-hfp-small-objects.vtc
7 files changed, 7 insertions(+), 7 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/13/403213/1

diff --git a/modules/varnish/files/tests/upload/01-basic-caching.vtc 
b/modules/varnish/files/tests/upload/01-basic-caching.vtc
index 222e2ae..e2927f4 100644
--- a/modules/varnish/files/tests/upload/01-basic-caching.vtc
+++ b/modules/varnish/files/tests/upload/01-basic-caching.vtc
@@ -5,7 +5,7 @@
 txresp -bodylen 1024
 } -start
 
-varnish v1 -arg "-p vcc_err_unref=false" -vcl+backend {
+varnish v1 -arg "-p vcc_err_unref=false -p vcc_allow_inline_c=true" 
-vcl+backend {
 backend vtc_backend {
 .host = "${s1_addr}"; .port = "${s1_port}";
 }
diff --git a/modules/varnish/files/tests/upload/02-unset-xrange.vtc 
b/modules/varnish/files/tests/upload/02-unset-xrange.vtc
index bc5e198..5952d78 100644
--- a/modules/varnish/files/tests/upload/02-unset-xrange.vtc
+++ b/modules/varnish/files/tests/upload/02-unset-xrange.vtc
@@ -7,7 +7,7 @@
 txresp
 } -start
 
-varnish v1 -arg "-p vcc_err_unref=false" -vcl+backend {
+varnish v1 -arg "-p vcc_err_unref=false -p vcc_allow_inline_c=true" 
-vcl+backend {
 backend vtc_backend {
 .host = "${s1_addr}"; .port = "${s1_port}";
 }
diff --git a/modules/varnish/files/tests/upload/03-backend-if-cached.vtc 
b/modules/varnish/files/tests/upload/03-backend-if-cached.vtc
index 1c67fbc..34c85c7 100644
--- a/modules/varnish/files/tests/upload/03-backend-if-cached.vtc
+++ b/modules/varnish/files/tests/upload/03-backend-if-cached.vtc
@@ -5,7 +5,7 @@
 txresp -hdr "ETag: 15f0fff99ed5aae4edffdd6496d7131f" -bodylen 1024
 } -start
 
-varnish v1 -arg "-p vcc_err_unref=false" -vcl+backend {
+varnish v1 -arg "-p vcc_err_unref=false -p vcc_allow_inline_c=true" 
-vcl+backend {
 backend vtc_backend {
 .host = "${s1_addr}"; .port = "${s1_port}";
 }
diff --git a/modules/varnish/files/tests/upload/05-range-requests.vtc 
b/modules/varnish/files/tests/upload/05-range-requests.vtc
index 53f3a9b..5239a57 100644
--- a/modules/varnish/files/tests/upload/05-range-requests.vtc
+++ b/modules/varnish/files/tests/upload/05-range-requests.vtc
@@ -11,7 +11,7 @@
 txresp -bodylen 20
 } -start
 
-varnish v1 -arg "-p vcc_err_unref=false" -vcl+backend {
+varnish v1 -arg "-p vcc_err_unref=false -p vcc_allow_inline_c=true" 
-vcl+backend {
 backend vtc_backend {
 .host = "${s1_addr}"; .port = "${s1_port}";
 }
diff --git 
a/modules/varnish/files/tests/upload/06-range-requests-convert-into-pass.vtc 
b/modules/varnish/files/tests/upload/06-range-requests-convert-into-pass.vtc
index 3e4ea8e..d6d21f2 100644
--- a/modules/varnish/files/tests/upload/06-range-requests-convert-into-pass.vtc
+++ b/modules/varnish/files/tests/upload/06-range-requests-convert-into-pass.vtc
@@ -7,7 +7,7 @@
 txresp -bodylen 20
 } -start
 
-varnish v1 -arg "-p vcc_err_unref=false" -vcl+backend {
+varnish v1 -arg "-p vcc_err_unref=false -p vcc_allow_inline_c=true" 
-vcl+backend {
 backend vtc_backend {
 .host = "${s1_addr}"; .port = "${s1_port}";
 }
diff --git a/modules/varnish/files/tests/upload/07-x-mediawiki-original.vtc 
b/modules/varnish/files/tests/upload/07-x-mediawiki-original.vtc
index 6e1545d..8dec18a 100644
--- a/modules/varnish/files/tests/upload/07-x-mediawiki-original.vtc
+++ b/modules/varnish/files/tests/upload/07-x-mediawiki-original.vtc
@@ -5,7 +5,7 @@
 txresp -bodylen 1024
 } -start
 
-varnish v1 -arg "-p vcc_err_unref=false" -vcl+backend {
+varnish v1 -arg "-p vcc_err_unref=false -p vcc_allow_inline_c=true" 
-vcl+backend {
 backend vtc_backend {
 .host = "${s1_addr}"; .port = "${s1_port}";
 }
diff --git a/modules/varnish/files/tests/upload/13-upload-hfp-small-objects.vtc 
b/modules/varnish/files/tests/upload/13-upload-hfp-small-objects.vtc
index 80e162c..2a88974 100644
--- a/modules/varnish/files/tests/upload/13-upload-hfp-small-objects.vtc
+++ b/modules/varnish/files/tests/upload/13-upload-hfp-small-objects.vtc
@@ -14,7 +14,7 @@
 txresp -bodylen 1025
 } -start
 
-varnish v1 -arg "-p 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Add DELETE to list of allowed methods for text varnish

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/402433 )

Change subject: Add DELETE to list of allowed methods for text varnish
..


Add DELETE to list of allowed methods for text varnish

Follow-up to I6f7fba56731da3d72dab34f8eb6b3eebc57f4879.
DELETE is also used by reading lists service.

Bug: T182825
Change-Id: Ifd779d9090aa555dcdb961470405a0496b5e054c
---
M modules/profile/manifests/cache/text.pp
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Dbrant: Looks good to me, but someone else must approve
  Ema: Verified; Looks good to me, approved



diff --git a/modules/profile/manifests/cache/text.pp 
b/modules/profile/manifests/cache/text.pp
index d4225b8..a691da4 100644
--- a/modules/profile/manifests/cache/text.pp
+++ b/modules/profile/manifests/cache/text.pp
@@ -49,7 +49,7 @@
 }
 
 $common_vcl_config = {
-'allowed_methods'  => '^(GET|HEAD|OPTIONS|POST|PURGE|PUT)$',
+'allowed_methods'  => '^(GET|HEAD|OPTIONS|POST|PURGE|PUT|DELETE)$',
 'purge_host_regex' => 
$::profile::cache::base::purge_host_not_upload_re,
 'static_host'  => $static_host,
 'top_domain'   => $top_domain,

-- 
To view, visit https://gerrit.wikimedia.org/r/402433
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ifd779d9090aa555dcdb961470405a0496b5e054c
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gergő Tisza 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Dbrant 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: lvs: rename lvs1007 eth interfaces

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402859 )

Change subject: lvs: rename lvs1007 eth interfaces
..

lvs: rename lvs1007 eth interfaces

After the installation of the new HP network interface on lvs1007, eth2
and eth3 get renamed as follows:

  systemd-udevd[506]: renamed network interface eth2 to eth11
  systemd-udevd[509]: renamed network interface eth3 to eth10

Update hiera config accordingly.

Bug: T167299
Change-Id: Ie59d8927bf4ad2b7e1010c144dde9d48df393fcb
---
M hieradata/common/lvs/interfaces.yaml
1 file changed, 4 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/59/402859/1

diff --git a/hieradata/common/lvs/interfaces.yaml 
b/hieradata/common/lvs/interfaces.yaml
index 47985d1..c456994 100644
--- a/hieradata/common/lvs/interfaces.yaml
+++ b/hieradata/common/lvs/interfaces.yaml
@@ -20,7 +20,7 @@
'lvs1004': 'eth0:10.64.17.4'
'lvs1005': 'eth0:10.64.17.5'
'lvs1006': 'eth0:10.64.17.6'
-   'lvs1007': 'eth2:10.64.17.7'
+   'lvs1007': 'eth11:10.64.17.7'
'lvs1008': 'eth2:10.64.17.8'
'lvs1009': 'eth2:10.64.17.9'
'lvs1010': 'eth2:10.64.17.10'
@@ -47,7 +47,7 @@
   'lvs1004': 'eth3:10.64.49.4'
   'lvs1005': 'eth3:10.64.49.5'
   'lvs1006': 'eth3:10.64.49.6'
-  'lvs1007': 'eth3:10.64.49.7'
+  'lvs1007': 'eth10:10.64.49.7'
   'lvs1008': 'eth3:10.64.49.8'
   'lvs1009': 'eth3:10.64.49.9'
   'lvs1010': 'eth3:10.64.49.10'
@@ -69,7 +69,7 @@
   'lvs1001': 'eth1:208.80.154.140'
   'lvs1002': 'eth1:208.80.154.141'
   'lvs1003': 'eth1:208.80.154.142'
-  'lvs1007': 'eth2:208.80.154.161'
+  'lvs1007': 'eth11:208.80.154.161'
   'lvs1008': 'eth2:208.80.154.162'
   'lvs1009': 'eth2:208.80.154.163'
   'lvs1010': 'eth2:208.80.154.164'
@@ -97,7 +97,7 @@
   'lvs1004': 'eth3:208.80.155.103'
   'lvs1005': 'eth3:208.80.155.104'
   'lvs1006': 'eth3:208.80.155.105'
-  'lvs1007': 'eth3:208.80.155.111'
+  'lvs1007': 'eth10:208.80.155.111'
   'lvs1008': 'eth3:208.80.155.112'
   'lvs1009': 'eth3:208.80.155.113'
   'lvs1010': 'eth3:208.80.155.114'

-- 
To view, visit https://gerrit.wikimedia.org/r/402859
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ie59d8927bf4ad2b7e1010c144dde9d48df393fcb
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnishmtail: notify daemons upon mtail program modification

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/402353 )

Change subject: varnishmtail: notify daemons upon mtail program modification
..


varnishmtail: notify daemons upon mtail program modification

Bug: T177199
Change-Id: Ia65841a7b7d2360878a45abd73d30a91ed303ee5
---
M modules/mtail/manifests/program.pp
M modules/varnish/manifests/logging/media.pp
M modules/varnish/manifests/logging/reqstats.pp
M modules/varnish/manifests/logging/rls.pp
M modules/varnish/manifests/logging/statsd.pp
M modules/varnish/manifests/logging/xcache.pp
M modules/varnish/manifests/logging/xcps.pp
7 files changed, 7 insertions(+), 0 deletions(-)

Approvals:
  Ema: Verified; Looks good to me, approved
  Filippo Giunchedi: Looks good to me, but someone else must approve



diff --git a/modules/mtail/manifests/program.pp 
b/modules/mtail/manifests/program.pp
index 3f9fec4..bcbcfca 100644
--- a/modules/mtail/manifests/program.pp
+++ b/modules/mtail/manifests/program.pp
@@ -46,6 +46,7 @@
 ensure  => $ensure,
 content => $content,
 source  => $source,
+notify  => $notify,
 require => File[$destination],
 }
 }
diff --git a/modules/varnish/manifests/logging/media.pp 
b/modules/varnish/manifests/logging/media.pp
index 6b14669..d96f36a 100644
--- a/modules/varnish/manifests/logging/media.pp
+++ b/modules/varnish/manifests/logging/media.pp
@@ -45,5 +45,6 @@
 
 mtail::program { 'varnishmedia':
 source => 'puppet:///modules/mtail/programs/varnishmedia.mtail',
+notify => Service['varnishmtail'],
 }
 }
diff --git a/modules/varnish/manifests/logging/reqstats.pp 
b/modules/varnish/manifests/logging/reqstats.pp
index 0992746..86bce90 100644
--- a/modules/varnish/manifests/logging/reqstats.pp
+++ b/modules/varnish/manifests/logging/reqstats.pp
@@ -63,5 +63,6 @@
 
 mtail::program { 'varnishreqstats':
 source => 'puppet:///modules/mtail/programs/varnishreqstats.mtail',
+notify => Service['varnishmtail'],
 }
 }
diff --git a/modules/varnish/manifests/logging/rls.pp 
b/modules/varnish/manifests/logging/rls.pp
index bcd3dc6..368550e 100644
--- a/modules/varnish/manifests/logging/rls.pp
+++ b/modules/varnish/manifests/logging/rls.pp
@@ -47,5 +47,6 @@
 
 mtail::program { 'varnishrls':
 source => 'puppet:///modules/mtail/programs/varnishrls.mtail',
+notify => Service['varnishmtail'],
 }
 }
diff --git a/modules/varnish/manifests/logging/statsd.pp 
b/modules/varnish/manifests/logging/statsd.pp
index 60be264..1960e02 100644
--- a/modules/varnish/manifests/logging/statsd.pp
+++ b/modules/varnish/manifests/logging/statsd.pp
@@ -64,5 +64,6 @@
 mtail::program { 'varnishbackend':
 source  => 'puppet:///modules/mtail/programs/varnishbackend.mtail',
 destination => '/etc/varnishmtail-backend',
+notify  => Service['varnishmtail-backend'],
 }
 }
diff --git a/modules/varnish/manifests/logging/xcache.pp 
b/modules/varnish/manifests/logging/xcache.pp
index d901c81..069cfbb 100644
--- a/modules/varnish/manifests/logging/xcache.pp
+++ b/modules/varnish/manifests/logging/xcache.pp
@@ -52,5 +52,6 @@
 
 mtail::program { 'varnishxcache':
 source => 'puppet:///modules/mtail/programs/varnishxcache.mtail',
+notify => Service['varnishmtail'],
 }
 }
diff --git a/modules/varnish/manifests/logging/xcps.pp 
b/modules/varnish/manifests/logging/xcps.pp
index e353d21..6383d0e 100644
--- a/modules/varnish/manifests/logging/xcps.pp
+++ b/modules/varnish/manifests/logging/xcps.pp
@@ -47,5 +47,6 @@
 
 mtail::program { 'varnishxcps':
 source => 'puppet:///modules/mtail/programs/varnishxcps.mtail',
+notify => Service['varnishmtail'],
 }
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/402353
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ia65841a7b7d2360878a45abd73d30a91ed303ee5
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnishmtail: notify daemons upon mtail program modification

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402353 )

Change subject: varnishmtail: notify daemons upon mtail program modification
..

varnishmtail: notify daemons upon mtail program modification

Bug: T177199
Change-Id: Ia65841a7b7d2360878a45abd73d30a91ed303ee5
---
M modules/mtail/manifests/program.pp
M modules/varnish/manifests/logging/media.pp
M modules/varnish/manifests/logging/reqstats.pp
M modules/varnish/manifests/logging/rls.pp
M modules/varnish/manifests/logging/statsd.pp
M modules/varnish/manifests/logging/xcache.pp
M modules/varnish/manifests/logging/xcps.pp
7 files changed, 7 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/53/402353/1

diff --git a/modules/mtail/manifests/program.pp 
b/modules/mtail/manifests/program.pp
index 3f9fec4..bcbcfca 100644
--- a/modules/mtail/manifests/program.pp
+++ b/modules/mtail/manifests/program.pp
@@ -46,6 +46,7 @@
 ensure  => $ensure,
 content => $content,
 source  => $source,
+notify  => $notify,
 require => File[$destination],
 }
 }
diff --git a/modules/varnish/manifests/logging/media.pp 
b/modules/varnish/manifests/logging/media.pp
index 6b14669..d96f36a 100644
--- a/modules/varnish/manifests/logging/media.pp
+++ b/modules/varnish/manifests/logging/media.pp
@@ -45,5 +45,6 @@
 
 mtail::program { 'varnishmedia':
 source => 'puppet:///modules/mtail/programs/varnishmedia.mtail',
+notify => Service['varnishmtail'],
 }
 }
diff --git a/modules/varnish/manifests/logging/reqstats.pp 
b/modules/varnish/manifests/logging/reqstats.pp
index 0992746..86bce90 100644
--- a/modules/varnish/manifests/logging/reqstats.pp
+++ b/modules/varnish/manifests/logging/reqstats.pp
@@ -63,5 +63,6 @@
 
 mtail::program { 'varnishreqstats':
 source => 'puppet:///modules/mtail/programs/varnishreqstats.mtail',
+notify => Service['varnishmtail'],
 }
 }
diff --git a/modules/varnish/manifests/logging/rls.pp 
b/modules/varnish/manifests/logging/rls.pp
index bcd3dc6..368550e 100644
--- a/modules/varnish/manifests/logging/rls.pp
+++ b/modules/varnish/manifests/logging/rls.pp
@@ -47,5 +47,6 @@
 
 mtail::program { 'varnishrls':
 source => 'puppet:///modules/mtail/programs/varnishrls.mtail',
+notify => Service['varnishmtail'],
 }
 }
diff --git a/modules/varnish/manifests/logging/statsd.pp 
b/modules/varnish/manifests/logging/statsd.pp
index 60be264..1960e02 100644
--- a/modules/varnish/manifests/logging/statsd.pp
+++ b/modules/varnish/manifests/logging/statsd.pp
@@ -64,5 +64,6 @@
 mtail::program { 'varnishbackend':
 source  => 'puppet:///modules/mtail/programs/varnishbackend.mtail',
 destination => '/etc/varnishmtail-backend',
+notify  => Service['varnishmtail-backend'],
 }
 }
diff --git a/modules/varnish/manifests/logging/xcache.pp 
b/modules/varnish/manifests/logging/xcache.pp
index d901c81..069cfbb 100644
--- a/modules/varnish/manifests/logging/xcache.pp
+++ b/modules/varnish/manifests/logging/xcache.pp
@@ -52,5 +52,6 @@
 
 mtail::program { 'varnishxcache':
 source => 'puppet:///modules/mtail/programs/varnishxcache.mtail',
+notify => Service['varnishmtail'],
 }
 }
diff --git a/modules/varnish/manifests/logging/xcps.pp 
b/modules/varnish/manifests/logging/xcps.pp
index e353d21..6383d0e 100644
--- a/modules/varnish/manifests/logging/xcps.pp
+++ b/modules/varnish/manifests/logging/xcps.pp
@@ -47,5 +47,6 @@
 
 mtail::program { 'varnishxcps':
 source => 'puppet:///modules/mtail/programs/varnishxcps.mtail',
+notify => Service['varnishmtail'],
 }
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/402353
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia65841a7b7d2360878a45abd73d30a91ed303ee5
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnishmtail: specify reload action

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402342 )

Change subject: varnishmtail: specify reload action
..

varnishmtail: specify reload action

Sending the HUP signal to mtail causes the daemon to recompile and
reload all programs in the -progs dir.

Send the HUP sigal to all processes in the varnishmtail{,-backend} units
upon daemon reload.

Bug: T177199
Change-Id: I8a27c4eed8b10c9898d357632dee1ed0e7e3c3a9
---
M modules/varnish/templates/initscripts/varnishmtail-backend.systemd.erb
M modules/varnish/templates/initscripts/varnishmtail.systemd.erb
2 files changed, 2 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/42/402342/1

diff --git 
a/modules/varnish/templates/initscripts/varnishmtail-backend.systemd.erb 
b/modules/varnish/templates/initscripts/varnishmtail-backend.systemd.erb
index 632846a..0e00f80 100644
--- a/modules/varnish/templates/initscripts/varnishmtail-backend.systemd.erb
+++ b/modules/varnish/templates/initscripts/varnishmtail-backend.systemd.erb
@@ -7,6 +7,7 @@
 SyslogIdentifier=varnishmtail-backend
 Restart=always
 ExecStart=/usr/local/bin/varnishmtail-backend <%= @varnishmtail_backend_progs 
%> <%= @varnishmtail_backend_port %>
+ExecReload=/bin/systemctl kill -s HUP varnishmtail-backend.service
 
 [Install]
 WantedBy=multi-user.target
diff --git a/modules/varnish/templates/initscripts/varnishmtail.systemd.erb 
b/modules/varnish/templates/initscripts/varnishmtail.systemd.erb
index fe9012a..fc06567 100644
--- a/modules/varnish/templates/initscripts/varnishmtail.systemd.erb
+++ b/modules/varnish/templates/initscripts/varnishmtail.systemd.erb
@@ -7,6 +7,7 @@
 SyslogIdentifier=varnishmtail
 Restart=always
 ExecStart=/usr/local/bin/varnishmtail <%= @mtail_progs %>
+ExecReload=/bin/systemctl kill -s HUP varnishmtail.service
 
 [Install]
 WantedBy=multi-user.target

-- 
To view, visit https://gerrit.wikimedia.org/r/402342
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I8a27c4eed8b10c9898d357632dee1ed0e7e3c3a9
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::cache::text: do not include ipsec role for pinkunicorn

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/402067 )

Change subject: role::cache::text: do not include ipsec role for pinkunicorn
..


role::cache::text: do not include ipsec role for pinkunicorn

Change-Id: Ic3eb83e872312c4ab0e29cb470291682c1cbf9ec
---
M modules/role/manifests/cache/text.pp
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Ema: Verified; Looks good to me, approved



diff --git a/modules/role/manifests/cache/text.pp 
b/modules/role/manifests/cache/text.pp
index 8d362c7..d7d9c33 100644
--- a/modules/role/manifests/cache/text.pp
+++ b/modules/role/manifests/cache/text.pp
@@ -15,7 +15,7 @@
 include ::profile::cache::kafka::statsv
 
 # TODO: refactor all this so that we have separate roles for production 
and labs
-if $::realm == 'production' {
+if $::realm == 'production' and $::hostname != 'cp1008' {
 include ::role::ipsec
 }
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/402067
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ic3eb83e872312c4ab0e29cb470291682c1cbf9ec
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::cache::text: do not include ipsec role for pinkunicorn

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402067 )

Change subject: role::cache::text: do not include ipsec role for pinkunicorn
..

role::cache::text: do not include ipsec role for pinkunicorn

Change-Id: Ic3eb83e872312c4ab0e29cb470291682c1cbf9ec
---
M modules/role/manifests/cache/text.pp
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/67/402067/1

diff --git a/modules/role/manifests/cache/text.pp 
b/modules/role/manifests/cache/text.pp
index 8d362c7..d7d9c33 100644
--- a/modules/role/manifests/cache/text.pp
+++ b/modules/role/manifests/cache/text.pp
@@ -15,7 +15,7 @@
 include ::profile::cache::kafka::statsv
 
 # TODO: refactor all this so that we have separate roles for production 
and labs
-if $::realm == 'production' {
+if $::realm == 'production' and $::hostname != 'cp1008' {
 include ::role::ipsec
 }
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/402067
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic3eb83e872312c4ab0e29cb470291682c1cbf9ec
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_canary: use main Kafka cluster(s)

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402061 )

Change subject: cache_canary: use main Kafka cluster(s)
..

cache_canary: use main Kafka cluster(s)

The change introduced in 31874a8 for cache_text should be applied to
cache_canary too.

Change-Id: Iec83fb3acb34806409c40751fe11769824adbc25
---
M hieradata/role/common/cache/canary.yaml
1 file changed, 1 insertion(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/61/402061/1

diff --git a/hieradata/role/common/cache/canary.yaml 
b/hieradata/role/common/cache/canary.yaml
index 1e844c0..40bb4c2 100644
--- a/hieradata/role/common/cache/canary.yaml
+++ b/hieradata/role/common/cache/canary.yaml
@@ -94,3 +94,4 @@
 # Profile::cache::ssl::unified
 profile::cache::ssl::unified::monitoring: true
 profile::cache::ssl::unified::letsencrypt: false
+profile::cache::kafka::statsv::kafka_cluster_name: main-eqiad

-- 
To view, visit https://gerrit.wikimedia.org/r/402061
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Iec83fb3acb34806409c40751fe11769824adbc25
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: mtail: update varnishbackend.mtail regex

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/402022 )

Change subject: mtail: update varnishbackend.mtail regex
..


mtail: update varnishbackend.mtail regex

Varnish backends are not always named "vcl-$uuid.$backend_name", but can
also be defined as "boot.$backend_name". Update regular expression
accordingly.

Bug: T177199
Change-Id: I89f458f92cf32f7c30062546507c24234bb07bc3
---
M modules/mtail/files/programs/varnishbackend.mtail
M modules/mtail/files/test/logs/varnishbackend.test
M modules/mtail/files/test/varnish_test.py
3 files changed, 3 insertions(+), 1 deletion(-)

Approvals:
  Ema: Looks good to me, approved
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, but someone else must approve



diff --git a/modules/mtail/files/programs/varnishbackend.mtail 
b/modules/mtail/files/programs/varnishbackend.mtail
index df4c527..06e7843 100644
--- a/modules/mtail/files/programs/varnishbackend.mtail
+++ b/modules/mtail/files/programs/varnishbackend.mtail
@@ -5,6 +5,6 @@
 
 # TODO(filippo): add proper ttfb histograms once a mtail version with
 # https://github.com/google/mtail/issues/106 is deployed.
-/http_status (?P[0-9][0-9][0-9])\thttp_method 
(?P[A-Z]+)\tbackend vcl-[a-z0-9-]+\.(?P\S+)\t/ {
+/http_status (?P[0-9][0-9][0-9])\thttp_method 
(?P[A-Z]+)\tbackend (vcl-[a-z0-9-]+|boot)\.(?P\S+)\t/ {
   varnish_backend_requests_seconds_count[$status][$method][$backend]++
 }
diff --git a/modules/mtail/files/test/logs/varnishbackend.test 
b/modules/mtail/files/test/logs/varnishbackend.test
index c376434..7c07f66 100644
--- a/modules/mtail/files/test/logs/varnishbackend.test
+++ b/modules/mtail/files/test/logs/varnishbackend.test
@@ -1,3 +1,4 @@
+http_status 301http_method GET backend boot.be_cp1065_eqiad_wmnet  
ttfb 0.001797   
 http_status 200http_method GET backend 
vcl-root-4d6ee02a-4455-43f6-aebd-f6a5f538b139.be_wdqs_svc_eqiad_wmnet   ttfb 
0.071747   
 http_status 200http_method GET backend 
vcl-root-4d6ee02a-4455-43f6-aebd-f6a5f538b139.be_wdqs_svc_eqiad_wmnet   ttfb 
0.015312   
 http_status 200http_method GET backend 
vcl-root-4d6ee02a-4455-43f6-aebd-f6a5f538b139.be_wdqs_svc_eqiad_wmnet   ttfb 
0.018608   
diff --git a/modules/mtail/files/test/varnish_test.py 
b/modules/mtail/files/test/varnish_test.py
index 56c6997..c48fb80 100644
--- a/modules/mtail/files/test/varnish_test.py
+++ b/modules/mtail/files/test/varnish_test.py
@@ -92,3 +92,4 @@
 
self.assertIn(('status=200,method=GET,backend=be_wdqs_svc_eqiad_wmnet', 12), s)
 self.assertIn(('status=204,method=GET,backend=be_bohrium_eqiad_wmnet', 
2), s)
 
self.assertIn(('status=200,method=POST,backend=be_bohrium_eqiad_wmnet', 1), s)
+self.assertIn(('status=301,method=GET,backend=be_cp1065_eqiad_wmnet', 
1), s)

-- 
To view, visit https://gerrit.wikimedia.org/r/402022
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I89f458f92cf32f7c30062546507c24234bb07bc3
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: mtail: update varnishbackend.mtail regex

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/402022 )

Change subject: mtail: update varnishbackend.mtail regex
..

mtail: update varnishbackend.mtail regex

Varnish backends are not always named "vcl-$uuid.$backend_name", but can
also be defined as "boot.$backend_name". Update regular expression
accordingly.

Bug: T177199
Change-Id: I89f458f92cf32f7c30062546507c24234bb07bc3
---
M modules/mtail/files/programs/varnishbackend.mtail
M modules/varnishkafka
2 files changed, 2 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/22/402022/1

diff --git a/modules/mtail/files/programs/varnishbackend.mtail 
b/modules/mtail/files/programs/varnishbackend.mtail
index df4c527..06e7843 100644
--- a/modules/mtail/files/programs/varnishbackend.mtail
+++ b/modules/mtail/files/programs/varnishbackend.mtail
@@ -5,6 +5,6 @@
 
 # TODO(filippo): add proper ttfb histograms once a mtail version with
 # https://github.com/google/mtail/issues/106 is deployed.
-/http_status (?P[0-9][0-9][0-9])\thttp_method 
(?P[A-Z]+)\tbackend vcl-[a-z0-9-]+\.(?P\S+)\t/ {
+/http_status (?P[0-9][0-9][0-9])\thttp_method 
(?P[A-Z]+)\tbackend (vcl-[a-z0-9-]+|boot)\.(?P\S+)\t/ {
   varnish_backend_requests_seconds_count[$status][$method][$backend]++
 }
diff --git a/modules/varnishkafka b/modules/varnishkafka
index 573a656..36fafe3 16
--- a/modules/varnishkafka
+++ b/modules/varnishkafka
@@ -1 +1 @@
-Subproject commit 573a65641dc226104aa272bd8a76dd8e92fec81f
+Subproject commit 36fafe3832330623d4bef1d9cef517bf7c407e0f

-- 
To view, visit https://gerrit.wikimedia.org/r/402022
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I89f458f92cf32f7c30062546507c24234bb07bc3
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: mtail: add program to count varnish backend metrics

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/401535 )

Change subject: mtail: add program to count varnish backend metrics
..


mtail: add program to count varnish backend metrics

Skip ttfb handling for now, the "+=" operation with floats is broken in the 
mtail version
we're deploying. See also upstream issue 
https://github.com/google/mtail/issues/106

Bug: T177199
Change-Id: Ia2fa39674be38ef340a7785e8f10722bd04373bf
---
A modules/mtail/files/programs/varnishbackend.mtail
A modules/mtail/files/test/logs/varnishbackend.test
M modules/mtail/files/test/varnish_test.py
M modules/varnish/manifests/logging/statsd.pp
4 files changed, 48 insertions(+), 0 deletions(-)

Approvals:
  Ema: Verified; Looks good to me, approved



diff --git a/modules/mtail/files/programs/varnishbackend.mtail 
b/modules/mtail/files/programs/varnishbackend.mtail
new file mode 100644
index 000..df4c527
--- /dev/null
+++ b/modules/mtail/files/programs/varnishbackend.mtail
@@ -0,0 +1,10 @@
+# Three counters to implement Prometheus histograms
+counter varnish_backend_requests_seconds_bucket by le, status, method, backend
+counter varnish_backend_requests_seconds_sum by status, method, backend
+counter varnish_backend_requests_seconds_count by status, method, backend
+
+# TODO(filippo): add proper ttfb histograms once a mtail version with
+# https://github.com/google/mtail/issues/106 is deployed.
+/http_status (?P[0-9][0-9][0-9])\thttp_method 
(?P[A-Z]+)\tbackend vcl-[a-z0-9-]+\.(?P\S+)\t/ {
+  varnish_backend_requests_seconds_count[$status][$method][$backend]++
+}
diff --git a/modules/mtail/files/test/logs/varnishbackend.test 
b/modules/mtail/files/test/logs/varnishbackend.test
new file mode 100644
index 000..c376434
--- /dev/null
+++ b/modules/mtail/files/test/logs/varnishbackend.test
@@ -0,0 +1,20 @@
+http_status 200http_method GET backend 
vcl-root-4d6ee02a-4455-43f6-aebd-f6a5f538b139.be_wdqs_svc_eqiad_wmnet   ttfb 
0.071747   
+http_status 200http_method GET backend 
vcl-root-4d6ee02a-4455-43f6-aebd-f6a5f538b139.be_wdqs_svc_eqiad_wmnet   ttfb 
0.015312   
+http_status 200http_method GET backend 
vcl-root-4d6ee02a-4455-43f6-aebd-f6a5f538b139.be_wdqs_svc_eqiad_wmnet   ttfb 
0.018608   
+http_status 200http_method GET backend 
vcl-root-4d6ee02a-4455-43f6-aebd-f6a5f538b139.be_phab1001_eqiad_wmnet   ttfb 
0.630876   
+http_status 200http_method GET backend 
vcl-root-4d6ee02a-4455-43f6-aebd-f6a5f538b139.be_wdqs_svc_eqiad_wmnet   ttfb 
0.014456   
+http_status 200http_method GET backend 
vcl-root-4d6ee02a-4455-43f6-aebd-f6a5f538b139.be_wdqs_svc_eqiad_wmnet   ttfb 
0.015593   
+http_status 200http_method GET backend 
vcl-root-4d6ee02a-4455-43f6-aebd-f6a5f538b139.be_labmon1001_eqiad_wmnet ttfb 
0.013696   
+http_status 200http_method GET backend 
vcl-root-4d6ee02a-4455-43f6-aebd-f6a5f538b139.be_labmon1001_eqiad_wmnet ttfb 
0.015762   
+http_status 200http_method GET backend 
vcl-root-4d6ee02a-4455-43f6-aebd-f6a5f538b139.be_labmon1001_eqiad_wmnet ttfb 
0.019773   
+http_status 200http_method GET backend 
vcl-root-4d6ee02a-4455-43f6-aebd-f6a5f538b139.be_phab1001_eqiad_wmnet   ttfb 
0.615119   
+http_status 200http_method GET backend 
vcl-root-4d6ee02a-4455-43f6-aebd-f6a5f538b139.be_wdqs_svc_eqiad_wmnet   ttfb 
0.068601   
+http_status 200http_method GET backend 
vcl-root-4d6ee02a-4455-43f6-aebd-f6a5f538b139.be_wdqs_svc_eqiad_wmnet   ttfb 
0.106901   
+http_status 200http_method GET backend 
vcl-root-4d6ee02a-4455-43f6-aebd-f6a5f538b139.be_wdqs_svc_eqiad_wmnet   ttfb 
0.069430   
+http_status 204http_method GET backend 
vcl-root-4d6ee02a-4455-43f6-aebd-f6a5f538b139.be_bohrium_eqiad_wmnetttfb 
0.052067   
+http_status 200http_method GET backend 
vcl-root-4d6ee02a-4455-43f6-aebd-f6a5f538b139.be_wdqs_svc_eqiad_wmnet   ttfb 
0.059147   
+http_status 200http_method GET backend 
vcl-root-4d6ee02a-4455-43f6-aebd-f6a5f538b139.be_wdqs_svc_eqiad_wmnet   ttfb 
0.705404   
+http_status 200http_method GET backend 
vcl-root-4d6ee02a-4455-43f6-aebd-f6a5f538b139.be_wdqs_svc_eqiad_wmnet   ttfb 
0.015081   
+http_status 200http_method POSTbackend 
vcl-root-4d6ee02a-4455-43f6-aebd-f6a5f538b139.be_bohrium_eqiad_wmnetttfb 
0.061224   
+http_status 200http_method GET backend 
vcl-root-4d6ee02a-4455-43f6-aebd-f6a5f538b139.be_wdqs_svc_eqiad_wmnet   ttfb 
0.068993   
+http_status 204http_method GET backend 
vcl-root-4d6ee02a-4455-43f6-aebd-f6a5f538b139.be_bohrium_eqiad_wmnetttfb 
0.052269   
diff --git a/modules/mtail/files/test/varnish_test.py 
b/modules/mtail/files/test/varnish_test.py
index c22e198..56c6997 100644
--- a/modules/mtail/files/test/varnish_test.py
+++ b/modules/mtail/files/test/varnish_test.py
@@ -79,3 +79,16 @@
 self.assertIn(('status=200,method=GET', 3), 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnish: add varnishmtail instance for varnish backends

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/401526 )

Change subject: varnish: add varnishmtail instance for varnish backends
..


varnish: add varnishmtail instance for varnish backends

We have introduced a service called varnishmtail starting with
4159504325de721e7236c16e8f40073d5a113a2b with the goal of gathering and
exposing metrics related to varnish frontend instances.

Add a new service called varnishmtail-backend, responsible for doing
similar work with varnish backends. We want to keep the two systems
separate for various reasons, including the fact that the work done by
varnish on the backend layer could in the future be done by an entirely
different software. Also, logically speaking serving client requests and
routing HTTP requests through our CDN are different tasks, which should
be cleanly separated at the monitoring level too.

Bug: T177199
Change-Id: Ia4b55413ae2fa2fd0a88bd937366a59474d03871
---
A modules/varnish/files/varnishmtail-backend
M modules/varnish/manifests/logging.pp
A modules/varnish/templates/initscripts/varnishmtail-backend.systemd.erb
3 files changed, 58 insertions(+), 0 deletions(-)

Approvals:
  Ema: Verified; Looks good to me, approved
  Filippo Giunchedi: Looks good to me, but someone else must approve



diff --git a/modules/varnish/files/varnishmtail-backend 
b/modules/varnish/files/varnishmtail-backend
new file mode 100644
index 000..0d9f383
--- /dev/null
+++ b/modules/varnish/files/varnishmtail-backend
@@ -0,0 +1,20 @@
+#!/bin/bash
+#
+# varnishmtail-backend - pipe varnishncsa output to mtail
+
+PROGS="${1:-/etc/mtail}"
+PORT="${2:-3903}"
+
+fmt_http_status='http_status %s'
+fmt_http_method='http_method %m'
+fmt_backend='backend %{VSL:BackendOpen[2]}x'
+fmt_ttfb='ttfb %{Varnish:time_firstbyte}x'
+
+FMT="${fmt_http_status}\t${fmt_http_method}\t${fmt_backend}\t${fmt_ttfb}\t"
+
+# Exclude client requests resulting in a pipe as they do not generate a backend
+# request. Varnish blindly passes on bytes in both directions in that case, so
+# there is no status and no ttfb.
+VSL_QUERY='BereqMethod ne "PURGE" and VCL_call ne "PIPE"'
+
+/usr/bin/varnishncsa -b -q "${VSL_QUERY}" -F "${FMT}" | mtail -progs 
"${PROGS}" -logfds 0 -port "${PORT}"
diff --git a/modules/varnish/manifests/logging.pp 
b/modules/varnish/manifests/logging.pp
index a236470..95491fd 100644
--- a/modules/varnish/manifests/logging.pp
+++ b/modules/varnish/manifests/logging.pp
@@ -17,11 +17,21 @@
 # [*mtail_progs*]
 #   Directory with mtail programs. Defaults to /etc/mtail.
 #
+# [*varnishmtail_backend_progs*]
+#   Directory with varnish backend mtail programs.
+#   Defaults to /etc/varnishmtail-backend/.
+#
+# [*varnishmtail_backend_port*]
+#   Port on which to bind the varnish backend mtail instance.
+#   Defaults to 3904.
+#
 class varnish::logging(
 $cache_cluster,
 $statsd_host,
 $forward_syslog='',
 $mtail_progs='/etc/mtail',
+$varnishmtail_backend_progs='/etc/varnishmtail-backend/',
+$varnishmtail_backend_port=3904,
 ){
 rsyslog::conf { 'varnish':
 content  => template('varnish/rsyslog.conf.erb'),
@@ -44,6 +54,22 @@
 require => File['/usr/local/bin/varnishmtail'],
 }
 
+file { '/usr/local/bin/varnishmtail-backend':
+ensure => present,
+owner  => 'root',
+group  => 'root',
+mode   => '0555',
+source => 'puppet:///modules/varnish/varnishmtail-backend',
+notify => Systemd::Service['varnishmtail-backend'],
+}
+
+systemd::service { 'varnishmtail-backend':
+ensure  => present,
+content => systemd_template('varnishmtail-backend'),
+restart => true,
+require => File['/usr/local/bin/varnishmtail-backend'],
+}
+
 # Client connection stats from the 'X-Connection-Properties'
 # header set by the SSL terminators.
 ::varnish::logging::xcps { 'xcps':
diff --git 
a/modules/varnish/templates/initscripts/varnishmtail-backend.systemd.erb 
b/modules/varnish/templates/initscripts/varnishmtail-backend.systemd.erb
new file mode 100644
index 000..632846a
--- /dev/null
+++ b/modules/varnish/templates/initscripts/varnishmtail-backend.systemd.erb
@@ -0,0 +1,12 @@
+[Unit]
+Description=Varnish Backend mtail
+After=varnish.service
+Requires=varnish.service
+
+[Service]
+SyslogIdentifier=varnishmtail-backend
+Restart=always
+ExecStart=/usr/local/bin/varnishmtail-backend <%= @varnishmtail_backend_progs 
%> <%= @varnishmtail_backend_port %>
+
+[Install]
+WantedBy=multi-user.target

-- 
To view, visit https://gerrit.wikimedia.org/r/401526
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ia4b55413ae2fa2fd0a88bd937366a59474d03871
Gerrit-PatchSet: 7
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Ema 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: admin: add yubikey for ema

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/401702 )

Change subject: admin: add yubikey for ema
..


admin: add yubikey for ema

Change-Id: I73a6eed15fdd239fc6cd3fe4311593549f7fc542
---
M modules/admin/data/data.yaml
1 file changed, 1 insertion(+), 0 deletions(-)

Approvals:
  Ema: Verified; Looks good to me, approved



diff --git a/modules/admin/data/data.yaml b/modules/admin/data/data.yaml
index 83fb8bf..5abd19a 100644
--- a/modules/admin/data/data.yaml
+++ b/modules/admin/data/data.yaml
@@ -2061,6 +2061,7 @@
 realname: Emanuele Rocca
 ssh_keys:
   - ssh-rsa 
B3NzaC1yc2EDAQABAAABAQCfoCT4+pXcLRk3jsZsxFsAtVP38Rs4pfdxSS05+hkMcN1yxi2R1ZeNFkQkCPo6s25Gq863ru5E/vxSjy9mDS4HeaxNSJlPPWD0iEmGdyCg+wEGAZ01n1muFDyb+oiSNy5JcgOk0ZqAPNPYlISKVp+vLro5txdEK8cySy4BIbiM5ygKIjmq1xP+dM9zVdd/UbVQPPBFS2UDBzLX3AeYen4biMx2GkHBpZ/nJLqtb9JE4LLzQAtrqi4+J9jKQGepeTemVQ/n3lbheiGmTDZbRsUx7GTXmdIOPqN4k6FEXEgMIe5plxV8YeAoWqMsGpln4J4bz9HCtM1dL5mdLsa7tkq/
 ema@orion
+  - ssh-rsa 
B3NzaC1yc2EDAQABAAABAQCZ4V90QtsYJ4QuykOcVJQPJ64S5wkQdus7N4W8km9LChzvD/dgu/K2NbvRpUSVki+JGGIi8rPyUpGi+ELYd0E6ul3izSnIsMfyTSHqM/craBe8CPmoiFEHKIkXmOGvl1HHczaeziotFVkBm6D5e7MEuz2QJc83XT95trwjZW+e8Cz6f6xW+SGcWDCqOceI6n3wc0PwnWUKZVum2nU71FFZlcCg/fL5srLPis3GWU3zfTmZpvHmXnBE5fFEd6wzQOoSNPHVpehtbgIIrFh5KU5j5HhksHyRZAez+LM19VuZyYOeqUkmIiGq/3hHn+npCOR/r51Lf1Smua+GPyEQNqbn
 ema@newbikey
 uid: 4565
 email: ero...@wikimedia.org
   elukey:

-- 
To view, visit https://gerrit.wikimedia.org/r/401702
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I73a6eed15fdd239fc6cd3fe4311593549f7fc542
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: mtail::program: allow to specify destination directory

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/401533 )

Change subject: mtail::program: allow to specify destination directory
..


mtail::program: allow to specify destination directory

Change-Id: Ie5543848b249f9a1d74baa560ee31a95de5e7961
---
M modules/mtail/manifests/program.pp
1 file changed, 20 insertions(+), 4 deletions(-)

Approvals:
  Ema: Looks good to me, approved
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, but someone else must approve



diff --git a/modules/mtail/manifests/program.pp 
b/modules/mtail/manifests/program.pp
index e54fd74..3f9fec4 100644
--- a/modules/mtail/manifests/program.pp
+++ b/modules/mtail/manifests/program.pp
@@ -15,21 +15,37 @@
 #   The content of the file provided as a puppet:/// file reference.
 #   Either this or 'content' must be specified.
 #
+# [*destination*]
+#   The directory where the mtail script will be installed provided as a
+#   string. Defaults to '/etc/mtail'.
+#
 define mtail::program(
-$ensure   = present,
-$content  = undef,
-$source   = undef,
+$ensure  = present,
+$content = undef,
+$source  = undef,
+$destination = '/etc/mtail',
 ) {
 validate_ensure($ensure)
+validate_absolute_path($destination)
 
 include ::mtail
 
 $basename = regsubst($title, '\W', '-', 'G')
-$filename = "/etc/mtail/${basename}.mtail"
+$filename = "${destination}/${basename}.mtail"
+
+if !defined(File[$destination]) {
+file { $destination:
+ensure => directory,
+owner  => 'root',
+group  => 'root',
+mode   => '0755',
+}
+}
 
 file { $filename:
 ensure  => $ensure,
 content => $content,
 source  => $source,
+require => File[$destination],
 }
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/401533
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ie5543848b249f9a1d74baa560ee31a95de5e7961
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: mtail::program: allow to specify destination directory

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/401533 )

Change subject: mtail::program: allow to specify destination directory
..

mtail::program: allow to specify destination directory

Change-Id: Ie5543848b249f9a1d74baa560ee31a95de5e7961
---
M modules/mtail/manifests/program.pp
1 file changed, 10 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/33/401533/1

diff --git a/modules/mtail/manifests/program.pp 
b/modules/mtail/manifests/program.pp
index e54fd74..75057f1 100644
--- a/modules/mtail/manifests/program.pp
+++ b/modules/mtail/manifests/program.pp
@@ -15,17 +15,23 @@
 #   The content of the file provided as a puppet:/// file reference.
 #   Either this or 'content' must be specified.
 #
+# [*destination*]
+#   The directory where the mtail script will be installed provided as a
+#   string. Defaults to '/etc/mtail'.
+#
 define mtail::program(
-$ensure   = present,
-$content  = undef,
-$source   = undef,
+$ensure  = present,
+$content = undef,
+$source  = undef,
+$destination = '/etc/mtail',
 ) {
 validate_ensure($ensure)
+validate_absolute_path($destination)
 
 include ::mtail
 
 $basename = regsubst($title, '\W', '-', 'G')
-$filename = "/etc/mtail/${basename}.mtail"
+$filename = "${destination}/${basename}.mtail"
 
 file { $filename:
 ensure  => $ensure,

-- 
To view, visit https://gerrit.wikimedia.org/r/401533
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ie5543848b249f9a1d74baa560ee31a95de5e7961
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnish: add varnishmtail instance for varnish backends

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/401526 )

Change subject: varnish: add varnishmtail instance for varnish backends
..

varnish: add varnishmtail instance for varnish backends

We have introduced a service called varnishmtail starting with
4159504325de721e7236c16e8f40073d5a113a2b with the goal of gathering and
exposing metrics related to varnish frontend instances.

Add a new service called varnishmtail-backend, responsible for doing
similar work with varnish backends. We want to keep the two systems
separate for various reasons, including the fact that the work done by
varnish on the backend layer could in the future be done by an entirely
different software. Also, logically speaking serving client requests and
routing HTTP requests through our CDN are different tasks, which should
be cleanly separated at the monitoring level too.

Bug: T177199
Change-Id: Ia4b55413ae2fa2fd0a88bd937366a59474d03871
---
A modules/varnish/files/varnishmtail-backend
M modules/varnish/manifests/logging.pp
A modules/varnish/templates/initscripts/varnishmtail-backend.systemd.erb
3 files changed, 58 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/26/401526/1

diff --git a/modules/varnish/files/varnishmtail-backend 
b/modules/varnish/files/varnishmtail-backend
new file mode 100644
index 000..0d9f383
--- /dev/null
+++ b/modules/varnish/files/varnishmtail-backend
@@ -0,0 +1,20 @@
+#!/bin/bash
+#
+# varnishmtail-backend - pipe varnishncsa output to mtail
+
+PROGS="${1:-/etc/mtail}"
+PORT="${2:-3903}"
+
+fmt_http_status='http_status %s'
+fmt_http_method='http_method %m'
+fmt_backend='backend %{VSL:BackendOpen[2]}x'
+fmt_ttfb='ttfb %{Varnish:time_firstbyte}x'
+
+FMT="${fmt_http_status}\t${fmt_http_method}\t${fmt_backend}\t${fmt_ttfb}\t"
+
+# Exclude client requests resulting in a pipe as they do not generate a backend
+# request. Varnish blindly passes on bytes in both directions in that case, so
+# there is no status and no ttfb.
+VSL_QUERY='BereqMethod ne "PURGE" and VCL_call ne "PIPE"'
+
+/usr/bin/varnishncsa -b -q "${VSL_QUERY}" -F "${FMT}" | mtail -progs 
"${PROGS}" -logfds 0 -port "${PORT}"
diff --git a/modules/varnish/manifests/logging.pp 
b/modules/varnish/manifests/logging.pp
index a236470..95491fd 100644
--- a/modules/varnish/manifests/logging.pp
+++ b/modules/varnish/manifests/logging.pp
@@ -17,11 +17,21 @@
 # [*mtail_progs*]
 #   Directory with mtail programs. Defaults to /etc/mtail.
 #
+# [*varnishmtail_backend_progs*]
+#   Directory with varnish backend mtail programs.
+#   Defaults to /etc/varnishmtail-backend/.
+#
+# [*varnishmtail_backend_port*]
+#   Port on which to bind the varnish backend mtail instance.
+#   Defaults to 3904.
+#
 class varnish::logging(
 $cache_cluster,
 $statsd_host,
 $forward_syslog='',
 $mtail_progs='/etc/mtail',
+$varnishmtail_backend_progs='/etc/varnishmtail-backend/',
+$varnishmtail_backend_port=3904,
 ){
 rsyslog::conf { 'varnish':
 content  => template('varnish/rsyslog.conf.erb'),
@@ -44,6 +54,22 @@
 require => File['/usr/local/bin/varnishmtail'],
 }
 
+file { '/usr/local/bin/varnishmtail-backend':
+ensure => present,
+owner  => 'root',
+group  => 'root',
+mode   => '0555',
+source => 'puppet:///modules/varnish/varnishmtail-backend',
+notify => Systemd::Service['varnishmtail-backend'],
+}
+
+systemd::service { 'varnishmtail-backend':
+ensure  => present,
+content => systemd_template('varnishmtail-backend'),
+restart => true,
+require => File['/usr/local/bin/varnishmtail-backend'],
+}
+
 # Client connection stats from the 'X-Connection-Properties'
 # header set by the SSL terminators.
 ::varnish::logging::xcps { 'xcps':
diff --git 
a/modules/varnish/templates/initscripts/varnishmtail-backend.systemd.erb 
b/modules/varnish/templates/initscripts/varnishmtail-backend.systemd.erb
new file mode 100644
index 000..632846a
--- /dev/null
+++ b/modules/varnish/templates/initscripts/varnishmtail-backend.systemd.erb
@@ -0,0 +1,12 @@
+[Unit]
+Description=Varnish Backend mtail
+After=varnish.service
+Requires=varnish.service
+
+[Service]
+SyslogIdentifier=varnishmtail-backend
+Restart=always
+ExecStart=/usr/local/bin/varnishmtail-backend <%= @varnishmtail_backend_progs 
%> <%= @varnishmtail_backend_port %>
+
+[Install]
+WantedBy=multi-user.target

-- 
To view, visit https://gerrit.wikimedia.org/r/401526
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia4b55413ae2fa2fd0a88bd937366a59474d03871
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list

[MediaWiki-commits] [Gerrit] operations/puppet[production]: prometheus: add reqstats aggregation rule

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/399199 )

Change subject: prometheus: add reqstats aggregation rule
..


prometheus: add reqstats aggregation rule

Add aggregation rule for varnish request stats to allow writing
prometheus queries across DCs.

Bug: T177199
Change-Id: Iedc48adae27c32a34bdba3849138a650d0ced391
---
M modules/role/files/prometheus/rules_ops.conf
1 file changed, 3 insertions(+), 0 deletions(-)

Approvals:
  Ema: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/role/files/prometheus/rules_ops.conf 
b/modules/role/files/prometheus/rules_ops.conf
index b6bbede..b315385 100644
--- a/modules/role/files/prometheus/rules_ops.conf
+++ b/modules/role/files/prometheus/rules_ops.conf
@@ -136,6 +136,9 @@
 none:xcps_h2:sum = sum(xcps_h2)
 none:xcps_tls_sess_reused:sum = sum(xcps_tls_sess_reused)
 
+# reqstats aggregation
+job_method_status:varnish_requests:sum = sum(varnish_requests) by (job, 
method, status)
+
 # MySQL aggregated stats
 job_role_shard:mysql_global_status_queries:rate5m = sum by (job, role, shard) 
(rate(mysql_global_status_queries[5m]))
 job_role_shard:mysql_global_status_handlers_write_total:rate5m = sum by (job, 
role, shard) 
(rate(mysql_global_status_handlers_total{handler=~"(write|update|delete).*"}[5m]))

-- 
To view, visit https://gerrit.wikimedia.org/r/399199
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Iedc48adae27c32a34bdba3849138a650d0ced391
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: prometheus: add reqstats aggregation rule

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/399199 )

Change subject: prometheus: add reqstats aggregation rule
..

prometheus: add reqstats aggregation rule

Add aggregation rule for varnish request stats to allow writing
prometheus queries across DCs.

Bug: T177199
Change-Id: Iedc48adae27c32a34bdba3849138a650d0ced391
---
M modules/role/files/prometheus/rules_ops.conf
1 file changed, 3 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/99/399199/1

diff --git a/modules/role/files/prometheus/rules_ops.conf 
b/modules/role/files/prometheus/rules_ops.conf
index b6bbede..60eed31 100644
--- a/modules/role/files/prometheus/rules_ops.conf
+++ b/modules/role/files/prometheus/rules_ops.conf
@@ -136,6 +136,9 @@
 none:xcps_h2:sum = sum(xcps_h2)
 none:xcps_tls_sess_reused:sum = sum(xcps_tls_sess_reused)
 
+# reqstats aggregation
+method_status:varnish_requests:sum = sum(varnish_requests) by (method, status)
+
 # MySQL aggregated stats
 job_role_shard:mysql_global_status_queries:rate5m = sum by (job, role, shard) 
(rate(mysql_global_status_queries[5m]))
 job_role_shard:mysql_global_status_handlers_write_total:rate5m = sum by (job, 
role, shard) 
(rate(mysql_global_status_handlers_total{handler=~"(write|update|delete).*"}[5m]))

-- 
To view, visit https://gerrit.wikimedia.org/r/399199
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Iedc48adae27c32a34bdba3849138a650d0ced391
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: mtail: add varnishreqstats.mtail

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398819 )

Change subject: mtail: add varnishreqstats.mtail
..


mtail: add varnishreqstats.mtail

Add a mtail script introducing a new metric called `varnish_requests`.
The metric counts HTTP request methods and response status codes.

Stop filtering out PURGEs in varnishmtail as they need to be taken into
account by varnishreqstats.mtail. As a way to ensure no issues arise by
making all other mtail scripts deal with PURGE requests, and as a
potential performace improvement, update all other mtail scripts to
ignore PURGEs.

Bug: T177199
Change-Id: I85e362d434acaf941e46c485ef1aed70a74165dc
---
M modules/mtail/files/programs/varnishmedia.mtail
A modules/mtail/files/programs/varnishreqstats.mtail
M modules/mtail/files/programs/varnishrls.mtail
M modules/mtail/files/programs/varnishxcache.mtail
M modules/mtail/files/programs/varnishxcps.mtail
M modules/mtail/files/test/logs/varnish.test
M modules/mtail/files/test/varnish_test.py
M modules/varnish/files/varnishmtail
M modules/varnish/manifests/logging/reqstats.pp
9 files changed, 65 insertions(+), 27 deletions(-)

Approvals:
  Ema: Looks good to me, approved
  BBlack: Looks good to me, but someone else must approve
  jenkins-bot: Verified



diff --git a/modules/mtail/files/programs/varnishmedia.mtail 
b/modules/mtail/files/programs/varnishmedia.mtail
index 885af9e..f8d4ebc 100644
--- a/modules/mtail/files/programs/varnishmedia.mtail
+++ b/modules/mtail/files/programs/varnishmedia.mtail
@@ -1,5 +1,9 @@
 counter varnish_thumbnails by status
 
-/^url .*\/thumb\/.*\thttp_status (?P.*)\tcache_control/ {
-varnish_thumbnails[$http_status]++
+/\thttp_method PURGE\t/ {
+# noop
+} else {
+/^url .*\/thumb\/.*\thttp_status (?P[0-9][0-9][0-9])\t/ {
+varnish_thumbnails[$http_status]++
+}
 }
diff --git a/modules/mtail/files/programs/varnishreqstats.mtail 
b/modules/mtail/files/programs/varnishreqstats.mtail
new file mode 100644
index 000..764f0e2
--- /dev/null
+++ b/modules/mtail/files/programs/varnishreqstats.mtail
@@ -0,0 +1,5 @@
+counter varnish_requests by status, method
+
+/\thttp_status (?P[0-9][0-9][0-9])\thttp_method (?P[A-Z]+)\t/ {
+varnish_requests[$status][$method]++
+}
diff --git a/modules/mtail/files/programs/varnishrls.mtail 
b/modules/mtail/files/programs/varnishrls.mtail
index 0705e44..210a0c9 100644
--- a/modules/mtail/files/programs/varnishrls.mtail
+++ b/modules/mtail/files/programs/varnishrls.mtail
@@ -1,7 +1,11 @@
 counter varnish_resourceloader_inm
 
-/^url \/w\/load.php.*\tinm (?P.*)$/ {
-$inm != "-" {
-varnish_resourceloader_inm++
+/\thttp_method PURGE\t/ {
+# noop
+} else {
+/^url \/w\/load.php.*\tinm (?P.*)$/ {
+$inm != "-" {
+varnish_resourceloader_inm++
+}
 }
 }
diff --git a/modules/mtail/files/programs/varnishxcache.mtail 
b/modules/mtail/files/programs/varnishxcache.mtail
index df4a197..cd56dfd 100644
--- a/modules/mtail/files/programs/varnishxcache.mtail
+++ b/modules/mtail/files/programs/varnishxcache.mtail
@@ -1,5 +1,9 @@
 counter varnish_x_cache by x_cache
 
-/^.*\tcache_status (?P.*)\thttp_status/ {
-varnish_x_cache[$x_cache]++
+/\thttp_method PURGE\t/ {
+# noop
+} else {
+/^.*\tcache_status (?P.*)\thttp_status/ {
+varnish_x_cache[$x_cache]++
+}
 }
diff --git a/modules/mtail/files/programs/varnishxcps.mtail 
b/modules/mtail/files/programs/varnishxcps.mtail
index 0aebb96..4281b1b 100644
--- a/modules/mtail/files/programs/varnishxcps.mtail
+++ b/modules/mtail/files/programs/varnishxcps.mtail
@@ -2,14 +2,18 @@
 counter xcps_tls_sess_reused
 counter xcps_tls by version, key_exchange, auth, cipher
 
-/\th2 1\t/ {
-xcps_h2++
-}
+/\thttp_method PURGE\t/ {
+# noop
+} else {
+/\th2 1\t/ {
+xcps_h2++
+}
 
-/\tsession_reused 1\t/ {
-xcps_tls_sess_reused++
-}
+/\tsession_reused 1\t/ {
+xcps_tls_sess_reused++
+}
 
-/\ttls_version (?PTLSv[0-9\.]+)\t.*\tkey_exchange 
(?P[a-zA-Z0-9-_]+)\tauth (?P[a-zA-Z0-9-_]+)\tcipher 
(?P[a-zA-Z0-9-_]+)\t/ {
-xcps_tls[$version][$key_exchange][$auth][$cipher]++
+/\ttls_version (?PTLSv[0-9\.]+)\t.*\tkey_exchange 
(?P[a-zA-Z0-9-_]+)\tauth (?P[a-zA-Z0-9-_]+)\tcipher 
(?P[a-zA-Z0-9-_]+)\t/ {
+xcps_tls[$version][$key_exchange][$auth][$cipher]++
+}
 }
diff --git a/modules/mtail/files/test/logs/varnish.test 
b/modules/mtail/files/test/logs/varnish.test
index 5cd96e5..a8f0da0 100644
--- a/modules/mtail/files/test/logs/varnish.test
+++ b/modules/mtail/files/test/logs/varnish.test
@@ -1,9 +1,9 @@
-url /  cache_status int-front  http_status 301 cache_control - inm -   h2 0
tls_version session_reused 0key_exchangeauthcipher  
full_cipher 
-url /w/index.php   cache_status hit-front  http_status 304 cache_control 
private, s-maxage=0, max-age=0, must-revalidate   inm -   h2 1

[MediaWiki-commits] [Gerrit] operations/puppet[production]: mtail: add varnishreqstats.mtail

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/398819 )

Change subject: mtail: add varnishreqstats.mtail
..

mtail: add varnishreqstats.mtail

Add an mtail script introducing a new metric called `varnish_requests`.
The metric counts HTTP request methods and response status codes.

Bug: T177199
Change-Id: I85e362d434acaf941e46c485ef1aed70a74165dc
---
M modules/mtail/files/programs/varnishmedia.mtail
A modules/mtail/files/programs/varnishreqstats.mtail
M modules/mtail/files/test/logs/varnish.test
M modules/mtail/files/test/varnish_test.py
M modules/varnish/files/varnishmtail
5 files changed, 31 insertions(+), 13 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/19/398819/1

diff --git a/modules/mtail/files/programs/varnishmedia.mtail 
b/modules/mtail/files/programs/varnishmedia.mtail
index 885af9e..30325a0 100644
--- a/modules/mtail/files/programs/varnishmedia.mtail
+++ b/modules/mtail/files/programs/varnishmedia.mtail
@@ -1,5 +1,5 @@
 counter varnish_thumbnails by status
 
-/^url .*\/thumb\/.*\thttp_status (?P.*)\tcache_control/ {
+/^url .*\/thumb\/.*\thttp_status (?P[1-5][0-9][0-9])\t/ {
 varnish_thumbnails[$http_status]++
 }
diff --git a/modules/mtail/files/programs/varnishreqstats.mtail 
b/modules/mtail/files/programs/varnishreqstats.mtail
new file mode 100644
index 000..12c4d0e
--- /dev/null
+++ b/modules/mtail/files/programs/varnishreqstats.mtail
@@ -0,0 +1,5 @@
+counter varnish_requests by status, method
+
+/\thttp_status (?P[1-5][0-9][0-9])\thttp_method (?P[A-Z]+)\t/ {
+varnish_requests[$status][$method]++
+}
diff --git a/modules/mtail/files/test/logs/varnish.test 
b/modules/mtail/files/test/logs/varnish.test
index 5cd96e5..a8f0da0 100644
--- a/modules/mtail/files/test/logs/varnish.test
+++ b/modules/mtail/files/test/logs/varnish.test
@@ -1,9 +1,9 @@
-url /  cache_status int-front  http_status 301 cache_control - inm -   h2 0
tls_version session_reused 0key_exchangeauthcipher  
full_cipher 
-url /w/index.php   cache_status hit-front  http_status 304 cache_control 
private, s-maxage=0, max-age=0, must-revalidate   inm -   h2 1tls_version 
TLSv1.2 session_reused 1key_exchange X25519 auth ECDSA  
cipher CHACHA20-POLY1305full_cipher ECDHE-ECDSA-CHACHA20-POLY1305   
-url 
/api/rest_v1/metrics/pageviews/per-article/fr.wikipedia.org/all-access/user/S%C3%A9lection_sexuelle/daily/20171126/20171201
cache_status hit-front  http_status 200 cache_control s-maxage=86400, 
max-age=86400 inm -
-url /w/load.phpcache_status hit-front  http_status 200 cache_control 
public, max-age=2592000, s-maxage=2592000 inm -
-url /w/load.phpcache_status hit-front  http_status 200 cache_control 
public, max-age=2592000, s-maxage=2592000 inm -
-url /w/load.phpcache_status hit-front  http_status 304 cache_control 
public, max-age=300, s-maxage=300 inm W/\"1adp3u3\"
-url /wikipedia/commons/5/51/Tang_Shaoyi.jpgcache_status int-front  
http_status 301 cache_control - inm -
-url 
/wikipedia/commons/thumb/5/51/Flag_of_North_Korea.svg/250px-Flag_of_North_Korea.svg.png
cache_status hit-front  http_status 200 cache_control - inm -
-url /wikipedia/en/thumb/f/fd/Portal-puzzle.svg/16px-Portal-puzzle.svg.png  
cache_status hit-front  http_status 200 cache_control - inm -
+url /  cache_status int-front  http_status 301 http_method GET cache_control - 
inm -   h2 0tls_version session_reused 0key_exchangeauth
cipher  full_cipher 
+url /w/index.php   cache_status hit-front  http_status 304 http_method GET 
cache_control private, s-maxage=0, max-age=0, must-revalidate   inm -   h2 1
tls_version TLSv1.2 session_reused 1key_exchange X25519 auth 
ECDSA  cipher CHACHA20-POLY1305full_cipher 
ECDHE-ECDSA-CHACHA20-POLY1305   
+url 
/api/rest_v1/metrics/pageviews/per-article/fr.wikipedia.org/all-access/user/S%C3%A9lection_sexuelle/daily/20171126/20171201
cache_status hit-front  http_status 200 http_method HEAD
cache_control s-maxage=86400, max-age=86400 inm -
+url /w/load.phpcache_status hit-front  http_status 200 http_method GET 
cache_control public, max-age=2592000, s-maxage=2592000 inm -
+url /w/load.phpcache_status hit-front  http_status 200 http_method 
HEADcache_control public, max-age=2592000, s-maxage=2592000 inm -
+url /w/load.phpcache_status hit-front  http_status 304 http_method GET 
cache_control public, max-age=300, s-maxage=300 inm W/\"1adp3u3\"
+url /wikipedia/commons/5/51/Tang_Shaoyi.jpgcache_status int-front  
http_status 301 http_method GET cache_control - inm -
+url 
/wikipedia/commons/thumb/5/51/Flag_of_North_Korea.svg/250px-Flag_of_North_Korea.svg.png
cache_status hit-front  http_status 200 http_method GET cache_control - inm 
-
+url 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: prometheus: add xcps aggregation rules

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398441 )

Change subject: prometheus: add xcps aggregation rules
..


prometheus: add xcps aggregation rules

Add aggregation rules for stats related to X-Connection-Properties. This
allows us to write prometheus queries across DCs. XCPS carries
information regarding TLS ciphersuites, TLS session reuse and HTTP2
adoption.

Bug: T177199
Change-Id: Ieb0c9cd45a3fd160ffb82f621482ee08e1202899
---
M modules/role/files/prometheus/rules_ops.conf
M modules/role/manifests/prometheus/global.pp
2 files changed, 5 insertions(+), 0 deletions(-)

Approvals:
  Ema: Looks good to me, approved
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, but someone else must approve



diff --git a/modules/role/files/prometheus/rules_ops.conf 
b/modules/role/files/prometheus/rules_ops.conf
index 3fcda12..c7a6f8f 100644
--- a/modules/role/files/prometheus/rules_ops.conf
+++ b/modules/role/files/prometheus/rules_ops.conf
@@ -131,6 +131,10 @@
 backend:varnish_backend_pipe_out:sum = sum(varnish_backend_pipe_out) without 
(server)
 backend:varnish_backend_req:sum = sum(varnish_backend_req) without (server)
 
+# X-Connection-Properties stats aggregation
+auth_cipher_key_exchange_version:xcps_tls:sum = sum(xcps_tls) by (auth, 
cipher, key_exchange, version)
+none:xcps_h2:sum = sum(xcps_h2)
+none:xcps_tls_sess_reused:sum = sum(xcps_tls_sess_reused)
 
 # MySQL aggregated stats
 job_role_shard:mysql_global_status_queries:rate5m = sum by (job, role, shard) 
(rate(mysql_global_status_queries[5m]))
diff --git a/modules/role/manifests/prometheus/global.pp 
b/modules/role/manifests/prometheus/global.pp
index a1302c0..1c0533e 100644
--- a/modules/role/manifests/prometheus/global.pp
+++ b/modules/role/manifests/prometheus/global.pp
@@ -35,6 +35,7 @@
 '{__name__=~"^.*:mysql_.*"}',
 '{__name__=~"^.*:memcached_.*"}',
 '{__name__=~"^.*:varnish_.*"}',
+'{__name__=~"^.*:xcps_.*"}',
 # blackbox_exporter probes results
 '{__name__=~"^probe_.*"}',
   ],

-- 
To view, visit https://gerrit.wikimedia.org/r/398441
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ieb0c9cd45a3fd160ffb82f621482ee08e1202899
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: prometheus: add xcps aggregation rules

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/398441 )

Change subject: prometheus: add xcps aggregation rules
..

prometheus: add xcps aggregation rules

Add aggregation rules for stats related to X-Connection-Properties. This
allows us to write prometheus queries across DCs. XCPS carries
information regarding TLS ciphersuites, TLS session reuse and HTTP2
adoption.

Bug: T177199
Change-Id: Ieb0c9cd45a3fd160ffb82f621482ee08e1202899
---
M modules/role/files/prometheus/rules_ops.conf
1 file changed, 4 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/41/398441/1

diff --git a/modules/role/files/prometheus/rules_ops.conf 
b/modules/role/files/prometheus/rules_ops.conf
index 3fcda12..695350b 100644
--- a/modules/role/files/prometheus/rules_ops.conf
+++ b/modules/role/files/prometheus/rules_ops.conf
@@ -131,6 +131,10 @@
 backend:varnish_backend_pipe_out:sum = sum(varnish_backend_pipe_out) without 
(server)
 backend:varnish_backend_req:sum = sum(varnish_backend_req) without (server)
 
+# X-Connection-Properties stats aggregation
+xcps:xcps_tls:sum = sum(xcps_tls) by (auth, cipher, key_exchange, version)
+xcps:xcps_h2:sum = sum(xcps_h2)
+xcps:xcps_tls_sess_reused:sum = sum(xcps_tls_sess_reused)
 
 # MySQL aggregated stats
 job_role_shard:mysql_global_status_queries:rate5m = sum by (job, role, shard) 
(rate(mysql_global_status_queries[5m]))

-- 
To view, visit https://gerrit.wikimedia.org/r/398441
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ieb0c9cd45a3fd160ffb82f621482ee08e1202899
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: vcl: remove X-CP-Full-Cipher

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/398314 )

Change subject: vcl: remove X-CP-Full-Cipher
..

vcl: remove X-CP-Full-Cipher

Change-Id: I056fb1a07dfbe9dea43c832dae795937e480c3dd
---
M modules/mtail/files/test/logs/varnish.test
M modules/varnish/files/tests/upload/16-x-connection-properties.vtc
M modules/varnish/files/varnishmtail
M modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
4 files changed, 4 insertions(+), 13 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/14/398314/1

diff --git a/modules/mtail/files/test/logs/varnish.test 
b/modules/mtail/files/test/logs/varnish.test
index 1ee9f9b..074d08a 100644
--- a/modules/mtail/files/test/logs/varnish.test
+++ b/modules/mtail/files/test/logs/varnish.test
@@ -1,5 +1,5 @@
-url /  cache_status int-front  http_status 301 cache_control - inm -   h2 0
tls_version session_reused 0key_exchangeauthcipher  
full_cipher 
-url /w/index.php   cache_status hit-front  http_status 304 cache_control 
private, s-maxage=0, max-age=0, must-revalidate   inm -   h2 1tls_version 
TLSv1.2 session_reused 1key_exchange X25519 auth ECDSA  
cipher CHACHA20-POLY1305-SHA256 full_cipher 
ECDHE-ECDSA-CHACHA20-POLY1305-SHA256
+url /  cache_status int-front  http_status 301 cache_control - inm -   h2 0
tls_version session_reused 0key_exchangeauthcipher  
+url /w/index.php   cache_status hit-front  http_status 304 cache_control 
private, s-maxage=0, max-age=0, must-revalidate   inm -   h2 1tls_version 
TLSv1.2 session_reused 1key_exchange X25519 auth ECDSA  
cipher CHACHA20-POLY1305-SHA256 
 url 
/api/rest_v1/metrics/pageviews/per-article/fr.wikipedia.org/all-access/user/S%C3%A9lection_sexuelle/daily/20171126/20171201
cache_status hit-front  http_status 200 cache_control s-maxage=86400, 
max-age=86400 inm -
 url /w/load.phpcache_status hit-front  http_status 200 cache_control 
public, max-age=2592000, s-maxage=2592000 inm -
 url /w/load.phpcache_status hit-front  http_status 200 cache_control 
public, max-age=2592000, s-maxage=2592000 inm -
diff --git a/modules/varnish/files/tests/upload/16-x-connection-properties.vtc 
b/modules/varnish/files/tests/upload/16-x-connection-properties.vtc
index eaa4037..389143f 100644
--- a/modules/varnish/files/tests/upload/16-x-connection-properties.vtc
+++ b/modules/varnish/files/tests/upload/16-x-connection-properties.vtc
@@ -10,7 +10,6 @@
 expect req.http.X-CP-Key-Exchange == "prime256v1"
 expect req.http.X-CP-Auth == "ECDSA"
 expect req.http.X-CP-Cipher == "AES256-GCM-SHA384"
-expect req.http.X-CP-Full-Cipher == "ECDHE-ECDSA-AES256-GCM-SHA384"
 
 txresp
 
@@ -23,7 +22,6 @@
 expect req.http.X-CP-Key-Exchange == "prime256v1"
 expect req.http.X-CP-Auth == "ECDSA"
 expect req.http.X-CP-Cipher == "AES128-SHA"
-expect req.http.X-CP-Full-Cipher == "ECDHE-ECDSA-AES128-SHA"
 
 txresp
 
@@ -36,7 +34,6 @@
 expect req.http.X-CP-Key-Exchange == "X25519"
 expect req.http.X-CP-Auth == "ECDSA"
 expect req.http.X-CP-Cipher == "AES256-GCM-SHA384"
-expect req.http.X-CP-Full-Cipher == "ECDHE-ECDSA-AES256-GCM-SHA384"
 
 txresp
 
@@ -49,7 +46,6 @@
 expect req.http.X-CP-Key-Exchange == "RSA"
 expect req.http.X-CP-Auth == "RSA"
 expect req.http.X-CP-Cipher == "AES128-SHA"
-expect req.http.X-CP-Full-Cipher == "AES128-SHA"
 
 txresp
 } -start
diff --git a/modules/varnish/files/varnishmtail 
b/modules/varnish/files/varnishmtail
index ba2de95..cd53184 100644
--- a/modules/varnish/files/varnishmtail
+++ b/modules/varnish/files/varnishmtail
@@ -15,9 +15,8 @@
 fmt_key_exchange='key_exchange %{VCL_Log:CP-Key-Exchange}x'
 fmt_auth='auth %{VCL_Log:CP-Auth}x'
 fmt_cipher='cipher %{VCL_Log:CP-Cipher}x'
-fmt_full_cipher='full_cipher %{VCL_Log:CP-Full-Cipher}x'
 
-FMT="${fmt_url}\t${fmt_cache_status}\t${fmt_http_status}\t${fmt_cache_control}\t${fmt_inm}\t${fmt_h2}\t${fmt_tls_version}\t${fmt_session_reused}\t${fmt_key_exchange}\t${fmt_auth}\t${fmt_cipher}\t${fmt_full_cipher}\t"
+FMT="${fmt_url}\t${fmt_cache_status}\t${fmt_http_status}\t${fmt_cache_control}\t${fmt_inm}\t${fmt_h2}\t${fmt_tls_version}\t${fmt_session_reused}\t${fmt_key_exchange}\t${fmt_auth}\t${fmt_cipher}\t"
 
 /usr/bin/varnishncsa -n frontend -q 'ReqMethod ne "PURGE"' -F "${FMT}" |
 mtail -progs "${PROGS}" -logfds 0
diff --git a/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb 
b/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
index b4d7551..237d5ed 100644
--- a/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
+++ b/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
@@ -290,9 +290,7 @@
 
set req.http.X-CP-Key-Exchange = 
regsub(req.http.X-Connection-Properties, ".* EC=([A-Za-z0-9]+);.*", "\1");
 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: vcl: add hash function name to CHACHA20-POLY1305 cipher

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/398311 )

Change subject: vcl: add hash function name to CHACHA20-POLY1305 cipher
..

vcl: add hash function name to CHACHA20-POLY1305 cipher

The hash function used by all ciphersuites described in rfc7905 is
SHA-256. Starting with TLSv1.3, CHACHA20-POLY1305 will be renamed into
CHACHA20-POLY1305-SHA256. Do the renaming now in our VCL to avoid stats
getting skewed later on.

Ref: https://tools.ietf.org/html/rfc7905#section-2
Change-Id: I9dec5f879c1b53be2232da83bbbf76170b49a18c
---
M modules/mtail/files/test/logs/varnish.test
M modules/mtail/files/test/varnish_test.py
M modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
3 files changed, 5 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/11/398311/1

diff --git a/modules/mtail/files/test/logs/varnish.test 
b/modules/mtail/files/test/logs/varnish.test
index 5cd96e5..1ee9f9b 100644
--- a/modules/mtail/files/test/logs/varnish.test
+++ b/modules/mtail/files/test/logs/varnish.test
@@ -1,5 +1,5 @@
 url /  cache_status int-front  http_status 301 cache_control - inm -   h2 0
tls_version session_reused 0key_exchangeauthcipher  
full_cipher 
-url /w/index.php   cache_status hit-front  http_status 304 cache_control 
private, s-maxage=0, max-age=0, must-revalidate   inm -   h2 1tls_version 
TLSv1.2 session_reused 1key_exchange X25519 auth ECDSA  
cipher CHACHA20-POLY1305full_cipher ECDHE-ECDSA-CHACHA20-POLY1305   
+url /w/index.php   cache_status hit-front  http_status 304 cache_control 
private, s-maxage=0, max-age=0, must-revalidate   inm -   h2 1tls_version 
TLSv1.2 session_reused 1key_exchange X25519 auth ECDSA  
cipher CHACHA20-POLY1305-SHA256 full_cipher 
ECDHE-ECDSA-CHACHA20-POLY1305-SHA256
 url 
/api/rest_v1/metrics/pageviews/per-article/fr.wikipedia.org/all-access/user/S%C3%A9lection_sexuelle/daily/20171126/20171201
cache_status hit-front  http_status 200 cache_control s-maxage=86400, 
max-age=86400 inm -
 url /w/load.phpcache_status hit-front  http_status 200 cache_control 
public, max-age=2592000, s-maxage=2592000 inm -
 url /w/load.phpcache_status hit-front  http_status 200 cache_control 
public, max-age=2592000, s-maxage=2592000 inm -
diff --git a/modules/mtail/files/test/varnish_test.py 
b/modules/mtail/files/test/varnish_test.py
index 40c6fc4..d45b9c6 100644
--- a/modules/mtail/files/test/varnish_test.py
+++ b/modules/mtail/files/test/varnish_test.py
@@ -60,7 +60,7 @@
 'version=TLSv1.2',
 'key_exchange=X25519',
 'auth=ECDSA',
-'cipher=CHACHA20-POLY1305',
+'cipher=CHACHA20-POLY1305-SHA256',
 ]
 for value in expected:
 self.assertIn(value, labels)
diff --git a/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb 
b/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
index e8c0153..b4d7551 100644
--- a/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
+++ b/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
@@ -314,6 +314,9 @@
set req.http.X-CP-Key-Exchange = "RSA";
}
 
+   // Starting with TLSv1.3
+   set req.http.X-CP-Cipher = regsub(req.http.X-CP-Cipher, 
"^CHACHA20-POLY1305$", "CHACHA20-POLY1305-SHA256");
+
// Log values to shared memory logs. They can be extracted with:
// varnishncsa -F "%{VCL_Log:CP-TLS-Version}x"
std.log("CP-HTTP2: " + req.http.X-CP-HTTP2);

-- 
To view, visit https://gerrit.wikimedia.org/r/398311
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I9dec5f879c1b53be2232da83bbbf76170b49a18c
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnishxcps.mtail: use prometheus labels

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/397876 )

Change subject: varnishxcps.mtail: use prometheus labels
..


varnishxcps.mtail: use prometheus labels

Turn tls version, key_exchange, auth, and cipher into labels. By doing
this, we make it possible to use various combinations of those stats
together and answer questions such as: how many TLSv1.2 connections used
x25519 for key exchange?

Leave HTTP2 and TLS session reuse stats as separate counters.

Bug: T177199
Change-Id: I79c67927cb86b3bbbe0b8dccaba7c767b5296a7a
---
M modules/mtail/files/programs/varnishxcps.mtail
M modules/mtail/files/test/varnish_test.py
2 files changed, 30 insertions(+), 63 deletions(-)

Approvals:
  Ema: Verified; Looks good to me, approved
  Filippo Giunchedi: Looks good to me, but someone else must approve



diff --git a/modules/mtail/files/programs/varnishxcps.mtail 
b/modules/mtail/files/programs/varnishxcps.mtail
index b74f9cb..0aebb96 100644
--- a/modules/mtail/files/programs/varnishxcps.mtail
+++ b/modules/mtail/files/programs/varnishxcps.mtail
@@ -1,36 +1,15 @@
-counter tls_h2
-counter tls_version by version
-counter tls_sess_reused
-counter tls_key_exchange by type
-counter tls_auth by type
-counter tls_cipher by name
-counter tls_full_cipher by name
+counter xcps_h2
+counter xcps_tls_sess_reused
+counter xcps_tls by version, key_exchange, auth, cipher
 
 /\th2 1\t/ {
-tls_h2++
-}
-
-/\ttls_version (?PTLSv[0-9\.]+)\t/ {
-tls_version[$version]++
+xcps_h2++
 }
 
 /\tsession_reused 1\t/ {
-tls_sess_reused++
+xcps_tls_sess_reused++
 }
 
-/\tkey_exchange (?P[a-zA-Z0-9-_]+)\t/ {
-tls_key_exchange[$type]++
-}
-
-/\tauth (?P[a-zA-Z0-9-_]+)\t/ {
-tls_auth[$type]++
-}
-
-
-/\tcipher (?P[a-zA-Z0-9-_]+)\t/ {
-tls_cipher[$name]++
-}
-
-/\tfull_cipher (?P[a-zA-Z0-9-_]+)\t/ {
-tls_full_cipher[$name]++
+/\ttls_version (?PTLSv[0-9\.]+)\t.*\tkey_exchange 
(?P[a-zA-Z0-9-_]+)\tauth (?P[a-zA-Z0-9-_]+)\tcipher 
(?P[a-zA-Z0-9-_]+)\t/ {
+xcps_tls[$version][$key_exchange][$auth][$cipher]++
 }
diff --git a/modules/mtail/files/test/varnish_test.py 
b/modules/mtail/files/test/varnish_test.py
index ccc0d15..40c6fc4 100644
--- a/modules/mtail/files/test/varnish_test.py
+++ b/modules/mtail/files/test/varnish_test.py
@@ -12,9 +12,9 @@
 os.path.join(test_dir, 'logs/varnish.test'))
 
 def testCacheStatus(self):
-m = self.store.get_metric('varnish_x_cache')
-self.assertEqual(2, m._value)
-self.assertIn('x_cache=int-front', m._labelpairs)
+s = self.store.get_samples('varnish_x_cache')
+self.assertIn(('x_cache=int-front', 2), s)
+self.assertIn(('x_cache=hit-front', 7), s)
 
 
 class VarnishRlsTest(unittest.TestCase):
@@ -24,8 +24,8 @@
 os.path.join(test_dir, 'logs/varnish.test'))
 
 def testIfNoneMatch(self):
-m = self.store.get_metric('varnish_resourceloader_inm')
-self.assertEquals(m._value, 1)
+s = self.store.get_samples('varnish_resourceloader_inm')
+self.assertIn(('', 1), s)
 
 
 class VarnishMediaTest(unittest.TestCase):
@@ -35,9 +35,8 @@
 os.path.join(test_dir, 'logs/varnish.test'))
 
 def testThumbnails(self):
-m = self.store.get_metric('varnish_thumbnails')
-self.assertEquals(2, m._value)
-self.assertIn('status=200', m._labelpairs)
+s = self.store.get_samples('varnish_thumbnails')
+self.assertIn(('status=200', 2), s)
 
 
 class VarnishXcpsTest(unittest.TestCase):
@@ -47,34 +46,23 @@
 os.path.join(test_dir, 'logs/varnish.test'))
 
 def testH2(self):
-m = self.store.get_metric('tls_h2')
-self.assertEqual(1, m._value)
+s = self.store.get_samples('xcps_h2')
+self.assertIn(('', 1), s)
 
 def testReusedSessions(self):
-m = self.store.get_metric('tls_sess_reused')
-self.assertEqual(1, m._value)
+s = self.store.get_samples('xcps_tls_sess_reused')
+self.assertIn(('', 1), s)
 
-def testTLSversion(self):
-m = self.store.get_metric('tls_version')
-self.assertEqual(1, m._value)
-self.assertIn('version=TLSv1.2', m._labelpairs)
+def testTLS(self):
+s = self.store.get_samples('xcps_tls')
+labels, count = s[0][0], s[0][1]
+expected = [
+'version=TLSv1.2',
+'key_exchange=X25519',
+'auth=ECDSA',
+'cipher=CHACHA20-POLY1305',
+]
+for value in expected:
+self.assertIn(value, labels)
 
-def testTLSKeyExchange(self):
-m = self.store.get_metric('tls_key_exchange')
-self.assertEqual(1, m._value)
-self.assertIn('type=X25519', m._labelpairs)
-
-def testTLSAuth(self):
-m = self.store.get_metric('tls_auth')
-self.assertEqual(1, m._value)
-self.assertIn('type=ECDSA', m._labelpairs)
-

[MediaWiki-commits] [Gerrit] operations/puppet[production]: tox: run mtail tests

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/394552 )

Change subject: tox: run mtail tests
..


tox: run mtail tests

Bug: T181794
Depends-On: I41f9487b57306723bcc0c15a8033f3572a84fa08
Change-Id: I1a13e0c4445060822c732398dfa30bcdaf0ca03c
---
M tox.ini
1 file changed, 4 insertions(+), 1 deletion(-)

Approvals:
  Ema: Verified; Looks good to me, approved



diff --git a/tox.ini b/tox.ini
index 0075b02..62496d1 100644
--- a/tox.ini
+++ b/tox.ini
@@ -2,7 +2,7 @@
 minversion = 1.6
 skipsdist = True
 
-envlist = webperf, commit-message, admin, apache, pep8
+envlist = webperf, commit-message, admin, apache, pep8, mtail
 
 [flake8]
 max-line-length = 100
@@ -56,3 +56,6 @@
 basepython = python2.7
 deps = flake8==3.3.0
 commands = flake8 {posargs}
+
+[testenv:mtail]
+commands = nosetests modules/mtail/files

-- 
To view, visit https://gerrit.wikimedia.org/r/394552
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I1a13e0c4445060822c732398dfa30bcdaf0ca03c
Gerrit-PatchSet: 5
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Hashar 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Add PUT to list of allowed methods for text varnish

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/398197 )

Change subject: Add PUT to list of allowed methods for text varnish
..


Add PUT to list of allowed methods for text varnish

HTTP PUT is used by the reading lists REST API (on scb) to update
list metadata.

Bug: T182825
Change-Id: I6f7fba56731da3d72dab34f8eb6b3eebc57f4879
---
M modules/profile/manifests/cache/text.pp
1 file changed, 1 insertion(+), 0 deletions(-)

Approvals:
  Ema: Verified; Looks good to me, approved



diff --git a/modules/profile/manifests/cache/text.pp 
b/modules/profile/manifests/cache/text.pp
index 367c248..9df5d94 100644
--- a/modules/profile/manifests/cache/text.pp
+++ b/modules/profile/manifests/cache/text.pp
@@ -49,6 +49,7 @@
 }
 
 $common_vcl_config = {
+'allowed_methods'  => '^(GET|HEAD|OPTIONS|POST|PURGE|PUT)$',
 'purge_host_regex' => 
$::profile::cache::base::purge_host_not_upload_re,
 'static_host'  => $static_host,
 'top_domain'   => $top_domain,

-- 
To view, visit https://gerrit.wikimedia.org/r/398197
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I6f7fba56731da3d72dab34f8eb6b3eebc57f4879
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gergő Tisza 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: prometheus: add mtail to varnish-upload job

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/397851 )

Change subject: prometheus: add mtail to varnish-upload job
..


prometheus: add mtail to varnish-upload job

Bug: T177199
Change-Id: Ic2632cb819e3cfeb281eb776b2ebaabafa30947d
---
M modules/role/manifests/prometheus/ops.pp
1 file changed, 6 insertions(+), 0 deletions(-)

Approvals:
  Ema: Verified; Looks good to me, approved



diff --git a/modules/role/manifests/prometheus/ops.pp 
b/modules/role/manifests/prometheus/ops.pp
index 226cc09..b914de8 100644
--- a/modules/role/manifests/prometheus/ops.pp
+++ b/modules/role/manifests/prometheus/ops.pp
@@ -232,6 +232,12 @@
 class_name => 'role::cache::misc',
 port   => '3903',
 }
+prometheus::class_config{ "varnish-upload_mtail_${::site}":
+dest   => "${targets_path}/varnish-upload_mtail_${::site}.yaml",
+site   => $::site,
+class_name => 'role::cache::upload',
+port   => '3903',
+}
 
 # Job definition for memcache_exporter
 $memcached_jobs = [

-- 
To view, visit https://gerrit.wikimedia.org/r/397851
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ic2632cb819e3cfeb281eb776b2ebaabafa30947d
Gerrit-PatchSet: 4
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnishxcps.mtail: use prometheus labels

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/397876 )

Change subject: varnishxcps.mtail: use prometheus labels
..

varnishxcps.mtail: use prometheus labels

Turn tls version, key_exchange, auth, and cipher into labels. By doing
this, we make it possible to use various combinations of those stats
together and answer questions such as: how many TLSv1.2 connections used
x25519 for key exchange?

Leave HTTP2, TLS session reuse and TLS full cipher stats as separate
counters.

Bug: T177199
Change-Id: I79c67927cb86b3bbbe0b8dccaba7c767b5296a7a
---
M modules/mtail/files/programs/varnishxcps.mtail
1 file changed, 9 insertions(+), 25 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/76/397876/1

diff --git a/modules/mtail/files/programs/varnishxcps.mtail 
b/modules/mtail/files/programs/varnishxcps.mtail
index b74f9cb..5ed7d84 100644
--- a/modules/mtail/files/programs/varnishxcps.mtail
+++ b/modules/mtail/files/programs/varnishxcps.mtail
@@ -1,36 +1,20 @@
-counter tls_h2
-counter tls_version by version
-counter tls_sess_reused
-counter tls_key_exchange by type
-counter tls_auth by type
-counter tls_cipher by name
-counter tls_full_cipher by name
+counter xcps_h2
+counter xcps_tls_sess_reused
+counter xcps_tls by version, key_exchange, auth, cipher
+counter xcps_tls_full_cipher by name
 
 /\th2 1\t/ {
-tls_h2++
-}
-
-/\ttls_version (?PTLSv[0-9\.]+)\t/ {
-tls_version[$version]++
+xcps_h2++
 }
 
 /\tsession_reused 1\t/ {
-tls_sess_reused++
+xcps_tls_sess_reused++
 }
 
-/\tkey_exchange (?P[a-zA-Z0-9-_]+)\t/ {
-tls_key_exchange[$type]++
-}
-
-/\tauth (?P[a-zA-Z0-9-_]+)\t/ {
-tls_auth[$type]++
-}
-
-
-/\tcipher (?P[a-zA-Z0-9-_]+)\t/ {
-tls_cipher[$name]++
+/\ttls_version (?PTLSv[0-9\.]+)\t.*\tkey_exchange 
(?P[a-zA-Z0-9-_]+)\tauth (?P[a-zA-Z0-9-_]+)\tcipher 
(?P[a-zA-Z0-9-_]+)\t/ {
+xcps_tls[$version][$key_exchange][$auth][$cipher]++
 }
 
 /\tfull_cipher (?P[a-zA-Z0-9-_]+)\t/ {
-tls_full_cipher[$name]++
+xcps_tls_full_cipher[$name]++
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/397876
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I79c67927cb86b3bbbe0b8dccaba7c767b5296a7a
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache: install varnishxcps.mtail

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/397831 )

Change subject: cache: install varnishxcps.mtail
..


cache: install varnishxcps.mtail

Bug: T177199
Change-Id: Ie18a00809f86ea815ee03ea00857c6d5eb0bce96
---
M modules/varnish/manifests/logging/xcps.pp
1 file changed, 4 insertions(+), 0 deletions(-)

Approvals:
  Ema: Looks good to me, approved
  jenkins-bot: Verified
  Filippo Giunchedi: Looks good to me, but someone else must approve



diff --git a/modules/varnish/manifests/logging/xcps.pp 
b/modules/varnish/manifests/logging/xcps.pp
index 51b136e..e353d21 100644
--- a/modules/varnish/manifests/logging/xcps.pp
+++ b/modules/varnish/manifests/logging/xcps.pp
@@ -44,4 +44,8 @@
 description  => 'Varnish traffic logger - varnishxcps',
 nrpe_command => '/usr/lib/nagios/plugins/check_procs -w 1:1 -a 
"/usr/local/bin/varnishxcps" -u root',
 }
+
+mtail::program { 'varnishxcps':
+source => 'puppet:///modules/mtail/programs/varnishxcps.mtail',
+}
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/397831
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ie18a00809f86ea815ee03ea00857c6d5eb0bce96
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache: install varnishxcps.mtail

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/397831 )

Change subject: cache: install varnishxcps.mtail
..

cache: install varnishxcps.mtail

Bug: T177199
Change-Id: Ie18a00809f86ea815ee03ea00857c6d5eb0bce96
---
M modules/varnish/manifests/logging/xcps.pp
1 file changed, 4 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/31/397831/1

diff --git a/modules/varnish/manifests/logging/xcps.pp 
b/modules/varnish/manifests/logging/xcps.pp
index 51b136e..e353d21 100644
--- a/modules/varnish/manifests/logging/xcps.pp
+++ b/modules/varnish/manifests/logging/xcps.pp
@@ -44,4 +44,8 @@
 description  => 'Varnish traffic logger - varnishxcps',
 nrpe_command => '/usr/lib/nagios/plugins/check_procs -w 1:1 -a 
"/usr/local/bin/varnishxcps" -u root',
 }
+
+mtail::program { 'varnishxcps':
+source => 'puppet:///modules/mtail/programs/varnishxcps.mtail',
+}
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/397831
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ie18a00809f86ea815ee03ea00857c6d5eb0bce96
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: mtail: port varnishxcps

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/395578 )

Change subject: mtail: port varnishxcps
..


mtail: port varnishxcps

Bug: T177199
Change-Id: I79b5cefa202db3e6137904a2b9b411f50e3781e1
---
A modules/mtail/files/programs/varnishxcps.mtail
M modules/mtail/files/test/logs/varnish.test
M modules/mtail/files/test/varnish_test.py
M modules/varnish/files/varnishmtail
4 files changed, 92 insertions(+), 3 deletions(-)

Approvals:
  Ema: Verified; Looks good to me, approved
  Filippo Giunchedi: Looks good to me, but someone else must approve



diff --git a/modules/mtail/files/programs/varnishxcps.mtail 
b/modules/mtail/files/programs/varnishxcps.mtail
new file mode 100644
index 000..b74f9cb
--- /dev/null
+++ b/modules/mtail/files/programs/varnishxcps.mtail
@@ -0,0 +1,36 @@
+counter tls_h2
+counter tls_version by version
+counter tls_sess_reused
+counter tls_key_exchange by type
+counter tls_auth by type
+counter tls_cipher by name
+counter tls_full_cipher by name
+
+/\th2 1\t/ {
+tls_h2++
+}
+
+/\ttls_version (?PTLSv[0-9\.]+)\t/ {
+tls_version[$version]++
+}
+
+/\tsession_reused 1\t/ {
+tls_sess_reused++
+}
+
+/\tkey_exchange (?P[a-zA-Z0-9-_]+)\t/ {
+tls_key_exchange[$type]++
+}
+
+/\tauth (?P[a-zA-Z0-9-_]+)\t/ {
+tls_auth[$type]++
+}
+
+
+/\tcipher (?P[a-zA-Z0-9-_]+)\t/ {
+tls_cipher[$name]++
+}
+
+/\tfull_cipher (?P[a-zA-Z0-9-_]+)\t/ {
+tls_full_cipher[$name]++
+}
diff --git a/modules/mtail/files/test/logs/varnish.test 
b/modules/mtail/files/test/logs/varnish.test
index b2d4b53..5cd96e5 100644
--- a/modules/mtail/files/test/logs/varnish.test
+++ b/modules/mtail/files/test/logs/varnish.test
@@ -1,5 +1,5 @@
-url /  cache_status int-front  http_status 301 cache_control - inm -
-url /w/index.php   cache_status hit-front  http_status 304 cache_control 
private, s-maxage=0, max-age=0, must-revalidate   inm -
+url /  cache_status int-front  http_status 301 cache_control - inm -   h2 0
tls_version session_reused 0key_exchangeauthcipher  
full_cipher 
+url /w/index.php   cache_status hit-front  http_status 304 cache_control 
private, s-maxage=0, max-age=0, must-revalidate   inm -   h2 1tls_version 
TLSv1.2 session_reused 1key_exchange X25519 auth ECDSA  
cipher CHACHA20-POLY1305full_cipher ECDHE-ECDSA-CHACHA20-POLY1305   
 url 
/api/rest_v1/metrics/pageviews/per-article/fr.wikipedia.org/all-access/user/S%C3%A9lection_sexuelle/daily/20171126/20171201
cache_status hit-front  http_status 200 cache_control s-maxage=86400, 
max-age=86400 inm -
 url /w/load.phpcache_status hit-front  http_status 200 cache_control 
public, max-age=2592000, s-maxage=2592000 inm -
 url /w/load.phpcache_status hit-front  http_status 200 cache_control 
public, max-age=2592000, s-maxage=2592000 inm -
diff --git a/modules/mtail/files/test/varnish_test.py 
b/modules/mtail/files/test/varnish_test.py
index ec04995..ccc0d15 100644
--- a/modules/mtail/files/test/varnish_test.py
+++ b/modules/mtail/files/test/varnish_test.py
@@ -38,3 +38,43 @@
 m = self.store.get_metric('varnish_thumbnails')
 self.assertEquals(2, m._value)
 self.assertIn('status=200', m._labelpairs)
+
+
+class VarnishXcpsTest(unittest.TestCase):
+def setUp(self):
+self.store = mtail_store.MtailMetricStore(
+os.path.join(test_dir, '../programs/varnishxcps.mtail'),
+os.path.join(test_dir, 'logs/varnish.test'))
+
+def testH2(self):
+m = self.store.get_metric('tls_h2')
+self.assertEqual(1, m._value)
+
+def testReusedSessions(self):
+m = self.store.get_metric('tls_sess_reused')
+self.assertEqual(1, m._value)
+
+def testTLSversion(self):
+m = self.store.get_metric('tls_version')
+self.assertEqual(1, m._value)
+self.assertIn('version=TLSv1.2', m._labelpairs)
+
+def testTLSKeyExchange(self):
+m = self.store.get_metric('tls_key_exchange')
+self.assertEqual(1, m._value)
+self.assertIn('type=X25519', m._labelpairs)
+
+def testTLSAuth(self):
+m = self.store.get_metric('tls_auth')
+self.assertEqual(1, m._value)
+self.assertIn('type=ECDSA', m._labelpairs)
+
+def testTLSCipher(self):
+m = self.store.get_metric('tls_cipher')
+self.assertEqual(1, m._value)
+self.assertIn('name=CHACHA20-POLY1305', m._labelpairs)
+
+def testTLSFullCipher(self):
+m = self.store.get_metric('tls_full_cipher')
+self.assertEqual(1, m._value)
+self.assertIn('name=ECDHE-ECDSA-CHACHA20-POLY1305', m._labelpairs)
diff --git a/modules/varnish/files/varnishmtail 
b/modules/varnish/files/varnishmtail
index 14c56ba..ba2de95 100644
--- a/modules/varnish/files/varnishmtail
+++ b/modules/varnish/files/varnishmtail
@@ -4,7 +4,20 @@
 
 PROGS="${1:-/etc/mtail}"
 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: vcl: add hostname/layer info to syntethic healthcheck response

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/393251 )

Change subject: vcl: add hostname/layer info to syntethic healthcheck response
..


vcl: add hostname/layer info to syntethic healthcheck response

Varnish currently returns an empty 200 OK as healthcheck response. Add
hostname/layer informationon to the body.

Change-Id: I3d0a8f4f75a92105e1192925da3beafa5cdfcc0e
---
A modules/varnish/files/tests/misc/13-healtchecks.vtc
M modules/varnish/templates/vcl/wikimedia-backend.vcl.erb
M modules/varnish/templates/vcl/wikimedia-common.inc.vcl.erb
M modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
4 files changed, 29 insertions(+), 1 deletion(-)

Approvals:
  Ema: Verified; Looks good to me, approved



diff --git a/modules/varnish/files/tests/misc/13-healtchecks.vtc 
b/modules/varnish/files/tests/misc/13-healtchecks.vtc
new file mode 100644
index 000..ca80fa5
--- /dev/null
+++ b/modules/varnish/files/tests/misc/13-healtchecks.vtc
@@ -0,0 +1,18 @@
+varnishtest "Varnish healthchecks"
+
+server s1 {} -start
+
+varnish v1 -arg "-p vcc_err_unref=false" -vcl+backend {
+backend vtc_backend {
+.host = "${s1_addr}"; .port = "${s1_port}";
+}
+
+include "/usr/share/varnish/tests/wikimedia_misc-backend.vcl";
+} -start
+
+client c1 {
+txreq -hdr "Host: varnishcheck.wikimedia.org"
+rxresp
+expect resp.status == 200
+expect resp.body ~ "Varnish backend running on"
+} -run
diff --git a/modules/varnish/templates/vcl/wikimedia-backend.vcl.erb 
b/modules/varnish/templates/vcl/wikimedia-backend.vcl.erb
index 59236fb..2d94a8e 100644
--- a/modules/varnish/templates/vcl/wikimedia-backend.vcl.erb
+++ b/modules/varnish/templates/vcl/wikimedia-backend.vcl.erb
@@ -220,6 +220,7 @@
if (resp.status >= 400) {
call synth_errorpage;
}
+   call wm_common_synth;
return (deliver);
 }
 
diff --git a/modules/varnish/templates/vcl/wikimedia-common.inc.vcl.erb 
b/modules/varnish/templates/vcl/wikimedia-common.inc.vcl.erb
index a4d32e0..2a14a7c 100644
--- a/modules/varnish/templates/vcl/wikimedia-common.inc.vcl.erb
+++ b/modules/varnish/templates/vcl/wikimedia-common.inc.vcl.erb
@@ -346,7 +346,7 @@
}
 
if ( req.http.host ~ "^varnishcheck" ) {
-   return (synth(200, "OK"));
+   return (synth(200, "healthcheck"));
}
 }
 
@@ -382,6 +382,13 @@
set req.http.X-CDIS = "pass";
 }
 
+sub wm_common_synth {
+   if (resp.reason == "healthcheck") {
+   set resp.reason = "OK";
+   synthetic("Varnish <%= @inst %> running on <%= @hostname %> is 
up");
+   }
+}
+
 sub wm_common_backend_response {
// This prevents the application layer from setting this in a response.
// We'll be setting this same variable internally in VCL in hit-for-pass
diff --git a/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb 
b/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
index 3719c2f..e8c0153 100644
--- a/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
+++ b/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
@@ -571,6 +571,8 @@
call synth_errorpage;
}
}
+
+   call wm_common_synth;
return (deliver);
 }
 

-- 
To view, visit https://gerrit.wikimedia.org/r/393251
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I3d0a8f4f75a92105e1192925da3beafa5cdfcc0e
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: mtail: port varnishxcps

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/395578 )

Change subject: mtail: port varnishxcps
..

mtail: port varnishxcps

Bug: T177199
Change-Id: I79b5cefa202db3e6137904a2b9b411f50e3781e1
---
A modules/mtail/files/programs/varnishxcps.mtail
M modules/mtail/files/test/logs/varnish.test
M modules/mtail/files/test/varnish_test.py
M modules/varnish/files/varnishmtail
4 files changed, 98 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/78/395578/1

diff --git a/modules/mtail/files/programs/varnishxcps.mtail 
b/modules/mtail/files/programs/varnishxcps.mtail
new file mode 100644
index 000..01a3e1d
--- /dev/null
+++ b/modules/mtail/files/programs/varnishxcps.mtail
@@ -0,0 +1,36 @@
+counter tls_h2
+counter tls_version by version
+counter tls_sess_reused
+counter tls_key_exchange by type
+counter tls_auth by type
+counter tls_cipher by name
+counter tls_full_cipher by name
+
+/^h2 1$/ {
+tls_h2++
+}
+
+/^tls_version (?PTLSv1.*)$/ {
+tls_version[$version]++
+}
+
+/^session_reused 1$/ {
+tls_sess_reused++
+}
+
+/^key_exchange (?P[a-zA-Z0-9]+)$/ {
+tls_key_exchange[$type]++
+}
+
+/^auth (?P[a-zA-Z0-9]+)$/ {
+tls_auth[$type]++
+}
+
+
+/^cipher (?P[-a-zA-Z0-9]+)$/ {
+tls_cipher[$name]++
+}
+
+/^full_cipher (?P[-a-zA-Z0-9]+)$/ {
+tls_full_cipher[$name]++
+}
diff --git a/modules/mtail/files/test/logs/varnish.test 
b/modules/mtail/files/test/logs/varnish.test
index b2d4b53..60fc85a 100644
--- a/modules/mtail/files/test/logs/varnish.test
+++ b/modules/mtail/files/test/logs/varnish.test
@@ -1,5 +1,19 @@
 url /  cache_status int-front  http_status 301 cache_control - inm -
+h2 0
+tls_version
+session_reused 0
+key_exchange
+auth
+cipher
+full_cipher
 url /w/index.php   cache_status hit-front  http_status 304 cache_control 
private, s-maxage=0, max-age=0, must-revalidate   inm -
+h2 1
+tls_version TLSv1.2
+session_reused 1
+key_exchange X25519
+auth ECDSA
+cipher CHACHA20-POLY1305
+full_cipher ECDHE-ECDSA-CHACHA20-POLY1305
 url 
/api/rest_v1/metrics/pageviews/per-article/fr.wikipedia.org/all-access/user/S%C3%A9lection_sexuelle/daily/20171126/20171201
cache_status hit-front  http_status 200 cache_control s-maxage=86400, 
max-age=86400 inm -
 url /w/load.phpcache_status hit-front  http_status 200 cache_control 
public, max-age=2592000, s-maxage=2592000 inm -
 url /w/load.phpcache_status hit-front  http_status 200 cache_control 
public, max-age=2592000, s-maxage=2592000 inm -
diff --git a/modules/mtail/files/test/varnish_test.py 
b/modules/mtail/files/test/varnish_test.py
index ec04995..ccc0d15 100644
--- a/modules/mtail/files/test/varnish_test.py
+++ b/modules/mtail/files/test/varnish_test.py
@@ -38,3 +38,43 @@
 m = self.store.get_metric('varnish_thumbnails')
 self.assertEquals(2, m._value)
 self.assertIn('status=200', m._labelpairs)
+
+
+class VarnishXcpsTest(unittest.TestCase):
+def setUp(self):
+self.store = mtail_store.MtailMetricStore(
+os.path.join(test_dir, '../programs/varnishxcps.mtail'),
+os.path.join(test_dir, 'logs/varnish.test'))
+
+def testH2(self):
+m = self.store.get_metric('tls_h2')
+self.assertEqual(1, m._value)
+
+def testReusedSessions(self):
+m = self.store.get_metric('tls_sess_reused')
+self.assertEqual(1, m._value)
+
+def testTLSversion(self):
+m = self.store.get_metric('tls_version')
+self.assertEqual(1, m._value)
+self.assertIn('version=TLSv1.2', m._labelpairs)
+
+def testTLSKeyExchange(self):
+m = self.store.get_metric('tls_key_exchange')
+self.assertEqual(1, m._value)
+self.assertIn('type=X25519', m._labelpairs)
+
+def testTLSAuth(self):
+m = self.store.get_metric('tls_auth')
+self.assertEqual(1, m._value)
+self.assertIn('type=ECDSA', m._labelpairs)
+
+def testTLSCipher(self):
+m = self.store.get_metric('tls_cipher')
+self.assertEqual(1, m._value)
+self.assertIn('name=CHACHA20-POLY1305', m._labelpairs)
+
+def testTLSFullCipher(self):
+m = self.store.get_metric('tls_full_cipher')
+self.assertEqual(1, m._value)
+self.assertIn('name=ECDHE-ECDSA-CHACHA20-POLY1305', m._labelpairs)
diff --git a/modules/varnish/files/varnishmtail 
b/modules/varnish/files/varnishmtail
index 14c56ba..2da382b 100644
--- a/modules/varnish/files/varnishmtail
+++ b/modules/varnish/files/varnishmtail
@@ -4,7 +4,14 @@
 
 PROGS="${1:-/etc/mtail}"
 
-FMT='url %U\tcache_status %{X-Cache-Status}o\thttp_status %s\tcache_control 
%{Cache-Control}o\tinm %{If-None-Match}i'
+FMT='url %U\tcache_status %{X-Cache-Status}o\thttp_status %s\tcache_control 
%{Cache-Control}o\tinm %{If-None-Match}i
+h2 %{VCL_Log:CP-HTTP2}x
+tls_version %{VCL_Log:CP-TLS-Version}x
+session_reused 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: mtail: add varnishmtail tests

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/394597 )

Change subject: mtail: add varnishmtail tests
..


mtail: add varnishmtail tests

Bug: T177199
Change-Id: I7c33a3dc1754e62408b0c450a80018d71d516b34
---
A modules/mtail/files/test/logs/varnish.test
A modules/mtail/files/test/varnish_test.py
2 files changed, 49 insertions(+), 0 deletions(-)

Approvals:
  Ema: Verified; Looks good to me, approved
  Filippo Giunchedi: Looks good to me, but someone else must approve



diff --git a/modules/mtail/files/test/logs/varnish.test 
b/modules/mtail/files/test/logs/varnish.test
new file mode 100644
index 000..b2d4b53
--- /dev/null
+++ b/modules/mtail/files/test/logs/varnish.test
@@ -0,0 +1,9 @@
+url /  cache_status int-front  http_status 301 cache_control - inm -
+url /w/index.php   cache_status hit-front  http_status 304 cache_control 
private, s-maxage=0, max-age=0, must-revalidate   inm -
+url 
/api/rest_v1/metrics/pageviews/per-article/fr.wikipedia.org/all-access/user/S%C3%A9lection_sexuelle/daily/20171126/20171201
cache_status hit-front  http_status 200 cache_control s-maxage=86400, 
max-age=86400 inm -
+url /w/load.phpcache_status hit-front  http_status 200 cache_control 
public, max-age=2592000, s-maxage=2592000 inm -
+url /w/load.phpcache_status hit-front  http_status 200 cache_control 
public, max-age=2592000, s-maxage=2592000 inm -
+url /w/load.phpcache_status hit-front  http_status 304 cache_control 
public, max-age=300, s-maxage=300 inm W/\"1adp3u3\"
+url /wikipedia/commons/5/51/Tang_Shaoyi.jpgcache_status int-front  
http_status 301 cache_control - inm -
+url 
/wikipedia/commons/thumb/5/51/Flag_of_North_Korea.svg/250px-Flag_of_North_Korea.svg.png
cache_status hit-front  http_status 200 cache_control - inm -
+url /wikipedia/en/thumb/f/fd/Portal-puzzle.svg/16px-Portal-puzzle.svg.png  
cache_status hit-front  http_status 200 cache_control - inm -
diff --git a/modules/mtail/files/test/varnish_test.py 
b/modules/mtail/files/test/varnish_test.py
new file mode 100644
index 000..ec04995
--- /dev/null
+++ b/modules/mtail/files/test/varnish_test.py
@@ -0,0 +1,40 @@
+import mtail_store
+import unittest
+import os
+
+test_dir = os.path.join(os.path.dirname(__file__))
+
+
+class VarnishXcacheTest(unittest.TestCase):
+def setUp(self):
+self.store = mtail_store.MtailMetricStore(
+os.path.join(test_dir, '../programs/varnishxcache.mtail'),
+os.path.join(test_dir, 'logs/varnish.test'))
+
+def testCacheStatus(self):
+m = self.store.get_metric('varnish_x_cache')
+self.assertEqual(2, m._value)
+self.assertIn('x_cache=int-front', m._labelpairs)
+
+
+class VarnishRlsTest(unittest.TestCase):
+def setUp(self):
+self.store = mtail_store.MtailMetricStore(
+os.path.join(test_dir, '../programs/varnishrls.mtail'),
+os.path.join(test_dir, 'logs/varnish.test'))
+
+def testIfNoneMatch(self):
+m = self.store.get_metric('varnish_resourceloader_inm')
+self.assertEquals(m._value, 1)
+
+
+class VarnishMediaTest(unittest.TestCase):
+def setUp(self):
+self.store = mtail_store.MtailMetricStore(
+os.path.join(test_dir, '../programs/varnishmedia.mtail'),
+os.path.join(test_dir, 'logs/varnish.test'))
+
+def testThumbnails(self):
+m = self.store.get_metric('varnish_thumbnails')
+self.assertEquals(2, m._value)
+self.assertIn('status=200', m._labelpairs)

-- 
To view, visit https://gerrit.wikimedia.org/r/394597
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I7c33a3dc1754e62408b0c450a80018d71d516b34
Gerrit-PatchSet: 5
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Volans 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: VCL: log TLS information to VSM

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/388064 )

Change subject: VCL: log TLS information to VSM
..


VCL: log TLS information to VSM

Parse X-Connection-Properties and log TLS information to VSM. The fields
extracted here are the same that varnishxcps generates.

We can then access the relevant entries from VSM with varnishncsa format
strings such as '%{VCL_Log:CP-Key-Exchange}x'

Bug: T177199
Change-Id: I692fc914b5032912efef43bdaa4cf78121f5014a
---
A modules/varnish/files/tests/upload/16-x-connection-properties.vtc
M modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
2 files changed, 130 insertions(+), 0 deletions(-)

Approvals:
  Ema: Looks good to me, approved
  BBlack: Looks good to me, but someone else must approve
  jenkins-bot: Verified



diff --git a/modules/varnish/files/tests/upload/16-x-connection-properties.vtc 
b/modules/varnish/files/tests/upload/16-x-connection-properties.vtc
new file mode 100644
index 000..eaa4037
--- /dev/null
+++ b/modules/varnish/files/tests/upload/16-x-connection-properties.vtc
@@ -0,0 +1,77 @@
+varnishtest "X-Connection-Properties"
+
+server s1 {
+rxreq
+expect req.url == "/1"
+
+expect req.http.X-CP-HTTP2 == "0"
+expect req.http.X-CP-TLS-Version == "TLSv1.2"
+expect req.http.X-CP-TLS-Session-Reused == 0
+expect req.http.X-CP-Key-Exchange == "prime256v1"
+expect req.http.X-CP-Auth == "ECDSA"
+expect req.http.X-CP-Cipher == "AES256-GCM-SHA384"
+expect req.http.X-CP-Full-Cipher == "ECDHE-ECDSA-AES256-GCM-SHA384"
+
+txresp
+
+rxreq
+expect req.url == "/2"
+
+expect req.http.X-CP-HTTP2 == "0"
+expect req.http.X-CP-TLS-Version == "TLSv1"
+expect req.http.X-CP-TLS-Session-Reused == 1
+expect req.http.X-CP-Key-Exchange == "prime256v1"
+expect req.http.X-CP-Auth == "ECDSA"
+expect req.http.X-CP-Cipher == "AES128-SHA"
+expect req.http.X-CP-Full-Cipher == "ECDHE-ECDSA-AES128-SHA"
+
+txresp
+
+rxreq
+expect req.url == "/3"
+
+expect req.http.X-CP-HTTP2 == "1"
+expect req.http.X-CP-TLS-Version == "TLSv1.2"
+expect req.http.X-CP-TLS-Session-Reused == 1
+expect req.http.X-CP-Key-Exchange == "X25519"
+expect req.http.X-CP-Auth == "ECDSA"
+expect req.http.X-CP-Cipher == "AES256-GCM-SHA384"
+expect req.http.X-CP-Full-Cipher == "ECDHE-ECDSA-AES256-GCM-SHA384"
+
+txresp
+
+rxreq
+expect req.url == "/4"
+
+expect req.http.X-CP-HTTP2 == "0"
+expect req.http.X-CP-TLS-Version == "TLSv1"
+expect req.http.X-CP-TLS-Session-Reused == 1
+expect req.http.X-CP-Key-Exchange == "RSA"
+expect req.http.X-CP-Auth == "RSA"
+expect req.http.X-CP-Cipher == "AES128-SHA"
+expect req.http.X-CP-Full-Cipher == "AES128-SHA"
+
+txresp
+} -start
+
+varnish v1 -arg "-p vcc_err_unref=false -p vcc_allow_inline_c=true" 
-vcl+backend {
+backend vtc_backend {
+.host = "${s1_addr}"; .port = "${s1_port}";
+}
+
+include "/usr/share/varnish/tests/wikimedia_upload-frontend.vcl";
+} -start
+
+client c1 {
+txreq -url "/1" -hdr "X-Forwarded-Proto: https" -hdr "Host: 
upload.wikimedia.org" -hdr "X-Connection-Properties: H2=0; SSR=0; SSL=TLSv1.2; 
C=ECDHE-ECDSA-AES256-GCM-SHA384; EC=prime256v1;"
+rxresp
+
+txreq -url "/2" -hdr "X-Forwarded-Proto: https" -hdr "Host: 
upload.wikimedia.org" -hdr "X-Connection-Properties: H2=0; SSR=1; SSL=TLSv1; 
C=ECDHE-ECDSA-AES128-SHA; EC=prime256v1;"
+rxresp
+
+txreq -url "/3" -hdr "X-Forwarded-Proto: https" -hdr "Host: 
upload.wikimedia.org" -hdr "X-Connection-Properties: H2=1; SSR=1; SSL=TLSv1.2; 
C=ECDHE-ECDSA-AES256-GCM-SHA384; EC=X25519;"
+rxresp
+
+txreq -url "/4" -hdr "X-Forwarded-Proto: https" -hdr "Host: 
upload.wikimedia.org" -hdr "X-Connection-Properties: H2=0; SSR=1; SSL=TLSv1; 
C=AES128-SHA; EC=UNDEF;"
+rxresp
+} -run
diff --git a/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb 
b/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
index a3f5826..3719c2f 100644
--- a/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
+++ b/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
@@ -282,6 +282,59 @@
call cluster_fe_recv_pre_purge;
call wm_common_recv_purge;
 
+   set req.http.X-CP-HTTP2 = regsub(req.http.X-Connection-Properties, 
"^H2=([01]);.*", "\1");
+
+   set req.http.X-CP-TLS-Version = 
regsub(req.http.X-Connection-Properties, ".* SSL=(TLSv1(\.[123])?);.*", "\1");
+
+   set req.http.X-CP-TLS-Session-Reused = 
regsub(req.http.X-Connection-Properties, ".* SSR=([01]);.*", "\1");
+
+   set req.http.X-CP-Key-Exchange = 
regsub(req.http.X-Connection-Properties, ".* EC=([A-Za-z0-9]+);.*", "\1");
+
+   set req.http.X-CP-Full-Cipher = 
regsub(req.http.X-Connection-Properties, ".* C=([A-Z0-9-]+);.*", "\1");
+
+   set req.http.X-CP-Auth = req.http.X-CP-Full-Cipher;
+
+   if (req.http.X-CP-Auth ~ 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnish: prometheus equivalent of statsd metrics daemons

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/394543 )

Change subject: varnish: prometheus equivalent of statsd metrics daemons
..


varnish: prometheus equivalent of statsd metrics daemons

We are currently running a few python scripts to extract specific
varnish statistics and send them to statsd. Every script starts and
reads from a varnishncsa process.

The goal of producing such statistics and making them available to
prometheus can be accomplished with a single varnishncsa instance and
mtail.

This commit introduces the following mtail scripts:

- varnishxcache.mtail
- varnishrls.mtail
- varnishmedia.mtail

A script called varnishmtail, also introduced in this commit along with
its systemd unit, is responsible for calling varnishncsa with the right
parameters and format string, and piping its output to mtail.

Disable standard mtail daemon startup in hiera, given that we explicitly
run it ourselves.

Bug: T177199
Change-Id: I31115573a5d7f43268eef3a1bcee92e18d5fa957
---
M hieradata/role/common/cache/canary.yaml
M hieradata/role/common/cache/misc.yaml
M hieradata/role/common/cache/text.yaml
M hieradata/role/common/cache/upload.yaml
A modules/mtail/files/programs/varnishmedia.mtail
A modules/mtail/files/programs/varnishrls.mtail
A modules/mtail/files/programs/varnishxcache.mtail
A modules/varnish/files/varnishmtail
M modules/varnish/manifests/logging.pp
M modules/varnish/manifests/logging/media.pp
M modules/varnish/manifests/logging/rls.pp
M modules/varnish/manifests/logging/xcache.pp
A modules/varnish/templates/initscripts/varnishmtail.systemd.erb
13 files changed, 80 insertions(+), 4 deletions(-)

Approvals:
  Ema: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/role/common/cache/canary.yaml 
b/hieradata/role/common/cache/canary.yaml
index a6356ec..cc92afc 100644
--- a/hieradata/role/common/cache/canary.yaml
+++ b/hieradata/role/common/cache/canary.yaml
@@ -10,6 +10,7 @@
 prometheus::node_exporter::collectors_extra:
   - qdisc
   - meminfo_numa
+mtail::ensure: 'stopped'
 # The contents of this hash control our DC->DC routing for varnish backend
 # daemons.  There should be a key for every cache datacenter.  The values must
 # be a core datacenter (eqiad or codfw), or at least must lead indirectly to
diff --git a/hieradata/role/common/cache/misc.yaml 
b/hieradata/role/common/cache/misc.yaml
index 6d4bbd0..461c2a5 100644
--- a/hieradata/role/common/cache/misc.yaml
+++ b/hieradata/role/common/cache/misc.yaml
@@ -7,6 +7,7 @@
 prometheus::node_exporter::collectors_extra:
   - qdisc
   - meminfo_numa
+mtail::ensure: 'stopped'
 # note this only affects tlsproxy now, should be moved to param there...
 cache::websocket_support: true
 # The contents of this hash control our DC->DC routing for varnish backend
diff --git a/hieradata/role/common/cache/text.yaml 
b/hieradata/role/common/cache/text.yaml
index 6c0b1d4..a50aa85 100644
--- a/hieradata/role/common/cache/text.yaml
+++ b/hieradata/role/common/cache/text.yaml
@@ -6,6 +6,7 @@
 prometheus::node_exporter::collectors_extra:
   - qdisc
   - meminfo_numa
+mtail::ensure: 'stopped'
 standard::has_ganglia: false
 # The contents of this hash control our DC->DC routing for varnish backend
 # daemons.  There should be a key for every cache datacenter.  The values must
diff --git a/hieradata/role/common/cache/upload.yaml 
b/hieradata/role/common/cache/upload.yaml
index 39e7844..88606de 100644
--- a/hieradata/role/common/cache/upload.yaml
+++ b/hieradata/role/common/cache/upload.yaml
@@ -6,6 +6,7 @@
 prometheus::node_exporter::collectors_extra:
   - qdisc
   - meminfo_numa
+mtail::ensure: 'stopped'
 standard::has_ganglia: false
 cache::tune_for_media: true
 # The contents of this hash control our DC->DC routing for varnish backend
diff --git a/modules/mtail/files/programs/varnishmedia.mtail 
b/modules/mtail/files/programs/varnishmedia.mtail
new file mode 100644
index 000..885af9e
--- /dev/null
+++ b/modules/mtail/files/programs/varnishmedia.mtail
@@ -0,0 +1,5 @@
+counter varnish_thumbnails by status
+
+/^url .*\/thumb\/.*\thttp_status (?P.*)\tcache_control/ {
+varnish_thumbnails[$http_status]++
+}
diff --git a/modules/mtail/files/programs/varnishrls.mtail 
b/modules/mtail/files/programs/varnishrls.mtail
new file mode 100644
index 000..0705e44
--- /dev/null
+++ b/modules/mtail/files/programs/varnishrls.mtail
@@ -0,0 +1,7 @@
+counter varnish_resourceloader_inm
+
+/^url \/w\/load.php.*\tinm (?P.*)$/ {
+$inm != "-" {
+varnish_resourceloader_inm++
+}
+}
diff --git a/modules/mtail/files/programs/varnishxcache.mtail 
b/modules/mtail/files/programs/varnishxcache.mtail
new file mode 100644
index 000..df4a197
--- /dev/null
+++ b/modules/mtail/files/programs/varnishxcache.mtail
@@ -0,0 +1,5 @@
+counter varnish_x_cache by x_cache
+
+/^.*\tcache_status (?P.*)\thttp_status/ {
+varnish_x_cache[$x_cache]++
+}
diff 

[MediaWiki-commits] [Gerrit] operations/puppet[production]: mtail: add 'ensure' parameter, remove 'enabled'

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/394957 )

Change subject: mtail: add 'ensure' parameter, remove 'enabled'
..


mtail: add 'ensure' parameter, remove 'enabled'

Add a new parameter called 'ensure': if set to 'running', the default,
puppet will make sure that the mtail service is running. The parameter
can otherwise be set to the self-explanatory value of 'stopped'.

Get rid of 'enabled', only used by the SysV init script.

Change-Id: I0f785e13de989d0ffa4c612c5ad44c609f5b531d
---
M modules/mtail/manifests/init.pp
M modules/mtail/templates/default.erb
2 files changed, 10 insertions(+), 8 deletions(-)

Approvals:
  Ema: Verified; Looks good to me, approved
  Filippo Giunchedi: Looks good to me, but someone else must approve



diff --git a/modules/mtail/manifests/init.pp b/modules/mtail/manifests/init.pp
index af1197b..fc3fd1d 100644
--- a/modules/mtail/manifests/init.pp
+++ b/modules/mtail/manifests/init.pp
@@ -13,21 +13,21 @@
 # [*graphite_hostport*]
 #   Also send metrics via graphite line-oriented protocol to this host:port.
 #
-# [*enabled*]
-#   Whether to start mtail at boot
+# [*ensure*]
+#   Whether mtail should be running or stopped.
 
 class mtail (
   $logs = ['/var/log/syslog'],
   $port = '3903',
   $graphite_hostport = 'graphite-in.eqiad.wmnet:2003',
   $graphite_prefix = "mtail.${::hostname}.",
-  $enabled = '1',
+  $ensure = 'running',
   $group = 'root',
 ) {
 validate_array($logs)
 validate_re($port, '^[0-9]+$')
 validate_string($graphite_hostport)
-validate_string($enabled)
+validate_re($ensure, '^(running|stopped)$')
 
 require_package('mtail')
 
@@ -41,8 +41,11 @@
 }
 
 systemd::service { 'mtail':
-ensure  => present,
-content => systemd_template('mtail'),
-restart => true,
+ensure => present,
+content=> systemd_template('mtail'),
+restart=> true,
+service_params => {
+ensure => $ensure,
+},
 }
 }
diff --git a/modules/mtail/templates/default.erb 
b/modules/mtail/templates/default.erb
index 0dae7fb..e26726f 100644
--- a/modules/mtail/templates/default.erb
+++ b/modules/mtail/templates/default.erb
@@ -1,5 +1,4 @@
 # Arguments used by SysV init script, systemd service file uses only EXTRA_ARGS
-#ENABLED=<%= @enabled %>
 #GRAPHITE_HOSTPORT=<%= @graphite_hostport %>
 #PORT=<%= @port %>
 #LOGS=<%= @logs.join(',') %>

-- 
To view, visit https://gerrit.wikimedia.org/r/394957
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I0f785e13de989d0ffa4c612c5ad44c609f5b531d
Gerrit-PatchSet: 4
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: vcl: distinguish between hfp and hfm

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/391171 )

Change subject: vcl: distinguish between hfp and hfm
..


vcl: distinguish between hfp and hfm

Convert all miss-once cases (uncacheable+ttl=0) into short-lived HFMs on
Varnish 5. Leave them untouched on Varnish 4.

Change HFP syntax according to the Varnish version in use.

Bug: T180434
Change-Id: Iec92a3d8370e20907b667f31bac00428d3e18cf6
---
M modules/varnish/templates/misc-backend.inc.vcl.erb
M modules/varnish/templates/misc-frontend.inc.vcl.erb
M modules/varnish/templates/text-backend.inc.vcl.erb
M modules/varnish/templates/text-common.inc.vcl.erb
M modules/varnish/templates/text-frontend.inc.vcl.erb
M modules/varnish/templates/upload-backend.inc.vcl.erb
M modules/varnish/templates/upload-common.inc.vcl.erb
M modules/varnish/templates/upload-frontend.inc.vcl.erb
M modules/varnish/templates/vcl/wikimedia-common.inc.vcl.erb
M modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
10 files changed, 61 insertions(+), 12 deletions(-)

Approvals:
  Ema: Verified; Looks good to me, approved
  BBlack: Looks good to me, but someone else must approve
  jenkins-bot: Verified



diff --git a/modules/varnish/templates/misc-backend.inc.vcl.erb 
b/modules/varnish/templates/misc-backend.inc.vcl.erb
index 19259fd..a8eba57 100644
--- a/modules/varnish/templates/misc-backend.inc.vcl.erb
+++ b/modules/varnish/templates/misc-backend.inc.vcl.erb
@@ -18,9 +18,9 @@
 sub cluster_be_backend_response {
 // Do not cache explicit lengths >= ~1GB in backends in general
 if (beresp.http.Content-Length ~ "^[0-9]{10}") {
-   set beresp.http.X-CDIS = "pass";
-set beresp.uncacheable = true;
-return (deliver);
+// HFP
+set beresp.http.X-CDIS = "pass";
+return(pass(beresp.ttl));
 }
 
 <% if not @varnish_testing -%>
diff --git a/modules/varnish/templates/misc-frontend.inc.vcl.erb 
b/modules/varnish/templates/misc-frontend.inc.vcl.erb
index 49581dd..0ee09d8 100644
--- a/modules/varnish/templates/misc-frontend.inc.vcl.erb
+++ b/modules/varnish/templates/misc-frontend.inc.vcl.erb
@@ -50,9 +50,9 @@
 sub cluster_fe_backend_response {
 // hit_for_pass on objects >= ~10MB or no CL
 if (!beresp.http.Content-Length || beresp.http.Content-Length ~ 
"^[0-9]{8}") {
+// HFP
 set beresp.http.X-CDIS = "pass";
-set beresp.uncacheable = true;
-return (deliver);
+return(pass(beresp.ttl));
 }
 }
 
diff --git a/modules/varnish/templates/text-backend.inc.vcl.erb 
b/modules/varnish/templates/text-backend.inc.vcl.erb
index 0ab419d..05372b3 100644
--- a/modules/varnish/templates/text-backend.inc.vcl.erb
+++ b/modules/varnish/templates/text-backend.inc.vcl.erb
@@ -68,7 +68,12 @@
bereq.url !~ "^/wiki/Special:HideBanners") {
std.log("Cacheable object with Set-Cookie found. bereq.url: " + 
bereq.url + " Cache-Control: " + beresp.http.Cache-Control + " Set-Cookie: " + 
beresp.http.Set-Cookie);
set beresp.http.Cache-Control = "private, max-age=0, 
s-maxage=0";
+   <%- if @varnish_version == 5 -%>
+   // HFM
+   set beresp.ttl = 10m;
+   <%- else -%>
set beresp.ttl = 0s;
+   <%- end -%>
set beresp.grace = 0s;
set beresp.keep = 0s;
set beresp.http.X-CDIS = "pass";
diff --git a/modules/varnish/templates/text-common.inc.vcl.erb 
b/modules/varnish/templates/text-common.inc.vcl.erb
index 67c1762..a907d8d 100644
--- a/modules/varnish/templates/text-common.inc.vcl.erb
+++ b/modules/varnish/templates/text-common.inc.vcl.erb
@@ -135,11 +135,16 @@
bereq.http.Cookie == "Token=1"
&& beresp.http.Vary ~ "(?i)(^|,)\s*Cookie\s*(,|$)"
) {
-   set beresp.ttl = 607s;
set beresp.grace = 31s;
set beresp.keep = 0s;
set beresp.http.X-CDIS = "pass";
+   // HFP
+   <%- if @varnish_version == 5 -%>
+   return(pass(607s));
+   <%- else -%>
+   set beresp.ttl = 607s;
set beresp.uncacheable = true;
return (deliver);
+   <%- end -%>
}
 }
diff --git a/modules/varnish/templates/text-frontend.inc.vcl.erb 
b/modules/varnish/templates/text-frontend.inc.vcl.erb
index 00b129f..ea45b7c 100644
--- a/modules/varnish/templates/text-frontend.inc.vcl.erb
+++ b/modules/varnish/templates/text-frontend.inc.vcl.erb
@@ -249,7 +249,12 @@
if (beresp.status == 200
 && bereq.http.X-CDIS == "miss"
 && beresp.http.X-Cache-Int !~ " hit/([4-9]|[0-9]{2,})$") {
+   <%- if @varnish_version == 5 -%>
+   // HFM
+   set beresp.ttl = 10m;
+   <%- else -%>
set beresp.ttl = 0s;
+   

[MediaWiki-commits] [Gerrit] operations/puppet[production]: mtail: add 'ensure' parameter, remove 'enabled'

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/394957 )

Change subject: mtail: add 'ensure' parameter, remove 'enabled'
..

mtail: add 'ensure' parameter, remove 'enabled'

Add a new parameter called 'ensure': if set to 'running', the default,
puppet will make sure that the mtail service is running. The parameter
can otherwise be set to the self-explanatory value of 'stopped'.

Get rid of 'enabled', only used by the SysV init script.

Change-Id: I0f785e13de989d0ffa4c612c5ad44c609f5b531d
---
M modules/mtail/manifests/init.pp
M modules/mtail/templates/default.erb
2 files changed, 10 insertions(+), 8 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/57/394957/1

diff --git a/modules/mtail/manifests/init.pp b/modules/mtail/manifests/init.pp
index af1197b..adf31f8 100644
--- a/modules/mtail/manifests/init.pp
+++ b/modules/mtail/manifests/init.pp
@@ -13,21 +13,21 @@
 # [*graphite_hostport*]
 #   Also send metrics via graphite line-oriented protocol to this host:port.
 #
-# [*enabled*]
-#   Whether to start mtail at boot
+# [*ensure*]
+#   Whether mtail should be running or stopped.
 
 class mtail (
   $logs = ['/var/log/syslog'],
   $port = '3903',
   $graphite_hostport = 'graphite-in.eqiad.wmnet:2003',
   $graphite_prefix = "mtail.${::hostname}.",
-  $enabled = '1',
+  $ensure = 'running',
   $group = 'root',
 ) {
 validate_array($logs)
 validate_re($port, '^[0-9]+$')
 validate_string($graphite_hostport)
-validate_string($enabled)
+validate_ensure($ensure)
 
 require_package('mtail')
 
@@ -41,8 +41,11 @@
 }
 
 systemd::service { 'mtail':
-ensure  => present,
-content => systemd_template('mtail'),
-restart => true,
+ensure => present,
+content=> systemd_template('mtail'),
+restart=> true,
+service_params => {
+ensure => $ensure,
+},
 }
 }
diff --git a/modules/mtail/templates/default.erb 
b/modules/mtail/templates/default.erb
index 0dae7fb..e26726f 100644
--- a/modules/mtail/templates/default.erb
+++ b/modules/mtail/templates/default.erb
@@ -1,5 +1,4 @@
 # Arguments used by SysV init script, systemd service file uses only EXTRA_ARGS
-#ENABLED=<%= @enabled %>
 #GRAPHITE_HOSTPORT=<%= @graphite_hostport %>
 #PORT=<%= @port %>
 #LOGS=<%= @logs.join(',') %>

-- 
To view, visit https://gerrit.wikimedia.org/r/394957
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I0f785e13de989d0ffa4c612c5ad44c609f5b531d
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: mtail: add varnishmtail tests

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/394597 )

Change subject: mtail: add varnishmtail tests
..

mtail: add varnishmtail tests

Add unit tests for all varnish mtail scripts.

Add a new method to MtailMetricStore, get_labels_dict, responsible for
returning a dictionary out of metric labels such as:

varnish_x_cache{x_cache="int-front"} 1
varnish_x_cache{x_cache="hit-front"} 5

Bug: T177199
Change-Id: I7c33a3dc1754e62408b0c450a80018d71d516b34
---
A modules/mtail/files/test/logs/varnish.test
M modules/mtail/files/test/mtail_store.py
A modules/mtail/files/test/varnish_test.py
3 files changed, 60 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/97/394597/1

diff --git a/modules/mtail/files/test/logs/varnish.test 
b/modules/mtail/files/test/logs/varnish.test
new file mode 100644
index 000..b2d4b53
--- /dev/null
+++ b/modules/mtail/files/test/logs/varnish.test
@@ -0,0 +1,9 @@
+url /  cache_status int-front  http_status 301 cache_control - inm -
+url /w/index.php   cache_status hit-front  http_status 304 cache_control 
private, s-maxage=0, max-age=0, must-revalidate   inm -
+url 
/api/rest_v1/metrics/pageviews/per-article/fr.wikipedia.org/all-access/user/S%C3%A9lection_sexuelle/daily/20171126/20171201
cache_status hit-front  http_status 200 cache_control s-maxage=86400, 
max-age=86400 inm -
+url /w/load.phpcache_status hit-front  http_status 200 cache_control 
public, max-age=2592000, s-maxage=2592000 inm -
+url /w/load.phpcache_status hit-front  http_status 200 cache_control 
public, max-age=2592000, s-maxage=2592000 inm -
+url /w/load.phpcache_status hit-front  http_status 304 cache_control 
public, max-age=300, s-maxage=300 inm W/\"1adp3u3\"
+url /wikipedia/commons/5/51/Tang_Shaoyi.jpgcache_status int-front  
http_status 301 cache_control - inm -
+url 
/wikipedia/commons/thumb/5/51/Flag_of_North_Korea.svg/250px-Flag_of_North_Korea.svg.png
cache_status hit-front  http_status 200 cache_control - inm -
+url /wikipedia/en/thumb/f/fd/Portal-puzzle.svg/16px-Portal-puzzle.svg.png  
cache_status hit-front  http_status 200 cache_control - inm -
diff --git a/modules/mtail/files/test/mtail_store.py 
b/modules/mtail/files/test/mtail_store.py
index 527e315..847ed7f 100644
--- a/modules/mtail/files/test/mtail_store.py
+++ b/modules/mtail/files/test/mtail_store.py
@@ -36,6 +36,18 @@

self._store[name][0]['LabelValues'][0].get('Labels', []),

self._store[name][0]['LabelValues'][0]['Value']['Value'])
 
+def get_labels_dict(self, name):
+if name not in self._store:
+raise ValueError('metric %s not found in store', name)
+
+ret = {}
+for label in self._store[name][0]['LabelValues']:
+key = label['Labels'][0]
+value = label['Value']['Value']
+ret[key] = value
+
+return ret
+
 
 class MtailMetric(object):
 def __init__(self, keys, labels, value):
diff --git a/modules/mtail/files/test/varnish_test.py 
b/modules/mtail/files/test/varnish_test.py
new file mode 100644
index 000..31c1034
--- /dev/null
+++ b/modules/mtail/files/test/varnish_test.py
@@ -0,0 +1,39 @@
+import mtail_store
+import unittest
+import os
+
+test_dir = os.path.join(os.path.dirname(__file__))
+
+
+class VarnishXcacheTest(unittest.TestCase):
+def setUp(self):
+self.store = mtail_store.MtailMetricStore(
+os.path.join(test_dir, '../programs/varnishxcache.mtail'),
+os.path.join(test_dir, 'logs/varnish.test'))
+
+def testCacheStatus(self):
+m = self.store.get_labels_dict('varnish_x_cache')
+self.assertEquals(m['hit-front'], 7)
+self.assertEquals(m['int-front'], 2)
+
+
+class VarnishRlsTest(unittest.TestCase):
+def setUp(self):
+self.store = mtail_store.MtailMetricStore(
+os.path.join(test_dir, '../programs/varnishrls.mtail'),
+os.path.join(test_dir, 'logs/varnish.test'))
+
+def testIfNoneMatch(self):
+m = self.store.get_metric('varnish_resourceloader_inm')
+self.assertEquals(m._value, 1)
+
+
+class VarnishMediaTest(unittest.TestCase):
+def setUp(self):
+self.store = mtail_store.MtailMetricStore(
+os.path.join(test_dir, '../programs/varnishmedia.mtail'),
+os.path.join(test_dir, 'logs/varnish.test'))
+
+def testThumbnails(self):
+m = self.store.get_labels_dict('varnish_thumbnails')
+self.assertEquals(m['200'], 2)

-- 
To view, visit https://gerrit.wikimedia.org/r/394597
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I7c33a3dc1754e62408b0c450a80018d71d516b34
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production

[MediaWiki-commits] [Gerrit] operations/puppet[production]: WIP: varnish: prometheus equivalent of statsd metrics daemons

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/394543 )

Change subject: WIP: varnish: prometheus equivalent of statsd metrics daemons
..

WIP: varnish: prometheus equivalent of statsd metrics daemons

Introduce the following mtail scripts:

- xcache.mtail (varnishxcache)
- resourceloader.mtail (varnishrls)
- media.mtail (varnishmedia)

Add and use mtail::script to install mtail scripts.

Bug: T177199
Change-Id: I31115573a5d7f43268eef3a1bcee92e18d5fa957
---
A modules/mtail/manifests/script.pp
A modules/varnish/files/mtail/media.mtail
A modules/varnish/files/mtail/resourceloader.mtail
A modules/varnish/files/mtail/xcache.mtail
M modules/varnish/manifests/logging/media.pp
M modules/varnish/manifests/logging/rls.pp
M modules/varnish/manifests/logging/xcache.pp
7 files changed, 70 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/43/394543/1

diff --git a/modules/mtail/manifests/script.pp 
b/modules/mtail/manifests/script.pp
new file mode 100644
index 000..b544b80
--- /dev/null
+++ b/modules/mtail/manifests/script.pp
@@ -0,0 +1,36 @@
+# Definition: mtail::script
+#
+# Install the given mtail script under the mtail scripts directory. There is no
+# need to notify any running mtail instance, changes are picked up by mtail
+# in a automatic fashion upon file modification.
+#
+# Parameters
+#   $source
+#   The file containing the mtail script to be installed (required).
+#   $destdir
+#   Destination directory. Defaults to /usr/local/share/mtail.
+#
+# Usage example:
+#   mtail::script { 'xcache':
+#   source => 'puppet:///modules/varnish/mtail/xcache.mtail',
+#   }
+#
+define mtail::script($source, $destdir='/usr/local/share/mtail') {
+validate_string($source)
+validate_absolute_path($destdir)
+
+file { $destdir:
+ensure => directory,
+owner  => 'root',
+group  => 'root',
+mode   => '0444',
+}
+
+file { "${destdir}/${title}.mtail":
+source  => $source,
+owner   => 'root',
+group   => 'root',
+mode=> '0444',
+require => File[$destdir],
+}
+}
diff --git a/modules/varnish/files/mtail/media.mtail 
b/modules/varnish/files/mtail/media.mtail
new file mode 100644
index 000..a84f184
--- /dev/null
+++ b/modules/varnish/files/mtail/media.mtail
@@ -0,0 +1,5 @@
+counter varnish_thumbnails by status
+
+/^url \/thumb\/.*\thttp_status (?P.*)\tcache_control/ {
+varnish_thumbnails[$http_status]++
+}
diff --git a/modules/varnish/files/mtail/resourceloader.mtail 
b/modules/varnish/files/mtail/resourceloader.mtail
new file mode 100644
index 000..0705e44
--- /dev/null
+++ b/modules/varnish/files/mtail/resourceloader.mtail
@@ -0,0 +1,7 @@
+counter varnish_resourceloader_inm
+
+/^url \/w\/load.php.*\tinm (?P.*)$/ {
+$inm != "-" {
+varnish_resourceloader_inm++
+}
+}
diff --git a/modules/varnish/files/mtail/xcache.mtail 
b/modules/varnish/files/mtail/xcache.mtail
new file mode 100644
index 000..df4a197
--- /dev/null
+++ b/modules/varnish/files/mtail/xcache.mtail
@@ -0,0 +1,5 @@
+counter varnish_x_cache by x_cache
+
+/^.*\tcache_status (?P.*)\thttp_status/ {
+varnish_x_cache[$x_cache]++
+}
diff --git a/modules/varnish/manifests/logging/media.pp 
b/modules/varnish/manifests/logging/media.pp
index a86e001..e9f7e33 100644
--- a/modules/varnish/manifests/logging/media.pp
+++ b/modules/varnish/manifests/logging/media.pp
@@ -1,7 +1,7 @@
 # == Define: varnish::logging::media
 #
 #  Accumulate browser cache hit ratio and total request volume statistics
-#  for Media requests and report to StatsD.
+#  for Media requests and report to StatsD. Expose metrics to prometheus.
 #
 # === Parameters
 #
@@ -42,4 +42,8 @@
 description  => 'Varnish traffic logger - varnishmedia',
 nrpe_command => '/usr/lib/nagios/plugins/check_procs -w 1:1 -a 
"/usr/local/bin/varnishmedia" -u root',
 }
+
+mtail::script { 'media':
+source => 'puppet:///modules/varnish/mtail/media.mtail',
+}
 }
diff --git a/modules/varnish/manifests/logging/rls.pp 
b/modules/varnish/manifests/logging/rls.pp
index 9ee16e7..4c4629a 100644
--- a/modules/varnish/manifests/logging/rls.pp
+++ b/modules/varnish/manifests/logging/rls.pp
@@ -1,7 +1,8 @@
 # == Define: varnish::logging::rls
 #
-#  Accumulate browser cache hit ratio and total request volume statistics
-#  for ResourceLoader requests (/w/load.php) and report to StatsD.
+#  Accumulate browser cache hit ratio and total request volume statistics for
+#  ResourceLoader requests (/w/load.php) and report to StatsD. Expose metrics
+#  to prometheus.
 #
 # === Parameters
 #
@@ -43,4 +44,8 @@
 description  => 'Varnish traffic logger - varnishrls',
 nrpe_command => '/usr/lib/nagios/plugins/check_procs -w 1:1 -a 
"/usr/local/bin/varnishrls" -u root',
 }
+
+mtail::script { 'resourceloader':

[MediaWiki-commits] [Gerrit] operations/puppet[production]: prometheus: add varnish-canary job definition

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/394063 )

Change subject: prometheus: add varnish-canary job definition
..


prometheus: add varnish-canary job definition

The only cache::canary host, pinkunicorn, is not being included in
prometheus' target list. Add a job definiton for varnish-canary, and
call prometheus::varnish_2layer for canary too.

Change-Id: I36d816d2ca4af62c77145759fea29df81b15e5e5
---
M modules/role/manifests/prometheus/ops.pp
1 file changed, 12 insertions(+), 0 deletions(-)

Approvals:
  Ema: Verified; Looks good to me, approved
  Filippo Giunchedi: Looks good to me, but someone else must approve



diff --git a/modules/role/manifests/prometheus/ops.pp 
b/modules/role/manifests/prometheus/ops.pp
index f4c11fe..8ef5dab 100644
--- a/modules/role/manifests/prometheus/ops.pp
+++ b/modules/role/manifests/prometheus/ops.pp
@@ -210,6 +210,13 @@
 ],
 'metric_relabel_configs' => [$varnish_be_uuid_relabel],
   },
+  {
+'job_name'=> 'varnish-canary',
+'file_sd_configs' => [
+  { 'files' => [ "${targets_path}/varnish-canary_*.yaml"] },
+],
+'metric_relabel_configs' => [$varnish_be_uuid_relabel],
+  },
 ]
 
 # Job definition for memcache_exporter
@@ -683,6 +690,11 @@
 cache_name   => 'upload',
 }
 
+prometheus::varnish_2layer{ 'canary':
+targets_path => $targets_path,
+cache_name   => 'canary',
+}
+
 # Move Prometheus metrics to new HW - T148408
 include rsync::server
 

-- 
To view, visit https://gerrit.wikimedia.org/r/394063
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I36d816d2ca4af62c77145759fea29df81b15e5e5
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: prometheus: add varnish-canary job definition

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/394063 )

Change subject: prometheus: add varnish-canary job definition
..

prometheus: add varnish-canary job definition

The only cache::canary host, pinkunicorn, is not being included in
prometheus' target list. Add a job definiton for varnish-canary, and
call prometheus::varnish_2layer for canary too.

Change-Id: I36d816d2ca4af62c77145759fea29df81b15e5e5
---
M modules/role/manifests/prometheus/ops.pp
1 file changed, 12 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/63/394063/1

diff --git a/modules/role/manifests/prometheus/ops.pp 
b/modules/role/manifests/prometheus/ops.pp
index 6361a18..cf5b10b 100644
--- a/modules/role/manifests/prometheus/ops.pp
+++ b/modules/role/manifests/prometheus/ops.pp
@@ -210,6 +210,13 @@
 ],
 'metric_relabel_configs' => [$varnish_be_uuid_relabel],
   },
+  {
+'job_name'=> 'varnish-canary',
+'file_sd_configs' => [
+  { 'files' => [ "${targets_path}/varnish-canary_*.yaml"] },
+],
+'metric_relabel_configs' => [$varnish_be_uuid_relabel],
+  },
 ]
 
 # Job definition for memcache_exporter
@@ -653,6 +660,11 @@
 cache_name   => 'upload',
 }
 
+prometheus::varnish_2layer{ 'canary':
+targets_path => $targets_path,
+cache_name   => 'canary',
+}
+
 # Move Prometheus metrics to new HW - T148408
 include rsync::server
 

-- 
To view, visit https://gerrit.wikimedia.org/r/394063
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I36d816d2ca4af62c77145759fea29df81b15e5e5
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Log more detailed info in Varnish slow request log

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/393751 )

Change subject: Log more detailed info in Varnish slow request log
..


Log more detailed info in Varnish slow request log

Break down the various parts of Varnish processing

Bug: T181315
Change-Id: Ia5ee2a723dc030920cc4931fbcdb782a8c47c578
---
M modules/varnish/templates/initscripts/varnish-slowreqs.systemd.erb
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Ema: Verified; Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/varnish/templates/initscripts/varnish-slowreqs.systemd.erb 
b/modules/varnish/templates/initscripts/varnish-slowreqs.systemd.erb
index 17977d3..619ffc7 100644
--- a/modules/varnish/templates/initscripts/varnish-slowreqs.systemd.erb
+++ b/modules/varnish/templates/initscripts/varnish-slowreqs.systemd.erb
@@ -10,7 +10,7 @@
 Restart=always
 # Note the usage of the %% specifier here.
 # See systemd.unit(5), section SPECIFIERS.
-ExecStart=/usr/bin/varnishncsa <%= @extraopts %> -q 'ReqMethod ne "PURGE" and 
Timestamp:Resp[2] > <%= scope['::varnish::common::log_slow_request_threshold'] 
%>' -F '%{VSL:Timestamp:Resp}x %%r %%s'
+ExecStart=/usr/bin/varnishncsa <%= @extraopts %> -q 'ReqMethod ne "PURGE" and 
Timestamp:Resp[2] > <%= scope['::varnish::common::log_slow_request_threshold'] 
%>' -F '%{VSL:Timestamp:Start[1]}x Start: %{VSL:Timestamp:Start[3]}x Req: 
%{VSL:Timestamp:Req[3]}x Fetch: %{VSL:Timestamp:Fetch[3]}x Process: 
%{VSL:Timestamp:Process[3]}x Resp: %{VSL:Timestamp:Resp[3]}x %%r %%s'
 
 [Install]
 WantedBy=multi-user.target

-- 
To view, visit https://gerrit.wikimedia.org/r/393751
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ia5ee2a723dc030920cc4931fbcdb782a8c47c578
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Gilles 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Imarlier 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: vcl: add hostname/layer info to syntethic healthcheck response

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/393251 )

Change subject: vcl: add hostname/layer info to syntethic healthcheck response
..

vcl: add hostname/layer info to syntethic healthcheck response

Varnish currently returns an empty 200 OK as healthcheck response. Add
hostname/layer informationon to the body.

Change-Id: I3d0a8f4f75a92105e1192925da3beafa5cdfcc0e
---
M modules/varnish/templates/vcl/wikimedia-backend.vcl.erb
M modules/varnish/templates/vcl/wikimedia-common.inc.vcl.erb
M modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
3 files changed, 11 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/51/393251/1

diff --git a/modules/varnish/templates/vcl/wikimedia-backend.vcl.erb 
b/modules/varnish/templates/vcl/wikimedia-backend.vcl.erb
index 59236fb..2d94a8e 100644
--- a/modules/varnish/templates/vcl/wikimedia-backend.vcl.erb
+++ b/modules/varnish/templates/vcl/wikimedia-backend.vcl.erb
@@ -220,6 +220,7 @@
if (resp.status >= 400) {
call synth_errorpage;
}
+   call wm_common_synth;
return (deliver);
 }
 
diff --git a/modules/varnish/templates/vcl/wikimedia-common.inc.vcl.erb 
b/modules/varnish/templates/vcl/wikimedia-common.inc.vcl.erb
index 9419e35..969ee33 100644
--- a/modules/varnish/templates/vcl/wikimedia-common.inc.vcl.erb
+++ b/modules/varnish/templates/vcl/wikimedia-common.inc.vcl.erb
@@ -346,7 +346,7 @@
}
 
if ( req.http.host ~ "^varnishcheck" ) {
-   return (synth(200, "OK"));
+   return (synth(200, "healthcheck"));
}
 }
 
@@ -382,6 +382,13 @@
set req.http.X-CDIS = "pass";
 }
 
+sub wm_common_synth {
+   if (resp.reason == "healthcheck") {
+   set resp.reason = "OK";
+   synthetic("Varnish <%= @inst %> running on <%= @hostname %> is 
up");
+   }
+}
+
 sub wm_common_backend_response {
// This prevents the application layer from setting this in a response.
// We'll be setting this same variable internally in VCL in hit-for-pass
diff --git a/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb 
b/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
index 29ed71d..2afc976 100644
--- a/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
+++ b/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
@@ -514,6 +514,8 @@
call synth_errorpage;
}
}
+
+   call wm_common_synth;
return (deliver);
 }
 

-- 
To view, visit https://gerrit.wikimedia.org/r/393251
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I3d0a8f4f75a92105e1192925da3beafa5cdfcc0e
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: WIP: cache: size-based cutoff for exp caching policy

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/393227 )

Change subject: WIP: cache: size-based cutoff for exp caching policy
..

WIP: cache: size-based cutoff for exp caching policy

- Limit the exp caching policy to Varnish 5 (v4 has no HFM)
- Unconditionally return HFM for exp
- Add a size-based HFP cutoff

Bug: T144187
Change-Id: I5a326e128153af9e3f21840eaf53164a4eb586d6
---
M modules/varnish/templates/vcl/wikimedia-common.inc.vcl.erb
M modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
2 files changed, 18 insertions(+), 6 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/27/393227/1

diff --git a/modules/varnish/templates/vcl/wikimedia-common.inc.vcl.erb 
b/modules/varnish/templates/vcl/wikimedia-common.inc.vcl.erb
index a4d32e0..bcbd68c 100644
--- a/modules/varnish/templates/vcl/wikimedia-common.inc.vcl.erb
+++ b/modules/varnish/templates/vcl/wikimedia-common.inc.vcl.erb
@@ -447,6 +447,22 @@
return (deliver);
<%- end -%>
}
+
+<% if @varnish_version == 5 && @vcl_config.fetch("admission_policy", "nhw") == 
"exp" -%>
+   // XXX: this should most likely not be defined here, as the exp policy 
need
+   // to be applied on the frontend layer only
+
+   // We want to apply the "exp" caching policy only to objects with CL 
below
+   // 10M (but present). The "exp" caching policy is defined in
+   // wikimedia-frontend.vcl.erb, but the logic below can not be added 
there
+   // because by returning HFP we would skip calling various VCL 
subroutines such
+   // as cluster_fe_backend_response_early.
+   if (beresp.status == 200 && bereq.http.X-CDIS == "miss"
+   && std.integer(beresp.http.Content-Length, 0) >= 1024 * 1024 * 
10) {
+   // HFP
+   return(pass(120s));
+   }
+<%- end -%>
 }
 
 // call just before wm_common_xcache_deliver, but only in vcl_deliver, not 
vcl_synth
diff --git a/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb 
b/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
index a3f5826..7ea9c50 100644
--- a/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
+++ b/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
@@ -2,7 +2,7 @@
 
 vcl 4.0;
 
-<% if @vcl_config.fetch("admission_policy", "nhw") == "exp" -%>
+<% if @varnish_version == 5 && @vcl_config.fetch("admission_policy", "nhw") == 
"exp" -%>
 // Includes for Exp cache admission policy, admission probability exponentially
 // decreasing with size. See vcl_backend_response. T144187
 C{
@@ -358,7 +358,7 @@
if (beresp.status == 503 && bereq.retries == 0 && bereq.method ~ 
"^(GET|HEAD|OPTIONS|PUT|DELETE)$") {
return(retry);
}
-<% if @vcl_config.fetch("admission_policy", "nhw") == "exp" -%>
+<% if @varnish_version == 5 && @vcl_config.fetch("admission_policy", "nhw") == 
"exp" -%>
 if (beresp.status == 200 && bereq.http.X-CDIS == "miss") {
 C{
const struct gethdr_s hdr = { HDR_BERESP, "\017Content-Length:" };
@@ -391,12 +391,8 @@
 
// If admission test succeeds, mark as uncacheable
if (admissionprob < urand) {
-   <%- if @varnish_version == 5 -%>
// HFM with ttl=67 to avoid stalling
VRT_l_beresp_ttl(ctx,67);
-   <%- else -%>
-   VRT_l_beresp_ttl(ctx,0);
-   <%- end -%>
VRT_l_beresp_uncacheable(ctx,1);
}
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/393227
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I5a326e128153af9e3f21840eaf53164a4eb586d6
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_misc: use grafana.w.o instead of git.w.o in VTC tests

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/393195 )

Change subject: cache_misc: use grafana.w.o instead of git.w.o in VTC tests
..


cache_misc: use grafana.w.o instead of git.w.o in VTC tests

To reduce the number of different Host header values in VTC, replace
git.w.o with grafana.w.o.

Change-Id: I7839310dee12135c9c21aaeebdd24407667316eb
---
M modules/varnish/files/tests/misc/01-basic-caching.vtc
M modules/varnish/files/tests/misc/09-chunked-response-add-cl.vtc
M modules/varnish/files/tests/misc/10-deliver-synth-x-client-ip.vtc
3 files changed, 6 insertions(+), 6 deletions(-)

Approvals:
  Ema: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/varnish/files/tests/misc/01-basic-caching.vtc 
b/modules/varnish/files/tests/misc/01-basic-caching.vtc
index eae1ec4..936e855 100644
--- a/modules/varnish/files/tests/misc/01-basic-caching.vtc
+++ b/modules/varnish/files/tests/misc/01-basic-caching.vtc
@@ -26,7 +26,7 @@
 }
 
 client c3 {
-txreq -hdr "Host: git.wikimedia.org"
+txreq -hdr "Host: grafana.wikimedia.org"
 rxresp
 expect resp.status == 200
 }
@@ -49,13 +49,13 @@
 varnish v1 -expect cache_hit == 0
 varnish v1 -expect n_object == 0
 
-# Cache miss with Host: git.wikimedia.org
+# Cache miss with Host: grafana.wikimedia.org
 client c3 -run
 varnish v1 -expect cache_miss == 1
 varnish v1 -expect cache_hit == 0
 varnish v1 -expect n_object == 1
 
-# Cache hit with Host: git.wikimedia.org
+# Cache hit with Host: grafana.wikimedia.org
 client c3 -run
 varnish v1 -expect cache_miss == 1
 varnish v1 -expect cache_hit == 1
diff --git a/modules/varnish/files/tests/misc/09-chunked-response-add-cl.vtc 
b/modules/varnish/files/tests/misc/09-chunked-response-add-cl.vtc
index bb3cca8..967d590 100644
--- a/modules/varnish/files/tests/misc/09-chunked-response-add-cl.vtc
+++ b/modules/varnish/files/tests/misc/09-chunked-response-add-cl.vtc
@@ -17,7 +17,7 @@
 } -start
 
 client c1 {
-txreq -hdr "Host: git.wikimedia.org" -hdr "X-Forwarded-Proto: https"
+txreq -hdr "Host: grafana.wikimedia.org" -hdr "X-Forwarded-Proto: https"
 rxresp
 expect resp.status == 200
 # We expect Content-Length to be set to 5 (hello) by varnish
diff --git a/modules/varnish/files/tests/misc/10-deliver-synth-x-client-ip.vtc 
b/modules/varnish/files/tests/misc/10-deliver-synth-x-client-ip.vtc
index 42ed3c2..5364738 100644
--- a/modules/varnish/files/tests/misc/10-deliver-synth-x-client-ip.vtc
+++ b/modules/varnish/files/tests/misc/10-deliver-synth-x-client-ip.vtc
@@ -16,12 +16,12 @@
 } -start
 
 client c1 {
-txreq -hdr "Host: git.wikimedia.org" -hdr "X-Forwarded-Proto: https"
+txreq -hdr "Host: grafana.wikimedia.org" -hdr "X-Forwarded-Proto: https"
 rxresp
 expect resp.status == 200
 expect resp.http.X-Client-IP == "127.0.0.1"
 
-txreq -hdr "Host: git.wikimedia.org"
+txreq -hdr "Host: grafana.wikimedia.org"
 rxresp
 # http -> https redirect through _synth, we should still get X-Client-IP
 expect resp.status == 301

-- 
To view, visit https://gerrit.wikimedia.org/r/393195
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I7839310dee12135c9c21aaeebdd24407667316eb
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_misc: use grafana.w.o instead of git.w.o in VTC tests

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/393195 )

Change subject: cache_misc: use grafana.w.o instead of git.w.o in VTC tests
..

cache_misc: use grafana.w.o instead of git.w.o in VTC tests

To reduce the number of different Host header values in VTC, replace
git.w.o with grafana.w.o.

Change-Id: I7839310dee12135c9c21aaeebdd24407667316eb
---
M modules/varnish/files/tests/misc/01-basic-caching.vtc
M modules/varnish/files/tests/misc/09-chunked-response-add-cl.vtc
M modules/varnish/files/tests/misc/10-deliver-synth-x-client-ip.vtc
3 files changed, 6 insertions(+), 6 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/95/393195/1

diff --git a/modules/varnish/files/tests/misc/01-basic-caching.vtc 
b/modules/varnish/files/tests/misc/01-basic-caching.vtc
index eae1ec4..936e855 100644
--- a/modules/varnish/files/tests/misc/01-basic-caching.vtc
+++ b/modules/varnish/files/tests/misc/01-basic-caching.vtc
@@ -26,7 +26,7 @@
 }
 
 client c3 {
-txreq -hdr "Host: git.wikimedia.org"
+txreq -hdr "Host: grafana.wikimedia.org"
 rxresp
 expect resp.status == 200
 }
@@ -49,13 +49,13 @@
 varnish v1 -expect cache_hit == 0
 varnish v1 -expect n_object == 0
 
-# Cache miss with Host: git.wikimedia.org
+# Cache miss with Host: grafana.wikimedia.org
 client c3 -run
 varnish v1 -expect cache_miss == 1
 varnish v1 -expect cache_hit == 0
 varnish v1 -expect n_object == 1
 
-# Cache hit with Host: git.wikimedia.org
+# Cache hit with Host: grafana.wikimedia.org
 client c3 -run
 varnish v1 -expect cache_miss == 1
 varnish v1 -expect cache_hit == 1
diff --git a/modules/varnish/files/tests/misc/09-chunked-response-add-cl.vtc 
b/modules/varnish/files/tests/misc/09-chunked-response-add-cl.vtc
index bb3cca8..967d590 100644
--- a/modules/varnish/files/tests/misc/09-chunked-response-add-cl.vtc
+++ b/modules/varnish/files/tests/misc/09-chunked-response-add-cl.vtc
@@ -17,7 +17,7 @@
 } -start
 
 client c1 {
-txreq -hdr "Host: git.wikimedia.org" -hdr "X-Forwarded-Proto: https"
+txreq -hdr "Host: grafana.wikimedia.org" -hdr "X-Forwarded-Proto: https"
 rxresp
 expect resp.status == 200
 # We expect Content-Length to be set to 5 (hello) by varnish
diff --git a/modules/varnish/files/tests/misc/10-deliver-synth-x-client-ip.vtc 
b/modules/varnish/files/tests/misc/10-deliver-synth-x-client-ip.vtc
index 42ed3c2..5364738 100644
--- a/modules/varnish/files/tests/misc/10-deliver-synth-x-client-ip.vtc
+++ b/modules/varnish/files/tests/misc/10-deliver-synth-x-client-ip.vtc
@@ -16,12 +16,12 @@
 } -start
 
 client c1 {
-txreq -hdr "Host: git.wikimedia.org" -hdr "X-Forwarded-Proto: https"
+txreq -hdr "Host: grafana.wikimedia.org" -hdr "X-Forwarded-Proto: https"
 rxresp
 expect resp.status == 200
 expect resp.http.X-Client-IP == "127.0.0.1"
 
-txreq -hdr "Host: git.wikimedia.org"
+txreq -hdr "Host: grafana.wikimedia.org"
 rxresp
 # http -> https redirect through _synth, we should still get X-Client-IP
 expect resp.status == 301

-- 
To view, visit https://gerrit.wikimedia.org/r/393195
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I7839310dee12135c9c21aaeebdd24407667316eb
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Lower depool threshold for Apache to 0.8 (80%)

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/389964 )

Change subject: Lower depool threshold for Apache to 0.8 (80%)
..


Lower depool threshold for Apache to 0.8 (80%)

With 0.9 one can only depool 7 out of 66 eqiad app servers. Given that our usual
CPU usage is around 15% that seems overly cautiuos, let's reduce it to
0.8 to leave more leeway for cluster maintenance (allowing bigger cluster parts
to be depooled/upgraded at once).

Bug: T178799
Change-Id: I9a14131ed74c8dd6832cceaff1e96b2029ae9ee4
---
M hieradata/common/lvs/configuration.yaml
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Ema: Verified; Looks good to me, approved
  BBlack: Looks good to me, but someone else must approve
  Alexandros Kosiaris: Looks good to me, but someone else must approve
  Elukey: Looks good to me, but someone else must approve



diff --git a/hieradata/common/lvs/configuration.yaml 
b/hieradata/common/lvs/configuration.yaml
index 79717af..fa4914a 100644
--- a/hieradata/common/lvs/configuration.yaml
+++ b/hieradata/common/lvs/configuration.yaml
@@ -343,7 +343,7 @@
 - eqiad
 - codfw
 ip: *ip_block007
-depool-threshold: '.9'
+depool-threshold: '.8'
 monitors:
   ProxyFetch:
 url:

-- 
To view, visit https://gerrit.wikimedia.org/r/389964
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I9a14131ed74c8dd6832cceaff1e96b2029ae9ee4
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Muehlenhoff 
Gerrit-Reviewer: Alexandros Kosiaris 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...varnish4[debian-wmf]: 5.1.3-1wm3: fix VSV00002

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/391541 )

Change subject: 5.1.3-1wm3: fix VSV2
..


5.1.3-1wm3: fix VSV2

Avoid buffer read overflow on vcl_error and -sfile (VSV2,
CVE-2017-8807).

Change-Id: I4e98e725d154affe0bb4a8c71b3445472f4586ce
---
M debian/changelog
A debian/patches/0008-vsv2-5.1.patch
M debian/patches/series
3 files changed, 41 insertions(+), 0 deletions(-)

Approvals:
  Muehlenhoff: Looks good to me, but someone else must approve
  Ema: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/debian/changelog b/debian/changelog
index 02eb745..a947366 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,10 @@
+varnish (5.1.3-1wm3) jessie-wikimedia; urgency=medium
+
+  * Avoid buffer read overflow on vcl_error and -sfile
+(VSV2, CVE-2017-8807)
+
+ -- Emanuele Rocca   Wed, 15 Nov 2017 12:29:06 +0100
+
 varnish (5.1.3-1wm2) jessie-wikimedia; urgency=medium
 
   * Add transaction_timeout: maximum amount of time in seconds for the whole
diff --git a/debian/patches/0008-vsv2-5.1.patch 
b/debian/patches/0008-vsv2-5.1.patch
new file mode 100644
index 000..df22b66
--- /dev/null
+++ b/debian/patches/0008-vsv2-5.1.patch
@@ -0,0 +1,33 @@
+From 6143df703e93f6a599ffcbd8258af7ce45d14576 Mon Sep 17 00:00:00 2001
+From: Martin Blix Grydeland 
+Date: Mon, 18 Sep 2017 16:04:53 +0200
+Subject: [PATCH] Avoid buffer read overflow on vcl_error and -sfile
+
+The file stevedore may return a buffer larger than asked for when
+requesting storage. Due to lack of check for this condition, the code
+to copy the synthetic error memory buffer from vcl_error would overrun
+the buffer.
+
+Patch by @shamger
+
+Fixes: #2429
+---
+ bin/varnishd/cache/cache_fetch.c | 2 ++
+ 1 file changed, 2 insertions(+)
+
+diff --git a/bin/varnishd/cache/cache_fetch.c 
b/bin/varnishd/cache/cache_fetch.c
+index 670dc8f..bd3a9b6 100644
+--- a/bin/varnishd/cache/cache_fetch.c
 b/bin/varnishd/cache/cache_fetch.c
+@@ -899,6 +899,8 @@ vbf_stp_error(struct worker *wrk, struct busyobj *bo)
+   l = ll;
+   if (VFP_GetStorage(bo->vfc, , ) != VFP_OK)
+   break;
++  if (l > ll)
++  l = ll;
+   memcpy(ptr, VSB_data(synth_body) + o, l);
+   VFP_Extend(bo->vfc, l);
+   ll -= l;
+-- 
+2.1.4
+
diff --git a/debian/patches/series b/debian/patches/series
index b0183a0..a58e661 100644
--- a/debian/patches/series
+++ b/debian/patches/series
@@ -5,3 +5,4 @@
 0005-stats-shortlived.patch
 0006-transaction-timeout.patch
 0007-varnishncsa-record-prefix.patch
+0008-vsv2-5.1.patch

-- 
To view, visit https://gerrit.wikimedia.org/r/391541
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I4e98e725d154affe0bb4a8c71b3445472f4586ce
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/varnish4
Gerrit-Branch: debian-wmf
Gerrit-Owner: Ema 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Muehlenhoff 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...varnish4[debian-wmf-4.1]: 4.1.8-1wm2: fix VSV00002

[Ema (Code Review)]

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/391538 )

Change subject: 4.1.8-1wm2: fix VSV2
..


4.1.8-1wm2: fix VSV2

Avoid buffer read overflow on vcl_error and -sfile (VSV2,
CVE-2017-8807).

Change-Id: Ibb4ed766d11ac366603eb74d6a86a584e5c306f6
---
M debian/changelog
A debian/patches/0006-vsv2-4.1.patch
M debian/patches/series
3 files changed, 41 insertions(+), 0 deletions(-)

Approvals:
  Muehlenhoff: Looks good to me, but someone else must approve
  Ema: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/debian/changelog b/debian/changelog
index c7a289f..9448e3e 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,10 @@
+varnish (4.1.8-1wm2) jessie-wikimedia; urgency=medium
+
+  * Avoid buffer read overflow on vcl_error and -sfile
+(VSV2, CVE-2017-8807)
+
+ -- Emanuele Rocca   Wed, 15 Nov 2017 12:24:09 +0100
+
 varnish (4.1.8-1wm1) jessie-wikimedia; urgency=medium
 
   * New upstream release
diff --git a/debian/patches/0006-vsv2-4.1.patch 
b/debian/patches/0006-vsv2-4.1.patch
new file mode 100644
index 000..650787c
--- /dev/null
+++ b/debian/patches/0006-vsv2-4.1.patch
@@ -0,0 +1,33 @@
+From 19a73184c6470a54f843c7c226c641a0b4ac2e8e Mon Sep 17 00:00:00 2001
+From: Martin Blix Grydeland 
+Date: Mon, 18 Sep 2017 16:04:53 +0200
+Subject: [PATCH] Avoid buffer read overflow on vcl_error and -sfile
+
+The file stevedore may return a buffer larger than asked for when
+requesting storage. Due to lack of check for this condition, the code
+to copy the synthetic error memory buffer from vcl_error would overrun
+the buffer.
+
+Patch by @shamger
+
+Fixes: #2429
+---
+ bin/varnishd/cache/cache_fetch.c | 2 ++
+ 1 file changed, 2 insertions(+)
+
+diff --git a/bin/varnishd/cache/cache_fetch.c 
b/bin/varnishd/cache/cache_fetch.c
+index d36377c..70f953f 100644
+--- a/bin/varnishd/cache/cache_fetch.c
 b/bin/varnishd/cache/cache_fetch.c
+@@ -873,6 +873,8 @@ vbf_stp_error(struct worker *wrk, struct busyobj *bo)
+   l = ll;
+   if (VFP_GetStorage(bo->vfc, , ) != VFP_OK)
+   break;
++  if (l > ll)
++  l = ll;
+   memcpy(ptr, VSB_data(synth_body) + o, l);
+   VBO_extend(bo, l);
+   ll -= l;
+-- 
+2.1.4
+
diff --git a/debian/patches/series b/debian/patches/series
index 3deaf68..d0d9770 100644
--- a/debian/patches/series
+++ b/debian/patches/series
@@ -3,3 +3,4 @@
 0003-vsm-perms.patch
 0004-storage-file-off-t.patch
 0005-stats-shortlived.patch
+0006-vsv2-4.1.patch

-- 
To view, visit https://gerrit.wikimedia.org/r/391538
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ibb4ed766d11ac366603eb74d6a86a584e5c306f6
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/varnish4
Gerrit-Branch: debian-wmf-4.1
Gerrit-Owner: Ema 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Muehlenhoff 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...varnish4[debian-wmf]: 5.1.3-1wm3: fix VSV00002

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/391541 )

Change subject: 5.1.3-1wm3: fix VSV2
..

5.1.3-1wm3: fix VSV2

Avoid buffer read overflow on vcl_error and -sfile (VSV2,
CVE-2017-8807).

Change-Id: I4e98e725d154affe0bb4a8c71b3445472f4586ce
---
M debian/changelog
A debian/patches/0008-vsv2-5.1.patch
M debian/patches/series
3 files changed, 41 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/varnish4 
refs/changes/41/391541/1

diff --git a/debian/changelog b/debian/changelog
index 02eb745..a947366 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,10 @@
+varnish (5.1.3-1wm3) jessie-wikimedia; urgency=medium
+
+  * Avoid buffer read overflow on vcl_error and -sfile
+(VSV2, CVE-2017-8807)
+
+ -- Emanuele Rocca   Wed, 15 Nov 2017 12:29:06 +0100
+
 varnish (5.1.3-1wm2) jessie-wikimedia; urgency=medium
 
   * Add transaction_timeout: maximum amount of time in seconds for the whole
diff --git a/debian/patches/0008-vsv2-5.1.patch 
b/debian/patches/0008-vsv2-5.1.patch
new file mode 100644
index 000..df22b66
--- /dev/null
+++ b/debian/patches/0008-vsv2-5.1.patch
@@ -0,0 +1,33 @@
+From 6143df703e93f6a599ffcbd8258af7ce45d14576 Mon Sep 17 00:00:00 2001
+From: Martin Blix Grydeland 
+Date: Mon, 18 Sep 2017 16:04:53 +0200
+Subject: [PATCH] Avoid buffer read overflow on vcl_error and -sfile
+
+The file stevedore may return a buffer larger than asked for when
+requesting storage. Due to lack of check for this condition, the code
+to copy the synthetic error memory buffer from vcl_error would overrun
+the buffer.
+
+Patch by @shamger
+
+Fixes: #2429
+---
+ bin/varnishd/cache/cache_fetch.c | 2 ++
+ 1 file changed, 2 insertions(+)
+
+diff --git a/bin/varnishd/cache/cache_fetch.c 
b/bin/varnishd/cache/cache_fetch.c
+index 670dc8f..bd3a9b6 100644
+--- a/bin/varnishd/cache/cache_fetch.c
 b/bin/varnishd/cache/cache_fetch.c
+@@ -899,6 +899,8 @@ vbf_stp_error(struct worker *wrk, struct busyobj *bo)
+   l = ll;
+   if (VFP_GetStorage(bo->vfc, , ) != VFP_OK)
+   break;
++  if (l > ll)
++  l = ll;
+   memcpy(ptr, VSB_data(synth_body) + o, l);
+   VFP_Extend(bo->vfc, l);
+   ll -= l;
+-- 
+2.1.4
+
diff --git a/debian/patches/series b/debian/patches/series
index b0183a0..a58e661 100644
--- a/debian/patches/series
+++ b/debian/patches/series
@@ -5,3 +5,4 @@
 0005-stats-shortlived.patch
 0006-transaction-timeout.patch
 0007-varnishncsa-record-prefix.patch
+0008-vsv2-5.1.patch

-- 
To view, visit https://gerrit.wikimedia.org/r/391541
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I4e98e725d154affe0bb4a8c71b3445472f4586ce
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/varnish4
Gerrit-Branch: debian-wmf
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits