Ottomata has submitted this change and it was merged.
Change subject: Require 2 ACKs from kafka brokers per default
......................................................................
Require 2 ACKs from kafka brokers per default
We already required 2 ACKs from kafka brokers for bits, mobile, and
text. Since that did not cause issues, we now require 2 ACKs per
default, and thereby also require 2 ACKs for upload.
Bug: 69667
Change-Id: I72acc4afd3f9699d101db18fa8e0e29ca49ed2ab
---
M manifests/role/cache.pp
1 file changed, 4 insertions(+), 25 deletions(-)
Approvals:
Ottomata: Looks good to me, approved
jenkins-bot: Verified
diff --git a/manifests/role/cache.pp b/manifests/role/cache.pp
index 4a65e55..55b406f 100644
--- a/manifests/role/cache.pp
+++ b/manifests/role/cache.pp
@@ -507,30 +507,6 @@
priority => 70,
}
- # The default of 1 ACK per batch causes issues (i.e.:
- # missing messages) when a broker drops out of its leader
- # role. Previous tests showed that requiring of 2 ACKs per
- # batch solved the issue in 4 of 4 instances. So it seems
- # 2 ACKs are sufficient, and we don't need to require ACKs
- # of all ISR machines.
- #
- # While the expected increase in network traffic should
- # not affect network performance, we're only slowly
- # ramping up the number of machines that request more
- # ACKs.
- #
- # For now, it's only bits, mobile, and text.
- # That's roughly 55% of requests and covers the production
- # use of the Analytics cluster.
- # Upload cache cluster will follow, if this works well.
- $topic_request_required_acks = $topic ? {
- 'webrequest_bits' => '2',
- 'webrequest_mobile' => '2',
- 'webrequest_text' => '2',
- default => '1',
- }
-
-
class { '::varnishkafka':
brokers => $kafka_brokers,
topic => $topic,
@@ -550,7 +526,10 @@
batch_num_messages => 6000,
# large timeout to account for potential cross DC latencies
topic_request_timeout_ms => 30000, # request ack timeout
- topic_request_required_acks => $topic_request_required_acks,
+ # By requiring 2 ACKs per message batch, we survive a
+ # single broker dropping out of its leader role,
+ # without seeing lost messages.
+ topic_request_required_acks => '2',
# Write out stats to varnishkafka.stats.json
# this often. This is set at 15 so that
# stats will be fresh when polled from gmetad.
--
To view, visit https://gerrit.wikimedia.org/r/167553
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I72acc4afd3f9699d101db18fa8e0e29ca49ed2ab
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: QChris <[email protected]>
Gerrit-Reviewer: Gage <[email protected]>
Gerrit-Reviewer: Ottomata <[email protected]>
Gerrit-Reviewer: QChris <[email protected]>
Gerrit-Reviewer: jenkins-bot <>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
