Elukey has submitted this change and it was merged.
Change subject: Set Kafka default cleanup policy to 'delete' to avoid any
compaction with 0.9
......................................................................
Set Kafka default cleanup policy to 'delete' to avoid any compaction with 0.9
The actual Kafka 0.9 config triggers a strange behavior after broker restart,
namely all log data get mtime=now skewing our weekely log deletion policies.
While checking the logs on kafka1013, the last broker that got restarted and
showed this behavior, I found:
[2016-05-24 13:12:05,573] INFO Compaction for partition [webrequest_upload,9]
is resumed (kafka.log.LogCleaner)
[2016-05-24 13:12:05,790] INFO Compaction for partition [webrequest_text,2]
is resumed (kafka.log.LogCleaner)
This is really weird since we don't set anywhere the new "compact" deletion
policy but we rely on the fact that "delete" is the default. Compaction
might explain why we get mtime=now for data logs. An interesting note from
the 0.9 changelog:
"Configuration parameter log.cleaner.enable is now true by default.
This means topics with a cleanup.policy=compact will now be compacted by
default"
From what I can see the kafka default class is not applied anymore (in which
we used to set the "delete" cleanup setting), so I am wondering if Kafka
enables compaction for some reason when cleanup is not explicitly set.
Change-Id: Ifd3514010016766bba9fd41262fbb086cf28e1bc
---
M modules/confluent/manifests/kafka/broker.pp
1 file changed, 4 insertions(+), 4 deletions(-)
Approvals:
Elukey: Looks good to me, approved
Ottomata: Looks good to me, but someone else must approve
jenkins-bot: Verified
diff --git a/modules/confluent/manifests/kafka/broker.pp
b/modules/confluent/manifests/kafka/broker.pp
index 63df4dd..abbc2f2 100644
--- a/modules/confluent/manifests/kafka/broker.pp
+++ b/modules/confluent/manifests/kafka/broker.pp
@@ -124,9 +124,9 @@
#
# [*log_cleanup_policy*]
# Designates the retention policy to use on old log segments. 'delete' will
-# discard old segments when their retention time or size limit has been
-# reached. 'compact' will enable log compaction.
-# Default: undef (delete)
+# discard old segments when their retention time or size limit has been
+# reached. 'compact' will enable log compaction.
+# Default: delete
#
# [*offsets_retention_minutes*]
# Log retention window in minutes for offsets topic.
@@ -212,7 +212,7 @@
$log_segment_bytes = undef,
$log_retention_check_interval_ms = undef,
- $log_cleanup_policy = undef,
+ $log_cleanup_policy = 'delete',
$offsets_retention_minutes = 10080, # 1 week
--
To view, visit https://gerrit.wikimedia.org/r/291697
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ifd3514010016766bba9fd41262fbb086cf28e1bc
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey <[email protected]>
Gerrit-Reviewer: Elukey <[email protected]>
Gerrit-Reviewer: Joal <[email protected]>
Gerrit-Reviewer: Ottomata <[email protected]>
Gerrit-Reviewer: jenkins-bot <>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
