[MediaWiki-commits] [Gerrit] Updating with recent upstream changes to varnishkafka.conf - change (operations...varnishkafka)

2013-10-25 Thread Ottomata (Code Review) 
Ottomata has submitted this change and it was merged.

Change subject: Updating with recent upstream changes to varnishkafka.conf
..


Updating with recent upstream changes to varnishkafka.conf

Change-Id: Id856d1f9a2cbc1bb6f22dc7bd95c2df950a10976
---
M manifests/defaults.pp
M manifests/init.pp
M templates/varnishkafka.conf.erb
3 files changed, 111 insertions(+), 32 deletions(-)

Approvals:
  Ottomata: Verified; Looks good to me, approved



diff --git a/manifests/defaults.pp b/manifests/defaults.pp
index 36e975b..756c444 100644
--- a/manifests/defaults.pp
+++ b/manifests/defaults.pp
@@ -7,9 +7,9 @@
 
 $output = 'kafka'
 $format_type= 'string'
-$format = '%l  %n  %t  
%{Varnish:time_firstbyte}x  %h  %{Varnish:handling}x/%s %b  %m  
http://%{Host}i%U%q -   %{Content-Type}o%{Referer}i 
%{X-Forwarded-For}i %{User-agent!escape}i'
+$format = '%l  %n  %t  
%{Varnish:time_firstbyte}x  %h  %{Varnish:handling}x/%s %b  %m  
http://%{Host}i%U%q -   %{Content-Type}o%{Referer}i 
%{X-Forwarded-For}i %{User-agent!escape}i   %{Accept-Language}i'
 
-$format_key_type= 'kafka'
+$format_key_type= 'string'
 $format_key = undef
 
 $partition  = -1
@@ -17,17 +17,20 @@
 $message_send_max_retries   = 3
 $topic_request_required_acks= 1
 $topic_message_timeout_ms   = 6
+$compression_codec  = 'none'
 
 $varnish_opts   = {
 'm' = 'RxRequest:^(?!PURGE$)',
 }
 $log_data_copy  = true
+$tag_size_max   = 2048
+$log_line_scratch_size  = 4096
+$log_hash_size  = 5000
+$log_hash_max   = 5
 
 $log_level  = 6
-$log_stderr = true
+$log_stderr = false
 $log_syslog = true
-
-$daemon_opts= undef
 
 $conf_template  = 'varnishkafka/varnishkafka.conf.erb'
 $default_template   = 'varnishkafka/varnishkafka.default.erb'
diff --git a/manifests/init.pp b/manifests/init.pp
index 3a39eae..1b9898f 100644
--- a/manifests/init.pp
+++ b/manifests/init.pp
@@ -30,10 +30,24 @@
 # $topic_request_required_acks  - Required ack level.  Default: 1
 # $topic_message_timeout_ms - Local message timeout (milliseconds).
 # Default: 6
+# $compression_codec- Compression codec to use when sending 
batched messages
+# Kafka.  Valid values are 'none', 'gzip', 
and 'snappy'.
+# Default: none
 # $varnish_opts - Arbitrary hash of varnish CLI options.
 # Default: { 'm' = 
'RxRequest:^(?!PURGE$)' }
 # $log_data_copy- If true, log tag data read from VSL files
 # should be copied instantly when read.  
Default true.
+# $tag_size_max - Maximum size of an individual field.  
Field will be truncated
+# if it is larger than this.  Default: 2048
+# $log_line_scratch_size- Size of static log line buffer.  If a 
line is larger than
+# this buffer, temp buffers will be 
allocated.  Set this
+# slighly larger than your expected line 
size.
+# Default: 4096
+# $log_hash_size- Number of hash buckets.  Set this to 
avg_requests_per_second / 5.
+# Default: 5000
+# $log_hash_max - Max number of log lines / bucket.  Set 
this to
+# avg_requests_per_second / $log_hash_size.
+# Default: 5
 # $log_level- varnishkafka log level.  Default 6 
(info).
 # $log_stderr   - Boolean.  Whether to log to stderr.  
Default: true
 # $log_syslog   - Boolean.  Whether to log to syslog.  
Default: true
@@ -58,15 +72,18 @@
 $message_send_max_retries   = 
$varnishkafka::defaults::message_send_max_retries,
 $topic_request_required_acks= 
$varnishkafka::defaults::topic_request_required_acks,
 $topic_message_timeout_ms   = 
$varnishkafka::defaults::topic_message_timeout_ms,
+$compression_codec  = 
$varnishkafka::defaults::compression_codec,
 
 $varnish_opts   = $varnishkafka::defaults::varnish_opts,
 $log_data_copy  =

[MediaWiki-commits] [Gerrit] Updating with recent upstream changes to varnishkafka.conf - change (operations...varnishkafka)

2013-10-24 Thread Ottomata (Code Review) 
Ottomata has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/91664


Change subject: Updating with recent upstream changes to varnishkafka.conf
..

Updating with recent upstream changes to varnishkafka.conf

Change-Id: Id856d1f9a2cbc1bb6f22dc7bd95c2df950a10976
---
M manifests/defaults.pp
M manifests/init.pp
M templates/varnishkafka.conf.erb
3 files changed, 72 insertions(+), 25 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet/varnishkafka 
refs/changes/64/91664/1

diff --git a/manifests/defaults.pp b/manifests/defaults.pp
index 36e975b..e23e4ea 100644
--- a/manifests/defaults.pp
+++ b/manifests/defaults.pp
@@ -7,9 +7,9 @@
 
 $output = 'kafka'
 $format_type= 'string'
-$format = '%l  %n  %t  
%{Varnish:time_firstbyte}x  %h  %{Varnish:handling}x/%s %b  %m  
http://%{Host}i%U%q -   %{Content-Type}o%{Referer}i 
%{X-Forwarded-For}i %{User-agent!escape}i'
+$format = '%l  %n  %t  
%{Varnish:time_firstbyte}x  %h  %{Varnish:handling}x/%s %b  %m  
http://%{Host}i%U%q -   %{Content-Type}o%{Referer}i 
%{X-Forwarded-For}i %{User-agent!escape}i   %{Accept-Language}i'
 
-$format_key_type= 'kafka'
+$format_key_type= 'string'
 $format_key = undef
 
 $partition  = -1
@@ -17,6 +17,7 @@
 $message_send_max_retries   = 3
 $topic_request_required_acks= 1
 $topic_message_timeout_ms   = 6
+$compression_codec  = 'none'
 
 $varnish_opts   = {
 'm' = 'RxRequest:^(?!PURGE$)',
@@ -24,10 +25,8 @@
 $log_data_copy  = true
 
 $log_level  = 6
-$log_stderr = true
+$log_stderr = false
 $log_syslog = true
-
-$daemon_opts= undef
 
 $conf_template  = 'varnishkafka/varnishkafka.conf.erb'
 $default_template   = 'varnishkafka/varnishkafka.default.erb'
diff --git a/manifests/init.pp b/manifests/init.pp
index 3a39eae..5c6edb2 100644
--- a/manifests/init.pp
+++ b/manifests/init.pp
@@ -30,6 +30,9 @@
 # $topic_request_required_acks  - Required ack level.  Default: 1
 # $topic_message_timeout_ms - Local message timeout (milliseconds).
 # Default: 6
+# $compression_codec- Compression codec to use when sending 
batched messages
+# Kafka.  Valid values are 'none', 'gzip', 
and 'snappy'.
+# Default: none
 # $varnish_opts - Arbitrary hash of varnish CLI options.
 # Default: { 'm' = 
'RxRequest:^(?!PURGE$)' }
 # $log_data_copy- If true, log tag data read from VSL files
@@ -58,6 +61,7 @@
 $message_send_max_retries   = 
$varnishkafka::defaults::message_send_max_retries,
 $topic_request_required_acks= 
$varnishkafka::defaults::topic_request_required_acks,
 $topic_message_timeout_ms   = 
$varnishkafka::defaults::topic_message_timeout_ms,
+$compression_codec  = 
$varnishkafka::defaults::compression_codec,
 
 $varnish_opts   = $varnishkafka::defaults::varnish_opts,
 $log_data_copy  = $varnishkafka::defaults::log_data_copy,
@@ -65,8 +69,6 @@
 $log_level  = $varnishkafka::defaults::log_level,
 $log_stderr = $varnishkafka::defaults::log_stderr,
 $log_syslog = $varnishkafka::defaults::log_syslog,
-
-$daemon_opts= $varnishkafka::defaults::daemon_opts,
 
 $conf_template  = $varnishkafka::defaults::conf_template,
 $default_template   = $varnishkafka::defaults::default_template
diff --git a/templates/varnishkafka.conf.erb b/templates/varnishkafka.conf.erb
index 54982be..6d1f32d 100644
--- a/templates/varnishkafka.conf.erb
+++ b/templates/varnishkafka.conf.erb
@@ -87,20 +87,20 @@
 format.type = %= @format_type %
 format = %= @format %
 
-# Optional secondary formatting.
-#   'output = kafka':  The rendered 'format.key' will be provided as the
-#  Kafka message Key
-#   'output = string': Print string to stdout.
-# Supports the same formating and type as 'format' and 'format.type'.
-#
 % if @format_key -%
 format.key.type = %= @format_key_type %
 format.key = %= @format_key %
 % else -%
 # format.key.type = %= @format_key_type %
-# format.key = %h
+# format.key = %l
 % end -%
 
+
+
+# The maximum accepted log tag size.
+# Larger tags will be truncated to this size.
+# Defaults to 2048