Misc. minor house-keeping fixes: add reasonable GC settings, fix up README, fix up example configs, fix the logging for tools, use a log directory for logs instead of the root directory.
Project: http://git-wip-us.apache.org/repos/asf/kafka/repo Commit: http://git-wip-us.apache.org/repos/asf/kafka/commit/14af7132 Tree: http://git-wip-us.apache.org/repos/asf/kafka/tree/14af7132 Diff: http://git-wip-us.apache.org/repos/asf/kafka/diff/14af7132 Branch: refs/heads/trunk Commit: 14af713252f016929f8150f2f6c2df9d0d447ddb Parents: f3f8fa5 Author: Jay Kreps <[email protected]> Authored: Thu Jul 11 08:06:43 2013 -0700 Committer: Jay Kreps <[email protected]> Committed: Mon Aug 5 10:10:18 2013 -0700 ---------------------------------------------------------------------- README.md | 55 +++++++++++------------- bin/kafka-add-partitions.sh | 1 - bin/kafka-check-reassignment-status.sh | 3 +- bin/kafka-console-consumer-log4j.properties | 21 --------- bin/kafka-console-consumer.sh | 5 +-- bin/kafka-console-producer.sh | 4 +- bin/kafka-consumer-perf-test.sh | 1 + bin/kafka-create-topic.sh | 4 +- bin/kafka-delete-topic.sh | 19 -------- bin/kafka-list-topic.sh | 4 +- bin/kafka-preferred-replica-election.sh | 3 +- bin/kafka-producer-perf-test.sh | 1 + bin/kafka-reassign-partitions.sh | 3 +- bin/kafka-replay-log-producer.sh | 4 +- bin/kafka-run-class.sh | 53 +++++++++++++++++++---- bin/kafka-server-start.sh | 8 ++-- bin/kafka-simple-consumer-perf-test.sh | 1 + bin/zookeeper-server-start.sh | 5 ++- config/log4j.properties | 8 ++-- config/producer.properties | 2 +- config/server.properties | 18 +++----- config/tools-log4j.properties | 22 ++++++++++ 22 files changed, 123 insertions(+), 122 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/README.md ---------------------------------------------------------------------- diff --git a/README.md b/README.md index cf0d59e..9879427 100644 --- a/README.md +++ b/README.md @@ -1,59 +1,52 @@ -# Kafka is a distributed publish/subscribe messaging system # +# Apache Kafka # -It is designed to support the following +See our [web site](http://kafka.apache.org) details on the project. -* Persistent messaging with O(1) disk structures that provide constant time performance even with many TB of stored messages. -* High-throughput: even with very modest hardware Kafka can support hundreds of thousands of messages per second. -* Explicit support for partitioning messages over Kafka servers and distributing consumption over a cluster of consumer machines while maintaining per-partition ordering semantics. -* Support for parallel data load into Hadoop. - -Kafka is aimed at providing a publish-subscribe solution that can handle all activity stream data and processing on a consumer-scale web site. This kind of activity (page views, searches, and other user actions) are a key ingredient in many of the social feature on the modern web. This data is typically handled by "logging" and ad hoc log aggregation solutions due to the throughput requirements. This kind of ad hoc solution is a viable solution to providing logging data to an offline analysis system like Hadoop, but is very limiting for building real-time processing. Kafka aims to unify offline and online processing by providing a mechanism for parallel load into Hadoop as well as the ability to partition real-time consumption over a cluster of machines. - -See our [web site](http://kafka.apache.org/) for more details on the project. - -## Contribution ## - -Kafka is a new project, and we are interested in building the community; we would welcome any thoughts or [patches](https://issues.apache.org/jira/browse/KAFKA). You can reach us [on the Apache mailing lists](http://kafka.apache.org/contact.html). - -The Kafka code is available from: - * git clone http://git-wip-us.apache.org/repos/asf/kafka.git kafka - -To contribute you can follow: - * https://cwiki.apache.org/confluence/display/KAFKA/Git+Workflow +## Building It ## To build for all supported versions of Scala: - 1. ./sbt +package To build for a particular version of Scala (either 2.8.0, 2.8.2, 2.9.1 or 2.9.2): 1. ./sbt "++2.8.0 package" *or* ./sbt "++2.8.2 package" *or* ./sbt "++2.9.1 package" *or* ./sbt "++2.9.2 package" +## Running It ## + +To run follow the instructions here: +1. http://kafka.apache.org/08/quickstart.html + +## Other Build Tips ## + Here are some useful sbt commands, to be executed at the sbt command prompt (./sbt). Prefixing with "++<version> " runs the command for a specific Scala version, prefixing with "+" will perform the action for all versions of Scala, and no prefix runs the command for the default (2.8.0) version of Scala. - tasks : Lists all the sbt commands and their descriptions - clean : Deletes all generated files (the target directory). - compile : Compile all the sub projects, but not create the jars - test : Run all unit tests in all sub projects - release-zip : Create all the jars, run unit tests and create a deployable release zip - package: Creates jars for src, test, docs etc - projects : List all the sub projects - project sub_project_name : Switch to a particular sub-project. For example, to switch to the core kafka code, use "project core-kafka" -Following commands can be run only on a particular sub project - - +The following commands can be run only on a particular sub project - test-only package.test.TestName : Runs only the specified test in the current sub project - run : Provides options to run any of the classes that have a main method. For example, you can switch to project java-examples, and run the examples there by executing "project java-examples" followed by "run" For more details please see the [SBT documentation](https://github.com/harrah/xsbt/wiki) +## Contribution ## + +Kafka is a new project, and we are interested in building the community; we would welcome any thoughts or [patches](https://issues.apache.org/jira/browse/KAFKA). You can reach us [on the Apache mailing lists](http://kafka.apache.org/contact.html). + +To contribute follow the instructions here: + * http://kafka.apache.org/contributing.html + +We also welcome patches for the website and documentation which can be found here: + * https://svn.apache.org/repos/asf/kafka/site + + + + http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/bin/kafka-add-partitions.sh ---------------------------------------------------------------------- diff --git a/bin/kafka-add-partitions.sh b/bin/kafka-add-partitions.sh index 7d217e2..c08b837 100755 --- a/bin/kafka-add-partitions.sh +++ b/bin/kafka-add-partitions.sh @@ -15,5 +15,4 @@ # limitations under the License. base_dir=$(dirname $0) -export KAFKA_OPTS="-Xmx512M -server -Dcom.sun.management.jmxremote -Dlog4j.configuration=file:$base_dir/kafka-console-consumer-log4j.properties" $base_dir/kafka-run-class.sh kafka.admin.AddPartitionsCommand $@ http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/bin/kafka-check-reassignment-status.sh ---------------------------------------------------------------------- diff --git a/bin/kafka-check-reassignment-status.sh b/bin/kafka-check-reassignment-status.sh index 4eb9aca..1f21858 100755 --- a/bin/kafka-check-reassignment-status.sh +++ b/bin/kafka-check-reassignment-status.sh @@ -14,5 +14,4 @@ # See the License for the specific language governing permissions and # limitations under the License. -base_dir=$(dirname $0) -$base_dir/kafka-run-class.sh kafka.admin.CheckReassignmentStatus $@ +$(dirname $0)/kafka-run-class.sh kafka.admin.CheckReassignmentStatus $@ http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/bin/kafka-console-consumer-log4j.properties ---------------------------------------------------------------------- diff --git a/bin/kafka-console-consumer-log4j.properties b/bin/kafka-console-consumer-log4j.properties deleted file mode 100644 index 6b76444..0000000 --- a/bin/kafka-console-consumer-log4j.properties +++ /dev/null @@ -1,21 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -log4j.rootLogger=INFO, stderr - -log4j.appender.stderr=org.apache.log4j.ConsoleAppender -log4j.appender.stderr.target=System.err -log4j.appender.stderr.layout=org.apache.log4j.PatternLayout -log4j.appender.stderr.layout.ConversionPattern=[%d] %p %m (%c)%n - http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/bin/kafka-console-consumer.sh ---------------------------------------------------------------------- diff --git a/bin/kafka-console-consumer.sh b/bin/kafka-console-consumer.sh index 8a76c6c..95edc9e 100755 --- a/bin/kafka-console-consumer.sh +++ b/bin/kafka-console-consumer.sh @@ -14,6 +14,5 @@ # See the License for the specific language governing permissions and # limitations under the License. -base_dir=$(dirname $0) -export KAFKA_OPTS="-Xmx512M -server -Dcom.sun.management.jmxremote -Dlog4j.configuration=file:$base_dir/kafka-console-consumer-log4j.properties" -$base_dir/kafka-run-class.sh kafka.consumer.ConsoleConsumer $@ +export KAFKA_HEAP_OPTS="-Xmx512M" +$(dirname $0)/kafka-run-class.sh kafka.consumer.ConsoleConsumer $@ http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/bin/kafka-console-producer.sh ---------------------------------------------------------------------- diff --git a/bin/kafka-console-producer.sh b/bin/kafka-console-producer.sh index 99d2dc6..f102c22 100755 --- a/bin/kafka-console-producer.sh +++ b/bin/kafka-console-producer.sh @@ -14,5 +14,5 @@ # See the License for the specific language governing permissions and # limitations under the License. -base_dir=$(dirname $0) -$base_dir/kafka-run-class.sh kafka.producer.ConsoleProducer $@ +export KAFKA_HEAP_OPTS="-Xmx512M" +$(dirname $0)/kafka-run-class.sh kafka.producer.ConsoleProducer $@ http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/bin/kafka-consumer-perf-test.sh ---------------------------------------------------------------------- diff --git a/bin/kafka-consumer-perf-test.sh b/bin/kafka-consumer-perf-test.sh index 7745ea8..39434d1 100755 --- a/bin/kafka-consumer-perf-test.sh +++ b/bin/kafka-consumer-perf-test.sh @@ -14,4 +14,5 @@ # See the License for the specific language governing permissions and # limitations under the License. +export KAFKA_HEAP_OPTS="-Xmx512M" $(dirname $0)/kafka-run-class.sh kafka.perf.ConsumerPerformance $@ http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/bin/kafka-create-topic.sh ---------------------------------------------------------------------- diff --git a/bin/kafka-create-topic.sh b/bin/kafka-create-topic.sh index fccda7b..7a5f297 100755 --- a/bin/kafka-create-topic.sh +++ b/bin/kafka-create-topic.sh @@ -14,6 +14,4 @@ # See the License for the specific language governing permissions and # limitations under the License. -base_dir=$(dirname $0) -export KAFKA_OPTS="-Xmx512M -server -Dcom.sun.management.jmxremote -Dlog4j.configuration=file:$base_dir/kafka-console-consumer-log4j.properties" -$base_dir/kafka-run-class.sh kafka.admin.CreateTopicCommand $@ +$(dirname $0)/kafka-run-class.sh kafka.admin.CreateTopicCommand $@ http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/bin/kafka-delete-topic.sh ---------------------------------------------------------------------- diff --git a/bin/kafka-delete-topic.sh b/bin/kafka-delete-topic.sh deleted file mode 100755 index f266ae3..0000000 --- a/bin/kafka-delete-topic.sh +++ /dev/null @@ -1,19 +0,0 @@ -#!/bin/bash -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -base_dir=$(dirname $0) -export KAFKA_OPTS="-Xmx512M -server -Dcom.sun.management.jmxremote -Dlog4j.configuration=file:$base_dir/kafka-console-consumer-log4j.properties" -$base_dir/kafka-run-class.sh kafka.admin.DeleteTopicCommand $@ http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/bin/kafka-list-topic.sh ---------------------------------------------------------------------- diff --git a/bin/kafka-list-topic.sh b/bin/kafka-list-topic.sh index 1235ad0..bb2dde5 100755 --- a/bin/kafka-list-topic.sh +++ b/bin/kafka-list-topic.sh @@ -14,6 +14,4 @@ # See the License for the specific language governing permissions and # limitations under the License. -base_dir=$(dirname $0) -export KAFKA_OPTS="-Xmx512M -server -Dcom.sun.management.jmxremote -Dlog4j.configuration=file:$base_dir/kafka-console-consumer-log4j.properties" -$base_dir/kafka-run-class.sh kafka.admin.ListTopicCommand $@ +$(dirname $0)/kafka-run-class.sh kafka.admin.ListTopicCommand $@ http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/bin/kafka-preferred-replica-election.sh ---------------------------------------------------------------------- diff --git a/bin/kafka-preferred-replica-election.sh b/bin/kafka-preferred-replica-election.sh index 9c63e43..6416dc1 100755 --- a/bin/kafka-preferred-replica-election.sh +++ b/bin/kafka-preferred-replica-election.sh @@ -14,5 +14,4 @@ # See the License for the specific language governing permissions and # limitations under the License. -base_dir=$(dirname $0) -$base_dir/kafka-run-class.sh kafka.admin.PreferredReplicaLeaderElectionCommand $@ +$(dirname $0)/kafka-run-class.sh kafka.admin.PreferredReplicaLeaderElectionCommand $@ http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/bin/kafka-producer-perf-test.sh ---------------------------------------------------------------------- diff --git a/bin/kafka-producer-perf-test.sh b/bin/kafka-producer-perf-test.sh index 25c4dce..1935522 100755 --- a/bin/kafka-producer-perf-test.sh +++ b/bin/kafka-producer-perf-test.sh @@ -14,4 +14,5 @@ # See the License for the specific language governing permissions and # limitations under the License. +export KAFKA_HEAP_OPTS="-Xmx512M" $(dirname $0)/kafka-run-class.sh kafka.perf.ProducerPerformance $@ http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/bin/kafka-reassign-partitions.sh ---------------------------------------------------------------------- diff --git a/bin/kafka-reassign-partitions.sh b/bin/kafka-reassign-partitions.sh index 162872c..a6bd148 100755 --- a/bin/kafka-reassign-partitions.sh +++ b/bin/kafka-reassign-partitions.sh @@ -14,5 +14,4 @@ # See the License for the specific language governing permissions and # limitations under the License. -base_dir=$(dirname $0) -$base_dir/kafka-run-class.sh kafka.admin.ReassignPartitionsCommand $@ +$(dirname $0)/kafka-run-class.sh kafka.admin.ReassignPartitionsCommand $@ http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/bin/kafka-replay-log-producer.sh ---------------------------------------------------------------------- diff --git a/bin/kafka-replay-log-producer.sh b/bin/kafka-replay-log-producer.sh index f36e0e2..86943ee 100755 --- a/bin/kafka-replay-log-producer.sh +++ b/bin/kafka-replay-log-producer.sh @@ -14,6 +14,4 @@ # See the License for the specific language governing permissions and # limitations under the License. -base_dir=$(dirname $0) -export KAFKA_OPTS="-Xmx512M -server -Dcom.sun.management.jmxremote -Dlog4j.configuration=file:$base_dir/../config/log4j.properties" -$base_dir/kafka-run-class.sh kafka.tools.ReplayLogProducer $@ +$(dirname $0)/kafka-run-class.sh kafka.tools.ReplayLogProducer $@ http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/bin/kafka-run-class.sh ---------------------------------------------------------------------- diff --git a/bin/kafka-run-class.sh b/bin/kafka-run-class.sh index 1b66655..eb6ff1b 100755 --- a/bin/kafka-run-class.sh +++ b/bin/kafka-run-class.sh @@ -22,10 +22,18 @@ fi base_dir=$(dirname $0)/.. -SCALA_VERSION=2.8.0 +# create logs directory +LOG_DIR=$base_dir/logs +if [ ! -d $LOG_DIR ]; then + mkdir $LOG_DIR +fi + +if [ -z "$SCALA_VERSION" ]; then + SCALA_VERSION=2.8.0 +fi # assume all dependencies have been packaged into one jar with sbt-assembly's task "assembly-package-dependency" -for file in $base_dir/core/target/scala-2.8.0/*.jar; +for file in $base_dir/core/target/scala-${SCALA_VERSION}/*.jar; do CLASSPATH=$CLASSPATH:$file done @@ -46,30 +54,59 @@ do CLASSPATH=$CLASSPATH:$file done +# JMX settings if [ -z "$KAFKA_JMX_OPTS" ]; then KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false " fi -if [ -z "$KAFKA_OPTS" ]; then - KAFKA_OPTS="-Xmx512M -server -Dlog4j.configuration=file:$base_dir/config/log4j.properties" -fi - +# JMX port to use if [ $JMX_PORT ]; then KAFKA_JMX_OPTS="$KAFKA_JMX_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT " fi +# Log4j settings +if [ -z "$KAFKA_LOG4J_OPTS" ]; then + KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$base_dir/config/tools-log4j.properties" +fi + +# Generic jvm settings you want to add +if [ -z "$KAFKA_OPTS" ]; then + KAFKA_OPTS="" +fi + +# Which java to use if [ -z "$JAVA_HOME" ]; then JAVA="java" else JAVA="$JAVA_HOME/bin/java" fi -$JAVA $KAFKA_OPTS $KAFKA_JMX_OPTS -cp $CLASSPATH "$@" +# Memory options +if [ -z "$KAFKA_HEAP_OPTS" ]; then + KAFKA_HEAP_OPTS="-Xmx256M" +fi + +# JVM performance options +if [ -z "$KAFKA_JVM_PERFORMANCE_OPTS" ]; then + KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:+UseCompressedOops -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC" +fi + +# GC options +GC_FILE_SUFFIX='-gc.log' +GC_LOG_FILE_NAME='' +if [ "$1" = "daemon" ] && [ -z "$KAFKA_GC_LOG_OPTS"] ; then + shift + GC_LOG_FILE_NAME=$1$GC_FILE_SUFFIX + shift + KAFKA_GC_LOG_OPTS="-Xloggc:$LOG_DIR/$GC_LOG_FILE_NAME -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps " +fi + +$JAVA $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp $CLASSPATH $KAFKA_OPTS "$@" exitval=$? if [ $exitval -eq "1" ] ; then - $JAVA $KAFKA_OPTS $KAFKA_JMX_OPTS -cp $CLASSPATH "$@" >& exception.txt + $JAVA $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS -cp $CLASSPATH $KAFKA_OPTS "$@" >& exception.txt exception=`cat exception.txt` noBuildMessage='Please build the project using sbt. Documentation is available at http://kafka.apache.org/' pattern="(Could not find or load main class)|(java\.lang\.NoClassDefFoundError)" http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/bin/kafka-server-start.sh ---------------------------------------------------------------------- diff --git a/bin/kafka-server-start.sh b/bin/kafka-server-start.sh index 307f76c..4005289 100755 --- a/bin/kafka-server-start.sh +++ b/bin/kafka-server-start.sh @@ -19,7 +19,7 @@ then echo "USAGE: $0 server.properties" exit 1 fi - -export JMX_PORT=${JMX_PORT:-9999} - -$(dirname $0)/kafka-run-class.sh kafka.Kafka $@ +base_dir=$(dirname $0) +export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$base_dir/../config/log4j.properties" +export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G" +$base_dir/kafka-run-class.sh daemon kafkaServer kafka.Kafka $@ http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/bin/kafka-simple-consumer-perf-test.sh ---------------------------------------------------------------------- diff --git a/bin/kafka-simple-consumer-perf-test.sh b/bin/kafka-simple-consumer-perf-test.sh index a9246cb..94c9ed0 100755 --- a/bin/kafka-simple-consumer-perf-test.sh +++ b/bin/kafka-simple-consumer-perf-test.sh @@ -14,4 +14,5 @@ # See the License for the specific language governing permissions and # limitations under the License. +export KAFKA_HEAP_OPTS="-Xmx512M" $(dirname $0)/kafka-run-class.sh kafka.perf.SimpleConsumerPerformance $@ http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/bin/zookeeper-server-start.sh ---------------------------------------------------------------------- diff --git a/bin/zookeeper-server-start.sh b/bin/zookeeper-server-start.sh index 184a10b..e4746a8 100755 --- a/bin/zookeeper-server-start.sh +++ b/bin/zookeeper-server-start.sh @@ -19,5 +19,8 @@ then echo "USAGE: $0 zookeeper.properties" exit 1 fi +base_dir=$(dirname $0) +export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$base_dir/../config/log4j.properties" +export KAFKA_HEAP_OPTS="-Xmx512M -Xms512M" +$base_dir/kafka-run-class.sh daemon zookeeper org.apache.zookeeper.server.quorum.QuorumPeerMain $@ -$(dirname $0)/kafka-run-class.sh org.apache.zookeeper.server.quorum.QuorumPeerMain $@ http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/config/log4j.properties ---------------------------------------------------------------------- diff --git a/config/log4j.properties b/config/log4j.properties index b76bc94..dcf48f5 100644 --- a/config/log4j.properties +++ b/config/log4j.properties @@ -20,25 +20,25 @@ log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c)%n log4j.appender.kafkaAppender=org.apache.log4j.DailyRollingFileAppender log4j.appender.kafkaAppender.DatePattern='.'yyyy-MM-dd-HH -log4j.appender.kafkaAppender.File=server.log +log4j.appender.kafkaAppender.File=logs/server.log log4j.appender.kafkaAppender.layout=org.apache.log4j.PatternLayout log4j.appender.kafkaAppender.layout.ConversionPattern=[%d] %p %m (%c)%n log4j.appender.stateChangeAppender=org.apache.log4j.DailyRollingFileAppender log4j.appender.stateChangeAppender.DatePattern='.'yyyy-MM-dd-HH -log4j.appender.stateChangeAppender.File=state-change.log +log4j.appender.stateChangeAppender.File=logs/state-change.log log4j.appender.stateChangeAppender.layout=org.apache.log4j.PatternLayout log4j.appender.stateChangeAppender.layout.ConversionPattern=[%d] %p %m (%c)%n log4j.appender.requestAppender=org.apache.log4j.DailyRollingFileAppender log4j.appender.requestAppender.DatePattern='.'yyyy-MM-dd-HH -log4j.appender.requestAppender.File=kafka-request.log +log4j.appender.requestAppender.File=logs/kafka-request.log log4j.appender.requestAppender.layout=org.apache.log4j.PatternLayout log4j.appender.requestAppender.layout.ConversionPattern=[%d] %p %m (%c)%n log4j.appender.controllerAppender=org.apache.log4j.DailyRollingFileAppender log4j.appender.controllerAppender.DatePattern='.'yyyy-MM-dd-HH -log4j.appender.controllerAppender.File=controller.log +log4j.appender.controllerAppender.File=logs/controller.log log4j.appender.controllerAppender.layout=org.apache.log4j.PatternLayout log4j.appender.controllerAppender.layout.ConversionPattern=[%d] %p %m (%c)%n http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/config/producer.properties ---------------------------------------------------------------------- diff --git a/config/producer.properties b/config/producer.properties index 162b8a6..52a7611 100644 --- a/config/producer.properties +++ b/config/producer.properties @@ -16,7 +16,7 @@ ############################# Producer Basics ############################# -# list of brokers used for bootstrapping +# list of brokers used for bootstrapping knowledge about the rest of the cluster # format: host1:port1,host2:port2 ... metadata.broker.list=localhost:9092 http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/config/server.properties ---------------------------------------------------------------------- diff --git a/config/server.properties b/config/server.properties index 0589c71..977b8da 100644 --- a/config/server.properties +++ b/config/server.properties @@ -47,8 +47,8 @@ socket.request.max.bytes=104857600 ############################# Log Basics ############################# -# The directory under which to store log files -log.dir=/tmp/kafka-logs +# A comma seperated list of directories under which to store log files +log.dirs=/tmp/kafka-logs # The number of logical partitions per topic per server. More partitions allow greater parallelism # for consumption, but also mean more files. @@ -56,12 +56,12 @@ num.partitions=2 ############################# Log Flush Policy ############################# -# The following configurations control the flush of data to disk. This is the most +# The following configurations control the flush of data to disk. This is among the most # important performance knob in kafka. # There are a few important trade-offs here: -# 1. Durability: Unflushed data is at greater risk of loss in the event of a crash. -# 2. Latency: Data is not made available to consumers until it is flushed (which adds latency). -# 3. Throughput: The flush is generally the most expensive operation. +# 1. Durability: Unflushed data may be lost if you are not using replication. +# 2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush. +# 3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to exceessive seeks. # The settings below allow one to configure the flush policy to flush data after a period of time or # every N messages (or both). This can be done globally and overridden on a per-topic basis. @@ -107,10 +107,4 @@ zookeeper.connect=localhost:2181 # Timeout in ms for connecting to zookeeper zookeeper.connection.timeout.ms=1000000 -# metrics reporter properties -kafka.metrics.polling.interval.secs=5 -kafka.metrics.reporters=kafka.metrics.KafkaCSVMetricsReporter -kafka.csv.metrics.dir=/tmp/kafka_metrics -# Disable csv reporting by default. -kafka.csv.metrics.reporter.enabled=false http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/config/tools-log4j.properties ---------------------------------------------------------------------- diff --git a/config/tools-log4j.properties b/config/tools-log4j.properties new file mode 100644 index 0000000..7924049 --- /dev/null +++ b/config/tools-log4j.properties @@ -0,0 +1,22 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +log4j.rootLogger=WARN, stdout + +log4j.appender.stdout=org.apache.log4j.ConsoleAppender +log4j.appender.stdout.layout=org.apache.log4j.PatternLayout +log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c)%n + +
