Ottomata has uploaded a new change for review. https://gerrit.wikimedia.org/r/227768
Change subject: Debianize 0.8.2.1 tag ...................................................................... Debianize 0.8.2.1 tag This renames the kafka-client package to kafka-cli, in order to disambiguate the package name from the kafka-clients.jar shipped with kafka-common. kafka-examples are no longer shipped as part of the kafka-common package, and kafka-perf no longer exists as its own jar. Any ext/ jars that are shipped along with the packages have been intentially named with their version numbers, so as to avoid potential naming conflicts with packages that might come from Debian or Ubuntu. This won't avoid all conflicts, but it will likely avoid most. NOTE: I had trouble with merge conflicts not resolving properly when merging 0.8.2.1 into the old debian branch (master). Even when always choosing 'theirs', many of the automatically merged files ended up not matching what was in the 0.8.2.1 tag, which caused the build process to fail when comparing the source in the debian branch to the orig.tar.gz file created from the tag. To work around, I have recreated the debianization directly on top of the 0.8.2.1 tag by manually copying the old debian/ and ext/ dirs, and then modifying them to work with 0.8.2.1. The old git history for the debian/ and ext/ dirs can be found in the master branch of this repository (unless we decide to rename this branch to avoid confusion). Change-Id: I69b23e4270eafb5b2912723c8a8591cbf0f82f63 --- A debian/README.Debian A debian/bin/kafka A debian/changelog A debian/compat A debian/control A debian/copyright A debian/gbp.conf A debian/kafka-cli.dirs A debian/kafka-cli.install A debian/kafka-cli.postinst A debian/kafka-cli.postrm A debian/kafka-common.dirs A debian/kafka-common.install A debian/kafka-common.links A debian/kafka-mirror.default A debian/kafka-mirror.dirs A debian/kafka-mirror.init A debian/kafka-server.dirs A debian/kafka-server.kafka.default A debian/kafka-server.kafka.init A debian/kafka-server.postinst A debian/patches/kafka_data_dirs_fixes.patch A debian/patches/logging_to_var_log.patch A debian/patches/our-own-build-system.patch A debian/patches/series A debian/rules A debian/source/format A debian/source/include-binaries A debian/source/options A ext/avro-1.4.0.jar A ext/commons-collections-2.1.1.jar A ext/easymock-3.0.jar A ext/hadoop-core-0.20.2.jar A ext/jasper-compiler-5.5.12.jar A ext/jasper-runtime-5.5.12.jar A ext/jopt-simple-3.2.jar A ext/jsp-2.1-6.1.14.jar A ext/jsp-api-2.1-6.1.14.jar A ext/kafka-ganglia-1.0.0.jar A ext/kfs-0.3.jar A ext/lz4-1.2.0.jar A ext/metrics-annotation-2.2.0.jar A ext/metrics-core-2.2.0.jar A ext/metrics-ganglia-2.2.0.jar A ext/metrics-graphite-2.2.0.jar A ext/paranamer-2.1.jar A ext/paranamer-ant-2.2.jar A ext/paranamer-generator-2.2.jar A ext/pig-0.8.0.jar A ext/piggybank-0.12.0.jar A ext/scalatest-1.2.jar A ext/snappy-java-1.0.4.1.jar A ext/xmlenc-0.52.jar A ext/zkclient-0.3.jar 54 files changed, 1,839 insertions(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/debs/kafka refs/changes/68/227768/1 diff --git a/debian/README.Debian b/debian/README.Debian new file mode 100644 index 0000000..c92fa6d --- /dev/null +++ b/debian/README.Debian @@ -0,0 +1,138 @@ +kafka for Debian +---------------- +This package is created using a custom build system based on Makefiles +instead of the standard sbt/gradle build system used by upstream. The reason +for this is to satisfy debian policy that no internet connection should +be required during package building as well as security concers about +the downloaded JAR files by sbt + +How to build this package +------------------------- + +This repository is be used to build the kafka package using +git-buildpackage. Below we illustrate one of the ways this can work for +you: + +* Install some mandatory packages + + $ apt-get install git-buildpackage + +* And some not so mandatory but nice to have packages + + $ apt-get install pbuilder cowbuilder + +Note that pbuilder/cowbuilder is not strictly required but it will save +you from installing the dependencies manually + +* Create a cowbuilder environment (we assume you are building against +precise pangolin here) + + $ sudo DIST=jessie git-pbuilder create --components "main universe" + +* Clone the repo + + $ git clone http://gerrit.wikimedia.org/r/operations/debs/kafka + $ cd kafka + +* Build + + $ DIST=jessie git-buildpackage -us -uc + +or if you dont want to use git-pbuilder: + + $ git-buildpackage -us -uc --git-builder=debuild + +* Grab your packages from the directory above the one your are in, unless +you have configured pbuilder otherwise + +Build a new version +------------------- + +* Should the current version in the repo not be enough, set upstream and + fetch + + $ git remote add kafka http://git-wip-us.apache.org/repos/asf/kafka.git + $ git fetch kafka + +* Tag the version you want to build against + + $ git tag upstream/<VERSION> <commit-ish> + +<VERSION> should be a valid upstream version according to the debian +specification: + +https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Version + +<commit-sh> can be any upstream commit like git tag, branch or a single +commit. The best result will probably be achieved with a valid git tag +from upstream that matches the debian upstream version specs. In that +case <VERSION> == <commit-ish> + +* Then update the debian branch to reflect that + + $ git checkout debian # Assuming you are not there already + $ git merge upstream/<VERSION> + +* Solve any potential conflicts that might arise. A very helpful way to +do this and avoid doing too much work is: + + $ git checkout --theirs -- . + +This works because all of our changes are in debian/ so in any kind of +conflict the newer version from upstream should be used + +* Add a new debian/changelog entry + + $ dch -i (don't forget to commit) + +* And build as above + +* Solve any problems that might arise and repeat the build + +* Should you be happy with the result, rebuild with a new debian tag. +Just add the --git-tag argument on the previous command and a new +debian/<VERSION>-revision should be present. + +* Push :-). git review might not be your best friend in this btw +Don't forget to push your new tag + +* NOTE: Do not push new branches either upstream or yours unless you really +know what you are doing + +Build trunk +----------- + +The above procedure also works for building from trunk with a few +changes: + +* You will have to come up with a <VERSION> to put in the changelog + +* <commit-ish> == trunk + +* Instead of + + $ git checkout master + $ git merge upstream/<VERSION> + +you do a: + + $ git checkout trunk (it should be linked to kafka/trunk) + $ git checkout master + $ git merge trunk + +* The build is going to be a bit different since we now base it on a branch +and not a tag: + + $ DIST=precise git-buildpackage -us -uc --git-upstream-tree=branch \ +--git-upstream-branch=trunk + +* You will probably need to solve various issues. Some might be changed +files in trunk that have other versions in master, files that might need +to be deleted/altered by the debian package or a variety of other +issues. The kafka source tree is not the fairest of them all. +Hints that might help: + +git checkout trunk -- <path> might come in handy +debian/patches will also need changes + +but you will need your good sense in most cases diff --git a/debian/bin/kafka b/debian/bin/kafka new file mode 100755 index 0000000..14cf876 --- /dev/null +++ b/debian/bin/kafka @@ -0,0 +1,372 @@ +#!/bin/sh + +# kafka +# +# Wrapper script for Kafka executable classes. +# This script is a single replacement for the +# shell scripts shipped with Kafka source bin/ +# directory. +# + +SCRIPT_NAME=$(basename "$0") +KAFKA_CONFIG=${KAFKA_CONFIG:-/etc/kafka} +LOG_DIR=${LOG_DIR:-/var/log/kafka} +SYS_CLASSDIR=${SYS_CLASSDIR:-/usr/share/java} +KAFKA_CLASSDIR=${KAFKA_CLASSDIR:-/usr/share/kafka} + +# Set ZOOKEEPER_OPT if ZOOKEEPER_URL is set and --zookeeper has not +# also been passed in as a CLI arg. This will be included +# in command functions that take a --zookeeper argument. +if [ -n "${ZOOKEEPER_URL}" -a -z "$(echo $@ | grep -- --zookeeper)" ]; then + ZOOKEEPER_OPT="--zookeeper ${ZOOKEEPER_URL}" +fi + +# Set BROKER_LIST_OPT if BROKER_LIST is set and --broker-list has not +# also been passed in as a CLI arg. This will be included +# in command functions that take a --broker-list argument. +if [ -n "${BROKER_LIST}" -a -z "$(echo $@ | grep -- --broker-list)" ]; then + BROKER_LIST_OPT="--broker-list ${BROKER_LIST}" +fi + + +USAGE="Usage: + +$SCRIPT_NAME <command> [opts] +Run $SCRIPT_NAME <command> with zero arguments/options to see command usage. + +Commands: + $SCRIPT_NAME topic [opts] + + $SCRIPT_NAME console-producer [opts] + $SCRIPT_NAME console-consumer [opts] + $SCRIPT_NAME simple-consumer-shell [opts] + $SCRIPT_NAME replay-log-producer [opts] + + $SCRIPT_NAME mirror-maker [opts] + $SCRIPT_NAME consumer-offset-checker [opts] + + $SCRIPT_NAME reassign-partitions [opts] + $SCRIPT_NAME preferred-replica-election [opts] + $SCRIPT_NAME replica-verification-tool [opts] + + $SCRIPT_NAME producer-perf-test [opts] + $SCRIPT_NAME consumer-perf-test [opts] + $SCRIPT_NAME simple-consumer-perf-test [opts] + + $SCRIPT_NAME server-start [-daemon] <server.properties> (Default: $KAFKA_CONFIG/server.properties) + $SCRIPT_NAME server-stop + + $SCRIPT_NAME zookeeper-start [-daemon] <zookeeper.properties> (Default: $KAFKA_CONFIG/zookeeper.properties) + $SCRIPT_NAME zookeeper-stop + $SCRIPT_NAME zookeeper-shell [opts] + +Environment Variables: + ZOOKEEPER_URL - If this is set, any commands that take a --zookeeper flag will be passed with this value. + BROKER_LIST - If this is set, any commands that take a --broker-list flag will be passed with this value. + KAFKA_CONFIG - location of Kafka config files. Default: /etc/kafka + JMX_PORT - Set this to expose JMX. This is set by default for brokers and producers. + KAFKA_JVM_PERFORMANCE_OPTS - Any special JVM perfomance options. This is set by default. + KAFKA_HEAP_OPTS - Any special JVM memory heap options. This is set by default. + KAFKA_LOG4J_OPTS - Any log4j options. Especially -Dlog4j.configuration. This is set by default. + JAVA_OPTS - Any extra JVM options. + KAFKA_OPTS - Any extra options you want to pass. +" + +usage() { echo "${USAGE}"; } + + +# Builds CLASSPATH and execs java with appropriate main class +kafka_run_class() { + if [ $# -lt 1 ]; + then + echo "USAGE: $0 [-daemon] [-name servicename] [-loggc] classname [opts]" + exit 1 + fi + + if [ -z "$SCALA_VERSION" ]; then + SCALA_VERSION=2.9.2 + fi + + # if these were not yet set (by user or defaults file), + # then go ahead and set them. + if [ -z "$KAFKA_JMX_OPTS" ]; then + KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false " + fi + if [ -n "$JMX_PORT" ]; then + KAFKA_JMX_OPTS="$KAFKA_JMX_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT " + fi + + # Log4j settings + if [ -z "$KAFKA_LOG4J_OPTS" ]; then + # Use tools-log4j.properties to configure loggers for CLIs. + # Zookeeper uses SLF4J simple logger, and is noisy by default, so make + # it always WARN level. + KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$KAFKA_CONFIG/tools-log4j.properties -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN" + fi + + if [ -z "$KAFKA_OPTS" ]; then + KAFKA_OPTS="" + fi + + if [ -z "$JAVA_HOME" ]; then + JAVA="java" + else + JAVA="$JAVA_HOME/bin/java" + fi + + # Memory options + if [ -z "$KAFKA_HEAP_OPTS" ]; then + KAFKA_HEAP_OPTS="-Xmx256M" + fi + + # JVM performance options + if [ -z "$KAFKA_JVM_PERFORMANCE_OPTS" ]; then + KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:+UseCompressedOops -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true" + fi + + # Other Java options + if [ -z "$JAVA_OPTS" ]; then + JAVA_OPTS="" + fi + + while [ $# -gt 0 ]; do + COMMAND=$1 + case $COMMAND in + -name) + DAEMON_NAME=$2 + CONSOLE_OUTPUT_FILE=$LOG_DIR/$DAEMON_NAME.out + shift 2 + ;; + -loggc) + if [ -z "$KAFKA_GC_LOG_OPTS"] ; then + GC_LOG_ENABLED="true" + fi + shift + ;; + -daemon) + DAEMON_MODE="true" + shift + ;; + *) + break + ;; + esac + done + + # GC options + GC_FILE_SUFFIX='-gc.log' + GC_LOG_FILE_NAME='' + if [ "x$GC_LOG_ENABLED" = "xtrue" ]; then + GC_LOG_FILE_NAME=$DAEMON_NAME$GC_FILE_SUFFIX + KAFKA_GC_LOG_OPTS="-Xloggc:$LOG_DIR/$GC_LOG_FILE_NAME -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps " + fi + + # Setting the classpath to all the needed dependencies. + SYSTEM_JARS=$SYS_CLASSDIR/asm3.jar:$SYS_CLASSDIR/cglib.jar:$SYS_CLASSDIR/commons-cli.jar:$SYS_CLASSDIR/commons-codec.jar:$SYS_CLASSDIR/commons-compress.jar:$SYS_CLASSDIR/commons-el.jar:$SYS_CLASSDIR/commons-httpclient.jar:$SYS_CLASSDIR/commons-io.jar:$SYS_CLASSDIR/commons-lang.jar:$SYS_CLASSDIR/commons-logging.jar:$SYS_CLASSDIR/commons-net.jar:$SYS_CLASSDIR/hsqldb.jar:$SYS_CLASSDIR/jackson-core.jar:$SYS_CLASSDIR/jackson-mapper.jar:$SYS_CLASSDIR/jets3t.jar:$SYS_CLASSDIR/jetty-util.jar:$SYS_CLASSDIR/jetty.jar:$SYS_CLASSDIR/joda-time.jar:$SYS_CLASSDIR/jsp-api-2.1.jar:$SYS_CLASSDIR/junit4.jar:$SYS_CLASSDIR/log4j-1.2.jar:$SYS_CLASSDIR/objenesis.jar:$SYS_CLASSDIR/oro.jar:$SYS_CLASSDIR/qdox.jar:$SYS_CLASSDIR/scala-library.jar:$SYS_CLASSDIR/servlet-api-2.5.jar:$SYS_CLASSDIR/slf4j-api.jar:$SYS_CLASSDIR/slf4j-simple.jar:$SYS_CLASSDIR/velocity-1.7.jar:$SYS_CLASSDIR/zookeeper.jar + KAFKA_INCLUDED_JARS=$SYS_CLASSDIR/avro-1.4.0.jar:$SYS_CLASSDIR/commons-collections-2.1.1.jar.jar:$SYS_CLASSDIR/easymock-3.0.jar:$SYS_CLASSDIR/hadoop-core-0.20.2.jar:$SYS_CLASSDIR/jasper-compiler-5.5.12.jar:$SYS_CLASSDIR/jasper-runtime-5.5.12.jar:$SYS_CLASSDIR/jopt-simple-3.2.jar:$SYS_CLASSDIR/jsp-2.1-6.1.14.jar:$SYS_CLASSDIR/jsp-api-2.1-6.1.14.jar:$SYS_CLASSDIR/kfs-0.3.jar:$SYS_CLASSDIR/lz4-1.2.0.jar:$SYS_CLASSDIR/metrics-annotation-2.2.0.jar:$SYS_CLASSDIR/metrics-core-2.2.0.jar:$SYS_CLASSDIR/metrics-ganglia-2.2.0.jar:$SYS_CLASSDIR/metrics-graphite-2.2.0.jar:$SYS_CLASSDIR/paranamer-ant-2.2.jar:$SYS_CLASSDIR/paranamer-generator-2.2.jar:$SYS_CLASSDIR/paranamer-2.1.jar:$SYS_CLASSDIR/pig-0.8.0.jar:$SYS_CLASSDIR/piggybank-0.12.0.jar:$SYS_CLASSDIR/scalatest-1.2.jar:$SYS_CLASSDIR/snappy-java-1.0.4.1.jar:$SYS_CLASSDIR/xmlenc-0.52.jar:$SYS_CLASSDIR/zkclient-0.3.jar + KAFKA_JARS=$KAFKA_CLASSDIR/hadoop-consumer.jar:$KAFKA_CLASSDIR/hadoop-producer.jar:$KAFKA_CLASSDIR/kafka-clients.jar:$KAFKA_CLASSDIR/kafka-ganglia-1.0.0.jar:$KAFKA_CLASSDIR/kafka.jar + + CLASSPATH=$CLASSPATH:$SYSTEM_JARS:$KAFKA_INCLUDED_JARS:$KAFKA_JARS + + # Launch mode + if [ "x$DAEMON_MODE" = "xtrue" ]; then + nohup $JAVA $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS $JAVA_OPTS -cp $CLASSPATH $KAFKA_OPTS "$@" > "$CONSOLE_OUTPUT_FILE" 2>&1 < /dev/null & + else + exec $JAVA $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS $JAVA_OPTS -cp $CLASSPATH $KAFKA_OPTS "$@" + fi +} + +cmd_topic() { + kafka_run_class kafka.admin.TopicCommand $ZOOKEEPER_OPT "$@" +} + +cmd_console_producer() { + export JMX_PORT="${JMX_PORT:-9990}" + export KAFKA_HEAP_OPTS="${KAFKA_HEAP_OPTS:--Xmx512M}" + kafka_run_class kafka.tools.ConsoleProducer $BROKER_LIST_OPT "$@" +} + +cmd_console_consumer() { + export KAFKA_HEAP_OPTS="${KAFKA_HEAP_OPTS:--Xmx512M}" + kafka_run_class kafka.tools.ConsoleConsumer $ZOOKEEPER_OPT "$@" +} + +cmd_simple_consumer_shell() { + kafka_run_class kafka.tools.SimpleConsumerShell "$@" +} + +cmd_consumer_offset_checker() { + kafka_run_class kafka.tools.ConsumerOffsetChecker $ZOOKEEPER_OPT "$@" +} + +cmd_replay_log_producer() { + export JMX_PORT="${JMX_PORT:-9992}" + kafka_run_class kafka.tools.ReplayLogProducer $BROKER_LIST_OPT $ZOOKEEPER_OPT "$@" +} + +cmd_replica_verification_tool() { + export JMX_PORT="${JMX_PORT:-9992}" + kafka_run_class kafka.tools.ReplicaVerificationTool "$@" +} + +cmd_mirror_maker() { + EXTRA_ARGS="-name kafkaMirror -loggc" + + COMMAND=$1 + case $COMMAND in + -daemon) + EXTRA_ARGS="-daemon "$EXTRA_ARGS + shift + ;; + *) + ;; + esac + + export JMX_PORT="${JMX_PORT:-9993}" + export KAFKA_LOG4J_OPTS="${KAFKA_LOG4J_OPTS:--Dlog4j.configuration=file:${KAFKA_CONFIG}/log4j.properties}" + export KAFKA_HEAP_OPTS="${KAFKA_HEAP_OPTS:--Xmx1G -Xms1G}" + kafka_run_class ${EXTRA_ARGS} kafka.tools.MirrorMaker "$@" +} + +cmd_consumer_offset_checker() { + kafka_run_class kafka.tools.ConsumerOffsetChecker $ZOOKEEPER_OPT "$@" +} + +cmd_reassign_partitions() { + kafka_run_class kafka.admin.ReassignPartitionsCommand $BROKER_LIST_OPT $ZOOKEEPER_OPT "$@" +} + +cmd_preferred_replica_election() { + kafka_run_class kafka.admin.PreferredReplicaLeaderElectionCommand $ZOOKEEPER_OPT "$@" +} + +cmd_producer_perf_test() { + export KAFKA_HEAP_OPTS="${KAFKA_HEAP_OPTS:-"-Xmx512M"}" + kafka_run_class kafka.tools.ProducerPerformance $BROKER_LIST_OPT "$@" +} +cmd_consumer_perf_test() { + export KAFKA_HEAP_OPTS="${KAFKA_HEAP_OPTS:-"-Xmx512M"}" + kafka_run_class kafka.tools.ConsumerPerformance $ZOOKEEPER_OPT "$@" +} + +cmd_server_start() { + EXTRA_ARGS="-name kafkaServer -loggc" + + COMMAND=$1 + case $COMMAND in + -daemon) + EXTRA_ARGS="-daemon "$EXTRA_ARGS + shift + ;; + *) + ;; + esac + + # default server.properties to $KAFKA_CONFIG/server.properties + server_properties=${1:-"${KAFKA_CONFIG}/server.properties"} + + export JMX_PORT="${JMX_PORT:-9999}" + export KAFKA_LOG4J_OPTS="${KAFKA_LOG4J_OPTS:--Dlog4j.configuration=file:${KAFKA_CONFIG}/log4j.properties}" + export KAFKA_HEAP_OPTS="${KAFKA_HEAP_OPTS:--Xmx1G -Xms1G}" + + kafka_run_class ${EXTRA_ARGS} kafka.Kafka "${server_properties}" +} + +cmd_server_stop() { + ps ax | grep -i 'kafka\.Kafka' | grep java | grep -v grep | awk '{print $1}' | xargs kill -SIGTERM +} + +cmd_zookeeper_start() { + EXTRA_ARGS="-name zookeeper -loggc" + + COMMAND=$1 + case "${COMMAND}" in + -daemon) + EXTRA_ARGS="-daemon ${EXTRA_ARGS}" + shift + ;; + *) + ;; + esac + + # default zookeeper.properties to $KAFKA_CONFIG/zookeeper.properties + zookeeper_properties=${1:-"${KAFKA_CONFIG}/zookeeper.properties"} + + export JMX_PORT="${JMX_PORT:-9998}" + export KAFKA_LOG4J_OPTS="${KAFKA_LOG4J_OPTS:--Dlog4j.configuration=file:${KAFKA_CONFIG}/log4j.properties}" + export KAFKA_HEAP_OPTS="${KAFKA_HEAP_OPTS:--Xmx512M -Xms512M}" + + kafka_run_class ${EXTRA_ARGS} org.apache.zookeeper.server.quorum.QuorumPeerMain "${zookeeper_properties}" +} + +cmd_zookeeper_stop() { + ps ax | grep -i 'zookeeper' | grep -v grep | awk '{print $1}' | xargs kill -SIGINT +} + +cmd_zookeeper_shell() { + if [ -z "${ZOOKEEPER_URL}" -a $# -eq 0 ]; + then + echo "USAGE: ${SCRIPT_NAME} zookeeper-shell zookeeper_host:port[/path] [args...]" + exit 1 + fi + + # Prefer the passed in $1 over ZOOKEEPER_URL for -server arg + if [ -n "$1" ]; + then + ARGS="-server $@" + else + ARGS="-server ${ZOOKEEPER_URL} $@" + fi + + kafka_run_class org.apache.zookeeper.ZooKeeperMain $ARGS +} + + + + +# if no commands, print usage. +if [ $# -lt 1 ]; then + usage && exit 0 +fi + +# parse cli args +while test $# != 0 +do + case "$1" in + topic | \ + console-producer | \ + console-consumer | \ + simple-consumer-shell | \ + replay-log-producer | \ + mirror-maker | \ + consumer-offset-checker | \ + reassign-partitions | \ + preferred-replica-election | \ + replica-verification-tool | \ + producer-perf-test | \ + consumer-perf-test | \ + simple-consumer-perf-test | \ + server-start | \ + server-stop | \ + zookeeper-start | \ + zookeeper-stop | \ + zookeeper-shell ) + command="$1"; shift; break; + shift; + break; + ;; + -h|--help|help) + usage + exit 0 + ;; + *) + echo "Invalid command : '$1'. Aborting." >&2 + exit 1 + ;; + esac +done + +# convert dashes to underscores +function_name=$(echo "${command}" | tr - _) +# call the command function +"cmd_$function_name" "$@" diff --git a/debian/changelog b/debian/changelog new file mode 100644 index 0000000..7f7d889 --- /dev/null +++ b/debian/changelog @@ -0,0 +1,75 @@ +kafka (0.8.2.1-1) jessie; urgency=low + + * Upstream release + + -- Andrew Otto (WMF) <[email protected]> Thu, 23 Jul 2015 16:09:40 +0000 + +kafka (0.8.1.1-6) precise; urgency=low + + * Use generic snappy version in bin/kafka CLASSPATH + * Remove duplicate slf4j-api.jar in bin/kafka CLASSPATH definition + * Remove same duplicate in debian/rules + * Fix typo in debian/rules + + -- Federico Giraud <[email protected]> Wed, 26 Nov 2014 12:12:30 -0800 + +kafka (0.8.1.1-5) precise; urgency=low + + * Increase kill timeout on shutdown. + + -- Federico Giraud <[email protected]> Mon, 03 Nov 2014 14:09:20 -0800 + +kafka (0.8.1.1-4) precise; urgency=low + + * Export $CLASSPATH from init scripts + + -- Patrick Lucas <[email protected]> Wed, 22 Oct 2014 18:17:57 -0700 + +kafka (0.8.1.1-3) precise; urgency=low + + * chown /var/spool/kafka/ in kafka-server.postinst + * Support extra java opts from $JAVA_OPTS + + -- Patrick Lucas <[email protected]> Tue, 23 Sep 2014 13:44:55 -0700 + +kafka (0.8.1.1-2) precise; urgency=low + + * kafka-mirror is now in its own package + + -- Andrew Otto (WMF) <[email protected]> Thu, 31 Jul 2014 19:07:40 +0000 + +kafka (0.8.1.1-1) precise; urgency=low + + * Build new version 0.8.1.1 + + -- Andrew Otto (WMF) <[email protected]> Wed, 16 Jul 2014 18:22:45 +0000 + +kafka (0.8.0-2) precise; urgency=low + + * Adding support for setting ulimit open files. + + -- Andrew Otto (WMF) <[email protected]> Wed, 05 Feb 2014 15:49:05 +0000 + +kafka (0.8.0-1) precise; urgency=low + + * Stable release of 0.8.0 tag. + + -- Andrew Otto (WMF) <[email protected]> Wed, 04 Dec 2013 15:01:05 +0000 + +kafka (0.8~20130913-1) precise; urgency=low + + * Bumping version to reflect latest commit (aebf7461) in 0.8 branch. + + -- Alexandros Kosiaris <[email protected]> Wed, 09 Sep 2013 16:44:14 -0700 + +kafka (0.8~20130827-1) precise; urgency=low + + * Bumping version to reflect latest commit in 0.8 branch. + + -- Alexandros Kosiaris <[email protected]> Wed, 28 Aug 2013 16:50:14 +0000 + +kafka (0.8~20130610-1) unstable; urgency=low + + * Initial release. + + -- Alexandros Kosiaris <[email protected]> Wed, 29 May 2013 23:45:30 +0000 diff --git a/debian/compat b/debian/compat new file mode 100644 index 0000000..ec63514 --- /dev/null +++ b/debian/compat @@ -0,0 +1 @@ +9 diff --git a/debian/control b/debian/control new file mode 100644 index 0000000..1d8e1d3 --- /dev/null +++ b/debian/control @@ -0,0 +1,98 @@ +Source: kafka +Section: misc +Priority: optional +Maintainer: Alexandros Kosiaris <[email protected]> +Build-Depends: debhelper (>= 9), openjdk-7-jdk | java7-jdk, javahelper (>= 0.40), + scala, + junit4, + velocity, + libasm3-java, + libcglib-java, + libcommons-cli-java, + libcommons-codec-java, + libcommons-compress-java, + libcommons-el-java, + libcommons-httpclient-java, + libcommons-io-java, + libcommons-lang-java, + libcommons-logging-java, + libcommons-net1-java, + libhsqldb-java, + libjackson-json-java, + libjets3t-java, + libjetty-java, + libjoda-time-java, + liblog4j1.2-java, + libobjenesis-java, + liboro-java, + libqdox-java, + libservlet2.5-java, + libslf4j-java, + libzookeeper-java +Standards-Version: 3.9.3 +Homepage: http://kafka.apache.org + +Package: kafka-common +Architecture: all +Depends: openjdk-7-jre | java7-runtime, scala-library, adduser, ${java:Depends}, ${misc:Depends} +Recommends: ${java:Recommends} +Description: Apache Kafka is a distributed publish-subscribe messaging system + Apache Kafka is designed to support the following + * Persistent messaging with O(1) disk structures + that provide constant time performance even with many TB of stored messages. + * High-throughput: even with very modest hardware Kafka can support hundreds + of thousands of messages per second. + * Explicit support for partitioning messages over Kafka servers and + distributing consumption over a cluster of consumer machines while + maintaining per-partition ordering semantics. + * Support for parallel data load into Hadoop. + . + This package includes common Kafka .jar files. + +Package: kafka-cli +Architecture: all +Depends: kafka-common +Description: Apache Kafka is a distributed publish-subscribe messaging system + Apache Kafka is designed to support the following + * Persistent messaging with O(1) disk structures + that provide constant time performance even with many TB of stored messages. + * High-throughput: even with very modest hardware Kafka can support hundreds + of thousands of messages per second. + * Explicit support for partitioning messages over Kafka servers and + distributing consumption over a cluster of consumer machines while + maintaining per-partition ordering semantics. + * Support for parallel data load into Hadoop. + . + This package includes a kafka shell script for easy interfacing with Kafka brokers. + +Package: kafka-server +Architecture: all +Depends: kafka-cli +Description: Apache Kafka is a distributed publish-subscribe messaging system + Apache Kafka is designed to support the following + * Persistent messaging with O(1) disk structures + that provide constant time performance even with many TB of stored messages. + * High-throughput: even with very modest hardware Kafka can support hundreds + of thousands of messages per second. + * Explicit support for partitioning messages over Kafka servers and + distributing consumption over a cluster of consumer machines while + maintaining per-partition ordering semantics. + * Support for parallel data load into Hadoop. + . + This package includes the init.d script and default file for a Kafka Broker. + +Package: kafka-mirror +Architecture: all +Depends: kafka-cli +Description: Apache Kafka is a distributed publish-subscribe messaging system + Apache Kafka is designed to support the following + * Persistent messaging with O(1) disk structures + that provide constant time performance even with many TB of stored messages. + * High-throughput: even with very modest hardware Kafka can support hundreds + of thousands of messages per second. + * Explicit support for partitioning messages over Kafka servers and + distributing consumption over a cluster of consumer machines while + maintaining per-partition ordering semantics. + * Support for parallel data load into Hadoop. + . + This package includes init.d script and default file for a Kafka MirrorMaker. diff --git a/debian/copyright b/debian/copyright new file mode 100644 index 0000000..fca2805 --- /dev/null +++ b/debian/copyright @@ -0,0 +1,77 @@ +Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/ +Upstream-Name: kafka +Source: http://kafka.apache.org/ + +Files: * +Copyright: 2012 The Apache Software Foundation. +License: Apache-2 + +Files: lib/sbt-lauch.jar +Copyright: 2008, 2009, 2010 Mark Harrah, Jason Zaugg +License: BSD-3-clause + +Files: debian/* +Copyright: 2013 Alexandros Kosiaris <[email protected]> +License: GPL-2+ + +Files: ext/* +Copyright: FIXME +License: FIXME + +License: Apache-2 + Licensed under the Apache License, Version 2.0 (the "License"); you may not + use this file except in compliance with the License. You may obtain a copy of + the License at + . + http://www.apache.org/licenses/LICENSE-2.0 + . + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language overning permissions and limitations under + the License. + . + On Debian systems, the Apache 2.0 license can be found in + /usr/share/common-licenses/Apache-2.0. + +License: BSD-3-clause + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions + are met: + 1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + 2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + 3. The name of the author may not be used to endorse or promote products + derived from this software without specific prior written permission. + . + THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR + IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES + OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. + IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, + INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT + NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF + THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +License: GPL-2+ + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 2 of the License, or + (at your option) any later version. + . + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + . + You should have received a copy of the GNU General Public License along + with this program; if not, write to the Free Software Foundation, Inc., + 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + . + On Debian systems, the full text of the GNU General Public + License version 2 can be found in the file + `/usr/share/common-licenses/GPL-2'. diff --git a/debian/gbp.conf b/debian/gbp.conf new file mode 100644 index 0000000..3865e29 --- /dev/null +++ b/debian/gbp.conf @@ -0,0 +1,9 @@ +[DEFAULT] +upstream-tree=tag +upstream-tag=upstream/%(version)s +debian-tag=debian/%(version)s +debian-branch=debian +cleaner=fakeroot debian/rules clean +# this is how we invoke pbuilder, arguments passed to git-buildpackage will be +# passed to dpkg-buildpackge in the chroot +builder = /usr/bin/git-pbuilder diff --git a/debian/kafka-cli.dirs b/debian/kafka-cli.dirs new file mode 100644 index 0000000..6f181bb --- /dev/null +++ b/debian/kafka-cli.dirs @@ -0,0 +1,2 @@ +etc/kafka +var/log/kafka diff --git a/debian/kafka-cli.install b/debian/kafka-cli.install new file mode 100644 index 0000000..4884114 --- /dev/null +++ b/debian/kafka-cli.install @@ -0,0 +1 @@ +debian/bin/kafka usr/sbin diff --git a/debian/kafka-cli.postinst b/debian/kafka-cli.postinst new file mode 100755 index 0000000..1479dd0 --- /dev/null +++ b/debian/kafka-cli.postinst @@ -0,0 +1,41 @@ +#!/bin/sh + +# postinst script for kafka-cli +# +# see: dh_installdeb(1) + +set -e + +case "$1" in + configure|reconfigure) + if ! getent passwd kafka >/dev/null; then + # Adding system user: kafka. + adduser \ + --system \ + --group \ + --no-create-home \ + --home /nonexistent \ + --gecos "Apache Kafka" \ + --shell /bin/false \ + kafka >/dev/null 2>/dev/null || : + fi + + # chown kafka directories + chown kafka:kafka /var/log/kafka + ;; + + abort-upgrade|abort-remove|abort-deconfigure) + ;; + + *) + echo "postinst called with unknown argument \`$1'" >&2 + exit 1 + ;; +esac + +# dh_installdeb will replace this with shell code automatically +# generated by other debhelper scripts. + +#DEBHELPER# + +exit 0 diff --git a/debian/kafka-cli.postrm b/debian/kafka-cli.postrm new file mode 100755 index 0000000..9fa6b8a --- /dev/null +++ b/debian/kafka-cli.postrm @@ -0,0 +1,28 @@ +#!/bin/sh + +# postrm script for kafka-cli +# +# see: dh_installdeb(1) + +set -e + +case "$1" in + purge) + deluser --quiet --system kafka >/dev/null || true + deluser --quiet --group --system --only-if-empty --remove-home kafka || true + rm -rf /var/log/kafka + ;; + remove|upgrade|failed-upgrade|abort-install|abort-upgrade|disappear) + ;; + *) + echo "postrm called with unknown argument \`$1'" >&2 + exit 1 + ;; +esac + +# dh_installdeb will replace this with shell code automatically +# generated by other debhelper scripts. + +#DEBHELPER# + +exit 0 diff --git a/debian/kafka-common.dirs b/debian/kafka-common.dirs new file mode 100644 index 0000000..f3aaeb8 --- /dev/null +++ b/debian/kafka-common.dirs @@ -0,0 +1 @@ +usr/share/kafka diff --git a/debian/kafka-common.install b/debian/kafka-common.install new file mode 100644 index 0000000..c06b4ec --- /dev/null +++ b/debian/kafka-common.install @@ -0,0 +1,25 @@ +ext/avro-1.4.0.jar usr/share/java +ext/commons-collections-2.1.1.jar usr/share/java +ext/easymock-3.0.jar usr/share/java +ext/hadoop-core-0.20.2.jar usr/share/java +ext/jasper-compiler-5.5.12.jar usr/share/java +ext/jasper-runtime-5.5.12.jar usr/share/java +ext/jopt-simple-3.2.jar usr/share/java +ext/jsp-2.1-6.1.14.jar usr/share/java +ext/jsp-api-2.1-6.1.14.jar usr/share/java +ext/kfs-0.3.jar usr/share/java +ext/lz4-1.2.0.jar usr/share/java +ext/metrics-annotation-2.2.0.jar usr/share/java +ext/metrics-core-2.2.0.jar usr/share/java +ext/metrics-ganglia-2.2.0.jar usr/share/java +ext/metrics-graphite-2.2.0.jar usr/share/java +ext/paranamer-ant-2.2.jar usr/share/java +ext/paranamer-generator-2.2.jar usr/share/java +ext/paranamer-2.1.jar usr/share/java +ext/pig-0.8.0.jar usr/share/java +ext/piggybank-0.12.0.jar usr/share/java +ext/scalatest-1.2.jar usr/share/java +ext/snappy-java-1.0.4.1.jar usr/share/java +ext/xmlenc-0.52.jar usr/share/java +ext/zkclient-0.3.jar usr/share/java +ext/kafka-ganglia-1.0.0.jar usr/share/kafka diff --git a/debian/kafka-common.links b/debian/kafka-common.links new file mode 100644 index 0000000..24f05a6 --- /dev/null +++ b/debian/kafka-common.links @@ -0,0 +1,5 @@ +usr/share/kafka/kafka-0.8.2.1.jar usr/share/kafka/kafka.jar +usr/share/kafka/kafka-clients-0.8.2.1.jar usr/share/kafka/kafka-clients.jar +usr/share/kafka/kafka-perf-0.8.2.1.jar usr/share/kafka/kafka-perf.jar +usr/share/kafka/hadoop-consumer-0.8.2.1.jar usr/share/kafka/hadoop-consumer.jar +usr/share/kafka/hadoop-producer-0.8.2.1.jar usr/share/kafka/hadoop-producer.jar diff --git a/debian/kafka-mirror.default b/debian/kafka-mirror.default new file mode 100644 index 0000000..cdb324c --- /dev/null +++ b/debian/kafka-mirror.default @@ -0,0 +1,45 @@ +# whether to allow init.d script to start a Kafka MirrorMaker daemon ("yes", "no") +KAFKA_MIRROR_START=no + +# User and group to run as +KAFKA_USER=kafka +KAFKA_GROUP=kafka +KAFKA_CONFIG=/etc/kafka + +# The default JMX_PORT for Kafka MirrorMaker is 9993. +# Set JMX_PORT to something else to override this. +#JMX_PORT=9993 + +#KAFKA_JMX_OPTS=${KAFKA_JMX_OPTS:="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"} + +# Memory sizes, and logging configuration +#KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$KAFKA_CONFIG/log4j.properties" + +# Default GC settings. +#KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:+UseCompressedOops -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC" +# LinkedIn recommended GC settings. See: https://kafka.apache.org/081/ops.html +# You need Java 7 u51 or greater to use the G1 GC. +#KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:PermSize=48m -XX:MaxPermSize=48m -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35" + +#KAFKA_HEAP_OPTS="-Xmx512M -Xms512M" +#KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:${KAFKA_CONFIG}/log4j.properties" +#KAFKA_OPTS="" + +# +# MirrorMaker options: +# + +# Set this to a space separated list of consumer.properties files. +# By default, /etc/init.d/kafka-mirror will use any files that match +# /etc/kafka/mirror/consumer*.properties +#KAFKA_MIRROR_CONSUMER_CONFIGS= + +#KAFKA_MIRROR_PRODUCER_CONFIG=/etc/kafka/mirror/producer.properties + +# Only set one of the following. +#KAFKA_MIRROR_WHITELIST='.*' +#KAFKA_MIRROR_BLACKLIST='' + +#KAFKA_MIRROR_NUM_STREAMS=1 +#KAFKA_MIRROR_NUM_PRODUCERS=1 +#KAFKA_MIRROR_QUEUE_SIZE=10000 diff --git a/debian/kafka-mirror.dirs b/debian/kafka-mirror.dirs new file mode 100644 index 0000000..cdba6d0 --- /dev/null +++ b/debian/kafka-mirror.dirs @@ -0,0 +1,2 @@ +etc/kafka/mirror +var/log/kafka diff --git a/debian/kafka-mirror.init b/debian/kafka-mirror.init new file mode 100755 index 0000000..1b67752 --- /dev/null +++ b/debian/kafka-mirror.init @@ -0,0 +1,251 @@ +#!/bin/sh +# +# /etc/init.d/kafka-mirror -- startup script for the Kafka MirrorMaker. +# See: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+mirroring+%28MirrorMaker%29 +# +# Written by Alexandros Kosiaris <[email protected]> and Andrew Otto <[email protected]> +# +### BEGIN INIT INFO +# Provides: kafka-mirror +# Required-Start: $local_fs $remote_fs $network +# Required-Stop: $local_fs $remote_fs $network +# Should-Start: $named +# Should-Stop: $named +# Default-Start: 2 3 4 5 +# Default-Stop: 0 1 6 +# Short-Description: Start kafka MirrorMaker +# Description: Start the Kafka MirrorMaker tool used to consume from multiple kafka clusters and produce into another. +### END INIT INFO + +set -e + +PATH=/bin:/usr/bin:/sbin:/usr/sbin +NAME="$(basename "$0" | sed 's/^[KS][0-9]\{2\}//')" +DESC="MirrorMaker" +DEFAULT=/etc/default/$NAME + +if [ `id -u` -ne 0 ]; then + echo "You need root privileges to run this script" + exit 1 +fi + +# Make sure kafka is started with system locale +if [ -r /etc/default/locale ]; then + . /etc/default/locale + export LANG +fi + +. /lib/lsb/init-functions + +if [ -r /etc/default/rcS ]; then + . /etc/default/rcS +fi + +# The following variables can be overwritten in $DEFAULT + +# Run kafka as this user ID and group ID +KAFKA_USER=kafka +KAFKA_GROUP=kafka +KAFKA_MIRROR_START=yes +KAFKA_CONFIG=/etc/kafka + +# Read in consumer config files from /etc/kafka/mirror. +KAFKA_MIRROR_CONSUMER_CONFIGS=$(ls $KAFKA_CONFIG/mirror/consumer*.properties || true) # Don't error out yet if files don't exist. +KAFKA_MIRROR_PRODUCER_CONFIG=$KAFKA_CONFIG/mirror/producer.properties + +KAFKA_MIRROR_NUM_STREAMS=1 +KAFKA_MIRROR_NUM_PRODUCERS=1 +KAFKA_MIRROR_QUEUE_SIZE=10000 +KAFKA_MIRROR_WHITELIST='.*' +KAFKA_MIRROR_BLACKLIST='' + +JDK_DIRS="/usr/lib/jvm/java-7-openjdk-amd64 /usr/lib/jvm/default-java /usr/lib/jvm/java-6-sun /usr/lib/jvm/java-1.5.0-sun /usr/lib/j2sdk1.5-sun /usr/lib/j2sdk1.5-ibm" + +# Look for the right JVM to use. +for jdir in $JDK_DIRS; do + if [ -r "$jdir/bin/java" -a -z "${JAVA_HOME}" ]; then + JAVA_HOME="$jdir" + fi +done +export JAVA_HOME + +# Default Java options +# Set java.awt.headless=true if JAVA_OPTS is not set so the +# Xalan XSL transformer can work without X11 display on JDK 1.4+ +if [ -z "$JAVA_OPTS" ]; then + JAVA_OPTS="-Djava.awt.headless=true" +fi + +# End of variables that can be overwritten in $DEFAULT + +# overwrite settings from default file +if [ -f "$DEFAULT" ]; then + . "$DEFAULT" +fi + +# Define other required variables +KAFKA_MIRROR_PID="/var/run/$NAME.pid" + + +kafka_mirror_sh() { + # Escape any double quotes in the value of JAVA_OPTS + JAVA_OPTS="$(echo $JAVA_OPTS | sed 's/\"/\\\"/g')" + KAFKA_OPTS="${KAFKA_OPTS} ${JAVA_OPTS}" + + # Export Kafka environment variables from $DEFAULT file so + # /usr/sbin/kafka will recognize them + export KAFKA_CONFIG JMX_PORT KAFKA_JMX_OPTS SCALA_VERSION KAFKA_JVM_PERFORMANCE_OPTS KAFKA_HEAP_OPTS KAFKA_LOG4J_OPTS KAFKA_OPTS CLASSPATH + + + # Define the command to run kafka MirrorMaker as a daemon + # set -a tells sh to export assigned variables to spawned shells. + KAFKA_MIRROR_ARGS=" +--num.streams $KAFKA_MIRROR_NUM_STREAMS \ +--num.producers $KAFKA_MIRROR_NUM_PRODUCERS \ +--queue.size $KAFKA_MIRROR_QUEUE_SIZE \ +--producer.config $KAFKA_MIRROR_PRODUCER_CONFIG " + + if [ -n "$KAFKA_MIRROR_WHITELIST" ]; then + KAFKA_MIRROR_ARGS="$KAFKA_MIRROR_ARGS --whitelist '$KAFKA_MIRROR_WHITELIST' " + else + KAFKA_MIRROR_ARGS="$KAFKA_MIRROR_ARGS --blacklist '$KAFKA_MIRROR_BLACKLIST' " + fi + + # Add all consumer config files to KAFKA_MIRROR_ARGS + for CONSUMER_CONFIG in $KAFKA_MIRROR_CONSUMER_CONFIGS; do + KAFKA_MIRROR_ARGS="$KAFKA_MIRROR_ARGS --consumer.config $CONSUMER_CONFIG" + done + + # Run as a daemon + set +e + + start-stop-daemon --start -b -u "$KAFKA_USER" -g "$KAFKA_GROUP" \ + -c "$KAFKA_USER" -m -p "$KAFKA_MIRROR_PID" \ + -x "/usr/sbin/kafka" -- "mirror-maker" $KAFKA_MIRROR_ARGS + status="$?" + set +a -e + return $status +} + +case "$1" in + start) + if [ -n "$KAFKA_MIRROR_START" -a "$KAFKA_MIRROR_START" != "yes" ]; then + log_failure_msg "KAFKA_MIRROR_START not set to 'yes' in $DEFAULT, not starting" + exit 0 + fi + + if [ -z "$JAVA_HOME" ]; then + log_failure_msg "no JDK found - please set JAVA_HOME" + exit 1 + fi + + # + # MirrorMaker argument checking: + # + + if [ -z "$KAFKA_MIRROR_CONSUMER_CONFIGS" ]; then + echo "No consumer config files provided." + exit 1 + fi + if [ -z "$KAFKA_MIRROR_PRODUCER_CONFIG" ]; then + echo "No producer config file provided." + exit 1 + fi + + if [ -n "$KAFKA_MIRROR_WHITELIST" -a -n "$KAFKA_MIRROR_BLACKLIST" ]; then + echo "Must set exactly one of KAFKA_MIRROR_WHITELIST and KAFKA_MIRROR_BLACKLIST, not both." + exit 1 + elif [ -z "$KAFKA_MIRROR_WHITELIST" -a -z "$KAFKA_MIRROR_BLACKLIST"]; then + echo "Must set exactly one of KAFKA_MIRROR_WHITELIST or KAFKA_MIRROR_BLACKLIST." + exit 1 + fi + + if [ ! -r "$KAFKA_MIRROR_PRODUCER_CONFIG" ]; then + echo "Cannot read producer config file '$KAFKA_MIRROR_PRODUCER_CONFIG'." + exit 1 + fi + + log_daemon_msg "Starting $DESC" "$NAME" + if start-stop-daemon --test --start --pidfile "$KAFKA_MIRROR_PID" \ + --user $KAFKA_USER --exec "$JAVA_HOME/bin/java" \ + >/dev/null; then + + kafka_mirror_sh start + sleep 5 + if start-stop-daemon --test --start --pidfile "$KAFKA_MIRROR_PID" \ + --user $KAFKA_USER --exec "$JAVA_HOME/bin/java" \ + >/dev/null; then + if [ -f "$KAFKA_MIRROR_PID" ]; then + rm -f "$KAFKA_MIRROR_PID" + fi + log_end_msg 1 + else + log_end_msg 0 + fi + else + log_progress_msg "(already running)" + log_end_msg 0 + fi + ;; + stop) + log_daemon_msg "Stopping $DESC" "$NAME" + + set +e + if [ -f "$KAFKA_MIRROR_PID" ]; then + start-stop-daemon --stop --pidfile "$KAFKA_MIRROR_PID" \ + --user "$KAFKA_USER" \ + --retry=TERM/60/KILL/5 >/dev/null + if [ $? -eq 1 ]; then + log_progress_msg "$DESC is not running but pid file exists, cleaning up" + elif [ $? -eq 3 ]; then + PID="`cat $KAFKA_MIRROR_PID`" + log_failure_msg "Failed to stop $NAME (pid $PID)" + exit 1 + fi + rm -f "$KAFKA_MIRROR_PID" + else + log_progress_msg "(not running)" + fi + log_end_msg 0 + set -e + ;; + status) + set +e + start-stop-daemon --test --start --pidfile "$KAFKA_MIRROR_PID" \ + --user $KAFKA_USER --exec "$JAVA_HOME/bin/java" \ + >/dev/null 2>&1 + if [ "$?" = "0" ]; then + + if [ -f "$KAFKA_MIRROR_PID" ]; then + log_success_msg "$DESC is not running, but pid file exists." + exit 1 + else + log_success_msg "$DESC is not running." + exit 3 + fi + else + log_success_msg "$DESC is running with pid `cat $KAFKA_MIRROR_PID`" + fi + set -e + ;; + restart|force-reload) + if [ -f "$KAFKA_MIRROR_PID" ]; then + $0 stop + sleep 1 + fi + $0 start + ;; + try-restart) + if start-stop-daemon --test --start --pidfile "$KAFKA_MIRROR_PID" \ + --user $KAFKA_USER --exec "$JAVA_HOME/bin/java" \ + >/dev/null; then + $0 start + fi + ;; + *) + log_success_msg "Usage: $0 {start|stop|restart|try-restart|force-reload|status}" + exit 1 + ;; +esac + +exit 0 diff --git a/debian/kafka-server.dirs b/debian/kafka-server.dirs new file mode 100644 index 0000000..df0a60d --- /dev/null +++ b/debian/kafka-server.dirs @@ -0,0 +1,3 @@ +etc/kafka +var/log/kafka +var/spool/kafka diff --git a/debian/kafka-server.kafka.default b/debian/kafka-server.kafka.default new file mode 100755 index 0000000..1285165 --- /dev/null +++ b/debian/kafka-server.kafka.default @@ -0,0 +1,28 @@ +# whether to allow init.d script to start a kafka broker ("yes", "no") +KAFKA_START=no + +# User and group to run as +KAFKA_USER=kafka +KAFKA_GROUP=kafka +KAFKA_CONFIG=/etc/kafka + +# ulimit open files limit for kafka user. +KAFKA_NOFILES_ULIMIT=8192 + +# The default JMX_PORT for Kafka Brokers is 9999. +# Set JMX_PORT to something else to override this. +#JMX_PORT=9999 +#KAFKA_JMX_OPTS=${KAFKA_JMX_OPTS:="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"} + +# Memory sizes, and logging configuration +#KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$KAFKA_CONFIG/log4j.properties" + +# Default GC settings. +#KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:+UseCompressedOops -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC" +# LinkedIn recommended GC settings. See: https://kafka.apache.org/081/ops.html +# You need Java 7 u51 or greater to use the G1 GC. +#KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:PermSize=48m -XX:MaxPermSize=48m -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35" + +#KAFKA_HEAP_OPTS="-Xmx1G -Xms1G" +#KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:${KAFKA_CONFIG}/log4j.properties" +#KAFKA_OPTS="" diff --git a/debian/kafka-server.kafka.init b/debian/kafka-server.kafka.init new file mode 100755 index 0000000..8024ead --- /dev/null +++ b/debian/kafka-server.kafka.init @@ -0,0 +1,203 @@ +#!/bin/sh +# +# /etc/init.d/kafka -- startup script for the kafka distributed publish-subscribe messaging system +# +# Written by Alexandros Kosiaris <[email protected]> +# +### BEGIN INIT INFO +# Provides: kafka +# Required-Start: $local_fs $remote_fs $network +# Required-Stop: $local_fs $remote_fs $network +# Should-Start: $named +# Should-Stop: $named +# Default-Start: 2 3 4 5 +# Default-Stop: 0 1 6 +# Short-Description: Start kafka +# Description: Start the kafka distributed publish-subscribe messaging system +### END INIT INFO + +set -e + +PATH=/bin:/usr/bin:/sbin:/usr/sbin +NAME="$(basename "$0" | sed 's/^[KS][0-9]\{2\}//')" +DESC="broker" +DEFAULT=/etc/default/$NAME + +if [ `id -u` -ne 0 ]; then + echo "You need root privileges to run this script" + exit 1 +fi + +# Make sure kafka is started with system locale +if [ -r /etc/default/locale ]; then + . /etc/default/locale + export LANG +fi + +. /lib/lsb/init-functions + +if [ -r /etc/default/rcS ]; then + . /etc/default/rcS +fi + +# # The following variables can be overwritten in $DEFAULT +# +# Run kafka as this user ID and group ID +KAFKA_USER=kafka +KAFKA_GROUP=kafka + +KAFKA_CONFIG=/etc/kafka +KAFKA_START=yes + +JDK_DIRS="/usr/lib/jvm/java-7-openjdk-amd64 /usr/lib/jvm/default-java /usr/lib/jvm/java-6-sun /usr/lib/jvm/java-1.5.0-sun /usr/lib/j2sdk1.5-sun /usr/lib/j2sdk1.5-ibm" + +# Look for the right JVM to use +for jdir in $JDK_DIRS; do + if [ -r "$jdir/bin/java" -a -z "${JAVA_HOME}" ]; then + JAVA_HOME="$jdir" + fi +done +export JAVA_HOME + +# Default Java options +# Set java.awt.headless=true if JAVA_OPTS is not set so the +# Xalan XSL transformer can work without X11 display on JDK 1.4+ +if [ -z "$JAVA_OPTS" ]; then + JAVA_OPTS="-Djava.awt.headless=true" +fi + +# End of variables that can be overwritten in $DEFAULT + +# overwrite settings from default file +if [ -f "$DEFAULT" ]; then + . "$DEFAULT" +fi + +# If ulimit nofiles was set in $DEFAULT, +# then change the limit for this process now. +# This will not set nofiles higher than +# system limit. +if [ -n "$KAFKA_NOFILES_ULIMIT" ]; then + ulimit -n "$KAFKA_NOFILES_ULIMIT" +fi + +# Define other required variables +KAFKA_PID="/var/run/$NAME.pid" +DAEMON="${JAVA_HOME}/bin/java" + +kafka_sh() { + # Escape any double quotes in the value of JAVA_OPTS + JAVA_OPTS="$(echo $JAVA_OPTS | sed 's/\"/\\\"/g')" + KAFKA_OPTS="${KAFKA_OPTS} ${JAVA_OPTS}" + + # Export Kafka environment variables from $DEFAULT file so + # /usr/sbin/kafka will recognize them + export KAFKA_CONFIG JMX_PORT KAFKA_JMX_OPTS SCALA_VERSION KAFKA_JVM_PERFORMANCE_OPTS KAFKA_HEAP_OPTS KAFKA_LOG4J_OPTS KAFKA_OPTS CLASSPATH + + # Run as a daemon + set +e + + start-stop-daemon --start -b -u "$KAFKA_USER" -g "$KAFKA_GROUP" \ + -c "$KAFKA_USER" -m -p "$KAFKA_PID" \ + -x "/usr/sbin/kafka" -- "server-start" "${KAFKA_CONFIG}/server.properties" + status="$?" + set +a -e + return $status +} + +case "$1" in + start) + if [ -n "$KAFKA_START" -a "$KAFKA_START" != "yes" ]; then + log_failure_msg "KAFKA_START not set to 'yes' in $DEFAULT, not starting" + exit 0 + fi + + if [ -z "$JAVA_HOME" ]; then + log_failure_msg "no JDK found - please set JAVA_HOME" + exit 1 + fi + + log_daemon_msg "Starting $DESC" "$NAME" + if start-stop-daemon --test --start --pidfile "$KAFKA_PID" \ + --user "$KAFKA_USER" --exec "${JAVA_HOME}/bin/java" \ + >/dev/null; then + + kafka_sh start + sleep 5 + if start-stop-daemon --test --start --pidfile "$KAFKA_PID" \ + --user "$KAFKA_USER" --exec "${JAVA_HOME}/bin/java" \ + >/dev/null; then + if [ -f "$KAFKA_PID" ]; then + rm -f "$KAFKA_PID" + fi + log_end_msg 1 + else + log_end_msg 0 + fi + else + log_progress_msg "(already running)" + log_end_msg 0 + fi + ;; + stop) + log_daemon_msg "Stopping $DESC" "$NAME" + + set +e + if [ -f "$KAFKA_PID" ]; then + start-stop-daemon --stop --pidfile "$KAFKA_PID" \ + --user "$KAFKA_USER" \ + --retry=TERM/60/KILL/5 >/dev/null + if [ $? -eq 1 ]; then + log_progress_msg "$DESC is not running but pid file exists, cleaning up" + elif [ $? -eq 3 ]; then + PID="`cat $KAFKA_PID`" + log_failure_msg "Failed to stop $NAME (pid $PID)" + exit 1 + fi + rm -f "$KAFKA_PID" + else + log_progress_msg "(not running)" + fi + log_end_msg 0 + set -e + ;; + status) + set +e + start-stop-daemon --test --start --pidfile "$KAFKA_PID" \ + --user "$KAFKA_USER" --exec "${JAVA_HOME}/bin/java" \ + >/dev/null 2>&1 + if [ "$?" = "0" ]; then + + if [ -f "$KAFKA_PID" ]; then + log_success_msg "$DESC is not running, but pid file exists." + exit 1 + else + log_success_msg "$DESC is not running." + exit 3 + fi + else + log_success_msg "$DESC is running with pid `cat $KAFKA_PID`" + fi + set -e + ;; + restart|force-reload) + if [ -f "$KAFKA_PID" ]; then + $0 stop + sleep 1 + fi + $0 start + ;; + try-restart) + if start-stop-daemon --test --start --pidfile "$KAFKA_PID" \ + --user "$KAFKA_USER" --exec "${JAVA_HOME}/bin/java" \ + >/dev/null; then + $0 start + fi + ;; + *) + log_success_msg "Usage: $0 {start|stop|restart|try-restart|force-reload|status}" + exit 1 + ;; +esac + +exit 0 diff --git a/debian/kafka-server.postinst b/debian/kafka-server.postinst new file mode 100755 index 0000000..2bb6b10 --- /dev/null +++ b/debian/kafka-server.postinst @@ -0,0 +1,28 @@ +#!/bin/sh + +# postinst script for kafka-server +# +# see: dh_installdeb(1) + +set -e + +case "$1" in + configure|reconfigure) + chown kafka:kafka /var/spool/kafka + ;; + + abort-upgrade|abort-remove|abort-deconfigure) + ;; + + *) + echo "postinst called with unknown argument \`$1'" >&2 + exit 1 + ;; +esac + +# dh_installdeb will replace this with shell code automatically +# generated by other debhelper scripts. + +#DEBHELPER# + +exit 0 diff --git a/debian/patches/kafka_data_dirs_fixes.patch b/debian/patches/kafka_data_dirs_fixes.patch new file mode 100644 index 0000000..c898d1d --- /dev/null +++ b/debian/patches/kafka_data_dirs_fixes.patch @@ -0,0 +1,13 @@ +diff -r -u a/config/server.properties b/config/server.properties +--- a/config/server.properties 2013-06-12 10:09:28.531748537 +0000 ++++ b/config/server.properties 2013-06-12 10:50:27.995707650 +0000 +@@ -55,7 +55,7 @@ + ############################# Log Basics ############################# + + # A comma seperated list of directories under which to store log files +-log.dirs=/tmp/kafka-logs ++log.dirs=/var/spool/kafka + + # The default number of log partitions per topic. More partitions allow greater + # parallelism for consumption, but this will also result in more files across + diff --git a/debian/patches/logging_to_var_log.patch b/debian/patches/logging_to_var_log.patch new file mode 100644 index 0000000..5e9cfb3 --- /dev/null +++ b/debian/patches/logging_to_var_log.patch @@ -0,0 +1,68 @@ +--- a/config/log4j.properties 2013-06-12 10:09:28.531748537 +0000 ++++ b/config/log4j.properties 2013-06-12 15:09:40.967449096 +0000 +@@ -15,17 +15,23 @@ + + kafka.logs.dir=logs + +-log4j.rootLogger=INFO, stdout ++log4j.rootLogger=INFO, stdout, fileAppender + + log4j.appender.stdout=org.apache.log4j.ConsoleAppender + log4j.appender.stdout.layout=org.apache.log4j.PatternLayout + log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c)%n ++log4j.appender.stdout.Threshold=WARN + +-log4j.appender.kafkaAppender=org.apache.log4j.DailyRollingFileAppender +-log4j.appender.kafkaAppender.DatePattern='.'yyyy-MM-dd-HH +-log4j.appender.kafkaAppender.File=${kafka.logs.dir}/server.log +-log4j.appender.kafkaAppender.layout=org.apache.log4j.PatternLayout +-log4j.appender.kafkaAppender.layout.ConversionPattern=[%d] %p %m (%c)%n ++log4j.appender.fileAppender=org.apache.log4j.RollingFileAppender ++log4j.appender.fileAppender.File=/var/log/kafka/kafka.log ++log4j.appender.fileAppender.layout=org.apache.log4j.PatternLayout ++log4j.appender.fileAppender.layout.ConversionPattern=[%d] %-4r [%t] %-5p %c %x - %m%n ++log4j.appender.fileAppender.MaxFileSize=100MB ++log4j.appender.fileAppender.MaxBackupIndex=2 ++log4j.appender.fileAppender.Threshold=INFO ++ ++# Turn on all our debugging info ++#log4j.logger.kafka=INFO + + log4j.appender.stateChangeAppender=org.apache.log4j.DailyRollingFileAppender + log4j.appender.stateChangeAppender.DatePattern='.'yyyy-MM-dd-HH +@@ -57,22 +63,21 @@ + #log4j.logger.kafka.perf=DEBUG, kafkaAppender + #log4j.logger.kafka.perf.ProducerPerformance$ProducerThread=DEBUG, kafkaAppender + #log4j.logger.org.I0Itec.zkclient.ZkClient=DEBUG +-log4j.logger.kafka=INFO, kafkaAppender + +-log4j.logger.kafka.network.RequestChannel$=WARN, requestAppender +-log4j.additivity.kafka.network.RequestChannel$=false ++#log4j.logger.kafka.network.RequestChannel$=WARN, requestAppender ++#log4j.additivity.kafka.network.RequestChannel$=false + + #log4j.logger.kafka.network.Processor=TRACE, requestAppender + #log4j.logger.kafka.server.KafkaApis=TRACE, requestAppender + #log4j.additivity.kafka.server.KafkaApis=false +-log4j.logger.kafka.request.logger=WARN, requestAppender +-log4j.additivity.kafka.request.logger=false +- +-log4j.logger.kafka.controller=TRACE, controllerAppender +-log4j.additivity.kafka.controller=false +- +-log4j.logger.kafka.log.LogCleaner=INFO, cleanerAppender +-log4j.additivity.kafka.log.LogCleaner=false +- +-log4j.logger.state.change.logger=TRACE, stateChangeAppender +-log4j.additivity.state.change.logger=false ++#log4j.logger.kafka.request.logger=WARN, requestAppender ++#log4j.additivity.kafka.request.logger=false ++# ++#log4j.logger.kafka.controller=TRACE, controllerAppender ++#log4j.additivity.kafka.controller=false ++# ++#log4j.logger.kafka.log.LogCleaner=INFO, cleanerAppender ++#log4j.additivity.kafka.log.LogCleaner=false ++# ++#log4j.logger.state.change.logger=TRACE, stateChangeAppender ++#log4j.additivity.state.change.logger=false diff --git a/debian/patches/our-own-build-system.patch b/debian/patches/our-own-build-system.patch new file mode 100644 index 0000000..07670c0 --- /dev/null +++ b/debian/patches/our-own-build-system.patch @@ -0,0 +1,259 @@ +From: Alexandros Kosiaris <[email protected]>, Andrew Otto <[email protected]> +Date: Tue, 28 May 2013 17:46:15 +0000 +Subject: Introducing our own build system + +Introducing our own build system for the kafka project based on +calling scalac and a set of simple Makefiles. Will compile per +directory {core, perf, examples, contrib}, install configuration +files and depend on either system provided JAR libraries or the +ones shipped in ext/. Install targets by default go to +/usr/local/share/java, packaging system will probably override that. +It supports PREFIX, DESTDIR, SYSCONFDIR. +CLASSPATH variable is meant to be overriden more by the caller. Some +sane defaults are discovered but they should not be relied on too much. +The version discovery system is rather naive and will fail in many +cases to correctly identify the version. Since this is not trivial to +fix we allow the caller to override version. + +Updated for 0.8.2.1 on 2015-07-28 by Andrew Otto. +--- + Makefile | 66 ++++++++++++++++++++++++++++++++++++++++ + clients/Makefile | 18 +++++++++++ + config/Makefile | 15 +++++++++ + contrib/Makefile | 24 +++++++++++++++ + contrib/hadoop-consumer/Makefile | 18 +++++++++++ + contrib/hadoop-producer/Makefile | 19 ++++++++++++ + core/Makefile | 21 +++++++++++++ + 9 files changed, 220 insertions(+) + create mode 100644 Makefile + create mode 100644 clients/Makefile + create mode 100644 config/Makefile + create mode 100644 contrib/Makefile + create mode 100644 contrib/hadoop-consumer/Makefile + create mode 100644 contrib/hadoop-producer/Makefile + create mode 100644 core/Makefile + +diff --git a/Makefile b/Makefile +new file mode 100644 +index 0000000..cf05093 +--- /dev/null ++++ b/Makefile +@@ -0,0 +1,66 @@ ++export SCALAC=scalac ++export JAVAC=javac ++export MKDIR_P=mkdir -p ++export RM_RF=rm -rf ++export JAR=jar ++export INSTALL=install -D -m 644 ++ ++# Figure this out as well as other dependencies ++export SCALA_LIBRARY=/usr/share/java/scala-library-2.9.2.jar ++ ++# DESTDIR ++ifndef PREFIX ++export PREFIX=/usr/local ++endif ++ifndef SYSCONFDIR ++export SYSCONFDIR=/usr/local/etc/kafka ++endif ++export LIBPATH=$(PREFIX)/share/kafka ++export DESTDIR ++ ++# Release specifics ++tag=$(shell git describe --abbrev=0) ++ver=$(shell echo $(tag) | sed -e 's/kafka-//' -e 's/-incubating-candidate-[0-9]//') ++ifndef VERSION ++export VERSION=$(ver) ++endif ++ ++# This is a fallback in case CLASSPATH is not defined by the caller ++# It will find and add to the CLASSPATH variable all jars in $(PREFIX)/usr/share ++# Hopefully CLASSPATH will be defined and this will be unused ++SYS_CLASSDIR:=$(PREFIX)/share/java ++SYS_CLASSJARS:=$(shell find $(SYS_CLASSDIR) -name "*.jar") ++export SYS_CLASSPATH:=$(shell echo $(SYS_CLASSJARS) | sed -e 's/ /:/g') ++ ++# Same here, just this is done for the shipped jar dependencies. ++EXT_CLASSDIR:=$(CURDIR)/ext ++EXT_CLASSJARS:=$(shell find $(EXT_CLASSDIR) -name "*.jar") ++export EXT_CLASSPATH:=$(shell echo $(EXT_CLASSJARS) | sed -e 's/ /:/g') ++ ++ifndef CLASSPATH ++export CLASSPATH=$(SYS_CLASSPATH):$(EXT_CLASSPATH) ++endif ++export CFLAGS=-usejavacp ++ ++# Don't bother building examples as part of main build process. ++SUBDIRS=clients core contrib config ++ ++BUILDDIRS=$(SUBDIRS:%=build-%) ++CLEANDIRS=$(SUBDIRS:%=clean-%) ++INSTALLDIRS=$(SUBDIRS:%=install-%) ++ ++all: $(BUILDDIRS) ++ ++install: all $(INSTALLDIRS) ++ ++clean: $(CLEANDIRS) ++ ++$(BUILDDIRS): $(TARGET_PATH) ++ $(MAKE) -C $(@:build-%=%) all ++ ++$(CLEANDIRS): $(TARGET_PATH) ++ $(MAKE) -C $(@:clean-%=%) clean ++ ++$(INSTALLDIRS): $(TARGET_PATH) ++ $(MAKE) -C $(@:install-%=%) install ++.PHONY: clean install all +diff --git a/clients/Makefile b/clients/Makefile +new file mode 100644 +index 0000000..3c0f6bd +--- /dev/null ++++ b/clients/Makefile +@@ -0,0 +1,18 @@ ++SOURCES=$(shell find src/main -name "*.java") ++ ++TARGET_PATH=$(CURDIR)/target ++CLASSPATH:=$(CLASSPATH):$(TARGET_PATH) ++ ++$(TARGET_PATH): ++ $(MKDIR_P) $(TARGET_PATH) ++ ++all: $(SOURCES) $(TARGET_PATH) ++ $(JAVAC) -d $(TARGET_PATH) $(SOURCES) ++ ++clean: ++ $(RM_RF) $(TARGET_PATH) ++ $(RM_RF) kafka-clients-$(VERSION).jar ++ ++install: all ++ $(JAR) cfv kafka-clients-$(VERSION).jar -C target . ++ $(INSTALL) -o root -g root kafka-clients-$(VERSION).jar $(DESTDIR)$(LIBPATH)/kafka-clients-$(VERSION).jar +diff --git a/config/Makefile b/config/Makefile +new file mode 100644 +index 0000000..f78a68d +--- /dev/null ++++ b/config/Makefile +@@ -0,0 +1,15 @@ ++CONFFILES=log4j.properties tools-log4j.properties server.properties zookeeper.properties ++MIRROR_CONFFILES=consumer.properties producer.properties ++MIRROR_CONFFILES_EXAMPLES=$(MIRROR_CONFFILES:.properties=.properties.example) ++ ++all: ++ ++clean: ++ ++install: all $(MIRROR_CONFFILES_EXAMPLES) ++ $(MKDIR_P) $(DESTDIR)/$(SYSCONFDIR) ++ $(INSTALL) -o root -g root $(CONFFILES) $(DESTDIR)/$(SYSCONFDIR) ++ $(MKDIR_P) $(DESTDIR)/$(SYSCONFDIR)/mirror ++%.properties.example: %.properties ++ $(MKDIR_P) $(DESTDIR)/$(SYSCONFDIR)/mirror ++ $(INSTALL) -T -o root -g root $< $(DESTDIR)/$(SYSCONFDIR)/mirror/$@ +diff --git a/contrib/Makefile b/contrib/Makefile +new file mode 100644 +index 0000000..1db7722 +--- /dev/null ++++ b/contrib/Makefile +@@ -0,0 +1,24 @@ ++SUBDIRS=hadoop-consumer hadoop-producer ++BUILDDIRS=$(SUBDIRS:%=build-%) ++CLEANDIRS=$(SUBDIRS:%=clean-%) ++INSTALLDIRS=$(SUBDIRS:%=install-%) ++ ++export CORE_TARGET_PATH=$(dir $(CURDIR))core/target ++export CLIENTS_TARGET_PATH=$(dir $(CURDIR))clients/target ++ ++.PHONY: all install clean ++ ++all: $(BUILDDIRS) ++install: $(INSTALLDIRS) all ++clean: $(CLEANDIRS) ++ ++ ++$(BUILDDIRS): $(TARGET_PATH) ++ $(MAKE) -C $(@:build-%=%) all ++ ++$(CLEANDIRS): $(TARGET_PATH) ++ $(MAKE) -C $(@:clean-%=%) clean ++ ++$(INSTALLDIRS): $(TARGET_PATH) ++ $(MAKE) -C $(@:install-%=%) install ++ +diff --git a/contrib/hadoop-consumer/Makefile b/contrib/hadoop-consumer/Makefile +new file mode 100644 +index 0000000..e2e3e93 +--- /dev/null ++++ b/contrib/hadoop-consumer/Makefile +@@ -0,0 +1,18 @@ ++SOURCES=$(shell find -name "*.java") ++ ++TARGET_PATH=$(CURDIR)/target ++CLASSPATH:=$(CLASSPATH):$(SCALA_LIBRARY):$(CLIENTS_TARGET_PATH):$(CORE_TARGET_PATH) ++ ++$(TARGET_PATH): ++ $(MKDIR_P) $(TARGET_PATH) ++ ++all: $(SOURCES) $(TARGET_PATH) ++ $(JAVAC) -d $(TARGET_PATH) $(SOURCES) ++ ++clean: ++ $(RM_RF) $(TARGET_PATH) ++ $(RM_RF) hadoop-consumer-$(VERSION).jar ++ ++install: all ++ $(JAR) cfv hadoop-consumer-$(VERSION).jar -C target . ++ $(INSTALL) -o root -g root hadoop-consumer-$(VERSION).jar $(DESTDIR)$(LIBPATH)/hadoop-consumer-$(VERSION).jar +diff --git a/contrib/hadoop-producer/Makefile b/contrib/hadoop-producer/Makefile +new file mode 100644 +index 0000000..3c5d123 +--- /dev/null ++++ b/contrib/hadoop-producer/Makefile +@@ -0,0 +1,19 @@ ++SOURCES=$(shell find -name "*.java") ++ ++TARGET_PATH=$(CURDIR)/target ++CLASSPATH:=$(CLASSPATH):$(SCALA_LIBRARY):$(CLIENTS_TARGET_PATH):$(CORE_TARGET_PATH) ++ ++$(TARGET_PATH): ++ $(MKDIR_P) $(TARGET_PATH) ++ ++all: $(SOURCES) $(TARGET_PATH) ++ $(JAVAC) -d $(TARGET_PATH) $(SOURCES) ++ ++clean: ++ $(RM_RF) $(TARGET_PATH) ++ $(RM_RF) hadoop-producer-$(VERSION).jar ++ ++install: all ++ $(JAR) cfve hadoop-producer-$(VERSION).jar kafka.bridge.examples.TextPublisher -C target . ++ $(INSTALL) -o root -g root hadoop-producer-$(VERSION).jar $(DESTDIR)$(LIBPATH)/hadoop-producer-$(VERSION).jar ++ +diff --git a/core/Makefile b/core/Makefile +new file mode 100644 +index 0000000..2c107e2 +--- /dev/null ++++ b/core/Makefile +@@ -0,0 +1,21 @@ ++SOURCES=$(shell find src/main -name "*.scala" -or -name "*.java") ++JAVA_SOURCES=$(shell find src/main -name "*.java") ++ ++TARGET_PATH=$(CURDIR)/target ++CLIENTS_TARGET_PATH=$(dir $(CURDIR))clients/target ++CLASSPATH:=$(CLASSPATH):$(CLIENTS_TARGET_PATH):$(TARGET_PATH) ++ ++$(TARGET_PATH): ++ $(MKDIR_P) $(TARGET_PATH) ++ ++all: $(SOURCES) $(TARGET_PATH) ++ $(SCALAC) $(CFLAGS) -d $(TARGET_PATH) $(SOURCES) ++ $(JAVAC) -d $(TARGET_PATH) $(JAVA_SOURCES) ++ ++clean: ++ $(RM_RF) $(TARGET_PATH) ++ $(RM_RF) kafka-$(VERSION).jar ++ ++install: ++ $(JAR) cfv kafka-$(VERSION).jar -C target . ++ $(INSTALL) -o root -g root kafka-$(VERSION).jar $(DESTDIR)$(LIBPATH)/kafka-$(VERSION).jar +-- diff --git a/debian/patches/series b/debian/patches/series new file mode 100644 index 0000000..8d89416 --- /dev/null +++ b/debian/patches/series @@ -0,0 +1,3 @@ +our-own-build-system.patch +kafka_data_dirs_fixes.patch +logging_to_var_log.patch diff --git a/debian/rules b/debian/rules new file mode 100755 index 0000000..493c3be --- /dev/null +++ b/debian/rules @@ -0,0 +1,36 @@ +#!/usr/bin/make -f + +export JAVA_HOME?=/usr/lib/jvm/java-1.7.0-openjdk-amd64 +export PREFIX=/usr +export SYSCONFDIR=/etc/kafka +export VERSION=0.8.2.1 + +# Put build dependency jars here. +export CLASSPATH=/usr/share/java/asm3.jar:/usr/share/java/cglib.jar:/usr/share/java/commons-cli.jar:/usr/share/java/commons-codec.jar:/usr/share/java/commons-compress.jar:/usr/share/java/commons-el.jar:/usr/share/java/commons-httpclient.jar:/usr/share/java/commons-io.jar:/usr/share/java/commons-lang.jar:/usr/share/java/commons-logging.jar:/usr/share/java/commons-net.jar:/usr/share/java/hsqldb.jar:/usr/share/java/jackson-core.jar:/usr/share/java/jackson-mapper.jar:/usr/share/java/jets3t.jar:/usr/share/java/jetty-util.jar:/usr/share/java/jetty.jar:/usr/share/java/joda-time.jar:/usr/share/java/jsp-api-2.1.jar:/usr/share/java/junit4.jar:/usr/share/java/log4j-1.2.jar:/usr/share/java/objenesis.jar:/usr/share/java/oro.jar:/usr/share/java/qdox.jar:/usr/share/java/scala-library.jar:/usr/share/java/servlet-api-2.5.jar:/usr/share/java/slf4j-api.jar:/usr/share/java/slf4j-simple.jar:/usr/share/java/velocity-1.7.jar:/usr/share/java/zookeeper.jar:$(CURDIR)/ext/avro-1.4.0.jar:$(CURDIR)/ext/commons-collections-2.1.1.jar.jar:$(CURDIR)/ext/easymock-3.0.jar:$(CURDIR)/ext/hadoop-core-0.20.2.jar:$(CURDIR)/ext/jasper-compiler-5.5.12.jar:$(CURDIR)/ext/jasper-runtime-5.5.12.jar:$(CURDIR)/ext/jopt-simple-3.2.jar:$(CURDIR)/ext/jsp-2.1-6.1.14.jar:$(CURDIR)/ext/jsp-api-2.1-6.1.14.jar:$(CURDIR)/ext/kfs-0.3.jar:$(CURDIR)/ext/lz4-1.2.0.jar:$(CURDIR)/ext/metrics-annotation-2.2.0.jar:$(CURDIR)/ext/metrics-core-2.2.0.jar:$(CURDIR)/ext/metrics-ganglia-2.2.0.jar:$(CURDIR)/ext/metrics-graphite-2.2.0.jar:$(CURDIR)/ext/paranamer-ant-2.2.jar:$(CURDIR)/ext/paranamer-generator-2.2.jar:$(CURDIR)/ext/paranamer-2.1.jar:$(CURDIR)/ext/pig-0.8.0.jar:$(CURDIR)/ext/piggybank-0.12.0.jar:$(CURDIR)/ext/scalatest-1.2.jar:$(CURDIR)/ext/snappy-java-1.0.4.1.jar:$(CURDIR)/ext/xmlenc-0.52.jar:$(CURDIR)/ext/zkclient-0.3.jar + +%: + dh $@ --with javahelper + +override_dh_install: + dh_install + # make install installs files into debian/tmp. + # These files are part of the kafka-common package. + cp -rv $(CURDIR)/debian/tmp/usr/share/kafka/* $(CURDIR)/debian/kafka-common/usr/share/kafka/ + + # These files are part of the kafka-cli package. + cp -rv $(CURDIR)/debian/tmp/etc/kafka/tools-log4j.properties $(CURDIR)/debian/kafka-cli/etc/kafka/ + cp -rv $(CURDIR)/debian/tmp/etc/kafka/log4j.properties $(CURDIR)/debian/kafka-cli/etc/kafka/ + + # These files are part of the kafka-server package. + cp -rv $(CURDIR)/debian/tmp/etc/kafka/server.properties $(CURDIR)/debian/kafka-server/etc/kafka/ + cp -rv $(CURDIR)/debian/tmp/etc/kafka/zookeeper.properties $(CURDIR)/debian/kafka-server/etc/kafka/ + + # These files are part of the kafka-mirror package + cp -rv $(CURDIR)/debian/tmp/etc/kafka/mirror $(CURDIR)/debian/kafka-mirror/etc/kafka/ + +override_dh_installinit: + # Install kafka-server.kafka init.d and default scripts under the name 'kafka'. + dh_installinit --name=kafka + + # Install kafka-mirror init.d and default scripts under their default name 'kafka-mirror'. + dh_installinit diff --git a/debian/source/format b/debian/source/format new file mode 100644 index 0000000..163aaf8 --- /dev/null +++ b/debian/source/format @@ -0,0 +1 @@ +3.0 (quilt) diff --git a/debian/source/include-binaries b/debian/source/include-binaries new file mode 100644 index 0000000..b7d748b --- /dev/null +++ b/debian/source/include-binaries @@ -0,0 +1,25 @@ +ext/avro-1.4.0.jar +ext/commons-collections-2.1.1.jar +ext/easymock-3.0.jar +ext/hadoop-core-0.20.2.jar +ext/jasper-compiler-5.5.12.jar +ext/jasper-runtime-5.5.12.jar +ext/jopt-simple-3.2.jar +ext/jsp-2.1-6.1.14.jar +ext/jsp-api-2.1-6.1.14.jar +ext/kfs-0.3.jar +ext/lz4-1.2.0.jar +ext/metrics-annotation-2.2.0.jar +ext/metrics-core-2.2.0.jar +ext/metrics-ganglia-2.2.0.jar +ext/metrics-graphite-2.2.0.jar +ext/paranamer-ant-2.2.jar +ext/paranamer-generator-2.2.jar +ext/paranamer-2.1.jar +ext/pig-0.8.0.jar +ext/piggybank-0.12.0.jar +ext/scalatest-1.2.jar +ext/snappy-java-1.0.4.1.jar +ext/xmlenc-0.52.jar +ext/zkclient-0.3.jar +ext/kafka-ganglia-1.0.0.jar diff --git a/debian/source/options b/debian/source/options new file mode 100644 index 0000000..2df3e3f --- /dev/null +++ b/debian/source/options @@ -0,0 +1 @@ +extend-diff-ignore = '^\.gitreview$' diff --git a/ext/avro-1.4.0.jar b/ext/avro-1.4.0.jar new file mode 100644 index 0000000..533d0a3 --- /dev/null +++ b/ext/avro-1.4.0.jar Binary files differ diff --git a/ext/commons-collections-2.1.1.jar b/ext/commons-collections-2.1.1.jar new file mode 100644 index 0000000..c461445 --- /dev/null +++ b/ext/commons-collections-2.1.1.jar Binary files differ diff --git a/ext/easymock-3.0.jar b/ext/easymock-3.0.jar new file mode 100644 index 0000000..2728943 --- /dev/null +++ b/ext/easymock-3.0.jar Binary files differ diff --git a/ext/hadoop-core-0.20.2.jar b/ext/hadoop-core-0.20.2.jar new file mode 100644 index 0000000..0568eaf --- /dev/null +++ b/ext/hadoop-core-0.20.2.jar Binary files differ diff --git a/ext/jasper-compiler-5.5.12.jar b/ext/jasper-compiler-5.5.12.jar new file mode 100644 index 0000000..2a410b4 --- /dev/null +++ b/ext/jasper-compiler-5.5.12.jar Binary files differ diff --git a/ext/jasper-runtime-5.5.12.jar b/ext/jasper-runtime-5.5.12.jar new file mode 100644 index 0000000..743d906 --- /dev/null +++ b/ext/jasper-runtime-5.5.12.jar Binary files differ diff --git a/ext/jopt-simple-3.2.jar b/ext/jopt-simple-3.2.jar new file mode 100644 index 0000000..5637362 --- /dev/null +++ b/ext/jopt-simple-3.2.jar Binary files differ diff --git a/ext/jsp-2.1-6.1.14.jar b/ext/jsp-2.1-6.1.14.jar new file mode 100644 index 0000000..debfe56 --- /dev/null +++ b/ext/jsp-2.1-6.1.14.jar Binary files differ diff --git a/ext/jsp-api-2.1-6.1.14.jar b/ext/jsp-api-2.1-6.1.14.jar new file mode 100644 index 0000000..1be50ab --- /dev/null +++ b/ext/jsp-api-2.1-6.1.14.jar Binary files differ diff --git a/ext/kafka-ganglia-1.0.0.jar b/ext/kafka-ganglia-1.0.0.jar new file mode 100644 index 0000000..d2d6a21 --- /dev/null +++ b/ext/kafka-ganglia-1.0.0.jar Binary files differ diff --git a/ext/kfs-0.3.jar b/ext/kfs-0.3.jar new file mode 100644 index 0000000..e517c09 --- /dev/null +++ b/ext/kfs-0.3.jar Binary files differ diff --git a/ext/lz4-1.2.0.jar b/ext/lz4-1.2.0.jar new file mode 100644 index 0000000..9cdebd2 --- /dev/null +++ b/ext/lz4-1.2.0.jar Binary files differ diff --git a/ext/metrics-annotation-2.2.0.jar b/ext/metrics-annotation-2.2.0.jar new file mode 100644 index 0000000..0acdfae --- /dev/null +++ b/ext/metrics-annotation-2.2.0.jar Binary files differ diff --git a/ext/metrics-core-2.2.0.jar b/ext/metrics-core-2.2.0.jar new file mode 100644 index 0000000..0f6d1cb --- /dev/null +++ b/ext/metrics-core-2.2.0.jar Binary files differ diff --git a/ext/metrics-ganglia-2.2.0.jar b/ext/metrics-ganglia-2.2.0.jar new file mode 100644 index 0000000..8ba2241 --- /dev/null +++ b/ext/metrics-ganglia-2.2.0.jar Binary files differ diff --git a/ext/metrics-graphite-2.2.0.jar b/ext/metrics-graphite-2.2.0.jar new file mode 100644 index 0000000..4c80152 --- /dev/null +++ b/ext/metrics-graphite-2.2.0.jar Binary files differ diff --git a/ext/paranamer-2.1.jar b/ext/paranamer-2.1.jar new file mode 100644 index 0000000..4abfd52 --- /dev/null +++ b/ext/paranamer-2.1.jar Binary files differ diff --git a/ext/paranamer-ant-2.2.jar b/ext/paranamer-ant-2.2.jar new file mode 100644 index 0000000..169bb24 --- /dev/null +++ b/ext/paranamer-ant-2.2.jar Binary files differ diff --git a/ext/paranamer-generator-2.2.jar b/ext/paranamer-generator-2.2.jar new file mode 100644 index 0000000..d4ae50d --- /dev/null +++ b/ext/paranamer-generator-2.2.jar Binary files differ diff --git a/ext/pig-0.8.0.jar b/ext/pig-0.8.0.jar new file mode 100644 index 0000000..ab8452f --- /dev/null +++ b/ext/pig-0.8.0.jar Binary files differ diff --git a/ext/piggybank-0.12.0.jar b/ext/piggybank-0.12.0.jar new file mode 100644 index 0000000..252bb64 --- /dev/null +++ b/ext/piggybank-0.12.0.jar Binary files differ diff --git a/ext/scalatest-1.2.jar b/ext/scalatest-1.2.jar new file mode 100644 index 0000000..1ba1f35 --- /dev/null +++ b/ext/scalatest-1.2.jar Binary files differ diff --git a/ext/snappy-java-1.0.4.1.jar b/ext/snappy-java-1.0.4.1.jar new file mode 100644 index 0000000..c64231f --- /dev/null +++ b/ext/snappy-java-1.0.4.1.jar Binary files differ diff --git a/ext/xmlenc-0.52.jar b/ext/xmlenc-0.52.jar new file mode 100644 index 0000000..fab67e4 --- /dev/null +++ b/ext/xmlenc-0.52.jar Binary files differ diff --git a/ext/zkclient-0.3.jar b/ext/zkclient-0.3.jar new file mode 100644 index 0000000..6dbd23c --- /dev/null +++ b/ext/zkclient-0.3.jar Binary files differ -- To view, visit https://gerrit.wikimedia.org/r/227768 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I69b23e4270eafb5b2912723c8a8591cbf0f82f63 Gerrit-PatchSet: 1 Gerrit-Project: operations/debs/kafka Gerrit-Branch: debian Gerrit-Owner: Ottomata <[email protected]> _______________________________________________ MediaWiki-commits mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
