Repository: metron-bro-plugin-kafka Updated Branches: refs/heads/master 25e40be1c -> c035becc1
METRON-1303 Reorganize the metron-bro-plugin-kafka (JonZeolla) closes apache/metron-bro-plugin-kafka#1 Project: http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/repo Commit: http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/commit/c035becc Tree: http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/tree/c035becc Diff: http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/diff/c035becc Branch: refs/heads/master Commit: c035becc16cfa5f0b37ecdc6c09beb360955d4c0 Parents: 25e40be Author: JonZeolla <zeo...@gmail.com> Authored: Tue Nov 14 07:35:00 2017 -0500 Committer: jonzeolla <jonzeo...@apache.org> Committed: Tue Nov 14 07:35:00 2017 -0500 ---------------------------------------------------------------------- CMakeLists.txt | 2 +- README.md | 29 ++++++++++++----------------- src/Plugin.cc | 6 +++--- src/Plugin.h | 2 +- tests/kafka/show-plugin.bro | 2 +- 5 files changed, 18 insertions(+), 23 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/c035becc/CMakeLists.txt ---------------------------------------------------------------------- diff --git a/CMakeLists.txt b/CMakeLists.txt index 30bf3b5..2bc8d4b 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -23,7 +23,7 @@ find_package(OpenSSL) if (LIBRDKAFKA_FOUND AND OPENSSL_FOUND) include_directories(BEFORE ${LibRDKafka_INCLUDE_DIR} ${OpenSSL_INCLUDE_DIR}) - bro_plugin_begin(BRO KAFKA) + bro_plugin_begin(APACHE KAFKA) bro_plugin_cc(src/KafkaWriter.cc) bro_plugin_cc(src/Plugin.cc) bro_plugin_cc(src/TaggedJSON.cc) http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/c035becc/README.md ---------------------------------------------------------------------- diff --git a/README.md b/README.md index 427988b..2901e18 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,8 @@ -Logging Bro Output to Kafka -=========================== +# Logging Bro Output to Kafka A Bro log writer that sends logging output to Kafka. This provides a convenient means for tools in the Hadoop ecosystem, such as Storm, Spark, and others, to process the data generated by Bro. -Installation ------------- +## Installation 1. Install [librdkafka](https://github.com/edenhill/librdkafka), a native client library for Kafka. This plugin has been tested against the latest release of librdkafka, which at the time of this writing is v0.9.4. @@ -29,12 +27,11 @@ Installation 1. Run the following command to ensure that the plugin was installed successfully. ``` - $ bro -N Bro::Kafka - Bro::Kafka - Writes logs to Kafka (dynamic, version 0.1) + $ bro -N Apache::Kafka + Apache::Kafka - Writes logs to Kafka (dynamic, version 0.1) ``` -Activation ----------- +## Activation The following examples highlight different ways that the plugin can be used. Simply add the Bro script language to your `local.bro` file (for example, `/usr/share/bro/site/local.bro`) as shown to demonstrate the example. @@ -46,7 +43,7 @@ The goal in this example is to send all HTTP and DNS records to a Kafka topic na * Defining `logs_to_send` will ensure that only HTTP and DNS records are sent. ``` -@load Bro/Kafka/logs-to-kafka.bro +@load Apache/Kafka/logs-to-kafka.bro redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG); redef Kafka::topic_name = "bro"; redef Kafka::kafka_conf = table( @@ -63,7 +60,7 @@ It is also possible to send each log stream to a uniquely named topic. The goal * Each log writer accepts a separate configuration table. ``` -@load Bro/Kafka/logs-to-kafka.bro +@load Apache/Kafka/logs-to-kafka.bro redef Kafka::topic_name = ""; redef Kafka::tag_json = T; @@ -101,7 +98,7 @@ You may want to configure bro to filter log messages with certain characteristic * If the log message contains a 128 byte long source or destination IP address, the log is not sent to kafka. ``` -@load Bro/Kafka/logs-to-kafka.bro +@load Apache/Kafka/logs-to-kafka.bro redef Kafka::topic_name = "bro"; redef Kafka::tag_json = T; @@ -141,11 +138,10 @@ event bro_init() &priority=-5 #### Notes * `logs_to_send` is mutually exclusive with `$pred`, thus for each log you want to set `$pred` on, you must individually setup a `Log::add_filter` and refrain from including that log in `logs_to_send`. - * You can also filter IPv6 logs from within your Metron cluster [using Stellar](../../metron-stellar/stellar-common#IS_IP). In that case, you wouldn't apply a predicate in your bro configuration, and instead Stellar would filter the logs out before they were processed by the enrichment layer of Metron. + * You can also filter IPv6 logs from within your Metron cluster [using Stellar](https://github.com/apache/metron/tree/master/metron-stellar/stellar-common#is_ip). In that case, you wouldn't apply a predicate in your bro configuration, and instead Stellar would filter the logs out before they were processed by the enrichment layer of Metron. * It is also possible to use the `is_v6_subnet()` bro function in your predicate, as of their [2.5 release](https://www.bro.org/sphinx-git/install/release-notes.html#bro-2-5), however the above example should work on [bro 2.4](https://www.bro.org/sphinx-git/install/release-notes.html#bro-2-4) and newer, which has been the focus of the kafka plugin. -Settings --------- +## Settings ### `kafka_conf` @@ -204,8 +200,7 @@ enable. The available contexts are: * feature * all -Kerberos --------- +## Kerberos This plugin supports producing messages from a kerberized kafka. There are a couple of prerequisites and a couple of settings to set. @@ -242,7 +237,7 @@ ${KAFKA_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.Si The following is how the `${BRO_HOME}/share/bro/site/local.bro` looks: ``` -@load Bro/Kafka/logs-to-kafka.bro +@load Apache/Kafka/logs-to-kafka.bro redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG); redef Kafka::topic_name = "bro"; redef Kafka::tag_json = T; http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/c035becc/src/Plugin.cc ---------------------------------------------------------------------- diff --git a/src/Plugin.cc b/src/Plugin.cc index f3dec8b..f2f63a5 100644 --- a/src/Plugin.cc +++ b/src/Plugin.cc @@ -17,18 +17,18 @@ #include "Plugin.h" -namespace plugin { namespace Bro_Kafka { +namespace plugin { namespace Apache_Kafka { Plugin plugin; }} -using namespace plugin::Bro_Kafka; +using namespace plugin::Apache_Kafka; plugin::Configuration Plugin::Configure() { AddComponent(new ::logging::Component("KafkaWriter", ::logging::writer::KafkaWriter::Instantiate)); plugin::Configuration config; - config.name = "Bro::Kafka"; + config.name = "Apache::Kafka"; config.description = "Writes logs to Kafka"; config.version.major = 0; config.version.minor = 1; http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/c035becc/src/Plugin.h ---------------------------------------------------------------------- diff --git a/src/Plugin.h b/src/Plugin.h index a169076..e99e91b 100644 --- a/src/Plugin.h +++ b/src/Plugin.h @@ -21,7 +21,7 @@ #include "KafkaWriter.h" #include <plugin/Plugin.h> -namespace plugin { namespace Bro_Kafka { +namespace plugin { namespace Apache_Kafka { class Plugin : public ::plugin::Plugin { protected: http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/c035becc/tests/kafka/show-plugin.bro ---------------------------------------------------------------------- diff --git a/tests/kafka/show-plugin.bro b/tests/kafka/show-plugin.bro index 4e8dd6a..1eb3dde 100644 --- a/tests/kafka/show-plugin.bro +++ b/tests/kafka/show-plugin.bro @@ -15,5 +15,5 @@ # limitations under the License. # -# @TEST-EXEC: bro -NN Bro::Kafka >output +# @TEST-EXEC: bro -NN Apache::Kafka >output # @TEST-EXEC: btest-diff output