Repository: metron-bro-plugin-kafka Updated Branches: refs/heads/master 37f05f8f4 -> 8d0400fef (forced update)
METRON-1866 Improve metron-bro-plugin-kafka documentation (JonZeolla via jonzeolla) closes apache/metron-bro-plugin-kafka#17 Project: http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/repo Commit: http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/commit/8d0400fe Tree: http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/tree/8d0400fe Diff: http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/diff/8d0400fe Branch: refs/heads/master Commit: 8d0400fef9488908e46e4a0922e01d19f52b04c2 Parents: 772788d Author: JonZeolla <[email protected]> Authored: Sat Nov 10 14:29:03 2018 -0500 Committer: Jon Zeolla <[email protected]> Committed: Sat Nov 10 14:52:35 2018 -0500 ---------------------------------------------------------------------- README.md | 98 +++++++++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 76 insertions(+), 22 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/8d0400fe/README.md ---------------------------------------------------------------------- diff --git a/README.md b/README.md index ae6b260..ee21a7e 100644 --- a/README.md +++ b/README.md @@ -12,24 +12,79 @@ This software is a part of the [Apache Metron](http://metron.apache.org/) projec ## Installation +### `bro-pkg` Installation + +`bro-pkg` is the preferred mechanism for installing this plugin, as it will dynamically retrieve, build, test, and load the plugin. Note, that you will still need to [activate](#activation) and configure the plugin after your installation. + +1. Install [librdkafka](https://github.com/edenhill/librdkafka), a native client library for Kafka. This plugin has been tested against the latest release of librdkafka, which at the time of this writing is v0.9.4. + + In order to use this plugin within a kerberized Kafka environment, you will also need `libsasl2` installed and will need to pass `--enable-sasl` to the `configure` script. + + ``` + $ curl -L https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz | tar xvz + $ cd librdkafka-0.9.4/ + $ ./configure --enable-sasl + $ make + $ sudo make install + ``` + +1. Configure `bro-pkg` by following the quickstart guide [here](https://bro-package-manager.readthedocs.io/en/stable/quickstart.html). + +1. Install the plugin using `bro-pkg install`. + + ``` + $ bro-pkg install apache/metron-bro-plugin-kafka --version master + The following packages will be INSTALLED: + bro/apache/metron-bro-plugin-kafka (master) + + Verify the following REQUIRED external dependencies: + (Ensure their installation on all relevant systems before proceeding): + from bro/apache/metron-bro-plugin-kafka (master): + librdkafka ~0.9.4 + + Proceed? [Y/n] + bro/apache/metron-bro-plugin-kafka asks for LIBRDKAFKA_ROOT (Path to librdkafka installation tree) ? [/usr/local/lib] + Saved answers to config file: /home/jonzeolla/.bro-pkg/config + Running unit tests for "bro/apache/metron-bro-plugin-kafka" + all 10 tests successful + + + Installing "bro/apache/metron-bro-plugin-kafka"........ + Installed "bro/apache/metron-bro-plugin-kafka" (master) + Loaded "bro/apache/metron-bro-plugin-kafka" + ``` + +1. Run the following command to ensure that the plugin was installed successfully. + + ``` + $ bro -N Apache::Kafka + Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3) + ``` + +### Manual Installation + +Manually installing the plugin should only occur in situations where installing and configuring `bro-pkg` is not reasonable, such as in a docker container. If you are running bro in an environment where you do not have Internet connectivity, investigate [bundles](https://bro-package-manager.readthedocs.io/en/stable/bro-pkg.html#bundle) or creating an internal [package source](https://bro-package-manager.readthedocs.io/en/stable/source.html). + +These instructions could also be helpful if you were interested in distributing this as a package (such as a deb or rpm). + 1. Install [librdkafka](https://github.com/edenhill/librdkafka), a native client library for Kafka. This plugin has been tested against the latest release of librdkafka, which at the time of this writing is v0.9.4. In order to use this plugin within a kerberized Kafka environment, you will also need `libsasl2` installed and will need to pass `--enable-sasl` to the `configure` script. ``` - curl -L https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz | tar xvz - cd librdkafka-0.9.4/ - ./configure --enable-sasl - make - sudo make install + $ curl -L https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz | tar xvz + $ cd librdkafka-0.9.4/ + $ ./configure --enable-sasl + $ make + $ sudo make install ``` 1. Build the plugin using the following commands. ``` - ./configure --bro-dist=$BRO_SRC - make - sudo make install + $ ./configure --bro-dist=$BRO_SRC + $ make + $ sudo make install ``` 1. Run the following command to ensure that the plugin was installed successfully. @@ -43,22 +98,21 @@ This software is a part of the [Apache Metron](http://metron.apache.org/) projec The following examples highlight different ways that the plugin can be used. Simply add the Bro script language to your `local.bro` file (for example, `/usr/share/bro/site/local.bro`) as shown to demonstrate the example. -### Example 1 +### Example 1 - Send a list of logs to kafka The goal in this example is to send all HTTP and DNS records to a Kafka topic named `bro`. * Any configuration value accepted by librdkafka can be added to the `kafka_conf` configuration table. - * By defining `topic_name` all records will be sent to the same Kafka topic. - * Defining `logs_to_send` will ensure that only HTTP and DNS records are sent. + * The `topic_name` will default to send all records to a single Kafka topic called 'bro'. + * Defining `logs_to_send` will send the HTTP and DNS records to the brokers specified in your `Kafka::kafka_conf`. ``` @load packages/metron-bro-plugin-kafka/Apache/Kafka redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG); -redef Kafka::topic_name = "bro"; redef Kafka::kafka_conf = table( - ["metadata.broker.list"] = "localhost:9092" + ["metadata.broker.list"] = "server1.example.com:9092,server2.example.com:9092" ); ``` -### Example 2 +### Example 2 - Send all active logs This plugin has the ability send all active logs to kafka with the following configuration. @@ -70,7 +124,7 @@ redef Kafka::kafka_conf = table( ); ``` -### Example 3 +### Example 3 - Send all active logs with exclusions You can also specify a blacklist of bro logs to ensure they aren't being sent to kafka regardless of the `Kafka::send_all_active_logs` and `Kafka::logs_to_send` configurations. In this example, we will send all of the enabled logs except for the Conn log. @@ -84,7 +138,7 @@ redef Kafka::kafka_conf = table( ); ``` -### Example 4 +### Example 4 - Send logs to unique topics It is also possible to send each log stream to a uniquely named topic. The goal in this example is to send all HTTP records to a Kafka topic named `http` and all DNS records to a separate Kafka topic named `dns`. * The `topic_name` value must be set to an empty string. @@ -97,7 +151,7 @@ It is also possible to send each log stream to a uniquely named topic. The goal redef Kafka::topic_name = ""; redef Kafka::tag_json = T; -event bro_init() +event bro_init() &priority=-10 { # handles HTTP local http_filter: Log::Filter = [ @@ -123,19 +177,18 @@ event bro_init() } ``` -### Example 5 +### Example 5 - Bro log filtering You may want to configure bro to filter log messages with certain characteristics from being sent to your kafka topics. For instance, Metron currently doesn't support IPv6 source or destination IPs in the default enrichments, so it may be helpful to filter those log messages from being sent to kafka (although there are [multiple ways](#notes) to approach this). In this example we will do that that, and are assuming a somewhat standard bro kafka plugin configuration, such that: - * All bro logs are sent to the `bro` topic, by configuring `Kafka::topic_name`. - * Each JSON message is tagged with the appropriate log type (such as `http`, `dns`, or `conn`), by setting `tag_json` to true. + * All bro logs are sent to the default `bro` topic. + * Each JSON message is tagged with the appropriate log type (such as `http`, `dns`, or `conn`), by setting `Kafka::tag_json` to true. * If the log message contains a 128 byte long source or destination IP address, the log is not sent to kafka. ``` @load packages/metron-bro-plugin-kafka/Apache/Kafka -redef Kafka::topic_name = "bro"; redef Kafka::tag_json = T; -event bro_init() &priority=-5 +event bro_init() &priority=-10 { # handles HTTP Log::add_filter(HTTP::LOG, [ @@ -171,6 +224,7 @@ event bro_init() &priority=-5 #### Notes * `logs_to_send` is mutually exclusive with `$pred`, thus for each log you want to set `$pred` on, you must individually setup a `Log::add_filter` and refrain from including that log in `logs_to_send`. + * In Bro 2.5.x the bro project introduced a [logger function](https://www.bro.org/sphinx/cluster/index.html#logger) which removes the logging functions from the manager thread, and taking advantage of that is highly recommended. If you are running this plugin on Bro 2.4.x, you may encounter issues where the manager thread is taking on too much responsibility and pinning a single CPU core without the ability to spread the load across additional cores. In this case, it may be in your best interest to prefer using a bro logging predicate over filtering in your Metron cluster [using Stellar](https://github.com/apache/metron/tree/master/metron-stellar/stellar-common) in order to lessen the load of that thread. * You can also filter IPv6 logs from within your Metron cluster [using Stellar](https://github.com/apache/metron/tree/master/metron-stellar/stellar-common#is_ip). In that case, you wouldn't apply a predicate in your bro configuration, and instead Stellar would filter the logs out before they were processed by the enrichment layer of Metron. * It is also possible to use the `is_v6_subnet()` bro function in your predicate, as of their [2.5 release](https://www.bro.org/sphinx-git/install/release-notes.html#bro-2-5), however the above example should work on [bro 2.4](https://www.bro.org/sphinx-git/install/release-notes.html#bro-2-4) and newer, which has been the focus of the kafka plugin.
