[
https://issues.apache.org/jira/browse/METRON-812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951058#comment-15951058
]
ASF GitHub Bot commented on METRON-812:
---------------------------------------
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/501#discussion_r109178396
--- Diff: metron-sensors/bro-plugin-kafka/README.md ---
@@ -0,0 +1,160 @@
+Bro Logging Output to Kafka
+===========================
+
+A Bro log writer that sends logging output to Kafka. This provides a
convenient
+means for tools in the Hadoop ecosystem, such as Storm, Spark, and others,
to
+process the data generated by Bro.
+
+Installation
+------------
+
+Install librdkafka (https://github.com/edenhill/librdkafka), a native
client
+library for Kafka. This plugin has been tested against the latest release
of
+librdkafka, which at the time of this writing is v0.9.4. In order to
support interacting
+with a kerberized kafka, you will need libsasl2 installed
+
+ # curl -L https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz
| tar xvz
+ # cd librdkafka-0.9.4/
+ # ./configure --enable-sasl=true
+ # make
+ # sudo make install
+
+Then compile this Bro plugin using the following commands.
+
+ # ./configure --bro-dist=$BRO_SRC
+ # make
+ # sudo make install
+
+Run the following command to ensure that the plugin was installed
successfully.
+
+ # bro -N Bro::Kafka
+ Bro::Kafka - Writes logs to Kafka (dynamic, version 0.1)
+
+Activation
+----------
+
+The easiest way to enable Kafka output is to load the plugin's
+``logs-to-kafka.bro`` script. If you are using BroControl, the following
lines
+added to local.bro will activate it.
+
+```
+@load Bro/Kafka/logs-to-kafka.bro
+redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG);
+redef Kafka::topic_name = "bro";
+redef Kafka::kafka_conf = table(
+ ["metadata.broker.list"] = "localhost:9092"
+);
+```
+
+This example will send all HTTP, DNS, and Conn logs to a Kafka broker
running on
+the localhost to a topic called ``bro``. Any configuration value accepted
by
+librdkafka can be added to the ``kafka_conf`` configuration table.
+
+Settings
+--------
+
+### ``kafka_conf``
+
+The global configuration settings for Kafka. These values are passed
through
+directly to librdkafka. Any valid librdkafka settings can be defined in
this
+table. The full set of valid librdkafka settings are available
+[here](https://github.com/edenhill/librdkafka/blob/v0.9.4/CONFIGURATION.md).
+
+```
+redef Kafka::kafka_conf = table(
+ ["metadata.broker.list"] = "localhost:9092",
+ ["client.id"] = "bro"
+);
+```
+
+### ``topic_name``
+
+The name of the topic in Kafka where all Bro logs will be sent to.
+
+```
+redef Kafka::topic_name = "bro";
+```
+
+### ``max_wait_on_shutdown``
+
+The maximum number of milliseconds that the plugin will wait for any
backlog of
+queued messages to be sent to Kafka before forced shutdown.
+
+```
+redef Kafka::max_wait_on_shutdown = 3000;
+```
+
+### ``tag_json``
+
+If true, a log stream identifier is appended to each JSON-formatted
message. For
+example, a Conn::LOG message will look like ``{ 'conn' : { ... }}``.
+
+```
+redef Kafka::tag_json = T;
+```
+
+### ``debug``
+
+A comma separated list of debug contexts in librdkafka which you want to
+enable. The available contexts are:
+* generic
+* broker
+* topic
+* metadata
+* queue
+* msg
+* protocol
+* cgrp
+* security
+* fetch
+* feature
+* all
+
+Kerberos
+--------
+
+This plugin supports producing messages from a kerberized kafka. There
+are a couple of prerequisites and a couple of settings to set.
+
+### SASL
+If you are using SASL as a security protocol for kafka, then you must have
+libsasl or libsasl2 installed. You can tell if sasl is enabled by
+running the following from the directory in which you have build
+librdkafka:
+```
+examples/rdkafka_example -X builtin.features
+builtin.features = gzip,snappy,ssl,sasl,regex
+```
+
+### Producer Config
+
+As stated above, you can configure the producer kafka configs in
+`${BRO_HOME}/share/bro/site/local.bro`. There are a few configs
+necessary to set, which are described
+[here](https://github.com/edenhill/librdkafka/wiki/Using-SASL-with-librdkafka).
+For an environment where the following is true:
+* The broker is `node1:6667`
+* This kafka is using `SASL_PLAINTEXT` as the security protocol
+* The keytab used is the `metron` keytab
+* The service principal for `metron` is `[email protected]`
+
+The kafka topic `bro` has been given permission for the `metron` user to
+write:
+```
+# login using the metron user
+kinit -kt /etc/security/keytabs/metron.headless.keytab [email protected]
+${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer
kafka.security.auth.SimpleAclAuthorizer --authorizer-properties
zookeeper.connect=node1:2181 --add --allow-principal User:metron --topic bro
--- End diff --
Maybe we can just specify ${KAFKA_HOME} instead of ${HDP_HOME}. There is
nothing HDP-specific about this.
> Make the bro-kafka plugin work with kerberos
> --------------------------------------------
>
> Key: METRON-812
> URL: https://issues.apache.org/jira/browse/METRON-812
> Project: Metron
> Issue Type: Improvement
> Reporter: Casey Stella
> Assignee: Casey Stella
> Labels: kerberos
>
> The bro-kafka plugin does not currently support kerberos. This JIRA should
> * make the version of librdkafka supported 0.9.4
> * ensure the plugin can write to a kerberized kafka
> * provide instructions on how to configure the plugin to write to a
> kerberized kafka
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
