Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/547#discussion_r113483664
--- Diff: metron-sensors/bro-plugin-kafka/README.md ---
@@ -94,6 +95,52 @@ event bro_init()
}
```
+### Example 3
+
+As documented in
[METRON-285](https://issues.apache.org/jira/browse/METRON-285) and
[METRON-286](https://issues.apache.org/jira/browse/METRON-286), various
components in Metron do not currently support IPv6. Because of this, you may
not want to send bro logs that contain IPv6 source or destination IPs into
Metron. In this example, we are assuming a somewhat standard bro configuration
for sending logs into a Metron cluster, such that:
+ * Each type of bro log is sent to the `bro` topic, but is tagged with the
appropriate log type (such as `http`, `dns`, or `conn`). This is done by
setting `topic_name` to `bro`, setting `$path` to an empty string (or leaving
it unset), and by setting `tag_json` to true.
+ * The Kafka writer is set appropriately to send logs to the `bro` Kafka
topic being used in your Metron cluster. This requires that your `kafka_conf`
and `$config` tables are appropriately configured.
+
+```
+@load Bro/Kafka/logs-to-kafka.bro
+redef Kafka::topic_name = "bro";
+redef Kafka::tag_json = T;
+redef Kafka::kafka_conf = table(
+ ["metadata.broker.list"] = "localhost:9092"
+);
+
+
+event bro_init() &priority=-5
+{
+ # handles HTTP
+ Log::add_filter(HTTP::LOG, [$name = "kafka-http",
+ $writer = Log::WRITER_KAFKAWRITER,
+ $pred(rec: HTTP::Info) = { return ! (( |rec$id$orig_h| == 128 ||
|rec$id$resp_h| == 128 )); },
+ $config = table(["stream_id"] = fmt("%s", HTTP::LOG))
+ ]);
+
+ # handles DNS
+ Log::add_filter(DNS::LOG, [$name = "kafka-dns",
+ $writer = Log::WRITER_KAFKAWRITER,
+ $pred(rec: DNS::Info) = { return ! (( |rec$id$orig_h| == 128 ||
|rec$id$resp_h| == 128 )); },
+ $config = table(["stream_id"] = fmt("%s", DNS::LOG))
+ ]);
+
+ # handles Conn
+ Log::add_filter(Conn::LOG, [$name = "kafka-conn",
+ $writer = Log::WRITER_KAFKAWRITER,
+ $pred(rec: Conn::Info) = { return ! (( |rec$id$orig_h| == 128 ||
|rec$id$resp_h| == 128 )); },
+ $config = table(["stream_id"] = fmt("%s", Conn::LOG))
+ ]);
+}
--- End diff --
With this script, I would expect to find a total of 6 log filters having
been created. The first 3 created by `Bro/Kafka/logs-to-kafka.bro` and then
the last 3 created by your `bro_init()` function. To avoid this, I think what
you want to do something more like this...
```
@load Bro/Kafka/logs-to-kafka.bro
redef Kafka::topic_name = "";
redef Kafka::tag_json = T;
event bro_init() &priority=-5
{
# handles HTTP
Log::add_filter(HTTP::LOG, [
$name = "kafka-http",
$writer = Log::WRITER_KAFKAWRITER,
$pred(rec: HTTP::Info) = { return ! (( |rec$id$orig_h| == 128 ||
|rec$id$resp_h| == 128 )); },
$config = table(
["stream_id"] = fmt("%s", HTTP::LOG),
["metadata.broker.list"] = "localhost:9092"
)
]);
# handles DNS
Log::add_filter(DNS::LOG, [
$name = "kafka-dns",
$writer = Log::WRITER_KAFKAWRITER,
$pred(rec: DNS::Info) = { return ! (( |rec$id$orig_h| == 128 ||
|rec$id$resp_h| == 128 )); },
$config = table(
["stream_id"] = fmt("%s", DNS::LOG),
["metadata.broker.list"] = "localhost:9092"
)
]);
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
