GitHub user JonZeolla opened a pull request:
https://github.com/apache/incubator-metron/pull/547
METRON-858 bro-plugin-kafka is throwing segfaults
## Contributor Comments
This PR is a follow-on of #545. Please DO NOT MERGE until the outstanding
items are all completed.
#### Outstanding items:
- [ ] Thoroughly test example 3
- [ ] Test at scale
The primary change here resolves a thread safety issue that is only seen
when under load. It has been reported in numerous places, but I've seen it
best documented [here](https://github.com/bro/bro-plugins/issues/43).
### Testing
The following steps can be used to validate the PR. (Mostly extracted from
METRON-883's testing steps)
1. Create a working directory.
```
mkdir metron-858
cd metron-858
```
1. Launch a CentOS host.
```
vagrant init bento/centos-6.7
vagrant up
vagrant ssh
```
1. Install some dependencies.
```
sudo su -
yum -y install epel-release
yum -y install "@Development tools" java-1.8.0-openjdk cmake
libpcap-devel openssl-devel python-devel
```
1. Create a new `HDP.repo` Yum repository; this will allow us to install
Kafka.
```
cat << EOF > /etc/yum.repos.d/HDP.repo
[HDP-2.5]
name=HDP-2.5
baseurl=http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0
path=/
enabled=1
gpgcheck=0
EOF
```
1. Install and start Kafka.
```
yum -y install kafka
export PATH=$PATH:/usr/hdp/current/kafka-broker/bin
zookeeper-server start
kafka start
```
1. Install Librdkafka 0.9.4.
```
wget https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz -O -
| tar -xz
cd librdkafka-0.9.4/
./configure --prefix=/usr
make
make install
```
1. Add Librdkafka to our default load path.
```
echo "/usr/lib" >> /etc/ld.so.conf.d/bro-plugin.conf
ldconfig -v
```
1. Build and install Bro.
```
yum -y install cmake libpcap-devel openssl-devel python-devel
wget https://www.bro.org/downloads/release/bro-2.4.1.tar.gz -O
~/bro-2.4.1.tar.gz
tar -xzf ~/bro-2.4.1.tar.gz -C ~
cd ~/bro-2.4.1
./configure --prefix=/usr
make
make install
```
1. Fetch the code from this PR.
```
git clone https://github.com/apache/incubator-metron ~/incubator-metron
cd ~/incubator-metron
git pull origin pull/XXX/head
```
1. Install the Bro Plugin.
```
cd metron-sensors/bro-plugin-kafka
./configure --bro-dist=/root/bro-2.4.1
--install-root=/usr/lib/bro/plugins/ --with-librdkafka=/usr
make
make install
```
1. Modify your `/usr/share/bro/site/local.bro`:
```
cat << EOF >> /usr/share/bro/site/local.bro
@load Bro/Kafka/logs-to-kafka.bro
redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG);
redef Kafka::topic_name = "bro";
redef Kafka::tag_json = T;
redef Kafka::kafka_conf = table( ["metadata.broker.list"] =
"localhost:9092" );
EOF
```
1. Create a virtual interface called `tap0` to listen on.
```
yum install -y tunctl
tunctl -p
ifconfig tap0 10.0.0.1 up
ip link set tap0 promisc on
```
1. Configure Bro to listen on virtual interface.
```
sed -i 's/eth0/tap0/g' /usr/etc/node.cfg
```
1. Create a Kafka topic called `bro`.
```
kafka-topics.sh --zookeeper localhost:2181 --create --topic bro
--partitions 1 --replication-factor 1
```
1. Make sure the Bro changes are installed and start Bro.
```
broctl deploy
```
1. Grab an example pcap file and replay some packet data through `tap0`.
Keep this running in a separate session.
```
yum -y install tcpreplay
wget
https://github.com/apache/incubator-metron/raw/master/metron-deployment/roles/sensor-test-mode/files/example.pcap
tcpreplay -i tap0 --loop=0 --stats=5 example.pcap
```
1. Ensure that data is hitting the `bro` topic in Kafka.
```
# kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro
--from-beginning
OpenJDK 64-Bit Server VM warning: If the number of processors is
expected to increase from one, then you should configure the number of parallel
GC threads appropriately using -XX:ParallelGCThreads=N
{metadata.broker.list=localhost:9092, request.timeout.ms=30000,
client.id=console-consumer-99442, security.protocol=PLAINTEXT}
{"dns":
{"ts":1493145915.795376,"uid":"CNfwFh1xJrsdwezojd","id.orig_h":"192.168.138.158","id.orig_p":60078,"id.resp_h":"192.168.138.2","id.resp_p":53,"proto":"udp","trans_id":18350,"query":"va872g.g90e1h.b8.642b63u.j985a2.v33e.37.pa269cc.e8mfzdgrf7g0.groupprograms.in","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["62.75.195.236"],"TTLs":[29.0],"rejected":false}}
{"dns":
{"ts":1493145916.433874,"uid":"CL3LrkiZoYceFU2Nh","id.orig_h":"192.168.138.158","id.orig_p":65315,"id.resp_h":"192.168.138.2","id.resp_p":53,"proto":"udp","trans_id":27248,"query":"ubb67.3c147o.u806a4.w07d919.o5f.f1.b80w.r0faf9.e8mfzdgrf7g0.groupprograms.in","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["62.75.195.236"],"TTLs":[29.0],"rejected":false}}
{"dns":
{"ts":1493145916.434025,"uid":"CbNL2S3VggZKyweUA6","id.orig_h":"192.168.138.158","id.orig_p":50683,"id.resp_h":"192.168.138.2","id.resp_p":53,"proto":"udp","trans_id":62139,"query":"r03afd2.c3008e.xc07r.b0f.a39.h7f0fa5eu.vb8fbl.e8mfzdgrf7g0.groupprograms.in","qclass":1,"qclass_name":"C_INTERNET","qtype":1,"qtype_name":"A","rcode":0,"rcode_name":"NOERROR","AA":false,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["62.75.195.236"],"TTLs":[29.0],"rejected":false}}
```
## Pull Request Checklist
In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:
### For all changes:
- [X] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [X] Does your PR title start with METRON-XXXX where XXXX is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [X] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [N/A] Have you included steps to reproduce the behavior or problem that
is being changed or addressed? (**See Contributor Comments**)
- [X] Have you included steps or a guide to how the change may be verified
and tested manually?
- [X] Have you ensured that the full suite of tests and checks have been
executed in the root incubating-metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```
- [N/A] Have you written or updated unit tests and or integration tests to
verify your changes?
- [N/A] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [X] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [X] Have you ensured that format looks appropriate for the output in
which it is rendered by building and verifying the site-book? If not then run
the following commands and the verify changes via
`site-book/target/site/index.html`:
```
cd site-book
bin/generate-md.sh
mvn site:site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up
for your personal repository such that your branches are built there before
submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JonZeolla/incubator-metron METRON-858
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/547.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #547
----
commit 2249b31211dd5856a9ae63fa03623091aeb5a983
Author: Nick Allen <[email protected]>
Date: 2017-04-24T17:35:32Z
METRON-883 Capture Bro Plugin Enhancements from bro/bro-plugins
commit 087533cd951ec7b17749e5d4b479dfac9f6ea42e
Author: Nick Allen <[email protected]>
Date: 2017-04-24T20:55:12Z
Improved README
commit ca05efe565e877f519771772a08fd12abb457902
Author: Jon Zeolla <[email protected]>
Date: 2017-04-25T13:33:36Z
Merge branch 'pr-545'
commit ac3daa48c4923c1c623ec0ccb37814c347f870cb
Author: Jon Zeolla <[email protected]>
Date: 2017-04-25T13:34:41Z
Thread safety fix
commit 08cd7b3a3df69931f3969d4fb41014b34b8fe2f5
Author: Jon Zeolla <[email protected]>
Date: 2017-04-25T17:20:04Z
Fix for cmake case sensitivity
commit fe32365ef854e369c7ebfb2578b84093fc7bd43f
Author: Jon Zeolla <[email protected]>
Date: 2017-04-25T17:20:34Z
Document logging predicates in bro
commit 4aa19c8a3dc5f1c886ad718e118dd35f6c5936d4
Author: Jon Zeolla <[email protected]>
Date: 2017-04-25T18:58:20Z
Merge branch 'master' of https://github.com/apache/incubator-metron into
METRON-858
commit 3f728a7b8d4d30f12e2d8114de749add41a90973
Author: Jon Zeolla <[email protected]>
Date: 2017-04-25T19:26:37Z
Add the Bro kafka plugin to the Metron Sensors README
commit 01127b686d3cd5936ad7da5cf9e5c48f09256709
Author: Jon Zeolla <[email protected]>
Date: 2017-04-25T19:27:29Z
Fix link for site-book
commit 8b5e845eab153ec9c8408f4da3f90be5252059cf
Author: Jon Zeolla <[email protected]>
Date: 2017-04-25T19:29:33Z
Be more succinct in the example
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
