[
https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009015#comment-16009015
]
ASF GitHub Bot commented on METRON-508:
---------------------------------------
GitHub user JonZeolla opened a pull request:
https://github.com/apache/metron/pull/586
METRON-508 Expand Elasticsearch templates to support the standard bro logs
## Contributor Comments
This PR makes it easier for someone with an existing bro install to send
some of their log files into Metron, based off of a combination of the [bro
documentation](https://www.bro.org/sphinx/script-reference/log-files.html) and
a fresh install of bro 2.5. There are future plans to expand on this via
[METRON-518](https://issues.apache.org/jira/browse/METRON-518) and
[METRON-908](https://issues.apache.org/jira/browse/METRON-908). Specifically,
this attempts to provide initial support the default-on fields of the following
logs:
-
[Conn](https://www.bro.org/sphinx/scripts/base/protocols/conn/main.bro.html#type-Conn::Info)
-
[DPD](https://www.bro.org/sphinx-git/scripts/base/frameworks/dpd/main.bro.html#type-DPD::Info)
-
[FTP](https://www.bro.org/sphinx/scripts/base/protocols/ftp/info.bro.html#type-FTP::Info)
-
[Files](https://www.bro.org/sphinx/scripts/base/frameworks/files/main.bro.html#type-Files::Info)
-
[Known::CertsInfo](https://www.bro.org/sphinx/scripts/policy/protocols/ssl/known-certs.bro.html#type-Known::CertsInfo)
-
[SMTP](https://www.bro.org/sphinx/scripts/base/protocols/smtp/main.bro.html#type-SMTP::Info)
-
[SSL](https://www.bro.org/sphinx/scripts/base/protocols/ssl/main.bro.html#type-SSL::Info)
-
[Weird](https://www.bro.org/sphinx/scripts/base/frameworks/notice/weird.bro.html#type-Weird::Info)
-
[Notice](https://www.bro.org/sphinx/scripts/base/frameworks/notice/main.bro.html#type-Notice::Info)
-
[DHCP](https://www.bro.org/sphinx/scripts/base/protocols/dhcp/main.bro.html#type-DHCP::Info)
-
[SSH](https://www.bro.org/sphinx/scripts/base/protocols/ssh/main.bro.html#type-SSH::Info)
-
[Software](https://www.bro.org/sphinx/scripts/base/frameworks/software/main.bro.html#type-Software::Info)
-
[Radius](https://www.bro.org/sphinx/scripts/base/protocols/radius/main.bro.html#type-RADIUS::Info)
-
[X509](https://www.bro.org/sphinx/scripts/base/files/x509/main.bro.html#type-X509::Info)
## Testing
1. Create a working directory and pull in this PR
```
mkdir ~/metron-508
git clone https://github.com/apache/incubator-metron
~/metron-508/incubator-metron
cd ~/metron-508/incubator-metron
git remote add jonzeolla https://github.com/jonzeolla/incubator-metron
git pull jonzeolla METRON-508
```
1. Modify
[this](https://github.com/JonZeolla/incubator-metron/blob/METRON-508/metron-deployment/vagrant/full-dev-platform/Vagrantfile#L20)
to remove `sensors,` (to spin up the real sensors).
```
sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/"
metron-deployment/vagrant/full-dev-platform/Vagrantfile
```
1. Start up full-dev.
```
cd metron-deployment/vagrant/full-dev-platform
vagrant up
```
1. Set up the environment in full-dev.
```
vagrant ssh
sudo su -
export PATH=$PATH:/usr/local/bro/bin
service monit stop && service sensor-stubs stop bro && broctl stop
yum -y install jq wireshark
```
1. Configure kafka in local.bro.
```
sed -i 's/redef Kafka::logs_to_send = .*/redef Kafka::logs_to_send =
set(HTTP::LOG, DNS::LOG, Conn::LOG, DPD::LOG, DHCP::LOG, FTP::LOG, SSH::LOG,
SSL::LOG, SMTP::LOG, RADIUS::LOG, Weird::LOG, Files::LOG, Notice::LOG,
Software::LOG, Known::CERTS_LOG, X509::LOG);/'
/usr/local/bro/share/bro/site/local.bro
echo "redef Kafka::debug = \"all\";" >>
/usr/local/bro/share/bro/site/local.bro
echo "redef Known::cert_tracking = ALL_HOSTS;" >>
/usr/local/bro/share/bro/site/local.bro
echo "redef Software::asset_tracking = ALL_HOSTS;" >>
/usr/local/bro/share/bro/site/local.bro
```
1. Monitor the bro kafka topic
```
# Open a new terminal
cd
~/metron-508/incubator-metron/metron-deployment/vagrant/full-dev-platform
vagrant ssh
sudo su -
export PATH=$PATH:/usr/local/bro/bin:/usr/hdp/current/kafka-broker/bin
kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro
```
1. Monitor the storm logs.
```
# Open a new terminal
cd
~/metron-508/incubator-metron/metron-deployment/vagrant/full-dev-platform
vagrant ssh
sudo su -
export PATH=$PATH:/usr/local/bro/bin:/usr/hdp/current/kafka-broker/bin
# Look at the storm logs (The "failed to parse" errors for ip_src_addr
and ip_dst_addr are expected, and should be addressed as a part of METRON-939)
tail -f /var/log/storm/workers-artifacts/indexing-*/*/worker.log
```
1. Run bro against some public pcaps.
```
# In the first of your three terminals
# These are kept separate so that the flat file log output won't stomp
the prior ones, for ingest validation
mkdir -p ~/brotmp/nitroba ~/brotmp/example-traffic ~/brotmp/ssh
~/brotmp/ftp ~/brotmp/radius
wget https://www.bro.org/static/traces/exercise-traffic.pcap -O
~/brotmp/example-traffic/exercise-traffic.pcap
wget
http://downloads.digitalcorpora.org/corpora/network-packet-dumps/2008-nitroba/nitroba.pcap
-O ~/brotmp/nitroba/nitroba.pcap
wget https://www.bro.org/static/traces/ssh.pcap -O ~/brotmp/ssh/ssh.pcap
wget
https://github.com/markofu/pcaps/blob/master/PracticalPacketAnalysis/ppa-capture-files/ftp.pcap?raw=true
-O ~/brotmp/ftp/ftp.pcap
wget
https://github.com/EmpowerSecurityAcademy/wireshark/blob/master/radius_localhost.pcapng?raw=true
-O ~/brotmp/radius/radius_localhost.pcapng
cd ~/brotmp/example-traffic
bro -r exercise-traffic.pcap /usr/local/bro/share/bro/site/local.bro -C
cd ~/brotmp/nitroba
bro -r nitroba.pcap /usr/local/bro/share/bro/site/local.bro -C
cd ~/brotmp/ssh
bro -r ssh.pcap /usr/local/bro/share/bro/site/local.bro -C
cd ~/brotmp/ftp
bro -r ftp.pcap /usr/local/bro/share/bro/site/local.bro -C
cd ~/brotmp/radius
editcap -F libpcap radius_localhost.pcapng radius_localhost.pcap
bro -r radius_localhost.pcap /usr/local/bro/share/bro/site/local.bro -C
```
1. Validate that terminals 2 and 3 don't have errors.
1. Verify proper indexing in ES and availability in kibana.
```
# Check around and make sure things look okay
declare -a exists notexists; for protocol in http dns conn dpd dhcp ftp
ssh ssl smtp radius weird files notice software known_certs x509; do if [[
$(curl -s -XGET "node1:9200/bro*/_search?q=protocol:${protocol}" | jq
'.hits.hits') == '[]' ]]; then notexists+=" ${protocol}"; else exists+="
${protocol}"; fi; done; if [ ${#notexists[@]} -ne 0 ]; then echo -e
"\n\n\033[0mThe following do exist in ES: ${exists[@]}\n\033[0;31mThe following
do NOT exist in ES: ${notexists[@]}\033[0m"; else echo 'All of the protocols
are in ES! Success!'; fi; unset exists notexists
# Check Kibana. For example:
http://node1:5000/app/kibana#/visualize/create?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-15y,mode:quick,to:now))&_a=(filters:!(),linked:!f,query:(query_string:(analyze_wildcard:!t,query:'*')),uiState:(),vis:(aggs:!((id:'3',params:(field:protocol,orderBy:'2',size:20),schema:segment,type:terms),(id:'2',schema:metric,type:count)),type:histogram))&indexPattern=bro*&type=histogram
# OPTIONAL testing
# Run `/usr/share/elasticsearch/bin/plugin install
mobz/elasticsearch-head` and look around
curl -XGET node1:9200/_cat/indices # First column should be all green
curl -XGET "node1:9200/bro*/_count" # Check the count of entries in the
bro index, you can re-run bro against specific PCAPs and watch this increase,
etc.
```
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.
Please refer also to our [Build Verification
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:
### For all changes:
- [X] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [X] Does your PR title start with METRON-XXXX where XXXX is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [X] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [X] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [X] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up
for your personal repository such that your branches are built there before
submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JonZeolla/incubator-metron METRON-508
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/586.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #586
----
commit 28990c61fb249c286f6eaac09be33e529a9dd7f6
Author: Jon Zeolla <[email protected]>
Date: 2017-05-12T13:31:32Z
METRON-508 Expand Elasticsearch templates to support the standard bro logs
----
> Expand Elasticsearch templates to support the standard bro logs
> ---------------------------------------------------------------
>
> Key: METRON-508
> URL: https://issues.apache.org/jira/browse/METRON-508
> Project: Metron
> Issue Type: Sub-task
> Reporter: Jon Zeolla
> Assignee: Jon Zeolla
> Priority: Minor
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> The current elasticsearch templates do not support any logs other than Conn,
> HTTP, and DNS. We should provide additional templates so that an
> out-of-the-box bro install can send all of its logs into Metron and they will
> get probably indexed in elasticsearch.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
