GitHub user nickwallen opened a pull request:
https://github.com/apache/incubator-metron/pull/85
METRON-115 Fetch Threat Intel Data using TAXII
Created an Ansible role that performs automated deployment and setup of
OpenTAXII services. Functionality has also been added to fetch threat intel
data from 3rd parties using TAXII.
This role has not been integrated into any of the deployments; Amazon or
Vagrant. The integration will occur once all work related to the "pcap ingest
refresh" is complete.
Other things to keep in mind...
- Python 2.7 is needed to support OpenTAXII, but CentOS 6 comes with Python
2.6. The `centos-release-scl` repository is used to install Python 2.7 without
impacting the existing Python 2.6.
- OpenTAXII is installed in a virtual environment to maintain isolation.
- The installation is currently backed by a sqlite database. OpenTAXII can
be configured with a proper database like MySQL, but that work has not been
done.
---
# OpenTAXII
Installs [OpenTAXII](https://github.com/EclecticIQ/OpenTAXII) as a deamon
that can be launched via a SysV service script. The complementary client
implementation, [Cabby](https://github.com/EclecticIQ/cabby) is also installed.
OpenTAXII is a robust Python implementation of TAXII Services that delivers
a rich feature set and friendly pythonic API.
[TAXII](https://stixproject.github.io/) (Trusted Automated eXchange of
Indicator Information) is a collection of specifications defining a set of
services and message exchanges used for sharing cyber threat intelligence
information between parties.
## Getting Started
After deployment completes the OpenTAXII service is installed and running.
A set of [Hail a TAXII](http://hailataxii.com/) threat intel collections have
been defined and configured. Use the `status` option to view the collections
that have been defined.
```
$ service opentaxii status
Checking opentaxii... Running
guest.phishtank_com 0
guest.Abuse_ch 0
guest.CyberCrime_Tracker 0
guest.EmergingThreats_rules 0
guest.Lehigh_edu 0
guest.MalwareDomainList_Hostlist 0
guest.blutmagie_de_torExits 0
guest.dataForLast_7daysOnly 0
guest.dshield_BlockList 0
```
Notice that each collections contain zero records. None of the data is
automatically synced during deployment. To sync the data manually use the
`sync` option as defined below. The following example does not provide a begin
and end time so the data will be fetched for the current day only.
```
# service opentaxii sync guest.blutmagie_de_torExits
2016-04-21 20:34:42,511 INFO: Starting new HTTP connection (1): localhost
2016-04-21 20:34:42,540 INFO: Response received for Inbox_Message from
http://localhost:9000/services/inbox
2016-04-21 20:34:42,542 INFO: Sending Inbox_Message to
http://localhost:9000/services/inbox
...
2016-04-21 20:34:42,719 INFO: Response received for Poll_Request from
http://localhost:9000/services/poll
2016-04-21 20:34:42,719 INFO: Content blocks count: 1618, is partial: False
```
The OpenTAXII service now contains 1,618 threat intel records indicating
Tor Exit nodes.
```
[root@source ~]# service opentaxii status
Checking opentaxii... Running
guest.phishtank_com 0
guest.Abuse_ch 0
guest.CyberCrime_Tracker 0
guest.EmergingThreats_rules 0
guest.Lehigh_edu 0
guest.MalwareDomainList_Hostlist 0
guest.blutmagie_de_torExits 1618
guest.dataForLast_7daysOnly 0
guest.dshield_BlockList 0
```
## Usage
A standard SysV script has been installed to manage OpenTAXII. The
following functions are available.
`start` `stop` `restart` the OpenTAXII service
`status` of the OpenTAXII service. The command displays the collections
that have been defined and the number of records in each.
```
$ service opentaxii status
Checking opentaxii... Running
guest.phishtank_com 984
guest.Abuse_ch 45
guest.CyberCrime_Tracker 482
guest.EmergingThreats_rules 0
guest.Lehigh_edu 1030
guest.MalwareDomainList_Hostlist 84
guest.blutmagie_de_torExits 3236
guest.dataForLast_7daysOnly 3377
guest.dshield_BlockList 0
```
`setup` Initializes the services and collections required to operate the
OpenTAXII service. This will destroy all existing data. The user is prompted
to continue before any data is destroyed.
```
# service opentaxii setup
WARNING: force reset and destroy all opentaxii data? [Ny]: y
Stopping opentaxii ..Ok
2016-04-21T19:56:01.886157Z [opentaxii.server] info: api.persistence.loaded
{timestamp=2016-04-21T19:56:01.886157Z, logger=opentaxii.server,
api_class=SQLDatabaseAPI, event=api.persistence.loaded, level=info}
2016-04-21T19:56:01.896503Z [opentaxii.server] info: api.auth.loaded
{timestamp=2016-04-21T19:56:01.896503Z, logger=opentaxii.server,
api_class=SQLDatabaseAPI, event=api.auth.loaded, level=info}
2016-04-21T19:56:01.896655Z [opentaxii.server] info: taxiiserver.configured
{timestamp=2016-04-21T19:56:01.896655Z, logger=opentaxii.server,
event=taxiiserver.configured, level=info}
...
Ok
```
`sync [collection] [begin-at] [end-at]` Syncs the threat intel data
available at [Hail a TAXII](http://hailataxii.com/). If no begin and end date
is provided then data is synced over the current day only.
- `collection` Name of the collection to sync.
- `begin-at` Exclusive begin of time window; ISO8601
- `end-at` Inclusive end of time window; ISO8601
```
$ service opentaxii sync guest.phishtank_com
+ /usr/local/opentaxii/opentaxii-venv/bin/taxii-proxy --poll-path
http://hailataxii.com/taxii-data --poll-collection guest.phishtank_com
--inbox-path http://localhost:9000/services/guest.phishtank_com-inbox
--inbox-collection guest.phishtank_com --binding urn:stix.mitre.org:xml:1.1.1
--begin 2016-04-21 --end 2016-04-22
2016-04-21 17:36:23,778 INFO: Sending Poll_Request to
http://hailataxii.com/taxii-data
2016-04-21 17:36:23,784 INFO: Starting new HTTP connection (1):
hailataxii.com
2016-04-21 17:36:24,175 INFO: Response received for Poll_Request from
http://hailataxii.com/taxii-data
2016-04-21 17:36:24,274 INFO: Sending Inbox_Message to
http://localhost:9000/services/guest.phishtank_com-inbox
...
2016-04-21 17:36:34,867 INFO: Response received for Poll_Request from
http://localhost:9000/services/guest.phishtank_com-poll
2016-04-21 17:36:34,868 INFO: Content blocks count: 6993, is partial: False
```
### Troubleshooting
Should you need to explore the installation, here are instructions on doing
so.
OpenTAXII is installed in a virtual environment. Before exploring the
environment run the following commands to perform the necessary setup. The
specific paths may change depending on your Ansible settings.
```
export LD_LIBRARY_PATH=/opt/rh/python27/root/usr/lib64
export OPENTAXII_CONFIG=/usr/local/opentaxii/etc/opentaxii-conf.yml
cd /usr/local/opentaxii
. opentaxii-venv/bin/activate
```
Discover available services.
```
taxii-discovery --discovery http://localhost:9000/services/discovery
taxii-discovery --discovery http://hailataxii.com/taxii-data
```
Explore available collections.
```
taxii-collections --discovery http://localhost:9000/services/discovery
taxii-collections --discovery http://hailataxii.com/taxii-data
```
Read data from a collection.
```
taxii-poll --discovery http://localhost:9000/services/discovery -c
guest.phishtank_com
taxii-poll --discovery http://hailataxii.com/taxii-data -c
guest.phishtank_com --begin 2016-04-20
```
Manually load data into a collection.
```
taxii-push \
--discovery http://localhost:9000/services/discovery \
--dest phishtank \
--content-file data.xml \
--username guest \
--password guest
```
Fetch data from a remote service and mirror it locally.
```
taxii-proxy --poll-path http://hailataxii.com/taxii-data \
--poll-collection guest.phishtank_com \
--inbox-path
http://localhost:9000/services/guest.phishtank_com-inbox \
--inbox-collection guest.phishtank_com \
--binding urn:stix.mitre.org:xml:1.1.1 \
--inbox-username guest \
--inbox-password guest \
--begin 2016-04-20
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/incubator-metron METRON-115
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/85.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #85
----
commit 656a5972da224a89e2e28790a8330ff5c4c32a3c
Author: Nick Allen <[email protected]>
Date: 2016-04-19T19:45:09Z
METRON-114 Created Ansible role to perform automated deployment of
OpenTAXII services.
commit 92932c787c7b6f7b680af7b17415f1f16ef6a7f1
Author: Nick Allen <[email protected]>
Date: 2016-04-20T19:13:23Z
METRON-115 Automate fetch of 3rd party threat intel data through TAXII
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
