GitHub user nickwallen opened a pull request:
https://github.com/apache/incubator-metron/pull/334
METRON-529 Allow Configurable Periods for Profiler Client
## [METRON-529](https://issues.apache.org/jira/browse/METRON-529)
By default, the Profiler creates Profiles with a period duration of 15
minutes. This means that data is accumulated, summarized and flushed every 15
minutes. The Client API must also have knowledge of this duration to correctly
retrieve the profile data. If the client API is expecting 15 minute periods, it
will not be able to read data generated by a Profiler that has been configured
with a 1 hour period.
The period duration can be configured in the Profiler by altering the
Profiler topology's static properties file. The Stellar Client API currently
provides no means to configure the period duration and defaults also to 15
minutes. This means that the Stellar Client API can only read profiles with a
period duration of 15 minutes.
This PR addresses this limitation so that the Profiler Client can be
configured to read profile data no matter what configuration settings were used
to generate that data.
### Testing
I tested this change in the "Quick Dev" environment. I used the Profiler
to create a Profile using a 1 minute period. I then confirmed that I was
unable to read the data using the Profiler client with the default client
configuration. I then changed the client configuration to a 1 minute duration.
I was then able to read the profile data as expected.
#### Steps
1. Ensure the Snort sensor and the Snort parser are running. Stop all
other topologies and sensors.
2. Edit `/usr/metron/0.2.1BETA/config/profiler.properties` and change the
period duration to 1 minute. Also, point the Profiler at the `enrichments`
topic to read data directly from the Snort Parser.
```
profiler.input.topic=enrichments
profiler.period.duration=1
profiler.period.duration.units=MINUTES
```
3. Follow the Profiler's [Getting
Started](https://github.com/apache/incubator-metron/tree/master/metron-analytics/metron-profiler#getting-started)
instructions to start generating profile data.
4. Configure the client to read data with a 1 minute period. To do so,
edit `/usr/metron/0.2.1BETA/config/zookeeper/global.properties`.
```
{
"es.clustername": "metron",
"es.ip": "node1",
"es.port": "9300",
"es.date.format": "yyyy.MM.dd.HH",
"profiler.client.period.duration": "1",
"profiler.client.period.duration.units": "MINUTES"
}
```
5. Push the previous global configuration change to Zookeeper.
```
bin/zk_load_configs.sh -m PUSH -i config/zookeeper/ -z node1:2181
```
6. Open up the Stellar shell and use the Profile client to read the data.
```
[root@node1 0.2.1BETA]# bin/stellar -z node1:2181
Stellar, Go!
Please note that functions are loading lazily in the background and will
be unavailable until loaded fully.
{es.clustername=metron, es.ip=node1, es.port=9300,
es.date.format=yyyy.MM.dd.HH, profiler.client.hbase.table=profiler,
profiler.client.column.family=P,
profiler.client.hbase.table.provider=org.apache.metron.hbase.HTableProvider,
profiler.client.period.duration=1,
profiler.client.period.duration.units=MINUTES,
profiler.client.salt.divisor=1000}
[Stellar]>>> PROFILE_GET('test','192.168.138.158', 2, 'DAYS')
Functions loaded, you may refer to functions now...
[161.0]
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/incubator-metron METRON-529
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/334.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #334
----
commit a902c2a7a6f11ea5131fc84be39b1f8b4913dece
Author: Nick Allen <[email protected]>
Date: 2016-11-01T13:42:02Z
METRON-529 Allow Configurable Periods for Profiler Client
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
