Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/415
Testing Instructions beyond the normal smoke test (i.e. letting data
flow through to the indices and checking them).
## Preliminaries
Since I will use the squid topology to pass data through in a controlled
way, we must install squid and generate one point of data:
* `yum install -y squid`
* `service squid start`
* `squidclient http://www.yahoo.com`
Also, set an environment variable to indicate `METRON_HOME`:
* `export METRON_HOME=/usr/metron/0.3.0`
## Free Up Space on the virtual machine
First, let's free up some headroom on the virtual machine. If you are
running this on a
multinode cluster, you would not have to do this.
* Kill monit via `service monit stop`
* Kill tcpreplay via `for i in $(ps -ef | grep tcpreplay | awk '{print
$2}');do kill -9 $i;done`
* Kill existing parser topologies via
* `storm kill snort`
* `storm kill bro`
* Kill flume via `for i in $(ps -ef | grep flume | awk '{print $2}');do
kill -9 $i;done`
* Kill yaf via `for i in $(ps -ef | grep yaf | awk '{print $2}');do kill -9
$i;done`
* Kill bro via `for i in $(ps -ef | grep bro | awk '{print $2}');do kill -9
$i;done`
## Deploy the squid parser
* Create the squid kafka topic:
`/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper node1:2181
--create --topic squid --partitions 1 --replication-factor 1`
* Start via `$METRON_HOME/bin/start_parser_topology.sh -k node1:6667 -z
node1:2181 -s squid`
### Test Case 1: Adjusting batch sizes
* Delete any squid index that currently exists (if any do).
* Create a file at `$METRON_HOME/config/zookeeper/indexing/squid.json` with
the following contents:
```
{
"index" : "squid",
"batchSize" : 5
}
```
* Send 4 data points through and ensure that there are no data points in
the index:
* `cat /var/log/squid/access.log /var/log/squid/access.log
/var/log/squid/access.log /var/log/squid/access.log |
/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list
node1:6667 --topic squid`
* `curl "http://localhost:9200/squid*/_search?pretty=true&q=*:*"; 2>
/dev/null| grep "full_hostname" | wc -l` should yield `0`
* Send a final data point through and ensure that we have 5 data points:
* `cat /var/log/squid/access.log |
/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list
node1:6667 --topic squid`
* `curl "http://localhost:9200/squid*/_search?pretty=true&q=*:*"; 2>
/dev/null| grep "full_hostname" | wc -l` should yield `5`
### Test Case 2: Update configs from the CLI
* Edit the file at `$METRON_HOME/config/zookeeper/indexing/squid.json` to
the following contents:
```
{
"index" : "squid",
"batchSize" : 10
}
```
* Push the configs: `$METRON_HOME/bin/zk_load_configs.sh -m PUSH -i
$METRON_HOME/config/zookeeper -z node1:2181`
* Dump the configs and verify the squid indexing config is correct:
`$METRON_HOME/bin/zk_load_configs.sh -m DUMP -z node1:2181`
### Test Case 3: Stellar Management Functions
* Execute the following in the stellar shell:
```
Stellar, Go!
Please note that functions are loading lazily in the background and will
be unavailable until loaded fully.
{es.clustername=metron, es.ip=node1, es.port=9300,
es.date.format=yyyy.MM.dd.HH}
[Stellar]>>> # Grab the indexing config
[Stellar]>>> squid_config := CONFIG_GET('INDEXING', 'squid', true)
Functions loaded, you may refer to functions now...
[Stellar]>>> # Update the index and batch size
[Stellar]>>> squid_config := INDEXING_SET_BATCH(
INDEXING_SET_INDEX(squid_config, 'squid'), 1)
[Stellar]>>> # Push the config to zookeeper
[Stellar]>>> CONFIG_PUT('INDEXING', squid_config, 'squid')
[Stellar]>>> # Grab the updated config from zookeeper
[Stellar]>>> CONFIG_GET('INDEXING', 'squid')
{
"index" : "squid",
"batchSize" : 1
}
```
* Confirm that the dump command from `$METRON_HOME/bin/zk_load_configs.sh
-m DUMP -z node1:2181` contains the config with batch size of `1`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
