[
https://issues.apache.org/jira/browse/METRON-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15828406#comment-15828406
]
ASF GitHub Bot commented on METRON-664:
---------------------------------------
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/419
Testing Instructions beyond the normal smoke test (i.e. letting data
flow through to the indices and checking them).
## Preliminaries
Since I will use the squid topology to pass data through in a controlled
way, we must install squid and generate one point of data:
* `yum install -y squid`
* `service squid start`
* `squidclient http://www.yahoo.com`
Also, set an environment variable to indicate `METRON_HOME`:
* `export METRON_HOME=/usr/metron/0.3.0`
## Free Up Space on the virtual machine
First, let's free up some headroom on the virtual machine. If you are
running this on a
multinode cluster, you would not have to do this.
* Kill monit via `service monit stop`
* Kill tcpreplay via `for i in $(ps -ef | grep tcpreplay | awk '{print
$2}');do kill -9 $i;done`
* Kill existing parser topologies via
* `storm kill snort`
* `storm kill bro`
* Kill flume via `for i in $(ps -ef | grep flume | awk '{print $2}');do
kill -9 $i;done`
* Kill yaf via `for i in $(ps -ef | grep yaf | awk '{print $2}');do kill -9
$i;done`
* Kill bro via `for i in $(ps -ef | grep bro | awk '{print $2}');do kill -9
$i;done`
## Deploy the squid parser
* Create the squid kafka topic:
`/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper node1:2181
--create --topic squid --partitions 1 --replication-factor 1`
* Start via `$METRON_HOME/bin/start_parser_topology.sh -k node1:6667 -z
node1:2181 -s squid`
### Test Case 0: Base Case Test
* Delete any squid index that currently exists (if any do) via `curl
-XDELETE "http://localhost:9200/squid*"`
* Send 1 data points through and ensure that there are no data points in
the index:
* `cat /var/log/squid/access.log |
/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list
node1:6667 --topic squid`
* `curl "http://localhost:9200/squid*/_search?pretty=true&q=*:*"; 2>
/dev/null| grep "full_hostname" | wc -l` should yield `1`
* Validate that the Storm UI for the indexing topology indicates a warning
in the console for both the "hdfsIndexingBolt" and "indexingBolt" to the effect
of `java.lang.Exception: WARNING: Default and (likely) unoptimized writer
config used for hdfs writer and sensor squid` and `java.lang.Exception:
WARNING: Default and (likely) unoptimized writer config used for elasticsearch
writer and sensor squid` respectively
### Test Case 1: Adjusting batch sizes independently
* Delete any squid index that currently exists (if any do) via `curl
-XDELETE "http://localhost:9200/squid*"`
* Create a file at `$METRON_HOME/config/zookeeper/indexing/squid.json` with
the following contents:
```
{
"hdfs" : {
"index": "squid",
"batchSize": 1,
"enabled" : true
},
"elasticsearch" : {
"index": "squid",
"batchSize": 5,
"enabled" : true
}
}
```
* Push the configs via `$METRON_HOME/bin/zk_load_configs.sh -m PUSH -i
$METRON_HOME/config/zookeeper -z node1:2181`
* Send 4 data points through and ensure:
* `cat /var/log/squid/access.log /var/log/squid/access.log
/var/log/squid/access.log /var/log/squid/access.log |
/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list
node1:6667 --topic squid`
* `curl "http://localhost:9200/squid*/_search?pretty=true&q=*:*"; 2>
/dev/null| grep "full_hostname" | wc -l` should yield `0`
* `hadoop fs -cat /apps/metron/indexing/indexed/squid/enrichment-null* |
wc -l` should yield `4`
* Send a final data point through and ensure:
* `cat /var/log/squid/access.log |
/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list
node1:6667 --topic squid`
* `curl "http://localhost:9200/squid*/_search?pretty=true&q=*:*"; 2>
/dev/null| grep "full_hostname" | wc -l` should yield `5`
* `hadoop fs -cat /apps/metron/indexing/indexed/squid/enrichment-null* |
wc -l` should yield `5`
### Test Case 2: Turn off HDFS writer
* Delete any squid index that currently exists (if any do) via `curl
-XDELETE "http://localhost:9200/squid*"`
* Edit the file at `$METRON_HOME/config/zookeeper/indexing/squid.json` to
the following contents:
```
{
"hdfs" : {
"index": "squid",
"batchSize": 1,
"enabled" : false
},
"elasticsearch" : {
"index": "squid",
"batchSize": 1,
"enabled" : true
}
}
```
* Send 1 data points through and ensure:
* `cat /var/log/squid/access.log |
/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list
node1:6667 --topic squid`
* `curl "http://localhost:9200/squid*/_search?pretty=true&q=*:*"; 2>
/dev/null| grep "full_hostname" | wc -l` should yield `1`
* `hadoop fs -cat /apps/metron/indexing/indexed/squid/enrichment-null* |
wc -l` should yield `0`
### Test Case 3: Stellar Management Functions
* Execute the following in the stellar shell:
```
Stellar, Go!
Please note that functions are loading lazily in the background and will be
unavailable until loaded fully.
{es.clustername=metron, es.ip=node1, es.port=9300,
es.date.format=yyyy.MM.dd.HH}
[Stellar]>>> # Grab the indexing config
[Stellar]>>> squid_config := CONFIG_GET('INDEXING', 'squid', true)
[Stellar]>>>
[Stellar]>>> # Update the index and batch size
[Stellar]>>> squid_config := INDEXING_SET_BATCH(
INDEXING_SET_INDEX(squid_config, 'hdfs', 'squid'), 'hdfs', 2)
[Stellar]>>> # Push the config to zookeeper
[Stellar]>>> CONFIG_PUT('INDEXING', squid_config, 'squid')
[Stellar]>>> # Grab the updated config from zookeeper
[Stellar]>>> CONFIG_GET('INDEXING', 'squid')
{
"hdfs" : {
"index" : "squid",
"batchSize" : 2,
"enabled" : false
},
"elasticsearch" : {
"index" : "squid",
"batchSize" : 1,
"enabled" : true
}
}
```
* Confirm that the dump command from `$METRON_HOME/bin/zk_load_configs.sh
-m DUMP -z node1:2181` contains the config with batch size of `1`
* Now pull the configs locally via `$METRON_HOME/bin/zk_load_configs.sh -m
PULL -z node1:2181 -o $METRON_HOME/config/zookeeper -f`
* Check that the "hdfs" config at
`$METRON_HOME/config/zookeeper/indexing/squid.json` is indeed:
```
{
"index" : "squid",
"batchSize" : 2,
"enabled" : false
}
```
> Make the index configuration per-writer with enabled/disabled
> -------------------------------------------------------------
>
> Key: METRON-664
> URL: https://issues.apache.org/jira/browse/METRON-664
> Project: Metron
> Issue Type: Sub-task
> Reporter: Casey Stella
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
