Hi Laurens, Does this exercise have to be executed on all 3 nodes?
You also mention that if the queues are empty this should work. Is there a way of clearing the queues? The reason I ask is that A few days ago I shut down yaf, bro, snort, etc. but I’m still processing millions of events which I suspect is the backlog of events that have been queued for processing. Kindest Frank From: Laurens Vets [mailto:[email protected]] Sent: Wednesday, September 6, 2017 6:17 PM To: [email protected] Cc: Frank Horsfall <[email protected]> Subject: Re: Clearing of data to start over Hi Frank, If you all your queues (Kafka/Storm) are empty, the following should work: - Deleting your elasticsearch indices: curl -X DELETE 'http://localhost:9200/snort_index_*', curl -X DELETE 'http://localhost:9200/yaf_index_*', etc... - Deleting your Hadoop data: Become the hdfs user: sudo su - hdfs Show what's been indexed in Hadoop: hdfs dfs -ls /apps/metron/indexing/indexed/ Output should show the following probably: /apps/metron/indexing/indexed/error /apps/metron/indexing/indexed/snort /apps/metron/indexing/indexed/yaf ... You can remove these with: hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/error/ hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/snort/ Or the individial files with hdfs dfs -rmr -skipTrash /apps/metron/indexing/indexed/error/FILENAME On 2017-09-06 13:59, Frank Horsfall wrote: Hello all, I have installed a 3 node system using the bare metal Centos 7 guideline. https://cwiki.apache.org/confluence/display/METRON/Metron+0.4.0+with+HDP+2.5+bare-metal+install+on+Centos+7+with+MariaDB+for+Metron+REST It has taken me a while to have all components working properly and I left the yaf,bro,snort apps running so quite a lot of data has been generated. Currently, I have almost 18 million events identified in Kibana. 16+ million are yaf based, and 2+ million are snort …. 190 events are my new squid telemetry, ☺. It looks like it still has a while to go before it catches up to current day. I recently shutdown the apps. My questions are: 1. Is there a way to wipe all my data and indices clean so that I may now begin with a fresh dataset? 2. Is there a way to configure yaf so that its data is meaningful ? It is currently creating what looks to be test data? 3. I have commented out the test snort rule but it is still generating the odd record which looks once again looks like test data. Can this be stopped as well? Kindest regards, Frank
