Re: Error deploying Metron 0.3.1 single Node

2018-11-29 Thread Laurens Vets
I would suggest to try with a newer version (0.6.0), 0.3.1 is very old. On 2018-11-29 6:20 p.m., Babak Abbaschian wrote: Followed this link: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68718548 With the following info:

Re: Upgrading to Elasticsearch 5.6

2018-10-05 Thread Laurens Vets
Hi Farrukh, I can only confirm that ES 5.6 works with Metron 0.6.0 as that's what I'm currently using. Hopefully someone else on the list can confirm whether ES 6 works... Kind regards, Laurens On 2018-10-05 10:05, Farrukh Naveed Anjum wrote: > I am trying to upgrade to 0.6, hope it

Re: Best Metron version for development

2018-02-15 Thread Laurens Vets
I'm not sure I understand the question completely, but my guess would be the latest release, i.e. 0.4.2? On 2018-02-15 10:19, Helder Reia wrote: > Hi, I am trying to build a intrusion detection system and I was thinking on > using Apache Metron, but I have a question: which is the best version

Re: elasticsearch template question.

2018-02-07 Thread Laurens Vets
Elastic and Solr. So, we never used, nor need nested template, and tend to just use the ‘:’ separated fields to define the hierarchy. Is there a particular use case you need the nesting for? Simon On 7 Feb 2018, at 01:26, Laurens Vets <laur...@daemon.be> wrote: I hope there's an elastic

elasticsearch template question.

2018-02-06 Thread Laurens Vets
I hope there's an elasticsearch expert on the mailing list :D I have a field called "responseElements:subnets" which can either contain: { "subnetIdentifier": "subnet-abcdefgh", "subnetStatus": "Active", "subnetAvailabilityZone": { "name": "us-west-2c" } }, { "subnetIdentifier":

Some Metron Alerts UI questions

2018-01-19 Thread Laurens Vets
Hi list, I have some general Alerts UI questions/comments/remarks, I hope you don't mind :) I'm using the UI that's part of Metron 0.4.2. These apply to my specific use case, so I might be completely wrong in how I use the UI... - When you're talking about 'alerts', from what I can see in

Upgrade from 0.4.1 to 0.4.2 fails on Alerts UI

2018-01-17 Thread Laurens Vets
Hello List, Targeting a wider audience here, see bug report https://issues.apache.org/jira/browse/METRON-1408. Basically, when I upgrade from 0.4.1 to 0.4.2 I run into issues with the Alerts UI. I built the Metron 0.4.2 RPMs and did an upgrade of my current 0.4.1 install with: "rpm -Uvh

Re: Metron 0.4.0 on CentOS (Ambari Host Regisration Problem)

2017-12-19 Thread Laurens Vets
ission. Do ambari create its own user ? > > On Tue, Dec 19, 2017 at 9:58 PM, Laurens Vets <laur...@daemon.be> wrote: > > Something strange is going on... > > How are you doing the install exactly? Everything manual? Or with an Ansible > playbook? > > Is it possible

Re: Metron 0.4.0 on CentOS (Ambari Host Regisration Problem)

2017-12-19 Thread Laurens Vets
Hi Farrukh, How come you don't have the commands 'mkdir', 'chown' & 'chmod' on node1? On 2017-12-19 02:42, Farrukh Naveed Anjum wrote: > Hi, > I am trying to install the Metron 0.4.0 ( Cent OS 6) following error is > coming up > > == > Creating target directory...

Re: Kibana Error

2017-10-25 Thread Laurens Vets
1 thing of the top of my head. You might have to make sure elasticsearch is configured as master & datanode. On 2017-10-25 10:13, Syed Hammad Tahir wrote: > I killed it via terminal and then restarted it. Still the same thing, cant > load the page when I go to elasticsearch health shortlink in

Re: event correlation on metron

2017-10-17 Thread Laurens Vets
Hi Youzha, Either check how the snort logs on the full dev installation are ingested (I believe it's with a script) or check the Apache NiFi project which makes it very easy to read logs from almost any format and ingest them to Metron via Kafka. On 2017-10-17 08:53, Youzha wrote: > is it

Re: Enable geo enrichment

2017-10-16 Thread Laurens Vets
ote: > > http://metron.apache.org/current-book/metron-platform/metron-enrichment/index.html > [2] > > Shows you how to configure geo enrichment. > > Simon > > On 5 Oct 2017, at 22:33, Laurens Vets <laur...@daemon.be> wrote: > > What's t

Enable geo enrichment

2017-10-05 Thread Laurens Vets
What's the quickest way to enable geo enrichment on a source ip address in 0.4.1-release? Is there a simple document somewhere with instructions?

Re: Metron Alerts UI not working (anymore)

2017-09-27 Thread Laurens Vets
I mean, I can go to the Swagger UI page on port 8082 and I see an overview of API actions. But how can actually log on? On 2017-09-27 14:55, Laurens Vets wrote: > How can I log into the Swagger UI? > > On 2017-09-27 14:38, Ryan Merriman wrote: > Nevermind it is proxying to me

Re: Not seeing any Metron alerts.

2017-09-26 Thread Laurens Vets
d. > > Jon > > On Mon, Sep 25, 2017, 19:34 Laurens Vets <laur...@daemon.be> wrote: > >> Next problem: >> >> I'm setting the "is_alert" field to true. It shows up in Kibana, but I >> don't get a threat.triage.level field which means t

Re: 192.168.138.158 address in yaf index

2017-09-20 Thread Laurens Vets
I think these addresses are used in the example.pcap (/opt/pcap-replay/example.pcap). The fact that you're receiving this means that pcap-replay is probably running in the background. You can check this with Monit ("monit summary"). On 2017-09-20 07:29, Frank Horsfall wrote: > Morning all, >

Re: Metron Installation

2017-09-15 Thread Laurens Vets
Hi Syed, Getting the full-dev environment up & running (in Virtualbox) works on my Ubuntu 16.04 LTS machine. However, 8 GB RAM might not be enough... For a bare metal install, 8 GB RAM will be an issue as well. It might work, but your experience will not be that good. On 2017-09-15 08:12, Syed

Re: Alerts UI password not working

2017-09-12 Thread Laurens Vets
browser javascript > console for errors. > > Ryan > > On Mon, Sep 11, 2017 at 6:55 PM, Laurens Vets <laur...@daemon.be> wrote: > >> I'm trying out the Alerts UI and it's not working. It seems the default >> admin/password doesn't work. >> >> I've ins

Re: Clearing of data to start over

2017-09-06 Thread Laurens Vets
Hi Frank, If you all your queues (Kafka/Storm) are empty, the following should work: - Deleting your elasticsearch indices: curl -X DELETE 'http://localhost:9200/snort_index_*', curl -X DELETE 'http://localhost:9200/yaf_index_*', etc... - Deleting your Hadoop data: Become the hdfs user:

hdfsIndexingBolt error messages

2017-08-17 Thread Laurens Vets
Hello, I suddenly receive the following error messages: java.nio.channels.ClosedChannelException at org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:1521) at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:104) at

Re: New install question

2017-08-17 Thread Laurens Vets
Hi Frank, No, docker is only needed on the host you're building Metron on. Kind regards, Laurens On 2017-08-17 07:46, Frank Horsfall wrote: > Hello I am going through the install procedure for 3 nodes at > >

Re: Storm indexing topology possible memory issues

2017-08-14 Thread Laurens Vets
are consuming. Since you say one > node is overloaded and one is barely utilized, I would first look at > redistributing your services so that the load is more balanced. You would > almost certainly want ES and Storm on different nodes. > > Ryan > > On Mon, Aug 14, 2017 a

Storm indexing topology possible memory issues

2017-08-14 Thread Laurens Vets
Hi List, I'm seeing the following errors in our indexing topology: kafkaSpout: java.lang.OutOfMemoryError: GC overhead limit exceeded at org.apache.kafka.common.utils.Utils.toArray(Utils.java:272) at org.apache.kafka.common.utils.Utils.toArray(Utils.java:265) at

Offset lag tool?

2017-08-14 Thread Laurens Vets
From the Performance-tuning-guide.md: "You will find the offset lag tool indispensable while verifying your settings." Probably because it's Monday, but I can't seem to find this offset lag tool anywhere...

Re: Issues with indexing topology

2017-08-13 Thread Laurens Vets
Hi Guillem, Did you eventually fix the problem? On 2017-08-01 11:00, Guillem Mateos wrote: > On the elasticsearch.properties file, right now, I have the following > regarding workers and executors: > > # Storm # > indexing.workers=1 > indexing.executors=1 >

Re: Problem with metron reference App

2017-08-10 Thread Laurens Vets
Can you check in /etc/elasticsearch/elasticsearch.yml whether both node.data and node.master are true? I remember having to set this manually. Also check "expected_data_nodes" = "0" & "gateway_recover_after_data_nodes" = "1" in Ambari. It's part of the guide for CentOS 6, I might not have it

Storm enrichmentJoinBolt Join cache reached max size limit.

2017-08-04 Thread Laurens Vets
Hi list, I see the following error in my enrichmentJoinBolt Storm UI: java.lang.Exception: Join cache reached max size limit. Increase the maxCacheSize setting or add more tasks to enrichment/threatintel join bolt. at

Re: Adding custom enrichment.

2017-07-31 Thread Laurens Vets
At the very least, I should get something back for "ENRICHMENT_GET('COMPANY', OurSubnets, 'enrichment', 't')" in the Stellar shell right? On 2017-07-28 13:47, Laurens Vets wrote: Hi list, I want to enrich AWS Cloudtrail events with an extra field "is_us" ("yes"

Adding custom enrichment.

2017-07-28 Thread Laurens Vets
Hi list, I want to enrich AWS Cloudtrail events with an extra field "is_us" ("yes" or "no") which shows whether the source ip address in my events is from our network or not. I created the file my_subnets.csv with the following content: 1.2.3.0/24;AS1230;Company1 1.2.4.0/24;AS1240;Company2

Re: Upgrade from 0.4.0-rc to 0.4.0-release

2017-07-06 Thread Laurens Vets
What would be the best way to upgrade from 0.4.0-rc to 0.4.0-release? Can I just do "rpm -Uvh metron*.rpm" or do I need to do something in Ambari? I'll answer myself. I've upgraded the old metron-* rpms (0.4.0-rc) to the new rpms (0.4.0-release) with "rpm -Uvh metron*.rpm", restarted my

Storm indexing out of memory

2017-06-14 Thread Laurens Vets
Hello list, One of the Storm workers dies with the following error message: 2017-06-14 11:17:32.503 o.a.s.util [ERROR] Async loop died! java.lang.OutOfMemoryError: Java heap space at org.apache.kafka.common.utils.Utils.toArray(Utils.java:272) ~[stormjar.jar:?] at

AWS deployment with 5 hosts.

2017-05-25 Thread Laurens Vets
Deploying the standard 10 instance setup works. However, for our current needs, 10 m4.xlarge instances seem overkill and we want to deploy Metron on only 5 hosts for now. I would think that editing metron/metron-deployment/amazon-ec2/playbook.yml would be enough. I changed the following:

Install Metron 0.4.0 on CentOS 7 with MySQL (MariaDB) for Metron REST.

2017-05-08 Thread Laurens Vets
Hi list, I'm not sure where to post this, but I've got a simple document which explains installing Metron 0.4.0. I've been trying to install Metron 0.4.0 in 3 VMs the past couple of days and with the help of Ryan, Jon & Otto succeeded today. I've got Metron 0.4.0 installed on CentOS 7 with a

Re: Unable to build Metron, stuck at rpm-docker

2017-05-03 Thread Laurens Vets
I "fixed" it by disabling selinux... On 2017-05-03 08:33, Laurens Vets wrote: Hi List, I'm following this guide: https://cwiki.apache.org/confluence/display/METRON/Metron+with+HDP+2.5+bare-metal+install and Maven seems to fail after this: "cd metron-deployment/packaging/docker/r

Unable to build Metron, stuck at rpm-docker

2017-05-03 Thread Laurens Vets
Hi List, I'm following this guide: https://cwiki.apache.org/confluence/display/METRON/Metron+with+HDP+2.5+bare-metal+install and Maven seems to fail after this: "cd metron-deployment/packaging/docker/rpm-docker" "mvn clean install -DskipTests -PHDP-2.5.0.0" Removing intermediate container