Hi List,
I'm following this guide:
https://cwiki.apache.org/confluence/display/METRON/Metron+with+HDP+2.5+bare-metal+install
and Maven seems to fail after this:
"cd metron-deployment/packaging/docker/rpm-docker"
"mvn clean install -DskipTests -PHDP-2.5.0.0"
Removing intermediate container 864
I "fixed" it by disabling selinux...
On 2017-05-03 08:33, Laurens Vets wrote:
Hi List,
I'm following this guide:
https://cwiki.apache.org/confluence/display/METRON/Metron+with+HDP+2.5+bare-metal+install
and Maven seems to fail after this:
"cd metron-deployment/packaging/doc
Hi list,
I'm not sure where to post this, but I've got a simple document which
explains installing Metron 0.4.0. I've been trying to install Metron
0.4.0 in 3 VMs the past couple of days and with the help of Ryan, Jon &
Otto succeeded today.
I've got Metron 0.4.0 installed on CentOS 7 with a M
running.
Sent with AquaMail for Android
http://www.aqua-mail.com
On May 8, 2017 3:08:48 PM Laurens Vets wrote:
Hi list,
I'm not sure where to post this, but I've got a simple document which
explains installing Metron 0.4.0. I've been trying to install Metron
0.4.0 in 3 VMs the pa
the markdown editor of your
choice :)
Sent with AquaMail for Android
http://www.aqua-mail.com
On May 11, 2017 8:27:24 AM Laurens Vets wrote:
Hi Earl,
See attached. The document is in Markdown format. There's still a
couple
of things which aren't working as expected like Kiba
It was mostly due to timeout errors. Trying to bring up ec2 instances
and then trying to connect to them that failed...
I'll get the logs for you.
On 2017-05-24 10:29, Nick Allen wrote:
> I think the core problem you're hitting is an issue with the NPM install.
> This seems related to an out
Deploying the standard 10 instance setup works. However, for our current
needs, 10 m4.xlarge instances seem overkill and we want to deploy Metron
on only 5 hosts for now.
I would think that editing
metron/metron-deployment/amazon-ec2/playbook.yml would be enough. I
changed the following:
>
> If you are wanting to run Metron in AWS for any period of time, a better
> approach is to define your VPC, spin up your EC2 hosts, install Ambari, then
> use Metron's MPack to install Metron.
>
> On Thu, May 25, 2017 at 1:00 PM, Laurens Vets wrote:
>
>&g
etc.
- Dima
On 05/12/2017 02:14 AM, Laurens Vets wrote:
On 2017-05-11 12:13, Earl Hinkle wrote:
So can it be used with out kibana working? Is that because of the
mariadb config? Also, the .md extension what app would this be opened
with?
I think that the Kibana stuff is due to my (weird?) test setup
Hello list,
One of the Storm workers dies with the following error message:
2017-06-14 11:17:32.503 o.a.s.util [ERROR] Async loop died!
java.lang.OutOfMemoryError: Java heap space
at org.apache.kafka.common.utils.Utils.toArray(Utils.java:272)
~[stormjar.jar:?]
at org.apache.kafk
Where and how exactly? I don't seem to find that 64MB setting in
Ambari...
On 2017-06-14 22:22, Nick Allen wrote:
> Yes, allocate more memory to your Storm workers.
>
> On Wed, Jun 14, 2017 at 1:37 PM, Laurens Vets wrote:
>
>> Hello list,
>>
>> One
Hi list,
What would be the best way to upgrade from 0.4.0-rc to 0.4.0-release?
Can I just do "rpm -Uvh metron*.rpm" or do I need to do something in
Ambari?
What would be the best way to upgrade from 0.4.0-rc to 0.4.0-release?
Can I just do "rpm -Uvh metron*.rpm" or do I need to do something in
Ambari?
I'll answer myself. I've upgraded the old metron-* rpms (0.4.0-rc) to
the new rpms (0.4.0-release) with "rpm -Uvh metron*.rpm", restarted my
machi
Hi Kxuan,
You mentioned Ubuntu 16. What was the problem there? Creating a full-dev
environment on Ubuntu works for me. Can you try the following on your
Ubuntu machine:
- Install VirtualBox, vagrant, maven & docker.io ("sudo apt-get install
virtualbox vagrant maven docker.io")
- "git clone -b
Before I open a JIRA ticket, does anyone know why starting Metron REST
fails, but manually starting it with "service metron-rest start
" works?
In /var/log/metron/metron-rest.log, I see the following:
. ___ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ |
11, 2017 at 5:13 PM, Nick Allen wrote:
>
> Yes, I have seen this. It is a bug. I believe Ryan submitted a fix in one of
> his open PRs. He can chime in with the exact one.
>
> On Jul 11, 2017 6:08 PM, "Laurens Vets" wrote:
> Before I open a JIRA ticket, does any
Hi list,
I want to enrich AWS Cloudtrail events with an extra field "is_us"
("yes" or "no") which shows whether the source ip address in my events
is from our network or not.
I created the file my_subnets.csv with the following content:
1.2.3.0/24;AS1230;Company1
1.2.4.0/24;AS1240;Company2
T
At the very least, I should get something back for
"ENRICHMENT_GET('COMPANY', OurSubnets, 'enrichment', 't')" in the
Stellar shell right?
On 2017-07-28 13:47, Laurens Vets wrote:
Hi list,
I want to enrich AWS Cloudtrail events with an extra field &qu
Hi list,
I see the following error in my enrichmentJoinBolt Storm UI:
java.lang.Exception: Join cache reached max size limit. Increase the
maxCacheSize setting or add more tasks to enrichment/threatintel join
bolt. at
org.apache.metron.enrichment.bolt.JoinBolt$JoinRemoveListener.onRemoval(Joi
Can you check in /etc/elasticsearch/elasticsearch.yml whether both
node.data and node.master are true? I remember having to set this
manually. Also check "expected_data_nodes" = "0" &
"gateway_recover_after_data_nodes" = "1" in Ambari.
It's part of the guide for CentOS 6, I might not have it copi
Hi Guillem,
Did you eventually fix the problem?
On 2017-08-01 11:00, Guillem Mateos wrote:
> On the elasticsearch.properties file, right now, I have the following
> regarding workers and executors:
>
> # Storm #
> indexing.workers=1
> indexing.executors=1
> topology.worker.childopts=
From the Performance-tuning-guide.md: "You will find the offset lag tool
indispensable while verifying your settings."
Probably because it's Monday, but I can't seem to find this offset lag
tool anywhere...
Hi List,
I'm seeing the following errors in our indexing topology:
kafkaSpout:
java.lang.OutOfMemoryError: GC overhead limit exceeded at
org.apache.kafka.common.utils.Utils.toArray(Utils.java:272) at
org.apache.kafka.common.utils.Utils.toArray(Utils.java:265) at
org.apache.kafka.clients.consu
mory might still be an issue?
On 2017-08-14 09:57, zeo...@gmail.com wrote:
> Try increasing nofile and nproc for your storm service account.
>
> Jon
>
> On Mon, Aug 14, 2017, 12:46 Laurens Vets wrote:
>
>> Hi List,
>>
>> I'm seeing the following
he resources they are consuming. Since you say one
> node is overloaded and one is barely utilized, I would first look at
> redistributing your services so that the load is more balanced. You would
> almost certainly want ES and Storm on different nodes.
>
> Ryan
>
> On
Hi Frank,
No, docker is only needed on the host you're building Metron on.
Kind regards,
Laurens
On 2017-08-17 07:46, Frank Horsfall wrote:
> Hello I am going through the install procedure for 3 nodes at
>
> https://cwiki.apache.org/confluence/display/METRON/Metron+0.4.0+with+HDP+2.5+bar
Hello,
I suddenly receive the following error messages:
java.nio.channels.ClosedChannelException at
org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:1521)
at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:104)
at
org.apache.hadoop.fs.FSDataOutputStream$Po
Hi Frank,
If you all your queues (Kafka/Storm) are empty, the following should
work:
- Deleting your elasticsearch indices: curl -X DELETE
'http://localhost:9200/snort_index_*', curl -X DELETE
'http://localhost:9200/yaf_index_*', etc...
- Deleting your Hadoop data:
Become the hdfs user: sud
he reason I ask is that
>
> A few days ago I shut down yaf, bro, snort, etc. but I'm still processing
> millions of events which I suspect is the backlog of events that have been
> queued for processing.
>
> Kindest
>
> Frank
>
> FROM: Laure
I'm trying out the Alerts UI and it's not working. It seems the default
admin/password doesn't work.
I've installed the UI via
https://github.com/apache/metron/tree/master/metron-interface/metron-alerts#installing-on-an-existing-cluster
but I also made sure that I changed the hostname in
metr
the browser javascript
> console for errors.
>
> Ryan
>
> On Mon, Sep 11, 2017 at 6:55 PM, Laurens Vets wrote:
>
>> I'm trying out the Alerts UI and it's not working. It seems the default
>> admin/password doesn't work.
>>
>> I've i
Hi Syed,
Getting the full-dev environment up & running (in Virtualbox) works on
my Ubuntu 16.04 LTS machine. However, 8 GB RAM might not be enough...
For a bare metal install, 8 GB RAM will be an issue as well. It might
work, but your experience will not be that good.
On 2017-09-15 08:12, Syed
I think these addresses are used in the example.pcap
(/opt/pcap-replay/example.pcap). The fact that you're receiving this
means that pcap-replay is probably running in the background. You can
check this with Monit ("monit summary").
On 2017-09-20 07:29, Frank Horsfall wrote:
> Morning all,
>
>
Hi Frank,
This works for me on CentOS 6 going from master (between 0.4.0 & 0.4.1)
to 0.4.1-release:
- Stop everything. Including ambari-server ("ambari-server stop").
- Build Metron RPMs
- Install/Upgrade the RPMs with "rpm -Uvh metron*.rpm"
- Install/Upgrade mpack with "ambari-server install-
I have the following configuration:
"threatIntel": {
"fieldMap": {},
"fieldToTypeMap": {},
"config": {},
"triageConfig": {
"riskLevelRules": [
{
"name": "Rule1",
"comment": "Checks whatever 1.",
"rule": "test == \"false\"",
"scor
to triage?
Simon
On 25 Sep 2017, at 18:46, Laurens Vets wrote:
I have the following configuration:
"threatIntel": {
"fieldMap": {},
"fieldToTypeMap": {},
"config": {},
"triageConfig": {
"riskLevelRules": [
t indicator, e.g. is it 2x std_devs, or 4x std_devs as
different rule levels. We’re adding the ability to make score a
stellar statement which simplifies this further by allowing score to
be a function, but thresholds are still useful to determine the text
content of the alert for example.
Simon
On 25 S
is_alert := is_alert || something_else.
&& is bitwise and
|| is bitwise or
Simon
On 25 Sep 2017, at 21:12, Laurens Vets wrote:
Thanks! Followup question, the below is_alert 'rules' in the snippet
from
http://apache.website-solution.net/metron/0.4.1/site-book/use-cases/geograp
on on why my rules
might not be working? (Metron UI accepts my JSON without issues)
On 2017-09-25 13:39, Laurens Vets wrote:
Thanks!
On 2017-09-25 13:16, Simon Elliston Ball wrote:
The second statement overwrites the first, but also uses the previous
value.
Technically that is an or. Note this
level field.
>
> Jon
>
> On Mon, Sep 25, 2017, 19:34 Laurens Vets wrote:
>
>> Next problem:
>>
>> I'm setting the "is_alert" field to true. It shows up in Kibana, but I
>> don't get a threat.triage.level field which means that either my
level field.
>
> Jon
>
> On Mon, Sep 25, 2017, 19:34 Laurens Vets wrote:
>
>> Next problem:
>>
>> I'm setting the "is_alert" field to true. It shows up in Kibana, but I
>> don't get a threat.triage.level field which means that eith
've
> got any custom templates there, and make sure you refresh the fields in
> kibana's index config.
>
> Simon
>
> On 26 Sep 2017, at 17:13, Laurens Vets wrote:
>
> After setting is_alert to true, this field is now shown in my event in
> Kibana.
When I installed the Metron Alerts UI on my 0.4.0 install, I could log
in with the Metron Management UI user.
I upgraded to 0.4.1, did the same as mentioned on
https://github.com/apache/metron/tree/apache-metron-0.4.1-release/metron-interface/metron-alerts#installing-on-an-existing-cluster,
bu
you won't
> even have to install it manually (my guess is it makes it in a couple days
> from today).
>
> Ryan
>
> On Wed, Sep 27, 2017 at 4:33 PM, Laurens Vets wrote:
> When I installed the Metron Alerts UI on my 0.4.0 install, I could log in
> with the Me
I mean, I can go to the Swagger UI page on port 8082 and I see an
overview of API actions. But how can actually log on?
On 2017-09-27 14:55, Laurens Vets wrote:
> How can I log into the Swagger UI?
>
> On 2017-09-27 14:38, Ryan Merriman wrote:
> Nevermind it is proxying to metro
roxy isn't setup correctly.
>
> On Wed, Sep 27, 2017 at 5:04 PM, Laurens Vets wrote:
>
> I mean, I can go to the Swagger UI page on port 8082 and I see an overview of
> API actions. But how can actually log on?
>
> On 2017-09-27 14:55, Laurens Vets wrote:
>
> H
Nevermind, I found the issue.
This works:
./bin/start_alerts_ui.sh -p 4201 -r http://:8082 [7]
This doesn't apparently:
./bin/start_alerts_ui.sh -p 4201 -r :8082
On 2017-09-27 15:58, Laurens Vets wrote:
> Once the alerts-ui package has been compiled & installed where exactl
Hello,
I've got the Alerts UI up and running. However, I do not see any alerts.
I can see events in Kibana with "is_alert" set to "true" and with a
score as well, but they do not show up in the Alerts UI.
How and where does the Alerts UI get actual alerts?
7;, 'snort', 'asa', 'bro', 'yaf'. It does not show
records under .kibana as they are not the alerts generated by the
system. Usually the index names for the sensors would have a sensor
name prefix followed by timestamp Ex: snort_index_2017.09.28.18
-Raghu
On
What's the quickest way to enable geo enrichment on a source ip address
in 0.4.1-release? Is there a simple document somewhere with
instructions?
nt-book/metron-platform/metron-enrichment/index.html
> [2]
>
> Shows you how to configure geo enrichment.
>
> Simon
>
> On 5 Oct 2017, at 22:33, Laurens Vets wrote:
>
> What's the quickest way to enable geo enrichment on a source ip address in
> 0.4.1-release?
Hi Youzha,
Either check how the snort logs on the full dev installation are
ingested (I believe it's with a script) or check the Apache NiFi project
which makes it very easy to read logs from almost any format and ingest
them to Metron via Kafka.
On 2017-10-17 08:53, Youzha wrote:
> is it poss
Can you post the full error message?
On 2017-10-20 07:33, kotipalli venkatesh wrote:
> Hi All,
>
> While, we are installing "apache Matron 0.4.1 with hdp 2.5 bare metal on
> CentOS7", we are facing an issue with "Kibana Server Install".
>
> We defined the "Port" for the "kibana_es_url" ins
Hi Syed,
See inline.
On 2017-10-20 00:32, Syed Hammad Tahir wrote:
I have installed the snort manually. Now I need help with :
1- Capturing the data of my lan and dumping it via snort :Snort cant
see the traffic outside vagrant vm, how do I make it see that traffic?
To be honest, configurin
1 thing of the top of my head. You might have to make sure elasticsearch
is configured as master & datanode.
On 2017-10-25 10:13, Syed Hammad Tahir wrote:
> I killed it via terminal and then restarted it. Still the same thing, cant
> load the page when I go to elasticsearch health shortlink in
Hi Farrukh,
How come you don't have the commands 'mkdir', 'chown' & 'chmod' on
node1?
On 2017-12-19 02:42, Farrukh Naveed Anjum wrote:
> Hi,
> I am trying to install the Metron 0.4.0 ( Cent OS 6) following error is
> coming up
>
> ==
> Creating target directory...
>
and NPM which resulted in removal
> of clear command. Any suggestion how can get it fixed up.
>
> On the contrary, when i use mkdir, chown and chmod on node1 it is working.
>
> On Tue, Dec 19, 2017 at 9:50 PM, Laurens Vets wrote:
>
> Hi Farrukh,
>
> How come you
user ?
>
> On Tue, Dec 19, 2017 at 9:58 PM, Laurens Vets wrote:
>
> Something strange is going on...
>
> How are you doing the install exactly? Everything manual? Or with an Ansible
> playbook?
>
> Is it possible you created a user which is not allowed to run
Hi Gaurav,
If you click on the red squares in the upper right corners of your
processors, what error messages do you see?
On 2018-01-14 19:29, Gaurav Bapat wrote:
> Hey Jon,
>
> I have Storm UI and the logs are coming from firewalls, servers, etc from
> other machines(HP ArcSight Logger).
>
Hello List,
Targeting a wider audience here, see bug report
https://issues.apache.org/jira/browse/METRON-1408.
Basically, when I upgrade from 0.4.1 to 0.4.2 I run into issues with the
Alerts UI. I built the Metron 0.4.2 RPMs and did an upgrade of my
current 0.4.1 install with: "rpm -Uvh metr
Hi list,
I have some general Alerts UI questions/comments/remarks, I hope you
don't mind :) I'm using the UI that's part of Metron 0.4.2. These apply
to my specific use case, so I might be completely wrong in how I use the
UI...
- When you're talking about 'alerts', from what I can see in th
I hope there's an elasticsearch expert on the mailing list :D
I have a field called "responseElements:subnets" which can either
contain:
{
"subnetIdentifier": "subnet-abcdefgh",
"subnetStatus": "Active",
"subnetAvailabilityZone": {
"name": "us-west-2c"
}
},
{
"subnetIdentifier":
lucene based
indices like Elastic and Solr. So, we never used, nor need nested
template, and tend to just use the ‘:’ separated fields to define the
hierarchy.
Is there a particular use case you need the nesting for?
Simon
On 7 Feb 2018, at 01:26, Laurens Vets wrote:
I hope there's an elast
I'm not sure I understand the question completely, but my guess would be
the latest release, i.e. 0.4.2?
On 2018-02-15 10:19, Helder Reia wrote:
> Hi, I am trying to build a intrusion detection system and I was thinking on
> using Apache Metron, but I have a question: which is the best version
Hi Farrukh,
I can only confirm that ES 5.6 works with Metron 0.6.0 as that's what
I'm currently using.
Hopefully someone else on the list can confirm whether ES 6 works...
Kind regards,
Laurens
On 2018-10-05 10:05, Farrukh Naveed Anjum wrote:
> I am trying to upgrade to 0.6, hope it supp
I would suggest to try with a newer version (0.6.0), 0.3.1 is very old.
On 2018-11-29 6:20 p.m., Babak Abbaschian wrote:
Followed this link:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68718548
With the following info:
***
66 matches
Mail list logo
