Re: Facing error whith mvn clean install

I highly suspect the fact that you're trying to compile Metron on Windows is the problem :) The only supported OS at the moment is CentOS 6 I believe. On 2017-07-27 19:07, bharath phatak wrote: Hi Michael, I was able pass on with the above issue.Its resolved. I am facing issue with

Re: [DISCUSS] Relocate Docker

On 2017-07-13 09:04, Nick Allen wrote: Having metron-docker at the top-level of the project seems to catch the attention of new users. Some then start using metron-docker to explore/try-out/demo Metron. The metron-docker code that we have is not well-suited for this purpose. It is only really

Upgrade from 0.4.0-rc to 0.4.0-release

Can someone confirm that the following instructions are correct for upgrading from 0.4.0-rc to 0.4.0-release? They seem to work for me, but my testing is rather limited. # cd metron # git checkout Metron_0.4.0 # mvn clean package -DskipTests -T 2C -P HDP-2.5.0.0,mpack # cd

[REQUEST] Contributor rights in Jira

Hello, Could a PMC member please grant my Jira account contributor rights? I'd like to start helping out with various smaller tasks. I promise I won't mess stuff up and go to IRC first for any questions/comments/additions. Otto has been extremely helpful there already :) Thanks, Laurens

Re: [Question] Stopping Storm, Metron & Kafka doesn't stop all Storm processes?

t 12:49 PM Laurens Vets <laur...@daemon.be> wrote: Hi, Simple question, when I stop Metron, Kafka & Storm via Ambari, I still see the storm worker processes running, is this expected?

[Question] Stopping Storm, Metron & Kafka doesn't stop all Storm processes?

Hi, Simple question, when I stop Metron, Kafka & Storm via Ambari, I still see the storm worker processes running, is this expected?

Re: [Question] Stopping Storm, Metron & Kafka doesn't stop all Storm processes?

with. It would be helpful to have more metrics about resource utilization over time in your cluster, are you able to gather anything and maybe put it into a visualization tool? Jon On Thu, Aug 17, 2017, 16:35 Laurens Vets <laur...@daemon.be> wrote: That seems close to the issue that I'm

Re: [Question] Stopping Storm, Metron & Kafka doesn't stop all Storm processes?

played with. > > It would be helpful to have more metrics about resource utilization over time > in your cluster, are you able to gather anything and maybe put it into a > visualization tool? > > Jon > > On Thu, Aug 17, 2017, 16:35 Laurens Vets <laur...@daemon.be> wrote:

Unable to build Metron, stuck at rpm-docker

Hi List, I'm following this guide: https://cwiki.apache.org/confluence/display/METRON/Metron+with+HDP+2.5+bare-metal+install and Maven seems to fail after this: "cd metron-deployment/packaging/docker/rpm-docker" "mvn clean install -DskipTests -PHDP-2.5.0.0" Removing intermediate container

Re: Elasticsearch: network.publish_host needed it seems

in the Elasticsearch config and leave network_publish_host empty. -D... On Thu, May 11, 2017 at 2:51 PM, Laurens Vets <laur...@daemon.be> wrote: Environment: - 2 VMs, each with 2 ip addresses (interfaces enp0s3 & enp0s8) called node1 and node3 - ES master on node1, data node on nod

Elasticsearch: network.publish_host needed it seems

Environment: - 2 VMs, each with 2 ip addresses (interfaces enp0s3 & enp0s8) called node1 and node3 - ES master on node1, data node on node3 - CentOS 7 For some reason, elasticsearch uses the ip attached to enp0s3 as it's publish address. Due to the way my test environment is set up, this will

Re: Trying to spin up Metron in EC2: Failed

atingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162) ... 63 more Any ideas on what might be going or why the timeout

Re: Trying to spin up Metron in EC2: Failed

on the Metron host? On Wed, May 17, 2017 at 1:36 PM, Laurens Vets <laur...@daemon.be> wrote: For testing purposes, I decided to spin up the default Metron AWS config. This resulted in a hang from ansible here: TASK [librdkafka : include] ** ** tas

Re: Trying to spin up Metron in EC2: Failed

Also, it's only 1 out of 10 hosts that failing it seems. On 2017-05-17 13:25, Laurens Vets wrote: When you say Metron host, do you mean the host I'm pushing from to ec2 or the master ambari server in ec2? On 2017-05-17 11:51, Ryan Merriman wrote: That happens when you don't have the zookeeper

Re: AWS single_node_vm

,zeppelin - include: tasks/create-hosts.yml host_count=2 host_type=ambari_slave,search,ec2 - include: tasks/create-hosts.yml host_count=1 host_type=ambari_slave,web,ec2 to this (6-node) tasks: - include: tasks/create-keypair.yml - include: tasks/create-vpc.yml - include:

Re: AWS single_node_vm

de: tasks/create-hosts.yml host_count=1 host_type=sensors,ambari_ master,ec2,monit - include: tasks/create-hosts.yml host_count=1 host_type=pcap_server,monit,ec2 - include: tasks/create-hosts.yml host_count=1 host_type=ambari_slave, enrichment,metron,ec2,zeppelin - include: task

Re: Trying to spin up Metron in EC2: Failed

could try standing up a small node in AWS and then use that to run the Metron deployment from. That always works much more smoothly. On Wed, May 17, 2017 at 4:41 PM, Laurens Vets <laur...@daemon.be> wrote: Also, it's only 1 out of 10 hosts that failing it seems. On 2017-05-17

Re: AWS single_node_vm

telling you that the blueprint that Ansible submitted is invalid. Likely a result of a required hostgroup not being defined. Can you (re?) post what you've put in amazon-ec2/playbook.yml? -D... On Mon, May 29, 2017 at 12:02 PM, Laurens Vets <laur...@daemon.be> wrote: Same problem

Re: AWS single_node_vm

(ES Master and Kibana) since there were fewer hosts. -D... On Tue, May 30, 2017 at 12:49 PM, Laurens Vets <laur...@daemon.be> wrote: I put in what you said previously :) - include: tasks/create-hosts.yml host_count=1 host_type=sensors,ambari_master,ec2,monit - include: tasks/

Re: [Discussion] About the wiki….

On 2017-06-13 14:09, Otto Fowler wrote: I think there are things in the wiki that are very very out of date, to the extent that they are confusing people looking at Metron. Basically anyone going to DOCS HOME from the site is being thrown into documentation that is sure to confuse them. Does

Re: Metron with HDP 2.5 bare-metal install fails

need to use Ambari 2.4.2+. Here's the link for 2.4.2: http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.4.2.0/ambari.repo -O /etc/yum.repos.d/ambari.repo -D... On Thu, May 4, 2017 at 6:16 PM, Laurens Vets <laur...@daemon.be> wrote: I'm installing Metron in 3 VMs following

Metron REST not starting

Hello list, I've installed Metron via Ambari and everything works except the Metron REST interface. It fails with the following error message: "Cannot load driver class: com.mysql.jdbc.Driver". I got the MySQL configuration items from here:

[BUG] Ambari-server login loop

Hi list, I have a strange issue suddenly... Ambari lets me log in, but immediately logs me out again with the following messages in /var/log/ambari-server/ambari-server.log: 11 Sep 2017 23:09:34,790 INFO [ambari-client-thread-78] AbstractProviderModule:424 - Metrics Collector Host or host

Re: Elasticsearch masters_also_are_datanodes doesn't work.

it seems to > be an Ambari or install issue rather than an ES issue per se. > > Thanks, > > --Matt > > FROM: Michael Miklavcic <michael.miklav...@gmail.com> > DATE: Wednesday, September 13, 2017 at 2:08 PM > TO: Matt Foley <mfo...@hortonworks.com> >

Re: Elasticsearch masters_also_are_datanodes doesn't work.

ield? Are you perhaps not logged into Ambari with an ambari user id that > has privs to change this config? Did you use non-default user configuration > with Ambari, and if so are you running with an admin-priv ambari account? > > FROM: Laurens Vets <laur...@daemon.be> > D

Re: SUM aggregator not working?

because ES doesn't handle those .'s well. Hey, maybe ES 5 is more sane about that sort of thing and we can avoid doing that transformation. Casey On Wed, Oct 4, 2017 at 4:38 PM, Laurens Vets <laur...@daemon.be> wrote: No idea whether it's a bug yet, I just need a 2nd set of eyes :) This

Re: Cloudtrail use case

Yes, that's what I meant :) I sent my mail too soon. On 2017-10-05 15:48, Nick Allen wrote: If you mean that you would be willing to do the work, then yes absolutely! I think that would be great. :) On Thu, Oct 5, 2017 at 6:45 PM, Laurens Vets <laur...@daemon.be> wrote: Hi, Would

Cloudtrail use case

Hi, Would anyone be interested in adding a full AWS Cloudtrail use case to the Metron documentation? I would roughly consist of: - Apache NiFi configuration to retrieve Cloudtrail logs from S3 and send it to Metron via Kafka. - Complete Metron sensor configuration (enrichment, alerting,

Upgrade from Metron 0.4.1-rc to 0.4.1-rc4 problems.

Hello, After upgrading from 0.4.1-rc (from last week) to rc4, both Metron Management UI and Metron REST fail to start with an error related to "METRON_ESCALATION_TOPIC="{{metron_escalation_topic}}"". Does anyone know what might be going on here? Metron Management UI Start output: Traceback

Re: Ambari Metrics Collector failing...

, 2017, 18:22 Laurens Vets <laur...@daemon.be> wrote: In preparation of 0.4.1-rc, I'm trying to install the current github master and I'm running into an issue with Ambari-metrics-collector. "Metrics Collector" seems to start, but immediately turns red again Ambari and stops. Any

Metron master installation

Hi list, In preparation of 0.4.1-rc, I'm trying to install the current github master and I'm running into an issue with Ambari-metrics-collector. "Metrics Collector" seems to start, but immediately turns red again Ambari. Any idea what might be going on or where I can start troubleshooting

Ambari Metrics Collector failing...

In preparation of 0.4.1-rc, I'm trying to install the current github master and I'm running into an issue with Ambari-metrics-collector. "Metrics Collector" seems to start, but immediately turns red again Ambari and stops. Any idea what might be going on or where I can start troubleshooting

Re: Elasticsearch masters_also_are_datanodes doesn't work.

On Tue, Sep 12, 2017 at 4:00 PM, Michael Miklavcic > <michael.miklav...@gmail.com> wrote: > > I think this is our default setup for full dev. It's only a 1-node VM, so I'm > pretty sure that it would not work otherwise. I'm spinning up full dev now > and will look into it al

Re: Upgrade from Metron 0.4.1-rc to 0.4.1-rc4 problems.

ea if this will work and, if not, what we should direct people to do who are migrating? On Sat, Sep 9, 2017 at 17:22 Laurens Vets <laur...@daemon.be> wrote: Hello, After upgrading from 0.4.1-rc (from last week) to rc4, both Metron Management UI and Metron REST fail to start with an error related t

Elasticsearch masters_also_are_datanodes doesn't work.

Another issue I noticed. Setting "masters_also_are_datanodes" in Ambari to "true" does not work. The settings in /etc/elasticsearch/elasticsearch.yml will always be false when restarting elasticsearch...

Re: [BUG] Ambari-server login loop

This seems to be related to Ambari Metrics somehow which I've put in Maintenance Mode because it's not working. Can I safely delete Ambari Metrics to see whether this is indeed the problem? On 2017-09-11 16:21, Laurens Vets wrote: Hi list, I have a strange issue suddenly... Ambari lets me

Re: Elasticsearch masters_also_are_datanodes doesn't work.

(?) prevents ES from working on a single-node deployment? On 9/10/17, 4:01 PM, "Laurens Vets" <laur...@daemon.be> wrote: Another issue I noticed. Setting "masters_also_are_datanodes" in Ambari to "true" does not work. The settings in /etc/elastic

Re: [BUG] Ambari-server login loop

I have no idea how this happened, but it is fixed now. - I've removed Ambari Metrics and reinstalled it again on a different host. - Rebooted all Metron nodes After this, the login works again. On 2017-09-12 08:15, Laurens Vets wrote: This seems to be related to Ambari Metrics somehow which

Re: [DISCUSS] Community meeting on Tuesday, Sept.23 10AM PST

11:30 won't work for me, but that's fine. I only had 1 comment on Otto's video: What happens when we have 2 parsers/sensors with the same name. If there's ever a parser/sensor repository, this might be an issue. On 2017-09-25 17:38, Otto Fowler wrote: 11:30 your time. Sorry I have to pick my

Re: [DISCUSS] How should Management UI save changes?

Maybe change the text on the button on the primary panel to "write" instead of "save"? Also, I want wider child panels in the management UI if at all possible. Especially the "RAW JSON" feels cramped. On 2017-09-20 14:37, Ryan Merriman wrote: Recently @nickwallen brought up some good points

[DISCUSS] Is there a reason for separate Management & Alerts UIs?

As the subject says, is there a specific reason to have the Management & Alerts UI separate? Having another option under "Operations" called "Alerts" in the Management UI seems to make more sense to me... If it's because they are called Management UI and Alerts UI, maybe we should make it

Error message when changing riskLevelRules

I have the following riskLevelRules: "riskLevelRules": [ { "name": "Not WORK", "comment": "Checks whether the field is_work is true or false.", "rule": "is_work == false", "score": 20, "reason": "FORMAT('%s

Re: [DISCUSS] Build broken due to transitive dependencies

I can confirm 0.4.1 (on CentOS 6!) builds for me as well. Are we sure it isn't due to the version of node shipped with the OS? On 2017-10-02 08:04, zeo...@gmail.com wrote: Hmm, 0.4.1 built fine for me. Jon On Mon, Oct 2, 2017 at 10:44 AM Casey Stella wrote: Ok, the

Re: [DISCUSS] Build broken due to transitive dependencies

fig On 2017-10-02 08:16, Laurens Vets wrote: I can confirm 0.4.1 (on CentOS 6!) builds for me as well. Are we sure it isn't due to the version of node shipped with the OS? On 2017-10-02 08:04, zeo...@gmail.com wrote: Hmm, 0.4.1 built fine for me. Jon On Mon, Oct 2, 2017 at 10:44 AM Casey Ste

Re: Question about SourceHandler and HdfsWriter

Hi Otto, Might this be related to the issues I was seeing? If/when indexing topology got broken, I couldn't recover until I cleared all queues. On 2017-09-04 08:23, Otto Fowler wrote: It looks like if the SourceHandler has a problem with it’s output stream, it will never recover. The

SUM aggregator not working?

No idea whether it's a bug yet, I just need a 2nd set of eyes :) This is my event as indexed in ES (Obviously some parts have been obfuscated): { "_index": "cloudtrail_index_2017.10.04.19", "_type": "cloudtrail_doc", "_id": "95617686-bd39-46ff-b5c0-db3aeb5b6bab", "_score": null,

Re: Cloudtrail use case

On 2017-10-05 15:45, Laurens Vets wrote: Hi, Would anyone be interested in adding a full AWS Cloudtrail use case to the Metron documentation? I would roughly consist of: - Apache NiFi configuration to retrieve Cloudtrail logs from S3 and send it to Metron via Kafka. - Complete Metron sensor

Re: [DISCUSS] Build broken due to transitive dependencies

PM, Laurens Vets <laur...@daemon.be> wrote: ... [INFO] --- frontend-maven-plugin:1.3:npm (ng build) @ metron-config --- [DEBUG] Configuring mojo com.github.eirslett:frontend-maven-plugin:1.3:npm from plugin realm ClassRealm[plugin>com.github.e irslett:frontend-maven-plugin:1.

Re: [DISCUSS] Community Meetings

Sounds good to me :) On 2017-12-14 05:59, Otto Fowler wrote: Ok, So we will be concerned with two types of meetings. I’ll take responsibility for calling the meetings and ‘moderation’. Dev meetings - feedback on how things are going overall - discussions on specific technical problems -

Re: [DISCUSS] Lowering the barrier to entry to for new users

On 2017-12-19 06:19, Justin Leet wrote: One of the topics that came up in recent community meeting was about lowering the barrier to entry for new users. This is a fairly broad topic that I think covers a few different subtopics. 1) Addressing (or making it easier to address) some of the

Re: [DISCUSS] Stellar in a Zeppelin Notebook

On 2017-12-19 07:03, Nick Allen wrote: > (1) I love the REPL, but I hate how inaccessible it is. > > (2) I love our use cases [1] and examples [2], but I hate how difficult it is > for a new user to run them. > > (3) I love the extensibility of Metron, but I hate looking at JSON. > > (4) I

Re: [DEV COMMUNITY MEETING] Call for Ideas and Schedule

I'll try to attend :) On 2017-12-14 12:43, Otto Fowler wrote: Dev Community Meeting Call I would like to propose a developer community meeting. I propose that we set the meeting early next week, and will throw out Monday, December 18th at 09:30AM PST, 12:30 on the East Coast and 5:30 in

Re: [DISCUSS] - Remove Kibana

absolutely still do it, I'm simply saying it would not be managed by us. On Nov 1, 2017 12:20 PM, "Laurens Vets" <laur...@daemon.be> wrote: If there's a viable way of looking at raw processed events (not necessarily alerts), then I'm all for removeing Kibana. I use Discover a lot to

Re: [DISCUSS] - Remove Kibana

If there's a viable way of looking at raw processed events (not necessarily alerts), then I'm all for removeing Kibana. I use Discover a lot to filter and look at events and create new policies from that. Is there currently a simple way to do this without Kibana? On 2017-11-01 09:13, Michael

Re: [DISCUSS] Field conversions

ES 2.x support officially ended 4 months ago (https://www.elastic.co/support/eol), so why still support ':' at all? :) Additionally, 2.x isn't even supported at all on the last 2 Ubuntu LTS releases (16.04 & 18.05). Therefor, move everything to use '.' and provide a conversion/upgrade script

Re: Some more upgrade fallout... Can't restart Metron Indexing

On 2018-01-18 09:14, Casey Stella wrote: So, the challenge here is that our install script isn't smart enough right now to skip creating tables that are already created. One thing you could do is 1. rename the hbase tables for metron (see

Re: [DISCUSS] Using JSON Path to support more complex documents with the JSONMap Parser

On 2018-01-25 07:57, Otto Fowler wrote: While it would be preferred if all data streamed into the parsers is already in ‘stream’ form, as opposed to ‘batched’ form, it may not always be possible, or possible at every step of system development. I was wondering if it would be worth adding

Upgrade from 0.4.1 to 0.4.2 fails on Alerts UI

Hello List, Targeting a wider audience here, see bug report https://issues.apache.org/jira/browse/METRON-1408. Basically, when I upgrade from 0.4.1 to 0.4.2 I run into issues with the Alerts UI. I built the Metron 0.4.2 RPMs and did an upgrade of my current 0.4.1 install with: "rpm -Uvh

Windows full-dev

Hey list, Has anyone gotten full-dev automatically set up on Windows by any chance or are we all using Linux & macOS?