Re: Integrating with a separate kafka cluster

Good afternoon Euan, In this group and in others I often see Nifi characterized (in my own words), as the bridge that can be used to connect various places you want to move data. If you haven't explored it yet, then it might be worthwhile to take an afternoon and have a look. I'm exploring a

RE: Drop events from Metron parser

servers and Kafka? Cheers, Tom. On 2020-05-05 17:25:01-07:00 Dima Kovalyov wrote: I would drop them on ingestion using NiFi's RouteOnContent. On Tue, May 5, 2020, 17:53 Yerex, Tom mailto:tom.ye...@ubc.ca>> wrote: Good afternoon, Our incoming data is not always perfect, in some cases

Drop events from Metron parser

Good afternoon, Our incoming data is not always perfect, in some cases events are simply missing fields. We would like a way to drop events when particular fields are empty (or have values we don't care about). One way we thought to do this might be to write a custom Stellar function. Does

RE: Metron alerts ui

Good afternoon Jai, I think that might be how the system currently behaves, if I am reading this post correctly: hxxps://community.cloudera.com/t5/Support-Questions/Metron-Alerts-UI/td-p/198406 Cheers, Tom On 2020-04-03 21:46:29-07:00 Geeks Girls wrote: Hi, How can I control the log entry

RE: How to secure Apache Metron cluster

Good morning Thuy, We are focused on multiple layers of security, beginning with the firewall but also local access control and monitoring down to individual processes running in the environment. Kerberos is a mechanism that is discussed as a security mechansim and I have had it working with

RE: Metron 0.7.1 - HttpServerException: 500 Server Error

ing metron, appeared in Ranger I switched the setting for usersync to use our Active Directory again. Once metron appeared in the user list, I updated the policy to allow metron and made several other changes to unblock storm, etc., and the error went away. Cheers, Tom. On 2020-04-03 15:21

RE: Metron 0.7.1 - HttpServerException: 500 Server Error

. -- Tom Yerex Cybersecurity Analyst, Information Technology Cybersecurity | CISO Office The University of British Columbia | Musqueam Traditional Territory Ponderosa Office Annex A | Vancouver BC | V6T1Z2 Canada Phone 604 822 6531 Privacy Matters @ UBC On 2020-04-03 15:21:15-07:00 Yerex, Tom

Metron 0.7.1 - HttpServerException: 500 Server Error

Running Metron 0.7.1 on CentOS7 in a Kerberos-enabled cluster. Metron Admin gui is having problems, the following error appears in metron-rest.log; my hunch is there is a document I need to read and follow, but since there is a few copies of various documents I thought it best to check with the

Metron with Knox and reload issue

Good evening, Working with the instructions from hxxps://github.com/apache/metron/tree/master/metron-interface This is a new installation and we are using LDAP with Metron and now attempting to use Knox for access control. Using Apache Metron Management and Alerts UI directly, the login works

RE: Connecting Metron to Elasticsearch with credentials

Metron setup because of covid19, but as far as I remember there is nothing special here, you have the login / password fields available to configure your credentials in Elasticsearch definition. What is your trouble? From: Yerex, Tom [mailto:tom.ye...@ubc.ca] Sent: Tuesday, March 24, 2020 22:57

Connecting Metron to Elasticsearch with credentials

Good afternoon, Our Elasticsearch install requires credentials to connect over port 9200. Has anyone set up a connection between Metron and Elasticsearch using credentials and/or can offer some guidance on how to achieve this? Cheers, Tom.

Re: Using something other than colons in field names?

r performance in sub documents in elastic search 2.x. It’s really a legacy thing now, and the NOOP should be considered the best path going forward. Simon On Mon, 3 Feb 2020 at 12:41, Yerex, Tom wrote: Thank you Vladimir. Before I go diving into making a lot of changes from the default,

Re: Using something other than colons in field names?

e: "elasticsearch": { "batchSize": 100, "enabled": true, "index": "myindex", "fieldNameConverter": "NOOP" }, On 2020/02/01 00:00:04, "Yerex, Tom&quo

Using something other than colons in field names?

Good afternoon, Our Metron installation uses colons in the field names. For example, geo ip enriched data appears as “enrichments:geo:ip_dst_addr:country”. Under Kibana (and from what I read Banana), the colon cannot be properly escaped for use with Timelion. My question: has anyone

Re: Stellar and distinct count

"period.end": 1426350194842, "timestamp": 1426350199232, "is_alert": true } (III) Threat Triage Now create a set of threat triage rules that score this new source of telemetry coming from the Profiler. The 'entity' field will contain th

Re: Enrichment (enrichments?) in Metron

r loading enrichment data into HBase. This supports loading smaller data sets from a local file or much larger data sets using an MR job against data stored in HDFS. [1] https://github.com/nickwallen/metron/tree/master/metron-platform/metron-data-management#flatfile-loader On Fri, Jan 17, 2

Re: Metron Tutorial - Fundamentals....Part 3?

Re: Metron Tutorial - FundamentalsPart 3? Hi Tom, For as long as I've been on the project, since our incubator days, there hasn't been a part 3. It skips from 2 to 4. On Thu, Jan 16, 2020 at 7:25 PM Yerex, Tom wrote: Good afternoon, I have been working through some internal docu

Metron Tutorial - Fundamentals....Part 3?

Good afternoon, I have been working through some internal documentation to reflect the official documentation (along with some edits), and I can’t seem to find part 3 of the Metron Tutorial that would come after part 2.

Stellar and distinct count

Good afternoon, We are working on enhancements from the geographic login outliers from hxxps://metron.apache.org/current-book/use-cases/geographic_login_outliers/index.html. The original solution works very well, thank you to those who put in the work creating documentation, developing

Metron enrichment

Good afternoon, I wanted to see if the information posted at https://metron.apache.org/current-book/metron-platform/metron-enrichment/index.html is still accurate. Specifically, under “Deployment Options”, the sentence, “There is currently one option for running enrichments in Metron, which

Seeking comments - Solr and Elasticsearch

Good afternoon, I’m fishing for some insight and experience, hopefully someone has a strong opinion and is willing to share. We are currently exploring the indexing options available in Metron. From what I can gather Elasticsearch has a great marketing budget and Solr has some large

Re: Altering the group by and filter fields in Metron Alerts

urce IP: `ip_src_addr` Country: `enrichments:geo:ip_dst_addr:country` Ideally, the fields available for grouping could be made configurable, but that change is not trivial. On Tue, Dec 3, 2019 at 6:18 PM Yerex, Tom wrote: Good afternoon, This applies to Metron 0.7.1. I am

Altering the group by and filter fields in Metron Alerts

Good afternoon, This applies to Metron 0.7.1. I am working with the Metron Alerts interface to expose data from the Metron geographic outliers case study (https://metron.apache.org/current-book/use-cases/geographic_login_outliers/index.html). We leverage the Elasticsearch common

Re: Re-establish /apps/metron directory

etron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_commands.py#L183 Best, Mike Miklavcic On Tue, Nov 5, 2019 at 10:44 AM Yerex, Tom wrote: Good morning, I wanted to share with the group that once the top-level “/apps/metron” folder is created

Re: Error stemming from hbaseBolt

ing that in a reply. That should give the community some better detail to help. Thanks, Mike On Tue, Nov 5, 2019 at 5:47 PM Yerex, Tom wrote: Good evening, I am working with metron 0.7.2. I have tried to implement a variation on the solution at hxxps://metron.

Error stemming from hbaseBolt

Good evening, I am working with metron 0.7.2. I have tried to implement a variation on the solution at hxxps://metron.apache.org/current-book/use-cases/geographic_login_outliers/index.html, with the modification that data is coming into a Kafka topic (JSON format) instead of an import

Re: Re-establish /apps/metron directory

Good morning, I wanted to share with the group that once the top-level “/apps/metron” folder is created, the rest of the folders seem to eventually reappear on their own after a few app restarts in Ambari. I have spun up a development instance of Metron (Centos7), to compare, so far it

Re-establish /apps/metron directory

Good afternoon, During development I squashed the /apps/metron/… directory in HDFS. Would someone be able to pinpoint for me the code that builds the /apps/metron directory and sub-directories so I can fix the state of those folders without a reinstallation? Cheers, Tom.

Antivirus in cluster?

Good morning, Has anyone deployed an antivirus product in their Metron cluster? If you have, I am assuming you have compiled a list of files/directories to exclude, would you be willing to share that list? If your organization has opted to not install any antivirus product, it would be

Re: Fields with a period/dot in the name

urrent-book/metron-platform/metron-parsers/index.html On Fri, Nov 1, 2019 at 1:21 PM Yerex, Tom wrote: Good day to everyone. I'm working on our own variation of the Geographic Login Outliers use case (https://metron.apache.org/current-book/use-cases/geographic_login_outlie

Fields with a period/dot in the name

Good day to everyone. I'm working on our own variation of the Geographic Login Outliers use case (https://metron.apache.org/current-book/use-cases/geographic_login_outliers/index.html). I noticed that our fields names arrive with a period in the name, for example "client.ip" and "user.id".

Re: apache storm error

I am working from memory so I am not entirely certain, but I think we had a similar error that was resolved by increasing the JVM heap for Elasticsearch from the default. In Ambari, under “Advanced elastic-jvm-options”, the “heap_size” setting. In our environment it is set to 2048m.

Re: CSV formatted data to Metron and deploying ML models

Hi Hema, I think the CSV formatted data can be imported into HBase and then applied from there. I don’t know how accurate this is, but I was able to do some very basic enrichment using the steps here:

RE: Apache Metron production deployment

Hello Marcus, Currently, we are using Metron in a development environment, running on a VMware product. Our deployment is deployed via Ansible with additional steps added to the Ansible code from the git repo. The operating system is CentOS 7 (minimal), which requires some additional steps to

Re: [DISCUSS] How are you using in Metron?

This is a great topic, thanks for posting it. 1. My team is focussed or about to use 1, 2, 3, 7, 8, 9, 10, 11 (ES), 13, 14. We are in for the long haul and I'm personally very excited to have a (very small), part to play working in Metron. 2. I have only been working with Metron a short time,

Re: use another geoIP db for enrichment

pstack.com/product You will need to write the functionality as Nick Allen mentioned in his response. --Tom. On 2019-04-01, 9:09 AM, "tkg_cangkul" wrote: Hi, well actually i'm looking for free geoIP db. Cheers, Tkg_cangkul On 01/04/19 22:

Re: use another geoIP db for enrichment

Good morning, Does it have to be free or not? Cheers, Tom. On 2019-04-01, 8:49 AM, "tkg_cangkul" wrote: Hi, Is there any ways to use another geoIP for metron.? I wanna try to use another geoIP other than geolite. if it's possible, pls give me some reference link to do