Re: So we graduated...

That's great! Congratulation everybody. On Fri, Apr 21, 2017 at 12:54 PM, Michael Miklavcic < michael.miklav...@gmail.com> wrote: > Congrats all > > On Apr 20, 2017 8:38 PM, "zeo...@gmail.com" wrote: > > > Well done everybody! Congrats > > > > Jon > > > > On Thu, Apr 20, 2017

Re: Post-parsing and Enrichment test framework

. > > > > [Stellar]>>> > > [Stellar]>>> ip_src_addr := "10.0.0.2" > > [Stellar]>>> ip_dst_addr := "10.0.0.3" > > [Stellar]>>> ip_src_port := 22 > > [Stellar]>>> ip_dst_port := 12345 > > [Stel

Elasticsearch 5.x upgrade

Hi all, I've heard there is a plan to upgrade Elasticsearch from 2.x to 5.x regarding Metron and Ambari mpack. I was wondering when that will happen. Is there any part in Metron Elasticsearch indexing that will be impacted by this upgrade? Like any change from the way of bulk-indexing? Cheers,

Post-parsing and Enrichment test framework

Hi all, I was wondering if there is a test framework we can use for Stellar post-parsing and enrichment use cases. It is very time-consuming to verify use cases end-to-end. Therefore, I am looking for a way of mocking use cases step by step to speed up our development. Regards, Ali

Re: Post-parsing and Enrichment test framework

track of changes. Cheers, Ali On Wed, Jul 5, 2017 at 12:06 AM, Simon Elliston Ball < si...@simonellistonball.com> wrote: > You should probably use the Stellar REPL (../metron/bin/stellar -z $ZK) > which gives you a kind of Stellar playground. > > Simon > > > On 4 Jul 2

Re: Normalization topology or separate normalization bolt for parsing topology

Stella <ceste...@gmail.com> wrote: > So, further transformation post-parse was one of the motivating reasons for > Stellar (to do that transformation post-parse). Is there a capability that > it's lacking that we can add to fit your usecase? > > On Wed, Apr 26, 2017 at

Re: Normalization topology or separate normalization bolt for parsing topology

, Otto Fowler <ottobackwa...@gmail.com> wrote: > Hi, > > Are you doing this cleansing all in the parser or are you using any > Stellar to do it? > Can you create a jira? > > > > On April 26, 2017 at 08:59:16, Ali Nazemian (alinazem...@gmail.com) wrote: > > Hi all

Re: Normalization topology or separate normalization bolt for parsing topology

(best effort tyep of thing) through the parser and do > the normalization post-parse..or is there a problem with that? > > On Wed, Apr 26, 2017 at 9:33 AM, Ali Nazemian <alinazem...@gmail.com> > wrote: > > > Hi Casey, > > > > It is actually pre-parse proce

Re: Normalization topology or separate normalization bolt for parsing topology

I've created a Jira ticket regarding this feature. https://issues.apache.org/jira/browse/METRON-893 On Wed, Apr 26, 2017 at 11:11 PM, Ali Nazemian <alinazem...@gmail.com> wrote: > Currently, we are using normal regex at the Java source code to handle > those situations. Howev

Re: Normalization topology or separate normalization bolt for parsing topology

n from > malformed logs, rather than throwing exceptions, but that's more about the > way we write parsers than having some kind of pre-clean. > > Simon > > Sent from my iPad > > > On 27 Apr 2017, at 08:04, Ali Nazemian <alinazem...@gmail.com> wrote: > &g

Re: performance benchmarks on the asa parser

Hi Simon, We have noticed those issues as well. Can you share the changes you have made? so we can merge it with our version. We have implemented about 40-50 more ciscotags so far. It would be great if we can optimize it and contribute back to the community. However, we may end up reimplement it

Re: performance benchmarks on the asa parser

Simon, I have read all emails and now I understand what you are saying. However, I couldn't understand the effect of predictability of latency on enrichments. On Fri, Jun 9, 2017 at 2:45 PM, Ali Nazemian <alinazem...@gmail.com> wrote: > Hi Simon, > > We have noticed those issu

[Discuss] Cyber Security Asset Management for Metron

Hi all, We are going to design and develop an asset database for Metron. For this purpose, I have been thinking of a graph schema model to map assets as Nodes and provide relations as Edges. This can be extended to event level to have a particular relation to assets as well as an event to event

Re: [Discuss] Cyber Security Asset Management for Metron

; > > > It might be good to discuss in the community specific use cases that > would > > be enabled by a graph database. That might help to flesh out the > technical > > aspects of it. > > > > > > > > > > > > On Wed, May 2

Re: Normalization topology or separate normalization bolt for parsing topology

arser that matches your definition of "minimum set". > > My main point here is that I am not seeing a need to re-architect > anything. I think we have the right tools, IMHO. > > > > > > > > > > On Tue, May 2, 2017 at 10:33 AM, Ali Nazemian <alinaze

Re: Normalization topology or separate normalization bolt for parsing topology

tackle a problem like this. Not all data can be > trusted. > > > > > > > > On Thu, Apr 27, 2017 at 9:54 AM, Ali Nazemian <alinazem...@gmail.com> > wrote: > > > Are you sure? The syslog_host name is way more complicated than something > > that can be a

Metron 0.4.2 release date

Hi all, I was wondering when Metron 0.4.2 will be released and whether it includes Metron-777 and Elasticsearch 5.x or not? Cheers, Ali

Re: Metron 0.4.2 release date

l.com>: > > > There's an ongoing conversation regarding client support in Metron here > > > <https://lists.apache.org/thread.html/0c5a837c901dd057420dd8c6b673dc > > 33ba88a8d97545d5b58856cfe8@%3Cdev.metron.apache.org%3E> > > > . > > >

Re: Metron 0.4.2 release date

hs, and trying to look further into the future than that at this > point would be difficult. > > That said, if anybody else has a more detailed timeline in mind, I would > love to hear more. > > Jon > > On Sun, Oct 8, 2017, 09:05 Ali Nazemian <alinazem...@gmail.com> wr

Re: Using Storm Resource Aware Scheduler

Any help regarding this question would be appreciated. On Thu, Nov 23, 2017 at 8:57 AM, Ali Nazemian <alinazem...@gmail.com> wrote: > 30 mins average of CPU load by checking Ambari. > > On 23 Nov. 2017 00:51, "Otto Fowler" <ottobackwa...@gmail.com> w

Re: Using Storm Resource Aware Scheduler

is no need for the new JIRA ( > https://issues.apache.org/jira/browse/METRON-1330 < > https://issues.apache.org/jira/browse/METRON-1330>). It should be closed > as a duplicate of https://issues.apache.org/jira/browse/METRON-1161 < > https://issues.apache.org/jira/browse/

Re: [DISCUSS] Are/how are you using the ES data pruner?

We tried to use it, but we had the same issue. It was not documented. We tried to use it, and we had some issues. It also was not exactly what we wanted, so we decided to create something from scratch by using Elasticsearch Curator. We wanted to have an ability to manage different prune mechanism

Using Storm Resource Aware Scheduler

Hi all, One of the issues that we are dealing with is the fact that not all of the Metron feeds have the same type of resource requirements. For example, we have some feeds that even a single Strom slot is way more than what it needs. We thought we could make it more utilised in total by

Re: [DISCUSS] Are/how are you using the ES data pruner?

ke > > On Nov 22, 2017 8:53 PM, "Ali Nazemian" <alinazem...@gmail.com> wrote: > > > We tried to use it, but we had the same issue. It was not documented. We > > tried to use it, and we had some issues. It also was not exactly what we > > wanted, so we decid

Re: [DISCUSS] Are/how are you using the ES data pruner?

on, Nov 27, 2017 at 3:46 PM, James Sirota <jsir...@apache.org> wrote: > > > One thing to keep in mind, as we will be introducing Solr shortly, is to > > find if something similar to curator exists for Solr. But we'll cross > that > > bridge when we get there > &g

Re: Using Storm Resource Aware Scheduler

30 mins average of CPU load by checking Ambari. On 23 Nov. 2017 00:51, "Otto Fowler" <ottobackwa...@gmail.com> wrote: How are you measuring the utilization? On November 22, 2017 at 08:12:51, Ali Nazemian (alinazem...@gmail.com) wrote: Hi all, One of the issues tha

Re: Heterogeneous indexing batch size for different Metron feeds

Any thoughts? On Sun, Dec 3, 2017 at 11:27 PM, Ali Nazemian <alinazem...@gmail.com> wrote: > Hi, > > We have noticed recently that no matter what batch size we use for Metron > indexing feeds, as long as we start using different batch size for > different Metron fee

Re: Heterogeneous indexing batch size for different Metron feeds

No specific error in the logs. I haven't enabled debug/trace, though. On Tue, Dec 5, 2017 at 11:54 AM, Otto Fowler <ottobackwa...@gmail.com> wrote: > My first thought is what are the errors when you get a high error rate? > > > On December 4, 2017 at 19:34:29, Ali Nazemian (ali

Re: Heterogeneous indexing batch size for different Metron feeds

That code does not have any logging to speak of… well debug / trace > logging that would help here either. > > > > On December 6, 2017 at 08:18:01, Ali Nazemian (alinazem...@gmail.com) > wrote: > > Everything looks normal except the high number of failed tuples. Do you > know how the

Re: Heterogeneous indexing batch size for different Metron feeds

com> wrote: > What do you see in the storm ui for the indexing topology? > > > On December 6, 2017 at 07:10:17, Ali Nazemian (alinazem...@gmail.com) > wrote: > > Both hdfs and Elasticsearch batch sizes. There is no error in the logs. It > mpacts topology error rate a

Re: Heterogeneous indexing batch size for different Metron feeds

er 5, 2017 at 08:03:46, Otto Fowler (ottobackwa...@gmail.com) wrote: Which of the indexing options are you changing the batch size for? HDFS? Elasticsearch? Both? Can you give an example? On December 5, 2017 at 02:09:29, Ali Nazemian (alinazem...@gmail.com) wrote: No specific error in the

Re: Heterogeneous indexing batch size for different Metron feeds

d be wrong though. > > > > > On December 7, 2017 at 06:47:15, Ali Nazemian (alinazem...@gmail.com) > wrote: > > Thank you very much. Unfortunately, reproducing all the situations are > very costly for us at this moment. We are kind of avoiding to hit that > issue by u

Re: Heterogeneous indexing batch size for different Metron feeds

he levers outlined above, iterate with each change in rapid >succession, and record your results. > > 1. > https://github.com/apache/metron/blob/master/metron- > platform/Performance-tuning-guide.md > > Sample command without Kerberos enabled (see link [1] for more detail wit

Heterogeneous indexing batch size for different Metron feeds

Hi, We have noticed recently that no matter what batch size we use for Metron indexing feeds, as long as we start using different batch size for different Metron feeds, indexing topology throughput will start dropping due to the high error rate! So I was wondering whether based on the current

Streaming Machine Learning use case

Hi all, I was wondering if someone has used Metron with any streaming ML framework such as SAMOA? I know that Metron provides Machine Learning separately via MAAS. However, it is hard to manage it from operational perspective especially if we want to have a pretty dynamic and evolving model.

Re: Streaming Machine Learning use case

hat we'd be better off looking at algorithms in Spark for > things like frequent pattern mining, though there the FP growth algorithm > is of course primarily a batch implementation. > > Are there any SAMOA algorithms in particular that you think would be > relevant to Metron use cases? > &g

Re: Using Java Rest Client instead of Transport Client for Elasticsearch

n Thu, Jun 14, 2018 at 2:28 PM Ali Nazemian wrote: > Hi Michael and Casey, > > It looks like ES believe Java Rest Client is mature enough to be pushed to > different products at this stage. However, I haven't used it personally. I > will share the question regarding x-pack

Re: Metron Alert UI and zero-down time Elasticsearch re-index

; > 01.01.2018, 22:30, "Ali Nazemian" <alinazem...@gmail.com>: > > Hi All, > > > > We are using an older version of Metron Alert-UI (Received in Oct 2017) > > which sends search queries to ES directly without using Metron Rest API. > We > >

Re: [DISCUSS] Update Metron Elasticsearch index names to metron_

Hi All, I just wanted to say it would be great if we can be careful with these type of changes. From the development point of view, it is just a few lines of code which can provide multiple advantages, but for live large-scale Metron platforms, some of these changes might be really expensive to

Re: Metron Alert UI and zero-down time Elasticsearch re-index

It would be great if we can have some help on this issue. Cheers, Ali On Sat, Jan 6, 2018 at 12:33 PM, Ali Nazemian <alinazem...@gmail.com> wrote: > Hi James, > > Due to changes in the field format, I want to create a new index with the > new format. Create an alias to refer t

Re: Enrichment and indexing routing mechanism

o I think if you change it in stellar it should work. Have you tried and failed? On January 29, 2018 at 07:22:23, Ali Nazemian (alinazem...@gmail.com) wrote: Yes, exactly. On Mon, Jan 29, 2018 at 11:15 PM, Otto Fowler <ottobackwa...@gmail.com> wrote: > Are you trying to change th

Enrichment and indexing routing mechanism

Hi All, I was wondering how the routing mechanism works in Metron currently. Can somebody please explain how Enrichment Storm topology understands a single event is related to which Metron feed? What about indexing? is that based on "source.type" field? Cheers, Ali

Re: Enrichment and indexing routing mechanism

And I am trying to understand if I set a post-parser Stellar transformation to change the value of "source.type" will it impact enrichment routing or it will get overwritten by an internal method? On Mon, Jan 29, 2018 at 11:22 PM, Ali Nazemian <alinazem...@gmail.com> wrote:

Re: Enrichment and indexing routing mechanism

, Simon Elliston Ball < si...@simonellistonball.com> wrote: > Yes, it is. > > Sent from my iPhone > > > On 29 Jan 2018, at 09:33, Ali Nazemian <alinazem...@gmail.com> wrote: > > > > Hi All, > > > > I was wondering how the routing mechanism works in

Re: Enrichment and indexing routing mechanism

> Kafka (indexing) -> Indexing topologies (ES / Solr / HDFS) configured based > on the indexing config named the same as source.type -> wherever the > indexer tells it to be. > > Simon > > > On 29 Jan 2018, at 11:53, Ali Nazemian <alinazem...@gmail.com> wrote: > >

Disable Metron parser output writer entirely

Hi All, I am trying to investigate whether we can disable a Metron parser output writer entirely and manage it via KAFKA_PUT Stellar function instead. First, is it possible via configuration? Second, will be any performance difference between normal Kafka writer and the Stellar version of it

Re: Disable Metron parser output writer entirely

ses only. It’s a very non-stellar construct > (non-expression, no return, side-effect dependent…) Also, it creates a > producer for every call, so your are definitely not going to get > performance out of it. > > Simon > > > On 5 Feb 2018, at 06:32, Ali Nazemian

Re: Disable Metron parser output writer entirely

What about the performance difference? On Fri, Feb 2, 2018 at 10:41 PM, Otto Fowler <ottobackwa...@gmail.com> wrote: > You cannot. > > > > On February 1, 2018 at 23:51:28, Ali Nazemian (alinazem...@gmail.com) > wrote: > > Hi All, > > I am trying to investi

[DISCUSS] community view/roadmap of threat intel

Hi All, I would like to understand Metron community view on Threat Intel aggregators as well as the roadmap of threat intelligence and threat hunting. There are some open source options available regarding threat intel aggregator such as Minemeld, Hippocampe, etc. Is there any plan to build that

Re: [DISCUSS] community view/roadmap of threat intel

gt; intel loader, or even through a direct to hbase streaming connector. > > Simon > > > On 14 Feb 2018, at 03:13, Ali Nazemian <alinazem...@gmail.com> wrote: > > > > Hi All, > > > > I would like to understand Metron community view on Threat Intel > >

Re: [DISCUSS] community view/roadmap of threat intel

gt; > > today the default Metron schema seems to lack any similar concept? Do > we > > > have plans to address it? > > > > > > 3. Atemporal matching - Given the use of big data technologies it seems > > to > > > me Metron should be able to look into past

ES mpack to include more ES 5 stack properties

Hi All, Is there any plan to include more ES 5+ specific properties to Metron mpack? For example, if we want to use dedicated nodes for Master Nodes, Data Nodes, Ingestion Nodes and ML Nodes and different configurations for them, how can we proceed? It may be out of the scope of the current

Metron Alert UI and zero-down time Elasticsearch re-index

Hi All, We are using an older version of Metron Alert-UI (Received in Oct 2017) which sends search queries to ES directly without using Metron Rest API. We wanted to run a zero-downtime ES reindex process by using ES aliasing. However, I am not sure how it will impact the search part of Alert-UI

Metron nested object

Hi all, We have recently faced some data sources that generate data in a nested format. For example, AWS Cloudtrail generates data in the following JSON format: { "Records": [ { "eventVersion": *"2.0"*, "userIdentity": { "type": *"IAMUser"*,

Re: Metron nested object

; > On December 21, 2017 at 08:28:13, Ali Nazemian (alinazem...@gmail.com) > wrote: > > Hi all, > > > We have recently faced some data sources that generate data in a nested > format. For example, AWS Cloudtrail generates data in the following JSON > format: > >

Change field separator in Metron to make it Hive and ORC friendly

Hi All, I was wondering if we can change the field separators in Metron to be able to make it Hive/ORC friendly. I could find the following PR, but neither dot nor colon is very Hive and ORC friendly and they will cause some issues. Hence, I wanted to see if it is possible to change the field

Re: Change field separator in Metron to make it Hive and ORC friendly

. > > Simon > > Sent from my iPhone > > > On 14 Aug 2018, at 11:42, deepak kumar wrote: > > > > I agree Ali. > > May be it can be configuration parameter. > > > >> On Tue, Aug 14, 2018 at 3:e t24 PM Ali Nazemian > wrote: > >> >

Re: Change field separator in Metron to make it Hive and ORC friendly

; Do you have any suggestions for what would make sense as a delimiter? > > On 9 August 2018 at 05:57, Ali Nazemian wrote: > > > Hi All, > > > > I was wondering if we can change the field separators in Metron to be > able > > to make it Hive/ORC friendly. I could

Re: [DISCUSS] Getting to a 1.0 release

One thing that we could imagine for v1.0 might be an ability to extend Metron from adding more pipelines to it. For example, being able to extend Metron to be integrated with other endpoints more easily from Storm perspective. For example, what if we would like to create other topologies to write

Re: [ANNOUNCE] - Apache Metron Slack channel

Can I be invited as well? On Thu, Aug 16, 2018 at 4:37 AM Otto Fowler wrote: > Done > > > On August 15, 2018 at 14:22:45, Vets, Laurens (laur...@daemon.be) wrote: > > Could I be invited? > > On 15-Aug-18 09:48, Michael Miklavcic wrote: > > + Metron user list > > > > On Wed, Aug 15, 2018 at

Re: [DISCUSS] Generic Syslog Parsing capability for parsers

Just adding more details regarding what different parts are: There are three stages here that need to be understood: 1- pre-parsing 2- chain of parsing (wrapping one type of message in another format) 3- post-parsing aka normalization Pre-parsing stage is where we need to specify what specific

Re: HCP in Cloud infrastructures such as AWS , GCP, AZURE

Depending on the model of security, you may have some challenges with the Ranger integration with your cloud storage especially if you are thinking of using TDE for the encryption at rest. Otherwise, using Metron in that way should be quite feasible. However, you may face some performance issues

Re: [DISCUSS] Slack Channel Use

I kind of expect to have Slack for more dev related discussions rather than user QA. I guess it is quite common to expect mailing list to be used for the purpose of knowledge sharing to make sure it will be accessible by other users as well. Of course, it is a trade-off that most of the other

Re: [DISCUSS] Recurrent Large Indexing Error Messages

I think if you look at the indexing error management, it is pretty much similar to parser and enrichment error use cases. It is even more common to expect something ended up in error topics. I think a wider independent job can be used to take care of error management. It can be decided to add a

Re: Authorization for Configuration

n > > items as subtasks on the FB Jira so that we can crosscheck what entry > > points have been implemented against the test scripts. Do you think this > > will impact streaming enrichments or the profiler at all? That is to say, > > as Ali asked, just how far are you looking to t

Re: [DISCUSS] Deprecating MySQL

Great feature to move to LDAP integration and hopefully Ranger integration afterwards. Does it need to support LDAP and AD separately? Cheers, Ali On Sat, Nov 17, 2018 at 3:29 AM Otto Fowler wrote: > I would like to understand the work required to move our JDBC support ( or > adapt the current

Re: [DISCUSS] Authorization for Configuration

Hi Justin, By configuration do you mean the sensor related configurations only? Are you limiting the scope of this activity to the management-UI or also Alert-UI as well? For example, defining different roles (pre-defined or customizable) and the fine-grained integration with Ranger? Cheers, Ali

Re: Authorization for Configuration

Hi Justin, By configuration do you mean the sensor related configurations only? Are you limiting the scope of this activity to the management-UI or also Alert-UI as well? For example, defining different roles (pre-defined or customizable) and the fine-grained integration with Ranger? Cheers, Ali

Re: [DISCUSS] Deprecate split-join enrichment topology in favor of unified enrichment topology

Hi, One thing to point out here is there were a few timestamp fields that exist for Split-join enrichment topology that haven't been made to the unified one. For example, there is no threat intel bolt timestamp. There might be some SLA related use cases regarding these timestamp fields that might

Re: [DISCUSS] Handling dropped messages in REGEX_SELECT with Kafka topic routing

Just one thing to bear in mind, publishing an error may cause some operational challenges as it fills up the error topic as well as storm logs which may not be necessary. To wear a Metron user hat, dropping a message with a debug/trace level log to specify the event is filter out makes sense. I

Re: [DISCUSS] Internal Metron fields

Totally agree with replacing dot with something else. We have had so much drama to use either dot or column with ORC either via Hive or Spark. Although we have replaced it with an underscore, it may not be a good idea as it can be confusing with underscores in the internal field names. Cheers,

[DISCUSS] Real-time processing engine: Storm, Spark, Flink or Cloud Native

Hi All, As far as I understood, there is a plan to change the real-time engine of Metron due to some issues that user and developer have been facing with it. I would like to explain some critical issues that customer have been facing to clarify it for the development team what the best approach

Re: [DISCUSS] Real-time processing engine: Storm, Spark, Flink or Cloud Native

have nearly completed > decoupling our core infrastructure from Storm at this point, which opens us > up to a number of possibilities going forward. > > Best, > Mike Miklavcic > > > On Thu, Apr 4, 2019 at 1:35 AM Ali Nazemian wrote: > > > Hi All, > > > >