[DISCUSS] deprecate misleading install methods and docs?

Following many discussions on the user and dev lists in the past, a number of users seem to have problems with the old ansible methods for installing AWS. I am not aware of anyone who is maintaining this area (please shout if you are willing to take on bringing this up to date) and we have a lo

Re: Threat Intel hailataxii

Looks to me like your discovery server is not working properly, hence the failure message. This could be a temporary connectivity issue, but if it’s repeatable I would look into your opentaxii config. Simon > On 29 Oct 2019, at 13:23, Thiago Rahal Disposti > wrote: > >  > Anyone knows wha

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

: > If anyone can think of the things that need to be backed up, please > comment the jira. > > > > > On August 27, 2019 at 17:07:20, Otto Fowler (ottobackwa...@gmail.com) > wrote: > > Good idea METRON–2239 [blocker]. > > > > On August 27, 20

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

You could always submit a Jira :) On Tue, 27 Aug 2019 at 21:27, Otto Fowler wrote: > You are right, that is much better than backup_metron_configs.sh. > > > > > On August 27, 2019 at 16:05:38, Simon Elliston Ball ( > si...@simonellistonball.com) wrote: > > You can d

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

sion running on HDP >>> 3.x. If there is any discrepancy between the two or additional settings >>> will be required, those will be documented in the release notes. From the >>> Metron perspective, this upgrade would be no different than simply >>> upgrading to the

Re: Good first issues to get started with?

sted in already had some work going on. > > Thanks > -jim spring > -- -- simon elliston ball @sireb

Re: Build Failed for 0.7.2

cher.java:415) > at > org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356) > [INFO] > > [INFO] BUILD FAILURE > [INFO] > > > > -- > *Best Regards* > Farrukh Naveed Anjum > *M:* +92 321 5083954 (WhatsApp Enabled) > *W:* https://www.farrukh.cc/ > -- -- simon elliston ball @sireb

Re: [DISCUSS] JsonMapParser original string functionality

processes, allowing for recreation of original form for forensics and evidentiary purposes. Simon > On 11 May 2019, at 00:10, Otto Fowler wrote: > > What about parser chaining? Should the original string be from kafka, or > the last parsed? > > > On May 10, 2019 at 19:03:39

Re: [DISCUSS] JsonMapParser original string functionality

The only scenario I can think of where a parser might treat original string differently, or even need to know about it would be different encoding locales. For example, if the string were to be encoded in a locale specific to the device and choose the encoding based on metadata or parsed content

Re: [DISCUSS] Upgrading HBase and Kafka support

The Docker option sounds like a much better and cleaner option for integration testing (closer to real too). My one question would be whether this would significantly increase test run time, and whether that would need Travis changes? Either way, the docker option sounds best. Simon > On 8 M

Re: [DISCUSS] Knox SSO feature branch review and features

knows when that gets released). > > What do we do in the meantime? I would also like to point out that > Metron > > is inherently different than other Hadoop stack services. We are a > > full-blown application with multiple UIs so the way we expose services > > through Knox m

Re: Running MAAS in batch

mass scoring? > > Thanks > Deepak > > On Fri, Nov 16, 2018 at 9:15 PM Otto Fowler > wrote: > >> That may be the best MAAS explanation I’ve seen Simon. >> >> >> On November 16, 2018 at 10:28:57, Simon Elliston Ball ( >> si...@simonellistonball.com) wr

Re: [DISCUSS] Knox SSO feature branch review and features

;>>>>> > > >>>>>>> > > >>>>>> > > >>>> > > >>> > > >> > > > > > > > https://github.com/angular/angular-cli/wiki/build#base-tag-handling-in-indexhtml > > >>

Re: Running MAAS in batch

tron platform. > Is there any way to run the models deployed in MAAS on the batch events / > data that have been indexed into hdfs ? > If anyone have tried this batch model , please share some insights. > Thanks > Deepak. > > -- -- simon elliston ball @sireb

Re: [DISCUSS] Knox SSO feature branch review and features

>> now >>>>>>>>> by >>>>>>>>> default. I imagine we'll deprecate JDBC-based authentication >> at >>>> some >>>>>>>>> point >>>>>>>>> so that may be a good time to switch. >>>>&

Re: [DISCUSS] Deprecating MySQL

een in > Metron. Provide all the options should a user desire them, but abstract > away the complexity in the UIs. > > Best, > Mike > > > On Tue, Nov 13, 2018 at 5:42 AM Simon Elliston Ball < > si...@simonellistonball.com> wrote: > >> I've been

[DISCUSS] Deprecating MySQL

I've been coming across a number of organisations who are blocked from installing Metron by the MySQL auth database. The main problems with our MySQL default are: * What? Un-ecrypted passwords?!? - which frankly is embarrassing in a security platform and usually where the deployment conversation

Re: [DISCUSS] Knox SSO feature branch review and features

of how to do that or can point to > some documentation, please share. > > On Mon, Nov 12, 2018 at 8:54 AM Simon Elliston Ball < > si...@simonellistonball.com> wrote: > > > Doing the Knox proxy work first certainly does make a lot of sense vs the > > SSO first app

Re: [DISCUSS] Knox SSO feature branch review and features

gt; > > > > > >> > > > I've spent some more time reading through Simon's response and > > the > > > >> > added > > > >> > > > sequence diagram. This is definitely helpful - thank you > Simon. > > > >> > > > > > > >> > >

Re: Revert PR #1218

e before introducing this change. > > I am going to revert the change on master, which will introduce an > additional commit that is an "undo" of the original commit. I will then > open a separate PR that introduces this new functionality. > > https://github.com/apache/metron/pull/1218 > > Thanks > -- -- simon elliston ball @sireb

Re: [DISCUSS] Knox SSO feature branch review and features

To clarify some of this I've put some documentation into https://github.com/apache/metron/pull/1203 under METRON-1755 ( https://issues.apache.org/jira/browse/METRON-1755). Hopefully the diagrams there should make it clearer. Simon On Tue, 18 Sep 2018 at 14:17, Simon Elliston Ball

Re: [DISCUSS] PCAP data for testing and development

ep would be a good approach? > Or sensor stubs for pcap would be a better way? > > I would be curious about your thoughts! > > Thanks, > Tibor > -- -- simon elliston ball @sireb

Re: [DISCUSS] Knox SSO feature branch review and features

; > >REST layer, and to provide a routing platform for later > > microservices." > > > - > > >https://issues.apache.org/jira/browse/METRON-1665. > > > - Microservices is a pretty loaded term. I know there had been > some > > > discussion a while back during the PCAP feature branch start, > but I > > > don't > > > recall ever reaching a consensus on it. More detail in this > thread > > - > > > > > > > > > https://lists.apache.org/thread.html/1db7c6fa1b0f364f8c03520db9989b4f7a446de82eb4d9786055048c@%3Cdev.metron.apache.org%3E > > > . > > > Can we get some clarification on what is meant by microservices > > > in the case > > > of this FB and relevant PR's, what that architecture looks like, > > and > > > how > > > it's achieved with the proposed changes in this PR/FB? It seems > > Zuul > > > is > > > also pertinent to this discussion, but there are many ways to > > > skin this cat > > > so I don't want to presume - > > > > > > https://blog.heroku.com/using_netflix_zuul_to_proxy_your_microservices > > > 6. Zuul, Spring Boot, and microservices - Closely related to > > point 5 > > >above. It seems that we weren't quite ready for this when it was > > > brought up > > >in May, or at the very least we had some concern of what direction > to > > > go. > > >What is the operational impact, mpack impact, and how we propose to > > > manage > > >it with Kerberos, etc.? > > > > > > > > > https://lists.apache.org/thread.html/c19904681e6a6d9ea3131be3d1a65b24447dca31b4aff588b263fd87@%3Cdev.metron.apache.org%3E > > > > > > There is a lot to like in this feature branch, imo. Great feature > > addition > > > with Knox and SSO. Introduction of LDAP support for authentication for > > > Metron UI's. Simplification/unification of our server hosting > > > infrastructure. I'm hoping we can flesh out some of the details pointed > > out > > > above a bit more and get this feature through. Great work so far! > > > > > > Best, > > > Mike Miklavcic > > > > > > -- -- simon elliston ball @sireb

Re: [DISCUSS] Contributing a General Purpose Regex Parser

dType and regex. The >expression that is evaluated is based on the output of the > recordTypeRegex >- Note: recordTypeRegex and messageHeaderRegex could be specified as >lists also (as a JSON array), where the list will be evaluated in order >until a matching regular expression is found. > > > > > > If there are no objections to having this type of Parser within Metron, we > will open a JIRA/PR for code review. > > *Jagdeep Singh* > -- -- simon elliston ball @sireb

Re: [DISCUSS] Getting to a 1.0 release

it is on the roadmap”. > > Regardless of the implementation, conceptually, security of data at rest is > important, and is a major outstanding item or the core metron proposition. > > > > >> On August 15, 2018 at 16:03:19, Simon Elliston Ball >> (si...@simonelli

Re: [DISCUSS] Getting to a 1.0 release

and closing it > > > >> On August 15, 2018 at 15:53:02, Otto Fowler (ottobackwa...@gmail.com) wrote: >> >> https://issues.apache.org/jira/browse/METRON-343 >> >>> On August 15, 2018 at 15:47:24, Simon Elliston Ball >>> (si...@simonellistonba

Re: [DISCUSS] Getting to a 1.0 release

ote: > > https://issues.apache.org/jira/browse/METRON-343 > >> On August 15, 2018 at 15:47:24, Simon Elliston Ball >> (si...@simonellistonball.com) wrote: >> >> What would you see as secure? I’ve seen people use TDE for the HDFS store, >> but it’s harder to encry

Re: [DISCUSS] Getting to a 1.0 release

What would you see as secure? I’ve seen people use TDE for the HDFS store, but it’s harder to encrypt storage with solr / es. Something I was thinking of doing to follow up on the Knox Feature was to add Ranger integration for securing and auditing configs, and potentially extending to the index

Slack Channel

Hello dev team, may I please join your slack channel :)

Re: [ANNOUNCE] - Apache Metron Slack channel

>3. Use your Apache email for your login > >4. Click "Channels" and look for #metron (Created by ottO June 15, > 2018) > > > > Best > > Mike Miklavcic > > > -- -- simon elliston ball @sireb

Re: Change field separator in Metron to make it Hive and ORC friendly

on separator. Maybe it would be nice to have an ability to >> change the separator to any other character and let users decide what they >> want to use. >> >> Cheers, >> Ali >> >> On Tue, Aug 14, 2018 at 12:14 AM Simon Elliston Ball < >> si...@si

Re: Change field separator in Metron to make it Hive and ORC friendly

Elasticsearch > and HDFS. > > https://github.com/apache/metron/pull/1022 > > Cheers, > Ali > -- -- simon elliston ball @sireb

Re: [DISCUSS] Metron Parsers in Nifi

g, and this approach will > make that not possible, as another consideration. > > > > On August 13, 2018 at 06:50:09, Simon Elliston Ball ( > si...@simonellistonball.com) wrote: > > Maybe the edge use case will clarify the config issue a little. The reason > I would want t

Re: [DISCUSS] Metron Parsers in Nifi

ook a look at how RecordReader could be leveraged (e.g. > > > > >>> CSVRecordReader), but this is pretty tightly tied into schemas > > > > >>> and is meant > > > > >>> to be used by ControllerServices, which are then used by > > Processors. > > > > >>> There's friction involved there in terms of schemas, but also in > > > > terms of > > > > >>> > > > > >>> access to ZK configs and things like parser chaining. We might > > > > >>> be able to > > > > >>> leverage it, but it seems like it'd be fairly shoehorned in > > > > >>> without getting > > > > >>> the schema and other benefits. > > > > >>> > > > > >>> We won’t have to provide our ‘no schema processors’ ( grok, csv, > > > json > > > > ). > > > > >>> > > > > >>> All the remaining processors DO have schemas that we know about. > We > > > > can > > > > >>> just provide the avro schemas the same way we provide the ES > > > schemas. > > > > >>> > > > > >>> The “parsing” should not be conflated with the transform/stellar > in > > > > >>> NiFi. We should make that separate. Running Stellar over Records > > > > would be > > > > >>> the best thing. > > > > >>> > > > > >>> - This Processor would work similarly to Storm: bytes[] in -> > JSON > > > > >>> out. > > > > >>> - There is a Processor > > > > >>> < > > > > >>> > > > > > > > > > https://github.com/apache/nifi/blob/master/nifi-nar- > bundles/nifi-standard-bundle/nifi-standard-processors/src/ > main/java/org/apache/nifi/processors/standard/JoltTransformJSON.java > > > > >>> > > > > > >>> that > > > > >>> handles loading other JARs that we can model a > > > > >>> MetronParserProcessor off of > > > > >>> that handles classpath/classloader issues (basically just sets > up a > > > > >>> classloader specific to what's being loaded and swaps out the > > > Thread's > > > > >>> loader when it calls to outside resources). > > > > >>> > > > > >>> There should be no reason to load modules outside the NAR. Why do > > > you > > > > >>> expect to? If each Metron Processor equiv of a Metron Storm > Parser > > > is > > > > just > > > > >>> parsing to json it shouldn’t need much.And we could package them > in > > > > the > > > > >>> NAR. I would suggest we have a Processor per Parser to allow for > > > > >>> specialization. It should all be in the nar. > > > > >>> > > > > >>> The Stellar Processor, if you would support the works would > > possibly > > > > need > > > > >>> this. > > > > >>> > > > > >>> 3. Create a MetronZkControllerService to supply our configs to > our > > > > >>> processors. > > > > >>> - This is a pretty established NiFi pattern for being able to > > > provide > > > > >>> access to other services needed by a Processor (e.g. databases or > > > > large > > > > >>> configurations files). > > > > >>> - The same controller service can be used by all Processors to > > > manage > > > > >>> configs in a consistent manner. > > > > >>> > > > > >>> I think controller services would make sense where needed, I’m > just > > > > not > > > > >>> sure what you imagine them being needed for? > > > > >>> > > > > >>> If the user has NiFi, and a Registry etc, are you saying you > > imagine > > > > them > > > > >>> using Metron + ZK to manage configurations? Or to be using BOTH > > > storm > > > > >>> processors and Nifi Processors? > > > > >>> > > > > >>> At that point, we can just NAR our controller service and parser > > > > processor > > > > >>> > > > > >>> up as needed, deploy them to NiFi, and let the user provide a > > config > > > > for > > > > >>> where their custom parsers can be provided (i.e. their parser > jar). > > > > This > > > > >>> would be 3 nars (processor, controller-service, and > > > > controller-service-api > > > > >>> > > > > >>> in order to bind the other two together). > > > > >>> > > > > >>> Once deployed, our ability to use parsers should fit well into > the > > > > >>> standard > > > > >>> NiFi workflow: > > > > >>> > > > > >>> 1. Create a MetronZkControllerService. > > > > >>> 2. Configure the service to point at zookeeper. > > > > >>> 3. Create a MetronParser. > > > > >>> 4. Configure it to use the controller service + parser jar > location > > > + > > > > >>> any other needed configs. > > > > >>> 5. Use the outputs as needed downstream (either writing out to > > Kafka > > > > or > > > > >>> feeding into more MetronParsers, etc.) > > > > >>> > > > > >>> Chaining parsers should ideally become a matter of chaining > > > > MetronParsers > > > > >>> > > > > >>> (and making sure the enveloping configs carry through properly). > > For > > > > >>> parser > > > > >>> aggregation, I'd just avoid it entirely until we know it's needed > > in > > > > NiFi. > > > > >>> > > > > >>> Justin > > > > > > > > --- > > > > Thank you, > > > > > > > > James Sirota > > > > PMC- Apache Metron > > > > jsirota AT apache DOT org > > > > > > > > > > > > > > > > -- -- simon elliston ball @sireb

Knox SSO feature branch PRs: a quick demo

I've recently put in a number of PRs on the Knox feature branch, and thought it might be useful to post a quick 'sprint demo' style explanation of what the various PRs and functionality entails: https://youtu.be/9OJz6hg0N1I Hope this helps with review process. There are a couple of areas where tha

Re: [DISCUSS] Batch Profiler

> [1] > https://lists.apache.org/thread.html/d28d18cc9358f5d9c276c7c304ff4e > e601041fb47bfc97acb6825083@%3Cdev... > > < > https://lists.apache.org/thread.html/d28d18cc9358f5d9c276c7c304ff4e > e601041fb47bfc97acb6825083@%3Cdev.metron.apache.org%3E> > > [2] https://issues.apache.org/jira/browse/METRON-1699 > -- -- simon elliston ball @sireb

Re: Security Feature Branch?

ery well Simon. I am >>> not sure what would be different about your submittal from other >> submittals >>> where that argument failed. >>> >>> On July 12, 2018 at 11:07:02, Simon Elliston Ball ( >>> si...@simonellistonball.com) wrote: >>

Re: Security Feature Branch?

ast on such things is to require that they are broken > into small reviewable chunks on a feature branch, even if the end to end > working version was more ‘usable’. > > > > On July 12, 2018 at 10:51:30, Simon Elliston Ball ( > si...@simonellistonball.com) wrote: > > I'

Security Feature Branch?

I've been doing some work on getting the Metron UIs and REST layers to work with Apache KnoxSSO, and LDAP authentication, to remove the need to store passwords in MySQL, allow AD integration, secure up our authentication points. I'm also working in a Knox service to allow the gateway to provide ful

Re: Performance comparison between Grok and Java regex

A streaming token parser might well get you good performance for that format... maybe something like an antlr grammar or even a simple scanner. Regex is not the only pattern :) It would also be great to see such a parser contributed back to the community of possible, and I sure we would be hap

Re: Architectural reason to split in 4 topologies / impact on the kafka ressources

t was the architectural reason to split the > > >> > ingestion in metron in 4 differents toppologies that all read/write > > to > > >> > kafka? > > >> > > > >> > For example, why the parsing and enrichment topologies have not > been > > >> > merged? Would it not be possible when you parse the message to > > directly > > >> > enricht it? > > >> > > > >> > Im asking that because splitting in several topologies means that > > all of > > >> > the topologies read/write to Kafka, which produce a bigger load on > > the > > >> > kafka cluster and then a need for way more infrastructure/servers. > > The > > >> cost > > >> > is especially true when we speak about TBs of data ingested every > > day. > > >> > > > >> > Im sure there were a very good reason, I was just curious. > > >> > > > >> > Thanks, > > >> > Michel > > >> > > > > > --- > > Thank you, > > > > James Sirota > > PMC- Apache Metron > > jsirota AT apache DOT org > > > > > -- -- simon elliston ball @sireb

Re: Writing enrichment data directly from NiFi with PutHBaseJSON

t;I like the streaming enrichment solutions but it depends on how you are > > getting the data in. If you get the data in a csv file just call the flat > > file loader from a script processor. No special Nifi required. > > > > > >If the enrichments don’t arrive in bulk, the st

Re: Writing enrichment data directly from NiFi with PutHBaseJSON

12 June 2018 20:33 >>> To: dev@metron.apache.org >>> Subject: Re: Writing enrichment data directly from NiFi with PutHBaseJSON >>> >>> I like the streaming enrichment solutions but it depends on how you are >> getting the data in. If you get the data in a csv

Re: Writing enrichment data directly from NiFi with PutHBaseJSON

ct format? > > -or- > > Does this matter-can Metron use the human-readable ROW ids? > > > > Charlie Joynt > > > > -- > > G-RESEARCH believes the information provided herein is reliable. While > > every care has been taken to ensure accuracy, the information is > > furnished to the recipients with no warranty as to the completeness > > and accuracy of its contents and on condition that any errors or > > omissions shall not be made the basis of any claim, demand or cause of > action. > > The information in this email is intended only for the named recipient. > > If you are not the intended recipient please notify us immediately and > > do not copy, distribute or take action based on this e-mail. > > All messages sent to and from this e-mail address will be logged by > > G-RESEARCH and are subject to archival storage, monitoring, review and > > disclosure. > > G-RESEARCH is the trading name of Trenchant Limited, 5th Floor, > > Whittington House, 19-30 Alfred Place, London WC1E 7EA. > > Trenchant Limited is a company registered in England with company > > number 08127121. > > -- > > > -- -- simon elliston ball @sireb

Re: Writing enrichment data directly from NiFi with PutHBaseJSON

>> >> On June 5, 2018 at 14:07:22, Simon Elliston Ball ( >> si...@simonellistonball.com) wrote: >> >> To be honest, I would expect this to be heavily linked to the Metron >> releases, since it's going to use other metron classes and dependencies to >&g

Re: Writing enrichment data directly from NiFi with PutHBaseJSON

out of cycle. > > > > On June 5, 2018 at 13:17:55, Simon Elliston Ball ( > si...@simonellistonball.com) wrote: > > Do you mean in the sense of a separate module, or are you suggesting we go > as far as a sub-project? > > On 5 June 2018 at 10:08, Otto Fowler wrote: > >

Re: Writing enrichment data directly from NiFi with PutHBaseJSON

Do you mean in the sense of a separate module, or are you suggesting we go as far as a sub-project? On 5 June 2018 at 10:08, Otto Fowler wrote: > If we do that, we should have it as a separate component maybe. > > > On June 5, 2018 at 12:42:57, Simon Elliston

Re: Writing enrichment data directly from NiFi with PutHBaseJSON

I'd be in strong support of that, Simon. I think we should have some other > NiFi components in Metron to enable users to interact with our > infrastructure from NiFi (e.g. being able to transform via stellar, etc). > > On Tue, Jun 5, 2018 at 10:32 AM Simon Elliston Ball < > s

Re: [DISCUSS] Field conversions

what version of ES they are running). If I am wrong and there is a >> better approach that works, then we should just revert #1022. >> >> On Tue, Jun 5, 2018 at 9:37 AM, Simon Elliston Ball < >> si...@simonellistonball.com> wrote: >> >>> I would definitely

Re: Writing enrichment data directly from NiFi with PutHBaseJSON

taken to ensure accuracy, the information is > furnished > > to the recipients with no warranty as to the completeness and accuracy of > > its contents and on condition that any errors or omissions shall not be > > made the basis of any claim, demand or cause of action. > > The information in this email is intended only for the named recipient. > > If you are not the intended recipient please notify us immediately and do > > not copy, distribute or take action based on this e-mail. > > All messages sent to and from this e-mail address will be logged by > > G-RESEARCH and are subject to archival storage, monitoring, review and > > disclosure. > > G-RESEARCH is the trading name of Trenchant Limited, 5th Floor, > > Whittington House, 19-30 Alfred Place, London WC1E 7EA. > > Trenchant Limited is a company registered in England with company number > > 08127121. > > -- > > > -- -- simon elliston ball @sireb

Re: [DISCUSS] Field conversions

ld be nice to have a script that read and > > > transformed fields for templates and indices to replace the colons with > > > dots in ES. > > > > > > Simon > > > > > > On 5 June 2018 at 06:40, Casey Stella wrote: > > > &g

Re: [DISCUSS] Field conversions

his kind of migration transformation easily? > > On Tue, Jun 5, 2018 at 9:37 AM Simon Elliston Ball < > si...@simonellistonball.com> wrote: > > > I would definitely agree that the transformation should be removed. We > have > > now however added a complex gener

Re: [DISCUSS] Field conversions

Allow user to change field > > name > > > >> conversion when indexing) only applies to indexing and not querying. > > > The > > > >> others only apply to a single field which does not scale well. Now > we > > > >> have > > > >> an issue with another field in > > > >> https://issues.apache.org/jira/browse/METRON-1600. Rather than > > > >> continuing > > > >> with a patchwork of different fixes I want to attempt to design a > > > >> system-wide solution. > > > >> > > > >> My first thought is to expand > > > https://github.com/apache/metron/pull/1022 > > > >> to > > > >> apply globally. However this is not trivial and would require > > > significant > > > >> changes. It would also make https://github.com/apache/ > > metron/pull/1010 > > > >> obsolete and we might end up having to revert all of it. > > > >> > > > >> Does anyone have any ideas or opinions? I am still researching > > > solutions > > > >> but would love some guidance from the community. > > > >> > > > > > > > > > > -- -- simon elliston ball @sireb

Re: [DISCUSS] parser ES + Solr schema abstraction

nce in the configurations from last to new >>> version >>> -> if there is a difference that effects the ‘schema’ in any configuration >>> -> build master schema from configurations >>> -> version, store, deploy >>> >>> or something.

Re: [DISCUSS] parser ES + Solr schema abstraction

here is a difference that effects the ‘schema’ in any configuration > -> build master schema from configurations > -> version, store, deploy > > or something. I’m sure there are things about clean slate deploy vs. new > version deploy. > > On May 22, 2018 at 09:59:06

Re: [DISCUSS] parser ES + Solr schema abstraction

it to > either ES or Solr. > > Thoughts? > -- -- simon elliston ball @sireb

Re: [DISCUSS] Pcap panel architecture

tegy, or underlying implementation details but these are > items > > we > > > > > > should discuss at some point. > > > > > > > > > > > > On Tue, May 8, 2018 at 5:38 PM, Michael Miklavcic < > > > > > > michael.miklav...@gmail.com> wrote: > &g

Re: [DISCUSS] Release?

tial > performance > > changes in since the last release. I think we might have a justification > > for a release. > > > > Casey > > > -- -- simon elliston ball @sireb

Re: [DISCUSS] Pcap panel architecture

t; > > > > > > > > > > > > > (Youhouuu my first reply on this kind of mail chain^^) > > > > > > > > > > > > > > > > > > > > > > > > > > > > If I may, I would like to share my view on the following 3 > > points. > > > > > > > > > > > > > > - Backend: > > > > > > > > > > > > > > The current metron-api is totally seperate, it will be logic > for > > me > > > > to > > > > > > have > > > > > > > it at the same place as the others rest api. Especially when > > more > > > > > > security > > > > > > > will be added, it will not be needed to do the job twice. > > > > > > > The current implementation send back a pcap object which still > > need > > > > to > > > > > > be > > > > > > > decoded. In the opensoc, the decoding was done with tshard on > > the > > > > > > frontend. > > > > > > > It will be good to have this decoding happening directly on the > > > > backend > > > > > > to > > > > > > > not create a load on frontend. An option will be to install > > tshark > > > on > > > > > > the > > > > > > > rest server and to use to convert the pcap to xml and then to a > > > json > > > > > > that > > > > > > > will be send to the frontend. > > > > > > > > > > > > > > I tried to start directly the map/reduce job to search over all > > the > > > > > pcap > > > > > > > data from the rest server and as Ryan mention it, we had > > trouble. I > > > > > will > > > > > > > try to find back the error. > > > > > > > > > > > > > > Then in the POC, what we tried is to use the pcap_query script > > and > > > > this > > > > > > > work fine. I just modified it that he sends back directly the > > > job_id > > > > of > > > > > > > yarn and not waiting that the job is finished. Then it will > > allow > > > the > > > > > UI > > > > > > > and the rest server to know what the status of the research by > > > > querying > > > > > > the > > > > > > > yarn rest api. This will allow the UI and the rest server to be > > > async > > > > > > > without any blocking phase. What do you think about that? > > > > > > > > > > > > > > > > > > > > > > > > > > > > Having the job submitted directly from the code of the rest > > server > > > > will > > > > > > be > > > > > > > perfect, but it will need a lot of investigation I think (but > > I'm > > > not > > > > > > the > > > > > > > expert so I might be completely wrong ^^). > > > > > > > > > > > > > > We know that the pcap_query scritp work fine so why not calling > > it? > > > > Is > > > > > > it > > > > > > > that bad? (maybe stupid question, but I really don’t see a lot > > of > > > > > > drawback) > > > > > > > > > > > > > > > > > > > > > > > > > > > > - Front end: > > > > > > > > > > > > > > Adding the the pcap search to the alert UI is, I think, the > > easiest > > > > way > > > > > > to > > > > > > > move forward. But indeed, it will then be the “Alert UI and > > > > pcapquery”. > > > > > > > Maybe the name of the UI should just change to something like > > > > > > “Monitoring & > > > > > > > Investigation UI” ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > Is there any roadmap or plan for the different UI? I mean did > > you > > > > > > already > > > > > > > had discussion on how you see the ui evolving with the new > > feature > > > > that > > > > > > > will come in the future? > > > > > > > > > > > > > > > > > > > > > > > > > > > > - Microservices: > > > > > > > > > > > > > > > > > > > > > > > > > > > > What do you mean exactly by microservices? Is it to separate > all > > > the > > > > > > > features in different projects? Or something like having the > > > > different > > > > > > > components in container like kubernet? (again maybe stupid > > > question, > > > > > but > > > > > > I > > > > > > > don’t clearly understand what you mean J ) > > > > > > > > > > > > > > > > > > > > > > > > > > > > Michel > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- -- simon elliston ball @sireb

Re: Streaming Machine Learning use case

from the integration point of view with Metron, so I wanted to > see if anyone had tried SAMOA in practice and especially with Metron use > cases. > > Regards, > Ali > -- -- simon elliston ball @sireb

Re: GeoLite deprecating legacy DBs

Don’t we already use the GeoLite2 database? Mine are all /apps/metron/geo/default/GeoLite2-City.mmdb.gz downloaded from http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz which seems to match the new format page. Am I missing something Jon, or are you referring to the old

Re: [DISCUSS] Time to remove github updates from dev?

I would say we should also update our website with subscription information. Simon > On 4 Apr 2018, at 18:51, Nick Allen wrote: > > https://lists.apache.org/list.html?iss...@metron.apache.org​ > > On Tue, Mar 20, 2018 at 5:06 PM, Otto Fowler > wrote: > >> How about a link? >> >> >> >> On

Re: [DISCUSS] Generic Syslog Parsing capability for parsers

It seems like parser chaining is becomes a hot topic on the repo too with https://github.com/apache/metron/pull/969#partial-pull-merging I would like to discuss the option, and how we might architect, of configuring parsers to ope

Re: [DISCUSS] Time to remove github updates from dev?

Should we not add the new lists to the website? Simon > On 19 Mar 2018, at 14:02, Casey Stella wrote: > > +1 > > > On Mon, Mar 19, 2018 at 8:16 AM Andre wrote: > >> Folks, >> >> All rejoice. This has been finally implemented. >> >> Cheers >> >> On 7 Feb 2018 08:33, "Andre" wrote: >> >>

Re: [DISCUSS] community view/roadmap of threat intel

hing - Given the use of big data technologies it seems to > me Metron should be able to look into past enrichment data in order to > classify traffic. I am not sure this is possible today? > > > Cheers > > > On Mon, Feb 19, 2018 at 8:48 PM, Simon Elliston Ball < > si...

Re: [DISCUSS] community view/roadmap of threat intel

tor Y, we can integrate it with Metron based on integration > points. > > Cheers, > Ali > > On Wed, Feb 14, 2018 at 11:28 PM, Simon Elliston Ball < > si...@simonellistonball.com> wrote: > >> We used to install soltra edge in the old ansible builds (which have >

Re: [DISCUSS] community view/roadmap of threat intel

We used to install soltra edge in the old ansible builds (which have thankfully now been pared back in the interests of stability in full dev). Soltra has not been a good option since they went proprietary, so since then we’ve included opentaxii (BSD 3) as a discovery and aggregator. Most of t

Re: Disable Metron parser output writer entirely

I expect the performance would be dire. If you really wanted to do something like this, a custom writer might make sense. KAFKA_PUT is really meant for debugging use cases only. It’s a very non-stellar construct (non-expression, no return, side-effect dependent…) Also, it creates a producer for

Re: [DISCUSS] Persistence store for user profile settings

t are already very familiar with RDBMS > solutions and have the infrastructure in place to manage those. For users > that don't need HA/DR, just use the DB that gets spun-up with Ambari. > > > > > > On Fri, Feb 2, 2018 at 7:17 AM Simon Elliston Ball < > si

Re: [DISCUSS] Persistence store for user profile settings

ing permissions, grouping and crud around that, and preloading, before > just throwing everything in RDBMS -or- HBASE. > > > > On February 2, 2018 at 08:08:24, Simon Elliston Ball > (si...@simonellistonball.com <mailto:si...@simonellistonball.com>) wrote: > >>

Re: [DISCUSS] Persistence store for user profile settings

vs. personal version in jira. Would RDBMS help > with that? > > > > On February 2, 2018 at 07:17:04, Simon Elliston Ball > (si...@simonellistonball.com <mailto:si...@simonellistonball.com>) wrote: > >> Introducing a RDBMS to the stack seems unnecessary for this. &

Re: [DISCUSS] Persistence store for user profile settings

Introducing a RDBMS to the stack seems unnecessary for this. If we consider the data access patterns for user profiles, we are unlikely to query into them, or indeed do anything other than look them up, or write them out by a username key. To that end, using an ORM to translate a a nested config

Re: When things change in hdfs, how do we know

I take it your service would just be a thin daemon along the lines of the PoC you linked, which makes a lot of sense, delegating the actual notification to the zookeeper bits we already have. That makes sense to me. One other question would be around the availability of that service (which is n

Re: Enrichment and indexing routing mechanism

lementation? I am trying to understand If I > change it in post-parser Stellar, will it be overwritten at the last step > of Parser topology or not? > > Cheers, > Ali > > On Mon, Jan 29, 2018 at 8:55 PM, Simon Elliston Ball < > si...@simonellistonball.com> wrote: >

Re: Enrichment and indexing routing mechanism

Yes, it is. Sent from my iPhone > On 29 Jan 2018, at 09:33, Ali Nazemian wrote: > > Hi All, > > I was wondering how the routing mechanism works in Metron currently. Can > somebody please explain how Enrichment Storm topology understands a single > event is related to which Metron feed? What ab

Re: [DISCUSS] Update Metron Elasticsearch index names to metron_

+1 on this. The idea of a default broad matching template should also include an order entry to avoid conflicts with more specific templates, and we should then document the need for a higher order value in all per-source index templates. In terms of production migration, I think we may want t

Re: Metron User Community Meeting Call

This is going to be a really exciting call. Looking forward to seeing how the GCR Canary sings :) I’m going to volunteer https://hortonworks.zoom.us/my/simonellistonball as a location for the meeting. I would also support the idea of a quick poll on what people are doing with Metron, and mayb

Re: When things change in hdfs, how do we know

13:27, Otto Fowler wrote: > > https://github.com/ottobackwards/hdfs-inotify-zookeeper > <https://github.com/ottobackwards/hdfs-inotify-zookeeper> > > Working on a poc > > > > On January 26, 2018 at 07:41:44, Simon Elliston Ball > (si...@simonellistonball.

Re: When things change in hdfs, how do we know

Should we consider using the Inotify interface to trigger reconfiguration, in same way we trigger config changes in curator? We also need to fix caching and lifecycle in the Grok parser to make the zookeeper changes propagate pattern changes while we’re at it. Simon > On 26 Jan 2018, at 03:16

Re: Metron nested object

ar your thoughts > > > Cheers > > > > [1] I appreciate the architecture is flexible... > [-] Apologies for the delay but I suspect my previous message got stuck in > moderation > > On Fri, Dec 22, 2017 at 3:59 AM, Simon Elliston Ball < > si...@simonellistonball.

Re: [DISCUSS] Generating and Interacting with serialized summary objects

There is some really cool stuff happening here, if only I’d been allowed to see the lists over Christmas... :) A few thoughts... I like Otto’s generalisation of the problem to include specific local stellar objects in a cache loaded from a store (HDFS seems a natural, but not only place, maybe

Re: Metron nested object

Correct, nested objects in lucene indexes lead to sub-documents, which leads to a massive drop in ingest and query rates, this is why the JSONMap parser for example deliberately flattens the Metorn JSON object. Before this decision was made, very early versions of OpenSOC nested enrichments for

Re: Metron - Emailing Alerts

ant to discuss and flesh out > > Thanks, > James > > 13.12.2017, 14:26, "Simon Elliston Ball" : >> We can already do that with profiles I would have thought. Create a profile >> that only picks alerts and then base your emails only from the alert events >>

Re: Metron - Emailing Alerts

s is probably a feature worthy of > consideration for Metron. > > 13.12.2017, 12:19, "Simon Elliston Ball" : >> Metron generates alerts onto a Kafka queue, which can be used to integrate >> with Alert management tools, usually some sort of existing alert aggregation

Re: Metron - Emailing Alerts

Metron generates alerts onto a Kafka queue, which can be used to integrate with Alert management tools, usually some sort of existing alert aggregation tool. An alternative approach common with this is to have a tool like Apache NiFi attach to the Metron alert feed and send email. The solution

Re: [DISCUSS] Community Meetings

nity meeting itself - this gives > others in other timezones and commitments review and voice in the decisions. > > If it didn't happen on the mailing lists then it didn't happen. :) > > > On Tue, Dec 12, 2017 at 1:39 PM, Simon Elliston Ball < > si...@simonellis

Re: [DISCUSS] Community Meetings

Yes, I do. I suspect the best bet will be to post recordings somewhere on the apache.org <http://apache.org/> metron site. Simon > On 12 Dec 2017, at 18:36, Otto Fowler wrote: > > Excellent, do you have the > 40 min + record option? > > > On December 12, 2017

Re: [DISCUSS] Community Meetings

Happy to volunteer a zoom room. That seems to have worked for most in the past. Simon > On 12 Dec 2017, at 18:09, Otto Fowler wrote: > > Thanks! I think I’d like something hosted though. > > > On December 12, 2017 at 11:18:52, Ahmed Shah (ahmeds...@cmail.carleton.ca) > wrote: > > Hello, >

Re: Wiki Docs links seem wrong

Awesome, many thanks! > On 7 Dec 2017, at 13:08, Kyle Richardson wrote: > > Fixed. > > -Kyle > > On Thu, Dec 7, 2017 at 7:20 AM, Simon Elliston Ball < > si...@simonellistonball.com> wrote: > >> https://cwiki.apache.org/confluence/display/METRON/ &g

Wiki Docs links seem wrong

https://cwiki.apache.org/confluence/display/METRON/Metron+User+Guide+-+per+release The links don’t seem to correspond to the versions on this page. Would be happy to fix, but I don’t have wiki perms. Simon

Re: DISCUSS: Quick change to parser config

] >}, > { > "transformation": "COMPLETE", > "output" : [ "ip_src_addr", "ip_dst_addr", "message"] >} > ] > } > > I think having these two treated separately makes sense because sometime

Re: DISCUSS: Quick change to parser config

uot;: ["ip_src_addr", "ip_dst_addr"], >> "config": { >> "ip_src_addr": "ipSrc", >> "ip_dest_addr": "ipDst" >> } , >> { >> "transformation": "STELLAR", >> “operatio

Re: DISCUSS: Quick change to parser config

quot; > } , > { > "transformation": "STELLAR", > “operation": “SomeOtherThing", > "output": [“foo", “bar"], > "config": { > “foo": “TO_UPPER(foo)", > “bar": “TO_LOWER(bar)" > } > } > ] &

DISCUSS: Quick change to parser config

I’m looking at the way parser config works, and transformation of field from their native names in, for example the ASA or CEF parsers, into a standard data model. At the moment I would do something like this: assuming I have fields [ipSrc, ipDst, pointlessExtraStuff, message] I might have:

Re: [DISCUSS] NPM / Node Problems

> > On November 27, 2017 at 07:02:51, Simon Elliston Ball > (si...@simonellistonball.com <mailto:si...@simonellistonball.com>) wrote: > >> Thinking about this, doesn’t our build plugin explicitly install it’s own >> node? So actually all the node version thing

Re: [DISCUSS] NPM / Node Problems

rely suggest a min version required to build UI successfully. > > -Raghu > > > > On Fri, Nov 24, 2017 at 10:21 PM, Simon Elliston Ball > wrote: >> Agreeing with Nick, it seems like the main reason people are building >> themselves, and hitting all these env

Re: Using Storm Resource Aware Scheduler

rent > parsers. > > I will create a Jira ticket to add an ability in UI to tune Metron parser > feeds at Storm level. Right now it is a little hard to maintain tuning > configurations per each parser, and as soon as somebody restarts them from > Management-UI/Ambari, it will be o

Re: [DISCUSS] NPM / Node Problems

Agreeing with Nick, it seems like the main reason people are building themselves, and hitting all these environmental issues, is that we do not as a project produce binary release artefacts (the rpms which users could just install) and instead leave that for the commercial distributors to do.

Re: Using Storm Resource Aware Scheduler

Implementing the resource aware scheduler would be decidedly non-trivial. Every topology will need additional configuration to tune for things like memory sizes, which is not going to buy you much change. So, at the micro-tuning level of parser this doesn’t make a lot of sense. However, it may

Re: analytics exchange platform

The analytics exchange concept is not really part of Apache Metron, but some commercial offerings include it. In terms of Metron itself, are you maybe thinking about Model as a Service: http://metron.apache.org/current-book/metron-analytics/metron-maas-service/index.html

  1   2   >