Re: Metron's Future / Alternatives

ercepted, corrupted, >> lost, destroyed, arrive late or be incomplete. If you receive a suspicious >> or unexpected email from us, or purporting to have been sent on our behalf, >> particularly containing different bank details, please do not reply to the >> email, click on any links,

Re: How to customise the streams visible in metron ui

; Hema > -- -- simon elliston ball @sireb

Re: 3rd party stellar functions

tron-stellar/stellar-common/3rdPartyStellar.html > , and it just doesn’t work. I changed the Global configs, uploaded the > “jar” file to HDFS, and then ran the stellar environment to test it. No > errors at all, but the function isn’t even listed when I do “%functions”. > > > >

Re: Having more than one use case on a Metron instance

that user A can only access > Alerts UI A, user B can only access Alerts UI B? > > Thanks again for your input, it is very much appreciated. > > On Wed, Feb 19, 2020 at 5:59 PM Simon Elliston Ball < > si...@simonellistonball.com> wrote: > >> I would suggest using

Re: Having more than one use case on a Metron instance

lysts could use different screens for the different use cases. > > Is there any way to configure this? And if not, does anyone in the > community have suggestions on how to approach this? > > Thanks in advance for the help. > -- -- simon elliston ball @sireb

Re: Using something other than colons in field names?

has anyone figured out a way to escape colons in their > query or another work around in general? Is there a setting somewhere that > can be used to change the default from a colon to a period or another > character? > > > > > > > > Thank you, > > > > > > > > Tom. > > > > > > > > > > -- -- simon elliston ball @sireb

Re: Mysterious Metron UI screenshot

a times > (screens are dated 2016). Is it Metron Management UI + Kibana iframes? > Can anyone shed more light on how these screens were created? > > Thank you. > > p.s. Can you please invite me to the slack channel? > > - Dima > > -- -- simon elliston ball @sireb

Re: How can i send batch of data to MaaS

December 10, 2019 at 09:39:27, Hema malini (nhemamalin...@gmail.com) >> wrote: >> >> Hi, >> >> Is there any way to pass a batch of data to Metron MaaS. We have some >> models like LSTM, which requires data to be aggregated and passed to model >> .Can you please suggest whether is it possible. >> >> Thanks and Regards, >> Hema >> >> -- -- simon elliston ball @sireb

Re: ingesting syslog and asa log into metron

out and > add grok patterns based on the log messages. > > > > On Mon, 25 Nov, 2019, 7:44 PM Simon Elliston Ball, < > si...@simonellistonball.com> wrote: > >> Use the nifi listen syslog processor to push Asa logs into a Kafka topic, >> then the metron as

Re: ingesting syslog and asa log into metron

and for now, I went to ask how can I ingest Syslog and asa log into apache > metron using nifi? > -- -- simon elliston ball @sireb

Re: Score not being issued by ThreatIntel Enrichment

"riskLevelRules": [ > >{ > > > "name": "All_threat", > > > "comment": "", > > > "rule": "ip_src_addr == ‘8.8.8.8’ ", > > > "reason": null, > > > "score": "5" > >} > >], > >"aggregator": "MAX", > >"aggregationConfig": {} > >} > > }, > > "configuration": {} > > } > > > > > > > > Appreciate any help. > > Thanks > -- -- simon elliston ball @sireb

Re: Enable optional fields in csv parser

6, Hema malini (nhemamalin...@gmail.com) >> wrote: >> >> Hi all, >> >> Is there any way to mark some columns as optional in column mapping in >> CSV parser. >> >> Thanks and Regards, >> Hema >> >> -- -- simon elliston ball @sireb

Re: CSV parser

Perhaps you could post your config? You should have a dictionary in it called columns which maps column name to index. Simon On Tue, 12 Nov 2019 at 16:05, Hema malini wrote: > Yes. I uploaded as mentioned in the document. > > On Tue, 12 Nov, 2019, 9:31 PM Simon Elliston Bal

Re: CSV parser

> getting still column metadata not defined.please let me know what I am > missing in this. > > On Tue, 12 Nov, 2019, 9:23 PM Simon Elliston Ball, < > si...@simonellistonball.com> wrote: > >> You modify the column data in the parser config. I suggest checking the >>

Re: Metron parser for firewall

rns. > > Thanks and Regards, > Hema > > On Fri, 8 Nov, 2019, 8:32 PM Simon Elliston Ball, < > si...@simonellistonball.com> wrote: > >> There is a Cisco ASA parser built into metron. I suggest using that. >> >> Simon >> >> On Fri, 8 Nov 2019 at 04:50, H

Re: Apache Metron production deployment

I would recommend against using the AWS deploy method on the github. It’s not really that well maintained, and the Ambari method is definitely the preferred at present, but then I tend to use a distro to install, or full dev if it’s just for local testing. Simon > On 29 Oct 2019, at 13:35,

Re: Threat Intel hailataxii

Looks to me like your discovery server is not working properly, hence the failure message. This could be a temporary connectivity issue, but if it’s repeatable I would look into your opentaxii config. Simon > On 29 Oct 2019, at 13:23, Thiago Rahal Disposti > wrote: > >  > Anyone knows

Re: Apache Metron production deployment

ge OS. > > > Thanks alot in advance! > > Best Regards > Marcus > -- -- simon elliston ball @sireb

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

: > If anyone can think of the things that need to be backed up, please > comment the jira. > > > > > On August 27, 2019 at 17:07:20, Otto Fowler (ottobackwa...@gmail.com) > wrote: > > Good idea METRON–2239 [blocker]. > > > > On August 27, 20

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

You could always submit a Jira :) On Tue, 27 Aug 2019 at 21:27, Otto Fowler wrote: > You are right, that is much better than backup_metron_configs.sh. > > > > > On August 27, 2019 at 16:05:38, Simon Elliston Ball ( > si...@simonellistonball.com) wrote: > > You can d

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

rsion running on HDP >>> 3.x. If there is any discrepancy between the two or additional settings >>> will be required, those will be documented in the release notes. From the >>> Metron perspective, this upgrade would be no different than simply >>> upgrading to the n

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

lease? If we do need 1.x, do we still > see upgrades as a gating function? The main issue is that this has the > potential to drag out the upgrade and further couple it with other > features. And with Storm 1.x being eol'ed, I'm not sure this is something > we can wait much longer for. I'll think on this and send out my own > thoughts once folks have had a chance to review. > > Best, > Mike Miklavcic > Apache Metron, PMC, committer > > > -- -- simon elliston ball @sireb

Re: Profiler Examples Not working

624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.NullPointerException > at > org.apache.metron.profiler.repl.ProfilerFunctions$ProfilerApply.apply(ProfilerFunctions.java:140) > at > org.apache.metron.stellar.common.StellarCompiler.lambda$exitTransformationFunc$13(StellarCompiler.java:664) > at > org.apache.metron.stellar.common.StellarCompiler$Expression.apply(StellarCompiler.java:259) > at > org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:151) > ... 7 more > > > -- > *Best Regards* > Farrukh Naveed Anjum > *M:* +92 321 5083954 (WhatsApp Enabled) > *W:* https://www.farrukh.cc/ > -- -- simon elliston ball @sireb

Re: [ask] create profile for profiler with multiple fields on foreach

t; > "profiles": [ > > { > > "profile": "hello-world", > > "onlyif": "exists(ip_src_addr) AND exists(ip_dst_addr)", > > "foreach": "ip_src_addr AND ip_dst_addr, > > "init":{ "count": "0" }, > > "update": { "count": "count + 1" }, > > "result": "count" > > } > > ] > > } > > -- -- simon elliston ball @sireb

Re: batch indexing in JSON format

Most users will have a batch process converting the JSON short term output into ORC or Parquet files, often adding them to hive tables at the same time. I usually do this with a spark job run every hour, or even every 15mins or less in some cases for high throughput environments. Anecdotally,

Re: flatfile_summarizer

n(SimpleFlatFileSummarizer.java:38) >> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >> at java.lang.reflect.Method.invoke(Method.java:498) >> >> at org.apache.hadoop.util.RunJar.run(RunJar.java:233) >> >> at org.apache.hadoop.util.RunJar.main(RunJar.java:148) >> >> >> >> Am I doing something wrong? Also, is there a better alternative to the >> “CSV” extractor? I’m ideally looking to load the entire line, regardless >> of any specific characters (regex may contain commas for example). >> >> >> >> Thanks in advance, >> >> David Auclair >> >> >> > -- -- simon elliston ball @sireb

Re: TASK [bro : Download bro] - fatal: [node1]: FAILED!

s C++11 compliant >> -- >> Linux ub1604in2017 4.4.0-148-generic #174-Ubuntu SMP Tue May 7 12:20:14 >> UTC 2019 x86_64 x86_64 x86_64 GNU/Linux >> -- >> Total System Memory = 15994.3 MB >> Processor Model: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz >> Processor Speed: 3361.617 MHz >> Processor Speed: 3448.476 MHz >> Processor Speed: 3323.898 MHz >> Processor Speed: 3400.664 MHz >> Processor Speed: 3359.625 MHz >> Processor Speed: 3283.125 MHz >> Processor Speed: 3382.335 MHz >> Processor Speed: 3275.023 MHz >> Total Physical Processors: 8 >> Total cores: 32 >> Disk information: >> /dev/sdb195G 52G 38G 58% / > > > > *Best regards!* > Pablo de Azevedo > -- -- simon elliston ball @sireb

Re: [ask] detect unsual login duration

ll give some alert > to us. > > If this possible, how to do that? > Pls help. > > > Best Regards, > > Tkg_cangkul > > -- -- simon elliston ball @sireb

Re: Issue when trying to load JSON

6.5.1050-37.jar:1.1.0.2.6.5.1050-37] > > at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] > > at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112] > > > > > > How can I debug this? > > > > Thanks > > > > Stéphane > > _ > > Ce message et ses pieces jointes peuvent contenir des informations > confidentielles ou privilegiees et ne doivent donc > pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu > ce message par erreur, veuillez le signaler > a l'expediteur et le detruire ainsi que les pieces jointes. Les messages > electroniques etant susceptibles d'alteration, > Orange decline toute responsabilite si ce message a ete altere, deforme ou > falsifie. Merci. > > This message and its attachments may contain confidential or privileged > information that may be protected by law; > they should not be distributed, used or copied without authorisation. > If you have received this email in error, please notify the sender and delete > this message and its attachments. > As emails may be altered, Orange is not liable for messages that have been > modified, changed or falsified. > Thank you. > > -- -- simon elliston ball @sireb

Re: Question about "parser_invalid"

Timestamp in Metron is always a unix epoch to avoid things like timezone issues. In this case, you can resolve this using a field transformation at the parsing stage, with the TO_EPOCH_TIMESTAMP function. Some custom parsers already do this, but for those that don’t, a simple bit of stellar

Re: Metron concept

vez recu > ce message par erreur, veuillez le signaler > a l'expediteur et le detruire ainsi que les pieces jointes. Les messages > electroniques etant susceptibles d'alteration, > Orange decline toute responsabilite si ce message a ete altere, d

Re: Metron concept

alsifie. Merci. > > This message and its attachments may contain confidential or privileged > information that may be protected by law; > they should not be distributed, used or copied without authorisation. > If you have received this email in error, please notify the sender and delete > this message and its attachments. > As emails may be altered, Orange is not liable for messages that have been > modified, changed or falsified. > Thank you. > > -- -- simon elliston ball @sireb

Re: Metron-REST is always stopping

Did you check to see if it was listening? Sometimes this can misreport in ambari if you have an incorrect version of the python requests library installed. Simon > On 4 Apr 2019, at 22:47, > wrote: > > Hello all, > > I’ve installed Metron last week and everything was working correctly.

Re: Metron logs for parser

Metron runs as a series of storm topologies, so you find the logs in storm. Your best route to get there is via the storm ui from ambari. Simon > On 25 Mar 2019, at 23:53, Meenakshi.S > wrote: > > Hi , > > I installed Metron using Ambari Server . > I was able to see the logs for

Re: Use case question

we will have to build our models etc. but given that all >> the heavy lifting is already done, I'm tempted to try Metron for this use >> case (instead of re-inventing the wheel). >> >> Is this possible/recommended? Or would you recommend using Metron >> strictly for network related analysis? >> >> Best Regards, >> Sanket >> > -- -- simon elliston ball @sireb

Re: Help regarding Parser Configuration

You might like to look into parser chaining for this: https://metron.apache.org/current-book/metron-platform/metron-parsers/ParserChaining.html Simon > On 20 Feb 2019, at 16:47, Farrukh Naveed Anjum > wrote: > > Yes, I am using BRO Parser, Can I sub divide the message field > >> On Wed,

Re: Graphs based on Metron or PCAP data

Graph enables a number of interesting use cases, and it really depends on what you’re after as to which tech makes sense. Spark graphx is a strong contender for analytics of things like betweenness and community linkage on HDFS indexed data. That would tend to be batch and through something

Re: Raw Message Strategy "Envelope"

e chained parser and another identical for the direct ingestion) or 2 > Kafka topics (1 for the syslog parser, 1 for both, the enveloped/chained > source and the "default" source). > > Appreciate your thoughts and comments. > > Best, > Stefan > -- > Stefan Kupstaitis-Dunkler > https://datahovel.com/ > https://www.meetup.com/Hadoop-User-Group-Vienna/ > https://twitter.com/StefanDunkler > -- -- simon elliston ball @sireb

Re: Error deploying Metron 0.3.1 single Node

Are you looking to install a dev build? If not and you just want to use the system, you may be better off with a pre-built distribution. Simon > On 30 Nov 2018, at 12:48, Babak Abbaschian wrote: > > It’s two weeks that I’m trying to install metron 0.6.1, but I end up with an > error with

Re: Issue with BasicIseParser

eer > +919447946359 > irshadkt@gmail.com > Skype : muhammed.irshad.k.t > -- -- simon elliston ball @sireb

Re: https access to Metron Alert UI

Metron doesn’t fully support this yet out of the box, but you can hack it up by changing the templates for the spring yaml config. More commonly, put it behind a reverse proxy for the ssl. There was talk about integrating that with Knox for ssl proxying, but that’s on pause now. Simon Sent

Re: Metron Not Reading From Kafka?

are not the intended recipient, you should delete this message. Any > disclosure, copying, or distribution of this message, or the taking of any > action based on it, is strictly prohibited. > -- -- simon elliston ball @sireb

Re: CEF Parser not Indexing data via Nifi (SysLogs)

What you need to do is NOT ParseCEF in NiFi. Metron should handle be CEF parsing. Just use NiFi to do the listen syslog (no need to parse in NiFi) then SplitText to get one line of CEF per kafka message (if your syslog is batching, this may not be necessary. Set up a sensor in Metron using

Re: How to delete the original message field once the message parsed?

res are not being used, or don’t exist yet (replay) can someone not tune > them down for their scenario with some understanding of the tradeoffs? > > I don’t think there is currently a way to do this, but it is worth having a > discussion on the issue. > > >> On June 25, 2018

Re: Alerts Not Being Generated?

Hi David, One quick thing just in case, is_alert, not is_alarm. That said that should not affect what’s in the alerts ui. You should see data from your geo source as well (whatever you called it). It is possible there may be a problem with your elastic template. You might be interested in

Re: Best Metron version for development

The full dev platform may be the easiest to test things like that on. It can be a little brittle if you’re running it in limited RAM, but it also has things like the sensor-stubs, which provides an easy means to fake up some input traffic. That may be useful for your development and testing.

Re: Error when trying to install Apache Metron CentOS7

To be honest, rather than messing about with grub for this, I would follow the alternative route outlines in the wiki page. To be even more honest, I wouldn’t use that method from the wiki and would probably go with something like the full dev VM platform if you’re looking to do development

Re: Stellar post-parsing transformation conditional statement

You either want a MAP_GET in your IF or a match statement in there I expect. See the match statement at https://github.com/apache/metron/blob/master/metron-stellar/stellar-common/README.md under core functions (it’s relatively new) Simon Sent from my iPhone > On 9 Feb 2018, at 03:55, Ali

Re: CentOS and Ubuntu

Not particularly. The centos builds seem to be used by more people on dev, probably because they’ve been around for longer, and so are arguably more tested. The area where it’s most likely to be relevant is in the install of repos for ES and potentially the fastcapa pcap probe (don’t quote me

Re: elasticsearch template question.

Hi Laurens, In Metron all fields tend to get flattened into an un-nested structure of keys and values. Some of the keys do represent a flattened tree structure (for example our standard enrichment fields). The reason for this is essentially ingest speed for nested documents in lucene based

Re: Define a function that can be used in Stellar

dy been implemented in Metron that has a > config file associated with it? I am trying to get an idea of how it works. > > On 3 Feb. 2018 00:44, "Simon Elliston Ball" <si...@simonellistonball.com > <mailto:si...@simonellistonball.com>> wrote: > Depends how you write t

Re: Apache Metron functions implementation

Hi Helder, It is very much possible, and very easy to create your own functions and models on top of Metron. There are two main ways in which you would do this, depending on the type of use case you’re looking at. Metron uses a language called Stellar as part of the enrichment stage (and

Re: Define a function that can be used in Stellar

n.apache.org>> > Subject: Re: Define a function that can be used in Stellar > > > > > > > > If something we have already does not fit the bill, I would recommend > creating that function in Java. Since you described it as "a bit complex"

Re: HBase enrichment vs Stellar enrichment for HBase look up

There shouldn’t be. Both run through the same kind of bolt-side caching, so you should be able to use the Stellar version, and in fact that’s the general direction the project is heading. We haven’t quite deprecated the plain HBase Bolt… but Stellar is definitely the preferred option. Simon

Re: Indexing Bolt Error

Yes, configure your indexing. https://metron.apache.org/current-book/metron-platform/metron-indexing/index.html Note it’s a warning, not an error, that default values are being used because you do not have a

Re: Some Metron Alerts UI questions

Hi Laurens, A few quick answers inline… Simon > On 20 Jan 2018, at 00:37, Laurens Vets wrote: > > Hi list, > > I have some general Alerts UI questions/comments/remarks, I hope you don't > mind :) I'm using the UI that's part of Metron 0.4.2. These apply to my > specific

Re: SysLog using CEF Parser (RSysLogs)

Are there any errors in the logs for the indexing bolt? I would expect the errors are probably at the elastic ingest point, and probably caused by an incorrect elastic template for the CEF data. Simon > On 22 Jan 2018, at 08:24, Farrukh Naveed Anjum > wrote: > >

Re: Define a function that can be used in Stellar

; >>> Yeah, I agree. It will be much easier to define functions on the fly and >>> use them afterwards. It could be defined as Lambda or custom function. >>> >>> Regards, >>> Ali >>> >>> >>> >>>> On Wed

Re: Define a function that can be used in Stellar

there any example regarding > adding a Stellar function in Java? Hopefully, we don't need to rebuild the > corresponding modules for this? > > Cheers, > Ali > > On Wed, Jan 17, 2018 at 8:40 PM, Simon Elliston Ball > <si...@simonellistonball.com <mailto:si...@simo

Re: Metron Reference Application (Profiling Your Streams Fails)

Looks like a docs typo on the wiki: What you need is CONFIG_PUT(“PROFILER”, profilerConfig) Simon > On 15 Jan 2018, at 10:45, Farrukh Naveed Anjum > wrote: > > Can you help on this ? > > On Mon, Jan 15, 2018 at 3:42 PM, Farrukh Naveed Anjum >

Re: Metron Rest Kerberos -- Kafka topic ACL

The ansible roles and playbooks included with Metron install Ambari to handle the setup of the Metron and the Hadoop, Kafka etc. components, so yes. > On 10 Jan 2018, at 03:18, varsha mordi wrote: > > Can Ambari UI work with Ansible? > > On Wed, Jan 10, 2018 at

Re: Metron Version

Are the logs you’re sending with syslog in CEF format? You will note that the CEF sensor uses the CEF parser, which means unless your logs are in CEF format, they will fail to parse and be dropped into the error index (worth checking the error index in kibana via the Metron Error Dashboard.

Re: metron vs ossec

In many ways it’s a matter of scale. OSSIM is a kind of lite version of AlienVault, and used by them. I’ve seen people move from an OSSIM architecture to Metron specifically to get better scaling, things like PCAP capabilities etc. but also retain the OSSEC agents to handle endpoint and

Re: machine learning libraries supported

gt; Is there a performance problem or how would you justify that phrase? > > thanks > > Le 07/12/2017 à 13:55, Simon Elliston Ball a écrit : >> I would recommend starting out with something like Spark, but the short >> answer is that anything that will run inside a yarn cont

Re: machine learning libraries supported

I would recommend starting out with something like Spark, but the short answer is that anything that will run inside a yarn container, so the answer is most ML libraries. Using Spark to train models on the historical store is a good bet, and then using the trained models with model as a

Re: Basic analysis

> The issue is the requirement for people on the user list to go to the source. > > > On December 6, 2017 at 09:16:39, Simon Elliston Ball > (si...@simonellistonball.com <mailto:si...@simonellistonball.com>) wrote: > >> No problem, I’ll grant you it’s not in the most i

Re: Basic analysis

providers I believe to add more samples of both dashboards and use cases. Simon > On 6 Dec 2017, at 14:12, Otto Fowler <ottobackwa...@gmail.com> wrote: > > Thanks Simon > > > On December 6, 2017 at 09:11:50, Simon Elliston Ball > (si...@simonellistonball.com <mail

Re: Basic analysis

<https://cwiki.apache.org/confluence/display/METRON/Enhancing+Metron+Dashboard> Should at least get us started. Simon > On 6 Dec 2017, at 14:00, Otto Fowler <ottobackwa...@gmail.com> wrote: > > Links? > > > On December 6, 2017 at 08:18:23, Simon Elliston Ball >

Re: Basic analysis

Yes. Consider a zeppelin notebook, or kibana dashboard for this. If you want to use these values for detection, consider building a profile based on the stats objects (see the profiler section of the documentation under analytics. Simon > On 6 Dec 2017, at 07:42, Syed Hammad Tahir

Re: Not able to run metron.

Otto Fowler (ottobackwa...@gmail.com > <mailto:ottobackwa...@gmail.com>) wrote: > >> I build on mac, and have : >> >> -- >> node >> v6.10.2 >> -- >> npm >> 3.10.10 >> >> for my node versions. >> >> >&

Re: Not able to run metron.

Sorry, you’re right, you do need ansible. Make sure the version is EXACTLY the version in the docs. Simon > On 22 Nov 2017, at 13:03, Otto Fowler wrote: > > You DO need ansible for full_dev deployment. > You do need Docker installed and running > > > > On November

Re: Not able to run metron.

You shouldn’t need ansible for the full-dev build, but you will need maven, docker and an up-to-date nodejs and npm package to do the actual build. I would recommend against using the OS provided nodejs and go with the packages from nodesource instead. The full-dev build is also the best

Re: Snort enrichment issue

Did you setup and load the geo enrichment database? https://metron.apache.org/current-book/metron-platform/metron-data-management/index.html#GeoLite2_Loader Also, we can’t really see the

Re: Kibana Error

ambari > after changing heapsize. Now doing it via console > > > > On Wed, Oct 25, 2017 at 5:13 PM, Simon Elliston Ball > <si...@simonellistonball.com <mailto:si...@simonellistonball.com>> wrote: > That just shows running, not health. The problem is that it is not

Re: Kibana Error

s this > > > > On Wed, Oct 25, 2017 at 5:07 PM, Simon Elliston Ball > <si...@simonellistonball.com <mailto:si...@simonellistonball.com>> wrote: > Did you check the elastic service was running and healthy with the health > checks. Try a few of the quick links

Re: Kibana Error

t; > > > On Wed, Oct 25, 2017 at 3:45 PM, Simon Elliston Ball > <si...@simonellistonball.com <mailto:si...@simonellistonball.com>> wrote: > I strongly suggest you spend some time learning about elastic search and some > of the basic components. This is not a bug, it’

Re: Kibana Error

cs16...@itu.edu.pk> wrote: >>> SHould I do it from here? If yes then please guide me how to >>> >>> >>> >>>> On Wed, Oct 25, 2017 at 1:17 PM, Simon Elliston Ball >>>> <si...@simonellistonball.com> wrote: >>>> Your elastic search instance

Re: SysLog Parser in Metron

Short answer: grok parsers. Longer answer: syslog is more a transport, not just a log format, so it encapsulates a wide variety of data sources. Your best bet is probably to use NiFi to listen for syslog from a remote host (ListenSyslog) and then route each application in the syslog to a

Re: multiple pattern grok parser in 1 file

the issue might be that your patterns flat out don’t match the logs. Simon > On 23 Oct 2017, at 10:36, tkg_cangkul <yuza.ras...@gmail.com> wrote: > > Hi Simon, > > I've tried your suggestion but i have an error msg like below : > > > > On 23/10/17

Re: multiple pattern grok parser in 1 file

That is not valid grok. Pattern names should be unique in the grok. What you probably mean is something like: AUTHLOG1 %{NUMBER:timestamp} %{SYSLOGHOST:syslog_host} %{DATA:syslog_program}(?:\[%{POSINT}\])?: %{WORD:login} password for %{USERNAME:username} from %{IP:ip} %{GREEDYDATA} AUTHLOG2

Re: event correlation on metron

; that’s what i mean. > what sensor that i need if i want to do this case? > especially when i wanna parse some host logs into metron enrichment and > indexing > >> On Wed, 18 Oct 2017 at 01.03 Simon Elliston Ball >> <si...@simonellistonball.com> wrote: >> W

Re: event correlation on metron

What you want to do in this setting is just TailFile, the just push to Kafka. The grok piece is more efficiently handled in the Metron grok parser. Push to a kafka topic named for your sensor, then setup a sensor (a parser topology to do the grok parsing and any transformation you need). Each

Re: Metron Error in Barematel Installation

id some dependency got updated... > and its is breaking it. > > On Mon, Oct 16, 2017 at 4:25 PM, Simon Elliston Ball > <si...@simonellistonball.com <mailto:si...@simonellistonball.com>> wrote: > This looks like an error in the frontend build. Sometimes this is transi

Re: Metron Error in Barematel Installation

This looks like an error in the frontend build. Sometimes this is transient (problems downloading npm packages) so a retry may help. However, we really should be looking at pinning the dependency versions, as this can also be caused by third-party npm packages being updated in the wild and

Re: Initial Testing

Syed, I would strongly suggest you go through the Squid based tutorial to get an idea of how enrichment and indexing works. See: https://cwiki.apache.org/confluence/display/METRON/Metron+Reference+Application >

Re: Metron Alerts UI, no alerts

Right now, you can't. I believe we should be taking the lost of index prefixes we use in the ui from the index config via the rest api, we can pull the names from each sensor index config and use that as the prefix in the ui. That way we pickup any new index automatically. Simon > On 28 Sep

Re: Not seeing any Metron alerts.

yways in the event after I refreshed the fields in > Kibana right? > > On 2017-09-26 09:16, Simon Elliston Ball wrote: > >> There should be, though you may need to update your templates in ES if >> you've got any custom templates there, and make sure you refre

Re: Not seeing any Metron alerts.

the _score field is actually an elastic search matching score field, and is not relevant to metron. You should see the scores in the threat:triage:score field. However, your rules will only be run if the telemetry has is_alert set true, so you should ensure that the enrichment phase sets

Re: Unable to add the hosts

The list says it wants one host per line, you have given it comma separated. > On 25 Sep 2017, at 09:31, kotipalli venkatesh > wrote: > > > Hi All, > > Please help on the below error, Target host, we added nodes and import the > id_rsa file on the main node.

Re: 192.168.138.158 address in yaf index

That sounds like an address from the standard example.pcap used to demo metron capability. In a real deployment you should not run pcap-replay which is what inserts this demo data. Simon > On 21 Sep 2017, at 00:29, Frank Horsfall > wrote: > > Morning all, >

Re: Clearing of data to start over

Multiple Kafka brokers will help a lot. The wizard allows you too add more by using the plus symbol next to Kafka on the master selection screen. After the fact you can add more with the add service button on the hosts screen in ambari. When adding brokers, don't forget to also alter your

Re: Threat triage rules using stellar geo enrichment

A much better way of doing this is to run the geo enrichment as part of the regular enrichment process and then just use the output field for the rule. Your config already does this, so your rule is in effect running the same enrichment twice. Just use enrichments.geo.ip_dst_addr.country !=

Re: How to change Elasticsearch indexing policy

You could change the index data format. One word of caution here though; the last time I saw this done it caused huge problems with locking on ingest against people running queries on the current day’s data and tended to knock recent relevant indexes out of disk cache at the OS level. It might

Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase Instance

kafka.start=WHERE_I_LEFT_OFF > > > > Regards, > > Balakrishna > > > From: Simon Elliston Ball [mailto:si...@simonellistonball.com] > Sent: Wednesday, July 12, 2017 3:28 PM > To: user@metron.apache.org > Subject: Re: Metron Profiler 0.3.0: HbaseBol

Re: Metron Profiler 0.3.0: HbaseBolt not storing data to HBase Instance

Looks like you’ve set the profile to purge (expires) every 30 ms, and your period is set to 30 minutes, so the data is being expired long before it has a change to write. Simon > On 12 Jul 2017, at 06:17, Krishna Dhanekula > wrote: > > I have an problem where

Re: Metron in-memory enrichment

Surely the caching should make this effectively an in memory lookup. Does the stellar enrichment function not use the same clientside caching as the Hbase bolt? Simon > On 19 Jun 2017, at 06:21, Casey Stella wrote: > > In order to do that, the easiest thing to do is to

Re: Question on Windows event log ingest and parse

gt; wrote: > > Correction, deploying the Storm topology is this: > > /usr/metron/$METRON_VERSION/bin/start_parser_topology.sh -z `hostname > -f`:2181 -k `hostname -f`:6667 -s winlogbeat > > > > > > From: Simon Elliston Ball <si...@simonellistonball.com> &