Re: Using Metron 0.7.2 in production

*You can certainly use master (sorry, typo/missing word) On Mon, Mar 9, 2020 at 9:38 AM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > You certainly use master - just realize that we haven't gone through the > release formalities yet as a community, so YMMV. This typica

Re: Elastic search in metron

The community would surely appreciate this contribution. If you're pressed for time, even a Jira with the documented steps outlined in the description could be useful for someone else that wants to pick it up and formalize some documentation or a migration script. On Sun, Mar 8, 2020 at 9:25 AM

Re: Using Metron 0.7.2 in production

You certainly use master - just realize that we haven't gone through the release formalities yet as a community, so YMMV. This typically includes some rounds of release candidate testing and ensuring we don't have any outstanding issues that we believe should be included in the next release. There

Re: linux-syslog(centos 7) parsing in apache metron error

That's how we route errors. Looks like the syslog parser had trouble with one of your syslog messages On Mon, Feb 24, 2020, 5:41 AM updates on tube wrote: > i get such error on kibana dashboard no error in storm > com.github.palindromicity.syslog.dsl.ParseException: Syntax error @ 1:0 no >

Re: Alerts UI user metrics dashboard

Here is some further detail in the rest app docs https://github.com/apache/metron/blob/master/metron-interface/metron-rest/README.md#logging On Sat, Feb 15, 2020, 9:53 AM Nick Allen wrote: > Try enabling debug logging for the REST service. You should be able to > add this option under Ambari >

Re: When does the Metron start using the new Maxmind GeoIP data after updating the data files in HDFS?

As soon as the property is changed in global config, it should initiate a reload in the enrichment topology - https://github.com/apache/metron/tree/master/metron-platform/metron-enrichment/metron-enrichment-common#geohdfsfile. I don't recall us writing code that checks a file sha-1 or anything

Re: Metron Tutorial - Fundamentals....Part 3?

Hi Tom, For as long as I've been on the project, since our incubator days, there hasn't been a part 3. It skips from 2 to 4. On Thu, Jan 16, 2020 at 7:25 PM Yerex, Tom wrote: > Good afternoon, > > > > I have been working through some internal documentation to reflect the > official

Re: asa elasticsearch template

There isn't one - you'll need to create it. You can use the bro, yaf, etc templates as examples. On Mon, Dec 30, 2019, 10:28 PM updates on tube wrote: > can any one tell me how to get elasticsearch tempate for asa i cant find > it. >

Re: Issue: reindexing of some events on parsers restart

> > > > > > Kills the topology with the name topology-name. Storm will first > deactivate > > > the topology's spouts for the duration of the topology's message > timeout to > > > allow all messages currently being processed to finish processing. > St

Re: Issue: reindexing of some events on parsers restart

and to stop > and wait until it finishes processing current events and commits changes to > kafka? > > > On 2019/12/10 18:18:28, Michael Miklavcic > wrote: > > Where are you seeing this? As far as I can tell, the UI and REST > endpoints > > default to a grace

Re: Issue: reindexing of some events on parsers restart

Where are you seeing this? As far as I can tell, the UI and REST endpoints default to a graceful shutdown. https://github.com/apache/metron/blob/master/metron-interface/metron-config/src/app/service/storm.service.ts#L154

Re: [DISCUSS] How are you using in Metron?

or perhaps a different “flavour” of > metron that caters for finance domain and can be built as a separate > project, although not sure how to go about it. Is that something the > community/project owners might be interested in considering or supporting? > > > > Best regards, >

[ANNOUNCE] Upgrade helper script available in master

Hi all, I wanted to share that we have made available an upgrade helper script. This will help manage backup and restore of: - Ambari configurations pertinent to Metron - Zookeeper configurations for Metron This will not backup any custom jars you may have placed in parser_contrib or

Re: Metron Enrichment Error

f4j/impl/StaticLoggerBinder.class] > > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] > > Took 0.4232 secondsERROR ArgumentError: Table enrichment does not exist. > > > > NOTE: About the SLF4J I did check the error in the website > http://www.slf4j.org/codes.html#multiple_bindings and it suggests to > remove one of the “.jar” files. I don’t think that’s a safe approach, so I > kept both of them. May it be the problem? > > > > Thanks > > > > *De:* Michael Miklavcic > *Enviada:* 6 de novembro de 2019 17:47 > *Para:* user@metron.apache.org > *Assunto:* Re: Metron Enrichment Error > > > > Hrm, I'm not sure how REST and mysql have anything to do with fixing that > particular issue, but I'm glad you were able to get it working! >

Re: Metron Enrichment Error

Hrm, I'm not sure how REST and mysql have anything to do with fixing that particular issue, but I'm glad you were able to get it working! On Wed, Nov 6, 2019 at 8:27 AM Gonçalo Pedras wrote: > Resolved. > > Deleted Metron service and added the service again but this time I > configured REST by

Re: Push data from elastic search to Metron alerts

It sounds like you might have some issues with Elasticsearch templates. See here for more detail - https://github.com/apache/metron/tree/master/metron-platform/metron-elasticsearch/metron-elasticsearch-common On Wed, Nov 6, 2019 at 8:25 AM Hema malini wrote: > Hi all, > > I pushed data to

Re: Metron Enrichment Error

You shouldn't need to create the HBase tables - the Ambari Metron MPack install will (should) do that for you. Do you have a file in $METRON_HOME/config/ named "metron_enrichment_hbase_configured"? We add a coprocessor to the HBase enrichment table and it looks like your startup is having issues

Re: Error stemming from hbaseBolt

Hi Tom, How did you build Metron? Our latest official release is 0.7.1 -> https://archive.apache.org/dist/metron/. Are you building via latest HEAD in the master branch? You might see a bland/vague exception like this if using an IDE (like Eclipse) for compilation. If you're not doing so already,

Re: Re-establish /apps/metron directory

Hey Tom, Just an fyi, the Ambari install MPack code lays down a series of "configured" flag files when various bits of the system have been configured. Ambari has no concept of this natively, so the Metron community had to come up with an intelligent compromise to work around that limitation. If

Re: Could not resolve dependencies

Hi, if you haven't built Metron itself, you should also do that from the root project first. mvn clean install -DskipTests Build-rpms will ONLY build the rpms using the existing build artifacts, so if the main build has not been run, those artifacts will not exist. On Mon, Nov 4, 2019 at 10:42

Re: Apache Metron production deployment

d. > > Do you have a plan that you can when in near future your will move away > from ambari direct dependancy? 1 month 6 months or something? > > Br > Marcus > Den 29 okt. 2019, kI 19:25, Michael Miklavcic > skrev: >> >> Also agreed on Nick's deployment com

Re: Apache Metron production deployment

Also agreed on Nick's deployment comments. Deploying on AWS manually is fairly trivial. The Ansible scripts have not been touched in months (years, even). You could also install Hadoop services manually (Big Top, for instance), but YMMV. Most of our testing to this point has been on Ambari,

[DISCUSS] How are you using in Metron?

I'd like to kick off a discussion to get a sense of how the broader community is currently using Metron. 1. What features are you using or seriously considering? e.g. 1. enrichments 2. streaming enrichments 3. profiler 4. pcap 5. flatfile summarizer 6. MaaS

Re: [ANNOUNCE] Apache Metron-bro-plugin-kafka release 0.3.0

Congrats all! > I’m pleased to announce the release of Metron 0.3.0! Metron-bro-plugin-kafka (to mitigate any potential future mailing list search confusion) On Thu, Oct 17, 2019 at 8:51 AM Otto Fowler wrote: > Just a reminder, if you used my script to verify the RC, please comment : >

Re: [MaaS] Repeated Events in ElasticSearch

Hi Thiago, Thanks for checking out Metron! Great to hear you're trying out MaaS, we love the feedback. I'm not sure what the expected response time should be for your model - that's probably worth a separate look (Casey Stella - any thoughts here?). If your model is taking 5-10 seconds to return

Re: Help deploying in AWS

t; so, I'm open to suggestions :) > > Thanks, > Eric > > On Thu, 12 Sep 2019 at 16:39, Michael Miklavcic > wrote: > > > > I think it's been quite some time since this has been tested (that I'm > aware of, anyhow). YMMV. I don't see any errors in the stacktrace you >

Re: Help deploying in AWS

I think it's been quite some time since this has been tested (that I'm aware of, anyhow). YMMV. I don't see any errors in the stacktrace you copied - those are all warnings. Near the bottom, the Maven output references "metron-profiler-spark", which is the part of the output we'd want to see. Are

Re: Kafka error in metron

ntManager.scala:53) >>>> at >>>> kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply(ControllerEventManager.scala:53) >>>> at >>>> kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWor

Re: Issues in sending syslog

Also look in the error indexing topic if Solr or ES. And there should also be data written to HDFS, even in case of error. hdfs dfs -ls /apps/metron/indexing On Wed, Sep 4, 2019 at 10:17 AM Hema malini wrote: > Hi, > > I am using Metron 0.7.2. I have setup of three node cluster in centos 7. I >

Re: ACTIONS button is not working in Alert UI

alert status from the dropdown but you do not see a change in status? >>>>>> The buttons itself seems to work fine for me using the latest from the >>>>>> master branch and testing on full dev. >>>>>> >>>>>> On a related note

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

can upgrade. >> >> The issue of what metron *is* features wise may be another one we want >> to take up at some point. The idea being can we separate the metron >> _integration parts from the metron core functionality such that we can work >> on them separately and thus s

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

t; integrations/applications. Of course definition of metron’s value beyond > integration, and what those features and application boundaries are would > be necessary. > > > > > On August 26, 2019 at 18:52:57, Michael Miklavcic ( > michael.miklav...@gmail.com) wrote: > > Hi

[DISCUSS] HDP 3.1 Upgrade and release strategy

Hi devs and users, Some questions were asked in the Slack channel about our ongoing HDP/Hadoop upgrade and I'd like to get a discussion rolling. The original Hadoop upgrade discuss thread can be found here

Re: ACTIONS button is not working in Alert UI

I don't have the UI up currently. Can Tibor, Shane, or Tamas provide any comment on this? On Wed, Jul 24, 2019 at 9:54 PM Rendi 7936 wrote: > Good morning, > Hi there, > > I have implemented Apache Metron 0.7.1 with Hortonworks Cyber Security > Platform. My state now, i can display alert with a

Re: batch indexing in JSON format

Adding to what Ryan said (and I agree), there are a couple additional consequences: 1. There are questions around just how optimal an ORC file written in real-time can actually be. In order to get columns of data striped effectively, you need a sizable number of k rows. That's probably

Re: flatfile_summarizer

d, Jul 10, 2019 at 9:25 AM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > Hi David, > > In this case you would probably want to write your own extractor by > implementing the following interface and setting it as your extractor > implementation

Re: flatfile_summarizer

Hi David, In this case you would probably want to write your own extractor by implementing the following interface and setting it as your extractor implementation -

Re: Stream joins/enrichments

Hi Sanket, Strictly speaking, there isn't a streaming join function. However, you can accomplish something reasonably similar using a streaming enrichment, as you pointed out. You would ultimately pull the enrichment based on the user id, which would then add all the additional fields from your

Re: Metron 0.7/HDP-2.5 installation issues

is 2.6.5, because it seems that HCP isn’t currently > ready for HDP 3 > > > > > > > > > > *From:* Michael Miklavcic [mailto:michael.miklav...@gmail.com] > *Sent:* Thursday, May 23, 2019 17:56 > *To:* user@metron.apache.org > *Subject:* Re: Metron 0.7/HDP-2.5

Re: Metron 0.7/HDP-2.5 installation issues

Someone may have a more recent set of instructions they can share, but the instructions you're referring to are very outdated. Ymmv On Thu, May 23, 2019, 9:50 AM Sanket Sharma wrote: > Hi, > > I am trying to install Metron on a single node (CentOS 7) using the > instructions here >

Re: Very low throuput on topologies

gt; > This completeLatency looks very high doesn’t it? > > > > And for bolt: > > { > > "emitted": 0, > > "requestedMemOnHeap": 128, > > "errorTime": null, > > "tasks": 12, > >

Re: Problems with geolocation enrichment

What version are you running? Take a look at this - https://github.com/apache/metron/pull/1299 - maxmind changed their format. On Mon, May 20, 2019 at 10:04 AM Thiago Rahal Disposti < thiago.ra...@kryptus.com> wrote: > Hi everyone, > > After a long time with everything working perfectly this

Re: Very low throuput on topologies

You could use curl from the cli. But if this is something you're testing out on your local machine, I'd probably start without Kerberos enabled and work the perf knobs there first. You should be able to see the "complete latency" from the Storm UI on each running topology. On Wed, May 15, 2019 at

Re: Invitation to Slack - Vidy - Apache Metron Channel

Invite sent. Fyi, we generally try to send users back to the mailing lists for help so that others can benefit from the archived threads. On Tue, May 14, 2019 at 9:37 AM Vidyasagar G wrote: > Hello, > > Am Vidy, can someone please invite me to Slack channel? Facing few issues > with deployment

Re: About Elastic templates

Thanks for sharing Stephane! Just an fyi, we do also recommend setting the type mapping for strings like you've shown, as indicated here - https://github.com/apache/metron/tree/master/metron-platform/metron-elasticsearch#type-mappings . On Thu, Apr 25, 2019 at 6:33 AM wrote: > I realize that

Re: Unable to execute REST_GET from stellar command line

the version as “0.5.1.1.6.0.0” in the ambari components list for >> metron. >> >> >> >> *From:* Michael Miklavcic [mailto:michael.miklav...@gmail.com] >> *Sent:* Tuesday, April 23, 2019 7:28 PM >> *To:* user@metron.apache.org >> *Subject:* Re: Unable

Re: Unable to execute REST_GET from stellar command line

I think that sounds like an HCP version, not the Apache version. Should be able to grab that from the Ambari component version list. On Tue, Apr 23, 2019, 7:50 PM Anil Donthireddy wrote: > We are using 1.6.0.0-7 version of metron. > > > > *From:* Nick Allen [mailto:n...@nickallen.org] > *Sent:*

Re: Metron Writes partial JSON to HDFS

Hi, this looks like you may be getting failures when writing to HDFS. For example, if there's a problem with a batch, it's possible that it will partially be written to HDFS. I would expect that in instances like this you will see duplicate entries in your HDFS records. The reason for this is that

Re: Load_tool.sh issues

Thiago, have you had any luck with this? I haven't seen this particular issue before. Is this happening for all of your topologies, or just a specific set? Which ones? Also, can you check for any errors in the kafka broker logs along with your Storm topology logs when the hang occurs? One other

Re: Snort logs flow issue

gt;> On Tue, Apr 9, 2019, 12:44 PM Hema malini >> wrote: >> >>> Hi Michael, >>> >>> Thanks for your reply. I couldn't find any errors in metron alerts UI >>> log . I clicked the search and changed the date range too. Still no >>> records

Re: Load_tool.sh issues

For starters, can you elaborate on this a bit? "After a few months running every 15min on the servers, it just stopped working, like this" You had an automated 15-min test that was running fine for months and just stopped working...? Am I interpreting that correctly, and any upgrades in that time?

Re: Snort logs flow issue

quot;," >>>> parallelenricher.enrich.end.ts":"1554384505342","threat. >>>> triage.rules.0.reason":null,"tos":"0","adapter. >>>> hostfromjsonlistadapter.begin.ts":"1554384503452",

Re: Unable to load Custom Stellar functions from HDFS

Hi Athul, Can you post your global.json? On Fri, Mar 29, 2019 at 8:38 AM Athul Parambath wrote: > Hi Team, > > > > We have HCP cluster installed along with HDP and here is the stack > versions: > > Ambari-2.6.2.2 > HDP-2.6.5.0 > HCP-1.8.0.0(Which includes Apache metron-0.7.0) > > > We are

Re: Snort logs flow issue

ui and in Storm topology logs > > On Fri, Apr 5, 2019, 10:53 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > >> How did you validate the logs are making it to the indexing topology? >> >> On Fri, Apr 5, 2019 at 8:12 AM Hema malini >> wrote: >> &

Re: Snort logs flow issue

How did you validate the logs are making it to the indexing topology? On Fri, Apr 5, 2019 at 8:12 AM Hema malini wrote: > > Hi, > > > > We have installed Metron 0.7.1 in centos 7 using Amabari.Using Nifi we > sent the sample snort logs copied from metron git repo to snort kafka > topic.We did

Re: Not seeing feeds in metron -alerts ui

I think I need a bit more context. Are you saying it makes it to indexing and then never makes it to ES or Solr? Are you running fulldev or another type of manual installation? Which index tool are you using, es or solr? On Wed, Apr 3, 2019, 5:26 AM Meenakshi.S <

Re: Cloud Install fails when trying to deploy cluster with Ambari

Follow the link for the Ambari host. The login should be admin:admin. There should be a link at the top that will show you tasks that Ambari executed - dig in there and you should find what failed. On Fri, Mar 29, 2019, 11:37 AM Carmela Stuart wrote: > Hi, > > > > I’m trying to install Metron

Re: Logstash as available parser

Hi Stéphane, Welcome, and thanks for the interest in the project! The Logstash parser you found is one of the original parsers we inherited from the original open-sourced OpenSoc project. We don't have any documentation specific to that parser (or unit tests as I'm looking at this), but it's

Re: Syslog5424 topic missing in kafka

There are a series of flag files we create in the metron home directory to assist with managing state on restarts. Look there for any flag files pertaining to Kafka/parsers. Deleting the file will inform Ambari to reinstall. On Thu, Mar 28, 2019, 5:38 AM Meenakshi.S <

Re: Metron 0.7.1 (METRON INDEXING) bug

m> wrote: >> >>> It is 1 >>> >>> On Sun, Mar 24, 2019, 10:47 PM Michael Miklavcic < >>> michael.miklav...@gmail.com> wrote: >>> >>>> Check out the storm UI. I'm not in front of a computer or I'd try to >>>> share

Re: Metron 0.7.1 (METRON INDEXING) bug

d i look slots are there > > On Sun, Mar 24, 2019, 10:37 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > >> To my knowledge, we haven't seen this issue before. If you're not getting >> anything in enrichment I'd look for exceptions in the parser logs.

Re: Metron 0.7.1 (METRON INDEXING) bug

To my knowledge, we haven't seen this issue before. If you're not getting anything in enrichment I'd look for exceptions in the parser logs. It sounds like messages aren't making it to the enrichment topic. Afa the indexing topology you might double check that you have enough Storm slots

Re: Issues with Metron profiler

Hi Anil, Can you share your profile definition? For the STATS_BIN issue you're seeing, it seems like your bins are wrong, however you're using the pre-defined DECILE bin list, which is unit tested (and the tests are passing). It may be that the stats object isn't quite right, but I can't tell

Re: Central Navigation Use Case

It sounds like we could provide an additional endpoint for the central navigation that we can configure to expose the other 2 endpoints (Management UI, Alerts) based on the user's roles and/or groups. On Mon, Mar 11, 2019 at 9:00 AM Justin Leet wrote: > This feels like our personas should

Re: Use case question

Sanket, you should definitely be able to use Metron for what you've described. Here are some examples that you might find useful for comparison - https://github.com/apache/metron/tree/master/use-cases Best, Mike On Mon, Mar 4, 2019 at 5:24 AM Sanket Sharma wrote: > Hi Simon, > > Thank you for

Re: Centos VM Install Fails, Python Exception Syntax

Otto is correct - I believe this is an issue with Ansible 2.7 using Python 3, which ends up causing some trouble. Try with Ansible 2.6.5. It looks like you're using Homebrew (which is what our README recommends, so that's not surprising), and you can use some combo of the following commands: # get

Re: Metron installation fails

Looks like you may have an issue with your configured users/groups in the blueprint. I'm not sure where you got the blueprint from, but per the stack trace it looks like the "metron" key doesn't exist in the user_to_groups_dict. File

Re: Feature/Wish: Make it possible to tag, group or search sensors in the Metron Mgmt UI

Hey Stefan, that's a cool idea. I'm not aware of any current Jiras or work for this, but feel free to create one if one doesn't exist. I'm not sure if the UI folks are planning to work on anything like this, eg we need to update the management UI to accommodate parser chaining and aggregation. It

Re: Managing Whitelists and Blacklists via UI

Would you see this utilized exclusively for test purposes, or should we look at this as a general feature req to expose our enrichment loading in the UI? To that end, does it make sense to make this broader than whitelists and blacklists? On Thu, Dec 20, 2018 at 2:30 AM Stefan Kupstaitis-Dunkler

Re: Building Metron with Vagrant from Master Branch on Mac OS X using SOLR

Thanks for the feedback Scott. I see 3 categories: 1. Some setup and prereq doc clarification for newcomers, e.g. need for xcode install, clarify doc version compatibility 2. Added details for configuring Solr instead of ES 3. Some added details around self-help on debugging the install, e.g.

Re: Build Errors

Hi David, building the RPMs requires building full Metron first. Switch to the root project directory for Metron and run "mvn clean install" from there. It will build and test all modules which are then able to be bundled up with RPM. You might also check out this page for some options on

Re: Hello

Welcome Scott! I'd probably use the dev list as a contributor, and as Otto and Jon stated, feel free to reach out on Slack. Mike On Sun, Oct 21, 2018 at 9:39 AM zeo...@gmail.com wrote: > Welcome to the community Scott! Let us know if you have any issues > spinning things up, I just recently

Re: Unable to Start Metron REST (With X-PACK)

It's in the rest app docs On Fri, Oct 19, 2018 at 8:14 AM Farrukh Naveed Anjum < anjum.farr...@gmail.com> wrote: > How to so do mysql setup ? > > On Fri, Oct 19, 2018, 6:49 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > >> That looks like 2

Re: Configure storm hostname in metron

It's not specific to enrichment. Look in Advanced metron-env On Fri, Oct 19, 2018 at 6:14 AM Habi S Ravi wrote: > Hi, > > I am setting up metron 0.6.0. When I start metron, it is searching storm > in a different host. > Also In ambari >metron > configs > enrichment section. I do not see any >

Re: Unable to Start Metron REST (With X-PACK)

That looks like 2 separate issues. For REST, check the type of db that's configured in Ambari. Only h2 will automatically create tables, users, and roles. In the version you're running, I believe DB setup steps are manual for other DB install options. On Thu, Oct 18, 2018 at 11:45 PM Farrukh

Re: Upgrading to Elasticsearch 5.6

ACK with 0.6 Release Ambari does > not show kibana and elasticsearch there. What version of Ambari Server > should I use for 0.6 Release ? (Metron Documentation is still outdated > pointing to 0.4.1) > > > > > > > On Fri, Oct 5, 2018 at 10:40 PM Michael Miklavcic < >

Re: Upgrading to Elasticsearch 5.6

We do not have support for ES 6.x yet. Metron's current supported/tested version is 5.6.2. Some users have reported success with 5.6.8 as well, though the reason we're specific about the patch version at this point is because we are still relying on the TransportClient for connecting to ES. There

Re: Upgrading to Elasticsearch 5.6

Yeah, that's basically saying that your indexing topology is still running the 2.0 ES client code. Have you upgraded all of your libraries yet? You'll also need to restart the indexing topology. That should pretty much do it. On Wed, Oct 3, 2018 at 12:50 AM Farrukh Naveed Anjum <

Re: Upgrading to ElasticSearch 6 (GEOPoints)

. > > Can you help ? I am not sure geo_point exists in BRO Templeate do it ? > > On Tue, Oct 2, 2018 at 6:56 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > >> I'm not sure I completely follow. We also have a location point in the >> templa

Re: Upgrading to ElasticSearch 6 (GEOPoints)

I'm not sure I completely follow. We also have a location point in the template. Can you elaborate a bit more on what you're trying to accomplish? { "geo_location_point": { "match": "enrichments:geo:*:location_point", "match_mapping_type": "*", "mapping": { "type": "geo_point"

Re: Metron and Edge Analytics

Hi Julian, Welcome, and thanks for reaching out! If you're looking for processing at the edge, then I think you're on the right track with NiFi and MiNiFi. Where Metron would be of potential use to you is if you're looking to perform additional analytics, ie as you stated things like "how long

Re: Metron Not Reading From Kafka?

That's great to hear David. Thanks for reporting back! On Fri, Sep 14, 2018, 2:52 PM David McGinnis wrote: > All, > > So I think we've found the root cause of this issue,, so I figured I'd > report back. It appears that storm itself was not properly configured, and > thus it was trying to pull

Re: Indexing topology keep crashing

This should be handled fine by Storm. You can set topology.max.spout.pending as back pressure. https://github.com/apache/metron/blob/master/metron-platform/Performance-tuning-guide.md#storm-tuning When there are bursts you may have some backup in the Kafka topics, but the topologies will catch

Re: Metron Not Reading From Kafka?

rtain the issue is with the parser topology. That's why we've tried to > narrow in on any issues showing up in the logs for that particular > topology. > > On Tue, Sep 11, 2018 at 9:27 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > >> Does that error show up ev

Re: Metron Not Reading From Kafka?

Does that error show up every time you restart the parser topology? I'm not sure why you would get that error. I believe at least as far as HDP is concerned, Ganglia hasn't been used in quite some time. Ambari Metrics Server is the approach that is now used. Before we get into the weeds, I would

Re: [ANNOUNCE] - Apache Metron Slack channel

+ Metron user list On Wed, Aug 15, 2018 at 10:38 AM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > Turns out we are able to invite folks on an ad-hoc basis. See instructions > here - > https://cwiki.apache.org/confluence/display/METRON/Community+Resources > > &

Re: metron / spot comparison

Just a heads up, there is active work on upgraded Solr support - https://github.com/apache/metron/tree/feature/METRON-1416-upgrade-solr On Mon, Apr 23, 2018 at 7:53 AM, David McGinnis wrote: > Tom, > > While I'm not a Metron or Security expert by any means, I

Re: No data in HDFS at /apps/metron/indexing/indexed after complete deployment of Full Development VM

I think could be an issue for kibana. What's your view ? >> >> Regards >> RK Sharma >> >> On Thu, Feb 8, 2018 at 9:11 PM, Michael Miklavcic < >> michael.miklav...@gmail.com> wrote: >> >>> We now have 2 topologies for indexing - random a

Re: No data in HDFS at /apps/metron/indexing/indexed after complete deployment of Full Development VM

We now have 2 topologies for indexing - random access and batch. Double check that both are currently running - our full dev environment is pretty full with resources currently. random_access_indexing batch_indexing random_access_indexing is responsible for getting data into Elasticsearch. You

Re: Hello and install issue

The quickest and easiest way is to run the full Dev build - metron-deployment/vagrant/full-dev. If you need to deploy on a dedicated machine, the rpms can be built by following the doc here, under "build rpms" - https://github.com/apache/metron/blob/master/metron-deployment/README.md. Hope this

Re: Kibana Error

What do you see when you go here? http://node1:9200/_cat/health?v You can also get the Elasticsearch Head Plugin for Chrome, which is very useful and will be compatible with 5.x versions of Elasticsearch when Metron upgrades (plugins from 2.x are no longer available in v5.6).

Re: Offset lag tool?

Hi Laurens, There are instructions and examples listed under Tooling->Kafka, and in the section "Example - Viewing Kafka Offset Lags" in the tuning guide. Best, Mike On Mon, Aug 14, 2017 at 9:34 AM, Casey Stella wrote: > It's part of kafka, actually. You can find it

Re: Adding custom enrichment.

Hi Laurens, I believe the way you're referring to 'subnet' in the second argument is as a variable. Did you set subnet := '192.168...' or whatever in advance of that call? You could also just pass in the value explicitly. Best, Mike On Mon, Jul 31, 2017 at 3:33 PM, Laurens Vets

Re: Failure installing Full-Dev Vagrant VM

Hi Mark, I'll comment on the sensors - we have a lightweight set of sensors (stubs) that we're using for dev purposes. Basically, they emit random demo data at intervals. The reason this was done was to keep the dev images lighter weight. Check this doc out -

Re: No longer incubating, but newly hatched!

Woo-hoo! On Mon, Apr 24, 2017 at 7:43 AM, Nick Allen wrote: > Woop woop! > > On Mon, Apr 24, 2017 at 9:28 AM, Casey Stella wrote: > >> Hi All, >> >> Some of you know this already and some of you might not, but as of the >> last ASF board meeting we