Re: Reducing Warnings in Build

2017-04-21 Thread Michael Miklavcic
Personally, I think it makes sense to have an inbound/outbound charset setting for the parsers. We should generally standardize on UTF-8 across Metron and its topologies, but accept potentially different charsets from the inbound sensors. That is to say, standardize the defaults for the things we

Re: [DISCUSS] Modify bylaws to allow speculative branches

2017-04-24 Thread Michael Miklavcic
"Branch committers ... do not cast binding votes or vetoes in the project." It sounds similar to what we do for other PRs except in this case it's 1..n contributors instead of just one. The step for getting the feature branch merged into trunk sounds reasonable and clear enough, but as Casey

Re: Facing error whith mvn clean install

2017-07-27 Thread Michael Miklavcic
ven/io.confluent/kafka-avro-serializer/pom.xml 116 Tue Feb 24 19:49:38 MST 2015 META-INF/maven/io.confluent/kafka-avro-serializer/pom.properties Best, Michael Miklavcic On Thu, Jul 27, 2017 at 10:47 AM, bharath phatak <bharath.pha...@gmail.com> wrote: > Hi All, > > I followe

Re: METRON-1004 Build improvements are in master

2017-06-30 Thread Michael Miklavcic
My branch before your changes Ran for 49 min 59 sec My branch after Ran for 26 min 51 sec You guys are my hero, great job! Best, Mike On Fri, Jun 30, 2017 at 8:11 AM, Justin Leet wrote: > METRON-1004 was made to address several issues we were having with the > Travis

Re: Ambari Wizard: Repo Tab

2017-04-26 Thread Michael Miklavcic
was going by the HW community page. > > Ok, Let me try it > > > > On April 25, 2017 at 16:04:07, David Lyle (dlyle65...@gmail.com) wrote: > > That would do it. It requires 2.4.2+. I would have sworn I put that in the > README, but I must have only annotated the PR.

Re: Ambari Wizard: Repo Tab

2017-04-26 Thread Michael Miklavcic
d it otherwise the file was not valid YAML and > it > > would crash trying to load the config file. > > > > network.host: ["_lo:ipv4_","_eth0:ipv4_"] > > > > On Wed, Apr 26, 2017 at 3:09 PM, Otto Fowler <ottobackwa...@gmail.com> > > wrote: >

Re: Ambari Wizard: Repo Tab

2017-04-26 Thread Michael Miklavcic
wing > > > > https://community.hortonworks.com/articles/60805/deploying- > a-fresh-metron-cluster-using-ambari-serv.html > > I DO have master and data node together. > > Why is that a problem? > > > > I will try again with master and data node separate. > >

Kerberos setup docs - bad merge and impending fix

2017-04-27 Thread Michael Miklavcic
Just a heads up to the dev list that a couple PRs on the Kerberos docs resulted in a bad merge that pushed some new instructions and formatting changes to the wrong place. I'm working on a fix now - https://issues.apache.org/jira/browse/METRON-899 If you are running the Ambari Kerberos install

Re: Status of METRON-153

2017-04-25 Thread Michael Miklavcic
We didn't come up with another approach Otto. I closed the PR at the time bc I didn't think we wanted to break with Centos 6 yet, and my PR probably would have done so. Can you elaborate on "prior capability wrt centos 6.x work again" a bit? On Apr 25, 2017 10:59 AM, "Otto Fowler"

Re: Quick Dev - Atlas Images

2017-04-25 Thread Michael Miklavcic
I may have missed a discuss thread on this, but it looks like we are no longer using -SNAPSHOT in our current dev master https://github.com/apache/incubator-metron/blob/master/pom.xml#L19 On Tue, Apr 25, 2017 at 12:34 PM, Nick Allen wrote: > >> I hadn't really reasoned

Re: Ambari Wizard: Repo Tab

2017-04-25 Thread Michael Miklavcic
epos/centos7', 'action': ['create'], > 'components': [u'HDP-UTILS', 'main'], 'repo_template': > '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list > %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif > %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'HDP-UTILS'

Re: Facing error whith mvn clean install

2017-07-28 Thread Michael Miklavcic
nable full debug > logging. > > [ERROR] > > [ERROR] For more information about the errors and possible solutions, > > please read the following articles: > > [ERROR] [Help 1] > > http://cwiki.apache.org/confluence/display/MAVEN/

Re: [DISCUSS] Easing the ramp-up into contributing

2017-07-28 Thread Michael Miklavcic
Hi Artem, You could probably get away with not running them locally since we use Travis, which functions similarly to Jenkins from a testing perspective. Additionally, you can hook up Travis to run on your own github account as many Metron committers have already done. The upshot is that if your

Re: Code style change requires vote?

2017-07-31 Thread Michael Miklavcic
Justin, this is great! Thanks again for the contribution as well as for the follow-up and clarification on dev guidelines as well as community vote, etc. I was under the impression that while the end goal is implicit, ie using the new standard henceforth, that there was some follow on work towards

Re: [DISCUSS] platform_info.sh and PR template / Dev Guidelines

2017-05-03 Thread Michael Miklavcic
I agree with Nick about the PR template insofar as we should keep it simple. And for any checklist items, we should seek to automate as much as possible. We've already made some strides with Justin Leet's changes to include site-book building in the main build, for instance. I'd prefer to add this

Re: Error building in Travis after taking master

2017-05-03 Thread Michael Miklavcic
Also Otto, are you able to build locally with that branch and same merge with master? On Wed, May 3, 2017 at 9:51 AM, Justin Leet wrote: > I'm also curious if it works if you change the two 'jacoco:prepare-agent' > to 'org.jacoco:prepare-agent'. Master has built off of

Re: Error building in Travis after taking master

2017-05-03 Thread Michael Miklavcic
<justinjl...@gmail.com> wrote: > Is it just that one branch that's failing, or is it also master? To try to > narrow things down. > > On Wed, May 3, 2017 at 1:43 PM, Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > Hahaha, I can't imagine why you wouldn't r

Re: Error building in Travis after taking master

2017-05-03 Thread Michael Miklavcic
t -b whyyounowork apache/master > > > > same issue locally > > > > > > > > > > On May 3, 2017 at 13:46:14, Justin Leet (justinjl...@gmail.com) wrote: > > > > Is it just that one branch that's failing, or is it also master? To try > >

Re: [DISCUSS] Code Style

2017-05-08 Thread Michael Miklavcic
+1 Justin. Thanks. On Mon, May 8, 2017 at 2:55 PM, Matt Foley wrote: > +1. I originally suggested the Sun style as a starting point, and I find > Justin’s arguments convincing, especially if there is a pre-packaged Google > style checking plug-in. > --Matt > > On

Re: Elasticsearch masters_also_are_datanodes doesn't work.

2017-09-13 Thread Michael Miklavcic
t; wrote: > Sorry, you’ve beat me. > > > > Mike, what have you found in full-dev? Does ES work correctly there? > > > > > > *From: *Laurens Vets <laur...@daemon.be> > *Date: *Wednesday, September 13, 2017 at 1:14 PM > *To: *Matt Foley <mfo...@hortonworks.c

Re: [DISCUSS] Dropping support for elastic 2.x

2017-10-04 Thread Michael Miklavcic
> > > > > > On 4 Oct 2017, at 18:05, Casey Stella <ceste...@gmail.com> wrote: > > > > > > > > So, how would this work in an upgrade scenario that does not involve > > > losing > > > > the existing indexed data? > >

Re: [DISCUSS] Dropping support for elastic 2.x

2017-10-04 Thread Michael Miklavcic
The client I'm currently working on moving towards would *not* be backwards compatible. https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-high-compatibility.html " The High Level Client is guaranteed to be able to communicate with any Elasticsearch node running on

Re: [DISCUSS] Dropping support for elastic 2.x

2017-10-04 Thread Michael Miklavcic
I should note that there's a difference between supporting INSTALLING multiple versions versus being able to manage them. On Wed, Oct 4, 2017 at 11:20 AM, Michael Miklavcic < michael.miklav...@gmail.com> wrote: > The question comes back to the DISCUSS I opened the other day about >

Re: Metron 0.4.2 release date

2017-10-09 Thread Michael Miklavcic
Hey Ali, I'm currently deep in ES 5.x migration work. If you have any specific concerns or requests that have not yet been covered on the mailing list, please feel free to comment. Best, Mike On Mon, Oct 9, 2017 at 9:05 AM, James Sirota wrote: > I would expect ES 5.x

Re: [DISCUSS] Upgrading Elasticsearch from 2.x to 5.x

2017-10-05 Thread Michael Miklavcic
s better or worse in the end. Best, Mike 1. https://www.elastic.co/blog/state-of-the-official- elasticsearch-java-clients 2. https://www.elastic.co/guide/en/elasticsearch/client/java- rest/current/java-rest-high-compatibility.html <https://www.elastic.co/guide/en/elasticsearch/client

Re: [DISCUSS] Build broken due to transitive dependencies

2017-10-13 Thread Michael Miklavcic
[INFO] Cannot find module 'tough-cookie' - It's as if the build has become sentient and now mocks us On Fri, Oct 13, 2017 at 12:00 PM, Laurens Vets wrote: > ... > [INFO] --- frontend-maven-plugin:1.3:npm (ng build) @ metron-config --- > [DEBUG] Configuring mojo

Re: Elasticsearch masters_also_are_datanodes doesn't work.

2017-09-12 Thread Michael Miklavcic
At the very least, the value provided by default seems to have changed to a "1" instead of "true" without the tooltip having been updated to match. [image: Inline image 1] On Tue, Sep 12, 2017 at 4:00 PM, Michael Miklavcic < michael.miklav...@gmail.com> wrote: > I th

Re: Elasticsearch masters_also_are_datanodes doesn't work.

2017-09-12 Thread Michael Miklavcic
I think this is our default setup for full dev. It's only a 1-node VM, so I'm pretty sure that it would not work otherwise. I'm spinning up full dev now and will look into it also. On Tue, Sep 12, 2017 at 3:04 PM, Laurens Vets wrote: >

Re: [DISCUSS] How should Management UI save changes?

2017-09-28 Thread Michael Miklavcic
Do we have any screen mockups/pdfs that can be shared? It might be easier for the community to discuss. On Wed, Sep 20, 2017 at 3:37 PM, Ryan Merriman wrote: > Recently @nickwallen brought up some good points about the usability of the > Management UI here: >

Re: [DISCUSS] Metron release 0.4.1

2017-08-30 Thread Michael Miklavcic
I have some work around fixing how we handle config with Ambari that I'd like to see go in. No PR yet, but coming soon. I expect to have this by the RC deadline. Mike On Wed, Aug 30, 2017 at 8:57 AM, Nick Allen wrote: > The following PRs are usability enhancements for the

Re: [DISCUSS] Upgrading Elasticsearch from 2.x to 5.x

2017-10-05 Thread Michael Miklavcic
riting for every version of ES. > > Unless that's something we have a demand for (or someone else persuades me > otherwise), I'm in favor of using the high level client. It seems like > it'd be easier to migrate to also, given the similarities API-wise to the > current client we're usi

Re: [DISCUSS] Upgrading Elasticsearch from 2.x to 5.x

2017-10-11 Thread Michael Miklavcic
sey Stella <ceste...@gmail.com> wrote: > Yeah, I agree with what Michael "fine whine" Miklavcic said; I'm in favor > of the high level client. > > On Thu, Oct 5, 2017 at 3:35 PM, Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > Justin, thank

Re: [DISCUSS] Upgrading Elasticsearch from 2.x to 5.x

2017-10-11 Thread Michael Miklavcic
I attached a PDF - shows up on my end. Is that not coming through? On Wed, Oct 11, 2017 at 6:42 PM, Otto Fowler <ottobackwa...@gmail.com> wrote: > I think there is a missing attachment? > > > On October 11, 2017 at 20:22:33, Michael Miklavcic ( > michael.mikla

Re: [DISCUSS] Upgrading Elasticsearch from 2.x to 5.x

2017-10-11 Thread Michael Miklavcic
erver and include the URL in the mail." I'm open to suggestions for better ways to share, but for now how's this? https://imgur.com/a/xSnPr On Wed, Oct 11, 2017 at 6:57 PM, Otto Fowler <ottobackwa...@gmail.com> wrote: > I don’t see one :( > > > On October 11, 2017 at

Re: Metron 0.4.2 release date

2017-10-11 Thread Michael Miklavcic
ort > it, I am fine. > > Regards, > Ali > > On Tue, Oct 10, 2017 at 3:33 AM, Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > Hey Ali, I'm currently deep in ES 5.x migration work. If you have any > > specific concerns or requests that have not yet

Re: [DISCUSS] Upgrading Elasticsearch from 2.x to 5.x

2017-10-12 Thread Michael Miklavcic
tachments, via comments on PRs. Not necessarily > intuitive for a “Discussion”. There would be more sensible places to put > files for discussion in github, such as “wiki” or “issues”, but those > aren’t enabled on apache projects in github. > > On 10/11/17, 10:39 PM, "Micha

Re: [DISCUSS] Are/how are you using the ES data pruner?

2017-11-22 Thread Michael Miklavcic
> Cheers, > Ali > > On Thu, Nov 23, 2017 at 12:20 PM, Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > From what I can tell, the data pruner isn't documented anywhere, so I'm > > curious if anybody is using this, and if so, how are you using

Re: [DISCUSS] e2e test infrastructure

2017-11-29 Thread Michael Miklavcic
do > > remote debugging of them). Obviously, that's easier said than done, but > > what I'd like to avoid us having essentially two different ways to do the > > same thing (spin up some our of dependency components and run code > against > > them). I'm worried that's quic

Re: [DISCUSS] e2e test infrastructure

2017-11-29 Thread Michael Miklavcic
vs full dev all over again. But without us > being able to easily kill one because half of tests depend on one and half > on the other. > > On Wed, Nov 29, 2017 at 1:22 AM, Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > What about just spinning up e

Re: [DISCUSS] Are/how are you using the ES data pruner?

2017-11-27 Thread Michael Miklavcic
hortly. It was > a > > quick and dirty work actually just to fix this temporarily. However, it > > might be a good starting point. > > > > On Thu, Nov 23, 2017 at 3:31 PM, Michael Miklavcic < > > michael.miklav...@gmail.com> wrote: > > > >> Thanks

Re: [DISCUSS] Stellar Documentation Autogeneration

2017-12-14 Thread Michael Miklavcic
+1 from me, great idea Justin. I did a bit of digging around also and the Doclet approach you're already using seems the way to go. I didn't come across any libraries that would make this easier or better. Not sure if Swagger has anything along these lines? On Thu, Dec 14, 2017 at 1:00 PM, Otto

Re: [DEV COMMUNITY MEETING] Call for Ideas and Schedule

2017-12-15 Thread Michael Miklavcic
Sounds good Otto. We probably also want to touch on the ES 5.6 upgrade along with our current release status and short-term release roadmap that Nick Allen has been guiding. On Fri, Dec 15, 2017 at 9:02 AM, Laurens Vets wrote: > I'll try to attend :) > > > On 2017-12-14

Re: [DISCUSS] - Remove Kibana

2017-11-01 Thread Michael Miklavcic
I use Discover a > lot to filter and look at events and create new policies from that. > > Is there currently a simple way to do this without Kibana? > > On 2017-11-01 09:13, Michael Miklavcic wrote: > >> As part of the ES upgrade, I got to thinking that it makes sens

Re: [DISCUSS] - Remove Kibana

2017-11-01 Thread Michael Miklavcic
. > > Or would Kibana always be required, just not installed by Metron? > > On 2017-11-01 11:34, Michael Miklavcic wrote: > >> You could absolutely still do it, I'm simply saying it would not be >> managed >> by us. >> >> On Nov 1, 2017 12:20 PM, "Lau

Re: [DISCUSS] Upcoming Release

2017-11-06 Thread Michael Miklavcic
I agree with your proposal, Nick. I think having a stabilizing release prior to upgrading ES/Kibana makes sense. On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen wrote: > I would like to start a discussion around upcoming releases. We have a > couple separate significant tracks

Re: [DISCUSS] Upcoming Release

2017-11-06 Thread Michael Miklavcic
y ES but > getting a handle on kibana and the mpack situation as well. > > > > > On November 6, 2017 at 12:48:45, Michael Miklavcic ( > michael.miklav...@gmail.com) wrote: > > I agree with your proposal, Nick. I think having a stabilizing release > prior to upgrading ES/Kib

Re: [DISCUSS] - Remove Kibana

2017-11-01 Thread Michael Miklavcic
gt; mean having two mpacks to build for a while though. > > I agree with others that, at least for now, Kibana is an integral part of > the Metron user experience. > > -Kyle > > On Wed, Nov 1, 2017 at 7:37 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > >

Re: Heterogeneous indexing batch size for different Metron feeds

2017-12-10 Thread Michael Miklavcic
\ --bootstrap-server $BROKERLIST \ --new-consumer Hope this helps. Cheers, Michael Miklavcic On Sun, Dec 10, 2017 at 5:38 AM, Ali Nazemian <alinazem...@gmail.com> wrote: > This seems not the same as our observations. Whenever there are some > messages in the indexing or

Re: [DISCUSS] Using Yarn package manager for metron-alerts

2017-10-30 Thread Michael Miklavcic
Would love to revive this - I think this could help drastically reduce our build times for metron-interface, which locally just took me 9 minutes in non-parallel mode with -DskipTests set. This is a really good suggestion even just for the offline install and version locking, as pointed out by

Re: [DISCUSS] Pcap panel architecture

2018-05-07 Thread Michael Miklavcic
That sounds fine - I'd imagine we'd be looking to hit the classpath related problems asap when merging the modules. For the module, we just have a pom that supplies external dependencies. Rather than every metron module depending on Guava or Jackson directly, or via transitive dependencies, we

Re: [DISCUSS] Pcap panel architecture

2018-05-03 Thread Michael Miklavcic
point is that maybe we should have a discuss about: > > > > * PCAP UI, goals etc > > * Where it would live and why, what that would mean etc > > * Backend ( this original mail ) > > > > > > > > On May 3, 2018 at 18:34:00, Michael Miklavcic ( >

Re: [DISCUSS] Pcap panel architecture

2018-05-08 Thread Michael Miklavcic
this all the time > > and > > > it's really useful. > > > > > > The metron-rest jar is already shaded. > > > > > > Reverse engineering the yarn jar command was the next thing I was going > > to > > > try. W

Re: [DISCUSS] Pcap panel architecture

2018-05-08 Thread Michael Miklavcic
d get results. All you should have to do > is generate some pcap data first. > > On Tue, May 8, 2018 at 4:17 PM, Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > @Ryan - pulled your branch and experimented with a few things. In doing > so, > >

Re: [DISCUSS] Pcap panel architecture

2018-05-07 Thread Michael Miklavcic
What order did you add the hadoop or yarn classpath? The "shaded" package stands out to me in this name "org.apache.hadoop.hbase.*shaded* .org.codehaus.jackson.jaxrs.JacksonJaxbJsonProvider." Maybe try adding those packages earlier on the classpath. I think that find command needs a "jar tvf",

Re: [DISCUSS] Release?

2018-05-09 Thread Michael Miklavcic
Is this what you mean Otto? https://github.com/apache/metron/blob/24822dddc68c264f59723f5e17d423cd497f6807/dev-utilities/release-utils/validate-jira-for-release On Wed, May 9, 2018 at 9:52 AM, Casey Stella wrote: > I wasn't aware we had a script for that..is that in >

Re: [DISCUSS] Release?

2018-05-09 Thread Michael Miklavcic
https://issues.apache.org/jira/browse/METRON-1511 METRON-1528Done Michael Miklavcic https://issues.apache.org/jira/browse/METRON-1528 METRON-1445Done Michael Miklavcic https://issues.apache.org/jira/browse

Re: [DISCUSS] Release?

2018-05-09 Thread Michael Miklavcic
+1 On Wed, May 9, 2018 at 9:13 AM, Casey Stella wrote: > Is it about time for a release? I know we got some substantial performance > changes in since the last release. I think we might have a justification > for a release. > > Casey >

Re: [DISCUSS] Release?

2018-05-09 Thread Michael Miklavcic
V via mmiklavc) closes apache/metron#889 > > >> 4 months ago METRON-1393: Fix bro Elasticsearch template (mmiklavc via > > >> mmiklavc) closes apache/metron#893 > > >> 4 months ago METRON-1379: Add an OBJECT_GET stellar function closes &

Re: [DISCUSS] Release?

2018-05-09 Thread Michael Miklavcic
the other major ES and Solr changes. On Wed, May 9, 2018 at 12:13 PM, Michael Miklavcic < michael.miklav...@gmail.com> wrote: > I'm also a +1 on 0.5.0. This is a fairly big release. > > On Wed, May 9, 2018 at 12:05 PM, Nick Allen <n...@nickallen.org> wrote: > >> +1 to 0.5.0

Re: [DISCUSS] Pcap panel architecture

2018-05-09 Thread Michael Miklavcic
LETE /api/v1/pcap/ > > This endpoint will delete pcap query results. Not sure yet how this fits > in with our broader cleanup strategy. > > This should get us started. What did I miss and what would you change > about these? I did not include much detail related to security, c

Re: [DISCUSS] Pcap UI user requirements

2018-05-09 Thread Michael Miklavcic
We are limited by Yarn and MapReduce applications in the case of pause/resume - I could be wrong, but I don't think that's something that's supported unless you're talking about multiple MR jobs strung together.

Re: [DISCUSS] Release?

2018-05-09 Thread Michael Miklavcic
ady in master waiting to be released that I don't > see a need to include it in the next release. I'd be happy to see that > Solr work drive a follow-on release. > > > > > On Wed, May 9, 2018 at 2:16 PM, Michael Miklavcic < > michael.miklav...@gmail.com> wrote: >

Re: [DISCUSS] Release?

2018-05-09 Thread Michael Miklavcic
and related > > docs) to make sure it's accurate as well. > > > > Jon > > > > On Wed, May 9, 2018 at 1:30 PM Michael Miklavcic < > > michael.miklav...@gmail.com> wrote: > > > > > Good call - I thought that made our last release, but this would b

Re: [DISCUSS] Release?

2018-05-10 Thread Michael Miklavcic
> > this something that is a blocker for the release? > > > > > > > > IMO, there is so much already in master waiting to be released that I > > > don't > > > > see a need to include it in the next release. I'd be happy to see > that > &g

Re: [DISCUSS] Release Manager

2018-05-10 Thread Michael Miklavcic
Thanks Matt for doing this for the community. Justin Leet as new lord commander of the Night's Watch? Aye, dilly, dilly. On Thu, May 10, 2018 at 9:07 AM, Justin Leet wrote: > I'd be happy to to volunteer to take over for a while. > > Thanks to Matt for all the help

Re: [DISCUSS] Metron release 0.5.0

2018-05-16 Thread Michael Miklavcic
Agreed Nick - the ES upgrade was pretty extensive on its own. On Wed, May 16, 2018 at 5:24 AM, Nick Allen wrote: > Going to 0.5.0 is well justified without Solr IMO. > > On Wed, May 16, 2018, 7:01 AM Otto Fowler wrote: > > > My question is: Is

Re: Using Java Rest Client instead of Transport Client for Elasticsearch

2018-06-13 Thread Michael Miklavcic
I think there is some level of support for auth via their REST api, but I don't see anything specific to X-Pack as you mentioned. However, the major reason we did not adopt it at the time of upgrade was because a number of features were not available to REST yet and the effort to simultaneously

Re: [DISCUSS] Refactoring

2018-05-30 Thread Michael Miklavcic
as to the intent >> and effect. >> >> +1 for the idea, we should have a vote round or edit round on the doc’s >> specific text. >> Although I will say, that some things it doesn’t matter how much you break >> them up wrt reviews. >> We shou

Re: [DISCUSS] Refactoring

2018-05-30 Thread Michael Miklavcic
y for cosmetic refactoring purposes aimed solely >> readability; it makes code review >> and merging difficult. It’s okay to fix an occasional comment or >> indentation, but if >> wholesale comment, whitespace or other refactoring changes are needed, >> make >>

[DISCUSS] Refactoring

2018-05-29 Thread Michael Miklavcic
play/METRON/Development+Guidelines Best, Michael Miklavcic

Re: [DISCUSS] Pcap panel architecture

2018-05-03 Thread Michael Miklavcic
Thanks for the write-up, Ryan. A few questions and comments. 1. metron-api 1. "It hasn't been used in a while and will need some end to end testing to make sure it still functions properly" > I was probably one of the last developers to touch this code a year or more ago -

Re: [DISCUSS] Pcap panel architecture

2018-05-03 Thread Michael Miklavcic
Otto, what are you and your customers finding useful and/or difficult from a split management/alerts UI perspective? It might help us to restate the original scope and intent around maintaining separate management and alert UI's, to your point about "contrary to previous direction." I personally

Re: [DISCUSS] Pcap panel architecture

2018-05-03 Thread Michael Miklavcic
Comments inline below. On Thu, May 3, 2018 at 3:25 PM, Ryan Merriman wrote: > Otto, > > I'm assuming just adding it to the Alerts UI is less work but I wouldn't be > strongly opposed to it being it's own UI. What are the reasons for doing > that? > > I don't know that we

Re: [DISCUSS] Pcap panel architecture

2018-05-03 Thread Michael Miklavcic
an etc > * Backend ( this original mail ) > > > > On May 3, 2018 at 18:34:00, Michael Miklavcic (michael.miklav...@gmail.com) > wrote: > > Otto, what are you and your customers finding useful and/or difficult from > a split management/alerts UI perspective? It might help us to r

Re: [DISCUSS] Generating and Interacting with serialized summary objects

2018-01-05 Thread Michael Miklavcic
, but it does potentially offer the > user a way to quickly grasp what the JSON blob is actually specifying. > Making it easy to understand, even if it's not the ideal way to interact is > potentially still a win. > > On Thu, Jan 4, 2018 at 1:28 PM, Michael Miklavcic < > michael.

Re: [DISCUSS] Generating and Interacting with serialized summary objects

2018-01-04 Thread Michael Miklavcic
ames Sirota <jsir...@apache.org> wrote: > > > >> I just went through these pull requests as well and also agree this is > >> good work. I think it's a good first pass. I would be careful with > trying > >> to boil the ocean here. I think for the i

Re: [DISCUSS] Generating and Interacting with serialized summary objects

2018-01-05 Thread Michael Miklavcic
s complex, but it might make it more > > easily > > > digestible if it's split up by idea (parsing, transformation, etc.). > > > > > > Re: Mike's point, I don't think we want the actual processing broken up > > as > > > ETL, but the representation t

[DISCUSS] Update Metron Elasticsearch index names to metron_

2018-01-24 Thread Michael Miklavcic
With the completion of https://github.com/apache/metron/pull/840 (METRON-939: Upgrade ElasticSearch and Kibana), we have the makings for a major release rev of Metron in the upcoming release (currently slotted to 0.4.3, I believe). Since there are non-backwards compatible changes pertaining to ES

Re: Dependency Checks

2018-01-24 Thread Michael Miklavcic
A big +1 to this. Good suggestion. On Jan 24, 2018 2:44 PM, "Nick Allen" wrote: > We should re-jigger `platform-info.sh` (or create a new tool) that very > obviously passes or fails based on what it discovers in the user's > environment. Right now, a user just runs the

Re: Master is failed in Travis

2018-01-24 Thread Michael Miklavcic
Fix worked on my fork's Travis build - https://travis-ci.org/mmiklavc/metron/builds/332889008?utm_source=email_medium=notification. Just waiting on the mainline build to run and we should be good to go. On Wed, Jan 24, 2018 at 10:36 AM, Michael Miklavcic < michael.miklav...@gmail.com>

Re: [DISCUSS] Update Metron Elasticsearch index names to metron_

2018-01-24 Thread Michael Miklavcic
": { "mapping": { "type": "float" }, "match": "threat:triage:*score", "match_mapping_type": "*" } }, { "threat_triage_reason": {

Re: [DISCUSS] Update Metron Elasticsearch index names to metron_

2018-01-24 Thread Michael Miklavcic
1 to a standard prefix for all Metron indices. I've had the same thought > > myself and you laid out the advantages well. > > > > > > > > > > > > On Wed, Jan 24, 2018 at 3:47 PM zeo...@gmail.com <zeo...@gmail.com> > wrote: > > > > > I a

Re: Master is failed in Travis

2018-01-24 Thread Michael Miklavcic
cted, because no message in 1 ms. > > This thread appears to shed light on the problem and I'm attempting to > lock the remap-istanbul version now. https://github.com/ > jhipster/generator-jhipster/issues/7031 > > I'd like to re-up my strong support to get this working m

Re: [DISCUSS] Update Metron Elasticsearch index names to metron_

2018-01-26 Thread Michael Miklavcic
;> { > >> "geo_longitude": { > >> "match": "enrichments:geo:*:longitude", > >> "match_mapping_type": "*", > >> "mapping": { > >>

Re: [DISCUSS] Move SHELL type functions from management to stellar common

2018-01-31 Thread Michael Miklavcic
Agreed On Jan 31, 2018 7:51 AM, "Justin Leet" wrote: > Agreed, I think it makes sense to move them there. > > On Wed, Jan 31, 2018 at 9:28 AM, Casey Stella wrote: > > > I'd be in favor of that. That is general purpose stuff. > > > > On Wed, Jan 31,

Re: [DISCUSS] Persistence store for user profile settings

2018-02-01 Thread Michael Miklavcic
Personally, I'd be in favor of something like Maria DB as an open source repo. Or any other ansi sql store. On the positive side, it should mesh seamlessly with ORM tools. And the schema for this should be pretty vanilla, I'd imagine. I might even consider skipping ORM for straight JDBC and simple

Re: [DISCUSS] The requested URL returned error: 404 Not Found

2018-02-05 Thread Michael Miklavcic
I had this problem on a machine running Vagrant 1.8.1. I thought it was only Ubuntu at first, so I removed some additional boxes and found it to be a problem for all of them. I didn't see any relevant articles other than some old stuff talking about a bundled curl command being a problem, but that

Re: [DISCUSS] The requested URL returned error: 404 Not Found

2018-02-05 Thread Michael Miklavcic
Preparing network interfaces based on configuration... > node1: Adapter 1: nat > node1: Adapter 2: hostonly > ==> node1: Forwarding ports... > node1: 22 (guest) => (host) (adapter 1) > ==> node1: Running 'pre-boot' VM customizations... > ==> node1: Booting

Re: [DISCUSS] Persistence store for user profile settings

2018-02-12 Thread Michael Miklavcic
t use the DB that gets spun-up with > Ambari. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Feb 2, 2018 at 7:17 AM Simon Elliston Ball < > > > > > si...@simonellistonb

Re: Apache Website Required Links

2018-02-16 Thread Michael Miklavcic
That's awesome Anand, thanks for tackling this! On Fri, Feb 16, 2018 at 7:41 AM, Anand Subramanian < asubraman...@hortonworks.com> wrote: > Btw, here is the output from running site-scan on my local version of the > changes: > > ➜ tools git:(master) ruby site-scan.rb http://127.0.0.1:4000 > 127

[DISCUSS] Split Elasticsearch and Kibana into separate MPack from Metron

2018-02-16 Thread Michael Miklavcic
This came up earlier when discussing work around the ES upgrade: https://lists.apache.org/thread.html/66280bc061afbba2c353221c3c05fd74b247b970921c009c29edc815@%3Cdev.metron.apache.org%3E

Re: [DISCUSS] Split Elasticsearch and Kibana into separate MPack from Metron

2018-02-21 Thread Michael Miklavcic
just too many installation > configurations for us to reasonably support; especially on larger > installations. Supporting the Elasticsearch MPack is a project unto > itself. That being said, the functionality will still be there for those > that want to use it. > > > > On Fri, Feb 16, 20

Re: [DISCUSS] Generating and Interacting with serialized summary objects

2018-01-03 Thread Michael Miklavcic
I'm liking this design and growth strategy, Casey. I also think Nick and Otto have some valid points. I always find there's a natural tension between too little, just enough, and boiling the ocean and these discuss threads really help drive what the short and long term visions should look like.

Re: [DISCUSS] Generating and Interacting with serialized summary objects

2018-01-03 Thread Michael Miklavcic
are that this is excellent. On Wed, Jan 3, 2018 at 12:43 PM, Michael Miklavcic < michael.miklav...@gmail.com> wrote: > I'm liking this design and growth strategy, Casey. I also think Nick and > Otto have some valid points. I always find there's a natural tension > between too little, just enou

Re: Knox SSO feature branch PRs: a quick demo

2018-08-02 Thread Michael Miklavcic
Nice! Thanks for the walk-through, Simon. Agreed that this should assist reviewers in understanding the feature branch and PR break-down. Cheers, Mike On Thu, Aug 2, 2018 at 8:51 AM larry mccay wrote: > Hi Simon - > > I like how you walk through those various PRs and describe what is done at >

Re: Feature Branch Process

2018-07-30 Thread Michael Miklavcic
Hey Nick, thanks for starting this thread. Some thought of mine from recent work in feature branches for Solr and the Pcap query panel: 1. As a general rule, feature branches should be started for code changes that are not able to be delivered in 1-2 PR's of no more than 1-2k lines, but

Re: [DISCUSS] Batch Profiler

2018-07-28 Thread Michael Miklavcic
+1 on the feature branch, Nick. I'll start reviewing the write-ups shortly. On Fri, Jul 27, 2018, 9:29 AM Nick Allen wrote: > Hi Everyone - > > A while back I opened up a discuss thread around the general idea of a > Batch Profiler [1]. I'd like to start making progress on a first draft of >

Re: [DISCUSS] Metron Parsers in Nifi

2018-08-08 Thread Michael Miklavcic
I think it also provides customers greater control over their architecture by giving them the flexibility to choose where/how to host their parsers. To Justin's point about the API, my biggest concern about the RecordReader approach is that it is not stable. We already have a similar problem in

Pcap Query Panel feature branch status

2018-08-07 Thread Michael Miklavcic
I'd like to put up a DISCUSS thread in the next day or so regarding getting the pcap feature branch merged. In preparation, I am going to share an accounting of completed and outstanding tasks. Can folks that have contributed update their Jira status in the subtasks? It looks like the current

Re: [ANNOUNCE] - Apache Metron Slack channel

2018-08-15 Thread Michael Miklavcic
On 15 August 2018 at 16:09, Nick Allen wrote: > > > Thanks for the instructions! > > > > On Wed, Aug 15, 2018 at 10:22 AM, Michael Miklavcic < > > michael.miklav...@gmail.com> wrote: > > > > > The Metron community has a Slack channel available

  1   2   3   4   >