Re: Master build - failed due to Maven download fail

2019-05-24 Thread Michael Miklavcic
; >> it seems like we should be able to cache it to avoid downloading every > >> time. > >> > >> On Mon, May 20, 2019 at 6:14 PM Michael Miklavcic < > >> michael.miklav...@gmail.com> wrote: > >> > >> > FYI I kicked off a rebuild for master. For some reason the wget > command > >> to > >> > grab Maven 3.3.9 failed. > >> > > >> > > >

Re: [DISCUSS] Travis CI Build Time Limits

2019-05-23 Thread Michael Miklavcic
I think this is a neat idea, however I have a couple concerns with this: 1. The build is really a tree of modules. With our recent de-coupling from Storm, this has become all the more exaggerated. This is a good thing, however the benefit of the parent pom approach we currently have is

Re: [DISCUSS] Travis CI Build Time Limits

2019-05-23 Thread Michael Miklavcic
I think we're talking about killing the cache just 1 time for the purpose of verifying that the https:// repo endpoints will work. It's possible I'm misunderstanding how we've configured our caches, but it seems like we'd just need to kick travis 1x after the first failed build that did all the

Re: [DISCUSS] Build RPM/DEBs in Travis?

2019-05-22 Thread Michael Miklavcic
..@gmail.com) wrote: > > Yep, that's all I was doing. I'm in favor of adding it to Travis. > > On Wed, May 22, 2019 at 10:28 AM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > I think Justin was just giving rationale, not suggesting we shouldn't > ma

Re: [DISCUSS] Parser Aggregation in Management UI

2019-05-22 Thread Michael Miklavcic
n Wed, May 8, 2019 at 3:58 PM Otto Fowler > wrote: > > > You need to be a committer, that is all I think. > > I would not use the github UI for it though, I do it through the cli > > > > > > > > On May 8, 2019 at 09:45:24, Michael Miklavcic ( > michael.miklav...@g

Re: [DISCUSS] Build RPM/DEBs in Travis?

2019-05-22 Thread Michael Miklavcic
I think Justin was just giving rationale, not suggesting we shouldn't make the improvements. I think at one point we also had some ambiguity about whether or how Docker was going to run in Travis, but I believe some folks have found a path through that? I agree with you both - let's get this added

Master build - failed due to Maven download fail

2019-05-20 Thread Michael Miklavcic
FYI I kicked off a rebuild for master. For some reason the wget command to grab Maven 3.3.9 failed.

Re: [DISCUSS] JsonMapParser original string functionality

2019-05-14 Thread Michael Miklavcic
y. If on, > >> you get the original string in pristine condition. If off, no original > >> string is appended for those who care more about storage space. > >> > >> I can't think of a reason where one kind of parser would have a > different > >> orig

Re: [RESULT][VOTE] Metron Release Candidate 0.7.1-RC2

2019-05-13 Thread Michael Miklavcic
 On Mon, May 13, 2019, 8:28 PM Justin Leet wrote: > The vote has passed. Including my +1, the voting was: > 3 binding +1’s > no 0’s > no -1’s. > > I'll work on finishing out the release process tomorrow and will notify the > dev and user lists. >

Re: [VOTE] Metron Release Candidate 0.7.1-RC2

2019-05-10 Thread Michael Miklavcic
+1 binding Validated same as Nick. Mike On Thu, May 9, 2019 at 5:54 PM Nick Allen wrote: > +1 binding > > I validated the release tarball, ran the full test suite and validated the > CentOS 6 development environment. Everything looks solid. Let's ship it. > > On Wed, May 8, 2019 at 6:50 PM

Re: [VOTE] Metron Release Candidate 0.7.1-RC2

2019-05-10 Thread Michael Miklavcic
; I really enjoyed the retro, 3-digit vibe on that one. > > On Fri, May 10, 2019 at 4:38 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > "METRON-685" - wow, that one was a long time coming. > > > > On Thu, May 9, 2019 at 5:54 PM Nick

Re: [DISCUSS] JsonMapParser original string functionality

2019-05-10 Thread Michael Miklavcic
e720773cc/metron-platform/metron-parsing/metron-parsers-common/src/main/java/org/apache/metron/parsers/ParserRunnerImpl.java#L149-L158 > > On Fri, May 10, 2019 at 4:11 PM Otto Fowler > wrote: > > > +1 > > > > > > On May 10, 2019 at 13:57:55, Michael Miklavcic

Re: [VOTE] Metron Release Candidate 0.7.1-RC2

2019-05-10 Thread Michael Miklavcic
"METRON-685" - wow, that one was a long time coming. On Thu, May 9, 2019 at 5:54 PM Nick Allen wrote: > +1 binding > > I validated the release tarball, ran the full test suite and validated the > CentOS 6 development environment. Everything looks solid. Let's ship it. > > On Wed, May 8, 2019

[DISCUSS] JsonMapParser original string functionality

2019-05-10 Thread Michael Miklavcic
When adding the capability for parsing messages in the JsonMapParser using JSON Path expressions the original behavior for managing original strings was changed.

Re: [RESULT] [VOTE] Update dev guidelines with format for sharing architecture source files and rendered images

2019-05-08 Thread Michael Miklavcic
Michael Miklavcic < michael.miklav...@gmail.com> wrote: > Vote has passed. Voting was: > 2 binding +1's (Mike Miklavcic, Otto Fowler) > 1 non-binding +1 (Jon Zeolla) > > I will start making the changes to the dev guidelines today. > > Vote thread - > https:

Re: [DISCUSS] Metron Release - 0.7.1 next steps

2019-05-08 Thread Michael Miklavcic
ing it as both for quite some amount of > time. It *is* both dev and demo, and we recommend it as such on the list > all the time. > > So there isn’t a decision to be made here as far as the status quo -> we > use full dev as both dev and demo. > > > > > On May 2,

[RESULT] [VOTE] Update dev guidelines with format for sharing architecture source files and rendered images

2019-05-08 Thread Michael Miklavcic
Vote has passed. Voting was: 2 binding +1's (Mike Miklavcic, Otto Fowler) 1 non-binding +1 (Jon Zeolla) I will start making the changes to the dev guidelines today. Vote thread -

Re: [DISCUSS] Parser Aggregation in Management UI

2019-05-08 Thread Michael Miklavcic
ers together and do the job to handle this edge case on the > > backend accordingly. > > > > What do you think, guys? > > > > Hope this helps. > > > > Tamas > > > > On Tue, May 7, 2019 at 4:34 PM Michael Miklavcic < > > michael.miklav

Re: [DISCUSS] Parser Aggregation in Management UI

2019-05-07 Thread Michael Miklavcic
only used for the aggregation feature and doesn't go beyond > > that. > > > I think the way I worded that sentence may have caused confusion. I > just > > > meant we use it to manage more pieces of state within the aggregation > > > feature than just previous and current sta

Re: [DISCUSS] Parser Aggregation in Management UI

2019-05-07 Thread Michael Miklavcic
te within the aggregation > feature than just previous and current state of grouped parsers. > > On Fri, May 3, 2019 at 1:32 AM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > Shane, thanks for putting this together. The updates on the Jira are >

Re: [VOTE] Update dev guidelines with format for sharing architecture source files and rendered images

2019-05-03 Thread Michael Miklavcic
n" but I'm good either way. > > - Jon Zeolla > zeo...@gmail.com > > > On Fri, May 3, 2019 at 10:18 AM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > Yes, it is free James. We made sure of that in the original discussion. > > > > On Thu, May

Re: [DISCUSS] Full-dev role in PR testign

2019-05-03 Thread Michael Miklavcic
ive. It was sufficient for our > > purposes, but there might have to be compromises here. We depend on a lot > > more of the Hadoop stack. > > > > Migrating to JUnit 5 in general *should* be pretty easy. I don't think we > > really use any of the stuff that migrated in o

Re: [VOTE] Update dev guidelines with format for sharing architecture source files and rendered images

2019-05-03 Thread Michael Miklavcic
Yes, it is free James. We made sure of that in the original discussion. On Thu, May 2, 2019 at 9:33 PM James Sirota wrote: > i am ok with it as long as we are not forcing people to buy stuff > > 02.05.2019, 18:18, "Michael Miklavcic" : > > Here's the latest

Re: [DISCUSS] Full-dev role in PR testign

2019-05-03 Thread Michael Miklavcic
gt; > see https://faustxvi.github.io/junit5-docker/. My main hesitation > there > > is more around how much effort to migrate it is. I think that's almost > > certainly a cleaner long term solution, but I suspect the 80% solution of > > migrating what we have *might* be eas

[VOTE] Update dev guidelines with format for sharing architecture source files and rendered images

2019-05-02 Thread Michael Miklavcic
Here's the latest discussion on the subject: https://lists.apache.org/thread.html/0aa2b0b9ed4a0f0b0d8bb018c618e62de196565f9af71f347e504076@%3Cdev.metron.apache.org%3E I'd like to propose a vote to change our dev guidelines which will clarify the tooling we use to produce diagrams and share the

Re: [DISCUSS] Parser Aggregation in Management UI

2019-05-02 Thread Michael Miklavcic
Shane, thanks for putting this together. The updates on the Jira are useful as well. > (we used it for more than just that in this feature, but that was the initial reasoning) What are you using NgRx for in the submitted work that goes beyond the aggregation feature? On Thu, May 2, 2019 at

Re: [DISCUSS] Metron Release - 0.7.1 next steps

2019-05-02 Thread Michael Miklavcic
p. > > https://github.com/apache/metron/pull/1398 > > On Thu, May 2, 2019 at 1:37 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > Here's the ticket I created to track it, which also references the Jira > > for the new UI feature. > >

Re: [DISCUSS] Dev guideline changes for architecture diagrams

2019-05-02 Thread Michael Miklavcic
age and image-source as > opposed to diagram and diagram source? Isn’t it more descriptive? Is > there another reason ( like for doc gen )? > > > > On May 1, 2019 at 20:12:47, Michael Miklavcic (michael.miklav...@gmail.com > ) > wrote: > > Picking up where things left off

Re: [DISCUSS] Metron Release - 0.7.1 next steps

2019-05-02 Thread Michael Miklavcic
g/thread.html/ad206bdd59594cf74560770dfdbfcde0addd120d6fa8ea73f1a92a6b@%3Cdev.metron.apache.org%3E) it looks like we're in good shape for referencing past discussions for this feature and have any remaining gaps being covered. On Thu, May 2, 2019 at 7:31 AM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > I am still i

Re: [DISCUSS] Metron Release - 0.7.1 next steps

2019-05-02 Thread Michael Miklavcic
PR is up. I added the doc change to the metron-deployment README since this serves as the gateway doc for all the VM instances. All of which would be affected by the feature gap. https://github.com/apache/metron/pull/1398 On Thu, May 2, 2019 at 1:37 PM Michael Miklavcic < michael.mik

Re: [DISCUSS] Metron Release - 0.7.1 next steps

2019-05-02 Thread Michael Miklavcic
Here's the ticket I created to track it, which also references the Jira for the new UI feature. https://issues.apache.org/jira/browse/METRON-2100 On Thu, May 2, 2019 at 12:34 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > :-) > > I expect to have #2 out sometime to

Re: [DISCUSS] Metron Release - 0.7.1 next steps

2019-05-02 Thread Michael Miklavcic
rom a personal perspective like to see > what > his opinion is as the person who originally brought it up. >- Mike said he's going to kick out a PR that addresses #2 >- After that undergoes the normal review process and is merged, we >proceed normally and cut RC2. > &

Re: [DISCUSS] Metron Release - 0.7.1 next steps

2019-05-02 Thread Michael Miklavcic
nd scheme, if > we > > want to cut releases regularly, then we need to bear down and just do it. > > Case in point, I opened the initial discussion for this release on March > > 13th [1] and it is now May 2nd and we have yet to release 7 weeks later. > > > > -- >

Re: [DISCUSS] Metron Release - 0.7.1 next steps

2019-05-02 Thread Michael Miklavcic
mbari, then full-dev manages all the setup via REST. At > > that > > > point, we can easily tell everyone to just use the management UI. > > > > > > On Wed, May 1, 2019 at 7:23 AM Otto Fowler > > > wrote: > > > > > > > I think it would h

Re: [DISCUSS] Metron Release - 0.7.1 next steps

2019-05-02 Thread Michael Miklavcic
wo overlapping domains. I'd love to see parsers > > > ripped out of Ambari, then full-dev manages all the setup via REST. At > > that > > > point, we can easily tell everyone to just use the management UI. > > > > > > On Wed, May 1, 2019 at 7:23 AM Otto Fowle

Re: [DISCUSS] Full-dev role in PR testign

2019-05-01 Thread Michael Miklavcic
Justin, what are your thoughts on leveraging this approach along with long-lived Docker containers? I think the lifecycle would look like: 1. I need components A, B, C 2. If not started, start A, B, C 3. If started, clean/reset it 4. Setup pre-test state 5. Run test(s) 6.

Re: [DISCUSS] Full-dev role in PR testign

2019-05-01 Thread Michael Miklavcic
> Our general procedures. Should we not be requiring full-dev on all PRs, but maybe just asking for a justification why not (e.g. "I don't need full-dev for this, because I added unit tests here and here and integration tests for the these components?"). And then a reviewer can request a full-dev

[DISCUSS] Dev guideline changes for architecture diagrams

2019-05-01 Thread Michael Miklavcic
Picking up where things left off in the VOTE thread on the subject, I'm presenting a revision to my original proposal below. I'd like to get this signed off on before submitting it for another vote. Otto, picking up where we left off, let me know if this looks good to you. I'd like to propose a

[DISCUSS] Metron Release - 0.7.1 next steps

2019-04-30 Thread Michael Miklavcic
The vote for RC1 did not pass and I'd like to kickstart some discussion about what we should do. I started taking a look at PR#1360 and it looks like this isn't quite as close to being able go in as I had originally expected. I want to talk about options here. It seems to me that we can: 1.

Re: [RESULT][VOTE] Metron Release Candidate 0.7.1-RC1

2019-04-30 Thread Michael Miklavcic
btw, it's 2 "binding" -1’s (Otto, Mike) I started taking a look at that PR and it looks like this isn't quite as close to being able go in as I had originally expected. I want to talk about options here. It seems to me that we can: 1. Wait for PR#1360 to go in, but this is likely going to

Re: [VOTE] Update dev guidelines with format for sharing architecture source files and rendered images

2019-04-30 Thread Michael Miklavcic
Based on some requested changes from Otto, I'm pulling the vote on this. I haven't had a chance to make the revisions yet, and will restart the vote when I have a chance to get those finished. On Fri, Apr 26, 2019 at 1:27 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > T

Re: [VOTE] Metron Release Candidate 0.7.1-RC1

2019-04-29 Thread Michael Miklavcic
; > > ensuring we get that PR cleaned up and in and/or additional > > documentation > > > > regarding the current limitations of this feature. > > > > > > > > > > > > On Sat, Apr 27, 2019 at 2:38 PM Anand Subramanian < > > > > asubraman...@hortonw

Re: [VOTE] Metron Release Candidate 0.7.1-RC1

2019-04-27 Thread Michael Miklavcic
t; * Management UI, Alerts UI and Swagger UI sanity check > > Regards, > Anand > > On 4/26/19, 5:18 AM, "Nick Allen" wrote: > > +1 Verified release with all documented steps and ran up Full Dev. > > On Thu, Apr 25, 2019 at 6:10 PM Michael Miklavcic < > michae

Re: [VOTE] Update dev guidelines with format for sharing architecture source files and rendered images

2019-04-26 Thread Michael Miklavcic
That works too On Fri, Apr 26, 2019, 12:20 PM Otto Fowler wrote: > When I say module, I mean at the root of any module, so each module could > have it’s own diagrams. > And the project root wold have diagrams for the ‘overall' > > > On April 26, 2019 at 13:44:30,

Re: [VOTE] Update dev guidelines with format for sharing architecture source files and rendered images

2019-04-26 Thread Michael Miklavcic
would be that the artifacts are split from their respective modules, but I honestly don't see that as a problem. On Fri, Apr 26, 2019, 11:40 AM Otto Fowler wrote: > On April 26, 2019 at 13:19:05, Michael Miklavcic ( > michael.miklav...@gmail.com) wrote: > > @otto when I get your respons

Re: [VOTE] Update dev guidelines with format for sharing architecture source files and rendered images

2019-04-26 Thread Michael Miklavcic
there > > I'm happy to add this as well, +1 to that. > > On April 25, 2019 at 12:57:47, Michael Miklavcic ( > michael.miklav...@gmail.com) wrote: > > I'd like to propose a vote to change our dev guidelines which will clarify > the tooling we use to produce diagrams and sha

Re: [VOTE] Metron Release Candidate 0.7.1-RC1

2019-04-25 Thread Michael Miklavcic
Just curious, did you also do AWS? I haven't run that in a while. Wondered if it worked ok. On Thu, Apr 25, 2019, 5:48 PM Nick Allen wrote: > +1 Verified release with all documented steps and ran up Full Dev. > > On Thu, Apr 25, 2019 at 6:10 PM Michael Miklavcic < > michael.mikl

Re: [VOTE] Metron Release Candidate 0.7.1-RC1

2019-04-25 Thread Michael Miklavcic
+1, I verified everything in the outlined steps except for AWS. On Thu, Apr 25, 2019 at 2:32 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > We're also not "incubator-metron" any longer. Do we require any kind of > voting or +1 on that verification pa

Re: [VOTE] Metron Release Candidate 0.7.1-RC1

2019-04-25 Thread Michael Miklavcic
We're also not "incubator-metron" any longer. Do we require any kind of voting or +1 on that verification page to make corrections to it? On Thu, Apr 25, 2019 at 2:29 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > fyi, the steps in this doc have changed sligh

Re: [VOTE] Metron Release Candidate 0.7.1-RC1

2019-04-25 Thread Michael Miklavcic
fyi, the steps in this doc have changed slightly per this naming convention change as well - https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds. On Thu, Apr 25, 2019 at 1:25 PM Justin Leet wrote: > For everyone taking the time to validate and vote on the RC, there is a >

[VOTE] Update dev guidelines with format for sharing architecture source files and rendered images

2019-04-25 Thread Michael Miklavcic
I'd like to propose a vote to change our dev guidelines which will clarify the tooling we use to produce diagrams and share the source files for those diagrams. The original discuss thread is noted at the end of this email. I propose the dev guidelines

Re: [DISCUSS] Next Release

2019-04-25 Thread Michael Miklavcic
0.7.1 Ryan > > Merriman > > METRON-2067Done 0.7.1 Michael > > Miklavcic > > METRON-2074Done 0.7.1 Michael > > Miklavcic > > METRON-2082Done

Re: [DISCUSS] Upgrade to HDP 3.1.0

2019-04-22 Thread Michael Miklavcic
+1 to starting a feature branch for this. +1 to removing our custom implementations if the newest revs are in fact stable now. Regarding the profile option - if it's possible to keep 2.6.5 for a bit and not require separate branches or code trees, this is probably OK. Otherwise, I'm inclined to

Re: [DISCUSS] Next Release

2019-04-17 Thread Michael Miklavcic
release. The last build failure on master I see was from 21 days > > ago. > > > > Otherwise, I'd really like to kick off the next release. > > > > On Tue, Apr 16, 2019 at 11:25 AM Michael Miklavcic < > > michael.miklav...@gmail.com> wrote: > >

Re: [DISCUSS] Stellar MAP data type features

2019-04-16 Thread Michael Miklavcic
another at this point. I'm inclined to focus on https://issues.apache.org/jira/browse/METRON-2072 first in order to enable some additional in-memory enrichment use cases. Feedback welcome. On Fri, Apr 12, 2019 at 11:01 AM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > Fol

[DISCUSS] Format for sharing architecture source files and rendered images

2019-04-16 Thread Michael Miklavcic
Hi devs, We've talked about this fairly ad hoc on PRs before, and I thought it worth bringing up more formally. Architecture diagrams are a strategically useful addition to the core developer documentation. Examples include: 1.

Re: [DISCUSS] Next Release

2019-04-16 Thread Michael Miklavcic
FYI, I just saw one of our reported intermittent test failures pop up again today - https://issues.apache.org/jira/browse/METRON-1814 On Mon, Apr 15, 2019 at 2:22 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > I want to thread the needle on this. I just reviewed a PR

Re: [DISCUSS] Next Release

2019-04-15 Thread Michael Miklavcic
n's comments in > > > > > > > https://lists.apache.org/thread.html/50b89b919bd8bef3f7fcdef167cbd7e489fa74a1e2da3e4fddb08b13@ > > > > > > Jon Zeolla > > > > On Thu, Mar 28, 2019, 2:16 PM Michael Miklavcic < > > michael.miklav...@gmail.com&g

Re: Problems with site book generation

2019-04-14 Thread Michael Miklavcic
Hey Dale, Try 'mvn clean site' - there might be some errant files hanging out there that are ignored by git. You might also do a 'mvn clean' from the root dev dir prior to building the site-book as well. Best, Mike On Sun, Apr 14, 2019 at 3:08 AM Dale Richardson wrote: > Just running a

Re: [DISCUSS] Stellar MAP data type features

2019-04-12 Thread Michael Miklavcic
the REDUCE function to this purpose - https://github.com/apache/metron/tree/master/metron-stellar/stellar-common#reduce . Best, Mike On Fri, Apr 5, 2019 at 2:04 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > Hi all, > > We have a number of data structures and function

Re: [DISCUSS] Real-time processing engine: Storm, Spark, Flink or Cloud Native

2019-04-11 Thread Michael Miklavcic
Hi Ali, Thank you for taking the time to share your experiences with us. I've been thinking about this a while now and wanted to take some time for reflection before responding. I need to kick out a proper dev list DISCUSS thread on this, but if you've seen a couple of the recent refactoring

Re: Problems with Dev deployment.

2019-04-10 Thread Michael Miklavcic
ill exist to link to, but aren't actually linked within the > site-book. > > On Wed, Apr 10, 2019 at 12:08 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > Dale - to spin back around to what would best help you, for now you could > > reference docs

Re: Problems with Dev deployment.

2019-04-10 Thread Michael Miklavcic
as long as you're seeing version 0.7.0 in the menu bar, you're good to go. No need to traverse the READMEs or build locally. Just follow the TOC there. Best, Mike On Wed, Apr 10, 2019 at 10:00 AM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > So does anyone know, have we

Re: Problems with Dev deployment.

2019-04-10 Thread Michael Miklavcic
buntu/Debian single-node VM with Vagrant and Ambari - > > Metron - Apache Software Foundation - cwiki.apache.org< > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68718548> > > Contributed by Umesh Kaushik < umesh.kaus...@bhujang.net > Introducti

Re: Problems with Dev deployment.

2019-04-09 Thread Michael Miklavcic
Where are you seeing 2.0.0.2 for Ansible? Should be 2.4.0+ now. https://github.com/apache/metron/blob/master/metron-deployment/development/centos6/README.md On Tue, Apr 9, 2019, 6:59 PM Michael Miklavcic wrote: > That would be awesome man! Yes, Jira tickets for every change. I don't >

Re: Problems with Dev deployment.

2019-04-09 Thread Michael Miklavcic
That would be awesome man! Yes, Jira tickets for every change. I don't recall off the top of my head the version - I'll have to look when I get back in front of a computer. On Tue, Apr 9, 2019, 6:56 PM Dale Richardson wrote: > Hi Michael, > Yep that was the issue, looks like "brew install

Re: Problems with Dev deployment.

2019-04-07 Thread Michael Miklavcic
Your ansible version is pulling in Python 3, which won't work. Try installing ansible with pip. On Sun, Apr 7, 2019, 6:33 AM Dale Richardson wrote: > Hi Folks, > I've been walking through some of the documentations on building and > deploying the Metron development image. > I am building Metron

[DISCUSS] Stellar MAP data type features

2019-04-05 Thread Michael Miklavcic
Hi all, We have a number of data structures and functions available via Stellar for manipulating those structures. For example, we provide the following functions for initializing and working with Bloom filters: [Stellar]>>> ?BLOOM_ ?BLOOM_ADD ?BLOOM_EXISTS ?BLOOM_INIT ?BLOOM_MERGE For Maps,

Re: [DISCUSS] Reverting METRON-2023 (PR #1364)

2019-04-03 Thread Michael Miklavcic
+1 from me also On Wed, Apr 3, 2019 at 6:18 AM Nick Allen wrote: > +1 > Thanks for digging into this. > > On Wed, Apr 3, 2019, 6:07 AM Shane Ardell > wrote: > > > Hello everyone, > > > > I'm sending out this email to notify the community of my intention to > > revert the commit for METRON-2023

Re: [DISCUSS] Ambari MPacks, Metron upgrades

2019-03-29 Thread Michael Miklavcic
ikely > more time will be used on mpack? > > Can you elaborate? > > > > On March 29, 2019 at 13:12:53, Michael Miklavcic ( > michael.miklav...@gmail.com) wrote: > > Hi devs, > > I wanted to kickstart a high level discussion on what the future of > de

[DISCUSS] Ambari MPacks, Metron upgrades

2019-03-29 Thread Michael Miklavcic
Hi devs, I wanted to kickstart a high level discussion on what the future of development on Ambari MPacks and upgrades will/should look like ongoing for Metron. The long and the short of it is that I don't think there is likely to be much more time invested in enhancing, upgrading, or expanding

Re: Slack channel access

2019-03-28 Thread Michael Miklavcic
Sure thing Wendy, welcome! On Thu, Mar 28, 2019 at 3:59 PM Wendy Edwards wrote: > Could I also be added to the Metron slack channel? > > Thanks, > Wendy > > On Thu, Mar 28, 2019 at 4:37 PM Yerex, Tom wrote: > > > Good afternoon, > > > > > > > > Please add me to the Metron Slack Channel. > > >

Re: Slack channel access

2019-03-28 Thread Michael Miklavcic
Hi Tom, check your inbox for the invite. On Thu, Mar 28, 2019 at 3:37 PM Yerex, Tom wrote: > Good afternoon, > > > > Please add me to the Metron Slack Channel. > > > > Thank you. > > >

Re: [DISCUSS] Next Release

2019-03-28 Thread Michael Miklavcic
; > > We should likely get METRON-2014 in, based on > > > > > https://lists.apache.org/thread.html/13bd0ed5606ad4f3427f24a8e759d6bcb61ace76d4afcc9f48310a00@%3Cdev.metron.apache.org%3E > > > > On Thu, Mar 14, 2019 at 4:24 PM Michael Miklavcic < > > michael

Re: [DISCUSS] Next Release

2019-03-14 Thread Michael Miklavcic
Good catch On Thu, Mar 14, 2019 at 2:53 PM zeo...@gmail.com wrote: > We should likely get METRON-2014 in, based on > > https://lists.apache.org/thread.html/13bd0ed5606ad4f3427f24a8e759d6bcb61ace76d4afcc9f48310a00@%3Cdev.metron.apache.org%3E > > On Thu, Mar 14, 2019 at 4:24 PM Mi

Re: [DISCUSS] Next Release

2019-03-14 Thread Michael Miklavcic
t if someone > comes up with something, it's not hard to pivot. > > On Wed, Mar 13, 2019, 13:08 Michael Miklavcic > > wrote: > > > I'd like to see this fixed for the next release. > > https://issues.apache.org/jira/browse/METRON-2036. Even though it's a >

Re: [DISCUSS] Next Release

2019-03-13 Thread Michael Miklavcic
I'd like to see this fixed for the next release. https://issues.apache.org/jira/browse/METRON-2036. Even though it's a non-prod issue, this is a core part of our infrastructure/development lifecycle that is currently broken and fits with our previous agreements of holding a release until all

Re: Issues with Metron profiler

2019-03-12 Thread Michael Miklavcic
Hi Anil, Can you share your profile definition? For the STATS_BIN issue you're seeing, it seems like your bins are wrong, however you're using the pre-defined DECILE bin list, which is unit tested (and the tests are passing). It may be that the stats object isn't quite right, but I can't tell

Re: [DISCUSS] Upgrading HBase and Kafka support

2019-03-08 Thread Michael Miklavcic
I'm -1 on #1 unless there's some desperately compelling reason to go that route. It would be a regression in our test coverage, and at that point it's really just duplicating our unit tests as opposed to checking our integration. I'm good with 3. Gating factors for a successful implementation

Re: [DISCUSS] Upgrading HBase and Kafka support

2019-03-08 Thread Michael Miklavcic
Jon, I didn't get a chance to review your PR, but it immediately came to mind wrt this task - did you guys have that working in Travis with Maven? What's the container lifecycle you came up with look like? On Fri, Mar 8, 2019 at 8:30 AM zeo...@gmail.com wrote: > +1 to option 3 on both. Also

Re: [DISCUSS] Central Navigation for Alerts and Management UI

2019-03-05 Thread Michael Miklavcic
The original design was done with the intent to keep the user profiles (soc analyst vs ops personnel) separate and enable a microservices-oriented architecture. I don't have a strong opinion one way or the other, but I'd be interested to hear whether others in the community find this wall useful,

Re: [DISCUSS] Architecture documentation

2019-02-26 Thread Michael Miklavcic
gt; > > > I feel like the code itself is pretty well documented. I updated existing > > javadocs and added javadocs to classes that didn't have them before this > > PR. In my opinion the level of documentation for these classes has > > increased significantly. > > &

Re: [DISCUSS] Charset PR merge effects

2019-02-25 Thread Michael Miklavcic
Justin, I like your proposal. I just had a couple thoughts that I think are worth pointing out. I'm not 100% clear about this point, but I don't believe we currently handle the possibility of various external data sources with heterogeneous encoding schemes in our Kafka topics. As it stands, I

Re: [DISCUSS] Architecture documentation

2019-02-25 Thread Michael Miklavcic
Tentatively agreed on further clarification of what we consider in/out of scope for documentation re: document something that wasn't documented before. Ryan, can you give a quick summary of what you *have* added/updated in documentation on this PR vs what you want to leave out? My initial concern

Re: [DISCUSS] Writer class refactor

2019-01-18 Thread Michael Miklavcic
Thanks for the write up, Ryan. I had to touch on some of this when refactoring the kafka writer away from the async model so we could guarantee delivery. We had potential to drop messages before that change because of the async producer calls, which would ack the Storm tuple as soon as the writer

Re: [DISCUSS] Metron documentation improvements

2018-12-21 Thread Michael Miklavcic
et to ensure you're not > checking domains linked from our docs and turn travis into google. ;) > > > On Thu, Dec 20, 2018 at 3:19 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > Well golly, I love this. I'll give that a whirl! > > > > On Th

Re: [DISCUSS] Metron documentation improvements

2018-12-20 Thread Michael Miklavcic
we want to go. I'm a +0 > > On Thu, Dec 20, 2018 at 2:47 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > We recently had our site-book doc generation break due to our not > including > > it in the Travis build. The fix for a broken build seems si

[DISCUSS] Metron documentation improvements

2018-12-20 Thread Michael Miklavcic
We recently had our site-book doc generation break due to our not including it in the Travis build. The fix for a broken build seems simple enough - add it to our build process and assuming it doesn't cause build timeout issues, we should be good to go. Beyond that, there are additional issues

Re: Site-book broken in master

2018-12-20 Thread Michael Miklavcic
+1 for merging a fix. > > > > On December 20, 2018 at 12:43:57, Michael Miklavcic ( > michael.miklav...@gmail.com) wrote: > > Thanks for the feedback Otto. I'm not too familiar with our infrastructure > afa impact of switching to ascii doc, but I think it's worth consi

Re: Site-book broken in master

2018-12-20 Thread Michael Miklavcic
Fowler (ottobackwa...@gmail.com) > wrote: > > What would help is an automated test the tests all the links as well. > > > > On December 19, 2018 at 19:55:03, Michael Miklavcic ( > michael.miklav...@gmail.com) wrote: > > I'd also like to submit a call for suggestions for making

Re: Site-book broken in master

2018-12-19 Thread Michael Miklavcic
seen at least a few PR's in recent memory unwittingly fall into this trap. On Wed, Dec 19, 2018 at 5:47 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > Fix is out in this PR: https://github.com/apache/metron/pull/1309 > Jira: https://issues.apache.org/jira/browse

Re: Site-book broken in master

2018-12-19 Thread Michael Miklavcic
should serve as a recipe for remedying them in the future. Best, Mike On Wed, Dec 19, 2018 at 3:46 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > FYI, site-book is currently broken in master. I think I've found the > source of the issue and will be issuing a fix short

Site-book broken in master

2018-12-19 Thread Michael Miklavcic
FYI, site-book is currently broken in master. I think I've found the source of the issue and will be issuing a fix shortly. Best, Mike

Re: [DISCUSS] Handling dropped messages in REGEX_SELECT with Kafka topic routing

2018-12-19 Thread Michael Miklavcic
ped messages otherwise they'll be in > a replay loop. Not acking is a flat-out bug IMO. > > On Wed, Dec 19, 2018 at 2:37 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > When a message is filtered by the message filtering mechanism, we > > explicitly

[DISCUSS] Handling dropped messages in REGEX_SELECT with Kafka topic routing

2018-12-19 Thread Michael Miklavcic
When a message is filtered by the message filtering mechanism, we explicitly drop the message (and presumably ack it in Storm), as explained here - https://github.com/apache/metron/tree/master/metron-platform/metron-parsing#filtered. When using the REGEX_SELECT field transformation (see here -

Re: [ANNOUNCE] Apache Metron release 0.7.0

2018-12-17 Thread Michael Miklavcic
And a big thanks to Justin Leet for being our release manager. Great work Justin! On Mon, Dec 17, 2018 at 10:07 AM Justin Leet wrote: > Hi all, > > I’m pleased to announce the release of Metron 0.7.0! There's been a lot > of work on improvements, upgrades, discussion, and more. Thanks to

Re: [DISCUSS] Mandatory relocation of Apache git repositories on git-wip-us.apache.org

2018-12-10 Thread Michael Miklavcic
If it's the option I'm picturing, I've used that for merging in a PR against my PR. It's pretty simple and I don't recall it allowing options for fitting our format. Unless I'm mistaken or that has changed, I'd recommend our current workflow. On Mon, Dec 10, 2018 at 3:26 PM Justin Leet wrote: >

Re: [DISCUSS] Remove `/api/v1/update/replace` endpoint

2018-12-07 Thread Michael Miklavcic
+1 to that Nick On Fri, Dec 7, 2018 at 11:13 AM Nick Allen wrote: > Heads up - I have not received a response from anyone for about 3 days. I > am going to take that as no one has a problem with this change. I will > likely merge the PR today. > > > > On Tue, Dec 4, 2018 at 4:05 PM Nick Allen

Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?

2018-12-03 Thread Michael Miklavcic
I have one more intermittent failure to add to the list from a timeout in the profiler integration tests. https://issues.apache.org/jira/browse/METRON-1918 On Mon, Dec 3, 2018 at 2:54 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > fwiw, I have not been able to

Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?

2018-12-03 Thread Michael Miklavcic
d before we're set to move forward with a core > release (or we choose not to fix it, given the current version is > affected), I'm happy to put out a new RC. > > On Mon, Dec 3, 2018 at 4:12 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > +1 Nick > &

  1   2   3   4   >