Re: Travis failing due to same Maven issue

2019-06-13 Thread Nick Allen
I am not sure if this will actually solve the issue, but here is another attempt. https://github.com/apache/metron/pull/1442 On Wed, Jun 12, 2019 at 9:31 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > I've seen this now with 2 back-to-back commits to master that end with >

Re: Storm 1.0.x eol

2019-06-04 Thread Nick Allen
As part of the upgrade to HDP 3.1, we should consider ending all Metron support for Storm 1.0.x. We have a few build options and workarounds in the code base (see storm-kafka-override, Maven profile HDP-1.5.0.0, etc) that would need cleaned up. On Mon, Jun 3, 2019 at 5:39 PM Otto Fowler wrote:

Re: [DISCUSS] Metron RPM spec file changelog

2019-05-31 Thread Nick Allen
I am also in favor of not maintaining a separate change log; that is what git is for. We just need to ensure that the rpmlint check can be satisfied in some way if we do not maintain the change log. On Fri, May 31, 2019 at 1:56 PM Ryan Merriman wrote: > I vote we get rid of them. It's easy

Re: Master build - failed due to Maven download fail

2019-05-24 Thread Nick Allen
https://github.com/apache/metron/pull/1433 On Fri, May 24, 2019 at 12:07 PM Nick Allen wrote: > FYI, I just created this to track the issue. > > https://issues.apache.org/jira/browse/METRON-2143 > > On Mon, May 20, 2019 at 6:39 PM Justin Leet wrote: > >> I saw

Re: Master build - failed due to Maven download fail

2019-05-24 Thread Nick Allen
FYI, I just created this to track the issue. https://issues.apache.org/jira/browse/METRON-2143 On Mon, May 20, 2019 at 6:39 PM Justin Leet wrote: > I saw the same thing on https://github.com/apache/metron/pull/1407, but > bouncing Travis fixed it. Not sure if it's a connection issue or what,

Re: [DISCUSS] Travis CI Build Time Limits

2019-05-23 Thread Nick Allen
ink we should be extra vigilant on getting >those build/test times *down*, rather than enabling them to increase. > Sure, >there are other options for solving that problem, but Travis is a simple >equalizer because it's impartial to whatever local hardware engineers >

Re: [DISCUSS] Build RPM/DEBs in Travis?

2019-05-22 Thread Nick Allen
uilding them, because nobody is going to run both unless they specifically > done something they'd expect to affect both. > > On Wed, May 22, 2019 at 9:30 AM Nick Allen wrote: > > > In light of issues like this https://github.com/apache/metron/pull/1419, > > has anyone looked into

Re: [DISCUSS] Travis CI Build Time Limits

2019-05-22 Thread Nick Allen
FYI - Here is a POC build of this concept running. This only runs the Parser and Enrichment integration tests, but as separate jobs in Travis. https://travis-ci.org/nickwallen/metron/builds/535791047 On Wed, May 22, 2019 at 9:20 AM Nick Allen wrote: > > > Justin Leet said

[DISCUSS] Build RPM/DEBs in Travis?

2019-05-22 Thread Nick Allen
In light of issues like this https://github.com/apache/metron/pull/1419, has anyone looked into building our RPMs and DEBs in Travis? This is a very common and easy mistake to make and our CI builds really should be able to catch this.

[DISCUSS] Travis CI Build Time Limits

2019-05-22 Thread Nick Allen
> Justin Leet said [1]: Looking at our build times, I'm actually concerned that if we kill the caches our builds won't complete. The integration tests take >45 minutes and it's very possible redownloading everything goes over our

Re: [DISCUSS] JsonMapParser original string functionality

2019-05-10 Thread Nick Allen
Fri, May 10, 2019 at 5:53 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > I think that's an excellent idea. Can anyone think of a situation where we > wouldn't want to add this the same way for all parsers? I suppose we could > always allow this to be overridden, also. &g

Re: [DISCUSS] JsonMapParser original string functionality

2019-05-10 Thread Nick Allen
I think maintaining the integrity of the original data makes a lot of sense for any parser. And ideally the original string should be what came out of Kafka with only the minimally necessary processing. With that in mind, we could solve this one level up. Instead of relying on each parser to do

Re: [VOTE] Metron Release Candidate 0.7.1-RC2

2019-05-10 Thread Nick Allen
I really enjoyed the retro, 3-digit vibe on that one. On Fri, May 10, 2019 at 4:38 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > "METRON-685" - wow, that one was a long time coming. > > On Thu, May 9, 2019 at 5:54 PM Nick Allen wrote: > > &

Re: [VOTE] Metron Release Candidate 0.7.1-RC2

2019-05-09 Thread Nick Allen
+1 binding I validated the release tarball, ran the full test suite and validated the CentOS 6 development environment. Everything looks solid. Let's ship it. On Wed, May 8, 2019 at 6:50 PM Justin Leet wrote: > This is a call to vote on releasing Apache Metron 0.7.1 > > Full list of changes

Re: [DISCUSS] Parser Aggregation in Management UI

2019-05-06 Thread Nick Allen
Have you considered creating a feature branch for the effort? This would allow you to break the effort into chunks, where the result of each PR may not be a fully working "master-ready" result. I am sure you guys tackled the work in chunks when developing it, so consider just replaying those

Re: [DISCUSS] Full-dev role in PR testign

2019-05-03 Thread Nick Allen
I'm exploring the use of TestContainers right now as part of the HDP 3.1 effort. Still exploring feasibility, but it is looking promising. On Fri, May 3, 2019 at 10:46 AM Justin Leet wrote: > I think everything Casey mentioned is a good call-out as things start to > build into specifics. I

Re: [DISCUSS] Metron Release - 0.7.1 next steps

2019-05-02 Thread Nick Allen
I think any open source project needs to strive to cut releases regularly. This is healthy for the project and community. It gets new features and functionality out to the community so we can get feedback, find what is working and what is not, iterate and improve. You probably agree with this.

Re: [DISCUSS] Metron Release - 0.7.1 next steps

2019-05-02 Thread Nick Allen
To echo Justin's comments, I am in favor of #2, which provides a clear, well-defined path to a release. - Why hold back a release, especially a point release containing 89 improvements, for one issue that will not affect most users? - It is one thing to stall a release to address a bug

Re: [VOTE] Metron Release Candidate 0.7.1-RC1

2019-04-28 Thread Nick Allen
an ( > > > asubraman...@hortonworks.com) wrote: > > > > > > +1 (non-binding) > > > > > > * Built RPMs and mpacks. > > > * Brought up Metron stack on 12-node CentOS 7 openstack cluster. > > > * Ran sensor-st

Re: [VOTE] Metron Release Candidate 0.7.1-RC1

2019-04-25 Thread Nick Allen
No, not AWS. On Thu, Apr 25, 2019, 8:49 PM Michael Miklavcic wrote: > Just curious, did you also do AWS? I haven't run that in a while. Wondered > if it worked ok. > > On Thu, Apr 25, 2019, 5:48 PM Nick Allen wrote: > > > +1 Verified release with all documented step

Re: [VOTE] Metron Release Candidate 0.7.1-RC1

2019-04-25 Thread Nick Allen
r 25, 2019 at 3:45 PM Nick Allen wrote: > > > No voting required. Those are just docs. Whoever is willing to correct > > and has access, should be able to. Good catch. > > > > On Thu, Apr 25, 2019 at 4:32 PM Michael Miklavcic < > > michael.miklav...@gmail.

Re: [VOTE] Metron Release Candidate 0.7.1-RC1

2019-04-25 Thread Nick Allen
No voting required. Those are just docs. Whoever is willing to correct and has access, should be able to. Good catch. On Thu, Apr 25, 2019 at 4:32 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > We're also not "incubator-metron" any longer. Do we require any kind of > voting or

Re: [DISCUSS] Next Release

2019-04-25 Thread Nick Allen
0.7.1 Anand Subramanian METRON-2038Done 0.7.1 Nick Allen METRON-2035Done 0.7.1 Nick Allen METRON-2041Done 0.7.1 Michael Miklavcic METRON-2036

Re: [DISCUSS] Next Release

2019-04-23 Thread Nick Allen
a list of tickets to update to the appropriate contributors. > > Thanks, > Justin > > > > On Tue, Apr 23, 2019, 09:54 Nick Allen wrote: > > > Justin - Can you kick-off the release process? > > > > On Wed, Apr 17, 2019 at 11:51 AM Michael Miklavcic < > &

Re: [DISCUSS] Upgrade to HDP 3.1.0

2019-04-23 Thread Nick Allen
ome possible risk if Ambari and the individual components introduce >changes here) > > Fortunately, Zookeeper appears to have stayed the same across versions. It > might be worthwhile to get a chart of the versions for each platform added > to the epic Jira for reference while performi

Re: [DISCUSS] Next Release

2019-04-23 Thread Nick Allen
m first. Having reviewed > the test-failures list, I'm in favor of moving forward with the release. > > On Wed, Apr 17, 2019 at 9:29 AM Nick Allen wrote: > > > Oh, and that failure from 21 days ago was caused by a downstream issue > > brought in by PR #1364, that we then later

[DISCUSS] Upgrade to HDP 3.1.0

2019-04-22 Thread Nick Allen
We currently support running Metron on an HDP 2.6.5 cluster. I'd like to get Metron updated to run in an HDP 3.1.0

Re: [DISCUSS] Next Release

2019-04-17 Thread Nick Allen
/cad679a5948ce0ee9866e61bbf75b1f5f255682c On Wed, Apr 17, 2019 at 11:25 AM Nick Allen wrote: > Are you suggesting the release should wait on this? Personally, I don't > feel that we have any *persistently intermittent* test failures that would > block a release. The last build failure on master I see

Re: [DISCUSS] Next Release

2019-04-17 Thread Nick Allen
to complete prior to a > >> release > >> > and I agree with Justin's comments in > >> > > >> > > >> > > >> > https://lists.apache.org/thread.html/50b89b919bd8bef3f7fcdef167cbd7e489fa74a1e2da3e4fddb08b13@ > >> > >

Re: Problems with Dev deployment.

2019-04-11 Thread Nick Allen
That script (metron-deployment/scripts/platform-info.sh) should be solid. I was going to suggest that you run that on your computer and send us the output. That has often helped us debug issues like this before. If it is not finding Node/NPM, then that is a problem that needs addressed. On

Re: [DISCUSS] Reverting METRON-2023 (PR #1364)

2019-04-03 Thread Nick Allen
+1 Thanks for digging into this. On Wed, Apr 3, 2019, 6:07 AM Shane Ardell wrote: > Hello everyone, > > I'm sending out this email to notify the community of my intention to > revert the commit for METRON-2023 (PR #1364 > ). This commit introduced a >

Re: [DISCUSS] Next Release

2019-03-28 Thread Nick Allen
e/METRON-2036. Even though it's > a > > > > non-prod issue, this is a core part of our infrastructure/development > > > > lifecycle that is currently broken and fits with our previous > > agreements > > > of > > > > holding a release until all int

[DISCUSS] Next Release

2019-03-13 Thread Nick Allen
I would like to open a discussion in regards to the next release. Our last 0.7.0 release was on Dec 11th. I believe we have a significant number of bug fixes and performance improvements that would make a worthy point release; 0.7.1. Although, we should review the change log and see if there are

Re: [DISCUSS] Upgrading HBase and Kafka support

2019-03-08 Thread Nick Allen
+1 for option 3. I am in favor of using Docker for the integration tests for all the reasons that you mentioned. On Fri, Mar 8, 2019 at 9:47 AM Ryan Merriman wrote: > I have been researching the effort involved to upgrade to HDP 3. Along the > way I've found a couple challenging issues that

Re: [DISCUSS] Architecture documentation

2019-02-26 Thread Nick Allen
And just to be clear, this is just a continuation of the discussion in this thread. This is not in any way a blocker for Ryan's PR, of course. On Tue, Feb 26, 2019 at 1:44 PM Nick Allen wrote: > > We could also enhance > "metron/dev-utilities/release-utils/validate-jira-f

Re: [DISCUSS] Architecture documentation

2019-02-26 Thread Nick Allen
rchitecture.md" in metron root that replaces this - > > > > > > > > > > > https://github.com/apache/metron/blob/d7d4fd9afb19e2bd2e66babb7e1514a19eae07d0/README.md#navigating-the-architecture > > > and covers the broad architecture with lin

Re: [DISCUSS] Architecture documentation

2019-02-25 Thread Nick Allen
I don't think we should hold up this work to document something that wasn't previously documented. A follow-on is sufficient. On Mon, Feb 25, 2019 at 8:50 AM Ryan Merriman wrote: > Recently I submitted a PR > that > introduces a large number of

Re: [DISCUSS] Architecture documentation

2019-02-25 Thread Nick Allen
> Procedurally, do we have a way to ensure that any follow-on documentation > happens prior to a release (anything in the wiki, etc.)? If someone thinks the code base needs X before the next release, then they can bring up X during the release discussion. We don't need additional procedure

Re: [VOTE] Metron Release Candidate 0.7.0-RC1

2018-12-12 Thread Nick Allen
+1 binding - All of the tarballs, checksums, and signatures are correct - All of the tests and integration tests ran successfully. - The release also spun-up correctly in the development environment. FYI - I had to slightly modify the metron-rc-check script for this to work. See the

Re: [VOTE] Metron Release Candidate 0.7.0-RC1

2018-12-11 Thread Nick Allen
having a > script to publish just the KEYS file?) . Given that this causes problems > with the RC check, it seems like we need to update something one way or the > other. It might just be updating the RC check script in the short term. > > > On Tue, Dec 11, 2018 at 3:20 PM Nic

Re: [VOTE] Metron Release Candidate 0.7.0-RC1

2018-12-11 Thread Nick Allen
Should there be a KEYS file with the release candidate at https://dist.apache.org/repos/dist/dev/metron/0.7.0-RC1/KEYS? Or was that changed to https://dist.apache.org/repos/dist/release/metron/KEYS ? ``` $ ~/Development/metron/dev-utilities/release-utils/metron-rc-check --version=0.7.0

Re: [DISCUSS] Mandatory relocation of Apache git repositories on git-wip-us.apache.org

2018-12-10 Thread Nick Allen
1 is created for updating the scripts with the new GitBox > > location. I'll start with this once the repositories are moved. > > > > - Roy > > > > Op ma 10 dec. 2018 om 15:52 schreef Nick Allen : > > > >> +1 Thanks for the heads up. We will do whatev

Re: [DISCUSS] Mandatory relocation of Apache git repositories on git-wip-us.apache.org

2018-12-10 Thread Nick Allen
+1 Thanks for the heads up. We will do whatever we need to to help with the transition. On Sun, Dec 9, 2018 at 11:03 AM Otto Fowler wrote: > +1 > > We will need jiras and PR’s for updating our scripts post move however. > > > On December 9, 2018 at 06:32:44, Roy Lenferink

Re: [DISCUSS] Remove `/api/v1/update/replace` endpoint

2018-12-07 Thread Nick Allen
Heads up - I have not received a response from anyone for about 3 days. I am going to take that as no one has a problem with this change. I will likely merge the PR today. On Tue, Dec 4, 2018 at 4:05 PM Nick Allen wrote: > PR #1284 completely removes the `/api/v1/update/replace` endpo

Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?

2018-12-05 Thread Nick Allen
lose to going in? > > >> > > >> If the plugin gets fixed before we're set to move forward with a core > > >> release (or we choose not to fix it, given the current version is > > >> affected), I'm happy to put out a new RC. > > >> > > >&g

[DISCUSS] Remove `/api/v1/update/replace` endpoint

2018-12-04 Thread Nick Allen
PR #1284 completely removes the `/api/v1/update/replace` endpoint. This endpoint is not being used by any other services in Metron currently. As part of the work I am doing to allow Elasticsearch to auto-generate the document ID, I would have had to make code changes to this endpoint to continue

Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?

2018-12-03 Thread Nick Allen
OK, well either way, I see no need to hold up Metron 0.6.1. On Mon, Dec 3, 2018 at 3:51 PM zeo...@gmail.com wrote: > I believe that 0.2.0 is impacted by the bug. > > Jon > > On Mon, Dec 3, 2018 at 3:50 PM Nick Allen wrote: > > > In light of this comment [1], I prop

Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?

2018-12-03 Thread Nick Allen
lugin-kafka 0.3 release is good to go from my side. Thanks > > for all of the reviews Nick > > > > On Wed, Nov 21, 2018 at 11:16 AM Nick Allen wrote: > > > > > Ha. Yes, that definitely counts and makes a ton of sense. Thanks! > > > > > >

Re: Unzipping Cypress

2018-11-26 Thread Nick Allen
Yes, I have noticed that too. If not a way to reduce the time, we should not be logging the unzipping process percentile-by-percentile in the Travis CI builds. On Sat, Nov 24, 2018 at 9:49 AM Otto Fowler wrote: > Anyone else seeing a lot of time taken downloading and unzipping Cypress on >

Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?

2018-11-21 Thread Nick Allen
E statements can cause parse errors in Stellar > (justinleet) closes apache/metron#1268 > METRON-1872 Move rat plugin away from snapshot version (justinleet) > closes apache/metron#1264 > > On Wed, Nov 21, 2018 at 10:59 AM Nick Allen wrote: > > > Also, I'd like to get this one

Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?

2018-11-21 Thread Nick Allen
ichael Miklavcic < > >> > > michael.miklav...@gmail.com> wrote: > >> > > > >> > >> And I do think we will be ready to roll another Metron release in > the > >> > near > >> > >> future as well. > >> > >> > &g

Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?

2018-11-21 Thread Nick Allen
; > > Jon > >> > > > >> > > On Wed, Oct 17, 2018 at 10:37 AM Michael Miklavcic < > >> > > michael.miklav...@gmail.com> wrote: > >> > > > >> > >> And I do think we wil

Re: [DISCUSS] Slack Channel Use

2018-11-12 Thread Nick Allen
hat's not to say that we couldn't stand > more > > > discussion on the lists, but a lot of the dev discussion happens on > > github > > > and JIRA and I'm happy to see an uptick in user traffic. > > > > > > On Wed, Oct 24, 2018 at 10:05 AM Otto Fowler > >

Re: [DISCUSS] Deprecate split-join enrichment topology in favor of unified enrichment topology

2018-11-01 Thread Nick Allen
+1 On Thu, Nov 1, 2018, 6:27 PM Justin Leet wrote: > +1, I haven't seen any case where the split-join topology isn't made > obsolete by the unified topology. > > On Thu, Nov 1, 2018 at 6:17 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > Fellow Metronians, > > > > We've had

Fwd: [DISCUSS] Day 1 User Experience - Getting Metron Running

2018-10-26 Thread Nick Allen
all of metron for this then. > > > On October 26, 2018 at 14:52:27, Nick Allen (n...@nickallen.org) wrote: > > From what I can tell, Katakoda functions mainly through hosting Docker > containers. So if I were to create Katakoda demo like "Introduction to > Stellar REPL", I would need to create

Re: [DISCUSS] Day 1 User Experience - Getting Metron Running

2018-10-26 Thread Nick Allen
nd hosts your container for each user session. That is my assumption from looking through some of the demos that currently exist. On Fri, Oct 26, 2018 at 2:42 PM Otto Fowler wrote: > What is the metron on docker part? > > > On October 26, 2018 at 14:37:48, Nick Allen (n...@nickallen

Re: [DISCUSS] Day 1 User Experience - Getting Metron Running

2018-10-26 Thread Nick Allen
> first timers to Metron. Katakoda seems very interesting - simple and > > straight-forward. I loved the way you can provide instructions, commands > > (that can be directly clicked!), links, explanation and so on. > > > > Regards, > > Anand > > > > On

[DISCUSS] Day 1 User Experience - Getting Metron Running

2018-10-25 Thread Nick Allen
We all know spinning up the development environment is a pain. Unfortunately, it is the only way for a new user to get a feel for Metron. We need a better way to introduce new users to Metron. I am hoping we can brainstorm ways to improve that experience. Here are a few thoughts that might help

Re: metron-elasticsearch integration tests failing after merging in master

2018-10-24 Thread Nick Allen
I usually create a JIRA when I see an intermittent failure like this. But I do not see a record of this particular issue before. https://jira.apache.org/jira/issues/?jql=text%20~%20%22Intermittent%22%20AND%20project%20%3D%20Metron%20AND%20resolution%20%3D%20Unresolved On Wed, Oct 24, 2018 at

Re: [DISCUSS] Slack Channel Use

2018-10-24 Thread Nick Allen
at 8:23 AM Casey Stella wrote: > > > Agreed, the benefit of the mailing list is that it’s searchable by > ponymail > > and the major search engines. > > On Mon, Oct 22, 2018 at 17:18 Nick Allen wrote: > > > > > I don't know that it is the same kind of sear

Re: Revert PR #1218

2018-10-23 Thread Nick Allen
o? > > Simon > > On Tue, 23 Oct 2018 at 19:58, Nick Allen wrote: > > > Hi Guys - > > > > @rmerriman tracked down some problems that were introduced with my PR > > #1218. Thanks to him for finding this. The change was intended to > improve > &g

Revert PR #1218

2018-10-23 Thread Nick Allen
Hi Guys - @rmerriman tracked down some problems that were introduced with my PR #1218. Thanks to him for finding this. The change was intended to improve Elasticsearch write performance by allowing Elasticsearch to set its own document ID. The problem is that if you then go to the Alerts UI

Re: [DISCUSS] Slack Channel Use

2018-10-22 Thread Nick Allen
sts for the reasons you stated > (and > >> > also because not everyone has access to slack generally). On the other > >> > hand, I think an interactive medium like Slack has a lot of advantages > >> in > >> > terms of user satisfaction. Ultimately, tho

Re: [DISCUSS] Slack Channel Use

2018-10-22 Thread Nick Allen
nce is that > there are more than Jon and I answering now. > > > On October 22, 2018 at 12:18:08, Nick Allen (n...@nickallen.org) wrote: > > It seems that we are seeing a lot of Metron usage and support questions on > the Slack Channel. > These are questions that previously would hav

[DISCUSS] Slack Channel Use

2018-10-22 Thread Nick Allen
It seems that we are seeing a lot of Metron usage and support questions on the Slack Channel. These are questions that previously would have been directed to the User or Dev mailing lists. Since this is occurring in the Slack Channel, the conversations are not archived. In my opinion, this is

[DISCUSS] Recurrent Large Indexing Error Messages

2018-10-19 Thread Nick Allen
I want to discuss solutions for the problem that I have described in METRON-1832; Recurrent Large Indexing Error Messages. I feel this is a very easy trap to fall into when using the default settings that currently come with Metron. ## Problem https://issues.apache.org/jira/browse/METRON-1832

Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?

2018-10-16 Thread Nick Allen
I am in favor of a release for both. There are a lot of really useful bug fixes, management of pcap through Ambari, more flexibility for configuring JAAS in Ambari, increased Elasticsearch performance, the Syslog parser, and the Batch Profiler, among others. I would be happy with calling it a

Re: [DISCUSS] Switching to a better alternative of Pikaday.js

2018-10-10 Thread Nick Allen
> Before making a decision on what's next, I'd to ask you a question. Is it > really a priority and is it really worth the effort to touch our currently used date picker component to get ~15% reduction in the bundle size by removing moment? As an aside, I think there is a greater benefit here

Re: [DISCUSS] Add e2e step to PR checklist

2018-10-10 Thread Nick Allen
feedback, Nick. I agree, we need them to be reliable > > enough to not be a source of constant false positives prior to putting > them > > into the checklist. > > On Thu, Oct 4, 2018 at 15:34 Nick Allen wrote: > > > > > I think we still have an issue of reliability

Re: [DISCUSS] Add e2e step to PR checklist

2018-10-04 Thread Nick Allen
I think we still have an issue of reliability. I can never reliably get them all to pass. I have no idea which failures are real. Am I the only one that experiences this? We need a reliable pass/fail on these before we talk about adding them to the checklist. For example, I just tried to run

Re: Metron dev environments moving to require Ansible 2.4+

2018-10-01 Thread Nick Allen
I have been able to spin-up a development environment today without a problem. On Mon, Oct 1, 2018 at 3:58 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > Anyone run latest master today with full dev? I'm seeing > NoClassDefFoundError exceptions on starting enrichments. I upgraded

Re: [DISCUSS] Batch Profiler Feature Branch

2018-09-28 Thread Nick Allen
hanks a lot for the contribution, this is great stuff! > >> > >> On Wed, Sep 26, 2018 at 6:26 PM Nick Allen wrote: > >> > >> > Or support to be offered for merging this feature branch into master? > >> > > >> > On Wed, Se

Re: [DISCUSS] Batch Profiler Feature Branch

2018-09-26 Thread Nick Allen
Or support to be offered for merging this feature branch into master? On Wed, Sep 26, 2018 at 6:20 PM Nick Allen wrote: > Thanks for the review. With https://github.com/apache/metron/pull/1209 > complete, > I think the feature branch is ready to be merged. Sounds like I have

Re: [DISCUSS] Batch Profiler Feature Branch

2018-09-26 Thread Nick Allen
com> wrote: > I just made a couple minor comments on that PR, and I am in agreement about > the readiness for merging with master. Good stuff Nick. > > On Fri, Sep 21, 2018 at 12:37 PM Nick Allen wrote: > > > Here is a PR that adds the input time constraints to the Batc

Re: [DISCUSS] Batch Profiler Feature Branch

2018-09-21 Thread Nick Allen
branch has been merged. Let me know if you disagree. Thanks On Thu, Sep 20, 2018 at 1:55 PM Nick Allen wrote: > Yeah, agreed. Per use case 3, when deploying to production there really > wouldn't be a huge overlap like 3 months of already profiled data. Its day > 1, the profile was just

Re: [DISCUSS] Batch Profiler Feature Branch

2018-09-20 Thread Nick Allen
job > over all the data. It might also make it easier to deal with remediation, > ie an error doesn't force you to re-run over the entire history. Same goes > for testing out the profile seeing batch job in the first place. > > On Thu, Sep 20, 2018 at 11:23 AM Nick Allen wrote: > >

Re: [DISCUSS] Batch Profiler Feature Branch

2018-09-20 Thread Nick Allen
its current > form the batch job runs over all 9 months? > > On Thu, Sep 20, 2018 at 11:13 AM Nick Allen wrote: > > > > How do we establish "tm" from 1.1 above? Any concerns about overlap or > > gaps after the seeding is performed? > > > > Good p

Re: [DISCUSS] Batch Profiler Feature Branch

2018-09-20 Thread Nick Allen
goes by and now I want to add a > new > >profile. > > 1. Same first step above > > 2. I would run the batch loader with *only* that new profile > > definition to seed? > > > > Forgive me if I missed this in PR's and discussion in the FB, but how

Re: [DISCUSS] Batch Profiler Feature Branch

2018-09-20 Thread Nick Allen
* that new profile > definition to seed? > > Forgive me if I missed this in PR's and discussion in the FB, but how do we > establish "tm" from 1.1 above? Any concerns about overlap or gaps after the > seeding is performed? > > On Thu, Sep 20, 2018 at 10:26

Re: [DISCUSS] Batch Profiler Feature Branch

2018-09-20 Thread Nick Allen
being able to read from ZK feels like a fairly > substantial, > > if subtle, set of potential problems. I'd like to see that in either > > before merging or at least pretty soon after merging. Is it a lot of > work > > to add that functionality based on where things are

Re: [DISCUSS] Batch Profiler Feature Branch

2018-09-20 Thread Nick Allen
gt; I think what you have outlined above is a good initial stab at the > > feature. Manual install of spark is not a big deal. Configuring via > > command line while we mature this feature is ok as well. Doesn't look > like > > configuration steps are too hard. I think you shou

[DISCUSS] Batch Profiler Feature Branch

2018-09-19 Thread Nick Allen
I would like to open a discussion to get the Batch Profiler feature branch merged into master as part of METRON-1699 [1] Create Batch Profiler. All of the work that I had in mind for our first draft of the Batch Profiler has been completed. Please take a look through what I have and let me know

Re: [DISCUSS] PCAP data for testing and development

2018-09-19 Thread Nick Allen
I would just be worried about resource constraints on the VM. But Simon's idea of 'do a loop and stop' might be a good solution. We could probably orchestrate that with Ansible tags actually. If you pass the tag, it will 'do a loop and stop', but by default it keeps the current behavior.

Re: [RESULT][VOTE] Metron Release Candidate 0.6.0-RC1

2018-09-12 Thread Nick Allen
Woop woop! On Wed, Sep 12, 2018 at 12:15 PM Justin Leet wrote: > The vote has passed. Including my +1, the voting was: > 3 binding +1’s > 1 non-binding +1’s > no 0’s > no -1’s. >

Re: [VOTE] Metron Release Candidate 0.6.0-RC1

2018-09-09 Thread Nick Allen
+1 Release this package as Apache Metron 0.6.0 Ran through all of the validation steps. No problems, except one transient integration test failure. Not a blocker for the release. On Fri, Sep 7, 2018 at 9:45 AM Nick Allen wrote: > I think the change you made was valuable clean-up. I do

Re: [DISCUSS] Split apart releases for core Metron and the Bro plugin

2018-09-07 Thread Nick Allen
e release infra gets improved and > scripted out more, I don't think it'll end up being much more than bundling > it together. > > On Fri, Sep 7, 2018 at 11:27 AM Nick Allen wrote: > > > > Other projects, e.g. NiFi <http://apache.org/dist/nifi/>, split apart > >

Re: [DISCUSS] Feature branches post-merge

2018-09-07 Thread Nick Allen
+1 delete old feature branches. BTW, there is a branch out there called METRON-113 that we probably need to clean-up. I'm not sure where that came from or why its still around. Probably a fat-finger from long ago. On Fri, Sep 7, 2018 at 12:00 PM Otto Fowler wrote: > I would drop them. > I’ve

Re: [DISCUSS] Split apart releases for core Metron and the Bro plugin

2018-09-07 Thread Nick Allen
> Other projects, e.g. NiFi , split apart these releases within their dist directories. I prefer the way Nifi organizes it. Definitely seems more logically organized. > If we split them apart, we can make the releases independently. This fixes the problem of

Re: [VOTE] Metron Release Candidate 0.6.0-RC1

2018-09-07 Thread Nick Allen
I think the change you made was valuable clean-up. I don't think we need to cancel the vote. We can either manually validate the RC or just use the script from your branch to do the validation. I will be testing the RC this morning. On Fri, Sep 7, 2018 at 9:34 AM Justin Leet wrote: > As full

Re: [DISCUSS] Getting to a 1.0 release

2018-09-07 Thread Nick Allen
We should also publish binaries for each release. That might include any of the following... - Publish the jars to a Maven repository - Publish the RPMs and DEBs to a repository - Host a pre-built "Full Dev" in Vagrant Cloud On Wed, Aug 15, 2018 at 2:49 PM zeo...@gmail.com wrote: >

Re: [DISCUSS] Contributing a General Purpose Regex Parser

2018-08-28 Thread Nick Allen
I'd love to see a PR for this. I know there are others in the community looking for something similar. On Sun, Aug 26, 2018 at 7:28 PM wrote: > Hello, > > > > We have implemented a general purpose regex parser for Metron that we are > interested in contributing back to the community. > > > >

Re: package.lock changes during build?

2018-08-25 Thread Nick Allen
Yes, I have noticed that also, but have not looked deeper. On Sat, Aug 25, 2018 at 10:32 AM Otto Fowler wrote: > I just did a PR, can saw that the package.lock file for alerts-ui was > changed, with updated versions. > I did *not* change the file, nor anything in metron-interface. That seems >

Re: [DISCUSS] Getting to a 1.0 release

2018-08-18 Thread Nick Allen
, 10:57 AM Kyle Richardson wrote: > +1 to separating developer docs and user docs. How should we approach that. > Have a separate doc book? I haven’t had a ton of time to contribute to code > lately but I’d be happy to help write some of these. > > On Sat, Aug 18, 2018 at 9:48 AM Ni

Re: [DISCUSS] Getting to a 1.0 release

2018-08-18 Thread Nick Allen
the value of Metron. On Sat, Aug 18, 2018 at 9:42 AM, Nick Allen wrote: > I'd like to see us focus on improving our docs before a version 1.0. > Right now we just stitch together a bunch of READMEs, which is a great > stride from where we started, but is not ideal. > > Our docs

Re: [DISCUSS] Getting to a 1.0 release

2018-08-18 Thread Nick Allen
I'd like to see us focus on improving our docs before a version 1.0. Right now we just stitch together a bunch of READMEs, which is a great stride from where we started, but is not ideal. Our docs should focused on the user and use cases; What can I do with Metron? Who does it help? How do I do

Re: [DISCUSS] Batch Profiler

2018-08-16 Thread Nick Allen
FYI - Work is progressing on the Batch Profiler in Spark. For those interested, feel free to take a look at any of the PRs that are open on this feature branch. https://github.com/apache/metron/pulls/nickwallen On Mon, Jul 30, 2018 at 10:50 AM, Nick Allen wrote: > >> 1. We will nee

Re: [ANNOUNCE] - Apache Metron Slack channel

2018-08-15 Thread Nick Allen
Thanks for the instructions! On Wed, Aug 15, 2018 at 10:22 AM, Michael Miklavcic < michael.miklav...@gmail.com> wrote: > The Metron community has a Slack channel available for communication > (similar to the existing IRC channel, only on Slack). > > To join: > >1. Go to slack.com. >2.

Re: [DISCUSS] Metron Release 0.6.0?

2018-08-15 Thread Nick Allen
+1 to a 0.6.0 release that includes the Pcap Panel and Solr work. +1 to doing a 0.2.0 release for metron-bro-plugin-kafka. I *think* we need to do the plugin release first, so that the 0.6.0 Metron release will point to plugin 0.2.0. FWIW, here are the changes since the last release. 6 days

Apache Self Service

2018-08-03 Thread Nick Allen
I was having a problem getting a new feature branch to sync-up with Github. One of the Apache Infra guys pointed me to this self-service tool. Worked like a charm. For future reference... https://selfserve.apache.org

Re: [DISCUSS] Batch Profiler

2018-07-30 Thread Nick Allen
ke the caps >- > >UI / Stellar base support >- > >Build out of Batch Profiler service on top of that >- > >Build out of replay service on top of that ( plus all the replay stuff >that needs to also be done - like are you replacing data or having two >

  1   2   3   4   >