Re: [DISCUSS] Next Release - Life After 0.7.1

[Shane Ardell]

Quick update from my end: I just left a +1 on
https://github.com/apache/metron/pull/1527 and will merge it into master
shortly. We are actively looking into the cause of the bug I encountered in
https://github.com/apache/metron/pull/1533. It would be nice to have this
in the release, but I would not categorize it as critical like #1527. I'm
optimistic we will have this resolved and merged into master by this
weekend, but I'm fine reducing scope to not include it.

On Fri, Dec 13, 2019 at 11:24 AM Nick Allen  wrote:

> Are we just waiting on the following PRs as release blockers?  Any others?
>
>- https://github.com/apache/metron/pull/1533
>- https://github.com/apache/metron/pull/1527
>
> Being towards the end of the year, people are going to be on holiday. It
> would be great if we could focus on reducing scope and getting a release
> cut.
>
>
> On Sat, Dec 7, 2019 at 10:04 AM Justin Leet  wrote:
>
> > https://github.com/apache/metron/pull/1568 and
> > https://github.com/apache/metron/pull/1554 are in master now.
> >
> > On Fri, Dec 6, 2019 at 7:16 PM Justin Leet 
> wrote:
> >
> > > I'd like to throw https://github.com/apache/metron/pull/1552 on the
> > pile.
> > > Per https://issues.apache.org/jira/browse/LEGAL-491, we should just
> note
> > > the contribution comes from dependabot. Would someone more familiar
> with
> > > the implications of upgrading that be able to review it, or give some
> > > advice on what we should be looking for in the review?
> > >
> > > On Thu, Dec 5, 2019 at 12:06 PM Shane Ardell  >
> > > wrote:
> > >
> > >> Speaking on the UI-related PRs that Justin mentioned, I also would
> like
> > to
> > >> see both of them merged before a release. At the moment, #1527 does
> not
> > >> address a few "stale data state" message inconsistencies that become
> > >> apparent as a result of that PR's work (you can read more about it in
> > the
> > >> PR comments). That said, I think those inconsistencies can be tracked
> > and
> > >> addressed separately from the current PR.
> > >>
> > >> On Thu, Dec 5, 2019 at 11:51 AM Michael Miklavcic <
> > >> michael.miklav...@gmail.com> wrote:
> > >>
> > >> > I think the junit upgrade should go in also. I'm almost finished
> > >> reviewing
> > >> > that.
> > >> >
> > >> > On Thu, Dec 5, 2019, 8:50 AM Justin Leet 
> > wrote:
> > >> >
> > >> > >  If we're going to do a bug fix release, I'd like to see some of
> the
> > >> low
> > >> > > hanging fix PRs get finished and merged prior to the release.
> We've
> > >> been
> > >> > > lax about getting them cleaned up, so I'd like to use a release as
> > an
> > >> > > opportunity to whittle the PRs down and put out a really solid
> > >> release.
> > >> > >
> > >> > > https://github.com/apache/metron/pull/1568 should be in before
> > >> release.
> > >> > It
> > >> > > addresses an issue with our validation of
> dependencies_with_url.csv
> > >> and
> > >> > > it's validation.
> > >> > > Should https://github.com/apache/metron/pull/1282 be in? Seems
> like
> > >> that
> > >> > > should have been merged awhile ago.
> > >> > >
> > >> > > There's also a couple UI performance / bug fixes PRs (e.g.
> > >> > > https://github.com/apache/metron/pull/1533 and
> > >> > > https://github.com/apache/metron/pull/1527) that have been
> sitting
> > >> > awhile.
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Thu, Dec 5, 2019 at 10:32 AM Nick Allen 
> > >> wrote:
> > >> > >
> > >> > > > Hello Metron'ers -
> > >> > > >
> > >> > > > I would like to make the case that it is time for us to cut the
> > next
> > >> > > Apache
> > >> > > > Metron release.
> > >> > > >
> > >> > > >- Our last release was 0.7.1 on May 15th
> > >> > > ><
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://lists.apache.org/thread.html/e2e532cbb63be757d0875718b082c069a268f57a9087510f196be09b%40%3Cdev.metron.apache.org%3E

Re: [DISCUSS] Feature Branch wiki page

[Shane Ardell]

I would gladly help with this in whatever way I can. As someone who started
contributing not that long ago, this proposed documentation would've saved
me a lot of headaches and pain. We should have clear guidelines and
expectations for how this should work, just as we do with smaller
contributions.

On Tue, Nov 19, 2019 at 9:35 AM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> We've been working with feature branches for nearly 2 years now on this
> project. We started fairly ad-hoc, and as it currently stands, we still
> appear to be fairly ad-hoc after looking for documentation on feature
> branch processes :). I propose we create a wiki page that includes some
> basic instructions to help anyone in the community that wants to contribute
> a larger feature to the project. This would included some general
> guidelines/expectations comparing the requirements for PRs going into a
> feature branch versus those going against master. It would also include
> some general instructions around handling merges and what the process for
> opening and finalizing a feature branch looks like. I'd be interested in
> some help and/or a volunteer to contribute to this work. This may require a
> vote since we're talking about dev guidelines, and I expect that to come up
> in this discussion as well.
>
> Best,
> Mike
>

[DISCUSS] Metron Alerts UI - possibly deprecate query builder and meta alerts

[Shane Ardell]

Hello devs and users,

A few of us working on Metron Alerts UI are seeking some user feedback on
two existing features. Specifically, we are wondering:

* How often do you use the query builder feature of the alerts search bar,
especially since the implementation of the manual search query option?
* How important is the meta alert feature to you? How often do you use it?

Both of these features are pretty intertwined with the search bar's
functionality. The query builder is lacking encapsulation from the search
bar, which leads to a lot of developer confusion when managing state for
the search bar. Meanwhile, meta alerts requires us to manipulate a user's
query string on the front-end before the request is sent to the REST layer,
which, among other things, makes it difficult for us to "seamlessly" switch
back and forth between manual and query builder mode. In addition, it's
another layer of complexity for a user entering their own query string in
manual search mode.

The implementation of these features is making it extremely hard to
maintain and implement new features if it involves the search bar. We are
talking about putting some time into cleaning up the implementation of
these features and improving their functionality, but if they don't have
much user value than perhaps it's better we put this to a vote to see if
these features should actually be deprecated.

Thanks in advance,
Shane

Re: PR Review Request

[Shane Ardell]

I reviewed this PR and left a +1. Great work, Tibor!

Because of the size of the PR (ie. near the threshold of being big enough
to require a feature branch), I'm going to leave it open until tomorrow for
others to chime in if they want. Given your detailed description and great
work, I'm sure others will agree with me that this PR is a welcome
improvement to the UI.

On Mon, Sep 23, 2019 at 7:01 AM Tibor Meller  wrote:

> Hi all,
>
> Could some of our committers take a look at this? Tamas (ruffle1996)
> already left a non-binding +1.
> https://github.com/apache/metron/pull/1514
>
> I found a few issue in the master what I would like to fix in the top of
> this. (Tickets created and mentioned in the PR desc).
>
> Thanks in advance!
>
> Tibor
>

Re: [DISCUSS] Parser Aggregation in Management UI

[Shane Ardell]

>
> > > I am sure you guys tackled the work in chunks when developing it, so
> > > consider just replaying those chunks onto the feature branch as
> separate
> > > PRs.
> > >
> > >
> > >
> > > On Mon, May 6, 2019 at 5:24 AM Tibor Meller 
> > > wrote:
> > >
> > > > I wondered on the weekend how we could split that PR to smaller
> chunks.
> > > > That PR is a result of almost 2 months of development and I don't see
> > how
> > > > to split that to multiple WORKING parts. It is as it is a whole
> working
> > > > feature. If we split it by packages or files we could provide smaller
> > > > non-functional PR's, but can end up having a broken Management UI
> after
> > > > having the 1st PR part merged into master. I don't think that would
> be
> > > > acceptable by the community (or even by me) so I would like to
> suggest
> > > two
> > > > other option to help review PR#1360.
> > > >
> > > > #1 We could extend that PR with our own author comments in Github.
> That
> > > > would help following which code part belongs to where and why it was
> > > > necessary.
> > > > #2 We can schedule an interactive code walkthrough call with the ones
> > who
> > > > interested in reviewing or the particular changeset.
> > > >
> > > > Please share your thoughts on this! Which version would support you
> the
> > > > best? Or if you have any other idea let us know.
> > > >
> > > > PS: I think the size of our PR's depends on how small independently
> > > > deliverable changesets we can identify before we starting to
> implement
> > a
> > > > relatively big new feature. Unfortunately, we missed to do that with
> > this
> > > > feature.
> > > >
> > > > On Fri, May 3, 2019 at 1:49 PM Shane Ardell <
> shane.m.ard...@gmail.com>
> > > > wrote:
> > > >
> > > > > NgRx was only used for the aggregation feature and doesn't go
> beyond
> > > > that.
> > > > > I think the way I worded that sentence may have caused confusion. I
> > > just
> > > > > meant we use it to manage more pieces of state within the
> aggregation
> > > > > feature than just previous and current state of grouped parsers.
> > > > >
> > > > > On Fri, May 3, 2019 at 1:32 AM Michael Miklavcic <
> > > > > michael.miklav...@gmail.com> wrote:
> > > > >
> > > > > > Shane, thanks for putting this together. The updates on the Jira
> > are
> > > > > useful
> > > > > > as well.
> > > > > >
> > > > > > > (we used it for more than just that in this feature, but that
> was
> > > the
> > > > > > initial reasoning)
> > > > > > What are you using NgRx for in the submitted work that goes
> beyond
> > > the
> > > > > > aggregation feature?
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, May 2, 2019 at 12:22 PM Shane Ardell <
> > > shane.m.ard...@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hello everyone,
> > > > > > >
> > > > > > > In response to discussions in the 0.7.1 release thread, I
> wanted
> > to
> > > > > > start a
> > > > > > > thread regarding the parser aggregation work for the Management
> > UI.
> > > > For
> > > > > > > anyone who has not already read and tested the PR locally, I've
> > > > added a
> > > > > > > detailed description of what we did and why to the JIRA ticket
> > > here:
> > > > > > > https://issues.apache.org/jira/browse/METRON-1856
> > > > > > >
> > > > > > > I'm wondering what the community thinks about what we've built
> > thus
> > > > > far.
> > > > > > Do
> > > > > > > you see anything missing that must be part of this new feature
> in
> > > the
> > > > > UI?
> > > > > > > Are there any strong objections to how we implemented it?
> > > > > > >
> > > > > > > I’m also looking to see if anyone has any thoughts on ho

Re: [DISCUSS] Parser Aggregation in Management UI

[Shane Ardell]

NgRx was only used for the aggregation feature and doesn't go beyond that.
I think the way I worded that sentence may have caused confusion. I just
meant we use it to manage more pieces of state within the aggregation
feature than just previous and current state of grouped parsers.

On Fri, May 3, 2019 at 1:32 AM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Shane, thanks for putting this together. The updates on the Jira are useful
> as well.
>
> > (we used it for more than just that in this feature, but that was the
> initial reasoning)
> What are you using NgRx for in the submitted work that goes beyond the
> aggregation feature?
>
>
>
> On Thu, May 2, 2019 at 12:22 PM Shane Ardell 
> wrote:
>
> > Hello everyone,
> >
> > In response to discussions in the 0.7.1 release thread, I wanted to
> start a
> > thread regarding the parser aggregation work for the Management UI. For
> > anyone who has not already read and tested the PR locally, I've added a
> > detailed description of what we did and why to the JIRA ticket here:
> > https://issues.apache.org/jira/browse/METRON-1856
> >
> > I'm wondering what the community thinks about what we've built thus far.
> Do
> > you see anything missing that must be part of this new feature in the UI?
> > Are there any strong objections to how we implemented it?
> >
> > I’m also looking to see if anyone has any thoughts on how we can possibly
> > simplify this PR. Right now it's pretty big, and there are a lot of
> commits
> > to parse through, but I'm not sure how we could break this work out into
> > separate, smaller PRs opened against master. We could try to cherry-pick
> > the commits into smaller PRs and then merge them into a feature branch,
> but
> > I'm not sure if that's worth the effort since that will only reduce the
> > number commits to review, not the lines changed.
> >
> > As an aside, I also want to give a little background into the
> introduction
> > of NgRx in this PR. To give a little background on why we chose to do
> this,
> > you can refer to the discussion thread here:
> >
> >
> https://lists.apache.org/thread.html/06a59ea42e8d9a9dea5f90aab4011e44434555f8b7f3cf21297c7c87@%3Cdev.metron.apache.org%3E
> >
> > We previously discussed introducing a better way to manage application
> > state in both UIs in that thread. It was decided that NgRx was a great
> tool
> > for many reasons, one of them being that we can piecemeal it into the
> > application rather than doing a huge rewrite of all the application state
> > at once. The contributors in this PR (myself included) decided this would
> > be a perfect opportunity to introduce NgRx into the Management UI since
> we
> > need to manage the previous and current state with the grouping feature
> so
> > that users can undo the changes they've made (we used it for more than
> just
> > that in this feature, but that was the initial reasoning). In addition,
> we
> > greatly benefited from this when it came time to debug our work in the UI
> > (the discussion in the above thread link goes a little more into the
> > advantages of debugging with NgRx and DevTools). Removing NgRx from this
> > work would reduce the numbers of lines changed slightly, but it would
> still
> > be a big PR and a lot of that code would just move to the component or
> > service level in the Angular application.
> >
> > Shane
> >
>

Re: [DISCUSS] Metron Release - 0.7.1 next steps

[Shane Ardell]

Here's a link to the ngrx discussion thread from a few months back:
https://lists.apache.org/thread.html/06a59ea42e8d9a9dea5f90aab4011e44434555f8b7f3cf21297c7c87@%3Cdev.metron.apache.org%3E

On Thu, May 2, 2019 at 1:17 PM Otto Fowler  wrote:

> If you can find a link in the archives for that thread, it would really
> help.
>
> I don’t think sending them up as one sensor would work…. as something
> quick.  I think it is an interesting idea from a higher level that would
> need some more thought though ( IE: what if every sensor in the ui was a
> sensor group, and the existing  entries where just groups of 1 ).
>
> As far as I can see, we have brought up the idea of a release ourselves, I
> don’t see why we don’t just swarm this issue and get it right then release.
>
>
>
> On May 2, 2019 at 04:16:31, Tamás Fodor (ftamas.m...@gmail.com) wrote:
>
> In PR#1360 we introduced a new state management strategy involving a new
> module called Ngrx. We had a discussion thread on this a few months ago and
> we successfully convinced you about the benefits. This is one of the
> reasons why this PR is going to be still huge after cleaning up the commit
> history. After you having a look at the changes and the feature itself,
> there's likely have questions about why certain parts work as they do. The
> thing what I'd like to point out is that, yes, it probably takes more time
> to get it in.
>
> In order to being able to release the RC, wouldn't it be an easy and quick
> fix on the backend if it sent the aggregated parsers to the client as they
> were one sensor? It's just an idea, it might be wrong, but at least we
> shouldn't have to wait until the aforementioned PR gets ready to be merged
> to the master.
>
> On Wed, May 1, 2019 at 4:16 PM Justin Leet  wrote:
>
> > Short version: I'm in favor of #2 of 0.7.1 and #1 as a blocker for 0.8.0.
> > #3 seems like a total waste of time and effort.
> >
> > The wall of text version:
> > I agree this isn't "just the wrong thing shown", but for completely
> > different reasons.
> >
> > To be extremely clear about what the problem is: Our "dev" environment
> > (whose very name implies the audience is develops) uses a
> performance-based
> > advanced feature to ensure that all our of sample flows are regularly run
> > and produce data. This feature has a bare minimal implementation to be
> > enabled via Ambari, which it currently is by default. This is because of
> > the limited resources available that previously resulted in us turning
> off
> > Yaf, and therefore testing it during regular full dev runs. Right now
> > however, this feature is not exposed through the management UI, and
> > therefore it isn't obvious what the implications are. Am I missing
> anything
> > here?
> >
> > For users actually choosing to use the parser aggregation feature in a
> > non-full-dev environment, I'd expect substantially more care to be
> involved
> > given the lack of easy configuration for it (after all, why would you
> > bother running the aggregated parser alongside the regular parser? This
> > could be more explicitly stated, but again that feels like a doc problem.
> > Right now I could essentially provide two of the same parser and create
> the
> > same problem, so right now aggregation is only special because it runs on
> > dev by default). This is, in my opinion, primarily a first impression
> > problem and likely one of many areas that could use improved
> documentation.
> >
> > Quite frankly, I think the issue pointed out here could mostly be
> resolved
> > by documenting how the current aggregation is done in dev, and telling
> how
> > to change it. Especially for a 0.x.1 release, which is primarily bug
> > fixes. As can be inferred from my vote, I don't think this problem is a
> > problem that needs solving in a point release. I would support improving
> > the documentation, both full-dev and for aggregation in general for the
> > 0.7.1 point release, while making a 0.8.0 release contingent upon the
> > outstanding PRs to enable it in the management UI.
> >
> > There are a couple deeper issues, imo, that I care substantially more
> about
> > than this in particular
> > * The dev environment is being used as our intro for users, because it's
> > convenient for us to not maintain more environments (which has been a
> major
> > pain point in the past). Worse, the dev environment strongly implies it's
> > for Metron developers, rather than people looking to build on top of
> > Metron. We need an actual strategy for providing end users a clean
> > impression of Metron (this could be clarifying what the expectations of
> > full dev are, renaming it to something like "full-demo", something more
> > involved, etc.). This is something that we've needed for awhile in
> general,
> > and includes larger topics like improving our website, potentially
> > improving the site book, actually publishing our Javadocs somewhere so
> > people can develop things easier, publishing out info about Stellar

[DISCUSS] Reverting METRON-2023 (PR #1364)

[Shane Ardell]

Hello everyone,

I'm sending out this email to notify the community of my intention to
revert the commit for METRON-2023 (PR #1364
). This commit introduced a bug
stemming from yauzl, the tool that Cypress uses to unzip itself. This bug
causes an error in Travis and is intermittent, which is how it passed
multiple build attempts before being merged into master. The issue is
currently being discussed with the Cypress team and can be viewed here
.

If anyone has any strong objection to this, please let me know here.
Otherwise, I plan on reverting this in the next 24 hours.

Shane

Re: [DISCUSS] Central Navigation for Alerts and Management UI

[Shane Ardell]

Good point, Otto. Just posted there now.

On Mon, Mar 11, 2019 at 12:11 PM Otto Fowler 
wrote:

> Maybe you should post to the users@ list
>
>
> On March 11, 2019 at 06:25:48, Shane Ardell (shane.m.ard...@gmail.com)
> wrote:
>
> Thank you both for explaining the original design choice. That does make
> sense.
>
> However, like both of you point out, I am also curious if anyone has direct
> user feedback indicating a need for either persona to switch between
> screens.
>
> Shane
>
> On Tue, Mar 5, 2019 at 7:25 PM Rita McKissick 
> wrote:
>
> > That was my thought, too. The Management UI is meant for the Operations
> > persona.
> > And the Alerts UI is meant for the SOC analyst persona. If we see a need
> > for either of these
> > personas to use both of the UIs, then the ability to switch between the
> > two UIs would
> > be great. Otherwise, I'm not sure that ability is necessary.
> >
> > As an aside, as the tech writer I would love to be able to switch between
> > the two UIs, but I'm not
> > really one of our supported personas __ Darn!
> >
> > Rita
> >
> > Rita McKissick ! Sr. Technical Writer
> > rmckiss...@hortonworks.com
> > (mobile) 831-234-3676
> >
> >
> > On 3/5/19, 9:50 AM, "Michael Miklavcic" 
> > wrote:
> >
> > The original design was done with the intent to keep the user profiles
> > (soc
> > analyst vs ops personnel) separate and enable a microservices-oriented
> > architecture. I don't have a strong opinion one way or the other, but
> > I'd
> > be interested to hear whether others in the community find this wall
> > useful, or if we should come back to a single pain of glass.
> >
> > Mike
> >
> > On Tue, Mar 5, 2019 at 9:12 AM Shane Ardell 
> > wrote:
> >
> > > Hello everyone,
> > >
> > > I recently started experimenting with implementing a navigation bar
> > in both
> > > the Alerts and Management UI. It would allow us to navigate between
> > the two
> > > UIs through links instead of manually entering a url or opening
> > separate
> > > tabs from Ambari.
> > >
> > > I'm just wondering what everyone's thoughts are. Is this something
> > we want
> > > in Metron?
> > >
> >
> >
> >
>

Re: [DISCUSS] Central Navigation for Alerts and Management UI

[Shane Ardell]

Thank you both for explaining the original design choice. That does make
sense.

However, like both of you point out, I am also curious if anyone has direct
user feedback indicating a need for either persona to switch between
screens.

Shane

On Tue, Mar 5, 2019 at 7:25 PM Rita McKissick 
wrote:

> That was my thought, too. The Management UI is meant for the Operations
> persona.
> And the Alerts UI is meant for the SOC analyst persona. If we see a need
> for either of these
> personas to use both of the UIs, then the ability to switch between the
> two UIs would
> be great. Otherwise, I'm not sure that ability is necessary.
>
> As an aside, as the tech writer I would love to be able to switch between
> the two UIs, but I'm not
> really one of our supported personas __ Darn!
>
> Rita
>
> Rita McKissick ! Sr. Technical Writer
> rmckiss...@hortonworks.com
> (mobile) 831-234-3676
>
>
> On 3/5/19, 9:50 AM, "Michael Miklavcic" 
> wrote:
>
> The original design was done with the intent to keep the user profiles
> (soc
> analyst vs ops personnel) separate and enable a microservices-oriented
> architecture. I don't have a strong opinion one way or the other, but
> I'd
> be interested to hear whether others in the community find this wall
> useful, or if we should come back to a single pain of glass.
>
> Mike
>
> On Tue, Mar 5, 2019 at 9:12 AM Shane Ardell 
> wrote:
>
> > Hello everyone,
> >
> > I recently started experimenting with implementing a navigation bar
> in both
> > the Alerts and Management UI. It would allow us to navigate between
> the two
> > UIs through links instead of manually entering a url or opening
> separate
> > tabs from Ambari.
> >
> > I'm just wondering what everyone's thoughts are. Is this something
> we want
> > in Metron?
> >
>
>
>

[DISCUSS] Central Navigation for Alerts and Management UI

[Shane Ardell]

Hello everyone,

I recently started experimenting with implementing a navigation bar in both
the Alerts and Management UI. It would allow us to navigate between the two
UIs through links instead of manually entering a url or opening separate
tabs from Ambari.

I'm just wondering what everyone's thoughts are. Is this something we want
in Metron?

Re: Unzipping Cypress

[Shane Ardell]

No problem! Glad I could help.

On Wed, Nov 28, 2018 at 5:03 PM Otto Fowler  wrote:

> OK,
>
> I think what is happening is that in my PR, I’m building metron in Docker
> and deploying to vagrant.  I have updated my PR to map the cypress cache
> into the Docker container.
> Thanks!
>
>
> On November 28, 2018 at 10:29:25, Shane Ardell (shane.m.ard...@gmail.com)
> wrote:
>
> For me, it's at ~/Library/Caches/Cypress, but the path depends on your OS:
>
> https://docs.cypress.io/guides/getting-started/installing-cypress.html#Binary-cache
>
> On Wed, Nov 28, 2018 at 4:19 PM Otto Fowler 
> wrote:
>
> > Where is the cache path?
> >
> >
> > On November 28, 2018 at 09:34:18, Shane Ardell (shane.m.ard...@gmail.com
> )
> > wrote:
> >
> > https://github.com/cypress-io/cypress/issues/1813
> >
>

Re: Unzipping Cypress

[Shane Ardell]

For me, it's at ~/Library/Caches/Cypress, but the path depends on your OS:
https://docs.cypress.io/guides/getting-started/installing-cypress.html#Binary-cache

On Wed, Nov 28, 2018 at 4:19 PM Otto Fowler  wrote:

> Where is the cache path?
>
>
> On November 28, 2018 at 09:34:18, Shane Ardell (shane.m.ard...@gmail.com)
> wrote:
>
> https://github.com/cypress-io/cypress/issues/1813
>

Re: Unzipping Cypress

[Shane Ardell]

Otto,

I'm not able to reproduce this issue locally. My build uses a binary stored
in a global cache directory. I did a little research and it looks like the
cached global binary not being used was an issue with Cypress v3.0.0
<https://github.com/cypress-io/cypress/issues/1813>, but it was resolved in
v3.0.1. We use v3.1.0.

I noticed that we do download Chromium every build. Just to verify, there
wasn't some confusion between Cypress and Chromium, correct?


[1]
https://docs.cypress.io/guides/getting-started/installing-cypress.html#npm-install

On Tue, Nov 27, 2018 at 6:40 PM Otto Fowler  wrote:

> I’m sorry for the confusion, but I see this locally.  I’m not looking at
> travis.
> I have been doing a lot of full dev deployments.
>
>
>
> On November 27, 2018 at 11:40:42, Shane Ardell (shane.m.ard...@gmail.com)
> wrote:
>
> Otto,
>
> Do you have a Travis log you can share that shows Cypress downloaded vs.
> using a cached version? It looks like my latest merge to master uses a
> cached version: https://travis-ci.org/apache/metron/jobs/458440883#L7292.
>
> Thanks in advance.
>
> On Mon, Nov 26, 2018 at 6:14 PM Shane Ardell 
> wrote:
>
> > It seems we can pretty easily configure the .travis.yml config file to
> > cache our npm modules:
> >
> https://docs.cypress.io/guides/guides/continuous-integration.html#Caching-the-Cypress-binary
> >
> > It also looks like we are already trying to cache our npm modules in the
> > Travis config, but, obviously, it's not working as intended. I can take
> a
> > look into why tomorrow.
> >
> > On Mon, Nov 26, 2018 at 5:33 PM Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> >> Shane, Tibor - Can you guys chime in on this?
> >>
> >> On Mon, Nov 26, 2018 at 9:13 AM Otto Fowler 
> >> wrote:
> >>
> >> > Isn’t there a way we can cache it?
> >> >
> >> >
> >> > On November 26, 2018 at 10:59:20, Nick Allen (n...@nickallen.org)
> >> wrote:
> >> >
> >> > Yes, I have noticed that too. If not a way to reduce the time, we
> should
> >> > not be logging the unzipping process percentile-by-percentile in the
> >> Travis
> >> > CI builds.
> >> >
> >> > On Sat, Nov 24, 2018 at 9:49 AM Otto Fowler 
>
> >> > wrote:
> >> >
> >> > > Anyone else seeing a lot of time taken downloading and unzipping
> >> Cypress
> >> > on
> >> > > builds?
> >> > > What is up with that?™
> >> > >
> >> > > ottO
> >> > >
> >> >
> >>
> >
>
>

Re: Unzipping Cypress

[Shane Ardell]

Otto,

Do you have a Travis log you can share that shows Cypress downloaded vs.
using a cached version? It looks like my latest merge to master uses a
cached version: https://travis-ci.org/apache/metron/jobs/458440883#L7292.

Thanks in advance.

On Mon, Nov 26, 2018 at 6:14 PM Shane Ardell 
wrote:

> It seems we can pretty easily configure the .travis.yml config file to
> cache our npm modules:
> https://docs.cypress.io/guides/guides/continuous-integration.html#Caching-the-Cypress-binary
>
> It also looks like we are already trying to cache our npm modules in the
> Travis config, but, obviously, it's not working as intended. I can take a
> look into why tomorrow.
>
> On Mon, Nov 26, 2018 at 5:33 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
>> Shane, Tibor - Can you guys chime in on this?
>>
>> On Mon, Nov 26, 2018 at 9:13 AM Otto Fowler 
>> wrote:
>>
>> > Isn’t there a way we can cache it?
>> >
>> >
>> > On November 26, 2018 at 10:59:20, Nick Allen (n...@nickallen.org)
>> wrote:
>> >
>> > Yes, I have noticed that too. If not a way to reduce the time, we should
>> > not be logging the unzipping process percentile-by-percentile in the
>> Travis
>> > CI builds.
>> >
>> > On Sat, Nov 24, 2018 at 9:49 AM Otto Fowler 
>> > wrote:
>> >
>> > > Anyone else seeing a lot of time taken downloading and unzipping
>> Cypress
>> > on
>> > > builds?
>> > > What is up with that?™
>> > >
>> > > ottO
>> > >
>> >
>>
>

Re: [DISCUSS] Add ngrx to handle state management in Angular

[Shane Ardell]

There are a couple of good examples in Metron Alerts showing different
advantages we would gain using NgRx. First, you’ll notice that a lot of
state does not persist between view changes. For example, if a user inputs
search criteria into the search bar, switches to the PCAP view, then
switches back, that search input is not persisted. If we used NgRx, the
application store can persist this state between view changes if we wanted
to. The importance of this will continue to become more apparent as the
application scales up and different views are added.

In addition, there is quite a bit of shared application data between
components. For example, you’ll see that both the GroupByComponent and
AlertFiltersComponent display the number of facets available. When you
filter by a facet, these numbers update. You can filter by clicking on the
facets available in the AlertFiltersComponent, or you can search with a
filter keyword and value in the search bar. Multiple areas of the
application are showing the same data, and multiple areas of the
application are manipulating that data. While passing data through @Input
and @Output decorators and services is fine for basic sharing of
application data, you’ll eventually create a complex web of data sharing
and mutation that can be hard to follow and debug. In my opinion, we are
nearing that point in our application. NgRx solves this with its
one-direction data flow and immutable data structures. The store is the
single source of truth which your component derives state from rather than
having components holding their own state and having to manage, communicate
and pass data between them.

There are also many debugging advantages. I don’t have a specific example
at the moment, but you can probably imagine how much easier it is to “time
travel” with NgRx debugging tools. What I mean by “time travel” is the
ability to move back and forth among the previous application states and
view the results in real-time rather than setting a breakpoint, reloading
the application, inspecting, setting another breakpoint to inspect a state
previous to your current breakpoint, reloading the application, etc. This
is a very repetitive and time-consuming thing to do when debugging on the
front-end, and is very common. For a visual example of this, here is a
video containing a decent overview of debugging with NgRx:
https://www.youtube.com/watch?v=70ojPxMA7Ig

On Tue, Nov 27, 2018 at 5:30 AM James Sirota  wrote:

> I found a helpful article here:
> https://brianflove.com/2018/01/08/ngrx-the-basics/
>
> A lot of this goes over my head, but in a nutshell, it's a tree-based
> state management object for JS.  Its main drawback seems to be added
> complexity, but if the guys who are more familiar with UI say we would
> benefit from it I am inclined to take them at their word.
>
> 26.11.2018, 13:09, "Michael Miklavcic" :
> > Shane, thanks for sharing this. Can you perhaps describe a sample use
> case
> > in the UI currently and explain for us how it currently works (or
> doesn't,
> > ha) versus how it would be modified and improved with using NgRx?
> >
> > Thanks,
> > Mike
> >
> > On Fri, Nov 23, 2018 at 7:44 AM Shane Ardell 
> > wrote:
> >
> >>  What I'm referring to is roughly the entire contents of the UI
> >>  application's memory.
> >>
> >>  On Thu, Nov 22, 2018 at 6:29 PM Otto Fowler 
> >>  wrote:
> >>
> >>  > Can you describe what you mean by “state” in a little more detail?
> Not a
> >>  > complete description, maybe just a crib list.
> >>  >
> >>  >
> >>  > On November 22, 2018 at 07:21:43, Shane Ardell (
> shane.m.ard...@gmail.com
> >>  )
> >>  > wrote:
> >>  >
> >>  > As both the Management and Alerts UI grow in size, managing
> application
> >>  > state continues to become more and more complex. To help us deal with
> >>  > managing all of this state and ensuring our application derives state
> >>  from
> >>  > a single source of truth, I suggest we start using NgRx, a state
> >>  > management
> >>  > library based on the Redux pattern but built for Angular. It is by
> far
> >>  the
> >>  > most popular library of this type for Angular. As you can see in the
> >>  > project's GitHub Insights tab <
> https://github.com/ngrx/platform/pulse>,
> >>  > it's quite actively worked on and releases are pretty frequent. The
> >>  > project
> >>  > is licensed under MIT.
> >>  >
> >>  > As far as an approach to integration, I don't think we necessarily
> need a
> >>  > big refactoring right off the bat. I feel something like this can be
> done
> >>  > in a piecemeal approach over time. I think we can start by
> introducing it
> >>  > into the project the next time we have a new application feature.
> >>  >
> >>  > What are everyone's thoughts around this?
> >>  >
> >>  > Cheers,
> >>  > Shane
> >>  >
> >>  >
>
> ---
> Thank you,
>
> James Sirota
> PMC- Apache Metron
> jsirota AT apache DOT org
>
>

Re: Unzipping Cypress

[Shane Ardell]

It seems we can pretty easily configure the .travis.yml config file to
cache our npm modules:
https://docs.cypress.io/guides/guides/continuous-integration.html#Caching-the-Cypress-binary

It also looks like we are already trying to cache our npm modules in the
Travis config, but, obviously, it's not working as intended. I can take a
look into why tomorrow.

On Mon, Nov 26, 2018 at 5:33 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Shane, Tibor - Can you guys chime in on this?
>
> On Mon, Nov 26, 2018 at 9:13 AM Otto Fowler 
> wrote:
>
> > Isn’t there a way we can cache it?
> >
> >
> > On November 26, 2018 at 10:59:20, Nick Allen (n...@nickallen.org) wrote:
> >
> > Yes, I have noticed that too. If not a way to reduce the time, we should
> > not be logging the unzipping process percentile-by-percentile in the
> Travis
> > CI builds.
> >
> > On Sat, Nov 24, 2018 at 9:49 AM Otto Fowler 
> > wrote:
> >
> > > Anyone else seeing a lot of time taken downloading and unzipping
> Cypress
> > on
> > > builds?
> > > What is up with that?™
> > >
> > > ottO
> > >
> >
>

Re: [DISCUSS] Add ngrx to handle state management in Angular

[Shane Ardell]

What I'm referring to is roughly the entire contents of the UI
application's memory.

On Thu, Nov 22, 2018 at 6:29 PM Otto Fowler  wrote:

> Can you describe what you mean by “state” in a little more detail?  Not a
> complete description, maybe just a crib list.
>
>
> On November 22, 2018 at 07:21:43, Shane Ardell (shane.m.ard...@gmail.com)
> wrote:
>
> As both the Management and Alerts UI grow in size, managing application
> state continues to become more and more complex. To help us deal with
> managing all of this state and ensuring our application derives state from
> a single source of truth, I suggest we start using NgRx, a state
> management
> library based on the Redux pattern but built for Angular. It is by far the
> most popular library of this type for Angular. As you can see in the
> project's GitHub Insights tab <https://github.com/ngrx/platform/pulse>,
> it's quite actively worked on and releases are pretty frequent. The
> project
> is licensed under MIT.
>
> As far as an approach to integration, I don't think we necessarily need a
> big refactoring right off the bat. I feel something like this can be done
> in a piecemeal approach over time. I think we can start by introducing it
> into the project the next time we have a new application feature.
>
> What are everyone's thoughts around this?
>
> Cheers,
> Shane
>
>

[DISCUSS] Add ngrx to handle state management in Angular

[Shane Ardell]

As both the Management and Alerts UI grow in size, managing application
state continues to become more and more complex. To help us deal with
managing all of this state and ensuring our application derives state from
a single source of truth, I suggest we start using NgRx, a state management
library based on the Redux pattern but built for Angular. It is by far the
most popular library of this type for Angular. As you can see in the
project's GitHub Insights tab ,
it's quite actively worked on and releases are pretty frequent. The project
is licensed under MIT.

As far as an approach to integration, I don't think we necessarily need a
big refactoring right off the bat. I feel something like this can be done
in a piecemeal approach over time. I think we can start by introducing it
into the project the next time we have a new application feature.

What are everyone's thoughts around this?

Cheers,
Shane

Re: [ANNOUNCE] Shane Ardell is a committer

[Shane Ardell]

I want to extend a huge thank you to everyone part of Apache Metron's PMC
for offering me this opportunity!

Cheers,
Shane

On Mon, Nov 19, 2018 at 4:53 PM zeo...@gmail.com  wrote:

> Congrats Shane!
>
> Jon
>
> On Mon, Nov 19, 2018 at 10:43 AM Anand Subramanian <
> asubraman...@hortonworks.com> wrote:
>
> > Many congratulations, Shane!
> >
> > Cheers,
> > Anand
> >
> > On 11/19/18, 8:36 PM, "James Sirota"  wrote:
> >
> >
> > The Project Management Committee (PMC) for Apache Metron has invited
> > Shane Ardell to become a committer and we are pleased to announce that he
> > has accepted.  I wanted to congratulate Shane on this achievement.
> >
> >
> > Being a committer enables easier contribution to the project since
> > there is no need to go via the patch submission process. This should
> enable
> > better productivity. Being a PMC member enables assistance with the
> > management and to guide the direction of the project.
> > ---
> > Thank you,
> >
> > James Sirota
> > PMC- Apache Metron
> > jsirota AT apache DOT org
> >
> >
> >
> > --
>
> Jon Zeolla
>

Re: [DISCUSS] Switching to a better alternative of Pikaday.js

[Shane Ardell]

We are definitely on the same page.

On Mon, Oct 15, 2018 at 2:06 PM Tibor Meller  wrote:

> @Shane It seems like we're agreed on this. :D
>
> On Mon, Oct 15, 2018 at 1:18 PM Tibor Meller 
> wrote:
>
> > Hi Guys,
> >
> > I think we could consider moving to ng bootstrap. It solves most of our
> > problems and makes our code base cleaner easier to maintain.
> >
> > Here are some benefits I see:
> > 1. we could eliminate jQuery from the code base
> > Currently, we're using bootstrap but an implementation based on jQuery.
> > Angular and jQuery doesn't build to live together.
> > 2. NgBootstrap made to being used with Angular
> > It uses Angular instead of hacking the rendering/dom manipulation with
> > jQuery.
> > 3. It contains a date and a time picker.
> > It's easy to combine to a datetime picker.
> > 4. No dependencies.
> > By changing currently used bootstrap to NgBootstrap we could clear
> jquery,
> > moment, pickaday, pickaday-time libraries from our dependencies.
> >
> > Another huge advantage of NgBootstrap is that we don't have to rewrite
> > anything we don't want to. Our UI already uses Bootstrap.
> >
> > What do you guys think?
> >
> > On Wed, Oct 10, 2018 at 4:19 PM Nick Allen  wrote:
> >
> >> > Before making a decision on what's next, I'd to ask you a question. Is
> >> it really
> >> a priority and is it really worth the effort to touch our currently used
> >> date picker component to get ~15% reduction in the bundle size by
> removing
> >> moment?
> >>
> >> As an aside, I think there is a greater benefit here too.  We need to
> make
> >> a conscious effort to identify libraries that we are using that are
> >> deprecated, lack community support, and are unlikely to be maintained
> and
> >> updated for security vulnerabilities.  We need to actively identify and
> >> replace those.
> >>
> >>
> >>
> >>
> >>
> >> On Wed, Oct 10, 2018 at 9:33 AM Tamás Fodor 
> >> wrote:
> >>
> >> > I'd like to open a discussion about switching to a new date picker
> >> library
> >> > in the Metron Alerts UI regarding to the following:
> >> >
> >> >
> >> >
> >>
> https://lists.apache.org/thread.html/2e4fafa4256ce14ebcd4433420974e24962884204418ade51f0e3bfb@%3Cdev.metron.apache.org%3E
> >> >
> >> > https://github.com/apache/metron/pull/1219#discussion_r223733562
> >> >
> >> > A week ago, I opened a PR about removing moment.js from the code base
> to
> >> > decrease the size of the production javascript bundle. I could achieve
> >> 15%
> >> > loss in the final bundle size which is admittedly not a game changer
> but
> >> > still. Not to mention if we want to heavily rely on date manipulator
> >> > functions in the future it's better to get rid of it at this early
> >> stage.
> >> > Go here  to read more
> about
> >> > the
> >> > background and the results. I tried to provide as many details as I
> >> could.
> >> >
> >> > So far, so good. But then I stumbled upon an obstacle, Pikaday
> >> > .
> >> >
> >> > Before going further, let me thank Tibor Meller, Michael Miklavcic and
> >> > Nicholas Allen for taking their time to go through my proposal to deal
> >> with
> >> > the aforementioned issue. At the end, we agreed on basically not going
> >> with
> >> > my temporary solution that intended to solve the related problems of
> >> > Pikaday and we'd rather like to find and change for a better
> >> alternative.
> >> >
> >> > To be fair, Pikaday is a pretty good date picker module, its only
> >> problem
> >> > is the moment dependency if it's installed via npm. But other than
> >> that, it
> >> > functions perfectly. Zero dependencies, small, etc. Long story short,
> >> it's
> >> > good for us unless we want to get rid of moment.js.
> >> >
> >> > Before making a decision on what's next, I'd to ask you a question. Is
> >> it
> >> > really a priority and is it really worth the effort to touch our
> >> currently
> >> > used date picker component to get ~15% reduction in the bundle size by
> >> > removing moment?
> >> >
> >> > I'm asking it because if we want to do so, considering that it's a
> huge
> >> > topic, the following questions might come up:
> >> >
> >> > *A: What component do we want to use instead of Pikaday?*
> >> >
> >> > I'm not satisfied with the alternative individual solutions out there
> on
> >> > npm. I'd rather pick a component library like the angular port of
> >> Bootstrap
> >> >  or the angular material
> library
> >> > . Both of them have a date picker
> >> component
> >> > and many other components to rely on and reuse throughout the Metron
> >> app.
> >> >
> >> > *B: What component library do we want to use?*
> >> >
> >> > Introducing a new component library requires a lot of research and
> there
> >> > are many things we have to agreed on. Since it's a long term plan
> >> because
> >> > it would be great to use it consistently instead of picking a new 

Re: [DISCUSS] Switching to a better alternative of Pikaday.js

[Shane Ardell]

>
> I'm not satisfied with the alternative individual solutions out there on
> npm. I'd rather pick a component library like the angular port of Bootstrap
>  or the angular material library
> . Both of them have a date picker component
> and many other components to rely on and reuse throughout the Metron app.


I’m really glad you brought this up. We currently include the entire
jQuery-based version of Bootstrap in our project, yet we are only really
taking advantage of component styling. The few places where we are using
Bootstrap component's for their javascript behavior, we aren't using them
in the "Angular way". For example, we currently do direct DOM manipulations
with jQuery inside the dialog box class instead of using Angular’s built-in
tooling
.
In my opinion, we are unnecessarily increasing our bundle size with a
dependency we don't really need, which is also susceptible to security
vulnerabilities in the future.

If we switch to using Ng Bootstrap, there are a few advantages:
1. As mentioned, we could remove moment.js as a dependency by using Ng
Bootstrap's datepicker instead of Pickaday.
2. We could remove jQuery as a dependency.
3. We import only the components we need vs importing the entire compiled
Bootstrap library.
4. We get to take advantage of Angular's rendering engine when using
components from said library.

I don't mean to hijack this discussion thread, but I felt it was worth
pointing out the other advantages and reasons for switching to a library
like Ng Bootstrap.


On Wed, Oct 10, 2018 at 4:19 PM Nick Allen  wrote:
>
> > Before making a decision on what's next, I'd to ask you a question. Is
it really
> a priority and is it really worth the effort to touch our currently used
> date picker component to get ~15% reduction in the bundle size by removing
> moment?
>
> As an aside, I think there is a greater benefit here too.  We need to make
> a conscious effort to identify libraries that we are using that are
> deprecated, lack community support, and are unlikely to be maintained and
> updated for security vulnerabilities.  We need to actively identify and
> replace those.
>
>
>
>
>
> On Wed, Oct 10, 2018 at 9:33 AM Tamás Fodor  wrote:
>
> > I'd like to open a discussion about switching to a new date picker
library
> > in the Metron Alerts UI regarding to the following:
> >
> >
> >
https://lists.apache.org/thread.html/2e4fafa4256ce14ebcd4433420974e24962884204418ade51f0e3bfb@%3Cdev.metron.apache.org%3E
> >
> > https://github.com/apache/metron/pull/1219#discussion_r223733562
> >
> > A week ago, I opened a PR about removing moment.js from the code base to
> > decrease the size of the production javascript bundle. I could achieve
15%
> > loss in the final bundle size which is admittedly not a game changer but
> > still. Not to mention if we want to heavily rely on date manipulator
> > functions in the future it's better to get rid of it at this early
stage.
> > Go here  to read more about
> > the
> > background and the results. I tried to provide as many details as I
could.
> >
> > So far, so good. But then I stumbled upon an obstacle, Pikaday
> > .
> >
> > Before going further, let me thank Tibor Meller, Michael Miklavcic and
> > Nicholas Allen for taking their time to go through my proposal to deal
with
> > the aforementioned issue. At the end, we agreed on basically not going
with
> > my temporary solution that intended to solve the related problems of
> > Pikaday and we'd rather like to find and change for a better
alternative.
> >
> > To be fair, Pikaday is a pretty good date picker module, its only
problem
> > is the moment dependency if it's installed via npm. But other than
that, it
> > functions perfectly. Zero dependencies, small, etc. Long story short,
it's
> > good for us unless we want to get rid of moment.js.
> >
> > Before making a decision on what's next, I'd to ask you a question. Is
it
> > really a priority and is it really worth the effort to touch our
currently
> > used date picker component to get ~15% reduction in the bundle size by
> > removing moment?
> >
> > I'm asking it because if we want to do so, considering that it's a huge
> > topic, the following questions might come up:
> >
> > *A: What component do we want to use instead of Pikaday?*
> >
> > I'm not satisfied with the alternative individual solutions out there on
> > npm. I'd rather pick a component library like the angular port of
Bootstrap
> >  or the angular material library
> > . Both of them have a date picker
component
> > and many other components to rely on and reuse throughout the Metron
app.
> >
> > *B: What component library do we want to use?*
> >
> > Introducing a 

Re: [DISCUSS] Add e2e step to PR checklist

[Shane Ardell]

Nick - To be clear, when you say you can never get them all to pass, do you
mean you can never get all the tests to pass without protractor flake
re-running the failing tests (ie. eventually all the tests pass in the
end), or do you mean you still have failing tests even after
protractor-flake does its work? I want to look into this, but before I do I
want to make sure I understand you correctly.

On Fri, Oct 5, 2018 at 12:40 PM Casey Stella  wrote:

> This is really good feedback, Nick. I agree, we need them to be reliable
> enough to not be a source of constant false positives prior to putting them
> into the checklist.
> On Thu, Oct 4, 2018 at 15:34 Nick Allen  wrote:
>
> > I think we still have an issue of reliability.  I can never reliably get
> > them all to pass.  I have no idea which failures are real.  Am I the only
> > one that experiences this?
> >
> > We need a reliable pass/fail on these before we talk about adding them to
> > the checklist.  For example, I just tried to run them on METRON-1771.  I
> > don't think we have a problem with these changes, but I have not been
> able
> > to get one run to fully pass.  See the attached output of those runs.
> >
> >
> >
> > On Wed, Oct 3, 2018 at 7:36 AM Shane Ardell 
> > wrote:
> >
> >> I ran them locally a handful of times just now, and on average they took
> >> approximately 15 minutes to complete.
> >>
> >> On Tue, Oct 2, 2018, 18:22 Michael Miklavcic <
> michael.miklav...@gmail.com
> >> >
> >> wrote:
> >>
> >> > @Shane Just how much time are we talking about, on average? I don't
> >> think
> >> > many in the community have had much exposure to running the e2e tests
> in
> >> > their current form. It might still be worth it in the short term.
> >> >
> >> > On Tue, Oct 2, 2018 at 10:20 AM Shane Ardell <
> shane.m.ard...@gmail.com>
> >> > wrote:
> >> >
> >> > > The protractor-flake package should catch and re-run false failures,
> >> so
> >> > > people shouldn't get failing tests when they are done running. I
> just
> >> > meant
> >> > > that we often re-run flaky tests with protractor-flake, so it can
> >> take a
> >> > > while to run and could increase the build time considerably.
> >> > >
> >> > > On Tue, Oct 2, 2018, 18:00 Casey Stella  wrote:
> >> > >
> >> > > > Are the tests so brittle that, even with flaky, people will run
> upon
> >> > > false
> >> > > > failures as part of contributing a PR?  If so, do we have a list
> of
> >> the
> >> > > > brittle ones (and the things that would disambiguate a true
> failure
> >> > from
> >> > > a
> >> > > > false failure) that we can add to the documentation?
> >> > > >
> >> > > > On Tue, Oct 2, 2018 at 11:58 AM Shane Ardell <
> >> shane.m.ard...@gmail.com
> >> > >
> >> > > > wrote:
> >> > > >
> >> > > > > I also would like to eventually have these tests automated.
> There
> >> > are a
> >> > > > > couple hurdles to setting up our e2e tests to run with our
> build.
> >> I
> >> > > think
> >> > > > > the biggest hurdle is setting up a dedicated server with data
> for
> >> the
> >> > > e2e
> >> > > > > tests to use. I would assume this requires funding, engineering
> >> > > support,
> >> > > > > obfuscated data, etc. I also think we should migrate our e2e
> >> tests to
> >> > > > > Cypress first because Protractor lacks debugging tools that
> would
> >> > make
> >> > > > our
> >> > > > > life much easier if, for example, we had a failure in our CI
> build
> >> > but
> >> > > > > could not reproduce locally. In addition, our current Protractor
> >> > tests
> >> > > > are
> >> > > > > brittle and extremely slow.
> >> > > > >
> >> > > > > All that said, it seems we agree that we could add another PR
> >> > checklist
> >> > > > > item in the meantime. Clarifying those e2e test instructions
> >> should
> >> > be
> >> > > > part
> >> > > > > of that task.
> >> > > &

Invite to Slack Channel

[Shane Ardell]

Hello everyone,

Is it possible for someone to send me an invite to the Metron Slack channel?

Regards,
Shane

Re: [DISCUSS] Add e2e step to PR checklist

[Shane Ardell]

I ran them locally a handful of times just now, and on average they took
approximately 15 minutes to complete.

On Tue, Oct 2, 2018, 18:22 Michael Miklavcic 
wrote:

> @Shane Just how much time are we talking about, on average? I don't think
> many in the community have had much exposure to running the e2e tests in
> their current form. It might still be worth it in the short term.
>
> On Tue, Oct 2, 2018 at 10:20 AM Shane Ardell 
> wrote:
>
> > The protractor-flake package should catch and re-run false failures, so
> > people shouldn't get failing tests when they are done running. I just
> meant
> > that we often re-run flaky tests with protractor-flake, so it can take a
> > while to run and could increase the build time considerably.
> >
> > On Tue, Oct 2, 2018, 18:00 Casey Stella  wrote:
> >
> > > Are the tests so brittle that, even with flaky, people will run upon
> > false
> > > failures as part of contributing a PR?  If so, do we have a list of the
> > > brittle ones (and the things that would disambiguate a true failure
> from
> > a
> > > false failure) that we can add to the documentation?
> > >
> > > On Tue, Oct 2, 2018 at 11:58 AM Shane Ardell  >
> > > wrote:
> > >
> > > > I also would like to eventually have these tests automated. There
> are a
> > > > couple hurdles to setting up our e2e tests to run with our build. I
> > think
> > > > the biggest hurdle is setting up a dedicated server with data for the
> > e2e
> > > > tests to use. I would assume this requires funding, engineering
> > support,
> > > > obfuscated data, etc. I also think we should migrate our e2e tests to
> > > > Cypress first because Protractor lacks debugging tools that would
> make
> > > our
> > > > life much easier if, for example, we had a failure in our CI build
> but
> > > > could not reproduce locally. In addition, our current Protractor
> tests
> > > are
> > > > brittle and extremely slow.
> > > >
> > > > All that said, it seems we agree that we could add another PR
> checklist
> > > > item in the meantime. Clarifying those e2e test instructions should
> be
> > > part
> > > > of that task.
> > > >
> > > > On Mon, Oct 1, 2018 at 2:36 PM Casey Stella 
> > wrote:
> > > >
> > > > > I'd also like to make sure that clear instructions are provided (or
> > > > linked
> > > > > to) about how to run them.  Also, we need to make sure the
> > instructions
> > > > are
> > > > > rock-solid for running them.
> > > > > Looking at
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/metron/tree/master/metron-interface/metron-alerts#e2e-tests
> > > > > ,
> > > > > would someone who doesn't have much or any knowledge of the UI be
> > able
> > > to
> > > > > run that without assistance?
> > > > >
> > > > > For instance, we use full-dev, do we need to stop data from being
> > > played
> > > > > into full-dev for the tests to work?
> > > > >
> > > > > Casey
> > > > >
> > > > > On Mon, Oct 1, 2018 at 8:29 AM Casey Stella 
> > > wrote:
> > > > >
> > > > > > I'm not super keen on expanding the steps to contribute,
> especially
> > > in
> > > > an
> > > > > > avenue that should be automated.
> > > > > > That being said, I think that until we get to the point of
> > automating
> > > > the
> > > > > > e2e tests, it's sensible to add them to the checklist.
> > > > > > So, I would support it, but I would also urge us to move forward
> > the
> > > > > > efforts of running these tests as part of the CI build.
> > > > > >
> > > > > > What is the current gap there?
> > > > > >
> > > > > > Casey
> > > > > >
> > > > > > On Mon, Oct 1, 2018 at 7:41 AM Shane Ardell <
> > > shane.m.ard...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > >> Hello everyone,
> > > > > >>
> > > > > >> In another discussion thread from July, I briefly mentioned the
> > idea
> > > > of
> > > > > >> adding a step to the pull request checklist asking contributors
> to
> > > run
> > > > > the
> > > > > >> UI end-to-end tests. Since we aren't running e2e tests as part
> of
> > > the
> > > > CI
> > > > > >> build, it's easy for contributors to unintentionally break these
> > > > tests.
> > > > > >> Reminding contributors to run these tests will hopefully help
> > catch
> > > > > >> situations like this before opening a pull request.
> > > > > >>
> > > > > >> Does this make sense to everyone?
> > > > > >>
> > > > > >> Regards,
> > > > > >> Shane
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Add e2e step to PR checklist

[Shane Ardell]

The protractor-flake package should catch and re-run false failures, so
people shouldn't get failing tests when they are done running. I just meant
that we often re-run flaky tests with protractor-flake, so it can take a
while to run and could increase the build time considerably.

On Tue, Oct 2, 2018, 18:00 Casey Stella  wrote:

> Are the tests so brittle that, even with flaky, people will run upon false
> failures as part of contributing a PR?  If so, do we have a list of the
> brittle ones (and the things that would disambiguate a true failure from a
> false failure) that we can add to the documentation?
>
> On Tue, Oct 2, 2018 at 11:58 AM Shane Ardell 
> wrote:
>
> > I also would like to eventually have these tests automated. There are a
> > couple hurdles to setting up our e2e tests to run with our build. I think
> > the biggest hurdle is setting up a dedicated server with data for the e2e
> > tests to use. I would assume this requires funding, engineering support,
> > obfuscated data, etc. I also think we should migrate our e2e tests to
> > Cypress first because Protractor lacks debugging tools that would make
> our
> > life much easier if, for example, we had a failure in our CI build but
> > could not reproduce locally. In addition, our current Protractor tests
> are
> > brittle and extremely slow.
> >
> > All that said, it seems we agree that we could add another PR checklist
> > item in the meantime. Clarifying those e2e test instructions should be
> part
> > of that task.
> >
> > On Mon, Oct 1, 2018 at 2:36 PM Casey Stella  wrote:
> >
> > > I'd also like to make sure that clear instructions are provided (or
> > linked
> > > to) about how to run them.  Also, we need to make sure the instructions
> > are
> > > rock-solid for running them.
> > > Looking at
> > >
> > >
> >
> https://github.com/apache/metron/tree/master/metron-interface/metron-alerts#e2e-tests
> > > ,
> > > would someone who doesn't have much or any knowledge of the UI be able
> to
> > > run that without assistance?
> > >
> > > For instance, we use full-dev, do we need to stop data from being
> played
> > > into full-dev for the tests to work?
> > >
> > > Casey
> > >
> > > On Mon, Oct 1, 2018 at 8:29 AM Casey Stella 
> wrote:
> > >
> > > > I'm not super keen on expanding the steps to contribute, especially
> in
> > an
> > > > avenue that should be automated.
> > > > That being said, I think that until we get to the point of automating
> > the
> > > > e2e tests, it's sensible to add them to the checklist.
> > > > So, I would support it, but I would also urge us to move forward the
> > > > efforts of running these tests as part of the CI build.
> > > >
> > > > What is the current gap there?
> > > >
> > > > Casey
> > > >
> > > > On Mon, Oct 1, 2018 at 7:41 AM Shane Ardell <
> shane.m.ard...@gmail.com>
> > > > wrote:
> > > >
> > > >> Hello everyone,
> > > >>
> > > >> In another discussion thread from July, I briefly mentioned the idea
> > of
> > > >> adding a step to the pull request checklist asking contributors to
> run
> > > the
> > > >> UI end-to-end tests. Since we aren't running e2e tests as part of
> the
> > CI
> > > >> build, it's easy for contributors to unintentionally break these
> > tests.
> > > >> Reminding contributors to run these tests will hopefully help catch
> > > >> situations like this before opening a pull request.
> > > >>
> > > >> Does this make sense to everyone?
> > > >>
> > > >> Regards,
> > > >> Shane
> > > >>
> > > >
> > >
> >
>

Re: [DISCUSS] Add e2e step to PR checklist

[Shane Ardell]

I also would like to eventually have these tests automated. There are a
couple hurdles to setting up our e2e tests to run with our build. I think
the biggest hurdle is setting up a dedicated server with data for the e2e
tests to use. I would assume this requires funding, engineering support,
obfuscated data, etc. I also think we should migrate our e2e tests to
Cypress first because Protractor lacks debugging tools that would make our
life much easier if, for example, we had a failure in our CI build but
could not reproduce locally. In addition, our current Protractor tests are
brittle and extremely slow.

All that said, it seems we agree that we could add another PR checklist
item in the meantime. Clarifying those e2e test instructions should be part
of that task.

On Mon, Oct 1, 2018 at 2:36 PM Casey Stella  wrote:

> I'd also like to make sure that clear instructions are provided (or linked
> to) about how to run them.  Also, we need to make sure the instructions are
> rock-solid for running them.
> Looking at
>
> https://github.com/apache/metron/tree/master/metron-interface/metron-alerts#e2e-tests
> ,
> would someone who doesn't have much or any knowledge of the UI be able to
> run that without assistance?
>
> For instance, we use full-dev, do we need to stop data from being played
> into full-dev for the tests to work?
>
> Casey
>
> On Mon, Oct 1, 2018 at 8:29 AM Casey Stella  wrote:
>
> > I'm not super keen on expanding the steps to contribute, especially in an
> > avenue that should be automated.
> > That being said, I think that until we get to the point of automating the
> > e2e tests, it's sensible to add them to the checklist.
> > So, I would support it, but I would also urge us to move forward the
> > efforts of running these tests as part of the CI build.
> >
> > What is the current gap there?
> >
> > Casey
> >
> > On Mon, Oct 1, 2018 at 7:41 AM Shane Ardell 
> > wrote:
> >
> >> Hello everyone,
> >>
> >> In another discussion thread from July, I briefly mentioned the idea of
> >> adding a step to the pull request checklist asking contributors to run
> the
> >> UI end-to-end tests. Since we aren't running e2e tests as part of the CI
> >> build, it's easy for contributors to unintentionally break these tests.
> >> Reminding contributors to run these tests will hopefully help catch
> >> situations like this before opening a pull request.
> >>
> >> Does this make sense to everyone?
> >>
> >> Regards,
> >> Shane
> >>
> >
>

[DISCUSS] Add e2e step to PR checklist

[Shane Ardell]

Hello everyone,

In another discussion thread from July, I briefly mentioned the idea of
adding a step to the pull request checklist asking contributors to run the
UI end-to-end tests. Since we aren't running e2e tests as part of the CI
build, it's easy for contributors to unintentionally break these tests.
Reminding contributors to run these tests will hopefully help catch
situations like this before opening a pull request.

Does this make sense to everyone?

Regards,
Shane

Re: [DISCUSS] Migrate from Protractor to Cypress

[Shane Ardell]

I think Tibor's idea of using PCAP tests as an introduction to Cypress for
Metron is a great idea. As he pointed out, PCAP tests can take advantage of
Cypress' capability to mock responses, and we can set it up to run in
Travis. Once the community is able to see the benefits from an actual set
of Cypress tests inside the project and running in Travis, I think any
questions about migrating the rest of the existing tests from Protractor to
Cypress will be settled. However, if for some reason we run into issues
implementing or running the tests, we will have invested a fraction of time
vs. migrating all the tests right away.

On Wed, Sep 26, 2018 at 2:12 PM Tibor Meller  wrote:

> Hi Team,
>
> Many of us agreed on that Cypress could be a more capable tool for us to
> write high-level UI tests, whether those be e2e, integration or automated
> regression tests. If there is no open question left about cypress we could
> to bring it a test drive. My suggestion is to implement the PCAP UI tests
> with Cypress. Some services and PCAP semple data yet not available from our
> CI environment so protractor is hardly applicable here. This would be a
> great opportunity for cypress to shine. With Cypress, we are able to mock
> out those responses and make it run in Travis.
> Anytime we make PCAP data available in Travis we could be able to plug out
> those mocks and run the same test as integration or e2e tests if we like.
>
> Because it is relatively easy to migrate across cypress and protractor I
> see no major risks here if we decide to stick with Protractor for some
> reason.
>
> What do you think?
>
> Thanks for your feedback,
> Tibor
>
> On Wed, Sep 19, 2018 at 1:49 PM Shane Ardell 
> wrote:
>
> > Hello everyone,
> >
> > Currently, we use Protractor to run our UI "end-to-end" tests. However,
> > there are a handful of major advantages we can gain from switching to
> > Cypress: https://www.cypress.io/features/.
> >
> >- As with most Selenium-based e2e testing frameworks, Protractor
> suffers
> >from test flakiness. This is because Selenium runs outside of the
> > browser
> >and executes remote commands across the network. To work around this
> at
> > the
> >moment, we are using protractor-flake to re-run failed tests, but this
> > is
> >more of a crutch than a fix. Cypress executes in the same run loop as
> > the
> >application it's testing, and as a result does not suffer from the
> same
> >flakiness.
> >- As a result of its architecture, Cypress runs much faster than
> >Protractor. This is especially critical if e2e tests are added to the
> CI
> >build in the future.
> >- Protractor is incredibly hard to debug. In contrast, Cypress comes
> >with a plethora of debugging features, some of which you can see in
> > action
> >here: https://vimeo.com/242961930#t=264s
> >
> > Does anyone else have thoughts or opinions on switching to Cypress or
> > staying with Protractor?
> >
> > Cheers,
> > Shane
> >
>

Re: [DISCUSS] Migrate from Protractor to Cypress

[Shane Ardell]

While the Cypress team suggests taking advantage of stubs where you can,
especially during development, we would definitely be able to test real
endpoints [1]. It can be used exactly like how Protractor is used, but with
the many benefits and features it provides [2]. Cypress also offers tools
for unit testing [3], which I think may be causing confusion as to what
exactly the library does. Cypress' main focus is e2e tests, but because of
its architecture, it can be used for all types of tests.

I agree with everything you mentioned, Mike. I think our approach now is
fine, but in the future I do think it's worth considering the Cypress
team's suggestions for when and when not to stub, but there are no hard and
fast rules [4][5].

I currently have a branch available on my fork where I've migrated over
some e2e tests from Protractor to Cypress. With the exception of a little
code cleanup, these tests perform the same steps as they do with
Protractor. I have yet to include instructions in the README or include an
npm script, but if anyone wants to see it in action they can do the
following:

   - download this branch:
   https://github.com/sardell/metron/tree/METRON-1648,
   - run `npm ci` from meron-alerts,
   - start the e2e test server,
   - run  `./node_modules/.bin/cypress open`
   - start a single test by clicking on a file name in the Cypress user
   interface, or run them all by clicking the play button.

I'll try to send some sort of benchmarks when I get a chance to show the
speed difference between the two libraries.

[1] https://docs.cypress.io/api/commands/request.html
[2] https://www.cypress.io/features/
[3] https://docs.cypress.io/guides/guides/stubs-spies-and-clocks.html
[4]
https://docs.cypress.io/guides/guides/network-requests.html#Testing-Strategies
.
[5]
https://docs.cypress.io/guides/getting-started/testing-your-app.html#Stubbing-the-Server

On Thu, Sep 20, 2018 at 12:09 AM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Shane,
>
> Can you elaborate on the testing model you're proposing? I looked through
> the overview and some of the documentation. As far as I can tell, this
> would effectively be and e2e test for the UI *only*, so we would be missing
> testing the actual integration points with the REST API or any other
> potential endpoints.
>
>1. Are you proposing we migrate all existing e2e tests, including those
>that currently hit Elasticsearch?
>2. Would shifting to Cypress mean that all e2e tests would be isolated
>to only what is rendered via the browser? i.e. our e2e suite is no
> longer
>testing integration to a backend?
>
> My assumption with the term e2e testing is that you are testing an entire
> vertical slice with no substantive mock/stub/fake/spy/dummy [1] in the way
> except for maybe some strategic cross-cutting concerns. It sounds like
> Cypress does NOT mean full e2e. My initial reaction to this is that there's
> a place for both forms of testing. If Cypress would help UI developers work
> on incremental changes, similar to how unit tests via JUnit help Java
> developers iterate on features, then I think that's great. I'm all for
> that! But unit tests are only one form of testing - we also do integration
> testing, which can flex multiple classes/components together, as well as
> more broad stack integration/functional testing that verifies everything
> works when integrated together. Generally speaking, total # of unit tests >
> # of integration tests > # functional/acceptance tests. I think we should
> carve out and define a testing approach for each. Can you elaborate a bit
> on your vision for how to manage the test interactions, or lack thereof,
> with the REST API as an integration endpoint? [2]
>
> At the time the write-up James shared was written, it appears that Cypress
> was not yet open source. Now, it's MIT license -
> https://github.com/cypress-io/cypress/blob/develop/LICENSE.md.
>
> Mike
>
> 1.
>
> https://martinfowler.com/articles/mocksArentStubs.html#TheDifferenceBetweenMocksAndStubs
> 2. https://martinfowler.com/articles/practical-test-pyramid.html#UiTests
>
>
> On Wed, Sep 19, 2018 at 8:47 AM James Sirota  wrote:
>
> > This article comparing the two is not favorable for Cypress.  Are any of
> > these concerns relevant to us?  If not, then I think Cypress is fine
> >
> >
> >
> https://hackernoon.com/cypress-io-vs-protractor-e2e-testing-battle-d124ece91dc7
> >
> >
>

[DISCUSS] Migrate from Protractor to Cypress

[Shane Ardell]

Hello everyone,

Currently, we use Protractor to run our UI "end-to-end" tests. However,
there are a handful of major advantages we can gain from switching to
Cypress: https://www.cypress.io/features/.

   - As with most Selenium-based e2e testing frameworks, Protractor suffers
   from test flakiness. This is because Selenium runs outside of the browser
   and executes remote commands across the network. To work around this at the
   moment, we are using protractor-flake to re-run failed tests, but this is
   more of a crutch than a fix. Cypress executes in the same run loop as the
   application it's testing, and as a result does not suffer from the same
   flakiness.
   - As a result of its architecture, Cypress runs much faster than
   Protractor. This is especially critical if e2e tests are added to the CI
   build in the future.
   - Protractor is incredibly hard to debug. In contrast, Cypress comes
   with a plethora of debugging features, some of which you can see in action
   here: https://vimeo.com/242961930#t=264s

Does anyone else have thoughts or opinions on switching to Cypress or
staying with Protractor?

Cheers,
Shane

Re: package.lock changes during build?

[Shane Ardell]

NPM's use of lock files has been quite controversial. I won't go into it
too deep here as there are endless posts criticizing and justifying their
approach, but `npm install` will install all modules listed as dependencies
in package.json and update package-lock.json accordingly instead of
referencing the lock file. This caused a lot of outrage in the community (I
would argue rightfully so), which led to a compromise in release 5.7.1 with
`npm ci`. This command installs exactly what is specified in the
package-lock.json.
https://blog.npmjs.org/post/171556855892/introducing-npm-ci-for-faster-more-reliable

Metron's build currently uses `npm install`, which is why we are seeing the
package-lock.json update whenever we build locally. Coincidentally, I just
addressed this by switching to `npm ci` in an open PR of mine because I
noticed the same happening locally and I was already updating npm commands
in the pom.xml.
https://github.com/apache/metron/pull/1096/files#diff-e8f55f2d9e4f18085052a36d750e9648L60



On Sat, Aug 25, 2018 at 7:13 PM Casey Stella  wrote:

> Yeah, that's what I thought too, but I wonder if it triggers a change if
> there's a dependency that is not version locked (i.e. the most recent
> version of dependency x moved from y to z).
>
> On Sat, Aug 25, 2018 at 11:52 AM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > Somewhere along the line the dependencies appear to have changed, but the
> > file never got checked in. I don't like that this part of our build also
> > seems to be non-deterministic. If I build metron 0.4.x today, for
> instance,
> > what will I get? If the answer is "who knows?" that's unacceptable, imo.
> > I've glanced at the package file and see carrots littering the
> > dependencies, which as I understand it means "get me anything later than
> > this version." I do not think we should be doing that.
> >
> >
> > On Sat, Aug 25, 2018, 9:14 AM Casey Stella  wrote:
> >
> > > I have looked into this for other reasons and the guidance that I've
> seen
> > > is to check in package-lock.json into source control.  I'll leave this
> > > stack overflow thread here:
> > >
> > >
> >
> https://stackoverflow.com/questions/44206782/do-i-commit-the-package-lock-json-file-created-by-npm-5
> > >
> > > I want to point out that I hate that this changes as part of the build.
> > I
> > > haven't gotten a complete handle on exactly why package-lock is
> changing
> > > seemingly non-deterministically yet.
> > >
> > > Casey
> > >
> > > On Sat, Aug 25, 2018 at 11:05 AM Nick Allen 
> wrote:
> > >
> > > > Yes, I have noticed that also, but have not looked deeper.
> > > >
> > > > On Sat, Aug 25, 2018 at 10:32 AM Otto Fowler <
> ottobackwa...@gmail.com>
> > > > wrote:
> > > >
> > > > > I just did a PR, can saw that the package.lock file for alerts-ui
> was
> > > > > changed, with updated versions.
> > > > > I did *not* change the file, nor anything in metron-interface. That
> > > seems
> > > > > to imply that this file is changed or updated by
> > > > > something that happens during building or deploying full dev.
> > > > >
> > > > > Is this true?  How does this work?  Is this on purpose?
> > > > >
> > > > > ottO
> > > > >
> > > >
> > >
> >
>

Re: Metron-alerts E2E testing

[Shane Ardell]

For now, I think the easiest action for us to take here is to add a step to
the pull request checklist asking contributors to run the e2e tests. In the
future, we may want to look into moving to Cypress for e2e testing. One of
the many advantages of Cypress is that it can be run by Travis
.

On Tue, Jul 3, 2018 at 11:13 AM Tibor Meller  wrote:

> Hi All,
>
> Probably some of you already know that Shane and I working on stabilizing
> and extending e2e tests in metron-interface/metron-alerts.
> Currently, the tests using Protractor which is the default e2e testing
> framework of Angular.
> The tests injecting data right into ElasticSearch and making assertation on
> the alerts UI. This means the tests only end to end from a UI perspective.
> However, even with that, they are very useful in the validation of UI and
> to prevent regressions.
>
> It the current state of the development it would be crucial to these tests
> being run by every developer who making changes on the alerts UI code or in
> the Metron REST service.
> Without that, it is very hard to tell whether the tests are just flaky or
> the changes brake some of them.
>
> So please start to experimentaling with this tool, and let us know how it
> works out for you.
>
> You can find more information about setup and run on this link:
>
> https://github.com/tiborm/metron/blob/master/metron-interface/metron-alerts/README.md#e2e-tests
>
> Also if you have any opinion or idea on how to improve just let us know.
>
> Cheers,
> Tibor
>