Re: [DISCUSS] Batch Profiler Feature Branch

[Nick Allen]

Or support to be offered for merging this feature branch into master?

On Wed, Sep 26, 2018 at 6:20 PM Nick Allen  wrote:

> Thanks for the review.  With  https://github.com/apache/metron/pull/1209 
> complete,
> I think the feature branch is ready to be merged.  Sounds like I have
> Mike's support.  Anyone else have comments, concerns, questions?
>
> On Tue, Sep 25, 2018 at 10:33 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
>> I just made a couple minor comments on that PR, and I am in agreement
>> about
>> the readiness for merging with master. Good stuff Nick.
>>
>> On Fri, Sep 21, 2018 at 12:37 PM Nick Allen  wrote:
>>
>> > Here is a PR that adds the input time constraints to the Batch Profiler
>> > (METRON-1787);  https://github.com/apache/metron/pull/1209.
>> >
>> > It seems that the consensus is that this is probably the last feature we
>> > need before merging the FB into master.  The other two can wait until
>> after
>> > the feature branch has been merged.  Let me know if you disagree.
>> >
>> > Thanks
>> >
>> >
>> > On Thu, Sep 20, 2018 at 1:55 PM Nick Allen  wrote:
>> >
>> > > Yeah, agreed.  Per use case 3, when deploying to production there
>> really
>> > > wouldn't be a huge overlap like 3 months of already profiled data.
>> Its
>> > day
>> > > 1, the profile was just deployed around the same time as you are
>> running
>> > > the Batch Profiler, so the overlap is in minutes, maybe hours.  But I
>> can
>> > > definitely see the usefulness of the feature for re-runs, etc as you
>> have
>> > > described.
>> > >
>> > > Based on this discussion, I created a few JIRAs.  Thanks all for the
>> > great
>> > > feedback and keep it coming.
>> > >
>> > > [1] METRON-1787 - Input Time Constraints for Batch Profiler
>> > > [2] METRON-1788 - Fetch Profile Definitions from Zk for Batch Profiler
>> > > [3] METRON-1789 - MPack Should Define Default Input Path for Batch
>> > > Profiler
>> > >
>> > >
>> > > --
>> > > [1] https://issues.apache.org/jira/browse/METRON-1787
>> > > [2] https://issues.apache.org/jira/browse/METRON-1788
>> > > [3] https://issues.apache.org/jira/browse/METRON-1789
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > On Thu, Sep 20, 2018 at 1:34 PM Michael Miklavcic <
>> > > michael.miklav...@gmail.com> wrote:
>> > >
>> > >> I think we might want to allow the flexibility to choose the date
>> range
>> > >> then. I don't yet feel like I have a good enough understanding of all
>> > the
>> > >> ways in which users would want to seed to force them to run the batch
>> > job
>> > >> over all the data. It might also make it easier to deal with
>> > remediation,
>> > >> ie an error doesn't force you to re-run over the entire history. Same
>> > goes
>> > >> for testing out the profile seeing batch job in the first place.
>> > >>
>> > >> On Thu, Sep 20, 2018 at 11:23 AM Nick Allen 
>> wrote:
>> > >>
>> > >> > Assuming you have 9 months of data archived, yes.
>> > >> >
>> > >> > On Thu, Sep 20, 2018 at 1:22 PM Michael Miklavcic <
>> > >> > michael.miklav...@gmail.com> wrote:
>> > >> >
>> > >> > > So in the case of 3 - if you had 6 months of data that hadn't
>> been
>> > >> > profiled
>> > >> > > and another 3 that had been profiled (9 months total data), in
>> its
>> > >> > current
>> > >> > > form the batch job runs over all 9 months?
>> > >> > >
>> > >> > > On Thu, Sep 20, 2018 at 11:13 AM Nick Allen 
>> > >> wrote:
>> > >> > >
>> > >> > > > > How do we establish "tm" from 1.1 above? Any concerns about
>> > >> overlap
>> > >> > or
>> > >> > > > gaps after the seeding is performed?
>> > >> > > >
>> > >> > > > Good point.  Right now, if the Streaming and Batch Profiler
>> > overlap
>> > >> the
>> > >> > > > last write wins.  And presumably the output of the Streaming
>> and
>> > >> Batch
>> > >> > > > Profiler are the same, so no worries, right? :)
>> > >> > > >
>> > >> > > > So it kind of works, but it is definitely not ideal for use
>> case
>> > >> 3.  I
>> > >> > > > could add --begin and --end args to constrain the time frame
>> over
>> > >> which
>> > >> > > the
>> > >> > > > Batch Profiler runs.  I do not have that in the feature branch.
>> > It
>> > >> > would
>> > >> > > > be easy enough to add though.
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > > On Thu, Sep 20, 2018 at 12:41 PM Michael Miklavcic <
>> > >> > > > michael.miklav...@gmail.com> wrote:
>> > >> > > >
>> > >> > > > > Ok, makes sense. That's sort of what I was thinking as well,
>> > Nick.
>> > >> > > > Pulling
>> > >> > > > > at this thread just a bit more...
>> > >> > > > >
>> > >> > > > >1. I have an existing system that's been up a while, and I
>> > have
>> > >> > > added
>> > >> > > > k
>> > >> > > > >profiles - assume these are the first profiles I've
>> created.
>> > >> > > > >   1. I would have t0 - tm (where m is the time when the
>> > >> profiles
>> > >> > > were
>> > >> > > > >   first installed) worth of data that has not been
>> profiled
>> > >> yet.
>> > >> > > > >   2. The b

Re: [DISCUSS] Batch Profiler Feature Branch

[Nick Allen]

Thanks for the review.  With
https://github.com/apache/metron/pull/1209 complete,
I think the feature branch is ready to be merged.  Sounds like I have
Mike's support.  Anyone else have comments, concerns, questions?

On Tue, Sep 25, 2018 at 10:33 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> I just made a couple minor comments on that PR, and I am in agreement about
> the readiness for merging with master. Good stuff Nick.
>
> On Fri, Sep 21, 2018 at 12:37 PM Nick Allen  wrote:
>
> > Here is a PR that adds the input time constraints to the Batch Profiler
> > (METRON-1787);  https://github.com/apache/metron/pull/1209.
> >
> > It seems that the consensus is that this is probably the last feature we
> > need before merging the FB into master.  The other two can wait until
> after
> > the feature branch has been merged.  Let me know if you disagree.
> >
> > Thanks
> >
> >
> > On Thu, Sep 20, 2018 at 1:55 PM Nick Allen  wrote:
> >
> > > Yeah, agreed.  Per use case 3, when deploying to production there
> really
> > > wouldn't be a huge overlap like 3 months of already profiled data.  Its
> > day
> > > 1, the profile was just deployed around the same time as you are
> running
> > > the Batch Profiler, so the overlap is in minutes, maybe hours.  But I
> can
> > > definitely see the usefulness of the feature for re-runs, etc as you
> have
> > > described.
> > >
> > > Based on this discussion, I created a few JIRAs.  Thanks all for the
> > great
> > > feedback and keep it coming.
> > >
> > > [1] METRON-1787 - Input Time Constraints for Batch Profiler
> > > [2] METRON-1788 - Fetch Profile Definitions from Zk for Batch Profiler
> > > [3] METRON-1789 - MPack Should Define Default Input Path for Batch
> > > Profiler
> > >
> > >
> > > --
> > > [1] https://issues.apache.org/jira/browse/METRON-1787
> > > [2] https://issues.apache.org/jira/browse/METRON-1788
> > > [3] https://issues.apache.org/jira/browse/METRON-1789
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Thu, Sep 20, 2018 at 1:34 PM Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > >> I think we might want to allow the flexibility to choose the date
> range
> > >> then. I don't yet feel like I have a good enough understanding of all
> > the
> > >> ways in which users would want to seed to force them to run the batch
> > job
> > >> over all the data. It might also make it easier to deal with
> > remediation,
> > >> ie an error doesn't force you to re-run over the entire history. Same
> > goes
> > >> for testing out the profile seeing batch job in the first place.
> > >>
> > >> On Thu, Sep 20, 2018 at 11:23 AM Nick Allen 
> wrote:
> > >>
> > >> > Assuming you have 9 months of data archived, yes.
> > >> >
> > >> > On Thu, Sep 20, 2018 at 1:22 PM Michael Miklavcic <
> > >> > michael.miklav...@gmail.com> wrote:
> > >> >
> > >> > > So in the case of 3 - if you had 6 months of data that hadn't been
> > >> > profiled
> > >> > > and another 3 that had been profiled (9 months total data), in its
> > >> > current
> > >> > > form the batch job runs over all 9 months?
> > >> > >
> > >> > > On Thu, Sep 20, 2018 at 11:13 AM Nick Allen 
> > >> wrote:
> > >> > >
> > >> > > > > How do we establish "tm" from 1.1 above? Any concerns about
> > >> overlap
> > >> > or
> > >> > > > gaps after the seeding is performed?
> > >> > > >
> > >> > > > Good point.  Right now, if the Streaming and Batch Profiler
> > overlap
> > >> the
> > >> > > > last write wins.  And presumably the output of the Streaming and
> > >> Batch
> > >> > > > Profiler are the same, so no worries, right? :)
> > >> > > >
> > >> > > > So it kind of works, but it is definitely not ideal for use case
> > >> 3.  I
> > >> > > > could add --begin and --end args to constrain the time frame
> over
> > >> which
> > >> > > the
> > >> > > > Batch Profiler runs.  I do not have that in the feature branch.
> > It
> > >> > would
> > >> > > > be easy enough to add though.
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > On Thu, Sep 20, 2018 at 12:41 PM Michael Miklavcic <
> > >> > > > michael.miklav...@gmail.com> wrote:
> > >> > > >
> > >> > > > > Ok, makes sense. That's sort of what I was thinking as well,
> > Nick.
> > >> > > > Pulling
> > >> > > > > at this thread just a bit more...
> > >> > > > >
> > >> > > > >1. I have an existing system that's been up a while, and I
> > have
> > >> > > added
> > >> > > > k
> > >> > > > >profiles - assume these are the first profiles I've
> created.
> > >> > > > >   1. I would have t0 - tm (where m is the time when the
> > >> profiles
> > >> > > were
> > >> > > > >   first installed) worth of data that has not been
> profiled
> > >> yet.
> > >> > > > >   2. The batch profiler process would be to take that
> exact
> > >> > profile
> > >> > > > >   definition from ZK and run the batch loader with that
> from
> > >> the
> > >> > > CLI.
> > >> > > > >   3. Profiles are now up to date from t0 - tCurrent
> > >> > > > >2. I've already d

Re: [DISCUSS] Migrate from Protractor to Cypress

[Michael Miklavcic]

I'm good with it. We can see some tests in action (and hopefully running in
Travis! :-D) and then migrate and deprecate Protractor accordingly if we
still agree that's the way to go. When you submit the first PR, please link
to this DISCUSS via permalink from the mailing list archives. Thanks guys.

Cheers,
Mike

On Wed, Sep 26, 2018 at 7:17 AM Shane Ardell 
wrote:

> I think Tibor's idea of using PCAP tests as an introduction to Cypress for
> Metron is a great idea. As he pointed out, PCAP tests can take advantage of
> Cypress' capability to mock responses, and we can set it up to run in
> Travis. Once the community is able to see the benefits from an actual set
> of Cypress tests inside the project and running in Travis, I think any
> questions about migrating the rest of the existing tests from Protractor to
> Cypress will be settled. However, if for some reason we run into issues
> implementing or running the tests, we will have invested a fraction of time
> vs. migrating all the tests right away.
>
> On Wed, Sep 26, 2018 at 2:12 PM Tibor Meller 
> wrote:
>
> > Hi Team,
> >
> > Many of us agreed on that Cypress could be a more capable tool for us to
> > write high-level UI tests, whether those be e2e, integration or automated
> > regression tests. If there is no open question left about cypress we
> could
> > to bring it a test drive. My suggestion is to implement the PCAP UI tests
> > with Cypress. Some services and PCAP semple data yet not available from
> our
> > CI environment so protractor is hardly applicable here. This would be a
> > great opportunity for cypress to shine. With Cypress, we are able to mock
> > out those responses and make it run in Travis.
> > Anytime we make PCAP data available in Travis we could be able to plug
> out
> > those mocks and run the same test as integration or e2e tests if we like.
> >
> > Because it is relatively easy to migrate across cypress and protractor I
> > see no major risks here if we decide to stick with Protractor for some
> > reason.
> >
> > What do you think?
> >
> > Thanks for your feedback,
> > Tibor
> >
> > On Wed, Sep 19, 2018 at 1:49 PM Shane Ardell 
> > wrote:
> >
> > > Hello everyone,
> > >
> > > Currently, we use Protractor to run our UI "end-to-end" tests. However,
> > > there are a handful of major advantages we can gain from switching to
> > > Cypress: https://www.cypress.io/features/.
> > >
> > >- As with most Selenium-based e2e testing frameworks, Protractor
> > suffers
> > >from test flakiness. This is because Selenium runs outside of the
> > > browser
> > >and executes remote commands across the network. To work around this
> > at
> > > the
> > >moment, we are using protractor-flake to re-run failed tests, but
> this
> > > is
> > >more of a crutch than a fix. Cypress executes in the same run loop
> as
> > > the
> > >application it's testing, and as a result does not suffer from the
> > same
> > >flakiness.
> > >- As a result of its architecture, Cypress runs much faster than
> > >Protractor. This is especially critical if e2e tests are added to
> the
> > CI
> > >build in the future.
> > >- Protractor is incredibly hard to debug. In contrast, Cypress comes
> > >with a plethora of debugging features, some of which you can see in
> > > action
> > >here: https://vimeo.com/242961930#t=264s
> > >
> > > Does anyone else have thoughts or opinions on switching to Cypress or
> > > staying with Protractor?
> > >
> > > Cheers,
> > > Shane
> > >
> >
>

Re: [DISCUSS] Replacing Moment.js with date-fns or native functions

[Michael Miklavcic]

Sounds reasonable to me as well. Thank you for detailed explanation,
examples, and risk assessment! When you submit the PR for this, please link
back to the DISCUSS via permalink from the mailing list archives. +1

Thanks,
Mike

On Wed, Sep 26, 2018 at 6:40 AM Casey Stella  wrote:

> I think it's fine.  My only concern would be that we aren't accidentally
> using moment.js somewhere for something that date-fns doesn't do.  I
> suspect whoever picks up the ticket will figure that out pretty quick
> though. ;) . I'm +1 on the move; you convinced me.
>
> On Wed, Sep 26, 2018 at 8:36 AM Tamás Fodor  wrote:
>
> > I'd like to open a discussion about replacing Moment.js
> >  in Alerts UI. date-fns 
> has
> > almost the same functionality to manipulate date/time in the application.
> > Moment.js is a great library but it has a huge footprint in the bundle
> and
> > it's significant when it comes to performance optimization in order to
> > decrease the initial load time.
> >
> > Here you can find a brief introduction of the problem and a few ideas how
> > to replace it with date-fns or native functions.
> > https://github.com/you-dont-need/You-Dont-Need-Momentjs
> >
> > *The problem:*
> > In order to use Moment.js, we have to import the whole library which has
> > significant impact on the size of the production bundle of our javascript
> > code.
> >
> > *A few examples:*
> >
> >
> https://github.com/apache/metron/blob/e66cfc80e6a6fa53110c3f2fa8ee0d31ea997bf6/metron-interface/metron-alerts/src/app/utils/utils.ts#L18
> >
> >
> https://github.com/apache/metron/blob/ccdbeff5076553382091d4b9423ed48ccdba10ee/metron-interface/metron-alerts/src/app/shared/pipes/time-lapse.pipe.ts#L20
> >
> > Even though, we have tree-shaking
> >  which is a great feature
> of
> > javascript bundlers (in our case Webpack), because of how Moment.js is
> > structured, it prevents the bundler from doing tree-shaking properly
> > <
> >
> https://github.com/you-dont-need/You-Dont-Need-Momentjs/blob/master/README.md
> > >.
> > date-fns is just fine with that
> >  >.
> > For example, to use the `format` functions we don't have to import the
> > whole library, we can import only the format function
> >  individually.
> >
> > What do you think?
> >
> > Tamas
> >
>

Re: [DISCUSS] Migrate from Protractor to Cypress

[Shane Ardell]

I think Tibor's idea of using PCAP tests as an introduction to Cypress for
Metron is a great idea. As he pointed out, PCAP tests can take advantage of
Cypress' capability to mock responses, and we can set it up to run in
Travis. Once the community is able to see the benefits from an actual set
of Cypress tests inside the project and running in Travis, I think any
questions about migrating the rest of the existing tests from Protractor to
Cypress will be settled. However, if for some reason we run into issues
implementing or running the tests, we will have invested a fraction of time
vs. migrating all the tests right away.

On Wed, Sep 26, 2018 at 2:12 PM Tibor Meller  wrote:

> Hi Team,
>
> Many of us agreed on that Cypress could be a more capable tool for us to
> write high-level UI tests, whether those be e2e, integration or automated
> regression tests. If there is no open question left about cypress we could
> to bring it a test drive. My suggestion is to implement the PCAP UI tests
> with Cypress. Some services and PCAP semple data yet not available from our
> CI environment so protractor is hardly applicable here. This would be a
> great opportunity for cypress to shine. With Cypress, we are able to mock
> out those responses and make it run in Travis.
> Anytime we make PCAP data available in Travis we could be able to plug out
> those mocks and run the same test as integration or e2e tests if we like.
>
> Because it is relatively easy to migrate across cypress and protractor I
> see no major risks here if we decide to stick with Protractor for some
> reason.
>
> What do you think?
>
> Thanks for your feedback,
> Tibor
>
> On Wed, Sep 19, 2018 at 1:49 PM Shane Ardell 
> wrote:
>
> > Hello everyone,
> >
> > Currently, we use Protractor to run our UI "end-to-end" tests. However,
> > there are a handful of major advantages we can gain from switching to
> > Cypress: https://www.cypress.io/features/.
> >
> >- As with most Selenium-based e2e testing frameworks, Protractor
> suffers
> >from test flakiness. This is because Selenium runs outside of the
> > browser
> >and executes remote commands across the network. To work around this
> at
> > the
> >moment, we are using protractor-flake to re-run failed tests, but this
> > is
> >more of a crutch than a fix. Cypress executes in the same run loop as
> > the
> >application it's testing, and as a result does not suffer from the
> same
> >flakiness.
> >- As a result of its architecture, Cypress runs much faster than
> >Protractor. This is especially critical if e2e tests are added to the
> CI
> >build in the future.
> >- Protractor is incredibly hard to debug. In contrast, Cypress comes
> >with a plethora of debugging features, some of which you can see in
> > action
> >here: https://vimeo.com/242961930#t=264s
> >
> > Does anyone else have thoughts or opinions on switching to Cypress or
> > staying with Protractor?
> >
> > Cheers,
> > Shane
> >
>

Re: [DISCUSS] Replacing Moment.js with date-fns or native functions

[Casey Stella]

I think it's fine.  My only concern would be that we aren't accidentally
using moment.js somewhere for something that date-fns doesn't do.  I
suspect whoever picks up the ticket will figure that out pretty quick
though. ;) . I'm +1 on the move; you convinced me.

On Wed, Sep 26, 2018 at 8:36 AM Tamás Fodor  wrote:

> I'd like to open a discussion about replacing Moment.js
>  in Alerts UI. date-fns  has
> almost the same functionality to manipulate date/time in the application.
> Moment.js is a great library but it has a huge footprint in the bundle and
> it's significant when it comes to performance optimization in order to
> decrease the initial load time.
>
> Here you can find a brief introduction of the problem and a few ideas how
> to replace it with date-fns or native functions.
> https://github.com/you-dont-need/You-Dont-Need-Momentjs
>
> *The problem:*
> In order to use Moment.js, we have to import the whole library which has
> significant impact on the size of the production bundle of our javascript
> code.
>
> *A few examples:*
>
> https://github.com/apache/metron/blob/e66cfc80e6a6fa53110c3f2fa8ee0d31ea997bf6/metron-interface/metron-alerts/src/app/utils/utils.ts#L18
>
> https://github.com/apache/metron/blob/ccdbeff5076553382091d4b9423ed48ccdba10ee/metron-interface/metron-alerts/src/app/shared/pipes/time-lapse.pipe.ts#L20
>
> Even though, we have tree-shaking
>  which is a great feature of
> javascript bundlers (in our case Webpack), because of how Moment.js is
> structured, it prevents the bundler from doing tree-shaking properly
> <
> https://github.com/you-dont-need/You-Dont-Need-Momentjs/blob/master/README.md
> >.
> date-fns is just fine with that
> .
> For example, to use the `format` functions we don't have to import the
> whole library, we can import only the format function
>  individually.
>
> What do you think?
>
> Tamas
>

[DISCUSS] Replacing Moment.js with date-fns or native functions

[Tamás Fodor]

I'd like to open a discussion about replacing Moment.js
 in Alerts UI. date-fns  has
almost the same functionality to manipulate date/time in the application.
Moment.js is a great library but it has a huge footprint in the bundle and
it's significant when it comes to performance optimization in order to
decrease the initial load time.

Here you can find a brief introduction of the problem and a few ideas how
to replace it with date-fns or native functions.
https://github.com/you-dont-need/You-Dont-Need-Momentjs

*The problem:*
In order to use Moment.js, we have to import the whole library which has
significant impact on the size of the production bundle of our javascript
code.

*A few examples:*
https://github.com/apache/metron/blob/e66cfc80e6a6fa53110c3f2fa8ee0d31ea997bf6/metron-interface/metron-alerts/src/app/utils/utils.ts#L18
https://github.com/apache/metron/blob/ccdbeff5076553382091d4b9423ed48ccdba10ee/metron-interface/metron-alerts/src/app/shared/pipes/time-lapse.pipe.ts#L20

Even though, we have tree-shaking
 which is a great feature of
javascript bundlers (in our case Webpack), because of how Moment.js is
structured, it prevents the bundler from doing tree-shaking properly
.
date-fns is just fine with that
.
For example, to use the `format` functions we don't have to import the
whole library, we can import only the format function
 individually.

What do you think?

Tamas

Re: [DISCUSS] Migrate from Protractor to Cypress

[Tibor Meller]

Hi Team,

Many of us agreed on that Cypress could be a more capable tool for us to
write high-level UI tests, whether those be e2e, integration or automated
regression tests. If there is no open question left about cypress we could
to bring it a test drive. My suggestion is to implement the PCAP UI tests
with Cypress. Some services and PCAP semple data yet not available from our
CI environment so protractor is hardly applicable here. This would be a
great opportunity for cypress to shine. With Cypress, we are able to mock
out those responses and make it run in Travis.
Anytime we make PCAP data available in Travis we could be able to plug out
those mocks and run the same test as integration or e2e tests if we like.

Because it is relatively easy to migrate across cypress and protractor I
see no major risks here if we decide to stick with Protractor for some
reason.

What do you think?

Thanks for your feedback,
Tibor

On Wed, Sep 19, 2018 at 1:49 PM Shane Ardell 
wrote:

> Hello everyone,
>
> Currently, we use Protractor to run our UI "end-to-end" tests. However,
> there are a handful of major advantages we can gain from switching to
> Cypress: https://www.cypress.io/features/.
>
>- As with most Selenium-based e2e testing frameworks, Protractor suffers
>from test flakiness. This is because Selenium runs outside of the
> browser
>and executes remote commands across the network. To work around this at
> the
>moment, we are using protractor-flake to re-run failed tests, but this
> is
>more of a crutch than a fix. Cypress executes in the same run loop as
> the
>application it's testing, and as a result does not suffer from the same
>flakiness.
>- As a result of its architecture, Cypress runs much faster than
>Protractor. This is especially critical if e2e tests are added to the CI
>build in the future.
>- Protractor is incredibly hard to debug. In contrast, Cypress comes
>with a plethora of debugging features, some of which you can see in
> action
>here: https://vimeo.com/242961930#t=264s
>
> Does anyone else have thoughts or opinions on switching to Cypress or
> staying with Protractor?
>
> Cheers,
> Shane
>