Re: [DISCUSS] Deprecate split-join enrichment topology in favor of unified enrichment topology

[Michael Miklavcic]

Hey @Ali - check this out - https://github.com/apache/metron/pull/1286.
Obviously, we're not going to have all of the same timestamp properties in
the unified topology since there are no more splitters and joiners, but
this should fill in the outstanding gaps around the adapter timestamps. I
see this as a short-term stopgap solution for metrics. I add these
properties back in with the caveat that users should expect them to be
deprecated at some point in favor of a more robust metrics solution. Let me
know if there's anything you feel we're still missing.

Best,
Mike Miklavcic

On Tue, Nov 20, 2018 at 11:01 AM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Thanks for pointing that out Ali. I created a ticket to track it -
> https://issues.apache.org/jira/browse/METRON-1889
>
> On Mon, Nov 19, 2018 at 11:51 PM Ali Nazemian 
> wrote:
>
>> Hi,
>>
>> One thing to point out here is there were a few timestamp fields that
>> exist for Split-join enrichment topology that haven't been made to the
>> unified one. For example, there is no threat intel bolt timestamp. There
>> might be some SLA related use cases regarding these timestamp fields that
>> might be nice to have before depreciation of the split-join one. Generally
>> speaking, makes sense to deprecate split-join topology, though.
>>
>> Cheers,
>> Ali
>>
>> On Fri, Nov 16, 2018 at 3:40 AM James Sirota  wrote:
>>
>> > This is excellent work, Mike and long overdue.  Thanks for doing this
>> >
>> > 05.11.2018, 16:46, "Michael Miklavcic" :
>> > > The PR has now been merged into master and closed.
>> > >
>> > > https://issues.apache.org/jira/browse/METRON-1855
>> > >
>> > > On Sat, Nov 3, 2018 at 6:47 PM Michael Miklavcic <
>> > > michael.miklav...@gmail.com> wrote:
>> > >
>> > >>  PR is out here - https://github.com/apache/metron/pull/1252
>> > >>
>> > >>  I made the unified enrichment topology the new default and marked
>> the
>> > >>  split-join topology as deprecated in various parts of the
>> > documentation. I
>> > >>  think we should have a release with the deprecation notes and new
>> > default
>> > >>  and then move to remove split-join entirely shortly thereafter.
>> > >>
>> > >>  Best,
>> > >>  Mike
>> > >>
>> > >>  On Fri, Nov 2, 2018 at 5:47 AM Mohan Venkateshaiah <
>> > >>  mvenkatesha...@hortonworks.com> wrote:
>> > >>
>> > >>>  +1 (non-binding)
>> > >>>
>> > >>>  Thanks
>> > >>>  Mohan DV
>> > >>>
>> > >>>  On 11/2/18, 3:29 PM, "zeo...@gmail.com"  wrote:
>> > >>>
>> > >>>  +1 totally agree.
>> > >>>
>> > >>>  Jon
>> > >>>
>> > >>>  On Fri, Nov 2, 2018, 1:31 AM Anand Subramanian <
>> > >>>  asubraman...@hortonworks.com>
>> > >>>  wrote:
>> > >>>
>> > >>>  > Piling on my +1 (non-binding) as well.
>> > >>>  >
>> > >>>  > On 11/2/18, 4:41 AM, "Ryan Merriman" 
>> > wrote:
>> > >>>  >
>> > >>>  > +1
>> > >>>  >
>> > >>>  > On Thu, Nov 1, 2018 at 5:38 PM Casey Stella <
>> ceste...@gmail.com
>> > >>>  >
>> > >>>  > wrote:
>> > >>>  >
>> > >>>  > > +1
>> > >>>  > > On Thu, Nov 1, 2018 at 18:34 Nick Allen <
>> n...@nickallen.org>
>> > >>>  wrote:
>> > >>>  > >
>> > >>>  > > > +1
>> > >>>  > > >
>> > >>>  > > > On Thu, Nov 1, 2018, 6:27 PM Justin Leet <
>> > >>>  justinjl...@gmail.com>
>> > >>>  > wrote:
>> > >>>  > > >
>> > >>>  > > > > +1, I haven't seen any case where the split-join
>> topology
>> > >>>  isn't
>> > >>>  > made
>> > >>>  > > > > obsolete by the unified topology.
>> > >>>  > > > >
>> > >>>  > > > > On Thu, Nov 1, 2018 at 6:17 PM Michael Miklavcic <
>> > >>>  > > > > michael.miklav...@gmail.com> wrote:
>> > >>>  > > > >
>> > >>>  > > > > > Fellow Metronians,
>> > >>>  > > > > >
>> > >>>  > > > > > We've had the unified enrichment topology around for
>> a
>> > >>>  number
>> > >>>  > of
>> > >>>  > > months
>> > >>>  > > > > > now, it has proved itself stable, and there is yet to
>> > >>>  be a
>> > >>>  > time that
>> > >>>  > > I
>> > >>>  > > > > have
>> > >>>  > > > > > seen the split-join topology outperform the unified
>> > >>>  one. Here
>> > >>>  > are
>> > >>>  > > some
>> > >>>  > > > > > simple reasons to deprecate the split-join topology.
>> > >>>  > > > > >
>> > >>>  > > > > > 1. Unified topology performs better.
>> > >>>  > > > > > 2. The configuration, especially for performance
>> > >>>  tuning is
>> > >>>  > much,
>> > >>>  > > > much
>> > >>>  > > > > > simpler in the unified model.
>> > >>>  > > > > > 3. The footprint within the cluster is smaller.
>> > >>>  > > > > > 4. One of the first activities for any install is
>> > >>>  that we
>> > >>>  > spend
>> > >>>  > > time
>> > >>>  > > > > > instructing users to switch to the unified topology.
>> > >>>  > > > > > 5. One less moving part to maintain.
>> > >>>  > > > > >
>> > >>>  > > > > > I'd like to recommend that we deprecate the
>> split-join

Re: [DISCUSS] Managing intermittent test failures

[Michael Miklavcic]

I added a comment here -
https://cwiki.apache.org/confluence/display/METRON/Reporting+Issues

On Thu, Nov 29, 2018 at 9:22 AM Casey Stella  wrote:

> +1, I'd say mention it on the dev list and slack channel.
>
> On Thu, Nov 29, 2018 at 10:26 AM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > Every now and then we see intermittent test failures, and rather than
> > sweeping them under the rug, we should have a simple method to track and
> > handle them. I started creating Jiras for tests that I've seen fail, but
> > that don't fail consistently, or even fail more than once. For example,
> > https://issues.apache.org/jira/browse/METRON-1851.
> >
> > I think we're all taking steps to varying degrees already, but I want to
> > call it out formally. I propose we create a ticket and add the label
> > "test-failure." It might also make sense to send a quick note to the dev
> > list or Slack channel, so attention can be brought to it and anyone else
> > that may have run into an issue with the test can chime in. We can clean
> > them out every few months - maybe do a review going into a release and
> > close any that have not been reproduced for some time. What do you all
> > think?
> >
> > Mike
> >
>

Re: [DISCUSS] Managing intermittent test failures

[Casey Stella]

+1, I'd say mention it on the dev list and slack channel.

On Thu, Nov 29, 2018 at 10:26 AM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Every now and then we see intermittent test failures, and rather than
> sweeping them under the rug, we should have a simple method to track and
> handle them. I started creating Jiras for tests that I've seen fail, but
> that don't fail consistently, or even fail more than once. For example,
> https://issues.apache.org/jira/browse/METRON-1851.
>
> I think we're all taking steps to varying degrees already, but I want to
> call it out formally. I propose we create a ticket and add the label
> "test-failure." It might also make sense to send a quick note to the dev
> list or Slack channel, so attention can be brought to it and anyone else
> that may have run into an issue with the test can chime in. We can clean
> them out every few months - maybe do a review going into a release and
> close any that have not been reproduced for some time. What do you all
> think?
>
> Mike
>

Re: [DISCUSS] Managing intermittent test failures

[Otto Fowler]

+1


On November 29, 2018 at 10:26:53, Michael Miklavcic (
michael.miklav...@gmail.com) wrote:

Every now and then we see intermittent test failures, and rather than
sweeping them under the rug, we should have a simple method to track and
handle them. I started creating Jiras for tests that I've seen fail, but
that don't fail consistently, or even fail more than once. For example,
https://issues.apache.org/jira/browse/METRON-1851.

I think we're all taking steps to varying degrees already, but I want to
call it out formally. I propose we create a ticket and add the label
"test-failure." It might also make sense to send a quick note to the dev
list or Slack channel, so attention can be brought to it and anyone else
that may have run into an issue with the test can chime in. We can clean
them out every few months - maybe do a review going into a release and
close any that have not been reproduced for some time. What do you all
think?

Mike

Re: [DISCUSS] Managing intermittent test failures

[Justin Leet]

Well, since I just put out pretty much this same request on the dev thread,
I'm +1.

On Thu, Nov 29, 2018 at 10:26 AM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Every now and then we see intermittent test failures, and rather than
> sweeping them under the rug, we should have a simple method to track and
> handle them. I started creating Jiras for tests that I've seen fail, but
> that don't fail consistently, or even fail more than once. For example,
> https://issues.apache.org/jira/browse/METRON-1851.
>
> I think we're all taking steps to varying degrees already, but I want to
> call it out formally. I propose we create a ticket and add the label
> "test-failure." It might also make sense to send a quick note to the dev
> list or Slack channel, so attention can be brought to it and anyone else
> that may have run into an issue with the test can chime in. We can clean
> them out every few months - maybe do a review going into a release and
> close any that have not been reproduced for some time. What do you all
> think?
>
> Mike
>

Re: Metron Release 0.6.1 and/or Plugin release 0.3.0?

[Justin Leet]

There's a few issues I would like to see at least triaged and preferably
addressed prior to the release of the main repo. In Jira, we have a
"test-failures" label, that has a few things attached to it. If we know of
any other Jiras that should have this label attached, please do so and I'd
appreciate it if you replied to the thread.  See test-failures

.

The Jiras are:
METRON-1810 
METRON-1814 
METRON-1851 

On Wed, Nov 21, 2018 at 2:20 PM zeo...@gmail.com  wrote:

> A metron-bro-plugin-kafka 0.3 release is good to go from my side.  Thanks
> for all of the reviews Nick
>
> On Wed, Nov 21, 2018 at 11:16 AM Nick Allen  wrote:
>
> > Ha.  Yes, that definitely counts and makes a ton of sense.  Thanks!
> >
> > On Wed, Nov 21, 2018 at 11:00 AM Justin Leet 
> > wrote:
> >
> > > Does "I forgot to pull master fresh before running the command" count
> as
> > a
> > > reason?
> > >
> > > The missing Jiras are:
> > >
> > > METRON-1890 Metron Vagrant should disable audio (ottobackwards)
> > closes
> > > apache/metron#1277
> > > METRON-1874 Create a Parser Debugger (nickwallen) closes
> > > apache/metron#1265
> > > METRON-1880 Use Caffeine for Profiler Caching (nickwallen) closes
> > > apache/metron#1270
> > > METRON-1877 Nested IF ELSE statements can cause parse errors in
> > Stellar
> > > (justinleet) closes apache/metron#1268
> > > METRON-1872 Move rat plugin away from snapshot version (justinleet)
> > > closes apache/metron#1264
> > >
> > > On Wed, Nov 21, 2018 at 10:59 AM Nick Allen 
> wrote:
> > >
> > > > Also, I'd like to get this one included in the release.  This is
> really
> > > > annoying for people just wanting to try out the Profiler.  And this
> was
> > > > 'broken' after the last release, so there currently is no release
> with
> > > this
> > > > problem and I'd like to keep it that way. :)
> > > >
> > > > https://github.com/apache/metron/pull/1276
> > > >
> > > > On Wed, Nov 21, 2018 at 10:11 AM Justin Leet 
> > > > wrote:
> > > >
> > > > > Realized I'd never sent the updated list of Jiras.  I changed the
> > > command
> > > > > slightly (to remove a clause I thought we'd already removed re:
> http,
> > > and
> > > > > added the awk to remove dupes resulting from multiple commits for a
> > > > single
> > > > > Jira. I'll do a PR for these changes).
> > > > >
> > > > > *apache/metron*
> > > > > git log "master" "^tags/apache-metron-0.6.0-release" --no-merges |
> > grep
> > > > -E
> > > > > "^[[:blank:]]+METRON" | sed 's/\[//g' | sed 's/\]//g' | awk
> > '!x[$1]++'
> > > > > METRON-1875 Expose configurable global settings in the Alerts
> UI
> > > > > (merrimanr) closes apache/metron#1266
> > > > > METRON-1834: Migrate Elasticsearch from TransportClient to new
> > Java
> > > > > REST API (mmiklavc via mmiklavc) closes apache/metron#1242
> > > > > METRON-1749 Update Angular to latest release in Management UI
> > > > (sardell
> > > > > via nickwallen) closes apache/metron#1217
> > > > > METRON-1870 Intermittent Stellar REST test failures (merrimanr
> > via
> > > > > nickwallen) closes apache/metron#1263
> > > > > METRON-1868 metron-committer-common incorrectly checking
> > REPO_NAME
> > > > > (JonZeolla via jonzeolla) closes apache/metron#1260
> > > > > METRON-1740 Improve Palo Alto parser to handle CONFIG and
> SYSTEM
> > > > syslog
> > > > > messages (liuy-tnz via nickwallen) closes apache/metron#1171
> > > > > METRON-1847 Create reusable script with functions from
> > > prepare-commit
> > > > > (ottobackwards) closes apache/metron#1248
> > > > > METRON-1850 Stellar REST function (merrimanr) closes
> > > > apache/metron#1250
> > > > > METRON-1858 BasicFireEyeParser check style cleanup and
> > optimization
> > > > > (ottobackwards) closes apache/metron#1255
> > > > > METRON-1864 Stellar date format test fails after daylight
> saving
> > > > > (ottobackwards) closes apache/metron#1258
> > > > > METRON-1861 METRON-1861: REST fails to start when LDAP enabled
> > and
> > > > > 'Active Spring profiles' config is empty (anandsubbu via
> justinleet)
> > > > closes
> > > > > apache/metron#1256
> > > > > METRON-1853: Add shutdown hook to Stellar BaseFunctionResolver
> > > > > (mmiklavc via mmiklavc) closes apache/metron#1251
> > > > > METRON-1857 Fix Metaalert Nested Alert Field Name in Index
> > Template
> > > > > (nickwallen) closes apache/metron#1253
> > > > > METRON-1855: Make unified enrichment topology the default and
> > > > deprecate
> > > > > split-join (mmiklavc via mmiklavc) closes apache/metron#1252
> > > > > METRON-1790 Unsubscribe from every observable in the pcap panel
> > UI
> > > > > component (ruffle via nickwallen) closes apache/metron#1208
> > > > > METR

[DISCUSS] Managing intermittent test failures

[Michael Miklavcic]

Every now and then we see intermittent test failures, and rather than
sweeping them under the rug, we should have a simple method to track and
handle them. I started creating Jiras for tests that I've seen fail, but
that don't fail consistently, or even fail more than once. For example,
https://issues.apache.org/jira/browse/METRON-1851.

I think we're all taking steps to varying degrees already, but I want to
call it out formally. I propose we create a ticket and add the label
"test-failure." It might also make sense to send a quick note to the dev
list or Slack channel, so attention can be brought to it and anyone else
that may have run into an issue with the test can chime in. We can clean
them out every few months - maybe do a review going into a release and
close any that have not been reproduced for some time. What do you all
think?

Mike

Re: Unzipping Cypress

[Shane Ardell]

No problem! Glad I could help.

On Wed, Nov 28, 2018 at 5:03 PM Otto Fowler  wrote:

> OK,
>
> I think what is happening is that in my PR, I’m building metron in Docker
> and deploying to vagrant.  I have updated my PR to map the cypress cache
> into the Docker container.
> Thanks!
>
>
> On November 28, 2018 at 10:29:25, Shane Ardell (shane.m.ard...@gmail.com)
> wrote:
>
> For me, it's at ~/Library/Caches/Cypress, but the path depends on your OS:
>
> https://docs.cypress.io/guides/getting-started/installing-cypress.html#Binary-cache
>
> On Wed, Nov 28, 2018 at 4:19 PM Otto Fowler 
> wrote:
>
> > Where is the cache path?
> >
> >
> > On November 28, 2018 at 09:34:18, Shane Ardell (shane.m.ard...@gmail.com
> )
> > wrote:
> >
> > https://github.com/cypress-io/cypress/issues/1813
> >
>