Re: [VOTE] Update dev guidelines with format for sharing architecture source files and rendered images

[Michael Miklavcic]

Good idea - I will make that addendum. I would consider that, ipso facto,
acceptable to everyone on the thread unless they say otherwise -
considering the entire point of this vote is to standardize the diagram
tool, it would be inconsistent with that goal to use the word "should."
"Must" makes sense here.

On Fri, May 3, 2019, 3:19 PM zeo...@gmail.com  wrote:

> +1 non-binding
>
> I would only prefer that we change "Appropriate architecture diagrams
> should be created in" to "Appropriate architecture diagrams must be created
> in" but I'm good either way.
>
> - Jon Zeolla
> zeo...@gmail.com
>
>
> On Fri, May 3, 2019 at 10:18 AM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > Yes, it is free James. We made sure of that in the original discussion.
> >
> > On Thu, May 2, 2019 at 9:33 PM James Sirota  wrote:
> >
> > > i am ok with it as long as we are not forcing people to buy stuff
> > >
> > > 02.05.2019, 18:18, "Michael Miklavcic" :
> > > > Here's the latest discussion on the subject:
> > > >
> > >
> >
> https://lists.apache.org/thread.html/0aa2b0b9ed4a0f0b0d8bb018c618e62de196565f9af71f347e504076@%3Cdev.metron.apache.org%3E
> > > >
> > > > I'd like to propose a vote to change our dev guidelines which will
> > > clarify
> > > > the tooling we use to produce diagrams and share the source files for
> > > those
> > > > diagrams. I propose the dev guidelines
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines
> > > and
> > > > PR checklist
> > > >
> > >
> >
> https://github.com/apache/metron/blob/master/.github/PULL_REQUEST_TEMPLATE.md#for-documentation-related-changes
> > > > be
> > > > changed in the following ways:
> > > >
> > > >1. Under "1.1 Contributing A Code Change"
> > > >   1. Change <<"New features and significant bug fixes should be
> > > >   documented in the JIRA and appropriate architecture diagrams
> > > should be
> > > >   attached. Major features may require a vote.">> to <<"New
> > features
> > > >   and significant bug fixes should be documented in the JIRA.
> > > Appropriate
> > > >   architecture diagrams should be created in
> https://www.draw.io/
> > > > and committed
> > > >   to source control as per section 2.4. Diagrams may be requested
> > of
> > > PR
> > > >   submitters during review either as documentation or as an aid
> to
> > > the
> > > >   reviewer. Major features may also require a vote.">>
> > > >2. Under "2.4 Documentation"
> > > >   1. New line item <<"Diagrams - We save architecture diagram
> > source
> > > >   files in an xml format rendered by draw.io (instructions
> below).
> > > This
> > > >   is the free tool of choice that we've agreed to use for
> > exchanging
> > > >   diagrams and their source files in Metron.">>
> > > >   2. New line item < > > >   "/images-source" and rendered diagrams and images
> > > belong in
> > > >   "/images."
> > > >   3. New subsection <<"Creating and Modifying Diagrams">>. This
> > > section
> > > >   would provide basic instructions for downloading source files
> > from
> > > >   draw.io.
> > > >3. Add a new checkbox item under PR checklist heading "For
> > > documentation
> > > >related changes" with the following text
> > > >   1. Have you ensured that any documentation diagrams have been
> > > >   updated, along with their source files, using draw.io? See
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines
> > > > for
> > > >   instructions.
> > > >4. Here is the Jira for migrating/redoing existing diagrams
> > > >   1. https://issues.apache.org/jira/browse/METRON-2099
> > > >
> > > > We require a minimum of 72 hours for a vote, not typically including
> > > > weekend days. I'd like to leave this vote open until Wednesday 5/8,
> > 12PM
> > > > EDT. Please vote +1, -1, or 0 to abstain, and also indicate if your
> > vote
> > > is
> > > > binding or non-binding.
> > >
> > > ---
> > > Thank you,
> > >
> > > James Sirota
> > > PMC- Apache Metron
> > > jsirota AT apache DOT org
> > >
> > >
> >
>

Re: [VOTE] Update dev guidelines with format for sharing architecture source files and rendered images

[zeo...@gmail.com]

+1 non-binding

I would only prefer that we change "Appropriate architecture diagrams
should be created in" to "Appropriate architecture diagrams must be created
in" but I'm good either way.

- Jon Zeolla
zeo...@gmail.com


On Fri, May 3, 2019 at 10:18 AM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Yes, it is free James. We made sure of that in the original discussion.
>
> On Thu, May 2, 2019 at 9:33 PM James Sirota  wrote:
>
> > i am ok with it as long as we are not forcing people to buy stuff
> >
> > 02.05.2019, 18:18, "Michael Miklavcic" :
> > > Here's the latest discussion on the subject:
> > >
> >
> https://lists.apache.org/thread.html/0aa2b0b9ed4a0f0b0d8bb018c618e62de196565f9af71f347e504076@%3Cdev.metron.apache.org%3E
> > >
> > > I'd like to propose a vote to change our dev guidelines which will
> > clarify
> > > the tooling we use to produce diagrams and share the source files for
> > those
> > > diagrams. I propose the dev guidelines
> > >
> >
> https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines
> > and
> > > PR checklist
> > >
> >
> https://github.com/apache/metron/blob/master/.github/PULL_REQUEST_TEMPLATE.md#for-documentation-related-changes
> > > be
> > > changed in the following ways:
> > >
> > >1. Under "1.1 Contributing A Code Change"
> > >   1. Change <<"New features and significant bug fixes should be
> > >   documented in the JIRA and appropriate architecture diagrams
> > should be
> > >   attached. Major features may require a vote.">> to <<"New
> features
> > >   and significant bug fixes should be documented in the JIRA.
> > Appropriate
> > >   architecture diagrams should be created in https://www.draw.io/
> > > and committed
> > >   to source control as per section 2.4. Diagrams may be requested
> of
> > PR
> > >   submitters during review either as documentation or as an aid to
> > the
> > >   reviewer. Major features may also require a vote.">>
> > >2. Under "2.4 Documentation"
> > >   1. New line item <<"Diagrams - We save architecture diagram
> source
> > >   files in an xml format rendered by draw.io (instructions below).
> > This
> > >   is the free tool of choice that we've agreed to use for
> exchanging
> > >   diagrams and their source files in Metron.">>
> > >   2. New line item < > >   "/images-source" and rendered diagrams and images
> > belong in
> > >   "/images."
> > >   3. New subsection <<"Creating and Modifying Diagrams">>. This
> > section
> > >   would provide basic instructions for downloading source files
> from
> > >   draw.io.
> > >3. Add a new checkbox item under PR checklist heading "For
> > documentation
> > >related changes" with the following text
> > >   1. Have you ensured that any documentation diagrams have been
> > >   updated, along with their source files, using draw.io? See
> > >
> >
> https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines
> > > for
> > >   instructions.
> > >4. Here is the Jira for migrating/redoing existing diagrams
> > >   1. https://issues.apache.org/jira/browse/METRON-2099
> > >
> > > We require a minimum of 72 hours for a vote, not typically including
> > > weekend days. I'd like to leave this vote open until Wednesday 5/8,
> 12PM
> > > EDT. Please vote +1, -1, or 0 to abstain, and also indicate if your
> vote
> > is
> > > binding or non-binding.
> >
> > ---
> > Thank you,
> >
> > James Sirota
> > PMC- Apache Metron
> > jsirota AT apache DOT org
> >
> >
>

Re: [DISCUSS] Full-dev role in PR testign

[Nick Allen]

I'm exploring the use of TestContainers right now as part of the HDP 3.1
effort.  Still exploring feasibility, but it is looking promising.

On Fri, May 3, 2019 at 10:46 AM Justin Leet  wrote:

> I think everything Casey mentioned is a good call-out as things start to
> build into specifics. I definitely agree it's a very nontrivial amount of
> work, but that lowering the barrier of entry to a lot of PRs eases the
> burden on both new and existing contributors by a substantial amount.
>
> @Mike,
> As a heads up, I (super briefly) looked into the Docker stuff a bit, and
> the extension idea may not work with the Docker stuff to the extent we want
> (at least without us doing some additional work ourselves).  It seems like
> at least what I linked earlier and some other stuff actually provide direct
> annotations rather than plugging directly into the same extensions idea.
>
> Before we dive into it too much, it might be worth playing around with it
> more and coming back to the group with a couple options. If you're
> interested in looking into it, I *suspect* it'll boil down to a couple
> options
> * Use Docker with something like the above link or testcontainers
> . It's possible the Docker stuff ends up
> being lightweight / fast enough to use on at least a per class basis like
> we do now, rather than trying to generalize across all tests immediately.
> * Roll our own code to have more fine-grained control over the Docker
> lifecycle and which components need to be spun up for extensions.
> * Figure out how to make the prior options play nice
> * Other
>
> I'll probably dig a bit on my own, but I'm not sure how much focused time
> I'll be able to put into it in the absolute immediate term.  I can probably
> whip up a quick demo of the extensions stuff over the next week or so in a
> one-off project to give a bit of a demo and maybe help out with some of
> experimentation. Feel free to reach out if there's anything in particular
> that would be helpful to look into.
>
> The extensions stuff does have a lot of benefits, but I had less components
> to work with and didn't have the same classpath worries that made real
> instances (i.e. Docker) more attractive. It was sufficient for our
> purposes, but there might have to be compromises here. We depend on a lot
> more of the Hadoop stack.
>
> Migrating to JUnit 5 in general *should* be pretty easy. I don't think we
> really use any of the stuff that migrated in odd ways, so it should mostly
> just be updating annotations and imports (@Before to @BeforeEach, etc.).
> I'm sure this glosses over at least a few gotchas, though.
>
> On Fri, May 3, 2019 at 10:09 AM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > I didn't get a chance to say so earlier, but Justin, I also like the
> JUnit
> > 5 extension suggestion. I've gone through some en-masse changes before,
> > e.g. standardizing the log4j construction idiom, and it honestly wasn't
> too
> > bad. Just a thought, it might make sense to kick this off by upgrading
> > overall JUnit 4 to 5 across the code base, and then diving into some of
> the
> > more 5-specific changes you're recommending as needed. I created this
> Jira
> > a bit ago - https://issues.apache.org/jira/browse/METRON-2037. That was
> to
> > upgrade to 4.13, but we might be able to kill 2 birds with one stone if
> we
> > go to JUnit 5. I'm volunteering to look into this and/or see the work
> > through to completion. What do you think?
> >
> > > - debuggability (right now we run the tests in the same JVM and setting
> >breakpoints is trivial, even in the innards of Hadoop.  This is very
> >valuable for figuring out what's going wrong and we'll need SOME
> > solution
> >for it)
> >
> > Yeah Casey, I remember this from the last time we discussed it. That's
> the
> > most import issue to be sure we have a handle on, imo. We'll need to
> figure
> > out remote debugging in Docker containers. Not to mention, the execution
> > path becomes a bit more spread out when we're running multiple components
> > as nature intended across multiple processes.
> >
> >
> >
> > On Fri, May 3, 2019 at 7:14 AM Casey Stella  wrote:
> >
> > > I just want to chime in and say I'm STRONGLY in favor of a docker-based
> > > approach to testing (I specifically like the JUnit 5 extensions
> > > suggestion).  I think that forcing a full-dev evaluation for every
> small
> > PR
> > > is a barrier to entry that I'd like to overcome.  I also think that
> this
> > is
> > > going to not be trivial.
> > >
> > > There will be weirdness/drama with:
> > >
> > >- cleanup
> > >- setup in situations where multi-components are used
> > >- debuggability (right now we run the tests in the same JVM and
> > setting
> > >breakpoints is trivial, even in the innards of Hadoop.  This is very
> > >valuable for figuring out what's going wrong and we'll need SOME
> > > solution
> > >for it)
> > >- possible resource 

Re: [VOTE] Update dev guidelines with format for sharing architecture source files and rendered images

[Michael Miklavcic]

Yes, it is free James. We made sure of that in the original discussion.

On Thu, May 2, 2019 at 9:33 PM James Sirota  wrote:

> i am ok with it as long as we are not forcing people to buy stuff
>
> 02.05.2019, 18:18, "Michael Miklavcic" :
> > Here's the latest discussion on the subject:
> >
> https://lists.apache.org/thread.html/0aa2b0b9ed4a0f0b0d8bb018c618e62de196565f9af71f347e504076@%3Cdev.metron.apache.org%3E
> >
> > I'd like to propose a vote to change our dev guidelines which will
> clarify
> > the tooling we use to produce diagrams and share the source files for
> those
> > diagrams. I propose the dev guidelines
> >
> https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines
> and
> > PR checklist
> >
> https://github.com/apache/metron/blob/master/.github/PULL_REQUEST_TEMPLATE.md#for-documentation-related-changes
> > be
> > changed in the following ways:
> >
> >1. Under "1.1 Contributing A Code Change"
> >   1. Change <<"New features and significant bug fixes should be
> >   documented in the JIRA and appropriate architecture diagrams
> should be
> >   attached. Major features may require a vote.">> to <<"New features
> >   and significant bug fixes should be documented in the JIRA.
> Appropriate
> >   architecture diagrams should be created in https://www.draw.io/
> > and committed
> >   to source control as per section 2.4. Diagrams may be requested of
> PR
> >   submitters during review either as documentation or as an aid to
> the
> >   reviewer. Major features may also require a vote.">>
> >2. Under "2.4 Documentation"
> >   1. New line item <<"Diagrams - We save architecture diagram source
> >   files in an xml format rendered by draw.io (instructions below).
> This
> >   is the free tool of choice that we've agreed to use for exchanging
> >   diagrams and their source files in Metron.">>
> >   2. New line item < >   "/images-source" and rendered diagrams and images
> belong in
> >   "/images."
> >   3. New subsection <<"Creating and Modifying Diagrams">>. This
> section
> >   would provide basic instructions for downloading source files from
> >   draw.io.
> >3. Add a new checkbox item under PR checklist heading "For
> documentation
> >related changes" with the following text
> >   1. Have you ensured that any documentation diagrams have been
> >   updated, along with their source files, using draw.io? See
> >
> https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines
> > for
> >   instructions.
> >4. Here is the Jira for migrating/redoing existing diagrams
> >   1. https://issues.apache.org/jira/browse/METRON-2099
> >
> > We require a minimum of 72 hours for a vote, not typically including
> > weekend days. I'd like to leave this vote open until Wednesday 5/8, 12PM
> > EDT. Please vote +1, -1, or 0 to abstain, and also indicate if your vote
> is
> > binding or non-binding.
>
> ---
> Thank you,
>
> James Sirota
> PMC- Apache Metron
> jsirota AT apache DOT org
>
>

Re: [DISCUSS] Full-dev role in PR testign

[Michael Miklavcic]

I didn't get a chance to say so earlier, but Justin, I also like the JUnit
5 extension suggestion. I've gone through some en-masse changes before,
e.g. standardizing the log4j construction idiom, and it honestly wasn't too
bad. Just a thought, it might make sense to kick this off by upgrading
overall JUnit 4 to 5 across the code base, and then diving into some of the
more 5-specific changes you're recommending as needed. I created this Jira
a bit ago - https://issues.apache.org/jira/browse/METRON-2037. That was to
upgrade to 4.13, but we might be able to kill 2 birds with one stone if we
go to JUnit 5. I'm volunteering to look into this and/or see the work
through to completion. What do you think?

> - debuggability (right now we run the tests in the same JVM and setting
   breakpoints is trivial, even in the innards of Hadoop.  This is very
   valuable for figuring out what's going wrong and we'll need SOME solution
   for it)

Yeah Casey, I remember this from the last time we discussed it. That's the
most import issue to be sure we have a handle on, imo. We'll need to figure
out remote debugging in Docker containers. Not to mention, the execution
path becomes a bit more spread out when we're running multiple components
as nature intended across multiple processes.



On Fri, May 3, 2019 at 7:14 AM Casey Stella  wrote:

> I just want to chime in and say I'm STRONGLY in favor of a docker-based
> approach to testing (I specifically like the JUnit 5 extensions
> suggestion).  I think that forcing a full-dev evaluation for every small PR
> is a barrier to entry that I'd like to overcome.  I also think that this is
> going to not be trivial.
>
> There will be weirdness/drama with:
>
>- cleanup
>- setup in situations where multi-components are used
>- debuggability (right now we run the tests in the same JVM and setting
>breakpoints is trivial, even in the innards of Hadoop.  This is very
>valuable for figuring out what's going wrong and we'll need SOME
> solution
>for it)
>- possible resource limitations in travis for running tests with
>multiple components
>
> Even so, with ALL of that being said, I still think the value outweighs the
> difficulty by a factor of 10.  Being able to be confident after a travis
> run that people aren't introducing subtle classpath or cross-component
> interaction issues would open up 80% of the class of PRs that don't require
> full-dev review.  That being said, I still don't think it's 100%.
> Specifically, PRs which can credibly be argued that they touch installation
> pathways would still need to be verified in full-dev as it's the only path
> to validating that (otherwise we would be regressing in test coverage).
>
> On Wed, May 1, 2019 at 9:33 PM Justin Leet  wrote:
>
> > >
> > > My impression is that this is already the status quo. But, if we think
> we
> > > need to be more clear on this, let's put up a vote to change the coding
> > > guidelines and PR checklist. I've done this many times in the past, the
> > > most obvious instances are when I've made doc changes or unit test
> > > modifications because those will not impact full dev. I will own this
> > item.
> > > I think it can probably get rolled in with my dev guideline changes for
> > > architecture diagrams.
> >
> >
> > For completeness in our PR checklist: "- [ ] Have you verified the basic
> > functionality of the build by building and running locally with Vagrant
> > full-dev environment or the equivalent?"  In practice, you're right, but
> > any newer contributors aren't necessarily going to know this.
> >
> > 1. I think we should create Jiras with the end deliverable being that our
> > > private vs public API endpoints are clearly delineated. From there, we
> > > create another round of javadoc - for the public APIs let the javadocs
> > rain
> > > from the heavens to your heart's content. It's for public consumption
> and
> > > should assist end users. See Mockito, for example -
> > >
> > >
> >
> https://static.javadoc.io/org.mockito/mockito-core/2.27.0/org/mockito/Mockito.html
> > > .
> > > For developer docs, I'm of the *extremely strong* opinion that this
> > should
> > > be limited. Emphasize module, package, class, and method naming
> > conventions
> > > over all else. If it doesn't make sense just reading the code, take a
> > > minute to summarize what you're doing and consider refactoring. For
> > > legitimately more complex and necessary code passages, add a note. For
> > > multi-class interactions that provide a larger story arc, add dev docs
> to
> > > the relevant READMEs. Our use of Zookeeper Curator and its interaction
> > with
> > > our topology config loading is a perfect example of a feature that
> would
> > > fit this need.
> > >
> > 2. I'm an immediate -1 on any documentation that looks like " /** Open
> the
> > > car door **/ public void openCarDoor() {...}" :-). The code speaks for
> > > itself.
> > > 3. Publish javadocs for public APIs, not our internal 

Re: [DISCUSS] Full-dev role in PR testign

[Casey Stella]

I just want to chime in and say I'm STRONGLY in favor of a docker-based
approach to testing (I specifically like the JUnit 5 extensions
suggestion).  I think that forcing a full-dev evaluation for every small PR
is a barrier to entry that I'd like to overcome.  I also think that this is
going to not be trivial.

There will be weirdness/drama with:

   - cleanup
   - setup in situations where multi-components are used
   - debuggability (right now we run the tests in the same JVM and setting
   breakpoints is trivial, even in the innards of Hadoop.  This is very
   valuable for figuring out what's going wrong and we'll need SOME solution
   for it)
   - possible resource limitations in travis for running tests with
   multiple components

Even so, with ALL of that being said, I still think the value outweighs the
difficulty by a factor of 10.  Being able to be confident after a travis
run that people aren't introducing subtle classpath or cross-component
interaction issues would open up 80% of the class of PRs that don't require
full-dev review.  That being said, I still don't think it's 100%.
Specifically, PRs which can credibly be argued that they touch installation
pathways would still need to be verified in full-dev as it's the only path
to validating that (otherwise we would be regressing in test coverage).

On Wed, May 1, 2019 at 9:33 PM Justin Leet  wrote:

> >
> > My impression is that this is already the status quo. But, if we think we
> > need to be more clear on this, let's put up a vote to change the coding
> > guidelines and PR checklist. I've done this many times in the past, the
> > most obvious instances are when I've made doc changes or unit test
> > modifications because those will not impact full dev. I will own this
> item.
> > I think it can probably get rolled in with my dev guideline changes for
> > architecture diagrams.
>
>
> For completeness in our PR checklist: "- [ ] Have you verified the basic
> functionality of the build by building and running locally with Vagrant
> full-dev environment or the equivalent?"  In practice, you're right, but
> any newer contributors aren't necessarily going to know this.
>
> 1. I think we should create Jiras with the end deliverable being that our
> > private vs public API endpoints are clearly delineated. From there, we
> > create another round of javadoc - for the public APIs let the javadocs
> rain
> > from the heavens to your heart's content. It's for public consumption and
> > should assist end users. See Mockito, for example -
> >
> >
> https://static.javadoc.io/org.mockito/mockito-core/2.27.0/org/mockito/Mockito.html
> > .
> > For developer docs, I'm of the *extremely strong* opinion that this
> should
> > be limited. Emphasize module, package, class, and method naming
> conventions
> > over all else. If it doesn't make sense just reading the code, take a
> > minute to summarize what you're doing and consider refactoring. For
> > legitimately more complex and necessary code passages, add a note. For
> > multi-class interactions that provide a larger story arc, add dev docs to
> > the relevant READMEs. Our use of Zookeeper Curator and its interaction
> with
> > our topology config loading is a perfect example of a feature that would
> > fit this need.
> >
> 2. I'm an immediate -1 on any documentation that looks like " /** Open the
> > car door **/ public void openCarDoor() {...}" :-). The code speaks for
> > itself.
> > 3. Publish javadocs for public APIs, not our internal dev APIs. Let your
> > fav IDE fill in the gaps for devs.
>
>
> I'm +1 on delineating public vs private APIs like you've outlined there.  I
> think our dev stuff is *better* than our general usage guides, but there's
> room for improvement. I'm fairly agnostic on the dev docs because to be
> honest, a ton of our older code is not at all self explanatory, and to be
> blunt refactoring a lot of it is a substantial lift (as we've all seen
> multiple times trying to refactor it).  If this were greenfield, I'd be in
> much stronger agreement with you, but I suspect in practice there's a lot
> of stuff nobody's going to refactor for awhile.
>
>
> > Full dev until we vote to replace the existing setup and can be confident
> > that the new approach 1. is stable, 2. takes <= the amount of time to
> > complete as full dev. I am +1 for migrating towards this approach and
> think
> > we should do so when it's dialed in.
>
>
> Great, I look forward to that getting in.
>
> Justin, what are your thoughts on leveraging this approach along with
> > long-lived Docker containers?
> >
>
> Apparently, there's actually an extension for running Docker containers,
> see  https://faustxvi.github.io/junit5-docker/.  My main hesitation there
> is more around how much effort to migrate it is. I think that's almost
> certainly a cleaner long term solution, but I suspect the 80% solution of
> migrating what we have *might* be easier.  There might also be ways of just
> leveraging this by moving stuff 

Re: [DISCUSS] Metron Release - 0.7.1 next steps

[Otto Fowler]

Despite the name, we *have* been using it as both for quite some amount of
time.  It *is* both dev and demo, and we recommend it as such on the list
all the time.

So there isn’t a decision to be made here as far as the status quo -> we
use full dev as both dev and demo.




On May 2, 2019 at 18:53:37, Michael Miklavcic (michael.miklav...@gmail.com)
wrote:

Whether or not full dev is, first and foremost, "dev" I think your
questions being up a good point. If not full_dev for introducing new users,
then what? If we want to provide a different env for letting people tinker
and try it out than we do for development, that's completely fine. But we
don't have that right now. So we can treat full_dev as multipurpose, or we
can stop directing non-devs to it, or we can add something new. I honestly
don't have any recommendations here. We've talked about docker instances
for replacing in-memory components, but I'm still not sure that solves this
problem, or adds more complexity. Given the current options on the table,
I'm inclined to go with "full_dev" serves both dev and demo purposes. Otto,
what do you think?

On Thu, May 2, 2019, 4:32 PM Otto Fowler  wrote:

> I’ve commented on the PR, and I won’t repeat it here as well, I will
> however ask again if we know and can list all of the usability issues
that
> surround this problem. IE. All the things that can happen or may happen
> for people who are not Metron developers and committers who are using
> full dev, because we keep recommending it.
>
>
>
> On May 2, 2019 at 17:38:30, Michael Miklavcic (michael.miklav...@gmail.com
> )
> wrote:
>
> PR is up. I added the doc change to the metron-deployment README since
this
> serves as the gateway doc for all the VM instances. All of which would be
> affected by the feature gap.
>
> https://github.com/apache/metron/pull/1398
>
> On Thu, May 2, 2019 at 1:37 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > Here's the ticket I created to track it, which also references the Jira
> > for the new UI feature.
> > https://issues.apache.org/jira/browse/METRON-2100
> >
> > On Thu, May 2, 2019 at 12:34 PM Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> >> :-)
> >>
> >> I expect to have #2 out sometime today.
> >>
> >> On Thu, May 2, 2019, 12:11 PM Justin Leet 
> wrote:
> >>
> >>> >
> >>> > I personally
> >>> > don't like this feature gap in full dev. It seems Otto agrees, and
> >>> Casey at
> >>> > the very least sees it as enough of an issue to gate us from 0.8.
> >>> >
> >>>
> >>> +1 on all of this. I don't like it either.
> >>>
> >>>
> >>> > Our vote landed 2-2. We are having a discussion about what to do
with
> >>> the
> >>> > release. This is that discussion.
> >>>
> >>>
> >>> I'm going to be honest, my response was a combination of misreading
> what
> >>> you said and thinking you were proposing delaying the release more
> >>> seriously and feeling a bit blindsided by a perceived move from the
> >>> initial
> >>> "take more time than originally anticipated" (which in my head I took
> as
> >>> a
> >>> couple days) to "versus next week, or the week after" (where delaying
> >>> things weeks is something I personally would like not buried so far
> down
> >>> in
> >>> the thread). Totally my bad, sorry about that.
> >>>
> >>> Other than that, it sounds like we're pretty much in agreement.
> >>>
> >>> Here's my current understanding of the state and consensus as of
right
> >>> now
> >>> (which is subject to change as more discussion happens):
> >>>
> >>> - Most of the people in the thread are in favor of #2 for 0.7.1 and
#3
> >>> for 0.8.0.
> >>> - I don't believe I've seen an explicit response from Otto on what
> >>> he
> >>> thinks about doing this, and from a personal perspective like to
> >>> see what
> >>> his opinion is as the person who originally brought it up.
> >>> - Mike said he's going to kick out a PR that addresses #2
> >>> - After that undergoes the normal review process and is merged, we
> >>> proceed normally and cut RC2.
> >>>
> >>>
> >>> On Thu, May 2, 2019 at 1:14 PM Michael Miklavcic <
> >>> michael.miklav...@gmail.com> wrote:
> >>>
> >>> > I think your later point about using a project release version,
from
> >>> the
> >>> > example of using other projects, is a valid one. To exactly that
> >>> point, a
> >>> > community member (Otto) brought up an issue/bug they found through
> >>> testing
> >>> > that they were previously unaware of and did not find documented.
> >>> Which was
> >>> > argued would be confusing to someone wanting to use a published
> >>> release. We
> >>> > discussed the implications of this bug/feature gap. And
incidentally,
> >>> it
> >>> > sounds like some people see full dev as more useful than just a dev
> >>> box,
> >>> > others do not, independent of what we chose to name it. That came
> from
> >>> our
> >>> > discussion about it.
> >>> >
> >>> > The expectation I had from my discussion with the contributors was
> that
> >>> > this fix for aggregation 

Re: [VOTE] Update dev guidelines with format for sharing architecture source files and rendered images

[Otto Fowler]

+1


On May 2, 2019 at 21:18:21, Michael Miklavcic (michael.miklav...@gmail.com)
wrote:

Here's the latest discussion on the subject:
https://lists.apache.org/thread.html/0aa2b0b9ed4a0f0b0d8bb018c618e62de196565f9af71f347e504076@%3Cdev.metron.apache.org%3E

I'd like to propose a vote to change our dev guidelines which will clarify
the tooling we use to produce diagrams and share the source files for those
diagrams. I propose the dev guidelines
https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines
and
PR checklist
https://github.com/apache/metron/blob/master/.github/PULL_REQUEST_TEMPLATE.md#for-documentation-related-changes
be
changed in the following ways:

1. Under "1.1 Contributing A Code Change"
1. Change <<"New features and significant bug fixes should be
documented in the JIRA and appropriate architecture diagrams should be
attached. Major features may require a vote.">> to <<"New features
and significant bug fixes should be documented in the JIRA. Appropriate
architecture diagrams should be created in https://www.draw.io/
and committed
to source control as per section 2.4. Diagrams may be requested of PR
submitters during review either as documentation or as an aid to the
reviewer. Major features may also require a vote.">>
2. Under "2.4 Documentation"
1. New line item <<"Diagrams - We save architecture diagram source
files in an xml format rendered by draw.io (instructions below). This
is the free tool of choice that we've agreed to use for exchanging
diagrams and their source files in Metron.">>
2. New line item >. This section
would provide basic instructions for downloading source files from
draw.io.
3. Add a new checkbox item under PR checklist heading "For documentation
related changes" with the following text
1. Have you ensured that any documentation diagrams have been
updated, along with their source files, using draw.io? See
https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines
for
instructions.
4. Here is the Jira for migrating/redoing existing diagrams
1. https://issues.apache.org/jira/browse/METRON-2099


We require a minimum of 72 hours for a vote, not typically including
weekend days. I'd like to leave this vote open until Wednesday 5/8, 12PM
EDT. Please vote +1, -1, or 0 to abstain, and also indicate if your vote is
binding or non-binding.

Re: [DISCUSS] Parser Aggregation in Management UI

[Shane Ardell]

NgRx was only used for the aggregation feature and doesn't go beyond that.
I think the way I worded that sentence may have caused confusion. I just
meant we use it to manage more pieces of state within the aggregation
feature than just previous and current state of grouped parsers.

On Fri, May 3, 2019 at 1:32 AM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Shane, thanks for putting this together. The updates on the Jira are useful
> as well.
>
> > (we used it for more than just that in this feature, but that was the
> initial reasoning)
> What are you using NgRx for in the submitted work that goes beyond the
> aggregation feature?
>
>
>
> On Thu, May 2, 2019 at 12:22 PM Shane Ardell 
> wrote:
>
> > Hello everyone,
> >
> > In response to discussions in the 0.7.1 release thread, I wanted to
> start a
> > thread regarding the parser aggregation work for the Management UI. For
> > anyone who has not already read and tested the PR locally, I've added a
> > detailed description of what we did and why to the JIRA ticket here:
> > https://issues.apache.org/jira/browse/METRON-1856
> >
> > I'm wondering what the community thinks about what we've built thus far.
> Do
> > you see anything missing that must be part of this new feature in the UI?
> > Are there any strong objections to how we implemented it?
> >
> > I’m also looking to see if anyone has any thoughts on how we can possibly
> > simplify this PR. Right now it's pretty big, and there are a lot of
> commits
> > to parse through, but I'm not sure how we could break this work out into
> > separate, smaller PRs opened against master. We could try to cherry-pick
> > the commits into smaller PRs and then merge them into a feature branch,
> but
> > I'm not sure if that's worth the effort since that will only reduce the
> > number commits to review, not the lines changed.
> >
> > As an aside, I also want to give a little background into the
> introduction
> > of NgRx in this PR. To give a little background on why we chose to do
> this,
> > you can refer to the discussion thread here:
> >
> >
> https://lists.apache.org/thread.html/06a59ea42e8d9a9dea5f90aab4011e44434555f8b7f3cf21297c7c87@%3Cdev.metron.apache.org%3E
> >
> > We previously discussed introducing a better way to manage application
> > state in both UIs in that thread. It was decided that NgRx was a great
> tool
> > for many reasons, one of them being that we can piecemeal it into the
> > application rather than doing a huge rewrite of all the application state
> > at once. The contributors in this PR (myself included) decided this would
> > be a perfect opportunity to introduce NgRx into the Management UI since
> we
> > need to manage the previous and current state with the grouping feature
> so
> > that users can undo the changes they've made (we used it for more than
> just
> > that in this feature, but that was the initial reasoning). In addition,
> we
> > greatly benefited from this when it came time to debug our work in the UI
> > (the discussion in the above thread link goes a little more into the
> > advantages of debugging with NgRx and DevTools). Removing NgRx from this
> > work would reduce the numbers of lines changed slightly, but it would
> still
> > be a big PR and a lot of that code would just move to the component or
> > service level in the Angular application.
> >
> > Shane
> >
>