Re: [DISCUSS] next release proposal

[Justin Leet]

METRON-799 is in master.  I know at least METRON-859 now has a conflict on
Kerberos-setup.md, so if anyone has anything similar, please make sure to
update your branch.

On Thu, Apr 20, 2017 at 9:25 AM, Otto Fowler 
wrote:

> Maybe in the next release we should have theme labels for the release in
> jira
>
>
> On April 20, 2017 at 09:24:14, Casey Stella (ceste...@gmail.com) wrote:
>
> +1, I agree. We should see this through. I will point out also that while
> not strictly required, METRON-861 (
> https://github.com/apache/incubator-metron/pull/534) would be required to
> use the CLI utilities if people are planning on setting acl's on the znodes
> for our config in their installation in a kerberized environment. I
> mention it only because it doesn't appear to be kerberos related, but it
> was inspired by a kerberos-related problem.
>
> Casey
>
> On Thu, Apr 20, 2017 at 9:17 AM, Otto Fowler 
> wrote:
>
> > +1 on the proposed changes.
> >
> > I think that it is important that if it is in reach, we ‘complete’ any
> > major theme of the release.
> > In this case kerberos. I think that given where pycapa and rest are, we
> > should make the effort
> > to get them in.
> >
> >
> >
> > On April 20, 2017 at 08:52:11, Ryan Merriman (merrim...@gmail.com)
> wrote:
> >
> > Matt,
> >
> > I started a discussion around Kerberos support as a prerequisite for
> MPack
> > work and the consensus was that a service should support Kerberos before
> > it's included in the MPack. There is a PR out there for Kerberos support
> > for REST (https://github.com/apache/incubator-metron/pull/535) but it
> has
> > not been reviewed yet. There is also likely a small amount of work to be
> > done once METRON-799 is accepted. Because of these 2 dependencies I think
> > it's probably best to wait on METRON-795.
> >
> > Ryan
> >
> > On Thu, Apr 20, 2017 at 1:13 AM, Matt Foley  wrote:
> >
> > > Hi all,
> > > I’ve put together RC1 for the 0.4.0 release of Metron, along with its
> > > book-site. It is available for your review at
> > > https://dist.apache.org/repos/dist/dev/incubator/metron/0.4.
> > > 0-RC1-incubating/
> > >
> > > I’m not putting it to VOTE yet, because I think some additional fixes
> are
> > > probably necessary:
> > > • We should add documentation for the remaining backward-incompatible
> > > changes
> > > • We should add these important bug fixes that have been committed to
> > > master since the 0.4.0 branch was cut:
> > > o METRON-634 fixes for Mpack for Centos7
> > > o METRON-856 Ansible rpm build wipes out prior binary build
> > > o METRON-821 Minor fixes in full dev kerberos setup instructions
> > > o Please give me your +1 for these additions
> > > • These PRs are currently open, and seem important to complete the
> > > Kerberos picture:
> > > o METRON-799: The MPack should function in a kerberized cluster
> > > o METRON-835 Use Profiler with Kerberos
> > > o METRON-836 Use Pycapa with Kerberos
> > > o METRON-859 Use REST application with Kerberos
> > > o Please give me your evaluation of whether these can be committed
> > > Real Soon Now, or we should not wait for them.
> > > • This PR is open and seems important to complete the REST picture:
> > > o METRON-795: Install Metron REST with Ambari MPack
> > > o Please give me your evaluation of whether this can be committed Real
> > > Soon Now, or we should not wait for it.
> > > • Anything else? I’ve deliberately left out the commits on master that
> > > represent new functionality not already in (or mostly in) the 0.4.0
> > branch.
> > >
> > > Thanks,
> > > --Matt
> > >
> > >
> > > On 4/18/17, 5:30 PM, "Matt Foley"  wrote:
> > >
> > > Thanks, we’re now up to 4 backward-incompatible issues. Any others
> > > should be so marked?
> > >
> > > On 4/17/17, 4:43 PM, "Matt Foley"  wrote:
> > >
> > > Hi all,
> > > Out of the 58 Jiras resolved, completely or partially, between
> > > 0.3.1 and 0.4.0, only one is labeled “backward-incompatible” and has
> text
> > > in the “Docs Text” field. And it’s super minor (METRON-771).
> > >
> > > Is this really true? If so, great, but if not, please help people
> > > upgrade without glitches: Fix these fields in your jiras, so they can
> be
> > > included in the Release Notes.
> > > a) In the “Labels” field, add “backward-incompatible”. (It will
> > > autocomplete for you.)
> > > b) In the “Doc Text” field, say what the issue is and what a
> > > person upgrading should do about it, if anything.
> > >
> > > As usual, non-response will be considered positive confirmation
> > > that no response is necessary :-)
> > > Please try to address in the next day or so.
> > >
> > > Thanks,
> > > Your humble Release Manager
> > >
> > >
> > > On 4/12/17, 10:59 AM, "zeo...@gmail.com"  wrote:
> > >
> > > I agree conceptually but haven't looked at them each
> > > individually to see
> > > how much they impact and if a short timeline for 

Re: Reducing Warnings in Build

[Justin Leet]

First off, I would absolutely love to see the warnings reduced and the
quality of our code improved.  I'm in favor of all four of the points, and
I think it's a good start towards weeding out a lot of issues.

So regarding question 1:

That awkwardness comes about because org.simple.json is all untyped Maps
under the hood (
https://github.com/fangyidong/json-simple/blob/master/src/main/java/org/json/simple/JSONObject.java#L19).
So anything that touches those classes starts requiring casts everywhere. A
couple months ago a PR was made against the library that seems to have
added a lot of this stuff (
https://github.com/fangyidong/json-simple/pull/113).  Basically, it was
forked (https://github.com/cliftonlabs/json-simple and
https://cliftonlabs.github.io/json-simple/) and they tried to contribute it
back and it's been sitting there.

I strongly think we should consider swapping over to the fork.  We'd have
to do a bit of research, but it seems the intent was to be completely
backwards compatible while updating things like generics and some various
issues they presumably saw with the original.

For 2, and I'm not an expert in this by any means, but I would be in favor
of defaulting to UTF-8.  It's most likely what's happening under the hood
anyway (even though it's technically platform dependent), so defaulting
probably makes it both more explicit and removes potential for issues if a
given system has a different underlying default.

Justin

On Fri, Apr 21, 2017 at 8:50 AM, JJ Meyer  wrote:

> Hello everybody,
>
> Currently our build has a significant amount of warnings (most from the
> error prone plugin). A lot of this has been documented here:
> https://issues.apache.org/jira/browse/METRON-617
>
> I want to continue to work on this Jira. I really want to reduce the number
> of warnings in our build. As the Jira points out, a lot of them are
> unchecked casts or the implicit use of default charset.
>
> When starting this, I want to start with a specific module. I plan on
> starting with `metron-interface` as it's fairly self contained and is
> pretty new. Below I want to layout what I plan on doing. Please let me know
> what you all think:
>
> 1. Suppress warnings where generics are not supported or checking type is
> not possible.
> 2. For all unit tests that require supplying a charset I'll supply the
> UTF-8 charset.
> 3. Update the API to have a charset configuration for each resource that
> needs one.
> 4. Remove @SuppressWarnings("ALL") on all unit tests. I found out error
> prone doesn't support this level of suppression. Which is probably a good
> thing. We will need to explicitly state what we want to suppress instead.
>
> Two big questions came up right away when I was thinking about the above:
>
> 1. When suppressing warnings. I see we sometimes cast a JSONObject to
> Map. I know it extends Map, but is it really safe to do
> something like the following? Should we have a utility that truly builds a
> map that uses generics? I plan on doing some testing around this, but if
> anyone has any experience with this it would be greatly appreciated. Here
> is an example of what I am talking about:
>
> JSONObject message = ...;
> @SuppressWarnings("unchecked")
> Map state = (Map) message;
>
>
> 2. This one is about configuring charset (#3 above). Specifically in
> metron-rest. Right now, I believe there are 3 sensor resources (index,
> enrichment, and parser) that throw warnings due to implicitly using the
> default charset. I propose that we have 3 configs (listed below). These
> configs would take any valid charset, default, or not set. If not set then
> UTF-8 would be default. Does this seem fair?
>
> sensor:
>   index.encoding: UTF-8
>   enrichment.encoding: UTF-8
>   parser.encoding: UTF-8
>
>
> Does anyone see any problems that may occur if we go about it this way? Any
> help on this would be very much appreciated.
>
> Thanks,
> JJ
>

Re: Having difficulty setting up both QuickDev and FullDev

[Justin Leet]

If it didn't resolve the issue, then yeah, we might as well just undo it,
and I can update the PR with that instead.

Justin

On Thu, Apr 20, 2017 at 2:11 PM, David Lyle <dlyle65...@gmail.com> wrote:

> Being as the dependency additon didn't actually resolve the out-of-order
> build issue, would it make sense to undo it?
>
> -D...
>
>
> On Thu, Apr 20, 2017 at 1:26 PM, Justin Leet <justinjl...@gmail.com>
> wrote:
>
> > Our instructions on building the RPMs are slightly out of date.  The pom
> > for metron-deployment changed slightly to have a dependency on a couple
> > other repos. This means during the project build before the RPM build it
> > needs to be mvn clean install, instead of mvn clean package.
> >
> > Basically from the root pom, do a mvn clean install -DskipTests
> >
> > I'd create a ticket and PR on the doc, but it looks like Apache Jira just
> > went down.
> >
> > Justin
> >
> > On Thu, Apr 20, 2017 at 12:53 PM, Anand Subramanian <
> > asubraman...@hortonworks.com> wrote:
> >
> > > I am seeing the same issue that Rita mentioned with full-dev on latest
> > > master. I have docker up and running, but the RPMs are still not built.
> > >
> > > Here is the output from mvn build-rpms:
> > > --
> > > ➜  metron-deployment git:(master) mvn clean package -Pbuild-rpms
> > > -DskipTests
> > > [INFO] Scanning for projects...
> > > [INFO] 
> > > 
> > > [INFO] Reactor Build Order:
> > > [INFO]
> > > [INFO] metron-deployment
> > > [INFO] metron-rpm
> > > [INFO]
> > > [INFO] 
> > > 
> > > [INFO] Building metron-deployment 0.4.0
> > > [INFO] 
> > > 
> > > [INFO]
> > > [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @
> > > metron-deployment ---
> > > [INFO]
> > > [INFO] 
> > > 
> > > [INFO] Building metron-rpm 0.4.0
> > > [INFO] 
> > > 
> > > Downloading: http://repo.hortonworks.com/
> content/repositories/releases/
> > > org/apache/metron/metron-platform/0.4.0/metron-platform-0.4.0.pom
> > > [WARNING] The POM for org.apache.metron:metron-platform:pom:0.4.0 is
> > > missing, no dependency information available
> > > Downloading: http://repo.hortonworks.com/
> content/repositories/releases/
> > > org/apache/metron/metron-analytics/0.4.0/metron-analytics-0.4.0.pom
> > > [WARNING] The POM for org.apache.metron:metron-analytics:pom:0.4.0 is
> > > missing, no dependency information available
> > > [INFO] 
> > > 
> > > [INFO] Reactor Summary:
> > > [INFO]
> > > [INFO] metron-deployment .. SUCCESS [
> > > 0.113 s]
> > > [INFO] metron-rpm . FAILURE [
> > > 2.353 s]
> > > [INFO] 
> > > 
> > > [INFO] BUILD FAILURE
> > > [INFO] 
> > > 
> > > [INFO] Total time: 2.569 s
> > > [INFO] Finished at: 2017-04-20T22:17:37+05:30
> > > [INFO] Final Memory: 12M/309M
> > > [INFO] 
> > > 
> > > [ERROR] Failed to execute goal on project metron-rpm: Could not resolve
> > > dependencies for project org.apache.metron:metron-rpm:pom:0.4.0: The
> > > following artifacts could not be resolved: org.apache.metron:metron-
> > platform:pom:0.4.0,
> > > org.apache.metron:metron-analytics:pom:0.4.0: Failure to find
> > > org.apache.metron:metron-platform:pom:0.4.0 in http://clojars.org/repo
> > > was cached in the local repository, resolution will not be reattempted
> > > until the update interval of clojars.org has elapsed or updates are
> > > forced -> [Help 1]
> > > [ERROR]
> > > [ERROR] To see the full stack trace of the errors, re-run Maven with
> the
> > > -e switch.
> > > [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> > > [ERROR]
> > > [ER

Re: [DISCUSS] Modify bylaws to allow speculative branches

[Justin Leet]

I'd prefer to still keep "Review then Commit" but I'd be interested in
lowering the review bar for it as long as the branch gets a solid final
review before merge.  I don't think a feature branch needs to be
micromanaged, but people should still take a quick glance at the commits
that come through. My stance on this may change if we find feature commits
stagnating for periods of time.

Basically, what I'd be looking for out of a feature branch is something
that's:
* Implementing a feature too large or with too much experimentation for a
single PR.
* Is at least looked at semi-regularly to have the opportunity for the
community to point out issues or scalability concerns without minor
concerns being blockers to progress. Otherwise, it's no different from a
giant PR that doesn't get examined until the PR is made.
* Reviews should only very rarely be blockers to progress continuing along
the branch. The branch shouldn't grind to a halt periodically or even
significantly slow progress to cleanup syntax or similar things.  The
tradeoff for this is I'd expect refactoring in a lot of cases before it
hits master to catch up on any of these things that we do care about, but
that aren't as important while the branch gets built up initially.
* Kept reasonably up to date with master. I really don't want a feature
branch sitting out for three months without being synced and then suddenly
everything explodes when we try to bring it back. It just causes all sorts
of review and testing issues.
* Ideally managed so that squashing multiple people's commits appropriately
is not a horrible nightmare.
* (Hopefully) able to be merged into master more smoothly as it's gotten
the major concerns cleaned out along the way.

Ideally, we also use this sort of framework to avoid some of the large PRs
we've seen around different features that felt very hard to get a grasp of
and review.

I'm pretty ambivalent on branch committers, and I'd be curious to get some
opinions (or possibly experiences if anyone has some) before I form a solid
opinion.


On Mon, Apr 24, 2017 at 12:01 PM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> "Branch committers ... do not cast binding votes or vetoes in the project."
> It sounds similar to what we do for other PRs except in this case it's 1..n
> contributors instead of just one. The step for getting the feature branch
> merged into trunk sounds reasonable and clear enough, but as Casey
> mentioned I'm also curious what the commit/review lifecycle looks like
> while the feature branch is active.
>
> On Fri, Apr 21, 2017 at 4:00 PM, Casey Stella  wrote:
>
> > So, what would that look like from a practical perspective?
> > * I presume commits would still associate to a JIRA, right?
> > * Are you proposing changing the strategy from Review then Commit to
> Commit
> > then Review for these branches?
> >
> > I know that we have some people who are active in the Hadoop project on
> the
> > PMC as mentors or are in more established projects with this kind of
> thing,
> > can you guys chime in about how that looks like and make some suggestions
> > as to some gotchas?
> >
> > Casey
> >
> > On Fri, Apr 21, 2017 at 5:19 PM, James Sirota 
> wrote:
> >
> > > Looking at Hadoop's bylaws
> > >
> > > https://hadoop.apache.org/bylaws.html
> > >
> > > They have this:
> > >
> > > Significant, pervasive features are often developed in a speculative
> > > branch of the repository. The PMC may grant commit rights on the branch
> > to
> > > its consistent contributors, while the initiative is active. Branch
> > > committers are responsible for shepherding their feature into an active
> > > release and do not cast binding votes or vetoes in the project.
> > >
> > > I would like to have something similar in our bylaws where we can have
> > > feature branches with their own set of committers.  We may have
> > > prototype-grade features that need to be rapidly evolved with the
> > feedback
> > > from the community.  So we would need a lighter process to evolve these
> > > features rapidly, while still keeping the community involved.  A more
> > > rigorous review process can then be applied when the feature is more
> > mature
> > > and is ready to be committed into master.
> > >
> > > What do you think?
> > >
> > > ---
> > > Thank you,
> > >
> > > James Sirota
> > > PPMC- Apache Metron (Incubating)
> > > jsirota AT apache DOT org
> > >
> >
>

Re: Code style change requires vote?

[Justin Leet]

That PR apparently got merged; I should have noted on the PR that this
thread got started.

Now there's a new potentially a new question: If that change required a
vote, what's the way we want to handle it?  Do we revert it or leave it and
just follow-on with a PR for any changes we choose to make during
discussion / vote?

Obviously, this is moot if the answer is we don't need a vote since that PR
did have the appropriate +1's.  At that point, I'd send an email calling it
out and make the appropriate wiki changes for getting people setup.

On Mon, Jul 31, 2017 at 1:50 PM, Justin Leet <justinjl...@gmail.com> wrote:

> Hi all,
>
> There was a bit of confusion on code style on at least one ticket,
> indirectly related to https://github.com/apache/metron/pull/577.  For a
> refresher (and also because GitHub is being iffy today), that PR is about
> moving to the Google Code Style as discussed in [DISCUSS] Code Style
> <https://lists.apache.org/thread.html/bd8afff9ad4fe83feb5e7fdfd5d136947665fd6e4e9856807d06e9a3@%3Cdev.metron.apache.org%3E>
>
> The feedback in that thread was positive, but no formal vote was done.
> I've found at least one vote thread for dev guidelines in the past, but
> it's also not part of the bylaws page itself.  Does this require just
> discussion/community consensus, or does it require (or do we just want) a
> vote?  And is that in line with what other projects tend to do (if we
> choose to fo
>
> If we're of the opinion that it requires a vote, I'll happy to kick up a
> discuss thread with the proposed change, get feedback, and run a vote.  It
> just delays when the change hits.
>

Re: MaaS and Metron Architecture talks at DataWorks Summit SJ 2017

[Justin Leet]

Could we put these up on the wiki page for tech talks in the community?
That page could probably use some love, although I know we've had
discussions about what we should do with wiki content.

https://cwiki.apache.org/confluence/display/METRON/Tech+Talks

On Thu, Aug 3, 2017 at 10:32 AM, Casey Stella  wrote:

> The Videos of talks that Simon Ball and I gave at DataWorks Summit are now
> up and on youtube:
>
> * Solving Cyber at Scale (business-level track) - https://www.youtube.com/
> watch?v=zVdRhwfum4Q
> * Model as a Service (technical track) - https://www.youtube.com/
> watch?v=LkrOKvyAc0s
> * Metron Architecture (with demo from LANL data) (technical track) -
> https://www.youtube.com/watch?v=0LrrAQXhqGY
>
> These talks are mostly current based on the existing architecture and the
> demos reflect the alerting UI that is not committed yet.  There are blogs
> coming out in support of this over the next week or so.
>
> If anyone has any questions about the talks or want any more information,
> feel free to ask. :)
>
> Best,
>
> Casey
>

Re: [DISCUSS] Code Reformatting next steps

[Justin Leet]

Regarding Matt's point, I can pretty easily split the manual vs automatic
changes.  I won't have time today, but I (probably) can Sunday.  Commits
should be relatively low velocity then, so it should mostly naturally occur.

Regarding PRs, I'm personally okay with best effort, and just correcting
any remainders during any reformats (or possibly pairing a license fix with
a reformat or whatever practical solution to the general idea we want).  I
expect there to be some drift as PRs come in, but I'm going with the idea
that perfect is the enemy of good.  At that point, we just do as little as
possible for 777, and just take care of it during or near that particular
reformat. Seems like it mostly comes with the potential overhead of pairing
some tickets together, which is annoying, but not too bad.

It might be better to just submit a paired license fix ticket + a reformat
by module, if we'd prefer that.  I could just kill current 1087, we pick a
module, and submit a license fix ticket/PR + a refactor ticket/PR.  At that
point, we should mostly stay out 777's way, and we don't stop commits for
such a widely spread PR.  It's pretty much the same amount of work, and is
probably cleaner all around.

Justin


On Fri, Aug 11, 2017 at 2:23 PM, Otto Fowler <ottobackwa...@gmail.com>
wrote:

> What do we do for PR’s?
>
> I don’t think I should run this on 777 for example while it is under
> review.  But, I don’t think running it as a last commit before landing is
> correct either.
>
> Will it be:
> PR as is ( if you haven’t reformatted, don’t )
> Follow on PR with only reformatting
> ??
>
>
>
> On August 11, 2017 at 14:17:35, Matt Foley (mfo...@hortonworks.com) wrote:
>
> Regarding METRON-1087, I’m in favor of freezing commits for a day, to let
> Justin re-run the script for METRON-1087 over all of current master, and
> commit it.
> Perhaps, for assurance, this commit should only include the fully-automated
> “vast majority”; the couple dozen files that needed manual fixes could be
> split out into a patch that gets manual review. I don’t have a problem
> saying +1 on what is evidently script-automated (L1:’s#/**#/*#’ and blank
> line after license block).
> All of us with open PRs will then have to update to master, but we do that
> periodically anyway. Github is really quite good about not causing extra
> review work for update merges.
> IMHO,
> --Matt
>
> On 8/11/17, 5:14 AM, "Justin Leet" <justinjl...@gmail.com> wrote:
>
> Now that METRON-746 <https://github.com/apache/metron/pull/577> is in, we
> have a consistent code formatting setup where (for the most part) it can be
> autoapplied.
>
> Barring one existing PR, the main thing is just picking a module, doing the
> format, and testing it out. Obviously, I'd like to avoid collisions with
> any major PRs out there (say METRON-777
> <https://github.com/apache/metron/pull/530>), so things like parsers are
> out.
> Does anybody have any thoughts on what should be grabbed first? Maybe
> metron-stellar since it's pretty easy to test and even though it gets PRs,
> they're typically fairly small and contained?
>
> The main hiccup before being able to do any blanket reformatting is this
> PR:
> METRON-1087: Adjust license headers to be comments instead of Javadoc
> <https://github.com/apache/metron/pull/685>
>
> Without it, all the license headers get reformatted in Javadoc style when
> autoformatted (this actually happened before, but since we didn't actually
> format our code much it was pretty benign). Once this is in, I'm fine doing
> any headers that sneak in via PRs, it's a pretty easy find/replace in the
> editor.
>
> Once we're confident this works fairly smoothly, it should be pretty easy
> to branch out and start hitting more modules in whatever priority order we
> want.
>
> Justin
>

Re: Metron Alerts bombing in Travis?

[Justin Leet]

I just merged Raghu's PR into master, so we should be good to go.

On Fri, Aug 11, 2017 at 12:53 PM, Justin Leet <justinjl...@gmail.com> wrote:

> I disagree with the hardcoded versions being bad, the obvious concern I
> have being the repeatability of the build.
>
> I'd also like to note that we release source as the Apache release
> artifacts, anything else we put out is convenience.  Say we have a
> situation where we rely on library Foo version 2.0+. At version 3.0, they
> (or one of their deps) change license to a Cat-X license.  At that point,
> any builds (and even prior released we have!) use the Cat-X dependency by
> default.  That seems bad, but I have no idea what the standard Apache
> thinking on that is.
>
>
>
> On Fri, Aug 11, 2017 at 11:12 AM, RaghuMitra Kandikonda <
> raghumitra@gmail.com> wrote:
>
>> I agree that we can have a separate discussion about moving to yarn.
>> Hardcoding all the versions in package.json is a bad idea considering that
>> JS world releases happen very frequently and it's easy to miss on a good
>> feature.
>>
>> All the ASF projected that I know of have moved to yarn and not hard coded
>> the versions in package.json file.  I can fix the current issue and start
>> a
>> discussion about Yarn. The yarn migration is pretty straightforward (
>> which
>> I should have done in the first place) we can have it in before we have
>> another build casualty.
>>
>> What do you guys say ?.
>>
>> -Raghu Mitra
>>
>> On Fri, Aug 11, 2017 at 8:24 PM, Nick Allen <n...@nickallen.org> wrote:
>>
>> > Yes, I agree Justin.  I think we need to have another (maybe separate?)
>> > discussion on changing the dependency tool set and discuss pros and
>> cons.
>> >
>> > On Fri, Aug 11, 2017 at 10:45 AM Justin Leet <justinjl...@gmail.com>
>> > wrote:
>> >
>> > > I'm honestly not very familiar with this stuff, but this seems more
>> like
>> > a
>> > > "how we use it issue", rather than an issue with the tech itself.  Do
>> we
>> > > have a more compelling reason, or set of reasons, beyond just fixing
>> to
>> > > specific version of our dependencies?
>> > >
>> > > On Fri, Aug 11, 2017 at 10:37 AM, Nick Allen <n...@nickallen.org>
>> wrote:
>> > >
>> > > > And I suggest that as a short-term (or medium-term) fix, separate
>> from
>> > > > moving wholesale to another dependency mechanism.
>> > > >
>> > > >
>> > > >
>> > > > On Fri, Aug 11, 2017 at 10:36 AM Nick Allen <n...@nickallen.org>
>> > wrote:
>> > > >
>> > > > > Would it make sense to just 'fix' at a specific version all NPM
>> > > > > dependencies, instead of allowing any version after x.y.z ?
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Fri, Aug 11, 2017 at 10:34 AM Casey Stella <ceste...@gmail.com
>> >
>> > > > wrote:
>> > > > >
>> > > > >> I agree; we should have a long-term JIRA for moving to yarn (so
>> > > > confusing
>> > > > >> *sigh*) and a tactical JIRA + PR to fix the build.  We should
>> > > prioritize
>> > > > >> review of that tactical JIRA.
>> > > > >>
>> > > > >> Casey
>> > > > >>
>> > > > >> On Fri, Aug 11, 2017 at 10:19 AM, Otto Fowler <
>> > > ottobackwa...@gmail.com>
>> > > > >> wrote:
>> > > > >>
>> > > > >> > Can you create a jira for moving to yarn please.
>> > > > >> >
>> > > > >> >
>> > > > >> > On August 11, 2017 at 10:17:40, Otto Fowler (
>> > > ottobackwa...@gmail.com)
>> > > > >> > wrote:
>> > > > >> >
>> > > > >> > Are there any other NPM modules where we are exposed to this
>> risk?
>> > > > >> >
>> > > > >> >
>> > > > >> > On August 11, 2017 at 10:11:22, RaghuMitra Kandikonda (
>> > > > >> > raghumitra@gmail.com) wrote:
>> > > > >> >
>> > > > >> > Ok, Bootstrap made a release 10hrs ago and they removed few
>> files
>> > > that
>>

Re: [DISCUSS] Code Reformatting next steps

[Justin Leet]

https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines
is updated to reflect the code style standards.  It also includes
instructions for setting up warnings and autoformatting in both IntelliJ
and Eclipse.

However, the current standard is to not update files that don't match the
overall style.  The larger, one-off, reformats are just to get things into
a consistent style such that we expect everything to actually follow our
style.  IntelliJ lets you reformat at module level (including file masks),
so any module level reformat (best case) is to just reformat on a *.java
mask and make sure nothing went wrong.  I have tested this before on our
project, hence 1087.

Updating the PR template is a good callout though. I'll create a ticket for
it.

On Fri, Aug 11, 2017 at 9:18 AM, Otto Fowler <ottobackwa...@gmail.com>
wrote:

> We should add something to the PR template about this.  Did we change the
> contributor’s guide?
>
> Also, are there common steps to take with Intellij, so that we all do the
> same thing
> for mass reformats?
>
> On August 11, 2017 at 08:14:21, Justin Leet (justinjl...@gmail.com) wrote:
>
> Now that METRON-746 <https://github.com/apache/metron/pull/577> is in, we
> have a consistent code formatting setup where (for the most part) it can
> be
> autoapplied.
>
> Barring one existing PR, the main thing is just picking a module, doing
> the
> format, and testing it out. Obviously, I'd like to avoid collisions with
> any major PRs out there (say METRON-777
> <https://github.com/apache/metron/pull/530>), so things like parsers are
> out.
> Does anybody have any thoughts on what should be grabbed first? Maybe
> metron-stellar since it's pretty easy to test and even though it gets PRs,
> they're typically fairly small and contained?
>
> The main hiccup before being able to do any blanket reformatting is this
> PR:
> METRON-1087: Adjust license headers to be comments instead of Javadoc
> <https://github.com/apache/metron/pull/685>
>
> Without it, all the license headers get reformatted in Javadoc style when
> autoformatted (this actually happened before, but since we didn't actually
> format our code much it was pretty benign). Once this is in, I'm fine
> doing
> any headers that sneak in via PRs, it's a pretty easy find/replace in the
> editor.
>
> Once we're confident this works fairly smoothly, it should be pretty easy
> to branch out and start hitting more modules in whatever priority order we
> want.
>
> Justin
>
>

[DISCUSS] Code Reformatting next steps

[Justin Leet]

Now that METRON-746  is in, we
have a consistent code formatting setup where (for the most part) it can be
autoapplied.

Barring one existing PR, the main thing is just picking a module, doing the
format, and testing it out. Obviously, I'd like to avoid collisions with
any major PRs out there (say METRON-777
), so things like parsers are
out.
Does anybody have any thoughts on what should be grabbed first? Maybe
metron-stellar since it's pretty easy to test and even though it gets PRs,
they're typically fairly small and contained?

The main hiccup before being able to do any blanket reformatting is this PR:
METRON-1087: Adjust license headers to be comments instead of Javadoc


Without it, all the license headers get reformatted in Javadoc style when
autoformatted (this actually happened before, but since we didn't actually
format our code much it was pretty benign). Once this is in, I'm fine doing
any headers that sneak in via PRs, it's a pretty easy find/replace in the
editor.

Once we're confident this works fairly smoothly, it should be pretty easy
to branch out and start hitting more modules in whatever priority order we
want.

Justin

Re: Metron Alerts bombing in Travis?

[Justin Leet]

I disagree with the hardcoded versions being bad, the obvious concern I
have being the repeatability of the build.

I'd also like to note that we release source as the Apache release
artifacts, anything else we put out is convenience.  Say we have a
situation where we rely on library Foo version 2.0+. At version 3.0, they
(or one of their deps) change license to a Cat-X license.  At that point,
any builds (and even prior released we have!) use the Cat-X dependency by
default.  That seems bad, but I have no idea what the standard Apache
thinking on that is.



On Fri, Aug 11, 2017 at 11:12 AM, RaghuMitra Kandikonda <
raghumitra@gmail.com> wrote:

> I agree that we can have a separate discussion about moving to yarn.
> Hardcoding all the versions in package.json is a bad idea considering that
> JS world releases happen very frequently and it's easy to miss on a good
> feature.
>
> All the ASF projected that I know of have moved to yarn and not hard coded
> the versions in package.json file.  I can fix the current issue and start a
> discussion about Yarn. The yarn migration is pretty straightforward ( which
> I should have done in the first place) we can have it in before we have
> another build casualty.
>
> What do you guys say ?.
>
> -Raghu Mitra
>
> On Fri, Aug 11, 2017 at 8:24 PM, Nick Allen <n...@nickallen.org> wrote:
>
> > Yes, I agree Justin.  I think we need to have another (maybe separate?)
> > discussion on changing the dependency tool set and discuss pros and cons.
> >
> > On Fri, Aug 11, 2017 at 10:45 AM Justin Leet <justinjl...@gmail.com>
> > wrote:
> >
> > > I'm honestly not very familiar with this stuff, but this seems more
> like
> > a
> > > "how we use it issue", rather than an issue with the tech itself.  Do
> we
> > > have a more compelling reason, or set of reasons, beyond just fixing to
> > > specific version of our dependencies?
> > >
> > > On Fri, Aug 11, 2017 at 10:37 AM, Nick Allen <n...@nickallen.org>
> wrote:
> > >
> > > > And I suggest that as a short-term (or medium-term) fix, separate
> from
> > > > moving wholesale to another dependency mechanism.
> > > >
> > > >
> > > >
> > > > On Fri, Aug 11, 2017 at 10:36 AM Nick Allen <n...@nickallen.org>
> > wrote:
> > > >
> > > > > Would it make sense to just 'fix' at a specific version all NPM
> > > > > dependencies, instead of allowing any version after x.y.z ?
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Aug 11, 2017 at 10:34 AM Casey Stella <ceste...@gmail.com>
> > > > wrote:
> > > > >
> > > > >> I agree; we should have a long-term JIRA for moving to yarn (so
> > > > confusing
> > > > >> *sigh*) and a tactical JIRA + PR to fix the build.  We should
> > > prioritize
> > > > >> review of that tactical JIRA.
> > > > >>
> > > > >> Casey
> > > > >>
> > > > >> On Fri, Aug 11, 2017 at 10:19 AM, Otto Fowler <
> > > ottobackwa...@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > Can you create a jira for moving to yarn please.
> > > > >> >
> > > > >> >
> > > > >> > On August 11, 2017 at 10:17:40, Otto Fowler (
> > > ottobackwa...@gmail.com)
> > > > >> > wrote:
> > > > >> >
> > > > >> > Are there any other NPM modules where we are exposed to this
> risk?
> > > > >> >
> > > > >> >
> > > > >> > On August 11, 2017 at 10:11:22, RaghuMitra Kandikonda (
> > > > >> > raghumitra@gmail.com) wrote:
> > > > >> >
> > > > >> > Ok, Bootstrap made a release 10hrs ago and they removed few
> files
> > > that
> > > > >> we
> > > > >> > were using hence the issue. We need to update package.json in
> > > > >> metron-alerts
> > > > >> > to use bootstrap '4.0.0-alpha.6' not any version after that. So
> > the
> > > > >> change
> > > > >> > would be metron-alerts/package.json:25 change bootstrap from
> > > > >> > '^4.0.0-alpha.6' to '4.0.0-alpha.6'.
> > > > >> >
> > > > >> > The long term fix should be to move to a better build tool like
> > > > 'yarn' (
>

Re: Metron Alerts bombing in Travis?

[Justin Leet]

I'm honestly not very familiar with this stuff, but this seems more like a
"how we use it issue", rather than an issue with the tech itself.  Do we
have a more compelling reason, or set of reasons, beyond just fixing to
specific version of our dependencies?

On Fri, Aug 11, 2017 at 10:37 AM, Nick Allen  wrote:

> And I suggest that as a short-term (or medium-term) fix, separate from
> moving wholesale to another dependency mechanism.
>
>
>
> On Fri, Aug 11, 2017 at 10:36 AM Nick Allen  wrote:
>
> > Would it make sense to just 'fix' at a specific version all NPM
> > dependencies, instead of allowing any version after x.y.z ?
> >
> >
> >
> > On Fri, Aug 11, 2017 at 10:34 AM Casey Stella 
> wrote:
> >
> >> I agree; we should have a long-term JIRA for moving to yarn (so
> confusing
> >> *sigh*) and a tactical JIRA + PR to fix the build.  We should prioritize
> >> review of that tactical JIRA.
> >>
> >> Casey
> >>
> >> On Fri, Aug 11, 2017 at 10:19 AM, Otto Fowler 
> >> wrote:
> >>
> >> > Can you create a jira for moving to yarn please.
> >> >
> >> >
> >> > On August 11, 2017 at 10:17:40, Otto Fowler (ottobackwa...@gmail.com)
> >> > wrote:
> >> >
> >> > Are there any other NPM modules where we are exposed to this risk?
> >> >
> >> >
> >> > On August 11, 2017 at 10:11:22, RaghuMitra Kandikonda (
> >> > raghumitra@gmail.com) wrote:
> >> >
> >> > Ok, Bootstrap made a release 10hrs ago and they removed few files that
> >> we
> >> > were using hence the issue. We need to update package.json in
> >> metron-alerts
> >> > to use bootstrap '4.0.0-alpha.6' not any version after that. So the
> >> change
> >> > would be metron-alerts/package.json:25 change bootstrap from
> >> > '^4.0.0-alpha.6' to '4.0.0-alpha.6'.
> >> >
> >> > The long term fix should be to move to a better build tool like
> 'yarn' (
> >> > https://yarnpkg.com/en/) instead of npm. But for now, i will submit a
> >> PR
> >> > to
> >> > fix the bootstrap version.
> >> >
> >> > If you wish to test the fix locally or have a working build before
> >> taking
> >> > my PR, edit package.json file and run 'npm install' and run 'npm run
> >> > build'.
> >> >
> >> > https://github.com/twbs/bootstrap/releases
> >> >
> >> > -Raghu Mitra
> >> >
> >> > On Fri, Aug 11, 2017 at 7:25 PM, Otto Fowler  >
> >> > wrote:
> >> >
> >> > > Where are the bootstrap files?
> >> > >
> >> > > the @import ~bootstrap/….. looks like it is failing?
> >> > >
> >> > >
> >> > >
> >> > > On August 11, 2017 at 09:51:03, RaghuMitra Kandikonda (
> >> > > raghumitra@gmail.com) wrote:
> >> > >
> >> > > I am running a clean install now, I will keep you posted.
> >> > >
> >> > > -Raghu Mitra
> >> > >
> >> > > On Fri, Aug 11, 2017 at 7:02 PM, Casey Stella 
> >> > wrote:
> >> > >
> >> > > > Yes, I'm getting it locally too, here's more context:
> >> > > >
> >> > > > [INFO] ERROR in multi script-loader!./~/jquery/dist/jquery.js
> >> > > > script-loader!./~/tether/dist/js/tether.js
> >> > > > script-loader!./~/ace-builds/src-noconflict/ace.js
> >> > > > [INFO] Module not found: Error: Can't resolve
> >> > > > '/Users/cstella/Documents/workspace/metron/fork/
> >> > incubator-metron/metron-
> >> > > > interface/metron-alerts/node_modules/jquery/dist/jquery.js'
> >> > > > in
> >> > > > '/Users/cstella/Documents/workspace/metron/fork/
> >> > incubator-metron/metron-
> >> > > > interface/metron-alerts/node_modules/@angular/cli/models/
> >> > > webpack-configs'
> >> > > > [INFO] @ multi script-loader!./~/jquery/dist/jquery.js
> >> > > > script-loader!./~/tether/dist/js/tether.js
> >> > > > script-loader!./~/ace-builds/src-noconflict/ace.js
> >> > > > [INFO]
> >> > > > [INFO] ERROR in multi script-loader!./~/jquery/dist/jquery.js
> >> > > > script-loader!./~/tether/dist/js/tether.js
> >> > > > script-loader!./~/ace-builds/src-noconflict/ace.js
> >> > > > [INFO] Module not found: Error: Can't resolve
> >> > > > '/Users/cstella/Documents/workspace/metron/fork/
> >> > incubator-metron/metron-
> >> > > > interface/metron-alerts/node_modules/tether/dist/js/tether.js'
> >> > > > in
> >> > > > '/Users/cstella/Documents/workspace/metron/fork/
> >> > incubator-metron/metron-
> >> > > > interface/metron-alerts/node_modules/@angular/cli/models/
> >> > > webpack-configs'
> >> > > > [INFO] @ multi script-loader!./~/jquery/dist/jquery.js
> >> > > > script-loader!./~/tether/dist/js/tether.js
> >> > > > script-loader!./~/ace-builds/src-noconflict/ace.js
> >> > > > [INFO]
> >> > > > [INFO] ERROR in ./src/vendor.scss
> >> > > > [INFO] Module build failed:
> >> > > > [INFO] @import "~bootstrap/scss/normalize";
> >> > > > [INFO] ^
> >> > > > [INFO] File to import not found or unreadable:
> >> > > > ~bootstrap/scss/normalize.
> >> > > > [INFO] Parent style sheet: stdin
> >> > > > [INFO] in
> >> > > > /Users/cstella/Documents/workspace/metron/fork/
> >> > incubator-metron/metron-
> >> > > > 

Re: [DISCUSS] Code Reformatting next steps

[Justin Leet]

Unfortunately, I did not getting around to splitting out the changes over
the weekend.

Matt, it looks like your reply was pretty close to my email, so I'm not
sure you saw it.  Would you have objections to just doing the license fix
and reformat as paired tickets, module-by-module?  I think that'll just be
easier all around to coordinate, and in retrospect I think it'll make my
life easier too.  At that point, I'll just kill the existing PR and put out
new ones when I have some free time.

On Fri, Aug 11, 2017 at 2:42 PM, Matt Foley <mfo...@hortonworks.com> wrote:

> We can wait until 777 goes in, but any smaller PRs should just roll with
> it.
> I’ve found (after doing it wrong a couple times) that if you just do the
> simple update merge:
> #in local git repo branch METRON-XXX, with additional git remote
> ‘apache’
> git fetch apache
> git merge apache/master
> #resolve merge conflicts
> git commit
> git push origin METRON-XXX
>
> then github very nicely doesn’t make the reviewers for the PR review all
> the additional files that got changed by the merge (unless they want to).
> So it’s low cost.
>
> So to be clear, we should wait for 777, then let Justin do the commit, and
> all open PRs immediately do an update merge – which in the vast majority of
> cases will be fully automatic in the merge, and unnecessary to review in
> the PR.
> --Matt
>
> From: Otto Fowler <ottobackwa...@gmail.com>
> Date: Friday, August 11, 2017 at 11:23 AM
> To: Matt Foley <mfo...@hortonworks.com>, "dev@metron.apache.org" <
> dev@metron.apache.org>
> Subject: Re: [DISCUSS] Code Reformatting next steps
>
> What do we do for PR’s?
>
> I don’t think I should run this on 777 for example while it is under
> review.  But, I don’t think running it as a last commit before landing is
> correct either.
>
> Will it be:
> PR as is ( if you haven’t reformatted, don’t )
> Follow on PR with only reformatting
> ??
>
>
>
>
> On August 11, 2017 at 14:17:35, Matt Foley (mfo...@hortonworks.com mfo...@hortonworks.com>) wrote:
> Regarding METRON-1087, I’m in favor of freezing commits for a day, to let
> Justin re-run the script for METRON-1087 over all of current master, and
> commit it.
> Perhaps, for assurance, this commit should only include the
> fully-automated “vast majority”; the couple dozen files that needed manual
> fixes could be split out into a patch that gets manual review. I don’t have
> a problem saying +1 on what is evidently script-automated (L1:’s#/**#/*#’
> and blank line after license block).
> All of us with open PRs will then have to update to master, but we do that
> periodically anyway. Github is really quite good about not causing extra
> review work for update merges.
> IMHO,
> --Matt
>
> On 8/11/17, 5:14 AM, "Justin Leet" <justinjl...@gmail.com justinjl...@gmail.com>> wrote:
>
> Now that METRON-746 <https://github.com/apache/metron/pull/577> is in, we
> have a consistent code formatting setup where (for the most part) it can be
> autoapplied.
>
> Barring one existing PR, the main thing is just picking a module, doing the
> format, and testing it out. Obviously, I'd like to avoid collisions with
> any major PRs out there (say METRON-777
> <https://github.com/apache/metron/pull/530>), so things like parsers are
> out.
> Does anybody have any thoughts on what should be grabbed first? Maybe
> metron-stellar since it's pretty easy to test and even though it gets PRs,
> they're typically fairly small and contained?
>
> The main hiccup before being able to do any blanket reformatting is this
> PR:
> METRON-1087: Adjust license headers to be comments instead of Javadoc
> <https://github.com/apache/metron/pull/685>
>
> Without it, all the license headers get reformatted in Javadoc style when
> autoformatted (this actually happened before, but since we didn't actually
> format our code much it was pretty benign). Once this is in, I'm fine doing
> any headers that sneak in via PRs, it's a pretty easy find/replace in the
> editor.
>
> Once we're confident this works fairly smoothly, it should be pretty easy
> to branch out and start hitting more modules in whatever priority order we
> want.
>
> Justin
>
>

Re: ability to submit patches

[Justin Leet]

Hi Artem,

We do our patches and reviews via pull requests on GitHub (
https://github.com/apache/metron)

If you aren't familiar with the GitHub PR process, take a look at
https://help.github.com/articles/creating-a-pull-request-from-a-fork/

Let us know if there's anything in particular you need help with there and
we can definitely help out.

Justin

On Mon, Jul 24, 2017 at 9:07 AM, Artem Ervits  wrote:

> Hello community, please grant user dbist13 ability to contribute patches on
> Jira. Opened JIRA METRON-1058 and attached patch, can't submit it.
>
> Thanks
>

METRON-1004 Build improvements are in master

[Justin Leet]

METRON-1004 was made to address several issues we were having with the
Travis builds.  Specifically that the builds went on too long.

After the merge into master, builds should be more consistent, stable, and
take about 25-30 minutes.

Please merge master into PRs if you are experiencing issues getting stable
Travis builds.  Feel free to reach out to me if any assistance is needed.

A short summary of the highlights.

* Moving Travis to a VM build, which improves stability and lowers variance
across builds
* Reusing test infrastructure across tests, to avoid spinup/spindown costs
* A variety of tests have been refactored for performance.  For example,
the parser tests have been modified to be less full integration test for
each individual parser.
* Hopefully stopping the infamous Storm slot issue that killed builds.
* Fixing an intermittent bug involving a Spring interaction with Kafka.

See https://github.com/apache/metron/pull/624 for more details.

Re: [VOTE] Apache Metron 0.4.0 release

[Justin Leet]

+1 (Non-binding)

* Verified Keys
* Verified mvn clean install completed successfully
* Ran full dev: saw data flow through, ran a couple of the REST APIs, and
opened up and clicked through a bit of the Management API.
* Examined site-book and didn't see any issues

On Thu, Jun 29, 2017 at 11:46 AM, Casey Stella  wrote:

> +1 (binding)
> * Verified keys
> * Verified mvn build
> * Verified unit and integration tests run
> * Verified license check runs
> * Verified fulldev spun up with smoketest
>
> On Wed, Jun 28, 2017 at 8:10 PM, Anand Subramanian <
> asubraman...@hortonworks.com> wrote:
>
> > +1 (non-binding)
> >
> > * Brought up Metron stack on 12-node CentOS7 openstack cluster
> > * Verify all services come up fine [PASS]
> > * Bro, YAF and snort - ingest into respective kafka topics and write
> > indices [PASS]
> > * Add squid telemetry, ingest into kafka topic and write indices [PASS]
> > * Metron YAF Zeppelin dashboard with sample ingested YAF data [PASS]
> > * Management UI and REST Swagger UI sanity check [PASS]
> >
> >
> > -Anand
> >
> >
> >
> >
> >
> > On 6/28/17, 12:06 AM, "Matt Foley"  wrote:
> >
> > >This is a call to vote on releasing this rc4 as “Apache Metron 0.4.0”.
> > >(Note: this is rc4 because the release candidate needed to be modified
> > with another commit after the rc3 tag was pushed to public.)
> > >
> > >Full list of changes in this release:
> > >https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/RELEASE_NOTES
> > >
> > >The tag/commit to be voted upon is:
> > >d52f574f8294e453ecad3871526858a0c3c2033d (tag apache-metron-0.4.0-rc4)
> > >
> > >The source archive being voted upon can be found here:
> > >https://dist.apache.org/repos/dist/dev/metron/0.4.0-
> > RC4/apache-metron-0.4.0-rc4.tar.gz
> > >and in github at:
> > >https://github.com/apache/metron/tree/Metron_0.4.0
> > >
> > >Other release files, signatures and digests can be found here:
> > >https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/KEYS
> > >
> > >The release artifacts are signed with the following key:
> > >https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/KEYS
> > >pub   rsa4096/4169AA27ECB31663 2011-07-31 [SCEA]
> > >Key fingerprint = 7854 36A7 8258 6B71 829C  67A0 4169 AA27 ECB3 1663
> > >uid = Matthew Foley (CODE SIGNING KEY) 
> > >
> > >Please vote on releasing this package as Apache Metron 0.4.0.
> > >When voting, please list the actions taken to verify the release.
> > >
> > >Recommended build validation and verification instructions are posted
> > here:
> > >https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds
> > >
> > >This vote will be open for at least 72 hours.  Please vote one of the
> > following responses:
> > >+1 Release this package as Apache Metron 0.4.0-RC4
> > >0 No opinion
> > >-1 Do not release this package because...
> > >
> > >Thank you,
> > >--Matt
> > >(your friendly release manager)
> > >
> > >
> > >
> >
>

Re: PLEASE DON'T COMMIT TO APACHE METRON until propagation issue is fixed

[Justin Leet]

Looks like it's already been closed, with a5b1377 as the latest commit.

We should be good to go at this point.  Any objections, Matt?

On Mon, Jul 3, 2017 at 2:39 PM, Matt Foley  wrote:

> https://issues.apache.org/jira/browse/INFRA-14502 has been opened.
>
> On 7/3/17, 11:25 AM, "Matt Foley"  wrote:
>
> It seems propagation from git-wip-us.apache.org/repos/asf/metron to
> github.com/apache/metron is currently broken.  If you browse
> https://git1-us-west.apache.org/repos/asf?p=metron.git you see that
> commit a5b13777a, for METRON-877, was committed yesterday but is still not
> present in github: https://github.com/apache/metron/commits/master (as of
> 11:20am PDT).
>
> Making commits to Apache from an incompletely merged starting point
> could cause deletion or corruption of previous commits (although of course
> experts with git could do it right if they’re aware of the problem).  I’m
> opening an INFRA ticket to request help restarting the propagation.
>
> Thanks,
> --Matt
>
> On Mon, Jul 3, 2017 at 4:16 AM -0700, "Otto Fowler" <
> ottobackwa...@gmail.com> wrote:
>
>
> This still has not hit.  Should we be concerned about
> merges/commits before
> it does?
>
>
>
> On July 2, 2017 at 16:27:43, mattf-horton (g...@git.apache.org)
> wrote:
>
> Github user mattf-horton commented on the issue:
>
> https://github.com/apache/metron/pull/616
>
> This has been committed to git-wip-us.apache.org/repos/asf/metron.
> Propagation seems a little slow today.
>
>
>
>
>
>

UI pivotting / aggregation backend

[Justin Leet]

I wanted to bring up a some stuff on the backend of our UI, and get
thoughts (+ things I overlooked, etc.).  There's also a couple points at
the end that merit discussion about how we handle things, since it gets
into how we handle our ES templates (since we generally want to aggregate
on raw fields, not analyzed ones).

To set the use case a bit, when we're looking through alerts in the UI,
we're going to want to be able to start pivoting and grouping in the UI.

For example, given a list of alerts, we may want to follow a ordering of
groupings like so:

All Alerts
--> Bucketed by User
> Then further by Destination IP
--> Then further by Severity

The stuff I expect we'll want to be able to do:
* Pivot through multiple layers (as in the example above).
* Get counts within each bucket (Do we have a lot of high severity alerts?
Mostly medium? etc?)
* Get a subset of fields (I assume we don't want every entire doc that
comes back in the bucket)
* Pagination (if I have > X docs, show me X and let me retrieve more as
needed)
* Sorting within a bucket (I may want to sort by time, by userid, etc.)
* Filtering (Be able to do this stuff while only showing high severity
alerts)

In terms of actually implementing this, to the best of my limited knowledge
(and playing around with ES looking into this), this seems like pretty
doable stuff, out of the box. See:
https://www.elastic.co/guide/en/elasticsearch/reference/2.4/search-aggregations-bucket-terms-aggregation.html

There are two main pain points I see in this:
* Actually constructing these queries.  I don't know that we've explicitly
said we want a layer of abstraction between the UI and the real time store,
but I strongly suggest we have one.  Theoretically, we should be able to
support (at least) Solr and ES in the UI, not just one.  Unfortunately,
since they aren't the same syntax, this means we have two impls, and I'd
personally like to see an abstraction that delegates appropriately.

* Aggregations in ES function post analysis. This means that we'll
typically want the raw field value to be able to aggregated on.  In ES
implementation, this means a "not_analyzed" field. Glancing (incredibly)
briefly through our templates, we do have some string values that are
analyzed (and I have no idea if they're generally relevant to this UI or
not, I just didn't look).  I'm also assuming Stellar enrichments are
analyzed right now.  I'm also unsure what happens to metadata (
https://github.com/apache/metron/pull/621)  Essentially the question is:
"How do we handle this, particularly since we're a pretty dynamic system?"

Re: [ANNOUNCE] Apache Metron 0.4.0 release

[Justin Leet]

Congrats, everyone!  A lot of people helped out across the board, and I
look forward to everyone's contributions moving ahead.

On Wed, Jul 5, 2017 at 4:47 PM, Otto Fowler  wrote:

> Thank you Matt, your first metron release as well!
> Congratulations to the community.
>
>
> On July 5, 2017 at 16:35:47, Matt Foley (ma...@apache.org) wrote:
>
> Friends and Colleagues,
> I’m happy to announce the completion and release of Apache Metron 0.4.0.
> Besides a bunch of great new features, this is also our first release as a
> TLP.
>
> The public website at http://metron.apache.org/ has been updated and has
> correct links to the new downloads and docs.
>
> Full list of changes in this release is in the Release Notes:
> http://www.apache.org/dyn/closer.cgi/metron/0.4.0/RELEASE_NOTES
>
> And the release point is tagged in github as
> “apache-metron-0.4.0-release”, viewable at:
> https://github.com/apache/metron/tree/apache-metron-0.4.0-release
> This is also the same has HEAD of Metron_0.4.0 branch.
>
> Every release is a team effort, and this one with 140+ tasks and bug
> fixes, certainly involved the whole community. Congratulations and THANK
> YOU to everyone who contributed, whether ideas, code, reviews,
> documentation, or testing.
>
> Best regards,
> --Matt (as Release Manager)
>
>
>
>

Re: [DISCUSS] Regression introduced in Full Dev

[Justin Leet]

The problem seems to be that full/quick dev rely on having the RPMs in the
/target folder

>From metron-deployment/roles/metron-rpms/defaults:
metron_rpm_glob: "{{ playbook_dir
}}/../packaging/docker/rpm-docker/target/RPMS/noarch/*.rpm"

However, that PR appears to no longer copy to the /target folder run as
expected, so quick and full dev are unable to find the RPMs.

On Tue, Apr 25, 2017 at 9:26 PM, Otto Fowler 
wrote:

> Ryan, did you see why it was not working?
>
>
> On April 25, 2017 at 18:17:29, Ryan Merriman (merrim...@gmail.com) wrote:
>
> A regression was introduced recently that breaks full dev. I've narrowed
> down the commit that introduced it and have submitted a PR to revert that
> commit: https://github.com/apache/incubator-metron/pull/549.
>
> Given there has been confusion recently over our deployment build process,
> I think it's appropriate that we discuss and come to a consensus and on
> how this should work.
>
> Ryan
>

Re: [DISCUSS] Storm topology.classpath setting

[Justin Leet]

topology.classpath is a topology level config, so Mike's potential
alternative is to migrate that down from Ambari into the actual topologies
we launch (assuming it works as expected).

Basically we're trading ease of use in Ambari vs. additional setup for each
topology that's run.

I'm in favor of pushing the config down to each topology (again, assuming
it works), despite the additional topology work, because it seems more
likely that users are going to see the issues from not restarting Storm and
Metron immediately after install/start than they are from missing configs
at the Flux level.  The topology config just becomes a bullet point in
instructions on creating a new topology, rather than a surprising and
nonintuitive issue resulting from install.

Once Metron installs and starts through Ambari, I would expect it to work
immediately.

On Tue, Apr 25, 2017 at 5:14 PM, Casey Stella  wrote:

> I don't know that I'm offended by either approach, but the way we have it
> now isn't offending me overly, honestly.
>
> Casey
>
> On Tue, Apr 25, 2017 at 5:11 PM, David Lyle  wrote:
>
> > I think you have one more restart than is strictly required but I get
> your
> > point. IIRC, when I did the original implementation there was a bit of a
> > wrinkle getting the right configs on the right classpath so I punted in
> > favor of topology.classpath.
> >
> > If you've got a more efficient way, I'm all for it.
> >
> > -D...
> >
> >
> > On Tue, Apr 25, 2017 at 4:55 PM, Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > We currently define topology.classpath at the global level for Storm.
> The
> > > reason for this feature was to enable setting the classpath to include
> > > things like hbase-site and core-site without modifying the topology jar
> > > files on the cluster. That made things simpler, but this also has the
> > > consequence that Metron's installation process via Ambari looks
> something
> > > like this:
> > >
> > >1. Install Metron
> > >2. Start Metron - part of install process
> > >3. Restart Storm  - bc of the new property that has been added that
> > >needs to be distributed
> > >4. Restart Metron - the topologies won't get the new classpath
> > otherwise
> > >
> > > We're in effect restarting Metron a couple times, which can take some
> > time.
> > >
> > > A possible alternative here is to set the classpath on the individual
> > > topologies. By doing so, we remove the need to restart Storm and Metron
> > > during the install. The down side is that this requires us to duplicate
> > > this setting for every flux file or topology, including newly-added
> > > topologies. What do folks think about which would be better?
> > >
> > > Best,
> > > Mike
> > >
> >
>

Re: [DISCUSS] Easing the ramp-up into contributing

[Justin Leet]

I created METRON-1071 <https://issues.apache.org/jira/browse/METRON-1071> to
track the CONTRIBUTING.md.  I also called out personal Travis setup
instructions, because that's pretty closely tied our dev experience and
seems like a good place to let it live.

On Fri, Jul 28, 2017 at 1:43 PM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Hi Artem, You could probably get away with not running them locally since
> we use Travis, which functions similarly to Jenkins from a testing
> perspective. Additionally, you can hook up Travis to run on your own github
> account as many Metron committers have already done. The upshot is that if
> your PR breaks an unrelated test, it would be up to you to debug and fix
> before it's able to be +1'ed and committed. Does that make sense?
>
> Best,
> Mike
>
> On Fri, Jul 28, 2017 at 11:05 AM, Artem Ervits <artemerv...@gmail.com>
> wrote:
>
>> Being new to this community and attempting to contribute METRON-1058,
>> METRON-1059 and METRON-1063, my experience is that requirement to run the
>> full tests and integration tests in order to contribute even a simple
>> patch
>> is overkill. I understand that community wants high quality contributions
>> but leaving the burden of running full test suites to a contributor is too
>> much. I've contributed to other communities and hurdle of running full
>> tests were on Jenkins side.
>> my 2c.
>>
>> On Fri, Jul 28, 2017 at 11:51 AM, Otto Fowler <ottobackwa...@gmail.com>
>> wrote:
>>
>> > +1 for CONTRIBUTING.md
>> >
>> >
>> > On July 28, 2017 at 09:14:38, Justin Leet (justinjl...@gmail.com)
>> wrote:
>> >
>> > I'm gonna break replies into chunks, since I'm responding to a few
>> people
>> > here.
>> >
>> > Re Nick:
>> > I'm honesty not sure how we'd want to improve that portion of the
>> landing
>> > experience, and I'm going to be totally honest the elevator pitch
>> > articulation of things is something I'm notably awful at. Having said
>> > that, I guarantee if that's your thought; other people have thought the
>> > same thing. I'd be cool with having an improved "What is Metron?", but
>> I'd
>> > probably need someone else to at least start that discussion up before I
>> > have a particularly useful opinion.
>> >
>> > Re Otto:
>> > What do we think about adding a CONTRIBUTING.md file, rolling it into
>> the
>> > site-book, and adding a short blurb linking to it on the main README.md?
>> > Github talks about it here:
>> > https://github.com/blog/1184-contributing-guidelines.
>> >
>> > I think this is also a good way to roll in Otto's comment about
>> encouraging
>> > reviews of PRs and emphasizing that we're definitely accepting of
>> > contributions across all spectrums.
>> >
>> > At that point, I think people should pretty easily be able to find it
>> from
>> > the main landing page and get a good overview of how people can hop in
>> and
>> > help. If we think that would be helpful, I'm happy to create a jira and
>> > grab it, even if it might take a bit to get to. It's also a good way to
>> > migrate of that info out of the wiki like Jon mentioned.
>> >
>> > Re Matt:
>> > Thanks for taking offering to take a look at it. I agree, I don't think
>> > it's the most important thing, but it is (hopefully) a pretty low
>> hanging
>> > fruit that provides value to people just hopping in (or even hopping
>> into a
>> > new area). I think the label is a good idea, because I think we're
>> > probably going to start having tickets to improve and clean some of
>> that up
>> > as we mature.
>> >
>> > On Thu, Jul 27, 2017 at 3:33 PM, Matt Foley <mfo...@hortonworks.com>
>> > wrote:
>> >
>> > > Really good discussion thread, thanks for opening it Justin.
>> > >
>> > > I’m a fan of adding javadocs to the publicly available doc set. It’s
>> not
>> > > the most important of the items listed below, but it is easy, and will
>> > push
>> > > people to be more attentive to dev documentation.
>> > >
>> > > METRON-759 is open for that, and I’ll take a crack at it.
>> > > I also suggest we start using “javadoc” (NOT “javadocs” plural) as a
>> > Label
>> > > on javadoc-related jiras.
>> > >
>> > > --Matt
>> > >
>> > > On 7/27/17, 10:36 AM, "Otto Fowler" &

Re: Java Code style is now Google Java Style Guide

[Justin Leet]

The wiki at Development Guidelines
<https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines>
has been updated, please take a look and if any updates are required, feel
free to reply and I'll update the page.

Justin

On Mon, Jul 31, 2017 at 10:23 PM, Justin Leet <justinjl...@gmail.com> wrote:

> Hi all,
>
> Following the merge of METRON-746
> <https://issues.apache.org/jira/browse/METRON-746> in
> https://github.com/apache/metron/pull/577, our code style is to follow
> the Google Java Style Guide
> <https://google.github.io/styleguide/javaguide.html>, where previously it
> was a two-space indented variant of the Sun standards.  Please take a look
> over the style guide, but it should mostly conform to what everyone is used
> to other than some minor things. This does not include any automated
> enforcement of code formatting, it's still the contributor's responsibility
> (and instructions for IntelliJ IDEA warnings and code formatting setup are
> included at the bottom).
>
> Right now, there is a ticket for reformatting our existing code, but
> specifics haven't been worked out (See: METRON-747
> <https://issues.apache.org/jira/browse/METRON-747>).  Please follow the
> existing practices of not reformatting files for unrelated changes and
> following the general formatting of a noncomplying files.
>
> The wiki has not been updated yet (I'll need permissions, if someone wants
> to work with me to update it).  Both the updated style and a set of IDE
> instructions will be added.  I'll follow up this email with a link when
> completed, but a wall of text setup is included both here and on the PR in
> the meantime
>
> Here's how to get warnings and formatting in IntelliJ IDEA.  Eclipse
> refused to show any warnings or errors with our project, so if someone is
> successfully using Eclipse (or wants to), I'm happy to help out and update
> the instructions. The PR's formatting might be more readable, so feel free
> to refer to it.
>
> The steps to get warnings in the IDE are:
>
>- Install the plugin (Available in IntelliJ's directly)
>- Go into Settings -> Other Settings -> Checkstyle
>- Change "Checkstyle version" to 8.0.
>- Apply. Otherwise the new file won't match the version and an error
>will be thrown.
>- Add a new Checkstyle file
>   - Use the checkstyle.xml file included in the root directory of the
>   project. This file is identical to google_checks.xml
>   
> <https://raw.githubusercontent.com/checkstyle/checkstyle/master/src/main/resources/google_checks.xml>,
>   but we may choose to change it in the future.
>   - If you have errors, you most likely need to make sure you're set
>   to the right version and have applied it.
>- Select the checkbox for the new style
>- Apply
>
> New warnings should show up in Java files, such as whitespace issues, etc.
> See Google's Java Style Checkstyle Coverage
> <http://checkstyle.sourceforge.net/google_style.html> for what exactly
> what Checkstyle can look for and what needs an actual human set of eyes.
>
> For code formatting:
>
> There are two options for IDE code formatting. One is to import the
> checkstyle.xml file from the project root. The other is to import Google's
> IntelliJ editor settings for coverage. In practice, Google's direct
> IntelliJ settings seem to handle autoformatting better.
>
> Follow the instructions for setting up formatting at
> https://github.com/HPI-Information-Systems/Metanome/
> wiki/Installing-the-google-styleguide-settings-in-intellij-and-eclipse.
> The direct Checkstyle can be imported similarly by choosing the CheckStyle
> Configuration when doing Import Scheme.
>
> At this point, running the code formatter should take care of most
> formatting issues.  In my experience several issues, such as commas being
> at the start of the line, will be not automatically resolved (although the
> IDE will give a warning).
>

Java Code style is now Google Java Style Guide

[Justin Leet]

Hi all,

Following the merge of METRON-746
 in
https://github.com/apache/metron/pull/577, our code style is to follow
the Google
Java Style Guide ,
where previously it was a two-space indented variant of the Sun standards.
Please take a look over the style guide, but it should mostly conform to
what everyone is used to other than some minor things. This does not
include any automated enforcement of code formatting, it's still the
contributor's responsibility (and instructions for IntelliJ IDEA warnings
and code formatting setup are included at the bottom).

Right now, there is a ticket for reformatting our existing code, but
specifics haven't been worked out (See: METRON-747
).  Please follow the
existing practices of not reformatting files for unrelated changes and
following the general formatting of a noncomplying files.

The wiki has not been updated yet (I'll need permissions, if someone wants
to work with me to update it).  Both the updated style and a set of IDE
instructions will be added.  I'll follow up this email with a link when
completed, but a wall of text setup is included both here and on the PR in
the meantime

Here's how to get warnings and formatting in IntelliJ IDEA.  Eclipse
refused to show any warnings or errors with our project, so if someone is
successfully using Eclipse (or wants to), I'm happy to help out and update
the instructions. The PR's formatting might be more readable, so feel free
to refer to it.

The steps to get warnings in the IDE are:

   - Install the plugin (Available in IntelliJ's directly)
   - Go into Settings -> Other Settings -> Checkstyle
   - Change "Checkstyle version" to 8.0.
   - Apply. Otherwise the new file won't match the version and an error
   will be thrown.
   - Add a new Checkstyle file
  - Use the checkstyle.xml file included in the root directory of the
  project. This file is identical to google_checks.xml
  
,
  but we may choose to change it in the future.
  - If you have errors, you most likely need to make sure you're set to
  the right version and have applied it.
   - Select the checkbox for the new style
   - Apply

New warnings should show up in Java files, such as whitespace issues, etc.
See Google's Java Style Checkstyle Coverage
 for what exactly what
Checkstyle can look for and what needs an actual human set of eyes.

For code formatting:

There are two options for IDE code formatting. One is to import the
checkstyle.xml file from the project root. The other is to import Google's
IntelliJ editor settings for coverage. In practice, Google's direct
IntelliJ settings seem to handle autoformatting better.

Follow the instructions for setting up formatting at
https://github.com/HPI-Information-Systems/Metanome/wiki/Installing-the-google-styleguide-settings-in-intellij-and-eclipse.
The direct Checkstyle can be imported similarly by choosing the CheckStyle
Configuration when doing Import Scheme.

At this point, running the code formatter should take care of most
formatting issues.  In my experience several issues, such as commas being
at the start of the line, will be not automatically resolved (although the
IDE will give a warning).

Re: Error building in Travis after taking master

[Justin Leet]

Does it run locally with:

mvn -q -T 2C -DskipTests install &&
 mvn -q -T 2C org.jacoco:prepare-agent surefire:test@unit-tests && mvn -q
org.jacoco:prepare-agent surefire:test@integration-tests  &&  mvn -q
org.jacoco:prepare-agent test --projects metron-interface/metron-config &&
 build_utils/verify_licenses.sh

On Wed, May 3, 2017 at 1:03 PM, Otto Fowler <ottobackwa...@gmail.com> wrote:

> No I cannot build locally with that command:
>
>
> [ERROR] No plugin found for prefix 'jacoco' in the current project and in
> the plugin groups [org.apache.maven.plugins, org.codehaus.mojo] available
> from the repositories [local (/Users/ottofowler/.m2/repository), central (
> https://repo.maven.apache.org/maven2)] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions,
> please read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/
> NoPluginFoundForPrefixException
>
>
> mvn package however, works fine.
>
>
>
> On May 3, 2017 at 12:51:34, Otto Fowler (ottobackwa...@gmail.com) wrote:
>
> I am going to be honest,
> I don’t usually build locally with :  mvn -q -T 2C -DskipTests install &&
>  mvn -q -T 2C jacoco:prepare-agent surefire:test@unit-tests && mvn -q
> jacoco:prepare-agent surefire:test@integration-tests  &&  mvn -q
> jacoco:prepare-agent test --projects metron-interface/metron-config &&
>  build_utils/verify_licenses.sh
>
> I build with mvn package
>
> I’m trying it now.
>
>
> On May 3, 2017 at 12:19:34, Michael Miklavcic (michael.miklav...@gmail.com
> )
> wrote:
>
> Also Otto, are you able to build locally with that branch and same merge
> with master?
>
> On Wed, May 3, 2017 at 9:51 AM, Justin Leet <justinjl...@gmail.com> wrote:
>
> > I'm also curious if it works if you change the two 'jacoco:prepare-agent'
> > to 'org.jacoco:prepare-agent'. Master has built off of this fine, so I'm
> > wondering if there's a difference in that build that breaks the plugin
> > resolution.
> >
> > On Wed, May 3, 2017 at 11:35 AM, Ryan Merriman <merrim...@gmail.com>
> > wrote:
> >
> > > You might want to try clearing the mvn cache. I've had travis get into
> a
> > > bad state before because of corrupt maven artifacts.
> > >
> > > On Wed, May 3, 2017 at 10:26 AM, Otto Fowler <ottobackwa...@gmail.com>
> > > wrote:
> > >
> > > > https://travis-ci.org/ottobackwards/incubator-
> > > metron/builds/228364894?utm_
> > > > source=email_medium=notification
> > > >
> > > > On May 3, 2017 at 11:12:15, Justin Leet (justinjl...@gmail.com)
> wrote:
> > > >
> > > > > Jacoco is introduced from
> > > > > https://github.com/apache/incubator-metron/pull/459.
> > > > >
> > > > > I'm not sure why you'd be getting a plugin error for it though,
> > because
> > > > > it's available in the standard repos and I was able to build off a
> > > > > completely clean maven cache (and Travis has been able to build it
> as
> > > > > well).
> > > > >
> > > > > What exactly you were running?
> > > > >
> > > > > On Wed, May 3, 2017 at 11:03 AM, Otto Fowler <
> > ottobackwa...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > Anyone know what this is?
> > > > >
> > > > > [ERROR] No plugin found for prefix 'jacoco' in the current project
> > and
> > > in
> > > > > the plugin groups [org.apache.maven.plugins, org.codehaus.mojo]
> > > available
> > > > > from the repositories [local (/home/travis/.m2/repository), central
> (
> > > > > https://repo.maven.apache.org/maven2)] -> [Help 1]
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: Error building in Travis after taking master

[Justin Leet]

I'm on 3.5 and not having issues with it.

On Wed, May 3, 2017 at 2:19 PM, Otto Fowler <ottobackwa...@gmail.com> wrote:

> Did we change the requirements for pre-commit testing to run the command
> that way?
> I did a commit this morning with the old command.
>
>
> On May 3, 2017 at 14:04:32, Otto Fowler (ottobackwa...@gmail.com) wrote:
>
> > git checkout -b whyyounowork apache/master
>
> same issue locally
>
>
>
>
> On May 3, 2017 at 13:46:14, Justin Leet (justinjl...@gmail.com) wrote:
>
> Is it just that one branch that's failing, or is it also master?  To try
> to narrow things down.
>
> On Wed, May 3, 2017 at 1:43 PM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
>> Hahaha, I can't imagine why you wouldn't run that command ¯\_(ツ)_/¯
>> Yeah, mvn clean package or mvn clean install are what I typically run.
>> (Install has the added benefit that if you've modified only files in one
>> module like metron-parsers, you can choose to build only that project bc
>> it
>> will grab dependency artifacts from your local cache). I wanted to see if
>> we could isolate the problem.
>>
>> On Wed, May 3, 2017 at 10:51 AM, Otto Fowler <ottobackwa...@gmail.com>
>> wrote:
>>
>> > I am going to be honest,
>> > I don’t usually build locally with :  mvn -q -T 2C -DskipTests install
>> &&
>> >  mvn -q -T 2C jacoco:prepare-agent surefire:test@unit-tests && mvn -q
>> > jacoco:prepare-agent surefire:test@integration-tests  &&  mvn -q
>> > jacoco:prepare-agent test --projects metron-interface/metron-config &&
>> >  build_utils/verify_licenses.sh
>> >
>> > I build with mvn package
>> >
>> > I’m trying it now.
>> >
>> >
>> > On May 3, 2017 at 12:19:34, Michael Miklavcic (
>> michael.miklav...@gmail.com)
>> > wrote:
>> >
>> > Also Otto, are you able to build locally with that branch and same merge
>> > with master?
>> >
>> > On Wed, May 3, 2017 at 9:51 AM, Justin Leet <justinjl...@gmail.com>
>> > wrote:
>> >
>> > > I'm also curious if it works if you change the two
>> > 'jacoco:prepare-agent'
>> > > to 'org.jacoco:prepare-agent'. Master has built off of this fine, so
>> I'm
>> > > wondering if there's a difference in that build that breaks the plugin
>> > > resolution.
>> > >
>> > > On Wed, May 3, 2017 at 11:35 AM, Ryan Merriman <merrim...@gmail.com>
>> > > wrote:
>> > >
>> > > > You might want to try clearing the mvn cache. I've had travis get
>> into
>> > a
>> > > > bad state before because of corrupt maven artifacts.
>> > > >
>> > > > On Wed, May 3, 2017 at 10:26 AM, Otto Fowler <
>> ottobackwa...@gmail.com>
>> >
>> > > > wrote:
>> > > >
>> > > > > https://travis-ci.org/ottobackwards/incubator-
>> > > > metron/builds/228364894?utm_
>> > > > > source=email_medium=notification
>> > > > >
>> > > > > On May 3, 2017 at 11:12:15, Justin Leet (justinjl...@gmail.com)
>> > wrote:
>> > > > >
>> > > > > > Jacoco is introduced from
>> > > > > > https://github.com/apache/incubator-metron/pull/459.
>> > > > > >
>> > > > > > I'm not sure why you'd be getting a plugin error for it though,
>> > > because
>> > > > > > it's available in the standard repos and I was able to build
>> off a
>> > > > > > completely clean maven cache (and Travis has been able to build
>> it
>> > as
>> > > > > > well).
>> > > > > >
>> > > > > > What exactly you were running?
>> > > > > >
>> > > > > > On Wed, May 3, 2017 at 11:03 AM, Otto Fowler <
>> > > ottobackwa...@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > Anyone know what this is?
>> > > > > >
>> > > > > > [ERROR] No plugin found for prefix 'jacoco' in the current
>> project
>> > > and
>> > > > in
>> > > > > > the plugin groups [org.apache.maven.plugins, org.codehaus.mojo]
>> > > > available
>> > > > > > from the repositories [local (/home/travis/.m2/repository),
>> > central (
>> > > > > > https://repo.maven.apache.org/maven2)] -> [Help 1]
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> >
>>
>
>

Re: Site-Book: mvn site never completes [MASTER BROKEN?]

[Justin Leet]

I'd argue it is. People can't validate at least some PRs, since if there's
doc changes, they can't make sure docs are good.

Sidenote, we could pretty easily loop the site into the Travis build to
avoid this type of thing.  I'd probably like to avoid building the javadocs
as part of that (if only because of the time involved).  We should be able
to use maven.javadoc.skip to do that, but I'd have to test it first.

On Tue, May 9, 2017 at 1:50 PM, Otto Fowler <ottobackwa...@gmail.com> wrote:

> So, is the build considered broken or what?
>
>
> On May 9, 2017 at 11:34:23, Justin Leet (justinjl...@gmail.com) wrote:
>
> https://github.com/apache/incubator-metron/blob/
> 716bda326f26b60c98318cdc1cc949b351a0b71a/metron-interface/
> metron-rest/README.md
> appears to be the problem. I can run the site successfully on the commit
> before, and this commit hangs.
>
> Just glancing through, the file appears fine on Github, but there might be
> an issue with the rendering the docs. I'm not really familiar enough with
> what's going on under the hood to know what the problem is, and I'd have
> to
> dig in more to have an informed opinion.
>
> Can someone double check me that 716bda32 fails and 38d26d43 succeeds?
>
> On Tue, May 9, 2017 at 11:26 AM, Otto Fowler <ottobackwa...@gmail.com>
> wrote:
>
> > I also did a >git checkout -b test-site apache/master
> > same issue.
> >
> >
> > On May 9, 2017 at 11:09:32, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
> >
> > Hi,
> > I am running mvn site for the site book, but it never completes. I am
> not
> > sure if there is anything I can do to trouble shoot this?
> > I took master into my branch this morning.
> >
> >
> > [INFO]
> > [INFO] --- maven-site-plugin:3.4:site (default-site) @ site-book ---
> > [INFO] Relativizing decoration links with respect to project URL:
> > https://metron.apache.org/
> > [INFO] Rendering site with 
> > org.apache.maven.skins:maven-fluido-skin:jar:1.3.0
>
> > skin.
> > [INFO] Rendering 65 Doxia documents: 65 markdown
> >
> >
> > That is the last output.
> >
> >
> >
>
>

Re: Site-Book: mvn site never completes

[Justin Leet]

Looks like the table of environment variables kills it.  An incredibly ugly
(like actually looks horrible) hack is to just wrap it in a code block and
let it be ugly. Anybody know a less horrible workaround?  I'll throw up a
PR with this and see if anyone has any ideas, too.  At least it'll let
people see there's a way to get around it if they need to.

I'm guessing we're probably hitting
https://issues.apache.org/jira/browse/DOXIA-554.

Long story short, the doxia-module-markdown depends on a deprecated
library, pegdown, which notably has performance issues (and can result in
hangs or infinite loops).  It'll be fixed, but not until 1.8 (we're on 1.6,
current is 1.7)

As an example, my favorite comment is "The issue with pegdown is more
serious than that. The simplest pathological case is consecutive
[[[, each open bracket doubles parsing time. 18 of
these take 1.3 seconds to parse."

On Tue, May 9, 2017 at 11:34 AM, Justin Leet <justinjl...@gmail.com> wrote:

> https://github.com/apache/incubator-metron/blob/
> 716bda326f26b60c98318cdc1cc949b351a0b71a/metron-interface/
> metron-rest/README.md appears to be the problem.  I can run the site
> successfully on the commit before, and this commit hangs.
>
> Just glancing through, the file appears fine on Github, but there might be
> an issue with the rendering the docs.  I'm not really familiar enough with
> what's going on under the hood to know what the problem is, and I'd have to
> dig in more to have an informed opinion.
>
> Can someone double check me that 716bda32 fails and 38d26d43 succeeds?
>
> On Tue, May 9, 2017 at 11:26 AM, Otto Fowler <ottobackwa...@gmail.com>
> wrote:
>
>> I also did a >git checkout -b test-site apache/master
>> same issue.
>>
>>
>> On May 9, 2017 at 11:09:32, Otto Fowler (ottobackwa...@gmail.com) wrote:
>>
>> Hi,
>> I am running mvn site for the site book, but it never completes.  I am
>> not sure if there is anything I can do to trouble shoot this?
>> I took master into my branch this morning.
>>
>>
>> [INFO]
>> [INFO] --- maven-site-plugin:3.4:site (default-site) @ site-book ---
>> [INFO] Relativizing decoration links with respect to project URL:
>> https://metron.apache.org/
>> [INFO] Rendering site with org.apache.maven.skins:maven-fluido-skin:jar:1.3.0
>> skin.
>> [INFO] Rendering 65 Doxia documents: 65 markdown
>>
>>
>> That is the last output.
>>
>>
>>
>

Re: Site-Book: mvn site never completes

[Justin Leet]

Just merged it

On Tue, May 9, 2017 at 4:26 PM, Otto Fowler <ottobackwa...@gmail.com> wrote:

> Who is going to merge this?
>
>
> On May 9, 2017 at 15:21:07, Justin Leet (justinjl...@gmail.com) wrote:
>
> https://github.com/apache/incubator-metron/pull/575
>
> Somewhat spiffy, since I took the 30 second task of splitting the table
> apart so the parser is less trash and it worked.
>
> On Tue, May 9, 2017 at 3:14 PM, Otto Fowler <ottobackwa...@gmail.com>
> wrote:
>
>> If putting it in a block lets it complete, lets do this and get master
>> going.
>> and we can enter a ticket about making it spiffy like.
>>
>>
>>
>>
>> On May 9, 2017 at 15:05:17, Justin Leet (justinjl...@gmail.com) wrote:
>>
>> Looks like the table of environment variables kills it. An incredibly ugly
>> (like actually looks horrible) hack is to just wrap it in a code block and
>> let it be ugly. Anybody know a less horrible workaround? I'll throw up a
>> PR with this and see if anyone has any ideas, too. At least it'll let
>> people see there's a way to get around it if they need to.
>>
>> I'm guessing we're probably hitting
>> https://issues.apache.org/jira/browse/DOXIA-554.
>>
>> Long story short, the doxia-module-markdown depends on a deprecated
>> library, pegdown, which notably has performance issues (and can result in
>> hangs or infinite loops). It'll be fixed, but not until 1.8 (we're on 1.6,
>> current is 1.7)
>>
>> As an example, my favorite comment is "The issue with pegdown is more
>> serious than that. The simplest pathological case is consecutive
>> [[[, each open bracket doubles parsing time. 18 of
>> these take 1.3 seconds to parse."
>>
>> On Tue, May 9, 2017 at 11:34 AM, Justin Leet <justinjl...@gmail.com>
>> wrote:
>>
>> > https://github.com/apache/incubator-metron/blob/
>> > 716bda326f26b60c98318cdc1cc949b351a0b71a/metron-interface/
>> > metron-rest/README.md appears to be the problem. I can run the site
>> > successfully on the commit before, and this commit hangs.
>> >
>> > Just glancing through, the file appears fine on Github, but there might
>> be
>> > an issue with the rendering the docs. I'm not really familiar enough
>> with
>> > what's going on under the hood to know what the problem is, and I'd
>> have to
>> > dig in more to have an informed opinion.
>> >
>> > Can someone double check me that 716bda32 fails and 38d26d43 succeeds?
>> >
>> > On Tue, May 9, 2017 at 11:26 AM, Otto Fowler <ottobackwa...@gmail.com>
>> > wrote:
>> >
>> >> I also did a >git checkout -b test-site apache/master
>> >> same issue.
>> >>
>> >>
>> >> On May 9, 2017 at 11:09:32, Otto Fowler (ottobackwa...@gmail.com)
>> wrote:
>> >>
>> >> Hi,
>> >> I am running mvn site for the site book, but it never completes. I am
>> >> not sure if there is anything I can do to trouble shoot this?
>> >> I took master into my branch this morning.
>> >>
>> >>
>> >> [INFO]
>> >> [INFO] --- maven-site-plugin:3.4:site (default-site) @ site-book ---
>> >> [INFO] Relativizing decoration links with respect to project URL:
>> >> https://metron.apache.org/
>> >> [INFO] Rendering site with org.apache.maven.skins:maven-f
>> luido-skin:jar:1.3.0
>> >> skin.
>> >> [INFO] Rendering 65 Doxia documents: 65 markdown
>> >>
>> >>
>> >> That is the last output.
>> >>
>> >>
>> >>
>> >
>>
>>
>

Re: [DISCUSS] Code Style

[Justin Leet]

I'll have a PR up a little later this week once I have a chance to sit down
and test/organize notes a little more.

On Mon, May 8, 2017 at 10:04 PM, zeo...@gmail.com <zeo...@gmail.com> wrote:

> +1 definitely a good idea
>
> On Mon, May 8, 2017, 9:07 PM Michael Miklavcic <
> michael.miklav...@gmail.com>
> wrote:
>
> > +1 Justin. Thanks.
> >
> > On Mon, May 8, 2017 at 2:55 PM, Matt Foley <mfo...@hortonworks.com>
> wrote:
> >
> > > +1.  I originally suggested the Sun style as a starting point, and I
> find
> > > Justin’s arguments convincing, especially if there is a pre-packaged
> > Google
> > > style checking plug-in.
> > > --Matt
> > >
> > > On 5/8/17, 9:47 AM, "Kyle Richardson" <kylerichards...@gmail.com>
> wrote:
> > >
> > > +1 Thanks, Justin. I'm all for adopting the Google style guide.
> > >
> > > -Kyle
> > >
> > > On Mon, May 8, 2017 at 10:24 AM, Justin Leet <
> justinjl...@gmail.com>
> > > wrote:
> > >
> > > > Sigh, the message was hidden below the fold.
> > > >
> > > > Part of what I've been playing around with is doing this.  It's
> > > pretty easy
> > > > to set up both autoformatting and warnings in idea.  There's a
> > > checkstyle
> > > > plugin, and you can just import the appropriate file on each.
> As a
> > > fair
> > > > warning, until we do the blanket reformatting there will be a lot
> > of
> > > > warning highlighting in a lot of the Java files.
> > > >
> > > > I'll be documenting that as part of the ticket (regardless of
> > > checkstyle
> > > > configuration specifics), and I'll also update the wiki (or point
> > to
> > > links
> > > > for it, I believe Checkstyle's docs walk through how to do it) as
> > > part of
> > > > the exercise.
> > > >
> > > > Sorry about spamming the list.
> > > >
> > > > On Mon, May 8, 2017 at 10:21 AM, Justin Leet <
> > justinjl...@gmail.com>
> > > > wrote:
> > > >
> > > > > Accidentally sent this to only Otto, instead of the whole
> group.
> > > > >
> > > > > Part of what I've been playing around with is doing this.  It's
> > > pretty
> > > > > easy to set up both autoformatting and warnings in idea.
> > There's a
> > > > > checkstyle plugin, and you can just import the appropriate file
> > on
> > > each.
> > > > > As a fair warning, until we do the blanket reformatting there
> > will
> > > be a
> > > > lot
> > > > > of warning highlighting in a lot of the Java files.
> > > > >
> > > > > I'll be documenting that as part of the ticket (regardless of
> > > checkstyle
> > > > > configuration specifics), and I'll also update the wiki (or
> point
> > > to
> > > > links
> > > > > for it, I believe Checkstyle's docs walk through how to do it)
> as
> > > part of
> > > > > the exercise.
> > > > >
> > > > > On Mon, May 8, 2017 at 9:40 AM, Otto Fowler <
> > > ottobackwa...@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> +1.
> > > > >> Does anyone have an idea setup for this?
> > > > >>
> > > > >>
> > > > >> On May 8, 2017 at 09:29:15, Justin Leet (
> justinjl...@gmail.com)
> > > wrote:
> > > > >>
> > > > >> I've been taking a look at setting up checkstyle per
> > > > >> https://issues.apache.org/jira/browse/METRON-746.
> > > > >>
> > > > >> Given that we don't actually enforce any style right now
> (saying
> > > we use
> > > > >> Sun's is not the same as enforcing it!), and plan to do a
> > blanket
> > > format
> > > > >> after we get checkstyle setup, this seems like a good
> > opportunity
> > > to
> > > > >> discuss what we want to actually implement.
> > > > >>
> > > > >> In particular, I'm proposing moving to Google's code style
> guide
> >

Re: [DISCUSS] Code Style

[Justin Leet]

https://github.com/apache/incubator-metron/pull/577 is up, based on this
discussion.  It includes instructions for setting up both warnings in
IntelliJ and autoformatting settings. They're a bit reconstructed after the
fact, so if anything is wrong, let me know and I'll update them.

https://issues.apache.org/jira/browse/METRON-747 can follow more or less
whenever.  It's pretty easy to do a bulk reformat of only Java files.  Like
I said before, we probably want to regenerate anything autogenerated
afterwards (i.e. anything resulting from the Stellar grammar), since I'm
not convinced that they'll actually be ignored.  Checkstyle supports a
couple different ways of handling warnings, so the way those files use
might not actually work as expected.  I'd rather just blanket format,
regenerated, and then call it.

I am inclined to not turn on enforcement of anything until we think through
exactly how we want it to work (e.g. handling just avoiding introducing new
errors vs just not allowing any errors vs. setting max errors per module).
I'm not sure how other projects handle this, and we probably want to look
into it.

On Tue, May 9, 2017 at 6:07 AM, Justin Leet <justinjl...@gmail.com> wrote:

> I'll have a PR up a little later this week once I have a chance to sit
> down and test/organize notes a little more.
>
> On Mon, May 8, 2017 at 10:04 PM, zeo...@gmail.com <zeo...@gmail.com>
> wrote:
>
>> +1 definitely a good idea
>>
>> On Mon, May 8, 2017, 9:07 PM Michael Miklavcic <
>> michael.miklav...@gmail.com>
>> wrote:
>>
>> > +1 Justin. Thanks.
>> >
>> > On Mon, May 8, 2017 at 2:55 PM, Matt Foley <mfo...@hortonworks.com>
>> wrote:
>> >
>> > > +1.  I originally suggested the Sun style as a starting point, and I
>> find
>> > > Justin’s arguments convincing, especially if there is a pre-packaged
>> > Google
>> > > style checking plug-in.
>> > > --Matt
>> > >
>> > > On 5/8/17, 9:47 AM, "Kyle Richardson" <kylerichards...@gmail.com>
>> wrote:
>> > >
>> > > +1 Thanks, Justin. I'm all for adopting the Google style guide.
>> > >
>> > > -Kyle
>> > >
>> > > On Mon, May 8, 2017 at 10:24 AM, Justin Leet <
>> justinjl...@gmail.com>
>> > > wrote:
>> > >
>> > > > Sigh, the message was hidden below the fold.
>> > > >
>> > > > Part of what I've been playing around with is doing this.  It's
>> > > pretty easy
>> > > > to set up both autoformatting and warnings in idea.  There's a
>> > > checkstyle
>> > > > plugin, and you can just import the appropriate file on each.
>> As a
>> > > fair
>> > > > warning, until we do the blanket reformatting there will be a
>> lot
>> > of
>> > > > warning highlighting in a lot of the Java files.
>> > > >
>> > > > I'll be documenting that as part of the ticket (regardless of
>> > > checkstyle
>> > > > configuration specifics), and I'll also update the wiki (or
>> point
>> > to
>> > > links
>> > > > for it, I believe Checkstyle's docs walk through how to do it)
>> as
>> > > part of
>> > > > the exercise.
>> > > >
>> > > > Sorry about spamming the list.
>> > > >
>> > > > On Mon, May 8, 2017 at 10:21 AM, Justin Leet <
>> > justinjl...@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Accidentally sent this to only Otto, instead of the whole
>> group.
>> > > > >
>> > > > > Part of what I've been playing around with is doing this.
>> It's
>> > > pretty
>> > > > > easy to set up both autoformatting and warnings in idea.
>> > There's a
>> > > > > checkstyle plugin, and you can just import the appropriate
>> file
>> > on
>> > > each.
>> > > > > As a fair warning, until we do the blanket reformatting there
>> > will
>> > > be a
>> > > > lot
>> > > > > of warning highlighting in a lot of the Java files.
>> > > > >
>> > > > > I'll be documenting that as part of the ticket (regardless of
>> > > checkstyle
>> > > > > configuration specifics), and I'll also update the wiki (or
>&

Re: [DISCUSS] Code Style

[Justin Leet]

Absolutely.  I am definitely not proposing that for right away.  That's
probably a separate discuss thread anyway as we get closer to make sure we
air out any concerns and make sure we aren't leaving a lot of PRs with
difficult merges and so on.

On Tue, May 9, 2017 at 7:27 PM, Matt Foley <ma...@apache.org> wrote:

> Could I ask please that any major reformat wait until the Metron 0.4.0
> release is out?
> Such across-the-board changes usually induce instability for a while, no
> matter how carefully done.
>
> Thanks,
> --Matt
> (with RM hat on)
>
> On 5/9/17, 4:12 PM, "Justin Leet" <justinjl...@gmail.com> wrote:
>
> https://github.com/apache/incubator-metron/pull/577 is up, based on
> this
> discussion.  It includes instructions for setting up both warnings in
> IntelliJ and autoformatting settings. They're a bit reconstructed
> after the
> fact, so if anything is wrong, let me know and I'll update them.
>
> https://issues.apache.org/jira/browse/METRON-747 can follow more or
> less
> whenever.  It's pretty easy to do a bulk reformat of only Java files.
> Like
> I said before, we probably want to regenerate anything autogenerated
> afterwards (i.e. anything resulting from the Stellar grammar), since
> I'm
> not convinced that they'll actually be ignored.  Checkstyle supports a
> couple different ways of handling warnings, so the way those files use
> might not actually work as expected.  I'd rather just blanket format,
> regenerated, and then call it.
>
> I am inclined to not turn on enforcement of anything until we think
> through
> exactly how we want it to work (e.g. handling just avoiding
> introducing new
> errors vs just not allowing any errors vs. setting max errors per
> module).
>     I'm not sure how other projects handle this, and we probably want to
> look
> into it.
>
> On Tue, May 9, 2017 at 6:07 AM, Justin Leet <justinjl...@gmail.com>
> wrote:
>
> > I'll have a PR up a little later this week once I have a chance to
> sit
> > down and test/organize notes a little more.
> >
> > On Mon, May 8, 2017 at 10:04 PM, zeo...@gmail.com <zeo...@gmail.com>
> > wrote:
> >
> >> +1 definitely a good idea
> >>
> >> On Mon, May 8, 2017, 9:07 PM Michael Miklavcic <
> >> michael.miklav...@gmail.com>
> >> wrote:
> >>
> >> > +1 Justin. Thanks.
> >> >
> >> > On Mon, May 8, 2017 at 2:55 PM, Matt Foley <
> mfo...@hortonworks.com>
> >> wrote:
> >> >
> >> > > +1.  I originally suggested the Sun style as a starting point,
> and I
> >> find
> >> > > Justin’s arguments convincing, especially if there is a
> pre-packaged
> >> > Google
> >> > > style checking plug-in.
> >> > > --Matt
> >> > >
> >> > > On 5/8/17, 9:47 AM, "Kyle Richardson" <
> kylerichards...@gmail.com>
> >> wrote:
> >> > >
> >> > > +1 Thanks, Justin. I'm all for adopting the Google style
> guide.
> >> > >
> >> > > -Kyle
> >> > >
> >> > > On Mon, May 8, 2017 at 10:24 AM, Justin Leet <
> >> justinjl...@gmail.com>
> >> > > wrote:
> >> > >
> >> > > > Sigh, the message was hidden below the fold.
> >> > > >
> >> > > > Part of what I've been playing around with is doing
> this.  It's
> >> > > pretty easy
> >> > > > to set up both autoformatting and warnings in idea.
> There's a
> >> > > checkstyle
> >> > > > plugin, and you can just import the appropriate file on
> each.
> >> As a
> >> > > fair
> >> > > > warning, until we do the blanket reformatting there will
> be a
> >> lot
> >> > of
> >> > > > warning highlighting in a lot of the Java files.
> >> > > >
> >> > > > I'll be documenting that as part of the ticket
> (regardless of
> >> > > checkstyle
> >> > > > configuration specifics), and I'll also update the wiki
> (or
> >> point
> >> > to
> >> > > links
> >> > > > for it, I bel

Re: Site-Book: mvn site never completes

[Justin Leet]

https://github.com/apache/incubator-metron/blob/716bda326f26b60c98318cdc1cc949b351a0b71a/metron-interface/metron-rest/README.md
appears to be the problem.  I can run the site successfully on the commit
before, and this commit hangs.

Just glancing through, the file appears fine on Github, but there might be
an issue with the rendering the docs.  I'm not really familiar enough with
what's going on under the hood to know what the problem is, and I'd have to
dig in more to have an informed opinion.

Can someone double check me that 716bda32 fails and 38d26d43 succeeds?

On Tue, May 9, 2017 at 11:26 AM, Otto Fowler 
wrote:

> I also did a >git checkout -b test-site apache/master
> same issue.
>
>
> On May 9, 2017 at 11:09:32, Otto Fowler (ottobackwa...@gmail.com) wrote:
>
> Hi,
> I am running mvn site for the site book, but it never completes.  I am not
> sure if there is anything I can do to trouble shoot this?
> I took master into my branch this morning.
>
>
> [INFO]
> [INFO] --- maven-site-plugin:3.4:site (default-site) @ site-book ---
> [INFO] Relativizing decoration links with respect to project URL:
> https://metron.apache.org/
> [INFO] Rendering site with org.apache.maven.skins:maven-fluido-skin:jar:1.3.0
> skin.
> [INFO] Rendering 65 Doxia documents: 65 markdown
>
>
> That is the last output.
>
>
>

Re: [DISCUSS] Mutation of Indexed Data

[Justin Leet]

First off, I agree with the characteristics.

For the data stores, we'll need to be able to make sure we can actually
handle the collapsing of the updates into a single view.  Casey mentioned
making the long term stores transparent, but there's potentially work for
the near real time stores: we need to make sure we actually do updates,
rather than create new docs that aren't linked to the old ones. This should
be entirely transparent and handled by a service layer, rather than
anything hardcoded to a datastore.

For ES at least, the only way to do this is to retrieve, mutate it, and
then reindex (even the updates API does that dance under the hood for you,
and since we're potentially doing non trivial changes we might need to
manage it ourselves).  This implies the existence of a key, even if one
isn't enforced by ES (Which I don't believe it will be).  We need to be
able to grab the doc(s?) to be updated, not end up with similar ones that
shouldn't be mutated.  I assume this is also true (at least the
generalities) of Solr as well.

In concert with your other thread, couldn't part of this key end up being
metadata (either user defined or environment defined)?  For example, in a
situation where customer id is applied as metadata, it's possible two
customers feed off the same datasource, but may need to mutate
independently.  At this point, we have metadata that is effectively keyed.
We don't want to update both docs, but there's not a real way to
distinguish them.  And maybe that's something we push off for the short
term, but it seems potentially nontrivial.

In terms of consistency, I'd definitely agree that the long-term storage
can be eventually consistent.  Any type of bulk spelunking, Spark jobs,
dashboarding, etc. shouldn't need up to the millisecond data.

Basically, I'm thinking the real time store is the snapshot of current
state, and the long term store is the full record complete with the lineage
history.

I'm also interested in people's opinions on how we want to manage HDFS.
Assuming we do use HBase to store our updates, that means that every HDFS
op has to join onto that HBase table to get any updates that HDFS is
missing (unless we implement some writeback and merge for HDFS data).  I'm
worried that our two datastores are really: ES, HDFS+HBase.  And that
keeping that data actually synced to end users is going to be painful.

Justin


On Wed, Jun 21, 2017 at 10:18 PM, Simon Elliston Ball <
si...@simonellistonball.com> wrote:

> I'd say that was an excellent set of requirements (very similar to the one
> we arrived on with the last discuss thread on this)
>
> My vote remains a transaction log in hbase given the relatively low volume
> (human scale) i would not expect this to need anything fancy like
> compaction into hdfs state, but that does make a good argument for a long
> term dataframe solution for spark, with a short term stop gap using a
> joined data frame and shc.
>
> Simon
>
> Sent from my iPhone
>
> > On 22 Jun 2017, at 05:11, Otto Fowler  wrote:
> >
> > Can you clarify what data stores are at play here?
> >
> >
> > On June 21, 2017 at 17:07:42, Casey Stella (ceste...@gmail.com) wrote:
> >
> > Hi All,
> >
> > I know we've had a couple of these already, but we're due for another
> > discussion of a sensible approach to mutating indexed data. The
> motivation
> > for this is users will want to update fields to correct and augment data.
> > These corrections are invaluable for things like feedback for ML models
> or
> > just plain providing better context when evaluating alerts, etc.
> >
> > Rather than posing a solution, I'd like to pose the characteristics of a
> > solution and we can fight about those first. ;)
> >
> > In my mind, the following are the characteristics that I'd look for:
> >
> > - Changes should be considered additional or replacement fields for
> > existing fields
> > - Changes need to be available in the web view in near real time (on the
> > order of milliseconds)
> > - Changes should be available in the batch view
> > - I'd be ok with eventually consistent with the web view, thoughts?
> > - Changes should have lineage preserved
> > - Current value is the optimized path
> > - Lineage search is the less optimized path
> > - If HBase is part of a solution
> > - maintain a scan-free solution
> > - maintain a coprocessor-free solution
> >
> > Most of what I've thought of is something along the lines:
> >
> > - Diffs are stored in columns in a HBase row(s)
> > - row: GUID:current would have one column with the current
> > representation
> > - row: GUID:lineage would have an ordered set of columns representing
> > the lineage diffs
> > - Mutable indices is directly updated (e.g. solr or ES)
> > - We'd probably want to provide transparent read support downstream
> > which supports merging for batch read:
> > - a spark dataframe
> > - a hive serde
> >
> > What I'd like to get out of this discussion is an architecture document
> > with a 

Re: [DISCUSS] Mutation of Indexed Data

[Justin Leet]

Thanks, Jon, that looks like it should work for the key.  I didn't realize
that guid got handled that way, which makes life much easier there.  Almost
like we already needed to identify messages or something.  At the point we
should be good, since we can easily retrieve, update, put on it.

We'll also need to make sure any long term storage solution also uses it.


On Thu, Jun 22, 2017 at 12:52 PM, zeo...@gmail.com <zeo...@gmail.com> wrote:

> The key should be a solved problem as of METRON-765
> <https://github.com/apache/metron/commit/27b0d6e31de94317b085766349a892
> 395f0d3309>,
> right?  It provides a single key for a given message that globally stored
> with the message, regardless of where/how.
>
> Jon
>
> On Thu, Jun 22, 2017 at 9:01 AM Justin Leet <justinjl...@gmail.com> wrote:
>
> > First off, I agree with the characteristics.
> >
> > For the data stores, we'll need to be able to make sure we can actually
> > handle the collapsing of the updates into a single view.  Casey mentioned
> > making the long term stores transparent, but there's potentially work for
> > the near real time stores: we need to make sure we actually do updates,
> > rather than create new docs that aren't linked to the old ones. This
> should
> > be entirely transparent and handled by a service layer, rather than
> > anything hardcoded to a datastore.
> >
> > For ES at least, the only way to do this is to retrieve, mutate it, and
> > then reindex (even the updates API does that dance under the hood for
> you,
> > and since we're potentially doing non trivial changes we might need to
> > manage it ourselves).  This implies the existence of a key, even if one
> > isn't enforced by ES (Which I don't believe it will be).  We need to be
> > able to grab the doc(s?) to be updated, not end up with similar ones that
> > shouldn't be mutated.  I assume this is also true (at least the
> > generalities) of Solr as well.
> >
> > In concert with your other thread, couldn't part of this key end up being
> > metadata (either user defined or environment defined)?  For example, in a
> > situation where customer id is applied as metadata, it's possible two
> > customers feed off the same datasource, but may need to mutate
> > independently.  At this point, we have metadata that is effectively
> keyed.
> > We don't want to update both docs, but there's not a real way to
> > distinguish them.  And maybe that's something we push off for the short
> > term, but it seems potentially nontrivial.
> >
> > In terms of consistency, I'd definitely agree that the long-term storage
> > can be eventually consistent.  Any type of bulk spelunking, Spark jobs,
> > dashboarding, etc. shouldn't need up to the millisecond data.
> >
> > Basically, I'm thinking the real time store is the snapshot of current
> > state, and the long term store is the full record complete with the
> lineage
> > history.
> >
> > I'm also interested in people's opinions on how we want to manage HDFS.
> > Assuming we do use HBase to store our updates, that means that every HDFS
> > op has to join onto that HBase table to get any updates that HDFS is
> > missing (unless we implement some writeback and merge for HDFS data).
> I'm
> > worried that our two datastores are really: ES, HDFS+HBase.  And that
> > keeping that data actually synced to end users is going to be painful.
> >
> > Justin
> >
> >
> > On Wed, Jun 21, 2017 at 10:18 PM, Simon Elliston Ball <
> > si...@simonellistonball.com> wrote:
> >
> > > I'd say that was an excellent set of requirements (very similar to the
> > one
> > > we arrived on with the last discuss thread on this)
> > >
> > > My vote remains a transaction log in hbase given the relatively low
> > volume
> > > (human scale) i would not expect this to need anything fancy like
> > > compaction into hdfs state, but that does make a good argument for a
> long
> > > term dataframe solution for spark, with a short term stop gap using a
> > > joined data frame and shc.
> > >
> > > Simon
> > >
> > > Sent from my iPhone
> > >
> > > > On 22 Jun 2017, at 05:11, Otto Fowler <ottobackwa...@gmail.com>
> wrote:
> > > >
> > > > Can you clarify what data stores are at play here?
> > > >
> > > >
> > > > On June 21, 2017 at 17:07:42, Casey Stella (ceste...@gmail.com)
> wrote:
> > > >
> > > > Hi All,
> > > >
> > > > I know we've had a couple of these already, but we

Re: METRON-777

[Justin Leet]

I know I'd like people (probably myself included) to start digging in.

I'm in a bit of the same boat where I'll have to do some work (a lot?) to
familiarize myself better with everything to properly review it, but
hopefully if we have a few sets of eyes looking at it we'll get everything
squared away.  I plan on taking time this week to at least do a super high
level glance and get at least some suggestions in, so any needed iteration
can start sooner rather than later.

On Wed, May 31, 2017 at 3:51 PM, zeo...@gmail.com  wrote:

> I was wondering, is anybody planning to or currently taking a look at
> Metron 777?  I think this is a great contribution and very important to
> improving the usability of the platform (along with some of it's follow on
> PRs).  I would be happy to help with functional testing and security static
> code analysis but unfortunately I'm not familiar enough with the existing
> code base to give it a +1 based off of the architectural changes.
>
> Jon
> --
>
> Jon
>

[DISCUSS] Code Style

[Justin Leet]

I've been taking a look at setting up checkstyle per
https://issues.apache.org/jira/browse/METRON-746.

Given that we don't actually enforce any style right now (saying we use
Sun's is not the same as enforcing it!), and plan to do a blanket format
after we get checkstyle setup, this seems like a good opportunity to
discuss what we want to actually implement.

In particular, I'm proposing moving to Google's code style guide and away
from Sun's.

   - Google's style guide already address the (admittedly) minor issues we
   have with the Sun Style, in particular two spaces and line length (100 in
   Google's). I'd prefer to just use something built-in as much as possible,
   and it seems like Google's style is closer to what we're looking for out of
   the box.
   - Not only is it built in, the explanation for each rule (and where
   checkstyle falls short) actually exists.
  - http://checkstyle.sourceforge.net/google_style.html
  - http://checkstyle.sourceforge.net/sun_style.html
  - Storm uses it, so it makes our code's style more compatible with
   one of our core dependencies (and one we've had to dig into repeatedly
   during upgrades and such).
  - https://github.com/apache/storm/blob/master/pom.xml#L1100
  - I personally find Google's guide easier to read and digest.  I'm
   curious how other people feel.

For ease of comparison:
Sun's guide: http://www.oracle.com/technetwork/java/codeconvtoc-136057.html
Google's guide: https://google.github.io/styleguide/javaguide.html

Re: [DISCUSS] Code Style

[Justin Leet]

Accidentally sent this to only Otto, instead of the whole group.

Part of what I've been playing around with is doing this.  It's pretty easy
to set up both autoformatting and warnings in idea.  There's a checkstyle
plugin, and you can just import the appropriate file on each.  As a fair
warning, until we do the blanket reformatting there will be a lot of
warning highlighting in a lot of the Java files.

I'll be documenting that as part of the ticket (regardless of checkstyle
configuration specifics), and I'll also update the wiki (or point to links
for it, I believe Checkstyle's docs walk through how to do it) as part of
the exercise.

On Mon, May 8, 2017 at 9:40 AM, Otto Fowler <ottobackwa...@gmail.com> wrote:

> +1.
> Does anyone have an idea setup for this?
>
>
> On May 8, 2017 at 09:29:15, Justin Leet (justinjl...@gmail.com) wrote:
>
> I've been taking a look at setting up checkstyle per
> https://issues.apache.org/jira/browse/METRON-746.
>
> Given that we don't actually enforce any style right now (saying we use
> Sun's is not the same as enforcing it!), and plan to do a blanket format
> after we get checkstyle setup, this seems like a good opportunity to
> discuss what we want to actually implement.
>
> In particular, I'm proposing moving to Google's code style guide and away
> from Sun's.
>
> - Google's style guide already address the (admittedly) minor issues we
> have with the Sun Style, in particular two spaces and line length (100 in
> Google's). I'd prefer to just use something built-in as much as possible,
> and it seems like Google's style is closer to what we're looking for out
> of
> the box.
> - Not only is it built in, the explanation for each rule (and where
> checkstyle falls short) actually exists.
> - http://checkstyle.sourceforge.net/google_style.html
> - http://checkstyle.sourceforge.net/sun_style.html
> - Storm uses it, so it makes our code's style more compatible with
> one of our core dependencies (and one we've had to dig into repeatedly
> during upgrades and such).
> - https://github.com/apache/storm/blob/master/pom.xml#L1100
> - I personally find Google's guide easier to read and digest. I'm
> curious how other people feel.
>
> For ease of comparison:
> Sun's guide: http://www.oracle.com/technetwork/java/codeconvtoc-
> 136057.html
> Google's guide: https://google.github.io/styleguide/javaguide.html
>
>

Re: Recent commit without JIRA in commit message

[Justin Leet]

To add some context, the problem here is that when I squashed METRON-799 it
became 3 commits, in order to keep proper attribution.  I missed having the
jira on the last of the 3 commits during the squash.  The other two commits
do still have it:

commit 41b5b1050cf29e5dcb5d0c36b5a9dbd1cafa745e
The MPack should function in a kerberized cluster (justinleet) closes
apache/incubator-metron#518

commit 55062fb79a85722c909906874e0772f51b18953d
METRON-799 The MPack should function in a kerberized cluster (dlyle via
justinleet) closes apache/incubator-metron#518

commit 187ef373a6cc0f89696349acc35ddeddbfd836ef
METRON-799 The MPack should function in a kerberized cluster
(justinleet) closes apache/incubator-metron#518

On Sun, Apr 30, 2017 at 8:03 PM, zeo...@gmail.com  wrote:

> It looks like METRON-799  >
> (#518 ) got commit
>  41b5b1050cf29e5dcb5d0c36b5a9dbd1cafa745e>
> without having the JIRA in the title.  Is this enough of a concern
> to reword the commit message?  I know there have been some concerns with
> changing ever changing history, no matter how small, but I figured I would
> bring it up to discuss.  Thanks,
>
> Jon
> --
>
> Jon
>

Re: Cloudtrail use case

[Justin Leet]

I totally forgot you added that.  100% think it belongs there.

On Fri, Oct 6, 2017 at 9:26 AM, Casey Stella <ceste...@gmail.com> wrote:

> There is actually a use-cases top level directory with worked examples in
> them.  They get picked up by the doc book too!  I'd suggest putting it
> there, thoughts?
>
> On Fri, Oct 6, 2017 at 8:44 AM, Nick Allen <n...@nickallen.org> wrote:
>
> > Yes, agreed, Justin.  I guess my main point to Laurens was meant to be
> that
> > the actual destination of the use case should be the least of our
> worries.
> > However Laurens wants to write it up will work. If you type it up, throw
> it
> > in an envelope, seal it with a stamp, and physically mail it to me, I
> will
> > make sure it lands in the right place. :)
> >
> >
> >
> > On Thu, Oct 5, 2017 at 9:20 PM Justin Leet <justinjl...@gmail.com>
> wrote:
> >
> > > I know we've had discussions about migrating stuff into docs before.
> It
> > > might be worth resurrecting a more use case focused version of that,
> > > instead of starting on the wiki.  I assume the end goal is availability
> > in
> > > the site-book, so even if it's not in a perfect place, I'd rather the
> > > effort be spent on making it pretty there.
> > >
> > > I think there's a few floating around that could use a home, so the
> > > discussion might make life easier for multiple things.  Some from the
> > wiki,
> > > some from random READMEs we could relocate and link, some from
> > > presentations and so on.
> > >
> > > Having said all that, I know discuss threads can take a few days to
> > > resolve, so wiki and then convert might be the lesser of two evils.
> > >
> > >
> > > On Thu, Oct 5, 2017 at 6:54 PM, Nick Allen <n...@nickallen.org> wrote:
> > >
> > > > We don't really have a location in the source code for use cases like
> > > this
> > > > right now.  But I think it is so important that we get use cases like
> > > this
> > > > published somewhere.  For now, you could add this to the Wiki.  Then
> > > later
> > > > on we can figure out how to handle that.
> > > >
> > > > On Thu, Oct 5, 2017 at 6:49 PM, Laurens Vets <laur...@daemon.be>
> > wrote:
> > > >
> > > > > On 2017-10-05 15:45, Laurens Vets wrote:
> > > > >
> > > > >> Hi,
> > > > >>
> > > > >> Would anyone be interested in adding a full AWS Cloudtrail use
> case
> > to
> > > > >> the Metron documentation? I would roughly consist of:
> > > > >> - Apache NiFi configuration to retrieve Cloudtrail logs from S3
> and
> > > > >> send it to Metron via Kafka.
> > > > >> - Complete Metron sensor configuration (enrichment, alerting,
> > etc...)
> > > > for
> > > > >> this.
> > > > >>
> > > > >
> > > > > Sent too soon :(
> > > > >
> > > > > If anyone would be interested in this documentation, where would
> add
> > > this
> > > > > in the source?
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Dropping support for elastic 2.x

[Justin Leet]

Forgot the link
https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-upgrade.html

On Wed, Oct 4, 2017 at 1:07 PM, Simon Elliston Ball <
si...@simonellistonball.com> wrote:

> The simplest option would probably be to upgrade the ES and then reindex
> from the HDFS store. Alternatively there are means to do inplace upgrades
> from 2.x to 5.x I believe.
>
> Simon
>
> > On 4 Oct 2017, at 18:05, Casey Stella  wrote:
> >
> > So, how would this work in an upgrade scenario that does not involve
> losing
> > the existing indexed data?
> >
> > On Wed, Oct 4, 2017 at 12:55 PM, Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> >> The client I'm currently working on moving towards would *not* be
> backwards
> >> compatible.
> >> https://www.elastic.co/guide/en/elasticsearch/client/java-
> >> rest/current/java-rest-high-compatibility.html
> >>
> >> "
> >> The High Level Client is guaranteed to be able to communicate with any
> >> Elasticsearch node running on the same major version and greater or
> equal
> >> minor version. It doesn’t need to be in the same minor version as the
> >> Elasticsearch nodes it communicates with, as it is forward compatible
> >> meaning that it supports communicating with later versions of
> Elasticsearch
> >> than the one it was developed for.
> >>
> >> The 5.6 client can communicate with any 5.6.x Elasticsearch node.
> Previous
> >> 5.x minor versions like 5.5.x, 5.4.x etc. are not (fully) supported.
> >> "
> >>
> >> Best,
> >> Mike
> >>
> >>
> >> On Wed, Oct 4, 2017 at 10:45 AM, Simon Elliston Ball <
> >> si...@simonellistonball.com> wrote:
> >>
> >>> A number of people are currently working on upgrading the ES support in
> >>> Metron to 5.x (including the clients, and the mpack managed install).
> >>>
> >>> Would anyone have any objections to dropping formal support for 2.x as
> a
> >>> result of this work? In theory the clients should be backward
> compatible
> >>> against older data stores, so metron could be upgraded without needing
> an
> >>> elastic upgrade.
> >>>
> >>> In practice, we would need to do pretty extensive testing and I
> wouldn’t
> >>> want us to have to code around long term support on older clients if
> >> no-one
> >>> in the community cares enough about the older ES. Do we think there is
> a
> >>> case to be made for maintaining long term support for older clients?
> >>>
> >>> Simon
> >>
>
>

Re: [DISCUSS] Dropping support for elastic 2.x

[Justin Leet]

ES should be upgradeable without wiping.  It's the client itself that isn't
backwards compatible.  It'll require both an upgrade of Metron and an ES
cluster.

On Wed, Oct 4, 2017 at 1:05 PM, Casey Stella  wrote:

> So, how would this work in an upgrade scenario that does not involve losing
> the existing indexed data?
>
> On Wed, Oct 4, 2017 at 12:55 PM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > The client I'm currently working on moving towards would *not* be
> backwards
> > compatible.
> > https://www.elastic.co/guide/en/elasticsearch/client/java-
> > rest/current/java-rest-high-compatibility.html
> >
> > "
> > The High Level Client is guaranteed to be able to communicate with any
> > Elasticsearch node running on the same major version and greater or equal
> > minor version. It doesn’t need to be in the same minor version as the
> > Elasticsearch nodes it communicates with, as it is forward compatible
> > meaning that it supports communicating with later versions of
> Elasticsearch
> > than the one it was developed for.
> >
> > The 5.6 client can communicate with any 5.6.x Elasticsearch node.
> Previous
> > 5.x minor versions like 5.5.x, 5.4.x etc. are not (fully) supported.
> > "
> >
> > Best,
> > Mike
> >
> >
> > On Wed, Oct 4, 2017 at 10:45 AM, Simon Elliston Ball <
> > si...@simonellistonball.com> wrote:
> >
> > > A number of people are currently working on upgrading the ES support in
> > > Metron to 5.x (including the clients, and the mpack managed install).
> > >
> > > Would anyone have any objections to dropping formal support for 2.x as
> a
> > > result of this work? In theory the clients should be backward
> compatible
> > > against older data stores, so metron could be upgraded without needing
> an
> > > elastic upgrade.
> > >
> > > In practice, we would need to do pretty extensive testing and I
> wouldn’t
> > > want us to have to code around long term support on older clients if
> > no-one
> > > in the community cares enough about the older ES. Do we think there is
> a
> > > case to be made for maintaining long term support for older clients?
> > >
> > > Simon
> >
>

Re: [DISCUSS] Upgrading Elasticsearch from 2.x to 5.x

[Justin Leet]

Do we intend on (or have interest in) supporting ES across major version
for a given version of Metron?  I'm not convinced it's worth the work of
using the low level client.

This really only seems useful for ES clusters that are being used outside
Metron and need to be on a different ES major version. Is that a use case
we want/need to support? I'm willing to bet it's significantly more work
and means we're modifying queries and even templates/mappings based on what
ES version we're interacting with (e.g. meta alerts in 5.x can exploit a
query param to not screw around with the mapping, but that param doesn't
exist in 2.x). At that point, we're either back to writing for ES 2.x or
writing for every version of ES.

Unless that's something we have a demand for (or someone else persuades me
otherwise), I'm in favor of using the high level client.  It seems like
it'd be easier to migrate to also, given the similarities API-wise to the
current client we're using.

On Thu, Oct 5, 2017 at 1:52 PM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> I think it might help the discussion to share my impressions of looking
> over the new API recommendations from ES. I've summarized some info
> provided by ES back in December 2016 regarding the reasons for switching to
> a new client model. [1]
>
> *Summary points:*
>
> Pre-5.x had Java API - binary exchange format used for node-to-node
> communications.
> In 5.x a low level REST API was added. Now there's also a high level REST
> client that handles request marshalling and response un-marshalling.
>
> *Benefits of existing Java API*
>
>1. Theoretically faster - binary format, no JSON parsing
>2. Hardened, used for internal ES node to node communications
>
> *Cons of Java API*
>
>1. Benchmarks show it's not really that much faster.
>2. Backwards compatibility - Java API changes often.
>3. Upgrades more challenging - need to refactor client code for new and
>deprecated features.
>4. Minor releases may contain breaking changes in the Java API
>5. Client and server *should* be on same JVM version (not as important
>post 2.x, but still potentially necessary bc of serialization w/binary
>format)
>6. Requires dependency on the entire elasticsearch server in order to
>use the client. We end up shading jars.
>
> *Benefits of new REST API*
>
>1. Upgrades
>   1. Breaking changes only made in major releases - "We are very
>   careful with backwards compatibility on the REST layer where breaking
>   changes are made only in major releases."
>   2. "The REST interface is much more stable and can be upgraded out of
>   step with the Elasticsearch cluster."
>2. REST client and server can be on different JVM's
>3. Dependencies for the low level client are very slim. No need for
>shading.
>4. The RestHighLevelClient supports the same request and response
>objects as the TransportClient
>5. Can be secured via HTTPS
>
> There are some additional benefits to the new API, however they depend on
> whether we choose to go with the high or low level client. More comments
> below.
>
> *Cons of new API*
>
>1. Dependencies - The high level client still requires the full ES
>dependency, though this will slim down in future releases.
>
> *Other comments specific to Metron*
>
> There's a question of whether we should use the low or high level REST
> client. The main differences between the two are how they handle lib
> dependencies and marshaling/unmarshaling. The low level client cleans up
> the dependencies dramatically, whereas the high level client still requires
> you to depend on elasticsearch core. On the other hand, the low level
> client does no work to handle marshaling/unmarshaling the
> requests/responses from the HTTP calls while the high level client handles
> this for you and exposes api-specific methods. The high level client
> accepts the same request arguments as the TransportClient and returns the
> same response objects. One more thing to note is that the low level client
> claims to be compatible with all versions of ES whereas the high level
> client appears to be only major version compatible.
>
> "The 5.6 client can communicate with any 5.6.x Elasticsearch node. Previous
> 5.x minor versions like 5.5.x, 5.4.x etc. are not (fully) supported." [2]
>
> Just as an example, here's a simple comparison of an index request in the
> low and high level API's.
>
> *Low Level*
>
> Map params = Collections.emptyMap();
> String jsonString = "{" +
> "\"user\":\"kimchy\"," +
> "\"postDate\":\"2013-01-30\"," +
> "\"message\":\"trying out Elasticsearch\"" +
> "}";
> HttpEntity entity = new NStringEntity(jsonString,
> ContentType.APPLICATION_JSON);
> Response response = restClient.performRequest("PUT", "/posts/doc/1",
> params, entity);
>
> *High Level*
>
> IndexRequest indexRequest = new IndexRequest("posts", "doc", "1")

Quick Dev

[Justin Leet]

So what are we going to do with Quick Dev?  I'm pretty sure everybody's
been using full dev for awhile now (and quick dev is probably broken since
I'm sure we haven't been regularly updating it).

I just realized our website links to a wiki page that says to use quick
dev.  Given that quick dev is broken or behind or both, this is going to be
incredibly confusing and misleading to anyone wandering in.

A short term fix is to just point that wiki page to full dev, so we aren't
at least broken to anyone coming in.

After that we should figure out what we're doing with quick and full dev
and take care of whatever needs to be done.

Thoughts?

Justin

Re: [DISCUSS] Meta alert Elasticsearch new template requirement ramifications

[Justin Leet]

I put up a preliminary PR at https://github.com/apache/metron/pull/780. As
noted there, this should almost certainly be under a different heading, and
possibly a different README, so feel free to chime in on that.  Primary
goal is to make sure the content makes sense and get adjustments as needed.

On Fri, Sep 29, 2017 at 10:10 AM, Otto Fowler <ottobackwa...@gmail.com>
wrote:

> We can also consider this when thinking about creating parsers with
> archetypes that contain ‘default’
> elasticsearch templates.
>
>
>
> On September 29, 2017 at 10:00:03, Justin Leet (justinjl...@gmail.com)
> wrote:
>
> As part of building a backend for meta-alerts (
> https://github.com/apache/metron/pull/734), there's an additional
> requirement for the Elasticsearch templates for new sensors. Although
> seemingly minor, this should be called out explicitly because of the wider
> implications of leaving it out of ANY sensor. Specifically, this can
> result in the UI and other queries not returning results, because
> Elasticsearch throws an error.
>
> A nested "alert" field must be added in the form:
>
> "alert": {
> "type": "nested"
> },
>
> This results from Elasticsearch 2.x requiring the type of searches that
> meta alerts wrap to have the fields exist in all indices or the query
> fails.
>
> In Elasticsearch 5.x, there is a per query parameter that can be set to
> avoid this: see
> https://www.elastic.co/guide/en/elasticsearch/reference/
> current/search-request-sort.html#_ignoring_unmapped_fields
> .
>
> The obvious short-term thing that needs to happen with this is improved
> documentation. A ticket for documenting this (with some more specific
> details and what the error looks like is at
> https://issues.apache.org/jira/browse/METRON-1220). Where should that
> documentation live? It seems like our documentation in general around this
> type of stuff is a little lacking. Right now, I'm putting a prelim version
> into the metron-indexing README, but either now or as a followon we should
> have a more robust version that lays requirements for things like
> templating.
>
> There are a couple options I see to address this more robustly.
> 1) Just upgrade to ES 5.x and modify the meta alert query appropriately. A
> beginning basis for this change exists in
> https://github.com/apache/metron/pull/619. More works needs to happen
> there to finalize it
> 2) Add in ZK hooks for when a new sensor is added. The DAOs could receive
> word that a new sensor has been added in ZK and then build and submit the
> modified template itself. This (or a variant) is probably something that
> should happen anyway, in order to be more consistent with the other pieces
> that monitor and act on ZK updates.
> 3) There may be some mitigating that can be done here, e.g. if a query
> fails with the relevant error, rerun a different variant that may not hit
> the meta alerts, but doesn't fail as extravagantly.
>
> Is there a preference on either where the new documentation lives? And is
> there a preference on how we address this going forward?
>
> Justin
>
>

[DISCUSS] Meta alert Elasticsearch new template requirement ramifications

[Justin Leet]

As part of building a backend for meta-alerts (
https://github.com/apache/metron/pull/734), there's an additional
requirement for the Elasticsearch templates for new sensors.  Although
seemingly minor, this should be called out explicitly because of the wider
implications of leaving it out of ANY sensor.  Specifically, this can
result in the UI and other queries not returning results, because
Elasticsearch throws an error.

A nested "alert" field must be added in the form:

"alert": {
   "type": "nested"
},

This results from Elasticsearch 2.x requiring the type of searches that
meta alerts wrap to have the fields exist in all indices or the query fails.

In Elasticsearch 5.x, there is a per query parameter that can be set to
avoid this: see
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html#_ignoring_unmapped_fields
.

The obvious short-term thing that needs to happen with this is improved
documentation.  A ticket for documenting this (with some more specific
details and what the error looks like is at
https://issues.apache.org/jira/browse/METRON-1220).  Where should that
documentation live?  It seems like our documentation in general around this
type of stuff is a little lacking.  Right now, I'm putting a prelim version
into the metron-indexing README, but either now or as a followon we should
have a more robust version that lays requirements for things like
templating.

There are a couple options I see to address this more robustly.
1) Just upgrade to ES 5.x and modify the meta alert query appropriately. A
beginning basis for this change exists in
https://github.com/apache/metron/pull/619.  More works needs to happen
there to finalize it
2) Add in ZK hooks for when a new sensor is added.  The DAOs could receive
word that a new sensor has been added in ZK and then build and submit the
modified template itself.  This (or a variant) is probably something that
should happen anyway, in order to be more consistent with the other pieces
that monitor and act on ZK updates.
3) There may be some mitigating that can be done here, e.g. if a query
fails with the relevant error, rerun a different variant that may not hit
the meta alerts, but doesn't fail as extravagantly.

Is there a preference on either where the new documentation lives?  And is
there a preference on how we address this going forward?

Justin

Re: Quick Dev

[Justin Leet]

Wiki updated.  It now points to full dev and the link just says "Dev
Platform"

On Fri, Oct 6, 2017 at 8:39 AM, Nick Allen <n...@nickallen.org> wrote:

> +1 To killing Quick Dev and updating the Wiki.  Quick Dev has been broken
> for eons.  Simon's point about "profusion of installs" makes a lot of sense
> too.
>
>
>
> On Fri, Oct 6, 2017 at 8:33 AM Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
>
> > +1 we see a lot of people struggling with the profusion of install and
> run
> > methods as it is, if we can reduce that surface area, life will be a lot
> > easier on the user list.
> >
> > > On 6 Oct 2017, at 13:28, zeo...@gmail.com <zeo...@gmail.com> wrote:
> > >
> > > I say we kill it and repoint the site.  That will give us one less
> thing
> > to
> > > upgrade to centos 7 as well.
> > >
> > > Jon
> > >
> > > On Fri, Oct 6, 2017, 08:27 Justin Leet <justinjl...@gmail.com> wrote:
> > >
> > >> So what are we going to do with Quick Dev?  I'm pretty sure
> everybody's
> > >> been using full dev for awhile now (and quick dev is probably broken
> > since
> > >> I'm sure we haven't been regularly updating it).
> > >>
> > >> I just realized our website links to a wiki page that says to use
> quick
> > >> dev.  Given that quick dev is broken or behind or both, this is going
> > to be
> > >> incredibly confusing and misleading to anyone wandering in.
> > >>
> > >> A short term fix is to just point that wiki page to full dev, so we
> > aren't
> > >> at least broken to anyone coming in.
> > >>
> > >> After that we should figure out what we're doing with quick and full
> dev
> > >> and take care of whatever needs to be done.
> > >>
> > >> Thoughts?
> > >>
> > >> Justin
> > >>
> > > --
> > >
> > > Jon
> >
> >
>

Re: Quick Dev

[Justin Leet]

What is the point of that anyway? It just looks like full dev with no
skipTags? That seems like it should just be documented that you can run
with Vagrant with '--ansible-skip-tags'? We probably should document any
tags you might want to skip anyway.

I'm pretty in favor of killing that.

On Fri, Oct 6, 2017 at 8:46 AM, Nick Allen <n...@nickallen.org> wrote:

> The same case might be made for the Code Lab Platform
> `metron-deployment/vagrant/codelab-platform`?
>
> On Fri, Oct 6, 2017 at 8:44 AM Justin Leet <justinjl...@gmail.com> wrote:
>
> > Wiki updated.  It now points to full dev and the link just says "Dev
> > Platform"
> >
> > On Fri, Oct 6, 2017 at 8:39 AM, Nick Allen <n...@nickallen.org> wrote:
> >
> > > +1 To killing Quick Dev and updating the Wiki.  Quick Dev has been
> broken
> > > for eons.  Simon's point about "profusion of installs" makes a lot of
> > sense
> > > too.
> > >
> > >
> > >
> > > On Fri, Oct 6, 2017 at 8:33 AM Simon Elliston Ball <
> > > si...@simonellistonball.com> wrote:
> > >
> > > > +1 we see a lot of people struggling with the profusion of install
> and
> > > run
> > > > methods as it is, if we can reduce that surface area, life will be a
> > lot
> > > > easier on the user list.
> > > >
> > > > > On 6 Oct 2017, at 13:28, zeo...@gmail.com <zeo...@gmail.com>
> wrote:
> > > > >
> > > > > I say we kill it and repoint the site.  That will give us one less
> > > thing
> > > > to
> > > > > upgrade to centos 7 as well.
> > > > >
> > > > > Jon
> > > > >
> > > > > On Fri, Oct 6, 2017, 08:27 Justin Leet <justinjl...@gmail.com>
> > wrote:
> > > > >
> > > > >> So what are we going to do with Quick Dev?  I'm pretty sure
> > > everybody's
> > > > >> been using full dev for awhile now (and quick dev is probably
> broken
> > > > since
> > > > >> I'm sure we haven't been regularly updating it).
> > > > >>
> > > > >> I just realized our website links to a wiki page that says to use
> > > quick
> > > > >> dev.  Given that quick dev is broken or behind or both, this is
> > going
> > > > to be
> > > > >> incredibly confusing and misleading to anyone wandering in.
> > > > >>
> > > > >> A short term fix is to just point that wiki page to full dev, so
> we
> > > > aren't
> > > > >> at least broken to anyone coming in.
> > > > >>
> > > > >> After that we should figure out what we're doing with quick and
> full
> > > dev
> > > > >> and take care of whatever needs to be done.
> > > > >>
> > > > >> Thoughts?
> > > > >>
> > > > >> Justin
> > > > >>
> > > > > --
> > > > >
> > > > > Jon
> > > >
> > > >
> > >
> >
>

Re: Cloudtrail use case

[Justin Leet]

I know we've had discussions about migrating stuff into docs before.  It
might be worth resurrecting a more use case focused version of that,
instead of starting on the wiki.  I assume the end goal is availability in
the site-book, so even if it's not in a perfect place, I'd rather the
effort be spent on making it pretty there.

I think there's a few floating around that could use a home, so the
discussion might make life easier for multiple things.  Some from the wiki,
some from random READMEs we could relocate and link, some from
presentations and so on.

Having said all that, I know discuss threads can take a few days to
resolve, so wiki and then convert might be the lesser of two evils.


On Thu, Oct 5, 2017 at 6:54 PM, Nick Allen  wrote:

> We don't really have a location in the source code for use cases like this
> right now.  But I think it is so important that we get use cases like this
> published somewhere.  For now, you could add this to the Wiki.  Then later
> on we can figure out how to handle that.
>
> On Thu, Oct 5, 2017 at 6:49 PM, Laurens Vets  wrote:
>
> > On 2017-10-05 15:45, Laurens Vets wrote:
> >
> >> Hi,
> >>
> >> Would anyone be interested in adding a full AWS Cloudtrail use case to
> >> the Metron documentation? I would roughly consist of:
> >> - Apache NiFi configuration to retrieve Cloudtrail logs from S3 and
> >> send it to Metron via Kafka.
> >> - Complete Metron sensor configuration (enrichment, alerting, etc...)
> for
> >> this.
> >>
> >
> > Sent too soon :(
> >
> > If anyone would be interested in this documentation, where would add this
> > in the source?
> >
>

Re: [DISCUSS] Release Process Update

[Justin Leet]

I'd argue it shouldn't be in the site-book at all. Presumably we aren't
releasing while Travis is broken, so it's not useful information for anyone
looking at docs. It just carries over from the main README.  Seems like we
should just scrub it when we do the other fixes to the READMEs to make them
suitable for site-book.  At that point it's just gone entirely. from the
next release.

Doesn't solve the problem of prior releases (assuming we care enough to do
anything).

On Mon, Oct 23, 2017 at 9:32 AM, zeo...@gmail.com  wrote:

> Today I was poking around the Metron site and documentation, and I noticed
> that the site-book's travis build status image is pointing to master for
> all of our releases.  We should probably update the release process
>  to
> pin
> this to the specific release.
>
> I will happily handle the documentation update but I wanted to ask, what
> should it be pointing to?  Repointing is incredibly straightforward
> , but I'm not sure would be
> more appropriate to use - the release branches (e.g. Metron_0.4.1), or
> the release tags (e.g. apache-metron-0.4.1-release)?  I'm not clear on
> their specific uses in our environment.  In reviewing our current process,
> it appears that we _could_ use either.
>
> I also wanted to ask, does anybody think that this should get fixed
> historically?  I think that this might be an excessive amount of hassle,
> but I wanted to put it out there since I'm not intimately familiar with
> what we'd need to do in order to clean this up.
>
> Jon
> --
>
> Jon
>

Master build failures in Travis

[Justin Leet]

See: METRON-1274  and
https://travis-ci.org/apache/metron/builds

An example failure is https://travis-ci.org/apache/metron/builds/290806887

This is possibly intermittent and possibly a result of METRON-1241
, but it hasn't been dug
into yet.  More details are in the ticket.

Please refrain from merging PRs into master until this is resolved.

Thanks,
Justin

Re: [DISCUSS] e2e test infrastructure

[Justin Leet]

As an additional consideration, it would be really nice to get our current
set of integration tests to be able to be run on this infrastructure as
well. Or at least able to be converted in a known manner. Eventually, we
could probably split out the integration tests from the unit tests
entirely. It would likely improve the build times if we we're reusing the
components between test classes (keep in mind right now, we only reuse
between test cases in a given class).

In my mind, ideally we have a single infra for integration and e2e tests.
I'd like to be able to run them from IntelliJ and debug them directly (or
at least be able to easily, and in a well documented manner, be able to do
remote debugging of them). Obviously, that's easier said than done, but
what I'd like to avoid us having essentially two different ways to do the
same thing (spin up some our of dependency components and run code against
them). I'm worried that's quick vs full dev all over again.  But without us
being able to easily kill one because half of tests depend on one and half
on the other.

On Wed, Nov 29, 2017 at 1:22 AM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> What about just spinning up each of the components in their own process?
> It's even lighter weight, doesn't have the complications for HDFS (you can
> use the local FS easily, for example), and doesn't have any issues around
> ports and port mapping with the containers.
>
> On Tue, Nov 28, 2017 at 3:48 PM, Otto Fowler 
> wrote:
>
> > As long as there is not a large chuck of custom deployment that has to be
> > maintained docker sounds ideal.
> > I would like to understand what it would take to create the docker e2e
> env.
> >
> >
> >
> > On November 28, 2017 at 17:27:13, Ryan Merriman (merrim...@gmail.com)
> > wrote:
> >
> > Currently the e2e tests for our Alerts UI depends on full dev being up
> and
> > running. This is not a good long term solution because it forces a
> > contributor/reviewer to run the tests manually with full dev running. It
> > would be better if the backend services could be made available to the
> e2e
> > tests while running in Travis. This would allow us to add the e2e tests
> to
> > our automated build process.
> >
> > What is the right approach? Here are some options I can think of:
> >
> > - Use the in-memory components we use for the backend integration tests
> > - Use a Docker approach
> > - Use mock components designed for the e2e tests
> >
> > Mocking the backend would be my least favorite option because it would
> > introduce a complex module of code that we have to maintain.
> >
> > The in-memory approach has some shortcomings but we may be able to solve
> > some of those by moving components to their own process and spinning them
> > up/down at the beginning/end of tests. Plus we are already using them.
> >
> > My preference would be Docker because it most closely mimics a real
> > installation and gives you isolation, networking and dependency
> management
> > features OOTB. In many cases Dockerfiles are maintained and published by
> a
> > third party and require no work other than some setup like loading data
> or
> > templates/schemas. Elasticsearch is a good example.
> >
> > I believe we could make any of these approaches work in Travis. What does
> > everyone think?
> >
> > Ryan
> >
>

Re: [DISCUSS] Integration/e2e test infrastructure requirements

[Justin Leet]

I think this covers the main things I want to see from a first cut (which
shouldn't be surprising to anyone who followed the PR thread).

The potential follow-on I'd like to see, is having the integration + e2e
tests handled during the maven integration test phase with the failsafe
plugin, instead of just being unit tests. Having said that, it's definitely
low priority, even once this stuff is complete.

On Wed, Dec 13, 2017 at 6:24 PM, Otto Fowler 
wrote:

> Awesome Ryan!
> Have you thought about confluence?
>
>
> On December 13, 2017 at 18:11:39, Ryan Merriman (merrim...@gmail.com)
> wrote:
>
> I took a first pass at adding tasks and will continue adding more as I
> think of them. I will wait for feedback on which modules to include before
> I add all those (only added metron-elasticsearch for now). I left all but
> a couple unassigned so that anyone can pick up a task if they want.
>
> On Wed, Dec 13, 2017 at 4:41 PM, Ryan Merriman 
> wrote:
>
> > Jira is here: https://issues.apache.org/jira/browse/METRON-1352. I am
> > starting to create sub-tasks based on the requirements outlined above and
> > included in that Jira description.
> >
> > I am compiling a list of modules that we'll need to convert to the
> testing
> > infrastructure. Based on imports of ComponentRunner, I get these modules:
> >
> > - metron-elasticsearch
> > - metron-enrichment
> > - metron-indexing
> > - metron-integration-test
> > - metron-maas-service
> > - metron-management
> > - metron-pcap-backend
> > - metron-profiler
> > - metron-rest
> > - metron-solr
> >
> > I am planning on creating sub-tasks for each of these. I know that
> > metron-common should also be converted because it uses the Zookeeper in
> > memory server but doesn't use ComponentRunner to manage it. Are there
> > other modules like this that you know of?
> >
> > On Wed, Dec 13, 2017 at 2:44 PM, Otto Fowler 
> > wrote:
> >
> >> Same as the feature branch name? I just want to find it and set a watch
> >> on it ;)
> >>
> >>
> >> On December 13, 2017 at 15:29:00, Ryan Merriman (merrim...@gmail.com)
> >> wrote:
> >>
> >> I'm open to ideas. What do you think the title should be?
> >>
> >> On Wed, Dec 13, 2017 at 2:13 PM, Otto Fowler 
> >> wrote:
> >>
> >> > What is the Master Jira going to be?
> >> >
> >> >
> >> >
> >> > On December 13, 2017 at 14:36:50, Ryan Merriman (merrim...@gmail.com)
> >> > wrote:
> >> >
> >> > I am going to start the process of creating Jiras out of these initial
> >> > requirements. I agree with them and think they are a good starting
> >> point.
> >> > Feel free to join in at anytime and add/change/remove requirements as
> >> > needed. I will update the thread once I have the initial Jiras created
> >> and
> >> > we can go from there.
> >> >
> >> > On Mon, Dec 11, 2017 at 4:10 PM, Ryan Merriman 
> >> > wrote:
> >> >
> >> > > The purpose of this discussion is map out what is required to get
> the
> >> > POC
> >> > > started with https://github.com/apache/metron/pull/858 into master.
> >> > >
> >> > > The following features were added in the previously mentioned PR:
> >> > >
> >> > > - Dockerfile for Metron REST
> >> > > - Dockerfile for Metron UIs
> >> > > - Docker Compose application including Metron images, Elasticsearch,
> >> > > Kafka, Zookeeper
> >> > > - Modified travis file that manages the Docker environment and runs
> >> > > the e2e tests as part of the build
> >> > > - Maven pom.xml that installs all the required assets into the
> Docker
> >> > > e2e module
> >> > > - Modified metron-alerts pom.xml that allows e2e tests to be run
> >> > > through Maven
> >> > > - An example integration test that has been converted to use the new
> >> > > infrastructure
> >> > >
> >> > > Here are the initial features proposed for acceptance into master:
> >> > >
> >> > > - All e2e and integration tests run on common infrastructure.
> >> > > - All e2e and integration tests are run automatically in the Travis
> >> > > build.
> >> > > - All e2e and integration tests run repeatably and reliably in the
> >> > > Travis build.
> >> > > - Debugging options are available and documented.
> >> > > - The new infra and how to interact with it is documented.
> >> > > - Old infrastructure removed (anything unused or commented out is
> >> > > deleted, instead of staying).
> >> > >
> >> > > Are there other requirements people want to add to this list?
> >> > >
> >> > >
> >> > >
> >> > >
> >> >
> >> >
> >>
> >>
> >
>

[DISCUSS] Stellar Documentation Autogeneration

[Justin Leet]

I think it would be valuable to have the documentation around Stellar being
autogenerated.  We have most of the info we'd want in the @Stellar
annotation, and ideally, we could just pull this info out and produce some
docs similar to what we already manually maintain.  This came up a bit in
the context of https://issues.apache.org/jira/browse/METRON-1361

I put together a super, super (super!) rough POC of using the approach of
Javadoc-style doclet processing that reads the annotations and kicks out
something pretty close to the current docs (without any fancy stuff like
the table of contents and so on).

Right now, there'd be a good deal more to do that to make it usable.  Off
the top of my head, the main things I wanted to look at before really even
taking an actual stab at it are

1) abstracting out the markdown formatting from the annotation parsing
2) Making sure we can integrate this approach without breaking current
Javadocs
3) Managing things across projects (since we put in Stellar functions all
over).
4) Slightly more though about how we'd manage it.

Otto's alluded to having a couple thoughts, and I'm more than happy to get
a better idea of what we want the end state to look like (either this or
something else, e.g. an annotation processor during compile phase or if
someone knows a tool that takes care of this sort of thing.)

Any thoughts?

[DISCUSS] Lowering the barrier to entry to for new users

[Justin Leet]

One of the topics that came up in recent community meeting was about
lowering the barrier to entry for new users.

This is a fairly broad topic that I think covers a few different subtopics.

1) Addressing (or making it easier to address) some of the things we've
seen on the user group from people getting started.
2) Making contributing easier and the ways to do so more obvious.  This
includes things like making it easier to find on our site (compare our page
to Storm's, for example).  It also includes things like reassessing our PR
template (For example, is everything still useful enough to keep it?).
3) Anything else that would make help users adopt Metron and become
actively involved in reviewing, fixes, docs, and all the other sorts of
things that make our stuff better.

I'm mostly going to open this up to a general discussion and brainstorming,
and presumably we come out with some tickets at the end of this.

Re: [DISCUSS] Stellar Documentation Autogeneration

[Justin Leet]

I created a ticket, https://issues.apache.org/jira/browse/METRON-1363 to
track this.  I also noticed that we might be able to swipe from Nifi.
Looks like they generate a nice page for their stuff based on annotations
and we may be able to adapt it to what we have. I haven't looked at their
code at all yet, but feel free to check out:
https://github.com/apache/nifi/tree/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-documentation/src/main/java/org/apache/nifi/documentation

On Thu, Dec 14, 2017 at 6:21 PM, Matt Foley <ma...@apache.org> wrote:

> +1 from me too, Justin.  Great idea.
>
>
> On 12/14/17, 12:44 PM, "zeo...@gmail.com" <zeo...@gmail.com> wrote:
>
> A huge +1 from me.  This would be great
>
> On Thu, Dec 14, 2017 at 3:39 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > +1 from me, great idea Justin. I did a bit of digging around also
> and the
> > Doclet approach you're already using seems the way to go. I didn't
> come
> > across any libraries that would make this easier or better. Not sure
> if
> > Swagger has anything along these lines?
> >
> > On Thu, Dec 14, 2017 at 1:00 PM, Otto Fowler <
> ottobackwa...@gmail.com>
> > wrote:
> >
> > > I think this is a great idea, and I looked at the POC and it isn’t
> as bad
> > > as you make it out to be;)
> > >
> > > What I would like to see is documentation for Stellar functions, by
> > > namespace generated. I would also
> > > like the capability to document at the namespace level.
> > >
> > > Often we have namespace level concepts that don’t fit into any
> given
> > > function’s documentation.
> > > Setting aside the how of the namespace documentation for a moment,
> based
> > on
> > > the POC I would
> > > suggest that we
> > >
> > > * find all namespaces
> > > * create a page per namespace
> > > * document each function in it’s namespace’s page
> > > * include the namespace doc in that page
> > >
> > > Each module that exports stellar function’s should have it’s own
> > > documentation.  As part of breaking stellar out to it’s own module
> > > we should remove stellar documentation from stellar common that
> applies
> > to
> > > functions outside that module.
> > >
> > >
> > >
> > > On December 14, 2017 at 14:32:56, Justin Leet (
> justinjl...@gmail.com)
> > > wrote:
> > >
> > > I think it would be valuable to have the documentation around
> Stellar
> > being
> > > autogenerated. We have most of the info we'd want in the @Stellar
> > > annotation, and ideally, we could just pull this info out and
> produce
> > some
> > > docs similar to what we already manually maintain. This came up a
> bit in
> > > the context of https://issues.apache.org/jira/browse/METRON-1361
> > >
> > > I put together a super, super (super!) rough POC of using the
> approach of
> > > Javadoc-style doclet processing that reads the annotations and
> kicks out
> > > something pretty close to the current docs (without any fancy
> stuff like
> > > the table of contents and so on).
> > >
> > > Right now, there'd be a good deal more to do that to make it
> usable. Off
> > > the top of my head, the main things I wanted to look at before
> really
> > even
> > > taking an actual stab at it are
> > >
> > > 1) abstracting out the markdown formatting from the annotation
> parsing
> > > 2) Making sure we can integrate this approach without breaking
> current
> > > Javadocs
> > > 3) Managing things across projects (since we put in Stellar
> functions all
> > > over).
> > > 4) Slightly more though about how we'd manage it.
> > >
> > > Otto's alluded to having a couple thoughts, and I'm more than
> happy to
> > get
> > > a better idea of what we want the end state to look like (either
> this or
> > > something else, e.g. an annotation processor during compile phase
> or if
> > > someone knows a tool that takes care of this sort of thing.)
> > >
> > > Any thoughts?
> > >
> >
> --
>
> Jon
>
>
>
>

Re: [DISCUSS] Support Ubuntu Installs in the MPack

[Justin Leet]

If we start adding direct support for systems other than CentOS (which I'm
in favor of), I suggest we make sure to very explicitly document what the
level of support, testing, etc. for everything is.

If we're not requiring everything to be tested against Ubuntu, we should
make sure to document exactly what the difference in expectation is along
with it being in some categorization that makes it obvious where it falls
(e.g. CentOS is primary support, Ubuntu is best effort, or whatever).

Also, it sounds like Nick is suggesting that we don't have devs test
against both CentOS and Ubuntu, but potentially that's only true for
non-Ubuntu PRs. If CentOS is the gold standard, I expect not only that
Ubuntu work, but that CentOS work as well.  This implies that Ubuntu
changes get tested against both (i.e. dev needs to be run up for both OSs).

On Thu, Dec 14, 2017 at 6:28 PM, Otto Fowler 
wrote:

> This sounds awesome.  The hortonworks article is getting older ever day.
> This seems like a feature branch candidate.
>
> On December 14, 2017 at 18:22:33, Nick Allen (n...@nickallen.org) wrote:
>
> I've done some work to get the MPack working on Ubuntu. I'd like to get
> that work packaged up and contributed back to Apache. I think it would be
> genuinly useful to the community.
>
> Here is how I was thinking about tackling that through a series of PRs.
>
> 1. Create the DEBs necessary for installing on Ubuntu. See PR #868.
>
> 2. Submit 3 or 4 separate PRs that enhance the existing MPack so that it
> works on both CentOS and Ubuntu. I honestly am not sure how many will fall
> out of the work that I've done, but I will try to chop it up logically so
> that it is easy to review.
>
> 3. Create a "Full Dev" equivalent for Ubuntu so that we can see the
> end-to-end install work for Ubuntu in an automated fashion.
>
>
> ** I do not expect developers to test their PRs on both CentOS and Ubuntu.
> I think the existing CentOS "Full Dev" should remain as the gold standard
> that we test PRs against. No changes there.
>
> Let me know if you have feedback or thoughts on this.
>
> Chao
>

Re: [DISCUSS] Support Ubuntu Installs in the MPack

[Justin Leet]

Yeah, I definitely agree with folding testing Ubuntu into an RC.  It would
be nice if we could fold that testing into a schedule, e.g. weekly, to
avoid unpleasant surprises at RC time.  Not really sure what the best way
to go about that would be. I think a Jira on the testing topic is a good
idea.

On Fri, Dec 15, 2017 at 10:54 AM, Casey Stella  wrote:

> Nick is right that the ASF does not provide support in an explicit way
> (i.e. there are no pathways to get *prioritized* support via SLAs, etc.),
> but it is expected that apache projects provide support via mailing lists
> and answered by volunteers.  Specifically, this is the crux of the
> "community over code" credo.  That philosophical point aside, I think what
> Justin may be intending is "support" in the sense of how much do we fold
> Ubuntu into our testing cycle.  It could be said that we tacitly "support"
> configurations which we test, beyond that caveat emptor.  Which is to say
> that questions on the mailing lists for Metron on Centos will likely be
> answered whereas Metron on OpenBSD might be met with more skepticism or not
> answered.
>
> I would argue that we start with Nick's very generous contribution without
> forcing developers to test their code against it.  Eventually, when we have
> a full-dev that spins up ubuntu, I'd argue that we could consider folding
> it into our testing plans for an RC.
>
> Regarding whether it fits in a feature branch, I think that as long as each
> PR stands alone in providing value, we can avoid a feature branch.  It
> might be worthwhile constructing a JIRA in apache to capture the follow-on
> tasks required to bring Ubuntu into a status where it's more prominent in
> our testing cycle.
>
> On Fri, Dec 15, 2017 at 10:45 AM, Nick Allen  wrote:
>
> > > The end goal is Ubuntu Ambari + Deb and full-dev-ubuntu right?
> >
> > That list sounds good to me.
> >
> > (Plus, some way of dealing with Justin's point about support.)
> >
> >
> >
> > On Fri, Dec 15, 2017 at 10:11 AM Otto Fowler 
> > wrote:
> >
> > > I’m ok if it is not. Suggesting because it is a series of prs.
> > >
> > > The end goal is Ubuntu Ambari + Deb and full-dev-ubuntu right?
> > >
> > > On December 15, 2017 at 10:03:23, Nick Allen (n...@nickallen.org)
> wrote:
> > >
> > > > This seems like a feature branch candidate.
> > >
> > > Personally, I don't see the need for a feature branch on this one.  It
> > > won't involve big, architectural changes.  The touch points are
> > > constrained.  Everything that we currently have will continue to work
> as
> > it
> > > always had after each PR.  If you feel strongly the other way, please
> > > provide your reasoning to help me understand.
> > >
> > >
> > >
> > >
> > > On Thu, Dec 14, 2017 at 6:28 PM, Otto Fowler 
> > > wrote:
> > >
> > >> This sounds awesome.  The hortonworks article is getting older ever
> day.
> > >> This seems like a feature branch candidate.
> > >>
> > >> On December 14, 2017 at 18:22:33, Nick Allen (n...@nickallen.org)
> > wrote:
> > >>
> > >> I've done some work to get the MPack working on Ubuntu. I'd like to
> get
> > >> that work packaged up and contributed back to Apache. I think it would
> > be
> > >> genuinly useful to the community.
> > >>
> > >> Here is how I was thinking about tackling that through a series of
> PRs.
> > >>
> > >> 1. Create the DEBs necessary for installing on Ubuntu. See PR #868.
> > >>
> > >> 2. Submit 3 or 4 separate PRs that enhance the existing MPack so that
> it
> > >> works on both CentOS and Ubuntu. I honestly am not sure how many will
> > fall
> > >> out of the work that I've done, but I will try to chop it up logically
> > so
> > >> that it is easy to review.
> > >>
> > >> 3. Create a "Full Dev" equivalent for Ubuntu so that we can see the
> > >> end-to-end install work for Ubuntu in an automated fashion.
> > >>
> > >>
> > >> ** I do not expect developers to test their PRs on both CentOS and
> > Ubuntu.
> > >> I think the existing CentOS "Full Dev" should remain as the gold
> > standard
> > >> that we test PRs against. No changes there.
> > >>
> > >> Let me know if you have feedback or thoughts on this.
> > >>
> > >> Chao
> > >>
> > >>
> > >
> >
>

Re: [DISCUSS] Support Ubuntu Installs in the MPack

[Justin Leet]

By 'direct support" I just meant that it becomes an installation target we
semi-actively maintain a specific installation method for.

Right now we don't need to communicate that all because we don't provide
anything other RPMs.  The cutoff is implicit: There's convenience RPMs you
can build, or you need to get it working manually.  Post Debs, this isn't
the case anymore.  The proposal is "We have convenience RPMs that are
regularly run up.  We have convenience Debs that are at least occasionally
checked and you probably should be able to run up.  Finally, you can do
things manually".

All I'm saying is that I would like that made explicit in our docs when we
integrate this work, so that people know what they're getting into.


On Fri, Dec 15, 2017 at 10:11 AM, Otto Fowler 
wrote:

> I’m ok if it is not. Suggesting because it is a series of prs.
>
> The end goal is Ubuntu Ambari + Deb and full-dev-ubuntu right?
>
> On December 15, 2017 at 10:03:23, Nick Allen (n...@nickallen.org) wrote:
>
> > This seems like a feature branch candidate.
>
> Personally, I don't see the need for a feature branch on this one.  It
> won't involve big, architectural changes.  The touch points are
> constrained.  Everything that we currently have will continue to work as it
> always had after each PR.  If you feel strongly the other way, please
> provide your reasoning to help me understand.
>
>
>
>
> On Thu, Dec 14, 2017 at 6:28 PM, Otto Fowler 
> wrote:
>
> > This sounds awesome.  The hortonworks article is getting older ever day.
> > This seems like a feature branch candidate.
> >
> > On December 14, 2017 at 18:22:33, Nick Allen (n...@nickallen.org) wrote:
> >
> > I've done some work to get the MPack working on Ubuntu. I'd like to get
> > that work packaged up and contributed back to Apache. I think it would be
> > genuinly useful to the community.
> >
> > Here is how I was thinking about tackling that through a series of PRs.
> >
> > 1. Create the DEBs necessary for installing on Ubuntu. See PR #868.
> >
> > 2. Submit 3 or 4 separate PRs that enhance the existing MPack so that it
> > works on both CentOS and Ubuntu. I honestly am not sure how many will
> fall
> > out of the work that I've done, but I will try to chop it up logically so
> > that it is easy to review.
> >
> > 3. Create a "Full Dev" equivalent for Ubuntu so that we can see the
> > end-to-end install work for Ubuntu in an automated fashion.
> >
> >
> > ** I do not expect developers to test their PRs on both CentOS and
> Ubuntu.
> > I think the existing CentOS "Full Dev" should remain as the gold standard
> > that we test PRs against. No changes there.
> >
> > Let me know if you have feedback or thoughts on this.
> >
> > Chao
> >
> >
>

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

[Justin Leet]

Do we have any further discussion on this?  Pardon me if I misstate
anyone's position, but it seems like we have a couple people (Otto and Jon
and slightly Matt?) in favor of (a), Nick in favor of (b), and presumably a
section of people like myself without a particular horse in the race.

It seems like we need to come to some sort of consensus so that we can get
the release bus moving again, and right now it seems like (a) is gathering
more explicit support.  Do we have a compelling reason to not do (a)? To be
honest, my main worry is more "If we do (a) are we going to be miserable if
we need to iterate or adjust?" I'm not seeing anything that suggests
anything too terrible, so unless we see some more discussion, I suggest we
move forward with (a).

On Mon, Dec 4, 2017 at 9:34 PM, zeo...@gmail.com  wrote:

> I would prefer a, but I was initially thinking of doing the plugin first
> and then get in the two PRs out to use this new tag, which are already +1'd
> and just waiting on this conversation.  For reference,
> https://github.com/apache/metron/pull/847 and
> https://github.com/apache/metron-bro-plugin-kafka/pull/4
>
> Jon
>
> On Mon, Dec 4, 2017, 20:54 Otto Fowler  wrote:
>
> > It seems to me, as I believe I have stated before that a) feels like the
> > proper way to handle this.  It is how I have seen other projects like
> NiFi
> > handle things as well.
> >
> >
> >
> > On December 4, 2017 at 17:14:41, Matt Foley (ma...@apache.org) wrote:
> >
> > Okay, looking at this from the perspective of making a release:
> >
> >
> >
> > We have two choices:
> >
> > a) I can simply make a 0.1 (or 1.0 or 0.4.2) release of
> > metron-bro-plugin-kafka, at the same time and using the same process
> > (modulo the necessary) as Metron.  This is dirt simple.
> >
> > b) I or someone needs to:
> >
> > - open a jira,
> >
> > - add the submodule to the Metron code tree,
> >
> > - possibly (optionally) add build mechanism to the maven poms, and
> >
> > - document as much as we think appropriate regarding what it is, how
> to
> > build it, and how to update it,
> >
> > and commit that before the 0.4.2 release.
> >
> >
> >
> > What is the will of the community?
> >
> > Thanks,
> >
> > --Matt
> >
> >
> >
> > From: Nick Allen 
> > Reply-To: "dev@metron.apache.org" 
> > Date: Tuesday, November 28, 2017 at 9:09 AM
> > To: "dev@metron.apache.org" 
> > Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for
> Bro'
> >
> >
> >
> > I'll add a bit to Jon's technical comments.
> >
> >
> >
> > * We only created a separate repo because it was a technical requirement
> to
> > leverage the bro-pkg mechanism.
> >
> > * Leveraging the new bro-pkg mechanism has many advantages as outlined by
> > Jon.
> >
> > * Enabling the bro-pkg mechanism is backwards compatible. I can install
> the
> > plugin exactly how we use to.
> >
> >
> >
> > While I agree with Jon's technical comments, I disagree with the
> > non-technical ones.
> >
> >
> >
> > (1) I do not want to change our release management process. While we
> needed
> > to make a new repo (a technical change), I did not want that to change
> how
> > we operate as a community (our procedures, policies, versioning and
> release
> > cycles).
> >
> >
> >
> > (2) I see no value in adopting a separate release management process for
> > the Bro plugin alone. Having a separate release process does not make the
> > plugin *more* available to the Bro community.
> >
> >
> >
> > (3) I also see no value in positioning the plugin to be spun-out of the
> > Metron project. It is a part of Metron and I want to see it benefit and
> > evolve "the Apache-way".
> >
> >
> >
> > In my mind, the best way to accommodate the additional repo, while
> > minimizing changes to our release management process, is to treat the new
> > repo as a submodule. I fail to see significant downsides to this
> approach.
> > A few extract command-line options do not seem overly onerous to me.
> >
> >
> >
> > Many thanks go to Jon for all the hard work he has put in enhancing the
> > plugin.
> >
> >
> >
> >
> >
> > On Mon, Nov 27, 2017 at 10:07 PM, zeo...@gmail.com 
> > wrote:
> >
> > In an attempt to keep this from becoming unbearably long, I will try to
> > keep my responses short, but I would be happy to elaborate. That's a
> fairly
> > good timeline and summary, but here are some clarifications in
> > corresponding order:
> >
> >
> >
> > - The plugin history is quite short and you can probably get a good bit
> of
> > context just by looking at the commits.
> >
> > - The plugin is only useful to the bro community, but it is rather
> popular.
> >
> > - The Bro team created the idea of bro packages, which can include bro
> > plugins, bro scripts, or BroControl plugins. So, instead of having a
> > 'plugins' repo, they moved to have a 'packages' repo which is by default
> > referenced by 

Re: MASTER is Broken

[Justin Leet]

Casey's PR has been committed into master.  Please update any branches/PRs
as needed.

On Wed, Oct 25, 2017 at 3:56 PM, Casey Stella <ceste...@gmail.com> wrote:

> I expect that can be done via an infra request.  Not sure though.
>
> On Wed, Oct 25, 2017 at 3:33 PM, Justin Leet <justinjl...@gmail.com>
> wrote:
>
> > To hop onto Otto's comment, it would be really nice to have master build
> > failures go to the dev list.  I have no idea who has permissions to set
> > that up though, so if anyone does that'd be super helpful.
> >
> > On Wed, Oct 25, 2017 at 3:14 PM, Casey Stella <ceste...@gmail.com>
> wrote:
> >
> > > I noticed it on my personal travis the when I merged in apache master.
> > > BTW: the PR for the fix is https://github.com/apache/metron/pull/816
> > >
> > >
> > >
> > > On Wed, Oct 25, 2017 at 3:13 PM, Otto Fowler <ottobackwa...@gmail.com>
> > > wrote:
> > >
> > > > Do you get the travis alerts for failed builds?  If not who does?
> > > >
> > > >
> > > > On October 25, 2017 at 15:12:46, Otto Fowler (
> ottobackwa...@gmail.com)
> > > > wrote:
> > > >
> > > > Thanks Casey
> > > >
> > > >
> > > > On October 25, 2017 at 15:11:34, Casey Stella (ceste...@gmail.com)
> > > wrote:
> > > >
> > > > I got it. I made a JIRA and have a fix in the middle of testing:
> > > > https://issues.apache.org/jira/browse/METRON-1280
> > > >
> > > > On Wed, Oct 25, 2017 at 3:09 PM, Otto Fowler <
> ottobackwa...@gmail.com>
> > > > wrote:
> > > >
> > > > > https://travis-ci.org/search/apache%252Fmetron
> > > > >
> > > > > I believe it is to do with https://github.com/apache/
> metron/pull/767
> > ,
> > > > and
> > > > > the missed addition of metron-zookeeper.
> > > > > My Pr’s that I’m merging master into are fixed by upping the
> > > > > metron-zookeeper version to 0.4.2.
> > > > > I would fix this, but I have to run out. If nobody grabs it, i’ll
> fix
> > > it
> > > > > later.
> > > > >
> > > >
> > > >
> > >
> >
>

Re: Master build failures in Travis

[Justin Leet]

One last thing, I'd recommend that master get merged into everyone's
branches/PRs to make sure the tests work consistently during any necessary
Travis runs.

On Wed, Oct 25, 2017 at 9:29 AM, Justin Leet <justinjl...@gmail.com> wrote:

> Master is building again!  Thanks to Ryan for taking care of it. We should
> be good to go with committing again.
>
> Thanks for your patience everyone.
>
> On Mon, Oct 23, 2017 at 10:06 AM, Casey Stella <ceste...@gmail.com> wrote:
>
>> Looks like Ryan got there first, which is awesome.  Thanks for cleaning up
>> my mess :)
>>
>> On Mon, Oct 23, 2017 at 10:04 AM, Casey Stella <ceste...@gmail.com>
>> wrote:
>>
>> > Yeah, that could be a consequence.  With the cache in place, the calls
>> to
>> > delete are async.  This isn't generally a problem in an actual
>> > installation, but in the integration tests, it can take some time to
>> sync
>> > up (depending on the load).  I ran it 20 or so times teasing these out,
>> but
>> > it's never quite enough.  PR pending. :)
>> >
>> > On Mon, Oct 23, 2017 at 9:58 AM, Justin Leet <justinjl...@gmail.com>
>> > wrote:
>> >
>> >> See: METRON-1274 <https://issues.apache.org/jira/browse/METRON-1274>
>> and
>> >> https://travis-ci.org/apache/metron/builds
>> >>
>> >> An example failure is https://travis-ci.org/apache/m
>> >> etron/builds/290806887
>> >>
>> >> This is possibly intermittent and possibly a result of METRON-1241
>> >> <https://issues.apache.org/jira/browse/METRON-1241>, but it hasn't
>> been
>> >> dug
>> >> into yet.  More details are in the ticket.
>> >>
>> >> Please refrain from merging PRs into master until this is resolved.
>> >>
>> >> Thanks,
>> >> Justin
>> >>
>> >
>> >
>>
>
>

Re: MASTER is Broken

[Justin Leet]

To hop onto Otto's comment, it would be really nice to have master build
failures go to the dev list.  I have no idea who has permissions to set
that up though, so if anyone does that'd be super helpful.

On Wed, Oct 25, 2017 at 3:14 PM, Casey Stella  wrote:

> I noticed it on my personal travis the when I merged in apache master.
> BTW: the PR for the fix is https://github.com/apache/metron/pull/816
>
>
>
> On Wed, Oct 25, 2017 at 3:13 PM, Otto Fowler 
> wrote:
>
> > Do you get the travis alerts for failed builds?  If not who does?
> >
> >
> > On October 25, 2017 at 15:12:46, Otto Fowler (ottobackwa...@gmail.com)
> > wrote:
> >
> > Thanks Casey
> >
> >
> > On October 25, 2017 at 15:11:34, Casey Stella (ceste...@gmail.com)
> wrote:
> >
> > I got it. I made a JIRA and have a fix in the middle of testing:
> > https://issues.apache.org/jira/browse/METRON-1280
> >
> > On Wed, Oct 25, 2017 at 3:09 PM, Otto Fowler 
> > wrote:
> >
> > > https://travis-ci.org/search/apache%252Fmetron
> > >
> > > I believe it is to do with https://github.com/apache/metron/pull/767,
> > and
> > > the missed addition of metron-zookeeper.
> > > My Pr’s that I’m merging master into are fixed by upping the
> > > metron-zookeeper version to 0.4.2.
> > > I would fix this, but I have to run out. If nobody grabs it, i’ll fix
> it
> > > later.
> > >
> >
> >
>

Re: [DISCUSS] Release Manager

[Justin Leet]

I'd be happy to to volunteer to take over for a while.

Thanks to Matt for all the help through the last couple releases!

Justin

On Thu, May 10, 2018 at 11:06 AM, Casey Stella  wrote:

> Hi All,
>
> Matt Foley, our esteemed Release manager for the last couple releases, has
> asked to be relieved.  So, I'm calling on volunteers for the next release
> manager.  It should be a committer and there are a few things that require
> a PMC member, I believe, but the release manager can ask for help from a
> PMC member.
>
> So, Matt's watch has ended, who wants to volunteer?
>
> Casey
>

Re: [DISCUSS] Metron release 0.5.0

[Justin Leet]

I have a minor adjustment to the proposed timeframe.  I'd like to move the
tentative date for starting the RC process to Tuesday the 22nd.  I have a
prior engagement on Monday that will consume the majority of the day.  I
was just too excited about releasing. Sorry about the trouble.

Justin

On Tue, May 15, 2018 at 4:26 PM, Justin Leet <justinjl...@gmail.com> wrote:

> Hi all,
>
> Based on a thread from last week
> <https://lists.apache.org/thread.html/8e795ad4faecb4256f1cfa36ddbf757c9ef9b3d104eab9df6573d9a4@%3Cdev.metron.apache.org%3E>,
> I'll be taking over as the release manager for our next release. Thanks
> again to Matt for his work shepherding our previous releases!
>
> Version Number
> This thread
> <https://lists.apache.org/thread.html/d906ce1491b775d8719bdb1d9bca72b9ec7fe255260ab955b5cdd695@%3Cdev.metron.apache.org%3E>
>  had
> a lot of support for the release bumping to 0.5.0 so I'd like to work off
> that, but we can adjust as needed.
>
> Proposed Timeframe
> I would tentatively like to start work on the RC this coming Monday 21st.
> This can be shifted as needed, either to make time for more PRs to make it
> in or if we'd like to get things moving faster.
>
> I'm proposing we create this release from the Metron master branch, plus
> any commits the community considers necessary for the release and can get
> in the very near future.
>
> JIRA status
> There are 24 open PRs at https://github.com/apache/metron/pulls.  If we
> consider any of these necessary for this release, we should work on getting
> them closed.
>
> There have been 123 commits since the current release (see the end of the
> message).  I'm sure there'll be updates in Jira required to get everything
> up to date.  I will follow up on that, and I'd ask that we work on
> preemptively cleaning our Jira up.
>
> Please respond with any specific PRs we'd like to have in for 0.5.0, along
> with any Jira issues we feel are worth working on.  In particular, in the
> previous discussion, https://issues.apache.org/jira/browse/METRON-1550 came
> up as something to potentially pull in, but I didn't see a consensus form.
>
> Completed PRs as of May 15 as generated by Nick's nifty one liner
> git log --pretty="%cr %s" tags/apache-metron-0.4.2-release..HEAD
>
> Thanks,
> Justin
>
> 3 hours ago METRON-1552: Add gzip file validation check to the geo loader
> (mmiklavc via mmiklavc) closes apache/metron#1011
> 23 hours ago METRON-1551 Profiler Should Not Use Java Serialization
> (nickwallen) closes apache/metron#1012
> 4 days ago METRON-1549: Add empty object test to WriterBoltIntegrationTest
> implementation (mmiklavc via mmiklavc) closes apache/metron#1009
> 6 days ago METRON-1541 Mvn clean results in git status having deleted
> files. (justinleet via nickwallen) closes apache/metron#1003
> 6 days ago METRON-1461 MIN MAX stellar function should take a stats or
> list object and return min/max (MohanDV via nickwallen) closes
> apache/metron#942
> 6 days ago METRON-1184 EC2 Deployment - Updating control_path to
> accommodate for Linux (Ahmed Shah via ottobackwards) closes
> apache/metron#754
> 6 days ago METRON-1530 Default proxy config settings in metron-contrib
> need to be updated (sardell via merrimanr) closes apache/metron#998
> 11 days ago METRON-1545 Upgrade Spring and Spring Boot (merrimanr) closes
> apache/metron#1008
> 13 days ago METRON-1543 Unable to Set Parser Output Topic in Sensor Config
> (nickwallen) closes apache/metron#1007
> 3 weeks ago METRON-1539: Specialized RENAME field transformer closes
> apache/incubator-metron#1002
> 3 weeks ago METRON-1520: Add caching for stellar field transformations
> closes apache/incubator-metron#990
> 3 weeks ago METRON-1529 CONFIG_GET Fails to Retrieve Latest Config When
> Run in Zeppelin REPL (nickwallen) closes apache/metron#997
> 3 weeks ago METRON-1511 Unable to Serialize Profiler Configuration
> (nickwallen) closes apache/metron#982
> 4 weeks ago METRON-1528: Fix missing file in metron.spec (mmiklavc via
> mmiklavc) closes apache/metron#996
> 4 weeks ago METRON-1445: Update performance tuning guide with more
> explicit parameter instructions (mmiklavc via mmiklavc) closes
> apache/metron#988
> 4 weeks ago METRON-1502 Upgrade Doxia plugin to 1.8 (justinleet) closes
> apache/metron#974
> 4 weeks ago METRON-1527: Remove dead test file sitting in source folder
> (mmiklavc via mmiklavc) closes apache/metron#994
> 4 weeks ago METRON-1499 Enable Configuration of Unified Enrichment
> Topology via Ambari (nickwallen) closes apache/metron#984
> 4 weeks ago METRON-1515: Errors loading stellar functions currently bomb
> the entire topology, they should be recoverable closes
> apache/incubator-metron#985
> 5 weeks a

[DISCUSS] Metron release 0.5.0

[Justin Leet]

Hi all,

Based on a thread from last week
,
I'll be taking over as the release manager for our next release. Thanks
again to Matt for his work shepherding our previous releases!

Version Number
This thread

had
a lot of support for the release bumping to 0.5.0 so I'd like to work off
that, but we can adjust as needed.

Proposed Timeframe
I would tentatively like to start work on the RC this coming Monday 21st.
This can be shifted as needed, either to make time for more PRs to make it
in or if we'd like to get things moving faster.

I'm proposing we create this release from the Metron master branch, plus
any commits the community considers necessary for the release and can get
in the very near future.

JIRA status
There are 24 open PRs at https://github.com/apache/metron/pulls.  If we
consider any of these necessary for this release, we should work on getting
them closed.

There have been 123 commits since the current release (see the end of the
message).  I'm sure there'll be updates in Jira required to get everything
up to date.  I will follow up on that, and I'd ask that we work on
preemptively cleaning our Jira up.

Please respond with any specific PRs we'd like to have in for 0.5.0, along
with any Jira issues we feel are worth working on.  In particular, in the
previous discussion, https://issues.apache.org/jira/browse/METRON-1550 came
up as something to potentially pull in, but I didn't see a consensus form.

Completed PRs as of May 15 as generated by Nick's nifty one liner
git log --pretty="%cr %s" tags/apache-metron-0.4.2-release..HEAD

Thanks,
Justin

3 hours ago METRON-1552: Add gzip file validation check to the geo loader
(mmiklavc via mmiklavc) closes apache/metron#1011
23 hours ago METRON-1551 Profiler Should Not Use Java Serialization
(nickwallen) closes apache/metron#1012
4 days ago METRON-1549: Add empty object test to WriterBoltIntegrationTest
implementation (mmiklavc via mmiklavc) closes apache/metron#1009
6 days ago METRON-1541 Mvn clean results in git status having deleted
files. (justinleet via nickwallen) closes apache/metron#1003
6 days ago METRON-1461 MIN MAX stellar function should take a stats or list
object and return min/max (MohanDV via nickwallen) closes apache/metron#942
6 days ago METRON-1184 EC2 Deployment - Updating control_path to
accommodate for Linux (Ahmed Shah via ottobackwards) closes
apache/metron#754
6 days ago METRON-1530 Default proxy config settings in metron-contrib need
to be updated (sardell via merrimanr) closes apache/metron#998
11 days ago METRON-1545 Upgrade Spring and Spring Boot (merrimanr) closes
apache/metron#1008
13 days ago METRON-1543 Unable to Set Parser Output Topic in Sensor Config
(nickwallen) closes apache/metron#1007
3 weeks ago METRON-1539: Specialized RENAME field transformer closes
apache/incubator-metron#1002
3 weeks ago METRON-1520: Add caching for stellar field transformations
closes apache/incubator-metron#990
3 weeks ago METRON-1529 CONFIG_GET Fails to Retrieve Latest Config When Run
in Zeppelin REPL (nickwallen) closes apache/metron#997
3 weeks ago METRON-1511 Unable to Serialize Profiler Configuration
(nickwallen) closes apache/metron#982
4 weeks ago METRON-1528: Fix missing file in metron.spec (mmiklavc via
mmiklavc) closes apache/metron#996
4 weeks ago METRON-1445: Update performance tuning guide with more explicit
parameter instructions (mmiklavc via mmiklavc) closes apache/metron#988
4 weeks ago METRON-1502 Upgrade Doxia plugin to 1.8 (justinleet) closes
apache/metron#974
4 weeks ago METRON-1527: Remove dead test file sitting in source folder
(mmiklavc via mmiklavc) closes apache/metron#994
4 weeks ago METRON-1499 Enable Configuration of Unified Enrichment Topology
via Ambari (nickwallen) closes apache/metron#984
4 weeks ago METRON-1515: Errors loading stellar functions currently bomb
the entire topology, they should be recoverable closes
apache/incubator-metron#985
5 weeks ago METRON-1522 Fix the typo errors at profile debugger readme
(MohanDV via nickwallen) closes apache/metron#992
5 weeks ago METRON-1519 Indexing Error Topic Property Not Displayed in
MPack (nickwallen) closes apache/metron#987
5 weeks ago METRON-1347: Indexing Topology should fail tuples without a
source.type (cstella via mmiklavc) closes apache/metron#863
5 weeks ago Add support for Vagrant Cachier plugin if present
(simonellistonball via mmiklavc) closes apache/metron#993
5 weeks ago METRON-1521: JSONMapParser is no longer serializable closes
apache/incubator-metron#991
5 weeks ago METRON-1516 Support for Ansible 2.5.0 (ottobackwards) closes
apache/metron#989
5 weeks ago METRON-1494 Profiler Emits Messages to Kafka When Not Needed
(nickwallen) closes apache/metron#967
5 weeks ago METRON-1510 Update Metron website 

[DISCUSS] Metron 0.5.0 Release Status Update

[Justin Leet]

Hi all,

I'm working on kicking off our release process
.  As
you might have seen today, a large number of tickets have been updated, in
particular aligning things like Assignee, Status, and Fix Version.  Please
see Metron Jira - 0.5.0
 for more
details on the individual tickets in the context of the release and to
ensure I've updated your tickets properly.

One additional Jira has been created and placed in progress: METRON-1574:
Update version to 0.5.0 .
A PR for this is available at PR 1026
. Once this ticket is reviewed
and merged, I plan on creating the release branch and release candidate
from then master, barring any blockers being found prior to this.


Thanks,
Justin

Re: [DISCUSS] Metron 0.5.0 Release Status Update

[Justin Leet]

Thanks, Nick!

Current status is that I've gotten my public key setup in the KEYS file,
and started through the RC creation process.

I hit the Font Awesome bundle.css issue we've seen twice before. See
https://issues.apache.org/jira/browse/METRON-1576

Previous occurrences:
https://issues.apache.org/jira/browse/METRON-1373
https://issues.apache.org/jira/browse/METRON-1163

I have a PR to address this at https://github.com/apache/metron/pull/1029.
Once this is handled, we should be able to continue towards getting an RC
for the release.

Justin

On Wed, May 23, 2018 at 12:20 PM, Nick Allen <n...@nickallen.org> wrote:

> METRON-1565 was merged into master, but not updated for 0.5.0.  I just
> fixed that.
>
> https://issues.apache.org/jira/browse/METRON-1565
>
>
>
> On Tue, May 22, 2018 at 8:28 PM, Justin Leet <justinjl...@gmail.com>
> wrote:
>
> > Hi all,
> >
> > I'm working on kicking off our release process
> > <https://cwiki.apache.org/confluence/display/METRON/Release+Process>.
> As
> > you might have seen today, a large number of tickets have been updated,
> in
> > particular aligning things like Assignee, Status, and Fix Version.
> Please
> > see Metron Jira - 0.5.0
> > <https://issues.apache.org/jira/projects/METRON/versions/12343342> for
> > more
> > details on the individual tickets in the context of the release and to
> > ensure I've updated your tickets properly.
> >
> > One additional Jira has been created and placed in progress: METRON-1574:
> > Update version to 0.5.0 <https://issues.apache.org/
> jira/browse/METRON-1574
> > >.
> > A PR for this is available at PR 1026
> > <https://github.com/apache/metron/pull/1026>. Once this ticket is
> reviewed
> > and merged, I plan on creating the release branch and release candidate
> > from then master, barring any blockers being found prior to this.
> >
> >
> > Thanks,
> > Justin
> >
>

Bro plugin release process docs?

[Justin Leet]

At the risk of exposing my ignorance, do we have the bro plugin release
process documented anywhere?  We have a doc for the main release (
https://cwiki.apache.org/confluence/display/METRON/Release+Process), but I
haven't noticed one for the bro plugin.

For the current RC, it's not included and it wasn't pushed for (it has less
changes for obvious reasons).  However, we should be making sure to
validate if its necessary to release and having the process documented.

Justin

[ANNOUNCE] Apache Metron release 0.5.0

[Justin Leet]

Hi All,

I’m happy to announce the release of Metron 0.5.0!  Everyone has put in a
lot of working into improvements, new features, and discussion.  Thanks to
everyone who contributed, and I look forward to having users enjoy our new
features and improvements.

Details:
The official release source code tarballs may be obtained at any of the
mirrors listed in
http://www.apache.org/dyn/closer.cgi/metron/0.5.0

As usual, the secure signatures and confirming hashes may be obtained at
https://dist.apache.org/repos/dist/release/metron/0.5.0

The release branches in github is
https://github.com/apache/metron/tree/Metron_0.5.0 (tag
apache-metron-0.5.0-release)

The release doc book is at http://metron.apache.org/current-book/index.html
The Apache Metron web site at http://metron.apache.org/ has been updated;
please refresh your web browser cache if the new links do not immediately
appear.

Change lists and Release Notes may be obtained at the same locations as the
tarballs.
For your reading pleasure, the change list is appended to this message.

Metron CHANGES (in reverse chronological order):

METRON-1586 Defaulting for the source type field in alerts UI does
not work (merrimanr via justinleet) closes apache/metron#1038
METRON-1569: Allow user to change field name conversion when
indexing to Elasticsearch (nickwallen via mmiklavc) closes
apache/metron#1022
METRON-1544 Flaky test:
org.apache.metron.stellar.common.CachingStellarProcessorTest#testCaching
(nickwallen) closes apache/metron#1015
METRON-1580 Release candidate check script requires Bro Plugin
(nickwallen via ottobackwards) closes apache/metron#1034
METRON-1532 Getting started documentation improvements (sardell
via nickwallen) closes apache/metron#1001
METRON-1576 bundle.css RAT failure for
metron-interface/metron-alerts (justinleet) closes apache/metron#1029
METRON-1575 Add leet gpg public key to the KEYS file (justinleet)
closes apache/metron#1028
METRON-1574 Update version to 0.5.0 (justinleet) closes apache/metron#1026
METRON-1566 Alert updates are not propagated to metaalert child
alerts (merrimanr) closes apache/metron#1018
METRON-1565 Metaalerts fix denormalization after moving to active
status (merrimanr) closes apache/metron#1017
METRON-1548 Remove hardcoded source:type from Alerts UI
(justinleet) closes apache/metron#1010
METRON-1548 Remove hardcoded source:type from Alerts UI (sardell
via justinleet) closes apache/metron#1010
METRON-1564 Full dev kafka has offsets.topic.replication.factor
set to 3 instead of 1 (justinleet) closes apache/metron#1016
METRON-1552: Add gzip file validation check to the geo loader
(mmiklavc via mmiklavc) closes apache/metron#1011
METRON-1551 Profiler Should Not Use Java Serialization
(nickwallen) closes apache/metron#1012
METRON-1549: Add empty object test to WriterBoltIntegrationTest
implementation (mmiklavc via mmiklavc) closes apache/metron#1009
METRON-1541 Mvn clean results in git status having deleted files.
(justinleet via nickwallen) closes apache/metron#1003
METRON-1461 MIN MAX stellar function should take a stats or list
object and return min/max (MohanDV via nickwallen) closes
apache/metron#942
METRON-1184 EC2 Deployment - Updating control_path to accommodate
for Linux (Ahmed Shah via ottobackwards) closes apache/metron#754
METRON-1530 Default proxy config settings in metron-contrib need
to be updated (sardell via merrimanr) closes apache/metron#998
METRON-1545 Upgrade Spring and Spring Boot (merrimanr) closes
apache/metron#1008
METRON-1543 Unable to Set Parser Output Topic in Sensor Config
(nickwallen) closes apache/metron#1007
METRON-1539: Specialized RENAME field transformer closes
apache/incubator-metron#1002
METRON-1520: Add caching for stellar field transformations closes
apache/incubator-metron#990
METRON-1529 CONFIG_GET Fails to Retrieve Latest Config When Run in
Zeppelin REPL (nickwallen) closes apache/metron#997
METRON-1511 Unable to Serialize Profiler Configuration
(nickwallen) closes apache/metron#982
METRON-1528: Fix missing file in metron.spec (mmiklavc via
mmiklavc) closes apache/metron#996
METRON-1445: Update performance tuning guide with more explicit
parameter instructions (mmiklavc via mmiklavc) closes
apache/metron#988
METRON-1502 Upgrade Doxia plugin to 1.8 (justinleet) closes
apache/metron#974
METRON-1527: Remove dead test file sitting in source folder
(mmiklavc via mmiklavc) closes apache/metron#994
METRON-1499 Enable Configuration of Unified Enrichment Topology
via Ambari (nickwallen) closes apache/metron#984
METRON-1515: Errors loading stellar functions currently bomb the
entire topology, they should be recoverable closes
apache/incubator-metron#985
METRON-1522 Fix the typo errors at profile debugger readme
(MohanDV via nickwallen) closes apache/metron#992
METRON-1519 Indexing Error Topic Property Not Displayed in MPack
(nickwallen) closes 

Re: [VOTE] Metron Release Candidate 0.5.0-RC1

[Justin Leet]

I'm going to go ahead and cancel RC1, since METRON-1544 looks pretty set.

A new release candidate will be cut.

Results (including my own vote):
+1
Nick Allen


-1
Otto Fowler
Justin Leet

On Tue, May 29, 2018 at 10:39 AM, Justin Leet  wrote:

> I didn't realize METRON-1544 wasn't in.  I'm definitely okay with
> cancelling the vote, and kicking out a new RC.
>
> On Tue, May 29, 2018 at 7:11 AM, Otto Fowler 
> wrote:
>
>> -1 (binding)
>>
>> My yield for building this it terrible.  1 in 3.
>>
>> I propose https://github.com/apache/metron/pull/1015 inclusion.
>>
>>
>> On May 27, 2018 at 17:50:43, Nick Allen (n...@nickallen.org) wrote:
>>
>> No, the PR for this transient issue is still under review.
>>
>> On Sun, May 27, 2018 at 10:53 AM, Otto Fowler 
>> wrote:
>>
>> >
>> > Failed tests:
>> >   CachingStellarProcessorTest.testCaching:73 expected:<6> but was:<5>
>> >
>> > I thought we landed a fix for this?
>> >
>> >
>> > On May 27, 2018 at 08:24:19, zeo...@gmail.com (zeo...@gmail.com) wrote:
>> >
>> > We did discuss doing a release since there were two new commits, but I
>> > don't think it was included in this round.
>> >
>> > Jon
>> >
>> > On Sat, May 26, 2018, 10:22 Otto Fowler 
>> wrote:
>> >
>> > > Is there a BRO RC # for this?
>> > >
>> > >
>> > > On May 25, 2018 at 14:53:25, Nick Allen (n...@nickallen.org) wrote:
>> > >
>> > > +1 Release this package as Apache Metron 0.5.0-RC1
>> > >
>> > > Ran through all validation steps using the `metron-rc-check` script,
>> > which
>> > > included running all the tests, license checks, and spun-up the CentOS
>> > dev
>> > > environment successfully.
>> > >
>> > >
>> > >
>> > > On Fri, May 25, 2018 at 1:40 PM, Nick Allen 
>> wrote:
>> > >
>> > > > This release does not contain an updated Bro plugin so the RC check
>> > > script
>> > > > does not currently work. Try using the patch at
>> > > > https://github.com/apache/metron/pull/1034.
>> > > >
>> > > >
>> > > > On Thu, May 24, 2018 at 3:23 PM, Justin Leet 
>> wrote:
>> > > >
>> > > >> Hi all,
>> > > >>
>> > > >> This is a call to vote on releasing Apache Metron 0.5.0
>> > > >>
>> > > >> Full list of changes in this release:
>> > > >> https://dist.apache.org/repos/dist/dev/metron/0.5.0-RC1/CHANGES
>> > > >>
>> > > >> The tag/commit to be voted upon is:
>> > > >>
>> > > >> (apache/metron) apache-metron-0.5.0-rc1
>> > > >>
>> > > >> The source archive being voted upon can be found here:
>> > > >> https://dist.apache.org/repos/dist/dev/metron/0.5.0-RC1/
>> > > >> apache-metron-0.5.0-rc1.tar.gz
>> > > >>
>> > > >> Other release files, signatures and digests can be found here:
>> > > >> https://dist.apache.org/repos/dist/dev/metron/0.5.0-RC1/
>> > > >>
>> > > >> The release artifacts are signed with the following key:
>> > > >> https://dist.apache.org/repos/dist/dev/metron/0.5.0-RC1/KEYS
>> > > >>
>> > > >> Please vote on releasing this package as Apache Metron 0.5.0-RC1
>> > > >>
>> > > >> When voting, please list the actions taken to verify the release.
>> > > >>
>> > > >> Recommended build validation and verification instructions are
>> posted
>> > > >> here:
>> > > >> https://cwiki.apache.org/confluence/display/METRON/Verifying
>> +Builds
>> > > >>
>> > > >> This vote will be open until 4pm EDT on Tuesday May 29 2018, to
>> > account
>> > > >> for
>> > > >> the weekend.
>> > > >>
>> > > >> [ ] +1 Release this package as Apache Metron 0.3.0-RC1
>> > > >>
>> > > >> [ ] 0 No opinion
>> > > >>
>> > > >> [ ] -1 Do not release this package because...
>> > > >>
>> > > >
>> > > >
>> > >
>> > --
>> >
>> > Jon
>> >
>> >
>>
>
>

Re: [VOTE] Metron Release Candidate 0.5.0-RC1

[Justin Leet]

Hi all,

Before I put out a new RC, I'm inclined to hold off until
https://issues.apache.org/jira/browse/METRON-1586 is in.  The impact is
that individual alerts in the UI can't be viewed without alerting the
global config manually.  It can be worked around, but I think it's a little
onerous to have everyone have to edit the global config.

Does anyone object to waiting for a new RC until that's taken care of?

Justin

On Tue, May 29, 2018 at 5:18 PM, Casey Stella  wrote:

> Just a question, do we need anything new in the Upgrading.md for this
> release?  Any migration that we expect people to do?
>
> On Tue, May 29, 2018 at 11:30 AM Nick Allen  wrote:
>
>> METRON-1544 was just merged into master.
>>
>>
>> On Tue, May 29, 2018 at 2:16 PM, Justin Leet 
>> wrote:
>>
>> > I'm going to go ahead and cancel RC1, since METRON-1544 looks pretty
>> set.
>> >
>> > A new release candidate will be cut.
>> >
>> > Results (including my own vote):
>> > +1
>> > Nick Allen
>> >
>> >
>> > -1
>> > Otto Fowler
>> > Justin Leet
>> >
>> > On Tue, May 29, 2018 at 10:39 AM, Justin Leet 
>> > wrote:
>> >
>> >> I didn't realize METRON-1544 wasn't in.  I'm definitely okay with
>> >> cancelling the vote, and kicking out a new RC.
>> >>
>> >> On Tue, May 29, 2018 at 7:11 AM, Otto Fowler 
>> >> wrote:
>> >>
>> >>> -1 (binding)
>> >>>
>> >>> My yield for building this it terrible.  1 in 3.
>> >>>
>> >>> I propose https://github.com/apache/metron/pull/1015 inclusion.
>> >>>
>> >>>
>> >>> On May 27, 2018 at 17:50:43, Nick Allen (n...@nickallen.org) wrote:
>> >>>
>> >>> No, the PR for this transient issue is still under review.
>> >>>
>> >>> On Sun, May 27, 2018 at 10:53 AM, Otto Fowler <
>> ottobackwa...@gmail.com>
>> >>> wrote:
>> >>>
>> >>> >
>> >>> > Failed tests:
>> >>> >   CachingStellarProcessorTest.testCaching:73 expected:<6> but
>> was:<5>
>> >>> >
>> >>> > I thought we landed a fix for this?
>> >>> >
>> >>> >
>> >>> > On May 27, 2018 at 08:24:19, zeo...@gmail.com (zeo...@gmail.com)
>> >>> wrote:
>> >>> >
>> >>> > We did discuss doing a release since there were two new commits,
>> but I
>> >>> > don't think it was included in this round.
>> >>> >
>> >>> > Jon
>> >>> >
>> >>> > On Sat, May 26, 2018, 10:22 Otto Fowler 
>> >>> wrote:
>> >>> >
>> >>> > > Is there a BRO RC # for this?
>> >>> > >
>> >>> > >
>> >>> > > On May 25, 2018 at 14:53:25, Nick Allen (n...@nickallen.org)
>> wrote:
>> >>> > >
>> >>> > > +1 Release this package as Apache Metron 0.5.0-RC1
>> >>> > >
>> >>> > > Ran through all validation steps using the `metron-rc-check`
>> script,
>> >>> > which
>> >>> > > included running all the tests, license checks, and spun-up the
>> >>> CentOS
>> >>> > dev
>> >>> > > environment successfully.
>> >>> > >
>> >>> > >
>> >>> > >
>> >>> > > On Fri, May 25, 2018 at 1:40 PM, Nick Allen 
>> >>> wrote:
>> >>> > >
>> >>> > > > This release does not contain an updated Bro plugin so the RC
>> check
>> >>> > > script
>> >>> > > > does not currently work. Try using the patch at
>> >>> > > > https://github.com/apache/metron/pull/1034.
>> >>> > > >
>> >>> > > >
>> >>> > > > On Thu, May 24, 2018 at 3:23 PM, Justin Leet 
>> >>> wrote:
>> >>> > > >
>> >>> > > >> Hi all,
>> >>> > > >>
>> >>> > > >> This is a call to vote on releasing Apache Metron 0.5.0
>> >>> > > >>
>> >>> > > >> Full list of changes in this release:
>> >>> > > >> https://dist.apache.org/repos/dist/dev/metron/0.5.0-RC1/
>> CHANGES
>> >>> > > >>
>> >>> > > >> The tag/commit to be voted upon is:
>> >>> > > >>
>> >>> > > >> (apache/metron) apache-metron-0.5.0-rc1
>> >>> > > >>
>> >>> > > >> The source archive being voted upon can be found here:
>> >>> > > >> https://dist.apache.org/repos/dist/dev/metron/0.5.0-RC1/
>> >>> > > >> apache-metron-0.5.0-rc1.tar.gz
>> >>> > > >>
>> >>> > > >> Other release files, signatures and digests can be found here:
>> >>> > > >> https://dist.apache.org/repos/dist/dev/metron/0.5.0-RC1/
>> >>> > > >>
>> >>> > > >> The release artifacts are signed with the following key:
>> >>> > > >> https://dist.apache.org/repos/dist/dev/metron/0.5.0-RC1/KEYS
>> >>> > > >>
>> >>> > > >> Please vote on releasing this package as Apache Metron
>> 0.5.0-RC1
>> >>> > > >>
>> >>> > > >> When voting, please list the actions taken to verify the
>> release.
>> >>> > > >>
>> >>> > > >> Recommended build validation and verification instructions are
>> >>> posted
>> >>> > > >> here:
>> >>> > > >> https://cwiki.apache.org/confluence/display/METRON/Verifying
>> >>> +Builds
>> >>> > > >>
>> >>> > > >> This vote will be open until 4pm EDT on Tuesday May 29 2018, to
>> >>> > account
>> >>> > > >> for
>> >>> > > >> the weekend.
>> >>> > > >>
>> >>> > > >> [ ] +1 Release this package as Apache Metron 0.3.0-RC1
>> >>> > > >>
>> >>> > > >> [ ] 0 No opinion
>> >>> > > >>
>> >>> > > >> [ ] -1 Do not release this package because...
>> >>> > > >>
>> >>> > > >
>> >>> > > >
>> >>> > >
>> >>> > --
>> >>> >
>> >>> > Jon
>> >>> >
>> >>> >
>> >>>
>> >>
>> >>
>> >
>>
>

[VOTE] Metron Release Candidate 0.5.0-RC2

[Justin Leet]

Hi all,

This is a call to vote on releasing Apache Metron 0.5.0

Full list of changes in this release:
https://dist.apache.org/repos/dist/dev/metron/0.5.0-RC2/CHANGES

The tag/commit to be voted upon is:

(apache/metron) apache-metron-0.5.0-rc2

The source archive being voted upon can be found here:
https://dist.apache.org/repos/dist/dev/metron/0.5.0-RC2/apac
he-metron-0.5.0-rc2.tar.gz

Other release files, signatures and digests can be found here:
https://dist.apache.org/repos/dist/dev/metron/0.5.0-RC2/

The release artifacts are signed with the following key:
https://dist.apache.org/repos/dist/dev/metron/0.5.0-RC2/KEYS

Please vote on releasing this package as Apache Metron 0.5.0-RC2

When voting, please list the actions taken to verify the release.

Recommended build validation and verification instructions are posted
here:
https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds

This vote will be open until 3pm EDT on Tuesday June 5 2018, to account for
the weekend.

[ ] +1 Release this package as Apache Metron 0.5.0-RC2

[ ] 0 No opinion

[ ] -1 Do not release this package because...

Re: [VOTE] Metron Release Candidate 0.5.0-RC2

[Justin Leet]

This includes a couple fixes from master, in particular two issues that
were problematic, METRON-1586
<https://issues.apache.org/jira/browse/METRON-1586> and METRON-1544
<https://issues.apache.org/jira/browse/METRON-1544>.

Justin

On Thu, May 31, 2018 at 2:30 PM, Justin Leet  wrote:

> Hi all,
>
> This is a call to vote on releasing Apache Metron 0.5.0
>
> Full list of changes in this release:
> https://dist.apache.org/repos/dist/dev/metron/0.5.0-RC2/CHANGES
>
> The tag/commit to be voted upon is:
>
> (apache/metron) apache-metron-0.5.0-rc2
>
> The source archive being voted upon can be found here:
> https://dist.apache.org/repos/dist/dev/metron/0.5.0-RC2/apac
> he-metron-0.5.0-rc2.tar.gz
>
> Other release files, signatures and digests can be found here:
> https://dist.apache.org/repos/dist/dev/metron/0.5.0-RC2/
>
> The release artifacts are signed with the following key:
> https://dist.apache.org/repos/dist/dev/metron/0.5.0-RC2/KEYS
>
> Please vote on releasing this package as Apache Metron 0.5.0-RC2
>
> When voting, please list the actions taken to verify the release.
>
> Recommended build validation and verification instructions are posted
> here:
> https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds
>
> This vote will be open until 3pm EDT on Tuesday June 5 2018, to account
> for the weekend.
>
> [ ] +1 Release this package as Apache Metron 0.5.0-RC2
>
> [ ] 0 No opinion
>
> [ ] -1 Do not release this package because...
>

[RESULT][VOTE] Metron Release Candidate 0.5.0-RC2

[Justin Leet]

The vote has passed.  Including my +1, the voting was:
3 binding +1’s
1 non-binding +1’s
no 0’s
no -1’s.

Release process is underway and will be announced to dev@ and user@ lists.

Justin

Re: [VOTE] Metron Release Candidate 0.5.0-RC1

[Justin Leet]

I didn't realize METRON-1544 wasn't in.  I'm definitely okay with
cancelling the vote, and kicking out a new RC.

On Tue, May 29, 2018 at 7:11 AM, Otto Fowler 
wrote:

> -1 (binding)
>
> My yield for building this it terrible.  1 in 3.
>
> I propose https://github.com/apache/metron/pull/1015 inclusion.
>
>
> On May 27, 2018 at 17:50:43, Nick Allen (n...@nickallen.org) wrote:
>
> No, the PR for this transient issue is still under review.
>
> On Sun, May 27, 2018 at 10:53 AM, Otto Fowler 
> wrote:
>
> >
> > Failed tests:
> >   CachingStellarProcessorTest.testCaching:73 expected:<6> but was:<5>
> >
> > I thought we landed a fix for this?
> >
> >
> > On May 27, 2018 at 08:24:19, zeo...@gmail.com (zeo...@gmail.com) wrote:
> >
> > We did discuss doing a release since there were two new commits, but I
> > don't think it was included in this round.
> >
> > Jon
> >
> > On Sat, May 26, 2018, 10:22 Otto Fowler  wrote:
> >
> > > Is there a BRO RC # for this?
> > >
> > >
> > > On May 25, 2018 at 14:53:25, Nick Allen (n...@nickallen.org) wrote:
> > >
> > > +1 Release this package as Apache Metron 0.5.0-RC1
> > >
> > > Ran through all validation steps using the `metron-rc-check` script,
> > which
> > > included running all the tests, license checks, and spun-up the CentOS
> > dev
> > > environment successfully.
> > >
> > >
> > >
> > > On Fri, May 25, 2018 at 1:40 PM, Nick Allen 
> wrote:
> > >
> > > > This release does not contain an updated Bro plugin so the RC check
> > > script
> > > > does not currently work. Try using the patch at
> > > > https://github.com/apache/metron/pull/1034.
> > > >
> > > >
> > > > On Thu, May 24, 2018 at 3:23 PM, Justin Leet 
> wrote:
> > > >
> > > >> Hi all,
> > > >>
> > > >> This is a call to vote on releasing Apache Metron 0.5.0
> > > >>
> > > >> Full list of changes in this release:
> > > >> https://dist.apache.org/repos/dist/dev/metron/0.5.0-RC1/CHANGES
> > > >>
> > > >> The tag/commit to be voted upon is:
> > > >>
> > > >> (apache/metron) apache-metron-0.5.0-rc1
> > > >>
> > > >> The source archive being voted upon can be found here:
> > > >> https://dist.apache.org/repos/dist/dev/metron/0.5.0-RC1/
> > > >> apache-metron-0.5.0-rc1.tar.gz
> > > >>
> > > >> Other release files, signatures and digests can be found here:
> > > >> https://dist.apache.org/repos/dist/dev/metron/0.5.0-RC1/
> > > >>
> > > >> The release artifacts are signed with the following key:
> > > >> https://dist.apache.org/repos/dist/dev/metron/0.5.0-RC1/KEYS
> > > >>
> > > >> Please vote on releasing this package as Apache Metron 0.5.0-RC1
> > > >>
> > > >> When voting, please list the actions taken to verify the release.
> > > >>
> > > >> Recommended build validation and verification instructions are
> posted
> > > >> here:
> > > >> https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds
> > > >>
> > > >> This vote will be open until 4pm EDT on Tuesday May 29 2018, to
> > account
> > > >> for
> > > >> the weekend.
> > > >>
> > > >> [ ] +1 Release this package as Apache Metron 0.3.0-RC1
> > > >>
> > > >> [ ] 0 No opinion
> > > >>
> > > >> [ ] -1 Do not release this package because...
> > > >>
> > > >
> > > >
> > >
> > --
> >
> > Jon
> >
> >
>

Re: [DISCUSS] Merging Solr feature branch (METRON-1416) into master

[Justin Leet]

The PR has two +1's at this point (and I'm implicitly +1). In the interest
of full disclosure, both are from people who made contributions of varying
degrees to the branch.

Are there any objections to merging the feature branch into master at this
point?

On Fri, Jun 22, 2018 at 1:12 PM Justin Leet  wrote:

> That's more or less why I didn't flesh out testing.  Might be worth
> spinning up full dev and the site-book to smoke test, but the branch should
> be in a good state.  I figured if we get a couple +1's on the PR, it's
> essentially voting anyway, but this is pretty new in terms of process.
>
>
>
> On Fri, Jun 22, 2018 at 12:53 PM Otto Fowler 
> wrote:
>
>> If all the PR’s are on master->feature branch.  Why do we need testing?
>> this is almost a vote situation.
>>
>>
>>
>>
>> On June 22, 2018 at 12:01:11, Justin Leet (justinjl...@gmail.com) wrote:
>>
>> The (formerly) active PRs are now merged in and closed.
>>
>> We don't seem to have defined way to merge a feature branch into master
>> (unless I missed it), so I went ahead and opened a PR against the parent
>> ticket. Please see #1076 <https://github.com/apache/metron/pull/1076>.
>>
>> I haven't fleshed out testing and so on for the PR description, although
>> if
>> we'd like it compiled from the various child PRs against the branch, I
>> can
>> certainly do so.
>>
>> Justin
>>
>> On Thu, Jun 21, 2018 at 6:46 PM Michael Miklavcic <
>> michael.miklav...@gmail.com> wrote:
>>
>> > +1 let's do it.
>> >
>> > On Thu, Jun 21, 2018, 2:01 PM Nick Allen  wrote:
>> >
>> > > +1 I think we should merge ASAP and kill the feature branch. I think
>> the
>> > > work has well surpassed the level required to get it into master.
>> > >
>> > > On Thu, Jun 21, 2018 at 1:20 PM, Justin Leet 
>> > > wrote:
>> > >
>> > > > Hi All,
>> > > >
>> > > > The Solr branch (/feature/METRON-1416-upgrade-solr
>> > > > <
>> > https://github.com/apache/metron/tree/feature/METRON-1416-upgrade-solr
>> > > >),
>> > > > has been progressing for a while now. I'd like to open up
>> discussion
>> > > > around what it takes to get it into master.
>> > > >
>> > > > The JIRA for tracking this feature branch is METRON-1416
>> > > > <https://issues.apache.org/jira/browse/METRON-1416>.
>> > > >
>> > > > As shown in the JIRA, the majority of tasks are complete, with a
>> few
>> > > > outstanding issues. Of these, I believe these are the main ones of
>> > > interest
>> > > > to this discussion.
>> > > >
>> > > > - METRON-1629 <https://issues.apache.org/jira/browse/METRON-1629>
>> -
>> > > > There is an active PR #1072 <
>> > > https://github.com/apache/metron/pull/1072
>> > > > >
>> > > > - METRON-1609 <https://issues.apache.org/jira/browse/METRON-1609>
>> -
>> > > > There is an active PR #1056 <
>> > > https://github.com/apache/metron/pull/1056
>> > > > >
>> > > > - METRON-1602 <https://issues.apache.org/jira/browse/METRON-1602>
>> -
>> > > > Full
>> > > > dev can run with Solr without this, it would simply be more
>> > > convenient.
>> > > > - METRON-1632 <https://issues.apache.org/jira/browse/METRON-1632>
>> -
>> > > > Causes a metaalert specific issue where UI filtering on
>> > > > source.type:metaalert fails. More detail is on the Jira.
>> > > > - Two validation tickets. It's been run up on multinode, and manual
>> > > > testing has happened (and I'm will be seen a bit more on the final
>> > PR
>> > > by
>> > > > various reviewers), so I'm inclined to just leave these open until
>> > > we're
>> > > > good to go. Let me know if we want to handle this differently.
>> > > >
>> > > > I'm of the opinion both of the active PRs need to be merged before
>> we
>> > > merge
>> > > > this into master, especially the documentation one. The other two
>> > > tickets
>> > > > can be done in the future; one can be worked around and one is a
>> > > metaalert
>> > > > specific issue that primarily effects the alerts UI.
>> > > >
>> > 

Re: [DISCUSS] Merging Solr feature branch (METRON-1416) into master

[Justin Leet]

The (formerly) active PRs are now merged in and closed.

We don't seem to have defined way to merge a feature branch into master
(unless I missed it), so I went ahead and opened a PR against the parent
ticket. Please see #1076 <https://github.com/apache/metron/pull/1076>.

I haven't fleshed out testing and so on for the PR description, although if
we'd like it compiled from the various child PRs against the branch, I can
certainly do so.

Justin

On Thu, Jun 21, 2018 at 6:46 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> +1 let's do it.
>
> On Thu, Jun 21, 2018, 2:01 PM Nick Allen  wrote:
>
> > +1 I think we should merge ASAP and kill the feature branch.  I think the
> > work has well surpassed the level required to get it into master.
> >
> > On Thu, Jun 21, 2018 at 1:20 PM, Justin Leet 
> > wrote:
> >
> > > Hi All,
> > >
> > > The Solr branch (/feature/METRON-1416-upgrade-solr
> > > <
> https://github.com/apache/metron/tree/feature/METRON-1416-upgrade-solr
> > >),
> > > has been progressing for a while now.  I'd like to open up discussion
> > > around what it takes to get it into master.
> > >
> > > The JIRA for tracking this feature branch is METRON-1416
> > > <https://issues.apache.org/jira/browse/METRON-1416>.
> > >
> > > As shown in the JIRA, the majority of tasks are complete, with a few
> > > outstanding issues. Of these, I believe these are the main ones of
> > interest
> > > to this discussion.
> > >
> > >- METRON-1629 <https://issues.apache.org/jira/browse/METRON-1629> -
> > >There is an active PR #1072 <
> > https://github.com/apache/metron/pull/1072
> > > >
> > >- METRON-1609 <https://issues.apache.org/jira/browse/METRON-1609> -
> > >There is an active PR #1056 <
> > https://github.com/apache/metron/pull/1056
> > > >
> > >- METRON-1602 <https://issues.apache.org/jira/browse/METRON-1602> -
> > > Full
> > >dev can run with Solr without this, it would simply be more
> > convenient.
> > >- METRON-1632 <https://issues.apache.org/jira/browse/METRON-1632> -
> > >Causes a metaalert specific issue where UI filtering on
> > >source.type:metaalert fails. More detail is on the Jira.
> > >- Two validation tickets.  It's been run up on multinode, and manual
> > >testing has happened (and I'm will be seen a bit more on the final
> PR
> > by
> > >various reviewers), so I'm inclined to just leave these open until
> > we're
> > >good to go.  Let me know if we want to handle this differently.
> > >
> > > I'm of the opinion both of the active PRs need to be merged before we
> > merge
> > > this into master, especially the documentation one.  The other two
> > tickets
> > > can be done in the future; one can be worked around and one is a
> > metaalert
> > > specific issue that primarily effects the alerts UI.
> > >
> > > As the branch has grown and diverged from master, it's gotten
> > increasingly
> > > unwieldy to maintain (and I think it's worth a follow-on discussion
> about
> > > how we manage refactorings that happen in these sorts of branches).  I
> > know
> > > there's been at least a couple merges from master that have been
> > > nontrivially difficult and required careful testing, particularly
> around
> > > the DAO layer, to avoid regressions in both code and tests.
> > >
> > > The feature set is pretty complete.  The UI works, barring the
> metaalert
> > > issue.  Much of the backend has been refactored and seen improved test
> > > coverage benefiting both Solr and Elasticsearch.  The main difference
> > > between ES and Solr is the lack of the equivalent visualizations to
> > > Kibana.  I don't believe the feature branch needs to wait for this, as
> > it's
> > > pretty standalone work that can be added as usage and demand dictates.
> > >
> > > I'm of the opinion that the benefits of getting the branch into master
> > > outweighs the issues still present, especially in terms of making
> > > refactoring and features available and easing the dev burden.  The
> > > remaining tickets are Solr specific, and ES functions as it does in
> > master.
> > >
> > > Are there any must-haves before we bring this branch back?  Are there
> any
> > > other concerns we have before a final PR is opened (pending completion
> of
> > > active PRs and any other must-haves)?
> > >
> > > Justin
> > >
> >
>

Re: [DISCUSS] Merging Solr feature branch (METRON-1416) into master

[Justin Leet]

The Solr feature branch is in now in master.  Note that there is no
METRON-1416 commit in the logs because all subtasks are committed under
their own JIRA and are in the history to maintain attribution.

On Tue, Jun 26, 2018 at 1:26 PM Otto Fowler  wrote:

> +1
>
>
> On June 26, 2018 at 11:43:39, Justin Leet (justinjl...@gmail.com) wrote:
>
> The PR has two +1's at this point (and I'm implicitly +1). In the interest
> of full disclosure, both are from people who made contributions of varying
> degrees to the branch.
>
> Are there any objections to merging the feature branch into master at this
> point?
>
> On Fri, Jun 22, 2018 at 1:12 PM Justin Leet  wrote:
>
>> That's more or less why I didn't flesh out testing.  Might be worth
>> spinning up full dev and the site-book to smoke test, but the branch should
>> be in a good state.  I figured if we get a couple +1's on the PR, it's
>> essentially voting anyway, but this is pretty new in terms of process.
>>
>>
>>
>> On Fri, Jun 22, 2018 at 12:53 PM Otto Fowler 
>> wrote:
>>
>>> If all the PR’s are on master->feature branch.  Why do we need testing?
>>> this is almost a vote situation.
>>>
>>>
>>>
>>>
>>> On June 22, 2018 at 12:01:11, Justin Leet (justinjl...@gmail.com) wrote:
>>>
>>> The (formerly) active PRs are now merged in and closed.
>>>
>>> We don't seem to have defined way to merge a feature branch into master
>>> (unless I missed it), so I went ahead and opened a PR against the parent
>>> ticket. Please see #1076 <https://github.com/apache/metron/pull/1076>.
>>>
>>> I haven't fleshed out testing and so on for the PR description, although
>>> if
>>> we'd like it compiled from the various child PRs against the branch, I
>>> can
>>> certainly do so.
>>>
>>> Justin
>>>
>>> On Thu, Jun 21, 2018 at 6:46 PM Michael Miklavcic <
>>> michael.miklav...@gmail.com> wrote:
>>>
>>> > +1 let's do it.
>>> >
>>> > On Thu, Jun 21, 2018, 2:01 PM Nick Allen  wrote:
>>> >
>>> > > +1 I think we should merge ASAP and kill the feature branch. I think
>>> the
>>> > > work has well surpassed the level required to get it into master.
>>> > >
>>> > > On Thu, Jun 21, 2018 at 1:20 PM, Justin Leet 
>>> > > wrote:
>>> > >
>>> > > > Hi All,
>>> > > >
>>> > > > The Solr branch (/feature/METRON-1416-upgrade-solr
>>> > > > <
>>> > https://github.com/apache/metron/tree/feature/METRON-1416-upgrade-solr
>>> > > >),
>>> > > > has been progressing for a while now. I'd like to open up
>>> discussion
>>> > > > around what it takes to get it into master.
>>> > > >
>>> > > > The JIRA for tracking this feature branch is METRON-1416
>>> > > > <https://issues.apache.org/jira/browse/METRON-1416>.
>>> > > >
>>> > > > As shown in the JIRA, the majority of tasks are complete, with a
>>> few
>>> > > > outstanding issues. Of these, I believe these are the main ones of
>>> > > interest
>>> > > > to this discussion.
>>> > > >
>>> > > > - METRON-1629 <https://issues.apache.org/jira/browse/METRON-1629>
>>> -
>>> > > > There is an active PR #1072 <
>>> > > https://github.com/apache/metron/pull/1072
>>> > > > >
>>> > > > - METRON-1609 <https://issues.apache.org/jira/browse/METRON-1609>
>>> -
>>> > > > There is an active PR #1056 <
>>> > > https://github.com/apache/metron/pull/1056
>>> > > > >
>>> > > > - METRON-1602 <https://issues.apache.org/jira/browse/METRON-1602>
>>> -
>>> > > > Full
>>> > > > dev can run with Solr without this, it would simply be more
>>> > > convenient.
>>> > > > - METRON-1632 <https://issues.apache.org/jira/browse/METRON-1632>
>>> -
>>> > > > Causes a metaalert specific issue where UI filtering on
>>> > > > source.type:metaalert fails. More detail is on the Jira.
>>> > > > - Two validation tickets. It's been run up on multinode, and manual
>>> > > > testing has happened (and I'm will be seen a bit more on the final
&

Re: [DISCUSS] Removing Markdown files from rat exclusion

[Justin Leet]

I've updated the PR to add the header to a new MD file that went in.

I've also commented on all PRs that I saw that would potentially be
problematic were they to go into master if they weren't merged first.

Once the updated PR gets the +1's reaffirmed, it will be merged into master
and Markdown headers will be enforced properly going forwad.

On Sun, Dec 24, 2017 at 8:09 PM, Justin Leet <justinjl...@gmail.com> wrote:

> I'm gonna let this percolate until Wednesday or so, assuming conversation
> doesn't reach a natural tipping point.  I'm inclined to agree with Nick,
> but I also don't want to resolve anything in a way that even potentially
> causes master problems until at least after Christmas has a chance to
> settle down for people.  At that point, assuming current course, I'll take
> a real run through of the PRs (and leave comments as appropriate, before
> merging.
>
> Obviously if anyone has suggestions or alternatives, still feel encouraged
> to respond.
>
> On Sat, Dec 23, 2017 at 11:17 AM, Nick Allen <n...@nickallen.org> wrote:
>
>> > This would result in master breaking (although it's a pretty easy fix).
>>
>> I am not concerned and don't think we need to wait on merging PR #883.
>>
>> Can you add a comment to each of the PRs that you identified?  We can make
>> sure that each gets merged with master before they go in.
>>
>>
>>
>> On Sat, Dec 23, 2017 at 11:08 AM, Justin Leet <justinjl...@gmail.com>
>> wrote:
>>
>> > I have a PR currently out (https://github.com/apache/metron/pull/883)
>> that
>> > removes the rat exclusion on Markdown files. There was a discuss thread
>> > awhile back about adding the header and removing the exclusion where it
>> was
>> > agreed that we should do this to meet Apache requirements.
>> Unfortunately,
>> > it didn't get any follow on.
>> >
>> > Right now the PR has two +1s, but it could potentially be problematic
>> with
>> > existing PRs.
>> >
>> > Any PR that meets two conditions could potentially be problematic
>> > 1. It adds a new Markdown file
>> > 2. Travis was run before the exclusion PR was merged.
>> >
>> > This is because whoever does the merge might not realize that master
>> should
>> > be merged in and the markdown file updated with the Apache header.  This
>> > would result in master breaking (although it's a pretty easy fix).
>> >
>> > Are we okay with merging this now/soon, or do we want to take additional
>> > steps to ensure we don't run into issues? If we want, I can run through
>> the
>> > PRs and add comments before merging.  Is this sufficient to at least
>> > mitigate the most obvious problems?
>> >
>> > I took a very quick glance through some of the most recent PRs and only
>> two
>> > really stood out to me (although I'm sure there are older ones that are
>> > still being worked on or looked at)
>> >
>> > METRON-1380 https://github.com/apache/metron/pull/882 - Adds a new
>> > markdown
>> > file, but Travis failed. If it gets fixed before this PR is merged we
>> could
>> > run into the problem
>> > METRON-1351 https://github.com/apache/metron/pull/868 - Adds a new
>> > markdown
>> > file and Travis succeeded. This would break master if merged as-is
>> after my
>> > PR.
>> >
>>
>
>

Re: [VOTE] Metron Release Candidate 0.4.2-RC2

[Justin Leet]

+1 validated with Otto's script

* Checksums
* Signatures
* Build
* Tests

On Tue, Dec 19, 2017 at 11:47 PM, Anand Subramanian <
asubraman...@hortonworks.com> wrote:

>
> * mvn clean package at root level
> * mvn clean package -Pbuild-rpms at metron-deployment level and generate
> RPMs
> * Brought up Metron stack on 12-node CentOS 7 openstack cluster using the
> generated RPMs
> * Bro, YAF and snort - ingest into kafka topics and validated indices
> * Add squid telemetry, ingest into kafka topic and validated indices
> * Management UI, Alerts UI and Swagger UI sanity check
>
> +1 (non-binding)
>
>
> Regards,
> Anand
>
>
>
>
> On 12/20/17, 3:11 AM, "Casey Stella"  wrote:
>
> >+1 validated via Otto's script
> >* Checksums
> >* Sigs
> >* Build
> >* Full dev validation
> >
> >On Tue, Dec 19, 2017 at 2:45 PM, Nick Allen  wrote:
> >
> >> +1  I validated using Otto's great script.
> >>
> >> * Validated the list of changes
> >> * Checksums
> >> * Sigs
> >> * Build
> >> * Tests
> >> * Full Dev
> >>
> >> On Tue, Dec 19, 2017 at 6:23 AM, Matt Foley  wrote:
> >>
> >> > Colleagues,
> >> > This is a call to vote on releasing Apache Metron 0.4.2 and its
> >> associated
> >> > metron-bro-plugin-kafka 0.1.0.
> >> > The release candidate is available at https://dist.apache.org/repos/
> >> > dist/dev/metron/0.4.2-RC2/
> >> >
> >> > Full list of changes in this release:
> >> > https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC2/CHANGES and
> >> > https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC2/
> >> CHANGES.bro-plugin
> >> >
> >> > The github tags to be voted upon are:
> >> > (apache/metron) apache-metron-0.4.2-rc2 and (apache/metron-bro-plugin-
> >> kafka)
> >> > 0.1
> >> >
> >> > The source archives being voted upon can be found here:
> >> > https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC2/
> >> > apache-metron-0.4.2-rc2.tar.gz
> >> > https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC2/
> >> > apache-metron-bro-plugin-kafka_0.1.0.tar.gz
> >> >
> >> > The site-book is at:
> >> > https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC2/
> >> > site-book/index.html
> >> >
> >> > Other release files, signatures and digests can be found here:
> >> > https://dist.apache.org/repos/dist/dev/metron/0.4.2-RC2/
> >> >
> >> > The release artifacts are signed with the following key:
> >> > 4169 AA27 ECB3 1663 in https://dist.apache.org/repos/
> >> > dist/dev/metron/0.4.2-RC2/KEYS
> >> >
> >> > Please vote on releasing this package as Apache Metron 0.4.2 and
> Apache
> >> > Metron-bro-plugin-kafka 0.1.0
> >> >
> >> > When voting, please list the actions taken to verify the release.
> >> >
> >> > Recommended build validation and verification instructions are posted
> >> here:
> >> > https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds
> >> > or you are encouraged to try the new release verification script that
> >> Otto
> >> > published via email on 11 Dec, available at
> >> > https://github.com/ottobackwards/Metron-and-Nifi-
> >> > Scripts/blob/master/metron/metron-rc-check
> >> >
> >> > This vote will be open until 9am PST on Friday 22 Dec 2017.
> >> >
> >> > Thank you,
> >> > --Matt
> >> >
> >> >
> >> >
> >> >
> >> >
> >>
>

Re: [DISCUSS] Removing Markdown files from rat exclusion

[Justin Leet]

I'm gonna let this percolate until Wednesday or so, assuming conversation
doesn't reach a natural tipping point.  I'm inclined to agree with Nick,
but I also don't want to resolve anything in a way that even potentially
causes master problems until at least after Christmas has a chance to
settle down for people.  At that point, assuming current course, I'll take
a real run through of the PRs (and leave comments as appropriate, before
merging.

Obviously if anyone has suggestions or alternatives, still feel encouraged
to respond.

On Sat, Dec 23, 2017 at 11:17 AM, Nick Allen <n...@nickallen.org> wrote:

> > This would result in master breaking (although it's a pretty easy fix).
>
> I am not concerned and don't think we need to wait on merging PR #883.
>
> Can you add a comment to each of the PRs that you identified?  We can make
> sure that each gets merged with master before they go in.
>
>
>
> On Sat, Dec 23, 2017 at 11:08 AM, Justin Leet <justinjl...@gmail.com>
> wrote:
>
> > I have a PR currently out (https://github.com/apache/metron/pull/883)
> that
> > removes the rat exclusion on Markdown files. There was a discuss thread
> > awhile back about adding the header and removing the exclusion where it
> was
> > agreed that we should do this to meet Apache requirements.
> Unfortunately,
> > it didn't get any follow on.
> >
> > Right now the PR has two +1s, but it could potentially be problematic
> with
> > existing PRs.
> >
> > Any PR that meets two conditions could potentially be problematic
> > 1. It adds a new Markdown file
> > 2. Travis was run before the exclusion PR was merged.
> >
> > This is because whoever does the merge might not realize that master
> should
> > be merged in and the markdown file updated with the Apache header.  This
> > would result in master breaking (although it's a pretty easy fix).
> >
> > Are we okay with merging this now/soon, or do we want to take additional
> > steps to ensure we don't run into issues? If we want, I can run through
> the
> > PRs and add comments before merging.  Is this sufficient to at least
> > mitigate the most obvious problems?
> >
> > I took a very quick glance through some of the most recent PRs and only
> two
> > really stood out to me (although I'm sure there are older ones that are
> > still being worked on or looked at)
> >
> > METRON-1380 https://github.com/apache/metron/pull/882 - Adds a new
> > markdown
> > file, but Travis failed. If it gets fixed before this PR is merged we
> could
> > run into the problem
> > METRON-1351 https://github.com/apache/metron/pull/868 - Adds a new
> > markdown
> > file and Travis succeeded. This would break master if merged as-is after
> my
> > PR.
> >
>

Re: [DISCUSS] Generating and Interacting with serialized summary objects

[Justin Leet]

I agree with the general sentiment that we can tailor specific use cases
via UI, and I'm worried that the use case specific solution (particularly
in light of the note that it's not even general to the class of bloom
filter problems, let alone an actually general problem) becomes more work
than this as soon as about 2 more uses cases actually get realized.
Pushing that to the UI lets people solve a variety of problems if they
really want to dig in, while still giving flexibility to provide a more
tailored experience for what we discover the 80% cases are in practice.

Keeping in mind I am mostly unfamiliar with the extractor config itself, I
am wondering if it makes sense to split up the config a bit.  While a lot
of implementation details are shared, maybe the extractor config itself
should be refactored into a couple parts analogous to ETL (as a follow on
task, I think if this is true, it predates Casey's proposed change).  It
doesn't necessarily make it less complex, but it might make it more easily
digestible if it's split up by idea (parsing, transformation, etc.).

Re: Mike's point, I don't think we want the actual processing broken up as
ETL, but the representation to the user in terms of configuration could be
similar (Since we're already doing parsing and transformation). We don't
have to implement it as an ETL pipeline, but it does potentially offer the
user a way to quickly grasp what the JSON blob is actually specifying.
Making it easy to understand, even if it's not the ideal way to interact is
potentially still a win.

On Thu, Jan 4, 2018 at 1:28 PM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> I mentioned this earlier, but I'll reiterate that I think this approach
> gives us the ability to make specific use cases via a UI, or other
> interface should we choose to add one, while keeping the core adaptable and
> flexible. This is ideal for middle tier as I think this effectively gives
> us the ability to pivot to other use cases very easily while not being so
> generic as to be useless. The fact that you were able to create this as
> quickly as you did seems to me directly related to the fact we made the
> decision to keep the loader somewhat flexible rather than very specific.
> The operation ordering and state carry from one phase of processing to the
> next would simply have been inscrutable, if not impossible, with a CLI
> option-only approach. Sure, it's not as simple as "put infile.txt
> outfile.txt", but the alternatives are not that clear either. One might
> argue we could split up the processing pieces as in traditional Hadoop, eg
> ETL: Sqoop ingest -> HDFS -> mapreduce, pig, hive, or spark transform. But
> quite frankly that's going in the *opposite* direction I think we want
> here. That's more complex in terms of moving parts. The config approach
> with pluggable Stellar insulates users from specific implementations, but
> also gives you the ability to pass lower level constructs, eg Spark SQL or
> HiveQL, should the need arise.
>
> In summary, my impressions are that at this point the features and level of
> abstraction feel appropriate to me. I think it buys us 1) learning from a
> starting typosquatting use case, 2) flexibility to change and adapt it
> without affecting users, and 3) enough concrete capability to make more
> specific use cases easy to deliver with a UI.
>
> Cheers,
> Mike
>
> On Jan 4, 2018 9:59 AM, "Casey Stella"  wrote:
>
> > It also occurs to me that even in this situation, it's not a sufficient
> > generalization for just Bloom, but this is a bloom filter of the output
> of
> > the all the typosquatted domains for the domain in each row.  If we
> wanted
> > to hard code, we'd have to hard code specifically the bloom filter *for*
> > typosquatting use-case.  Hard coding this would prevent things like bloom
> > filters containing malicious IPs from a reference source, for instance.
> >
> > On Thu, Jan 4, 2018 at 10:46 AM, Casey Stella 
> wrote:
> >
> > > So, there is value outside of just bloom usage.  The most specific
> > example
> > > of this would be in order to configure a bloom filter, we need to know
> at
> > > least an upper bound of the number of items that are going to be added
> to
> > > the bloom filter.  In order to do that, we need to count the number of
> > > typosquatted domains.  Specifically at https://github.com/
> > > cestella/incubator-metron/tree/typosquat_merge/use-
> > > cases/typosquat_detection#configure-the-bloom-filter you can see how
> we
> > > use the CONSOLE writer with an extractor config to count the number of
> > > typosquatted domains in the alexa top 10k dataset so we can size the
> > filter
> > > appropriately.
> > >
> > > I'd argue that other types of probabalistic data structures could also
> > > make sense here as well, like statistical sketches. Consider, for
> > instance,
> > > a cheap and dirty DGA indicator where we take the Alexa top 1M and look
> > at
> > > 

Re: Checkstyle - have we run it?

[Justin Leet]

I would expect so. Honestly, it's probably more just getting PRs out on it,
and adjusting if it ends up causing problems.  Given that we just did a
release, this is probably the best time to at least do a couple modules,
see if any problems are caused, and clean it up afterwards.

On Wed, Jan 17, 2018 at 12:20 PM, Otto Fowler <ottobackwa...@gmail.com>
wrote:

> Thanks,
> I have check style up and integrated, and I have been running it on *new*
> files etc.
> But now when I work in existing, I obviously see the issues.
>
> I *think* in the end module by module is the only feasible way is it not?
>
>
> On January 17, 2018 at 12:15:33, Justin Leet (justinjl...@gmail.com)
> wrote:
>
> It exists, we have a style that can be imported and setup in IntelliJ with
> the Checkstyle plugin
>
> Reformatting can also be done in IntelliJ (which will help a lot, but not
> all issues). This can be done on a file mask basis (e.g. just do "*.java"
> files to avoid reformatting other things), and could be done module wide
> or
> project wide or whatever. I would turn off autoformatting of Javadocs in
> IntelliJ (because a lot of the Apache license headers are Javadocs instead
> of comments). Other than that, I don't think there are any other problems.
>
> The main problem is more taking the time to do it, avoiding issues with
> existing PRs, and making it manageable to review and take care of. Do we
> do it module by module (keeping in mind we have a whole lot)? Things like
> that. I'm happy to help out, but I just really haven't put in the effort
> to
> get things moving forward.
>
>
> On Wed, Jan 17, 2018 at 9:12 AM, Otto Fowler <ottobackwa...@gmail.com>
> wrote:
>
> > Where are we with the check style integration? How are we handling check
> > style in existing modules?
> > I seem to remember talk of a script or something to reformat?
> >
> > It would be nice to get some of the warnings out of the builds, how
> should
> > we go about it?
> >
> > ottO
> >
>
>

Re: Checkstyle - have we run it?

[Justin Leet]

It exists, we have a style that can be imported and setup in IntelliJ with
the Checkstyle plugin

Reformatting can also be done in IntelliJ (which will help a lot, but not
all issues). This can be done on a file mask basis (e.g. just do "*.java"
files to avoid reformatting other things), and could be done module wide or
project wide or whatever. I would turn off autoformatting of Javadocs in
IntelliJ (because a lot of the Apache license headers are Javadocs instead
of comments).  Other than that, I don't think there are any other problems.

The main problem is more taking the time to do it, avoiding issues with
existing PRs, and making it manageable to review and take care of.  Do we
do it module by module (keeping in mind we have a whole lot)? Things like
that. I'm happy to help out, but I just really haven't put in the effort to
get things moving forward.


On Wed, Jan 17, 2018 at 9:12 AM, Otto Fowler 
wrote:

> Where are we with the check style integration?  How are we handling check
> style in existing modules?
> I seem to remember talk of a script or something to reformat?
>
> It would be nice to get some of the warnings out of the builds, how should
> we go about it?
>
> ottO
>

[DISCUSS] Upgrading Solr

[Justin Leet]

Now that we have ES at a modern version, we should consider bringing Solr
to a modern version as well.

The focus of this work would be to get us in a place where Solr is
upgraded, along with the related work of building out the Solr
functionality to parity with Elasticsearch. The goal would not be to add
net new functionality, just to get Solr and ES in the same place for the
alerts UI and REST interface.  Additionally, it would include the various
supporting necessities such as ensuring associated DAOs are testable, and
so on.

Given the testing, reviewing, and iteration involved, I'd like to propose
doing this work in a feature in a feature branch.

Jiras would be created based on this discussion once it dies down a bit.

Re: [DISCUSS] Move SHELL type functions from management to stellar common

[Justin Leet]

Agreed, I think it makes sense to move them there.

On Wed, Jan 31, 2018 at 9:28 AM, Casey Stella  wrote:

> I'd be in favor of that.  That is general purpose stuff.
>
> On Wed, Jan 31, 2018 at 9:12 AM, Otto Fowler 
> wrote:
>
> > Per:  https://issues.apache.org/jira/browse/METRON-876
> >
> > I think we should move the shell/console type functions from stellar
> > management to stellar-common, and guard them with CONSOLE capability.
> > Thoughts?
> >
> > ottO
> >
>

Apache Website Required Links

[Justin Leet]

I'd created a Jira awhile ago, but it deserves a callout to the community.
Especially if someone wants to grab it, it's probably something pretty easy
(and valuable!) to do.

There's a set of required links on Apache web pages, which can be seen
at Website
Navigation Links Policy


Reporting is at Site Check For Project - Metron


This ticket is available at:
METRON-1386 

Re: Apache Website Required Links

[Justin Leet]

Whimsy has run the site checks since then, and unfortunately we're still
showing that the links are missing.  Like I said, I think it's because the
links aren't actually on the homepage itself.

On Thu, Feb 15, 2018 at 5:02 PM, Justin Leet <justinjl...@gmail.com> wrote:

> Hey, I took a look at this, and am unsure of the implementation.
> Specifically, the links are in a subpage, not on the homepage itself.  I'm
> unsure if that's allowable or not, but I'm pretty sure whimsy won't pick it
> up unless they do something more clever than the script I glanced at.
>
> I checked out whimsy and ran tools/site-scan.rb and got this result (which
> shows nulls for the links):
>
>> {16:52}~/Documents/workspace/whimsy/tools:master ✓ ➭ ruby site-scan.rb
>>> http://metron.apache.org/
>>
>> Metron http://metron.apache.org/ recent
>>
>> {
>>
>>   "Metron": {
>>
>> "display_name": "Metron",
>>
>> "uri": "http://metron.apache.org/;,
>>
>> "events": null,
>>
>> "foundation": null,
>>
>> "license": "http://www.apache.org/licenses/LICENSE-2.0;,
>>
>> "sponsorship": null,
>>
>> "security": null,
>>
>> "trademarks": "Apache Metron and its logo are trademarks of the The
>>> Apache Software Foundation.",
>>
>> "copyright": "Copyright © 2018, The Apache Software Foundation.",
>>
>> "image": null,
>>
>> "copyparent": true
>>
>>   }
>>
>> }
>>
>>
> If you hit the subpage itself, things look better
>
>> {16:53}~/Documents/workspace/whimsy/tools:master ✓ ➭ ruby site-scan.rb
>>> http://metron.apache.org/asf
>>
>> Metron http://metron.apache.org/asf/ missing
>>
>> {
>>
>>   "Metron": {
>>
>> "display_name": "Metron",
>>
>> "uri": "http://metron.apache.org/asf/;,
>>
>> "events": "https://www.apache.org/events/current-event;,
>>
>> "foundation": "APACHE",
>>
>> "license": "http://www.apache.org/licenses/LICENSE-2.0;,
>>
>> "sponsorship": "https://www.apache.org/foundation/sponsorship.html;,
>>
>> "security": "https://www.apache.org/security/;,
>>
>> "trademarks": "Apache Metron and its logo are trademarks of the The
>>> Apache Software Foundation.",
>>
>> "copyright": "Copyright © 2018, The Apache Software Foundation.",
>>
>> "image": null,
>>
>> "thanks": "https://www.apache.org/foundation/thanks.html;,
>>
>> "copyparent": true
>>
>>   }
>>
>> }
>>
>>
>
> On Thu, Feb 15, 2018 at 3:38 PM, Casey Stella <ceste...@gmail.com> wrote:
>
>> Just reporting back that Anand's PR METRON-1386 (
>> https://github.com/apache/metron/pull/935) has been merged into master
>> and
>> the asf-site branch.
>> Kudos to Anand!
>>
>> Casey
>>
>> On Wed, Feb 7, 2018 at 9:11 AM, Anand Subramanian <
>> asubraman...@hortonworks.com> wrote:
>>
>> > I can take a shot at this if there are no other takers.
>> >
>> > Regards,
>> > Anand
>> >
>> > On 2/5/18, 8:59 PM, "Justin Leet" <justinjl...@gmail.com> wrote:
>> >
>> > I'd created a Jira awhile ago, but it deserves a callout to the
>> > community.
>> > Especially if someone wants to grab it, it's probably something
>> pretty
>> > easy
>> > (and valuable!) to do.
>> >
>> > There's a set of required links on Apache web pages, which can be
>> seen
>> > at Website
>> > Navigation Links Policy
>> > <https://www.apache.org/foundation/marks/pmcs#navigation>
>> >
>> > Reporting is at Site Check For Project - Metron
>> > <https://whimsy.apache.org/site/project/metron>
>> >
>> > This ticket is available at:
>> > METRON-1386 <https://issues.apache.org/jira/browse/METRON-1386>
>> >
>> >
>> >
>>
>
>

Re: Apache Website Required Links

[Justin Leet]

Hey, I took a look at this, and am unsure of the implementation.
Specifically, the links are in a subpage, not on the homepage itself.  I'm
unsure if that's allowable or not, but I'm pretty sure whimsy won't pick it
up unless they do something more clever than the script I glanced at.

I checked out whimsy and ran tools/site-scan.rb and got this result (which
shows nulls for the links):

> {16:52}~/Documents/workspace/whimsy/tools:master ✓ ➭ ruby site-scan.rb
>> http://metron.apache.org/
>
> Metron http://metron.apache.org/ recent
>
> {
>
>   "Metron": {
>
> "display_name": "Metron",
>
> "uri": "http://metron.apache.org/;,
>
> "events": null,
>
> "foundation": null,
>
> "license": "http://www.apache.org/licenses/LICENSE-2.0;,
>
> "sponsorship": null,
>
> "security": null,
>
> "trademarks": "Apache Metron and its logo are trademarks of the The
>> Apache Software Foundation.",
>
> "copyright": "Copyright © 2018, The Apache Software Foundation.",
>
> "image": null,
>
> "copyparent": true
>
>   }
>
> }
>
>
If you hit the subpage itself, things look better

> {16:53}~/Documents/workspace/whimsy/tools:master ✓ ➭ ruby site-scan.rb
>> http://metron.apache.org/asf
>
> Metron http://metron.apache.org/asf/ missing
>
> {
>
>   "Metron": {
>
> "display_name": "Metron",
>
> "uri": "http://metron.apache.org/asf/;,
>
> "events": "https://www.apache.org/events/current-event;,
>
> "foundation": "APACHE",
>
> "license": "http://www.apache.org/licenses/LICENSE-2.0;,
>
> "sponsorship": "https://www.apache.org/foundation/sponsorship.html;,
>
> "security": "https://www.apache.org/security/;,
>
> "trademarks": "Apache Metron and its logo are trademarks of the The
>> Apache Software Foundation.",
>
> "copyright": "Copyright © 2018, The Apache Software Foundation.",
>
> "image": null,
>
> "thanks": "https://www.apache.org/foundation/thanks.html;,
>
> "copyparent": true
>
>   }
>
> }
>
>

On Thu, Feb 15, 2018 at 3:38 PM, Casey Stella <ceste...@gmail.com> wrote:

> Just reporting back that Anand's PR METRON-1386 (
> https://github.com/apache/metron/pull/935) has been merged into master and
> the asf-site branch.
> Kudos to Anand!
>
> Casey
>
> On Wed, Feb 7, 2018 at 9:11 AM, Anand Subramanian <
> asubraman...@hortonworks.com> wrote:
>
> > I can take a shot at this if there are no other takers.
> >
> > Regards,
> > Anand
> >
> > On 2/5/18, 8:59 PM, "Justin Leet" <justinjl...@gmail.com> wrote:
> >
> > I'd created a Jira awhile ago, but it deserves a callout to the
> > community.
> > Especially if someone wants to grab it, it's probably something
> pretty
> > easy
> > (and valuable!) to do.
> >
> > There's a set of required links on Apache web pages, which can be
> seen
> > at Website
> > Navigation Links Policy
> > <https://www.apache.org/foundation/marks/pmcs#navigation>
> >
> > Reporting is at Site Check For Project - Metron
> > <https://whimsy.apache.org/site/project/metron>
> >
> > This ticket is available at:
> > METRON-1386 <https://issues.apache.org/jira/browse/METRON-1386>
> >
> >
> >
>

Re: [DISCUSS] Removing Markdown files from rat exclusion

[Justin Leet]

The PR is merged into master, and all relevant PRs have a comment noting
that adding the header is required.

As a reminder, this means Apache headers are required on all markdown files
and this will be enforced by rat.

On Sat, Dec 30, 2017 at 8:33 AM, Justin Leet <justinjl...@gmail.com> wrote:

> I've updated the PR to add the header to a new MD file that went in.
>
> I've also commented on all PRs that I saw that would potentially be
> problematic were they to go into master if they weren't merged first.
>
> Once the updated PR gets the +1's reaffirmed, it will be merged into
> master and Markdown headers will be enforced properly going forwad.
>
> On Sun, Dec 24, 2017 at 8:09 PM, Justin Leet <justinjl...@gmail.com>
> wrote:
>
>> I'm gonna let this percolate until Wednesday or so, assuming conversation
>> doesn't reach a natural tipping point.  I'm inclined to agree with Nick,
>> but I also don't want to resolve anything in a way that even potentially
>> causes master problems until at least after Christmas has a chance to
>> settle down for people.  At that point, assuming current course, I'll take
>> a real run through of the PRs (and leave comments as appropriate, before
>> merging.
>>
>> Obviously if anyone has suggestions or alternatives, still feel
>> encouraged to respond.
>>
>> On Sat, Dec 23, 2017 at 11:17 AM, Nick Allen <n...@nickallen.org> wrote:
>>
>>> > This would result in master breaking (although it's a pretty easy fix).
>>>
>>> I am not concerned and don't think we need to wait on merging PR #883.
>>>
>>> Can you add a comment to each of the PRs that you identified?  We can
>>> make
>>> sure that each gets merged with master before they go in.
>>>
>>>
>>>
>>> On Sat, Dec 23, 2017 at 11:08 AM, Justin Leet <justinjl...@gmail.com>
>>> wrote:
>>>
>>> > I have a PR currently out (https://github.com/apache/metron/pull/883)
>>> that
>>> > removes the rat exclusion on Markdown files. There was a discuss thread
>>> > awhile back about adding the header and removing the exclusion where
>>> it was
>>> > agreed that we should do this to meet Apache requirements.
>>> Unfortunately,
>>> > it didn't get any follow on.
>>> >
>>> > Right now the PR has two +1s, but it could potentially be problematic
>>> with
>>> > existing PRs.
>>> >
>>> > Any PR that meets two conditions could potentially be problematic
>>> > 1. It adds a new Markdown file
>>> > 2. Travis was run before the exclusion PR was merged.
>>> >
>>> > This is because whoever does the merge might not realize that master
>>> should
>>> > be merged in and the markdown file updated with the Apache header.
>>> This
>>> > would result in master breaking (although it's a pretty easy fix).
>>> >
>>> > Are we okay with merging this now/soon, or do we want to take
>>> additional
>>> > steps to ensure we don't run into issues? If we want, I can run
>>> through the
>>> > PRs and add comments before merging.  Is this sufficient to at least
>>> > mitigate the most obvious problems?
>>> >
>>> > I took a very quick glance through some of the most recent PRs and
>>> only two
>>> > really stood out to me (although I'm sure there are older ones that are
>>> > still being worked on or looked at)
>>> >
>>> > METRON-1380 https://github.com/apache/metron/pull/882 - Adds a new
>>> > markdown
>>> > file, but Travis failed. If it gets fixed before this PR is merged we
>>> could
>>> > run into the problem
>>> > METRON-1351 https://github.com/apache/metron/pull/868 - Adds a new
>>> > markdown
>>> > file and Travis succeeded. This would break master if merged as-is
>>> after my
>>> > PR.
>>> >
>>>
>>
>>
>

[DISCUSS] Merging Solr feature branch (METRON-1416) into master

[Justin Leet]

Hi All,

The Solr branch (/feature/METRON-1416-upgrade-solr
),
has been progressing for a while now.  I'd like to open up discussion
around what it takes to get it into master.

The JIRA for tracking this feature branch is METRON-1416
.

As shown in the JIRA, the majority of tasks are complete, with a few
outstanding issues. Of these, I believe these are the main ones of interest
to this discussion.

   - METRON-1629  -
   There is an active PR #1072 
   - METRON-1609  -
   There is an active PR #1056 
   - METRON-1602  - Full
   dev can run with Solr without this, it would simply be more convenient.
   - METRON-1632  -
   Causes a metaalert specific issue where UI filtering on
   source.type:metaalert fails. More detail is on the Jira.
   - Two validation tickets.  It's been run up on multinode, and manual
   testing has happened (and I'm will be seen a bit more on the final PR by
   various reviewers), so I'm inclined to just leave these open until we're
   good to go.  Let me know if we want to handle this differently.

I'm of the opinion both of the active PRs need to be merged before we merge
this into master, especially the documentation one.  The other two tickets
can be done in the future; one can be worked around and one is a metaalert
specific issue that primarily effects the alerts UI.

As the branch has grown and diverged from master, it's gotten increasingly
unwieldy to maintain (and I think it's worth a follow-on discussion about
how we manage refactorings that happen in these sorts of branches).  I know
there's been at least a couple merges from master that have been
nontrivially difficult and required careful testing, particularly around
the DAO layer, to avoid regressions in both code and tests.

The feature set is pretty complete.  The UI works, barring the metaalert
issue.  Much of the backend has been refactored and seen improved test
coverage benefiting both Solr and Elasticsearch.  The main difference
between ES and Solr is the lack of the equivalent visualizations to
Kibana.  I don't believe the feature branch needs to wait for this, as it's
pretty standalone work that can be added as usage and demand dictates.

I'm of the opinion that the benefits of getting the branch into master
outweighs the issues still present, especially in terms of making
refactoring and features available and easing the dev burden.  The
remaining tickets are Solr specific, and ES functions as it does in master.

Are there any must-haves before we bring this branch back?  Are there any
other concerns we have before a final PR is opened (pending completion of
active PRs and any other must-haves)?

Justin

Re: [DISCUSS] Metron Parsers in Nifi

[Justin Leet]

I'll add onto Mike's discussion with the original set of requirements I had
in mind (and apply feedback on these as necessary!). This is largely
overlap with what Mike said, but I want to make sure it's clear where my
proposal was coming from, so we can improve on it as needed.  James and
Mike are also right, I think I skipped over the benefits of NiFi in general
a bit, so thanks for chiming in there.

- Deploy our bundled parsers without needing custom wrapping on all of them.
- Don't prevent ourselves from building custom wrapping as needed.
- Custom Java parsers with an easy way to hook in, similar to what we
already do in Storm.
- One stop (or at least one format) configuration, for the case when we're
doing some thing in NiFi (parsers) and some elsewhere (enrichment and
indexing). I don't think it'll always be "start in NiFi, end in Storm",
especially as we build out Stellar capability, but I also don't want users
learning a different set of configs and config tools for every platform we
run on.
- Ability to build out parsers and other systems fairly easily, e.g. Spark.
- Support our current use cases (in particular parser chaining as a more
advanced use case).

It really boils down to providing a relatively simple user path to be able
to migrate to NiFi as needed or desired as simply as possible in a very
general way, while not preventing parser by parser enhancements.

On Wed, Aug 8, 2018 at 7:14 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> I think it also provides customers greater control over their architecture
> by giving them the flexibility to choose where/how to host their parsers.
>
> To Justin's point about the API, my biggest concern about the RecordReader
> approach is that it is not stable. We already have a similar problem in
> having the TransportClient in ElasticSearch - they are prone to changing it
> in minor versions with the advent of their newer REST API, which is
> problematic for ensuring a stable installation.
>
> From my own perspective, our goal with NiFi, at least in part, should be
> the ability to deploy our core parsing infrastructure, i.e.
>
>- pre-built parsers
>- custom java parsers
>- Stellar transforms
>- custom stellar transforms
>
> And have the ability to configure it similarly to how we configure parsers
> within Storm. Consistent with our recent parser chaining and aggregation
> feature, users should be able to construct and deploy similar constructs in
> NiFi. The core architectural shift would be that parser code should be
> platform agnostic. We provide the plumbing in Storm, NiFi, and  Streaming?, other> and platform architects and devops teams can choose how
> and where to deploy.
>
> Best,
> Mike
>
>
> On Wed, Aug 8, 2018 at 9:57 AM James Sirota  wrote:
>
> > Integration with NiFi would be useful for parsing low-volume telemetries
> > at the edge.  This is a much more resource friendly way to do it than
> > setting up dedicated storm topologies.  The integration would be that the
> > NiFi processor parses the data and pushes it straight into the enrichment
> > topic, saving us the resources of having multiple parsers in storm
> >
> > Thanks,
> > James
> >
> > 07.08.2018, 11:29, "Otto Fowler" :
> > > Why do we start over. We are going back and forth on implementation,
> and
> > I
> > > don’t think we have the same goals or concerns.
> > >
> > > What would be the requirements or goals of metron integration with
> Nifi?
> > > How many levels or options for integration do we have?
> > > What are the approaches to choose from?
> > > Who are the target users?
> > >
> > > On August 7, 2018 at 12:24:56, Justin Leet (justinjl...@gmail.com)
> > wrote:
> > >
> > > So how does the MetronRecordReader roll into everything? It seems like
> > it'd
> > > be more useful on the reader per format approach, but otherwise it
> > doesn't
> > > really seem like we gain much, and it requires getting everything
> linked
> > up
> > > properly to be used. Assuming we looked at doing it that way, is the
> idea
> > > that we'd setup a ControllerService with the MetronRecordReader and a
> > > MetronRecordWriter and then have the StellarTransformRecord processor
> > > configured with those ControllerServices? How do we manage the
> > > configurations of the everything that way? How does the
> ControllerService
> > > get configured with whatever parser(s) are needed in the flow?
> Basically,
> > > what's your vision for how everything would tie together?
> > >
> > > I also forgot to mention this in the original writeup, but there's
>

Re: [DISCUSS] Metron Parsers in Nifi

[Justin Leet]

That's definitely good info, thanks for reaching out to them about it.

In terms of exposing/sharing, I don't think we have to couple them tightly
(in fact, I think we should loosen the coupling as much as possible without
forcing reimplementation of things). I think there's definitely a way to do
that terms of the general purpose processor I proposed (or in terms of
RecordReader or another implementation).

It would definitely be easy enough to configure it to either pull from ZK
or to use a parser config json extract as a parameter (to maintain the same
formatting and make migration easy).  And we can still build specific
NiFi-oriented parsers as needed (that manage things like Schema via the
registry and other Nifi mechanisms).  This keeps parsers entirely decoupled
from a metron installation.

Alternatively, we extract our config handling to a module and scripts we
can package up and easily deploy configs against ZK (or the maybe Nifi's
StateController's or whatever).  We definitely shouldn't need absolutely
everything installed to be able to run just parsers on Nifi.

Having said that, right now the easiest way we have to maintain on the fly
updatable configs (and updatable is important!) is via ZK.  Params in Nifi
aren't quite that flexible, to the best of my knowledge (i.e. you have to
stop, update config and restart). We might be able to exploit the
StateController to manage this for us, but I'm honestly not familiar enough
with it and for deployments split between NiFi and Storm, it means
configuration gets managed in a couple different ways (which may with users
since there is a fairly brightline delineation which makes it easier to
accept).  There some complicated configs like fieldTransforms, which is
part of why I would like things to be configured in the same format (if not
the same mechanism).

Ideally, in my mind, the parsers shared between both NiFi and Storm just
implement the very general MessageParser interface (which is pretty
minimal, a couple setup methods, validation, and the actual parse).  This
is pretty lightweight and the split of metron-parsers into
metron-parsers-common et al. would loosen the coupling between parsers and
the rest of metron into that core needed to support that.

IMO, at that point, we'd have a pretty minimal NAR (or NARs depending on
config management) that lets us run our set of parsers, lets users build
new parsers (and don't block specialized NiFi implementations that exploit
NiFi's feature set), and lets us get things configured in a relatively
consistent manner, without losing features, and hopefully requiring a
pretty minimal slice of Metron to be useful.

On Thu, Aug 9, 2018 at 10:06 AM Otto Fowler  wrote:

> I think the benefits are clear.  What is unclear is if the goal is to
> expose or share or re-use Metron capabilities ( stellar, parsing ) in nifi
> in a way that is native to nifi ( configured and managed in nifi ), where
> you may not even need metron ( say you just want to parse asa ) or if the
> goal is to have a hybrid approach coupling the processors/readers to the
> metron installation.
>
>
> On August 9, 2018 at 09:14:58, Justin Leet (justinjl...@gmail.com) wrote:
>
> I'll add onto Mike's discussion with the original set of requirements I
> had
> in mind (and apply feedback on these as necessary!). This is largely
> overlap with what Mike said, but I want to make sure it's clear where my
> proposal was coming from, so we can improve on it as needed. James and
> Mike are also right, I think I skipped over the benefits of NiFi in
> general
> a bit, so thanks for chiming in there.
>
> - Deploy our bundled parsers without needing custom wrapping on all of
> them.
> - Don't prevent ourselves from building custom wrapping as needed.
> - Custom Java parsers with an easy way to hook in, similar to what we
> already do in Storm.
> - One stop (or at least one format) configuration, for the case when we're
> doing some thing in NiFi (parsers) and some elsewhere (enrichment and
> indexing). I don't think it'll always be "start in NiFi, end in Storm",
> especially as we build out Stellar capability, but I also don't want users
> learning a different set of configs and config tools for every platform we
> run on.
> - Ability to build out parsers and other systems fairly easily, e.g.
> Spark.
> - Support our current use cases (in particular parser chaining as a more
> advanced use case).
>
> It really boils down to providing a relatively simple user path to be able
> to migrate to NiFi as needed or desired as simply as possible in a very
> general way, while not preventing parser by parser enhancements.
>
> On Wed, Aug 8, 2018 at 7:14 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > I think it also provides customers greater control over their
> architecture
> > by giving them t

[DISCUSS] Release cadence

[Justin Leet]

Hi all,

In concert with the discuss thread on a potential 0.6.0 release, I'd also
like start a discussion about our release cadence.  We've generally been
pretty relaxed around doing releases, and I'm curious what people's
thoughts are on adopting a somewhat more regular schedule.

Couple questions I think are relevant
1. Is this something we should work towards and, if we do, how do we want
to go about it?

   - "Whenever someone feels like pushing out a discuss thread"?
   - "Let's just start a discuss thread every X and if we want to release
   we release"?
   - "let's try to get a release out every X and what's on the bus is on
   the bus"?
   - Something else?

2. Assuming we do want to do more regular releases, what's the timeframe
we'd like to shoot for?

Personally, I'd like to just start a discuss thread regularly, with the
built-in expectation that not every thread should necessarily lead to a
release. I don't want to be forcing release overhead when there's not
enough to merit a release, but releasing more often than we often do now
would provide a lot of values to users.

In terms of timeframe, I tend to think a 2-3 month cadence for the threads
is reasonable. It's long enough to potentially accrue enough features to
merit a release, but short enough that when we pass on a release we're
probably fine just waiting for another cycle to come around.  The last
release was ~2 months ago and we have a good amount of stuff here, but I
also don't expect two feature branches going in to be the norm.

I'd expect whatever comes out of this thread to also be relatively
informal. At least right now, I don't feel like we need a rigid schedule,
and I'd still like people to feel encouraged to propose a release,
particularly when there are a couple major features or critical fixes.
Alternatively, I would expect some of these discuss threads to conclude,
"We should do a release, but let's wait a couple waits for these tickets to
finish up" (e.g. like the Pcap query panel).

Justin

Re: [DISCUSS] Metron Release 0.6.0?

[Justin Leet]

I was actually going to kick out another thread in a little bit around our
release schedule, it should be out shortly.

Good point on the metron-bro-plugin-kafka.  I'm in favor of putting out a
0.2.0 release of it simultaneously.

On Wed, Aug 15, 2018 at 9:48 AM zeo...@gmail.com  wrote:

> I agree - I would love to see a release not long after the PCAP FB gets
> into master, and 0.6.0 makes sense to me.
>
> I'd also like to see a 0.2 release of metron-bro-plugin-kafka.  There is
> one new commit, and I have a PR open which is waiting on some tests before
> it's ready to be evaluated/merged.  I will try to get that work done asap.
> As of right now metron's dev ansible scripts pin to a specific release of
> metron-bro-plugin-kafka (0.1
> <0.1
> https://github.com/apache/metron/blob/master/metron-deployment/ansible/roles/bro/vars/main.yml
> >),
> and I'm fine leaving that as is until after the coming release, but we
> could also do a metron-bro-plugin-kafka release first and then update
> metron to point the dev environment to the new package prior to the
> upcoming RC.
>
> I would also like to discuss what the roadmap looks like for a 1.0 release
> and perhaps a more regular release schedule.  I have some thoughts but
> don't want to hijack this thread.
>
> Jon
>
> On Wed, Aug 15, 2018 at 9:11 AM Justin Leet  wrote:
>
> > Hi all,
> >
> > It's been a little while since the last release, and a couple major items
> > have gone in since then (or are hopefully close to going in!).  In
> > particular, I'd personally like to see a release with our Solr work
> > <https://issues.apache.org/jira/browse/METRON-1416> and the
> > close-to-completion PCAP Query Panel
> > <https://issues.apache.org/jira/browse/METRON-1554>.  There is a thread
> > <
> >
> https://lists.apache.org/thread.html/94ebc9be23f6f2ec8c53f8f6b71e97d6919baf415caf534e2b25ba9b@%3Cdev.metron.apache.org%3E
> > >
> > around what's left before merging the PCAP feature branch, I encourage
> you
> > to take a look. There are also some nice-to-haves as well as some Apache
> > cleanup around the RAT tool and typescript files
> > <https://github.com/apache/metron/pull/1126>.
> >
> > Version Number
> > I'm proposing bumping to 0.6.0, in particular because of the Solr and
> PCAP
> > efforts. We can adjust that as necessary.
> >
> > I'm proposing we release this from the Metron master branch, plus any
> > commits the community considers necessary.  Note that I'm proposing that
> > this release occur after the PCAP feature branch is merged into master.
> >
> > Proposed Timeframe
> > I would tentatively like to start work on the RC Wednesday, September
> 5th.
> > It's a little further out than usual, but I wanted to kick off the
> > discussion before Labor Day and to give ongoing  time to settle. And also
> > because I'll be unavailable around Labor Day.
> >
> > JIRA Status
> > There are 31 open PRs at https://github.com/apache/metron/pulls. We
> should
> > work on getting anything we feel merits inclusion closed out. Please
> > respond with any tickets we'd like included.
> >
> > A couple of these are for the PCAP feature branch, and there will be at
> > least one more for documentation.
> >
> > There will be updates necessary to get our Jira up to date.  I'll follow
> up
> > on that, and ask that everyone double check their tickets.
> >
> > There have been 106 commits since the 0.5.0 release (listed at the end of
> > message). There will be a few more when we pull in the PCAP feature
> branch.
> >
> > Completed PRs as of Aug 15 as generated by git log --pretty="%cr %s"
> > tags/apache-metron-0.5.0-release..HEAD.
> >
> > 5 days ago METRON-1730: Update steps to run pycapa on Centos 6 (mmiklavc
> > via mmiklavc) closes apache/metron#1152
> > 13 days ago METRON-1701 Update General notes on the installation of
> Pycapa
> > on Kerberized cluster (MohanDV via nickwallen) closes apache/metron#1136
> > 3 weeks ago METRON-1650 Packaging docker containers are too large
> > (jameslamb via merrimanr) closes apache/metron#1091
> > 3 weeks ago METRON-1604 : Add RHEL 7 power pc to OS family for the HCP
> > management pack repo info closes apache/incubator-metron#1052
> > 3 weeks ago METRON-1687: Upgrade the rat plugin to 0.13-SNAPSHOT closes
> > apache/incubator-metron#1126
> > 3 weeks ago METRON-1694: Clean up Metron REST docs closes
> > apache/incubator-metron#1131
> > 4 weeks ago METRON-1606 Add a wrap to incoming messages in
> the
> > metron json parser (ottobackwards) closes ap

Re: [DISCUSS] Metron Release 0.6.0?

[Justin Leet]

with Storm
> versions >= 1.1.0 (nickwallen) closes apache/metron#1042
> 2 months ago METRON-1598 NoClassDefFoundError when running with
> Elasticsearch X-Pack (nickwallen) closes apache/metron#1048
> 2 months ago METRON-1589 '/api/v1/search/search' fails when 'Solr Zookeeper
> Urls' has comma separated multiple zookeeper urls (justinleet) closes
> apache/metron#1040
> 2 months ago METRON-1593 Setting Metron rest additional classpath removes
> HBase and Hadoop configs from classpath (merrimanr) closes
> apache/metron#1044
> 3 months ago METRON-1571 Correct KAFKA_TAIL Seek to End Logic (nickwallen)
> closes apache/metron#1023
> 3 months ago METRON-1579: Stellar should return the expression that failed
> in the exception closes apache/incubator-metron#1033
> 3 months ago METRON-1586 Defaulting for the source type field in alerts UI
> does not work (merrimanr via justinleet) closes apache/metron#1038
> 3 months ago METRON-1569: Allow user to change field name conversion when
> indexing to Elasticsearch (nickwallen via mmiklavc) closes
> apache/metron#1022
> 3 months ago METRON-1544 Flaky test: org.apache.metron.stellar.common.
> CachingStellarProcessorTest#testCaching (nickwallen) closes
> apache/metron#1015
> 3 months ago METRON-1580 Release candidate check script requires Bro Plugin
> (nickwallen via ottobackwards) closes apache/metron#1034
> 3 months ago METRON-1532 Getting started documentation improvements
> (sardell via nickwallen) closes apache/metron#1001
> 3 months ago METRON-1577 Solr searches dont include the index of the
> result (merrimanr) closes apache/metron#1031
> 3 months ago METRON-1421 Create a SolrMetaAlertDao (justinleet) closes
> apache/metron#970
> 3 months ago Merge branch 'master' into feature/METRON-1416-upgrade-solr
> 3 months ago METRON-1567 Large error message cant be written in Solr
> (justinleet) closes apache/metron#1020
> 4 months ago METRON-1540 Solr Integration tests should use actual schemas
> (justinleet) closes apache/metron#1005
> 4 months ago Merge remote-tracking branch 'origin/master' into
> feature/METRON-1416-upgrade-solr
> 4 months ago METRON-1526 Location field types cause DocValuesField appear
> more than once error (merrimanr via justinleet) closes apache/metron#995
> 4 months ago METRON-1503 Alerts are not getting populated in alerts UI when
> search engine is Solr (merrimanr) closes apache/metron#975
> 5 months ago METRON-1424 Kerberos: Solr (merrimanr) closes
> apache/metron#960
> 5 months ago METRON-1482 Update REST to work with Solr (merrimanr) closes
> apache/metron#957
> 5 months ago METRON-1464 Convert schemas to be compatible with Solr 5.5.2
> (merrimanr) closes apache/metron#945
> 6 months ago METRON-1423 Ambari work to handle Solr configuration
> (merrimanr) closes apache/metron#934
> 6 months ago Merge branch 'master' into feature/METRON-1416-upgrade-solr
> 6 months ago METRON-1448: Update SolrWriter to conform to new collection
> strategy this closes apache/incubator-metron#929
> 6 months ago Merge branch 'master' into feature/METRON-1416-upgrade-solr
> 6 months ago Merge branch 'master' into feature/METRON-1416-upgrade-solr
> 6 months ago METRON-1441: Create complementary Solr schemas for the main
> sensors this closes apache/metron#922
> 6 months ago METRON-1436: Manually Install Solr Cloud in Full Dev (mmiklavc
> via mmiklavc) closes apache/metron#918
> 7 months ago METRON-1419: Create a SolrDao this closes
> apache/incubator-metron#911
>
>
>
> On Wed, Aug 15, 2018 at 9:48 AM, zeo...@gmail.com 
> wrote:
>
> > I agree - I would love to see a release not long after the PCAP FB gets
> > into master, and 0.6.0 makes sense to me.
> >
> > I'd also like to see a 0.2 release of metron-bro-plugin-kafka.  There is
> > one new commit, and I have a PR open which is waiting on some tests
> before
> > it's ready to be evaluated/merged.  I will try to get that work done
> asap.
> > As of right now metron's dev ansible scripts pin to a specific release of
> > metron-bro-plugin-kafka (0.1
> > <0.1https://github.com/apache/metron/blob/master/metron-
> > deployment/ansible/roles/bro/vars/main.yml>),
> > and I'm fine leaving that as is until after the coming release, but we
> > could also do a metron-bro-plugin-kafka release first and then update
> > metron to point the dev environment to the new package prior to the
> > upcoming RC.
> >
> > I would also like to discuss what the roadmap looks like for a 1.0
> release
> > and perhaps a more regular release schedule.  I have some thoughts but
> > don't want to hijack this thread.
> >
> > Jon
> >
> > On Wed, Aug 15, 2018 at 9:11 AM Justin Leet 
> wrote:
> >
> > > Hi

[DISCUSS] Metron Release 0.6.0?

[Justin Leet]

Hi all,

It's been a little while since the last release, and a couple major items
have gone in since then (or are hopefully close to going in!).  In
particular, I'd personally like to see a release with our Solr work
 and the
close-to-completion PCAP Query Panel
.  There is a thread

around what's left before merging the PCAP feature branch, I encourage you
to take a look. There are also some nice-to-haves as well as some Apache
cleanup around the RAT tool and typescript files
.

Version Number
I'm proposing bumping to 0.6.0, in particular because of the Solr and PCAP
efforts. We can adjust that as necessary.

I'm proposing we release this from the Metron master branch, plus any
commits the community considers necessary.  Note that I'm proposing that
this release occur after the PCAP feature branch is merged into master.

Proposed Timeframe
I would tentatively like to start work on the RC Wednesday, September 5th.
It's a little further out than usual, but I wanted to kick off the
discussion before Labor Day and to give ongoing  time to settle. And also
because I'll be unavailable around Labor Day.

JIRA Status
There are 31 open PRs at https://github.com/apache/metron/pulls. We should
work on getting anything we feel merits inclusion closed out. Please
respond with any tickets we'd like included.

A couple of these are for the PCAP feature branch, and there will be at
least one more for documentation.

There will be updates necessary to get our Jira up to date.  I'll follow up
on that, and ask that everyone double check their tickets.

There have been 106 commits since the 0.5.0 release (listed at the end of
message). There will be a few more when we pull in the PCAP feature branch.

Completed PRs as of Aug 15 as generated by git log --pretty="%cr %s"
tags/apache-metron-0.5.0-release..HEAD.

5 days ago METRON-1730: Update steps to run pycapa on Centos 6 (mmiklavc
via mmiklavc) closes apache/metron#1152
13 days ago METRON-1701 Update General notes on the installation of Pycapa
on Kerberized cluster (MohanDV via nickwallen) closes apache/metron#1136
3 weeks ago METRON-1650 Packaging docker containers are too large
(jameslamb via merrimanr) closes apache/metron#1091
3 weeks ago METRON-1604 : Add RHEL 7 power pc to OS family for the HCP
management pack repo info closes apache/incubator-metron#1052
3 weeks ago METRON-1687: Upgrade the rat plugin to 0.13-SNAPSHOT closes
apache/incubator-metron#1126
3 weeks ago METRON-1694: Clean up Metron REST docs closes
apache/incubator-metron#1131
4 weeks ago METRON-1606 Add a wrap to incoming messages in the
metron json parser (ottobackwards) closes apache/metron#1054
4 weeks ago METRON-1672 Add metron-alertss UI unit tests to travis
build process (justinleet) closes apache/metron#1106
4 weeks ago METRON-1684 Fix Markdown problems in 3rdPartyParser.md
(justinleet) closes apache/metron#1110
4 weeks ago METRON-1657 Parser aggregation in storm (justinleet) closes
apache/metron#1099
4 weeks ago METRON-1651 Fixing failing protractor e2e test (tiborm via
merrimanr) closes apache/metron#1095
4 weeks ago METRON-1673 Fix Javadoc errors (justinleet) closes
apache/metron#1107
4 weeks ago METRON-1620: Fixes for forensic clustering use case example
(mmiklavc via mmiklavc) closes apache/metron#1065
4 weeks ago METRON-1659: The platform-info.sh should check for the vagrant
hostmanager plugin closes apache/incubator-metron#1100
4 weeks ago METRON-1658: Upgrade bro to 2.5.4 closes
apache/incubator-metron#1101
4 weeks ago METRON-1236 Add start/stop/restart commands that execute
successfully, when ambari agents run as non-root user closes
apache/incubator-metron#1105
4 weeks ago METRON-1670: Stellar WEEK_OF_YEAR test is locale sensitive
closes apache/incubator-metron#1104
5 weeks ago METRON-1660 On Solr, sorting by threat score fails (justinleet)
closes apache/metron#1102
5 weeks ago METRON-1656 Create KAKFA_SEEK function (nickwallen) closes
apache/metron#1097
5 weeks ago METRON-1644: Support parser chaining closes
apache/incubator-metron#1084
5 weeks ago METRON-1655 Make REGEXP_MATCH take multiple regexs in the 2nd
arg (ottobackwards) closes apache/metron#1098
6 weeks ago METRON-1643: Create a REGEX_ROUTING field transformation closes
apache/incubator-metron#1083
6 weeks ago METRON-1652 Document X-Pack Common Problem (nickwallen) closes
apache/metron#1092
6 weeks ago METRON-1649 Intermittent Test Failure
ProfileBuilderBoltTest#testFlushExpiredProfiles (nickwallen) closes
apache/metron#1090
6 weeks ago METRON-1635 Alerts UI status update doesnt immediately
show up (merrimanr) closes apache/metron#1080
6 weeks ago METRON-1642: KafkaWriter should be able choose the topic from a
field in addition to topology construction time 

[DISCUSS] Metron Parsers in Nifi

[Justin Leet]

Hi all,

There's interest in being able to run Metron parsers in NiFi, rather than
inside Storm. I dug into this a bit, and have some thoughts on how we could
go about this.  I'd love feedback on this, along with anything we'd
consider must haves as well as future enhancements.

   1. Separate metron-parsers into metron-parsers-common and metron-storm
   and create metron-parsers-nifi. For this code to be reusable across
   platforms (NiFi, Storm, and anything else in the future), we'll need to
   decouple our parsers and Storm.
  - There's also some nice fringe benefits around refactoring our code
  to be substantially more clear and understandable; something
which came up
  while allowing for parser aggregation.
   2. Create a MetronProcessor that can run our parsers.
  - I took a look at how RecordReader could be leveraged (e.g.
  CSVRecordReader), but this is pretty tightly tied into schemas
and is meant
  to be used by ControllerServices, which are then used by Processors.
  There's friction involved there in terms of schemas, but also in terms of
  access to ZK configs and things like parser chaining.  We might
be able to
  leverage it, but it seems like it'd be fairly shoehorned in
without getting
  the schema and other benefits.
  - This Processor would work similarly to Storm: bytes[] in -> JSON
  out.
  - There is a Processor
  

that
  handles loading other JARs that we can model a
MetronParserProcessor off of
  that handles classpath/classloader issues (basically just sets up a
  classloader specific to what's being loaded and swaps out the Thread's
  loader when it calls to outside resources).
  3. Create a MetronZkControllerService to supply our configs to our
   processors.
  - This is a pretty established NiFi pattern for being able to provide
  access to other services needed by a Processor (e.g. databases or large
  configurations files).
  - The same controller service can be used by all Processors to manage
  configs in a consistent manner.

At that point, we can just NAR our controller service and parser processor
up as needed, deploy them to NiFi, and let the user provide a config for
where their custom parsers can be provided (i.e. their parser jar).  This
would be 3 nars (processor, controller-service, and controller-service-api
in order to bind the other two together).

Once deployed, our ability to use parsers should fit well into the standard
NiFi workflow:

   1. Create a MetronZkControllerService.
   2. Configure the service to point at zookeeper.
   3. Create a MetronParser.
   4. Configure it to use the controller service + parser jar location +
   any other needed configs.
   5. Use the outputs as needed downstream (either writing out to Kafka or
   feeding into more MetronParsers, etc.)

Chaining parsers should ideally become a matter of chaining MetronParsers
(and making sure the enveloping configs carry through properly). For parser
aggregation, I'd just avoid it entirely until we know it's needed in NiFi.

Justin

Re: [DISCUSS] Metron Parsers in Nifi

[Justin Leet]

Thanks for the comments, Otto, this is definitely great feedback.  I'd love
to respond inline, but the email's already starting to lose it's
formatting, so I'll go with the classic "wall of text".  Let me know if I
didn't address everything.

Loading modules (or jars or whatever) outside of our Processor gives us the
benefit of making it incredibly easy for a users to create their own
parsers. I would definitely expect our own bundled parsers to be included
in our base NAR, but loading modules enables users to only have to learn
how Metron wants our stuff lined up and just plug it in. Having said that,
I could see having a wrapper for our bundled parsers that makes it really
easy to just say you want an MetronAsaParser or MetronBroParser, etc. That
would give us the best of both worlds, where it's easy to get setup our
bundled parsers and also trivial to pull in non-bundled parsers. What doing
this gives us is an easy way to support (hopefully) every parser that gets
made, right out of the box, without us needing to build a specialized
version of everything until we decide to and without users having to jump
through hoops.

None of this prevents anyone from creating specialized parsers (for perf
reasons, or to use the schema registries, or anything else).  It's probably
worthwhile to package up some of built-in parsers and customize them to use
more specialized feature appropriately as we see things get used in the
wild.  Like you said, we could likely provide Avro schemas for some of this
and give users a more robust experience on what we choose to support and
provide guidance for other things.  I'm also worried that building
specialized schemas becomes problematic for things like parser chaining
(where our routers wrap the underlying messages and add on their own info).
Going down that road potentially requires anything wrapped to have a
specialized schema for the wrapped version in addition to a vanilla version
(although please correct me if I'm missing something there, I'll openly
admit to some shakiness on how that would be handled).

I also disagree that this is un-Nifi-like, although I'm admittedly not as
skilled there.  The basis for doing this is directly inspired by the
JoltTransformer, which is extremely similar to the proposed setup for our
parsers: Simply take a spec (in this case the configs, including the
fieldTransformations), and delegate a mapping from bytes[] to JSON.  The
Jolt library even has an Expression Language (check out
https://community.hortonworks.com/articles/105965/expression-language-with-jolt-in-apache-nifi.html),
so it's not a foreign concept. I believe Simon Ball has already done some
experimenting around with getting Stellar running in NiFi, and I'd love to
see Stellar more readily available in NiFi in general.

Re: the ControllerService, I see this as a way to maintain Metron's use of
ZK as the source of config truth.  Users could definitely be using NiFi and
Storm in tandem (parse in NiFi + enrich and index from Storm, for
example).  Using the ControllerService gives us a ZK instance as the single
source of truth.  That way we aren't forcing users to go to two different
places to manage configs.  This also lets us leverage our existing scripts
and our existing infrastructure around configs and their management and
validation very easily.  It also gives users a way to port from NiFi to
Storm or vice-versa without having to migrate configs as well. We could
also provide the option to configure the Processor itself with the data
(just don't set up a controller service and provide the json or whatever as
one of our properties).


On Tue, Aug 7, 2018 at 10:12 AM Otto Fowler  wrote:

> I think this is a good idea.  As I mentioned in the other thread I’ve been
> doing a lot of work on Nifi recently.
> I think the important thing is that what is done should be done the NiFi
> way, not bolting the Metron composition
> onto Nifi.  Think of it like the Tao of Unix, the parsers and components
> should be single purpose and simple, allowing
> exceptional flexibility in composition.
>
> Comments inline.
>
> On August 7, 2018 at 09:27:01, Justin Leet (justinjl...@gmail.com) wrote:
>
> Hi all,
>
> There's interest in being able to run Metron parsers in NiFi, rather than
> inside Storm. I dug into this a bit, and have some thoughts on how we could
>
> go about this. I'd love feedback on this, along with anything we'd
> consider must haves as well as future enhancements.
>
> 1. Separate metron-parsers into metron-parsers-common and metron-storm
> and create metron-parsers-nifi. For this code to be reusable across
> platforms (NiFi, Storm, and anything else in the future), we'll need to
> decouple our parsers and Storm.
>
> +1.  The “parsing code” should be a library that implements an interface (
> another library ).
>
> The Processors and the Storm things can share them.
>
>