This looks good to me, honestly. Anything to make the build more
understandable and help find classpath issues easier is a good idea IMO.
Just curious, did you test that PR in both solr and ES (you added an
exclude in the ES portion of the code) and did you spin it up in full-dev
I just want to chime in and say I'm STRONGLY in favor of a docker-based
approach to testing (I specifically like the JUnit 5 extensions
suggestion). I think that forcing a full-dev evaluation for every small PR
is a barrier to entry that I'd like to overcome. I also think that this is
FWIW, I'm in favor of 2. I think it's a relatively minor bug and the
impact is limited. I do agree that it should be a blocker for 0.8.0 though.
On Thu, May 2, 2019 at 9:31 AM Michael Miklavcic <
> I am still in favor of option 2. I will volunteer and submit
On Thu, Dec 20, 2018 at 1:08 PM Casey Stella wrote:
> > I definitely agree with option 3; that's a no-brainer IMO. I thought for
> > sure this was already happening, honestly.
> > As for 2, we could even script the broken link check by:
I definitely agree with option 3; that's a no-brainer IMO. I thought for
sure this was already happening, honestly.
As for 2, we could even script the broken link check by:
- Serving up the site locally via python with `python -m http.server`
from the site-book output directory
We absolutely should be acking the dropped messages otherwise they'll be in
a replay loop. Not acking is a flat-out bug IMO.
On Wed, Dec 19, 2018 at 2:37 PM Michael Miklavcic <
> When a message is filtered by the message filtering mechanism, we
+1 to that!!
On Mon, Dec 17, 2018 at 13:16 Michael Miklavcic
> And a big thanks to Justin Leet for being our release manager. Great work
> On Mon, Dec 17, 2018 at 10:07 AM Justin Leet wrote:
>> Hi all,
>> I’m pleased to announce the release of Metron 0.7.0! There's been
+1, I'd say mention it on the dev list and slack channel.
On Thu, Nov 29, 2018 at 10:26 AM Michael Miklavcic <
> Every now and then we see intermittent test failures, and rather than
> sweeping them under the rug, we should have a simple method to track and
Congrats Shane! Well deserved!
On Tue, Nov 27, 2018 at 7:24 AM Tamás Fodor wrote:
> Congratulations Shane!
> On Mon, Nov 19, 2018 at 5:58 PM Mohan Venkateshaiah <
> mvenkatesha...@hortonworks.com> wrote:
> > Congrats Shane !!
> > Thanks
> > Mohan DV
> > On 11/19/18, 10:26 PM,
for Git to handle without
> requiring redoing merges and another full round of testing. I'd much prefer
> to avoid that in this instance.
> This PR is ready to be merged into master. It's recent and very close to
> fully up to date in the branch. Latest master merges cleanly. There is an
> > > like this in Slack should be redirected to the user list.
> > >
> > > Is this a reasonable way separate our concerns here?
> > >
> > > On Wed, Oct 24, 2018 at 11:37 AM Michael Miklavcic <
> > > michael.miklav...@g
On Thu, Nov 1, 2018 at 18:34 Nick Allen wrote:
> On Thu, Nov 1, 2018, 6:27 PM Justin Leet wrote:
> > +1, I haven't seen any case where the split-join topology isn't made
> > obsolete by the unified topology.
> > On Thu, Nov 1, 2018 at 6:17 PM Michael Miklavcic <
quick clarification, I said "a lot of dev discussion happens on github and
JIRA". I want to make sure I didn't mean to imply that larger decisions
were being made outside of the appropriate place, the dev list.
On Wed, Oct 24, 2018 at 10:08 AM Casey Stella wrote:
> > Metron is sort of dead just because the mailing list is not so active
> > anymore!
> > Cheers,
> > Ali
> > On Tue, Oct 23, 2018 at 8:23 AM Casey Stella wrote:
> > > Agreed, the benefit of the mailing list is that it’s searchable
t; > >
> > > On Mon, Oct 22, 2018 at 10:51 AM Justin Leet
> > > wrote:
> > >
> > >> If we want to push more discussion to the dev list, my obvious follow
> > >> question then is "What are we hoping to get out of Slack/irc/other
I am of 2 minds, but I tend to agree. On the one hand, it's definitely the
preference that we use the mailing lists for the reasons you stated (and
also because not everyone has access to slack generally). On the other
hand, I think an interactive medium like Slack has a lot of advantages in
I think it makes a lot of sense. A couple of questions:
- What actions do you see the REST verbs corresponding to? I would
understand GET (which is in effect "evaluate an expression", right?), but
I'm not sure about the others.
- We should probably be careful about caching stellar
>> > > people shouldn't get failing tests when they are done running. I just
>> > meant
>> > > that we often re-run flaky tests with protractor-flake, so it can
>> take a
>> > > while to run and could increase the build time considerably.
uce locally. In addition, our current Protractor tests are
> brittle and extremely slow.
> All that said, it seems we agree that we could add another PR checklist
> item in the meantime. Clarifying those e2e test instructions should be part
> of that task.
> On Mon, Oct 1, 2
doesn't have much or any knowledge of the UI be able to
run that without assistance?
For instance, we use full-dev, do we need to stop data from being played
into full-dev for the tests to work?
On Mon, Oct 1, 2018 at 8:29 AM Casey Stella wrote:
> I'm not super keen on expanding the st
I'm not super keen on expanding the steps to contribute, especially in an
avenue that should be automated.
That being said, I think that until we get to the point of automating the
e2e tests, it's sensible to add them to the checklist.
So, I would support it, but I would also urge us to move
ed along the way and formalize them as Casey is recommending
> > in our bylaws. I'll be following up with more specific thoughts on
> > language.
> > Best,
> > Mike
> > On Fri, Sep 28, 2018 at 10:13 AM Justin Leet
> > wrote:
I just noticed this, but googling "metron bylaws" yields
http://metron.apache.org/develop/bylaws.html which is not our bylaws. Our
bylaws are on
We should fix that.
On Fri, Sep 28, 2018 at 12:02 PM Casey Ste
Given discussions about the current high-profile feature branch (Knox SSO),
I thought it might be appropriate to have a conversation about what
constitutes a feature branch and get some of this encoded in the community
Specifically, there was the request made that we split up
I'm coming in late to the game here, but for my mind a feature branch
should involve the minimum architectural change to accomplish a given
The feature in question is SSO integration. It seems to me that the
operative question is can we do the feature without making the OTHER
I think it's fine. My only concern would be that we aren't accidentally
using moment.js somewhere for something that date-fns doesn't do. I
suspect whoever picks up the ticket will figure that out pretty quick
though. ;) . I'm +1 on the move; you convinced me.
On Wed, Sep 26, 2018 at 8:36 AM
> I'm not sure I fully understand what is out of date. I know I have
> personally modified our licenses a couple times in the past and used an
> automated script that, I believe, Casey Stella had created for doing the
> check. I even made some improvements to it a long ways ba
Mike, did you mean to reply to this on the dev list or were you aiming to
make this comment on the PR? If you were aiming to make this comment on
the PR, then I think you need to go through github's UI.
On Fri, Sep 7, 2018 at 1:34 PM Michael Miklavcic <
+1 to defer for this release and complete separation. Good fences make
good submodules. ;)
On Fri, Sep 7, 2018 at 2:33 PM zeo...@gmail.com wrote:
> +1 to defer for this release and +1 to Justin's suggested release/dist
> directory breakout and complete separation.
> On Fri, Sep 7,
I’d get rid of them.
On Thu, Sep 6, 2018 at 13:42 Michael Miklavcic
> What are we doing with feature branches once they're complete and merged
> into master? Is our expectation that we'll keep feature branches in
> perpetuity, or should we plan to do some house cleaning once they've been
wait, I'm an op? Coming up in the world! Do we need this still? I'm
currently afk, but will get to it tomorrow.
On Wed, Aug 29, 2018 at 4:23 PM Otto Fowler wrote:
> Damn, I was hoping not. It will never happen now
> On August 29, 2018 at 15:49:26, zeo...@gmail.com (zeo...@gmail.com)
+1, I look forward to the PR.
On Tue, Aug 28, 2018 at 8:37 AM Nick Allen wrote:
> I'd love to see a PR for this. I know there are others in the community
> looking for something similar.
> On Sun, Aug 26, 2018 at 7:28 PM wrote:
> > Hello,
> > We have implemented a general
never we build locally. Coincidentally, I
> > addressed this by switching to `npm ci` in an open PR of mine because I
> > noticed the same happening locally and I was already updating npm
> > in the pom.xml.
> I've glanced at the package file and see carrots littering the
> dependencies, which as I understand it means "get me anything later than
> this version." I do not think we should be doing that.
> On Sat, Aug 25, 2018, 9:14 AM Casey Stella
I have looked into this for other reasons and the guidance that I've seen
is to check in package-lock.json into source control. I'll leave this
stack overflow thread here:
I want to point out
I completely agree, Mike. Our docs are either very high level or very low
level (and possibly stale) and, worse, aren't aimed at the actors that
I think that the HBase project does a good job of providing coherent and
useable documentation in their "HBase Book" (see
I'm +1 on the merge. This is great work and congrats to those who
contributed to it!
On Thu, Aug 16, 2018 at 8:27 AM Otto Fowler wrote:
> Looks good, thanks!
> On August 15, 2018 at 19:38:12, Ryan Merriman (merrim...@gmail.com) wrote:
> Otto, I believe the items you requested are in the
which are also
> large-ish and will fit nicely into the next cycle (pending completion, of
>1. NiFi Metron parsers
>2. Profiler enhancements - bootstrapping, etc.
>3. Knox SSO
> On Wed, Aug 15, 2018 at 11:10 AM Casey Stella wrote:
Strictly selfishly, I'd love for a release to happen quickly enough to have
something to announce to the board during the reports. Once every 2 months
or when a sufficiently complicated change happens sounds like a sensible
I very much support a "how do we get to 1.0" discussion, maybe
+1 to both releases, this is plenty for an 0.6.0 and a 0.2.0
On Wed, Aug 15, 2018 at 11:04 AM Justin Leet wrote:
> I just sent a thread about release cadence. Jon, I'd recommend starting a
> thread on a 1.0 roadmap. I thought about merging the threads, but I think
> that's just going to result
Sorry Simon, I retract the comment! I didn't realize it was possible, but
it is possible to invite.
On Wed, Aug 15, 2018 at 1:01 PM Casey Stella wrote:
> Sadly, it's the ASF slack and I believe it requires an @apache.org email
> On Wed, Aug 15, 2018 at 12:57 PM Simon E
Sadly, it's the ASF slack and I believe it requires an @apache.org email
On Wed, Aug 15, 2018 at 12:57 PM Simon Elliston Ball <
> Hello dev team, may I please join your slack channel :)
nd Metron wherein possible threat details can be
> > communicated
> > > to Knox to take action on at authentication/authorization time.
> > > Knox could also potentially push interesting events like possible brute
> > > force login attempts to Metron.
> > >
I added the feature branch: feature/METRON-1663-knoxsso
On Thu, Jul 12, 2018 at 11:13 AM Otto Fowler
> I think I understand what you are saying very very very well Simon. I am
I would support this being a feature branch. It sounds like a valuable but
On Thu, Jul 12, 2018 at 10:51 AM Simon Elliston Ball <
> I've been doing some work on getting the Metron UIs and REST layers to work
> with Apache KnoxSSO, and LDAP
I have no objection and would consider it to be a prerequisite to bringing
in the PR unless there's someone depending on it out there. You might want
to cc user@ as well, to get a broader set of input for the "are people
using it?" question.
On Fri, Jun 29, 2018 at 5:21 PM Ryan Merriman wrote:
Those are good questions and there were some reasons surrounding that. In
fact, historically, we had fewer topologies (e.g. indexing and enrichment
were merged). Even earlier on, we had just one giant topology per parser
that enriched and indexed. The long story short is that we
I think that we should merge now, but I’m perhaps biased since I did one of
the hard merges. I think that since the major outstanding bug is being
worked and we are otherwise feature complete, the feature branch did its
job and we are ready to merge.
On Thu, Jun 21, 2018 at 10:21 Justin Leet
I created a PR for the empty collection falseyness as well:
https://github.com/apache/metron/pull/1064 so we can choose either of them
if we so desire.
On Sat, Jun 16, 2018 at 1:10 PM Casey Stella wrote:
> I created a PR for this functionality, in case we decided for it:
On Sat, Jun 16, 2018 at 10:17 AM Casey Stella wrote:
> Right now, because fields may not exist, users can have an awkward time.
> For instance, checking for is_alert, you end up having to preface checks
> with exists(is_alert).
> For instance, in one of our use-cases:
Right now, because fields may not exist, users can have an awkward time.
For instance, checking for is_alert, you end up having to preface checks
For instance, in one of our use-cases:
> >>> 2016/06/16/Metron+Tutorial+-+Fundamentals+Part+6%3A+Streaming+Enrichm
> >>> ent
> >>> 4. Create a new "dns" sensor in Metron 5. Use the CSVParser and
> >>> SimpleHbaseEnrichment
Great job all! This was a big release with a lot of good stuff. I
especially like the performance improvements :)
On Fri, Jun 8, 2018 at 8:54 AM Justin Leet wrote:
> Hi All,
> I’m happy to announce the release of Metron 0.5.0! Everyone has put in a
> lot of working into
s it worth us creating an nifi-metron-bundle. Happy to kick that off,
> > > since I'm half way there.
> > >
> > > Simon
> > >
> > >
> > >
> > > On 5 June 2018 at 08:41, Otto Fowler wrote:
> > >
> > > > We have jiras a
> It is still our user list and dev list that will have the burden of
> talking folks through that.
> On June 5, 2018 at 09:58:32, Casey Stella (ceste...@gmail.com) wrote:
> To be clear, I'm not even suggesting that we create any tooling here. I'd
> say ju
> Do we, the community, think it would be a good idea to create a
> PutMetronEnrichment NiFi processor for this use case? It seems a number of
> people want to use NiFi to manage and schedule loading of enrichments for
> On 5 June 2018
hat read and
> > transformed fields for templates and indices to replace the colons with
> > dots in ES.
> > Simon
> > On 5 June 2018 at 06:40, Casey Stella wrote:
> > > +1 to that, Simon. Do we have a sense of if there are utilities
The problem, as you correctly diagnosed, is the key in HBase. We construct
the key very specifically in Metron, so it's unlikely to work out of the
box with the NiFi processor unfortunately. The key that we use is formed
here in the codebase:
nd transforming existing data.
> > On Tue, Jun 5, 2018 at 8:21 AM, Casey Stella wrote:
> > > Well, on write it is a transformation, on read it's a translation.
> > is
> > > to say that you're providing a mapping on read to translate field
Well, on write it is a transformation, on read it's a translation. This is
to say that you're providing a mapping on read to translate field names
given the index you're using. The other approach that I was considering
last night is a field transformation REST call which translates field names
Before we construct a super generic solution, can we get an analysis of all
the places in the UI where we're hard-coding fields? It seems like pulling
the field from the global config is the strategy that we've gone with that
could be expanded upon in https://github.com/apache/metron/pull/1010
> On May 30, 2018 at 11:53:51, Casey Stella (ceste...@gmail.com) wrote:
> Yep, I think we can, mike.
> Let me start with a emendation:
> "Don’t combine code changes with lots of edits of whitespace, comments, or
> code changes specifically for cosmetic refactoring pur
up a vote
> thread following with the final proposed changes?
> On Wed, May 30, 2018 at 9:46 AM, Casey Stella wrote:
>> I'm torn on this, honestly. I completely agree that cosmetic refactoring
>> gets in the way of review and the risk can be more than the re
I'm torn on this, honestly. I completely agree that cosmetic refactoring
gets in the way of review and the risk can be more than the reward,
especially in a subtle bit of code.
That being said, I'm a big fan of opportunistically refactoring to
generalize or correct faulty assumptions. Often, I
Just a question, do we need anything new in the Upgrading.md for this
release? Any migration that we expect people to do?
On Tue, May 29, 2018 at 11:30 AM Nick Allen wrote:
> METRON-1544 was just merged into master.
> On Tue, May 29, 2018 at 2:16 PM, Justin Leet
> > I'm going
Cool! I'd welcome a syslog parser!
On Fri, May 18, 2018 at 10:02 AM Otto Fowler
> There have been some issues and talk about they way we parse syslog, and
> the deficiencies of our grok and regex based approaches, mainly not
> supporting structured data as I
18 at 9:07 AM, Justin Leet <justinjl...@gmail.com>
> > wrote:
> > > I'd be happy to to volunteer to take over for a while.
> > >
> > > Thanks to Matt for all the help through the last couple releases!
> > >
> > > Justin
Matt Foley, our esteemed Release manager for the last couple releases, has
asked to be relieved. So, I'm calling on volunteers for the next release
manager. It should be a committer and there are a few things that require
a PMC member, I believe, but the release manager can ask for help
A couple of thoughts on cluster overuse:
* Definitely can't pause/resume MR jobs, unfortunately
* The traditional approach to managing overuse of cluster resources and
prioritization in Yarn is via the scheduler. I'd suggest rather than
building this ourselves, we allow users to be associated
> > >> > > > > >> 3 months ago METRON-1432 JDK Install Fails on Ubuntu
> > >> Development
> > >> > > > > >> Environment (nickwallen) closes apache/metron#913
> > >> > >
> > > >> 4 months ago METRON-1428: Travis build failing from
> > > > > >> (mmiklavc via mmiklavc) closes apache/metron#908
> > > > > >> 4 months ago METRON-1302: Split up Indexing Topology into batch
> > and
> > > &g
> 5 months ago METRON-1349 Full Dev Builds Metron Twice (nickwallen) closes
> 5 months ago METRON-1343 Swagger UI for User Controller needs request
> method (MohanDV via ottobackwards) closes apache/metron#862
> 5 months ago METRON-1306: When index template install
I wasn't aware we had a script for that..is that in
On Wed, May 9, 2018 at 11:41 AM Otto Fowler <ottobackwa...@gmail.com> wrote:
> Can you run the issues included script and post that for us to see?
> On May 9, 2018 at 11:14:11, Casey Stella (
Is it about time for a release? I know we got some substantial performance
changes in since the last release. I think we might have a justification
for a release.
On Fri, Apr 20, 2018 at 11:17 AM David Lyle wrote:
> +1 sounds good to me.
> On Fri, Apr 20, 2018 at 11:09 AM, zeo...@gmail.com
> > +1 (non-binding)
> > On Fri, Apr 20, 2018 at 9:42 AM Michel Sumbul
I think I'd prefer to see the changelog only include the release entries,
rather than individual entries per dev. We keep the spec file in source
control to determine the individual changes between releases. I'm happy to
have my mind changed, though.
On Wed, Apr 18, 2018 at 9:47 AM Michael
I have not personally seen that one yet, but I will not deny that it
exists. It could be very intermittent or triggered under load in travis
too. Either way, we should probably investigate and fix.
On Wed, Apr 11, 2018 at 3:57 PM Otto Fowler wrote:
> I had a PR build
Nevermind, it's just the internal apache release status wasn't updated.
Sorry, I updated it as part of the board report. Let me make sure I update
teh docs for releasing.
On Tue, Apr 10, 2018 at 10:35 AM Casey Stella <ceste...@gmail.com> wrote:
> It seems that 0.4.2 never got
k cluster using the
> > generated RPMs
> > * Bro, YAF and snort - ingest into kafka topics and validated indices
> > * Add squid telemetry, ingest into kafka topic and validated indices
> > * Management UI, Alerts UI and Swagger UI sanity check
> > +1 (non-binding)
get this actioned.
> >> They would need to assist with:
> >> 1. Creation of the new "issues" list
> >> 2. redirect both GitHub and JIRA integrations to the new list
> >> Cheers
> >> On Sat, Jan 27, 201
FYI, the PR for this is up at https://github.com/apache/metron/pull/940
For those interested, please comment on the actual implementation there.
On Thu, Feb 22, 2018 at 12:43 PM, Casey Stella <ceste...@gmail.com> wrote:
> So, these are good questions, as usual Otto :)
> > ho
> * how does this effect the distribution of work through the cluster, and
> resiliency of the topologies?
> * Is anyone else doing it like this?
> * Can we have multiple thread pools and group tasks together ( or separate
> them ) wrt hbase?
> On February 22,
I've been thinking and working on something that I wanted to get some
feedback on. The way that we do our enrichments, the split/join
architecture was created to effectively to parallel enrichments in a
storm-like way in contrast to OpenSoc.
There are some good parts to this
Just reporting back that Anand's PR METRON-1386 (
https://github.com/apache/metron/pull/935) has been merged into master and
the asf-site branch.
Kudos to Anand!
On Wed, Feb 7, 2018 at 9:11 AM, Anand Subramanian <
> I can take a shot at this if there
So, I'll answer your question with some questions:
- No matter the data store we use upgrading will take some care, right?
- Do we currently depend on a RDBMS anywhere? I want to say that we do
in the REST layer already, right?
- If we don't use a RDBMs, what's the other option?
is a newish-api, and there should be 1 listeners.
> Having 1 listener shouldn’t be an issue.
> On January 31, 2018 at 11:45:54, Casey Stella (ceste...@gmail.com) wrote:
> Hmm, I have heard this feedback before. Perhaps a more low-key approach
> would be eit
Hmm, I have heard this feedback before. Perhaps a more low-key approach
would be either a static timer that checked or a timer bolt that sent a
periodic timer and the parser bolt reconfigured the parser (or indeed we
added a Reloadable interface with a 'reload' method). We could be smart
I assumed he was talking about the SHELL_EDIT stuff and maybe the file
loading bits. The config stuff is metron specific
On Wed, Jan 31, 2018 at 10:06 AM, Nick Allen wrote:
> > I think we should move the shell/console type functions from stellar
> What functions
I'd be in favor of that. That is general purpose stuff.
On Wed, Jan 31, 2018 at 9:12 AM, Otto Fowler
> Per: https://issues.apache.org/jira/browse/METRON-876
> I think we should move the shell/console type functions from stellar
> management to stellar-common,
I can't wait! This is going to be really cool :)
On Fri, Jan 26, 2018 at 5:25 PM, James Sirota wrote:
> Yeah very interested in the presentation as well
> 26.01.2018, 15:15, "Simon Elliston Ball" :
> > This is going to be a really exciting
This could be one of those intermittent test failures related to timing.
Time elapsed: 0.064 sec <<< FAILURE!
java.lang.AssertionError: Status expected:<404> but was:<200>
I could get behind that.
On Fri, Jan 19, 2018 at 3:31 PM, Andre wrote:
> May I suggest Metron follows the NiFi mailing list strategy (we got
> inspired by another project but I don't recall the name) and remove the
> github comments from the dev list?
+1 to both the feature branch and user@ announcement.
On Thu, Jan 18, 2018 at 2:45 PM, Otto Fowler
> +1 to the feature branch.
> Also, there have been some questions about solr support recently, I think
> when the feature branch
> is ready you should announce
So, the challenge here is that our install script isn't smart enough right
now to skip creating tables that are already created. One thing you could
1. rename the hbase tables for metron (see
I made an infra ticket: https://issues.apache.org/jira/browse/INFRA-15865
On Thu, Jan 18, 2018 at 11:42 AM, Otto Fowler
> 24hr long build is blocking up master’s travis build.
> Who can nuke it?
The Project Management Committee (PMC) for Apache Metron has invited Anand
Subramanian to become a committer and we are pleased to announce that they
Congratulations and welcome, Anand!
I haven't seen that one. I spun one up from master on Friday and it seemed
ok. Sorry, "works for me!" isn't super helpful, but it may be relevant
since master is close to 0.4.2 :)
On Mon, Jan 8, 2018 at 11:11 AM, Otto Fowler
> I just started up full dev from
> I would imagine the ‘stellar-object-repo’ would be part of the global
> configuration or configuration passed to the command.
> why specify in the function itself?
> On January 5, 2018 at 11:22:32, Casey Stella (ceste...@gmail.com) wrote:
features and level
> > abstraction feel appropriate to me. I think it buys us 1) learning from a
> > starting typosquatting use case, 2) flexibility to change and adapt it
> > without affecting users, and 3) enough concrete capability to make more
> > specific use ca
typosquatting use-case. Hard coding this would prevent things like bloom
filters containing malicious IPs from a reference source, for instance.
On Thu, Jan 4, 2018 at 10:46 AM, Casey Stella <ceste...@gmail.com> wrote:
> So, there is value outside of just bloom usage. The most specifi
1 - 100 of 208 matches
Mail list logo