So we graduated...

2017-04-20 Thread Casey Stella
For anyone paying attention to incubator-general, it will come as no surprise that we graduated as of last night's board meeting. We have a press released queued up and planned for monday along with a PR (METRON-687 at https://github.com/apache/incubator-metron/pull/539). It escaped my notice

Re: Failure to Deploy "Quick Dev"

2017-04-21 Thread Casey Stella
I'm betting we need to regenerate the quickdev image after all the mpack changes. On Fri, Apr 21, 2017 at 10:45 AM, Otto Fowler wrote: > From lira: > > I 'think' that quickdev is actually build from full_dev, with metron > installed already. So it may be that we need a

Re: Ansible Docker fails to build latest

2017-04-21 Thread Casey Stella
Sorry, can you go through how you're getting to this error? I'm not super familiar with this part of the stack..is this compiling Metron inside of a docker image? On Tue, Apr 18, 2017 at 1:45 PM, Otto Fowler wrote: > Is it something to do with the relocation of

Re: Silly question about zk_load_configs.sh

2017-04-24 Thread Casey Stella
, Otto Fowler <ottobackwa...@gmail.com> wrote: > OK, not to be thick, I just don’t see where any class from metron-parsers > gets called or instantiated. > ¯\_(ツ)_/¯ > > > On April 24, 2017 at 09:55:38, Casey Stella (ceste...@gmail.com) wrote: > > All of the current

Re: [DISCUSS] Persisting user data

2017-08-03 Thread Casey Stella
I'd vote for a DB-based solution, but I'd argue that any solution shouldn't be database specific (i.e. postgres), but JDBC-generic. People and organizations have very strong views regarding databases and I'd prefer to side-step those holy wars by being agnostic. On Wed, Aug 2, 2017 at 9:36 PM,

Re: MaaS and Metron Architecture talks at DataWorks Summit SJ 2017

2017-08-03 Thread Casey Stella
Ok, those talks are added. On Thu, Aug 3, 2017 at 3:44 PM, Casey Stella <ceste...@gmail.com> wrote: > Absolutely! > > On Thu, Aug 3, 2017 at 3:41 PM, Justin Leet <justinjl...@gmail.com> wrote: > >> Could we put these up on the wiki page for tech talks in the

Re: SimpleEnrichmentFlatFileLoaderIntegrationTest …..

2017-08-03 Thread Casey Stella
That is puzzling for sure. Another intermittent failing test mystery (dum dum dum), brought to you by the letter 'o' and the number Math.rand(). On Thu, Aug 3, 2017 at 3:15 PM, Otto Fowler wrote: > Results : > > Failed tests: > >

MaaS and Metron Architecture talks at DataWorks Summit SJ 2017

2017-08-03 Thread Casey Stella
The Videos of talks that Simon Ball and I gave at DataWorks Summit are now up and on youtube: * Solving Cyber at Scale (business-level track) - https://www.youtube.com/watch?v=zVdRhwfum4Q * Model as a Service (technical track) - https://www.youtube.com/watch?v=LkrOKvyAc0s * Metron Architecture

Re: MaaS and Metron Architecture talks at DataWorks Summit SJ 2017

2017-08-03 Thread Casey Stella
tent. > > https://cwiki.apache.org/confluence/display/METRON/Tech+Talks > > On Thu, Aug 3, 2017 at 10:32 AM, Casey Stella <ceste...@gmail.com> wrote: > >> The Videos of talks that Simon Ball and I gave at DataWorks Summit are >> now up and on youtube: >> >>

Re: Metron Alerts bombing in Travis?

2017-08-11 Thread Casey Stella
Yes, I'm getting it locally too, here's more context: [INFO] ERROR in multi script-loader!./~/jquery/dist/jquery.js script-loader!./~/tether/dist/js/tether.js script-loader!./~/ace-builds/src-noconflict/ace.js [INFO] Module not found: Error: Can't resolve

Re: Metron Alerts bombing in Travis?

2017-08-11 Thread Casey Stella
t; > wrote: > > > Where are the bootstrap files? > > > > the @import ~bootstrap/….. looks like it is failing? > > > > > > > > On August 11, 2017 at 09:51:03, RaghuMitra Kandikonda ( > > raghumitra....@gmail.com) wrote: > > > > I am ru

Re: threatintel_taxii_load.sh throws exception

2017-07-10 Thread Casey Stella
Thanks Vladimir, https://github.com/apache/metron/pull/643 was submitted a few moments ago. On Mon, Jul 10, 2017 at 3:35 PM, Vladimir Shlyakhtin < vladimir.shlyakh...@sstech.us> wrote: > Filed METRON-1026 > > - Vladimir > > From: Otto Fowler

Re: UI pivotting / aggregation backend

2017-07-07 Thread Casey Stella
I just want to chime in and support the notion of an abstraction layer between the UI and the indexed stores. I think that having an API that people can conform to is going to be important as people want to plug in their own backing indices in the future. Casey On Thu, Jul 6, 2017 at 2:11 PM,

Re: [DISCUSSION] METRON-1046 -> Stellar Files for multiple statement execution

2017-07-14 Thread Casey Stella
Just chiming in on a part of this: definitely we do not want to lose automatic config updates (at least, I'd be strongly, strongly STRONGLY against it). I definitely agree that JSON files could easily get unwieldy. I don't know anything about JSON pointers, could you cover that briefly, Matt?

Re: threatintel_taxii_load.sh throws exception

2017-07-10 Thread Casey Stella
Also, please reply here with the JIRA and I'll submit a PR. It's a very easy fix. On Mon, Jul 10, 2017 at 1:24 PM, Casey Stella <ceste...@gmail.com> wrote: > This is absolutely a bug and you should open a JIRA. > > On Mon, Jul 10, 2017 at 1:11 PM, Vladimir Shlyakhtin <

Re: threatintel_taxii_load.sh throws exception

2017-07-10 Thread Casey Stella
This is absolutely a bug and you should open a JIRA. On Mon, Jul 10, 2017 at 1:11 PM, Vladimir Shlyakhtin < vladimir.shlyakh...@sstech.us> wrote: > Hello, > > After upgrading from 0.3.0 version we noticed that taxii loader does not > work. > > Here is details: > > #

Re: [VOTE] Apache Metron 0.4.0 release

2017-06-29 Thread Casey Stella
+1 (binding) * Verified keys * Verified mvn build * Verified unit and integration tests run * Verified license check runs * Verified fulldev spun up with smoketest On Wed, Jun 28, 2017 at 8:10 PM, Anand Subramanian < asubraman...@hortonworks.com> wrote: > +1 (non-binding) > > * Brought up Metron

Re: [DISCUSS] Regression introduced in Full Dev

2017-04-25 Thread Casey Stella
Yeah, I tend to agree that a rundown of the various methods and when you would use them is in order. I will say that full-dev is especially important to have working since it is required for validating PRs. On Tue, Apr 25, 2017 at 18:56 zeo...@gmail.com wrote: > Can somebody

Re: Normalization topology or separate normalization bolt for parsing topology

2017-04-26 Thread Casey Stella
nd give us parsing > exception. The timestamp example was not a good one because that is > actually a post-parse exception. > > On Wed, Apr 26, 2017 at 11:28 PM, Casey Stella <ceste...@gmail.com> wrote: > > > So, further transformation post-parse was one of th

Re: Normalization topology or separate normalization bolt for parsing topology

2017-04-26 Thread Casey Stella
you going to deal with normalization after the parsing if > that noise affects the parsing? For some reason, the incoming data do not > look like in the way that has to be. > > On Wed, Apr 26, 2017 at 11:37 PM, Casey Stella <ceste...@gmail.com> wrote: > > > Ok, that's an

Re: Normalization topology or separate normalization bolt for parsing topology

2017-04-26 Thread Casey Stella
y field. We do after all > store the original_string for you if you really absolutely have to had > everything, so a more schema-on-read philosophy certainly applies and will > likely side-step a lot of your issues. > > Simon > > > On 26 Apr 2017, at 14:37, Casey Ste

Re: Normalization topology or separate normalization bolt for parsing topology

2017-04-26 Thread Casey Stella
So, further transformation post-parse was one of the motivating reasons for Stellar (to do that transformation post-parse). Is there a capability that it's lacking that we can add to fit your usecase? On Wed, Apr 26, 2017 at 9:24 AM, Ali Nazemian wrote: > I've created a

Re: Normalization topology or separate normalization bolt for parsing topology

2017-04-26 Thread Casey Stella
So, having stellar operate on the whole message is definitely something that would be cool. That being said, it's also nice to motivate the construction of functions to do simple transformations/normalizations. That way, common useful capabilities may be reused all the places Stellar is used

Re: [DISCUSS] Modify bylaws to allow speculative branches

2017-04-21 Thread Casey Stella
So, what would that look like from a practical perspective? * I presume commits would still associate to a JIRA, right? * Are you proposing changing the strategy from Review then Commit to Commit then Review for these branches? I know that we have some people who are active in the Hadoop project

Re: Don't forget to un-incubate

2017-04-28 Thread Casey Stella
I'm going to do my best to go through it tonight or tomorrow and file the appropriate tickets. If I run into issues, I'll ping you on monday Taylor. Sorry, I've been a bit busy. Best, Casey On Fri, Apr 28, 2017 at 5:12 PM, P. Taylor Goetz wrote: > general@ moved to BCC. >

Re: Don't forget to un-incubate

2017-04-28 Thread Casey Stella
Thanks Greg, I'm on it; been a busy week. Sorry about that. Best, Casey On Fri, Apr 28, 2017 at 4:11 PM, Greg Stein wrote: > Apache Metron, > > You need to move to a TLP. It has been over a week since the Board meeting, > and Infrastructure has not seen any tickets filed

Re: Silly question about zk_load_configs.sh

2017-04-24 Thread Casey Stella
rs, except what is brought in from > common, why doesn’t this just call common? > You don’t need parsers jar for any of this do you? > > > On April 24, 2017 at 09:22:06, Casey Stella (ceste...@gmail.com) wrote: > > Because stellar statements are validated prior to pushing, we needed

No longer incubating, but newly hatched!

2017-04-24 Thread Casey Stella
Hi All, Some of you know this already and some of you might not, but as of the last ASF board meeting we became a top level project with me serving as the Vice President of Apache Metron. The good people at the ASF press office scheduled some press early this morning. - NASDAQ GlobeNewswire

Re: [VOTE] Metron 0.4.0 release (RC2)

2017-04-28 Thread Casey Stella
I'd prefer to not release something where ES is getting killed due to misconfiguration. I'm -1 binding unless someone has more context. On Fri, Apr 28, 2017 at 6:56 AM, Anand Subramanian < asubraman...@hortonworks.com> wrote: > -1 (non-binding) > > Validated on a 12-node openstack CentOS 7

Re: Profiler statistics NaN

2017-08-09 Thread Casey Stella
Ok, so the problem here is that your profile is returning integers (specifically HLLP cardinalities) rather than stats objects. When you're doing: STATS_PERCENTILE(STATS_MERGE( PROFILE_GET('host-talks-to', '99.191.183.156', PROFILE_FIXED(10, 'HOURS')), 90) You are calling STATS_MERGE on a

Re: Profiler statistics NaN

2017-08-09 Thread Casey Stella
t; > "result": "HLLP_CARDINALITY(outcoming)" > > to this... > > "result": "outcoming" > > ? > > On Wed, Aug 9, 2017 at 3:48 PM Casey Stella <ceste...@gmail.com> wrote: > > > Ok, so the problem here is that your profile is re

Re: Profiler statistics NaN

2017-08-09 Thread Casey Stella
describe or complain that it doesn't know how to handle a > > list. Easy fix though. > > > > [Stellar]>>> STATS_MEAN(STATS_ADD(null, 1, 2, 3)) > > 2.0 > > > > [Stellar]>>> STATS_MEAN(STATS_ADD(null, [1,2,3])) > > 1.0 > &g

Re: Profiler statistics NaN

2017-08-09 Thread Casey Stella
friendly set of > facade functions, > or some thing would work. > > > > On August 9, 2017 at 16:38:48, Casey Stella (ceste...@gmail.com) wrote: > > Yeah, I'm leaning toward STATS_ADD or STATS_INIT taking a list of numbers. > STATS_MERGE seems confusing. > >

Dataworks summit Sydney

2017-05-03 Thread Casey Stella
The Australia/Pacific version of Dataworks Summit is in Sydney this year, September 20-21. This is a great place to talk about work you are doing in Apache Metron or how you are using Metron. Information on submitting an abstract is at

Re: Infrastructure migrated

2017-05-15 Thread Casey Stella
in the JIRA, please let me know and I'll add you. On Mon, May 15, 2017 at 9:30 AM, Casey Stella <ceste...@gmail.com> wrote: > For those who haven't noticed, the git infrastructure has migrated over. > If you are a committer, you will need to adjust the URLs for the apache > remote re

Infrastructure migrated

2017-05-15 Thread Casey Stella
For those who haven't noticed, the git infrastructure has migrated over. If you are a committer, you will need to adjust the URLs for the apache remote repo via: git remote set-url apache https://git-wip-us.apache.org/repos/asf/metron.git Obviously substitute "apache" for "upstream" if you

[DISCUSS] Enrichment Split/Join issues

2017-05-16 Thread Casey Stella
Hi All, Last week, I encountered some weirdness in the Enrichment topology. Doing some somewhat high-latency enrichment work, I noticed that at some point, data stopped flowing through the enrichment topology. I tracked down the problem to the join bolt. For those who aren't aware, we do a

Re: [DISCUSS] Enrichment Split/Join issues

2017-05-16 Thread Casey Stella
the knobs to tune, this architecture works, I believe. On Tue, May 16, 2017 at 12:09 PM, Casey Stella <ceste...@gmail.com> wrote: > We could definitely parallelize within the bolt, but you're right, it does > break the storm model. I also like making things other people's problems

Re: [DISCUSS] Enrichment Split/Join issues

2017-05-16 Thread Casey Stella
efficient (i.e. first labeling, and > “completing” a message and then dependent of label and completeness do > different other enrichments). > > So you have a +1 from me for serial rather than parallel enrichment. > > > BR, >Christian > > On 16.05.17, 16:58, "Casey Stel

Re: [DISCUSS] Enrichment Split/Join issues

2017-05-16 Thread Casey Stella
simplify the Enrichment > topology. I don't think we would not need the split/join pattern if we did > this. No? > > On Tue, May 16, 2017 at 11:54 AM, Casey Stella <ceste...@gmail.com> wrote: > > > The problem is that an enrichment type won't necessarily have a fixed >

Re: [INCOMING] Metron 0.4.0 release (RC3)

2017-06-09 Thread Casey Stella
I'm willing to be shot down about this, but METRON-995 fixes a pretty significant bug around using map variables and temporary variables in stellar enrichments. I'd appreciate if it could get in, but I'll understand if we think it shouldn't. On Sat, Jun 3, 2017 at 4:31 AM, Christian Tramnitz

Re: [INCOMING] Metron 0.4.0 release (RC3)

2017-06-19 Thread Casey Stella
Ok METRON-995 is in, so we should be able to release any time from my perspective. I think we're just waiting on some resolution for 941 at this point (that thread seems dead). On Fri, Jun 9, 2017 at 6:38 PM, Casey Stella <ceste...@gmail.com> wrote: > I'm willing to be shot d

[DISCUSS] Mutation of Indexed Data

2017-06-21 Thread Casey Stella
Hi All, I know we've had a couple of these already, but we're due for another discussion of a sensible approach to mutating indexed data. The motivation for this is users will want to update fields to correct and augment data. These corrections are invaluable for things like feedback for ML

Re: storm-kafka-client use in Metron

2017-06-21 Thread Casey Stella
Thanks for the heads up, kris. This is really good context and we should be aware of it. So we could avoid this by avoiding subscription objects which rely on consumer.subscribe and prefer consumer.assign, correct? On Tue, Jun 20, 2017 at 10:07 PM, Kristopher Kane wrote:

[DISCUSS] Metadata Ingest

2017-06-21 Thread Casey Stella
Hi All, I wanted to call attention to a JIRA (METRON-1001) that I just submitted and possibly discuss it more broader than on the PR. Currently, we only ingest data in Metron. Often, there is valuable metadata constructed up-stream of Metron that is relevant to enrichment and cross-cuts many

Re: [DISCUSS] Metadata Ingest

2017-06-22 Thread Casey Stella
d like to talk about if some of this > activity > > is more enrichment than not, and should be handled/exposed there, where > we > > have the splitter/joiner pattern already. > > > > - Other than exposing the metadata, I am not sure I understand the > > difference

Re: [DISCUSS] Mutation of Indexed Data

2017-06-26 Thread Casey Stella
be > painful. > >> > > >> > Justin > >> > > >> > > >> > On Wed, Jun 21, 2017 at 10:18 PM, Simon Elliston Ball < > >> > si...@simonellistonball.com> wrote: > >> > > >> > > I'd say that w

Re: Metron Release rules

2017-06-27 Thread Casey Stella
+1 to removing it. Other top level projects do not have a disclaimer (see, for example, hbase: http://www-eu.apache.org/dist/hbase/stable) On Tue, Jun 27, 2017 at 4:40 AM, Otto Fowler wrote: > Is there an equivalent disclaimer for full products? > > On June 26, 2017 at

Re: [DISCUSS] Metron IRC channel

2017-05-25 Thread Casey Stella
ask for what we > > mentioned above? > > > > Jon > > > > On Sun, Dec 18, 2016 at 1:58 PM Kyle Richardson < > kylerichards...@gmail.com> > > wrote: > > > >> I'll second the JIRA and Git integrations. I also like the meeting > minutes > >

Re: [Discuss] Cyber Security Asset Management for Metron

2017-05-25 Thread Casey Stella
I definitely sympathize with the desire to have a graph database part of the architecture, but I concur with Ali; the reputations for scalable graph databases aren't the best. I have resisted in pushing it so far because of the concern about stability of an implementation. I think we should

Re: [DISCUSS] Enrichment Split/Join issues

2017-05-16 Thread Casey Stella
and Geo enrichment bolts from the Enrichment > > topology. Stellar provides a user with much greater flexibility than the > > existing HBase and Geo enrichment bolts. > > > > A side effect of this would be to greatly simplify the Enrichment > > topology. I don't thi

Re: Question about the customization of Metron with my machine learining algo.

2017-06-05 Thread Casey Stella
We do not ship any ML models currently with metron, just the infrastructure to deploy your own models and interact with those models from within Metron. That being said, you might be interested in https://gist.github.com/cestella/8dd83031b8898a732b6a5a60fce1b616 That's the code to take a DGA

Re: Question about the customization of Metron with my machine learining algo.

2017-06-06 Thread Casey Stella
tation process. > > For example if I have an SVM algo that I would test into Metron and that > ML algortihm has been developed in python using scikit-py. > > How can I do that? > > Thank you and I'm sorry for the very basic question. > > Best Regards, > > Simone >

Re: [ANNOUNCE] Apache Metron Release 0.4.1

2017-09-19 Thread Casey Stella
Fantastic! I'm really proud of this release and a great job was done by Matt and the community for getting this out! On Tue, Sep 19, 2017 at 1:24 PM, Frank Horsfall < frankhorsf...@cunet.carleton.ca> wrote: > Congrats guys! > > > > Frank > > > > > > *From:* zeo...@gmail.com

Re: Assign issue to user (754: METRON-1184 , 713: METRON-1130 )

2017-09-13 Thread Casey Stella
done, thanks for your contribution! On Wed, Sep 13, 2017 at 12:02 PM, Ahmed Shah wrote: > Hello, > > > I recently submitted a new PR and JIRA Issue. > > According to the dev guidelines apache.org/confluence/pages/viewpage.action?pageId=61332235>,

Re: [VOTE] Metron Release Candidate 0.4.1-RC4

2017-09-14 Thread Casey Stella
I ran it up and forgot to vote. +1 binding - ran in full-dev - verified signatures - stellar works On Thu, Sep 14, 2017 at 9:52 AM, Matt Foley wrote: > Hi Team, > Unfortunately, it appears I was incorrect to say it’s passing just yet. > On reviewing the rules, I find that a

Re: [DISCUSS] Upgrading Elasticsearch from 2.x to 5.x

2017-10-06 Thread Casey Stella
Yeah, I agree with what Michael "fine whine" Miklavcic said; I'm in favor of the high level client. On Thu, Oct 5, 2017 at 3:35 PM, Michael Miklavcic < michael.miklav...@gmail.com> wrote: > Justin, thanks for the feedback! I'm inclined to agree with you about using > the high level client. It's

Re: Cloudtrail use case

2017-10-06 Thread Casey Stella
There is actually a use-cases top level directory with worked examples in them. They get picked up by the doc book too! I'd suggest putting it there, thoughts? On Fri, Oct 6, 2017 at 8:44 AM, Nick Allen wrote: > Yes, agreed, Justin. I guess my main point to Laurens was

Re: [DISCUSS] Dropping support for elastic 2.x

2017-10-04 Thread Casey Stella
So, how would this work in an upgrade scenario that does not involve losing the existing indexed data? On Wed, Oct 4, 2017 at 12:55 PM, Michael Miklavcic < michael.miklav...@gmail.com> wrote: > The client I'm currently working on moving towards would *not* be backwards > compatible. >

Re: [DISCUSS] Dropping support for elastic 2.x

2017-10-04 Thread Casey Stella
rk, but it looks like it's just calling > > shell commands from python, e.g. https://github.com/ > > apache/metron/blob/master/metron-deployment/packaging/ > > ambari/metron-mpack/src/main/resources/common-services/ > > ELASTICSEARCH/2.3.3/package/scripts/elastic_master.p

Re: [DISCUSS] Dropping support for elastic 2.x

2017-10-04 Thread Casey Stella
> > from the HDFS store. Alternatively there are means to do inplace upgrades > > from 2.x to 5.x I believe. > > > > Simon > > > > > On 4 Oct 2017, at 18:05, Casey Stella <ceste...@gmail.com> wrote: > > > > > > So, how would this work in an upgrad

Re: [VOTE] Metron Release Candidate 0.4.1-RC3

2017-09-08 Thread Casey Stella
Yeah, I fixed the dependencies_with_url problem in METRON-1169. Let's pull that one in and the rat check one and cut a new RC. I'm -1 On Fri, Sep 8, 2017 at 7:38 PM, Matt Foley wrote: > Couple pieces of info that may affect your vote: > > 1. This does not include today’s

Re: Unclear recent commit

2017-09-08 Thread Casey Stella
My guess is that what happened was otto did a pull of METRON-1061 into his master, but it wasn't *quite* in sync with master in apache and there was a conflict in the dependencies_with_url.csv, so that commit was made locally to fix the conflict. both commits, the squashed and the merge conflict

Re: Unclear recent commit

2017-09-08 Thread Casey Stella
I don't have an issue with this; it's good to have the explanation. I also found that we weren't running the dependency analyzer as part of travis (or, we thought we were, but we weren't) until METRON-1169. Thanks otto for the explanation and jon for catching it. On Fri, Sep 8, 2017 at 10:09

Re: [VOTE] Metron Release Candidate 0.4.1-RC3

2017-09-08 Thread Casey Stella
Just a FYI, I think all the relevant PRs to fix the rat check and dependency analyzer are committed now and we can cut another RC whenever we're ready. On Fri, Sep 8, 2017 at 8:24 PM, Casey Stella <ceste...@gmail.com> wrote: > Yeah, I fixed the dependencies_with_url problem in ME

Re: Unclear recent commit

2017-09-08 Thread Casey Stella
That's very weird. On Fri, Sep 8, 2017 at 10:12 PM, Otto Fowler <ottobackwa...@gmail.com> wrote: > without re-writing my other reply: > I use the prepare-commit into a clean repo every time, so I believe that > this issue > *was* in the apache repo. > > > On September

Re: [DISCUSS] Metron release 0.4.1

2017-08-29 Thread Casey Stella
For my PRs, I'd vote for METRON-1122 being in (commit very imminent). I'd very much like METRON-1134 to be in as well. Beyond that, I'm ok On Tue, Aug 22, 2017 at 4:37 PM, Nick Allen wrote: > Thanks for starting the process, Matt. > > These are my own open PRs that I would

Re: [DISCUSS] Metron release 0.4.1

2017-08-29 Thread Casey Stella
do you have an ETA for these two PRs? (PR#709 and 717) > Thanks, > --Matt > > On 8/29/17, 9:34 AM, "Casey Stella" <ceste...@gmail.com> wrote: > > For my PRs, I'd vote for METRON-1122 being in (commit very imminent). > I'd very much like METRON-1134 to be in as well. &

Re: Upgrade from Metron 0.4.1-rc to 0.4.1-rc4 problems.

2017-09-09 Thread Casey Stella
So the escalation topic is a new parameter for the REST service in 0.4.1. It appears that the ambari upgrade story is a bit weak. Is it possible to modify /var/lib/ambari-agent/cache/ common-services/METRON/0.4.1/package/templates/metron.j2 and - create a kafka topic called 'metron_escalation'

Re: [DISCUSS] Metron release 0.4.1

2017-09-05 Thread Casey Stella
t; wrote: >> >> > > >> >> > >> The following PRs are usability enhancements for the >> Profiler. They >> >> are >> >> > >> fairly simple and I think are very helpful for >> troubleshooting. I >> &g

Re: Unclear recent commit

2017-09-08 Thread Casey Stella
d against policy? What should I have done? > I think the committer’s guide is archived btw. > > > On September 8, 2017 at 22:12:27, Casey Stella (ceste...@gmail.com) wrote: > > I don't have an issue with this; it's good to have the explanation. I also > found that we weren't r

[DISCUSS] Splitting up the Indexing Topology

2017-09-25 Thread Casey Stella
One of the lessons that have bubbled up in doing some performance analysis is that having the indexing topology share both the ES and the HDFS writer in the same topology can be problematic from a tuning perspective. Specifically, it's hard to square that circle and make both perform fast enough

[DISCUSS] Build broken due to transitive dependencies

2017-10-02 Thread Casey Stella
Ok, the build is broken in metron-config due to some transitive changes that happened in npm-land: [INFO] /Users/cstella/Documents/workspace/metron/fork/incubator-metron/metron-interface/metron-config/node_modules/toposort/index.js:32 [INFO] throw new Error('Cyclic dependency:

Re: [DISCUSS] Build broken due to transitive dependencies

2017-10-02 Thread Casey Stella
rking at any moment because its dependencies can change. Prevent this by > migrating to Yarn: https://bower.io/blog/2017/how-to-migrate-away-from- > bower/ > > > On October 2, 2017 at 11:29:50, Casey Stella (ceste...@gmail.com) wrote: > > Ok, I can verify that 0.4.1 did build

Re: [DISCUSS] Build broken due to transitive dependencies

2017-10-02 Thread Casey Stella
ntly in a pr, though I can’t > remember which one > > > On October 2, 2017 at 11:45:48, Casey Stella (ceste...@gmail.com) wrote: > > Yeah, seems like it might be worth while to seriously investigate > migrating to yarn if that gets us a consistent build. > > On Mon, Oct

Re: [ANNOUNCE] Metron community meeting

2017-08-24 Thread Casey Stella
forever as far as I'm concerned. They don't really close so much as they are forgotten. :) On Thu, Aug 24, 2017 at 11:08 AM, Otto Fowler <ottobackwa...@gmail.com> wrote: > Casey, what is the time frame for the discuss thread to ‘close’? > > > On August 24, 2017 at 10:06:12, C

Re: [ANNOUNCE] Metron community meeting

2017-08-24 Thread Casey Stella
Yeah, sorry about that; technology (and webex in particular) is such a fragile thing sometimes. I think the synopsis is pretty complete. That being said, if there's anything that you want clarity on, we can hammer it out on the discuss thread too. On Wed, Aug 23, 2017 at 4:28 PM, James Sirota

Re: [ANNOUNCE] Metron community meeting

2017-08-24 Thread Casey Stella
weekend plus a business day has been elapsed - Some general consensus is reached - A full day has gone by without any dissent If nobody responds, then treat that as consent by silence. That's what I normally use. :) On Thu, Aug 24, 2017 at 11:09 AM, Casey Stella <ceste...@gmail.com>

Re: Question on Hdfs Writer

2017-08-28 Thread Casey Stella
The storm metron-writer bolt has to write to many different sensor's HDFS files. The hdfs bolts provided by storm only support writing to one file (or set of files). The alternative would have been a bolt per sensor, which did not support our architecture. In short, the hdfs bolt from storm

Re: [DISCUSS] Metron release 0.4.1

2017-08-30 Thread Casey Stella
ait until Friday, if > necessary, to cut an RC with 717 in it? > Thanks, > --Matt > > On 8/29/17, 11:45 AM, "Casey Stella" <ceste...@gmail.com> wrote: > > 709 is in and 717 is under concerted review by Otto. I'd like to see > it in > by Friday. >

Re: SUM aggregator not working?

2017-10-04 Thread Casey Stella
Ok, so this is subtle. Your rules are wrong and I totally understand why you thought they were right. When we index into ES, we take . and convert them to :, however PRIOR to indexing (when threat triage is running) those fields have .'s not :'s Therefore, your rules should be:

Re: Stellar support for switch/case style conditionals

2017-10-17 Thread Casey Stella
Ugh, I forgot to preface this with DISCUSS: Sorry! On Tue, Oct 17, 2017 at 12:05 PM, Casey Stella <ceste...@gmail.com> wrote: > Hi All, > > It's high time that Stellar supports some form of conditional that is > beyond if/then/else. Right now, the way to do fall-th

Re: Stellar support for switch/case style conditionals

2017-10-17 Thread Casey Stella
it. > > So default would be a keyword? > > and a lambda that uses x can be used on the right side of the : > > > > On October 17, 2017 at 14:21:01, Casey Stella (ceste...@gmail.com) wrote: > > So, just to map this onto the example, you mean: > match(longer_variable ->

Re: Stellar support for switch/case style conditionals

2017-10-17 Thread Casey Stella
nd it’s value is returned > * no matches returns null or return of optional final statement, which is > a LAMBDA without a BOOLEAN_STATEMENT > > > On October 17, 2017 at 12:06:05, Casey Stella (ceste...@gmail.com) wrote: > > Ugh, I forgot to preface this with DISCUSS: Sorry! &

Re: Stellar support for switch/case style conditionals

2017-10-17 Thread Casey Stella
There's no string concat candy, so that'd probably be default: FORMAT('critical-%s%s', x, y) On Tue, Oct 17, 2017 at 2:28 PM, Otto Fowler wrote: > match(longer_variable -> x, other_variable -> y) { x < 10 : 'info', x <= > 20 : 'warn’, x < y : ‘oh boy’, default:

Re: Master build failures in Travis

2017-10-23 Thread Casey Stella
Yeah, that could be a consequence. With the cache in place, the calls to delete are async. This isn't generally a problem in an actual installation, but in the integration tests, it can take some time to sync up (depending on the load). I ran it 20 or so times teasing these out, but it's never

Re: Master build failures in Travis

2017-10-23 Thread Casey Stella
Looks like Ryan got there first, which is awesome. Thanks for cleaning up my mess :) On Mon, Oct 23, 2017 at 10:04 AM, Casey Stella <ceste...@gmail.com> wrote: > Yeah, that could be a consequence. With the cache in place, the calls to > delete are async. This isn't general

Re: [DISCUSS] Upcoming Release

2017-11-15 Thread Casey Stella
I'd say that if a release is this imminent that we had better notify the release manager who will make a release announcement, Nick. Matt, are you tuning in to this? On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen wrote: > Hi Guys - > > I want to follow-up on this discussion.

Re: [DISCUSS] e2e test infrastructure

2017-11-29 Thread Casey Stella
Honestly, I'm ok with either the in-memory component approach or the docker approach as long as: - It runs in travis - The infrastructure components are spun up in a way that isolates their classpath - The UI e2e test and the integration tests both use the same infrastructure I

Re: [DISCUSS] Upcoming Release

2017-12-04 Thread Casey Stella
I would be in favor of a release at this point. On Mon, Dec 4, 2017 at 4:57 PM, Matt Foley wrote: > Hey all, > I see METRON-1252 was resolved over the weekend. Shall I go ahead and > start the process with 0.4.2 release? > Does anyone have any commits they feel strongly

Re: DISCUSS: Quick change to parser config

2017-12-04 Thread Casey Stella
So, just chiming in here. It seems to me that we have a problem with extraneous fields in a couple of different ways: * Temporary Variables I think that the problem of temporary variables is one beyond just the parser. What I'd like to see is the Stellar field transformations operate similar

Re: [DISCUSS] Stellar Documentation Autogeneration

2017-12-14 Thread Casey Stella
chiming in with a +1 on my end too. This would be fantastic. On Thu, Dec 14, 2017 at 2:51 PM, Nick Allen wrote: > +1 I think it is a great idea, Justin and the only way that we'll keep the > docs in-sync with the code. > > > > > > On Thu, Dec 14, 2017 at 2:32 PM Justin Leet

Re: [DISCUSS] Support Ubuntu Installs in the MPack

2017-12-15 Thread Casey Stella
Nick is right that the ASF does not provide support in an explicit way (i.e. there are no pathways to get *prioritized* support via SLAs, etc.), but it is expected that apache projects provide support via mailing lists and answered by volunteers. Specifically, this is the crux of the "community

Re: [DISCUSS] Upcoming Release

2017-12-15 Thread Casey Stella
, Matt Foley <mfo...@hortonworks.com> wrote: > Perhaps under “build_utils” we should add a subdirectory for > “release_utils”. > > From: Casey Stella <ceste...@gmail.com> > Date: Friday, December 15, 2017 at 10:50 AM > To: "dev@metron.apache.org" <d

Re: [DISCUSS] Upcoming Release

2017-12-15 Thread Casey Stella
METRON-1228Done > Unassigned https://issues.apache.org/jira/browse/METRON-1228 > <https://issues.apache.org/jira/browse/METRON-1228>* > *METRON-1218 To Do Ryan > Merriman https://issues.apache.org/jira/browse/METRON-1

Re: [VOTE] Metron Release Candidate 0.4.2-RC2

2017-12-19 Thread Casey Stella
+1 validated via Otto's script * Checksums * Sigs * Build * Full dev validation On Tue, Dec 19, 2017 at 2:45 PM, Nick Allen wrote: > +1 I validated using Otto's great script. > > * Validated the list of changes > * Checksums > * Sigs > * Build > * Tests > * Full Dev > > On

Re: [DISCUSS] Stellar in a Zeppelin Notebook

2017-12-19 Thread Casey Stella
I love it! I wonder if we could get more of the REPL-like experience (i.e. I crave autocomplete ;) if we integrated it with jquery shell like they did with nanook (https://github.com/aeshell/nanook). I know zeppelin lets you integrate with more complex javascript. Regardless, this is awesome,

Re: [DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

2017-11-16 Thread Casey Stella
I'd recommend restarting this thread with this subject and including [MENTORS] in the subject line. At least I don't know the answer to this and I'd want broader visibility so we get more responses. On Thu, Nov 16, 2017 at 9:10 AM, Nick Allen wrote: > The code of the 'Kafka

New PMC members

2017-12-07 Thread Casey Stella
The Project Management Committee (PMC) for Apache Impala has invited Otto Fowler, Michael Miklavcic and Justin Leet to become a PMC member and we are pleased to announce that they have accepted. Congratulations and welcome!

Re: New PMC members

2017-12-07 Thread Casey Stella
Well, obviously, I meant Metron instead of Impala. To this point, we should have a wiki page around templates for this, similar to the impala project. :) On Thu, Dec 7, 2017 at 10:06 AM, Casey Stella <ceste...@gmail.com> wrote: > The Project Management Committee (PMC) for Apache I

  1   2   3   >