Re: Beware Homebrew Netcat ( 0.7.1 )

https://issues.apache.org/jira/browse/METRON-713 On Fri, Feb 10, 2017 at 3:32 PM, Otto Fowler <ottobackwa...@gmail.com> wrote: > Jira? > > > On February 10, 2017 at 15:25:15, Casey Stella (ceste...@gmail.com) wrote: > > Thanks for running that down, Otto. We should re

[VOTE] Releasing Apache Metron (incubating) 0.3.1-RC4

This is a call to vote on releasing Apache Metron 0.3.1-RC4 incubating Full list of changes in this release: https://dist.apache.org/repos/dist/dev/incubator/metron/0.3.1-RC4-incubating/CHANGES The tag/commit to be voted upon is apache-metron-0.3.1-rc4-incubating:

[RESULT][VOTE] Releasing Apache Metron (incubating) 0.3.1-RC3

We forgot to Rev the MPack version number; as such, it was decided to get that in before cutting the release. Results: -1: Casey Jon Zeolla (non-binding)

Re: [ANNOUNCE] Metron Apache Community Demo for Metron_0.3.1

e to make it. > > Jon > > On Wed, Feb 8, 2017 at 10:14 AM Casey Stella <ceste...@gmail.com> wrote: > > > I'd like to show the modifications/enhancements to the flat file loader > > > > On Wed, Feb 8, 2017 at 4:01 AM, Jimson K James <tomsma...@aczire.com>

Re: Unable to build Ansible 2.0.0.2 on macOS using our instructions

we can avoid it. > > -D... > > > On Fri, Feb 10, 2017 at 9:26 AM, Casey Stella <ceste...@gmail.com> wrote: > > > Hmm, that's disconcerting. I certainly don't mind documenting the > > workaround that you've proposed in METRON-708 > > > > On Thu, Feb 9,

Re: Unable to build Ansible 2.0.0.2 on macOS using our instructions

Hmm, that's disconcerting. I certainly don't mind documenting the workaround that you've proposed in METRON-708 On Thu, Feb 9, 2017 at 1:05 PM, zeo...@gmail.com wrote: > Has anybody attempted to set up ansible 2.0.0.2 recently on a fresh mac > using brew and the steps we

Re: [VOTE] Releasing Apache Metron (incubating) 0.3.1-RC3

PM zeo...@gmail.com <zeo...@gmail.com> wrote: > > > -1 (non-binding) due to mpack revision number > > > > On Thu, Feb 9, 2017, 2:42 PM Casey Stella <ceste...@gmail.com> wrote: > > > > -1, we didn't rev the mpack. Discussion going on currently as to what &g

Re: [VOTE] Releasing Apache Metron (incubating) 0.3.1-RC3

-1, we didn't rev the mpack. Discussion going on currently as to what version it should be and Justin volunteered to do the work. I vote we wait for that and cut another RC. On Thu, Feb 9, 2017 at 9:23 AM, Casey Stella <ceste...@gmail.com> wrote: > This is a call to vote on releasi

Re: Rev additional metron components?

I do agree that the MPack should be rev'd and a new RC should be cut. Is there a way to name the versioning of the management pack so that it indicates the oldest version of Metron that can be installed with that version? So, in this case, maybe 0.3.1.0? Also, I'm looking for volunteers to take

[VOTE] Releasing Apache Metron (incubating) 0.3.1-RC3

This is a call to vote on releasing Apache Metron 0.3.1-RC3 incubating Full list of changes in this release: https://dist.apache.org/repos/dist/dev/incubator/metron/0.3.1-RC3-incubating/CHANGES The tag/commit to be voted upon is apache-metron-0.3.1-rc3-incubating:

[RESULT][VOTE] Releasing Apache Metron (incubating) 0.3.1-RC2

The vote fails due to issues with the ansible deploy which have been corrected as of METRON-707. RC3 will be out momentarily. Results: +1 Nick Allen Anand Subramanian (non-binding) Casey Stella -1 James Sirota

Re: [VOTE] Releasing Apache Metron (incubating) 0.3.1-RC2 (corrected links, same RC)

<n...@nickallen.org> wrote: > Is this error encountered when deploying on Quick Dev? I'm confused when > this bug was introduced. I never ran into this when testing RC1 or RC2. > > On Wed, Feb 8, 2017 at 3:47 PM, Casey Stella <ceste...@gmail.com> wrote: > > > Th

Re: [ANNOUNCE] Metron Apache Community Demo for Metron_0.3.1

I'd like to show the modifications/enhancements to the flat file loader On Wed, Feb 8, 2017 at 4:01 AM, Jimson K James wrote: > Hi Team, > > It'd be useful if anyone can walkme through metron-docker working with a > Kafka sensor and doing basic enrichment. > > Thanks &

[VOTE] Releasing Apache Metron (incubating) 0.3.1-RC2 (corrected links, same RC)

This is a call to vote on releasing Apache Metron 0.3.1-RC2 incubating Full list of changes in this release: https://dist.apache.org/repos/dist/dev/incubator/metron/0.3.1-RC2-incubating/CHANGES The tag/commit to be voted upon is apache-metron-0.3.1-rc2-incubating:

Re: [VOTE] Releasing Apache Metron (incubating) 0.3.1-RC2

count three, starting > with the first line :-) There is also an instance of “0.3.0”. > Thanks, > --Matt > > On 2/7/17, 8:18 AM, "Casey Stella" <ceste...@gmail.com> wrote: > > This is a call to vote on releasing Apache Metron 0.3.1-RC1 incubating > > &g

Re: [DISCUSS] Build Times are getting out of hand

haha there was some desperation there, I'll admit. ;) On Tue, Feb 7, 2017 at 3:12 PM, Otto Fowler <ottobackwa...@gmail.com> wrote: > This PR gets a star just for the commit messages, it isn’t even Friday > Casey > > > On February 7, 2017 at 14:49:22, Casey Stella (cest

Re: [DISCUSS] Build Times are getting out of hand

Casey suggested." :) > > > > -D... > > > > > > On Tue, Feb 7, 2017 at 10:45 AM, Michael Miklavcic < > > michael.miklav...@gmail.com> wrote: > > > > > I agree with this. I don't think we should switch to an alternate > system > > > until

[VOTE] Releasing Apache Metron (incubating) 0.3.1-RC2

This is a call to vote on releasing Apache Metron 0.3.1-RC1 incubating Full list of changes in this release: https://dist.apache.org/repos/dist/dev/incubator/metron/0.3.1-RC2-incubating/CHANGES The tag/commit to be voted upon is apache-metron-0.3.0-rc1-incubating:

Re: [DISCUSS] Build Times are getting out of hand

kwa...@gmail.com> wrote: > Is there an alternative to Travis? Do other like sized apache projects > have these problems? Do they use travis? > > > On February 6, 2017 at 17:02:37, Casey Stella (ceste...@gmail.com) wrote: > > For those with pending/building pull requests,

Re: [DISCUSS] Build Times are getting out of hand

.. SUCCESS [09:38 min] > metron-data-management . SUCCESS [09:15 min] > elasticsearch-shaded ... SUCCESS [08:05 min] > > I'm going to take a look at Travis and also see what pom dependencies I can > start excluding.

[RESULT][VOTE] Releasing Apache Metron (incubating) 0.3.1-RC1

The vote fails; a new release candidate will be cut when METRON-703 is accepted. Results: +1 Nick Allen James Sirota Casey Stella -1 David Lyle

Re: [VOTE] Releasing Apache Metron (incubating) 0.3.1-RC1

> > Build successful > > > Integration tests successful > > > Deploy of "Full Dev" successful > > > Deploy of "Quick Dev" successful > > > > > > On Mon, Feb 6, 2017 at 3:43 PM, Casey Stella <ceste...@gmail.com> > wrote: >

[DISCUSS] Build Times are getting out of hand

For those with pending/building pull requests, it will come as no surprise that our build times are increasing at a pace that is worrisome. In fact, we have hit a fundamental limit associated with Travis over the weekend. We have creeped up into the 40+ minute build territory and travis seems to

[VOTE] Releasing Apache Metron (incubating) 0.3.1-RC1

This is a call to vote on releasing Apache Metron 0.3.1-RC1 incubating Full list of changes in this release: https://dist.apache.org/repos/dist/dev/incubator/metron/0.3.1-RC1-incubating/CHANGES The tag/commit to be voted upon is apache-metron-0.3.0-rc1-incubating:

Re: [Discuss] Direction of metron-docker

So, I'm late chiming in here, but I'll go ahead anyway. :) There are a couple of questions here that stand out: *Is the docker infrastructure sufficient to replace vagrant at the moment?* I do not consider it to be a sufficient environment to acceptance test features because it does not install

Re: [DISCUSS] Contributions with multiple authors and requisite modifications to the development guide

>> Just to be clear - I’m not pushing for this, I was more curious about > > how > > >> to do it if we wanted to. My main interest is that the process we > agree > > to > > >> fosters and encourages collaboration rather than make it such a pain > > that >

Re: Code Review - METRON-571

Just chiming in, also wanted to thank you for taking it on. Justin is right, the right venue for feedback here is on a pull request submitted via github. The steps roughly are: 1. Fork https://github.com/apache/incubator-metron.git by going to https://github.com/apache/incubator-metron

Re: [DISCUSS] Next Release (0.3.1) Content

to change, but could you comment on this thread in the dev list so I know if we need to wait for METRON-650. Casey On Thu, Feb 2, 2017 at 4:58 PM, Casey Stella <ceste...@gmail.com> wrote: > Ok, I've created the upgrading document for 0.3.0 to 0.3.1 and included > the things that

Re: [DISCUSS] Next Release (0.3.1) Content

constitutes a mention in the development > guidelines > <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235 > > > once > it's in master. > > Jon > > On Fri, Jan 27, 2017 at 9:05 AM Casey Stella <ceste...@gmail.com> wrote: > > >

Re: [DISCUSS] Expansion of the capabilities of PROFILE_GET

31, 2017 at 2:25 PM, Casey Stella <ceste...@gmail.com> wrote: > +1 to that. Here here! :) > > On Tue, Jan 31, 2017 at 2:21 PM, Nick Allen <n...@nickallen.org> wrote: > >> Yes, agreed. However we specify the schedule, it would be decoupled from >> the Profile

Re: [Discuss] Improve Alerting

p of another proposal that takes this idea a little further. What > if we treated the Profiler as another source of telemetry? > > On Wed, Feb 1, 2017 at 2:23 PM, Casey Stella <ceste...@gmail.com> wrote: > > > Regarding point 2, could we enable the profiler to write data to

Re: [Discuss] Improve Alerting

I like the direction. One thing that we may want is for comment to just be a stellar expression and construct a function to essentially do String.format(). So, that'd become: "triageConfig" : { "riskLevelRules" : [ { "name" : "Abnormal Value", "comment" : "FORMAT('For %s; the

Re: [Discuss] Improve Alerting

Regarding point 2, could we enable the profiler to write data to kafka and the enrichment queue? I'm proposing the profiler do something like this: - Count the number of inbound flows - On the tick, send a message to the enrichment queue containing: - the number of flows - A

Re: [DISCUSS] Expansion of the capabilities of PROFILE_GET

ofiles := PROFILE_GET("profile1", "entity1", timestamps) > > > Or > > timestamps := STELLAR_DSL("on every other Tuesday") > profiles := PROFILE_GET("profile1", "entity1", timestamps) > > > > > On Tue, Jan 31, 2017 at 2:01 PM

Re: [DISCUSS] Expansion of the capabilities of PROFILE_GET

d, if possible, ended) so that I can say “Get that old profile > from November 14-21, whatever its metadata was” without needing to know its > period, groups, etc. Obviously it would be nice to be able to query the > metadata itself, too, but just having the metadata gives us the right start.

Re: [DISCUSS] Expansion of the capabilities of PROFILE_GET

One more point, one of the reasons for decoupling the PROFILE_GET from PROFILE_LOOKUP means that we could ahve alternative implementations of PROFILE_LOOKUP. We could have a PROFILE_LOOKUP_CRON as well. On Tue, Jan 31, 2017 at 1:43 PM, Casey Stella <ceste...@gmail.com> wrote: >

Re: [DISCUSS] Expansion of the capabilities of PROFILE_GET

uot; except @USHolidays or something like that. I'd > have to look into this some more. > > And there are also nice online Cron expression "translators" that we could > mimic in a Metron user interface. For example, https://crontab.guru. > > > > > On Tue, Jan 31,

Re: [DISCUSS] Expansion of the capabilities of PROFILE_GET

s well-known semantics proven over many years > > > > > > On Tue, Jan 31, 2017 at 12:00 PM, Casey Stella <ceste...@gmail.com> wrote: > > > I actually did consider cron initially but dismissed it for the following > > reasons: > > > >- Cron synt

Re: [DISCUSS] Expansion of the capabilities of PROFILE_GET

> > On Mon, Jan 23, 2017 at 3:01 PM, Casey Stella <ceste...@gmail.com> wrote: > > > Hi All, > > > > I'm planning to expand the capabilities of PROFILE_GET and wanted to pass > > an idea past the community. > > > > *Current State* > > > > Curr

Re: [DISCUSS] Contributions with multiple authors and requisite modifications to the development guide

ull+Requests > > > On January 31, 2017 at 09:04:38, Casey Stella (ceste...@gmail.com) wrote: > > The problem is that a single commit cannot have multiple authors and it's > difficult to construct a minimal set of commits for multiple authors in > the > same PR by

Re: [DISCUSS] Contributions with multiple authors and requisite modifications to the development guide

thor: 1 METRON-123 Fool all the bars - But, there was this little deal > > I think it's hard to cover all the cases, so maybe a statement of the > principle we're trying to maintain? > > -D... > > > On Fri, Jan 27, 2017 at 5:50 PM, Casey Stella <ceste...@gmail.com> w

Re: [DISCUSS] Contributions with multiple authors and requisite modifications to the development guide

ficulties > reviewing and testing. The changeset should be reviewed and tested as a > single unit. If there is more than one author, simply don't squash. > > -D... > > > On Fri, Jan 27, 2017 at 5:44 PM, Casey Stella <ceste...@gmail.com> wrote: > > > It is important for eve

Re: [MENTORS] ICLA for non-committer contributions

> probably make that clear in the development guide if it isn't already. > > > > Jon > > > > On Fri, Jan 27, 2017, 5:08 PM Casey Stella <ceste...@gmail.com> wrote: > > > > > I should add, each of them should have the same PR title, just >

Re: [MENTORS] ICLA for non-committer contributions

wrote: > What Ryan has suggested has happened twice before from memory. We should > probably make that clear in the development guide if it isn't already. > > Jon > > On Fri, Jan 27, 2017, 5:08 PM Casey Stella <ceste...@gmail.com> wrote: > > > I should add, each of t

[DISCUSS] Contributions with multiple authors and requisite modifications to the development guide

It is important for every contribution to attribute the author, our git log is a legal papertrail for attribution. As a consequence of that, I recommend the Development Guidelines section 1.2 be amended to the following (bolded change): Everyone is encouraged to review open pull requests. We

Re: [MENTORS] ICLA for non-committer contributions

I should add, each of them should have the same PR title, just different authors. On Fri, Jan 27, 2017 at 5:07 PM, Casey Stella <ceste...@gmail.com> wrote: > I'd suggest making multiple PRs, one depending on the other one possibly, > to each be committed with the appropriate

Re: [MENTORS] ICLA for non-committer contributions

> request? > > On Fri, Jan 27, 2017 at 4:02 PM, Casey Stella <ceste...@gmail.com> wrote: > > > Yes, we should definitely not destroy authorship information. To my > > knowledge that hasn't happened yet and we should ensure it does not > happen > > in the future. &g

Re: [MENTORS] ICLA for non-committer contributions

rge a pull request, the authorship information is maintained. > Just make sure you don’t squash other people’s commits. > > -Taylor > > > On Jan 27, 2017, at 4:36 PM, Casey Stella <ceste...@gmail.com> wrote: > > > > Hi Mentors, > > > > I was wondering if yo

[MENTORS] ICLA for non-committer contributions

Hi Mentors, I was wondering if you could help me settle a question. What is the ASF's stance on ICLAs for non-committer contributions? Are they required? On the one hand, https://www.apache.org/dev/committers.html#applying-patches requires only that we attribute appropriately to form a legal

Re: [DISCUSS] Next Release (0.3.1) Content

I should add, you may be thinking something more rigorous and step-by-step. If so, you think you might be interested in volunteering to do a first draft as a PR that we can adjust? On Fri, Jan 27, 2017 at 9:01 AM, Casey Stella <ceste...@gmail.com> wrote: > So, I agree with the Upg

Re: [DISCUSS] Next Release (0.3.1) Content

.hortonworks.com/search.html?f==question= > search%2Fsearch=relevance=scp_if_ssh> > on the HCC boards, so I'm fairly confident that it is not an issue with my > local environment. > > Jon > > On Thu, Jan 26, 2017 at 6:18 PM Casey Stella <ceste...@gmail.com> wrote: > > I took the li

Re: [DISCUSS] How to do Sliding Windows in Profiler

and think this would be much better for users with either some > templates or other form of abstraction to simplify common patterns. > > Mike > > > On Wed, Jan 25, 2017 at 8:02 AM, Casey Stella <ceste...@gmail.com> wrote: > > > So, I think the key takeaway, at least f

Re: [DISCUSS] Next Release (0.3.1) Content

is METRON-650 because the reliance on the opensoc github repo might cause issues with the release being acceptable (see discussion by Mike Miklavcic with the mentors a couple weeks ago). On Thu, Jan 26, 2017 at 6:06 PM, Casey Stella <ceste...@gmail.com> wrote: > Hello Everyone! > > It'

[DISCUSS] Next Release (0.3.1) Content

Hello Everyone! It's been almost 2 months since the last major release and I think it might be time to do a minor release of 0.3.1. The purpose of this email is multifold: - to take stock of what we have already committed - figure out what (if anything) is missing to make a release that

Re: Build failing

; Feel free to add / correct anything in that ticket. > > Justin > > On Tue, Jan 24, 2017 at 8:09 AM, Casey Stella <ceste...@gmail.com> wrote: > > > One thing that I would caution though is that this is likely a heisenbug. > > The more logging I added earlier mad

Re: [DISCUSS] Gratuating to Apache Top Level Project

Ok, well, I'm sold then. :) On Wed, Jan 25, 2017 at 6:14 PM, Billie Rinaldi wrote: > As a mentor, I would also support Metron graduating. > > Billie > > On Mon, Jan 23, 2017 at 4:09 PM, James Sirota wrote: > > > I think the Apache Incubation was very

Re: [Discuss] Geo Enrichment with Stellar

Maybe off topic for this discussion, but what is the additional work? I *think* Justin's PR could be used to completely replace the non-stellar geo enrichment, but I'm curious if I missed something. On Wed, Jan 25, 2017 at 4:32 PM, Nick Allen wrote: > As part of Justin's

Re: [DISCUSS] Gratuating to Apache Top Level Project

On the whole, I think we're ready to graduate based on the maturity checklist posted. I think the issues with the 0.3.0 build were resolved as of METRON-600 (the new website with proper download links and appropriate indication that we are an incubated project). The only thing that gives has

Re: Build failing

project). On Tue, Jan 24, 2017 at 07:59 Casey Stella <ceste...@gmail.com> wrote: > Agreed to both counts. I was able to reproduce it locally, but not in an > IDE by the way. > On Tue, Jan 24, 2017 at 07:57 Justin Leet <justinjl...@gmail.com> wrote: > > I definitely agree that

Re: Build failing

ke to > propose that part of that ticket is adding logging. Right now, I'm > concerned we don't have enough info from the Travis builds to be able to > (easily) debug failure or reproduce locally. > > Justin > > On Mon, Jan 23, 2017 at 4:16 PM, Casey Stella <ceste...@gmail

Re: Build failing

One more thing, just for posterity here, it always freezes at 6 records written to HDFS. That's the reason I thought it was a flushing issue. On Mon, Jan 23, 2017 at 3:38 PM, Casey Stella <ceste...@gmail.com> wrote: > Ok, so now I'm concerned that this isn't a fluke. Here's an exc

Re: Build failing

On Mon, Jan 23, 2017 at 2:09 PM, Casey Stella <ceste...@gmail.com> wrote: > Yeah, I adjusted the timeout on the indexing integration tests as part of > https://github.com/apache/incubator-metron/pull/420 which I'll merge in > today. > > On Mon, Jan 23, 2017 at 2:01 PM, ze

Re: Build failing

Yeah, I adjusted the timeout on the indexing integration tests as part of https://github.com/apache/incubator-metron/pull/420 which I'll merge in today. On Mon, Jan 23, 2017 at 2:01 PM, zeo...@gmail.com wrote: > Okay, now we have back to back failures, and it looks like it may

Re: [VOTE] Release Process

+1 (binding) On Tue, Jan 17, 2017 at 11:17 PM, James Sirota wrote: > I made the revisions based on the discuss thread > > The document is available here: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66854770 > > And is also attached for reference to

Re: [DISCUSS] Release Process

Speaking as the current release manager, it's pretty obvious when you've done something wrong and as part of the vote in incubator general the collateral is pretty heavily scrutinized. On Mon, Jan 16, 2017 at 3:52 PM, Otto Fowler wrote: > If you are the person

Re: [DISCUSS] Release Process

Hey Matt, Github actually constructs the tarball for us using git archive (for every tag, for that matter). We don't explicitly push any tarball to github. The reason why we pull the tarball from github directly is that we ship a .gitattributes to define what gets put in the tarball. (we don't

Re: [DISCUSS] Moving GeoIP management away from MySQL

Yep, just what I was thinking On Mon, Jan 16, 2017 at 7:22 PM, Matt Foley <ma...@apache.org> wrote: > Sounds good! And use a versioning scheme via subdirectories in HDFS, so > you can revert back if you want. > > On 1/16/17, 4:11 PM, "Casey Stella" <ceste..

Re: [DISCUSS] Moving GeoIP management away from MySQL

pull > > > an > > > > > > instance > > > > > > > > into > > > > > > > > > >> each supervisor, so it makes a lot of sense for > > > > relatively small > &

Re: [DISCUSS] Turning off indexing writers feature discussion

recap, what I am +1 on is Nick's proposed syntax with the following > >> modifications: > >> 1. An explicit enabled field > >> 2. A default on for unspecified to match current semantics > > > > I'm +1 on all of this. > > > > On Sat, Jan 14, 2017 at 1

Re: [DISCUSS] Ambari Metron Configuration Management consequences and call to action

h > > >>> node) to force the data to be reloaded from Ambari-server. Best > > solution > > >>> is don’t cheat. > > >>> > > >>> Also, there may be circumstances under which the Ambari-agent will > > detect > > >>>

Re: [DISCUSS] Moving GeoIP management away from MySQL

his mean we would have to sync the enrichment data to > each > > > > Storm > > > > >>> supervisor? HDFS could (probably would) have a network hop too, > no? > > > > >>> > > > > >>> Fwiw - > >

Re: [DISCUSS] Moving GeoIP management away from MySQL

> > update their db files. Inside the code, we wrap the MapDB portions to > make > > it transparent to downstream code." > > > > Seems a bit more complex than "refresh the hbase table". Afaik, either > > approach would require some sort of translation bet

Re: [DISCUSS] Moving GeoIP management away from MySQL

I think that it's a sensible thing to use MapDB for the geo enrichment. Let me state my reasoning: - An HBase implementation would necessitate a HBase scan possibly hitting HDFS, which is expensive per-message. - An HBase implementation would necessitate a network hop and MapDB would

Re: [DISCUSS] Turning off indexing writers feature discussion

hahaha :) On Mon, Jan 16, 2017 at 10:49 AM, Nick Allen <n...@nickallen.org> wrote: > I don't quite support it for #1 and #2, but you absolutely sold me on #3. > Good sell. +1 > > > On Mon, Jan 16, 2017 at 10:46 AM, Casey Stella <ceste...@gmail.com> wrote: > &g

Re: [DISCUSS] Turning off indexing writers feature discussion

operty. I also like the idea of a path property for HDFS. > > > > -Kyle > > > > > On Jan 14, 2017, at 10:51 AM, Casey Stella <ceste...@gmail.com> wrote: > > > > > > I'm +1 on an explicit enabled property and a filter (or when) > property. I >

Re: [DISCUSS] Turning off indexing writers feature discussion

ault on? > > If I have a running system, then upgrade and a new indexer is added, would > that default to on and change the behavior of my system? Maybe this is a > fair trade-off that does not happen too frequently. > > On Sat, Jan 14, 2017 at 10:51 AM, Casey Stella

Re: [DISCUSS] Turning off indexing writers feature discussion

t; > > And to use ES with some overrides but not HDFS or solr it would look > like: > > > > { > >'elasticsearch': { > > 'enabled': 'true', > > 'index': 'foo', > > 'batchSize': 100 > > }, > >'hdfs': { > > 'e

Re: [DISCUSS] Turning off indexing writers feature discussion

, it still passes through to the index using defaults. In this new scheme it would not. This changes the default semantics for the system and I think it changes it for the worse. I would strongly prefer a on-by-default indexing config as we have now. On Fri, Jan 13, 2017 at 17:13 Casey Stella

Re: [DISCUSS] Turning off indexing writers feature discussion

t that, if we support but are > disabling a writer, that the platform inserts a default when(false) to be > explicit. > > Jon > > On Fri, Jan 13, 2017 at 11:59 AM Casey Stella <ceste...@gmail.com> wrote: > > > Let me noodle on this over the weekend. Your syntax is looking less >

Re: [DISCUSS] Ambari Metron Configuration Management consequences and call to action

bari. We’ll need to dig in to find out how flexible this is. > > I do think we should continue supporting non-Ambari use, and if we put all > configs in ZK, that gets way easier to do in a simple and consistent way. > (Propagation problem solved). More thoughts after I have ‘em :-) >

Re: [DISCUSS] Turning off indexing writers feature discussion

gt; a1#writer-specific-case-without-filters>Writer-specific > > > case without filters > > > > { > > 'elasticsearch': { > > 'index': 'foo', > > 'batchSize': 1 > > }, > > 'hdfs': { > > 'index': 'foo', > > 'batchSize': 100 > >

Re: [DISCUSS] Turning off indexing writers feature discussion

te step, before the process which actually > writes out to each platform. It may also make sense to have a concept of a > routing key built up by earlier enrichment to allow shuffle control in > storm, rather than a full stellar statement for routing, to avoid overhead. > > Simon >

Re: [DISCUSS] Turning off indexing writers feature discussion

w? I think it's > global control of all index types + overrides on a per-type basis. Fwiw, > I'm totally for that, but I want to make sure I'm not imposing my > pre-concieved notions on your consensus-driven ones. > > -D > > On Fri, Jan 13, 2017 at 10:44 AM, Casey Stella &l

Re: [DISCUSS] Turning off indexing writers feature discussion

ly I like my original example as there are fewer sub-structures, > like 'writerConfig', which makes the whole thing simpler and easier to > grok. But maybe others will think your proposal is just as easy to grok. > > > > On Fri, Jan 13, 2017 at 10:01 AM, Casey Stella <ceste...@gmai

Re: [DISCUSS] Turning off indexing writers feature discussion

at 10:08 AM, Otto Fowler <ottobackwa...@gmail.com> > wrote: > > > How does it look with 50 whens > > > > > > On January 13, 2017 at 10:02:02, Casey Stella (ceste...@gmail.com) > wrote: > > > > Ok, so here's what I'm thinking based on the discussion: > &

Re: [DISCUSS] Ambari Metron Configuration Management consequences and call to action

i > had also designed for or not. > > On Fri, Jan 13, 2017 at 10:14 AM, Casey Stella <ceste...@gmail.com> wrote: > > > Polling the Ambari server via REST (or their API if they have one), would > > entail all workers hitting one server and create a single point of >

Re: [DISCUSS] Ambari Metron Configuration Management consequences and call to action

ionally) reason > > > > I think we'd need to do some detailed design around how to handle what we > > expect to be dynamic configs, but the main principle should (imo) be to > > always know who and why and make sure that Ambari is aware and is the > > static backing store for Zooke

Re: [DISCUSS] Turning off indexing writers feature discussion

ists(field2) or" , "exists(field3)" ] Thinks that's a bad idea? Casey On Fri, Jan 13, 2017 at 10:08 AM, Otto Fowler <ottobackwa...@gmail.com> wrote: > How does it look with 50 whens > > > On January 13, 2017 at 10:02:02, Casey Stella (

Re: [DISCUSS] Turning off indexing writers feature discussion

Jan 13, 2017 at 9:44 AM, Carolyn Duby <cd...@hortonworks.com> wrote: > For larger installations you need to control what is indexed so you don’t > end up with a nasty elastic search situation and so you can mine the data > later for reports and training ml models. > &g

Re: [DISCUSS] Turning off indexing writers feature discussion

: { > "when": "true", > "batchSize": 1000, > "index": "squid" > } > } > > > > > > > > > On Fri, Jan 13, 2017 at 9:10 AM, Casey Stella <ceste...@gmail.com> wrote: > > > Ye

Re: [DISCUSS] Ambari Metron Configuration Management consequences and call to action

t; That's exactly correct, Casey. Basically, an expansion of what we're > currently doing with global.json, enrichment.properties and > elasticsearch.properties. > > -D... > > > On Fri, Jan 13, 2017 at 9:12 AM, Casey Stella <ceste...@gmail.com> wrote: > > > I woul

Re: [DISCUSS] Ambari Metron Configuration Management consequences and call to action

path > would be to do stuff (human or machine) via Ambari. > > -D... > > > On Fri, Jan 13, 2017 at 9:01 AM, Casey Stella <ceste...@gmail.com> wrote: > > > Just piling on in support for Ambari. I really, really don't like > > reinventing wheels, especially ha

Re: [DISCUSS] Ambari Metron Configuration Management consequences and call to action

> Thanks for starting this thread. I believe you are correct in your > > assessment of the 4 options for updating configs in Metron. When using > more > > than one of these options we can get into a split-brain scenario. A basic > > example is updating the gl

Re: [DISCUSS] Turning off indexing writers feature discussion

gt; > -Kyle > > On Thu, Jan 12, 2017 at 6:51 PM, Matt Foley <ma...@apache.org> wrote: > > > Ah, I see. If overriding the default index name allows using the same > > name for multiple sensors, then the goal can be achieved. > > Thanks, > > --Matt > > > &g

Re: [DISCUSS] Ambari Metron Configuration Management consequences and call to action

than one of these options we can get into a split-brain scenario. A basic > > example is updating the global config on disk and using the > > zk_load_configs.sh. Later, if a user decides to restart Ambari, the > cached > > version stored by Ambari (it's in the MySQL or other

Re: [DISCUSS] Turning off indexing writers feature discussion

ted and available > with common field names, why doesn’t it make sense to index the messages > together, for CEP querying? I think Splunk has illustrated this model. > > On 1/12/17, 3:00 PM, "Casey Stella" <ceste...@gmail.com> wrote: > > yeah, I mean, honestly

Re: [PROPOSAL] up-to-date versioned documentation

7 at 6:19 PM, Casey Stella <ceste...@gmail.com> wrote: > So, I do think this would be better than what we currently do. I like a > few things in particular: > >- I don't like the wiki one bit. >- We have a LOT of documentation in the README.md's and it's sometimes >

Re: [PROPOSAL] up-to-date versioned documentation

So, I do think this would be better than what we currently do. I like a few things in particular: - I don't like the wiki one bit. - We have a LOT of documentation in the README.md's and it's sometimes poorly organized - I like a documentation preprocessing pipeline to be present.

Re: [DISCUSS] Turning off indexing writers feature discussion

ions over the whole mass of data rather than just vertical > slices of it. > > On 1/12/17, 2:15 PM, "Casey Stella" <ceste...@gmail.com> wrote: > > Hey Matt, > > Thanks for the comment! > 1. At the moment, we only have one index name, the default of whi

Re: [DISCUSS] Dev Guide and Committer Review Guide additions?

Regarding 3, Matt, I just started a dev list discussion about configs and the various components that manage them and how they interact. Hopefully we end up in a coherent approach, but in the lead of that, I'd say yes, valid need for such an architecture. Please chime in on that thread or even

<    1   2   3   4   >