[GitHub] incubator-metron issue #416: METRON-656: Make Stellar 'in' closer to functio...

Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/416 @jjmeyer0 Thanks so much for the comments! 1. Yeah, definitely, but this is a language feature rather than a function that we're testing, so I put it in the `StellarTest` 2. I

Re: Kibana Tile Map not available

Hi Dima - I forgot to email earlier, but the tile maps are working fine for me using Quick Dev. Still having this problem? On Wed, Jan 11, 2017 at 6:02 PM, Dima Kovalyov wrote: > Hello, > > I have attached screenshot of how my Tile Map looks like right now >

[GitHub] incubator-metron issue #416: METRON-656: Make Stellar 'in' closer to functio...

Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/416 +1 (non-binding). I'll create a Jira based on the last comments between @cestella and me. --- If your project is set up for it, you can reply to this email and have your reply appear on

Re: Kibana Tile Map not available

Hello Nick, Thanks for the reply! Yeah, I still have it. What CDN does you Kibana is using? Are you able to access URLs I listed in my original email? - Dima On 01/13/2017 10:23 PM, Nick Allen wrote: > Hi Dima - I forgot to email earlier, but the tile maps are working fine for > me using Quick

[GitHub] incubator-metron pull request #416: METRON-656: Make Stellar 'in' closer to ...

Github user jjmeyer0 commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/416#discussion_r96071886 --- Diff: metron-platform/metron-common/README.md --- @@ -39,11 +40,17 @@ The following keywords need to be single quote escaped in order to be

[GitHub] incubator-metron issue #416: METRON-656: Make Stellar 'in' closer to functio...

Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/416 I think this is a great feature. I do have a few suggestions/questions: 1. Does it make sense to start moving away from `StellarTest` and break out into specific test classes (eg.

[GitHub] incubator-metron issue #316: METRON-503: Metron REST API

Github user merrimanr commented on the issue: https://github.com/apache/incubator-metron/pull/316 That use case makes sense. I will leave it in. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] incubator-metron issue #394: METRON-623: Management UI

Github user merrimanr commented on the issue: https://github.com/apache/incubator-metron/pull/394 I'm going to close this until METRON-503 gets merged. Sorry for the distraction. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] incubator-metron issue #416: METRON-656: Make Stellar 'in' closer to functio...

Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/416 I found one additional corner case that may need to be addressed. It looks the like expression, `null in [ null, 'something' ]`, returns false, but should return true. --- If your

[GitHub] incubator-metron issue #416: METRON-656: Make Stellar 'in' closer to functio...

Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/416 Disregard number 2. It was an oversight on my part. Sorry about that. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

Re: Kibana Tile Map not available

No, I cannot reach those URLs. On Fri, Jan 13, 2017 at 4:34 PM, Dima Kovalyov wrote: > Hello Nick, > > Thanks for the reply! Yeah, I still have it. What CDN does you Kibana is > using? > Are you able to access URLs I listed in my original email? > > - Dima > > On

[GitHub] incubator-metron pull request #394: METRON-623: Management UI

Github user merrimanr closed the pull request at: https://github.com/apache/incubator-metron/pull/394 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

Re: [DISCUSS] Turning off indexing writers feature discussion

I like the suggestions you made, Nick. The only thing I would add is that it's also nice to see an explicit when(false), as people newer to the platform may not know where to expect configs for the different writers. Being able to do it either way, which I think is already assumed in your model,

[GitHub] incubator-metron issue #416: METRON-656: Make Stellar 'in' closer to functio...

Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/416 @jjmeyer0 Oh good one! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] incubator-metron pull request #416: METRON-656: Make Stellar 'in' closer to ...

Github user cestella commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/416#discussion_r96073648 --- Diff: metron-platform/metron-common/README.md --- @@ -39,11 +40,17 @@ The following keywords need to be single quote escaped in order to be

[GitHub] incubator-metron issue #416: METRON-656: Make Stellar 'in' closer to functio...

Github user mmiklavc commented on the issue: https://github.com/apache/incubator-metron/pull/416 I like the semantics here. One small comment on the tests, +1 pending that adjustment and Travis. --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] incubator-metron pull request #416: METRON-656: Make Stellar 'in' closer to ...

Github user cestella commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/416#discussion_r96046421 --- Diff: metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarTest.java --- @@ -418,6 +417,33 @@ public void

Re: [DISCUSS] Turning off indexing writers feature discussion

I will add also that instead of global overrides, like index, we should use configuration key names that are more appropriate to the output. For example, does 'index' really make sense for HDFS? Or would 'path' be more appropriate? { 'elasticsearch': { 'index': 'foo',

Re: [DISCUSS] Turning off indexing writers feature discussion

Let me noodle on this over the weekend. Your syntax is looking less onerous to me and I like the following statement from Otto: "In the end, each write destination ‘type’ will need it’s own configuration. This is an extension point." I may come around to your way of thinking. On Fri, Jan 13,

[GitHub] incubator-metron pull request #416: METRON-656: Make Stellar 'in' closer to ...

GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/416 METRON-656: Make Stellar 'in' closer to functioning like python We have an `in` operator in stellar, but it could be much better. This should bring it at parity with the `in` operator in

Re: [DISCUSS] Ambari Metron Configuration Management consequences and call to action

Gosh I hate being two hours behind you guys! :-) I’ll go back through the thread and collect open questions, but wanted to put a word in about Zookeeper integration. I had been about to ask what the benefits of using ZK are, and I’ve now heard two features: - A logical “single place” that is

Re: [DISCUSS] Turning off indexing writers feature discussion

Hmm, I'm not sure I agree that in most cases users would accept the default batch size, especially in sizeable environments. In search tiers like ES it is very important, and should be tuned to the specific data that you're sending because it depends on the number of bytes, not necessarily number

Re: [DISCUSS] Turning off indexing writers feature discussion

I think Simon has a very valid suggestion. Additionally, I have a two questions. For the following config: { "index" : "foo" ,"batchSize" : 100 } Are now all logs going to the same index? I read this as a writer-specific override of the sensor-specific defaults to use an index name of foo*

[GitHub] incubator-metron pull request #416: METRON-656: Make Stellar 'in' closer to ...

Github user mmiklavc commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/416#discussion_r96046647 --- Diff: metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarTest.java --- @@ -418,6 +417,33 @@ public void

Re: [DISCUSS] Turning off indexing writers feature discussion

In the end, each write destination ‘type’ will need it’s own configuration. This is an extension point. { HDFS:{ outputAdapters:[ {name: avro, settings:{ avro stuff…. when:{ }, { name: sequence file, ….. or some such. On January 13, 2017 at 11:51:15, Nick Allen (n...@nickallen.org) wrote: I

[GitHub] incubator-metron pull request #404: METRON-624: Updated Comparison/Equality ...

Github user asfgit closed the pull request at: https://github.com/apache/incubator-metron/pull/404 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

Re: [DISCUSS] Turning off indexing writers feature discussion

> > Nick's concerns about my suggestion were that it was overly complex and > hard to grok and that we could dispense with backwards compatibility and > make people do a bit more work on the default case for the benefits of a > simpler advanced case. (Nick, make sure I don't misstate your

[GitHub] incubator-metron pull request #416: METRON-656: Make Stellar 'in' closer to ...

Github user mmiklavc commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/416#discussion_r96045537 --- Diff: metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarTest.java --- @@ -418,6 +417,33 @@ public void

Re: [DISCUSS] Ambari Metron Configuration Management consequences and call to action

Ok, I'll try to give the historical context for the parser and enrichment and indexing topologies since I was the one who made a lot of the mistakes. ;) In the beginning there was only configuration changes at topology start and this was done via those flux properties. As the configs got more

Re: [DISCUSS] Turning off indexing writers feature discussion

One thing that I really like about Nick's suggestion is that it allows writer-specific configs in a clear and simple way. It is more complex for the default case (all writers write to indices named the same thing with a fixed batch size), which I do not like, but maybe it's worth the compromise

Re: [DISCUSS] Hosting Kraken maven artifacts in incubator-metron git repo

Has anyone used git's sub-modules before? My understanding is you just point to an external repository. So *technically* I do not think the code would be hosted in the main repo. Even if that was allowed, I have concerns about how inactive the repo is. Could we fork this, make our changes, and

Re: [DISCUSS] Hosting Kraken maven artifacts in incubator-metron git repo

Perhaps it would be more appropriate to put it under https://dist.apache.org/repos/dist/release/incubator/metron/ , perhaps as https://dist.apache.org/repos/dist/release/incubator/metron/mvn-repo ? We should not host anything with a license that isn’t compatible with inclusion in an Apache

Re: [DISCUSS] Ambari Metron Configuration Management consequences and call to action

+1 I strongly agree with Jon's view. Requiring a restart would be a big step backwards. I think the power of the platform is that the user can act on live streaming data in a quick, iterative fashion. Adding enrichments, creating triage rules, adjusting profiles are all operational activities

Re: [DISCUSS] Turning off indexing writers feature discussion

Thanks Casey! I think I had the right of it, but wanted to make sure. I'm +1 on defaults in global with overrides in sensor-specific. At least in the first iteration. I (like Otto) suspect we'll have a few go-arounds on this. -D... On Fri, Jan 13, 2017 at 11:09 AM, Otto Fowler

Re: [DISCUSS] Ambari Metron Configuration Management consequences and call to action

That's good feedback, Jon. I think that puts us at: - Expand ambari to manage the remaining sensor-specific configs - Refactor the push calls to zookeeper (in ConfigurationUtils, I think) to push to ambari and take an Ambari user/pw and (optionally) reason - We shall retain current

Re: [DISCUSS] Turning off indexing writers feature discussion

Yeah, I tend to like the first option too. Any opposition to that from anyone? The points brought up are good ones and I think that it may be worth a broader discussion of the requirements of indexing in a separate dev list thread. Maybe a list of desires with coherent use-cases justifying them

Re: [DISCUSS] Ambari Metron Configuration Management consequences and call to action

I would suggest not having Ambari replace zookeeper. I think the proposal is to have Ambari replace the editable store (like the JSON files on disk). Zookeeper woudl be the source of truth for the running topologies and ambari would be sync'd to it. Correct if I misspeak, dave or matt. Casey

Re: [DISCUSS] Turning off indexing writers feature discussion

OH that's a good idea! On Fri, Jan 13, 2017 at 9:39 AM, Nick Allen wrote: > I like the "Index Filtering" option based on the flexibility that it > provides. Should each output (HDFS, ES, etc) have its own configuration > settings? For example, aren't things like batching

Re: [PROPOSAL] up-to-date versioned documentation

I think something that does what you have laid out here, no matter the implementation details would be ideal On January 12, 2017 at 18:05:24, Matt Foley (ma...@apache.org) wrote: We currently have three forms of documentation, with the following advantages and disadvantages: || Docs || Pro ||

Re: [DISCUSS] Turning off indexing writers feature discussion

I prefer option1 with stellar, although I’m concerned that in a real world scenario the amount of filters and rules might be large, and some thought about the structure of the rule expressions for maintainability etc will need to be considered. On January 12, 2017 at 15:52:03, Casey Stella

Re: [DISCUSS] Turning off indexing writers feature discussion

For larger installations you need to control what is indexed so you don’t end up with a nasty elastic search situation and so you can mine the data later for reports and training ml models. Thanks Carolyn On 1/13/17, 9:40 AM, "Casey Stella" wrote: >OH that's a good

Re: [DISCUSS] Ambari Metron Configuration Management consequences and call to action

The only tooling I'm aware of that Ambari isn't already using is the Stellar stuff, is there more? Regardless, I'd always push from Ambari to zookeeper and let other tooling talk to Ambari (Casey's first bullet). The only wrinkle is we have to decide if we want to support manual installation.

Re: [DISCUSS] Ambari Metron Configuration Management consequences and call to action

That's exactly correct, Casey. Basically, an expansion of what we're currently doing with global.json, enrichment.properties and elasticsearch.properties. -D... On Fri, Jan 13, 2017 at 9:12 AM, Casey Stella wrote: > I would suggest not having Ambari replace zookeeper. I

Re: [DISCUSS] Ambari Metron Configuration Management consequences and call to action

So, the reason to have the push operations push to ambari and then have ambari sync to zookeeper (btw: is this possible, do we have a hook like that in ambari?) is to make sure that users can specify a comment about what changed, correct? If we pushed to zookeeper and had ambari listen (not sure

Re: [DISCUSS] Ambari Metron Configuration Management consequences and call to action

In the main yes- I've made some changes: - Expand ambari to manage the remaining sensor-specific configs - Refactor the push calls to zookeeper (in ConfigurationUtils, I think) to push to ambari and take an Ambari user/pw and (optionally) reason - (Ambari can push to zookeeper, but it

[GitHub] incubator-metron issue #415: METRON-652: Extract indexing config from enrich...

Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/415 @JonZeolla Yes it is! Whoops, my bad. I guess my JIRA search-fu isn't as good as I thought. --- If your project is set up for it, you can reply to this email and have your reply appear

Re: [DISCUSS] Ambari Metron Configuration Management consequences and call to action

Let me ask a stupid question. What does Zookeeper do for us that Ambari cannot? Why keep Zookeeper in the mix? On Fri, Jan 13, 2017 at 9:28 AM, David Lyle wrote: > In the main yes- I've made some changes: > > - Expand ambari to manage the remaining sensor-specific

Re: [DISCUSS] Turning off indexing writers feature discussion

Darn it Nick, you beat me to the punch. =) YES, please. I think I discussed this a while back in my ES tuning conversations, but that's _super_ important. I have this documented here under Elasticsearch > On

Re: [DISCUSS] Ambari Metron Configuration Management consequences and call to action

Just piling on in support for Ambari. I really, really don't like reinventing wheels, especially hard ones. I guess my questions now are mainly around technical feasibility. Seems to me that we can either: - retrofit the tooling that currently manages configs to use the Ambari API's as

Re: [DISCUSS] Ambari Metron Configuration Management consequences and call to action

So, basically, your proposed changes, broken into tangible gobbets of work: - Expand ambari to manage the remaining sensor-specific configs - Refactor the push calls to zookeeper (in ConfigurationUtils, I think) to push to ambari and take a reason - Question remains about whether

Re: [DISCUSS] Turning off indexing writers feature discussion

I like the "Index Filtering" option based on the flexibility that it provides. Should each output (HDFS, ES, etc) have its own configuration settings? For example, aren't things like batching handled separately for HDFS versus Elasticsearch? Something along the lines of... { "hdfs" : {

Re: [DISCUSS] Turning off indexing writers feature discussion

One thing that I thought of that I very strenuous do not like in Nick's proposal is that if a writer config is not specified then it is turned off (I think; if I misunderstood let me know). In the situation where we have a new sensor, right now if there are no index config and no enrichment

Re: [DISCUSS] Ambari Metron Configuration Management consequences and call to action

I'm in complete agreement with all the points Matt made. I think the way forward should be to expose ALL user-modifiable configs via Ambari and let Ambari actively manage them. We should keep the command line tools as the backend and Ambari should continue to leverage them. This will allow manual

Re: [DISCUSS] Turning off indexing writers feature discussion

Ok, so here's what I'm thinking based on the discussion: - Keeping the configs that we have now (batchSize and index) as defaults for the unspecified writer-specific case - Adding the config Nick suggested *Base Case*: { } - all writers write all messages - index named the same

Re: [PROPOSAL] up-to-date versioned documentation

+1 I think it is sorely needed. If we can come up with a really slick solution like Spark, then great. I am also not against a half-baked solution that can later evolve into something else. For example, create an index README.md that links together all the existing READMEs and run Pandoc on it.

Re: [DISCUSS] Ambari Metron Configuration Management consequences and call to action

Polling the Ambari server via REST (or their API if they have one), would entail all workers hitting one server and create a single point of failure (the ambari server is what serves up REST). Zookeeper's intent is to not have a single point of failure like this and (one of its main) use-cases is

Re: [DISCUSS] Turning off indexing writers feature discussion

Like most things, we are best off to try something and iterate. I just think we should be aware from the beginning ( have tests etc ) of how it works when there are many filters. On January 13, 2017 at 10:11:47, Casey Stella (ceste...@gmail.com) wrote: I imagined one stellar statement and if

Re: [DISCUSS] Turning off indexing writers feature discussion

We also need to account for the complexity of the statements On January 13, 2017 at 10:27:51, Otto Fowler (ottobackwa...@gmail.com) wrote: Like most things, we are best off to try something and iterate. I just think we should be aware from the beginning ( have tests etc ) of how it works when

Re: [DISCUSS] Turning off indexing writers feature discussion

I imagined one stellar statement and if you wanted an "or" in there, you could put it there. I was also planning on doing the JSOn trick of accepting either a string or list of strings to let you do multiline. e.g. "when" : [ "exists(field1) or" , "exists(field2) or" ,

Re: [DISCUSS] Ambari Metron Configuration Management consequences and call to action

ZooKeeper is more efficient if you want to maintain an update to the topologies without requiring a restart. Is this useful going forward? I think it is for development but production environments you would generally only be updating during a maintenance window so requiring a restart is not

Re: [DISCUSS] Ambari Metron Configuration Management consequences and call to action

No, it was good to bring up, Nick. I might have it wrong re: Ambari. Casey On Fri, Jan 13, 2017 at 10:27 AM, Nick Allen wrote: > That makes sense. I wasn't sure based on Matt's original > suggestion/description of Ambari, whether that was something that Ambari > had also

Re: [DISCUSS] Turning off indexing writers feature discussion

Are you saying we support all of these variants? I realize you are trying to have some backwards compatibility, but this also makes it harder for a user to grok (for me at least). Personally I like my original example as there are fewer sub-structures, like 'writerConfig', which makes the whole

Re: [DISCUSS] Ambari Metron Configuration Management consequences and call to action

The driver for using Zookeeper is that it is asynchronous and accepts callbacks. Ambari would need to have that capability, otherwise we have to poll which is a deal breaker in my opinion. On Fri, Jan 13, 2017 at 9:28 AM, Casey Stella wrote: > No, it was good to bring up,

Re: [DISCUSS] Turning off indexing writers feature discussion

Yep, that makes sense, Casey. I understand multiline is still just the same when statement. I was more responding to Otto's concern about dealing with 50 whens. In regards to multiline, I don't know if adding that is worth the potential confusion. I prefer very simple configs that are stupid

Re: [DISCUSS] Turning off indexing writers feature discussion

I am suggesting that, yes. The configs are essentially the same as yours, except there is an override specified at the top level. Without that, in order to specify both HDFS and ES have batch sizes of 100, you have to explicitly configure each. It's less that I'm trying to have backwards

Re: [DISCUSS] Ambari Metron Configuration Management consequences and call to action

Makes sense, Dave. I am totally clear on the proposal. I just wanted to ask the stupid question to bring the conversation full circle, leave no stone unturned, insert favorite idiom here. On Fri, Jan 13, 2017 at 10:46 AM, David Lyle wrote: > To be clear- NOBODY is

Re: [DISCUSS] Turning off indexing writers feature discussion

This is an excellent point On January 13, 2017 at 10:54:07, Simon Elliston Ball (si...@simonellistonball.com) wrote: Some thing else to consider here is the possibility of multiple indices within a given target technology. For example, if I’m indexing data from a given sensor into, say

Re: [DISCUSS] Turning off indexing writers feature discussion

I was thinking there would only be one 'when' for each output. So if we have Elasticsearch and HDFS, you would have only 2 'when's. Each when statement could be as simple or complex as you need. On Fri, Jan 13, 2017 at 10:08 AM, Otto Fowler wrote: > How does it look

Re: [DISCUSS] Turning off indexing writers feature discussion

Nick, Yep, that's the example I showed. I'm just suggesting that that when use the multiline JSON trick here . A single "when" statement with a couple "or"'s So: "when" : [ "exists(field1) or" , "exists(field2) or" ,

Re: [DISCUSS] Ambari Metron Configuration Management consequences and call to action

To be clear- NOBODY is suggesting replacing Zookeeper with Ambari. So, as a bit of a reset- here's what's being proposed: - Expand ambari to manage the remaining sensor-specific configs - Refactor the push calls to zookeeper (in ConfigurationUtils, I think) to push to ambari and take an

Re: [DISCUSS] Turning off indexing writers feature discussion

Some thing else to consider here is the possibility of multiple indices within a given target technology. For example, if I’m indexing data from a given sensor into, say solr, I may want it filtered differently into two different indices. This would enable me to create different ‘views’ which

Re: [DISCUSS] Turning off indexing writers feature discussion

Casey, Can you give me a level set of what your thinking is now? I think it's global control of all index types + overrides on a per-type basis. Fwiw, I'm totally for that, but I want to make sure I'm not imposing my pre-concieved notions on your consensus-driven ones. -D On Fri, Jan 13,

Re: [DISCUSS] Ambari Metron Configuration Management consequences and call to action

Right, good conversation to bring up for sure. Just to comment on production generally only being updated during maintenance windows - I can tell you that my plans are to make my dev, test, and prod Metron a very dynamic and frequently changing environment which will have coordinated but frequent

Re: [DISCUSS] Turning off indexing writers feature discussion

Dave, For the benefit of posterity and people who might not be as deeply entangled in the system as we have been, I'll recap things and hopefully answer your question in the process. Historically the index configuration is split between the enrichment configs and the global configs. - The

Re: [DISCUSS] Turning off indexing writers feature discussion

Simon, Great thought. I had considered it, but didn't want to bite off all that as part of a PR. I thought baby-steps for the moment woudl be best. Perhaps this deserves its own JIRA and discussion? On Fri, Jan 13, 2017 at 10:53 AM, Simon Elliston Ball < si...@simonellistonball.com> wrote: >

Re: [DISCUSS] Hosting Kraken maven artifacts in incubator-metron git repo

No, we can't host artifacts in a git repo, or on a website. It would be like distributing a release that hasn't been voted upon. Regarding message threading, in Gmail adding a [tag] to the subject does not create a new thread. So the change is not visible in my mailbox unless the rest of the