[GitHub] incubator-metron issue #438: METRON-686 Record Rule Set that Fired During Th...

2017-02-27 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/438 Ok, I'm +1 on this pending successful travis. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] incubator-metron issue #438: METRON-686 Record Rule Set that Fired During Th...

2017-02-24 Thread nickwallen
Github user nickwallen commented on the issue: https://github.com/apache/incubator-metron/pull/438 Ok, I flattened the threat score with the latest commit. The PR description has also been updated to match current state. Please take a gander. --- If your project is set up for it,

[GitHub] incubator-metron issue #438: METRON-686 Record Rule Set that Fired During Th...

2017-02-24 Thread nickwallen
Github user nickwallen commented on the issue: https://github.com/apache/incubator-metron/pull/438 Sounds good, looks like there are more votes for option 2. I'll go that route. Gracias! --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] incubator-metron issue #438: METRON-686 Record Rule Set that Fired During Th...

2017-02-22 Thread ottobackwards
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/438 I think what we are saying is 1. I think that is the best way to move forward. Just make sure that this commit isn't merged into an RC for 0.3.1 --- If your project is set up

[GitHub] incubator-metron issue #438: METRON-686 Record Rule Set that Fired During Th...

2017-02-22 Thread nickwallen
Github user nickwallen commented on the issue: https://github.com/apache/incubator-metron/pull/438 @ottobackwards @cestella @justinleet - Let me know what you guys think on my previous comment about how to move forward on this PR. I think we have a rough outline of the

[GitHub] incubator-metron issue #438: METRON-686 Record Rule Set that Fired During Th...

2017-02-22 Thread nickwallen
Github user nickwallen commented on the issue: https://github.com/apache/incubator-metron/pull/438 So assuming we put aside the issue of flattening the data per METRON-735, what should be the go-forward for this PR? I outlined two options above. Please share your opinions on those

[GitHub] incubator-metron issue #438: METRON-686 Record Rule Set that Fired During Th...

2017-02-22 Thread ottobackwards
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/438 I commented on 735 so it wins. #settled --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] incubator-metron issue #438: METRON-686 Record Rule Set that Fired During Th...

2017-02-22 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/438 rgr, deleting 736 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] incubator-metron issue #438: METRON-686 Record Rule Set that Fired During Th...

2017-02-22 Thread ottobackwards
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/438 let me know which one to comment on! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] incubator-metron issue #438: METRON-686 Record Rule Set that Fired During Th...

2017-02-22 Thread nickwallen
Github user nickwallen commented on the issue: https://github.com/apache/incubator-metron/pull/438 Ha... oops. Yep, we agree. Add your commentary to METRON-735. You're suggesting a specific implementation, which is good. --- If your project is set up for it, you can reply to this

[GitHub] incubator-metron issue #438: METRON-686 Record Rule Set that Fired During Th...

2017-02-22 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/438 Ok, it appears that I tramped you, @nickwallen Our JIRAs seem to be suggesting the same thing, am I right? If you say so, I'll delete mine. --- If your project is set up for it, you can

[GitHub] incubator-metron issue #438: METRON-686 Record Rule Set that Fired During Th...

2017-02-22 Thread nickwallen
Github user nickwallen commented on the issue: https://github.com/apache/incubator-metron/pull/438 Created [METRON-735](https://issues.apache.org/jira/browse/METRON-735), in case this idea gains wider support from the community. --- If your project is set up for it, you can reply to

[GitHub] incubator-metron issue #438: METRON-686 Record Rule Set that Fired During Th...

2017-02-22 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/438 I'm of the opinion that the flattening should be writer-specific and that should be a function of the writer config with the default to be specified by the writer implementation. This

[GitHub] incubator-metron issue #438: METRON-686 Record Rule Set that Fired During Th...

2017-02-22 Thread nickwallen
Github user nickwallen commented on the issue: https://github.com/apache/incubator-metron/pull/438 @ottobackwards Yep, good call out. We should address the JSON mapper as you described also. --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] incubator-metron issue #438: METRON-686 Record Rule Set that Fired During Th...

2017-02-22 Thread ottobackwards
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/438 That makes sense, thanks for the explanation. I am +1 for having SOLR specific flattening, although documenting well needs to be a priority. Also, if we do this, please log a

[GitHub] incubator-metron issue #438: METRON-686 Record Rule Set that Fired During Th...

2017-02-22 Thread nickwallen
Github user nickwallen commented on the issue: https://github.com/apache/incubator-metron/pull/438 > if you look at the JSONMapParser, Casey and I implemented a flattener... @ottobackwards Thanks for pointing that out. If we end up going with option 2, I will try and reuse

[GitHub] incubator-metron issue #438: METRON-686 Record Rule Set that Fired During Th...

2017-02-22 Thread ottobackwards
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/438 Nick - if you look at the JSONMapParser, Casey and I implemented a flattener that you would use for this. So, all of our json stuff from that _was_ already flattened. I am more

[GitHub] incubator-metron issue #438: METRON-686 Record Rule Set that Fired During Th...

2017-02-22 Thread nickwallen
Github user nickwallen commented on the issue: https://github.com/apache/incubator-metron/pull/438 I really could use some opinions on the go-forward here. I see two options. (1) Assume that any indexer that cannot handle complex types is currently broken. * Since

[GitHub] incubator-metron issue #438: METRON-686 Record Rule Set that Fired During Th...

2017-02-21 Thread nickwallen
Github user nickwallen commented on the issue: https://github.com/apache/incubator-metron/pull/438 Bump? Haven't received feedback on my last comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] incubator-metron issue #438: METRON-686 Record Rule Set that Fired During Th...

2017-02-12 Thread james-sirota
Github user james-sirota commented on the issue: https://github.com/apache/incubator-metron/pull/438 I think a better approach is to bake in an enforcement layer within Metron to only allow flat maps (key-value pairs where the value cannot be a complex object). You would enforce

Re: [GitHub] incubator-metron issue #438: METRON-686 Record Rule Set that Fired During Th...

2017-02-10 Thread Otto Fowler
Could you re-use the jsonmap parser stuff? On February 10, 2017 at 13:58:54, cestella (g...@git.apache.org) wrote: Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/438 It occurs to me that one thing we might consider going forward is to flatten the

[GitHub] incubator-metron issue #438: METRON-686 Record Rule Set that Fired During Th...

2017-02-10 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/438 It occurs to me that one thing we might consider going forward is to flatten the map upon writing to the index. We could configure the message flattening as a capability on a per-writer

[GitHub] incubator-metron issue #438: METRON-686 Record Rule Set that Fired During Th...

2017-02-10 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/438 I think the only thing that's concerning me here is that we have avoided complex types in values in the past. I believe it had to do with ES performance and the fact that not all indexes

[GitHub] incubator-metron issue #438: METRON-686 Record Rule Set that Fired During Th...

2017-02-10 Thread justinleet
Github user justinleet commented on the issue: https://github.com/apache/incubator-metron/pull/438 +1, Everything looks good. As noted above, both the issue I had and the review comment were addressed by adding new Jiras and aren't needed here. Thanks a lot, Nick. This is

[GitHub] incubator-metron issue #438: METRON-686 Record Rule Set that Fired During Th...

2017-02-09 Thread justinleet
Github user justinleet commented on the issue: https://github.com/apache/incubator-metron/pull/438 After talking with Casey, it's an issue with StellarShell, not this PR. I'll make a ticket and get it done. So feel free to ignore this issue, @nickwallen --- If your project is

[GitHub] incubator-metron issue #438: METRON-686 Record Rule Set that Fired During Th...

2017-02-09 Thread justinleet
Github user justinleet commented on the issue: https://github.com/apache/incubator-metron/pull/438 Seems like THREAT_TRIAGE_REMOVE just behaves really badly ``` [Stellar]>>> conf := THREAT_TRIAGE_ADD(conf, [triage]) [Stellar]>>> conf := THREAT_TRIAGE_REMOVE(conf,

[GitHub] incubator-metron issue #438: METRON-686 Record Rule Set that Fired During Th...

2017-02-09 Thread justinleet
Github user justinleet commented on the issue: https://github.com/apache/incubator-metron/pull/438 I saw some odd behavior I think is unrelated to this PR itself while testing. I tried to remove the threat triage rule, messed up, fixed it, and then borked my conf variable.