[GitHub] incubator-metron pull request #543: METRON-857 Ability to completely build p...

2017-04-21 Thread ottobackwards
GitHub user ottobackwards opened a pull request: https://github.com/apache/incubator-metron/pull/543 METRON-857 Ability to completely build project in single Docker Image ## Contributor Comments Before ambari integration, we had the ability to use the ansible docker image t

Re: Ansible Docker fails to build latest

2017-04-21 Thread Otto Fowler
OK, the issue is the old old maven that is in the ansible image. Changing it to 3.3.9 resolved. On April 21, 2017 at 17:21:02, Otto Fowler (ottobackwa...@gmail.com) wrote: Yes. I am following the readme in there and starting the image, then just running mvn package -DskipTests On April 21,

Re: [DISCUSS] Modify bylaws to allow speculative branches

2017-04-21 Thread Casey Stella
So, what would that look like from a practical perspective? * I presume commits would still associate to a JIRA, right? * Are you proposing changing the strategy from Review then Commit to Commit then Review for these branches? I know that we have some people who are active in the Hadoop project o

Re: Ansible Docker fails to build latest

2017-04-21 Thread Otto Fowler
Yes. I am following the readme in there and starting the image, then just running mvn package -DskipTests On April 21, 2017 at 15:28:15, Casey Stella (ceste...@gmail.com) wrote: Sorry, can you go through how you're getting to this error? I'm not super familiar with this part of the stack..is

[DISCUSS] Modify bylaws to allow speculative branches

2017-04-21 Thread James Sirota
Looking at Hadoop's bylaws https://hadoop.apache.org/bylaws.html They have this: Significant, pervasive features are often developed in a speculative branch of the repository. The PMC may grant commit rights on the branch to its consistent contributors, while the initiative is active. Branch

Re: Ansible Docker fails to build latest

2017-04-21 Thread Casey Stella
Sorry, can you go through how you're getting to this error? I'm not super familiar with this part of the stack..is this compiling Metron inside of a docker image? On Tue, Apr 18, 2017 at 1:45 PM, Otto Fowler wrote: > Is it something to do with the relocation of google.common that happens in > t

[GitHub] incubator-metron issue #542: METRON-873: Stellar string literals do not supp...

2017-04-21 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/542 @ottobackwards you're totally right. We desperately are in need of better stellar documentation: * A language reference * A set of introductory lessons in Stellar --- If your pro

[GitHub] incubator-metron pull request #542: METRON-873: Stellar string literals do n...

2017-04-21 Thread cestella
GitHub user cestella reopened a pull request: https://github.com/apache/incubator-metron/pull/542 METRON-873: Stellar string literals do not support quote escaping ## Contributor Comments Right now, in stellar, we cannot represent a string literal that contains `'foo'` if the st

[GitHub] incubator-metron pull request #542: METRON-873: Stellar string literals do n...

2017-04-21 Thread cestella
Github user cestella closed the pull request at: https://github.com/apache/incubator-metron/pull/542 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the fea

[GitHub] incubator-metron issue #542: METRON-873: Stellar string literals do not supp...

2017-04-21 Thread ottobackwards
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/542 We need a stellar guide, more suited to learning stellar, than just documenting the facts of it. Something that gets you in the shell and gives you some exercises. --- If your proj

[GitHub] incubator-metron issue #542: METRON-873: Stellar string literals do not supp...

2017-04-21 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/542 Yes, you're right. Take a look now. I also added support for \n, \t and \r while I was there. --- If your project is set up for it, you can reply to this email and have your reply appea

[GitHub] incubator-metron issue #542: METRON-873: Stellar string literals do not supp...

2017-04-21 Thread justinleet
Github user justinleet commented on the issue: https://github.com/apache/incubator-metron/pull/542 ``` /code [Stellar]>>> " bar foo " bar \\\ foo ``` Shouldn't that produce 'bar \\ foo' (One less backslash)? It seems like it's not handling chained backslash escap

[GitHub] incubator-metron issue #542: METRON-873: Stellar string literals do not supp...

2017-04-21 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/542 We're not using SCHAR anymore. The current commit should address the backslash issue. Give it a try and let me know what you think. :) --- If your project is set up for it, you can repl

Re: Reducing Warnings in Build

2017-04-21 Thread Otto Fowler
I am sorry, I mean for encodings. We should improve the warnings, and design and plan how to handle non utf-8 separately. On April 21, 2017 at 13:28:09, Michael Miklavcic ( michael.miklav...@gmail.com) wrote: Otto, are you referring to the 80% plus case of encodings or the compiler warnings i

Re: Reducing Warnings in Build

2017-04-21 Thread Michael Miklavcic
Otto, are you referring to the 80% plus case of encodings or the compiler warnings in general? I'm not sure I agree about leaving the encoding as is, but I do agree that we should split the work out from the other javac compiler warnings. Right now we're at the whim of the OS default when we haven'

[GitHub] incubator-metron issue #542: METRON-873: Stellar string literals do not supp...

2017-04-21 Thread justinleet
Github user justinleet commented on the issue: https://github.com/apache/incubator-metron/pull/542 What if I want to put a backslash character in my string? e.g. I want to produce 'some \ string'. I don't think this worked before, but it's particularly relevant because now w

[GitHub] incubator-metron issue #534: METRON-861: Allow JVM args to be passed to CLI ...

2017-04-21 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/534 I could get behind that. Anyone else have other ideas? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project doe

[GitHub] incubator-metron issue #534: METRON-861: Allow JVM args to be passed to CLI ...

2017-04-21 Thread merrimanr
Github user merrimanr commented on the issue: https://github.com/apache/incubator-metron/pull/534 METRON_JVMFLAGS? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wis

[GitHub] incubator-metron issue #534: METRON-861: Allow JVM args to be passed to CLI ...

2017-04-21 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/534 You know, we definitely could. I chose JVMFLAGS because it's what zookeeper calls them, but I'm open to other options. Any suggestions? --- If your project is set up for it, you can rep

[GitHub] incubator-metron issue #534: METRON-861: Allow JVM args to be passed to CLI ...

2017-04-21 Thread merrimanr
Github user merrimanr commented on the issue: https://github.com/apache/incubator-metron/pull/534 Looks good to me. This should be generally useful, even outside of the jaas use case. Would it make sense to namespace the JVMFLAGS environment variable so it's specific to Me

Re: Reducing Warnings in Build

2017-04-21 Thread Otto Fowler
Until there is a use case supporting changing to other then the 80% plus case, this should not be changed. or a customer requirement etc. This would need some design and discussion time as well to map out all the implications for the complete pipeline. I think configurable charset should be separ

[GitHub] incubator-metron pull request #542: METRON-873: Stellar string literals do n...

2017-04-21 Thread cestella
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/542 METRON-873: Stellar string literals do not support quote escaping ## Contributor Comments Right now, in stellar, we cannot represent a string literal that contains `'foo'` if the stri

Re: Reducing Warnings in Build

2017-04-21 Thread JJ Meyer
Nick, You're not off base at all. This is exactly the push back I wanted to hear. I wasn't really sure if what I was proposing was the proper way of going about it. My understanding of charsets is limited, and I've usually just defaulted to UTF-8. That being said, right or wrong, my thought proces

Re: Reducing Warnings in Build

2017-04-21 Thread Ryan Merriman
I think Nick brings up some good points. Would there ever be a reason to not use UTF8 as the default from parsing a message on? All the tools we use for analytics work with UTF8 (am I wrong?). The only case I can see needing a configurable charset would be if a message coming from a sensor were

Re: Reducing Warnings in Build

2017-04-21 Thread Michael Miklavcic
Personally, I think it makes sense to have an inbound/outbound charset setting for the parsers. We should generally standardize on UTF-8 across Metron and its topologies, but accept potentially different charsets from the inbound sensors. That is to say, standardize the defaults for the things we d

Re: Reducing Warnings in Build

2017-04-21 Thread Nick Allen
Per (2), I think it makes sense to make the charset configurable, but with the proposal of 3 separate settings, wouldn't things blow up horribly if the Parsers are producing UTF-8, but Enrichment is expecting UTF-16? They are not even speaking the same language, no? This makes me think that we ne

Re: Failure to Deploy "Quick Dev"

2017-04-21 Thread Casey Stella
I'm betting we need to regenerate the quickdev image after all the mpack changes. On Fri, Apr 21, 2017 at 10:45 AM, Otto Fowler wrote: > From lira: > > I 'think' that quickdev is actually build from full_dev, with metron > installed already. So it may be that we need a new image built to make th

[GitHub] incubator-metron issue #541: METRON-870: Add filtering by packet payload to ...

2017-04-21 Thread nishihatapalmer
Github user nishihatapalmer commented on the issue: https://github.com/apache/incubator-metron/pull/541 Correct, there's no NFA or DFA under the hood of the SequenceMatcher. You can create sequences using the regex syntax using the SequenceMatcherCompiler, as long as only sy

[GitHub] incubator-metron issue #539: METRON-867: In the event that we graduate, remo...

2017-04-21 Thread justinleet
Github user justinleet commented on the issue: https://github.com/apache/incubator-metron/pull/539 I'm fine with looking at it Monday. I may take a quick look if there's any JIRAs from other projects we can steal as a starting template, but if not we can start running through the gra

Re: Failure to Deploy "Quick Dev"

2017-04-21 Thread Otto Fowler
>From lira: I 'think' that quickdev is actually build from full_dev, with metron installed already. So it may be that we need a new image built to make this not an upgrade situation? On April 21, 2017 at 10:37:39, Nick Allen (n...@nickallen.org) wrote: Not sure if I am doing something stupid, b

Failure to Deploy "Quick Dev"

2017-04-21 Thread Nick Allen
Not sure if I am doing something stupid, but I cannot deploy "Quick Dev" from master currently. Full details in https://issues.apache.org/jira/browse/METRON-872.

[GitHub] incubator-metron issue #541: METRON-870: Add filtering by packet payload to ...

2017-04-21 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/541 Currently, I'm using the SequenceMatcher to compile a matching expression and then using a searcher to search in the byte array for that expression (code is [here](https://github.com/cest

[GitHub] incubator-metron issue #539: METRON-867: In the event that we graduate, remo...

2017-04-21 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/539 No, no follow-on tickets as of yet. I was planning on looking at it Monday after the press release and formulating a plan and discussing with the community any impact that might have.

[GitHub] incubator-metron issue #541: METRON-870: Add filtering by packet payload to ...

2017-04-21 Thread nishihatapalmer
Github user nishihatapalmer commented on the issue: https://github.com/apache/incubator-metron/pull/541 When you say use regexes, do you mean use the regex syntax to create fixed length sequences, or do you mean use full regex functionality? Full regex exists using NFAs and DFAs, but

[GitHub] incubator-metron issue #541: METRON-870: Add filtering by packet payload to ...

2017-04-21 Thread nishihatapalmer
Github user nishihatapalmer commented on the issue: https://github.com/apache/incubator-metron/pull/541 There is a slightly out of date (note to self: update this!) syntax document at: https://github.com/nishihatapalmer/byteseek/blob/master/src/main/java/net/byteseek/parser/regex/

[GitHub] incubator-metron pull request #541: METRON-870: Add filtering by packet payl...

2017-04-21 Thread cestella
GitHub user cestella reopened a pull request: https://github.com/apache/incubator-metron/pull/541 METRON-870: Add filtering by packet payload to the pcap query ## Contributor Comments Currently we have the ability to filter packets in the pcap query tool by header information (s

[GitHub] incubator-metron pull request #541: METRON-870: Add filtering by packet payl...

2017-04-21 Thread cestella
Github user cestella closed the pull request at: https://github.com/apache/incubator-metron/pull/541 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the fea

[GitHub] incubator-metron issue #539: METRON-867: In the event that we graduate, remo...

2017-04-21 Thread justinleet
Github user justinleet commented on the issue: https://github.com/apache/incubator-metron/pull/539 +1, I took a look through the site and didn't see anything wrong and also searched the code a bit for signs of incubation. Do we have tickets for any of the follow-on work mentio

Re: Reducing Warnings in Build

2017-04-21 Thread Justin Leet
First off, I would absolutely love to see the warnings reduced and the quality of our code improved. I'm in favor of all four of the points, and I think it's a good start towards weeding out a lot of issues. So regarding question 1: That awkwardness comes about because org.simple.json is all unt

[GitHub] incubator-metron issue #541: METRON-870: Add filtering by packet payload to ...

2017-04-21 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/541 Hey, thanks for that feedback @nishihatapalmer ! I adjusted to use the suggested searcher. I did have one more question, I'm looking to document the possible regex's available for binary

Reducing Warnings in Build

2017-04-21 Thread JJ Meyer
Hello everybody, Currently our build has a significant amount of warnings (most from the error prone plugin). A lot of this has been documented here: https://issues.apache.org/jira/browse/METRON-617 I want to continue to work on this Jira. I really want to reduce the number of warnings in our bui

[GitHub] incubator-metron issue #541: METRON-870: Add filtering by packet payload to ...

2017-04-21 Thread nishihatapalmer
Github user nishihatapalmer commented on the issue: https://github.com/apache/incubator-metron/pull/541 Hi, byteseek author here. I notice you're using the standard BoyerMooreHorspool searcher. Just to let you know that the HorspoolFinalFlagSearcher offers much better performance th