[GitHub] incubator-metron pull request: Initial code for a website
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/1#issuecomment-164833748 Everyone should have the ability to comment or submit a JIRA. On Tue, Dec 15, 2015 at 11:03 AM, ddutta <notificati...@github.com> wrote: > Many people might not have access to JIRA while everyone should be able to > comment on github. What should we use? > > â > Reply to this email directly or view it on GitHub > <https://github.com/apache/incubator-metron/pull/1#issuecomment-164827846> > . > --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #143: METRON-197: Validation should be the las...
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/143 METRON-197: Validation should be the last step in the ParserBolt Right now we are doing the validation prior to the messageFilter. We should only validate the parsed messages which passes through the filter. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron METRON-197 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/143.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #143 commit 53e3f61fe795b1620ee5847448ba1a2a13274120 Author: cstella <ceste...@gmail.com> Date: 2016-06-03T02:25:16Z Fixing global validation to run at the end of the validation pipeline. commit 134e4cd02a149452f9f774a6a85d6c7ccefabc17 Author: cstella <ceste...@gmail.com> Date: 2016-06-03T03:07:24Z Fixing ParserBolt to do validations in the proper place. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #146: METRON-203 General best practice and bug...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/146#discussion_r66321208
--- Diff:
metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/nonbulk/taxii/TaxiiHandler.java
---
@@ -61,346 +63,334 @@
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.dom.DOMSource;
import javax.xml.transform.stream.StreamResult;
-import java.io.*;
+import java.io.IOException;
+import java.io.StringWriter;
import java.net.URI;
import java.net.URISyntaxException;
import java.net.URL;
import java.text.SimpleDateFormat;
import java.util.*;
public class TaxiiHandler extends TimerTask {
-private static final Logger LOG = Logger.getLogger(TaxiiHandler.class);
+ private static final Logger LOG = Logger.getLogger(TaxiiHandler.class);
-private static ThreadLocal xmlFactory = new
ThreadLocal() {
-@Override
-protected TaxiiXmlFactory initialValue() {
-return new TaxiiXmlFactory();
-}
-};
-private static ThreadLocal messageFactory = new
ThreadLocal() {
-@Override
-protected ObjectFactory initialValue() {
-return new ObjectFactory();
-}
-};
-
-private HttpClient taxiiClient;
-private URL endpoint;
-private Extractor extractor;
-private String hbaseTable;
-private String columnFamily;
-private Map<String, HTableInterface> connectionCache = new HashMap<>();
-private HttpClientContext context;
-private String collection;
-private String subscriptionId;
-private EnrichmentConverter converter = new EnrichmentConverter();
-private Date beginTime;
-private Configuration config;
-private boolean inProgress = false;
-private Set allowedIndicatorTypes;
-public TaxiiHandler( TaxiiConnectionConfig connectionConfig
- , Extractor extractor
- , Configuration config
- ) throws Exception
-{
-LOG.info("Loading configuration: " + connectionConfig);
-this.allowedIndicatorTypes =
connectionConfig.getAllowedIndicatorTypes();
-this.extractor = extractor;
-this.collection = connectionConfig.getCollection();
-this.subscriptionId = connectionConfig.getSubscriptionId();
-hbaseTable = connectionConfig.getTable();
-columnFamily = connectionConfig.getColumnFamily();
-this.beginTime = connectionConfig.getBeginTime();
-this.config = config;
-initializeClient(connectionConfig);
-LOG.info("Configured, starting polling " + endpoint + " for " +
collection);
+ private static ThreadLocal xmlFactory = new
ThreadLocal() {
+@Override
+protected TaxiiXmlFactory initialValue() {
+ return new TaxiiXmlFactory();
}
+ };
+ private static ThreadLocal messageFactory = new
ThreadLocal() {
+@Override
+protected ObjectFactory initialValue() {
+ return new ObjectFactory();
+}
+ };
-protected synchronized HTableInterface getTable(String table) throws
IOException {
-HTableInterface ret = connectionCache.get(table);
-if(ret == null) {
-ret = createHTable(table);
-connectionCache.put(table, ret);
-}
-return ret;
+ private HttpClient taxiiClient;
+ private URL endpoint;
+ private Extractor extractor;
+ private String hbaseTable;
+ private String columnFamily;
+ private Map<String, HTableInterface> connectionCache = new HashMap<>();
+ private HttpClientContext context;
+ private String collection;
+ private String subscriptionId;
+ private EnrichmentConverter converter = new EnrichmentConverter();
+ private Date beginTime;
+ private Configuration config;
+ private boolean inProgress = false;
+ private Set allowedIndicatorTypes;
+ public TaxiiHandler( TaxiiConnectionConfig connectionConfig
+ , Extractor extractor
+ , Configuration config
+ ) throws Exception
+ {
+LOG.info("Loading configuration: " + connectionConfig);
+this.allowedIndicatorTypes =
connectionConfig.getAllowedIndicatorTypes();
+this.extractor = extractor;
+this.collection = connectionConfig.getCollection();
+this.subscriptionId = connectionConfig.getSubscriptionId();
+hbaseTable = connectionConfig.getTable();
+columnFamily = connectionConfig.getColumnFamily();
+this.beginTime = connectionConfig.getBeginTime();
+
[GitHub] incubator-metron pull request #148: METRON-215: Fixing an NPE in the Message...
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/148 METRON-215: Fixing an NPE in the MessageParser Optional.of does not accept nulls. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron METRON-215 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/148.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #148 commit 98032f455ef84f4390502731731be4f76e10a661 Author: cstella <ceste...@gmail.com> Date: 2016-06-09T13:51:45Z METRON-215: Fixing an NPE in the MessageParser due to Optional.of not accepting nulls --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #147: METRON-208 Add Google Analytics tags to ...
Github user cestella commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/147#discussion_r66463687 --- Diff: site/diff --- @@ -0,0 +1,186 @@ +diff --git a/site/_includes/footer.html b/site/_includes/footer.html --- End diff -- Did this file get committed accidentally? Probably don't want the diff here. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #149: METRON-217: Found a grabbag of bugs
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/149 METRON-217: Found a grabbag of bugs Doing some scale testing, I found a grabbag of bugs: * The elasticsearch writer should allow multiple ES hosts to be passed in via a List. This should be backwards compatible. * There is a concurrent modification exception while cloning the message field * The parser topology numSpoutTasks and numParserTasks are swapped * The grok parser should throw an exception if it cannot parse so the message that could not be parsed is sent to the error queue. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron bug_squash_1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/149.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #149 commit 3738df5112bf7514fb1a9e9b6c7d812ce97a8a69 Author: cstella <ceste...@gmail.com> Date: 2016-06-10T19:17:57Z Fixing a grabbag of bugs. commit 08609a91483683c241e7f3e706630f79e25bffcb Author: cstella <ceste...@gmail.com> Date: 2016-06-10T19:28:11Z Updating writer bolt. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-186: Create a fieldMapping f...
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/136 METRON-186: Create a fieldMapping functionality which allows for parsed fields to be transformed Currently the parsers take care of transforming raw data to the parsed JSON representation. Allow for a layer to be placed at a sensor level to transform input fields from the parsed messages to create new fields. For instance, mapping IANA protocol numbers to a standardized textual representation (i.e. 6 maps to TCP). This should be generalized, so that we can provide common functions and users can define the input fields and the output field to use. This configuration should live with the sensor configuration in zookeeper. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron transformers Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/136.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #136 commit 0d3b31be29b4238e1923d4d812e40dca4afe64f2 Author: cstella <ceste...@gmail.com> Date: 2016-05-25T19:01:55Z Creating mapping infrastructure for parsing. commit 7e1366735a0ed8eb78ec7103759b6a2574beff2f Author: cstella <ceste...@gmail.com> Date: 2016-05-25T21:16:47Z Renamed and refactored a bit. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/127#discussion_r64669063
--- Diff:
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
---
@@ -60,7 +93,25 @@ public void prepare(Map stormConf, TopologyContext
context, OutputCollector coll
super.prepare(stormConf, context, collector);
this.collector = collector;
parser.init();
-writer.init();
+
+if(isBulk) {
+ writerTransformer = config -> new ParserWriterConfiguration(config);
--- End diff --
The config object can change if zookeeper is updated, so we want an
indirection here.
On Wed, May 25, 2016 at 19:11 merrimanr <notificati...@github.com> wrote:
> In
>
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
> <https://github.com/apache/incubator-metron/pull/127#discussion_r64668872>
> :
>
> > @@ -60,7 +93,25 @@ public void prepare(Map stormConf, TopologyContext
context, OutputCollector coll
> > super.prepare(stormConf, context, collector);
> > this.collector = collector;
> > parser.init();
> > -writer.init();
> > +
> > +if(isBulk) {
> > + writerTransformer = config -> new
ParserWriterConfiguration(config);
>
> Curious why a Function is used here. Why not just instantiate a
> WriterConfiguration object for each case and pass that to
> messageWriter.init? Is there a benefit to doing it this way?
>
> â
> You are receiving this because you authored the thread.
> Reply to this email directly or view it on GitHub
>
<https://github.com/apache/incubator-metron/pull/127/files/e448abd8cccf6f912095474f034668198e58f442#r64668872>
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #150: METRON-222: Address ConcurrentModificationExcep...
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/150 I don't see a difference in the clone method vs the method currently implemented. I think we should evaluate in a follow-on JIRA whether any of the clones are needed. It seems that none of them should be needed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #151: METRON-223: Invalid and Erroneous messag...
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/151 METRON-223: Invalid and Erroneous messages should go to kafka for further analysis in the Parser Topology Right now we send error messages and invalid messages to separate streams in the Parser bolt. These streams should be picked up and sent to kafka for further analysis and indexing. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron parser_error_kafka Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/151.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #151 commit b0365161989acade67f9f0a2957dcb3dcc6d3eb7 Author: cstella <ceste...@gmail.com> Date: 2016-06-09T22:29:29Z generalizing the writers from the parser bolt. commit 266a1e53fe6d661c356398ba90135a75f4208ad2 Author: cstella <ceste...@gmail.com> Date: 2016-06-10T00:47:53Z tests added and corrected. commit 25f4f3432ea57f5376a9668a5ee46ba9f36a2d46 Author: cstella <ceste...@gmail.com> Date: 2016-06-10T16:41:04Z Added integration test. commit 8f1a249c8da3fd0a33557f1a17103d0896e28a0f Author: cstella <ceste...@gmail.com> Date: 2016-06-10T18:17:56Z updating. commit d6310496fa1245973f272f6d5139ffe66304b8dd Author: cstella <ceste...@gmail.com> Date: 2016-06-14T13:34:05Z Merge branch 'master' into parser_error_kafka --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #142: METRON-204: Field Transformation Domain Specifi...
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/142 So, I think the main disadvantage of using groovy or any of the scripting engine integrations is speed and expressibility. Having a targeted domain specific language is going to be a lot simpler to parse than something more complex like groovy (or javascript for that matter). It's hardly even a DSL, frankly, just the ability to compose functions, which seems like a sweet spot between the expressivity and complexity. If we want to implement a FieldTransformation which is a scripting language, there's nothing keeping us from doing this. The other reason I like a simple DSL with just function composition is that I'd *really* prefer to not give users enough rope to hang themselves with here if we can help it. This DSL really exposes only the things you can do in one line of code. If it's more than one line of code, we should be considering that a feature in the language. If we really want to integrate a scripting engine in the future, we can create a separate FieldTransformer to do it in just one class. On Mon, Jun 13, 2016 at 9:38 AM, Nick Allen <notificati...@github.com> wrote: > Love the functionality that you are creating with this. I am curious if it > would make more sense to provide this functionality via an existing > JVM-capable language like Groovy, etc, rather than inventing our own DSL. > At first glance, it seems there are a lot of advantages in doing it this > way. What were the pros/cons when you thought through this? > > â > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <https://github.com/apache/incubator-metron/pull/142#issuecomment-225582895>, > or mute the thread > <https://github.com/notifications/unsubscribe/AAg-x_DQ6RC1lRloLDnlWEgjS3FCFC0Eks5qLV05gaJpZM4ItJMU> > . > --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #144: METRON-211 Enable creation of new Kafka topics ...
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/144 +1 this looks good --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #145: METRON-212: Allow additional Elasticsearch temp...
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/145 +1 looks good --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #142: METRON-204: Field Transformation Domain ...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/142#discussion_r66876932
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/query/generated/PredicateLexer.java
---
@@ -114,59 +118,74 @@ public PredicateLexer(CharStream input) {
public ATN getATN() { return _ATN; }
public static final String _serializedATN =
-
"\3\u0430\ud6d1\u8206\uad2d\u4417\uaef1\u8d80\uaadd\2\26\u00af\b\1\4\2"+
+
"\3\u0430\ud6d1\u8206\uad2d\u4417\uaef1\u8d80\uaadd\2\34\u00d5\b\1\4\2"+
"\t\2\4\3\t\3\4\4\t\4\4\5\t\5\4\6\t\6\4\7\t\7\4\b\t\b\4\t\t\t\4\n\t\n\4"+
"\13\t\13\4\f\t\f\4\r\t\r\4\16\t\16\4\17\t\17\4\20\t\20\4\21\t\21\4\22"+
-
"\t\22\4\23\t\23\4\24\t\24\4\25\t\25\4\26\t\26\3\2\3\2\3\2\3\2\3\2\3\2"+
-
"\3\2\3\2\5\2\66\n\2\3\3\3\3\3\3\3\3\3\3\3\3\5\3>\n\3\3\4\3\4\3\4\3\4\3"+
-
"\4\3\4\5\4F\n\4\3\5\3\5\3\5\3\5\3\5\3\5\3\5\3\5\5\5P\n\5\3\6\3\6\3\6\3"+
-
"\6\3\6\3\6\3\6\3\6\3\6\3\6\5\6\\\n\6\3\7\3\7\3\7\3\b\3\b\3\b\3\t\3\t\3"+
-
"\n\3\n\3\13\3\13\3\f\3\f\3\r\3\r\3\16\3\16\3\16\3\17\3\17\3\17\3\17\3"+
-
"\17\3\17\3\17\3\20\3\20\3\20\3\20\3\20\3\20\3\20\3\21\3\21\7\21\u0081"+
-
"\n\21\f\21\16\21\u0084\13\21\3\22\3\22\3\23\3\23\7\23\u008a\n\23\f\23"+
-
"\16\23\u008d\13\23\3\23\3\23\3\23\7\23\u0092\n\23\f\23\16\23\u0095\13"+
-
"\23\3\23\5\23\u0098\n\23\3\24\3\24\3\25\3\25\3\25\3\25\6\25\u00a0\n\25"+
-
"\r\25\16\25\u00a1\3\25\5\25\u00a5\n\25\3\25\3\25\3\26\6\26\u00aa\n\26"+
-
"\r\26\16\26\u00ab\3\26\3\26\3\u00a1\2\27\3\3\5\4\7\5\t\6\13\7\r\b\17\t"+
-
"\21\n\23\13\25\f\27\r\31\16\33\17\35\20\37\21!\22#\2%\23\'\24)\25+\26"+
-
"\3\2\7\5\2C\\aac|\b\2\60\60\62;C\\^^aac|\7\2\f\f\17\17$$))^^\3\3\f\f\5"+
-
"\2\13\f\16\17\"\"\u00ba\2\3\3\2\2\2\2\5\3\2\2\2\2\7\3\2\2\2\2\t\3\2\2"+
-
"\2\2\13\3\2\2\2\2\r\3\2\2\2\2\17\3\2\2\2\2\21\3\2\2\2\2\23\3\2\2\2\2\25"+
-
"\3\2\2\2\2\27\3\2\2\2\2\31\3\2\2\2\2\33\3\2\2\2\2\35\3\2\2\2\2\37\3\2"+
-
"\2\2\2!\3\2\2\2\2%\3\2\2\2\2\'\3\2\2\2\2)\3\2\2\2\2+\3\2\2\2\3\65\3\2"+
-
"\2\2\5=\3\2\2\2\7E\3\2\2\2\tO\3\2\2\2\13[\3\2\2\2\r]\3\2\2\2\17`\3\2\2"+
-
"\2\21c\3\2\2\2\23e\3\2\2\2\25g\3\2\2\2\27i\3\2\2\2\31k\3\2\2\2\33m\3\2"+
-
"\2\2\35p\3\2\2\2\37w\3\2\2\2!~\3\2\2\2#\u0085\3\2\2\2%\u0097\3\2\2\2\'"+
-
"\u0099\3\2\2\2)\u009b\3\2\2\2+\u00a9\3\2\2\2-.\7c\2\2./\7p\2\2/\66\7f"+
-
"\2\2\60\61\7(\2\2\61\66\7(\2\2\62\63\7C\2\2\63\64\7P\2\2\64\66\7F\2\2"+
-
"\65-\3\2\2\2\65\60\3\2\2\2\65\62\3\2\2\2\66\4\3\2\2\2\678\7q\2\28>\7t"+
-
"\2\29:\7~\2\2:>\7~\2\2;<\7Q\2\2<>\7T\2\2=\67\3\2\2\2=9\3\2\2\2=;\3\2\2"+
-
"\2>\6\3\2\2\2?@\7p\2\2@A\7q\2\2AF\7v\2\2BC\7P\2\2CD\7Q\2\2DF\7V\2\2E?"+
-
"\3\2\2\2EB\3\2\2\2F\b\3\2\2\2GH\7v\2\2HI\7t\2\2IJ\7w\2\2JP\7g\2\2KL\7"+
-
"V\2\2LM\7T\2\2MN\7W\2\2NP\7G\2\2OG\3\2\2\2OK\3\2\2\2P\n\3\2\2\2QR\7h\2"+
-
"\2RS\7c\2\2ST\7n\2\2TU\7u\2\2U\\\7g\2\2VW\7H\2\2WX\7C\2\2XY\7N\2\2YZ\7"+
-
"U\2\2Z\\\7G\2\2[Q\3\2\2\2[V\3\2\2\2\\\f\3\2\2\2]^\7?\2\2^_\7?\2\2_\16"+
-
"\3\2\2\2`a\7#\2\2ab\7?\2\2b\20\3\2\2\2cd\7.\2\2d\22\3\2\2\2ef\7]\2\2f"+
-
"\24\3\2\2\2gh\7_\2\2h\26\3\2\2\2ij\7*\2\2j\30\3\2\2\2kl\7+\2\2l\32\3\2"+
-
"\2\2mn\7k\2\2no\7p\2\2o\34\3\2\2\2pq\7p\2\2qr\7q\2\2rs\7v\2\2st\7\"\2"+
-
"\2tu\7k\2\2uv\7p\2\2v\36\3\2\2\2wx\7g\2\2xy\7z\2\2yz\7k\2\2z{\7u\2\2{"+
- "|\7v\2\2|}\7u\2\2}
\3\2\2\2~\u0082\t\2\2\2\177\u0081\t\3\2\2\u0080\177"+
-
"\3\2\2\2\u0081\u0084\3\2\2\2\u0082\u0080\3\2\2\2\u0082\u0083\3\2\2\2\u0083"+
-
"\"\3\2\2\2\u0084\u0082\3\2\2\2\u0085\u0086\n\4\2\2\u0086$\3\2\2\2\u0087"+
-
"\u008b\7$\2\2\u0088\u008a\5#\22\2\u0089\u0088\3\2\2\2\u008a\u008d\3\2"+
-
"\2\2\u008b\u0089\3\2\2\2\u008b\u008c\3\2\2\2\u008c\u008e\3\2\2\2\u008d"+
-
"\u008b\3\2\2\2\u008e\u0098\7$\2\2\u008f\u0093\7)\2\2\u0090\u0092\5#\22"+
-
"\2\u0091\u0090\3\2\2\2\u0092\u0095\3\2\2\2\u0093\u0091\3\2\2\2\u0093\u0094"+
-
"\3\2\2\2\u0094\u0096\3\2\2\2\u0095\u0093\3\2\
[GitHub] incubator-metron pull request #142: METRON-204: Field Transformation Domain ...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/142#discussion_r66877352
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/aggregator/Aggregators.java
---
@@ -18,18 +18,21 @@
package org.apache.metron.common.aggregator;
+import org.apache.metron.common.utils.ConversionUtils;
+
import java.util.List;
import java.util.Map;
import java.util.function.BinaryOperator;
import java.util.function.Predicate;
public enum Aggregators implements Aggregator {
- MAX( (numbers, config) -> accumulate(0d, (x,y) ->
Math.max(x.doubleValue(),y.doubleValue()), numbers))
- ,MIN( (numbers, config) -> accumulate(0d, (x,y) ->
Math.min(x.doubleValue(),y.doubleValue()), numbers))
- ,SUM( (numbers, config) -> accumulate(0d, (x,y) -> x.doubleValue() +
y.doubleValue(), numbers))
+ MAX( (numbers, config) -> accumulate(0d, (x,y) ->
Math.max(x.doubleValue(),y.doubleValue()), numbers, config))
+ ,MIN( (numbers, config) -> accumulate(0d, (x,y) ->
Math.min(x.doubleValue(),y.doubleValue()), numbers, config))
+ ,SUM( (numbers, config) -> accumulate(0d, (x,y) -> x.doubleValue() +
y.doubleValue(), numbers, config))
,MEAN( (numbers, config) -> scale(SUM.aggregate(numbers, config),
numbers, n -> true))
- ,POSITIVE_MEAN( (numbers, config) -> scale(SUM.aggregate(numbers,
config), numbers, n -> n.doubleValue() > 0))
+ ,POSITIVE_MEAN( (numbers, config) -> positiveMean(numbers, config))
;
+ public static String NEGATIVE_VALUES_TRUMP_CONF = "negativeValuesTrump";
--- End diff --
I noticed an issue with threat triage in-that we have no ability to cause a
particular rule to trump the rest. For instance, if you have rules 1 through
5, but if you hit rule 5, you want to make sure that the triage level is very
high. For that use-case, if you associate a negative value with a triage rule,
the aggregators will return +Infinity.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #146: METRON-203 General best practice and bug fixes
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/146 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #150: METRON-222: Address ConcurrentModificati...
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/150 METRON-222: Address ConcurrentModificationException in BulkMessageWriter You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron METRON-222 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/150.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #150 commit 1843e1b083ca75a0e0fc2f60d863856a3c09ecf4 Author: cstella <ceste...@gmail.com> Date: 2016-06-14T02:24:12Z METRON-222: Address ConcurrentModificationException in BulkMessageWriter --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #142: METRON-204: Field Transformation Domain Specifi...
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/142 Also, it seems like a big PR, but honestly it's deceptive. There was a lot of file moving and the generated Antlr code inflates the size. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-190: Make start_parser_topol...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/139#issuecomment-222313782 @james-sirota Right you are! Just updated exposing the numTasks for the spout and the parser bolt. ``` -pnt,--parser_num_tasksParser Num Tasks -pp,--parser_p Parser Parallelism -snt,--spout_num_tasksSpout Num Tasks -sp,--spout_p Spout Parallelism ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/127#issuecomment-222638653 You sure Kafka is still up? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-190: Make start_parser_topol...
GitHub user cestella opened a pull request:
https://github.com/apache/incubator-metron/pull/139
METRON-190: Make start_parser_topology.sh more adaptable regarding storm
topology configuration
Add the ability to specify the message timeout, the max task parallelism,
the number of ackers, the number of workers. Also allow the user to pass in
JSON file containing a map of extra storm configs.
I added the following extra options to `start_parser_topology.sh`:
```
-e,--extra_options Extra options in the form of a
JSON file with a map for
content.
-mt,--message_timeoutMessage Timeout in Seconds
-mtp,--max_task_parallelism Max task parallelism
-na,--num_ackers Number of Ackers
-nw,--num_workersNumber of Workers
```
For instance, if you wanted to set some storm property on
the config called `topology.ticks.tuple.freq.secs` to 1000 and
`storm.local.dir` to `/opt/my/path`
you could create a file called `custom_config.json` containing
```
{
"topology.ticks.tuple.freq.secs" : 1000,
"storm.local.dir" : "/opt/my/path"
}
```
and pass `--extra_options custom_config.json` to `start_parser_topology.sh`.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cestella/incubator-metron
parser_parallelism_expose
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/139.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #139
commit 33815e0382b54fb6931bae4579223535e7abdc90
Author: cstella <ceste...@gmail.com>
Date: 2016-05-27T19:18:12Z
Exposed more storm topology config options.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-190: Make start_parser_topol...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/139#issuecomment-53339 Ok, I verified this on full dev vagrant. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-186: Create a fieldMapping f...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/136#issuecomment-221862988 Yeah, I was thinking about that too. I think transformation is a more descriptive term. You're the 2nd person (in addition to myself) who has made that comment. I think that means I should call it transformation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-100 GeoIP errors out silentl...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/134#discussion_r64742580
--- Diff:
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/jdbc/JdbcAdapter.java
---
@@ -38,6 +39,27 @@
private JdbcConfig config;
private String host;
+ protected boolean isConnectionClosed() {
+boolean isClosed = statement == null || connection == null;
+if(!isClosed) {
+ try {
+isClosed = statement.isClosed() || connection.isClosed();
+ } catch (SQLException e) {
+_LOG.error("Unable to maintain open JDBC connection: " +
e.getMessage(), e);
+isClosed = true;
+ }
+}
+return isClosed;
+ }
+
+ protected boolean resetConnectionIfNecessary() {
+if(isConnectionClosed())
+{
--- End diff --
So, I'll definitely make the class consistently K (I must've fat-fingered
the newline). We don't have a coherent style at the moment in this respect. I
prefer K bracing, so you'll see that in my code, but yeah, I agree, we need
to correct style as we see it.
Also, don't worry about nitpicking..we all have to live in this code. :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...
Github user cestella commented on the pull request:
https://github.com/apache/incubator-metron/pull/127#issuecomment-222534318
Looks like it can't find the writerClassname field. Are you sure you ran a
build from this branch before the deploy?
On Mon, May 30, 2016 at 13:48 James Sirota <notificati...@github.com> wrote:
> /usr/metron/0.1BETA/bin/zk_load_configs.sh -m DUMP -z 1xxx:2181
> log4j:WARN No appenders could be found for logger
> (org.apache.curator.framework.imps.CuratorFrameworkImpl).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
> more info.
> GLOBAL Config: global
> {
> "es.clustername": "metron",
> "es.ip": "xxx",
> "es.port": "9300",
> "es.date.format": ".MM.dd.HH"
> }
>
> PARSER Config: websphere
> {
>
>
"parserClassName":"org.apache.metron.parsers.websphere.GrokWebSphereParser",
> "sensorTopic":"websphere",
> "parserConfig":
> {
> "grokPath":"/patterns/websphere",
> "patternLabel":"WEBSPHERE",
> "timestampField":"timestamp_string",
> "dateFormat":" MMM dd HH:mm:ss"
> }
> }
>
> PARSER Config: bluecoat
> {
>
"parserClassName":"org.apache.metron.parsers.bluecoat.BasicBluecoatParser",
> "sensorTopic":"bluecoat",
> "parserConfig": {}
> }
>
> PARSER Config: squid
> {
> "parserClassName": "org.apache.metron.parsers.GrokParser",
> "sensorTopic": "squid",
> "parserConfig": {
> "grokPath": "/patterns/squid",
> "patternLabel": "SQUID_DELIMITED",
> "timestampField": "timestamp"
> }
> }
>
> Exception in thread "main" java.lang.RuntimeException: Unable to load {
>
>
> "parserClassName" : "org.apache.metron.parsers.csv.CSVParser"
> ,"writerClassName" :
> "org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter"
> ,"sensorTopic":"user"
> ,"parserConfig":
> {
> "shew.table" : "enrichment"
> ,"shew.cf" : "t"
> ,"shew.keyColumns" : "user"
> ,"shew.enrichmentType" : "user"
> ,"columns" : {
> "user" : 0
> ,"ip" : 1
> }
> }
> }
>
> at
org.apache.metron.common.configuration.ConfigurationType.lambda$static$1(ConfigurationType.java:47)
> at
org.apache.metron.common.configuration.ConfigurationType$$Lambda$9/1684106402.apply(Unknown
Source)
> at
org.apache.metron.common.configuration.ConfigurationType.deserialize(ConfigurationType.java:78)
> at
org.apache.metron.common.configuration.ConfigurationsUtils.lambda$dumpConfigs$0(ConfigurationsUtils.java:272)
> at
org.apache.metron.common.configuration.ConfigurationsUtils$$Lambda$7/785992331.visit(Unknown
Source)
> at
org.apache.metron.common.configuration.ConfigurationsUtils.visitConfigs(ConfigurationsUtils.java:264)
> at
org.apache.metron.common.configuration.ConfigurationsUtils.visitConfigs(ConfigurationsUtils.java:251)
> at
org.apache.metron.common.configuration.ConfigurationsUtils.dumpConfigs(ConfigurationsUtils.java:271)
> at
org.apache.metron.common.cli.ConfigurationManager.dump(ConfigurationManager.java:115)
> at
org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:177)
> at
org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:161)
> at
org.apache.metron.common.cli.ConfigurationManager.main(ConfigurationManager.java:198)
>
> Caused by:
> com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException:
> Unrecognized field "writerClassName" (class
> org.apache.metron.common.configuration.SensorParserConfig), not marked as
> ignorable (3 known properties: , "parserConfig", "parserClassName",
> "sensorTopic"])
> at Source: java.io.StringReader@23bb844
>
<https://github.com/java.io.StringReader/incubator-metron/commit/23bb8443>;
> line: 3, column: 26
> at
>
com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyExcept
[GitHub] incubator-metron pull request: METRON-183 Allow the simple hbase e...
Github user cestella commented on the pull request:
https://github.com/apache/incubator-metron/pull/131#issuecomment-222363251
The way to validate this is to
* Create a second column family on the enrichment HBase table, say `cf1`
* Push some enrichment data into the table in that column family.
Something similar to the malicious IP's from the blog post. I'd suggest
calling the enrichment type `malicious_ip`
* Modify the configs for one of the topologies you want to test in
`/usr/metron/0.1BETA/config/zookeeper/enrichment` to add a `config` section
under `threatIntel` marking that `malicious_ip` should come from column family
`cf1` like the following
```
{
"index": "bro",
"batchSize": 5,
"threatIntel": {
"fieldMap": {
"hbaseThreatIntel" : [ "ip_dst_addr" ]
},
"fieldToTypeMap": {
"ip_dst_addr" : [ "malicious_ip" ]
},
"config" : {
"typeToColumnFamily" : {
"malicious_ip" : "cf1"
}
}
}
}
```
* Run some data through and ensure that enrichments still exist.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-190: Make start_parser_topol...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/139#issuecomment-222363512 good catch, I'll make those changes now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...
Github user cestella commented on the pull request:
https://github.com/apache/incubator-metron/pull/127#issuecomment-222364875
In order to validate this, you can do the following:
* Configure a new parser, in this example I'll call it a `user` parser and
we'll parse some CSV data to map `username` to `ip` by creating a file
`/usr/metron/0.1BETA/config/zookeeper/enrichment/user.json` with
```
{
"parserClassName" : "org.apache.metron.parsers.csv.CSVParser"
,"writerClassName" :
"org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter"
,"sensorTopic":"user"
,"parserConfig":
{
"shew.table" : "enrichment"
,"shew.cf" : "t"
,"shew.keyColumns" : "user"
,"shew.enrichmentType" : "user"
,"columns" : {
"user" : 0
,"ip" : 1
}
}
}
```
* Add a new `user` enrichment type to `bro` data by adding `ip_src_addr` to
`hbaseEnrichment` and associating `user` as a field type for `ip_src_addr` in
`/usr/metron/0.1BETA/config/zookeeper/enrichment/bro.json` like so
```
{
"index": "bro",
"batchSize": 5,
"enrichment": {
"fieldMap": {
"geo": [
"ip_dst_addr",
"ip_src_addr"
],
"host": [
"host"
],
"hbaseEnrichment" : [ "ip_src_addr" ]
},
"fieldToTypeMap":
{
"ip_src_addr" : [ "user"]
}
},
"threatIntel":{
"fieldMap":
{
"hbaseThreatIntel": ["ip_dst_addr", "ip_src_addr"]
},
"fieldToTypeMap":
{
"ip_dst_addr" : [ "malicious_ip" ]
,"ip_src_addr" : [ "malicious_ip" ]
}
}
}```
* Create the Kafka Queue as in the tutorials
* Using `/usr/metron/0.1BETA/bin/zk_load_configs.sh` push up the config you
just created. `/usr/metron/0.1BETA/bin/zk_load_configs.sh -m PUSH -z node1:2181
-i /usr/metron/0.1BETA/config/zookeeper`
* Create some reference CSV reference data with that looks like
`jsirota,192.168.168.1` into a csv file named `user.csv`
* Use the kafka console producer to push data into the `user` topic via
`cat user.csv | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh
--broker-list node1:6667 --topic user`
* You should be able to check that the data gets into HBase by doing a
`scan 'enrichment'` from the `hbase shell`
* You should also be able to check, after new data has been run through,
that the data is enriched in elasticsearch. I would suggest bouncing the
enrichment topology to ensure that stale data in the caches get flushed, but
that is not strictly necessary.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-183 Allow the simple hbase e...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/131#issuecomment-222363326 As it stands, we have the docs for the configs with where the configurations live. I think you are absolutely right that we need to move them closer to the place where they are used, rather than where the configuration objects are defined. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-189: Add the ability to do g...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/138#issuecomment-222363626 Yeah, the documentation situation is a bit incorrect. Documentation for configuration started to be placed where the configuration objects existed, rather than where they were used. Since we put the configuration in commons, then the documentation went in commons. I suggest strongly that we do a follow-on after we flush the PR queue to move docs around. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #167: METRON-246 metron_streaming role needs to expli...
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/167 +1, looks good --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #152: METRON-228: Fixing NPE when enrichment c...
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/152 METRON-228: Fixing NPE when enrichment config does not exist. Because we are pulling the batch size from the enrichment sensor config, if it does not exist then it will throw a NPE. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron METRON-228 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/152.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #152 commit bd43de8be1169f06873c77da76a6a2e8c2b5ce69 Author: cstella <ceste...@gmail.com> Date: 2016-06-15T19:04:12Z METRON-228: Fixing NPE when enrichment config does not exist. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #155: METRON-231: Snort parser should throw ex...
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/155 METRON-231: Snort parser should throw exception Snort parser returns null when we should be throwing an exception in the event of an improperly formatted message. An exception will result in the message being sent to the error queue. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron METRON-231 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/155.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #155 commit 931e66ce2f87ad6ff6d384928447b1dd9c1c4bd1 Author: cstella <ceste...@gmail.com> Date: 2016-06-16T13:30:49Z METRON-231: Snort parser should throw exception --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #154: METRON-230: Bro parser should throw exce...
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/154 METRON-230: Bro parser should throw exception Right now, if an invalid message comes to the bro parser, it returns null, which is interpreted as no messages from the parser. INstead, we should throw an exception so the message can be routed to the error queue. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron METRON-230 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/154.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #154 commit 5c507f682f60d3ddf333d31f1401d907778ec6e3 Author: cstella <ceste...@gmail.com> Date: 2016-06-16T12:59:06Z METRON-230: Bro parser should throw exception rather than returning null. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #156: METRON-235 Expose filtering capability f...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/156#discussion_r67419245
--- Diff:
metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/CliParser.java
---
@@ -0,0 +1,83 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.pcap.query;
+
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.HelpFormatter;
+import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
+
+/**
+ * Provides commmon required fields for the PCAP filter jobs
+ */
+public class CliParser {
+
+ public Options buildOptions() {
+Options options = new Options();
+options.addOption(newOption("h", false, "Display help"));
+options.addOption(newOption("basePath", true, String.format("Base PCAP
data path. Default is '%s'", CliConfig.BASE_PATH_DEFAULT)));
--- End diff --
Is this a short option as well as a long option? I tend to like having
both at our disposal.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #156: METRON-235 Expose filtering capability f...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/156#discussion_r67420354
--- Diff:
metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/QueryCliParser.java
---
@@ -0,0 +1,57 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.pcap.query;
+
+import org.apache.commons.cli.*;
+
+public class QueryCliParser extends CliParser {
+ private Options queryOptions;
+
+ public QueryCliParser() {
+queryOptions = setupOptions();
+ }
+
+ private Options setupOptions() {
+Options options = buildOptions();
+options.addOption(newOption("query", true, "Query string to use as a
filter"));
+return options;
+ }
+
+ /**
+ * Parses query pcap filter options and required parameters common to
all filter types.
+ *
+ * @param args command line arguments to parse
+ * @return Configuration tailored to query pcap queries
+ * @throws ParseException
+ */
+ public QueryCliConfig parse(String[] args) throws ParseException {
+CommandLineParser parser = new BasicParser();
--- End diff --
All the existing utilities use PosixParser, can we move to that to conform?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #156: METRON-235 Expose filtering capability f...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/156#discussion_r67420651
--- Diff:
metron-platform/metron-pcap-backend/src/main/scripts/pcap_query.sh ---
@@ -0,0 +1,34 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+BIGTOP_DEFAULTS_DIR=${BIGTOP_DEFAULTS_DIR-/etc/default}
+[ -n "${BIGTOP_DEFAULTS_DIR}" -a -r ${BIGTOP_DEFAULTS_DIR}/hbase ] && .
${BIGTOP_DEFAULTS_DIR}/hbase
+
+# Autodetect JAVA_HOME if not defined
+if [ -e /usr/libexec/bigtop-detect-javahome ]; then
+ . /usr/libexec/bigtop-detect-javahome
+elif [ -e /usr/lib/bigtop-utils/bigtop-detect-javahome ]; then
+ . /usr/lib/bigtop-utils/bigtop-detect-javahome
+fi
+
+export METRON_VERSION=${project.version}
+export METRON_HOME=/usr/metron/$METRON_VERSION
+export API_JAR=${project.artifactId}-$METRON_VERSION.jar
--- End diff --
Can we call the `API_JAR` variable something else since it's coming from
the metron-pcap-backend project? Perhaps `PCAP_BACKEND_JAR`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #156: METRON-235 Expose filtering capability f...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/156#discussion_r67421199
--- Diff:
metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/PcapCli.java
---
@@ -0,0 +1,168 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.pcap.query;
+
+import org.apache.commons.cli.ParseException;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.metron.common.system.Clock;
+import org.apache.metron.common.utils.timestamp.TimestampConverters;
+import org.apache.metron.pcap.filter.fixed.FixedPcapFilter;
+import org.apache.metron.pcap.filter.query.QueryPcapFilter;
+import org.apache.metron.pcap.mr.PcapJob;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+public class PcapCli {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(PcapCli.class);
+ private final PcapJob jobRunner;
+ private final ResultsWriter resultsWriter;
+ private final Clock clock;
+
+ public static void main(String[] args) {
+int status = new PcapCli(new PcapJob(), new ResultsWriter(), new
Clock()).run(args);
+System.exit(status);
+ }
+
+ public PcapCli(PcapJob jobRunner, ResultsWriter resultsWriter, Clock
clock) {
+this.jobRunner = jobRunner;
+this.resultsWriter = resultsWriter;
+this.clock = clock;
+ }
+
+ public int run(String[] args) {
+if (args.length < 1) {
+ printBasicHelp();
+ return -1;
+}
+String jobType = args[0];
+List<byte[]> results = new ArrayList<>();
+if ("fixed".equals(jobType)) {
+ FixedCliParser fixedParser = new FixedCliParser();
+ FixedCliConfig config = null;
+ try {
+config = fixedParser.parse(Arrays.copyOfRange(args, 1,
args.length));
+ } catch (ParseException e) {
+System.out.println(e.getMessage());
+fixedParser.printHelp();
--- End diff --
Also, would be nice for printHelp to take the stream as an arg
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #156: METRON-235 Expose filtering capability f...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/156#discussion_r67421145
--- Diff:
metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/PcapCli.java
---
@@ -0,0 +1,168 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.pcap.query;
+
+import org.apache.commons.cli.ParseException;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.metron.common.system.Clock;
+import org.apache.metron.common.utils.timestamp.TimestampConverters;
+import org.apache.metron.pcap.filter.fixed.FixedPcapFilter;
+import org.apache.metron.pcap.filter.query.QueryPcapFilter;
+import org.apache.metron.pcap.mr.PcapJob;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+public class PcapCli {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(PcapCli.class);
+ private final PcapJob jobRunner;
+ private final ResultsWriter resultsWriter;
+ private final Clock clock;
+
+ public static void main(String[] args) {
+int status = new PcapCli(new PcapJob(), new ResultsWriter(), new
Clock()).run(args);
+System.exit(status);
+ }
+
+ public PcapCli(PcapJob jobRunner, ResultsWriter resultsWriter, Clock
clock) {
+this.jobRunner = jobRunner;
+this.resultsWriter = resultsWriter;
+this.clock = clock;
+ }
+
+ public int run(String[] args) {
+if (args.length < 1) {
+ printBasicHelp();
+ return -1;
+}
+String jobType = args[0];
+List<byte[]> results = new ArrayList<>();
+if ("fixed".equals(jobType)) {
+ FixedCliParser fixedParser = new FixedCliParser();
+ FixedCliConfig config = null;
+ try {
+config = fixedParser.parse(Arrays.copyOfRange(args, 1,
args.length));
+ } catch (ParseException e) {
+System.out.println(e.getMessage());
--- End diff --
We should probably print to stderr here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #156: METRON-235 Expose filtering capability f...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/156#discussion_r67422356
--- Diff:
metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/PcapCli.java
---
@@ -0,0 +1,168 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.pcap.query;
+
+import org.apache.commons.cli.ParseException;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.metron.common.system.Clock;
+import org.apache.metron.common.utils.timestamp.TimestampConverters;
+import org.apache.metron.pcap.filter.fixed.FixedPcapFilter;
+import org.apache.metron.pcap.filter.query.QueryPcapFilter;
+import org.apache.metron.pcap.mr.PcapJob;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+public class PcapCli {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(PcapCli.class);
+ private final PcapJob jobRunner;
+ private final ResultsWriter resultsWriter;
+ private final Clock clock;
+
+ public static void main(String[] args) {
+int status = new PcapCli(new PcapJob(), new ResultsWriter(), new
Clock()).run(args);
+System.exit(status);
+ }
+
+ public PcapCli(PcapJob jobRunner, ResultsWriter resultsWriter, Clock
clock) {
+this.jobRunner = jobRunner;
+this.resultsWriter = resultsWriter;
+this.clock = clock;
+ }
+
+ public int run(String[] args) {
+if (args.length < 1) {
+ printBasicHelp();
+ return -1;
+}
+String jobType = args[0];
+List<byte[]> results = new ArrayList<>();
+if ("fixed".equals(jobType)) {
+ FixedCliParser fixedParser = new FixedCliParser();
+ FixedCliConfig config = null;
+ try {
+config = fixedParser.parse(Arrays.copyOfRange(args, 1,
args.length));
--- End diff --
Nope, this is fine by me. I just wanted to make sure it wasn't to skip
over hadoop options because we weren't using GenericOptionsParser. I retract
the comment. :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #156: METRON-235 Expose filtering capability f...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/156#discussion_r67420022
--- Diff:
metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/PcapCli.java
---
@@ -0,0 +1,168 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.pcap.query;
+
+import org.apache.commons.cli.ParseException;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.metron.common.system.Clock;
+import org.apache.metron.common.utils.timestamp.TimestampConverters;
+import org.apache.metron.pcap.filter.fixed.FixedPcapFilter;
+import org.apache.metron.pcap.filter.query.QueryPcapFilter;
+import org.apache.metron.pcap.mr.PcapJob;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+public class PcapCli {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(PcapCli.class);
+ private final PcapJob jobRunner;
+ private final ResultsWriter resultsWriter;
+ private final Clock clock;
+
+ public static void main(String[] args) {
+int status = new PcapCli(new PcapJob(), new ResultsWriter(), new
Clock()).run(args);
+System.exit(status);
+ }
+
+ public PcapCli(PcapJob jobRunner, ResultsWriter resultsWriter, Clock
clock) {
+this.jobRunner = jobRunner;
+this.resultsWriter = resultsWriter;
+this.clock = clock;
+ }
+
+ public int run(String[] args) {
+if (args.length < 1) {
--- End diff --
Don't we need to use GenericOptionsParser(conf, args).getRemainingArgs()?
Otherwise we're going to get all the hadoop args if we need to specify specific
`-D` stuff for the MR job.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #156: METRON-235 Expose filtering capability f...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/156#discussion_r67420179
--- Diff:
metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/PcapCli.java
---
@@ -0,0 +1,168 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.pcap.query;
+
+import org.apache.commons.cli.ParseException;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.metron.common.system.Clock;
+import org.apache.metron.common.utils.timestamp.TimestampConverters;
+import org.apache.metron.pcap.filter.fixed.FixedPcapFilter;
+import org.apache.metron.pcap.filter.query.QueryPcapFilter;
+import org.apache.metron.pcap.mr.PcapJob;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+public class PcapCli {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(PcapCli.class);
+ private final PcapJob jobRunner;
+ private final ResultsWriter resultsWriter;
+ private final Clock clock;
+
+ public static void main(String[] args) {
+int status = new PcapCli(new PcapJob(), new ResultsWriter(), new
Clock()).run(args);
+System.exit(status);
+ }
+
+ public PcapCli(PcapJob jobRunner, ResultsWriter resultsWriter, Clock
clock) {
+this.jobRunner = jobRunner;
+this.resultsWriter = resultsWriter;
+this.clock = clock;
+ }
+
+ public int run(String[] args) {
+if (args.length < 1) {
+ printBasicHelp();
+ return -1;
+}
+String jobType = args[0];
+List<byte[]> results = new ArrayList<>();
+if ("fixed".equals(jobType)) {
+ FixedCliParser fixedParser = new FixedCliParser();
+ FixedCliConfig config = null;
+ try {
+config = fixedParser.parse(Arrays.copyOfRange(args, 1,
args.length));
--- End diff --
Why are we starting at 1 here? Is it to skip over some hadoop configs? If
so, see above comment about GenericOptionsParser for a more general approach.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #156: METRON-235 Expose filtering capability f...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/156#discussion_r67421802
--- Diff:
metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/CliParser.java
---
@@ -0,0 +1,83 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.pcap.query;
+
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.HelpFormatter;
+import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
+
+/**
+ * Provides commmon required fields for the PCAP filter jobs
+ */
+public class CliParser {
+
+ public Options buildOptions() {
+Options options = new Options();
+options.addOption(newOption("h", false, "Display help"));
+options.addOption(newOption("basePath", true, String.format("Base PCAP
data path. Default is '%s'", CliConfig.BASE_PATH_DEFAULT)));
+options.addOption(newOption("baseOutputPath", true,
String.format("Query result output path. Default is '%s'",
CliConfig.BASE_OUTPUT_PATH_DEFAULT)));
+options.addOption(newOption("startTime", true, "Packet start time
range. Default is '0'"));
--- End diff --
start time should be required. As it stands, the defaults, if unspecified
is to return everything. That is likely not what we want and will result in a
VERY big MR job in some of the installations.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #156: METRON-235 Expose filtering capability f...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/156#discussion_r67424320
--- Diff:
metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/FixedCliParser.java
---
@@ -0,0 +1,66 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.pcap.query;
+
+import org.apache.commons.cli.*;
+import org.apache.metron.common.Constants;
+
+public class FixedCliParser extends CliParser {
+ private Options fixedOptions;
+
+ public FixedCliParser() {
+fixedOptions = buildFixedOptions();
+ }
+
+ private Options buildFixedOptions() {
+Options options = buildOptions();
+options.addOption(newOption("srcAddr", true, "Source IP address"));
--- End diff --
Can we conform to the naming convention around src/dst addr/port and change:
* `srcAddr` to `ip_src_addr` as the long option
* `dstAddr` to `ip_dst_addr` as the long option
* `srcPort` to `ip_src_port` as the long option
* `dstPort` to `ip_dst_port` as the long option
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #156: METRON-235 Expose filtering capability f...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/156#discussion_r67422117
--- Diff:
metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/CliParser.java
---
@@ -0,0 +1,83 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.pcap.query;
+
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.HelpFormatter;
+import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
+
+/**
+ * Provides commmon required fields for the PCAP filter jobs
+ */
+public class CliParser {
+
+ public Options buildOptions() {
+Options options = new Options();
+options.addOption(newOption("h", false, "Display help"));
+options.addOption(newOption("basePath", true, String.format("Base PCAP
data path. Default is '%s'", CliConfig.BASE_PATH_DEFAULT)));
+options.addOption(newOption("baseOutputPath", true,
String.format("Query result output path. Default is '%s'",
CliConfig.BASE_OUTPUT_PATH_DEFAULT)));
+options.addOption(newOption("startTime", true, "Packet start time
range. Default is '0'"));
--- End diff --
Can we specify a date format to use to parse the start and end times? I
suspect that's a useful piece of sugar as it would require people to not have
to look up the epoch time translation every time they want to run this for a
time range. I'd just pass an arg with the date format to use. If it's not
specified, then assume epoch time.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #159: METRON-237 Remove metron-ui from the code base
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/159 If we're going to remove metron-ui, which I completely support btw, we are going to need to trim down the sections in the LICENSE around metron-ui components. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #156: METRON-235 Expose filtering capability f...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/156#discussion_r67512583
--- Diff:
metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/query/PcapCliTest.java
---
@@ -0,0 +1,275 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.pcap.query;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.metron.common.Constants;
+import org.apache.metron.common.system.Clock;
+import org.apache.metron.common.utils.timestamp.TimestampConverters;
+import org.apache.metron.pcap.filter.fixed.FixedPcapFilter;
+import org.apache.metron.pcap.filter.query.QueryPcapFilter;
+import org.apache.metron.pcap.mr.PcapJob;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mock;
+import org.mockito.Mockito;
+import org.mockito.MockitoAnnotations;
+
+import java.io.BufferedOutputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.PrintStream;
+import java.nio.charset.StandardCharsets;
+import java.text.ParseException;
+import java.text.SimpleDateFormat;
+import java.util.Arrays;
+import java.util.Date;
+import java.util.EnumMap;
+import java.util.List;
+
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.junit.Assert.assertThat;
+import static org.mockito.Matchers.*;
+import static org.mockito.Mockito.when;
+
+public class PcapCliTest {
+
+ @Mock
+ private PcapJob jobRunner;
+ @Mock
+ private ResultsWriter resultsWriter;
+ @Mock
+ private Clock clock;
+
+ @Before
+ public void setup() {
+MockitoAnnotations.initMocks(this);
+ }
+
+ @Test
+ public void runs_fixed_pcap_filter_job_with_default_argument_list()
throws Exception {
+String[] args = {
+"fixed",
+"-start_time", "500",
--- End diff --
I thought these were long args now, how does - work as opposed to --?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #160: METRON-233: Expose Kafka producer and co...
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/160 METRON-233: Expose Kafka producer and consumer configs for parser topologies We need the ability to tune KafkaSpout and KafkaWriter properties in a more granular way. Currently we do not expose more than a few parameters when creating the KafkaProducer in the KafkaWriter and do not expose almost any of the KafkaSpout configurations. This PR should allow for the KafkaWriter to have arbitrary KafkaProducer configs exposed in the zookeeper config and have the full range of KafkaSpout configs be exposed from the CLI used to start parser topologies. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron kafka_writer_config Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/160.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #160 commit 3eebff1f4c016b33d90fd2d0cfbacbe1125dc209 Author: cstella <ceste...@gmail.com> Date: 2016-06-17T17:47:56Z METRON-233: Expose Kafka producer and consumer configs for parser topologies --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #162: METRON-239: NOOP Bulk Message Writer
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/162 METRON-239: NOOP Bulk Message Writer We should have a NOOP Writer that will sleep at random. This will help differentiate write slowdowns between index misconfiguration and storm/kafka misconfiguration. It should be configurable from the object as well as from zookeeper. * In zookeeper, the "noopLatency" field should be a string defining the latency. * On the object, there should be a "withLatency" function which allows you to define the latency. The latency is defined via string in one of two modes: * An integer which provides a fixed latency * A comma separated range (i.e. "10, 20") which allows for a random latency between the bounds. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron noop_writer Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/162.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #162 commit 93170002ba3c10c82cacc69597082bfb9cea21f8 Author: cstella <ceste...@gmail.com> Date: 2016-06-20T15:37:21Z METRON-239: NOOP Bulk Message Writer --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #164: METRON-244 Added Documentation
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/164 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #156: METRON-235 Expose filtering capability for PCAP...
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/156 +1 from me --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #170: METRON-253: Reloading parser configurati...
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/170 METRON-253: Reloading parser configurations from zookeeper doesn't call init() This results in a NPE if you do not specify an output column (this output column gets inferred as part of init()) To validate this: * Get a running topology * Run `/usr/metron/0.2.0BETA/bin/zk_load_configs.sh -m PUSH -i /usr/metron/0.2.0BETA/config/zookeeper -z node1:2181` * Wait a full 2 minutes * Check the `parser_error` topic via the kafka console consumer and ensure you don't see any weird NPEs The problem here is that init() is not getting run on the sensor parser config, so some syntactic sugar in the FieldTransformer isn't getting updated. Specifically, if you only specify `input` and not `output`, then it will NPE. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron yaf_npe Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/170.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #170 commit b0d391c143e3f14a16cadcac6ab91e6863b1be8d Author: cstella <ceste...@gmail.com> Date: 2016-06-22T20:27:31Z METRON-253: Reloading parser configurations from zookeeper doesn't call init() --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #161: Develop
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/161 So, a couple of things. Please refactor this into separate PRs associated with JIRAs. As of now, it's just impossible to review and we don't really know what we're getting. It looks like a bunch of parsers, so maybe one JIRA per parser? Also, name the PRs with the JIRA name as the first part of the name. For instance, if your JIRA is METRON-123, then PR should be METRON-123: description. Thanks for the contribution, though! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #158: METRON-219 Create Default Metron Dashboard for ...
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/158 got my +1 after that --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #163: METRON-241: Update revision from 0.1BETA to 0.2...
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/163 This worked on `quick-dev-platform`. Data from the default sensors made it to the new kibana dashboard. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #142: METRON-204: Field Transformation Domain Specifi...
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/142 I have modified the squid parser to use the MTL transformer to extract the subdomain-less hostname rather than using grok. So, in order to test this, you should be able to * create the `squid` kafka topic * run the squid parser * pump data from the squid logs through using the console producer * Look at the `enrichment` kafka topic and verify the messages coming out have * `url` : the full URL * `full_hostname` : the full hostname of the URL * `domain_without_subdomains` : the domain without subdomains This is an improvement of the original functionality since it only stripped off the `www` if it exists. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #172: METRON-255: Expose the IPProtocolTransfo...
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/172 METRON-255: Expose the IPProtocolTransformer as a function in the Stellar query language We have a TransformationFunction to map IANA protocol numbers to normalized strings. This should be exposed in the query language. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron protocol_transformer_to_stellar Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/172.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #172 commit d1a9fba498ce29362b5ae6da088b34617b6ca939 Author: cstella <ceste...@gmail.com> Date: 2016-06-23T18:45:00Z Adding protocol to name function --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #169: METRON-252 Metron Dashboard Can Be Broken By Ad...
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/169 +1, spun up in vagrant --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #175: METRON-243 Ensure templates are installed on al...
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/175 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #173: METRON-256: Make the PCap values returne...
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/173 METRON-256: Make the PCap values returned ordered by timestamp Right now we are not specifying the partitioner for the MR job to filter PCaps as part of PCap retrieval, so we are getting back packets in the wrong order. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron pcap_single_reducer Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/173.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #173 commit 808f303e9cdb3145ed8b267b7046f6647c7fd5c4 Author: cstella <ceste...@gmail.com> Date: 2016-06-23T19:38:43Z Make the PCap values returned ordered by timestamp --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #159: METRON-237 Remove metron-ui from the code base
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/159 +1, looks good --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #156: METRON-235 Expose filtering capability f...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/156#discussion_r67512176
--- Diff:
metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/PcapCli.java
---
@@ -0,0 +1,168 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.pcap.query;
+
+import org.apache.commons.cli.ParseException;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.metron.common.system.Clock;
+import org.apache.metron.common.utils.timestamp.TimestampConverters;
+import org.apache.metron.pcap.filter.fixed.FixedPcapFilter;
+import org.apache.metron.pcap.filter.query.QueryPcapFilter;
+import org.apache.metron.pcap.mr.PcapJob;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+public class PcapCli {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(PcapCli.class);
+ private final PcapJob jobRunner;
+ private final ResultsWriter resultsWriter;
+ private final Clock clock;
+
+ public static void main(String[] args) {
+int status = new PcapCli(new PcapJob(), new ResultsWriter(), new
Clock()).run(args);
+System.exit(status);
+ }
+
+ public PcapCli(PcapJob jobRunner, ResultsWriter resultsWriter, Clock
clock) {
+this.jobRunner = jobRunner;
+this.resultsWriter = resultsWriter;
+this.clock = clock;
+ }
+
+ public int run(String[] args) {
+if (args.length < 1) {
--- End diff --
This is addressed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #171: METRON-254 pcap inspector emits fields t...
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/171 METRON-254 pcap inspector emits fields that are named based on the enum, rather than the standardized field names PcapInspector utility prints field names that are enum names (i.e. SRC_ADDR) as opposed to the standard field names (ip_src_addr) Check this by running some pcap data through (follow steps 1-7 from https://github.com/apache/incubator-metron/pull/156) You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron pcap_inspector_naming Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/171.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #171 commit 379bb0b274352f444d203747d9162010122cdf41 Author: cstella <ceste...@gmail.com> Date: 2016-06-23T18:10:59Z Changing the inspector to output field name, not enum name. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: [METRON-25] Create Bro Plugin to Se...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/17#issuecomment-180045345 I like the bro plugin solution. The one comment I have is for bro-plugin-kafka/scripts/init.bro Is there a way to pull the kafka broker list and topic name from something less hard-coded..like a config file? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-35 Implement threat intellig...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/22#issuecomment-183747542 Of course, I have attached a design doc to the [JIRA](https://issues.apache.org/jira/browse/METRON-35) . This is really a single feature as leaving out any part will leave the whole feature nonfunctional. It can seem a bit complex, but it fits within the overall architecture built for the enrichments. I detailed this and how it fits within the overall architecture in the design doc. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-35 Implement threat intellig...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/22#issuecomment-183613907 I want to point out a couple of other things this PR provides that aren't strictly associated with the feature above, but are general cleanup tasks: * Removed lingering hbase-site.xml which have a bad habit of finding their way onto the classpath and confusing HBase in integration tests * The split of integration tests (defined as a test that ends with "IntegrationTest") into the integration-test maven lifecycle phase * Using the shade maven plugin to relocate our guava dependency so that we can use a more recent version of Guava than 12 (which is the most recent that HBase will allow due to google's habit of aggressive removal of deprecated code). This comes up when running HBase in minicluster mode as well as in situations when running bolts which have to package the hbase-client. * General cleanup of the build to use the version properties instead of hard coding different versions of common components (e.g. hbase-client, storm-core, etc.) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-35 Implement threat intellig...
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/22 METRON-35 Implement threat intelligence message enrichment Create the infrastructure to * Bulk ingest threat intelligence feeds from CSV and Stix data sources into HBase * Enrich messages who have fields which match the threat intelligence data in HBase * Create the infrastructure to remove unused threat intelligence data * Augment the Packet capture topology to incorporate a malicious IP threat intel tagger The tagging infrastructure much meet the following criteria: * They are downstream of the enrichments * The threat intelligence bolts execute in parallel with a similar architecture as the enrichments (i.e. split and join). You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron Threat_Intel_Feeds Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/22.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22 commit 5cf5409472d9557f7725ad14a8bcca3663c364aa Author: cstella <ceste...@gmail.com> Date: 2016-02-03T21:30:13Z Added ThreatIntelBulkLoader commit 77105eb645dd357d512aa1d52e9d28e3641003f3 Author: cstella <ceste...@gmail.com> Date: 2016-02-04T16:00:16Z updating threat intel loader. commit 4fcaebcdc38cbf56df89137883c92725e80a88e6 Author: cstella <ceste...@gmail.com> Date: 2016-02-04T16:40:44Z Adding shell script to execute the threat intel feeds. commit 0d390fc0d86af24976649828a8853aec10ab9b0c Author: cstella <ceste...@gmail.com> Date: 2016-02-03T21:30:13Z Added ThreatIntelBulkLoader commit 8256e22f679896c18df8cbfc2dd0bc67a7718b32 Author: cstella <ceste...@gmail.com> Date: 2016-02-04T16:00:16Z updating threat intel loader. commit e5aeb99fb29da3d00eabe53252d88a3345d5e34a Author: cstella <ceste...@gmail.com> Date: 2016-02-04T16:40:44Z Adding shell script to execute the threat intel feeds. commit cfcd709bbbef3e24a5c75b41d07beae9934fe843 Author: cstella <ceste...@gmail.com> Date: 2016-02-04T16:52:37Z Merge branch 'Threat_Intel_Feeds' of github.com:cestella/incubator-metron into Threat_Intel_Feeds commit 5ca646a94f91ec6745abda8fe27a585f1a15904e Author: cstella <ceste...@gmail.com> Date: 2016-02-05T22:31:11Z Moving around some components to common, refactoring some dependencies to allow hbase integration tests in Metron-DataLoads, Implemented the Leastrecentlyusedevictor with bloom filters, integration tested ThreatIntelBulkLoader, Create MR job to evict not recently used keys. commit b7721d375c79e0380d0799ad895faa8b44546e76 Author: cstella <ceste...@gmail.com> Date: 2016-02-05T22:31:22Z Moving around some components to common, refactoring some dependencies to allow hbase integration tests in Metron-DataLoads, Implemented the Leastrecentlyusedevictor with bloom filters, integration tested ThreatIntelBulkLoader, Create MR job to evict not recently used keys. commit 6e026600e41e766a4af0e8c0caa0dc2c882d0bd9 Author: cstella <ceste...@gmail.com> Date: 2016-02-08T18:37:15Z Adding uni ttests for the bulk load/delete jobs. commit 32b198cd241a296f0f1c90cbcdbdb2bcaa3e9dd6 Author: cstella <ceste...@gmail.com> Date: 2016-02-08T19:17:40Z Merge branch 'master' into Threat_Intel_Feeds commit 5c0283c09217f29863ec75c49fd32b420d4e970c Author: cstella <ceste...@gmail.com> Date: 2016-02-09T17:52:02Z Updating to add new extractor, Stix extractor commit 110ed867a0ba7ed638fab7eeb99ffe5e03dcb17e Author: cstella <ceste...@gmail.com> Date: 2016-02-09T18:05:51Z Added test for stix extractor. commit 3cc67d58c08ef8b7cbe2d360512bdfa968e2888e Author: cstella <ceste...@gmail.com> Date: 2016-02-09T20:01:49Z Changed the bloom filter persistent access tracker to use HBase instead of HDFS commit d49496dcb34208fdf997c01a50379ef297a9f3e4 Author: cstella <ceste...@gmail.com> Date: 2016-02-09T20:21:58Z Updating poms to allow more memory. commit c46b4c5b2cd816e50bda050fa51c0e6b28fcf3c2 Author: cstella <ceste...@gmail.com> Date: 2016-02-09T23:15:51Z we really need to stop shipping hbase-site.xmls around. commit 920223ab2c39e834fddea18353997111d8693488 Author: cstella <ceste...@gmail.com> Date: 2016-02-10T20:18:49Z Made HBase Bolt more adaptable. commit 580257e27b917bd029eecab49a3b6b8aac375fde Author: cstella <ceste...@gmail.com> Date: 2016-02-10T20:27:00Z Merge branch 'master' into Threat_Intel_Feeds commit 560877b6c29903fd80b23cb846176dca801336dc Author: cstella <ceste...@gmail.com> Date: 2016-02-10T20:50:51Z HBaseBolt was so wrong. commit 5221eb9d9f4bef6cf580efbb6
[GitHub] incubator-metron pull request: Metron 22 Adding back the HBase bol...
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/16 Metron 22 Adding back the HBase bolt to the pcap topology In addition to adding the hbase bolt back into the pcap topology, the beginnings of an integration test framework was created in Metron-Topologies. Please see PcapIntegrationTest for an example. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron METRON-22 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/16.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #16 commit 28f75bdf848018c16b32b16adea184ac85d3fd05 Author: cstella <ceste...@gmail.com> Date: 2016-01-27T15:22:01Z Updating gitignore to not include intellij cruft. commit eaa3514a317723fa0b9fd40e155796d07afc3121 Author: cstella <ceste...@gmail.com> Date: 2016-01-28T17:43:22Z First cut at integration testing framework with in memory Flux and elastic search components. Also, an integration test for the PCap topology. commit f3c66bd206fcf03e839c5bbec89fbae62f5ffe7d Author: cstella <ceste...@gmail.com> Date: 2016-01-29T15:40:05Z Updating with HBase bolt added back to Flux PCap topology. commit 2c9847e315e8234be720691d92e55d72bb108240 Author: cstella <ceste...@gmail.com> Date: 2016-01-29T16:09:00Z Unit test for PCAP topology with hbase is now functional. commit b5bba8bf668250ee0f91860a2903037cbd273604 Author: cstella <ceste...@gmail.com> Date: 2016-01-29T19:17:22Z Updated pcap remote flux code. commit c6d5e1c27a0ae480ff9c7abac73ca94afd260abb Author: cstella <ceste...@gmail.com> Date: 2016-01-27T15:22:01Z Updating gitignore to not include intellij cruft. commit eb144f78fa06e8d5a536495665ae8e3e4972c963 Author: cstella <ceste...@gmail.com> Date: 2016-01-28T17:43:22Z First cut at integration testing framework with in memory Flux and elastic search components. Also, an integration test for the PCap topology. commit f92e8205a21ea735e002f43bafb1f64273df71a5 Author: cstella <ceste...@gmail.com> Date: 2016-01-29T15:40:05Z Updating with HBase bolt added back to Flux PCap topology. commit f17e82b4a51da4c8aa25fcf569778dfb59c82012 Author: cstella <ceste...@gmail.com> Date: 2016-01-29T16:09:00Z Unit test for PCAP topology with hbase is now functional. commit 1588edb675dccb8bbaa88a352f8b2c2f69cb4cf8 Author: cstella <ceste...@gmail.com> Date: 2016-01-29T19:17:22Z Updated pcap remote flux code. commit ab56b9a49902054cba09cb871875540ebba1a4c5 Author: cstella <ceste...@gmail.com> Date: 2016-01-29T20:17:04Z Fixed merge conflict commit 776209f55e75bbce0a1f9cb864b41c8956659e14 Author: cstella <ceste...@gmail.com> Date: 2016-02-01T20:41:59Z Updated ansible scripts to properly create config for pcap topology, updated local and remote topologies to use the kafka zookeeper list format to specify the zookeeper quorum for hbase, and updated the HBaseBolt to accept a kafka-style list instead of a host and port separated. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-50 Ingest threat intel data ...
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/29 METRON-50 Ingest threat intel data from Taxii feeds Ingest Taxii threat intel feeds via a polling service that will write into HBase as an companion to the BulkLoadThreatIntel job. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron Bulk_Load_Enhancement Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/29.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #29 commit ce336c931f0d808e2ed837eaa872787f6573babf Author: cstella <ceste...@gmail.com> Date: 2016-02-18T14:34:35Z Refactored the Extractor framework and bulk load framework to be more generic. commit 24f737dd5b3eee1b7055f617a7d1292b83ffe35b Author: cstella <ceste...@gmail.com> Date: 2016-02-18T18:08:37Z updating threat intel converters. commit bd1145ebdd45bbf780ecaf32d27aa4c2af180f03 Author: cstella <ceste...@gmail.com> Date: 2016-02-18T22:58:55Z Merging master into branch and adding licenses. commit 03dd47c4735901d411fb40aebd201bfb62244966 Author: cstella <ceste...@gmail.com> Date: 2016-02-22T20:41:34Z Fixing Taxii feeds to function properly. commit 70e34c61345ee9c8ecffa9e3954185c3e2eab975 Author: cstella <ceste...@gmail.com> Date: 2016-02-22T20:44:27Z Merge branch 'master' into Bulk_Load_Enhancement commit c007d583b383fe4f19fb4e42c1b85a9567b2a03d Author: cstella <ceste...@gmail.com> Date: 2016-02-23T01:10:13Z Updating to include integration test and some refactoring of some code into their own module specifically for unit testing. commit fd41f92a800569112079df6ee08d01bbd1779ba3 Author: cstella <ceste...@gmail.com> Date: 2016-02-24T02:38:34Z Correcting classpath issues and adding script that works against soltra commit 8058388ad619b9ba0c5531fdc3e94e845ab82a93 Author: cstella <ceste...@gmail.com> Date: 2016-02-24T02:49:15Z Merge branch 'master' into Bulk_Load_Enhancement commit 8cb970669d8ade72e7be799a10bbb4d7ffa012b5 Author: cstella <ceste...@gmail.com> Date: 2016-02-24T02:58:58Z UPdating licensing issues. commit a9e12e867b2436b4abf5821aaa1c3c03974a4ce7 Author: cstella <ceste...@gmail.com> Date: 2016-02-24T21:22:07Z Merge branch 'master' into Bulk_Load_Enhancement --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: Configure Cobertura plugin to produ...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/21#issuecomment-184820606 This looks good. Can you create a JIRA for it and I'll merge it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-37: Configure Cobertura Plug...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/23#issuecomment-184857855 As promised, +1 to this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-59 Automated Deployment of M...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/41#issuecomment-194475475 Ok +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: Metron 98 - Travis does not fail a ...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/69#issuecomment-206891621 +1 and I'm itching to get travis back and in working order. Just for posterity, the integration tests were failing and not reporting the failure back up to travis. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-95: Create data purging scri...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/70#issuecomment-207453329 I looked at it. Looks good to me. +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-93: Generalize the HBase thr...
Github user cestella commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/66#discussion_r58729427 --- Diff: metron-streaming/Metron-Common/pom.xml --- @@ -34,8 +34,11 @@ Metron Kraken Repository https://raw.github.com/opensoc/kraken/mvn-repo + + + --- End diff -- the newlines? Nope, I'll remove them. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-108 Created fast packet capt...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/73#issuecomment-208532061 It's unfortunate that we are unable to get this to work with singlenode vagrant. In lieu of that, can we have at least a utility and some sample data to publish to kafka so that we can test downstream processes in singlenode vagrant? Also, as we talked about earlier, please add a utility class and unit test showing the appropriate way to add headers to the raw data. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-108 Created fast packet capt...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/73#issuecomment-208544467 So, since this touches the ansible stuff, could we please get someone else to verify that singlenode vagrant isn't broken in a too-subtle-for-me-to-detect way. If that's ok and there's a followup PR with the testing utility, then I'm +1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-108 Created fast packet capt...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/73#issuecomment-208532922 Got it, makes sense. This looks great, btw. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-70 Create service wrapper fo...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/47#issuecomment-197539298 This looks good to me, but I'm going to defer to @merrimanr to give it a second glance since he's in that code right now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-57 Added Snort Community rul...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/34#issuecomment-193823068 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-60 Created service managemen...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/37#issuecomment-193815074 Looks like deployment/roles/snort/files/snortrules-snapshot-2962.tar.gz is giving an apache-rat license violation. Can you fix and update? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-62 Add writing to enriched d...
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/39 METRON-62 Add writing to enriched data back to the enrichment topology When we rearchitected the topologies to use flux, we never added back in the HDFS writing. This needs to be done. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron hdfs_writer Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/39.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #39 commit c9ccc1e6aadb41372769220925f3f7772a4f1b01 Author: cstella <ceste...@gmail.com> Date: 2016-03-04T19:14:56Z Get this branch back to stable. commit 8f622eb1836d27e9868a5b6aeae10adfee39fdbb Author: cstella <ceste...@gmail.com> Date: 2016-03-04T23:54:16Z Creating the HDFSWriter. This includes unit tests and the updated test topology. commit 0e5d33f23075b2f6ae7d3342712c0a2c449ade6b Author: cstella <ceste...@gmail.com> Date: 2016-03-07T16:39:58Z Enrichment integration test fixes, missing license headers, remote topology updates, oh my commit fa759f09964387a5bfc5b5c2e59303fd157bb01e Author: cstella <ceste...@gmail.com> Date: 2016-03-08T17:15:59Z merging master in and also fixing the SourceCOnfigUtils --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-58: Remediate Deployment Int...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/36#issuecomment-193502076 Ok +1 from me :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: Metron-56 Create unified enrichment...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/33#issuecomment-190815542 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-61: Allow selection of topol...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/38#issuecomment-193971678 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: Metron 86 - SolrWriter
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/67#discussion_r58221773
--- Diff:
metron-streaming/Metron-Solr/src/main/java/org/apache/metron/writer/solr/MetronSolrClient.java
---
@@ -0,0 +1,83 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.writer.solr;
+
+import org.apache.log4j.Logger;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.impl.CloudSolrClient;
+import org.apache.solr.client.solrj.request.QueryRequest;
+import org.apache.solr.common.params.CollectionParams;
+import org.apache.solr.common.params.ModifiableSolrParams;
+import org.apache.solr.common.util.NamedList;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+public class MetronSolrClient extends CloudSolrClient {
+
+ private static final Logger LOG =
Logger.getLogger(MetronSolrClient.class);
+
+
+ public MetronSolrClient(String zkHost) {
+super(zkHost);
+ }
+
+ public void createCollection(String name, int numShards, int
replicationFactor) {
+if(!listCollections().contains(name)) {
+ try {
+request(getCreasteCollectionsRequest(name, numShards,
replicationFactor));
+ } catch (SolrServerException | IOException e) {
+LOG.error(e, e);
+ }
+}
+ }
+
+ public QueryRequest getCreasteCollectionsRequest(String name, int
numShards, int replicationFactor) {
+ModifiableSolrParams params = new ModifiableSolrParams();
+params.set("action", CollectionParams.CollectionAction.CREATE.name());
+params.set("name", name);
+params.set("numShards", numShards);
+params.set("replicationFactor", replicationFactor);
+params.set("collection.configName", name);
+QueryRequest request = new QueryRequest(params);
+request.setPath("/admin/collections");
+return request;
+ }
+
+ public List listCollections() {
+List collections = new ArrayList<>();
+try {
+ NamedList response = request(getListCollectionsRequest(),
null);
+ collections = (List) response.get("collections");
+} catch (SolrServerException | IOException e) {
+ LOG.error(e, e);
--- End diff --
Can we bubble this exception up?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: Metron 86 - SolrWriter
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/67#discussion_r58223216
--- Diff:
metron-streaming/Metron-Common/src/main/java/org/apache/metron/bolt/ConfiguredBolt.java
---
@@ -77,15 +82,37 @@ public void childEvent(CuratorFramework client,
PathChildrenCacheEvent event) th
} catch (Exception e) {
throw new RuntimeException(e);
}
+
+long timeElapsed = 0;
--- End diff --
We should load the configs directly from zookeeper synchronously rather
than do a while, sleep until they get loaded asynchronously.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: Metron 86 - SolrWriter
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/67#discussion_r58221656
--- Diff:
metron-streaming/Metron-Solr/src/main/java/org/apache/metron/writer/solr/MetronSolrClient.java
---
@@ -0,0 +1,83 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.writer.solr;
+
+import org.apache.log4j.Logger;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.impl.CloudSolrClient;
+import org.apache.solr.client.solrj.request.QueryRequest;
+import org.apache.solr.common.params.CollectionParams;
+import org.apache.solr.common.params.ModifiableSolrParams;
+import org.apache.solr.common.util.NamedList;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+public class MetronSolrClient extends CloudSolrClient {
+
+ private static final Logger LOG =
Logger.getLogger(MetronSolrClient.class);
+
+
+ public MetronSolrClient(String zkHost) {
+super(zkHost);
+ }
+
+ public void createCollection(String name, int numShards, int
replicationFactor) {
+if(!listCollections().contains(name)) {
+ try {
+request(getCreasteCollectionsRequest(name, numShards,
replicationFactor));
+ } catch (SolrServerException | IOException e) {
+LOG.error(e, e);
--- End diff --
Should we be swallowing this exception? Seems like this should bubble up.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: Metron 86 - SolrWriter
Github user cestella commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/67#discussion_r58223047 --- Diff: deployment/inventory/multinode-vagrant/group_vars/all --- @@ -49,17 +49,23 @@ snort_version: "2.9.8.0-1" snort_alert_csv_path: "/var/log/snort/alert.csv" #data directories -zookeeper_data_dir: "/newdir/hadoop/zookeeper" -namenode_checkpoint_dir: "/newdir/hadoop/hdfs/namesecondary" -namenode_name_dir: "/newdir/hadoop/hdfs/namenode" -datanode_data_dir: "/newdir/hadoop/hdfs/data" -journalnode_edits_dir: "/newdir/hadoop/hdfs/journalnode" -nodemanager_local_dirs: "/newdir/hadoop/yarn/local" -timeline_ldb_store_path: "/newdir/hadoop/yarn/timeline" -timeline_ldb_state_path: "/newdir/hadoop/yarn/timeline" -nodemanager_log_dirs: "/newdir/hadoop/yarn/log" -jhs_recovery_store_ldb_path: "/newdir/hadoop/mapreduce/jhs" -storm_local_dir: "/newdir/hadoop/storm" -kafka_log_dirs: "/newdir/kafka-log" -elasticsearch_data_dir: "/newdir1/elasticsearch" +#zookeeper_data_dir: "/newdir/hadoop/zookeeper" --- End diff -- These should be removed, not commented out. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: Metron 86 - SolrWriter
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/67#discussion_r58223640
--- Diff: metron-streaming/Metron-Elasticsearch/pom.xml ---
@@ -0,0 +1,202 @@
+
+
+
+http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+org.apache.metron
+Metron-Streaming
+0.1BETA
+
+Metron-Elasticsearch
+
+UTF-8
+
UTF-8
+
+
+
+org.apache.metron
+Metron-Common
+${project.parent.version}
+
+
+org.elasticsearch
+elasticsearch
+${global_elasticsearch_version}
+
+
+org.apache.storm
+storm-core
+${global_storm_version}
+provided
+
+
+servlet-api
+javax.servlet
+
+
+log4j-over-slf4j
+org.slf4j
+
+
+
+
+org.apache.metron
+Metron-Testing
+${project.parent.version}
+test
+
+
+org.apache.metron
+Metron-Topologies
+${project.parent.version}
+
+
+org.mockito
+mockito-all
+1.9.5
--- End diff --
We should make this version a global variable
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: Metron 86 - SolrWriter
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/67#discussion_r58221412
--- Diff:
metron-streaming/Metron-Solr/src/main/java/org/apache/metron/writer/solr/MetronSolrClient.java
---
@@ -0,0 +1,83 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.writer.solr;
+
+import org.apache.log4j.Logger;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.impl.CloudSolrClient;
+import org.apache.solr.client.solrj.request.QueryRequest;
+import org.apache.solr.common.params.CollectionParams;
+import org.apache.solr.common.params.ModifiableSolrParams;
+import org.apache.solr.common.util.NamedList;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+public class MetronSolrClient extends CloudSolrClient {
+
+ private static final Logger LOG =
Logger.getLogger(MetronSolrClient.class);
+
+
+ public MetronSolrClient(String zkHost) {
+super(zkHost);
+ }
+
+ public void createCollection(String name, int numShards, int
replicationFactor) {
+if(!listCollections().contains(name)) {
+ try {
+request(getCreasteCollectionsRequest(name, numShards,
replicationFactor));
+ } catch (SolrServerException | IOException e) {
+LOG.error(e, e);
+ }
+}
+ }
+
+ public QueryRequest getCreasteCollectionsRequest(String name, int
numShards, int replicationFactor) {
+ModifiableSolrParams params = new ModifiableSolrParams();
+params.set("action", CollectionParams.CollectionAction.CREATE.name());
--- End diff --
Maybe make these keys a constant?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-93: Generalize the HBase thr...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/66#discussion_r58375098
--- Diff:
metron-streaming/Metron-Common/src/test/java/org/apache/metron/enrichment/EnrichmentConfigTest.java
---
@@ -0,0 +1,253 @@
+package org.apache.metron.enrichment;
+
+import org.apache.metron.Constants;
+import org.apache.metron.domain.SourceConfig;
+import org.apache.metron.utils.JSONUtils;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class EnrichmentConfigTest {
+ @Test
+ public void testThreatIntel() throws Exception {
+/*
+{
+ "index": "bro",
--- End diff --
Normally I'd agree, but those comments are there because multiline strings
in java are lacking, so if changes to the JSON structure happen, the changes
generally happen to the comment and then the string is replaced, letting the
IDE handle escaping quotes. One thing that we could consider is incorporating
something like https://github.com/benelog/multiline, which i've seen in other
projects for these kinds of situations.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-95: Create data purging scri...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/68#discussion_r58473310
--- Diff: deployment/roles/metron_streaming/defaults/main.yml ---
@@ -34,3 +39,6 @@ storm_topologies:
- "{{ metron_directory }}/config/topologies/yaf/remote.yaml"
- "{{ metron_directory }}/config/topologies/pcap/remote.yaml"
- "{{ metron_directory }}/config/topologies/enrichment/remote.yaml"
+hdfs_retention_days: 30
+hdfs_bro_purge_cronjob: "yarn jar {{ metron_directory }}/lib/{{
metron_dataloads_jar }} org.apache.metron.dataloads.bulk.HDFSDataPruner -f {{
hdfs_url }} -g '/apps/metron/enrichment/indexed/bro_doc/*enrichment-*' -s
$(date -d '{{ hdfs_retention_days }} days ago' +%m/%d/%Y) -n 1 >>
/var/log/bro-purge/cron-bro-purge.log 2>&1"
--- End diff --
Not quite sure what shadow the command line options means. Yeah, some
people do explicitly call `yarn jar foo.jar main.class args`, but I suspect
they'd rather be calling `my_script_name.sh args` every single time. Before we
had this for the zookeeper config utility, I'd grep the ansible scripts every
time I needed to update the sourceconfigs in zookeeper. When I needed to
submit flux topologies (similar pattern) I'd have to grep ansible to remember
the Flux command (similarly obtuse as the yarn jar command). I don't find that
to be very good user experience. Another reason is that I can put
$METRON_HOME/bin on the path and have the script autocomplete, whereas I'm
never getting that main.class autocompleted. :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: Updating missing licenses from metr...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/64#issuecomment-203428699 * Removed extraneous LICENSE files and COPYING files * Fixed the verbiage for the cisco contribution in the LICENSE file * Added link to the license files for the 3rd party bundled components under metron-ui/lib/public/vendor in the LICENSE file * Added inline licenses in the LICENSE file for the * fontawesome fonts * effective_tld_names.dat data file * bundled portions of kibana (where you saw the app.js without the header file) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-93: Generalize the HBase thr...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/66#discussion_r58468314
--- Diff:
metron-streaming/Metron-Common/src/test/java/org/apache/metron/enrichment/EnrichmentConfigTest.java
---
@@ -0,0 +1,253 @@
+package org.apache.metron.enrichment;
+
+import org.apache.metron.Constants;
+import org.apache.metron.domain.SourceConfig;
+import org.apache.metron.utils.JSONUtils;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class EnrichmentConfigTest {
+ @Test
+ public void testThreatIntel() throws Exception {
+/*
+{
+ "index": "bro",
--- End diff --
Ok, fair enough, I'm adding the multiline string utility for the unit tests
and moving the config JSON to there, so we don't have to maintain them in 2
different places. This is a similar pattern as they use in DataFu for their
unit tests.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-95: Create data purging scri...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/68#discussion_r58468994
--- Diff: deployment/roles/metron_streaming/defaults/main.yml ---
@@ -34,3 +39,6 @@ storm_topologies:
- "{{ metron_directory }}/config/topologies/yaf/remote.yaml"
- "{{ metron_directory }}/config/topologies/pcap/remote.yaml"
- "{{ metron_directory }}/config/topologies/enrichment/remote.yaml"
+hdfs_retention_days: 30
+hdfs_bro_purge_cronjob: "yarn jar {{ metron_directory }}/lib/{{
metron_dataloads_jar }} org.apache.metron.dataloads.bulk.HDFSDataPruner -f {{
hdfs_url }} -g '/apps/metron/enrichment/indexed/bro_doc/*enrichment-*' -s
$(date -d '{{ hdfs_retention_days }} days ago' +%m/%d/%Y) -n 1 >>
/var/log/bro-purge/cron-bro-purge.log 2>&1"
--- End diff --
Please make this an actual shell script and put it in an appropriate
src/main/bash, not the direct call to yarn jar.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: METRON-95: Create data purging scri...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/68#discussion_r58469110
--- Diff: metron-streaming/Metron-DataLoads/pom.xml ---
@@ -149,7 +149,12 @@
${project.parent.version}
test
-
+
+org.mockito
+mockito-core
+1.10.16
--- End diff --
Ryan's PR from last week includes mockito as well and I believe there's a
global mockito variable. When you get around to merging master (after ryan's
PR is merged), please make sure that this is referring to the global mockito
version.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: Updating missing licenses from metr...
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/64 Updating missing licenses from metron-ui. This is required for the apache release You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron 0.1BETA_rc6_cleanup Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/64.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #64 commit b44113ad2ca06e8701152f8b069ee4c8d76dfdb1 Author: cstella <ceste...@gmail.com> Date: 2016-03-30T01:25:08Z Updating missing licenses from metron-ui. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-119 Move PCAP infrastructure...
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/93#issuecomment-215694256 @dlyle65535 Definitely agreed, I'll submit a change this morning to remove the warnings. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request: METRON-119 Move PCAP infrastructure...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/93#discussion_r61566951
--- Diff:
metron-deployment/roles/metron_streaming/tasks/metron_topology.yml ---
@@ -16,18 +16,14 @@
#
---
- name: Submit Metron Parser topologies
- command: storm jar {{ metron_directory }}/lib/{{ metron_parsers_jar_name
}} org.apache.storm.flux.Flux --filter {{
metron_parsers_properties_config_path }} --remote {{ item }}
+ command: "{{ metron_directory }}/bin/start_parser_topology.sh {{ item }}"
with_items:
- "{{ storm_parser_topologies }}"
--- End diff --
So, adding pcap to the list of parser topologies won't do it because pcap
has a special script (start_pcap_topology.sh) due to it having a different
config file (all of the parser topologies share the same config). Also, it's
just a different sort of beast than a parser topology (i.e. we don't actually
parse anything, we just take the raw data, slap on a header and put it in HDFS).
That being said, what I think we need to do is start the pcap topology when
pycapa is installed. I'll have to look into where and how to do that in
ansible. If you have any thoughts or suggestions, I'd be all ears. ;)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
