[GitHub] incubator-metron pull request: [METRON-150] Adding WebSphere parse...

2016-05-10 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/115#discussion_r62781383 --- Diff: metron-platform/metron-integration-test/src/main/resources/sample/config/sensors/websphere.json --- @@ -0,0 +1,22

[GitHub] incubator-metron pull request: [METRON-150] Adding WebSphere parse...

2016-05-11 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/115#discussion_r62843136 --- Diff: metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/websphere/GrokWebSphereParserTest.java --- @@ -0,0 +1,115

[GitHub] incubator-metron pull request: [METRON-150] Adding WebSphere parse...

2016-05-11 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/115#issuecomment-218452417 Thanks so much for the contribution. This is going to be great when it gets in! --- If your project is set up for it, you can reply to this email and have

[GitHub] incubator-metron pull request: [METRON-150] Adding WebSphere parse...

2016-05-12 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/115#issuecomment-218772767 Great Job! Have you tested this on the full-dev-vagrant at all? --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] incubator-metron pull request: Metron 146 topology workers

2016-05-12 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/116#issuecomment-218777995 Ah, ok, so this is failing the integration tests because you're not setting the topology.workers property. You can either set it * In [this](

[GitHub] incubator-metron pull request: METRON-151: Remove yum update from ...

2016-05-12 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/117#issuecomment-218843462 Definitely +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] incubator-metron pull request: METRON-155 Added query filtering ca...

2016-05-12 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/119#discussion_r63087348 --- Diff: metron-platform/metron-api/src/main/java/org/apache/metron/pcapservice/PcapReceiverImplRestEasy.java --- @@ -97,6 +97,66 @@ private

[GitHub] incubator-metron pull request: METRON-155 Added query filtering ca...

2016-05-12 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/119#discussion_r63099841 --- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/query/PredicateProcessor.java --- @@ -18,12 +18,20

[GitHub] incubator-metron pull request: METRON-155 Added query filtering ca...

2016-05-12 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/119#discussion_r63099913 --- Diff: metron-platform/metron-common/src/test/java/org/apache/metron/common/query/QueryParserTest.java --- @@ -67,6 +67,7 @@ public void

[GitHub] incubator-metron pull request: METRON-155 Added query filtering ca...

2016-05-12 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/119#discussion_r63100294 --- Diff: metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/filter/PcapFilters.java --- @@ -0,0 +1,41 @@ +/** + * Licensed

[GitHub] incubator-metron pull request: METRON-155 Added query filtering ca...

2016-05-12 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/119#discussion_r63101688 --- Diff: metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/filter/query/QueryPcapFilter.java --- @@ -0,0 +1,74

[GitHub] incubator-metron pull request: METRON-155 Added query filtering ca...

2016-05-12 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/119#discussion_r63102232 --- Diff: metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/mr/PcapJob.java --- @@ -33,36 +33,39 @@ import

[GitHub] incubator-metron pull request: METRON-155 Added query filtering ca...

2016-05-12 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/119#discussion_r63102275 --- Diff: metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/mr/PcapJob.java --- @@ -33,36 +33,39 @@ import

[GitHub] incubator-metron pull request: METRON-155 Added query filtering ca...

2016-05-12 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/119#issuecomment-218894825 On the whole, this is great. Definitely a great feature and an impressive 2nd PR. Thanks for the contribution! Please make sure you didn&#

[GitHub] incubator-metron pull request: METRON-155 Added query filtering ca...

2016-05-12 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/119#discussion_r63105345 --- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/query/PredicateProcessor.java --- @@ -18,12 +18,20

[GitHub] incubator-metron pull request: METRON-155 Added query filtering ca...

2016-05-12 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/119#discussion_r63111447 --- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/query/PredicateProcessor.java --- @@ -18,12 +18,20

[GitHub] incubator-metron pull request: METRON-155 Added query filtering ca...

2016-05-12 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/119#discussion_r63112571 --- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/query/PredicateProcessor.java --- @@ -18,12 +18,20

[GitHub] incubator-metron pull request: METRON-155 Added query filtering ca...

2016-05-12 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/119#discussion_r63114735 --- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/query/PredicateProcessor.java --- @@ -18,12 +18,20

[GitHub] incubator-metron pull request: METRON-155 Added query filtering ca...

2016-05-12 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/119#discussion_r63115005 --- Diff: metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/filter/query/QueryPcapFilter.java --- @@ -0,0 +1,74

[GitHub] incubator-metron pull request: [METRON-150] Adding WebSphere parse...

2016-05-13 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/115#issuecomment-219048736 Ok, tested this. Sorry it took so long (vagrant drama). Ok, so this is what I did to test this. Spin up the full-dev-vagrant: 1. Do a build

[GitHub] incubator-metron pull request: METRON-155 Added query filtering ca...

2016-05-13 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/119#issuecomment-219049457 @nickwallen Agreed we should support BPF. This PR makes the filter pluggable and we already have the query language. We can have a follow-on PR for BPF

[GitHub] incubator-metron pull request: [METRON-150] Adding WebSphere parse...

2016-05-13 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/115#issuecomment-219048777 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] incubator-metron pull request: Added vagrant install hostmanager s...

2016-05-13 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/121#issuecomment-219068764 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] incubator-metron pull request: METRON-132 Monit Integration

2016-05-13 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/118#issuecomment-219158724 +1, looks great! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] incubator-metron pull request: Metron 142 Simplify Parser configur...

2016-05-16 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/120#issuecomment-219424532 This looks good to me. +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] incubator-metron pull request: METRON-155 Added query filtering ca...

2016-05-16 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/119#issuecomment-219425116 +1, got my vote. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] incubator-metron pull request: Specify shorter control_path to avo...

2016-05-20 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/122#issuecomment-220619587 +1 this looks good --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] incubator-metron pull request: METRON-156 Fixed Docs for Vagrant D...

2016-05-20 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/124#discussion_r64049158 --- Diff: metron-deployment/vagrant/full-dev-platform/README.md --- @@ -62,41 +60,46 @@ Now that the hard part is done, start the Metron

[GitHub] incubator-metron pull request: Metron-126 added dfs.replication=1 ...

2016-05-20 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/125#issuecomment-220620508 +1 looks good --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] incubator-metron pull request: METRON-173 Updating the the Apache ...

2016-05-20 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/126#issuecomment-220634820 +1, this looks great --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

2016-05-20 Thread cestella
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/127 METRON-174 Storm consumption of hbase enrichment reference data We should support streaming enrichment data into kafka and writing it out to HBase in a format suitable to be used in

[GitHub] incubator-metron pull request: METRON-156 Fixed Docs for Vagrant D...

2016-05-20 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/124#discussion_r64060254 --- Diff: metron-deployment/vagrant/full-dev-platform/README.md --- @@ -62,41 +60,46 @@ Now that the hard part is done, start the Metron

[GitHub] incubator-metron pull request: Metron 145: Automate Creation of Va...

2016-05-20 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/123#issuecomment-220647462 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] incubator-metron pull request: Metron 173

2016-05-20 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/128#issuecomment-220718342 Metron 173 is already merged. It looks like you have another JIRA (Metron 177) for the typos, please open up another PR with just the Metron-177 changes in

[GitHub] incubator-metron pull request: METRON-178 Expose the filter capabi...

2016-05-20 Thread cestella
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/129 METRON-178 Expose the filter capability in the Parser topologies. Allow users to specify the MessageFilter to use in the parser topologies. We already have this capability, we need to

[GitHub] incubator-metron pull request: METRON-177 Fix typos on the Metron ...

2016-05-23 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/130#issuecomment-220976450 Ok, this looks good, +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] incubator-metron pull request: Metron 173

2016-05-23 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/128#issuecomment-220976859 Since you made a METRON-177 PR, could you please close this one? --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] incubator-metron pull request: METRON-179 Fixed Docs for Vagrant D...

2016-05-23 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/124#issuecomment-220977119 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] incubator-metron pull request: METRON-183 Allow the simple hbase e...

2016-05-23 Thread cestella
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/131 METRON-183 Allow the simple hbase enrichment adapter and simple threat intel adapter to use multiple column families Allow the simple hbase enrichment adapter and simple threat intel

[GitHub] incubator-metron pull request: METRON-178 Expose the filter capabi...

2016-05-24 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/129#discussion_r64495854 --- Diff: metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java --- @@ -59,10 +61,22 @@ public ParserBolt

[GitHub] incubator-metron pull request: METRON-178 Expose the filter capabi...

2016-05-24 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/129#issuecomment-221437141 Yeah, upon further consideration, I think that probably an init method is probably the best thing for MessageFilter. I'll refactor. --- If your proje

[GitHub] incubator-metron pull request: METRON-178 Expose the filter capabi...

2016-05-25 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/129#discussion_r64564714 --- Diff: metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java --- @@ -59,10 +61,22 @@ public ParserBolt

[GitHub] incubator-metron pull request: METRON-100 GeoIP errors out silentl...

2016-05-25 Thread cestella
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/134 METRON-100 GeoIP errors out silently in vagrant When we transitioned from passing the value to the adapters to passing a CacheKey object, we never adjusted the SQL statements. Also

[GitHub] incubator-metron pull request: METRON-180 Enable each component to...

2016-05-25 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/133#issuecomment-221577078 Yeah, this makes sense. Do we need to adjust the `run_ansible_role.sh` to pass in the `--skip-tags` arg as well? --- If your project is set up for it, you

[GitHub] incubator-metron pull request: METRON-184 Fixed 'creates' path to ...

2016-05-25 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/132#issuecomment-221577928 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] incubator-metron pull request: METRON-180 Enable each component to...

2016-05-25 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/133#issuecomment-221611276 Also, is there an ec2.py somewhere that didn't get checked in? --- If your project is set up for it, you can reply to this email and have your reply a

[GitHub] incubator-metron pull request: METRON-186: Create a fieldMapping f...

2016-05-25 Thread cestella
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/136 METRON-186: Create a fieldMapping functionality which allows for parsed fields to be transformed Currently the parsers take care of transforming raw data to the parsed JSON

[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

2016-05-25 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/127#discussion_r64669063 --- Diff: metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java --- @@ -60,7 +93,25 @@ public void prepare

[GitHub] incubator-metron pull request: METRON-152: Add support for ES 2.3....

2016-05-25 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/135#issuecomment-221739471 +1, spun up in single node vagrant and looks good! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] incubator-metron pull request: METRON-186: Create a fieldMapping f...

2016-05-26 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/136#issuecomment-221862988 Yeah, I was thinking about that too. I think transformation is a more descriptive term. You're the 2nd person (in addition to myself) who has made

[GitHub] incubator-metron pull request: METRON-100 GeoIP errors out silentl...

2016-05-26 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/134#discussion_r64742580 --- Diff: metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/jdbc/JdbcAdapter.java --- @@ -38,6 +39,27

[GitHub] incubator-metron pull request: METRON-189: Add the ability to do g...

2016-05-27 Thread cestella
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/138 METRON-189: Add the ability to do global validations on messages passing through the parser. Allow the user to specify field level or message level validations to ensure messages coming

[GitHub] incubator-metron pull request: METRON-180 Enable each component to...

2016-05-27 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/133#issuecomment-222159692 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] incubator-metron pull request: METRON-190: Make start_parser_topol...

2016-05-27 Thread cestella
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/139 METRON-190: Make start_parser_topology.sh more adaptable regarding storm topology configuration Add the ability to specify the message timeout, the max task parallelism, the number of

[GitHub] incubator-metron pull request: METRON-190: Make start_parser_topol...

2016-05-27 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/139#issuecomment-53339 Ok, I verified this on full dev vagrant. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] incubator-metron pull request: METRON-190: Make start_parser_topol...

2016-05-28 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/139#issuecomment-222313782 @james-sirota Right you are! Just updated exposing the numTasks for the spout and the parser bolt. ``` -pnt,--parser_num_tasksParser Num

[GitHub] incubator-metron pull request: METRON-183 Allow the simple hbase e...

2016-05-29 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/131#issuecomment-222363251 The way to validate this is to * Create a second column family on the enrichment HBase table, say `cf1` * Push some enrichment data into the table in

[GitHub] incubator-metron pull request: METRON-183 Allow the simple hbase e...

2016-05-29 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/131#issuecomment-222363326 As it stands, we have the docs for the configs with where the configurations live. I think you are absolutely right that we need to move them closer to the

[GitHub] incubator-metron pull request: METRON-190: Make start_parser_topol...

2016-05-29 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/139#issuecomment-222363512 good catch, I'll make those changes now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as wel

[GitHub] incubator-metron pull request: METRON-189: Add the ability to do g...

2016-05-29 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/138#issuecomment-222363626 Yeah, the documentation situation is a bit incorrect. Documentation for configuration started to be placed where the configuration objects existed, rather

[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

2016-05-29 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/127#issuecomment-222364875 In order to validate this, you can do the following: * Configure a new parser, in this example I'll call it a `user` parser and we'll parse som

[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

2016-05-30 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/127#issuecomment-222452241 @james-sirota Did you push the new parser config to zookeeper via `/usr/metron/0.1BETA/bin/zk_load_configs.sh -m PUSH -z node1:2181 -i /usr/metron/0.1BETA

[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

2016-05-30 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/127#issuecomment-222452433 If you did push the config before trying to start the parser, then please confirm that the `user` topology is in zookeeper via inspecting the output of

[GitHub] incubator-metron pull request:

2016-05-30 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/commit/ab8163bcc64d4b725ad61a5f6d8a74aad812a24a#commitcomment-17664548 You should not need the zookeeper information in those config objects since it will get passed via the

[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

2016-05-30 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/127#issuecomment-222534318 Looks like it can't find the writerClassname field. Are you sure you ran a build from this branch before the deploy? On Mon, May 30, 2016 at

[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

2016-05-31 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/127#issuecomment-222638653 You sure Kafka is still up? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

2016-05-31 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/127#issuecomment-222640686 Try pulling data from that broker using the console consumer --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] incubator-metron pull request #138: METRON-189: Add the ability to do global...

2016-06-01 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/138#discussion_r65462256 --- Diff: metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java --- @@ -165,8 +174,17 @@ public void

[GitHub] incubator-metron pull request #138: METRON-189: Add the ability to do global...

2016-06-01 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/138#discussion_r65462284 --- Diff: metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java --- @@ -140,18 +142,25 @@ public void

[GitHub] incubator-metron pull request #142: METRON-204: Field Transformation Domain ...

2016-06-02 Thread cestella
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/142 METRON-204: Field Transformation Domain Specific Language Similar to the domain specific query language, it would be nice to have a domain specific language for transformations which is

[GitHub] incubator-metron pull request #143: METRON-197: Validation should be the las...

2016-06-02 Thread cestella
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/143 METRON-197: Validation should be the last step in the ParserBolt Right now we are doing the validation prior to the messageFilter. We should only validate the parsed messages which

[GitHub] incubator-metron issue #142: METRON-204: Field Transformation Domain Specifi...

2016-06-03 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/142 I will definitely create a doc on the JIRA about the DSL and would be happy to discuss. It's really just the extraction of the existing transformation functions from the query DSL

[GitHub] incubator-metron issue #142: METRON-204: Field Transformation Domain Specifi...

2016-06-03 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/142 Also, it seems like a big PR, but honestly it's deceptive. There was a lot of file moving and the generated Antlr code inflates the size. --- If your project is set up for it, yo

[GitHub] incubator-metron issue #142: METRON-204: Field Transformation Domain Specifi...

2016-06-03 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/142 I uploaded a design doc to the JIRA. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] incubator-metron pull request #143: METRON-197: Validation should be the las...

2016-06-08 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/143#discussion_r66248395 --- Diff: metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java --- @@ -143,14 +140,14 @@ public void

[GitHub] incubator-metron issue #143: METRON-197: Validation should be the last step ...

2016-06-08 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/143 @merrimanr @james-sirota I tend to agree, validations should happen prior to field transformations since the transformation may affect the validity of the message. I'll adjust

[GitHub] incubator-metron issue #141: Fix unreleased resources

2016-06-08 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/141 +1, I like it --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] incubator-metron pull request #146: METRON-203 General best practice and bug...

2016-06-08 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/146#discussion_r66321208 --- Diff: metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/nonbulk/taxii/TaxiiHandler.java --- @@ -61,346 +63,334

[GitHub] incubator-metron pull request #148: METRON-215: Fixing an NPE in the Message...

2016-06-09 Thread cestella
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/148 METRON-215: Fixing an NPE in the MessageParser Optional.of does not accept nulls. You can merge this pull request into a Git repository by running: $ git pull https://github.com

[GitHub] incubator-metron pull request #147: METRON-208 Add Google Analytics tags to ...

2016-06-09 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/147#discussion_r66463687 --- Diff: site/diff --- @@ -0,0 +1,186 @@ +diff --git a/site/_includes/footer.html b/site/_includes/footer.html --- End diff

[GitHub] incubator-metron issue #147: METRON-208 Add Google Analytics tags to Metron ...

2016-06-09 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/147 +1 looks good to me --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] incubator-metron pull request #149: METRON-217: Found a grabbag of bugs

2016-06-10 Thread cestella
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/149 METRON-217: Found a grabbag of bugs Doing some scale testing, I found a grabbag of bugs: * The elasticsearch writer should allow multiple ES hosts to be passed in via a List. This

[GitHub] incubator-metron issue #149: METRON-217: Found a grabbag of bugs

2016-06-10 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/149 So, that's the other side of the concurrent modification exception, @dlyle65535. Somehow we are modifying the JSONObject message while it's deserializing, like

[GitHub] incubator-metron issue #149: METRON-217: Found a grabbag of bugs

2016-06-10 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/149 Just for those poor souls who are debugging this on the internet, this is the sister exception (from the other thread): > 2016-06-10 13:34:44.687 b.s.util [ERROR] Async loop d

[GitHub] incubator-metron issue #149: METRON-217: Found a grabbag of bugs

2016-06-10 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/149 @dlyle65535 any chance you can run that again with this commit? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] incubator-metron issue #149: METRON-217: Found a grabbag of bugs

2016-06-10 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/149 I also went ahead and added a bit better logging if the CME returns, we should get a sense of what the other threads are doing. Definitely would like to see if you see any CMEs with the

[GitHub] incubator-metron issue #142: METRON-204: Field Transformation Domain Specifi...

2016-06-13 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/142 So, I think the main disadvantage of using groovy or any of the scripting engine integrations is speed and expressibility. Having a targeted domain specific language is going to be

[GitHub] incubator-metron issue #146: METRON-203 General best practice and bug fixes

2016-06-13 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/146 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] incubator-metron pull request #142: METRON-204: Field Transformation Domain ...

2016-06-13 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/142#discussion_r66876932 --- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/query/generated/PredicateLexer.java --- @@ -114,59 +118,74

[GitHub] incubator-metron pull request #142: METRON-204: Field Transformation Domain ...

2016-06-13 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/142#discussion_r66877352 --- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/aggregator/Aggregators.java --- @@ -18,18 +18,21

[GitHub] incubator-metron pull request #142: METRON-204: Field Transformation Domain ...

2016-06-13 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/142#discussion_r66885719 --- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/query/generated/PredicateLexer.java --- @@ -114,59 +118,74

[GitHub] incubator-metron pull request #150: METRON-222: Address ConcurrentModificati...

2016-06-13 Thread cestella
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/150 METRON-222: Address ConcurrentModificationException in BulkMessageWriter You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella

[GitHub] incubator-metron issue #150: METRON-222: Address ConcurrentModificationExcep...

2016-06-14 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/150 I don't see a difference in the clone method vs the method currently implemented. I think we should evaluate in a follow-on JIRA whether any of the clones are needed. It seems that

[GitHub] incubator-metron pull request #151: METRON-223: Invalid and Erroneous messag...

2016-06-14 Thread cestella
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/151 METRON-223: Invalid and Erroneous messages should go to kafka for further analysis in the Parser Topology Right now we send error messages and invalid messages to separate streams in

[GitHub] incubator-metron issue #144: METRON-211 Enable creation of new Kafka topics ...

2016-06-14 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/144 +1 this looks good --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] incubator-metron issue #145: METRON-212: Allow additional Elasticsearch temp...

2016-06-14 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/145 +1 looks good --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] incubator-metron pull request #152: METRON-228: Fixing NPE when enrichment c...

2016-06-15 Thread cestella
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/152 METRON-228: Fixing NPE when enrichment config does not exist. Because we are pulling the batch size from the enrichment sensor config, if it does not exist then it will throw a NPE. You

[GitHub] incubator-metron pull request #154: METRON-230: Bro parser should throw exce...

2016-06-16 Thread cestella
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/154 METRON-230: Bro parser should throw exception Right now, if an invalid message comes to the bro parser, it returns null, which is interpreted as no messages from the parser. INstead, we

[GitHub] incubator-metron pull request #155: METRON-231: Snort parser should throw ex...

2016-06-16 Thread cestella
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/155 METRON-231: Snort parser should throw exception Snort parser returns null when we should be throwing an exception in the event of an improperly formatted message. An exception will

[GitHub] incubator-metron issue #142: METRON-204: Field Transformation Domain Specifi...

2016-06-16 Thread cestella
Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/142 I have modified the squid parser to use the MTL transformer to extract the subdomain-less hostname rather than using grok. So, in order to test this, you should be able to * create

<    1   2   3   4   5   6   7   8   9   10   >