[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/530
  
@mmiklavc does that mean we can go back to 3.0.1?  Or should?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #685: METRON-1087: Adjust license headers to be comments...

[justinleet]

GitHub user justinleet opened a pull request:

https://github.com/apache/metron/pull/685

METRON-1087: Adjust license headers to be comments instead of Javadoc

## Contributor Comments
This does three things:
1. Changes Javadoc to just plain comment.
2. Fixes headers that have been affected by formatting (get rid of 
paragraph tags, weird formatting, and deleted spaces, etc.)
3. Adds a newline after the license, before the package statement.

Even though this affects a (very) large number of files, I left it as one 
PR.  This is mostly because even though there's many changes, the vast, vast 
majority are changing `/**` to `/*` and adding a newlines.  Only a couple dozen 
files actually have fixes beyond that.  If this is too large, too tedious, or 
GitHub does weird crap, let me know and I can split this up.  It was just 
easier when making the change to just find/replace and validate correctness 
afterwards.

Note that the newline in 3 is to meet the [Google Java Style 
Guide](https://google.github.io/styleguide/javaguide.html#s3-source-file-structure).
  This is not handled by the IDE formatter (at least in IntelliJ), 
unfortunately.
```
A source file consists of, in order:

1. License or copyright information, if present
2. Package statement
3. Import statements
4. Exactly one top-level class
Exactly one blank line separates each section that is present.
```

Validation is just opening a file and ensuring that autoformat doesn't 
change the license header.

I still need to spin up full dev as a precaution, but given that it's all 
comments and whitespace, I'm expecting no issues.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have ]you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/justinleet/metron license_comment_fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/685.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #685


commit 8a93d11389d8f2e961873470faea396bd311f83f
Author: justinjleet 
Date:   2017-08-04T19:18:13Z

Making the license not javadoc and putting newline before package

commit c584d88b4e177959e158436f11d444cac66eb562
Author: justinjleet 
Date:   2017-08-04T19:21:16Z

Fixing urls in licenses




---
If your project is set up for it, you can reply to this email 

[GitHub] metron pull request #684: DO NOT MERGE: METRON-1086: Create a Blockly-based ...

[merrimanr]

GitHub user merrimanr opened a pull request:

https://github.com/apache/metron/pull/684

DO NOT MERGE: METRON-1086: Create a Blockly-based user interface for Stellar

## Contributor Comments
This PR is a first look at the Blockly/Stellar integration with a working 
example.  This is POC quality code and has a long way to go.  This is just an 
initial preview to kick off the discussion, gather requirements, and work out a 
design.

To take it for a test drive, spin up full dev and bring up the Management 
UI.  Open the threat triage panel on the bro (or any) sensor and add the rule 
`EXISTS(ip_src_addr)`:
![screen shot 2017-08-04 at 5 21 51 
pm](https://user-images.githubusercontent.com/311928/28989205-8785e940-7939-11e7-87c0-957eaa959195.png)

Click the Save button and then click on the block icon for the newly 
created rule.  This will open the Stellar Blockly editor:
![screen shot 2017-08-04 at 5 24 10 
pm](https://user-images.githubusercontent.com/311928/28989252-c0c9403a-7939-11e7-8ea7-b616fbd51a2e.png)

You will notice the rule that was just created is now represented as a 
series of connected blocks with the rule displayed in a text box just below.  
You can edit the rule by dragging, dropping and interacting with various blocks 
that represent Stellar operators and functions.  Try changing the rule to 
`EXISTS(ip_src_addr) && IS_DOMAIN(host)` like this:
![screen shot 2017-08-04 at 5 33 57 
pm](https://user-images.githubusercontent.com/311928/28989475-1e08e8d0-793b-11e7-9115-a0b5378788ce.png)

The `&&` operator is under the **Boolean** category in the toolbox to the 
left.  The `IS_DOMAIN` function is under **Stellar -> Domain**.  The purple 
block with the value set to `host` represents a list of fields parsed from the 
json in the **SAMPLE** widget at the top.  This block is in the **Fields** 
category.  If the `host` field doesn't appear in the dropdown try getting new 
samples (right arrow) until you get a message that contains that field:
![screen shot 2017-08-04 at 5 43 59 
pm](https://user-images.githubusercontent.com/311928/28989668-88020284-793c-11e7-9f75-db82ce28e8a7.png)


Once you get the correct Stellar statement, save the sensor config:
![screen shot 2017-08-04 at 5 39 25 
pm](https://user-images.githubusercontent.com/311928/28989596-f1a9b656-793b-11e7-9c21-97674b97de0f.png)

Open that rule again with the Blockly editor and you should see the block 
statement you just created:
![screen shot 2017-08-04 at 5 41 12 
pm](https://user-images.githubusercontent.com/311928/28989617-22b9dcee-793c-11e7-8e29-8d629c984098.png)

This preview contains only a subset of Stellar features and is very much 
incomplete.  There is much more work to do.  For this demo it is shown in 
context of the threat triage editor but could be used anywhere Stellar is 
exposed.  For now I will leave you with this and let you form your own opinions 
about what it should evolve to.  After people have had a chance to see it, we 
can continue the discussion on where this should go next.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 

[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/530
  
Ok, I decided to try yet another combo here. I set the plugin version back 
to 3.0.1 and re-installed the archetype. Now I get a new file 
`~/.m2/repository/archetype-catalog.xml` with the archetype in it. Re-running 
the generate now works as expected.

I'm glad I was finally able to find a path through this, but it doesn't 
give me confidence that the Maven archetype plugin is working so 
inconsistently. Nevermind that the docs appear to be out of date now with the 
local catalog location - 
https://maven.apache.org/archetype/maven-archetype-plugin/specification/archetype-catalog.html


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/530
  
So another wrinkle here - when running the archetype plugin from a 
non-project (separate, clean) directory, it's going to default to 3.0.1. I'm 
trying this up in full dev right now and running into the same issues as I'm 
seeing on my Mac. I run `mvn clean install` on the extension archetype project 
and it shows up in archetype-catalog.xml. When I try to use the archetype, it 
doesn't show up. Again, this is Centos
```
[root@node1(192.168.66.121): ~/metron]
# metron-deployment/scripts/platform-info.sh
Metron 0.4.1
--
* METRON-777
--
commit 04727ca8c448e2845e48a8a95812e6bf27b2c13d
Author: Otto Fowler 
Date:   Thu Aug 3 13:30:29 2017 -0400

update document to remove references to the working dir
--
--
metron-deployment/scripts/platform-info.sh: line 48: ansible: command not 
found
--
metron-deployment/scripts/platform-info.sh: line 52: vagrant: command not 
found
--
Python 2.6.6
--
Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426; 
2017-04-03T19:39:06Z)
Maven home: /opt/maven
Java version: 1.8.0_77, vendor: Oracle Corporation
Java home: /usr/jdk64/jdk1.8.0_77/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-642.15.1.el6.x86_64", arch: "amd64", 
family: "unix"
--
metron-deployment/scripts/platform-info.sh: line 64: docker: command not 
found
--
node
v6.11.2
--
npm
3.10.10
--
Linux node1 2.6.32-642.15.1.el6.x86_64 #1 SMP Fri Feb 24 14:31:22 UTC 2017 
x86_64 x86_64 x86_64 GNU/Linux
--
Total System Memory = 7870.23 MB
Processor Model: Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
Processor Speed: 2294.688 MHz
Total Physical Processors: 4
Total cores: 16
Disk information:
/dev/mapper/vg_node1-lv_root
/dev/sda1 477M   63M  389M  14% /boot
This CPU may not support virtualization

# install archetypes

[root@node1(192.168.66.121): ~/metron/metron-maven-archetypes]
# mvn clean install
[INFO] Scanning for projects...
[INFO] 

[INFO] Reactor Build Order:
[INFO]
[INFO] metron-maven-parser-extension-archetype
[INFO] metron-maven-archetypes
[INFO]
[INFO] 

[INFO] Building metron-maven-parser-extension-archetype 0.4.1
[INFO] 

[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ 
metron-maven-parser-extension-archetype ---
[INFO] Deleting 
/root/metron/metron-maven-archetypes/metron-maven-parser-extension-archetype/target
[INFO]
[INFO] --- maven-resources-plugin:3.0.2:resources (default-resources) @ 
metron-maven-parser-extension-archetype ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered 
resources, i.e. build is platform dependent!
[INFO] Copying 22 resources
[INFO]
[INFO] --- maven-resources-plugin:3.0.2:testResources 
(default-testResources) @ metron-maven-parser-extension-archetype ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered 
resources, i.e. build is platform dependent!
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-archetype-plugin:2.4:jar (default-jar) @ 
metron-maven-parser-extension-archetype ---
[INFO] Building archetype jar: 
/root/metron/metron-maven-archetypes/metron-maven-parser-extension-archetype/target/metron-maven-parser-extension-archetype-0.4.1
[INFO]
[INFO] --- maven-archetype-plugin:2.4:integration-test 
(default-integration-test) @ metron-maven-parser-extension-archetype ---
[INFO] Processing Archetype IT project: basic
[INFO] 

[INFO] Using following parameters for creating project from Archetype: 
metron-maven-parser-extension-archetype:0.4.1
[INFO] 

[INFO] Parameter: groupId, Value: archetype.it
[INFO] Parameter: artifactId, Value: basic
[INFO] Parameter: version, Value: 0.1-SNAPSHOT
[INFO] Parameter: package, Value: it.pkg
[INFO] Parameter: packageInPathFormat, Value: it/pkg
[INFO] Parameter: parserName, Value: sample
[INFO] Parameter: version, Value: 0.1-SNAPSHOT
[INFO] Parameter: package, Value: it.pkg
[INFO] Parameter: metronVersion, Value: 0.4.0
[INFO] Parameter: groupId, Value: archetype.it
[INFO] Parameter: parserClassName, Value: Sample
[INFO] Parameter: artifactId, Value: basic
[INFO] Parent element not overwritten in 

[GitHub] metron issue #683: METRON-1084: Management UI web server license should be A...

[james-sirota]

Github user james-sirota commented on the issue:

https://github.com/apache/metron/pull/683
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #683: METRON-1084: Management UI web server license shou...

[merrimanr]

GitHub user merrimanr opened a pull request:

https://github.com/apache/metron/pull/683

METRON-1084: Management UI web server license should be Apache

## Contributor Comments
Minor license correction.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/merrimanr/incubator-metron METRON-1084

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/683.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #683


commit d6357d218bfe10e939a8591352adc6568ca74df8
Author: merrimanr 
Date:   2017-08-04T16:36:17Z

changed license to Apache




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #680: DO NOT MERGE: METRON-1078: Metron Indexing fails to stop ...

[anandsubbu]

Github user anandsubbu commented on the issue:

https://github.com/apache/metron/pull/680
  
Sounds great, @dlyle65535 !


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #680: DO NOT MERGE: METRON-1078: Metron Indexing fails to stop ...

[dlyle65535]

Github user dlyle65535 commented on the issue:

https://github.com/apache/metron/pull/680
  
Oh yeah, thanks for clarifying, @anandsubbu - I'm not going to change the 
'happy path' logic, so when the UI is up, it will work the way it did prior.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #680: DO NOT MERGE: METRON-1078: Metron Indexing fails to stop ...

[anandsubbu]

Github user anandsubbu commented on the issue:

https://github.com/apache/metron/pull/680
  
Hi @dlyle65535 can it be a two step validation as follows: 
a) We first verify if we are able to hit the storm URL, if that fails -> 
then we mark the service as stopped. 
b) If the storm URL is up, and the service/topology (indexing, enrichment, 
parsers etc) is missing, then we mark the service as stopped.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #678: METRON-1076: Management UI status check always rep...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/678


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #679: METRON-1075: Add faceted search capabilities

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/679


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #680: DO NOT MERGE: METRON-1078: Metron Indexing fails to stop ...

[dlyle65535]

Github user dlyle65535 commented on the issue:

https://github.com/apache/metron/pull/680
  
I'm starting to think the best approach would to assume the service is 
stopped if we can't get to the ui (which we kind of do anyway because of how 
status works).

Anybody absolutely hate that idea?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #680: DO NOT MERGE: METRON-1078: Metron Indexing fails to stop ...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/680
  
@dlyle65535 Thanks for the update. That's pretty surprising, it seems like 
this should take care of it.  Ping me when there's some new info, and I'll hop 
back on this review.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #680: METRON-1078: Metron Indexing fails to stop during "Stop A...

[dlyle65535]

Github user dlyle65535 commented on the issue:

https://github.com/apache/metron/pull/680
  
Thanks @justinleet  - I'm going to hold off on this, @anandsubbu did some 
additional testing and it didn't actually solve the problem. :( 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #680: METRON-1078: Metron Indexing fails to stop during "Stop A...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/680
  
+1 by inspection


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---