[jira] [Commented] (METRON-746) Build Custom Checkstyle and IDE formatting settings

2017-05-12 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16009087#comment-16009087
 ] 

ASF GitHub Bot commented on METRON-746:
---

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/577
  
Actually, that's a lie, autoformat works, but checkstyle doesn't highlight 
anything.  Well, actually, Eclipse won't highlight any errors, including 
outright compilation errors.  If someone has a working Eclipse environment, I 
can help setup checkstyle, but I'm not spending time debugging why Eclipse 
doesn't know how to import a cleanly cloned Maven project.


> Build Custom Checkstyle and IDE formatting settings
> ---
>
> Key: METRON-746
> URL: https://issues.apache.org/jira/browse/METRON-746
> Project: Metron
>  Issue Type: Improvement
>Reporter: Justin Leet
>Assignee: Justin Leet
>Priority: Minor
>
> We need a custom checkstyle.xml based off the sun convention checkstyle.  
> Based on a discussion thread, there are a few things that need to be setup
> * Two space indents
> * Line Length longer than 80 (pretty popular in the discussion, but not 
> officially part of our code style)
> * Appropriate warn/error levels set so we don't immediately start failing 
> every build.
> * Importable IntelliJ code style at minimum, but I'd also like to see Eclipse 
> if possible to avoid forcing a given dev environment.  IntelliJ allows for 
> importing a checkstyle.xml to use as the basis.  We can export the resulting 
> formatting settings for people.
> * Ensure Travis actually runs checkstyle during our builds



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (METRON-746) Build Custom Checkstyle and IDE formatting settings

2017-05-12 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16009083#comment-16009083
 ] 

ASF GitHub Bot commented on METRON-746:
---

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/577
  
This appears to also work in Eclipse (similar deal: Install Checkstyle from 
Eclipse Marketplace, import GoogleStyle with the same instruction link as 
IntelliJ).

Having said that, Eclipse really dislikes some plugins in our POMs, and it 
seems like an Eclipse thing.  I haven't dug in further, and to be blunt, am not 
interested in doing so unless someone really, really wants to work with Eclipse.


> Build Custom Checkstyle and IDE formatting settings
> ---
>
> Key: METRON-746
> URL: https://issues.apache.org/jira/browse/METRON-746
> Project: Metron
>  Issue Type: Improvement
>Reporter: Justin Leet
>Assignee: Justin Leet
>Priority: Minor
>
> We need a custom checkstyle.xml based off the sun convention checkstyle.  
> Based on a discussion thread, there are a few things that need to be setup
> * Two space indents
> * Line Length longer than 80 (pretty popular in the discussion, but not 
> officially part of our code style)
> * Appropriate warn/error levels set so we don't immediately start failing 
> every build.
> * Importable IntelliJ code style at minimum, but I'd also like to see Eclipse 
> if possible to avoid forcing a given dev environment.  IntelliJ allows for 
> importing a checkstyle.xml to use as the basis.  We can export the resulting 
> formatting settings for people.
> * Ensure Travis actually runs checkstyle during our builds



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (METRON-746) Build Custom Checkstyle and IDE formatting settings

2017-05-12 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16009082#comment-16009082
 ] 

ASF GitHub Bot commented on METRON-746:
---

GitHub user justinleet reopened a pull request:

https://github.com/apache/metron/pull/577

METRON-746: Build Custom Checkstyle and IDE formatting settings

## Contributor Comments

Pretty much just editing the POM to actually set up Checkstyle with the 
Google Code Style, per the dev thread.

See [Google Java Style 
Guide](https://google.github.io/styleguide/javaguide.html)

Note that the Checkstyle version is set to 7.7.  I've tested it with the 
plugin version and had no issues at all, but if anyone is concerned, we can 
lower it to either the default or another version.

### IntelliJ warnings
To pull into IntelliJ, please see 
https://github.com/jshiell/checkstyle-idea#configuration
Essentially the instructions are to:

-  Install the plugin (Available in IntelliJ's directly)
- Go into Settings -> Other Settings -> Checkstyle
- Change "Checkstyle version" to 7.7.
- Apply.  Otherwise the new file won't match the version and an error will 
be thrown.
- Add a new Checkstyle file
  - This can either be a local file or a remote one (remote will attempt to 
update periodically, so if you don't want to repeatedly connect to the URL, 
download the file somewhere and load it in).  Either works. For 7.7, use 
[google_checks.xml](https://raw.githubusercontent.com/checkstyle/checkstyle/master/src/main/resources/google_checks.xml)
  - If you have errors, you most likely need to make sure you're set to the 
right version and have applied it.
- Select the checkbox for the new style
- Apply

To ensure it's setup properly, open a file, e.g. 
`metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java`

New warnings should show up that are based on Checkstyle, e.g.
```
Checkstyle: WhitespaceAround: 'if' is not followed by whitespace. Empty 
blocks may only be represented as {} when not part of a multi-block statement
```

### Code formatting
There are two options for code formatting. One is to import the 
google_checks.xml file from above.  The other is to import Google's IntelliJ 
editor settings for coverage.  In practice, Google's direct IntelliJ settings 
seem to handle autoformatting better. The instructions are essentially the same 
for each.

Google's setup is described at 
https://github.com/HPI-Information-Systems/Metanome/wiki/Installing-the-google-styleguide-settings-in-intellij-and-eclipse.
  The direct Checkstyle can be imported similarly, but choosing the CheckStyle 
Configuration when doing Import Scheme.

### Required Follow-up
This will require updating the wiki at 
https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines 
(Section 2.2).  It needs to be switched to the Google code conventions, and the 
instructions above for formatting and warnings should be pulled in.

### Testing
I still need to spin up full-dev, but to get the checkstyle report do:
```
mvn clean install -DskipTests
mvn site site:stage-deploy site:deploy -Dmaven.javadoc.skip=true 
```
Navigate to file:///tmp/metron/site/checkstyle-aggregate.html in a browser.
You should see
`The following document contains the results of Checkstyle 7.7 with 
google_checks.xml ruleset.` and a set of warnings. In practice, most of them 
are fixed by batch autoformatting, per discussions, which will be a followup 
task.

### Misc
No enforcement of Checkstyle is made, and in fact, by default 
google_checks.xml sets everything to warning. The thought is that we'd need to 
pretty significantly reduce the number of warnings, and probably think through 
how we want to handle it.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention

[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs

2017-05-12 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16009015#comment-16009015
 ] 

ASF GitHub Bot commented on METRON-508:
---

GitHub user JonZeolla opened a pull request:

https://github.com/apache/metron/pull/586

METRON-508 Expand Elasticsearch templates to support the standard bro logs

## Contributor Comments
This PR makes it easier for someone with an existing bro install to send 
some of their log files into Metron, based off of a combination of the [bro 
documentation](https://www.bro.org/sphinx/script-reference/log-files.html) and 
a fresh install of bro 2.5.  There are future plans to expand on this via 
[METRON-518](https://issues.apache.org/jira/browse/METRON-518) and 
[METRON-908](https://issues.apache.org/jira/browse/METRON-908).  Specifically, 
this attempts to provide initial support the default-on fields of the following 
logs:
 - 
[Conn](https://www.bro.org/sphinx/scripts/base/protocols/conn/main.bro.html#type-Conn::Info)
 - 
[DPD](https://www.bro.org/sphinx-git/scripts/base/frameworks/dpd/main.bro.html#type-DPD::Info)
 - 
[FTP](https://www.bro.org/sphinx/scripts/base/protocols/ftp/info.bro.html#type-FTP::Info)
 - 
[Files](https://www.bro.org/sphinx/scripts/base/frameworks/files/main.bro.html#type-Files::Info)
 - 
[Known::CertsInfo](https://www.bro.org/sphinx/scripts/policy/protocols/ssl/known-certs.bro.html#type-Known::CertsInfo)
 - 
[SMTP](https://www.bro.org/sphinx/scripts/base/protocols/smtp/main.bro.html#type-SMTP::Info)
 - 
[SSL](https://www.bro.org/sphinx/scripts/base/protocols/ssl/main.bro.html#type-SSL::Info)
 - 
[Weird](https://www.bro.org/sphinx/scripts/base/frameworks/notice/weird.bro.html#type-Weird::Info)
 - 
[Notice](https://www.bro.org/sphinx/scripts/base/frameworks/notice/main.bro.html#type-Notice::Info)
 - 
[DHCP](https://www.bro.org/sphinx/scripts/base/protocols/dhcp/main.bro.html#type-DHCP::Info)
 - 
[SSH](https://www.bro.org/sphinx/scripts/base/protocols/ssh/main.bro.html#type-SSH::Info)
 - 
[Software](https://www.bro.org/sphinx/scripts/base/frameworks/software/main.bro.html#type-Software::Info)
 - 
[Radius](https://www.bro.org/sphinx/scripts/base/protocols/radius/main.bro.html#type-RADIUS::Info)
 - 
[X509](https://www.bro.org/sphinx/scripts/base/files/x509/main.bro.html#type-X509::Info)


## Testing
1.  Create a working directory and pull in this PR
```
mkdir ~/metron-508
git clone https://github.com/apache/incubator-metron 
~/metron-508/incubator-metron
cd ~/metron-508/incubator-metron
git remote add jonzeolla https://github.com/jonzeolla/incubator-metron
git pull jonzeolla METRON-508
```
1.  Modify 
[this](https://github.com/JonZeolla/incubator-metron/blob/METRON-508/metron-deployment/vagrant/full-dev-platform/Vagrantfile#L20)
 to remove `sensors,` (to spin up the real sensors).
```
sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" 
metron-deployment/vagrant/full-dev-platform/Vagrantfile
```
1.  Start up full-dev.
```
cd metron-deployment/vagrant/full-dev-platform
vagrant up
```
1.  Set up the environment in full-dev.
```
vagrant ssh
sudo su -
export PATH=$PATH:/usr/local/bro/bin
service monit stop && service sensor-stubs stop bro && broctl stop
yum -y install jq wireshark
```
1.  Configure kafka in local.bro.
```
sed -i 's/redef Kafka::logs_to_send = .*/redef Kafka::logs_to_send = 
set(HTTP::LOG, DNS::LOG, Conn::LOG, DPD::LOG, DHCP::LOG, FTP::LOG, SSH::LOG, 
SSL::LOG, SMTP::LOG, RADIUS::LOG, Weird::LOG, Files::LOG, Notice::LOG, 
Software::LOG, Known::CERTS_LOG, X509::LOG);/' 
/usr/local/bro/share/bro/site/local.bro
echo "redef Kafka::debug = \"all\";" >> 
/usr/local/bro/share/bro/site/local.bro
echo "redef Known::cert_tracking = ALL_HOSTS;" >> 
/usr/local/bro/share/bro/site/local.bro
echo "redef Software::asset_tracking = ALL_HOSTS;" >> 
/usr/local/bro/share/bro/site/local.bro
```
1.  Monitor the bro kafka topic
```
# Open a new terminal
cd 
~/metron-508/incubator-metron/metron-deployment/vagrant/full-dev-platform
vagrant ssh
sudo su -
export PATH=$PATH:/usr/local/bro/bin:/usr/hdp/current/kafka-broker/bin
kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro
```
1.  Monitor the storm logs.
```
# Open a new terminal
cd 
~/metron-508/incubator-metron/metron-deployment/vagrant/full-dev-platform
vagrant ssh
sudo su -
export PATH=$PATH:/usr/local/bro/bin:/usr/hdp/current/kafka-broker/bin
# Look at the storm logs (The "failed to parse" errors for ip_src_addr 
and ip_dst_addr are expected, and

[jira] [Commented] (METRON-936) Fixes to pcap for performance and testing

2017-05-12 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16008689#comment-16008689
 ] 

ASF GitHub Bot commented on METRON-936:
---

GitHub user mmiklavc opened a pull request:

https://github.com/apache/metron/pull/585

METRON-936: Fixes to pcap for performance and testing

https://issues.apache.org/jira/browse/METRON-936

I still have some cleanup and additional docs to add, but I wanted to get 
this out for review.

## Contributor Comments
[Please place any comments here.  A description of the problem/enhancement, 
how to reproduce the issue, your testing methodology, etc.]


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mmiklavc/incubator-metron pcap-fixes

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/585.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #585


commit 6bcc34d0f4d4ae473a768f5909248fb85ffdc487
Author: Michael Miklavcic 
Date:   2017-05-12T20:56:24Z

fixes to pcap for performance and testing




> Fixes to pcap for performance and testing
> -
>
> Key: METRON-936
> URL: https://issues.apache.org/jira/browse/METRON-936
> Project: Metron
>  Issue Type: Bug
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>
> Default in pcap.properties for kafka.pcap.start=END, and it's also not used 
> in the Flux file. topology.workers should also be provided via the properties 
> file.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (METRON-936) Miscellaneous PCAP fixes

2017-05-12 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16008686#comment-16008686
 ] 

ASF GitHub Bot commented on METRON-936:
---

Github user mmiklavc closed the pull request at:

https://github.com/apache/metron/pull/569


> Miscellaneous PCAP fixes
> 
>
> Key: METRON-936
> URL: https://issues.apache.org/jira/browse/METRON-936
> Project: Metron
>  Issue Type: Bug
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>
> Default in pcap.properties for kafka.pcap.start=END, and it's also not used 
> in the Flux file. topology.workers should also be provided via the properties 
> file.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (METRON-936) Fixes to pcap for performance and testing

2017-05-12 Thread Michael Miklavcic (JIRA) 

 [ 
https://issues.apache.org/jira/browse/METRON-936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Miklavcic updated METRON-936:
-
Summary: Fixes to pcap for performance and testing  (was: Miscellaneous 
PCAP fixes)

> Fixes to pcap for performance and testing
> -
>
> Key: METRON-936
> URL: https://issues.apache.org/jira/browse/METRON-936
> Project: Metron
>  Issue Type: Bug
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>
> Default in pcap.properties for kafka.pcap.start=END, and it's also not used 
> in the Flux file. topology.workers should also be provided via the properties 
> file.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (METRON-936) Miscellaneous PCAP fixes

2017-05-12 Thread Michael Miklavcic (JIRA) 

 [ 
https://issues.apache.org/jira/browse/METRON-936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Miklavcic updated METRON-936:
-
Summary: Miscellaneous PCAP fixes  (was: PCAP topology properties in flux 
not correctly using pcap.properties)

> Miscellaneous PCAP fixes
> 
>
> Key: METRON-936
> URL: https://issues.apache.org/jira/browse/METRON-936
> Project: Metron
>  Issue Type: Bug
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>
> Default in pcap.properties for kafka.pcap.start=END, and it's also not used 
> in the Flux file. topology.workers should also be provided via the properties 
> file.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (METRON-950) Migrate storm-kafka-client to 1.1

2017-05-12 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16008574#comment-16008574
 ] 

ASF GitHub Bot commented on METRON-950:
---

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/584
  
The test plan for this should be mainly a smoke test..make sure data flows 
through the default topologies in full dev.


> Migrate storm-kafka-client to 1.1
> -
>
> Key: METRON-950
> URL: https://issues.apache.org/jira/browse/METRON-950
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>Assignee: Casey Stella
>
> There are MAJOR performance issues with the storm-kafka-client.  Throughput 
> is roughly an order of magnitude faster in the 1.1.0 version vs the 1.0.3.
> This also removes the requirement for the non-HDP profile to rely on the HDP 
> repo.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (METRON-948) License field in npm package.json files is incorrect

2017-05-12 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16008506#comment-16008506
 ] 

ASF GitHub Bot commented on METRON-948:
---

Github user simonellistonball closed the pull request at:

https://github.com/apache/incubator-metron/pull/582


> License field in npm package.json files is incorrect
> 
>
> Key: METRON-948
> URL: https://issues.apache.org/jira/browse/METRON-948
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4
>Reporter: Simon Elliston Ball
>Priority: Blocker
>
> The npm package.json files have an abbreviated license field which 
> incorrectly states MIT instead of Apache (all files are licensed Apache)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (METRON-932) Change HLLP Stellar functions to accept empty lists

2017-05-12 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16008489#comment-16008489
 ] 

ASF GitHub Bot commented on METRON-932:
---

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/566


> Change HLLP Stellar functions to accept empty lists
> ---
>
> Key: METRON-932
> URL: https://issues.apache.org/jira/browse/METRON-932
> Project: Metron
>  Issue Type: Improvement
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>
> The current implementation, which disallows merging empty sets, makes 
> interactions via the Profiler more complex than they need to be. This Jira 
> would change the argument handling to allow empty lists. The resulting 
> functionality should treat cardinality of a null set as zero.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (METRON-949) Change from error to warning message for Metron REST in Ambari service_advisor

2017-05-12 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16008473#comment-16008473
 ] 

ASF GitHub Bot commented on METRON-949:
---

Github user anandsubbu closed the pull request at:

https://github.com/apache/incubator-metron/pull/583


> Change from error to warning message for Metron REST in Ambari service_advisor
> --
>
> Key: METRON-949
> URL: https://issues.apache.org/jira/browse/METRON-949
> Project: Metron
>  Issue Type: Bug
>Reporter: Anand Subramanian
>Assignee: Anand Subramanian
>Priority: Minor
> Attachments: Metron-REST-service_check.png
>
>
> In the "Assign Masters" page of the Ambari deployment wizard, there is a 
> check for the Metron REST host to have storm supervisor installed. This is 
> currently displayed as an Error message
> However in the flow of the wizard, the actual storm supervisor selection can 
> only be done in the subsequent screen. So, this message should be changed 
> into a Warning instead (exclamation inside a triangle). 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (METRON-932) Change HLLP Stellar functions to accept empty lists

2017-05-12 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16008468#comment-16008468
 ] 

ASF GitHub Bot commented on METRON-932:
---

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/566
  
+1 by inspection.  this looks great


> Change HLLP Stellar functions to accept empty lists
> ---
>
> Key: METRON-932
> URL: https://issues.apache.org/jira/browse/METRON-932
> Project: Metron
>  Issue Type: Improvement
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>
> The current implementation, which disallows merging empty sets, makes 
> interactions via the Profiler more complex than they need to be. This Jira 
> would change the argument handling to allow empty lists. The resulting 
> functionality should treat cardinality of a null set as zero.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (METRON-902) ES improperly indexes Bro logs

2017-05-12 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16008122#comment-16008122
 ] 

ASF GitHub Bot commented on METRON-902:
---

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/555


> ES improperly indexes Bro logs
> --
>
> Key: METRON-902
> URL: https://issues.apache.org/jira/browse/METRON-902
> Project: Metron
>  Issue Type: Bug
>Reporter: Jon Zeolla
>Assignee: Jon Zeolla
>
> It appears that an old issue has been reintroduced into the ES template for 
> indexing bro DNS logs.  It is possible that other issues have been 
> reintroduced as well, as I have not yet reviewed the template holistically.
> Initial fix:  
> https://github.com/apache/incubator-metron/blob/4bfb09c49fbc6204ce8b826887d99beff414f84a/metron-deployment/roles/metron_elasticsearch_templates/files/es_templates/bro_index.template#L165-L167
> Reintroduction:  
> https://github.com/apache/incubator-metron/blob/125dbef1e59ff808a62e4f5a7d265aafbcf6bf08/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/0.2.0BETA/package/files/bro_index.template#L165-L167



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (METRON-935) EC2 Deployment Failure - Could Not Create Blueprint

2017-05-12 Thread Nick Allen (JIRA) 

 [ 
https://issues.apache.org/jira/browse/METRON-935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Allen updated METRON-935:
--
Affects Version/s: 0.3.1
Fix Version/s: Next + 1

> EC2 Deployment Failure - Could Not Create Blueprint
> ---
>
> Key: METRON-935
> URL: https://issues.apache.org/jira/browse/METRON-935
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.3.1
>Reporter: Nick Allen
>Assignee: Ryan Merriman
>Priority: Blocker
> Fix For: Next + 1
>
>
> When deploying Metron with EC2, I ran into this problem.  
> {code}
> TASK [ambari_config : Deploy cluster with Ambari; 
> http://ec2-54-236-37-116.compute-1.amazonaws.com:8080] ***
> fatal: [ec2-54-236-37-116.compute-1.amazonaws.com]: FAILED! => {"changed": 
> false, "failed": true, 
> "msg": "Ambari client exception occurred: Could not create blueprint: request 
> code 400, 
> request message {\n  \"status\" : 400,\n  \"message\" : \"Blueprint 
> configuration validation failed: 
> Missing required properties.  Specify a value for these properties in the 
> blueprint configuration. 
> {metron={metron-env=[metron_jdbc_platform, metron_jdbc_driver, 
> metron_jdbc_username, metron_jdbc_url]}}\"\n}"}
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (METRON-912) Metron vagrant setup steps no longer work

2017-05-12 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16008105#comment-16008105
 ] 

ASF GitHub Bot commented on METRON-912:
---

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/560


> Metron vagrant setup steps no longer work
> -
>
> Key: METRON-912
> URL: https://issues.apache.org/jira/browse/METRON-912
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>
> The latest maven prereq steps no longer work, as they install versions 
> inconsistent with what we have tested.  



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (METRON-943) Create traffic connections report in zeppelin

2017-05-12 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16008074#comment-16008074
 ] 

ASF GitHub Bot commented on METRON-943:
---

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/573


> Create traffic connections report in zeppelin
> -
>
> Key: METRON-943
> URL: https://issues.apache.org/jira/browse/METRON-943
> Project: Metron
>  Issue Type: Improvement
>Reporter: Justin Leet
>Assignee: Justin Leet
>
> User types in CIDR range into a search box
> System generates connections report:
> Volume of outbound traffic (cumulative) for every IP in range
> Volume of inbound traffic (cumulative) for every IP in range



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (METRON-949) Change from error to warning message for Metron REST in Ambari service_advisor

2017-05-12 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16007860#comment-16007860
 ] 

ASF GitHub Bot commented on METRON-949:
---

GitHub user anandsubbu opened a pull request:

https://github.com/apache/incubator-metron/pull/583

METRON-949 Change from error to warning message for Metron REST in Ambari 
service_advisor

## Contributor Comments
Reduced the level to WARN in service_advisor.py.

Steps to validate:
- Fire up Ambari wizard after installing Mpack
- In the Assign Masters page, Metron REST should now show a 'WARN' icon 
instead of 'ERROR'

I have not carried out the full suite of tests, since this is a minor 
rename from ERROR to WARN.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [NA] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [NA] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [NA] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [NA] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [NA] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [NA] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/anandsubbu/incubator-metron METRON-949

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/583.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #583


commit e7c158d24ccb27f0973332d76c598233381f6d3e
Author: Anand Subramanian 
Date:   2017-05-12T09:26:16Z

Changed from ERROR to WARN




> Change from error to warning message for Metron REST in Ambari service_advisor
> --
>
> Key: METRON-949
> URL: https://issues.apache.org/jira/browse/METRON-949
> Project: Metron
>  Issue Type: Bug
>Reporter: Anand Subramanian
>Assignee: Anand Subramanian
>Priority: Minor
> Attachments: Metron-REST-service_check.png
>
>
> In the "Assign Masters" page of the Ambari deployment wizard, there is a 
> check for the Metron REST host to have storm supervisor installed. This is 
> currently displayed as an Error message
> However in the flow of the wizard, the actual storm supervisor selection can 
> only be done in the subsequent screen. So, this

[jira] [Commented] (METRON-914) Build Metron failure

2017-05-12 Thread GS Peter (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16007851#comment-16007851
 ] 

GS Peter commented on METRON-914:
-

Hi [~rmerriman]. I have done configed its, problem is metron version. But I 
don't ingest snort, bro, yaf log. Can you help me?

> Build Metron failure
> 
>
> Key: METRON-914
> URL: https://issues.apache.org/jira/browse/METRON-914
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4
> Environment: Ubuntu 16.04.1
>Reporter: GS Peter
> Fix For: 0.4
>
> Attachments: build.log.tar.gz, npm-debug.log
>
>   Original Estimate: 504h
>  Remaining Estimate: 504h
>
> [INFO] metron-config .. FAILURE [04:08 
> min]
> [INFO] metron-rest-client . SKIPPED
> [INFO] metron-rest  SKIPPED
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 58:04 min
> [INFO] Finished at: 2017-04-28T15:57:16+07:00
> [INFO] Final Memory: 141M/588M
> [INFO] 
> 
> [ERROR] Failed to execute goal 
> com.github.eirslett:frontend-maven-plugin:1.3:npm (npm install) on project 
> metron-config: Failed to run task: 'npm install' failed. (error code 1) -> 
> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please 
> read the following articles:
> [ERROR] [Help 1] 
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the 
> command
> [ERROR]   mvn  -rf :metron-config



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (METRON-914) Build Metron failure

2017-05-12 Thread GS Peter (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16007851#comment-16007851
 ] 

GS Peter edited comment on METRON-914 at 5/12/17 9:26 AM:
--

Hi [~rmerriman]. I have done configed it, problem is metron version. But I 
don't ingest snort, bro, yaf log. Can you help me?


was (Author: gspeter):
Hi [~rmerriman]. I have done configed its, problem is metron version. But I 
don't ingest snort, bro, yaf log. Can you help me?

> Build Metron failure
> 
>
> Key: METRON-914
> URL: https://issues.apache.org/jira/browse/METRON-914
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4
> Environment: Ubuntu 16.04.1
>Reporter: GS Peter
> Fix For: 0.4
>
> Attachments: build.log.tar.gz, npm-debug.log
>
>   Original Estimate: 504h
>  Remaining Estimate: 504h
>
> [INFO] metron-config .. FAILURE [04:08 
> min]
> [INFO] metron-rest-client . SKIPPED
> [INFO] metron-rest  SKIPPED
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 58:04 min
> [INFO] Finished at: 2017-04-28T15:57:16+07:00
> [INFO] Final Memory: 141M/588M
> [INFO] 
> 
> [ERROR] Failed to execute goal 
> com.github.eirslett:frontend-maven-plugin:1.3:npm (npm install) on project 
> metron-config: Failed to run task: 'npm install' failed. (error code 1) -> 
> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please 
> read the following articles:
> [ERROR] [Help 1] 
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the 
> command
> [ERROR]   mvn  -rf :metron-config



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (METRON-949) Change from error to warning message for Metron REST in Ambari service_advisor

2017-05-12 Thread Anand Subramanian (JIRA) 
Anand Subramanian created METRON-949:


 Summary: Change from error to warning message for Metron REST in 
Ambari service_advisor
 Key: METRON-949
 URL: https://issues.apache.org/jira/browse/METRON-949
 Project: Metron
  Issue Type: Bug
Reporter: Anand Subramanian
Assignee: Anand Subramanian
Priority: Minor
 Attachments: Metron-REST-service_check.png

In the "Assign Masters" page of the Ambari deployment wizard, there is a check 
for the Metron REST host to have storm supervisor installed. This is currently 
displayed as an Error message

However in the flow of the wizard, the actual storm supervisor selection can 
only be done in the subsequent screen. So, this message should be changed into 
a Warning instead (exclamation inside a triangle). 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)