date:20171115

[jira] [Commented] (METRON-1316) Fastcapa Fails to Compile in Test Environment

2017-11-15 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16254850#comment-16254850
 ] 

ASF GitHub Bot commented on METRON-1316:


Github user anandsubbu commented on the issue:

https://github.com/apache/metron/pull/841
  
Hi @nickwallen , tested on the vagrant fastcapa deployment. Deployed fine, 
and I was able to see this message as well:
```
TASK [debug] 
***
ok: [source] => {
"msg": "Successfully received packets sent from pcap-replay!"
}
```
During the deployment, I noticed the following message. I was not sure if 
this can be ignored or needs to be looked at.

```
TASK [fastcapa : Restart for modified kernel params] 
***
fatal: [sink]: FAILED! => {"changed": false, "failed": true, 
"module_stderr": "", "module_stdout": "", "msg": "MODULE FAILURE", "parsed": 
false}
...ignoring
```



> Fastcapa Fails to Compile in Test Environment
> -
>
> Key: METRON-1316
> URL: https://issues.apache.org/jira/browse/METRON-1316
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.1
>Reporter: Nick Allen
>Priority: Critical
> Fix For: Next + 1
>
>
> When deploying Fastcapa in the test environment, the following error is 
> encountered.
> {code}
> == Build lib/librte_eal/linuxapp/kni
>   CC [M]  
> /root/dpdk-stable-16.11.1/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/igb_main.o
> /root/dpdk-stable-16.11.1/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/igb_main.c:
>  In function ‘igb_ndo_bridge_getlink’:
> /root/dpdk-stable-16.11.1/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/igb_main.c:2288:2:
>  error: too few arguments to function ‘ndo_dflt_bridge_getlink’
>   return ndo_dflt_bridge_getlink(skb, pid, seq, dev, mode, 0, 0, nlflags);
>   ^
> In file included from 
> /usr/src/kernels/3.10.0-693.5.2.el7.x86_64/include/net/dst.h:13:0,
>  from 
> /usr/src/kernels/3.10.0-693.5.2.el7.x86_64/include/net/sock.h:72,
>  from 
> /usr/src/kernels/3.10.0-693.5.2.el7.x86_64/include/linux/tcp.h:23,
>  from 
> /root/dpdk-stable-16.11.1/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/igb_main.c:34:
> /usr/src/kernels/3.10.0-693.5.2.el7.x86_64/include/linux/rtnetlink.h:115:12: 
> note: declared here
>  extern int ndo_dflt_bridge_getlink(struct sk_buff *skb, u32 pid, u32 seq,
> ^
> /root/dpdk-stable-16.11.1/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/igb_main.c:
>  At top level:
> /root/dpdk-stable-16.11.1/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/igb_main.c:2317:2:
>  error: unknown field ‘ndo_set_vf_vlan’ specified in initializer
>   .ndo_set_vf_vlan = igb_ndo_set_vf_vlan,
>   ^
> /root/dpdk-stable-16.11.1/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/igb_main.c:
>  In function ‘igb_ndo_bridge_getlink’:
> /root/dpdk-stable-16.11.1/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/igb_main.c:2296:1:
>  error: control reaches end of non-void function [-Werror=return-type]
>  }
>  ^
> cc1: all warnings being treated as errors
> make[10]: *** 
> [/root/dpdk-stable-16.11.1/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/igb_main.o]
>  Error 1
> make[9]: *** 
> [_module_/root/dpdk-stable-16.11.1/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni]
>  Error 2
> make[8]: *** [sub-make] Error 2
> make[7]: *** [rte_kni.ko] Error 2
> make[6]: *** [kni] Error 2
> make[5]: *** [linuxapp] Error 2
> make[4]: *** [librte_eal] Error 2
> make[3]: *** [lib] Error 2
> make[2]: *** [all] Error 2
> make[1]: *** [pre_install] Error 2
> make: *** [install] Error 2
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1311) Service Check Should Check Elasticsearch Index Templates

2017-11-15 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-1311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16254663#comment-16254663
 ] 

ASF GitHub Bot commented on METRON-1311:


Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/839
  
+1 by inspection. Thanks @nickwallen!


> Service Check Should Check Elasticsearch Index Templates
> 
>
> Key: METRON-1311
> URL: https://issues.apache.org/jira/browse/METRON-1311
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.1
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: Next + 1
>
>
> The Service Check in Ambari does not validate that the Elasticsearch index 
> templates have been installed.  Without these index templates bad things can 
> happen.  For example, the Alerts UI will not be able to display any alerts.
> The Elasticsearch index templates that are installed by Ambari should also be 
> checked as part of the Metron Service Check.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

2017-11-15 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16254603#comment-16254603
 ] 

ASF GitHub Bot commented on METRON-1289:


Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/824


> Alert fields are lost when a MetaAlert is created
> -
>
> Key: METRON-1289
> URL: https://issues.apache.org/jira/browse/METRON-1289
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a MetaAlert is created, the included results are being updated 
> incorrectly with only the "metaalert" field.  This causes subsequent findOne 
> operations to only return the "metaalert field for that alert.  All fields 
> should continue to be present.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1316) Fastcapa Fails to Compile in Test Environment

2017-11-15 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16254392#comment-16254392
 ] 

ASF GitHub Bot commented on METRON-1316:


GitHub user nickwallen opened a pull request:

https://github.com/apache/metron/pull/841

METRON-1316 Fastcapa Fails to Compile in Test Environment


DPDK 16.11.1 will no longer compile against newer Linux kernels due to an 
interface change in the kernel.  

This can be replicated in the Fastcapa test environment.  We use the 
`bentos/centos-7.1` image for the test environment.  In this image, the 
underlying Linux kernel has been updated which makes it no longer work with 
DPDK 16.11.1.

This change upgrades DPDK to 17.08 in the test environment and also updates 
Fastcapa to accomodate a function signature change in the newer version of DPDK.

I also added the ability to run the same Fastcapa test code under multiple 
operating systems.  So now we can validate that Fastcapa runs in CentOS 7.1 and 
also CentOS 7.4, for example.  Additional platforms could easily be added.

I also removed an old Ansible role that is no longer being used.

## Testing

1. Run the following.
```
cd metron-sensors/fastcapa/centos-7.1
vagrant up
```

2.  The deployment should complete with a message like so.
```
Successfully received a Kafka message from fastcapa!



## Pull Request Checklist

- [ ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nickwallen/metron METRON-1316

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/841.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #841


commit a00166187794a996b37c910e525ea7469b0241fb
Author: Nick Allen 
Date:   2017-11-15T22:42:38Z

METRON-1316




> Fastcapa Fails to Compile in Test Environment
> -
>
> Key: METRON-1316
> URL: https://issues.apache.org/jira/browse/METRON-1316
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.1
>Reporter: Nick Allen
>Priority: Critical
> Fix For: Next + 1
>
>
> When deploying Fastcapa in the test environment, the following error is 
> encountered.
> {code}
> == Build lib/librte_eal/linuxapp/kni
>   CC [M]  
> /root/dpdk-stable-16.11.1/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/igb_main.o
> /root/dpdk-stable-16.11.1/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/igb_main.c:
>  In function ‘igb_ndo_bridge_getlink’:
> /root/dpdk-stable-16.11.1/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/igb_main.c:2288:2:
>  error: too few arguments to function ‘ndo_dflt_bridge_getlink’
>   return ndo_dflt_bridge_getlink(skb, pid, seq, dev, mode, 0, 0, nlflags);
>   ^
> In file included from 
> /usr/src/kernels/3.10.0-693.5.2.el7.x86_64/include/net/dst.h:13:0,
>  from 
> /usr/src/kernels/3.10.0-693.5.2.el7.x86_64/include/net/sock.h:72,
>  from 
> /usr/src/kernels/3.10.0-693.5.2.el7.x86_64/include/linux/tcp.h:23,
>  from 
> /root/dpdk-stable-16.11.1/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/igb_main.c:34:
> /usr/src/kernels/3.10.0-693.5.2.el7.x86_64/include/linux/rtnetlink.h:115:12: 
> note: declared here
>  extern int

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

2017-11-15 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16254393#comment-16254393
 ] 

ASF GitHub Bot commented on METRON-1289:


Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/824
  
+1, looks good. Thanks for all the work on the supplemental fixes. Feel 
free to skip attribution on the testing PR.


> Alert fields are lost when a MetaAlert is created
> -
>
> Key: METRON-1289
> URL: https://issues.apache.org/jira/browse/METRON-1289
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a MetaAlert is created, the included results are being updated 
> incorrectly with only the "metaalert" field.  This causes subsequent findOne 
> operations to only return the "metaalert field for that alert.  All fields 
> should continue to be present.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (METRON-1316) Fastcapa Fails to Compile in Test Environment

2017-11-15 Thread Nick Allen (JIRA)

Nick Allen created METRON-1316:
--

 Summary: Fastcapa Fails to Compile in Test Environment
 Key: METRON-1316
 URL: https://issues.apache.org/jira/browse/METRON-1316
 Project: Metron
  Issue Type: Bug
Affects Versions: 0.4.1
Reporter: Nick Allen
Priority: Critical
 Fix For: Next + 1


When deploying Fastcapa in the test environment, the following error is 
encountered.
{code}
== Build lib/librte_eal/linuxapp/kni
  CC [M]  
/root/dpdk-stable-16.11.1/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/igb_main.o
/root/dpdk-stable-16.11.1/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/igb_main.c:
 In function ‘igb_ndo_bridge_getlink’:
/root/dpdk-stable-16.11.1/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/igb_main.c:2288:2:
 error: too few arguments to function ‘ndo_dflt_bridge_getlink’
  return ndo_dflt_bridge_getlink(skb, pid, seq, dev, mode, 0, 0, nlflags);
  ^
In file included from 
/usr/src/kernels/3.10.0-693.5.2.el7.x86_64/include/net/dst.h:13:0,
 from 
/usr/src/kernels/3.10.0-693.5.2.el7.x86_64/include/net/sock.h:72,
 from 
/usr/src/kernels/3.10.0-693.5.2.el7.x86_64/include/linux/tcp.h:23,
 from 
/root/dpdk-stable-16.11.1/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/igb_main.c:34:
/usr/src/kernels/3.10.0-693.5.2.el7.x86_64/include/linux/rtnetlink.h:115:12: 
note: declared here
 extern int ndo_dflt_bridge_getlink(struct sk_buff *skb, u32 pid, u32 seq,
^
/root/dpdk-stable-16.11.1/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/igb_main.c:
 At top level:
/root/dpdk-stable-16.11.1/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/igb_main.c:2317:2:
 error: unknown field ‘ndo_set_vf_vlan’ specified in initializer
  .ndo_set_vf_vlan = igb_ndo_set_vf_vlan,
  ^
/root/dpdk-stable-16.11.1/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/igb_main.c:
 In function ‘igb_ndo_bridge_getlink’:
/root/dpdk-stable-16.11.1/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/igb_main.c:2296:1:
 error: control reaches end of non-void function [-Werror=return-type]
 }
 ^
cc1: all warnings being treated as errors
make[10]: *** 
[/root/dpdk-stable-16.11.1/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni/igb_main.o]
 Error 1
make[9]: *** 
[_module_/root/dpdk-stable-16.11.1/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/kni]
 Error 2
make[8]: *** [sub-make] Error 2
make[7]: *** [rte_kni.ko] Error 2
make[6]: *** [kni] Error 2
make[5]: *** [linuxapp] Error 2
make[4]: *** [librte_eal] Error 2
make[3]: *** [lib] Error 2
make[2]: *** [all] Error 2
make[1]: *** [pre_install] Error 2
make: *** [install] Error 2
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

2017-11-15 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16254250#comment-16254250
 ] 

ASF GitHub Bot commented on METRON-1289:


Github user cestella commented on the issue:

https://github.com/apache/metron/pull/824
  
This looks good to me.  I will not require attribution to my PR against 
this branch and I give this a +1 pending travis.


> Alert fields are lost when a MetaAlert is created
> -
>
> Key: METRON-1289
> URL: https://issues.apache.org/jira/browse/METRON-1289
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a MetaAlert is created, the included results are being updated 
> incorrectly with only the "metaalert" field.  This causes subsequent findOne 
> operations to only return the "metaalert field for that alert.  All fields 
> should continue to be present.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

2017-11-15 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16254099#comment-16254099
 ] 

ASF GitHub Bot commented on METRON-1289:


Github user cestella commented on the issue:

https://github.com/apache/metron/pull/824
  
@justinleet same here.  My +1 is imminent pending docs.


> Alert fields are lost when a MetaAlert is created
> -
>
> Key: METRON-1289
> URL: https://issues.apache.org/jira/browse/METRON-1289
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a MetaAlert is created, the included results are being updated 
> incorrectly with only the "metaalert" field.  This causes subsequent findOne 
> operations to only return the "metaalert field for that alert.  All fields 
> should continue to be present.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

2017-11-15 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16254095#comment-16254095
 ] 

ASF GitHub Bot commented on METRON-1289:


Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/824
  
Code-wise, I'm pretty good at this point.  Once the docs come in, I'll give 
them a once-over and hopefully we're good to go soon.

Thanks a lot for the hard work here!


> Alert fields are lost when a MetaAlert is created
> -
>
> Key: METRON-1289
> URL: https://issues.apache.org/jira/browse/METRON-1289
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a MetaAlert is created, the included results are being updated 
> incorrectly with only the "metaalert" field.  This causes subsequent findOne 
> operations to only return the "metaalert field for that alert.  All fields 
> should continue to be present.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

2017-11-15 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16253975#comment-16253975
 ] 

ASF GitHub Bot commented on METRON-1289:


Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/824
  
## Patch neither alert and status
Create a metaalerts and get the GUID for the following steps.

### Patch in new field
```
/api/v1/update/patch

curl -X PATCH --header 'Content-Type: application/json' --header 'Accept: 
*/*' -d '{
  "guid": "00eae5ba-6137-4601-ae3a-fbf0003e58e6",
  "index": "metaalert_index",
  "patch": [
{
  "op": "add"
, "path": "/name"
, "value": "My new meta alert name"
}
  ],
  "sensorType": "metaalert"
}' 'http://node1:8082/api/v1/update/patch'
```

### Retrieve the meta alert and ensure it contains the new 'name' field
```
/api/v1/search/findOne

curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '{
  "guid": "00eae5ba-6137-4601-ae3a-fbf0003e58e6",
  "index": "metaalert_index",
  "sensorType": "metaalert"
}' 'http://node1:8082/api/v1/search/findOne'
```



> Alert fields are lost when a MetaAlert is created
> -
>
> Key: METRON-1289
> URL: https://issues.apache.org/jira/browse/METRON-1289
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a MetaAlert is created, the included results are being updated 
> incorrectly with only the "metaalert" field.  This causes subsequent findOne 
> operations to only return the "metaalert field for that alert.  All fields 
> should continue to be present.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

2017-11-15 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16253970#comment-16253970
 ] 

ASF GitHub Bot commented on METRON-1289:


Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/824
  
## Patch alert and status
Create a metaalerts and get the GUID for the following steps.

### Attempt to update status field
```
/api/v1/update/patch

curl -X PATCH --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '{
  "guid": "00eae5ba-6137-4601-ae3a-fbf0003e58e6",
  "index": "metaalert_index",
  "patch": [
{
  "op": "replace"
, "path": "/status"
, "value": "failure"
}
  ],
  "sensorType": "metaalert"
}' 'http://node1:8082/api/v1/update/patch'
```

Should return 
```
{
  "responseCode": 500,
  "message": "Meta alert patches are not allowed for /alert or /status 
paths.  Please use the add/remove alert or update status functions instead.",
  "fullMessage": "IllegalArgumentException: Meta alert patches are not 
allowed for /alert or /status paths.  Please use the add/remove alert or update 
status functions instead."
}
```

### Attempt to update the alert list
```
/api/v1/update/patch

curl -X PATCH --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '{
  "guid": "00eae5ba-6137-4601-ae3a-fbf0003e58e6",
  "index": "metaalert_index",
  "patch": [
{
  "op": "replace"
, "path": "/alert"
, "value": [{
"alertOne":"test"
},
{
 "alertTwo":"test"
}
]
}
  ],
  "sensorType": "metaalert"
}' 'http://node1:8082/api/v1/update/patch'
```

Should return 
```
{
  "responseCode": 500,
  "message": "Meta alert patches are not allowed for /alert or /status 
paths.  Please use the add/remove alert or update status functions instead.",
  "fullMessage": "IllegalArgumentException: Meta alert patches are not 
allowed for /alert or /status paths.  Please use the add/remove alert or update 
status functions instead."
}
```


> Alert fields are lost when a MetaAlert is created
> -
>
> Key: METRON-1289
> URL: https://issues.apache.org/jira/browse/METRON-1289
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a MetaAlert is created, the included results are being updated 
> incorrectly with only the "metaalert" field.  This causes subsequent findOne 
> operations to only return the "metaalert field for that alert.  All fields 
> should continue to be present.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

2017-11-15 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16253954#comment-16253954
 ] 

ASF GitHub Bot commented on METRON-1289:


Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/824
  
## Create meta alert with more than 10 alerts

### Find more than 10 alerts alerts

```
/api/v1/search/search

curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '{
  "fields": [
"guid"
  ],
  "from": 0,
  "indices": [
"snort"
  ],
  "query": "ip_dst_addr:192.168.66.121",
  "size": 11
}' 'http://node1:8082/api/v1/search/search'
```

Note the alerts that come back
```
62a53a5f-78e6-417a-8078-fb850baa3e84
876b72cb-9d72-4706-ac99-46cf91a8f359
5fd8b0a0-1f68-494a-ae20-633542a7045d
aee597a0-4255-499a-a4e2-ec7d756babb2
bf9e0e73-e64c-4759-b4f7-efad0a60be82

5ab9ce98-30db-45b2-a4e6-6489f136c839
0a4a7019-04f8-4a8c-af0b-d2e3908ecdc9
3423fdca-cefa-402a-b57d-60b75a15f046
2eb63002-e5f2-467a-8675-30b653ae145b
53f38cfd-aa89-4e49-ba5f-827eb73774cd

5f71a515-4976-4b0d-be85-bb6879b1e151
```

### Create a metaalert with the alerts
```
/api/v1/metaalert/create

curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '{
  "alerts": [
{
  "guid": "62a53a5f-78e6-417a-8078-fb850baa3e84",
  "index": "snort_index_2017.11.15.17",
  "sensorType": "snort"
},
 {
  "guid":"876b72cb-9d72-4706-ac99-46cf91a8f359",
  "index": "snort_index_2017.11.15.17",
  "sensorType": "snort"
},
{
  "guid": "5fd8b0a0-1f68-494a-ae20-633542a7045d",
  "index": "snort_index_2017.11.15.17",
  "sensorType": "snort"
},
{
  "guid": "aee597a0-4255-499a-a4e2-ec7d756babb2",
  "index": "snort_index_2017.11.15.17",
  "sensorType": "snort"
},
 {
  "guid":"bf9e0e73-e64c-4759-b4f7-efad0a60be82",
  "index": "snort_index_2017.11.15.17",
  "sensorType": "snort"
},
{
  "guid": "5ab9ce98-30db-45b2-a4e6-6489f136c839",
  "index": "snort_index_2017.11.15.17",
  "sensorType": "snort"
},
 {
  "guid":"0a4a7019-04f8-4a8c-af0b-d2e3908ecdc9",
  "index": "snort_index_2017.11.15.17",
  "sensorType": "snort"
},
{
  "guid": "3423fdca-cefa-402a-b57d-60b75a15f046",
  "index": "snort_index_2017.11.15.17",
  "sensorType": "snort"
},
{
  "guid": "2eb63002-e5f2-467a-8675-30b653ae145b",
  "index": "snort_index_2017.11.15.17",
  "sensorType": "snort"
},
 {
  "guid":"53f38cfd-aa89-4e49-ba5f-827eb73774cd",
  "index": "snort_index_2017.11.15.17",
  "sensorType": "snort"
},
 {
  "guid":"5f71a515-4976-4b0d-be85-bb6879b1e151",
  "index": "snort_index_2017.11.15.17",
  "sensorType": "snort"
}
  ],
  "groups": [
"test"
  ]
}' 'http://node1:8082/api/v1/metaalert/create'
```


Make sure to get the resulting guid from the response.
```
00eae5ba-6137-4601-ae3a-fbf0003e58e6
```

### Retrieve the meta alert and ensure it contains the provided alerts
```
/api/v1/search/findOne

curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '{
  "guid": "00eae5ba-6137-4601-ae3a-fbf0003e58e6",
  "index": "metaalert_index",
  "sensorType": "metaalert"
}' 'http://node1:8082/api/v1/search/findOne'
```

### Retrieve the child alerts
Ensure all alerts have the 'metaalerts' field populated with the parent 
meta alert.

```
/api/v1/search/findOne

curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '{
  "guid":"62a53a5f-78e6-417a-8078-fb850baa3e84",
  "sensorType": "snort"
}' 'http://node1:8082/api/v1/search/findOne'

... // 10 more times
```


> Alert fields are lost when a MetaAlert is created
> -
>
> Key: METRON-1289
> URL: https://issues.apache.org/jira/browse/METRON-1289
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a MetaAlert is created, the included results are being updated 
> incorrectly with only the "metaalert" field.  This causes subsequent findOne 
> operations to only return the "metaalert field for that alert.  All fields 
> should continue to be present.



--
This message was sent by

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

2017-11-15 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16253938#comment-16253938
 ] 

ASF GitHub Bot commented on METRON-1289:


Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/824
  
## Changing Metaalert status

### Find two alerts
```
/api/v1/search/search

curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '{
  "fields": [
"guid"
  ],
  "from": 0,
  "indices": [
"snort"
  ],
  "query": "ip_dst_addr:192.168.66.121",
  "size": 2
}' 'http://node1:8082/api/v1/search/search'
```
Results in two guids:

```
8b8314d4-277b-44dc-a75b-04b0cdcedb40
4ac26cf7-ab93-4940-9a0e-8e7f4d67736d
```

### Create a metaalert with the alerts
```
/api/v1/metaalert/create

curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '{
  "alerts": [
{
  "guid": "8b8314d4-277b-44dc-a75b-04b0cdcedb40",
  "index": "snort_index_2017.11.15.17",
  "sensorType": "snort"
},
 {
  "guid":"4ac26cf7-ab93-4940-9a0e-8e7f4d67736d",
  "index": "snort_index_2017.11.15.17",
  "sensorType": "snort"
}

  ],
  "groups": [
"test"
  ]
}' 'http://node1:8082/api/v1/metaalert/create'

```

Make sure to get the resulting guid from the response.
```
da60ccc9-9e79-45c5-be07-0a322c8791f0
```

### Retrieve the meta alert and ensure it contains the provided alerts
```
/api/v1/search/findOne

curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '{
  "guid": "da60ccc9-9e79-45c5-be07-0a322c8791f0",
  "index": "metaalert_index",
  "sensorType": "metaalert"
}' 'http://node1:8082/api/v1/search/findOne'
```

### Change the meta alert status to active
This makes sure nothing happens when we set active status to the same 
active status.
```
/api/v1/metaalert/update/status/{guid}/{status}

curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' 
'http://node1:8082/api/v1/metaalert/update/status/da60ccc9-9e79-45c5-be07-0a322c8791f0/active'
```

It should return false, as no status has changed.

### Retrieve the metaalert and ensure it is still active
Look for the 'status' field.

```
/api/v1/search/findOne

curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '{
  "guid": "da60ccc9-9e79-45c5-be07-0a322c8791f0",
  "index": "metaalert_index",
  "sensorType": "metaalert"
}' 'http://node1:8082/api/v1/search/findOne'
```

### Retrieve the child alerts
Ensure both alerts have the 'metaalerts' field populated with the parent 
meta alert.

```
/api/v1/search/findOne

curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '{
  "guid":"8b8314d4-277b-44dc-a75b-04b0cdcedb40",
  "sensorType": "snort"
}' 'http://node1:8082/api/v1/search/findOne'

curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '{
  "guid":"4ac26cf7-ab93-4940-9a0e-8e7f4d67736d",
  "sensorType": "snort"
}' 'http://node1:8082/api/v1/search/findOne'
```

### Change the meta alert status to inactive
Look for the 'status' field.  Running this once will set it to 'inactive'.  
Subsequent runs have no effect ('inactive' -> 'inactive' does nothing).
```
/api/v1/metaalert/update/status/{guid}/{status}

curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' 
'http://node1:8082/api/v1/metaalert/update/status/da60ccc9-9e79-45c5-be07-0a322c8791f0/inactive'
```

It should return true, because the status has changed.

### Retrieve the metaalert and ensure it is inactive
```
/api/v1/search/findOne

curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '{
  "guid": "da60ccc9-9e79-45c5-be07-0a322c8791f0",
  "index": "metaalert_index",
  "sensorType": "metaalert"
}' 'http://node1:8082/api/v1/search/findOne'
```

### Retrieve the child alerts
Ensure neither alert has the 'metaalerts' field populated with the parent 
meta alert.

```
/api/v1/search/findOne

curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '{
  "guid":"8b8314d4-277b-44dc-a75b-04b0cdcedb40",
  "sensorType": "snort"
}' 'http://node1:8082/api/v1/search/findOne'

curl -X POST --header 'Content-Type: application/json' --header

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

2017-11-15 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16253921#comment-16253921
 ] 

ASF GitHub Bot commented on METRON-1289:


Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/824
  
## Removing alerts and removing an already removed alert

### Find two alerts
```
/api/v1/search/search

curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '{
  "fields": [
"guid"
  ],
  "from": 0,
  "indices": [
"snort"
  ],
  "query": "ip_dst_addr:192.168.66.121",
  "size": 2
}' 'http://node1:8082/api/v1/search/search'
```
Results in two guids:

```
8b8314d4-277b-44dc-a75b-04b0cdcedb40
4ac26cf7-ab93-4940-9a0e-8e7f4d67736d
```

### Create a metaalert with the alerts
```
/api/v1/metaalert/create

curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '{
  "alerts": [
{
  "guid": "8b8314d4-277b-44dc-a75b-04b0cdcedb40",
  "index": "snort_index_2017.11.15.17",
  "sensorType": "snort"
},
 {
  "guid":"4ac26cf7-ab93-4940-9a0e-8e7f4d67736d",
  "index": "snort_index_2017.11.15.17",
  "sensorType": "snort"
}

  ],
  "groups": [
"test"
  ]
}' 'http://node1:8082/api/v1/metaalert/create'

```

Make sure to get the resulting guid from the response.
```
b25b663e-39c9-42d5-a52c-e6380235d43f
```

### Retrieve the meta alert and ensure it contains the provided alerts
```
/api/v1/search/findOne

curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '{
  "guid": "b25b663e-39c9-42d5-a52c-e6380235d43f",
  "index": "metaalert_index",
  "sensorType": "metaalert"
}' 'http://node1:8082/api/v1/search/findOne'
```

### Remove one of the alerts
```
/api/v1/metaalert/remove/alert

curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '{
  "alerts": [
{
  "guid": "8b8314d4-277b-44dc-a75b-04b0cdcedb40",
  "index": "snort_index_2017.11.15.17",
  "sensorType": "snort"
}
  ],
  "metaAlertGuid": "b25b663e-39c9-42d5-a52c-e6380235d43f"
}' 'http://node1:8082/api/v1/metaalert/remove/alert'
```

### Retrieve the meta alert again, and ensure it only contains the second 
alert.
```
/api/v1/search/findOne

curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '{
  "guid": "b25b663e-39c9-42d5-a52c-e6380235d43f",
  "index": "metaalert_index",
  "sensorType": "metaalert"
}' 'http://node1:8082/api/v1/search/findOne'
```

### Rerun the delete
```
/api/v1/metaalert/remove/alert

curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '{
  "alerts": [
{
  "guid": "8b8314d4-277b-44dc-a75b-04b0cdcedb40",
  "index": "snort_index_2017.11.15.17",
  "sensorType": "snort"
}
  ],
  "metaAlertGuid": "b25b663e-39c9-42d5-a52c-e6380235d43f"
}' 'http://node1:8082/api/v1/metaalert/remove/alert'
```

### Retrieve the meta alert again, and ensure it only contains the second 
alert.
```
/api/v1/search/findOne

curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '{
  "guid": "b25b663e-39c9-42d5-a52c-e6380235d43f",
  "index": "metaalert_index",
  "sensorType": "metaalert"
}' 'http://node1:8082/api/v1/search/findOne'
```

### Retrieve the child alerts
Ensure only the second alert has the 'metaalerts' field populated with the 
parent met alert.

```
/api/v1/search/findOne

curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '{
  "guid":"8b8314d4-277b-44dc-a75b-04b0cdcedb40",
  "sensorType": "snort"
}' 'http://node1:8082/api/v1/search/findOne'

curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '{
  "guid":"4ac26cf7-ab93-4940-9a0e-8e7f4d67736d",
  "sensorType": "snort"
}' 'http://node1:8082/api/v1/search/findOne'
```


> Alert fields are lost when a MetaAlert is created
> -
>
> Key: METRON-1289
> URL: https://issues.apache.org/jira/browse/METRON-1289
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a MetaAlert is created, the included results are being updated 
> incorrectly with

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

2017-11-15 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16253899#comment-16253899
 ] 

ASF GitHub Bot commented on METRON-1289:


Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/824
  
## Adding alerts and adding a preexisting alert

### Find two alerts
```
/api/v1/search/search

curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '{
  "fields": [
"guid"
  ],
  "from": 0,
  "indices": [
"snort"
  ],
  "query": "ip_dst_addr:192.168.66.121",
  "size": 2
}' 'http://node1:8082/api/v1/search/search'
```
Results in two guids:

```
8b8314d4-277b-44dc-a75b-04b0cdcedb40
4ac26cf7-ab93-4940-9a0e-8e7f4d67736d
```

### Create a metaalert with only one of the alerts
```
/api/v1/metaalert/create

curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '{
  "alerts": [
{
  "guid": "8b8314d4-277b-44dc-a75b-04b0cdcedb40",
  "index": "snort_index_2017.11.15.17",
  "sensorType": "snort"
}
  ],
  "groups": [
"test"
  ]
}' 'http://node1:8082/api/v1/metaalert/create'
```

Make sure to get the resulting guid from the response.
```
6a4affe4-02ce-4d25-80b1-bfc4ca53f557
```

### Retrieve the meta alert and ensure it contains the provided alert
```
/api/v1/search/findOne

curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '{
  "guid": "6a4affe4-02ce-4d25-80b1-bfc4ca53f557",
  "index": "metaalert_index",
  "sensorType": "metaalert"
}' 'http://node1:8082/api/v1/search/findOne'
```

### Retrieve the child alert and ensure the 'metaalerts' field contains the 
new GUID of the new metaalert
```
/api/v1/search/findOne

curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '{
  "guid":"8b8314d4-277b-44dc-a75b-04b0cdcedb40",
  "sensorType": "snort"
}' 'http://node1:8082/api/v1/search/findOne'
```

### Add the same alert to the meta alert
```
/api/v1/metaalert/add/alert

curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '{
  "alerts": [
{
  "guid": "8b8314d4-277b-44dc-a75b-04b0cdcedb40",
  "index": "snort_index_2017.11.15.17",
  "sensorType": "snort"
}
  ],
  "metaAlertGuid": "6a4affe4-02ce-4d25-80b1-bfc4ca53f557"
}' 'http://node1:8082/api/v1/metaalert/add/alert'
```

It should return "false" as no alerts have been added.  The meta alert 
should be retrieved again to validate.

### Run the add alert again but with the second alert
```
/api/v1/metaalert/add/alert

curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '{
  "alerts": [
{
  "guid": "8b8314d4-277b-44dc-a75b-04b0cdcedb40",
  "index": "snort_index_2017.11.15.17",
  "sensorType": "snort"
},
 {
  "guid":"4ac26cf7-ab93-4940-9a0e-8e7f4d67736d",
  "index": "snort_index_2017.11.15.17",
  "sensorType": "snort"
}
  ],
  "metaAlertGuid": "6a4affe4-02ce-4d25-80b1-bfc4ca53f557"
}' 'http://node1:8082/api/v1/metaalert/add/alert'
```

It should return true, because the second alert will be added.  The meta 
alert should be retrieved again to validate.

### Retrieve the child alerts
Ensure they have the 'metaalerts' field populated with their parent.

```
/api/v1/search/findOne

curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '{
  "guid":"8b8314d4-277b-44dc-a75b-04b0cdcedb40",
  "sensorType": "snort"
}' 'http://node1:8082/api/v1/search/findOne'

curl -X POST --header 'Content-Type: application/json' --header 'Accept: 
application/json' -d '{
  "guid":"4ac26cf7-ab93-4940-9a0e-8e7f4d67736d",
  "sensorType": "snort"
}' 'http://node1:8082/api/v1/search/findOne'
```



> Alert fields are lost when a MetaAlert is created
> -
>
> Key: METRON-1289
> URL: https://issues.apache.org/jira/browse/METRON-1289
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a MetaAlert is created, the included results are being updated 
> incorrectly with only the "metaalert" field.  This causes subsequent findOne 
> operations to only return the "metaalert field for that alert.  All fields 
> should continue to be present.



--

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

2017-11-15 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16253868#comment-16253868
 ] 

ASF GitHub Bot commented on METRON-1289:


Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/824#discussion_r151203066
  
--- Diff: 
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/MetaAlertController.java
 ---
@@ -60,5 +63,37 @@
   ) throws RestException {
 return new ResponseEntity<>(metaAlertService.create(createRequest), 
HttpStatus.OK);
   }
+
+  @ApiOperation(value = "Create a meta alert")
--- End diff --

The descriptions and so on need to be updated on all of these.  I know 
you're working on documentation, but I wanted to call it out so it doesn't slip 
through.


> Alert fields are lost when a MetaAlert is created
> -
>
> Key: METRON-1289
> URL: https://issues.apache.org/jira/browse/METRON-1289
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a MetaAlert is created, the included results are being updated 
> incorrectly with only the "metaalert" field.  This causes subsequent findOne 
> operations to only return the "metaalert field for that alert.  All fields 
> should continue to be present.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1212) Bundles and Maven Plugin

2017-11-15 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-1212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16253847#comment-16253847
 ] 

ASF GitHub Bot commented on METRON-1212:


Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/774#discussion_r151198323
  
--- Diff: bundles-maven-plugin/README.md ---
@@ -0,0 +1,230 @@
+
+# Apache Metron Bundle Maven Plugin
+
+Apache Metron Bundles Maven Plugin helps to build Bundles Archives to 
support the classloader isolation model.
+
+## Table of Contents
+
+- [Requirements](#requirements)
+- [Building](#building)
+- [Getting Stared](#getting_started)
+- [Getting Help](#getting-help)
+- [License](#license)
+
+## Requirements
+* JDK 1.7 or higher
+* Apache Maven 3.1.0 or higher
+
+## Building 
+
+Building the bundles-maven-plugin module should be rare since it will be 
released infrequently compared to
+the main 'metron' code tree.
+
+- Build with `mvn clean install`
+- Presuming you need to make use of changes to the bundles-maven-plugin 
module, you should next
+  go to the [metron](../metron) directory and follow its instructions. 
+
+## Getting Started
+
+While it is most likely
+that a maven archetype is being utilized to create bundles, as part of a 
toolkit etc, you may want to create on manually, or may need to create a 
project for use in an archetype.
+
+The plugin is utilized by setting the packaging of a maven module to 
'bundle'.
+
+```xml
+bundle
+```
+
+This means that when you package this module, any of it's non-provided 
dependencies will be packaged into the produced bundle ( and all of their 
non-provided dependencies as well).
+Since a library may not always be distributed as part of a bundle with all 
it's dependencies, the bundle module
+shall be a separate module from the actual classes and dependencies to be 
bundled.
+
+A very simple example layout for a project that utilizes bundles would be:
+
+```bash
+├── README.md
+├── pom.xml
+├── testapp
+│   ├── pom.xml
+│   ├── src
+│   │   ├── main
+│   │   │   └── java
+│   │   │   └── org
+│   │   │   └── apache
+│   │   │   └── test
+│   │   │   └── App.java
+│   │   └── test
+│   │   └── java
+│   │   └── org
+│   │   └── apache
+│   │   └── test
+│   │   └── AppTest.java
+└── testappbundle
+├── pom.xml
+```
+Where testappbundle is the bundle module that creates a bundle of testapp, 
and contains the following pom.xml:
+```xml
+
+http://maven.apache.org/POM/4.0.0;
+  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+  
+test.bundles.plugin
+org.apache.test
+1.0-SNAPSHOT
+  
+  4.0.0
+
+  test.app.bundle
+
+  
+  bundle
+  
+  
+  
+
+  org.apache.test
+  test.app
+  1.0-SNAPSHOT
+
+  
+
+  
+  
+
+  
+
+  org.apache.metron
+  bundles-maven-plugin
+  0.4.2
+  true
+  
+  
+
+  
+
+
+  
+org.apache.metron
+bundles-maven-plugin
+0.4.2
+true
+  
+
+  
+
+```
+When the module is packaged, it packages all of it's  non-provided 
dependencies into the bundles /bundled-dependencies directory.
+Thus, to create a bundle of a module's jar and that jar's non-provided 
dependencies, you add that module to your
+bundle modules dependencies.  You can unzip and examine the bundle in the 
target directory, and verify 
+it's contents, which should be similar to :
+
+```bash
+-> % tree .
+.
+└── META-INF
+├── MANIFEST.MF
+├── bundled-dependencies
+│   ├── log4j-1.2.17.jar
+│   ├── metron-common-0.4.1.jar
+│   ├── slf4j-api-1.7.7.jar
+│   ├── slf4j-log4j12-1.7.7.jar
+│   └── test.app-1.0-SNAPSHOT.jar
+└── maven
+└── org.apache.test
+└── test.app.bundle
+├── pom.properties
+└── pom.xml
+```
+
+This reflects the testapp project, which has these dependencies :
+
+```xml
+
+
+  org.apache.metron
+  metron-common
+  0.4.1
+
+
+  junit
+  junit
+  3.8.1
+  test
+
+  
+```
+metron-common

[jira] [Commented] (METRON-1212) Bundles and Maven Plugin

2017-11-15 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-1212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16253835#comment-16253835
 ] 

ASF GitHub Bot commented on METRON-1212:


Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/774#discussion_r151196532
  
--- Diff: bundles-maven-plugin/README.md ---
@@ -0,0 +1,230 @@
+
+# Apache Metron Bundle Maven Plugin
+
+Apache Metron Bundles Maven Plugin helps to build Bundles Archives to 
support the classloader isolation model.
+
+## Table of Contents
+
+- [Requirements](#requirements)
+- [Building](#building)
+- [Getting Stared](#getting_started)
+- [Getting Help](#getting-help)
+- [License](#license)
+
+## Requirements
+* JDK 1.7 or higher
+* Apache Maven 3.1.0 or higher
+
+## Building 
+
+Building the bundles-maven-plugin module should be rare since it will be 
released infrequently compared to
+the main 'metron' code tree.
+
+- Build with `mvn clean install`
+- Presuming you need to make use of changes to the bundles-maven-plugin 
module, you should next
+  go to the [metron](../metron) directory and follow its instructions. 
+
+## Getting Started
+
+While it is most likely
+that a maven archetype is being utilized to create bundles, as part of a 
toolkit etc, you may want to create on manually, or may need to create a 
project for use in an archetype.
+
+The plugin is utilized by setting the packaging of a maven module to 
'bundle'.
+
+```xml
+bundle
+```
+
+This means that when you package this module, any of it's non-provided 
dependencies will be packaged into the produced bundle ( and all of their 
non-provided dependencies as well).
+Since a library may not always be distributed as part of a bundle with all 
it's dependencies, the bundle module
+shall be a separate module from the actual classes and dependencies to be 
bundled.
+
+A very simple example layout for a project that utilizes bundles would be:
+
+```bash
+├── README.md
+├── pom.xml
+├── testapp
+│   ├── pom.xml
+│   ├── src
+│   │   ├── main
+│   │   │   └── java
+│   │   │   └── org
+│   │   │   └── apache
+│   │   │   └── test
+│   │   │   └── App.java
+│   │   └── test
+│   │   └── java
+│   │   └── org
+│   │   └── apache
+│   │   └── test
+│   │   └── AppTest.java
+└── testappbundle
+├── pom.xml
+```
+Where testappbundle is the bundle module that creates a bundle of testapp, 
and contains the following pom.xml:
+```xml
+
+http://maven.apache.org/POM/4.0.0;
+  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+  
+test.bundles.plugin
+org.apache.test
+1.0-SNAPSHOT
+  
+  4.0.0
+
+  test.app.bundle
+
+  
+  bundle
+  
+  
+  
+
+  org.apache.test
+  test.app
+  1.0-SNAPSHOT
+
+  
+
+  
+  
+
+  
+
+  org.apache.metron
+  bundles-maven-plugin
+  0.4.2
+  true
+  
+  
+
+  
+
+
+  
+org.apache.metron
+bundles-maven-plugin
+0.4.2
+true
+  
+
+  
+
+```
+When the module is packaged, it packages all of it's  non-provided 
dependencies into the bundles /bundled-dependencies directory.
+Thus, to create a bundle of a module's jar and that jar's non-provided 
dependencies, you add that module to your
+bundle modules dependencies.  You can unzip and examine the bundle in the 
target directory, and verify 
+it's contents, which should be similar to :
+
+```bash
+-> % tree .
+.
+└── META-INF
+├── MANIFEST.MF
+├── bundled-dependencies
+│   ├── log4j-1.2.17.jar
+│   ├── metron-common-0.4.1.jar
+│   ├── slf4j-api-1.7.7.jar
+│   ├── slf4j-log4j12-1.7.7.jar
+│   └── test.app-1.0-SNAPSHOT.jar
+└── maven
+└── org.apache.test
+└── test.app.bundle
+├── pom.properties
+└── pom.xml
+```
+
+This reflects the testapp project, which has these dependencies :
+
+```xml
+
+
+  org.apache.metron
+  metron-common
+  0.4.1
+
+
+  junit
+  junit
+  3.8.1
+  test
+
+  
+```
+metron-common

[jira] [Commented] (METRON-1302) Split up Indexing Topology into batch and random access sections

2017-11-15 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-1302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16253809#comment-16253809
 ] 

ASF GitHub Bot commented on METRON-1302:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/831#discussion_r151190365
  
--- Diff: 
metron-platform/metron-indexing/src/main/scripts/start_hdfs_topology.sh ---
@@ -0,0 +1,22 @@
+#!/bin/bash
+# 
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+# 
+# http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+METRON_VERSION=${project.version}
+METRON_HOME=/usr/metron/$METRON_VERSION
+TOPOLOGY_JAR=metron-elasticsearch-$METRON_VERSION-uber.jar
+storm jar $METRON_HOME/lib/$TOPOLOGY_JAR org.apache.storm.flux.Flux 
--remote $METRON_HOME/flux/indexing/batch/remote.yaml --filter 
$METRON_HOME/config/hdfs.properties
--- End diff --

Yeah, I prefer to keep the scripts `hdfs` and `elasticsearch`


> Split up Indexing Topology into batch and random access sections
> 
>
> Key: METRON-1302
> URL: https://issues.apache.org/jira/browse/METRON-1302
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>
> Currently we have the indexing topology handle writing to both random access 
> indices (e.g. elasticsearch) as well as batch write indices (e.g. hdfs).  We 
> should split these up and configure them separately.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1302) Split up Indexing Topology into batch and random access sections

2017-11-15 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-1302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16253808#comment-16253808
 ] 

ASF GitHub Bot commented on METRON-1302:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/831#discussion_r151190210
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_commands.py
 ---
@@ -162,34 +164,77 @@ def start_indexing_topology(self, env):
   self.__params.metron_principal_name,
   
execute_user=self.__params.metron_user)
 
-start_cmd_template = 
"""{0}/bin/start_elasticsearch_topology.sh \
--s {1} \
--z {2}"""
-start_cmd = 
start_cmd_template.format(self.__params.metron_home,
-  self.__indexing_topology,
-  
self.__params.zookeeper_quorum)
+start_cmd_template = """{0}/bin/start_hdfs_topology.sh"""
+start_cmd = 
start_cmd_template.format(self.__params.metron_home)
 Execute(start_cmd, user=self.__params.metron_user, tries=3, 
try_sleep=5, logoutput=True)
 
 else:
-Logger.info('Indexing topology already running')
+Logger.info('Batch Indexing topology already running')
 
-Logger.info('Finished starting indexing topology')
+Logger.info('Finished starting batch indexing topology')
 
-def stop_indexing_topology(self, env):
-Logger.info('Stopping ' + self.__indexing_topology)
+def start_random_access_indexing_topology(self, env):
+Logger.info('Starting ' + self.__random_access_indexing_topology)
--- End diff --

Definitely, @nickwallen I think we're thinking the same thing.  This is 
literally just the first baby step toward a broader vision of pluggable 
writers.  I think we probably want to start here with a discuss thread.  Where 
I was thinking of going next in here was merging the flux files into one and 
pulling the configs from zookeeper, which would be a step towards your vision.


> Split up Indexing Topology into batch and random access sections
> 
>
> Key: METRON-1302
> URL: https://issues.apache.org/jira/browse/METRON-1302
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>
> Currently we have the indexing topology handle writing to both random access 
> indices (e.g. elasticsearch) as well as batch write indices (e.g. hdfs).  We 
> should split these up and configure them separately.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1302) Split up Indexing Topology into batch and random access sections

2017-11-15 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-1302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16253804#comment-16253804
 ] 

ASF GitHub Bot commented on METRON-1302:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/831#discussion_r151189543
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-indexing-env.xml
 ---
@@ -73,18 +98,31 @@
 Indexing Update Column Family
 
 
-indexing_workers
+ra_indexing_workers
+Number of Indexing Topology Workers
--- End diff --

That sounds good.  Which one would you guys prefer:
* display names in the mpack of "Elasticsearch" and "HDFS"
* variable prefixes of `es` and `hdfs`

I think I'd prefer the first as I really have been trying to set us up to a 
world where we can make things pluggable (this is the first baby step).


> Split up Indexing Topology into batch and random access sections
> 
>
> Key: METRON-1302
> URL: https://issues.apache.org/jira/browse/METRON-1302
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>
> Currently we have the indexing topology handle writing to both random access 
> indices (e.g. elasticsearch) as well as batch write indices (e.g. hdfs).  We 
> should split these up and configure them separately.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1302) Split up Indexing Topology into batch and random access sections

2017-11-15 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-1302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16253805#comment-16253805
 ] 

ASF GitHub Bot commented on METRON-1302:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/831#discussion_r151189633
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-indexing-env.xml
 ---
@@ -73,18 +98,31 @@
 Indexing Update Column Family
 
 
-indexing_workers
+ra_indexing_workers
+Number of Indexing Topology Workers
--- End diff --

I should add, or some other 3rd option that I haven't anticipated. ;)


> Split up Indexing Topology into batch and random access sections
> 
>
> Key: METRON-1302
> URL: https://issues.apache.org/jira/browse/METRON-1302
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>
> Currently we have the indexing topology handle writing to both random access 
> indices (e.g. elasticsearch) as well as batch write indices (e.g. hdfs).  We 
> should split these up and configure them separately.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1212) Bundles and Maven Plugin

2017-11-15 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-1212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16253791#comment-16253791
 ] 

ASF GitHub Bot commented on METRON-1212:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/774
  
@nickwallen I have added a usage document in the last commit.  Is this what 
you are looking for?


> Bundles and Maven Plugin
> 
>
> Key: METRON-1212
> URL: https://issues.apache.org/jira/browse/METRON-1212
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>
> The first effort will be to land the bundle system and supporting maven 
> plugin on master



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (METRON-908) Improve ES indexing for bro logs

2017-11-15 Thread Jon Zeolla (JIRA)


 [ 
https://issues.apache.org/jira/browse/METRON-908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jon Zeolla reassigned METRON-908:
-

Assignee: (was: Jon Zeolla)

> Improve ES indexing for bro logs
> 
>
> Key: METRON-908
> URL: https://issues.apache.org/jira/browse/METRON-908
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>
> Right now ES indexing is rather simple.  Because we know the schema of the 
> bro logs, we should investigate and implement more useful indexing and 
> tokenization methods.
> An initial offhand idea is to consider the path hierarchy tokenizer 
> https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-pathhierarchy-tokenizer.html#analysis-pathhierarchy-tokenizer
> We should also create a custom tokenizer for comma separated values, which 
> are how bro logs write sets into a field.  
> http://stackoverflow.com/questions/31143136/indexing-a-comma-separated-value-field-in-elastic-search



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1291) Kafka produce REST endpoint does not work in a Kerberized cluster

2017-11-15 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-1291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16253568#comment-16253568
 ] 

ASF GitHub Bot commented on METRON-1291:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/826#discussion_r151149320
  
--- Diff: 
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/KafkaConfig.java
 ---
@@ -108,6 +108,9 @@ public ZkUtils zkUtils() {
 producerConfig.put("key.serializer", 
"org.apache.kafka.common.serialization.StringSerializer");
 producerConfig.put("value.serializer", 
"org.apache.kafka.common.serialization.StringSerializer");
 producerConfig.put("request.required.acks", 1);
+if 
(environment.getProperty(MetronRestConstants.KERBEROS_ENABLED_SPRING_PROPERTY, 
Boolean.class, false)) {
+  producerConfig.put("security.protocol", "SASL_PLAINTEXT");
--- End diff --

I submitted a [PR](https://github.com/merrimanr/incubator-metron/pull/35) 
against your branch with my proposed solution to this.  It fixes the issue for 
both the producer and consumer configs.  If you choose to accept it, you can do 
it without my attribution.


> Kafka produce REST endpoint does not work in a Kerberized cluster
> -
>
> Key: METRON-1291
> URL: https://issues.apache.org/jira/browse/METRON-1291
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

2017-11-15 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16253533#comment-16253533
 ] 

ASF GitHub Bot commented on METRON-1289:


Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/824
  
Double check me on that logic though. I could definitely be masking an off 
by one error there.


> Alert fields are lost when a MetaAlert is created
> -
>
> Key: METRON-1289
> URL: https://issues.apache.org/jira/browse/METRON-1289
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a MetaAlert is created, the included results are being updated 
> incorrectly with only the "metaalert" field.  This causes subsequent findOne 
> operations to only return the "metaalert field for that alert.  All fields 
> should continue to be present.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

2017-11-15 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16253528#comment-16253528
 ] 

ASF GitHub Bot commented on METRON-1289:


Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/824
  
@merrimanr I'm okay with excluding metaalerts (although I need to review 
what you did there).  I wouldn't expect it to go down by two though.

Say I have two matches, I put one in a metaalert (so it should be hidden).  
I make the query again.  I would still expect to get the remaining, standalone 
match. The metaalert should never have showed up and the child alert should be 
hidden. So one result.  


> Alert fields are lost when a MetaAlert is created
> -
>
> Key: METRON-1289
> URL: https://issues.apache.org/jira/browse/METRON-1289
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a MetaAlert is created, the included results are being updated 
> incorrectly with only the "metaalert" field.  This causes subsequent findOne 
> operations to only return the "metaalert field for that alert.  All fields 
> should continue to be present.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-813) Migrate metron-bro-plugin-kafka to be a bro package

2017-11-15 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16253416#comment-16253416
 ] 

ASF GitHub Bot commented on METRON-813:
---

GitHub user JonZeolla opened a pull request:

https://github.com/apache/metron-bro-plugin-kafka/pull/3

METRON-813: Migrate metron-bro-plugin-kafka to be a bro package

This should turn this repo into a bro package containing a bro plugin.

# Testing
The below testing plan assumes CentOS/RHEL, but can be tweaked to work on 
most mainstream linux distros.


1.  Install [Kafka 
0.10.0.1](https://kafka.apache.org/0101/documentation.html#quickstart), 
[Zookeeper 
3.4.6](https://zookeeper.apache.org/doc/r3.4.6/zookeeperStarted.html) (The same 
versions from [HDP 
2.5.5](https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.5/bk_release-notes/content/ch01s01.html)),
 and any package dependancies for testing.
```
# cd
# yum -y install java screen
# wget 
https://archive.apache.org/dist/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz{,.sha1}
# if [[ "$(sha1sum zookeeper-3.4.6.tar.gz)" == "$(cat 
zookeeper-3.4.6.tar.gz.sha1)" ]]; then tar -xvf zookeeper-3.4.6.tar.gz; else 
echo "sha1 sums do not match"; fi
# cd zookeeper-3.4.6
# cp conf/zoo_sample.cfg conf/zoo.cfg
# bin/zkServer.sh start
# cd
# wget 
https://mirrors.sonic.net/apache/kafka/0.10.0.1/kafka_2.10-0.10.0.1.tgz
# wget 
https://dist.apache.org/repos/dist/release/kafka/0.10.0.1/kafka_2.10-0.10.0.1.tgz.md5
# # Compare MD5s using md5sum
# tar -xvf kafka_2.10-0.10.0.1.tgz
# cd kafka_2.10-0.10.0.1
# bin/kafka-server-start.sh config/server.properties &
# bin/kafka-topics.sh --create --zookeeper localhost:2181 
--replication-factor 1 --partitions 1 --topic bro
```
1.  [Install bro 2.5.2](https://www.bro.org/sphinx/install/install.html) 
and 
[bro-pkg](http://bro-package-manager.readthedocs.io/en/stable/quickstart.html#installation).
  Make sure you are running at least bro 2.5 and bro-pkg 1.2.0, and configure 
bro-pkg properly.
```
# export PATH=$PATH:/usr/local/bro/bin
# bro --version
bro version 2.5.2
# bro-pkg --version
bro-pkg 1.2.2
# bro-pkg autoconfig
```
1.  Create a working directory and pull in this PR (selfishly using my 
branch of `checkout-pr` from 
[metron-commit-stuff](https://github.com/jonzeolla/metron-commit-stuff/tree/support-bro-plugin)
 to test some recent updates)
```
# git clone https://github.com/jonzeolla/metron-commit-stuff 
~/metron-commit-stuff
# cd ~/metron-commit-stuff
# git checkout support-bro-plugin
# cd
# ~/metron-commit-stuff/checkout-pr 3
Please select a repository:
  1) metron
  2) metron-bro-plugin-kafka
Selection [metron]: bro

```
1.  Install the package, and all of its dependancies, from the PR branch.  
Ensure it passes its unit tests.
```
# # Install librdkafka by following ONLY instruction 1 
[here](https://github.com/apache/metron-bro-plugin-kafka#installation)
# cd ~/metron-bro-plugin-kafka-pr3/
# bro-pkg install .
```
1.  Configure the plugin.
```
cat << EOF >> /usr/local/bro/share/bro/site/local.bro

# Activate metron-bro-plugin-kafka
@load metron-bro-plugin-kafka-pr3/Bro/Kafka

# Configure metron-bro-plugin-kafka
redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, Conn::LOG);
EOF 
```
1.  Run bro manually while monitoring kafka to confirm things are working.
```
# mkdir -p ~/brotmp/nitroba ~/brotmp/example-traffic
# wget https://www.bro.org/static/traces/exercise-traffic.pcap -O 
~/brotmp/example-traffic/exercise-traffic.pcap
# wget 
http://downloads.digitalcorpora.org/corpora/network-packet-dumps/2008-nitroba/nitroba.pcap
 -O ~/brotmp/nitroba/nitroba.pcap
# export PATH=$PATH:~/kafka_2.11-0.10.1.0/bin
# screen
# kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro
# # Ctrl+A c to make a new screen window
# cd ~/brotmp/example-traffic
# bro -r exercise-traffic.pcap /usr/local/bro/share/bro/site/local.bro 
-C
# # Use Ctrl+A n to cycle through screen sessions for validation.  To 
run another test, on your second window, do
# cd ~/brotmp/nitroba
# bro -r nitroba.pcap /usr/local/bro/share/bro/site/local.bro -C
```

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JonZeolla/metron-bro-plugin-kafka METRON-813

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron-bro-plugin-kafka/pull/3.patch

To close this pull request,

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

2017-11-15 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16253382#comment-16253382
 ] 

ASF GitHub Bot commented on METRON-1289:


Github user iraghumitra commented on the issue:

https://github.com/apache/metron/pull/824
  
+1 for functionality all the API's are working fine. Great job @merrimanr  
 


> Alert fields are lost when a MetaAlert is created
> -
>
> Key: METRON-1289
> URL: https://issues.apache.org/jira/browse/METRON-1289
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a MetaAlert is created, the included results are being updated 
> incorrectly with only the "metaalert" field.  This causes subsequent findOne 
> operations to only return the "metaalert field for that alert.  All fields 
> should continue to be present.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1316) Fastcapa Fails to Compile in Test Environment

[jira] [Commented] (METRON-1311) Service Check Should Check Elasticsearch Index Templates

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

[jira] [Commented] (METRON-1316) Fastcapa Fails to Compile in Test Environment

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

[jira] [Created] (METRON-1316) Fastcapa Fails to Compile in Test Environment

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

[jira] [Commented] (METRON-1212) Bundles and Maven Plugin

[jira] [Commented] (METRON-1212) Bundles and Maven Plugin

[jira] [Commented] (METRON-1302) Split up Indexing Topology into batch and random access sections

[jira] [Commented] (METRON-1302) Split up Indexing Topology into batch and random access sections

[jira] [Commented] (METRON-1302) Split up Indexing Topology into batch and random access sections

[jira] [Commented] (METRON-1302) Split up Indexing Topology into batch and random access sections

[jira] [Commented] (METRON-1212) Bundles and Maven Plugin

[jira] [Assigned] (METRON-908) Improve ES indexing for bro logs

[jira] [Commented] (METRON-1291) Kafka produce REST endpoint does not work in a Kerberized cluster

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

[jira] [Commented] (METRON-813) Migrate metron-bro-plugin-kafka to be a bro package

[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created

29 matches

Site Navigation

Mail list logo

Footer information