[jira] [Commented] (METRON-943) Create traffic connections report in zeppelin
[ https://issues.apache.org/jira/browse/METRON-943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003991#comment-16003991 ] ASF GitHub Bot commented on METRON-943: --- Github user james-sirota commented on the issue: https://github.com/apache/incubator-metron/pull/573 perfect. + 1 on the dashboard. Looks like travis failed, though > Create traffic connections report in zeppelin > - > > Key: METRON-943 > URL: https://issues.apache.org/jira/browse/METRON-943 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Assignee: Justin Leet > > User types in CIDR range into a search box > System generates connections report: > Volume of outbound traffic (cumulative) for every IP in range > Volume of inbound traffic (cumulative) for every IP in range -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-946) Full/Quick dev should default network_host to [ _local_, _site_ ]
[ https://issues.apache.org/jira/browse/METRON-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003955#comment-16003955 ] Matt Foley commented on METRON-946: --- [~dlyle] also recommends as part of this to do a cleanup step: removing metron-deployment/inventory/devimage-vagrant/ and all its contents. It is obsolete and unused, and might be confusing to developers looking for examples to follow. > Full/Quick dev should default network_host to [ _local_, _site_ ] > -- > > Key: METRON-946 > URL: https://issues.apache.org/jira/browse/METRON-946 > Project: Metron > Issue Type: Bug >Reporter: David M. Lyle > > Add override to single_node_vm.yml -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-946) Full/Quick dev should default network_host to [ _local_, _site_ ]
[ https://issues.apache.org/jira/browse/METRON-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003953#comment-16003953 ] ASF GitHub Bot commented on METRON-946: --- GitHub user mattf-horton opened a pull request: https://github.com/apache/incubator-metron/pull/578 METRON-946 Full/Quick dev should default network_host to [ _local_, _… ## Contributor Comments @dlyle65535 observed that even single-node installs should have ES bind an external interface (as well as loopback) so that external clients can access ES reports. Based on that, FullDev assumes "node1" is bound to an external i/f, which didn't work with the latest. This patch will fix the problem. To test, install FullDev and see that the ES config is good. Also install single node manually and check ES config still good. There shouldn't be any other observable changes. ## Pull Request Checklist ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [NA] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [NA] Have you written or updated unit tests and or integration tests to verify your changes? - [NA] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: NA You can merge this pull request into a Git repository by running: $ git pull https://github.com/mattf-horton/incubator-metron METRON-946 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/578.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #578 commit 059cf98ce044c61935c8e8d82f46b57af42c8d50 Author: mattf-horton Date: 2017-05-10T02:45:44Z METRON-946 Full/Quick dev should default network_host to [ _local_, _site_ ] > Full/Quick dev should default network_host to [ _local_, _site_ ] > -- > > Key: METRON-946 > URL: https://issues.apache.org/jira/browse/METRON-946 > Project: Metron > Issue Type: Bug >Reporter: David M. Lyle > > Add override to single_node_vm.yml -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-946) Full/Quick dev should default network_host to [ _local_, _site_ ]
[ https://issues.apache.org/jira/browse/METRON-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003821#comment-16003821 ] Matt Foley commented on METRON-946: --- Turns out even single-node installs should have ES bind an external interface (as well as loopback) so that external clients can access ES reports. Based on that, FullDev assumes "node1" is bound to an external i/f, which didn't work with the latest. This patch will fix the problem. > Full/Quick dev should default network_host to [ _local_, _site_ ] > -- > > Key: METRON-946 > URL: https://issues.apache.org/jira/browse/METRON-946 > Project: Metron > Issue Type: Bug >Reporter: David M. Lyle > > Add override to single_node_vm.yml -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-746) Build Custom Checkstyle and IDE formatting settings
[
https://issues.apache.org/jira/browse/METRON-746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003709#comment-16003709
]
ASF GitHub Bot commented on METRON-746:
---
GitHub user justinleet opened a pull request:
https://github.com/apache/incubator-metron/pull/577
METRON-746: Build Custom Checkstyle and IDE formatting settings
## Contributor Comments
Pretty much just editing the POM to actually set up Checkstyle with the
Google Code Style, per the dev thread.
See [Google Java Style
Guide](https://google.github.io/styleguide/javaguide.html)
Note that the Checkstyle version is set to 7.7. I've tested it with the
plugin version and had no issues at all, but if anyone is concerned, we can
lower it to either the default or another version.
### IntelliJ warnings
To pull into IntelliJ, please see
https://github.com/jshiell/checkstyle-idea#configuration
Essentially the instructions are to:
- Install the plugin (Available in IntelliJ's directly)
- Go into Settings -> Other Settings -> Checkstyle
- Change "Checkstyle version" to 7.7.
- Apply. Otherwise the new file won't match the version and an error will
be thrown.
- Add a new Checkstyle file
- This can either be a local file or a remote one (remote will attempt to
update periodically, so if you don't want to repeatedly connect to the URL,
download the file somewhere and load it in). Either works. For 7.7, use
[google_checks.xml](https://raw.githubusercontent.com/checkstyle/checkstyle/master/src/main/resources/google_checks.xml)
- If you have errors, you most likely need to make sure you're set to the
right version and have applied it.
- Select the checkbox for the new style
- Apply
To ensure it's setup properly, open a file, e.g.
`metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java`
New warnings should show up that are based on Checkstyle, e.g.
```
Checkstyle: WhitespaceAround: 'if' is not followed by whitespace. Empty
blocks may only be represented as {} when not part of a multi-block statement
```
### Code formatting
There are two options for code formatting. One is to import the
google_checks.xml file from above. The other is to import Google's IntelliJ
editor settings for coverage. In practice, Google's direct IntelliJ settings
seem to handle autoformatting better. The instructions are essentially the same
for each.
Google's setup is described at
https://github.com/HPI-Information-Systems/Metanome/wiki/Installing-the-google-styleguide-settings-in-intellij-and-eclipse.
The direct Checkstyle can be imported similarly, but choosing the CheckStyle
Configuration when doing Import Scheme.
### Required Follow-up
This will require updating the wiki at
https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines
(Section 2.2). It needs to be switched to the Google code conventions, and the
instructions above for formatting and warnings should be pulled in.
### Testing
I still need to spin up full-dev, but to get the checkstyle report do:
```
mvn clean install -DskipTests
mvn site site:stage-deploy site:deploy -Dmaven.javadoc.skip=true
```
Navigate to file:///tmp/metron/site/checkstyle-aggregate.html in a browser.
You should see
`The following document contains the results of Checkstyle 7.7 with
google_checks.xml ruleset.` and a set of warnings. In practice, most of them
are fixed by batch autoformatting, per discussions, which will be a followup
task.
### Misc
No enforcement of Checkstyle is made, and in fact, by default
google_checks.xml sets everything to warning. The thought is that we'd need to
pretty significantly reduce the number of warnings, and probably think through
how we want to handle it.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.
Please refer also to our [Build Verification
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? P
[jira] [Commented] (METRON-945) Resolve merge conflict in metron.spec
[ https://issues.apache.org/jira/browse/METRON-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003658#comment-16003658 ] ASF GitHub Bot commented on METRON-945: --- GitHub user dlyle65535 opened a pull request: https://github.com/apache/incubator-metron/pull/576 METRON-945: Resolve merge conflict in metron.spec ## Contributor Comments This changeset will allow the rpm build to complete. Currently, master will fail trying to build the rpms. Run full-dev to verify. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [X] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [X] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [X] Have you included steps or a guide to how the change may be verified and tested manually? - [X] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [N/A] Have you written or updated unit tests and or integration tests to verify your changes? - [N/A] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? **Note** - you'll need to change network_host to [ _local_, _site_ ] for this to work, I've opened METRON-946 to track this. ### For documentation related changes: - [N/A] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dlyle65535/incubator-metron METRON-945 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/576.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #576 commit db009121cd552172e9e6bf7546e1b3df719da3d8 Author: David Lyle Date: 2017-05-09T22:13:43Z METRON-945: Resolve merge conflict in metron.spec > Resolve merge conflict in metron.spec > - > > Key: METRON-945 > URL: https://issues.apache.org/jira/browse/METRON-945 > Project: Metron > Issue Type: Bug >Reporter: David M. Lyle >Assignee: David M. Lyle > > METRON-913 mistakenly introduced a merge conflict in metron.spec. The correct > behavior was to keep both. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-946) Full/Quick dev should default network_host to [ _local_, _site_ ]
David M. Lyle created METRON-946: Summary: Full/Quick dev should default network_host to [ _local_, _site_ ] Key: METRON-946 URL: https://issues.apache.org/jira/browse/METRON-946 Project: Metron Issue Type: Bug Reporter: David M. Lyle Add override to single_node_vm.yml -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-945) Resolve merge conflict in metron.spec
David M. Lyle created METRON-945: Summary: Resolve merge conflict in metron.spec Key: METRON-945 URL: https://issues.apache.org/jira/browse/METRON-945 Project: Metron Issue Type: Bug Reporter: David M. Lyle METRON-913 mistakenly introduced a merge conflict in metron.spec. The correct behavior was to keep both. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (METRON-945) Resolve merge conflict in metron.spec
[ https://issues.apache.org/jira/browse/METRON-945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David M. Lyle reassigned METRON-945: Assignee: David M. Lyle > Resolve merge conflict in metron.spec > - > > Key: METRON-945 > URL: https://issues.apache.org/jira/browse/METRON-945 > Project: Metron > Issue Type: Bug >Reporter: David M. Lyle >Assignee: David M. Lyle > > METRON-913 mistakenly introduced a merge conflict in metron.spec. The correct > behavior was to keep both. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-931) Stellar REDUCE incorrectly returns null for fewer than 3 items in list
[
https://issues.apache.org/jira/browse/METRON-931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003536#comment-16003536
]
ASF GitHub Bot commented on METRON-931:
---
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/565
> Stellar REDUCE incorrectly returns null for fewer than 3 items in list
> --
>
> Key: METRON-931
> URL: https://issues.apache.org/jira/browse/METRON-931
> Project: Metron
> Issue Type: Bug
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>
> Examples:
> OK:
> {code}
> h1 := REDUCE(['foo', 'bar', 'baz'], (s, x) -> HLLP_ADD(s, x), HLLP_INIT(5, 6))
> s1 := REDUCE([1,2,3], (s, x) -> STATS_ADD(s, x), STATS_INIT())
> {code}
> Not OK:
> {code}
> h1 := REDUCE(['foo', 'bar'], (s, x) -> HLLP_ADD(s, x), HLLP_INIT(5, 6))
> s1 := REDUCE([1,2], (s, x) -> STATS_ADD(s, x), STATS_INIT())
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (METRON-944) markdown table causes infinite hanging in site-book's doxia-module-markdown
[ https://issues.apache.org/jira/browse/METRON-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003485#comment-16003485 ] ASF GitHub Bot commented on METRON-944: --- Github user asfgit closed the pull request at: https://github.com/apache/incubator-metron/pull/575 > markdown table causes infinite hanging in site-book's doxia-module-markdown > --- > > Key: METRON-944 > URL: https://issues.apache.org/jira/browse/METRON-944 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Assignee: Justin Leet > > It hangs forever. Don't like it. > Most likely https://issues.apache.org/jira/browse/DOXIA-554 > It's theoretically fixed, but the newest version isn't released yet, so we > can't use it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-944) markdown table causes infinite hanging in site-book's doxia-module-markdown
[ https://issues.apache.org/jira/browse/METRON-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003483#comment-16003483 ] ASF GitHub Bot commented on METRON-944: --- Github user mattf-horton commented on the issue: https://github.com/apache/incubator-metron/pull/575 +1 : confirmed current state same as one I tested. Thanks, @justinleet ! > markdown table causes infinite hanging in site-book's doxia-module-markdown > --- > > Key: METRON-944 > URL: https://issues.apache.org/jira/browse/METRON-944 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Assignee: Justin Leet > > It hangs forever. Don't like it. > Most likely https://issues.apache.org/jira/browse/DOXIA-554 > It's theoretically fixed, but the newest version isn't released yet, so we > can't use it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-897) Failed to Start Elasticsearch Deployed with MPack - SettingsException
[
https://issues.apache.org/jira/browse/METRON-897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003472#comment-16003472
]
Matt Foley commented on METRON-897:
---
METRON-905 is done, so marking this done too. Comments in PR#564 indicate this
was sufficient.
> Failed to Start Elasticsearch Deployed with MPack - SettingsException
> -
>
> Key: METRON-897
> URL: https://issues.apache.org/jira/browse/METRON-897
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Matt Foley
>
> {code}
> SettingsException[Failed to load settings from [elasticsearch.yml]]
> {code}
> When deploying Elasticsearch with the Ambari MPack, it fails to start. This
> is in an environment with 1 master node and 2 data nodes all running on 3
> separate hosts. The exception is this.
> {code}
> Apr 26 16:45:41 y113 systemd: Starting Elasticsearch...
> Apr 26 16:45:41 y113 systemd: Started Elasticsearch.
> Apr 26 16:45:41 y113 elasticsearch: Exception in thread "main"
> SettingsException[Failed to load settings from [elasticsearch.yml]]; nested:
> ParserException[while parsing a block mapping
> Apr 26 16:45:41 y113 elasticsearch: in 'reader', line 2, column 1:
> Apr 26 16:45:41 y113 elasticsearch: cluster:
> Apr 26 16:45:41 y113 elasticsearch: ^
> Apr 26 16:45:41 y113 elasticsearch: expected , but found FlowEntry
> Apr 26 16:45:41 y113 elasticsearch: in 'reader', line 67, column 26:
> Apr 26 16:45:41 y113 elasticsearch: network.host: "_lo:ipv4_","_eth0:ipv4_"
> Apr 26 16:45:41 y113 elasticsearch: ^
> Apr 26 16:45:41 y113 elasticsearch: ];
> Apr 26 16:45:41 y113 elasticsearch: Likely root cause: while parsing a block
> mapping
> Apr 26 16:45:41 y113 elasticsearch: in 'reader', line 2, column 1:
> Apr 26 16:45:41 y113 elasticsearch: cluster:
> Apr 26 16:45:41 y113 elasticsearch: ^
> Apr 26 16:45:41 y113 elasticsearch: expected , but found FlowEntry
> Apr 26 16:45:41 y113 elasticsearch: in 'reader', line 67, column 26:
> Apr 26 16:45:41 y113 elasticsearch: network.host: "_lo:ipv4_","_eth0:ipv4_"
> Apr 26 16:45:41 y113 elasticsearch: ^
> Apr 26 16:45:41 y113 elasticsearch: at
> com.fasterxml.jackson.dataformat.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingKey.produce(ParserImpl.java:570)
> Apr 26 16:45:41 y113 elasticsearch: at
> com.fasterxml.jackson.dataformat.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:158)
> Apr 26 16:45:41 y113 elasticsearch: at
> com.fasterxml.jackson.dataformat.yaml.snakeyaml.parser.ParserImpl.getEvent(ParserImpl.java:168)
> Apr 26 16:45:41 y113 elasticsearch: at
> com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:342)
> Apr 26 16:45:41 y113 elasticsearch: at
> org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:53)
> Apr 26 16:45:41 y113 elasticsearch: at
> org.elasticsearch.common.settings.loader.XContentSettingsLoader.serializeObject(XContentSettingsLoader.java:99)
> Apr 26 16:45:41 y113 elasticsearch: at
> org.elasticsearch.common.settings.loader.XContentSettingsLoader.load(XContentSettingsLoader.java:67)
> Apr 26 16:45:41 y113 elasticsearch: at
> org.elasticsearch.common.settings.loader.XContentSettingsLoader.load(XContentSettingsLoader.java:45)
> Apr 26 16:45:41 y113 elasticsearch: at
> org.elasticsearch.common.settings.loader.YamlSettingsLoader.load(YamlSettingsLoader.java:46)
> Apr 26 16:45:41 y113 elasticsearch: at
> org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1080)
> Apr 26 16:45:41 y113 elasticsearch: at
> org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1067)
> Apr 26 16:45:41 y113 elasticsearch: at
> org.elasticsearch.node.internal.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:88)
> Apr 26 16:45:41 y113 elasticsearch: at
> org.elasticsearch.bootstrap.Bootstrap.initialSettings(Bootstrap.java:202)
> Apr 26 16:45:41 y113 elasticsearch: at
> org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:241)
> Apr 26 16:45:41 y113 elasticsearch: at
> org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
> Apr 26 16:45:41 y113 elasticsearch: Refer to the log for complete error
> details.
> Apr 26 16:45:41 y113 systemd: elasticsearch.service: main process exited,
> code=exited, status=1/FAILURE
> Apr 26 16:45:41 y113 systemd: Unit elasticsearch.service entered failed state.
> Apr 26 16:45:41 y113 systemd: elasticsearch.service failed.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (METRON-905) Fix square-bracket behavior and default network interface bindings for ES
[ https://issues.apache.org/jira/browse/METRON-905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003468#comment-16003468 ] ASF GitHub Bot commented on METRON-905: --- Github user asfgit closed the pull request at: https://github.com/apache/incubator-metron/pull/564 > Fix square-bracket behavior and default network interface bindings for ES > - > > Key: METRON-905 > URL: https://issues.apache.org/jira/browse/METRON-905 > Project: Metron > Issue Type: Sub-task >Affects Versions: 0.3.1 >Reporter: Matt Foley >Assignee: Matt Foley > Fix For: 0.3.1 > > > Community agrees we should change Elasticsearch default binding to "all > interfaces" (0.0.0.0). > In addition, taking opportunity to fix the problem with square brackets on > these parameters: > * zen_discovery_ping_unicast_hosts > * network_host > For all other parameters in ES and other Metron sub-components, the square > brackets around lists are not needed in the parameter value, but are instead > provided in the template text. Making it so here, with the additional step > that if the user provides them (due to old habit) the extra square brackets > will be stripped before they cause a problem. This should make the change > less backward-incompatible. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-944) markdown table causes infinite hanging in site-book's doxia-module-markdown
[ https://issues.apache.org/jira/browse/METRON-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003403#comment-16003403 ] ASF GitHub Bot commented on METRON-944: --- Github user justinleet commented on the issue: https://github.com/apache/incubator-metron/pull/575 @mattf-horton Thanks for catching that. Merged it in. > markdown table causes infinite hanging in site-book's doxia-module-markdown > --- > > Key: METRON-944 > URL: https://issues.apache.org/jira/browse/METRON-944 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Assignee: Justin Leet > > It hangs forever. Don't like it. > Most likely https://issues.apache.org/jira/browse/DOXIA-554 > It's theoretically fixed, but the newest version isn't released yet, so we > can't use it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-944) markdown table causes infinite hanging in site-book's doxia-module-markdown
[ https://issues.apache.org/jira/browse/METRON-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003398#comment-16003398 ] ASF GitHub Bot commented on METRON-944: --- Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/575 +1 - generated site and opened in browser, clicked around. Thanks for jumping on this > markdown table causes infinite hanging in site-book's doxia-module-markdown > --- > > Key: METRON-944 > URL: https://issues.apache.org/jira/browse/METRON-944 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Assignee: Justin Leet > > It hangs forever. Don't like it. > Most likely https://issues.apache.org/jira/browse/DOXIA-554 > It's theoretically fixed, but the newest version isn't released yet, so we > can't use it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-944) markdown table causes infinite hanging in site-book's doxia-module-markdown
[ https://issues.apache.org/jira/browse/METRON-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003383#comment-16003383 ] ASF GitHub Bot commented on METRON-944: --- Github user mattf-horton commented on the issue: https://github.com/apache/incubator-metron/pull/575 @justinleet , there's also a problem on this page with code blocks after numeric list items. They need to be indented so they don't cause restarts of the numbering. I've submitted PR#5 to this fork to fix them. You may merge without attribution. I've regenerated, and confirmed that while site takes longer than it used to, it works and returns within a reasonable amount of time. Good work, Justin! > markdown table causes infinite hanging in site-book's doxia-module-markdown > --- > > Key: METRON-944 > URL: https://issues.apache.org/jira/browse/METRON-944 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Assignee: Justin Leet > > It hangs forever. Don't like it. > Most likely https://issues.apache.org/jira/browse/DOXIA-554 > It's theoretically fixed, but the newest version isn't released yet, so we > can't use it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (METRON-944) markdown table causes infinite hanging in site-book's doxia-module-markdown
[ https://issues.apache.org/jira/browse/METRON-944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-944: --- Description: It hangs forever. Don't like it. Most likely https://issues.apache.org/jira/browse/DOXIA-554 It's theoretically fixed, but the newest version isn't released yet, so we can't use it. was:It hangs forever. Don't like it. > markdown table causes infinite hanging in site-book's doxia-module-markdown > --- > > Key: METRON-944 > URL: https://issues.apache.org/jira/browse/METRON-944 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Assignee: Justin Leet > > It hangs forever. Don't like it. > Most likely https://issues.apache.org/jira/browse/DOXIA-554 > It's theoretically fixed, but the newest version isn't released yet, so we > can't use it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-944) markdown table causes infinite hanging in site-book's doxia-module-markdown
[ https://issues.apache.org/jira/browse/METRON-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003358#comment-16003358 ] ASF GitHub Bot commented on METRON-944: --- Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/575 +1 by inspection > markdown table causes infinite hanging in site-book's doxia-module-markdown > --- > > Key: METRON-944 > URL: https://issues.apache.org/jira/browse/METRON-944 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Assignee: Justin Leet > > It hangs forever. Don't like it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-944) markdown table causes infinite hanging in site-book's doxia-module-markdown
[ https://issues.apache.org/jira/browse/METRON-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003351#comment-16003351 ] ASF GitHub Bot commented on METRON-944: --- Github user justinleet commented on the issue: https://github.com/apache/incubator-metron/pull/575 Just spin up the site-book and make sure it a) completes (~30s) and b) doesn't look like hot garbage in the site-book > markdown table causes infinite hanging in site-book's doxia-module-markdown > --- > > Key: METRON-944 > URL: https://issues.apache.org/jira/browse/METRON-944 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Assignee: Justin Leet > > It hangs forever. Don't like it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-944) markdown table causes infinite hanging in site-book's doxia-module-markdown
[ https://issues.apache.org/jira/browse/METRON-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003348#comment-16003348 ] ASF GitHub Bot commented on METRON-944: --- GitHub user justinleet opened a pull request: https://github.com/apache/incubator-metron/pull/575 METRON-944: markdown table causes infinite hanging in site-book's doxia-module-markdown ## Contributor Comments Hangs less. Still not performant (takes ~30 seconds), but at least the tables aren't trash. Stripped out as many columns as reasonable, split into different tables, etc. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/justinleet/incubator-metron readme_fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/575.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #575 commit a7aa10a1651308e3b7dd625a3fe89b9a28e3047e Author: justinjleet Date: 2017-05-09T19:14:54Z reformatting table > markdown table causes infinite hanging in site-book's doxia-module-markdown > --- > > Key: METRON-944 > URL: https://issues.apache.org/jira/browse/METRON-944 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Assignee: Justin Leet > > It hangs forever. Don't like it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-944) markdown table causes infinite hanging in site-book's doxia-module-markdown
Justin Leet created METRON-944: -- Summary: markdown table causes infinite hanging in site-book's doxia-module-markdown Key: METRON-944 URL: https://issues.apache.org/jira/browse/METRON-944 Project: Metron Issue Type: Improvement Reporter: Justin Leet Assignee: Justin Leet It hangs forever. Don't like it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-914) Build Metron failure
[ https://issues.apache.org/jira/browse/METRON-914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003312#comment-16003312 ] Otto Fowler commented on METRON-914: I'm going to tag [~rmerriman], he knows more about node than I do > Build Metron failure > > > Key: METRON-914 > URL: https://issues.apache.org/jira/browse/METRON-914 > Project: Metron > Issue Type: Bug >Affects Versions: 0.4 > Environment: Ubuntu 16.04.1 >Reporter: GS Peter > Fix For: 0.4 > > Attachments: build.log.tar.gz, npm-debug.log > > Original Estimate: 504h > Remaining Estimate: 504h > > [INFO] metron-config .. FAILURE [04:08 > min] > [INFO] metron-rest-client . SKIPPED > [INFO] metron-rest SKIPPED > [INFO] > > [INFO] BUILD FAILURE > [INFO] > > [INFO] Total time: 58:04 min > [INFO] Finished at: 2017-04-28T15:57:16+07:00 > [INFO] Final Memory: 141M/588M > [INFO] > > [ERROR] Failed to execute goal > com.github.eirslett:frontend-maven-plugin:1.3:npm (npm install) on project > metron-config: Failed to run task: 'npm install' failed. (error code 1) -> > [Help 1] > [ERROR] > [ERROR] To see the full stack trace of the errors, re-run Maven with the -e > switch. > [ERROR] Re-run Maven using the -X switch to enable full debug logging. > [ERROR] > [ERROR] For more information about the errors and possible solutions, please > read the following articles: > [ERROR] [Help 1] > http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException > [ERROR] > [ERROR] After correcting the problems, you can resume the build with the > command > [ERROR] mvn -rf :metron-config -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-935) EC2 Deployment Failure - Could Not Create Blueprint
[
https://issues.apache.org/jira/browse/METRON-935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003240#comment-16003240
]
ASF GitHub Bot commented on METRON-935:
---
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/568
> EC2 Deployment Failure - Could Not Create Blueprint
> ---
>
> Key: METRON-935
> URL: https://issues.apache.org/jira/browse/METRON-935
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Priority: Blocker
>
> When deploying Metron with EC2, I ran into this problem.
> {code}
> TASK [ambari_config : Deploy cluster with Ambari;
> http://ec2-54-236-37-116.compute-1.amazonaws.com:8080] ***
> fatal: [ec2-54-236-37-116.compute-1.amazonaws.com]: FAILED! => {"changed":
> false, "failed": true,
> "msg": "Ambari client exception occurred: Could not create blueprint: request
> code 400,
> request message {\n \"status\" : 400,\n \"message\" : \"Blueprint
> configuration validation failed:
> Missing required properties. Specify a value for these properties in the
> blueprint configuration.
> {metron={metron-env=[metron_jdbc_platform, metron_jdbc_driver,
> metron_jdbc_username, metron_jdbc_url]}}\"\n}"}
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (METRON-913) Create IP Report in Zeppelin
[ https://issues.apache.org/jira/browse/METRON-913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003227#comment-16003227 ] ASF GitHub Bot commented on METRON-913: --- Github user asfgit closed the pull request at: https://github.com/apache/incubator-metron/pull/561 > Create IP Report in Zeppelin > > > Key: METRON-913 > URL: https://issues.apache.org/jira/browse/METRON-913 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.4 >Reporter: David M. Lyle >Assignee: David M. Lyle > Fix For: 0.4 > > > User types IP into a search box > System generates IP report > Most frequent connections (last 24 hours) > Recent connections (last 1 hour) > Top DNS queries made (last 24 hours) > All ports used (last 24 hours) > List of all HTTP user agents (last 24 hours) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (METRON-913) Create IP Report in Zeppelin
[ https://issues.apache.org/jira/browse/METRON-913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David M. Lyle updated METRON-913: - Fix Version/s: (was: Next + 1) 0.4 > Create IP Report in Zeppelin > > > Key: METRON-913 > URL: https://issues.apache.org/jira/browse/METRON-913 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.4 >Reporter: David M. Lyle >Assignee: David M. Lyle > Fix For: 0.4 > > > User types IP into a search box > System generates IP report > Most frequent connections (last 24 hours) > Recent connections (last 1 hour) > Top DNS queries made (last 24 hours) > All ports used (last 24 hours) > List of all HTTP user agents (last 24 hours) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-937) Pycapa - Consume Messages from Begin, End, or Stored Offsets
[ https://issues.apache.org/jira/browse/METRON-937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002929#comment-16002929 ] ASF GitHub Bot commented on METRON-937: --- GitHub user nickwallen reopened a pull request: https://github.com/apache/incubator-metron/pull/570 METRON-937 Pycapa Consume Messages from Begin, End, or Stored Offsets ## Changes * I have altered Pycapa so that it can consume messages from either the beginning, end or the previously stored offsets of a topic. This can be done using the `-o` or `--kafka-offset` flag that can be either `begin`, `end`, or `stored`. This defaults to `end`. * Added additional information when using the `--pretty-print` flag. It will now show the partition and offset for a packet, both on the producer-side (sending packets to Kafka) or the consumer-side (receiving packets from Kafka). ## Testing 1. Install Pycapa. 1. Launch a Kafka Broker on your localhost. For example, using Kafka installed via Brew on my Mac. ``` export KAFKA_HOME=/usr/local/Cellar/kafka/0.10.1.1/ export KAFKA_CONF=$KAFKA_HOME/libexec/config zookeeper-server-start $KAFKA_CONF/zookeeper.properties kafka-server-start $KAFKA_CONF/server.properties ``` 1. Produce some packets. Assumes you're actively using `en0` and Kafka is at `localhost:9092`. Let it run until it completes and captures 50 packets. ``` pycapa --producer \ --interface en0 \ --kafka-broker localhost:9092 \ --kafka-topic pcap \ --max-packets 50 ``` 1. Run the consumer. The consumer will not consume any packets. It starts from the end of the topic, by default, and since there are no active producers, there is nothing new to consume. ``` pycapa --consumer \ --kafka-broker localhost:9092 \ --kafka-topic pcap --pretty-print 1 ``` 1. Run the consumer again, but start from the beginning of the topic. You should be able to consume all 50 packets. ``` pycapa --consumer \ --kafka-broker localhost:9092 \ --kafka-topic pcap --pretty-print 1 --kafka-offset begin ``` ## Pull Request Checklist - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via: - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: You can merge this pull request into a Git repository by running: $ git pull https://github.com/nickwallen/incubator-metron METRON-937 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/570.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #570 commit 23af8e9535ee396ce1a722b51672d4e8a9a69b4d Author: Nick Allen Date: 2017-05-05T21:51:05Z METRON-937 Pycapa Consume Messages from Begin, End, or Stored Offsets > Pycapa - Consume Messages from Begin, End, or Stored Offsets > > > Key: METRON-937 > URL: https://issues.apache.org/jira/browse/METRON-937 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen > > Enable Pycapa to consume
[jira] [Commented] (METRON-937) Pycapa - Consume Messages from Begin, End, or Stored Offsets
[ https://issues.apache.org/jira/browse/METRON-937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002927#comment-16002927 ] ASF GitHub Bot commented on METRON-937: --- Github user nickwallen commented on the issue: https://github.com/apache/incubator-metron/pull/570 travis! ``` --- T E S T S --- Running org.apache.metron.pcap.integration.PcapTopologyIntegrationTest Formatting using clusterid: testClusterID Processing: ip_dst_addr => 207.28.210.1 Processing: protocol => foo Processing: ip_dst_port => 22 Processing: ip_dst_port => 22 Processing: ip_dst_port => 22 Processing: ip_dst_port => 22 Processing: ip_dst_port => 22 Processing: ip_dst_port => 22 Processing: ip_dst_port => 22 Processing: ip_dst_port => 22 Processing: ip_dst_port => 22 Processing: ip_dst_port => 22 Ended 2017-05-09 14:39:20 ERROR ReadClusterState:345 - Error trying to shutdown workers in Thread[SLOT_1027,5,main] java.lang.IllegalStateException: It took over 6ms to shut down slot Thread[SLOT_1027,5,main] at org.apache.storm.daemon.supervisor.ReadClusterState$1.call(ReadClusterState.java:294) at org.apache.storm.daemon.supervisor.ReadClusterState$1.call(ReadClusterState.java:292) at org.apache.storm.daemon.supervisor.ReadClusterState$3.call(ReadClusterState.java:307) at org.apache.storm.daemon.supervisor.ReadClusterState$3.call(ReadClusterState.java:304) at org.apache.storm.daemon.supervisor.ReadClusterState.shutdownAllWorkers(ReadClusterState.java:333) at org.apache.storm.daemon.supervisor.Supervisor.shutdownAllWorkers(Supervisor.java:330) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) at org.apache.storm.testing$kill_local_storm_cluster.invoke(testing.clj:204) at org.apache.storm.LocalCluster$_shutdown.invoke(LocalCluster.clj:66) at org.apache.storm.LocalCluster.shutdown(Unknown Source) at org.apache.metron.integration.components.FluxTopologyComponent.stop(FluxTopologyComponent.java:159) ``` > Pycapa - Consume Messages from Begin, End, or Stored Offsets > > > Key: METRON-937 > URL: https://issues.apache.org/jira/browse/METRON-937 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen > > Enable Pycapa to consume packets from either the beginning, ending or stored > offsets for a topic. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-937) Pycapa - Consume Messages from Begin, End, or Stored Offsets
[ https://issues.apache.org/jira/browse/METRON-937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002928#comment-16002928 ] ASF GitHub Bot commented on METRON-937: --- Github user nickwallen closed the pull request at: https://github.com/apache/incubator-metron/pull/570 > Pycapa - Consume Messages from Begin, End, or Stored Offsets > > > Key: METRON-937 > URL: https://issues.apache.org/jira/browse/METRON-937 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen > > Enable Pycapa to consume packets from either the beginning, ending or stored > offsets for a topic. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-912) Metron vagrant setup steps no longer work
[ https://issues.apache.org/jira/browse/METRON-912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002919#comment-16002919 ] ASF GitHub Bot commented on METRON-912: --- Github user nickwallen commented on the issue: https://github.com/apache/incubator-metron/pull/560 > I have not been able to find anybody who got Metron working on vagrant after only installing docker via `brew cask install docker`. Installing Docker with `brew cask install docker` and then building Metron RPMs works for me (that's the part that needs Docker). What problems have you run into with this? You do need to start the Docker daemon, but I have to do that all the time (like when I reboot my laptop or just get tired of looking at the whale.) > Metron vagrant setup steps no longer work > - > > Key: METRON-912 > URL: https://issues.apache.org/jira/browse/METRON-912 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla > > The latest maven prereq steps no longer work, as they install versions > inconsistent with what we have tested. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-934) Component and task id are missing in the indexing topology Hdfs file names
[ https://issues.apache.org/jira/browse/METRON-934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002904#comment-16002904 ] ASF GitHub Bot commented on METRON-934: --- GitHub user dlyle65535 opened a pull request: https://github.com/apache/incubator-metron/pull/574 METRON-934: Component and task id are missing in the indexing topology Hdfs file names. ## Contributor Comments Prior to this changeset, all files written to hdfs will have the format enrichment-null-0-0-timestamp.json. This causes lock errors in environments that use more than 1 task per writer. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [X] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [X] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [X] Have you included steps or a guide to how the change may be verified and tested manually? - [X] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [X] Have you written or updated unit tests and or integration tests to verify your changes? - [N/A] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [N/A] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dlyle65535/incubator-metron METRON-934 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/574.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #574 commit f24575b4345efde9c07d77b126850743af77ea3d Author: David Lyle Date: 2017-05-09T15:35:44Z METRON-934: Component and task id are missing in the indexing topology Hdfs file names. > Component and task id are missing in the indexing topology Hdfs file names > -- > > Key: METRON-934 > URL: https://issues.apache.org/jira/browse/METRON-934 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman > > The HdfsWriter class creates a new PathExtensionFileNameFormat object and > passes it to SourceHandler class. However, the > PathExtensionFileNameFormat.prepare method is never called, resulting in > uninitialized DefaultFileNameFormat.componentId and > DefaultFileNameFormat.taskId fields. > This becomes a problem as soon as a BulkMessageWriterBolt with HdfsWriter > configured is given more than one task because the HDFS file name is not > unique among tasks (taskId defaults to 0 if not initialized). -- This message was sent by Atlassian JIRA (v6.3.15#
[jira] [Updated] (METRON-934) Component and task id are missing in the indexing topology Hdfs file names
[ https://issues.apache.org/jira/browse/METRON-934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David M. Lyle updated METRON-934: - Summary: Component and task id are missing in the indexing topology Hdfs file names (was: Component and task id are missing in the ndexing topology Hdfs file names) > Component and task id are missing in the indexing topology Hdfs file names > -- > > Key: METRON-934 > URL: https://issues.apache.org/jira/browse/METRON-934 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman > > The HdfsWriter class creates a new PathExtensionFileNameFormat object and > passes it to SourceHandler class. However, the > PathExtensionFileNameFormat.prepare method is never called, resulting in > uninitialized DefaultFileNameFormat.componentId and > DefaultFileNameFormat.taskId fields. > This becomes a problem as soon as a BulkMessageWriterBolt with HdfsWriter > configured is given more than one task because the HDFS file name is not > unique among tasks (taskId defaults to 0 if not initialized). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-840) All "ambari_*" hosts need to have a /localrepo folder
[ https://issues.apache.org/jira/browse/METRON-840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002818#comment-16002818 ] ASF GitHub Bot commented on METRON-840: --- Github user asfgit closed the pull request at: https://github.com/apache/incubator-metron/pull/563 > All "ambari_*" hosts need to have a /localrepo folder > - > > Key: METRON-840 > URL: https://issues.apache.org/jira/browse/METRON-840 > Project: Metron > Issue Type: Bug >Reporter: David M. Lyle >Assignee: David M. Lyle >Priority: Blocker > Fix For: 0.4 > > > Ambari pushes out the repo definitions to all hosts. If the /localrepo folder > doesn't exist and have a repoinfo subfolder, installation will fail. > Currently, installation fails on EC2. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-935) EC2 Deployment Failure - Could Not Create Blueprint
[
https://issues.apache.org/jira/browse/METRON-935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002811#comment-16002811
]
ASF GitHub Bot commented on METRON-935:
---
Github user merrimanr closed the pull request at:
https://github.com/apache/incubator-metron/pull/568
> EC2 Deployment Failure - Could Not Create Blueprint
> ---
>
> Key: METRON-935
> URL: https://issues.apache.org/jira/browse/METRON-935
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Priority: Blocker
>
> When deploying Metron with EC2, I ran into this problem.
> {code}
> TASK [ambari_config : Deploy cluster with Ambari;
> http://ec2-54-236-37-116.compute-1.amazonaws.com:8080] ***
> fatal: [ec2-54-236-37-116.compute-1.amazonaws.com]: FAILED! => {"changed":
> false, "failed": true,
> "msg": "Ambari client exception occurred: Could not create blueprint: request
> code 400,
> request message {\n \"status\" : 400,\n \"message\" : \"Blueprint
> configuration validation failed:
> Missing required properties. Specify a value for these properties in the
> blueprint configuration.
> {metron={metron-env=[metron_jdbc_platform, metron_jdbc_driver,
> metron_jdbc_username, metron_jdbc_url]}}\"\n}"}
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (METRON-935) EC2 Deployment Failure - Could Not Create Blueprint
[
https://issues.apache.org/jira/browse/METRON-935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002812#comment-16002812
]
ASF GitHub Bot commented on METRON-935:
---
GitHub user merrimanr reopened a pull request:
https://github.com/apache/incubator-metron/pull/568
METRON-935: EC2 Deployment Failure - Could Not Create Blueprint
## Contributor Comments
Has not been tested on ec2 yet but should resolve this issue or at least
fix the error in the Jira description.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.
Please refer also to our [Build Verification
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been
executed in the root incubating-metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in
which it is rendered by building and verifying the site-book? If not then run
the following commands and the verify changes via
`site-book/target/site/index.html`:
```
cd site-book
mvn site
```
Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up
for your personal repository such that your branches are built there before
submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/merrimanr/incubator-metron METRON-935
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/568.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #568
commit b19f11bdea38e53f70b56f2fd8f1ee737c9af66a
Author: merrimanr
Date: 2017-05-08T12:52:20Z
Added jdbc defaults
> EC2 Deployment Failure - Could Not Create Blueprint
> ---
>
> Key: METRON-935
> URL: https://issues.apache.org/jira/browse/METRON-935
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Priority: Blocker
>
> When deploying Metron with EC2, I ran into this problem.
> {code}
> TASK [ambari_config : Deploy cluster with Ambari;
> http://ec2-54-236-37-116.compute-1.amazonaws.com:8080] ***
> fatal: [ec2-54-236-37-116.compute-1.amazonaws.com]: FAILED! => {"changed":
> false, "failed": true,
> "msg": "Ambari client exception occurred: Could not create blueprint: request
> code 400,
> request message {\n \"status\" : 400,\n \"message\" : \"Blueprint
> configuration validation failed:
> Missing required properties. Specify a value for these properties in the
> blueprint configuration.
> {metron={metron-env=[metron_jdbc_platform, metron_jdbc_driver,
> metron_jdbc_username, metron_jdbc_url]}}\"\n}"}
> {code}
--
This message wa
[jira] [Assigned] (METRON-906) Rest service storm configuration does not allow for proper URLs
[ https://issues.apache.org/jira/browse/METRON-906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-906: -- Assignee: Justin Leet > Rest service storm configuration does not allow for proper URLs > --- > > Key: METRON-906 > URL: https://issues.apache.org/jira/browse/METRON-906 > Project: Metron > Issue Type: Bug >Affects Versions: 0.3.1 >Reporter: Simon Elliston Ball >Assignee: Justin Leet > Labels: newbie > > The storm.ui.url field does not accept a URL, but instead accepts a > hostname, port and path, without the schema. The code then prepends http:// > This is somewhat confusing, and prevents us later moving to support ssl > access to the api. The field should retain its name as url, but accept an > actual url. > Note this is a breaking config change, and will required documentation. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-833) Update MaaS documentation to explain how it interacts with kerberos
[ https://issues.apache.org/jira/browse/METRON-833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002789#comment-16002789 ] ASF GitHub Bot commented on METRON-833: --- Github user nickwallen commented on the issue: https://github.com/apache/incubator-metron/pull/520 @cestella Reminder to resolve the conflicts here so we can get this merged in. > Update MaaS documentation to explain how it interacts with kerberos > --- > > Key: METRON-833 > URL: https://issues.apache.org/jira/browse/METRON-833 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-943) Create traffic connections report in zeppelin
[ https://issues.apache.org/jira/browse/METRON-943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002786#comment-16002786 ] ASF GitHub Bot commented on METRON-943: --- GitHub user justinleet opened a pull request: https://github.com/apache/incubator-metron/pull/573 METRON-943: Create traffic connections report in zeppelin ## Contributor Comments Adds a Zeppelin dashboard that lets the user get connection counts filtered by a CIDR block. The implementation is a little kludgy because Spark/Hive don't have easy operations to handle things like IPs and CIDRs. I'd have liked to keep it in one paragraph, but the clunkiness around handling the ips and cidr made that a pain that seemed significantly worse. If someone knows an easy way to take care of it, I can quick try it. Do we want/need similar paragraphs for the other sources? Given that we're primarily looking for volume of connections, it seems unnecessary, but it might be nice to have (possibly as a follow-on?) It's pretty to repeat the paragraphs as needed. Also, let me know if there are any verbiage changes (since that came up on another dashboard), or other adjustments that should be made. ## Test Plan To test, spin up full-dev. To get Yaf data, it'll be necessary to start the sensor-stub ``` service sensor-stubs start yaf ``` It'll also be necessary to add Yaf to the list of sensors run in Ambari. To do so, stop Metron, edit "Metron Parsers" to include "yaf" (or be yaf only). Start Metron and ensure a yaf topology is present. Let data flow through. Once some data has gone through, we'll need to have an instance of Zeppelin. Because of the size of the Vagrant instance, we'll want to shut down unneeded services. Shutdown Metron, Kibana, Storm, Kafka, and HBase. Install Zeppelin from "Actions - Add Service". It'll prompt you to install Spark and Hive, do so. Configuration is pretty trivial, all that's necessary is to set an arbitrary Hive database password. Let this run. The Hive service check likes to fail on our Vagrant, but it's benign (some impersonation configuration issue unrelated to actually running our queries). Ignore it and accept the installation. From Metron's "Service Actions", run the "Zeppelin Notebook Import", to load our notebooks into Zeppelin. Use the quick links to navigate to the Zeppelin UI. Go into the "Metron - Connection Volume Report" notebook. Queries can be made by CIDR, e.g. 192.0.0.0/8 to get total amount of traffic by source or by destination IP range. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/justinleet/incubator-metron zepp_conn_vol Alte
[jira] [Commented] (METRON-931) Stellar REDUCE incorrectly returns null for fewer than 3 items in list
[
https://issues.apache.org/jira/browse/METRON-931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002768#comment-16002768
]
ASF GitHub Bot commented on METRON-931:
---
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/565
+1 Good catch
> Stellar REDUCE incorrectly returns null for fewer than 3 items in list
> --
>
> Key: METRON-931
> URL: https://issues.apache.org/jira/browse/METRON-931
> Project: Metron
> Issue Type: Bug
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>
> Examples:
> OK:
> {code}
> h1 := REDUCE(['foo', 'bar', 'baz'], (s, x) -> HLLP_ADD(s, x), HLLP_INIT(5, 6))
> s1 := REDUCE([1,2,3], (s, x) -> STATS_ADD(s, x), STATS_INIT())
> {code}
> Not OK:
> {code}
> h1 := REDUCE(['foo', 'bar'], (s, x) -> HLLP_ADD(s, x), HLLP_INIT(5, 6))
> s1 := REDUCE([1,2], (s, x) -> STATS_ADD(s, x), STATS_INIT())
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (METRON-943) Create traffic connections report in zeppelin
Justin Leet created METRON-943: -- Summary: Create traffic connections report in zeppelin Key: METRON-943 URL: https://issues.apache.org/jira/browse/METRON-943 Project: Metron Issue Type: Improvement Reporter: Justin Leet Assignee: Justin Leet User types in CIDR range into a search box System generates connections report: Volume of outbound traffic (cumulative) for every IP in range Volume of inbound traffic (cumulative) for every IP in range -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-937) Pycapa - Consume Messages from Begin, End, or Stored Offsets
[ https://issues.apache.org/jira/browse/METRON-937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002740#comment-16002740 ] ASF GitHub Bot commented on METRON-937: --- GitHub user nickwallen reopened a pull request: https://github.com/apache/incubator-metron/pull/570 METRON-937 Pycapa Consume Messages from Begin, End, or Stored Offsets ## Changes * I have altered Pycapa so that it can consume messages from either the beginning, end or the previously stored offsets of a topic. This can be done using the `-o` or `--kafka-offset` flag that can be either `begin`, `end`, or `stored`. This defaults to `end`. * Added additional information when using the `--pretty-print` flag. It will now show the partition and offset for a packet, both on the producer-side (sending packets to Kafka) or the consumer-side (receiving packets from Kafka). ## Testing 1. Install Pycapa. 1. Launch a Kafka Broker on your localhost. For example, using Kafka installed via Brew on my Mac. ``` export KAFKA_HOME=/usr/local/Cellar/kafka/0.10.1.1/ export KAFKA_CONF=$KAFKA_HOME/libexec/config zookeeper-server-start $KAFKA_CONF/zookeeper.properties kafka-server-start $KAFKA_CONF/server.properties ``` 1. Produce some packets. Assumes you're actively using `en0` and Kafka is at `localhost:9092`. Let it run until it completes and captures 50 packets. ``` pycapa --producer \ --interface en0 \ --kafka-broker localhost:9092 \ --kafka-topic pcap \ --max-packets 50 ``` 1. Run the consumer. The consumer will not consume any packets. It starts from the end of the topic, by default, and since there are no active producers, there is nothing new to consume. ``` pycapa --consumer \ --kafka-broker localhost:9092 \ --kafka-topic pcap --pretty-print 1 ``` 1. Run the consumer again, but start from the beginning of the topic. You should be able to consume all 50 packets. ``` pycapa --consumer \ --kafka-broker localhost:9092 \ --kafka-topic pcap --pretty-print 1 --kafka-offset begin ``` ## Pull Request Checklist - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via: - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: You can merge this pull request into a Git repository by running: $ git pull https://github.com/nickwallen/incubator-metron METRON-937 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/570.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #570 commit 23af8e9535ee396ce1a722b51672d4e8a9a69b4d Author: Nick Allen Date: 2017-05-05T21:51:05Z METRON-937 Pycapa Consume Messages from Begin, End, or Stored Offsets > Pycapa - Consume Messages from Begin, End, or Stored Offsets > > > Key: METRON-937 > URL: https://issues.apache.org/jira/browse/METRON-937 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen > > Enable Pycapa to consume
[jira] [Commented] (METRON-937) Pycapa - Consume Messages from Begin, End, or Stored Offsets
[ https://issues.apache.org/jira/browse/METRON-937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002738#comment-16002738 ] ASF GitHub Bot commented on METRON-937: --- Github user nickwallen commented on the issue: https://github.com/apache/incubator-metron/pull/570 travis! ``` Running org.apache.metron.parsers.integration.YafIntegrationTest Running Sample Data Validation on sensorType yaf 2017-05-09 13:31:23 ERROR ParserTopologyComponent:114 - Storm slots didn't shut down entirely cleanly *sigh*. I gave them the old one-two-skadoo and killed the slots with prejudice. If tests fail, we'll have to find a better way of killing them. java.lang.IllegalStateException: It took over 6ms to shut down slot Thread[SLOT_1024,5,main] ``` > Pycapa - Consume Messages from Begin, End, or Stored Offsets > > > Key: METRON-937 > URL: https://issues.apache.org/jira/browse/METRON-937 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen > > Enable Pycapa to consume packets from either the beginning, ending or stored > offsets for a topic. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-937) Pycapa - Consume Messages from Begin, End, or Stored Offsets
[ https://issues.apache.org/jira/browse/METRON-937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002739#comment-16002739 ] ASF GitHub Bot commented on METRON-937: --- Github user nickwallen closed the pull request at: https://github.com/apache/incubator-metron/pull/570 > Pycapa - Consume Messages from Begin, End, or Stored Offsets > > > Key: METRON-937 > URL: https://issues.apache.org/jira/browse/METRON-937 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen > > Enable Pycapa to consume packets from either the beginning, ending or stored > offsets for a topic. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-942) REST Support for Parser Extensions
[ https://issues.apache.org/jira/browse/METRON-942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002731#comment-16002731 ] Otto Fowler commented on METRON-942: Should not be merged until METRON-777 is merged > REST Support for Parser Extensions > -- > > Key: METRON-942 > URL: https://issues.apache.org/jira/browse/METRON-942 > Project: Metron > Issue Type: New Feature >Reporter: Otto Fowler >Assignee: Otto Fowler > > If and after METRON-777 lands, support will be required for installing parser > extensions into the system, as well as uninstalling them. > The first step in this is to have support in metron-rest for this. > The rest interface should support installing a parser extension from the > assembly tar.gz produced by the archetype. > The result of the installation should be: > * the bundle is installed in the alternate extension library location in hdfs > * the patterns for all parsers in the extension should be deployed to hdfs > * the configurations for each pattern should be installed into ZK ( parser, > indexing, enrichment ) > * an extension configuration, that includes the parsers it produces, and the > default configurations ( the above and elasticsearch etc ) should be entered > into ZK > The reading of these configurations should also be supported. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-942) REST Support for Parser Extensions
Otto Fowler created METRON-942: -- Summary: REST Support for Parser Extensions Key: METRON-942 URL: https://issues.apache.org/jira/browse/METRON-942 Project: Metron Issue Type: New Feature Reporter: Otto Fowler Assignee: Otto Fowler If and after METRON-777 lands, support will be required for installing parser extensions into the system, as well as uninstalling them. The first step in this is to have support in metron-rest for this. The rest interface should support installing a parser extension from the assembly tar.gz produced by the archetype. The result of the installation should be: * the bundle is installed in the alternate extension library location in hdfs * the patterns for all parsers in the extension should be deployed to hdfs * the configurations for each pattern should be installed into ZK ( parser, indexing, enrichment ) * an extension configuration, that includes the parsers it produces, and the default configurations ( the above and elasticsearch etc ) should be entered into ZK The reading of these configurations should also be supported. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-902) ES improperly indexes Bro logs
[ https://issues.apache.org/jira/browse/METRON-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002651#comment-16002651 ] ASF GitHub Bot commented on METRON-902: --- Github user nickwallen commented on the issue: https://github.com/apache/incubator-metron/pull/555 +1 by inspection > ES improperly indexes Bro logs > -- > > Key: METRON-902 > URL: https://issues.apache.org/jira/browse/METRON-902 > Project: Metron > Issue Type: Bug >Reporter: Jon Zeolla >Assignee: Jon Zeolla > > It appears that an old issue has been reintroduced into the ES template for > indexing bro DNS logs. It is possible that other issues have been > reintroduced as well, as I have not yet reviewed the template holistically. > Initial fix: > https://github.com/apache/incubator-metron/blob/4bfb09c49fbc6204ce8b826887d99beff414f84a/metron-deployment/roles/metron_elasticsearch_templates/files/es_templates/bro_index.template#L165-L167 > Reintroduction: > https://github.com/apache/incubator-metron/blob/125dbef1e59ff808a62e4f5a7d265aafbcf6bf08/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/0.2.0BETA/package/files/bro_index.template#L165-L167 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-937) Pycapa - Consume Messages from Begin, End, or Stored Offsets
[ https://issues.apache.org/jira/browse/METRON-937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002647#comment-16002647 ] ASF GitHub Bot commented on METRON-937: --- Github user nickwallen closed the pull request at: https://github.com/apache/incubator-metron/pull/570 > Pycapa - Consume Messages from Begin, End, or Stored Offsets > > > Key: METRON-937 > URL: https://issues.apache.org/jira/browse/METRON-937 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen > > Enable Pycapa to consume packets from either the beginning, ending or stored > offsets for a topic. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-937) Pycapa - Consume Messages from Begin, End, or Stored Offsets
[ https://issues.apache.org/jira/browse/METRON-937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002646#comment-16002646 ] ASF GitHub Bot commented on METRON-937: --- Github user nickwallen commented on the issue: https://github.com/apache/incubator-metron/pull/570 kick travis > Pycapa - Consume Messages from Begin, End, or Stored Offsets > > > Key: METRON-937 > URL: https://issues.apache.org/jira/browse/METRON-937 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen > > Enable Pycapa to consume packets from either the beginning, ending or stored > offsets for a topic. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-937) Pycapa - Consume Messages from Begin, End, or Stored Offsets
[ https://issues.apache.org/jira/browse/METRON-937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002648#comment-16002648 ] ASF GitHub Bot commented on METRON-937: --- GitHub user nickwallen reopened a pull request: https://github.com/apache/incubator-metron/pull/570 METRON-937 Pycapa Consume Messages from Begin, End, or Stored Offsets ## Changes * I have altered Pycapa so that it can consume messages from either the beginning, end or the previously stored offsets of a topic. This can be done using the `-o` or `--kafka-offset` flag that can be either `begin`, `end`, or `stored`. This defaults to `end`. * Added additional information when using the `--pretty-print` flag. It will now show the partition and offset for a packet, both on the producer-side (sending packets to Kafka) or the consumer-side (receiving packets from Kafka). ## Testing 1. Install Pycapa. 1. Launch a Kafka Broker on your localhost. For example, using Kafka installed via Brew on my Mac. ``` export KAFKA_HOME=/usr/local/Cellar/kafka/0.10.1.1/ export KAFKA_CONF=$KAFKA_HOME/libexec/config zookeeper-server-start $KAFKA_CONF/zookeeper.properties kafka-server-start $KAFKA_CONF/server.properties ``` 1. Produce some packets. Assumes you're actively using `en0` and Kafka is at `localhost:9092`. Let it run until it completes and captures 50 packets. ``` pycapa --producer \ --interface en0 \ --kafka-broker localhost:9092 \ --kafka-topic pcap \ --max-packets 50 ``` 1. Run the consumer. The consumer will not consume any packets. It starts from the end of the topic, by default, and since there are no active producers, there is nothing new to consume. ``` pycapa --consumer \ --kafka-broker localhost:9092 \ --kafka-topic pcap --pretty-print 1 ``` 1. Run the consumer again, but start from the beginning of the topic. You should be able to consume all 50 packets. ``` pycapa --consumer \ --kafka-broker localhost:9092 \ --kafka-topic pcap --pretty-print 1 --kafka-offset begin ``` ## Pull Request Checklist - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via: - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: You can merge this pull request into a Git repository by running: $ git pull https://github.com/nickwallen/incubator-metron METRON-937 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/570.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #570 commit 23af8e9535ee396ce1a722b51672d4e8a9a69b4d Author: Nick Allen Date: 2017-05-05T21:51:05Z METRON-937 Pycapa Consume Messages from Begin, End, or Stored Offsets > Pycapa - Consume Messages from Begin, End, or Stored Offsets > > > Key: METRON-937 > URL: https://issues.apache.org/jira/browse/METRON-937 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen > > Enable Pycapa to consume
[jira] [Created] (METRON-941) native PaloAlto parser corrupts message when having a comma in the payload
Christian Tramnitz created METRON-941:
-
Summary: native PaloAlto parser corrupts message when having a
comma in the payload
Key: METRON-941
URL: https://issues.apache.org/jira/browse/METRON-941
Project: Metron
Issue Type: Bug
Affects Versions: 0.4
Environment: full-dev master
Reporter: Christian Tramnitz
Priority: Minor
When a data field contains a comma (i.e. the URL, not too uncommon), the
split(",") kicks in and the rest of the message if off by few fields due to
positional definition.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (METRON-940) problems with current Palo Alto schema for CEF parser
[
https://issues.apache.org/jira/browse/METRON-940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christian Tramnitz updated METRON-940:
--
Description:
The current Palo Alto parser (schema on top of CEF parser) seems to use a
custom field definition.
As far as I can tell there is no "standard" definition for a CEF message in
PaloAlto as the scheme can be freely defined. However, there is a documented
example and I would suggest to base the Metron parser upon this documented
definition (rather than a custom definition).
Alternatively we could come up with our message format definition for Palo Alto
CEF, but then we need to document what needs to be done on the Firewall to get
these.
This is a sanitized sample message for threat and traffic:
{noformat}
<14>1 2017-05-08T23:22:00+00:00 10.1.1.1 - - - - CEF:0|Palo Alto
Networks|PAN-OS|7.0.0|url|THREAT|1|rt=May 08 2017 23:22:00 GMT
deviceExternalId=000 src=192.168.1.2 dst=10.28.1.1
sourceTranslatedAddress=10.200.0.1 destinationTranslatedAddress=10.28.1.1
cs1Label=Rule cs1=rulename suser= duser= app=ssl cs3Label=Virtual System
cs3=vsys1 cs4Label=Source Zone cs4=private cs5Label=Destination Zone
cs5=untrust deviceInboundInterface=vlan.1001 deviceOutboundInterface=vlan.1
cs6Label=LogProfile cs6=Syslog cn1Label=SessionID cn1=53493 cnt=1 spt=59950
dpt=443 sourceTranslatedPort=30630 destinationTranslatedPort=443
flexString1Label=Flags flexString1=0x40b000 proto=tcp act=alert
request=\"www.example.com/\" cs2Label=URL Category cs2=unknown
flexString2Label=Direction flexString2=client-to-server externalId=9868673
requestContext= cat=() filePath= fileId=0 fileHash=
requestClientApplication= fileType= panosxforwarderfor= panosreferer= suid=
msg= duid= oldFileId=0 PanOSDGl1=16 PanOSDGl2=11 PanOSDGl3=0 PanOSDGl4=0
PanOSVsysName= dvchost=firewall
{noformat}
{noformat}
<14>1 2017-05-08T23:22:00+00:00 10.12.1.1 - - - - CEF:0|Palo Alto
Networks|PAN-OS|7.0.0|drop|TRAFFIC|1|rt=May 08 2017 23:21:59 GMT
deviceExternalId=000 src=100.1.2.3 dst=120.1.2.3
sourceTranslatedAddress=0.0.0.0 destinationTranslatedAddress=0.0.0.0
cs1Label=Rule cs1=DropLog suser= duser= app=not-applicable cs3Label=Virtual
System cs3=vsys1 cs4Label=Source Zone cs4=untrust cs5Label=Destination Zone
cs5=untrust deviceInboundInterface=vlan.1 deviceOutboundInterface=
cs6Label=LogProfile cs6=Syslog cn1Label=SessionID cn1=0 cnt=1 spt=7297 dpt=123
sourceTranslatedPort=0 destinationTranslatedPort=0 flexString1Label=Flags
flexString1=0x0 proto=udp act=deny flexNumber1Label=Total bytes flexNumber1=67
in=67 out=0 cn2Label=Packets cn2=1 PanOSPacketsReceived=0 PanOSPacketsSent=1
start=May 08 2017 23:21:59 GMT cn3Label=Elapsed time in seconds cn3=0
cs2Label=URL Category cs2=any externalId=3342330262 reason=policy-deny
PanOSDGl1=16 PanOSDGl2=11 PanOSDGl3=0 PanOSDGl4=0 PanOSVsysName=
dvchost=firewall cat=from-policy
{noformat}
Using the following definitions:
{noformat}
Traffic:
CEF:0|Palo Alto
Networks|PAN-OS|7.0.0|$subtype|$type|1|rt=$cefformatted-receive_time
deviceExternalId=$serial src=$src dst=$dst sourceTranslatedAddress=$natsrc
destinationTranslatedAddress=$natdst cs1Label=Rule cs1=$rule suser=$srcuser
duser=$dstuser app=$app cs3Label=Virtual System cs3=$vsys cs4Label=Source Zone
cs4=$from cs5Label=Destination Zone cs5=$to deviceInboundInterface=$inbound_if
deviceOutboundInterface=$outbound_if cs6Label=LogProfile cs6=$logset
cn1Label=SessionID cn1=$sessionid cnt=$repeatcnt spt=$sport dpt=$dport
sourceTranslatedPort=$natsport destinationTranslatedPort=$natdport
flexString1Label=Flags flexString1=$flags proto=$proto act=$action
flexNumber1Label=Total bytes flexNumber1=$bytes in=$bytes_sent
out=$bytes_received cn2Label=Packets cn2=$packets
PanOSPacketsReceived=$pkts_received PanOSPacketsSent=$pkts_sent
start=$cef-formatted-time_generated cn3Label=Elapsed time in seconds
cn3=$elapsed cs2Label=URL Category cs2=$category externalId=$seqno
reason=$session_end_reason PanOSDGl1=$dg_hier_level_1
PanOSDGl2=$dg_hier_level_2 PanOSDGl3=$dg_hier_level_3
PanOSDGl4=$dg_hier_level_4 PanOSVsysName=$vsys_name dvchost=$device_name
cat=$action_source
Threat:
CEF:0|Palo Alto
Networks|PAN-OS|7.0.0|$subtype|$type|$number-ofseverity|rt=$cef-formatted-receive_time
deviceExternalId=$serial src=$src dst=$dst sourceTranslatedAddress=$natsrc
destinationTranslatedAddress=$natdst cs1Label=Rule cs1=$rule suser=$srcuser
duser=$dstuser app=$app cs3Label=Virtual System cs3=$vsys cs4Label=Source Zone
cs4=$from cs5Label=Destination Zone cs5=$to deviceInboundInterface=$inbound_if
deviceOutboundInterface=$outbound_if cs6Label=LogProfile cs6=$logset
cn1Label=SessionID cn1=$sessionid cnt=$repeatcnt spt=$sport dpt=$dport
sourceTranslatedPort=$natsport destinationTranslatedPort=$natdport
flexString1Label=Flags flexString1=$flags proto=$proto act=$action
request=$misc cs2Label=URL Category cs2=$category
[jira] [Updated] (METRON-940) problems with current Palo Alto schema for CEF parser
[
https://issues.apache.org/jira/browse/METRON-940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christian Tramnitz updated METRON-940:
--
Description:
The current Palo Alto parser (schema on top of CEF parser) seems to use a
custom field definition.
As far as I can tell there is no "standard" definition for a CEF message in
PaloAlto as the scheme can be freely defined. However, there is a documented
example and I would suggest to base the Metron upon this documented definition
rather than a custom definition.
Alternatively we could come up with our message format definition for Palo Alto
CEF, but then we need to document what needs to be done on the Firewall to get
these.
This is a sanitized sample message for threat and traffic:
{noformat}
<14>1 2017-05-08T23:22:00+00:00 10.1.1.1 - - - - CEF:0|Palo Alto
Networks|PAN-OS|7.0.0|url|THREAT|1|rt=May 08 2017 23:22:00 GMT
deviceExternalId=000 src=192.168.1.2 dst=10.28.1.1
sourceTranslatedAddress=10.200.0.1 destinationTranslatedAddress=10.28.1.1
cs1Label=Rule cs1=rulename suser= duser= app=ssl cs3Label=Virtual System
cs3=vsys1 cs4Label=Source Zone cs4=private cs5Label=Destination Zone
cs5=untrust deviceInboundInterface=vlan.1001 deviceOutboundInterface=vlan.1
cs6Label=LogProfile cs6=Syslog cn1Label=SessionID cn1=53493 cnt=1 spt=59950
dpt=443 sourceTranslatedPort=30630 destinationTranslatedPort=443
flexString1Label=Flags flexString1=0x40b000 proto=tcp act=alert
request=\"www.example.com/\" cs2Label=URL Category cs2=unknown
flexString2Label=Direction flexString2=client-to-server externalId=9868673
requestContext= cat=() filePath= fileId=0 fileHash=
requestClientApplication= fileType= panosxforwarderfor= panosreferer= suid=
msg= duid= oldFileId=0 PanOSDGl1=16 PanOSDGl2=11 PanOSDGl3=0 PanOSDGl4=0
PanOSVsysName= dvchost=firewall
{noformat}
{noformat}
<14>1 2017-05-08T23:22:00+00:00 10.12.1.1 - - - - CEF:0|Palo Alto
Networks|PAN-OS|7.0.0|drop|TRAFFIC|1|rt=May 08 2017 23:21:59 GMT
deviceExternalId=000 src=100.1.2.3 dst=120.1.2.3
sourceTranslatedAddress=0.0.0.0 destinationTranslatedAddress=0.0.0.0
cs1Label=Rule cs1=DropLog suser= duser= app=not-applicable cs3Label=Virtual
System cs3=vsys1 cs4Label=Source Zone cs4=untrust cs5Label=Destination Zone
cs5=untrust deviceInboundInterface=vlan.1 deviceOutboundInterface=
cs6Label=LogProfile cs6=Syslog cn1Label=SessionID cn1=0 cnt=1 spt=7297 dpt=123
sourceTranslatedPort=0 destinationTranslatedPort=0 flexString1Label=Flags
flexString1=0x0 proto=udp act=deny flexNumber1Label=Total bytes flexNumber1=67
in=67 out=0 cn2Label=Packets cn2=1 PanOSPacketsReceived=0 PanOSPacketsSent=1
start=May 08 2017 23:21:59 GMT cn3Label=Elapsed time in seconds cn3=0
cs2Label=URL Category cs2=any externalId=3342330262 reason=policy-deny
PanOSDGl1=16 PanOSDGl2=11 PanOSDGl3=0 PanOSDGl4=0 PanOSVsysName=
dvchost=firewall cat=from-policy
{noformat}
Using the following definitions:
{noformat}
Traffic:
CEF:0|Palo Alto
Networks|PAN-OS|7.0.0|$subtype|$type|1|rt=$cefformatted-receive_time
deviceExternalId=$serial src=$src dst=$dst sourceTranslatedAddress=$natsrc
destinationTranslatedAddress=$natdst cs1Label=Rule cs1=$rule suser=$srcuser
duser=$dstuser app=$app cs3Label=Virtual System cs3=$vsys cs4Label=Source Zone
cs4=$from cs5Label=Destination Zone cs5=$to deviceInboundInterface=$inbound_if
deviceOutboundInterface=$outbound_if cs6Label=LogProfile cs6=$logset
cn1Label=SessionID cn1=$sessionid cnt=$repeatcnt spt=$sport dpt=$dport
sourceTranslatedPort=$natsport destinationTranslatedPort=$natdport
flexString1Label=Flags flexString1=$flags proto=$proto act=$action
flexNumber1Label=Total bytes flexNumber1=$bytes in=$bytes_sent
out=$bytes_received cn2Label=Packets cn2=$packets
PanOSPacketsReceived=$pkts_received PanOSPacketsSent=$pkts_sent
start=$cef-formatted-time_generated cn3Label=Elapsed time in seconds
cn3=$elapsed cs2Label=URL Category cs2=$category externalId=$seqno
reason=$session_end_reason PanOSDGl1=$dg_hier_level_1
PanOSDGl2=$dg_hier_level_2 PanOSDGl3=$dg_hier_level_3
PanOSDGl4=$dg_hier_level_4 PanOSVsysName=$vsys_name dvchost=$device_name
cat=$action_source
Threat:
CEF:0|Palo Alto
Networks|PAN-OS|7.0.0|$subtype|$type|$number-ofseverity|rt=$cef-formatted-receive_time
deviceExternalId=$serial src=$src dst=$dst sourceTranslatedAddress=$natsrc
destinationTranslatedAddress=$natdst cs1Label=Rule cs1=$rule suser=$srcuser
duser=$dstuser app=$app cs3Label=Virtual System cs3=$vsys cs4Label=Source Zone
cs4=$from cs5Label=Destination Zone cs5=$to deviceInboundInterface=$inbound_if
deviceOutboundInterface=$outbound_if cs6Label=LogProfile cs6=$logset
cn1Label=SessionID cn1=$sessionid cnt=$repeatcnt spt=$sport dpt=$dport
sourceTranslatedPort=$natsport destinationTranslatedPort=$natdport
flexString1Label=Flags flexString1=$flags proto=$proto act=$action
request=$misc cs2Label=URL Category cs2=$category flexStri
[jira] [Updated] (METRON-940) problems with current Palo Alto schema for CEF parser
[
https://issues.apache.org/jira/browse/METRON-940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christian Tramnitz updated METRON-940:
--
Attachment: pan-os-70-CEF-guide.pdf
pan-os-61-CEF-guide.pdf
> problems with current Palo Alto schema for CEF parser
> -
>
> Key: METRON-940
> URL: https://issues.apache.org/jira/browse/METRON-940
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4
> Environment: full-dev 0.4.0 master
>Reporter: Christian Tramnitz
> Attachments: pan-os-61-CEF-guide.pdf, pan-os-70-CEF-guide.pdf
>
>
> The current Palo Alto parser (schema on top of CEF parser) seems to use a
> custom field definition.
> As far as I can tell there is no "standard" definition for a CEF message in
> PaloAlto as the scheme can be freely defined. However, there is a documented
> example and I would suggest to base the Metron upon this documented
> definition rather than a custom definition.
> Alternatively we could come up with our message format definition for Palo
> Alto CEF, but then we need to document what needs to be done on the Firewall
> to get these.
> This is a sanitized sample message for threat and traffic:
> {noformat}
> <14>1 2017-05-08T23:22:00+00:00 10.1.1.1 - - - - CEF:0|Palo Alto
> Networks|PAN-OS|7.0.0|url|THREAT|1|rt=May 08 2017 23:22:00 GMT
> deviceExternalId=000 src=192.168.1.2 dst=10.28.1.1
> sourceTranslatedAddress=10.200.0.1 destinationTranslatedAddress=10.28.1.1
> cs1Label=Rule cs1=rulename suser= duser= app=ssl cs3Label=Virtual System
> cs3=vsys1 cs4Label=Source Zone cs4=private cs5Label=Destination Zone
> cs5=untrust deviceInboundInterface=vlan.1001 deviceOutboundInterface=vlan.1
> cs6Label=LogProfile cs6=Syslog cn1Label=SessionID cn1=53493 cnt=1 spt=59950
> dpt=443 sourceTranslatedPort=30630 destinationTranslatedPort=443
> flexString1Label=Flags flexString1=0x40b000 proto=tcp act=alert
> request=\"www.example.com/\" cs2Label=URL Category cs2=unknown
> flexString2Label=Direction flexString2=client-to-server externalId=9868673
> requestContext= cat=() filePath= fileId=0 fileHash=
> requestClientApplication= fileType= panosxforwarderfor= panosreferer= suid=
> msg= duid= oldFileId=0 PanOSDGl1=16 PanOSDGl2=11 PanOSDGl3=0 PanOSDGl4=0
> PanOSVsysName= dvchost=firewall
> {noformat}
> {noformat}
> <14>1 2017-05-08T23:22:00+00:00 10.12.1.1 - - - - CEF:0|Palo Alto
> Networks|PAN-OS|7.0.0|drop|TRAFFIC|1|rt=May 08 2017 23:21:59 GMT
> deviceExternalId=000 src=100.1.2.3 dst=120.1.2.3
> sourceTranslatedAddress=0.0.0.0 destinationTranslatedAddress=0.0.0.0
> cs1Label=Rule cs1=DropLog suser= duser= app=not-applicable cs3Label=Virtual
> System cs3=vsys1 cs4Label=Source Zone cs4=untrust cs5Label=Destination Zone
> cs5=untrust deviceInboundInterface=vlan.1 deviceOutboundInterface=
> cs6Label=LogProfile cs6=Syslog cn1Label=SessionID cn1=0 cnt=1 spt=7297
> dpt=123 sourceTranslatedPort=0 destinationTranslatedPort=0
> flexString1Label=Flags flexString1=0x0 proto=udp act=deny
> flexNumber1Label=Total bytes flexNumber1=67 in=67 out=0 cn2Label=Packets
> cn2=1 PanOSPacketsReceived=0 PanOSPacketsSent=1 start=May 08 2017 23:21:59
> GMT cn3Label=Elapsed time in seconds cn3=0 cs2Label=URL Category cs2=any
> externalId=3342330262 reason=policy-deny PanOSDGl1=16 PanOSDGl2=11
> PanOSDGl3=0 PanOSDGl4=0 PanOSVsysName= dvchost=firewall cat=from-policy
> {noformat}
> Using the following definitions:
> {noformat}
> Traffic:
> CEF:0|Palo Alto
> Networks|PAN-OS|7.0.0|$subtype|$type|1|rt=$cefformatted-receive_time
> deviceExternalId=$serial src=$src dst=$dst sourceTranslatedAddress=$natsrc
> destinationTranslatedAddress=$natdst cs1Label=Rule cs1=$rule suser=$srcuser
> duser=$dstuser app=$app cs3Label=Virtual System cs3=$vsys cs4Label=Source
> Zone cs4=$from cs5Label=Destination Zone cs5=$to
> deviceInboundInterface=$inbound_if deviceOutboundInterface=$outbound_if
> cs6Label=LogProfile cs6=$logset cn1Label=SessionID cn1=$sessionid
> cnt=$repeatcnt spt=$sport dpt=$dport sourceTranslatedPort=$natsport
> destinationTranslatedPort=$natdport flexString1Label=Flags flexString1=$flags
> proto=$proto act=$action flexNumber1Label=Total bytes flexNumber1=$bytes
> in=$bytes_sent out=$bytes_received cn2Label=Packets cn2=$packets
> PanOSPacketsReceived=$pkts_received PanOSPacketsSent=$pkts_sent
> start=$cef-formatted-time_generated cn3Label=Elapsed time in seconds
> cn3=$elapsed cs2Label=URL Category cs2=$category externalId=$seqno
> reason=$session_end_reason PanOSDGl1=$dg_hier_level_1
> PanOSDGl2=$dg_hier_level_2 PanOSDGl3=$dg_hier_level_3
> PanOSDGl4=$dg_hier_level_4 PanOSVsysName=$vsys_name dvchost=$device_name
> cat=$action_source
> Threat:
> CEF:0|Palo Alto
> Networks|PAN-OS|7.0.0|$subtype|$type|$number-ofseverity|rt=$cef-formatted
[jira] [Created] (METRON-940) problems with current Palo Alto schema for CEF parser
Christian Tramnitz created METRON-940:
-
Summary: problems with current Palo Alto schema for CEF parser
Key: METRON-940
URL: https://issues.apache.org/jira/browse/METRON-940
Project: Metron
Issue Type: Bug
Affects Versions: 0.4
Environment: full-dev 0.4.0 master
Reporter: Christian Tramnitz
The current Palo Alto parser (schema on top of CEF parser) seems to use a
custom field definition.
As far as I can tell there is no "standard" definition for a CEF message in
PaloAlto as the scheme can be freely defined. However, there is a documented
example and I would suggest to base the Metron upon this documented definition
rather than a custom definition.
Alternatively we could come up with our message format definition for Palo Alto
CEF, but then we need to document what needs to be done on the Firewall to get
these.
This is a sanitized sample message for threat and traffic:
{noformat}
<14>1 2017-05-08T23:22:00+00:00 10.1.1.1 - - - - CEF:0|Palo Alto
Networks|PAN-OS|7.0.0|url|THREAT|1|rt=May 08 2017 23:22:00 GMT
deviceExternalId=000 src=192.168.1.2 dst=10.28.1.1
sourceTranslatedAddress=10.200.0.1 destinationTranslatedAddress=10.28.1.1
cs1Label=Rule cs1=rulename suser= duser= app=ssl cs3Label=Virtual System
cs3=vsys1 cs4Label=Source Zone cs4=private cs5Label=Destination Zone
cs5=untrust deviceInboundInterface=vlan.1001 deviceOutboundInterface=vlan.1
cs6Label=LogProfile cs6=Syslog cn1Label=SessionID cn1=53493 cnt=1 spt=59950
dpt=443 sourceTranslatedPort=30630 destinationTranslatedPort=443
flexString1Label=Flags flexString1=0x40b000 proto=tcp act=alert
request=\"www.example.com/\" cs2Label=URL Category cs2=unknown
flexString2Label=Direction flexString2=client-to-server externalId=9868673
requestContext= cat=() filePath= fileId=0 fileHash=
requestClientApplication= fileType= panosxforwarderfor= panosreferer= suid=
msg= duid= oldFileId=0 PanOSDGl1=16 PanOSDGl2=11 PanOSDGl3=0 PanOSDGl4=0
PanOSVsysName= dvchost=firewall
{noformat}
{noformat}
<14>1 2017-05-08T23:22:00+00:00 10.12.1.1 - - - - CEF:0|Palo Alto
Networks|PAN-OS|7.0.0|drop|TRAFFIC|1|rt=May 08 2017 23:21:59 GMT
deviceExternalId=000 src=100.1.2.3 dst=120.1.2.3
sourceTranslatedAddress=0.0.0.0 destinationTranslatedAddress=0.0.0.0
cs1Label=Rule cs1=DropLog suser= duser= app=not-applicable cs3Label=Virtual
System cs3=vsys1 cs4Label=Source Zone cs4=untrust cs5Label=Destination Zone
cs5=untrust deviceInboundInterface=vlan.1 deviceOutboundInterface=
cs6Label=LogProfile cs6=Syslog cn1Label=SessionID cn1=0 cnt=1 spt=7297 dpt=123
sourceTranslatedPort=0 destinationTranslatedPort=0 flexString1Label=Flags
flexString1=0x0 proto=udp act=deny flexNumber1Label=Total bytes flexNumber1=67
in=67 out=0 cn2Label=Packets cn2=1 PanOSPacketsReceived=0 PanOSPacketsSent=1
start=May 08 2017 23:21:59 GMT cn3Label=Elapsed time in seconds cn3=0
cs2Label=URL Category cs2=any externalId=3342330262 reason=policy-deny
PanOSDGl1=16 PanOSDGl2=11 PanOSDGl3=0 PanOSDGl4=0 PanOSVsysName=
dvchost=firewall cat=from-policy
{noformat}
Using the following definitions:
{noformat}
Traffic:
CEF:0|Palo Alto
Networks|PAN-OS|7.0.0|$subtype|$type|1|rt=$cefformatted-receive_time
deviceExternalId=$serial src=$src dst=$dst sourceTranslatedAddress=$natsrc
destinationTranslatedAddress=$natdst cs1Label=Rule cs1=$rule suser=$srcuser
duser=$dstuser app=$app cs3Label=Virtual System cs3=$vsys cs4Label=Source Zone
cs4=$from cs5Label=Destination Zone cs5=$to deviceInboundInterface=$inbound_if
deviceOutboundInterface=$outbound_if cs6Label=LogProfile cs6=$logset
cn1Label=SessionID cn1=$sessionid cnt=$repeatcnt spt=$sport dpt=$dport
sourceTranslatedPort=$natsport destinationTranslatedPort=$natdport
flexString1Label=Flags flexString1=$flags proto=$proto act=$action
flexNumber1Label=Total bytes flexNumber1=$bytes in=$bytes_sent
out=$bytes_received cn2Label=Packets cn2=$packets
PanOSPacketsReceived=$pkts_received PanOSPacketsSent=$pkts_sent
start=$cef-formatted-time_generated cn3Label=Elapsed time in seconds
cn3=$elapsed cs2Label=URL Category cs2=$category externalId=$seqno
reason=$session_end_reason PanOSDGl1=$dg_hier_level_1
PanOSDGl2=$dg_hier_level_2 PanOSDGl3=$dg_hier_level_3
PanOSDGl4=$dg_hier_level_4 PanOSVsysName=$vsys_name dvchost=$device_name
cat=$action_source
Threat:
CEF:0|Palo Alto
Networks|PAN-OS|7.0.0|$subtype|$type|$number-ofseverity|rt=$cef-formatted-receive_time
deviceExternalId=$serial src=$src dst=$dst sourceTranslatedAddress=$natsrc
destinationTranslatedAddress=$natdst cs1Label=Rule cs1=$rule suser=$srcuser
duser=$dstuser app=$app cs3Label=Virtual System cs3=$vsys cs4Label=Source Zone
cs4=$from cs5Label=Destination Zone cs5=$to deviceInboundInterface=$inbound_if
deviceOutboundInterface=$outbound_if cs6Label=LogProfile cs6=$logset
cn1Label=SessionID cn1=$sessionid cnt=$repeatcnt spt=$sport
