[jira] [Commented] (METRON-1441) Create complementary Solr schemas for the main sensors
[ https://issues.apache.org/jira/browse/METRON-1441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352970#comment-16352970 ] ASF GitHub Bot commented on METRON-1441: Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/922 I would say collection for each parser and an error collection. It's similar to ES templates where we define fields, types, etc. > Create complementary Solr schemas for the main sensors > -- > > Key: METRON-1441 > URL: https://issues.apache.org/jira/browse/METRON-1441 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Priority: Major > > We have ES templates for bro, snort, yaf, and error, we need corresponding > solr schemas for these collections. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1441) Create complementary Solr schemas for the main sensors
[ https://issues.apache.org/jira/browse/METRON-1441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352968#comment-16352968 ] ASF GitHub Bot commented on METRON-1441: Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/922 So, the idea is that we have a 'core' for each parser? and for errors? > Create complementary Solr schemas for the main sensors > -- > > Key: METRON-1441 > URL: https://issues.apache.org/jira/browse/METRON-1441 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Priority: Major > > We have ES templates for bro, snort, yaf, and error, we need corresponding > solr schemas for these collections. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1446) Fix openjdk issue with Ubuntu
[ https://issues.apache.org/jira/browse/METRON-1446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352926#comment-16352926 ] ASF GitHub Bot commented on METRON-1446: GitHub user mmiklavc opened a pull request: https://github.com/apache/metron/pull/926 METRON-1446: Fix openjdk issue with Ubuntu ## Contributor Comments https://issues.apache.org/jira/browse/METRON-1446 Spin up full dev for Ubuntu. It should run successfully all the way through, but specifically it should make it past "TASK [java_jdk : Install openjdk repository]" **Note**, I had issues with Vagrant 1.8.1 that caused trouble with outdated Hashicorp URLs that will give you a 404. Vagrant 2.0.2 worked fine. ## Pull Request Checklist In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? You can merge this pull request into a Git repository by running: $ git pull https://github.com/mmiklavc/metron ubuntu-jdk-fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/926.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #926 commit 9b2dbb5542442e38e444375394f6b6a728994354 Author: Michael Miklavcic Date: 2018-02-05T20:28:22Z Fix openjdk issue with Ubuntu > Fix openjdk issue with Ubuntu > - > > Key: METRON-1446 > URL: https://issues.apache.org/jira/browse/METRON-1446 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > > There appears to have been an update to how openjdk packages are setup with > Ubuntu. The setup requests user feedback that is hanging the ansible command. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-1446) Fix openjdk issue with Ubuntu
Michael Miklavcic created METRON-1446: - Summary: Fix openjdk issue with Ubuntu Key: METRON-1446 URL: https://issues.apache.org/jira/browse/METRON-1446 Project: Metron Issue Type: Bug Reporter: Michael Miklavcic Assignee: Michael Miklavcic There appears to have been an update to how openjdk packages are setup with Ubuntu. The setup requests user feedback that is hanging the ansible command. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-1445) Update performance tuning guide with more explicit parameter instructions
Michael Miklavcic created METRON-1445: - Summary: Update performance tuning guide with more explicit parameter instructions Key: METRON-1445 URL: https://issues.apache.org/jira/browse/METRON-1445 Project: Metron Issue Type: Improvement Reporter: Michael Miklavcic Assignee: Michael Miklavcic -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1444) Add Ubuntu Repositories for Elasticsearch to the Mpack
[ https://issues.apache.org/jira/browse/METRON-1444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-1444: --- Description: Currently the user must manually do something like the following to manually add the package repos before installing Metron with the MPack. These repositories need to be added as part of the MPack so that the manual step is not required. {{cat >/etc/apt/sources.list.d/elasticsearch.list << EOL}} {{deb [https://packages.elastic.co/curator/5/debian] stable main}} {{deb [https://artifacts.elastic.co/packages/5.x/apt] stable main}} {{EOL}} {{apt-get update}} NOTE: On the first attempt in fixing this defect, repository validation fails in Ambari and packages cannot be installed from these repos. We need to better understand how Ambari is validating the repos. was: Currently the user must manually do something like the following to manually add the package repos before installing Metron with the MPack. These repositories need to be added as part of the MPack so that the manual step is not required. {{cat >/etc/apt/sources.list.d/metron-ubuntu.list << EOL}} {{deb https://s3.amazonaws.com/dev.hortonworks.com/HCP/ubuntu12/1.x/BUILDS/1.4.0.0-$BUILD HCP main}} {{deb https://packages.elastic.co/curator/5/debian stable main}} {{deb https://artifacts.elastic.co/packages/5.x/apt stable main}} {{deb https://deb.nodesource.com/node_6.x trusty main}} {{EOL}} {{apt-get update}} NOTE: On the first attempt in fixing this defect, repository validation fails in Ambari and packages cannot be installed from these repos. We need to better understand how Ambari is validating the repos. > Add Ubuntu Repositories for Elasticsearch to the Mpack > -- > > Key: METRON-1444 > URL: https://issues.apache.org/jira/browse/METRON-1444 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Priority: Major > > Currently the user must manually do something like the following to manually > add the package repos before installing Metron with the MPack. These > repositories need to be added as part of the MPack so that the manual step is > not required. > {{cat >/etc/apt/sources.list.d/elasticsearch.list << EOL}} > {{deb [https://packages.elastic.co/curator/5/debian] stable main}} > {{deb [https://artifacts.elastic.co/packages/5.x/apt] stable main}} > {{EOL}} > {{apt-get update}} > NOTE: On the first attempt in fixing this defect, repository validation fails > in Ambari and packages cannot be installed from these repos. We need to > better understand how Ambari is validating the repos. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1442) Rest endpoints for operations on Indexing topology is broken after splitting the indexing topology
[ https://issues.apache.org/jira/browse/METRON-1442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352488#comment-16352488 ] ASF GitHub Bot commented on METRON-1442: Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/923 > Rest endpoints for operations on Indexing topology is broken after splitting > the indexing topology > -- > > Key: METRON-1442 > URL: https://issues.apache.org/jira/browse/METRON-1442 > Project: Metron > Issue Type: Bug >Reporter: Mohan >Assignee: Mohan >Priority: Major > > Rest endpoints for operations on Indexing topology is broken after splitting > the indexing topology into random_access_indexing and batch_indexing -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1441) Create complementary Solr schemas for the main sensors
[ https://issues.apache.org/jira/browse/METRON-1441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352467#comment-16352467 ] ASF GitHub Bot commented on METRON-1441: Github user cestella commented on the issue: https://github.com/apache/metron/pull/922 @merrimanr Things should be fixed at this point. I'm going to spin it up and validate things this morning. > Create complementary Solr schemas for the main sensors > -- > > Key: METRON-1441 > URL: https://issues.apache.org/jira/browse/METRON-1441 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Priority: Major > > We have ES templates for bro, snort, yaf, and error, we need corresponding > solr schemas for these collections. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1441) Create complementary Solr schemas for the main sensors
[ https://issues.apache.org/jira/browse/METRON-1441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352465#comment-16352465 ] ASF GitHub Bot commented on METRON-1441: Github user cestella commented on the issue: https://github.com/apache/metron/pull/922 @merrimanr Ah! Yes, we really should set a `guid` field for errors. > Create complementary Solr schemas for the main sensors > -- > > Key: METRON-1441 > URL: https://issues.apache.org/jira/browse/METRON-1441 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Priority: Major > > We have ES templates for bro, snort, yaf, and error, we need corresponding > solr schemas for these collections. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1441) Create complementary Solr schemas for the main sensors
[ https://issues.apache.org/jira/browse/METRON-1441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352462#comment-16352462 ] ASF GitHub Bot commented on METRON-1441: Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/922 The errors I found happened when I tried to create collections with each schema. The error related to the `guid` field happens because it is defined as the unique key but not included in the list of fields (for error schema). > Create complementary Solr schemas for the main sensors > -- > > Key: METRON-1441 > URL: https://issues.apache.org/jira/browse/METRON-1441 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Priority: Major > > We have ES templates for bro, snort, yaf, and error, we need corresponding > solr schemas for these collections. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1443) Missing Critical MPack Install Instruction for Ubuntu
[ https://issues.apache.org/jira/browse/METRON-1443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-1443: --- Fix Version/s: Next + 1 > Missing Critical MPack Install Instruction for Ubuntu > - > > Key: METRON-1443 > URL: https://issues.apache.org/jira/browse/METRON-1443 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: Next + 1 > > > When installing Elasticsearch with the MPack on Ubuntu, you must manually > install the Elasticsearch repositories. The Mpack itself does not do this, > like it does on CentOS. > When the development environment on Ubuntu is spun-up this step is performed > within Ansible as a prerequisite to the Mpack install. Until this can be > fixed so that it matches what happens in CentOS this needs to be at least > documented. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1433) Only emit debugging timing fields in enrichment when debugging is turned on
[ https://issues.apache.org/jira/browse/METRON-1433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352453#comment-16352453 ] ASF GitHub Bot commented on METRON-1433: Github user cestella commented on the issue: https://github.com/apache/metron/pull/915 @mraliagha Nah, they're not a troublemaker. I'm removing the PR since there's interest in them staying turned on by default. > Only emit debugging timing fields in enrichment when debugging is turned on > --- > > Key: METRON-1433 > URL: https://issues.apache.org/jira/browse/METRON-1433 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Priority: Major > > Right now we always emit performance debugging fields in the split/join > bolts. We should only do that when debug logging is turned on for > org.apache.metron.enrichment.bolt -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1433) Only emit debugging timing fields in enrichment when debugging is turned on
[ https://issues.apache.org/jira/browse/METRON-1433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352454#comment-16352454 ] ASF GitHub Bot commented on METRON-1433: Github user cestella closed the pull request at: https://github.com/apache/metron/pull/915 > Only emit debugging timing fields in enrichment when debugging is turned on > --- > > Key: METRON-1433 > URL: https://issues.apache.org/jira/browse/METRON-1433 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Priority: Major > > Right now we always emit performance debugging fields in the split/join > bolts. We should only do that when debug logging is turned on for > org.apache.metron.enrichment.bolt -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1441) Create complementary Solr schemas for the main sensors
[ https://issues.apache.org/jira/browse/METRON-1441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352444#comment-16352444 ] ASF GitHub Bot commented on METRON-1441: Github user cestella commented on the issue: https://github.com/apache/metron/pull/922 @merrimanr I'm confused about the guid field. I had to take that out given the sample data that I generated from full-dev (they didn't have a `guid` field). How did you create the errors? > Create complementary Solr schemas for the main sensors > -- > > Key: METRON-1441 > URL: https://issues.apache.org/jira/browse/METRON-1441 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Priority: Major > > We have ES templates for bro, snort, yaf, and error, we need corresponding > solr schemas for these collections. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1442) Rest endpoints for operations on Indexing topology is broken after splitting the indexing topology
[ https://issues.apache.org/jira/browse/METRON-1442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352440#comment-16352440 ] ASF GitHub Bot commented on METRON-1442: Github user cestella commented on the issue: https://github.com/apache/metron/pull/923 +1 by inspection; great job Mohan. > Rest endpoints for operations on Indexing topology is broken after splitting > the indexing topology > -- > > Key: METRON-1442 > URL: https://issues.apache.org/jira/browse/METRON-1442 > Project: Metron > Issue Type: Bug >Reporter: Mohan >Assignee: Mohan >Priority: Major > > Rest endpoints for operations on Indexing topology is broken after splitting > the indexing topology into random_access_indexing and batch_indexing -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1443) Missing Critical MPack Install Instruction for Ubuntu
[ https://issues.apache.org/jira/browse/METRON-1443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352437#comment-16352437 ] ASF GitHub Bot commented on METRON-1443: Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/925 > Missing Critical MPack Install Instruction for Ubuntu > - > > Key: METRON-1443 > URL: https://issues.apache.org/jira/browse/METRON-1443 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > > When installing Elasticsearch with the MPack on Ubuntu, you must manually > install the Elasticsearch repositories. The Mpack itself does not do this, > like it does on CentOS. > When the development environment on Ubuntu is spun-up this step is performed > within Ansible as a prerequisite to the Mpack install. Until this can be > fixed so that it matches what happens in CentOS this needs to be at least > documented. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1299) MetronError tests fail if hostname isn't set
[ https://issues.apache.org/jira/browse/METRON-1299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352433#comment-16352433 ] ASF GitHub Bot commented on METRON-1299: Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/924#discussion_r165982935 --- Diff: metron-platform/metron-common/src/test/java/org/apache/metron/common/error/MetronErrorTest.java --- @@ -53,7 +52,14 @@ public void getJSONObjectShouldReturnBasicInformation() { assertEquals(Constants.ErrorType.PARSER_ERROR.getType(), errorJSON.get(Constants.ErrorFields.ERROR_TYPE.getName())); assertEquals("error", errorJSON.get(Constants.SENSOR_TYPE)); assertEquals("sensorType", errorJSON.get(Constants.ErrorFields.FAILED_SENSOR_TYPE.getName())); -assertTrue(((String) errorJSON.get(Constants.ErrorFields.HOSTNAME.getName())).length() > 0); + +try { + String hostName = InetAddress.getLocalHost().getHostName(); --- End diff -- So, in the instance where this happens, is hostName is null or is it that there's an Exception thrown? If it's null, can't we do a: ``` if(hostName == null) { LOG.warn("Unable to resolve local hostname, skipping validation." } else { assertTrue(((String) errorJSON.get(Constants.ErrorFields.HOSTNAME.getName())).length() > 0); assertEquals(hostName, (String) errorJSON.get(Constants.ErrorFields.HOSTNAME.getName())); } ``` I ask because I worry about extraneous *other* exceptions cropping up and us ignoring them. It's a very small nit, let me know what you think. :) > MetronError tests fail if hostname isn't set > > > Key: METRON-1299 > URL: https://issues.apache.org/jira/browse/METRON-1299 > Project: Metron > Issue Type: Bug >Affects Versions: 0.4.1 > Environment: openSUSE Tumbleweed 20171102, OpenJDK 1.8.0_144 >Reporter: Stuart Bertram >Assignee: Otto Fowler >Priority: Major > > If I run "mvn package" in the root Metron directory then compilation fails > because of a null reference in > {{./metron-platform/metron-common/src/test/java/org/apache/metron/common/error/MetronErrorTest.java}}. > This happens in {{getJSONObjectShouldReturnBasicInformation}} on line 56 > because {{errorJSON.get(Constants.ErrorFields.HOSTNAME.getName())}} is > assumed to return a string and the {{length()}} method is called on it. > Because the assert doesn't use a message then no obvious reason why it fails > is logged. > The underlying problem is that > {{metron-platform/metron-common/src/main/java/org/apache/metron/common/error/MetronError.java}} > falls through to a {{catch}} block in {{addHostname()}} when > {{netAddress.getLocalHost().getHostName()}} exceptions with {{Name or service > not known}} for the host name. > Setting a hostname that resolves is a build requirement that is out of > Metron's control, but if the code specifically handles the fact that it won't > always be retrieved then it seems problematic to have tests that assume it > works _and_ not make it clear what the failure is when it occurs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1443) Missing Critical MPack Install Instruction for Ubuntu
[ https://issues.apache.org/jira/browse/METRON-1443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352429#comment-16352429 ] ASF GitHub Bot commented on METRON-1443: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/925 https://issues.apache.org/jira/browse/METRON-1444 > Missing Critical MPack Install Instruction for Ubuntu > - > > Key: METRON-1443 > URL: https://issues.apache.org/jira/browse/METRON-1443 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > > When installing Elasticsearch with the MPack on Ubuntu, you must manually > install the Elasticsearch repositories. The Mpack itself does not do this, > like it does on CentOS. > When the development environment on Ubuntu is spun-up this step is performed > within Ansible as a prerequisite to the Mpack install. Until this can be > fixed so that it matches what happens in CentOS this needs to be at least > documented. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-1444) Add Ubuntu Repositories for Elasticsearch to the Mpack
Nick Allen created METRON-1444: -- Summary: Add Ubuntu Repositories for Elasticsearch to the Mpack Key: METRON-1444 URL: https://issues.apache.org/jira/browse/METRON-1444 Project: Metron Issue Type: Bug Reporter: Nick Allen Currently the user must manually do something like the following to manually add the package repos before installing Metron with the MPack. These repositories need to be added as part of the MPack so that the manual step is not required. {{cat >/etc/apt/sources.list.d/metron-ubuntu.list << EOL}} {{deb https://s3.amazonaws.com/dev.hortonworks.com/HCP/ubuntu12/1.x/BUILDS/1.4.0.0-$BUILD HCP main}} {{deb https://packages.elastic.co/curator/5/debian stable main}} {{deb https://artifacts.elastic.co/packages/5.x/apt stable main}} {{deb https://deb.nodesource.com/node_6.x trusty main}} {{EOL}} {{apt-get update}} NOTE: On the first attempt in fixing this defect, repository validation fails in Ambari and packages cannot be installed from these repos. We need to better understand how Ambari is validating the repos. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1443) Missing Critical MPack Install Instruction for Ubuntu
[ https://issues.apache.org/jira/browse/METRON-1443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352424#comment-16352424 ] ASF GitHub Bot commented on METRON-1443: Github user cestella commented on the issue: https://github.com/apache/metron/pull/925 +1 by inspection > Missing Critical MPack Install Instruction for Ubuntu > - > > Key: METRON-1443 > URL: https://issues.apache.org/jira/browse/METRON-1443 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > > When installing Elasticsearch with the MPack on Ubuntu, you must manually > install the Elasticsearch repositories. The Mpack itself does not do this, > like it does on CentOS. > When the development environment on Ubuntu is spun-up this step is performed > within Ansible as a prerequisite to the Mpack install. Until this can be > fixed so that it matches what happens in CentOS this needs to be at least > documented. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)