[jira] [Assigned] (METRON-799) The MPack should function in a kerberized cluster
[ https://issues.apache.org/jira/browse/METRON-799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-799: -- Assignee: Justin Leet > The MPack should function in a kerberized cluster > - > > Key: METRON-799 > URL: https://issues.apache.org/jira/browse/METRON-799 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Justin Leet > Labels: kerberos > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-818) Ambari elasticsearch.properties template is missing topology.worker.childopts
Justin Leet created METRON-818: -- Summary: Ambari elasticsearch.properties template is missing topology.worker.childopts Key: METRON-818 URL: https://issues.apache.org/jira/browse/METRON-818 Project: Metron Issue Type: Bug Reporter: Justin Leet Assignee: Justin Leet Indexing fails on Ambari deploys, because indexing topology can't resolve a property {code} Error: Could not find or load main class ${topology.worker.childopts} {code} "topology.worker.childopts=" needs to be added to metron-env.xml. An empty value is fine (it's only set to nonempty for Kerberos) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-817) Customise output file path patterns for HDFS indexing
Justin Leet created METRON-817: -- Summary: Customise output file path patterns for HDFS indexing Key: METRON-817 URL: https://issues.apache.org/jira/browse/METRON-817 Project: Metron Issue Type: Improvement Reporter: Justin Leet Assignee: Justin Leet We need to be able to customize the filepaths for HDFS indexing, to allow fields to be part of the naming. For example, if I have a 'tenant' field, I should be able to direct it to a path {code} /apps/metron/.../{tenant}/{sensor}/{date}/filename-34324-432434.json {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (METRON-617) Eliminate Javac warnings in the build
[ https://issues.apache.org/jira/browse/METRON-617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-617: -- Assignee: (was: Justin Leet) > Eliminate Javac warnings in the build > - > > Key: METRON-617 > URL: https://issues.apache.org/jira/browse/METRON-617 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.0 >Reporter: Justin Leet >Priority: Minor > > We have a lot of Javac warnings (several hundred) in the build right now. The > primary cause of these is unchecked types (for reasons including > org.simple.json uses raw types under the hood). These should be labelled > with @SuppressWarnings("unchecked") where appropriate. In addition, things > like @SafeVarargs should be added where appropriate, etc. > Given the number of changes in this ticket is significantly larger than even > the Error Prone warnings (and may require tinkering with generics at > appropriate points), I'm going to split this up by module (Analytics and > Platform), and then split platform as necessary (parsers and enrichment in > particular have a lot). Otherwise, anything resulting is likely to be too > large and unmanageable to be properly tested and reviewed. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (METRON-349) Switch ownership of topologies and files to metron user and update perms
[ https://issues.apache.org/jira/browse/METRON-349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-349: --- Description: Metron services are typically run under the storm user (e.g. spinning up topologies). The mpack deploy creates a Metron user and group. This install should be updated to be running and deploying as the metron user. In addition, many of the files are created or owned by users like the storm user (e.g. in HDFS). These files should also be owned by the metron user, and permissions restricted from 775. Notably, METRON-796 resulted from a partial fix to this. This ticket is the more complete solution to the ownership problem (796 is intended only to get things back in working order, and will actually revert ownership from metron:metron to metron:hadoop to allow storm user to write) was:Currently, Metron services are run under the root user- change this to run under a Metron user. Summary: Switch ownership of topologies and files to metron user and update perms (was: Switch Metron User from root to metron) Updated this ticket to be more complete per discussion on: https://github.com/apache/incubator-metron/pull/488 The original intent is still valid, but a bit outdated and incomplete, so this ticket title and description is updated appropriately. > Switch ownership of topologies and files to metron user and update perms > > > Key: METRON-349 > URL: https://issues.apache.org/jira/browse/METRON-349 > Project: Metron > Issue Type: Improvement >Reporter: David M. Lyle > Labels: deployment, platform > > Metron services are typically run under the storm user (e.g. spinning up > topologies). The mpack deploy creates a Metron user and group. This install > should be updated to be running and deploying as the metron user. > In addition, many of the files are created or owned by users like the storm > user (e.g. in HDFS). These files should also be owned by the metron user, > and permissions restricted from 775. > Notably, METRON-796 resulted from a partial fix to this. This ticket is the > more complete solution to the ownership problem (796 is intended only to get > things back in working order, and will actually revert ownership from > metron:metron to metron:hadoop to allow storm user to write) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-796) Mpack uses wrong group for owning HDFS directories
Justin Leet created METRON-796: -- Summary: Mpack uses wrong group for owning HDFS directories Key: METRON-796 URL: https://issues.apache.org/jira/browse/METRON-796 Project: Metron Issue Type: Bug Reporter: Justin Leet Assignee: Justin Leet org.apache.hadoop.security.AccessControlException: Permission denied: user=storm, access=WRITE, inode="/apps/metron/indexing/indexed/snort/enrichment-null-0-0-1490305873514.json":metron:metron:drwxrwx The group got changed a bit ago from cluster_env.user_group (hadoop) to cluster_env.metron_group (metron). However, because everything right now runs as the storm user (which is in the hadoop group), it doesn't have perms to write anymore. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-791) Add links to website and downloads to top level POM
Justin Leet created METRON-791: -- Summary: Add links to website and downloads to top level POM Key: METRON-791 URL: https://issues.apache.org/jira/browse/METRON-791 Project: Metron Issue Type: Improvement Affects Versions: 0.3.1 Reporter: Justin Leet Assignee: Justin Leet Priority: Minor Right now our GitHub landing page doesn't link back to the Apache page. Both the main link, and the link to our releases section should be added. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-773) Intermittent unit test errors in the KafkaControllerIntegrationTest
[ https://issues.apache.org/jira/browse/METRON-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15929984#comment-15929984 ] Justin Leet commented on METRON-773: I've also seen this in the fixes for the site build, intermittently. https://travis-ci.org/justinleet/incubator-metron/builds/209355250 > Intermittent unit test errors in the KafkaControllerIntegrationTest > --- > > Key: METRON-773 > URL: https://issues.apache.org/jira/browse/METRON-773 > Project: Metron > Issue Type: Bug >Reporter: Casey Stella >Assignee: Casey Stella > > Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 8.514 sec <<< > FAILURE! - in org.apache.metron.rest.controller.KafkaControllerIntegrationTest > test(org.apache.metron.rest.controller.KafkaControllerIntegrationTest) Time > elapsed: 8.415 sec <<< FAILURE! > java.lang.AssertionError: Status expected:<200> but was:<404> > at > org.springframework.test.util.AssertionErrors.fail(AssertionErrors.java:54) > at > org.springframework.test.util.AssertionErrors.assertEquals(AssertionErrors.java:81) > at > org.springframework.test.web.servlet.result.StatusResultMatchers$10.match(StatusResultMatchers.java:664) > at > org.springframework.test.web.servlet.MockMvc$1.andExpect(MockMvc.java:171) > at > org.apache.metron.rest.controller.KafkaControllerIntegrationTest.test(KafkaControllerIntegrationTest.java:173) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:483) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at > org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86) > at > org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:252) > at > org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:94) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at > org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61) > at > org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at > org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:191) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128) > at > org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (METRON-746) Build Custom Checkstyle and IDE formatting settings
[ https://issues.apache.org/jira/browse/METRON-746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-746: --- Description: We need a custom checkstyle.xml based off the sun convention checkstyle. Based on a discussion thread, there are a few things that need to be setup * Two space indents * Line Length longer than 80 (pretty popular in the discussion, but not officially part of our code style) * Appropriate warn/error levels set so we don't immediately start failing every build. * Importable IntelliJ code style at minimum, but I'd also like to see Eclipse if possible to avoid forcing a given dev environment. IntelliJ allows for importing a checkstyle.xml to use as the basis. We can export the resulting formatting settings for people. * Ensure Travis actually runs checkstyle during our builds was: We need a custom checkstyle.xml based off the sun convention checkstyle. Based on a discussion thread, there are a few things that need to be setup * Two space indents * Line Length longer than 80 (pretty popular in the discussion, but not officially part of our code style) * Appropriate warn/error levels set so we don't immediately start failing every build. * Importable IntelliJ code style at minimum, but I'd also like to see Eclipse if possible to avoid forcing a given dev environment. IntelliJ allows for importing a checkstyle.xml to use as the basis. We can export the resulting formatting settings for people. > Build Custom Checkstyle and IDE formatting settings > --- > > Key: METRON-746 > URL: https://issues.apache.org/jira/browse/METRON-746 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Priority: Minor > > We need a custom checkstyle.xml based off the sun convention checkstyle. > Based on a discussion thread, there are a few things that need to be setup > * Two space indents > * Line Length longer than 80 (pretty popular in the discussion, but not > officially part of our code style) > * Appropriate warn/error levels set so we don't immediately start failing > every build. > * Importable IntelliJ code style at minimum, but I'd also like to see Eclipse > if possible to avoid forcing a given dev environment. IntelliJ allows for > importing a checkstyle.xml to use as the basis. We can export the resulting > formatting settings for people. > * Ensure Travis actually runs checkstyle during our builds -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-747) Blanket Reformat of code based on checkstyle configs
Justin Leet created METRON-747: -- Summary: Blanket Reformat of code based on checkstyle configs Key: METRON-747 URL: https://issues.apache.org/jira/browse/METRON-747 Project: Metron Issue Type: Improvement Reporter: Justin Leet Priority: Minor Once METRON-746 is done, we should perform a blanket reformat of our code per discussion on the dev list. At this point we should start enforcing our code via checkstyle (so make sure the appropriate code styling is set to error). We might need to be careful about autogenerated code here (otherwise we might start breaking builds on things we don't care about). I'm unsure if you can whitelist specific files or dirs in checkstyle or not. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-746) Build Custom Checkstyle and IDE formatting settings
Justin Leet created METRON-746: -- Summary: Build Custom Checkstyle and IDE formatting settings Key: METRON-746 URL: https://issues.apache.org/jira/browse/METRON-746 Project: Metron Issue Type: Improvement Reporter: Justin Leet Priority: Minor We need a custom checkstyle.xml based off the sun convention checkstyle. Based on a discussion thread, there are a few things that need to be setup * Two space indents * Line Length longer than 80 (pretty popular in the discussion, but not officially part of our code style) * Appropriate warn/error levels set so we don't immediately start failing every build. * Importable IntelliJ code style at minimum, but I'd also like to see Eclipse if possible to avoid forcing a given dev environment. IntelliJ allows for importing a checkstyle.xml to use as the basis. We can export the resulting formatting settings for people. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-745) Create Error Dashboards
Justin Leet created METRON-745: -- Summary: Create Error Dashboards Key: METRON-745 URL: https://issues.apache.org/jira/browse/METRON-745 Project: Metron Issue Type: Improvement Reporter: Justin Leet Assignee: Justin Leet With errors being indexed once METRON-694 and the associated PR are pulled in (https://github.com/apache/incubator-metron/pull/453), we should create Kibana dashboards to be able to get some summary information and view the errors we get. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (METRON-725) Javadoc is broken by the use of apiNote
[ https://issues.apache.org/jira/browse/METRON-725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-725: --- Fix Version/s: 0.3.1 > Javadoc is broken by the use of apiNote > --- > > Key: METRON-725 > URL: https://issues.apache.org/jira/browse/METRON-725 > Project: Metron > Issue Type: Bug >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Minor > Fix For: 0.3.1 > > > Error seen: > {code} > >mvn javadoc:javadoc > ... > [ERROR] > /Users/jleet/Documents/workspace/incubator-metron/metron-platform/metron-common/src/main/java/org/apache/metron/common/utils/file/ReaderSpliterator.java:127: > error: unknown tag: apiNote > ... > {code} > {{@apiNote}} doesn't work by default when generating Javadocs. Apparently, > it's intended to be language level information rather than a widely adopted > tag. > This only shows up in ReaderSpliterator, in docs copied directly from the > language construct. Given that all these methods are {{@Override}}, it seems > reasonable to just drop the docs entirely, given that they inherit anyway. > If desired, we could explicitly inherit the parent docs. > Finally, we could enable the use of {{@apiNote}}, but given the intended > usage and our copied use, I'm inclined to just change our code directly. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (METRON-734) Builds failing because of MaxMind DB transitive dependency
[ https://issues.apache.org/jira/browse/METRON-734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-734: --- Fix Version/s: 0.3.1 > Builds failing because of MaxMind DB transitive dependency > -- > > Key: METRON-734 > URL: https://issues.apache.org/jira/browse/METRON-734 > Project: Metron > Issue Type: Bug >Reporter: Justin Leet >Assignee: Justin Leet > Fix For: 0.3.1 > > > Original Error > {code} > [ERROR] Failed to execute goal on project metron-enrichment: Could not > resolve dependencies for project > org.apache.metron:metron-enrichment:jar:0.3.1: Failed to collect dependencies > at com.maxmind.geoip2:geoip2:jar:2.8.0 -> com.maxmind.db:maxmind-db:jar:1.2.1 > -> com.fasterxml.jackson.core:jackson-databind:jar:2.9.0-SNAPSHOT: Failed to > read artifact descriptor for > com.fasterxml.jackson.core:jackson-databind:jar:2.9.0-SNAPSHOT: Failure to > find com.fasterxml:oss-parent:pom:28 in http://clojars.org/repo was cached in > the local repository, resolution will not be reattempted until the update > interval of clojars.org has elapsed or updates are forced -> [Help 1] > {code} > Discussion on the lib > https://github.com/maxmind/GeoIP2-java/issues/77 > The geoip2 lib from MaxMind has a transitive dependency on open ended version > range of jackson-databind. This causes it to try to use 2.9.0-SNAPSHOT, > rather than a distinct version. > The easy answer is to just exclude jackson from being pulled in by geoip2 > lib, and use our own. The catch is that the version geoip declares in its > POM is higher than the version we use and I don't know if that causes any > problems (I'm guessing not, since it's going from 2.7.x to 2.8.x, but I > haven't tested it). > If there's a way to skip snapshots that might also contribute to solving our > problem, but I still don't like the unrepeatability of the build if they > depend on an open ended range. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (METRON-734) Builds failing because of MaxMind DB transitive dependency
[ https://issues.apache.org/jira/browse/METRON-734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-734: --- Description: Original Error {code} [ERROR] Failed to execute goal on project metron-enrichment: Could not resolve dependencies for project org.apache.metron:metron-enrichment:jar:0.3.1: Failed to collect dependencies at com.maxmind.geoip2:geoip2:jar:2.8.0 -> com.maxmind.db:maxmind-db:jar:1.2.1 -> com.fasterxml.jackson.core:jackson-databind:jar:2.9.0-SNAPSHOT: Failed to read artifact descriptor for com.fasterxml.jackson.core:jackson-databind:jar:2.9.0-SNAPSHOT: Failure to find com.fasterxml:oss-parent:pom:28 in http://clojars.org/repo was cached in the local repository, resolution will not be reattempted until the update interval of clojars.org has elapsed or updates are forced -> [Help 1] {code} Discussion on the lib https://github.com/maxmind/GeoIP2-java/issues/77 The geoip2 lib from MaxMind has a transitive dependency on open ended version range of jackson-databind. This causes it to try to use 2.9.0-SNAPSHOT, rather than a distinct version. The easy answer is to just exclude jackson from being pulled in by geoip2 lib, and use our own. The catch is that the version geoip declares in its POM is higher than the version we use and I don't know if that causes any problems (I'm guessing not, since it's going from 2.7.x to 2.8.x, but I haven't tested it). If there's a way to skip snapshots that might also contribute to solving our problem, but I still don't like the unrepeatability of the build if they depend on an open ended range. was: Original Error {code} [ERROR] Failed to execute goal on project metron-enrichment: Could not resolve dependencies for project org.apache.metron:metron-enrichment:jar:0.3.1: Failed to collect dependencies at com.maxmind.geoip2:geoip2:jar:2.8.0 -> com.maxmind.db:maxmind-db:jar:1.2.1 -> com.fasterxml.jackson.core:jackson-databind:jar:2.9.0-SNAPSHOT: Failed to read artifact descriptor for com.fasterxml.jackson.core:jackson-databind:jar:2.9.0-SNAPSHOT: Failure to find com.fasterxml:oss-parent:pom:28 in http://clojars.org/repo was cached in the local repository, resolution will not be reattempted until the update interval of clojars.org has elapsed or updates are forced -> [Help 1] {code} The geoip2 lib from MaxMind has a transitive dependency on open ended version range of jackson-databind. This causes it to try to use 2.9.0-SNAPSHOT, rather than a distinct version. The easy answer is to just exclude jackson from being pulled in by geoip2 lib, and use our own. The catch is that the version geoip declares in its POM is higher than the version we use and I don't know if that causes any problems (I'm guessing not, since it's going from 2.7.x to 2.8.x, but I haven't tested it). If there's a way to skip snapshots that might also contribute to solving our problem, but I still don't like the unrepeatability of the build if they depend on an open ended range. > Builds failing because of MaxMind DB transitive dependency > -- > > Key: METRON-734 > URL: https://issues.apache.org/jira/browse/METRON-734 > Project: Metron > Issue Type: Bug >Reporter: Justin Leet >Assignee: Justin Leet > > Original Error > {code} > [ERROR] Failed to execute goal on project metron-enrichment: Could not > resolve dependencies for project > org.apache.metron:metron-enrichment:jar:0.3.1: Failed to collect dependencies > at com.maxmind.geoip2:geoip2:jar:2.8.0 -> com.maxmind.db:maxmind-db:jar:1.2.1 > -> com.fasterxml.jackson.core:jackson-databind:jar:2.9.0-SNAPSHOT: Failed to > read artifact descriptor for > com.fasterxml.jackson.core:jackson-databind:jar:2.9.0-SNAPSHOT: Failure to > find com.fasterxml:oss-parent:pom:28 in http://clojars.org/repo was cached in > the local repository, resolution will not be reattempted until the update > interval of clojars.org has elapsed or updates are forced -> [Help 1] > {code} > Discussion on the lib > https://github.com/maxmind/GeoIP2-java/issues/77 > The geoip2 lib from MaxMind has a transitive dependency on open ended version > range of jackson-databind. This causes it to try to use 2.9.0-SNAPSHOT, > rather than a distinct version. > The easy answer is to just exclude jackson from being pulled in by geoip2 > lib, and use our own. The catch is that the version geoip declares in its > POM is higher than the version we use and I don't know if that causes any > problems (I'm guessing not, since it's going from 2.7.x to 2.8.x, but I > haven't tested it). > If there's a way to skip snapshots that might also contribute to solving our > problem, but I still don't like the unrepeatability of the build if they > depend on an open ended range. -- This message was sent by Atlassian JIRA (v6.3.15#
[jira] [Created] (METRON-734) Builds failing because of MaxMind DB transitive dependency
Justin Leet created METRON-734: -- Summary: Builds failing because of MaxMind DB transitive dependency Key: METRON-734 URL: https://issues.apache.org/jira/browse/METRON-734 Project: Metron Issue Type: Bug Reporter: Justin Leet Assignee: Justin Leet Original Error {code} [ERROR] Failed to execute goal on project metron-enrichment: Could not resolve dependencies for project org.apache.metron:metron-enrichment:jar:0.3.1: Failed to collect dependencies at com.maxmind.geoip2:geoip2:jar:2.8.0 -> com.maxmind.db:maxmind-db:jar:1.2.1 -> com.fasterxml.jackson.core:jackson-databind:jar:2.9.0-SNAPSHOT: Failed to read artifact descriptor for com.fasterxml.jackson.core:jackson-databind:jar:2.9.0-SNAPSHOT: Failure to find com.fasterxml:oss-parent:pom:28 in http://clojars.org/repo was cached in the local repository, resolution will not be reattempted until the update interval of clojars.org has elapsed or updates are forced -> [Help 1] {code} The geoip2 lib from MaxMind has a transitive dependency on open ended version range of jackson-databind. This causes it to try to use 2.9.0-SNAPSHOT, rather than a distinct version. The easy answer is to just exclude jackson from being pulled in by geoip2 lib, and use our own. The catch is that the version geoip declares in its POM is higher than the version we use and I don't know if that causes any problems (I'm guessing not, since it's going from 2.7.x to 2.8.x, but I haven't tested it). If there's a way to skip snapshots that might also contribute to solving our problem, but I still don't like the unrepeatability of the build if they depend on an open ended range. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-733) Remove Geo database from ParserBolt
Justin Leet created METRON-733: -- Summary: Remove Geo database from ParserBolt Key: METRON-733 URL: https://issues.apache.org/jira/browse/METRON-733 Project: Metron Issue Type: Bug Reporter: Justin Leet Assignee: Justin Leet The ParserBolt inits the Geo DB in its prepare() method. Parsers, unlike enrichments, do not use geo as a base capability. They should be able to run without the DB file existing in HDFS. This init causes issues if the file is missing, even if GEO_GET is unused in the parser definition. This change should preserve the ability of Parsers to employ Stellar's GEO_GET. The Stellar function already handles that init, so it shouldn't be an issue. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (METRON-728) ReaderSpliteratorTest fails randomly and extremely rarely
[ https://issues.apache.org/jira/browse/METRON-728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-728: -- Assignee: Justin Leet > ReaderSpliteratorTest fails randomly and extremely rarely > - > > Key: METRON-728 > URL: https://issues.apache.org/jira/browse/METRON-728 > Project: Metron > Issue Type: Bug >Affects Versions: 0.3.1 >Reporter: Justin Leet >Assignee: Justin Leet > > See logs at > https://travis-ci.org/justinleet/incubator-metron/builds/203298348 > I was able to reproduce this locally by calling > {{testActuallyParallel_mediumBatch}} in a {{while(true)}} loop. It can also > occur in {{testActuallyParallel_mediumBatchImplicitlyParallel()}}. I also > had to add > {{forkJoinPool.shutdownNow();}} to the end of the test, because otherwise OOM > errors occur. > My current assumption is that there's no guarantee you ever actually end up > running in parallel, so in extremely rare cases you just end up running one > thread. > I've had it vary wildly when I hit it, from within a second or two to running > for over a minute before an assertion failure occurs. > We could just alter the assertion to be > {code}Assert.assertTrue(threads.size() <= (int) Math.ceil(9.0 / 2) && > threads.size() >= 1);{code} > This defeats the purpose of testing the parallelism a bit, but if there's no > guarantee we actually get parallelism there's not a fantastic way to test it. > Given the extreme rarity, we might want to just live with the fact that > occasionally {{threads.size() >= 1}} hits the threads.size() == 1 case on the > two tests. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-728) ReaderSpliteratorTest fails randomly and extremely rarely
Justin Leet created METRON-728: -- Summary: ReaderSpliteratorTest fails randomly and extremely rarely Key: METRON-728 URL: https://issues.apache.org/jira/browse/METRON-728 Project: Metron Issue Type: Bug Affects Versions: 0.3.1 Reporter: Justin Leet See logs at https://travis-ci.org/justinleet/incubator-metron/builds/203298348 I was able to reproduce this locally by calling {{testActuallyParallel_mediumBatch}} in a {{while(true)}} loop. It can also occur in {{testActuallyParallel_mediumBatchImplicitlyParallel()}}. I also had to add {{forkJoinPool.shutdownNow();}} to the end of the test, because otherwise OOM errors occur. My current assumption is that there's no guarantee you ever actually end up running in parallel, so in extremely rare cases you just end up running one thread. I've had it vary wildly when I hit it, from within a second or two to running for over a minute before an assertion failure occurs. We could just alter the assertion to be {code}Assert.assertTrue(threads.size() <= (int) Math.ceil(9.0 / 2) && threads.size() >= 1);{code} This defeats the purpose of testing the parallelism a bit, but if there's no guarantee we actually get parallelism there's not a fantastic way to test it. Given the extreme rarity, we might want to just live with the fact that occasionally {{threads.size() >= 1}} hits the threads.size() == 1 case on the two tests. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-726) Clean up mvn site generation
Justin Leet created METRON-726: -- Summary: Clean up mvn site generation Key: METRON-726 URL: https://issues.apache.org/jira/browse/METRON-726 Project: Metron Issue Type: Bug Reporter: Justin Leet Assignee: Justin Leet Priority: Minor Right now there's a couple issues with running mvn:site. The most obvious is that EMMA appears to not work at all, but in attempting to fix that, several other issues came to light. Error seen: {code} [ERROR] Failed to execute goal org.apache.maven.plugins:maven-site-plugin:3.3:site (default-site) on project metron-maas-common: failed to get report for org.codehaus.mojo:emma-maven-plugin: Failed to execute goal org.codehaus.mojo:emma-maven-plugin:1.0-alpha-3:instrument (report:emma) on project metron-maas-common: Execution report:emma of goal org.codehaus.mojo:emma-maven-plugin:1.0-alpha-3:instrument failed: CONSTANT_info: invalid tag value [18] -> [Help 1] {code} After commenting out everything EMMA, there are still some issues seen: {code} [WARNING] Unable to process class org/apache/metron/test/converters/BinaryConverters.class in JarAnalyzer File /Users/jleet/.m2/repository/org/apache/metron/metron-test-utilities/0.3.1/metron-test-utilities-0.3.1.jar org.apache.bcel.classfile.ClassFormatException: Invalid byte tag in constant pool: 18 {code} This seems to be a Java 8 issue, which means that EMMA likely is impossible to make work. I'm unsure that's the root cause, but given the age of EMMA plus (the outdated version of) BCEL thowing a very similar issue implies that the Java version is related. This also implies that our {{mvn site}} hasn't worked in a long time. Cleaning this up should include at least * Getting code coverage working again * Consolidating reporting in our poms. A lot of it is repeated everywhere we have Java. * Ensure we can actually generate and look through the site. * METRON-725 fixes Javadoc, so reporting will still have issues until that is taken care of. * Apparently checkstyle got dropped at some point. It's easy enough to add in, and can be taken care of here. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-725) Javadoc is broken by the use of apiNote
Justin Leet created METRON-725: -- Summary: Javadoc is broken by the use of apiNote Key: METRON-725 URL: https://issues.apache.org/jira/browse/METRON-725 Project: Metron Issue Type: Bug Reporter: Justin Leet Assignee: Justin Leet Priority: Minor Error seen: {code} >mvn javadoc:javadoc ... [ERROR] /Users/jleet/Documents/workspace/incubator-metron/metron-platform/metron-common/src/main/java/org/apache/metron/common/utils/file/ReaderSpliterator.java:127: error: unknown tag: apiNote ... {code} {{@apiNote}} doesn't work by default when generating Javadocs. Apparently, it's intended to be language level information rather than a widely adopted tag. This only shows up in ReaderSpliterator, in docs copied directly from the language construct. Given that all these methods are {{@Override}}, it seems reasonable to just drop the docs entirely, given that they inherit anyway. If desired, we could explicitly inherit the parent docs. Finally, we could enable the use of {{@apiNote}}, but given the intended usage and our copied use, I'm inclined to just change our code directly. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-711) StellarShell assigns variables even if an exception was thrown in the statement.
Justin Leet created METRON-711: -- Summary: StellarShell assigns variables even if an exception was thrown in the statement. Key: METRON-711 URL: https://issues.apache.org/jira/browse/METRON-711 Project: Metron Issue Type: Bug Reporter: Justin Leet Assignee: Justin Leet Priority: Minor Discovered while reviewing https://github.com/apache/incubator-metron/pull/438. If an exception is thrown during Stellar execution, the exception will be logged, and null is returned. The assignment then goes through as normal, leaving the assigned variable null. Seen using THREAT_TRIAGE_REMOVE with a String arg, instead of a List. Resulted in a null conf variable. {code} [Stellar]>>> conf := THREAT_TRIAGE_ADD(conf, [triage]) [Stellar]>>> conf := THREAT_TRIAGE_REMOVE(conf, 'Abnormal DNS Port') [!] Unable to execute: java.lang.String cannot be cast to java.util.List org.apache.metron.common.dsl.ParseException: Unable to execute: java.lang.String cannot be cast to java.util.List at org.apache.metron.common.stellar.StellarCompiler.getResult(StellarCompiler.java:409) at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:127) at org.apache.metron.common.stellar.shell.StellarExecutor.execute(StellarExecutor.java:275) at org.apache.metron.common.stellar.shell.StellarShell.executeStellar(StellarShell.java:373) at org.apache.metron.common.stellar.shell.StellarShell.handleStellar(StellarShell.java:276) at org.apache.metron.common.stellar.shell.StellarShell.execute(StellarShell.java:412) at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to java.util.List at org.apache.metron.management.ThreatTriageFunctions$RemoveStellarTransformation.apply(ThreatTriageFunctions.java:232) at org.apache.metron.common.stellar.StellarCompiler.exitTransformationFunc(StellarCompiler.java:267) at org.apache.metron.common.stellar.generated.StellarParser$TransformationFuncContext.exitRule(StellarParser.java:1689) at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422) at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632) at org.apache.metron.common.stellar.generated.StellarParser.functions(StellarParser.java:1712) at org.apache.metron.common.stellar.generated.StellarParser.arithmetic_operands(StellarParser.java:1846) at org.apache.metron.common.stellar.generated.StellarParser.arithmetic_expr_mul(StellarParser.java:1609) at org.apache.metron.common.stellar.generated.StellarParser.arithmetic_expr(StellarParser.java:1469) at org.apache.metron.common.stellar.generated.StellarParser.transformation_expr(StellarParser.java:308) at org.apache.metron.common.stellar.generated.StellarParser.transformation(StellarParser.java:149) at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:126) ... 8 more [Stellar]>>> conf [Stellar]>>> conf [Stellar]>>> conf := THREAT_TRIAGE_REMOVE(conf, ['Abnormal DNS Port']) [Stellar]>>> conf { "enrichment" : { "fieldMap" : { }, "fieldToTypeMap" : { }, "config" : { } }, "threatIntel" : { "fieldMap" : { }, "fieldToTypeMap" : { }, "config" : { }, "triageConfig" : { "riskLevelRules" : [ ], "aggregator" : "MAX", "aggregationConfig" : { } } }, "configuration" : { } } {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-710) Rev MPack Version to 0.3.1.0
Justin Leet created METRON-710: -- Summary: Rev MPack Version to 0.3.1.0 Key: METRON-710 URL: https://issues.apache.org/jira/browse/METRON-710 Project: Metron Issue Type: Bug Reporter: Justin Leet Assignee: Justin Leet Fix For: 0.3.1 For the release, need to rev mpack version. Dev list has so far agreed on 0.3.1.0 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (METRON-680) GeoLiteDatabase incorrectly using country geoname_id instead of city
[ https://issues.apache.org/jira/browse/METRON-680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-680: --- Fix Version/s: 0.3.1 > GeoLiteDatabase incorrectly using country geoname_id instead of city > > > Key: METRON-680 > URL: https://issues.apache.org/jira/browse/METRON-680 > Project: Metron > Issue Type: Bug >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Minor > Fix For: 0.3.1 > > > Due to misunderstanding exactly how things tied together with the updated > database, the wrong field is used for the locId. Instead of using the city's > geoname_id, we are using the country's. > This will effect Kibana dashboards and anything that depends on the locId, > because it will be retrieved at the country level instead of the city level. > The change will not break anything (anything not at the city level uses the > country's code, e.g. if the IP is for Japan in general, the city code is > 1861060, not empty or null). This example from the plaintext database can be > seen in the second and third fields at: > bq. 1.112.0.0/15,1861060,1861060,,0,0,,35.6900,139.6900,500 > The offending code is in `GeoLiteDatabase.java` and should be > `geoInfo.put("locID", convertNullToEmptyString(country.getGeoNameId()));` > This should be updated to grab the city's geoname, and tests should be > updated to reflect this (they didn't catch this error because of the > misunderstood data change, not an error in coding). > Ideally, this field is renamed and better documented as part of METRON-679 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-693) Stellar REPL lets reserved tokens be used as unusable variable names
Justin Leet created METRON-693: -- Summary: Stellar REPL lets reserved tokens be used as unusable variable names Key: METRON-693 URL: https://issues.apache.org/jira/browse/METRON-693 Project: Metron Issue Type: Bug Reporter: Justin Leet Assignee: Justin Leet Priority: Minor Reserved tokens like '=' can be assigned with the variable assignment operator. These "variables" can no longer be used, but show up in the %vars command. Reserved tokens should be disallowed. Example {code} [Stellar]>>> test := 1 [Stellar]>>> 2 + 2 := 5 [Stellar]>>> = := 6 [Stellar]>>> %vars 2 + 2 = 5 via 5 test = 1 via 1 = = 6 via 6 {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] (METRON-680) GeoLiteDatabase incorrectly using country geoname_id instead of city
Title: Message Title Justin Leet commented on METRON-680 Re: GeoLiteDatabase incorrectly using country geoname_id instead of city We actually use the field directly for the "Unique-Location(s)" visualization in Kibana. How we choose to populate the field changes the meaning of the count, which is where that question comes from. If we're fine counting only the city's geoname_id for that field, the switch to city is a one line change + test fix. The swapped data is in the same format as the current data, so the change to city is something I could have a PR for very quickly. Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (METRON-680) GeoLiteDatabase incorrectly using country geoname_id instead of city
Title: Message Title Justin Leet commented on METRON-680 Re: GeoLiteDatabase incorrectly using country geoname_id instead of city The part about including the country's code as the city isn't always true. See: 23.129.1.0/24,,6252001,,0,0 It's the U.S. (http://www.geonames.org/6252001/united-states.html), but doesn't attach any city information to it. Do we need to be doing a fallback of "Use city if available, else use country"? Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (METRON-680) GeoLiteDatabase incorrectly using country geoname_id instead of city
Title: Message Title Justin Leet created an issue Metron / METRON-680 GeoLiteDatabase incorrectly using country geoname_id instead of city Issue Type: Bug Assignee: Justin Leet Created: 31/Jan/17 14:12 Priority: Major Reporter: Justin Leet Due to misunderstanding exactly how things tied together with the updated database, the wrong field is used for the locId. Instead of using the city's geoname_id, we are using the country's. This will effect Kibana dashboards and anything that depends on the locId, because it will be retrieved at the country level instead of the city level. The change will not break anything (anything not at the city level uses the country's code, e.g. if the IP is for Japan in general, the city code is 1861060, not empty or null). This example from the plaintext database can be seen in the second and third fields at: 1.112.0.0/15,1861060,1861060,,0,0,,35.6900,139.6900,500 The offending code is in `GeoLiteDatabase.java` and should be `geoInfo.put("locID", convertNullToEmptyString(country.getGeoNameId()));` This should be updated to grab the city's geoname, and tests should be updated to reflect this (they didn't catch this error because of the misunderstood data change, not an error in coding). Ideally, this field is renamed and better documented as part of METRON-679
[jira] (METRON-679) Investigate including additional GeoIP fields
Title: Message Title Justin Leet commented on METRON-679 Re: Investigate including additional GeoIP fields As a note, geoname_id is actually referencing an outside database, e.g. http://www.geonames.org/1861060/japan.html 1861060 is the geoname_id for Japan (and is used as the city id where there is no particular city for the IP). We should add documentation around this, and ideally update that field name and the associated dashboards to make this relationship explicit (rather than masking it by calling it locId) Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (METRON-679) Investigate including additional GeoIP fields
Title: Message Title Justin Leet commented on METRON-679 Re: Investigate including additional GeoIP fields James Sirota You're more familiar with what's potentially useful than I am, any insight here? Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (METRON-679) Investigate including additional GeoIP fields
Title: Message Title Justin Leet created an issue Metron / METRON-679 Investigate including additional GeoIP fields Issue Type: Improvement Assignee: Unassigned Created: 31/Jan/17 13:56 Priority: Minor Reporter: Justin Leet The newer downloadable database we're using provides a wider array of fields than our old solution. We may also want to update some naming (we've been using metro code and calling it dma code, even in the old solution, for example). Data pulled from plaintext download at: http://dev.maxmind.com/geoip/geoip2/geolite2/ Currently used fields: geoname_id (renaming it LocId). city_name country_iso_code metro_code (but renaming it dma code) postal_code latitude longitude network (indirectly, it's the field used for pulling back data for IP and is incl
[jira] [Commented] (METRON-674) Archived Telemetry Filenames in HDFS Contain 'Null'
[ https://issues.apache.org/jira/browse/METRON-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15838562#comment-15838562 ] Justin Leet commented on METRON-674: http://storm.apache.org/releases/1.0.1/storm-hdfs.html Per the docs, the format is {code} {prefix}{componentId}-{taskId}-{rotationNum}-{timestamp}{extension} {code} Given a file like: enrichment-null-0-0-1484752251563.json The null field is componentId. I'd have to dig into why it's actually null though. > Archived Telemetry Filenames in HDFS Contain 'Null' > --- > > Key: METRON-674 > URL: https://issues.apache.org/jira/browse/METRON-674 > Project: Metron > Issue Type: Bug >Affects Versions: 0.3.0 >Reporter: Nick Allen >Priority: Minor > > When running "Quick Dev", I have noticed that all of the archived telemetry > files in HDFS contain null in the name. > {code} > [root@node1 0.3.0]# hdfs dfs -ls /apps/metron/indexing/indexed/* > Found 2 items > -rw-r--r-- 1 storm hadoop 644753 2017-01-19 17:47 > /apps/metron/indexing/indexed/bro/enrichment-null-0-0-1484847868551.json > -rw-r--r-- 1 storm hadoop 14107767 2017-01-19 18:46 > /apps/metron/indexing/indexed/bro/enrichment-null-0-0-1484848728527.json > Found 5 items > -rwxrwxrwx 1 storm hadoop 205699 2017-01-16 21:57 > /apps/metron/indexing/indexed/snort/enrichment-null-0-0-1484603710250.json > -rwxrwxrwx 1 storm hadoop5773871 2017-01-17 14:34 > /apps/metron/indexing/indexed/snort/enrichment-null-0-0-1484603925156.json > -rwxrwxrwx 1 storm hadoop 253870 2017-01-17 13:43 > /apps/metron/indexing/indexed/snort/enrichment-null-0-0-1484660437793.json > -rwxrwxrwx 1 storm hadoop 24023035 2017-01-17 19:45 > /apps/metron/indexing/indexed/snort/enrichment-null-0-0-1484660672723.json > -rwxrwxrwx 1 storm hadoop2063857 2017-01-17 19:02 > /apps/metron/indexing/indexed/snort/enrichment-null-0-0-1484679265343.json > Found 147 items > -rwxrwxrwx 1 storm hadoop 18199681 2017-01-18 16:35 > /apps/metron/indexing/indexed/yaf/enrichment-null-0-0-1484752251563.json > -rwxrwxrwx 1 storm hadoop 216895 2017-01-19 17:47 > /apps/metron/indexing/indexed/yaf/enrichment-null-0-0-1484846918122.json > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (METRON-283) Migrate Geo Enrichment outside of MySQL
[ https://issues.apache.org/jira/browse/METRON-283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-283: -- Assignee: Justin Leet > Migrate Geo Enrichment outside of MySQL > --- > > Key: METRON-283 > URL: https://issues.apache.org/jira/browse/METRON-283 > Project: Metron > Issue Type: Improvement >Reporter: James Sirota >Assignee: Justin Leet >Priority: Minor > > We need to migrate our enrichment SQL store from MySQL to Phoenix or some > other SQL on Hbase library. Or alternatively come up with a way to do this > without using SQL. This way we don't have a dependency on MySQL and there is > one less thing that we need to install on our platform -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-649) Improve handling of dates in parsers when year is missing from data
Justin Leet created METRON-649: -- Summary: Improve handling of dates in parsers when year is missing from data Key: METRON-649 URL: https://issues.apache.org/jira/browse/METRON-649 Project: Metron Issue Type: Bug Reporter: Justin Leet METRON-647 makes a change in the tests to match expected output to the existing parser behavior to alleviate year related issues. The larger issue discovered as we looked into it is that some parsers (at least GrokWebSphereParser) assume everything is in the current year. E.g. if it's Jan 4th, 2017 and we get a message for April 15 the assumption is made that the message is from April 15, 2017. It seems unlikely we want to just toss any data like this as being bad, but we need some handling or rules around what happens for formats like this. A potential solution is to assume that the year should assume a past date if it's from a future date (assume April 15, 2016 instead of April 15, 2017). However, there are potential replay use cases where this might not be sufficient (e.g. we've have data for a few years and it's actually from April 15, 2013). We need to figure out a strategy for how we handle cases like this, in addition to ensuring any existing parsers handle it appropriately. In addition, the solution needs to be appropriately documented. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-647) Parser unit test failures due to assumed year values
[ https://issues.apache.org/jira/browse/METRON-647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-647: --- Fix Version/s: Next + 1 > Parser unit test failures due to assumed year values > > > Key: METRON-647 > URL: https://issues.apache.org/jira/browse/METRON-647 > Project: Metron > Issue Type: Bug >Reporter: Kyle Richardson >Assignee: Justin Leet >Priority: Blocker > Fix For: Next + 1 > > > Unit Tests Failing: > * BasicAsaParserTest.testIp6Addr:151 > * GrokWebSphereParserTest.testParseLoginLine:60 > * GrokWebSphereParserTest.testParseMalformedLoginLine:151 > * GrokWebSphereParserTest.tetsParseLogoutLine:84 > * GrokWebSphereParserTest.tetsParseMalformedLogoutLine:175 > * GrokWebSphereParserTest.tetsParseMalformedOtherLine:220 > * GrokWebSphereParserTest.tetsParseMalformedRBMLine:198 > * GrokWebSphereParserTest.tetsParseOtherLine:129 > * GrokWebSphereParserTest.tetsParseRBMLine:107 > The issue appears to be the same for all. The original syslog message did not > include a year (fairly common); however, we have hard coded an assertion > based on the year the code was written (in our case 2016). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (METRON-647) Parser unit test failures due to assumed year values
[ https://issues.apache.org/jira/browse/METRON-647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-647: -- Assignee: Justin Leet > Parser unit test failures due to assumed year values > > > Key: METRON-647 > URL: https://issues.apache.org/jira/browse/METRON-647 > Project: Metron > Issue Type: Bug >Reporter: Kyle Richardson >Assignee: Justin Leet >Priority: Blocker > > Unit Tests Failing: > * BasicAsaParserTest.testIp6Addr:151 > * GrokWebSphereParserTest.testParseLoginLine:60 > * GrokWebSphereParserTest.testParseMalformedLoginLine:151 > * GrokWebSphereParserTest.tetsParseLogoutLine:84 > * GrokWebSphereParserTest.tetsParseMalformedLogoutLine:175 > * GrokWebSphereParserTest.tetsParseMalformedOtherLine:220 > * GrokWebSphereParserTest.tetsParseMalformedRBMLine:198 > * GrokWebSphereParserTest.tetsParseOtherLine:129 > * GrokWebSphereParserTest.tetsParseRBMLine:107 > The issue appears to be the same for all. The original syslog message did not > include a year (fairly common); however, we have hard coded an assertion > based on the year the code was written (in our case 2016). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-647) Parser unit test failures due to assumed year values
[ https://issues.apache.org/jira/browse/METRON-647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15798409#comment-15798409 ] Justin Leet commented on METRON-647: [~kylerichardson] Were you planning on picking this up? If not, let me know, and I'll grab it and throw up a PR so this gets cleaned up for everyone. > Parser unit test failures due to assumed year values > > > Key: METRON-647 > URL: https://issues.apache.org/jira/browse/METRON-647 > Project: Metron > Issue Type: Bug >Reporter: Kyle Richardson >Priority: Blocker > > Unit Tests Failing: > * BasicAsaParserTest.testIp6Addr:151 > * GrokWebSphereParserTest.testParseLoginLine:60 > * GrokWebSphereParserTest.testParseMalformedLoginLine:151 > * GrokWebSphereParserTest.tetsParseLogoutLine:84 > * GrokWebSphereParserTest.tetsParseMalformedLogoutLine:175 > * GrokWebSphereParserTest.tetsParseMalformedOtherLine:220 > * GrokWebSphereParserTest.tetsParseMalformedRBMLine:198 > * GrokWebSphereParserTest.tetsParseOtherLine:129 > * GrokWebSphereParserTest.tetsParseRBMLine:107 > The issue appears to be the same for all. The original syslog message did not > include a year (fairly common); however, we have hard coded an assertion > based on the year the code was written (in our case 2016). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-618) Eliminate Javac Warnings in metron-analytics
[ https://issues.apache.org/jira/browse/METRON-618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-618: --- Fix Version/s: Next + 1 > Eliminate Javac Warnings in metron-analytics > > > Key: METRON-618 > URL: https://issues.apache.org/jira/browse/METRON-618 > Project: Metron > Issue Type: Sub-task >Affects Versions: 0.3.0 >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Minor > Fix For: Next + 1 > > > Kill off the java compiler warnings in maas project, as much as possible. > Most of these are related to missing @SuppressWarnings("unchecked") on code > that either should have them, or be refactored so it's not necessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-619) Eliminate Javac warnings in metron-parsers
[ https://issues.apache.org/jira/browse/METRON-619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-619: --- Assignee: (was: Justin Leet) > Eliminate Javac warnings in metron-parsers > -- > > Key: METRON-619 > URL: https://issues.apache.org/jira/browse/METRON-619 > Project: Metron > Issue Type: Sub-task >Affects Versions: 0.3.0 >Reporter: Justin Leet >Priority: Minor > > Metron-parsers has a lot of Javac warnings. Clean up the code as necessary > to get rid of them. Probably mostly just missing > @SuppressWarnings("unchecked") because things like our JSON library use raw > types under the hood. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-621) Eliminate Javac warnings in metron-platform outside of parsers and enrichment
[ https://issues.apache.org/jira/browse/METRON-621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-621: --- Assignee: (was: Justin Leet) > Eliminate Javac warnings in metron-platform outside of parsers and enrichment > - > > Key: METRON-621 > URL: https://issues.apache.org/jira/browse/METRON-621 > Project: Metron > Issue Type: Sub-task >Affects Versions: 0.3.0 >Reporter: Justin Leet >Priority: Minor > > metron-platform has a lot of Javac warnings. Clean up the code as necessary > to get rid of them. Probably mostly just missing > @SuppressWarnings("unchecked") because things like our JSON library use raw > types under the hood. > metron-parsers and metron-enrichment are covered by other tickets based on > how many warnings they have. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-620) Eliminate Javac warnings in metron-enrichment
[ https://issues.apache.org/jira/browse/METRON-620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-620: --- Assignee: (was: Justin Leet) > Eliminate Javac warnings in metron-enrichment > - > > Key: METRON-620 > URL: https://issues.apache.org/jira/browse/METRON-620 > Project: Metron > Issue Type: Sub-task >Affects Versions: 0.3.0 >Reporter: Justin Leet >Priority: Minor > > metron-enrichment has a lot of Javac warnings. Clean up the code as > necessary to get rid of them. Probably mostly just missing > @SuppressWarnings("unchecked") because things like our JSON library use raw > types under the hood. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-621) Eliminate Javac warnings in metron-platform outside of parsers and enrichment
Justin Leet created METRON-621: -- Summary: Eliminate Javac warnings in metron-platform outside of parsers and enrichment Key: METRON-621 URL: https://issues.apache.org/jira/browse/METRON-621 Project: Metron Issue Type: Sub-task Affects Versions: 0.3.0 Reporter: Justin Leet Assignee: Justin Leet Priority: Minor metron-platform has a lot of Javac warnings. Clean up the code as necessary to get rid of them. Probably mostly just missing @SuppressWarnings("unchecked") because things like our JSON library use raw types under the hood. metron-parsers and metron-enrichment are covered by other tickets based on how many warnings they have. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-620) Eliminate Javac warnings in metron-enrichment
Justin Leet created METRON-620: -- Summary: Eliminate Javac warnings in metron-enrichment Key: METRON-620 URL: https://issues.apache.org/jira/browse/METRON-620 Project: Metron Issue Type: Sub-task Affects Versions: 0.3.0 Reporter: Justin Leet Assignee: Justin Leet Priority: Minor metron-enrichment has a lot of Javac warnings. Clean up the code as necessary to get rid of them. Probably mostly just missing @SuppressWarnings("unchecked") because things like our JSON library use raw types under the hood. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-619) Eliminate Javac warnings in metron-parsers
Justin Leet created METRON-619: -- Summary: Eliminate Javac warnings in metron-parsers Key: METRON-619 URL: https://issues.apache.org/jira/browse/METRON-619 Project: Metron Issue Type: Sub-task Affects Versions: 0.3.0 Reporter: Justin Leet Assignee: Justin Leet Priority: Minor Metron-parsers has a lot of Javac warnings. Clean up the code as necessary to get rid of them. Probably mostly just missing @SuppressWarnings("unchecked") because things like our JSON library use raw types under the hood. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-618) Eliminate Javac Warnings in metron-analytics
Justin Leet created METRON-618: -- Summary: Eliminate Javac Warnings in metron-analytics Key: METRON-618 URL: https://issues.apache.org/jira/browse/METRON-618 Project: Metron Issue Type: Sub-task Affects Versions: 0.3.0 Reporter: Justin Leet Assignee: Justin Leet Priority: Minor Kill off the java compiler warnings in maas project, as much as possible. Most of these are related to missing @SuppressWarnings("unchecked") on code that either should have them, or be refactored so it's not necessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-617) Eliminate Javac warnings in the build
Justin Leet created METRON-617: -- Summary: Eliminate Javac warnings in the build Key: METRON-617 URL: https://issues.apache.org/jira/browse/METRON-617 Project: Metron Issue Type: Improvement Affects Versions: 0.3.0 Reporter: Justin Leet Assignee: Justin Leet Priority: Minor We have a lot of Javac warnings (several hundred) in the build right now. The primary cause of these is unchecked types (for reasons including org.simple.json uses raw types under the hood). These should be labelled with @SuppressWarnings("unchecked") where appropriate. In addition, things like @SafeVarargs should be added where appropriate, etc. Given the number of changes in this ticket is significantly larger than even the Error Prone warnings (and may require tinkering with generics at appropriate points), I'm going to split this up by module (Analytics and Platform), and then split platform as necessary (parsers and enrichment in particular have a lot). Otherwise, anything resulting is likely to be too large and unmanageable to be properly tested and reviewed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-615) Prevent warnings in code from future contributions
[ https://issues.apache.org/jira/browse/METRON-615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15730011#comment-15730011 ] Justin Leet commented on METRON-615: Yeah, I agree about new (and even current) contributors. Whatever we do, documenting it for contributors is important, because I also don't want people to get turned off because things blow up that they don't expect. There should also be documentation around what tools we're using and how we enable / disable things like this. These particular things aren't restrictive while something is being built or improved, but I can see cases where you'd want to disable/downgrade errors or warnings while you're working locally. To the best of my knowledge, this doesn't exist in a fleshed out form and should probably end up on the wiki somewhere. > Prevent warnings in code from future contributions > -- > > Key: METRON-615 > URL: https://issues.apache.org/jira/browse/METRON-615 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Priority: Minor > > After cleaning up most of the warnings in METRON-612, there's a question of > locking out additional warnings in the code from Error Prone. This leads to > the more general question of avoiding the addition of warnings in the future > in general, not just from Error Prone but from javac or any other tooling we > may add. > The Hadoop project itself does something pretty extensive, e.g. [Hadoop > QA|https://issues.apache.org/jira/browse/HADOOP-13827?focusedCommentId=15678418&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15678418]. > Settings up something like that might be pretty involved and overkill for > where we are, but it might be a nice end goal. > As an parallel measure, we can potentially turn Error Prone's warnings into > errors for warnings we've stamped out. METRON-612's PR ([Github > PR|https://github.com/apache/incubator-metron/pull/389]), assuming no > additional warnings have come in on master, allows for 4 to be upgraded. > [SynchronizeOnNonFinalField|http://errorprone.info/bugpattern/SynchronizeOnNonFinalField] > [BoxedPrimitiveConstructor|http://errorprone.info/bugpattern/BoxedPrimitiveConstructor] > [ClassCanBeStatic|http://errorprone.info/bugpattern/ClassCanBeStatic] > [ClassNewInstance|http://errorprone.info/bugpattern/ClassNewInstance] > Of these, I'm personally in favor of promoting SynchronizeOnNonFinalField, > BoxedPrimitiveConstructor, and ClassNewInstance. ClassCanbeStatic I'm pretty > neutral on. > In particular, SynchronizeOnNonFinalField is a correctness issue. > BoxedPrimitiveConstructor is a perf / code quality issue. ClassNewInstance > avoids some issues around reflection. Both BoxedPrimitiveConstructor and > ClassNewInstance are expected to have the originating problems become > deprecated in Java 9 in favor of the more correct construction. > [~dlyle]'s opinion per the PR: > {quote} > Totally For Erroring on: > SynchronizeOnNonFinalField > Not against it for: > BoxedPrimitiveConstructor > ClassCanBeStatic > Okay with it for ClassNewInstance with additional input. I get why it's > harmful, but is it more harmful than reflection in general? Or do you just > want to not insert known future deprecated code? > {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-612) Clean up Error Prone generated warnings
[ https://issues.apache.org/jira/browse/METRON-612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-612: --- Fix Version/s: Next + 1 > Clean up Error Prone generated warnings > --- > > Key: METRON-612 > URL: https://issues.apache.org/jira/browse/METRON-612 > Project: Metron > Issue Type: Bug >Affects Versions: 0.3.0 >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Minor > Fix For: Next + 1 > > > As a result of METRON-593, we have a new set of warnings in our code that > should be addressed. 593 already addressed errors (as those would break the > build otherwise). > These can be seen in the mvn compile output of the various modules. It > includes things like missing Override annotations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-615) Prevent warnings in code from future contributions
Justin Leet created METRON-615: -- Summary: Prevent warnings in code from future contributions Key: METRON-615 URL: https://issues.apache.org/jira/browse/METRON-615 Project: Metron Issue Type: Improvement Reporter: Justin Leet Priority: Minor After cleaning up most of the warnings in METRON-612, there's a question of locking out additional warnings in the code from Error Prone. This leads to the more general question of avoiding the addition of warnings in the future in general, not just from Error Prone but from javac or any other tooling we may add. The Hadoop project itself does something pretty extensive, e.g. [Hadoop QA|https://issues.apache.org/jira/browse/HADOOP-13827?focusedCommentId=15678418&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15678418]. Settings up something like that might be pretty involved and overkill for where we are, but it might be a nice end goal. As an parallel measure, we can potentially turn Error Prone's warnings into errors for warnings we've stamped out. METRON-612's PR ([Github PR|https://github.com/apache/incubator-metron/pull/389]), assuming no additional warnings have come in on master, allows for 4 to be upgraded. [SynchronizeOnNonFinalField|http://errorprone.info/bugpattern/SynchronizeOnNonFinalField] [BoxedPrimitiveConstructor|http://errorprone.info/bugpattern/BoxedPrimitiveConstructor] [ClassCanBeStatic|http://errorprone.info/bugpattern/ClassCanBeStatic] [ClassNewInstance|http://errorprone.info/bugpattern/ClassNewInstance] Of these, I'm personally in favor of promoting SynchronizeOnNonFinalField, BoxedPrimitiveConstructor, and ClassNewInstance. ClassCanbeStatic I'm pretty neutral on. In particular, SynchronizeOnNonFinalField is a correctness issue. BoxedPrimitiveConstructor is a perf / code quality issue. ClassNewInstance avoids some issues around reflection. Both BoxedPrimitiveConstructor and ClassNewInstance are expected to have the originating problems become deprecated in Java 9 in favor of the more correct construction. [~dlyle]'s opinion per the PR: {quote} Totally For Erroring on: SynchronizeOnNonFinalField Not against it for: BoxedPrimitiveConstructor ClassCanBeStatic Okay with it for ClassNewInstance with additional input. I get why it's harmful, but is it more harmful than reflection in general? Or do you just want to not insert known future deprecated code? {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-614) Eliminate use of the default Charset
[ https://issues.apache.org/jira/browse/METRON-614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-614: --- Description: During work on METRON-612, I noticed a lot of our warnings are from implicit use of the default Charset (e.g. when creating InputStreamReaders). Given that this value is platform dependent, we should at least consider explicitly using a particular Charset. In addition, after fixing this, I would like to see it upgraded to a compile error, because in my experience almost everyone tends to use methods that use the implicit default Charset. This can be done by changing the compiler arg in the pom to instead be {code} -Xlint:unchecked -Xep:DefaultCharset:ERROR {code} My assumption is that we'll want UTF-8, but given a lack of expertise and that usually there's a lot of opinions on character set issues, we may want to consider other options. http://errorprone.info/bugpattern/DefaultCharset https://docs.oracle.com/javase/8/docs/api/java/nio/charset/Charset.html lists standard Charsets and provides more details. {code} Charset Description US-ASCIISeven-bit ASCII, a.k.a. ISO646-US, a.k.a. the Basic Latin block of the Unicode character set ISO-8859-1 ISO Latin Alphabet No. 1, a.k.a. ISO-LATIN-1 UTF-8 Eight-bit UCS Transformation Format UTF-16BESixteen-bit UCS Transformation Format, big-endian byte order UTF-16LESixteen-bit UCS Transformation Format, little-endian byte order UTF-16 Sixteen-bit UCS Transformation Format, byte order identified by an optional byte-order mark {code} was: During work on Metron-612, I noticed a lot of our warnings are from implicit use of the default Charset (e.g. when creating InputStreamReaders). Given that this value is platform dependent, we should at least consider explicitly using a particular Charset. In addition, after fixing this, I would like to see it upgraded to a compile error, because in my experience almost everyone tends to use methods that use the implicit default Charset. This can be done by changing the compiler arg in the pom to instead be {code} -Xlint:unchecked -Xep:DefaultCharset:ERROR {code} My assumption is that we'll want UTF-8, but given a lack of expertise and that usually there's a lot of opinions on character set issues, we may want to consider other options. http://errorprone.info/bugpattern/DefaultCharset https://docs.oracle.com/javase/8/docs/api/java/nio/charset/Charset.html lists standard Charsets and provides more details. {code} Charset Description US-ASCIISeven-bit ASCII, a.k.a. ISO646-US, a.k.a. the Basic Latin block of the Unicode character set ISO-8859-1 ISO Latin Alphabet No. 1, a.k.a. ISO-LATIN-1 UTF-8 Eight-bit UCS Transformation Format UTF-16BESixteen-bit UCS Transformation Format, big-endian byte order UTF-16LESixteen-bit UCS Transformation Format, little-endian byte order UTF-16 Sixteen-bit UCS Transformation Format, byte order identified by an optional byte-order mark {code} > Eliminate use of the default Charset > > > Key: METRON-614 > URL: https://issues.apache.org/jira/browse/METRON-614 > Project: Metron > Issue Type: Bug >Reporter: Justin Leet >Priority: Minor > > During work on METRON-612, I noticed a lot of our warnings are from implicit > use of the default Charset (e.g. when creating InputStreamReaders). Given > that this value is platform dependent, we should at least consider explicitly > using a particular Charset. > In addition, after fixing this, I would like to see it upgraded to a compile > error, because in my experience almost everyone tends to use methods that use > the implicit default Charset. This can be done by changing the compiler arg > in the pom to instead be > {code} > > -Xlint:unchecked > -Xep:DefaultCharset:ERROR > > {code} > My assumption is that we'll want UTF-8, but given a lack of expertise and > that usually there's a lot of opinions on character set issues, we may want > to consider other options. > http://errorprone.info/bugpattern/DefaultCharset > https://docs.oracle.com/javase/8/docs/api/java/nio/charset/Charset.html lists > standard Charsets and provides more details. > {code} > Charset Description > US-ASCII Seven-bit ASCII, a.k.a. ISO646-US, a.k.a. the Basic Latin block > of the Unicode character set > ISO-8859-1ISO Latin Alphabet No. 1, a.k.a. ISO-LATIN-1 > UTF-8 Eight-bit UCS Transformation Format > UTF-16BE Sixteen-bit UCS Transformation Format, big-endian byte order > UTF-16LE Sixteen-bit UCS Transformation Format, little-endian byte order > UTF-16Sixteen-bit UCS Transformation Format, byte order identified by > an optional byte-order mark > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-614) Eliminate use of the default Charset
Justin Leet created METRON-614: -- Summary: Eliminate use of the default Charset Key: METRON-614 URL: https://issues.apache.org/jira/browse/METRON-614 Project: Metron Issue Type: Bug Reporter: Justin Leet Priority: Minor During work on Metron-612, I noticed a lot of our warnings are from implicit use of the default Charset (e.g. when creating InputStreamReaders). Given that this value is platform dependent, we should at least consider explicitly using a particular Charset. In addition, after fixing this, I would like to see it upgraded to a compile error, because in my experience almost everyone tends to use methods that use the implicit default Charset. This can be done by changing the compiler arg in the pom to instead be {code} -Xlint:unchecked -Xep:DefaultCharset:ERROR {code} My assumption is that we'll want UTF-8, but given a lack of expertise and that usually there's a lot of opinions on character set issues, we may want to consider other options. http://errorprone.info/bugpattern/DefaultCharset https://docs.oracle.com/javase/8/docs/api/java/nio/charset/Charset.html lists standard Charsets and provides more details. {code} Charset Description US-ASCIISeven-bit ASCII, a.k.a. ISO646-US, a.k.a. the Basic Latin block of the Unicode character set ISO-8859-1 ISO Latin Alphabet No. 1, a.k.a. ISO-LATIN-1 UTF-8 Eight-bit UCS Transformation Format UTF-16BESixteen-bit UCS Transformation Format, big-endian byte order UTF-16LESixteen-bit UCS Transformation Format, little-endian byte order UTF-16 Sixteen-bit UCS Transformation Format, byte order identified by an optional byte-order mark {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-593) Enable an automated static analysis tool in the build
[ https://issues.apache.org/jira/browse/METRON-593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-593: --- Fix Version/s: Next + 1 > Enable an automated static analysis tool in the build > - > > Key: METRON-593 > URL: https://issues.apache.org/jira/browse/METRON-593 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.0 >Reporter: Justin Leet >Assignee: Justin Leet > Fix For: Next + 1 > > > From on a discussion thread I kicked off on the dev list. Some newer notes > follow > h4. Original Email > I wanted to kick off a discussion about integrating a static analysis tool > into our builds. > The main discussion points I wanted to start up (and feel encouraged to add > more): > 1) Most importantly, do we get enough value by adding a tool? > 2) What are we looking for out of a tool (Extensibility to add our own > checks, plugged into build cycle directly, ease of use, customizability, > etc.)? > 3) Are there any particular tools people have experience with? > 4) Assuming we want to roll something out, what's the best path? My current > assumption is that it's probably easiest to handle things on a pom by pom > basis, rather than trying to do everything at once, but there may be more > nuance people want to add. > The main one I've used FindBugs, but there's a been discussion lately about > issues with their community which led me to take a (very) brief glance at > into Google's errorprone. It seems like it's an alternative worth considering > from what I've seen. > Some links to errorprone info: > http://errorprone.info/ > https://github.com/google/error-prone > http://errorprone.info/bugpatterns > I played around with it for about 2 minutes, and was able to get it up and > running and happily complaining about metron-common during it's build cycle. > Haven't dug too much into the errors/warnings to get a sense of signal to > noise ratio. If anybody is interested in playing around with that setup for > metron-common, I have a branch at: > https://github.com/justinleet/incubator-metron/tree/errorprone > Just go to metron-platform/metron-common and run: > mvn compile > Gist with the error prone output. > https://gist.github.com/justinleet/8d514727a0caeb5db2b4f76de0607214 > h4. New notes > After playing around with error prone a bit more (I was able to get it > running on most of our modules and fix build breaking errors pretty quickly), > it provides significantly less check coverage than findbugs, but has the > benefit of being directly tied into the compile (meaning people can get > feedback and errors pretty quickly). Related to this, the errors that error > prone gave out were actual issues (although relatively minor in our codebase, > e.g. catching issues with format() calls). > It seems the benefits of error prone fall primarily on it coming earlier in > the lifecycle to give feedback quicker and being actively maintained and > improved. > The main benefit of findbugs is being a broader set of checks, but at the > expense of being later in the build cycle (because it operates on bytecode), > and the community being in an odd place right now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-612) Clean up Error Prone generated warnings
[ https://issues.apache.org/jira/browse/METRON-612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15725463#comment-15725463 ] Justin Leet commented on METRON-612: We have a (very) large number of warnings of the form {code} /home/travis/build/justinleet/incubator-metron/metron-analytics/metron-maas-common/src/main/java/org/apache/metron/maas/util/RESTUtil.java:64: warning: [DefaultCharset] Implicit use of the platform default charset, which can result in e.g. non-ASCII characters being silently replaced with '?' in many environments return new BufferedReader(new InputStreamReader(response.getEntity().getContent())) ^ (see http://errorprone.info/bugpattern/DefaultCharset) Did you mean 'return new BufferedReader(new InputStreamReader(response.getEntity().getContent(), UTF_8))' or 'return new BufferedReader(new InputStreamReader(response.getEntity().getContent(), Charset.defaultCharset()))'? {code} See http://docs.oracle.com/javase/8/docs/api/java/nio/charset/Charset.html for standard Charsets. Given that the default Charset is platform dependent, we should at least consider making a conscious choice on this. If we don't want to make a decision, it is also possible to set this warning to ignore, but I'm not generally a fan of doing that for warnings. Does anybody with more knowledge / expertise have any input on this? > Clean up Error Prone generated warnings > --- > > Key: METRON-612 > URL: https://issues.apache.org/jira/browse/METRON-612 > Project: Metron > Issue Type: Bug >Affects Versions: 0.3.0 >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Minor > > As a result of METRON-593, we have a new set of warnings in our code that > should be addressed. 593 already addressed errors (as those would break the > build otherwise). > These can be seen in the mvn compile output of the various modules. It > includes things like missing Override annotations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-609) Enhance Mpack to handle single-node and small-cluster installs of Elasticsearch
[ https://issues.apache.org/jira/browse/METRON-609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15725452#comment-15725452 ] Justin Leet commented on METRON-609: For some additional context, the master only and data configurations only ES nodes differ in two properties boolean properties (at least for now): Master: {code} node: data: false master: true {code} These configs are reversed for a Data only node. They can both be true and it'll be a combined node. Some other configs may have to change to support this, e.g. {code} gateway_recover_after_data_nodes 3 Recover as long as this many data or master nodes have joined the cluster. {code} In Ambari service definitions, a cardinality is given for each component (In this case Master 1+ and Data 3+). Off the top of my seems like there's a couple approaches to combined nodes (working within the current setup): 1. Separate component for combined nodes: If we have combined master / data nodes we want to be able to enforce something like: (1+ C node OR (1+ M AND 3+D)). To the best of my (very) limited knowledge, Ambari doesn't support this. 2. Making Master nodes have a config option for holding data: We could make the data option for Master be controlled by configuration and set cardinality of D to 0+. I don't know if a rational cluster definition can be enforced with this type of setup. > Enhance Mpack to handle single-node and small-cluster installs of > Elasticsearch > --- > > Key: METRON-609 > URL: https://issues.apache.org/jira/browse/METRON-609 > Project: Metron > Issue Type: Improvement >Reporter: Matt Foley >Assignee: Matt Foley > > The current Mpack for Ambari install of Metron requires that Elasticsearch be > installed with 1+ dedicated Masters and 3+ dedicated Slaves. It does not > support combined master/datanode configuration (non-"dedicated" Master), > which is the recommended way to run a single-node configuration of > Elasticsearch; nor does it allow less than 3 dedicated Slaves, thereby > requiring the cluster have at least 4 nodes. > This jira proposes to enable 1-, 2-, and 3-node clusters to have a working > Elasticsearch install via the Mpack. It will also allow non-dedicated > master/datanodes, which are required for single-node and useful for few-node > clusters. > I intend to also include the following enhancements and fixes: > * Determine whether elastic-sysconfig:data_dir and elastic-site:path_data > should have same default value and if so fix it. (I think they should, and > they don't. Probably there should only be one variable instead of two.) > * Provide Quick Links in Ambari service page for Elasticsearch to the > self-report pages for Elasticsearch health, node list, and other "_cat" > status. May include some "_cluster" status. > * Improve various mouse-over Description fields in the GUI > * Improve the order of fields in the GUI, to group at the beginning the > fields that must be modified or are commonly modified. > * Provide a README.md that describes what is going on with the various > resource files in the Mpack src > * Enhance the wiki page for cluster installation, with regard to single-node > install with Mpack. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-596) Eliminate Maven warnings from build
[ https://issues.apache.org/jira/browse/METRON-596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-596: --- Fix Version/s: Next + 1 > Eliminate Maven warnings from build > --- > > Key: METRON-596 > URL: https://issues.apache.org/jira/browse/METRON-596 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.0 >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Minor > Fix For: Next + 1 > > > Right now our builds throw some warnings in Maven. Things like not specifying > a version for plugins, etc. These should be cleaned up. > Current Build warnings: > {code} > [INFO] Scanning for projects... > [WARNING] > [WARNING] Some problems were encountered while building the effective model > for org.apache.metron:metron-common:jar:0.3.0 > [WARNING] 'build.plugins.plugin.version' for > org.apache.maven.plugins:maven-jar-plugin is missing. @ > org.apache.metron:metron-common:[unknown-version], > /Users/jleet/Documents/workspace/incubator-metron/metron-platform/metron-common/pom.xml, > line 360, column 21 > [WARNING] > [WARNING] Some problems were encountered while building the effective model > for org.apache.metron:metron-enrichment:jar:0.3.0 > [WARNING] 'build.plugins.plugin.version' for > org.apache.maven.plugins:maven-jar-plugin is missing. @ > org.apache.metron:metron-enrichment:[unknown-version], > /Users/jleet/Documents/workspace/incubator-metron/metron-platform/metron-enrichment/pom.xml, > line 287, column 21 > [WARNING] > [WARNING] Some problems were encountered while building the effective model > for org.apache.metron:metron-parsers:jar:0.3.0 > [WARNING] 'build.plugins.plugin.version' for > org.apache.maven.plugins:maven-jar-plugin is missing. @ > org.apache.metron:metron-parsers:[unknown-version], > /Users/jleet/Documents/workspace/incubator-metron/metron-platform/metron-parsers/pom.xml, > line 250, column 21 > [WARNING] > [WARNING] Some problems were encountered while building the effective model > for org.apache.metron:metron-indexing:jar:0.3.0 > [WARNING] 'build.plugins.plugin.version' for > org.apache.maven.plugins:maven-jar-plugin is missing. @ > org.apache.metron:metron-indexing:[unknown-version], > /Users/jleet/Documents/workspace/incubator-metron/metron-platform/metron-indexing/pom.xml, > line 183, column 21 > [WARNING] > [WARNING] Some problems were encountered while building the effective model > for org.apache.metron:metron-management:jar:0.3.0 > [WARNING] The expression ${parent.version} is deprecated. Please use > ${project.parent.version} instead. > [WARNING] The expression ${parent.version} is deprecated. Please use > ${project.parent.version} instead. > [WARNING] The expression ${parent.version} is deprecated. Please use > ${project.parent.version} instead. > [WARNING] > [WARNING] Some problems were encountered while building the effective model > for org.apache.metron:metron-hbase:jar:0.3.0 > [WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must > be unique: org.apache.hadoop:hadoop-hdfs:jar:tests -> duplicate declaration > of version ${global_hadoop_version} @ > org.apache.metron:metron-hbase:[unknown-version], > /Users/jleet/Documents/workspace/incubator-metron/metron-platform/metron-hbase/pom.xml, > line 166, column 21 > [WARNING] > [WARNING] It is highly recommended to fix these problems because they > threaten the stability of your build. > [WARNING] > [WARNING] For this reason, future Maven versions might no longer support > building such malformed projects. > [WARNING] > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-604) Mpack installs do not work on clean machines due to missing Elastic Curator repo
[ https://issues.apache.org/jira/browse/METRON-604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-604: --- Fix Version/s: Next + 1 > Mpack installs do not work on clean machines due to missing Elastic Curator > repo > > > Key: METRON-604 > URL: https://issues.apache.org/jira/browse/METRON-604 > Project: Metron > Issue Type: Bug >Affects Versions: 0.3.0 >Reporter: David M. Lyle >Assignee: Justin Leet > Fix For: Next + 1 > > > I cut fresh rpms/mpack for myself following [~justinleet]'s [readme| > https://github.com/justinleet/metron-rpm] > Add the mpack to Ambari and then build a fresh 10 node cluster, HDP 2.5.3 and > Ambar 2.4.2 (note my last attempt was with HDP 2.5.0 and Ambari 2.4.1 iirc) > Add the services, hit deploy, everything goes fine until it attempts to > deploy kibana, the install fails because it cannot find the > python-elasticsearch package within the repos we deploy using the mpack. > If I manually add the following repo to the server where Ambari is attempting > to install Kibana then I can retry and everything works fine. > {noformat} > [curator-4] > name=CentOS/RHEL 7 repository for Elasticsearch Curator 4.x packages > baseurl=http://packages.elastic.co/curator/4/centos/7 > gpgcheck=1 > gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch > enabled=1 > {noformat} > I don’t know for sure this is the correct repository for us to use based on > the version of ES/Kibana that we’re using, I just know it contains a package > which satisfies the dependency requirements and allows the install to > complete successfully. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-604) Mpack installs do not work on clean machines due to missing Elastic Curator repo
[ https://issues.apache.org/jira/browse/METRON-604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-604: --- Summary: Mpack installs do not work on clean machines due to missing Elastic Curator repo (was: Mpack installs do not work on clean machines due to missing Elastic repo) > Mpack installs do not work on clean machines due to missing Elastic Curator > repo > > > Key: METRON-604 > URL: https://issues.apache.org/jira/browse/METRON-604 > Project: Metron > Issue Type: Bug >Affects Versions: 0.3.0 >Reporter: David M. Lyle >Assignee: Justin Leet > > I cut fresh rpms/mpack for myself following [~justinleet]'s [readme| > https://github.com/justinleet/metron-rpm] > Add the mpack to Ambari and then build a fresh 10 node cluster, HDP 2.5.3 and > Ambar 2.4.2 (note my last attempt was with HDP 2.5.0 and Ambari 2.4.1 iirc) > Add the services, hit deploy, everything goes fine until it attempts to > deploy kibana, the install fails because it cannot find the > python-elasticsearch package within the repos we deploy using the mpack. > If I manually add the following repo to the server where Ambari is attempting > to install Kibana then I can retry and everything works fine. > {noformat} > [curator-4] > name=CentOS/RHEL 7 repository for Elasticsearch Curator 4.x packages > baseurl=http://packages.elastic.co/curator/4/centos/7 > gpgcheck=1 > gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch > enabled=1 > {noformat} > I don’t know for sure this is the correct repository for us to use based on > the version of ES/Kibana that we’re using, I just know it contains a package > which satisfies the dependency requirements and allows the install to > complete successfully. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-612) Clean up Error Prone generated warnings
Justin Leet created METRON-612: -- Summary: Clean up Error Prone generated warnings Key: METRON-612 URL: https://issues.apache.org/jira/browse/METRON-612 Project: Metron Issue Type: Bug Affects Versions: 0.3.0 Reporter: Justin Leet Assignee: Justin Leet Priority: Minor As a result of METRON-593, we have a new set of warnings in our code that should be addressed. 593 already addressed errors (as those would break the build otherwise). These can be seen in the mvn compile output of the various modules. It includes things like missing Override annotations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (METRON-604) Mpack installs do not work on clean machines due to missing Elastic repo
[ https://issues.apache.org/jira/browse/METRON-604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-604: -- Assignee: Justin Leet > Mpack installs do not work on clean machines due to missing Elastic repo > > > Key: METRON-604 > URL: https://issues.apache.org/jira/browse/METRON-604 > Project: Metron > Issue Type: Bug >Affects Versions: 0.3.0 >Reporter: David M. Lyle >Assignee: Justin Leet > > I cut fresh rpms/mpack for myself following [~justinleet]'s [readme| > https://github.com/justinleet/metron-rpm] > Add the mpack to Ambari and then build a fresh 10 node cluster, HDP 2.5.3 and > Ambar 2.4.2 (note my last attempt was with HDP 2.5.0 and Ambari 2.4.1 iirc) > Add the services, hit deploy, everything goes fine until it attempts to > deploy kibana, the install fails because it cannot find the > python-elasticsearch package within the repos we deploy using the mpack. > If I manually add the following repo to the server where Ambari is attempting > to install Kibana then I can retry and everything works fine. > {noformat} > [curator-4] > name=CentOS/RHEL 7 repository for Elasticsearch Curator 4.x packages > baseurl=http://packages.elastic.co/curator/4/centos/7 > gpgcheck=1 > gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch > enabled=1 > {noformat} > I don’t know for sure this is the correct repository for us to use based on > the version of ES/Kibana that we’re using, I just know it contains a package > which satisfies the dependency requirements and allows the install to > complete successfully. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-597) Sporadic Failures of Profiler Integration Tests
[ https://issues.apache.org/jira/browse/METRON-597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15708818#comment-15708818 ] Justin Leet commented on METRON-597: https://github.com/apache/incubator-metron/pull/378 has only one commit difference with apache/master: METRON-588. It also just failed again on this test > Sporadic Failures of Profiler Integration Tests > --- > > Key: METRON-597 > URL: https://issues.apache.org/jira/browse/METRON-597 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen > > Seems to be some kind of timing issue. > {code} > --- > T E S T S > --- > Running org.apache.metron.profiler.integration.ProfilerIntegrationTest > Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 66.564 sec > <<< FAILURE! > testExample3(org.apache.metron.profiler.integration.ProfilerIntegrationTest) > Time elapsed: 16.759 sec <<< FAILURE! > java.lang.AssertionError: expected:<20.0> but was: > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:834) > at org.junit.Assert.assertEquals(Assert.java:553) > at org.junit.Assert.assertEquals(Assert.java:683) > at > org.apache.metron.profiler.integration.ProfilerIntegrationTest.testExample3(ProfilerIntegrationTest.java:203) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:483) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:483) > at > org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) > at > org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) > at > org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75) > Results : > Failed tests: > testExample3(org.apache.metron.profiler.integration.ProfilerIntegrationTest): > expected:<20.0> but was: > Tests run: 5, Failures: 1, Errors: 0, Skipped: 0 > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-597) Sporadic Failures of Profiler Integration Tests
[ https://issues.apache.org/jira/browse/METRON-597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15708659#comment-15708659 ] Justin Leet commented on METRON-597: I'll try to keep an eye out for test failures in the other packages, too [~nickwallen]. You're right that we could have larger integration test issues, but given that those have had some refactoring lately, I'm not sure I want to group the others into the same bucket yet. > Sporadic Failures of Profiler Integration Tests > --- > > Key: METRON-597 > URL: https://issues.apache.org/jira/browse/METRON-597 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen > > Seems to be some kind of timing issue. > {code} > --- > T E S T S > --- > Running org.apache.metron.profiler.integration.ProfilerIntegrationTest > Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 66.564 sec > <<< FAILURE! > testExample3(org.apache.metron.profiler.integration.ProfilerIntegrationTest) > Time elapsed: 16.759 sec <<< FAILURE! > java.lang.AssertionError: expected:<20.0> but was: > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:834) > at org.junit.Assert.assertEquals(Assert.java:553) > at org.junit.Assert.assertEquals(Assert.java:683) > at > org.apache.metron.profiler.integration.ProfilerIntegrationTest.testExample3(ProfilerIntegrationTest.java:203) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:483) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:483) > at > org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) > at > org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) > at > org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75) > Results : > Failed tests: > testExample3(org.apache.metron.profiler.integration.ProfilerIntegrationTest): > expected:<20.0> but was: > Tests run: 5, Failures: 1, Errors: 0, Skipped: 0 > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-596) Eliminate Maven warnings from build
Justin Leet created METRON-596: -- Summary: Eliminate Maven warnings from build Key: METRON-596 URL: https://issues.apache.org/jira/browse/METRON-596 Project: Metron Issue Type: Improvement Affects Versions: 0.3.0 Reporter: Justin Leet Assignee: Justin Leet Priority: Minor Right now our builds throw some warnings in Maven. Things like not specifying a version for plugins, etc. These should be cleaned up. Current Build warnings: {code} [INFO] Scanning for projects... [WARNING] [WARNING] Some problems were encountered while building the effective model for org.apache.metron:metron-common:jar:0.3.0 [WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-jar-plugin is missing. @ org.apache.metron:metron-common:[unknown-version], /Users/jleet/Documents/workspace/incubator-metron/metron-platform/metron-common/pom.xml, line 360, column 21 [WARNING] [WARNING] Some problems were encountered while building the effective model for org.apache.metron:metron-enrichment:jar:0.3.0 [WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-jar-plugin is missing. @ org.apache.metron:metron-enrichment:[unknown-version], /Users/jleet/Documents/workspace/incubator-metron/metron-platform/metron-enrichment/pom.xml, line 287, column 21 [WARNING] [WARNING] Some problems were encountered while building the effective model for org.apache.metron:metron-parsers:jar:0.3.0 [WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-jar-plugin is missing. @ org.apache.metron:metron-parsers:[unknown-version], /Users/jleet/Documents/workspace/incubator-metron/metron-platform/metron-parsers/pom.xml, line 250, column 21 [WARNING] [WARNING] Some problems were encountered while building the effective model for org.apache.metron:metron-indexing:jar:0.3.0 [WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-jar-plugin is missing. @ org.apache.metron:metron-indexing:[unknown-version], /Users/jleet/Documents/workspace/incubator-metron/metron-platform/metron-indexing/pom.xml, line 183, column 21 [WARNING] [WARNING] Some problems were encountered while building the effective model for org.apache.metron:metron-management:jar:0.3.0 [WARNING] The expression ${parent.version} is deprecated. Please use ${project.parent.version} instead. [WARNING] The expression ${parent.version} is deprecated. Please use ${project.parent.version} instead. [WARNING] The expression ${parent.version} is deprecated. Please use ${project.parent.version} instead. [WARNING] [WARNING] Some problems were encountered while building the effective model for org.apache.metron:metron-hbase:jar:0.3.0 [WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.apache.hadoop:hadoop-hdfs:jar:tests -> duplicate declaration of version ${global_hadoop_version} @ org.apache.metron:metron-hbase:[unknown-version], /Users/jleet/Documents/workspace/incubator-metron/metron-platform/metron-hbase/pom.xml, line 166, column 21 [WARNING] [WARNING] It is highly recommended to fix these problems because they threaten the stability of your build. [WARNING] [WARNING] For this reason, future Maven versions might no longer support building such malformed projects. [WARNING] {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-593) Enable an automated static analysis tool in the build
[ https://issues.apache.org/jira/browse/METRON-593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15705751#comment-15705751 ] Justin Leet commented on METRON-593: It's built in to a given version of error-prone. The actual bugpatterns are implemented at [Bug Patterns|https://github.com/google/error-prone/tree/master/core/src/main/java/com/google/errorprone/bugpatterns] in their repo. The webpage's docs are, as noted on the http://errorprone.info/bugpatterns, are autogenerated from that source. For example, MissingOverride.java results in http://errorprone.info/bugpattern/MissingOverride, which would be how the compilation knows what link to generate (I assume, but I haven't dug into their source that much). At that point, updating to latest ruleset is a matter of updating error-prone (which should be updating the version in the pom, and fixing any new errors that arise). As a sidenote, as we fix warnings, we can start promoting those to errors (or turn them off if we determine they're not something we're actually concerned bout) if we want as described in http://errorprone.info/docs/flags. > Enable an automated static analysis tool in the build > - > > Key: METRON-593 > URL: https://issues.apache.org/jira/browse/METRON-593 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.0 >Reporter: Justin Leet >Assignee: Justin Leet > > From on a discussion thread I kicked off on the dev list. Some newer notes > follow > h4. Original Email > I wanted to kick off a discussion about integrating a static analysis tool > into our builds. > The main discussion points I wanted to start up (and feel encouraged to add > more): > 1) Most importantly, do we get enough value by adding a tool? > 2) What are we looking for out of a tool (Extensibility to add our own > checks, plugged into build cycle directly, ease of use, customizability, > etc.)? > 3) Are there any particular tools people have experience with? > 4) Assuming we want to roll something out, what's the best path? My current > assumption is that it's probably easiest to handle things on a pom by pom > basis, rather than trying to do everything at once, but there may be more > nuance people want to add. > The main one I've used FindBugs, but there's a been discussion lately about > issues with their community which led me to take a (very) brief glance at > into Google's errorprone. It seems like it's an alternative worth considering > from what I've seen. > Some links to errorprone info: > http://errorprone.info/ > https://github.com/google/error-prone > http://errorprone.info/bugpatterns > I played around with it for about 2 minutes, and was able to get it up and > running and happily complaining about metron-common during it's build cycle. > Haven't dug too much into the errors/warnings to get a sense of signal to > noise ratio. If anybody is interested in playing around with that setup for > metron-common, I have a branch at: > https://github.com/justinleet/incubator-metron/tree/errorprone > Just go to metron-platform/metron-common and run: > mvn compile > Gist with the error prone output. > https://gist.github.com/justinleet/8d514727a0caeb5db2b4f76de0607214 > h4. New notes > After playing around with error prone a bit more (I was able to get it > running on most of our modules and fix build breaking errors pretty quickly), > it provides significantly less check coverage than findbugs, but has the > benefit of being directly tied into the compile (meaning people can get > feedback and errors pretty quickly). Related to this, the errors that error > prone gave out were actual issues (although relatively minor in our codebase, > e.g. catching issues with format() calls). > It seems the benefits of error prone fall primarily on it coming earlier in > the lifecycle to give feedback quicker and being actively maintained and > improved. > The main benefit of findbugs is being a broader set of checks, but at the > expense of being later in the build cycle (because it operates on bytecode), > and the community being in an odd place right now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-593) Enable an automated static analysis tool in the build
[ https://issues.apache.org/jira/browse/METRON-593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-593: --- Affects Version/s: 0.3.0 > Enable an automated static analysis tool in the build > - > > Key: METRON-593 > URL: https://issues.apache.org/jira/browse/METRON-593 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.0 >Reporter: Justin Leet >Assignee: Justin Leet > > From on a discussion thread I kicked off on the dev list. Some newer notes > follow > h4. Original Email > I wanted to kick off a discussion about integrating a static analysis tool > into our builds. > The main discussion points I wanted to start up (and feel encouraged to add > more): > 1) Most importantly, do we get enough value by adding a tool? > 2) What are we looking for out of a tool (Extensibility to add our own > checks, plugged into build cycle directly, ease of use, customizability, > etc.)? > 3) Are there any particular tools people have experience with? > 4) Assuming we want to roll something out, what's the best path? My current > assumption is that it's probably easiest to handle things on a pom by pom > basis, rather than trying to do everything at once, but there may be more > nuance people want to add. > The main one I've used FindBugs, but there's a been discussion lately about > issues with their community which led me to take a (very) brief glance at > into Google's errorprone. It seems like it's an alternative worth considering > from what I've seen. > Some links to errorprone info: > http://errorprone.info/ > https://github.com/google/error-prone > http://errorprone.info/bugpatterns > I played around with it for about 2 minutes, and was able to get it up and > running and happily complaining about metron-common during it's build cycle. > Haven't dug too much into the errors/warnings to get a sense of signal to > noise ratio. If anybody is interested in playing around with that setup for > metron-common, I have a branch at: > https://github.com/justinleet/incubator-metron/tree/errorprone > Just go to metron-platform/metron-common and run: > mvn compile > Gist with the error prone output. > https://gist.github.com/justinleet/8d514727a0caeb5db2b4f76de0607214 > h4. New notes > After playing around with error prone a bit more (I was able to get it > running on most of our modules and fix build breaking errors pretty quickly), > it provides significantly less check coverage than findbugs, but has the > benefit of being directly tied into the compile (meaning people can get > feedback and errors pretty quickly). Related to this, the errors that error > prone gave out were actual issues (although relatively minor in our codebase, > e.g. catching issues with format() calls). > It seems the benefits of error prone fall primarily on it coming earlier in > the lifecycle to give feedback quicker and being actively maintained and > improved. > The main benefit of findbugs is being a broader set of checks, but at the > expense of being later in the build cycle (because it operates on bytecode), > and the community being in an odd place right now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-593) Enable an automated static analysis tool in the build
[ https://issues.apache.org/jira/browse/METRON-593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15705606#comment-15705606 ] Justin Leet commented on METRON-593: For adding error-prone to all the projects, with all error level issues fixed. Warnings have not been fixed (and result in appropriate additional build output). My personal branch is updated with these changes, and I'm putting it out so people can look at it. Tests have been run, but quick-dev has not been run yet. Output of the build: https://gist.github.com/justinleet/1082bc86dd53abad1f34aa7bf0519a67 > Enable an automated static analysis tool in the build > - > > Key: METRON-593 > URL: https://issues.apache.org/jira/browse/METRON-593 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Assignee: Justin Leet > > From on a discussion thread I kicked off on the dev list. Some newer notes > follow > h4. Original Email > I wanted to kick off a discussion about integrating a static analysis tool > into our builds. > The main discussion points I wanted to start up (and feel encouraged to add > more): > 1) Most importantly, do we get enough value by adding a tool? > 2) What are we looking for out of a tool (Extensibility to add our own > checks, plugged into build cycle directly, ease of use, customizability, > etc.)? > 3) Are there any particular tools people have experience with? > 4) Assuming we want to roll something out, what's the best path? My current > assumption is that it's probably easiest to handle things on a pom by pom > basis, rather than trying to do everything at once, but there may be more > nuance people want to add. > The main one I've used FindBugs, but there's a been discussion lately about > issues with their community which led me to take a (very) brief glance at > into Google's errorprone. It seems like it's an alternative worth considering > from what I've seen. > Some links to errorprone info: > http://errorprone.info/ > https://github.com/google/error-prone > http://errorprone.info/bugpatterns > I played around with it for about 2 minutes, and was able to get it up and > running and happily complaining about metron-common during it's build cycle. > Haven't dug too much into the errors/warnings to get a sense of signal to > noise ratio. If anybody is interested in playing around with that setup for > metron-common, I have a branch at: > https://github.com/justinleet/incubator-metron/tree/errorprone > Just go to metron-platform/metron-common and run: > mvn compile > Gist with the error prone output. > https://gist.github.com/justinleet/8d514727a0caeb5db2b4f76de0607214 > h4. New notes > After playing around with error prone a bit more (I was able to get it > running on most of our modules and fix build breaking errors pretty quickly), > it provides significantly less check coverage than findbugs, but has the > benefit of being directly tied into the compile (meaning people can get > feedback and errors pretty quickly). Related to this, the errors that error > prone gave out were actual issues (although relatively minor in our codebase, > e.g. catching issues with format() calls). > It seems the benefits of error prone fall primarily on it coming earlier in > the lifecycle to give feedback quicker and being actively maintained and > improved. > The main benefit of findbugs is being a broader set of checks, but at the > expense of being later in the build cycle (because it operates on bytecode), > and the community being in an odd place right now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-593) Enable an automated static analysis tool in the build
Justin Leet created METRON-593: -- Summary: Enable an automated static analysis tool in the build Key: METRON-593 URL: https://issues.apache.org/jira/browse/METRON-593 Project: Metron Issue Type: Improvement Reporter: Justin Leet Assignee: Justin Leet >From on a discussion thread I kicked off on the dev list. Some newer notes >follow h4. Original Email I wanted to kick off a discussion about integrating a static analysis tool into our builds. The main discussion points I wanted to start up (and feel encouraged to add more): 1) Most importantly, do we get enough value by adding a tool? 2) What are we looking for out of a tool (Extensibility to add our own checks, plugged into build cycle directly, ease of use, customizability, etc.)? 3) Are there any particular tools people have experience with? 4) Assuming we want to roll something out, what's the best path? My current assumption is that it's probably easiest to handle things on a pom by pom basis, rather than trying to do everything at once, but there may be more nuance people want to add. The main one I've used FindBugs, but there's a been discussion lately about issues with their community which led me to take a (very) brief glance at into Google's errorprone. It seems like it's an alternative worth considering from what I've seen. Some links to errorprone info: http://errorprone.info/ https://github.com/google/error-prone http://errorprone.info/bugpatterns I played around with it for about 2 minutes, and was able to get it up and running and happily complaining about metron-common during it's build cycle. Haven't dug too much into the errors/warnings to get a sense of signal to noise ratio. If anybody is interested in playing around with that setup for metron-common, I have a branch at: https://github.com/justinleet/incubator-metron/tree/errorprone Just go to metron-platform/metron-common and run: mvn compile Gist with the error prone output. https://gist.github.com/justinleet/8d514727a0caeb5db2b4f76de0607214 h4. New notes After playing around with error prone a bit more (I was able to get it running on most of our modules and fix build breaking errors pretty quickly), it provides significantly less check coverage than findbugs, but has the benefit of being directly tied into the compile (meaning people can get feedback and errors pretty quickly). Related to this, the errors that error prone gave out were actual issues (although relatively minor in our codebase, e.g. catching issues with format() calls). It seems the benefits of error prone fall primarily on it coming earlier in the lifecycle to give feedback quicker and being actively maintained and improved. The main benefit of findbugs is being a broader set of checks, but at the expense of being later in the build cycle (because it operates on bytecode), and the community being in an odd place right now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (METRON-522) Need to mandate Client installation on Metron Host
[ https://issues.apache.org/jira/browse/METRON-522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-522: -- Assignee: Justin Leet > Need to mandate Client installation on Metron Host > -- > > Key: METRON-522 > URL: https://issues.apache.org/jira/browse/METRON-522 > Project: Metron > Issue Type: Bug >Affects Versions: 0.2.1BETA >Reporter: Anand Subramanian >Assignee: Justin Leet > > This is while deploying Metron using Ambari mpacks on a existing HDP cluster > by adding Metron services subsequently. > While the Metron components colocation criteria is addressed by defect > [Metron-464|https://issues.apache.org/jira/browse/METRON-464], there needs to > be another check for ensuring that the Clients (ZK, HBASE etc.) must be > installed on the host where Metron components are installed. > This defect is created to track the enhancement for checking the presence of > client on the host selected for Metron components. > Note that one can use the Host->MetronHost->Add Clients option to install > client on the Metron host and subsequently "Add Service" again. However, it > would be good to check for this when the wizard is fired up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-552) Ambari Mpack should be able to manage pcap topology
[ https://issues.apache.org/jira/browse/METRON-552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15683694#comment-15683694 ] Justin Leet commented on METRON-552: Addendum to the addendum. The metron user doesn't have a /user/metron created on HDFS, so it's necessary to create and chown it appropriately. At that point, the script should be run as the Metron user. This should all be handled by the Ambari service. {code} su - hdfs hadoop fs -mkdir /user/metron hadoop fs -chown metron:hadoop /user/metron {code} > Ambari Mpack should be able to manage pcap topology > --- > > Key: METRON-552 > URL: https://issues.apache.org/jira/browse/METRON-552 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.2.1BETA >Reporter: Justin Leet > > Right now, the mpack installs the pcap RPM, but doesn't do any management of > it. This should be handled by Ambari. With the current boundaries of the > mpack (i.e. responsibility begins at kafka, not before), using pycapa to > produce data to Kafka wouldn't be managed by Ambari. > As a workaround (which will also have to happen in the Ambari service), pcap > topology can be run on an Ambari managed Metron instance by > 1) Updating /usr/metron/0.2.1BETA/config/pcap.properties by editing kafka.zk > manually. Ambari should manage these configs itself, once the service is up. > 2) Creating /apps/metron/pcap on HDFS, owning it to metron:hadoop, and > updating permissions to 775 on the folder. > At this point, /usr/metron/0.2.1BETA/bin/start_pcap_topology.sh should be > able to be run normally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-578) Missing error handling bolts for enrichment and threat intel
Justin Leet created METRON-578: -- Summary: Missing error handling bolts for enrichment and threat intel Key: METRON-578 URL: https://issues.apache.org/jira/browse/METRON-578 Project: Metron Issue Type: Improvement Affects Versions: 0.2.1BETA Reporter: Justin Leet Assignee: Justin Leet TL;DR - we need to add error handling to enrichments/threat intel Metron has parsers, enrichment + threat intel, and indexing topologies currently. Parsers and and enrichment have bolts that write to error topics in Kafka # indexing_error # parser_error # parser_invalid The GenericEnrichmentBolt handles errors gracefully by passing along failed enrichment tuples un-enriched and additionally emitting the tuple to an "error" stream, however there is currently no plumbing to handle the error stream. {code:java} } catch (Exception e) { LOG.error("[Metron] Unable to enrich message: " + rawMessage, e); JSONObject error = ErrorUtils.generateErrorMessage("Enrichment problem: " + rawMessage, e); if (key != null) { collector.emit(enrichmentType, new Values(key, enrichedMessage, subGroup)); } collector.emit("error", new Values(error)); } {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-552) Ambari Mpack should be able to manage pcap topology
[ https://issues.apache.org/jira/browse/METRON-552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15667358#comment-15667358 ] Justin Leet commented on METRON-552: As a addendum to this, before the topology can be started, it's also necessary to create the pcap kafka topic (doesn't need to be populated with data or anything, but it needs to actually exist to be read from in the topology). > Ambari Mpack should be able to manage pcap topology > --- > > Key: METRON-552 > URL: https://issues.apache.org/jira/browse/METRON-552 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.2.1BETA >Reporter: Justin Leet > > Right now, the mpack installs the pcap RPM, but doesn't do any management of > it. This should be handled by Ambari. With the current boundaries of the > mpack (i.e. responsibility begins at kafka, not before), using pycapa to > produce data to Kafka wouldn't be managed by Ambari. > As a workaround (which will also have to happen in the Ambari service), pcap > topology can be run on an Ambari managed Metron instance by > 1) Updating /usr/metron/0.2.1BETA/config/pcap.properties by editing kafka.zk > manually. Ambari should manage these configs itself, once the service is up. > 2) Creating /apps/metron/pcap on HDFS, owning it to metron:hadoop, and > updating permissions to 775 on the folder. > At this point, /usr/metron/0.2.1BETA/bin/start_pcap_topology.sh should be > able to be run normally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-563) Ambari Metron service uses incorrect port for installing Elasticsearch templates
[ https://issues.apache.org/jira/browse/METRON-563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-563: --- Affects Version/s: (was: 0.3.0) 0.2.1BETA Fix Version/s: 0.3.0 > Ambari Metron service uses incorrect port for installing Elasticsearch > templates > > > Key: METRON-563 > URL: https://issues.apache.org/jira/browse/METRON-563 > Project: Metron > Issue Type: Bug >Affects Versions: 0.2.1BETA >Reporter: Justin Leet >Assignee: Justin Leet > Fix For: 0.3.0 > > > The port entered by the user at install time should be the binary port > (typically 9300), which is used by the indexing bolt. The installation of > templates expects the HTTP port for Elasticsearch (typically 9200). > There are (probably) two options for fixes here: > 1) Split the ES url and have something like ES Server, ES HTTP port, ES > Binary port > 2) Have the template loading go through the binary port. I assume this can > be done, but I don't know enough about it to say for sure. > This can be worked around by setting the ES url to :9200, installing > the templates, and swapping back to :9300. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-564) Mpack shows UI error when changing configuration
[ https://issues.apache.org/jira/browse/METRON-564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-564: --- Affects Version/s: (was: 0.3.0) 0.2.1BETA Fix Version/s: 0.3.0 > Mpack shows UI error when changing configuration > > > Key: METRON-564 > URL: https://issues.apache.org/jira/browse/METRON-564 > Project: Metron > Issue Type: Bug >Affects Versions: 0.2.1BETA >Reporter: Justin Leet >Assignee: Justin Leet > Fix For: 0.3.0 > > > When confirming configuration changes, the Ambari UI shows an error that it > cannot validate configuration consistency. > An error in the logs refers to the service advisor > (getServiceConfigurationRecommendations), in particular a method that > actually appears twice in the file, which is probably the root cause. This > first instance of this method should be removed, and the error fixed. > {code} > Traceback (most recent call last): > File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 158, > in > main(sys.argv) > File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 115, > in main > result = stackAdvisor.validateConfigurations(services, hosts) > File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py", > line 536, in validateConfigurations > validationItems = self.getConfigurationsValidationItems(services, hosts) > File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py", > line 612, in getConfigurationsValidationItems > recommendations = self.recommendConfigurations(services, hosts) > File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py", > line 766, in recommendConfigurations > serviceAdvisor.getServiceConfigurationRecommendations(configurations, > clusterSummary, services, hosts) > File > "/var/lib/ambari-server/resources/common-services/METRON/0.3.0/service_advisor.py", > line 103, in getServiceConfigurationRecommendations > stormUIServerPort = > services["configurations"]["storm-site"]["properties"]["ui.port"] > KeyError: 'storm-site' > 10 Nov 2016 15:29:58,186 WARN [ambari-client-thread-28] > AbstractResourceProvider:90 - Error occurred during validation > org.apache.ambari.server.api.services.stackadvisor.StackAdvisorException: > Stack Advisor reported an error: KeyError: 'storm-site' > StdOut file: /var/run/ambari-server/stack-recommendations/18/stackadvisor.out > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-563) Ambari Metron service uses incorrect port for installing Elasticsearch templates
[ https://issues.apache.org/jira/browse/METRON-563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-563: --- Affects Version/s: (was: 0.2.1BETA) 0.3.0 > Ambari Metron service uses incorrect port for installing Elasticsearch > templates > > > Key: METRON-563 > URL: https://issues.apache.org/jira/browse/METRON-563 > Project: Metron > Issue Type: Bug >Affects Versions: 0.3.0 >Reporter: Justin Leet >Assignee: Justin Leet > > The port entered by the user at install time should be the binary port > (typically 9300), which is used by the indexing bolt. The installation of > templates expects the HTTP port for Elasticsearch (typically 9200). > There are (probably) two options for fixes here: > 1) Split the ES url and have something like ES Server, ES HTTP port, ES > Binary port > 2) Have the template loading go through the binary port. I assume this can > be done, but I don't know enough about it to say for sure. > This can be worked around by setting the ES url to :9200, installing > the templates, and swapping back to :9300. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-564) Mpack shows UI error when changing configuration
[ https://issues.apache.org/jira/browse/METRON-564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-564: --- Affects Version/s: (was: 0.2.1BETA) 0.3.0 > Mpack shows UI error when changing configuration > > > Key: METRON-564 > URL: https://issues.apache.org/jira/browse/METRON-564 > Project: Metron > Issue Type: Bug >Affects Versions: 0.3.0 >Reporter: Justin Leet >Assignee: Justin Leet > > When confirming configuration changes, the Ambari UI shows an error that it > cannot validate configuration consistency. > An error in the logs refers to the service advisor > (getServiceConfigurationRecommendations), in particular a method that > actually appears twice in the file, which is probably the root cause. This > first instance of this method should be removed, and the error fixed. > {code} > Traceback (most recent call last): > File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 158, > in > main(sys.argv) > File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 115, > in main > result = stackAdvisor.validateConfigurations(services, hosts) > File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py", > line 536, in validateConfigurations > validationItems = self.getConfigurationsValidationItems(services, hosts) > File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py", > line 612, in getConfigurationsValidationItems > recommendations = self.recommendConfigurations(services, hosts) > File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py", > line 766, in recommendConfigurations > serviceAdvisor.getServiceConfigurationRecommendations(configurations, > clusterSummary, services, hosts) > File > "/var/lib/ambari-server/resources/common-services/METRON/0.3.0/service_advisor.py", > line 103, in getServiceConfigurationRecommendations > stormUIServerPort = > services["configurations"]["storm-site"]["properties"]["ui.port"] > KeyError: 'storm-site' > 10 Nov 2016 15:29:58,186 WARN [ambari-client-thread-28] > AbstractResourceProvider:90 - Error occurred during validation > org.apache.ambari.server.api.services.stackadvisor.StackAdvisorException: > Stack Advisor reported an error: KeyError: 'storm-site' > StdOut file: /var/run/ambari-server/stack-recommendations/18/stackadvisor.out > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (METRON-563) Ambari Metron service uses incorrect port for installing Elasticsearch templates
[ https://issues.apache.org/jira/browse/METRON-563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-563: -- Assignee: Justin Leet > Ambari Metron service uses incorrect port for installing Elasticsearch > templates > > > Key: METRON-563 > URL: https://issues.apache.org/jira/browse/METRON-563 > Project: Metron > Issue Type: Bug >Affects Versions: 0.2.1BETA >Reporter: Justin Leet >Assignee: Justin Leet > > The port entered by the user at install time should be the binary port > (typically 9300), which is used by the indexing bolt. The installation of > templates expects the HTTP port for Elasticsearch (typically 9200). > There are (probably) two options for fixes here: > 1) Split the ES url and have something like ES Server, ES HTTP port, ES > Binary port > 2) Have the template loading go through the binary port. I assume this can > be done, but I don't know enough about it to say for sure. > This can be worked around by setting the ES url to :9200, installing > the templates, and swapping back to :9300. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-564) Mpack shows UI error when changing configuration
[ https://issues.apache.org/jira/browse/METRON-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15655158#comment-15655158 ] Justin Leet commented on METRON-564: This does not appear to prevent config changes from actually happening. > Mpack shows UI error when changing configuration > > > Key: METRON-564 > URL: https://issues.apache.org/jira/browse/METRON-564 > Project: Metron > Issue Type: Bug >Affects Versions: 0.2.1BETA >Reporter: Justin Leet >Assignee: Justin Leet > > When confirming configuration changes, the Ambari UI shows an error that it > cannot validate configuration consistency. > An error in the logs refers to the service advisor > (getServiceConfigurationRecommendations), in particular a method that > actually appears twice in the file, which is probably the root cause. This > first instance of this method should be removed, and the error fixed. > {code} > Traceback (most recent call last): > File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 158, > in > main(sys.argv) > File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 115, > in main > result = stackAdvisor.validateConfigurations(services, hosts) > File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py", > line 536, in validateConfigurations > validationItems = self.getConfigurationsValidationItems(services, hosts) > File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py", > line 612, in getConfigurationsValidationItems > recommendations = self.recommendConfigurations(services, hosts) > File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py", > line 766, in recommendConfigurations > serviceAdvisor.getServiceConfigurationRecommendations(configurations, > clusterSummary, services, hosts) > File > "/var/lib/ambari-server/resources/common-services/METRON/0.3.0/service_advisor.py", > line 103, in getServiceConfigurationRecommendations > stormUIServerPort = > services["configurations"]["storm-site"]["properties"]["ui.port"] > KeyError: 'storm-site' > 10 Nov 2016 15:29:58,186 WARN [ambari-client-thread-28] > AbstractResourceProvider:90 - Error occurred during validation > org.apache.ambari.server.api.services.stackadvisor.StackAdvisorException: > Stack Advisor reported an error: KeyError: 'storm-site' > StdOut file: /var/run/ambari-server/stack-recommendations/18/stackadvisor.out > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-564) Mpack shows UI error when changing configuration
[ https://issues.apache.org/jira/browse/METRON-564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-564: --- Description: When confirming configuration changes, the Ambari UI shows an error that it cannot validate configuration consistency. An error in the logs refers to the service advisor (getServiceConfigurationRecommendations), in particular a method that actually appears twice in the file, which is probably the root cause. This first instance of this method should be removed, and the error fixed. {code} Traceback (most recent call last): File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 158, in main(sys.argv) File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 115, in main result = stackAdvisor.validateConfigurations(services, hosts) File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py", line 536, in validateConfigurations validationItems = self.getConfigurationsValidationItems(services, hosts) File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py", line 612, in getConfigurationsValidationItems recommendations = self.recommendConfigurations(services, hosts) File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py", line 766, in recommendConfigurations serviceAdvisor.getServiceConfigurationRecommendations(configurations, clusterSummary, services, hosts) File "/var/lib/ambari-server/resources/common-services/METRON/0.3.0/service_advisor.py", line 103, in getServiceConfigurationRecommendations stormUIServerPort = services["configurations"]["storm-site"]["properties"]["ui.port"] KeyError: 'storm-site' 10 Nov 2016 15:29:58,186 WARN [ambari-client-thread-28] AbstractResourceProvider:90 - Error occurred during validation org.apache.ambari.server.api.services.stackadvisor.StackAdvisorException: Stack Advisor reported an error: KeyError: 'storm-site' StdOut file: /var/run/ambari-server/stack-recommendations/18/stackadvisor.out {code} was: When confirming configuration changes, the Ambari UI shows an error that it cannot validate configuration consistency. An error in the logs refers to the service advisor (getServiceConfigurationRecommendations), in particular a method that actually appears twice in the file, which is probably the root cause. This first instance of this method should be removed, and the error fixed. Traceback (most recent call last): File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 158, in main(sys.argv) File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 115, in main result = stackAdvisor.validateConfigurations(services, hosts) File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py", line 536, in validateConfigurations validationItems = self.getConfigurationsValidationItems(services, hosts) File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py", line 612, in getConfigurationsValidationItems recommendations = self.recommendConfigurations(services, hosts) File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py", line 766, in recommendConfigurations serviceAdvisor.getServiceConfigurationRecommendations(configurations, clusterSummary, services, hosts) File "/var/lib/ambari-server/resources/common-services/METRON/0.3.0/service_advisor.py", line 103, in getServiceConfigurationRecommendations stormUIServerPort = services["configurations"]["storm-site"]["properties"]["ui.port"] KeyError: 'storm-site' 10 Nov 2016 15:29:58,186 WARN [ambari-client-thread-28] AbstractResourceProvider:90 - Error occurred during validation org.apache.ambari.server.api.services.stackadvisor.StackAdvisorException: Stack Advisor reported an error: KeyError: 'storm-site' StdOut file: /var/run/ambari-server/stack-recommendations/18/stackadvisor.out > Mpack shows UI error when changing configuration > > > Key: METRON-564 > URL: https://issues.apache.org/jira/browse/METRON-564 > Project: Metron > Issue Type: Bug >Affects Versions: 0.2.1BETA >Reporter: Justin Leet >Assignee: Justin Leet > > When confirming configuration changes, the Ambari UI shows an error that it > cannot validate configuration consistency. > An error in the logs refers to the service advisor > (getServiceConfigurationRecommendations), in particular a method that > actually appears twice in the file, which is probably the root cause. This > first instance of this method should be removed, and the error fixed. > {code} > Traceback (most recent call last): > File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 158, > in > main(sys.argv) > File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 115,
[jira] [Created] (METRON-564) Mpack shows UI error when changing configuration
Justin Leet created METRON-564: -- Summary: Mpack shows UI error when changing configuration Key: METRON-564 URL: https://issues.apache.org/jira/browse/METRON-564 Project: Metron Issue Type: Bug Affects Versions: 0.2.1BETA Reporter: Justin Leet Assignee: Justin Leet When confirming configuration changes, the Ambari UI shows an error that it cannot validate configuration consistency. An error in the logs refers to the service advisor (getServiceConfigurationRecommendations), in particular a method that actually appears twice in the file, which is probably the root cause. This first instance of this method should be removed, and the error fixed. Traceback (most recent call last): File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 158, in main(sys.argv) File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 115, in main result = stackAdvisor.validateConfigurations(services, hosts) File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py", line 536, in validateConfigurations validationItems = self.getConfigurationsValidationItems(services, hosts) File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py", line 612, in getConfigurationsValidationItems recommendations = self.recommendConfigurations(services, hosts) File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py", line 766, in recommendConfigurations serviceAdvisor.getServiceConfigurationRecommendations(configurations, clusterSummary, services, hosts) File "/var/lib/ambari-server/resources/common-services/METRON/0.3.0/service_advisor.py", line 103, in getServiceConfigurationRecommendations stormUIServerPort = services["configurations"]["storm-site"]["properties"]["ui.port"] KeyError: 'storm-site' 10 Nov 2016 15:29:58,186 WARN [ambari-client-thread-28] AbstractResourceProvider:90 - Error occurred during validation org.apache.ambari.server.api.services.stackadvisor.StackAdvisorException: Stack Advisor reported an error: KeyError: 'storm-site' StdOut file: /var/run/ambari-server/stack-recommendations/18/stackadvisor.out -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-563) Ambari Metron service uses incorrect port for installing Elasticsearch templates
Justin Leet created METRON-563: -- Summary: Ambari Metron service uses incorrect port for installing Elasticsearch templates Key: METRON-563 URL: https://issues.apache.org/jira/browse/METRON-563 Project: Metron Issue Type: Bug Affects Versions: 0.2.1BETA Reporter: Justin Leet The port entered by the user at install time should be the binary port (typically 9300), which is used by the indexing bolt. The installation of templates expects the HTTP port for Elasticsearch (typically 9200). There are (probably) two options for fixes here: 1) Split the ES url and have something like ES Server, ES HTTP port, ES Binary port 2) Have the template loading go through the binary port. I assume this can be done, but I don't know enough about it to say for sure. This can be worked around by setting the ES url to :9200, installing the templates, and swapping back to :9300. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-525) Unable to start PCAP topology
[ https://issues.apache.org/jira/browse/METRON-525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15644318#comment-15644318 ] Justin Leet commented on METRON-525: There's a couple things going on related to this: First, the PCAP topology has a different startup script (start_pcap_topology.sh). This is the core problem and [~james.sirota] updated some docs to make this more clear and accessible at: https://cwiki.apache.org/confluence/display/METRON/Launching+Metron+Topologies https://cwiki.apache.org/confluence/display/METRON/Bulk+Loading+Enrichments+and+Pruning+Data However, this doesn't apply to the Ambari Mpack, because until recently the PCAP RPM wasn't even installed and there is no service for managing it. I took a bit of time to spin it up and provide a manual workaround. It couple manual steps right now (assuming pycapa is up and running). Both need to be done before running start_pcap_topology.sh 1) /usr/metron/0.2.1BETA/config/pcap.properties needs to have kafka.zk set appropriately. 2) /apps/metron/pcap on HDFS needs to be created, chowned to metron:hadoop, and chmoded to 775 At this point, I was able to get both the topology and the pcap_query.sh running and producing data. To fix these as a service, 1) is just exposing the pcap.properties in Ambari and 2) is just creating the directory with appropriate perms (we do exactly the same thing for /apps/metron/enrichment), plus the surrounding build out of the Ambari definition. > Unable to start PCAP topology > - > > Key: METRON-525 > URL: https://issues.apache.org/jira/browse/METRON-525 > Project: Metron > Issue Type: Bug >Affects Versions: 0.2.2BETA >Reporter: Neha Sinha > > The following error is seen while starting PCAP topology :- > = > [root@metron-s-10 ~]# /usr/metron/0.2.1BETA/bin/start_parser_topology.sh -k > metron-s-10.openstacklocal:6667 -z metron-s-10.openstacklocal:2181 -s pcap > Running: /usr/jdk64/jdk1.8.0_77/bin/java -client -Ddaemon.name= > -Dstorm.options= -Dstorm.home=/grid/0/hdp/2.4.3.0-227/storm > -Dstorm.log.dir=/grid/0/log/storm > -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > -Dstorm.conf.file= -cp > /grid/0/hdp/2.4.3.0-227/storm/lib/log4j-api-2.1.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/cheshire-5.3.1.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/compojure-1.1.3.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/tools.logging-0.2.3.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/core.incubator-0.1.0.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/jline-0.9.94.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/ring-core-1.1.5.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/java.classpath-0.2.2.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/slf4j-api-1.7.7.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/zookeeper.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/disruptor-2.10.1.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/log4j-core-2.1.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/jackson-core-2.3.1.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/tigris-0.1.1.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/reflectasm-1.07-shaded.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/clj-stacktrace-0.2.7.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/commons-codec-1.6.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/clojure-1.6.0.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/ring-jetty-adapter-1.3.0.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/ring-json-0.3.1.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/servlet-api-2.5.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/tools.namespace-0.2.4.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/clj-time-0.8.0.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/ring-devel-1.3.0.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/hadoop-auth-2.7.1.2.4.3.0-227.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/jackson-dataformat-smile-2.3.1.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/hiccup-0.3.6.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/asm-4.0.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/storm-core-0.10.0.2.4.3.0-227.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/clout-1.0.1.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/ns-tracker-0.2.2.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/minlog-1.2.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/oncrpc-1.0.7.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/log4j-slf4j-impl-2.1.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/gmetric4j-1.0.7.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/ring-servlet-1.3.0.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/kryo-2.21.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/metron/0.2.1BETA/lib/metron-parsers-0.2.1BETA-uber.jar:/usr/hdp/current/storm-supervisor/conf:/grid/0/hdp/2.4.3.0-227/storm/bin > -Dstorm.jar=/usr/metron/0.2.1BETA/lib/metron-parsers-0.2.1BETA-uber.jar > org.apache.metron.parsers.topology.ParserTopologyCLI -k > metron-s-10.openstacklocal:6667 -z metron-s-10.openstacklocal:2181 -s pcap > 05:59:01.065 [main] INFO o.a.c.f.i.Cur
[jira] [Created] (METRON-553) Ambari Metron config should have "Test Connection" button for MySql and Elasticsearch
Justin Leet created METRON-553: -- Summary: Ambari Metron config should have "Test Connection" button for MySql and Elasticsearch Key: METRON-553 URL: https://issues.apache.org/jira/browse/METRON-553 Project: Metron Issue Type: Improvement Affects Versions: 0.2.1BETA Reporter: Justin Leet Right now, configs are entered while installing (or altering) Metron in Ambari. It would be nice to have a button to test connection for MySql and Elasticsearch so that you know the configs are correct before installing and spinning up everything. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-552) Ambari Mpack should be able to manage pcap topology
Justin Leet created METRON-552: -- Summary: Ambari Mpack should be able to manage pcap topology Key: METRON-552 URL: https://issues.apache.org/jira/browse/METRON-552 Project: Metron Issue Type: Improvement Affects Versions: 0.2.1BETA Reporter: Justin Leet Right now, the mpack installs the pcap RPM, but doesn't do any management of it. This should be handled by Ambari. With the current boundaries of the mpack (i.e. responsibility begins at kafka, not before), using pycapa to produce data to Kafka wouldn't be managed by Ambari. As a workaround (which will also have to happen in the Ambari service), pcap topology can be run on an Ambari managed Metron instance by 1) Updating /usr/metron/0.2.1BETA/config/pcap.properties by editing kafka.zk manually. Ambari should manage these configs itself, once the service is up. 2) Creating /apps/metron/pcap on HDFS, owning it to metron:hadoop, and updating permissions to 775 on the folder. At this point, /usr/metron/0.2.1BETA/bin/start_pcap_topology.sh should be able to be run normally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-520) /apps/metron/enrichment directory does not get created for Metron cluster deployed via Ambari
[ https://issues.apache.org/jira/browse/METRON-520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-520: --- Fix Version/s: 0.2.2BETA > /apps/metron/enrichment directory does not get created for Metron cluster > deployed via Ambari > - > > Key: METRON-520 > URL: https://issues.apache.org/jira/browse/METRON-520 > Project: Metron > Issue Type: Bug >Affects Versions: 0.2.1BETA >Reporter: Neha Sinha >Assignee: Justin Leet > Fix For: 0.2.2BETA > > > 1.Deploy Metron cluster via Ambari > 2. Replay Bro logs to generate bro elasticsearch indices > 3. The bro enriched and indexed data should be written to the HDFS at :- > /apps/metron/enrichment > The indexed data gets written to "/apps/metron/enrichment" for metron setups > that get deployed via Ansible, however, this path does not get created for > clusters deployed via Ambari. > Output for "hdfs dfs -ls" command for clusters deployed via Ansible > [hdfs@metron-ansible-3 ~]$ hdfs dfs -ls /apps/metron > Found 2 items > drwxrwxr-x - storm hadoop 0 2016-10-24 11:41 > /apps/metron/enrichment > drwxrwxr-x - hdfs hadoop 0 2016-10-24 11:03 /apps/metron/patterns > Output for "hdfs dfs -ls" command for clusters deployed via Ambari > [hdfs@metron-s-10 ~]$ hdfs dfs -ls /apps/metron > Found 7 items > -rwxr-xr-x 3 hdfs hdfs 13427 2016-10-25 10:02 /apps/metron/asa > -rwxr-xr-x 3 hdfs hdfs 5203 2016-10-25 10:02 /apps/metron/common > -rwxr-xr-x 3 hdfs hdfs524 2016-10-25 10:02 /apps/metron/fireeye > -rwxr-xr-x 3 hdfs hdfs 2552 2016-10-25 10:02 /apps/metron/sourcefire > -rwxr-xr-x 3 hdfs hdfs180 2016-10-25 10:02 /apps/metron/squid > -rwxr-xr-x 3 hdfs hdfs 2221 2016-10-25 10:02 /apps/metron/websphere > -rwxr-xr-x 3 hdfs hdfs879 2016-10-25 10:02 /apps/metron/yaf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (METRON-520) /apps/metron/enrichment directory does not get created for Metron cluster deployed via Ambari
[ https://issues.apache.org/jira/browse/METRON-520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-520: -- Assignee: Justin Leet > /apps/metron/enrichment directory does not get created for Metron cluster > deployed via Ambari > - > > Key: METRON-520 > URL: https://issues.apache.org/jira/browse/METRON-520 > Project: Metron > Issue Type: Bug >Affects Versions: 0.2.1BETA >Reporter: Neha Sinha >Assignee: Justin Leet > > 1.Deploy Metron cluster via Ambari > 2. Replay Bro logs to generate bro elasticsearch indices > 3. The bro enriched and indexed data should be written to the HDFS at :- > /apps/metron/enrichment > The indexed data gets written to "/apps/metron/enrichment" for metron setups > that get deployed via Ansible, however, this path does not get created for > clusters deployed via Ambari. > Output for "hdfs dfs -ls" command for clusters deployed via Ansible > [hdfs@metron-ansible-3 ~]$ hdfs dfs -ls /apps/metron > Found 2 items > drwxrwxr-x - storm hadoop 0 2016-10-24 11:41 > /apps/metron/enrichment > drwxrwxr-x - hdfs hadoop 0 2016-10-24 11:03 /apps/metron/patterns > Output for "hdfs dfs -ls" command for clusters deployed via Ambari > [hdfs@metron-s-10 ~]$ hdfs dfs -ls /apps/metron > Found 7 items > -rwxr-xr-x 3 hdfs hdfs 13427 2016-10-25 10:02 /apps/metron/asa > -rwxr-xr-x 3 hdfs hdfs 5203 2016-10-25 10:02 /apps/metron/common > -rwxr-xr-x 3 hdfs hdfs524 2016-10-25 10:02 /apps/metron/fireeye > -rwxr-xr-x 3 hdfs hdfs 2552 2016-10-25 10:02 /apps/metron/sourcefire > -rwxr-xr-x 3 hdfs hdfs180 2016-10-25 10:02 /apps/metron/squid > -rwxr-xr-x 3 hdfs hdfs 2221 2016-10-25 10:02 /apps/metron/websphere > -rwxr-xr-x 3 hdfs hdfs879 2016-10-25 10:02 /apps/metron/yaf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-527) Connection pool shut down seen when running threatintel_taxii_load.sh script
Justin Leet created METRON-527: -- Summary: Connection pool shut down seen when running threatintel_taxii_load.sh script Key: METRON-527 URL: https://issues.apache.org/jira/browse/METRON-527 Project: Metron Issue Type: Bug Reporter: Justin Leet Assignee: Justin Leet Discovered by [~anandsubbu]: I have setup opentaxii server on a 12-node metron cluster. I was able to scan and pull records into the taxi server. {code} [root@metron-test2-8 ~]# service opentaxii status Checking opentaxii... Running guest.phishtank_com5891 guest.Abuse_ch 49 guest.CyberCrime_Tracker 0 guest.EmergingThreats_rules0 guest.Lehigh_edu 2025 guest.MalwareDomainList_Hostlist 50 guest.blutmagie_de_torExits14897 guest.dataForLast_7daysOnly23210 guest.dshield_BlockList0 {code} When I ran the threatintel_taxii_load.sh script, it is able to push some of the records into hbase 'threatintel' table, but fails with a "Connection pool shut down" error shortly after. This was observed for all the subscriptions that I attempted. Here is the command that was used: {code} [root@metron-test2-8 ~]# /usr/metron/0.2.0BETA/bin//threatintel_taxii_load.sh -b "2015-01-01 00:00:00" -c ~/connection_config.json -e ~/extractor.json -p 30 {code} The error message is pasted below. {code} 16/08/30 09:24:14 ERROR taxii.TaxiiHandler: Connection pool shut down java.lang.IllegalStateException: Connection pool shut down at org.apache.metron.httpcore.dataload.util.Asserts.check(Asserts.java:34) at org.apache.metron.httpcore.dataload.pool.AbstractConnPool.lease(AbstractConnPool.java:169) at org.apache.metron.httpcore.dataload.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:217) at org.apache.metron.httpcore.dataload.impl.execchain.MainClientExec.execute(MainClientExec.java:158) at org.apache.metron.httpcore.dataload.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195) at org.apache.metron.httpcore.dataload.impl.execchain.RetryExec.execute(RetryExec.java:85) at org.apache.metron.httpcore.dataload.impl.execchain.RedirectExec.execute(RedirectExec.java:108) at org.apache.metron.httpcore.dataload.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186) at org.apache.metron.httpcore.dataload.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.mitre.taxii.client.HttpClient.callTaxiiService(HttpClient.java:297) at org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.call(TaxiiHandler.java:336) at org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.call(TaxiiHandler.java:242) at org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.run(TaxiiHandler.java:171) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) Exception in thread "Timer-0" java.lang.RuntimeException: Unable to make request at org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.run(TaxiiHandler.java:214) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) Caused by: java.lang.IllegalStateException: Connection pool shut down at org.apache.metron.httpcore.dataload.util.Asserts.check(Asserts.java:34) at org.apache.metron.httpcore.dataload.pool.AbstractConnPool.lease(AbstractConnPool.java:169) at org.apache.metron.httpcore.dataload.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:217) at org.apache.metron.httpcore.dataload.impl.execchain.MainClientExec.execute(MainClientExec.java:158) at org.apache.metron.httpcore.dataload.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195) at org.apache.metron.httpcore.dataload.impl.execchain.RetryExec.execute(RetryExec.java:85) at org.apache.metron.httpcore.dataload.impl.execchain.RedirectExec.execute(RedirectExec.java:108) at org.apache.metron.httpcore.dataload.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186) at org.apache.metron.httpcore.dataload.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.mitre.taxii.client.HttpClient.callTaxiiService(HttpClient.java:297) at org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.call(TaxiiHandler.java:336) at org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.call(TaxiiHandler.java:242) at org.apache.metron.dataloads.nonbulk.taxii.TaxiiHandler.run(TaxiiHandler
[jira] [Assigned] (METRON-495) Upgrade Storm to 1.0.x
[ https://issues.apache.org/jira/browse/METRON-495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-495: -- Assignee: Justin Leet > Upgrade Storm to 1.0.x > -- > > Key: METRON-495 > URL: https://issues.apache.org/jira/browse/METRON-495 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Assignee: Justin Leet > > As listed at https://storm.apache.org/2016/04/12/storm100-released.html, > there's a variety of improvements with Storm 1.0. The obvious and likely > most important improvement is to performance, but a variety of other > improvements are noted on that link. > There are several changes that have to occur in order to make this upgrade. > As noted in http://storm.apache.org/releases/current/index.html, Storm's code > packages moved from backtype.storm to org.apache.storm, meaning all > topologies have to be recompiled if that change. There is a runtime > converter to run things in place with backtype.storm, but this doesn't appear > to be enough for our case because a couple interfaces change from byte[] to > ByteBuffer (somewhat, but not entirely, related to > https://issues.apache.org/jira/browse/STORM-1449). Even without this issue, > the long term solution is to use the new package naming. > In addition, our dev instances right now spin up an HDP 2.4 instance, which > matches our current version of Storm. HDP 2.5 uses Storm 1.0.1, so to migrate > to Storm 1.0.x, I'd prefer to match that version, rather than going to 1.0.2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-495) Upgrade Storm to 1.0.x
Justin Leet created METRON-495: -- Summary: Upgrade Storm to 1.0.x Key: METRON-495 URL: https://issues.apache.org/jira/browse/METRON-495 Project: Metron Issue Type: Improvement Reporter: Justin Leet As listed at https://storm.apache.org/2016/04/12/storm100-released.html, there's a variety of improvements with Storm 1.0. The obvious and likely most important improvement is to performance, but a variety of other improvements are noted on that link. There are several changes that have to occur in order to make this upgrade. As noted in http://storm.apache.org/releases/current/index.html, Storm's code packages moved from backtype.storm to org.apache.storm, meaning all topologies have to be recompiled if that change. There is a runtime converter to run things in place with backtype.storm, but this doesn't appear to be enough for our case because a couple interfaces change from byte[] to ByteBuffer (somewhat, but not entirely, related to https://issues.apache.org/jira/browse/STORM-1449). Even without this issue, the long term solution is to use the new package naming. In addition, our dev instances right now spin up an HDP 2.4 instance, which matches our current version of Storm. HDP 2.5 uses Storm 1.0.1, so to migrate to Storm 1.0.x, I'd prefer to match that version, rather than going to 1.0.2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (METRON-425) Stellar transformation fails to handle special characters
[ https://issues.apache.org/jira/browse/METRON-425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-425: -- Assignee: Justin Leet > Stellar transformation fails to handle special characters > - > > Key: METRON-425 > URL: https://issues.apache.org/jira/browse/METRON-425 > Project: Metron > Issue Type: Bug >Reporter: Neha Sinha >Assignee: Justin Leet > > I updated the snort parser file to have the following stellar transformation > :- > PARSER Config: snort > { > "parserClassName":"org.apache.metron.parsers.snort.BasicSnortParser", > "sensorTopic":"snort", > "parserConfig": {}, > "fieldTransformations" : [ > { > "transformation" : "STELLAR" > ,"output" : [ "is_alert","newStellarField","isAlert"] > ,"config" : > { "is_alert" : "false", > "isAlert" : "false", > "newStellarField" : "<>" } > } > ] > } > I get the following exception/error for the snort logs :- > 2016-09-13 11:30:32.765 o.a.m.p.BasicParser [TRACE] [Metron] Message conforms > to schema: {"msg":"\"'snort test > alert'\"","sig_rev":"0","ip_dst_port":"80","ethsrc":"00:00:00:00:00:00","tcpseq":"0x5869E532","dgmlen":"40","icmpid":"","tcplen":"","tcpwindow":"0xFA02","icmpseq":"","tcpack":"0x3E05E218","protocol":"TCP","ip_dst_addr":"72.34.49.86","original_string":"09\/13-11:30:25.703857 > ,1,999158,0,\"'snort test > alert'\",TCP,192.168.138.158,49204,72.34.49.86,80,00:00:00:00:00:00,00:00:00:00:00:00,0x3C,***A,0x5869E532,0x3E05E218,,0xFA02,128,0,2508,40,40960","icmpcode":"","tos":"0","id":"2508","ip_src_addr":"192.168.138.158","timestamp":1473766928857,"ethdst":"00:00:00:00:00:00","is_alert":"true","ttl":"128","ethlen":"0x3C","iplen":"40960","icmptype":"","ip_src_port":"49204","tcpflags":"***A","sig_id":"999158","sig_generator":"1"} > 2016-09-13 11:30:32.766 b.s.d.executor [ERROR] > org.apache.metron.common.dsl.ParseException: Syntax error @ 1:0 no viable > alternative at input '<' > at > org.apache.metron.common.dsl.ErrorListener.syntaxError(ErrorListener.java:34) > ~[stormjar.jar:?] > at > org.antlr.v4.runtime.ProxyErrorListener.syntaxError(ProxyErrorListener.java:65) > ~[stormjar.jar:?] > at org.antlr.v4.runtime.Parser.notifyErrorListeners(Parser.java:558) > ~[stormjar.jar:?] > at > org.antlr.v4.runtime.DefaultErrorStrategy.reportNoViableAlternative(DefaultErrorStrategy.java:310) > ~[stormjar.jar:?] > at > org.antlr.v4.runtime.DefaultErrorStrategy.reportError(DefaultErrorStrategy.java:147) > ~[stormjar.jar:?] > at > org.apache.metron.common.stellar.generated.StellarParser.transformation_expr(StellarParser.java:300) > ~[stormjar.jar:?] > at > org.apache.metron.common.stellar.generated.StellarParser.transformation(StellarParser.java:146) > ~[stormjar.jar:?] > at > org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:92) > ~[stormjar.jar:?] > at > org.apache.metron.common.field.transformation.StellarTransformation.map(StellarTransformation.java:46) > ~[stormjar.jar:?] > at > org.apache.metron.common.configuration.FieldTransformer.transform(FieldTransformer.java:111) > ~[stormjar.jar:?] > at > org.apache.metron.common.configuration.FieldTransformer.transformAndUpdate(FieldTransformer.java:123) > ~[stormjar.jar:?] > at > org.apache.metron.parsers.bolt.ParserBolt.execute(ParserBolt.java:125) > [stormjar.jar:?] > at > backtype.storm.daemon.executor$fn__5492$tuple_action_fn__5494.invoke(executor.clj:684) > [storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258] > at > backtype.storm.daemon.executor$mk_task_receiver$fn__5415.invoke(executor.clj:431) > [storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258] > at > backtype.storm.disruptor$clojure_handler$reify__4991.onEvent(disruptor.clj:58) > [storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258] > at > backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125) > [storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258] > at > backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) > [storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258] > at > backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) > [storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258] > at > backtype.storm.daemon.executor$fn__5492$fn__5505$fn__5556.invoke(executor.clj:813) > [storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258] > at backtype.storm.util$async_loop$fn__644.invoke(util.clj:479) > [storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258] > at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?] > at java.lang.Thread.run(Thread.java:745) [?:1.8.0_60] > Caused by: org.antlr.v4.runtime.NoViableAltException >
[jira] [Created] (METRON-482) Add logging to GrokParser to indicate supplied TimeZone
Justin Leet created METRON-482: -- Summary: Add logging to GrokParser to indicate supplied TimeZone Key: METRON-482 URL: https://issues.apache.org/jira/browse/METRON-482 Project: Metron Issue Type: Bug Reporter: Justin Leet Assignee: Justin Leet TimeZone can be passed to the GrokParser and is used in timestamp conversions. However, it is not logged when this is used, making it difficult to tell why the conversion process resulted in the given values. Adding logging will make this easier to understand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (METRON-326) Error Handling in ElasticsearchWriter
[ https://issues.apache.org/jira/browse/METRON-326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-326: -- Assignee: Justin Leet > Error Handling in ElasticsearchWriter > - > > Key: METRON-326 > URL: https://issues.apache.org/jira/browse/METRON-326 > Project: Metron > Issue Type: Bug >Reporter: Ajay Yadav >Assignee: Justin Leet > > In Elasticsearch writer we raise a exception if BulkResponse object has > failures and that results in failing the whole batch even if some objects > failed in it. This has spiral effect specially when there is continuous > stream of bad messages and errorStream is tied to indexingBolt. > If possible we should iterate through items in BulkResponse object and send > only failed messages to errorStream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-410) mysql_server's MySQL install causes mutually assured destruction when installed on the same machine as the Ambari Hive MySQL
[ https://issues.apache.org/jira/browse/METRON-410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15523421#comment-15523421 ] Justin Leet commented on METRON-410: Upgraded this to its own issue. METRON-427 was merged in to make it available to everyone as an MVP > mysql_server's MySQL install causes mutually assured destruction when > installed on the same machine as the Ambari Hive MySQL > > > Key: METRON-410 > URL: https://issues.apache.org/jira/browse/METRON-410 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Priority: Minor > Original Estimate: 48h > Remaining Estimate: 48h > > Metron's mysql_server MySQL install causes mutually assured destruction when > installed on the same machine as the Ambari Hive MySQL. Here is the startup > error you get afterwards. > https://gist.github.com/JonZeolla/2ed4161c141ba32a3e8a0d6ce9718779 > In the short term, maybe add a check so they won't live on the same box. > Long term, allow them to coexist? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-410) mysql_server's MySQL install causes mutually assured destruction when installed on the same machine as the Ambari Hive MySQL
[ https://issues.apache.org/jira/browse/METRON-410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-410: --- Issue Type: Improvement (was: Sub-task) Parent: (was: METRON-427) > mysql_server's MySQL install causes mutually assured destruction when > installed on the same machine as the Ambari Hive MySQL > > > Key: METRON-410 > URL: https://issues.apache.org/jira/browse/METRON-410 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Priority: Minor > Original Estimate: 48h > Remaining Estimate: 48h > > Metron's mysql_server MySQL install causes mutually assured destruction when > installed on the same machine as the Ambari Hive MySQL. Here is the startup > error you get afterwards. > https://gist.github.com/JonZeolla/2ed4161c141ba32a3e8a0d6ce9718779 > In the short term, maybe add a check so they won't live on the same box. > Long term, allow them to coexist? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (METRON-427) Create Ambari Management Pack for Metron Installation
[ https://issues.apache.org/jira/browse/METRON-427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-427: -- Assignee: Justin Leet > Create Ambari Management Pack for Metron Installation > - > > Key: METRON-427 > URL: https://issues.apache.org/jira/browse/METRON-427 > Project: Metron > Issue Type: New Feature >Reporter: Justin Leet >Assignee: Justin Leet > > Right now, Metron depends on Ambari blueprints, in the Ansible scripts, to > deploy onto a cluster. > To ease installation, a full Ambari Management Pack > (https://cwiki.apache.org/confluence/display/AMBARI/Management+Packs) can be > used to lay down topologies, etc. > The current expectation is that the boundaries of this would cover from Kafka > to the indexes. The dev list has additional discussion about whether or not > sensor install and what should exist beyond minimum viable product. > Additional follow-on tickets would be created based on the results of both > that discussion and any discussion on this ticket. > A minimum viable product for this would cover > * Laying down topologies (parsers, enrichment, and indexing) > * Starting and stopping topologies > * Setting up configuration > * Setting up bits (Using RPMs currently built locally) > * Set up infra dependencies (MySql and Elasticsearch) > At this point, the MVP could take data from Kafka, run it through the > topologies, and make it available in the output Elasticsearch Indexes. > A good deal of the ground work for this is already completed (several Service > Definitions, along with the RPM creation). Relevant Jira's are: > * METRON-383 (Create Ambari Service Definition for Metron Parsers) > * METRON-385 (Create Ambari Service Definition for Indexing) > * METRON-386 (Create Ambari Service Definition for Elasticsearch) > * METRON-357 (Create Ambari Service Definition for Kibana) > * METRON-214 (Build RPM Packages for Deployment) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-427) Create Ambari Management Pack for Metron Installation
Justin Leet created METRON-427: -- Summary: Create Ambari Management Pack for Metron Installation Key: METRON-427 URL: https://issues.apache.org/jira/browse/METRON-427 Project: Metron Issue Type: New Feature Reporter: Justin Leet Right now, Metron depends on Ambari blueprints, in the Ansible scripts, to deploy onto a cluster. To ease installation, a full Ambari Management Pack (https://cwiki.apache.org/confluence/display/AMBARI/Management+Packs) can be used to lay down topologies, etc. The current expectation is that the boundaries of this would cover from Kafka to the indexes. The dev list has additional discussion about whether or not sensor install and what should exist beyond minimum viable product. Additional follow-on tickets would be created based on the results of both that discussion and any discussion on this ticket. A minimum viable product for this would cover * Laying down topologies (parsers, enrichment, and indexing) * Starting and stopping topologies * Setting up configuration * Setting up bits (Using RPMs currently built locally) * Set up infra dependencies (MySql and Elasticsearch) At this point, the MVP could take data from Kafka, run it through the topologies, and make it available in the output Elasticsearch Indexes. A good deal of the ground work for this is already completed (several Service Definitions, along with the RPM creation). Relevant Jira's are: * METRON-383 (Create Ambari Service Definition for Metron Parsers) * METRON-385 (Create Ambari Service Definition for Indexing) * METRON-386 (Create Ambari Service Definition for Elasticsearch) * METRON-357 (Create Ambari Service Definition for Kibana) * METRON-214 (Build RPM Packages for Deployment) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-405) Create Ambari Service Definition for Enrichment
Justin Leet created METRON-405: -- Summary: Create Ambari Service Definition for Enrichment Key: METRON-405 URL: https://issues.apache.org/jira/browse/METRON-405 Project: Metron Issue Type: New Feature Reporter: Justin Leet Assignee: Justin Leet To pull everything into an easier install through Ambari, create a service definition to automatically install and handle the enrichment topology appropriately. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-386) Create Ambari Service Definition for Elasticsearch
Justin Leet created METRON-386: -- Summary: Create Ambari Service Definition for Elasticsearch Key: METRON-386 URL: https://issues.apache.org/jira/browse/METRON-386 Project: Metron Issue Type: New Feature Reporter: Justin Leet Assignee: Justin Leet Spinning up Elasticsearch through Ambari would make deployment and management a lot easier and comprehensive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-385) Create Ambari Service Definition for Indexing
Justin Leet created METRON-385: -- Summary: Create Ambari Service Definition for Indexing Key: METRON-385 URL: https://issues.apache.org/jira/browse/METRON-385 Project: Metron Issue Type: New Feature Reporter: Justin Leet Assignee: Justin Leet To pull everything into an easier install through Ambari, create a service definition to automatically install and handle the indexing topology appropriately. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-334) Travis CI cache Maven dependencies
Justin Leet created METRON-334: -- Summary: Travis CI cache Maven dependencies Key: METRON-334 URL: https://issues.apache.org/jira/browse/METRON-334 Project: Metron Issue Type: Improvement Reporter: Justin Leet Assignee: Justin Leet Priority: Minor Per https://docs.travis-ci.com/user/caching/ Dependencies can be cached to avoid having them be redownloaded every build. Specifically we want to cache $HOME/.m2 per the docs This should speed up Travis builds a little bit, although the majority of the time will still be on testing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (METRON-319) Automate RPM Builds
[ https://issues.apache.org/jira/browse/METRON-319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-319: -- Assignee: Justin Leet > Automate RPM Builds > --- > > Key: METRON-319 > URL: https://issues.apache.org/jira/browse/METRON-319 > Project: Metron > Issue Type: New Feature >Reporter: David M. Lyle >Assignee: Justin Leet > Fix For: 0.2.1BETA > > > Create automation that will build the RPMs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (METRON-305) Build Solr Deployment RPM
[ https://issues.apache.org/jira/browse/METRON-305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-305: -- Assignee: Justin Leet > Build Solr Deployment RPM > - > > Key: METRON-305 > URL: https://issues.apache.org/jira/browse/METRON-305 > Project: Metron > Issue Type: Sub-task >Reporter: David M. Lyle >Assignee: Justin Leet > Fix For: 0.2.1BETA > > > Relocate metron_streaming Solr file copy and initial setup to RPM -- This message was sent by Atlassian JIRA (v6.3.4#6332)