[jira] [Created] (METRON-861) Allow JVM args to be passed to CLI utilities
Casey Stella created METRON-861: --- Summary: Allow JVM args to be passed to CLI utilities Key: METRON-861 URL: https://issues.apache.org/jira/browse/METRON-861 Project: Metron Issue Type: Improvement Reporter: Casey Stella Assignee: Casey Stella This is motivated by the fact that if one sets the acl's on the znodes that metron creates (e.g. /metron/topology/global) to read/write by metron (from the zkcli `setAcl /metron/topology/global sasl:metron:crwda`), then permissions prohibit the CLI tools from functioning because the JAAS config is not loaded. This JIRA allows users to pass java properties to the CLI tools. While important for Kerberos, it's useful in general if one needs to adjust the heap required, etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (METRON-842) Add dynamic templates for risk score fields
[ https://issues.apache.org/jira/browse/METRON-842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella reassigned METRON-842: --- Assignee: Casey Stella > Add dynamic templates for risk score fields > --- > > Key: METRON-842 > URL: https://issues.apache.org/jira/browse/METRON-842 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella > > When risk scores come in as null initially, ES will mistake the field for a > string which can cause errors once the scores become numbers in follow-on > messages. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-842) Add dynamic templates for risk score fields
Casey Stella created METRON-842: --- Summary: Add dynamic templates for risk score fields Key: METRON-842 URL: https://issues.apache.org/jira/browse/METRON-842 Project: Metron Issue Type: Improvement Reporter: Casey Stella When risk scores come in as null initially, ES will mistake the field for a string which can cause errors once the scores become numbers in follow-on messages. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-841) Failed to connect node to cluster due to: java.lang.NullPointerException
[
https://issues.apache.org/jira/browse/METRON-841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15963379#comment-15963379
]
Casey Stella commented on METRON-841:
-
Looks like a nifi issue.
> Failed to connect node to cluster due to: java.lang.NullPointerException
> -
>
> Key: METRON-841
> URL: https://issues.apache.org/jira/browse/METRON-841
> Project: Metron
> Issue Type: Bug
> Environment: SUSE 11 SP3, NIFI 1.1.0.2.1.2.0-10, Java jdk1.8.0_60, 2
> node clustered Nifi
>Reporter: Joseph Niemiec
>
> We had a server fill its root directory, we stopped NiFi, moved the
> repositories to mounts with space, updated the configs and now we have a
> single node that will not join the cluster. There are no major errors but we
> do run into some WARNS talking about an NPE before Jetty explodes.
> 2017-04-10 14:55:57,293 ERROR [main] o.a.nifi.controller.StandardFlowService
> Failed to load flow from cluster due to:
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to
> cluster due to: java.lang.NullPointerException
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to
> cluster due to: java.lang.NullPointerException
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:901)
> ~[nifi-framework-core-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]
> at
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
> ~[nifi-framework-core-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:831)
> [nifi-jetty-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]
> at org.apache.nifi.NiFi.(NiFi.java:156)
> [nifi-runtime-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]
> at org.apache.nifi.NiFi.main(NiFi.java:262)
> [nifi-runtime-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]
> Caused by: java.lang.NullPointerException: null
> at
> org.apache.nifi.controller.repository.SchemaRepositoryRecordSerde.deserializeRecord(SchemaRepositoryRecordSerde.java:119)
> ~[nifi-framework-core-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]
> at
> org.apache.nifi.controller.repository.SchemaRepositoryRecordSerde.deserializeEdit(SchemaRepositoryRecordSerde.java:109)
> ~[nifi-framework-core-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]
> at
> org.apache.nifi.controller.repository.SchemaRepositoryRecordSerde.deserializeEdit(SchemaRepositoryRecordSerde.java:46)
> ~[nifi-framework-core-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]
> at
> org.wali.MinimalLockingWriteAheadLog$Partition.recoverNextTransaction(MinimalLockingWriteAheadLog.java:1072)
> ~[nifi-write-ahead-log-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]
> at
> org.wali.MinimalLockingWriteAheadLog.recoverFromEdits(MinimalLockingWriteAheadLog.java:459)
> ~[nifi-write-ahead-log-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]
> at
> org.wali.MinimalLockingWriteAheadLog.recoverRecords(MinimalLockingWriteAheadLog.java:301)
> ~[nifi-write-ahead-log-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]
> at
> org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.loadFlowFiles(WriteAheadFlowFileRepository.java:346)
> ~[nifi-framework-core-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]
> at
> org.apache.nifi.controller.FlowController.initializeFlow(FlowController.java:699)
> ~[nifi-framework-core-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]
> at
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:701)
> ~[nifi-framework-core-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:872)
> ~[nifi-framework-core-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]
> ... 4 common frames omitted
> 2017-04-10 14:55:57,293 INFO [main] o.a.n.c.c.node.NodeClusterCoordinator
> hpce2r01n11.abc.com:9090 requested disconnection from cluster due to
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to
> cluster due to: java.lang.NullPointerException
> 2017-04-10 14:55:57,293 INFO [main] o.a.n.c.c.node.NodeClusterCoordinator
> Status of hpce2r01n11.abc.com:9090 changed from
> NodeConnectionStatus[nodeId=hpce2r01n11.abc.com:9090, state=CONNECTING,
> updateId=60] to NodeConnectionStatus[nodeId=hpce2r01n11.abc.com:9090,
> state=DISCONNECTED, Disconnect Code=Node Failed to Startup Properly,
> Disconnect Reason=org.apache.nifi.cluster.ConnectionException: Failed to
> connect node to cluster due to: java.lang.NullPointerException, updateId=60]
> 2017-04-10 14:55:57,293 DEBUG [main] o.a.n.c.c.node.NodeClusterCoordinator
> State of cluster nodes is now
> {hpce2r01n12.abc.com:9090=NodeConnectionStatus[nodeId=hpce2r01n12.abc.c
[jira] [Deleted] (METRON-841) Failed to connect node to cluster due to: java.lang.NullPointerException
[
https://issues.apache.org/jira/browse/METRON-841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Casey Stella deleted METRON-841:
> Failed to connect node to cluster due to: java.lang.NullPointerException
> -
>
> Key: METRON-841
> URL: https://issues.apache.org/jira/browse/METRON-841
> Project: Metron
> Issue Type: Bug
> Environment: SUSE 11 SP3, NIFI 1.1.0.2.1.2.0-10, Java jdk1.8.0_60, 2
> node clustered Nifi
>Reporter: Joseph Niemiec
>
> We had a server fill its root directory, we stopped NiFi, moved the
> repositories to mounts with space, updated the configs and now we have a
> single node that will not join the cluster. There are no major errors but we
> do run into some WARNS talking about an NPE before Jetty explodes.
> 2017-04-10 14:55:57,293 ERROR [main] o.a.nifi.controller.StandardFlowService
> Failed to load flow from cluster due to:
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to
> cluster due to: java.lang.NullPointerException
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to
> cluster due to: java.lang.NullPointerException
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:901)
> ~[nifi-framework-core-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]
> at
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
> ~[nifi-framework-core-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:831)
> [nifi-jetty-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]
> at org.apache.nifi.NiFi.(NiFi.java:156)
> [nifi-runtime-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]
> at org.apache.nifi.NiFi.main(NiFi.java:262)
> [nifi-runtime-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]
> Caused by: java.lang.NullPointerException: null
> at
> org.apache.nifi.controller.repository.SchemaRepositoryRecordSerde.deserializeRecord(SchemaRepositoryRecordSerde.java:119)
> ~[nifi-framework-core-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]
> at
> org.apache.nifi.controller.repository.SchemaRepositoryRecordSerde.deserializeEdit(SchemaRepositoryRecordSerde.java:109)
> ~[nifi-framework-core-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]
> at
> org.apache.nifi.controller.repository.SchemaRepositoryRecordSerde.deserializeEdit(SchemaRepositoryRecordSerde.java:46)
> ~[nifi-framework-core-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]
> at
> org.wali.MinimalLockingWriteAheadLog$Partition.recoverNextTransaction(MinimalLockingWriteAheadLog.java:1072)
> ~[nifi-write-ahead-log-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]
> at
> org.wali.MinimalLockingWriteAheadLog.recoverFromEdits(MinimalLockingWriteAheadLog.java:459)
> ~[nifi-write-ahead-log-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]
> at
> org.wali.MinimalLockingWriteAheadLog.recoverRecords(MinimalLockingWriteAheadLog.java:301)
> ~[nifi-write-ahead-log-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]
> at
> org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.loadFlowFiles(WriteAheadFlowFileRepository.java:346)
> ~[nifi-framework-core-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]
> at
> org.apache.nifi.controller.FlowController.initializeFlow(FlowController.java:699)
> ~[nifi-framework-core-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]
> at
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:701)
> ~[nifi-framework-core-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:872)
> ~[nifi-framework-core-1.1.0.2.1.2.0-10.jar:1.1.0.2.1.2.0-10]
> ... 4 common frames omitted
> 2017-04-10 14:55:57,293 INFO [main] o.a.n.c.c.node.NodeClusterCoordinator
> hpce2r01n11.abc.com:9090 requested disconnection from cluster due to
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to
> cluster due to: java.lang.NullPointerException
> 2017-04-10 14:55:57,293 INFO [main] o.a.n.c.c.node.NodeClusterCoordinator
> Status of hpce2r01n11.abc.com:9090 changed from
> NodeConnectionStatus[nodeId=hpce2r01n11.abc.com:9090, state=CONNECTING,
> updateId=60] to NodeConnectionStatus[nodeId=hpce2r01n11.abc.com:9090,
> state=DISCONNECTED, Disconnect Code=Node Failed to Startup Properly,
> Disconnect Reason=org.apache.nifi.cluster.ConnectionException: Failed to
> connect node to cluster due to: java.lang.NullPointerException, updateId=60]
> 2017-04-10 14:55:57,293 DEBUG [main] o.a.n.c.c.node.NodeClusterCoordinator
> State of cluster nodes is now
> {hpce2r01n12.abc.com:9090=NodeConnectionStatus[nodeId=hpce2r01n12.abc.com:9090,
> state=CONNECTED, updateId=54],
> hpce2r01n11.abc.com:9090=NodeConnect
[jira] [Created] (METRON-833) Update MaaS documentation to explain how it interacts with kerberos
Casey Stella created METRON-833: --- Summary: Update MaaS documentation to explain how it interacts with kerberos Key: METRON-833 URL: https://issues.apache.org/jira/browse/METRON-833 Project: Metron Issue Type: Improvement Reporter: Casey Stella -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (METRON-626) Add Higher-Order Map Function to Stellar
[
https://issues.apache.org/jira/browse/METRON-626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Casey Stella updated METRON-626:
This ended up being an inadvertent dup of 821 which has a PR on it. Do you
think we should close this one or repurpose this one to make MAP function on
multi-dimensional structures like java maps?
> Add Higher-Order Map Function to Stellar
>
>
> Key: METRON-626
> URL: https://issues.apache.org/jira/browse/METRON-626
> Project: Metron
> Issue Type: Improvement
>Reporter: Nick Allen
>
> It would be extremely useful to have a 'map' higher-order function in
> Stellar. A 'flat map' might also be needed.
> [https://en.wikipedia.org/wiki/Map_(higher-order_function)]
> A few examples that I am thinking of. Ignore my choice of syntax. I am just
> trying to express the use cases here.
> Fetch all of the profile measurements over the past 2 hours. There may be
> many. What is the mean of each of those measurements?
> {code}
> stats := PROFILE_GET('p1', 'e1', 2, "HOURS")
> stats.MAP( s -> STATS_MEAN(s))
> {code}
> Get profile measurements taken 1 week ago, 2 weeks ago, and 3 weeks ago, so
> that I can then compare them.
> {code}
> { "1 week", "2 weeks", "3 weeks" }.MAP( lookback -> PROFILE_GET('p1','e1',
> lookback)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (METRON-831) Add lambda expressions and rudimentary functional programming primitives to Stellar
[ https://issues.apache.org/jira/browse/METRON-831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-831: Summary: Add lambda expressions and rudimentary functional programming primitives to Stellar (was: Add lambda expressions and rudimentary functional programming primitives to stellar) > Add lambda expressions and rudimentary functional programming primitives to > Stellar > --- > > Key: METRON-831 > URL: https://issues.apache.org/jira/browse/METRON-831 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella > > To enable use-cases where the user may want to do simple iteration, we can > expose simple functional primitives such as: > * MAP > * REDUCE > * FILTER > To fully support this, we will need lambda expressions as arguments to these > functions. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (METRON-831) Add lambda expressions and rudimentary functional programming primitives to stellar
[ https://issues.apache.org/jira/browse/METRON-831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-831: Description: To enable use-cases where the user may want to do simple iteration, we can expose simple functional primitives such as: * MAP * REDUCE * FILTER To fully support this, we will need lambda expressions as arguments to these functions. Summary: Add lambda expressions and rudimentary functional programming primitives to stellar (was: Add lambda functions and rudimentary functional primitives) > Add lambda expressions and rudimentary functional programming primitives to > stellar > --- > > Key: METRON-831 > URL: https://issues.apache.org/jira/browse/METRON-831 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella > > To enable use-cases where the user may want to do simple iteration, we can > expose simple functional primitives such as: > * MAP > * REDUCE > * FILTER > To fully support this, we will need lambda expressions as arguments to these > functions. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-831) Add lambda functions and rudimentary functional primitives
Casey Stella created METRON-831: --- Summary: Add lambda functions and rudimentary functional primitives Key: METRON-831 URL: https://issues.apache.org/jira/browse/METRON-831 Project: Metron Issue Type: Improvement Reporter: Casey Stella -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-820) StellarProcessor should have a static expression cache
Casey Stella created METRON-820: --- Summary: StellarProcessor should have a static expression cache Key: METRON-820 URL: https://issues.apache.org/jira/browse/METRON-820 Project: Metron Issue Type: Bug Reporter: Casey Stella We recreate StellarProcessor objects all over the place which do not reuse their expression caches. This results in a significant performance regression as we are compiling the expression and then processing for every message, rather than compiling once and reusing. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (METRON-812) Make the bro-kafka plugin work with kerberos
[ https://issues.apache.org/jira/browse/METRON-812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-812: Description: The bro-kafka plugin does not currently support kerberos. This JIRA should * make the version of librdkafka supported 0.9.4 * ensure the plugin can write to a kerberized kafka * provide instructions on how to configure the plugin to write to a kerberized kafka > Make the bro-kafka plugin work with kerberos > > > Key: METRON-812 > URL: https://issues.apache.org/jira/browse/METRON-812 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella > Labels: kerberos > > The bro-kafka plugin does not currently support kerberos. This JIRA should > * make the version of librdkafka supported 0.9.4 > * ensure the plugin can write to a kerberized kafka > * provide instructions on how to configure the plugin to write to a > kerberized kafka -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-812) Make the bro-kafka plugin work with kerberos
Casey Stella created METRON-812: --- Summary: Make the bro-kafka plugin work with kerberos Key: METRON-812 URL: https://issues.apache.org/jira/browse/METRON-812 Project: Metron Issue Type: Improvement Reporter: Casey Stella Assignee: Casey Stella -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (METRON-797) Pass security.protocol and enable auto-renew for the storm topologies
[ https://issues.apache.org/jira/browse/METRON-797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-797: Description: METRON-793 migrated the storm topologies to the storm-kafka-client spout, which supports kerberos. To complete the kerberos work on the existing topologies, we need to be able to enable the spouts and kafka writers to use security protocols other than PLAINTEXT. Also, enabling auto-renew plugins for storm will enable the topologies to run for extended durations in a kerberized cluster. This work was inspired by a portion of the investigatory work done at https://github.com/dlyle65535/incubator-metron/tree/kerb-testing?files=1 by: * @dlyle65535 * @merrimanr * @mmiklavc This carves out a specific piece of that functionality with the following differences: * The mpack work is not included, but the properties are set up to enable it as a follow-on * It presumes METRON-793, so it uses storm-kafka-client rather than storm-kafka * It adds a flag when starting the parsers to pass the security protocol and sets up the writers and the spout automatically rather than relying on the set of extra kafka configs (though both approaches would work here). NOTE: This does not encompass MPack changes to enable kerberos (METRON-799) or fix the sensors to work with a kerberized kafka (METRON-798). That would be follow-on work. was: METRON-793 migrated the storm topologies to the storm-kafka-client spout, which supports kerberos. To complete the kerberos work on the existing topologies, we need to be able to enable the spouts and kafka writers to use security protocols other than PLAINTEXT. Also, enabling auto-renew plugins for storm will enable the topologies to run for extended durations in a kerberized cluster. NOTE: This does not encompass MPack changes to enable kerberos (METRON-799) or fix the sensors to work with a kerberized kafka (METRON-798). That would be follow-on work. > Pass security.protocol and enable auto-renew for the storm topologies > - > > Key: METRON-797 > URL: https://issues.apache.org/jira/browse/METRON-797 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella > Labels: kerberos > > METRON-793 migrated the storm topologies to the storm-kafka-client spout, > which supports kerberos. To complete the kerberos work on the existing > topologies, we need to be able to enable the spouts and kafka writers to use > security protocols other than PLAINTEXT. Also, enabling auto-renew plugins > for storm will enable the topologies to run for extended durations in a > kerberized cluster. > This work was inspired by a portion of the investigatory work done at > https://github.com/dlyle65535/incubator-metron/tree/kerb-testing?files=1 by: > * @dlyle65535 > * @merrimanr > * @mmiklavc > This carves out a specific piece of that functionality with the following > differences: > * The mpack work is not included, but the properties are set up to enable it > as a follow-on > * It presumes METRON-793, so it uses storm-kafka-client rather than > storm-kafka > * It adds a flag when starting the parsers to pass the security protocol and > sets up the writers and the spout automatically rather than relying on the > set of extra kafka configs (though both approaches would work here). > NOTE: This does not encompass MPack changes to enable kerberos (METRON-799) > or fix the sensors to work with a kerberized kafka (METRON-798). That would > be follow-on work. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-804) Create a document to describe kerberizing vagrant
Casey Stella created METRON-804: --- Summary: Create a document to describe kerberizing vagrant Key: METRON-804 URL: https://issues.apache.org/jira/browse/METRON-804 Project: Metron Issue Type: Improvement Reporter: Casey Stella This should cover step-by-step how to kerberize vagrant with a local KDC and the manual setup to get Metron running data through into the indices. This will enable testing in a kerberized environment. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-803) Ensure REST API works with kerberized cluster
Casey Stella created METRON-803: --- Summary: Ensure REST API works with kerberized cluster Key: METRON-803 URL: https://issues.apache.org/jira/browse/METRON-803 Project: Metron Issue Type: Improvement Reporter: Casey Stella -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (METRON-801) Ensure pycapa works with kerberized cluster
[ https://issues.apache.org/jira/browse/METRON-801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-801: Labels: kerberos (was: ) > Ensure pycapa works with kerberized cluster > --- > > Key: METRON-801 > URL: https://issues.apache.org/jira/browse/METRON-801 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella > Labels: kerberos > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (METRON-797) Pass security.protocol and enable auto-renew for the storm topologies
[ https://issues.apache.org/jira/browse/METRON-797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-797: Labels: kerberos (was: ) > Pass security.protocol and enable auto-renew for the storm topologies > - > > Key: METRON-797 > URL: https://issues.apache.org/jira/browse/METRON-797 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella > Labels: kerberos > > METRON-793 migrated the storm topologies to the storm-kafka-client spout, > which supports kerberos. To complete the kerberos work on the existing > topologies, we need to be able to enable the spouts and kafka writers to use > security protocols other than PLAINTEXT. Also, enabling auto-renew plugins > for storm will enable the topologies to run for extended durations in a > kerberized cluster. > NOTE: This does not encompass MPack changes to enable kerberos or fix the > sensors to work with a kerberized kafka. That would be follow-on work. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-802) Ensure fastcapa works with a kerberized cluster
Casey Stella created METRON-802: --- Summary: Ensure fastcapa works with a kerberized cluster Key: METRON-802 URL: https://issues.apache.org/jira/browse/METRON-802 Project: Metron Issue Type: Improvement Reporter: Casey Stella -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-801) Ensure pycapa works with kerberized cluster
Casey Stella created METRON-801: --- Summary: Ensure pycapa works with kerberized cluster Key: METRON-801 URL: https://issues.apache.org/jira/browse/METRON-801 Project: Metron Issue Type: Improvement Reporter: Casey Stella -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-800) Ensure MaaS works with Kerberized Cluster
Casey Stella created METRON-800: --- Summary: Ensure MaaS works with Kerberized Cluster Key: METRON-800 URL: https://issues.apache.org/jira/browse/METRON-800 Project: Metron Issue Type: Improvement Reporter: Casey Stella -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-799) The MPack should function in a kerberized cluster
Casey Stella created METRON-799: --- Summary: The MPack should function in a kerberized cluster Key: METRON-799 URL: https://issues.apache.org/jira/browse/METRON-799 Project: Metron Issue Type: Improvement Reporter: Casey Stella -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-798) Bro plugin should be made to talk to a kerberized kafka
Casey Stella created METRON-798: --- Summary: Bro plugin should be made to talk to a kerberized kafka Key: METRON-798 URL: https://issues.apache.org/jira/browse/METRON-798 Project: Metron Issue Type: Improvement Reporter: Casey Stella -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (METRON-793) Migrate to storm-kafka-client kafka spout from storm-kafka
[ https://issues.apache.org/jira/browse/METRON-793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-793: Labels: kerberos (was: ) > Migrate to storm-kafka-client kafka spout from storm-kafka > -- > > Key: METRON-793 > URL: https://issues.apache.org/jira/browse/METRON-793 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella > Labels: kerberos > > In order to eventually support kerberos, the suggested path is to migrate to > the new kafka spout (org.apache.storm:storm-kafka-client) which uses the new > consumer API. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-797) Pass security.protocol and enable auto-renew for the storm topologies
[ https://issues.apache.org/jira/browse/METRON-797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940918#comment-15940918 ] Casey Stella commented on METRON-797: - We require the new storm kafka bolt to pass protocol to kafka. > Pass security.protocol and enable auto-renew for the storm topologies > - > > Key: METRON-797 > URL: https://issues.apache.org/jira/browse/METRON-797 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella > > METRON-793 migrated the storm topologies to the storm-kafka-client spout, > which supports kerberos. To complete the kerberos work on the existing > topologies, we need to be able to enable the spouts and kafka writers to use > security protocols other than PLAINTEXT. Also, enabling auto-renew plugins > for storm will enable the topologies to run for extended durations in a > kerberized cluster. > NOTE: This does not encompass MPack changes to enable kerberos or fix the > sensors to work with a kerberized kafka. That would be follow-on work. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (METRON-797) Pass security.protocol and enable auto-renew for the storm topologies
[ https://issues.apache.org/jira/browse/METRON-797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella reassigned METRON-797: --- Assignee: Casey Stella > Pass security.protocol and enable auto-renew for the storm topologies > - > > Key: METRON-797 > URL: https://issues.apache.org/jira/browse/METRON-797 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella > > METRON-793 migrated the storm topologies to the storm-kafka-client spout, > which supports kerberos. To complete the kerberos work on the existing > topologies, we need to be able to enable the spouts and kafka writers to use > security protocols other than PLAINTEXT. Also, enabling auto-renew plugins > for storm will enable the topologies to run for extended durations in a > kerberized cluster. > NOTE: This does not encompass MPack changes to enable kerberos or fix the > sensors to work with a kerberized kafka. That would be follow-on work. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-797) Pass security.protocol and enable auto-renew for the storm topologies
Casey Stella created METRON-797: --- Summary: Pass security.protocol and enable auto-renew for the storm topologies Key: METRON-797 URL: https://issues.apache.org/jira/browse/METRON-797 Project: Metron Issue Type: Improvement Reporter: Casey Stella METRON-793 migrated the storm topologies to the storm-kafka-client spout, which supports kerberos. To complete the kerberos work on the existing topologies, we need to be able to enable the spouts and kafka writers to use security protocols other than PLAINTEXT. Also, enabling auto-renew plugins for storm will enable the topologies to run for extended durations in a kerberized cluster. NOTE: This does not encompass MPack changes to enable kerberos or fix the sensors to work with a kerberized kafka. That would be follow-on work. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-794) When we move to Storm 1.1.x, migrate the storm-kafka-client dependency back to Storm from HDP-specific
Casey Stella created METRON-794: --- Summary: When we move to Storm 1.1.x, migrate the storm-kafka-client dependency back to Storm from HDP-specific Key: METRON-794 URL: https://issues.apache.org/jira/browse/METRON-794 Project: Metron Issue Type: Task Reporter: Casey Stella Right now in the top level pom, we are depending on a HDP version of the storm-kafka-client due to it supporting kafka 0.10.x. As of storm 1.1.x+ we should be able to move back to the apache version. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-793) Migrate to storm-kafka-client kafka spout from storm-kafka
Casey Stella created METRON-793: --- Summary: Migrate to storm-kafka-client kafka spout from storm-kafka Key: METRON-793 URL: https://issues.apache.org/jira/browse/METRON-793 Project: Metron Issue Type: Improvement Reporter: Casey Stella In order to eventually support kerberos, the suggested path is to migrate to the new kafka spout (org.apache.storm:storm-kafka-client) which uses the new consumer API. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-773) Intermittent unit test errors in the KafkaControllerIntegrationTest
Casey Stella created METRON-773: --- Summary: Intermittent unit test errors in the KafkaControllerIntegrationTest Key: METRON-773 URL: https://issues.apache.org/jira/browse/METRON-773 Project: Metron Issue Type: Bug Reporter: Casey Stella Assignee: Casey Stella Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 8.514 sec <<< FAILURE! - in org.apache.metron.rest.controller.KafkaControllerIntegrationTest test(org.apache.metron.rest.controller.KafkaControllerIntegrationTest) Time elapsed: 8.415 sec <<< FAILURE! java.lang.AssertionError: Status expected:<200> but was:<404> at org.springframework.test.util.AssertionErrors.fail(AssertionErrors.java:54) at org.springframework.test.util.AssertionErrors.assertEquals(AssertionErrors.java:81) at org.springframework.test.web.servlet.result.StatusResultMatchers$10.match(StatusResultMatchers.java:664) at org.springframework.test.web.servlet.MockMvc$1.andExpect(MockMvc.java:171) at org.apache.metron.rest.controller.KafkaControllerIntegrationTest.test(KafkaControllerIntegrationTest.java:173) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86) at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:252) at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:94) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61) at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:191) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-768) Remove the overreaching Rat exclusions from the build
Casey Stella created METRON-768: --- Summary: Remove the overreaching Rat exclusions from the build Key: METRON-768 URL: https://issues.apache.org/jira/browse/METRON-768 Project: Metron Issue Type: Improvement Reporter: Casey Stella Right now the rat exclusions are broad. We should investigate whether they are still relevant, put a comment to justify their relevance and look into moving them into exclusions for the individual modules where applicable. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (METRON-767) Clean up license
[ https://issues.apache.org/jira/browse/METRON-767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-767: Summary: Clean up license (was: Clean up license from METRON-622) > Clean up license > > > Key: METRON-767 > URL: https://issues.apache.org/jira/browse/METRON-767 > Project: Metron > Issue Type: Task >Reporter: Casey Stella > > There's some verbiage at the end that shouldn't exist and we're missing > inclusion of the actual licenses. I corrected both. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (METRON-767) Clean up license from METRON-622
[ https://issues.apache.org/jira/browse/METRON-767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-767: Description: There's some verbiage at the end that shouldn't exist and we're missing inclusion of the actual licenses. I corrected both. (was: Metron-622 put some verbiage at the end of the license that isn't necessary and should be removed.) > Clean up license from METRON-622 > > > Key: METRON-767 > URL: https://issues.apache.org/jira/browse/METRON-767 > Project: Metron > Issue Type: Task >Reporter: Casey Stella > > There's some verbiage at the end that shouldn't exist and we're missing > inclusion of the actual licenses. I corrected both. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-767) Clean up license from METRON-622
Casey Stella created METRON-767: --- Summary: Clean up license from METRON-622 Key: METRON-767 URL: https://issues.apache.org/jira/browse/METRON-767 Project: Metron Issue Type: Task Reporter: Casey Stella Metron-622 put some verbiage at the end of the license that isn't necessary and should be removed. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (METRON-766) Release 0.3.1
[ https://issues.apache.org/jira/browse/METRON-766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-766: Summary: Release 0.3.1 (was: Announce the 0.3.1 release) > Release 0.3.1 > - > > Key: METRON-766 > URL: https://issues.apache.org/jira/browse/METRON-766 > Project: Metron > Issue Type: Task >Reporter: Casey Stella >Assignee: Casey Stella > > * Remove the 0.3.0 bits in svn from dist > * Add the 0.3.1 bits from RC5 to dist > * Update the webpage with the 0.3.1 release -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-766) Announce the 0.3.1 release
Casey Stella created METRON-766: --- Summary: Announce the 0.3.1 release Key: METRON-766 URL: https://issues.apache.org/jira/browse/METRON-766 Project: Metron Issue Type: Task Reporter: Casey Stella Assignee: Casey Stella * Remove the 0.3.0 bits in svn from dist * Add the 0.3.1 bits from RC5 to dist * Update the webpage with the 0.3.1 release -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-763) When the indicator column is not present in the extractor config for enrichment loading, we should not NPE
Casey Stella created METRON-763: --- Summary: When the indicator column is not present in the extractor config for enrichment loading, we should not NPE Key: METRON-763 URL: https://issues.apache.org/jira/browse/METRON-763 Project: Metron Issue Type: Bug Reporter: Casey Stella -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-762) Storm to HDFS Microbatch writer option for PCAP
Casey Stella created METRON-762: --- Summary: Storm to HDFS Microbatch writer option for PCAP Key: METRON-762 URL: https://issues.apache.org/jira/browse/METRON-762 Project: Metron Issue Type: Improvement Reporter: Casey Stella Assignee: Casey Stella Currently storm writes a packet at a time to HDFS. This has performance implications. Rather, we should allow the user to define a batch size so that writes are batched. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-748) MaaS errors are inscrutible and horrible
Casey Stella created METRON-748: --- Summary: MaaS errors are inscrutible and horrible Key: METRON-748 URL: https://issues.apache.org/jira/browse/METRON-748 Project: Metron Issue Type: Bug Reporter: Casey Stella If a container fails, we get an exception about the shell script erroring out in the App Master log. We should package up and send the stderr and stdout to app master. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-744) Allow Stellar functions to be loaded from HDFS
Casey Stella created METRON-744: --- Summary: Allow Stellar functions to be loaded from HDFS Key: METRON-744 URL: https://issues.apache.org/jira/browse/METRON-744 Project: Metron Issue Type: New Feature Reporter: Casey Stella The benefit of Stellar is that adding new functionality is as simple as providing a Jar. This enables people who want to integrate with Metron to easy add enrichments or other functionality. The snag currently with this is that we provide a single jar, so all stellar functions that we have available must be dependencies of the main jar that drives the topology plus what local directories we can configure via the storm configs. This makes the process of adding 3rd party jars not as easy as it could be. Adjust the the following to additionally load classes from a location in HDFS /apps/metron/stellar using something like accumulo ( https://accumulo.apache.org/blog/2014/05/03/accumulo-classloader.html) * Profiler topology * Parser topology * Enrichment topology * Enrichment Flat file loader * Enrichment MR loader -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-743) Sort the files when reading results from Pcap
Casey Stella created METRON-743: --- Summary: Sort the files when reading results from Pcap Key: METRON-743 URL: https://issues.apache.org/jira/browse/METRON-743 Project: Metron Issue Type: Bug Reporter: Casey Stella The FileSystem.listFiles() call does not return the files in sorted order, which we assume for all FileSystem implementations. We should sort this to be certain. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-742) Generated code for profile window selector DSL did not get committed as part of METRON-690
Casey Stella created METRON-742: --- Summary: Generated code for profile window selector DSL did not get committed as part of METRON-690 Key: METRON-742 URL: https://issues.apache.org/jira/browse/METRON-742 Project: Metron Issue Type: Bug Reporter: Casey Stella -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-741) Stellar Field Transformations should execute all of the transformations, not just the output
Casey Stella created METRON-741: --- Summary: Stellar Field Transformations should execute all of the transformations, not just the output Key: METRON-741 URL: https://issues.apache.org/jira/browse/METRON-741 Project: Metron Issue Type: Improvement Reporter: Casey Stella Currently, the only stellar field transformations that are executed are those in the output field. We should execute all of the stellar statements in the order they are specified. Stellar statements which are not output may be used as intermediate variables for fields which are output. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-740) Add the ability to pass a log4j properties to MaaS CLI
Casey Stella created METRON-740: --- Summary: Add the ability to pass a log4j properties to MaaS CLI Key: METRON-740 URL: https://issues.apache.org/jira/browse/METRON-740 Project: Metron Issue Type: Improvement Reporter: Casey Stella right now deploy and service CLI programs for MaaS do not allow the user to override the log4j properties. The flatfile loader does and MaaS should follow suit. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Deleted] (METRON-736) Create a structure transformer for messages that is specifiable in the writer config
[ https://issues.apache.org/jira/browse/METRON-736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella deleted METRON-736: > Create a structure transformer for messages that is specifiable in the writer > config > > > Key: METRON-736 > URL: https://issues.apache.org/jira/browse/METRON-736 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella > > Currently we have a requirement that structures indexed be flattened and > modified in some small ways (keys are transformed to : from . in ES). We > should have a pluggable layer to enforce structure rules that is configurable > by writer. The default should be specified by the writer implementation. > This way we can have our cake and eat it too. Also, we could ALREADY be in a > situation where messages aren't flat (imagine a situation where a stellar > function returns a map or a list and it get assigned to a field). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-736) Create a structure transformer for messages that is specifiable in the writer config
Casey Stella created METRON-736: --- Summary: Create a structure transformer for messages that is specifiable in the writer config Key: METRON-736 URL: https://issues.apache.org/jira/browse/METRON-736 Project: Metron Issue Type: Improvement Reporter: Casey Stella Currently we have a requirement that structures indexed be flattened and modified in some small ways (keys are transformed to : from . in ES). We should have a pluggable layer to enforce structure rules that is configurable by writer. The default should be specified by the writer implementation. This way we can have our cake and eat it too. Also, we could ALREADY be in a situation where messages aren't flat (imagine a situation where a stellar function returns a map or a list and it get assigned to a field). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-732) When an error happens in an composed function in stellar, the real exception is not bubbled up
Casey Stella created METRON-732:
---
Summary: When an error happens in an composed function in stellar,
the real exception is not bubbled up
Key: METRON-732
URL: https://issues.apache.org/jira/browse/METRON-732
Project: Metron
Issue Type: Bug
Reporter: Casey Stella
If you have a function which throws an exception and it is an inner function of
a composition, then the exception isn't bubbled up.
For instance, if {{PROFILE_WINDOW}} throws an exception in the following:
{{STATS_MEAN(STATS_MERGE(PROFILE_GET('stat', 'global', PROFILE_WINDOW('5 minute
window starting ago'}}
We will get
{{org.apache.metron.common.dsl.ParseException: Unable to execute: expected at
least 3 argument(s), found 2}}
Rather than the actual exception thrown from PROFILE_WINDOW
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (METRON-713) Port dummy_rest.sh to jython from bash
[ https://issues.apache.org/jira/browse/METRON-713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15861855#comment-15861855 ] Casey Stella commented on METRON-713: - Everyone building Metron has java on their system, not everyone has bash (problem with the current impl) or python. The other option is just java, honestly. > Port dummy_rest.sh to jython from bash > -- > > Key: METRON-713 > URL: https://issues.apache.org/jira/browse/METRON-713 > Project: Metron > Issue Type: Bug >Reporter: Casey Stella > > Currently dummy_rest.sh, a part of the integration test package for Metron, > uses netcat. Modern versions of netcat won't let it bind to the port it's > trying. It's probably worth while to port this service over to jython so > that it's more cross-platform friendly. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (METRON-713) Port dummy_rest.sh to jython from bash
[ https://issues.apache.org/jira/browse/METRON-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-713: Summary: Port dummy_rest.sh to jython from bash (was: Port dummy_rest.sh to jython from shell) > Port dummy_rest.sh to jython from bash > -- > > Key: METRON-713 > URL: https://issues.apache.org/jira/browse/METRON-713 > Project: Metron > Issue Type: Bug >Reporter: Casey Stella > > Currently dummy_rest.sh, a part of the integration test package for Metron, > uses netcat. Modern versions of netcat won't let it bind to the port it's > trying. It's probably worth while to port this service over to jython so > that it's more cross-platform friendly. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-713) Port dummy_rest.sh to jython from shell
Casey Stella created METRON-713: --- Summary: Port dummy_rest.sh to jython from shell Key: METRON-713 URL: https://issues.apache.org/jira/browse/METRON-713 Project: Metron Issue Type: Bug Reporter: Casey Stella Currently dummy_rest.sh, a part of the integration test package for Metron, uses netcat. Modern versions of netcat won't let it bind to the port it's trying. It's probably worth while to port this service over to jython so that it's more cross-platform friendly. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (METRON-712) Separate evaluation from parsing in Stellar
[ https://issues.apache.org/jira/browse/METRON-712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-712: Description: With the current implementation of Stellar, we cannot cache the parse tree and then apply it after the fact. It's just an artifact of how we do the parsing: we actually execute the statement as we parse rather than constructing an AST that can then be evaluated later given a message. Essentially what I'm proposing is that we build the equivalent of Pattern.compile() in Java except for Stellar. We should for multiple reasons: * code clarity - decoupling the stellar language from the generated parser code * performance - saving lexing and parsing for every message was: With the current implementation of Stellar, we cannot cache the parse tree and then apply it after the fact. It's just an artifact of how we do the parsing: we actually execute the statement as we parse rather than constructing an AST that can then be evaluated later given a message. We should for multiple reasons: * code clarity - decoupling the stellar language from the generated parser code * performance - saving lexing and parsing for every message > Separate evaluation from parsing in Stellar > --- > > Key: METRON-712 > URL: https://issues.apache.org/jira/browse/METRON-712 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella > > With the current implementation of Stellar, we cannot cache the parse tree > and then apply it after the fact. It's just an artifact of how we do the > parsing: we actually execute the statement as we parse rather than > constructing an AST that can then be evaluated later given a message. > Essentially what I'm proposing is that we build the equivalent of > Pattern.compile() in Java except for Stellar. > We should for multiple reasons: > * code clarity - decoupling the stellar language from the generated parser > code > * performance - saving lexing and parsing for every message -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-712) Separate evaluation from parsing in Stellar
Casey Stella created METRON-712: --- Summary: Separate evaluation from parsing in Stellar Key: METRON-712 URL: https://issues.apache.org/jira/browse/METRON-712 Project: Metron Issue Type: Improvement Reporter: Casey Stella With the current implementation of Stellar, we cannot cache the parse tree and then apply it after the fact. It's just an artifact of how we do the parsing: we actually execute the statement as we parse rather than constructing an AST that can then be evaluated later given a message. We should for multiple reasons: * code clarity - decoupling the stellar language from the generated parser code * performance - saving lexing and parsing for every message -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-707) Correct ansible to execute threat intel bulk loading via the flat file script
Casey Stella created METRON-707: --- Summary: Correct ansible to execute threat intel bulk loading via the flat file script Key: METRON-707 URL: https://issues.apache.org/jira/browse/METRON-707 Project: Metron Issue Type: Bug Reporter: Casey Stella Assignee: Casey Stella Fix For: 0.3.1 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (METRON-689) Create Cron-syntax based timestamp lookup for profiler to enable sparse windows
[ https://issues.apache.org/jira/browse/METRON-689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella reassigned METRON-689: --- Assignee: Casey Stella > Create Cron-syntax based timestamp lookup for profiler to enable sparse > windows > --- > > Key: METRON-689 > URL: https://issues.apache.org/jira/browse/METRON-689 > Project: Metron > Issue Type: New Feature >Reporter: Casey Stella >Assignee: Casey Stella > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (METRON-699) Update metron-statistics documentation
[ https://issues.apache.org/jira/browse/METRON-699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-699: Fix Version/s: 0.3.1 > Update metron-statistics documentation > -- > > Key: METRON-699 > URL: https://issues.apache.org/jira/browse/METRON-699 > Project: Metron > Issue Type: Bug >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Trivial > Fix For: 0.3.1 > > > Update the metron-statistics documentation. Currently some of the > documentation is not properly hierarchical. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (METRON-705) Parallelize the build in travis to the extent that is obvious
[ https://issues.apache.org/jira/browse/METRON-705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella reassigned METRON-705: --- Assignee: Casey Stella > Parallelize the build in travis to the extent that is obvious > - > > Key: METRON-705 > URL: https://issues.apache.org/jira/browse/METRON-705 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella > > Travis suggests at > https://blog.travis-ci.com/2012-11-28-speeding-up-your-tests-by-parallelizing-them/ > that for situations where the integration get chunky, we can parallelize > them using their build matrix. Also, if we can separate those out, we can > also process-parallelize the unit and build. > This is just a stopgap that requires no code changes to lower build > wall-clock times. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (METRON-678) Multithread the flat file loader
[ https://issues.apache.org/jira/browse/METRON-678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-678: Fix Version/s: 0.3.1 > Multithread the flat file loader > > > Key: METRON-678 > URL: https://issues.apache.org/jira/browse/METRON-678 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella > Fix For: 0.3.1 > > > Currently the flat file loader is single threaded in its writing to HBase. > We could make this a lot faster by multithreading the HBase puts. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (METRON-684) Decouple Timestamp calculation from PROFILE_GET
[ https://issues.apache.org/jira/browse/METRON-684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-684: Fix Version/s: 0.3.1 > Decouple Timestamp calculation from PROFILE_GET > --- > > Key: METRON-684 > URL: https://issues.apache.org/jira/browse/METRON-684 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella > Fix For: 0.3.1 > > > Currently PROFILE_GET only supports a static lookback of a fixed duration. > As we have more complicated, potentially sparse, lookbacks (e.g. the same > time slice every tuesday for a month), it would be nice to decouple the > construction of timestamps from PROFILE_GET into its own set of functions. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-705) Parallelize the build in travis to the extent that is obvious
Casey Stella created METRON-705: --- Summary: Parallelize the build in travis to the extent that is obvious Key: METRON-705 URL: https://issues.apache.org/jira/browse/METRON-705 Project: Metron Issue Type: Improvement Reporter: Casey Stella Travis suggests at https://blog.travis-ci.com/2012-11-28-speeding-up-your-tests-by-parallelizing-them/ that for situations where the integration get chunky, we can parallelize them using their build matrix. Also, if we can separate those out, we can also process-parallelize the unit and build. This is just a stopgap that requires no code changes to lower build wall-clock times. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-703) Rev the version from 0.3.0 to 0.3.1
Casey Stella created METRON-703: --- Summary: Rev the version from 0.3.0 to 0.3.1 Key: METRON-703 URL: https://issues.apache.org/jira/browse/METRON-703 Project: Metron Issue Type: Task Reporter: Casey Stella Assignee: Casey Stella -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (METRON-703) Rev the version from 0.3.0 to 0.3.1
[ https://issues.apache.org/jira/browse/METRON-703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-703: Fix Version/s: 0.3.1 > Rev the version from 0.3.0 to 0.3.1 > --- > > Key: METRON-703 > URL: https://issues.apache.org/jira/browse/METRON-703 > Project: Metron > Issue Type: Task >Reporter: Casey Stella >Assignee: Casey Stella > Fix For: 0.3.1 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-701) Triage Metrics Produced by the Profiler
[ https://issues.apache.org/jira/browse/METRON-701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15854460#comment-15854460 ] Casey Stella commented on METRON-701: - So, I don't necessarily have an issue with hard coding the write paths in the topologies (option 2), but could you elaborate on why you think that way is more scalable than constructing a write bolt handles the write phase of the profiles? Essentially you have a separate bolt which would execute the stellar statements in the write phase of the profiles associated with the writes being done. > Triage Metrics Produced by the Profiler > --- > > Key: METRON-701 > URL: https://issues.apache.org/jira/browse/METRON-701 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen > > h3. Problem > The motivating example is that I would like to create an alert if the number > of inbound flows to any host over a 15 minute interval is abnormal. > The value being interrogated here, the number of inbound flows, is not a > static value contained within any single telemetry message. This value is > calculated across multiple messages by the Profiler. The current Threat > Triage process cannot be used to interrogate values calculated by the > Profiler. > h3. Proposed Solution > I am proposing that we treat the Profiler as a source of telemetry. The > measurements captured by the Profiler would be enqueued into a Kafka topic. > We would then treat those Profiler messages like any other telemetry. We > would parse, enrich, triage, and index those messages. > This would have the following advantages. > 1. We would be able to reuse the same threat triage mechanism for values > calculated by the Profiler. > 2. We would be able to generate profiles from the profiled data - aka > meta-profiles anyone? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (METRON-692) Update Upgrading.md for 0.3.0 -> 0.3.1
[ https://issues.apache.org/jira/browse/METRON-692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella reassigned METRON-692: --- Assignee: Casey Stella > Update Upgrading.md for 0.3.0 -> 0.3.1 > -- > > Key: METRON-692 > URL: https://issues.apache.org/jira/browse/METRON-692 > Project: Metron > Issue Type: Task >Reporter: Casey Stella >Assignee: Casey Stella > Fix For: 0.3.1 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (METRON-692) Update Upgrading.md for 0.3.0 -> 0.3.1
[ https://issues.apache.org/jira/browse/METRON-692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-692: Fix Version/s: 0.3.1 > Update Upgrading.md for 0.3.0 -> 0.3.1 > -- > > Key: METRON-692 > URL: https://issues.apache.org/jira/browse/METRON-692 > Project: Metron > Issue Type: Task >Reporter: Casey Stella >Assignee: Casey Stella > Fix For: 0.3.1 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-692) Update Upgrading.md for 0.3.0 -> 0.3.1
Casey Stella created METRON-692: --- Summary: Update Upgrading.md for 0.3.0 -> 0.3.1 Key: METRON-692 URL: https://issues.apache.org/jira/browse/METRON-692 Project: Metron Issue Type: Task Reporter: Casey Stella -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-690) Create a DSL-based timestamp lookup for profiler to enable sparse windows
Casey Stella created METRON-690: --- Summary: Create a DSL-based timestamp lookup for profiler to enable sparse windows Key: METRON-690 URL: https://issues.apache.org/jira/browse/METRON-690 Project: Metron Issue Type: New Feature Reporter: Casey Stella I propose that we support the following features: * A starting point that is not current time * Sparse bins (i.e. the last hour for every tuesday for the last month) * The ability to skip events (e.g. weekends, holidays) This would result in a new function with the following arguments: from - The lookback starting point (default to now) fromUnits - The units for the lookback starting point to - The ending point for the lookback window (default to from + binSize) toUnits - The units for the lookback ending point including - A list of conditions which we would skip. weekend holiday sunday through saturday excluding - A list of conditions which we would skip. weekend holiday sunday through saturday binSize - The size of the lookback bin binUnits - The units of the lookback bin Given the number of arguments and their complexity and the fact that many, many are optional, PROFILE_LOOKBACK accept a string backed by a DSL to express these criteria Base Case: A lookback of 1 hour ago PROFILE_LOOKBACK( '1 hour bins from now') Example 1: The same time window every tuesday for the last month starting one hour ago Just to make this as clear as possible, if this is run at 3PM on Monday January 23rd, 2017, it would include the following bins: January 17th, 2PM - 3PM January 10th, 2PM - 3PM January 3rd, 2PM - 3PM December 27th, 2PM - 3PM PROFILE_LOOKBACK( '1 hour bins from 1 hour to 1 month including tuesdays') Example 2: The same time window every sunday for the last month starting one hour ago skipping holidays Just to make this as clear as possible, if this is run at 3PM on Monday January 22rd, 2017, it would include the following bins: January 16th, 2PM - 3PM January 9th, 2PM - 3PM January 2rd, 2PM - 3PM NOT December 25th PROFILE_LOOKBACK( '1 hour bins from 1 hour to 1 month including tuesdays excluding holidays') -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-689) Create Cron-syntax based timestamp lookup for profiler to enable sparse windows
Casey Stella created METRON-689: --- Summary: Create Cron-syntax based timestamp lookup for profiler to enable sparse windows Key: METRON-689 URL: https://issues.apache.org/jira/browse/METRON-689 Project: Metron Issue Type: New Feature Reporter: Casey Stella -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-684) Decouple Timestamp calculation from PROFILE_GET
Casey Stella created METRON-684: --- Summary: Decouple Timestamp calculation from PROFILE_GET Key: METRON-684 URL: https://issues.apache.org/jira/browse/METRON-684 Project: Metron Issue Type: Improvement Reporter: Casey Stella Assignee: Casey Stella Currently PROFILE_GET only supports a static lookback of a fixed duration. As we have more complicated, potentially sparse, lookbacks (e.g. the same time slice every tuesday for a month), it would be nice to decouple the construction of timestamps from PROFILE_GET into its own set of functions. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-683) Threat Triage aggregators should just be a stellar expression
Casey Stella created METRON-683: --- Summary: Threat Triage aggregators should just be a stellar expression Key: METRON-683 URL: https://issues.apache.org/jira/browse/METRON-683 Project: Metron Issue Type: Improvement Reporter: Casey Stella -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-682) Unify and Improve the Flat File Loader
Casey Stella created METRON-682: --- Summary: Unify and Improve the Flat File Loader Key: METRON-682 URL: https://issues.apache.org/jira/browse/METRON-682 Project: Metron Issue Type: Improvement Reporter: Casey Stella Currently the flat file loader is deficient in a couple ways: * It only supports importing local data despite there being a separate, poorly named, application which supports importing enrichment via MapReduce called threat_intel_loader.sh * It does not support local imports from HDFS * It does not support local imports from URLs * It does not support importing zipped archives locally * You cannot import more than one file at once This JIRA will: * Unify the MapReduce and local imports into one program and allow the user to specify the import mode with a CLI flag * Support local imports from HDFS and URLs * Support local imports from zipped files * Support importing more than one file at once -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (METRON-678) Multithread the flat file loader
[ https://issues.apache.org/jira/browse/METRON-678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-678: Description: Currently the flat file loader is single threaded in its writing to HBase. We could make this a lot faster by multithreading the HBase puts. > Multithread the flat file loader > > > Key: METRON-678 > URL: https://issues.apache.org/jira/browse/METRON-678 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella > > Currently the flat file loader is single threaded in its writing to HBase. > We could make this a lot faster by multithreading the HBase puts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-678) Multithread the flat file loader
Casey Stella created METRON-678: --- Summary: Multithread the flat file loader Key: METRON-678 URL: https://issues.apache.org/jira/browse/METRON-678 Project: Metron Issue Type: Improvement Reporter: Casey Stella Assignee: Casey Stella -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-627) Add HyperLogLogPlus implementation to Stellar
[ https://issues.apache.org/jira/browse/METRON-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-627: Fix Version/s: 0.3.1 > Add HyperLogLogPlus implementation to Stellar > - > > Key: METRON-627 > URL: https://issues.apache.org/jira/browse/METRON-627 > Project: Metron > Issue Type: Improvement >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic > Fix For: 0.3.1 > > > Calculating set cardinality can be a useful tool for a security analyst. For > instance, a large volume of non-unique src ip addresses hitting your network > may be an indication that you are currently under attack. There have been > many advancements in distinct value (DV) estimation over the years. We have > seen implementations evolve from K-Minimum-Values (KMV), to LogLog, to > HyperLogLog, and now to Google's much-improved HyperLogLogPlu algorithm. The > key improvements in this latest manifestation of the algorithm are: > moves to a 64-bit hash > handles sparse sets > is more accurate with small cardinality > This Jira tracks the effort to add a HyperLogLogPlus implementation to Metron. > References: > https://research.neustar.biz/2013/01/24/hyperloglog-googles-take-on-engineering-hll/ > http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/40671.pdf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-641) Fix Kibana install file
[
https://issues.apache.org/jira/browse/METRON-641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Casey Stella updated METRON-641:
Assignee: (was: Casey Stella)
> Fix Kibana install file
> ---
>
> Key: METRON-641
> URL: https://issues.apache.org/jira/browse/METRON-641
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.3.0
>Reporter: Dima Kovalyov
>Priority: Minor
> Fix For: 0.3.1
>
>
> Kibana installed as part of Metron mpack winthin Ambari will fail during
> start with following error:
> {code}
> ValueError: zero length field name in format
> {code}
> We can fix it with:
> {code}
> sed -i 's@{}/kibana@{0}/kibana@g'
> /incubator-metron/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/KIBANA/4.5.1/package/scripts/kibana_master.py
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (METRON-642) Fix ElasticSearch Mpack issues in Metron
[
https://issues.apache.org/jira/browse/METRON-642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Casey Stella updated METRON-642:
Assignee: (was: Casey Stella)
> Fix ElasticSearch Mpack issues in Metron
>
>
> Key: METRON-642
> URL: https://issues.apache.org/jira/browse/METRON-642
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.3.0
>Reporter: Dima Kovalyov
>Priority: Minor
> Fix For: 0.3.1
>
>
> First
> Currently in Metron Mpack es.ip in Metron configuration is set to: es_url.
> Which is incorrect and will fail, instead it should be replaced with:
> "es.ip": "",
> "es.port": "9300",
> But I do not know how to get elasticsearch_master_hostname. Also, value of
> es.port seems like can be defaulted to es_binary_port or even omitted as it
> is already specified in es_binary_port.
> This should be chanted in:
> https://github.com/apache/incubator-metron/blob/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml
> On line 208.
> First
> Second
> In file
> https://github.com/apache/incubator-metron/blob/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/configuration/elastic-sysconfig.xml
> On lines 93 and 95 symbols "/" is forgotten. e.g:
> ES_JAVA_OPTS="-verbose:gc -Xloggc:{{log_dir}}/elasticsearch_gc.log
> -XX:-CMSConcurrentMTEnabled
> should be:
> ES_JAVA_OPTS="-verbose:gc -Xloggc:{{log_dir}}/elasticsearch_gc.log
> -XX:-CMSConcurrentMTEnabled
> Second
> Third
> After ES install it will fail to start with error"
> {code}
> Likely root cause: java.nio.file.AccessDeniedException:
> /etc/elasticsearch/scripts
> {code}
> The reason for the is root permission on these files:
> {code}
> # ls -la /etc/elasticsearch
> ...
> -rwxr-x---. 1 root elasticsearch 2571 May 12 09:24 logging.yml
> drwxr-x---. 2 root elasticsearch 4096 May 17 11:49 scripts
> {code}
> I was not able to trace down origin of these files, so do not know how to fix
> it.
> Third
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (METRON-642) Fix ElasticSearch Mpack issues in Metron
[
https://issues.apache.org/jira/browse/METRON-642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Casey Stella updated METRON-642:
Fix Version/s: 0.3.1
> Fix ElasticSearch Mpack issues in Metron
>
>
> Key: METRON-642
> URL: https://issues.apache.org/jira/browse/METRON-642
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.3.0
>Reporter: Dima Kovalyov
>Priority: Minor
> Fix For: 0.3.1
>
>
> First
> Currently in Metron Mpack es.ip in Metron configuration is set to: es_url.
> Which is incorrect and will fail, instead it should be replaced with:
> "es.ip": "",
> "es.port": "9300",
> But I do not know how to get elasticsearch_master_hostname. Also, value of
> es.port seems like can be defaulted to es_binary_port or even omitted as it
> is already specified in es_binary_port.
> This should be chanted in:
> https://github.com/apache/incubator-metron/blob/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml
> On line 208.
> First
> Second
> In file
> https://github.com/apache/incubator-metron/blob/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/configuration/elastic-sysconfig.xml
> On lines 93 and 95 symbols "/" is forgotten. e.g:
> ES_JAVA_OPTS="-verbose:gc -Xloggc:{{log_dir}}/elasticsearch_gc.log
> -XX:-CMSConcurrentMTEnabled
> should be:
> ES_JAVA_OPTS="-verbose:gc -Xloggc:{{log_dir}}/elasticsearch_gc.log
> -XX:-CMSConcurrentMTEnabled
> Second
> Third
> After ES install it will fail to start with error"
> {code}
> Likely root cause: java.nio.file.AccessDeniedException:
> /etc/elasticsearch/scripts
> {code}
> The reason for the is root permission on these files:
> {code}
> # ls -la /etc/elasticsearch
> ...
> -rwxr-x---. 1 root elasticsearch 2571 May 12 09:24 logging.yml
> drwxr-x---. 2 root elasticsearch 4096 May 17 11:49 scripts
> {code}
> I was not able to trace down origin of these files, so do not know how to fix
> it.
> Third
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (METRON-641) Fix Kibana install file
[
https://issues.apache.org/jira/browse/METRON-641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Casey Stella updated METRON-641:
Fix Version/s: 0.3.1
> Fix Kibana install file
> ---
>
> Key: METRON-641
> URL: https://issues.apache.org/jira/browse/METRON-641
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.3.0
>Reporter: Dima Kovalyov
>Priority: Minor
> Fix For: 0.3.1
>
>
> Kibana installed as part of Metron mpack winthin Ambari will fail during
> start with following error:
> {code}
> ValueError: zero length field name in format
> {code}
> We can fix it with:
> {code}
> sed -i 's@{}/kibana@{0}/kibana@g'
> /incubator-metron/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/KIBANA/4.5.1/package/scripts/kibana_master.py
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (METRON-608) Mpack to install a single-node test cluster
[ https://issues.apache.org/jira/browse/METRON-608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-608: Fix Version/s: (was: Next + 1) 0.3.1 > Mpack to install a single-node test cluster > --- > > Key: METRON-608 > URL: https://issues.apache.org/jira/browse/METRON-608 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.0 > Environment: Linux, Ambari installation >Reporter: Matt Foley >Assignee: Matt Foley > Fix For: 0.3.1 > > > The current Mpack for Ambari install of Metron fails to correctly install > Elasticsearch if restricted to a single-node cluster. Yet a single-node > install of Elasticsearch is certainly feasible, as shown by our quick-dev > environment. > This is a short-term fix by providing a completely separate Mpack just for > the single-node scenario. I'm also opening METRON-609 to enhance the > existing Mpack to handle the single-node and small-number-of-nodes scenario, > but that one will require much deeper testing and is likely to take a while > to complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-635) Vagrant provisioning fails on CentOS hosts
[ https://issues.apache.org/jira/browse/METRON-635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-635: Fix Version/s: 0.3.1 > Vagrant provisioning fails on CentOS hosts > -- > > Key: METRON-635 > URL: https://issues.apache.org/jira/browse/METRON-635 > Project: Metron > Issue Type: Bug >Affects Versions: 0.3.0 >Reporter: Jon Zeolla >Assignee: Jon Zeolla > Fix For: 0.3.1 > > > Vagrant provisioning fails on CentOS hosts with an sftp error message. > Adding `scp_if_ssh = True` in ansible.cfg fixes things. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-670) Monit Incorrectly Reports Status
[ https://issues.apache.org/jira/browse/METRON-670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-670: Assignee: Nick Allen (was: Casey Stella) > Monit Incorrectly Reports Status > > > Key: METRON-670 > URL: https://issues.apache.org/jira/browse/METRON-670 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen > Fix For: 0.3.1 > > > In a constrained environment, like 'Quick Dev', Monit will often incorrectly > report the status of a Metron topology. This occurs when the environment is > under load and a query of topology status exceeds the default timeout of 30 > seconds. Need to add a parameter so that the timeout for a status check can > be extended under these conditions. This was previously done for starting > and stopping a topology, but not for a status checks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-670) Monit Incorrectly Reports Status
[ https://issues.apache.org/jira/browse/METRON-670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-670: Fix Version/s: 0.3.1 > Monit Incorrectly Reports Status > > > Key: METRON-670 > URL: https://issues.apache.org/jira/browse/METRON-670 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen > Fix For: 0.3.1 > > > In a constrained environment, like 'Quick Dev', Monit will often incorrectly > report the status of a Metron topology. This occurs when the environment is > under load and a query of topology status exceeds the default timeout of 30 > seconds. Need to add a parameter so that the timeout for a status check can > be extended under these conditions. This was previously done for starting > and stopping a topology, but not for a status checks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-270) Add Zeppelin to the platform
[ https://issues.apache.org/jira/browse/METRON-270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-270: Fix Version/s: 0.3.1 > Add Zeppelin to the platform > > > Key: METRON-270 > URL: https://issues.apache.org/jira/browse/METRON-270 > Project: Metron > Issue Type: New Feature >Reporter: James Sirota >Assignee: Justin Leet > Labels: METRON_ML > Fix For: 0.3.1 > > > I propose adding Zeppelin to the platform to aid in interactive dashboarding > and data visualizations -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-609) Enhance Mpack to handle single-node and small-cluster installs of Elasticsearch
[ https://issues.apache.org/jira/browse/METRON-609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-609: Fix Version/s: 0.3.1 > Enhance Mpack to handle single-node and small-cluster installs of > Elasticsearch > --- > > Key: METRON-609 > URL: https://issues.apache.org/jira/browse/METRON-609 > Project: Metron > Issue Type: Improvement >Reporter: Matt Foley >Assignee: Matt Foley > Fix For: 0.3.1 > > > The current Mpack for Ambari install of Metron requires that Elasticsearch be > installed with 1+ dedicated Masters and 3+ dedicated Slaves. It does not > support combined master/datanode configuration (non-"dedicated" Master), > which is the recommended way to run a single-node configuration of > Elasticsearch; nor does it allow less than 3 dedicated Slaves, thereby > requiring the cluster have at least 4 nodes. > This jira proposes to enable 1-, 2-, and 3-node clusters to have a working > Elasticsearch install via the Mpack. It will also allow non-dedicated > master/datanodes, which are required for single-node and useful for few-node > clusters. > I intend to also include the following enhancements and fixes: > * Determine whether elastic-sysconfig:data_dir and elastic-site:path_data > should have same default value and if so fix it. (I think they should, and > they don't. Probably there should only be one variable instead of two.) > * Provide Quick Links in Ambari service page for Elasticsearch to the > self-report pages for Elasticsearch health, node list, and other "_cat" > status. May include some "_cluster" status. > * Improve various mouse-over Description fields in the GUI > * Improve the order of fields in the GUI, to group at the beginning the > fields that must be modified or are commonly modified. Possibly separate > "basic" and "advanced" configs. > * Provide a README.md that describes what is going on with the various > resource files in the Mpack src > * Enhance the wiki page for cluster installation, with regard to single-node > install with Mpack. > * Database-preserving upgrade from version 1.0.0.0. > Also see > https://github.com/apache/incubator-metron/tree/master/metron-deployment#current-limitations > for some other known issues. And the documentation in that README.md page > is significantly out of date wrt the Mpack. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-609) Enhance Mpack to handle single-node and small-cluster installs of Elasticsearch
[ https://issues.apache.org/jira/browse/METRON-609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-609: Assignee: Matt Foley (was: Casey Stella) > Enhance Mpack to handle single-node and small-cluster installs of > Elasticsearch > --- > > Key: METRON-609 > URL: https://issues.apache.org/jira/browse/METRON-609 > Project: Metron > Issue Type: Improvement >Reporter: Matt Foley >Assignee: Matt Foley > > The current Mpack for Ambari install of Metron requires that Elasticsearch be > installed with 1+ dedicated Masters and 3+ dedicated Slaves. It does not > support combined master/datanode configuration (non-"dedicated" Master), > which is the recommended way to run a single-node configuration of > Elasticsearch; nor does it allow less than 3 dedicated Slaves, thereby > requiring the cluster have at least 4 nodes. > This jira proposes to enable 1-, 2-, and 3-node clusters to have a working > Elasticsearch install via the Mpack. It will also allow non-dedicated > master/datanodes, which are required for single-node and useful for few-node > clusters. > I intend to also include the following enhancements and fixes: > * Determine whether elastic-sysconfig:data_dir and elastic-site:path_data > should have same default value and if so fix it. (I think they should, and > they don't. Probably there should only be one variable instead of two.) > * Provide Quick Links in Ambari service page for Elasticsearch to the > self-report pages for Elasticsearch health, node list, and other "_cat" > status. May include some "_cluster" status. > * Improve various mouse-over Description fields in the GUI > * Improve the order of fields in the GUI, to group at the beginning the > fields that must be modified or are commonly modified. Possibly separate > "basic" and "advanced" configs. > * Provide a README.md that describes what is going on with the various > resource files in the Mpack src > * Enhance the wiki page for cluster installation, with regard to single-node > install with Mpack. > * Database-preserving upgrade from version 1.0.0.0. > Also see > https://github.com/apache/incubator-metron/tree/master/metron-deployment#current-limitations > for some other known issues. And the documentation in that README.md page > is significantly out of date wrt the Mpack. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-676) Create Zeppelin Notebook for YAF Telemetry
[ https://issues.apache.org/jira/browse/METRON-676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-676: Assignee: Nick Allen (was: Casey Stella) > Create Zeppelin Notebook for YAF Telemetry > -- > > Key: METRON-676 > URL: https://issues.apache.org/jira/browse/METRON-676 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Assignee: Nick Allen > Fix For: 0.3.1 > > Attachments: Metron - YAF Telemetry.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-675) Make Threat Triage rules able to be assigned names and comments
[ https://issues.apache.org/jira/browse/METRON-675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-675: Fix Version/s: 0.3.1 > Make Threat Triage rules able to be assigned names and comments > --- > > Key: METRON-675 > URL: https://issues.apache.org/jira/browse/METRON-675 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella > Fix For: 0.3.1 > > > There may be many, many threat triage rules. To help organize these, we > should make them slightly more complex than a simple key/value as we have it > now. We should add optional name and optional comment fields. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-676) Create Zeppelin Notebook for YAF Telemetry
[ https://issues.apache.org/jira/browse/METRON-676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-676: Fix Version/s: 0.3.1 > Create Zeppelin Notebook for YAF Telemetry > -- > > Key: METRON-676 > URL: https://issues.apache.org/jira/browse/METRON-676 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Assignee: Nick Allen > Fix For: 0.3.1 > > Attachments: Metron - YAF Telemetry.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-650) Remove Kraken Maven dependency on opensoc github repo
[
https://issues.apache.org/jira/browse/METRON-650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Casey Stella updated METRON-650:
Fix Version/s: 0.3.1
> Remove Kraken Maven dependency on opensoc github repo
> -
>
> Key: METRON-650
> URL: https://issues.apache.org/jira/browse/METRON-650
> Project: Metron
> Issue Type: Improvement
>Reporter: Michael Miklavcic
> Fix For: 0.3.1
>
>
> Metron common's pom is referencing the opensoc repo to get Kraken:
>
>
> Metron-Kraken-Repo
> Metron Kraken Repository
> https://raw.github.com/opensoc/kraken/mvn-repo
>
>
> ...
>
> org.krakenapps
> kraken-pcap
> ${global_pcap_version}
>
>
> slf4j-api
> org.slf4j
>
>
> slf4j-simple
> org.slf4j
>
>
>
> This library does not currently reside in mvn central (searched by package
> and libname) and should instead be staged in Apache with Metron.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (METRON-283) Migrate Geo Enrichment outside of MySQL
[ https://issues.apache.org/jira/browse/METRON-283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-283: Fix Version/s: 0.3.1 > Migrate Geo Enrichment outside of MySQL > --- > > Key: METRON-283 > URL: https://issues.apache.org/jira/browse/METRON-283 > Project: Metron > Issue Type: Improvement >Reporter: James Sirota >Assignee: Justin Leet >Priority: Minor > Fix For: 0.3.1 > > > We need to migrate our enrichment SQL store from MySQL to Phoenix or some > other SQL on Hbase library. Or alternatively come up with a way to do this > without using SQL. This way we don't have a dependency on MySQL and there is > one less thing that we need to install on our platform -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-668) Remove the "tickUpdate" profile config and make the "init" phase not reset variables
[
https://issues.apache.org/jira/browse/METRON-668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Casey Stella updated METRON-668:
Fix Version/s: 0.3.1
> Remove the "tickUpdate" profile config and make the "init" phase not reset
> variables
>
>
> Key: METRON-668
> URL: https://issues.apache.org/jira/browse/METRON-668
> Project: Metron
> Issue Type: Improvement
>Reporter: Casey Stella
>Assignee: Casey Stella
> Fix For: 0.3.1
>
>
> Originally during work on the MAD outlier work, I conceived of a need for a
> new callback in the profile configuration, "tickUpdate" that ran at the tick
> and had the variables from the tick that just completed available to it.
> This was done so that I could merge the state accumulated in the tick with a
> lookback window of state for MAD. The problem is that the "init" phase
> happens after this and blows away the changes done in "tickUpdate", so it
> never worked like intended.
> It occurs to me that what we really want is not to have two separate config
> phases, but only one, "init" and to not reset the variables on the tick for
> the profile. You can, of course, choose to update them by overwriting them
> in the "init" phase *or* you can choose to use them as part of your init.
> For context, this would make the example for MAD:
> {code:javascript}
> {
> "profiles": [
> {
> "profile": "sketchy_mad",
> "foreach": "'global'",
> "onlyif": "true",
> "init" : {
> "s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad',
> 'global', 5, 'MINUTES'))"
>},
> "tickUpdate": {
> "s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad',
> 'global', 5, 'MINUTES'), s)"
> },
> "update": {
> "s": "OUTLIER_MAD_ADD(s, value)"
> },
> "result": "s"
> }
> ]
> }
> {code}
> is functionally equivalent to
> {code:javascript}
> {
> "profiles": [
> {
> "profile": "sketchy_mad",
> "foreach": "'global'",
> "onlyif": "true",
> "init" : {
> "s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad',
> 'global', 5, 'MINUTES'))"
>},
> "update": {
> "s": "OUTLIER_MAD_ADD(s, value)"
> },
> "result": "s"
> }
> ]
> }
> {code}
> This resets the MAD state to the last 5 minute window. If we did NOT reset
> the state and keep accumulating state (provided we did not clear the
> variables on init, we could do the following:
> {code:javascript}
> {
> "profiles": [
> {
> "profile": "sketchy_mad",
> "foreach": "'global'",
> "onlyif": "true",
> "init" : {
> "s": "if exists(s) then s else
> OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad',
> 'global', 5, 'MINUTES'))"
>},
> "update": {
> "s": "OUTLIER_MAD_ADD(s, value)"
> },
> "result": "s"
> }
> ]
> }
> {code}
> s would get initialized sensibly and then always accumulate as long as the
> topology continued (rather than having a fixed lookback).
> In short, making init to not reset the variables shouldn't cause any harm and
> should provide another set of use-cases for the profiler. Also, tickUpdate
> has no function whatsoever and should be removed because it gets overwritten
> by init directly after being called.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (METRON-672) SolrIndexingIntegrationTest fails intermittently
[
https://issues.apache.org/jira/browse/METRON-672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Casey Stella updated METRON-672:
Fix Version/s: 0.3.1
> SolrIndexingIntegrationTest fails intermittently
>
>
> Key: METRON-672
> URL: https://issues.apache.org/jira/browse/METRON-672
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.3.0
>Reporter: Justin Leet
>Assignee: Casey Stella
> Fix For: 0.3.1
>
>
> Adapted from a dev list conversation
> h4. Initial Error in Travis
> Jon noted this in the Travis builds
> {code}
> `test(org.apache.metron.solr.integration.SolrIndexingIntegrationTest):
> Took too long to complete: 150582 > 15`, more details below:
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 166.167 sec <<< FAILURE!
> test(org.apache.metron.solr.integration.SolrIndexingIntegrationTest)
> Time elapsed: 166.071 sec <<< ERROR!
> {code}
> h4. Additional Notes
> Casey was able to reproduce this locally (but not in his IDE). Couple details
> in the dev list excerpt.
> Fixing this should ideally include adding more detailed logging to hopefully
> avoid these issues in the future. As a note, unfortunately in this case,
> Casey notes that logging seems to make this issue rarer. Still, logging to
> be able to understand the flow (and tuning logging level as appropriate)
> would help resolve issues in the future.
> h4. Dev List Excerpt
> Per Casey:
> {quote}
> Ok, so now I'm concerned that this isn't a fluke. Here's an excerpt from
> the failing logs on travis for my PR with substantially longer timeouts (
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/194575474/log.txt)
> {code}
> Running org.apache.metron.solr.integration.SolrIndexingIntegrationTest
> 0 vs 10 vs 0
> Processed
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Processed
> target/indexingIntegrationTest/hdfs/test/enrichment-null-0-0-1485200689038.json
> 10 vs 10 vs 6
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 317.056 sec <<< FAILURE!
> test(org.apache.metron.solr.integration.SolrIndexingIntegrationTest)
> Time elapsed: 316.949 sec <<< ERROR!
> java.lang.RuntimeException: Took too long to complete: 300783 > 30
> at
> org.apache.metron.integration.ComponentRunner.process(ComponentRunner.java:131)
> at
> org.apache.metron.indexing.integration.IndexingIntegrationTest.test(IndexingIntegrationTest.java:173)
> {code}
> I'm getting the impression that this isn't the timeout and we have a
> mystery on our hands. Each of those lines "10 vs 10 vs 6" happen 15
> seconds apart. That line means that it read 10 entries from kafka, 10
> entries from the indexed data and 6 entries from HDFS. It's that 6
> entries that is the problem. Also of note, this does not seem to
> happen to me locally AND it's not consistent on Travis. Given all
> that I'd say that it's
[jira] [Updated] (METRON-666) Fix javadoc doclint errors
[ https://issues.apache.org/jira/browse/METRON-666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-666: Fix Version/s: 0.3.1 > Fix javadoc doclint errors > -- > > Key: METRON-666 > URL: https://issues.apache.org/jira/browse/METRON-666 > Project: Metron > Issue Type: Bug >Affects Versions: 0.3.0 >Reporter: Matt Foley >Assignee: Matt Foley > Fix For: 0.3.1 > > > Java 8 includes "doclint" as part of javadocs. As a result, running javadoc > on current code base has fatal errors, mostly (not all) related to use of > "" (not allowed ever), or unmatched "" (not preceded by a matching > ""). It is, however, happy with unmatched "", so that's the thing to > use for paragraph separators. Put it on the same line as the next line of > text to avoid a warning about "empty ". > There are other errors fixed here too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-659) Emulate Sensors in Development Environments
[ https://issues.apache.org/jira/browse/METRON-659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-659: Fix Version/s: (was: Next + 1) 0.3.1 > Emulate Sensors in Development Environments > --- > > Key: METRON-659 > URL: https://issues.apache.org/jira/browse/METRON-659 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.0 >Reporter: Nick Allen >Assignee: Nick Allen > Fix For: 0.3.1 > > > Replace the Snort, Bro, and YAF sensors on the "Quick Dev" and "Full Dev" > environments with a mechanism that consumes less resources. > These environments are notoriously difficult to work with because the > installed services consume nearly all available memory. The Bro, YAF, and > Snort sensors, along with the PCAP Replay service, consume a considerable > amount of these limited resources. > Replacing these sensors with a lightweight mechanism should free-up > additional resources, make the environment easier to work with, and result in > faster deployments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-664) Make the index configuration per-writer with enabled/disabled
[ https://issues.apache.org/jira/browse/METRON-664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-664: Fix Version/s: 0.3.1 > Make the index configuration per-writer with enabled/disabled > - > > Key: METRON-664 > URL: https://issues.apache.org/jira/browse/METRON-664 > Project: Metron > Issue Type: Sub-task >Reporter: Casey Stella > Fix For: 0.3.1 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-600) Fix Metron Website
[ https://issues.apache.org/jira/browse/METRON-600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-600: Fix Version/s: 0.3.1 > Fix Metron Website > -- > > Key: METRON-600 > URL: https://issues.apache.org/jira/browse/METRON-600 > Project: Metron > Issue Type: Improvement >Reporter: James Sirota >Assignee: Ryan Merriman > Fix For: 0.3.1 > > > h3. Issue 1 > Podling web sites MUST include a clear disclaimer on their website and in all > documentation (including releases) stating that they are in incubation. > Podlings SHOULD use the following text for all disclaimers (replace the > underlined phrases as appropriate): > Apache Podling-Name is an effort undergoing incubation at The Apache Software > Foundation (ASF), sponsored by the name of Apache TLP sponsor. Incubation is > required of all newly accepted projects until a further review indicates that > the infrastructure, communications, and decision making process have > stabilized in a manner consistent with other successful ASF projects. While > incubation status is not necessarily a reflection of the completeness or > stability of the code, it does indicate that the project has yet to be fully > endorsed by the ASF. > h3. Issue 2 > Podlings websites SHOULD contain the Apache Incubator Project logo as sign of > affiliation > Apache Project Web Sites typically include several standard pages. Each page > is formatted with a navigation bar on the left and a project standard header > that includes the Incubator graphic. > [We need to make the Logo more prominent and move towards the top of the page > rather than having it on the bottom like we do] > h3. Issue 3 > The sources for every podling site sources should be maintained in the > podling's site SVN or git directory > h3. Issue 4 > Previous statement of the problem: > bq. A downloads page needs to be created with links per release. The link to > the artifact needs to be using the mirror site for apache. For example, the > 0.3.0 release would be > http://www.apache.org/dyn/closer.lua/incubator/metron/0.3.0/apache-metron-0.3.0-incubating.tar.gz. > The MD5, SHA and Signature can be from the apache release site. Look at > the storm page as an example: http://storm.apache.org/downloads.html (Note > https://maven.apache.org/download.cgi is a better example because it allows > changing the mirror if needed.) > A separate Downloads page is needed, rather than the current button, in order > to satisfy these three requirements of > http://www.apache.org/dev/release-download-pages.html : > * All links to the downloadable distribution artifacts MUST NOT reference the > main Apache dist web site (dist.apache.org). Instead, they should use the > standard scripting mechanisms to distribute the load between the mirror sites > (see http://www.apache.org/dyn/closer.cgi/ ). > * All links to checksums, detached signatures and public keys MUST reference > the main Apache dist web site (dist.apache.org), and use https (SSL). > * The site SHOULD provide clear and easy links to the public keys, checksums > and detached signatures from the download release page, and include a > reminder text with links to more information for users. > Detailed discussion and examples are provided in [Comment > 15723335|https://issues.apache.org/jira/browse/METRON-600?focusedCommentId=15723335] > below. > h3. Issue 5 > [Lets try to conform as much as possible to the following suggested template] > Project Home Page: the primary entry point to the site; contains project > description, news, invitation to join the project. > [We have this, great] > License Page: usually, the Apache License 2.0 > [We don't have this, we should probably put it under the about page] > Downloads: many projects in incubation will release code, and this page > describes them and has links to the download pages that redirect to Apache > Mirror sites. > [We have this, great] > Documentation: this page describes the project documentation, including > javadoc for Java projects; guides, tutorials, and links to external > documentation. > [We should probably just link to the wiki so we don't have to maintain this > in two places] > Committers: a list of current committers on the project. > [We need to update this from our status page that can be found here. Need to > make sure both are consistent. > http://incubator.apache.org/projects/metron.html > ] > Mailing Lists: there are several mailing lists that the community might be > interested in, and this page contains mailto: links that allow easy > subscription (and unsubscription) to any of them. > [We should probably put this under our community page and also link to the > apache status page to make sure it's consistent] > FAQ: frequently asked questions are answered here. > [We p
[jira] [Updated] (METRON-654) Create RPM Installer for Profiler
[ https://issues.apache.org/jira/browse/METRON-654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-654: Fix Version/s: (was: Next + 1) 0.3.1 > Create RPM Installer for Profiler > - > > Key: METRON-654 > URL: https://issues.apache.org/jira/browse/METRON-654 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.0 >Reporter: Nick Allen >Assignee: Nick Allen > Fix For: 0.3.1 > > > The Profiler currently must be installed using the Ansible deployment > scripts. Prior to getting the Profiler integrated with the rest of the > Ambari installation process, we need to package the Profiler in an RPM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-532) Define Profile Period When Calling PROFILE_GET
[ https://issues.apache.org/jira/browse/METRON-532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-532: Fix Version/s: (was: 0.3.0) 0.3.1 > Define Profile Period When Calling PROFILE_GET > -- > > Key: METRON-532 > URL: https://issues.apache.org/jira/browse/METRON-532 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Matt Foley > Fix For: 0.3.1 > > > The Profiler Client currently offers the PROFILE_GET Stellar function to > access profile data. The work done for METRON-529 allowed the user to > customize the profile period using Metron global properties. > A user may need to access historical profiles with different durations and > would want to specify the period as part of the call to the Profiler Client, > rather than in the Metron global properties. > This would be especially necessary should METRON-530 be completed allowing > different profiles to use different period durations simultaneously. > There is some discussion of this attached to METRON-529. > See also discussion in METRON-594 "Replay Telemetry Data through Profiler". > Note: At suggestion of [~cestella], scope of this work item was expanded to > include checking for config changes at run time, in the PROFILE_GET Stellar > function. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-652) Extract indexing config from enrichment config
[ https://issues.apache.org/jira/browse/METRON-652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-652: Fix Version/s: 0.3.1 > Extract indexing config from enrichment config > -- > > Key: METRON-652 > URL: https://issues.apache.org/jira/browse/METRON-652 > Project: Metron > Issue Type: Sub-task >Reporter: Casey Stella >Assignee: Casey Stella > Fix For: 0.3.1 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-656) Make Stellar 'in' closer to functioning like python
[
https://issues.apache.org/jira/browse/METRON-656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Casey Stella updated METRON-656:
Fix Version/s: 0.3.1
> Make Stellar 'in' closer to functioning like python
> ---
>
> Key: METRON-656
> URL: https://issues.apache.org/jira/browse/METRON-656
> Project: Metron
> Issue Type: New Feature
>Reporter: Casey Stella
>Assignee: Casey Stella
> Fix For: 0.3.1
>
>
> We have an 'in' operator in stellar, but it could be much better. This should
> bring it at parity with the 'in' operator in python
> * in should support string contains e.g. 'foo' in 'foobar'
> * in should support Collection contains e.g. 'foo' in [ 'foo', 'bar' ].
> Legacy was to only support lists
> * in should support map contains e.g. 'foo' in { 'foo' : 5 }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (METRON-624) Comparison Operators Evaluate Incorrectly
[
https://issues.apache.org/jira/browse/METRON-624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Casey Stella updated METRON-624:
Fix Version/s: 0.3.1
> Comparison Operators Evaluate Incorrectly
> -
>
> Key: METRON-624
> URL: https://issues.apache.org/jira/browse/METRON-624
> Project: Metron
> Issue Type: Bug
>Reporter: Josh Meyer
> Labels: stellar
> Fix For: 0.3.1
>
>
> Currently there is an issue with the way Stellar interprets comparison
> expressions. Currently it only compares numbers when both sides are numbers
> otherwise it converts both sides of the expression to a value and then
> compares them. Also, when looking at numbers it always gets double values to
> compare.
> Below is an example of a failing test that should pass.
> {code:theme=FadeToGrey|linenumbers=true|language=java|firstline=0001|collapse=false}
> @Test
> public void compareNumberAndStringWithSameValueShouldFail() throws
> Exception {
> final Map variableMap = new HashMap<>();
> assertFalse(runPredicate("1 == '1'", variableMap::get));
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
