[GitHub] metron pull request #1097: METRON-1656 Create KAKFA_SEEK function
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1097 ---
[jira] [Updated] (METRON-1656) Create KAKFA_SEEK function
[ https://issues.apache.org/jira/browse/METRON-1656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-1656: --- Fix Version/s: Next + 1 > Create KAKFA_SEEK function > -- > > Key: METRON-1656 > URL: https://issues.apache.org/jira/browse/METRON-1656 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: Next + 1 > > > A Stellar function to be used in the REPL that allows you to seek to a > specific offset within a Kafka topic. This complements the existing Kafka > functions like KAFKA_GET, KAFKA_TAIL, and KAFKA_FIND. Like those functions > this supports the "rich" message view. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1646) Sensor Stubs should work when kerberized
[ https://issues.apache.org/jira/browse/METRON-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-1646: --- Fix Version/s: Next + 1 > Sensor Stubs should work when kerberized > > > Key: METRON-1646 > URL: https://issues.apache.org/jira/browse/METRON-1646 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: Next + 1 > > > The sensor-stubs do not work in a kerberized development environment. This > makes it difficult to test issues related to kerberization. Enhancing > sensor-stubs to work in a kerberized environment will make testing in the > development environment under kerberization much simpler. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1652) Document X-Pack Common Problem
[ https://issues.apache.org/jira/browse/METRON-1652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-1652: --- Fix Version/s: Next + 1 > Document X-Pack Common Problem > -- > > Key: METRON-1652 > URL: https://issues.apache.org/jira/browse/METRON-1652 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Minor > Fix For: Next + 1 > > > Improvements to the Elasticsearch X-Pack documentation to document a common > problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1649) Intermittent Test Failure ProfileBuilderBoltTest#testFlushExpiredProfiles
[
https://issues.apache.org/jira/browse/METRON-1649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Allen updated METRON-1649:
---
Fix Version/s: Next + 1
> Intermittent Test Failure ProfileBuilderBoltTest#testFlushExpiredProfiles
> -
>
> Key: METRON-1649
> URL: https://issues.apache.org/jira/browse/METRON-1649
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
> Fix For: Next + 1
>
>
> {code}
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.412 sec <<<
> FAILURE! - in org.apache.metron.profiler.bolt.ProfileBuilderBoltTest
> testFlushExpiredProfiles(org.apache.metron.profiler.bolt.ProfileBuilderBoltTest)
> Time elapsed: 0.026 sec <<< FAILURE!
> org.mockito.exceptions.verification.TooManyActualInvocations:
> outputCollector.emit(
> "hbase",
>
> );
> Wanted 1 time:
> -> at
> org.apache.metron.profiler.bolt.ProfileBuilderBoltTest.getProfileMeasurements(ProfileBuilderBoltTest.java:265)
> But was 2 times. Undesired invocation:
> -> at org.apache.metron.profiler.bolt.HBaseEmitter.emit(HBaseEmitter.java:54)
> at
> org.apache.metron.profiler.bolt.ProfileBuilderBoltTest.getProfileMeasurements(ProfileBuilderBoltTest.java:265)
> at
> org.apache.metron.profiler.bolt.ProfileBuilderBoltTest.testFlushExpiredProfiles(ProfileBuilderBoltTest.java:205)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1609) Elasticsearch settings in Ambari should not be required if Solr is the indexer
[ https://issues.apache.org/jira/browse/METRON-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-1609: --- Fix Version/s: Next + 1 > Elasticsearch settings in Ambari should not be required if Solr is the indexer > -- > > Key: METRON-1609 > URL: https://issues.apache.org/jira/browse/METRON-1609 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: Next + 1 > > Attachments: Screen Shot 2018-06-06 at 12.34.13 PM.png, Screen Shot > 2018-06-06 at 12.37.08 PM.png > > > When a user deploys Metron with Solr, the Mpack requires that the user > populate the Index Settings > Elasticsearch Hosts field to continue, even > when deploying a Metron cluster that indexes to Solr. > The other Elasticsearch specific fields on the Index Settings page are also > required, although defaults are provided and thus the user is not required to > enter anything here. If you remove the provided defaults, you can see that > each of the fields are required, even when Elasticsearch will not be used. > * The Elasticsearch related properties under the Ambari 'Index settings' tab > should only be required if Elasticsearch is chosen as the indexer. > * The Solr related properties under the 'Index Settings' tab should only be > required if Solr is chosen as the indexer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1622) Allow user to define global property 'threat.triage.score.field' in Ambari
[ https://issues.apache.org/jira/browse/METRON-1622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-1622: --- Fix Version/s: Next + 1 > Allow user to define global property 'threat.triage.score.field' in Ambari > --- > > Key: METRON-1622 > URL: https://issues.apache.org/jira/browse/METRON-1622 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: Next + 1 > > > Based on METRON-1608 and METRON-1617, the user can specify the name of the > field containing the threat triage score in the global configuration. This is > used by the Alerts UI. > Currently a user can only change this value using the CLI. This property > should be exposed to the user via Ambari. The user should be able to define > this property directly in Ambari. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1599) Allow user to define global property 'source.type.field' in Ambari
[ https://issues.apache.org/jira/browse/METRON-1599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-1599: --- Fix Version/s: Next + 1 > Allow user to define global property 'source.type.field' in Ambari > --- > > Key: METRON-1599 > URL: https://issues.apache.org/jira/browse/METRON-1599 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: Next + 1 > > > The user can specify the name of the message field containing the source type > in the global configuration. This is used by the Alerts UI. > Currently this necessitates a user using the CLI to add the field. This > property should be exposed to the user via Ambari. The user should be able > to define this property directly in Ambari. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1598) NoClassDefFoundError when running with Elasticsearch X-Pack
[
https://issues.apache.org/jira/browse/METRON-1598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Allen updated METRON-1598:
---
Fix Version/s: Next + 1
> NoClassDefFoundError when running with Elasticsearch X-Pack
> ---
>
> Key: METRON-1598
> URL: https://issues.apache.org/jira/browse/METRON-1598
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.2
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
> Fix For: Next + 1
>
>
> When running on Elasticsearch with X-Pack enabled, the following exception is
> thrown in the Metron REST logs.
> {code}
> 2018-05-23T07:43:16.000 ERROR
> [org.apache.metron.elasticsearch.utils.ElasticsearchUtils] - Failed to
> convert search request to JSON java.lang.NoClassDefFoundError:
> com/fasterxml/jackson/dataformat/smile/SmileParser at
> org.elasticsearch.common.xcontent.XContentFactory.contentBuilder(XContentFactory.java:123)
> at
> org.elasticsearch.action.support.ToXContentToBytes.buildAsBytes(ToXContentToBytes.java:62)
> at
> org.elasticsearch.action.support.ToXContentToBytes.buildAsBytes(ToXContentToBytes.java:52)
> at
> org.apache.metron.elasticsearch.utils.ElasticsearchUtils.toJSON(ElasticsearchUtils.java:277)
> at
> org.apache.metron.elasticsearch.dao.ElasticsearchRequestSubmitter.submitSearch(ElasticsearchRequestSubmitter.java:58)
> at
> org.apache.metron.elasticsearch.dao.ElasticsearchDao.search(ElasticsearchDao.java:172)
> at
> org.apache.metron.elasticsearch.dao.ElasticsearchMetaAlertDao.search(ElasticsearchMetaAlertDao.java:403)
> at
> org.apache.metron.rest.service.impl.SearchServiceImpl.search(SearchServiceImpl.java:83)
> at
> org.apache.metron.rest.controller.SearchController.search(SearchController.java:54)
> {code}
> This does not happen with all queries, just some queries. For example, the
> following query completes successfully.
> {code}
> {
> "indices" : [],
> "query" : "*",
> "size" : 100,
> "from" : 0,
> "sort" : [ \{ "field" : "timestamp", "sortOrder" : "DESC" } ],
> "fields" : null,
> "facetFields" : null
> }
> {code}
> On the other hand, this query fails with the exception shown above.
> {code}
> {
> "indices" : [],
> "query" : "*",
> "size" : 100,
> "from" : 0,
> "sort" : [ \{ "field" : "timestamp", "sortOrder" : "DESC" } ],
> "fields" : null,
> "facetFields" : []
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1531) Web Start Scripts Incorrectly Report OK If Process Dies
[ https://issues.apache.org/jira/browse/METRON-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540239#comment-16540239 ] Nick Allen commented on METRON-1531: This PR was closed without being merged and will not be fixed. > Web Start Scripts Incorrectly Report OK If Process Dies > --- > > Key: METRON-1531 > URL: https://issues.apache.org/jira/browse/METRON-1531 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Labels: won't-fix > > The start scripts for the Alerts and Management UIs will incorrectly report > OK when running 'start', if the previously started process has died or been > killed. This is misleading. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1662) PCAP UI: Downloading PCAP file
[ https://issues.apache.org/jira/browse/METRON-1662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tibor Meller updated METRON-1662: - Summary: PCAP UI: Downloading PCAP file (was: PCAP UI: Downloading PDML) > PCAP UI: Downloading PCAP file > -- > > Key: METRON-1662 > URL: https://issues.apache.org/jira/browse/METRON-1662 > Project: Metron > Issue Type: New Feature >Reporter: Tibor Meller >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1656) Create KAKFA_SEEK function
[ https://issues.apache.org/jira/browse/METRON-1656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540224#comment-16540224 ] ASF GitHub Bot commented on METRON-1656: Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1097 > Create KAKFA_SEEK function > -- > > Key: METRON-1656 > URL: https://issues.apache.org/jira/browse/METRON-1656 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > > A Stellar function to be used in the REPL that allows you to seek to a > specific offset within a Kafka topic. This complements the existing Kafka > functions like KAFKA_GET, KAFKA_TAIL, and KAFKA_FIND. Like those functions > this supports the "rich" message view. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1592) Unable to use third party parser with Storm versions >= 1.1.0
[
https://issues.apache.org/jira/browse/METRON-1592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Allen updated METRON-1592:
---
Fix Version/s: Next + 1
> Unable to use third party parser with Storm versions >= 1.1.0
> -
>
> Key: METRON-1592
> URL: https://issues.apache.org/jira/browse/METRON-1592
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
> Fix For: Next + 1
>
>
> When launching a third party parser using
> $METRON_HOME/bin/start_parser_topology.sh in HDP 2.6, Storm 1.1.0 the
> following exception is thrown.
>
> {code:java}
> ClassNotFoundException:
> org.apache.metron.parsers.topology.MergeAndShadeTransformer{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1553) Validate JIRA Script Error
[
https://issues.apache.org/jira/browse/METRON-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Allen updated METRON-1553:
---
Fix Version/s: Next + 1
> Validate JIRA Script Error
> --
>
> Key: METRON-1553
> URL: https://issues.apache.org/jira/browse/METRON-1553
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
> Fix For: Next + 1
>
>
> {code:java}
> $ ./validate-jira-for-release --version="0.4.3"
> --start=tags/apache-metron-0.4.2-release
> Cloning into 'metron-0.4.3'...
> remote: Counting objects: 39871, done.
> remote: Compressing objects: 100% (13140/13140), done.
> remote: Total 39871 (delta 18078), reused 37550 (delta 17019)
> Receiving objects: 100% (39871/39871), 54.34 MiB | 6.23 MiB/s, done.
> Resolving deltas: 100% (18078/18078), done.
> fatal: Not a git repository (or any of the parent directories): .git
> Fetching origin
> JIRA STATUS FIX VERSION ASSIGNEE FIX
> METRON-1549 Done Michael Miklavcic
> https://issues.apache.org/jira/browse/METRON-1549
> METRON-1541 To Do Unassigned https://issues.apache.org/jira/browse/METRON-1541
> ...{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1197) Profiler topology fails to write profile data to hbase table on a kerberised cluster
[
https://issues.apache.org/jira/browse/METRON-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540244#comment-16540244
]
Nick Allen commented on METRON-1197:
Unable to replicate currently. May have been fixed indirectly by another
JIRA/PR.
> Profiler topology fails to write profile data to hbase table on a kerberised
> cluster
>
>
> Key: METRON-1197
> URL: https://issues.apache.org/jira/browse/METRON-1197
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.1
>Reporter: Mohan
>Assignee: Nick Allen
>Priority: Major
>
> Profiler fails to write the profile data to hbase table , I see
> 'javax.security.sasl.SaslException: GSS initiate failed' exception when the
> profiler tries to write the data to hbase table .
> I tried publishing below message to 'indexing' topic 5 times
> {code:java}
> { "ip_src_addr": "10.0.0.1", "protocol": "HTTPS", "length": "10", "bytes_in":
> 234, "timestamp": "1505909543000" }
> {code}
> below is the output for the console consumer for 'indexing' topic
> {code:java}
> [metron@nat-r7-fsvs-metron-1 bin]$ ./kafka-console-consumer.sh --zookeeper
> nat-r7-fsvs-metron-1.openstacklocal --topic indexing --security-protocol
> PLAINTEXTSASL
> {metadata.broker.list=nat-r7-fsvs-metron-3.openstacklocal:6667,nat-r7-fsvs-metron-7.openstacklocal:6667,nat-r7-fsvs-metron-9.openstacklocal:6667,nat-r7-fsvs-metron-11.openstacklocal:6667,nat-r7-fsvs-metron-8.openstacklocal:6667,nat-r7-fsvs-metron-5.openstacklocal:6667,nat-r7-fsvs-metron-12.openstacklocal:6667,nat-r7-fsvs-metron-1.openstacklocal:6667,nat-r7-fsvs-metron-6.openstacklocal:6667,nat-r7-fsvs-metron-2.openstacklocal:6667,nat-r7-fsvs-metron-10.openstacklocal:6667,
> request.timeout.ms=3, client.id=console-consumer-83924,
> security.protocol=PLAINTEXTSASL}
> { "ip_src_addr": "10.0.0.1", "protocol": "HTTPS", "length": "10", "bytes_in":
> 234, "timestamp": "1505909543000" }
> { "ip_src_addr": "10.0.0.1", "protocol": "HTTPS", "length": "10", "bytes_in":
> 234, "timestamp": "1505909543000" }
> { "ip_src_addr": "10.0.0.1", "protocol": "HTTPS", "length": "10", "bytes_in":
> 234, "timestamp": "1505909543000" }
> { "ip_src_addr": "10.0.0.1", "protocol": "HTTPS", "length": "10", "bytes_in":
> 234, "timestamp": "1505909543000" }
> { "ip_src_addr": "10.0.0.1", "protocol": "HTTPS", "length": "10", "bytes_in":
> 234, "timestamp": "1505909543000" }
> {"period.start":150591690,"period":1673241,"enrichmentsplitterbolt.splitter.end.ts":"1505917778229","profile":"calender-effects","enrichmentsplitterbolt.splitter.begin.ts":"1505917778228","is_alert":"true","source.type":"profiler","threatintelsplitterbolt.splitter.end.ts":"1505917778234","threatinteljoinbolt.joiner.ts":"1505917778237","enrichmentjoinbolt.joiner.ts":"1505917778232","period.end":150591780,"threatintelsplitterbolt.splitter.begin.ts":"1505917778234","entity":"10.0.0.1","timestamp":1505917778104}
> {"period.start":150591690,"period":1673241,"enrichmentsplitterbolt.splitter.end.ts":"1505917778230","profile":"source_ip_counter","enrichmentsplitterbolt.splitter.begin.ts":"1505917778230","is_alert":"true","source.type":"profiler","threatintelsplitterbolt.splitter.end.ts":"1505917778235","threatinteljoinbolt.joiner.ts":"1505917778238","enrichmentjoinbolt.joiner.ts":"1505917778233","period.end":150591780,"threatintelsplitterbolt.splitter.begin.ts":"1505917778235","entity":"10.0.0.1","timestamp":1505917778105}
> {code}
> After waiting for the specified 'profiler.period.duration', When profiler
> tries to write the data to hbase table Below is the exception on the profiler
> worker log
> {code:java}
> 2017-09-20 14:29:38.881 o.a.h.h.i.AbstractRpcClient [WARN] Exception
> encountered while connecting to the server :
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> 2017-09-20 14:29:38.882 o.a.h.h.i.AbstractRpcClient [ERROR] SASL
> authentication failed. The most likely cause is missing or invalid
> credentials. Consider 'kinit'.
> javax.security.sasl.SaslException: GSS initiate failed
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> ~[?:1.8.0_144]
> at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:609)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$600(RpcClientImpl.java:154)
> [stormjar.jar:?]
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:735)
> ~[stormjar.jar:?]
[jira] [Created] (METRON-1662) PCAP UI: Downloading PDML
Tibor Meller created METRON-1662: Summary: PCAP UI: Downloading PDML Key: METRON-1662 URL: https://issues.apache.org/jira/browse/METRON-1662 Project: Metron Issue Type: New Feature Reporter: Tibor Meller -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1662) PCAP UI: Downloading PDML
[ https://issues.apache.org/jira/browse/METRON-1662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540153#comment-16540153 ] Simon Elliston Ball commented on METRON-1662: - Is PDML a realistic format for download to an end user? I would have thought that PCAP format would be better for an end user download. > PCAP UI: Downloading PDML > - > > Key: METRON-1662 > URL: https://issues.apache.org/jira/browse/METRON-1662 > Project: Metron > Issue Type: New Feature >Reporter: Tibor Meller >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1572) Enhance KAFKA_PUT function
[ https://issues.apache.org/jira/browse/METRON-1572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-1572: --- Fix Version/s: Next + 1 > Enhance KAFKA_PUT function > -- > > Key: METRON-1572 > URL: https://issues.apache.org/jira/browse/METRON-1572 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: Next + 1 > > > Enhance the KAFKA_PUT function to accept either a List of String or a String. > This makes it simpler to send either 1 message or a bunch of messages. > KAFKA_PUT should queue up all messages to be sent before waiting on a > response. This improves response when sending a large number of messages. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1533) Create KAFKA_FIND Stellar Function
[
https://issues.apache.org/jira/browse/METRON-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Allen updated METRON-1533:
---
Fix Version/s: Next + 1
> Create KAFKA_FIND Stellar Function
> --
>
> Key: METRON-1533
> URL: https://issues.apache.org/jira/browse/METRON-1533
> Project: Metron
> Issue Type: Improvement
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Minor
> Fix For: Next + 1
>
>
> When creating enrichments, I often find that I want to validate that the
> enrichment I just created was successful on the live, incoming stream of
> telemetry. My workflow looks something like this.
> 1. Create and test the enrichment that I want to create.
> {code:java}
> [Stellar]>>> ip_src_addr := "72.34.49.86"
> 72.34.49.86
> [Stellar]>>> geo := GEO_GET(ip_src_addr)
> {country=US, dmaCode=803, city=Los Angeles, postalCode=90014,
> latitude=34.0438, location_point=34.0438,-118.2512, locID=5368361,
> longitude=-118.2512}
> {code}
> 2. That looks good to me. Now let's add that to my Bro telemetry.
> {code:java}
> [Stellar]>>> conf := SHELL_EDIT(conf)
> {
> "enrichment" : {
> "fieldMap": {
> "stellar": {
> "config": [
>"geo := GEO_GET(ip_src_addr)"
> ]
> }
> }
> },
> "threatIntel": {
> }
> }
> [Stellar]>>> CONFIG_PUT("ENRICHMENTS", e, "bro")
> {code}
>
> 3. It looks like that worked, but did that really work?
> At this point, I would run KAFKA_GET as many times as it takes to retrieve a
> Bro message. You would just have to get lucky and hope that the enrichment
> worked and secondly that you would pull down a Bro message (as opposed to a
> different sensor).
>
> I would rather have a function that lets me only pull back the messages that
> I care about. In this case I could either retrieve only Bro messages.
> {code:java}
> KAFKA_FIND('indexing', m -> MAP_GET('source.type', m) == 'bro')
> {code}
> Or I could look for messages that contain geolocation data.
> {code:java}
> KAFKA_FIND('indexing', m -> MAP_EXISTS('geo.city', m))
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1573) Enhance KAFKA_* functions to return partition and offset details
[ https://issues.apache.org/jira/browse/METRON-1573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-1573: --- Fix Version/s: Next + 1 > Enhance KAFKA_* functions to return partition and offset details > > > Key: METRON-1573 > URL: https://issues.apache.org/jira/browse/METRON-1573 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: Next + 1 > > > The KAFKA_* functions currently simply return the value of the message. > There are often times when it would be useful to get more detailed > information including the message partition, offset, key, etc. > The functions should be enhanced to allow a user to optionally get a more > detailed view. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1584) Indexing Topology Crashes with Invalid Message
[
https://issues.apache.org/jira/browse/METRON-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Allen updated METRON-1584:
---
Fix Version/s: Next + 1
> Indexing Topology Crashes with Invalid Message
> --
>
> Key: METRON-1584
> URL: https://issues.apache.org/jira/browse/METRON-1584
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
> Fix For: Next + 1
>
>
> Per Mohan Venkateshaiah:
> I published message "adkadknalkda;LK;ad;Da;dD;" to indexing topic , I see
> that the random access indexing topology worker thread died and couldn't
> recover until the kafka topic was deleted and recreated.
> {code:java}
> Caused by: java.lang.IllegalStateException: Unable to parse
> adkadknalkda;LK;ad;Da;dD; due to null
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:49)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:25)
> ~[stormjar.jar:?]
> at
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:237)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> ... 6 more
> Caused by: org.json.simple.parser.ParseException
> at org.json.simple.parser.Yylex.yylex(Yylex.java:610) ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.nextToken(JSONParser.java:269)
> ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.parse(JSONParser.java:118)
> ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.parse(JSONParser.java:81)
> ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.parse(JSONParser.java:75)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:47)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:25)
> ~[stormjar.jar:?]
> at
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:237)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> ... 6 more
> 2018-05-24 09:21:22.236 o.a.s.util Thread-9-indexingBolt-executor[3 3]
> [ERROR] Halting process: ("Worker died")
> java.lang.RuntimeException: ("Worker died")
> at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
> at org.apache.storm.daemon.worker$fn__10799$fn__10800.invoke(worker.clj:763)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.daemon.executor$mk_executor_data$fn__10011$fn__10012.invoke(executor.clj:276)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:494)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
> 2018-05-24 09:21:22.237 o.a.s.d.worker Thread-20 [INFO] Shutting down worker
> random_access_indexing-24-1527147389 703b5bf7-6c9d-46f3-8136-0c4877a69375 6700
> 2018-05-24 09:21:22.237 o.a.s.d.worker Thread-20 [INFO] Terminating messaging
> context
> 2018-05-24 09:21:22.238 o.a.s.d.worker Thread-20 [INFO] Shutting down
> executors
> 2018-05-24 09:21:22.238 o.a.s.d.executor Thread-20 [INFO] Shutting down
> executor
> __metricsorg.apache.hadoop.metrics2.sink.storm.StormTimelineMetricsSink:[2 2]
> 2018-05-24 09:21:22.239 o.a.s.util Thread-6-disruptor-executor[2
> 2]-send-queue [INFO] Async loop interrupted!
> 2018-05-24 09:21:22.239 o.a.s.util
>
[jira] [Updated] (METRON-1531) Web Start Scripts Incorrectly Report OK If Process Dies
[ https://issues.apache.org/jira/browse/METRON-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-1531: --- Labels: won't-fix (was: ) > Web Start Scripts Incorrectly Report OK If Process Dies > --- > > Key: METRON-1531 > URL: https://issues.apache.org/jira/browse/METRON-1531 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Labels: won't-fix > > The start scripts for the Alerts and Management UIs will incorrectly report > OK when running 'start', if the previously started process has died or been > killed. This is misleading. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1099: METRON-1657: Parser aggregation in storm
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1099 I ran through https://github.com/apache/metron/blob/master/use-cases/parser_chaining/README.md, but spinning it up as ``` $METRON_HOME/bin/start_parser_topology.sh -k $BROKERLIST -z $ZOOKEEPER -s cisco-6-302,cisco-5-304,pix_syslog_router ``` This results in indices (noting that I'd pushed the data to the topic a couple times, so the numbers won't line up directly if you run it): ``` [root@node1 ~]# curl -X GET "localhost:9200/_cat/indices?v" health status index uuid pri rep docs.count docs.deleted store.size pri.store.size yellow open cisco-5-304_index_2018.07.11.18 z-MyPPEJSN6cur7FJbFORA 5 1 450340.8kb340.8kb yellow open cisco-6-302_index_2018.07.11.18 vAFrok4sRpW49_RYt9RqbQ 5 16600 1.4mb 1.4mb ... ---
[jira] [Commented] (METRON-1657) Parser aggregation in storm
[ https://issues.apache.org/jira/browse/METRON-1657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540579#comment-16540579 ] ASF GitHub Bot commented on METRON-1657: Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1099 I ran through https://github.com/apache/metron/blob/master/use-cases/parser_chaining/README.md, but spinning it up as ``` $METRON_HOME/bin/start_parser_topology.sh -k $BROKERLIST -z $ZOOKEEPER -s cisco-6-302,cisco-5-304,pix_syslog_router ``` This results in indices (noting that I'd pushed the data to the topic a couple times, so the numbers won't line up directly if you run it): ``` [root@node1 ~]# curl -X GET "localhost:9200/_cat/indices?v" health status index uuid pri rep docs.count docs.deleted store.size pri.store.size yellow open cisco-5-304_index_2018.07.11.18 z-MyPPEJSN6cur7FJbFORA 5 1 450340.8kb340.8kb yellow open cisco-6-302_index_2018.07.11.18 vAFrok4sRpW49_RYt9RqbQ 5 16600 1.4mb 1.4mb ... > Parser aggregation in storm > --- > > Key: METRON-1657 > URL: https://issues.apache.org/jira/browse/METRON-1657 > Project: Metron > Issue Type: Bug >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > > Currently our parsing solution requires one storm topology per sensor. It has > been complained that this may be wasteful of resources and that, rather than > one storm topology per sensor, it would be advantageous to have multiple > sensors in the same topology. The benefit to this is that it would require > fewer storm slots. > The issue with this is that whenever we've aggregated functionality like this > before, we've run into issues appropriately being able to scale storm (e.g. > batch vs random access indexing in the same topology). The main point in > addressing this is to recommend that parsers with similar velocities and > complexity are grouped together. > Particularly for a first cut, leave the configuration mostly as-is, while > allowing for comma separated lists of sensors in start_parser_topology.sh > (e.g. bro,yaf creates a aggregated parser consisting of those two). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1660) On Solr, sorting by threat score fails
[
https://issues.apache.org/jira/browse/METRON-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540587#comment-16540587
]
ASF GitHub Bot commented on METRON-1660:
Github user asfgit closed the pull request at:
https://github.com/apache/metron/pull/1102
> On Solr, sorting by threat score fails
> --
>
> Key: METRON-1660
> URL: https://issues.apache.org/jira/browse/METRON-1660
> Project: Metron
> Issue Type: Bug
>Reporter: Justin Leet
>Assignee: Justin Leet
>Priority: Major
>
> * HCP 1.5.1.0 bld 16
> * 12-node Ubuntu 14.04 cluster on Openstack
> * Solr indexing server + Kerberized
> *Steps to Reproduce*
> 1. Ingest snort events
> 2. Create a metaalert using snort alerts
> 3. Sort by 'Score'
> Following error is seen in REST API:
> {code}
> {"responseCode":500,"message":"No live SolrServers available to handle this
> request:[http://nat-u14-jaks-metron-1.openstacklocal:8983/solr/metaalert_shard1_replica1,
>
> http://nat-u14-jaks-metron-1.openstacklocal:8983/solr/yaf_shard1_replica1]","fullMessage":"HttpSolrClient.RemoteSolrException:
> Error from server at
> http://nat-u14-jaks-metron-1.openstacklocal:8983/solr/metaalert_shard1_replica1:
> java.lang.Float cannot be cast to java.lang.String"}
> {code}
> Following exceptions are seen in solr.log
> {code}
> 2018-07-09 09:41:25,817 [qtp705265961-18] ERROR [c:metaalert s:shard1
> r:core_node1 x:metaalert_shard1_replica1]
> org.apache.solr.common.SolrException (SolrException.java:159) -
> null:java.lang.ClassCastException: java.lang.Float cannot be cast to
> java.lang.String
> at
> org.apache.solr.schema.FieldType.unmarshalStringSortValue(FieldType.java:1100)
> at org.apache.solr.schema.StrField.unmarshalSortValue(StrField.java:105)
> at
> org.apache.solr.handler.component.QueryComponent.unmarshalSortValues(QueryComponent.java:1251)
> at
> org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:1075)
> at
> org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:775)
> at
> org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:754)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:429)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:723)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:534)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
> at
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
>
[GitHub] metron pull request #1102: METRON-1660: On Solr, sorting by threat score fai...
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1102 ---
