[jira] [Updated] (METRON-829) Use Fastcapa with Kerberos
[ https://issues.apache.org/jira/browse/METRON-829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-829: -- Fix Version/s: Next + 1 > Use Fastcapa with Kerberos > -- > > Key: METRON-829 > URL: https://issues.apache.org/jira/browse/METRON-829 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.1 >Reporter: Nick Allen >Assignee: Nick Allen > Fix For: Next + 1 > > > How can we use Fastcapa in a Kerberized environment? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (METRON-829) Use Fastcapa with Kerberos
[ https://issues.apache.org/jira/browse/METRON-829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-829: -- Affects Version/s: 0.3.1 > Use Fastcapa with Kerberos > -- > > Key: METRON-829 > URL: https://issues.apache.org/jira/browse/METRON-829 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.1 >Reporter: Nick Allen >Assignee: Nick Allen > Fix For: Next + 1 > > > How can we use Fastcapa in a Kerberized environment? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-854) Create DHCPDump Parser
[
https://issues.apache.org/jira/browse/METRON-854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969028#comment-15969028
]
Nick Allen commented on METRON-854:
---
+1 very useful!
> Create DHCPDump Parser
> --
>
> Key: METRON-854
> URL: https://issues.apache.org/jira/browse/METRON-854
> Project: Metron
> Issue Type: New Feature
>Reporter: Bas van de Lustgraaf
>Priority: Minor
> Labels: parser
>
> Create a DHCPDump parser. This information can be used during enrichment to
> link ip-addresses to hostnames.
> {noformat}
> TIME: 2017-01-16 16:54:21.655|INTERFACE: eth2|OP:1 BOOTPREQUEST|CIADDR:
> 172.20.75.77|YIADDR: 0.0.0.0|SIADDR: 0.0.0.0|GIADDR: 172.20.75.8|CHADDR:
> fc:f8:ae:e8:ef:db:00:00:00:00:00:00:00:00:00:00|OPTION: 53 1 DHCP message
> type: 8 |DHCPINFORM|OPTION: 61 7 Client-identifier:
> 01:fc:f8:ae:e8:ef:db|OPTION: 12 5 Host name: Q1244|OPTION: 60 8 Vendor
> class identifier: MSFT 5.0|OPTION: 55 13 Parameter Request List: 1
> (Subnet mask)|| 15 (Domainname)|| 3 (Routers)|| 6 (DNS server)|| 44
> (NetBIOS name server)|| 46 (NetBIOS node type)|| 47 (NetBIOS scope)|| 31
> (Perform router discovery)|| 33 (Static route)||121 (Classless Static
> Route)||249 (MSFT - Classless route)|| 43 (Vendor specific info)||252 (MSFT -
> WinSock Proxy Auto Detect)|||IP: 10.10.10.177 > 172.20.1.11 |
> b8:ca:3a:67:95:8a > 0:50:56:84:68:43
> TIME: 2017-01-16 17:13:14.548|INTERFACE: eth2|OP:1 BOOTPREQUEST|CIADDR:
> 172.20.75.77|YIADDR: 0.0.0.0|SIADDR: 0.0.0.0|GIADDR: 172.20.75.8|CHADDR:
> fc:f8:ae:e8:ef:db:00:00:00:00:00:00:00:00:00:00|OPTION: 53 1 DHCP message
> type: 8 |DHCPINFORM|OPTION: 61 7 Client-identifier:
> 01:fc:f8:ae:e8:ef:db|OPTION: 12 5 Host name: Q1244|OPTION: 60 8 Vendor
> class identifier: MSFT 5.0|OPTION: 55 13 Parameter Request List: 1
> (Subnet mask)|| 15 (Domainname)|| 3 (Routers)|| 6 (DNS server)|| 44
> (NetBIOS name server)|| 46 (NetBIOS node type)|| 47 (NetBIOS scope)|| 31
> (Perform router discovery)|| 33 (Static route)||121 (Classless Static
> Route)||249 (MSFT - Classless route)|| 43 (Vendor specific info)||252 (MSFT -
> WinSock Proxy Auto Detect)|||IP: 10.10.10.177 > 172.20.1.10 |
> b8:ca:3a:67:95:8a > 0:50:56:b9:28:ac
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (METRON-836) Use Pycapa with Kerberos
Nick Allen created METRON-836: - Summary: Use Pycapa with Kerberos Key: METRON-836 URL: https://issues.apache.org/jira/browse/METRON-836 Project: Metron Issue Type: Improvement Affects Versions: 0.3.1 Reporter: Nick Allen Assignee: Nick Allen Fix For: Next + 1 How can we use Pycapa with Kerberos? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-835) Use Profiler with Kerberos
Nick Allen created METRON-835: - Summary: Use Profiler with Kerberos Key: METRON-835 URL: https://issues.apache.org/jira/browse/METRON-835 Project: Metron Issue Type: Improvement Affects Versions: 0.3.1 Reporter: Nick Allen Assignee: Nick Allen Fix For: Next + 1 Define how the Profiler can be used with Kerberos. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-626) Add Higher-Order Map Function to Stellar
[
https://issues.apache.org/jira/browse/METRON-626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960962#comment-15960962
]
Nick Allen commented on METRON-626:
---
Closed as this is a duplicate of METRON-831
> Add Higher-Order Map Function to Stellar
>
>
> Key: METRON-626
> URL: https://issues.apache.org/jira/browse/METRON-626
> Project: Metron
> Issue Type: Improvement
>Reporter: Nick Allen
>
> It would be extremely useful to have a 'map' higher-order function in
> Stellar. A 'flat map' might also be needed.
> [https://en.wikipedia.org/wiki/Map_(higher-order_function)]
> A few examples that I am thinking of. Ignore my choice of syntax. I am just
> trying to express the use cases here.
> Fetch all of the profile measurements over the past 2 hours. There may be
> many. What is the mean of each of those measurements?
> {code}
> stats := PROFILE_GET('p1', 'e1', 2, "HOURS")
> stats.MAP( s -> STATS_MEAN(s))
> {code}
> Get profile measurements taken 1 week ago, 2 weeks ago, and 3 weeks ago, so
> that I can then compare them.
> {code}
> { "1 week", "2 weeks", "3 weeks" }.MAP( lookback -> PROFILE_GET('p1','e1',
> lookback)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (METRON-626) Add Higher-Order Map Function to Stellar
[
https://issues.apache.org/jira/browse/METRON-626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960952#comment-15960952
]
Nick Allen commented on METRON-626:
---
Either way. We can close this one as a dup.
> Add Higher-Order Map Function to Stellar
>
>
> Key: METRON-626
> URL: https://issues.apache.org/jira/browse/METRON-626
> Project: Metron
> Issue Type: Improvement
>Reporter: Nick Allen
>
> It would be extremely useful to have a 'map' higher-order function in
> Stellar. A 'flat map' might also be needed.
> [https://en.wikipedia.org/wiki/Map_(higher-order_function)]
> A few examples that I am thinking of. Ignore my choice of syntax. I am just
> trying to express the use cases here.
> Fetch all of the profile measurements over the past 2 hours. There may be
> many. What is the mean of each of those measurements?
> {code}
> stats := PROFILE_GET('p1', 'e1', 2, "HOURS")
> stats.MAP( s -> STATS_MEAN(s))
> {code}
> Get profile measurements taken 1 week ago, 2 weeks ago, and 3 weeks ago, so
> that I can then compare them.
> {code}
> { "1 week", "2 weeks", "3 weeks" }.MAP( lookback -> PROFILE_GET('p1','e1',
> lookback)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (METRON-831) Add lambda expressions and rudimentary functional programming primitives to Stellar
[ https://issues.apache.org/jira/browse/METRON-831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960767#comment-15960767 ] Nick Allen commented on METRON-831: --- Is this the same as https://issues.apache.org/jira/browse/METRON-626? > Add lambda expressions and rudimentary functional programming primitives to > Stellar > --- > > Key: METRON-831 > URL: https://issues.apache.org/jira/browse/METRON-831 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella > > To enable use-cases where the user may want to do simple iteration, we can > expose simple functional primitives such as: > * MAP > * REDUCE > * FILTER > To fully support this, we will need lambda expressions as arguments to these > functions. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (METRON-822) Improve Fastcapa Performance
[ https://issues.apache.org/jira/browse/METRON-822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-822: -- Fix Version/s: Next + 1 > Improve Fastcapa Performance > > > Key: METRON-822 > URL: https://issues.apache.org/jira/browse/METRON-822 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.1 >Reporter: Nick Allen >Assignee: Nick Allen > Fix For: Next + 1 > > > Improve the performance and scalability of the Fastcapa probe. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (METRON-770) Unable to Launch Fastcapa Test Environment
[
https://issues.apache.org/jira/browse/METRON-770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Allen updated METRON-770:
--
Affects Version/s: 0.3.1
> Unable to Launch Fastcapa Test Environment
> --
>
> Key: METRON-770
> URL: https://issues.apache.org/jira/browse/METRON-770
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.3.1
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: Next + 1
>
>
> Unable to launch the Fastcapa test environment. Errors out during launch
> with the following error.
> {code}
> TASK [ambari_gather_facts : Ask Ambari: metron_hosts]
> **
> fatal: [source]: FAILED! => {"failed": true, "msg": "ERROR! 'dict object' has
> no attribute 'ambari_master'"}
> PLAY RECAP
> *
> source : ok=20 changed=15 unreachable=0failed=1
> Ansible failed to complete successfully. Any error output should be
> visible above. Please fix these errors and try again.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (METRON-770) Unable to Launch Fastcapa Test Environment
[
https://issues.apache.org/jira/browse/METRON-770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Allen updated METRON-770:
--
Fix Version/s: Next + 1
> Unable to Launch Fastcapa Test Environment
> --
>
> Key: METRON-770
> URL: https://issues.apache.org/jira/browse/METRON-770
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.3.1
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: Next + 1
>
>
> Unable to launch the Fastcapa test environment. Errors out during launch
> with the following error.
> {code}
> TASK [ambari_gather_facts : Ask Ambari: metron_hosts]
> **
> fatal: [source]: FAILED! => {"failed": true, "msg": "ERROR! 'dict object' has
> no attribute 'ambari_master'"}
> PLAY RECAP
> *
> source : ok=20 changed=15 unreachable=0failed=1
> Ansible failed to complete successfully. Any error output should be
> visible above. Please fix these errors and try again.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (METRON-822) Improve Fastcapa Performance
[ https://issues.apache.org/jira/browse/METRON-822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-822: -- Affects Version/s: 0.3.1 > Improve Fastcapa Performance > > > Key: METRON-822 > URL: https://issues.apache.org/jira/browse/METRON-822 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.1 >Reporter: Nick Allen >Assignee: Nick Allen > Fix For: Next + 1 > > > Improve the performance and scalability of the Fastcapa probe. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (METRON-670) Monit Incorrectly Reports Status
[ https://issues.apache.org/jira/browse/METRON-670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-670: -- Fix Version/s: (was: 0.3.1) > Monit Incorrectly Reports Status > > > Key: METRON-670 > URL: https://issues.apache.org/jira/browse/METRON-670 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen > > In a constrained environment, like 'Quick Dev', Monit will often incorrectly > report the status of a Metron topology. This occurs when the environment is > under load and a query of topology status exceeds the default timeout of 30 > seconds. Need to add a parameter so that the timeout for a status check can > be extended under these conditions. This was previously done for starting > and stopping a topology, but not for a status checks. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (METRON-701) Triage Metrics Produced by the Profiler
[ https://issues.apache.org/jira/browse/METRON-701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-701: -- Affects Version/s: 0.3.1 Labels: Profiler Triage (was: ) Fix Version/s: Next + 1 > Triage Metrics Produced by the Profiler > --- > > Key: METRON-701 > URL: https://issues.apache.org/jira/browse/METRON-701 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.1 >Reporter: Nick Allen >Assignee: Nick Allen > Labels: Profiler, Triage > Fix For: Next + 1 > > > h3. Problem > The motivating example is that I would like to create an alert if the number > of inbound flows to any host over a 15 minute interval is abnormal. > The value being interrogated here, the number of inbound flows, is not a > static value contained within any single telemetry message. This value is > calculated across multiple messages by the Profiler. The current Threat > Triage process cannot be used to interrogate values calculated by the > Profiler. > h3. Proposed Solution > I am proposing that we treat the Profiler as a source of telemetry. The > measurements captured by the Profiler would be enqueued into a Kafka topic. > We would then treat those Profiler messages like any other telemetry. We > would parse, enrich, triage, and index those messages. > This would have the following advantages. > 1. We would be able to reuse the same threat triage mechanism for values > calculated by the Profiler. > 2. We would be able to generate profiles from the profiled data - aka > meta-profiles anyone? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-829) Use Fastcapa with Kerberos
Nick Allen created METRON-829: - Summary: Use Fastcapa with Kerberos Key: METRON-829 URL: https://issues.apache.org/jira/browse/METRON-829 Project: Metron Issue Type: Improvement Reporter: Nick Allen Assignee: Nick Allen How can we use Fastcapa in a Kerberized environment? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-822) Improve Fastcapa Performance
[
https://issues.apache.org/jira/browse/METRON-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957203#comment-15957203
]
Nick Allen commented on METRON-822:
---
h2. Performance Test
h3. Test Environment
There were two hosts, both Cisco UCS boxes with 10G Cisco VNICs. There is a
point-to-point fiber connection between `enp9s0f0` on both hosts. There is
also a traditional switched link on `enp10s0f0` connecting these hosts.
Packets are generated on y137 on `enp9s0f0` and are then captured on y138 over
`enp9s0f0` by Fastcapa. Fastcapa then uses `en10s0f0` to package and send
those packets in bulk to Kafka running on y137.
h3. Results
The following shows the Fastcapa probe capturing ~1.1 Gbps successfully for
roughly 10 minutes. All metrics were inline and no significant backlog was
created. Effectively, no packets were dropped. The small number shown as a
loss are a rounded error and due to some noise on the line caused by a
misconfigured NIC. The test was successful.
||Event ||Packet Count||Delta||
| Packets Generated | 126,925,885 | - |
| Received @ Fastcapa | 126,926,236 | -351 |
| Received @ Kafka| 126,926,235 | ~0 |
h3. Steps
Create Kakfa Topic with 256 partitions and 1 broker
{code}
[root@y137 ~]# kafka-topics.sh --zookeeper y137:2181 --create --topic pcap256
--partitions 256 --replication-factor 1
Created topic "pcap256".
{code}
Count of the number of packets in the topic before the test. The count simply
uses the offset.
{code}
[root@y137 ~]# kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list
y137:9092 --topic pcap256 --time -1 | tail -n +2 | awk -F":" '{s+=$3} END
{print s}'
0
{code}
Start Fastcapa
* Rx descriptors limited to 512 due to VNIC
* Rx queues limited to 2 due to VNIC
* All other parameters as shown in log
{code}
[root@y138 fastcapa]# fastcapa -l 0,1,2,3,4,5,6,7 --huge-dir /mnt/huge_1GB --
-t pcap256 -c /etc/fastcapa.y137 -r 512 -q 2 -x 32768
EAL: Detected 32 lcore(s)
EAL: Probing VFIO support...
EAL: PCI device :01:00.0 on NUMA socket 0
EAL: probe driver: 8086:1521 net_e1000_igb
EAL: PCI device :01:00.1 on NUMA socket 0
EAL: probe driver: 8086:1521 net_e1000_igb
EAL: PCI device :09:00.0 on NUMA socket 0
EAL: probe driver: 1137:43 net_enic
PMD: rte_enic_pmd: Advanced Filters not available
PMD: rte_enic_pmd: vNIC MAC addr 00:35:1a:cd:6c:ad wq/rq 256/512 mtu 1500, max
mtu:9004
PMD: rte_enic_pmd: vNIC csum tx/rx yes/yes rss yes intr mode any type min timer
125 usec loopback tag 0x
PMD: rte_enic_pmd: vNIC resources avail: wq 1 rq 4 cq 5 intr 8
EAL: PCI device :0a:00.0 on NUMA socket 0
EAL: probe driver: 1137:43 net_enic
[ -t KAFKA_TOPIC ] defined as pcap256
[ -c KAFKA_CONFIG ] defined as /etc/fastcapa.y137
[ -r NB_RX_DESC ] defined as 512
[ -q NB_RX_QUEUE ] defined as 2
[ -x TX_RING_SIZE ] defined as 32768
[ -p PORT_MASK ] undefined; defaulting to 0x01
[ -b BURST_SIZE ] undefined; defaulting to 32
USER1: Initializing port 0
PMD: rte_enic_pmd: Rq 0 Scatter rx mode enabled
PMD: rte_enic_pmd: Rq 0 Scatter rx mode not being used
PMD: rte_enic_pmd: Using 512 rx descriptors (sop 512, data 0)
PMD: rte_enic_pmd: Rq 1 Scatter rx mode enabled
PMD: rte_enic_pmd: Rq 1 Scatter rx mode not being used
PMD: rte_enic_pmd: Using 512 rx descriptors (sop 512, data 0)
PMD: rte_enic_pmd: TX Queues - effective number of descs:32
PMD: rte_enic_pmd: vNIC resources used: wq 1 rq 4 cq 3 intr 0
USER1: Device setup successfully; port=0, mac=00 35 1a cd 6c ad
USER1: Launching receive worker; worker=0, core=1, queue=0
USER1: Receive worker started; core=1, socket=0, queue=0 attempts=32
USER1: Launching receive worker; worker=1, core=2, queue=1
USER1: Receive worker started; core=2, socket=0, queue=1 attempts=32
USER1: Launching transmit worker; worker=0, core=3 ring=0
USER1: Launching transmit worker; worker=1, core=4 ring=1
USER1: Transmit worker started; core=3, socket=0
USER1: Transmit worker started; core=4, socket=0
USER1: Launching transmit worker; worker=2, core=5 ring=0
USER1: Launching transmit worker; worker=3, core=6 ring=1
USER1: Transmit worker started; core=5, socket=0
USER1: Launching transmit worker; worker=4, core=7 ring=0
USER1: Transmit worker started; core=6, socket=0
USER1: Starting to monitor workers; core=0, socket=0
USER1: Transmit worker started; core=7, socket=0
- in - --- queued --- - out - drops
[nic] 2 - - -
[rx]2 - 2 0
[tx]2 - 2 0
[kaf] 2 0 1 0
...
{code}
Start packet generator.
{code}
[root@y137 ~]# time tcpreplay -i enp9s0f0 --loop=0 --stats=5 --preload-pcap
--mbps 1100 example.pcap
File Cache is enabled
Actual: 1055541 packets (687500724 bytes) sent in 5.00 seconds.
[jira] [Created] (METRON-828) Fastcapa - Roll Stats Output File
Nick Allen created METRON-828: - Summary: Fastcapa - Roll Stats Output File Key: METRON-828 URL: https://issues.apache.org/jira/browse/METRON-828 Project: Metron Issue Type: Improvement Reporter: Nick Allen Priority: Minor With Fastcapa, the `-s` command line option will write detailed statistics from the Kafka client library to an external file. This file can get rather larger depending on how frequently you have `statistics.interval.ms` set. The code should be updated to properly roll the file and handle faults. There is probably a nice logging lib that could handle this for us. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-822) Improve Fastcapa Performance
Nick Allen created METRON-822: - Summary: Improve Fastcapa Performance Key: METRON-822 URL: https://issues.apache.org/jira/browse/METRON-822 Project: Metron Issue Type: Improvement Reporter: Nick Allen Assignee: Nick Allen Improve the performance and scalability of the Fastcapa probe. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-770) Unable to Launch Fastcapa Test Environment
Nick Allen created METRON-770:
-
Summary: Unable to Launch Fastcapa Test Environment
Key: METRON-770
URL: https://issues.apache.org/jira/browse/METRON-770
Project: Metron
Issue Type: Bug
Reporter: Nick Allen
Assignee: Nick Allen
Unable to launch the Fastcapa test environment. Errors out during launch with
the following error.
{code}
TASK [ambari_gather_facts : Ask Ambari: metron_hosts] **
fatal: [source]: FAILED! => {"failed": true, "msg": "ERROR! 'dict object' has
no attribute 'ambari_master'"}
PLAY RECAP *
source : ok=20 changed=15 unreachable=0failed=1
Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (METRON-756) Potential for Unexpected Looping If Error Occurs in Indexing Topology
Nick Allen created METRON-756: - Summary: Potential for Unexpected Looping If Error Occurs in Indexing Topology Key: METRON-756 URL: https://issues.apache.org/jira/browse/METRON-756 Project: Metron Issue Type: Bug Reporter: Nick Allen If an error occurs in the indexing topology there is a potential for an unexpected failure loop. This was first called out on a PR discussion thread [1]. [1] https://github.com/apache/incubator-metron/pull/453#issuecomment-282159998 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-747) Blanket Reformat of code based on checkstyle configs
[ https://issues.apache.org/jira/browse/METRON-747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15890177#comment-15890177 ] Nick Allen commented on METRON-747: --- I have seen in the generated code from ANTRL specific comments that seem to be intended to solve this problem. For example, in org.apache.metron.common.stellar.generated.StellarLexer there is a comment on line 4 "//CHECKSTYLE:OFF". > Blanket Reformat of code based on checkstyle configs > > > Key: METRON-747 > URL: https://issues.apache.org/jira/browse/METRON-747 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Priority: Minor > > Once METRON-746 is done, we should perform a blanket reformat of our code per > discussion on the dev list. At this point we should start enforcing our code > via checkstyle (so make sure the appropriate code styling is set to error). > We might need to be careful about autogenerated code here (otherwise we might > start breaking builds on things we don't care about). I'm unsure if you can > whitelist specific files or dirs in checkstyle or not. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (METRON-601) MockHTable Put Log is Not Thread Safe
[ https://issues.apache.org/jira/browse/METRON-601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-601: -- Affects Version/s: 0.3.0 Fix Version/s: (was: Next + 1) 0.3.1 > MockHTable Put Log is Not Thread Safe > - > > Key: METRON-601 > URL: https://issues.apache.org/jira/browse/METRON-601 > Project: Metron > Issue Type: Sub-task >Affects Versions: 0.3.0 >Reporter: Nick Allen >Assignee: Nick Allen > Fix For: 0.3.1 > > > The MockHTable uses an ArrayList to store a log of Puts that have been > submitted against the MockHTable. The MockHTable, along with the put log, is > accessed from multiple threads during the integration tests. Access to the > Put log is not thread safe, which is likely at least one cause of METRON-597. > The Put log is used by multiple tests, but more so by the > ProfilerIntegrationTest. This tests polls the list to block the thread until > the expected number of Puts have been submitted. This is likely why this > test is more impacted by this issue than others. > The Put Log needs to made thread safe. See > `org.apache.metron.test.mock.MockHTable.putLog` -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (METRON-686) Record Rule Set that Fired During Threat Triage
[
https://issues.apache.org/jira/browse/METRON-686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Allen updated METRON-686:
--
Affects Version/s: 0.3.1
Fix Version/s: Next + 1
> Record Rule Set that Fired During Threat Triage
> ---
>
> Key: METRON-686
> URL: https://issues.apache.org/jira/browse/METRON-686
> Project: Metron
> Issue Type: Improvement
>Affects Versions: 0.3.1
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: Next + 1
>
>
> h3. Problem
> There is little transparency into the Threat Triage process itself. When
> Threat Triage runs, all I get is a score. I don't know how that score was
> arrived at, which rules were triggered, and the specific values that caused a
> rule to trigger.
> More specifically, there is no way to generate a message that looks like "The
> host 'powned.svr.bank.com' has '230' inbound flows, exceeding the threshold
> of '202'". This makes it difficult for an analyst to action the alert.
> h3. Proposed Solution
> To improve the transparency of the Threat Triage process, I am proposing
> these enhancements.
> (1) Threat Triage should attach to each message all of the rules that fired
> in addition to the total calculated threat triage score.
> (2) Threat Triage should allow a custom message to be generated for each
> rule. The custom message would allow for some form of string interpolation
> so that I can add specific values from each message to the generated alert.
> We could allow this in one or both of the new fields that Casey just added,
> name and comment.
> (3) The specific method of string interpolation will be implemented under a
> separate issue.
> h3. Example
> (1) In this example, we have a telemetry message with a field called 'value'
> that we need to monitor. In Enrichment, I calculate some sort of value
> threshold, over which an alert should be generated.
> (2) In Threat Triage, I use the calculated value threshold to alert on any
> message that has a value exceeding this threshold.
> (3) By leveraging a new field called 'reason', I can embed values from the
> message, like the hostname, value, and value threshold, into the alert
> produced by Threat Triage.
> {code}
> "triageConfig" : {
> "riskLevelRules" : [ {
> "name" : "Abnormal DNS Port",
> "rule" : "source.type == 'bro' and protocol == 'dns' and ip_dst_port
> != 53",
> "score" : 10.0,
> "reason" : "FORMAT('Abnormal DNS Port: expected: 53, found: %s:%d',
> ip_dst_addr, ip_dst_port)"
> } ],
> "aggregator" : "MAX",
> "aggregationConfig" : { }
> }
> {code}
> (4) The Threat Triage process today would add only the total calculated score.
> {code}
> "threat.triage.level": 10.0
> {code}
> With this proposal, Threat Triage would add the following to the message.
> {code}
> "threat.triage.level":{
>"score":10.0,
>"rules":[
> {
> "name":"Abnormal DNS Port",
> "comment":null
> "score":10.0,
> "reason":"Abnormal DNS Port: expected: 53, found: 224.0.0.251:5353",
> }
>]
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (METRON-687) Create String Formatting Function for Stellar
[ https://issues.apache.org/jira/browse/METRON-687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-687: -- Affects Version/s: 0.3.0 Fix Version/s: 0.3.1 > Create String Formatting Function for Stellar > - > > Key: METRON-687 > URL: https://issues.apache.org/jira/browse/METRON-687 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.0 >Reporter: Nick Allen >Assignee: Nick Allen > Fix For: 0.3.1 > > > Create a Stellar function that allows a user to create formatting strings aka > perform string interpolation. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (METRON-688) Add Threat Triage Reason Field
[ https://issues.apache.org/jira/browse/METRON-688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-688: -- Affects Version/s: 0.3.1 Fix Version/s: Next + 1 > Add Threat Triage Reason Field > -- > > Key: METRON-688 > URL: https://issues.apache.org/jira/browse/METRON-688 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.1 >Reporter: Nick Allen >Assignee: Nick Allen > Fix For: Next + 1 > > > Allow the Threat Triage comment field to contain a Stellar expression that > can be executed. This will allow a rule to produce a message with contextual > information like the specific values that caused the rule to trigger. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-688) Add Threat Triage Reason Field
[ https://issues.apache.org/jira/browse/METRON-688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15888678#comment-15888678 ] Nick Allen commented on METRON-688: --- Included in https://github.com/apache/incubator-metron/pull/438 > Add Threat Triage Reason Field > -- > > Key: METRON-688 > URL: https://issues.apache.org/jira/browse/METRON-688 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen > > Allow the Threat Triage comment field to contain a Stellar expression that > can be executed. This will allow a rule to produce a message with contextual > information like the specific values that caused the rule to trigger. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-739) Create Local Profile Runner
Nick Allen created METRON-739: - Summary: Create Local Profile Runner Key: METRON-739 URL: https://issues.apache.org/jira/browse/METRON-739 Project: Metron Issue Type: Improvement Reporter: Nick Allen Assignee: Nick Allen Troubleshooting issues when programming against a live stream of data can be difficult. This is the primary reason that drove the creation of the REPL. Similarly it would be useful to have a means to run a profile 'locally' to troubleshoot, validate, or debug the profile. I would want to somehow be able to throw canned data at the Profile message-by-message and interrogate the current state of the Profile. I would also need to be able to trigger a flush event and interrogate the result. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-738) Allow Dynamic Topics with KafkaMessageWriter
Nick Allen created METRON-738: - Summary: Allow Dynamic Topics with KafkaMessageWriter Key: METRON-738 URL: https://issues.apache.org/jira/browse/METRON-738 Project: Metron Issue Type: Improvement Affects Versions: 0.3.1 Reporter: Nick Allen When using a `BulkMessageWriterBolt` paired with a `KafkaMessageWriter`, I can only set the name of the destination topic statically. The name of the topic cannot be set or changed dynamically. For example, I cannot fetch the topic name from a Zookeeper configuration and set the value dynamically. This would have been useful for [METRON-701|https://issues.apache.org/jira/browse/METRON-701] and is referenced in this [Github conversation|https://github.com/apache/incubator-metron/pull/449#issuecomment-282028282]. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-735) Support Complex Data Types in Telemetry Messages
[ https://issues.apache.org/jira/browse/METRON-735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878721#comment-15878721 ] Nick Allen commented on METRON-735: --- ~cestella commented on this issue: https://github.com/apache/incubator-metron/pull/438 I'm of the opinion that the flattening should be writer-specific and that should be a function of the writer config with the default to be specified by the writer implementation. This way we can have our cake and eat it too. Also, we could ALREADY be in a situation where messages aren't flat (imagine a situation where a stellar function returns a map or a list and it get assigned to a field). The only safe way to do this is to enforce it at the writer, IMO. This is one of the stated benefits to extracting writer configs into their own structures. Regarding existing conventions, this one was around when I joined the project. I might be wrong, but it was an early convention. The reasoning, as I recall, was multi-fold: * Solr didn't handle it * Interacting with complex structures was deemed to be difficult * Indexing nested structures had some performance implications > Support Complex Data Types in Telemetry Messages > > > Key: METRON-735 > URL: https://issues.apache.org/jira/browse/METRON-735 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen > > Up until now, we have been working under the assumption that all telemetry > messages in Metron must not contain complex data types like lists or maps. > Complex data types were 'flattened' to remove any complex data types from the > telemetry messages. > Most parsers today produce flat telemetry messages. That is messages with no > complex data types. The problem is that even if I use a parser that > generates only flattened data, I could create an enrichment (like using > GEO_GET) that appends non-flat data to a message. Thus we have non-flat data > coursing through the veins of Metron. ;) > I think the original idea of flattening data was because one of our Indexers > could not handle non-flat, complex data types. At the time, we just decided, > well don't create any non-flat data. > But now, since we have a completely 'programmable' system, I don't think it > is safe to assume that the data will always be non-flat. A user could create > their own Stellar function to use during enrichment. Should we force on them > the burden of flattening the data? > It makes way more sense in my mind, to make the indexer transform the data > however it needs to , to correctly index the data. If the current issue is > with the Solr Indexer, then we should fix that to flatten any data that it > needs to. There would be one touch point to address this issue rather than > many. > This is a blanket JIRA that might result in multiple sub-tasks of changes > needed to allow telemetry messages to contain complex data types like lists > and maps. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-735) Support Complex Data Types in Telemetry Messages
Nick Allen created METRON-735: - Summary: Support Complex Data Types in Telemetry Messages Key: METRON-735 URL: https://issues.apache.org/jira/browse/METRON-735 Project: Metron Issue Type: Improvement Reporter: Nick Allen Up until now, we have been working under the assumption that all telemetry messages in Metron must not contain complex data types like lists or maps. Complex data types were 'flattened' to remove any complex data types from the telemetry messages. Most parsers today produce flat telemetry messages. That is messages with no complex data types. The problem is that even if I use a parser that generates only flattened data, I could create an enrichment (like using GEO_GET) that appends non-flat data to a message. Thus we have non-flat data coursing through the veins of Metron. ;) I think the original idea of flattening data was because one of our Indexers could not handle non-flat, complex data types. At the time, we just decided, well don't create any non-flat data. But now, since we have a completely 'programmable' system, I don't think it is safe to assume that the data will always be non-flat. A user could create their own Stellar function to use during enrichment. Should we force on them the burden of flattening the data? It makes way more sense in my mind, to make the indexer transform the data however it needs to , to correctly index the data. If the current issue is with the Solr Indexer, then we should fix that to flatten any data that it needs to. There would be one touch point to address this issue rather than many. This is a blanket JIRA that might result in multiple sub-tasks of changes needed to allow telemetry messages to contain complex data types like lists and maps. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (METRON-686) Record Rule Set that Fired During Threat Triage
[
https://issues.apache.org/jira/browse/METRON-686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Allen updated METRON-686:
--
Description:
h3. Problem
There is little transparency into the Threat Triage process itself. When
Threat Triage runs, all I get is a score. I don't know how that score was
arrived at, which rules were triggered, and the specific values that caused a
rule to trigger.
More specifically, there is no way to generate a message that looks like "The
host 'powned.svr.bank.com' has '230' inbound flows, exceeding the threshold of
'202'". This makes it difficult for an analyst to action the alert.
h3. Proposed Solution
To improve the transparency of the Threat Triage process, I am proposing these
enhancements.
(1) Threat Triage should attach to each message all of the rules that fired in
addition to the total calculated threat triage score.
(2) Threat Triage should allow a custom message to be generated for each rule.
The custom message would allow for some form of string interpolation so that I
can add specific values from each message to the generated alert. We could
allow this in one or both of the new fields that Casey just added, name and
comment.
(3) The specific method of string interpolation will be implemented under a
separate issue.
h3. Example
(1) In this example, we have a telemetry message with a field called 'value'
that we need to monitor. In Enrichment, I calculate some sort of value
threshold, over which an alert should be generated.
(2) In Threat Triage, I use the calculated value threshold to alert on any
message that has a value exceeding this threshold.
(3) By leveraging a new field called 'reason', I can embed values from the
message, like the hostname, value, and value threshold, into the alert produced
by Threat Triage.
{code}
"triageConfig" : {
"riskLevelRules" : [ {
"name" : "Abnormal DNS Port",
"rule" : "source.type == 'bro' and protocol == 'dns' and ip_dst_port !=
53",
"score" : 10.0,
"reason" : "FORMAT('Abnormal DNS Port: expected: 53, found: %s:%d',
ip_dst_addr, ip_dst_port)"
} ],
"aggregator" : "MAX",
"aggregationConfig" : { }
}
{code}
(4) The Threat Triage process today would add only the total calculated score.
{code}
"threat.triage.level": 10.0
{code}
With this proposal, Threat Triage would add the following to the message.
{code}
"threat.triage.level":{
"score":10.0,
"rules":[
{
"name":"Abnormal DNS Port",
"comment":null
"score":10.0,
"reason":"Abnormal DNS Port: expected: 53, found: 224.0.0.251:5353",
}
]
}
{code}
was:
h3. Problem
There is little transparency into the Threat Triage process itself. When
Threat Triage runs, all I get is a score. I don't know how that score was
arrived at, which rules were triggered, and the specific values that caused a
rule to trigger.
More specifically, there is no way to generate a message that looks like "The
host 'powned.svr.bank.com' has '230' inbound flows, exceeding the threshold of
'202'". This makes it difficult for an analyst to action the alert.
h3. Proposed Solution
To improve the transparency of the Threat Triage process, I am proposing these
enhancements.
(1) Threat Triage should attach to each message all of the rules that fired in
addition to the total calculated threat triage score.
(2) Threat Triage should allow a custom message to be generated for each rule.
The custom message would allow for some form of string interpolation so that I
can add specific values from each message to the generated alert. We could
allow this in one or both of the new fields that Casey just added, name and
comment.
(3) The specific method of string interpolation will be implemented under a
separate issue.
h3. Example
(1) In this example, we have a telemetry message with a field called 'value'
that we need to monitor. In Enrichment, I calculate some sort of value
threshold, over which an alert should be generated.
(2) In Threat Triage, I use the calculated value threshold to alert on any
message that has a value exceeding this threshold.
(3) I can embed values from the message, like the hostname, value, and value
threshold, into the alert produced by Threat Triage. Notice that I am using
{noformat}${this}{noformat} for string interpolation, but it could be any
syntax that we choose.
{code}
"triageConfig" : {
"riskLevelRules" : [
{
"name" : "Abnormal Value",
"comment" : "For ${hostname}; the value ${value} exceeds threshold of
${value_threshold}",
"rule" : "value > value_threshold",
"score" : 10
}
],
"aggregator" : "MAX"
}
{code}
(4) The Threat Triage process today would add only the total calculated score.
{code}
"threat.triage.level": 10.0
{code}
With this proposal, Threat Triage would add the following to the
[jira] [Created] (METRON-702) `KAFKA_GET` Hangs If Specified Kafka Broker Does Not Exist
Nick Allen created METRON-702: - Summary: `KAFKA_GET` Hangs If Specified Kafka Broker Does Not Exist Key: METRON-702 URL: https://issues.apache.org/jira/browse/METRON-702 Project: Metron Issue Type: Bug Reporter: Nick Allen By default, this function will attempt to connect to a Kafka Broker running at localhost:9092. If one does not exist there, it will hang indefinitely. It appears that the Kafka Consumer is not respecting the timeout settings when it attempts to connect to the broker. See associated Kafka issue. https://issues.apache.org/jira/browse/KAFKA-1894 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-701) Triage Metrics Produced by the Profiler
[ https://issues.apache.org/jira/browse/METRON-701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854566#comment-15854566 ] Nick Allen commented on METRON-701: --- Thanks. Originally, I was thinking of option 1 differently. I was thinking that the idea was to execute the "write" Stellar statement in the context of the existing "builder" bolt. Creating a separate "writer" bolt might improve option 1. If my understanding is now correct, the difference between the improved option 1 and option 2 is whether you have (option 1) a generic writer bolt that just executes a stellar statement or (option 2) you have a dedicated bolt to handle HBase and another for Kafka. How would we do something like batching if we have a generic bolt that just executes Stellar expressions? The Stellar function itself would have to implement the batching mechanism right? > Triage Metrics Produced by the Profiler > --- > > Key: METRON-701 > URL: https://issues.apache.org/jira/browse/METRON-701 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen > > h3. Problem > The motivating example is that I would like to create an alert if the number > of inbound flows to any host over a 15 minute interval is abnormal. > The value being interrogated here, the number of inbound flows, is not a > static value contained within any single telemetry message. This value is > calculated across multiple messages by the Profiler. The current Threat > Triage process cannot be used to interrogate values calculated by the > Profiler. > h3. Proposed Solution > I am proposing that we treat the Profiler as a source of telemetry. The > measurements captured by the Profiler would be enqueued into a Kafka topic. > We would then treat those Profiler messages like any other telemetry. We > would parse, enrich, triage, and index those messages. > This would have the following advantages. > 1. We would be able to reuse the same threat triage mechanism for values > calculated by the Profiler. > 2. We would be able to generate profiles from the profiled data - aka > meta-profiles anyone? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-701) Triage Metrics Produced by the Profiler
[ https://issues.apache.org/jira/browse/METRON-701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854407#comment-15854407 ] Nick Allen commented on METRON-701: --- There are multiple ways to consider to implementing a mechanism to write profile measurements to Kafka. Here are two to consider. (1) Alter the Profile Definitions The Profile author would choose if their profile data needs to be re-ingested as telemetry. The author would use a Stellar function to write the profile measurement to Kafka when the profile is flushed. This would expand the profile definition to accommodate this new feature. Pros * Simple - Simpler to implement * Flexibility - Profile data that is not triaged, does not need to be re-ingested and processed Cons * Difficult to Scale/Optimize - It will be difficult to optimize the writes to Kafka. An increase in either the number of profiles, or the number of entities being profiled (which can occur without any change to the Profiler itself) could cause scaling problems that cannot be easily addressed. * More Complex Profile Definition - Makes the Profile definition more complex and forces an additional decision on the user. * More Complex Triage - Introduces uncertainty about which profiles will be available for triage. Adding a new triage may require updating the threat triage rules AND updating the Profile definition. (2) Alter the Profiler Topology Alter the Profiler Topology, so that when a profile is flushed the measurement is sent to a downstream bolt for writing to HBase and another downstream bolt for writing to Kafka. There would be a global switch to write to Kafka or not. Either all Profiles write to Kafka or none do. Pros * Easier to Scale/Optimize - Provides a well-understood mechanism to optimize the writes to Kafka. Scaling the writes to Kafka could be done in the same manner as done for HBase. * Less Complex Profile Definition * Less Complex Triage - Less complex for the user as their is no uncertainty about which Profile data is available for threat triage. Cons * Less Flexible - Less flexible as either all Profiles write to Kafka or none do. > Triage Metrics Produced by the Profiler > --- > > Key: METRON-701 > URL: https://issues.apache.org/jira/browse/METRON-701 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen > > h3. Problem > The motivating example is that I would like to create an alert if the number > of inbound flows to any host over a 15 minute interval is abnormal. > The value being interrogated here, the number of inbound flows, is not a > static value contained within any single telemetry message. This value is > calculated across multiple messages by the Profiler. The current Threat > Triage process cannot be used to interrogate values calculated by the > Profiler. > h3. Proposed Solution > I am proposing that we treat the Profiler as a source of telemetry. The > measurements captured by the Profiler would be enqueued into a Kafka topic. > We would then treat those Profiler messages like any other telemetry. We > would parse, enrich, triage, and index those messages. > This would have the following advantages. > 1. We would be able to reuse the same threat triage mechanism for values > calculated by the Profiler. > 2. We would be able to generate profiles from the profiled data - aka > meta-profiles anyone? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-701) Triage Metrics Produced by the Profiler
Nick Allen created METRON-701: - Summary: Triage Metrics Produced by the Profiler Key: METRON-701 URL: https://issues.apache.org/jira/browse/METRON-701 Project: Metron Issue Type: Improvement Reporter: Nick Allen Assignee: Nick Allen h3. Problem The motivating example is that I would like to create an alert if the number of inbound flows to any host over a 15 minute interval is abnormal. The value being interrogated here, the number of inbound flows, is not a static value contained within any single telemetry message. This value is calculated across multiple messages by the Profiler. The current Threat Triage process cannot be used to interrogate values calculated by the Profiler. h3. Proposed Solution I am proposing that we treat the Profiler as a source of telemetry. The measurements captured by the Profiler would be enqueued into a Kafka topic. We would then treat those Profiler messages like any other telemetry. We would parse, enrich, triage, and index those messages. This would have the following advantages. 1. We would be able to reuse the same threat triage mechanism for values calculated by the Profiler. 2. We would be able to generate profiles from the profiled data - aka meta-profiles anyone? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (METRON-686) Record Rule Set that Fired During Threat Triage
[
https://issues.apache.org/jira/browse/METRON-686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Allen updated METRON-686:
--
Summary: Record Rule Set that Fired During Threat Triage (was: Record of
Rule Set that Fired During Threat Triage)
> Record Rule Set that Fired During Threat Triage
> ---
>
> Key: METRON-686
> URL: https://issues.apache.org/jira/browse/METRON-686
> Project: Metron
> Issue Type: Improvement
>Reporter: Nick Allen
>Assignee: Nick Allen
>
> h3. Problem
> There is little transparency into the Threat Triage process itself. When
> Threat Triage runs, all I get is a score. I don't know how that score was
> arrived at, which rules were triggered, and the specific values that caused a
> rule to trigger.
> More specifically, there is no way to generate a message that looks like "The
> host 'powned.svr.bank.com' has '230' inbound flows, exceeding the threshold
> of '202'". This makes it difficult for an analyst to action the alert.
> h3. Proposed Solution
> To improve the transparency of the Threat Triage process, I am proposing
> these enhancements.
> (1) Threat Triage should attach to each message all of the rules that fired
> in addition to the total calculated threat triage score.
> (2) Threat Triage should allow a custom message to be generated for each
> rule. The custom message would allow for some form of string interpolation
> so that I can add specific values from each message to the generated alert.
> We could allow this in one or both of the new fields that Casey just added,
> name and comment.
> (3) The specific method of string interpolation will be implemented under a
> separate issue.
> h3. Example
> (1) In this example, we have a telemetry message with a field called 'value'
> that we need to monitor. In Enrichment, I calculate some sort of value
> threshold, over which an alert should be generated.
> (2) In Threat Triage, I use the calculated value threshold to alert on any
> message that has a value exceeding this threshold.
> (3) I can embed values from the message, like the hostname, value, and value
> threshold, into the alert produced by Threat Triage. Notice that I am using
> {noformat}${this}{noformat} for string interpolation, but it could be any
> syntax that we choose.
> {code}
> "triageConfig" : {
> "riskLevelRules" : [
> {
> "name" : "Abnormal Value",
> "comment" : "For ${hostname}; the value ${value} exceeds threshold of
> ${value_threshold}",
> "rule" : "value > value_threshold",
> "score" : 10
> }
> ],
> "aggregator" : "MAX"
> }
> {code}
> (4) The Threat Triage process today would add only the total calculated score.
> {code}
> "threat.triage.level": 10.0
> {code}
> With this proposal, Threat Triage would add the following to the message.
> Notice how each of the {noformat}${variables}{noformat} have been replaced
> with the actual values extracted from the message. This allows for more
> contextual information to action the alert.
> {code}
> "threat.triage": {
> "score": 10.0,
> "rules": [
> {
> "name": "Abnormal Value",
> "comment" : "For 10.0.0.1; the value 101 exceeds threshold of 42",
> "score" : 10
> }
> ]
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (METRON-688) Allow Threat Triage Comment Field to Contain Stellar Expressions
[ https://issues.apache.org/jira/browse/METRON-688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850184#comment-15850184 ] Nick Allen commented on METRON-688: --- May be better to create an additional field like 'reason' or 'context' (or anything else) where additional context is provided instead of changing the use of 'comment'. Could still find use for a general, fixed description of the rule, which is what 'comment' provides. > Allow Threat Triage Comment Field to Contain Stellar Expressions > > > Key: METRON-688 > URL: https://issues.apache.org/jira/browse/METRON-688 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen > > Allow the Threat Triage comment field to contain a Stellar expression that > can be executed. This will allow a rule to produce a message with contextual > information like the specific values that caused the rule to trigger. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-686) Record of Rule Set that Fired During Threat Triage
Nick Allen created METRON-686:
-
Summary: Record of Rule Set that Fired During Threat Triage
Key: METRON-686
URL: https://issues.apache.org/jira/browse/METRON-686
Project: Metron
Issue Type: Improvement
Reporter: Nick Allen
Assignee: Nick Allen
h3. Problem
There is little transparency into the Threat Triage process itself. When
Threat Triage runs, all I get is a score. I don't know how that score was
arrived at, which rules were triggered, and the specific values that caused a
rule to trigger.
More specifically, there is no way to generate a message that looks like "The
host 'powned.svr.bank.com' has '230' inbound flows, exceeding the threshold of
'202'". This makes it difficult for an analyst to action the alert.
h3. Proposed Solution
To improve the transparency of the Threat Triage process, I am proposing these
enhancements.
(1) Threat Triage should attach to each message all of the rules that fired in
addition to the total calculated threat triage score.
(2) Threat Triage should allow a custom message to be generated for each rule.
The custom message would allow for some form of string interpolation so that I
can add specific values from each message to the generated alert. We could
allow this in one or both of the new fields that Casey just added, name and
comment.
(3) The specific method of string interpolation will be implemented under a
separate issue.
h3. Example
(1) In this example, we have a telemetry message with a field called 'value'
that we need to monitor. In Enrichment, I calculate some sort of value
threshold, over which an alert should be generated.
(2) In Threat Triage, I use the calculated value threshold to alert on any
message that has a value exceeding this threshold.
(3) I can embed values from the message, like the hostname, value, and value
threshold, into the alert produced by Threat Triage. Notice that I am using
{noformat}${this}{noformat} for string interpolation, but it could be any
syntax that we choose.
{code}
"triageConfig" : {
"riskLevelRules" : [
{
"name" : "Abnormal Value",
"comment" : "For ${hostname}; the value ${value} exceeds threshold of
${value_threshold}",
"rule" : "value > value_threshold",
"score" : 10
}
],
"aggregator" : "MAX"
}
{code}
(4) The Threat Triage process today would add only the total calculated score.
{code}
"threat.triage.level": 10.0
{code}
With this proposal, Threat Triage would add the following to the message.
Notice how each of the {noformat}${variables}{noformat} have been replaced with
the actual values extracted from the message. This allows for more contextual
information to action the alert.
{code}
"threat.triage": {
"score": 10.0,
"rules": [
{
"name": "Abnormal Value",
"comment" : "For 10.0.0.1; the value 101 exceeds threshold of 42",
"score" : 10
}
]
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] (METRON-676) Create Zeppelin Notebook for YAF Telemetry
Title: Message Title Nick Allen commented on METRON-676 Re: Create Zeppelin Notebook for YAF Telemetry Sure, I'll just send a link to this JIRA. Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] [Updated] (METRON-676) Create Zeppelin Notebook for YAF Telemetry
[ https://issues.apache.org/jira/browse/METRON-676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-676: -- Attachment: Metron - YAF Telemetry - Report Mode.png What the notebook looks-like in report mode, which is much cleaner, read-only view of the same notebook. > Create Zeppelin Notebook for YAF Telemetry > -- > > Key: METRON-676 > URL: https://issues.apache.org/jira/browse/METRON-676 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Assignee: Nick Allen > Fix For: 0.3.1 > > Attachments: Metron - YAF Telemetry.png, Metron - YAF Telemetry - > Report Mode.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-676) Create Zeppelin Notebook for YAF Telemetry
[ https://issues.apache.org/jira/browse/METRON-676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-676: -- Attachment: Metron - YAF Telemetry.png Attached a screen capture of what the Notebook looks like when it is backed with roughly 7 days worth of data. > Create Zeppelin Notebook for YAF Telemetry > -- > > Key: METRON-676 > URL: https://issues.apache.org/jira/browse/METRON-676 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Assignee: Nick Allen > Attachments: Metron - YAF Telemetry.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-676) Create Zeppelin Notebook for YAF Telemetry
Nick Allen created METRON-676: - Summary: Create Zeppelin Notebook for YAF Telemetry Key: METRON-676 URL: https://issues.apache.org/jira/browse/METRON-676 Project: Metron Issue Type: Sub-task Reporter: Nick Allen Assignee: Nick Allen -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-346) Zeppelin Notebooks for Query Data Services
[ https://issues.apache.org/jira/browse/METRON-346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-346: -- Attachment: Metron - YAF Telemetry.json Metron - YAF Telemetry.png A screenshot of the YAF Telemetry Zeppelin Notebook along with the Notebook itself. > Zeppelin Notebooks for Query Data Services > -- > > Key: METRON-346 > URL: https://issues.apache.org/jira/browse/METRON-346 > Project: Metron > Issue Type: New Feature >Reporter: George Vetticaden >Assignee: Nick Allen > Labels: platform > Attachments: Metron - YAF Telemetry.json, Metron - YAF Telemetry.png, > Welcome to Metron - Analyzing Telemetry.json > > > Create Some out of the box Zeppelin Notebooks / Dashboards for teh data int > eh Security Vault -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-674) Archived Telemetry Filenames in HDFS Contain 'Null'
Nick Allen created METRON-674:
-
Summary: Archived Telemetry Filenames in HDFS Contain 'Null'
Key: METRON-674
URL: https://issues.apache.org/jira/browse/METRON-674
Project: Metron
Issue Type: Bug
Affects Versions: 0.3.0
Reporter: Nick Allen
Priority: Minor
When running "Quick Dev", I have noticed that all of the archived telemetry
files in HDFS contain null in the name.
{code}
[root@node1 0.3.0]# hdfs dfs -ls /apps/metron/indexing/indexed/*
Found 2 items
-rw-r--r-- 1 storm hadoop 644753 2017-01-19 17:47
/apps/metron/indexing/indexed/bro/enrichment-null-0-0-1484847868551.json
-rw-r--r-- 1 storm hadoop 14107767 2017-01-19 18:46
/apps/metron/indexing/indexed/bro/enrichment-null-0-0-1484848728527.json
Found 5 items
-rwxrwxrwx 1 storm hadoop 205699 2017-01-16 21:57
/apps/metron/indexing/indexed/snort/enrichment-null-0-0-1484603710250.json
-rwxrwxrwx 1 storm hadoop5773871 2017-01-17 14:34
/apps/metron/indexing/indexed/snort/enrichment-null-0-0-1484603925156.json
-rwxrwxrwx 1 storm hadoop 253870 2017-01-17 13:43
/apps/metron/indexing/indexed/snort/enrichment-null-0-0-1484660437793.json
-rwxrwxrwx 1 storm hadoop 24023035 2017-01-17 19:45
/apps/metron/indexing/indexed/snort/enrichment-null-0-0-1484660672723.json
-rwxrwxrwx 1 storm hadoop2063857 2017-01-17 19:02
/apps/metron/indexing/indexed/snort/enrichment-null-0-0-1484679265343.json
Found 147 items
-rwxrwxrwx 1 storm hadoop 18199681 2017-01-18 16:35
/apps/metron/indexing/indexed/yaf/enrichment-null-0-0-1484752251563.json
-rwxrwxrwx 1 storm hadoop 216895 2017-01-19 17:47
/apps/metron/indexing/indexed/yaf/enrichment-null-0-0-1484846918122.json
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (METRON-670) Monit Incorrectly Reports Status
Nick Allen created METRON-670: - Summary: Monit Incorrectly Reports Status Key: METRON-670 URL: https://issues.apache.org/jira/browse/METRON-670 Project: Metron Issue Type: Bug Reporter: Nick Allen Assignee: Nick Allen In a constrained environment, like 'Quick Dev', Monit will often incorrectly report the status of a Metron topology. This occurs when the environment is under load and a query of topology status exceeds the default timeout of 30 seconds. Need to add a parameter so that the timeout for a status check can be extended under these conditions. This was previously done for starting and stopping a topology, but not for a status checks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-659) Emulate Sensors in Development Environments
Nick Allen created METRON-659: - Summary: Emulate Sensors in Development Environments Key: METRON-659 URL: https://issues.apache.org/jira/browse/METRON-659 Project: Metron Issue Type: Improvement Affects Versions: 0.3.0 Reporter: Nick Allen Assignee: Nick Allen Fix For: Next + 1 Replace the Snort, Bro, and YAF sensors on the "Quick Dev" and "Full Dev" environments with a mechanism that consumes less resources. These environments are notoriously difficult to work with because the installed services consume nearly all available memory. The Bro, YAF, and Snort sensors, along with the PCAP Replay service, consume a considerable amount of these limited resources. Replacing these sensors with a lightweight mechanism should free-up additional resources, make the environment easier to work with, and result in faster deployments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-654) Create RPM Installer for Profiler
Nick Allen created METRON-654: - Summary: Create RPM Installer for Profiler Key: METRON-654 URL: https://issues.apache.org/jira/browse/METRON-654 Project: Metron Issue Type: Improvement Affects Versions: 0.3.0 Reporter: Nick Allen Assignee: Nick Allen Fix For: Next + 1 The Profiler currently must be installed using the Ansible deployment scripts. Prior to getting the Profiler integrated with the rest of the Ambari installation process, we need to package the Profiler in an RPM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-645) Unable to Start Fastcapa Test Environment
[
https://issues.apache.org/jira/browse/METRON-645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Allen updated METRON-645:
--
Affects Version/s: 0.3.0
Fix Version/s: Next + 1
> Unable to Start Fastcapa Test Environment
> -
>
> Key: METRON-645
> URL: https://issues.apache.org/jira/browse/METRON-645
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.3.0
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: Next + 1
>
>
> {code}
> $ vagrant up
> Bringing machine 'source' up with 'virtualbox' provider...
> Bringing machine 'sink' up with 'virtualbox' provider...
> ...
> ==> sink: Running provisioner: ansible...
> sink: Running ansible-playbook...
> ERROR! role definitions must contain a role name
> The error appears to have been in
> '/Users/nallen/Development/incubator-metron/metron-deployment/vagrant/fastcapa-test-platform/playbook.yml':
> line 30, column 7, but may
> be elsewhere in the file depending on the exact syntax problem.
> The offending line appears to be:
> - { role: sensor-test-mode, pcap_replay: True, install_yaf: False,
> install_snort: False }
> - service: name=pcap-replay state=started
> ^ here
> Ansible failed to complete successfully. Any error output should be
> visible above. Please fix these errors and try again.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (METRON-645) Unable to Start Fastcapa Test Environment
[
https://issues.apache.org/jira/browse/METRON-645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15795799#comment-15795799
]
Nick Allen commented on METRON-645:
---
Also, running into a problem building DPDK 2.2.0 on the later versions of
CentOS 6.
{code}
== Build lib/librte_eal/linuxapp/igb_uio
LD
/root/dpdk-2.2.0/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio/built-in.o
CC [M]
/root/dpdk-2.2.0/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio/igb_uio.o
/root/dpdk-2.2.0/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio/igb_uio.c:
In function ‘igbuio_msix_mask_irq’:
/root/dpdk-2.2.0/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio/igb_uio.c:213:
error: ‘PCI_MSIX_ENTRY_CTRL_MASKBIT’ undeclared (first use in this function)
/root/dpdk-2.2.0/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio/igb_uio.c:213:
error: (Each undeclared identifier is reported only once
/root/dpdk-2.2.0/x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio/igb_uio.c:213:
error: for each function it appears in.)
{code}
> Unable to Start Fastcapa Test Environment
> -
>
> Key: METRON-645
> URL: https://issues.apache.org/jira/browse/METRON-645
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>
> {code}
> $ vagrant up
> Bringing machine 'source' up with 'virtualbox' provider...
> Bringing machine 'sink' up with 'virtualbox' provider...
> ...
> ==> sink: Running provisioner: ansible...
> sink: Running ansible-playbook...
> ERROR! role definitions must contain a role name
> The error appears to have been in
> '/Users/nallen/Development/incubator-metron/metron-deployment/vagrant/fastcapa-test-platform/playbook.yml':
> line 30, column 7, but may
> be elsewhere in the file depending on the exact syntax problem.
> The offending line appears to be:
> - { role: sensor-test-mode, pcap_replay: True, install_yaf: False,
> install_snort: False }
> - service: name=pcap-replay state=started
> ^ here
> Ansible failed to complete successfully. Any error output should be
> visible above. Please fix these errors and try again.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (METRON-645) Unable to Start Fastcapa Test Environment
Nick Allen created METRON-645:
-
Summary: Unable to Start Fastcapa Test Environment
Key: METRON-645
URL: https://issues.apache.org/jira/browse/METRON-645
Project: Metron
Issue Type: Bug
Reporter: Nick Allen
Assignee: Nick Allen
{code}
$ vagrant up
Bringing machine 'source' up with 'virtualbox' provider...
Bringing machine 'sink' up with 'virtualbox' provider...
...
==> sink: Running provisioner: ansible...
sink: Running ansible-playbook...
ERROR! role definitions must contain a role name
The error appears to have been in
'/Users/nallen/Development/incubator-metron/metron-deployment/vagrant/fastcapa-test-platform/playbook.yml':
line 30, column 7, but may
be elsewhere in the file depending on the exact syntax problem.
The offending line appears to be:
- { role: sensor-test-mode, pcap_replay: True, install_yaf: False,
install_snort: False }
- service: name=pcap-replay state=started
^ here
Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (METRON-590) Enable Use of Event Time in Profiler
[ https://issues.apache.org/jira/browse/METRON-590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15795259#comment-15795259 ] Nick Allen commented on METRON-590: --- Per previous community discussion, the work of refactoring the ConfiguredBolt is a prerequisite. > Enable Use of Event Time in Profiler > > > Key: METRON-590 > URL: https://issues.apache.org/jira/browse/METRON-590 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen > > There are at least two different times that are important to consider when > handling the telemetry messages received by Metron. > (1) Processing time is the time at which Metron processed the message. > (2) Event time is the time at which the event actually occurred. > If Metron is consuming live data and all is well, the processing and event > times may remain close and consistent. When processing time differs from > event time the data produced by the Profiler may be inaccurate. There are a > few scenarios under which these times might differ greatly which would > negatively impact the feature set produced by the Profiler. > (1) When the system has experienced an outage, for example, a scheduled > maintenance window. When restarted a high volume of messages will need to be > processed by the Profiler. The output of the Profiler will indicate an > increase in activity, although no change in activity actually occurred on the > target network. This could happen whether the outage was Metron itself or an > upstream system that feeds data to Metron. > (2) If the user attempts to replay historical telemetry through the Profiler, > the Profiler will attribute the activity to the time period in which it was > processed. Obviously the activity should be attributed to the time period in > which the raw telemetry events originated in. > There are some scenarios when processing time might be preferred and other > use cases where event time is preferred. The Profiler should be enhanced to > allow it to produce profiles based on either processing time or event time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-626) Add Higher-Order Map Function to Stellar
Nick Allen created METRON-626:
-
Summary: Add Higher-Order Map Function to Stellar
Key: METRON-626
URL: https://issues.apache.org/jira/browse/METRON-626
Project: Metron
Issue Type: Improvement
Reporter: Nick Allen
It would be extremely useful to have a 'map' higher-order function in Stellar.
A 'flat map' might also be needed.
[https://en.wikipedia.org/wiki/Map_(higher-order_function)]
A few examples that I am thinking of. Ignore my choice of syntax. I am just
trying to express the use cases here.
Fetch all of the profile measurements over the past 2 hours. There may be
many. What is the mean of each of those measurements?
{code}
stats := PROFILE_GET('p1', 'e1', 2, "HOURS")
stats.MAP( s -> STATS_MEAN(s))
{code}
Get profile measurements taken 1 week ago, 2 weeks ago, and 3 weeks ago, so
that I can then compare them.
{code}
{ "1 week", "2 weeks", "3 weeks" }.MAP( lookback -> PROFILE_GET('p1','e1',
lookback)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (METRON-585) Pcap Replay Fails to Stop
[ https://issues.apache.org/jira/browse/METRON-585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-585: -- Affects Version/s: 0.3.0 Fix Version/s: Next + 1 > Pcap Replay Fails to Stop > - > > Key: METRON-585 > URL: https://issues.apache.org/jira/browse/METRON-585 > Project: Metron > Issue Type: Bug >Affects Versions: 0.3.0 >Reporter: Nick Allen >Assignee: Nick Allen > Fix For: Next + 1 > > > The service script for Pcap Replay often fails to actually stop the tcpreplay > process. > The `tcpreplay` process is a huge consumer of resources when running on Quick > Dev and actually having it stop when asked, would be super nice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-613) Support Versioning of Values Written by the Profiler
Nick Allen created METRON-613: - Summary: Support Versioning of Values Written by the Profiler Key: METRON-613 URL: https://issues.apache.org/jira/browse/METRON-613 Project: Metron Issue Type: Improvement Reporter: Nick Allen The Profiler can write any serializable object, not just numeric values. This has tremendous value when leveraged with the Stellar Stats package. See [Example 4](https://github.com/apache/incubator-metron/tree/master/metron-analytics/metron-profiler#examples) as an example. The downside of this approach is that if the objects that have been serialized change over time, the persisted data can become unreadable. If the Profiler uses one version to serialize and write the data and the Profiler Client uses a different version to deserialize and read, it is possible that the versions are not compatible and would not be readable. For example, if at one point in time the Profiler persists Stats data that is backed by the t-dunning library version 3.0, then it is possible that at a later point in time when migrating to version 3.1 the data might not be readable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-606) Profiler Overwriting Previously Written Values
[ https://issues.apache.org/jira/browse/METRON-606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-606: -- Affects Version/s: 0.3.0 Fix Version/s: Next + 1 > Profiler Overwriting Previously Written Values > -- > > Key: METRON-606 > URL: https://issues.apache.org/jira/browse/METRON-606 > Project: Metron > Issue Type: Bug >Affects Versions: 0.3.0 >Reporter: Nick Allen >Assignee: Nick Allen > Fix For: Next + 1 > > > The Profiler is repetitively overwriting previously written values. Where I > would expect to see a series of multiple values in Hbase, one written per > flush, there is only a single value. This same value is being repetitively > overwritten with new data. > This is occurring because the same ProfileMeasurement is being re-used after > a flush event. Prior to METRON-575, a new ProfileMeasurement was created > after a flush event. Without the new ProfileMeasurement, the timestamp used > to generate the row key remains the same. > Drat. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-490) Stellar - Validation of Required Parameters
[ https://issues.apache.org/jira/browse/METRON-490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723053#comment-15723053 ] Nick Allen commented on METRON-490: --- Will not fix as described in PR. > Stellar - Validation of Required Parameters > --- > > Key: METRON-490 > URL: https://issues.apache.org/jira/browse/METRON-490 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen > Fix For: 0.3.0 > > > Currently, each Stellar function handles validation of required function > parameters in its own way. In some cases, we have functions that throw an > IndexOutOfBoundsException, in other cases the function will produce its own > error message and exception. > There needs to be a standard mechanism to validate required function > parameters. This mechanism should handle missing parameters gracefully and > provide an error message that makes sense to a user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-575) State from different profiles can be co-mingled incorrectly
[ https://issues.apache.org/jira/browse/METRON-575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-575: -- Affects Version/s: 0.3.0 Fix Version/s: Next + 1 > State from different profiles can be co-mingled incorrectly > --- > > Key: METRON-575 > URL: https://issues.apache.org/jira/browse/METRON-575 > Project: Metron > Issue Type: Bug >Affects Versions: 0.3.0 >Reporter: Nick Allen >Assignee: Nick Allen > Fix For: Next + 1 > > > The ProfileBuilderBolt incorrectly assumes that it will only ever see a > single [profile, entity] pair. The bolt maintains a single StellarExecutor > that is responsible for executing the init, update, result expressions. This > assumption is incorrect as Storm's field grouping only guarantees that the > same profile/entity pairs will go to the same task. Storm does not guarantee > that a task only receives a single profile/entity pair. > The easiest fix is to maintain a cache that maps a profile/entity to its > state. This would follow what is currently done in the Join bolt. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-557) Create Stellar Functions for Kafka
[ https://issues.apache.org/jira/browse/METRON-557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-557: -- Affects Version/s: 0.3.0 Fix Version/s: Next + 1 > Create Stellar Functions for Kafka > -- > > Key: METRON-557 > URL: https://issues.apache.org/jira/browse/METRON-557 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.0 >Reporter: Nick Allen >Assignee: Nick Allen > Fix For: Next + 1 > > > Add functions to read and write messages to Kafka topics. > This should allow a user the to debug a Grok expression or Enrichment > configuration inside the REPL. For example, I can pull a message off of the > error topic to see why it failed to parse. Then fix the configuration and > resubmit the message to try again. I also won’t have to go outside of the > Stellar REPL to monitor topic activity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-556) Profiler - Refactor 'Group By' Calculation
[ https://issues.apache.org/jira/browse/METRON-556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-556: -- Affects Version/s: 0.3.0 Fix Version/s: Next + 1 > Profiler - Refactor 'Group By' Calculation > -- > > Key: METRON-556 > URL: https://issues.apache.org/jira/browse/METRON-556 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.0 >Reporter: Nick Allen >Assignee: Nick Allen > Fix For: Next + 1 > > > Currently a Profile's 'group by' expression is calculated in the context of > the HBaseBolt. Specifically it is calculated in the ProfileHBaseMapper. > This mapper is run by the HBaseBolt to define how the Profile values stored > in a tuple get written to HBase.This can be refactored so that the 'group > by' expression is calculated at the same time as the 'result' expression in > the context of the ProfileBuilderBolt. > This has a few advantages. It allows any errors encountered in executing the > 'group by' expression to be contained within the ProfileBuilderBolt instance > responsible for the profile/entity pair alone. Any exceptions cannot impact > the functioning of the rest of the Profiler. This is also more consistent > and simple as the HBaseBolt will receive everything that it needs, > pre-calculated, in the form of a ProfileMeasurement object. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-570) Add Profiler Link to README
[ https://issues.apache.org/jira/browse/METRON-570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-570: -- Fix Version/s: 0.3.0 > Add Profiler Link to README > --- > > Key: METRON-570 > URL: https://issues.apache.org/jira/browse/METRON-570 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Minor > Fix For: 0.3.0 > > > Add a link to the Profiler as part of the project's root README. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-576) Stellar function resolution takes too long on running cluster
[ https://issues.apache.org/jira/browse/METRON-576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-576: -- Affects Version/s: 0.3.0 Fix Version/s: Next + 1 > Stellar function resolution takes too long on running cluster > - > > Key: METRON-576 > URL: https://issues.apache.org/jira/browse/METRON-576 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.0 >Reporter: Nick Allen >Assignee: Nick Allen > Fix For: Next + 1 > > > When running the Stellar REPL in a cluster on AWS, function resolution takes > 50-60 seconds. The user is not able to execute any functions in the REPL > until this process completes. > The default function resolver searches the classpath for Stellar functions. > The delay may be because there are just too many classes in the classpath to > search on a running cluster. As more libraries are added as dependencies > under /usr/metron//lib this problem just gets worse. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-597) Sporadic Failures of Profiler Integration Tests
[
https://issues.apache.org/jira/browse/METRON-597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Allen updated METRON-597:
--
Affects Version/s: 0.3.0
Fix Version/s: Next + 1
> Sporadic Failures of Profiler Integration Tests
> ---
>
> Key: METRON-597
> URL: https://issues.apache.org/jira/browse/METRON-597
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.3.0
>Reporter: Nick Allen
>Assignee: Nick Allen
> Fix For: Next + 1
>
>
> Seems to be some kind of timing issue.
> {code}
> ---
> T E S T S
> ---
> Running org.apache.metron.profiler.integration.ProfilerIntegrationTest
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 66.564 sec
> <<< FAILURE!
> testExample3(org.apache.metron.profiler.integration.ProfilerIntegrationTest)
> Time elapsed: 16.759 sec <<< FAILURE!
> java.lang.AssertionError: expected:<20.0> but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:553)
> at org.junit.Assert.assertEquals(Assert.java:683)
> at
> org.apache.metron.profiler.integration.ProfilerIntegrationTest.testExample3(ProfilerIntegrationTest.java:203)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
> at
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
> at
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
> Results :
> Failed tests:
> testExample3(org.apache.metron.profiler.integration.ProfilerIntegrationTest):
> expected:<20.0> but was:
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (METRON-601) MockHTable Put Log is Not Thread Safe
[ https://issues.apache.org/jira/browse/METRON-601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-601: -- Fix Version/s: Next + 1 > MockHTable Put Log is Not Thread Safe > - > > Key: METRON-601 > URL: https://issues.apache.org/jira/browse/METRON-601 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Assignee: Nick Allen > Fix For: Next + 1 > > > The MockHTable uses an ArrayList to store a log of Puts that have been > submitted against the MockHTable. The MockHTable, along with the put log, is > accessed from multiple threads during the integration tests. Access to the > Put log is not thread safe, which is likely at least one cause of METRON-597. > The Put log is used by multiple tests, but more so by the > ProfilerIntegrationTest. This tests polls the list to block the thread until > the expected number of Puts have been submitted. This is likely why this > test is more impacted by this issue than others. > The Put Log needs to made thread safe. See > `org.apache.metron.test.mock.MockHTable.putLog` -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-601) MockHTable Put Log is Not Thread Safe
[ https://issues.apache.org/jira/browse/METRON-601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-601: -- Fixed by METRON-601 > MockHTable Put Log is Not Thread Safe > - > > Key: METRON-601 > URL: https://issues.apache.org/jira/browse/METRON-601 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Assignee: Nick Allen > > The MockHTable uses an ArrayList to store a log of Puts that have been > submitted against the MockHTable. The MockHTable, along with the put log, is > accessed from multiple threads during the integration tests. Access to the > Put log is not thread safe, which is likely at least one cause of METRON-597. > The Put log is used by multiple tests, but more so by the > ProfilerIntegrationTest. This tests polls the list to block the thread until > the expected number of Puts have been submitted. This is likely why this > test is more impacted by this issue than others. > The Put Log needs to made thread safe. See > `org.apache.metron.test.mock.MockHTable.putLog` -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-364) Stellar - Adding Two Integers Does Not Equal an Integer
[
https://issues.apache.org/jira/browse/METRON-364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723021#comment-15723021
]
Nick Allen commented on METRON-364:
---
It does seem to still be a problem. I had a Stellar REPL running that I could
check this on real quick.
```
[Stellar]>>> TO_INTEGER(2 + 2)
4
[Stellar]>>> 2 + 2
4.0
```
> Stellar - Adding Two Integers Does Not Equal an Integer
> ---
>
> Key: METRON-364
> URL: https://issues.apache.org/jira/browse/METRON-364
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
> Labels: stellar
>
> A test showing the problem which would be part of
> org.apache.metron.common.stellar.StellarTest
> ```java
> @Test
> public void testAddIntegers() throws Exception {
> String query = "1 + 1";
> Assert.assertEquals(2, run(query, new HashMap<>()));
> //java.lang.AssertionError:
> //Expected :2
> //Actual :2.0
> }
> ```
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (METRON-364) Stellar - Adding Two Integers Does Not Equal an Integer
[
https://issues.apache.org/jira/browse/METRON-364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723015#comment-15723015
]
Nick Allen commented on METRON-364:
---
[~jjmeyer0] Feel free to tackle this. I do not know of anyone working this.
This *might* already be addressed by another fix that was made. Even if it has
been fixed, it would be valuable if you could confirm. If it is, mark the JIRA
as fixed and close it.
Thanks
> Stellar - Adding Two Integers Does Not Equal an Integer
> ---
>
> Key: METRON-364
> URL: https://issues.apache.org/jira/browse/METRON-364
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
> Labels: stellar
>
> A test showing the problem which would be part of
> org.apache.metron.common.stellar.StellarTest
> ```java
> @Test
> public void testAddIntegers() throws Exception {
> String query = "1 + 1";
> Assert.assertEquals(2, run(query, new HashMap<>()));
> //java.lang.AssertionError:
> //Expected :2
> //Actual :2.0
> }
> ```
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (METRON-532) Define Profile Period When Calling PROFILE_GET
[ https://issues.apache.org/jira/browse/METRON-532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722347#comment-15722347 ] Nick Allen commented on METRON-532: --- If it makes life easy, I would be glad to test it myself on Quick Dev. Just let me know. > Define Profile Period When Calling PROFILE_GET > -- > > Key: METRON-532 > URL: https://issues.apache.org/jira/browse/METRON-532 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Matt Foley > Fix For: 0.3.0 > > > The Profiler Client currently offers the PROFILE_GET Stellar function to > access profile data. The work done for METRON-529 allowed the user to > customize the profile period using Metron global properties. > A user may need to access historical profiles with different durations and > would want to specify the period as part of the call to the Profiler Client, > rather than in the Metron global properties. > This would be especially necessary should METRON-530 be completed allowing > different profiles to use different period durations simultaneously. > There is some discussion of this attached to METRON-529. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-606) Profiler Overwriting Previously Written Values
Nick Allen created METRON-606: - Summary: Profiler Overwriting Previously Written Values Key: METRON-606 URL: https://issues.apache.org/jira/browse/METRON-606 Project: Metron Issue Type: Bug Reporter: Nick Allen Assignee: Nick Allen The Profiler is repetitively overwriting previously written values. Where I would expect to see a series of multiple values in Hbase, one written per flush, there is only a single value. This same value is being repetitively overwritten with new data. This is occurring because the same ProfileMeasurement is being re-used after a flush event. Prior to METRON-575, a new ProfileMeasurement was created after a flush event. Without the new ProfileMeasurement, the timestamp used to generate the row key remains the same. Drat. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-602) MockHTable Scanner Not Returning Expected Results
[
https://issues.apache.org/jira/browse/METRON-602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15713040#comment-15713040
]
Nick Allen commented on METRON-602:
---
What's happening here is that sometimes the Profiler flushes a time or two more
than would be expected. An extra flush would occur between when the size of
the put log was validated and when the put log was actually retrieved to
validate the actual results. Just need to make sure the validation code is
less opinionated about how many flushes occurred.
> MockHTable Scanner Not Returning Expected Results
> -
>
> Key: METRON-602
> URL: https://issues.apache.org/jira/browse/METRON-602
> Project: Metron
> Issue Type: Sub-task
>Reporter: Nick Allen
>
> In some cases, even when the MockHTable Put Log shows that the expected data
> has been submitted to the table, the scanner implementation does not return
> the expected results.
> In this branch
> (https://github.com/nickwallen/incubator-metron/tree/METRON-597-DEBUG) I have
> added some additional debug statements that query the Put Log. In validating
> the results the test uses a scanner to retrieve results. The results
> retrieved do not match what was just extracted from the Put Log.
> {code}
> ---
> T E S T S
> ---
> Running org.apache.metron.profiler.integration.ProfilerIntegrationTest
> read
> put:
> {"totalColumns":1,"row":"\\xFF\\xFF\\xFF\\xF7example310.0.0.2\\x00\\x00\\x00\\x00\\x11\\xA6E\\xE8","families":{"P":[{"qualifier":"value","vlen":9,"tag":[],"timestamp":1480547731528}]}}
> cell:
> \xFF\xFF\xFF\xF7example310.0.0.2\x00\x00\x00\x00\x11\xA6E\xE8/P:value/1480547731528/Put/vlen=9/seqid=0
> value: 20.0
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 60.359 sec
> <<< FAILURE!
> testExample3(org.apache.metron.profiler.integration.ProfilerIntegrationTest)
> Time elapsed: 16.272 sec <<< FAILURE!
> java.lang.AssertionError: expected:<20.0> but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:553)
> at org.junit.Assert.assertEquals(Assert.java:683)
> at
> org.apache.metron.profiler.integration.ProfilerIntegrationTest.testExample3(ProfilerIntegrationTest.java:215)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
> at
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
> at
>
[jira] [Issue Comment Deleted] (METRON-602) MockHTable Scanner Not Returning Expected Results
[
https://issues.apache.org/jira/browse/METRON-602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Allen updated METRON-602:
--
Comment: was deleted
(was: What's happening here is that sometimes the Profiler flushes a time or
two more than would be expected. An extra flush would occur between when the
size of the put log was validated and when the put log was actually retrieved
to validate the actual results. Just need to make sure the validation code is
less opinionated about how many flushes occurred.)
> MockHTable Scanner Not Returning Expected Results
> -
>
> Key: METRON-602
> URL: https://issues.apache.org/jira/browse/METRON-602
> Project: Metron
> Issue Type: Sub-task
>Reporter: Nick Allen
>
> In some cases, even when the MockHTable Put Log shows that the expected data
> has been submitted to the table, the scanner implementation does not return
> the expected results.
> In this branch
> (https://github.com/nickwallen/incubator-metron/tree/METRON-597-DEBUG) I have
> added some additional debug statements that query the Put Log. In validating
> the results the test uses a scanner to retrieve results. The results
> retrieved do not match what was just extracted from the Put Log.
> {code}
> ---
> T E S T S
> ---
> Running org.apache.metron.profiler.integration.ProfilerIntegrationTest
> read
> put:
> {"totalColumns":1,"row":"\\xFF\\xFF\\xFF\\xF7example310.0.0.2\\x00\\x00\\x00\\x00\\x11\\xA6E\\xE8","families":{"P":[{"qualifier":"value","vlen":9,"tag":[],"timestamp":1480547731528}]}}
> cell:
> \xFF\xFF\xFF\xF7example310.0.0.2\x00\x00\x00\x00\x11\xA6E\xE8/P:value/1480547731528/Put/vlen=9/seqid=0
> value: 20.0
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 60.359 sec
> <<< FAILURE!
> testExample3(org.apache.metron.profiler.integration.ProfilerIntegrationTest)
> Time elapsed: 16.272 sec <<< FAILURE!
> java.lang.AssertionError: expected:<20.0> but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:553)
> at org.junit.Assert.assertEquals(Assert.java:683)
> at
> org.apache.metron.profiler.integration.ProfilerIntegrationTest.testExample3(ProfilerIntegrationTest.java:215)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
> at
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
> at
>
[jira] [Updated] (METRON-602) MockHTable Scanner Not Returning Expected Results
[
https://issues.apache.org/jira/browse/METRON-602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Allen updated METRON-602:
--
Description:
In some cases, even when the MockHTable Put Log shows that the expected data
has been submitted to the table, the scanner implementation does not return the
expected results.
In this branch
(https://github.com/nickwallen/incubator-metron/tree/METRON-597-DEBUG) I have
added some additional debug statements that query the Put Log. In validating
the results the test uses a scanner to retrieve results. The results retrieved
do not match what was just extracted from the Put Log.
{code}
---
T E S T S
---
Running org.apache.metron.profiler.integration.ProfilerIntegrationTest
read
put:
{"totalColumns":1,"row":"\\xFF\\xFF\\xFF\\xF7example310.0.0.2\\x00\\x00\\x00\\x00\\x11\\xA6E\\xE8","families":{"P":[{"qualifier":"value","vlen":9,"tag":[],"timestamp":1480547731528}]}}
cell:
\xFF\xFF\xFF\xF7example310.0.0.2\x00\x00\x00\x00\x11\xA6E\xE8/P:value/1480547731528/Put/vlen=9/seqid=0
value: 20.0
Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 60.359 sec <<<
FAILURE!
testExample3(org.apache.metron.profiler.integration.ProfilerIntegrationTest)
Time elapsed: 16.272 sec <<< FAILURE!
java.lang.AssertionError: expected:<20.0> but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:553)
at org.junit.Assert.assertEquals(Assert.java:683)
at
org.apache.metron.profiler.integration.ProfilerIntegrationTest.testExample3(ProfilerIntegrationTest.java:215)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
Results :
Failed tests:
testExample3(org.apache.metron.profiler.integration.ProfilerIntegrationTest):
expected:<20.0> but was:
{code}
This *could* be due to METRON-601, but this could also be a separate bug. More
investigation needed.
was:
In some cases, even when the MockHTable Put Log shows that the expected data
has been submitted to the table, the scanner implementation does not return the
expected results.
In this branch
(https://github.com/nickwallen/incubator-metron/tree/METRON-597-FIX) I have
added some additional debug statements that
[jira] [Created] (METRON-602) MockHTable Scanner Not Returning Expected Results
Nick Allen created METRON-602:
-
Summary: MockHTable Scanner Not Returning Expected Results
Key: METRON-602
URL: https://issues.apache.org/jira/browse/METRON-602
Project: Metron
Issue Type: Sub-task
Reporter: Nick Allen
In some cases, even when the MockHTable Put Log shows that the expected data
has been submitted to the table, the scanner implementation does not return the
expected results.
In this branch
(https://github.com/nickwallen/incubator-metron/tree/METRON-597-FIX) I have
added some additional debug statements that query the Put Log. In validating
the results the test uses a scanner to retrieve results. The results retrieved
do not match what was just extracted from the Put Log.
{code}
---
T E S T S
---
Running org.apache.metron.profiler.integration.ProfilerIntegrationTest
read
put:
{"totalColumns":1,"row":"\\xFF\\xFF\\xFF\\xF7example310.0.0.2\\x00\\x00\\x00\\x00\\x11\\xA6E\\xE8","families":{"P":[{"qualifier":"value","vlen":9,"tag":[],"timestamp":1480547731528}]}}
cell:
\xFF\xFF\xFF\xF7example310.0.0.2\x00\x00\x00\x00\x11\xA6E\xE8/P:value/1480547731528/Put/vlen=9/seqid=0
value: 20.0
Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 60.359 sec <<<
FAILURE!
testExample3(org.apache.metron.profiler.integration.ProfilerIntegrationTest)
Time elapsed: 16.272 sec <<< FAILURE!
java.lang.AssertionError: expected:<20.0> but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:553)
at org.junit.Assert.assertEquals(Assert.java:683)
at
org.apache.metron.profiler.integration.ProfilerIntegrationTest.testExample3(ProfilerIntegrationTest.java:215)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
Results :
Failed tests:
testExample3(org.apache.metron.profiler.integration.ProfilerIntegrationTest):
expected:<20.0> but was:
{code}
This *could* be due to METRON-601, but this could also be a separate bug. More
investigation needed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (METRON-601) MockHTable Put Log is Not Thread Safe
Nick Allen created METRON-601: - Summary: MockHTable Put Log is Not Thread Safe Key: METRON-601 URL: https://issues.apache.org/jira/browse/METRON-601 Project: Metron Issue Type: Sub-task Reporter: Nick Allen Assignee: Nick Allen The MockHTable uses an ArrayList to store a log of Puts that have been submitted against the MockHTable. The MockHTable, along with the put log, is accessed from multiple threads during the integration tests. Access to the Put log is not thread safe, which is likely at least one cause of METRON-597. The Put log is used by multiple tests, but more so by the ProfilerIntegrationTest. This tests polls the list to block the thread until the expected number of Puts have been submitted. This is likely why this test is more impacted by this issue than others. The Put Log needs to made thread safe. See `org.apache.metron.test.mock.MockHTable.putLog` -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (METRON-597) Sporadic Failures of Profiler Integration Tests
[
https://issues.apache.org/jira/browse/METRON-597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Allen reassigned METRON-597:
-
Assignee: Nick Allen
> Sporadic Failures of Profiler Integration Tests
> ---
>
> Key: METRON-597
> URL: https://issues.apache.org/jira/browse/METRON-597
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>
> Seems to be some kind of timing issue.
> {code}
> ---
> T E S T S
> ---
> Running org.apache.metron.profiler.integration.ProfilerIntegrationTest
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 66.564 sec
> <<< FAILURE!
> testExample3(org.apache.metron.profiler.integration.ProfilerIntegrationTest)
> Time elapsed: 16.759 sec <<< FAILURE!
> java.lang.AssertionError: expected:<20.0> but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:553)
> at org.junit.Assert.assertEquals(Assert.java:683)
> at
> org.apache.metron.profiler.integration.ProfilerIntegrationTest.testExample3(ProfilerIntegrationTest.java:203)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
> at
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
> at
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
> Results :
> Failed tests:
> testExample3(org.apache.metron.profiler.integration.ProfilerIntegrationTest):
> expected:<20.0> but was:
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (METRON-597) Sporadic Failures of Profiler Integration Tests
[
https://issues.apache.org/jira/browse/METRON-597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15708852#comment-15708852
]
Nick Allen commented on METRON-597:
---
Travis passed in my branch after disabling the ProfilerIntegrationTests. This
being sporadic, can't be sure, but good news so far. Would much prefer this be
specific to the profiler tests than some deep-rooted bug affecting all
integration tests. I'll open a PR, which will give it another Travis CI build.
> Sporadic Failures of Profiler Integration Tests
> ---
>
> Key: METRON-597
> URL: https://issues.apache.org/jira/browse/METRON-597
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>
> Seems to be some kind of timing issue.
> {code}
> ---
> T E S T S
> ---
> Running org.apache.metron.profiler.integration.ProfilerIntegrationTest
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 66.564 sec
> <<< FAILURE!
> testExample3(org.apache.metron.profiler.integration.ProfilerIntegrationTest)
> Time elapsed: 16.759 sec <<< FAILURE!
> java.lang.AssertionError: expected:<20.0> but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:553)
> at org.junit.Assert.assertEquals(Assert.java:683)
> at
> org.apache.metron.profiler.integration.ProfilerIntegrationTest.testExample3(ProfilerIntegrationTest.java:203)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
> at
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
> at
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
> Results :
> Failed tests:
> testExample3(org.apache.metron.profiler.integration.ProfilerIntegrationTest):
> expected:<20.0> but was:
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (METRON-597) Sporadic Failures of Profiler Integration Tests
[
https://issues.apache.org/jira/browse/METRON-597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15708733#comment-15708733
]
Nick Allen commented on METRON-597:
---
Well, might fix those. :) If you look at those builds, for example #1230, the
Enrichment, Asa, and Elasticsearch integration tests had not yet run. So would
have those failed subsequently?
I'll create a branch disabling the ProfileIntegrationTests just to tease this
out, and see how the Travis build fares.
> Sporadic Failures of Profiler Integration Tests
> ---
>
> Key: METRON-597
> URL: https://issues.apache.org/jira/browse/METRON-597
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>
> Seems to be some kind of timing issue.
> {code}
> ---
> T E S T S
> ---
> Running org.apache.metron.profiler.integration.ProfilerIntegrationTest
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 66.564 sec
> <<< FAILURE!
> testExample3(org.apache.metron.profiler.integration.ProfilerIntegrationTest)
> Time elapsed: 16.759 sec <<< FAILURE!
> java.lang.AssertionError: expected:<20.0> but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:553)
> at org.junit.Assert.assertEquals(Assert.java:683)
> at
> org.apache.metron.profiler.integration.ProfilerIntegrationTest.testExample3(ProfilerIntegrationTest.java:203)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
> at
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
> at
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
> Results :
> Failed tests:
> testExample3(org.apache.metron.profiler.integration.ProfilerIntegrationTest):
> expected:<20.0> but was:
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (METRON-597) Sporadic Failures of Profiler Integration Tests
[
https://issues.apache.org/jira/browse/METRON-597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15708707#comment-15708707
]
Nick Allen commented on METRON-597:
---
I'm not so sure this will solve our problems. I looked through at least the
first page of pull request test failures in Travis and while I do see a good
number of ProfilerIntegrationTest failures, I also see
ElasticsearchIntegrationTest, EnrichmentIntegrationTest, AsaIntegrationTest,
and ServiceDiscoveryIntegrationTest failures.
{code}
Build #1203 --> 8 days ago
testExample3(org.apache.metron.profiler.integration.ProfilerIntegrationTest)
Time elapsed: 16.858 sec <<< FAILURE!
Failed tests:
testExample3(org.apache.metron.profiler.integration.ProfilerIntegrationTest):
expected:<20.0> but was:
Build #1205 --> 7 days ago
Running org.apache.metron.maas.discovery.ServiceDiscoveryIntegrationTest
2016-11-23 12:33:25,177 ERROR [Curator-TreeCache-0] discovery.ServiceDiscoverer
(ServiceDiscoverer.java:updateState(171)) -
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 19.432 sec <<<
FAILURE! - in org.apache.metron.maas.discovery.ServiceDiscoveryIntegrationTest
testDiscovery(org.apache.metron.maas.discovery.ServiceDiscoveryIntegrationTest)
Time elapsed: 19.374 sec <<< FAILURE!
java.lang.AssertionError: expected:<3> but was:<0>
Failed tests: ServiceDiscoveryIntegrationTest.testDiscovery:96 expected:<3> but
was:<0>
Build #1206 --> 7 days ago
Running
org.apache.metron.elasticsearch.integration.ElasticsearchIndexingIntegrationTest
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 137.782 sec <<<
FAILURE!
test(org.apache.metron.elasticsearch.integration.ElasticsearchIndexingIntegrationTest)
Time elapsed: 137.671 sec <<< ERROR!
java.lang.RuntimeException: Took too long to complete: 120035 > 12
test(org.apache.metron.elasticsearch.integration.ElasticsearchIndexingIntegrationTest):
Took too long to complete: 120035 > 12
Build #1207 --> 7 days ago
Running org.apache.metron.profiler.integration.ProfilerIntegrationTest
Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 63.834 sec <<<
FAILURE!
testExample3(org.apache.metron.profiler.integration.ProfilerIntegrationTest)
Time elapsed: 16.956 sec <<< FAILURE!
java.lang.AssertionError: expected:<20.0> but was:
Failed tests:
testExample3(org.apache.metron.profiler.integration.ProfilerIntegrationTest):
expected:<20.0> but was:
Build #1219 --> 1 day ago
test(org.apache.metron.parsers.integration.AsaIntegrationTest) Time elapsed:
78.72 sec <<< ERROR!
java.lang.RuntimeException: Too many retries: 16
at
org.apache.metron.integration.ComponentRunner.process(ComponentRunner.java:140)
Tests in error: AsaIntegrationTest>ParserIntegrationTest.test:80 » Runtime Too
many retries: 1...
Build #1220 --> 1 day ago
Running org.apache.metron.parsers.integration.AsaIntegrationTest
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 78.738 sec <<<
FAILURE! - in org.apache.metron.parsers.integration.AsaIntegrationTest
test(org.apache.metron.parsers.integration.AsaIntegrationTest) Time elapsed:
78.738 sec <<< ERROR!
java.lang.RuntimeException: Too many retries: 16
Tests in error: AsaIntegrationTest>ParserIntegrationTest.test:80 » Runtime Too
many retries: 1...
Build #1222 -->
Running org.apache.metron.profiler.integration.ProfilerIntegrationTest
Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 57.989 sec <<<
FAILURE!
testExample3(org.apache.metron.profiler.integration.ProfilerIntegrationTest)
Time elapsed: 14.505 sec <<< FAILURE!
java.lang.AssertionError: expected:<20.0> but was:
Failed tests:
testExample3(org.apache.metron.profiler.integration.ProfilerIntegrationTest):
expected:<20.0> but was:
Build #1223 --> 19 hours ago
Running org.apache.metron.enrichment.integration.EnrichmentIntegrationTest
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 36.428 sec <<<
FAILURE! - in org.apache.metron.enrichment.integration.EnrichmentIntegrationTest
test(org.apache.metron.enrichment.integration.EnrichmentIntegrationTest) Time
elapsed: 36.423 sec <<< FAILURE!
java.lang.AssertionError: expected:<10> but was:<0>
Failed tests: EnrichmentIntegrationTest.test:173 expected:<10> but was:<0>
Build #1224 --> 19 hours ago
Running org.apache.metron.enrichment.integration.EnrichmentIntegrationTest
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 33.335 sec <<<
FAILURE! - in org.apache.metron.enrichment.integration.EnrichmentIntegrationTest
test(org.apache.metron.enrichment.integration.EnrichmentIntegrationTest) Time
elapsed: 33.328 sec <<< FAILURE!
Failed tests: EnrichmentIntegrationTest.test:173 expected:<10> but was:<0>
Build #1225 --> 19 hours ago
Running org.apache.metron.profiler.integration.ProfilerIntegrationTest
Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 75.54 sec <<<
FAILURE!
[jira] [Commented] (METRON-597) Sporadic Failures of Profiler Integration Tests
[
https://issues.apache.org/jira/browse/METRON-597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15708646#comment-15708646
]
Nick Allen commented on METRON-597:
---
But if it makes life simpler, I would not be against disabling the test.
> Sporadic Failures of Profiler Integration Tests
> ---
>
> Key: METRON-597
> URL: https://issues.apache.org/jira/browse/METRON-597
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>
> Seems to be some kind of timing issue.
> {code}
> ---
> T E S T S
> ---
> Running org.apache.metron.profiler.integration.ProfilerIntegrationTest
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 66.564 sec
> <<< FAILURE!
> testExample3(org.apache.metron.profiler.integration.ProfilerIntegrationTest)
> Time elapsed: 16.759 sec <<< FAILURE!
> java.lang.AssertionError: expected:<20.0> but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:553)
> at org.junit.Assert.assertEquals(Assert.java:683)
> at
> org.apache.metron.profiler.integration.ProfilerIntegrationTest.testExample3(ProfilerIntegrationTest.java:203)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
> at
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
> at
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
> Results :
> Failed tests:
> testExample3(org.apache.metron.profiler.integration.ProfilerIntegrationTest):
> expected:<20.0> but was:
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (METRON-597) Sporadic Failures of Profiler Integration Tests
[
https://issues.apache.org/jira/browse/METRON-597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15708642#comment-15708642
]
Nick Allen commented on METRON-597:
---
Are we sure this is the one that's really causing pain? When I've looked, many
times it has been the other integration tests failing like Enrichment,
Elasticsearch, etc. I think there are problems spanning all the integration
tests. I could be wrong though. I haven't looked at all the failures.
> Sporadic Failures of Profiler Integration Tests
> ---
>
> Key: METRON-597
> URL: https://issues.apache.org/jira/browse/METRON-597
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>
> Seems to be some kind of timing issue.
> {code}
> ---
> T E S T S
> ---
> Running org.apache.metron.profiler.integration.ProfilerIntegrationTest
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 66.564 sec
> <<< FAILURE!
> testExample3(org.apache.metron.profiler.integration.ProfilerIntegrationTest)
> Time elapsed: 16.759 sec <<< FAILURE!
> java.lang.AssertionError: expected:<20.0> but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:553)
> at org.junit.Assert.assertEquals(Assert.java:683)
> at
> org.apache.metron.profiler.integration.ProfilerIntegrationTest.testExample3(ProfilerIntegrationTest.java:203)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
> at
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
> at
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
> Results :
> Failed tests:
> testExample3(org.apache.metron.profiler.integration.ProfilerIntegrationTest):
> expected:<20.0> but was:
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (METRON-597) Sporadic Failures of Profiler Integration Tests
[
https://issues.apache.org/jira/browse/METRON-597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706744#comment-15706744
]
Nick Allen commented on METRON-597:
---
I had a theory before that this is caused by something in the `MockHTable`
class. Just a theory. I need to track this down some more.
> Sporadic Failures of Profiler Integration Tests
> ---
>
> Key: METRON-597
> URL: https://issues.apache.org/jira/browse/METRON-597
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>
> Seems to be some kind of timing issue.
> {code}
> ---
> T E S T S
> ---
> Running org.apache.metron.profiler.integration.ProfilerIntegrationTest
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 66.564 sec
> <<< FAILURE!
> testExample3(org.apache.metron.profiler.integration.ProfilerIntegrationTest)
> Time elapsed: 16.759 sec <<< FAILURE!
> java.lang.AssertionError: expected:<20.0> but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:553)
> at org.junit.Assert.assertEquals(Assert.java:683)
> at
> org.apache.metron.profiler.integration.ProfilerIntegrationTest.testExample3(ProfilerIntegrationTest.java:203)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
> at
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
> at
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
> Results :
> Failed tests:
> testExample3(org.apache.metron.profiler.integration.ProfilerIntegrationTest):
> expected:<20.0> but was:
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (METRON-597) Sporadic Failures of Profiler Integration Tests
[
https://issues.apache.org/jira/browse/METRON-597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Allen updated METRON-597:
--
Description:
Seems to be some kind of timing issue.
{code}
---
T E S T S
---
Running org.apache.metron.profiler.integration.ProfilerIntegrationTest
Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 66.564 sec <<<
FAILURE!
testExample3(org.apache.metron.profiler.integration.ProfilerIntegrationTest)
Time elapsed: 16.759 sec <<< FAILURE!
java.lang.AssertionError: expected:<20.0> but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:553)
at org.junit.Assert.assertEquals(Assert.java:683)
at
org.apache.metron.profiler.integration.ProfilerIntegrationTest.testExample3(ProfilerIntegrationTest.java:203)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
Results :
Failed tests:
testExample3(org.apache.metron.profiler.integration.ProfilerIntegrationTest):
expected:<20.0> but was:
Tests run: 5, Failures: 1, Errors: 0, Skipped: 0
{code}
was:
Seems to be some kind of timing issue.
---
T E S T S
---
Running org.apache.metron.profiler.integration.ProfilerIntegrationTest
Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 66.564 sec <<<
FAILURE!
testExample3(org.apache.metron.profiler.integration.ProfilerIntegrationTest)
Time elapsed: 16.759 sec <<< FAILURE!
java.lang.AssertionError: expected:<20.0> but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:553)
at org.junit.Assert.assertEquals(Assert.java:683)
at
org.apache.metron.profiler.integration.ProfilerIntegrationTest.testExample3(ProfilerIntegrationTest.java:203)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at
[jira] [Created] (METRON-597) Sporadic Failures of Profiler Integration Tests
Nick Allen created METRON-597:
-
Summary: Sporadic Failures of Profiler Integration Tests
Key: METRON-597
URL: https://issues.apache.org/jira/browse/METRON-597
Project: Metron
Issue Type: Bug
Reporter: Nick Allen
Seems to be some kind of timing issue.
---
T E S T S
---
Running org.apache.metron.profiler.integration.ProfilerIntegrationTest
Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 66.564 sec <<<
FAILURE!
testExample3(org.apache.metron.profiler.integration.ProfilerIntegrationTest)
Time elapsed: 16.759 sec <<< FAILURE!
java.lang.AssertionError: expected:<20.0> but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:553)
at org.junit.Assert.assertEquals(Assert.java:683)
at
org.apache.metron.profiler.integration.ProfilerIntegrationTest.testExample3(ProfilerIntegrationTest.java:203)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
{code}
Results :
Failed tests:
testExample3(org.apache.metron.profiler.integration.ProfilerIntegrationTest):
expected:<20.0> but was:
Tests run: 5, Failures: 1, Errors: 0, Skipped: 0
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (METRON-594) Replay Telemetry Data through Profiler
[ https://issues.apache.org/jira/browse/METRON-594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705868#comment-15705868 ] Nick Allen commented on METRON-594: --- I would like to get feedback from the community. How should this work? What use cases do you envision? What features do we need to support this? > Replay Telemetry Data through Profiler > -- > > Key: METRON-594 > URL: https://issues.apache.org/jira/browse/METRON-594 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen > > The Profiler currently consumes live telemetry, in real-time, as it is > streamed through Metron. A useful extension of this functionality would > allow the Profiler to also consume archived, historical telemetry. Allowing > a user to selectively replay archived, historical raw telemetry through the > Profiler has a number of applications. The following use cases help describe > why this might be useful. > Use Case 1 - Model Development > When developing a new model, I often need a feature set of historical data on > which to train my model. I can either wait days, weeks, months for the > Profiler to generate this based on live data or I could re-run the raw, > historical telemetry through the Profiler to get started immediately. It is > much simpler to use the same mechanism to create this historical data set, > than a separate batch-driven tool to recreate something that approximates the > historical feature set. > Use Case 2 - Model Deployment > When deploying an analytical model to a new environment, like production, on > day 1 there is often no historical data for the model to work with. This > often leaves a gap between when the model is deployed and when that model is > actually useful. If I could replay raw telemetry through the profiler a > historical feature set could be created as part of the deployment process. > This allows my model to start functioning on day 1. > Use Case 3 - Profile Validation > When creating a Profile, it is difficult to understand how the configured > profile might behave against the entire data set. By creating the profile > and watching it consume real-time streaming data, I only have an > understanding of how it behaves on that small segment of data. If I am able > to replay historical telemetry, I can instantly understand how it behaves on > a much larger data set; including all the anomalies and exceptions that > exist in all large data sets. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-594) Replay Telemetry Data through Profiler
[ https://issues.apache.org/jira/browse/METRON-594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705860#comment-15705860 ] Nick Allen commented on METRON-594: --- To replay telemetry data, the Profiler has to be able to use event time, rather than processing time as it currently does. > Replay Telemetry Data through Profiler > -- > > Key: METRON-594 > URL: https://issues.apache.org/jira/browse/METRON-594 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen > > The Profiler currently consumes live telemetry, in real-time, as it is > streamed through Metron. A useful extension of this functionality would > allow the Profiler to also consume archived, historical telemetry. Allowing > a user to selectively replay archived, historical raw telemetry through the > Profiler has a number of applications. The following use cases help describe > why this might be useful. > Use Case 1 - Model Development > When developing a new model, I often need a feature set of historical data on > which to train my model. I can either wait days, weeks, months for the > Profiler to generate this based on live data or I could re-run the raw, > historical telemetry through the Profiler to get started immediately. It is > much simpler to use the same mechanism to create this historical data set, > than a separate batch-driven tool to recreate something that approximates the > historical feature set. > Use Case 2 - Model Deployment > When deploying an analytical model to a new environment, like production, on > day 1 there is often no historical data for the model to work with. This > often leaves a gap between when the model is deployed and when that model is > actually useful. If I could replay raw telemetry through the profiler a > historical feature set could be created as part of the deployment process. > This allows my model to start functioning on day 1. > Use Case 3 - Profile Validation > When creating a Profile, it is difficult to understand how the configured > profile might behave against the entire data set. By creating the profile > and watching it consume real-time streaming data, I only have an > understanding of how it behaves on that small segment of data. If I am able > to replay historical telemetry, I can instantly understand how it behaves on > a much larger data set; including all the anomalies and exceptions that > exist in all large data sets. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-594) Replay Telemetry Data through Profiler
[ https://issues.apache.org/jira/browse/METRON-594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-594: -- Description: The Profiler currently consumes live telemetry, in real-time, as it is streamed through Metron. A useful extension of this functionality would allow the Profiler to also consume archived, historical telemetry. Allowing a user to selectively replay archived, historical raw telemetry through the Profiler has a number of applications. The following use cases help describe why this might be useful. Use Case 1 - Model Development When developing a new model, I often need a feature set of historical data on which to train my model. I can either wait days, weeks, months for the Profiler to generate this based on live data or I could re-run the raw, historical telemetry through the Profiler to get started immediately. It is much simpler to use the same mechanism to create this historical data set, than a separate batch-driven tool to recreate something that approximates the historical feature set. Use Case 2 - Model Deployment When deploying an analytical model to a new environment, like production, on day 1 there is often no historical data for the model to work with. This often leaves a gap between when the model is deployed and when that model is actually useful. If I could replay raw telemetry through the profiler a historical feature set could be created as part of the deployment process. This allows my model to start functioning on day 1. Use Case 3 - Profile Validation When creating a Profile, it is difficult to understand how the configured profile might behave against the entire data set. By creating the profile and watching it consume real-time streaming data, I only have an understanding of how it behaves on that small segment of data. If I am able to replay historical telemetry, I can instantly understand how it behaves on a much larger data set; including all the anomalies and exceptions that exist in all large data sets. was: The Profiler currently consumes live telemetry, in real-time, as it is streamed through Metron. A useful extension of this functionality would allow the Profiler to also consume archived, historical telemetry. Allowing a user to selectively replay archived, historical raw telemetry through the Profiler has a number of applications. The following use cases help describe why this might be useful. Use Case 1 - Model Development When developing a new model, I often need a feature set of historical data on which to train my model. I can either wait days, weeks, months for the Profiler to generate this based on live data or I could re-run the raw, historical telemetry through the Profiler to get started immediately. It is much simpler to use the same mechanism to create this historical data set, than a separate batch-driven tool to recreate something that approximates the historical feature set. Use Case 2 - Model Deployment When deploying an analytical model to a new environment, like production, on day 1 there is often no historical data for the model to work with. This often leaves a gap between when the model is deployed and when that model is actually useful. If I could replay raw telemetry through the profiler a historical feature set could be created as part of the deployment process. This allows my model to start functioning on day 1. Use Case 3 - Profile Validation When creating a Profile, it is difficult to understand how the configured profile might behave against the entire data set. By creating the profile and watching it consume real-time streaming data, I only have an understanding of how behaves on that small segment of data. If I am able to replay historical telemetry, I can instantly understand how it behaves on a much larger data set along with all the anomalies and exceptions that exist in all large data sets. > Replay Telemetry Data through Profiler > -- > > Key: METRON-594 > URL: https://issues.apache.org/jira/browse/METRON-594 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen > > The Profiler currently consumes live telemetry, in real-time, as it is > streamed through Metron. A useful extension of this functionality would > allow the Profiler to also consume archived, historical telemetry. Allowing > a user to selectively replay archived, historical raw telemetry through the > Profiler has a number of applications. The following use cases help describe > why this might be useful. > Use Case 1 - Model Development > When developing a new model, I often need a feature set of historical data on > which to train my model. I can either wait days, weeks, months for the > Profiler to generate this based on live data or I could re-run the raw, > historical
[jira] [Created] (METRON-594) Replay Telemetry Data through Profiler
Nick Allen created METRON-594: - Summary: Replay Telemetry Data through Profiler Key: METRON-594 URL: https://issues.apache.org/jira/browse/METRON-594 Project: Metron Issue Type: Improvement Reporter: Nick Allen The Profiler currently consumes live telemetry, in real-time, as it is streamed through Metron. A useful extension of this functionality would allow the Profiler to also consume archived, historical telemetry. Allowing a user to selectively replay archived, historical raw telemetry through the Profiler has a number of applications. The following use cases help describe why this might be useful. Use Case 1 - Model Development When developing a new model, I often need a feature set of historical data on which to train my model. I can either wait days, weeks, months for the Profiler to generate this based on live data or I could re-run the raw, historical telemetry through the Profiler to get started immediately. It is much simpler to use the same mechanism to create this historical data set, than a separate batch-driven tool to recreate something that approximates the historical feature set. Use Case 2 - Model Deployment When deploying an analytical model to a new environment, like production, on day 1 there is often no historical data for the model to work with. This often leaves a gap between when the model is deployed and when that model is actually useful. If I could replay raw telemetry through the profiler a historical feature set could be created as part of the deployment process. This allows my model to start functioning on day 1. Use Case 3 - Profile Validation When creating a Profile, it is difficult to understand how the configured profile might behave against the entire data set. By creating the profile and watching it consume real-time streaming data, I only have an understanding of how behaves on that small segment of data. If I am able to replay historical telemetry, I can instantly understand how it behaves on a much larger data set along with all the anomalies and exceptions that exist in all large data sets. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-590) Enable Use of Event Time in Profiler
[ https://issues.apache.org/jira/browse/METRON-590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-590: -- Description: There are at least two different times that are important to consider when handling the telemetry messages received by Metron. (1) Processing time is the time at which Metron processed the message. (2) Event time is the time at which the event actually occurred. If Metron is consuming live data and all is well, the processing and event times may remain close and consistent. When processing time differs from event time the data produced by the Profiler may be inaccurate. There are a few scenarios under which these times might differ greatly which would negatively impact the feature set produced by the Profiler. (1) When the system has experienced an outage, for example, a scheduled maintenance window. When restarted a high volume of messages will need to be processed by the Profiler. The output of the Profiler will indicate an increase in activity, although no change in activity actually occurred on the target network. This could happen whether the outage was Metron itself or an upstream system that feeds data to Metron. (2) If the user attempts to replay historical telemetry through the Profiler, the Profiler will attribute the activity to the time period in which it was processed. Obviously the activity should be attributed to the time period in which the raw telemetry events originated in. There are some scenarios when processing time might be preferred and other use cases where event time is preferred. The Profiler should be enhanced to allow it to produce profiles based on either processing time or event time. was: There are at least two different times that are important to consider when handling the telemetry messages received by Metron. (1) Processing time is the time at which Metron processed the message. (2) Event time is the time at which the event actually occurred. If Metron is consuming live data and all is well, these times may remain close and consistent. There are many scenarios under which these times might differ greatly. For example, if there are delays in processing messages due to heavy load or if older, archived telemetry is being replayed through the system. When processing time differs from event time the data produced by the Profiler may be inaccurate. There are some use cases when processing time might be preferred and other use cases where event time is preferred. The Profiler should be enhanced to allow it to produce profiles based on either processing time or event time. > Enable Use of Event Time in Profiler > > > Key: METRON-590 > URL: https://issues.apache.org/jira/browse/METRON-590 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen > > There are at least two different times that are important to consider when > handling the telemetry messages received by Metron. > (1) Processing time is the time at which Metron processed the message. > (2) Event time is the time at which the event actually occurred. > If Metron is consuming live data and all is well, the processing and event > times may remain close and consistent. When processing time differs from > event time the data produced by the Profiler may be inaccurate. There are a > few scenarios under which these times might differ greatly which would > negatively impact the feature set produced by the Profiler. > (1) When the system has experienced an outage, for example, a scheduled > maintenance window. When restarted a high volume of messages will need to be > processed by the Profiler. The output of the Profiler will indicate an > increase in activity, although no change in activity actually occurred on the > target network. This could happen whether the outage was Metron itself or an > upstream system that feeds data to Metron. > (2) If the user attempts to replay historical telemetry through the Profiler, > the Profiler will attribute the activity to the time period in which it was > processed. Obviously the activity should be attributed to the time period in > which the raw telemetry events originated in. > There are some scenarios when processing time might be preferred and other > use cases where event time is preferred. The Profiler should be enhanced to > allow it to produce profiles based on either processing time or event time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-590) Enable Use of Event Time in Profiler
[ https://issues.apache.org/jira/browse/METRON-590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-590: -- Description: There are at least two different times that are important to consider when handling the telemetry messages received by Metron. (1) Processing time is the time at which Metron processed the message. (2) Event time is the time at which the event actually occurred. If Metron is consuming live data and all is well, these times may remain close and consistent. There are many scenarios under which these times might differ greatly. For example, if there are delays in processing messages due to heavy load or if older, archived telemetry is being replayed through the system. When processing time differs from event time the data produced by the Profiler may be inaccurate. There are some use cases when processing time might be preferred and other use cases where event time is preferred. The Profiler should be enhanced to allow it to produce profiles based on either processing time or event time. was: There are at least two different times that are important to consider when handling the telemetry messages received by Metron. (1) Processing time is the time at which Metron processed the message. (2) Event time is the time at which the event actually occurred. If Metron is consuming live data and all is well, these times may remain close and consistent. There are many scenarios under which these times might differ greatly. For example, if there are delays in processing messages due to heavy load or if older, archived telemetry is being replayed through the system. When processing time differs from event time the data produced by the Profiler may be inaccurate. The Profiler should be enhanced to allow it to produce profiles based on either processing time or event time. > Enable Use of Event Time in Profiler > > > Key: METRON-590 > URL: https://issues.apache.org/jira/browse/METRON-590 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen > > There are at least two different times that are important to consider when > handling the telemetry messages received by Metron. > (1) Processing time is the time at which Metron processed the message. > (2) Event time is the time at which the event actually occurred. > If Metron is consuming live data and all is well, these times may remain > close and consistent. There are many scenarios under which these times might > differ greatly. For example, if there are delays in processing messages due > to heavy load or if older, archived telemetry is being replayed through the > system. > When processing time differs from event time the data produced by the > Profiler may be inaccurate. There are some use cases when processing time > might be preferred and other use cases where event time is preferred. The > Profiler should be enhanced to allow it to produce profiles based on either > processing time or event time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-590) Enable Use of Event Time in Profiler
Nick Allen created METRON-590: - Summary: Enable Use of Event Time in Profiler Key: METRON-590 URL: https://issues.apache.org/jira/browse/METRON-590 Project: Metron Issue Type: Improvement Reporter: Nick Allen Assignee: Nick Allen There are at least two different times that are important to consider when handling the telemetry messages received by Metron. (1) Processing time is the time at which Metron processed the message. (2) Event time is the time at which the event actually occurred. If Metron is consuming live data and all is well, these times may remain close and consistent. There are many scenarios under which these times might differ greatly. For example, if there are delays in processing messages due to heavy load or if older, archived telemetry is being replayed through the system. When processing time differs from event time the data produced by the Profiler may be inaccurate. The Profiler should be enhanced to allow it to produce profiles based on either processing time or event time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-585) Pcap Replay Fails to Stop
Nick Allen created METRON-585: - Summary: Pcap Replay Fails to Stop Key: METRON-585 URL: https://issues.apache.org/jira/browse/METRON-585 Project: Metron Issue Type: Bug Reporter: Nick Allen Assignee: Nick Allen The service script for Pcap Replay often fails to actually stop the tcpreplay process. The `tcpreplay` process is a huge consumer of resources when running on Quick Dev and actually having it stop when asked, would be super nice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-576) Stellar function resolution takes too long on running cluster
Nick Allen created METRON-576: - Summary: Stellar function resolution takes too long on running cluster Key: METRON-576 URL: https://issues.apache.org/jira/browse/METRON-576 Project: Metron Issue Type: Improvement Reporter: Nick Allen Assignee: Nick Allen When running the Stellar REPL in a cluster on AWS, function resolution takes 50-60 seconds. The user is not able to execute any functions in the REPL until this process completes. The default function resolver searches the classpath for Stellar functions. The delay may be because there are just too many classes in the classpath to search on a running cluster. As more libraries are added as dependencies under /usr/metron//lib this problem just gets worse. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-574) Allow user to interrupt function execution in the Stellar REPL
Nick Allen created METRON-574: - Summary: Allow user to interrupt function execution in the Stellar REPL Key: METRON-574 URL: https://issues.apache.org/jira/browse/METRON-574 Project: Metron Issue Type: Bug Reporter: Nick Allen There is no way for a user to interrupt a long-running function execution in the Stellar REPL. If the user does a CTRL-C, the entire REPL environment is killed. It would be useful for a user to be able to interrupt an errant function, but still maintain the Stellar execution environment. I've run into this quite a few times over the past few days. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-573) Cluster deployed via 'amazon-ec2' process does not survive reboot
[ https://issues.apache.org/jira/browse/METRON-573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15673889#comment-15673889 ] Nick Allen commented on METRON-573: --- If the deployment process were to use the internal DNS names, which do not change, then this *might* fix the problem. The only consideration being that these internal names can only be resolved within the same VPC (virtual private cloud; Amazon-speak). * Example of an external DNS name that can and will change on reboot; ec2-35-163-234-95.us-west-2.compute.amazonaws.com * Example of an internal DNS name that will not change on reboot; ip-10-0-0-143.us-west-2.compute.internal > Cluster deployed via 'amazon-ec2' process does not survive reboot > - > > Key: METRON-573 > URL: https://issues.apache.org/jira/browse/METRON-573 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen > > The current AWS cloud deployment process results in a cluster that does not > survive a reboot. The cluster is configured using the external DNS names. > These names can change when a host is rebooted. When the DNS names change > the configuration for all the core components like Storm, HBase, HDFS, etc > are invalid. None of the core components can even be started from Ambari. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-569) Enrichment topology duplicates messages
[ https://issues.apache.org/jira/browse/METRON-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15671850#comment-15671850 ] Nick Allen commented on METRON-569: --- It makes sense to me that this problem occurs when an enrichment takes longer than the storm timeout. Bravo for teasing that out. The join bolt has a cache containing the original messages that it is expecting enrichments for. This cache is invalidated after some period of time. If the cache invalidation time is less than the storm timeout, then in the scenario described, the 'late' enrichment message would reach the join bolt, but the join bolt would have already forgotten about the original message. The join bolt would then correctly ignore the 'late' enrichment message. To address the potential for misconfiguration, we could set the cache invalidation time to be some fraction of the storm timeout. > Enrichment topology duplicates messages > --- > > Key: METRON-569 > URL: https://issues.apache.org/jira/browse/METRON-569 > Project: Metron > Issue Type: Bug >Reporter: Domenic Puzio > Original Estimate: 1h > Remaining Estimate: 1h > > When running the 'enrichment' topology, I get duplicate message being > indexed. For example, I put 100 messages into the 'enrichment' Kafka queue > and I get 175 messages onto the 'indexing' Kafka queue. This happens when I > am running the 'enrichment' topology with one or more enrichment bolt. > This is an acking issue within the JoinBolt class. When a message does not > "complete" the join (like when it is the first message in a pair of message > to get joined) it does not get acked. This means that this message will get > replayed through Storm, causing message duplication further down the road and > tons of additional overhead. Adding the correct acking resolves this problem. > I will add the PR for this shortly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-570) Add Profiler Link to README
Nick Allen created METRON-570: - Summary: Add Profiler Link to README Key: METRON-570 URL: https://issues.apache.org/jira/browse/METRON-570 Project: Metron Issue Type: Bug Reporter: Nick Allen Assignee: Nick Allen Priority: Minor Add a link to the Profiler as part of the project's root README. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-557) Create Stellar Functions for Kafka
Nick Allen created METRON-557: - Summary: Create Stellar Functions for Kafka Key: METRON-557 URL: https://issues.apache.org/jira/browse/METRON-557 Project: Metron Issue Type: Improvement Reporter: Nick Allen Assignee: Nick Allen Add functions to read and write messages to Kafka topics. This should allow a user the to debug a Grok expression or Enrichment configuration inside the REPL. For example, I can pull a message off of the error topic to see why it failed to parse. Then fix the configuration and resubmit the message to try again. I also won’t have to go outside of the Stellar REPL to monitor topic activity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-556) Profiler - Refactor 'Group By' Calculation
Nick Allen created METRON-556: - Summary: Profiler - Refactor 'Group By' Calculation Key: METRON-556 URL: https://issues.apache.org/jira/browse/METRON-556 Project: Metron Issue Type: Improvement Reporter: Nick Allen Assignee: Nick Allen Currently, the 'group by' expression is calculated in the context of the HBaseBolt. Specifically it is calculated in the ProfileHBaseMapper. This mapper is run by the HBaseBolt to define how the Profile values stored in a tuple get written to HBase.This can be refactored so that the 'group by' expression can be calculated at the same time as the 'result' expression in the context of the ProfileBuilderBolt. This has a few advantages. it allows any errors encountered in executing the 'group by' expression to be contained within the ProfileBuilderBolt instance responsible for the profile. Any exceptions cannot impact the functioning of the rest of the Profiler. This is also more consistent and simple as the HBaseBolt will receive everything that it needs, pre-calculated, in the form of a ProfileMeasurement object. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-556) Profiler - Refactor 'Group By' Calculation
[ https://issues.apache.org/jira/browse/METRON-556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-556: -- Description: Currently a Profile's 'group by' expression is calculated in the context of the HBaseBolt. Specifically it is calculated in the ProfileHBaseMapper. This mapper is run by the HBaseBolt to define how the Profile values stored in a tuple get written to HBase.This can be refactored so that the 'group by' expression is calculated at the same time as the 'result' expression in the context of the ProfileBuilderBolt. This has a few advantages. It allows any errors encountered in executing the 'group by' expression to be contained within the ProfileBuilderBolt instance responsible for the profile/entity pair alone. Any exceptions cannot impact the functioning of the rest of the Profiler. This is also more consistent and simple as the HBaseBolt will receive everything that it needs, pre-calculated, in the form of a ProfileMeasurement object. was: Currently a Profile's 'group by' expression is calculated in the context of the HBaseBolt. Specifically it is calculated in the ProfileHBaseMapper. This mapper is run by the HBaseBolt to define how the Profile values stored in a tuple get written to HBase.This can be refactored so that the 'group by' expression can be calculated at the same time as the 'result' expression in the context of the ProfileBuilderBolt. This has a few advantages. it allows any errors encountered in executing the 'group by' expression to be contained within the ProfileBuilderBolt instance responsible for the profile. Any exceptions cannot impact the functioning of the rest of the Profiler. This is also more consistent and simple as the HBaseBolt will receive everything that it needs, pre-calculated, in the form of a ProfileMeasurement object. > Profiler - Refactor 'Group By' Calculation > -- > > Key: METRON-556 > URL: https://issues.apache.org/jira/browse/METRON-556 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen > > Currently a Profile's 'group by' expression is calculated in the context of > the HBaseBolt. Specifically it is calculated in the ProfileHBaseMapper. > This mapper is run by the HBaseBolt to define how the Profile values stored > in a tuple get written to HBase.This can be refactored so that the 'group > by' expression is calculated at the same time as the 'result' expression in > the context of the ProfileBuilderBolt. > This has a few advantages. It allows any errors encountered in executing the > 'group by' expression to be contained within the ProfileBuilderBolt instance > responsible for the profile/entity pair alone. Any exceptions cannot impact > the functioning of the rest of the Profiler. This is also more consistent > and simple as the HBaseBolt will receive everything that it needs, > pre-calculated, in the form of a ProfileMeasurement object. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-556) Profiler - Refactor 'Group By' Calculation
[ https://issues.apache.org/jira/browse/METRON-556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-556: -- Description: Currently a Profile's 'group by' expression is calculated in the context of the HBaseBolt. Specifically it is calculated in the ProfileHBaseMapper. This mapper is run by the HBaseBolt to define how the Profile values stored in a tuple get written to HBase.This can be refactored so that the 'group by' expression can be calculated at the same time as the 'result' expression in the context of the ProfileBuilderBolt. This has a few advantages. it allows any errors encountered in executing the 'group by' expression to be contained within the ProfileBuilderBolt instance responsible for the profile. Any exceptions cannot impact the functioning of the rest of the Profiler. This is also more consistent and simple as the HBaseBolt will receive everything that it needs, pre-calculated, in the form of a ProfileMeasurement object. was: Currently, the 'group by' expression is calculated in the context of the HBaseBolt. Specifically it is calculated in the ProfileHBaseMapper. This mapper is run by the HBaseBolt to define how the Profile values stored in a tuple get written to HBase.This can be refactored so that the 'group by' expression can be calculated at the same time as the 'result' expression in the context of the ProfileBuilderBolt. This has a few advantages. it allows any errors encountered in executing the 'group by' expression to be contained within the ProfileBuilderBolt instance responsible for the profile. Any exceptions cannot impact the functioning of the rest of the Profiler. This is also more consistent and simple as the HBaseBolt will receive everything that it needs, pre-calculated, in the form of a ProfileMeasurement object. > Profiler - Refactor 'Group By' Calculation > -- > > Key: METRON-556 > URL: https://issues.apache.org/jira/browse/METRON-556 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen > > Currently a Profile's 'group by' expression is calculated in the context of > the HBaseBolt. Specifically it is calculated in the ProfileHBaseMapper. > This mapper is run by the HBaseBolt to define how the Profile values stored > in a tuple get written to HBase.This can be refactored so that the 'group > by' expression can be calculated at the same time as the 'result' expression > in the context of the ProfileBuilderBolt. > This has a few advantages. it allows any errors encountered in executing the > 'group by' expression to be contained within the ProfileBuilderBolt instance > responsible for the profile. Any exceptions cannot impact the functioning of > the rest of the Profiler. This is also more consistent and simple as the > HBaseBolt will receive everything that it needs, pre-calculated, in the form > of a ProfileMeasurement object. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-548) Improve Profiler Documentation
[ https://issues.apache.org/jira/browse/METRON-548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-548: -- Fix Version/s: 0.2.2BETA > Improve Profiler Documentation > -- > > Key: METRON-548 > URL: https://issues.apache.org/jira/browse/METRON-548 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen > Fix For: 0.2.2BETA > > > Improve the documentation to highlight how the Profiler can store any > serializable object, not just numeric values, and how that plays nicely with > the Stellar STATS_* package. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
