[GitHub] metron issue #970: METRON-1421: Create a SolrMetaAlertDao

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/970
  
It is definitely looking better.  I am liking the progress.  But I still 
think having overly broad interfaces that do everything is causing problems; 
primarily `IndexDao` and also `MetaAlertDao`.

Neither of those need to exist.  For example...
* The `MetaAlertService` does not need a `MetaAlertDao`.
* If the `MetaAlertService` needs to update meta-alerts then it should 
directly use a `MetaAlertUpdateDao`.  
* If the `MetaAlertService` needs to search for meta-alerts then it should 
directly use a `MetaAlertSearchDao`.  

I can submit a PR to your branch, if you like.


---

[GitHub] metron issue #531: METRON-854 create dhcp dump parser

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/531
  
So, what is the status of this?  Can we at *least* sum up? 


---

[jira] [Commented] (METRON-854) Create DHCPDump Parser

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432293#comment-16432293
 ] 

ASF GitHub Bot commented on METRON-854:
---

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/531
  
So, what is the status of this?  Can we at *least* sum up? 


> Create DHCPDump Parser
> --
>
> Key: METRON-854
> URL: https://issues.apache.org/jira/browse/METRON-854
> Project: Metron
>  Issue Type: New Feature
>Reporter: Bas van de Lustgraaf
>Priority: Minor
>  Labels: parser
>
> Create a DHCPDump parser. This information can be used during enrichment to 
> link ip-addresses to hostnames.
> {noformat}
> TIME: 2017-01-16 16:54:21.655|INTERFACE: eth2|OP:1 BOOTPREQUEST|CIADDR: 
> 172.20.75.77|YIADDR: 0.0.0.0|SIADDR: 0.0.0.0|GIADDR: 172.20.75.8|CHADDR: 
> fc:f8:ae:e8:ef:db:00:00:00:00:00:00:00:00:00:00|OPTION:  53   1 DHCP message 
> type: 8 |DHCPINFORM|OPTION:  61   7 Client-identifier: 
> 01:fc:f8:ae:e8:ef:db|OPTION:  12   5 Host name: Q1244|OPTION:  60   8 Vendor 
> class identifier: MSFT 5.0|OPTION:  55  13 Parameter Request List:   1 
> (Subnet mask)|| 15 (Domainname)||  3 (Routers)||  6 (DNS server)|| 44 
> (NetBIOS name server)|| 46 (NetBIOS node type)|| 47 (NetBIOS scope)|| 31 
> (Perform router discovery)|| 33 (Static route)||121 (Classless Static 
> Route)||249 (MSFT - Classless route)|| 43 (Vendor specific info)||252 (MSFT - 
> WinSock Proxy Auto Detect)|||IP: 10.10.10.177 > 172.20.1.11 | 
> b8:ca:3a:67:95:8a > 0:50:56:84:68:43
> TIME: 2017-01-16 17:13:14.548|INTERFACE: eth2|OP:1 BOOTPREQUEST|CIADDR: 
> 172.20.75.77|YIADDR: 0.0.0.0|SIADDR: 0.0.0.0|GIADDR: 172.20.75.8|CHADDR: 
> fc:f8:ae:e8:ef:db:00:00:00:00:00:00:00:00:00:00|OPTION:  53   1 DHCP message 
> type: 8 |DHCPINFORM|OPTION:  61   7 Client-identifier: 
> 01:fc:f8:ae:e8:ef:db|OPTION:  12   5 Host name: Q1244|OPTION:  60   8 Vendor 
> class identifier: MSFT 5.0|OPTION:  55  13 Parameter Request List:   1 
> (Subnet mask)|| 15 (Domainname)||  3 (Routers)||  6 (DNS server)|| 44 
> (NetBIOS name server)|| 46 (NetBIOS node type)|| 47 (NetBIOS scope)|| 31 
> (Perform router discovery)|| 33 (Static route)||121 (Classless Static 
> Route)||249 (MSFT - Classless route)|| 43 (Vendor specific info)||252 (MSFT - 
> WinSock Proxy Auto Detect)|||IP: 10.10.10.177 > 172.20.1.10 | 
> b8:ca:3a:67:95:8a > 0:50:56:b9:28:ac
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #670: METRON-1065 grok pattern for cisco asa parser expects sys...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/670
  
What is the status of this?


---

[GitHub] metron issue #979: METRON-1507 Added support for vagrant cachier

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/979
  
I think to test we need to try with and without the plugin


---

[jira] [Commented] (METRON-1421) Create a SolrMetaAlertDao

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432348#comment-16432348
 ] 

ASF GitHub Bot commented on METRON-1421:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/970
  
> I'd like to know what specific problems we're going to solve that merit 
that level of change in this PR. 

The fundamental problem here is a lack of good unit tests on these search 
DAOs.  That is why I am interested in refactoring this.  

What I don't want to repeat is #832 or [the recent issue found in Full Dev 
on this PR](https://github.com/apache/metron/pull/970#issuecomment-379753506).  
Are these clues that we don't have sufficient testing?

We really need quality tests here in particular, because this is a complex 
and difficult to implement feature.  Because it is so difficult, we need to 
have **especially** good automated testing around this.

I completely appreciate not wanting to expand scope too much.  But when 
adding a big chunk of net-new functionality, you inherently have to refactor to 
keep things maintainable.  To what degree refactoring is needed, is a judgement 
call of course.






> Create a SolrMetaAlertDao
> -
>
> Key: METRON-1421
> URL: https://issues.apache.org/jira/browse/METRON-1421
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Justin Leet
>Assignee: Justin Leet
>Priority: Major
>
> Create an implementation of the MetaAlertDao for Solr. This will involve 
> implementing the various MetaAlertDao methods using the SolrJ library and 
> also providing a SolrMetaAlertIntegrationTest (similar to 
> ElasticsearchMetaAlertIntegrationTest).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #970: METRON-1421: Create a SolrMetaAlertDao

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/970
  
> I'd like to know what specific problems we're going to solve that merit 
that level of change in this PR. 

The fundamental problem here is a lack of good unit tests on these search 
DAOs.  That is why I am interested in refactoring this.  

What I don't want to repeat is #832 or [the recent issue found in Full Dev 
on this PR](https://github.com/apache/metron/pull/970#issuecomment-379753506).  
Are these clues that we don't have sufficient testing?

We really need quality tests here in particular, because this is a complex 
and difficult to implement feature.  Because it is so difficult, we need to 
have **especially** good automated testing around this.

I completely appreciate not wanting to expand scope too much.  But when 
adding a big chunk of net-new functionality, you inherently have to refactor to 
keep things maintainable.  To what degree refactoring is needed, is a judgement 
call of course.






---

[jira] [Commented] (METRON-1065) Grok pattern for Cisco ASA Parser expects syslog_pri

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432296#comment-16432296
 ] 

ASF GitHub Bot commented on METRON-1065:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/670
  
What is the status of this?


> Grok pattern for Cisco ASA Parser expects syslog_pri
> 
>
> Key: METRON-1065
> URL: https://issues.apache.org/jira/browse/METRON-1065
> Project: Metron
>  Issue Type: Improvement
>Affects Versions: 0.4.1
>Reporter: Bas van de Lustgraaf
>Priority: Minor
>
> The current grok pattern `CISCO_TAGGED_SYSLOG` expects to have a syslog 
> priority present at the start of each message. Unfortunately, this is not 
> always the case.
> *Currently supported:*
> {noformat}
> <162>Aug 05 2016 01:02:25: %ASA-2-106006: Deny inbound UDP from 
> 10.25.177.164/63279 to 10.2.52.71/161 on interface Inside
> {noformat}
> *Not supported by the current Grok pattern:*
> {noformat}
> Aug 05 2016 01:02:25: %ASA-2-106006: Deny inbound UDP from 
> 10.25.177.164/63279 to 10.2.52.71/161 on interface Inside
> {noformat}
> My suggestion would be to edit the `CISCO_TAGGED_SYSLOG` pattern to make the 
> following part optional: 
> {noformat}
> <%{POSINT:syslog_pri}>
> {noformat}
> And grep the severity from the `%ASA-4-106023` part. The part between the 
> hyphens, is the severity (source 
> http://www.cisco.com/c/en/us/td/docs/security/asa/syslog/b_syslog/syslogs-sev-level.html).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #684: DO NOT MERGE: METRON-1086: Create a Blockly-based user in...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/684
  
I will repeat that this should be a feature branch.  What are we going to 
do here?  This is just sitting here getting more and more out of date.


---

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/687
  
This is pretty old at this point, and there has been a lot of discussion 
all over the place about it.  I'm not sure I can sum up where we are.  
@cestella can you sum up your current feeling on how we can move this forward?


---

[jira] [Commented] (METRON-1086) Create a Blockly-based user interface for Stellar

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432297#comment-16432297
 ] 

ASF GitHub Bot commented on METRON-1086:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/684
  
I will repeat that this should be a feature branch.  What are we going to 
do here?  This is just sitting here getting more and more out of date.


> Create a Blockly-based user interface for Stellar
> -
>
> Key: METRON-1086
> URL: https://issues.apache.org/jira/browse/METRON-1086
> Project: Metron
>  Issue Type: New Feature
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
>
> Having a visual Stellar editor would be useful.  Blockly is a Google library 
> for building visual programming editors:  
> https://developers.google.com/blockly/.  This feature will integrate the two 
> and produce a reusable javascript widget for editing Stellar code.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1233) Remove description of Global configuration from Enrichment doc

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432308#comment-16432308
 ] 

ASF GitHub Bot commented on METRON-1233:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/789
  
Still needs conflict resolution, still needs relevancy check


> Remove description of Global configuration from Enrichment doc
> --
>
> Key: METRON-1233
> URL: https://issues.apache.org/jira/browse/METRON-1233
> Project: Metron
>  Issue Type: Improvement
>Reporter: Dima Kovalyov
>Priority: Minor
>
> It confuses reader. Description of Global configuration is available ahead of 
> the enrichment and listed here:
> https://metron.apache.org/current-book/metron-platform/metron-common/index.html
> Global config is used for validation of the fields.
> Enrichment is used for fields enrichment and probably can also be used for 
> validation of the fields (which is not covered in the doc).
> From available doc I can see that they serve two different purposes.
> So, I don't see any reason to mention global config in enrichment 
> description. What's the benefit of knowing about global configuration when 
> you want deep dive in enrichment?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1298) TimeRange Picker doesn't work on Safari

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432310#comment-16432310
 ] 

ASF GitHub Bot commented on METRON-1298:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/830
  
What is the status of this PR?


> TimeRange Picker doesn't work on Safari
> ---
>
> Key: METRON-1298
> URL: https://issues.apache.org/jira/browse/METRON-1298
> Project: Metron
>  Issue Type: Bug
>Reporter: RaghuMitra
>Assignee: RaghuMitra
>Priority: Major
>
> Date Picker doesn't work on safari the request goes with a timestamp of NaN



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1421) Create a SolrMetaAlertDao

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432312#comment-16432312
 ] 

ASF GitHub Bot commented on METRON-1421:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/970
  
> @justinleet Solve the code reuse problem for Solr and ES.

I am also noticing another positive side effect here. Getting rid of these 
interface hierarchies (like having the `MetaAlertService` use an 
`MetaAlertUpdateDao`) is that the service layer becomes where code lives that 
can be reused across ES and Solr.  And that makes a ton of logical sense to me.





> Create a SolrMetaAlertDao
> -
>
> Key: METRON-1421
> URL: https://issues.apache.org/jira/browse/METRON-1421
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Justin Leet
>Assignee: Justin Leet
>Priority: Major
>
> Create an implementation of the MetaAlertDao for Solr. This will involve 
> implementing the various MetaAlertDao methods using the SolrJ library and 
> also providing a SolrMetaAlertIntegrationTest (similar to 
> ElasticsearchMetaAlertIntegrationTest).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #970: METRON-1421: Create a SolrMetaAlertDao

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/970
  
> @justinleet Solve the code reuse problem for Solr and ES.

I am also noticing another positive side effect here. Getting rid of these 
interface hierarchies (like having the `MetaAlertService` use an 
`MetaAlertUpdateDao`) is that the service layer becomes where code lives that 
can be reused across ES and Solr.  And that makes a ton of logical sense to me.





---

[jira] [Commented] (METRON-1212) Bundles and Maven Plugin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432311#comment-16432311
 ] 

ASF GitHub Bot commented on METRON-1212:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/865
  
@mattf-horton any chance to look at this?


> Bundles and Maven Plugin
> 
>
> Key: METRON-1212
> URL: https://issues.apache.org/jira/browse/METRON-1212
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>  Labels: metron-feature-canidate, 
> metron-feature-extensions-parsers
>
> The first effort will be to land the bundle system and supporting maven 
> plugin on master



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1421) Create a SolrMetaAlertDao

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432325#comment-16432325
 ] 

ASF GitHub Bot commented on METRON-1421:


Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/970
  
@nickwallen
Right now those interfaces are basically a marker interface that requires 
everything that's needed for a complete DAO set. Prior to some of the Solr 
refactorings, it existed as a single DAO to provide a complete set of 
functionality to be implemented in order to allow an implementation to be 
complete (for some definition of complete) and available to be swapped out via 
configuration.  Even with the refactoring into multiple interfaces, the 
IndexDao and MetaAlertDao still exist to provide some definition of complete.

Getting rid of IndexDao in particular likely has potentially broader 
implications because right now that interface ties together the various layers. 
Making those changes in a complete and consistent manner is larger than 
MetaAlertService, because the SearchServiceImpl and others take an IndexDao 
(which could be a regular SolrDao or SolrMetaAlertDao, etc.) in order to allow 
regular searches to have meta alert adjustments, e.g. massage the queries to 
make sure metaalerts get searched and returned.

Another problem is there's refactoring to existing infrastructure and 
patterns around the creation and composition of IndexDaos. In addition, 
configuration changes will need to be made (and thought out) without that 
interface existing.  I'm not sure how large that refactoring is, but I think 
changing and testing that is nontrivial enough that it should be a separate PR.

In particular, I'm worried about increasing the scope of an already large 
and complicated PR and I'm disinclined to pull that task into this PR without 
compelling reasons. If removing those interfaces is a necessity for moving 
forward here, I'd like to see that as a PR against the parent feature branch 
that we pull into this branch when it's done. I'm happy to help out using this 
branch (or a derivative thereof) as a pilot / testing ground for that, but 
again I think removing the basic DAO interface itself is outside the scope of 
this.

I'd like to know what specific problems we're going to solve that merit 
that level of change in this PR. IMO, those problems would need to be pretty 
compelling (e.g. we find that reducing the duplication between ES and Solr 
necessitates it) to increase the scope that way unless the changes end up being 
substantially easier and less impactful than I'd expect.


> Create a SolrMetaAlertDao
> -
>
> Key: METRON-1421
> URL: https://issues.apache.org/jira/browse/METRON-1421
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Justin Leet
>Assignee: Justin Leet
>Priority: Major
>
> Create an implementation of the MetaAlertDao for Solr. This will involve 
> implementing the various MetaAlertDao methods using the SolrJ library and 
> also providing a SolrMetaAlertIntegrationTest (similar to 
> ElasticsearchMetaAlertIntegrationTest).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1516) Support for Ansible 2.5.0

[Otto Fowler (JIRA)]

Otto Fowler created METRON-1516:
---

 Summary: Support for Ansible 2.5.0
 Key: METRON-1516
 URL: https://issues.apache.org/jira/browse/METRON-1516
 Project: Metron
  Issue Type: New Feature
Reporter: Otto Fowler
Assignee: Otto Fowler


I was forced to update ansible to 2.5.0 to resolve errors with vagrant, but our 
ansible scripts don't support it.

 

I have 'patch' tested and had no issues.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1421) Create a SolrMetaAlertDao

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432221#comment-16432221
 ] 

ASF GitHub Bot commented on METRON-1421:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/970
  
It is definitely looking better.  I am liking the progress.  But I still 
think having overly broad interfaces that do everything is causing problems; 
primarily `IndexDao` and also `MetaAlertDao`.

Neither of those need to exist.  For example...
* The `MetaAlertService` does not need a `MetaAlertDao`.
* If the `MetaAlertService` needs to update meta-alerts then it should 
directly use a `MetaAlertUpdateDao`.  
* If the `MetaAlertService` needs to search for meta-alerts then it should 
directly use a `MetaAlertSearchDao`.  

I can submit a PR to your branch, if you like.


> Create a SolrMetaAlertDao
> -
>
> Key: METRON-1421
> URL: https://issues.apache.org/jira/browse/METRON-1421
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Justin Leet
>Assignee: Justin Leet
>Priority: Major
>
> Create an implementation of the MetaAlertDao for Solr. This will involve 
> implementing the various MetaAlertDao methods using the SolrJ library and 
> also providing a SolrMetaAlertIntegrationTest (similar to 
> ElasticsearchMetaAlertIntegrationTest).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1356) Add a mechanism in Java for discovering service host/ports

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432286#comment-16432286
 ] 

ASF GitHub Bot commented on METRON-1356:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/973
  
@merrimanr this looks great.  A couple of things:

1.  Can you give some steps on how to test this?  I would like to keep the 
FB moving
2. have you ever looked at https://www.testcontainers.org/?


> Add a mechanism in Java for discovering service host/ports
> --
>
> Key: METRON-1356
> URL: https://issues.apache.org/jira/browse/METRON-1356
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Ryan Merriman
>Priority: Major
>
> Integration tests will need to initialize clients with service urls.  These 
> may change depending on where and how the infrastructure is run (Docker 
> engine vs Docker for Mac).  It would be helpful to have a unified way of 
> retrieving these across all integration tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #973: METRON-1356: Add a mechanism in Java for discovering serv...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/973
  
@merrimanr this looks great.  A couple of things:

1.  Can you give some steps on how to test this?  I would like to keep the 
FB moving
2. have you ever looked at https://www.testcontainers.org/?


---

[GitHub] metron issue #713: METRON-1130 - Updating Amazon EC2 README.md

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/713
  
what is the status of this PR?


---

[jira] [Commented] (METRON-1130) Updating README.md - Amazon EC2 deployment instructions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432303#comment-16432303
 ] 

ASF GitHub Bot commented on METRON-1130:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/713
  
what is the status of this PR?


> Updating README.md - Amazon EC2 deployment instructions
> ---
>
> Key: METRON-1130
> URL: https://issues.apache.org/jira/browse/METRON-1130
> Project: Metron
>  Issue Type: Improvement
>Affects Versions: 0.4.0
> Environment: $ cat /etc/centos-release
> ClearOS release 7.4.0
> ---
> ClearOS is based on CentOS
> 
> [x@x]# mvn -version
> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 
> 2015-11-10T11:41:47-05:00)
> [x@x]# ansible --version
> ansible 2.2.2.0
> [x@x]# python -V
> Python 2.7.5
> [x@x]# git --version
> git version 1.8.3.1
> [x@x]# docker --version
> Docker version 1.12.6, build 88a4867/1.12.6
> [x@x]# npm --version
> 3.10.10
> [x@x]# node -v
> v6.11.1
> [x@x]# bzip2 -V
> bzip2, a block-sorting file compressor.  Version 1.0.6, 6-Sept-2010.
>Reporter: ashah
>Assignee: ashah
>Priority: Trivial
>  Labels: documentation, newbie, starter
> Fix For: 0.4.0
>
>
> PR is updating doc based on experience with installing 0.4.0 using a linux 
> env (ClearOS release 7.4.0 - ClearOS is based on CentOS).
> https://github.com/apache/metron/pull/713



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1507) Add support for Vagrant Cachier plugin if present

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432314#comment-16432314
 ] 

ASF GitHub Bot commented on METRON-1507:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/979
  
I think to test we need to try with and without the plugin


> Add support for Vagrant Cachier plugin if present
> -
>
> Key: METRON-1507
> URL: https://issues.apache.org/jira/browse/METRON-1507
> Project: Metron
>  Issue Type: Improvement
>Reporter: Simon Elliston Ball
>Priority: Trivial
>
> Add configuration in development vagrant files to support vagrant cachier 
> ([http://fgrehm.viewdocs.io/vagrant-cachier/)] to reduce development machine 
> install times. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #970: METRON-1421: Create a SolrMetaAlertDao

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/970
  
@nickwallen
Right now those interfaces are basically a marker interface that requires 
everything that's needed for a complete DAO set. Prior to some of the Solr 
refactorings, it existed as a single DAO to provide a complete set of 
functionality to be implemented in order to allow an implementation to be 
complete (for some definition of complete) and available to be swapped out via 
configuration.  Even with the refactoring into multiple interfaces, the 
IndexDao and MetaAlertDao still exist to provide some definition of complete.

Getting rid of IndexDao in particular likely has potentially broader 
implications because right now that interface ties together the various layers. 
Making those changes in a complete and consistent manner is larger than 
MetaAlertService, because the SearchServiceImpl and others take an IndexDao 
(which could be a regular SolrDao or SolrMetaAlertDao, etc.) in order to allow 
regular searches to have meta alert adjustments, e.g. massage the queries to 
make sure metaalerts get searched and returned.

Another problem is there's refactoring to existing infrastructure and 
patterns around the creation and composition of IndexDaos. In addition, 
configuration changes will need to be made (and thought out) without that 
interface existing.  I'm not sure how large that refactoring is, but I think 
changing and testing that is nontrivial enough that it should be a separate PR.

In particular, I'm worried about increasing the scope of an already large 
and complicated PR and I'm disinclined to pull that task into this PR without 
compelling reasons. If removing those interfaces is a necessity for moving 
forward here, I'd like to see that as a PR against the parent feature branch 
that we pull into this branch when it's done. I'm happy to help out using this 
branch (or a derivative thereof) as a pilot / testing ground for that, but 
again I think removing the basic DAO interface itself is outside the scope of 
this.

I'd like to know what specific problems we're going to solve that merit 
that level of change in this PR. IMO, those problems would need to be pretty 
compelling (e.g. we find that reducing the duplication between ES and Solr 
necessitates it) to increase the scope that way unless the changes end up being 
substantially easier and less impactful than I'd expect.


---

[jira] [Commented] (METRON-1184) EC2 Deployment - Updating control_path to accommodate for Linux

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432305#comment-16432305
 ] 

ASF GitHub Bot commented on METRON-1184:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/754
  
What is the status of this?  Can someone who can do EC2 deploys test it out?


> EC2 Deployment - Updating control_path to accommodate for Linux
> ---
>
> Key: METRON-1184
> URL: https://issues.apache.org/jira/browse/METRON-1184
> Project: Metron
>  Issue Type: Improvement
>Affects Versions: 0.4.1
> Environment: ClearOS[Distro of CentOS] npm 3.10.10, node 6.11.1, 
> docker 1.12.6, mvn 3.3.9, ansible 2.2.2.0, No Vagrant)
>Reporter: ashah
>Assignee: ashah
>Priority: Minor
>  Labels: amazon, cloud
> Fix For: Next + 1, 0.4.1
>
>
> PR: https://github.com/apache/metron/pull/754
> For 0.4.1-rc4, using "control_path = ~/.ssh/ansible-ssh-%%C" for a Linux 
> environment ( such as ClearOS[Distro of CentOS] npm 3.10.10, node 6.11.1, 
> docker 1.12.6, mvn 3.3.9, ansible 2.2.2.0, No Vagrant) could throw the 
> following error:
> ```
> "TASK [setup] 
> ***
> fatal: [ec2-xx-yy-0-130.us-west-2.compute.amazonaws.com]: UNREACHABLE! => 
> {"changed": false, "msg": "Failed to connect to the host via ssh: 
> percent_expand: unknown key %C\r\n", "unreachable": true}" 
> ```
> This PR is to change the line to "control_path = ~/.ssh/ansbile-ssh-%%h-%%r" 
> to make amazon-ec2 also potentially deployable with Linux while preserving a 
> small path length. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #775: [METRON-1214] rpm build fails due to npm absence in Docke...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/775
  
@DimDroll any update?  Or can you close this?


---

[GitHub] metron issue #439: METRON-571 add stellar external functions feature - code ...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/439
  
@devopsec : Tyler, can you update this or close it?


---

[jira] [Commented] (METRON-571) Add stellar keywords for executing local commands

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432291#comment-16432291
 ] 

ASF GitHub Bot commented on METRON-571:
---

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/439
  
@devopsec : Tyler, can you update this or close it?


> Add stellar keywords for executing local commands
> -
>
> Key: METRON-571
> URL: https://issues.apache.org/jira/browse/METRON-571
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Priority: Minor
> Attachments: ExternalFunctions.java, ExternalFunctionsTest.java
>
>
> Stellar should have the ability to execute scripts on a server in order to 
> undertake automated mitigation or alerting.  Perhaps SHELL_EXEC?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1090) STELLAR Add Assignment capability to Stellar language

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432302#comment-16432302
 ] 

ASF GitHub Bot commented on METRON-1090:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/687
  
This is pretty old at this point, and there has been a lot of discussion 
all over the place about it.  I'm not sure I can sum up where we are.  
@cestella can you sum up your current feeling on how we can move this forward?


> STELLAR Add Assignment capability to Stellar language
> -
>
> Key: METRON-1090
> URL: https://issues.apache.org/jira/browse/METRON-1090
> Project: Metron
>  Issue Type: New Feature
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Add support for =, +=, -=, *=, /= for variables in stellar.
> foo = 5  returns 5 AND sets foo to 5 if the VariableResolver supports it.
> The shell should be changed to assign variables using this syntax, because 
> exposing the left side to stellar allows catching errors for invalid keywords 
> in left side.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #754: METRON-1184 EC2 Deployment - Updating control_path to acc...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/754
  
What is the status of this?  Can someone who can do EC2 deploys test it out?


---

[jira] [Commented] (METRON-1184) EC2 Deployment - Updating control_path to accommodate for Linux

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432335#comment-16432335
 ] 

ASF GitHub Bot commented on METRON-1184:


Github user lvets commented on the issue:

https://github.com/apache/metron/pull/754
  
@ottobackwards I will try this out this week.


> EC2 Deployment - Updating control_path to accommodate for Linux
> ---
>
> Key: METRON-1184
> URL: https://issues.apache.org/jira/browse/METRON-1184
> Project: Metron
>  Issue Type: Improvement
>Affects Versions: 0.4.1
> Environment: ClearOS[Distro of CentOS] npm 3.10.10, node 6.11.1, 
> docker 1.12.6, mvn 3.3.9, ansible 2.2.2.0, No Vagrant)
>Reporter: ashah
>Assignee: ashah
>Priority: Minor
>  Labels: amazon, cloud
> Fix For: Next + 1, 0.4.1
>
>
> PR: https://github.com/apache/metron/pull/754
> For 0.4.1-rc4, using "control_path = ~/.ssh/ansible-ssh-%%C" for a Linux 
> environment ( such as ClearOS[Distro of CentOS] npm 3.10.10, node 6.11.1, 
> docker 1.12.6, mvn 3.3.9, ansible 2.2.2.0, No Vagrant) could throw the 
> following error:
> ```
> "TASK [setup] 
> ***
> fatal: [ec2-xx-yy-0-130.us-west-2.compute.amazonaws.com]: UNREACHABLE! => 
> {"changed": false, "msg": "Failed to connect to the host via ssh: 
> percent_expand: unknown key %C\r\n", "unreachable": true}" 
> ```
> This PR is to change the line to "control_path = ~/.ssh/ansbile-ssh-%%h-%%r" 
> to make amazon-ec2 also potentially deployable with Linux while preserving a 
> small path length. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #754: METRON-1184 EC2 Deployment - Updating control_path to acc...

[lvets]

Github user lvets commented on the issue:

https://github.com/apache/metron/pull/754
  
@ottobackwards I will try this out this week.


---

[GitHub] metron issue #970: METRON-1421: Create a SolrMetaAlertDao

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/970
  
Do you think we have enough unit tests on this functionality how it is?


---

[jira] [Commented] (METRON-1086) Create a Blockly-based user interface for Stellar

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432483#comment-16432483
 ] 

ASF GitHub Bot commented on METRON-1086:


Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/684
  
Sorry for ignoring this for so long.  This is going to be a fairly complex 
module to maintain and it is VERY tightly coupled with Stellar.  This means 
that if we want to officially add this feature it will place an additional and 
substantial burden on people making changes to Stellar.

I am happy to lead this but it should be something that is overwhelmingly 
supported by the community.  It will take a lot of effort from multiple people 
to make this comprehensive and robust.  I would need volunteers to help me.

Might make sense to revive this once we are confident Stellar is very 
stable and won't change much.  Not sure if that will ever happen since it's so 
flexible and has already evolved so much :)


> Create a Blockly-based user interface for Stellar
> -
>
> Key: METRON-1086
> URL: https://issues.apache.org/jira/browse/METRON-1086
> Project: Metron
>  Issue Type: New Feature
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
>
> Having a visual Stellar editor would be useful.  Blockly is a Google library 
> for building visual programming editors:  
> https://developers.google.com/blockly/.  This feature will integrate the two 
> and produce a reusable javascript widget for editing Stellar code.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #970: METRON-1421: Create a SolrMetaAlertDao

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/970
  
As we are already on a feature branch, can we not iterate and refactor 
this?  Does it all have to go into *this* pr?

If this is landed, what is keeping @nickwallen from throwing a PR at the 
feature branch?



---

[GitHub] metron issue #684: DO NOT MERGE: METRON-1086: Create a Blockly-based user in...

[lvets]

Github user lvets commented on the issue:

https://github.com/apache/metron/pull/684
  
I can help out with testing wherever possible. This is a very cool 
feature...


---

[jira] [Created] (METRON-1518) Build Failure When Using Profile HDP-2.5.0.0

[Nick Allen (JIRA)]

Nick Allen created METRON-1518:
--

 Summary: Build Failure When Using Profile HDP-2.5.0.0
 Key: METRON-1518
 URL: https://issues.apache.org/jira/browse/METRON-1518
 Project: Metron
  Issue Type: Bug
Reporter: Nick Allen
Assignee: Nick Allen


{code}
/Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:45:
error: cannot find symbol
import org.apache.storm.StormTimer;
   ^
  symbol:   class StormTimer
  location: package org.apache.storm
/Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:167:
error: cannot find symbol
  private StormTimer expiredFlushTimer;
  ^
  symbol:   class StormTimer
  location: class ProfileBuilderBolt
/Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:450:
error: cannot find symbol
  private StormTimer createTimer(String name) {
  ^
  symbol:   class StormTimer
  location: class ProfileBuilderBolt
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1421) Create a SolrMetaAlertDao

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432486#comment-16432486
 ] 

ASF GitHub Bot commented on METRON-1421:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/970
  
As we are already on a feature branch, can we not iterate and refactor 
this?  Does it all have to go into *this* pr?

If this is landed, what is keeping @nickwallen from throwing a PR at the 
feature branch?



> Create a SolrMetaAlertDao
> -
>
> Key: METRON-1421
> URL: https://issues.apache.org/jira/browse/METRON-1421
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Justin Leet
>Assignee: Justin Leet
>Priority: Major
>
> Create an implementation of the MetaAlertDao for Solr. This will involve 
> implementing the various MetaAlertDao methods using the SolrJ library and 
> also providing a SolrMetaAlertIntegrationTest (similar to 
> ElasticsearchMetaAlertIntegrationTest).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #684: DO NOT MERGE: METRON-1086: Create a Blockly-based user in...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/684
  
I would be happy to help, I think I have a decent handle with the stellar 
side, and could get up to speed on the ui stuff with your help.

Think about how we can go about it.
I think I would need to understand better how this works, what tightly 
coupled means etc.  Maybe we can slice this up between ui and backend?


---

[jira] [Commented] (METRON-1086) Create a Blockly-based user interface for Stellar

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432520#comment-16432520
 ] 

ASF GitHub Bot commented on METRON-1086:


Github user lvets commented on the issue:

https://github.com/apache/metron/pull/684
  
I can help out with testing wherever possible. This is a very cool 
feature...


> Create a Blockly-based user interface for Stellar
> -
>
> Key: METRON-1086
> URL: https://issues.apache.org/jira/browse/METRON-1086
> Project: Metron
>  Issue Type: New Feature
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
>
> Having a visual Stellar editor would be useful.  Blockly is a Google library 
> for building visual programming editors:  
> https://developers.google.com/blockly/.  This feature will integrate the two 
> and produce a reusable javascript widget for editing Stellar code.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #970: METRON-1421: Create a SolrMetaAlertDao

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/970
  
@ottobackwards Correct me if I'm misunderstanding your view, @nickwallen, 
but the discussion is around what's the required level of iteration / 
refactoring where this meets expectations for testing and maintainability 
before this can go into the feature branch at all. I'm fine iterating 
afterwards (and that may include things like overall DAO restructuring an other 
things, depending on what ends up happening here).  What we're trying to avoid 
is a situation where we immediately want to do large scale refactoring 
immediately (whether as part of the feature branch itself or master).  This is 
(potentially) one of the final (and largest + most complicated) components of 
the feature branch, so I think that's where the concern from both myself and 
Nick is.

Once this is landed (or before or whenever), we can throw whatever PRs at 
the feature branch.


---

[jira] [Commented] (METRON-1421) Create a SolrMetaAlertDao

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432539#comment-16432539
 ] 

ASF GitHub Bot commented on METRON-1421:


Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/970
  
@ottobackwards Correct me if I'm misunderstanding your view, @nickwallen, 
but the discussion is around what's the required level of iteration / 
refactoring where this meets expectations for testing and maintainability 
before this can go into the feature branch at all. I'm fine iterating 
afterwards (and that may include things like overall DAO restructuring an other 
things, depending on what ends up happening here).  What we're trying to avoid 
is a situation where we immediately want to do large scale refactoring 
immediately (whether as part of the feature branch itself or master).  This is 
(potentially) one of the final (and largest + most complicated) components of 
the feature branch, so I think that's where the concern from both myself and 
Nick is.

Once this is landed (or before or whenever), we can throw whatever PRs at 
the feature branch.


> Create a SolrMetaAlertDao
> -
>
> Key: METRON-1421
> URL: https://issues.apache.org/jira/browse/METRON-1421
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Justin Leet
>Assignee: Justin Leet
>Priority: Major
>
> Create an implementation of the MetaAlertDao for Solr. This will involve 
> implementing the various MetaAlertDao methods using the SolrJ library and 
> also providing a SolrMetaAlertIntegrationTest (similar to 
> ElasticsearchMetaAlertIntegrationTest).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1517) GeoLite discontinuation notice

[Laurens Vets (JIRA)]

Laurens Vets created METRON-1517:


 Summary: GeoLite discontinuation notice
 Key: METRON-1517
 URL: https://issues.apache.org/jira/browse/METRON-1517
 Project: Metron
  Issue Type: Bug
Reporter: Laurens Vets


I believe the geolite db used by Metron will be discontinued:

https://support.maxmind.com/geolite-legacy-discontinuation-notice/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1421) Create a SolrMetaAlertDao

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432619#comment-16432619
 ] 

ASF GitHub Bot commented on METRON-1421:


Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/970
  
@ottobackwards You're absolutely not confusing things.  It's important to 
have that perspective, and I am 100% appreciative of you hopping in and 
offering up your opinion.  I think you're right that we probably need to reach 
a solid compromise position rather than boiling the ocean.

@nickwallen What if, for this PR, we do the following:
1. Mostly keep the interface refactoring from the branch.
2. Instead of pushing everything directly into SolrUpdateMetaAlertDao and 
ElasticsearchUpdateMetaAlertDao, we push to AbstractLuceneUpdateMetaAlertDao 
minimally as needed to prevent duping (which is admittedly substantial)?  
Everything else pushes down to the specific DAO which extends the abstract 
DAOs. As in the test branch, the SolrMetaAlertDao just uses and passes along 
the appropriate impls as needed.
3. Repeat (2) for the other DAOs implementations
4. I take another pass at tests to make sure both the common and specific 
impls are properly tested.

At that point, it should somewhat clean up the type hierarchy which will 
enable us to make things more pluggable in the future (including potentially 
working to eliminate the aggregate marker interfaces). At the very least, I 
think it makes it more straightforward, although it's not perfect. This should 
improve maintainability and testing. The testing pass should make sure we're at 
least better positioned to be confident in future refactorings.

I 100% admit this doesn't completely address the composition vs. extension, 
but it at least makes progress relative towards that goal relative to where we 
were.  It has the benefit of being a doable fairly low-risk refactoring that 
enables the PR to make it into the branch.

Hopefully this has the elements of a good compromise: Nobody is overjoyed, 
but nobody feels cheated.  In addition, we at least have set ourselves up to 
make it easier for everyone to be happier in the future.


> Create a SolrMetaAlertDao
> -
>
> Key: METRON-1421
> URL: https://issues.apache.org/jira/browse/METRON-1421
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Justin Leet
>Assignee: Justin Leet
>Priority: Major
>
> Create an implementation of the MetaAlertDao for Solr. This will involve 
> implementing the various MetaAlertDao methods using the SolrJ library and 
> also providing a SolrMetaAlertIntegrationTest (similar to 
> ElasticsearchMetaAlertIntegrationTest).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #970: METRON-1421: Create a SolrMetaAlertDao

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/970
  
@merrimanr I should probably hit you up too before we confirm that's what 
the plan is, since you've run stuff up a couple times and been pretty involved 
in the Solr work.  Does the above plan for reaching an end state to the PR work 
for you?


---

[jira] [Commented] (METRON-1517) GeoLite discontinuation notice

[Laurens Vets (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432651#comment-16432651
 ] 

Laurens Vets commented on METRON-1517:
--

Don't we need to adapt Metron for:
h4. *Important Notice for GeoLite2 Users*

We have decided to discontinue providing “latitude” and “longitude” with the 
GeoLite2 City database. Once discontinued, any integration using these keys 
will no longer return the approximate latitude and longitude of the most 
granular geolocation returned for the IP queried: postal code, city, 
subdivision or country. Please take action to update your integration as needed.

> GeoLite discontinuation notice
> --
>
> Key: METRON-1517
> URL: https://issues.apache.org/jira/browse/METRON-1517
> Project: Metron
>  Issue Type: Bug
>Reporter: Laurens Vets
>Priority: Major
>
> I believe the geolite db used by Metron will be discontinued:
> https://support.maxmind.com/geolite-legacy-discontinuation-notice/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1517) GeoLite discontinuation notice

[Justin Leet (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432672#comment-16432672
 ] 

Justin Leet commented on METRON-1517:
-

Correct, we do use GeoLite2 (although we rename some fields internally to match 
the original GeoLite database.  There's a ticket to not do that, iirc.).

Latitude and longitude being removed is interesting though.

> GeoLite discontinuation notice
> --
>
> Key: METRON-1517
> URL: https://issues.apache.org/jira/browse/METRON-1517
> Project: Metron
>  Issue Type: Bug
>Reporter: Laurens Vets
>Priority: Major
>
> I believe the geolite db used by Metron will be discontinued:
> https://support.maxmind.com/geolite-legacy-discontinuation-notice/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #970: METRON-1421: Create a SolrMetaAlertDao

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/970
  
@justinleet @nickwallen , sorry I didn't mean to confuse things.  It seems 
to me that @nickwallen is willing to do some work on this per his 'vision' and 
that might be easier if this landed in a known state rather than thrashing the 
pr.  But I don't have skin in this review so to speak.  Just trying to help.


---

[jira] [Commented] (METRON-1421) Create a SolrMetaAlertDao

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432597#comment-16432597
 ] 

ASF GitHub Bot commented on METRON-1421:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/970
  
@justinleet @nickwallen , sorry I didn't mean to confuse things.  It seems 
to me that @nickwallen is willing to do some work on this per his 'vision' and 
that might be easier if this landed in a known state rather than thrashing the 
pr.  But I don't have skin in this review so to speak.  Just trying to help.


> Create a SolrMetaAlertDao
> -
>
> Key: METRON-1421
> URL: https://issues.apache.org/jira/browse/METRON-1421
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Justin Leet
>Assignee: Justin Leet
>Priority: Major
>
> Create an implementation of the MetaAlertDao for Solr. This will involve 
> implementing the various MetaAlertDao methods using the SolrJ library and 
> also providing a SolrMetaAlertIntegrationTest (similar to 
> ElasticsearchMetaAlertIntegrationTest).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #970: METRON-1421: Create a SolrMetaAlertDao

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/970
  
I agree with your plan @justinleet .  I am on-board.  Thanks for laying 
that out.


---

[jira] [Commented] (METRON-1214) rpm build fails due to npm absence in Dockerfile

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432306#comment-16432306
 ] 

ASF GitHub Bot commented on METRON-1214:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/775
  
@DimDroll any update?  Or can you close this?


> rpm build fails due to npm absence in Dockerfile 
> -
>
> Key: METRON-1214
> URL: https://issues.apache.org/jira/browse/METRON-1214
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.1
>Reporter: Dima Kovalyov
>Priority: Minor
>
> I've received error in metron/metron-deployment/packaging/docker/rpm-docker 
> that npm is not installed, which failed the whole RPM build. To fix it I've 
> added npm to line 28 in Dockerfile:
> {code}
> RUN yum -y install asciidoc rpm-build rpm2cpio tar unzip xmlto zip rpmlint 
> npm && yum clean all
> {code}
> Requirement of npm was introduced after ui and rest addition.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #789: METRON-1233: Remove description of Global configuration f...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/789
  
Still needs conflict resolution, still needs relevancy check


---

[GitHub] metron issue #830: METRON-1298: TimeRange Picker doesn't work on Safari

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/830
  
What is the status of this PR?


---

[jira] [Commented] (METRON-1421) Create a SolrMetaAlertDao

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432370#comment-16432370
 ] 

ASF GitHub Bot commented on METRON-1421:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/970
  
Do you think we have enough unit tests on this functionality how it is?


> Create a SolrMetaAlertDao
> -
>
> Key: METRON-1421
> URL: https://issues.apache.org/jira/browse/METRON-1421
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Justin Leet
>Assignee: Justin Leet
>Priority: Major
>
> Create an implementation of the MetaAlertDao for Solr. This will involve 
> implementing the various MetaAlertDao methods using the SolrJ library and 
> also providing a SolrMetaAlertIntegrationTest (similar to 
> ElasticsearchMetaAlertIntegrationTest).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1421) Create a SolrMetaAlertDao

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432442#comment-16432442
 ] 

ASF GitHub Bot commented on METRON-1421:


Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/970
  
@nickwallen 
Assuming we go through with the partial refactoring, the existence of those 
interfaces does not prevent us from testing the individual DAOs. A lot of the 
highly specific changes at the implementation level of MetaAlertDao are things 
that more or less need to be integration tested (e.g. making sure the magic 
Solr incantations do what's expected).  The other stuff can be unit tested 
regardless of what implementation class it's in.

Re: #832 Absolutely agree we should have as many unit tests as possible to 
avoid things like this.

The tests are substantially better than they were prior to this PR. I'm 
happy to take another pass to try to catch out any cases we're missing.  The 
initial pass took place during a refactoring and I wouldn't be surprised if 
there's more we can do.  The problem is that I need to have a stable code base 
that I'm testing against. That's the main reason I'm trying to get this 
narrowed down and resolve the overall code structure.  I'm not trying to say 
that we're done adding tests or that there's not room for improvement.  I'm 
trying to get to a point where I'm not trying to build the tests we'd both like 
to see on a pile of sand.  For me, killing that dedupe is a prerequisite 
because it could change how those unit tests happen in terms of avoiding 
duplication and ensuring that the variants on the implementations are more 
easily understood and tested.  Right now it's hard to build expectations 
against the code, because we're changing basic structure and how things are 
built.  Even if the test cases themselves cover the same thing, a lot of stuff 
like setup changes so there's nontrivial effort involved in keeping the tests 
aligned.

IMO (and feel free, or even encouraged, to argue with me on this), killing 
the interfaces doesn't buy us enough testability to merit adding that 
refactoring to this PR.  I do think rethinking the DAO structure is a 
definitely a worthwhile task that could improve a lot of things at once, but it 
should be a separate effort.

I think a reasonable level of testing for this PR is the integration 
testing at the level of broad interface DAOs + the already substantial increase 
in testing + maybe another pass once the implementation has stabilized. Let me 
know if your expectations are different, because it sounds like both of our 
expectations are influencing how much refactoring we want, so we need to be on 
the same page there.

Re: the issue on this PR itself, that appears to be an issue with using the 
correct source.type field for Solr. I'm working on a confirming and providing a 
separate fix for that.  Part of the problem appears to be the unit tests 
themselves are incorrect. The other problem is that our test schemas 
incorrectly use ':' instead of '.' and I carried this through. So the problem 
isn't that the tests are incomplete, but that it's compounding a bug in our 
tests themselves.  In fact, I called out that I thought it was weird we were 
using ':' in the PR description.


> Create a SolrMetaAlertDao
> -
>
> Key: METRON-1421
> URL: https://issues.apache.org/jira/browse/METRON-1421
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Justin Leet
>Assignee: Justin Leet
>Priority: Major
>
> Create an implementation of the MetaAlertDao for Solr. This will involve 
> implementing the various MetaAlertDao methods using the SolrJ library and 
> also providing a SolrMetaAlertIntegrationTest (similar to 
> ElasticsearchMetaAlertIntegrationTest).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #970: METRON-1421: Create a SolrMetaAlertDao

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/970
  
@nickwallen 
Assuming we go through with the partial refactoring, the existence of those 
interfaces does not prevent us from testing the individual DAOs. A lot of the 
highly specific changes at the implementation level of MetaAlertDao are things 
that more or less need to be integration tested (e.g. making sure the magic 
Solr incantations do what's expected).  The other stuff can be unit tested 
regardless of what implementation class it's in.

Re: #832 Absolutely agree we should have as many unit tests as possible to 
avoid things like this.

The tests are substantially better than they were prior to this PR. I'm 
happy to take another pass to try to catch out any cases we're missing.  The 
initial pass took place during a refactoring and I wouldn't be surprised if 
there's more we can do.  The problem is that I need to have a stable code base 
that I'm testing against. That's the main reason I'm trying to get this 
narrowed down and resolve the overall code structure.  I'm not trying to say 
that we're done adding tests or that there's not room for improvement.  I'm 
trying to get to a point where I'm not trying to build the tests we'd both like 
to see on a pile of sand.  For me, killing that dedupe is a prerequisite 
because it could change how those unit tests happen in terms of avoiding 
duplication and ensuring that the variants on the implementations are more 
easily understood and tested.  Right now it's hard to build expectations 
against the code, because we're changing basic structure and how things are 
built.  Even if
  the test cases themselves cover the same thing, a lot of stuff like setup 
changes so there's nontrivial effort involved in keeping the tests aligned.

IMO (and feel free, or even encouraged, to argue with me on this), killing 
the interfaces doesn't buy us enough testability to merit adding that 
refactoring to this PR.  I do think rethinking the DAO structure is a 
definitely a worthwhile task that could improve a lot of things at once, but it 
should be a separate effort.

I think a reasonable level of testing for this PR is the integration 
testing at the level of broad interface DAOs + the already substantial increase 
in testing + maybe another pass once the implementation has stabilized. Let me 
know if your expectations are different, because it sounds like both of our 
expectations are influencing how much refactoring we want, so we need to be on 
the same page there.

Re: the issue on this PR itself, that appears to be an issue with using the 
correct source.type field for Solr. I'm working on a confirming and providing a 
separate fix for that.  Part of the problem appears to be the unit tests 
themselves are incorrect. The other problem is that our test schemas 
incorrectly use ':' instead of '.' and I carried this through. So the problem 
isn't that the tests are incomplete, but that it's compounding a bug in our 
tests themselves.  In fact, I called out that I thought it was weird we were 
using ':' in the PR description.


---

[jira] [Updated] (METRON-1518) Build Failure When Using Profile HDP-2.5.0.0

[Nick Allen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/METRON-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Allen updated METRON-1518:
---
Description: 
{code}
/Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:45:
error: cannot find symbol
import org.apache.storm.StormTimer;
   ^
  symbol:   class StormTimer
  location: package org.apache.storm
/Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:167:
error: cannot find symbol
  private StormTimer expiredFlushTimer;
  ^
  symbol:   class StormTimer
  location: class ProfileBuilderBolt
/Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:450:
error: cannot find symbol
  private StormTimer createTimer(String name) {
  ^
  symbol:   class StormTimer
  location: class ProfileBuilderBolt
{code}


  was:
{code}
/Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:45:
error: cannot find symbol
import org.apache.storm.StormTimer;
   ^
  symbol:   class StormTimer
  location: package org.apache.storm
/Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:167:
error: cannot find symbol
  private StormTimer expiredFlushTimer;
  ^
  symbol:   class StormTimer
  location: class ProfileBuilderBolt
/Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:450:
error: cannot find symbol
  private StormTimer createTimer(String name) {
  ^
  symbol:   class StormTimer
  location: class ProfileBuilderBolt
{code}


> Build Failure When Using Profile HDP-2.5.0.0
> 
>
> Key: METRON-1518
> URL: https://issues.apache.org/jira/browse/METRON-1518
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> {code}
> /Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:45:
> error: cannot find symbol
> import org.apache.storm.StormTimer;
>^
>   symbol:   class StormTimer
>   location: package org.apache.storm
> /Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:167:
> error: cannot find symbol
>   private StormTimer expiredFlushTimer;
>   ^
>   symbol:   class StormTimer
>   location: class ProfileBuilderBolt
> /Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:450:
> error: cannot find symbol
>   private StormTimer createTimer(String name) {
>   ^
>   symbol:   class StormTimer
>   location: class ProfileBuilderBolt
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1518) Build Failure When Using Profile HDP-2.5.0.0

[Nick Allen (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432704#comment-16432704
 ] 

Nick Allen commented on METRON-1518:


This is caused by my PR; https://github.com/apache/metron/pull/977.

This problem occurs only when building Metron with the '-PHDP‌-2.5.0.0' Maven 
build profile.

When building Metron against this profile, it uses an HDP specific version of 
Storm (1.0.1.2.5.0.0-1245). When building normally, without that profile (how 
Metron is built in Apache-land) then it uses Storm 1.0.3.

The class in question (org.apache.storm.StormTimer) does not exist in HDP Storm 
1.0.1.2.5.0.0-1245, but does exist in Storm 1.0.3.

{code}
nallen@HW12402 : ~/.m2/repository/org/apache/storm/storm-core
$ jar -tvf 
~/.m2/repository/org/apache/storm/storm-core/1.0.3/storm-core-1.0.3.jar | grep 
StormTimer
   848 Tue Feb 07 15:25:56 EST 2017 org/apache/storm/StormTimer$1.class
   902 Tue Feb 07 15:25:56 EST 2017 org/apache/storm/StormTimer$2.class
   650 Tue Feb 07 15:25:56 EST 2017 org/apache/storm/StormTimer$QueueEntry.class
  1275 Tue Feb 07 15:25:56 EST 2017 
org/apache/storm/StormTimer$StormTimerTask$1.class
  2972 Tue Feb 07 15:25:56 EST 2017 
org/apache/storm/StormTimer$StormTimerTask.class
  3684 Tue Feb 07 15:25:56 EST 2017 org/apache/storm/StormTimer.class

nallen@HW12402 : ~/.m2/repository/org/apache/storm/storm-core
$ jar -tvf 
~/.m2/repository/org/apache/storm/storm-core/1.0.1.2.5.0.0-1245/storm-core-1.0.1.2.5.0.0-1245.jar
 | grep StormTimer
{code}

> Build Failure When Using Profile HDP-2.5.0.0
> 
>
> Key: METRON-1518
> URL: https://issues.apache.org/jira/browse/METRON-1518
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> {code}
> /Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:45:
> error: cannot find symbol
> import org.apache.storm.StormTimer;
>^
>   symbol:   class StormTimer
>   location: package org.apache.storm
> /Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:167:
> error: cannot find symbol
>   private StormTimer expiredFlushTimer;
>   ^
>   symbol:   class StormTimer
>   location: class ProfileBuilderBolt
> /Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:450:
> error: cannot find symbol
>   private StormTimer createTimer(String name) {
>   ^
>   symbol:   class StormTimer
>   location: class ProfileBuilderBolt
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1086) Create a Blockly-based user interface for Stellar

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432507#comment-16432507
 ] 

ASF GitHub Bot commented on METRON-1086:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/684
  
I would be happy to help, I think I have a decent handle with the stellar 
side, and could get up to speed on the ui stuff with your help.

Think about how we can go about it.
I think I would need to understand better how this works, what tightly 
coupled means etc.  Maybe we can slice this up between ui and backend?


> Create a Blockly-based user interface for Stellar
> -
>
> Key: METRON-1086
> URL: https://issues.apache.org/jira/browse/METRON-1086
> Project: Metron
>  Issue Type: New Feature
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
>
> Having a visual Stellar editor would be useful.  Blockly is a Google library 
> for building visual programming editors:  
> https://developers.google.com/blockly/.  This feature will integrate the two 
> and produce a reusable javascript widget for editing Stellar code.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1517) GeoLite discontinuation notice

[Nick Allen (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432607#comment-16432607
 ] 

Nick Allen commented on METRON-1517:


I read this as a discontinuation of GeoLite and that they are pushing everyone 
to GeoLite2.  AFAIK, Metron already uses GeoLite2.

https://dev.maxmind.com/geoip/geoip2/geolite2/

> GeoLite discontinuation notice
> --
>
> Key: METRON-1517
> URL: https://issues.apache.org/jira/browse/METRON-1517
> Project: Metron
>  Issue Type: Bug
>Reporter: Laurens Vets
>Priority: Major
>
> I believe the geolite db used by Metron will be discontinued:
> https://support.maxmind.com/geolite-legacy-discontinuation-notice/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1421) Create a SolrMetaAlertDao

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432640#comment-16432640
 ] 

ASF GitHub Bot commented on METRON-1421:


Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/970
  
@merrimanr I should probably hit you up too before we confirm that's what 
the plan is, since you've run stuff up a couple times and been pretty involved 
in the Solr work.  Does the above plan for reaching an end state to the PR work 
for you?


> Create a SolrMetaAlertDao
> -
>
> Key: METRON-1421
> URL: https://issues.apache.org/jira/browse/METRON-1421
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Justin Leet
>Assignee: Justin Leet
>Priority: Major
>
> Create an implementation of the MetaAlertDao for Solr. This will involve 
> implementing the various MetaAlertDao methods using the SolrJ library and 
> also providing a SolrMetaAlertIntegrationTest (similar to 
> ElasticsearchMetaAlertIntegrationTest).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #684: DO NOT MERGE: METRON-1086: Create a Blockly-based user in...

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/684
  
Sorry for ignoring this for so long.  This is going to be a fairly complex 
module to maintain and it is VERY tightly coupled with Stellar.  This means 
that if we want to officially add this feature it will place an additional and 
substantial burden on people making changes to Stellar.

I am happy to lead this but it should be something that is overwhelmingly 
supported by the community.  It will take a lot of effort from multiple people 
to make this comprehensive and robust.  I would need volunteers to help me.

Might make sense to revive this once we are confident Stellar is very 
stable and won't change much.  Not sure if that will ever happen since it's so 
flexible and has already evolved so much :)


---

[GitHub] metron issue #970: METRON-1421: Create a SolrMetaAlertDao

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/970
  
Just to summarize my position, my suggested refactorings are just one of 
many ways to improve the testability of all this.  We don't have to take this 
approach.

For this PR, I will be happy if we think we have sufficient unit test 
coverage on the functionality here.  If @justinleet thinks we are good on unit 
tests now, I will take another look at the unit tests today.

> @justinleet:  What we're trying to avoid is a situation where we 
immediately want to do large scale refactoring immediately (whether as part of 
the feature branch itself or master).

To be honest, ultimately, I would like to refactor this (iteratively) to be 
simpler and more testable, but it doesn't have to happen on this PR or even on 
this feature branch or even in the near-term.  I don't know when I'd be able to 
get to that.

@justinleet What do you think is a good path forward?  I want to keep an 
open mind and make sure we're not doing more than we need to here.




---

[jira] [Commented] (METRON-1421) Create a SolrMetaAlertDao

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432554#comment-16432554
 ] 

ASF GitHub Bot commented on METRON-1421:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/970
  
Just to summarize my position, my suggested refactorings are just one of 
many ways to improve the testability of all this.  We don't have to take this 
approach.

For this PR, I will be happy if we think we have sufficient unit test 
coverage on the functionality here.  If @justinleet thinks we are good on unit 
tests now, I will take another look at the unit tests today.

> @justinleet:  What we're trying to avoid is a situation where we 
immediately want to do large scale refactoring immediately (whether as part of 
the feature branch itself or master).

To be honest, ultimately, I would like to refactor this (iteratively) to be 
simpler and more testable, but it doesn't have to happen on this PR or even on 
this feature branch or even in the near-term.  I don't know when I'd be able to 
get to that.

@justinleet What do you think is a good path forward?  I want to keep an 
open mind and make sure we're not doing more than we need to here.




> Create a SolrMetaAlertDao
> -
>
> Key: METRON-1421
> URL: https://issues.apache.org/jira/browse/METRON-1421
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Justin Leet
>Assignee: Justin Leet
>Priority: Major
>
> Create an implementation of the MetaAlertDao for Solr. This will involve 
> implementing the various MetaAlertDao methods using the SolrJ library and 
> also providing a SolrMetaAlertIntegrationTest (similar to 
> ElasticsearchMetaAlertIntegrationTest).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1421) Create a SolrMetaAlertDao

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432634#comment-16432634
 ] 

ASF GitHub Bot commented on METRON-1421:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/970
  
I agree with your plan @justinleet .  I am on-board.  Thanks for laying 
that out.


> Create a SolrMetaAlertDao
> -
>
> Key: METRON-1421
> URL: https://issues.apache.org/jira/browse/METRON-1421
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Justin Leet
>Assignee: Justin Leet
>Priority: Major
>
> Create an implementation of the MetaAlertDao for Solr. This will involve 
> implementing the various MetaAlertDao methods using the SolrJ library and 
> also providing a SolrMetaAlertIntegrationTest (similar to 
> ElasticsearchMetaAlertIntegrationTest).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #986: METRON-1518 Build Failure When Using Profile HDP-2...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/986#discussion_r180544522
  
--- Diff: .travis.yml ---
@@ -33,7 +33,7 @@ before_install:
   - npm config set prefix $HOME/.npm-prefix --global
 
 install:
-  - time mvn -q -T 2C -DskipTests clean install
+  - time mvn -q -T 2C -DskipTests -PHDP-2.5.0.0 clean install
--- End diff --

Just so I'm completely clear, by "build matrix" I mean 
[this](https://docs.travis-ci.com/user/customizing-the-build#Build-Matrix)


---

[jira] [Commented] (METRON-1518) Build Failure When Using Profile HDP-2.5.0.0

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432841#comment-16432841
 ] 

ASF GitHub Bot commented on METRON-1518:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/986#discussion_r180544522
  
--- Diff: .travis.yml ---
@@ -33,7 +33,7 @@ before_install:
   - npm config set prefix $HOME/.npm-prefix --global
 
 install:
-  - time mvn -q -T 2C -DskipTests clean install
+  - time mvn -q -T 2C -DskipTests -PHDP-2.5.0.0 clean install
--- End diff --

Just so I'm completely clear, by "build matrix" I mean 
[this](https://docs.travis-ci.com/user/customizing-the-build#Build-Matrix)


> Build Failure When Using Profile HDP-2.5.0.0
> 
>
> Key: METRON-1518
> URL: https://issues.apache.org/jira/browse/METRON-1518
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> {code}
> /Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:45:
> error: cannot find symbol
> import org.apache.storm.StormTimer;
>^
>   symbol:   class StormTimer
>   location: package org.apache.storm
> /Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:167:
> error: cannot find symbol
>   private StormTimer expiredFlushTimer;
>   ^
>   symbol:   class StormTimer
>   location: class ProfileBuilderBolt
> /Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:450:
> error: cannot find symbol
>   private StormTimer createTimer(String name) {
>   ^
>   symbol:   class StormTimer
>   location: class ProfileBuilderBolt
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #970: METRON-1421: Create a SolrMetaAlertDao

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/970
  
@mmiklavc I think that's a good insight.  The original intention of 
metaalerts was to basically wrap and augment the functionality of the index 
alerts.  I think a portion of that problem we've hit is that wrapping that is 
both specific in implementation and general in purpose and it blurs that line 
between data store and domain model object. 

I definitely agree that a discuss thread on our access patterns is a good 
outcome of this.  Doing an second implementation is always revealing for things 
like this, and taking that insight and improving things for the future would be 
good.


---

[jira] [Commented] (METRON-1421) Create a SolrMetaAlertDao

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432852#comment-16432852
 ] 

ASF GitHub Bot commented on METRON-1421:


Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/970
  
@mmiklavc I think that's a good insight.  The original intention of 
metaalerts was to basically wrap and augment the functionality of the index 
alerts.  I think a portion of that problem we've hit is that wrapping that is 
both specific in implementation and general in purpose and it blurs that line 
between data store and domain model object. 

I definitely agree that a discuss thread on our access patterns is a good 
outcome of this.  Doing an second implementation is always revealing for things 
like this, and taking that insight and improving things for the future would be 
good.


> Create a SolrMetaAlertDao
> -
>
> Key: METRON-1421
> URL: https://issues.apache.org/jira/browse/METRON-1421
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Justin Leet
>Assignee: Justin Leet
>Priority: Major
>
> Create an implementation of the MetaAlertDao for Solr. This will involve 
> implementing the various MetaAlertDao methods using the SolrJ library and 
> also providing a SolrMetaAlertIntegrationTest (similar to 
> ElasticsearchMetaAlertIntegrationTest).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1518) Build Failure When Using Profile HDP-2.5.0.0

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432837#comment-16432837
 ] 

ASF GitHub Bot commented on METRON-1518:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/986#discussion_r180543990
  
--- Diff: .travis.yml ---
@@ -33,7 +33,7 @@ before_install:
   - npm config set prefix $HOME/.npm-prefix --global
 
 install:
-  - time mvn -q -T 2C -DskipTests clean install
+  - time mvn -q -T 2C -DskipTests -PHDP-2.5.0.0 clean install
--- End diff --

I believe the rationale for having travis run without the 2.5.0.0 profile 
and full-dev build with it is to ensure that we build with both per PR (since 
both are required for a +1).  I would be more comfortable with a follow-on task 
to make a proper travis matrix where both are run in travis per build.


> Build Failure When Using Profile HDP-2.5.0.0
> 
>
> Key: METRON-1518
> URL: https://issues.apache.org/jira/browse/METRON-1518
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> {code}
> /Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:45:
> error: cannot find symbol
> import org.apache.storm.StormTimer;
>^
>   symbol:   class StormTimer
>   location: package org.apache.storm
> /Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:167:
> error: cannot find symbol
>   private StormTimer expiredFlushTimer;
>   ^
>   symbol:   class StormTimer
>   location: class ProfileBuilderBolt
> /Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:450:
> error: cannot find symbol
>   private StormTimer createTimer(String name) {
>   ^
>   symbol:   class StormTimer
>   location: class ProfileBuilderBolt
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1518) Build Failure When Using Profile HDP-2.5.0.0

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432869#comment-16432869
 ] 

ASF GitHub Bot commented on METRON-1518:


Github user cestella commented on the issue:

https://github.com/apache/metron/pull/986
  
+1 by inspection, thanks!


> Build Failure When Using Profile HDP-2.5.0.0
> 
>
> Key: METRON-1518
> URL: https://issues.apache.org/jira/browse/METRON-1518
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> {code}
> /Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:45:
> error: cannot find symbol
> import org.apache.storm.StormTimer;
>^
>   symbol:   class StormTimer
>   location: package org.apache.storm
> /Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:167:
> error: cannot find symbol
>   private StormTimer expiredFlushTimer;
>   ^
>   symbol:   class StormTimer
>   location: class ProfileBuilderBolt
> /Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:450:
> error: cannot find symbol
>   private StormTimer createTimer(String name) {
>   ^
>   symbol:   class StormTimer
>   location: class ProfileBuilderBolt
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1518) Build Failure When Using Profile HDP-2.5.0.0

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432866#comment-16432866
 ] 

ASF GitHub Bot commented on METRON-1518:


Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/986#discussion_r180547851
  
--- Diff: .travis.yml ---
@@ -33,7 +33,7 @@ before_install:
   - npm config set prefix $HOME/.npm-prefix --global
 
 install:
-  - time mvn -q -T 2C -DskipTests clean install
+  - time mvn -q -T 2C -DskipTests -PHDP-2.5.0.0 clean install
--- End diff --

That works.


> Build Failure When Using Profile HDP-2.5.0.0
> 
>
> Key: METRON-1518
> URL: https://issues.apache.org/jira/browse/METRON-1518
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> {code}
> /Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:45:
> error: cannot find symbol
> import org.apache.storm.StormTimer;
>^
>   symbol:   class StormTimer
>   location: package org.apache.storm
> /Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:167:
> error: cannot find symbol
>   private StormTimer expiredFlushTimer;
>   ^
>   symbol:   class StormTimer
>   location: class ProfileBuilderBolt
> /Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:450:
> error: cannot find symbol
>   private StormTimer createTimer(String name) {
>   ^
>   symbol:   class StormTimer
>   location: class ProfileBuilderBolt
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #986: METRON-1518 Build Failure When Using Profile HDP-2.5.0.0

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/986
  
+1 by inspection, thanks!


---

[GitHub] metron pull request #986: METRON-1518 Build Failure When Using Profile HDP-2...

[nickwallen]

GitHub user nickwallen opened a pull request:

https://github.com/apache/metron/pull/986

METRON-1518 Build Failure When Using Profile HDP-2.5.0.0

The build fails when using the 'HDP-2.5.0.0' profile.
```

/Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:45:
error: cannot find symbol
import org.apache.storm.StormTimer;
   ^
  symbol:   class StormTimer
  location: package org.apache.storm

/Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:167:
error: cannot find symbol
  private StormTimer expiredFlushTimer;
  ^
  symbol:   class StormTimer
  location: class ProfileBuilderBolt

/Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:450:
error: cannot find symbol
  private StormTimer createTimer(String name) {
  ^
  symbol:   class StormTimer
  location: class ProfileBuilderBolt
```

This was caused by my PR; #977.  That code relied on a class that is only 
available in Storm 1.0.3+.  When using the Maven profile `HDP-2.5.0.0`, the 
code is built against Storm 1.0.1.

### Changes

* This completely removes the use of a`StormTimer`.  There is no need for 
it.  I had thought it would help me drive testing, but it has not ended up 
being useful at all.  

* An executor is used to drive a separate thread which flushes expired 
profiles on a regular basis.

* Updated the Travis CI builds so they are run with the HDP-2.5.0.0 
profile.  This ensures that the CI tests are run against the same version of 
Storm as the development environments are built against.  This will help ensure 
we catch similar issues in the future.

### Manual Testing

* Run a development environment and ensure that alerts are visible in the 
Alerts UI.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nickwallen/metron METRON-1518

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/986.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #986


commit 0c012fc8cfa6314dfc30c7d372970f1d68bfd908
Author: Nick Allen 
Date:   2018-04-10T12:46:47Z

METRON-1518 Build Failure When Using Profile HDP-2.5.0.0




---

[GitHub] metron issue #970: METRON-1421: Create a SolrMetaAlertDao

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/970
  
@justinleet @nickwallen I agree with the proposed plan for this PR's 
refactorings. That being said, I have some comments after having more or less 
caught up with the thread.

Generally speaking, it's not uncommon to provide an interface that 
encompasses all the desired methods for your data access layer. For instance:
- https://martinfowler.com/eaaCatalog/tableDataGateway.html
- https://martinfowler.com/eaaCatalog/dataMapper.html
- 
https://www.tutorialspoint.com/design_pattern/data_access_object_pattern.htm

I initially blinked at having the broad assortment of individual (CRUD) 
interfaces along with the aggregate ones, but I think that's fine. Not strictly 
necessary, but at least the intention is clear.

I think some of the trouble I/we're seeing with these interfaces and the 
inheritance is that it seems like there's some code smell around the 
abstractions we're using  wrt the meta alerts and index. "Index" and "meta 
alert" don't seem to be on the same plane of abstraction. Index is a data store 
whereas meta alert is a domain model object. I think it's probably time (as a 
follow on DISCUSS thread) to map out how we want our access patterns 
restructured to better accommodate the request for composition that @nickwallen 
pointed out along with addressing the model and DAO abstraction mismatch.


---

[GitHub] metron pull request #986: METRON-1518 Build Failure When Using Profile HDP-2...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/986#discussion_r180543990
  
--- Diff: .travis.yml ---
@@ -33,7 +33,7 @@ before_install:
   - npm config set prefix $HOME/.npm-prefix --global
 
 install:
-  - time mvn -q -T 2C -DskipTests clean install
+  - time mvn -q -T 2C -DskipTests -PHDP-2.5.0.0 clean install
--- End diff --

I believe the rationale for having travis run without the 2.5.0.0 profile 
and full-dev build with it is to ensure that we build with both per PR (since 
both are required for a +1).  I would be more comfortable with a follow-on task 
to make a proper travis matrix where both are run in travis per build.


---

[GitHub] metron issue #986: METRON-1518 Build Failure When Using Profile HDP-2.5.0.0

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/986
  
+1 ship it


---

[GitHub] metron issue #985: METRON-1515: Errors loading stellar functions currently b...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/985
  
Ok, I added a unit test for this as requested.


---

[jira] [Commented] (METRON-1515) Errors loading stellar functions currently bomb the entire topology, they should be recoverable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432763#comment-16432763
 ] 

ASF GitHub Bot commented on METRON-1515:


Github user cestella commented on the issue:

https://github.com/apache/metron/pull/985
  
Ok, I added a unit test for this as requested.


> Errors loading stellar functions currently bomb the entire topology, they 
> should be recoverable
> ---
>
> Key: METRON-1515
> URL: https://issues.apache.org/jira/browse/METRON-1515
> Project: Metron
>  Issue Type: Bug
>Reporter: Casey Stella
>Priority: Major
>
> If a bad stellar function gets on the classpath, the entire enrichment 
> topology (or shell) bombs.  We should just log an error and skip the function.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1518) Build Failure When Using Profile HDP-2.5.0.0

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432818#comment-16432818
 ] 

ASF GitHub Bot commented on METRON-1518:


GitHub user nickwallen opened a pull request:

https://github.com/apache/metron/pull/986

METRON-1518 Build Failure When Using Profile HDP-2.5.0.0

The build fails when using the 'HDP-2.5.0.0' profile.
```

/Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:45:
error: cannot find symbol
import org.apache.storm.StormTimer;
   ^
  symbol:   class StormTimer
  location: package org.apache.storm

/Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:167:
error: cannot find symbol
  private StormTimer expiredFlushTimer;
  ^
  symbol:   class StormTimer
  location: class ProfileBuilderBolt

/Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:450:
error: cannot find symbol
  private StormTimer createTimer(String name) {
  ^
  symbol:   class StormTimer
  location: class ProfileBuilderBolt
```

This was caused by my PR; #977.  That code relied on a class that is only 
available in Storm 1.0.3+.  When using the Maven profile `HDP-2.5.0.0`, the 
code is built against Storm 1.0.1.

### Changes

* This completely removes the use of a`StormTimer`.  There is no need for 
it.  I had thought it would help me drive testing, but it has not ended up 
being useful at all.  

* An executor is used to drive a separate thread which flushes expired 
profiles on a regular basis.

* Updated the Travis CI builds so they are run with the HDP-2.5.0.0 
profile.  This ensures that the CI tests are run against the same version of 
Storm as the development environments are built against.  This will help ensure 
we catch similar issues in the future.

### Manual Testing

* Run a development environment and ensure that alerts are visible in the 
Alerts UI.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nickwallen/metron METRON-1518

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/986.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #986


commit 0c012fc8cfa6314dfc30c7d372970f1d68bfd908
Author: Nick Allen 
Date:   2018-04-10T12:46:47Z

METRON-1518 Build Failure When Using Profile HDP-2.5.0.0




> Build Failure When Using Profile HDP-2.5.0.0
> 
>
> Key: METRON-1518
> URL: https://issues.apache.org/jira/browse/METRON-1518
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> {code}
> /Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:45:
> error: cannot find symbol
> import org.apache.storm.StormTimer;
>^
>   symbol:   class StormTimer
>   location: package org.apache.storm
> /Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:167:
> error: cannot find symbol
>   private StormTimer expiredFlushTimer;
>   ^
>   symbol:   class StormTimer
>   location: class ProfileBuilderBolt
> /Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:450:
> error: cannot find symbol
>   private StormTimer createTimer(String name) {
>   ^
>   symbol:   class StormTimer
>   location: class ProfileBuilderBolt
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1421) Create a SolrMetaAlertDao

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432821#comment-16432821
 ] 

ASF GitHub Bot commented on METRON-1421:


Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/970
  
@justinleet @nickwallen I agree with the proposed plan for this PR's 
refactorings. That being said, I have some comments after having more or less 
caught up with the thread.

Generally speaking, it's not uncommon to provide an interface that 
encompasses all the desired methods for your data access layer. For instance:
- https://martinfowler.com/eaaCatalog/tableDataGateway.html
- https://martinfowler.com/eaaCatalog/dataMapper.html
- 
https://www.tutorialspoint.com/design_pattern/data_access_object_pattern.htm

I initially blinked at having the broad assortment of individual (CRUD) 
interfaces along with the aggregate ones, but I think that's fine. Not strictly 
necessary, but at least the intention is clear.

I think some of the trouble I/we're seeing with these interfaces and the 
inheritance is that it seems like there's some code smell around the 
abstractions we're using  wrt the meta alerts and index. "Index" and "meta 
alert" don't seem to be on the same plane of abstraction. Index is a data store 
whereas meta alert is a domain model object. I think it's probably time (as a 
follow on DISCUSS thread) to map out how we want our access patterns 
restructured to better accommodate the request for composition that @nickwallen 
pointed out along with addressing the model and DAO abstraction mismatch.


> Create a SolrMetaAlertDao
> -
>
> Key: METRON-1421
> URL: https://issues.apache.org/jira/browse/METRON-1421
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Justin Leet
>Assignee: Justin Leet
>Priority: Major
>
> Create an implementation of the MetaAlertDao for Solr. This will involve 
> implementing the various MetaAlertDao methods using the SolrJ library and 
> also providing a SolrMetaAlertIntegrationTest (similar to 
> ElasticsearchMetaAlertIntegrationTest).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #986: METRON-1518 Build Failure When Using Profile HDP-2...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/986#discussion_r180547851
  
--- Diff: .travis.yml ---
@@ -33,7 +33,7 @@ before_install:
   - npm config set prefix $HOME/.npm-prefix --global
 
 install:
-  - time mvn -q -T 2C -DskipTests clean install
+  - time mvn -q -T 2C -DskipTests -PHDP-2.5.0.0 clean install
--- End diff --

That works.


---

[jira] [Commented] (METRON-1518) Build Failure When Using Profile HDP-2.5.0.0

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432881#comment-16432881
 ] 

ASF GitHub Bot commented on METRON-1518:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/986
  
+1 ship it


> Build Failure When Using Profile HDP-2.5.0.0
> 
>
> Key: METRON-1518
> URL: https://issues.apache.org/jira/browse/METRON-1518
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> {code}
> /Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:45:
> error: cannot find symbol
> import org.apache.storm.StormTimer;
>^
>   symbol:   class StormTimer
>   location: package org.apache.storm
> /Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:167:
> error: cannot find symbol
>   private StormTimer expiredFlushTimer;
>   ^
>   symbol:   class StormTimer
>   location: class ProfileBuilderBolt
> /Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:450:
> error: cannot find symbol
>   private StormTimer createTimer(String name) {
>   ^
>   symbol:   class StormTimer
>   location: class ProfileBuilderBolt
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #970: METRON-1421: Create a SolrMetaAlertDao

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/970
  
@ottobackwards You're absolutely not confusing things.  It's important to 
have that perspective, and I am 100% appreciative of you hopping in and 
offering up your opinion.  I think you're right that we probably need to reach 
a solid compromise position rather than boiling the ocean.

@nickwallen What if, for this PR, we do the following:
1. Mostly keep the interface refactoring from the branch.
2. Instead of pushing everything directly into SolrUpdateMetaAlertDao and 
ElasticsearchUpdateMetaAlertDao, we push to AbstractLuceneUpdateMetaAlertDao 
minimally as needed to prevent duping (which is admittedly substantial)?  
Everything else pushes down to the specific DAO which extends the abstract 
DAOs. As in the test branch, the SolrMetaAlertDao just uses and passes along 
the appropriate impls as needed.
3. Repeat (2) for the other DAOs implementations
4. I take another pass at tests to make sure both the common and specific 
impls are properly tested.

At that point, it should somewhat clean up the type hierarchy which will 
enable us to make things more pluggable in the future (including potentially 
working to eliminate the aggregate marker interfaces). At the very least, I 
think it makes it more straightforward, although it's not perfect. This should 
improve maintainability and testing. The testing pass should make sure we're at 
least better positioned to be confident in future refactorings.

I 100% admit this doesn't completely address the composition vs. extension, 
but it at least makes progress relative towards that goal relative to where we 
were.  It has the benefit of being a doable fairly low-risk refactoring that 
enables the PR to make it into the branch.

Hopefully this has the elements of a good compromise: Nobody is overjoyed, 
but nobody feels cheated.  In addition, we at least have set ourselves up to 
make it easier for everyone to be happier in the future.


---

[jira] [Commented] (METRON-1515) Errors loading stellar functions currently bomb the entire topology, they should be recoverable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433076#comment-16433076
 ] 

ASF GitHub Bot commented on METRON-1515:


Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/985#discussion_r180583454
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/resolver/ClasspathFunctionResolver.java
 ---
@@ -254,18 +266,24 @@ public void initialize(Context context) {
 Set classes = new HashSet<>();
 Set ret = new HashSet<>();
 for(ClassLoader cl : cls) {
-  for(Class c : ClassIndex.getAnnotated(Stellar.class, cl)) {
-LOG.debug("{}: Found class: {}", cl.getClass().getCanonicalName(), 
c.getCanonicalName());
-boolean isAssignable = StellarFunction.class.isAssignableFrom(c);
-boolean isFiltered = filterBuilder.apply(c.getCanonicalName());
-if( isAssignable && isFiltered ) {
-  String className = c.getName();
-  if(!classes.contains(className)) {
-LOG.debug("{}: Added class: {}", 
cl.getClass().getCanonicalName(), className);
-ret.add((Class) c);
-classes.add(className);
+  for(Class c : getStellarClasses(cl)) {
+try {
+  LOG.debug("{}: Found class: {}", 
cl.getClass().getCanonicalName(), c.getCanonicalName());
+  if (includeClass(c, filterBuilder)) {
+String className = c.getName();
+if (!classes.contains(className)) {
+  LOG.debug("{}: Added class: {}", 
cl.getClass().getCanonicalName(), className);
+  ret.add((Class) c);
+  classes.add(className);
+}
   }
 }
+catch(Error le) {
+  //we have had some error loading a stellar function.  This could 
mean that
+  //the classpath is unstable (e.g. old copies of jars are on the 
classpath).
+  LOG.error("Skipping class: " + le.getMessage()
--- End diff --

Commons has a method that handles this situation well.  I tend to always 
use that in these situations;  `ClassUtils.getCanonicalName(Class cls, 
String valueIfNull)`.  If I'm not mistaken, it would make your logic simpler; 
only one error log statement.  


> Errors loading stellar functions currently bomb the entire topology, they 
> should be recoverable
> ---
>
> Key: METRON-1515
> URL: https://issues.apache.org/jira/browse/METRON-1515
> Project: Metron
>  Issue Type: Bug
>Reporter: Casey Stella
>Priority: Major
>
> If a bad stellar function gets on the classpath, the entire enrichment 
> topology (or shell) bombs.  We should just log an error and skip the function.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #985: METRON-1515: Errors loading stellar functions curr...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/985#discussion_r180583454
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/resolver/ClasspathFunctionResolver.java
 ---
@@ -254,18 +266,24 @@ public void initialize(Context context) {
 Set classes = new HashSet<>();
 Set ret = new HashSet<>();
 for(ClassLoader cl : cls) {
-  for(Class c : ClassIndex.getAnnotated(Stellar.class, cl)) {
-LOG.debug("{}: Found class: {}", cl.getClass().getCanonicalName(), 
c.getCanonicalName());
-boolean isAssignable = StellarFunction.class.isAssignableFrom(c);
-boolean isFiltered = filterBuilder.apply(c.getCanonicalName());
-if( isAssignable && isFiltered ) {
-  String className = c.getName();
-  if(!classes.contains(className)) {
-LOG.debug("{}: Added class: {}", 
cl.getClass().getCanonicalName(), className);
-ret.add((Class) c);
-classes.add(className);
+  for(Class c : getStellarClasses(cl)) {
+try {
+  LOG.debug("{}: Found class: {}", 
cl.getClass().getCanonicalName(), c.getCanonicalName());
+  if (includeClass(c, filterBuilder)) {
+String className = c.getName();
+if (!classes.contains(className)) {
+  LOG.debug("{}: Added class: {}", 
cl.getClass().getCanonicalName(), className);
+  ret.add((Class) c);
+  classes.add(className);
+}
   }
 }
+catch(Error le) {
+  //we have had some error loading a stellar function.  This could 
mean that
+  //the classpath is unstable (e.g. old copies of jars are on the 
classpath).
+  LOG.error("Skipping class: " + le.getMessage()
--- End diff --

Commons has a method that handles this situation well.  I tend to always 
use that in these situations;  `ClassUtils.getCanonicalName(Class cls, 
String valueIfNull)`.  If I'm not mistaken, it would make your logic simpler; 
only one error log statement.  


---

[GitHub] metron pull request #987: METRON-1519 Indexing Error Topic Property Not Disp...

[nickwallen]

GitHub user nickwallen opened a pull request:

https://github.com/apache/metron/pull/987

METRON-1519 Indexing Error Topic Property Not Displayed in MPack

The user is not able to configure the Indexing Error Topic in the MPack. A 
property has been defined for this in the Mpack, but it is not configured 
correctly to be displayed. The name of the property is "Indexing Error Topic".

### Testing

1. Spin-up the development environment.

2. Navigate to Metron > Configs > Indexing.

3. Change the value for "Indexing Error Topic".

4. Restart the Indexing topology.

5. Ensure that the new topic is in use by checking the topology's 
properties.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nickwallen/metron METRON-1519

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/987.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #987


commit 30a9800987033ac806d9ec6a9e809f6c06b35422
Author: Nick Allen 
Date:   2018-04-10T22:01:24Z

METRON-1519 Indexing Error Topic Property Not Displayed in MPack




---

[jira] [Commented] (METRON-1519) Indexing Error Topic Property Not Displayed in MPack

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433065#comment-16433065
 ] 

ASF GitHub Bot commented on METRON-1519:


GitHub user nickwallen opened a pull request:

https://github.com/apache/metron/pull/987

METRON-1519 Indexing Error Topic Property Not Displayed in MPack

The user is not able to configure the Indexing Error Topic in the MPack. A 
property has been defined for this in the Mpack, but it is not configured 
correctly to be displayed. The name of the property is "Indexing Error Topic".

### Testing

1. Spin-up the development environment.

2. Navigate to Metron > Configs > Indexing.

3. Change the value for "Indexing Error Topic".

4. Restart the Indexing topology.

5. Ensure that the new topic is in use by checking the topology's 
properties.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nickwallen/metron METRON-1519

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/987.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #987


commit 30a9800987033ac806d9ec6a9e809f6c06b35422
Author: Nick Allen 
Date:   2018-04-10T22:01:24Z

METRON-1519 Indexing Error Topic Property Not Displayed in MPack




> Indexing Error Topic Property Not Displayed in MPack
> 
>
> Key: METRON-1519
> URL: https://issues.apache.org/jira/browse/METRON-1519
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Critical
>
> The user is not able to configure the Indexing Error Topic in the MPack. A 
> property has been defined for this in the Mpack, but it is not configured 
> correctly to be displayed. The name of the property is "Indexing Error Topic".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1515) Errors loading stellar functions currently bomb the entire topology, they should be recoverable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432888#comment-16432888
 ] 

ASF GitHub Bot commented on METRON-1515:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/985
  
+1 by inspection.  Thanks for the contribution!


> Errors loading stellar functions currently bomb the entire topology, they 
> should be recoverable
> ---
>
> Key: METRON-1515
> URL: https://issues.apache.org/jira/browse/METRON-1515
> Project: Metron
>  Issue Type: Bug
>Reporter: Casey Stella
>Priority: Major
>
> If a bad stellar function gets on the classpath, the entire enrichment 
> topology (or shell) bombs.  We should just log an error and skip the function.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1519) Indexing Error Topic Property Not Displayed in MPack

[Nick Allen (JIRA)]

Nick Allen created METRON-1519:
--

 Summary: Indexing Error Topic Property Not Displayed in MPack
 Key: METRON-1519
 URL: https://issues.apache.org/jira/browse/METRON-1519
 Project: Metron
  Issue Type: Bug
Reporter: Nick Allen
Assignee: Nick Allen


The user is not able to configure the Indexing Error Topic in the MPack. A 
property has been defined for this in the Mpack, but it is not configured 
correctly to be displayed. The name of the property is "Indexing Error Topic".





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #985: METRON-1515: Errors loading stellar functions currently b...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/985
  
+1 by inspection.  Thanks for the contribution!


---

[GitHub] metron pull request #986: METRON-1518 Build Failure When Using Profile HDP-2...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/986


---

[jira] [Commented] (METRON-1518) Build Failure When Using Profile HDP-2.5.0.0

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432998#comment-16432998
 ] 

ASF GitHub Bot commented on METRON-1518:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/986
  
Thanks for the reviews.  Just to be sure, I ran up the CentOS dev 
environment, validated that alerts are being created, created a profile and 
made sure that values were written.


> Build Failure When Using Profile HDP-2.5.0.0
> 
>
> Key: METRON-1518
> URL: https://issues.apache.org/jira/browse/METRON-1518
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> {code}
> /Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:45:
> error: cannot find symbol
> import org.apache.storm.StormTimer;
>^
>   symbol:   class StormTimer
>   location: package org.apache.storm
> /Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:167:
> error: cannot find symbol
>   private StormTimer expiredFlushTimer;
>   ^
>   symbol:   class StormTimer
>   location: class ProfileBuilderBolt
> /Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:450:
> error: cannot find symbol
>   private StormTimer createTimer(String name) {
>   ^
>   symbol:   class StormTimer
>   location: class ProfileBuilderBolt
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #985: METRON-1515: Errors loading stellar functions curr...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/985#discussion_r180576221
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/resolver/ClasspathFunctionResolver.java
 ---
@@ -254,18 +266,24 @@ public void initialize(Context context) {
 Set classes = new HashSet<>();
 Set ret = new HashSet<>();
 for(ClassLoader cl : cls) {
-  for(Class c : ClassIndex.getAnnotated(Stellar.class, cl)) {
-LOG.debug("{}: Found class: {}", cl.getClass().getCanonicalName(), 
c.getCanonicalName());
-boolean isAssignable = StellarFunction.class.isAssignableFrom(c);
-boolean isFiltered = filterBuilder.apply(c.getCanonicalName());
-if( isAssignable && isFiltered ) {
-  String className = c.getName();
-  if(!classes.contains(className)) {
-LOG.debug("{}: Added class: {}", 
cl.getClass().getCanonicalName(), className);
-ret.add((Class) c);
-classes.add(className);
+  for(Class c : getStellarClasses(cl)) {
+try {
+  LOG.debug("{}: Found class: {}", 
cl.getClass().getCanonicalName(), c.getCanonicalName());
+  if (includeClass(c, filterBuilder)) {
+String className = c.getName();
+if (!classes.contains(className)) {
+  LOG.debug("{}: Added class: {}", 
cl.getClass().getCanonicalName(), className);
+  ret.add((Class) c);
+  classes.add(className);
+}
   }
 }
+catch(Error le) {
+  //we have had some error loading a stellar function.  This could 
mean that
+  //the classpath is unstable (e.g. old copies of jars are on the 
classpath).
+  LOG.error("Skipping class: " + le.getMessage()
--- End diff --

The exception message might not always contain the class name.  We should 
explicitly log the class name, like we do in the debug statement on line 271.


---

[GitHub] metron issue #986: METRON-1518 Build Failure When Using Profile HDP-2.5.0.0

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/986
  
Thanks for the reviews.  Just to be sure, I ran up the CentOS dev 
environment, validated that alerts are being created, created a profile and 
made sure that values were written.


---

[GitHub] metron pull request #985: METRON-1515: Errors loading stellar functions curr...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/985#discussion_r180578439
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/resolver/ClasspathFunctionResolver.java
 ---
@@ -254,18 +266,24 @@ public void initialize(Context context) {
 Set classes = new HashSet<>();
 Set ret = new HashSet<>();
 for(ClassLoader cl : cls) {
-  for(Class c : ClassIndex.getAnnotated(Stellar.class, cl)) {
-LOG.debug("{}: Found class: {}", cl.getClass().getCanonicalName(), 
c.getCanonicalName());
-boolean isAssignable = StellarFunction.class.isAssignableFrom(c);
-boolean isFiltered = filterBuilder.apply(c.getCanonicalName());
-if( isAssignable && isFiltered ) {
-  String className = c.getName();
-  if(!classes.contains(className)) {
-LOG.debug("{}: Added class: {}", 
cl.getClass().getCanonicalName(), className);
-ret.add((Class) c);
-classes.add(className);
+  for(Class c : getStellarClasses(cl)) {
+try {
+  LOG.debug("{}: Found class: {}", 
cl.getClass().getCanonicalName(), c.getCanonicalName());
+  if (includeClass(c, filterBuilder)) {
+String className = c.getName();
+if (!classes.contains(className)) {
+  LOG.debug("{}: Added class: {}", 
cl.getClass().getCanonicalName(), className);
+  ret.add((Class) c);
+  classes.add(className);
+}
   }
 }
+catch(Error le) {
+  //we have had some error loading a stellar function.  This could 
mean that
+  //the classpath is unstable (e.g. old copies of jars are on the 
classpath).
+  LOG.error("Skipping class: " + le.getMessage()
--- End diff --

On second thought, I think I might've squared the circle there in the 
latest commit.  How do you like what I came up with?


---

[jira] [Commented] (METRON-1518) Build Failure When Using Profile HDP-2.5.0.0

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433016#comment-16433016
 ] 

ASF GitHub Bot commented on METRON-1518:


Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/986


> Build Failure When Using Profile HDP-2.5.0.0
> 
>
> Key: METRON-1518
> URL: https://issues.apache.org/jira/browse/METRON-1518
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> {code}
> /Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:45:
> error: cannot find symbol
> import org.apache.storm.StormTimer;
>^
>   symbol:   class StormTimer
>   location: package org.apache.storm
> /Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:167:
> error: cannot find symbol
>   private StormTimer expiredFlushTimer;
>   ^
>   symbol:   class StormTimer
>   location: class ProfileBuilderBolt
> /Users/ottofowler/src/apache/forks/metron/metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileBuilderBolt.java:450:
> error: cannot find symbol
>   private StormTimer createTimer(String name) {
>   ^
>   symbol:   class StormTimer
>   location: class ProfileBuilderBolt
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1515) Errors loading stellar functions currently bomb the entire topology, they should be recoverable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433042#comment-16433042
 ] 

ASF GitHub Bot commented on METRON-1515:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/985#discussion_r180576511
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/resolver/ClasspathFunctionResolver.java
 ---
@@ -254,18 +266,24 @@ public void initialize(Context context) {
 Set classes = new HashSet<>();
 Set ret = new HashSet<>();
 for(ClassLoader cl : cls) {
-  for(Class c : ClassIndex.getAnnotated(Stellar.class, cl)) {
-LOG.debug("{}: Found class: {}", cl.getClass().getCanonicalName(), 
c.getCanonicalName());
-boolean isAssignable = StellarFunction.class.isAssignableFrom(c);
-boolean isFiltered = filterBuilder.apply(c.getCanonicalName());
-if( isAssignable && isFiltered ) {
-  String className = c.getName();
-  if(!classes.contains(className)) {
-LOG.debug("{}: Added class: {}", 
cl.getClass().getCanonicalName(), className);
-ret.add((Class) c);
-classes.add(className);
+  for(Class c : getStellarClasses(cl)) {
+try {
+  LOG.debug("{}: Found class: {}", 
cl.getClass().getCanonicalName(), c.getCanonicalName());
+  if (includeClass(c, filterBuilder)) {
+String className = c.getName();
+if (!classes.contains(className)) {
+  LOG.debug("{}: Added class: {}", 
cl.getClass().getCanonicalName(), className);
+  ret.add((Class) c);
+  classes.add(className);
+}
   }
 }
+catch(Error le) {
+  //we have had some error loading a stellar function.  This could 
mean that
+  //the classpath is unstable (e.g. old copies of jars are on the 
classpath).
+  LOG.error("Skipping class: " + le.getMessage()
--- End diff --

Yeah, this is tricky, getting the classname may actually cause the 
exception (this is precisely the scenario that motivated this, sadly).  That's 
why I forewent that approach.


> Errors loading stellar functions currently bomb the entire topology, they 
> should be recoverable
> ---
>
> Key: METRON-1515
> URL: https://issues.apache.org/jira/browse/METRON-1515
> Project: Metron
>  Issue Type: Bug
>Reporter: Casey Stella
>Priority: Major
>
> If a bad stellar function gets on the classpath, the entire enrichment 
> topology (or shell) bombs.  We should just log an error and skip the function.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1515) Errors loading stellar functions currently bomb the entire topology, they should be recoverable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433041#comment-16433041
 ] 

ASF GitHub Bot commented on METRON-1515:


Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/985#discussion_r180576221
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/resolver/ClasspathFunctionResolver.java
 ---
@@ -254,18 +266,24 @@ public void initialize(Context context) {
 Set classes = new HashSet<>();
 Set ret = new HashSet<>();
 for(ClassLoader cl : cls) {
-  for(Class c : ClassIndex.getAnnotated(Stellar.class, cl)) {
-LOG.debug("{}: Found class: {}", cl.getClass().getCanonicalName(), 
c.getCanonicalName());
-boolean isAssignable = StellarFunction.class.isAssignableFrom(c);
-boolean isFiltered = filterBuilder.apply(c.getCanonicalName());
-if( isAssignable && isFiltered ) {
-  String className = c.getName();
-  if(!classes.contains(className)) {
-LOG.debug("{}: Added class: {}", 
cl.getClass().getCanonicalName(), className);
-ret.add((Class) c);
-classes.add(className);
+  for(Class c : getStellarClasses(cl)) {
+try {
+  LOG.debug("{}: Found class: {}", 
cl.getClass().getCanonicalName(), c.getCanonicalName());
+  if (includeClass(c, filterBuilder)) {
+String className = c.getName();
+if (!classes.contains(className)) {
+  LOG.debug("{}: Added class: {}", 
cl.getClass().getCanonicalName(), className);
+  ret.add((Class) c);
+  classes.add(className);
+}
   }
 }
+catch(Error le) {
+  //we have had some error loading a stellar function.  This could 
mean that
+  //the classpath is unstable (e.g. old copies of jars are on the 
classpath).
+  LOG.error("Skipping class: " + le.getMessage()
--- End diff --

The exception message might not always contain the class name.  We should 
explicitly log the class name, like we do in the debug statement on line 271.


> Errors loading stellar functions currently bomb the entire topology, they 
> should be recoverable
> ---
>
> Key: METRON-1515
> URL: https://issues.apache.org/jira/browse/METRON-1515
> Project: Metron
>  Issue Type: Bug
>Reporter: Casey Stella
>Priority: Major
>
> If a bad stellar function gets on the classpath, the entire enrichment 
> topology (or shell) bombs.  We should just log an error and skip the function.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)