[jira] [Commented] (METRON-1480) Add yarn as default build tool for the frontend

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389885#comment-16389885
 ] 

ASF GitHub Bot commented on METRON-1480:


Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/952
  
I'm having trouble understanding what this PR actually does.  I see that 
package-lock.json was removed and yarn.lock was added.  Is that all that's 
needed to switch to yarn?  Do we need to also update our build commands to 
match those [here](https://yarnpkg.com/en/docs/usage)?

What considerations are there for development?  Are we supposed to be using 
yarn commands now instead of npm commands?  Do we need to install yarn locally? 
 We probably need to review our documentation and rpm build processes too.  We 
use npm all over the place.


> Add yarn as default build tool for the frontend
> ---
>
> Key: METRON-1480
> URL: https://issues.apache.org/jira/browse/METRON-1480
> Project: Metron
>  Issue Type: Improvement
>Reporter: Daniel Toth
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1471) Migrate shuffle connections to local or shuffle

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390145#comment-16390145
 ] 

ASF GitHub Bot commented on METRON-1471:


Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/949
  
There are also shuffle groupings in the enrichment and profiler topologies. 
 Do we want to update those too?


> Migrate shuffle connections to local or shuffle
> ---
>
> Key: METRON-1471
> URL: https://issues.apache.org/jira/browse/METRON-1471
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>Priority: Major
>
> Currently, we use shuffle groupings when we do not want to group by field.  
> We should, instead, use local or shuffle groupings.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1480) Add yarn as default build tool for the frontend

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389913#comment-16389913
 ] 

ASF GitHub Bot commented on METRON-1480:


Github user xyztdanid4 commented on a diff in the pull request:

https://github.com/apache/metron/pull/952#discussion_r172936489
  
--- Diff: metron-interface/metron-config/package.json ---
@@ -33,6 +28,11 @@
 "@angular/platform-browser": "2.0.0",
 "@angular/platform-browser-dynamic": "2.0.0",
 "@angular/router": "3.0.0",
+"@types/ace": "0.0.32",
--- End diff --

Yes. I was not enough carefully. Yarn did the rearrangement, but I can move 
the dependencies back to their original place if you would like to.


> Add yarn as default build tool for the frontend
> ---
>
> Key: METRON-1480
> URL: https://issues.apache.org/jira/browse/METRON-1480
> Project: Metron
>  Issue Type: Improvement
>Reporter: Daniel Toth
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1480) Add yarn as default build tool for the frontend

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389920#comment-16389920
 ] 

ASF GitHub Bot commented on METRON-1480:


Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/952#discussion_r172937150
  
--- Diff: metron-interface/metron-config/package.json ---
@@ -33,6 +28,11 @@
 "@angular/platform-browser": "2.0.0",
 "@angular/platform-browser-dynamic": "2.0.0",
 "@angular/router": "3.0.0",
+"@types/ace": "0.0.32",
--- End diff --

No I was just curious.  If Yarn did this automatically we might as well 
leave it that way.  No need to revert.


> Add yarn as default build tool for the frontend
> ---
>
> Key: METRON-1480
> URL: https://issues.apache.org/jira/browse/METRON-1480
> Project: Metron
>  Issue Type: Improvement
>Reporter: Daniel Toth
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1480) Add yarn as default build tool for the frontend

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389902#comment-16389902
 ] 

ASF GitHub Bot commented on METRON-1480:


Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/952#discussion_r172934046
  
--- Diff: metron-interface/metron-config/package.json ---
@@ -33,6 +28,11 @@
 "@angular/platform-browser": "2.0.0",
 "@angular/platform-browser-dynamic": "2.0.0",
 "@angular/router": "3.0.0",
+"@types/ace": "0.0.32",
--- End diff --

What was the reason for rearranging the dependencies?  It's not a big deal 
but it's distracting and makes reviewing harder.


> Add yarn as default build tool for the frontend
> ---
>
> Key: METRON-1480
> URL: https://issues.apache.org/jira/browse/METRON-1480
> Project: Metron
>  Issue Type: Improvement
>Reporter: Daniel Toth
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1480) Add yarn as default build tool for the frontend

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389912#comment-16389912
 ] 

ASF GitHub Bot commented on METRON-1480:


Github user xyztdanid4 commented on the issue:

https://github.com/apache/metron/pull/952
  
@merrimanr 
Yes this is the right way to switch to yarn.
Yes we have install yarn locally.
I did not know that we have so much dependency on npm. But for me it was 
hard to add new dependencies using npm, cause it was constantly failing. And 
yes I would recommend to use yarn, at least for the front end builds. What yarn 
does is that reduces the possibility of version mismatch  of the dependencies 
by importing the dependencies subdependency in the right version. So yes we 
have to use yarn commands if we merge this PR.


> Add yarn as default build tool for the frontend
> ---
>
> Key: METRON-1480
> URL: https://issues.apache.org/jira/browse/METRON-1480
> Project: Metron
>  Issue Type: Improvement
>Reporter: Daniel Toth
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1480) Add yarn as default build tool for the frontend

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389927#comment-16389927
 ] 

ASF GitHub Bot commented on METRON-1480:


Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/952
  
I think we are on board with switching to yarn so no concerns there.  There 
was a discussion on it but if anyone now feels differently, speak up.

Sounds like we still have a lot of work to do.  I would suggest reviewing 
the metron-config and metron-alerts READMEs and update them appropriately.  In 
fact, I would just search the whole project for npm commands and switch to yarn 
as needed.  You will definitely need to run this up in full dev and verify 
everything works.

Our maven front end plugin will also need to be reviewed and updated:  
https://github.com/eirslett/frontend-maven-plugin.  We use that in 
metron-config and metron-alerts.


> Add yarn as default build tool for the frontend
> ---
>
> Key: METRON-1480
> URL: https://issues.apache.org/jira/browse/METRON-1480
> Project: Metron
>  Issue Type: Improvement
>Reporter: Daniel Toth
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1480) Add yarn as default build tool for the frontend

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389897#comment-16389897
 ] 

ASF GitHub Bot commented on METRON-1480:


Github user xyztdanid4 commented on the issue:

https://github.com/apache/metron/pull/952
  
@merrimanr


> Add yarn as default build tool for the frontend
> ---
>
> Key: METRON-1480
> URL: https://issues.apache.org/jira/browse/METRON-1480
> Project: Metron
>  Issue Type: Improvement
>Reporter: Daniel Toth
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1480) Add yarn as default build tool for the frontend

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389896#comment-16389896
 ] 

ASF GitHub Bot commented on METRON-1480:


Github user xyztdanid4 commented on a diff in the pull request:

https://github.com/apache/metron/pull/952#discussion_r172932830
  
--- Diff: metron-interface/metron-config/package.json ---
@@ -33,6 +28,11 @@
 "@angular/platform-browser": "2.0.0",
 "@angular/platform-browser-dynamic": "2.0.0",
 "@angular/router": "3.0.0",
+"@types/ace": "0.0.32",
--- End diff --

No there were no changes other than rearrange of the dependencies.
About the main attribute, in our case it will do nothing. It defines the 
entry point of the application if we would SystemJS or requireJS. It is the 
default setting for yarn.


> Add yarn as default build tool for the frontend
> ---
>
> Key: METRON-1480
> URL: https://issues.apache.org/jira/browse/METRON-1480
> Project: Metron
>  Issue Type: Improvement
>Reporter: Daniel Toth
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1464) Convert schemas to be compatible with Solr 5.5.2

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389919#comment-16389919
 ] 

ASF GitHub Bot commented on METRON-1464:


Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/945
  
When looking at the reference guide, it's not obvious to me what the 
differences are between Point and Trie fields as it relates to our use case.  
Point field types for the corresponding Trie field types don't exist in 5.5 
anyways so I suspect it's a moot point if supporting 5.5 is desirable.

Yes this has been tested with the HDP Search MPack (Solr 5.5.2) and the 
manually installed Solr 6.6.2.


> Convert schemas to be compatible with Solr 5.5.2
> 
>
> Key: METRON-1464
> URL: https://issues.apache.org/jira/browse/METRON-1464
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
>
> It would be ideal if our Solr schemas were compatible with Solr 5.5.2 or 
> 6.6.2.  This would be users the option of using a manually installed Solr or 
> HDP Search.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1480) Add yarn as default build tool for the frontend

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389917#comment-16389917
 ] 

ASF GitHub Bot commented on METRON-1480:


Github user xyztdanid4 commented on the issue:

https://github.com/apache/metron/pull/952
  
@merrimanr 
although it should work with npm as well, cause then it will use the 
package.json to install the dependencies.


> Add yarn as default build tool for the frontend
> ---
>
> Key: METRON-1480
> URL: https://issues.apache.org/jira/browse/METRON-1480
> Project: Metron
>  Issue Type: Improvement
>Reporter: Daniel Toth
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1474) Add normalizecss

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390294#comment-16390294
 ] 

ASF GitHub Bot commented on METRON-1474:


Github user xyztdanid4 closed the pull request at:

https://github.com/apache/metron/pull/955


> Add normalizecss
> 
>
> Key: METRON-1474
> URL: https://issues.apache.org/jira/browse/METRON-1474
> Project: Metron
>  Issue Type: Improvement
>Reporter: Daniel Toth
>Priority: Minor
>
> Add normalizecss to reduce css mistakes



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1482) Update REST to work with Solr

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390400#comment-16390400
 ] 

ASF GitHub Bot commented on METRON-1482:


GitHub user merrimanr opened a pull request:

https://github.com/apache/metron/pull/957

METRON-1482: Update REST to work with Solr

## Contributor Comments
This PR updates the REST start script to properly include the correct 
indexing jar on the classpath depending on the configured indexing writer 
(Elasticsearch or Solr).  To test this:

1. Spin up full dev and follow the instructions for manually installing 
Solr:  
https://github.com/apache/metron/tree/feature/METRON-1416-upgrade-solr/metron-platform/metron-solr#installing.
 
2. Change the Index Writer to "Solr" in Ambari (Metron > Indexing tab) and 
restart the Indexing component.
3. Verify data is landing in the Solr collections
4. Restart the REST component in Ambari
5. You should be able to execute search queries in Swagger

I had to include a stubbed SolrMetaAlertDao class that doesn't do anything 
to get everything to start up.  This will eventually be replaced by the work 
being done in https://issues.apache.org/jira/browse/METRON-1421.  I also fixed 
a bug in the SolrDao that was causing in issue in the Alerts UI.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/merrimanr/incubator-metron solr-rest

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/957.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #957


commit 6bb30af9d2005414e3ee44c0bdb0ea14540ce13c
Author: cstella 
Date:   2018-02-01T21:33:56Z

METRON-1441: Create complementary Solr schemas for the main sensors

commit f4ff0c401eff23d9c1b2ca3b264bd9b0d4e8f381
Author: cstella 
Date:   2018-02-01T21:47:12Z

Updating dao

commit 7e2ecb0f2f55ea16529128fec14920bc2a546b07
Author: cstella 
Date:   2018-02-02T21:43:38Z

Migrated data 

[jira] [Created] (METRON-1482) Update REST to work with Solr

[Ryan Merriman (JIRA)]

Ryan Merriman created METRON-1482:
-

 Summary: Update REST to work with Solr
 Key: METRON-1482
 URL: https://issues.apache.org/jira/browse/METRON-1482
 Project: Metron
  Issue Type: Sub-task
Reporter: Ryan Merriman
Assignee: Ryan Merriman


Ambari should start REST with the correct Solr indexing jar on the classpath.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1471) Migrate shuffle connections to local or shuffle

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390278#comment-16390278
 ] 

ASF GitHub Bot commented on METRON-1471:


Github user cestella commented on the issue:

https://github.com/apache/metron/pull/949
  
@merrimanr As of now, there should be no more SHUFFLE groupings in 
enrichments, but I'll change the profiler for sure.


> Migrate shuffle connections to local or shuffle
> ---
>
> Key: METRON-1471
> URL: https://issues.apache.org/jira/browse/METRON-1471
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>Priority: Major
>
> Currently, we use shuffle groupings when we do not want to group by field.  
> We should, instead, use local or shuffle groupings.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1474) Add normalizecss

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390312#comment-16390312
 ] 

ASF GitHub Bot commented on METRON-1474:


GitHub user xyztdanid4 opened a pull request:

https://github.com/apache/metron/pull/956

METRON-1474 Add normalizecss

## Contributor Comments
This PR adds normalizecss to the frontend projects. It is useful to bridge 
the differences between browsers.


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [X] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at 
[https://issues.apache.org/jira/browse/METRON-1474](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [X] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [X] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [X] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xyztdanid4/metron METRON-1474

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/956.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #956


commit 61e342b89d8ba7554d3679556969c09a37f575c8
Author: Daniel Toth 
Date:   2018-03-07T21:53:14Z

Add normalize.css to frontend projects




> Add normalizecss
> 
>
> Key: METRON-1474
> URL: https://issues.apache.org/jira/browse/METRON-1474
> Project: Metron
>  Issue Type: Improvement
>Reporter: Daniel Toth
>Priority: Minor
>
> Add normalizecss to reduce css mistakes



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1480) Add yarn as default build tool for the frontend

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389878#comment-16389878
 ] 

ASF GitHub Bot commented on METRON-1480:


Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/952#discussion_r172927486
  
--- Diff: metron-interface/metron-config/package.json ---
@@ -33,6 +28,11 @@
 "@angular/platform-browser": "2.0.0",
 "@angular/platform-browser-dynamic": "2.0.0",
 "@angular/router": "3.0.0",
+"@types/ace": "0.0.32",
--- End diff --

Were there any real changes made to this file?  It looks like some 
dependencies were simply rearranged.  I do see the addition of `"main": 
"index.js"`.  What is the purpose of that?


> Add yarn as default build tool for the frontend
> ---
>
> Key: METRON-1480
> URL: https://issues.apache.org/jira/browse/METRON-1480
> Project: Metron
>  Issue Type: Improvement
>Reporter: Daniel Toth
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1480) Add yarn as default build tool for the frontend

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389879#comment-16389879
 ] 

ASF GitHub Bot commented on METRON-1480:


Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/952#discussion_r172927751
  
--- Diff: metron-interface/metron-alerts/package.json ---
@@ -27,8 +27,8 @@
 "core-js": "^2.4.1",
 "font-awesome": "^4.7.0",
 "moment": "^2.18.1",
-"pikaday-time": "^1.6.1",
 "ng2-dragula": "^1.5.0",
+"pikaday-time": "^1.6.1",
--- End diff --

See comment for metron-config/package.json.


> Add yarn as default build tool for the frontend
> ---
>
> Key: METRON-1480
> URL: https://issues.apache.org/jira/browse/METRON-1480
> Project: Metron
>  Issue Type: Improvement
>Reporter: Daniel Toth
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1480) Add yarn as default build tool for the frontend

[Daniel Toth (JIRA)]

Daniel Toth created METRON-1480:
---

 Summary: Add yarn as default build tool for the frontend
 Key: METRON-1480
 URL: https://issues.apache.org/jira/browse/METRON-1480
 Project: Metron
  Issue Type: Improvement
Reporter: Daniel Toth






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1479) Add editorconfig - create guideline for code formatting in the angular part of the app

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389363#comment-16389363
 ] 

ASF GitHub Bot commented on METRON-1479:


GitHub user xyztdanid4 opened a pull request:

https://github.com/apache/metron/pull/951

Metron-1479 Introduce editorconfig, and format all of the code in alerts 
and config projects

It is necessary to use editorconfig, code should be look like the same in 
all files.

### For all changes:
- [X] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at 
[https://issues.apache.org/jira/browse/METRON-1479](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [X] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [X] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [X] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [X] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xyztdanid4/metron METRON-1479

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/951.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #951


commit f9c2a969d359e30cb55b5012757e22c32ce57e78
Author: Daniel Toth 
Date:   2018-03-07T08:34:13Z

Update gitignore

commit b0cfda918e574992c94e0c890b66ce83af772837
Author: Daniel Toth 
Date:   2018-03-07T08:35:03Z

Integrate editorconfig

commit 588ef781295de1b141cef7d4ca12a03a8606f2c5
Author: Daniel Toth 
Date:   2018-03-07T08:59:17Z

Format code of metron alerts

commit e36877357e14bf12ae8ae38a5491dbb6d219f32d
Author: Daniel Toth 
Date:   2018-03-07T10:15:59Z

Format metron config project




> Add editorconfig - create guideline for code formatting in the angular part 
> of the app
> --
>
> Key: METRON-1479
> URL: https://issues.apache.org/jira/browse/METRON-1479
> Project: Metron
>  Issue Type: Improvement
>Reporter: Daniel Toth
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1481) Add package.json to metron-alerts

[Daniel Toth (JIRA)]

Daniel Toth created METRON-1481:
---

 Summary: Add package.json to metron-alerts
 Key: METRON-1481
 URL: https://issues.apache.org/jira/browse/METRON-1481
 Project: Metron
  Issue Type: Improvement
Reporter: Daniel Toth






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1474) Add normalizecss

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389563#comment-16389563
 ] 

ASF GitHub Bot commented on METRON-1474:


GitHub user xyztdanid4 opened a pull request:

https://github.com/apache/metron/pull/955

METRON-1474 Add normalizecss

THIS IS WIP. FIRST #952 SHOULD BE MERGED.
## Contributor Comments
This PR adds normalizecss to the frontend projects. It is useful to bridge 
the differences between browsers.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at 
[https://issues.apache.org/jira/browse/METRON-1474](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [X] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [X] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [X] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [X] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [X] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xyztdanid4/metron METRON-1474

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/955.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #955


commit 0812c6eca0bc009ddd0658f7976f5734daedb859
Author: Daniel Toth 
Date:   2018-03-07T10:40:01Z

Add yarn as build tool

commit 9117dac298a42a0b434ab933354a54542724b6bc
Author: Daniel Toth 
Date:   2018-03-07T11:47:50Z

Update ts to avoid 'app module is not an ng module error on build'

commit c4bef1662835051db430d657b479691f137d34ec
Author: Daniel Toth 
Date:   2018-03-07T13:44:29Z

Add normalize css to the frontend




> Add normalizecss
> 
>
> Key: METRON-1474
> URL: https://issues.apache.org/jira/browse/METRON-1474
> Project: Metron
>  Issue Type: Improvement
>Reporter: Daniel Toth
>Priority: Minor
>
> Add normalizecss to reduce css mistakes



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1460) Create a complementary non-split-join enrichment topology

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389610#comment-16389610
 ] 

ASF GitHub Bot commented on METRON-1460:


Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/940


> Create a complementary non-split-join enrichment topology
> -
>
> Key: METRON-1460
> URL: https://issues.apache.org/jira/browse/METRON-1460
> Project: Metron
>  Issue Type: New Feature
>Reporter: Casey Stella
>Priority: Major
>
> There are some deficiencies to the split/join topology.
>  * It's hard to reason about
>  * Understanding the latency of enriching a message requires looking at 
> multiple bolts that each give summary statistics
>  * The join bolt's cache is really hard to reason about when performance 
> tuning
>  * During spikes in traffic, you can overload the join bolt's cache and drop 
> messages if you aren't careful
>  * In general, it's hard to associate a cache size and a duration kept in 
> cache with throughput and latency
>  * There are a lot of network hops per message
>  * Right now we are stuck at 2 stages of transformations being done 
> (enrichment and threat intel).  It's very possible that you might want 
> stellar enrichments to depend on the output of other stellar enrichments.  In 
> order to implement this in split/join you'd have to create a cycle in the 
> storm topology
>  
> I propose that we move to a model where we do enrichments in a single bolt in 
> parallel using a static threadpool (e.g. multiple workers in the same process 
> would share the threadpool).  IN all other ways, this would be backwards 
> compatible.  A transparent drop-in for the existing enrichment topology.
> There are some pros/cons about this too:
>  * Pro
>  * Easier to reason about from an individual message perspective
>  * Architecturally decoupled from Storm
>  * This sets us up if we want to consider other streaming technologies
>  * Fewer bolts
>  * spout -> enrichment bolt -> threatintel bolt -> output bolt
>  * Way fewer network hops per message
>  * currently 2n+1 where n is the number of enrichments used (if using stellar 
> subgroups, each subgroup is a hop)
>  * Easier to reason about from a performance perspective
>  * We trade cache size and eviction timeout for threadpool size
>  * We set ourselves up to have stellar subgroups with dependencies
>  * i.e. stellar subgroups that depend on the output of other subgroups
>  * If we do this, we can shrink the topology to just spout -> 
> enrichment/threat intel -> output
>  * Con
>  * We can no longer tune stellar enrichments independent from HBase 
> enrichments
>  * To be fair, with enrichments moving to stellar, this is the case in the 
> split/join approach too
>  * No idea about performance
> What I propose is to submit a PR that will deliver an alternative, completely 
> backwards compatible topology for enrichment that you can use by adjusting 
> the start_enrichment_topology.sh script to use remote-unified.yaml instead of 
> remote.yaml.  If we live with it for a while and have some good experiences 
> with it, maybe we can consider retiring the old enrichment topology.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1463) Adjust the groupings and shuffles in enrichment to be more efficient

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389613#comment-16389613
 ] 

ASF GitHub Bot commented on METRON-1463:


Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/944


> Adjust the groupings and shuffles in enrichment to be more efficient
> 
>
> Key: METRON-1463
> URL: https://issues.apache.org/jira/browse/METRON-1463
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>Priority: Major
>
> Currently there are some deficiencies in our grouping approach in the 
> enrichment topology:
>  * We have field grouping by key in places where we need LOCAL_OR_SHUFFLE 
> groupings
>  * We have shuffle groupings in places where we need LOCAL_OR_SHUFFLE 
> groupings
>  * We have field groupings by key in places where we need field grouping by 
> message (specifically in the connections from the splitter to the 
> enrichments).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (METRON-1473) Add and fix tslint

[Daniel Toth (JIRA)]

 [ 
https://issues.apache.org/jira/browse/METRON-1473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Toth updated METRON-1473:

Summary: Add and fix tslint  (was: Add tslint support)

> Add and fix tslint
> --
>
> Key: METRON-1473
> URL: https://issues.apache.org/jira/browse/METRON-1473
> Project: Metron
>  Issue Type: Improvement
>Reporter: Daniel Toth
>Priority: Minor
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> Add tslint support for better code quality



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1474) Add normalizecss

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389592#comment-16389592
 ] 

ASF GitHub Bot commented on METRON-1474:


Github user simonellistonball commented on the issue:

https://github.com/apache/metron/pull/955
  
Have the dependencies, License and Notices files been updated?


> Add normalizecss
> 
>
> Key: METRON-1474
> URL: https://issues.apache.org/jira/browse/METRON-1474
> Project: Metron
>  Issue Type: Improvement
>Reporter: Daniel Toth
>Priority: Minor
>
> Add normalizecss to reduce css mistakes



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1480) Add yarn as default build tool for the frontend

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389468#comment-16389468
 ] 

ASF GitHub Bot commented on METRON-1480:


GitHub user xyztdanid4 opened a pull request:

https://github.com/apache/metron/pull/952

Metron-1480 Add yarn as default build tool for the frontend

Updated the build tool to yarn for the frontend project to avoid npm 
instability during dependency adding.

### For all changes:
- [X] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at 
[https://issues.apache.org/jira/browse/METRON-1480](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [X] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [X] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [X] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [X] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xyztdanid4/metron METRON-1480

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/952.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #952


commit 0812c6eca0bc009ddd0658f7976f5734daedb859
Author: Daniel Toth 
Date:   2018-03-07T10:40:01Z

Add yarn as build tool

commit 9117dac298a42a0b434ab933354a54542724b6bc
Author: Daniel Toth 
Date:   2018-03-07T11:47:50Z

Update ts to avoid 'app module is not an ng module error on build'




> Add yarn as default build tool for the frontend
> ---
>
> Key: METRON-1480
> URL: https://issues.apache.org/jira/browse/METRON-1480
> Project: Metron
>  Issue Type: Improvement
>Reporter: Daniel Toth
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1472) Add stylelint support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389497#comment-16389497
 ] 

ASF GitHub Bot commented on METRON-1472:


GitHub user xyztdanid4 opened a pull request:

https://github.com/apache/metron/pull/953

Metron 1472 Add stylelint support

THIS IS WIP. FIRST #952 SHOULD MERGED.

Added support for stylelint to be able to validate scss syntax.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [X] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at 
[https://issues.apache.org/jira/browse/METRON-1472](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [X] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [X] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [X] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [X] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xyztdanid4/metron METRON-1472

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/953.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #953


commit 0812c6eca0bc009ddd0658f7976f5734daedb859
Author: Daniel Toth 
Date:   2018-03-07T10:40:01Z

Add yarn as build tool

commit 9117dac298a42a0b434ab933354a54542724b6bc
Author: Daniel Toth 
Date:   2018-03-07T11:47:50Z

Update ts to avoid 'app module is not an ng module error on build'

commit f5a8521552db0a3f0b5e7c11ff26c42241edf5d4
Author: Daniel Toth 
Date:   2018-03-07T12:30:37Z

Add stlyelint support




> Add stylelint support
> -
>
> Key: METRON-1472
> URL: https://issues.apache.org/jira/browse/METRON-1472
> Project: Metron
>  Issue Type: Improvement
>Reporter: Daniel Toth
>Priority: Minor
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> Add stylelint support for better code quality



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1464) Convert schemas to be compatible with Solr 5.5.2

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389678#comment-16389678
 ] 

ASF GitHub Bot commented on METRON-1464:


Github user simonellistonball commented on the issue:

https://github.com/apache/metron/pull/945
  
To be fair, my question is probably just as appropriate on a discuss thread 
and a separate ticket out of said thread if it comes to it.


> Convert schemas to be compatible with Solr 5.5.2
> 
>
> Key: METRON-1464
> URL: https://issues.apache.org/jira/browse/METRON-1464
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
>
> It would be ideal if our Solr schemas were compatible with Solr 5.5.2 or 
> 6.6.2.  This would be users the option of using a manually installed Solr or 
> HDP Search.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1480) Add yarn as default build tool for the frontend

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389690#comment-16389690
 ] 

ASF GitHub Bot commented on METRON-1480:


Github user xyztdanid4 commented on a diff in the pull request:

https://github.com/apache/metron/pull/952#discussion_r172879403
  
--- Diff: metron-interface/metron-alerts/package-lock.json ---
@@ -1,6427 +0,0 @@
-{
--- End diff --

I think that was the problem!
Thanks. @cestella 


> Add yarn as default build tool for the frontend
> ---
>
> Key: METRON-1480
> URL: https://issues.apache.org/jira/browse/METRON-1480
> Project: Metron
>  Issue Type: Improvement
>Reporter: Daniel Toth
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1464) Convert schemas to be compatible with Solr 5.5.2

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389647#comment-16389647
 ] 

ASF GitHub Bot commented on METRON-1464:


Github user cestella commented on the issue:

https://github.com/apache/metron/pull/945
  
@merrimanr you ran this up in Solr 5.5 as well as 6.6, right?  If so, then 
I'm content with the change and give a +1 pending (other than holding for an 
answer to Simon's question, which I was wondering as well).



> Convert schemas to be compatible with Solr 5.5.2
> 
>
> Key: METRON-1464
> URL: https://issues.apache.org/jira/browse/METRON-1464
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
>
> It would be ideal if our Solr schemas were compatible with Solr 5.5.2 or 
> 6.6.2.  This would be users the option of using a manually installed Solr or 
> HDP Search.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1467) Replace guava caches in places where the keyspace might be large

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389653#comment-16389653
 ] 

ASF GitHub Bot commented on METRON-1467:


Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/947#discussion_r172866140
  
--- Diff: 
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/JoinBolt.java
 ---
@@ -89,29 +91,25 @@ public void prepare(Map map, TopologyContext 
topologyContext, OutputCollector ou
 if (this.maxTimeRetain == null) {
   throw new IllegalStateException("maxTimeRetain must be specified");
 }
-loader = new CacheLoader>() {
-  @Override
-  public Map load(String key) throws Exception {
-return new HashMap<>();
-  }
-};
-cache = CacheBuilder.newBuilder().maximumSize(maxCacheSize)
-.expireAfterWrite(maxTimeRetain, 
TimeUnit.MINUTES).removalListener(new JoinRemoveListener())
-.build(loader);
+loader = s -> new HashMap<>();
+cache = Caffeine.newBuilder().maximumSize(maxCacheSize)
+ .expireAfterWrite(maxTimeRetain, TimeUnit.MINUTES)
+ .removalListener(new JoinRemoveListener())
--- End diff --

It seems like we only want notified of a full cache when ERROR logging is 
set. Is that the case? In the `JoinRemoveListener` we end up doing some work 
that we probably don't need to do unless ERROR logging is set.  One easy fix 
would be to only add the "remove listener" if `LOG.isDebugEnabled()`.


> Replace guava caches in places where the keyspace might be large
> 
>
> Key: METRON-1467
> URL: https://issues.apache.org/jira/browse/METRON-1467
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>Priority: Major
>
> Based on the performance tuning exercise as part of METRON-1460, guava has 
> difficulties with cache sizes over 10k.  We, unfortunately, are quite 
> demanding of guava in this regard so we should transition a few uses of guava 
> to Caffeine:
>  * Stellar processor cache
>  * The JoinBolt cache
>  * The Enrichment Bolt Cache
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1467) Replace guava caches in places where the keyspace might be large

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389665#comment-16389665
 ] 

ASF GitHub Bot commented on METRON-1467:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/947#discussion_r172870992
  
--- Diff: 
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/JoinBolt.java
 ---
@@ -89,29 +91,25 @@ public void prepare(Map map, TopologyContext 
topologyContext, OutputCollector ou
 if (this.maxTimeRetain == null) {
   throw new IllegalStateException("maxTimeRetain must be specified");
 }
-loader = new CacheLoader>() {
-  @Override
-  public Map load(String key) throws Exception {
-return new HashMap<>();
-  }
-};
-cache = CacheBuilder.newBuilder().maximumSize(maxCacheSize)
-.expireAfterWrite(maxTimeRetain, 
TimeUnit.MINUTES).removalListener(new JoinRemoveListener())
-.build(loader);
+loader = s -> new HashMap<>();
+cache = Caffeine.newBuilder().maximumSize(maxCacheSize)
+ .expireAfterWrite(maxTimeRetain, TimeUnit.MINUTES)
+ .removalListener(new JoinRemoveListener())
--- End diff --

So, I believe this was intentionally done before this PR (I migrated this 
to the new caching strategy) and the idea is that if a removal is happening 
from the join cache under specific circumstances, we want to know about it 
because a message could be being dropped because the cache is being 
overwhelmed.  @merrimanr Can you chime in here on the rationale?


> Replace guava caches in places where the keyspace might be large
> 
>
> Key: METRON-1467
> URL: https://issues.apache.org/jira/browse/METRON-1467
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>Priority: Major
>
> Based on the performance tuning exercise as part of METRON-1460, guava has 
> difficulties with cache sizes over 10k.  We, unfortunately, are quite 
> demanding of guava in this regard so we should transition a few uses of guava 
> to Caffeine:
>  * Stellar processor cache
>  * The JoinBolt cache
>  * The Enrichment Bolt Cache
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1467) Replace guava caches in places where the keyspace might be large

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389724#comment-16389724
 ] 

ASF GitHub Bot commented on METRON-1467:


Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/947#discussion_r172888378
  
--- Diff: 
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/JoinBolt.java
 ---
@@ -89,29 +91,25 @@ public void prepare(Map map, TopologyContext 
topologyContext, OutputCollector ou
 if (this.maxTimeRetain == null) {
   throw new IllegalStateException("maxTimeRetain must be specified");
 }
-loader = new CacheLoader>() {
-  @Override
-  public Map load(String key) throws Exception {
-return new HashMap<>();
-  }
-};
-cache = CacheBuilder.newBuilder().maximumSize(maxCacheSize)
-.expireAfterWrite(maxTimeRetain, 
TimeUnit.MINUTES).removalListener(new JoinRemoveListener())
-.build(loader);
+loader = s -> new HashMap<>();
+cache = Caffeine.newBuilder().maximumSize(maxCacheSize)
+ .expireAfterWrite(maxTimeRetain, TimeUnit.MINUTES)
+ .removalListener(new JoinRemoveListener())
--- End diff --

Yes, it is pre-existing.  We can address at a later time.

I remember now, maxing out this cache causes the Split/Join to fail, which 
is a major problem for the Split/Join topology.  And this cache here is only 
for the Split/Join, not the Unified topology.

We should probably look at adding similar logging (only when ERROR enabled) 
for the other places where we use the cache.  Or just some mechanism to 
periodically log cache stats.  Anywho, down the road.


> Replace guava caches in places where the keyspace might be large
> 
>
> Key: METRON-1467
> URL: https://issues.apache.org/jira/browse/METRON-1467
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>Priority: Major
>
> Based on the performance tuning exercise as part of METRON-1460, guava has 
> difficulties with cache sizes over 10k.  We, unfortunately, are quite 
> demanding of guava in this regard so we should transition a few uses of guava 
> to Caffeine:
>  * Stellar processor cache
>  * The JoinBolt cache
>  * The Enrichment Bolt Cache
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1467) Replace guava caches in places where the keyspace might be large

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389725#comment-16389725
 ] 

ASF GitHub Bot commented on METRON-1467:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/947
  
+1 LGTM


> Replace guava caches in places where the keyspace might be large
> 
>
> Key: METRON-1467
> URL: https://issues.apache.org/jira/browse/METRON-1467
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>Priority: Major
>
> Based on the performance tuning exercise as part of METRON-1460, guava has 
> difficulties with cache sizes over 10k.  We, unfortunately, are quite 
> demanding of guava in this regard so we should transition a few uses of guava 
> to Caffeine:
>  * Stellar processor cache
>  * The JoinBolt cache
>  * The Enrichment Bolt Cache
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1467) Replace guava caches in places where the keyspace might be large

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389754#comment-16389754
 ] 

ASF GitHub Bot commented on METRON-1467:


Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/947


> Replace guava caches in places where the keyspace might be large
> 
>
> Key: METRON-1467
> URL: https://issues.apache.org/jira/browse/METRON-1467
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>Priority: Major
>
> Based on the performance tuning exercise as part of METRON-1460, guava has 
> difficulties with cache sizes over 10k.  We, unfortunately, are quite 
> demanding of guava in this regard so we should transition a few uses of guava 
> to Caffeine:
>  * Stellar processor cache
>  * The JoinBolt cache
>  * The Enrichment Bolt Cache
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)