GitHub user merrimanr opened a pull request:
https://github.com/apache/metron/pull/1275
METRON-1878: Add Metron as a Knox service
## Contributor Comments
This PR adds REST and the Alerts UI as services in Knox. Currently this is
incomplete and meant as a starting point to give reviewers an idea of the
changes involved. The outstanding items are detailed below.
### Changed Included
- Fixed minor path issues in various UI related code. Files include:
- app.component.html - path should be relative
- authentication.service.ts - logout should match other endpoints
- alert-search.directive.ts - path should be relative
- index.html - a relative base href means Knox does not have to rewrite
links
- Created Metron Knox topology and service definition files for REST and
Alerts UI. The service definition files are relatively simple and should look
similar to other services.
- Created a script to deploy Metron files to Knox
- Added a Spring "knox" profile that configures Swagger with the right
endpoint paths
- Added service definition files to rpm spec
### Testing
This feature can be tested in full dev with some manual configuration.
After spinning up full dev:
1. Add Knox as a Service in Ambari (Admin > Stacks and Versions > Add
Service next to Knox)
2. Follow the instructions
[here](https://github.com/apache/metron/tree/master/metron-deployment/development#knox-demo-ldap)
to enable the Knox LDAP in full dev. You should be able to authentication
with REST using guest/guest-password after this is done.
3. Install the Metron topology and service definition files by running the
script at `/usr/metron/0.6.1/bin/install_metron_knox.sh`. This script copies
the metron.xml topology file to `/usr/hdp/current/knox-server/conf/topologies`
and the service definition files to
`/usr/hdp/current/knox-server/data/services/metron-rest/0.6.1` and
`/usr/hdp/current/knox-server/data/services/metron-alerts/0.6.1`
4. Update the REST API path for the Alerts UI. Currently expressjs proxies
REST requests from the Alerts UI but this will now be done by Knox instead.
The `apiRoot` setting in `app-config.json` is used to construct the url for
REST requests. Change this setting in
`/usr/metron/0.6.1/web/alerts-ui/assets/app-config.json` from:
```
{
"apiRoot": "/api/v1"
}
```
to:
```
{
"apiRoot": "/gateway/metron/metron-rest/api/v1"
}
```
5. Update the REST API path for Swagger. The Swagger interface is served
by the REST application and provides links for the various endpoints. Swagger
must be configured with the new Knox root path for REST. Navigate to `Ambari >
Metron > Configs > REST`. Change the `Active Spring profiles` setting from
`dev` to `dev,knox` and change the `Metron Spring options` to
`--knox.root=/gateway/metron/metron-rest`. This configures Swagger to prepend
the endpoint paths with the `knox.root` Spring property.
6. Restart REST with Ambari and you should now be able to access both
Swagger and the Alerts UI through Knox:
- Swagger - https://node1:8443/gateway/metron/metron-rest/swagger-ui.html
- Alerts UI - https://node1:8443/gateway/metron/metron-alerts/alerts-list
Everything should function normally including login and logout.
### Outstanding Items
#### Installing Metron in Knox
Installing a service in Knox involves 2 steps:
1. Deploy the service definition files mentioned earlier by copying them to
Knox directories on the machine where Knox is installed
2. Either update an existing Knox topology file or create a new topology
file. This file configures a collection of services. In our case this
includes authentication and urls to the services exposed.
What is the best way to install these files in Knox? The approach in this
PR only works if Metron is installed on the same machine as the Knox gateway.
I also opted to create a dedicated Metron topology file. I think this will
give us more control and allow us to expose properties in a more user-friendly
way. Knox has a default topology that you can configure in Ambari but it's
exposed as is: a big xml file. Does anyone think we should use the default
topology file instead?
#### Adding Knox to the stack
This process requires that we add Knox separately through Ambari. Do we
want to make Knox a dependency for Metron similar to Kafka, Storm, etc?
Does it make sense to require that Knox and Metron are colocated? Doing
this simplifies installation and configuration and I would vote we make this a
requirement unless there is a good reason not to. Does anyone think there are
cases where users would need to install them on different machines? If we do
that, how do we install the service definition files?
#### Ambari automation
Currently setting up Metron with Knox is a manual process. I believe all
of this can easily be automated with Ambari. How far do we want to go in this
initial pass? Do we want to just document it for now? If we did want to
automate this setup with Ambari, we would need to:
- Put the app-config.json under Ambari control and expose the `apiRoot`
setting
- Add the `knox.root` setting to the REST config file
(`rest_application.yml`) and expose it through an Ambari setting
- Add an Ambari setting similar to the `LDAP enabled` setting that
automatically adds the `knox` spring profile to the list of active profiles
- Possibly put the metron.xml topology file under Ambari control
- Add a call to `install_metron_knox.sh` in the appropriate Metron Mpack
script
#### Quick Links
At this point I do not know how to make Ambari quick links dynamic. You
can bind the port to a property but I don't know if you can do that for the
scheme, host, and path. This means that after you follow these instructions
for setting up Metron with Knox, the quick links will still point to the urls
without Knox. Does anyone know how to change quick links to point to Knox
automatically? There has been discussion of enabling Knox by default so in
that case we can just set the quick links to the Knox urls.
Hopefully a working example will provide more context as we plan out this
feature.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.
Please refer also to our [Build Verification
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
```
mvn -q clean integration-test install &&
dev-utilities/build-utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered by building and verifying the site-book? If not then run
the following commands and the verify changes via
`site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up
for your personal repository such that your branches are built there before
submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/merrimanr/incubator-metron knox-mpack
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/1275.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1275
----
commit c0345e2faba1f10f495a4c3c3c5ec8b5137e2d1c
Author: merrimanr <merrimanr@...>
Date: 2018-11-16T20:58:46Z
initial commit
commit 6cf7303af1d5cb06fea0d70a52f553c82878de3f
Author: merrimanr <merrimanr@...>
Date: 2018-11-19T21:05:24Z
missing license headers and rpm spec updates
----
---
