[
https://issues.apache.org/jira/browse/METRON-1555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519458#comment-16519458
]
ASF GitHub Bot commented on METRON-1555:
----------------------------------------
GitHub user merrimanr reopened a pull request:
https://github.com/apache/metron/pull/1019
METRON-1555: Update REST to run YARN and MR jobs
## Contributor Comments
This PR sets us up to run YARN and MR jobs inside our REST application.
Changes include:
- addition of maven dependencies
- addition of -Dhdp.version parameter to the REST startup script
- MPack now supplies the hdp.version parameter
- MPack now sets up a "metron" service user HDFS directory needed for
running MR jobs
- MPack now sets up a pcap HDFS directory
- addition of a Pcap controller with a single Fixed Pcap Query endpoint and
service to demonstrate running MR jobs in REST
The fixed pcap query endpoint submitted here should match the functionality
in the metron-api module with a few minor differences:
- the default input and output paths are spring properties instead of
hardcoded in classes (this will make it easier to expose them in Ambari if we
choose to)
- query results are not cleaned up automatically since that work is
captured in a separate Jira
- num reducers is defaulted to 1 instead of 10
Unit and integration tests are included and this has been tested in full
dev. I tested this by generating sample pcap data with the
PcapTopologyIntegrationTest. You can do this by either:
- running the test in your IDE and pausing it after the topology has
generated data
- commenting out `clearOutDir(outDir);` and running the test
Pcap data should be present in
`/metron/metron-platform/metron-pcap-backend/target/pcap/data_dir`. Upload the
`pcap*` files to the `/apps/metron/pcap` directory in HDFS. You should be able
to perform the tests in PcapTopologyIntegrationTest using REST and get the same
results.
For example:
```
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"endTime": 1458240269424,
"startTime": 1458240269419
}' 'http://node1:8082/api/v1/pcap/fixed'
```
should return 2 pcap results.
```
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{}' 'http://node1:8082/api/v1/pcap/fixed'
```
should return 20 pcap results.
```
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"ipDstAddr":"207.28.210.1"
}' 'http://node1:8082/api/v1/pcap/fixed'
```
should return no pcap results.
```
curl -X POST --header 'Content-Type: application/json' --header 'Accept:
application/json' -d '{
"ipDstPort": 22
}' 'http://node1:8082/api/v1/pcap/fixed'
```
should return 10 pcap results.
Related discussion can be found here:
http://mail-archives.apache.org/mod_mbox/metron-dev/201805.mbox/%3ccaevkqpbxzjnu_wgrbfwnz-mvqnkb7mthedveq9plyhwfit7...@mail.gmail.com%3E
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.
Please refer also to our [Build Verification
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified
and tested manually?
- [] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
```
mvn -q clean integration-test install &&
dev-utilities/build-utils/verify_licenses.sh
```
- [ ] Have you written or updated unit tests and or integration tests to
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in
which it is rendered by building and verifying the site-book? If not then run
the following commands and the verify changes via
`site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up
for your personal repository such that your branches are built there before
submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/merrimanr/incubator-metron pcap-rest-test
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/1019.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1019
----
commit 4dcee51dd544e6c064dcc9dd1478c923a00c8281
Author: merrimanr <merrimanr@...>
Date: 2018-05-07T20:52:09Z
added simple pcap endpoint to rest
commit 22fe5e9ff3c167b42ebeb7a9f1000753a409aff1
Author: merrimanr <merrimanr@...>
Date: 2018-05-08T22:32:03Z
pcap query runs in rest
commit a65d7f5bcda61c2272a7753c6d9ffebb227968cb
Author: merrimanr <merrimanr@...>
Date: 2018-05-16T21:35:01Z
updated dependencies_with_url.csv
commit 75b50a5fcf51fd6b88311e4ba004bb070f8bd27c
Author: merrimanr <merrimanr@...>
Date: 2018-06-07T20:29:08Z
removed service classes
commit 22f127af06a4d1eb53cba0bfb238c9668fff76fc
Author: merrimanr <merrimanr@...>
Date: 2018-06-13T13:25:48Z
Revert "removed service classes"
This reverts commit 75b50a5fcf51fd6b88311e4ba004bb070f8bd27c.
commit e83d645910e8c03e0a3e4e3e8de73a9c664d2aa6
Author: merrimanr <merrimanr@...>
Date: 2018-06-13T19:07:37Z
Merge remote-tracking branch 'mirror/feature/METRON-1554-pcap-query-panel'
into pcap-rest-test
commit 1573cb62684ac075bb31ea2d4fb6f604e18cf512
Author: merrimanr <merrimanr@...>
Date: 2018-06-15T13:13:53Z
added pcap service
commit fbbd230bece5cc6284f752f26b83a2987098afed
Author: merrimanr <merrimanr@...>
Date: 2018-06-18T21:27:59Z
added tests and minor fixes
----
> Update REST to run YARN and MR jobs
> -----------------------------------
>
> Key: METRON-1555
> URL: https://issues.apache.org/jira/browse/METRON-1555
> Project: Metron
> Issue Type: Sub-task
> Reporter: Ryan Merriman
> Assignee: Ryan Merriman
> Priority: Major
>
> This task involves enabling REST to submit YARN or MR jobs. We will likely
> need to:
> * update Maven dependencies to include YARN and MR libraries in the
> classpath and resolve any version conflicts
> * update REST start script to include properties required for YARN
> * update the MPack for any additional setup work (create user HDFS directory
> for example) and properties needed
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
