GitHub user nickwallen reopened a pull request:
https://github.com/apache/metron/pull/1174
METRON-1748 Improve Storm Profiler Integration Test
Improved the Storm Profiler integration tests based on improvements I made
for the Spark Profiler feature branch.
* Validate the output of the Profiler using the client library;
`PROFILE_GET`. This is better validation that things are working end-to-end.
* Using more telemetry messages to validate event time processing.
* Added logging around cache maintenence.
* Added logging of the tuple windows that are passed from Storm's
WindowManager. This can help debug issues where the time lag and window size
are incorrectly set.
## Testing
This only changes the integration tests and adds logging. Running the
integration tests is sufficient to test the changes there. Running up the
development environment is sufficient for the extra caution.
## Pull Request Checklist
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [x] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
- [x] Have you written or updated unit tests and or integration tests to
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/metron METRON-1748
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/1174.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1174
----
commit 56b70f666dd75a2861d17a23b7ed307124ba1f74
Author: Nick Allen <nick@...>
Date: 2018-08-17T17:13:10Z
METRON-1748 Improve Storm Profiler Integration Test
commit 33516a52788a764005a7bc3946034bb6a2858201
Author: Nick Allen <nick@...>
Date: 2018-08-23T18:54:35Z
Small changes to tuple window logging
commit fbb2fcd50df7e5812ed63479891f56b65adc899f
Author: Nick Allen <nick@...>
Date: 2018-08-24T13:01:44Z
Need to wait windowLagMillis + periodDurationMillis before sending another
message to close the event window
commit bf160d521ba8530f556fdc610c6845bf3065e8fb
Author: Nick Allen <nick@...>
Date: 2018-08-24T13:49:08Z
Updated the event time test so that both periods contained in the data are
validated
----
---
