[
https://issues.apache.org/jira/browse/METRON-1925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16711892#comment-16711892
]
ASF GitHub Bot commented on METRON-1925:
----------------------------------------
GitHub user nickwallen opened a pull request:
https://github.com/apache/metron/pull/1292
METRON-1925 Provide Verbose View of Profile Results in REPL
## Motivation
When viewing profile measurements in the REPL using PROFILE_GET, you simply
get back a list of values. It is not easy to determine from which time period
the measurements were taken.
For example, are the following values all sequential? Are there any gaps
in the measurements taken over the past 30 minutes? When was the first
measurement taken?
```
[Stellar]>>> PROFILE_GET("hello-world","global",PROFILE_FIXED(30,
"MINUTES"))
[2655, 1170, 1185, 1170, 1185, 1215, 1200, 1170]
```
The `PROFILE_GET` function was designed to return values that serve as
input to other functions. It was not designed to return values in a
human-readable form that can be easily understood. We need another way to query
for profile measurements in the REPL that provides a user with a better
understanding of the profile measurements.
## Solution
This PR provides a new function called `PROFILE_VIEW`. It is effectively a
"verbose mode" for `PROFILE_GET`.
For lack of a better name, I just called it `PROFILE_VIEW`. I would be
open to alternatives. I did not want to add additional options to the already
complex `PROFILE_GET`.
* Description: Retrieves a series of measurements from a stored profile.
Provides a more verbose view of each measurement than PROFILE_GET. Returns a
map containing the profile name, entity, period id, period start, period end
for each profile measurement.
* Arguments:
profile - The name of the profile.
entity - The name of the entity.
periods - The list of profile periods to fetch. Use PROFILE_WINDOW or
PROFILE_FIXED.
groups - Optional, The groups to retrieve. Must correspond to the
'groupBy' list used during profile creation. Defaults to an empty list, meaning
no groups.
* Returns: A map for each profile measurement containing the profile name,
entity, period, and value.
## Test Drive
1. Spin-up Full Dev and create a profile. Follow the Profiler README.
Reduce the profile period if you are impatient.
1. Open up the REPL and retrieve the values using `PROFILE_GET`. Notice
that I have no idea when the first measurement was taken, if the values are
sequential, if there are gaps in the values and how big.
```
[Stellar]>>> PROFILE_GET("hello-world","global",PROFILE_FIXED(30,
"MINUTES"))
[1185, 1170, 1185, 1215, 1200, 1170, 5425, 1155, 1215, 1200]
```
1. Now use `PROFILE_VIEW` to retrieve the same results.
```
[Stellar]>>> results :=
PROFILE_VIEW("hello-world","global",PROFILE_FIXED(30, "MINUTES"))
[{period.start=1544119560000, period=12867663, profile=hello-world,
period.end=1544119680000, groups=[], value=1185, entity=global},
{period.start=1544119680000, period=12867664, profile=hello-world,
period.end=1544119800000, groups=[], value=1170, entity=global},
{period.start=1544119800000, period=12867665, profile=hello-world,
period.end=1544119920000, groups=[], value=1185, entity=global},
{period.start=1544119920000, period=12867666, profile=hello-world,
period.end=1544120040000, groups=[], value=1215, entity=global},
{period.start=1544120040000, period=12867667, profile=hello-world,
period.end=1544120160000, groups=[], value=1200, entity=global},
{period.start=1544120160000, period=12867668, profile=hello-world,
period.end=1544120280000, groups=[], value=1170, entity=global},
{period.start=1544120880000, period=12867674, profile=hello-world,
period.end=1544121000000, groups=[], value=5425, entity=global},
{period.start=1544121000000, period=12867675, profile=hello-world,
period.end=1544121120000, groups=[], value=1155, entity=global},
{period.start=1544121120000, period=12867676, profile=hello-world,
period.end=1544121240000, groups=[], value=1215, entity=global},
{period.start=1544121240000, period=12867677, profile=hello-world,
period.end=1544121360000, groups=[], value=1200, entity=global}]
```
1. For each measurement, I have a map containing the period, period start,
period end, profile name, entity, groups, and value. With this I can better
answer some of the questions above.
```
{
profile=hello-world,
entity=global,
period=12867663,
period.start=1544119560000,
period.end=1544119680000,
groups=[],
value=1185
}
```
1. When was the first measurement taken?
```
[Stellar]>>> GET(results, 0)
{period.start=1544119560000, period=12867663, profile=hello-world,
period.end=1544119680000, groups=[], value=1185, entity=global}
```
I can see that the first period started at 1544119560000 or Thu Dec 06
2018 18:06:00 UTC.
1. Are these measurements sequential? Are there any gaps?
I can use `MAP` on the list of results to extract just the periods from
each profile measurement. The period is a monotonically increasing value.
```
[Stellar]>>> MAP(results, m -> MAP_GET("period", m))
[12867663, 12867664, 12867665, 12867666, 12867667, 12867668, 12867674,
12867675, 12867676, 12867677]
```
From this I can tell that there is a gap in the measurements.
* The first period here is 12867663 and the last is 12867677.
* I can see that the first 6 measurements are sequential up to period
12867668.
* Then there is a gap of 5 periods (12867669 - 12867673) before the
measurements resume at period 12867674.
1. How big was that gap?
We know that we missed 5 periods, but how big is that gap?
Here is when each period starts in epoch milliseconds.
```
[Stellar]>>> MAP(results, m -> MAP_GET("period.start", m))
[1544119560000, 1544119680000, 1544119800000, 1544119920000,
1544120040000, 1544120160000, 1544120880000, 1544121000000, 1544121120000,
1544121240000]
```
I can find the difference between the 5th and 6th period. The
different is about 720000 milliseconds, which is 12 minutes.
```
[Stellar]>>> MAP_GET("period.start", GET(results, 5))
1544120160000
[Stellar]>>> MAP_GET("period.start", GET(results, 6))
1544120880000
[Stellar]>>> MAP_GET("period.start", GET(results, 6)) -
MAP_GET("period.start", GET(results, 5))
720000
```
## Changes
* Altered the ProfilerClient so that it returns the `ProfileMeasurement`
values instead of just a list of the raw values stored in HBase.
* Removed one of the methods of ProfileClient that was not being used to
simplify things.
* Updated `PROFILE_GET` to work with the altered ProfilerClient.
* Added `PROFILE_VIEW`.
## Pull Request Checklist
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [ ] Does your PR title start with METRON-XXXX where XXXX is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [ ] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
- [ ] Have you written or updated unit tests and or integration tests to
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/metron METRON-1925
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/1292.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1292
----
commit 8a058e6274d712ed046325ccb408063f8ac9edbf
Author: Nick Allen <nick@...>
Date: 2018-12-04T13:54:45Z
METRON-1925 Provide Verbose View of Profile Results in REPL
----
> Provide Verbose View of Profile Results in REPL
> -----------------------------------------------
>
> Key: METRON-1925
> URL: https://issues.apache.org/jira/browse/METRON-1925
> Project: Metron
> Issue Type: Bug
> Reporter: Nick Allen
> Assignee: Nick Allen
> Priority: Major
>
> When viewing profile measurements in the REPL using PROFILE_GET, you simply
> get back a list of values. It is not easy to determine from which time period
> the measurements were taken.
> For example, are the following values all sequential? Are there any gaps in
> the measurements taken over the past 30 minutes? When was the first
> measurement taken?
> {code:java}
> [Stellar]>>> PROFILE_GET("hello-world","global",PROFILE_FIXED(30, "MINUTES"))
> [2655, 1170, 1185, 1170, 1185, 1215, 1200, 1170]{code}
> The `PROFILE_GET` function was designed to return values that serve as input
> to other functions. It was not designed to return values in a human-readable
> form that can be easily understood.
> We need another way to query for profile measurements in the REPL that gives
> me a better understanding of the profile measurements.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
