[
https://issues.apache.org/jira/browse/METRON-606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15717077#comment-15717077
]
ASF GitHub Bot commented on METRON-606:
---------------------------------------
GitHub user nickwallen opened a pull request:
https://github.com/apache/incubator-metron/pull/387
METRON-606
[METRON-606](https://issues.apache.org/jira/browse/METRON-606)
The Profiler is repetitively overwriting previously written values. Where I
would expect to see a series of multiple values in Hbase, one written per
flush, there is only a single value. This same value is being repetitively
overwritten with new data.
The following changes are included in the PR.
* Refactored the ProfileBuilderBolt so that the logic for building a
profile no longer lives in the bolt itself, but in a separate ProfileBuilder.
The bolt simply maintains a cache of ProfileBuilder instances.
* Ensured that ProfileMeasurement objects are not reused, which reduces the
likelihood of another defect like this in the future.
* Added a Clock interface that allows me to advance time in the unit tests.
This was necessary for testing this specific defect. This will also help with
METRON-590 in dealing with processing versus event time.
* Added a specific unit test for this defect.
* Added a unit test for the recent 'tickUpdate' addition to the Profiler.
The change was tested on a live AWS cluster along with the 'Quick Dev'
environment.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/incubator-metron METRON-606
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/387.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #387
----
commit 4e9fc6307356e9dcb71ec1d9ed78644b1e7f3eb3
Author: Nick Allen <[email protected]>
Date: 2016-11-25T23:24:22Z
METRON-606 Extracted ProfileBuilder logic from the ProfileBuilderBolt.
commit deaaf4cb5066e1c03f20b45bb62a577724cdf22f
Author: Nick Allen <[email protected]>
Date: 2016-12-02T23:07:43Z
METRON-606 Ensure that a new profile measurement is created on each flush
commit cad4674df2eaf67bcfbc8de3cc8fa74a8f1a2b77
Author: Nick Allen <[email protected]>
Date: 2016-12-02T23:34:46Z
METRON-606 Added a unit test specifically for METRON-606
commit d5bfa09248018ad009ac20ccf4032082f4c06d74
Author: Nick Allen <[email protected]>
Date: 2016-12-02T23:53:04Z
METRON-606 Added missing ASF license headers
commit 8b0d84336503898300cf17ddc8ac78c8c25932d3
Author: Nick Allen <[email protected]>
Date: 2016-12-03T01:15:57Z
METRON-606 Exception message can cause an NPE
----
> Profiler Overwriting Previously Written Values
> ----------------------------------------------
>
> Key: METRON-606
> URL: https://issues.apache.org/jira/browse/METRON-606
> Project: Metron
> Issue Type: Bug
> Reporter: Nick Allen
> Assignee: Nick Allen
>
> The Profiler is repetitively overwriting previously written values. Where I
> would expect to see a series of multiple values in Hbase, one written per
> flush, there is only a single value. This same value is being repetitively
> overwritten with new data.
> This is occurring because the same ProfileMeasurement is being re-used after
> a flush event. Prior to METRON-575, a new ProfileMeasurement was created
> after a flush event. Without the new ProfileMeasurement, the timestamp used
> to generate the row key remains the same.
> Drat.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
