[
https://issues.apache.org/jira/browse/METRON-610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15725946#comment-15725946
]
ASF GitHub Bot commented on METRON-610:
---------------------------------------
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/388
Ok, I tested this in vagrant. The testing methodology was as follows:
* Created a simple global profile that adds a `value` field to a stats
object and serializes it:
```
{
"profiles": [
{
"profile": "stats",
"foreach": "'global'",
"onlyif": "true",
"init" : {
"s": "STATS_INIT()"
},
"update": {
"s": "STATS_ADD(s, value)"
},
"result": "s"
}
]
}
```
* Created a simple CSV parser called `mad` that takes in comma separated
pairs and adds the mean from the last 10 minute window:
```
{
"parserClassName" : "org.apache.metron.parsers.csv.CSVParser"
,"sensorTopic" : "mad"
,"parserConfig" : {
"columns" : {
"value_str" : 0
,"source" : 1
}
}
,"fieldTransformations" : [
{
"transformation" : "STELLAR"
,"output" : [ "value" , "mean"]
,"config" : {
"value" : "TO_DOUBLE(value_str)",
"mean" : "STATS_MEAN(STATS_MERGE(PROFILE_GET('stats', 'global', 10,
'MINUTES')))"
}
}
]
}
```
* Created an enrichment that adds a field `enrichment_mean` via a stellar
enrichment:
```
{
"index": "mad",
"batchSize": 1,
"enrichment": {
"fieldMap": {
"stellar" : {
"config" : {
"enrichment_mean" : "STATS_MEAN(STATS_MERGE(PROFILE_GET('stats',
'global', 10, 'MINUTES')))"
}
}
}
,"fieldToTypeMap": { }
},
"threatIntel": {
"fieldMap": { },
"fieldToTypeMap": { },
"triageConfig" : {
"riskLevelRules" : {
},
"aggregator" : "MAX"
}
}
}
```
* Generated a stream of data of the form `value,type` (e.g. 2.45,synthetic)
* Open up the stellar shell after 10 minutes and run
`STATS_MEAN(STATS_MERGE(PROFILE_GET('stats', 'global', 10, 'MINUTES')))` and
ensure I get something that is not `NaN`
This ensures that statistics objects are able to be serialized and
deserialized correctly from the:
* Enrichment topology
* Parser topology
* Stellar REPL
> OnlineStatisticsProvider serialization is broken at random in the REPL
> ----------------------------------------------------------------------
>
> Key: METRON-610
> URL: https://issues.apache.org/jira/browse/METRON-610
> Project: Metron
> Issue Type: Bug
> Reporter: Casey Stella
> Assignee: Casey Stella
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> We rely on the t-digest library version 3.1 and elasticsearch brings along
> 3.0. There is a small API incompatibility between the two versions (namely
> the static method TDigest.createAvlTreeDigest() is available in 3.1 but not
> 3.0).
> If the classpath for the Stellar REPL chooses the 3.0 version of the library,
> then deserialization is broken.
> Strictly speaking this is not a problem of the serialized form being
> incorrect, but a problem in the custom kryo serialization code in the class.
> It relies on the default constructor being called and then the digest being
> deserialized using the code within the t-digest library. Because the default
> constructor initializes the digest via a call that does not exist in 3.0, it
> breaks. The serialization logic is safe to use in both versions, but the
> object can't be constructed in 3.0.
> This fix directly instantiates the AvlTreeDigest, which exists in both
> versions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
