[
https://issues.apache.org/jira/browse/METRON-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16954000#comment-16954000
]
Michael Miklavcic commented on METRON-2284:
-------------------------------------------
Can we have you try the following exercise from the REPL using your profile?
https://github.com/apache/metron/blob/master/metron-analytics/metron-profiler-repl/README.md#getting-started
{code:java}
[Stellar]>>> val := SET_INIT()
[]
[Stellar]>>> devicehostname := 'windows9.something.com'
windows9.something.com
[Stellar]>>> val := SET_ADD(val, IS_EMPTY(devicehostname))
[false]
[Stellar]>>> conf := SHELL_EDIT()
# add the following profile contents in the vi editor that comes up:
{
"profiles":[
{
"profile":"batchtest5",
"onlyif":"devicehostname == 'windows9.something.com'",
"foreach":"devicehostname",
"init":{
"val":"SET_INIT()"
},
"update":{
"val":"SET_ADD(val, IS_EMPTY(devicehostname))"
},
"result":{
"profile":"val"
}
}
],
"timestampField":"timestamp"
}
[Stellar]>>> profiler := PROFILER_INIT(conf)
Profiler{1 profile(s), 0 messages(s), 0 route(s)}
[Stellar]>>> msg := SHELL_EDIT()
# add this record
{"devicehostname": "windows9.something.com", "timestamp": 1567241981000}
[Stellar]>>> PROFILER_APPLY(msg, profiler)
Profiler{1 profile(s), 1 messages(s), 1 route(s)}
[Stellar]>>> values := PROFILER_FLUSH(profiler)
[{period={duration=900000, period=1741379, start=1567241100000,
end=1567242000000}, profile=batchtest5, groups=[], value=[false],
entity=windows9.something.com}]
{code}
I'm seeing "value" set to false, as expected, at least from the REPL. Let's see
if we can verify that part of the functionality matches up as expected and go
from there.
> Metron Profiler for Spark doesn't work as expected
> --------------------------------------------------
>
> Key: METRON-2284
> URL: https://issues.apache.org/jira/browse/METRON-2284
> Project: Metron
> Issue Type: Bug
> Affects Versions: 0.7.1
> Reporter: Maxim Dashenko
> Priority: Major
>
> Used command:
> {code}
> /usr/hdp/current/spark2-client/bin/spark-submit --class
> org.apache.metron.profiler.spark.cli.BatchProfilerCLI --properties-file
> /usr/hcp/current/metron/config/batch-profiler.properties
> ~/metron-profiler-spark-0.7.1.1.9.1.0-6.jar --config
> /usr/hcp/current/metron/config/batch-profiler.properties --profiles
> ~/profiler.json
> {code}
> cat /usr/hcp/current/metron/config/batch-profiler.properties
> {code}
> profiler.batch.input.path=/tmp/test_data.logs
> profiler.batch.input.format=json
> profiler.period.duration=15
> profiler.period.duration.units=MINUTES
> {code}
>
> cat ~/profiler.json
> {code}
> {
> "profiles":[
> {
> "profile":"batchtest5",
> "onlyif":"source.type == 'testsource' and devicehostname ==
> 'windows9.something.com'",
> "foreach":"devicehostname",
> "init":{
> "val":"SET_INIT()"
> },
> "update":{
> "val":"SET_ADD(val, IS_EMPTY(devicehostname))"
> },
> "result":{
> "profile":"val"
> }
> }
> ],
> "timestampField":"timestamp"
> }
> {code}
> cat test_data.logs
> {code}
> {"devicehostname": "windows9.something.com", "timestamp": 1567241981000,
> "source.type": "testsource"}
> {code}
> Stellar statement
> {code}
> PROFILE_GET('batchtest5', 'windows9.something.com', PROFILE_FIXED(100,
> 'DAYS'))
> {code}
> Returns:
> {code}
> [[true]]
> {code}
> Expected result:
> {code}
> [[false]]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
