[
https://issues.apache.org/jira/browse/METRON-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16969431#comment-16969431
]
Nick Allen commented on METRON-2284:
------------------------------------
This definitely looks to be a bug. The behavior between the Profiler in the
REPL and in the Batch Profiler in Spark should be identical, but it seems not
to be.
What are you trying to do with this profile? Maybe I can help you with a
workaround until we can fix the problem.
BTW, thank you for providing such a clear bug report with the exact steps to
replicate. Very helpful!
> Metron Profiler for Spark doesn't work as expected
> --------------------------------------------------
>
> Key: METRON-2284
> URL: https://issues.apache.org/jira/browse/METRON-2284
> Project: Metron
> Issue Type: Bug
> Affects Versions: 0.7.1
> Reporter: Maxim Dashenko
> Priority: Major
>
> Used command:
> {code}
> /usr/hdp/current/spark2-client/bin/spark-submit --class
> org.apache.metron.profiler.spark.cli.BatchProfilerCLI --properties-file
> /usr/hcp/current/metron/config/batch-profiler.properties
> ~/metron-profiler-spark-0.7.1.1.9.1.0-6.jar --config
> /usr/hcp/current/metron/config/batch-profiler.properties --profiles
> ~/profiler.json
> {code}
> cat /usr/hcp/current/metron/config/batch-profiler.properties
> {code}
> profiler.batch.input.path=/tmp/test_data.logs
> profiler.batch.input.format=json
> profiler.period.duration=15
> profiler.period.duration.units=MINUTES
> {code}
>
> cat ~/profiler.json
> {code}
> {
> "profiles":[
> {
> "profile":"batchtest5",
> "onlyif":"source.type == 'testsource' and devicehostname ==
> 'windows9.something.com'",
> "foreach":"devicehostname",
> "init":{
> "val":"SET_INIT()"
> },
> "update":{
> "val":"SET_ADD(val, IS_EMPTY(devicehostname))"
> },
> "result":{
> "profile":"val"
> }
> }
> ],
> "timestampField":"timestamp"
> }
> {code}
> cat test_data.logs
> {code}
> {"devicehostname": "windows9.something.com", "timestamp": 1567241981000,
> "source.type": "testsource"}
> {code}
> Stellar statement
> {code}
> PROFILE_GET('batchtest5', 'windows9.something.com', PROFILE_FIXED(100,
> 'DAYS'))
> {code}
> Returns:
> {code}
> [[true]]
> {code}
> Expected result:
> {code}
> [[false]]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
