I would like to know if using PROFILE_GET is appropriate in profiler definition?
I am seeing un-appropriate results. I need to use PROFILE_GET in enrichment configuration as well as in profiler definition. I am seeing very un-required results when I experiment having PROFILE_GET in enrichment config and profiler definition. Thanks, Anil. From: Casey Stella [mailto:[email protected]] Sent: Saturday, April 28, 2018 12:26 AM To: [email protected] Cc: Satish Abburi <[email protected]> Subject: Re: Parse Exception at SplitterBolt in Profiler Topology: 2018-04-27 10:57:16.575 o.a.m.p.b.ProfileSplitterBolt Thread-6-splitterBolt-executor[7 7] [ERROR] Unexpected failure: message='null' That exception appears to me to be a problem in parsing the message coming into the profiler as opposed to having trouble parsing the profiler config. That list of integers are the raw characters in the message. It may be worthwhile to try to take the array of integers and try to turn them into a string (in python that could be as simple as ''.join(chr(i) for i in [123,***********Lot of integer values here in the logs end with****34, 48, 34, 125]) from the python REPL) to see exactly what was in the raw message. On Fri, Apr 27, 2018 at 2:40 PM Anil Donthireddy <[email protected]<mailto:[email protected]>> wrote: Hi, I defined a profile definition for my requirement to perform profile statistics. The calculation of profiler statistics for current profiling period will depend on the previous profile flushed stats. So, I am trying to read the previous stats during initialization ( “init” field in json ) part of profile for the current profiling period. The definition works fine when there are profile stats from the previous period, which means my profile definition working fine for first 15 minutes (my profile period is set to 15 mins). When profiler flushes the profile stats into hbase for first 15 minutes and starts for next period, I am getting the parse exception as seen in attached doc at Profiler topology and SplitterBolt. I have executed all lines of code in the stellar shell (instantiated with zookeeper config ./bin/stellat –z XXXX). It works fine without any issues in all the cases. Which means, I am able to fetch the previous profile stats, initialize for current record and execute all lines of code in profiler definition with expected result. I could not able to figure out what else is going wrong during SplitterBolt execution. I request someone to go through the attached logs and help me to troubleshoot the issue further. High Level Exception stack trace: *************************************** Exception seen at profiler topology logs ******************************* 2018-04-27 10:56:06.483 o.a.h.m.s.s.StormTimelineMetricsSink Thread-23 [WARN] Unable to send metrics to collector by address:http://null:6188/ws/v1/timeline/metrics 2018-04-27 10:57:16.575 o.a.m.p.b.ProfileSplitterBolt Thread-6-splitterBolt-executor[7 7] [ERROR] Unexpected failure: message='null', tuple='{value=[123,***********Lot of integer values here in the logs end with****34, 48, 34, 125]}' org.json.simple.parser.ParseException: null at org.json.simple.parser.Yylex.yylex(Yylex.java:610) ~[stormjar.jar:?] at org.json.simple.parser.JSONParser.nextToken(JSONParser.java:269) ~[stormjar.jar:?] at org.json.simple.parser.JSONParser.parse(JSONParser.java:118) ~[stormjar.jar:?] at org.json.simple.parser.JSONParser.parse(JSONParser.java:81) ~[stormjar.jar:?] at org.json.simple.parser.JSONParser.parse(JSONParser.java:75) ~[stormjar.jar:?] at org.apache.metron.profiler.bolt.ProfileSplitterBolt.doExecute(ProfileSplitterBolt.java:108) ~[stormjar.jar:?] at org.apache.metron.profiler.bolt.ProfileSplitterBolt.execute(ProfileSplitterBolt.java:94) [stormjar.jar:?] at org.apache.storm.daemon.executor$fn__10250$tuple_action_fn__10252.invoke(executor.clj:730) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235] at org.apache.storm.daemon.executor$mk_task_receiver$fn__10171.invoke(executor.clj:462) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235] at org.apache.storm.disruptor$clojure_handler$reify__9685.onEvent(disruptor.clj:40) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235] at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235] at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235] at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235] at org.apache.storm.daemon.executor$fn__10250$fn__10263$fn__10316.invoke(executor.clj:849) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235] at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235] at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161] ****************************** Exception at SplitterBolt: *********************** Unexpected character (.) at position 885. at org.json.simple.parser.Yylex.yylex(Yylex.java:610) at org.json.simple.parser.JSONParser.nextToken(JSONParser.java:269) at org.json.simple.parser.JSONParser.parse(JSONParser.java:118) at org.json.simple.parser.JSONParser.parse(JSONParser.java:81) at org.json.simple.parser.JSONParser.parse(JSONParser.java:75) at org.apache.metron.profiler.bolt.ProfileSplitterBolt.doExecute(ProfileSplitterBolt.java:108) at org.apache.metron.profiler.bolt.ProfileSplitterBolt.execute(ProfileSplitterBolt.java:94) at org.apache.storm.daemon.executor$fn__10250$tuple_action_fn__10252.invoke(executor.clj:730) at org.apache.storm.daemon.executor$mk_task_receiver$fn__10171.invoke(executor.clj:462) at org.apache.storm.disruptor$clojure_handler$reify__9685.onEvent(disruptor.clj:40) at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472) at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451) at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) at org.apache.storm.daemon.executor$fn__10250$fn__10263$fn__10316.invoke(executor.clj:849) at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:748) Thanks, Anil.
