Thanks for your great responses. Let me try and figure it out.
But what I wonder is there is some bug in the implementation of enrichment + profiling as I am seeing wage results. We should figureout what might be causing the iisue. I would like to know if there is any dependency in enrichment module and profiler module ? It seems like there is bug realted to dependency on enrichment and profiler. Thanks. Anil. From: Casey Stella [mailto:ceste...@gmail.com] Sent: Saturday, April 28, 2018 3:20 AM To: user@metron.apache.org Cc: Satish Abburi <satish.abb...@sstech.us> Subject: Re: Parse Exception at SplitterBolt in Profiler Topology: 2018-04-27 10:57:16.575 o.a.m.p.b.ProfileSplitterBolt Thread-6-splitterBolt-executor[7 7] [ERROR] Unexpected failure: message='null' It can be, depending on the use-case. In your case, PROFILE_GET('list-of-domains-per-user', ...) returns a list of the values stored in the 'list-of-domains-per-user', so it's a list of (probably) lists. In the REPL, you can make the PROFILE_GET call and see what it returns, but I'm suspecting it's a list of lists. You probably want to flatten that list of lists. I wish I could tell you that there's a FLATTEN or LIST_MERGE stellar function, but I don't see one (opportunity for contribution!). We can cobble one together, however, for you with reduce, like so: "init" : { "p" : "PROFILE_GET('list-of-domains-per-user', ...)", "lateral_init_set" : "REDUCE( p, (s, x) -> REDUCE(x, (a,b) -> LIST_ADD(a, b), s), [])" ... } [Aside for the reader: That is hideous, right? Definitely want that LIST_MERGE function. Any ambitious person want to get their feet wet in open source contribution and provide it? Just add it right next to LIST_ADD in DataStructureFunctions.java] Now, if you were willing for lateral_init_set to be a SET rather than a list, then we can do the following: "init" : { "p" : "PROFILE_GET('list-of-domains-per-user', ...)", "lateral_init_set" : "SET_MERGE(p)" ... } I suspect the second option would work out better since you don't care about duplicates in that list of domains. On Fri, Apr 27, 2018 at 16:44 Anil Donthireddy <anil.donthire...@sstech.us<mailto:anil.donthire...@sstech.us>> wrote: I would like to know if using PROFILE_GET is appropriate in profiler definition? I am seeing un-appropriate results. I need to use PROFILE_GET in enrichment configuration as well as in profiler definition. I am seeing very un-required results when I experiment having PROFILE_GET in enrichment config and profiler definition. Thanks, Anil. From: Casey Stella [mailto:ceste...@gmail.com<mailto:ceste...@gmail.com>] Sent: Saturday, April 28, 2018 12:26 AM To: user@metron.apache.org<mailto:user@metron.apache.org> Cc: Satish Abburi <satish.abb...@sstech.us<mailto:satish.abb...@sstech.us>> Subject: Re: Parse Exception at SplitterBolt in Profiler Topology: 2018-04-27 10:57:16.575 o.a.m.p.b.ProfileSplitterBolt Thread-6-splitterBolt-executor[7 7] [ERROR] Unexpected failure: message='null' That exception appears to me to be a problem in parsing the message coming into the profiler as opposed to having trouble parsing the profiler config. That list of integers are the raw characters in the message. It may be worthwhile to try to take the array of integers and try to turn them into a string (in python that could be as simple as ''.join(chr(i) for i in [123,***********Lot of integer values here in the logs end with****34, 48, 34, 125]) from the python REPL) to see exactly what was in the raw message. On Fri, Apr 27, 2018 at 2:40 PM Anil Donthireddy <anil.donthire...@sstech.us<mailto:anil.donthire...@sstech.us>> wrote: Hi, I defined a profile definition for my requirement to perform profile statistics. The calculation of profiler statistics for current profiling period will depend on the previous profile flushed stats. So, I am trying to read the previous stats during initialization ( “init” field in json ) part of profile for the current profiling period. The definition works fine when there are profile stats from the previous period, which means my profile definition working fine for first 15 minutes (my profile period is set to 15 mins). When profiler flushes the profile stats into hbase for first 15 minutes and starts for next period, I am getting the parse exception as seen in attached doc at Profiler topology and SplitterBolt. I have executed all lines of code in the stellar shell (instantiated with zookeeper config ./bin/stellat –z XXXX). It works fine without any issues in all the cases. Which means, I am able to fetch the previous profile stats, initialize for current record and execute all lines of code in profiler definition with expected result. I could not able to figure out what else is going wrong during SplitterBolt execution. I request someone to go through the attached logs and help me to troubleshoot the issue further. High Level Exception stack trace: *************************************** Exception seen at profiler topology logs ******************************* 2018-04-27 10:56:06.483 o.a.h.m.s.s.StormTimelineMetricsSink Thread-23 [WARN] Unable to send metrics to collector by address:http://null:6188/ws/v1/timeline/metrics 2018-04-27 10:57:16.575 o.a.m.p.b.ProfileSplitterBolt Thread-6-splitterBolt-executor[7 7] [ERROR] Unexpected failure: message='null', tuple='{value=[123,***********Lot of integer values here in the logs end with****34, 48, 34, 125]}' org.json.simple.parser.ParseException: null at org.json.simple.parser.Yylex.yylex(Yylex.java:610) ~[stormjar.jar:?] at org.json.simple.parser.JSONParser.nextToken(JSONParser.java:269) ~[stormjar.jar:?] at org.json.simple.parser.JSONParser.parse(JSONParser.java:118) ~[stormjar.jar:?] at org.json.simple.parser.JSONParser.parse(JSONParser.java:81) ~[stormjar.jar:?] at org.json.simple.parser.JSONParser.parse(JSONParser.java:75) ~[stormjar.jar:?] at org.apache.metron.profiler.bolt.ProfileSplitterBolt.doExecute(ProfileSplitterBolt.java:108) ~[stormjar.jar:?] at org.apache.metron.profiler.bolt.ProfileSplitterBolt.execute(ProfileSplitterBolt.java:94) [stormjar.jar:?] at org.apache.storm.daemon.executor$fn__10250$tuple_action_fn__10252.invoke(executor.clj:730) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235] at org.apache.storm.daemon.executor$mk_task_receiver$fn__10171.invoke(executor.clj:462) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235] at org.apache.storm.disruptor$clojure_handler$reify__9685.onEvent(disruptor.clj:40) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235] at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235] at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235] at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235] at org.apache.storm.daemon.executor$fn__10250$fn__10263$fn__10316.invoke(executor.clj:849) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235] at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235] at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161] ****************************** Exception at SplitterBolt: *********************** Unexpected character (.) at position 885. at org.json.simple.parser.Yylex.yylex(Yylex.java:610) at org.json.simple.parser.JSONParser.nextToken(JSONParser.java:269) at org.json.simple.parser.JSONParser.parse(JSONParser.java:118) at org.json.simple.parser.JSONParser.parse(JSONParser.java:81) at org.json.simple.parser.JSONParser.parse(JSONParser.java:75) at org.apache.metron.profiler.bolt.ProfileSplitterBolt.doExecute(ProfileSplitterBolt.java:108) at org.apache.metron.profiler.bolt.ProfileSplitterBolt.execute(ProfileSplitterBolt.java:94) at org.apache.storm.daemon.executor$fn__10250$tuple_action_fn__10252.invoke(executor.clj:730) at org.apache.storm.daemon.executor$mk_task_receiver$fn__10171.invoke(executor.clj:462) at org.apache.storm.disruptor$clojure_handler$reify__9685.onEvent(disruptor.clj:40) at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472) at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451) at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) at org.apache.storm.daemon.executor$fn__10250$fn__10263$fn__10316.invoke(executor.clj:849) at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:748) Thanks, Anil.
