Hi Anil, Can you share your profile definition? For the STATS_BIN issue you're seeing, it seems like your bins are wrong, however you're using the pre-defined DECILE bin list, which is unit tested (and the tests are passing). It may be that the stats object isn't quite right, but I can't tell from this snippet alone. Your activity_count at least appears populated.
On Mon, Mar 11, 2019 at 6:16 PM Anil Donthireddy <anil.donthire...@sstech.us> wrote: > Hi, > > > > I am seeing below issues with Metron profiler. > > > > 1) Profiler is writing the duplicate data to Hbase. For example I > have inserted few records couple of days back to my test sensor topic and > written a profile that counts the number of records. I see that profiler is > still storing the same count continuously in to Hbase. As I observe when I > see the WARN message below in profiler log, I see the profiler inserts data > to Hbase that number of times (I am not sure if there is any relation) > > a. 2019-03-07 15:56:28.419 o.a.m.p.DefaultMessageDistributor > pool-17-thread-1 [WARN] Profile expired from active cache; > profile=demo_user_activity_count_ar, entity=Test1User5 > > 2) I am storing the STAT objects to Hbase and trying to calculate > the bin in another profiler by pulling Stats from Hbase. I see below > exception after a while > > a. 2019-03-11 11:24:51.039 o.a.m.p.DefaultProfileBuilder > pool-18-thread-1 [ERROR] Bad 'update' expression: error='Unable to parse: > if EXISTS(s) then STATS_BIN(s, activity_count, 'DECILE') else -1 due to: > Your bins must be non-decreasing with relevant variables > s=org.apache.metron.statistics.OnlineStatisticsProvider@808c9b26,activity_count=24', > expr='if EXISTS(s) then STATS_BIN(s, activity_count, 'DECILE') else -1', > profile='demo_user_activity_bin_ar', entity='Test2User4', > variables-available='[period.start, period, > enrichmentsplitterbolt.splitter.end.ts, bin, profile, > enrichmentsplitterbolt.splitter.begin.ts, is_alert, source.type, > threatintelsplitterbolt.splitter.end.ts, s, activity_count, > threatinteljoinbolt.joiner.ts, enrichmentjoinbolt.joiner.ts, period.end, > threatintelsplitterbolt.splitter.begin.ts, entity, timestamp]' > > b. org.apache.metron.stellar.dsl.ParseException: Unable to parse: if > EXISTS(s) then STATS_BIN(s, activity_count, 'DECILE') else -1 due to: Your > bins must be non-decreasing with relevant variables > s=org.apache.metron.statistics.OnlineStatisticsProvider@808c9b26,activity_count=24 > > > > I would like to understand what does it meant by “[WARN] Profile expired > from active cache; profile=demo_user_activity_count_ar, entity=Test1User5”, > when can this happen and how to resolve this? > > > > I would also like to understand how to resolve the issue “[ERROR] Bad > 'update' expression: error='Unable to parse: if EXISTS(s) then STATS_BIN(s, > activity_count, 'DECILE') else -1 due to: Your bins must be non-decreasing > with relevant variables > s=org.apache.metron.statistics.OnlineStatisticsProvider@808c9b26,activity_count=24', > expr='if EXISTS(s) then STATS_BIN(s, activity_count, 'DECILE') else -1', > profile='demo_user_activity_bin_ar', entity='Test2User4', > variables-available='[period.start, period, > enrichmentsplitterbolt.splitter.end.ts, bin, profile, > enrichmentsplitterbolt.splitter.begin.ts, is_alert, source.type, > threatintelsplitterbolt.splitter.end.ts, s, activity_count, > threatinteljoinbolt.joiner.ts, enrichmentjoinbolt.joiner.ts, period.end, > threatintelsplitterbolt.splitter.begin.ts, entity, timestamp]'” > > > > Looking forward for some help from Metron team or community to understand > and resolve the above issues. > > > > Thanking you. > > > > Thanks, > > Anil. >