Hi Gonçalo - What could be happening is that your Profiler is not tuned to be able to keep up with the amount of incoming data that you have. I would guess that the Profiler "keeps counting beyond that period of time" because it is still processing old data that is queued up.
- How much data is hitting your indexing topic (events per second)? - Have you tried to tune any of the Profiler settings? Here are some other things to keep in mind. - Check the Storm UI to see what the Profiler topology is doing and whether it is keeping up. - Increase the resources available to the Profiler so that it can keep up with the incoming data. - Use "event time" processing (set the "timestamp field") <https://metron.apache.org/current-book/metron-analytics/metron-profiler/index.html#timestampField> which will ensure that the profiles are being written using the timestamps contained within your data, rather than wall clock time. This will prevent your profiles from becoming skewed if processing falls behind. Best of luck On Mon, Jan 20, 2020 at 6:49 AM Gonçalo Pedras <goncalo.ped...@ctd.pt> wrote: > Hi, > > I’ve deployed Apache Metron with HDP 3.1 support provided by the GitHub > repository ( > https://github.com/apache/metron/blob/feature/METRON-2088-support-hdp-3.1 > ). > > I’ve some questions about the Profiler and somehow confused. I’m testing > the ASA parser and i’ve deployed two profiles: > > 1. Counting ip_src_addr. > > 2. Counting syslog_severity. > > The profiler properties have the default settings. > > I ran the parser last friday for a couple of seconds and it generated > about three thousand records. Today I ran the ‘PROFILER_GET’ in Stellar for > a ‘PROFILE_FIXED’ of 72 hours and I checked it against the Elasticsearch > index and I realised the counts don’t match. For exemple, for a specific IP > source “a” in that period of time I got 21 hits and in the result of > ‘PROFILER_GET’ returned a stream of results that make no sense to me. My > source of the ASA parser wasn’t sending any records to Kafka and somehow > the profiler managed to keep counting beyond that period of time. Where it > should be something like: [21], it returned [27, 27, 27, 54, 27, 27, …] . > My question is: > > · Is the Profiler working fine? And if it is, can someone explain > it to me? > > · And if it is not woking well, what is the problem, and how to > fix it? > > > > Thanks > > >
