Anil, Do you have any examples you can share about the use of profiler jars in your Java code?
Thanks, Stéphane From: Anil Donthireddy [mailto:[email protected]] Sent: Wednesday, April 24, 2019 19:25 To: DAVY Stephane OBS/CSO Cc: [email protected] Subject: RE: Various questions around profiler Hi Stephane. Please find my comments below in yello. From: [email protected] [mailto:[email protected]] Sent: Wednesday, April 24, 2019 1:30 AM To: [email protected] Subject: Various questions around profiler Hello everybody, I've been playing with Metron for a few weeks now, it is really a very exciting project and I'd like first to thanks all the contributors. I'm currently investigating around the use of profiler. I've tested it with the basic example of counting IP address as explained in the doc, and now it's time for questions: - In the various examples I've found, each profile computes only one value. Is it possible to do something like that: { "profiles": [ { "profile": "test", "foreach": "ip_src_addr", "onlyif": "exists(ip_src_addr)", "init": { "count": 0 , "sum_rcvd_bytes": 0}, "update": { "count": "count + 1", "sum_rcvd_bytes": "sum_rcvd_bytes + rcvdbyte" }, "result": "{'count': count, 'sum_rcvd_bytes': sum_rcvd_bytes}" } ] } >From the Stellar CLI, it seems to works fine, but when I try during data >ingest, I see no data coming into the profiler table. Please note that I've >waited for the 15mn, and that I have deleted data in the profiler table using >the "truncate_preserve" command in hbase. You need to have profile field in relust which goes to hbase table as below "result":{ "profile":"<valueToWriteToHabseForEachEntity>" } However I am not quite sure if you can send multiple values like count, sum_rcvd_bytes to profile table. - In case of issue, what is the right procedure to reinitialize all the profiler stack? I am not sure if there will be any issue with truncate preserve. Upto my understanding it should cause any issue but I never tried - The data in the profiler table can be pulled with Stellar functions which include some advanced features like statistics, cardinality, but is it possible to access all of this from a Java / Scala / any other language? Metron have profiler client implemented in Java, We should able to import the metron profiler client jars from the supported languages like Java/Scala to query the profiler statistics from code. We are doing it in Java. - The MaaS service seems to apply to the incoming data only, how is it possible to use it only on the aggregated profile data? As far I have used MAAS, I used it in enrichment, if you need to do something related to each event, then you can query profiler data in enrichment, store it to new column and apply MAAS in enrichment. If you want to apply MAAS at entity level, then first we need to check if MAAS function is available in class path of profiler execution. If it exists then we can call MAAS at possible places like init, update part of profiler definition or can even have nested profiler definition based on the use case. Maybe too much questions in the same mail? Thanks, Stéphane _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.
