OK, I finally found the problem when pasting the whole error stack in the mail:
Caused by: java.lang.RuntimeException: Unexpected version format: 11.0.3 The first java in my path is java 11. When switching to Java 8 it worked correctly Stéphane From: Nick Allen [mailto:[email protected]] Sent: Thursday, April 25, 2019 17:35 To: [email protected] Subject: Re: Various questions around profiler (1) Did you launch the REPL with the -z <ZOOKEEPER> option? (2) What user are you running the REPL as? (3) Can you scan the table using the 'hbase shell'? Something like this... echo "scan 'profiler'" | hbase shell (4) Can you show the full session from launching the REPL, running the PROFILE_GET, and including the full stack trace and error? On Thu, Apr 25, 2019 at 9:44 AM <[email protected]<mailto:[email protected]>> wrote: Well, it seems that I have another issue right now: [Stellar]>>> PROFILE_GET('simple_count','22.0.35.5', PROFILE_FIXED(30, 'MINUTES')) [!] Unable to parse: PROFILE_GET('simple_count','22.0.35.5', PROFILE_FIXED(30, 'MINUTES')) due to: Unable to access table: profiler It looks like a permission issue but I haven’t setup any security right now. Any idea? From: Nick Allen [mailto:[email protected]<mailto:[email protected]>] Sent: Thursday, April 25, 2019 14:29 To: [email protected]<mailto:[email protected]> Subject: Re: Various questions around profiler Try querying for that record in the REPL with PROFILE_GET and then using MAP_GET on it. It will likely work as you expect. value := PROFILE_GET("test", ...) MAP_GET("sum_rcvd_bytes", value) On Thu, Apr 25, 2019 at 2:38 AM <[email protected]<mailto:[email protected]>> wrote: Hello Nick, Thanks for your answer. Well, I don’t know what was the issue, but after a restart profiling data went to HBASE. With the configuration below ("result": "{'count': count, 'sum_rcvd_bytes': sum_rcvd_bytes}"), I get something like this in HBASE: value=\x01\x00java.util.HashMa\xF0\x01\x02\x03\x01coun\xF4\x02\x8A\x03\x03\x0 \x00\x00\x00\x00\x00\x1Ab\x18 1sum_rcvd_byte\xF3\x09\xDC\x9D\xC3\x09 So, not sure that it is usable. Do not hesitate to share your upcoming work on all of this ☺ Cordialement / Best regards Stéphane From: Nick Allen [mailto:[email protected]<mailto:[email protected]>] Sent: Wednesday, April 24, 2019 15:08 To: [email protected]<mailto:[email protected]> Subject: Re: Various questions around profiler > In the various examples I’ve found, each profile computes only one value. Is > it possible to do something like that: Currently, each profile produces only a single value. But I've had the same thought as you. It would be nice to be able to persist multiple values in one shot. Feel free to create a JIRA if you have thoughts on how that might work. > From the Stellar CLI, it seems to works fine, but when I try during data > ingest, I see no data coming into the profiler table. Please note that I’ve > waited for the 15mn, and that I have deleted data in the profiler table using > the “truncate_preserve” command in hbase. (1) Have you changed any of the default Profiler settings like the profile period? (2) In the Stellar REPL (CLI) did you test with the real telemetry from Kafka or just mock messages? Retrieve some of the real streaming telemetry using `KAFKA_GET` and test that against your profile definition. Maybe there is a difference between the mock data that you tested with versus the real streaming data. (3) To debug further, in the Storm UI go to the Profiler topology, then turn on DEBUG logging for the package "org.apache.metron.profiler". Leave this extra logging running for at least the length of your profile period (by default 15 minutes) and see what the logs show. > The data in the profiler table can be pulled with Stellar functions which > include some advanced features like statistics, cardinality, but is it > possible to access all of this from a Java / Scala / any other language? There is an internal class that the PROFILE_GET Stellar function uses to actually retrieve the profiles from HBase. You could possibly use this as a Java API. But this is definitely not ideal. I have done some proof-of-concept work to store the profile data using Phoenix in Hbase, which would allow you to access all of the profile data using SQL/JDBC/ODBC. That would allow you to retrieve that data with the tooling of your choice, even including something like a Zeppelin Notebook. When I have some cycles, I'd like to share that with the community. On Wed, Apr 24, 2019 at 4:30 AM <[email protected]<mailto:[email protected]>> wrote: Hello everybody, I’ve been playing with Metron for a few weeks now, it is really a very exciting project and I’d like first to thanks all the contributors. I’m currently investigating around the use of profiler. I’ve tested it with the basic example of counting IP address as explained in the doc, and now it’s time for questions: - In the various examples I’ve found, each profile computes only one value. Is it possible to do something like that: { "profiles": [ { "profile": "test", "foreach": "ip_src_addr", "onlyif": "exists(ip_src_addr)", "init": { "count": 0 , "sum_rcvd_bytes": 0}, "update": { "count": "count + 1", "sum_rcvd_bytes": "sum_rcvd_bytes + rcvdbyte" }, "result": "{'count': count, 'sum_rcvd_bytes': sum_rcvd_bytes}" } ] } From the Stellar CLI, it seems to works fine, but when I try during data ingest, I see no data coming into the profiler table. Please note that I’ve waited for the 15mn, and that I have deleted data in the profiler table using the “truncate_preserve” command in hbase. - In case of issue, what is the right procedure to reinitialize all the profiler stack? - The data in the profiler table can be pulled with Stellar functions which include some advanced features like statistics, cardinality, but is it possible to access all of this from a Java / Scala / any other language? - The MaaS service seems to apply to the incoming data only, how is it possible to use it only on the aggregated profile data? Maybe too much questions in the same mail? Thanks, Stéphane _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.
