Well, it seems that I have another issue right now:
[Stellar]>>> PROFILE_GET('simple_count','22.0.35.5', PROFILE_FIXED(30,
'MINUTES'))
[!] Unable to parse: PROFILE_GET('simple_count','22.0.35.5', PROFILE_FIXED(30,
'MINUTES')) due to: Unable to access table: profiler
It looks like a permission issue but I haven’t setup any security right now.
Any idea?
From: Nick Allen [mailto:[email protected]]
Sent: Thursday, April 25, 2019 14:29
To: [email protected]
Subject: Re: Various questions around profiler
Try querying for that record in the REPL with PROFILE_GET and then using
MAP_GET on it. It will likely work as you expect.
value := PROFILE_GET("test", ...)
MAP_GET("sum_rcvd_bytes", value)
On Thu, Apr 25, 2019 at 2:38 AM
<[email protected]<mailto:[email protected]>> wrote:
Hello Nick,
Thanks for your answer. Well, I don’t know what was the issue, but after a
restart profiling data went to HBASE. With the configuration below ("result":
"{'count': count, 'sum_rcvd_bytes': sum_rcvd_bytes}"), I get something like
this in HBASE:
value=\x01\x00java.util.HashMa\xF0\x01\x02\x03\x01coun\xF4\x02\x8A\x03\x03\x0
\x00\x00\x00\x00\x00\x1Ab\x18 1sum_rcvd_byte\xF3\x09\xDC\x9D\xC3\x09
So, not sure that it is usable. Do not hesitate to share your upcoming work on
all of this ☺
Cordialement / Best regards
Stéphane
From: Nick Allen [mailto:[email protected]<mailto:[email protected]>]
Sent: Wednesday, April 24, 2019 15:08
To: [email protected]<mailto:[email protected]>
Subject: Re: Various questions around profiler
> In the various examples I’ve found, each profile computes only one value. Is
> it possible to do something like that:
Currently, each profile produces only a single value. But I've had the same
thought as you. It would be nice to be able to persist multiple values in one
shot. Feel free to create a JIRA if you have thoughts on how that might work.
> From the Stellar CLI, it seems to works fine, but when I try during data
> ingest, I see no data coming into the profiler table. Please note that I’ve
> waited for the 15mn, and that I have deleted data in the profiler table using
> the “truncate_preserve” command in hbase.
(1) Have you changed any of the default Profiler settings like the profile
period?
(2) In the Stellar REPL (CLI) did you test with the real telemetry from Kafka
or just mock messages? Retrieve some of the real streaming telemetry using
`KAFKA_GET` and test that against your profile definition. Maybe there is a
difference between the mock data that you tested with versus the real streaming
data.
(3) To debug further, in the Storm UI go to the Profiler topology, then turn on
DEBUG logging for the package "org.apache.metron.profiler". Leave this extra
logging running for at least the length of your profile period (by default 15
minutes) and see what the logs show.
> The data in the profiler table can be pulled with Stellar functions which
> include some advanced features like statistics, cardinality, but is it
> possible to access all of this from a Java / Scala / any other language?
There is an internal class that the PROFILE_GET Stellar function uses to
actually retrieve the profiles from HBase. You could possibly use this as a
Java API. But this is definitely not ideal.
I have done some proof-of-concept work to store the profile data using Phoenix
in Hbase, which would allow you to access all of the profile data using
SQL/JDBC/ODBC. That would allow you to retrieve that data with the tooling of
your choice, even including something like a Zeppelin Notebook. When I have
some cycles, I'd like to share that with the community.
On Wed, Apr 24, 2019 at 4:30 AM
<[email protected]<mailto:[email protected]>> wrote:
Hello everybody,
I’ve been playing with Metron for a few weeks now, it is really a very exciting
project and I’d like first to thanks all the contributors. I’m currently
investigating around the use of profiler. I’ve tested it with the basic example
of counting IP address as explained in the doc, and now it’s time for questions:
- In the various examples I’ve found, each profile computes only one
value. Is it possible to do something like that:
{
"profiles": [
{
"profile": "test",
"foreach": "ip_src_addr",
"onlyif": "exists(ip_src_addr)",
"init": { "count": 0 , "sum_rcvd_bytes": 0},
"update": { "count": "count + 1",
"sum_rcvd_bytes": "sum_rcvd_bytes + rcvdbyte" },
"result": "{'count': count, 'sum_rcvd_bytes': sum_rcvd_bytes}"
}
]
}
From the Stellar CLI, it seems to works fine, but when I try during data
ingest, I see no data coming into the profiler table. Please note that I’ve
waited for the 15mn, and that I have deleted data in the profiler table using
the “truncate_preserve” command in hbase.
- In case of issue, what is the right procedure to reinitialize all
the profiler stack?
- The data in the profiler table can be pulled with Stellar functions
which include some advanced features like statistics, cardinality, but is it
possible to access all of this from a Java / Scala / any other language?
- The MaaS service seems to apply to the incoming data only, how is it
possible to use it only on the aggregated profile data?
Maybe too much questions in the same mail?
Thanks,
Stéphane
_________________________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce
message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou
falsifie. Merci.
This message and its attachments may contain confidential or privileged
information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete
this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been
modified, changed or falsified.
Thank you.
_________________________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce
message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou
falsifie. Merci.
This message and its attachments may contain confidential or privileged
information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete
this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been
modified, changed or falsified.
Thank you.
_________________________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce
message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou
falsifie. Merci.
This message and its attachments may contain confidential or privileged
information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete
this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been
modified, changed or falsified.
Thank you.
