(1) Did you launch the REPL with the -z <ZOOKEEPER> option?
(2) What user are you running the REPL as?
(3) Can you scan the table using the 'hbase shell'? Something like this...
echo "scan 'profiler'" | hbase shell
(4) Can you show the full session from launching the REPL, running the
PROFILE_GET, and including the full stack trace and error?
On Thu, Apr 25, 2019 at 9:44 AM <[email protected]> wrote:
> Well, it seems that I have another issue right now:
>
>
>
> [Stellar]>>> PROFILE_GET('simple_count','22.0.35.5', PROFILE_FIXED(30,
> 'MINUTES'))
>
> [!] Unable to parse: PROFILE_GET('simple_count','22.0.35.5',
> PROFILE_FIXED(30, 'MINUTES')) due to: *Unable to access table: profiler*
>
>
>
> It looks like a permission issue but I haven’t setup any security right
> now.
>
>
>
> Any idea?
>
>
>
>
>
> *From:* Nick Allen [mailto:[email protected]]
> *Sent:* Thursday, April 25, 2019 14:29
> *To:* [email protected]
> *Subject:* Re: Various questions around profiler
>
>
>
>
>
> Try querying for that record in the REPL with PROFILE_GET and then using
> MAP_GET on it. It will likely work as you expect.
>
>
>
> value := PROFILE_GET("test", ...)
>
> MAP_GET("sum_rcvd_bytes", value)
>
>
>
>
>
>
>
> On Thu, Apr 25, 2019 at 2:38 AM <[email protected]> wrote:
>
> Hello Nick,
>
>
>
> Thanks for your answer. Well, I don’t know what was the issue, but after a
> restart profiling data went to HBASE. With the configuration below
> ("result": "{'count': count, 'sum_rcvd_bytes': sum_rcvd_bytes}"), I get
> something like this in HBASE:
>
>
>
> value=\x01\x00java.util.HashMa\xF0\x01\x02\x03\x01coun\xF4\x02\x8A\x03\x03\x0
> \x00\x00\x00\x00\x00\x1Ab\x18
> 1sum_rcvd_byte\xF3\x09\xDC\x9D\xC3\x09
>
>
>
> So, not sure that it is usable. Do not hesitate to share your upcoming
> work on all of this J
>
>
>
> Cordialement / Best regards
>
>
>
> Stéphane
>
>
>
> *From:* Nick Allen [mailto:[email protected]]
> *Sent:* Wednesday, April 24, 2019 15:08
> *To:* [email protected]
> *Subject:* Re: Various questions around profiler
>
>
>
> > In the various examples I’ve found, each profile computes only one
> value. Is it possible to do something like that:
>
>
>
> Currently, each profile produces only a single value. But I've had the
> same thought as you. It would be nice to be able to persist multiple
> values in one shot. Feel free to create a JIRA if you have thoughts on how
> that might work.
>
>
>
>
>
> > From the Stellar CLI, it seems to works fine, but when I try during data
> ingest, I see no data coming into the profiler table. Please note that I’ve
> waited for the 15mn, and that I have deleted data in the profiler table
> using the “truncate_preserve” command in hbase.
>
>
>
> (1) Have you changed any of the default Profiler settings like the profile
> period?
>
>
>
> (2) In the Stellar REPL (CLI) did you test with the real telemetry from
> Kafka or just mock messages? Retrieve some of the real streaming telemetry
> using `KAFKA_GET` and test that against your profile definition. Maybe
> there is a difference between the mock data that you tested with versus the
> real streaming data.
>
>
>
> (3) To debug further, in the Storm UI go to the Profiler topology, then
> turn on DEBUG logging for the package "org.apache.metron.profiler". Leave
> this extra logging running for at least the length of your profile period
> (by default 15 minutes) and see what the logs show.
>
>
>
>
>
> > The data in the profiler table can be pulled with Stellar functions
> which include some advanced features like statistics, cardinality, but is
> it possible to access all of this from a Java / Scala / any other language?
>
>
>
> There is an internal class that the PROFILE_GET Stellar function uses to
> actually retrieve the profiles from HBase. You could possibly use this as
> a Java API. But this is definitely not ideal.
>
>
>
> I have done some proof-of-concept work to store the profile data using
> Phoenix in Hbase, which would allow you to access all of the profile data
> using SQL/JDBC/ODBC. That would allow you to retrieve that data with the
> tooling of your choice, even including something like a Zeppelin Notebook.
> When I have some cycles, I'd like to share that with the community.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Wed, Apr 24, 2019 at 4:30 AM <[email protected]> wrote:
>
> Hello everybody,
>
>
>
> I’ve been playing with Metron for a few weeks now, it is really a very
> exciting project and I’d like first to thanks all the contributors. I’m
> currently investigating around the use of profiler. I’ve tested it with the
> basic example of counting IP address as explained in the doc, and now it’s
> time for questions:
>
> - In the various examples I’ve found, each profile computes only
> one value. Is it possible to do something like that:
>
> {
>
> "profiles": [
>
> {
>
> "profile": "test",
>
> "foreach": "ip_src_addr",
>
> "onlyif": "exists(ip_src_addr)",
>
> "init": { "count": 0 , "sum_rcvd_bytes": 0},
>
> "update": { "count": "count + 1",
>
> "sum_rcvd_bytes": "sum_rcvd_bytes + rcvdbyte" },
>
> "result": "{'count': count, 'sum_rcvd_bytes': sum_rcvd_bytes}"
>
> }
>
> ]
>
> }
>
>
>
> From the Stellar CLI, it seems to works fine, but when I try during data
> ingest, I see no data coming into the profiler table. Please note that I’ve
> waited for the 15mn, and that I have deleted data in the profiler table
> using the “truncate_preserve” command in hbase.
>
> - In case of issue, what is the right procedure to reinitialize
> all the profiler stack?
>
> - The data in the profiler table can be pulled with Stellar
> functions which include some advanced features like statistics,
> cardinality, but is it possible to access all of this from a Java / Scala /
> any other language?
>
> - The MaaS service seems to apply to the incoming data only, how
> is it possible to use it only on the aggregated profile data?
>
>
>
>
>
> Maybe too much questions in the same mail?
>
>
>
> Thanks,
>
>
>
> Stéphane
>
> _________________________________________________________________________________________________________________________
>
>
>
> Ce message et ses pieces jointes peuvent contenir des informations
> confidentielles ou privilegiees et ne doivent donc
>
> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu
> ce message par erreur, veuillez le signaler
>
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
> electroniques etant susceptibles d'alteration,
>
> Orange decline toute responsabilite si ce message a ete altere, deforme ou
> falsifie. Merci.
>
>
>
> This message and its attachments may contain confidential or privileged
> information that may be protected by law;
>
> they should not be distributed, used or copied without authorisation.
>
> If you have received this email in error, please notify the sender and delete
> this message and its attachments.
>
> As emails may be altered, Orange is not liable for messages that have been
> modified, changed or falsified.
>
> Thank you.
>
> _________________________________________________________________________________________________________________________
>
>
>
> Ce message et ses pieces jointes peuvent contenir des informations
> confidentielles ou privilegiees et ne doivent donc
>
> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu
> ce message par erreur, veuillez le signaler
>
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
> electroniques etant susceptibles d'alteration,
>
> Orange decline toute responsabilite si ce message a ete altere, deforme ou
> falsifie. Merci.
>
>
>
> This message and its attachments may contain confidential or privileged
> information that may be protected by law;
>
> they should not be distributed, used or copied without authorisation.
>
> If you have received this email in error, please notify the sender and delete
> this message and its attachments.
>
> As emails may be altered, Orange is not liable for messages that have been
> modified, changed or falsified.
>
> Thank you.
>
> _________________________________________________________________________________________________________________________
>
> Ce message et ses pieces jointes peuvent contenir des informations
> confidentielles ou privilegiees et ne doivent donc
> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu
> ce message par erreur, veuillez le signaler
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
> electroniques etant susceptibles d'alteration,
> Orange decline toute responsabilite si ce message a ete altere, deforme ou
> falsifie. Merci.
>
> This message and its attachments may contain confidential or privileged
> information that may be protected by law;
> they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and delete
> this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been
> modified, changed or falsified.
> Thank you.
>
>
