[
https://issues.apache.org/jira/browse/PHOENIX-4701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16452569#comment-16452569
]
James Taylor edited comment on PHOENIX-4701 at 4/25/18 4:13 PM:
----------------------------------------------------------------
Thanks for the response, [[email protected]] and [~samarthjain]. I think the
best approach would be to persist our [client
metrics|http://phoenix.apache.org/metrics.html] in the SYSTEM.LOG instead of
inventing a new mechanism. The metrics captures all the same information as
your QueryLogInfo (and much more), rolls all the information up to a single set
of metrics for each Phoenix statement (aggregating/merging parallel scans,
etc), and can emits a single log line (which could be written in a single
upsert statement). At SFDC, we emit this information in a layer above (and use
Splunk to produce nifty dashboard for monitoring), but this could easily be
emitted directly in Phoenix and go through your asynchronous write path (and
then use Phoenix queries to produce the same kind of dashboards). The only
piece would be to add the concept of a log level to each metric to enable
statically controlling which metrics are output.
was (Author: jamestaylor):
Thanks for the response, [[email protected]] and [~samarthjain]. I think the
best approach would be to persist our [client
metrics|http://phoenix.apache.org/metrics.html] in the SYSTEM.LOG instead of
inventing a new mechanism. The metrics captures all the same information as
your QueryLogInfo (and much more), rolls all the information up to a single set
of metrics for each Phoenix statement (aggregating/merging parallel scans,
etc), and can emits a single log line (which could be written in a single
upsert statement). At SFDC, we emit this information in a layer above, but this
could easily be emitted directly in Phoenix and go through your asynchronous
write path. The only piece would be to add the concept of a log level to each
metric to enable statically controlling which metrics are output.
> Improve schema of SYSTEM.LOG table
> ----------------------------------
>
> Key: PHOENIX-4701
> URL: https://issues.apache.org/jira/browse/PHOENIX-4701
> Project: Phoenix
> Issue Type: Bug
> Reporter: James Taylor
> Assignee: James Taylor
> Priority: Major
> Fix For: 4.14.0, 5.0.0
>
> Attachments: PHOENIX-4701_wip1.patch, PHOENIX-4701_wip2.patch
>
>
> If possible, the SYSTEM.LOG table would benefit greatly (3-5x perf gain)
> from being declared as immutable with a column encoding of 1 byte and a
> storage format of SINGLE_CELL_ARRAY_WITH_OFFSETS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
