[
https://issues.apache.org/jira/browse/HBASE-23938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17102395#comment-17102395
]
Viraj Jasani edited comment on HBASE-23938 at 5/8/20, 9:24 AM:
---------------------------------------------------------------
While I agree HDFS level persistence should be good, the reason why I prefer
system table is because user can easily retrieve all records on shell, use
ColumnValue filters, and HBase will take care of StoreFile management, rolling,
compaction etc.
In both cases (system table and direct HDFS), persisting complete data is going
to require extra IO anyways: HBase and HDFS levels or just HDFS level. With
system table, we are not concerned about FileSystem layer.
On the other hand, Mutate calls might be too slow but they can't be slow
indefinitely right? Maybe instead of persisting all of them right after keeping
them in ring buffer, we can have a cron running every 20 min which can look
into in memory ring buffer entries and persist them all to System table. What
do you think ?
[~anoop.hbase] [~apurtell] [~busbey]
Edit: Initial discussions reg persistence of logs is present in parent Jira:
HBASE-22978
was (Author: vjasani):
While I agree HDFS level persistence should be good, the reason why I prefer
system table is because user can easily retrieve all records on shell, use
ColumnValue filters, and HBase will take care of StoreFile management.
In both cases (system table and direct HDFS), persisting complete data is going
to require extra IO anyways: HBase and HDFS levels or just HDFS level. With
system table, we are not concerned about FileSystem layer.
On the other hand, Mutate calls might be too slow but they can't be slow
indefinitely right? Maybe instead of persisting all of them right after keeping
them in ring buffer, we can have a cron running every 20 min which can look
into in memory ring buffer entries and persist them all to System table. What
do you think ?
[~anoop.hbase] [~apurtell] [~busbey]
> Replicate slow/large RPC calls to HDFS
> --------------------------------------
>
> Key: HBASE-23938
> URL: https://issues.apache.org/jira/browse/HBASE-23938
> Project: HBase
> Issue Type: Sub-task
> Affects Versions: 3.0.0-alpha-1, 2.3.0, 1.7.0
> Reporter: Viraj Jasani
> Assignee: Viraj Jasani
> Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0
>
> Attachments: Screen Shot 2020-05-07 at 12.01.26 AM.png
>
>
> We should provide capability to replicate complete slow and large RPC logs to
> HDFS or create new system table in addition to Ring Buffer. This way we don't
> lose any of slow logs and operator can retrieve all the slow/large logs.
> Replicating logs to HDFS / creating new system table should be configurable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
