[
https://issues.apache.org/jira/browse/HBASE-26345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427647#comment-17427647
]
Viraj Jasani edited comment on HBASE-26345 at 10/12/21, 3:00 PM:
-----------------------------------------------------------------
Thanks [~apurtell] for the review!
{quote}The doc is missing a minor point about ring buffers in that it is ok and
expected to begin overwriting entries after a wrap-around, this is how they
re-use the oldest entries for storing the newest, but it can be inferred.
{quote}
Agree, even though it is inferred, I think adding this extra line is definitely
beneficial. Will add this before commit.
{quote}If I didn't know anything about this feature and I was reading the doc,
I would wonder if the slow log persistence stuff can be reused for another NQ
and how I might go about doing that.
{quote}
Sounds good, let me add this one additional paragraph:
_If the optional persistence layer is required, the NamedQueueService
implementation can use an additional service layer to provide the usecase
specific permanent storage and add the records to it after they are inserted in
the ring-buffer queue. Slow/LargeLog usecase prefers system table to persist
all entries, the system table access is provided by SlowLogPersistentService
(HBASE-23938). The similar implementation can be extended for other potential
usecases. Persistence service layer is responsible for permanent storage of
records unlike the in-memory ring-buffer queue where the queue is used to
access the recent records only and hence, the persistence service can use any
storage - HDFS FileSystem, S3 like object stores, HBase system tables,
Zookeeper, Bookkeeper etc. Hence, this service layer is flexible and new
usecase should provide their own persistence layer with the custom logic.
SlowLogPersistentService use hbase:slowlog system table. The benefit in the
case of using system table is that we can reuse hbase-client Table APIs for
CRUD operations and operators can scan the table using shell commands._
was (Author: vjasani):
Thanks [~apurtell] for the review!
{quote}The doc is missing a minor point about ring buffers in that it is ok and
expected to begin overwriting entries after a wrap-around, this is how they
re-use the oldest entries for storing the newest, but it can be inferred.
{quote}
Agree, even though it is inferred, I think adding this extra line is definitely
beneficial. Will add this before commit.
{quote}If I didn't know anything about this feature and I was reading the doc,
I would wonder if the slow log persistence stuff can be reused for another NQ
and how I might go about doing that.
{quote}
Sounds good, let me add this one additional paragraph:
_If the optional persistence layer is required, the NamedQueueService
implementation can use an additional service layer to provide the usecase
specific permanent storage and add the records to it after they are inserted in
the ring-buffer queue. Slow/LargeLog usecase prefers system table to persist
all entries, the system table access is provided by SlowLogPersistentService
(HBASE-23938). The similar implementation can be extended for other potential
usecases. Persistence service layer is responsible for permanent storage of
records unlike the in-memory ring-buffer queue where the queue is used to
access the recent records only and hence, the persistence service can use any
storage - HDFS FileSystem, S3 like object stores, HBase system tables,
Zookeeper, Bookkeeper etc. Hence, this service layer is flexible and new
usecase should provide their own persistence layer with the custom logic. If
the persistence preference is HBase system table, SlowLogPersistentService
already provides pluggable access to hbase:slowlog system table. The benefit in
the case of using system table is that we can reuse hbase-client Table APIs for
CRUD operations and operators can scan the table using shell commands._
> Design doc for HBase NamedQueue framework
> -----------------------------------------
>
> Key: HBASE-26345
> URL: https://issues.apache.org/jira/browse/HBASE-26345
> Project: HBase
> Issue Type: Task
> Reporter: Viraj Jasani
> Assignee: Viraj Jasani
> Priority: Major
> Fix For: 3.0.0-alpha-2
>
> Attachments: Online NamedQueue framework with optional persistence
> layer.pdf
>
>
> We have a couple of usecases already being served by the NamedQueue
> framework. It would be good to also provide a design doc to provide the high
> level system components view and let other usecases be onboarded with ease.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
