[jira] [Commented] (HBASE-20951) Ratis LogService backed WALs
[
https://issues.apache.org/jira/browse/HBASE-20951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17553775#comment-17553775
]
Josh Elser commented on HBASE-20951:
{quote}I am thinking of resolving this and all subtasks as WontFix or Abandoned.
{quote}
Yeah, I think a "Later" is appropriate. Let me do this.
> Ratis LogService backed WALs
>
>
> Key: HBASE-20951
> URL: https://issues.apache.org/jira/browse/HBASE-20951
> Project: HBase
> Issue Type: New Feature
> Components: wal
>Reporter: Josh Elser
>Priority: Major
>
> Umbrella issue for the Ratis+WAL work:
> Design doc:
> [https://docs.google.com/document/d/1Su5py_T5Ytfh9RoTTX2s20KbSJwBHVxbO7ge5ORqbCk/edit#|https://docs.google.com/document/d/1Su5py_T5Ytfh9RoTTX2s20KbSJwBHVxbO7ge5ORqbCk/edit]
> The (over-simplified) goal is to re-think the current WAL APIs we have now,
> ensure that they are de-coupled from the notion of being backed by HDFS, swap
> the current implementations over to the new API, and then wire up the Ratis
> LogService to the new WAL API.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-20951) Ratis LogService backed WALs
[ https://issues.apache.org/jira/browse/HBASE-20951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17553189#comment-17553189 ] Andrew Kyle Purtell commented on HBASE-20951: - How likely is this to be resurrected? I am thinking of resolving this and all subtasks as WontFix or Abandoned. > Ratis LogService backed WALs > > > Key: HBASE-20951 > URL: https://issues.apache.org/jira/browse/HBASE-20951 > Project: HBase > Issue Type: New Feature > Components: wal >Reporter: Josh Elser >Priority: Major > > Umbrella issue for the Ratis+WAL work: > Design doc: > [https://docs.google.com/document/d/1Su5py_T5Ytfh9RoTTX2s20KbSJwBHVxbO7ge5ORqbCk/edit#|https://docs.google.com/document/d/1Su5py_T5Ytfh9RoTTX2s20KbSJwBHVxbO7ge5ORqbCk/edit] > The (over-simplified) goal is to re-think the current WAL APIs we have now, > ensure that they are de-coupled from the notion of being backed by HDFS, swap > the current implementations over to the new API, and then wire up the Ratis > LogService to the new WAL API. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (HBASE-20951) Ratis LogService backed WALs
[
https://issues.apache.org/jira/browse/HBASE-20951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16796712#comment-16796712
]
Allan Yang commented on HBASE-20951:
{quote}
Meaning, we would be able to keep everything inside of HBase already.
{quote}
Thanks for your reply, I got it, [~elserj]!
> Ratis LogService backed WALs
>
>
> Key: HBASE-20951
> URL: https://issues.apache.org/jira/browse/HBASE-20951
> Project: HBase
> Issue Type: New Feature
> Components: wal
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
>
> Umbrella issue for the Ratis+WAL work:
> Design doc:
> [https://docs.google.com/document/d/1Su5py_T5Ytfh9RoTTX2s20KbSJwBHVxbO7ge5ORqbCk/edit#|https://docs.google.com/document/d/1Su5py_T5Ytfh9RoTTX2s20KbSJwBHVxbO7ge5ORqbCk/edit]
> The (over-simplified) goal is to re-think the current WAL APIs we have now,
> ensure that they are de-coupled from the notion of being backed by HDFS, swap
> the current implementations over to the new API, and then wire up the Ratis
> LogService to the new WAL API.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20951) Ratis LogService backed WALs
[ https://issues.apache.org/jira/browse/HBASE-20951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16796115#comment-16796115 ] Josh Elser commented on HBASE-20951: bq. The goal of this project is to separate WAL from HDFS and store it into a log service to achieve lower cost and flexibility in cloud compare to HDFS(decommission is panic as said in the doc). But, what is the storage for this log service? Local disks in EC2 instance? Correct. The "WAL data" would really become the RAFT log which is stored on local disk. bq. How to achieve the goal of flexibility itself？Does it mean that we need to maintain another cluster for this log service? Nope. The "novel" part of Ratis is that it is a library that we would embed, not another service that we need to run. Meaning, we would be able to keep everything inside of HBase already. bq. isn't it panic too when decommissioning? Yes, the big difference operationally is that we would have to be much more careful about losing nodes, whereas we don't really care if a lot of RS disappear all at once. If we lose too many RS, we would potentially be unable to read a recently written WAL. Does that help? Happy to answer any questions :) > Ratis LogService backed WALs > > > Key: HBASE-20951 > URL: https://issues.apache.org/jira/browse/HBASE-20951 > Project: HBase > Issue Type: New Feature > Components: wal >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > > Umbrella issue for the Ratis+WAL work: > Design doc: > [https://docs.google.com/document/d/1Su5py_T5Ytfh9RoTTX2s20KbSJwBHVxbO7ge5ORqbCk/edit#|https://docs.google.com/document/d/1Su5py_T5Ytfh9RoTTX2s20KbSJwBHVxbO7ge5ORqbCk/edit] > The (over-simplified) goal is to re-think the current WAL APIs we have now, > ensure that they are de-coupled from the notion of being backed by HDFS, swap > the current implementations over to the new API, and then wire up the Ratis > LogService to the new WAL API. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20951) Ratis LogService backed WALs
[ https://issues.apache.org/jira/browse/HBASE-20951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795593#comment-16795593 ] Allan Yang commented on HBASE-20951: [~elserj], I have a question for the ratis based log service. The goal of this project is to separate WAL from HDFS and store it into a log service to achieve lower cost and flexibility in cloud compare to HDFS(decommission is panic as said in the doc). But, what is the storage for this log service? Local disks in EC2 instance? How to achieve the goal of flexibility itself？Does it mean that we need to maintain another cluster for this log service? isn't it panic too when decommissioning? > Ratis LogService backed WALs > > > Key: HBASE-20951 > URL: https://issues.apache.org/jira/browse/HBASE-20951 > Project: HBase > Issue Type: New Feature > Components: wal >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > > Umbrella issue for the Ratis+WAL work: > Design doc: > [https://docs.google.com/document/d/1Su5py_T5Ytfh9RoTTX2s20KbSJwBHVxbO7ge5ORqbCk/edit#|https://docs.google.com/document/d/1Su5py_T5Ytfh9RoTTX2s20KbSJwBHVxbO7ge5ORqbCk/edit] > The (over-simplified) goal is to re-think the current WAL APIs we have now, > ensure that they are de-coupled from the notion of being backed by HDFS, swap > the current implementations over to the new API, and then wire up the Ratis > LogService to the new WAL API. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20951) Ratis LogService backed WALs
[ https://issues.apache.org/jira/browse/HBASE-20951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16630901#comment-16630901 ] Josh Elser commented on HBASE-20951: FYI HBASE-20952 was spun out into a standalone issue (from a child issue) to better support making small changes to the WAL API. Linked it off of this issue. > Ratis LogService backed WALs > > > Key: HBASE-20951 > URL: https://issues.apache.org/jira/browse/HBASE-20951 > Project: HBase > Issue Type: New Feature > Components: wal >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > > Umbrella issue for the Ratis+WAL work: > Design doc: > [https://docs.google.com/document/d/1Su5py_T5Ytfh9RoTTX2s20KbSJwBHVxbO7ge5ORqbCk/edit#|https://docs.google.com/document/d/1Su5py_T5Ytfh9RoTTX2s20KbSJwBHVxbO7ge5ORqbCk/edit] > The (over-simplified) goal is to re-think the current WAL APIs we have now, > ensure that they are de-coupled from the notion of being backed by HDFS, swap > the current implementations over to the new API, and then wire up the Ratis > LogService to the new WAL API. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20951) Ratis LogService backed WALs
[
https://issues.apache.org/jira/browse/HBASE-20951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16582640#comment-16582640
]
Josh Elser commented on HBASE-20951:
{quote}Can we make it generic? Then we can have following, it still provides
the flexibility to support FSBased system (maybe not only HDFS if on the
Cloud?), and interface is cleaner.
{quote}
Sounds like a good idea to me, [~reidchan]. Thanks for the suggestion!
> Ratis LogService backed WALs
>
>
> Key: HBASE-20951
> URL: https://issues.apache.org/jira/browse/HBASE-20951
> Project: HBase
> Issue Type: New Feature
> Components: wal
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
>
> Umbrella issue for the Ratis+WAL work:
> Design doc:
> [https://docs.google.com/document/d/1Su5py_T5Ytfh9RoTTX2s20KbSJwBHVxbO7ge5ORqbCk/edit#|https://docs.google.com/document/d/1Su5py_T5Ytfh9RoTTX2s20KbSJwBHVxbO7ge5ORqbCk/edit]
> The (over-simplified) goal is to re-think the current WAL APIs we have now,
> ensure that they are de-coupled from the notion of being backed by HDFS, swap
> the current implementations over to the new API, and then wire up the Ratis
> LogService to the new WAL API.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20951) Ratis LogService backed WALs
[
https://issues.apache.org/jira/browse/HBASE-20951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16580667#comment-16580667
]
Reid Chan commented on HBASE-20951:
---
Took a look at the {{WALInfo}}.
Can we make it generic? Then we can have following, it still provides the
flexibility to support FSBased system (maybe not only HDFS if on the Cloud?),
and interface is cleaner.
{code} T getPath(); {code}
Not sure if it is too early to comment, pardon me. And thanks the info,
[~elserj]!
> Ratis LogService backed WALs
>
>
> Key: HBASE-20951
> URL: https://issues.apache.org/jira/browse/HBASE-20951
> Project: HBase
> Issue Type: New Feature
> Components: wal
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
>
> Umbrella issue for the Ratis+WAL work:
> Design doc:
> [https://docs.google.com/document/d/1Su5py_T5Ytfh9RoTTX2s20KbSJwBHVxbO7ge5ORqbCk/edit#|https://docs.google.com/document/d/1Su5py_T5Ytfh9RoTTX2s20KbSJwBHVxbO7ge5ORqbCk/edit]
> The (over-simplified) goal is to re-think the current WAL APIs we have now,
> ensure that they are de-coupled from the notion of being backed by HDFS, swap
> the current implementations over to the new API, and then wire up the Ratis
> LogService to the new WAL API.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20951) Ratis LogService backed WALs
[ https://issues.apache.org/jira/browse/HBASE-20951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16580488#comment-16580488 ] Josh Elser commented on HBASE-20951: Those with a close eye might stumble onto https://github.com/hbase-temp-wal-refactor/hbase/tree/wal-refactor. To prevent anyone from thinking something heinous is going on: between myself, Ted, Ankit, and Sergey who were trying to get this effort off the ground, we have been struggling to get the ball rolling. To try to iterate faster and get something of better quality into Apache for an initial API/interface review, we've spun out this organization/fork. Please rest assured that it is not a fork to do feature development outside of Apache, nor is it an attempt to circumvent those in Apache who have expressed interest in working on this. It is only a place for us to iterate on potential API changes more quickly, in order to bring a better first patch into Apache for review by the community. Please reach out to me if there are any concerns about this. > Ratis LogService backed WALs > > > Key: HBASE-20951 > URL: https://issues.apache.org/jira/browse/HBASE-20951 > Project: HBase > Issue Type: New Feature > Components: wal >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > > Umbrella issue for the Ratis+WAL work: > Design doc: > [https://docs.google.com/document/d/1Su5py_T5Ytfh9RoTTX2s20KbSJwBHVxbO7ge5ORqbCk/edit#|https://docs.google.com/document/d/1Su5py_T5Ytfh9RoTTX2s20KbSJwBHVxbO7ge5ORqbCk/edit] > The (over-simplified) goal is to re-think the current WAL APIs we have now, > ensure that they are de-coupled from the notion of being backed by HDFS, swap > the current implementations over to the new API, and then wire up the Ratis > LogService to the new WAL API. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
