[jira] [Comment Edited] (HBASE-21246) Introduce WALIdentity interface
[ https://issues.apache.org/jira/browse/HBASE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16675528#comment-16675528 ] Ted Yu edited comment on HBASE-21246 at 11/5/18 6:20 PM: - In patch v24, I dropped the static methods from WALFactory - there are used in test code. I also removed reference to AbstractFSWALProvider in WALFactory since the Reader creation is done by the provider. was (Author: yuzhih...@gmail.com): In patch v24, I dropped the static methods from WALFactory - there are used in test code. I also removed reference to AbstractFSWALProvider since the Reader creation is done by the provider. > Introduce WALIdentity interface > --- > > Key: HBASE-21246 > URL: https://issues.apache.org/jira/browse/HBASE-21246 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Fix For: HBASE-20952 > > Attachments: 21246.003.patch, 21246.20.txt, 21246.21.txt, > 21246.23.txt, 21246.24.txt, 21246.HBASE-20952.001.patch, > 21246.HBASE-20952.002.patch, 21246.HBASE-20952.004.patch, > 21246.HBASE-20952.005.patch, 21246.HBASE-20952.007.patch, > 21246.HBASE-20952.008.patch, replication-src-creates-wal-reader.jpg, > wal-factory-providers.png, wal-providers.png, wal-splitter-reader.jpg, > wal-splitter-writer.jpg > > > We are introducing WALIdentity interface so that the WAL representation can > be decoupled from distributed filesystem. > The interface provides getName method whose return value can represent > filename in distributed filesystem environment or, the name of the stream > when the WAL is backed by log stream. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21246) Introduce WALIdentity interface
[ https://issues.apache.org/jira/browse/HBASE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670472#comment-16670472 ] Ted Yu edited comment on HBASE-21246 at 11/2/18 12:27 AM: -- wal-providers.png is diagram for class hierarchy between WALProvider interface and implementing WAL Provider classes. wal-factory-providers.png is diagram involving WALProvider related classes and WALFactory class. It also shows WALIdentity (and FSWALIdentity) which replaces Path in the existing WAL APIs. In the center of the upper half of the diagram is WALFactory whose functionality is to create WALProvider instances. WALSplitter uses the WALProvider instance created by WALFactory to access WAL. WALSplitter previously refers to the file being split using FileStatus. Now it uses WALIdentity to refer to the entity being split. Below WALIdentity is FSWALIdentity which implements WALIdentity and represents distributed FileSystem based identity (with Path field). To the left of WALFactory is the WALProvider interface. The interface is implemented by the following classes: * RegionGroupingProvider * AbstractFSWALProvider * SyncReplicationWALProvider * DisabledWALProvider The AsyncFSWALProvider and FSHLogProvider classes build on top of (extends) AbstractFSWALProvider. The refactored WAL API, as shown in these diagrams, illustrate how we abstract from distributed FileSystem-centric concepts. was (Author: yuzhih...@gmail.com): wal-providers.png is diagram for class hierarchy between WALProvider interface and implementing WAL Provider classes. wal-factory-providers.png is diagram involving WALProvider related classes and WALFactory class. It also shows WALIdentity (and FSWALIdentity) which replaces Path in the existing WAL APIs. In the center of the upper half of the diagram is WALFactory whose functionality is to create WALProvider instances. WALSplitter uses the WALProvider instance created by WALFactory to access WAL. WALSplitter previously refers to the file being split using FileStatus. Now it uses WALIdentity to refer to the entity being split. Below WALIdentity is FSWALIdentity which implements WALIdentity and represents distributed FileSystem based identity (with Path field). To the left of WALFactory is the WALProvider interface. The interface is implemented by the following classes: * RegionGroupingProvider * AbstractFSWALProvider * SyncReplicationWALProvider * DisabledWALProvider The AsyncFSWALProvider class builds on top of (extends) AbstractFSWALProvider. The refactored WAL API, as shown in these diagrams, illustrate how we abstract from distributed FileSystem-centric concepts. > Introduce WALIdentity interface > --- > > Key: HBASE-21246 > URL: https://issues.apache.org/jira/browse/HBASE-21246 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Fix For: HBASE-20952 > > Attachments: 21246.003.patch, 21246.20.txt, 21246.21.txt, > 21246.23.txt, 21246.HBASE-20952.001.patch, 21246.HBASE-20952.002.patch, > 21246.HBASE-20952.004.patch, 21246.HBASE-20952.005.patch, > 21246.HBASE-20952.007.patch, 21246.HBASE-20952.008.patch, > wal-factory-providers.png, wal-providers.png > > > We are introducing WALIdentity interface so that the WAL representation can > be decoupled from distributed filesystem. > The interface provides getName method whose return value can represent > filename in distributed filesystem environment or, the name of the stream > when the WAL is backed by log stream. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21246) Introduce WALIdentity interface
[ https://issues.apache.org/jira/browse/HBASE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670472#comment-16670472 ] Ted Yu edited comment on HBASE-21246 at 11/2/18 12:04 AM: -- wal-providers.png is diagram for class hierarchy between WALProvider interface and implementing WAL Provider classes. wal-factory-providers.png is diagram involving WALProvider related classes and WALFactory class. It also shows WALIdentity (and FSWALIdentity) which replaces Path in the existing WAL APIs. In the center of the upper half of the diagram is WALFactory whose functionality is to create WALProvider instances. WALSplitter uses the WALProvider instance created by WALFactory to access WAL. WALSplitter previously refers to the file being split using FileStatus. Now it uses WALIdentity to refer to the entity being split. Below WALIdentity is FSWALIdentity which implements WALIdentity and represents distributed FileSystem based identity (with Path field). To the left of WALFactory is the WALProvider interface. The interface is implemented by the following classes: * RegionGroupingProvider * AbstractFSWALProvider * SyncReplicationWALProvider * DisabledWALProvider The AsyncFSWALProvider class builds on top of (extends) AbstractFSWALProvider. The refactored WAL API, as shown in these diagrams, illustrate how we abstract from distributed FileSystem-centric concepts. was (Author: yuzhih...@gmail.com): wal-providers.png is diagram for class hierarchy between WALProvider interface and implementing WAL Provider classes. wal-factory-providers.png is diagram involving WALProvider related classes and WALFactory class. It also shows WALIdentity (and FSWALIdentity) which replaces Path in the existing WAL APIs. In the center of the upper half of the diagram is WALFactory whose functionality is to create WALProvider instances. WALSplitter uses the WALProvider instance created by WALFactory to access WAL. To the left of WALFactory is the WALProvider interface. The interface is implemented by the following classes: * RegionGroupingProvider * AbstractFSWALProvider * SyncReplicationWALProvider * DisabledWALProvider The AsyncFSWALProvider class builds on top of (extends) AbstractFSWALProvider. The refactored WAL API, as shown in these diagrams, illustrate how we abstract from distributed FileSystem-centric concepts. > Introduce WALIdentity interface > --- > > Key: HBASE-21246 > URL: https://issues.apache.org/jira/browse/HBASE-21246 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Fix For: HBASE-20952 > > Attachments: 21246.003.patch, 21246.20.txt, 21246.21.txt, > 21246.23.txt, 21246.HBASE-20952.001.patch, 21246.HBASE-20952.002.patch, > 21246.HBASE-20952.004.patch, 21246.HBASE-20952.005.patch, > 21246.HBASE-20952.007.patch, 21246.HBASE-20952.008.patch, > wal-factory-providers.png, wal-providers.png > > > We are introducing WALIdentity interface so that the WAL representation can > be decoupled from distributed filesystem. > The interface provides getName method whose return value can represent > filename in distributed filesystem environment or, the name of the stream > when the WAL is backed by log stream. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21246) Introduce WALIdentity interface
[ https://issues.apache.org/jira/browse/HBASE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670472#comment-16670472 ] Ted Yu edited comment on HBASE-21246 at 11/1/18 11:58 PM: -- wal-providers.png is diagram for class hierarchy between WALProvider interface and implementing WAL Provider classes. wal-factory-providers.png is diagram involving WALProvider related classes and WALFactory class. It also shows WALIdentity (and FSWALIdentity) which replaces Path in the existing WAL APIs. In the center of the upper half of the diagram is WALFactory whose functionality is to create WALProvider instances. WALSplitter uses the WALProvider instance created by WALFactory to access WAL. To the left of WALFactory is the WALProvider interface. The interface is implemented by the following classes: * RegionGroupingProvider * AbstractFSWALProvider * SyncReplicationWALProvider * DisabledWALProvider The AsyncFSWALProvider class builds on top of (extends) AbstractFSWALProvider. The refactored WAL API, as shown in these diagrams, illustrate how we abstract from distributed FileSystem-centric concepts. was (Author: yuzhih...@gmail.com): wal-providers.png is diagram for class hierarchy between WALProvider interface and implementing WAL Provider classes. wal-factory-providers.png is diagram involving WALProvider related classes and WALFactory class. It also shows WALIdentity (and FSWALIdentity) which replaces Path in the existing WAL APIs. The refactored WAL API, as shown in these diagrams, illustrate how we abstract from distributed FileSystem-centric concepts. > Introduce WALIdentity interface > --- > > Key: HBASE-21246 > URL: https://issues.apache.org/jira/browse/HBASE-21246 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Fix For: HBASE-20952 > > Attachments: 21246.003.patch, 21246.20.txt, 21246.21.txt, > 21246.23.txt, 21246.HBASE-20952.001.patch, 21246.HBASE-20952.002.patch, > 21246.HBASE-20952.004.patch, 21246.HBASE-20952.005.patch, > 21246.HBASE-20952.007.patch, 21246.HBASE-20952.008.patch, > wal-factory-providers.png, wal-providers.png > > > We are introducing WALIdentity interface so that the WAL representation can > be decoupled from distributed filesystem. > The interface provides getName method whose return value can represent > filename in distributed filesystem environment or, the name of the stream > when the WAL is backed by log stream. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21246) Introduce WALIdentity interface
[ https://issues.apache.org/jira/browse/HBASE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670472#comment-16670472 ] Ted Yu edited comment on HBASE-21246 at 11/1/18 5:03 PM: - wal-providers.png is diagram for class hierarchy between WALProvider interface and implementing WAL Provider classes. wal-factory-providers.png is diagram involving WALProvider related classes and WALFactory class. It also shows WALIdentity (and FSWALIdentity) which replaces Path in the existing WAL APIs. The refactored WAL API, as shown in these diagrams, illustrate how we abstract from distributed FileSystem-centric concepts. was (Author: yuzhih...@gmail.com): wal-providers.png is diagram for WALProvider related classes. wal-factory-providers.png is diagram involving WALProvider related classes and WALFactory class. It also shows WALIdentity (and FSWALIdentity) which replaces Path in the existing WAL APIs. The refactored WAL API, as shown in these diagrams, illustrate how we abstract from distributed FileSystem-centric concepts. > Introduce WALIdentity interface > --- > > Key: HBASE-21246 > URL: https://issues.apache.org/jira/browse/HBASE-21246 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Fix For: HBASE-20952 > > Attachments: 21246.003.patch, 21246.20.txt, 21246.21.txt, > 21246.23.txt, 21246.HBASE-20952.001.patch, 21246.HBASE-20952.002.patch, > 21246.HBASE-20952.004.patch, 21246.HBASE-20952.005.patch, > 21246.HBASE-20952.007.patch, 21246.HBASE-20952.008.patch, > wal-factory-providers.png, wal-providers.png > > > We are introducing WALIdentity interface so that the WAL representation can > be decoupled from distributed filesystem. > The interface provides getName method whose return value can represent > filename in distributed filesystem environment or, the name of the stream > when the WAL is backed by log stream. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21246) Introduce WALIdentity interface
[ https://issues.apache.org/jira/browse/HBASE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670472#comment-16670472 ] Ted Yu edited comment on HBASE-21246 at 11/1/18 4:26 PM: - wal-providers.png is diagram for WALProvider related classes. wal-factory-providers.png is diagram involving WALProvider related classes and WALFactory class. It also shows WALIdentity (and FSWALIdentity) which replaces Path in the existing WAL APIs. The refactored WAL API, as shown in these diagrams, illustrate how we abstract from distributed FileSystem-centric concepts. was (Author: yuzhih...@gmail.com): wal-providers.png is diagram for WALProvider related classes. wal-factory-providers.png is diagram involving WALProvider related classes and WALFactory class. > Introduce WALIdentity interface > --- > > Key: HBASE-21246 > URL: https://issues.apache.org/jira/browse/HBASE-21246 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Fix For: HBASE-20952 > > Attachments: 21246.003.patch, 21246.20.txt, 21246.21.txt, > 21246.23.txt, 21246.HBASE-20952.001.patch, 21246.HBASE-20952.002.patch, > 21246.HBASE-20952.004.patch, 21246.HBASE-20952.005.patch, > 21246.HBASE-20952.007.patch, 21246.HBASE-20952.008.patch, > wal-factory-providers.png, wal-providers.png > > > We are introducing WALIdentity interface so that the WAL representation can > be decoupled from distributed filesystem. > The interface provides getName method whose return value can represent > filename in distributed filesystem environment or, the name of the stream > when the WAL is backed by log stream. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21246) Introduce WALIdentity interface
[
https://issues.apache.org/jira/browse/HBASE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659867#comment-16659867
]
Ted Yu edited comment on HBASE-21246 at 10/23/18 5:03 PM:
--
bq. Why do we have a String constructor anyways?
One example usage for the ctor taking String is in {{refreshSources}} method of
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
:
{code}
for (SortedSet walsByGroup :
walsByIdRecoveredQueues.get(queueId).values()) {
walsByGroup.forEach(wal ->
src.enqueueLog(this.walProvider.createWALIdentity(wal)));
{code}
The {{wal}} variable above is the String representation of WAL.
In handling failed replication queue(s), for each queue Id, there are Set of
WALs represented using String (persisted form on zookeeper).
In order to handle the String WAL name (from zookeeper), we would need
{{walProvider.createWALIdentity}} which accepts String parameter.
So {{walProvider.createWALIdentity}} call is deserialization from String form
of WAL name to WALIdentity.
Here is corresponding code from current master branch:
{code}
walsByGroup.forEach(wal -> src.enqueueLog(new Path(wal)));
{code}
createWALIdentity() ends up calling FSWALIdentity ctor accepting String when
WAL is backed by hdfs. For other WAL provider, different WALIdentity instance
would be created.
was (Author: yuzhih...@gmail.com):
bq. Why do we have a String constructor anyways?
One example usage for the ctor taking String is in {{refreshSources}} method of
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
:
{code}
for (SortedSet walsByGroup :
walsByIdRecoveredQueues.get(queueId).values()) {
walsByGroup.forEach(wal ->
src.enqueueLog(this.walProvider.createWALIdentity(wal)));
{code}
The {{wal}} variable above is the String representation of WAL.
In handling failed replication queue(s), for each queue Id, there are Set of
WALs represented using String (persisted form on zookeeper).
So {{walProvider.createWALIdentity}} call is deserialization from String form
of WAL name to WALIdentity.
Here is corresponding code from current master branch:
{code}
walsByGroup.forEach(wal -> src.enqueueLog(new Path(wal)));
{code}
createWALIdentity() ends up calling FSWALIdentity ctor accepting String when
WAL is backed by hdfs. For other WAL provider, different WALIdentity instance
would be created.
> Introduce WALIdentity interface
> ---
>
> Key: HBASE-21246
> URL: https://issues.apache.org/jira/browse/HBASE-21246
> Project: HBase
> Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: HBASE-20952
>
> Attachments: 21246.003.patch, 21246.HBASE-20952.001.patch,
> 21246.HBASE-20952.002.patch, 21246.HBASE-20952.004.patch,
> 21246.HBASE-20952.005.patch, 21246.HBASE-20952.007.patch,
> 21246.HBASE-20952.008.patch
>
>
> We are introducing WALIdentity interface so that the WAL representation can
> be decoupled from distributed filesystem.
> The interface provides getName method whose return value can represent
> filename in distributed filesystem environment or, the name of the stream
> when the WAL is backed by log stream.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21246) Introduce WALIdentity interface
[
https://issues.apache.org/jira/browse/HBASE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659867#comment-16659867
]
Ted Yu edited comment on HBASE-21246 at 10/23/18 3:33 AM:
--
bq. Why do we have a String constructor anyways?
One example usage for the ctor taking String is in {{refreshSources}} method of
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
:
{code}
for (SortedSet walsByGroup :
walsByIdRecoveredQueues.get(queueId).values()) {
walsByGroup.forEach(wal ->
src.enqueueLog(this.walProvider.createWALIdentity(wal)));
{code}
The {{wal}} variable above is the String representation of WAL.
In handling failed replication queue(s), for each queue Id, there are Set of
WALs represented using String (persisted form on zookeeper).
So {{walProvider.createWALIdentity}} call is deserialization from String form
of WAL name to WALIdentity.
Here is corresponding code from current master branch:
{code}
walsByGroup.forEach(wal -> src.enqueueLog(new Path(wal)));
{code}
createWALIdentity() ends up calling FSWALIdentity ctor accepting String when
WAL is backed by hdfs. For other WAL provider, different WALIdentity instance
would be created.
was (Author: yuzhih...@gmail.com):
bq. Why do we have a String constructor anyways?
One example usage for the ctor taking String is in refreshSources of
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
:
{code}
for (SortedSet walsByGroup :
walsByIdRecoveredQueues.get(queueId).values()) {
walsByGroup.forEach(wal ->
src.enqueueLog(this.walProvider.createWALIdentity(wal)));
{code}
createWALIdentity() ends up calling FSWALIdentity ctor accepting String (when
WAL is backed by hdfs).
> Introduce WALIdentity interface
> ---
>
> Key: HBASE-21246
> URL: https://issues.apache.org/jira/browse/HBASE-21246
> Project: HBase
> Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: HBASE-20952
>
> Attachments: 21246.003.patch, 21246.HBASE-20952.001.patch,
> 21246.HBASE-20952.002.patch, 21246.HBASE-20952.004.patch,
> 21246.HBASE-20952.005.patch, 21246.HBASE-20952.007.patch,
> 21246.HBASE-20952.008.patch
>
>
> We are introducing WALIdentity interface so that the WAL representation can
> be decoupled from distributed filesystem.
> The interface provides getName method whose return value can represent
> filename in distributed filesystem environment or, the name of the stream
> when the WAL is backed by log stream.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21246) Introduce WALIdentity interface
[
https://issues.apache.org/jira/browse/HBASE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659991#comment-16659991
]
Josh Elser edited comment on HBASE-21246 at 10/23/18 3:03 AM:
--
{quote}
> Why do we have a String constructor anyways?
createWALIdentity() ends up calling FSWALIdentity ctor accepting String (when
WAL is backed by hdfs).
{quote}
I don't understand why you're telling me this. I could look at the wal-refactor
code if I wanted to know who called it. I am asking you why this HDFS-based
implementation for a WAL is using anything other than a {{Path}} to refer to a
WAL.
The question (worry) in-between the lines is: do we have references to WALs
other than by Path? Looking at {{master}}, the original code from
ReplicationSourceManager was using Path.
was (Author: elserj):
{quote}{quote}Why do we have a String constructor anyways?
{quote}
createWALIdentity() ends up calling FSWALIdentity ctor accepting String (when
WAL is backed by hdfs).
{quote}
I don't understand why you're telling me this. I could look at the wal-refactor
code if I wanted to know who called it. I am asking you why this HDFS-based
implementation for a WAL is using anything other than a {{Path}} to refer to a
WAL.
The question (worry) in-between the lines is: do we have references to WALs
other than by Path? Looking at {{master}}, the original code from
ReplicationSourceManager was using Path.
> Introduce WALIdentity interface
> ---
>
> Key: HBASE-21246
> URL: https://issues.apache.org/jira/browse/HBASE-21246
> Project: HBase
> Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: HBASE-20952
>
> Attachments: 21246.003.patch, 21246.HBASE-20952.001.patch,
> 21246.HBASE-20952.002.patch, 21246.HBASE-20952.004.patch,
> 21246.HBASE-20952.005.patch, 21246.HBASE-20952.007.patch,
> 21246.HBASE-20952.008.patch
>
>
> We are introducing WALIdentity interface so that the WAL representation can
> be decoupled from distributed filesystem.
> The interface provides getName method whose return value can represent
> filename in distributed filesystem environment or, the name of the stream
> when the WAL is backed by log stream.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21246) Introduce WALIdentity interface
[ https://issues.apache.org/jira/browse/HBASE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659666#comment-16659666 ] Ted Yu edited comment on HBASE-21246 at 10/22/18 9:03 PM: -- Stack's questions seem to be at higher level. So allow me to respond to them first. bq. Where is WALIdentity in the design doc? See this thread on the doc: https://docs.google.com/document/d/1o552MkKq9wF3BXY2nVcsCXBAImUH6r132Cxv9WHL3D8/edit?disco=COODUIA bq. Designed or are we trying variations? Or a particular variation? What is presented in the patch on this JIRA follows what is outlined in the design doc. w.r.t. the concrete fields / methods of WALIdentity (and implementing classes), we're open to discussion and will respect the community's consensus. was (Author: yuzhih...@gmail.com): Stack's questions seem to be at higher level. So allow me to respond to them first. bq. Where is WALIdentity in the design doc? See this thread on the doc: https://docs.google.com/document/d/1o552MkKq9wF3BXY2nVcsCXBAImUH6r132Cxv9WHL3D8/edit bq. Designed or are we trying variations? Or a particular variation? What is presented in the patch on this JIRA follows what is outlined in the design doc. w.r.t. the concrete fields / methods of WALIdentity (and implementing classes), we're open to discussion and will respect the community's consensus. > Introduce WALIdentity interface > --- > > Key: HBASE-21246 > URL: https://issues.apache.org/jira/browse/HBASE-21246 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Fix For: HBASE-20952 > > Attachments: 21246.003.patch, 21246.HBASE-20952.001.patch, > 21246.HBASE-20952.002.patch, 21246.HBASE-20952.004.patch, > 21246.HBASE-20952.005.patch, 21246.HBASE-20952.007.patch, > 21246.HBASE-20952.008.patch > > > We are introducing WALIdentity interface so that the WAL representation can > be decoupled from distributed filesystem. > The interface provides getName method whose return value can represent > filename in distributed filesystem environment or, the name of the stream > when the WAL is backed by log stream. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21246) Introduce WALIdentity interface
[
https://issues.apache.org/jira/browse/HBASE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652899#comment-16652899
]
Ted Yu edited comment on HBASE-21246 at 10/17/18 4:16 AM:
--
bq. the strange thing is that we have a method which creates WALIdentity
directly from a String.
Good point.
In ReplicationSourceManager, we can change:
{code}
private final ConcurrentMap>>
walsByIdRecoveredQueues;
{code}
to:
{code}
private final ConcurrentMap>>
walsByIdRecoveredQueues;
{code}
This way, in refreshSources(), we retrieve WALIdentity from the Map and don't
need to create identity.
I am making changes in ReplicationSourceManager and SyncReplicationWALProvider
in wal-refactor repo for evaluation.
I will drop createWALIdentity in the next patch once evaluation passes.
was (Author: yuzhih...@gmail.com):
bq. the strange thing is that we have a method which creates WALIdentity
directly from a String.
Good point.
In ReplicationSourceManager, we can change:
{code}
private final ConcurrentMap>>
walsByIdRecoveredQueues;
{code}
to:
{code}
private final ConcurrentMap>>
walsByIdRecoveredQueues;
{code}
This way, in refreshSources(), we retrieve WALIdentity from the Map and don't
need to create identity.
I will drop createWALIdentity in the next patch.
> Introduce WALIdentity interface
> ---
>
> Key: HBASE-21246
> URL: https://issues.apache.org/jira/browse/HBASE-21246
> Project: HBase
> Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: HBASE-20952
>
> Attachments: 21246.003.patch, 21246.HBASE-20952.001.patch,
> 21246.HBASE-20952.002.patch, 21246.HBASE-20952.004.patch,
> 21246.HBASE-20952.005.patch
>
>
> We are introducing WALIdentity interface so that the WAL representation can
> be decoupled from distributed filesystem.
> The interface provides getName method whose return value can represent
> filename in distributed filesystem environment or, the name of the stream
> when the WAL is backed by log stream.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21246) Introduce WALIdentity interface
[ https://issues.apache.org/jira/browse/HBASE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638994#comment-16638994 ] Ted Yu edited comment on HBASE-21246 at 10/10/18 3:56 PM: -- The first question is addressed by patch v2 which drops the method. The abstraction of WAL identity should allow all backends (hdfs, Ratis, Kafka) to express the information the underlying backend carries for each WAL. e.g. for Kafka backend, we can use topic to represent region, the partitions under the topic would correspond to the WALs for the region. Therefore a String alone doesn't have as much expressibility as an interface. That is why we chose to use an interface to represent WAL identity. was (Author: yuzhih...@gmail.com): Now that design doc is back to review mode, please allow some more time till I address the above comment. > Introduce WALIdentity interface > --- > > Key: HBASE-21246 > URL: https://issues.apache.org/jira/browse/HBASE-21246 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Attachments: 21246.HBASE-20952.001.patch, 21246.HBASE-20952.002.patch > > > We are introducing WALIdentity interface so that the WAL representation can > be decoupled from distributed filesystem. > The interface provides getName method whose return value can represent > filename in distributed filesystem environment or, the name of the stream > when the WAL is backed by log stream. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
