[
https://issues.apache.org/jira/browse/HBASE-14439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ben Lau updated HBASE-14439:
----------------------------
Description:
Ticket for work in progress on new FileSystem abstractions. Previously, we
(Yahoo) submitted a ticket that would add support for humongous (1 million
region+) tables via a hierarchical layout (HBASE-13991). However open source
is moving in a similar but not identical direction in the future and so the
patch will not be merged into open source.
We will be working on a different patch now with folks from open source. It
will create/add to 2 layers-- a path abstraction layer and a use-oriented
abstraction layer. The path abstraction layer is epitomized by classes like
FsUtils (and in the patch new classes like AFsLayout). The use oriented
abstraction layer is epitomized by existing classes like
MasterFileSystem/HRegionFileSystem (and possibly new classes later) that build
on the path abstraction layer and focus on 'doing things' (eg creating regions)
and less on the gritty details like the paths.
This work on abstracting and isolating the paths from the use cases will help
Yahoo not diverge too much from open source with its internal 'Humongous' table
hierarchical layout, while also helping open source move further towards the
eventual goal of redoing the FS layout in a similar (but different)
hierarchical layout later that focuses on data directory uniformity (unlike the
humongous patch) and storing hierarchy in the meta table instead which enables
new optimizations (see HBASE-14090.)
Attached to this ticket is some work we've done at Yahoo so far that will be
put into an open source HBase branch for further collaboration. The patch is
not meant to be complete yet and is a work in progress. (Please wait on patch
comments/reviews.) It also includes some Yahoo-specific 'humongous' layout
code that will be removed before submission in open source.
was:
Ticket for work in progress on new FileSystem abstractions. Previously, we
(Yahoo) submitted a ticket that would add support for humongous (1 million
region+) tables via a hierarchical layout (HBASE-13991). However open source
is moving in a similar but not identical direction in the future and so the
patch will not be merged into open source.
We will be working with Cloudera on a different patch now. It will create/add
to 2 layers-- a path abstraction layer and a use-oriented abstraction layer.
The path abstraction layer is epitomized by classes like FsUtils (and in the
patch new classes like AFsLayout). The use oriented abstraction layer is
epitomized by existing classes like MasterFileSystem/HRegionFileSystem (and
possibly new classes later) that build on the path abstraction layer and focus
on 'doing things' (eg creating regions) and less on the gritty details like the
paths.
This work on abstracting and isolating the paths from the use cases will help
Yahoo not diverge too much from open source with its internal 'Humongous' table
hierarchical layout, while also helping open source move further towards the
eventual goal of redoing the FS layout in a similar (but different)
hierarchical layout later that focuses on data directory uniformity (unlike the
humongous patch) and storing hierarchy in the meta table instead which enables
new optimizations (see HBASE-14090.)
Attached to this ticket is some work we've done at Yahoo so far that will be
put into an open source HBase branch for further collaboration. The patch is
not meant to be complete yet and is a work in progress. (Please wait on patch
comments/reviews.) It also includes some Yahoo-specific 'humongous' layout
code that will be removed before submission in open source.
> New/Improved Filesystem Abstractions
> ------------------------------------
>
> Key: HBASE-14439
> URL: https://issues.apache.org/jira/browse/HBASE-14439
> Project: HBase
> Issue Type: Sub-task
> Reporter: Ben Lau
> Assignee: Matteo Bertozzi
> Attachments: abstraction.patch
>
>
> Ticket for work in progress on new FileSystem abstractions. Previously, we
> (Yahoo) submitted a ticket that would add support for humongous (1 million
> region+) tables via a hierarchical layout (HBASE-13991). However open source
> is moving in a similar but not identical direction in the future and so the
> patch will not be merged into open source.
> We will be working on a different patch now with folks from open source. It
> will create/add to 2 layers-- a path abstraction layer and a use-oriented
> abstraction layer. The path abstraction layer is epitomized by classes like
> FsUtils (and in the patch new classes like AFsLayout). The use oriented
> abstraction layer is epitomized by existing classes like
> MasterFileSystem/HRegionFileSystem (and possibly new classes later) that
> build on the path abstraction layer and focus on 'doing things' (eg creating
> regions) and less on the gritty details like the paths.
> This work on abstracting and isolating the paths from the use cases will help
> Yahoo not diverge too much from open source with its internal 'Humongous'
> table hierarchical layout, while also helping open source move further
> towards the eventual goal of redoing the FS layout in a similar (but
> different) hierarchical layout later that focuses on data directory
> uniformity (unlike the humongous patch) and storing hierarchy in the meta
> table instead which enables new optimizations (see HBASE-14090.)
> Attached to this ticket is some work we've done at Yahoo so far that will be
> put into an open source HBase branch for further collaboration. The patch is
> not meant to be complete yet and is a work in progress. (Please wait on
> patch comments/reviews.) It also includes some Yahoo-specific 'humongous'
> layout code that will be removed before submission in open source.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
