[
https://issues.apache.org/jira/browse/HBASE-28216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788857#comment-17788857
]
Wei-Chiu Chuang commented on HBASE-28216:
-----------------------------------------
No that's fine. We're pursuing that in a separate branch HBASE-27740.
(reminder to myself: finish review&merge HBASE-27769 today)
I was under the impression that setErasureCodingPolicy requires HDFS admin user
privilege. But checking the code again looks like it requires just the write
privilege of that directory.
We recently added EC support in Apache Impala (check out Cloudera doc
https://docs.cloudera.com/cdw-runtime/1.5.1/impala-reference/topics/impala-ec-policies.html
the doc talks about Ozone EC but it works the same way for HDFS EC) but we did
not add the support for Impala to update table EC properties.
> HDFS erasure coding support for table data dirs
> -----------------------------------------------
>
> Key: HBASE-28216
> URL: https://issues.apache.org/jira/browse/HBASE-28216
> Project: HBase
> Issue Type: New Feature
> Reporter: Bryan Beaudreault
> Priority: Major
>
> [Erasure
> coding|https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html]
> (EC) is a hadoop-3 feature which can drastically reduce storage
> requirements, at the expense of locality. At my company we have a few hbase
> clusters which are extremely data dense and take mostly write traffic, fewer
> reads (cold data). We'd like to reduce the cost of these clusters, and EC is
> a great way to do that since it can reduce replication related storage costs
> by 50%.
> It's possible to enable EC policies on sub directories of HDFS. One can
> manually set this with {{{}hdfs ec -setPolicy -path
> /hbase/data/default/usertable -policy xxxx{}}}. This can work without any
> hbase support.
> One problem with that is a lack of visibility by operators into which tables
> might have EC enabled. I think this is where HBase can help. Here's my
> proposal:
> * Add a new TableDescriptor and ColumnDescriptor field ERASURE_CODING_POLICY
> * In ModifyTableProcedure preflightChecks, if ERASURE_CODING_POLICY is set,
> verify that the requested policy is available and enabled via
> DistributedFileSystem.
> getErasureCodingPolicies().
> * During ModifyTableProcedure, add a new state for
> MODIFY_TABLE_SYNC_ERASURE_CODING_POLICY.
> ** When adding or changing a policy, use DistributedFileSystem.
> setErasureCodingPolicy to sync it for the data and archive dir of that table
> (or column in table)
> ** When removing the property or setting it to empty, use
> DistributedFileSystem.
> unsetErasureCodingPolicy to remove it from the data and archive dir.
> Since this new API is in hadoop-3 only, we'll need to add a reflection
> wrapper class for managing the calls and verifying that the API is available.
> We'll similarly do that API check in preflightChecks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
