[
https://issues.apache.org/jira/browse/HBASE-24289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mengqing Peng updated HBASE-24289:
----------------------------------
Release Note:
Enhance DateTieredCompaction to support HDFS storage policy within one class
family.
# First you need enable DTCP.
To turn on Date Tiered Compaction (It is not recommended to turn on for the
whole cluster because that will put meta table on it too and random get on meta
table will be impacted):
hbase.hstore.compaction.compaction.policy=org.apache.hadoop.hbase.regionserver.compactions.DateTieredCompactionPolicy
## Parameters for Date Tiered Compaction:
hbase.hstore.compaction.date.tiered.max.storefile.age.millis: Files with
max-timestamp smaller than this will no longer be compacted.Default at
Long.MAX_VALUE.
hbase.hstore.compaction.date.tiered.base.window.millis: base window size in
milliseconds. Default at 6 hours.
hbase.hstore.compaction.date.tiered.windows.per.tier: number of windows per
tier. Default at 4.
hbase.hstore.compaction.date.tiered.incoming.window.min: minimal number of
files to compact in the incoming window. Set it to expected number of files in
the window to avoid wasteful compaction. Default at 6.
# Then enable HDTCP(Heterogeneous Date Tiered Compaction) as follow example
configurations:
hbase.hstore.compaction.date.tiered.storage.policy.enable=true
hbase.hstore.compaction.date.tiered.hot.window.age.millis=3600000
hbase.hstore.compaction.date.tiered.hot.window.storage.policy=ALL_SSD
hbase.hstore.compaction.date.tiered.warm.window.age.millis=20600000
hbase.hstore.compaction.date.tiered.warm.window.storage.policy=ONE_SSD
hbase.hstore.compaction.date.tiered.cold.window.storage.policy=HOT
## It is better to enable WAL and flushing HFile storage policy with HDTCP. You
can tune follow settings as well:
hbase.wal.storage.policy=ALL_SSD
create
'table',{NAME=>'f1',CONFIGURATION=>{'hbase.hstore.block.storage.policy'=>'ALL_SSD'}}
# Disable HDTCP as follow:
hbase.hstore.compaction.date.tiered.storage.policy.enable=false
was:
Enhance DateTieredCompaction to support HDFS storage policy within one class
family.
# First you need enable DTCP.
To turn on Date Tiered Compaction (It is not recommended to turn on for the
whole cluster because that will put meta table on it too and random get on meta
table will be impacted):
hbase.hstore.compaction.compaction.policy=org.apache.hadoop.hbase.regionserver.compactions.DateTieredCompactionPolicy
## Parameters for Date Tiered Compaction:
hbase.hstore.compaction.date.tiered.max.storefile.age.millis: Files with
max-timestamp smaller than this will no longer be compacted.Default at
Long.MAX_VALUE.
hbase.hstore.compaction.date.tiered.base.window.millis: base window size in
milliseconds. Default at 6 hours.
hbase.hstore.compaction.date.tiered.windows.per.tier: number of windows per
tier. Default at 4.
hbase.hstore.compaction.date.tiered.incoming.window.min: minimal number of
files to compact in the incoming window. Set it to expected number of files in
the window to avoid wasteful compaction. Default at 6.
# Then enable HDTCP(Heterogeneous Date Tiered Compaction) as follow example
configurations:
hbase.hstore.compaction.date.tiered.storage.policy.enable=true
hbase.hstore.compaction.date.tiered.hot.window.age.millis=3600000
hbase.hstore.compaction.date.tiered.hot.window.storage.policy=ALL_SSD
hbase.hstore.compaction.date.tiered.warm.window.age.millis=20600000
hbase.hstore.compaction.date.tiered.warm.window.storage.policy=ONE_SSD
hbase.hstore.compaction.date.tiered.cold.window.storage.policy=HOT
## It is better to enable WAL and flushing HFile storage policy with HDTCP. You
can tune follow settings as well:
hbase.wal.storage.policy=ALL_SSD
create
'table',{NAME=>'f1',CONFIGURATION=>{'hbase.hstore.block.storage.policy'=>'ONE_SSD'}}
# Disable HDTCP as follow:
hbase.hstore.compaction.date.tiered.storage.policy.enable=false
> Heterogeneous Storage for Date Tiered Compaction
> ------------------------------------------------
>
> Key: HBASE-24289
> URL: https://issues.apache.org/jira/browse/HBASE-24289
> Project: HBase
> Issue Type: New Feature
> Components: Compaction
> Reporter: Mengqing Peng
> Assignee: Mengqing Peng
> Priority: Major
> Fix For: 3.0.0-alpha-1, 2.4.0
>
>
> Support DateTiredCompaction(HBASE-15181) for cold and hot data separation,
> support different storage policies for different time periods of data to get
> better performance, for example, we can configure the data of last 1 month in
> SSD, and 1 month ago data was in HDD.
> design doc:
> https://docs.google.com/document/d/1fk_EWLNnxniwt3gDjUS_apQ3cPzn90AmvDT1wkirvKE/edit?usp=sharing
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
