[jira] [Updated] (HBASE-15314) Allow more than one backing file in bucketcache
[
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-15314:
--
Fix Version/s: (was: 2.0)
2.0.0
> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
> Issue Type: Sub-task
> Components: BucketCache
>Reporter: stack
>Assignee: chunhui shen
> Fix For: 2.0.0
>
> Attachments: FileIOEngine.java, HBASE-15314.master.001.patch,
> HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch,
> HBASE-15314-v3.patch, HBASE-15314-v4.patch, HBASE-15314-v5.patch,
> HBASE-15314-v6.patch, HBASE-15314-v7.patch, HBASE-15314-v8.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more
> than one SSD in it.
> Usage (Setting the following configurations in hbase-site.xml):
> {quote}
>
> hbase.bucketcache.ioengine
>
> files:/mnt/disk1/bucketcache,/mnt/disk2/bucketcache,/mnt/disk3/bucketcache,/mnt/disk4/bucketcache
>
>
> hbase.bucketcache.size
> 1048576
>
> {quote}
> The above setting means the total capacity of cache is 1048576MB(1TB), each
> file length will be set to 0.25TB.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-15314) Allow more than one backing file in bucketcache
[
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chunhui shen updated HBASE-15314:
-
Description:
Allow bucketcache use more than just one backing file: e.g. chassis has more
than one SSD in it.
Usage (Setting the following configurations in hbase-site.xml):
{quote}
hbase.bucketcache.ioengine
files:/mnt/disk1/bucketcache,/mnt/disk2/bucketcache,/mnt/disk3/bucketcache,/mnt/disk4/bucketcache
hbase.bucketcache.size
1048576
{quote}
The above setting means the total capacity of cache is 1048576MB(1TB), each
file length will be set to 0.25TB.
was:
Allow bucketcache use more than just one backing file: e.g. chassis has more
than one SSD in it.
Usage:
{quote}
Setting the following configurations in hbase-site.xml:
hbase.bucketcache.ioengine
files:/mnt/disk1/bucketcache,/mnt/disk2/bucketcache,/mnt/disk3/bucketcache,/mnt/disk4/bucketcache
hbase.bucketcache.size
1048576
{quote}
The above setting means the total capacity of cache is 1048576MB(1TB), each
file length will be set to 0.25TB.
> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
> Issue Type: Sub-task
> Components: BucketCache
>Reporter: stack
>Assignee: chunhui shen
> Fix For: 2.0
>
> Attachments: FileIOEngine.java, HBASE-15314.master.001.patch,
> HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch,
> HBASE-15314-v3.patch, HBASE-15314-v4.patch, HBASE-15314-v5.patch,
> HBASE-15314-v6.patch, HBASE-15314-v7.patch, HBASE-15314-v8.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more
> than one SSD in it.
> Usage (Setting the following configurations in hbase-site.xml):
> {quote}
>
> hbase.bucketcache.ioengine
>
> files:/mnt/disk1/bucketcache,/mnt/disk2/bucketcache,/mnt/disk3/bucketcache,/mnt/disk4/bucketcache
>
>
> hbase.bucketcache.size
> 1048576
>
> {quote}
> The above setting means the total capacity of cache is 1048576MB(1TB), each
> file length will be set to 0.25TB.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-15314) Allow more than one backing file in bucketcache
[
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chunhui shen updated HBASE-15314:
-
Description:
Allow bucketcache use more than just one backing file: e.g. chassis has more
than one SSD in it.
Usage:
{quote}
Setting the following configurations in hbase-site.xml:
hbase.bucketcache.ioengine
files:/mnt/disk1/bucketcache,/mnt/disk2/bucketcache,/mnt/disk3/bucketcache,/mnt/disk4/bucketcache
hbase.bucketcache.size
1048576
{quote}
The above setting means the total capacity of cache is 1048576MB(1TB), each
file length will be set to 0.25TB.
was:Allow bucketcache use more than just one backing file: e.g. chassis has
more than one SSD in it.
> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
> Issue Type: Sub-task
> Components: BucketCache
>Reporter: stack
>Assignee: chunhui shen
> Fix For: 2.0
>
> Attachments: FileIOEngine.java, HBASE-15314.master.001.patch,
> HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch,
> HBASE-15314-v3.patch, HBASE-15314-v4.patch, HBASE-15314-v5.patch,
> HBASE-15314-v6.patch, HBASE-15314-v7.patch, HBASE-15314-v8.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more
> than one SSD in it.
> Usage:
> {quote}
> Setting the following configurations in hbase-site.xml:
>
> hbase.bucketcache.ioengine
>
> files:/mnt/disk1/bucketcache,/mnt/disk2/bucketcache,/mnt/disk3/bucketcache,/mnt/disk4/bucketcache
>
>
> hbase.bucketcache.size
> 1048576
>
> {quote}
> The above setting means the total capacity of cache is 1048576MB(1TB), each
> file length will be set to 0.25TB.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-15314) Allow more than one backing file in bucketcache
[ https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15314: --- Resolution: Fixed Assignee: chunhui shen (was: Aaron Tokhy) Hadoop Flags: Reviewed Fix Version/s: 2.0 Release Note: (was: The following patch adds a couple of features without making any big changes to the existing BucketCache allocator. This approach introduces the notion of 'segmentation' (or in other words, the underlying IOEngine can be made up of non-contiguous segments. Two methods are added to expose this information to the BucketCache allocator. boolean IOEngine#isSegmented() boolean IOEngine#doesAllocationCrossSegments(long offset, long len) BucketCache calls these methods to determine if a 'contiguous' allocation of a particular block size can occur. It does this by checking if doesAllocationCrossSegments(offset, len) is true. If an allocation crosses a segment, another call to allocate is made for the same block. The first block is wasted (it is marked allocated). The worst case is 1 'largest block' per file. If an allocation fails for any reason, all/any allocated blocks (including wasted ones) are freed again for subsequent allocation requests. This is very similar to a 'JBOD' configuration (there is no striping of any kind). To configure HBase to use multiple files when configuring the HBase bucket cache, set the following configurations in hbase-site.xml: ... hbase.bucketcache.ioengine files:/mnt/disk1/bucketcache,/mnt/disk2/bucketcache,/mnt/disk3/bucketcache,/mnt/disk4/bucketcache hbase.bucketcache.size 1048756 ... /mnt/disk1, /mnt/disk2, /mnt/disk3 and /mnt/disk4 are expected to be separate filesystems (one mountpoint per volume). HBase would need to have read/write permissions to these directories. In this configuration, 4 sparsely allocated 256GB files (each named 'bucketcache') are created in each directory on HBase RegionServer startup. Ideally such a configuration is useful when the underlying HBase filesystem is non-local. ) Status: Resolved (was: Patch Available) Pushed to master. Thanks for all the your efforts [~zyork] and @aaron Thanks for the patch [~zjushch]. Thanks for the reviews. [~zjushch] Can you update the release note with the latest change? If not let me know I will try to phrase it. > Allow more than one backing file in bucketcache > --- > > Key: HBASE-15314 > URL: https://issues.apache.org/jira/browse/HBASE-15314 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack >Assignee: chunhui shen > Fix For: 2.0 > > Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, > HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, > HBASE-15314-v3.patch, HBASE-15314-v4.patch, HBASE-15314-v5.patch, > HBASE-15314-v6.patch, HBASE-15314-v7.patch, HBASE-15314-v8.patch > > > Allow bucketcache use more than just one backing file: e.g. chassis has more > than one SSD in it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-15314) Allow more than one backing file in bucketcache
[ https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chunhui shen updated HBASE-15314: - Attachment: HBASE-15314-v8.patch Update the patch as rb comment > Allow more than one backing file in bucketcache > --- > > Key: HBASE-15314 > URL: https://issues.apache.org/jira/browse/HBASE-15314 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack >Assignee: Aaron Tokhy > Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, > HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, > HBASE-15314-v3.patch, HBASE-15314-v4.patch, HBASE-15314-v5.patch, > HBASE-15314-v6.patch, HBASE-15314-v7.patch, HBASE-15314-v8.patch > > > Allow bucketcache use more than just one backing file: e.g. chassis has more > than one SSD in it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-15314) Allow more than one backing file in bucketcache
[ https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chunhui shen updated HBASE-15314: - Attachment: HBASE-15314-v7.patch v7.patch added some annotations as comment on review board. > Allow more than one backing file in bucketcache > --- > > Key: HBASE-15314 > URL: https://issues.apache.org/jira/browse/HBASE-15314 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack >Assignee: Aaron Tokhy > Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, > HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, > HBASE-15314-v3.patch, HBASE-15314-v4.patch, HBASE-15314-v5.patch, > HBASE-15314-v6.patch, HBASE-15314-v7.patch > > > Allow bucketcache use more than just one backing file: e.g. chassis has more > than one SSD in it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-15314) Allow more than one backing file in bucketcache
[ https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chunhui shen updated HBASE-15314: - Attachment: HBASE-15314-v6.patch Uploaded the latest patch (15314-v6.patch) on review board > Allow more than one backing file in bucketcache > --- > > Key: HBASE-15314 > URL: https://issues.apache.org/jira/browse/HBASE-15314 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack >Assignee: Aaron Tokhy > Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, > HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, > HBASE-15314-v3.patch, HBASE-15314-v4.patch, HBASE-15314-v5.patch, > HBASE-15314-v6.patch > > > Allow bucketcache use more than just one backing file: e.g. chassis has more > than one SSD in it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-15314) Allow more than one backing file in bucketcache
[ https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chunhui shen updated HBASE-15314: - Attachment: HBASE-15314-v5.patch > Allow more than one backing file in bucketcache > --- > > Key: HBASE-15314 > URL: https://issues.apache.org/jira/browse/HBASE-15314 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack >Assignee: Aaron Tokhy > Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, > HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, > HBASE-15314-v3.patch, HBASE-15314-v4.patch, HBASE-15314-v5.patch > > > Allow bucketcache use more than just one backing file: e.g. chassis has more > than one SSD in it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-15314) Allow more than one backing file in bucketcache
[ https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chunhui shen updated HBASE-15314: - Attachment: HBASE-15314-v4.patch > Allow more than one backing file in bucketcache > --- > > Key: HBASE-15314 > URL: https://issues.apache.org/jira/browse/HBASE-15314 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack >Assignee: Aaron Tokhy > Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, > HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, > HBASE-15314-v3.patch, HBASE-15314-v4.patch > > > Allow bucketcache use more than just one backing file: e.g. chassis has more > than one SSD in it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-15314) Allow more than one backing file in bucketcache
[ https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chunhui shen updated HBASE-15314: - Attachment: FileIOEngine.java > Allow more than one backing file in bucketcache > --- > > Key: HBASE-15314 > URL: https://issues.apache.org/jira/browse/HBASE-15314 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack >Assignee: Aaron Tokhy > Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, > HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, > HBASE-15314-v3.patch > > > Allow bucketcache use more than just one backing file: e.g. chassis has more > than one SSD in it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15314) Allow more than one backing file in bucketcache
[ https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15314: -- Attachment: HBASE-15314.master.001.patch > Allow more than one backing file in bucketcache > --- > > Key: HBASE-15314 > URL: https://issues.apache.org/jira/browse/HBASE-15314 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack >Assignee: Aaron Tokhy > Attachments: HBASE-15314.master.001.patch, > HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, > HBASE-15314-v3.patch > > > Allow bucketcache use more than just one backing file: e.g. chassis has more > than one SSD in it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15314) Allow more than one backing file in bucketcache
[ https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15314: -- Attachment: (was: HBASE-15314-v3.patch) > Allow more than one backing file in bucketcache > --- > > Key: HBASE-15314 > URL: https://issues.apache.org/jira/browse/HBASE-15314 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack >Assignee: Aaron Tokhy > Attachments: HBASE-15314.master.001.patch, > HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, > HBASE-15314-v3.patch > > > Allow bucketcache use more than just one backing file: e.g. chassis has more > than one SSD in it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15314) Allow more than one backing file in bucketcache
[ https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15314: -- Attachment: HBASE-15314-v3.patch > Allow more than one backing file in bucketcache > --- > > Key: HBASE-15314 > URL: https://issues.apache.org/jira/browse/HBASE-15314 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack >Assignee: Aaron Tokhy > Attachments: HBASE-15314.master.001.patch, HBASE-15314.patch, > HBASE-15314-v2.patch, HBASE-15314-v3.patch, HBASE-15314-v3.patch > > > Allow bucketcache use more than just one backing file: e.g. chassis has more > than one SSD in it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15314) Allow more than one backing file in bucketcache
[ https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Tokhy updated HBASE-15314: Attachment: HBASE-15314.master.001.patch > Allow more than one backing file in bucketcache > --- > > Key: HBASE-15314 > URL: https://issues.apache.org/jira/browse/HBASE-15314 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack >Assignee: Aaron Tokhy > Attachments: HBASE-15314.master.001.patch, HBASE-15314.patch, > HBASE-15314-v2.patch, HBASE-15314-v3.patch > > > Allow bucketcache use more than just one backing file: e.g. chassis has more > than one SSD in it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15314) Allow more than one backing file in bucketcache
[ https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Tokhy updated HBASE-15314: Attachment: (was: HBASE-15314.master.001.patch) > Allow more than one backing file in bucketcache > --- > > Key: HBASE-15314 > URL: https://issues.apache.org/jira/browse/HBASE-15314 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack >Assignee: Aaron Tokhy > Attachments: HBASE-15314.patch, HBASE-15314-v2.patch, > HBASE-15314-v3.patch > > > Allow bucketcache use more than just one backing file: e.g. chassis has more > than one SSD in it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15314) Allow more than one backing file in bucketcache
[ https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Tokhy updated HBASE-15314: Attachment: HBASE-15314.master.001.patch > Allow more than one backing file in bucketcache > --- > > Key: HBASE-15314 > URL: https://issues.apache.org/jira/browse/HBASE-15314 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack >Assignee: Aaron Tokhy > Attachments: HBASE-15314.master.001.patch, HBASE-15314.patch, > HBASE-15314-v2.patch, HBASE-15314-v3.patch > > > Allow bucketcache use more than just one backing file: e.g. chassis has more > than one SSD in it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15314) Allow more than one backing file in bucketcache
[ https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Tokhy updated HBASE-15314: Release Note: The following patch adds a couple of features without making any big changes to the existing BucketCache allocator. This approach introduces the notion of 'segmentation' (or in other words, the underlying IOEngine can be made up of non-contiguous segments. Two methods are added to expose this information to the BucketCache allocator. boolean IOEngine#isSegmented() boolean IOEngine#doesAllocationCrossSegments(long offset, long len) BucketCache calls these methods to determine if a 'contiguous' allocation of a particular block size can occur. It does this by checking if doesAllocationCrossSegments(offset, len) is true. If an allocation crosses a segment, another call to allocate is made for the same block. The first block is wasted (it is marked allocated). The worst case is 1 'largest block' per file. If an allocation fails for any reason, all/any allocated blocks (including wasted ones) are freed again for subsequent allocation requests. This is very similar to a 'JBOD' configuration (there is no striping of any kind). To configure HBase to use multiple files when configuring the HBase bucket cache, set the following configurations in hbase-site.xml: ... hbase.bucketcache.ioengine files:/mnt/disk1/bucketcache,/mnt/disk2/bucketcache,/mnt/disk3/bucketcache,/mnt/disk4/bucketcache hbase.bucketcache.size 1048756 ... /mnt/disk1, /mnt/disk2, /mnt/disk3 and /mnt/disk4 are expected to be separate filesystems (one mountpoint per volume). HBase would need to have read/write permissions to these directories. In this configuration, 4 sparsely allocated 256GB files (each named 'bucketcache') are created in each directory on HBase RegionServer startup. Ideally such a configuration is useful when the underlying HBase filesystem is non-local. was: The following patch adds a couple of features without making any big changes to the existing BucketCache allocator. This approach introduces the notion of 'segmentation' (or in other words, the underlying IOEngine can be made up of non-contiguous segments. Two methods are added to expose this information to the BucketCache allocator. boolean IOEngine#isSegmented() boolean IOEngine#doesAllocationCrossSegments(long offset, long len) BucketCache calls these methods to determine if a 'contiguous' allocation of a particular block size can occur. It does this by checking if doesAllocationCrossSegments(offset, len) is true. If an allocation crosses a segment, another call to allocate is made for the same block. The first block is wasted (it is marked allocated). The worst case is 1 'largest block' per file. If an allocation fails for any reason, all/any allocated blocks (including wasted ones) are freed again for subsequent allocation requests. This is very similar to a 'JBOD' configuration (there is no striping of any kind). To configure HBase to use multiple files when configuring the HBase bucket cache, set the following configurations in hbase-site.xml: ... hbase.bucketcache.ioengine files:/mnt/disk1/bucketcache,/mnt/disk2/bucketcache,/mnt/disk3/bucketcache,/mnt/disk4/bucketcache hbase.bucketcache.size 1048756 ... /mnt/disk1, /mnt/disk2, /mnt/disk3 and /mnt/disk4 are expected to be separate filesystems (one mountpoint per volume). HBase would need to have read/write permissions to these directories. In this configuration, 4 sparsely allocated 256GB files (each named 'bucketcache') are created in each directory on HBase RegionServer startup. Ideally such a configuration is useful when the underlying HBase filesystem is non-local. There are also some additional fixes to the previous patch: 1) The 'total size' aligns with the 'total aggregate file size'. This is done by doing a ceiling division, and rounding up the 'totalSize' so that each segment is equally sized. segmentSize = ceil(totalSize / numFiles) totalSize = segmentSize * numFiles 2) All failed allocations, including extra ones made due to crossing segments, are cleaned up > Allow more than one backing file in bucketcache > --- > > Key: HBASE-15314 > URL: https://issues.apache.org/jira/browse/HBASE-15314 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack >Assignee: Aaron Tokhy > Attachments: HBASE-15314-v2.patch, HBASE-15314-v3.patch, > HBASE-15314.patch > > > Allow bucketcache use more than just one backing file: e.g. chassis has more > than one SSD in it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15314) Allow more than one backing file in bucketcache
[ https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Tokhy updated HBASE-15314: Release Note: The following patch adds a couple of features without making any big changes to the existing BucketCache allocator. This approach introduces the notion of 'segmentation' (or in other words, the underlying IOEngine can be made up of non-contiguous segments. Two methods are added to expose this information to the BucketCache allocator. boolean IOEngine#isSegmented() boolean IOEngine#doesAllocationCrossSegments(long offset, long len) BucketCache calls these methods to determine if a 'contiguous' allocation of a particular block size can occur. It does this by checking if doesAllocationCrossSegments(offset, len) is true. If an allocation crosses a segment, another call to allocate is made for the same block. The first block is wasted (it is marked allocated). The worst case is 1 'largest block' per file. If an allocation fails for any reason, all/any allocated blocks (including wasted ones) are freed again for subsequent allocation requests. This is very similar to a 'JBOD' configuration (there is no striping of any kind). To configure HBase to use multiple files when configuring the HBase bucket cache, set the following configurations in hbase-site.xml: ... hbase.bucketcache.ioengine files:/mnt/disk1/bucketcache,/mnt/disk2/bucketcache,/mnt/disk3/bucketcache,/mnt/disk4/bucketcache hbase.bucketcache.size 1048756 ... /mnt/disk1, /mnt/disk2, /mnt/disk3 and /mnt/disk4 are expected to be separate filesystems (one mountpoint per volume). HBase would need to have read/write permissions to these directories. In this configuration, 4 sparsely allocated 256GB files (each named 'bucketcache') are created in each directory on HBase RegionServer startup. Ideally such a configuration is useful when the underlying HBase filesystem is non-local. There are also some additional fixes to the previous patch: 1) The 'total size' aligns with the 'total aggregate file size'. This is done by doing a ceiling division, and rounding up the 'totalSize' so that each segment is equally sized. segmentSize = ceil(totalSize / numFiles) totalSize = segmentSize * numFiles 2) All failed allocations, including extra ones made due to crossing segments, are cleaned up was: The following patch adds a couple of features without making any big changes to the existing BucketCache allocator. This approach introduces the notion of 'segmentation' (or in other words, the underlying IOEngine can be made up of non-contiguous segments. Two methods are added to expose this information to the BucketCache allocator. boolean IOEngine#isSegmented() boolean IOEngine#doesAllocationCrossSegments(long offset, long len) BucketCache calls these methods to determine if a 'contiguous' allocation of a particular block size can occur. It does this by checking if doesAllocationCrossSegments(offset, len) is true. If an allocation crosses a segment, another call to allocate is made for the same block. The first block is wasted (it is marked allocated). The worst case is 1 'largest block' per file. If an allocation fails for any reason, all/any allocated blocks (including wasted ones) are freed again for subsequent allocation requests. This is very similar to a 'JBOD' configuration (there is no striping of any kind). To configure HBase to use multiple files when configuring the HBase bucket cache, There are also some additional fixes to the previous patch: 1) The 'total size' aligns with the 'total aggregate file size'. This is done by doing a ceiling division, and rounding up the 'totalSize' so that each segment is equally sized. segmentSize = ceil(totalSize / numFiles) totalSize = segmentSize * numFiles 2) All failed allocations, including extra ones made due to crossing segments, are cleaned up > Allow more than one backing file in bucketcache > --- > > Key: HBASE-15314 > URL: https://issues.apache.org/jira/browse/HBASE-15314 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack >Assignee: Aaron Tokhy > Attachments: HBASE-15314-v2.patch, HBASE-15314-v3.patch, > HBASE-15314.patch > > > Allow bucketcache use more than just one backing file: e.g. chassis has more > than one SSD in it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15314) Allow more than one backing file in bucketcache
[ https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Tokhy updated HBASE-15314: Release Note: The following patch adds a couple of features without making any big changes to the existing BucketCache allocator. This approach introduces the notion of 'segmentation' (or in other words, the underlying IOEngine can be made up of non-contiguous segments. Two methods are added to expose this information to the BucketCache allocator. boolean IOEngine#isSegmented() boolean IOEngine#doesAllocationCrossSegments(long offset, long len) BucketCache calls these methods to determine if a 'contiguous' allocation of a particular block size can occur. It does this by checking if doesAllocationCrossSegments(offset, len) is true. If an allocation crosses a segment, another call to allocate is made for the same block. The first block is wasted (it is marked allocated). The worst case is 1 'largest block' per file. If an allocation fails for any reason, all/any allocated blocks (including wasted ones) are freed again for subsequent allocation requests. This is very similar to a 'JBOD' configuration (there is no striping of any kind). To configure HBase to use multiple files when configuring the HBase bucket cache, There are also some additional fixes to the previous patch: 1) The 'total size' aligns with the 'total aggregate file size'. This is done by doing a ceiling division, and rounding up the 'totalSize' so that each segment is equally sized. segmentSize = ceil(totalSize / numFiles) totalSize = segmentSize * numFiles 2) All failed allocations, including extra ones made due to crossing segments, are cleaned up was: The following patch adds a couple of features without making any big changes to the existing BucketCache allocator. This approach introduces the notion of 'segmentation' (or in other words, the underlying IOEngine can be made up of non-contiguous segments. Two methods are added to expose this information to the BucketCache allocator. boolean IOEngine#isSegmented() boolean IOEngine#doesAllocationCrossSegments(long offset, long len) BucketCache calls these methods to determine if a 'contiguous' allocation of a particular block size can occur. It does this by checking if doesAllocationCrossSegments(offset, len) is true. If an allocation crosses a segment, another call to allocate is made for the same block. The first block is wasted (it is marked allocated). The worst case is 1 'largest block' per file. If an allocation fails for any reason, all/any allocated blocks (including wasted ones) are freed again for subsequent allocation requests. This is very similar to a 'JBOD' configuration (there is no striping of any kind). There are also some additional fixes: 1) The 'total size' aligns with the 'total aggregate file size'. This is done by doing a ceiling division, and rounding up the 'totalSize' so that each segment is equally sized. segmentSize = ceil(totalSize / numFiles) totalSize = segmentSize * numFiles 2) All failed allocations, including extra ones made due to crossing segments, are cleaned up > Allow more than one backing file in bucketcache > --- > > Key: HBASE-15314 > URL: https://issues.apache.org/jira/browse/HBASE-15314 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack >Assignee: Aaron Tokhy > Attachments: HBASE-15314-v2.patch, HBASE-15314-v3.patch, > HBASE-15314.patch > > > Allow bucketcache use more than just one backing file: e.g. chassis has more > than one SSD in it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15314) Allow more than one backing file in bucketcache
[ https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15314: -- Release Note: The following patch adds a couple of features without making any big changes to the existing BucketCache allocator. This approach introduces the notion of 'segmentation' (or in other words, the underlying IOEngine can be made up of non-contiguous segments. Two methods are added to expose this information to the BucketCache allocator. boolean IOEngine#isSegmented() boolean IOEngine#doesAllocationCrossSegments(long offset, long len) BucketCache calls these methods to determine if a 'contiguous' allocation of a particular block size can occur. It does this by checking if doesAllocationCrossSegments(offset, len) is true. If an allocation crosses a segment, another call to allocate is made for the same block. The first block is wasted (it is marked allocated). The worst case is 1 'largest block' per file. If an allocation fails for any reason, all/any allocated blocks (including wasted ones) are freed again for subsequent allocation requests. This is very similar to a 'JBOD' configuration (there is no striping of any kind). There are also some additional fixes: 1) The 'total size' aligns with the 'total aggregate file size'. This is done by doing a ceiling division, and rounding up the 'totalSize' so that each segment is equally sized. segmentSize = ceil(totalSize / numFiles) totalSize = segmentSize * numFiles 2) All failed allocations, including extra ones made due to crossing segments, are cleaned up was: The following patch adds a couple of features without making any big changes to the existing BucketCache allocator. This approach introduces the notion of 'segmentation' (or in other words, the underlying IOEngine can be made up of non-contiguous segments. Two methods are added to expose this information to the BucketCache allocator. boolean IOEngine#isSegmented() boolean IOEngine#doesAllocationCrossSegments(long offset, long len) BucketCache calls these methods to determine if a 'contiguous' allocation of a particular block size can occur. It does this by checking if doesAllocationCrossSegments(offset, len) is true. If an allocation crosses a segment, another call to allocate is made for the same block. The first block is wasted (it is marked allocated). The worst case is 1 'largest block' per file. If an allocation fails for any reason, all/any allocated blocks (including wasted ones) are freed again for subsequent allocation requests. This is very similar to a 'JBOD' configuration (there is no striping of any kind). There are also some additional fixes: 1) The 'total size' aligns with the 'total aggregate file size'. This is done by doing a ceiling division, and rounding up the 'totalSize' so that each segment is equally sized. segmentSize = ceil(totalSize / numFiles) totalSize = segmentSize * numFiles 2) All failed allocations, including extra ones made due to crossing segments, are cleaned up > Allow more than one backing file in bucketcache > --- > > Key: HBASE-15314 > URL: https://issues.apache.org/jira/browse/HBASE-15314 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack >Assignee: Aaron Tokhy > Attachments: HBASE-15314-v2.patch, HBASE-15314-v3.patch, > HBASE-15314.patch > > > Allow bucketcache use more than just one backing file: e.g. chassis has more > than one SSD in it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15314) Allow more than one backing file in bucketcache
[ https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15314: -- Release Note: The following patch adds a couple of features without making any big changes to the existing BucketCache allocator. This approach introduces the notion of 'segmentation' (or in other words, the underlying IOEngine can be made up of non-contiguous segments. Two methods are added to expose this information to the BucketCache allocator. boolean IOEngine#isSegmented() boolean IOEngine#doesAllocationCrossSegments(long offset, long len) BucketCache calls these methods to determine if a 'contiguous' allocation of a particular block size can occur. It does this by checking if doesAllocationCrossSegments(offset, len) is true. If an allocation crosses a segment, another call to allocate is made for the same block. The first block is wasted (it is marked allocated). The worst case is 1 'largest block' per file. If an allocation fails for any reason, all/any allocated blocks (including wasted ones) are freed again for subsequent allocation requests. This is very similar to a 'JBOD' configuration (there is no striping of any kind). There are also some additional fixes: 1) The 'total size' aligns with the 'total aggregate file size'. This is done by doing a ceiling division, and rounding up the 'totalSize' so that each segment is equally sized. segmentSize = ceil(totalSize / numFiles) totalSize = segmentSize * numFiles 2) All failed allocations, including extra ones made due to crossing segments, are cleaned up Move description to release note. > Allow more than one backing file in bucketcache > --- > > Key: HBASE-15314 > URL: https://issues.apache.org/jira/browse/HBASE-15314 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack >Assignee: Aaron Tokhy > Attachments: HBASE-15314-v2.patch, HBASE-15314-v3.patch, > HBASE-15314.patch > > > Allow bucketcache use more than just one backing file: e.g. chassis has more > than one SSD in it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15314) Allow more than one backing file in bucketcache
[ https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15314: -- Status: Patch Available (was: Open) > Allow more than one backing file in bucketcache > --- > > Key: HBASE-15314 > URL: https://issues.apache.org/jira/browse/HBASE-15314 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack >Assignee: Aaron Tokhy > Attachments: HBASE-15314-v2.patch, HBASE-15314-v3.patch, > HBASE-15314.patch > > > Allow bucketcache use more than just one backing file: e.g. chassis has more > than one SSD in it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15314) Allow more than one backing file in bucketcache
[ https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Tokhy updated HBASE-15314: Attachment: HBASE-15314-v3.patch > Allow more than one backing file in bucketcache > --- > > Key: HBASE-15314 > URL: https://issues.apache.org/jira/browse/HBASE-15314 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack >Assignee: Aaron Tokhy > Attachments: HBASE-15314-v2.patch, HBASE-15314-v3.patch, > HBASE-15314.patch > > > Allow bucketcache use more than just one backing file: e.g. chassis has more > than one SSD in it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15314) Allow more than one backing file in bucketcache
[ https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-15314: --- Assignee: Aaron Tokhy (was: Amal Joshy) > Allow more than one backing file in bucketcache > --- > > Key: HBASE-15314 > URL: https://issues.apache.org/jira/browse/HBASE-15314 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack >Assignee: Aaron Tokhy > Attachments: HBASE-15314-v2.patch, HBASE-15314.patch > > > Allow bucketcache use more than just one backing file: e.g. chassis has more > than one SSD in it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15314) Allow more than one backing file in bucketcache
[ https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amal Joshy updated HBASE-15314: --- Attachment: HBASE-15314-v2.patch > Allow more than one backing file in bucketcache > --- > > Key: HBASE-15314 > URL: https://issues.apache.org/jira/browse/HBASE-15314 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack >Assignee: Amal Joshy > Attachments: HBASE-15314-v2.patch, HBASE-15314.patch > > > Allow bucketcache use more than just one backing file: e.g. chassis has more > than one SSD in it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15314) Allow more than one backing file in bucketcache
[ https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amal Joshy updated HBASE-15314: --- Attachment: HBASE-15314.patch > Allow more than one backing file in bucketcache > --- > > Key: HBASE-15314 > URL: https://issues.apache.org/jira/browse/HBASE-15314 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack >Assignee: Amal Joshy > Attachments: HBASE-15314.patch > > > Allow bucketcache use more than just one backing file: e.g. chassis has more > than one SSD in it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
