[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426300#comment-16426300
 ] 

stack commented on HBASE-15314:
---

Got committed with a wonky JIRA ID

commit e67eb6c424d76ee259f5076c277454a73e3a2bf4
Author: Ramkrishna 
Date:   Thu Mar 16 16:11:35 2017 +0530

HBSE-15314 Allow more than one backing file in bucketcache (Chunhui Shen)

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: chunhui shen
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: FileIOEngine.java, HBASE-15314-v2.patch, 
> HBASE-15314-v3.patch, HBASE-15314-v4.patch, HBASE-15314-v5.patch, 
> HBASE-15314-v6.patch, HBASE-15314-v7.patch, HBASE-15314-v8.patch, 
> HBASE-15314.master.001.patch, HBASE-15314.master.001.patch, HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.
> Usage (Setting the following configurations in hbase-site.xml):
> {quote}
> 
>   hbase.bucketcache.ioengine
>   
> files:/mnt/disk1/bucketcache,/mnt/disk2/bucketcache,/mnt/disk3/bucketcache,/mnt/disk4/bucketcache
> 
> 
>   hbase.bucketcache.size
>   1048576
> 
> {quote}
> The above setting means the total capacity of cache is 1048576MB(1TB), each 
> file length will be set to 0.25TB.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15949162#comment-15949162
 ] 

Hudson commented on HBASE-15314:


SUCCESS: Integrated in Jenkins build HBase-1.3-JDK7 #135 (See 
[https://builds.apache.org/job/HBase-1.3-JDK7/135/])
HBASE-17837 Backport HBASE-15314 to branch-1.3  (Chunhui shen & (ramkrishna: 
rev 36da029edd596bf543a8f6eee3005c55eb313644)
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/FileIOEngine.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/bucket/TestFileIOEngine.java


> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: chunhui shen
> Fix For: 2.0
>
> Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, 
> HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, 
> HBASE-15314-v3.patch, HBASE-15314-v4.patch, HBASE-15314-v5.patch, 
> HBASE-15314-v6.patch, HBASE-15314-v7.patch, HBASE-15314-v8.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.
> Usage (Setting the following configurations in hbase-site.xml):
> {quote}
> 
>   hbase.bucketcache.ioengine
>   
> files:/mnt/disk1/bucketcache,/mnt/disk2/bucketcache,/mnt/disk3/bucketcache,/mnt/disk4/bucketcache
> 
> 
>   hbase.bucketcache.size
>   1048576
> 
> {quote}
> The above setting means the total capacity of cache is 1048576MB(1TB), each 
> file length will be set to 0.25TB.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15949130#comment-15949130
 ] 

Hudson commented on HBASE-15314:


SUCCESS: Integrated in Jenkins build HBase-1.3-JDK8 #144 (See 
[https://builds.apache.org/job/HBase-1.3-JDK8/144/])
HBASE-17837 Backport HBASE-15314 to branch-1.3  (Chunhui shen & (ramkrishna: 
rev 36da029edd596bf543a8f6eee3005c55eb313644)
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/FileIOEngine.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/bucket/TestFileIOEngine.java


> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: chunhui shen
> Fix For: 2.0
>
> Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, 
> HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, 
> HBASE-15314-v3.patch, HBASE-15314-v4.patch, HBASE-15314-v5.patch, 
> HBASE-15314-v6.patch, HBASE-15314-v7.patch, HBASE-15314-v8.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.
> Usage (Setting the following configurations in hbase-site.xml):
> {quote}
> 
>   hbase.bucketcache.ioengine
>   
> files:/mnt/disk1/bucketcache,/mnt/disk2/bucketcache,/mnt/disk3/bucketcache,/mnt/disk4/bucketcache
> 
> 
>   hbase.bucketcache.size
>   1048576
> 
> {quote}
> The above setting means the total capacity of cache is 1048576MB(1TB), each 
> file length will be set to 0.25TB.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15949094#comment-15949094
 ] 

Hudson commented on HBASE-15314:


SUCCESS: Integrated in Jenkins build HBase-1.4 #684 (See 
[https://builds.apache.org/job/HBase-1.4/684/])
HBASE-17837 Backport HBASE-15314 to branch-1.3  (Chunhui shen & (ramkrishna: 
rev 589a0e2efb7b604d5f13f186143445b57438db0b)
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/bucket/TestFileIOEngine.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/FileIOEngine.java


> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: chunhui shen
> Fix For: 2.0
>
> Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, 
> HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, 
> HBASE-15314-v3.patch, HBASE-15314-v4.patch, HBASE-15314-v5.patch, 
> HBASE-15314-v6.patch, HBASE-15314-v7.patch, HBASE-15314-v8.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.
> Usage (Setting the following configurations in hbase-site.xml):
> {quote}
> 
>   hbase.bucketcache.ioengine
>   
> files:/mnt/disk1/bucketcache,/mnt/disk2/bucketcache,/mnt/disk3/bucketcache,/mnt/disk4/bucketcache
> 
> 
>   hbase.bucketcache.size
>   1048576
> 
> {quote}
> The above setting means the total capacity of cache is 1048576MB(1TB), each 
> file length will be set to 0.25TB.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15948921#comment-15948921
 ] 

Hudson commented on HBASE-15314:


SUCCESS: Integrated in Jenkins build HBase-1.3-IT #17 (See 
[https://builds.apache.org/job/HBase-1.3-IT/17/])
HBASE-17837 Backport HBASE-15314 to branch-1.3  (Chunhui shen & (ramkrishna: 
rev 36da029edd596bf543a8f6eee3005c55eb313644)
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/FileIOEngine.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/bucket/TestFileIOEngine.java


> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: chunhui shen
> Fix For: 2.0
>
> Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, 
> HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, 
> HBASE-15314-v3.patch, HBASE-15314-v4.patch, HBASE-15314-v5.patch, 
> HBASE-15314-v6.patch, HBASE-15314-v7.patch, HBASE-15314-v8.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.
> Usage (Setting the following configurations in hbase-site.xml):
> {quote}
> 
>   hbase.bucketcache.ioengine
>   
> files:/mnt/disk1/bucketcache,/mnt/disk2/bucketcache,/mnt/disk3/bucketcache,/mnt/disk4/bucketcache
> 
> 
>   hbase.bucketcache.size
>   1048576
> 
> {quote}
> The above setting means the total capacity of cache is 1048576MB(1TB), each 
> file length will be set to 0.25TB.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941591#comment-15941591
 ] 

ramkrishna.s.vasudevan commented on HBASE-15314:


We need this for 1.3 for some testing purpose. Let us know if we can take up 
the back port. Excuse typos.

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: chunhui shen
> Fix For: 2.0
>
> Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, 
> HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, 
> HBASE-15314-v3.patch, HBASE-15314-v4.patch, HBASE-15314-v5.patch, 
> HBASE-15314-v6.patch, HBASE-15314-v7.patch, HBASE-15314-v8.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.
> Usage (Setting the following configurations in hbase-site.xml):
> {quote}
> 
>   hbase.bucketcache.ioengine
>   
> files:/mnt/disk1/bucketcache,/mnt/disk2/bucketcache,/mnt/disk3/bucketcache,/mnt/disk4/bucketcache
> 
> 
>   hbase.bucketcache.size
>   1048576
> 
> {quote}
> The above setting means the total capacity of cache is 1048576MB(1TB), each 
> file length will be set to 0.25TB.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[chunhui shen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15940586#comment-15940586
 ] 

chunhui shen commented on HBASE-15314:
--

[~zyork] 
Could you help to backport it?  The patch should be nearly same  with trunk's.
I'm not convenient these days.

Thank you very much!

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: chunhui shen
> Fix For: 2.0
>
> Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, 
> HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, 
> HBASE-15314-v3.patch, HBASE-15314-v4.patch, HBASE-15314-v5.patch, 
> HBASE-15314-v6.patch, HBASE-15314-v7.patch, HBASE-15314-v8.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.
> Usage (Setting the following configurations in hbase-site.xml):
> {quote}
> 
>   hbase.bucketcache.ioengine
>   
> files:/mnt/disk1/bucketcache,/mnt/disk2/bucketcache,/mnt/disk3/bucketcache,/mnt/disk4/bucketcache
> 
> 
>   hbase.bucketcache.size
>   1048576
> 
> {quote}
> The above setting means the total capacity of cache is 1048576MB(1TB), each 
> file length will be set to 0.25TB.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939639#comment-15939639
 ] 

Anoop Sam John commented on HBASE-15314:


+1 to backport this to branch-1

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: chunhui shen
> Fix For: 2.0
>
> Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, 
> HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, 
> HBASE-15314-v3.patch, HBASE-15314-v4.patch, HBASE-15314-v5.patch, 
> HBASE-15314-v6.patch, HBASE-15314-v7.patch, HBASE-15314-v8.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.
> Usage (Setting the following configurations in hbase-site.xml):
> {quote}
> 
>   hbase.bucketcache.ioengine
>   
> files:/mnt/disk1/bucketcache,/mnt/disk2/bucketcache,/mnt/disk3/bucketcache,/mnt/disk4/bucketcache
> 
> 
>   hbase.bucketcache.size
>   1048576
> 
> {quote}
> The above setting means the total capacity of cache is 1048576MB(1TB), each 
> file length will be set to 0.25TB.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Zach York (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939356#comment-15939356
 ] 

Zach York commented on HBASE-15314:
---

[~zjushch] Any chance we can get this backported to branch-1? Otherwise, I can 
try to take a look soon.

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: chunhui shen
> Fix For: 2.0
>
> Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, 
> HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, 
> HBASE-15314-v3.patch, HBASE-15314-v4.patch, HBASE-15314-v5.patch, 
> HBASE-15314-v6.patch, HBASE-15314-v7.patch, HBASE-15314-v8.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.
> Usage (Setting the following configurations in hbase-site.xml):
> {quote}
> 
>   hbase.bucketcache.ioengine
>   
> files:/mnt/disk1/bucketcache,/mnt/disk2/bucketcache,/mnt/disk3/bucketcache,/mnt/disk4/bucketcache
> 
> 
>   hbase.bucketcache.size
>   1048576
> 
> {quote}
> The above setting means the total capacity of cache is 1048576MB(1TB), each 
> file length will be set to 0.25TB.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[chunhui shen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927306#comment-15927306
 ] 

chunhui shen commented on HBASE-15314:
--

[~ram_krish]
Please help to commit it,  thanks   :)

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Aaron Tokhy
> Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, 
> HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, 
> HBASE-15314-v3.patch, HBASE-15314-v4.patch, HBASE-15314-v5.patch, 
> HBASE-15314-v6.patch, HBASE-15314-v7.patch, HBASE-15314-v8.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925855#comment-15925855
 ] 

ramkrishna.s.vasudevan commented on HBASE-15314:


[~zjushch]
I think there are no objections and I think you can go ahead and commit the 
patch if you have the time to do it.

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Aaron Tokhy
> Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, 
> HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, 
> HBASE-15314-v3.patch, HBASE-15314-v4.patch, HBASE-15314-v5.patch, 
> HBASE-15314-v6.patch, HBASE-15314-v7.patch, HBASE-15314-v8.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15923740#comment-15923740
 ] 

ramkrishna.s.vasudevan commented on HBASE-15314:


Since all the comments have been fixed. I plan to commit this today  night my 
time. So pls speak up if there are any other objections. Thanks [~zjushch].

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Aaron Tokhy
> Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, 
> HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, 
> HBASE-15314-v3.patch, HBASE-15314-v4.patch, HBASE-15314-v5.patch, 
> HBASE-15314-v6.patch, HBASE-15314-v7.patch, HBASE-15314-v8.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15923566#comment-15923566
 ] 

Hadoop QA commented on HBASE-15314:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
5s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
27m 7s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 101m 12s 
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
16s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 140m 43s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12858602/HBASE-15314-v8.patch |
| JIRA Issue | HBASE-15314 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux f78e3634e2ed 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 201c838 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/6086/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/6086/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Aaron Tokhy
> Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, 
> HBASE-15314.master.001.patch, 

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15907770#comment-15907770
 ] 

ramkrishna.s.vasudevan commented on HBASE-15314:


[~zjushch]
There was one more comment in RB. Are you going to add a new patch here fixing 
it? If there is a consensus we can commit this and close this JIRA.

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Aaron Tokhy
> Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, 
> HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, 
> HBASE-15314-v3.patch, HBASE-15314-v4.patch, HBASE-15314-v5.patch, 
> HBASE-15314-v6.patch, HBASE-15314-v7.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15899836#comment-15899836
 ] 

Hadoop QA commented on HBASE-15314:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
26m 43s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 111m 24s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
19s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 150m 42s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.balancer.TestStochasticLoadBalancer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12856620/HBASE-15314-v7.patch |
| JIRA Issue | HBASE-15314 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 054b83598131 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / b635414 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/5986/artifact/patchprocess/patch-unit-hbase-server.txt
 |
| unit test logs |  
https://builds.apache.org/job/PreCommit-HBASE-Build/5986/artifact/patchprocess/patch-unit-hbase-server.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/5986/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/5986/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> 

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15899778#comment-15899778
 ] 

Anoop Sam John commented on HBASE-15314:


Oh did not notice ur comment.  

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Aaron Tokhy
> Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, 
> HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, 
> HBASE-15314-v3.patch, HBASE-15314-v4.patch, HBASE-15314-v5.patch, 
> HBASE-15314-v6.patch, HBASE-15314-v7.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15899733#comment-15899733
 ] 

stack commented on HBASE-15314:
---

What I said above [~anoop.hbase] "I started to look at rb but looking at the 
quality of the comments by ramkrishna vasudevan, Aaron Tokhy and Zach York, I'd 
only be getting in the way." -- smile.

I took a look anyways. Patch looks nice and clean.

Needs a release note if only to describe new 'files' option. Should also talk 
about behavior laying down blocks, the 'striping' effect.

[~zyork] [~aartokhy] you good w/ Chunhui's patch?

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Aaron Tokhy
> Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, 
> HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, 
> HBASE-15314-v3.patch, HBASE-15314-v4.patch, HBASE-15314-v5.patch, 
> HBASE-15314-v6.patch, HBASE-15314-v7.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15899660#comment-15899660
 ] 

Anoop Sam John commented on HBASE-15314:


+1 from me.  Want to look at this [~saint@gmail.com]?  

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Aaron Tokhy
> Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, 
> HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, 
> HBASE-15314-v3.patch, HBASE-15314-v4.patch, HBASE-15314-v5.patch, 
> HBASE-15314-v6.patch, HBASE-15314-v7.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15899298#comment-15899298
 ] 

ramkrishna.s.vasudevan commented on HBASE-15314:


Is this good to be committed now?
Any more comments.

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Aaron Tokhy
> Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, 
> HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, 
> HBASE-15314-v3.patch, HBASE-15314-v4.patch, HBASE-15314-v5.patch, 
> HBASE-15314-v6.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898772#comment-15898772
 ] 

Hadoop QA commented on HBASE-15314:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
27m 1s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 126m 35s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
16s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 165m 59s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestHRegionWithInMemoryFlush |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12856419/HBASE-15314-v6.patch |
| JIRA Issue | HBASE-15314 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 5a6e87738b81 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 
10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 81cb298 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/5975/artifact/patchprocess/patch-unit-hbase-server.txt
 |
| unit test logs |  
https://builds.apache.org/job/PreCommit-HBASE-Build/5975/artifact/patchprocess/patch-unit-hbase-server.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/5975/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/5975/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
>   

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896787#comment-15896787
 ] 

ramkrishna.s.vasudevan commented on HBASE-15314:


Will commit this latest patch from [~zjushch] - if there are no further 
comments.

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Aaron Tokhy
> Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, 
> HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, 
> HBASE-15314-v3.patch, HBASE-15314-v4.patch, HBASE-15314-v5.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893741#comment-15893741
 ] 

ramkrishna.s.vasudevan commented on HBASE-15314:


[~aartokhy], [~zyork]
What do you feel about the latest patch in rb? If you are all fine, then shall 
we go ahead with [~zjushch]'s patch?

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Aaron Tokhy
> Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, 
> HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, 
> HBASE-15314-v3.patch, HBASE-15314-v4.patch, HBASE-15314-v5.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893558#comment-15893558
 ] 

stack commented on HBASE-15314:
---

I started to look at rb but looking at the quality of the comments by 
[~ramkrishna.s.vasude...@gmail.com], [~aartokhy] and [~zyork], I'd only be 
getting in the way.

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Aaron Tokhy
> Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, 
> HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, 
> HBASE-15314-v3.patch, HBASE-15314-v4.patch, HBASE-15314-v5.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887406#comment-15887406
 ] 

ramkrishna.s.vasudevan commented on HBASE-15314:


Posted some questions in RB [~zjushch].
Ping [~saint@gmail.com], [~aartokhy], [~zyork] - What do  you think of the 
latest patch?.

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Aaron Tokhy
> Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, 
> HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, 
> HBASE-15314-v3.patch, HBASE-15314-v4.patch, HBASE-15314-v5.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[chunhui shen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15884497#comment-15884497
 ] 

chunhui shen commented on HBASE-15314:
--

{quote}
Shouldn't sizePerFile be adjusted since total space is lower than sizePerFile ?
When total space is lower than sizePerFile, the above would raise exception, 
right ?
{quote}
Yes, if the space is less than sizePerFile,  FileIOEngine will throw exception 
when initializing,  it's unnecessary to adjust the sizePerFile.

{quote}
For getFileNum(long offset), RuntimeException may be thrown. Should it declare 
to throw IOE ?
{quote}
It won't be thrown as expected, RuntimeException  seems ok.


{quote}
+   * Get the absolute offset in given file with the relative global offset.
What does "relative global" mean ?
{quote}
The FileIOEngine provide a logic contiguous storage  with several physical 
files,  the global offset is oriented to the logic storage.

Thanks Ted, Other comments are applied in patch-v5.

Move to RB https://reviews.apache.org/r/57068/ for more review.


> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Aaron Tokhy
> Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, 
> HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, 
> HBASE-15314-v3.patch, HBASE-15314-v4.patch, HBASE-15314-v5.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15884242#comment-15884242
 ] 

Hadoop QA commented on HBASE-15314:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
6s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
27m 43s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 95m 42s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
15s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 136m 11s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12854658/HBASE-15314-v4.patch |
| JIRA Issue | HBASE-15314 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux ae48b6364038 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / f444b3b |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| whitespace | 
https://builds.apache.org/job/PreCommit-HBASE-Build/5837/artifact/patchprocess/whitespace-eol.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/5837/artifact/patchprocess/patch-unit-hbase-server.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/5837/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/5837/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> 

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15884214#comment-15884214
 ] 

Ted Yu commented on HBASE-15314:


{code}
+  return new FileIOEngine(capacity, 
ioEngineName.substring(5).split(FileIOEngine.FILE_DELIMITER));
...
+write(ByteBuffer.wrap(srcBuffer.array(), srcBuffer.arrayOffset(), 
srcBuffer.remaining()), offset);
{code}
Wrap long line.
{code}
+  + " total space under " + filePath + ", not enough for applied "
{code}
"applied" -> "requested"

For the if block:
{code}
+if (totalSpace < sizePerFile) {
{code}
Shouldn't sizePerFile be adjusted since total space is lower than sizePerFile ?
{code}
+rafs[i].setLength(sizePerFile);
{code}
When total space is lower than sizePerFile, the above would raise exception, 
right ?

For shutdown(), should the try block be moved inside the for loop ?
{code}
+   * Get the absolute offset in given file with the relative global offset.
{code}
What does "relative global" mean ?

For getFileNum(long offset), RuntimeException may be thrown. Should it declare 
to throw IOE ?


> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Aaron Tokhy
> Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, 
> HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, 
> HBASE-15314-v3.patch, HBASE-15314-v4.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[chunhui shen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15884201#comment-15884201
 ] 

chunhui shen commented on HBASE-15314:
--

[~ram_krish]
With the uploaded v4 patch,  we could config the single file or multi-files as 
the following
{quote}
 hbase.bucketcache.ioengine 
 file:/mnt/disk1/bucketcache

 hbase.bucketcache.ioengine 
 
file:/mnt/disk1/bucketcache,/mnt/disk2/bucketcache,/mnt/disk3/bucketcache
{quote}

Thanks.

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Aaron Tokhy
> Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, 
> HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, 
> HBASE-15314-v3.patch, HBASE-15314-v4.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15882021#comment-15882021
 ] 

ramkrishna.s.vasudevan commented on HBASE-15314:


[~zjushch]
I think in your .java file that you have attached it is not complete. Can you 
prepare as a patch? There are some changes in BucketAllocator for 
enabling/disabling alloction range and also evict blocks in that range.

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Aaron Tokhy
> Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, 
> HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, 
> HBASE-15314-v3.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[chunhui shen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881836#comment-15881836
 ] 

chunhui shen commented on HBASE-15314:
--

bq. How would you handle the case of striping
Read the block from two files if crossed, just like the handling of 
ByteBufferIOEngine which would read block from multiple ByteBuffer.
This logic is implemented  by FileIOEngine#accessFile from the attachment 
'FileIOEngine.java'. 

Thanks

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Aaron Tokhy
> Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, 
> HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, 
> HBASE-15314-v3.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15880915#comment-15880915
 ] 

ramkrishna.s.vasudevan commented on HBASE-15314:


[~zjushch], [~zyork], [~aartokhy]
What is the decision here? 
Going thro the discussions and the patch since we have now more than one file 
backing the bucket cache we are now able to make use of all the SSDs mounted so 
it also means when we have random reads we are able to parallely access the 
data in all SSDs. But I think none of the patches here change the way the 
writes are written to the SSDs via the allocators that [~danielpol] had 
mentioned.


> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Aaron Tokhy
> Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, 
> HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, 
> HBASE-15314-v3.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[chunhui shen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850958#comment-15850958
 ] 

chunhui shen commented on HBASE-15314:
--

Sorry, I still haven't got the benefit point if  blocks are guaranteed to 
reside in a single segment.

With either of the two approaches, I think the BucketCache can still continue 
operating if device failure.

In fact, the attachment 'FileIOEngine.java ' also includes the logic of 
tolerating device failure.

bq.This does not happen in the patch I submitted, but it could be an 
improvement that you wouldn't be able to easily handle if HBase blocks crossed 
segments.
Thus, I don't t agree that it wouldn't be able to easily handle if HBase blocks 
crossed segments.

At last, I still think the BucketCache shouldn't be aware of multi-segments, 
the FileIOEngine should provide a contiguous storage with several physical 
devices,  just like RAID. 



> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Aaron Tokhy
> Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, 
> HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, 
> HBASE-15314-v3.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Aaron Tokhy (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840635#comment-15840635
 ] 

Aaron Tokhy commented on HBASE-15314:
-

bq. 1. It will take an extra IO if an allocation cross files, but the 
probability is about 32KB/320GB ≈ 1/10M ( example for block size 32KB, one file 
capacity 320GB). So, I think it's no effect for perfomance

The performance difference with either approach is negligible as you mentioned. 
 The guarantee is that an allocation is guaranteed to reside on a single file.

bq. 2. With crossing files allocation, no extra logic is needed if a single 
file fails due to an IO error. We will free the whole alloction if failed (see 
this logic from BucketCahe#writeToCache).

Yes this would then require no additional changes in BucketCache.

bq. IMPO, make things simple if no obvious benefit, I prefer to use the thought 
of 1st patch.

My patch employs almost the exact same approach as [~anoop.hbase]'s (the first 
patch) and employs the work [~Amal Joshy] added to avoid crossing file 
boundaries, while adding plenty of tests.

There are a couple of things you can gain by ensuring that HBase blocks are 
guaranteed to reside in a single segment:

- Turning off files in a JBOD configuration when an IOException occurs.  Only 
the allocations that resided in the segment that resided on the device that 
failed can be invalided, and the BucketCache can continue.  This does not 
happen in the patch I submitted, but it could be an improvement that you 
wouldn't be able to easily handle if HBase blocks crossed segments.

- Implementing striding/striping without adding too much logic in BucketCache.  
If a single 'RAID 0' volume fails (in a RAID 0 IOEngine decorator), only the 
allocations that resided in one of the drives could be invalidated, and the 
BucketCache can still continue operating.

This could be added as a 'cache invalidation on device failure' feature for 
BucketCache.

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Aaron Tokhy
> Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, 
> HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, 
> HBASE-15314-v3.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[chunhui shen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837098#comment-15837098
 ] 

chunhui shen commented on HBASE-15314:
--

IMPO, make things simple if no obvious benefit,  I prefer to use the thought of 
1st patch.

Thanks.

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Aaron Tokhy
> Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, 
> HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, 
> HBASE-15314-v3.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[chunhui shen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837087#comment-15837087
 ] 

chunhui shen commented on HBASE-15314:
--

bq.but at least we can guarantee that every allocation will reside in its own 
file
I also have considered this scene, but seems not meaningful :
1. It will take an extra IO if  an allocation cross files, but the probability 
is about 32KB/320GB ≈ 1/10M ( example for block size 32KB, one file capacity 
320GB).  So,  I think it's no effect for perfomance
2. With crossing files allocation, no extra logic is needed if a single file 
fails due to an IO error.  We will free the whole alloction if failed ( see 
this logic from BucketCahe#writeToCache).

In addition,  it seems introducing an odd stuff which is not understandable 
easily for user , from the release notes：
{quote}
The first block is wasted (it is marked allocated). The worst case is 1 
'largest block' per file. If an allocation fails for any reason, all/any 
allocated blocks (including wasted ones) are freed again for subsequent 
allocation requests. This is very similar to a 'JBOD' configuration (there is 
no striping of any kind).
{quote}


> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Aaron Tokhy
> Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, 
> HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, 
> HBASE-15314-v3.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Zach York (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837044#comment-15837044
 ] 

Zach York commented on HBASE-15314:
---

I agree with [~aartokhy] that this makes allocations safer. I think another 
approach that addresses striping should be addressed later, but I think this 
patch itself is a good approach.

[~stack] [~zjushch] [~anoop.hbase] What do you think? Can we move forward on 
this patch.

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Aaron Tokhy
> Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, 
> HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, 
> HBASE-15314-v3.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[chunhui shen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15823572#comment-15823572
 ] 

chunhui shen commented on HBASE-15314:
--

bq. So this is similar to what the 1st patches in this jira doing

Yes, but the 1st patch doesn't handle the case which read/write data cross 
multiple files.  
The function of FileIOEngine#accessFile could be copied to the patch from the 
attachment 'FileIOEngine.java'

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Aaron Tokhy
> Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, 
> HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, 
> HBASE-15314-v3.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15823546#comment-15823546
 ] 

Anoop Sam John commented on HBASE-15314:


So this is similar to what the 1st patches in this jira doing. Correct Chunhui? 
 I Agree to ur point of being simple...

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Aaron Tokhy
> Attachments: FileIOEngine.java, HBASE-15314.master.001.patch, 
> HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, 
> HBASE-15314-v3.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[chunhui shen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15823401#comment-15823401
 ] 

chunhui shen commented on HBASE-15314:
--

Sorry for the late response,
but I think it's not a good choice that adding the notion of 'segmentation', 
it will make things complex and is difficult to understand about allocating 
block for newer.

Each IOEngine should provide a logic contiguous storage even if using several 
physical  IO devices, so the design and implementation  will become simple.

Just my thought, FYI
Thanks

PS:
The uploaded FileIOEngine.java support multi-files storage which is used in our 
internal branch. 
Less than 50 lines are needed to add this feature.

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Aaron Tokhy
> Attachments: HBASE-15314.master.001.patch, 
> HBASE-15314.master.001.patch, HBASE-15314.patch, HBASE-15314-v2.patch, 
> HBASE-15314-v3.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822672#comment-15822672
 ] 

Hadoop QA commented on HBASE-15314:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
25m 38s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 84m 51s 
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
20s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 122m 17s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12847467/HBASE-15314.master.001.patch
 |
| JIRA Issue | HBASE-15314 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux f8d065f8de8f 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 9fd5dab1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/5258/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/5258/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Aaron Tokhy
> Attachments: HBASE-15314.master.001.patch, 
> HBASE-15314.master.001.patch, HBASE-15314.patch, 

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822630#comment-15822630
 ] 

stack commented on HBASE-15314:
---

Thanks for using submit-patch tool [~aartokhy].

Let me retry your patch. Failures looks totally unrelated.

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Aaron Tokhy
> Attachments: HBASE-15314.master.001.patch, HBASE-15314.patch, 
> HBASE-15314-v2.patch, HBASE-15314-v3.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822521#comment-15822521
 ] 

Hadoop QA commented on HBASE-15314:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 40s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 
31s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
5s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 58s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
38m 44s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 146m 47s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
33s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 204m 36s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.hbase.security.access.TestCoprocessorWhitelistMasterObserver |
|   | hadoop.hbase.client.TestZKAsyncRegistry |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12847410/HBASE-15314.master.001.patch
 |
| JIRA Issue | HBASE-15314 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 982ce211250f 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / cdee1a7 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/5255/artifact/patchprocess/patch-unit-hbase-server.txt
 |
| unit test logs |  
https://builds.apache.org/job/PreCommit-HBASE-Build/5255/artifact/patchprocess/patch-unit-hbase-server.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/5255/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/5255/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> Allow more than one backing file in bucketcache
> 

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Aaron Tokhy (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822282#comment-15822282
 ] 

Aaron Tokhy commented on HBASE-15314:
-

Made the appropriate whitespace fix, this time posting this patch and 
review-board posting with the nifty 'dev-support/submit-patch.py' tool

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Aaron Tokhy
> Attachments: HBASE-15314.master.001.patch, HBASE-15314.patch, 
> HBASE-15314-v2.patch, HBASE-15314-v3.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820007#comment-15820007
 ] 

Hadoop QA commented on HBASE-15314:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
54s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
55s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 52s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
19s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
36m 12s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 108m 8s 
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
22s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 160m 31s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12841815/HBASE-15314-v3.patch |
| JIRA Issue | HBASE-15314 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 709d50616ed9 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
 |
| git revision | master / f7d0f15 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace | 
https://builds.apache.org/job/PreCommit-HBASE-Build/5247/artifact/patchprocess/whitespace-eol.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/5247/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/5247/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
> 

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Aaron Tokhy (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819742#comment-15819742
 ] 

Aaron Tokhy commented on HBASE-15314:
-

Modified the release notes with information on configuring this new 'files' 
mode.

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Aaron Tokhy
> Attachments: HBASE-15314-v2.patch, HBASE-15314-v3.patch, 
> HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Aaron Tokhy (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819707#comment-15819707
 ] 

Aaron Tokhy commented on HBASE-15314:
-

Sure, I will go ahead an add a comment on the 'files:' argument, with an 
example of its usage.

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Aaron Tokhy
> Attachments: HBASE-15314-v2.patch, HBASE-15314-v3.patch, 
> HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819689#comment-15819689
 ] 

stack commented on HBASE-15314:
---

Thanks [~aartokhy] Nice. I put the above as release note. Any chance of adding 
a little on new 'files:' ioengine type saying what it offers. Could then add 
sentence on how it uses new 'segmentation' facility?

On the patch, looks good. I hit submit to see how it does against patch build. 
Thanks.



> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Aaron Tokhy
> Attachments: HBASE-15314-v2.patch, HBASE-15314-v3.patch, 
> HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Zach York (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819585#comment-15819585
 ] 

Zach York commented on HBASE-15314:
---

[~stack] Do you need any further explanation? I think eventually we should 
introduce a striping implementation, but that can be an improvement after this 
JIRA is resolved.

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Aaron Tokhy
> Attachments: HBASE-15314-v2.patch, HBASE-15314-v3.patch, 
> HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Aaron Tokhy (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812759#comment-15812759
 ] 

Aaron Tokhy commented on HBASE-15314:
-

Sorry I was out for a couple of weeks.  Here is the description:

The following patch adds a couple of features without making any big changes to 
the existing BucketCache allocator.  This approach introduces the notion of 
'segmentation' (or in other words, the underlying IOEngine can be made up of 
non-contiguous segments.  Two methods are added to expose this information to 
the BucketCache allocator.

boolean IOEngine#isSegmented()
boolean IOEngine#doesAllocationCrossSegments(long offset, long len)

BucketCache calls these methods to determine if a 'contiguous' allocation of a 
particular block size can occur.  It does this by checking if 
doesAllocationCrossSegments(offset, len) is true.  If an allocation crosses a 
segment, another call to allocate is made for the same block.  The first block 
is wasted (it is marked allocated).  The worst case is 1 'largest block' per 
file.

If an allocation fails for any reason, all/any allocated blocks (including 
wasted ones) are freed again for subsequent allocation requests.

This is very similar to a 'JBOD' configuration (there is no striping of any 
kind).

There are also some additional fixes:

1) The 'total size' aligns with the 'total aggregate file size'.  This is done 
by doing a ceiling division, and rounding up the 'totalSize' so that each 
segment is equally sized.

segmentSize = ceil(totalSize / numFiles)
totalSize = segmentSize * numFiles

2) All failed allocations, including extra ones made due to crossing segments, 
are cleaned up



> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Aaron Tokhy
> Attachments: HBASE-15314-v2.patch, HBASE-15314-v3.patch, 
> HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15737243#comment-15737243
 ] 

stack commented on HBASE-15314:
---

Sorry for the delay [~aartokhy]. Patch looks good. Any chance of a short 
description of how it works (see [~danielpol] remarks above; would be good if 
you could say what you do from his suggestion and what you do not). How do we 
fill the multiple files? Can you say more in the new methods isSegmented what 
this means for implementors?

Please fill out the release note here with an example of how to use/configure 
multifiles.

On the patch:

'files:' is a nice way of specifying the multifile ioengine.

Otherwise after a skim, it looks good.

Thanks.



> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Aaron Tokhy
> Attachments: HBASE-15314-v2.patch, HBASE-15314-v3.patch, 
> HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Zach York (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15729846#comment-15729846
 ] 

Zach York commented on HBASE-15314:
---

[~stack] [~anoop.hbase] can you take a look at the patch?

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Aaron Tokhy
> Attachments: HBASE-15314-v2.patch, HBASE-15314-v3.patch, 
> HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Aaron Tokhy (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723424#comment-15723424
 ] 

Aaron Tokhy commented on HBASE-15314:
-

Attached the patch just now.

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Aaron Tokhy
> Attachments: HBASE-15314-v2.patch, HBASE-15314-v3.patch, 
> HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15717715#comment-15717715
 ] 

Anoop Sam John commented on HBASE-15314:


Assigned to Aaron.  Can u attach the patch pls?

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Aaron Tokhy
> Attachments: HBASE-15314-v2.patch, HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716076#comment-15716076
 ] 

stack commented on HBASE-15314:
---

Oh, are you running w/ your patch?

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Amal Joshy
> Attachments: HBASE-15314-v2.patch, HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716075#comment-15716075
 ] 

stack commented on HBASE-15314:
---

I added you [~aartokhy] I started to review up on gh but probably best to do it 
here.  Use the little tool ./dev-support/submit-patch.py  Thanks for working on 
this.

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Amal Joshy
> Attachments: HBASE-15314-v2.patch, HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Zach York (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715676#comment-15715676
 ] 

Zach York commented on HBASE-15314:
---

+1 I have reviewed the patch and think it is much better than the original v2.

[~stack] can you review this? If it needs to be a patch (instead of a PR), can 
you change the assignee so that Aaron can attach the patch?

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Amal Joshy
> Attachments: HBASE-15314-v2.patch, HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Taklon Stephen Wu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15712744#comment-15712744
 ] 

Taklon Stephen Wu commented on HBASE-15314:
---

+1

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Amal Joshy
> Attachments: HBASE-15314-v2.patch, HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Aaron Tokhy (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710487#comment-15710487
 ] 

Aaron Tokhy commented on HBASE-15314:
-

I created a github pull request with my patch applied as I am unable to attach 
my patch here:
https://github.com/apache/hbase/pull/42

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Amal Joshy
> Attachments: HBASE-15314-v2.patch, HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Aaron Tokhy (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710483#comment-15710483
 ] 

Aaron Tokhy commented on HBASE-15314:
-

I created a github pull request with my patch applied as I am unable to attach 
my patch here:

https://github.com/apache/hbase/pull/42

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Amal Joshy
> Attachments: HBASE-15314-v2.patch, HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Aaron Tokhy (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691553#comment-15691553
 ] 

Aaron Tokhy commented on HBASE-15314:
-

I have started working on improving v2 of this patch to add some additional 
tests, while keeping the implementation mostly the same.  I've addressed some 
issues such as guaranteeing the total aggregate file size is greater than or 
equal to the total allocatable size (using long ceiling division).  This also 
adds some cleanup logic if an exception is thrown on allocation failure.

Included with the change is a set of parameterized tests as well as some 
changes to have TestBucketCache to also test various configurable IOEngine 
types.

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Amal Joshy
> Attachments: HBASE-15314-v2.patch, HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Amal Joshy (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15199267#comment-15199267
 ] 

Amal Joshy commented on HBASE-15314:


[~danielpol] Sorry i couldn't get back to you sooner. Thanks for pointing out 
the issue. I have attached a new patch to check if a particular block crosses 
the file boundary. If it does, a different offset will be used for its 
allocation.
 Any other suggestions to solve this issue are welcome.

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Amal Joshy
> Attachments: HBASE-15314-v2.patch, HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Daniel Pol (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15189324#comment-15189324
 ] 

Daniel Pol commented on HBASE-15314:


Getting back to the available patch. It looks like there could be an issue when 
you end up using a block that straps the file boundary. Basically when 
fileOffset + length > filesize. 

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Amal Joshy
> Attachments: HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15189247#comment-15189247
 ] 

Anoop Sam John commented on HBASE-15314:


So in order to have the clear randomness while filling in files itself, we may 
have to have all sized buckets (we have by def 14 buckets of diff sizes) in all 
files.  The buckets allocation in entire BC's capacity (offset of each bucket 
being decided) and allocation for each of the HFile block happens at an upper 
layer to IOEngine impl (BucketAllocator)..  I feel this will be a much bigger 
change.  Can we do that as a follow on issue?
In read case we will have randomness mostly.. That depends on which blocks we 
will read..

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Amal Joshy
> Attachments: HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15188804#comment-15188804
 ] 

Anoop Sam John commented on HBASE-15314:


As what I checked in BC code area, yes we are not utilizing full area of BC.  
Seems buckets in some of the specific size will not get used.. This is when u 
have all the tables with same block size.   Ya the issue with block sizes in 
HFiles is that we really can not guarentee the size...  Checking these area as 
well..

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Amal Joshy
> Attachments: HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Daniel Pol (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15187759#comment-15187759
 ] 

Daniel Pol commented on HBASE-15314:


My favorite use case is when you have the hottest table that you want to cache 
completely and make sure you get the best performance for it (and the table 
doesn't fit into RAM). Right now its a matter of adding software raid on top of 
multiple SSDs to achieve that. I would like to remove the software raid 
overhead by doing the parallelism in Hbase. I agree it's not easy, mostly 
because the Hbase blocksize is not really fixed. So you have to end up adding 
logic to handle that. 
Funny you mentioned the buckets of fixed size. I'm thinking about filling 
another JIRA related to a lot of wasted space in bucketcache because of that. 
When you have a small bucketcache that's not an issue, but when you get to a 
few TiB and you end up with half the space unused but allocated it becomes a 
serious issue.

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Amal Joshy
> Attachments: HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15187363#comment-15187363
 ] 

Anoop Sam John commented on HBASE-15314:


Amal said it.
The decision of where to write (I mean which offset) is being taken NOT within 
IOEngine impl but at BucketAllocator level.
It is NOT always true that we will 1st write and fill 1st file and then go to 
2nd.  There is a bucket allocation happening 1st.  The buckets will be having 
some fixed size. By default we have 4+1 KB, 8+1 KB... 512+1 KB sized buckets.  
(This is configurable)..  So we will make buckets and fix offset to each of 
these buckets.When a new block comes to be cached, we will 1st see which 
bucket it can go. We will pick up a size which is least size with size >= block 
size.   And we will see within that where to write. This offset is being passed.
So in order to make sure we have equi distribution while writing blocks, we 
will need change in top layer of bucket allocator.
Ya once the cache is almost filled, the read will be distributed. Write happens 
in an async way. So perf gain because of parallel writes to diff disks might 
not be that relevant.   Ya still it will be best we can distribute load from 
begin and write load also..  

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Amal Joshy
> Attachments: HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Daniel Pol (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15187053#comment-15187053
 ] 

Daniel Pol commented on HBASE-15314:


I was thinking about something that basically salts the offset. Like a modulo 
function for example.
public FileChannel getFileChannel(long offset) {
return fileChannels.get(offset % NumberOfFiles);
}


> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Amal Joshy
> Attachments: HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Amal Joshy (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186620#comment-15186620
 ] 

Amal Joshy commented on HBASE-15314:


{quote}
The point and need here is that as when you are filling up try to parallelize 
the files instead of filling it up one by one. So that initial 20G itself 
should be distributed among the four files. Is that so?
{quote}
Yes, that is what we are trying to achieve. But the problem is that the 
BucketAllocator finds us an offset to store a block and this offset will be 
generated in such a way that all the blocks will be mapped into the same file 
until it is filled completely. We cannot obtain any random writes due to this. 

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Amal Joshy
> Attachments: HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186584#comment-15186584
 ] 

ramkrishna.s.vasudevan commented on HBASE-15314:


bq.If your bucketcache usedsize is only 20GB you get the performance of a 
single physical disk with the proposed patch. If you use all files in parallel 
you get the performance of all 4 disks.
I am trying learn more here. So here the point is as and when the cache is 
filling up only then the usedSize will be growing and reaching upto 100GB 
(which is the total size of bucketCache).
The point and need here is that as when you are filling up try to parallelize 
the files instead of filling it up one by one. So that initial 20G itself 
should be distributed among the four files. Is that so?

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Amal Joshy
> Attachments: HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Amal Joshy (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186575#comment-15186575
 ] 

Amal Joshy commented on HBASE-15314:


{quote}
this is more like fileEndingOffset?
{quote}
Exactly, I'll rename it in the next patch. Thanks [~ram_krish]

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Amal Joshy
> Attachments: HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186552#comment-15186552
 ] 

ramkrishna.s.vasudevan commented on HBASE-15314:


bq.fileEndSizes
this is more like fileEndingOffset?
bq. atleast once all the files are filled, we can guarantee random reads 
utilizing all the disks.
Yes this is true. Reads are random always. 


> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Amal Joshy
> Attachments: HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186544#comment-15186544
 ] 

stack commented on HBASE-15314:
---

Thanks [~Amal Joshy]

bq. But it is not possible to obtain parallel writes with the current 
BucketAllocator implementation. 

Why is this please? A bucket can't have an associated file? That'd be breaking 
change?

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Amal Joshy
> Attachments: HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Amal Joshy (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15186497#comment-15186497
 ] 

Amal Joshy commented on HBASE-15314:


I understand your concern [~danielpol]. But it is not possible to obtain 
parallel writes with the current BucketAllocator implementation. Either we need 
to change the current implementation or plugin  a different BucketAllocator 
only for FileIOEngine. With this patch, atleast once all the files are filled, 
we can guarantee random reads utilizing all the disks.

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Amal Joshy
> Attachments: HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Daniel Pol (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15185544#comment-15185544
 ] 

Daniel Pol commented on HBASE-15314:


If I understand the code correctly, you fill one file before you switch to the 
next one. For performance reasons I would recommend filling all files at once 
in parallel. Take for example a 100GB bucketcache on 4 files (25GB per file), 
each file on a separate physical disk. If your bucketcache usedsize is only 
20GB you get the performance of a single physical disk with the proposed patch. 
If you use all files in parallel you get the performance of all 4 disks. 

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Amal Joshy
> Attachments: HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Amal Joshy (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15184443#comment-15184443
 ] 

Amal Joshy commented on HBASE-15314:


Submitted initial patch. Please review.

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Amal Joshy
> Attachments: HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15175071#comment-15175071
 ] 

Anoop Sam John commented on HBASE-15314:


Ya that just a first thought and comment..  We can include this in the current 
FileIOEngine itself.  Now we read single file path. We can allow users to pass 
a comma separated list of files.. The path of files they can give such that all 
mounted SSDs will get used.Thanks [~Amal Joshy]..  

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Amal Joshy
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15175063#comment-15175063
 ] 

stack commented on HBASE-15314:
---

Thank you [~Amal Joshy] Idea is you have more than one SSD plugged into the 
chassis and you'd like to make use of all.

[~anoop.hbase] Should it be another ioengine? Why not current file rather than 
add another choice? You just list a set of files to use?

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Amal Joshy
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Amal Joshy (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15175049#comment-15175049
 ] 

Amal Joshy commented on HBASE-15314:


I have started working on this.

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Amal Joshy
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15160211#comment-15160211
 ] 

Anoop Sam John commented on HBASE-15314:


Implement as another IOEngine (?)   MultiFile

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)