subject:"\[jira\] \[Updated\] \(HBASE\-13259\) mmap\(\) based BucketCache IOEngine"


 [ 
https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-13259:
---
Release Note: 
mmap() based bucket cache can be configured by specifying the property
{code}

  hbase.bucketcache.ioengine
  mmap://filepath

{code}
This mode of bucket cache is ideal when your file based bucket cache size is 
lesser than then available RAM. When the cache is bigger than the available RAM 
then the kernel page faults will make this cache perform lesser particularly in 
case of scans. 

  was:
mmap() based bucket cache can be configured by specifying the property
{code}

  hbase.bucketcache.ioengine
  mmap://

{code}
This mode of bucket cache is ideal when your file based bucket cache size is 
lesser than then available RAM. When the cache is bigger than the available RAM 
then the kernel page faults will make this cache perform lesser particularly in 
case of scans. 


> mmap() based BucketCache IOEngine
> -
>
> Key: HBASE-13259
> URL: https://issues.apache.org/jira/browse/HBASE-13259
> Project: HBase
>  Issue Type: New Feature
>  Components: BlockCache
>Affects Versions: 0.98.10
>Reporter: Zee Chen
>Assignee: Zee Chen
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-13259-v2.patch, HBASE-13259.patch, 
> HBASE-13259_v3.patch, HBASE-13259_v4.patch, HBASE-13259_v5.patch, 
> HBASE-13259_v6.patch, ioread-1.svg, mmap-0.98-v1.patch, mmap-1.svg, 
> mmap-trunk-v1.patch
>
>
> Of the existing BucketCache IOEngines, FileIOEngine uses pread() to copy data 
> from kernel space to user space. This is a good choice when the total working 
> set size is much bigger than the available RAM and the latency is dominated 
> by IO access. However, when the entire working set is small enough to fit in 
> the RAM, using mmap() (and subsequent memcpy()) to move data from kernel 
> space to user space is faster. I have run some short keyval gets tests and 
> the results indicate a reduction of 2%-7% of kernel CPU on my system, 
> depending on the load. On the gets, the latency histograms from mmap() are 
> identical to those from pread(), but peak throughput is close to 40% higher.
> This patch modifies ByteByfferArray to allow it to specify a backing file.
> Example for using this feature: set  hbase.bucketcache.ioengine to 
> mmap:/dev/shm/bucketcache.0 in hbase-site.xml.
> Attached perf measured CPU usage breakdown in flames graph.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13259) mmap() based BucketCache IOEngine


 [ 
https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-13259:
---
Release Note: 
mmap() based bucket cache can be configured by specifying the property
{code}

  hbase.bucketcache.ioengine
   mmap://filepath 

{code}
This mode of bucket cache is ideal when your file based bucket cache size is 
lesser than then available RAM. When the cache is bigger than the available RAM 
then the kernel page faults will make this cache perform lesser particularly in 
case of scans. 

  was:
mmap() based bucket cache can be configured by specifying the property
{code}

  hbase.bucketcache.ioengine
  mmap://filepath

{code}
This mode of bucket cache is ideal when your file based bucket cache size is 
lesser than then available RAM. When the cache is bigger than the available RAM 
then the kernel page faults will make this cache perform lesser particularly in 
case of scans. 


> mmap() based BucketCache IOEngine
> -
>
> Key: HBASE-13259
> URL: https://issues.apache.org/jira/browse/HBASE-13259
> Project: HBase
>  Issue Type: New Feature
>  Components: BlockCache
>Affects Versions: 0.98.10
>Reporter: Zee Chen
>Assignee: Zee Chen
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-13259-v2.patch, HBASE-13259.patch, 
> HBASE-13259_v3.patch, HBASE-13259_v4.patch, HBASE-13259_v5.patch, 
> HBASE-13259_v6.patch, ioread-1.svg, mmap-0.98-v1.patch, mmap-1.svg, 
> mmap-trunk-v1.patch
>
>
> Of the existing BucketCache IOEngines, FileIOEngine uses pread() to copy data 
> from kernel space to user space. This is a good choice when the total working 
> set size is much bigger than the available RAM and the latency is dominated 
> by IO access. However, when the entire working set is small enough to fit in 
> the RAM, using mmap() (and subsequent memcpy()) to move data from kernel 
> space to user space is faster. I have run some short keyval gets tests and 
> the results indicate a reduction of 2%-7% of kernel CPU on my system, 
> depending on the load. On the gets, the latency histograms from mmap() are 
> identical to those from pread(), but peak throughput is close to 40% higher.
> This patch modifies ByteByfferArray to allow it to specify a backing file.
> Example for using this feature: set  hbase.bucketcache.ioengine to 
> mmap:/dev/shm/bucketcache.0 in hbase-site.xml.
> Attached perf measured CPU usage breakdown in flames graph.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13259) mmap() based BucketCache IOEngine


 [ 
https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-13259:
---
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: (was: 1.3.0)
 Release Note: 
mmap() based bucket cache can be configured by specifying the property
{code}

  hbase.bucketcache.ioengine
  mmap://

{code}
This mode of bucket cache is ideal when your file based bucket cache size is 
lesser than then available RAM. When the cache is bigger than the available RAM 
then the kernel page faults will make this cache perform lesser particularly in 
case of scans. 
   Status: Resolved  (was: Patch Available)

> mmap() based BucketCache IOEngine
> -
>
> Key: HBASE-13259
> URL: https://issues.apache.org/jira/browse/HBASE-13259
> Project: HBase
>  Issue Type: New Feature
>  Components: BlockCache
>Affects Versions: 0.98.10
>Reporter: Zee Chen
>Assignee: Zee Chen
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-13259-v2.patch, HBASE-13259.patch, 
> HBASE-13259_v3.patch, HBASE-13259_v4.patch, HBASE-13259_v5.patch, 
> HBASE-13259_v6.patch, ioread-1.svg, mmap-0.98-v1.patch, mmap-1.svg, 
> mmap-trunk-v1.patch
>
>
> Of the existing BucketCache IOEngines, FileIOEngine uses pread() to copy data 
> from kernel space to user space. This is a good choice when the total working 
> set size is much bigger than the available RAM and the latency is dominated 
> by IO access. However, when the entire working set is small enough to fit in 
> the RAM, using mmap() (and subsequent memcpy()) to move data from kernel 
> space to user space is faster. I have run some short keyval gets tests and 
> the results indicate a reduction of 2%-7% of kernel CPU on my system, 
> depending on the load. On the gets, the latency histograms from mmap() are 
> identical to those from pread(), but peak throughput is close to 40% higher.
> This patch modifies ByteByfferArray to allow it to specify a backing file.
> Example for using this feature: set  hbase.bucketcache.ioengine to 
> mmap:/dev/shm/bucketcache.0 in hbase-site.xml.
> Attached perf measured CPU usage breakdown in flames graph.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13259) mmap() based BucketCache IOEngine


 [ 
https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-13259:
---
Attachment: HBASE-13259_v6.patch

This is what was committed. Test failure was unrelated. Corrected the 
whitespace and the checkstyle comment.

> mmap() based BucketCache IOEngine
> -
>
> Key: HBASE-13259
> URL: https://issues.apache.org/jira/browse/HBASE-13259
> Project: HBase
>  Issue Type: New Feature
>  Components: BlockCache
>Affects Versions: 0.98.10
>Reporter: Zee Chen
>Assignee: Zee Chen
>Priority: Critical
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-13259-v2.patch, HBASE-13259.patch, 
> HBASE-13259_v3.patch, HBASE-13259_v4.patch, HBASE-13259_v5.patch, 
> HBASE-13259_v6.patch, ioread-1.svg, mmap-0.98-v1.patch, mmap-1.svg, 
> mmap-trunk-v1.patch
>
>
> Of the existing BucketCache IOEngines, FileIOEngine uses pread() to copy data 
> from kernel space to user space. This is a good choice when the total working 
> set size is much bigger than the available RAM and the latency is dominated 
> by IO access. However, when the entire working set is small enough to fit in 
> the RAM, using mmap() (and subsequent memcpy()) to move data from kernel 
> space to user space is faster. I have run some short keyval gets tests and 
> the results indicate a reduction of 2%-7% of kernel CPU on my system, 
> depending on the load. On the gets, the latency histograms from mmap() are 
> identical to those from pread(), but peak throughput is close to 40% higher.
> This patch modifies ByteByfferArray to allow it to specify a backing file.
> Example for using this feature: set  hbase.bucketcache.ioengine to 
> mmap:/dev/shm/bucketcache.0 in hbase-site.xml.
> Attached perf measured CPU usage breakdown in flames graph.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13259) mmap() based BucketCache IOEngine


 [ 
https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-13259:
---
Status: Patch Available  (was: Open)

> mmap() based BucketCache IOEngine
> -
>
> Key: HBASE-13259
> URL: https://issues.apache.org/jira/browse/HBASE-13259
> Project: HBase
>  Issue Type: New Feature
>  Components: BlockCache
>Affects Versions: 0.98.10
>Reporter: Zee Chen
>Assignee: Zee Chen
>Priority: Critical
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-13259-v2.patch, HBASE-13259.patch, 
> HBASE-13259_v3.patch, HBASE-13259_v4.patch, HBASE-13259_v5.patch, 
> ioread-1.svg, mmap-0.98-v1.patch, mmap-1.svg, mmap-trunk-v1.patch
>
>
> Of the existing BucketCache IOEngines, FileIOEngine uses pread() to copy data 
> from kernel space to user space. This is a good choice when the total working 
> set size is much bigger than the available RAM and the latency is dominated 
> by IO access. However, when the entire working set is small enough to fit in 
> the RAM, using mmap() (and subsequent memcpy()) to move data from kernel 
> space to user space is faster. I have run some short keyval gets tests and 
> the results indicate a reduction of 2%-7% of kernel CPU on my system, 
> depending on the load. On the gets, the latency histograms from mmap() are 
> identical to those from pread(), but peak throughput is close to 40% higher.
> This patch modifies ByteByfferArray to allow it to specify a backing file.
> Example for using this feature: set  hbase.bucketcache.ioengine to 
> mmap:/dev/shm/bucketcache.0 in hbase-site.xml.
> Attached perf measured CPU usage breakdown in flames graph.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13259) mmap() based BucketCache IOEngine


 [ 
https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-13259:
---
Attachment: HBASE-13259_v5.patch

Updated patch. This applies cleanly and solves the whitespaces and checkstyle 
comments.

> mmap() based BucketCache IOEngine
> -
>
> Key: HBASE-13259
> URL: https://issues.apache.org/jira/browse/HBASE-13259
> Project: HBase
>  Issue Type: New Feature
>  Components: BlockCache
>Affects Versions: 0.98.10
>Reporter: Zee Chen
>Assignee: Zee Chen
>Priority: Critical
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-13259-v2.patch, HBASE-13259.patch, 
> HBASE-13259_v3.patch, HBASE-13259_v4.patch, HBASE-13259_v5.patch, 
> ioread-1.svg, mmap-0.98-v1.patch, mmap-1.svg, mmap-trunk-v1.patch
>
>
> Of the existing BucketCache IOEngines, FileIOEngine uses pread() to copy data 
> from kernel space to user space. This is a good choice when the total working 
> set size is much bigger than the available RAM and the latency is dominated 
> by IO access. However, when the entire working set is small enough to fit in 
> the RAM, using mmap() (and subsequent memcpy()) to move data from kernel 
> space to user space is faster. I have run some short keyval gets tests and 
> the results indicate a reduction of 2%-7% of kernel CPU on my system, 
> depending on the load. On the gets, the latency histograms from mmap() are 
> identical to those from pread(), but peak throughput is close to 40% higher.
> This patch modifies ByteByfferArray to allow it to specify a backing file.
> Example for using this feature: set  hbase.bucketcache.ioengine to 
> mmap:/dev/shm/bucketcache.0 in hbase-site.xml.
> Attached perf measured CPU usage breakdown in flames graph.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13259) mmap() based BucketCache IOEngine


 [ 
https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-13259:
---
Status: Open  (was: Patch Available)

> mmap() based BucketCache IOEngine
> -
>
> Key: HBASE-13259
> URL: https://issues.apache.org/jira/browse/HBASE-13259
> Project: HBase
>  Issue Type: New Feature
>  Components: BlockCache
>Affects Versions: 0.98.10
>Reporter: Zee Chen
>Assignee: Zee Chen
>Priority: Critical
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-13259-v2.patch, HBASE-13259.patch, 
> HBASE-13259_v3.patch, HBASE-13259_v4.patch, HBASE-13259_v5.patch, 
> ioread-1.svg, mmap-0.98-v1.patch, mmap-1.svg, mmap-trunk-v1.patch
>
>
> Of the existing BucketCache IOEngines, FileIOEngine uses pread() to copy data 
> from kernel space to user space. This is a good choice when the total working 
> set size is much bigger than the available RAM and the latency is dominated 
> by IO access. However, when the entire working set is small enough to fit in 
> the RAM, using mmap() (and subsequent memcpy()) to move data from kernel 
> space to user space is faster. I have run some short keyval gets tests and 
> the results indicate a reduction of 2%-7% of kernel CPU on my system, 
> depending on the load. On the gets, the latency histograms from mmap() are 
> identical to those from pread(), but peak throughput is close to 40% higher.
> This patch modifies ByteByfferArray to allow it to specify a backing file.
> Example for using this feature: set  hbase.bucketcache.ioengine to 
> mmap:/dev/shm/bucketcache.0 in hbase-site.xml.
> Attached perf measured CPU usage breakdown in flames graph.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13259) mmap() based BucketCache IOEngine


 [ 
https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-13259:
---
Status: Patch Available  (was: Open)

> mmap() based BucketCache IOEngine
> -
>
> Key: HBASE-13259
> URL: https://issues.apache.org/jira/browse/HBASE-13259
> Project: HBase
>  Issue Type: New Feature
>  Components: BlockCache
>Affects Versions: 0.98.10
>Reporter: Zee Chen
>Assignee: Zee Chen
>Priority: Critical
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-13259-v2.patch, HBASE-13259.patch, 
> HBASE-13259_v3.patch, HBASE-13259_v4.patch, ioread-1.svg, mmap-0.98-v1.patch, 
> mmap-1.svg, mmap-trunk-v1.patch
>
>
> Of the existing BucketCache IOEngines, FileIOEngine uses pread() to copy data 
> from kernel space to user space. This is a good choice when the total working 
> set size is much bigger than the available RAM and the latency is dominated 
> by IO access. However, when the entire working set is small enough to fit in 
> the RAM, using mmap() (and subsequent memcpy()) to move data from kernel 
> space to user space is faster. I have run some short keyval gets tests and 
> the results indicate a reduction of 2%-7% of kernel CPU on my system, 
> depending on the load. On the gets, the latency histograms from mmap() are 
> identical to those from pread(), but peak throughput is close to 40% higher.
> This patch modifies ByteByfferArray to allow it to specify a backing file.
> Example for using this feature: set  hbase.bucketcache.ioengine to 
> mmap:/dev/shm/bucketcache.0 in hbase-site.xml.
> Attached perf measured CPU usage breakdown in flames graph.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13259) mmap() based BucketCache IOEngine


 [ 
https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-13259:
---
Attachment: HBASE-13259_v4.patch

Updated patch. Removed ByteBufferallocator#sync(). I tried with a small program 
but with MappedBB.force() or fileChannel.force() itself I was able to read and 
write. May be those things or for forcibly ensuring that things are written to 
the storage. 
[~zeocio]
Do you think this patch is good with filechannel.force() instead of 
MappedByteBuffers.force()?

> mmap() based BucketCache IOEngine
> -
>
> Key: HBASE-13259
> URL: https://issues.apache.org/jira/browse/HBASE-13259
> Project: HBase
>  Issue Type: New Feature
>  Components: BlockCache
>Affects Versions: 0.98.10
>Reporter: Zee Chen
>Assignee: Zee Chen
>Priority: Critical
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-13259-v2.patch, HBASE-13259.patch, 
> HBASE-13259_v3.patch, HBASE-13259_v4.patch, ioread-1.svg, mmap-0.98-v1.patch, 
> mmap-1.svg, mmap-trunk-v1.patch
>
>
> Of the existing BucketCache IOEngines, FileIOEngine uses pread() to copy data 
> from kernel space to user space. This is a good choice when the total working 
> set size is much bigger than the available RAM and the latency is dominated 
> by IO access. However, when the entire working set is small enough to fit in 
> the RAM, using mmap() (and subsequent memcpy()) to move data from kernel 
> space to user space is faster. I have run some short keyval gets tests and 
> the results indicate a reduction of 2%-7% of kernel CPU on my system, 
> depending on the load. On the gets, the latency histograms from mmap() are 
> identical to those from pread(), but peak throughput is close to 40% higher.
> This patch modifies ByteByfferArray to allow it to specify a backing file.
> Example for using this feature: set  hbase.bucketcache.ioengine to 
> mmap:/dev/shm/bucketcache.0 in hbase-site.xml.
> Attached perf measured CPU usage breakdown in flames graph.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13259) mmap() based BucketCache IOEngine


 [ 
https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-13259:
---
Status: Open  (was: Patch Available)

> mmap() based BucketCache IOEngine
> -
>
> Key: HBASE-13259
> URL: https://issues.apache.org/jira/browse/HBASE-13259
> Project: HBase
>  Issue Type: New Feature
>  Components: BlockCache
>Affects Versions: 0.98.10
>Reporter: Zee Chen
>Assignee: Zee Chen
>Priority: Critical
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-13259-v2.patch, HBASE-13259.patch, 
> HBASE-13259_v3.patch, ioread-1.svg, mmap-0.98-v1.patch, mmap-1.svg, 
> mmap-trunk-v1.patch
>
>
> Of the existing BucketCache IOEngines, FileIOEngine uses pread() to copy data 
> from kernel space to user space. This is a good choice when the total working 
> set size is much bigger than the available RAM and the latency is dominated 
> by IO access. However, when the entire working set is small enough to fit in 
> the RAM, using mmap() (and subsequent memcpy()) to move data from kernel 
> space to user space is faster. I have run some short keyval gets tests and 
> the results indicate a reduction of 2%-7% of kernel CPU on my system, 
> depending on the load. On the gets, the latency histograms from mmap() are 
> identical to those from pread(), but peak throughput is close to 40% higher.
> This patch modifies ByteByfferArray to allow it to specify a backing file.
> Example for using this feature: set  hbase.bucketcache.ioengine to 
> mmap:/dev/shm/bucketcache.0 in hbase-site.xml.
> Attached perf measured CPU usage breakdown in flames graph.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13259) mmap() based BucketCache IOEngine

2016-01-28 Thread ramkrishna.s.vasudevan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-13259:
---
Attachment: HBASE-13259_v3.patch

Updated patch based on trunk. Need to do the testing in test cluster. Will 
report back the findings. Posting here for having a look at the current patch. 
[~zeocio]
Are you fine with the latest patch?

> mmap() based BucketCache IOEngine
> -
>
> Key: HBASE-13259
> URL: https://issues.apache.org/jira/browse/HBASE-13259
> Project: HBase
>  Issue Type: New Feature
>  Components: BlockCache
>Affects Versions: 0.98.10
>Reporter: Zee Chen
>Assignee: Zee Chen
>Priority: Critical
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-13259-v2.patch, HBASE-13259.patch, 
> HBASE-13259_v3.patch, ioread-1.svg, mmap-0.98-v1.patch, mmap-1.svg, 
> mmap-trunk-v1.patch
>
>
> Of the existing BucketCache IOEngines, FileIOEngine uses pread() to copy data 
> from kernel space to user space. This is a good choice when the total working 
> set size is much bigger than the available RAM and the latency is dominated 
> by IO access. However, when the entire working set is small enough to fit in 
> the RAM, using mmap() (and subsequent memcpy()) to move data from kernel 
> space to user space is faster. I have run some short keyval gets tests and 
> the results indicate a reduction of 2%-7% of kernel CPU on my system, 
> depending on the load. On the gets, the latency histograms from mmap() are 
> identical to those from pread(), but peak throughput is close to 40% higher.
> This patch modifies ByteByfferArray to allow it to specify a backing file.
> Example for using this feature: set  hbase.bucketcache.ioengine to 
> mmap:/dev/shm/bucketcache.0 in hbase-site.xml.
> Attached perf measured CPU usage breakdown in flames graph.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13259) mmap() based BucketCache IOEngine

2016-01-28 Thread ramkrishna.s.vasudevan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-13259:
---
Status: Patch Available  (was: Open)

> mmap() based BucketCache IOEngine
> -
>
> Key: HBASE-13259
> URL: https://issues.apache.org/jira/browse/HBASE-13259
> Project: HBase
>  Issue Type: New Feature
>  Components: BlockCache
>Affects Versions: 0.98.10
>Reporter: Zee Chen
>Assignee: Zee Chen
>Priority: Critical
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-13259-v2.patch, HBASE-13259.patch, 
> HBASE-13259_v3.patch, ioread-1.svg, mmap-0.98-v1.patch, mmap-1.svg, 
> mmap-trunk-v1.patch
>
>
> Of the existing BucketCache IOEngines, FileIOEngine uses pread() to copy data 
> from kernel space to user space. This is a good choice when the total working 
> set size is much bigger than the available RAM and the latency is dominated 
> by IO access. However, when the entire working set is small enough to fit in 
> the RAM, using mmap() (and subsequent memcpy()) to move data from kernel 
> space to user space is faster. I have run some short keyval gets tests and 
> the results indicate a reduction of 2%-7% of kernel CPU on my system, 
> depending on the load. On the gets, the latency histograms from mmap() are 
> identical to those from pread(), but peak throughput is close to 40% higher.
> This patch modifies ByteByfferArray to allow it to specify a backing file.
> Example for using this feature: set  hbase.bucketcache.ioengine to 
> mmap:/dev/shm/bucketcache.0 in hbase-site.xml.
> Attached perf measured CPU usage breakdown in flames graph.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13259) mmap() based BucketCache IOEngine

2016-01-28 Thread ramkrishna.s.vasudevan (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-13259:
---
Status: Open  (was: Patch Available)

> mmap() based BucketCache IOEngine
> -
>
> Key: HBASE-13259
> URL: https://issues.apache.org/jira/browse/HBASE-13259
> Project: HBase
>  Issue Type: New Feature
>  Components: BlockCache
>Affects Versions: 0.98.10
>Reporter: Zee Chen
>Assignee: Zee Chen
>Priority: Critical
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-13259-v2.patch, HBASE-13259.patch, ioread-1.svg, 
> mmap-0.98-v1.patch, mmap-1.svg, mmap-trunk-v1.patch
>
>
> Of the existing BucketCache IOEngines, FileIOEngine uses pread() to copy data 
> from kernel space to user space. This is a good choice when the total working 
> set size is much bigger than the available RAM and the latency is dominated 
> by IO access. However, when the entire working set is small enough to fit in 
> the RAM, using mmap() (and subsequent memcpy()) to move data from kernel 
> space to user space is faster. I have run some short keyval gets tests and 
> the results indicate a reduction of 2%-7% of kernel CPU on my system, 
> depending on the load. On the gets, the latency histograms from mmap() are 
> identical to those from pread(), but peak throughput is close to 40% higher.
> This patch modifies ByteByfferArray to allow it to specify a backing file.
> Example for using this feature: set  hbase.bucketcache.ioengine to 
> mmap:/dev/shm/bucketcache.0 in hbase-site.xml.
> Attached perf measured CPU usage breakdown in flames graph.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13259) mmap() based BucketCache IOEngine

2015-12-17 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-13259:
--
Priority: Critical  (was: Major)

I just remembered this issue. We should get this in. Makes sense. Making it 
critical for consideration for 1.3 and 2.0.

> mmap() based BucketCache IOEngine
> -
>
> Key: HBASE-13259
> URL: https://issues.apache.org/jira/browse/HBASE-13259
> Project: HBase
>  Issue Type: New Feature
>  Components: BlockCache
>Affects Versions: 0.98.10
>Reporter: Zee Chen
>Priority: Critical
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-13259-v2.patch, HBASE-13259.patch, ioread-1.svg, 
> mmap-0.98-v1.patch, mmap-1.svg, mmap-trunk-v1.patch
>
>
> Of the existing BucketCache IOEngines, FileIOEngine uses pread() to copy data 
> from kernel space to user space. This is a good choice when the total working 
> set size is much bigger than the available RAM and the latency is dominated 
> by IO access. However, when the entire working set is small enough to fit in 
> the RAM, using mmap() (and subsequent memcpy()) to move data from kernel 
> space to user space is faster. I have run some short keyval gets tests and 
> the results indicate a reduction of 2%-7% of kernel CPU on my system, 
> depending on the load. On the gets, the latency histograms from mmap() are 
> identical to those from pread(), but peak throughput is close to 40% higher.
> This patch modifies ByteByfferArray to allow it to specify a backing file.
> Example for using this feature: set  hbase.bucketcache.ioengine to 
> mmap:/dev/shm/bucketcache.0 in hbase-site.xml.
> Attached perf measured CPU usage breakdown in flames graph.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13259) mmap() based BucketCache IOEngine

2015-12-17 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-13259:
--
Assignee: Zee Chen

> mmap() based BucketCache IOEngine
> -
>
> Key: HBASE-13259
> URL: https://issues.apache.org/jira/browse/HBASE-13259
> Project: HBase
>  Issue Type: New Feature
>  Components: BlockCache
>Affects Versions: 0.98.10
>Reporter: Zee Chen
>Assignee: Zee Chen
>Priority: Critical
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-13259-v2.patch, HBASE-13259.patch, ioread-1.svg, 
> mmap-0.98-v1.patch, mmap-1.svg, mmap-trunk-v1.patch
>
>
> Of the existing BucketCache IOEngines, FileIOEngine uses pread() to copy data 
> from kernel space to user space. This is a good choice when the total working 
> set size is much bigger than the available RAM and the latency is dominated 
> by IO access. However, when the entire working set is small enough to fit in 
> the RAM, using mmap() (and subsequent memcpy()) to move data from kernel 
> space to user space is faster. I have run some short keyval gets tests and 
> the results indicate a reduction of 2%-7% of kernel CPU on my system, 
> depending on the load. On the gets, the latency histograms from mmap() are 
> identical to those from pread(), but peak throughput is close to 40% higher.
> This patch modifies ByteByfferArray to allow it to specify a backing file.
> Example for using this feature: set  hbase.bucketcache.ioengine to 
> mmap:/dev/shm/bucketcache.0 in hbase-site.xml.
> Attached perf measured CPU usage breakdown in flames graph.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13259) mmap() based BucketCache IOEngine

2015-06-22 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-13259:

Fix Version/s: (was: 1.2.0)
   1.3.0

bumping out to 1.3

 mmap() based BucketCache IOEngine
 -

 Key: HBASE-13259
 URL: https://issues.apache.org/jira/browse/HBASE-13259
 Project: HBase
  Issue Type: New Feature
  Components: BlockCache
Affects Versions: 0.98.10
Reporter: Zee Chen
 Fix For: 2.0.0, 1.3.0

 Attachments: HBASE-13259-v2.patch, HBASE-13259.patch, ioread-1.svg, 
 mmap-0.98-v1.patch, mmap-1.svg, mmap-trunk-v1.patch


 Of the existing BucketCache IOEngines, FileIOEngine uses pread() to copy data 
 from kernel space to user space. This is a good choice when the total working 
 set size is much bigger than the available RAM and the latency is dominated 
 by IO access. However, when the entire working set is small enough to fit in 
 the RAM, using mmap() (and subsequent memcpy()) to move data from kernel 
 space to user space is faster. I have run some short keyval gets tests and 
 the results indicate a reduction of 2%-7% of kernel CPU on my system, 
 depending on the load. On the gets, the latency histograms from mmap() are 
 identical to those from pread(), but peak throughput is close to 40% higher.
 This patch modifies ByteByfferArray to allow it to specify a backing file.
 Example for using this feature: set  hbase.bucketcache.ioengine to 
 mmap:/dev/shm/bucketcache.0 in hbase-site.xml.
 Attached perf measured CPU usage breakdown in flames graph.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13259) mmap() based BucketCache IOEngine

2015-04-20 Thread Nick Dimiduk (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Nick Dimiduk updated HBASE-13259:
-
Fix Version/s: (was: 1.1.0)
1.2.0

mmap() based BucketCache IOEngine
-

Key: HBASE-13259
URL: https://issues.apache.org/jira/browse/HBASE-13259
Project: HBase
Issue Type: New Feature
Components: BlockCache
Affects Versions: 0.98.10
Reporter: Zee Chen
Fix For: 2.0.0, 1.2.0

Attachments: HBASE-13259-v2.patch, HBASE-13259.patch, ioread-1.svg,
mmap-0.98-v1.patch, mmap-1.svg, mmap-trunk-v1.patch

Of the existing BucketCache IOEngines, FileIOEngine uses pread() to copy data
from kernel space to user space. This is a good choice when the total working
set size is much bigger than the available RAM and the latency is dominated
by IO access. However, when the entire working set is small enough to fit in
the RAM, using mmap() (and subsequent memcpy()) to move data from kernel
space to user space is faster. I have run some short keyval gets tests and
the results indicate a reduction of 2%-7% of kernel CPU on my system,
depending on the load. On the gets, the latency histograms from mmap() are
identical to those from pread(), but peak throughput is close to 40% higher.
This patch modifies ByteByfferArray to allow it to specify a backing file.
Example for using this feature: set hbase.bucketcache.ioengine to
mmap:/dev/shm/bucketcache.0 in hbase-site.xml.
Attached perf measured CPU usage breakdown in flames graph.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13259) mmap() based BucketCache IOEngine

2015-04-15 Thread Nick Dimiduk (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Nick Dimiduk updated HBASE-13259:
-
Fix Version/s: (was: 2.2.0)
1.1.0
2.0.0

mmap() based BucketCache IOEngine
-

Attachments: HBASE-13259-v2.patch, HBASE-13259.patch, ioread-1.svg,
mmap-0.98-v1.patch, mmap-1.svg, mmap-trunk-v1.patch

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13259) mmap() based BucketCache IOEngine

[
https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Zee Chen updated HBASE-13259:
-
Attachment: HBASE-13259.patch

mmap() based BucketCache IOEngine
-

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13259) mmap() based BucketCache IOEngine

[
https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Zee Chen updated HBASE-13259:
-
Status: Patch Available (was: Open)

mmap() based BucketCache IOEngine
-

Key: HBASE-13259
URL: https://issues.apache.org/jira/browse/HBASE-13259
Project: HBase
Issue Type: New Feature
Components: BlockCache
Affects Versions: 0.98.10
Reporter: Zee Chen
Fix For: 2.2.0

Attachments: HBASE-13259-v2.patch, HBASE-13259.patch, ioread-1.svg,
mmap-0.98-v1.patch, mmap-1.svg, mmap-trunk-v1.patch

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13259) mmap() based BucketCache IOEngine

[
https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Zee Chen updated HBASE-13259:
-
Status: Open (was: Patch Available)

mmap() based BucketCache IOEngine
-

Key: HBASE-13259
URL: https://issues.apache.org/jira/browse/HBASE-13259
Project: HBase
Issue Type: New Feature
Components: BlockCache
Affects Versions: 0.98.10
Reporter: Zee Chen
Fix For: 2.2.0

Attachments: HBASE-13259-v2.patch, HBASE-13259.patch, ioread-1.svg,
mmap-0.98-v1.patch, mmap-1.svg, mmap-trunk-v1.patch

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13259) mmap() based BucketCache IOEngine

[
https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Zee Chen updated HBASE-13259:
-
Attachment: HBASE-13259-v2.patch

mmap() based BucketCache IOEngine
-

Key: HBASE-13259
URL: https://issues.apache.org/jira/browse/HBASE-13259
Project: HBase
Issue Type: New Feature
Components: BlockCache
Affects Versions: 0.98.10
Reporter: Zee Chen
Fix For: 2.2.0

Attachments: HBASE-13259-v2.patch, HBASE-13259.patch, ioread-1.svg,
mmap-0.98-v1.patch, mmap-1.svg, mmap-trunk-v1.patch

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13259) mmap() based BucketCache IOEngine

[
https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Zee Chen updated HBASE-13259:
-
Fix Version/s: 2.2.0
Status: Patch Available (was: Open)

mmap() based BucketCache IOEngine
-

Key: HBASE-13259
URL: https://issues.apache.org/jira/browse/HBASE-13259
Project: HBase
Issue Type: New Feature
Components: BlockCache
Affects Versions: 0.98.10
Reporter: Zee Chen
Fix For: 2.2.0

Attachments: HBASE-13259.patch, ioread-1.svg, mmap-0.98-v1.patch,
mmap-1.svg, mmap-trunk-v1.patch

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13259) mmap() based BucketCache IOEngine


 [ 
https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zee Chen updated HBASE-13259:
-
Attachment: mmap-trunk-v1.patch

 mmap() based BucketCache IOEngine
 -

 Key: HBASE-13259
 URL: https://issues.apache.org/jira/browse/HBASE-13259
 Project: HBase
  Issue Type: New Feature
  Components: BlockCache
Affects Versions: 0.98.10
Reporter: Zee Chen
 Attachments: mmap-0.98-v1.patch, mmap-trunk-v1.patch


 Of the existing BucketCache IOEngines, FileIOEngine uses pread() to copy data 
 from kernel space to user space. This is a good choice when the total working 
 set size is much bigger than the available RAM and the latency is dominated 
 by IO access. However, when the entire working set is small enough to fit in 
 the RAM, using mmap() (and subsequent memcpy()) to move data from kernel 
 space to user space is faster. I have run some short keyval gets tests and 
 the results indicate a reduction of 2%-7% of kernel CPU on my system, 
 depending on the load. On the gets, the latency histograms from mmap() are 
 identical to those from pread(), but peak throughput is close to 40% higher.
 This patch modifies ByteByfferArray to allow it to specify a backing file.
 Example for using this feature: set  hbase.bucketcache.ioengine to 
 mmap:/dev/shm/bucketcache.0 in hbase-site.xml.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13259) mmap() based BucketCache IOEngine


 [ 
https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zee Chen updated HBASE-13259:
-
Attachment: (was: mmap-trunk-v1.patch)

 mmap() based BucketCache IOEngine
 -

 Key: HBASE-13259
 URL: https://issues.apache.org/jira/browse/HBASE-13259
 Project: HBase
  Issue Type: New Feature
  Components: BlockCache
Affects Versions: 0.98.10
Reporter: Zee Chen
 Attachments: mmap-0.98-v1.patch


 Of the existing BucketCache IOEngines, FileIOEngine uses pread() to copy data 
 from kernel space to user space. This is a good choice when the total working 
 set size is much bigger than the available RAM and the latency is dominated 
 by IO access. However, when the entire working set is small enough to fit in 
 the RAM, using mmap() (and subsequent memcpy()) to move data from kernel 
 space to user space is faster. I have run some short keyval gets tests and 
 the results indicate a reduction of 2%-7% of kernel CPU on my system, 
 depending on the load. On the gets, the latency histograms from mmap() are 
 identical to those from pread(), but peak throughput is close to 40% higher.
 This patch modifies ByteByfferArray to allow it to specify a backing file.
 Example for using this feature: set  hbase.bucketcache.ioengine to 
 mmap:/dev/shm/bucketcache.0 in hbase-site.xml.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13259) mmap() based BucketCache IOEngine

[
https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Zee Chen updated HBASE-13259:
-
Description:
Of the existing BucketCache IOEngines, FileIOEngine uses pread() to copy data
from kernel space to user space. This is a good choice when the total working
set size is much bigger than the available RAM and the latency is dominated by
IO access. However, when the entire working set is small enough to fit in the
RAM, using mmap() (and subsequent memcpy()) to move data from kernel space to
user space is faster. I have run some short keyval gets tests and the results
indicate a reduction of 2%-7% of kernel CPU on my system, depending on the
load. On the gets, the latency histograms from mmap() are identical to those
from pread(), but peak throughput is close to 40% higher.

This patch modifies ByteByfferArray to allow it to specify a backing file.

Example for using this feature: set hbase.bucketcache.ioengine to
mmap:/dev/shm/bucketcache.0 in hbase-site.xml.

Attached perf measured CPU usage breakdown in flames graph.

was:
Of the existing BucketCache IOEngines, FileIOEngine uses pread() to copy data
from kernel space to user space. This is a good choice when the total working
set size is much bigger than the available RAM and the latency is dominated by
IO access. However, when the entire working set is small enough to fit in the
RAM, using mmap() (and subsequent memcpy()) to move data from kernel space to
user space is faster. I have run some short keyval gets tests and the results
indicate a reduction of 2%-7% of kernel CPU on my system, depending on the
load. On the gets, the latency histograms from mmap() are identical to those
from pread(), but peak throughput is close to 40% higher.

This patch modifies ByteByfferArray to allow it to specify a backing file.

Example for using this feature: set hbase.bucketcache.ioengine to
mmap:/dev/shm/bucketcache.0 in hbase-site.xml.

mmap() based BucketCache IOEngine
-

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13259) mmap() based BucketCache IOEngine


 [ 
https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zee Chen updated HBASE-13259:
-
Attachment: mmap-0.98-v1.patch

 mmap() based BucketCache IOEngine
 -

 Key: HBASE-13259
 URL: https://issues.apache.org/jira/browse/HBASE-13259
 Project: HBase
  Issue Type: New Feature
  Components: BlockCache
Affects Versions: 0.98.10
Reporter: Zee Chen
 Attachments: mmap-0.98-v1.patch


 Of the existing BucketCache IOEngines, FileIOEngine uses pread() to
 copy data from kernel space to user space. This is a good choice when the
 total working set size is much bigger than the available RAM and the latency 
 is dominated by IO access. However, when the entire working set is
 small enough to fit in the RAM, using mmap() (and subsequent memcpy()) to 
 move data from kernel space to user space is faster. I have run some short 
 keyval gets tests and the results indicate a reduction of 2%-7% of kernel CPU 
 on my system, depending on the load. On the gets, the latency
 histograms from mmap() are identical to those from pread(), but peak
 throughput is close to 40% higher.
 This patch modifies ByteByfferArray to allow it to specify a backing file.
 Example for using this feature: set  hbase.bucketcache.ioengine to 
 mmap:/dev/shm/bucketcache.0 in hbase-site.xml.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13259) mmap() based BucketCache IOEngine

[
https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

This patch modifies ByteByfferArray to allow it to specify a backing file.

Example for using this feature: set hbase.bucketcache.ioengine to
mmap:/dev/shm/bucketcache.0 in hbase-site.xml.

was:
Of the existing BucketCache IOEngines, FileIOEngine uses pread() to
copy data from kernel space to user space. This is a good choice when the
total working set size is much bigger than the available RAM and the latency is
dominated by IO access. However, when the entire working set is
small enough to fit in the RAM, using mmap() (and subsequent memcpy()) to move
data from kernel space to user space is faster. I have run some short keyval
gets tests and the results indicate a reduction of 2%-7% of kernel CPU on my
system, depending on the load. On the gets, the latency
histograms from mmap() are identical to those from pread(), but peak
throughput is close to 40% higher.

This patch modifies ByteByfferArray to allow it to specify a backing file.

Example for using this feature: set hbase.bucketcache.ioengine to
mmap:/dev/shm/bucketcache.0 in hbase-site.xml.

mmap() based BucketCache IOEngine
-

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13259) mmap() based BucketCache IOEngine


 [ 
https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zee Chen updated HBASE-13259:
-
Attachment: mmap-trunk-v1.patch

 mmap() based BucketCache IOEngine
 -

 Key: HBASE-13259
 URL: https://issues.apache.org/jira/browse/HBASE-13259
 Project: HBase
  Issue Type: New Feature
  Components: BlockCache
Affects Versions: 0.98.10
Reporter: Zee Chen
 Attachments: mmap-0.98-v1.patch, mmap-trunk-v1.patch


 Of the existing BucketCache IOEngines, FileIOEngine uses pread() to copy data 
 from kernel space to user space. This is a good choice when the total working 
 set size is much bigger than the available RAM and the latency is dominated 
 by IO access. However, when the entire working set is small enough to fit in 
 the RAM, using mmap() (and subsequent memcpy()) to move data from kernel 
 space to user space is faster. I have run some short keyval gets tests and 
 the results indicate a reduction of 2%-7% of kernel CPU on my system, 
 depending on the load. On the gets, the latency histograms from mmap() are 
 identical to those from pread(), but peak throughput is close to 40% higher.
 This patch modifies ByteByfferArray to allow it to specify a backing file.
 Example for using this feature: set  hbase.bucketcache.ioengine to 
 mmap:/dev/shm/bucketcache.0 in hbase-site.xml.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13259) mmap() based BucketCache IOEngine


 [ 
https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zee Chen updated HBASE-13259:
-
Attachment: ioread-1.svg

 mmap() based BucketCache IOEngine
 -

 Key: HBASE-13259
 URL: https://issues.apache.org/jira/browse/HBASE-13259
 Project: HBase
  Issue Type: New Feature
  Components: BlockCache
Affects Versions: 0.98.10
Reporter: Zee Chen
 Attachments: ioread-1.svg, mmap-0.98-v1.patch, mmap-1.svg, 
 mmap-trunk-v1.patch


 Of the existing BucketCache IOEngines, FileIOEngine uses pread() to copy data 
 from kernel space to user space. This is a good choice when the total working 
 set size is much bigger than the available RAM and the latency is dominated 
 by IO access. However, when the entire working set is small enough to fit in 
 the RAM, using mmap() (and subsequent memcpy()) to move data from kernel 
 space to user space is faster. I have run some short keyval gets tests and 
 the results indicate a reduction of 2%-7% of kernel CPU on my system, 
 depending on the load. On the gets, the latency histograms from mmap() are 
 identical to those from pread(), but peak throughput is close to 40% higher.
 This patch modifies ByteByfferArray to allow it to specify a backing file.
 Example for using this feature: set  hbase.bucketcache.ioengine to 
 mmap:/dev/shm/bucketcache.0 in hbase-site.xml.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13259) mmap() based BucketCache IOEngine