[jira] [Commented] (HBASE-7266) [89-fb] Using pread for non-compaction read request
[ https://issues.apache.org/jira/browse/HBASE-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126804#comment-16126804 ] vinisha commented on HBASE-7266: Which HBase versions does it affect? > [89-fb] Using pread for non-compaction read request > --- > > Key: HBASE-7266 > URL: https://issues.apache.org/jira/browse/HBASE-7266 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: Liyin Tang > > There are 2 kinds of read operations in HBase: pread and seek+read. > Pread, positional read, is stateless and create a new connection between the > DFSClient and DataNode for each operation. While seek+read is to seek to a > specific postion and prefetch blocks from data nodes. The benefit of > seek+read is that it will cache the prefetch result but the downside is it is > stateful and needs to synchronized. > So far, both compaction and scan are using seek+read, which caused some > resource contention. So using the pread for the scan request can avoid the > resource contention. In addition, the region server is able to do the > prefetch for the scan request (HBASE-6874) so that it won't be necessary to > let the DFSClient to prefetch the data any more. > I will run through the scan benchmark (with no block cache) with verify the > performance. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-7266) [89-fb] Using pread for non-compaction read request
[ https://issues.apache.org/jira/browse/HBASE-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13726207#comment-13726207 ] Chao Shi commented on HBASE-7266: - Lars, yes, we're using an early 0.94 version. Thanks for the fix and we will try to benchmark a newer version. I think the fix is too tricky and I would prefer the idea Liyin proposed. Let's discus in the new issue. [89-fb] Using pread for non-compaction read request --- Key: HBASE-7266 URL: https://issues.apache.org/jira/browse/HBASE-7266 Project: HBase Issue Type: Improvement Reporter: Liyin Tang There are 2 kinds of read operations in HBase: pread and seek+read. Pread, positional read, is stateless and create a new connection between the DFSClient and DataNode for each operation. While seek+read is to seek to a specific postion and prefetch blocks from data nodes. The benefit of seek+read is that it will cache the prefetch result but the downside is it is stateful and needs to synchronized. So far, both compaction and scan are using seek+read, which caused some resource contention. So using the pread for the scan request can avoid the resource contention. In addition, the region server is able to do the prefetch for the scan request (HBASE-6874) so that it won't be necessary to let the DFSClient to prefetch the data any more. I will run through the scan benchmark (with no block cache) with verify the performance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7266) [89-fb] Using pread for non-compaction read request
[ https://issues.apache.org/jira/browse/HBASE-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13724944#comment-13724944 ] Chao Shi commented on HBASE-7266: - We are experiencing this too. This happens when a large number of concurrent scan requests on a single region. Because of the lock (on seek+read), only one handler thread is effectively working. Our scan range is very small (just a few adjacent rows, fits into 1 block). [89-fb] Using pread for non-compaction read request --- Key: HBASE-7266 URL: https://issues.apache.org/jira/browse/HBASE-7266 Project: HBase Issue Type: Improvement Reporter: Liyin Tang There are 2 kinds of read operations in HBase: pread and seek+read. Pread, positional read, is stateless and create a new connection between the DFSClient and DataNode for each operation. While seek+read is to seek to a specific postion and prefetch blocks from data nodes. The benefit of seek+read is that it will cache the prefetch result but the downside is it is stateful and needs to synchronized. So far, both compaction and scan are using seek+read, which caused some resource contention. So using the pread for the scan request can avoid the resource contention. In addition, the region server is able to do the prefetch for the scan request (HBASE-6874) so that it won't be necessary to let the DFSClient to prefetch the data any more. I will run through the scan benchmark (with no block cache) with verify the performance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7266) [89-fb] Using pread for non-compaction read request
[ https://issues.apache.org/jira/browse/HBASE-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725387#comment-13725387 ] Lars Hofhansl commented on HBASE-7266: -- In 0.94+ I fixed this by falling back to pread if the lock istream is taken (HBASE-7336). [89-fb] Using pread for non-compaction read request --- Key: HBASE-7266 URL: https://issues.apache.org/jira/browse/HBASE-7266 Project: HBase Issue Type: Improvement Reporter: Liyin Tang There are 2 kinds of read operations in HBase: pread and seek+read. Pread, positional read, is stateless and create a new connection between the DFSClient and DataNode for each operation. While seek+read is to seek to a specific postion and prefetch blocks from data nodes. The benefit of seek+read is that it will cache the prefetch result but the downside is it is stateful and needs to synchronized. So far, both compaction and scan are using seek+read, which caused some resource contention. So using the pread for the scan request can avoid the resource contention. In addition, the region server is able to do the prefetch for the scan request (HBASE-6874) so that it won't be necessary to let the DFSClient to prefetch the data any more. I will run through the scan benchmark (with no block cache) with verify the performance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7266) [89-fb] Using pread for non-compaction read request
[ https://issues.apache.org/jira/browse/HBASE-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725443#comment-13725443 ] Liyin Tang commented on HBASE-7266: --- Chao,we have switched all the read operation to pread in the 89-fb branch. There are 2 followup tasks for the pread. 1) The DFSClient maintains a connection pool instead of creating new connection for each pread operation. 2) HBase will actively pre-load the next several blocks in a stream fashion for large sequential scans (HBASE-9102) [89-fb] Using pread for non-compaction read request --- Key: HBASE-7266 URL: https://issues.apache.org/jira/browse/HBASE-7266 Project: HBase Issue Type: Improvement Reporter: Liyin Tang There are 2 kinds of read operations in HBase: pread and seek+read. Pread, positional read, is stateless and create a new connection between the DFSClient and DataNode for each operation. While seek+read is to seek to a specific postion and prefetch blocks from data nodes. The benefit of seek+read is that it will cache the prefetch result but the downside is it is stateful and needs to synchronized. So far, both compaction and scan are using seek+read, which caused some resource contention. So using the pread for the scan request can avoid the resource contention. In addition, the region server is able to do the prefetch for the scan request (HBASE-6874) so that it won't be necessary to let the DFSClient to prefetch the data any more. I will run through the scan benchmark (with no block cache) with verify the performance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7266) [89-fb] Using pread for non-compaction read request
[ https://issues.apache.org/jira/browse/HBASE-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509392#comment-13509392 ] Ted Yu commented on HBASE-7266: --- bq. So far, both compaction and scan are using pread, which caused some resource contention. So using the pread for the scan request can avoid the resource contention. Looks like the above description needs some rephrase. [89-fb] Using pread for non-compaction read request --- Key: HBASE-7266 URL: https://issues.apache.org/jira/browse/HBASE-7266 Project: HBase Issue Type: Improvement Reporter: Liyin Tang There are 2 kinds of read operations in HBase: pread and seek+read. Pread, positional read, is stateless and create a new connection between the DFSClient and DataNode for each operation. While seek+read is to seek to a specific postion and prefetch blocks from data nodes. The benefit of seek+read is that it will cache the prefetch result but the downside is it is stateful and needs to synchronized. So far, both compaction and scan are using pread, which caused some resource contention. So using the pread for the scan request can avoid the resource contention. In addition, the region server is able to do the prefetch for the scan request (HBASE-6874) so that it won't be necessary to let the DFSClient to prefetch the data any more. I will run through the scan benchmark (with no block cache) with verify the performance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7266) [89-fb] Using pread for non-compaction read request
[ https://issues.apache.org/jira/browse/HBASE-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509398#comment-13509398 ] Lars Hofhansl commented on HBASE-7266: -- Looks like in 0.94+ we're already doing this (see HFileReaderV2) [89-fb] Using pread for non-compaction read request --- Key: HBASE-7266 URL: https://issues.apache.org/jira/browse/HBASE-7266 Project: HBase Issue Type: Improvement Reporter: Liyin Tang There are 2 kinds of read operations in HBase: pread and seek+read. Pread, positional read, is stateless and create a new connection between the DFSClient and DataNode for each operation. While seek+read is to seek to a specific postion and prefetch blocks from data nodes. The benefit of seek+read is that it will cache the prefetch result but the downside is it is stateful and needs to synchronized. So far, both compaction and scan are using seek+read, which caused some resource contention. So using the pread for the scan request can avoid the resource contention. In addition, the region server is able to do the prefetch for the scan request (HBASE-6874) so that it won't be necessary to let the DFSClient to prefetch the data any more. I will run through the scan benchmark (with no block cache) with verify the performance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7266) [89-fb] Using pread for non-compaction read request
[
https://issues.apache.org/jira/browse/HBASE-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509466#comment-13509466
]
binlijin commented on HBASE-7266:
-
[~lhofhansl] i don't get it, when scan the parameter 'pread' is determined by
Scan.isGetScan(),which is
{code}
Scan
public boolean isGetScan() {
return this.startRow != null this.startRow.length 0
Bytes.equals(this.startRow, this.stopRow);
}
{code}
[89-fb] Using pread for non-compaction read request
---
Key: HBASE-7266
URL: https://issues.apache.org/jira/browse/HBASE-7266
Project: HBase
Issue Type: Improvement
Reporter: Liyin Tang
There are 2 kinds of read operations in HBase: pread and seek+read.
Pread, positional read, is stateless and create a new connection between the
DFSClient and DataNode for each operation. While seek+read is to seek to a
specific postion and prefetch blocks from data nodes. The benefit of
seek+read is that it will cache the prefetch result but the downside is it is
stateful and needs to synchronized.
So far, both compaction and scan are using seek+read, which caused some
resource contention. So using the pread for the scan request can avoid the
resource contention. In addition, the region server is able to do the
prefetch for the scan request (HBASE-6874) so that it won't be necessary to
let the DFSClient to prefetch the data any more.
I will run through the scan benchmark (with no block cache) with verify the
performance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
