[jira] [Comment Edited] (HBASE-17910) Use separated StoreFileReader for streaming read
[
https://issues.apache.org/jira/browse/HBASE-17910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17551386#comment-17551386
]
Minwoo Kang edited comment on HBASE-17910 at 6/8/22 10:34 AM:
--
I have the same problem. Cpu usage of namenode is too high because of Switch to
stream read(trySwitchToStreamRead method).
In my case, I changed hbase.storescanner.use.pread to true to prevent the
change from pread to stream read.
was (Author: minwoo.kang):
I have the same problem. Cpu usage of namenode is too high because of Switch to
stream read(trySwitchToStreamRead method).
> Use separated StoreFileReader for streaming read
>
>
> Key: HBASE-17910
> URL: https://issues.apache.org/jira/browse/HBASE-17910
> Project: HBase
> Issue Type: Improvement
> Components: regionserver, Scanners
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.0.0
>
>
> For now we have already supportted using private readers for compaction, by
> creating a new StoreFile copy. I think a better way is to allow creating
> multiple readers from a single StoreFile instance, thus we can avoid the ugly
> cloning, and the reader can also be used for streaming scan, not only for
> compaction.
> The reason we want to do this is that, we found a read amplification when
> using short circult read. {{BlockReaderLocal}} will use an internal buffer to
> read data first, the buffer size is based on the configured buffer size and
> the readahead option in CachingStrategy. For normal pread request, we should
> just bypass the buffer, this can be achieved by setting readahead to 0. But
> for streaming read I think the buffer is somehow still useful? So we need to
> use different FSDataInputStream for pread and streaming read.
> And one more thing is that, we can also remove the streamLock if streaming
> read always use its own reader.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Comment Edited] (HBASE-17910) Use separated StoreFileReader for streaming read
[
https://issues.apache.org/jira/browse/HBASE-17910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17551386#comment-17551386
]
Minwoo Kang edited comment on HBASE-17910 at 6/8/22 4:43 AM:
-
I have the same problem. Cpu usage of namenode is too high because of Switch to
stream read(trySwitchToStreamRead method).
was (Author: minwoo.kang):
I have the same problem. Cpu usage of namenode is too high because of stream
read.
> Use separated StoreFileReader for streaming read
>
>
> Key: HBASE-17910
> URL: https://issues.apache.org/jira/browse/HBASE-17910
> Project: HBase
> Issue Type: Improvement
> Components: regionserver, Scanners
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.0.0
>
>
> For now we have already supportted using private readers for compaction, by
> creating a new StoreFile copy. I think a better way is to allow creating
> multiple readers from a single StoreFile instance, thus we can avoid the ugly
> cloning, and the reader can also be used for streaming scan, not only for
> compaction.
> The reason we want to do this is that, we found a read amplification when
> using short circult read. {{BlockReaderLocal}} will use an internal buffer to
> read data first, the buffer size is based on the configured buffer size and
> the readahead option in CachingStrategy. For normal pread request, we should
> just bypass the buffer, this can be achieved by setting readahead to 0. But
> for streaming read I think the buffer is somehow still useful? So we need to
> use different FSDataInputStream for pread and streaming read.
> And one more thing is that, we can also remove the streamLock if streaming
> read always use its own reader.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Comment Edited] (HBASE-17910) Use separated StoreFileReader for streaming read
[
https://issues.apache.org/jira/browse/HBASE-17910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17551386#comment-17551386
]
Minwoo Kang edited comment on HBASE-17910 at 6/8/22 4:28 AM:
-
I have the same problem. Cpu usage of namenode is too high because of stream
read.
was (Author: minwoo.kang):
I have the same problem. Cpu usage of our namenode is too high because of
stream read.
> Use separated StoreFileReader for streaming read
>
>
> Key: HBASE-17910
> URL: https://issues.apache.org/jira/browse/HBASE-17910
> Project: HBase
> Issue Type: Improvement
> Components: regionserver, Scanners
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.0.0
>
>
> For now we have already supportted using private readers for compaction, by
> creating a new StoreFile copy. I think a better way is to allow creating
> multiple readers from a single StoreFile instance, thus we can avoid the ugly
> cloning, and the reader can also be used for streaming scan, not only for
> compaction.
> The reason we want to do this is that, we found a read amplification when
> using short circult read. {{BlockReaderLocal}} will use an internal buffer to
> read data first, the buffer size is based on the configured buffer size and
> the readahead option in CachingStrategy. For normal pread request, we should
> just bypass the buffer, this can be achieved by setting readahead to 0. But
> for streaming read I think the buffer is somehow still useful? So we need to
> use different FSDataInputStream for pread and streaming read.
> And one more thing is that, we can also remove the streamLock if streaming
> read always use its own reader.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Comment Edited] (HBASE-17910) Use separated StoreFileReader for streaming read
[
https://issues.apache.org/jira/browse/HBASE-17910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226438#comment-17226438
]
Karthik Palanisamy edited comment on HBASE-17910 at 11/5/20, 1:17 AM:
--
[~anoop.hbase] [~zhangduo] [~busbey] [~elserj]
Recently, one of our user reported high CPU usage in namenode. On our
troubleshooting, found millions of OPEN and GetFileInfo calls continuously to
namenode, is because of readType STREAM which creates multiple scanners. I
understand we switch readType to STREAM automatically but I don't find any flag
to disable STREAM. I am curious if that is the expected design?
The switch happens below,
If the scan become get.
if the scan with startrow and stoprow.
if the scan keeps running for long time. I.e kv bytesRead > preadMaxBytes.
(Default preadMaxBytes is 4*blockSize, which is 4*64KB).
Maybe this spike could be at every cluster but the user might not be noticed
yet. At this moment, I am trying to work around with
"hbase.storescanner.pread.max.bytes" and
"hbase.cells.scanned.per.heartbeat.check". Will post more updates next week
with the root cause.
{code:java}
...
private StoreScanner(HStore store, Scan scan, ScanInfo scanInfo,
int numColumns, long readPt, boolean cacheBlocks, ScanType scanType) {
..
get = scan.isGetScan();
..
this.maxRowSize = scanInfo.getTableMaxRowSize();
if (get) {
this.readType = Scan.ReadType.PREAD;
this.scanUsePread = true;
}
...
public void shipped() throws IOException {
..
clearAndClose(scannersForDelayedClose);
if (this.heap != null) {
..
trySwitchToStreamRead();
}
}
..
void trySwitchToStreamRead() {
if (readType != Scan.ReadType.DEFAULT || !scanUsePread || closing ||
heap.peek() == null || bytesRead < preadMaxBytes) {
return;
}
LOG.debug("Switch to stream read (scanned={} bytes) of {}", bytesRead,
this.store.getColumnFamilyName());
..
}
{code}
was (Author: kpalanisamy):
[~anoop.hbase] [~zhangduo] [~busbey] [~elserj]
Recently, one of our user reported high CPU usage in namenode. On our
troubleshooting, found millions of OPEN and GetFileInfo calls continuously to
namenode, is because of readType STREAM which creates multiple scanners. I
understand we switch readType to STREAM automatically but I don't find any flag
to disable STREAM. I am curious if that is the expected design?
The switch happens below,
If the scan become get.
if the scan with startrow and stoprow.
if the scan keeps running for long time. I.e kv bytesRead > preadMaxBytes.
(Default preadMaxBytes is 4*blockSize, which is 4*64KB).
Maybe this spike could be at every cluster but the user might not be noticed
yet. At this moment, I am trying to work around with
"hbase.storescanner.pread.max.bytes" and
"hbase.cells.scanned.per.heartbeat.check". Will post more updates next week
with the root cause.
{code:java}
this(family, minVersions, maxVersions, ttl, keepDeletedCells,
timeToPurgeDeletes, comparator,
conf.getLong(HConstants.TABLE_MAX_ROWSIZE_KEY,
HConstants.TABLE_MAX_ROWSIZE_DEFAULT),
conf.getBoolean("hbase.storescanner.use.pread", false),
getCellsPerTimeoutCheck(conf),
conf.getBoolean(StoreScanner.STORESCANNER_PARALLEL_SEEK_ENABLE, false),
conf.getLong(StoreScanner.STORESCANNER_PREAD_MAX_BYTES, 4 * blockSize),
newVersionBehavior);
...
private StoreScanner(HStore store, Scan scan, ScanInfo scanInfo,
int numColumns, long readPt, boolean cacheBlocks, ScanType scanType) {
..
get = scan.isGetScan();
..
this.maxRowSize = scanInfo.getTableMaxRowSize();
if (get) {
this.readType = Scan.ReadType.PREAD;
this.scanUsePread = true;
}
...{code}
> Use separated StoreFileReader for streaming read
>
>
> Key: HBASE-17910
> URL: https://issues.apache.org/jira/browse/HBASE-17910
> Project: HBase
> Issue Type: Improvement
> Components: regionserver, Scanners
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.0.0
>
>
> For now we have already supportted using private readers for compaction, by
> creating a new StoreFile copy. I think a better way is to allow creating
> multiple readers from a single StoreFile instance, thus we can avoid the ugly
> cloning, and the reader can also be used for streaming scan, not only for
> compaction.
> The reason we want to do this is that, we found a read amplification when
> using short circult read. {{BlockReaderLocal}} will use an internal buffer to
> read data first, the buffer size is based on the configured buffer size and
> the readahead option in CachingStrategy. For normal pread request, we should
> just bypass the buffer, this can be achieved by setting readahead to 0. But
> for streaming read I
[jira] [Comment Edited] (HBASE-17910) Use separated StoreFileReader for streaming read
[
https://issues.apache.org/jira/browse/HBASE-17910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226438#comment-17226438
]
Karthik Palanisamy edited comment on HBASE-17910 at 11/5/20, 1:15 AM:
--
[~anoop.hbase] [~zhangduo] [~busbey] [~elserj]
Recently, one of our user reported high CPU usage in namenode. On our
troubleshooting, found millions of OPEN and GetFileInfo calls continuously to
namenode, is because of readType STREAM which creates multiple scanners. I
understand we switch readType to STREAM automatically but I don't find any flag
to disable STREAM. I am curious if that is the expected design?
The switch happens below,
If the scan become get.
if the scan with startrow and stoprow.
if the scan keeps running for long time. I.e kv bytesRead > preadMaxBytes.
(Default preadMaxBytes is 4*blockSize, which is 4*64KB).
Maybe this spike could be at every cluster but the user might not be noticed
yet. At this moment, I am trying to work around with
"hbase.storescanner.pread.max.bytes" and
"hbase.cells.scanned.per.heartbeat.check". Will post more updates next week
with the root cause.
{code:java}
this(family, minVersions, maxVersions, ttl, keepDeletedCells,
timeToPurgeDeletes, comparator,
conf.getLong(HConstants.TABLE_MAX_ROWSIZE_KEY,
HConstants.TABLE_MAX_ROWSIZE_DEFAULT),
conf.getBoolean("hbase.storescanner.use.pread", false),
getCellsPerTimeoutCheck(conf),
conf.getBoolean(StoreScanner.STORESCANNER_PARALLEL_SEEK_ENABLE, false),
conf.getLong(StoreScanner.STORESCANNER_PREAD_MAX_BYTES, 4 * blockSize),
newVersionBehavior);
...
private StoreScanner(HStore store, Scan scan, ScanInfo scanInfo,
int numColumns, long readPt, boolean cacheBlocks, ScanType scanType) {
..
get = scan.isGetScan();
..
this.maxRowSize = scanInfo.getTableMaxRowSize();
if (get) {
this.readType = Scan.ReadType.PREAD;
this.scanUsePread = true;
}
...{code}
was (Author: kpalanisamy):
[~anoop.hbase] [~zhangduo] [~busbey] [~elserj]
Recently, one of our user reported high CPU usage in namenode. On our
troubleshooting, found millions of OPEN and GetFileInfo calls continuously to
namenode, is because of readType STREAM which creates multiple scanners. I
understand we switch readType to STREAM automatically but I don't find any flag
to disable STREAM. I am curious if that is the expected design?
The switch happens below,
If the scan become get.
if the scan with startrow and stoprow.
if the scan keeps running for long time. I.e kv bytesRead > preadMaxBytes.
(Default preadMaxBytes is 4*blockSize, which is 4*64KB).
Maybe this spike could be at every cluster but the user might not be noticed
yet. At this moment, I am trying to work around with
"hbase.storescanner.pread.max.bytes" and
"hbase.cells.scanned.per.heartbeat.check". Will post more updates next week
with the root cause.
{code:java}
this(family, minVersions, maxVersions, ttl, keepDeletedCells,
timeToPurgeDeletes, comparator,
conf.getLong(HConstants.TABLE_MAX_ROWSIZE_KEY,
HConstants.TABLE_MAX_ROWSIZE_DEFAULT),
conf.getBoolean("hbase.storescanner.use.pread", false),
getCellsPerTimeoutCheck(conf),
conf.getBoolean(StoreScanner.STORESCANNER_PARALLEL_SEEK_ENABLE, false),
conf.getLong(StoreScanner.STORESCANNER_PREAD_MAX_BYTES, 4 * blockSize),
newVersionBehavior);{code}
> Use separated StoreFileReader for streaming read
>
>
> Key: HBASE-17910
> URL: https://issues.apache.org/jira/browse/HBASE-17910
> Project: HBase
> Issue Type: Improvement
> Components: regionserver, Scanners
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.0.0
>
>
> For now we have already supportted using private readers for compaction, by
> creating a new StoreFile copy. I think a better way is to allow creating
> multiple readers from a single StoreFile instance, thus we can avoid the ugly
> cloning, and the reader can also be used for streaming scan, not only for
> compaction.
> The reason we want to do this is that, we found a read amplification when
> using short circult read. {{BlockReaderLocal}} will use an internal buffer to
> read data first, the buffer size is based on the configured buffer size and
> the readahead option in CachingStrategy. For normal pread request, we should
> just bypass the buffer, this can be achieved by setting readahead to 0. But
> for streaming read I think the buffer is somehow still useful? So we need to
> use different FSDataInputStream for pread and streaming read.
> And one more thing is that, we can also remove the streamLock if streaming
> read always use its own reader.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-17910) Use separated StoreFileReader for streaming read
[
https://issues.apache.org/jira/browse/HBASE-17910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226438#comment-17226438
]
Karthik Palanisamy edited comment on HBASE-17910 at 11/5/20, 1:11 AM:
--
[~anoop.hbase] [~zhangduo] [~busbey] [~elserj]
Recently, one of our user reported high CPU usage in namenode. On our
troubleshooting, found millions of OPEN and GetFileInfo calls continuously to
namenode, is because of readType STREAM which creates multiple scanners. I
understand we switch readType to STREAM automatically but I don't find any flag
to disable STREAM. I am curious if that is the expected design?
The switch happens below,
If the scan become get.
if the scan with startrow and stoprow.
if the scan keeps running for long time. I.e kv bytesRead > preadMaxBytes.
(Default preadMaxBytes is 4*blockSize, which is 4*64KB).
Maybe this spike could be at every cluster but the user might not be noticed
yet. At this moment, I am trying to work around with
"hbase.storescanner.pread.max.bytes" and
"hbase.cells.scanned.per.heartbeat.check". Will post more updates next week
with the root cause.
{code:java}
this(family, minVersions, maxVersions, ttl, keepDeletedCells,
timeToPurgeDeletes, comparator,
conf.getLong(HConstants.TABLE_MAX_ROWSIZE_KEY,
HConstants.TABLE_MAX_ROWSIZE_DEFAULT),
conf.getBoolean("hbase.storescanner.use.pread", false),
getCellsPerTimeoutCheck(conf),
conf.getBoolean(StoreScanner.STORESCANNER_PARALLEL_SEEK_ENABLE, false),
conf.getLong(StoreScanner.STORESCANNER_PREAD_MAX_BYTES, 4 * blockSize),
newVersionBehavior);{code}
was (Author: kpalanisamy):
[~anoop.hbase] [~zhangduo] [~busbey] [~elserj]
Recently, one of our user reported high CPU usage in namenode. On our
troubleshooting, found millions of OPEN and GetFileInfo calls continuously to
namenode, is because of readType STREAM which creates multiple scanners. I
understand we switch readType to STREAM automatically but I don't find any flag
to disable STREAM. I am curious if that is the expected design?
If the scan become get.
if the scan with startrow and stoprow.
if the scan keeps running for long time. I.e kv bytesRead > preadMaxBytes.
(Default preadMaxBytes is 4*blockSize, which is 4*64KB).
Maybe this spike could be at every cluster but the user might not be noticed
yet. At this moment, I am trying to work around with
"hbase.storescanner.pread.max.bytes" and
"hbase.cells.scanned.per.heartbeat.check". Will post more updates next week
with the root cause.
{code:java}
this(family, minVersions, maxVersions, ttl, keepDeletedCells,
timeToPurgeDeletes, comparator,
conf.getLong(HConstants.TABLE_MAX_ROWSIZE_KEY,
HConstants.TABLE_MAX_ROWSIZE_DEFAULT),
conf.getBoolean("hbase.storescanner.use.pread", false),
getCellsPerTimeoutCheck(conf),
conf.getBoolean(StoreScanner.STORESCANNER_PARALLEL_SEEK_ENABLE, false),
conf.getLong(StoreScanner.STORESCANNER_PREAD_MAX_BYTES, 4 * blockSize),
newVersionBehavior);{code}
> Use separated StoreFileReader for streaming read
>
>
> Key: HBASE-17910
> URL: https://issues.apache.org/jira/browse/HBASE-17910
> Project: HBase
> Issue Type: Improvement
> Components: regionserver, Scanners
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.0.0
>
>
> For now we have already supportted using private readers for compaction, by
> creating a new StoreFile copy. I think a better way is to allow creating
> multiple readers from a single StoreFile instance, thus we can avoid the ugly
> cloning, and the reader can also be used for streaming scan, not only for
> compaction.
> The reason we want to do this is that, we found a read amplification when
> using short circult read. {{BlockReaderLocal}} will use an internal buffer to
> read data first, the buffer size is based on the configured buffer size and
> the readahead option in CachingStrategy. For normal pread request, we should
> just bypass the buffer, this can be achieved by setting readahead to 0. But
> for streaming read I think the buffer is somehow still useful? So we need to
> use different FSDataInputStream for pread and streaming read.
> And one more thing is that, we can also remove the streamLock if streaming
> read always use its own reader.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
