[jira] [Commented] (HBASE-17917) Use pread by default for all user scan and switch to streaming read if needed
[ https://issues.apache.org/jira/browse/HBASE-17917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389475#comment-16389475 ] Duo Zhang commented on HBASE-17917: --- Fine. Will prepare one soon. > Use pread by default for all user scan and switch to streaming read if needed > - > > Key: HBASE-17917 > URL: https://issues.apache.org/jira/browse/HBASE-17917 > Project: HBase > Issue Type: Sub-task > Components: scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0 > > Attachments: HBASE-17917-v1.patch, HBASE-17917-v2.patch, > HBASE-17917-v2.patch, HBASE-17917-v3.patch, HBASE-17917-v4.patch, > HBASE-17917-v5.patch, HBASE-17917-v6.patch, HBASE-17917-v6.patch, > HBASE-17917.patch > > > As said in the parent issue. We need some benchmark here first. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-17917) Use pread by default for all user scan and switch to streaming read if needed
[ https://issues.apache.org/jira/browse/HBASE-17917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389465#comment-16389465 ] Anoop Sam John commented on HBASE-17917: It would be great if this jira is having a Release Notes [~Apache9] detailing the change and when switch to stream read and what are its implications (if any) > Use pread by default for all user scan and switch to streaming read if needed > - > > Key: HBASE-17917 > URL: https://issues.apache.org/jira/browse/HBASE-17917 > Project: HBase > Issue Type: Sub-task > Components: scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 2.0.0 > > Attachments: HBASE-17917-v1.patch, HBASE-17917-v2.patch, > HBASE-17917-v2.patch, HBASE-17917-v3.patch, HBASE-17917-v4.patch, > HBASE-17917-v5.patch, HBASE-17917-v6.patch, HBASE-17917-v6.patch, > HBASE-17917.patch > > > As said in the parent issue. We need some benchmark here first. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-17917) Use pread by default for all user scan and switch to streaming read if needed
[ https://issues.apache.org/jira/browse/HBASE-17917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020843#comment-16020843 ] Hudson commented on HBASE-17917: FAILURE: Integrated in Jenkins build HBase-HBASE-14614 #244 (See [https://builds.apache.org/job/HBase-HBASE-14614/244/]) HBASE-17917 Use pread by default for all user scan (zhangduo: rev 0ae0edcd630aa1dcb6c47ea11fa4367ae0a5baa8) * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSwitchToStreamRead.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/TestIOFencing.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultCompactSelection.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactingMemStore.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanInfo.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksScanned.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileReader.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SegmentScanner.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/querymatcher/TestUserScanQueryMatcher.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/DelegatingKeyValueScanner.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultMemStore.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/MockStoreFile.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMajorCompaction.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/querymatcher/TestCompactionScanQueryMatcher.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestCoprocessorScanPolicy.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReversedStoreScanner.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScanner.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/NonLazyKeyValueScanner.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileInfo.java > Use pread by default for all user scan and switch to streaming read if needed > - > > Key: HBASE-17917 > URL: https://issues.apache.org/jira/browse/HBASE-17917 > Project: HBase > Issue Type: Sub-task > Components: scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17917.patch, HBASE-17917-v1.patch, > HBASE-17917-v2.patch, HBASE-17917-v2.patch, HBASE-17917-v3.patch, > HBASE-17917-v4.patch, HBASE-17917-v5.patch, HBASE-17917-v6.patch, > HBASE-17917-v6.patch > > > As said in the parent issue. We need some benchmark here first. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17917) Use pread by default for all user scan and switch to streaming read if needed
[ https://issues.apache.org/jira/browse/HBASE-17917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16006046#comment-16006046 ] Hudson commented on HBASE-17917: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2989 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2989/]) HBASE-17917 Use pread by default for all user scan (zhangduo: rev 0ae0edcd630aa1dcb6c47ea11fa4367ae0a5baa8) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/DelegatingKeyValueScanner.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactingMemStore.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/NonLazyKeyValueScanner.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScanner.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SegmentScanner.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileReader.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultMemStore.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanInfo.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileInfo.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSwitchToStreamRead.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestCoprocessorScanPolicy.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReversedStoreScanner.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultCompactSelection.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/MockStoreFile.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/querymatcher/TestUserScanQueryMatcher.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/querymatcher/TestCompactionScanQueryMatcher.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMajorCompaction.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/TestIOFencing.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksScanned.java > Use pread by default for all user scan and switch to streaming read if needed > - > > Key: HBASE-17917 > URL: https://issues.apache.org/jira/browse/HBASE-17917 > Project: HBase > Issue Type: Sub-task > Components: scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17917.patch, HBASE-17917-v1.patch, > HBASE-17917-v2.patch, HBASE-17917-v2.patch, HBASE-17917-v3.patch, > HBASE-17917-v4.patch, HBASE-17917-v5.patch, HBASE-17917-v6.patch, > HBASE-17917-v6.patch > > > As said in the parent issue. We need some benchmark here first. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17917) Use pread by default for all user scan and switch to streaming read if needed
[ https://issues.apache.org/jira/browse/HBASE-17917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16005834#comment-16005834 ] Duo Zhang commented on HBASE-17917: --- The failed UTs are unrelated and can pass locally. Will commit shortly. > Use pread by default for all user scan and switch to streaming read if needed > - > > Key: HBASE-17917 > URL: https://issues.apache.org/jira/browse/HBASE-17917 > Project: HBase > Issue Type: Sub-task > Components: scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17917.patch, HBASE-17917-v1.patch, > HBASE-17917-v2.patch, HBASE-17917-v2.patch, HBASE-17917-v3.patch, > HBASE-17917-v4.patch, HBASE-17917-v5.patch, HBASE-17917-v6.patch, > HBASE-17917-v6.patch > > > As said in the parent issue. We need some benchmark here first. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17917) Use pread by default for all user scan and switch to streaming read if needed
[
https://issues.apache.org/jira/browse/HBASE-17917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16004924#comment-16004924
]
Hadoop QA commented on HBASE-17917:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 28s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 16 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
8s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 18s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 1s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
56m 20s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 108m 51s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
1s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 191m 11s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestPerColumnFamilyFlush |
| Timed out junit tests |
org.apache.hadoop.hbase.replication.regionserver.TestWALEntryStream |
| | org.apache.hadoop.hbase.io.encoding.TestDataBlockEncoders |
| | org.apache.hadoop.hbase.replication.TestReplicationSmallTests |
| | org.apache.hadoop.hbase.io.encoding.TestLoadAndSwitchEncodeOnDisk |
| | org.apache.hadoop.hbase.filter.TestFuzzyRowFilterEndToEnd |
| | org.apache.hadoop.hbase.io.encoding.TestEncodedSeekers |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12867328/HBASE-17917-v6.patch |
| JIRA Issue | HBASE-17917 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 31cf2d91c44e 4.8.3-std-1 #1 SMP Fri Oct 21 11:15:43 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / c5cc81d |
| Default Java | 1.8.0_131 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/6751/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/6751/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
[jira] [Commented] (HBASE-17917) Use pread by default for all user scan and switch to streaming read if needed
[ https://issues.apache.org/jira/browse/HBASE-17917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16004128#comment-16004128 ] Guanghao Zhang commented on HBASE-17917: +1. bq. And we can also keep improving the trySwitchToStreamRead method to make it more intelligent. Open a new scanner for streaming read will lead a new request to NN. So look forward to see this improvement in the future. > Use pread by default for all user scan and switch to streaming read if needed > - > > Key: HBASE-17917 > URL: https://issues.apache.org/jira/browse/HBASE-17917 > Project: HBase > Issue Type: Sub-task > Components: scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17917.patch, HBASE-17917-v1.patch, > HBASE-17917-v2.patch, HBASE-17917-v2.patch, HBASE-17917-v3.patch, > HBASE-17917-v4.patch, HBASE-17917-v5.patch > > > As said in the parent issue. We need some benchmark here first. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17917) Use pread by default for all user scan and switch to streaming read if needed
[
https://issues.apache.org/jira/browse/HBASE-17917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16004099#comment-16004099
]
Duo Zhang commented on HBASE-17917:
---
{quote}
Can stream open be done in background? We keep preading till NN comes back?
(Can be new issue).
{quote}
Yes, this is an important optimization. Can do it in a follow on issue.
Thanks sir.
> Use pread by default for all user scan and switch to streaming read if needed
> -
>
> Key: HBASE-17917
> URL: https://issues.apache.org/jira/browse/HBASE-17917
> Project: HBase
> Issue Type: Sub-task
> Components: scan
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17917.patch, HBASE-17917-v1.patch,
> HBASE-17917-v2.patch, HBASE-17917-v2.patch, HBASE-17917-v3.patch,
> HBASE-17917-v4.patch, HBASE-17917-v5.patch
>
>
> As said in the parent issue. We need some benchmark here first.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17917) Use pread by default for all user scan and switch to streaming read if needed
[ https://issues.apache.org/jira/browse/HBASE-17917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16004079#comment-16004079 ] stack commented on HBASE-17917: --- [~Apache9] Skimmed latest. It looks great. Nice cleanup. I like the making stuff private. [~lhofhansl] ! bq. and if the kvs we scanned reaches this limit, we will reopen the scanner with stream. Can stream open be done in background? We keep preading till NN comes back? (Can be new issue). +1 from me. > Use pread by default for all user scan and switch to streaming read if needed > - > > Key: HBASE-17917 > URL: https://issues.apache.org/jira/browse/HBASE-17917 > Project: HBase > Issue Type: Sub-task > Components: scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17917.patch, HBASE-17917-v1.patch, > HBASE-17917-v2.patch, HBASE-17917-v2.patch, HBASE-17917-v3.patch, > HBASE-17917-v4.patch, HBASE-17917-v5.patch > > > As said in the parent issue. We need some benchmark here first. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17917) Use pread by default for all user scan and switch to streaming read if needed
[ https://issues.apache.org/jira/browse/HBASE-17917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16003893#comment-16003893 ] Duo Zhang commented on HBASE-17917: --- Ping [~lhofhansl] [~stack]. Thanks. > Use pread by default for all user scan and switch to streaming read if needed > - > > Key: HBASE-17917 > URL: https://issues.apache.org/jira/browse/HBASE-17917 > Project: HBase > Issue Type: Sub-task > Components: scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17917.patch, HBASE-17917-v1.patch, > HBASE-17917-v2.patch, HBASE-17917-v2.patch, HBASE-17917-v3.patch, > HBASE-17917-v4.patch, HBASE-17917-v5.patch > > > As said in the parent issue. We need some benchmark here first. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17917) Use pread by default for all user scan and switch to streaming read if needed
[ https://issues.apache.org/jira/browse/HBASE-17917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997739#comment-15997739 ] Duo Zhang commented on HBASE-17917: --- Ping [~lhofhansl]. And [~stack] What dou think of this approach sir? Thanks. > Use pread by default for all user scan and switch to streaming read if needed > - > > Key: HBASE-17917 > URL: https://issues.apache.org/jira/browse/HBASE-17917 > Project: HBase > Issue Type: Sub-task > Components: scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17917.patch, HBASE-17917-v1.patch, > HBASE-17917-v2.patch, HBASE-17917-v2.patch, HBASE-17917-v3.patch, > HBASE-17917-v4.patch, HBASE-17917-v5.patch > > > As said in the parent issue. We need some benchmark here first. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
