[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544805#comment-16544805 ] Jerry He commented on HBASE-20565: -- [~openinx] Thanks for the update. I can twist the application to use ColumnRangeFilter (startColumn, endColumn), and ColumnPaginationFilter(limit, column-name-offset). It is ok to do it, and easy when starting from the beginning since column-name-offset is the startColumn. But then we will have to know the last retrieved column name to set the next round. > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 1.4.4 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Attachments: debug.diff, debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20889) PE scan is failing with NullPointer
[
https://issues.apache.org/jira/browse/HBASE-20889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544798#comment-16544798
]
Vikas Vishwakarma commented on HBASE-20889:
---
+1 v2 looks good to me [~yuzhih...@gmail.com]
> PE scan is failing with NullPointer
> ---
>
> Key: HBASE-20889
> URL: https://issues.apache.org/jira/browse/HBASE-20889
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.3
>Reporter: Vikas Vishwakarma
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20889.branch-1.3.txt, 20889.branch-1.3.v2.txt
>
>
> Command used
> {code:java}
> ~/current/bigdata-hbase/hbase/hbase/bin/hbase pe --nomapred scan 1 >
> scan1{code}
> PE scan 1 is failing with NullPointer
> {code:java}
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.doLocalClients(PerformanceEvaluation.java:447)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:1920)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.run(PerformanceEvaluation.java:2305)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:2326)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$ScanTest.testTakedown(PerformanceEvaluation.java:1530)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:1165)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1896)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$1.call(PerformanceEvaluation.java:429)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$1.call(PerformanceEvaluation.java:424)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544730#comment-16544730 ] Zheng Hu commented on HBASE-20565: -- [~jinghe], I've talked with [~anoop.hbase], I think we have a basic idea to fix this bug. I'll upload the patch in these days. Thanks. > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 1.4.4 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Attachments: debug.diff, debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20866) HBase 1.x scan performance degradation compared to 0.98 version
[
https://issues.apache.org/jira/browse/HBASE-20866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544702#comment-16544702
]
Hudson commented on HBASE-20866:
Results for branch branch-1.3
[build #393 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/393/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/393//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/393//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/393//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> HBase 1.x scan performance degradation compared to 0.98 version
> ---
>
> Key: HBASE-20866
> URL: https://issues.apache.org/jira/browse/HBASE-20866
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.2
>Reporter: Vikas Vishwakarma
>Assignee: Vikas Vishwakarma
>Priority: Critical
> Fix For: 1.5.0, 1.2.7, 1.3.3, 1.4.6
>
> Attachments: HBASE-20866.branch-1.3.001.patch,
> HBASE-20866.branch-1.3.002.patch, HBASE-20866.branch-1.3.003.patch
>
>
> Internally while testing 1.3 as part of migration from 0.98 to 1.3 we
> observed perf degradation in scan performance for phoenix queries varying
> from few 10's to upto 200% depending on the query being executed. We tried
> simple native HBase scan and there also we saw upto 40% degradation in
> performance when the number of column qualifiers are high (40-50+)
> To identify the root cause of performance diff between 0.98 and 1.3 we
> carried out lot of experiments with profiling and git bisect iterations,
> however we were not able to identify any particular source of scan
> performance degradation and it looked like this is an accumulated degradation
> of 5-10% over various enhancements and refactoring.
> We identified few major enhancements like partialResult handling,
> ScannerContext with heartbeat processing, time/size limiting, RPC
> refactoring, etc that could have contributed to small degradation in
> performance which put together could be leading to large overall degradation.
> One of the changes is
> [HBASE-11544|https://jira.apache.org/jira/browse/HBASE-11544] which
> implements partialResult handling. In ClientScanner.java the results received
> from server are cached on the client side by converting the result array into
> an ArrayList. This function gets called in a loop depending on the number of
> rows in the scan result. Example for ten’s of millions of rows scanned, this
> can be called in the order of millions of times.
> In almost all the cases 99% of the time (except for handling partial results,
> etc). We are just taking the resultsFromServer converting it into a ArrayList
> resultsToAddToCache in addResultsToList(..) and then iterating over the list
> again and adding it to cache in loadCache(..) as given in the code path below
> In ClientScanner → loadCache(..) → getResultsToAddToCache(..) →
> addResultsToList(..) →
> {code:java}
> loadCache() {
> ...
> List resultsToAddToCache =
> getResultsToAddToCache(values, callable.isHeartbeatMessage());
> ...
> …
> for (Result rs : resultsToAddToCache) {
> rs = filterLoadedCell(rs);
> cache.add(rs);
> ...
> }
> }
> getResultsToAddToCache(..) {
> ..
> final boolean isBatchSet = scan != null && scan.getBatch() > 0;
> final boolean allowPartials = scan != null &&
> scan.getAllowPartialResults();
> ..
> if (allowPartials || isBatchSet) {
> addResultsToList(resultsToAddToCache, resultsFromServer, 0,
> (null == resultsFromServer ? 0 : resultsFromServer.length));
> return resultsToAddToCache;
> }
> ...
> }
> private void addResultsToList(List outputList, Result[] inputArray,
> int start, int end) {
> if (inputArray == null || start < 0 || end > inputArray.length) return;
> for (int i = start; i < end; i++) {
> outputList.add(inputArray[i]);
> }
> }{code}
>
> It looks like we can avoid the result array to arraylist conversion
> (resultsFromServer --> resultsToAddToCache ) for the first case which is also
> the most frequent case and instead directly take the values arraay returned
> by callable and add it to the cache without converting it into ArrayList.
> I have taken both these flags allowPartials and isBatchSet out in loadcahe()
> and I am directly adding
[jira] [Commented] (HBASE-18477) Umbrella JIRA for HBase Read Replica clusters
[
https://issues.apache.org/jira/browse/HBASE-18477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544679#comment-16544679
]
Hudson commented on HBASE-18477:
Results for branch HBASE-18477
[build #265 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/265/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/265//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/265//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/265//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/265//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> Umbrella JIRA for HBase Read Replica clusters
> -
>
> Key: HBASE-18477
> URL: https://issues.apache.org/jira/browse/HBASE-18477
> Project: HBase
> Issue Type: New Feature
>Reporter: Zach York
>Assignee: Zach York
>Priority: Major
> Attachments: HBase Read-Replica Clusters Scope doc.docx, HBase
> Read-Replica Clusters Scope doc.pdf, HBase Read-Replica Clusters Scope
> doc_v2.docx, HBase Read-Replica Clusters Scope doc_v2.pdf
>
>
> Recently, changes (such as HBASE-17437) have unblocked HBase to run with a
> root directory external to the cluster (such as in Amazon S3). This means
> that the data is stored outside of the cluster and can be accessible after
> the cluster has been terminated. One use case that is often asked about is
> pointing multiple clusters to one root directory (sharing the data) to have
> read resiliency in the case of a cluster failure.
>
> This JIRA is an umbrella JIRA to contain all the tasks necessary to create a
> read-replica HBase cluster that is pointed at the same root directory.
>
> This requires making the Read-Replica cluster Read-Only (no metadata
> operation or data operations).
> Separating the hbase:meta table for each cluster (Otherwise HBase gets
> confused with multiple clusters trying to update the meta table with their ip
> addresses)
> Adding refresh functionality for the meta table to ensure new metadata is
> picked up on the read replica cluster.
> Adding refresh functionality for HFiles for a given table to ensure new data
> is picked up on the read replica cluster.
>
> This can be used with any existing cluster that is backed by an external
> filesystem.
>
> Please note that this feature is still quite manual (with the potential for
> automation later).
>
> More information on this particular feature can be found here:
> https://aws.amazon.com/blogs/big-data/setting-up-read-replica-clusters-with-hbase-on-amazon-s3/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-20892) [UI] Start / End keys are empty on table.jsp
Ted Yu created HBASE-20892: -- Summary: [UI] Start / End keys are empty on table.jsp Key: HBASE-20892 URL: https://issues.apache.org/jira/browse/HBASE-20892 Project: HBase Issue Type: Bug Affects Versions: 2.0.1 Reporter: Ted Yu When viewing table.jsp?name=TestTable , I found that the Start / End keys for all the regions were simply dashes without real value. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20889) PE scan is failing with NullPointer
[
https://issues.apache.org/jira/browse/HBASE-20889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544601#comment-16544601
]
Hadoop QA commented on HBASE-20889:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m
2s{color} | {color:blue} The patch file was not named according to hbase's
naming conventions. Please see
https://yetus.apache.org/documentation/0.7.0/precommit-patchnames for
instructions. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-1.3 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
19s{color} | {color:green} branch-1.3 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} branch-1.3 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} branch-1.3 passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
21s{color} | {color:green} branch-1.3 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
28s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} branch-1.3 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} branch-1.3 passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
24s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 29s{color} | {color:green} Patch does not cause any errors with Hadoop 2.4.1
2.5.2 2.6.5 2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 84m
17s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}114m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:c57ccf7 |
| JIRA Issue | HBASE-20889 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12931704/20889.branch-1.3.v2.txt
|
[jira] [Commented] (HBASE-20865) CreateTableProcedure is stuck in retry loop in CREATE_TABLE_WRITE_FS_LAYOUT state
[
https://issues.apache.org/jira/browse/HBASE-20865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544592#comment-16544592
]
Hudson commented on HBASE-20865:
Results for branch branch-2.0
[build #551 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/551/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/551//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/551//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/551//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> CreateTableProcedure is stuck in retry loop in CREATE_TABLE_WRITE_FS_LAYOUT
> state
> -
>
> Key: HBASE-20865
> URL: https://issues.apache.org/jira/browse/HBASE-20865
> Project: HBase
> Issue Type: Bug
> Components: amv2
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1
>
> Attachments: HBASE-20865.master.001.patch
>
>
> Similar to HBASE-20616, CreateTableProcedure gets stuck in retry loop in
> CREATE_TABLE_WRITE_FS_LAYOUT state when writing HDFS fails.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20887) Backport HBASE-20865 "CreateTableProcedure is stuck in retry loop in CREATE_TABLE_WRITE_FS_LAYOUT state"
[
https://issues.apache.org/jira/browse/HBASE-20887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544591#comment-16544591
]
Hudson commented on HBASE-20887:
Results for branch branch-2.0
[build #551 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/551/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/551//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/551//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/551//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Backport HBASE-20865 "CreateTableProcedure is stuck in retry loop in
> CREATE_TABLE_WRITE_FS_LAYOUT state"
>
>
> Key: HBASE-20887
> URL: https://issues.apache.org/jira/browse/HBASE-20887
> Project: HBase
> Issue Type: Sub-task
> Components: backport
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
>Priority: Major
> Fix For: 2.0.2
>
> Attachments: HBASE-20887.branch-2.0.001.patch
>
>
> Backport parent issue to branch-2.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-18201) add UT and docs for DataBlockEncodingTool
[
https://issues.apache.org/jira/browse/HBASE-18201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544586#comment-16544586
]
Hadoop QA commented on HBASE-18201:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
29s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 5m
44s{color} | {color:blue} branch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
50s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
21s{color} | {color:red} root: The patch generated 6 new + 9 unchanged - 0
fixed = 15 total (was 9) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 5m
12s{color} | {color:blue} patch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
54s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 29s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 31m 24s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}111m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-18201 |
| JIRA
[jira] [Commented] (HBASE-20889) PE scan is failing with NullPointer
[
https://issues.apache.org/jira/browse/HBASE-20889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544569#comment-16544569
]
Ted Yu commented on HBASE-20889:
See if v2 is better.
I reuse the {{Scan}} variable of ScanTest.
> PE scan is failing with NullPointer
> ---
>
> Key: HBASE-20889
> URL: https://issues.apache.org/jira/browse/HBASE-20889
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.3
>Reporter: Vikas Vishwakarma
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20889.branch-1.3.txt, 20889.branch-1.3.v2.txt
>
>
> Command used
> {code:java}
> ~/current/bigdata-hbase/hbase/hbase/bin/hbase pe --nomapred scan 1 >
> scan1{code}
> PE scan 1 is failing with NullPointer
> {code:java}
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.doLocalClients(PerformanceEvaluation.java:447)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:1920)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.run(PerformanceEvaluation.java:2305)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:2326)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$ScanTest.testTakedown(PerformanceEvaluation.java:1530)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:1165)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1896)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$1.call(PerformanceEvaluation.java:429)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$1.call(PerformanceEvaluation.java:424)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20889) PE scan is failing with NullPointer
[
https://issues.apache.org/jira/browse/HBASE-20889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-20889:
---
Attachment: 20889.branch-1.3.v2.txt
> PE scan is failing with NullPointer
> ---
>
> Key: HBASE-20889
> URL: https://issues.apache.org/jira/browse/HBASE-20889
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.3
>Reporter: Vikas Vishwakarma
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20889.branch-1.3.txt, 20889.branch-1.3.v2.txt
>
>
> Command used
> {code:java}
> ~/current/bigdata-hbase/hbase/hbase/bin/hbase pe --nomapred scan 1 >
> scan1{code}
> PE scan 1 is failing with NullPointer
> {code:java}
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.doLocalClients(PerformanceEvaluation.java:447)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:1920)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.run(PerformanceEvaluation.java:2305)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:2326)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$ScanTest.testTakedown(PerformanceEvaluation.java:1530)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:1165)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1896)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$1.call(PerformanceEvaluation.java:429)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$1.call(PerformanceEvaluation.java:424)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20890) PE filterScan seems to be stuck forever
[
https://issues.apache.org/jira/browse/HBASE-20890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544565#comment-16544565
]
Vikas Vishwakarma commented on HBASE-20890:
---
oh ! thanks for debugging [~yuzhih...@gmail.com] , probably we should add
another parameter here to limit the number of iterations for filterScan. My
weekend test runs got completely taken over by filterScan test :)
> PE filterScan seems to be stuck forever
> ---
>
> Key: HBASE-20890
> URL: https://issues.apache.org/jira/browse/HBASE-20890
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.3
>Reporter: Vikas Vishwakarma
>Priority: Minor
>
> Command Used
> {code:java}
> ~/current/bigdata-hbase/hbase/hbase/bin/hbase pe --nomapred randomWrite 1 >
> write 2>&1
> ~/current/bigdata-hbase/hbase/hbase/bin/hbase pe --nomapred filterScan 1 >
> filterScan 2>&1
> {code}
>
> Output
> This kept running for several hours just printing the below messages in logs
>
> {code:java}
> -bash-4.1$ grep "Advancing internal scanner to startKey" filterScan.1 | head
> 2018-07-13 10:44:45,188 DEBUG [TestClient-0] client.ClientScanner - Advancing
> internal scanner to startKey at '52359'
> 2018-07-13 10:44:45,976 DEBUG [TestClient-0] client.ClientScanner - Advancing
> internal scanner to startKey at '52359'
> 2018-07-13 10:44:46,695 DEBUG [TestClient-0] client.ClientScanner - Advancing
> internal scanner to startKey at '52359'
> .
> -bash-4.1$ grep "Advancing internal scanner to startKey" filterScan.1 | tail
> 2018-07-15 06:20:22,353 DEBUG [TestClient-0] client.ClientScanner - Advancing
> internal scanner to startKey at '52359'
> 2018-07-15 06:20:23,044 DEBUG [TestClient-0] client.ClientScanner - Advancing
> internal scanner to startKey at '52359'
> 2018-07-15 06:20:23,768 DEBUG [TestClient-0] client.ClientScanner - Advancing
> internal scanner to startKey at '52359'
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20889) PE scan is failing with NullPointer
[
https://issues.apache.org/jira/browse/HBASE-20889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544559#comment-16544559
]
Vikas Vishwakarma edited comment on HBASE-20889 at 7/15/18 3:00 PM:
Thanks [~yuzhih...@gmail.com] for the quick fix. I am still wondering a bit
though why we have different handling for RandomScanWithRangeTest and ScanTest.
In ScanTest also shouldn't we be calling
updateScanMetrics(scan.getScanMetrics()); after every testRow() and not just in
testTakedown(). Probably open a new Jira for handling the ScanMetrics for
ScanTest or we could just move the updateScanMetrics() from takedown() to
testRow() in a final block as part of this Jira, that should also solve the
NullPointer as well as metric update issue ?
was (Author: vik.karma):
Thanks [~yuzhih...@gmail.com] for the quick fix. I am still wondering a bit
though why we have different handling for RandomScanWithRangeTest and ScanTest.
In ScanTest also shouldn't we be calling
updateScanMetrics(scan.getScanMetrics()); after every testRow() and not just in
testTakedown(). Probably open a new Jira for handling the ScanMetrics for
ScanTest or we could just move the updateScanMetrics() from takedown() to
testRow() in a final block as part of this Jira ?
> PE scan is failing with NullPointer
> ---
>
> Key: HBASE-20889
> URL: https://issues.apache.org/jira/browse/HBASE-20889
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.3
>Reporter: Vikas Vishwakarma
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20889.branch-1.3.txt
>
>
> Command used
> {code:java}
> ~/current/bigdata-hbase/hbase/hbase/bin/hbase pe --nomapred scan 1 >
> scan1{code}
> PE scan 1 is failing with NullPointer
> {code:java}
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.doLocalClients(PerformanceEvaluation.java:447)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:1920)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.run(PerformanceEvaluation.java:2305)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:2326)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$ScanTest.testTakedown(PerformanceEvaluation.java:1530)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:1165)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1896)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$1.call(PerformanceEvaluation.java:429)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$1.call(PerformanceEvaluation.java:424)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20889) PE scan is failing with NullPointer
[
https://issues.apache.org/jira/browse/HBASE-20889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544559#comment-16544559
]
Vikas Vishwakarma edited comment on HBASE-20889 at 7/15/18 2:59 PM:
Thanks [~yuzhih...@gmail.com] for the quick fix. I am still wondering a bit
though why we have different handling for RandomScanWithRangeTest and ScanTest.
In ScanTest also shouldn't we be calling
updateScanMetrics(scan.getScanMetrics()); after every testRow() and not just in
testTakedown(). Probably open a new Jira for handling the ScanMetrics for
ScanTest or we could just move the updateScanMetrics() from takedown() to
testRow() in a final block as part of this Jira ?
was (Author: vik.karma):
Thanks [~yuzhih...@gmail.com] for the quick fix. I am still wondering a bit
though why we have different handling for RandomScanWithRangeTest and ScanTest.
In ScanTest also shouldn't we be calling
updateScanMetrics(scan.getScanMetrics()); after every testRow() and not just in
testTakedown(). Probably open a new Jira for handling the ScanMetrics for
ScanTest?
> PE scan is failing with NullPointer
> ---
>
> Key: HBASE-20889
> URL: https://issues.apache.org/jira/browse/HBASE-20889
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.3
>Reporter: Vikas Vishwakarma
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20889.branch-1.3.txt
>
>
> Command used
> {code:java}
> ~/current/bigdata-hbase/hbase/hbase/bin/hbase pe --nomapred scan 1 >
> scan1{code}
> PE scan 1 is failing with NullPointer
> {code:java}
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.doLocalClients(PerformanceEvaluation.java:447)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:1920)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.run(PerformanceEvaluation.java:2305)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:2326)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$ScanTest.testTakedown(PerformanceEvaluation.java:1530)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:1165)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1896)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$1.call(PerformanceEvaluation.java:429)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$1.call(PerformanceEvaluation.java:424)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20890) PE filterScan seems to be stuck forever
[
https://issues.apache.org/jira/browse/HBASE-20890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544563#comment-16544563
]
Ted Yu commented on HBASE-20890:
It was just that FilteredScanTest is called for every row.
>From my cluster, it took about ~2 seconds for one row (5 regions for
>TestTable). Rough calculation shows that filterScan would take 23 days to
>finish :-)
> PE filterScan seems to be stuck forever
> ---
>
> Key: HBASE-20890
> URL: https://issues.apache.org/jira/browse/HBASE-20890
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.3
>Reporter: Vikas Vishwakarma
>Priority: Minor
>
> Command Used
> {code:java}
> ~/current/bigdata-hbase/hbase/hbase/bin/hbase pe --nomapred randomWrite 1 >
> write 2>&1
> ~/current/bigdata-hbase/hbase/hbase/bin/hbase pe --nomapred filterScan 1 >
> filterScan 2>&1
> {code}
>
> Output
> This kept running for several hours just printing the below messages in logs
>
> {code:java}
> -bash-4.1$ grep "Advancing internal scanner to startKey" filterScan.1 | head
> 2018-07-13 10:44:45,188 DEBUG [TestClient-0] client.ClientScanner - Advancing
> internal scanner to startKey at '52359'
> 2018-07-13 10:44:45,976 DEBUG [TestClient-0] client.ClientScanner - Advancing
> internal scanner to startKey at '52359'
> 2018-07-13 10:44:46,695 DEBUG [TestClient-0] client.ClientScanner - Advancing
> internal scanner to startKey at '52359'
> .
> -bash-4.1$ grep "Advancing internal scanner to startKey" filterScan.1 | tail
> 2018-07-15 06:20:22,353 DEBUG [TestClient-0] client.ClientScanner - Advancing
> internal scanner to startKey at '52359'
> 2018-07-15 06:20:23,044 DEBUG [TestClient-0] client.ClientScanner - Advancing
> internal scanner to startKey at '52359'
> 2018-07-15 06:20:23,768 DEBUG [TestClient-0] client.ClientScanner - Advancing
> internal scanner to startKey at '52359'
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20889) PE scan is failing with NullPointer
[
https://issues.apache.org/jira/browse/HBASE-20889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544559#comment-16544559
]
Vikas Vishwakarma commented on HBASE-20889:
---
Thanks [~yuzhih...@gmail.com] for the quick fix. I am still wondering a bit
though why we have different handling for RandomScanWithRangeTest and ScanTest.
In ScanTest also shouldn't we be calling
updateScanMetrics(scan.getScanMetrics()); after every testRow() and not just in
testTakedown(). Probably open a new Jira for handling the ScanMetrics for
ScanTest?
> PE scan is failing with NullPointer
> ---
>
> Key: HBASE-20889
> URL: https://issues.apache.org/jira/browse/HBASE-20889
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.3
>Reporter: Vikas Vishwakarma
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20889.branch-1.3.txt
>
>
> Command used
> {code:java}
> ~/current/bigdata-hbase/hbase/hbase/bin/hbase pe --nomapred scan 1 >
> scan1{code}
> PE scan 1 is failing with NullPointer
> {code:java}
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.doLocalClients(PerformanceEvaluation.java:447)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:1920)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.run(PerformanceEvaluation.java:2305)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:2326)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$ScanTest.testTakedown(PerformanceEvaluation.java:1530)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:1165)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1896)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$1.call(PerformanceEvaluation.java:429)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$1.call(PerformanceEvaluation.java:424)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-18201) add UT and docs for DataBlockEncodingTool
[ https://issues.apache.org/jira/browse/HBASE-18201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544557#comment-16544557 ] Kuan-Po Tseng commented on HBASE-18201: --- Patch 004 Bugs * When HFile includeTags is true, it does'nt mean all Cells in HFile contains tags, those don't contains tags will have a tagsLen((short) 0) part after the KeyValue structure. During the decoding process, decoder ROW_INDEX_V1 will not put tagsLen part back in cell if it is zero. Tests * Add DataBlockEncodingTool testing for HFile with cells contasin tags or not. Others * Fix some coding style problems. > add UT and docs for DataBlockEncodingTool > - > > Key: HBASE-18201 > URL: https://issues.apache.org/jira/browse/HBASE-18201 > Project: HBase > Issue Type: Sub-task > Components: tooling >Reporter: Chia-Ping Tsai >Assignee: Kuan-Po Tseng >Priority: Minor > Labels: beginner > Attachments: HBASE-18201.master.001.patch, > HBASE-18201.master.002.patch, HBASE-18201.master.002.patch, > HBASE-18201.master.003.patch, HBASE-18201.master.004.patch > > > There is no example, documents, or tests for DataBlockEncodingTool. We should > have it friendly if any use case exists. Otherwise, we should just get rid of > it because DataBlockEncodingTool presumes that the implementation of cell > returned from DataBlockEncoder is KeyValue. The presume may obstruct the > cleanup of KeyValue references in the code base of read/write path. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-18201) add UT and docs for DataBlockEncodingTool
[ https://issues.apache.org/jira/browse/HBASE-18201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuan-Po Tseng updated HBASE-18201: -- Attachment: HBASE-18201.master.004.patch > add UT and docs for DataBlockEncodingTool > - > > Key: HBASE-18201 > URL: https://issues.apache.org/jira/browse/HBASE-18201 > Project: HBase > Issue Type: Sub-task > Components: tooling >Reporter: Chia-Ping Tsai >Assignee: Kuan-Po Tseng >Priority: Minor > Labels: beginner > Attachments: HBASE-18201.master.001.patch, > HBASE-18201.master.002.patch, HBASE-18201.master.002.patch, > HBASE-18201.master.003.patch, HBASE-18201.master.004.patch > > > There is no example, documents, or tests for DataBlockEncodingTool. We should > have it friendly if any use case exists. Otherwise, we should just get rid of > it because DataBlockEncodingTool presumes that the implementation of cell > returned from DataBlockEncoder is KeyValue. The presume may obstruct the > cleanup of KeyValue references in the code base of read/write path. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20889) PE scan is failing with NullPointer
[
https://issues.apache.org/jira/browse/HBASE-20889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544551#comment-16544551
]
Hadoop QA commented on HBASE-20889:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m
2s{color} | {color:blue} The patch file was not named according to hbase's
naming conventions. Please see
https://yetus.apache.org/documentation/0.7.0/precommit-patchnames for
instructions. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-1.3 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
39s{color} | {color:green} branch-1.3 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} branch-1.3 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} branch-1.3 passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s{color} | {color:green} branch-1.3 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
13s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} branch-1.3 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} branch-1.3 passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
13s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
5m 51s{color} | {color:green} Patch does not cause any errors with Hadoop 2.4.1
2.5.2 2.6.5 2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}114m 33s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}135m 43s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.replication.TestReplicationSmallTests |
| | hadoop.hbase.master.normalizer.TestSimpleRegionNormalizerOnCluster |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce
[jira] [Updated] (HBASE-20889) PE scan is failing with NullPointer
[
https://issues.apache.org/jira/browse/HBASE-20889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-20889:
---
Priority: Major (was: Minor)
> PE scan is failing with NullPointer
> ---
>
> Key: HBASE-20889
> URL: https://issues.apache.org/jira/browse/HBASE-20889
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.3
>Reporter: Vikas Vishwakarma
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20889.branch-1.3.txt
>
>
> Command used
> {code:java}
> ~/current/bigdata-hbase/hbase/hbase/bin/hbase pe --nomapred scan 1 >
> scan1{code}
> PE scan 1 is failing with NullPointer
> {code:java}
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.doLocalClients(PerformanceEvaluation.java:447)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:1920)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.run(PerformanceEvaluation.java:2305)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:2326)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$ScanTest.testTakedown(PerformanceEvaluation.java:1530)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:1165)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1896)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$1.call(PerformanceEvaluation.java:429)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$1.call(PerformanceEvaluation.java:424)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20889) PE scan is failing with NullPointer
[
https://issues.apache.org/jira/browse/HBASE-20889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-20889:
---
Attachment: 20889.branch-1.3.txt
> PE scan is failing with NullPointer
> ---
>
> Key: HBASE-20889
> URL: https://issues.apache.org/jira/browse/HBASE-20889
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.3
>Reporter: Vikas Vishwakarma
>Priority: Minor
> Attachments: 20889.branch-1.3.txt
>
>
> Command used
> {code:java}
> ~/current/bigdata-hbase/hbase/hbase/bin/hbase pe --nomapred scan 1 >
> scan1{code}
> PE scan 1 is failing with NullPointer
> {code:java}
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.doLocalClients(PerformanceEvaluation.java:447)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:1920)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.run(PerformanceEvaluation.java:2305)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:2326)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$ScanTest.testTakedown(PerformanceEvaluation.java:1530)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:1165)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1896)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$1.call(PerformanceEvaluation.java:429)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$1.call(PerformanceEvaluation.java:424)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HBASE-20889) PE scan is failing with NullPointer
[
https://issues.apache.org/jira/browse/HBASE-20889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu reassigned HBASE-20889:
--
Assignee: Ted Yu
> PE scan is failing with NullPointer
> ---
>
> Key: HBASE-20889
> URL: https://issues.apache.org/jira/browse/HBASE-20889
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.3
>Reporter: Vikas Vishwakarma
>Assignee: Ted Yu
>Priority: Minor
> Attachments: 20889.branch-1.3.txt
>
>
> Command used
> {code:java}
> ~/current/bigdata-hbase/hbase/hbase/bin/hbase pe --nomapred scan 1 >
> scan1{code}
> PE scan 1 is failing with NullPointer
> {code:java}
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.doLocalClients(PerformanceEvaluation.java:447)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:1920)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.run(PerformanceEvaluation.java:2305)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:2326)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$ScanTest.testTakedown(PerformanceEvaluation.java:1530)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:1165)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1896)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$1.call(PerformanceEvaluation.java:429)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$1.call(PerformanceEvaluation.java:424)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20889) PE scan is failing with NullPointer
[
https://issues.apache.org/jira/browse/HBASE-20889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-20889:
---
Status: Patch Available (was: Open)
> PE scan is failing with NullPointer
> ---
>
> Key: HBASE-20889
> URL: https://issues.apache.org/jira/browse/HBASE-20889
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.3
>Reporter: Vikas Vishwakarma
>Assignee: Ted Yu
>Priority: Minor
> Attachments: 20889.branch-1.3.txt
>
>
> Command used
> {code:java}
> ~/current/bigdata-hbase/hbase/hbase/bin/hbase pe --nomapred scan 1 >
> scan1{code}
> PE scan 1 is failing with NullPointer
> {code:java}
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.doLocalClients(PerformanceEvaluation.java:447)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:1920)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.run(PerformanceEvaluation.java:2305)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:2326)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$ScanTest.testTakedown(PerformanceEvaluation.java:1530)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:1165)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1896)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$1.call(PerformanceEvaluation.java:429)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$1.call(PerformanceEvaluation.java:424)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20889) PE scan is failing with NullPointer
[
https://issues.apache.org/jira/browse/HBASE-20889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544531#comment-16544531
]
Ted Yu edited comment on HBASE-20889 at 7/15/18 11:54 AM:
--
Around line 1540:
{code}
void testRow(final int i) throws IOException {
if (this.testScanner == null) {
Scan scan = new Scan(format(opts.startRow));
{code}
It seems the Scan object should not be local to {{testRow}}.
The {{Scan}} variable of ScanTest should be modified instead.
was (Author: yuzhih...@gmail.com):
I looked at PerformanceEvaluation.java in branch-1.3 but the line numbers don't
match what were given in description. Can you tell me the commit hash you used
for PE run ?
Thanks
> PE scan is failing with NullPointer
> ---
>
> Key: HBASE-20889
> URL: https://issues.apache.org/jira/browse/HBASE-20889
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.3
>Reporter: Vikas Vishwakarma
>Priority: Minor
>
> Command used
> {code:java}
> ~/current/bigdata-hbase/hbase/hbase/bin/hbase pe --nomapred scan 1 >
> scan1{code}
> PE scan 1 is failing with NullPointer
> {code:java}
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.doLocalClients(PerformanceEvaluation.java:447)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:1920)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.run(PerformanceEvaluation.java:2305)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:2326)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$ScanTest.testTakedown(PerformanceEvaluation.java:1530)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:1165)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1896)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$1.call(PerformanceEvaluation.java:429)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$1.call(PerformanceEvaluation.java:424)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20889) PE scan is failing with NullPointer
[
https://issues.apache.org/jira/browse/HBASE-20889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544531#comment-16544531
]
Ted Yu commented on HBASE-20889:
I looked at PerformanceEvaluation.java in branch-1.3 but the line numbers don't
match what were given in description. Can you tell me the commit hash you used
for PE run ?
Thanks
> PE scan is failing with NullPointer
> ---
>
> Key: HBASE-20889
> URL: https://issues.apache.org/jira/browse/HBASE-20889
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.3
>Reporter: Vikas Vishwakarma
>Priority: Minor
>
> Command used
> {code:java}
> ~/current/bigdata-hbase/hbase/hbase/bin/hbase pe --nomapred scan 1 >
> scan1{code}
> PE scan 1 is failing with NullPointer
> {code:java}
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.doLocalClients(PerformanceEvaluation.java:447)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:1920)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.run(PerformanceEvaluation.java:2305)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:2326)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$ScanTest.testTakedown(PerformanceEvaluation.java:1530)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:1165)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1896)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$1.call(PerformanceEvaluation.java:429)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$1.call(PerformanceEvaluation.java:424)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20889) PE scan is failing with NullPointer
[
https://issues.apache.org/jira/browse/HBASE-20889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544527#comment-16544527
]
Ted Yu edited comment on HBASE-20889 at 7/15/18 11:38 AM:
--
In 2.0, I ran first:
{code}
hbase pe sequentialWrite 1
{code}
the above command gave me (no NPE):
{code}
2018-07-15 11:37:22,264 INFO [TestClient-0] hbase.PerformanceEvaluation:
Finished TestClient-0 in 87943ms over 1048576 rows
2018-07-15 11:37:22,265 INFO [main] hbase.PerformanceEvaluation: [ScanTest]
Summary of timings (ms): [87943]
2018-07-15 11:37:22,275 INFO [main] hbase.PerformanceEvaluation: [ScanTest
duration ] Min: 87943msMax: 87943msAvg: 87943ms
2018-07-15 11:37:22,275 INFO [main] hbase.PerformanceEvaluation: [ Avg latency
(us)] 83
2018-07-15 11:37:22,275 INFO [main] hbase.PerformanceEvaluation: [ Avg
TPS/QPS]11923row per second
{code}
was (Author: yuzhih...@gmail.com):
In 2.0, the above command gave me:
{code}
java.lang.IllegalStateException: Must specify an existing table for read
commands. Run a write command first.
at
org.apache.hadoop.hbase.PerformanceEvaluation.checkTable(PerformanceEvaluation.java:344)
at
org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:2379)
at
org.apache.hadoop.hbase.PerformanceEvaluation.run(PerformanceEvaluation.java:2882)
{code}
> PE scan is failing with NullPointer
> ---
>
> Key: HBASE-20889
> URL: https://issues.apache.org/jira/browse/HBASE-20889
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.3
>Reporter: Vikas Vishwakarma
>Priority: Minor
>
> Command used
> {code:java}
> ~/current/bigdata-hbase/hbase/hbase/bin/hbase pe --nomapred scan 1 >
> scan1{code}
> PE scan 1 is failing with NullPointer
> {code:java}
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.doLocalClients(PerformanceEvaluation.java:447)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:1920)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.run(PerformanceEvaluation.java:2305)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:2326)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$ScanTest.testTakedown(PerformanceEvaluation.java:1530)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:1165)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1896)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$1.call(PerformanceEvaluation.java:429)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$1.call(PerformanceEvaluation.java:424)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20889) PE scan is failing with NullPointer
[
https://issues.apache.org/jira/browse/HBASE-20889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544527#comment-16544527
]
Ted Yu commented on HBASE-20889:
In 2.0, the above command gave me:
{code}
java.lang.IllegalStateException: Must specify an existing table for read
commands. Run a write command first.
at
org.apache.hadoop.hbase.PerformanceEvaluation.checkTable(PerformanceEvaluation.java:344)
at
org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:2379)
at
org.apache.hadoop.hbase.PerformanceEvaluation.run(PerformanceEvaluation.java:2882)
{code}
> PE scan is failing with NullPointer
> ---
>
> Key: HBASE-20889
> URL: https://issues.apache.org/jira/browse/HBASE-20889
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.3
>Reporter: Vikas Vishwakarma
>Priority: Minor
>
> Command used
> {code:java}
> ~/current/bigdata-hbase/hbase/hbase/bin/hbase pe --nomapred scan 1 >
> scan1{code}
> PE scan 1 is failing with NullPointer
> {code:java}
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.doLocalClients(PerformanceEvaluation.java:447)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:1920)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.run(PerformanceEvaluation.java:2305)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:2326)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$ScanTest.testTakedown(PerformanceEvaluation.java:1530)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:1165)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1896)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$1.call(PerformanceEvaluation.java:429)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$1.call(PerformanceEvaluation.java:424)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20887) Backport HBASE-20865 "CreateTableProcedure is stuck in retry loop in CREATE_TABLE_WRITE_FS_LAYOUT state"
[ https://issues.apache.org/jira/browse/HBASE-20887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20887: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.0.2 Status: Resolved (was: Patch Available) Pushed to branch-2.0. Thanks [~brfrn169] for contributing. > Backport HBASE-20865 "CreateTableProcedure is stuck in retry loop in > CREATE_TABLE_WRITE_FS_LAYOUT state" > > > Key: HBASE-20887 > URL: https://issues.apache.org/jira/browse/HBASE-20887 > Project: HBase > Issue Type: Sub-task > Components: backport >Reporter: Toshihiro Suzuki >Assignee: Toshihiro Suzuki >Priority: Major > Fix For: 2.0.2 > > Attachments: HBASE-20887.branch-2.0.001.patch > > > Backport parent issue to branch-2.0. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20891) Avoid intermediate array to arraylist inter-conversions while loading scan cache
[ https://issues.apache.org/jira/browse/HBASE-20891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikas Vishwakarma updated HBASE-20891: -- Summary: Avoid intermediate array to arraylist inter-conversions while loading scan cache (was: Avoid intermediate array to arraylist conversions while loading scan cache) > Avoid intermediate array to arraylist inter-conversions while loading scan > cache > > > Key: HBASE-20891 > URL: https://issues.apache.org/jira/browse/HBASE-20891 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 3.0.0, 2.1.0 >Reporter: Vikas Vishwakarma >Assignee: Vikas Vishwakarma >Priority: Major > > As discussed in HBASE-20866, we would like to avoid array to arraylist > conversions while loading scan cache which is currently happening as part of > partial result handling. In HBASE-20866 we are handling the changes for > branch-1.x. In this request we will handle it for branch-2 and master branch, > since the code has been refactored and will require more changes compared to > branch-1 > Also preliminary look at the master branch shows that result handling has > been separated out into AllowPartialScanResultCache, BatchScanResultCache and > CompleteScanResultCache. > In case of BatchScanResultCache we are actually converting Result[] to > List for result grooming and then List back to toArray > before returning to loadCache() where it is added to cache. > So in case of BatchScanResultCache if we are able to directly load the > results to cache then we would be avoiding two intermediate conversions > * result Array to ArrayList in BatchScanResultCache for result grooming > * ArrayList to array conversion while returning to loadCache() > Which will probably give higher performance improvement compared to branch-1 > case handled in HBASE-20866 where we avoided just one result array to > arraylist conversion and saw upto 10% improvement in scan performance > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20891) Avoid intermediate array to arraylist conversions while loading scan cache
Vikas Vishwakarma created HBASE-20891: - Summary: Avoid intermediate array to arraylist conversions while loading scan cache Key: HBASE-20891 URL: https://issues.apache.org/jira/browse/HBASE-20891 Project: HBase Issue Type: Bug Components: Client Affects Versions: 3.0.0, 2.1.0 Reporter: Vikas Vishwakarma Assignee: Vikas Vishwakarma As discussed in HBASE-20866, we would like to avoid array to arraylist conversions while loading scan cache which is currently happening as part of partial result handling. In HBASE-20866 we are handling the changes for branch-1.x. In this request we will handle it for branch-2 and master branch, since the code has been refactored and will require more changes compared to branch-1 Also preliminary look at the master branch shows that result handling has been separated out into AllowPartialScanResultCache, BatchScanResultCache and CompleteScanResultCache. In case of BatchScanResultCache we are actually converting Result[] to List for result grooming and then List back to toArray before returning to loadCache() where it is added to cache. So in case of BatchScanResultCache if we are able to directly load the results to cache then we would be avoiding two intermediate conversions * result Array to ArrayList in BatchScanResultCache for result grooming * ArrayList to array conversion while returning to loadCache() Which will probably give higher performance improvement compared to branch-1 case handled in HBASE-20866 where we avoided just one result array to arraylist conversion and saw upto 10% improvement in scan performance -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20890) PE filterScan seems to be stuck forever
Vikas Vishwakarma created HBASE-20890:
-
Summary: PE filterScan seems to be stuck forever
Key: HBASE-20890
URL: https://issues.apache.org/jira/browse/HBASE-20890
Project: HBase
Issue Type: Bug
Affects Versions: 1.3.3
Reporter: Vikas Vishwakarma
Command Used
{code:java}
~/current/bigdata-hbase/hbase/hbase/bin/hbase pe --nomapred randomWrite 1 >
write 2>&1
~/current/bigdata-hbase/hbase/hbase/bin/hbase pe --nomapred filterScan 1 >
filterScan 2>&1
{code}
Output
This kept running for several hours just printing the below messages in logs
{code:java}
-bash-4.1$ grep "Advancing internal scanner to startKey" filterScan.1 | head
2018-07-13 10:44:45,188 DEBUG [TestClient-0] client.ClientScanner - Advancing
internal scanner to startKey at '52359'
2018-07-13 10:44:45,976 DEBUG [TestClient-0] client.ClientScanner - Advancing
internal scanner to startKey at '52359'
2018-07-13 10:44:46,695 DEBUG [TestClient-0] client.ClientScanner - Advancing
internal scanner to startKey at '52359'
.
-bash-4.1$ grep "Advancing internal scanner to startKey" filterScan.1 | tail
2018-07-15 06:20:22,353 DEBUG [TestClient-0] client.ClientScanner - Advancing
internal scanner to startKey at '52359'
2018-07-15 06:20:23,044 DEBUG [TestClient-0] client.ClientScanner - Advancing
internal scanner to startKey at '52359'
2018-07-15 06:20:23,768 DEBUG [TestClient-0] client.ClientScanner - Advancing
internal scanner to startKey at '52359'
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20889) PE scan is failing with NullPointer
[
https://issues.apache.org/jira/browse/HBASE-20889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikas Vishwakarma updated HBASE-20889:
--
Summary: PE scan is failing with NullPointer (was: PE scan 1 is failing
with NullPointer)
> PE scan is failing with NullPointer
> ---
>
> Key: HBASE-20889
> URL: https://issues.apache.org/jira/browse/HBASE-20889
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.3
>Reporter: Vikas Vishwakarma
>Priority: Minor
>
> Command used
> {code:java}
> ~/current/bigdata-hbase/hbase/hbase/bin/hbase pe --nomapred scan 1 >
> scan1{code}
> PE scan 1 is failing with NullPointer
> {code:java}
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.doLocalClients(PerformanceEvaluation.java:447)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:1920)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.run(PerformanceEvaluation.java:2305)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:2326)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$ScanTest.testTakedown(PerformanceEvaluation.java:1530)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:1165)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1896)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$1.call(PerformanceEvaluation.java:429)
> at
> org.apache.hadoop.hbase.PerformanceEvaluation$1.call(PerformanceEvaluation.java:424)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-20889) PE scan 1 is failing with NullPointer
Vikas Vishwakarma created HBASE-20889:
-
Summary: PE scan 1 is failing with NullPointer
Key: HBASE-20889
URL: https://issues.apache.org/jira/browse/HBASE-20889
Project: HBase
Issue Type: Bug
Affects Versions: 1.3.3
Reporter: Vikas Vishwakarma
Command used
{code:java}
~/current/bigdata-hbase/hbase/hbase/bin/hbase pe --nomapred scan 1 > scan1{code}
PE scan 1 is failing with NullPointer
{code:java}
java.io.IOException: java.lang.NullPointerException
at
org.apache.hadoop.hbase.PerformanceEvaluation.doLocalClients(PerformanceEvaluation.java:447)
at
org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:1920)
at
org.apache.hadoop.hbase.PerformanceEvaluation.run(PerformanceEvaluation.java:2305)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at
org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:2326)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hbase.PerformanceEvaluation$ScanTest.testTakedown(PerformanceEvaluation.java:1530)
at
org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:1165)
at
org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1896)
at
org.apache.hadoop.hbase.PerformanceEvaluation$1.call(PerformanceEvaluation.java:429)
at
org.apache.hadoop.hbase.PerformanceEvaluation$1.call(PerformanceEvaluation.java:424)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-20888) null values appearing in spark dataframe while reading data from apache phoenix
Raj Hans created HBASE-20888: Summary: null values appearing in spark dataframe while reading data from apache phoenix Key: HBASE-20888 URL: https://issues.apache.org/jira/browse/HBASE-20888 Project: HBase Issue Type: Bug Reporter: Raj Hans one column contains xml data. if xml size is small, data comes correctly in dataframe but as soon as xml length increase after a limit, for that particular row , few of the other column value show as null. i am using spark sql jdbc to read data from phoenix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20866) HBase 1.x scan performance degradation compared to 0.98 version
[
https://issues.apache.org/jira/browse/HBASE-20866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544416#comment-16544416
]
Hudson commented on HBASE-20866:
SUCCESS: Integrated in Jenkins build HBase-1.3-IT #432 (See
[https://builds.apache.org/job/HBase-1.3-IT/432/])
HBASE-20866 HBase 1.x scan performance degradation compared to 0.98 (vikasv:
rev f4fb871a197c1c021db55ad101f25b63c956a1c7)
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java
> HBase 1.x scan performance degradation compared to 0.98 version
> ---
>
> Key: HBASE-20866
> URL: https://issues.apache.org/jira/browse/HBASE-20866
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.2
>Reporter: Vikas Vishwakarma
>Assignee: Vikas Vishwakarma
>Priority: Critical
> Fix For: 1.5.0, 1.2.7, 1.3.3, 1.4.6
>
> Attachments: HBASE-20866.branch-1.3.001.patch,
> HBASE-20866.branch-1.3.002.patch, HBASE-20866.branch-1.3.003.patch
>
>
> Internally while testing 1.3 as part of migration from 0.98 to 1.3 we
> observed perf degradation in scan performance for phoenix queries varying
> from few 10's to upto 200% depending on the query being executed. We tried
> simple native HBase scan and there also we saw upto 40% degradation in
> performance when the number of column qualifiers are high (40-50+)
> To identify the root cause of performance diff between 0.98 and 1.3 we
> carried out lot of experiments with profiling and git bisect iterations,
> however we were not able to identify any particular source of scan
> performance degradation and it looked like this is an accumulated degradation
> of 5-10% over various enhancements and refactoring.
> We identified few major enhancements like partialResult handling,
> ScannerContext with heartbeat processing, time/size limiting, RPC
> refactoring, etc that could have contributed to small degradation in
> performance which put together could be leading to large overall degradation.
> One of the changes is
> [HBASE-11544|https://jira.apache.org/jira/browse/HBASE-11544] which
> implements partialResult handling. In ClientScanner.java the results received
> from server are cached on the client side by converting the result array into
> an ArrayList. This function gets called in a loop depending on the number of
> rows in the scan result. Example for ten’s of millions of rows scanned, this
> can be called in the order of millions of times.
> In almost all the cases 99% of the time (except for handling partial results,
> etc). We are just taking the resultsFromServer converting it into a ArrayList
> resultsToAddToCache in addResultsToList(..) and then iterating over the list
> again and adding it to cache in loadCache(..) as given in the code path below
> In ClientScanner → loadCache(..) → getResultsToAddToCache(..) →
> addResultsToList(..) →
> {code:java}
> loadCache() {
> ...
> List resultsToAddToCache =
> getResultsToAddToCache(values, callable.isHeartbeatMessage());
> ...
> …
> for (Result rs : resultsToAddToCache) {
> rs = filterLoadedCell(rs);
> cache.add(rs);
> ...
> }
> }
> getResultsToAddToCache(..) {
> ..
> final boolean isBatchSet = scan != null && scan.getBatch() > 0;
> final boolean allowPartials = scan != null &&
> scan.getAllowPartialResults();
> ..
> if (allowPartials || isBatchSet) {
> addResultsToList(resultsToAddToCache, resultsFromServer, 0,
> (null == resultsFromServer ? 0 : resultsFromServer.length));
> return resultsToAddToCache;
> }
> ...
> }
> private void addResultsToList(List outputList, Result[] inputArray,
> int start, int end) {
> if (inputArray == null || start < 0 || end > inputArray.length) return;
> for (int i = start; i < end; i++) {
> outputList.add(inputArray[i]);
> }
> }{code}
>
> It looks like we can avoid the result array to arraylist conversion
> (resultsFromServer --> resultsToAddToCache ) for the first case which is also
> the most frequent case and instead directly take the values arraay returned
> by callable and add it to the cache without converting it into ArrayList.
> I have taken both these flags allowPartials and isBatchSet out in loadcahe()
> and I am directly adding values to scanner cache if the above condition is
> pass instead of coverting it into arrayList by calling
> getResultsToAddToCache(). For example:
> {code:java}
> protected void loadCache() throws IOException {
> Result[] values = null;
> ..
> final boolean isBatchSet = scan != null && scan.getBatch() > 0;
> final boolean allowPartials = scan != null && scan.getAllowPartialResults();
> ..
> for (;;) {
> try {
> values = call(callable, caller, scannerTimeout);
> ..
> } catch (DoNotRetryIOException |
[jira] [Commented] (HBASE-20866) HBase 1.x scan performance degradation compared to 0.98 version
[
https://issues.apache.org/jira/browse/HBASE-20866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544414#comment-16544414
]
Vikas Vishwakarma commented on HBASE-20866:
---
thanks [~apurtell] [~yuzhih...@gmail.com] , i was able to commit to 1.3 but
looks like will need changes for other branches. Will add and commit the same.
Separately i am seeing some issues with pe . The filterScan test never
completes and was running for over 16 hours, also i am seeing nullpointers in
some cases, will log separate Jira for those.
> HBase 1.x scan performance degradation compared to 0.98 version
> ---
>
> Key: HBASE-20866
> URL: https://issues.apache.org/jira/browse/HBASE-20866
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.2
>Reporter: Vikas Vishwakarma
>Assignee: Vikas Vishwakarma
>Priority: Critical
> Fix For: 1.5.0, 1.2.7, 1.3.3, 1.4.6
>
> Attachments: HBASE-20866.branch-1.3.001.patch,
> HBASE-20866.branch-1.3.002.patch, HBASE-20866.branch-1.3.003.patch
>
>
> Internally while testing 1.3 as part of migration from 0.98 to 1.3 we
> observed perf degradation in scan performance for phoenix queries varying
> from few 10's to upto 200% depending on the query being executed. We tried
> simple native HBase scan and there also we saw upto 40% degradation in
> performance when the number of column qualifiers are high (40-50+)
> To identify the root cause of performance diff between 0.98 and 1.3 we
> carried out lot of experiments with profiling and git bisect iterations,
> however we were not able to identify any particular source of scan
> performance degradation and it looked like this is an accumulated degradation
> of 5-10% over various enhancements and refactoring.
> We identified few major enhancements like partialResult handling,
> ScannerContext with heartbeat processing, time/size limiting, RPC
> refactoring, etc that could have contributed to small degradation in
> performance which put together could be leading to large overall degradation.
> One of the changes is
> [HBASE-11544|https://jira.apache.org/jira/browse/HBASE-11544] which
> implements partialResult handling. In ClientScanner.java the results received
> from server are cached on the client side by converting the result array into
> an ArrayList. This function gets called in a loop depending on the number of
> rows in the scan result. Example for ten’s of millions of rows scanned, this
> can be called in the order of millions of times.
> In almost all the cases 99% of the time (except for handling partial results,
> etc). We are just taking the resultsFromServer converting it into a ArrayList
> resultsToAddToCache in addResultsToList(..) and then iterating over the list
> again and adding it to cache in loadCache(..) as given in the code path below
> In ClientScanner → loadCache(..) → getResultsToAddToCache(..) →
> addResultsToList(..) →
> {code:java}
> loadCache() {
> ...
> List resultsToAddToCache =
> getResultsToAddToCache(values, callable.isHeartbeatMessage());
> ...
> …
> for (Result rs : resultsToAddToCache) {
> rs = filterLoadedCell(rs);
> cache.add(rs);
> ...
> }
> }
> getResultsToAddToCache(..) {
> ..
> final boolean isBatchSet = scan != null && scan.getBatch() > 0;
> final boolean allowPartials = scan != null &&
> scan.getAllowPartialResults();
> ..
> if (allowPartials || isBatchSet) {
> addResultsToList(resultsToAddToCache, resultsFromServer, 0,
> (null == resultsFromServer ? 0 : resultsFromServer.length));
> return resultsToAddToCache;
> }
> ...
> }
> private void addResultsToList(List outputList, Result[] inputArray,
> int start, int end) {
> if (inputArray == null || start < 0 || end > inputArray.length) return;
> for (int i = start; i < end; i++) {
> outputList.add(inputArray[i]);
> }
> }{code}
>
> It looks like we can avoid the result array to arraylist conversion
> (resultsFromServer --> resultsToAddToCache ) for the first case which is also
> the most frequent case and instead directly take the values arraay returned
> by callable and add it to the cache without converting it into ArrayList.
> I have taken both these flags allowPartials and isBatchSet out in loadcahe()
> and I am directly adding values to scanner cache if the above condition is
> pass instead of coverting it into arrayList by calling
> getResultsToAddToCache(). For example:
> {code:java}
> protected void loadCache() throws IOException {
> Result[] values = null;
> ..
> final boolean isBatchSet = scan != null && scan.getBatch() > 0;
> final boolean allowPartials = scan != null && scan.getAllowPartialResults();
> ..
> for (;;) {
> try {
> values = call(callable, caller, scannerTimeout);
> ..
