[jira] [Created] (HBASE-17999) Pyspark HBase Connector
Waqar Muhammad created HBASE-17999: -- Summary: Pyspark HBase Connector Key: HBASE-17999 URL: https://issues.apache.org/jira/browse/HBASE-17999 Project: HBase Issue Type: Brainstorming Components: API Affects Versions: 1.2.4 Environment: Centos7, Python Reporter: Waqar Muhammad Is there a way/connector to connect hbase from pyspark and perform queries? Is there any official documentation for that? Would be awsome if someone could point me in the right direction Thanks in advance -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17899) VerifyReplication not exiting for invalid arguments even though we have check for it
[
https://issues.apache.org/jira/browse/HBASE-17899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997794#comment-15997794
]
Maddineni Sukumar commented on HBASE-17899:
---
[~apurtell] , I marked this issue fix is part of other JIra(HBASE-16466) and
marked it as resolved. Correct me if I did anything wrong here.
> VerifyReplication not exiting for invalid arguments even though we have check
> for it
>
>
> Key: HBASE-17899
> URL: https://issues.apache.org/jira/browse/HBASE-17899
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0, 1.3.1
>Reporter: Maddineni Sukumar
>Assignee: Maddineni Sukumar
>Priority: Minor
>
> In doCommandLine() method when ever we see invalid argument, instead of
> exiting we are continuing VerifyReplication job due to "return false;"
> missing in below if condition.
> if (cmd.startsWith("--")) {
> printUsage("Invalid argument '" + cmd + "'");
> }
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (HBASE-17899) VerifyReplication not exiting for invalid arguments even though we have check for it
[
https://issues.apache.org/jira/browse/HBASE-17899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maddineni Sukumar resolved HBASE-17899.
---
Resolution: Fixed
> VerifyReplication not exiting for invalid arguments even though we have check
> for it
>
>
> Key: HBASE-17899
> URL: https://issues.apache.org/jira/browse/HBASE-17899
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0, 1.3.1
>Reporter: Maddineni Sukumar
>Assignee: Maddineni Sukumar
>Priority: Minor
>
> In doCommandLine() method when ever we see invalid argument, instead of
> exiting we are continuing VerifyReplication job due to "return false;"
> missing in below if condition.
> if (cmd.startsWith("--")) {
> printUsage("Invalid argument '" + cmd + "'");
> }
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (HBASE-17900) VerifyReplication - input param variables declared as static causing issues to run VerifyReplication multiple times in single JVM(mainly for unit tests)
[ https://issues.apache.org/jira/browse/HBASE-17900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maddineni Sukumar resolved HBASE-17900. --- Resolution: Resolved Fix Version/s: 2.0.0 > VerifyReplication - input param variables declared as static causing issues > to run VerifyReplication multiple times in single JVM(mainly for unit tests) > - > > Key: HBASE-17900 > URL: https://issues.apache.org/jira/browse/HBASE-17900 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.3.1 >Reporter: Maddineni Sukumar >Assignee: Maddineni Sukumar >Priority: Minor > Fix For: 2.0.0 > > > VerifyReplication - input param variables declared as static causing issues > to run VerifyReplication multiple times in single JVM(mainly for unit tests) > Input params related variables are declared as static like below . > static long startTime = 0; > static long endTime = Long.MAX_VALUE; > So if I want to run VerifyReplication of table A and table B one after > another in single JVM then while running for Table B it is running with both > options provided during Table A and Table B because of static input > variables. > We need to make these variables as class level variables or use some cleanup > before each run. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17131) Avoid livelock caused by HRegion#processRowsWithLocks
[
https://issues.apache.org/jira/browse/HBASE-17131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997780#comment-15997780
]
Chia-Ping Tsai commented on HBASE-17131:
bq. anything interesting about that issue, was that in our code base?
Not much. It is inconvenient to use the transaction within single region, but i
had noticed another issue HBASE-16933.
> Avoid livelock caused by HRegion#processRowsWithLocks
> -
>
> Key: HBASE-17131
> URL: https://issues.apache.org/jira/browse/HBASE-17131
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.4.0, 1.3.1
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17131.branch-1.v0.patch, HBASE-17131.v0.patch
>
>
> {code:title=HRegion.java|borderStyle=solid}
> try {
> // STEP 2. Acquire the row lock(s)
> acquiredRowLocks = new ArrayList(rowsToLock.size());
> for (byte[] row : rowsToLock) {
> // Attempt to lock all involved rows, throw if any lock times out
> // use a writer lock for mixed reads and writes
> acquiredRowLocks.add(getRowLockInternal(row, false));
> }
> // STEP 3. Region lock
> lock(this.updatesLock.readLock(), acquiredRowLocks.size() == 0 ? 1 :
> acquiredRowLocks.size());
> locked = true;
> boolean success = false;
> long now = EnvironmentEdgeManager.currentTime();
> try {
> {code}
> We should lock all involved rows in the second try-finally. Otherwise, we
> won’t release the previous locks if any subsequent lock times out.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-15199) Move jruby jar so only on hbase-shell module classpath; currently globally available
[
https://issues.apache.org/jira/browse/HBASE-15199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997752#comment-15997752
]
Sean Busbey commented on HBASE-15199:
-
{quote}
However, there is a catch - if JRUBY_HOME is set as environment variable and
you use bin/start-hbase.sh, the jruby jar would appear in the classpath:
java.class.path=/a/jruby-1.6.8/lib/jruby.jar
{quote}
this part might not be HBASE-17878. One shouldn't have JRUBY_HOME exported in
the environment when launching the server processes normally. You should only
have it exported when you actually need your specific JRuby in the classpath.
> Move jruby jar so only on hbase-shell module classpath; currently globally
> available
>
>
> Key: HBASE-15199
> URL: https://issues.apache.org/jira/browse/HBASE-15199
> Project: HBase
> Issue Type: Task
> Components: dependencies, jruby, shell
>Reporter: stack
>Assignee: Xiang Li
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 15199.txt, HBASE-15199.master.001.patch,
> HBASE-15199.master.002.patch, HBASE-15199.master.003.patch
>
>
> A suggestion that came up out of internal issue (filed by Mr Jan Van Besien)
> was to move the scope of the jruby include down so it is only a dependency
> for the hbase-shell. jruby jar brings in a bunch of dependencies (joda time
> for example) which can clash with the includes of others. Our Sean suggests
> that could be good to shut down exploit possibilities if jruby was not
> globally available. Only downside I can think is that it may no longer be
> available to our bin/*rb scripts if we move the jar but perhaps these can be
> changed so they can find the ruby jar in new location.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17917) Use pread by default for all user scan and switch to streaming read if needed
[ https://issues.apache.org/jira/browse/HBASE-17917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997739#comment-15997739 ] Duo Zhang commented on HBASE-17917: --- Ping [~lhofhansl]. And [~stack] What dou think of this approach sir? Thanks. > Use pread by default for all user scan and switch to streaming read if needed > - > > Key: HBASE-17917 > URL: https://issues.apache.org/jira/browse/HBASE-17917 > Project: HBase > Issue Type: Sub-task > Components: scan >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17917.patch, HBASE-17917-v1.patch, > HBASE-17917-v2.patch, HBASE-17917-v2.patch, HBASE-17917-v3.patch, > HBASE-17917-v4.patch, HBASE-17917-v5.patch > > > As said in the parent issue. We need some benchmark here first. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997737#comment-15997737
]
Hadoop QA commented on HBASE-17786:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 24s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
7s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 20s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
27s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 6s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
55m 2s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 110m 7s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
2s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 191m 35s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.snapshot.TestMobSecureExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestMobExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestExportSnapshot |
| | org.apache.hadoop.hbase.TestPartialResultsFromClientSide |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12866513/HBASE-17786.002.patch
|
| JIRA Issue | HBASE-17786 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux f3a76649f0ee 4.8.3-std-1 #1 SMP Fri Oct 21 11:15:43 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 01af270 |
| Default Java | 1.8.0_131 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/6701/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/6701/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6701/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
[jira] [Updated] (HBASE-15199) Move jruby jar so only on hbase-shell module classpath; currently globally available
[ https://issues.apache.org/jira/browse/HBASE-15199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-15199: The original patch correctly accounts for JRUBY_HOME, no need for an addendum. Ted, it sounds like you should bring the s3 issue up on the jira that got marked as a DUP of this one. The problem is the same before and after. > Move jruby jar so only on hbase-shell module classpath; currently globally > available > > > Key: HBASE-15199 > URL: https://issues.apache.org/jira/browse/HBASE-15199 > Project: HBase > Issue Type: Task > Components: dependencies, jruby, shell >Reporter: stack >Assignee: Xiang Li >Priority: Critical > Fix For: 2.0.0 > > Attachments: 15199.txt, HBASE-15199.master.001.patch, > HBASE-15199.master.002.patch, HBASE-15199.master.003.patch > > > A suggestion that came up out of internal issue (filed by Mr Jan Van Besien) > was to move the scope of the jruby include down so it is only a dependency > for the hbase-shell. jruby jar brings in a bunch of dependencies (joda time > for example) which can clash with the includes of others. Our Sean suggests > that could be good to shut down exploit possibilities if jruby was not > globally available. Only downside I can think is that it may no longer be > available to our bin/*rb scripts if we move the jar but perhaps these can be > changed so they can find the ruby jar in new location. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17489) ClientScanner may send a next request to a RegionScanner which has been exhausted
[ https://issues.apache.org/jira/browse/HBASE-17489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997701#comment-15997701 ] Duo Zhang commented on HBASE-17489: --- Yeah in the hbase client we only use the scanner id when opening scanner since when calling next or close you need to send the scanner id to RS so you must have known the scanner id already... Mind openning a issue for this? I think it is easy to fix. We need to add some comments and also a UT to confirm that we do not remove it again in the future. Thanks. > ClientScanner may send a next request to a RegionScanner which has been > exhausted > - > > Key: HBASE-17489 > URL: https://issues.apache.org/jira/browse/HBASE-17489 > Project: HBase > Issue Type: Bug > Components: Client, scan >Affects Versions: 2.0.0, 1.3.0, 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.0.0, 1.4.0, 1.3.1 > > Attachments: HBASE-17489-branch-1.3.patch, > HBASE-17489-branch-1.patch, HBASE-17489.patch, HBASE-17489-v1.patch, > HBASE-17489-v2.patch, HBASE-17489-v3.patch, HBASE-17489-v4.patch, > HBASE-17489-v4.patch, HBASE-17489-v5.patch, HBASE-17489-v6.patch > > > Found it when implementing HBASE-17045. Seems the final result of the scan is > correct but no doubt the logic is broken. We need to fix it to stop things > get worse in the future. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15199) Move jruby jar so only on hbase-shell module classpath; currently globally available
[ https://issues.apache.org/jira/browse/HBASE-15199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997684#comment-15997684 ] Jerry He commented on HBASE-15199: -- Addendum to put this case in the 'if jruby_needed' of the original patch too? > Move jruby jar so only on hbase-shell module classpath; currently globally > available > > > Key: HBASE-15199 > URL: https://issues.apache.org/jira/browse/HBASE-15199 > Project: HBase > Issue Type: Task > Components: dependencies, jruby, shell >Reporter: stack >Assignee: Xiang Li >Priority: Critical > Fix For: 2.0.0 > > Attachments: 15199.txt, HBASE-15199.master.001.patch, > HBASE-15199.master.002.patch, HBASE-15199.master.003.patch > > > A suggestion that came up out of internal issue (filed by Mr Jan Van Besien) > was to move the scope of the jruby include down so it is only a dependency > for the hbase-shell. jruby jar brings in a bunch of dependencies (joda time > for example) which can clash with the includes of others. Our Sean suggests > that could be good to shut down exploit possibilities if jruby was not > globally available. Only downside I can think is that it may no longer be > available to our bin/*rb scripts if we move the jar but perhaps these can be > changed so they can find the ruby jar in new location. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17489) ClientScanner may send a next request to a RegionScanner which has been exhausted
[ https://issues.apache.org/jira/browse/HBASE-17489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997677#comment-15997677 ] Lars Hofhansl commented on HBASE-17489: --- Hey [~Apache9]. Quick question: It seems that now we only set the scanner id in the ScanResponse when we actually created a new region scanner (but not on next or close). Is that by design? There is a problem with this now with OpenTSBD (well AsyncHBase) which ties requests and responses together using the scanner id. > ClientScanner may send a next request to a RegionScanner which has been > exhausted > - > > Key: HBASE-17489 > URL: https://issues.apache.org/jira/browse/HBASE-17489 > Project: HBase > Issue Type: Bug > Components: Client, scan >Affects Versions: 2.0.0, 1.3.0, 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.0.0, 1.4.0, 1.3.1 > > Attachments: HBASE-17489-branch-1.3.patch, > HBASE-17489-branch-1.patch, HBASE-17489.patch, HBASE-17489-v1.patch, > HBASE-17489-v2.patch, HBASE-17489-v3.patch, HBASE-17489-v4.patch, > HBASE-17489-v4.patch, HBASE-17489-v5.patch, HBASE-17489-v6.patch > > > Found it when implementing HBASE-17045. Seems the final result of the scan is > correct but no doubt the logic is broken. We need to fix it to stop things > get worse in the future. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-15199) Move jruby jar so only on hbase-shell module classpath; currently globally available
[
https://issues.apache.org/jira/browse/HBASE-15199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997606#comment-15997606
]
Ted Yu edited comment on HBASE-15199 at 5/5/17 12:09 AM:
-
The above procedure allows shell to work.
However, there is a catch - if JRUBY_HOME is set as environment variable and
you use bin/start-hbase.sh, the jruby jar would appear in the classpath:
java.class.path=/a/jruby-1.6.8/lib/jruby.jar
leading to the following exception when rootdir is on s3(a):
{code}
2017-05-04 23:53:48,192 FATAL [cn012:46854.activeMasterManager] master.HMaster:
Failed to become active master
java.nio.file.AccessDeniedException: : getFileStatus on :
com.amazonaws.services.s3.model.AmazonS3Exception: AWS authentication requires
a valid Date or x-amz-date header (Service: Amazon S3; Status Code: 403; Error
Code: AccessDenied; Request ID: B1725B25836D2604), S3 Extended Request ID:
f8+vQ1XvtF3iCC9RBCtOnsQrMdQTvWSJb930LTkS4BJxRqZHuUNoI/fFELMV8ndMBnYgRp4x91g=
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:158)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1635)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:117)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1447)
at org.apache.hadoop.fs.s3a.S3AFileSystem.exists(S3AFileSystem.java:2040)
at
org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:462)
at
org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:162)
at
org.apache.hadoop.hbase.master.MasterFileSystem.(MasterFileSystem.java:142)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:724)
{code}
Should we document this ?
If so, I can open a JIRA.
was (Author: yuzhih...@gmail.com):
The above procedure allows shell to work.
However, there is a catch - if JRUBY_HOME is set as environment variable and
you use bin/start-hbase.sh, the jruby jar would appear in the classpath:
java.class.path=/a/jruby-1.6.8/lib/jruby.jar
leading to the following exception:
{code}
2017-05-04 23:53:48,192 FATAL [cn012:46854.activeMasterManager] master.HMaster:
Failed to become active master
java.nio.file.AccessDeniedException: : getFileStatus on :
com.amazonaws.services.s3.model.AmazonS3Exception: AWS authentication requires
a valid Date or x-amz-date header (Service: Amazon S3; Status Code: 403; Error
Code: AccessDenied; Request ID: B1725B25836D2604), S3 Extended Request ID:
f8+vQ1XvtF3iCC9RBCtOnsQrMdQTvWSJb930LTkS4BJxRqZHuUNoI/fFELMV8ndMBnYgRp4x91g=
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:158)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1635)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:117)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1447)
at org.apache.hadoop.fs.s3a.S3AFileSystem.exists(S3AFileSystem.java:2040)
at
org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:462)
at
org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:162)
at
org.apache.hadoop.hbase.master.MasterFileSystem.(MasterFileSystem.java:142)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:724)
{code}
Should we document this ?
If so, I can open a JIRA.
> Move jruby jar so only on hbase-shell module classpath; currently globally
> available
>
>
> Key: HBASE-15199
> URL: https://issues.apache.org/jira/browse/HBASE-15199
> Project: HBase
> Issue Type: Task
> Components: dependencies, jruby, shell
>Reporter: stack
>Assignee: Xiang Li
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 15199.txt, HBASE-15199.master.001.patch,
> HBASE-15199.master.002.patch, HBASE-15199.master.003.patch
>
>
> A suggestion that came up out of internal issue (filed by Mr Jan Van Besien)
> was to move the scope of the jruby include down so it is only a dependency
> for the hbase-shell. jruby jar brings in a bunch of dependencies (joda time
> for example) which can clash with the includes of others. Our Sean suggests
> that could be good to shut down exploit possibilities if jruby was not
> globally available. Only downside I can think is that it may no longer be
> available to our bin/*rb scripts if we move the jar but perhaps these can be
> changed so they can find the ruby jar in new location.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-15199) Move jruby jar so only on hbase-shell module classpath; currently globally available
[
https://issues.apache.org/jira/browse/HBASE-15199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997606#comment-15997606
]
Ted Yu commented on HBASE-15199:
The above procedure allows shell to work.
However, there is a catch - if JRUBY_HOME is set as environment variable and
you use bin/start-hbase.sh, the jruby jar would appear in the classpath:
java.class.path=/a/jruby-1.6.8/lib/jruby.jar
leading to the following exception:
{code}
2017-05-04 23:53:48,192 FATAL [cn012:46854.activeMasterManager] master.HMaster:
Failed to become active master
java.nio.file.AccessDeniedException: : getFileStatus on :
com.amazonaws.services.s3.model.AmazonS3Exception: AWS authentication requires
a valid Date or x-amz-date header (Service: Amazon S3; Status Code: 403; Error
Code: AccessDenied; Request ID: B1725B25836D2604), S3 Extended Request ID:
f8+vQ1XvtF3iCC9RBCtOnsQrMdQTvWSJb930LTkS4BJxRqZHuUNoI/fFELMV8ndMBnYgRp4x91g=
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:158)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1635)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:117)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1447)
at org.apache.hadoop.fs.s3a.S3AFileSystem.exists(S3AFileSystem.java:2040)
at
org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:462)
at
org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:162)
at
org.apache.hadoop.hbase.master.MasterFileSystem.(MasterFileSystem.java:142)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:724)
{code}
Should we document this ?
If so, I can open a JIRA.
> Move jruby jar so only on hbase-shell module classpath; currently globally
> available
>
>
> Key: HBASE-15199
> URL: https://issues.apache.org/jira/browse/HBASE-15199
> Project: HBase
> Issue Type: Task
> Components: dependencies, jruby, shell
>Reporter: stack
>Assignee: Xiang Li
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 15199.txt, HBASE-15199.master.001.patch,
> HBASE-15199.master.002.patch, HBASE-15199.master.003.patch
>
>
> A suggestion that came up out of internal issue (filed by Mr Jan Van Besien)
> was to move the scope of the jruby include down so it is only a dependency
> for the hbase-shell. jruby jar brings in a bunch of dependencies (joda time
> for example) which can clash with the includes of others. Our Sean suggests
> that could be good to shut down exploit possibilities if jruby was not
> globally available. Only downside I can think is that it may no longer be
> available to our bin/*rb scripts if we move the jar but perhaps these can be
> changed so they can find the ruby jar in new location.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-15199) Move jruby jar so only on hbase-shell module classpath; currently globally available
[
https://issues.apache.org/jira/browse/HBASE-15199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997598#comment-15997598
]
Sean Busbey commented on HBASE-15199:
-
JRUBY_HOME needs to be jruby install, not just a location that includes a
renamed jruby-complete jar. jruby.jar that it looks for isn't meant to be the
whole shebang (note how the shell sets the jruby support files to look in
JRUBY_HOME/lib, that's where it expects to find e.g. the ruby support files
that your invocation is complaining about).
{code}
Downloads busbey$ curl -O
https://s3.amazonaws.com/jruby.org/downloads/1.6.8/jruby-bin-1.6.8.tar.gz
% Total% Received % Xferd Average Speed TimeTime Time Current
Dload Upload Total SpentLeft Speed
100 14.9M 100 14.9M0 0 3960k 0 0:00:03 0:00:03 --:--:-- 3960k
Downloads busbey$ tar xzf jruby-bin-1.6.8.tar.gz
Downloads busbey$ cd hbase-2.0.0-SNAPSHOT
hbase-2.0.0-SNAPSHOT busbey$ JRUBY_HOME=~/Downloads/jruby-1.6.8 ./bin/hbase
shell
2017-05-04 18:37:26,740 WARN [main] util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/Users/busbey/tmp_projects/hbase/hbase-assembly/target/hbase-2.0.0-SNAPSHOT/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/local/Cellar/hadoop/2.7.3/libexec/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
HBase Shell
Use "help" to get list of supported commands.
Use "exit" to quit this interactive shell.
Version 2.0.0-SNAPSHOT, r01af27061e97dd82de79f6900189c868f8919a2c, Thu May 4
18:22:39 CDT 2017
Took 0.0440 seconds
1.8.7-p357 :001 > exit
{code}
> Move jruby jar so only on hbase-shell module classpath; currently globally
> available
>
>
> Key: HBASE-15199
> URL: https://issues.apache.org/jira/browse/HBASE-15199
> Project: HBase
> Issue Type: Task
> Components: dependencies, jruby, shell
>Reporter: stack
>Assignee: Xiang Li
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 15199.txt, HBASE-15199.master.001.patch,
> HBASE-15199.master.002.patch, HBASE-15199.master.003.patch
>
>
> A suggestion that came up out of internal issue (filed by Mr Jan Van Besien)
> was to move the scope of the jruby include down so it is only a dependency
> for the hbase-shell. jruby jar brings in a bunch of dependencies (joda time
> for example) which can clash with the includes of others. Our Sean suggests
> that could be good to shut down exploit possibilities if jruby was not
> globally available. Only downside I can think is that it may no longer be
> available to our bin/*rb scripts if we move the jar but perhaps these can be
> changed so they can find the ruby jar in new location.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997592#comment-15997592
]
Umesh Agashe commented on HBASE-17786:
--
Linked reviewboard: https://reviews.apache.org/r/59003/.
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17786.001.patch, HBASE-17786.002.patch
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997589#comment-15997589
]
Sean Busbey commented on HBASE-17786:
-
thanks! can you link to the reviewboard here and I'll use it to look at v2?
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17786.001.patch, HBASE-17786.002.patch
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Umesh Agashe updated HBASE-17786:
-
Attachment: HBASE-17786.002.patch
Thanks [~busbey] for the review. I have done the changes per your comments.
Please see the 002 patch.
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17786.001.patch, HBASE-17786.002.patch
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17995) improve log messages during snapshot related tests
[ https://issues.apache.org/jira/browse/HBASE-17995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997583#comment-15997583 ] Sean Busbey commented on HBASE-17995: - I had to retest that test when working on HBASE-17964 earlier. AFAICT it's unrelated. > improve log messages during snapshot related tests > -- > > Key: HBASE-17995 > URL: https://issues.apache.org/jira/browse/HBASE-17995 > Project: HBase > Issue Type: Improvement > Components: integration tests, mapreduce, snapshots, test >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-17995.0.patch > > > while verifying the changes for HBASE-17964 I had to chase down a failure > related to having the wrong hbase configs. Adding some additional logging > detail let me see what was happening. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997579#comment-15997579
]
Umesh Agashe commented on HBASE-17786:
--
bq. nit: the file header should use "/*" instead of "/**" because it's not
meant to be a javadoc parsed comment. I think we make this mistake in lots of
places already, so not a big deal if you leave it.
Makes sense. Fixed. 2428 out of 3528 Java files use /**. In future, git add
hook would be useful to check this.
bq. We've been inconsistent on wether non-Test classes in the test packages
need InterfaceAudience annotations. Since this is a tool, best to be on the
safe side and label it IA.LimitedPrivate(TOOL).
Fixed.
bq. So this is general cleanup while you're in the area?
Yes
bq. Comments like this would be good to have as output while running as DEBUG
or INFO messages (perhaps on stderr if we can't use a logger for some reason).
Fixed. Added a INFO log message for it.
bq. Why this variable rather than just using
HConstants.DEFAULT_REGIONSERVER_PORT?
Fixed.
bq. Class description should include an example of how we expect folks to
invoke this.
Fixed.
{quote}
bq. What's the rationale for using System.out directly instead of a logger?
Thinking about this more, and looking at e.g. the existing
PerformanceEvaluation tool, I think we should update this to use commons-logger
as most of the rest of our stuff does.
{quote}
I referred to tests HFilePerformanceEvaluation and
ProcedureWALPerformanceEvaluation which have System.out.print statements. Most
PerformanceEvaluation tools (9/12) have System.out.print statements. Looking at
the output I think the rational is: logs get printed to stderr while explicit
print statements print to stdout. stderr can be redirected to get user
presentable output with the tool.
{quote}
We should only need the stochastic load balancer settings if the load balancer
is the stochastic load balancer, I think?
Also, we should not set these values if the incoming configuration had them set
by the user running the perf tool.
{quote}
Removed. User can set these in conf, defaults will be used otherwise.
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17786.001.patch
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997564#comment-15997564
]
Hadoop QA commented on HBASE-17786:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 26s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 19s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
31s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 3s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
60m 50s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 116m 12s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
57s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 203m 32s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.snapshot.TestMobSecureExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestMobExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestExportSnapshot |
| | org.apache.hadoop.hbase.filter.TestFuzzyRowFilterEndToEnd |
| | org.apache.hadoop.hbase.TestPartialResultsFromClientSide |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12866478/HBASE-17786.001.patch
|
| JIRA Issue | HBASE-17786 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 2b4546525e28 4.8.3-std-1 #1 SMP Fri Oct 21 11:15:43 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 0fb1fd1 |
| Default Java | 1.8.0_131 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/6700/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/6700/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6700/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console
[jira] [Created] (HBASE-17998) Improve HBase RPC write throttling size estimation
Karan Mehta created HBASE-17998: --- Summary: Improve HBase RPC write throttling size estimation Key: HBASE-17998 URL: https://issues.apache.org/jira/browse/HBASE-17998 Project: HBase Issue Type: Improvement Reporter: Karan Mehta Assignee: Karan Mehta Currently when RPC throttling, the size of each put is estimated using a hardcoded value 100 bytes. This can be improved by using the protobuf size as an estimate, without having to deserialize or do a big refactoring. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17990) Refactor TestFSUtils to use Before
[ https://issues.apache.org/jira/browse/HBASE-17990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-17990: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.0.0 Status: Resolved (was: Patch Available) HBASE-17990 was missing in the subject of the patch. I added it before committing. > Refactor TestFSUtils to use Before > -- > > Key: HBASE-17990 > URL: https://issues.apache.org/jira/browse/HBASE-17990 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Zach York >Assignee: Zach York >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-17990.master.001.patch > > > TestFSUtils has many tests that reuse functionality. These should be changed > to use a common setup method to avoid code duplication. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-14925) Develop HBase shell command/tool to list table's region info through command line
[ https://issues.apache.org/jira/browse/HBASE-14925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997479#comment-15997479 ] Andrew Purtell commented on HBASE-14925: The addendum patch contains more changes than we typically accept for an addendum, which are only the specific changes to address review feedback that comes in late. That said it all looks useful. The change to use toStringBinary was done. Let me apply this patch and get an idea of what the format looks like. Unless I come back soon to say otherwise I'm +1 to commit provided at least one of the others providing feedback agree. > Develop HBase shell command/tool to list table's region info through command > line > - > > Key: HBASE-14925 > URL: https://issues.apache.org/jira/browse/HBASE-14925 > Project: HBase > Issue Type: Improvement > Components: shell >Reporter: Romil Choksi >Assignee: Karan Mehta > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-14925.002.patch, > HBASE-14925.003.addendum.001.patch, HBASE-14925.003.patch, HBASE-14925.patch > > > I am going through the hbase shell commands to see if there is anything I can > use to get all the regions info just for a particular table. I don’t see any > such command that provides me that information. > It would be better to have a command that provides region info, start key, > end key etc taking a table name as the input parameter. This is available > through HBase UI on clicking on a particular table's link > A tool/shell command to get a list of regions for a table or all tables in a > tabular structured output (that is machine readable) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15199) Move jruby jar so only on hbase-shell module classpath; currently globally available
[
https://issues.apache.org/jira/browse/HBASE-15199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997472#comment-15997472
]
Ted Yu commented on HBASE-15199:
Here is the dir listing:
{code}
dir/a/jruby:
total 4
drwxr-xr-x 2 hbase hadoop 4096 May 4 17:57 lib
dir/a/jruby/lib:
total 13512
-rw-r--r-- 1 hbase hadoop 13832273 May 4 17:57 jruby.jar
{code}
jruby.jar is actually jruby-complete-1.6.8.jar
> Move jruby jar so only on hbase-shell module classpath; currently globally
> available
>
>
> Key: HBASE-15199
> URL: https://issues.apache.org/jira/browse/HBASE-15199
> Project: HBase
> Issue Type: Task
> Components: dependencies, jruby, shell
>Reporter: stack
>Assignee: Xiang Li
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 15199.txt, HBASE-15199.master.001.patch,
> HBASE-15199.master.002.patch, HBASE-15199.master.003.patch
>
>
> A suggestion that came up out of internal issue (filed by Mr Jan Van Besien)
> was to move the scope of the jruby include down so it is only a dependency
> for the hbase-shell. jruby jar brings in a bunch of dependencies (joda time
> for example) which can clash with the includes of others. Our Sean suggests
> that could be good to shut down exploit possibilities if jruby was not
> globally available. Only downside I can think is that it may no longer be
> available to our bin/*rb scripts if we move the jar but perhaps these can be
> changed so they can find the ruby jar in new location.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-15199) Move jruby jar so only on hbase-shell module classpath; currently globally available
[ https://issues.apache.org/jira/browse/HBASE-15199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997467#comment-15997467 ] Sean Busbey commented on HBASE-15199: - Does your install work with the shipped jruby? Does it work if you export JRUBY_HOME prior to invocation? Can you paste bin a recursive ls of wherever JRUBY_HOME is pointing? > Move jruby jar so only on hbase-shell module classpath; currently globally > available > > > Key: HBASE-15199 > URL: https://issues.apache.org/jira/browse/HBASE-15199 > Project: HBase > Issue Type: Task > Components: dependencies, jruby, shell >Reporter: stack >Assignee: Xiang Li >Priority: Critical > Fix For: 2.0.0 > > Attachments: 15199.txt, HBASE-15199.master.001.patch, > HBASE-15199.master.002.patch, HBASE-15199.master.003.patch > > > A suggestion that came up out of internal issue (filed by Mr Jan Van Besien) > was to move the scope of the jruby include down so it is only a dependency > for the hbase-shell. jruby jar brings in a bunch of dependencies (joda time > for example) which can clash with the includes of others. Our Sean suggests > that could be good to shut down exploit possibilities if jruby was not > globally available. Only downside I can think is that it may no longer be > available to our bin/*rb scripts if we move the jar but perhaps these can be > changed so they can find the ruby jar in new location. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15199) Move jruby jar so only on hbase-shell module classpath; currently globally available
[
https://issues.apache.org/jira/browse/HBASE-15199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997439#comment-15997439
]
Ted Yu commented on HBASE-15199:
This was what I did:
. produced hbase-2.0.0-SNAPSHOT-bin.tar.gz
. expanded the tar ball under /tmp
. made sure JRUBY_HOME points to a directory where lib/jruby.jar exists (copied
from jruby-complete jar)
. used the following command to launch shell:
{code}
JRUBY_HOME=/xx/jruby bin/hbase shell
{code}
I got:
{code}
LoadError: no such file to load -- irb/completion
require at org/jruby/RubyKernel.java:1062
(root) at /tmp/hbase-2.0.0-SNAPSHOT/bin/../bin/hirb.rb:41
{code}
What did I miss ?
> Move jruby jar so only on hbase-shell module classpath; currently globally
> available
>
>
> Key: HBASE-15199
> URL: https://issues.apache.org/jira/browse/HBASE-15199
> Project: HBase
> Issue Type: Task
> Components: dependencies, jruby, shell
>Reporter: stack
>Assignee: Xiang Li
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 15199.txt, HBASE-15199.master.001.patch,
> HBASE-15199.master.002.patch, HBASE-15199.master.003.patch
>
>
> A suggestion that came up out of internal issue (filed by Mr Jan Van Besien)
> was to move the scope of the jruby include down so it is only a dependency
> for the hbase-shell. jruby jar brings in a bunch of dependencies (joda time
> for example) which can clash with the includes of others. Our Sean suggests
> that could be good to shut down exploit possibilities if jruby was not
> globally available. Only downside I can think is that it may no longer be
> available to our bin/*rb scripts if we move the jar but perhaps these can be
> changed so they can find the ruby jar in new location.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997402#comment-15997402
]
Sean Busbey commented on HBASE-17786:
-
{code}
+ // Non-default configurations.
+ private void setupConf() {
+conf.setFloat("hbase.master.balancer.stochastic.maxMovePercent", 1.0f);
+conf.setLong(StochasticLoadBalancer.MAX_STEPS_KEY, 200L);
+conf.setFloat("hbase.master.balancer.stochastic.localityCost", 0);
+conf.setLong("hbase.master.balancer.stochastic.maxRunningTime", 300 *
1000); // 300 sec
+conf.setFloat("hbase.master.balancer.stochastic.minCostNeedBalance",
0.05f);
+
+conf.setClass(HConstants.HBASE_MASTER_LOADBALANCER_CLASS,
loadBalancerClazz, LoadBalancer.class);
+loadBalancer = LoadBalancerFactory.getLoadBalancer(conf);
+ }
{code}
We should only need the stochastic load balancer settings if the load balancer
is the stochastic load balancer, I think?
Also, we should not set these values if the incoming configuration had them set
by the user running the perf tool.
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17786.001.patch
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997393#comment-15997393
]
Sean Busbey commented on HBASE-17786:
-
bq. What's the rationale for using System.out directly instead of a logger?
Thinking about this more, and looking at e.g. the existing
PerformanceEvaluation tool, I think we should update this to use commons-logger
as most of the rest of our stuff does.
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17786.001.patch
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997390#comment-15997390
]
Sean Busbey commented on HBASE-17786:
-
{code}
+/**
+ * Tool to test performance of different {@link
org.apache.hadoop.hbase.master.LoadBalancer}
+ * implementations.
+ */
+public class LoadBalancerPerformanceEvaluation extends AbstractHBaseTool {
{code}
Class description should include an example of how we expect folks to invoke
this.
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17786.001.patch
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17977) Enabled Master observer to delete quotas on table deletion by default
[ https://issues.apache.org/jira/browse/HBASE-17977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997366#comment-15997366 ] Andrew Purtell commented on HBASE-17977: +1 > Enabled Master observer to delete quotas on table deletion by default > - > > Key: HBASE-17977 > URL: https://issues.apache.org/jira/browse/HBASE-17977 > Project: HBase > Issue Type: Sub-task >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: HBASE-16961 > > Attachments: HBASE-17977.001.HBASE-16961.patch, > HBASE-17977.002.HBASE-16961.patch > > > Deleting a table's quota when we delete the table should be the common case, > not the exception. > Load the observer by default (when quota support is enabled), and switch the > flag to disable loading it. Make sure to update the docs too. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17964) ensure hbase-metrics-api is included in mapreduce job classpaths
[
https://issues.apache.org/jira/browse/HBASE-17964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997359#comment-15997359
]
Hudson commented on HBASE-17964:
SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2950 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/2950/])
HBASE-17964 ensure hbase-metrics-api is included in mapreduce job (busbey: rev
0fb1fd12005f5c51924aa67d114813be896b5cc1)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
> ensure hbase-metrics-api is included in mapreduce job classpaths
>
>
> Key: HBASE-17964
> URL: https://issues.apache.org/jira/browse/HBASE-17964
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-17964.0.patch
>
>
> HBASE-9774 moved metrics stuff into a couple of additional modules.
> currently, MR against snapshots fail because it's unable to find the
> metrics-api classes it needs for ClientSideRegionScanner.
> {code}
> Error: java.io.IOException: java.lang.NoClassDefFoundError:
> org/apache/hadoop/hbase/metrics/Snapshot
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initializeStores(HRegion.java:1027)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:883)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:858)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6694)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6651)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6622)
> at
> org.apache.hadoop.hbase.client.ClientSideRegionScanner.(ClientSideRegionScanner.java:59)
> at
> org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormatImpl$RecordReader.initialize(TableSnapshotInputFormatImpl.java:211)
> at
> org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat$TableSnapshotRegionRecordReader.initialize(TableSnapshotInputFormat.java:140)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:548)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17991) Add more details about compaction queue on /dump
[
https://issues.apache.org/jira/browse/HBASE-17991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997358#comment-15997358
]
Hudson commented on HBASE-17991:
SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2950 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/2950/])
HBASE-17991 Add more details about compaction queue on /dump (Guangxu (tedyu:
rev 52119200bcfcc9b449c829c5e1b7e30afdb87be7)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java
> Add more details about compaction queue on /dump
>
>
> Key: HBASE-17991
> URL: https://issues.apache.org/jira/browse/HBASE-17991
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17991-master-v1.patch,
> HBASE-17991-master-v2.patch, HBASE-17991-master-v3.patch
>
>
> RS dump information as follows
> {code}
> RS Queue:
> ===
> Compaction/Split Queue summary: compaction_queue=(20:0), split_queue=0,
> merge_queue=0
> Compaction/Split Queue dump:
> LargeCompation Queue:
> Request =
> regionName=usertable,user4180497275766179957,1491904095205.1d4085e4438752f3611f66e7b043fe44.,
> storeName=0, fileCount=1, fileSize=1.9 G (1.9 G), priority=1,
> time=21697920409804647
> Request =
> regionName=usertable,user4568009753557153251,1491904099262.95bf004e3c9b35a58c60ca5d5b11d190.,
> storeName=0, fileCount=1, fileSize=1.9 G (1.9 G), priority=1,
> time=21697920413223800
> SmallCompation Queue:
> Store = b, pri = 108
> Store = b, pri = 108
> Store = b, pri = 108
> Store = b, pri = 108
> Store = b, pri = 108
> Store = b, pri = 109
> {code}
> Compaction queue information will be displayed on page /dump.
> If compation has selected the file, it will print the details information of
> the compation, otherwise only print storename and priority(Store = b, pri =
> 108) which is useless for us.
> So, we should also print more detailed information, such as regionName,
> starttime etc.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17534) SecureBulkLoadClient squashes DoNotRetryIOExceptions from the server
[ https://issues.apache.org/jira/browse/HBASE-17534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997353#comment-15997353 ] Hudson commented on HBASE-17534: SUCCESS: Integrated in Jenkins build HBase-1.3-JDK7 #157 (See [https://builds.apache.org/job/HBase-1.3-JDK7/157/]) HBASE-17534 Avoid re-wrapping IOExceptions as IOExceptions (elserj: rev 2ddb30c570ce239c4fb81213d468452709b14d78) * (add) hbase-client/src/test/java/org/apache/hadoop/hbase/client/coprocessor/TestSecureBulkLoadClient.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/coprocessor/SecureBulkLoadClient.java > SecureBulkLoadClient squashes DoNotRetryIOExceptions from the server > > > Key: HBASE-17534 > URL: https://issues.apache.org/jira/browse/HBASE-17534 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: 1.4.0, 1.2.6, 1.3.2, 1.1.11 > > Attachments: HBASE-17534.001.branch-1.patch, > HBASE-17534.002.branch-1.patch, HBASE-17534.003.branch-1.patch, > HBASE-17534.004.branch-1.patch, HBASE-17534.005.branch-1.patch, > HBASE-17534.006.branch-1.patch > > > While writing some tests against 1.x, I noticed that what should have been a > DoNotRetryIOException sent to the client from a RegionServer was getting > retried until it reached the hbase client retries limit. > Upon inspection, I found that the SecureBulkLoadClient was wrapping all > Exceptions from the RPC as an IOException. I believe this is creating a case > where the RPC system doesn't notice that there's a DNRIOException wrapped > beneath it, thinking it's a transient error. > This results in clients having to wait for the retry limit to be reached > before they get acknowledgement that something failed. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997352#comment-15997352
]
Sean Busbey commented on HBASE-17786:
-
{code}
+ public static final int SERVER_PORT = 61000;
...
+ ServerName sn = ServerName.valueOf("srv" + i, SERVER_PORT, i);
{code}
Why this variable rather than just using
{{HConstants.DEFAULT_REGIONSERVER_PORT}}?
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17786.001.patch
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17534) SecureBulkLoadClient squashes DoNotRetryIOExceptions from the server
[ https://issues.apache.org/jira/browse/HBASE-17534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997351#comment-15997351 ] Hudson commented on HBASE-17534: SUCCESS: Integrated in Jenkins build HBase-1.3-JDK8 #171 (See [https://builds.apache.org/job/HBase-1.3-JDK8/171/]) HBASE-17534 Avoid re-wrapping IOExceptions as IOExceptions (elserj: rev 2ddb30c570ce239c4fb81213d468452709b14d78) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/coprocessor/SecureBulkLoadClient.java * (add) hbase-client/src/test/java/org/apache/hadoop/hbase/client/coprocessor/TestSecureBulkLoadClient.java > SecureBulkLoadClient squashes DoNotRetryIOExceptions from the server > > > Key: HBASE-17534 > URL: https://issues.apache.org/jira/browse/HBASE-17534 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: 1.4.0, 1.2.6, 1.3.2, 1.1.11 > > Attachments: HBASE-17534.001.branch-1.patch, > HBASE-17534.002.branch-1.patch, HBASE-17534.003.branch-1.patch, > HBASE-17534.004.branch-1.patch, HBASE-17534.005.branch-1.patch, > HBASE-17534.006.branch-1.patch > > > While writing some tests against 1.x, I noticed that what should have been a > DoNotRetryIOException sent to the client from a RegionServer was getting > retried until it reached the hbase client retries limit. > Upon inspection, I found that the SecureBulkLoadClient was wrapping all > Exceptions from the RPC as an IOException. I believe this is creating a case > where the RPC system doesn't notice that there's a DNRIOException wrapped > beneath it, thinking it's a transient error. > This results in clients having to wait for the retry limit to be reached > before they get acknowledgement that something failed. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997346#comment-15997346
]
Sean Busbey commented on HBASE-17786:
-
{code}
+// Do round-robin assignment
{code}
Comments like this would be good to have as output while running as DEBUG or
INFO messages (perhaps on stderr if we can't use a logger for some reason).
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17786.001.patch
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997345#comment-15997345
]
Sean Busbey commented on HBASE-17786:
-
What's the rationale for using System.out directly instead of a logger?
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17786.001.patch
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17534) SecureBulkLoadClient squashes DoNotRetryIOExceptions from the server
[ https://issues.apache.org/jira/browse/HBASE-17534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997342#comment-15997342 ] Hudson commented on HBASE-17534: SUCCESS: Integrated in Jenkins build HBase-1.2-JDK7 #131 (See [https://builds.apache.org/job/HBase-1.2-JDK7/131/]) HBASE-17534 Avoid re-wrapping IOExceptions as IOExceptions (elserj: rev afb67a8d9daf9dacce580067e6dc96c8d8956a0e) * (add) hbase-client/src/test/java/org/apache/hadoop/hbase/client/coprocessor/TestSecureBulkLoadClient.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/coprocessor/SecureBulkLoadClient.java > SecureBulkLoadClient squashes DoNotRetryIOExceptions from the server > > > Key: HBASE-17534 > URL: https://issues.apache.org/jira/browse/HBASE-17534 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: 1.4.0, 1.2.6, 1.3.2, 1.1.11 > > Attachments: HBASE-17534.001.branch-1.patch, > HBASE-17534.002.branch-1.patch, HBASE-17534.003.branch-1.patch, > HBASE-17534.004.branch-1.patch, HBASE-17534.005.branch-1.patch, > HBASE-17534.006.branch-1.patch > > > While writing some tests against 1.x, I noticed that what should have been a > DoNotRetryIOException sent to the client from a RegionServer was getting > retried until it reached the hbase client retries limit. > Upon inspection, I found that the SecureBulkLoadClient was wrapping all > Exceptions from the RPC as an IOException. I believe this is creating a case > where the RPC system doesn't notice that there's a DNRIOException wrapped > beneath it, thinking it's a transient error. > This results in clients having to wait for the retry limit to be reached > before they get acknowledgement that something failed. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17995) improve log messages during snapshot related tests
[
https://issues.apache.org/jira/browse/HBASE-17995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997331#comment-15997331
]
Hadoop QA commented on HBASE-17995:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m 5s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
8s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 50s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 117m 48s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 166m 40s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.util.TestHBaseFsckTwoRS |
| Timed out junit tests | org.apache.hadoop.hbase.util.TestHBaseFsckTwoRS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12866433/HBASE-17995.0.patch |
| JIRA Issue | HBASE-17995 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux dcf4e894c891 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 0fb1fd1 |
| Default Java | 1.8.0_131 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/6699/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/6699/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6699/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6699/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> improve log messages during snapshot related tests
>
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997333#comment-15997333
]
Sean Busbey commented on HBASE-17786:
-
{code}
+ public static final int DEFAULT_NUM_REGIONS = 100;
{code}
nit: all these defaults can start as private since we don't need to reference
them outside of the class.
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17786.001.patch
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997327#comment-15997327
]
Sean Busbey commented on HBASE-17786:
-
{quote}
bq. Why this change to make it mutable?
I think all field declaration in the body of an interface are implicitly
public, static, and final.
{quote}
Ah, good point. Missed that we were in an interface. So this is general cleanup
while you're in the area? I noticed some unused import changes as well. Usually
we try to keep cleanup and functionality changes in different commits so that
it's easier to review the two separately.
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17786.001.patch
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17534) SecureBulkLoadClient squashes DoNotRetryIOExceptions from the server
[ https://issues.apache.org/jira/browse/HBASE-17534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997326#comment-15997326 ] Hudson commented on HBASE-17534: SUCCESS: Integrated in Jenkins build HBase-1.2-JDK8 #127 (See [https://builds.apache.org/job/HBase-1.2-JDK8/127/]) HBASE-17534 Avoid re-wrapping IOExceptions as IOExceptions (elserj: rev afb67a8d9daf9dacce580067e6dc96c8d8956a0e) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/coprocessor/SecureBulkLoadClient.java * (add) hbase-client/src/test/java/org/apache/hadoop/hbase/client/coprocessor/TestSecureBulkLoadClient.java > SecureBulkLoadClient squashes DoNotRetryIOExceptions from the server > > > Key: HBASE-17534 > URL: https://issues.apache.org/jira/browse/HBASE-17534 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: 1.4.0, 1.2.6, 1.3.2, 1.1.11 > > Attachments: HBASE-17534.001.branch-1.patch, > HBASE-17534.002.branch-1.patch, HBASE-17534.003.branch-1.patch, > HBASE-17534.004.branch-1.patch, HBASE-17534.005.branch-1.patch, > HBASE-17534.006.branch-1.patch > > > While writing some tests against 1.x, I noticed that what should have been a > DoNotRetryIOException sent to the client from a RegionServer was getting > retried until it reached the hbase client retries limit. > Upon inspection, I found that the SecureBulkLoadClient was wrapping all > Exceptions from the RPC as an IOException. I believe this is creating a case > where the RPC system doesn't notice that there's a DNRIOException wrapped > beneath it, thinking it's a transient error. > This results in clients having to wait for the retry limit to be reached > before they get acknowledgement that something failed. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17534) SecureBulkLoadClient squashes DoNotRetryIOExceptions from the server
[ https://issues.apache.org/jira/browse/HBASE-17534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997323#comment-15997323 ] Hudson commented on HBASE-17534: SUCCESS: Integrated in Jenkins build HBase-1.1-JDK7 #1867 (See [https://builds.apache.org/job/HBase-1.1-JDK7/1867/]) HBASE-17534 Avoid re-wrapping IOExceptions as IOExceptions (elserj: rev 5de7799e015f2190fdfb6ecf64637706af5c76f7) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/coprocessor/SecureBulkLoadClient.java * (add) hbase-client/src/test/java/org/apache/hadoop/hbase/client/coprocessor/TestSecureBulkLoadClient.java > SecureBulkLoadClient squashes DoNotRetryIOExceptions from the server > > > Key: HBASE-17534 > URL: https://issues.apache.org/jira/browse/HBASE-17534 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: 1.4.0, 1.2.6, 1.3.2, 1.1.11 > > Attachments: HBASE-17534.001.branch-1.patch, > HBASE-17534.002.branch-1.patch, HBASE-17534.003.branch-1.patch, > HBASE-17534.004.branch-1.patch, HBASE-17534.005.branch-1.patch, > HBASE-17534.006.branch-1.patch > > > While writing some tests against 1.x, I noticed that what should have been a > DoNotRetryIOException sent to the client from a RegionServer was getting > retried until it reached the hbase client retries limit. > Upon inspection, I found that the SecureBulkLoadClient was wrapping all > Exceptions from the RPC as an IOException. I believe this is creating a case > where the RPC system doesn't notice that there's a DNRIOException wrapped > beneath it, thinking it's a transient error. > This results in clients having to wait for the retry limit to be reached > before they get acknowledgement that something failed. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17534) SecureBulkLoadClient squashes DoNotRetryIOExceptions from the server
[ https://issues.apache.org/jira/browse/HBASE-17534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997324#comment-15997324 ] Hudson commented on HBASE-17534: FAILURE: Integrated in Jenkins build HBase-1.4 #722 (See [https://builds.apache.org/job/HBase-1.4/722/]) HBASE-17534 Avoid re-wrapping IOExceptions as IOExceptions (elserj: rev 737b489eddac94bf14390aa410661136811ea181) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/coprocessor/SecureBulkLoadClient.java * (add) hbase-client/src/test/java/org/apache/hadoop/hbase/client/coprocessor/TestSecureBulkLoadClient.java > SecureBulkLoadClient squashes DoNotRetryIOExceptions from the server > > > Key: HBASE-17534 > URL: https://issues.apache.org/jira/browse/HBASE-17534 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: 1.4.0, 1.2.6, 1.3.2, 1.1.11 > > Attachments: HBASE-17534.001.branch-1.patch, > HBASE-17534.002.branch-1.patch, HBASE-17534.003.branch-1.patch, > HBASE-17534.004.branch-1.patch, HBASE-17534.005.branch-1.patch, > HBASE-17534.006.branch-1.patch > > > While writing some tests against 1.x, I noticed that what should have been a > DoNotRetryIOException sent to the client from a RegionServer was getting > retried until it reached the hbase client retries limit. > Upon inspection, I found that the SecureBulkLoadClient was wrapping all > Exceptions from the RPC as an IOException. I believe this is creating a case > where the RPC system doesn't notice that there's a DNRIOException wrapped > beneath it, thinking it's a transient error. > This results in clients having to wait for the retry limit to be reached > before they get acknowledgement that something failed. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997322#comment-15997322
]
Sean Busbey commented on HBASE-17786:
-
{code}
+/**
+ * Tool to test performance of different {@link
org.apache.hadoop.hbase.master.LoadBalancer}
+ * implementations.
+ */
+public class LoadBalancerPerformanceEvaluation extends AbstractHBaseTool {
+
{code}
We've been inconsistent on wether non-Test classes in the test packages need
{{InterfaceAudience}} annotations. Since this is a tool, best to be on the safe
side and label it IA.LimitedPrivate(TOOL).
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17786.001.patch
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997316#comment-15997316
]
Sean Busbey commented on HBASE-17786:
-
{code}
index
..afb67198c3d131bfc32d3a0581a78070f4f82e7c
--- /dev/null
+++
b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/LoadBalancerPerformanceEvaluation.java
@@ -0,0 +1,169 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
{code}
nit: the file header should use "/\*" instead of "/\*\*" because it's not meant
to be a javadoc parsed comment. I think we make this mistake in lots of places
already, so not a big deal if you leave it.
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17786.001.patch
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997318#comment-15997318
]
Umesh Agashe commented on HBASE-17786:
--
bq. Why this change to make it mutable?
I think all field declaration in the body of an interface are implicitly
public, static, and final.
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17786.001.patch
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997306#comment-15997306
]
Sean Busbey commented on HBASE-17786:
-
{code}
// Used to signal to the caller that the region(s) cannot be assigned
- static final ServerName BOGUS_SERVER_NAME =
ServerName.valueOf("bogus.example.com,1,1");
+ ServerName BOGUS_SERVER_NAME = ServerName.valueOf("bogus.example.com,1,1");
{code}
Why this change to make it mutable?
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17786.001.patch
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Umesh Agashe updated HBASE-17786:
-
Status: Patch Available (was: In Progress)
Added perf-test tool for benchmarking different LoadBalancer implementations.
Tested from command line.
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17786.001.patch
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Umesh Agashe updated HBASE-17786:
-
Attachment: HBASE-17786.001.patch
Perf-test to tool for LoadBalancer algorithms.
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-17786.001.patch
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17534) SecureBulkLoadClient squashes DoNotRetryIOExceptions from the server
[ https://issues.apache.org/jira/browse/HBASE-17534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997288#comment-15997288 ] Hudson commented on HBASE-17534: SUCCESS: Integrated in Jenkins build HBase-1.1-JDK8 #1950 (See [https://builds.apache.org/job/HBase-1.1-JDK8/1950/]) HBASE-17534 Avoid re-wrapping IOExceptions as IOExceptions (elserj: rev 5de7799e015f2190fdfb6ecf64637706af5c76f7) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/coprocessor/SecureBulkLoadClient.java * (add) hbase-client/src/test/java/org/apache/hadoop/hbase/client/coprocessor/TestSecureBulkLoadClient.java > SecureBulkLoadClient squashes DoNotRetryIOExceptions from the server > > > Key: HBASE-17534 > URL: https://issues.apache.org/jira/browse/HBASE-17534 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: 1.4.0, 1.2.6, 1.3.2, 1.1.11 > > Attachments: HBASE-17534.001.branch-1.patch, > HBASE-17534.002.branch-1.patch, HBASE-17534.003.branch-1.patch, > HBASE-17534.004.branch-1.patch, HBASE-17534.005.branch-1.patch, > HBASE-17534.006.branch-1.patch > > > While writing some tests against 1.x, I noticed that what should have been a > DoNotRetryIOException sent to the client from a RegionServer was getting > retried until it reached the hbase client retries limit. > Upon inspection, I found that the SecureBulkLoadClient was wrapping all > Exceptions from the RPC as an IOException. I believe this is creating a case > where the RPC system doesn't notice that there's a DNRIOException wrapped > beneath it, thinking it's a transient error. > This results in clients having to wait for the retry limit to be reached > before they get acknowledgement that something failed. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17131) Avoid livelock caused by HRegion#processRowsWithLocks
[
https://issues.apache.org/jira/browse/HBASE-17131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997290#comment-15997290
]
Esteban Gutierrez commented on HBASE-17131:
---
[~chia7712] anything interesting about that issue, was that in our code base?
Thanks!
> Avoid livelock caused by HRegion#processRowsWithLocks
> -
>
> Key: HBASE-17131
> URL: https://issues.apache.org/jira/browse/HBASE-17131
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.4.0, 1.3.1
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17131.branch-1.v0.patch, HBASE-17131.v0.patch
>
>
> {code:title=HRegion.java|borderStyle=solid}
> try {
> // STEP 2. Acquire the row lock(s)
> acquiredRowLocks = new ArrayList(rowsToLock.size());
> for (byte[] row : rowsToLock) {
> // Attempt to lock all involved rows, throw if any lock times out
> // use a writer lock for mixed reads and writes
> acquiredRowLocks.add(getRowLockInternal(row, false));
> }
> // STEP 3. Region lock
> lock(this.updatesLock.readLock(), acquiredRowLocks.size() == 0 ? 1 :
> acquiredRowLocks.size());
> locked = true;
> boolean success = false;
> long now = EnvironmentEdgeManager.currentTime();
> try {
> {code}
> We should lock all involved rows in the second try-finally. Otherwise, we
> won’t release the previous locks if any subsequent lock times out.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17131) Avoid livelock caused by HRegion#processRowsWithLocks
[
https://issues.apache.org/jira/browse/HBASE-17131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997283#comment-15997283
]
Chia-Ping Tsai commented on HBASE-17131:
code reading. I investigated the transaction over hbase and then found this
issue. Does the patch cause any problem? If yes, I will fix it asap.
> Avoid livelock caused by HRegion#processRowsWithLocks
> -
>
> Key: HBASE-17131
> URL: https://issues.apache.org/jira/browse/HBASE-17131
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.4.0, 1.3.1
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17131.branch-1.v0.patch, HBASE-17131.v0.patch
>
>
> {code:title=HRegion.java|borderStyle=solid}
> try {
> // STEP 2. Acquire the row lock(s)
> acquiredRowLocks = new ArrayList(rowsToLock.size());
> for (byte[] row : rowsToLock) {
> // Attempt to lock all involved rows, throw if any lock times out
> // use a writer lock for mixed reads and writes
> acquiredRowLocks.add(getRowLockInternal(row, false));
> }
> // STEP 3. Region lock
> lock(this.updatesLock.readLock(), acquiredRowLocks.size() == 0 ? 1 :
> acquiredRowLocks.size());
> locked = true;
> boolean success = false;
> long now = EnvironmentEdgeManager.currentTime();
> try {
> {code}
> We should lock all involved rows in the second try-finally. Otherwise, we
> won’t release the previous locks if any subsequent lock times out.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17131) Avoid livelock caused by HRegion#processRowsWithLocks
[
https://issues.apache.org/jira/browse/HBASE-17131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997229#comment-15997229
]
stack commented on HBASE-17131:
---
You saw this issue in production or this code reading [~chia7712]?
> Avoid livelock caused by HRegion#processRowsWithLocks
> -
>
> Key: HBASE-17131
> URL: https://issues.apache.org/jira/browse/HBASE-17131
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.4.0, 1.3.1
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17131.branch-1.v0.patch, HBASE-17131.v0.patch
>
>
> {code:title=HRegion.java|borderStyle=solid}
> try {
> // STEP 2. Acquire the row lock(s)
> acquiredRowLocks = new ArrayList(rowsToLock.size());
> for (byte[] row : rowsToLock) {
> // Attempt to lock all involved rows, throw if any lock times out
> // use a writer lock for mixed reads and writes
> acquiredRowLocks.add(getRowLockInternal(row, false));
> }
> // STEP 3. Region lock
> lock(this.updatesLock.readLock(), acquiredRowLocks.size() == 0 ? 1 :
> acquiredRowLocks.size());
> locked = true;
> boolean success = false;
> long now = EnvironmentEdgeManager.currentTime();
> try {
> {code}
> We should lock all involved rows in the second try-finally. Otherwise, we
> won’t release the previous locks if any subsequent lock times out.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (HBASE-17997) jruby-complete-1.6.8.jar is in cached_classpath.txt
Ted Yu created HBASE-17997:
--
Summary: jruby-complete-1.6.8.jar is in cached_classpath.txt
Key: HBASE-17997
URL: https://issues.apache.org/jira/browse/HBASE-17997
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
HBASE-15199 moves jruby-complete-1.6.8.jar to lib/ruby directory.
However, jruby-complete-1.6.8.jar still appears in cached_classpath.txt
This means that user would see exception similar to the following when starting
hbase in standalone mode with s3a as rootdir :
{code}
2017-05-04 16:41:32,854 WARN
[RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=38659]
internal.S3MetadataResponseHandler: Unable to parse last modified date: Thu, 04
May 2017 16:27:09 GMT
java.lang.IllegalStateException: Joda-time 2.2 or later version is required,
but found version: null
at com.amazonaws.util.DateUtils.handleException(DateUtils.java:149)
at com.amazonaws.util.DateUtils.parseRFC822Date(DateUtils.java:195)
at
com.amazonaws.services.s3.internal.ServiceUtils.parseRfc822Date(ServiceUtils.java:78)
at
com.amazonaws.services.s3.internal.AbstractS3ResponseHandler.populateObjectMetadata(AbstractS3ResponseHandler.java:115)
at
com.amazonaws.services.s3.internal.S3ObjectResponseHandler.handle(S3ObjectResponseHandler.java:52)
at
com.amazonaws.services.s3.internal.S3ObjectResponseHandler.handle(S3ObjectResponseHandler.java:30)
at
com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:1072)
at
com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:746)
at
com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)
at
com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1191)
at org.apache.hadoop.fs.s3a.S3AInputStream.reopen(S3AInputStream.java:148)
at org.apache.hadoop.fs.s3a.S3AInputStream.lazySeek(S3AInputStream.java:281)
at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:364)
at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:75)
at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:92)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock.positionalReadWithExtra(HFileBlock.java:722)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1420)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1677)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1504)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:439)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:904)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:267)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:169)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:363)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:217)
at org.apache.hadoop.hbase.regionserver.HStore.createScanner(HStore.java:2132)
at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2122)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:5687)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2679)
at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2665)
at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2647)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6906)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6885)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2007)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17977) Enabled Master observer to delete quotas on table deletion by default
[ https://issues.apache.org/jira/browse/HBASE-17977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997211#comment-15997211 ] Josh Elser commented on HBASE-17977: The above test failures either pass locally or also fail on `master`. Ted/Andrew, if you have a moment to re-check, please. > Enabled Master observer to delete quotas on table deletion by default > - > > Key: HBASE-17977 > URL: https://issues.apache.org/jira/browse/HBASE-17977 > Project: HBase > Issue Type: Sub-task >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: HBASE-16961 > > Attachments: HBASE-17977.001.HBASE-16961.patch, > HBASE-17977.002.HBASE-16961.patch > > > Deleting a table's quota when we delete the table should be the common case, > not the exception. > Load the observer by default (when quota support is enabled), and switch the > flag to disable loading it. Make sure to update the docs too. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Work started] (HBASE-17786) Create LoadBalancer perf-tests (test balancer algorithm decoupled from workload)
[
https://issues.apache.org/jira/browse/HBASE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HBASE-17786 started by Umesh Agashe.
> Create LoadBalancer perf-tests (test balancer algorithm decoupled from
> workload)
>
>
> Key: HBASE-17786
> URL: https://issues.apache.org/jira/browse/HBASE-17786
> Project: HBase
> Issue Type: Sub-task
> Components: Balancer, proc-v2
>Reporter: stack
>Assignee: Umesh Agashe
> Labels: beginner
> Fix For: 2.0.0
>
>
> (Below is a quote from [~mbertozzi] taken from an internal issue that I'm
> moving out here)
> Add perf tools and keep monitored balancer performance (a BalancerPE-type
> thing).
> Most of the balancers should be instantiable without requiring a
> mini-cluster, and it easy to create tons of RegionInfo and ServerNames with a
> for loop.
> The balancer is just creating a map RegionInfo:ServerName.
> There are two methods to test roundRobinAssignment() and retainAssignment()
> {code}
> Map roundRobinAssignment(
> List regions,
> List servers
> ) throws HBaseIOException;
> Map retainAssignment(
> Map regions,
> List servers
> ) throws HBaseIOException;
> {code}
> There are a bunch of obvious optimization that everyone can see just by
> looking at the code. (like replacing array with set when we do
> contains/remove operations). It will be nice to have a baseline and start
> improving from there.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-16488) Starting namespace and quota services in master startup asynchronizely
[
https://issues.apache.org/jira/browse/HBASE-16488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997200#comment-15997200
]
Josh Elser commented on HBASE-16488:
{noformat}
@@ -2599,11 +2625,26 @@ public class HMaster extends HRegionServer implements
MasterServices, Server {
void checkNamespaceManagerReady() throws IOException {
checkInitialized();
-if (tableNamespaceManager == null ||
-!tableNamespaceManager.isTableAvailableAndInitialized(true)) {
+if (tableNamespaceManager == null) {
throw new IOException("Table Namespace Manager not ready yet, try again
later");
+} else if (!tableNamespaceManager.isTableAvailableAndInitialized(true)) {
+ try {
+// Wait some time.
+long startTime = EnvironmentEdgeManager.currentTime();
+int timeout = conf.getInt("hbase.master.namespace.waitforready",
3);
+while (!tableNamespaceManager.isTableNamespaceManagerStarted() &&
+EnvironmentEdgeManager.currentTime() - startTime < timeout) {
+ Thread.sleep(100);
+}
+ } catch (InterruptedException e) {
+throw (InterruptedIOException) new
InterruptedIOException().initCause(e);
+ }
+ if (!tableNamespaceManager.isTableNamespaceManagerStarted()) {
+throw new IOException("Table Namespace Manager not fully initialized,
try again later");
+ }
}
}
{noformat}
This sits a little funny with me. Ideally, we'd have the caller do the sleeping
so that we're not blocking a thread inside of the Master (or worse an RPC
handler). Your change here is definitely easier to implement, but I wonder how
hard it would be to leave the exception throw and implement retry logic in the
callers (other methods in HMaster or hbase client).
Unrelated: shouldn't {{tableNamespaceManager}} be volatile if we're checking it
across different threads? Or, make it final and use an {{AtomicReference}}?
{noformat}
@@ -93,7 +94,7 @@ public class TableNamespaceManager {
long startTime = EnvironmentEdgeManager.currentTime();
int timeout = conf.getInt(NS_INIT_TIMEOUT, DEFAULT_NS_INIT_TIMEOUT);
while (!isTableAvailableAndInitialized(false)) {
-if (EnvironmentEdgeManager.currentTime() - startTime + 100 > timeout) {
+if (EnvironmentEdgeManager.currentTime() - startTime > timeout) {
// We can't do anything if ns is not online.
throw new IOException("Timedout " + timeout + "ms waiting for
namespace table to " +
"be assigned");
{noformat}
Do you know of the reason we were previously augmenting this "runtime" by 100ms?
{noformat}
diff --git
hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
index f60be66..c75d4bc 100644
--- hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
+++ hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
@@ -105,6 +105,7 @@ import org.apache.hadoop.hbase.security.HBaseKerberosUtils;
import org.apache.hadoop.hbase.security.User;
import org.apache.hadoop.hbase.security.visibility.VisibilityLabelsCache;
import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
import org.apache.hadoop.hbase.util.FSTableDescriptors;
import org.apache.hadoop.hbase.util.FSUtils;
import org.apache.hadoop.hbase.util.JVMClusterUtil;
@@ -1459,6 +1460,7 @@ public class HBaseTestingUtility extends
HBaseCommonTestingUtility {
.setMaxVersions(numVersions);
desc.addFamily(hcd);
}
+waitUntilTableNamespaceManagerStarted();
getHBaseAdmin().createTable(desc, startKey, endKey, numRegions);
// HBaseAdmin only waits for regions to appear in hbase:meta we should
wait until they are assigned
waitUntilAllRegionsAssigned(tableName);
@@ -1497,6 +1499,7 @@ public class HBaseTestingUtility extends
HBaseCommonTestingUtility {
hcd.setBloomFilterType(BloomType.NONE);
htd.addFamily(hcd);
}
+waitUntilTableNamespaceManagerStarted();
getHBaseAdmin().createTable(htd, splitKeys);
// HBaseAdmin only waits for regions to appear in hbase:meta we should
wait until they are
// assigned
{noformat}
Do this once in {{MiniHBaseCluster startMiniHBaseCluster(int, int, Class,
Class, boolean, boolean)}} instead of having it littered across
HBaseTestingUtility?
Nice test additions!
> Starting namespace and quota services in master startup asynchronizely
> --
>
> Key: HBASE-16488
> URL: https://issues.apache.org/jira/browse/HBASE-16488
> Project: HBase
> Issue Type: Improvement
> Components: master
>Affects Versions: 2.0.0, 1.3.0, 1.0.3, 1.4.0, 1.1.5, 1.2.2
>Reporter: Stephen Yuan Jiang
>
[jira] [Commented] (HBASE-16466) HBase snapshots support in VerifyReplication tool to reduce load on live HBase cluster with large tables
[ https://issues.apache.org/jira/browse/HBASE-16466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997184#comment-15997184 ] Maddineni Sukumar commented on HBASE-16466: --- Thanks [~apurtell] , Will submit patch for 1.3. I tried and its failing with same error. Looks like I need to change unit test for 1.3 . > HBase snapshots support in VerifyReplication tool to reduce load on live > HBase cluster with large tables > > > Key: HBASE-16466 > URL: https://issues.apache.org/jira/browse/HBASE-16466 > Project: HBase > Issue Type: Improvement > Components: hbase >Affects Versions: 0.98.21 >Reporter: Sukumar Maddineni >Assignee: Maddineni Sukumar > Fix For: 2.0.0 > > Attachments: HBASE-16466.branch-1.3.001.patch, HBASE-16466.v1.patch, > HBASE-16466.v2.patch, HBASE-16466.v3.patch, HBASE-16466.v4.patch, > HBASE-16466.v5.patch > > > As of now VerifyReplicatin tool is running using normal HBase scanners. If > you want to run VerifyReplication multiple times on a production live > cluster with large tables then it creates extra load on HBase layer. So if we > implement snapshot based support then both in source and target we can read > data from snapshots which reduces load on HBase -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17667) Implement async flush/compact region methods
[
https://issues.apache.org/jira/browse/HBASE-17667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997168#comment-15997168
]
Hadoop QA commented on HBASE-17667:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 28s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 22s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
56s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 8s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
32s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
27s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 36s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 23s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
60m 50s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 28s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 120m 43s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
58s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 222m 27s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestRegionReplicaFailover |
| | hadoop.hbase.regionserver.TestEncryptionKeyRotation |
| | hadoop.hbase.regionserver.TestCorruptedRegionStoreFile |
| Timed out junit tests |
org.apache.hadoop.hbase.replication.regionserver.TestWALEntryStream |
| | org.apache.hadoop.hbase.replication.TestReplicationSmallTests |
| | org.apache.hadoop.hbase.TestIOFencing |
| | org.apache.hadoop.hbase.filter.TestFuzzyRowFilterEndToEnd |
| | org.apache.hadoop.hbase.io.encoding.TestBufferedDataBlockEncoder |
| | org.apache.hadoop.hbase.io.encoding.TestEncodedSeekers |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12866417/HBASE-17667.v3.patch |
| JIRA Issue | HBASE-17667 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 5559a0b532c9 4.8.3-std-1 #1 SMP Fri Oct 21 11:15:43 UTC 2016
x86_64 x86_64 x86_64
[jira] [Commented] (HBASE-16436) Add the CellChunkMap variant
[
https://issues.apache.org/jira/browse/HBASE-16436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997136#comment-15997136
]
Hadoop QA commented on HBASE-16436:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 25s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 16s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
29s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
61m 31s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 113m 25s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
51s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 202m 37s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestPerColumnFamilyFlush |
| Timed out junit tests |
org.apache.hadoop.hbase.replication.regionserver.TestWALEntryStream |
| | org.apache.hadoop.hbase.replication.TestReplicationSmallTests |
| | org.apache.hadoop.hbase.quotas.TestQuotaThrottle |
| | org.apache.hadoop.hbase.filter.TestFuzzyRowFilterEndToEnd |
| | org.apache.hadoop.hbase.io.TestFileLink |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12866412/HBASE-16436-V10.patch
|
| JIRA Issue | HBASE-16436 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 42851975256b 4.8.3-std-1 #1 SMP Fri Oct 21 11:15:43 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 2de6b05 |
| Default Java | 1.8.0_131 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/6697/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/6697/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6697/testReport/ |
| modules | C: hbase-server U: hbase-server |
|
[jira] [Commented] (HBASE-17991) Add more details about compaction queue on /dump
[
https://issues.apache.org/jira/browse/HBASE-17991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997125#comment-15997125
]
Hudson commented on HBASE-17991:
FAILURE: Integrated in Jenkins build HBase-1.4 #721 (See
[https://builds.apache.org/job/HBase-1.4/721/])
HBASE-17991 Add more details about compaction queue on /dump (Guangxu (tedyu:
rev dd321c4a356a917901025bd916abdf60a83d1560)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java
> Add more details about compaction queue on /dump
>
>
> Key: HBASE-17991
> URL: https://issues.apache.org/jira/browse/HBASE-17991
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17991-master-v1.patch,
> HBASE-17991-master-v2.patch, HBASE-17991-master-v3.patch
>
>
> RS dump information as follows
> {code}
> RS Queue:
> ===
> Compaction/Split Queue summary: compaction_queue=(20:0), split_queue=0,
> merge_queue=0
> Compaction/Split Queue dump:
> LargeCompation Queue:
> Request =
> regionName=usertable,user4180497275766179957,1491904095205.1d4085e4438752f3611f66e7b043fe44.,
> storeName=0, fileCount=1, fileSize=1.9 G (1.9 G), priority=1,
> time=21697920409804647
> Request =
> regionName=usertable,user4568009753557153251,1491904099262.95bf004e3c9b35a58c60ca5d5b11d190.,
> storeName=0, fileCount=1, fileSize=1.9 G (1.9 G), priority=1,
> time=21697920413223800
> SmallCompation Queue:
> Store = b, pri = 108
> Store = b, pri = 108
> Store = b, pri = 108
> Store = b, pri = 108
> Store = b, pri = 108
> Store = b, pri = 109
> {code}
> Compaction queue information will be displayed on page /dump.
> If compation has selected the file, it will print the details information of
> the compation, otherwise only print storename and priority(Store = b, pri =
> 108) which is useless for us.
> So, we should also print more detailed information, such as regionName,
> starttime etc.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17534) SecureBulkLoadClient squashes DoNotRetryIOExceptions from the server
[ https://issues.apache.org/jira/browse/HBASE-17534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997124#comment-15997124 ] Hudson commented on HBASE-17534: SUCCESS: Integrated in Jenkins build HBase-1.2-IT #866 (See [https://builds.apache.org/job/HBase-1.2-IT/866/]) HBASE-17534 Avoid re-wrapping IOExceptions as IOExceptions (elserj: rev afb67a8d9daf9dacce580067e6dc96c8d8956a0e) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/coprocessor/SecureBulkLoadClient.java * (add) hbase-client/src/test/java/org/apache/hadoop/hbase/client/coprocessor/TestSecureBulkLoadClient.java > SecureBulkLoadClient squashes DoNotRetryIOExceptions from the server > > > Key: HBASE-17534 > URL: https://issues.apache.org/jira/browse/HBASE-17534 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: 1.4.0, 1.2.6, 1.3.2, 1.1.11 > > Attachments: HBASE-17534.001.branch-1.patch, > HBASE-17534.002.branch-1.patch, HBASE-17534.003.branch-1.patch, > HBASE-17534.004.branch-1.patch, HBASE-17534.005.branch-1.patch, > HBASE-17534.006.branch-1.patch > > > While writing some tests against 1.x, I noticed that what should have been a > DoNotRetryIOException sent to the client from a RegionServer was getting > retried until it reached the hbase client retries limit. > Upon inspection, I found that the SecureBulkLoadClient was wrapping all > Exceptions from the RPC as an IOException. I believe this is creating a case > where the RPC system doesn't notice that there's a DNRIOException wrapped > beneath it, thinking it's a transient error. > This results in clients having to wait for the retry limit to be reached > before they get acknowledgement that something failed. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17996) HBase master fails to start sometimes on RHEL7
[
https://issues.apache.org/jira/browse/HBASE-17996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Knupp updated HBASE-17996:
Attachment:
hbase-jenkins-master-impala-boost-static-burst-slave-el7-11ef.vpc.cloudera.com.out
hbase-jenkins-master-impala-boost-static-burst-slave-el7-02f4.vpc.cloudera.com.out
Attaching logs from two different failures.
> HBase master fails to start sometimes on RHEL7
> --
>
> Key: HBASE-17996
> URL: https://issues.apache.org/jira/browse/HBASE-17996
> Project: HBase
> Issue Type: Bug
> Components: master
>Reporter: David Knupp
> Attachments:
> hbase-jenkins-master-impala-boost-static-burst-slave-el7-02f4.vpc.cloudera.com.out,
>
> hbase-jenkins-master-impala-boost-static-burst-slave-el7-11ef.vpc.cloudera.com.out
>
>
> Impala includes HBase in its local test environment, and we have found that
> intermittently, the HBase master node fails to start when we are testing on
> RHEL7.
> In these failures, what we typically see in the logs is this:
> {noformat}
> 17/04/29 21:33:47 INFO zookeeper.ClientCnxn: Session establishment complete
> on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x15bbd21b797000a,
> negotiated timeout = 9
> 17/04/29 21:33:47 INFO client.ZooKeeperRegistry: ClusterId read in ZooKeeper
> is null
> 17/04/29 21:33:48 INFO master.ActiveMasterManager: Deleting ZNode for
> /hbase/backup-masters/localhost,16000,1493526758211 from backup master
> directory
> {noformat}
> On a successful startup, the log looks like this:
> {noformat}
> 17/04/16 21:32:29 INFO zookeeper.ClientCnxn: Session establishment complete
> on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x15b7a2ed6860005,
> negotiated timeout = 9
> 17/04/16 21:32:29 INFO client.ZooKeeperRegistry: ClusterId read in ZooKeeper
> is null
> 17/04/16 21:32:30 INFO util.FSUtils: Created version file at
> hdfs://localhost:20500/hbase with version=8
> 17/04/16 21:32:31 INFO master.MasterFileSystem: BOOTSTRAP: creating
> hbase:meta region
> {noformat}
> So the event that we don't see in the failed start up attempts is
> {{master.MasterFileSystem: BOOTSTRAP: creating hbase:meta region}}.
> The full logs will be attached.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17996) HBase master fails to start sometimes on RHEL7
[
https://issues.apache.org/jira/browse/HBASE-17996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Knupp updated HBASE-17996:
Description:
Impala includes HBase in its local test environment, and we have found that
intermittently, the HBase master node fails to start when we are testing on
RHEL7.
In these failures, what we typically see in the logs is this:
{noformat}
17/04/29 21:33:47 INFO zookeeper.ClientCnxn: Session establishment complete on
server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x15bbd21b797000a,
negotiated timeout = 9
17/04/29 21:33:47 INFO client.ZooKeeperRegistry: ClusterId read in ZooKeeper is
null
17/04/29 21:33:48 INFO master.ActiveMasterManager: Deleting ZNode for
/hbase/backup-masters/localhost,16000,1493526758211 from backup master directory
{noformat}
On a successful startup, the log looks like this:
{noformat}
17/04/16 21:32:29 INFO zookeeper.ClientCnxn: Session establishment complete on
server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x15b7a2ed6860005,
negotiated timeout = 9
17/04/16 21:32:29 INFO client.ZooKeeperRegistry: ClusterId read in ZooKeeper is
null
17/04/16 21:32:30 INFO util.FSUtils: Created version file at
hdfs://localhost:20500/hbase with version=8
17/04/16 21:32:31 INFO master.MasterFileSystem: BOOTSTRAP: creating hbase:meta
region
{noformat}
So the event that we don't see in the failed start up attempts is
{{master.MasterFileSystem: BOOTSTRAP: creating hbase:meta region}}.
The full logs will be attached.
was:
Impala includes HBase in its local test environment, and we have found that
intermittently, the HBase master node fails to start when we are testing on
RHEL7.
In these failures, what we typically see in the logs is this:
{noformat}
17/04/29 21:33:47 INFO zookeeper.ClientCnxn: Session establishment complete on
server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x15bbd21b797000a,
negotiated timeout = 9
17/04/29 21:33:47 INFO client.ZooKeeperRegistry: ClusterId read in ZooKeeper is
null
17/04/29 21:33:48 INFO master.ActiveMasterManager: Deleting ZNode for
/hbase/backup-masters/localhost,16000,1493526758211 from backup master directory
{noformat}
On a successful startup, the log looks like this:
{noformat}
17/04/16 21:32:29 INFO zookeeper.ClientCnxn: Session establishment complete on
server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x15b7a2ed6860005,
negotiated timeout = 9
17/04/16 21:32:29 INFO client.ZooKeeperRegistry: ClusterId read in ZooKeeper is
null
17/04/16 21:32:30 INFO util.FSUtils: Created version file at
hdfs://localhost:20500/hbase with version=8
17/04/16 21:32:31 INFO master.MasterFileSystem: BOOTSTRAP: creating hbase:meta
region
{noformat}
So the event that seems to be lacking is {{master.MasterFileSystem: BOOTSTRAP:
creating hbase:meta region}}.
The full logs will be attached.
> HBase master fails to start sometimes on RHEL7
> --
>
> Key: HBASE-17996
> URL: https://issues.apache.org/jira/browse/HBASE-17996
> Project: HBase
> Issue Type: Bug
> Components: master
>Reporter: David Knupp
>
> Impala includes HBase in its local test environment, and we have found that
> intermittently, the HBase master node fails to start when we are testing on
> RHEL7.
> In these failures, what we typically see in the logs is this:
> {noformat}
> 17/04/29 21:33:47 INFO zookeeper.ClientCnxn: Session establishment complete
> on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x15bbd21b797000a,
> negotiated timeout = 9
> 17/04/29 21:33:47 INFO client.ZooKeeperRegistry: ClusterId read in ZooKeeper
> is null
> 17/04/29 21:33:48 INFO master.ActiveMasterManager: Deleting ZNode for
> /hbase/backup-masters/localhost,16000,1493526758211 from backup master
> directory
> {noformat}
> On a successful startup, the log looks like this:
> {noformat}
> 17/04/16 21:32:29 INFO zookeeper.ClientCnxn: Session establishment complete
> on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x15b7a2ed6860005,
> negotiated timeout = 9
> 17/04/16 21:32:29 INFO client.ZooKeeperRegistry: ClusterId read in ZooKeeper
> is null
> 17/04/16 21:32:30 INFO util.FSUtils: Created version file at
> hdfs://localhost:20500/hbase with version=8
> 17/04/16 21:32:31 INFO master.MasterFileSystem: BOOTSTRAP: creating
> hbase:meta region
> {noformat}
> So the event that we don't see in the failed start up attempts is
> {{master.MasterFileSystem: BOOTSTRAP: creating hbase:meta region}}.
> The full logs will be attached.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (HBASE-17996) HBase master fails to start sometimes on RHEL7
David Knupp created HBASE-17996:
---
Summary: HBase master fails to start sometimes on RHEL7
Key: HBASE-17996
URL: https://issues.apache.org/jira/browse/HBASE-17996
Project: HBase
Issue Type: Bug
Components: master
Reporter: David Knupp
Impala includes HBase in its local test environment, and we have found that
intermittently, the HBase master node fails to start when we are testing on
RHEL7.
In these failures, what we typically see in the logs is this:
{noformat}
17/04/29 21:33:47 INFO zookeeper.ClientCnxn: Session establishment complete on
server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x15bbd21b797000a,
negotiated timeout = 9
17/04/29 21:33:47 INFO client.ZooKeeperRegistry: ClusterId read in ZooKeeper is
null
17/04/29 21:33:48 INFO master.ActiveMasterManager: Deleting ZNode for
/hbase/backup-masters/localhost,16000,1493526758211 from backup master directory
{noformat}
On a successful startup, the log looks like this:
{noformat}
17/04/16 21:32:29 INFO zookeeper.ClientCnxn: Session establishment complete on
server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x15b7a2ed6860005,
negotiated timeout = 9
17/04/16 21:32:29 INFO client.ZooKeeperRegistry: ClusterId read in ZooKeeper is
null
17/04/16 21:32:30 INFO util.FSUtils: Created version file at
hdfs://localhost:20500/hbase with version=8
17/04/16 21:32:31 INFO master.MasterFileSystem: BOOTSTRAP: creating hbase:meta
region
{noformat}
So the event that seems to be lacking is {{master.MasterFileSystem: BOOTSTRAP:
creating hbase:meta region}}.
The full logs will be attached.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17534) SecureBulkLoadClient squashes DoNotRetryIOExceptions from the server
[ https://issues.apache.org/jira/browse/HBASE-17534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997109#comment-15997109 ] Hudson commented on HBASE-17534: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #38 (See [https://builds.apache.org/job/HBase-1.3-IT/38/]) HBASE-17534 Avoid re-wrapping IOExceptions as IOExceptions (elserj: rev 2ddb30c570ce239c4fb81213d468452709b14d78) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/coprocessor/SecureBulkLoadClient.java * (add) hbase-client/src/test/java/org/apache/hadoop/hbase/client/coprocessor/TestSecureBulkLoadClient.java > SecureBulkLoadClient squashes DoNotRetryIOExceptions from the server > > > Key: HBASE-17534 > URL: https://issues.apache.org/jira/browse/HBASE-17534 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: 1.4.0, 1.2.6, 1.3.2, 1.1.11 > > Attachments: HBASE-17534.001.branch-1.patch, > HBASE-17534.002.branch-1.patch, HBASE-17534.003.branch-1.patch, > HBASE-17534.004.branch-1.patch, HBASE-17534.005.branch-1.patch, > HBASE-17534.006.branch-1.patch > > > While writing some tests against 1.x, I noticed that what should have been a > DoNotRetryIOException sent to the client from a RegionServer was getting > retried until it reached the hbase client retries limit. > Upon inspection, I found that the SecureBulkLoadClient was wrapping all > Exceptions from the RPC as an IOException. I believe this is creating a case > where the RPC system doesn't notice that there's a DNRIOException wrapped > beneath it, thinking it's a transient error. > This results in clients having to wait for the retry limit to be reached > before they get acknowledgement that something failed. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17534) SecureBulkLoadClient squashes DoNotRetryIOExceptions from the server
[ https://issues.apache.org/jira/browse/HBASE-17534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-17534: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks again for the review, Sean. For posterity, 2.0 (master) does not need this change due to SecureBulkLoadClient moving out of the coprocessor realm and into the standard client RPC realm. > SecureBulkLoadClient squashes DoNotRetryIOExceptions from the server > > > Key: HBASE-17534 > URL: https://issues.apache.org/jira/browse/HBASE-17534 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: 1.4.0, 1.2.6, 1.3.2, 1.1.11 > > Attachments: HBASE-17534.001.branch-1.patch, > HBASE-17534.002.branch-1.patch, HBASE-17534.003.branch-1.patch, > HBASE-17534.004.branch-1.patch, HBASE-17534.005.branch-1.patch, > HBASE-17534.006.branch-1.patch > > > While writing some tests against 1.x, I noticed that what should have been a > DoNotRetryIOException sent to the client from a RegionServer was getting > retried until it reached the hbase client retries limit. > Upon inspection, I found that the SecureBulkLoadClient was wrapping all > Exceptions from the RPC as an IOException. I believe this is creating a case > where the RPC system doesn't notice that there's a DNRIOException wrapped > beneath it, thinking it's a transient error. > This results in clients having to wait for the retry limit to be reached > before they get acknowledgement that something failed. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17995) improve log messages during snapshot related tests
[ https://issues.apache.org/jira/browse/HBASE-17995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-17995: Status: Patch Available (was: Open) > improve log messages during snapshot related tests > -- > > Key: HBASE-17995 > URL: https://issues.apache.org/jira/browse/HBASE-17995 > Project: HBase > Issue Type: Improvement > Components: integration tests, mapreduce, snapshots, test >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-17995.0.patch > > > while verifying the changes for HBASE-17964 I had to chase down a failure > related to having the wrong hbase configs. Adding some additional logging > detail let me see what was happening. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17995) improve log messages during snapshot related tests
[ https://issues.apache.org/jira/browse/HBASE-17995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-17995: Attachment: HBASE-17995.0.patch -00 - adds log messages detailing steps of some snapshot tests - adds details to failure messages > improve log messages during snapshot related tests > -- > > Key: HBASE-17995 > URL: https://issues.apache.org/jira/browse/HBASE-17995 > Project: HBase > Issue Type: Improvement > Components: integration tests, mapreduce, snapshots, test >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-17995.0.patch > > > while verifying the changes for HBASE-17964 I had to chase down a failure > related to having the wrong hbase configs. Adding some additional logging > detail let me see what was happening. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17534) SecureBulkLoadClient squashes DoNotRetryIOExceptions from the server
[ https://issues.apache.org/jira/browse/HBASE-17534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997043#comment-15997043 ] Josh Elser commented on HBASE-17534: Alright, QA (now that it's happy) appears OK. Findbugs were already there, and TestReplicasClient passes locally (seems to be about call cancellation, nothing to do with bulk loads). Going to commit to branch-1. > SecureBulkLoadClient squashes DoNotRetryIOExceptions from the server > > > Key: HBASE-17534 > URL: https://issues.apache.org/jira/browse/HBASE-17534 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: 1.4.0, 1.2.6, 1.3.2, 1.1.11 > > Attachments: HBASE-17534.001.branch-1.patch, > HBASE-17534.002.branch-1.patch, HBASE-17534.003.branch-1.patch, > HBASE-17534.004.branch-1.patch, HBASE-17534.005.branch-1.patch, > HBASE-17534.006.branch-1.patch > > > While writing some tests against 1.x, I noticed that what should have been a > DoNotRetryIOException sent to the client from a RegionServer was getting > retried until it reached the hbase client retries limit. > Upon inspection, I found that the SecureBulkLoadClient was wrapping all > Exceptions from the RPC as an IOException. I believe this is creating a case > where the RPC system doesn't notice that there's a DNRIOException wrapped > beneath it, thinking it's a transient error. > This results in clients having to wait for the retry limit to be reached > before they get acknowledgement that something failed. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17995) improve log messages during snapshot related tests
Sean Busbey created HBASE-17995: --- Summary: improve log messages during snapshot related tests Key: HBASE-17995 URL: https://issues.apache.org/jira/browse/HBASE-17995 Project: HBase Issue Type: Improvement Components: integration tests, mapreduce, snapshots, test Affects Versions: 2.0.0 Reporter: Sean Busbey Assignee: Sean Busbey Priority: Trivial Fix For: 2.0.0 while verifying the changes for HBASE-17964 I had to chase down a failure related to having the wrong hbase configs. Adding some additional logging detail let me see what was happening. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17964) ensure hbase-metrics-api is included in mapreduce job classpaths
[
https://issues.apache.org/jira/browse/HBASE-17964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-17964:
Resolution: Fixed
Status: Resolved (was: Patch Available)
okay, finally got to verify this change running the snapshot integration test
against a cluster.
I have some logging improvements that let me figure out what I was doing wrong;
I'll make a follow on with those.
> ensure hbase-metrics-api is included in mapreduce job classpaths
>
>
> Key: HBASE-17964
> URL: https://issues.apache.org/jira/browse/HBASE-17964
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-17964.0.patch
>
>
> HBASE-9774 moved metrics stuff into a couple of additional modules.
> currently, MR against snapshots fail because it's unable to find the
> metrics-api classes it needs for ClientSideRegionScanner.
> {code}
> Error: java.io.IOException: java.lang.NoClassDefFoundError:
> org/apache/hadoop/hbase/metrics/Snapshot
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initializeStores(HRegion.java:1027)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:883)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:858)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6694)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6651)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6622)
> at
> org.apache.hadoop.hbase.client.ClientSideRegionScanner.(ClientSideRegionScanner.java:59)
> at
> org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormatImpl$RecordReader.initialize(TableSnapshotInputFormatImpl.java:211)
> at
> org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat$TableSnapshotRegionRecordReader.initialize(TableSnapshotInputFormat.java:140)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:548)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17991) Add more details about compaction queue on /dump
[
https://issues.apache.org/jira/browse/HBASE-17991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-17991:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks for the patch.
> Add more details about compaction queue on /dump
>
>
> Key: HBASE-17991
> URL: https://issues.apache.org/jira/browse/HBASE-17991
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17991-master-v1.patch,
> HBASE-17991-master-v2.patch, HBASE-17991-master-v3.patch
>
>
> RS dump information as follows
> {code}
> RS Queue:
> ===
> Compaction/Split Queue summary: compaction_queue=(20:0), split_queue=0,
> merge_queue=0
> Compaction/Split Queue dump:
> LargeCompation Queue:
> Request =
> regionName=usertable,user4180497275766179957,1491904095205.1d4085e4438752f3611f66e7b043fe44.,
> storeName=0, fileCount=1, fileSize=1.9 G (1.9 G), priority=1,
> time=21697920409804647
> Request =
> regionName=usertable,user4568009753557153251,1491904099262.95bf004e3c9b35a58c60ca5d5b11d190.,
> storeName=0, fileCount=1, fileSize=1.9 G (1.9 G), priority=1,
> time=21697920413223800
> SmallCompation Queue:
> Store = b, pri = 108
> Store = b, pri = 108
> Store = b, pri = 108
> Store = b, pri = 108
> Store = b, pri = 108
> Store = b, pri = 109
> {code}
> Compaction queue information will be displayed on page /dump.
> If compation has selected the file, it will print the details information of
> the compation, otherwise only print storename and priority(Store = b, pri =
> 108) which is useless for us.
> So, we should also print more detailed information, such as regionName,
> starttime etc.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17991) Add more details about compaction queue on /dump
[
https://issues.apache.org/jira/browse/HBASE-17991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-17991:
---
Summary: Add more details about compaction queue on /dump (was: Add more
details about compation queue on /dump)
> Add more details about compaction queue on /dump
>
>
> Key: HBASE-17991
> URL: https://issues.apache.org/jira/browse/HBASE-17991
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17991-master-v1.patch,
> HBASE-17991-master-v2.patch, HBASE-17991-master-v3.patch
>
>
> RS dump information as follows
> {code}
> RS Queue:
> ===
> Compaction/Split Queue summary: compaction_queue=(20:0), split_queue=0,
> merge_queue=0
> Compaction/Split Queue dump:
> LargeCompation Queue:
> Request =
> regionName=usertable,user4180497275766179957,1491904095205.1d4085e4438752f3611f66e7b043fe44.,
> storeName=0, fileCount=1, fileSize=1.9 G (1.9 G), priority=1,
> time=21697920409804647
> Request =
> regionName=usertable,user4568009753557153251,1491904099262.95bf004e3c9b35a58c60ca5d5b11d190.,
> storeName=0, fileCount=1, fileSize=1.9 G (1.9 G), priority=1,
> time=21697920413223800
> SmallCompation Queue:
> Store = b, pri = 108
> Store = b, pri = 108
> Store = b, pri = 108
> Store = b, pri = 108
> Store = b, pri = 108
> Store = b, pri = 109
> {code}
> Compaction queue information will be displayed on page /dump.
> If compation has selected the file, it will print the details information of
> the compation, otherwise only print storename and priority(Store = b, pri =
> 108) which is useless for us.
> So, we should also print more detailed information, such as regionName,
> starttime etc.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17993) Delete useless info log in RpcServer.processResponse
[ https://issues.apache.org/jira/browse/HBASE-17993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996871#comment-15996871 ] Hudson commented on HBASE-17993: FAILURE: Integrated in Jenkins build HBase-1.4 #720 (See [https://builds.apache.org/job/HBase-1.4/720/]) HBASE-17993 Delete useless info log in RpcServer.processResponse (zhangduo: rev a5f52360f60df48df056bfc47a71215a540fa76c) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java > Delete useless info log in RpcServer.processResponse > > > Key: HBASE-17993 > URL: https://issues.apache.org/jira/browse/HBASE-17993 > Project: HBase > Issue Type: Bug > Components: rpc >Affects Versions: 1.4.0 >Reporter: Jingyun Tian >Assignee: Jingyun Tian > Labels: trivial > Fix For: 1.4.0 > > Attachments: HBASE-17993-branch-1.patch > > > There is a log which is to track what current call is. It is used to > debugging, we'd better delete it in released version. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17991) Add more details about compation queue on /dump
[
https://issues.apache.org/jira/browse/HBASE-17991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996810#comment-15996810
]
Hadoop QA commented on HBASE-17991:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 58s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 111m 23s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 154m 18s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12866382/HBASE-17991-master-v3.patch
|
| JIRA Issue | HBASE-17991 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux d91a7934aa06 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 2de6b05 |
| Default Java | 1.8.0_131 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/6696/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6696/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6696/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Add more details about compation queue on /dump
> ---
>
> Key: HBASE-17991
> URL: https://issues.apache.org/jira/browse/HBASE-17991
> Project: HBase
>
[jira] [Updated] (HBASE-17667) Implement async flush/compact region methods
[
https://issues.apache.org/jira/browse/HBASE-17667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-17667:
---
Attachment: HBASE-17667.v3.patch
> Implement async flush/compact region methods
> --
>
> Key: HBASE-17667
> URL: https://issues.apache.org/jira/browse/HBASE-17667
> Project: HBase
> Issue Type: Sub-task
> Components: Admin, asyncclient, Client
>Affects Versions: 2.0.0
>Reporter: Zheng Hu
>Assignee: Zheng Hu
> Fix For: 2.0.0
>
> Attachments: HBASE-17667.v1.patch, HBASE-17667.v1.patch,
> HBASE-17667.v2.patch, HBASE-17667.v3.patch, HBASE-17667.v3.patch
>
>
> Implement following methods for async admin:
> {code}
> 1. flush table ;
> 2. flush region;
> 3. compact table;
> 4. compact region;
> 5. compact region server;
> 6. major compact for table;
> 7. major compact for region;
> 8. major compact for CF;
> 9. major compact for specific region and specific CF;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-16436) Add the CellChunkMap variant
[ https://issues.apache.org/jira/browse/HBASE-16436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anastasia Braginsky updated HBASE-16436: Attachment: HBASE-16436-V10.patch > Add the CellChunkMap variant > > > Key: HBASE-16436 > URL: https://issues.apache.org/jira/browse/HBASE-16436 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: Anastasia Braginsky > Fix For: 2.0.0 > > Attachments: HBASE-16436-V07.patch, HBASE-16436-V09.patch, > HBASE-16436-V10.patch, HBASE-16436-V43.patch > > > This sub-task is specifically to add the CellChunkMap created by [~anastas] > and [~eshcar] with specific tests and integrate it with the in memory > flush/compaction flow. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17994) Add async client test to Performance Evaluation tool
Guanghao Zhang created HBASE-17994: -- Summary: Add async client test to Performance Evaluation tool Key: HBASE-17994 URL: https://issues.apache.org/jira/browse/HBASE-17994 Project: HBase Issue Type: New Feature Affects Versions: 2.0.0 Reporter: Guanghao Zhang -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17991) Add more details about compation queue on /dump
[
https://issues.apache.org/jira/browse/HBASE-17991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996768#comment-15996768
]
Hadoop QA commented on HBASE-17991:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 34s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
54s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 40s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
37s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 3s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
62m 22s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 222m 22s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 316m 17s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestAsyncProcedureAdminApi |
| | hadoop.hbase.client.TestAsyncBalancerAdminApi |
| | hadoop.hbase.client.TestMobSnapshotCloneIndependence |
| | hadoop.hbase.client.TestAsyncTableAdminApi |
| | hadoop.hbase.util.TestHBaseFsckReplicas |
| | hadoop.hbase.client.TestAsyncSnapshotAdminApi |
| Timed out junit tests |
org.apache.hadoop.hbase.backup.TestBackupDeleteRestore |
| | org.apache.hadoop.hbase.filter.TestFuzzyRowFilterEndToEnd |
| | org.apache.hadoop.hbase.util.TestHBaseFsckTwoRS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12866358/HBASE-17991-master-v2.patch
|
| JIRA Issue | HBASE-17991 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 1c6c76fa1b2a 4.8.3-std-1 #1 SMP Fri Oct 21 11:15:43 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 2de6b05 |
| Default Java | 1.8.0_131 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/6692/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test
[jira] [Commented] (HBASE-17667) Implement async flush/compact region methods
[
https://issues.apache.org/jira/browse/HBASE-17667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996767#comment-15996767
]
Hadoop QA commented on HBASE-17667:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 28s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 40s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 12s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
29s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 28s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
4s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 55s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 55s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
63m 5s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 0s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 127m 31s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
43s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 234m 12s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestRegionReplicaFailover |
| | hadoop.hbase.regionserver.TestEncryptionKeyRotation |
| | hadoop.hbase.regionserver.TestPerColumnFamilyFlush |
| | hadoop.hbase.regionserver.TestCorruptedRegionStoreFile |
| Timed out junit tests |
org.apache.hadoop.hbase.replication.multiwal.TestReplicationSyncUpToolWithMultipleWAL
|
| | org.apache.hadoop.hbase.TestIOFencing |
| | org.apache.hadoop.hbase.replication.TestReplicationKillMasterRSCompressed
|
| |
org.apache.hadoop.hbase.replication.regionserver.TestTableBasedReplicationSourceManagerImpl
|
| | org.apache.hadoop.hbase.filter.TestFuzzyRowFilterEndToEnd |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12866372/HBASE-17667.v3.patch |
| JIRA Issue | HBASE-17667 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux bc68a72ca1a3 4.8.3-std-1 #1 SMP
[jira] [Updated] (HBASE-17993) Delete useless info log in RpcServer.processResponse
[ https://issues.apache.org/jira/browse/HBASE-17993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17993: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to branch-1. Thanks [~tianjingyun] for contributing. > Delete useless info log in RpcServer.processResponse > > > Key: HBASE-17993 > URL: https://issues.apache.org/jira/browse/HBASE-17993 > Project: HBase > Issue Type: Bug > Components: rpc >Affects Versions: 1.4.0 >Reporter: Jingyun Tian >Assignee: Jingyun Tian > Labels: trivial > Fix For: 1.4.0 > > Attachments: HBASE-17993-branch-1.patch > > > There is a log which is to track what current call is. It is used to > debugging, we'd better delete it in released version. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HBASE-11013) Clone Snapshots on Secure Cluster Should provide option to apply Retained User Permissions
[
https://issues.apache.org/jira/browse/HBASE-11013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Hu reassigned HBASE-11013:
Assignee: Zheng Hu
> Clone Snapshots on Secure Cluster Should provide option to apply Retained
> User Permissions
> --
>
> Key: HBASE-11013
> URL: https://issues.apache.org/jira/browse/HBASE-11013
> Project: HBase
> Issue Type: Improvement
> Components: snapshots
>Reporter: Ted Yu
>Assignee: Zheng Hu
>
> Currently,
> {code}
> sudo su - test_user
> create 't1', 'f1'
> sudo su - hbase
> snapshot 't1', 'snap_one'
> clone_snapshot 'snap_one', 't2'
> {code}
> In this scenario the user - test_user would not have permissions for the
> clone table t2.
> We need to add improvement feature such that the permissions of the original
> table are recorded in snapshot metadata and an option is provided for
> applying them to the new table as part of the clone process.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17991) Add more details about compation queue on /dump
[
https://issues.apache.org/jira/browse/HBASE-17991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guangxu Cheng updated HBASE-17991:
--
Attachment: HBASE-17991-master-v3.patch
v3 : a minor change. use long instead of Long
> Add more details about compation queue on /dump
> ---
>
> Key: HBASE-17991
> URL: https://issues.apache.org/jira/browse/HBASE-17991
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17991-master-v1.patch,
> HBASE-17991-master-v2.patch, HBASE-17991-master-v3.patch
>
>
> RS dump information as follows
> {code}
> RS Queue:
> ===
> Compaction/Split Queue summary: compaction_queue=(20:0), split_queue=0,
> merge_queue=0
> Compaction/Split Queue dump:
> LargeCompation Queue:
> Request =
> regionName=usertable,user4180497275766179957,1491904095205.1d4085e4438752f3611f66e7b043fe44.,
> storeName=0, fileCount=1, fileSize=1.9 G (1.9 G), priority=1,
> time=21697920409804647
> Request =
> regionName=usertable,user4568009753557153251,1491904099262.95bf004e3c9b35a58c60ca5d5b11d190.,
> storeName=0, fileCount=1, fileSize=1.9 G (1.9 G), priority=1,
> time=21697920413223800
> SmallCompation Queue:
> Store = b, pri = 108
> Store = b, pri = 108
> Store = b, pri = 108
> Store = b, pri = 108
> Store = b, pri = 108
> Store = b, pri = 109
> {code}
> Compaction queue information will be displayed on page /dump.
> If compation has selected the file, it will print the details information of
> the compation, otherwise only print storename and priority(Store = b, pri =
> 108) which is useless for us.
> So, we should also print more detailed information, such as regionName,
> starttime etc.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17993) Delete useless info log in RpcServer.processResponse
[
https://issues.apache.org/jira/browse/HBASE-17993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996578#comment-15996578
]
Hadoop QA commented on HBASE-17993:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 23s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
45s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} branch-1 passed with JDK v1.8.0_131 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 0s
{color} | {color:red} hbase-server in branch-1 has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} branch-1 passed with JDK v1.8.0_131 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.8.0_131 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
15m 33s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed with JDK v1.8.0_131 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 88m 30s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 118m 54s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestReplicasClient |
| | hadoop.hbase.regionserver.TestRSKilledWhenInitializing |
| | hadoop.hbase.regionserver.TestCompactionInDeadRegionServer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:58c504e |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12866362/HBASE-17993-branch-1.patch
|
| JIRA Issue | HBASE-17993 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
|
[jira] [Assigned] (HBASE-17690) Clean up MOB code
[ https://issues.apache.org/jira/browse/HBASE-17690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du reassigned HBASE-17690: Assignee: Jingcheng Du (was: Xiang Li) > Clean up MOB code > - > > Key: HBASE-17690 > URL: https://issues.apache.org/jira/browse/HBASE-17690 > Project: HBase > Issue Type: Improvement > Components: mob >Reporter: Jingcheng Du >Assignee: Jingcheng Du > Fix For: 2.0.0 > > Attachments: HBASE-17690.patch, HBASE-17690-v1.patch > > > Clean up the code in MOB. > # Fix the incorrect description in comments. > # Fix the warning and remove redundant import in code. > # Remove the references to the deprecated code. > # Add throughput controller for DefaultMobStoreFlusher and > DefaultMobStoreCompactor. > The RB link is https://reviews.apache.org/r/57078/. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17667) Implement async flush/compact region methods
[
https://issues.apache.org/jira/browse/HBASE-17667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Hu updated HBASE-17667:
-
Attachment: HBASE-17667.v3.patch
Rebase & implement flush() method.
> Implement async flush/compact region methods
> --
>
> Key: HBASE-17667
> URL: https://issues.apache.org/jira/browse/HBASE-17667
> Project: HBase
> Issue Type: Sub-task
> Components: Admin, asyncclient, Client
>Affects Versions: 2.0.0
>Reporter: Zheng Hu
>Assignee: Zheng Hu
> Fix For: 2.0.0
>
> Attachments: HBASE-17667.v1.patch, HBASE-17667.v1.patch,
> HBASE-17667.v2.patch, HBASE-17667.v3.patch
>
>
> Implement following methods for async admin:
> {code}
> 1. flush table ;
> 2. flush region;
> 3. compact table;
> 4. compact region;
> 5. compact region server;
> 6. major compact for table;
> 7. major compact for region;
> 8. major compact for CF;
> 9. major compact for specific region and specific CF;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17977) Enabled Master observer to delete quotas on table deletion by default
[
https://issues.apache.org/jira/browse/HBASE-17977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996457#comment-15996457
]
Hadoop QA commented on HBASE-17977:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 29s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 39s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
38s {color} | {color:green} HBASE-16961 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 16s
{color} | {color:green} HBASE-16961 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
56s {color} | {color:green} HBASE-16961 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m
11s {color} | {color:green} HBASE-16961 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
20s {color} | {color:green} HBASE-16961 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 1s
{color} | {color:green} HBASE-16961 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 20s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
0s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 59s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 59s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m 19s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.4.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m 40s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.4.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m 59s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.5.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 9m 19s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.5.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 11m 37s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.5.2. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 48s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 111m 53s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 209m 17s
{color} | {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
36s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 458m 1s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestRegionReplicaFailover |
| | hadoop.hbase.client.TestAsyncRegionAdminApi |
| | hadoop.hbase.snapshot.TestExportSnapshot |
| |
[jira] [Updated] (HBASE-17993) Delete useless info log in RpcServer.processResponse
[ https://issues.apache.org/jira/browse/HBASE-17993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingyun Tian updated HBASE-17993: - Attachment: (was: HBASE-17993.patch) > Delete useless info log in RpcServer.processResponse > > > Key: HBASE-17993 > URL: https://issues.apache.org/jira/browse/HBASE-17993 > Project: HBase > Issue Type: Bug > Components: rpc >Affects Versions: 1.4.0 >Reporter: Jingyun Tian >Assignee: Jingyun Tian > Labels: trivial > Fix For: 1.4.0 > > Attachments: HBASE-17993-branch-1.patch > > > There is a log which is to track what current call is. It is used to > debugging, we'd better delete it in released version. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HBASE-17993) Delete useless info log in RpcServer.processResponse
[ https://issues.apache.org/jira/browse/HBASE-17993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reassigned HBASE-17993: - Assignee: Jingyun Tian (was: Jingyun Tian) > Delete useless info log in RpcServer.processResponse > > > Key: HBASE-17993 > URL: https://issues.apache.org/jira/browse/HBASE-17993 > Project: HBase > Issue Type: Bug > Components: rpc >Affects Versions: 1.4.0 >Reporter: Jingyun Tian >Assignee: Jingyun Tian > Labels: trivial > Fix For: 1.4.0 > > Attachments: HBASE-17993-branch-1.patch, HBASE-17993.patch > > > There is a log which is to track what current call is. It is used to > debugging, we'd better delete it in released version. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17993) Delete useless info log in RpcServer.processResponse
[ https://issues.apache.org/jira/browse/HBASE-17993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17993: -- Attachment: HBASE-17993-branch-1.patch Rename the patch. > Delete useless info log in RpcServer.processResponse > > > Key: HBASE-17993 > URL: https://issues.apache.org/jira/browse/HBASE-17993 > Project: HBase > Issue Type: Bug > Components: rpc >Affects Versions: 1.4.0 >Reporter: Jingyun Tian >Assignee: Jingyun Tian > Labels: trivial > Fix For: 1.4.0 > > Attachments: HBASE-17993-branch-1.patch, HBASE-17993.patch > > > There is a log which is to track what current call is. It is used to > debugging, we'd better delete it in released version. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17993) Delete useless info log in RpcServer.processResponse
[
https://issues.apache.org/jira/browse/HBASE-17993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996416#comment-15996416
]
Hadoop QA commented on HBASE-17993:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s {color}
| {color:red} HBASE-17993 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12866361/HBASE-17993.patch |
| JIRA Issue | HBASE-17993 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6693/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Delete useless info log in RpcServer.processResponse
>
>
> Key: HBASE-17993
> URL: https://issues.apache.org/jira/browse/HBASE-17993
> Project: HBase
> Issue Type: Bug
> Components: rpc
>Affects Versions: 1.4.0
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
> Labels: trivial
> Fix For: 1.4.0
>
> Attachments: HBASE-17993.patch
>
>
> There is a log which is to track what current call is. It is used to
> debugging, we'd better delete it in released version.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17993) Delete useless info log in RpcServer.processResponse
[ https://issues.apache.org/jira/browse/HBASE-17993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996407#comment-15996407 ] Duo Zhang commented on HBASE-17993: --- +1, this is a trivial patch. Will commit after we get the pre commit result. > Delete useless info log in RpcServer.processResponse > > > Key: HBASE-17993 > URL: https://issues.apache.org/jira/browse/HBASE-17993 > Project: HBase > Issue Type: Bug > Components: rpc >Affects Versions: 1.4.0 >Reporter: Jingyun Tian >Assignee: Jingyun Tian > Labels: trivial > Fix For: 1.4.0 > > Attachments: HBASE-17993.patch > > > There is a log which is to track what current call is. It is used to > debugging, we'd better delete it in released version. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17993) Delete useless info log in RpcServer.processResponse
[ https://issues.apache.org/jira/browse/HBASE-17993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17993: -- Status: Patch Available (was: Open) > Delete useless info log in RpcServer.processResponse > > > Key: HBASE-17993 > URL: https://issues.apache.org/jira/browse/HBASE-17993 > Project: HBase > Issue Type: Bug > Components: rpc >Affects Versions: 1.4.0 >Reporter: Jingyun Tian >Assignee: Jingyun Tian > Labels: trivial > Fix For: 1.4.0 > > Attachments: HBASE-17993.patch > > > There is a log which is to track what current call is. It is used to > debugging, we'd better delete it in released version. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17993) Delete useless info log in RpcServer.processResponse
[ https://issues.apache.org/jira/browse/HBASE-17993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17993: -- Affects Version/s: (was: 1.0.0) 1.4.0 Fix Version/s: 1.4.0 > Delete useless info log in RpcServer.processResponse > > > Key: HBASE-17993 > URL: https://issues.apache.org/jira/browse/HBASE-17993 > Project: HBase > Issue Type: Bug > Components: rpc >Affects Versions: 1.4.0 >Reporter: Jingyun Tian >Assignee: Jingyun Tian > Labels: trivial > Fix For: 1.4.0 > > Attachments: HBASE-17993.patch > > > There is a log which is to track what current call is. It is used to > debugging, we'd better delete it in released version. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HBASE-17993) Delete useless info log in RpcServer.processResponse
[ https://issues.apache.org/jira/browse/HBASE-17993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reassigned HBASE-17993: - Assignee: Jingyun Tian > Delete useless info log in RpcServer.processResponse > > > Key: HBASE-17993 > URL: https://issues.apache.org/jira/browse/HBASE-17993 > Project: HBase > Issue Type: Bug > Components: rpc >Affects Versions: 1.0.0 >Reporter: Jingyun Tian >Assignee: Jingyun Tian > Labels: trivial > Attachments: HBASE-17993.patch > > > There is a log which is to track what current call is. It is used to > debugging, we'd better delete it in released version. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17993) Delete useless info log in RpcServer.processResponse
[ https://issues.apache.org/jira/browse/HBASE-17993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingyun Tian updated HBASE-17993: - Fix Version/s: (was: 1.4.0) > Delete useless info log in RpcServer.processResponse > > > Key: HBASE-17993 > URL: https://issues.apache.org/jira/browse/HBASE-17993 > Project: HBase > Issue Type: Bug > Components: rpc >Affects Versions: 1.0.0 >Reporter: Jingyun Tian > Labels: trivial > Attachments: HBASE-17993.patch > > > There is a log which is to track what current call is. It is used to > debugging, we'd better delete it in released version. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17993) Delete useless info log in RpcServer.processResponse
[ https://issues.apache.org/jira/browse/HBASE-17993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996394#comment-15996394 ] Duo Zhang commented on HBASE-17993: --- Yeah I think it was introduced by me when backporting the new rpc implementation to branch-1... > Delete useless info log in RpcServer.processResponse > > > Key: HBASE-17993 > URL: https://issues.apache.org/jira/browse/HBASE-17993 > Project: HBase > Issue Type: Bug > Components: rpc >Affects Versions: 1.0.0 >Reporter: Jingyun Tian > Labels: trivial > Attachments: HBASE-17993.patch > > > There is a log which is to track what current call is. It is used to > debugging, we'd better delete it in released version. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
