[jira] [Resolved] (HBASE-13360) Standalone example docs (for Raft / HBASE-12259)
[
https://issues.apache.org/jira/browse/HBASE-13360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jean-Marc Spaggiari resolved HBASE-13360.
-
Resolution: Invalid
Standalone example docs (for Raft / HBASE-12259)
Key: HBASE-13360
URL: https://issues.apache.org/jira/browse/HBASE-13360
Project: HBase
Issue Type: Improvement
Components: Consensus
Affects Versions: HBASE-12259
Reporter: Justin SB
Priority: Minor
I'm very interested in the Raft implementation (in particular, whether it can
be reused elsewhere, given that this effort seems likely to produce the
canonical Raft implementation).
I'd like to try it standalone, but I'm not yet having much success.
I start a cluster of 3 servers (N=0,1,2):
java -cp
conf:hbase-consensus/target/hbase-consensus-2.0.0-SNAPSHOT.jar:hbase-consensus/target/dependency/*
org.apache.hadoop.hbase.consensus.server.LocalConsensusServer -region 1
-servers 127.0.0.1:1,127.0.0.1:10001,127.0.0.1:10002 -debug DEBUG
-localIndex ${N}
And then I try running the load-test client:
java -cp
hbase-consensus/target/hbase-consensus-2.0.0-SNAPSHOT.jar:hbase-consensus/target/dependency/*
org.apache.hadoop.hbase.consensus.client.QuorumLoadTestClient -region 1
-servers 127.0.0.1:1,127.0.0.1:10001,127.0.0.1:10002
That gives an immediate NPE at
org.apache.hadoop.hbase.consensus.quorum.QuorumInfo.populateInternalMaps(QuorumInfo.java:315).
Are there any docs on how to get started playing with running this branch?
Thanks!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13360) Standalone example docs (for Raft / HBASE-12259)
[
https://issues.apache.org/jira/browse/HBASE-13360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14386253#comment-14386253
]
Jean-Marc Spaggiari commented on HBASE-13360:
-
Hi Justin,
If you have questions about how to run HBase, or where to find the
documentation, etc. the right place to ask is the HBase mailing list.
http://hbase.apache.org/mail-lists.html
Thanks.
Standalone example docs (for Raft / HBASE-12259)
Key: HBASE-13360
URL: https://issues.apache.org/jira/browse/HBASE-13360
Project: HBase
Issue Type: Improvement
Components: Consensus
Affects Versions: HBASE-12259
Reporter: Justin SB
Priority: Minor
I'm very interested in the Raft implementation (in particular, whether it can
be reused elsewhere, given that this effort seems likely to produce the
canonical Raft implementation).
I'd like to try it standalone, but I'm not yet having much success.
I start a cluster of 3 servers (N=0,1,2):
java -cp
conf:hbase-consensus/target/hbase-consensus-2.0.0-SNAPSHOT.jar:hbase-consensus/target/dependency/*
org.apache.hadoop.hbase.consensus.server.LocalConsensusServer -region 1
-servers 127.0.0.1:1,127.0.0.1:10001,127.0.0.1:10002 -debug DEBUG
-localIndex ${N}
And then I try running the load-test client:
java -cp
hbase-consensus/target/hbase-consensus-2.0.0-SNAPSHOT.jar:hbase-consensus/target/dependency/*
org.apache.hadoop.hbase.consensus.client.QuorumLoadTestClient -region 1
-servers 127.0.0.1:1,127.0.0.1:10001,127.0.0.1:10002
That gives an immediate NPE at
org.apache.hadoop.hbase.consensus.quorum.QuorumInfo.populateInternalMaps(QuorumInfo.java:315).
Are there any docs on how to get started playing with running this branch?
Thanks!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13336) Consistent rules for security meta table protections
[ https://issues.apache.org/jira/browse/HBASE-13336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14386302#comment-14386302 ] Anoop Sam John commented on HBASE-13336: +1 for the rules what [~apurtell] said.. Consistent rules for security meta table protections Key: HBASE-13336 URL: https://issues.apache.org/jira/browse/HBASE-13336 Project: HBase Issue Type: Improvement Reporter: Andrew Purtell Assignee: Srikanth Srungarapu Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.13 The AccessController and VisibilityController do different things regarding protecting their meta tables. The AC allows schema changes and disable/enable if the user has permission. The VC unconditionally disallows all admin actions. Generally, bad things will happen if these meta tables are damaged, disabled, or dropped. The likely outcome is random frequent (or constant) server side op failures with nasty stack traces. On the other hand some things like column family and table attribute changes can have valid use cases. We should have consistent and sensible rules for protecting security meta tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-10728) get_counter value is never used.
[ https://issues.apache.org/jira/browse/HBASE-10728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars George updated HBASE-10728: Resolution: Fixed Status: Resolved (was: Patch Available) get_counter value is never used. Key: HBASE-10728 URL: https://issues.apache.org/jira/browse/HBASE-10728 Project: HBase Issue Type: Bug Affects Versions: 0.96.2, 0.98.1, 0.99.0, 1.0.0 Reporter: Jean-Marc Spaggiari Assignee: Jean-Marc Spaggiari Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.13 Attachments: 0001-HBASE-10728-get_counter-value-is-never-used.patch, 0002-HBASE-10728-get_counter-value-is-never-used.patch, HBASE-10728-v0-0.96.patch, HBASE-10728-v0-0.98.patch, HBASE-10728-v0-trunk.patch, HBASE-10728-v1-0.96.patch, HBASE-10728-v1-0.98.patch, HBASE-10728-v1-trunk.patch, HBASE-10728-v2-trunk.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10728) get_counter value is never used.
[ https://issues.apache.org/jira/browse/HBASE-10728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14386407#comment-14386407 ] Lars George commented on HBASE-10728: - Committed to 0.98, branch-1, branch-1.0, and master. Thanks [~jmspaggi] for the issue. get_counter value is never used. Key: HBASE-10728 URL: https://issues.apache.org/jira/browse/HBASE-10728 Project: HBase Issue Type: Bug Affects Versions: 0.96.2, 0.98.1, 0.99.0, 1.0.0 Reporter: Jean-Marc Spaggiari Assignee: Jean-Marc Spaggiari Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.13 Attachments: 0001-HBASE-10728-get_counter-value-is-never-used.patch, 0002-HBASE-10728-get_counter-value-is-never-used.patch, HBASE-10728-v0-0.96.patch, HBASE-10728-v0-0.98.patch, HBASE-10728-v0-trunk.patch, HBASE-10728-v1-0.96.patch, HBASE-10728-v1-0.98.patch, HBASE-10728-v1-trunk.patch, HBASE-10728-v2-trunk.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10728) get_counter value is never used.
[ https://issues.apache.org/jira/browse/HBASE-10728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14386410#comment-14386410 ] Hudson commented on HBASE-10728: FAILURE: Integrated in HBase-1.0 #838 (See [https://builds.apache.org/job/HBase-1.0/838/]) HBASE-10728 get_counter value is never used. (larsgeorge: rev 34abf48f8795f1d04f06c97f013a3fb4b9116873) * hbase-shell/src/main/ruby/shell/commands/incr.rb * hbase-shell/src/main/ruby/shell/commands/get_counter.rb * hbase-shell/src/main/ruby/hbase/table.rb get_counter value is never used. Key: HBASE-10728 URL: https://issues.apache.org/jira/browse/HBASE-10728 Project: HBase Issue Type: Bug Affects Versions: 0.96.2, 0.98.1, 0.99.0, 1.0.0 Reporter: Jean-Marc Spaggiari Assignee: Jean-Marc Spaggiari Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.13 Attachments: 0001-HBASE-10728-get_counter-value-is-never-used.patch, 0002-HBASE-10728-get_counter-value-is-never-used.patch, HBASE-10728-v0-0.96.patch, HBASE-10728-v0-0.98.patch, HBASE-10728-v0-trunk.patch, HBASE-10728-v1-0.96.patch, HBASE-10728-v1-0.98.patch, HBASE-10728-v1-trunk.patch, HBASE-10728-v2-trunk.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-10728) get_counter value is never used.
[ https://issues.apache.org/jira/browse/HBASE-10728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars George updated HBASE-10728: Attachment: 0001-HBASE-10728-get_counter-value-is-never-used-branch-1.patch 0001-HBASE-10728-get_counter-value-is-never-used-branch-1.0.patch 0001-HBASE-10728-get_counter-value-is-never-used-0.98.patch get_counter value is never used. Key: HBASE-10728 URL: https://issues.apache.org/jira/browse/HBASE-10728 Project: HBase Issue Type: Bug Affects Versions: 0.96.2, 0.98.1, 0.99.0, 1.0.0 Reporter: Jean-Marc Spaggiari Assignee: Jean-Marc Spaggiari Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.13 Attachments: 0001-HBASE-10728-get_counter-value-is-never-used-0.98.patch, 0001-HBASE-10728-get_counter-value-is-never-used-branch-1.0.patch, 0001-HBASE-10728-get_counter-value-is-never-used-branch-1.patch, 0001-HBASE-10728-get_counter-value-is-never-used.patch, 0002-HBASE-10728-get_counter-value-is-never-used.patch, HBASE-10728-v0-0.96.patch, HBASE-10728-v0-0.98.patch, HBASE-10728-v0-trunk.patch, HBASE-10728-v1-0.96.patch, HBASE-10728-v1-0.98.patch, HBASE-10728-v1-trunk.patch, HBASE-10728-v2-trunk.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10728) get_counter value is never used.
[ https://issues.apache.org/jira/browse/HBASE-10728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14386451#comment-14386451 ] Hudson commented on HBASE-10728: FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #879 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/879/]) HBASE-10728 get_counter value is never used. (larsgeorge: rev 37e423aa399a6df937e89ee19ca0578962d855fe) * hbase-shell/src/main/ruby/hbase/table.rb * hbase-shell/src/main/ruby/shell/commands/get_counter.rb * hbase-shell/src/main/ruby/shell/commands/incr.rb get_counter value is never used. Key: HBASE-10728 URL: https://issues.apache.org/jira/browse/HBASE-10728 Project: HBase Issue Type: Bug Affects Versions: 0.96.2, 0.98.1, 0.99.0, 1.0.0 Reporter: Jean-Marc Spaggiari Assignee: Jean-Marc Spaggiari Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.13 Attachments: 0001-HBASE-10728-get_counter-value-is-never-used-0.98.patch, 0001-HBASE-10728-get_counter-value-is-never-used-branch-1.0.patch, 0001-HBASE-10728-get_counter-value-is-never-used-branch-1.patch, 0001-HBASE-10728-get_counter-value-is-never-used.patch, 0002-HBASE-10728-get_counter-value-is-never-used.patch, HBASE-10728-v0-0.96.patch, HBASE-10728-v0-0.98.patch, HBASE-10728-v0-trunk.patch, HBASE-10728-v1-0.96.patch, HBASE-10728-v1-0.98.patch, HBASE-10728-v1-trunk.patch, HBASE-10728-v2-trunk.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10728) get_counter value is never used.
[ https://issues.apache.org/jira/browse/HBASE-10728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14386531#comment-14386531 ] Hudson commented on HBASE-10728: FAILURE: Integrated in HBase-1.1 #338 (See [https://builds.apache.org/job/HBase-1.1/338/]) HBASE-10728 get_counter value is never used. (larsgeorge: rev 26f4c32d5407a3a9fff6977768d954d357628f6e) * hbase-shell/src/main/ruby/hbase/table.rb * hbase-shell/src/main/ruby/shell/commands/get_counter.rb * hbase-shell/src/main/ruby/shell/commands/incr.rb get_counter value is never used. Key: HBASE-10728 URL: https://issues.apache.org/jira/browse/HBASE-10728 Project: HBase Issue Type: Bug Affects Versions: 0.96.2, 0.98.1, 0.99.0, 1.0.0 Reporter: Jean-Marc Spaggiari Assignee: Jean-Marc Spaggiari Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.13 Attachments: 0001-HBASE-10728-get_counter-value-is-never-used-0.98.patch, 0001-HBASE-10728-get_counter-value-is-never-used-branch-1.0.patch, 0001-HBASE-10728-get_counter-value-is-never-used-branch-1.patch, 0001-HBASE-10728-get_counter-value-is-never-used.patch, 0002-HBASE-10728-get_counter-value-is-never-used.patch, HBASE-10728-v0-0.96.patch, HBASE-10728-v0-0.98.patch, HBASE-10728-v0-trunk.patch, HBASE-10728-v1-0.96.patch, HBASE-10728-v1-0.98.patch, HBASE-10728-v1-trunk.patch, HBASE-10728-v2-trunk.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10728) get_counter value is never used.
[ https://issues.apache.org/jira/browse/HBASE-10728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14386674#comment-14386674 ] Hudson commented on HBASE-10728: SUCCESS: Integrated in HBase-0.98 #925 (See [https://builds.apache.org/job/HBase-0.98/925/]) HBASE-10728 get_counter value is never used. (larsgeorge: rev 37e423aa399a6df937e89ee19ca0578962d855fe) * hbase-shell/src/main/ruby/shell/commands/incr.rb * hbase-shell/src/main/ruby/shell/commands/get_counter.rb * hbase-shell/src/main/ruby/hbase/table.rb get_counter value is never used. Key: HBASE-10728 URL: https://issues.apache.org/jira/browse/HBASE-10728 Project: HBase Issue Type: Bug Affects Versions: 0.96.2, 0.98.1, 0.99.0, 1.0.0 Reporter: Jean-Marc Spaggiari Assignee: Jean-Marc Spaggiari Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.13 Attachments: 0001-HBASE-10728-get_counter-value-is-never-used-0.98.patch, 0001-HBASE-10728-get_counter-value-is-never-used-branch-1.0.patch, 0001-HBASE-10728-get_counter-value-is-never-used-branch-1.patch, 0001-HBASE-10728-get_counter-value-is-never-used.patch, 0002-HBASE-10728-get_counter-value-is-never-used.patch, HBASE-10728-v0-0.96.patch, HBASE-10728-v0-0.98.patch, HBASE-10728-v0-trunk.patch, HBASE-10728-v1-0.96.patch, HBASE-10728-v1-0.98.patch, HBASE-10728-v1-trunk.patch, HBASE-10728-v2-trunk.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10728) get_counter value is never used.
[ https://issues.apache.org/jira/browse/HBASE-10728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14386572#comment-14386572 ] Hudson commented on HBASE-10728: SUCCESS: Integrated in HBase-TRUNK #6321 (See [https://builds.apache.org/job/HBase-TRUNK/6321/]) HBASE-10728 get_counter value is never used. (larsgeorge: rev 7f8745453ed6b5eb085de1fcb52c1a975cc1471a) * hbase-shell/src/main/ruby/shell/commands/get_counter.rb * hbase-shell/src/main/ruby/shell/commands/incr.rb * hbase-shell/src/main/ruby/hbase/table.rb get_counter value is never used. Key: HBASE-10728 URL: https://issues.apache.org/jira/browse/HBASE-10728 Project: HBase Issue Type: Bug Affects Versions: 0.96.2, 0.98.1, 0.99.0, 1.0.0 Reporter: Jean-Marc Spaggiari Assignee: Jean-Marc Spaggiari Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.13 Attachments: 0001-HBASE-10728-get_counter-value-is-never-used-0.98.patch, 0001-HBASE-10728-get_counter-value-is-never-used-branch-1.0.patch, 0001-HBASE-10728-get_counter-value-is-never-used-branch-1.patch, 0001-HBASE-10728-get_counter-value-is-never-used.patch, 0002-HBASE-10728-get_counter-value-is-never-used.patch, HBASE-10728-v0-0.96.patch, HBASE-10728-v0-0.98.patch, HBASE-10728-v0-trunk.patch, HBASE-10728-v1-0.96.patch, HBASE-10728-v1-0.98.patch, HBASE-10728-v1-trunk.patch, HBASE-10728-v2-trunk.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13361) Remove or undeprecate {get|set}ScannerCaching in HTable
[
https://issues.apache.org/jira/browse/HBASE-13361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-13361:
--
Resolution: Fixed
Fix Version/s: 2.0.0
Release Note: Removed getScannerCaching and setScannerCaching from Table
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks [~lars_francke] Applied to master branch.
Remove or undeprecate {get|set}ScannerCaching in HTable
---
Key: HBASE-13361
URL: https://issues.apache.org/jira/browse/HBASE-13361
Project: HBase
Issue Type: Task
Components: Client
Affects Versions: 1.0.0
Reporter: Lars Francke
Assignee: Lars Francke
Priority: Minor
Fix For: 2.0.0
Attachments: HBASE-13361.patch
{{getScannerCaching}} and {{setScannerCaching}} have been deprecated since
2011 (HBASE-4439). I think it's time to either remove them or undeprecate.
I was not able to find any other issues in JIRA or discussions on mailing
lists regarding deprecation removal policies but I think 3+ years is probably
enough warning.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13348) Separate the thread number configs for meta server and server operations
[
https://issues.apache.org/jira/browse/HBASE-13348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14386828#comment-14386828
]
stack commented on HBASE-13348:
---
Looks good to me. +1. On commit put a '.' between meta and serverops so
'meta.serverop' but this is nit.
Separate the thread number configs for meta server and server operations
Key: HBASE-13348
URL: https://issues.apache.org/jira/browse/HBASE-13348
Project: HBase
Issue Type: Improvement
Components: master
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor
Fix For: 2.0.0
Attachments: HBASE-13348-v1.diff
Currently, the config keys for thread number of meta server and server
operations in HMaster are same:
See: HMaster.java #993
{code}
this.service.startExecutorService(ExecutorType.MASTER_SERVER_OPERATIONS,
conf.getInt(hbase.master.executor.serverops.threads, 5));
this.service.startExecutorService(ExecutorType.MASTER_META_SERVER_OPERATIONS,
conf.getInt(hbase.master.executor.serverops.threads, 5));
{code}
In large cluster, we usually enlarge the thread number for server operation
separately to make the master handle regionserver shutdown events quickly
in some extremely cases.
So I think we need separate the thread number config for the two operations.
Suggestions are welcomed. Thanks~
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13362) set max result size from client only (like caching)?
[ https://issues.apache.org/jira/browse/HBASE-13362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14386846#comment-14386846 ] stack commented on HBASE-13362: --- Sounds good to me. [~jonathan.lawlor] What you think? set max result size from client only (like caching)? Key: HBASE-13362 URL: https://issues.apache.org/jira/browse/HBASE-13362 Project: HBase Issue Type: Brainstorming Reporter: Lars Hofhansl With the recent problems we've been seeing client/server result size mismatch, I was thinking: Why was this not a problem with scanner caching? There are two reasons: # number of rows is easy to calculate (and we did it correctly) # caching is only controlled from the client, never set on the server alone We did fix both #1 and #2 in HBASE-13262. Still, I'd like to discuss the following: * default the client sent max result size to 2mb * remove any server only result sizing * continue to use hbase.client.scanner.max.result.size but enforce it via the client only (as the name implies anyway). Comments? Concerns? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13209) Procedure V2 - master Add/Modify/Delete Column Family
[ https://issues.apache.org/jira/browse/HBASE-13209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14386968#comment-14386968 ] Stephen Yuan Jiang commented on HBASE-13209: The patch is posted in review board: https://reviews.apache.org/r/32627/ Procedure V2 - master Add/Modify/Delete Column Family - Key: HBASE-13209 URL: https://issues.apache.org/jira/browse/HBASE-13209 Project: HBase Issue Type: Sub-task Components: master Affects Versions: 2.0.0, 1.1.0 Reporter: Stephen Yuan Jiang Assignee: Stephen Yuan Jiang Labels: reliability Attachments: AlterColumnFamilyServerPV2-draft.v0-master.patch Original Estimate: 168h Time Spent: 72h Remaining Estimate: 96h master side, part of HBASE-12439 starts up the procedure executor on the master and replaces the add/modify/delete handlers with the procedure version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13361) Remove or undeprecate {get|set}ScannerCaching in HTable
[
https://issues.apache.org/jira/browse/HBASE-13361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14386979#comment-14386979
]
Hudson commented on HBASE-13361:
FAILURE: Integrated in HBase-TRUNK #6322 (See
[https://builds.apache.org/job/HBase-TRUNK/6322/])
HBASE-13361 Remove or undeprecate {get|set}ScannerCaching in HTable (Lars
Francke) (stack: rev 3815a33e3400cee73f63e205afac8f1a2d9c174f)
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
Remove or undeprecate {get|set}ScannerCaching in HTable
---
Key: HBASE-13361
URL: https://issues.apache.org/jira/browse/HBASE-13361
Project: HBase
Issue Type: Task
Components: Client
Affects Versions: 1.0.0
Reporter: Lars Francke
Assignee: Lars Francke
Priority: Minor
Fix For: 2.0.0
Attachments: HBASE-13361.patch
{{getScannerCaching}} and {{setScannerCaching}} have been deprecated since
2011 (HBASE-4439). I think it's time to either remove them or undeprecate.
I was not able to find any other issues in JIRA or discussions on mailing
lists regarding deprecation removal policies but I think 3+ years is probably
enough warning.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13210) Procedure V2 - master Modify table
[ https://issues.apache.org/jira/browse/HBASE-13210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-13210: --- Attachment: (was: ModifyTableProcedureServerPV2-draft.v0-master.patch) Procedure V2 - master Modify table -- Key: HBASE-13210 URL: https://issues.apache.org/jira/browse/HBASE-13210 Project: HBase Issue Type: Sub-task Components: master Affects Versions: 2.0.0, 1.1.0 Reporter: Stephen Yuan Jiang Assignee: Stephen Yuan Jiang Labels: reliablity Original Estimate: 72h Time Spent: 48h Remaining Estimate: 24h master side, part of HBASE-12439 starts up the procedure executor on the master and replaces the modify table handlers with the procedure version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13213) Split out locality metrics among primary and secondary region
[ https://issues.apache.org/jira/browse/HBASE-13213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387037#comment-14387037 ] Devaraj Das commented on HBASE-13213: - I don't follow the change you made in TestRegionServerMetrics. You should have made a change in TestMetricsRegionServer in order to have the change in MetricsRegionServerWrapperStub.java have any effect... Split out locality metrics among primary and secondary region - Key: HBASE-13213 URL: https://issues.apache.org/jira/browse/HBASE-13213 Project: HBase Issue Type: Sub-task Reporter: Ted Yu Assignee: Ted Yu Attachments: 13213-v1.patch This task provides the ability to track locality of primary and secondary region replicas. We already have percentFilesLocal metric. There should be two sets of metrics - one for primaries and another for secondary / tertiary regions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13362) set max result size from client only (like caching)?
[ https://issues.apache.org/jira/browse/HBASE-13362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14386983#comment-14386983 ] Jonathan Lawlor commented on HBASE-13362: - This sounds like a great idea. As [~lhofhansl] pointed out with the test in HBASE-13297, it easy for the client and server to have different configurations for the default max result size. Prior to HBASE-13262 this would have meant data loss. With HBASE-13262 we no longer have data loss, it's just ugly. If instead it was dealt with in the same manner as caching (e.g. the caching value must be carried in the ScanRequest to the server) it would be much cleaner. The 2mb default sounds good. Probably obvious, but just to be clear, we would still need to support instances where the client uses a negative maxResultSize to indicate that the response should not be limited by the result size (i.e. negative maxResultSize is equivalent to maxResultSize = Long.MAX_VALUE). If backported prior to branch-1, it would be nice to accompany this change with a change in the default caching value (from the current default of 100 to Integer.MAX_VALUE) so that the size limit is reached by default, rather than the caching/row limit (I say prior to branch-1 because the defaults of caching/maxResultSize in branch-1+ will already produce this behavior). Granted, this accompanying change would probably be dealt with best in a separate JIRA. set max result size from client only (like caching)? Key: HBASE-13362 URL: https://issues.apache.org/jira/browse/HBASE-13362 Project: HBase Issue Type: Brainstorming Reporter: Lars Hofhansl With the recent problems we've been seeing client/server result size mismatch, I was thinking: Why was this not a problem with scanner caching? There are two reasons: # number of rows is easy to calculate (and we did it correctly) # caching is only controlled from the client, never set on the server alone We did fix both #1 and #2 in HBASE-13262. Still, I'd like to discuss the following: * default the client sent max result size to 2mb * remove any server only result sizing * continue to use hbase.client.scanner.max.result.size but enforce it via the client only (as the name implies anyway). Comments? Concerns? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13363) Allow tables to be read only while receiving replication
Elliott Clark created HBASE-13363: - Summary: Allow tables to be read only while receiving replication Key: HBASE-13363 URL: https://issues.apache.org/jira/browse/HBASE-13363 Project: HBase Issue Type: Bug Reporter: Elliott Clark Allow a token to determine if a table is writable. Allow passing the token around to which cluster is hot. Hence the token will be a hot potato. Or a HotMutato -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13209) Procedure V2 - master Add/Modify/Delete Column Family
[ https://issues.apache.org/jira/browse/HBASE-13209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14386972#comment-14386972 ] Stephen Yuan Jiang commented on HBASE-13209: Should there be a timeout for the wait ? Today, we wait forever on sync operation - this is not changed from PV2 The TODO's were meant for delete column family, right ? TODO for rollback steps What if IOE is thrown from mfs.deleteFamilyFromFS() ? The procedure would fail. Procedure V2 - master Add/Modify/Delete Column Family - Key: HBASE-13209 URL: https://issues.apache.org/jira/browse/HBASE-13209 Project: HBase Issue Type: Sub-task Components: master Affects Versions: 2.0.0, 1.1.0 Reporter: Stephen Yuan Jiang Assignee: Stephen Yuan Jiang Labels: reliability Attachments: AlterColumnFamilyServerPV2-draft.v0-master.patch Original Estimate: 168h Time Spent: 72h Remaining Estimate: 96h master side, part of HBASE-12439 starts up the procedure executor on the master and replaces the add/modify/delete handlers with the procedure version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13209) Procedure V2 - master Add/Modify/Delete Column Family
[ https://issues.apache.org/jira/browse/HBASE-13209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-13209: --- Attachment: AlterColumnFamilyPV2-draft.v0-master.patch Procedure V2 - master Add/Modify/Delete Column Family - Key: HBASE-13209 URL: https://issues.apache.org/jira/browse/HBASE-13209 Project: HBase Issue Type: Sub-task Components: master Affects Versions: 2.0.0, 1.1.0 Reporter: Stephen Yuan Jiang Assignee: Stephen Yuan Jiang Labels: reliability Attachments: AlterColumnFamilyPV2-draft.v0-master.patch Original Estimate: 168h Time Spent: 72h Remaining Estimate: 96h master side, part of HBASE-12439 starts up the procedure executor on the master and replaces the add/modify/delete handlers with the procedure version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13209) Procedure V2 - master Add/Modify/Delete Column Family
[ https://issues.apache.org/jira/browse/HBASE-13209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-13209: --- Attachment: (was: AlterColumnFamilyServerPV2-draft.v0-master.patch) Procedure V2 - master Add/Modify/Delete Column Family - Key: HBASE-13209 URL: https://issues.apache.org/jira/browse/HBASE-13209 Project: HBase Issue Type: Sub-task Components: master Affects Versions: 2.0.0, 1.1.0 Reporter: Stephen Yuan Jiang Assignee: Stephen Yuan Jiang Labels: reliability Attachments: AlterColumnFamilyPV2-draft.v0-master.patch Original Estimate: 168h Time Spent: 72h Remaining Estimate: 96h master side, part of HBASE-12439 starts up the procedure executor on the master and replaces the add/modify/delete handlers with the procedure version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13210) Procedure V2 - master Modify table
[ https://issues.apache.org/jira/browse/HBASE-13210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-13210: --- Attachment: ModifyTableProcedureServerPV2-draft.v0-master.patch Procedure V2 - master Modify table -- Key: HBASE-13210 URL: https://issues.apache.org/jira/browse/HBASE-13210 Project: HBase Issue Type: Sub-task Components: master Affects Versions: 2.0.0, 1.1.0 Reporter: Stephen Yuan Jiang Assignee: Stephen Yuan Jiang Labels: reliablity Attachments: ModifyTableProcedureServerPV2-draft.v0-master.patch Original Estimate: 72h Time Spent: 48h Remaining Estimate: 24h master side, part of HBASE-12439 starts up the procedure executor on the master and replaces the modify table handlers with the procedure version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13213) Split out locality metrics among primary and secondary region
[ https://issues.apache.org/jira/browse/HBASE-13213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-13213: --- Attachment: locality-metrics-4.txt Split out locality metrics among primary and secondary region - Key: HBASE-13213 URL: https://issues.apache.org/jira/browse/HBASE-13213 Project: HBase Issue Type: Sub-task Reporter: Ted Yu Assignee: Ted Yu Attachments: 13213-v1.patch, locality-metrics-4.txt This task provides the ability to track locality of primary and secondary region replicas. We already have percentFilesLocal metric. There should be two sets of metrics - one for primaries and another for secondary / tertiary regions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12975) Supportable SplitTransaction and RegionMergeTransaction interfaces
[ https://issues.apache.org/jira/browse/HBASE-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387170#comment-14387170 ] Andrew Purtell commented on HBASE-12975: This is held up by test failures in the branch-1 backport. I will put up new master and branch-1 patches here for Jenkins runs when ready. Supportable SplitTransaction and RegionMergeTransaction interfaces -- Key: HBASE-12975 URL: https://issues.apache.org/jira/browse/HBASE-12975 Project: HBase Issue Type: Improvement Reporter: Rajeshbabu Chintaguntla Assignee: Andrew Purtell Fix For: 2.0.0, 1.1.0 Attachments: HBASE-12975.patch, HBASE-12975.patch, HBASE-12975.patch Making SplitTransaction, RegionMergeTransaction limited private is required to support local indexing feature in Phoenix to ensure regions colocation. We can ensure region split, regions merge in the coprocessors in few method calls without touching internals like creating zk's, file layout changes or assignments. 1) stepsBeforePONR, stepsAfterPONR we can ensure split. 2) meta entries can pass through coprocessors to atomically update with the normal split/merge. 3) rollback on failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12972) Region, a supportable public/evolving subset of HRegion
[
https://issues.apache.org/jira/browse/HBASE-12972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387169#comment-14387169
]
Andrew Purtell commented on HBASE-12972:
I was held up with test failures in branch-1 on HBASE-12975, but there are no
problems with this change on its own. I need to fix up for new commits to
trunk. Let me put up master and branch-1 patches here for Jenkins and then
proceed.
Region, a supportable public/evolving subset of HRegion
---
Key: HBASE-12972
URL: https://issues.apache.org/jira/browse/HBASE-12972
Project: HBase
Issue Type: New Feature
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-12972-0.98.patch, HBASE-12972.patch,
HBASE-12972.patch, HBASE-12972.patch, HBASE-12972.patch
On HBASE-12566, [~lhofhansl] proposed:
{quote}
Maybe we can have a {{Region}} interface that is to {{HRegion}} is what
{{Store}} is to {{HStore}}. Store marked with {{@InterfaceAudience.Private}}
but used in some coprocessor hooks.
{quote}
By example, now coprocessors have to reach into HRegion in order to
participate in row and region locking protocols, this is one area where the
functionality is legitimate for coprocessors but not for users, so an
in-between interface make sense.
In addition we should promote {{Store}}'s interface audience to
LimitedPrivate(COPROC).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13364) Make using the default javac on by default
[ https://issues.apache.org/jira/browse/HBASE-13364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387352#comment-14387352 ] Elliott Clark commented on HBASE-13364: --- Either that or turn it off by default yeah. Thoughts? Make using the default javac on by default -- Key: HBASE-13364 URL: https://issues.apache.org/jira/browse/HBASE-13364 Project: HBase Issue Type: Bug Reporter: Elliott Clark Assignee: Elliott Clark Errorprone doesn't work with java 8 and java 8 is becoming more and more standard everywhere. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13317) Region server reportForDuty stuck looping if there is a master change
[
https://issues.apache.org/jira/browse/HBASE-13317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387189#comment-14387189
]
Enis Soztutar commented on HBASE-13317:
---
bq. Do you want me to clean up this in this JIRA?
I think it is fine either way. The patch should work from my reading.
Region server reportForDuty stuck looping if there is a master change
-
Key: HBASE-13317
URL: https://issues.apache.org/jira/browse/HBASE-13317
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 1.0.0, 2.0.0, 0.98.12
Reporter: Jerry He
Assignee: Jerry He
Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.13
Attachments: HBASE-13317-0.98-v2.patch, HBASE-13317-0.98-v3.patch,
HBASE-13317-0.98-v4.patch, HBASE-13317-0.98-v5.patch, HBASE-13317-0.98.patch,
HBASE-13317-master-v5.patch
During cluster startup, region server reportForDuty gets stuck looping if
there is a master change.
{noformat}
2015-03-22 11:15:16,186 INFO [regionserver60020] regionserver.HRegionServer:
reportForDuty to master=bigaperf274,6,1427045883965 with port=60020,
startcode=1427048115174
2015-03-22 11:15:16,272 WARN [regionserver60020] regionserver.HRegionServer:
error telling master we are up
com.google.protobuf.ServiceException: java.net.ConnectException: Connection
refused
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1678)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
at
org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8277)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2137)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:896)
at java.lang.Thread.run(Thread.java:745)
2015-03-22 11:15:16,274 WARN [regionserver60020] regionserver.HRegionServer:
reportForDuty failed; sleeping and then retrying.
2015-03-22 11:15:19,274 INFO [regionserver60020] regionserver.HRegionServer:
reportForDuty to master=bigaperf273,6,1427048108439 with port=60020,
startcode=1427048115174
2015-03-22 11:15:19,275 WARN [regionserver60020] regionserver.HRegionServer:
error telling master we are up
com.google.protobuf.ServiceException: java.net.ConnectException: Connection
refused
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1678)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
at
org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8277)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2137)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:896)
at java.lang.Thread.run(Thread.java:745)
2015-03-22 11:15:19,276 WARN [regionserver60020] regionserver.HRegionServer:
reportForDuty failed; sleeping and then retrying.
2015-03-22 11:15:22,276 INFO [regionserver60020] regionserver.HRegionServer:
reportForDuty to master=bigaperf273,6,1427048108439 with port=60020,
startcode=1427048115174
2015-03-22 11:15:22,296 DEBUG [regionserver60020] regionserver.HRegionServer:
Master is not running yet
2015-03-22 11:15:22,296 WARN [regionserver60020] regionserver.HRegionServer:
reportForDuty failed; sleeping and then retrying.
2015-03-22 11:15:25,296 INFO [regionserver60020] regionserver.HRegionServer:
reportForDuty to master=bigaperf273,6,1427048108439 with port=60020,
startcode=1427048115174
2015-03-22 11:15:25,299 DEBUG [regionserver60020] regionserver.HRegionServer:
Master is not running yet
2015-03-22 11:15:25,299 WARN [regionserver60020] regionserver.HRegionServer:
reportForDuty failed; sleeping and then retrying.
2015-03-22 11:15:28,299 INFO [regionserver60020] regionserver.HRegionServer:
reportForDuty to master=bigaperf273,6,1427048108439 with port=60020,
startcode=1427048115174
2015-03-22 11:15:28,302 DEBUG [regionserver60020] regionserver.HRegionServer:
Master is not running yet
2015-03-22 11:15:28,302 WARN [regionserver60020] regionserver.HRegionServer:
reportForDuty failed; sleeping and then retrying.
{noformat}
What happended is the region server first got
master=bigaperf274,6,1427045883965. Before it was able to report
successfully, the maser changed to bigaperf273,6,1427048108439.
We were supposed to open a new connection to the new master. But we never
did,
[jira] [Commented] (HBASE-12972) Region, a supportable public/evolving subset of HRegion
[
https://issues.apache.org/jira/browse/HBASE-12972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387249#comment-14387249
]
Andrew Purtell commented on HBASE-12972:
Attached refreshed patches
Region, a supportable public/evolving subset of HRegion
---
Key: HBASE-12972
URL: https://issues.apache.org/jira/browse/HBASE-12972
Project: HBase
Issue Type: New Feature
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-12972-0.98.patch, HBASE-12972-branch-1.patch,
HBASE-12972.patch, HBASE-12972.patch, HBASE-12972.patch, HBASE-12972.patch,
HBASE-12972.patch
On HBASE-12566, [~lhofhansl] proposed:
{quote}
Maybe we can have a {{Region}} interface that is to {{HRegion}} is what
{{Store}} is to {{HStore}}. Store marked with {{@InterfaceAudience.Private}}
but used in some coprocessor hooks.
{quote}
By example, now coprocessors have to reach into HRegion in order to
participate in row and region locking protocols, this is one area where the
functionality is legitimate for coprocessors but not for users, so an
in-between interface make sense.
In addition we should promote {{Store}}'s interface audience to
LimitedPrivate(COPROC).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12972) Region, a supportable public/evolving subset of HRegion
[
https://issues.apache.org/jira/browse/HBASE-12972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-12972:
---
Attachment: HBASE-12972-branch-1.patch
HBASE-12972.patch
Region, a supportable public/evolving subset of HRegion
---
Key: HBASE-12972
URL: https://issues.apache.org/jira/browse/HBASE-12972
Project: HBase
Issue Type: New Feature
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-12972-0.98.patch, HBASE-12972-branch-1.patch,
HBASE-12972.patch, HBASE-12972.patch, HBASE-12972.patch, HBASE-12972.patch,
HBASE-12972.patch
On HBASE-12566, [~lhofhansl] proposed:
{quote}
Maybe we can have a {{Region}} interface that is to {{HRegion}} is what
{{Store}} is to {{HStore}}. Store marked with {{@InterfaceAudience.Private}}
but used in some coprocessor hooks.
{quote}
By example, now coprocessors have to reach into HRegion in order to
participate in row and region locking protocols, this is one area where the
functionality is legitimate for coprocessors but not for users, so an
in-between interface make sense.
In addition we should promote {{Store}}'s interface audience to
LimitedPrivate(COPROC).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13213) Split out locality metrics among primary and secondary region
[
https://issues.apache.org/jira/browse/HBASE-13213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387314#comment-14387314
]
Hadoop QA commented on HBASE-13213:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12708197/locality-metrics-4.txt
against master branch at commit 3815a33e3400cee73f63e205afac8f1a2d9c174f.
ATTACHMENT ID: 12708197
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1928 checkstyle errors (more than the master's current 1927 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13487//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13487//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13487//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13487//console
This message is automatically generated.
Split out locality metrics among primary and secondary region
-
Key: HBASE-13213
URL: https://issues.apache.org/jira/browse/HBASE-13213
Project: HBase
Issue Type: Sub-task
Reporter: Ted Yu
Assignee: Ted Yu
Attachments: 13213-v1.patch, locality-metrics-4.txt
This task provides the ability to track locality of primary and secondary
region replicas.
We already have percentFilesLocal metric.
There should be two sets of metrics - one for primaries and another for
secondary / tertiary regions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-13364) Make using the default javac on by default
Elliott Clark created HBASE-13364: - Summary: Make using the default javac on by default Key: HBASE-13364 URL: https://issues.apache.org/jira/browse/HBASE-13364 Project: HBase Issue Type: Bug Reporter: Elliott Clark Assignee: Elliott Clark Errorprone doesn't work with java 8 and java 8 is becoming more and more standard everywhere. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-8410) Basic quota support for namespaces
[
https://issues.apache.org/jira/browse/HBASE-8410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-8410:
-
Release Note:
Namespace auditor provides basic quota support for namespaces in terms of
number of tables and number of regions. In order to use namespace quotas, quota
support must be enabled by setting
hbase.quota.enabled property to true in hbase-site.xml file.
The users can add quota information to namespace, while creating new namespaces
or by altering existing ones.
Examples:
1. create_namespace 'ns1', {'hbase.namespace.quota.maxregions'='10'}
2. create_namespace 'ns2',
{'hbase.namespace.quota.maxtables'='2','hbase.namespace.quota.maxregions'='5'}
3. alter_namespace 'ns3', {METHOD = 'set',
'hbase.namespace.quota.maxtables'='5','hbase.namespace.quota.maxregions'='25'}
The quotas can be modified/added to namespace at any point of time. To remove
quotas, the following command can be used:
alter_namespace 'ns3', {METHOD = 'unset', 'hbase.namespace.quota.maxtables'}
alter_namespace 'ns3', {METHOD = 'unset', 'hbase.namespace.quota.maxregions'}
was:
Namespace auditor provides basic quota support for namespaces in terms of
number of tables and number of regions. In order to use namespace quotas, quota
support must be enabled by setting
hbase.quota.enabled property to true in hbase-site.xml file.
The users can add quota information to namespace, while creating new namespaces
or by altering existing ones.
Examples:
1. create_namespace 'ns1', {'hbase.namespace.quota.maxregions'='10'}
2. create_namespace 'ns2',
{'hbase.namespace.quota.maxtables'='2','hbase.namespace.quota.maxregion'='5'}
3. alter_namespace 'ns3', {METHOD = 'set',
'hbase.namespace.quota.maxtables'='5','hbase.namespace.quota.maxregion'='25'}
The quotas can be modified/added to namespace at any point of time. To remove
quotas, the following command can be used:
alter_namespace 'ns3', {METHOD = 'unset', 'hbase.namespace.quota.maxtables'}
alter_namespace 'ns3', {METHOD = 'unset', 'hbase.namespace.quota.maxregions'}
Basic quota support for namespaces
--
Key: HBASE-8410
URL: https://issues.apache.org/jira/browse/HBASE-8410
Project: HBase
Issue Type: Sub-task
Reporter: Francis Liu
Assignee: Vandana Ayyalasomayajula
Fix For: 2.0.0
Attachments: 8410-master.9.patch, HBASE-8410-master.1.patch,
HBASE-8410-master.4.patch, HBASE-8410-master.5.patch,
HBASE-8410-master.6.patch, HBASE-8410-master.7.patch,
HBASE-8410-master.9.patch, HBASE-8410-master.patch, HBASE-8410.docx,
HBASE-8410_trunk_14.patch
This task involves creating an observer which provides basic quota support to
namespaces in terms of (1) number of tables and (2) number of regions. The
quota support can be enabled by setting:
property
namehbase.coprocessor.region.classes/name
valueorg.apache.hadoop.hbase.namespace.NamespaceController/value
/property
property
namehbase.coprocessor.master.classes/name
valueorg.apache.hadoop.hbase.namespace.NamespaceController/value
/property
in the hbase-site.xml.
To add quotas to namespace, while creating namespace properties need to be
added.
Examples:
1. namespace_create 'ns1', {'hbase.namespace.quota.maxregion'='10'}
2. 1. namespace_create 'ns2', {'hbase.namespace.quota.maxtables'='2'},
{'hbase.namespace.quota.maxregion'='5'}
The quotas can be modified/added to namespace at any point of time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-8410) Basic quota support for namespaces
[
https://issues.apache.org/jira/browse/HBASE-8410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-8410:
-
Release Note:
Namespace auditor provides basic quota support for namespaces in terms of
number of tables and number of regions. In order to use namespace quotas, quota
support must be enabled by setting
hbase.quota.enabled property to true in hbase-site.xml file.
The users can add quota information to namespace, while creating new namespaces
or by altering existing ones.
Examples:
1. create_namespace 'ns1', {'hbase.namespace.quota.maxregions'='10'}
2. create_namespace 'ns2',
{'hbase.namespace.quota.maxtables'='2','hbase.namespace.quota.maxregion'='5'}
3. alter_namespace 'ns3', {METHOD = 'set',
'hbase.namespace.quota.maxtables'='5','hbase.namespace.quota.maxregion'='25'}
The quotas can be modified/added to namespace at any point of time. To remove
quotas, the following command can be used:
alter_namespace 'ns3', {METHOD = 'unset', 'hbase.namespace.quota.maxtables'}
alter_namespace 'ns3', {METHOD = 'unset', 'hbase.namespace.quota.maxregions'}
was:
Namespace auditor provides basic quota support for namespaces in terms of
number of tables and number of regions. In order to use namespace quotas, quota
support must be enabled by setting
hbase.quota.enabled property to true in hbase-site.xml file.
The users can add quota information to namespace, while creating new namespaces
or by altering existing ones.
Examples:
1. namespace_create 'ns1', {'hbase.namespace.quota.maxregions'='10'}
2. namespace_create 'ns2',
{'hbase.namespace.quota.maxtables'='2','hbase.namespace.quota.maxregion'='5'}
3. alter_namespace 'ns3', {METHOD = 'set',
'hbase.namespace.quota.maxtables'='5','hbase.namespace.quota.maxregion'='25'}
The quotas can be modified/added to namespace at any point of time.
Basic quota support for namespaces
--
Key: HBASE-8410
URL: https://issues.apache.org/jira/browse/HBASE-8410
Project: HBase
Issue Type: Sub-task
Reporter: Francis Liu
Assignee: Vandana Ayyalasomayajula
Fix For: 2.0.0
Attachments: 8410-master.9.patch, HBASE-8410-master.1.patch,
HBASE-8410-master.4.patch, HBASE-8410-master.5.patch,
HBASE-8410-master.6.patch, HBASE-8410-master.7.patch,
HBASE-8410-master.9.patch, HBASE-8410-master.patch, HBASE-8410.docx,
HBASE-8410_trunk_14.patch
This task involves creating an observer which provides basic quota support to
namespaces in terms of (1) number of tables and (2) number of regions. The
quota support can be enabled by setting:
property
namehbase.coprocessor.region.classes/name
valueorg.apache.hadoop.hbase.namespace.NamespaceController/value
/property
property
namehbase.coprocessor.master.classes/name
valueorg.apache.hadoop.hbase.namespace.NamespaceController/value
/property
in the hbase-site.xml.
To add quotas to namespace, while creating namespace properties need to be
added.
Examples:
1. namespace_create 'ns1', {'hbase.namespace.quota.maxregion'='10'}
2. 1. namespace_create 'ns2', {'hbase.namespace.quota.maxtables'='2'},
{'hbase.namespace.quota.maxregion'='5'}
The quotas can be modified/added to namespace at any point of time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13364) Make using the default javac on by default
[ https://issues.apache.org/jira/browse/HBASE-13364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387336#comment-14387336 ] Andrew Purtell commented on HBASE-13364: We didn't go anywhere with it, unfortunately. Want to revert? Make using the default javac on by default -- Key: HBASE-13364 URL: https://issues.apache.org/jira/browse/HBASE-13364 Project: HBase Issue Type: Bug Reporter: Elliott Clark Assignee: Elliott Clark Errorprone doesn't work with java 8 and java 8 is becoming more and more standard everywhere. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12891) Parallel execution for Hbck checkRegionConsistency
[ https://issues.apache.org/jira/browse/HBASE-12891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Latham updated HBASE-12891: Attachment: HBASE-12891-v3.patch Thanks, Stephen, for jumping in, identifying the concurrency problem and the new patch. Churro is out right now and we've been up to some other things. I reviewed the patch, and I'd like to propose an alternative that separates the lazy load of regionsFromMeta into a synchronized method and leaves use of it alone. I've attached that as HBASE-12891-v3.patch Parallel execution for Hbck checkRegionConsistency -- Key: HBASE-12891 URL: https://issues.apache.org/jira/browse/HBASE-12891 Project: HBase Issue Type: Improvement Components: hbck Affects Versions: 2.0.0, 0.98.10, 1.1.0 Reporter: churro morales Assignee: Stephen Yuan Jiang Labels: performance, scalability Fix For: 2.0.0, 1.1.0, 0.98.13 Attachments: HBASE-12891-v1.patch, HBASE-12891-v3.patch, HBASE-12891.98.patch, HBASE-12891.patch, HBASE-12891.patch, HBASE-12891.v2-branch-1.patch, HBASE-12891.v2-master.patch, hbase-12891-addendum1.patch We have a lot of regions on our cluster ~500k and noticed that hbck took quite some time in checkAndFixConsistency(). [~davelatham] patched our cluster to do this check in parallel to speed things up. I'll attach the patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-8410) Basic quota support for namespaces
[
https://issues.apache.org/jira/browse/HBASE-8410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387220#comment-14387220
]
Enis Soztutar commented on HBASE-8410:
--
Thanks Ashish. I've edited the release notes.
Basic quota support for namespaces
--
Key: HBASE-8410
URL: https://issues.apache.org/jira/browse/HBASE-8410
Project: HBase
Issue Type: Sub-task
Reporter: Francis Liu
Assignee: Vandana Ayyalasomayajula
Fix For: 2.0.0
Attachments: 8410-master.9.patch, HBASE-8410-master.1.patch,
HBASE-8410-master.4.patch, HBASE-8410-master.5.patch,
HBASE-8410-master.6.patch, HBASE-8410-master.7.patch,
HBASE-8410-master.9.patch, HBASE-8410-master.patch, HBASE-8410.docx,
HBASE-8410_trunk_14.patch
This task involves creating an observer which provides basic quota support to
namespaces in terms of (1) number of tables and (2) number of regions. The
quota support can be enabled by setting:
property
namehbase.coprocessor.region.classes/name
valueorg.apache.hadoop.hbase.namespace.NamespaceController/value
/property
property
namehbase.coprocessor.master.classes/name
valueorg.apache.hadoop.hbase.namespace.NamespaceController/value
/property
in the hbase-site.xml.
To add quotas to namespace, while creating namespace properties need to be
added.
Examples:
1. namespace_create 'ns1', {'hbase.namespace.quota.maxregion'='10'}
2. 1. namespace_create 'ns2', {'hbase.namespace.quota.maxtables'='2'},
{'hbase.namespace.quota.maxregion'='5'}
The quotas can be modified/added to namespace at any point of time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-11544) [Ergonomics] hbase.client.scanner.caching is dogged and will try to return batch even if it means OOME
[ https://issues.apache.org/jira/browse/HBASE-11544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Lawlor updated HBASE-11544: Attachment: HBASE-11544-addendum-v2.patch Attaching a rebased version of the patch since recent changes on master prevented a clean apply. Anyone have any thoughts on how ScannerContexts fits into the scanner RPC workflow? Questions, ideas for improvement, alternative approaches? [Ergonomics] hbase.client.scanner.caching is dogged and will try to return batch even if it means OOME -- Key: HBASE-11544 URL: https://issues.apache.org/jira/browse/HBASE-11544 Project: HBase Issue Type: Bug Reporter: stack Assignee: Jonathan Lawlor Priority: Critical Fix For: 2.0.0, 1.1.0 Attachments: Allocation_Hot_Spots.html, HBASE-11544-addendum-v1.patch, HBASE-11544-addendum-v2.patch, HBASE-11544-branch_1_0-v1.patch, HBASE-11544-branch_1_0-v2.patch, HBASE-11544-v1.patch, HBASE-11544-v2.patch, HBASE-11544-v3.patch, HBASE-11544-v4.patch, HBASE-11544-v5.patch, HBASE-11544-v6.patch, HBASE-11544-v6.patch, HBASE-11544-v6.patch, HBASE-11544-v7.patch, HBASE-11544-v8-branch-1.patch, HBASE-11544-v8.patch, gc.j.png, h.png, hits.j.png, m.png, mean.png, net.j.png, q (2).png Running some tests, I set hbase.client.scanner.caching=1000. Dataset has large cells. I kept OOME'ing. Serverside, we should measure how much we've accumulated and return to the client whatever we've gathered once we pass out a certain size threshold rather than keep accumulating till we OOME. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13352) Add hbase.import.version to Import usage.
[ https://issues.apache.org/jira/browse/HBASE-13352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387255#comment-14387255 ] Enis Soztutar commented on HBASE-13352: --- Import can also filter records, in which case input records will not be equal to output records. So input != output is not necessarily a problem. We can expand on the warning, saying maybe some have been filtered out if configured. Alternatively, we can also add a counter for filtered out records, and do the math. Other than that +1. Add hbase.import.version to Import usage. - Key: HBASE-13352 URL: https://issues.apache.org/jira/browse/HBASE-13352 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Priority: Trivial Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.13 Attachments: 13352-v2.txt, 13352.txt We just tried to export some (small amount of) data out of an 0.94 cluster to 0.98 cluster. We used Export/Import for that. By default we found that the import M/R job correctly reports the number of records seen, but _silently_ does not import anything. After looking at the 0.98 it's obvious there's an hbase.import.version (-Dhbase.import.version=0.94) to make this work. Two issues: # -Dhbase.import.version=0.94 should be show with the the Import.usage # If not given it should not just silently not import anything In this issue I'll just a trivially add this option to the Import tool's usage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12967) Invalid FQCNs in alter table command leaves the table unusable
[ https://issues.apache.org/jira/browse/HBASE-12967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-12967: -- Fix Version/s: (was: 1.0.1) 1.0.2 Invalid FQCNs in alter table command leaves the table unusable -- Key: HBASE-12967 URL: https://issues.apache.org/jira/browse/HBASE-12967 Project: HBase Issue Type: Bug Reporter: ramkrishna.s.vasudevan Priority: Critical Fix For: 2.0.0, 1.1.0, 0.98.13, 1.0.2 Refer to this thread http://osdir.com/ml/general/2015-02/msg03547.html A user tries to alter a table with a new split policy. Due to an invalid classname the table does not get enabled and the table becomes unusable. I think Procedure V2 is a long term soln for this but I think we atleast need to provide a work around or a set of steps to come out of this. Any fix before Procedure V2 comes into place would useful for the already released versions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13078) IntegrationTestSendTraceRequests is a noop
[ https://issues.apache.org/jira/browse/HBASE-13078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-13078: -- Fix Version/s: (was: 1.0.1) 1.0.2 IntegrationTestSendTraceRequests is a noop -- Key: HBASE-13078 URL: https://issues.apache.org/jira/browse/HBASE-13078 Project: HBase Issue Type: Test Components: integration tests Reporter: Nick Dimiduk Priority: Critical Fix For: 2.0.0, 1.1.0, 0.98.13, 1.0.2 While pair-debugging with [~jeffreyz] on HBASE-13077, we noticed that IntegrationTestSendTraceRequests doesn't actually assert anything. This test should be converted to use a mini cluster, setup a POJOSpanReceiver, and then verify the spans collected. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12891) Parallel execution for Hbck checkRegionConsistency
[ https://issues.apache.org/jira/browse/HBASE-12891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Latham updated HBASE-12891: Attachment: HBASE-12891-v4.patch And here's v4 without the duplicate import statements. Parallel execution for Hbck checkRegionConsistency -- Key: HBASE-12891 URL: https://issues.apache.org/jira/browse/HBASE-12891 Project: HBase Issue Type: Improvement Components: hbck Affects Versions: 2.0.0, 0.98.10, 1.1.0 Reporter: churro morales Assignee: Stephen Yuan Jiang Labels: performance, scalability Fix For: 2.0.0, 1.1.0, 0.98.13 Attachments: HBASE-12891-v1.patch, HBASE-12891-v3.patch, HBASE-12891-v4.patch, HBASE-12891.98.patch, HBASE-12891.patch, HBASE-12891.patch, HBASE-12891.v2-branch-1.patch, HBASE-12891.v2-master.patch, hbase-12891-addendum1.patch We have a lot of regions on our cluster ~500k and noticed that hbck took quite some time in checkAndFixConsistency(). [~davelatham] patched our cluster to do this check in parallel to speed things up. I'll attach the patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13213) Split out locality metrics among primary and secondary region
[ https://issues.apache.org/jira/browse/HBASE-13213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-13213: --- Attachment: 13213-v4.txt Split out locality metrics among primary and secondary region - Key: HBASE-13213 URL: https://issues.apache.org/jira/browse/HBASE-13213 Project: HBase Issue Type: Sub-task Reporter: Ted Yu Assignee: Ted Yu Attachments: 13213-v1.patch, 13213-v4.txt, locality-metrics-4.txt This task provides the ability to track locality of primary and secondary region replicas. We already have percentFilesLocal metric. There should be two sets of metrics - one for primaries and another for secondary / tertiary regions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-9738) Delete table and loadbalancer interference
[
https://issues.apache.org/jira/browse/HBASE-9738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj Das resolved HBASE-9738.
Resolution: Fixed
Thanks for checking [~mantonov]. I think this got fixed by HBASE-10349.
Delete table and loadbalancer interference
--
Key: HBASE-9738
URL: https://issues.apache.org/jira/browse/HBASE-9738
Project: HBase
Issue Type: Bug
Reporter: Devaraj Das
Priority: Critical
Fix For: 2.0.0, 1.1.0
I have noticed that when the balancer is computing a plan for region moves,
and a delete table is issued, there is some interference.
1. At time t1, user deleted the table.
2. This led to the master updating the meta table to remove the line for the
regioninfo for a region f2a9e2e9d70894c03f54ee5902bebee6.
{noformat}
2013-10-04 08:42:52,495 INFO [MASTER_TABLE_OPERATIONS-hor15n05:6-0]
catalog.MetaEditor: Deleted [{ENCODED = f2a9e2e9d70894c03f54ee5902bebee6,
NAME = 'usertable,,1380876170581.f2a9e2e9d70894c03f54ee5902bebee6.',
STARTKEY = '', ENDKEY = ''}]
{noformat}
3. However around the same time, the balancer kicked in, and reassigned the
region and made it online somewhere. It didn't check the fact (nor anyone
else did) that the table was indeed deleted.
{noformat}
2013-10-04 08:42:53,215 INFO
[hor15n05.gq1.ygridcore.net,6,1380869262259-BalancerChore]
master.HMaster: balance
hri=usertable,,1380876170581.f2a9e2e9d70894c03f54ee5902bebee6.,
src=hor15n09.gq1.ygridcore.net,60020,1380869263722,
dest=hor15n11.gq1.ygridcore.net,60020,1380869263682
{noformat}
.
{noformat}
2013-10-04 08:42:53,592 INFO [AM.ZK.Worker-pool2-t829] master.RegionStates:
Onlined f2a9e2e9d70894c03f54ee5902bebee6 on
hor15n11.gq1.ygridcore.net,60020,1380869263682
{noformat}
4. Henceforth, all the drop tables started giving warnings like
{noformat}
2013-10-04 08:45:17,587 INFO [RpcServer.handler=8,port=6]
master.HMaster: Client=hrt_qa//68.142.246.151 delete usertable
2013-10-04 08:45:17,631 DEBUG [RpcServer.handler=8,port=6]
lock.ZKInterProcessLockBase: Acquired a lock for
/hbase/table-lock/usertable/write-master:600
2013-10-04 08:45:17,637 WARN [RpcServer.handler=8,port=6]
catalog.MetaReader: No serialized HRegionInfo in
keyvalues={usertable,,1380876170581.f2a9e2e9d70894c03f54ee5902bebee6./info:seqnumDuringOpen/1380876173509/Put/vlen=8/mvcc=0,
usertable,,1380876170581.f2a9e2e9d70894c03f54ee5902bebee6./info:server/1380876173509/Put/vlen=32/mvcc=0,
usertable,,1380876170581.f2a9e2e9d70894c03f54ee5902bebee6./info:serverstartcode/1380876173509/Put/vlen=8/mvcc=0}
{noformat}
5. The create of the same table also fails since there is still state
(reincarnated, maybe) about the table in the master.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13335) Update ClientSmallScanner and ClientSmallReversedScanner
[ https://issues.apache.org/jira/browse/HBASE-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-13335: --- Fix Version/s: 1.0.1 Update ClientSmallScanner and ClientSmallReversedScanner Key: HBASE-13335 URL: https://issues.apache.org/jira/browse/HBASE-13335 Project: HBase Issue Type: Sub-task Components: Client Reporter: Josh Elser Assignee: Josh Elser Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.13 Attachments: HBASE-13335.patch Some follow-on work for HBASE-13262: it's unlikely that clients using the small scanners would get enough data to run into the initial bug, but the scanner implementations should still adhere to the moreResultsInRegion flag when the server sends it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12891) Parallel execution for Hbck checkRegionConsistency
[
https://issues.apache.org/jira/browse/HBASE-12891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387616#comment-14387616
]
Hadoop QA commented on HBASE-12891:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12708237/HBASE-12891-v4.patch
against master branch at commit 3815a33e3400cee73f63e205afac8f1a2d9c174f.
ATTACHMENT ID: 12708237
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13491//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13491//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13491//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13491//console
This message is automatically generated.
Parallel execution for Hbck checkRegionConsistency
--
Key: HBASE-12891
URL: https://issues.apache.org/jira/browse/HBASE-12891
Project: HBase
Issue Type: Improvement
Components: hbck
Affects Versions: 2.0.0, 0.98.10, 1.1.0
Reporter: churro morales
Assignee: Stephen Yuan Jiang
Labels: performance, scalability
Fix For: 2.0.0, 1.1.0, 0.98.13
Attachments: HBASE-12891-v1.patch, HBASE-12891-v3.patch,
HBASE-12891-v4.patch, HBASE-12891.98.patch, HBASE-12891.patch,
HBASE-12891.patch, HBASE-12891.v2-branch-1.patch,
HBASE-12891.v2-master.patch, hbase-12891-addendum1.patch
We have a lot of regions on our cluster ~500k and noticed that hbck took
quite some time in checkAndFixConsistency(). [~davelatham] patched our
cluster to do this check in parallel to speed things up. I'll attach the
patch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13362) set max result size from client only (like caching)?
[ https://issues.apache.org/jira/browse/HBASE-13362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387615#comment-14387615 ] Enis Soztutar commented on HBASE-13362: --- Agreed with Josh. We probably want to protect against misbehaving (or misconfigured) clients from the server side. A client with 10GB max size set might bring down the RS. So we cannot blindly trust the max size from the client I think. set max result size from client only (like caching)? Key: HBASE-13362 URL: https://issues.apache.org/jira/browse/HBASE-13362 Project: HBase Issue Type: Brainstorming Reporter: Lars Hofhansl With the recent problems we've been seeing client/server result size mismatch, I was thinking: Why was this not a problem with scanner caching? There are two reasons: # number of rows is easy to calculate (and we did it correctly) # caching is only controlled from the client, never set on the server alone We did fix both #1 and #2 in HBASE-13262. Still, I'd like to discuss the following: * default the client sent max result size to 2mb * remove any server only result sizing * continue to use hbase.client.scanner.max.result.size but enforce it via the client only (as the name implies anyway). Comments? Concerns? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13362) set max result size from client only (like caching)?
[ https://issues.apache.org/jira/browse/HBASE-13362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387618#comment-14387618 ] Enis Soztutar commented on HBASE-13362: --- But I'm all +1 for simplifying this. set max result size from client only (like caching)? Key: HBASE-13362 URL: https://issues.apache.org/jira/browse/HBASE-13362 Project: HBase Issue Type: Brainstorming Reporter: Lars Hofhansl With the recent problems we've been seeing client/server result size mismatch, I was thinking: Why was this not a problem with scanner caching? There are two reasons: # number of rows is easy to calculate (and we did it correctly) # caching is only controlled from the client, never set on the server alone We did fix both #1 and #2 in HBASE-13262. Still, I'd like to discuss the following: * default the client sent max result size to 2mb * remove any server only result sizing * continue to use hbase.client.scanner.max.result.size but enforce it via the client only (as the name implies anyway). Comments? Concerns? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12891) Parallel execution for Hbck checkRegionConsistency
[
https://issues.apache.org/jira/browse/HBASE-12891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387364#comment-14387364
]
Hadoop QA commented on HBASE-12891:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12708210/HBASE-12891-v3.patch
against master branch at commit 3815a33e3400cee73f63e205afac8f1a2d9c174f.
ATTACHMENT ID: 12708210
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1935 checkstyle errors (more than the master's current 1927 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13488//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13488//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13488//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13488//console
This message is automatically generated.
Parallel execution for Hbck checkRegionConsistency
--
Key: HBASE-12891
URL: https://issues.apache.org/jira/browse/HBASE-12891
Project: HBase
Issue Type: Improvement
Components: hbck
Affects Versions: 2.0.0, 0.98.10, 1.1.0
Reporter: churro morales
Assignee: Stephen Yuan Jiang
Labels: performance, scalability
Fix For: 2.0.0, 1.1.0, 0.98.13
Attachments: HBASE-12891-v1.patch, HBASE-12891-v3.patch,
HBASE-12891.98.patch, HBASE-12891.patch, HBASE-12891.patch,
HBASE-12891.v2-branch-1.patch, HBASE-12891.v2-master.patch,
hbase-12891-addendum1.patch
We have a lot of regions on our cluster ~500k and noticed that hbck took
quite some time in checkAndFixConsistency(). [~davelatham] patched our
cluster to do this check in parallel to speed things up. I'll attach the
patch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13364) Make using the default javac on by default
[ https://issues.apache.org/jira/browse/HBASE-13364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387363#comment-14387363 ] Andrew Purtell commented on HBASE-13364: The new version of error-prone supports Java 8: https://groups.google.com/forum/#!topic/error-prone-discuss/syl8dKP_Aow They're now on GitHub of course: https://github.com/google/error-prone Maybe we change this issue into a task to upgrade to their latest? Or we can drop it from the build. Depends if anyone sees it as useful. I think, potentially, it can be very useful, but especially if we develop our own custom analyses. See discussion on the original motivating issue. Make using the default javac on by default -- Key: HBASE-13364 URL: https://issues.apache.org/jira/browse/HBASE-13364 Project: HBase Issue Type: Bug Reporter: Elliott Clark Assignee: Elliott Clark Errorprone doesn't work with java 8 and java 8 is becoming more and more standard everywhere. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13364) Make using the default javac on by default
[ https://issues.apache.org/jira/browse/HBASE-13364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387427#comment-14387427 ] Elliott Clark commented on HBASE-13364: --- They stopped using the javac that's in JAVA_HOME. I think the tool is really useful for jenkins or for test runs but I don't think that it should be run by default for builds. I think of it like code coverage. Something useful but I don't want it running on a production build. Make using the default javac on by default -- Key: HBASE-13364 URL: https://issues.apache.org/jira/browse/HBASE-13364 Project: HBase Issue Type: Bug Reporter: Elliott Clark Assignee: Elliott Clark Errorprone doesn't work with java 8 and java 8 is becoming more and more standard everywhere. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-7105) RS throws NPE on forcing compaction from HBase shell on a single bulk imported file.
[
https://issues.apache.org/jira/browse/HBASE-7105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-7105:
-
Fix Version/s: (was: 1.0.1)
1.0.2
RS throws NPE on forcing compaction from HBase shell on a single bulk
imported file.
Key: HBASE-7105
URL: https://issues.apache.org/jira/browse/HBASE-7105
Project: HBase
Issue Type: Bug
Components: regionserver
Reporter: Karthik Ranganathan
Assignee: Cosmin Lehene
Fix For: 2.0.0, 1.1.0, 1.0.2
Attachments:
0001-HBASE-7105-RS-throws-NPE-on-forcing-compaction-from-.patch,
0001-HBASE-7105-RS-throws-NPE-on-forcing-compaction-from-.patch
In StoreFile, we have:
private AtomicBoolean majorCompaction = null;
In StoreFile.open(), we do:
b = metadataMap.get(MAJOR_COMPACTION_KEY);
if (b != null) {
// init majorCompaction variable
}
Because the file was bulk imported, this is never initialized. Any subsequent
call to isMajorCompaction() NPE's.
Fix is to initialize it to false.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13221) HDFS Transparent Encryption breaks WAL writing
[ https://issues.apache.org/jira/browse/HBASE-13221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-13221: -- Fix Version/s: (was: 1.0.1) 1.0.2 HDFS Transparent Encryption breaks WAL writing -- Key: HBASE-13221 URL: https://issues.apache.org/jira/browse/HBASE-13221 Project: HBase Issue Type: Bug Components: wal Affects Versions: 0.98.0, 1.0.0 Reporter: Sean Busbey Assignee: Sean Busbey Priority: Critical Fix For: 2.0.0, 1.1.0, 0.98.13, 1.0.2 We need to detect when HDFS Transparent Encryption (Hadoop 2.6.0+) is enabled and fall back to more synchronization in the WAL to prevent catastrophic failure under load. See HADOOP-11708 for more details. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12987) HBCK should print status while scanning over many regions
[ https://issues.apache.org/jira/browse/HBASE-12987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387487#comment-14387487 ] Mikhail Antonov commented on HBASE-12987: - That was my backburner which I never got to tackling..so feel free to grab, sure. Thanks Josh. HBCK should print status while scanning over many regions - Key: HBASE-12987 URL: https://issues.apache.org/jira/browse/HBASE-12987 Project: HBase Issue Type: Improvement Components: hbck, Usability Reporter: Nick Dimiduk Assignee: Mikhail Antonov Labels: beginner Running simple commands like hbck -summary on a large table can take some time. We should print some information to let it be known how things are progressing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13283) Document the steps for rolling back the security on hbase.
[
https://issues.apache.org/jira/browse/HBASE-13283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-13283:
--
Fix Version/s: (was: 1.0.1)
1.0.2
Document the steps for rolling back the security on hbase.
--
Key: HBASE-13283
URL: https://issues.apache.org/jira/browse/HBASE-13283
Project: HBase
Issue Type: Sub-task
Reporter: Srikanth Srungarapu
Fix For: 2.0.0, 1.1.0, 0.98.12, 1.0.2
Would be great to document the steps the user should follow to rollback the
security. The following are
* shutdown hbase
* run hbase clean up script with --cleanAcls introduced in HBASE-13162.
* remove from the hbase-site.xml conf
{code}
hbase.security.authentication
hbase.regionserver.kerberos.principal
hbase.regionserver.keytab.file
hbase.master.kerberos.principal
hbase.master.keytab.file
{code}
* Removing AccessController coprocessors from hbase-site.xml
* Address hbase.security.authorization based on outcome of HBASE-13275.
cc: [~misty]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13143) TestCacheOnWrite is flaky and needs a diet
[ https://issues.apache.org/jira/browse/HBASE-13143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-13143: -- Fix Version/s: (was: 1.0.1) 1.0.2 TestCacheOnWrite is flaky and needs a diet -- Key: HBASE-13143 URL: https://issues.apache.org/jira/browse/HBASE-13143 Project: HBase Issue Type: Bug Affects Versions: 0.98.11 Reporter: Andrew Purtell Priority: Critical Fix For: 2.0.0, 1.1.0, 0.98.13, 1.0.2 TestCacheOnWrite passes locally but has been flaking in 0.98 builds on Jenkins, most recently https://builds.apache.org/job/HBase-0.98/878/ The test takes a long time to execute (338.492 sec) and is resource intensive (216 tests). Neither of these characteristics endear it to Jenkins. When I ran this unit test on a macbook after a minute the fan was running so fast I thought it would take flight. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13335) Update ClientSmallScanner and ClientSmallReversedScanner
[ https://issues.apache.org/jira/browse/HBASE-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-13335: --- Attachment: HBASE-13335.patch First attempt at a patch to update ClientSmallScanner and ClientSmallReversedScanner. Fairly straightforward -- tests included. Update ClientSmallScanner and ClientSmallReversedScanner Key: HBASE-13335 URL: https://issues.apache.org/jira/browse/HBASE-13335 Project: HBase Issue Type: Sub-task Components: Client Reporter: Josh Elser Assignee: Josh Elser Fix For: 2.0.0, 1.1.0, 0.98.13 Attachments: HBASE-13335.patch Some follow-on work for HBASE-13262: it's unlikely that clients using the small scanners would get enough data to run into the initial bug, but the scanner implementations should still adhere to the moreResultsInRegion flag when the server sends it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13335) Update ClientSmallScanner and ClientSmallReversedScanner
[ https://issues.apache.org/jira/browse/HBASE-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-13335: --- Status: Patch Available (was: Open) Update ClientSmallScanner and ClientSmallReversedScanner Key: HBASE-13335 URL: https://issues.apache.org/jira/browse/HBASE-13335 Project: HBase Issue Type: Sub-task Components: Client Reporter: Josh Elser Assignee: Josh Elser Fix For: 2.0.0, 1.1.0, 0.98.13 Attachments: HBASE-13335.patch Some follow-on work for HBASE-13262: it's unlikely that clients using the small scanners would get enough data to run into the initial bug, but the scanner implementations should still adhere to the moreResultsInRegion flag when the server sends it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-9738) Delete table and loadbalancer interference
[
https://issues.apache.org/jira/browse/HBASE-9738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387489#comment-14387489
]
Mikhail Antonov commented on HBASE-9738:
Thanks [~devaraj].
Delete table and loadbalancer interference
--
Key: HBASE-9738
URL: https://issues.apache.org/jira/browse/HBASE-9738
Project: HBase
Issue Type: Bug
Reporter: Devaraj Das
Priority: Critical
Fix For: 2.0.0, 1.1.0
I have noticed that when the balancer is computing a plan for region moves,
and a delete table is issued, there is some interference.
1. At time t1, user deleted the table.
2. This led to the master updating the meta table to remove the line for the
regioninfo for a region f2a9e2e9d70894c03f54ee5902bebee6.
{noformat}
2013-10-04 08:42:52,495 INFO [MASTER_TABLE_OPERATIONS-hor15n05:6-0]
catalog.MetaEditor: Deleted [{ENCODED = f2a9e2e9d70894c03f54ee5902bebee6,
NAME = 'usertable,,1380876170581.f2a9e2e9d70894c03f54ee5902bebee6.',
STARTKEY = '', ENDKEY = ''}]
{noformat}
3. However around the same time, the balancer kicked in, and reassigned the
region and made it online somewhere. It didn't check the fact (nor anyone
else did) that the table was indeed deleted.
{noformat}
2013-10-04 08:42:53,215 INFO
[hor15n05.gq1.ygridcore.net,6,1380869262259-BalancerChore]
master.HMaster: balance
hri=usertable,,1380876170581.f2a9e2e9d70894c03f54ee5902bebee6.,
src=hor15n09.gq1.ygridcore.net,60020,1380869263722,
dest=hor15n11.gq1.ygridcore.net,60020,1380869263682
{noformat}
.
{noformat}
2013-10-04 08:42:53,592 INFO [AM.ZK.Worker-pool2-t829] master.RegionStates:
Onlined f2a9e2e9d70894c03f54ee5902bebee6 on
hor15n11.gq1.ygridcore.net,60020,1380869263682
{noformat}
4. Henceforth, all the drop tables started giving warnings like
{noformat}
2013-10-04 08:45:17,587 INFO [RpcServer.handler=8,port=6]
master.HMaster: Client=hrt_qa//68.142.246.151 delete usertable
2013-10-04 08:45:17,631 DEBUG [RpcServer.handler=8,port=6]
lock.ZKInterProcessLockBase: Acquired a lock for
/hbase/table-lock/usertable/write-master:600
2013-10-04 08:45:17,637 WARN [RpcServer.handler=8,port=6]
catalog.MetaReader: No serialized HRegionInfo in
keyvalues={usertable,,1380876170581.f2a9e2e9d70894c03f54ee5902bebee6./info:seqnumDuringOpen/1380876173509/Put/vlen=8/mvcc=0,
usertable,,1380876170581.f2a9e2e9d70894c03f54ee5902bebee6./info:server/1380876173509/Put/vlen=32/mvcc=0,
usertable,,1380876170581.f2a9e2e9d70894c03f54ee5902bebee6./info:serverstartcode/1380876173509/Put/vlen=8/mvcc=0}
{noformat}
5. The create of the same table also fails since there is still state
(reincarnated, maybe) about the table in the master.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12943) Set sun.net.inetaddr.ttl in HBase
[ https://issues.apache.org/jira/browse/HBASE-12943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387549#comment-14387549 ] Enis Soztutar commented on HBASE-12943: --- Now that the subtask went in that fixes the stub management, we can set the TTL by default in this jira. Set sun.net.inetaddr.ttl in HBase - Key: HBASE-12943 URL: https://issues.apache.org/jira/browse/HBASE-12943 Project: HBase Issue Type: Bug Reporter: Liu Shaohui Assignee: Liu Shaohui Fix For: 1.1.0 Attachments: 12943-1-master.txt The default value of config: sun.net.inetaddr.ttl is -1 and the java processes will cache the mapping of hostname to ip address forever, See: http://docs.oracle.com/javase/7/docs/technotes/guides/net/properties.html But things go wrong when a regionserver with same hostname and different ip address rejoins the hbase cluster. The HMaster will get wrong ip address of the regionserver from this cache and every region assignment to this regionserver will be blocked for a time because the HMaster can't communicate with the regionserver. A tradeoff is to set the sun.net.inetaddr.ttl to 10m or 1h and make the wrong cache expired. Suggestions are welcomed. Thanks~ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12954) Ability impaired using HBase on multihomed hosts
[
https://issues.apache.org/jira/browse/HBASE-12954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387574#comment-14387574
]
Enis Soztutar commented on HBASE-12954:
---
I'm +1 for v14 for master and branch-1 given that it has been tested (manually)
to some extend. We can add a release note about the new configuration settings
with an explicit note about this being an expert level configuration.
Ability impaired using HBase on multihomed hosts
Key: HBASE-12954
URL: https://issues.apache.org/jira/browse/HBASE-12954
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.4
Reporter: Clay B.
Assignee: Ted Yu
Priority: Minor
Attachments: 12954-v1.txt, 12954-v10.txt, 12954-v11.txt,
12954-v12.txt, 12954-v12.txt, 12954-v12.txt, 12954-v13.txt, 12954-v14.txt,
12954-v7.txt, 12954-v8.txt, Hadoop Three Interfaces.png
For HBase clusters running on unusual networks (such as NAT'd cloud
environments or physical machines with multiple IP's per network interface)
it would be ideal to have a way to both specify:
# which IP interface to which HBase master or region-server will bind
# what hostname HBase will advertise in Zookeeper both for a master or
region-server process
While efforts such as HBASE-8640 go a long way to normalize these two sources
of information, it is not possible in the current design of the properties
available to an administrator for these to be unambiguously specified.
One has been able to request {{hbase.master.ipc.address}} or
{{hbase.regionserver.ipc.address}} but one can not specify the desired HBase
{{hbase.master.hostname}}. (It was removed in HBASE-1357, further I am
unaware of a region-server equivalent.)
I use a configuration management system to generate all of my configuration
files on a per-machine basis. As such, an option to generate a file
specifying exactly which hostname to use would be helpful.
Today, specifying the bind address for HBase works and one can use an
HBase-only DNS for faking what to put in Zookeeper but this is far from
ideal. Network interfaces have no intrinsic IP address, nor hostname.
Specifing a DNS server is awkward as the DNS server may differ from the
system's resolver and is a single IP address. Similarly, on hosts which use a
transient VIP (e.g. through keepalived) for other services, it means there's
a seemingly non-deterministic hostname choice made by HBase depending on the
state of the VIP at daemon start-up time.
I will attach two networking examples I use which become very difficult to
manage under the current properties.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13335) Update ClientSmallScanner and ClientSmallReversedScanner
[ https://issues.apache.org/jira/browse/HBASE-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387625#comment-14387625 ] Enis Soztutar commented on HBASE-13335: --- Maybe get rid of RetryingCallableFactory, and only keep SmallScanRetryingCallableFactory and put it in ClientSmallScanner. Or we can make a ScannerCallableFactory as a base and put the base in AbstractClientScanner. Update ClientSmallScanner and ClientSmallReversedScanner Key: HBASE-13335 URL: https://issues.apache.org/jira/browse/HBASE-13335 Project: HBase Issue Type: Sub-task Components: Client Reporter: Josh Elser Assignee: Josh Elser Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.13 Attachments: HBASE-13335.patch Some follow-on work for HBASE-13262: it's unlikely that clients using the small scanners would get enough data to run into the initial bug, but the scanner implementations should still adhere to the moreResultsInRegion flag when the server sends it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13299) Add setReturnResults() to Increment, like Append has
[
https://issues.apache.org/jira/browse/HBASE-13299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-13299:
--
Fix Version/s: (was: 1.0.1)
1.1.0
2.0.0
Add setReturnResults() to Increment, like Append has
Key: HBASE-13299
URL: https://issues.apache.org/jira/browse/HBASE-13299
Project: HBase
Issue Type: Bug
Components: API
Affects Versions: 1.0.0
Reporter: Lars George
Assignee: Ashish Singhi
Priority: Critical
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-13299.patch
It seems like a good idea to keep Mutations similar, in that vein, the
{{{set|get}ReturnResults()}} method of Append seems like something Increment
could also benefit from, no?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13299) Add setReturnResults() to Increment, like Append has
[
https://issues.apache.org/jira/browse/HBASE-13299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387443#comment-14387443
]
Enis Soztutar commented on HBASE-13299:
---
As much as I like this, I think we should put it to only 1.1+ to be on the safe
side. What do you guys think. Other than that, patch LGTM.
Add setReturnResults() to Increment, like Append has
Key: HBASE-13299
URL: https://issues.apache.org/jira/browse/HBASE-13299
Project: HBase
Issue Type: Bug
Components: API
Affects Versions: 1.0.0
Reporter: Lars George
Assignee: Ashish Singhi
Priority: Critical
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-13299.patch
It seems like a good idea to keep Mutations similar, in that vein, the
{{{set|get}ReturnResults()}} method of Append seems like something Increment
could also benefit from, no?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12987) HBCK should print status while scanning over many regions
[ https://issues.apache.org/jira/browse/HBASE-12987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387472#comment-14387472 ] Josh Elser commented on HBASE-12987: You still planning on working this [~mantonov]? If not, might I grab it from ya? HBCK should print status while scanning over many regions - Key: HBASE-12987 URL: https://issues.apache.org/jira/browse/HBASE-12987 Project: HBase Issue Type: Improvement Components: hbck, Usability Reporter: Nick Dimiduk Assignee: Mikhail Antonov Labels: beginner Running simple commands like hbck -summary on a large table can take some time. We should print some information to let it be known how things are progressing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13291) Lift the scan ceiling
[ https://issues.apache.org/jira/browse/HBASE-13291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387468#comment-14387468 ] stack commented on HBASE-13291: --- # The vint mvcc on end of keyvalue is expensive to parse; we can't make use of the unsafe toLong which seems a good deal faster than get bytes and bit noodling going by micro-benchmarking (I see 10x diff doing unsafe #getLong over get byte and left shifting). This fact and the way we have tags tagged on end (perhaps) frustrates go-fast efforts. # Doing a getLong instead of two getInts and then bit noodling is faster than doing two getInts (this is using unsafe -- I saw almost 10x diff). Added it in. # Bulk of our time is in StoreFileScanner#_next # Next is ScanQueryMatcher#match. It is not being very smart. It uses the Cell Interface to pull out elements to compare only each getXXXoffset or getXXXlength starts over from scratch reparsing key, value, row, etc. lengths extracting ints so as to do some offset math. This is a killer.The worst offender is the call from #match to #isCellTTLExpired In here we check to see if tags are enabled. We do this by parsing key and value ints to figure end of KV to poke for tags. All scans are paying for that time when DLR sets an MVCC on a KV. # If lots of cells being returned, then we will be doing lots of array resizing. These show in the profilings. The HFileReaderV3 and then HFileReaderV2 with an abstract under this makes it hard to navigate what is going on. Need to flatten now we are up on v3 file format and all is on by default. Lift the scan ceiling - Key: HBASE-13291 URL: https://issues.apache.org/jira/browse/HBASE-13291 Project: HBase Issue Type: Improvement Components: Scanners Affects Versions: 1.0.0 Reporter: stack Assignee: stack Attachments: 13291.inlining.txt, Screen Shot 2015-03-26 at 12.12.13 PM.png, Screen Shot 2015-03-26 at 3.39.33 PM.png, hack_to_bypass_bb.txt, nonBBposAndInineMvccVint.txt, q (1).png, traces.7.svg, traces.filterall.svg, traces.nofilter.svg, traces.small2.svg, traces.smaller.svg Scanning medium sized rows with multiple concurrent scanners exhibits interesting 'ceiling' properties. A server runs at about 6.7k ops a second using 450% of possible 1600% of CPUs when 4 clients each with 10 threads doing scan 1000 rows. If I add '--filterAll' argument (do not return results), then we run at 1450% of possible 1600% possible but we do 8k ops a second. Let me attach flame graphs for two cases. Unfortunately, there is some frustrating dark art going on. Let me try figure it... Filing issue in meantime to keep score in. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13222) Provide means of non-destructive balancer inspection
[ https://issues.apache.org/jira/browse/HBASE-13222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387467#comment-14387467 ] Josh Elser commented on HBASE-13222: Thanks for the praise and taking a look (even belated), Nick! Provide means of non-destructive balancer inspection Key: HBASE-13222 URL: https://issues.apache.org/jira/browse/HBASE-13222 Project: HBase Issue Type: Improvement Components: Balancer Reporter: Nick Dimiduk Assignee: Josh Elser Priority: Minor Fix For: 2.0.0, 1.1.0 Attachments: HBASE-13222-v1.patch, HBASE-13222-v2.patch, HBASE-13222-v3.patch, HBASE-13222.00-0.98.patch, HBASE-13222.patch, master.jpg At least on 0.98 (haven't checked master, 1.0), it seems we don't have a means for accessing balancer status w.o changing it in the process. Here's a little script to tickle ZK and see what the flag is. Would be nice if this was available from the shell, on the master status page, or something similar. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5878) Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
[
https://issues.apache.org/jira/browse/HBASE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-5878:
-
Fix Version/s: (was: 1.0.1)
1.0.2
Use getVisibleLength public api from HdfsDataInputStream from Hadoop-2.
---
Key: HBASE-5878
URL: https://issues.apache.org/jira/browse/HBASE-5878
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Uma Maheswara Rao G
Assignee: Ashish Singhi
Fix For: 2.0.0, 1.1.0, 1.0.2
Attachments: HBASE-5878-v2.patch, HBASE-5878-v3.patch,
HBASE-5878-v4.patch, HBASE-5878.patch
SequencFileLogReader:
Currently Hbase using getFileLength api from DFSInputStream class by
reflection. DFSInputStream is not exposed as public. So, this may change in
future. Now HDFS exposed HdfsDataInputStream as public API.
We can make use of it, when we are not able to find the getFileLength api
from DFSInputStream as a else condition. So, that we will not have any sudden
surprise like we are facing today.
Also, it is just logging one warn message and proceeding if it throws any
exception while getting the length. I think we can re-throw the exception
because there is no point in continuing with dataloss.
{code}
long adjust = 0;
try {
Field fIn = FilterInputStream.class.getDeclaredField(in);
fIn.setAccessible(true);
Object realIn = fIn.get(this.in);
// In hadoop 0.22, DFSInputStream is a standalone class. Before
this,
// it was an inner class of DFSClient.
if (realIn.getClass().getName().endsWith(DFSInputStream)) {
Method getFileLength = realIn.getClass().
getDeclaredMethod(getFileLength, new Class? []{});
getFileLength.setAccessible(true);
long realLength = ((Long)getFileLength.
invoke(realIn, new Object []{})).longValue();
assert(realLength = this.length);
adjust = realLength - this.length;
} else {
LOG.info(Input stream class: + realIn.getClass().getName() +
, not adjusting length);
}
} catch(Exception e) {
SequenceFileLogReader.LOG.warn(
Error while trying to get accurate file length. +
Truncation / data loss may occur if RegionServers die., e);
}
return adjust + super.getPos();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13356) HBase should provide an InputFormat supporting multiple scans in mapreduce jobs over snapshots
[ https://issues.apache.org/jira/browse/HBASE-13356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387504#comment-14387504 ] Enis Soztutar commented on HBASE-13356: --- Sounds good. bq. TableSnapshotInputFormatImpl.InputSplit take in a scan object and restoreDir path We can create sub dirs in the restoreDir from user for each scan. HBase should provide an InputFormat supporting multiple scans in mapreduce jobs over snapshots -- Key: HBASE-13356 URL: https://issues.apache.org/jira/browse/HBASE-13356 Project: HBase Issue Type: New Feature Components: mapreduce Reporter: Andrew Mains Priority: Minor Currently, HBase supports the pushing of multiple scans to mapreduce jobs over live tables (via MultiTableInputFormat) but only supports a single scan for mapreduce jobs over table snapshots. It would be handy to support multiple scans over snapshots as well, probably through another input format (MultiTableSnapshotInputFormat?). To mimic the functionality present in MultiTableInputFormat, the new input format would likely have to take in the names of all snapshots used in addition to the scans. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13365) Fix errors from new errorProne version
Elliott Clark created HBASE-13365: - Summary: Fix errors from new errorProne version Key: HBASE-13365 URL: https://issues.apache.org/jira/browse/HBASE-13365 Project: HBase Issue Type: Bug Reporter: Elliott Clark -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13317) Region server reportForDuty stuck looping if there is a master change
[
https://issues.apache.org/jira/browse/HBASE-13317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387603#comment-14387603
]
Jerry He commented on HBASE-13317:
--
thanks, [~enis]
Attached branch-1-v5 patch.
Region server reportForDuty stuck looping if there is a master change
-
Key: HBASE-13317
URL: https://issues.apache.org/jira/browse/HBASE-13317
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 1.0.0, 2.0.0, 0.98.12
Reporter: Jerry He
Assignee: Jerry He
Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.13
Attachments: HBASE-13317-0.98-v2.patch, HBASE-13317-0.98-v3.patch,
HBASE-13317-0.98-v4.patch, HBASE-13317-0.98-v5.patch, HBASE-13317-0.98.patch,
HBASE-13317-branch-1-v5.patch, HBASE-13317-master-v5.patch
During cluster startup, region server reportForDuty gets stuck looping if
there is a master change.
{noformat}
2015-03-22 11:15:16,186 INFO [regionserver60020] regionserver.HRegionServer:
reportForDuty to master=bigaperf274,6,1427045883965 with port=60020,
startcode=1427048115174
2015-03-22 11:15:16,272 WARN [regionserver60020] regionserver.HRegionServer:
error telling master we are up
com.google.protobuf.ServiceException: java.net.ConnectException: Connection
refused
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1678)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
at
org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8277)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2137)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:896)
at java.lang.Thread.run(Thread.java:745)
2015-03-22 11:15:16,274 WARN [regionserver60020] regionserver.HRegionServer:
reportForDuty failed; sleeping and then retrying.
2015-03-22 11:15:19,274 INFO [regionserver60020] regionserver.HRegionServer:
reportForDuty to master=bigaperf273,6,1427048108439 with port=60020,
startcode=1427048115174
2015-03-22 11:15:19,275 WARN [regionserver60020] regionserver.HRegionServer:
error telling master we are up
com.google.protobuf.ServiceException: java.net.ConnectException: Connection
refused
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1678)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
at
org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8277)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2137)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:896)
at java.lang.Thread.run(Thread.java:745)
2015-03-22 11:15:19,276 WARN [regionserver60020] regionserver.HRegionServer:
reportForDuty failed; sleeping and then retrying.
2015-03-22 11:15:22,276 INFO [regionserver60020] regionserver.HRegionServer:
reportForDuty to master=bigaperf273,6,1427048108439 with port=60020,
startcode=1427048115174
2015-03-22 11:15:22,296 DEBUG [regionserver60020] regionserver.HRegionServer:
Master is not running yet
2015-03-22 11:15:22,296 WARN [regionserver60020] regionserver.HRegionServer:
reportForDuty failed; sleeping and then retrying.
2015-03-22 11:15:25,296 INFO [regionserver60020] regionserver.HRegionServer:
reportForDuty to master=bigaperf273,6,1427048108439 with port=60020,
startcode=1427048115174
2015-03-22 11:15:25,299 DEBUG [regionserver60020] regionserver.HRegionServer:
Master is not running yet
2015-03-22 11:15:25,299 WARN [regionserver60020] regionserver.HRegionServer:
reportForDuty failed; sleeping and then retrying.
2015-03-22 11:15:28,299 INFO [regionserver60020] regionserver.HRegionServer:
reportForDuty to master=bigaperf273,6,1427048108439 with port=60020,
startcode=1427048115174
2015-03-22 11:15:28,302 DEBUG [regionserver60020] regionserver.HRegionServer:
Master is not running yet
2015-03-22 11:15:28,302 WARN [regionserver60020] regionserver.HRegionServer:
reportForDuty failed; sleeping and then retrying.
{noformat}
What happended is the region server first got
master=bigaperf274,6,1427045883965. Before it was able to report
successfully, the maser changed to bigaperf273,6,1427048108439.
We were supposed to open a new connection to the new master. But we never
did, looping and trying to old address forever.
--
This message
[jira] [Updated] (HBASE-13317) Region server reportForDuty stuck looping if there is a master change
[
https://issues.apache.org/jira/browse/HBASE-13317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jerry He updated HBASE-13317:
-
Attachment: HBASE-13317-branch-1-v5.patch
Region server reportForDuty stuck looping if there is a master change
-
Key: HBASE-13317
URL: https://issues.apache.org/jira/browse/HBASE-13317
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 1.0.0, 2.0.0, 0.98.12
Reporter: Jerry He
Assignee: Jerry He
Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.13
Attachments: HBASE-13317-0.98-v2.patch, HBASE-13317-0.98-v3.patch,
HBASE-13317-0.98-v4.patch, HBASE-13317-0.98-v5.patch, HBASE-13317-0.98.patch,
HBASE-13317-branch-1-v5.patch, HBASE-13317-master-v5.patch
During cluster startup, region server reportForDuty gets stuck looping if
there is a master change.
{noformat}
2015-03-22 11:15:16,186 INFO [regionserver60020] regionserver.HRegionServer:
reportForDuty to master=bigaperf274,6,1427045883965 with port=60020,
startcode=1427048115174
2015-03-22 11:15:16,272 WARN [regionserver60020] regionserver.HRegionServer:
error telling master we are up
com.google.protobuf.ServiceException: java.net.ConnectException: Connection
refused
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1678)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
at
org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8277)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2137)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:896)
at java.lang.Thread.run(Thread.java:745)
2015-03-22 11:15:16,274 WARN [regionserver60020] regionserver.HRegionServer:
reportForDuty failed; sleeping and then retrying.
2015-03-22 11:15:19,274 INFO [regionserver60020] regionserver.HRegionServer:
reportForDuty to master=bigaperf273,6,1427048108439 with port=60020,
startcode=1427048115174
2015-03-22 11:15:19,275 WARN [regionserver60020] regionserver.HRegionServer:
error telling master we are up
com.google.protobuf.ServiceException: java.net.ConnectException: Connection
refused
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1678)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
at
org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8277)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2137)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:896)
at java.lang.Thread.run(Thread.java:745)
2015-03-22 11:15:19,276 WARN [regionserver60020] regionserver.HRegionServer:
reportForDuty failed; sleeping and then retrying.
2015-03-22 11:15:22,276 INFO [regionserver60020] regionserver.HRegionServer:
reportForDuty to master=bigaperf273,6,1427048108439 with port=60020,
startcode=1427048115174
2015-03-22 11:15:22,296 DEBUG [regionserver60020] regionserver.HRegionServer:
Master is not running yet
2015-03-22 11:15:22,296 WARN [regionserver60020] regionserver.HRegionServer:
reportForDuty failed; sleeping and then retrying.
2015-03-22 11:15:25,296 INFO [regionserver60020] regionserver.HRegionServer:
reportForDuty to master=bigaperf273,6,1427048108439 with port=60020,
startcode=1427048115174
2015-03-22 11:15:25,299 DEBUG [regionserver60020] regionserver.HRegionServer:
Master is not running yet
2015-03-22 11:15:25,299 WARN [regionserver60020] regionserver.HRegionServer:
reportForDuty failed; sleeping and then retrying.
2015-03-22 11:15:28,299 INFO [regionserver60020] regionserver.HRegionServer:
reportForDuty to master=bigaperf273,6,1427048108439 with port=60020,
startcode=1427048115174
2015-03-22 11:15:28,302 DEBUG [regionserver60020] regionserver.HRegionServer:
Master is not running yet
2015-03-22 11:15:28,302 WARN [regionserver60020] regionserver.HRegionServer:
reportForDuty failed; sleeping and then retrying.
{noformat}
What happended is the region server first got
master=bigaperf274,6,1427045883965. Before it was able to report
successfully, the maser changed to bigaperf273,6,1427048108439.
We were supposed to open a new connection to the new master. But we never
did, looping and trying to old address forever.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12972) Region, a supportable public/evolving subset of HRegion
[
https://issues.apache.org/jira/browse/HBASE-12972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387446#comment-14387446
]
Hadoop QA commented on HBASE-12972:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12708220/HBASE-12972-branch-1.patch
against branch-1 branch at commit 3815a33e3400cee73f63e205afac8f1a2d9c174f.
ATTACHMENT ID: 12708220
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 359
new or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13489//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13489//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13489//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13489//console
This message is automatically generated.
Region, a supportable public/evolving subset of HRegion
---
Key: HBASE-12972
URL: https://issues.apache.org/jira/browse/HBASE-12972
Project: HBase
Issue Type: New Feature
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-12972-0.98.patch, HBASE-12972-branch-1.patch,
HBASE-12972.patch, HBASE-12972.patch, HBASE-12972.patch, HBASE-12972.patch,
HBASE-12972.patch
On HBASE-12566, [~lhofhansl] proposed:
{quote}
Maybe we can have a {{Region}} interface that is to {{HRegion}} is what
{{Store}} is to {{HStore}}. Store marked with {{@InterfaceAudience.Private}}
but used in some coprocessor hooks.
{quote}
By example, now coprocessors have to reach into HRegion in order to
participate in row and region locking protocols, this is one area where the
functionality is legitimate for coprocessors but not for users, so an
in-between interface make sense.
In addition we should promote {{Store}}'s interface audience to
LimitedPrivate(COPROC).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-7105) RS throws NPE on forcing compaction from HBase shell on a single bulk imported file.
[
https://issues.apache.org/jira/browse/HBASE-7105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387476#comment-14387476
]
Cosmin Lehene commented on HBASE-7105:
--
[~enis] this fell off my memory. I'll try to take a look this week.
RS throws NPE on forcing compaction from HBase shell on a single bulk
imported file.
Key: HBASE-7105
URL: https://issues.apache.org/jira/browse/HBASE-7105
Project: HBase
Issue Type: Bug
Components: regionserver
Reporter: Karthik Ranganathan
Assignee: Cosmin Lehene
Fix For: 2.0.0, 1.1.0, 1.0.2
Attachments:
0001-HBASE-7105-RS-throws-NPE-on-forcing-compaction-from-.patch,
0001-HBASE-7105-RS-throws-NPE-on-forcing-compaction-from-.patch
In StoreFile, we have:
private AtomicBoolean majorCompaction = null;
In StoreFile.open(), we do:
b = metadataMap.get(MAJOR_COMPACTION_KEY);
if (b != null) {
// init majorCompaction variable
}
Because the file was bulk imported, this is never initialized. Any subsequent
call to isMajorCompaction() NPE's.
Fix is to initialize it to false.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13247) Change BufferedMutatorExample to use addColumn() since add() is deprecated
[ https://issues.apache.org/jira/browse/HBASE-13247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-13247: -- Labels: beginner beginners newbie (was: beginner beginners) Change BufferedMutatorExample to use addColumn() since add() is deprecated -- Key: HBASE-13247 URL: https://issues.apache.org/jira/browse/HBASE-13247 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 1.0.0 Reporter: Lars George Assignee: Lars George Priority: Trivial Labels: beginner, beginners, newbie Fix For: 2.0.0, 1.1.0, 1.0.2 The add() call is deprecated and should be replaced by addColumn(), trivial change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12743) [ITBLL] Master fails rejoining cluster stuck splitting logs; Distributed log replay=true
[
https://issues.apache.org/jira/browse/HBASE-12743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-12743:
--
Fix Version/s: (was: 1.0.1)
1.0.2
[ITBLL] Master fails rejoining cluster stuck splitting logs; Distributed log
replay=true
Key: HBASE-12743
URL: https://issues.apache.org/jira/browse/HBASE-12743
Project: HBase
Issue Type: Bug
Reporter: stack
Fix For: 2.0.0, 1.1.0, 1.0.2
Attachments: 12743.hack.txt
Master is stuck for two days trying to rejoin cluster after monkey killed and
restarted it.
After retrying to get namespace 350 times, Master goes down:
{code}
2014-12-20 18:43:54,285 INFO [c2020:16020.activeMasterManager]
client.RpcRetryingCaller: Call exception, tries=349, retries=350,
started=6885331 ms ago, cancelled=false, msg=row 'default' on table
'hbase:namespace' at
region=hbase:namespace,,1417551886199.ecdcd0172cd3e32d291bc282771895da.,
hostname=c2023.halxg.cloudera.com,16020,1418988286696, seqNum=600190
2014-12-20 18:43:54,303 WARN [c2020:16020.activeMasterManager]
master.TableNamespaceManager: Caught exception in initializing namespace
table manager
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
attempts=350, exceptions:
Sat Dec 20 16:49:08 PST 2014,
RpcRetryingCaller{globalStartTime=1419122948954, pause=100, retries=350},
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: Region
hbase:namespace,,1417551886199.ecdcd0172cd3e32d291bc282771895da. is not
online on c2023.halxg.cloudera.com,16020,1418988286696
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2722)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:851)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:1695)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:30434)
{code}
Seems like 2014-12-20 16:49:03,665 INFO [RS_LOG_REPLAY_OPS-c2021:16020-0]
wal.WALSplitter: DistributedLogReplay = true
Seems easy enough to reproduce.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13364) Make using the default javac on by default
[ https://issues.apache.org/jira/browse/HBASE-13364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387546#comment-14387546 ] Andrew Purtell commented on HBASE-13364: +1 Patch also updates the error prone version which leads to build failures due to our code triggering some of the new checkers. That's not ideal, but since it's an optional profile disabled by default we can file another issue to fix those. Make using the default javac on by default -- Key: HBASE-13364 URL: https://issues.apache.org/jira/browse/HBASE-13364 Project: HBase Issue Type: Bug Components: build Affects Versions: 2.0.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0 Attachments: HBASE-13364.patch Errorprone doesn't work with java 8 and java 8 is becoming more and more standard everywhere. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13364) Make using the default javac on by default
[ https://issues.apache.org/jira/browse/HBASE-13364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387564#comment-14387564 ] Elliott Clark commented on HBASE-13364: --- bq.That's not ideal, but since it's an optional profile disabled by default we can file another issue to fix those. Yeah lets file that issue after this goes in. Most of the errors look innocuous, but still better safe than sorry. Make using the default javac on by default -- Key: HBASE-13364 URL: https://issues.apache.org/jira/browse/HBASE-13364 Project: HBase Issue Type: Bug Components: build Affects Versions: 2.0.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0 Attachments: HBASE-13364.patch Errorprone doesn't work with java 8 and java 8 is becoming more and more standard everywhere. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13364) Make using the default javac on by default
[ https://issues.apache.org/jira/browse/HBASE-13364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-13364: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Make using the default javac on by default -- Key: HBASE-13364 URL: https://issues.apache.org/jira/browse/HBASE-13364 Project: HBase Issue Type: Bug Components: build Affects Versions: 2.0.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0 Attachments: HBASE-13364.patch Errorprone doesn't work with java 8 and java 8 is becoming more and more standard everywhere. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13335) Update ClientSmallScanner and ClientSmallReversedScanner
[
https://issues.apache.org/jira/browse/HBASE-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387605#comment-14387605
]
Enis Soztutar commented on HBASE-13335:
---
This seems specific to scanner callables, not a generic callable factory
(callables can be any method).
{code}
+public interface RetryingCallableFactory {
{code}
Maybe we can make it an internal class. Other than that LGTM. Nice tests!
Update ClientSmallScanner and ClientSmallReversedScanner
Key: HBASE-13335
URL: https://issues.apache.org/jira/browse/HBASE-13335
Project: HBase
Issue Type: Sub-task
Components: Client
Reporter: Josh Elser
Assignee: Josh Elser
Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.13
Attachments: HBASE-13335.patch
Some follow-on work for HBASE-13262:
it's unlikely that clients using the small scanners would get enough data to
run into the initial bug, but the scanner implementations should still adhere
to the moreResultsInRegion flag when the server sends it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11811) Use binary search for seeking into a block
[ https://issues.apache.org/jira/browse/HBASE-11811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387375#comment-14387375 ] stack commented on HBASE-11811: --- Patch looks good. Would help case where repeated, random seeks in a small set of hot blocks. Won't help scans. We should write the index into the block at write time rather than calculate when reading? Use binary search for seeking into a block -- Key: HBASE-11811 URL: https://issues.apache.org/jira/browse/HBASE-11811 Project: HBase Issue Type: Brainstorming Reporter: Lars Hofhansl Attachments: 11811-wip-v2.txt, 11811-wip-v4.txt, block_index-v2.txt Currently upon every seek (including Gets) we need to linearly look through the block from the beginning until we find the Cell we are looking for. It should be possible to build a simple cache of offsets of Cells for each block as it is loaded and then use binary search to find the Cell in question. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13364) Make using the default javac on by default
[ https://issues.apache.org/jira/browse/HBASE-13364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387433#comment-14387433 ] Andrew Purtell commented on HBASE-13364: Works for me Make using the default javac on by default -- Key: HBASE-13364 URL: https://issues.apache.org/jira/browse/HBASE-13364 Project: HBase Issue Type: Bug Reporter: Elliott Clark Assignee: Elliott Clark Errorprone doesn't work with java 8 and java 8 is becoming more and more standard everywhere. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13222) Provide means of non-destructive balancer inspection
[ https://issues.apache.org/jira/browse/HBASE-13222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387461#comment-14387461 ] Nick Dimiduk commented on HBASE-13222: -- Belated +1. Nice work [~elserj]. Provide means of non-destructive balancer inspection Key: HBASE-13222 URL: https://issues.apache.org/jira/browse/HBASE-13222 Project: HBase Issue Type: Improvement Components: Balancer Reporter: Nick Dimiduk Assignee: Josh Elser Priority: Minor Fix For: 2.0.0, 1.1.0 Attachments: HBASE-13222-v1.patch, HBASE-13222-v2.patch, HBASE-13222-v3.patch, HBASE-13222.00-0.98.patch, HBASE-13222.patch, master.jpg At least on 0.98 (haven't checked master, 1.0), it seems we don't have a means for accessing balancer status w.o changing it in the process. Here's a little script to tickle ZK and see what the flag is. Would be nice if this was available from the shell, on the master status page, or something similar. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13247) Change BufferedMutatorExample to use addColumn() since add() is deprecated
[ https://issues.apache.org/jira/browse/HBASE-13247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-13247: -- Fix Version/s: (was: 1.0.1) 1.0.2 Change BufferedMutatorExample to use addColumn() since add() is deprecated -- Key: HBASE-13247 URL: https://issues.apache.org/jira/browse/HBASE-13247 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 1.0.0 Reporter: Lars George Assignee: Lars George Priority: Trivial Labels: beginner, beginners, newbie Fix For: 2.0.0, 1.1.0, 1.0.2 The add() call is deprecated and should be replaced by addColumn(), trivial change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13247) Change BufferedMutatorExample to use addColumn() since add() is deprecated
[ https://issues.apache.org/jira/browse/HBASE-13247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-13247: -- Labels: beginner beginners (was: ) Change BufferedMutatorExample to use addColumn() since add() is deprecated -- Key: HBASE-13247 URL: https://issues.apache.org/jira/browse/HBASE-13247 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 1.0.0 Reporter: Lars George Assignee: Lars George Priority: Trivial Labels: beginner, beginners, newbie Fix For: 2.0.0, 1.1.0, 1.0.2 The add() call is deprecated and should be replaced by addColumn(), trivial change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13317) Region server reportForDuty stuck looping if there is a master change
[
https://issues.apache.org/jira/browse/HBASE-13317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387495#comment-14387495
]
Enis Soztutar commented on HBASE-13317:
---
Test failure does not seem related. I'm +1 for the patch.
Region server reportForDuty stuck looping if there is a master change
-
Key: HBASE-13317
URL: https://issues.apache.org/jira/browse/HBASE-13317
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 1.0.0, 2.0.0, 0.98.12
Reporter: Jerry He
Assignee: Jerry He
Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.13
Attachments: HBASE-13317-0.98-v2.patch, HBASE-13317-0.98-v3.patch,
HBASE-13317-0.98-v4.patch, HBASE-13317-0.98-v5.patch, HBASE-13317-0.98.patch,
HBASE-13317-master-v5.patch
During cluster startup, region server reportForDuty gets stuck looping if
there is a master change.
{noformat}
2015-03-22 11:15:16,186 INFO [regionserver60020] regionserver.HRegionServer:
reportForDuty to master=bigaperf274,6,1427045883965 with port=60020,
startcode=1427048115174
2015-03-22 11:15:16,272 WARN [regionserver60020] regionserver.HRegionServer:
error telling master we are up
com.google.protobuf.ServiceException: java.net.ConnectException: Connection
refused
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1678)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
at
org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8277)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2137)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:896)
at java.lang.Thread.run(Thread.java:745)
2015-03-22 11:15:16,274 WARN [regionserver60020] regionserver.HRegionServer:
reportForDuty failed; sleeping and then retrying.
2015-03-22 11:15:19,274 INFO [regionserver60020] regionserver.HRegionServer:
reportForDuty to master=bigaperf273,6,1427048108439 with port=60020,
startcode=1427048115174
2015-03-22 11:15:19,275 WARN [regionserver60020] regionserver.HRegionServer:
error telling master we are up
com.google.protobuf.ServiceException: java.net.ConnectException: Connection
refused
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1678)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
at
org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8277)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2137)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:896)
at java.lang.Thread.run(Thread.java:745)
2015-03-22 11:15:19,276 WARN [regionserver60020] regionserver.HRegionServer:
reportForDuty failed; sleeping and then retrying.
2015-03-22 11:15:22,276 INFO [regionserver60020] regionserver.HRegionServer:
reportForDuty to master=bigaperf273,6,1427048108439 with port=60020,
startcode=1427048115174
2015-03-22 11:15:22,296 DEBUG [regionserver60020] regionserver.HRegionServer:
Master is not running yet
2015-03-22 11:15:22,296 WARN [regionserver60020] regionserver.HRegionServer:
reportForDuty failed; sleeping and then retrying.
2015-03-22 11:15:25,296 INFO [regionserver60020] regionserver.HRegionServer:
reportForDuty to master=bigaperf273,6,1427048108439 with port=60020,
startcode=1427048115174
2015-03-22 11:15:25,299 DEBUG [regionserver60020] regionserver.HRegionServer:
Master is not running yet
2015-03-22 11:15:25,299 WARN [regionserver60020] regionserver.HRegionServer:
reportForDuty failed; sleeping and then retrying.
2015-03-22 11:15:28,299 INFO [regionserver60020] regionserver.HRegionServer:
reportForDuty to master=bigaperf273,6,1427048108439 with port=60020,
startcode=1427048115174
2015-03-22 11:15:28,302 DEBUG [regionserver60020] regionserver.HRegionServer:
Master is not running yet
2015-03-22 11:15:28,302 WARN [regionserver60020] regionserver.HRegionServer:
reportForDuty failed; sleeping and then retrying.
{noformat}
What happended is the region server first got
master=bigaperf274,6,1427045883965. Before it was able to report
successfully, the maser changed to bigaperf273,6,1427048108439.
We were supposed to open a new connection to the new master. But we never
did, looping and trying to old address forever.
--
This message was
[jira] [Updated] (HBASE-13336) Consistent rules for security meta table protections
[ https://issues.apache.org/jira/browse/HBASE-13336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-13336: -- Fix Version/s: (was: 1.0.1) 1.0.2 Consistent rules for security meta table protections Key: HBASE-13336 URL: https://issues.apache.org/jira/browse/HBASE-13336 Project: HBase Issue Type: Improvement Reporter: Andrew Purtell Assignee: Srikanth Srungarapu Fix For: 2.0.0, 1.1.0, 0.98.13, 1.0.2 The AccessController and VisibilityController do different things regarding protecting their meta tables. The AC allows schema changes and disable/enable if the user has permission. The VC unconditionally disallows all admin actions. Generally, bad things will happen if these meta tables are damaged, disabled, or dropped. The likely outcome is random frequent (or constant) server side op failures with nasty stack traces. On the other hand some things like column family and table attribute changes can have valid use cases. We should have consistent and sensible rules for protecting security meta tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13364) Make using the default javac on by default
[ https://issues.apache.org/jira/browse/HBASE-13364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-13364: -- Component/s: build Make using the default javac on by default -- Key: HBASE-13364 URL: https://issues.apache.org/jira/browse/HBASE-13364 Project: HBase Issue Type: Bug Components: build Affects Versions: 2.0.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0 Attachments: HBASE-13364.patch Errorprone doesn't work with java 8 and java 8 is becoming more and more standard everywhere. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13364) Make using the default javac on by default
[ https://issues.apache.org/jira/browse/HBASE-13364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-13364: -- Fix Version/s: 2.0.0 Affects Version/s: 2.0.0 Status: Patch Available (was: Open) Make using the default javac on by default -- Key: HBASE-13364 URL: https://issues.apache.org/jira/browse/HBASE-13364 Project: HBase Issue Type: Bug Components: build Affects Versions: 2.0.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0 Attachments: HBASE-13364.patch Errorprone doesn't work with java 8 and java 8 is becoming more and more standard everywhere. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13364) Make using the default javac on by default
[ https://issues.apache.org/jira/browse/HBASE-13364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-13364: -- Attachment: HBASE-13364.patch Make using the default javac on by default -- Key: HBASE-13364 URL: https://issues.apache.org/jira/browse/HBASE-13364 Project: HBase Issue Type: Bug Components: build Affects Versions: 2.0.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0 Attachments: HBASE-13364.patch Errorprone doesn't work with java 8 and java 8 is becoming more and more standard everywhere. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12954) Ability impaired using HBase on multihomed hosts
[
https://issues.apache.org/jira/browse/HBASE-12954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-12954:
---
Fix Version/s: 1.1.0
2.0.0
Release Note:
The following config is added by this JIRA:
hbase.regionserver.hostname
This config is for experts: don't set its value unless you really know what you
are doing.
When set to a non-empty value, this represents the (external facing) hostname
for the underlying server.
See https://issues.apache.org/jira/browse/HBASE-12954 for details.
Hadoop Flags: Reviewed
Ability impaired using HBase on multihomed hosts
Key: HBASE-12954
URL: https://issues.apache.org/jira/browse/HBASE-12954
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.4
Reporter: Clay B.
Assignee: Ted Yu
Priority: Minor
Fix For: 2.0.0, 1.1.0
Attachments: 12954-v1.txt, 12954-v10.txt, 12954-v11.txt,
12954-v12.txt, 12954-v12.txt, 12954-v12.txt, 12954-v13.txt, 12954-v14.txt,
12954-v7.txt, 12954-v8.txt, Hadoop Three Interfaces.png
For HBase clusters running on unusual networks (such as NAT'd cloud
environments or physical machines with multiple IP's per network interface)
it would be ideal to have a way to both specify:
# which IP interface to which HBase master or region-server will bind
# what hostname HBase will advertise in Zookeeper both for a master or
region-server process
While efforts such as HBASE-8640 go a long way to normalize these two sources
of information, it is not possible in the current design of the properties
available to an administrator for these to be unambiguously specified.
One has been able to request {{hbase.master.ipc.address}} or
{{hbase.regionserver.ipc.address}} but one can not specify the desired HBase
{{hbase.master.hostname}}. (It was removed in HBASE-1357, further I am
unaware of a region-server equivalent.)
I use a configuration management system to generate all of my configuration
files on a per-machine basis. As such, an option to generate a file
specifying exactly which hostname to use would be helpful.
Today, specifying the bind address for HBase works and one can use an
HBase-only DNS for faking what to put in Zookeeper but this is far from
ideal. Network interfaces have no intrinsic IP address, nor hostname.
Specifing a DNS server is awkward as the DNS server may differ from the
system's resolver and is a single IP address. Similarly, on hosts which use a
transient VIP (e.g. through keepalived) for other services, it means there's
a seemingly non-deterministic hostname choice made by HBase depending on the
state of the VIP at daemon start-up time.
I will attach two networking examples I use which become very difficult to
manage under the current properties.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13362) set max result size from client only (like caching)?
[
https://issues.apache.org/jira/browse/HBASE-13362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387604#comment-14387604
]
Josh Elser commented on HBASE-13362:
{quote}
* remove any server only result sizing
* continue to use hbase.client.scanner.max.result.size but enforce it via the
client only (as the name implies anyway).
{quote}
Are you suggesting that the server shouldn't do anything with
hbase.client.scanner.max.result.size then? On its own, I agree that it's a bit
goofy for that property to be used server-side (I did a double-take the first
time I saw it).
But, assuming I understand you correctly, if we remove that property from the
server, we'd have to rely on quotas to restrict memory used in a server for
scans across all users, right? Best as I can tell, there isn't a
straightforward way to configure RegionServer scans should not exceed $X
memory as there is now (that parameter * number of rpc handlers or the scan
quota) which is a little worrisome at first glance as it would let clients use
a lot of memory out of the box if they set it in the Scan. I could just be
ignorant of other configuration properties though...
set max result size from client only (like caching)?
Key: HBASE-13362
URL: https://issues.apache.org/jira/browse/HBASE-13362
Project: HBase
Issue Type: Brainstorming
Reporter: Lars Hofhansl
With the recent problems we've been seeing client/server result size
mismatch, I was thinking: Why was this not a problem with scanner caching?
There are two reasons:
# number of rows is easy to calculate (and we did it correctly)
# caching is only controlled from the client, never set on the server alone
We did fix both #1 and #2 in HBASE-13262.
Still, I'd like to discuss the following:
* default the client sent max result size to 2mb
* remove any server only result sizing
* continue to use hbase.client.scanner.max.result.size but enforce it via the
client only (as the name implies anyway).
Comments? Concerns?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12891) Parallel execution for Hbck checkRegionConsistency
[ https://issues.apache.org/jira/browse/HBASE-12891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387602#comment-14387602 ] Stephen Yuan Jiang commented on HBASE-12891: Sounds good. Parallel execution for Hbck checkRegionConsistency -- Key: HBASE-12891 URL: https://issues.apache.org/jira/browse/HBASE-12891 Project: HBase Issue Type: Improvement Components: hbck Affects Versions: 2.0.0, 0.98.10, 1.1.0 Reporter: churro morales Assignee: Stephen Yuan Jiang Labels: performance, scalability Fix For: 2.0.0, 1.1.0, 0.98.13 Attachments: HBASE-12891-v1.patch, HBASE-12891-v3.patch, HBASE-12891-v4.patch, HBASE-12891.98.patch, HBASE-12891.patch, HBASE-12891.patch, HBASE-12891.v2-branch-1.patch, HBASE-12891.v2-master.patch, hbase-12891-addendum1.patch We have a lot of regions on our cluster ~500k and noticed that hbck took quite some time in checkAndFixConsistency(). [~davelatham] patched our cluster to do this check in parallel to speed things up. I'll attach the patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13335) Update ClientSmallScanner and ClientSmallReversedScanner
[
https://issues.apache.org/jira/browse/HBASE-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387611#comment-14387611
]
Josh Elser commented on HBASE-13335:
Thanks for taking a look.
bq. not a generic callable factory (callables can be any method).
That's a good point. I meant it was {{o.a.h.hbase.client.RetryingCallable}},
but the name doesn't quite denote that well :)
bq. Maybe we can make it an internal class
Suggestions? Inner class on ClientSmallScanner? Something else?
Update ClientSmallScanner and ClientSmallReversedScanner
Key: HBASE-13335
URL: https://issues.apache.org/jira/browse/HBASE-13335
Project: HBase
Issue Type: Sub-task
Components: Client
Reporter: Josh Elser
Assignee: Josh Elser
Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.13
Attachments: HBASE-13335.patch
Some follow-on work for HBASE-13262:
it's unlikely that clients using the small scanners would get enough data to
run into the initial bug, but the scanner implementations should still adhere
to the moreResultsInRegion flag when the server sends it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13213) Split out locality metrics among primary and secondary region
[
https://issues.apache.org/jira/browse/HBASE-13213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387385#comment-14387385
]
Ted Yu commented on HBASE-13213:
{code}
$ python dev-support/checkstyle_report.py trunkCheckstyle.xml
patchCheckstyle.xml
hbase-server/target/generated-jamon/org/apache/hadoop/hbase/tmpl/regionserver/ServerMetricsTmplImpl.java
{code}
The extra Checkstyle warning came from a generated file.
Split out locality metrics among primary and secondary region
-
Key: HBASE-13213
URL: https://issues.apache.org/jira/browse/HBASE-13213
Project: HBase
Issue Type: Sub-task
Reporter: Ted Yu
Assignee: Ted Yu
Attachments: 13213-v1.patch, locality-metrics-4.txt
This task provides the ability to track locality of primary and secondary
region replicas.
We already have percentFilesLocal metric.
There should be two sets of metrics - one for primaries and another for
secondary / tertiary regions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-5761) [Thrift2] TDelete.deleteType defaults to TDeleteType.DELETE_COLUMNS, but the docs suggest otherwise
[
https://issues.apache.org/jira/browse/HBASE-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-5761:
-
Fix Version/s: (was: 1.0.1)
1.0.2
[Thrift2] TDelete.deleteType defaults to TDeleteType.DELETE_COLUMNS, but the
docs suggest otherwise
---
Key: HBASE-5761
URL: https://issues.apache.org/jira/browse/HBASE-5761
Project: HBase
Issue Type: Bug
Components: documentation
Reporter: Wouter Bolsterlee
Priority: Trivial
Labels: beginner
Fix For: 2.0.0, 1.1.0, 1.0.2
It seems to me there is an inconsistency (or error) in the Thrift2
{{TDelete}} struct and its documentation. The docs for the {{TDelete}} struct
state:
{quote}
If no timestamp is specified the most recent version will be deleted. To
delete all previous versions, specify the DELETE_COLUMNS TDeleteType.
{quote}
...which implies that the default is {{TDeleteType.DELETE_COLUMN}}
(singular), not {{TDeleteType.DELETE_COLUMNS}} (plural).
However, the {{deleteType}} field in the {{TDelete}} struct defaults to the
value {{1}}, which is {{TDeleteType.DELETE_COLUMNS}} (plural) in
{{/src/main/resources/org/apache/hadoop/hbase/thrift2/hbase.thrift}}. The
field is currently (r1239241) defined as follows:
{{4: optional TDeleteType deleteType = 1,}}
I'd suggest that the default for this optional field is changed to
{{TDeleteType.DELETE_COLUMN}} (singular). The line above from the {{TDelete}}
struct would then become:
{{4: optional TDeleteType deleteType = 0,}}
Since this change just involves changing a {{1}} into a {{0}}, I'll leave the
trivial patch to someone who can also commit it in one go. Thanks in advance.
:)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12816) GC logs are lost upon Region Server restart if GCLogFileRotation is enabled
[ https://issues.apache.org/jira/browse/HBASE-12816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-12816: -- Fix Version/s: (was: 1.0.1) 1.0.2 GC logs are lost upon Region Server restart if GCLogFileRotation is enabled --- Key: HBASE-12816 URL: https://issues.apache.org/jira/browse/HBASE-12816 Project: HBase Issue Type: Bug Components: scripts Reporter: Abhishek Singh Chouhan Assignee: Abhishek Singh Chouhan Priority: Minor Fix For: 2.0.0, 1.1.0, 0.98.13, 1.0.2 Attachments: HBASE-12816.patch When -XX:+UseGCLogFileRotation is used gc log files end with .gc.0 instead of .gc. hbase_rotate_log () in hbase-daemon.sh does not handle this correctly and hence when a RS is restarted old gc logs are lost(overwritten). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-5761) [Thrift2] TDelete.deleteType defaults to TDeleteType.DELETE_COLUMNS, but the docs suggest otherwise
[
https://issues.apache.org/jira/browse/HBASE-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387485#comment-14387485
]
Enis Soztutar commented on HBASE-5761:
--
Agreed with changing the documentation rather than default behavior. [~uws]
mind putting up a patch?
[Thrift2] TDelete.deleteType defaults to TDeleteType.DELETE_COLUMNS, but the
docs suggest otherwise
---
Key: HBASE-5761
URL: https://issues.apache.org/jira/browse/HBASE-5761
Project: HBase
Issue Type: Bug
Components: documentation
Reporter: Wouter Bolsterlee
Priority: Trivial
Labels: beginner
Fix For: 2.0.0, 1.1.0, 1.0.2
It seems to me there is an inconsistency (or error) in the Thrift2
{{TDelete}} struct and its documentation. The docs for the {{TDelete}} struct
state:
{quote}
If no timestamp is specified the most recent version will be deleted. To
delete all previous versions, specify the DELETE_COLUMNS TDeleteType.
{quote}
...which implies that the default is {{TDeleteType.DELETE_COLUMN}}
(singular), not {{TDeleteType.DELETE_COLUMNS}} (plural).
However, the {{deleteType}} field in the {{TDelete}} struct defaults to the
value {{1}}, which is {{TDeleteType.DELETE_COLUMNS}} (plural) in
{{/src/main/resources/org/apache/hadoop/hbase/thrift2/hbase.thrift}}. The
field is currently (r1239241) defined as follows:
{{4: optional TDeleteType deleteType = 1,}}
I'd suggest that the default for this optional field is changed to
{{TDeleteType.DELETE_COLUMN}} (singular). The line above from the {{TDelete}}
struct would then become:
{{4: optional TDeleteType deleteType = 0,}}
Since this change just involves changing a {{1}} into a {{0}}, I'll leave the
trivial patch to someone who can also commit it in one go. Thanks in advance.
:)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11290) Unlock RegionStates
[ https://issues.apache.org/jira/browse/HBASE-11290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387530#comment-14387530 ] Mikhail Antonov commented on HBASE-11290: - Is that something we would like to pull in sometime soon (1.1?). Unlock RegionStates --- Key: HBASE-11290 URL: https://issues.apache.org/jira/browse/HBASE-11290 Project: HBase Issue Type: Sub-task Reporter: Francis Liu Assignee: Virag Kothari Fix For: 2.0.0, 1.1.0, 0.98.13 Attachments: HBASE-11290-0.98.patch, HBASE-11290-0.98_v2.patch, HBASE-11290.draft.patch Even though RegionStates is a highly accessed data structure in HMaster. Most of it's methods are synchronized. Which limits concurrency. Even simply making some of the getters non-synchronized by using concurrent data structures has helped with region assignments. We can go as simple as this approach or create locks per region or a bucket lock per region bucket. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
