[jira] [Commented] (HBASE-23185) High cpu usage because getTable()#put() gets config value every time
[ https://issues.apache.org/jira/browse/HBASE-23185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956570#comment-16956570 ] Andrew Kyle Purtell commented on HBASE-23185: - [~stack] Shout! I think the 1.4 that just went out is the last but there might be more 1.3s. So if not too much trouble... > High cpu usage because getTable()#put() gets config value every time > > > Key: HBASE-23185 > URL: https://issues.apache.org/jira/browse/HBASE-23185 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 1.5.0, 1.4.10, 1.2.12, 1.3.5 >Reporter: Shinya Yoshida >Assignee: Shinya Yoshida >Priority: Major > Labels: performance > Fix For: 1.6.0 > > Attachments: Screenshot from 2019-10-18 12-38-14.png, Screenshot from > 2019-10-18 13-03-24.png > > > When we analyzed the performance of our hbase application with many puts, we > found that Configuration methods use many CPU resources: > !Screenshot from 2019-10-18 12-38-14.png|width=460,height=205! > As you can see, getTable().put() is calling Configuration methods which cause > regex or synchronization by Hashtable. > This should not happen in 0.99.2 because > https://issues.apache.org/jira/browse/HBASE-12128 addressed such an issue. > However, it's reproducing nowadays by bugs or leakages after many code > evoluations between 0.9x and 1.x. > # > [https://github.com/apache/hbase/blob/dd9eadb00f9dcd071a246482a11dfc7d63845f00/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java#L369-L374] > ** finishSetup is called every new HTable() e.g. every con.getTable() > ** So getInt is called everytime and it does regex > # > [https://github.com/apache/hbase/blob/dd9eadb00f9dcd071a246482a11dfc7d63845f00/hbase-client/src/main/java/org/apache/hadoop/hbase/client/BufferedMutatorImpl.java#L115] > ** BufferedMutatorImpl is created every first put for HTable e.g. > con.getTable().put() > ** Create ConnectionConf every time in BufferedMutatorImpl constructor > ** ConnectionConf gets config value in the constructor > # > [https://github.com/apache/hbase/blob/dd9eadb00f9dcd071a246482a11dfc7d63845f00/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java#L326] > ** AsyncProcess is created in BufferedMutatorImpl constructor, so new > AsyncProcess is created by con.getTable().put() > ** AsyncProcess parse many configurations > So, con.getTable().put() is heavy operation for CPU because of getting config > value. > > With in-house patch for this issue, we observed about 10% improvement on > max-throughput (e.g. CPU usage) at client-side: > !Screenshot from 2019-10-18 13-03-24.png|width=508,height=223! > > Seems branch-2 is not affected because client implementation has been changed > dramatically. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23317) An option to fail only the region open if a coprocessor fails to load
Andrew Kyle Purtell created HBASE-23317: --- Summary: An option to fail only the region open if a coprocessor fails to load Key: HBASE-23317 URL: https://issues.apache.org/jira/browse/HBASE-23317 Project: HBase Issue Type: Improvement Reporter: Andrew Kyle Purtell If a table coprocessor fails to load, rather than aborting, throw an exception which prevents the region from opening. This will lead to unresolvable regions in transition but in some circumstances this may be preferable to process aborts. On the other hand, there would be a new risk that the failure to load is a symptom of or a cause of regionserver global state corruption that eventually leads to other problems. Should at least be an option, though. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23318) LoadTestTool doesn't start
[ https://issues.apache.org/jira/browse/HBASE-23318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-23318: Description: ./bin/hbase ltt after unpacking a binary tarball distribution doesn't start with a CNFE. We are missing the tests jar from hbase-zookeeper. The client tarball includes this but if one wants to launch it on a server or a general purpose deploy (i.e. not the client tarball) the test jar has to be in the server classpath as well. was:./bin/hbase ltt after unpacking a binary tarball distribution doesn't start with a CNFE. We are missing the tests jar from hbase-zookeeper. > LoadTestTool doesn't start > -- > > Key: HBASE-23318 > URL: https://issues.apache.org/jira/browse/HBASE-23318 > Project: HBase > Issue Type: Bug >Affects Versions: 2.2.1 >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Minor > Fix For: 3.0.0, 2.3.0, 2.2.3 > > > ./bin/hbase ltt after unpacking a binary tarball distribution doesn't start > with a CNFE. We are missing the tests jar from hbase-zookeeper. > The client tarball includes this but if one wants to launch it on a server or > a general purpose deploy (i.e. not the client tarball) the test jar has to be > in the server classpath as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23318) LoadTestTool doesn't start
[ https://issues.apache.org/jira/browse/HBASE-23318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-23318: Status: Patch Available (was: Open) > LoadTestTool doesn't start > -- > > Key: HBASE-23318 > URL: https://issues.apache.org/jira/browse/HBASE-23318 > Project: HBase > Issue Type: Bug >Affects Versions: 2.2.1 >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Minor > Fix For: 3.0.0, 2.3.0, 2.2.3 > > > ./bin/hbase ltt after unpacking a binary tarball distribution doesn't start > with a CNFE. We are missing the tests jar from hbase-zookeeper. > The client tarball includes this but if one wants to launch it on a server or > a general purpose deploy (i.e. not the client tarball) the test jar has to be > in the server classpath as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-18095) Provide an option for clients to find the server hosting META that does not involve the ZooKeeper client
[ https://issues.apache.org/jira/browse/HBASE-18095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977953#comment-16977953 ] Andrew Kyle Purtell edited comment on HBASE-18095 at 11/20/19 12:57 AM: We want this in our operations so that would be a vote for a branch-1 backport. Means we won't have to maintain a local patch. The upgrade story is clean. This is an additive change. Masters and zookeeper will both be able to provide service for a client configured to use either, until such time a site is completely migrated. Then as an operator you'd probably want to restrict cluster clients from accessing zookeeper service ports, but this would be up to the site operator. These separate means of discovery can coexist for as long as they need to. was (Author: apurtell): We want this in our operations so that would be a vote for a branch-1 backport. Means we won't have to maintain a local patch. The upgrade story is clean. This is an additive change. Masters and zookeeper will both be able to provide service for a client configured to use either, until such time a site is completely migrated. > Provide an option for clients to find the server hosting META that does not > involve the ZooKeeper client > > > Key: HBASE-18095 > URL: https://issues.apache.org/jira/browse/HBASE-18095 > Project: HBase > Issue Type: New Feature > Components: Client >Reporter: Andrew Kyle Purtell >Assignee: Bharath Vissapragada >Priority: Major > Attachments: HBASE-18095.master-v1.patch, HBASE-18095.master-v2.patch > > > Clients are required to connect to ZooKeeper to find the location of the > regionserver hosting the meta table region. Site configuration provides the > client a list of ZK quorum peers and the client uses an embedded ZK client to > query meta location. Timeouts and retry behavior of this embedded ZK client > are managed orthogonally to HBase layer settings and in some cases the ZK > cannot manage what in theory the HBase client can, i.e. fail fast upon outage > or network partition. > We should consider new configuration settings that provide a list of > well-known master and backup master locations, and with this information the > client can contact any of the master processes directly. Any master in either > active or passive state will track meta location and respond to requests for > it with its cached last known location. If this location is stale, the client > can ask again with a flag set that requests the master refresh its location > cache and return the up-to-date location. Every client interaction with the > cluster thus uses only HBase RPC as transport, with appropriate settings > applied to the connection. The configuration toggle that enables this > alternative meta location lookup should be false by default. > This removes the requirement that HBase clients embed the ZK client and > contact the ZK service directly at the beginning of the connection lifecycle. > This has several benefits. ZK service need not be exposed to clients, and > their potential abuse, yet no benefit ZK provides the HBase server cluster is > compromised. Normalizing HBase client and ZK client timeout settings and > retry behavior - in some cases, impossible, i.e. for fail-fast - is no longer > necessary. > And, from [~ghelmling]: There is an additional complication here for > token-based authentication. When a delegation token is used for SASL > authentication, the client uses the cluster ID obtained from Zookeeper to > select the token identifier to use. So there would also need to be some > Zookeeper-less, unauthenticated way to obtain the cluster ID as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-18095) Provide an option for clients to find the server hosting META that does not involve the ZooKeeper client
[ https://issues.apache.org/jira/browse/HBASE-18095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977956#comment-16977956 ] Andrew Kyle Purtell commented on HBASE-18095: - bq. why limit many operations to just the active master or even just the collection of masters I missed this question, sorry. A number of similar distributed systems have a notion of a bootstrap set, a list of well known addresses from which a client can discover all other service roles and locations necessary to know for operation. In today's HBase operations the zookeeper quorum peer list serves this role. When proposing this I thought zk quorum list -> hbase master list would be a nice simple lateral change easy to understand and familiar. > Provide an option for clients to find the server hosting META that does not > involve the ZooKeeper client > > > Key: HBASE-18095 > URL: https://issues.apache.org/jira/browse/HBASE-18095 > Project: HBase > Issue Type: New Feature > Components: Client >Reporter: Andrew Kyle Purtell >Assignee: Bharath Vissapragada >Priority: Major > Attachments: HBASE-18095.master-v1.patch, HBASE-18095.master-v2.patch > > > Clients are required to connect to ZooKeeper to find the location of the > regionserver hosting the meta table region. Site configuration provides the > client a list of ZK quorum peers and the client uses an embedded ZK client to > query meta location. Timeouts and retry behavior of this embedded ZK client > are managed orthogonally to HBase layer settings and in some cases the ZK > cannot manage what in theory the HBase client can, i.e. fail fast upon outage > or network partition. > We should consider new configuration settings that provide a list of > well-known master and backup master locations, and with this information the > client can contact any of the master processes directly. Any master in either > active or passive state will track meta location and respond to requests for > it with its cached last known location. If this location is stale, the client > can ask again with a flag set that requests the master refresh its location > cache and return the up-to-date location. Every client interaction with the > cluster thus uses only HBase RPC as transport, with appropriate settings > applied to the connection. The configuration toggle that enables this > alternative meta location lookup should be false by default. > This removes the requirement that HBase clients embed the ZK client and > contact the ZK service directly at the beginning of the connection lifecycle. > This has several benefits. ZK service need not be exposed to clients, and > their potential abuse, yet no benefit ZK provides the HBase server cluster is > compromised. Normalizing HBase client and ZK client timeout settings and > retry behavior - in some cases, impossible, i.e. for fail-fast - is no longer > necessary. > And, from [~ghelmling]: There is an additional complication here for > token-based authentication. When a delegation token is used for SASL > authentication, the client uses the cluster ID obtained from Zookeeper to > select the token identifier to use. So there would also need to be some > Zookeeper-less, unauthenticated way to obtain the cluster ID as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23317) Detect and sideline poison pill regions
[ https://issues.apache.org/jira/browse/HBASE-23317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-23317: Summary: Detect and sideline poison pill regions (was: An option to fail only the region open if a coprocessor fails to load) > Detect and sideline poison pill regions > --- > > Key: HBASE-23317 > URL: https://issues.apache.org/jira/browse/HBASE-23317 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Kyle Purtell >Priority: Minor > > If a table coprocessor fails to load, rather than aborting, throw an > exception which prevents the region from opening. This will lead to > unresolvable regions in transition but in some circumstances this may be > preferable to process aborts. On the other hand, there would be a new risk > that the failure to load is a symptom of or a cause of regionserver global > state corruption that eventually leads to other problems. Should at least be > an option, though. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23317) Detect and sideline poison pill regions
[ https://issues.apache.org/jira/browse/HBASE-23317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-23317: Description: The master can track that a region deploy has been repeatedly crashing regionservers and rather than continue to pass around the poison pill put its assignment into an administratively failed state. (was: If a table coprocessor fails to load, rather than aborting, throw an exception which prevents the region from opening. This will lead to unresolvable regions in transition but in some circumstances this may be preferable to process aborts. On the other hand, there would be a new risk that the failure to load is a symptom of or a cause of regionserver global state corruption that eventually leads to other problems. Should at least be an option, though. ) > Detect and sideline poison pill regions > --- > > Key: HBASE-23317 > URL: https://issues.apache.org/jira/browse/HBASE-23317 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Kyle Purtell >Priority: Minor > > The master can track that a region deploy has been repeatedly crashing > regionservers and rather than continue to pass around the poison pill put its > assignment into an administratively failed state. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23318) LoadTestTool doesn't start
[ https://issues.apache.org/jira/browse/HBASE-23318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-23318: Affects Version/s: 2.2.1 > LoadTestTool doesn't start > -- > > Key: HBASE-23318 > URL: https://issues.apache.org/jira/browse/HBASE-23318 > Project: HBase > Issue Type: Bug >Affects Versions: 2.2.1 >Reporter: Andrew Kyle Purtell >Priority: Minor > > ./bin/hbase ltt after unpacking a binary tarball distribution doesn't start > with a CNFE. We are missing the tests jar from hbase-zookeeper. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23318) LoadTestTool doesn't start
[ https://issues.apache.org/jira/browse/HBASE-23318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-23318: Fix Version/s: 2.2.3 2.3.0 3.0.0 > LoadTestTool doesn't start > -- > > Key: HBASE-23318 > URL: https://issues.apache.org/jira/browse/HBASE-23318 > Project: HBase > Issue Type: Bug >Affects Versions: 2.2.1 >Reporter: Andrew Kyle Purtell >Priority: Minor > Fix For: 3.0.0, 2.3.0, 2.2.3 > > > ./bin/hbase ltt after unpacking a binary tarball distribution doesn't start > with a CNFE. We are missing the tests jar from hbase-zookeeper. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-23318) LoadTestTool doesn't start
[ https://issues.apache.org/jira/browse/HBASE-23318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell reassigned HBASE-23318: --- Assignee: Andrew Kyle Purtell > LoadTestTool doesn't start > -- > > Key: HBASE-23318 > URL: https://issues.apache.org/jira/browse/HBASE-23318 > Project: HBase > Issue Type: Bug >Affects Versions: 2.2.1 >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Minor > Fix For: 3.0.0, 2.3.0, 2.2.3 > > > ./bin/hbase ltt after unpacking a binary tarball distribution doesn't start > with a CNFE. We are missing the tests jar from hbase-zookeeper. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-23288) Backport HBASE-23251 (Add Column Family and Table Names to HFileContext) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-23288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-23288. - Fix Version/s: 1.6.0 Hadoop Flags: Reviewed Resolution: Fixed > Backport HBASE-23251 (Add Column Family and Table Names to HFileContext) to > branch-1 > > > Key: HBASE-23288 > URL: https://issues.apache.org/jira/browse/HBASE-23288 > Project: HBase > Issue Type: Sub-task >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Fix For: 1.6.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23318) LoadTestTool doesn't start
[ https://issues.apache.org/jira/browse/HBASE-23318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-23318: Resolution: Fixed Status: Resolved (was: Patch Available) > LoadTestTool doesn't start > -- > > Key: HBASE-23318 > URL: https://issues.apache.org/jira/browse/HBASE-23318 > Project: HBase > Issue Type: Bug >Affects Versions: 2.2.1 >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Minor > Fix For: 3.0.0, 2.3.0, 2.2.3 > > > ./bin/hbase ltt after unpacking a binary tarball distribution doesn't start > with a CNFE. We are missing the tests jar from hbase-zookeeper. > The client tarball includes this but if one wants to launch it on a server or > a general purpose deploy (i.e. not the client tarball) the test jar has to be > in the server classpath as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23317) Detect and sideline poison pill regions
[ https://issues.apache.org/jira/browse/HBASE-23317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977021#comment-16977021 ] Andrew Kyle Purtell commented on HBASE-23317: - Nope. Repurposed > Detect and sideline poison pill regions > --- > > Key: HBASE-23317 > URL: https://issues.apache.org/jira/browse/HBASE-23317 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Kyle Purtell >Priority: Minor > > The master can track that a region deploy has been repeatedly crashing > regionservers and rather than continue to pass around the poison pill put its > assignment into an administratively failed state. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23318) LoadTestTool doesn't start
Andrew Kyle Purtell created HBASE-23318: --- Summary: LoadTestTool doesn't start Key: HBASE-23318 URL: https://issues.apache.org/jira/browse/HBASE-23318 Project: HBase Issue Type: Bug Reporter: Andrew Kyle Purtell ./bin/hbase ltt after unpacking a binary tarball distribution doesn't start with a CNFE. We are missing the tests jar from hbase-zookeeper. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-18095) Provide an option for clients to find the server hosting META that does not involve the ZooKeeper client
[ https://issues.apache.org/jira/browse/HBASE-18095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977953#comment-16977953 ] Andrew Kyle Purtell commented on HBASE-18095: - We want this in our operations so that would be a vote for a branch-1 backport. Means we won't have to maintain a local patch. The upgrade story is clean. This is an additive change. Masters and zookeeper will both be able to provide service for a client configured to use either, until such time a site is completely migrated. > Provide an option for clients to find the server hosting META that does not > involve the ZooKeeper client > > > Key: HBASE-18095 > URL: https://issues.apache.org/jira/browse/HBASE-18095 > Project: HBase > Issue Type: New Feature > Components: Client >Reporter: Andrew Kyle Purtell >Assignee: Bharath Vissapragada >Priority: Major > Attachments: HBASE-18095.master-v1.patch, HBASE-18095.master-v2.patch > > > Clients are required to connect to ZooKeeper to find the location of the > regionserver hosting the meta table region. Site configuration provides the > client a list of ZK quorum peers and the client uses an embedded ZK client to > query meta location. Timeouts and retry behavior of this embedded ZK client > are managed orthogonally to HBase layer settings and in some cases the ZK > cannot manage what in theory the HBase client can, i.e. fail fast upon outage > or network partition. > We should consider new configuration settings that provide a list of > well-known master and backup master locations, and with this information the > client can contact any of the master processes directly. Any master in either > active or passive state will track meta location and respond to requests for > it with its cached last known location. If this location is stale, the client > can ask again with a flag set that requests the master refresh its location > cache and return the up-to-date location. Every client interaction with the > cluster thus uses only HBase RPC as transport, with appropriate settings > applied to the connection. The configuration toggle that enables this > alternative meta location lookup should be false by default. > This removes the requirement that HBase clients embed the ZK client and > contact the ZK service directly at the beginning of the connection lifecycle. > This has several benefits. ZK service need not be exposed to clients, and > their potential abuse, yet no benefit ZK provides the HBase server cluster is > compromised. Normalizing HBase client and ZK client timeout settings and > retry behavior - in some cases, impossible, i.e. for fail-fast - is no longer > necessary. > And, from [~ghelmling]: There is an additional complication here for > token-based authentication. When a delegation token is used for SASL > authentication, the client uses the cluster ID obtained from Zookeeper to > select the token identifier to use. So there would also need to be some > Zookeeper-less, unauthenticated way to obtain the cluster ID as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23219) Re-enable ZKLess tests for branch-1 (Revert HBASE-14622)
[ https://issues.apache.org/jira/browse/HBASE-23219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16964978#comment-16964978 ] Andrew Kyle Purtell commented on HBASE-23219: - I ran tests relevant for the branch-1 patch 25 times in a loop and they looked fine. Now doing the same with 1.4, then 1.3. Will commit if they all look good. > Re-enable ZKLess tests for branch-1 (Revert HBASE-14622) > > > Key: HBASE-23219 > URL: https://issues.apache.org/jira/browse/HBASE-23219 > Project: HBase > Issue Type: Task > Components: test >Affects Versions: 1.3.6 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Trivial > Fix For: 1.4.12, 1.3.7 > > Attachments: HBASE-23219.branch-1.001.patch, > HBASE-23219.branch-1.3.001.patch, HBASE-23219.branch-1.4.001.patch > > > Since we are using zkless in our production setup, we would like to enable > these tests back in apache on branch-1. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-23128) Restore Region interface compatibility
[
https://issues.apache.org/jira/browse/HBASE-23128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell resolved HBASE-23128.
-
Resolution: Fixed
> Restore Region interface compatibility
> ---
>
> Key: HBASE-23128
> URL: https://issues.apache.org/jira/browse/HBASE-23128
> Project: HBase
> Issue Type: Bug
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Blocker
> Fix For: 1.5.0
>
>
> Adding methods to a Public interface is ok for a minor release, removing
> methods is not. We need to restore
> {code}
> abstract method boolean bulkLoadHFiles (
> Collection>, boolean, Region.BulkLoadListener)
> {code}
> to the Region interface in order to maintain binary compatibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23126) IntegrationTestRSGroup is useless now
[ https://issues.apache.org/jira/browse/HBASE-23126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946089#comment-16946089 ] Andrew Kyle Purtell commented on HBASE-23126: - Would you prefer to remove the IT or try to fix it? > IntegrationTestRSGroup is useless now > - > > Key: HBASE-23126 > URL: https://issues.apache.org/jira/browse/HBASE-23126 > Project: HBase > Issue Type: Bug > Components: rsgroup >Reporter: Duo Zhang >Priority: Major > > It extends TestRSGroupsBase and wants to run all the UTs defined in > TestRSGroupsBase, but after HBASE-21265, all the UTs have been moved to sub > classes... -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23110) Backport HBASE-23054 "Remove synchronization block from MetaTableMetrics and fix LossyCounting algorithm" to branch-1
[ https://issues.apache.org/jira/browse/HBASE-23110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-23110: Resolution: Fixed Status: Resolved (was: Patch Available) > Backport HBASE-23054 "Remove synchronization block from MetaTableMetrics and > fix LossyCounting algorithm" to branch-1 > - > > Key: HBASE-23110 > URL: https://issues.apache.org/jira/browse/HBASE-23110 > Project: HBase > Issue Type: Sub-task > Components: metrics >Reporter: Duo Zhang >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 1.5.0, 1.4.11 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-16141) Unwind use of UserGroupInformation.doAs() to convey requester identity in coprocessor upcalls
[ https://issues.apache.org/jira/browse/HBASE-16141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-16141: Fix Version/s: (was: 1.5.0) 1.6.0 > Unwind use of UserGroupInformation.doAs() to convey requester identity in > coprocessor upcalls > - > > Key: HBASE-16141 > URL: https://issues.apache.org/jira/browse/HBASE-16141 > Project: HBase > Issue Type: Improvement > Components: Coprocessors, security >Reporter: Gary Helmling >Assignee: Gary Helmling >Priority: Major > Fix For: 3.0.0, 2.3.0, 1.6.0 > > > In discussion on HBASE-16115, there is some discussion of whether > UserGroupInformation.doAs() is the right mechanism for propagating the > original requester's identify in certain system contexts (splits, > compactions, some procedure calls). It has the unfortunately of overriding > the current user, which makes for very confusing semantics for coprocessor > implementors. We should instead find an alternate mechanism for conveying > the caller identity, which does not override the current user context. > I think we should instead look at passing this through as part of the > ObserverContext passed to every coprocessor hook. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-18162) Backport 'Update jruby to a newer version' to branch-1
[ https://issues.apache.org/jira/browse/HBASE-18162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-18162: Fix Version/s: (was: 1.5.0) 1.6.0 > Backport 'Update jruby to a newer version' to branch-1 > -- > > Key: HBASE-18162 > URL: https://issues.apache.org/jira/browse/HBASE-18162 > Project: HBase > Issue Type: Task > Components: dependencies, shell >Affects Versions: 1.4.0 >Reporter: Sean Busbey >Priority: Major > Fix For: 1.6.0 > > > Work on HBASE-16196 ran into shell failures while attempting a branch-1 > backport. > Start from either the attempted backport or the master branch version. Work > out what's different in branch-1 that causes TestShell to fail. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-14927) Backport HBASE-13014 and HBASE-14749 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-14927: Fix Version/s: (was: 1.5.0) 1.6.0 > Backport HBASE-13014 and HBASE-14749 to branch-1 > > > Key: HBASE-14927 > URL: https://issues.apache.org/jira/browse/HBASE-14927 > Project: HBase > Issue Type: Improvement >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan >Priority: Major > Fix For: 1.6.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-15648) Reduce number of concurrent region location lookups when MetaCache entry is cleared
[ https://issues.apache.org/jira/browse/HBASE-15648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-15648: Fix Version/s: (was: 1.5.0) 1.6.0 > Reduce number of concurrent region location lookups when MetaCache entry is > cleared > --- > > Key: HBASE-15648 > URL: https://issues.apache.org/jira/browse/HBASE-15648 > Project: HBase > Issue Type: Sub-task > Components: Client >Affects Versions: 1.2.0, 1.3.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Major > Fix For: 3.0.0, 2.3.0, 1.6.0 > > Attachments: HBASE-15648-branch-1.3.v1.patch > > > It seems in HConnectionImplementation#locateRegionInMeta if region location > is removed from the cache, with large number of client threads we could have > many of them getting cache miss and doing meta scan, which looks unnecessary > - we could empty mechanism similar to what we have in IdLock in HFileReader > to fetch the block to cache, do ensure that if one thread is already looking > up location for region R1, other threads who need it's location wait until > first thread finishes his work. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-18415) The local timeout may cause Admin to submit duplicate request
[ https://issues.apache.org/jira/browse/HBASE-18415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-18415: Fix Version/s: (was: 1.5.0) 1.5.1 1.6.0 > The local timeout may cause Admin to submit duplicate request > - > > Key: HBASE-18415 > URL: https://issues.apache.org/jira/browse/HBASE-18415 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Major > Fix For: 3.0.0, 2.3.0, 1.6.0, 1.5.1 > > Attachments: HBASE-18415.branch-1.ut.patch, > HBASE-18415.branch-1.v0.patch, HBASE-18415.branch-1.v1.patch, > HBASE-18415.branch-1.v2.patch, HBASE-18415.branch-1.v3.patch, > HBASE-18415.branch-1.v3.patch, HBASE-18415.branch-1.v3.patch, > HBASE-18415.branch-1.v4.patch, HBASE-18415.branch-1.v4.patch, > HBASE-18415.branch-1.v4.patch > > > After a timeout occurs on first request, client will retry the request with > distinct group/nonce. The second request may bring the TableXXXException back > if the first request have changed the table state. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-20151) Bug with SingleColumnValueFilter and FamilyFilter
[
https://issues.apache.org/jira/browse/HBASE-20151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-20151:
Fix Version/s: (was: 1.5.0)
1.5.1
1.6.0
> Bug with SingleColumnValueFilter and FamilyFilter
> -
>
> Key: HBASE-20151
> URL: https://issues.apache.org/jira/browse/HBASE-20151
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.1.0, 2.0.1, 1.4.5
> Environment: MacOS 10.13.3
> HBase 1.3.1
>Reporter: Steven Sadowski
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 1.6.0, 1.5.1
>
> Attachments: HBASE-20151.master.001.patch,
> HBASE-20151.master.002.patch, HBASE-20151.master.003.patch,
> HBASE-20151.master.004.patch, HBASE-20151.master.004.patch,
> HBASE-20151.master.005.patch, HBASE-20151.master.006.patch,
> filter-list-type.v1.txt
>
>
> When running the following queries, the result is sometimes return correctly
> and other times incorrectly based on the qualifier queried.
> Setup:
> {code:java}
> create 'test', 'a', 'b'
> test = get_table 'test'
> test.put '1', 'a:1', nil
> test.put '1', 'a:10', nil
> test.put '1', 'b:2', nil
> {code}
>
> This query works fine when the SCVF's qualifier has length 1 (i.e. '1') :
> {code:java}
> test.scan({ FILTER => "(
> SingleColumnValueFilter('a','1',=,'binary:',true,true) AND
> FamilyFilter(=,'binary:b') )"})
> ROW COLUMN+CELL
> 1column=b:2,
> timestamp=1520455888059, value=
> 1 row(s) in 0.0060 seconds
> {code}
>
> The query should return the same result when passed a qualifier of length 2
> (i.e. '10') :
> {code:java}
> test.scan({ FILTER => "(
> SingleColumnValueFilter('a','10',=,'binary:',true,true) AND
> FamilyFilter(=,'binary:b') )"})
> ROW COLUMN+CELL
> 0 row(s) in 0.0110 seconds
> {code}
> However, in this case, it does not return any row (expected result would be
> to return the same result as the first query).
>
> Removing the family filter while the qualifier is '10' yields expected
> results:
> {code:java}
> test.scan({ FILTER => "(
> SingleColumnValueFilter('a','10',=,'binary:',true,true) )"})
> ROW COLUMN+CELL
> 1column=a:1,
> timestamp=1520455887954, value=
> 1column=a:10,
> timestamp=1520455888024, value=
> 1column=b:2,
> timestamp=1520455888059, value=
> 1 row(s) in 0.0140 seconds
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23069) periodic dependency bump for Sep 2019
[ https://issues.apache.org/jira/browse/HBASE-23069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-23069: Fix Version/s: (was: 1.5.0) 1.5.1 1.6.0 > periodic dependency bump for Sep 2019 > - > > Key: HBASE-23069 > URL: https://issues.apache.org/jira/browse/HBASE-23069 > Project: HBase > Issue Type: Improvement > Components: dependencies, hbase-thirdparty >Reporter: Sean Busbey >Priority: Critical > Fix For: 3.0.0, 2.3.0, 1.6.0, 1.5.1 > > > we should do a pass to see if there are any dependencies we can bump. (also > follow-on we should automate this check) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-13160) SplitLogWorker does not pick up the task immediately
[
https://issues.apache.org/jira/browse/HBASE-13160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-13160:
Fix Version/s: (was: 1.5.0)
1.6.0
> SplitLogWorker does not pick up the task immediately
>
>
> Key: HBASE-13160
> URL: https://issues.apache.org/jira/browse/HBASE-13160
> Project: HBase
> Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Major
> Fix For: 1.6.0
>
> Attachments: hbase-13160_v1.patch
>
>
> We were reading some code with Jeffrey, and we realized that the
> SplitLogWorker's internal task loop is weird. It does {{ls}} every second and
> sleeps, but have another mechanism to learn about new tasks, but does not
> make affective use of the zk notification.
> I have a simple patch which might improve this area.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-12943) Set sun.net.inetaddr.ttl in HBase
[ https://issues.apache.org/jira/browse/HBASE-12943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-12943: Fix Version/s: (was: 1.5.0) 1.5.1 1.6.0 > Set sun.net.inetaddr.ttl in HBase > - > > Key: HBASE-12943 > URL: https://issues.apache.org/jira/browse/HBASE-12943 > Project: HBase > Issue Type: Bug >Reporter: Shaohui Liu >Assignee: Shaohui Liu >Priority: Major > Fix For: 3.0.0, 2.3.0, 1.6.0, 1.5.1 > > Attachments: 12943-1-master.txt > > > The default value of config: sun.net.inetaddr.ttl is -1 and the java > processes will cache the mapping of hostname to ip address forever, See: > http://docs.oracle.com/javase/7/docs/technotes/guides/net/properties.html > But things go wrong when a regionserver with same hostname and different ip > address rejoins the hbase cluster. The HMaster will get wrong ip address of > the regionserver from this cache and every region assignment to this > regionserver will be blocked for a time because the HMaster can't communicate > with the regionserver. > A tradeoff is to set the sun.net.inetaddr.ttl to 10m or 1h and make the wrong > cache expired. > Suggestions are welcomed. Thanks~ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-20993) [Auth] IPC client fallback to simple auth allowed doesn't work
[
https://issues.apache.org/jira/browse/HBASE-20993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-20993:
Fix Version/s: (was: 1.5.0)
1.5.1
1.4.11
1.6.0
> [Auth] IPC client fallback to simple auth allowed doesn't work
> --
>
> Key: HBASE-20993
> URL: https://issues.apache.org/jira/browse/HBASE-20993
> Project: HBase
> Issue Type: Bug
> Components: Client, IPC/RPC, security
>Affects Versions: 1.2.6, 1.3.2, 1.2.7, 1.4.7
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Critical
> Fix For: 1.6.0, 1.4.11, 1.3.7, 1.5.1
>
> Attachments: HBASE-20993.001.patch,
> HBASE-20993.003.branch-1.flowchart.png, HBASE-20993.branch-1.002.patch,
> HBASE-20993.branch-1.003.patch, HBASE-20993.branch-1.004.patch,
> HBASE-20993.branch-1.005.patch, HBASE-20993.branch-1.006.patch,
> HBASE-20993.branch-1.007.patch, HBASE-20993.branch-1.008.patch,
> HBASE-20993.branch-1.009.patch, HBASE-20993.branch-1.009.patch,
> HBASE-20993.branch-1.010.patch, HBASE-20993.branch-1.011.patch,
> HBASE-20993.branch-1.012.patch, HBASE-20993.branch-1.013.patch,
> HBASE-20993.branch-1.2.001.patch, HBASE-20993.branch-1.wip.002.patch,
> HBASE-20993.branch-1.wip.patch, yetus-local-testpatch-output-009.txt
>
>
> It is easily reproducible.
> client's hbase-site.xml: hadoop.security.authentication:kerberos,
> hbase.security.authentication:kerberos,
> hbase.ipc.client.fallback-to-simple-auth-allowed:true, keytab and principal
> are right set
> A simple auth hbase cluster, a kerberized hbase client application.
> application trying to r/w/c/d table will have following exception:
> {code}
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:617)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$700(RpcClientImpl.java:162)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:743)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:740)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:740)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:906)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:873)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1241)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:58383)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.isMasterRunning(ConnectionManager.java:1592)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1530)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1552)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1581)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1738)
> at
> org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4297)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4289)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsyncV2(HBaseAdmin.java:753)
> at
>
[jira] [Updated] (HBASE-11290) Unlock RegionStates
[ https://issues.apache.org/jira/browse/HBASE-11290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-11290: Fix Version/s: (was: 1.5.0) 1.6.0 > Unlock RegionStates > --- > > Key: HBASE-11290 > URL: https://issues.apache.org/jira/browse/HBASE-11290 > Project: HBase > Issue Type: Sub-task >Affects Versions: 1.2.0, 1.3.0 >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > Fix For: 1.6.0 > > Attachments: HBASE-11290-0.98.patch, HBASE-11290-0.98_v2.patch, > HBASE-11290.01.branch-1.patch, HBASE-11290.02.patch, HBASE-11290.03.patch, > HBASE-11290.draft.patch, HBASE-11290_trunk.patch > > > Even though RegionStates is a highly accessed data structure in HMaster. Most > of it's methods are synchronized. Which limits concurrency. Even simply > making some of the getters non-synchronized by using concurrent data > structures has helped with region assignments. We can go as simple as this > approach or create locks per region or a bucket lock per region bucket. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-21800) RegionServer aborted due to NPE from MetaTableMetrics coprocessor
[
https://issues.apache.org/jira/browse/HBASE-21800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-21800:
Fix Version/s: (was: 1.5.0)
> RegionServer aborted due to NPE from MetaTableMetrics coprocessor
> -
>
> Key: HBASE-21800
> URL: https://issues.apache.org/jira/browse/HBASE-21800
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors, meta, metrics, Operability
>Reporter: Sakthi
>Assignee: Sakthi
>Priority: Critical
> Labels: Meta
> Fix For: 3.0.0, 2.2.0, 1.4.10, 2.3.0, 2.1.4
>
> Attachments: hbase-21800.branch-1.001.patch,
> hbase-21800.branch-1.002.patch, hbase-21800.branch-1.003.patch,
> hbase-21800.branch-1.004.patch, hbase-21800.master.001.patch,
> hbase-21800.master.002.patch, hbase-21800.master.003.patch
>
>
> I was just playing around the code, trying to capture "Top k" table metrics
> from MetaMetrics, when I bumped into this issue. Though currently we are not
> capturing "Top K" table metrics, but we can encounter this issue because of
> the "Top k Clients" that is implemented using the LossyAlgo.
>
> RegionServer gets aborted due to a NPE from MetaTableMetrics coprocessor. The
> log looks somewhat like this:
> {code:java}
> 2019-01-28 23:31:10,311 ERROR
> [RpcServer.priority.FPBQ.Fifo.handler=19,queue=1,port=16020]
> coprocessor.CoprocessorHost: The coprocessor
> org.apache.hadoop.hbase.coprocessor.MetaTableMetrics threw
> java.lang.NullPointerException
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.markMeterIfPresent(MetaTableMetrics.java:123)
> at
> org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.tableMetricRegisterAndMark2(MetaTableMetrics.java:233)
> at
> org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.preGetOp(MetaTableMetrics.java:82)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$19.call(RegionCoprocessorHost.java:840)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$19.call(RegionCoprocessorHost.java:837)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:551)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:625)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preGet(RegionCoprocessorHost.java:837)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2608)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2547)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41998)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> 2019-01-28 23:31:10,314 ERROR
> [RpcServer.priority.FPBQ.Fifo.handler=19,queue=1,port=16020]
> regionserver.HRegionServer: * ABORTING region server
> 10.0.0.24,16020,1548747043814: The coprocessor
> org.apache.hadoop.hbase.coprocessor.MetaTableMetrics threw
> java.lang.NullPointerException *
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.markMeterIfPresent(MetaTableMetrics.java:123)
> at
> org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.tableMetricRegisterAndMark2(MetaTableMetrics.java:233)
> at
> org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.preGetOp(MetaTableMetrics.java:82)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$19.call(RegionCoprocessorHost.java:840)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$19.call(RegionCoprocessorHost.java:837)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:551)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:625)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preGet(RegionCoprocessorHost.java:837)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2608)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2547)
> at
>
[jira] [Updated] (HBASE-22667) [Flush] NPE when region flushs
[
https://issues.apache.org/jira/browse/HBASE-22667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-22667:
Fix Version/s: (was: 1.4.7)
(was: 1.5.0)
> [Flush] NPE when region flushs
> --
>
> Key: HBASE-22667
> URL: https://issues.apache.org/jira/browse/HBASE-22667
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.6
>Reporter: Reid Chan
>Priority: Critical
>
> {code}
> 2019-07-09 08:02:14,262 FATAL
> org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server
> hostname,16020,1562233574704: Replay of WAL required. Forcing server shutdown
> org.apache.hadoop.hbase.DroppedSnapshotException: region:
> namespace:table,963,1562296120996.b8e2f19748d374d192b93f106a0f73b3.
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2646)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2322)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2284)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2170)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:2095)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:508)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:478)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$900(MemStoreFlusher.java:76)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:264)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at java.util.ArrayList.(ArrayList.java:177)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.updateReaders(StoreScanner.java:863)
> at
> org.apache.hadoop.hbase.regionserver.HStore.notifyChangedReadersObservers(HStore.java:1172)
> at
> org.apache.hadoop.hbase.regionserver.HStore.updateStorefiles(HStore.java:1145)
> at
> org.apache.hadoop.hbase.regionserver.HStore.access$900(HStore.java:122)
> at
> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.commit(HStore.java:2505)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2600)
> ... 9 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-22421) branch-1 docker mode for yetus fails
[
https://issues.apache.org/jira/browse/HBASE-22421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-22421:
Fix Version/s: (was: 1.5.0)
> branch-1 docker mode for yetus fails
>
>
> Key: HBASE-22421
> URL: https://issues.apache.org/jira/browse/HBASE-22421
> Project: HBase
> Issue Type: Bug
> Components: build, test
>Affects Versions: 1.5.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Fix For: 1.4.10, 1.3.5
>
> Attachments: HBASE-22421-branch-1.v0.patch,
> HBASE-22421-branch-1.v1.addendum.patch, HBASE-22421-branch-1.v1.patch
>
>
> started showing up on precommit runs yesterday. I presume a new version of
> rubocop came out that updated the minimum ruby version
> {code}
> 02:46:28 Building native extensions. This could take a while...
> 02:46:29 [91mERROR: Error installing rubocop:
> 02:46:29 rubocop requires Ruby version >= 2.3.0.
> 02:46:29 [0mSuccessfully installed jaro_winkler-1.5.2
> 02:46:30 The command '/bin/sh -c gem install rubocop' returned a non-zero
> code: 1
> 02:46:30 ERROR: Docker failed to build yetus/hbase:61288f8.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-21940) TestMasterShutdown fails on branch-1
[ https://issues.apache.org/jira/browse/HBASE-21940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-21940: Fix Version/s: (was: 1.5.0) > TestMasterShutdown fails on branch-1 > > > Key: HBASE-21940 > URL: https://issues.apache.org/jira/browse/HBASE-21940 > Project: HBase > Issue Type: Bug >Affects Versions: 1.5.0 >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Critical > > TestMasterShutdown is always failing on branch-1. > Git bisect pointed me to this commit: > f3f3798575527df45961f4cbdb6d4c1d04cfb1e3 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-22448) Scan is slow for Multiple Column prefixes
[ https://issues.apache.org/jira/browse/HBASE-22448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-22448: Fix Version/s: (was: 1.4.10) (was: 1.5.0) > Scan is slow for Multiple Column prefixes > - > > Key: HBASE-22448 > URL: https://issues.apache.org/jira/browse/HBASE-22448 > Project: HBase > Issue Type: Bug > Components: Scanners >Affects Versions: 1.4.8, 1.4.9 >Reporter: Karthick >Assignee: Zheng Hu >Priority: Critical > Labels: prefix, scan, scanner > Attachments: 0001-benchmark-UT.patch, HBaseFileImport.java, > filter-list-with-or-internal-2.png, > org.apache.hadoop.hbase.filter.TestSlowColumnPrefix-output.zip, > qualifiers.txt, scanquery.txt > > > While scanning a row (around 10 lakhs columns) with 100 column prefixes, it > takes around 4 seconds in hbase-1.2.5 and when the same query is executed in > hbase-1.4.9 it takes around 50 seconds. > Is there any way to optimise this? > > *P.S:* > We have applied the patch provided in > [-HBASE-21620-|https://jira.apache.org/jira/browse/HBASE-21620] and > [-HBASE-21734-|https://jira.apache.org/jira/browse/HBASE-21734] . Attached > *qualifiers*.*txt* file which contains the column keys. Use the > *HBaseFileImport.java* file provided to populate in your table and use > *scanquery.txt* to query. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-20047) AuthenticationTokenIdentifier should provide a toString
[
https://issues.apache.org/jira/browse/HBASE-20047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-20047:
Fix Version/s: (was: 1.5.0)
> AuthenticationTokenIdentifier should provide a toString
> ---
>
> Key: HBASE-20047
> URL: https://issues.apache.org/jira/browse/HBASE-20047
> Project: HBase
> Issue Type: Improvement
> Components: Usability
>Reporter: Sean Busbey
>Assignee: maoling
>Priority: Minor
> Labels: beginner
> Fix For: 3.0.0, 2.1.0, 1.3.3, 1.4.10
>
> Attachments: HBASE-20047.master.v0.patch, HBASE-20047.master.v1.patch
>
>
> It'd be easier to debug things like MapReduce and Spark jobs if our
> AuthenticationTokenIdentifier provided a toString method.
> For comparison, here's an example of a MapReduce job that has both an HDFS
> delegation token and our delegation token:
> {code:java}
> 18/02/21 20:40:06 INFO mapreduce.JobSubmitter: Kind: HBASE_AUTH_TOKEN,
> Service: 92a63bd8-9e00-4c04-ab61-da8e606068e1, Ident:
> (org.apache.hadoop.hbase.security.token.AuthenticationTokenIdentifier@17)
> 18/02/21 20:40:06 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN,
> Service: 172.31.118.118:8020, Ident: (token for some_user:
> HDFS_DELEGATION_TOKEN owner=some_u...@example.com, renewer=yarn, realUser=,
> issueDate=1519274405003, maxDate=1519879205003, sequenceNumber=23,
> masterKeyId=9)
> {code}
> Stuff in TokenIdentifier is supposed to be public, so we should be fine to
> dump everything, similar to Hadoop's AbstractDelegationTokenIdentifier.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-22822) Re/Un-schedule balancer chore for balance_switch
[ https://issues.apache.org/jira/browse/HBASE-22822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-22822: Fix Version/s: (was: 2.2.1) (was: 1.5.0) (was: 3.0.0) > Re/Un-schedule balancer chore for balance_switch > > > Key: HBASE-22822 > URL: https://issues.apache.org/jira/browse/HBASE-22822 > Project: HBase > Issue Type: Improvement > Components: Balancer, master >Affects Versions: 3.0.0, 1.5.0, 2.2.1 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > > balance_switch turns on/off balancer. When it is turned off, we don't remove > balancer chore from scheduled chores hence it keeps running only to > eventually find out that it is not supposed to perform any action(if balancer > was turned off). We can unschedule the chore to prevent the chore() execution > and reschedule it when it is turned on by balance_switch. > This should also facilitate running balancer immediately after triggering > balance_switch true, and then chore would continue running as per duration > provided in hbase.balancer.period. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-16848) Usage for show_peer_tableCFs command doesn't include peer
[
https://issues.apache.org/jira/browse/HBASE-16848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-16848:
Fix Version/s: (was: 1.5.0)
> Usage for show_peer_tableCFs command doesn't include peer
> -
>
> Key: HBASE-16848
> URL: https://issues.apache.org/jira/browse/HBASE-16848
> Project: HBase
> Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Peter Somogyi
>Priority: Minor
> Labels: documentation, replication, shell
> Fix For: 1.3.3, 1.4.3, 2.0.0, 1.2.7
>
> Attachments: HBASE-16848.master.001.patch,
> HBASE-16848.master.001.patch
>
>
> {code}
> hbase(main):003:0> show_peer_tableCFs
> ERROR: wrong number of arguments (0 for 1)
> Here is some help for this command:
> Show replicable table-cf config for the specified peer.
> hbase> show_peer_tableCFs
> {code}
> The sample usage should include peer id.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-22430) hbase-vote should tee build and test output to console
[ https://issues.apache.org/jira/browse/HBASE-22430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-22430: Fix Version/s: (was: 1.5.0) > hbase-vote should tee build and test output to console > -- > > Key: HBASE-22430 > URL: https://issues.apache.org/jira/browse/HBASE-22430 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Trivial > Fix For: 3.0.0, 2.2.0, 1.4.10, 2.3.0, 1.3.5 > > Attachments: HBASE-22430.patch, HBASE-22430.patch > > > The hbase-vote script should tee the build and test output to console in > addition to the output file so the user does not become suspicious about > progress. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23128) Restore Region interface compatibility
[
https://issues.apache.org/jira/browse/HBASE-23128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946161#comment-16946161
]
Andrew Kyle Purtell commented on HBASE-23128:
-
I'm going to merge the PR as soon as the compat report verifies the issue with
Region is fixed so I can get out the 1.5.0 RC4 but we can always circle back if
there's something that needs changing.
> Restore Region interface compatibility
> ---
>
> Key: HBASE-23128
> URL: https://issues.apache.org/jira/browse/HBASE-23128
> Project: HBase
> Issue Type: Bug
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Blocker
> Fix For: 1.5.0
>
>
> Adding methods to a Public interface is ok for a minor release, removing
> methods is not. We need to restore
> {code}
> abstract method boolean bulkLoadHFiles (
> Collection>, boolean, Region.BulkLoadListener)
> {code}
> to the Region interface in order to maintain binary compatibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-22356) API to get hdfs block distribution from regionservers
[ https://issues.apache.org/jira/browse/HBASE-22356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-22356: Fix Version/s: (was: 1.5.0) 1.5.1 1.6.0 > API to get hdfs block distribution from regionservers > - > > Key: HBASE-22356 > URL: https://issues.apache.org/jira/browse/HBASE-22356 > Project: HBase > Issue Type: Sub-task > Components: API, Balancer, regionserver >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Major > Labels: balancer > Fix For: 3.0.0, 1.6.0, 2.2.2, 1.5.1 > > Attachments: HBASE-22356.master.001.patch, > HBASE-22356.master.002.patch > > > A RegionServer API has to be added which will return HDFSBlockDistribution > for all the regions it hosts. RS already has this info cached and updated > when flush/compaction happens. Master can query and get instead of hitting > the namenode and caching. The larger the cluster becomes, the more costly it > becomes to get this information and more stale the cached information becomes. > This jira is only to add the API to regionserver. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-17890) FuzzyRowFilter fail if unaligned support is false
[
https://issues.apache.org/jira/browse/HBASE-17890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-17890:
Fix Version/s: (was: 1.5.0)
1.5.1
1.6.0
> FuzzyRowFilter fail if unaligned support is false
> -
>
> Key: HBASE-17890
> URL: https://issues.apache.org/jira/browse/HBASE-17890
> Project: HBase
> Issue Type: Sub-task
> Components: util
>Affects Versions: 1.2.5, 2.0.0
>Reporter: Jerry He
>Assignee: Chia-Ping Tsai
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 1.6.0, 1.5.1
>
> Attachments: HBASE-17890.v0.branch-1.patch, HBASE-17890.v0.patch,
> HBASE-17890.v1.branch-1.patch, HBASE-17890.v1.patch, HBASE-17890.v2.patch,
> HBASE-17890.v3.patch, HBASE-17890.v3.patch, HBASE-17890.v3.patch,
> HBASE-17890.v3.patch, HBASE-17890.v3.patch
>
>
> When unaligned support is false, FuzzyRow tests fail:
> {noformat}
> Failed tests:
> TestFuzzyRowAndColumnRangeFilter.Test:134->runTest:157->runScanner:186
> expected:<10> but was:<0>
> TestFuzzyRowFilter.testSatisfiesForward:81 expected: but was:
> TestFuzzyRowFilter.testSatisfiesReverse:121 expected: but
> was:
> TestFuzzyRowFilterEndToEnd.testEndToEnd:247->runTest1:278->runScanner:343
> expected:<6250> but was:<0>
> TestFuzzyRowFilterEndToEnd.testFilterList:385->runTest:417->runScanner:445
> expected:<5> but was:<0>
> TestFuzzyRowFilterEndToEnd.testHBASE14782:204 expected:<6> but was:<0>
> {noformat}
> This can be reproduced in the case described in HBASE-17869. Or on a platform
> really without unaligned support.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-21140) Backport 'HBASE-21136 NPE in MetricsTableSourceImpl.updateFlushTime' to branch-1 . (and backport HBASE-15728 for branch-1)
[ https://issues.apache.org/jira/browse/HBASE-21140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-21140: Fix Version/s: (was: 1.5.0) 1.5.1 1.6.0 > Backport 'HBASE-21136 NPE in MetricsTableSourceImpl.updateFlushTime' to > branch-1 . (and backport HBASE-15728 for branch-1) > --- > > Key: HBASE-21140 > URL: https://issues.apache.org/jira/browse/HBASE-21140 > Project: HBase > Issue Type: Bug > Components: metrics >Reporter: Duo Zhang >Assignee: Xu Cang >Priority: Major > Fix For: 1.6.0, 1.5.1 > > Attachments: > HBASE-21140.diff_against_cf198a65e8d704d28538c4c165a941b9e5bac678.branch-1.001.patch > > > There is no computeIfAbsent method on branch-1 as we still need to support > JDK7, so the fix will be different with branch-2+. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-21492) CellCodec Written To WAL Before It's Verified
[
https://issues.apache.org/jira/browse/HBASE-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-21492:
Fix Version/s: (was: 1.5.0)
> CellCodec Written To WAL Before It's Verified
> -
>
> Key: HBASE-21492
> URL: https://issues.apache.org/jira/browse/HBASE-21492
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 1.2.7, 2.0.2
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Critical
> Fix For: 3.0.0, 2.2.0, 2.1.2, 1.2.10, 2.0.4, 1.4.10, 1.3.4
>
> Attachments: HBASE-21492-branch-1.patch, HBASE-21492.1.patch,
> HBASE-21492.2.patch, HBASE-21492.2.patch
>
>
> The cell codec class name is written into the WAL file, but the cell codec
> class is not actually verified to exist. Therefore, users can inadvertently
> configure an invalid class name and it will be recorded into the WAL file.
> At that point, the WAL file becomes unreadable and blocks processing of all
> other WAL files.
> {code:java|title=AbstractProtobufLogWriter.java}
> private WALHeader buildWALHeader0(Configuration conf, WALHeader.Builder
> builder) {
> if (!builder.hasWriterClsName()) {
> builder.setWriterClsName(getWriterClassName());
> }
> if (!builder.hasCellCodecClsName()) {
> builder.setCellCodecClsName(WALCellCodec.getWALCellCodecClass(conf));
> }
> return builder.build();
> }
> {code}
> https://github.com/apache/hbase/blob/025ddce868eb06b4072b5152c5ffae5a01e7ae30/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractProtobufLogWriter.java#L78-L86
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-19468) FNFE during scans and flushes
[ https://issues.apache.org/jira/browse/HBASE-19468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-19468: Fix Version/s: (was: 1.5.0) > FNFE during scans and flushes > - > > Key: HBASE-19468 > URL: https://issues.apache.org/jira/browse/HBASE-19468 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Affects Versions: 1.3.1 >Reporter: Thiruvel Thirumoolan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 1.3.2, 1.4.1, 2.0.0 > > Attachments: HBASE-19468-poc.patch, HBASE-19468_1.4.patch, > HBASE-19468_master.patch > > > We see FNFE exceptions on our 1.3 clusters when scans and flushes happen at > the same time. This causes regionserver to throw a UnknownScannerException > and client retries. > This happens during the following sequence: > 1. Scanner open, client fetched some rows from regionserver and working on it > 2. Flush happens and storeScanner is updated with flushed files > (StoreScanner.updateReaders()) > 3. Compaction happens on the region while scanner is still open > 4. compaction discharger runs and cleans up the newly flushed file as we > don't have new scanners on it yet. > 5. Client issues scan.next and during StoreScanner.resetScannerStack(), we > get a FNFE. RegionServer throws a UnknownScannerThe client retries in 1.3. > With branch-1.4, the scan fails with a DoNotRetryIOException. > [~ram_krish], My proposal is to increment the reader count during > updateReaders() and decrement it during resetScannerStack(), so discharger > doesn't clean it up. Scan lease expiries also have to be taken care of. Am I > missing anything? Is there a better approach? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-21553) schedLock not released in MasterProcedureScheduler
[ https://issues.apache.org/jira/browse/HBASE-21553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-21553: Fix Version/s: (was: 1.5.0) > schedLock not released in MasterProcedureScheduler > -- > > Key: HBASE-21553 > URL: https://issues.apache.org/jira/browse/HBASE-21553 > Project: HBase > Issue Type: Bug > Components: proc-v2 >Reporter: Xu Cang >Assignee: Xu Cang >Priority: Critical > Fix For: 1.3.3, 1.4.10 > > Attachments: HBASE-21553-branch-1.001.patch, > HBASE-21553-branch-1.002.patch > > > https://github.com/apache/hbase/blob/branch-1/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/MasterProcedureScheduler.java#L749 > As shown above, we didn't unlock schedLock which can cause deadlock. > Besides this, there are other places in this class handles schedLock.unlock > in a risky manner. I'd like to move them to finally block to improve the > robustness of handling locks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-21590) Optimize trySkipToNextColumn in StoreScanner a bit
[ https://issues.apache.org/jira/browse/HBASE-21590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-21590: Fix Version/s: (was: 1.5.0) > Optimize trySkipToNextColumn in StoreScanner a bit > -- > > Key: HBASE-21590 > URL: https://issues.apache.org/jira/browse/HBASE-21590 > Project: HBase > Issue Type: Improvement > Components: Performance, Scanners >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Critical > Fix For: 3.0.0, 2.2.0, 2.0.4, 1.4.10, 2.1.3 > > Attachments: 21590-1.5.txt, HBASE-21590-master.txt, > HBASE-21590.addendum.v1.patch > > > See latest comment on HBASE-17958 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-21196) HTableMultiplexer clears the meta cache after every put operation
[
https://issues.apache.org/jira/browse/HBASE-21196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-21196:
Fix Version/s: (was: 1.5.0)
> HTableMultiplexer clears the meta cache after every put operation
> -
>
> Key: HBASE-21196
> URL: https://issues.apache.org/jira/browse/HBASE-21196
> Project: HBase
> Issue Type: Bug
> Components: Performance
>Affects Versions: 3.0.0, 1.3.3, 2.2.0
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Critical
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3, 1.4.10, 1.3.4, 1.2.11
>
> Attachments: HBASE-21196-branch-1.patch,
> HBASE-21196.master.001.patch, HBASE-21196.master.001.patch,
> HBASE-21196.master.002.patch, HTableMultiplexer1000Puts.UT.txt
>
>
> *Problem:* Operations which use
> {{AsyncRequestFutureImpl.receiveMultiAction(MultiAction, ServerName,
> MultiResponse, int)}} API with tablename set to null reset the meta cache of
> the corresponding server after each call. One such operation is put operation
> of HTableMultiplexer (Might not be the only one). This may impact the
> performance of the system severely as all new ops directed to that server
> will have to go to zk first to get the meta table address and then get the
> location of the table region as it will become empty after every
> htablemultiplexer put.
> From the logs below, one can see after every other put the cached region
> locations are cleared. As a side effect of this, before every put the server
> needs to contact zk and get meta table location and read meta to get region
> locations of the table.
> {noformat}
> 2018-09-13 22:21:15,467 TRACE [htable-pool11-t1] client.MetaCache(283):
> Removed all cached region locations that map to
> root1-thinkpad-t440p,35811,1536857446588
> 2018-09-13 22:21:15,467 DEBUG [HTableFlushWorker-5]
> client.HTableMultiplexer$FlushWorker(632): Processed 1 put requests for
> root1-ThinkPad-T440p:35811 and 0 failed, latency for this send: 5
> 2018-09-13 22:21:15,515 TRACE
> [RpcServer.reader=1,bindAddress=root1-ThinkPad-T440p,port=35811]
> ipc.RpcServer$Connection(1954): RequestHeader call_id: 218 method_name: "Get"
> request_param: true priority: 0 timeout: 6 totalRequestSize: 137 bytes
> 2018-09-13 22:21:15,515 TRACE
> [RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=35811]
> ipc.CallRunner(105): callId: 218 service: ClientService methodName: Get size:
> 137 connection: 127.0.0.1:42338 executing as root1
> 2018-09-13 22:21:15,515 TRACE
> [RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=35811]
> ipc.RpcServer(2356): callId: 218 service: ClientService methodName: Get size:
> 137 connection: 127.0.0.1:42338 param: region=
> testHTableMultiplexer_1,,1536857451720.304d914b641a738624937c7f9b4d684f.,
> row=\x00\x00\x00\xC4 connection: 127.0.0.1:42338, response result {
> associated_cell_count: 1 stale: false } queueTime: 0 processingTime: 0
> totalTime: 0
> 2018-09-13 22:21:15,516 TRACE
> [RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=35811]
> io.BoundedByteBufferPool(106): runningAverage=16384, totalCapacity=0,
> count=0, allocations=1
> 2018-09-13 22:21:15,516 TRACE [main] ipc.AbstractRpcClient(236): Call: Get,
> callTime: 2ms
> 2018-09-13 22:21:15,516 TRACE [main] client.ClientScanner(122): Scan
> table=hbase:meta,
> startRow=testHTableMultiplexer_1,\x00\x00\x00\xC5,99
> 2018-09-13 22:21:15,516 TRACE [main] client.ClientSmallReversedScanner(179):
> Advancing internal small scanner to startKey at
> 'testHTableMultiplexer_1,\x00\x00\x00\xC5,99'
> 2018-09-13 22:21:15,517 TRACE [main] client.ZooKeeperRegistry(59): Looking up
> meta region location in ZK,
> connection=org.apache.hadoop.hbase.client.ZooKeeperRegistry@599f571f
> {noformat}
> From the minicluster logs [^HTableMultiplexer1000Puts.UT.txt] one can see
> that the string "Removed all cached region locations that map" and "Looking
> up meta region location in ZK" are present for every put.
> *Analysis:*
> The problem occurs as we call the {{cleanServerCache}} method always clears
> the server cache in case tablename is null and exception is null. See
> [AsyncRequestFutureImpl.java#L918|https://github.com/apache/hbase/blob/5d14c1af65c02f4e87059337c35e4431505de91c/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRequestFutureImpl.java#L918]
> {code:java}
> private void cleanServerCache(ServerName server, Throwable regionException) {
> if (tableName == null &&
> ClientExceptionsUtil.isMetaClearingException(regionException)) {
> // For multi-actions, we don't have a table name, but we want to make
> sure to clear the
> // cache in case there were location-related exceptions. We don't to
> clear the cache
> // for
[jira] [Updated] (HBASE-22235) OperationStatus.{SUCCESS|FAILURE|NOT_RUN} are not visible to 3rd party coprocessors
[
https://issues.apache.org/jira/browse/HBASE-22235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-22235:
Fix Version/s: (was: 1.5.0)
> OperationStatus.{SUCCESS|FAILURE|NOT_RUN} are not visible to 3rd party
> coprocessors
> ---
>
> Key: HBASE-22235
> URL: https://issues.apache.org/jira/browse/HBASE-22235
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors
>Reporter: Lars Hofhansl
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 1.4.10, 2.3.0, 1.2.12, 2.1.5, 1.3.5
>
> Attachments: HBASE-22235-branch-1.patch, HBASE-22235.patch,
> HBASE-22235.patch, HBASE-22235.patch
>
>
> preBatchMutate is useless for some operation due to this.
> See also TEPHRA-299. This looks like an oversight.
> MiniBatchOperationInProgress has limited visibility for coprocessors.
> OperationStatus and OperationStatusCode should have the same.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-22184) [security] Support get|set LogLevel in HTTPS mode
[ https://issues.apache.org/jira/browse/HBASE-22184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-22184: Fix Version/s: (was: 1.5.0) > [security] Support get|set LogLevel in HTTPS mode > - > > Key: HBASE-22184 > URL: https://issues.apache.org/jira/browse/HBASE-22184 > Project: HBase > Issue Type: New Feature > Components: logging, security, website >Reporter: Reid Chan >Assignee: Wei-Chiu Chuang >Priority: Major > Labels: security > Fix For: 3.0.0, 1.4.10, 2.3.0 > > Attachments: HBASE-22184.branch-1.001.patch, > HBASE-22184.branch-1.002.patch, HBASE-22184.master.001.patch, > HBASE-22184.master.002.patch, HBASE-22184.master.003.patch > > > As title read. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-22291) Fix recovery of recovered.edits files under root dir
[ https://issues.apache.org/jira/browse/HBASE-22291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-22291: Fix Version/s: (was: 1.5.0) > Fix recovery of recovered.edits files under root dir > > > Key: HBASE-22291 > URL: https://issues.apache.org/jira/browse/HBASE-22291 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.4.9 >Reporter: Zach York >Assignee: Zach York >Priority: Major > Fix For: 3.0.0, 2.2.0, 1.4.10, 2.3.0, 2.1.5, 1.3.5 > > Attachments: HBASE-22291.branch-1.001.patch, > HBASE-22291.master.001.patch, HBASE-22291.master.002.patch > > > It looks like a few places are using incorrect FS instances in the > replayRecoveredEditsForPath method that was introduced in HBASE-20734. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-22270) master's jmx.clusterRequests could be negative in branch-1
[ https://issues.apache.org/jira/browse/HBASE-22270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-22270: Fix Version/s: (was: 1.5.0) > master's jmx.clusterRequests could be negative in branch-1 > -- > > Key: HBASE-22270 > URL: https://issues.apache.org/jira/browse/HBASE-22270 > Project: HBase > Issue Type: Bug > Components: master, regionserver >Affects Versions: 1.4.9, 1.3.4, 1.2.12 >Reporter: puleya7 >Assignee: puleya7 >Priority: Major > Fix For: 1.4.10, 1.3.5 > > > In 1.x branch, regionserver could report a negative (overflow) requestCount > to master, causing the master's jmx.clusterRequests to become smaller even > negative > HBASE-12444 fixed, but missed a little when backport to branch-1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-22330) Backport HBASE-20724 (Sometimes some compacted storefiles are still opened after region failover) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-22330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-22330: Fix Version/s: (was: 1.5.0) > Backport HBASE-20724 (Sometimes some compacted storefiles are still opened > after region failover) to branch-1 > - > > Key: HBASE-22330 > URL: https://issues.apache.org/jira/browse/HBASE-22330 > Project: HBase > Issue Type: Sub-task > Components: Compaction, regionserver >Affects Versions: 1.5.0, 1.4.9, 1.3.4 >Reporter: Andrew Kyle Purtell >Assignee: Abhishek Singh Chouhan >Priority: Major > Fix For: 1.4.10, 1.3.5 > > Attachments: HBASE-22330-addendum.branch-1.patch, > HBASE-22330.branch-1.001.patch, HBASE-22330.branch-1.002.patch, > HBASE-22330.branch-1.3.001.patch > > > There appears to be a race condition between close and split which when > combined with a side effect of HBASE-20704, leads to the parent region store > files getting archived and cleared while daughter regions still have > references to those parent region store files. > Here is the timeline of events observed for an affected region: > # RS1 faces ZooKeeper connectivity issue for master node and starts shutting > itself down. As part of this it starts to close the store and clean up the > compacted files (File A) > # Master starts bulk assigning regions and assign parent region to RS2 > # Region opens on RS2 and ends up opening compacted store file(s) (suspect > this is due to HBASE-20724) > # Now split happens and daughter regions open on RS2 and try to run a > compaction as part of post open > # Split request at this point is complete. However now archiving proceeds on > RS1 and ends up archiving the store file that is referenced by the daughter. > Compaction fails due to FileNotFoundException and all subsequent attempts to > open the region will fail until manual resolution. > We think having HBASE-20724 would help in such situations since we won't end > up loading compacted store files in the first place. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-22341) Add explicit guidelines for removing deprecations in book
[ https://issues.apache.org/jira/browse/HBASE-22341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-22341: Fix Version/s: (was: 1.5.0) > Add explicit guidelines for removing deprecations in book > - > > Key: HBASE-22341 > URL: https://issues.apache.org/jira/browse/HBASE-22341 > Project: HBase > Issue Type: Improvement > Components: API, community, documentation >Reporter: Jan Hentschel >Assignee: Jan Hentschel >Priority: Major > Fix For: 3.0.0, 2.2.0, 1.4.10, 2.3.0, 2.0.6, 2.1.5, 1.3.5 > > > Based on the discussion on the mailing list about the [removal of deprecated > versions|https://lists.apache.org/thread.html/6abaf9757c515213e530edb4cb80faa6c60007908d448437def77f54@%3Cdev.hbase.apache.org%3E], > the client API compatibility should be extended to make it clear when a > deprecated API will be removed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-22310) checkAndMutate used an incorrect row to check the condition
[ https://issues.apache.org/jira/browse/HBASE-22310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-22310: Fix Version/s: (was: 1.5.0) > checkAndMutate used an incorrect row to check the condition > --- > > Key: HBASE-22310 > URL: https://issues.apache.org/jira/browse/HBASE-22310 > Project: HBase > Issue Type: Bug > Components: API >Affects Versions: 1.3.3, 1.4.9 >Reporter: Adonis Ling >Assignee: Adonis Ling >Priority: Major > Fix For: 1.4.10, 1.3.4 > > Attachments: HBASE-22310-branch-1.patch, > HBASE-22310.branch-1.3.001.patch, HBASE-22310.branch-1.3.002.patch, > HBASE-22310.branch-1.4.001.patch, HBASE-22310.branch-1.4.002.patch, > HBASE-22310.branch-1.4.003.patch > > > In branch-1.4, checkAndMutate used the row of RowMutation to check the > condition which is incorrect. It will fail in the case which is checking a > row and mutate a different row. > The issue doesn't happen in the master branch. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-22392) Remove extra/useless +
[
https://issues.apache.org/jira/browse/HBASE-22392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-22392:
Fix Version/s: (was: 1.5.0)
> Remove extra/useless +
> ---
>
> Key: HBASE-22392
> URL: https://issues.apache.org/jira/browse/HBASE-22392
> Project: HBase
> Issue Type: Improvement
>Reporter: puleya7
>Assignee: puleya7
>Priority: Trivial
> Fix For: 3.0.0, 2.2.0, 1.4.10, 2.3.0, 2.0.6, 2.1.5, 1.3.5
>
> Attachments: HBASE-22392.branch-1.001.patch,
> HBASE-22392.master.001.patch
>
>
> {code:java}
> [HRegion.java]
> LOG.info("Flushing " + + storesToFlush.size() + "/" + stores.size() + "
> column families," +
> [HStore.java]
> ", sequenceid=" + +storeFile.getReader().getSequenceID() + ", filesize="
> [LoadIncrementalHFiles.java]
> LOG.info("Split occurred while grouping HFiles, retry attempt " + +count + "
> with " +{code}
>
> Has 3 extra/useless "+".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-22429) hbase-vote download step requires URL to end with '/'
[ https://issues.apache.org/jira/browse/HBASE-22429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-22429: Fix Version/s: (was: 1.5.0) > hbase-vote download step requires URL to end with '/' > -- > > Key: HBASE-22429 > URL: https://issues.apache.org/jira/browse/HBASE-22429 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Kyle Purtell >Assignee: Tak-Lon (Stephen) Wu >Priority: Trivial > Fix For: 3.0.0, 2.2.0, 1.4.10, 2.3.0, 1.3.5 > > > The hbase-vote script's download step requires the sourcedir URL be > terminated with a path separator or else the retrieval will escape the > candidate's directory and mirror way too much. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-22487) getMostLoadedRegions is unused
[
https://issues.apache.org/jira/browse/HBASE-22487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-22487:
Fix Version/s: (was: 1.5.0)
> getMostLoadedRegions is unused
> --
>
> Key: HBASE-22487
> URL: https://issues.apache.org/jira/browse/HBASE-22487
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Reporter: Clay B.
>Assignee: Clay B.
>Priority: Trivial
> Fix For: 3.0.0, 2.2.0, 1.4.10, 2.3.0, 1.3.5, 2.1.6
>
> Attachments: 0001-HBASE-22487-getMostLoadedRegions-is-unused.patch
>
>
> Reading {{HRegionServer.java}}, I noticed the function
> {{getMostLoadedRegions}} this seems to replicate functionality found now in
> the {{StochasticLoadBalancer.}} Further, it seems the only consumer of this
> function was removed in HBASE-805.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-21766) TestSimpleRpcScheduler is flaky (branch-1)
[ https://issues.apache.org/jira/browse/HBASE-21766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-21766: Fix Version/s: (was: 1.5.0) > TestSimpleRpcScheduler is flaky (branch-1) > -- > > Key: HBASE-21766 > URL: https://issues.apache.org/jira/browse/HBASE-21766 > Project: HBase > Issue Type: Bug > Components: rpc, test >Affects Versions: 1.5.0 >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Trivial > Fix For: 1.4.10, 1.3.4 > > Attachments: HBASE-21766-branch-1.patch > > > Trivial fix for flakiness in TestSimpleRpcScheduler when test host is under > load. See patch. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-22125) Fix more instances in make_rc.sh where we need -Dhttps.protocols=TLSv1.2
[ https://issues.apache.org/jira/browse/HBASE-22125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-22125: Fix Version/s: (was: 1.5.0) > Fix more instances in make_rc.sh where we need -Dhttps.protocols=TLSv1.2 > > > Key: HBASE-22125 > URL: https://issues.apache.org/jira/browse/HBASE-22125 > Project: HBase > Issue Type: Bug >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Trivial > Fix For: 1.4.10, 1.3.4, 1.2.12 > > Attachments: HBASE-22125-branch-1.patch > > > make_rc.sh on branch-1 is missing some places where we need to define the > system property https.protocols=TLSv1.2 in order for JDK 7 to succeed in > accessing Maven resources. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-21818) Incorrect list item in javadoc
[
https://issues.apache.org/jira/browse/HBASE-21818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-21818:
Fix Version/s: (was: 1.5.0)
> Incorrect list item in javadoc
> --
>
> Key: HBASE-21818
> URL: https://issues.apache.org/jira/browse/HBASE-21818
> Project: HBase
> Issue Type: Improvement
> Components: documentation, master
>Affects Versions: 1.1.0
> Environment: has nothing to do with env
>Reporter: qiang Liu
>Assignee: qiang Liu
>Priority: Trivial
> Labels: easyfix, javadoc
> Fix For: 1.4.10, 1.3.4
>
> Attachments: HBASE-21818.branch-1.001.patch,
> HBASE-21818.branch-1.002.patch, HBASE-21818.branch-1.003.patch,
> HBASE-21818.branch-1.004.patch, HBASE-21818.branch-1.005.patch,
> HBASE-21818.branch-1.006.patch, blankLineOfJavaDoc.png
>
> Original Estimate: 1m
> Remaining Estimate: 1m
>
> the java doc of this function type a as ，so dispaly a blanck line
> {code:java}
> //org.apache.hadoop.hbase.master.HMaster#finishActiveMasterInitialization
> {code}
> paste the function and it's doc here , please look the line "Ensure
> assignment of meta/namespace regions"
>
>
> {code:java}
> /**
> * Finish initialization of HMaster after becoming the primary master.
> *
> *
> * Initialize master components - file system manager, server manager,
> * assignment manager, region server tracker, etc
> * Start necessary service threads - balancer, catalog janior,
> * executor services, etc
> * Set cluster as UP in ZooKeeper
> * Wait for RegionServers to check-in
> * Split logs and perform data recovery, if necessary
> * Ensure assignment of meta/namespace regions
> * Handle either fresh cluster start or master failover
> *
> *
> * @throws IOException
> * @throws InterruptedException
> * @throws KeeperException
> * @throws CoordinatedStateException
> */
> private void finishActiveMasterInitialization(MonitoredTask status)
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23110) Backport HBASE-23054 "Remove synchronization block from MetaTableMetrics and fix LossyCounting algorithm" to branch-1
[ https://issues.apache.org/jira/browse/HBASE-23110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946094#comment-16946094 ] Andrew Kyle Purtell commented on HBASE-23110: - I'm going to merge the PR to make progress. > Backport HBASE-23054 "Remove synchronization block from MetaTableMetrics and > fix LossyCounting algorithm" to branch-1 > - > > Key: HBASE-23110 > URL: https://issues.apache.org/jira/browse/HBASE-23110 > Project: HBase > Issue Type: Sub-task > Components: metrics >Reporter: Duo Zhang >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 1.5.0, 1.4.11 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-20172) During coprocessor load, switch classloader only if it's a custom CP.
[ https://issues.apache.org/jira/browse/HBASE-20172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-20172: Fix Version/s: (was: 1.5.0) 1.6.0 > During coprocessor load, switch classloader only if it's a custom CP. > - > > Key: HBASE-20172 > URL: https://issues.apache.org/jira/browse/HBASE-20172 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.0 >Reporter: Ankit Singhal >Assignee: Ankit Singhal >Priority: Major > Fix For: 1.6.0 > > Attachments: HBASE-20172-branch-1.patch, HBASE-20172.patch > > > Current Impact:- > Metric registries will not be able to load its implementation through service > loader and etc. > We are not observing this with Java8 because ServiceLoader uses system class > loader if provided class loader is null but it gets exposed with Java7 > easily(TEPHRA-285) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23126) IntegrationTestRSGroup is useless now
[ https://issues.apache.org/jira/browse/HBASE-23126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946102#comment-16946102 ] Andrew Kyle Purtell commented on HBASE-23126: - Even before there was HBASE-20926 ... > IntegrationTestRSGroup is useless now > - > > Key: HBASE-23126 > URL: https://issues.apache.org/jira/browse/HBASE-23126 > Project: HBase > Issue Type: Bug > Components: rsgroup >Reporter: Duo Zhang >Priority: Major > > It extends TestRSGroupsBase and wants to run all the UTs defined in > TestRSGroupsBase, but after HBASE-21265, all the UTs have been moved to sub > classes... -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-20643) Getting HDFSBlockDist in Master by querying RegionServers
[ https://issues.apache.org/jira/browse/HBASE-20643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-20643: Fix Version/s: (was: 1.5.0) 1.6.0 3.0.0 > Getting HDFSBlockDist in Master by querying RegionServers > - > > Key: HBASE-20643 > URL: https://issues.apache.org/jira/browse/HBASE-20643 > Project: HBase > Issue Type: Improvement > Components: Balancer >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Major > Labels: balancer > Fix For: 3.0.0, 2.3.0, 1.6.0 > > > Region locality information is needed by the balancer to generate region > plans. Computing HDFSBlockDistribution is expensive on larger clusters and > adds load to the NameNode. This also needs to be recomputed on a master > restart. The proposal is to get the HDFSBlockDistribution from the > RegionServers instead of computing it in Master. RS already has this > information and we could just reuse it by querying it. RS already passes > dataLocality info via RegionLoad today. > Proposed Implementation: This is a high-level overview. > # A RegionServer API has to be added which will return HDFSBlockDistribution > for all the regions it hosts. RS already has this info. Since ClusterStatus > has already become bulky and we don’t need updated locality so fast, it’s > better to have another API rather than add this to RegionLoad and pass it > along with RSReport. > # Master will have a Chore to query all RegionServers and will cache the > HDFSBlockDistribution for those regions. This is easy and quick. Admins can > tune the frequency based on size of the cluster. On a ~90 nodes cluster with > 500k regions and a prototype implementation and no load, it took about 5 > seconds to get all HDFSBlockDistribution from RS. > # The cache will be an extension of RegionLocationFinder (subclass), if > needed to keep the implementation simple. Probably will get clear with > implementation. > # Balancer will use the new cache to get all HDFSBlockDistribution. If there > is a new region and Chore didn’t get the block distribution from RS during > its previous run, then it will be computed by RegionLocationFinder the same > way it has been done now. If the Chore runs more frequently like every hour, > then this recomputation will be drastically reduced. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-20926) IntegrationTestRSGroup is broken
[ https://issues.apache.org/jira/browse/HBASE-20926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-20926: Fix Version/s: (was: 1.5.0) 1.5.1 1.6.0 > IntegrationTestRSGroup is broken > - > > Key: HBASE-20926 > URL: https://issues.apache.org/jira/browse/HBASE-20926 > Project: HBase > Issue Type: Bug > Components: integration tests >Affects Versions: 2.1.0 >Reporter: Sergey Soldatov >Assignee: Sergey Soldatov >Priority: Major > Fix For: 3.0.0, 2.3.0, 1.6.0, 1.5.1 > > > There are several problems: > 1. It doesn't work in minicluster mode because in afterMethod() it using > IntegrationTestingUtility.restoreCluster() which just shutdown minicluster in > the not distributed mode > 2. It uses tests from TestRSGroups which was supposed to be common for both > unit and integration tests, but for last two years, there were a number of > tests added that are using internal API, not compatible with the distributed > mode. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-21685) Change repository urls to Gitbox
[ https://issues.apache.org/jira/browse/HBASE-21685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-21685: Fix Version/s: (was: 1.5.0) > Change repository urls to Gitbox > > > Key: HBASE-21685 > URL: https://issues.apache.org/jira/browse/HBASE-21685 > Project: HBase > Issue Type: Task >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Critical > Fix For: 3.0.0, 2.2.0, 1.4.10, 2.1.3, 2.0.5, 1.3.4, 1.2.11 > > Attachments: HBASE-21685.master.001.patch, > HBASE-21685.master.001.patch > > > Moving to Gitbox is approaching and references to git-wip-us need to be > changed to gitbox. > Some of the Jenkins jobs are referring to git-wip-us which if going to be > locked after the migration. We could move them to github so the build flow > will remain intact. > Previous discussion on dev@: > [https://lists.apache.org/thread.html/3496568d6cc002f74f5c3bcce46ed44b7ee9e90d7d53af2c65b6f785@%3Cdev.hbase.apache.org%3E] > After this notify INFRA to make the change -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-21711) Remove references to git.apache.org/hbase.git
[ https://issues.apache.org/jira/browse/HBASE-21711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-21711: Fix Version/s: (was: 1.5.0) > Remove references to git.apache.org/hbase.git > - > > Key: HBASE-21711 > URL: https://issues.apache.org/jira/browse/HBASE-21711 > Project: HBase > Issue Type: Sub-task >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Critical > Fix For: 3.0.0, 2.2.0, 1.4.10, 2.1.3, 2.0.5, 1.3.4, 1.2.11 > > Attachments: HBASE-21711.branch-1.001.patch, > HBASE-21711.master.001.patch > > > With the GitBox migration not only git-wip-us was removed but also > git.apache.org/hbase.git is not available anymore. (INFRA-17640) > We need to remove all references to this url. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-21618) Scan with the same startRow(inclusive=true) and stopRow(inclusive=false) returns one result
[
https://issues.apache.org/jira/browse/HBASE-21618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-21618:
Fix Version/s: (was: 1.5.0)
> Scan with the same startRow(inclusive=true) and stopRow(inclusive=false)
> returns one result
> ---
>
> Key: HBASE-21618
> URL: https://issues.apache.org/jira/browse/HBASE-21618
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.0.2
> Environment: hbase server 2.0.2
> hbase client 2.0.0
>Reporter: Jermy Li
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.2.0, 2.1.2, 2.0.4, 1.4.10
>
> Attachments: HBASE-21618.branch-1.001.patch,
> HBASE-21618.master.001.patch, HBASE-21618.master.002.patch,
> HBASE-21618.master.003.patch
>
>
> I expect the following code to return none result, but still return a row:
> {code:java}
> byte[] rowkey = "some key existed";
> Scan scan = new Scan();
> scan.withStartRow(rowkey, true);
> scan.withStopRow(rowkey, false);
> htable.getScanner(scan);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-21920) Ignoring 'empty' end_key while calculating end_key for new region in HBCK -fixHdfsOverlaps command can cause data loss
[
https://issues.apache.org/jira/browse/HBASE-21920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-21920:
Fix Version/s: (was: 1.5.0)
> Ignoring 'empty' end_key while calculating end_key for new region in HBCK
> -fixHdfsOverlaps command can cause data loss
> --
>
> Key: HBASE-21920
> URL: https://issues.apache.org/jira/browse/HBASE-21920
> Project: HBase
> Issue Type: Bug
> Components: hbck
>Reporter: Syeda Arshiya Tabreen
>Assignee: Syeda Arshiya Tabreen
>Priority: Major
> Fix For: 1.4.10, 1.3.5
>
> Attachments: HBASE-21920.branch-1.001.patch,
> HBASE-21920.branch-1.002.patch, HBASE-21920.branch-1.002.patch,
> HBASE-21920.branch-1.002.patch, HBASE-21920.branch-1.002.patch,
> HBASE-21920.branch-1.patch
>
>
> When running *-fixHdfsOverlaps* command due to overlap in the regions of the
> table ,it moves all the hfiles of overlapping regions into new region with
> start_key and end_key calculating based on minimum and maximum start_key and
> end_key of all overlapping regions.
> When calculating start_key and end_key for new region,end_key with 'empty' is
> not considered which leads to data loss when scanned using '*startrow'.*
> *For example:*
> 1.create table 't'
> 2.Insert records \{00,111,200} into the table 't'and flush the data
> 3.split the table 't' with split-key '100'
> 4.Now we have three regions( 1 parent and two daughter regions )
> 1.*Region-1*('Empty','Empty') => \{00,111,200}
> 2.*Region-2*('Empty','100')=>\{00}
> 3.*Region-3*('100','Empty')=>\{111,200}
> 5.Make sure parent region is not deleted in file system and run
> -*fixHdfsOverlaps* command
> This -*fixHdfsOverlaps* command will move all the hfiles of the three regions
> {*Region-1,Region- 2,Region-3*} into a new region(*Region-4*) created with
> start_key='*Empty'* and end_key='*100'*
> This is because it does not consider end_key=*'Empty'* and considers
> end_key=*'100'* as maximum which in turn makes all the hfiles of three
> regions to move into new region even if records in hfile is more than the
> end_key='*100'* and one empty region *Region -5 (100,Empty)* will be
> created because table region end key was not empty.
> Now we have 2 regions:
> 1.*Region-4*(Empty,100)=>\{00,111,200}
> 2.*Region-5*(100,Empty)=>{}
> when the entire table scan is done, all the records will be displayed, there
> wont be any data loss but scan with start_key is done below are the results:
> 1.scan 't', \{ STARTROW => '00'} => \{00,111,200}
> 2.scan 't', \{ STARTROW => '100'}=>{}
> The second scan will give empty result because it searches the rows in
> *Region -5*(100,Empty) which contains no records but records \{111,200} is
> present in *Region-4*(Empty,100).
> The problem exists only when end_key=*'Empty'* is present in any of the
> overlapping regions.I think if end_key is present in any of the overlapping
> regions,we have to consider it as maximum end_key.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-22152) Create a jenkins file for yetus to processing GitHub PR
[ https://issues.apache.org/jira/browse/HBASE-22152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-22152: Fix Version/s: (was: 1.5.0) > Create a jenkins file for yetus to processing GitHub PR > --- > > Key: HBASE-22152 > URL: https://issues.apache.org/jira/browse/HBASE-22152 > Project: HBase > Issue Type: Sub-task > Components: build >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0, 1.4.10, 1.3.4, 2.3.0, 2.0.6, 1.2.12, 2.1.5 > > Attachments: HBASE-22152-v1.patch, HBASE-22152.patch > > > I think we can just copy the jenkinsfile from the hadoop project. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-22045) Mutable range histogram reports incorrect outliers
[
https://issues.apache.org/jira/browse/HBASE-22045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-22045:
Fix Version/s: (was: 1.5.0)
> Mutable range histogram reports incorrect outliers
> --
>
> Key: HBASE-22045
> URL: https://issues.apache.org/jira/browse/HBASE-22045
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 1.5.0, 2.0.0, 1.4.9, 2.1.3, 2.2.1
>Reporter: Abhishek Singh Chouhan
>Assignee: Abhishek Singh Chouhan
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 1.4.10, 2.3.0, 2.1.5
>
> Attachments: HBASE-22045.master.001.patch
>
>
> MutableRangeHistogram during the snapshot calculates the outliers (eg.
> mutate_TimeRange_60-inf) and adds the counter with incorrect calculation
> by using the overall count of event and not the number of events in the
> snapshot.
> {code:java}
> long val = histogram.getCount();
> if (val - cumNum > 0) {
> metricsRecordBuilder.addCounter(
> Interns.info(name + "_" + rangeType + "_" + ranges[ranges.length -
> 1] + "-inf", desc),
> val - cumNum);
> }{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-22067) Fix log line in StochasticLoadBalancer when balancer is an ill-fit for cluster size
[ https://issues.apache.org/jira/browse/HBASE-22067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-22067: Fix Version/s: (was: 1.5.0) > Fix log line in StochasticLoadBalancer when balancer is an ill-fit for > cluster size > --- > > Key: HBASE-22067 > URL: https://issues.apache.org/jira/browse/HBASE-22067 > Project: HBase > Issue Type: Bug > Components: Balancer >Reporter: Xu Cang >Assignee: Xu Cang >Priority: Major > Labels: balancer > Fix For: 3.0.0, 1.4.10, 1.3.4, 2.3.0 > > Attachments: HBASE-22067.branch-1.001.patch, > HBASE-22067.master.001.patch > > > HBASE-21338 Added log lines regarding load balancer warnings. There is a bug > in log that uses wrong parameter. > 'maxRunningTime' is used , should be maxSteps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-22225) Profiler tab on Master/RS UI not working w/o comprehensive message
[
https://issues.apache.org/jira/browse/HBASE-5?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-5:
Fix Version/s: (was: 1.5.0)
> Profiler tab on Master/RS UI not working w/o comprehensive message
> --
>
> Key: HBASE-5
> URL: https://issues.apache.org/jira/browse/HBASE-5
> Project: HBase
> Issue Type: Improvement
> Components: UI
>Affects Versions: 1.5.0
>Reporter: Yu Li
>Assignee: Andrew Kyle Purtell
>Priority: Minor
> Fix For: 3.0.0, 2.2.0, 1.4.10, 2.3.0, 2.1.5, 1.3.5
>
> Attachments: HBASE-5-branch-1.patch, HBASE-5.patch, Screen
> Shot 2019-04-26 at 4.50.01 PM.jpg
>
>
> As titled, when checking 1.5.0 RC3 binary package, clicking the "Profiler"
> tab on HMaster/RegionServer web UI, it complains page not found error like
> below:
> {noformat}
> Problem accessing /prof. Reason:
> NOT_FOUND
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-21963) Add a script for building and verifying release candidate
[
https://issues.apache.org/jira/browse/HBASE-21963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-21963:
Fix Version/s: (was: 1.5.0)
> Add a script for building and verifying release candidate
> -
>
> Key: HBASE-21963
> URL: https://issues.apache.org/jira/browse/HBASE-21963
> Project: HBase
> Issue Type: Test
> Components: community, scripts
>Affects Versions: 3.0.0, 2.1.3
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Minor
> Fix For: 3.0.0, 2.2.0, 1.4.10, 1.3.4, 2.3.0, 2.0.6, 1.2.12, 2.1.5
>
> Attachments: HBASE-21963.master.001.patch,
> HBASE-21963.master.002.patch, HBASE-21963.master.003.patch,
> HBASE-21963.master.004.patch, HBASE-21963.master.005.patch,
> HBASE-21963.master.006.patch
>
>
> During the release vote for HBase 2.1.3RC1, a driver/helper script was
> mentioned and can potentially help contributors prepare to vote for a release
> candidate. As recommended, we decided to move toward this tool to under
> {{dev-support/}}
> Here the driver script provides the following automation:
> 1. Import and check publisher key(s)
> 2. Checksum of sources and binaries
> 3. Signature of sources and binaries
> 4. Rat check
> 5. Built from source
> 6. Verify unit tests
> {code}
> # example usage
> $ bash dev-support/hbase-vote.sh -s
> https://dist.apache.org/repos/dist/dev/hbase/hbase-1.5.0RC2/
> $ bash dev-support/hbase-vote.sh -h
> hbase-vote. A script for standard vote which verifies the following items
> 1. Checksum of sources and binaries
> 2. Signature of sources and binaries
> 3. Rat check
> 4. Built from source
> 5. Unit tests
> Usage: hbase-vote.sh -s | --source [-k | --key ] [-f |
> --keys-file-url ]
>hbase-vote.sh -h | --help
> -h | --help Show this screen.
> -s | --source '' A URL pointing to the release candidate
> sources and binaries
> e.g.
> https://dist.apache.org/repos/dist/dev/hbase/hbase-RC0/
> -k | --key '' A signature of the public key, e.g. 9AD2AE49
> -f | --keys-file-url '' the URL of the key file, default is
> http://www.apache.org/dist/hbase/KEYS
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-22379) Fix Markdown for "Voting on Release Candidates" in book
[ https://issues.apache.org/jira/browse/HBASE-22379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-22379: Fix Version/s: (was: 1.5.0) > Fix Markdown for "Voting on Release Candidates" in book > --- > > Key: HBASE-22379 > URL: https://issues.apache.org/jira/browse/HBASE-22379 > Project: HBase > Issue Type: Improvement > Components: community, documentation >Reporter: Jan Hentschel >Assignee: Jan Hentschel >Priority: Minor > Fix For: 3.0.0, 2.2.0, 1.4.10, 2.3.0, 2.0.6, 2.1.5, 1.3.5 > > > The Markdown in the section "Voting on Release Candidates" of the HBase book > seems to be broken. It looks like that there should be a quote, which isn't > displayed correctly. Same is true for the formatting of the Maven RAT command. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-22274) Cell size limit check on append should consider cell's previous size.
[
https://issues.apache.org/jira/browse/HBASE-22274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-22274:
Fix Version/s: (was: 1.5.0)
> Cell size limit check on append should consider cell's previous size.
> -
>
> Key: HBASE-22274
> URL: https://issues.apache.org/jira/browse/HBASE-22274
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.0.0, 1.3.5
>Reporter: Xu Cang
>Assignee: Xu Cang
>Priority: Minor
> Fix For: 3.0.0, 2.2.0, 1.4.10, 2.3.0, 2.1.5
>
> Attachments: HBASE-22274-branch-1.001.patch,
> HBASE-22274-branch-1.002.patch, HBASE-22274-branch-1.003.patch,
> HBASE-22274-branch-1.003.patch, HBASE-22274-branch-1.004.patch,
> HBASE-22274-branch-1.005.patch, HBASE-22274-master.001.patch,
> HBASE-22274-master.002.patch, HBASE-22274-master.002.patch,
> HBASE-22274-master.003.patch
>
>
> Now we have cell size limit check based on this parameter
> *hbase.server.keyvalue.maxsize*
> One case was missing: appending to a cell only take append op's cell size
> into account against this limit check. we should check against the potential
> final cell size after the append.'
> It's easy to reproduce this :
>
> Apply this diff
>
> {code:java}
> diff --git
> a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
>
> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> index 5a285ef6ba..8633177ebe 100644 ---
> a/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> +++
> b/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
> @@ -6455,7 +6455,7
> - t.append(new Append(ROW).addColumn(FAMILY, QUALIFIER, new byte[10 *
> 1024]));
> + t.append(new Append(ROW).addColumn(FAMILY, QUALIFIER, new byte[2 * 1024]));
> {code}
>
> Fix is to add this check in #reckonDeltas in HRegion class, where we have
> already got the appended cell's size.
> Will throw DoNotRetryIOException if checks is failed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-22070) Checking restoreDir in RestoreSnapshotHelper
[ https://issues.apache.org/jira/browse/HBASE-22070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-22070: Fix Version/s: (was: 1.5.0) > Checking restoreDir in RestoreSnapshotHelper > - > > Key: HBASE-22070 > URL: https://issues.apache.org/jira/browse/HBASE-22070 > Project: HBase > Issue Type: Bug > Components: snapshots >Affects Versions: 2.0.0 >Reporter: Vincent Choi >Assignee: Vincent Choi >Priority: Minor > Fix For: 3.0.0, 2.2.0, 1.4.10, 1.3.4, 2.3.0, 2.0.6, 2.1.5 > > Attachments: > 0001-fixed-a-bug-of-checking-if-restoreDir-is-subDir-of-r.patch, > HBASE-22070-master-001.patch, HBASE-22070.patch > > > The restoreDir shouldn't be a sub directory of rootDir. The code check it > with a prefix check > "restoreDir.toUri().getPath().startsWith(rootDir.toUri().getPath())". But it > goes error in some reasonable cases. > eg: rootDir = hdfs://user/hbase restoreDir = hdfs://user/hbase_restore. So I > think it's more reasonable to chang the code to > "restoreDir.toUri().getPath().startsWith(rootDir.toUri().getPath() + "/" )". -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-22106) Log cause of the failure when coprocessor specification parsing fails.
[
https://issues.apache.org/jira/browse/HBASE-22106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-22106:
Fix Version/s: (was: 1.5.0)
> Log cause of the failure when coprocessor specification parsing fails.
> --
>
> Key: HBASE-22106
> URL: https://issues.apache.org/jira/browse/HBASE-22106
> Project: HBase
> Issue Type: Bug
> Components: logging
>Affects Versions: 1.4.9
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Minor
> Fix For: 1.4.10, 1.3.4
>
> Attachments: HBASE-22106.branch-1.4.001.patch
>
>
> {code}
> 2019-02-15 22:56:33,008 ERROR [RS_OPEN_REGION-n79:16020-16]
> regionserver.RegionCoprocessorHost: Malformed table coprocessor
> specification: key=coprocessor$2, spec:
> |org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver|805306366|
> 2019-02-16 00:39:33,651 ERROR [RS_OPEN_REGION-n76:16020-14]
> regionserver.RegionCoprocessorHost: Malformed table coprocessor
> specification: key=coprocessor$1, spec:
> |org.apache.phoenix.coprocessor.ScanRegionObserver|805306366|
> 2019-02-16 00:39:33,651 ERROR [RS_OPEN_REGION-n76:16020-19]
> regionserver.RegionCoprocessorHost: Malformed table coprocessor
> specification: key=coprocessor$1, spec:
> |org.apache.phoenix.coprocessor.ScanRegionObserver|805306366|
> {code}
> I checked the same coprocessor specification(logged in exception) with the
> code and it is parsed successfully. And even after the restart , regionserver
> also didn't complain about it.
> I think we should be logging the cause of the exception while parsing table
> descriptor for the coprocessor for better debugging.
> https://github.com/apache/hbase/blob/branch-1.4/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java#L318
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-22375) Promote AccessChecker to LimitedPrivate(Coprocessor)
[ https://issues.apache.org/jira/browse/HBASE-22375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-22375: Fix Version/s: (was: 1.5.0) > Promote AccessChecker to LimitedPrivate(Coprocessor) > > > Key: HBASE-22375 > URL: https://issues.apache.org/jira/browse/HBASE-22375 > Project: HBase > Issue Type: Task > Components: Coprocessors, security >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Minor > Fix For: 3.0.0, 2.2.0, 1.4.10, 2.3.0, 2.1.5, 1.3.5 > > Attachments: HBASE-22375-branch-1.patch, HBASE-22375.patch > > > We refactored access checking so other components can reuse permissions > checks formerly encapsulated by the AccessController coprocessor. The new API > is AccessChecker. Currently it is marked Private but I propose to promote > this to LimitedPrivate(Coprocessor) with Evolving status. > Unless there is an objection I will make this change. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-22062) Backport HBASE-20917 addendum to branch-1 & branch-1.4
[ https://issues.apache.org/jira/browse/HBASE-22062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-22062: Fix Version/s: (was: 1.5.0) > Backport HBASE-20917 addendum to branch-1 & branch-1.4 > -- > > Key: HBASE-22062 > URL: https://issues.apache.org/jira/browse/HBASE-22062 > Project: HBase > Issue Type: Bug >Reporter: Sakthi >Assignee: Sakthi >Priority: Minor > Fix For: 1.4.10 > > Attachments: HBASE-22062.branch-1.4.001.patch > > > The original patch is present in the 2 branches but the addendum is missing > from the 2 branches. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-22358) Change rubocop configuration for method length
[ https://issues.apache.org/jira/browse/HBASE-22358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-22358: Fix Version/s: (was: 1.5.0) > Change rubocop configuration for method length > -- > > Key: HBASE-22358 > URL: https://issues.apache.org/jira/browse/HBASE-22358 > Project: HBase > Issue Type: Improvement > Components: community, shell >Reporter: Jan Hentschel >Assignee: Murtaza Hassan >Priority: Minor > Labels: beginner, beginners > Fix For: 3.0.0, 2.2.0, 1.4.10, 2.3.0, 2.0.6, 2.1.5, 1.3.5 > > > rubocop currently uses a maximum method length for the Ruby code of 10, which > is way too restrictive. In Checkstyle we're using 150 lines per method. Don't > know if it needs to be that much, but something between 50 and 75 seems to be > more realistic, especially for test cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-21884) Fix box/unbox findbugs warning in secure bulk load
[
https://issues.apache.org/jira/browse/HBASE-21884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-21884:
Fix Version/s: (was: 1.5.0)
> Fix box/unbox findbugs warning in secure bulk load
> --
>
> Key: HBASE-21884
> URL: https://issues.apache.org/jira/browse/HBASE-21884
> Project: HBase
> Issue Type: Task
>Affects Versions: 3.0.0, 1.5.0, 2.2.0, 2.1.1, 2.0.3, 2.1.2, 2.0.4, 1.4.10,
> 1.3.4, 1.2.11, 2.3.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Minor
> Fix For: 3.0.0, 2.2.0, 1.4.10, 2.0.5, 1.3.4, 1.2.11, 2.3.0, 2.1.4
>
> Attachments: HBASE-21884-branch-1.v0.patch,
> HBASE-21884-master.v0.patch, HBASE-21884-master.v1.patch
>
>
> {code}
> ReasonTests
> FindBugs module:hbase-server
> Boxed value is unboxed and then immediately reboxed in
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.incrementUgiReference(UserGroupInformation)
> At SecureBulkLoadEndpoint.java:then immediately reboxed in
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.incrementUgiReference(UserGroupInformation)
> At SecureBulkLoadEndpoint.java:[line 268]
> {code}
> Looking at branch-2 and master I suspect we're doing the same wasteful
> operation but findbugs can't see it through the lambda definition.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23128) Restore Region interface compatibility
[
https://issues.apache.org/jira/browse/HBASE-23128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-23128:
Description:
Adding methods to a Public interface is ok for a minor release, removing
methods is not. We need to restore
{code}
abstract method boolean bulkLoadHFiles (
Collection>, boolean, Region.BulkLoadListener)
{code}
to the Region interface in order to maintain binary compatibility.
was:
Adding methods to a Public interface is ok for a minor release, removing
methods is not. We need to restore
abstract method boolean bulkLoadHFiles ( Collection>,
boolean, Region.BulkLoadListener )
in order to maintain binary compatibility.
> Restore Region interface compatibility
> ---
>
> Key: HBASE-23128
> URL: https://issues.apache.org/jira/browse/HBASE-23128
> Project: HBase
> Issue Type: Bug
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Blocker
> Fix For: 1.5.0
>
>
> Adding methods to a Public interface is ok for a minor release, removing
> methods is not. We need to restore
> {code}
> abstract method boolean bulkLoadHFiles (
> Collection>, boolean, Region.BulkLoadListener)
> {code}
> to the Region interface in order to maintain binary compatibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HBASE-23128) Restore Region interface compatibility
[ https://issues.apache.org/jira/browse/HBASE-23128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell reassigned HBASE-23128: --- Assignee: Andrew Kyle Purtell > Restore Region interface compatibility > --- > > Key: HBASE-23128 > URL: https://issues.apache.org/jira/browse/HBASE-23128 > Project: HBase > Issue Type: Bug >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Blocker > Fix For: 1.5.0 > > > Adding methods to a Public interface is ok for a minor release, removing > methods is not. We need to restore > abstract method boolean bulkLoadHFiles ( Collection>, > boolean, Region.BulkLoadListener ) > in order to maintain binary compatibility. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23128) Restore Region interface compatibility
[ https://issues.apache.org/jira/browse/HBASE-23128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-23128: Fix Version/s: 1.5.0 > Restore Region interface compatibility > --- > > Key: HBASE-23128 > URL: https://issues.apache.org/jira/browse/HBASE-23128 > Project: HBase > Issue Type: Bug >Reporter: Andrew Kyle Purtell >Priority: Trivial > Fix For: 1.5.0 > > > Adding methods to a Public interface is ok for a minor release, removing > methods is not. We need to restore > abstract method boolean bulkLoadHFiles ( Collection>, > boolean, Region.BulkLoadListener ) > in order to maintain binary compatibility. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23128) Restore Region interface compatibility
[ https://issues.apache.org/jira/browse/HBASE-23128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-23128: Priority: Blocker (was: Trivial) > Restore Region interface compatibility > --- > > Key: HBASE-23128 > URL: https://issues.apache.org/jira/browse/HBASE-23128 > Project: HBase > Issue Type: Bug >Reporter: Andrew Kyle Purtell >Priority: Blocker > Fix For: 1.5.0 > > > Adding methods to a Public interface is ok for a minor release, removing > methods is not. We need to restore > abstract method boolean bulkLoadHFiles ( Collection>, > boolean, Region.BulkLoadListener ) > in order to maintain binary compatibility. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23128) Restore Region interface compatibility
Andrew Kyle Purtell created HBASE-23128: --- Summary: Restore Region interface compatibility Key: HBASE-23128 URL: https://issues.apache.org/jira/browse/HBASE-23128 Project: HBase Issue Type: Bug Reporter: Andrew Kyle Purtell Adding methods to a Public interface is ok for a minor release, removing methods is not. We need to restore abstract method boolean bulkLoadHFiles ( Collection>, boolean, Region.BulkLoadListener ) in order to maintain binary compatibility. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-21905) TestFIFOCompactionPolicy is flaky
[ https://issues.apache.org/jira/browse/HBASE-21905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-21905: Fix Version/s: (was: 1.5.0) 1.5.1 1.6.0 > TestFIFOCompactionPolicy is flaky > - > > Key: HBASE-21905 > URL: https://issues.apache.org/jira/browse/HBASE-21905 > Project: HBase > Issue Type: Test > Components: test >Affects Versions: 1.5.0 >Reporter: Andrew Kyle Purtell >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Labels: branch-1 > Fix For: 1.6.0, 1.5.1 > > Attachments: > org.apache.hadoop.hbase.regionserver.compactions.TestFIFOCompactionPolicy-output.txt > > > java.lang.IllegalArgumentException , overlaps with > For example: > [ERROR] > testFIFOCompactionPolicyExpiredEmptyHFiles(org.apache.hadoop.hbase.regionserver.compactions.TestFIFOCompactionPolicy) > Time elapsed: 3.321 s <<< ERROR! > java.io.IOException: > java.io.IOException: > [hdfs://localhost:41525/user/apurtell/test-data/734de07d-1f22-46a9-a1f5-96ad4578450b/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/c4f673438e09d7ef5a9b79b363639cde/f/c0c5836c1f714f78847cf00326586b69, > > hdfs://localhost:41525/user/apurtell/test-data/734de07d-1f22-46a9-a1f5-96ad4578450b/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/c4f673438e09d7ef5a9b79b363639cde/f/c65648691f614b2d8dd4b586c5923bfe] > overlaps with > [hdfs://localhost:41525/user/apurtell/test-data/734de07d-1f22-46a9-a1f5-96ad4578450b/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/c4f673438e09d7ef5a9b79b363639cde/f/c0c5836c1f714f78847cf00326586b69] > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2438) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:297) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277) > Caused by: java.lang.IllegalArgumentException: > [hdfs://localhost:41525/user/apurtell/test-data/734de07d-1f22-46a9-a1f5-96ad4578450b/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/c4f673438e09d7ef5a9b79b363639cde/f/c0c5836c1f714f78847cf00326586b69, > > hdfs://localhost:41525/user/apurtell/test-data/734de07d-1f22-46a9-a1f5-96ad4578450b/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/c4f673438e09d7ef5a9b79b363639cde/f/c65648691f614b2d8dd4b586c5923bfe] > overlaps with > [hdfs://localhost:41525/user/apurtell/test-data/734de07d-1f22-46a9-a1f5-96ad4578450b/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/c4f673438e09d7ef5a9b79b363639cde/f/c0c5836c1f714f78847cf00326586b69] > at > com.google.common.base.Preconditions.checkArgument(Preconditions.java:119) > at > org.apache.hadoop.hbase.regionserver.HStore.addToCompactingFiles(HStore.java:1824) > at > org.apache.hadoop.hbase.regionserver.HStore.requestCompaction(HStore.java:1798) > at > org.apache.hadoop.hbase.regionserver.CompactSplitThread.selectCompaction(CompactSplitThread.java:415) > at > org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompactionInternal(CompactSplitThread.java:388) > at > org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompactionInternal(CompactSplitThread.java:317) > at > org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompaction(CompactSplitThread.java:306) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.compactRegion(RSRpcServices.java:1513) > at > org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:23649) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2380) > ... 3 more -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-11685) Incr/decr on the reference count of HConnectionImplementation need be atomic
[
https://issues.apache.org/jira/browse/HBASE-11685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-11685:
Fix Version/s: (was: 1.5.0)
1.6.0
> Incr/decr on the reference count of HConnectionImplementation need be atomic
> -
>
> Key: HBASE-11685
> URL: https://issues.apache.org/jira/browse/HBASE-11685
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: Shaohui Liu
>Assignee: Shaohui Liu
>Priority: Minor
> Fix For: 1.6.0
>
> Attachments: HBASE-11685-trunk-v1.diff, HBASE-11685-trunk-v2.diff,
> HBASE-11685-trunk-v3.diff, HBASE-11685-trunk-v4.diff,
> HBASE-11685-trunk-v5.diff, HBASE-11685-trunk-v6.diff
>
>
> Currently, the incr/decr operation on the ref count of
> HConnectionImplementation are not atomic. This may cause that the ref count
> always be larger than 0 and the connection never be closed.
> {code}
> /**
> * Increment this client's reference count.
> */
> void incCount() {
> ++refCount;
> }
> /**
> * Decrement this client's reference count.
> */
> void decCount() {
> if (refCount > 0) {
> --refCount;
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-16071) The VisibilityLabelFilter and AccessControlFilter should not count the "delete cell"
[
https://issues.apache.org/jira/browse/HBASE-16071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-16071:
Fix Version/s: (was: 1.5.0)
1.6.0
> The VisibilityLabelFilter and AccessControlFilter should not count the
> "delete cell"
>
>
> Key: HBASE-16071
> URL: https://issues.apache.org/jira/browse/HBASE-16071
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Minor
> Fix For: 3.0.0, 2.3.0, 1.6.0
>
> Attachments: HBASE-16071-v1.patch, HBASE-16071-v2.patch,
> HBASE-16071-v3.patch
>
>
> The VisibilityLabelFilter will see and count the "delete cell" if the
> scan.isRaw() returns true, so the (put) cell will be skipped if it has lower
> version than "delete cell"
> The critical code is shown below:
> {code:title=VisibilityLabelFilter.java|borderStyle=solid}
> public ReturnCode filterKeyValue(Cell cell) throws IOException {
> if (curFamily.getBytes() == null
> || !(CellUtil.matchingFamily(cell, curFamily.getBytes(),
> curFamily.getOffset(),
> curFamily.getLength( {
> curFamily.set(cell.getFamilyArray(), cell.getFamilyOffset(),
> cell.getFamilyLength());
> // For this family, all the columns can have max of
> curFamilyMaxVersions versions. No need to
> // consider the older versions for visibility label check.
> // Ideally this should have been done at a lower layer by HBase (?)
> curFamilyMaxVersions = cfVsMaxVersions.get(curFamily);
> // Family is changed. Just unset curQualifier.
> curQualifier.unset();
> }
> if (curQualifier.getBytes() == null
> || !(CellUtil.matchingQualifier(cell, curQualifier.getBytes(),
> curQualifier.getOffset(),
> curQualifier.getLength( {
> curQualifier.set(cell.getQualifierArray(), cell.getQualifierOffset(),
> cell.getQualifierLength());
> curQualMetVersions = 0;
> }
> curQualMetVersions++;
> if (curQualMetVersions > curFamilyMaxVersions) {
> return ReturnCode.SKIP;
> }
> return this.expEvaluator.evaluate(cell) ? ReturnCode.INCLUDE :
> ReturnCode.SKIP;
> }
> {code}
> [VisibilityLabelFilter.java|https://github.com/apache/hbase/blob/d7a4499dfc8b3936a0eca867589fc2b23b597866/hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityLabelFilter.java]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-22126) TestBlocksRead is flaky
[ https://issues.apache.org/jira/browse/HBASE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-22126: Fix Version/s: (was: 1.5.0) 1.5.1 1.6.0 > TestBlocksRead is flaky > --- > > Key: HBASE-22126 > URL: https://issues.apache.org/jira/browse/HBASE-22126 > Project: HBase > Issue Type: Test > Components: test >Affects Versions: 1.5.0 >Reporter: Andrew Kyle Purtell >Assignee: Sandeep Pal >Priority: Major > Labels: branch-1 > Fix For: 1.6.0, 1.5.1 > > > TestBlocksRead does not fail when invoked by itself but is flaky when run as > part of the suite. > Some kind of race during setup. > [ERROR] > testBlocksStoredWhenCachingDisabled(org.apache.hadoop.hbase.regionserver.TestBlocksRead) > Time elapsed: 0.19 s <<< ERROR! > java.net.ConnectException: Call From $HOST/$IP to localhost:59658 failed on > connection exception: java.net.ConnectException: Connection refused; For more > details see: http://wiki.apache.org/hadoop/ConnectionRefused > at > org.apache.hadoop.hbase.regionserver.TestBlocksRead.initHRegion(TestBlocksRead.java:112) > at > org.apache.hadoop.hbase.regionserver.TestBlocksRead.testBlocksStoredWhenCachingDisabled(TestBlocksRead.java:389) > Caused by: java.net.ConnectException: Connection refused > at > org.apache.hadoop.hbase.regionserver.TestBlocksRead.initHRegion(TestBlocksRead.java:112) > at > org.apache.hadoop.hbase.regionserver.TestBlocksRead.testBlocksStoredWhenCachingDisabled(TestBlocksRead.java:389) > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-14964) Backport HBASE-14901 to brach-1 - There is duplicated code to create/manage encryption keys
[ https://issues.apache.org/jira/browse/HBASE-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-14964: Fix Version/s: (was: 1.5.0) 1.5.1 1.6.0 > Backport HBASE-14901 to brach-1 - There is duplicated code to create/manage > encryption keys > --- > > Key: HBASE-14964 > URL: https://issues.apache.org/jira/browse/HBASE-14964 > Project: HBase > Issue Type: Improvement > Components: encryption >Reporter: Nate Edel >Priority: Minor > Fix For: 1.6.0, 1.5.1 > > Attachments: HBASE-14964-branch-1.1.patch, > HBASE-14964.1.branch-1.patch, HBASE-14964.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > There is duplicated code from MobUtils.createEncryptionContext in HStore, and > there is a subset of that code in HFileReaderImpl. > Refactored key selection > Moved both to EncryptionUtil.java > Can't figure out how to write a unit test for this, but there's no new code > just refactoring. > A lot of the Mob stuff hasn't been backported, so this is a very small patch. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-20618) Skip large rows instead of throwing an exception to client
[
https://issues.apache.org/jira/browse/HBASE-20618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-20618:
Fix Version/s: (was: 1.5.0)
1.6.0
> Skip large rows instead of throwing an exception to client
> --
>
> Key: HBASE-20618
> URL: https://issues.apache.org/jira/browse/HBASE-20618
> Project: HBase
> Issue Type: New Feature
>Reporter: Swapna
>Priority: Minor
> Fix For: 3.0.0, 2.3.0, 1.6.0
>
> Attachments: HBASE-20618.hbasemaster.v01.patch,
> HBASE-20618.hbasemaster.v02.patch, HBASE-20618.v1.branch-1.patch,
> HBASE-20618.v1.branch-1.patch
>
>
> Currently HBase supports throwing RowTooBigException incase there is a row
> with one of the column family data exceeds the configured maximum
> https://issues.apache.org/jira/browse/HBASE-10925?attachmentOrder=desc
> We have some bad rows growing very large. We need a way to skip these rows
> for most of our jobs.
> Some of the options we considered:
> Option 1:
> Hbase client handle the exception and restart the scanner past bad row by
> capturing the row key where it failed. Can be by adding the rowkey to the
> exception stack trace, which seems brittle. Client would ignore the setting
> if its upgraded before server.
> Option 2:
> Skip through big rows on Server.Go with server level config similar to
> "hbase.table.max.rowsize" or request based by changing the scan request api.
> If allowed to do per request, based on the scan request config, Client will
> have to ignore the setting if its upgraded before server.
> {code}
> try {
> populateResult(results, this.storeHeap, scannerContext, current);
> } catch(RowTooBigException e) {
> LOG.info("Row exceeded the limit in storeheap. Skipping row with
> key:"+Bytes.toString(current.getRowArray()));
> this.storeHeap.reseek(PrivateCellUtil.createLastOnRow(current));
> results.clear();
> scannerContext.clearProgress();
> continue;
> }
> {code}
> Prefer the option 2 with server level config. Please share your inputs
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-22839) Make sure the batches within one region are shipped to the sink clusters in order (branch-1)
[ https://issues.apache.org/jira/browse/HBASE-22839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-22839: Fix Version/s: (was: 1.5.0) 1.6.0 > Make sure the batches within one region are shipped to the sink clusters in > order (branch-1) > > > Key: HBASE-22839 > URL: https://issues.apache.org/jira/browse/HBASE-22839 > Project: HBase > Issue Type: Improvement > Components: Replication >Affects Versions: 1.3.4, 1.3.5 >Reporter: Bin Shi >Assignee: Bin Shi >Priority: Major > Fix For: 3.0.0, 2.3.0, 1.6.0 > > > Problem Statement: > In the cross-cluster replication validation, we found some cells in source > and sink cluster can have the same row key, the same timestamp but different > values. The happens when mutations with the same row key are submitted in > batch without specifying the timestamp, and the same timestamp in the unit of > millisecond is assigned at the time when they are committed to the WAL. > When this happens, if the major compaction hasn’t happened yet and you scan > the table, you can find some cells have the same row key, the same timestamps > but different values, like the first three rows in the following table. > |Row Key 1|CF0::Column 1|Timestatmp 1|Value 1| > |Row Key 1|CF0::Column 1|Timestatmp 1|Value 2| > |Row Key 1|CF0::Column 1|Timestatmp 1|Value 3| > |Row Key 2|CF0::Column 1|Timestatmp 2|Value 4| > |Row Key 3|CF0::Column 1|Timestatmp 4|Value 5| > The ordering of the first three rows is indeterminate in the presence of the > cross-replication, so after compaction, in the master cluster you will see > “Row Key 1, CF0::Column1, Timestamp1” having the value 3, but in the slave > cluster, you might see the cell having one of the three possible values 1, 2, > 3, which results in data inconsistency issue between the master and slave > clusters. > Root Cause Analysis: > In HBaseInterClusterReplicationEndpoint.createBatches() of branch-1.3, the > WAL entries from the same region could be split into different batches > according to replication RPC limit and these batches are shipped by > ReplicationSource concurrently, so the batches for the same region could > arrive at the sink in the slave clusters then apply to the region > synchronously in indeterminate order. > Solution: > In HBase 3.0.0 and 2.1.0, [~Apache9]&[~openinx]&[~fenghh] provided Serial > Replication HBASE-20046 which guarantees the order of pushing logs to slave > clusters is same as the order of requests from client in the master cluster. > It contains mainly two changes: > # Recording the replication "barriers" in ZooKeeper to synchronize the > replication across old/failed RS and new RS to provide strict ordering > semantics even in the presence of region-move or RS failure. > # Make sure the batches within one region are shipped to the slave clusters > in order. > The second part of change is exactly what we need and the minimal change to > fix the issue in this JIRA. > To fix the issue in this JIRA, we have two options: > # Cherry-Pick HBASE-20046 to branch 1.3. Pros: It also fixes the data > inconsistency issue when there is region-move or RS failure and help to > reduce the noises in our cross-cluster replication/backup validation which is > our ultimate goal. Cons: the change is big and I'm not sure for now whether > the change is self-contained or it has other dependencies which need to port > to branch 1.3 too; and we need longer time to validate and stabilize. > # Port the minimal change or make the equivalent change as the second part > of HBASE-20046 to make sure the batches within one region are shipped to the > slave clusters in order." > With limited knowledge about HBase Release Schedule and Process, I prefer > option 2 because of cons of option 1, but I'm open to option 1 and other > options. Thoughts? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23120) Fix ref guide - Release process
[
https://issues.apache.org/jira/browse/HBASE-23120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-23120:
Fix Version/s: (was: 1.5.0)
1.6.0
> Fix ref guide - Release process
> ---
>
> Key: HBASE-23120
> URL: https://issues.apache.org/jira/browse/HBASE-23120
> Project: HBase
> Issue Type: Sub-task
>Reporter: Sakthi
>Assignee: Sakthi
>Priority: Minor
> Fix For: 3.0.0, 2.3.0, 1.6.0, 2.1.7, 2.2.2
>
>
> The example is of a sample ~/.m2/settings.xml file.
> {code:java}
>
> {code}
> is not a proper comment.
> Also I think here,
> {code:java}
>
> {code}
> the double hyphened --list-keys shouldn't be used. It leads to FATAL error
> like the one below:
> {code:java}
> [FATAL] Non-parseable settings
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-21776) Duplicate "Set storagePolicy" debug logging
[ https://issues.apache.org/jira/browse/HBASE-21776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-21776: Fix Version/s: (was: 1.5.0) 1.5.1 1.6.0 > Duplicate "Set storagePolicy" debug logging > --- > > Key: HBASE-21776 > URL: https://issues.apache.org/jira/browse/HBASE-21776 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 1.5.0 >Reporter: Andrew Kyle Purtell >Assignee: Sandeep Pal >Priority: Trivial > Labels: branch-1 > Fix For: 1.6.0, 1.5.1 > > > An example: > 2019-01-25 02:57:15,831 DEBUG > [regionserver/ip-172-31-13-83.us-west-2.compute.internal/172.31.13.83:8120-longCompactions-1548384634805] > util.CommonFSUtils: Set storagePolicy=HOT for > path=hdfs://ip-172-31-5-95.us-west-2.compute.internal:8020/hbase/data/default/IntegrationTestBigLinkedList/16899cdfdb4ab217208f4108ad0c4803/.tmp/meta > 2019-01-25 02:57:15,831 DEBUG > [regionserver/ip-172-31-13-83.us-west-2.compute.internal/172.31.13.83:8120-longCompactions-1548384634805] > util.CommonFSUtils: Set storagePolicy=HOT for > path=hdfs://ip-172-31-5-95.us-west-2.compute.internal:8020/hbase/data/default/IntegrationTestBigLinkedList/16899cdfdb4ab217208f4108ad0c4803/.tmp/meta > Seen most often during compactions. Ideally we only log once per directory > per flush or compaction. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-20912) Add import order config in dev support for eclipse
[ https://issues.apache.org/jira/browse/HBASE-20912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-20912: Fix Version/s: (was: 1.5.0) > Add import order config in dev support for eclipse > -- > > Key: HBASE-20912 > URL: https://issues.apache.org/jira/browse/HBASE-20912 > Project: HBase > Issue Type: Bug >Reporter: Ankit Singhal >Assignee: Ankit Singhal >Priority: Major > Fix For: 3.0.0, 2.2.0, 1.4.10, 1.3.4, 2.3.0, 2.1.5 > > Attachments: eclipse.importorder > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-21048) Get LogLevel is not working from console in secure environment
[
https://issues.apache.org/jira/browse/HBASE-21048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-21048:
Fix Version/s: (was: 1.5.0)
> Get LogLevel is not working from console in secure environment
> --
>
> Key: HBASE-21048
> URL: https://issues.apache.org/jira/browse/HBASE-21048
> Project: HBase
> Issue Type: Bug
>Reporter: Chandra Sekhar
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Labels: security
> Fix For: 3.0.0, 1.4.10, 2.3.0
>
> Attachments: HBASE-21048.001.patch, HBASE-21048.branch-1.001.patch,
> HBASE-21048.branch-1.002.patch, HBASE-21048.branch-1.4.001.patch,
> HBASE-21048.master.001.patch, HBASE-21048.master.002.patch,
> HBASE-21048.master.003.patch, HBASE-21048.master.004.patch,
> HBASE-21048.master.005.patch, HBASE-21048.master.006.patch,
> HBASE-21048.master.007.patch, HBASE-21048.master.008.patch
>
>
> When we try to get log level of specific package in secure environment,
> getting SocketException.
> {code:java}
> hbase/master/bin# ./hbase org.apache.hadoop.hbase.http.log.LogLevel -getlevel
> host-:16010 org.apache.hadoop.hbase
> Connecting to http://host-:16010/logLevel?log=org.apache.hadoop.hbase
> java.net.SocketException: Unexpected end of file from server
> {code}
> It is trying to connect http instead of https
> code snippet that handling only http in *LogLevel.java*
> {code:java}
> public static void main(String[] args) {
> if (args.length == 3 && "-getlevel".equals(args[0])) {
> process("http://; + args[1] + "/logLevel?log=" + args[2]);
> return;
> }
> else if (args.length == 4 && "-setlevel".equals(args[0])) {
> process("http://; + args[1] + "/logLevel?log=" + args[2]
> + "=" + args[3]);
> return;
> }
> System.err.println(USAGES);
> System.exit(-1);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
