[jira] [Commented] (HBASE-22335) do add hfile ref only when replication_scope is 1

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-22335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836197#comment-16836197
 ] 

Ashish Singhi commented on HBASE-22335:
---

LGTM. Please check if you can write a UT for this.

> do add hfile ref only when replication_scope is 1
> -
>
> Key: HBASE-22335
> URL: https://issues.apache.org/jira/browse/HBASE-22335
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: chenxu
>Assignee: chenxu
>Priority: Major
>  Labels: replication
> Attachments: HBASE-22335-master-v1.patch
>
>
> When bulkload replication enabled, every hfile's znode will add to 
> /hbase/replication/hfile-refs, no matter what the REPLICATION_SCOPE is.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-14939) Document bulk loaded hfile replication

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-14939:
--
   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Thanks for the contribution, [~jojochuang]!

> Document bulk loaded hfile replication
> --
>
> Key: HBASE-14939
> URL: https://issues.apache.org/jira/browse/HBASE-14939
> Project: HBase
>  Issue Type: Task
>  Components: documentation
>Reporter: Ashish Singhi
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-14939.master.001.patch
>
>
> After HBASE-13153 is committed we need to add that information under the 
> Cluster Replication section in HBase book.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-14939) Document bulk loaded hfile replication

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16725042#comment-16725042
 ] 

Ashish Singhi commented on HBASE-14939:
---

Thanks [~jojochuang]. The patch looks good to me.

> Document bulk loaded hfile replication
> --
>
> Key: HBASE-14939
> URL: https://issues.apache.org/jira/browse/HBASE-14939
> Project: HBase
>  Issue Type: Task
>  Components: documentation
>Reporter: Ashish Singhi
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HBASE-14939.master.001.patch
>
>
> After HBASE-13153 is committed we need to add that information under the 
> Cluster Replication section in HBase book.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21267) Spark in java examples throws 'A master URL must be set in your configuration'

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-21267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639514#comment-16639514
 ] 

Ashish Singhi commented on HBASE-21267:
---

 [~ram_krish], the change you propose makes sense.

> Spark in java examples throws 'A master URL must be set in your configuration'
> --
>
> Key: HBASE-21267
> URL: https://issues.apache.org/jira/browse/HBASE-21267
> Project: HBase
>  Issue Type: Test
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
>
> Trying out some spark on java examples I found this
> 'A master URL must be set in your configuration'.
> It comes from here
> bq.SparkConf sparkConf = new 
> SparkConf().setAppName("JavaHBaseBulkPutExample " + tableName);
> Adding the below line solved the problem, 
> bq. SparkConf sparkConf = new SparkConf().setAppName("JavaHBaseBulkPutExample 
> " + tableName).setMaster("local[2]")
> But am not sure if there is any better way.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-21001) ReplicationObserver fails to load in HBase 2.0.0

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16601158#comment-16601158
 ] 

Ashish Singhi commented on HBASE-21001:
---

lgtm

> ReplicationObserver fails to load in HBase 2.0.0
> 
>
> Key: HBASE-21001
> URL: https://issues.apache.org/jira/browse/HBASE-21001
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Wei-Chiu Chuang
>Assignee: Guangxu Cheng
>Priority: Major
> Attachments: HBASE-21001.master.001.patch
>
>
> ReplicationObserver was added in HBASE-17290 to prevent "Potential loss of 
> data for replication of bulk loaded hfiles".
> I tried to enable bulk loading replication feature 
> (hbase.replication.bulkload.enabled=true and configure 
> hbase.replication.cluster.id) on a HBase 2.0.0 cluster, but the RegionServer 
> started with the following error:
> {quote}
> 2018-08-02 18:20:36,365 INFO 
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost: System 
> coprocessor loading is enabled
> 2018-08-02 18:20:36,365 INFO 
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost: Table 
> coprocessor loading is enabled
> 2018-08-02 18:20:36,365 ERROR 
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost: 
> org.apache.hadoop.hbase.replication.regionserver.ReplicationObserver is not 
> of type
> RegionServerCoprocessor. Check the configuration of 
> hbase.coprocessor.regionserver.classes
> 2018-08-02 18:20:36,366 ERROR 
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost: Cannot load coprocessor 
> ReplicationObserver
> {quote}
> It looks like this was broken by HBASE-17732 to me, but I could be wrong. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-18822) Create table for peer cluster automatically when creating table in source cluster of using namespace replication.

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556966#comment-16556966
 ] 

Ashish Singhi commented on HBASE-18822:
---

bq. Ashish Singhi also have the same concern I guess.
No, mine is not that and I have tried my best to explain my concern above.

bq. I have only one concern, why are we replicating table during its creation, 
table will be replicated to the desired peers when replication is enabled for 
that. 
I think this should be ok, when a user create table in source cluster 
immediately it is created in peer cluster and then the user has to just change 
the replication scope. (Thinking from a user POV who is not using 
enable_table_replication command)

> Create table for peer cluster automatically when creating table in source 
> cluster of using namespace replication.
> -
>
> Key: HBASE-18822
> URL: https://issues.apache.org/jira/browse/HBASE-18822
> Project: HBase
>  Issue Type: Improvement
>  Components: Replication
>Affects Versions: 2.0.0-alpha-2
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-18822.v1.patch, HBASE-18822.v1.patch
>
>
> In our cluster of using namespace replication,   we always forget to create 
> table in peer cluster, which lead to replication get stuck. 
> We have implemented the feature in our cluster:  create table for peer 
> cluster automatically when creating table in source cluster of using 
> namespace replication.
>  
> I'm not sure if someone else needs this feature, so create an issue here for 
> discussing   



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-18822) Create table for peer cluster automatically when creating table in source cluster of using namespace replication.

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555816#comment-16555816
 ] 

Ashish Singhi commented on HBASE-18822:
---

I get that point [~openinx]. My only concern is, unless explicitly a user 
specify any table for replication all the tables will be synced to peer 
cluster, which IMO is not correct. If others here think otherwise also I'm fine 
with it. I don't want to be a blocker.

> Create table for peer cluster automatically when creating table in source 
> cluster of using namespace replication.
> -
>
> Key: HBASE-18822
> URL: https://issues.apache.org/jira/browse/HBASE-18822
> Project: HBase
>  Issue Type: Improvement
>  Components: Replication
>Affects Versions: 2.0.0-alpha-2
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-18822.v1.patch
>
>
> In our cluster of using namespace replication,   we always forget to create 
> table in peer cluster, which lead to replication get stuck. 
> We have implemented the feature in our cluster:  create table for peer 
> cluster automatically when creating table in source cluster of using 
> namespace replication.
>  
> I'm not sure if someone else needs this feature, so create an issue here for 
> discussing   



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-18822) Create table for peer cluster automatically when creating table in source cluster of using namespace replication.

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555605#comment-16555605
 ] 

Ashish Singhi commented on HBASE-18822:
---

{quote}Depends on the peer's replicateAllUserTables, if true, then will 
replicate to peer, if false then won't. 
{quote}
Exactly, by default it is true which means all the tables schema will be synced 
to peer cluster.

> Create table for peer cluster automatically when creating table in source 
> cluster of using namespace replication.
> -
>
> Key: HBASE-18822
> URL: https://issues.apache.org/jira/browse/HBASE-18822
> Project: HBase
>  Issue Type: Improvement
>  Components: Replication
>Affects Versions: 2.0.0-alpha-2
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-18822.v1.patch
>
>
> In our cluster of using namespace replication,   we always forget to create 
> table in peer cluster, which lead to replication get stuck. 
> We have implemented the feature in our cluster:  create table for peer 
> cluster automatically when creating table in source cluster of using 
> namespace replication.
>  
> I'm not sure if someone else needs this feature, so create an issue here for 
> discussing   



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-18822) Create table for peer cluster automatically when creating table in source cluster of using namespace replication.

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555467#comment-16555467
 ] 

Ashish Singhi commented on HBASE-18822:
---

What will happen when a user doesn't explicitly specify any namespace or table 
in replication peer config ?

Thanks.

> Create table for peer cluster automatically when creating table in source 
> cluster of using namespace replication.
> -
>
> Key: HBASE-18822
> URL: https://issues.apache.org/jira/browse/HBASE-18822
> Project: HBase
>  Issue Type: Improvement
>  Components: Replication
>Affects Versions: 2.0.0-alpha-2
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-18822.v1.patch
>
>
> In our cluster of using namespace replication,   we always forget to create 
> table in peer cluster, which lead to replication get stuck. 
> We have implemented the feature in our cluster:  create table for peer 
> cluster automatically when creating table in source cluster of using 
> namespace replication.
>  
> I'm not sure if someone else needs this feature, so create an issue here for 
> discussing   



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-18822) Create table for peer cluster automatically when creating table in source cluster of using namespace replication.

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555415#comment-16555415
 ] 

Ashish Singhi commented on HBASE-18822:
---

Looked at the patch at high level. 
This will create all the tables in the peer cluster after enabling it. May be 
some user are not interested in all the tables data replication but still we 
will end up in creating the table in all the peers. 
Can we instead add a boolean to DDL APIs,  by default false and on true we can 
replicate the same operation in peer clusters. We have done the same way in my 
previous employer. WDYT ?
//CC [~pankaj2461]

> Create table for peer cluster automatically when creating table in source 
> cluster of using namespace replication.
> -
>
> Key: HBASE-18822
> URL: https://issues.apache.org/jira/browse/HBASE-18822
> Project: HBase
>  Issue Type: Improvement
>  Components: Replication
>Affects Versions: 2.0.0-alpha-2
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-18822.v1.patch
>
>
> In our cluster of using namespace replication,   we always forget to create 
> table in peer cluster, which lead to replication get stuck. 
> We have implemented the feature in our cluster:  create table for peer 
> cluster automatically when creating table in source cluster of using 
> namespace replication.
>  
> I'm not sure if someone else needs this feature, so create an issue here for 
> discussing   



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20861) WAL entries should be replicated which are updated after peer addition

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16538103#comment-16538103
 ] 

Ashish Singhi commented on HBASE-20861:
---

We can use HBASE-9888 for the fix as the solution proposed there I think is 
same as what you propose here.

> WAL entries should be replicated which are updated after peer addition
> --
>
> Key: HBASE-20861
> URL: https://issues.apache.org/jira/browse/HBASE-20861
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
>
> Steps to reproduce:
> 1. Add a peer
> 2. Enable a table replication
> 3. Put data
> 4. remove peer
> 5. Truncate the table in both the cluster
> 6. Add peer
> Observation:
> Here the old data is getting replicated in the table in peer cluster.
> HBase replication by design upon peer addition always starts replication from 
> begining of WAL file, so all replicable wal entries get replicated 
> irrespective of peer addition time.
> This can be handled by skipping WAL entries which is having create time 
> before peer addition time.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20861) WAL entries should be replicated which are updated after peer addition

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16537044#comment-16537044
 ] 

Ashish Singhi commented on HBASE-20861:
---

See HBASE-9888

> WAL entries should be replicated which are updated after peer addition
> --
>
> Key: HBASE-20861
> URL: https://issues.apache.org/jira/browse/HBASE-20861
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
>
> Steps to reproduce:
> 1. Add a peer
> 2. Enable a table replication
> 3. Put data
> 4. remove peer
> 5. Truncate the table in both the cluster
> 6. Add peer
> Observation:
> Here the old data is getting replicated in the table in peer cluster.
> HBase replication by design upon peer addition always starts replication from 
> begining of WAL file, so all replicable wal entries get replicated 
> irrespective of peer addition time.
> This can be handled by skipping WAL entries which is having create time 
> before peer addition time.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HBASE-20357) AccessControlClient API Enhancement

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527273#comment-16527273
 ] 

Ashish Singhi edited comment on HBASE-20357 at 6/29/18 7:10 AM:


Do we support that scenario ?
As per our semantic versioning guidelines, to me it doesn't seems like:
{noformat}
Client code written to APIs available in a given patch release might not run 
against the old jars from an earlier patch version.
{noformat}



was (Author: ashish singhi):
Do we support that scenario ?

> AccessControlClient API Enhancement
> ---
>
> Key: HBASE-20357
> URL: https://issues.apache.org/jira/browse/HBASE-20357
> Project: HBase
>  Issue Type: Improvement
>  Components: security
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20357.master.001.patch, 
> HBASE-20357.master.002.patch, HBASE-20357.master.003.patch, 
> HBASE-20357.master.addendum.0.patch
>
>
> *Background:*
>  Currently HBase ACLs can be retrieved based on the namespace or table name 
> only. There is no direct API available to retrieve the permissions based on 
> the namespace, table name, column family and column qualifier for specific 
> user.
> Client has to write application logic in multiple steps to retrieve ACLs 
> based on table name, column name and column qualifier for specific user.
>  HBase should enhance AccessControlClient APIs to simplyfy this.
> *AccessControlClient API should be extended with following APIs,*    
>  # To retrieve permissions based on the namespace, table name, column family 
> and column qualifier for specific user.
>   Permissions can be retrieved based on the following inputs,
>        - Namespace/Table (already available)
>        - Namespace/Table + UserName
>        - Table + CF
>        - Table + CF + UserName
>        - Table + CF + CQ
>        - Table + CF + CQ + UserName
>           Scope of retrieving permission will be as follows,
>                  - Same as existing
>        2. To validate whether a user is allowed to perform specified 
> operations on a particular table, will be useful to check user privilege 
> instead of getting ACD during client                                    
> operation.
>              User validation can be performed based on following inputs, 
>                   - Table + CF + CQ + UserName + Actions
>             Scope of validating user privilege,
>                     User can perform self check without any special privilege 
> but ADMIN privilege will be required to perform check for other users.
>                     For example, suppose there are two users "userA" & 
> "userB" then there can be below scenarios,
>                         - when userA want to check whether userA have 
> privilege to perform mentioned actions
>                                 > userA don't need ADMIN privilege, as it's a 
> self query.
>                        - when userA want to check whether userB have 
> privilege to perform mentioned actions,
>                                 > userA must have ADMIN or superuser 
> privilege, as it's trying to query for other user.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20357) AccessControlClient API Enhancement

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527273#comment-16527273
 ] 

Ashish Singhi commented on HBASE-20357:
---

Do we support that scenario ?

> AccessControlClient API Enhancement
> ---
>
> Key: HBASE-20357
> URL: https://issues.apache.org/jira/browse/HBASE-20357
> Project: HBase
>  Issue Type: Improvement
>  Components: security
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20357.master.001.patch, 
> HBASE-20357.master.002.patch, HBASE-20357.master.003.patch, 
> HBASE-20357.master.addendum.0.patch
>
>
> *Background:*
>  Currently HBase ACLs can be retrieved based on the namespace or table name 
> only. There is no direct API available to retrieve the permissions based on 
> the namespace, table name, column family and column qualifier for specific 
> user.
> Client has to write application logic in multiple steps to retrieve ACLs 
> based on table name, column name and column qualifier for specific user.
>  HBase should enhance AccessControlClient APIs to simplyfy this.
> *AccessControlClient API should be extended with following APIs,*    
>  # To retrieve permissions based on the namespace, table name, column family 
> and column qualifier for specific user.
>   Permissions can be retrieved based on the following inputs,
>        - Namespace/Table (already available)
>        - Namespace/Table + UserName
>        - Table + CF
>        - Table + CF + UserName
>        - Table + CF + CQ
>        - Table + CF + CQ + UserName
>           Scope of retrieving permission will be as follows,
>                  - Same as existing
>        2. To validate whether a user is allowed to perform specified 
> operations on a particular table, will be useful to check user privilege 
> instead of getting ACD during client                                    
> operation.
>              User validation can be performed based on following inputs, 
>                   - Table + CF + CQ + UserName + Actions
>             Scope of validating user privilege,
>                     User can perform self check without any special privilege 
> but ADMIN privilege will be required to perform check for other users.
>                     For example, suppose there are two users "userA" & 
> "userB" then there can be below scenarios,
>                         - when userA want to check whether userA have 
> privilege to perform mentioned actions
>                                 > userA don't need ADMIN privilege, as it's a 
> self query.
>                        - when userA want to check whether userB have 
> privilege to perform mentioned actions,
>                                 > userA must have ADMIN or superuser 
> privilege, as it's trying to query for other user.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20357) AccessControlClient API Enhancement

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527251#comment-16527251
 ] 

Ashish Singhi commented on HBASE-20357:
---

{quote}If hbase 2.2 AccessControlClient (with this change) calls 
{{hasPermission}} on hbase 2.1 cluster, what would happen ?
{quote}
[~yuzhih...@gmail.com], you mean client is of 2.2 version and server is 2.1 
version ?

> AccessControlClient API Enhancement
> ---
>
> Key: HBASE-20357
> URL: https://issues.apache.org/jira/browse/HBASE-20357
> Project: HBase
>  Issue Type: Improvement
>  Components: security
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20357.master.001.patch, 
> HBASE-20357.master.002.patch, HBASE-20357.master.003.patch, 
> HBASE-20357.master.addendum.0.patch
>
>
> *Background:*
>  Currently HBase ACLs can be retrieved based on the namespace or table name 
> only. There is no direct API available to retrieve the permissions based on 
> the namespace, table name, column family and column qualifier for specific 
> user.
> Client has to write application logic in multiple steps to retrieve ACLs 
> based on table name, column name and column qualifier for specific user.
>  HBase should enhance AccessControlClient APIs to simplyfy this.
> *AccessControlClient API should be extended with following APIs,*    
>  # To retrieve permissions based on the namespace, table name, column family 
> and column qualifier for specific user.
>   Permissions can be retrieved based on the following inputs,
>        - Namespace/Table (already available)
>        - Namespace/Table + UserName
>        - Table + CF
>        - Table + CF + UserName
>        - Table + CF + CQ
>        - Table + CF + CQ + UserName
>           Scope of retrieving permission will be as follows,
>                  - Same as existing
>        2. To validate whether a user is allowed to perform specified 
> operations on a particular table, will be useful to check user privilege 
> instead of getting ACD during client                                    
> operation.
>              User validation can be performed based on following inputs, 
>                   - Table + CF + CQ + UserName + Actions
>             Scope of validating user privilege,
>                     User can perform self check without any special privilege 
> but ADMIN privilege will be required to perform check for other users.
>                     For example, suppose there are two users "userA" & 
> "userB" then there can be below scenarios,
>                         - when userA want to check whether userA have 
> privilege to perform mentioned actions
>                                 > userA don't need ADMIN privilege, as it's a 
> self query.
>                        - when userA want to check whether userB have 
> privilege to perform mentioned actions,
>                                 > userA must have ADMIN or superuser 
> privilege, as it's trying to query for other user.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20788) Write up a doc about how to rolling upgrade from 1.x to 2.x

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16524684#comment-16524684
 ] 

Ashish Singhi commented on HBASE-20788:
---

I see, thanks.

> Write up a doc about how to rolling upgrade from 1.x to 2.x
> ---
>
> Key: HBASE-20788
> URL: https://issues.apache.org/jira/browse/HBASE-20788
> Project: HBase
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Blocker
> Fix For: 2.1.0
>
> Attachments: HBASE-20788.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HBASE-14937) Make rpc call timeout for replication adaptive

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi reassigned HBASE-14937:
-

Assignee: (was: Ashish Singhi)

Unassigned as I will be not be able to work on it in the near future.

> Make rpc call timeout for replication adaptive
> --
>
> Key: HBASE-14937
> URL: https://issues.apache.org/jira/browse/HBASE-14937
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ashish Singhi
>Priority: Major
>  Labels: replication
> Attachments: HBASE-14937.patch
>
>
> When peer cluster replication is disabled and lot of writes are happening in 
> active cluster and later on peer cluster replication is enabled then there 
> are chances that replication requests to peer cluster may time out.
> This is possible after HBASE-13153 and it can also happen with many and many 
> WAL data replication still pending to replicate.
> Approach to this problem will be discussed in the comments.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HBASE-14938) Limit the number of znodes for ZK in bulk loaded hfile replication

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi reassigned HBASE-14938:
-

Assignee: (was: Ashish Singhi)

Unassigned as I will be not be able to work on it in the near future.

> Limit the number of znodes for ZK in bulk loaded hfile replication
> --
>
> Key: HBASE-14938
> URL: https://issues.apache.org/jira/browse/HBASE-14938
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ashish Singhi
>Priority: Major
> Attachments: HBASE-14938(1).patch, HBASE-14938-v1.patch, 
> HBASE-14938-v2(1).patch, HBASE-14938-v2.patch, HBASE-14938.patch
>
>
> In ZK the maximum allowable size of the data array is 1 MB. Until we have 
> fixed HBASE-10295 we need to handle this.
> Approach to this problem will be discussed in the comments section.
> Note: We have done internal testing with more than 3k nodes in ZK yet to be 
> replicated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HBASE-14939) Document bulk loaded hfile replication

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi reassigned HBASE-14939:
-

Assignee: (was: Ashish Singhi)

Unassigned as I will be not be able to work on it in the near future.

> Document bulk loaded hfile replication
> --
>
> Key: HBASE-14939
> URL: https://issues.apache.org/jira/browse/HBASE-14939
> Project: HBase
>  Issue Type: Task
>  Components: documentation
>Reporter: Ashish Singhi
>Priority: Major
>
> After HBASE-13153 is committed we need to add that information under the 
> Cluster Replication section in HBase book.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HBASE-15446) Do not abort RS when WAL append for bulk load event marker is failed

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi reassigned HBASE-15446:
-

Assignee: (was: Ashish Singhi)

Unassigned as I will be not be able to work on it in the near future.

> Do not abort RS when WAL append for bulk load event marker is failed
> 
>
> Key: HBASE-15446
> URL: https://issues.apache.org/jira/browse/HBASE-15446
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Ashish Singhi
>Priority: Major
>
> During bulk load process when the RS fails to append the bulk load event 
> marker in WAL we abort that RS which is actually not really required. Instead 
> just throw back the exception to the client and let the client retry. This 
> will be helpful in case of replication of bulk loaded data.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HBASE-15254) Support fixed domain name in Kerberos name for HBase replication cross realm trust setup

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi reassigned HBASE-15254:
-

Assignee: (was: Ashish Singhi)

Unassigned as I will be not be able to work on it in the near future.

> Support fixed domain name in Kerberos name for HBase replication cross realm 
> trust setup
> 
>
> Key: HBASE-15254
> URL: https://issues.apache.org/jira/browse/HBASE-15254
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ashish Singhi
>Priority: Major
>  Labels: kerberos, replication, security
>
> HBase replication will not work with Kerberos cross realm trust when domain 
> name in the principal is not hostname. 
> A mail was also sent related to this in user mailing list, [mail | 
> https://groups.google.com/forum/#!topic/nosql-databases/AYhQnU9Fc7E]
> The problem here is when ever a user adds a new host to cluster he/she also 
> needs to add a principal name for that host in KDC, generate a new keytab 
> file and update it across other hosts accordingly if required. 
> To save all this efforts users may prefer to have a fixed domain name in the 
> principal for all the hosts and in that case HBase replication will fail 
> because currently we are using client principal to create sasl client instead 
> we need to use server principal to create sasl client and establish the sasl 
> context



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20788) Write up a doc about how to rolling upgrade from 1.x to 2.x

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16524658#comment-16524658
 ] 

Ashish Singhi commented on HBASE-20788:
---

That is ok.. It feels so good to see we can do a rolling upgrade from 1.x to 
2.x. Thanks for trying it.

Can you please tell us what issues we might get if we use ZK for assignment ?

> Write up a doc about how to rolling upgrade from 1.x to 2.x
> ---
>
> Key: HBASE-20788
> URL: https://issues.apache.org/jira/browse/HBASE-20788
> Project: HBase
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Blocker
> Fix For: 2.1.0
>
> Attachments: HBASE-20788.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20788) Write up a doc about how to rolling upgrade from 1.x to 2.x

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16524652#comment-16524652
 ] 

Ashish Singhi commented on HBASE-20788:
---

bq. zk-less assignment is enabled, i.e, hbase.assignment.usezk is set true.
Isn't this contradictory ? We say to enable ZK less assignment and then we say 
to use ZK for assignment. 
If hbase.assignment.usezk is set true, means we will use ZK for region 
assignment. Correct me if I have got it wrong.

> Write up a doc about how to rolling upgrade from 1.x to 2.x
> ---
>
> Key: HBASE-20788
> URL: https://issues.apache.org/jira/browse/HBASE-20788
> Project: HBase
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Blocker
> Fix For: 2.1.0
>
> Attachments: HBASE-20788.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20357) AccessControlClient API Enhancement

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500116#comment-16500116
 ] 

Ashish Singhi commented on HBASE-20357:
---

Can you please upload the patch in RB.

> AccessControlClient API Enhancement
> ---
>
> Key: HBASE-20357
> URL: https://issues.apache.org/jira/browse/HBASE-20357
> Project: HBase
>  Issue Type: Improvement
>  Components: security
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Attachments: HBASE-20357.master.001.patch
>
>
> *Background:*
>  Currently HBase ACLs can be retrieved based on the namespace or table name 
> only. There is no direct API available to retrieve the permissions based on 
> the namespace, table name, column family and column qualifier for specific 
> user.
> Client has to write application logic in multiple steps to retrieve ACLs 
> based on table name, column name and column qualifier for specific user.
>  HBase should enhance AccessControlClient APIs to simplyfy this.
> *AccessControlClient API should be extended with following APIs,*    
>  # To retrieve permissions based on the namespace, table name, column family 
> and column qualifier for specific user.
>   Permissions can be retrieved based on the following inputs,
>        - Namespace/Table (already available)
>        - Namespace/Table + UserName
>        - Table + CF
>        - Table + CF + UserName
>        - Table + CF + CQ
>        - Table + CF + CQ + UserName
>           Scope of retrieving permission will be as follows,
>                  - Same as existing
>        2. To validate whether a user is allowed to perform specified 
> operations on a particular table, will be useful to check user privilege 
> instead of getting ACD during client                                    
> operation.
>              User validation can be performed based on following inputs, 
>                   - Table + CF + CQ + UserName + Actions
>             Scope of validating user privilege,
>                     User can perform self check without any special privilege 
> but ADMIN privilege will be required to perform check for other users.
>                     For example, suppose there are two users "userA" & 
> "userB" then there can be below scenarios,
>                         - when userA want to check whether userA have 
> privilege to perform mentioned actions
>                                 > userA don't need ADMIN privilege, as it's a 
> self query.
>                        - when userA want to check whether userB have 
> privilege to perform mentioned actions,
>                                 > userA must have ADMIN or superuser 
> privilege, as it's trying to query for other user.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20590) REST Java client is not able to negotiate with the server in the secure mode

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-20590:
--
Release Note: Adds a negotiation logic between a secure java REST client 
and server. After this jira the Java REST client will start responding to the 
Negotiate challenge sent by the server. Adds RESTDemoClient which can be used 
to verify whether the secure Java REST client works against secure REST server 
or not.  (was: Adds a negotiation logic between a secure REST client and 
server. After this jira the REST client will start responding to the Negotiate 
challenge sent by the server.)

> REST Java client is not able to negotiate with the server in the secure mode
> 
>
> Key: HBASE-20590
> URL: https://issues.apache.org/jira/browse/HBASE-20590
> Project: HBase
>  Issue Type: Bug
>  Components: REST, security
>Affects Versions: 1.3.1
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
>Priority: Critical
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.2.7, 1.3.3, 2.0.1, 1.4.5
>
> Attachments: HBASE-20590.branch-1.patch, HBASE-20590.patch, 
> HBASE-20590.v1.patch, HBASE-20590.v2.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20590) REST Java client is not able to negotiate with the server in the secure mode

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-20590:
--
 Hadoop Flags: Reviewed
Fix Version/s: 3.0.0
 Release Note: Adds a negotiation logic between a secure REST client and 
server. After this jira the REST client will start responding to the Negotiate 
challenge sent by the server.

Thanks [~stack] for the review.
Committed the patch to branch-1.2+

> REST Java client is not able to negotiate with the server in the secure mode
> 
>
> Key: HBASE-20590
> URL: https://issues.apache.org/jira/browse/HBASE-20590
> Project: HBase
>  Issue Type: Bug
>  Components: REST, security
>Affects Versions: 1.3.1
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
>Priority: Critical
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.2.7, 1.3.3, 2.0.1, 1.4.5
>
> Attachments: HBASE-20590.branch-1.patch, HBASE-20590.patch, 
> HBASE-20590.v1.patch, HBASE-20590.v2.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20590) REST Java client is not able to negotiate with the server in the secure mode

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-20590:
--
Fix Version/s: 1.4.5
   2.0.1
   1.3.3
   1.2.7
   1.5.0
   2.1.0

> REST Java client is not able to negotiate with the server in the secure mode
> 
>
> Key: HBASE-20590
> URL: https://issues.apache.org/jira/browse/HBASE-20590
> Project: HBase
>  Issue Type: Bug
>  Components: REST, security
>Affects Versions: 1.3.1
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
>Priority: Critical
> Fix For: 2.1.0, 1.5.0, 1.2.7, 1.3.3, 2.0.1, 1.4.5
>
> Attachments: HBASE-20590.branch-1.patch, HBASE-20590.patch, 
> HBASE-20590.v1.patch, HBASE-20590.v2.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20590) REST Java client is not able to negotiate with the server in the secure mode

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499814#comment-16499814
 ] 

Ashish Singhi commented on HBASE-20590:
---

Tested the patch manually by running, 
./hbase org.apache.hadoop.hbase.rest.RESTDemoClient  9090 true

Without the patch, below error is logged,
{noformat}
2018-06-04 11:01:47,281 WARN  [main] httpclient.HttpMethodDirector: Unable to 
respond to any of these challenges: {negotiate=Negotiate}
Exception in thread "main" java.security.PrivilegedActionException: 
java.io.IOException: put request failed with 401
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.hbase.rest.RESTDemoClient.main(RESTDemoClient.java:70)
Caused by: java.io.IOException: put request failed with 401
at 
org.apache.hadoop.hbase.rest.client.RemoteHTable.put(RemoteHTable.java:426)
at 
org.apache.hadoop.hbase.rest.RESTDemoClient.run(RESTDemoClient.java:88)
at 
org.apache.hadoop.hbase.rest.RESTDemoClient$1.run(RESTDemoClient.java:73)
at 
org.apache.hadoop.hbase.rest.RESTDemoClient$1.run(RESTDemoClient.java:70)
... 3 more
{noformat}

I will commit the patch shortly, if no objections.

> REST Java client is not able to negotiate with the server in the secure mode
> 
>
> Key: HBASE-20590
> URL: https://issues.apache.org/jira/browse/HBASE-20590
> Project: HBase
>  Issue Type: Bug
>  Components: REST, security
>Affects Versions: 1.3.1
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
>Priority: Critical
> Attachments: HBASE-20590.branch-1.patch, HBASE-20590.patch, 
> HBASE-20590.v1.patch, HBASE-20590.v2.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20590) REST Java client is not able to negotiate with the server in the secure mode

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499806#comment-16499806
 ] 

Ashish Singhi commented on HBASE-20590:
---

bq. hbase-rest in the patch failed.
I don't see any failure here,
{noformat}
[INFO] Results:
[INFO] 
[INFO] Tests run: 190, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
{noformat}

bq. The patch has 1 ill-formed XML file(s).
Not related to the patch.

> REST Java client is not able to negotiate with the server in the secure mode
> 
>
> Key: HBASE-20590
> URL: https://issues.apache.org/jira/browse/HBASE-20590
> Project: HBase
>  Issue Type: Bug
>  Components: REST, security
>Affects Versions: 1.3.1
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
>Priority: Critical
> Attachments: HBASE-20590.branch-1.patch, HBASE-20590.patch, 
> HBASE-20590.v1.patch, HBASE-20590.v2.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20590) REST Java client is not able to negotiate with the server in the secure mode

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-20590:
--
Attachment: HBASE-20590.branch-1.patch

> REST Java client is not able to negotiate with the server in the secure mode
> 
>
> Key: HBASE-20590
> URL: https://issues.apache.org/jira/browse/HBASE-20590
> Project: HBase
>  Issue Type: Bug
>  Components: REST, security
>Affects Versions: 1.3.1
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
>Priority: Critical
> Attachments: HBASE-20590.branch-1.patch, HBASE-20590.patch, 
> HBASE-20590.v1.patch, HBASE-20590.v2.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20590) REST Java client is not able to negotiate with the server in the secure mode

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-20590:
--
Attachment: HBASE-20590.v2.patch

> REST Java client is not able to negotiate with the server in the secure mode
> 
>
> Key: HBASE-20590
> URL: https://issues.apache.org/jira/browse/HBASE-20590
> Project: HBase
>  Issue Type: Bug
>  Components: REST, security
>Affects Versions: 1.3.1
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
>Priority: Critical
> Attachments: HBASE-20590.patch, HBASE-20590.v1.patch, 
> HBASE-20590.v2.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20590) REST Java client is not able to negotiate with the server in the secure mode

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497603#comment-16497603
 ] 

Ashish Singhi commented on HBASE-20590:
---

bq. Would a test that puts up REST server and does secure client be too hard to 
do?
Yes its little hard to do.. It will require code refactoring of RESTServer 
class to adapt to MiniKdc. So only to test this I came with Demo RestClient 
which can be used to verify it against a secure cluster. 

Regarding checkstyle warnings, I will also try to by-pass them as its coming 
from example code not from the production code, so it should be ok to do I 
think.

> REST Java client is not able to negotiate with the server in the secure mode
> 
>
> Key: HBASE-20590
> URL: https://issues.apache.org/jira/browse/HBASE-20590
> Project: HBase
>  Issue Type: Bug
>  Components: REST, security
>Affects Versions: 1.3.1
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
>Priority: Critical
> Attachments: HBASE-20590.patch, HBASE-20590.v1.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20590) REST Java client is not able to negotiate with the server in the secure mode

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16492667#comment-16492667
 ] 

Ashish Singhi commented on HBASE-20590:
---

Regarding checkstyle warnings,
{quote}
./hbase-examples/src/main/java/org/apache/hadoop/hbase/rest/RESTDemoClient.java:41:import
 org.apache.yetus.audience.InterfaceAudience;: Wrong order for 
'org.apache.yetus.audience.InterfaceAudience' import. [ImportOrder]
{quote}

Imports are lexicographically sorted, not sure what change I should do to 
correct this warning.
 
{quote}
./hbase-examples/src/main/java/org/apache/hadoop/hbase/rest/RESTDemoClient.java:137:
 new AppConfigurationEntry("com.sun.security.auth.module.Krb5LoginModule",: 
'array initialization' child have incorrect indentation level 12, expected 
level should be 10.
{quote}
I have formatted the code as per 
https://github.com/apache/hbase/blob/master/dev-support/hbase_eclipse_formatter.xml.
 Should I manually correct this ?

> REST Java client is not able to negotiate with the server in the secure mode
> 
>
> Key: HBASE-20590
> URL: https://issues.apache.org/jira/browse/HBASE-20590
> Project: HBase
>  Issue Type: Bug
>  Components: REST, security
>Affects Versions: 1.3.1
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
>Priority: Critical
> Attachments: HBASE-20590.patch, HBASE-20590.v1.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20590) REST Java client is not able to negotiate with the server in the secure mode

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-20590:
--
Attachment: HBASE-20590.v1.patch

> REST Java client is not able to negotiate with the server in the secure mode
> 
>
> Key: HBASE-20590
> URL: https://issues.apache.org/jira/browse/HBASE-20590
> Project: HBase
>  Issue Type: Bug
>  Components: REST, security
>Affects Versions: 1.3.1
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
>Priority: Critical
> Attachments: HBASE-20590.patch, HBASE-20590.v1.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20590) REST Java client is not able to negotiate with the server in the secure mode

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-20590:
--
Status: Patch Available  (was: Open)

I have also added a RestDemoClient on similar lines of Thrift DemoClient.

Please review.

> REST Java client is not able to negotiate with the server in the secure mode
> 
>
> Key: HBASE-20590
> URL: https://issues.apache.org/jira/browse/HBASE-20590
> Project: HBase
>  Issue Type: Bug
>  Components: REST, security
>Affects Versions: 1.3.1
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
>Priority: Critical
> Attachments: HBASE-20590.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20590) REST Java client is not able to negotiate with the server in the secure mode

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-20590:
--
Attachment: HBASE-20590.patch

> REST Java client is not able to negotiate with the server in the secure mode
> 
>
> Key: HBASE-20590
> URL: https://issues.apache.org/jira/browse/HBASE-20590
> Project: HBase
>  Issue Type: Bug
>  Components: REST, security
>Affects Versions: 1.3.1
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
>Priority: Critical
> Attachments: HBASE-20590.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HBASE-20602) hbase.master.quota.observer.ignore property seems to be not taking effect

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16482192#comment-16482192
 ] 

Ashish Singhi edited comment on HBASE-20602 at 5/21/18 5:22 AM:


{quote}Is it designed to be like that?
{quote}
Yes, check HBASE-13220 that is what I got the comment from the original author 
of this feature.


was (Author: ashish singhi):
Check HBASE-13220

> hbase.master.quota.observer.ignore property seems to be not taking effect
> -
>
> Key: HBASE-20602
> URL: https://issues.apache.org/jira/browse/HBASE-20602
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 2.0.0
>Reporter: Biju Nair
>Priority: Major
>
> From [doc|https://hbase.apache.org/book.html#ops.space.quota.deletion] 
> setting {{hbase.master.quota.observer.ignore property to true}} will retain 
> the space quota even after table is deleted. But doesn't seem to be the case 
> i.e. whether the property is not defined which sets the value to {{false}} or 
> set the property to {{true}} in {{site.xml}}, the quota gets removed when the 
> corresponding table is dropped. Will verify whether it works in 1.x. Did a 
> grep on the master source, did get a hit on the property in code.
> Steps to reproduce
>  * Add this property and restart {{hbase}}
> {noformat}
>     
>         hbase.master.quota.observer.ignore
>         true
>     {noformat}
>  * Through {{hbase}} shell
>  * 
> {noformat}
> hbase(main):003:0> set_quota TYPE => SPACE, TABLE => 't1', LIMIT => '1G', 
> POLICY => NO_INSERTS
> Took 0.0317 seconds{noformat}
>  * 
> {noformat}
> hbase(main):005:0> create 't1','cf1'
> Created table t1
> Took 0.7904 seconds{noformat}
>  * 
> {noformat}
> hbase(main):006:0> list_quotas
> OWNER QUOTAS
> TABLE => t1 TYPE => SPACE, TABLE => t1, LIMIT => 1073741824, VIOLATION_POLICY 
> => NO_INSERTS
> 1 row(s){noformat}
>  * 
> {noformat}
> hbase(main):007:0> disable 't1'
> Took 0.4909 seconds
> hbase(main):008:0> list_quotas
> OWNER QUOTAS
> TABLE => t1 TYPE => SPACE, TABLE => t1, LIMIT => 1073741824, VIOLATION_POLICY 
> => NO_INSERTS
> 1 row(s)
> Took 0.0420 seconds{noformat}
>  * 
> {noformat}
> hbase(main):009:0> drop 't1'
> Took 0.1407 seconds{noformat}
>  * 
> {noformat}
> hbase(main):010:0> list_quotas
> OWNER QUOTAS
> 0 row(s)
> Took 0.0307 seconds{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20602) hbase.master.quota.observer.ignore property seems to be not taking effect

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16482192#comment-16482192
 ] 

Ashish Singhi commented on HBASE-20602:
---

Check HBASE-13220

> hbase.master.quota.observer.ignore property seems to be not taking effect
> -
>
> Key: HBASE-20602
> URL: https://issues.apache.org/jira/browse/HBASE-20602
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 2.0.0
>Reporter: Biju Nair
>Priority: Major
>
> From [doc|https://hbase.apache.org/book.html#ops.space.quota.deletion] 
> setting {{hbase.master.quota.observer.ignore property to true}} will retain 
> the space quota even after table is deleted. But doesn't seem to be the case 
> i.e. whether the property is not defined which sets the value to {{false}} or 
> set the property to {{true}} in {{site.xml}}, the quota gets removed when the 
> corresponding table is dropped. Will verify whether it works in 1.x. Did a 
> grep on the master source, did get a hit on the property in code.
> Steps to reproduce
>  * Add this property and restart {{hbase}}
> {noformat}
>     
>         hbase.master.quota.observer.ignore
>         true
>     {noformat}
>  * Through {{hbase}} shell
>  * 
> {noformat}
> hbase(main):003:0> set_quota TYPE => SPACE, TABLE => 't1', LIMIT => '1G', 
> POLICY => NO_INSERTS
> Took 0.0317 seconds{noformat}
>  * 
> {noformat}
> hbase(main):005:0> create 't1','cf1'
> Created table t1
> Took 0.7904 seconds{noformat}
>  * 
> {noformat}
> hbase(main):006:0> list_quotas
> OWNER QUOTAS
> TABLE => t1 TYPE => SPACE, TABLE => t1, LIMIT => 1073741824, VIOLATION_POLICY 
> => NO_INSERTS
> 1 row(s){noformat}
>  * 
> {noformat}
> hbase(main):007:0> disable 't1'
> Took 0.4909 seconds
> hbase(main):008:0> list_quotas
> OWNER QUOTAS
> TABLE => t1 TYPE => SPACE, TABLE => t1, LIMIT => 1073741824, VIOLATION_POLICY 
> => NO_INSERTS
> 1 row(s)
> Took 0.0420 seconds{noformat}
>  * 
> {noformat}
> hbase(main):009:0> drop 't1'
> Took 0.1407 seconds{noformat}
>  * 
> {noformat}
> hbase(main):010:0> list_quotas
> OWNER QUOTAS
> 0 row(s)
> Took 0.0307 seconds{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-20590) REST Java client is not able to negotiate with the server in the secure mode

[Ashish Singhi (JIRA)]

Ashish Singhi created HBASE-20590:
-

 Summary: REST Java client is not able to negotiate with the server 
in the secure mode
 Key: HBASE-20590
 URL: https://issues.apache.org/jira/browse/HBASE-20590
 Project: HBase
  Issue Type: Bug
  Components: REST, security
Affects Versions: 1.3.1
Reporter: Ashish Singhi
Assignee: Ashish Singhi






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20004) Client is not able to execute REST queries in a secure cluster

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-20004:
--
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 1.4.5
   1.3.3
   1.2.7
   1.5.0
   2.1.0
   3.0.0
 Release Note: 
Added 'hbase.rest.http.allow.options.method' configuration property to allow 
user to decide whether Rest Server HTTP should allow OPTIONS method or not. By 
default it is enabled in HBase 2.1.0+ versions and in other versions it is 
disabled.
Similarly 'hbase.thrift.http.allow.options.method' is added HBase 1.5, 2.1.0 
and 3.0.0 versions. It is disabled by default.
   Status: Resolved  (was: Patch Available)

Pushed to 1.2+ versions.

> Client is not able to execute REST queries in a secure cluster
> --
>
> Key: HBASE-20004
> URL: https://issues.apache.org/jira/browse/HBASE-20004
> Project: HBase
>  Issue Type: Bug
>  Components: REST, security
>Affects Versions: 1.3.1
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
>Priority: Minor
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.2.7, 1.3.3, 2.0.1, 1.4.5
>
> Attachments: HBASE-20004.branch-1.patch, 
> HBASE-20004.branch-1.v1.patch, HBASE-20004.branch-1.v1.patch, 
> HBASE-20004.branch-2.0.patch, HBASE-20004.patch, HBASE-20004.v1.patch, 
> HBASE-20004.v2.patch
>
>
> Firefox browser is not able to negotiate REST queries with server in secure 
> mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20004) Client is not able to execute REST queries in a secure cluster

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470753#comment-16470753
 ] 

Ashish Singhi commented on HBASE-20004:
---

Thanks for the review [~busbey] and [~stack].

branch-1 failures not related to the match. I will commit this patch shortly

> Client is not able to execute REST queries in a secure cluster
> --
>
> Key: HBASE-20004
> URL: https://issues.apache.org/jira/browse/HBASE-20004
> Project: HBase
>  Issue Type: Bug
>  Components: REST, security
>Affects Versions: 1.3.1
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
>Priority: Minor
> Fix For: 2.0.1
>
> Attachments: HBASE-20004.branch-1.patch, 
> HBASE-20004.branch-1.v1.patch, HBASE-20004.branch-1.v1.patch, 
> HBASE-20004.branch-2.0.patch, HBASE-20004.patch, HBASE-20004.v1.patch, 
> HBASE-20004.v2.patch
>
>
> Firefox browser is not able to negotiate REST queries with server in secure 
> mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HBASE-20004) Client is not able to execute REST queries in a secure cluster

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470753#comment-16470753
 ] 

Ashish Singhi edited comment on HBASE-20004 at 5/10/18 5:07 PM:


Thanks for the review [~busbey] and [~stack].

branch-1 failures are not related to the patch. I will commit this patch shortly


was (Author: ashish singhi):
Thanks for the review [~busbey] and [~stack].

branch-1 failures not related to the match. I will commit this patch shortly

> Client is not able to execute REST queries in a secure cluster
> --
>
> Key: HBASE-20004
> URL: https://issues.apache.org/jira/browse/HBASE-20004
> Project: HBase
>  Issue Type: Bug
>  Components: REST, security
>Affects Versions: 1.3.1
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
>Priority: Minor
> Fix For: 2.0.1
>
> Attachments: HBASE-20004.branch-1.patch, 
> HBASE-20004.branch-1.v1.patch, HBASE-20004.branch-1.v1.patch, 
> HBASE-20004.branch-2.0.patch, HBASE-20004.patch, HBASE-20004.v1.patch, 
> HBASE-20004.v2.patch
>
>
> Firefox browser is not able to negotiate REST queries with server in secure 
> mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20004) Client is not able to execute REST queries in a secure cluster

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470376#comment-16470376
 ] 

Ashish Singhi commented on HBASE-20004:
---

Reattaching for another run for branch-1 patch.

Will wait for one more day for commit if QA is all ok. Waiting for for your 
blessings sir, [~stack].

> Client is not able to execute REST queries in a secure cluster
> --
>
> Key: HBASE-20004
> URL: https://issues.apache.org/jira/browse/HBASE-20004
> Project: HBase
>  Issue Type: Bug
>  Components: REST, security
>Affects Versions: 1.3.1
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
>Priority: Minor
> Attachments: HBASE-20004.branch-1.patch, 
> HBASE-20004.branch-1.v1.patch, HBASE-20004.branch-1.v1.patch, 
> HBASE-20004.branch-2.0.patch, HBASE-20004.patch, HBASE-20004.v1.patch, 
> HBASE-20004.v2.patch
>
>
> Firefox browser is not able to negotiate REST queries with server in secure 
> mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20004) Client is not able to execute REST queries in a secure cluster

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-20004:
--
Attachment: HBASE-20004.branch-1.v1.patch

> Client is not able to execute REST queries in a secure cluster
> --
>
> Key: HBASE-20004
> URL: https://issues.apache.org/jira/browse/HBASE-20004
> Project: HBase
>  Issue Type: Bug
>  Components: REST, security
>Affects Versions: 1.3.1
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
>Priority: Minor
> Attachments: HBASE-20004.branch-1.patch, 
> HBASE-20004.branch-1.v1.patch, HBASE-20004.branch-1.v1.patch, 
> HBASE-20004.branch-2.0.patch, HBASE-20004.patch, HBASE-20004.v1.patch, 
> HBASE-20004.v2.patch
>
>
> Firefox browser is not able to negotiate REST queries with server in secure 
> mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20004) Client is not able to execute REST queries in a secure cluster

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468680#comment-16468680
 ] 

Ashish Singhi commented on HBASE-20004:
---

I will update the release note based on the patch approval.

> Client is not able to execute REST queries in a secure cluster
> --
>
> Key: HBASE-20004
> URL: https://issues.apache.org/jira/browse/HBASE-20004
> Project: HBase
>  Issue Type: Bug
>  Components: REST, security
>Affects Versions: 1.3.1
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
>Priority: Minor
> Attachments: HBASE-20004.branch-1.patch, 
> HBASE-20004.branch-1.v1.patch, HBASE-20004.branch-2.0.patch, 
> HBASE-20004.patch, HBASE-20004.v1.patch, HBASE-20004.v2.patch
>
>
> Firefox browser is not able to negotiate REST queries with server in secure 
> mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20004) Client is not able to execute REST queries in a secure cluster

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468678#comment-16468678
 ] 

Ashish Singhi commented on HBASE-20004:
---

I have attached branch-2.0 and branch-1 patch which keeps the default behavior 
as the existing code.

Planning to commit this tomorrow unless objections. 

 [~stack], can I push this change to branch-2.0 ?

> Client is not able to execute REST queries in a secure cluster
> --
>
> Key: HBASE-20004
> URL: https://issues.apache.org/jira/browse/HBASE-20004
> Project: HBase
>  Issue Type: Bug
>  Components: REST, security
>Affects Versions: 1.3.1
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
>Priority: Minor
> Attachments: HBASE-20004.branch-1.patch, 
> HBASE-20004.branch-1.v1.patch, HBASE-20004.branch-2.0.patch, 
> HBASE-20004.patch, HBASE-20004.v1.patch, HBASE-20004.v2.patch
>
>
> Firefox browser is not able to negotiate REST queries with server in secure 
> mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20004) Client is not able to execute REST queries in a secure cluster

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-20004:
--
Attachment: HBASE-20004.branch-1.v1.patch

> Client is not able to execute REST queries in a secure cluster
> --
>
> Key: HBASE-20004
> URL: https://issues.apache.org/jira/browse/HBASE-20004
> Project: HBase
>  Issue Type: Bug
>  Components: REST, security
>Affects Versions: 1.3.1
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
>Priority: Minor
> Attachments: HBASE-20004.branch-1.patch, 
> HBASE-20004.branch-1.v1.patch, HBASE-20004.branch-2.0.patch, 
> HBASE-20004.patch, HBASE-20004.v1.patch, HBASE-20004.v2.patch
>
>
> Firefox browser is not able to negotiate REST queries with server in secure 
> mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20004) Client is not able to execute REST queries in a secure cluster

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-20004:
--
Attachment: (was: HBASE-20004.branch-2.0.patch)

> Client is not able to execute REST queries in a secure cluster
> --
>
> Key: HBASE-20004
> URL: https://issues.apache.org/jira/browse/HBASE-20004
> Project: HBase
>  Issue Type: Bug
>  Components: REST, security
>Affects Versions: 1.3.1
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
>Priority: Minor
> Attachments: HBASE-20004.branch-1.patch, 
> HBASE-20004.branch-1.v1.patch, HBASE-20004.branch-2.0.patch, 
> HBASE-20004.patch, HBASE-20004.v1.patch, HBASE-20004.v2.patch
>
>
> Firefox browser is not able to negotiate REST queries with server in secure 
> mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20004) Client is not able to execute REST queries in a secure cluster

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-20004:
--
Attachment: HBASE-20004.branch-2.0.patch

> Client is not able to execute REST queries in a secure cluster
> --
>
> Key: HBASE-20004
> URL: https://issues.apache.org/jira/browse/HBASE-20004
> Project: HBase
>  Issue Type: Bug
>  Components: REST, security
>Affects Versions: 1.3.1
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
>Priority: Minor
> Attachments: HBASE-20004.branch-1.patch, 
> HBASE-20004.branch-2.0.patch, HBASE-20004.branch-2.0.patch, 
> HBASE-20004.patch, HBASE-20004.v1.patch, HBASE-20004.v2.patch
>
>
> Firefox browser is not able to negotiate REST queries with server in secure 
> mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20004) Client is not able to execute REST queries in a secure cluster

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-20004:
--
Attachment: HBASE-20004.branch-2.0.patch

> Client is not able to execute REST queries in a secure cluster
> --
>
> Key: HBASE-20004
> URL: https://issues.apache.org/jira/browse/HBASE-20004
> Project: HBase
>  Issue Type: Bug
>  Components: REST, security
>Affects Versions: 1.3.1
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
>Priority: Minor
> Attachments: HBASE-20004.branch-1.patch, 
> HBASE-20004.branch-2.0.patch, HBASE-20004.patch, HBASE-20004.v1.patch, 
> HBASE-20004.v2.patch
>
>
> Firefox browser is not able to negotiate REST queries with server in secure 
> mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20004) Client is not able to execute REST queries in a secure cluster

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-20004:
--
Attachment: (was: HBASE-20004.branch-2.0.patch)

> Client is not able to execute REST queries in a secure cluster
> --
>
> Key: HBASE-20004
> URL: https://issues.apache.org/jira/browse/HBASE-20004
> Project: HBase
>  Issue Type: Bug
>  Components: REST, security
>Affects Versions: 1.3.1
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
>Priority: Minor
> Attachments: HBASE-20004.branch-1.patch, HBASE-20004.patch, 
> HBASE-20004.v1.patch, HBASE-20004.v2.patch
>
>
> Firefox browser is not able to negotiate REST queries with server in secure 
> mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20004) Client is not able to execute REST queries in a secure cluster

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-20004:
--
Attachment: HBASE-20004.branch-2.0.patch

> Client is not able to execute REST queries in a secure cluster
> --
>
> Key: HBASE-20004
> URL: https://issues.apache.org/jira/browse/HBASE-20004
> Project: HBase
>  Issue Type: Bug
>  Components: REST, security
>Affects Versions: 1.3.1
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
>Priority: Minor
> Attachments: HBASE-20004.branch-1.patch, 
> HBASE-20004.branch-2.0.patch, HBASE-20004.patch, HBASE-20004.v1.patch, 
> HBASE-20004.v2.patch
>
>
> Firefox browser is not able to negotiate REST queries with server in secure 
> mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20004) Client is not able to execute REST queries in a secure cluster

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467682#comment-16467682
 ] 

Ashish Singhi commented on HBASE-20004:
---

@Sean Busbey, can you also please review the patch.

> Client is not able to execute REST queries in a secure cluster
> --
>
> Key: HBASE-20004
> URL: https://issues.apache.org/jira/browse/HBASE-20004
> Project: HBase
>  Issue Type: Bug
>  Components: REST, security
>Affects Versions: 1.3.1
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
>Priority: Minor
> Attachments: HBASE-20004.branch-1.patch, HBASE-20004.patch, 
> HBASE-20004.v1.patch, HBASE-20004.v2.patch
>
>
> Firefox browser is not able to negotiate REST queries with server in secure 
> mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20004) Client is not able to execute REST queries in a secure cluster

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-20004:
--
Attachment: HBASE-20004.v2.patch

> Client is not able to execute REST queries in a secure cluster
> --
>
> Key: HBASE-20004
> URL: https://issues.apache.org/jira/browse/HBASE-20004
> Project: HBase
>  Issue Type: Bug
>  Components: REST, security
>Affects Versions: 1.3.1
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
>Priority: Minor
> Attachments: HBASE-20004.branch-1.patch, HBASE-20004.patch, 
> HBASE-20004.v1.patch, HBASE-20004.v2.patch
>
>
> Firefox browser is not able to negotiate REST queries with server in secure 
> mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20004) Client is not able to execute REST queries in a secure cluster

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467205#comment-16467205
 ] 

Ashish Singhi commented on HBASE-20004:
---

Attached v2 patch addressing Ted's comment.

> Client is not able to execute REST queries in a secure cluster
> --
>
> Key: HBASE-20004
> URL: https://issues.apache.org/jira/browse/HBASE-20004
> Project: HBase
>  Issue Type: Bug
>  Components: REST, security
>Affects Versions: 1.3.1
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
>Priority: Minor
> Attachments: HBASE-20004.branch-1.patch, HBASE-20004.patch, 
> HBASE-20004.v1.patch, HBASE-20004.v2.patch
>
>
> Firefox browser is not able to negotiate REST queries with server in secure 
> mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20004) Client is not able to execute REST queries in a secure cluster

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467170#comment-16467170
 ] 

Ashish Singhi commented on HBASE-20004:
---

I added that first then removed, thinking that it works as it is and adding one 
more new configuration which I felt whose value will not be changed from its 
default, made no sense to me. Do you think otherwise and want me to add ?

> Client is not able to execute REST queries in a secure cluster
> --
>
> Key: HBASE-20004
> URL: https://issues.apache.org/jira/browse/HBASE-20004
> Project: HBase
>  Issue Type: Bug
>  Components: REST, security
>Affects Versions: 1.3.1
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
>Priority: Minor
> Attachments: HBASE-20004.branch-1.patch, HBASE-20004.patch, 
> HBASE-20004.v1.patch
>
>
> Firefox browser is not able to negotiate REST queries with server in secure 
> mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HBASE-6970) hbase-deamon.sh creates/updates pid file even when that start failed.

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-6970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi reassigned HBASE-6970:


Assignee: (was: Ashish Singhi)

> hbase-deamon.sh creates/updates pid file even when that start failed.
> -
>
> Key: HBASE-6970
> URL: https://issues.apache.org/jira/browse/HBASE-6970
> Project: HBase
>  Issue Type: Bug
>  Components: Usability
>Reporter: Lars Hofhansl
>Priority: Major
>
> We just ran into a strange issue where could neither start nor stop services 
> with hbase-deamon.sh.
> The problem is this:
> {code}
> nohup nice -n $HBASE_NICENESS "$HBASE_HOME"/bin/hbase \
> --config "${HBASE_CONF_DIR}" \
> $command "$@" $startStop > "$logout" 2>&1 < /dev/null &
> echo $! > $pid
> {code}
> So the pid file is created or updated even when the start of the service 
> failed. The next stop command will then fail, because the pid file has the 
> wrong pid in it.
> Edit: Spelling and more spelling errors.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HBASE-5622) Improve efficiency of mapred vesion of RowCounter

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi reassigned HBASE-5622:


Assignee: (was: Ashish Singhi)

> Improve efficiency of mapred vesion of RowCounter
> -
>
> Key: HBASE-5622
> URL: https://issues.apache.org/jira/browse/HBASE-5622
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Lars Hofhansl
>Priority: Minor
> Attachments: HBASE-5622.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-5622) Improve efficiency of mapred vesion of RowCounter

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-5622:
-
Status: Open  (was: Patch Available)

> Improve efficiency of mapred vesion of RowCounter
> -
>
> Key: HBASE-5622
> URL: https://issues.apache.org/jira/browse/HBASE-5622
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Lars Hofhansl
>Priority: Minor
> Attachments: HBASE-5622.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (HBASE-5742) LoadIncrementalHFiles throws too generic of an exception

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-5742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi resolved HBASE-5742.
--
Resolution: Not A Problem

The constructor in the context no more exists.

> LoadIncrementalHFiles throws too generic of an exception
> 
>
> Key: HBASE-5742
> URL: https://issues.apache.org/jira/browse/HBASE-5742
> Project: HBase
>  Issue Type: Improvement
>  Components: mapreduce
>Reporter: David Capwell
>Assignee: Ashish Singhi
>Priority: Major
>
> In 0.90 the LoadIncrementalHFiles constructor did not throw an exception, now 
> it throws Exception.  The constructor should ether not throw an exception or 
> throw ZooKeeperConnectionException, and MasterNotRunningException since those 
> come from the HBaseAdmin call.
> https://github.com/apache/hbase/blob/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java#L105



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-13644) Expand AC testing coverage to include all combinations of scope and permissions for region interface

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-13644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-13644:
--
Labels: beginner  (was: )

> Expand AC testing coverage to include all combinations of scope and 
> permissions for region interface
> 
>
> Key: HBASE-13644
> URL: https://issues.apache.org/jira/browse/HBASE-13644
> Project: HBase
>  Issue Type: Sub-task
>  Components: security, test
>Reporter: Ashish Singhi
>Priority: Major
>  Labels: beginner
>
> As of now, the tests in TestAccessController and TestAccessController2 
> doesn't cover all the combinations of Scope and Permissions. Ideally, we 
> should have testing coverage for the entire region interface in [ACL 
> matrix|https://hbase.apache.org/book/appendix_acl_matrix.html].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HBASE-13644) Expand AC testing coverage to include all combinations of scope and permissions for region interface

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-13644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi reassigned HBASE-13644:
-

Assignee: (was: Ashish Singhi)

> Expand AC testing coverage to include all combinations of scope and 
> permissions for region interface
> 
>
> Key: HBASE-13644
> URL: https://issues.apache.org/jira/browse/HBASE-13644
> Project: HBase
>  Issue Type: Sub-task
>  Components: security, test
>Reporter: Ashish Singhi
>Priority: Major
>
> As of now, the tests in TestAccessController and TestAccessController2 
> doesn't cover all the combinations of Scope and Permissions. Ideally, we 
> should have testing coverage for the entire region interface in [ACL 
> matrix|https://hbase.apache.org/book/appendix_acl_matrix.html].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20004) Client is not able to execute REST queries in a secure cluster

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-20004:
--
Status: Patch Available  (was: In Progress)

> Client is not able to execute REST queries in a secure cluster
> --
>
> Key: HBASE-20004
> URL: https://issues.apache.org/jira/browse/HBASE-20004
> Project: HBase
>  Issue Type: Bug
>  Components: REST, security
>Affects Versions: 1.3.1
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
>Priority: Minor
> Attachments: HBASE-20004.branch-1.patch, HBASE-20004.patch, 
> HBASE-20004.v1.patch
>
>
> Firefox browser is not able to negotiate REST queries with server in secure 
> mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20004) Client is not able to execute REST queries in a secure cluster

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465881#comment-16465881
 ] 

Ashish Singhi commented on HBASE-20004:
---

Attached a patch making the option of enabling and disabling the OPTIONS method 
as configurable. By default OPTIONS method will not be added in constraint 
mapping and will be added in the other places.

Please review.

> Client is not able to execute REST queries in a secure cluster
> --
>
> Key: HBASE-20004
> URL: https://issues.apache.org/jira/browse/HBASE-20004
> Project: HBase
>  Issue Type: Bug
>  Components: REST, security
>Affects Versions: 1.3.1
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
>Priority: Minor
> Attachments: HBASE-20004.branch-1.patch, HBASE-20004.patch, 
> HBASE-20004.v1.patch
>
>
> Firefox browser is not able to negotiate REST queries with server in secure 
> mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20004) Client is not able to execute REST queries in a secure cluster

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-20004:
--
Attachment: HBASE-20004.v1.patch

> Client is not able to execute REST queries in a secure cluster
> --
>
> Key: HBASE-20004
> URL: https://issues.apache.org/jira/browse/HBASE-20004
> Project: HBase
>  Issue Type: Bug
>  Components: REST, security
>Affects Versions: 1.3.1
>Reporter: Ashish Singhi
>Assignee: Ashish Singhi
>Priority: Minor
> Attachments: HBASE-20004.branch-1.patch, HBASE-20004.patch, 
> HBASE-20004.v1.patch
>
>
> Firefox browser is not able to negotiate REST queries with server in secure 
> mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20485) Copy constructor of Scan doesn't copy the readType and replicaId

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452097#comment-16452097
 ] 

Ashish Singhi commented on HBASE-20485:
---

Good finding, [~nihaljain.cs]!!

> Copy constructor of Scan doesn't copy the readType and replicaId
> 
>
> Key: HBASE-20485
> URL: https://issues.apache.org/jira/browse/HBASE-20485
> Project: HBase
>  Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Nihal Jain
>Priority: Minor
>  Labels: beginner, beginners
> Fix For: 3.0.0
>
> Attachments: HBASE-20485.master.001.patch
>
>
> The field was introduced by HBASE-17045.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20485) Copy constructor of Scan doesn't copy the readType and replicaId

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451987#comment-16451987
 ] 

Ashish Singhi commented on HBASE-20485:
---

I think by some means if we can check the size of actual and copied object in 
test then that should be fine. 

> Copy constructor of Scan doesn't copy the readType and replicaId
> 
>
> Key: HBASE-20485
> URL: https://issues.apache.org/jira/browse/HBASE-20485
> Project: HBase
>  Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Nihal Jain
>Priority: Minor
>  Labels: beginner, beginners
> Fix For: 3.0.0
>
> Attachments: HBASE-20485.master.001.patch
>
>
> The field was introduced by HBASE-17045.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20485) Copy constructor of Scan doesn't copy the readType and replicaId

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451956#comment-16451956
 ] 

Ashish Singhi commented on HBASE-20485:
---

 Just thinking, is there any way to avoid such mistakes in the future ? Where 
any new property added for Scan is not missed from the copy constructor.

> Copy constructor of Scan doesn't copy the readType and replicaId
> 
>
> Key: HBASE-20485
> URL: https://issues.apache.org/jira/browse/HBASE-20485
> Project: HBase
>  Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Nihal Jain
>Priority: Minor
>  Labels: beginner, beginners
> Fix For: 3.0.0
>
> Attachments: HBASE-20485.master.001.patch
>
>
> The field was introduced by HBASE-17045.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20406) HBase Thrift HTTP - Shouldn't handle TRACE/OPTIONS methods

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438705#comment-16438705
 ] 

Ashish Singhi commented on HBASE-20406:
---

HBASE-20004. Plan was to make it configurable. I plan to work on it in the 
coming week.

> HBase Thrift HTTP - Shouldn't handle TRACE/OPTIONS methods
> --
>
> Key: HBASE-20406
> URL: https://issues.apache.org/jira/browse/HBASE-20406
> Project: HBase
>  Issue Type: Improvement
>  Components: security, Thrift
>Reporter: Kevin Risden
>Assignee: Kevin Risden
>Priority: Major
> Attachments: HBASE-20406.master.001.patch
>
>
> HBASE-10473 introduced a utility HttpServerUtil.constrainHttpMethods to 
> prevent Jetty from answering on TRACE and OPTIONS methods. This should be 
> added to Thrift in HTTP mode as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20349) [DOC] upgrade guide should call out removal of prefix-tree data block encoding

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16434895#comment-16434895
 ] 

Ashish Singhi commented on HBASE-20349:
---

LGTM

> [DOC] upgrade guide should call out removal of prefix-tree data block encoding
> --
>
> Key: HBASE-20349
> URL: https://issues.apache.org/jira/browse/HBASE-20349
> Project: HBase
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: Sean Busbey
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-20349.master.001.patch, 
> HBASE-20349.master.002.patch
>
>
> See HBASE-19179. Needs to be in the upgrade section. Right now there's just 
> an offhand mention in Appendix E about the removal.
> Since we can no longer read data encoded with prefix tree, we should include 
> instructions on rewriting data.
> Also ensure we don't have spurious references to it. (the help from ltt 
> that's in the ref guide still lists it, for example).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-15291) FileSystem not closed in secure bulkLoad

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433610#comment-16433610
 ] 

Ashish Singhi commented on HBASE-15291:
---

Attached branch-1 patch which I pushed to branch-1.3, branch-1.4 and 
branch-1.5. and branch-1.2 patch for branch-1.2 for reference.

> FileSystem not closed in secure bulkLoad
> 
>
> Key: HBASE-15291
> URL: https://issues.apache.org/jira/browse/HBASE-15291
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.0.2, 0.98.16.1
>Reporter: Yong Zhang
>Assignee: Ashish Singhi
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.2.7, 1.3.3, 1.4.4, 2.0.1
>
> Attachments: HBASE-15291-branch-1.2.patch, 
> HBASE-15291-branch-1.patch, HBASE-15291-revert-master.patch, 
> HBASE-15291.001.patch, HBASE-15291.002.patch, HBASE-15291.003.patch, 
> HBASE-15291.004.patch, HBASE-15291.addendum, HBASE-15291.patch, 
> HBASE-15291.v1.patch, HBASE-15291.v2.patch, HBASE-15291.v2.patch, patch
>
>
> FileSystem not closed in secure bulkLoad after bulkLoad  finish, it will 
> cause memory used more and more if too many bulkLoad .



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-15291) FileSystem not closed in secure bulkLoad

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-15291:
--
Attachment: HBASE-15291-branch-1.patch

> FileSystem not closed in secure bulkLoad
> 
>
> Key: HBASE-15291
> URL: https://issues.apache.org/jira/browse/HBASE-15291
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.0.2, 0.98.16.1
>Reporter: Yong Zhang
>Assignee: Ashish Singhi
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.2.7, 1.3.3, 1.4.4, 2.0.1
>
> Attachments: HBASE-15291-branch-1.2.patch, 
> HBASE-15291-branch-1.patch, HBASE-15291-revert-master.patch, 
> HBASE-15291.001.patch, HBASE-15291.002.patch, HBASE-15291.003.patch, 
> HBASE-15291.004.patch, HBASE-15291.addendum, HBASE-15291.patch, 
> HBASE-15291.v1.patch, HBASE-15291.v2.patch, HBASE-15291.v2.patch, patch
>
>
> FileSystem not closed in secure bulkLoad after bulkLoad  finish, it will 
> cause memory used more and more if too many bulkLoad .



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-15291) FileSystem not closed in secure bulkLoad

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-15291:
--
Attachment: HBASE-15291-branch-1.2.patch

> FileSystem not closed in secure bulkLoad
> 
>
> Key: HBASE-15291
> URL: https://issues.apache.org/jira/browse/HBASE-15291
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.0.2, 0.98.16.1
>Reporter: Yong Zhang
>Assignee: Ashish Singhi
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.2.7, 1.3.3, 1.4.4, 2.0.1
>
> Attachments: HBASE-15291-branch-1.2.patch, 
> HBASE-15291-branch-1.patch, HBASE-15291-revert-master.patch, 
> HBASE-15291.001.patch, HBASE-15291.002.patch, HBASE-15291.003.patch, 
> HBASE-15291.004.patch, HBASE-15291.addendum, HBASE-15291.patch, 
> HBASE-15291.v1.patch, HBASE-15291.v2.patch, HBASE-15291.v2.patch, patch
>
>
> FileSystem not closed in secure bulkLoad after bulkLoad  finish, it will 
> cause memory used more and more if too many bulkLoad .



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-15291) FileSystem not closed in secure bulkLoad

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-15291:
--
   Resolution: Fixed
Fix Version/s: 2.1.0
   Status: Resolved  (was: Patch Available)

Pushed to branch-1.2+

> FileSystem not closed in secure bulkLoad
> 
>
> Key: HBASE-15291
> URL: https://issues.apache.org/jira/browse/HBASE-15291
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.0.2, 0.98.16.1
>Reporter: Yong Zhang
>Assignee: Ashish Singhi
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.2.7, 1.3.3, 1.4.4, 2.0.1
>
> Attachments: HBASE-15291-revert-master.patch, HBASE-15291.001.patch, 
> HBASE-15291.002.patch, HBASE-15291.003.patch, HBASE-15291.004.patch, 
> HBASE-15291.addendum, HBASE-15291.patch, HBASE-15291.v1.patch, 
> HBASE-15291.v2.patch, HBASE-15291.v2.patch, patch
>
>
> FileSystem not closed in secure bulkLoad after bulkLoad  finish, it will 
> cause memory used more and more if too many bulkLoad .



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-15291) FileSystem not closed in secure bulkLoad

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433464#comment-16433464
 ] 

Ashish Singhi commented on HBASE-15291:
---

Thanks.

Let me commit this.

> FileSystem not closed in secure bulkLoad
> 
>
> Key: HBASE-15291
> URL: https://issues.apache.org/jira/browse/HBASE-15291
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.0.2, 0.98.16.1
>Reporter: Yong Zhang
>Assignee: Ashish Singhi
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.2.7, 1.3.3, 1.4.4, 2.0.1
>
> Attachments: HBASE-15291-revert-master.patch, HBASE-15291.001.patch, 
> HBASE-15291.002.patch, HBASE-15291.003.patch, HBASE-15291.004.patch, 
> HBASE-15291.addendum, HBASE-15291.patch, HBASE-15291.v1.patch, 
> HBASE-15291.v2.patch, HBASE-15291.v2.patch, patch
>
>
> FileSystem not closed in secure bulkLoad after bulkLoad  finish, it will 
> cause memory used more and more if too many bulkLoad .



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (HBASE-16499) slow replication for small HBase clusters

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi resolved HBASE-16499.
---
Resolution: Fixed

> slow replication for small HBase clusters
> -
>
> Key: HBASE-16499
> URL: https://issues.apache.org/jira/browse/HBASE-16499
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: Vikas Vishwakarma
>Assignee: Ashish Singhi
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-16499-addendum.patch, HBASE-16499.patch, 
> HBASE-16499.patch
>
>
> For small clusters 10-20 nodes we recently observed that replication is 
> progressing very slowly when we do bulk writes and there is lot of lag 
> accumulation on AgeOfLastShipped / SizeOfLogQueue. From the logs we observed 
> that the number of threads used for shipping wal edits in parallel comes from 
> the following equation in HBaseInterClusterReplicationEndpoint
> int n = Math.min(Math.min(this.maxThreads, entries.size()/100+1),
>   replicationSinkMgr.getSinks().size());
> ... 
>   for (int i=0; i entryLists.add(new ArrayList(entries.size()/n+1)); <-- 
> batch size
>   }
> ...
> for (int i=0; i  .
> // RuntimeExceptions encountered here bubble up and are handled 
> in ReplicationSource
> pool.submit(createReplicator(entryLists.get(i), i));  <-- 
> concurrency 
> futures++;
>   }
> }
> maxThreads is fixed & configurable and since we are taking min of the three 
> values n gets decided based replicationSinkMgr.getSinks().size() when we have 
> enough edits to replicate
> replicationSinkMgr.getSinks().size() is decided based on 
> int numSinks = (int) Math.ceil(slaveAddresses.size() * ratio);
> where ratio is this.ratio = conf.getFloat("replication.source.ratio", 
> DEFAULT_REPLICATION_SOURCE_RATIO);
> Currently DEFAULT_REPLICATION_SOURCE_RATIO is set to 10% so for small 
> clusters of size 10-20 RegionServers  the value we get for numSinks and hence 
> n is very small like 1 or 2. This substantially reduces the pool concurrency 
> used for shipping wal edits in parallel effectively slowing down replication 
> for small clusters and causing lot of lag accumulation in AgeOfLastShipped. 
> Sometimes it takes tens of hours to clear off the entire replication queue 
> even after the client has finished writing on the source side. 
> We are running tests by varying replication.source.ratio and have seen 
> multi-fold improvement in total replication time (will update the results 
> here). I wanted to propose here that we should increase the default value for 
> replication.source.ratio also so that we have sufficient concurrency even for 
> small clusters. We figured it out after lot of iterations and debugging so 
> probably slightly higher default will save the trouble. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-16499) slow replication for small HBase clusters

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426508#comment-16426508
 ] 

Ashish Singhi commented on HBASE-16499:
---

Thanks for the review. I have pushed the addendum to only master branch.

Addendum didn't apply as we have not committed HBASE-20273 in branch-2 and 
branch-2.0

> slow replication for small HBase clusters
> -
>
> Key: HBASE-16499
> URL: https://issues.apache.org/jira/browse/HBASE-16499
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: Vikas Vishwakarma
>Assignee: Ashish Singhi
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-16499-addendum.patch, HBASE-16499.patch, 
> HBASE-16499.patch
>
>
> For small clusters 10-20 nodes we recently observed that replication is 
> progressing very slowly when we do bulk writes and there is lot of lag 
> accumulation on AgeOfLastShipped / SizeOfLogQueue. From the logs we observed 
> that the number of threads used for shipping wal edits in parallel comes from 
> the following equation in HBaseInterClusterReplicationEndpoint
> int n = Math.min(Math.min(this.maxThreads, entries.size()/100+1),
>   replicationSinkMgr.getSinks().size());
> ... 
>   for (int i=0; i entryLists.add(new ArrayList(entries.size()/n+1)); <-- 
> batch size
>   }
> ...
> for (int i=0; i  .
> // RuntimeExceptions encountered here bubble up and are handled 
> in ReplicationSource
> pool.submit(createReplicator(entryLists.get(i), i));  <-- 
> concurrency 
> futures++;
>   }
> }
> maxThreads is fixed & configurable and since we are taking min of the three 
> values n gets decided based replicationSinkMgr.getSinks().size() when we have 
> enough edits to replicate
> replicationSinkMgr.getSinks().size() is decided based on 
> int numSinks = (int) Math.ceil(slaveAddresses.size() * ratio);
> where ratio is this.ratio = conf.getFloat("replication.source.ratio", 
> DEFAULT_REPLICATION_SOURCE_RATIO);
> Currently DEFAULT_REPLICATION_SOURCE_RATIO is set to 10% so for small 
> clusters of size 10-20 RegionServers  the value we get for numSinks and hence 
> n is very small like 1 or 2. This substantially reduces the pool concurrency 
> used for shipping wal edits in parallel effectively slowing down replication 
> for small clusters and causing lot of lag accumulation in AgeOfLastShipped. 
> Sometimes it takes tens of hours to clear off the entire replication queue 
> even after the client has finished writing on the source side. 
> We are running tests by varying replication.source.ratio and have seen 
> multi-fold improvement in total replication time (will update the results 
> here). I wanted to propose here that we should increase the default value for 
> replication.source.ratio also so that we have sufficient concurrency even for 
> small clusters. We figured it out after lot of iterations and debugging so 
> probably slightly higher default will save the trouble. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HBASE-16499) slow replication for small HBase clusters

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426508#comment-16426508
 ] 

Ashish Singhi edited comment on HBASE-16499 at 4/5/18 5:52 AM:
---

Thanks for the review. I have pushed the addendum to only master branch.

Addendum didn't apply to branch-2 and branch-2.0, as we have not committed 
HBASE-20273 in branch-2 and branch-2.0


was (Author: ashish singhi):
Thanks for the review. I have pushed the addendum to only master branch.

Addendum didn't apply as we have not committed HBASE-20273 in branch-2 and 
branch-2.0

> slow replication for small HBase clusters
> -
>
> Key: HBASE-16499
> URL: https://issues.apache.org/jira/browse/HBASE-16499
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: Vikas Vishwakarma
>Assignee: Ashish Singhi
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-16499-addendum.patch, HBASE-16499.patch, 
> HBASE-16499.patch
>
>
> For small clusters 10-20 nodes we recently observed that replication is 
> progressing very slowly when we do bulk writes and there is lot of lag 
> accumulation on AgeOfLastShipped / SizeOfLogQueue. From the logs we observed 
> that the number of threads used for shipping wal edits in parallel comes from 
> the following equation in HBaseInterClusterReplicationEndpoint
> int n = Math.min(Math.min(this.maxThreads, entries.size()/100+1),
>   replicationSinkMgr.getSinks().size());
> ... 
>   for (int i=0; i entryLists.add(new ArrayList(entries.size()/n+1)); <-- 
> batch size
>   }
> ...
> for (int i=0; i  .
> // RuntimeExceptions encountered here bubble up and are handled 
> in ReplicationSource
> pool.submit(createReplicator(entryLists.get(i), i));  <-- 
> concurrency 
> futures++;
>   }
> }
> maxThreads is fixed & configurable and since we are taking min of the three 
> values n gets decided based replicationSinkMgr.getSinks().size() when we have 
> enough edits to replicate
> replicationSinkMgr.getSinks().size() is decided based on 
> int numSinks = (int) Math.ceil(slaveAddresses.size() * ratio);
> where ratio is this.ratio = conf.getFloat("replication.source.ratio", 
> DEFAULT_REPLICATION_SOURCE_RATIO);
> Currently DEFAULT_REPLICATION_SOURCE_RATIO is set to 10% so for small 
> clusters of size 10-20 RegionServers  the value we get for numSinks and hence 
> n is very small like 1 or 2. This substantially reduces the pool concurrency 
> used for shipping wal edits in parallel effectively slowing down replication 
> for small clusters and causing lot of lag accumulation in AgeOfLastShipped. 
> Sometimes it takes tens of hours to clear off the entire replication queue 
> even after the client has finished writing on the source side. 
> We are running tests by varying replication.source.ratio and have seen 
> multi-fold improvement in total replication time (will update the results 
> here). I wanted to propose here that we should increase the default value for 
> replication.source.ratio also so that we have sufficient concurrency even for 
> small clusters. We figured it out after lot of iterations and debugging so 
> probably slightly higher default will save the trouble. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20273) [DOC] include call out of additional changed config defaults in 2.0 upgrade

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426507#comment-16426507
 ] 

Ashish Singhi commented on HBASE-20273:
---

Should we push this change to branch-2.0 also ? Because that is from where user 
will be referring the hbase book which will be part of release tar ball.

> [DOC] include call out of additional changed config defaults in 2.0 upgrade
> ---
>
> Key: HBASE-20273
> URL: https://issues.apache.org/jira/browse/HBASE-20273
> Project: HBase
>  Issue Type: Sub-task
>  Components: documentation
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Mike Drob
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20273.patch
>
>
> Copied from feedback on HBASE-19158 from [~mdrob]:
> {quote}
> Default settings/configuration properties changed/renamed:
> HBASE-19919
> HBASE-19148
> HBASE-18307
> HBASE-17314
> HBASE-15784
> HBASE-15027
> HBASE-14906
> HBASE-14521
> {quote}
> More detail later from [~mdrob]:
> {quote}
> would like to see notes that:
> hbase.master.cleaner.interval changed from 1 min to 10 min
> MasterProcedureConstants.MASTER_PROCEDURE_THREADS defaults to CPU/4 instead 
> of CPU
> hbase.rpc.server.nativetransport renamed to hbase.netty.nativetransport
> hbase.netty.rpc.server.worker.count renamed to base.netty.worker.count
> hbase.hfile.compactions.discharger.interval renamed to 
> hbase.hfile.compaction.discharger.interval
> hbase.hregion.percolumnfamilyflush.size.lower.bound removed at site level, 
> but can still be applied at table level
> hbase.client.reties.number now counts the total number of retries, not the 
> total number of tries
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-16499) slow replication for small HBase clusters

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426452#comment-16426452
 ] 

Ashish Singhi commented on HBASE-16499:
---

[~stack]  or [~busbey] can you please check the addendum attached.

> slow replication for small HBase clusters
> -
>
> Key: HBASE-16499
> URL: https://issues.apache.org/jira/browse/HBASE-16499
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: Vikas Vishwakarma
>Assignee: Ashish Singhi
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-16499-addendum.patch, HBASE-16499.patch, 
> HBASE-16499.patch
>
>
> For small clusters 10-20 nodes we recently observed that replication is 
> progressing very slowly when we do bulk writes and there is lot of lag 
> accumulation on AgeOfLastShipped / SizeOfLogQueue. From the logs we observed 
> that the number of threads used for shipping wal edits in parallel comes from 
> the following equation in HBaseInterClusterReplicationEndpoint
> int n = Math.min(Math.min(this.maxThreads, entries.size()/100+1),
>   replicationSinkMgr.getSinks().size());
> ... 
>   for (int i=0; i entryLists.add(new ArrayList(entries.size()/n+1)); <-- 
> batch size
>   }
> ...
> for (int i=0; i  .
> // RuntimeExceptions encountered here bubble up and are handled 
> in ReplicationSource
> pool.submit(createReplicator(entryLists.get(i), i));  <-- 
> concurrency 
> futures++;
>   }
> }
> maxThreads is fixed & configurable and since we are taking min of the three 
> values n gets decided based replicationSinkMgr.getSinks().size() when we have 
> enough edits to replicate
> replicationSinkMgr.getSinks().size() is decided based on 
> int numSinks = (int) Math.ceil(slaveAddresses.size() * ratio);
> where ratio is this.ratio = conf.getFloat("replication.source.ratio", 
> DEFAULT_REPLICATION_SOURCE_RATIO);
> Currently DEFAULT_REPLICATION_SOURCE_RATIO is set to 10% so for small 
> clusters of size 10-20 RegionServers  the value we get for numSinks and hence 
> n is very small like 1 or 2. This substantially reduces the pool concurrency 
> used for shipping wal edits in parallel effectively slowing down replication 
> for small clusters and causing lot of lag accumulation in AgeOfLastShipped. 
> Sometimes it takes tens of hours to clear off the entire replication queue 
> even after the client has finished writing on the source side. 
> We are running tests by varying replication.source.ratio and have seen 
> multi-fold improvement in total replication time (will update the results 
> here). I wanted to propose here that we should increase the default value for 
> replication.source.ratio also so that we have sufficient concurrency even for 
> small clusters. We figured it out after lot of iterations and debugging so 
> probably slightly higher default will save the trouble. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20231) Not able to delete column family from a row using RemoteHTable

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425226#comment-16425226
 ] 

Ashish Singhi commented on HBASE-20231:
---

Done. Thanks

> Not able to delete column family from a row using RemoteHTable
> --
>
> Key: HBASE-20231
> URL: https://issues.apache.org/jira/browse/HBASE-20231
> Project: HBase
>  Issue Type: Bug
>  Components: REST
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Fix For: 1.5.0, 1.2.7, 1.3.3, 1.4.4, 2.0.0
>
> Attachments: HBASE-20231-branch-1-v2.patch, 
> HBASE-20231-branch-1-v3.patch, HBASE-20231-branch-1.3.patch, 
> HBASE-20231-branch-1.patch, HBASE-20231-v2.patch, HBASE-20231-v3.patch, 
> HBASE-20231.patch
>
>
> Example code to reproduce the issue,
> {code:java}
> Cluster cluster = new Cluster();
> cluster.add("rest-server-IP", rest-server-port);
> Client client = new Client(cluster);
> RemoteHTable table = new RemoteHTable(client, "t1");
> // Insert few records,
> Put put = new Put(Bytes.toBytes("r1"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c2"), Bytes.toBytes("c2"));
> put.add(Bytes.toBytes("cf2"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> table.put(put);
> put = new Put(Bytes.toBytes("r2"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c2"), Bytes.toBytes("c2"));
> put.add(Bytes.toBytes("cf2"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> table.put(put);
> // Delete the entire column family from the row
> Delete del = new Delete(Bytes.toBytes("r2"));
> del.addFamily(Bytes.toBytes("cf1"));
> table.delete(del);
> {code}
> Here the problem is in building row specification in 
> RemoteHTable.buildRowSpec(). Row specification is framed as "/t1/r2/cf1:" 
> instead of "/t1/r2/cf1". 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HBASE-20231) Not able to delete column family from a row using RemoteHTable

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425226#comment-16425226
 ] 

Ashish Singhi edited comment on HBASE-20231 at 4/4/18 9:15 AM:
---

Committed. Thanks


was (Author: ashish singhi):
Done. Thanks

> Not able to delete column family from a row using RemoteHTable
> --
>
> Key: HBASE-20231
> URL: https://issues.apache.org/jira/browse/HBASE-20231
> Project: HBase
>  Issue Type: Bug
>  Components: REST
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Fix For: 1.5.0, 1.2.7, 1.3.3, 1.4.4, 2.0.0
>
> Attachments: HBASE-20231-branch-1-v2.patch, 
> HBASE-20231-branch-1-v3.patch, HBASE-20231-branch-1.3.patch, 
> HBASE-20231-branch-1.patch, HBASE-20231-v2.patch, HBASE-20231-v3.patch, 
> HBASE-20231.patch
>
>
> Example code to reproduce the issue,
> {code:java}
> Cluster cluster = new Cluster();
> cluster.add("rest-server-IP", rest-server-port);
> Client client = new Client(cluster);
> RemoteHTable table = new RemoteHTable(client, "t1");
> // Insert few records,
> Put put = new Put(Bytes.toBytes("r1"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c2"), Bytes.toBytes("c2"));
> put.add(Bytes.toBytes("cf2"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> table.put(put);
> put = new Put(Bytes.toBytes("r2"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c2"), Bytes.toBytes("c2"));
> put.add(Bytes.toBytes("cf2"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> table.put(put);
> // Delete the entire column family from the row
> Delete del = new Delete(Bytes.toBytes("r2"));
> del.addFamily(Bytes.toBytes("cf1"));
> table.delete(del);
> {code}
> Here the problem is in building row specification in 
> RemoteHTable.buildRowSpec(). Row specification is framed as "/t1/r2/cf1:" 
> instead of "/t1/r2/cf1". 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20231) Not able to delete column family from a row using RemoteHTable

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-20231:
--
Fix Version/s: 1.3.3
   1.2.7

> Not able to delete column family from a row using RemoteHTable
> --
>
> Key: HBASE-20231
> URL: https://issues.apache.org/jira/browse/HBASE-20231
> Project: HBase
>  Issue Type: Bug
>  Components: REST
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Fix For: 1.5.0, 1.2.7, 1.3.3, 1.4.4, 2.0.0
>
> Attachments: HBASE-20231-branch-1-v2.patch, 
> HBASE-20231-branch-1-v3.patch, HBASE-20231-branch-1.3.patch, 
> HBASE-20231-branch-1.patch, HBASE-20231-v2.patch, HBASE-20231-v3.patch, 
> HBASE-20231.patch
>
>
> Example code to reproduce the issue,
> {code:java}
> Cluster cluster = new Cluster();
> cluster.add("rest-server-IP", rest-server-port);
> Client client = new Client(cluster);
> RemoteHTable table = new RemoteHTable(client, "t1");
> // Insert few records,
> Put put = new Put(Bytes.toBytes("r1"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c2"), Bytes.toBytes("c2"));
> put.add(Bytes.toBytes("cf2"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> table.put(put);
> put = new Put(Bytes.toBytes("r2"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c2"), Bytes.toBytes("c2"));
> put.add(Bytes.toBytes("cf2"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> table.put(put);
> // Delete the entire column family from the row
> Delete del = new Delete(Bytes.toBytes("r2"));
> del.addFamily(Bytes.toBytes("cf1"));
> table.delete(del);
> {code}
> Here the problem is in building row specification in 
> RemoteHTable.buildRowSpec(). Row specification is framed as "/t1/r2/cf1:" 
> instead of "/t1/r2/cf1". 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-16499) slow replication for small HBase clusters

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425041#comment-16425041
 ] 

Ashish Singhi commented on HBASE-16499:
---

[~busbey], can you please review the addendum so that I can commit the same 
after your blessings.

> slow replication for small HBase clusters
> -
>
> Key: HBASE-16499
> URL: https://issues.apache.org/jira/browse/HBASE-16499
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: Vikas Vishwakarma
>Assignee: Ashish Singhi
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-16499-addendum.patch, HBASE-16499.patch, 
> HBASE-16499.patch
>
>
> For small clusters 10-20 nodes we recently observed that replication is 
> progressing very slowly when we do bulk writes and there is lot of lag 
> accumulation on AgeOfLastShipped / SizeOfLogQueue. From the logs we observed 
> that the number of threads used for shipping wal edits in parallel comes from 
> the following equation in HBaseInterClusterReplicationEndpoint
> int n = Math.min(Math.min(this.maxThreads, entries.size()/100+1),
>   replicationSinkMgr.getSinks().size());
> ... 
>   for (int i=0; i entryLists.add(new ArrayList(entries.size()/n+1)); <-- 
> batch size
>   }
> ...
> for (int i=0; i  .
> // RuntimeExceptions encountered here bubble up and are handled 
> in ReplicationSource
> pool.submit(createReplicator(entryLists.get(i), i));  <-- 
> concurrency 
> futures++;
>   }
> }
> maxThreads is fixed & configurable and since we are taking min of the three 
> values n gets decided based replicationSinkMgr.getSinks().size() when we have 
> enough edits to replicate
> replicationSinkMgr.getSinks().size() is decided based on 
> int numSinks = (int) Math.ceil(slaveAddresses.size() * ratio);
> where ratio is this.ratio = conf.getFloat("replication.source.ratio", 
> DEFAULT_REPLICATION_SOURCE_RATIO);
> Currently DEFAULT_REPLICATION_SOURCE_RATIO is set to 10% so for small 
> clusters of size 10-20 RegionServers  the value we get for numSinks and hence 
> n is very small like 1 or 2. This substantially reduces the pool concurrency 
> used for shipping wal edits in parallel effectively slowing down replication 
> for small clusters and causing lot of lag accumulation in AgeOfLastShipped. 
> Sometimes it takes tens of hours to clear off the entire replication queue 
> even after the client has finished writing on the source side. 
> We are running tests by varying replication.source.ratio and have seen 
> multi-fold improvement in total replication time (will update the results 
> here). I wanted to propose here that we should increase the default value for 
> replication.source.ratio also so that we have sufficient concurrency even for 
> small clusters. We figured it out after lot of iterations and debugging so 
> probably slightly higher default will save the trouble. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-16499) slow replication for small HBase clusters

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-16499:
--
Attachment: HBASE-16499-addendum.patch

> slow replication for small HBase clusters
> -
>
> Key: HBASE-16499
> URL: https://issues.apache.org/jira/browse/HBASE-16499
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: Vikas Vishwakarma
>Assignee: Ashish Singhi
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-16499-addendum.patch, HBASE-16499.patch, 
> HBASE-16499.patch
>
>
> For small clusters 10-20 nodes we recently observed that replication is 
> progressing very slowly when we do bulk writes and there is lot of lag 
> accumulation on AgeOfLastShipped / SizeOfLogQueue. From the logs we observed 
> that the number of threads used for shipping wal edits in parallel comes from 
> the following equation in HBaseInterClusterReplicationEndpoint
> int n = Math.min(Math.min(this.maxThreads, entries.size()/100+1),
>   replicationSinkMgr.getSinks().size());
> ... 
>   for (int i=0; i entryLists.add(new ArrayList(entries.size()/n+1)); <-- 
> batch size
>   }
> ...
> for (int i=0; i  .
> // RuntimeExceptions encountered here bubble up and are handled 
> in ReplicationSource
> pool.submit(createReplicator(entryLists.get(i), i));  <-- 
> concurrency 
> futures++;
>   }
> }
> maxThreads is fixed & configurable and since we are taking min of the three 
> values n gets decided based replicationSinkMgr.getSinks().size() when we have 
> enough edits to replicate
> replicationSinkMgr.getSinks().size() is decided based on 
> int numSinks = (int) Math.ceil(slaveAddresses.size() * ratio);
> where ratio is this.ratio = conf.getFloat("replication.source.ratio", 
> DEFAULT_REPLICATION_SOURCE_RATIO);
> Currently DEFAULT_REPLICATION_SOURCE_RATIO is set to 10% so for small 
> clusters of size 10-20 RegionServers  the value we get for numSinks and hence 
> n is very small like 1 or 2. This substantially reduces the pool concurrency 
> used for shipping wal edits in parallel effectively slowing down replication 
> for small clusters and causing lot of lag accumulation in AgeOfLastShipped. 
> Sometimes it takes tens of hours to clear off the entire replication queue 
> even after the client has finished writing on the source side. 
> We are running tests by varying replication.source.ratio and have seen 
> multi-fold improvement in total replication time (will update the results 
> here). I wanted to propose here that we should increase the default value for 
> replication.source.ratio also so that we have sufficient concurrency even for 
> small clusters. We figured it out after lot of iterations and debugging so 
> probably slightly higher default will save the trouble. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-16499) slow replication for small HBase clusters

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425037#comment-16425037
 ] 

Ashish Singhi commented on HBASE-16499:
---

Sure will add that immediately.. Sorry for not doing it before, I was not aware 
of it!

> slow replication for small HBase clusters
> -
>
> Key: HBASE-16499
> URL: https://issues.apache.org/jira/browse/HBASE-16499
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: Vikas Vishwakarma
>Assignee: Ashish Singhi
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-16499.patch, HBASE-16499.patch
>
>
> For small clusters 10-20 nodes we recently observed that replication is 
> progressing very slowly when we do bulk writes and there is lot of lag 
> accumulation on AgeOfLastShipped / SizeOfLogQueue. From the logs we observed 
> that the number of threads used for shipping wal edits in parallel comes from 
> the following equation in HBaseInterClusterReplicationEndpoint
> int n = Math.min(Math.min(this.maxThreads, entries.size()/100+1),
>   replicationSinkMgr.getSinks().size());
> ... 
>   for (int i=0; i entryLists.add(new ArrayList(entries.size()/n+1)); <-- 
> batch size
>   }
> ...
> for (int i=0; i  .
> // RuntimeExceptions encountered here bubble up and are handled 
> in ReplicationSource
> pool.submit(createReplicator(entryLists.get(i), i));  <-- 
> concurrency 
> futures++;
>   }
> }
> maxThreads is fixed & configurable and since we are taking min of the three 
> values n gets decided based replicationSinkMgr.getSinks().size() when we have 
> enough edits to replicate
> replicationSinkMgr.getSinks().size() is decided based on 
> int numSinks = (int) Math.ceil(slaveAddresses.size() * ratio);
> where ratio is this.ratio = conf.getFloat("replication.source.ratio", 
> DEFAULT_REPLICATION_SOURCE_RATIO);
> Currently DEFAULT_REPLICATION_SOURCE_RATIO is set to 10% so for small 
> clusters of size 10-20 RegionServers  the value we get for numSinks and hence 
> n is very small like 1 or 2. This substantially reduces the pool concurrency 
> used for shipping wal edits in parallel effectively slowing down replication 
> for small clusters and causing lot of lag accumulation in AgeOfLastShipped. 
> Sometimes it takes tens of hours to clear off the entire replication queue 
> even after the client has finished writing on the source side. 
> We are running tests by varying replication.source.ratio and have seen 
> multi-fold improvement in total replication time (will update the results 
> here). I wanted to propose here that we should increase the default value for 
> replication.source.ratio also so that we have sufficient concurrency even for 
> small clusters. We figured it out after lot of iterations and debugging so 
> probably slightly higher default will save the trouble. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-20231) Not able to delete column family from a row using RemoteHTable

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-20231:
--
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 2.0.0
   1.4.4
   Status: Resolved  (was: Patch Available)

> Not able to delete column family from a row using RemoteHTable
> --
>
> Key: HBASE-20231
> URL: https://issues.apache.org/jira/browse/HBASE-20231
> Project: HBase
>  Issue Type: Bug
>  Components: REST
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Fix For: 1.5.0, 1.4.4, 2.0.0
>
> Attachments: HBASE-20231-branch-1-v2.patch, 
> HBASE-20231-branch-1-v3.patch, HBASE-20231-branch-1.patch, 
> HBASE-20231-v2.patch, HBASE-20231-v3.patch, HBASE-20231.patch
>
>
> Example code to reproduce the issue,
> {code:java}
> Cluster cluster = new Cluster();
> cluster.add("rest-server-IP", rest-server-port);
> Client client = new Client(cluster);
> RemoteHTable table = new RemoteHTable(client, "t1");
> // Insert few records,
> Put put = new Put(Bytes.toBytes("r1"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c2"), Bytes.toBytes("c2"));
> put.add(Bytes.toBytes("cf2"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> table.put(put);
> put = new Put(Bytes.toBytes("r2"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c2"), Bytes.toBytes("c2"));
> put.add(Bytes.toBytes("cf2"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> table.put(put);
> // Delete the entire column family from the row
> Delete del = new Delete(Bytes.toBytes("r2"));
> del.addFamily(Bytes.toBytes("cf1"));
> table.delete(del);
> {code}
> Here the problem is in building row specification in 
> RemoteHTable.buildRowSpec(). Row specification is framed as "/t1/r2/cf1:" 
> instead of "/t1/r2/cf1". 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (HBASE-20231) Not able to delete column family from a row using RemoteHTable

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425008#comment-16425008
 ] 

Ashish Singhi edited comment on HBASE-20231 at 4/4/18 4:50 AM:
---

Thanks [~pankaj2461] for the patches and [~yuzhih...@gmail.com], [~uagashe] and 
[~chia7712] for the reviews.

I have pushed this change to branch-1.4+. [~pankaj2461], if you attach patch 
for branch-1.2 and branch-1.3, I will commit there as well.


was (Author: ashish singhi):
Thanks [~pankaj2461] for the patches and [~yuzhih...@gmail.com], [~uagashe] and 
[~chia7712] for the reviews.

I have pushed this change to branch-1.4+. If you attach patch for branch-1.2 
and branch-1.3, I will commit there as well.

> Not able to delete column family from a row using RemoteHTable
> --
>
> Key: HBASE-20231
> URL: https://issues.apache.org/jira/browse/HBASE-20231
> Project: HBase
>  Issue Type: Bug
>  Components: REST
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Fix For: 1.5.0
>
> Attachments: HBASE-20231-branch-1-v2.patch, 
> HBASE-20231-branch-1-v3.patch, HBASE-20231-branch-1.patch, 
> HBASE-20231-v2.patch, HBASE-20231-v3.patch, HBASE-20231.patch
>
>
> Example code to reproduce the issue,
> {code:java}
> Cluster cluster = new Cluster();
> cluster.add("rest-server-IP", rest-server-port);
> Client client = new Client(cluster);
> RemoteHTable table = new RemoteHTable(client, "t1");
> // Insert few records,
> Put put = new Put(Bytes.toBytes("r1"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c2"), Bytes.toBytes("c2"));
> put.add(Bytes.toBytes("cf2"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> table.put(put);
> put = new Put(Bytes.toBytes("r2"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c2"), Bytes.toBytes("c2"));
> put.add(Bytes.toBytes("cf2"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> table.put(put);
> // Delete the entire column family from the row
> Delete del = new Delete(Bytes.toBytes("r2"));
> del.addFamily(Bytes.toBytes("cf1"));
> table.delete(del);
> {code}
> Here the problem is in building row specification in 
> RemoteHTable.buildRowSpec(). Row specification is framed as "/t1/r2/cf1:" 
> instead of "/t1/r2/cf1". 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20231) Not able to delete column family from a row using RemoteHTable

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-20231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425008#comment-16425008
 ] 

Ashish Singhi commented on HBASE-20231:
---

Thanks [~pankaj2461] for the patches and [~yuzhih...@gmail.com], [~uagashe] and 
[~chia7712] for the reviews.

I have pushed this change to branch-1.4+. If you attach patch for branch-1.2 
and branch-1.3, I will commit there as well.

> Not able to delete column family from a row using RemoteHTable
> --
>
> Key: HBASE-20231
> URL: https://issues.apache.org/jira/browse/HBASE-20231
> Project: HBase
>  Issue Type: Bug
>  Components: REST
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Major
> Fix For: 1.5.0
>
> Attachments: HBASE-20231-branch-1-v2.patch, 
> HBASE-20231-branch-1-v3.patch, HBASE-20231-branch-1.patch, 
> HBASE-20231-v2.patch, HBASE-20231-v3.patch, HBASE-20231.patch
>
>
> Example code to reproduce the issue,
> {code:java}
> Cluster cluster = new Cluster();
> cluster.add("rest-server-IP", rest-server-port);
> Client client = new Client(cluster);
> RemoteHTable table = new RemoteHTable(client, "t1");
> // Insert few records,
> Put put = new Put(Bytes.toBytes("r1"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c2"), Bytes.toBytes("c2"));
> put.add(Bytes.toBytes("cf2"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> table.put(put);
> put = new Put(Bytes.toBytes("r2"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> put.add(Bytes.toBytes("cf1"), Bytes.toBytes("c2"), Bytes.toBytes("c2"));
> put.add(Bytes.toBytes("cf2"), Bytes.toBytes("c1"), Bytes.toBytes("c1"));
> table.put(put);
> // Delete the entire column family from the row
> Delete del = new Delete(Bytes.toBytes("r2"));
> del.addFamily(Bytes.toBytes("cf1"));
> table.delete(del);
> {code}
> Here the problem is in building row specification in 
> RemoteHTable.buildRowSpec(). Row specification is framed as "/t1/r2/cf1:" 
> instead of "/t1/r2/cf1". 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-16499) slow replication for small HBase clusters

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-16499:
--
  Resolution: Fixed
Hadoop Flags: Reviewed
Release Note: Changed the default value for replication.source.ratio from 
0.1 to 0.5. Which means now by default 50% of the total RegionServers in peer 
cluster(s) will participate in replication.
  Status: Resolved  (was: Patch Available)

Thanks [~stack] for the review.

I have pushed this to branch-2.0+

> slow replication for small HBase clusters
> -
>
> Key: HBASE-16499
> URL: https://issues.apache.org/jira/browse/HBASE-16499
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: Vikas Vishwakarma
>Assignee: Ashish Singhi
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-16499.patch, HBASE-16499.patch
>
>
> For small clusters 10-20 nodes we recently observed that replication is 
> progressing very slowly when we do bulk writes and there is lot of lag 
> accumulation on AgeOfLastShipped / SizeOfLogQueue. From the logs we observed 
> that the number of threads used for shipping wal edits in parallel comes from 
> the following equation in HBaseInterClusterReplicationEndpoint
> int n = Math.min(Math.min(this.maxThreads, entries.size()/100+1),
>   replicationSinkMgr.getSinks().size());
> ... 
>   for (int i=0; i entryLists.add(new ArrayList(entries.size()/n+1)); <-- 
> batch size
>   }
> ...
> for (int i=0; i  .
> // RuntimeExceptions encountered here bubble up and are handled 
> in ReplicationSource
> pool.submit(createReplicator(entryLists.get(i), i));  <-- 
> concurrency 
> futures++;
>   }
> }
> maxThreads is fixed & configurable and since we are taking min of the three 
> values n gets decided based replicationSinkMgr.getSinks().size() when we have 
> enough edits to replicate
> replicationSinkMgr.getSinks().size() is decided based on 
> int numSinks = (int) Math.ceil(slaveAddresses.size() * ratio);
> where ratio is this.ratio = conf.getFloat("replication.source.ratio", 
> DEFAULT_REPLICATION_SOURCE_RATIO);
> Currently DEFAULT_REPLICATION_SOURCE_RATIO is set to 10% so for small 
> clusters of size 10-20 RegionServers  the value we get for numSinks and hence 
> n is very small like 1 or 2. This substantially reduces the pool concurrency 
> used for shipping wal edits in parallel effectively slowing down replication 
> for small clusters and causing lot of lag accumulation in AgeOfLastShipped. 
> Sometimes it takes tens of hours to clear off the entire replication queue 
> even after the client has finished writing on the source side. 
> We are running tests by varying replication.source.ratio and have seen 
> multi-fold improvement in total replication time (will update the results 
> here). I wanted to propose here that we should increase the default value for 
> replication.source.ratio also so that we have sufficient concurrency even for 
> small clusters. We figured it out after lot of iterations and debugging so 
> probably slightly higher default will save the trouble. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-16499) slow replication for small HBase clusters

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-16499:
--
Attachment: HBASE-16499.patch

> slow replication for small HBase clusters
> -
>
> Key: HBASE-16499
> URL: https://issues.apache.org/jira/browse/HBASE-16499
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: Vikas Vishwakarma
>Assignee: Ashish Singhi
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-16499.patch, HBASE-16499.patch
>
>
> For small clusters 10-20 nodes we recently observed that replication is 
> progressing very slowly when we do bulk writes and there is lot of lag 
> accumulation on AgeOfLastShipped / SizeOfLogQueue. From the logs we observed 
> that the number of threads used for shipping wal edits in parallel comes from 
> the following equation in HBaseInterClusterReplicationEndpoint
> int n = Math.min(Math.min(this.maxThreads, entries.size()/100+1),
>   replicationSinkMgr.getSinks().size());
> ... 
>   for (int i=0; i entryLists.add(new ArrayList(entries.size()/n+1)); <-- 
> batch size
>   }
> ...
> for (int i=0; i  .
> // RuntimeExceptions encountered here bubble up and are handled 
> in ReplicationSource
> pool.submit(createReplicator(entryLists.get(i), i));  <-- 
> concurrency 
> futures++;
>   }
> }
> maxThreads is fixed & configurable and since we are taking min of the three 
> values n gets decided based replicationSinkMgr.getSinks().size() when we have 
> enough edits to replicate
> replicationSinkMgr.getSinks().size() is decided based on 
> int numSinks = (int) Math.ceil(slaveAddresses.size() * ratio);
> where ratio is this.ratio = conf.getFloat("replication.source.ratio", 
> DEFAULT_REPLICATION_SOURCE_RATIO);
> Currently DEFAULT_REPLICATION_SOURCE_RATIO is set to 10% so for small 
> clusters of size 10-20 RegionServers  the value we get for numSinks and hence 
> n is very small like 1 or 2. This substantially reduces the pool concurrency 
> used for shipping wal edits in parallel effectively slowing down replication 
> for small clusters and causing lot of lag accumulation in AgeOfLastShipped. 
> Sometimes it takes tens of hours to clear off the entire replication queue 
> even after the client has finished writing on the source side. 
> We are running tests by varying replication.source.ratio and have seen 
> multi-fold improvement in total replication time (will update the results 
> here). I wanted to propose here that we should increase the default value for 
> replication.source.ratio also so that we have sufficient concurrency even for 
> small clusters. We figured it out after lot of iterations and debugging so 
> probably slightly higher default will save the trouble. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-16499) slow replication for small HBase clusters

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423947#comment-16423947
 ] 

Ashish Singhi commented on HBASE-16499:
---

May be same RS in peer was getting selected for replication every time!

> slow replication for small HBase clusters
> -
>
> Key: HBASE-16499
> URL: https://issues.apache.org/jira/browse/HBASE-16499
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: Vikas Vishwakarma
>Assignee: Ashish Singhi
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-16499.patch, HBASE-16499.patch
>
>
> For small clusters 10-20 nodes we recently observed that replication is 
> progressing very slowly when we do bulk writes and there is lot of lag 
> accumulation on AgeOfLastShipped / SizeOfLogQueue. From the logs we observed 
> that the number of threads used for shipping wal edits in parallel comes from 
> the following equation in HBaseInterClusterReplicationEndpoint
> int n = Math.min(Math.min(this.maxThreads, entries.size()/100+1),
>   replicationSinkMgr.getSinks().size());
> ... 
>   for (int i=0; i entryLists.add(new ArrayList(entries.size()/n+1)); <-- 
> batch size
>   }
> ...
> for (int i=0; i  .
> // RuntimeExceptions encountered here bubble up and are handled 
> in ReplicationSource
> pool.submit(createReplicator(entryLists.get(i), i));  <-- 
> concurrency 
> futures++;
>   }
> }
> maxThreads is fixed & configurable and since we are taking min of the three 
> values n gets decided based replicationSinkMgr.getSinks().size() when we have 
> enough edits to replicate
> replicationSinkMgr.getSinks().size() is decided based on 
> int numSinks = (int) Math.ceil(slaveAddresses.size() * ratio);
> where ratio is this.ratio = conf.getFloat("replication.source.ratio", 
> DEFAULT_REPLICATION_SOURCE_RATIO);
> Currently DEFAULT_REPLICATION_SOURCE_RATIO is set to 10% so for small 
> clusters of size 10-20 RegionServers  the value we get for numSinks and hence 
> n is very small like 1 or 2. This substantially reduces the pool concurrency 
> used for shipping wal edits in parallel effectively slowing down replication 
> for small clusters and causing lot of lag accumulation in AgeOfLastShipped. 
> Sometimes it takes tens of hours to clear off the entire replication queue 
> even after the client has finished writing on the source side. 
> We are running tests by varying replication.source.ratio and have seen 
> multi-fold improvement in total replication time (will update the results 
> here). I wanted to propose here that we should increase the default value for 
> replication.source.ratio also so that we have sufficient concurrency even for 
> small clusters. We figured it out after lot of iterations and debugging so 
> probably slightly higher default will save the trouble. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-15291) FileSystem not closed in secure bulkLoad

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423946#comment-16423946
 ] 

Ashish Singhi commented on HBASE-15291:
---

Test failure is not related.

> FileSystem not closed in secure bulkLoad
> 
>
> Key: HBASE-15291
> URL: https://issues.apache.org/jira/browse/HBASE-15291
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.0.2, 0.98.16.1
>Reporter: Yong Zhang
>Assignee: Ashish Singhi
>Priority: Major
> Attachments: HBASE-15291-revert-master.patch, HBASE-15291.001.patch, 
> HBASE-15291.002.patch, HBASE-15291.003.patch, HBASE-15291.004.patch, 
> HBASE-15291.addendum, HBASE-15291.patch, HBASE-15291.v1.patch, 
> HBASE-15291.v2.patch, HBASE-15291.v2.patch, patch
>
>
> FileSystem not closed in secure bulkLoad after bulkLoad  finish, it will 
> cause memory used more and more if too many bulkLoad .



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-16499) slow replication for small HBase clusters

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418528#comment-16418528
 ] 

Ashish Singhi commented on HBASE-16499:
---

Thanks Stack for the Go.

I have attached patch for the same. Please review.

> slow replication for small HBase clusters
> -
>
> Key: HBASE-16499
> URL: https://issues.apache.org/jira/browse/HBASE-16499
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: Vikas Vishwakarma
>Assignee: Vikas Vishwakarma
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.0
>
> Attachments: HBASE-16499.patch
>
>
> For small clusters 10-20 nodes we recently observed that replication is 
> progressing very slowly when we do bulk writes and there is lot of lag 
> accumulation on AgeOfLastShipped / SizeOfLogQueue. From the logs we observed 
> that the number of threads used for shipping wal edits in parallel comes from 
> the following equation in HBaseInterClusterReplicationEndpoint
> int n = Math.min(Math.min(this.maxThreads, entries.size()/100+1),
>   replicationSinkMgr.getSinks().size());
> ... 
>   for (int i=0; i entryLists.add(new ArrayList(entries.size()/n+1)); <-- 
> batch size
>   }
> ...
> for (int i=0; i  .
> // RuntimeExceptions encountered here bubble up and are handled 
> in ReplicationSource
> pool.submit(createReplicator(entryLists.get(i), i));  <-- 
> concurrency 
> futures++;
>   }
> }
> maxThreads is fixed & configurable and since we are taking min of the three 
> values n gets decided based replicationSinkMgr.getSinks().size() when we have 
> enough edits to replicate
> replicationSinkMgr.getSinks().size() is decided based on 
> int numSinks = (int) Math.ceil(slaveAddresses.size() * ratio);
> where ratio is this.ratio = conf.getFloat("replication.source.ratio", 
> DEFAULT_REPLICATION_SOURCE_RATIO);
> Currently DEFAULT_REPLICATION_SOURCE_RATIO is set to 10% so for small 
> clusters of size 10-20 RegionServers  the value we get for numSinks and hence 
> n is very small like 1 or 2. This substantially reduces the pool concurrency 
> used for shipping wal edits in parallel effectively slowing down replication 
> for small clusters and causing lot of lag accumulation in AgeOfLastShipped. 
> Sometimes it takes tens of hours to clear off the entire replication queue 
> even after the client has finished writing on the source side. 
> We are running tests by varying replication.source.ratio and have seen 
> multi-fold improvement in total replication time (will update the results 
> here). I wanted to propose here that we should increase the default value for 
> replication.source.ratio also so that we have sufficient concurrency even for 
> small clusters. We figured it out after lot of iterations and debugging so 
> probably slightly higher default will save the trouble. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-16499) slow replication for small HBase clusters

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-16499:
--
Fix Version/s: 2.0.0
   2.1.0
   3.0.0
   Status: Patch Available  (was: Open)

> slow replication for small HBase clusters
> -
>
> Key: HBASE-16499
> URL: https://issues.apache.org/jira/browse/HBASE-16499
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: Vikas Vishwakarma
>Assignee: Vikas Vishwakarma
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.0
>
> Attachments: HBASE-16499.patch
>
>
> For small clusters 10-20 nodes we recently observed that replication is 
> progressing very slowly when we do bulk writes and there is lot of lag 
> accumulation on AgeOfLastShipped / SizeOfLogQueue. From the logs we observed 
> that the number of threads used for shipping wal edits in parallel comes from 
> the following equation in HBaseInterClusterReplicationEndpoint
> int n = Math.min(Math.min(this.maxThreads, entries.size()/100+1),
>   replicationSinkMgr.getSinks().size());
> ... 
>   for (int i=0; i entryLists.add(new ArrayList(entries.size()/n+1)); <-- 
> batch size
>   }
> ...
> for (int i=0; i  .
> // RuntimeExceptions encountered here bubble up and are handled 
> in ReplicationSource
> pool.submit(createReplicator(entryLists.get(i), i));  <-- 
> concurrency 
> futures++;
>   }
> }
> maxThreads is fixed & configurable and since we are taking min of the three 
> values n gets decided based replicationSinkMgr.getSinks().size() when we have 
> enough edits to replicate
> replicationSinkMgr.getSinks().size() is decided based on 
> int numSinks = (int) Math.ceil(slaveAddresses.size() * ratio);
> where ratio is this.ratio = conf.getFloat("replication.source.ratio", 
> DEFAULT_REPLICATION_SOURCE_RATIO);
> Currently DEFAULT_REPLICATION_SOURCE_RATIO is set to 10% so for small 
> clusters of size 10-20 RegionServers  the value we get for numSinks and hence 
> n is very small like 1 or 2. This substantially reduces the pool concurrency 
> used for shipping wal edits in parallel effectively slowing down replication 
> for small clusters and causing lot of lag accumulation in AgeOfLastShipped. 
> Sometimes it takes tens of hours to clear off the entire replication queue 
> even after the client has finished writing on the source side. 
> We are running tests by varying replication.source.ratio and have seen 
> multi-fold improvement in total replication time (will update the results 
> here). I wanted to propose here that we should increase the default value for 
> replication.source.ratio also so that we have sufficient concurrency even for 
> small clusters. We figured it out after lot of iterations and debugging so 
> probably slightly higher default will save the trouble. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-16499) slow replication for small HBase clusters

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-16499:
--
Attachment: HBASE-16499.patch

> slow replication for small HBase clusters
> -
>
> Key: HBASE-16499
> URL: https://issues.apache.org/jira/browse/HBASE-16499
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: Vikas Vishwakarma
>Assignee: Vikas Vishwakarma
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.0
>
> Attachments: HBASE-16499.patch
>
>
> For small clusters 10-20 nodes we recently observed that replication is 
> progressing very slowly when we do bulk writes and there is lot of lag 
> accumulation on AgeOfLastShipped / SizeOfLogQueue. From the logs we observed 
> that the number of threads used for shipping wal edits in parallel comes from 
> the following equation in HBaseInterClusterReplicationEndpoint
> int n = Math.min(Math.min(this.maxThreads, entries.size()/100+1),
>   replicationSinkMgr.getSinks().size());
> ... 
>   for (int i=0; i entryLists.add(new ArrayList(entries.size()/n+1)); <-- 
> batch size
>   }
> ...
> for (int i=0; i  .
> // RuntimeExceptions encountered here bubble up and are handled 
> in ReplicationSource
> pool.submit(createReplicator(entryLists.get(i), i));  <-- 
> concurrency 
> futures++;
>   }
> }
> maxThreads is fixed & configurable and since we are taking min of the three 
> values n gets decided based replicationSinkMgr.getSinks().size() when we have 
> enough edits to replicate
> replicationSinkMgr.getSinks().size() is decided based on 
> int numSinks = (int) Math.ceil(slaveAddresses.size() * ratio);
> where ratio is this.ratio = conf.getFloat("replication.source.ratio", 
> DEFAULT_REPLICATION_SOURCE_RATIO);
> Currently DEFAULT_REPLICATION_SOURCE_RATIO is set to 10% so for small 
> clusters of size 10-20 RegionServers  the value we get for numSinks and hence 
> n is very small like 1 or 2. This substantially reduces the pool concurrency 
> used for shipping wal edits in parallel effectively slowing down replication 
> for small clusters and causing lot of lag accumulation in AgeOfLastShipped. 
> Sometimes it takes tens of hours to clear off the entire replication queue 
> even after the client has finished writing on the source side. 
> We are running tests by varying replication.source.ratio and have seen 
> multi-fold improvement in total replication time (will update the results 
> here). I wanted to propose here that we should increase the default value for 
> replication.source.ratio also so that we have sufficient concurrency even for 
> small clusters. We figured it out after lot of iterations and debugging so 
> probably slightly higher default will save the trouble. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-16499) slow replication for small HBase clusters

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418444#comment-16418444
 ] 

Ashish Singhi commented on HBASE-16499:
---

We use replication for our disaster recovery solution. Where we setup two 
clusters, one is Active and the other is Standby(Read Only). With the default 
{{replication.source.ratio}} (0.1), the data replication from active to standby 
is found to be very slow and resource utilization of standby cluster was also 
not good, so we changed it to 0.5 and found that the replication was happening 
at a very good rate.

I don't have numbers with me to present, but 0.1 seems to be very less. I 
propose to change the default to 0.5 in branch-2.0+ versions. 
 [~stack], what do you say, shall I go ahead and change this ? Thanks.

> slow replication for small HBase clusters
> -
>
> Key: HBASE-16499
> URL: https://issues.apache.org/jira/browse/HBASE-16499
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: Vikas Vishwakarma
>Assignee: Vikas Vishwakarma
>Priority: Major
>
> For small clusters 10-20 nodes we recently observed that replication is 
> progressing very slowly when we do bulk writes and there is lot of lag 
> accumulation on AgeOfLastShipped / SizeOfLogQueue. From the logs we observed 
> that the number of threads used for shipping wal edits in parallel comes from 
> the following equation in HBaseInterClusterReplicationEndpoint
> int n = Math.min(Math.min(this.maxThreads, entries.size()/100+1),
>   replicationSinkMgr.getSinks().size());
> ... 
>   for (int i=0; i entryLists.add(new ArrayList(entries.size()/n+1)); <-- 
> batch size
>   }
> ...
> for (int i=0; i  .
> // RuntimeExceptions encountered here bubble up and are handled 
> in ReplicationSource
> pool.submit(createReplicator(entryLists.get(i), i));  <-- 
> concurrency 
> futures++;
>   }
> }
> maxThreads is fixed & configurable and since we are taking min of the three 
> values n gets decided based replicationSinkMgr.getSinks().size() when we have 
> enough edits to replicate
> replicationSinkMgr.getSinks().size() is decided based on 
> int numSinks = (int) Math.ceil(slaveAddresses.size() * ratio);
> where ratio is this.ratio = conf.getFloat("replication.source.ratio", 
> DEFAULT_REPLICATION_SOURCE_RATIO);
> Currently DEFAULT_REPLICATION_SOURCE_RATIO is set to 10% so for small 
> clusters of size 10-20 RegionServers  the value we get for numSinks and hence 
> n is very small like 1 or 2. This substantially reduces the pool concurrency 
> used for shipping wal edits in parallel effectively slowing down replication 
> for small clusters and causing lot of lag accumulation in AgeOfLastShipped. 
> Sometimes it takes tens of hours to clear off the entire replication queue 
> even after the client has finished writing on the source side. 
> We are running tests by varying replication.source.ratio and have seen 
> multi-fold improvement in total replication time (will update the results 
> here). I wanted to propose here that we should increase the default value for 
> replication.source.ratio also so that we have sufficient concurrency even for 
> small clusters. We figured it out after lot of iterations and debugging so 
> probably slightly higher default will save the trouble. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-15291) FileSystem not closed in secure bulkLoad

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16417260#comment-16417260
 ] 

Ashish Singhi commented on HBASE-15291:
---

I have a updated patch(v2) to address your comments. Please take a look.

Thanks

> FileSystem not closed in secure bulkLoad
> 
>
> Key: HBASE-15291
> URL: https://issues.apache.org/jira/browse/HBASE-15291
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.0.2, 0.98.16.1
>Reporter: Yong Zhang
>Assignee: Ashish Singhi
>Priority: Major
> Attachments: HBASE-15291-revert-master.patch, HBASE-15291.001.patch, 
> HBASE-15291.002.patch, HBASE-15291.003.patch, HBASE-15291.004.patch, 
> HBASE-15291.addendum, HBASE-15291.patch, HBASE-15291.v1.patch, 
> HBASE-15291.v2.patch, patch
>
>
> FileSystem not closed in secure bulkLoad after bulkLoad  finish, it will 
> cause memory used more and more if too many bulkLoad .



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HBASE-15291) FileSystem not closed in secure bulkLoad

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-15291:
--
Attachment: HBASE-15291.v2.patch

> FileSystem not closed in secure bulkLoad
> 
>
> Key: HBASE-15291
> URL: https://issues.apache.org/jira/browse/HBASE-15291
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.0.2, 0.98.16.1
>Reporter: Yong Zhang
>Assignee: Ashish Singhi
>Priority: Major
> Attachments: HBASE-15291-revert-master.patch, HBASE-15291.001.patch, 
> HBASE-15291.002.patch, HBASE-15291.003.patch, HBASE-15291.004.patch, 
> HBASE-15291.addendum, HBASE-15291.patch, HBASE-15291.v1.patch, 
> HBASE-15291.v2.patch, patch
>
>
> FileSystem not closed in secure bulkLoad after bulkLoad  finish, it will 
> cause memory used more and more if too many bulkLoad .



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-15291) FileSystem not closed in secure bulkLoad

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16417241#comment-16417241
 ] 

Ashish Singhi commented on HBASE-15291:
---

It does throw IOE.. My IDE was not showing it, so just checked the code [here 
|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java#L514]and
 find out now.. Let me make a patch for it.

Sorry for the inconvenience caused.

> FileSystem not closed in secure bulkLoad
> 
>
> Key: HBASE-15291
> URL: https://issues.apache.org/jira/browse/HBASE-15291
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.0.2, 0.98.16.1
>Reporter: Yong Zhang
>Assignee: Ashish Singhi
>Priority: Major
> Attachments: HBASE-15291-revert-master.patch, HBASE-15291.001.patch, 
> HBASE-15291.002.patch, HBASE-15291.003.patch, HBASE-15291.004.patch, 
> HBASE-15291.addendum, HBASE-15291.patch, HBASE-15291.v1.patch, patch
>
>
> FileSystem not closed in secure bulkLoad after bulkLoad  finish, it will 
> cause memory used more and more if too many bulkLoad .



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)