[jira] [Commented] (HBASE-14925) Develop HBase shell command/tool to list table's region info through command line

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989771#comment-15989771
 ] 

Ashish Singhi commented on HBASE-14925:
---

Thanks for taking a look.
If [~karanmehta93], doesn't show up for the above comments by Monday, then its 
my responsibility to put up an addendum to address the comments pointed out by 
[~enis] and release note as commented by [~busbey].

> Develop HBase shell command/tool to list table's region info through command 
> line
> -
>
> Key: HBASE-14925
> URL: https://issues.apache.org/jira/browse/HBASE-14925
> Project: HBase
>  Issue Type: Improvement
>  Components: shell
>Reporter: Romil Choksi
>Assignee: Karan Mehta
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-14925.002.patch, HBASE-14925.003.patch, 
> HBASE-14925.patch
>
>
> I am going through the hbase shell commands to see if there is anything I can 
> use to get all the regions info just for a particular table. I don’t see any 
> such command that provides me that information.
> It would be better to have a command that provides region info, start key, 
> end key etc taking a table name as the input parameter. This is available 
> through HBase UI on clicking on a particular table's link
> A tool/shell command to get a list of regions for a table or all tables in a 
> tabular structured output (that is machine readable)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (HBASE-14925) Develop HBase shell command/tool to list table's region info through command line

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989771#comment-15989771
 ] 

Ashish Singhi edited comment on HBASE-14925 at 4/29/17 5:14 AM:


Thanks for taking a look.
If in case [~karanmehta93] doesn't show up for the above comments by Monday, 
then its my responsibility to put up an addendum to address the comments 
pointed out by [~enis] and release note as commented by [~busbey].


was (Author: ashish singhi):
Thanks for taking a look.
If [~karanmehta93], doesn't show up for the above comments by Monday, then its 
my responsibility to put up an addendum to address the comments pointed out by 
[~enis] and release note as commented by [~busbey].

> Develop HBase shell command/tool to list table's region info through command 
> line
> -
>
> Key: HBASE-14925
> URL: https://issues.apache.org/jira/browse/HBASE-14925
> Project: HBase
>  Issue Type: Improvement
>  Components: shell
>Reporter: Romil Choksi
>Assignee: Karan Mehta
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-14925.002.patch, HBASE-14925.003.patch, 
> HBASE-14925.patch
>
>
> I am going through the hbase shell commands to see if there is anything I can 
> use to get all the regions info just for a particular table. I don’t see any 
> such command that provides me that information.
> It would be better to have a command that provides region info, start key, 
> end key etc taking a table name as the input parameter. This is available 
> through HBase UI on clicking on a particular table's link
> A tool/shell command to get a list of regions for a table or all tables in a 
> tabular structured output (that is machine readable)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17973) Create shell command to identify regions with poor locality

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15988729#comment-15988729
 ] 

Ashish Singhi commented on HBASE-17973:
---

You want a dedicated command for that ? if not we can make it part of 
{{list_regions}} command output added as part of HBASE-14925 ?

> Create shell command to identify regions with poor locality
> ---
>
> Key: HBASE-17973
> URL: https://issues.apache.org/jira/browse/HBASE-17973
> Project: HBase
>  Issue Type: Improvement
>  Components: shell
>Reporter: Josh Elser
>Assignee: Josh Elser
>
> The data locality of regions often plays a large role in the efficiency of 
> HBase. Compactions are also expensive to execute, especially on very large 
> tables. The balancer can do a good job trying to maintain locality (when 
> tuned properly), but it is not perfect.
> This creates a less-than-desirable situation where it's a costly operation to 
> take a cluster with spotty poor locality (e.g. a small percentage of 
> regionservers with poor locality).
> We already have this information available via the {{ClusterStatus}} proto. 
> We can easily write a shell command that can present regions which are 
> lacking a certain percentage of locality.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HBASE-14925) Develop HBase shell command/tool to list table's region info through command line

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-14925:
--
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Thanks for the contribution, Karan.
I have committed this to master and branch-1.

> Develop HBase shell command/tool to list table's region info through command 
> line
> -
>
> Key: HBASE-14925
> URL: https://issues.apache.org/jira/browse/HBASE-14925
> Project: HBase
>  Issue Type: Improvement
>  Components: shell
>Reporter: Romil Choksi
>Assignee: Karan Mehta
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-14925.002.patch, HBASE-14925.003.patch, 
> HBASE-14925.patch
>
>
> I am going through the hbase shell commands to see if there is anything I can 
> use to get all the regions info just for a particular table. I don’t see any 
> such command that provides me that information.
> It would be better to have a command that provides region info, start key, 
> end key etc taking a table name as the input parameter. This is available 
> through HBase UI on clicking on a particular table's link
> A tool/shell command to get a list of regions for a table or all tables in a 
> tabular structured output (that is machine readable)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HBASE-14925) Develop HBase shell command/tool to list table's region info through command line

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-14925:
--
Fix Version/s: 1.4.0
   2.0.0

> Develop HBase shell command/tool to list table's region info through command 
> line
> -
>
> Key: HBASE-14925
> URL: https://issues.apache.org/jira/browse/HBASE-14925
> Project: HBase
>  Issue Type: Improvement
>  Components: shell
>Reporter: Romil Choksi
>Assignee: Karan Mehta
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-14925.002.patch, HBASE-14925.003.patch, 
> HBASE-14925.patch
>
>
> I am going through the hbase shell commands to see if there is anything I can 
> use to get all the regions info just for a particular table. I don’t see any 
> such command that provides me that information.
> It would be better to have a command that provides region info, start key, 
> end key etc taking a table name as the input parameter. This is available 
> through HBase UI on clicking on a particular table's link
> A tool/shell command to get a list of regions for a table or all tables in a 
> tabular structured output (that is machine readable)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-14925) Develop HBase shell command/tool to list table's region info through command line

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985982#comment-15985982
 ] 

Ashish Singhi commented on HBASE-14925:
---

Changes are fine. +1
I will commit this tomorrow unless there is any objections. I hope the patch 
applies to branch-1 also.

> Develop HBase shell command/tool to list table's region info through command 
> line
> -
>
> Key: HBASE-14925
> URL: https://issues.apache.org/jira/browse/HBASE-14925
> Project: HBase
>  Issue Type: Improvement
>  Components: shell
>Reporter: Romil Choksi
>Assignee: Karan Mehta
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-14925.002.patch, HBASE-14925.003.patch, 
> HBASE-14925.patch
>
>
> I am going through the hbase shell commands to see if there is anything I can 
> use to get all the regions info just for a particular table. I don’t see any 
> such command that provides me that information.
> It would be better to have a command that provides region info, start key, 
> end key etc taking a table name as the input parameter. This is available 
> through HBase UI on clicking on a particular table's link
> A tool/shell command to get a list of regions for a table or all tables in a 
> tabular structured output (that is machine readable)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-13288) Fix naming of parameter in Delete constructor

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-13288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15976514#comment-15976514
 ] 

Ashish Singhi commented on HBASE-13288:
---

Hi [~chia7712], I am away from my system for few days. Can you help committing 
it ?

> Fix naming of parameter in Delete constructor
> -
>
> Key: HBASE-13288
> URL: https://issues.apache.org/jira/browse/HBASE-13288
> Project: HBase
>  Issue Type: Bug
>  Components: API
>Affects Versions: 1.0.0
>Reporter: Lars George
>Assignee: Ashish Singhi
>Priority: Trivial
>  Labels: beginner
> Attachments: HBASE-13288.patch
>
>
> We have these two variants:
> {code}
> Delete(byte[] row, long timestamp)
> Delete(final byte[] rowArray, final int rowOffset, final int rowLength, long 
> ts)
> {code}
> Both should use {{timestamp}} as the parameter name, not this mix.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-14925) Develop HBase shell command/tool to list table's region info through command line

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958305#comment-15958305
 ] 

Ashish Singhi commented on HBASE-14925:
---

bq. This is to determine the time taken for the command to execute, the 
start_time variable is initialized before the call to this function.
I could not find the start_time variable in the patch and also end_time 
variable value not being used anywhere in the patch. What am I missing ? Can 
you point me ?

> Develop HBase shell command/tool to list table's region info through command 
> line
> -
>
> Key: HBASE-14925
> URL: https://issues.apache.org/jira/browse/HBASE-14925
> Project: HBase
>  Issue Type: Improvement
>  Components: shell
>Reporter: Romil Choksi
>Assignee: Karan Mehta
> Attachments: HBASE-14925.002.patch, HBASE-14925.003.patch, 
> HBASE-14925.patch
>
>
> I am going through the hbase shell commands to see if there is anything I can 
> use to get all the regions info just for a particular table. I don’t see any 
> such command that provides me that information.
> It would be better to have a command that provides region info, start key, 
> end key etc taking a table name as the input parameter. This is available 
> through HBase UI on clicking on a particular table's link
> A tool/shell command to get a list of regions for a table or all tables in a 
> tabular structured output (that is machine readable)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-14925) Develop HBase shell command/tool to list table's region info through command line

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956548#comment-15956548
 ] 

Ashish Singhi commented on HBASE-14925:
---

Patch overall looks fine to me.
Where are we using this ?
{code}
 @end_time = Time.now
{code}

Can we improve the output format, the alignment of header and values ? It is 
little difficult to understand the output.
{noformat}
hbase(main):002:0> list_regions 't1'
 SERVER_NAME REGION_NAME START_KEY END_KEY SIZE REQUESTS
 host1,16045,1491380558177 t1,,1491380624003.2719831ece2827e7afa62c6b022f155c.  
10 0 0
 host1,16045,1491380558177 
t1,10,1491380624003.dbd617c150f536739ac4b145807c013b. 10 20 0 0
 host1,16045,1491380558177 
t1,20,1491380624003.9eea1596bf68009bf81137ba4c4f7bc6. 20 30 0 0
 host1,16045,1491380558177 
t1,30,1491380624003.4fa3232d2a35bcb13b9327fc4760. 30 40 0 0
 host1,16045,1491380558177 
t1,40,1491380624003.94cba65531a9ea8784e924266583d1d1. 40  0 0
5 row(s)
{noformat}

> Develop HBase shell command/tool to list table's region info through command 
> line
> -
>
> Key: HBASE-14925
> URL: https://issues.apache.org/jira/browse/HBASE-14925
> Project: HBase
>  Issue Type: Improvement
>  Components: shell
>Reporter: Romil Choksi
>Assignee: Karan Mehta
> Attachments: HBASE-14925.002.patch, HBASE-14925.patch
>
>
> I am going through the hbase shell commands to see if there is anything I can 
> use to get all the regions info just for a particular table. I don’t see any 
> such command that provides me that information.
> It would be better to have a command that provides region info, start key, 
> end key etc taking a table name as the input parameter. This is available 
> through HBase UI on clicking on a particular table's link
> A tool/shell command to get a list of regions for a table or all tables in a 
> tabular structured output (that is machine readable)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-14925) Develop HBase shell command/tool to list table's region info through command line

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954751#comment-15954751
 ] 

Ashish Singhi commented on HBASE-14925:
---

If a user doesn't give region_server_name then we will not list any regions of 
the table as per the above code, I think better would be to list all the 
regions in this case with server name against each region like your previous 
patch. If a user gives the region_server_name then we can do this way.
I think you will retain the size and requests count of a region in the output 
same as your previous patch.

> Develop HBase shell command/tool to list table's region info through command 
> line
> -
>
> Key: HBASE-14925
> URL: https://issues.apache.org/jira/browse/HBASE-14925
> Project: HBase
>  Issue Type: Improvement
>  Components: shell
>Reporter: Romil Choksi
>Assignee: Karan Mehta
> Attachments: HBASE-14925.patch
>
>
> I am going through the hbase shell commands to see if there is anything I can 
> use to get all the regions info just for a particular table. I don’t see any 
> such command that provides me that information.
> It would be better to have a command that provides region info, start key, 
> end key etc taking a table name as the input parameter. This is available 
> through HBase UI on clicking on a particular table's link
> A tool/shell command to get a list of regions for a table or all tables in a 
> tabular structured output (that is machine readable)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-9393:
-
Attachment: HBASE-9393.v16.patch

Attaching the same patch again to trigger the QA.

> Hbase does not closing a closed socket resulting in many CLOSE_WAIT 
> 
>
> Key: HBASE-9393
> URL: https://issues.apache.org/jira/browse/HBASE-9393
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.94.2, 0.98.0, 1.0.1.1, 1.1.2
> Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, 
> 7279 regions
>Reporter: Avi Zrachya
>Assignee: Ashish Singhi
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-9393.patch, HBASE-9393.v10.patch, 
> HBASE-9393.v11.patch, HBASE-9393.v12.patch, HBASE-9393.v13.patch, 
> HBASE-9393.v14.patch, HBASE-9393.v15.patch, HBASE-9393.v15.patch, 
> HBASE-9393.v16.patch, HBASE-9393.v16.patch, HBASE-9393.v1.patch, 
> HBASE-9393.v2.patch, HBASE-9393.v3.patch, HBASE-9393.v4.patch, 
> HBASE-9393.v5.patch, HBASE-9393.v5.patch, HBASE-9393.v5.patch, 
> HBASE-9393.v6.patch, HBASE-9393.v6.patch, HBASE-9393.v6.patch, 
> HBASE-9393.v7.patch, HBASE-9393.v8.patch, HBASE-9393.v9.patch
>
>
> HBase dose not close a dead connection with the datanode.
> This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect 
> to the datanode because too many mapped sockets from one host to another on 
> the same port.
> The example below is with low CLOSE_WAIT count because we had to restart 
> hbase to solve the porblem, later in time it will incease to 60-100K sockets 
> on CLOSE_WAIT
> [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l
> 13156
> [root@hd2-region3 ~]# ps -ef |grep 21592
> root 17255 17219  0 12:26 pts/000:00:00 grep 21592
> hbase21592 1 17 Aug29 ?03:29:06 
> /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m 
> -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode 
> -Dhbase.log.dir=/var/log/hbase 
> -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ...



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-9393:
-
Status: Patch Available  (was: Open)

> Hbase does not closing a closed socket resulting in many CLOSE_WAIT 
> 
>
> Key: HBASE-9393
> URL: https://issues.apache.org/jira/browse/HBASE-9393
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.1.2, 1.0.1.1, 0.98.0, 0.94.2
> Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, 
> 7279 regions
>Reporter: Avi Zrachya
>Assignee: Ashish Singhi
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-9393.patch, HBASE-9393.v10.patch, 
> HBASE-9393.v11.patch, HBASE-9393.v12.patch, HBASE-9393.v13.patch, 
> HBASE-9393.v14.patch, HBASE-9393.v15.patch, HBASE-9393.v15.patch, 
> HBASE-9393.v16.patch, HBASE-9393.v16.patch, HBASE-9393.v1.patch, 
> HBASE-9393.v2.patch, HBASE-9393.v3.patch, HBASE-9393.v4.patch, 
> HBASE-9393.v5.patch, HBASE-9393.v5.patch, HBASE-9393.v5.patch, 
> HBASE-9393.v6.patch, HBASE-9393.v6.patch, HBASE-9393.v6.patch, 
> HBASE-9393.v7.patch, HBASE-9393.v8.patch, HBASE-9393.v9.patch
>
>
> HBase dose not close a dead connection with the datanode.
> This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect 
> to the datanode because too many mapped sockets from one host to another on 
> the same port.
> The example below is with low CLOSE_WAIT count because we had to restart 
> hbase to solve the porblem, later in time it will incease to 60-100K sockets 
> on CLOSE_WAIT
> [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l
> 13156
> [root@hd2-region3 ~]# ps -ef |grep 21592
> root 17255 17219  0 12:26 pts/000:00:00 grep 21592
> hbase21592 1 17 Aug29 ?03:29:06 
> /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m 
> -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode 
> -Dhbase.log.dir=/var/log/hbase 
> -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ...



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-9393:
-
Status: Open  (was: Patch Available)

> Hbase does not closing a closed socket resulting in many CLOSE_WAIT 
> 
>
> Key: HBASE-9393
> URL: https://issues.apache.org/jira/browse/HBASE-9393
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.1.2, 1.0.1.1, 0.98.0, 0.94.2
> Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, 
> 7279 regions
>Reporter: Avi Zrachya
>Assignee: Ashish Singhi
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-9393.patch, HBASE-9393.v10.patch, 
> HBASE-9393.v11.patch, HBASE-9393.v12.patch, HBASE-9393.v13.patch, 
> HBASE-9393.v14.patch, HBASE-9393.v15.patch, HBASE-9393.v15.patch, 
> HBASE-9393.v16.patch, HBASE-9393.v1.patch, HBASE-9393.v2.patch, 
> HBASE-9393.v3.patch, HBASE-9393.v4.patch, HBASE-9393.v5.patch, 
> HBASE-9393.v5.patch, HBASE-9393.v5.patch, HBASE-9393.v6.patch, 
> HBASE-9393.v6.patch, HBASE-9393.v6.patch, HBASE-9393.v7.patch, 
> HBASE-9393.v8.patch, HBASE-9393.v9.patch
>
>
> HBase dose not close a dead connection with the datanode.
> This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect 
> to the datanode because too many mapped sockets from one host to another on 
> the same port.
> The example below is with low CLOSE_WAIT count because we had to restart 
> hbase to solve the porblem, later in time it will incease to 60-100K sockets 
> on CLOSE_WAIT
> [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l
> 13156
> [root@hd2-region3 ~]# ps -ef |grep 21592
> root 17255 17219  0 12:26 pts/000:00:00 grep 21592
> hbase21592 1 17 Aug29 ?03:29:06 
> /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m 
> -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode 
> -Dhbase.log.dir=/var/log/hbase 
> -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ...



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15950785#comment-15950785
 ] 

Ashish Singhi commented on HBASE-9393:
--

Attached patch v16 for further reviews. Once the master branch patch is 
accepted I will attach patches for other branches.
Response to [~busbey] comments,
{quote}
The addition of the unbuffer call here means that we need to update the 
javadocs for HFile.createReader(FileSystem, Path, FSDataInputStreamWrapper, 
long, CacheConfig, Configuration) and HFile.createReaderFromStream(Path, 
FSDataInputStream, long, CacheConfig, Configuration) to note that callers need 
to ensure no other threads have access to the passed FSDISW instance.
We should also ensure that existing calls to those methods are safely passing 
the FSDISW instance.
{quote}
No need, the new reference of FSDISW is just created and passed from this 
methods.

{quote}
Just want to make sure I'm following the rationale correctly here. This won't 
actually take care of unbuffering if the lock is held e.g. for reading. I think 
this is fine, since it implies someone else is still using the stream and 
presumably they will also attempt to unbuffer when they are done.
{quote}
Yes.

> Hbase does not closing a closed socket resulting in many CLOSE_WAIT 
> 
>
> Key: HBASE-9393
> URL: https://issues.apache.org/jira/browse/HBASE-9393
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.94.2, 0.98.0, 1.0.1.1, 1.1.2
> Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, 
> 7279 regions
>Reporter: Avi Zrachya
>Assignee: Ashish Singhi
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-9393.patch, HBASE-9393.v10.patch, 
> HBASE-9393.v11.patch, HBASE-9393.v12.patch, HBASE-9393.v13.patch, 
> HBASE-9393.v14.patch, HBASE-9393.v15.patch, HBASE-9393.v15.patch, 
> HBASE-9393.v16.patch, HBASE-9393.v1.patch, HBASE-9393.v2.patch, 
> HBASE-9393.v3.patch, HBASE-9393.v4.patch, HBASE-9393.v5.patch, 
> HBASE-9393.v5.patch, HBASE-9393.v5.patch, HBASE-9393.v6.patch, 
> HBASE-9393.v6.patch, HBASE-9393.v6.patch, HBASE-9393.v7.patch, 
> HBASE-9393.v8.patch, HBASE-9393.v9.patch
>
>
> HBase dose not close a dead connection with the datanode.
> This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect 
> to the datanode because too many mapped sockets from one host to another on 
> the same port.
> The example below is with low CLOSE_WAIT count because we had to restart 
> hbase to solve the porblem, later in time it will incease to 60-100K sockets 
> on CLOSE_WAIT
> [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l
> 13156
> [root@hd2-region3 ~]# ps -ef |grep 21592
> root 17255 17219  0 12:26 pts/000:00:00 grep 21592
> hbase21592 1 17 Aug29 ?03:29:06 
> /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m 
> -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode 
> -Dhbase.log.dir=/var/log/hbase 
> -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ...



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-9393:
-
Attachment: HBASE-9393.v16.patch

> Hbase does not closing a closed socket resulting in many CLOSE_WAIT 
> 
>
> Key: HBASE-9393
> URL: https://issues.apache.org/jira/browse/HBASE-9393
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.94.2, 0.98.0, 1.0.1.1, 1.1.2
> Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, 
> 7279 regions
>Reporter: Avi Zrachya
>Assignee: Ashish Singhi
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-9393.patch, HBASE-9393.v10.patch, 
> HBASE-9393.v11.patch, HBASE-9393.v12.patch, HBASE-9393.v13.patch, 
> HBASE-9393.v14.patch, HBASE-9393.v15.patch, HBASE-9393.v15.patch, 
> HBASE-9393.v16.patch, HBASE-9393.v1.patch, HBASE-9393.v2.patch, 
> HBASE-9393.v3.patch, HBASE-9393.v4.patch, HBASE-9393.v5.patch, 
> HBASE-9393.v5.patch, HBASE-9393.v5.patch, HBASE-9393.v6.patch, 
> HBASE-9393.v6.patch, HBASE-9393.v6.patch, HBASE-9393.v7.patch, 
> HBASE-9393.v8.patch, HBASE-9393.v9.patch
>
>
> HBase dose not close a dead connection with the datanode.
> This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect 
> to the datanode because too many mapped sockets from one host to another on 
> the same port.
> The example below is with low CLOSE_WAIT count because we had to restart 
> hbase to solve the porblem, later in time it will incease to 60-100K sockets 
> on CLOSE_WAIT
> [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l
> 13156
> [root@hd2-region3 ~]# ps -ef |grep 21592
> root 17255 17219  0 12:26 pts/000:00:00 grep 21592
> hbase21592 1 17 Aug29 ?03:29:06 
> /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m 
> -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode 
> -Dhbase.log.dir=/var/log/hbase 
> -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ...



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-14925) Develop HBase shell command/tool to list table's region info through command line

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927624#comment-15927624
 ] 

Ashish Singhi commented on HBASE-14925:
---

{code}
+for server_name in cluster_status.getServers()
+  for name, region in 
cluster_status.getLoad(server_name).getRegionsLoad()
+region_name = region.getNameAsString()
+regionStoreFileSize = region.getStorefileSizeMB()
+regionRequests = region.getRequestsCount()
+if region_name.start_with? tgtTable
{code}
Instead of loading all the regions available in the cluster we can get the 
regions of the target table only and print it details. We can get the target 
table regions location from Connection#getRegionLocator#getAllRegionLocations, 
otherwise imagine a cluster having 1M regions, the response of this command 
will be very slow.

Also similar like web UI you can log other information of Regions like start 
key, end key...

> Develop HBase shell command/tool to list table's region info through command 
> line
> -
>
> Key: HBASE-14925
> URL: https://issues.apache.org/jira/browse/HBASE-14925
> Project: HBase
>  Issue Type: Improvement
>  Components: shell
>Reporter: Romil Choksi
>Assignee: Karan Mehta
> Attachments: HBASE-14925.patch
>
>
> I am going through the hbase shell commands to see if there is anything I can 
> use to get all the regions info just for a particular table. I don’t see any 
> such command that provides me that information.
> It would be better to have a command that provides region info, start key, 
> end key etc taking a table name as the input parameter. This is available 
> through HBase UI on clicking on a particular table's link
> A tool/shell command to get a list of regions for a table or all tables in a 
> tabular structured output (that is machine readable)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17790) Mark ReplicationAdmin's peerAdded and listReplicationPeers as Deprecated

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927522#comment-15927522
 ] 

Ashish Singhi commented on HBASE-17790:
---

{code} @deprecated user {@link 
org.apache.hadoop.hbase.client.Admin#listReplicationPeers()} instead {code}
user -> use. 
You can change it on commit. +1

> Mark ReplicationAdmin's peerAdded and listReplicationPeers as Deprecated
> 
>
> Key: HBASE-17790
> URL: https://issues.apache.org/jira/browse/HBASE-17790
> Project: HBase
>  Issue Type: Sub-task
>  Components: Replication
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17790.master.001.patch, 
> HBASE-17790.master.002.patch
>
>
> Now most of public method in ReplicationAdmin have been moved to Admin and 
> marked as Deprecated. And peerAdded and listReplicationPeers method need be 
> marked as Deprecated, too.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17790) Mark ReplicationAdmin's peerAdded and listReplicationPeers as Deprecated

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925975#comment-15925975
 ] 

Ashish Singhi commented on HBASE-17790:
---

Can you add java doc for the user pointing which method to use instead of this 
deprecated method.
Otherwise +1 if Hadoop QA is green.

> Mark ReplicationAdmin's peerAdded and listReplicationPeers as Deprecated
> 
>
> Key: HBASE-17790
> URL: https://issues.apache.org/jira/browse/HBASE-17790
> Project: HBase
>  Issue Type: Sub-task
>  Components: Replication
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17790.master.001.patch
>
>
> Now most of public method in ReplicationAdmin have been moved to Admin and 
> marked as Deprecated. And peerAdded and listReplicationPeers method need be 
> marked as Deprecated, too.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17769) In webUI, master and regionserver address should use rpc-port instead of infoport

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905063#comment-15905063
 ] 

Ashish Singhi commented on HBASE-17769:
---

Can you attach the sceenshots before and after ?

{code}
-String host = masterServerName.getHostname() + ":" + infoPort;
+String host = masterServerName.getHostname() + ":" + port;
 String url = "//" + host + "/master-status";
{code}
The above change of yours will break the URL now. Please check whether on 
clicking the URL it still redirects to master page.

> In webUI, master and regionserver address should use rpc-port instead of 
> infoport
> -
>
> Key: HBASE-17769
> URL: https://issues.apache.org/jira/browse/HBASE-17769
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
> Attachments: HBASE-17769-v1.patch
>
>
> On table.jsp display regionserver address is ip:infoport, it may be more 
> appropriate to use ip:hbase.regionserver.port
> Similarly, the master address displayed on the regionserver page should be 
> ip:hbase.master.port



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16010) Put draining function through Admin API

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905034#comment-15905034
 ] 

Ashish Singhi commented on HBASE-16010:
---

I see this jira is in reopened state but the patch code is available in master 
branch. So what's the final conclusion ? 

> Put draining function through Admin API
> ---
>
> Key: HBASE-16010
> URL: https://issues.apache.org/jira/browse/HBASE-16010
> Project: HBase
>  Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: hbase-16010-v1.patch, hbase-16010-v2.patch, 
> HBASE-16010-v3.patch
>
>
> Currently, there is no Amdin API for draining function. Client has to 
> interact directly with Zookeeper draining node to add and remove draining 
> servers.
> For example, in draining_servers.rb:
> {code}
>   zkw = org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.new(config, 
> "draining_servers", nil)
>   parentZnode = zkw.drainingZNode
>   begin
> for server in servers
>   node = ZKUtil.joinZNode(parentZnode, server)
>   ZKUtil.createAndFailSilent(zkw, node)
> end
>   ensure
> zkw.close()
>   end
> {code}
> This is not good in cases like secure clusters with protected Zookeeper nodes.
> Let's put draining function through Admin API.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17762) Add logging to HBaseAdmin for user initiated tasks

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905002#comment-15905002
 ] 

Ashish Singhi commented on HBASE-17762:
---

We already have audit logs coming from AccessController class. Is that not 
enough ?

> Add logging to HBaseAdmin for user initiated tasks
> --
>
> Key: HBASE-17762
> URL: https://issues.apache.org/jira/browse/HBASE-17762
> Project: HBase
>  Issue Type: Task
>Reporter: churro morales
>Assignee: churro morales
> Fix For: 2.0.0, 1.4.0, 0.98.25
>
> Attachments: HBASE-17762.patch, HBASE-17762.v1.patch
>
>
> Things like auditing a forced major compaction are really useful and right 
> now there is no logging when this is triggered.  Other actions may require 
> logging as well. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17460) enable_table_replication can not perform cyclic replication of a table

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886013#comment-15886013
 ] 

Ashish Singhi commented on HBASE-17460:
---

bq. Also I am not 100% sure why the check needed in 
checkAndSyncTableDescToPeers.
If you are not sure then why add it, we can simply solve it by just comparing 
the HTDs without replication scope.

If we really require that check then that's a different issue and should be 
solved as part of a different jira.

I think we should do the same for master branch also. WDYT [~tedyu] ?

> enable_table_replication can not perform cyclic replication of a table
> --
>
> Key: HBASE-17460
> URL: https://issues.apache.org/jira/browse/HBASE-17460
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: NITIN VERMA
>Assignee: NITIN VERMA
>  Labels: incompatibleChange, replication
> Fix For: 2.0.0
>
> Attachments: 17460-addendum.txt, 17460-addendum.v2.txt, 
> 17460.branch-1.v3.txt, 17460.v5.txt, 
> HBASE-17460-branch-1-v5-plusaddendum-v2.patch, HBASE-17460.patch, 
> HBASE-17460_v2.patch, HBASE-17460_v3.patch, HBASE-17460_v4.patch
>
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>
> The enable_table_replication operation is broken for cyclic replication of 
> HBase table as we compare all the properties of column families (including 
> REPLICATION_SCOPE). 
> Below is exactly what happens:
> 1.  Running "enable_table_replication 'table1'  " opeartion on first cluster 
> will set the REPLICATION_SCOPE of all column families to peer id '1'. This 
> will also create a table on second cluster where REPLICATION_SCOPE is still 
> set to peer id '0'.
> 2. Now when we run "enable_table_replication 'table1'" on second cluster, we 
> compare all the properties of table (including REPLICATION_SCOPE_, which 
> obviously is different now. 
> I am proposing a fix for this issue where we should avoid comparing 
> REPLICATION_SCOPE inside HColumnDescriotor::compareTo() method, especially 
> when replication is not already enabled on the desired table.
> I have made that change and it is working. I will submit the patch soon.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17460) enable_table_replication can not perform cyclic replication of a table

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885795#comment-15885795
 ] 

Ashish Singhi commented on HBASE-17460:
---

ReplicationAdmin#enableTableRep is a public API from a interface audience 
public class, we cannot change the method signature just like that.

Earlier ReplicationAdmin#enableTableRep behavior was, user was allowed to 
enable replication on a CF later also if user didn't do it the first time, but 
with this change user will not be able to do this using this API.
Is it really required to check for localHtd.isReplicationEnabled() and throw 
exception even if its enabled on some CF and not all, what will happen if we do 
not do ?

Sorry for being later here,
The jira was raised for handling cyclic replication in 
ReplicationAdmin#enableTableRep API, for that I think the simple fix would have 
been just to avoid the replication scope check for source and peer HTD. But 
looks like we have introduced a behavior change in a public API :(

> enable_table_replication can not perform cyclic replication of a table
> --
>
> Key: HBASE-17460
> URL: https://issues.apache.org/jira/browse/HBASE-17460
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: NITIN VERMA
>Assignee: NITIN VERMA
>  Labels: incompatibleChange, replication
> Fix For: 2.0.0
>
> Attachments: 17460-addendum.txt, 17460-addendum.v2.txt, 
> 17460.branch-1.v3.txt, 17460.v5.txt, 
> HBASE-17460-branch-1-v5-plusaddendum-v2.patch, HBASE-17460.patch, 
> HBASE-17460_v2.patch, HBASE-17460_v3.patch, HBASE-17460_v4.patch
>
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>
> The enable_table_replication operation is broken for cyclic replication of 
> HBase table as we compare all the properties of column families (including 
> REPLICATION_SCOPE). 
> Below is exactly what happens:
> 1.  Running "enable_table_replication 'table1'  " opeartion on first cluster 
> will set the REPLICATION_SCOPE of all column families to peer id '1'. This 
> will also create a table on second cluster where REPLICATION_SCOPE is still 
> set to peer id '0'.
> 2. Now when we run "enable_table_replication 'table1'" on second cluster, we 
> compare all the properties of table (including REPLICATION_SCOPE_, which 
> obviously is different now. 
> I am proposing a fix for this issue where we should avoid comparing 
> REPLICATION_SCOPE inside HColumnDescriotor::compareTo() method, especially 
> when replication is not already enabled on the desired table.
> I have made that change and it is working. I will submit the patch soon.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-17687) hive on hbase table and phoenix table can't be selected

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881963#comment-15881963
 ] 

Ashish Singhi commented on HBASE-17687:
---

This issue seems to be a vendor specific. Please check with the vendor first.
Thanks

> hive on hbase table and phoenix table can't  be selected
> 
>
> Key: HBASE-17687
> URL: https://issues.apache.org/jira/browse/HBASE-17687
> Project: HBase
>  Issue Type: Improvement
>  Components: hbase
>Affects Versions: 1.0.2
> Environment: hadoop 2.7.2
> hbase 1.0.2
> phoenix 4.4
> hive 1.3
> all above are based on huawei FusionInsight HD(FusionInsight 
> V100R002C60U10SPC001)
>Reporter: yunliangchen
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> First , I created a table on phoenix, as this:
> ---
> DROP TABLE IF EXISTS bidwd_test01 CASCADE;
> CREATE TABLE IF NOT EXISTS bidwd_test01(
>rk VARCHAR,
>c1 integer,
>c2 VARCHAR,
>c3 VARCHAR,
>c4 VARCHAR
>constraint bidwd_test01_pk primary key(rk)
> )
> COMPRESSION='SNAPPY'
> ;
> ---
> And then , I upserted two rows into the table:
> ---
> upsert into bidwd_test01 values('001',1,'zhangsan','20170217','2017-02-17 
> 12:34:22');
> upsert into bidwd_test01 values('002',2,'lisi','20170216','2017-02-16 
> 12:34:22');
> ---
> At last , I scaned the table like this:
> ---
> select * from bidwd_test01;
> ---
> It's OK by now, but, I want to create a hive on hbase table ,that mapping to 
> the phoenix table , the script likes this:
> ---
> USE BIDWD;
> DROP TABLE test01;
> CREATE EXTERNAL TABLE test01
> (
>  rk string,
>  id int,
>  name string,
>  datekey string,
>  time_stamp string
> )
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'  
> WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,0:C1,0:C2,0:C3,0:C4")  
> TBLPROPERTIES ("hbase.table.name" = "BIDWD_TEST01");
> ---
> So,I also try to insert some data into the table,and scan this table:
> ---
> set hive.execution.engine=mr;
> insert into test01 values('003',3,'lisi2','20170215','2017-02-15 12:34:22');
> select * from test01;
> ---
> But,there are some problems like this:
> +++--+-+--+--+
> | test01.rk  | test01.id  | test01.name  | test01.datekey  |  
> test01.time_stamp   |
> +++--+-+--+--+
> | 001| NULL   | zhangsan | 20170217| 2017-02-17 
> 12:34:22  |
> | 002| NULL   | lisi | 20170216| 2017-02-16 
> 12:34:22  |
> | 003| 3  | lisi2| 20170215| 2017-02-15 
> 12:34:22  |
> +++--+-+--+--+
> the column "id" 's value was null,only the last row is ok.
> but,when I scan data in the phoenix ,there are some errors like this:
> Error: ERROR 201 (22000): Illegal data. Expected length of at least 115 
> bytes, but had 31 (state=22000,code=201)
> java.sql.SQLException: ERROR 201 (22000): Illegal data. Expected length of at 
> least 115 bytes, but had 31
>   at 
> org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:389)
>   at 
> org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:145)
>   at 
> org.apache.phoenix.schema.KeyValueSchema.next(KeyValueSchema.java:211)
>   at 
> org.apache.phoenix.expression.ProjectedColumnExpression.evaluate(ProjectedColumnExpression.java:113)
>   at 
> org.apache.phoenix.compile.ExpressionProjector.getValue(ExpressionProjector.java:69)
>   at 
> org.apache.phoenix.jdbc.PhoenixResultSet.getString(PhoenixResultSet.java:591)
>   at sqlline.Rows$Row.(Rows.java:183)
>   at sqlline.BufferedRows.(BufferedRows.java:38)
>   at sqlline.SqlLine.print(SqlLine.java:1546)
>   at sqlline.Commands.execute(Commands.java:833)
>   at sqlline.Commands.sql(Commands.java:732)
>   at sqlline.SqlLine.dispatch(SqlLine.java:702)
>   at sqlline.SqlLine.begin(SqlLine.java:575)
>   at 

[jira] [Commented] (HBASE-16119) Procedure v2 - Reimplement merge

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15825510#comment-15825510
 ] 

Ashish Singhi commented on HBASE-16119:
---

bq. Adds a method to Admin. Is this an incompat change? Allowed but should we 
tick the incompat box?
In master branch we will be supporting only java8 for now, so we can add 
default methods in the interface and avoid the binary incompatibility.
https://docs.oracle.com/javase/tutorial/java/IandI/defaultmethods.html

> Procedure v2 - Reimplement merge
> 
>
> Key: HBASE-16119
> URL: https://issues.apache.org/jira/browse/HBASE-16119
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2, Region Assignment
>Affects Versions: 2.0.0
>Reporter: Matteo Bertozzi
>Assignee: Stephen Yuan Jiang
> Fix For: 2.0.0
>
> Attachments: HBASE-16119.v1-master.patch, HBASE-16119.v2-master.patch
>
>
> use the proc-v2 state machine for merge. also update the logic to have a 
> single meta-writer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15825501#comment-15825501
 ] 

Ashish Singhi commented on HBASE-9393:
--

Will get back to this very soon with patches for all the versions.

> Hbase does not closing a closed socket resulting in many CLOSE_WAIT 
> 
>
> Key: HBASE-9393
> URL: https://issues.apache.org/jira/browse/HBASE-9393
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.94.2, 0.98.0, 1.0.1.1, 1.1.2
> Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, 
> 7279 regions
>Reporter: Avi Zrachya
>Assignee: Ashish Singhi
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-9393.patch, HBASE-9393.v10.patch, 
> HBASE-9393.v11.patch, HBASE-9393.v12.patch, HBASE-9393.v13.patch, 
> HBASE-9393.v14.patch, HBASE-9393.v15.patch, HBASE-9393.v15.patch, 
> HBASE-9393.v1.patch, HBASE-9393.v2.patch, HBASE-9393.v3.patch, 
> HBASE-9393.v4.patch, HBASE-9393.v5.patch, HBASE-9393.v5.patch, 
> HBASE-9393.v5.patch, HBASE-9393.v6.patch, HBASE-9393.v6.patch, 
> HBASE-9393.v6.patch, HBASE-9393.v7.patch, HBASE-9393.v8.patch, 
> HBASE-9393.v9.patch
>
>
> HBase dose not close a dead connection with the datanode.
> This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect 
> to the datanode because too many mapped sockets from one host to another on 
> the same port.
> The example below is with low CLOSE_WAIT count because we had to restart 
> hbase to solve the porblem, later in time it will incease to 60-100K sockets 
> on CLOSE_WAIT
> [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l
> 13156
> [root@hd2-region3 ~]# ps -ef |grep 21592
> root 17255 17219  0 12:26 pts/000:00:00 grep 21592
> hbase21592 1 17 Aug29 ?03:29:06 
> /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m 
> -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode 
> -Dhbase.log.dir=/var/log/hbase 
> -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17468) unread messages in TCP connections - possible connection leak

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15825498#comment-15825498
 ] 

Ashish Singhi commented on HBASE-17468:
---

We have been using that solution internally for around a year now.
bq. Would it work with 1.2.0?
Yes. We have back ported it to 1.0.2 version FYI

bq. but would it cause any side effects?
We didn't see any.
Note: HBASE-9393 patch will work only if your using one of the 2.8.0, 2.7.3, 
2.6.5, 3.0.0-alpha1 Hadoop version.

> unread messages in TCP connections - possible connection leak
> -
>
> Key: HBASE-17468
> URL: https://issues.apache.org/jira/browse/HBASE-17468
> Project: HBase
>  Issue Type: Bug
>Reporter: Shridhar Sahukar
>Priority: Critical
>
> We are running HBase 1.2.0-cdh5.7.1 (Cloudera distribution).
> On our Hadoop cluster, we are seeing that each HBase region server has large 
> number of TCP connections to all the HDFS data nodes and all these 
> connections have unread data in socket buffers. Some of these connections are 
> also in CLOSE_WAIT or FIN_WAIT1 state while the rest are in ESTABLISHED state.
> Looks like HBase is creating some connections requesting data from HDFS, but 
> its forgetting about those connections before it could read the data. Thus 
> the connections are left lingering around with large data stuck in their 
> receive buffers. Also, it seems HDFS closes these connections after a while, 
> but since there is data in receive buffer the connection is left in 
> CLOSE_WAIT/FIN_WAIT1 states.
> Below is a snapshot from one of the region servers:
> ## Total number of connections to HDFS  (pid of region server is 143722)
> [bda@md-bdadev-42 hbase]$ sudo netstat -anp|grep 143722 | wc -l
> 827
> ## Connections that are not in ESTABLISHED state
> [bda@md-bdadev-42 hbase]$ sudo netstat -anp|grep 143722 | grep -v ESTABLISHED 
> | wc -l
> 344
> ##Snapshot of some of these connections:
> tcp   133887  0 146.1.180.43:48533  146.1.180.40:50010  
> ESTABLISHED 143722/java
> tcp82934  0 146.1.180.43:59647  146.1.180.42:50010  
> ESTABLISHED 143722/java
> tcp0  0 146.1.180.43:50761  146.1.180.27:2181   
> ESTABLISHED 143722/java
> tcp   234084  0 146.1.180.43:58335  146.1.180.42:50010  
> ESTABLISHED 143722/java
> tcp   967667  0 146.1.180.43:56136  146.1.180.68:50010  
> ESTABLISHED 143722/java
> tcp   156037  0 146.1.180.43:59659  146.1.180.42:50010  
> ESTABLISHED 143722/java
> tcp   212488  0 146.1.180.43:56810  146.1.180.48:50010  
> ESTABLISHED 143722/java
> tcp61871  0 146.1.180.43:53593  146.1.180.35:50010  
> ESTABLISHED 143722/java
> tcp   121216  0 146.1.180.43:35324  146.1.180.38:50010  
> ESTABLISHED 143722/java
> tcp1  0 146.1.180.43:32982  146.1.180.42:50010  
> CLOSE_WAIT  143722/java
> tcp82934  0 146.1.180.43:42359  146.1.180.54:50010  
> ESTABLISHED 143722/java
> tcp   159422  0 146.1.180.43:59731  146.1.180.42:50010  
> ESTABLISHED 143722/java
> tcp   134573  0 146.1.180.43:60210  146.1.180.76:50010  
> ESTABLISHED 143722/java
> tcp82934  0 146.1.180.43:59713  146.1.180.42:50010  
> ESTABLISHED 143722/java
> tcp   135765  0 146.1.180.43:44412  146.1.180.29:50010  
> ESTABLISHED 143722/java
> tcp   161655  0 146.1.180.43:43117  146.1.180.42:50010  
> ESTABLISHED 143722/java
> tcp75990  0 146.1.180.43:59729  146.1.180.42:50010  
> ESTABLISHED 143722/java
> tcp78583  0 146.1.180.43:59971  146.1.180.42:50010  
> ESTABLISHED 143722/java
> tcp1  0 146.1.180.43:39893  146.1.180.67:50010  
> CLOSE_WAIT  143722/java
> tcp1  0 146.1.180.43:38834  146.1.180.47:50010  
> CLOSE_WAIT  143722/java
> tcp1  0 146.1.180.43:40707  146.1.180.50:50010  
> CLOSE_WAIT  143722/java
> tcp   106102  0 146.1.180.43:48208  146.1.180.75:50010  
> ESTABLISHED 143722/java
> tcp   332013  0 146.1.180.43:34795  146.1.180.37:50010  
> ESTABLISHED 143722/java
> tcp1  0 146.1.180.43:57644  146.1.180.67:50010  
> CLOSE_WAIT  143722/java
> tcp79119  0 146.1.180.43:54438  146.1.180.70:50010  
> ESTABLISHED 143722/java
> tcp77438  0 146.1.180.43:35259  146.1.180.38:50010  
> ESTABLISHED 143722/java
> tcp1  0 146.1.180.43:57579  146.1.180.41:50010  
> CLOSE_WAIT  143722/java
> tcp   318091  0 146.1.180.43:60124  

[jira] [Commented] (HBASE-17443) Move listReplicated/enableTableRep/disableTableRep methods from ReplicationAdmin to Admin

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15823922#comment-15823922
 ] 

Ashish Singhi commented on HBASE-17443:
---

+1 on v4

> Move listReplicated/enableTableRep/disableTableRep methods from 
> ReplicationAdmin to Admin
> -
>
> Key: HBASE-17443
> URL: https://issues.apache.org/jira/browse/HBASE-17443
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17443-v1.patch, HBASE-17443-v2.patch, 
> HBASE-17443-v2.patch, HBASE-17443-v3.patch, HBASE-17443-v3.patch, 
> HBASE-17443-v4.patch
>
>
> We have moved other replication requests to Admin and mark ReplicationAdmin 
> as Deprecated, so listReplicated/enableTableRep/disableTableRep methods need 
> move to Admin, too.
> Review board: https://reviews.apache.org/r/55534/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17443) Move listReplicated/enableTableRep/disableTableRep methods from ReplicationAdmin to Admin

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15823806#comment-15823806
 ] 

Ashish Singhi commented on HBASE-17443:
---

I have left some minor comments on the RB.
Overall looks good.

> Move listReplicated/enableTableRep/disableTableRep methods from 
> ReplicationAdmin to Admin
> -
>
> Key: HBASE-17443
> URL: https://issues.apache.org/jira/browse/HBASE-17443
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17443-v1.patch, HBASE-17443-v2.patch, 
> HBASE-17443-v2.patch, HBASE-17443-v3.patch, HBASE-17443-v3.patch
>
>
> We have moved other replication requests to Admin and mark ReplicationAdmin 
> as Deprecated, so listReplicated/enableTableRep/disableTableRep methods need 
> move to Admin, too.
> Review board: https://reviews.apache.org/r/55534/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HBASE-17472) Correct the semantic of permission grant

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15823695#comment-15823695
 ] 

Ashish Singhi edited comment on HBASE-17472 at 1/16/17 9:47 AM:


bq. Can we add hint for grant command after we change the behavior ?
Hints can be missed reading and especially when the API is called through java 
code. May be add a new command what [~Apache9] suggested.

Others sound OK to me.


was (Author: ashish singhi):
bq. Can we add hint for grant command after we change the behavior ?
Hints can be missed reading and especially when the API is called through java 
code. May be add a new command like [~Apache9] suggested.

Others sound OK to me.

> Correct the semantic of  permission grant
> -
>
> Key: HBASE-17472
> URL: https://issues.apache.org/jira/browse/HBASE-17472
> Project: HBase
>  Issue Type: Improvement
>  Components: Admin
>Reporter: huzheng
>Assignee: huzheng
>
> Currently, HBase grant operation has following semantic:
> {code}
> hbase(main):019:0> grant 'hbase_tst', 'RW', 'ycsb'
> 0 row(s) in 0.0960 seconds
> hbase(main):020:0> user_permission 'ycsb'
> User 
> Namespace,Table,Family,Qualifier:Permission   
>   
>   
> 
>  hbase_tst   default,ycsb,,: 
> [Permission:actions=READ,WRITE]   
>   
>   
> 1 row(s) in 0.0550 seconds
> hbase(main):021:0> grant 'hbase_tst', 'CA', 'ycsb'
> 0 row(s) in 0.0820 seconds
> hbase(main):022:0> user_permission 'ycsb'
> User 
> Namespace,Table,Family,Qualifier:Permission   
>   
>   
>  hbase_tst   default,ycsb,,: 
> [Permission: actions=CREATE,ADMIN]
>   
>   
> 1 row(s) in 0.0490 seconds
> {code}  
> Later permission will replace previous granted permissions, which confused 
> most of HBase administrator.
> It's seems more reasonable that HBase merge multiple granted permission.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17472) Correct the semantic of permission grant

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15823695#comment-15823695
 ] 

Ashish Singhi commented on HBASE-17472:
---

bq. Can we add hint for grant command after we change the behavior ?
Hints can be missed reading and especially when the API is called through java 
code. May be add a new command like [~Apache9] suggested.

Others sound OK to me.

> Correct the semantic of  permission grant
> -
>
> Key: HBASE-17472
> URL: https://issues.apache.org/jira/browse/HBASE-17472
> Project: HBase
>  Issue Type: Improvement
>  Components: Admin
>Reporter: huzheng
>Assignee: huzheng
>
> Currently, HBase grant operation has following semantic:
> {code}
> hbase(main):019:0> grant 'hbase_tst', 'RW', 'ycsb'
> 0 row(s) in 0.0960 seconds
> hbase(main):020:0> user_permission 'ycsb'
> User 
> Namespace,Table,Family,Qualifier:Permission   
>   
>   
> 
>  hbase_tst   default,ycsb,,: 
> [Permission:actions=READ,WRITE]   
>   
>   
> 1 row(s) in 0.0550 seconds
> hbase(main):021:0> grant 'hbase_tst', 'CA', 'ycsb'
> 0 row(s) in 0.0820 seconds
> hbase(main):022:0> user_permission 'ycsb'
> User 
> Namespace,Table,Family,Qualifier:Permission   
>   
>   
>  hbase_tst   default,ycsb,,: 
> [Permission: actions=CREATE,ADMIN]
>   
>   
> 1 row(s) in 0.0490 seconds
> {code}  
> Later permission will replace previous granted permissions, which confused 
> most of HBase administrator.
> It's seems more reasonable that HBase merge multiple granted permission.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17443) Move listReplicated/enableTableRep/disableTableRep methods from ReplicationAdmin to Admin

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15823493#comment-15823493
 ] 

Ashish Singhi commented on HBASE-17443:
---

I will review it today. Thanks for the ping.

> Move listReplicated/enableTableRep/disableTableRep methods from 
> ReplicationAdmin to Admin
> -
>
> Key: HBASE-17443
> URL: https://issues.apache.org/jira/browse/HBASE-17443
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17443-v1.patch, HBASE-17443-v2.patch, 
> HBASE-17443-v2.patch, HBASE-17443-v3.patch, HBASE-17443-v3.patch
>
>
> We have moved other replication requests to Admin and mark ReplicationAdmin 
> as Deprecated, so listReplicated/enableTableRep/disableTableRep methods need 
> move to Admin, too.
> Review board: https://reviews.apache.org/r/55534/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17472) Correct the semantic of permission grant

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15823484#comment-15823484
 ] 

Ashish Singhi commented on HBASE-17472:
---

This behavior exists in HBase from more than 3 years now. Will it not confuse 
the HBase administrators if we change the behavior now ?

> Correct the semantic of  permission grant
> -
>
> Key: HBASE-17472
> URL: https://issues.apache.org/jira/browse/HBASE-17472
> Project: HBase
>  Issue Type: Improvement
>  Components: Admin
>Reporter: huzheng
>Assignee: huzheng
>
> Currently, HBase grant operation has following semantic:
> {code}
> hbase(main):019:0> grant 'hbase_tst', 'RW', 'ycsb'
> 0 row(s) in 0.0960 seconds
> hbase(main):020:0> user_permission 'ycsb'
> User 
> Namespace,Table,Family,Qualifier:Permission   
>   
>   
> 
>  hbase_tst   default,ycsb,,: 
> [Permission:actions=READ,WRITE]   
>   
>   
> 1 row(s) in 0.0550 seconds
> hbase(main):021:0> grant 'hbase_tst', 'CA', 'ycsb'
> 0 row(s) in 0.0820 seconds
> hbase(main):022:0> user_permission 'ycsb'
> User 
> Namespace,Table,Family,Qualifier:Permission   
>   
>   
>  hbase_tst   default,ycsb,,: 
> [Permission: actions=CREATE,ADMIN]
>   
>   
> 1 row(s) in 0.0490 seconds
> {code}  
> Later permission will replace previous granted permissions, which confused 
> most of HBase administrator.
> It's seems more reasonable that HBase merge multiple granted permission.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17460) enable_table_replication can not perform cyclic replication of a table

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15823480#comment-15823480
 ] 

Ashish Singhi commented on HBASE-17460:
---

Isn't it sufficient to check only whether the names of CFs for which 
Replication is set is same or not across the cluster for replicating the table 
data ? If yes, then I think  just adding a method in ReplicationAdmin class 
which checks this is sufficient.

> enable_table_replication can not perform cyclic replication of a table
> --
>
> Key: HBASE-17460
> URL: https://issues.apache.org/jira/browse/HBASE-17460
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: NITIN VERMA
>Assignee: NITIN VERMA
>  Labels: replication
> Attachments: HBASE-17460.patch
>
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>
> The enable_table_replication operation is broken for cyclic replication of 
> HBase table as we compare all the properties of column families (including 
> REPLICATION_SCOPE). 
> Below is exactly what happens:
> 1.  Running "enable_table_replication 'table1'  " opeartion on first cluster 
> will set the REPLICATION_SCOPE of all column families to peer id '1'. This 
> will also create a table on second cluster where REPLICATION_SCOPE is still 
> set to peer id '0'.
> 2. Now when we run "enable_table_replication 'table1'" on second cluster, we 
> compare all the properties of table (including REPLICATION_SCOPE_, which 
> obviously is different now. 
> I am proposing a fix for this issue where we should avoid comparing 
> REPLICATION_SCOPE inside HColumnDescriotor::compareTo() method, especially 
> when replication is not already enabled on the desired table.
> I have made that change and it is working. I will submit the patch soon.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17468) unread messages in TCP connections - possible connection leak

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822654#comment-15822654
 ] 

Ashish Singhi commented on HBASE-17468:
---

Same as HBASE-9393.

> unread messages in TCP connections - possible connection leak
> -
>
> Key: HBASE-17468
> URL: https://issues.apache.org/jira/browse/HBASE-17468
> Project: HBase
>  Issue Type: Bug
>Reporter: Shridhar Sahukar
>Priority: Critical
>
> We are running HBase 1.2.0-cdh5.7.1 (Cloudera distribution).
> On our Hadoop cluster, we are seeing that each HBase region server has large 
> number of TCP connections to all the HDFS data nodes and all these 
> connections have unread data in socket buffers. Some of these connections are 
> also in CLOSE_WAIT or FIN_WAIT1 state while the rest are in ESTABLISHED state.
> Looks like HBase is creating some connections requesting data from HDFS, but 
> its forgetting about those connections before it could read the data. Thus 
> the connections are left lingering around with large data stuck in their 
> receive buffers. Also, it seems HDFS closes these connections after a while, 
> but since there is data in receive buffer the connection is left in 
> CLOSE_WAIT/FIN_WAIT1 states.
> Below is a snapshot from one of the region servers:
> ## Total number of connections to HDFS  (pid of region server is 143722)
> [bda@md-bdadev-42 hbase]$ sudo netstat -anp|grep 143722 | wc -l
> 827
> ## Connections that are not in ESTABLISHED state
> [bda@md-bdadev-42 hbase]$ sudo netstat -anp|grep 143722 | grep -v ESTABLISHED 
> | wc -l
> 344
> ##Snapshot of some of these connections:
> tcp   133887  0 146.1.180.43:48533  146.1.180.40:50010  
> ESTABLISHED 143722/java
> tcp82934  0 146.1.180.43:59647  146.1.180.42:50010  
> ESTABLISHED 143722/java
> tcp0  0 146.1.180.43:50761  146.1.180.27:2181   
> ESTABLISHED 143722/java
> tcp   234084  0 146.1.180.43:58335  146.1.180.42:50010  
> ESTABLISHED 143722/java
> tcp   967667  0 146.1.180.43:56136  146.1.180.68:50010  
> ESTABLISHED 143722/java
> tcp   156037  0 146.1.180.43:59659  146.1.180.42:50010  
> ESTABLISHED 143722/java
> tcp   212488  0 146.1.180.43:56810  146.1.180.48:50010  
> ESTABLISHED 143722/java
> tcp61871  0 146.1.180.43:53593  146.1.180.35:50010  
> ESTABLISHED 143722/java
> tcp   121216  0 146.1.180.43:35324  146.1.180.38:50010  
> ESTABLISHED 143722/java
> tcp1  0 146.1.180.43:32982  146.1.180.42:50010  
> CLOSE_WAIT  143722/java
> tcp82934  0 146.1.180.43:42359  146.1.180.54:50010  
> ESTABLISHED 143722/java
> tcp   159422  0 146.1.180.43:59731  146.1.180.42:50010  
> ESTABLISHED 143722/java
> tcp   134573  0 146.1.180.43:60210  146.1.180.76:50010  
> ESTABLISHED 143722/java
> tcp82934  0 146.1.180.43:59713  146.1.180.42:50010  
> ESTABLISHED 143722/java
> tcp   135765  0 146.1.180.43:44412  146.1.180.29:50010  
> ESTABLISHED 143722/java
> tcp   161655  0 146.1.180.43:43117  146.1.180.42:50010  
> ESTABLISHED 143722/java
> tcp75990  0 146.1.180.43:59729  146.1.180.42:50010  
> ESTABLISHED 143722/java
> tcp78583  0 146.1.180.43:59971  146.1.180.42:50010  
> ESTABLISHED 143722/java
> tcp1  0 146.1.180.43:39893  146.1.180.67:50010  
> CLOSE_WAIT  143722/java
> tcp1  0 146.1.180.43:38834  146.1.180.47:50010  
> CLOSE_WAIT  143722/java
> tcp1  0 146.1.180.43:40707  146.1.180.50:50010  
> CLOSE_WAIT  143722/java
> tcp   106102  0 146.1.180.43:48208  146.1.180.75:50010  
> ESTABLISHED 143722/java
> tcp   332013  0 146.1.180.43:34795  146.1.180.37:50010  
> ESTABLISHED 143722/java
> tcp1  0 146.1.180.43:57644  146.1.180.67:50010  
> CLOSE_WAIT  143722/java
> tcp79119  0 146.1.180.43:54438  146.1.180.70:50010  
> ESTABLISHED 143722/java
> tcp77438  0 146.1.180.43:35259  146.1.180.38:50010  
> ESTABLISHED 143722/java
> tcp1  0 146.1.180.43:57579  146.1.180.41:50010  
> CLOSE_WAIT  143722/java
> tcp   318091  0 146.1.180.43:60124  146.1.180.42:50010  
> ESTABLISHED 143722/java
> tcp1  0 146.1.180.43:51715  146.1.180.70:50010  
> CLOSE_WAIT  143722/java
> tcp   126519  0 146.1.180.43:36389  146.1.180.49:50010  
> ESTABLISHED 143722/java
> tcp1  0 

[jira] [Commented] (HBASE-17442) Move most of the replication related classes to hbase-server package

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821547#comment-15821547
 ] 

Ashish Singhi commented on HBASE-17442:
---

bq. could we move them to an hbase-replication module instead?
Good point. +1

> Move most of the replication related classes to hbase-server package
> 
>
> Key: HBASE-17442
> URL: https://issues.apache.org/jira/browse/HBASE-17442
> Project: HBase
>  Issue Type: Sub-task
>  Components: build, Replication
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0
>
>
> After the replication requests are routed through master, replication 
> implementation details didn't need be exposed to client. We should move most 
> of the replication related classes to hbase-server package.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17460) enable_table_replication can not perform cyclic replication of a table

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821478#comment-15821478
 ] 

Ashish Singhi commented on HBASE-17460:
---

Good find.
Regarding patch, It's not correct to change the compareTo logic of HCD for 
replication purpose, it can be used for other purpose also. 
It's better to have a own compareTo method for HCD for replication where we can 
compare only those properties of HCD which are really required for replication.

> enable_table_replication can not perform cyclic replication of a table
> --
>
> Key: HBASE-17460
> URL: https://issues.apache.org/jira/browse/HBASE-17460
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: NITIN VERMA
>  Labels: replication
> Attachments: HBASE-17460.patch
>
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>
> The enable_table_replication operation is broken for cyclic replication of 
> HBase table as we compare all the properties of column families (including 
> REPLICATION_SCOPE). 
> Below is exactly what happens:
> 1.  Running "enable_table_replication 'table1'  " opeartion on first cluster 
> will set the REPLICATION_SCOPE of all column families to peer id '1'. This 
> will also create a table on second cluster where REPLICATION_SCOPE is still 
> set to peer id '0'.
> 2. Now when we run "enable_table_replication 'table1'" on second cluster, we 
> compare all the properties of table (including REPLICATION_SCOPE_, which 
> obviously is different now. 
> I am proposing a fix for this issue where we should avoid comparing 
> REPLICATION_SCOPE inside HColumnDescriotor::compareTo() method, especially 
> when replication is not already enabled on the desired table.
> I have made that change and it is working. I will submit the patch soon.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17444) Namespaces are not showing up when grant 'R' Only permissions to group

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15814985#comment-15814985
 ] 

Ashish Singhi commented on HBASE-17444:
---

Did I say its a bug in HBASE 1.0.0 ?


> Namespaces are not showing up when grant  'R' Only permissions to group
> ---
>
> Key: HBASE-17444
> URL: https://issues.apache.org/jira/browse/HBASE-17444
> Project: HBase
>  Issue Type: Bug
>  Components: hbase, shell
>Affects Versions: 1.0.0
>Reporter: sandeepvura
> Attachments: TestNamespaceCommands.java
>
>
> Hi,
> We are unable to list namespace when given 'R' permission to specific group. 
> But we can able to list namespace when grant 'RWXC' permssion.
> Please confirm is this bug in existing version hbase 1.0.0
> Regards,
> Sandeep



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17444) Namespaces are not showing up when grant 'R' Only permissions to group

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15814937#comment-15814937
 ] 

Ashish Singhi commented on HBASE-17444:
---

branch-1.0 is EOLed.
I ran the attached test on branch-1.3 code and it behaves as expected.
As you mentioned the test should fail for user USER_NS_RCWX as it will return 
the namespaces but it doesn't.
To list namespaces, a user needs to have Admin privilege.

Can you run the attached test on branch-1.0 code ?

> Namespaces are not showing up when grant  'R' Only permissions to group
> ---
>
> Key: HBASE-17444
> URL: https://issues.apache.org/jira/browse/HBASE-17444
> Project: HBase
>  Issue Type: Bug
>  Components: hbase, shell
>Affects Versions: 1.0.0
>Reporter: sandeepvura
> Attachments: TestNamespaceCommands.java
>
>
> Hi,
> We are unable to list namespace when given 'R' permission to specific group. 
> But we can able to list namespace when grant 'RWXC' permssion.
> Please confirm is this bug in existing version hbase 1.0.0
> Regards,
> Sandeep



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17444) Namespaces are not showing up when grant 'R' Only permissions to group

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-17444:
--
Attachment: TestNamespaceCommands.java

> Namespaces are not showing up when grant  'R' Only permissions to group
> ---
>
> Key: HBASE-17444
> URL: https://issues.apache.org/jira/browse/HBASE-17444
> Project: HBase
>  Issue Type: Bug
>  Components: hbase, shell
>Affects Versions: 1.0.0
>Reporter: sandeepvura
> Attachments: TestNamespaceCommands.java
>
>
> Hi,
> We are unable to list namespace when given 'R' permission to specific group. 
> But we can able to list namespace when grant 'RWXC' permssion.
> Please confirm is this bug in existing version hbase 1.0.0
> Regards,
> Sandeep



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17337) list replication peers request should be routed through master

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15810838#comment-15810838
 ] 

Ashish Singhi commented on HBASE-17337:
---

+1, pending QA run.

> list replication peers request should be routed through master
> --
>
> Key: HBASE-17337
> URL: https://issues.apache.org/jira/browse/HBASE-17337
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17337-v1.patch, HBASE-17337-v2.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14061) Support CF-level Storage Policy

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15807447#comment-15807447
 ] 

Ashish Singhi commented on HBASE-14061:
---

lgtm

> Support CF-level Storage Policy
> ---
>
> Key: HBASE-14061
> URL: https://issues.apache.org/jira/browse/HBASE-14061
> Project: HBase
>  Issue Type: Sub-task
>  Components: HFile, regionserver
> Environment: hadoop-2.6.0
>Reporter: Victor Xu
>Assignee: Yu Li
> Attachments: HBASE-14061-master-v1.patch, HBASE-14061.v2.patch, 
> HBASE-14061.v3.patch, HBASE-14061.v4.patch
>
>
> After reading [HBASE-12848|https://issues.apache.org/jira/browse/HBASE-12848] 
> and [HBASE-12934|https://issues.apache.org/jira/browse/HBASE-12934], I wrote 
> a patch to implement cf-level storage policy. 
> My main purpose is to improve random-read performance for some really hot 
> data, which usually locates in certain column family of a big table.
> Usage:
> $ hbase shell
> > alter 'TABLE_NAME', METADATA => {'hbase.hstore.block.storage.policy' => 
> > 'POLICY_NAME'}
> > alter 'TABLE_NAME', {NAME=>'CF_NAME', METADATA => 
> > {'hbase.hstore.block.storage.policy' => 'POLICY_NAME'}}
> HDFS's setStoragePolicy can only take effect when new hfile is created in a 
> configured directory, so I had to make sub directories(for each cf) in 
> region's .tmp directory and set storage policy for them.
> Besides, I had to upgrade hadoop version to 2.6.0 because 
> dfs.getStoragePolicy cannot be easily written in reflection, and I needed 
> this api to finish my unit test.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17337) list replication peers request should be routed through master

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15807417#comment-15807417
 ] 

Ashish Singhi commented on HBASE-17337:
---

bq. No, this is listing operation, as opposed to a get with with peer_id. We 
should not throw an exception even if there are no peers.
Ya, makes sense. Thank you.

> list replication peers request should be routed through master
> --
>
> Key: HBASE-17337
> URL: https://issues.apache.org/jira/browse/HBASE-17337
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17337-v1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17337) list replication peers request should be routed through master

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15804452#comment-15804452
 ] 

Ashish Singhi commented on HBASE-17337:
---

Add javadoc and InterfaceAudience to ReplicationPeerDescription class.

{code}
111 for (String peerId : peerIds) {
112   if (pattern == null || (pattern != null && 
pattern.matcher(peerId).matches())) {
113 peers.add(new ReplicationPeerDescription(peerId, 
replicationPeers
114 .getStatusOfPeerFromBackingStore(peerId), replicationPeers
115 .getReplicationPeerConfig(peerId)));
116   }
117 }
{code}

If pattern is not null and once the pattern matches the peer id then we can 
break out of the for loop.

If user passes a pattern which doesn't match any existing peerIds, I think we 
should throw ReplicationPeerNotFoundException/IllegalArgumentException for it.



> list replication peers request should be routed through master
> --
>
> Key: HBASE-17337
> URL: https://issues.apache.org/jira/browse/HBASE-17337
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17337-v1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17290) Potential loss of data for replication of bulk loaded hfiles

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-17290:
--
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Pushed to master and branch-1.
Thanks for the reviews, Ted.

> Potential loss of data for replication of bulk loaded hfiles
> 
>
> Key: HBASE-17290
> URL: https://issues.apache.org/jira/browse/HBASE-17290
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Ted Yu
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17290.branch-1.patch, HBASE-17290.patch, 
> HBASE-17290.v1.patch
>
>
> Currently the support for replication of bulk loaded hfiles relies on bulk 
> load marker written in the WAL.
> The move of bulk loaded hfile(s) (into region directory) may succeed but the 
> write of bulk load marker may fail.
> This means that although bulk loaded hfile is being served in source cluster, 
> the replication wouldn't happen.
> Normally operator is supposed to retry the bulk load. But relying on human 
> retry is not robust solution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17290) Potential loss of data for replication of bulk loaded hfiles

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15804402#comment-15804402
 ] 

Ashish Singhi commented on HBASE-17290:
---

bq. Failed junit tests  hadoop.hbase.client.TestMetaWithReplicas
Test failure was not related to patch. It is failing on other build runs also 
[here|https://builds.apache.org/job/HBase-1.4/584/], 
[here|https://builds.apache.org/job/HBase-1.4/582/]...

> Potential loss of data for replication of bulk loaded hfiles
> 
>
> Key: HBASE-17290
> URL: https://issues.apache.org/jira/browse/HBASE-17290
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Ted Yu
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17290.branch-1.patch, HBASE-17290.patch, 
> HBASE-17290.v1.patch
>
>
> Currently the support for replication of bulk loaded hfiles relies on bulk 
> load marker written in the WAL.
> The move of bulk loaded hfile(s) (into region directory) may succeed but the 
> write of bulk load marker may fail.
> This means that although bulk loaded hfile is being served in source cluster, 
> the replication wouldn't happen.
> Normally operator is supposed to retry the bulk load. But relying on human 
> retry is not robust solution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17290) Potential loss of data for replication of bulk loaded hfiles

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-17290:
--
Attachment: HBASE-17290.branch-1.patch

Attached branch-1 patch.

> Potential loss of data for replication of bulk loaded hfiles
> 
>
> Key: HBASE-17290
> URL: https://issues.apache.org/jira/browse/HBASE-17290
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Ted Yu
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17290.branch-1.patch, HBASE-17290.patch, 
> HBASE-17290.v1.patch
>
>
> Currently the support for replication of bulk loaded hfiles relies on bulk 
> load marker written in the WAL.
> The move of bulk loaded hfile(s) (into region directory) may succeed but the 
> write of bulk load marker may fail.
> This means that although bulk loaded hfile is being served in source cluster, 
> the replication wouldn't happen.
> Normally operator is supposed to retry the bulk load. But relying on human 
> retry is not robust solution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17290) Potential loss of data for replication of bulk loaded hfiles

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15803638#comment-15803638
 ] 

Ashish Singhi commented on HBASE-17290:
---

Planning to commit this after an hour or so, if there is no further review 
comments.

> Potential loss of data for replication of bulk loaded hfiles
> 
>
> Key: HBASE-17290
> URL: https://issues.apache.org/jira/browse/HBASE-17290
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Ted Yu
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17290.patch, HBASE-17290.v1.patch
>
>
> Currently the support for replication of bulk loaded hfiles relies on bulk 
> load marker written in the WAL.
> The move of bulk loaded hfile(s) (into region directory) may succeed but the 
> write of bulk load marker may fail.
> This means that although bulk loaded hfile is being served in source cluster, 
> the replication wouldn't happen.
> Normally operator is supposed to retry the bulk load. But relying on human 
> retry is not robust solution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17290) Potential loss of data for replication of bulk loaded hfiles

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15801949#comment-15801949
 ] 

Ashish Singhi commented on HBASE-17290:
---

Addressed the comments.
Please review.

> Potential loss of data for replication of bulk loaded hfiles
> 
>
> Key: HBASE-17290
> URL: https://issues.apache.org/jira/browse/HBASE-17290
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Ted Yu
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17290.patch, HBASE-17290.v1.patch
>
>
> Currently the support for replication of bulk loaded hfiles relies on bulk 
> load marker written in the WAL.
> The move of bulk loaded hfile(s) (into region directory) may succeed but the 
> write of bulk load marker may fail.
> This means that although bulk loaded hfile is being served in source cluster, 
> the replication wouldn't happen.
> Normally operator is supposed to retry the bulk load. But relying on human 
> retry is not robust solution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17290) Potential loss of data for replication of bulk loaded hfiles

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-17290:
--
Attachment: HBASE-17290.v1.patch

> Potential loss of data for replication of bulk loaded hfiles
> 
>
> Key: HBASE-17290
> URL: https://issues.apache.org/jira/browse/HBASE-17290
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Ted Yu
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17290.patch, HBASE-17290.v1.patch
>
>
> Currently the support for replication of bulk loaded hfiles relies on bulk 
> load marker written in the WAL.
> The move of bulk loaded hfile(s) (into region directory) may succeed but the 
> write of bulk load marker may fail.
> This means that although bulk loaded hfile is being served in source cluster, 
> the replication wouldn't happen.
> Normally operator is supposed to retry the bulk load. But relying on human 
> retry is not robust solution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17290) Potential loss of data for replication of bulk loaded hfiles

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15801873#comment-15801873
 ] 

Ashish Singhi commented on HBASE-17290:
---

{quote} Can you point me to the code which handles the case where Path for bulk 
loaded hfile is recorded but the commit (move of hfile) fails ?
In that scenario, the file wouldn't be found at time of replication. {quote}
In that scenario we will end up here, 
https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java#L380

> Potential loss of data for replication of bulk loaded hfiles
> 
>
> Key: HBASE-17290
> URL: https://issues.apache.org/jira/browse/HBASE-17290
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Ted Yu
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17290.patch
>
>
> Currently the support for replication of bulk loaded hfiles relies on bulk 
> load marker written in the WAL.
> The move of bulk loaded hfile(s) (into region directory) may succeed but the 
> write of bulk load marker may fail.
> This means that although bulk loaded hfile is being served in source cluster, 
> the replication wouldn't happen.
> Normally operator is supposed to retry the bulk load. But relying on human 
> retry is not robust solution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HBASE-14061) Support CF-level Storage Policy

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15801374#comment-15801374
 ] 

Ashish Singhi edited comment on HBASE-14061 at 1/5/17 1:40 PM:
---

{code}
/**
1280   * Return the encryption algorithm in use by this family
1281   * 
1282   * Not using {@code enum} here because HDFS is not using {@code enum} 
for storage policy, see
1283   * 
org.apache.hadoop.hdfs.server.blockmanagement.BlockStoragePolicySuite for more 
details
1284   */
1285  public String getStoragePolicy() {
1286return getValue(STORAGE_POLICY);
1287  }

 /**
1290   * Set the encryption algorithm for use with this family
1291   * @param policy
1292   */
1293  public HColumnDescriptor setStoragePolicy(String policy) {
1294setValue(STORAGE_POLICY, policy);
1295return this;
1296  }
{code}
That javadoc is for HCD#get and setEncryptionType, need to correct it.
Otheriswe LGTM.


was (Author: ashish singhi):
{code}
/**
1280   * Return the encryption algorithm in use by this family
1281   * 
1282   * Not using {@code enum} here because HDFS is not using {@code enum} 
for storage policy, see
1283   * 
org.apache.hadoop.hdfs.server.blockmanagement.BlockStoragePolicySuite for more 
details
1284   */
1285  public String getStoragePolicy() {
1286return getValue(STORAGE_POLICY);
1287  }

 /**
1290   * Set the encryption algorithm for use with this family
1291   * @param policy
1292   */
1293  public HColumnDescriptor setStoragePolicy(String policy) {
1294setValue(STORAGE_POLICY, policy);
1295return this;
1296  }
{code}
That javadoc is for HCD#getEncryptionType, need to correct it.
Otheriswe LGTM.

> Support CF-level Storage Policy
> ---
>
> Key: HBASE-14061
> URL: https://issues.apache.org/jira/browse/HBASE-14061
> Project: HBase
>  Issue Type: Sub-task
>  Components: HFile, regionserver
> Environment: hadoop-2.6.0
>Reporter: Victor Xu
>Assignee: Yu Li
> Attachments: HBASE-14061-master-v1.patch, HBASE-14061.v2.patch, 
> HBASE-14061.v3.patch
>
>
> After reading [HBASE-12848|https://issues.apache.org/jira/browse/HBASE-12848] 
> and [HBASE-12934|https://issues.apache.org/jira/browse/HBASE-12934], I wrote 
> a patch to implement cf-level storage policy. 
> My main purpose is to improve random-read performance for some really hot 
> data, which usually locates in certain column family of a big table.
> Usage:
> $ hbase shell
> > alter 'TABLE_NAME', METADATA => {'hbase.hstore.block.storage.policy' => 
> > 'POLICY_NAME'}
> > alter 'TABLE_NAME', {NAME=>'CF_NAME', METADATA => 
> > {'hbase.hstore.block.storage.policy' => 'POLICY_NAME'}}
> HDFS's setStoragePolicy can only take effect when new hfile is created in a 
> configured directory, so I had to make sub directories(for each cf) in 
> region's .tmp directory and set storage policy for them.
> Besides, I had to upgrade hadoop version to 2.6.0 because 
> dfs.getStoragePolicy cannot be easily written in reflection, and I needed 
> this api to finish my unit test.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14061) Support CF-level Storage Policy

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15801374#comment-15801374
 ] 

Ashish Singhi commented on HBASE-14061:
---

{code}
/**
1280   * Return the encryption algorithm in use by this family
1281   * 
1282   * Not using {@code enum} here because HDFS is not using {@code enum} 
for storage policy, see
1283   * 
org.apache.hadoop.hdfs.server.blockmanagement.BlockStoragePolicySuite for more 
details
1284   */
1285  public String getStoragePolicy() {
1286return getValue(STORAGE_POLICY);
1287  }

 /**
1290   * Set the encryption algorithm for use with this family
1291   * @param policy
1292   */
1293  public HColumnDescriptor setStoragePolicy(String policy) {
1294setValue(STORAGE_POLICY, policy);
1295return this;
1296  }
{code}
That javadoc is for HCD#getEncryptionType, need to correct it.
Otheriswe LGTM.

> Support CF-level Storage Policy
> ---
>
> Key: HBASE-14061
> URL: https://issues.apache.org/jira/browse/HBASE-14061
> Project: HBase
>  Issue Type: Sub-task
>  Components: HFile, regionserver
> Environment: hadoop-2.6.0
>Reporter: Victor Xu
>Assignee: Yu Li
> Attachments: HBASE-14061-master-v1.patch, HBASE-14061.v2.patch, 
> HBASE-14061.v3.patch
>
>
> After reading [HBASE-12848|https://issues.apache.org/jira/browse/HBASE-12848] 
> and [HBASE-12934|https://issues.apache.org/jira/browse/HBASE-12934], I wrote 
> a patch to implement cf-level storage policy. 
> My main purpose is to improve random-read performance for some really hot 
> data, which usually locates in certain column family of a big table.
> Usage:
> $ hbase shell
> > alter 'TABLE_NAME', METADATA => {'hbase.hstore.block.storage.policy' => 
> > 'POLICY_NAME'}
> > alter 'TABLE_NAME', {NAME=>'CF_NAME', METADATA => 
> > {'hbase.hstore.block.storage.policy' => 'POLICY_NAME'}}
> HDFS's setStoragePolicy can only take effect when new hfile is created in a 
> configured directory, so I had to make sub directories(for each cf) in 
> region's .tmp directory and set storage policy for them.
> Besides, I had to upgrade hadoop version to 2.6.0 because 
> dfs.getStoragePolicy cannot be easily written in reflection, and I needed 
> this api to finish my unit test.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15172) Support setting storage policy in bulkload

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15801353#comment-15801353
 ] 

Ashish Singhi commented on HBASE-15172:
---

+1

> Support setting storage policy in bulkload
> --
>
> Key: HBASE-15172
> URL: https://issues.apache.org/jira/browse/HBASE-15172
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-15172.patch, HBASE-15172.v2.patch
>
>
> When using tiered HFile storage, we should be able to generating hfile with 
> correct storage type during bulkload. This JIRA is targeting at making it 
> possible.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HBASE-17290) Potential loss of data for replication of bulk loaded hfiles

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15801226#comment-15801226
 ] 

Ashish Singhi edited comment on HBASE-17290 at 1/5/17 12:25 PM:


Sorry for the delay, got stuck with company work.
I have attached the patch.

Added a new RS observer, ReplicationObserver to solve this bug.
Please review.


was (Author: ashish singhi):
Sorry, got stuck with company work.
I have attached the patch.

Added a new RS observer, ReplicationObserver to solve this bug.
Please review.

> Potential loss of data for replication of bulk loaded hfiles
> 
>
> Key: HBASE-17290
> URL: https://issues.apache.org/jira/browse/HBASE-17290
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Ted Yu
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17290.patch
>
>
> Currently the support for replication of bulk loaded hfiles relies on bulk 
> load marker written in the WAL.
> The move of bulk loaded hfile(s) (into region directory) may succeed but the 
> write of bulk load marker may fail.
> This means that although bulk loaded hfile is being served in source cluster, 
> the replication wouldn't happen.
> Normally operator is supposed to retry the bulk load. But relying on human 
> retry is not robust solution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HBASE-17290) Potential loss of data for replication of bulk loaded hfiles

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi reassigned HBASE-17290:
-

Assignee: Ashish Singhi

> Potential loss of data for replication of bulk loaded hfiles
> 
>
> Key: HBASE-17290
> URL: https://issues.apache.org/jira/browse/HBASE-17290
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Ted Yu
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17290.patch
>
>
> Currently the support for replication of bulk loaded hfiles relies on bulk 
> load marker written in the WAL.
> The move of bulk loaded hfile(s) (into region directory) may succeed but the 
> write of bulk load marker may fail.
> This means that although bulk loaded hfile is being served in source cluster, 
> the replication wouldn't happen.
> Normally operator is supposed to retry the bulk load. But relying on human 
> retry is not robust solution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17290) Potential loss of data for replication of bulk loaded hfiles

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15801226#comment-15801226
 ] 

Ashish Singhi commented on HBASE-17290:
---

Sorry, got stuck with company work.
I have attached the patch.

Added a new RS observer, ReplicationObserver to solve this bug.
Please review.

> Potential loss of data for replication of bulk loaded hfiles
> 
>
> Key: HBASE-17290
> URL: https://issues.apache.org/jira/browse/HBASE-17290
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Ted Yu
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17290.patch
>
>
> Currently the support for replication of bulk loaded hfiles relies on bulk 
> load marker written in the WAL.
> The move of bulk loaded hfile(s) (into region directory) may succeed but the 
> write of bulk load marker may fail.
> This means that although bulk loaded hfile is being served in source cluster, 
> the replication wouldn't happen.
> Normally operator is supposed to retry the bulk load. But relying on human 
> retry is not robust solution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17290) Potential loss of data for replication of bulk loaded hfiles

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-17290:
--
Fix Version/s: 1.4.0
   2.0.0
Affects Version/s: 1.3.0
   Status: Patch Available  (was: Open)

> Potential loss of data for replication of bulk loaded hfiles
> 
>
> Key: HBASE-17290
> URL: https://issues.apache.org/jira/browse/HBASE-17290
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Ted Yu
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17290.patch
>
>
> Currently the support for replication of bulk loaded hfiles relies on bulk 
> load marker written in the WAL.
> The move of bulk loaded hfile(s) (into region directory) may succeed but the 
> write of bulk load marker may fail.
> This means that although bulk loaded hfile is being served in source cluster, 
> the replication wouldn't happen.
> Normally operator is supposed to retry the bulk load. But relying on human 
> retry is not robust solution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17290) Potential loss of data for replication of bulk loaded hfiles

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-17290:
--
Attachment: HBASE-17290.patch

> Potential loss of data for replication of bulk loaded hfiles
> 
>
> Key: HBASE-17290
> URL: https://issues.apache.org/jira/browse/HBASE-17290
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
> Attachments: HBASE-17290.patch
>
>
> Currently the support for replication of bulk loaded hfiles relies on bulk 
> load marker written in the WAL.
> The move of bulk loaded hfile(s) (into region directory) may succeed but the 
> write of bulk load marker may fail.
> This means that although bulk loaded hfile is being served in source cluster, 
> the replication wouldn't happen.
> Normally operator is supposed to retry the bulk load. But relying on human 
> retry is not robust solution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17336) get/update replication peer config requests should be routed through master

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15780155#comment-15780155
 ] 

Ashish Singhi commented on HBASE-17336:
---

+1 on V4, pending QA run.

> get/update replication peer config requests should be routed through master
> ---
>
> Key: HBASE-17336
> URL: https://issues.apache.org/jira/browse/HBASE-17336
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17336-v1.patch, HBASE-17336-v2.patch, 
> HBASE-17336-v3.patch, HBASE-17336-v4.patch
>
>
> As HBASE-11392 description says, we should move replication operations to be 
> routed through master.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17336) get/update replication peer config requests should be routed through master

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15780153#comment-15780153
 ] 

Ashish Singhi commented on HBASE-17336:
---

I have seen that, but just from a user perspective it will me more helpful to 
have deprecate doc for each method suggesting which method to use.

> get/update replication peer config requests should be routed through master
> ---
>
> Key: HBASE-17336
> URL: https://issues.apache.org/jira/browse/HBASE-17336
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17336-v1.patch, HBASE-17336-v2.patch, 
> HBASE-17336-v3.patch, HBASE-17336-v4.patch
>
>
> As HBASE-11392 description says, we should move replication operations to be 
> routed through master.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17336) get/update replication peer config requests should be routed through master

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15779712#comment-15779712
 ] 

Ashish Singhi commented on HBASE-17336:
---

I have left some comments on the review board.
It would be good if you can add deprecate tag on the ReplicationAdmin methods 
and suggest devs to use which method of Admin class. May be you can do this in 
another sub-task.

> get/update replication peer config requests should be routed through master
> ---
>
> Key: HBASE-17336
> URL: https://issues.apache.org/jira/browse/HBASE-17336
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17336-v1.patch, HBASE-17336-v2.patch, 
> HBASE-17336-v3.patch
>
>
> As HBASE-11392 description says, we should move replication operations to be 
> routed through master.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11392) add/remove peer requests should be routed through master

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-11392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766177#comment-15766177
 ] 

Ashish Singhi commented on HBASE-11392:
---

v6 patch is good to me.

> add/remove peer requests should be routed through master
> 
>
> Key: HBASE-11392
> URL: https://issues.apache.org/jira/browse/HBASE-11392
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-11392-v1.patch, HBASE-11392-v2.patch, 
> HBASE-11392-v3.patch, HBASE-11392-v4.patch, HBASE-11392-v5.patch, 
> HBASE-11392-v6.patch
>
>
> ReplicationAdmin directly operates over the zookeeper data for replication 
> setup. We should move these operations to be routed through master for two 
> reasons: 
>  - Replication implementation details are exposed to client. We should move 
> most of the replication related classes to hbase-server package. 
>  - Routing the requests through master is the standard practice for all other 
> operations. It allows for decoupling implementation details from the client 
> and code.
> Review board: https://reviews.apache.org/r/54730/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17290) Potential loss of data for replication of bulk loaded hfiles

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15760585#comment-15760585
 ] 

Ashish Singhi commented on HBASE-17290:
---

[~tedyu], isn't this already handled as part of HBASE-15425 ?

> Potential loss of data for replication of bulk loaded hfiles
> 
>
> Key: HBASE-17290
> URL: https://issues.apache.org/jira/browse/HBASE-17290
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>
> Currently the support for replication of bulk loaded hfiles relies on bulk 
> load marker written in the WAL.
> The move of bulk loaded hfile(s) (into region directory) may succeed but the 
> write of bulk load marker may fail.
> This means that although bulk loaded hfile is being served in source cluster, 
> the replication wouldn't happen.
> Normally operator is supposed to retry the bulk load. But relying on human 
> retry is not robust solution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14417) Incremental backup and bulk loading

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715023#comment-15715023
 ] 

Ashish Singhi commented on HBASE-14417:
---

I don't want to block this jira. I will be on holiday for next two weeks so 
will not be able to check this before that. If any one else is fine pls go 
ahead and commit it. I will review it later.
Thanks

> Incremental backup and bulk loading
> ---
>
> Key: HBASE-14417
> URL: https://issues.apache.org/jira/browse/HBASE-14417
> Project: HBase
>  Issue Type: New Feature
>Affects Versions: 2.0.0
>Reporter: Vladimir Rodionov
>Assignee: Ted Yu
>Priority: Critical
>  Labels: backup
> Fix For: 2.0.0
>
> Attachments: 14417-tbl-ext.v9.txt, 14417.v1.txt, 14417.v11.txt, 
> 14417.v13.txt, 14417.v2.txt, 14417.v21.txt, 14417.v23.txt, 14417.v24.txt, 
> 14417.v25.txt, 14417.v6.txt
>
>
> Currently, incremental backup is based on WAL files. Bulk data loading 
> bypasses WALs for obvious reasons, breaking incremental backups. The only way 
> to continue backups after bulk loading is to create new full backup of a 
> table. This may not be feasible for customers who do bulk loading regularly 
> (say, every day).
> Google doc for design:
> https://docs.google.com/document/d/1ACCLsecHDvzVSasORgqqRNrloGx4mNYIbvAU7lq5lJE



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HBASE-14417) Incremental backup and bulk loading

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715023#comment-15715023
 ] 

Ashish Singhi edited comment on HBASE-14417 at 12/2/16 12:38 PM:
-

I don't want to block this jira. I will be on holiday for next two weeks so 
will not be able to check this before that. If any one else is fine with the 
patch pls go ahead and commit it. I will review it later.
Thanks


was (Author: ashish singhi):
I don't want to block this jira. I will be on holiday for next two weeks so 
will not be able to check this before that. If any one else is fine pls go 
ahead and commit it. I will review it later.
Thanks

> Incremental backup and bulk loading
> ---
>
> Key: HBASE-14417
> URL: https://issues.apache.org/jira/browse/HBASE-14417
> Project: HBase
>  Issue Type: New Feature
>Affects Versions: 2.0.0
>Reporter: Vladimir Rodionov
>Assignee: Ted Yu
>Priority: Critical
>  Labels: backup
> Fix For: 2.0.0
>
> Attachments: 14417-tbl-ext.v9.txt, 14417.v1.txt, 14417.v11.txt, 
> 14417.v13.txt, 14417.v2.txt, 14417.v21.txt, 14417.v23.txt, 14417.v24.txt, 
> 14417.v25.txt, 14417.v6.txt
>
>
> Currently, incremental backup is based on WAL files. Bulk data loading 
> bypasses WALs for obvious reasons, breaking incremental backups. The only way 
> to continue backups after bulk loading is to create new full backup of a 
> table. This may not be feasible for customers who do bulk loading regularly 
> (say, every day).
> Google doc for design:
> https://docs.google.com/document/d/1ACCLsecHDvzVSasORgqqRNrloGx4mNYIbvAU7lq5lJE



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16499) slow replication for small HBase clusters

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-16499:
--
Component/s: Replication

> slow replication for small HBase clusters
> -
>
> Key: HBASE-16499
> URL: https://issues.apache.org/jira/browse/HBASE-16499
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: Vikas Vishwakarma
>Assignee: Vikas Vishwakarma
>
> For small clusters 10-20 nodes we recently observed that replication is 
> progressing very slowly when we do bulk writes and there is lot of lag 
> accumulation on AgeOfLastShipped / SizeOfLogQueue. From the logs we observed 
> that the number of threads used for shipping wal edits in parallel comes from 
> the following equation in HBaseInterClusterReplicationEndpoint
> int n = Math.min(Math.min(this.maxThreads, entries.size()/100+1),
>   replicationSinkMgr.getSinks().size());
> ... 
>   for (int i=0; i entryLists.add(new ArrayList(entries.size()/n+1)); <-- 
> batch size
>   }
> ...
> for (int i=0; i  .
> // RuntimeExceptions encountered here bubble up and are handled 
> in ReplicationSource
> pool.submit(createReplicator(entryLists.get(i), i));  <-- 
> concurrency 
> futures++;
>   }
> }
> maxThreads is fixed & configurable and since we are taking min of the three 
> values n gets decided based replicationSinkMgr.getSinks().size() when we have 
> enough edits to replicate
> replicationSinkMgr.getSinks().size() is decided based on 
> int numSinks = (int) Math.ceil(slaveAddresses.size() * ratio);
> where ratio is this.ratio = conf.getFloat("replication.source.ratio", 
> DEFAULT_REPLICATION_SOURCE_RATIO);
> Currently DEFAULT_REPLICATION_SOURCE_RATIO is set to 10% so for small 
> clusters of size 10-20 RegionServers  the value we get for numSinks and hence 
> n is very small like 1 or 2. This substantially reduces the pool concurrency 
> used for shipping wal edits in parallel effectively slowing down replication 
> for small clusters and causing lot of lag accumulation in AgeOfLastShipped. 
> Sometimes it takes tens of hours to clear off the entire replication queue 
> even after the client has finished writing on the source side. 
> We are running tests by varying replication.source.ratio and have seen 
> multi-fold improvement in total replication time (will update the results 
> here). I wanted to propose here that we should increase the default value for 
> replication.source.ratio also so that we have sufficient concurrency even for 
> small clusters. We figured it out after lot of iterations and debugging so 
> probably slightly higher default will save the trouble. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16499) slow replication for small HBase clusters

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15711200#comment-15711200
 ] 

Ashish Singhi commented on HBASE-16499:
---

Any update here, [~vik.karma] ?

> slow replication for small HBase clusters
> -
>
> Key: HBASE-16499
> URL: https://issues.apache.org/jira/browse/HBASE-16499
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: Vikas Vishwakarma
>Assignee: Vikas Vishwakarma
>
> For small clusters 10-20 nodes we recently observed that replication is 
> progressing very slowly when we do bulk writes and there is lot of lag 
> accumulation on AgeOfLastShipped / SizeOfLogQueue. From the logs we observed 
> that the number of threads used for shipping wal edits in parallel comes from 
> the following equation in HBaseInterClusterReplicationEndpoint
> int n = Math.min(Math.min(this.maxThreads, entries.size()/100+1),
>   replicationSinkMgr.getSinks().size());
> ... 
>   for (int i=0; i entryLists.add(new ArrayList(entries.size()/n+1)); <-- 
> batch size
>   }
> ...
> for (int i=0; i  .
> // RuntimeExceptions encountered here bubble up and are handled 
> in ReplicationSource
> pool.submit(createReplicator(entryLists.get(i), i));  <-- 
> concurrency 
> futures++;
>   }
> }
> maxThreads is fixed & configurable and since we are taking min of the three 
> values n gets decided based replicationSinkMgr.getSinks().size() when we have 
> enough edits to replicate
> replicationSinkMgr.getSinks().size() is decided based on 
> int numSinks = (int) Math.ceil(slaveAddresses.size() * ratio);
> where ratio is this.ratio = conf.getFloat("replication.source.ratio", 
> DEFAULT_REPLICATION_SOURCE_RATIO);
> Currently DEFAULT_REPLICATION_SOURCE_RATIO is set to 10% so for small 
> clusters of size 10-20 RegionServers  the value we get for numSinks and hence 
> n is very small like 1 or 2. This substantially reduces the pool concurrency 
> used for shipping wal edits in parallel effectively slowing down replication 
> for small clusters and causing lot of lag accumulation in AgeOfLastShipped. 
> Sometimes it takes tens of hours to clear off the entire replication queue 
> even after the client has finished writing on the source side. 
> We are running tests by varying replication.source.ratio and have seen 
> multi-fold improvement in total replication time (will update the results 
> here). I wanted to propose here that we should increase the default value for 
> replication.source.ratio also so that we have sufficient concurrency even for 
> small clusters. We figured it out after lot of iterations and debugging so 
> probably slightly higher default will save the trouble. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17178) Add region balance throttling

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707949#comment-15707949
 ] 

Ashish Singhi commented on HBASE-17178:
---

+1

> Add region balance throttling
> -
>
> Key: HBASE-17178
> URL: https://issues.apache.org/jira/browse/HBASE-17178
> Project: HBase
>  Issue Type: Improvement
>  Components: Balancer
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-17178-branch-1.patch, HBASE-17178-v1.patch, 
> HBASE-17178-v2.patch, HBASE-17178-v3.patch, HBASE-17178-v4.patch, 
> HBASE-17178-v5.patch, HBASE-17178-v6.patch
>
>
> Our online cluster serves dozens of  tables and different tables serve for 
> different services. If the balancer moves too many regions in the same time, 
> it will decrease the availability for some table or some services. So we add 
> region balance throttling on our online serve cluster. 
> We introduce a new config hbase.balancer.max.balancing.regions, which means 
> the max number of regions in transition when balancing.
> If we config this to 1 and a table have 100 regions, then the table will have 
> 99 regions available at any time. It helps a lot for our use case and it has 
> been running a long time
> our production cluster.
> But for some use case, we need the balancer run faster. If a cluster has 100 
> regionservers, then it add 50 new regionservers for peak requests. Then it 
> need balancer run as soon as
> possible and let the cluster reach a balance state soon. Our idea is compute 
> max number of regions in transition by the max balancing time and the average 
> time of region in transition.
> Then the balancer use the computed value to throttling.
> Examples for understanding.
> A cluster has 100 regionservers, each regionserver has 200 regions and the 
> average time of region in transition is 1 seconds, we config the max 
> balancing time is 10 * 60 seconds.
> Case 1. One regionserver crash, the cluster at most need balance 200 regions. 
> Then 200 / (10 * 60s / 1s) < 1, it means the max number of regions in 
> transition is 1 when balancing. Then the balancer can move region one by one 
> and the cluster will have high availability  when balancing.
> Case 2. Add other 100 regionservers, the cluster at most need balance 1 
> regions. Then 1 / (10 * 60s / 1s) = 16.7, it means the max number of 
> regions in transition is 17 when balancing. Then the cluster can reach a 
> balance state within the max balancing time.
> Any suggestions are welcomed.
> Review board: https://reviews.apache.org/r/54191/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HBASE-17203) unable to see the namespaces and tables when grant read-only privleges

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707827#comment-15707827
 ] 

Ashish Singhi edited comment on HBASE-17203 at 11/30/16 7:23 AM:
-

This is correct.
To list namespaces, a user needs to have Admin privilege.
Please check https://hbase.apache.org/book.html#appendix_acl_matrix


was (Author: ashish singhi):
This is correct.
To see to namespaces a user needs to have Admin privilege.
Please check https://hbase.apache.org/book.html#appendix_acl_matrix

> unable to see the namespaces and tables when grant read-only privleges
> --
>
> Key: HBASE-17203
> URL: https://issues.apache.org/jira/browse/HBASE-17203
> Project: HBase
>  Issue Type: Task
>  Components: hbase, shell
>Affects Versions: 1.0.0
>Reporter: sandeep vura
>
> Hi,
> i have created namespaces and given grant  read-only privileges. When i run 
> "list_namespace" on hbase shell i am unable to see the namespaces and also 
> the tables.
> But namespaces and tables are able to see in HUE --> HBase browser.
> Executed commands:
> create_namespace 'q1'
> grant '@hadooptest','R','@q1'
> Please advise.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-17203) unable to see the namespaces and tables when grant read-only privleges

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi resolved HBASE-17203.
---
Resolution: Invalid

This is correct.
To see to namespaces a user needs to have Admin privilege.
Please check https://hbase.apache.org/book.html#appendix_acl_matrix

> unable to see the namespaces and tables when grant read-only privleges
> --
>
> Key: HBASE-17203
> URL: https://issues.apache.org/jira/browse/HBASE-17203
> Project: HBase
>  Issue Type: Task
>  Components: hbase, shell
>Affects Versions: 1.0.0
>Reporter: sandeep vura
>
> Hi,
> i have created namespaces and given grant  read-only privileges. When i run 
> "list_namespace" on hbase shell i am unable to see the namespaces and also 
> the tables.
> But namespaces and tables are able to see in HUE --> HBase browser.
> Executed commands:
> create_namespace 'q1'
> grant '@hadooptest','R','@q1'
> Please advise.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17178) Add region balance throttling

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707683#comment-15707683
 ] 

Ashish Singhi commented on HBASE-17178:
---

I left some very minor comments on RB.
Thanks.

> Add region balance throttling
> -
>
> Key: HBASE-17178
> URL: https://issues.apache.org/jira/browse/HBASE-17178
> Project: HBase
>  Issue Type: Improvement
>  Components: Balancer
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-17178-branch-1.patch, HBASE-17178-v1.patch, 
> HBASE-17178-v2.patch, HBASE-17178-v3.patch, HBASE-17178-v4.patch, 
> HBASE-17178-v5.patch
>
>
> Our online cluster serves dozens of  tables and different tables serve for 
> different services. If the balancer moves too many regions in the same time, 
> it will decrease the availability for some table or some services. So we add 
> region balance throttling on our online serve cluster. 
> We introduce a new config hbase.balancer.max.balancing.regions, which means 
> the max number of regions in transition when balancing.
> If we config this to 1 and a table have 100 regions, then the table will have 
> 99 regions available at any time. It helps a lot for our use case and it has 
> been running a long time
> our production cluster.
> But for some use case, we need the balancer run faster. If a cluster has 100 
> regionservers, then it add 50 new regionservers for peak requests. Then it 
> need balancer run as soon as
> possible and let the cluster reach a balance state soon. Our idea is compute 
> max number of regions in transition by the max balancing time and the average 
> time of region in transition.
> Then the balancer use the computed value to throttling.
> Examples for understanding.
> A cluster has 100 regionservers, each regionserver has 200 regions and the 
> average time of region in transition is 1 seconds, we config the max 
> balancing time is 10 * 60 seconds.
> Case 1. One regionserver crash, the cluster at most need balance 200 regions. 
> Then 200 / (10 * 60s / 1s) < 1, it means the max number of regions in 
> transition is 1 when balancing. Then the balancer can move region one by one 
> and the cluster will have high availability  when balancing.
> Case 2. Add other 100 regionservers, the cluster at most need balance 1 
> regions. Then 1 / (10 * 60s / 1s) = 16.7, it means the max number of 
> regions in transition is 17 when balancing. Then the cluster can reach a 
> balance state within the max balancing time.
> Any suggestions are welcomed.
> Review board: https://reviews.apache.org/r/54191/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16302) age of last shipped op and age of last applied op should be histograms

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-16302:
--
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: (was: 1.3.1)
   Status: Resolved  (was: Patch Available)

I have committed this to master and branch-1.

Thanks for the patch, Ashu.
Thanks for the review, Ted.

> age of last shipped op and age of last applied op should be histograms
> --
>
> Key: HBASE-16302
> URL: https://issues.apache.org/jira/browse/HBASE-16302
> Project: HBase
>  Issue Type: Improvement
>  Components: Replication
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16302.patch.v0.patch
>
>
> Replication exports metric ageOfLastShippedOp as an indication of how much 
> replication is lagging. But, with multiwal enabled, it's not representative 
> because replication could be lagging for a long time for one wal group 
> (something wrong with a particular region) while being fine for others. The 
> ageOfLastShippedOp becomes a useless metric for alerting in such a case.
> Also, since there is no mapping between individual replication sources and 
> replication sinks, the age of last applied op can be a highly spiky metric if 
> only certain replication sources are lagging.
> We should use histograms for these metrics and use maximum value of this 
> histogram to report replication lag when building stats.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14417) Incremental backup and bulk loading

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15694965#comment-15694965
 ] 

Ashish Singhi commented on HBASE-14417:
---

Can you upload the patch on RB. It will be easy to review.

> Incremental backup and bulk loading
> ---
>
> Key: HBASE-14417
> URL: https://issues.apache.org/jira/browse/HBASE-14417
> Project: HBase
>  Issue Type: New Feature
>Affects Versions: 2.0.0
>Reporter: Vladimir Rodionov
>Assignee: Ted Yu
>Priority: Critical
>  Labels: backup
> Fix For: 2.0.0
>
> Attachments: 14417-tbl-ext.v9.txt, 14417.v1.txt, 14417.v11.txt, 
> 14417.v13.txt, 14417.v2.txt, 14417.v21.txt, 14417.v23.txt, 14417.v24.txt, 
> 14417.v25.txt, 14417.v6.txt
>
>
> Currently, incremental backup is based on WAL files. Bulk data loading 
> bypasses WALs for obvious reasons, breaking incremental backups. The only way 
> to continue backups after bulk loading is to create new full backup of a 
> table. This may not be feasible for customers who do bulk loading regularly 
> (say, every day).
> Google doc for design:
> https://docs.google.com/document/d/1ACCLsecHDvzVSasORgqqRNrloGx4mNYIbvAU7lq5lJE



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17110) Add an "Overall Strategy" option(balanced both on table level and server level) to SimpleLoadBalancer

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15694901#comment-15694901
 ] 

Ashish Singhi commented on HBASE-17110:
---

LB is an interface and it may break client codes which are implementing this 
interface. But since it is marked audience private, seems fine to me adding 
another balanceCluster api.

> Add an "Overall Strategy" option(balanced both on table level and server 
> level) to SimpleLoadBalancer
> -
>
> Key: HBASE-17110
> URL: https://issues.apache.org/jira/browse/HBASE-17110
> Project: HBase
>  Issue Type: Improvement
>  Components: Balancer
>Affects Versions: 2.0.0, 1.2.4
>Reporter: Charlie Qiangeng Xu
>Assignee: Charlie Qiangeng Xu
> Attachments: HBASE-17110-V2.patch, HBASE-17110-V3.patch, 
> HBASE-17110-V4.patch, HBASE-17110-V5.patch, HBASE-17110.patch
>
>
> This jira is about an enhancement of simpleLoadBalancer. Here we introduce a 
> new strategy: "bytableOverall" which could be controlled by adding:
> {noformat}
> 
>   hbase.master.loadbalance.bytableOverall
>   true
> 
> {noformat}
> We have been using the strategy on our largest cluster for several months. 
> it's proven to be very helpful and stable, especially, the result is quite 
> visible to the users.
> Here is the reason why it's helpful:
> When operating large scale clusters(our case), some companies still prefer to 
> use {{SimpleLoadBalancer}} due to its simplicity, quick balance plan 
> generation, etc. Current SimpleLoadBalancer has two modes: 
> 1. byTable, which only guarantees that the regions of one table could be 
> uniformly distributed. 
> 2. byCluster, which ignores the distribution within tables and balance the 
> regions all together.
> If the pressures on different tables are different, the first byTable option 
> is the preferable one in most case. Yet, this choice sacrifice the cluster 
> level balance and would cause some servers to have significantly higher load, 
> e.g. 242 regions on server A but 417 regions on server B.(real world stats)
> Consider this case,  a cluster has 3 tables and 4 servers:
> {noformat}
>   server A has 3 regions: table1:1, table2:1, table3:1
>   server B has 3 regions: table1:2, table2:2, table3:2
>   server C has 3 regions: table1:3, table2:3, table3:3
>   server D has 0 regions.
> {noformat}
> From the byTable strategy's perspective, the cluster has already been 
> perfectly balanced on table level. But a perfect status should be like:
> {noformat}
>   server A has 2 regions: table2:1, table3:1
>   server B has 2 regions: table1:2, table3:2
>   server C has 3 regions: table1:3, table2:3, table3:3
>   server D has 2 regions: table1:1, table2:2
> {noformat}
> We can see the server loads change from 3,3,3,0 to 2,2,3,2, while the table1, 
> table2 and table3 still keep balanced.   
> And this is what the new mode "byTableOverall" can achieve.
> Two UTs have been added as well and the last one demonstrates the advantage 
> of the new strategy.
> Also, a onConfigurationChange method has been implemented to hot control the 
> "slop" variable.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17110) Add an "Overall Strategy" option(balanced both on table level and server level) to SimpleLoadBalancer

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15692290#comment-15692290
 ] 

Ashish Singhi commented on HBASE-17110:
---

Give me some time, I'm also checking it.
Thanks.

> Add an "Overall Strategy" option(balanced both on table level and server 
> level) to SimpleLoadBalancer
> -
>
> Key: HBASE-17110
> URL: https://issues.apache.org/jira/browse/HBASE-17110
> Project: HBase
>  Issue Type: Improvement
>  Components: Balancer
>Affects Versions: 2.0.0, 1.2.4
>Reporter: Charlie Qiangeng Xu
>Assignee: Charlie Qiangeng Xu
> Attachments: HBASE-17110-V2.patch, HBASE-17110-V3.patch, 
> HBASE-17110-V4.patch, HBASE-17110-V5.patch, HBASE-17110.patch
>
>
> This jira is about an enhancement of simpleLoadBalancer. Here we introduce a 
> new strategy: "bytableOverall" which could be controlled by adding:
> {noformat}
> 
>   hbase.master.loadbalance.bytableOverall
>   true
> 
> {noformat}
> We have been using the strategy on our largest cluster for several months. 
> it's proven to be very helpful and stable, especially, the result is quite 
> visible to the users.
> Here is the reason why it's helpful:
> When operating large scale clusters(our case), some companies still prefer to 
> use {{SimpleLoadBalancer}} due to its simplicity, quick balance plan 
> generation, etc. Current SimpleLoadBalancer has two modes: 
> 1. byTable, which only guarantees that the regions of one table could be 
> uniformly distributed. 
> 2. byCluster, which ignores the distribution within tables and balance the 
> regions all together.
> If the pressures on different tables are different, the first byTable option 
> is the preferable one in most case. Yet, this choice sacrifice the cluster 
> level balance and would cause some servers to have significantly higher load, 
> e.g. 242 regions on server A but 417 regions on server B.(real world stats)
> Consider this case,  a cluster has 3 tables and 4 servers:
> {noformat}
>   server A has 3 regions: table1:1, table2:1, table3:1
>   server B has 3 regions: table1:2, table2:2, table3:2
>   server C has 3 regions: table1:3, table2:3, table3:3
>   server D has 0 regions.
> {noformat}
> From the byTable strategy's perspective, the cluster has already been 
> perfectly balanced on table level. But a perfect status should be like:
> {noformat}
>   server A has 2 regions: table2:1, table3:1
>   server B has 2 regions: table1:2, table3:2
>   server C has 3 regions: table1:3, table2:3, table3:3
>   server D has 2 regions: table1:1, table2:2
> {noformat}
> We can see the server loads change from 3,3,3,0 to 2,2,3,2, while the table1, 
> table2 and table3 still keep balanced.   
> And this is what the new mode "byTableOverall" can achieve.
> Two UTs have been added as well and the last one demonstrates the advantage 
> of the new strategy.
> Also, a onConfigurationChange method has been implemented to hot control the 
> "slop" variable.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17031) Scanners should check for null start and end rows

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15643203#comment-15643203
 ] 

Ashish Singhi commented on HBASE-17031:
---

Dup of HBASE-16498 ?
[~pankaj2461]

> Scanners should check for null start and end rows
> -
>
> Key: HBASE-17031
> URL: https://issues.apache.org/jira/browse/HBASE-17031
> Project: HBase
>  Issue Type: Bug
>  Components: Scanners
>Reporter: Ashu Pachauri
>Priority: Minor
>
> If a scan is passed with a null start row, it fails very deep in the call 
> stack. We should validate start and end rows for not null before launching 
> the scan.
> Here is the associated jstack:
> {code}
> java.lang.RuntimeException: java.lang.NullPointerException
>   at 
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219)
>   at 
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:326)
>   at 
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:301)
>   at 
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:166)
>   at 
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:161)
>   at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:798)
> Caused by: java.lang.NullPointerException
>   at org.apache.hadoop.hbase.util.Bytes.compareTo(Bytes.java:1225)
>   at 
> org.apache.hadoop.hbase.util.Bytes$ByteArrayComparator.compare(Bytes.java:158)
>   at 
> org.apache.hadoop.hbase.util.Bytes$ByteArrayComparator.compare(Bytes.java:147)
>   at 
> org.apache.hadoop.hbase.types.CopyOnWriteArrayMap$ArrayHolder.find(CopyOnWriteArrayMap.java:892)
>   at 
> org.apache.hadoop.hbase.types.CopyOnWriteArrayMap.floorEntry(CopyOnWriteArrayMap.java:169)
>   at 
> org.apache.hadoop.hbase.client.MetaCache.getCachedLocation(MetaCache.java:79)
>   at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getCachedLocation(ConnectionManager.java:1391)
>   at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1231)
>   at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1183)
>   at 
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:305)
>   at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
>   at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
>   at 
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:211)
>   ... 30 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16910) Avoid NPE when starting StochasticLoadBalancer

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-16910:
--
Fix Version/s: 1.4.0

Thanks for the quick turn around. I have pushed this to branch-1.
Didn't push to branch-1.3, until [~mantonov] says for it.

> Avoid NPE when starting StochasticLoadBalancer
> --
>
> Key: HBASE-16910
> URL: https://issues.apache.org/jira/browse/HBASE-16910
> Project: HBase
>  Issue Type: Bug
>  Components: Balancer
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16910-branch-1.patch, HBASE-16910-v1.patch, 
> HBASE-16910.patch
>
>
> When master start, it initialize StochasticLoadBalancer.
> {code}
> this.balancer.setClusterStatus(getClusterStatus());
> this.balancer.setMasterServices(this);
> {code}
> It first setClusterStatus(), then setMasterService(). But in setClusterStatus 
> method, it use master service which is not initialized. So it will throw NPE.
> {code}
> int tablesCount = isByTable ? services.getTableDescriptors().getAll().size() 
> : 1;
> {code}
> It happens when set isByTable is ture.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16910) Avoid NPE when starting StochasticLoadBalancer

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15604316#comment-15604316
 ] 

Ashish Singhi commented on HBASE-16910:
---

We faced this issue yesterday in one of our test cluster also which is based on 
branch-1.0.
[~zghaobac], can you provide patch for other branches also. If not let me know 
I can prepare the patches.
Ideally bug fix should go to all the running versions.

> Avoid NPE when starting StochasticLoadBalancer
> --
>
> Key: HBASE-16910
> URL: https://issues.apache.org/jira/browse/HBASE-16910
> Project: HBase
>  Issue Type: Bug
>  Components: Balancer
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16910-v1.patch, HBASE-16910.patch
>
>
> When master start, it initialize StochasticLoadBalancer.
> {code}
> this.balancer.setClusterStatus(getClusterStatus());
> this.balancer.setMasterServices(this);
> {code}
> It first setClusterStatus(), then setMasterService(). But in setClusterStatus 
> method, it use master service which is not initialized. So it will throw NPE.
> {code}
> int tablesCount = isByTable ? services.getTableDescriptors().getAll().size() 
> : 1;
> {code}
> It happens when set isByTable is ture.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16724) Snapshot owner can't clone

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-16724:
--
Fix Version/s: 1.4.0

Pushed to branch-1.
On branch-1.3 the modified test in the patch failed. Didn't check further 
branches.
{code}
---
 T E S T S
---
Running org.apache.hadoop.hbase.security.access.TestAccessController
Tests run: 61, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 88.736 sec <<< 
FAILURE! - in org.apache.hadoop.hbase.security.access.TestAccessController
testSnapshot(org.apache.hadoop.hbase.security.access.TestAccessController)  
Time elapsed: 0.035 sec  <<< ERROR!
java.lang.NullPointerException: null
at 
org.apache.hadoop.hbase.snapshot.SnapshotDescriptionUtils.isSnapshotOwner(SnapshotDescriptionUtils.java:364)
at 
org.apache.hadoop.hbase.security.access.AccessController.preCloneSnapshot(AccessController.java:1335)
at 
org.apache.hadoop.hbase.security.access.TestAccessController$70.run(TestAccessController.java:2053)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at 
org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:340)
at 
org.apache.hadoop.hbase.security.access.SecureTestUtil.verifyAllowed(SecureTestUtil.java:177)
at 
org.apache.hadoop.hbase.security.access.SecureTestUtil.verifyAllowed(SecureTestUtil.java:193)
at 
org.apache.hadoop.hbase.security.access.TestAccessController.testSnapshot(TestAccessController.java:2063)


Results :

Tests in error:
  
TestAccessController.testSnapshot:2063->SecureTestUtil.verifyAllowed:193->SecureTestUtil.verifyAllowed:177
 ▒ NullPointer

Tests run: 61, Failures: 0, Errors: 1, Skipped: 0
{code}
Next time please run test locally also before attaching the patch.

> Snapshot owner can't clone
> --
>
> Key: HBASE-16724
> URL: https://issues.apache.org/jira/browse/HBASE-16724
> Project: HBase
>  Issue Type: Bug
>  Components: snapshots
>Affects Versions: 2.0.0
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16724-V2.patch, HBASE-16724-V3.patch, 
> HBASE-16724-branch-1.1.patch, HBASE-16724-branch-1.2.patch, 
> HBASE-16724-branch-1.3.patch, HBASE-16724-branch-1.patch, HBASE-16724.patch
>
>
> Currently only Global admin has the access of cloning a snapshot.
> In AccessController,
> {code}
>   @Override
>   public void preCloneSnapshot(final 
> ObserverContext ctx,
>   final SnapshotDescription snapshot, final HTableDescriptor 
> hTableDescriptor)
>   throws IOException {
> requirePermission(getActiveUser(ctx), "cloneSnapshot " + 
> snapshot.getName(), Action.ADMIN);
>   }
> {code}
> Snapshot owner should be able to  clone it, need to add a check like,
> {code}
> SnapshotDescriptionUtils.isSnapshotOwner(snapshot, user)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16821) Enhance LoadIncrementalHFiles to convey missing hfiles if any

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15574711#comment-15574711
 ] 

Ashish Singhi commented on HBASE-16821:
---

{code}
List missing = run(dirPath, null, tableName);
{code}
You can rename here too on commit.

> Enhance LoadIncrementalHFiles to convey missing hfiles if any
> -
>
> Key: HBASE-16821
> URL: https://issues.apache.org/jira/browse/HBASE-16821
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 16821.v1.txt, 16821.v2.txt
>
>
> When map parameter of run() method is not null:
> {code}
>   public int run(String dirPath, Map map, TableName 
> tableName) throws Exception{
> {code}
> the caller knows the exact files to be bulk loaded.
> This issue is to enhance the run() API so that when certain hfiles turn out 
> to be missing, the return value should indicate the missing files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16821) Enhance LoadIncrementalHFiles to convey missing hfiles if any

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15574708#comment-15574708
 ] 

Ashish Singhi commented on HBASE-16821:
---

OK, got it.
+1

> Enhance LoadIncrementalHFiles to convey missing hfiles if any
> -
>
> Key: HBASE-16821
> URL: https://issues.apache.org/jira/browse/HBASE-16821
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 16821.v1.txt, 16821.v2.txt
>
>
> When map parameter of run() method is not null:
> {code}
>   public int run(String dirPath, Map map, TableName 
> tableName) throws Exception{
> {code}
> the caller knows the exact files to be bulk loaded.
> This issue is to enhance the run() API so that when certain hfiles turn out 
> to be missing, the return value should indicate the missing files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16724) Snapshot owner can't clone

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-16724:
--
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 2.0.0
   Status: Resolved  (was: Patch Available)

Pushed to master branch.
Patch doesn't apply cleanly to branch-1. If it is attached will commit there 
also.
Thanks.

> Snapshot owner can't clone
> --
>
> Key: HBASE-16724
> URL: https://issues.apache.org/jira/browse/HBASE-16724
> Project: HBase
>  Issue Type: Bug
>  Components: snapshots
>Affects Versions: 2.0.0
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Fix For: 2.0.0
>
> Attachments: HBASE-16724-V2.patch, HBASE-16724-V3.patch, 
> HBASE-16724.patch
>
>
> Currently only Global admin has the access of cloning a snapshot.
> In AccessController,
> {code}
>   @Override
>   public void preCloneSnapshot(final 
> ObserverContext ctx,
>   final SnapshotDescription snapshot, final HTableDescriptor 
> hTableDescriptor)
>   throws IOException {
> requirePermission(getActiveUser(ctx), "cloneSnapshot " + 
> snapshot.getName(), Action.ADMIN);
>   }
> {code}
> Snapshot owner should be able to  clone it, need to add a check like,
> {code}
> SnapshotDescriptionUtils.isSnapshotOwner(snapshot, user)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16807) RegionServer will fail to report new active Hmaster until HMaster/RegionServer failover

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-16807:
--
Release Note:   (was: push to master. Thanks all the guys!)

> RegionServer will fail to report new active Hmaster until 
> HMaster/RegionServer failover
> ---
>
> Key: HBASE-16807
> URL: https://issues.apache.org/jira/browse/HBASE-16807
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Fix For: 2.0.0
>
> Attachments: HBASE-16807.patch
>
>
> It's little weird, but it happened in the product environment that few 
> RegionServer missed master znode create notification on master failover. In 
> that case ZooKeeperNodeTracker will not refresh the cached data and 
> MasterAddressTracker will always return old active HM detail to Region server 
> on ServiceException.
> Though We create region server stub on failure but without refreshing the 
> MasterAddressTracker data.
> In HRegionServer.createRegionServerStatusStub()
> {code}
>   boolean refresh = false; // for the first time, use cached data
> RegionServerStatusService.BlockingInterface intf = null;
> boolean interrupted = false;
> try {
>   while (keepLooping()) {
> sn = this.masterAddressTracker.getMasterAddress(refresh);
> if (sn == null) {
>   if (!keepLooping()) {
> // give up with no connection.
> LOG.debug("No master found and cluster is stopped; bailing out");
> return null;
>   }
>   if (System.currentTimeMillis() > (previousLogTime + 1000)) {
> LOG.debug("No master found; retry");
> previousLogTime = System.currentTimeMillis();
>   }
>   refresh = true; // let's try pull it from ZK directly
>   if (sleep(200)) {
> interrupted = true;
>   }
>   continue;
> }
> {code}
> Here we refresh node only when 'sn' is NULL otherwise it will use same cached 
> data. 
> So in above case RegionServer will never report active HMaster successfully 
> until HMaster failover or RegionServer restart.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16653) Backport HBASE-11393 to all branches which support namespace

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571031#comment-15571031
 ] 

Ashish Singhi commented on HBASE-16653:
---

bq. ReplicationPeerConfig class is InterfaceAudience.Public and v3 patch added 
two methods for it. Did this has compatibility issues?
This should be ok.

> Backport HBASE-11393 to all branches which support namespace
> 
>
> Key: HBASE-16653
> URL: https://issues.apache.org/jira/browse/HBASE-16653
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.4.0, 1.0.5, 1.3.1, 0.98.22, 1.1.7, 1.2.4
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 1.4.0
>
> Attachments: HBASE-16653-branch-1-v1.patch, 
> HBASE-16653-branch-1-v2.patch, HBASE-16653-branch-1-v3.patch
>
>
> As HBASE-11386 mentioned, the parse code about replication table-cfs config 
> will be wrong when table name contains namespace and we can only config the 
> default namespace's tables in the peer. It is a bug for all branches which 
> support namespace. HBASE-11393 resolved this by use a pb object but it was 
> only merged to master branch. Other branches still have this problem. I 
> thought we should fix this bug in all branches which support namespace.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15568251#comment-15568251
 ] 

Ashish Singhi commented on HBASE-13153:
---

[~tedyu], the above is handled in the code.
Take a look 
[here|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java#L372-L379].

> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-13153-branch-1-v20.patch, 
> HBASE-13153-branch-1-v21.patch, HBASE-13153-v1.patch, HBASE-13153-v10.patch, 
> HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch, 
> HBASE-13153-v14.patch, HBASE-13153-v15.patch, HBASE-13153-v16.patch, 
> HBASE-13153-v17.patch, HBASE-13153-v18.patch, HBASE-13153-v19.patch, 
> HBASE-13153-v2.patch, HBASE-13153-v20.patch, HBASE-13153-v21.patch, 
> HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch, 
> HBASE-13153-v6.patch, HBASE-13153-v7.patch, HBASE-13153-v8.patch, 
> HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk Load 
> Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load 
> Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG
>
>
> Currently we plan to use HBase Replication feature to deal with disaster 
> tolerance scenario.But we encounter an issue that we will use bulkload very 
> frequently,because bulkload bypass write path, and will not generate WAL, so 
> the data will not be replicated to backup cluster. It's inappropriate to 
> bukload twice both on active cluster and backup cluster. So i advise do some 
> modification to bulkload feature to enable bukload to both active cluster and 
> backup cluster



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HBASE-16663) JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15554397#comment-15554397
 ] 

Ashish Singhi edited comment on HBASE-16663 at 10/7/16 7:35 AM:


[~pankaj2461], after committing HBASE-16723 also I'm not able to get a clean 
test run for the newly added test class. Can you check once.


was (Author: ashish singhi):
[~pankaj2461], after committing HBASE-16723 also I'm able to get clean test run 
for the newly added test class. Can you check once.

> JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
> 
>
> Key: HBASE-16663
> URL: https://issues.apache.org/jira/browse/HBASE-16663
> Project: HBase
>  Issue Type: Bug
>  Components: metrics, security
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16663-0.98.patch, HBASE-16663-V2.patch, 
> HBASE-16663-V3.patch, HBASE-16663-branch-1.patch, HBASE-16663.patch
>
>
> After HBASE-16284, unauthorized user will not able allowed to stop 
> HM/RS/cluster, but while executing "cpHost.preStopMaster()", ConnectorServer 
> will be stopped before AccessController validation.
> hbase-site.xml,
> {noformat}
>  
>   hbase.coprocessor.master.classes
> 
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>  
>   
>   hbase.coprocessor.regionserver.classes
> 
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>   
> {noformat}
> HBaseAdmin.stopMaster(),
> {noformat}
> 2016-09-20 21:12:26,796 INFO  
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000] 
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 21:13:55,380 WARN  
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000] 
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for 
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 21:14:00,495 ERROR 
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000] 
> master.MasterRpcServices: Exception occurred while stopping master
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient 
> permissions for user 'P72981' (global, action=ADMIN)
>   at 
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
>   at 
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
>   at 
> org.apache.hadoop.hbase.security.access.AccessController.preStopMaster(AccessController.java:1297)
>   at 
> org.apache.hadoop.hbase.master.MasterCoprocessorHost$68.call(MasterCoprocessorHost.java:821)
>   at 
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:1188)
>   at 
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.preStopMaster(MasterCoprocessorHost.java:817)
>   at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2352)
>   at 
> org.apache.hadoop.hbase.master.MasterRpcServices.stopMaster(MasterRpcServices.java:1364)
> {noformat}
> HBaseAdmin.stopRegionServer(rs-host-port),
> {noformat}
> 2016-09-20 20:59:01,234 INFO  
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020] 
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 20:59:01,250 WARN  
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020] 
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for 
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 20:59:01,253 WARN  
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020] 
> regionserver.HRegionServer: The region server did not stop
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient 
> permissions for user 'P72981' (global, action=ADMIN)
>   at 
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
>   at 
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
>   at 
> org.apache.hadoop.hbase.security.access.AccessController.preStopRegionServer(AccessController.java:2501)
>   at 
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost$1.call(RegionServerCoprocessorHost.java:84)
>   at 
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.execOperation(RegionServerCoprocessorHost.java:256)
>   at 
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.preStop(RegionServerCoprocessorHost.java:80)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.stop(HRegionServer.java:1905)
>   at 
> 

[jira] [Commented] (HBASE-16663) JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15554397#comment-15554397
 ] 

Ashish Singhi commented on HBASE-16663:
---

[~pankaj2461], after committing HBASE-16723 also I'm able to get clean test run 
for the newly added test class. Can you check once.

> JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
> 
>
> Key: HBASE-16663
> URL: https://issues.apache.org/jira/browse/HBASE-16663
> Project: HBase
>  Issue Type: Bug
>  Components: metrics, security
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16663-0.98.patch, HBASE-16663-V2.patch, 
> HBASE-16663-V3.patch, HBASE-16663-branch-1.patch, HBASE-16663.patch
>
>
> After HBASE-16284, unauthorized user will not able allowed to stop 
> HM/RS/cluster, but while executing "cpHost.preStopMaster()", ConnectorServer 
> will be stopped before AccessController validation.
> hbase-site.xml,
> {noformat}
>  
>   hbase.coprocessor.master.classes
> 
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>  
>   
>   hbase.coprocessor.regionserver.classes
> 
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>   
> {noformat}
> HBaseAdmin.stopMaster(),
> {noformat}
> 2016-09-20 21:12:26,796 INFO  
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000] 
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 21:13:55,380 WARN  
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000] 
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for 
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 21:14:00,495 ERROR 
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000] 
> master.MasterRpcServices: Exception occurred while stopping master
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient 
> permissions for user 'P72981' (global, action=ADMIN)
>   at 
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
>   at 
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
>   at 
> org.apache.hadoop.hbase.security.access.AccessController.preStopMaster(AccessController.java:1297)
>   at 
> org.apache.hadoop.hbase.master.MasterCoprocessorHost$68.call(MasterCoprocessorHost.java:821)
>   at 
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:1188)
>   at 
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.preStopMaster(MasterCoprocessorHost.java:817)
>   at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2352)
>   at 
> org.apache.hadoop.hbase.master.MasterRpcServices.stopMaster(MasterRpcServices.java:1364)
> {noformat}
> HBaseAdmin.stopRegionServer(rs-host-port),
> {noformat}
> 2016-09-20 20:59:01,234 INFO  
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020] 
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 20:59:01,250 WARN  
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020] 
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for 
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 20:59:01,253 WARN  
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020] 
> regionserver.HRegionServer: The region server did not stop
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient 
> permissions for user 'P72981' (global, action=ADMIN)
>   at 
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
>   at 
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
>   at 
> org.apache.hadoop.hbase.security.access.AccessController.preStopRegionServer(AccessController.java:2501)
>   at 
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost$1.call(RegionServerCoprocessorHost.java:84)
>   at 
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.execOperation(RegionServerCoprocessorHost.java:256)
>   at 
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.preStop(RegionServerCoprocessorHost.java:80)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.stop(HRegionServer.java:1905)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.stopServer(RSRpcServices.java:1961)
> {noformat}
> HBaseAdmin.shutdown(),
> {noformat}
> 2016-09-21 12:09:08,259 INFO  
> 

[jira] [Updated] (HBASE-16723) RMI registry is not destroyed after stopping JMX Connector Server

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-16723:
--
Attachment: HBASE-16723-0.98.patch

Attached patch which I committed to 0.98 branch as it was not applying cleanly.

> RMI registry is not destroyed after stopping JMX Connector Server
> -
>
> Key: HBASE-16723
> URL: https://issues.apache.org/jira/browse/HBASE-16723
> Project: HBase
>  Issue Type: Bug
>  Components: metrics
>Affects Versions: 2.0.0, 1.4.0, 1.3.1, 1.2.3, 0.98.22
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.4, 0.98.24
>
> Attachments: HBASE-16723-0.98.patch, HBASE-16723-V2.patch, 
> HBASE-16723-V3.patch, HBASE-16723.patch
>
>
> We are creating RMI registry in JMXListener.startConnectorServer() ,
> {code}
> // Create the RMI registry
> LocateRegistry.createRegistry(rmiRegistryPort);
> {code}
> This registry is never deregistered, should be destoyed after stopping JMX 
> Connector server.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16723) RMI registry is not destroyed after stopping JMX Connector Server

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-16723:
--
Fix Version/s: 0.98.24
   1.2.4
   1.3.1

Pushed to branch-1.3, branch-1.2 & 0.98 also.

> RMI registry is not destroyed after stopping JMX Connector Server
> -
>
> Key: HBASE-16723
> URL: https://issues.apache.org/jira/browse/HBASE-16723
> Project: HBase
>  Issue Type: Bug
>  Components: metrics
>Affects Versions: 2.0.0, 1.4.0, 1.3.1, 1.2.3, 0.98.22
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.4, 0.98.24
>
> Attachments: HBASE-16723-V2.patch, HBASE-16723-V3.patch, 
> HBASE-16723.patch
>
>
> We are creating RMI registry in JMXListener.startConnectorServer() ,
> {code}
> // Create the RMI registry
> LocateRegistry.createRegistry(rmiRegistryPort);
> {code}
> This registry is never deregistered, should be destoyed after stopping JMX 
> Connector server.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16724) Snapshot owner can't clone

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15551831#comment-15551831
 ] 

Ashish Singhi commented on HBASE-16724:
---

What about just updating it to ? {code} cloneSnapshot | 
superuser|global(A)|(SnapshotOwner & TableName matches) {code}

> Snapshot owner can't clone
> --
>
> Key: HBASE-16724
> URL: https://issues.apache.org/jira/browse/HBASE-16724
> Project: HBase
>  Issue Type: Bug
>  Components: snapshots
>Affects Versions: 2.0.0
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Attachments: HBASE-16724-V2.patch, HBASE-16724.patch
>
>
> Currently only Global admin has the access of cloning a snapshot.
> In AccessController,
> {code}
>   @Override
>   public void preCloneSnapshot(final 
> ObserverContext ctx,
>   final SnapshotDescription snapshot, final HTableDescriptor 
> hTableDescriptor)
>   throws IOException {
> requirePermission(getActiveUser(ctx), "cloneSnapshot " + 
> snapshot.getName(), Action.ADMIN);
>   }
> {code}
> Snapshot owner should be able to  clone it, need to add a check like,
> {code}
> SnapshotDescriptionUtils.isSnapshotOwner(snapshot, user)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16724) Snapshot owner can't clone

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15548048#comment-15548048
 ] 

Ashish Singhi commented on HBASE-16724:
---

{code}
| cloneSnapshot | superuser\|global(A)\|SnapshotOwner
{code}
Not just snapshot owner it should be table owner also. So it should be {code} 
superuser\|global(A)\|SnapshotOwner & TableOwner{code}


> Snapshot owner can't clone
> --
>
> Key: HBASE-16724
> URL: https://issues.apache.org/jira/browse/HBASE-16724
> Project: HBase
>  Issue Type: Bug
>  Components: snapshots
>Affects Versions: 2.0.0
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Attachments: HBASE-16724-V2.patch, HBASE-16724.patch
>
>
> Currently only Global admin has the access of cloning a snapshot.
> In AccessController,
> {code}
>   @Override
>   public void preCloneSnapshot(final 
> ObserverContext ctx,
>   final SnapshotDescription snapshot, final HTableDescriptor 
> hTableDescriptor)
>   throws IOException {
> requirePermission(getActiveUser(ctx), "cloneSnapshot " + 
> snapshot.getName(), Action.ADMIN);
>   }
> {code}
> Snapshot owner should be able to  clone it, need to add a check like,
> {code}
> SnapshotDescriptionUtils.isSnapshotOwner(snapshot, user)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16724) Snapshot owner can't clone

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15545862#comment-15545862
 ] 

Ashish Singhi commented on HBASE-16724:
---

Need to update the ACL matrix table in HBase book also.

> Snapshot owner can't clone
> --
>
> Key: HBASE-16724
> URL: https://issues.apache.org/jira/browse/HBASE-16724
> Project: HBase
>  Issue Type: Bug
>  Components: snapshots
>Affects Versions: 2.0.0
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Attachments: HBASE-16724.patch
>
>
> Currently only Global admin has the access of cloning a snapshot.
> In AccessController,
> {code}
>   @Override
>   public void preCloneSnapshot(final 
> ObserverContext ctx,
>   final SnapshotDescription snapshot, final HTableDescriptor 
> hTableDescriptor)
>   throws IOException {
> requirePermission(getActiveUser(ctx), "cloneSnapshot " + 
> snapshot.getName(), Action.ADMIN);
>   }
> {code}
> Snapshot owner should be able to  clone it, need to add a check like,
> {code}
> SnapshotDescriptionUtils.isSnapshotOwner(snapshot, user)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-16493) MapReduce counters not updated with ScanMetrics

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi resolved HBASE-16493.
---
Resolution: Duplicate

Duplicate of HBASE-16678

> MapReduce counters not updated with ScanMetrics 
> 
>
> Key: HBASE-16493
> URL: https://issues.apache.org/jira/browse/HBASE-16493
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Affects Versions: 1.0.0, 2.0.0, 1.1.0
>Reporter: Jacobo Coll
>
> ScanMetrics were introduced in the [HBASE-4145]. These metrics were able to 
> work even in a parallel environment such as MapReduce.
> The TableRecordReader creates a Scanner with a copy of the given "scan", 
> called "currentScan". The ScanMetrics are captured by the Scanner and 
> modifies the given Scan instance, "currentScan". The TableRecordReader, after 
> reading the last value, updates the job counters to aggregate them. 
> But since [HBASE-13030], the TableRecordReader reads the scanMetrics from the 
> object "scan" instead of using the "currentScan"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16723) RMI registry is not destroyed after stopping JMX Connector Server

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15529419#comment-15529419
 ] 

Ashish Singhi commented on HBASE-16723:
---

+1

> RMI registry is not destroyed after stopping JMX Connector Server
> -
>
> Key: HBASE-16723
> URL: https://issues.apache.org/jira/browse/HBASE-16723
> Project: HBase
>  Issue Type: Bug
>  Components: metrics
>Affects Versions: 2.0.0, 1.4.0, 1.3.1, 1.2.3, 0.98.22
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Fix For: 2.0.0
>
> Attachments: HBASE-16723-V2.patch, HBASE-16723.patch
>
>
> We are creating RMI registry in JMXListener.startConnectorServer() ,
> {code}
> // Create the RMI registry
> LocateRegistry.createRegistry(rmiRegistryPort);
> {code}
> This registry is never deregistered, should be destoyed after stopping JMX 
> Connector server.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16724) Snapshot owner can't clone

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15529416#comment-15529416
 ] 

Ashish Singhi commented on HBASE-16724:
---

Scenario:
user 'user1' (have admin and create rights on the namespace in which it create 
the table below)
1. Create a table and takes the snapshot
2. Disable and drops the table
3. Performs restore_snapshot, which fails
Though there are sufficient permissions to the user 'user1' to execute 
restore_snapshot as per [ACL 
Matric|https://hbase.apache.org/book.html#appendix_acl_matrix] the access is 
denied.

Currently when we restore a snapshot if the table that doesn't exist then 
internally it will clone the snapshot which requires either superuser of global 
admin permission only as per the ACL matrix and user1 lacks them, so it fails. 
But the end user doesn't know about internal behavior.

So we think clone snapshot should also work with same permissions as 
restore_snapshot.

WDYT [~mbertozzi]/Others ?


> Snapshot owner can't clone
> --
>
> Key: HBASE-16724
> URL: https://issues.apache.org/jira/browse/HBASE-16724
> Project: HBase
>  Issue Type: Bug
>  Components: snapshots
>Affects Versions: 2.0.0
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>
> Currently only Global admin has the access of cloning a snapshot.
> In AccessController,
> {code}
>   @Override
>   public void preCloneSnapshot(final 
> ObserverContext ctx,
>   final SnapshotDescription snapshot, final HTableDescriptor 
> hTableDescriptor)
>   throws IOException {
> requirePermission(getActiveUser(ctx), "cloneSnapshot " + 
> snapshot.getName(), Action.ADMIN);
>   }
> {code}
> Snapshot owner should be able to  clone it, need to add a check like,
> {code}
> SnapshotDescriptionUtils.isSnapshotOwner(snapshot, user)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16723) RMI registry is not destroyed after stopping JMX Connector Server

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15529382#comment-15529382
 ] 

Ashish Singhi commented on HBASE-16723:
---

I think we should deregister RMI registry when there is an error to start 
JMXConnectorServer also. WDYT ?

> RMI registry is not destroyed after stopping JMX Connector Server
> -
>
> Key: HBASE-16723
> URL: https://issues.apache.org/jira/browse/HBASE-16723
> Project: HBase
>  Issue Type: Bug
>  Components: metrics
>Affects Versions: 2.0.0, 1.4.0, 1.3.1, 1.2.3, 0.98.22
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Fix For: 2.0.0
>
> Attachments: HBASE-16723.patch
>
>
> We are creating RMI registry in JMXListener.startConnectorServer() ,
> {code}
> // Create the RMI registry
> LocateRegistry.createRegistry(rmiRegistryPort);
> {code}
> This registry is never deregistered, should be destoyed after stopping JMX 
> Connector server.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16663) JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525283#comment-15525283
 ] 

Ashish Singhi commented on HBASE-16663:
---

[~pankaj2461], any update here ?
This is good finding and we will need to address this in the next release of 
branch-1.2 and 0.98.
/cc [~busbey], [~apurtell]

> JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
> 
>
> Key: HBASE-16663
> URL: https://issues.apache.org/jira/browse/HBASE-16663
> Project: HBase
>  Issue Type: Bug
>  Components: metrics, security
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
>
> After HBASE-16284, unauthorized user will not able allowed to stop 
> HM/RS/cluster, but while executing "cpHost.preStopMaster()", ConnectorServer 
> stopped will be stopped before AccessController validation.
> hbase-site.xml,
> {noformat}
>  
>   hbase.coprocessor.master.classes
> 
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>  
>   
>   hbase.coprocessor.regionserver.classes
> 
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>   
> {noformat}
> HBaseAdmin.stopMaster(),
> {noformat}
> 2016-09-20 21:12:26,796 INFO  
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000] 
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 21:13:55,380 WARN  
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000] 
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for 
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 21:14:00,495 ERROR 
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000] 
> master.MasterRpcServices: Exception occurred while stopping master
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient 
> permissions for user 'P72981' (global, action=ADMIN)
>   at 
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
>   at 
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
>   at 
> org.apache.hadoop.hbase.security.access.AccessController.preStopMaster(AccessController.java:1297)
>   at 
> org.apache.hadoop.hbase.master.MasterCoprocessorHost$68.call(MasterCoprocessorHost.java:821)
>   at 
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:1188)
>   at 
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.preStopMaster(MasterCoprocessorHost.java:817)
>   at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2352)
>   at 
> org.apache.hadoop.hbase.master.MasterRpcServices.stopMaster(MasterRpcServices.java:1364)
> {noformat}
> HBaseAdmin.stopRegionServer(rs-host-port),
> {noformat}
> 2016-09-20 20:59:01,234 INFO  
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020] 
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 20:59:01,250 WARN  
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020] 
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for 
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 20:59:01,253 WARN  
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020] 
> regionserver.HRegionServer: The region server did not stop
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient 
> permissions for user 'P72981' (global, action=ADMIN)
>   at 
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
>   at 
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
>   at 
> org.apache.hadoop.hbase.security.access.AccessController.preStopRegionServer(AccessController.java:2501)
>   at 
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost$1.call(RegionServerCoprocessorHost.java:84)
>   at 
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.execOperation(RegionServerCoprocessorHost.java:256)
>   at 
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.preStop(RegionServerCoprocessorHost.java:80)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.stop(HRegionServer.java:1905)
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.stopServer(RSRpcServices.java:1961)
> {noformat}
> HBaseAdmin.shutdown(),
> {noformat}
> 2016-09-21 12:09:08,259 INFO  
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000] 
> master.MasterRpcServices: Client=P72981//10.18.248.96 shutdown
> 2016-09-21 12:09:08,261 INFO  
> 

[jira] [Commented] (HBASE-16672) Add option for bulk load to copy hfile(s) instead of renaming

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15519242#comment-15519242
 ] 

Ashish Singhi commented on HBASE-16672:
---

+1 pending QA run.
{code}
/**
536* Create a protocol buffer bulk load request
537*
538* @param familyPaths
539* @param regionName
540* @param assignSeqNum
541* @return a bulk load request
542*/
543   public static BulkLoadHFileRequest buildBulkLoadHFileRequest(
544   final List> familyPaths,
545   final byte[] regionName, boolean assignSeqNum,
546   final Token userToken, final String bulkToken, boolean 
copyFiles) {
{code}
Add other params also, can do it on commit if no other review comments also 
please add release note to this.

> Add option for bulk load to copy hfile(s) instead of renaming
> -
>
> Key: HBASE-16672
> URL: https://issues.apache.org/jira/browse/HBASE-16672
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 16672.v1.txt, 16672.v10.txt, 16672.v2.txt, 16672.v3.txt, 
> 16672.v4.txt, 16672.v5.txt, 16672.v6.txt, 16672.v7.txt, 16672.v8.txt, 
> 16672.v9.txt
>
>
> This is related to HBASE-14417, to support incrementally restoring to 
> multiple destinations, this issue adds option which would always copy 
> hfile(s) during bulk load.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16672) Add option for bulk load to copy hfile(s) instead of renaming

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15519123#comment-15519123
 ] 

Ashish Singhi commented on HBASE-16672:
---

{code}
+finalPath = bulkLoadListener.prepareBulkLoad(familyName, path, 
copyFile);
+copyFile = false;
{code}

Why we are changing the value of copyFile here ? If it was true then we set 
here to false so the next file in the for loop will not be copied, correct ?

> Add option for bulk load to copy hfile(s) instead of renaming
> -
>
> Key: HBASE-16672
> URL: https://issues.apache.org/jira/browse/HBASE-16672
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 16672.v1.txt, 16672.v2.txt, 16672.v3.txt, 16672.v4.txt, 
> 16672.v5.txt, 16672.v6.txt, 16672.v7.txt, 16672.v8.txt
>
>
> This is related to HBASE-14417, to support incrementally restoring to 
> multiple destinations, this issue adds option which would always copy 
> hfile(s) during bulk load.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)