[jira] [Commented] (HBASE-17125) Inconsistent result when use filter to read data

[Guanghao Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055179#comment-16055179
 ] 

Guanghao Zhang commented on HBASE-17125:


The findbugs and whitespace were introduced by generated protobuf. And the 
failed tests were not rleated. They passed locally. Thanks.

> Inconsistent result when use filter to read data
> 
>
> Key: HBASE-17125
> URL: https://issues.apache.org/jira/browse/HBASE-17125
> Project: HBase
>  Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: example.diff, HBASE-17125.master.001.patch, 
> HBASE-17125.master.002.patch, HBASE-17125.master.002.patch, 
> HBASE-17125.master.003.patch, HBASE-17125.master.004.patch, 
> HBASE-17125.master.005.patch, HBASE-17125.master.006.patch, 
> HBASE-17125.master.007.patch, HBASE-17125.master.008.patch, 
> HBASE-17125.master.009.patch, HBASE-17125.master.009.patch, 
> HBASE-17125.master.010.patch, HBASE-17125.master.011.patch
>
>
> Assume a cloumn's max versions is 3, then we write 4 versions of this column. 
> The oldest version doesn't remove immediately. But from the user view, the 
> oldest version has gone. When user use a filter to query, if the filter skip 
> a new version, then the oldest version will be seen again. But after compact 
> the region, then the oldest version will never been seen. So it is weird for 
> user. The query will get inconsistent result before and after region 
> compaction.
> The reason is matchColumn method of UserScanQueryMatcher. It first check the 
> cell by filter, then check the number of versions needed. So if the filter 
> skip the new version, then the oldest version will be seen again when it is 
> not removed.
> Have a discussion offline with [~Apache9] and [~fenghh], now we have two 
> solution for this problem. The first idea is check the number of versions 
> first, then check the cell by filter. As the comment of setFilter, the filter 
> is called after all tests for ttl, column match, deletes and max versions 
> have been run.
> {code}
>   /**
>* Apply the specified server-side filter when performing the Query.
>* Only {@link Filter#filterKeyValue(Cell)} is called AFTER all tests
>* for ttl, column match, deletes and max versions have been run.
>* @param filter filter to run on the server
>* @return this for invocation chaining
>*/
>   public Query setFilter(Filter filter) {
> this.filter = filter;
> return this;
>   }
> {code}
> But this idea has another problem, if a column's max version is 5 and the 
> user query only need 3 versions. It first check the version's number, then 
> check the cell by filter. So the cells number of the result may less than 3. 
> But there are 2 versions which don't read anymore.
> So the second idea has three steps.
> 1. check by the max versions of this column
> 2. check the kv by filter
> 3. check the versions which user need.
> But this will lead the ScanQueryMatcher more complicated. And this will break 
> the javadoc of Query.setFilter.
> Now we don't have a final solution for this problem. Suggestions are welcomed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18233) We shouldn't wait for readlock in doMiniBatchMutation in case of deadlock

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055170#comment-16055170
 ] 

stack commented on HBASE-18233:
---

You fellows see the [~jerryhe] notes over on HBASE-18144 ? Also on HBASE-18144 
I was thinking a unit test easy enough to make given how clean the scenario is 
but sounds like the scenario can be reproduced with the unit test attached 
here? Thanks.

> We shouldn't wait for readlock in doMiniBatchMutation in case of deadlock
> -
>
> Key: HBASE-18233
> URL: https://issues.apache.org/jira/browse/HBASE-18233
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.7
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-18233-branch-1.2.patch, 
> HBASE-18233-branch-1.2.v2.patch, HBASE-18233-branch-1.2.v3.patch
>
>
> Please refer to the discuss in HBASE-18144
> https://issues.apache.org/jira/browse/HBASE-18144?focusedCommentId=16051701=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16051701



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18240) Add hbase-auxillary, a project with hbase utility including an hbase-shaded-thirdparty module with guava, netty, etc.

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055155#comment-16055155
 ] 

stack commented on HBASE-18240:
---

001 moves hbase-common to use the guava 22.0 that is in hbase-thirdparty-shaded 
module.

Of note is that guava 22.0 requires a few small changes and some pom-ery is 
necessary undoing transitive includes. Otherwise, its just a bit of mass find 
and replace.

> Add hbase-auxillary, a project with hbase utility including an 
> hbase-shaded-thirdparty module with guava, netty, etc.
> -
>
> Key: HBASE-18240
> URL: https://issues.apache.org/jira/browse/HBASE-18240
> Project: HBase
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0
>
> Attachments: HBASE-18240.master.001.patch, hbase-auxillary.tgz
>
>
> This issue is about adding a new related project to host hbase auxillary 
> utility. In this new project, the first thing we'd add is a module to host 
> shaded versions of third party libraries.
> This task comes of discussion held here 
> http://apache-hbase.679495.n3.nabble.com/DISCUSS-More-Shading-td4083025.html 
> where one conclusion of the DISCUSSION was "... pushing this part forward 
> with some code is the next logical step. Seems to be consensus about taking 
> our known internal dependencies and performing this shade magic."



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18212) In Standalone mode with local filesystem HBase logs Warning message:Failed to invoke 'unbuffer' method in class class org.apache.hadoop.fs.FSDataInputStream

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055154#comment-16055154
 ] 

Ashish Singhi commented on HBASE-18212:
---

Thanks Sean.
I was waiting for a binding +1 to commit. Since you are running tests and 
waiting for the result, I will leave it to you for commit.

> In Standalone mode with local filesystem HBase logs Warning message:Failed to 
> invoke 'unbuffer' method in class class org.apache.hadoop.fs.FSDataInputStream
> 
>
> Key: HBASE-18212
> URL: https://issues.apache.org/jira/browse/HBASE-18212
> Project: HBase
>  Issue Type: Bug
>  Components: Operability
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Umesh Agashe
>Assignee: Ashish Singhi
>Priority: Minor
> Attachments: HBASE-18212.patch
>
>
> New users may get nervous after seeing following warning level log messages 
> (considering new users will most likely run HBase in Standalone mode first):
> {code}
> WARN  [MemStoreFlusher.1] io.FSDataInputStreamWrapper: Failed to invoke 
> 'unbuffer' method in class class org.apache.hadoop.fs.FSDataInputStream . So 
> there may be a TCP socket connection left open in CLOSE_WAIT state.
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18240) Add hbase-auxillary, a project with hbase utility including an hbase-shaded-thirdparty module with guava, netty, etc.

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-18240:
--
Attachment: HBASE-18240.master.001.patch

> Add hbase-auxillary, a project with hbase utility including an 
> hbase-shaded-thirdparty module with guava, netty, etc.
> -
>
> Key: HBASE-18240
> URL: https://issues.apache.org/jira/browse/HBASE-18240
> Project: HBase
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0
>
> Attachments: HBASE-18240.master.001.patch, hbase-auxillary.tgz
>
>
> This issue is about adding a new related project to host hbase auxillary 
> utility. In this new project, the first thing we'd add is a module to host 
> shaded versions of third party libraries.
> This task comes of discussion held here 
> http://apache-hbase.679495.n3.nabble.com/DISCUSS-More-Shading-td4083025.html 
> where one conclusion of the DISCUSSION was "... pushing this part forward 
> with some code is the next logical step. Seems to be consensus about taking 
> our known internal dependencies and performing this shade magic."



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18167) OfflineMetaRepair tool may cause HMaster abort always

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055148#comment-16055148
 ] 

Ashish Singhi commented on HBASE-18167:
---

Minor comments:
1.
{code}
 LOG.warn(
  "Couldn't assign the regions as SSH was not finished 
within the specified time in hbase.master.initializationmonitor.timeout 
parameter.");
{code}
Better to log the value of configuration instead of property name. 

2. Delete the table and namespace in test in the finally block.

> OfflineMetaRepair tool may cause HMaster abort always
> -
>
> Key: HBASE-18167
> URL: https://issues.apache.org/jira/browse/HBASE-18167
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 1.4.0, 1.3.1, 1.3.2
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 1.4.0
>
> Attachments: HBASE-18167-branch-1.patch
>
>
> In the production environment, we met a weird scenario where some Meta table 
> HFile blocks were missing due to some reason.
> To recover the environment we tried to rebuild the meta using 
> OfflineMetaRepair tool and restart the cluster, but HMaster couldn't finish 
> it's initialization. It always timed out as namespace table region was never 
> assigned.
> Steps to reproduce
> ==
> 1. Assign meta table region to HMaster (it can be on any RS, just to 
> reproduce the  scenario)
> {noformat}
>   
> hbase.balancer.tablesOnMaster
> hbase:meta
> 
> {noformat}
> 2. Start HMaster and RegionServer
> 2. Create two namespace, say "ns1" & "ns2"
> 3. Create two tables "ns1:t1' & "ns2:t1'
> 4. flush 'hbase:meta"
> 5. Stop HMaster (graceful shutdown)
> 6. Kill -9 RegionServer (Abnormal shutdown)
> 7. Run OfflineMetaRepair as follows,
> {noformat}
>   hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair -fix
> {noformat}
> 8. Restart HMaster and RegionServer
> 9. HMaster will never be able to finish its initialization and abort always 
> with below message,
> {code}
> 2017-06-06 15:11:07,582 FATAL [Hostname:16000.activeMasterManager] 
> master.HMaster: Unhandled exception. Starting shutdown.
> java.io.IOException: Timedout 12ms waiting for namespace table to be 
> assigned
> at 
> org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:98)
> at 
> org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:1054)
> at 
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:848)
> at org.apache.hadoop.hbase.master.HMaster.access$600(HMaster.java:199)
> at org.apache.hadoop.hbase.master.HMaster$2.run(HMaster.java:1871)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Root cause
> ==
> 1. During HM start up AM assumes that it's a failover scenario based on the 
> existing old WAL files, so SSH/SCP will split WAL files and assign the 
> holding regions. 
> 2. During SSH/SCP it retrieves the server holding regions from meta/AM's 
> in-memory-state, but meta only had "regioninfo" entry (as already rebuild by 
> OfflineMetaRepair). So empty region will be returned and it wont trigger any 
> assignment.
> 3. HMaster which is waiting for namespace table to be assigned will timeout 
> and abort always.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-17125) Inconsistent result when use filter to read data

[Guanghao Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055141#comment-16055141
 ] 

Guanghao Zhang commented on HBASE-17125:


bq. which test in the patch exercises the above scenario ?
The SpecifiedNumVersionsColumnFilter control how many versions column will be 
returned. Add a unit test 
TestFromClientSide#testSpecifiedNumVersionsColumnFilter for it. Thanks.

> Inconsistent result when use filter to read data
> 
>
> Key: HBASE-17125
> URL: https://issues.apache.org/jira/browse/HBASE-17125
> Project: HBase
>  Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: example.diff, HBASE-17125.master.001.patch, 
> HBASE-17125.master.002.patch, HBASE-17125.master.002.patch, 
> HBASE-17125.master.003.patch, HBASE-17125.master.004.patch, 
> HBASE-17125.master.005.patch, HBASE-17125.master.006.patch, 
> HBASE-17125.master.007.patch, HBASE-17125.master.008.patch, 
> HBASE-17125.master.009.patch, HBASE-17125.master.009.patch, 
> HBASE-17125.master.010.patch, HBASE-17125.master.011.patch
>
>
> Assume a cloumn's max versions is 3, then we write 4 versions of this column. 
> The oldest version doesn't remove immediately. But from the user view, the 
> oldest version has gone. When user use a filter to query, if the filter skip 
> a new version, then the oldest version will be seen again. But after compact 
> the region, then the oldest version will never been seen. So it is weird for 
> user. The query will get inconsistent result before and after region 
> compaction.
> The reason is matchColumn method of UserScanQueryMatcher. It first check the 
> cell by filter, then check the number of versions needed. So if the filter 
> skip the new version, then the oldest version will be seen again when it is 
> not removed.
> Have a discussion offline with [~Apache9] and [~fenghh], now we have two 
> solution for this problem. The first idea is check the number of versions 
> first, then check the cell by filter. As the comment of setFilter, the filter 
> is called after all tests for ttl, column match, deletes and max versions 
> have been run.
> {code}
>   /**
>* Apply the specified server-side filter when performing the Query.
>* Only {@link Filter#filterKeyValue(Cell)} is called AFTER all tests
>* for ttl, column match, deletes and max versions have been run.
>* @param filter filter to run on the server
>* @return this for invocation chaining
>*/
>   public Query setFilter(Filter filter) {
> this.filter = filter;
> return this;
>   }
> {code}
> But this idea has another problem, if a column's max version is 5 and the 
> user query only need 3 versions. It first check the version's number, then 
> check the cell by filter. So the cells number of the result may less than 3. 
> But there are 2 versions which don't read anymore.
> So the second idea has three steps.
> 1. check by the max versions of this column
> 2. check the kv by filter
> 3. check the versions which user need.
> But this will lead the ScanQueryMatcher more complicated. And this will break 
> the javadoc of Query.setFilter.
> Now we don't have a final solution for this problem. Suggestions are welcomed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18234) Revisit the async admin api

[Guanghao Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guanghao Zhang updated HBASE-18234:
---
Description: 
1. Update the balance method name. 
balancer -> balance
setBalancerRunning -> setBalancerOn
isBalancerEnabled -> isBalancerOn
2. Use HRegionLocation instead of Pair
3. Remove the closeRegionWithEncodedRegionName method. Because all other api 
can handle region name or encoded region name both. So don't need a method for 
encoded name.
4. Unify the region name parameter's type to byte[]. And region name may be 
full name or encoded name.
5. Unify the server name parameter's type to ServerName. For smoe api, it 
support null for server name. So use Optional instead.
6. Unify the table name parameter's type to TableName.

  was:
1. Update the balance method name. 
balancer -> balance
setBalancerRunning -> setBalancerOn
isBalancerEnabled -> isBalancerOn
2. Use HRegionLocation instead of Pair
3. Remove the closeRegionWithEncodedRegionName method. Because all other api 
can handle region name or encoded region name both. So don't need a method for 
encoded name.


> Revisit the async admin api
> ---
>
> Key: HBASE-18234
> URL: https://issues.apache.org/jira/browse/HBASE-18234
> Project: HBase
>  Issue Type: Sub-task
>  Components: Client
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0, 3.0.0
>
> Attachments: HBASE-18234.master.001.patch, 
> HBASE-18234.master.002.patch
>
>
> 1. Update the balance method name. 
> balancer -> balance
> setBalancerRunning -> setBalancerOn
> isBalancerEnabled -> isBalancerOn
> 2. Use HRegionLocation instead of Pair
> 3. Remove the closeRegionWithEncodedRegionName method. Because all other api 
> can handle region name or encoded region name both. So don't need a method 
> for encoded name.
> 4. Unify the region name parameter's type to byte[]. And region name may be 
> full name or encoded name.
> 5. Unify the server name parameter's type to ServerName. For smoe api, it 
> support null for server name. So use Optional instead.
> 6. Unify the table name parameter's type to TableName.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18233) We shouldn't wait for readlock in doMiniBatchMutation in case of deadlock

[Allan Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055135#comment-16055135
 ] 

Allan Yang commented on HBASE-18233:


Added some comments in patch v3, thanks for your advice, [~stack], [~carp84].

> We shouldn't wait for readlock in doMiniBatchMutation in case of deadlock
> -
>
> Key: HBASE-18233
> URL: https://issues.apache.org/jira/browse/HBASE-18233
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.7
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-18233-branch-1.2.patch, 
> HBASE-18233-branch-1.2.v2.patch, HBASE-18233-branch-1.2.v3.patch
>
>
> Please refer to the discuss in HBASE-18144
> https://issues.apache.org/jira/browse/HBASE-18144?focusedCommentId=16051701=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16051701



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18233) We shouldn't wait for readlock in doMiniBatchMutation in case of deadlock

[Allan Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allan Yang updated HBASE-18233:
---
Attachment: HBASE-18233-branch-1.2.v3.patch

> We shouldn't wait for readlock in doMiniBatchMutation in case of deadlock
> -
>
> Key: HBASE-18233
> URL: https://issues.apache.org/jira/browse/HBASE-18233
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.7
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-18233-branch-1.2.patch, 
> HBASE-18233-branch-1.2.v2.patch, HBASE-18233-branch-1.2.v3.patch
>
>
> Please refer to the discuss in HBASE-18144
> https://issues.apache.org/jira/browse/HBASE-18144?focusedCommentId=16051701=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16051701



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18233) We shouldn't wait for readlock in doMiniBatchMutation in case of deadlock

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055129#comment-16055129
 ] 

stack commented on HBASE-18233:
---

Oh, good to know that the test can repro the 30 second timeout.

> We shouldn't wait for readlock in doMiniBatchMutation in case of deadlock
> -
>
> Key: HBASE-18233
> URL: https://issues.apache.org/jira/browse/HBASE-18233
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.7
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-18233-branch-1.2.patch, 
> HBASE-18233-branch-1.2.v2.patch
>
>
> Please refer to the discuss in HBASE-18144
> https://issues.apache.org/jira/browse/HBASE-18144?focusedCommentId=16051701=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16051701



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (HBASE-18233) We shouldn't wait for readlock in doMiniBatchMutation in case of deadlock

[Allan Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055106#comment-16055106
 ] 

Allan Yang edited comment on HBASE-18233 at 6/20/17 4:35 AM:
-

{quote}
Why (I just skimmed the test). This is with sort in place? Why we get to 30 
seconds waiting on a lock?
{quote}
We haven't commit HBASE-17924 to branch-1.2 yet, so we are not sorting batches 
here. Sorry that I may have made some confusion here, [~stack].


was (Author: allan163):
{quote}
Why (I just skimmed the test). This is with sort in place? Why we get to 30 
seconds waiting on a lock?
{quote}
We haven't commit HBASE-17924 to branch-1.2 yet, so we are not sorting batches 
here

> We shouldn't wait for readlock in doMiniBatchMutation in case of deadlock
> -
>
> Key: HBASE-18233
> URL: https://issues.apache.org/jira/browse/HBASE-18233
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.7
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-18233-branch-1.2.patch, 
> HBASE-18233-branch-1.2.v2.patch
>
>
> Please refer to the discuss in HBASE-18144
> https://issues.apache.org/jira/browse/HBASE-18144?focusedCommentId=16051701=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16051701



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18240) Add hbase-auxillary, a project with hbase utility including an hbase-shaded-thirdparty module with guava, netty, etc.

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055125#comment-16055125
 ] 

stack commented on HBASE-18240:
---

s/3rdparty/thirdparty/ in above.

> Add hbase-auxillary, a project with hbase utility including an 
> hbase-shaded-thirdparty module with guava, netty, etc.
> -
>
> Key: HBASE-18240
> URL: https://issues.apache.org/jira/browse/HBASE-18240
> Project: HBase
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0
>
> Attachments: hbase-auxillary.tgz
>
>
> This issue is about adding a new related project to host hbase auxillary 
> utility. In this new project, the first thing we'd add is a module to host 
> shaded versions of third party libraries.
> This task comes of discussion held here 
> http://apache-hbase.679495.n3.nabble.com/DISCUSS-More-Shading-td4083025.html 
> where one conclusion of the DISCUSSION was "... pushing this part forward 
> with some code is the next logical step. Seems to be consensus about taking 
> our known internal dependencies and performing this shade magic."



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18240) Add hbase-auxillary, a project with hbase utility including an hbase-shaded-thirdparty module with guava, netty, etc.

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-18240:
--
Attachment: hbase-auxillary.tgz

What I have so far.

Currently working on using it in mother project just to show it works.

> Add hbase-auxillary, a project with hbase utility including an 
> hbase-shaded-thirdparty module with guava, netty, etc.
> -
>
> Key: HBASE-18240
> URL: https://issues.apache.org/jira/browse/HBASE-18240
> Project: HBase
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0
>
> Attachments: hbase-auxillary.tgz
>
>
> This issue is about adding a new related project to host hbase auxillary 
> utility. In this new project, the first thing we'd add is a module to host 
> shaded versions of third party libraries.
> This task comes of discussion held here 
> http://apache-hbase.679495.n3.nabble.com/DISCUSS-More-Shading-td4083025.html 
> where one conclusion of the DISCUSSION was "... pushing this part forward 
> with some code is the next logical step. Seems to be consensus about taking 
> our known internal dependencies and performing this shade magic."



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18240) Add hbase-auxillary, a project with hbase utility including an hbase-shaded-thirdparty module with guava, netty, etc.

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055121#comment-16055121
 ] 

stack commented on HBASE-18240:
---

[~busbey] Help would be appreciated especially if you've been this route before.

I'm thinking new repo called hbase-auxillary. That ok? Was thinking we could 
host various little related projects here with the first one being 3rdparty 
shaded.

I'd have a root project and then a first module called hbase-3rdparty-shaded. 
This module would produce shaded and relocated 3rd party libs.

groupid = org.apache.hbase.auxillary

Relocation would be from com.google.* to 
org.apache.hbase.3rdparty.shaded.com.google.* etc.

Let me attach what I have currently.

But yeah, request for repo appreciated.

> Add hbase-auxillary, a project with hbase utility including an 
> hbase-shaded-thirdparty module with guava, netty, etc.
> -
>
> Key: HBASE-18240
> URL: https://issues.apache.org/jira/browse/HBASE-18240
> Project: HBase
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0
>
>
> This issue is about adding a new related project to host hbase auxillary 
> utility. In this new project, the first thing we'd add is a module to host 
> shaded versions of third party libraries.
> This task comes of discussion held here 
> http://apache-hbase.679495.n3.nabble.com/DISCUSS-More-Shading-td4083025.html 
> where one conclusion of the DISCUSSION was "... pushing this part forward 
> with some code is the next logical step. Seems to be consensus about taking 
> our known internal dependencies and performing this shade magic."



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18233) We shouldn't wait for readlock in doMiniBatchMutation in case of deadlock

[Yu Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055119#comment-16055119
 ] 

Yu Li commented on HBASE-18233:
---

Thanks [~stack] for the questions, maybe we should mention benefits of the 
change (fail fast to reduce contention and prevent possible deadlocks) into 
JIRA description.

And I second the suggestion to add more comments in codes to make it easier to 
understand.

Thanks [~allan163] for the refinements and fix in the new patch, lgtm.

> We shouldn't wait for readlock in doMiniBatchMutation in case of deadlock
> -
>
> Key: HBASE-18233
> URL: https://issues.apache.org/jira/browse/HBASE-18233
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.7
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-18233-branch-1.2.patch, 
> HBASE-18233-branch-1.2.v2.patch
>
>
> Please refer to the discuss in HBASE-18144
> https://issues.apache.org/jira/browse/HBASE-18144?focusedCommentId=16051701=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16051701



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-17546) Incorrect syntax at HBase-Spark Module Examples

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055107#comment-16055107
 ] 

Hudson commented on HBASE-17546:


FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3226 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/3226/])
HBASE-17546 Correct scala syntax to match Apache Spark examples (busbey: rev 
7b6eb90ac9d6f10324f5633baa741d49638064f9)
* (edit) 
hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/example/datasources/AvroSource.scala
* (edit) 
hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/example/datasources/HBaseSource.scala
* (edit) 
hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/example/datasources/DataType.scala


> Incorrect syntax at HBase-Spark Module Examples
> ---
>
> Key: HBASE-17546
> URL: https://issues.apache.org/jira/browse/HBASE-17546
> Project: HBase
>  Issue Type: Bug
>  Components: spark
>Affects Versions: 1.1.8
>Reporter: Chetan Khatri
>Assignee: Chetan Khatri
>Priority: Minor
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-17546.master.001.patch
>
>
> Corrected : Syntax errors at HBase-Spark module
> Description:
>  - Syntax errors has been correct at HBase-Spark module examples.
> - Spark Transformation : somewhere show() and somewhere show only, corrected 
> to show().



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18234) Revisit the async admin api

[Guanghao Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guanghao Zhang updated HBASE-18234:
---
Attachment: HBASE-18234.master.002.patch

> Revisit the async admin api
> ---
>
> Key: HBASE-18234
> URL: https://issues.apache.org/jira/browse/HBASE-18234
> Project: HBase
>  Issue Type: Sub-task
>  Components: Client
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0, 3.0.0
>
> Attachments: HBASE-18234.master.001.patch, 
> HBASE-18234.master.002.patch
>
>
> 1. Update the balance method name. 
> balancer -> balance
> setBalancerRunning -> setBalancerOn
> isBalancerEnabled -> isBalancerOn
> 2. Use HRegionLocation instead of Pair
> 3. Remove the closeRegionWithEncodedRegionName method. Because all other api 
> can handle region name or encoded region name both. So don't need a method 
> for encoded name.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18233) We shouldn't wait for readlock in doMiniBatchMutation in case of deadlock

[Allan Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055106#comment-16055106
 ] 

Allan Yang commented on HBASE-18233:


{quote}
Why (I just skimmed the test). This is with sort in place? Why we get to 30 
seconds waiting on a lock?
{quote}
We haven't commit HBASE-17924 to branch-1.2 yet, so we are not sorting batches 
here

> We shouldn't wait for readlock in doMiniBatchMutation in case of deadlock
> -
>
> Key: HBASE-18233
> URL: https://issues.apache.org/jira/browse/HBASE-18233
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.7
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-18233-branch-1.2.patch, 
> HBASE-18233-branch-1.2.v2.patch
>
>
> Please refer to the discuss in HBASE-18144
> https://issues.apache.org/jira/browse/HBASE-18144?focusedCommentId=16051701=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16051701



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-18241) Change client.Table and client.Admin to not use HTableDescriptor

[Biju Nair (JIRA)]

Biju Nair created HBASE-18241:
-

 Summary: Change client.Table and client.Admin to not use 
HTableDescriptor
 Key: HBASE-18241
 URL: https://issues.apache.org/jira/browse/HBASE-18241
 Project: HBase
  Issue Type: Bug
Reporter: Biju Nair
Priority: Minor


{{HTableDescriptor}} is deprecated and scheduled to be removed in 3.0. But 
[client.Table|https://github.com/apache/hbase/blob/a66d491892514fd4a188d6ca87d6260d8ae46184/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Table.java#L69]
 and 
[client.Admin|https://github.com/apache/hbase/blob/a66d491892514fd4a188d6ca87d6260d8ae46184/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java#L198]
 method {{getTableDescriptor}} returns {{HTableDescriptor}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-17546) Incorrect syntax at HBase-Spark Module Examples

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055101#comment-16055101
 ] 

Hudson commented on HBASE-17546:


SUCCESS: Integrated in Jenkins build HBase-2.0 #74 (See 
[https://builds.apache.org/job/HBase-2.0/74/])
HBASE-17546 Correct scala syntax to match Apache Spark examples (busbey: rev 
79607fda85b58fc0ba9f9045fd6990d2841e276c)
* (edit) 
hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/example/datasources/HBaseSource.scala
* (edit) 
hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/example/datasources/AvroSource.scala
* (edit) 
hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/example/datasources/DataType.scala


> Incorrect syntax at HBase-Spark Module Examples
> ---
>
> Key: HBASE-17546
> URL: https://issues.apache.org/jira/browse/HBASE-17546
> Project: HBase
>  Issue Type: Bug
>  Components: spark
>Affects Versions: 1.1.8
>Reporter: Chetan Khatri
>Assignee: Chetan Khatri
>Priority: Minor
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-17546.master.001.patch
>
>
> Corrected : Syntax errors at HBase-Spark module
> Description:
>  - Syntax errors has been correct at HBase-Spark module examples.
> - Spark Transformation : somewhere show() and somewhere show only, corrected 
> to show().



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18233) We shouldn't wait for readlock in doMiniBatchMutation in case of deadlock

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055096#comment-16055096
 ] 

stack commented on HBASE-18233:
---

bq. We should 'flush' batches with read locks and then come back to try the 
failed readlock again.

Ok.

I think this needs comment in the code. We are familiar w/ how it works now but 
who knows who is going to be in here next and it takes a while to figure its 
workings.

bq. As you can see in the new added UT 
org.apache.hadoop.hbase.client.TestMultiParallel.testMultiThreadWithRowLocks, 
if we wait for the lock, we will face 30 seconds timeout of getting locks

Why (I just skimmed the test). This is with sort in place? Why we get to 30 
seconds waiting on a lock?

Thanks.



> We shouldn't wait for readlock in doMiniBatchMutation in case of deadlock
> -
>
> Key: HBASE-18233
> URL: https://issues.apache.org/jira/browse/HBASE-18233
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.7
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-18233-branch-1.2.patch, 
> HBASE-18233-branch-1.2.v2.patch
>
>
> Please refer to the discuss in HBASE-18144
> https://issues.apache.org/jira/browse/HBASE-18144?focusedCommentId=16051701=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16051701



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18167) OfflineMetaRepair tool may cause HMaster abort always

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055095#comment-16055095
 ] 

Ted Yu commented on HBASE-18167:


Since hbase:meta may contain many rows, it would be faster if we don't scan the 
whole table.
That was why I suggested changing the return type to boolean.

> OfflineMetaRepair tool may cause HMaster abort always
> -
>
> Key: HBASE-18167
> URL: https://issues.apache.org/jira/browse/HBASE-18167
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 1.4.0, 1.3.1, 1.3.2
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 1.4.0
>
> Attachments: HBASE-18167-branch-1.patch
>
>
> In the production environment, we met a weird scenario where some Meta table 
> HFile blocks were missing due to some reason.
> To recover the environment we tried to rebuild the meta using 
> OfflineMetaRepair tool and restart the cluster, but HMaster couldn't finish 
> it's initialization. It always timed out as namespace table region was never 
> assigned.
> Steps to reproduce
> ==
> 1. Assign meta table region to HMaster (it can be on any RS, just to 
> reproduce the  scenario)
> {noformat}
>   
> hbase.balancer.tablesOnMaster
> hbase:meta
> 
> {noformat}
> 2. Start HMaster and RegionServer
> 2. Create two namespace, say "ns1" & "ns2"
> 3. Create two tables "ns1:t1' & "ns2:t1'
> 4. flush 'hbase:meta"
> 5. Stop HMaster (graceful shutdown)
> 6. Kill -9 RegionServer (Abnormal shutdown)
> 7. Run OfflineMetaRepair as follows,
> {noformat}
>   hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair -fix
> {noformat}
> 8. Restart HMaster and RegionServer
> 9. HMaster will never be able to finish its initialization and abort always 
> with below message,
> {code}
> 2017-06-06 15:11:07,582 FATAL [Hostname:16000.activeMasterManager] 
> master.HMaster: Unhandled exception. Starting shutdown.
> java.io.IOException: Timedout 12ms waiting for namespace table to be 
> assigned
> at 
> org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:98)
> at 
> org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:1054)
> at 
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:848)
> at org.apache.hadoop.hbase.master.HMaster.access$600(HMaster.java:199)
> at org.apache.hadoop.hbase.master.HMaster$2.run(HMaster.java:1871)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Root cause
> ==
> 1. During HM start up AM assumes that it's a failover scenario based on the 
> existing old WAL files, so SSH/SCP will split WAL files and assign the 
> holding regions. 
> 2. During SSH/SCP it retrieves the server holding regions from meta/AM's 
> in-memory-state, but meta only had "regioninfo" entry (as already rebuild by 
> OfflineMetaRepair). So empty region will be returned and it wont trigger any 
> assignment.
> 3. HMaster which is waiting for namespace table to be assigned will timeout 
> and abort always.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18167) OfflineMetaRepair tool may cause HMaster abort always

[Pankaj Kumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055094#comment-16055094
 ] 

Pankaj Kumar commented on HBASE-18167:
--

   bq. Suggest changing the return value to int or boolean.

IMHO It would be better to return Set instead of int or boolean, may be useful 
in some future use cases. 
what do you say [~tedyu]?

> OfflineMetaRepair tool may cause HMaster abort always
> -
>
> Key: HBASE-18167
> URL: https://issues.apache.org/jira/browse/HBASE-18167
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 1.4.0, 1.3.1, 1.3.2
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 1.4.0
>
> Attachments: HBASE-18167-branch-1.patch
>
>
> In the production environment, we met a weird scenario where some Meta table 
> HFile blocks were missing due to some reason.
> To recover the environment we tried to rebuild the meta using 
> OfflineMetaRepair tool and restart the cluster, but HMaster couldn't finish 
> it's initialization. It always timed out as namespace table region was never 
> assigned.
> Steps to reproduce
> ==
> 1. Assign meta table region to HMaster (it can be on any RS, just to 
> reproduce the  scenario)
> {noformat}
>   
> hbase.balancer.tablesOnMaster
> hbase:meta
> 
> {noformat}
> 2. Start HMaster and RegionServer
> 2. Create two namespace, say "ns1" & "ns2"
> 3. Create two tables "ns1:t1' & "ns2:t1'
> 4. flush 'hbase:meta"
> 5. Stop HMaster (graceful shutdown)
> 6. Kill -9 RegionServer (Abnormal shutdown)
> 7. Run OfflineMetaRepair as follows,
> {noformat}
>   hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair -fix
> {noformat}
> 8. Restart HMaster and RegionServer
> 9. HMaster will never be able to finish its initialization and abort always 
> with below message,
> {code}
> 2017-06-06 15:11:07,582 FATAL [Hostname:16000.activeMasterManager] 
> master.HMaster: Unhandled exception. Starting shutdown.
> java.io.IOException: Timedout 12ms waiting for namespace table to be 
> assigned
> at 
> org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:98)
> at 
> org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:1054)
> at 
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:848)
> at org.apache.hadoop.hbase.master.HMaster.access$600(HMaster.java:199)
> at org.apache.hadoop.hbase.master.HMaster$2.run(HMaster.java:1871)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Root cause
> ==
> 1. During HM start up AM assumes that it's a failover scenario based on the 
> existing old WAL files, so SSH/SCP will split WAL files and assign the 
> holding regions. 
> 2. During SSH/SCP it retrieves the server holding regions from meta/AM's 
> in-memory-state, but meta only had "regioninfo" entry (as already rebuild by 
> OfflineMetaRepair). So empty region will be returned and it wont trigger any 
> assignment.
> 3. HMaster which is waiting for namespace table to be assigned will timeout 
> and abort always.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18233) We shouldn't wait for readlock in doMiniBatchMutation in case of deadlock

[Allan Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allan Yang updated HBASE-18233:
---
Attachment: (was: HBASE-18233-branch-1.2.v2.patch)

> We shouldn't wait for readlock in doMiniBatchMutation in case of deadlock
> -
>
> Key: HBASE-18233
> URL: https://issues.apache.org/jira/browse/HBASE-18233
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.7
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-18233-branch-1.2.patch, 
> HBASE-18233-branch-1.2.v2.patch
>
>
> Please refer to the discuss in HBASE-18144
> https://issues.apache.org/jira/browse/HBASE-18144?focusedCommentId=16051701=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16051701



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18233) We shouldn't wait for readlock in doMiniBatchMutation in case of deadlock

[Allan Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allan Yang updated HBASE-18233:
---
Attachment: HBASE-18233-branch-1.2.v2.patch

> We shouldn't wait for readlock in doMiniBatchMutation in case of deadlock
> -
>
> Key: HBASE-18233
> URL: https://issues.apache.org/jira/browse/HBASE-18233
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.7
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-18233-branch-1.2.patch, 
> HBASE-18233-branch-1.2.v2.patch, HBASE-18233-branch-1.2.v2.patch
>
>
> Please refer to the discuss in HBASE-18144
> https://issues.apache.org/jira/browse/HBASE-18144?focusedCommentId=16051701=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16051701



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18214) Replace the folly::AtomicHashMap usage in the RPC layer

[Enis Soztutar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Enis Soztutar updated HBASE-18214:
--
Attachment: hbase-18214_v3.patch

Attaching v3 which compiles. Not tested, and the insert still needs some work. 

> Replace the folly::AtomicHashMap usage in the RPC layer
> ---
>
> Key: HBASE-18214
> URL: https://issues.apache.org/jira/browse/HBASE-18214
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Devaraj Das
>Assignee: Devaraj Das
> Attachments: 18214-1-1.txt, 18214-1-2.txt, hbase-18214_v3.patch
>
>
> In my tests, I saw that folly::AtomicHashMap usage is not appropriate for 
> one, rather common use case. It'd become sort of unusable (inserts would 
> hang) after a bunch of inserts and erases. This hashmap is used to keep track 
> of call-Id after a connection is set up in the RPC layer (insert a 
> call-id/msg pair when an RPC is sent, and erase the pair when the 
> corresponding response is received). Here is a simple program that will 
> demonstrate the issue:
> {code}
> folly::AtomicHashMap f(100);
> int i = 0;
> while (i < 1) {
> try {
>   f.insert(i,100);
>   LOG(INFO) << "Inserted " << i << "  " << f.size();
>   f.erase(i);
>   LOG(INFO) << "Deleted " << i << "  " << f.size();
>   i++;
> } catch (const std::exception ) {
>   LOG(INFO) << "Exception " << e.what();
>   break;
> }
> }
> {code}
> After poking around a little bit, it is indeed called out as a limitation 
> here 
> https://github.com/facebook/folly/blob/master/folly/docs/AtomicHashMap.md 
> (grep for 'erase'). Proposal is to replace this with something that will fit 
> in in the above usecase (thinking of using std::unordered_map).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18226) Disable reverse DNS lookup at HMaster and use default hostname provided by RegionServer

[Duo Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055089#comment-16055089
 ] 

Duo Xu commented on HBASE-18226:


[~enis]

I do not see anywhere else we need to change.

if the instance is HMaster, it will get the serverName from HRegionServer 
constructor and no further changes happen on the serverName variable.

{code}
String hostName = shouldUseThisHostnameInstead() ? useThisHostnameInstead :
  rpcServices.isa.getHostName();
serverName = ServerName.valueOf(hostName, rpcServices.isa.getPort(), 
startcode);
{code}

if the instance is HRegionServer, although it is going through the same piece 
of constructor code, in run() function, it will call reportForDuty() and in 
handleReportForDutyResponse(w), servername is overwritten by the hostname 
returned by HMaster.

{code}
// Try and register with the Master; tell it we are here.  Break if
  // server is stopped or the clusterup flag is down or hdfs went wacky.
  while (keepLooping()) {
RegionServerStartupResponse w = reportForDuty();
if (w == null) {
  LOG.warn("reportForDuty failed; sleeping and then retrying.");
  this.sleeper.sleep();
} else {
  handleReportForDutyResponse(w);
  break;
}
  }
{code}

{code}
protected void handleReportForDutyResponse(final RegionServerStartupResponse c)
  throws IOException {
try {
  boolean updateRootDir = false;
  for (NameStringPair e : c.getMapEntriesList()) {
String key = e.getName();
// The hostname the master sees us as.
if (key.equals(HConstants.KEY_FOR_HOSTNAME_SEEN_BY_MASTER)) {
  String hostnameFromMasterPOV = e.getValue();
  this.serverName = ServerName.valueOf(hostnameFromMasterPOV,
rpcServices.isa.getPort(), this.startcode);
{code}

So I think we do not need to add extra code for handling Master as a RS.

Let me know if i missed something.

> Disable reverse DNS lookup at HMaster and use default hostname provided by 
> RegionServer
> ---
>
> Key: HBASE-18226
> URL: https://issues.apache.org/jira/browse/HBASE-18226
> Project: HBase
>  Issue Type: Bug
>Reporter: Duo Xu
> Attachments: HBASE-18226.001.patch, HBASE-18226.002.patch, 
> HBASE-18226.003.patch, HBASE-18226.004.patch
>
>
> Description updated:
> In some unusual network environment, forward DNS lookup is supported while 
> reverse DNS lookup may not work properly.
> This JIRA is to address that HMaster uses the hostname passed from RS instead 
> of doing reverse DNS lookup to tells RS which hostname to use during 
> reportForDuty() . This has already been implemented by HBASE-12954 by adding 
> "useThisHostnameInstead" field in RegionServerStatusProtos.
> Currently "useThisHostnameInstead" is optional and RS by default only passes 
> port, server start code and server current time info to HMaster during RS 
> reportForDuty(). In order to use this field, users currently need to specify 
> "hbase.regionserver.hostname" on every regionserver node's hbase-site.xml. 
> This causes some trouble in
> 1. some deployments managed by some management tools like Ambari, which 
> maintains the same copy of hbase-site.xml across all the nodes.
> 2. HBASE-12954 is targeting multihomed hosts, which users want to manually 
> set the hostname value for each node. In the other cases (not multihomed), I 
> just want RS to use the hostname return by the node and set it in 
> useThisHostnameInstead and pass to HMaster during reportForDuty().
> I would like to introduce a setting that if the setting is set to true, 
> "useThisHostnameInstead" will be set to the hostname RS gets from the node. 
> Then HMaster will skip reverse DNS lookup because it sees 
> "useThisHostnameInstead" field is set in the request.
> "hbase.regionserver.hostname.reported.to.master", is it a good name?
> 
> Regarding the hostname returned by the RS node, I read the source code again 
> (including hadoop-common dns.java). By default RS gets hostname by calling 
> InetAddress.getLocalHost().getCanonicalHostName(). If users specify 
> "hbase.regionserver.dns.interface" or "hbase.regionserver.dns.nameserver" or 
> some underlying system configuration changes (eg. modifying 
> /etc/nsswitch.conf), it may first read from DNS or other sources instead of 
> first checking /etc/hosts file.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18233) We shouldn't wait for readlock in doMiniBatchMutation in case of deadlock

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055080#comment-16055080
 ] 

Ted Yu commented on HBASE-18233:


{code}
747 HTable table = new HTable(UTIL.getConfiguration(), TEST_TABLE2);
{code}
table should be closed upon return from the method.

Nice test.

> We shouldn't wait for readlock in doMiniBatchMutation in case of deadlock
> -
>
> Key: HBASE-18233
> URL: https://issues.apache.org/jira/browse/HBASE-18233
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.7
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-18233-branch-1.2.patch, 
> HBASE-18233-branch-1.2.v2.patch
>
>
> Please refer to the discuss in HBASE-18144
> https://issues.apache.org/jira/browse/HBASE-18144?focusedCommentId=16051701=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16051701



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18233) We shouldn't wait for readlock in doMiniBatchMutation in case of deadlock

[Allan Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055073#comment-16055073
 ] 

Allan Yang commented on HBASE-18233:


{quote}
Maybe return null directly in the else clause makes the logic clearer?
{quote}
Modified in the new patch
{quote}
Please also check the failed UT case and confirm whether relative. Thanks.
{quote}
The failed UTs are not related and passed locally. But the timeout case is 
related, I have fixed this case in the new patch
{quote}
I'm not that sure but it seems we also have the issue in branch-1/branch-1.3?
{quote}
They may have the same issue, let me check, I think even in branch-2 and master 
branch have this issue too. We should commit to this branch either.
[~carp84], thanks for your explanation, that's exactly what I’m thinking of!

{quote}
What about this question: "Where is the benefit? The benefit is that if some 
one comes through looking for a write lock, first we'll flush any read lock 
batches? Any benefit seen?"
Is the benefit that we no longer block? That we make-do with however many locks 
we were able to attain?
{quote}
Yes, you are right, boss [~stack]. As we discussed in HBASE-18144. The benefit 
is that we won't create a block situation here. We should 'flush' batches with 
read locks and then come back to try the failed readlock again. Even it is a 
readlock, blocking will create a temp deadlock, right?
{quote}
Any benefit seen in testing?
{quote}
As you can see in the new added UT 
org.apache.hadoop.hbase.client.TestMultiParallel.testMultiThreadWithRowLocks, 
if we wait for the lock, we will face 30 seconds timeout of getting locks
 


> We shouldn't wait for readlock in doMiniBatchMutation in case of deadlock
> -
>
> Key: HBASE-18233
> URL: https://issues.apache.org/jira/browse/HBASE-18233
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.7
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-18233-branch-1.2.patch, 
> HBASE-18233-branch-1.2.v2.patch
>
>
> Please refer to the discuss in HBASE-18144
> https://issues.apache.org/jira/browse/HBASE-18144?focusedCommentId=16051701=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16051701



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18180) Possible connection leak while closing BufferedMutator in TableOutputFormat

[Pankaj Kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pankaj Kumar updated HBASE-18180:
-
Attachment: HBASE-18180-branch-1.3.patch

Thanks [~tedyu] for committing this patch. How about fixing this issue in 
branch 1.3, attaching branch-1.3 patch. 

> Possible connection leak while closing BufferedMutator in TableOutputFormat
> ---
>
> Key: HBASE-18180
> URL: https://issues.apache.org/jira/browse/HBASE-18180
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Affects Versions: 1.4.0, 1.3.1, 1.3.2
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Fix For: 3.0.0, 1.4.0, 2.0.0-alpha-2
>
> Attachments: HBASE-18180-branch-1.3.patch, 
> HBASE-18180-branch-1.patch, HBASE-18180.patch
>
>
> In TableOutputFormat, connection will not be released in case when 
> "mutator.close()" throws exception.
> org.apache.hadoop.hbase.mapreduce.TableOutputFormat
> {code}
> public void close(TaskAttemptContext context)
> throws IOException {
>   mutator.close();
>   connection.close();
> }
> {code}
> org.apache.hadoop.hbase.mapred.TableOutputFormat
> {code}
> public void close(Reporter reporter) throws IOException {
>   this.m_mutator.close();
>   if (connection != null) {
> connection.close();
> connection = null;
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18233) We shouldn't wait for readlock in doMiniBatchMutation in case of deadlock

[Allan Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allan Yang updated HBASE-18233:
---
Attachment: HBASE-18233-branch-1.2.v2.patch

> We shouldn't wait for readlock in doMiniBatchMutation in case of deadlock
> -
>
> Key: HBASE-18233
> URL: https://issues.apache.org/jira/browse/HBASE-18233
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.7
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-18233-branch-1.2.patch, 
> HBASE-18233-branch-1.2.v2.patch
>
>
> Please refer to the discuss in HBASE-18144
> https://issues.apache.org/jira/browse/HBASE-18144?focusedCommentId=16051701=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16051701



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18240) Add hbase-auxillary, a project with hbase utility including an hbase-shaded-thirdparty module with guava, netty, etc.

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055045#comment-16055045
 ] 

Sean Busbey commented on HBASE-18240:
-

let me know if you need help navigating the git repo request or the rest of the 
task.

> Add hbase-auxillary, a project with hbase utility including an 
> hbase-shaded-thirdparty module with guava, netty, etc.
> -
>
> Key: HBASE-18240
> URL: https://issues.apache.org/jira/browse/HBASE-18240
> Project: HBase
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0
>
>
> This issue is about adding a new related project to host hbase auxillary 
> utility. In this new project, the first thing we'd add is a module to host 
> shaded versions of third party libraries.
> This task comes of discussion held here 
> http://apache-hbase.679495.n3.nabble.com/DISCUSS-More-Shading-td4083025.html 
> where one conclusion of the DISCUSSION was "... pushing this part forward 
> with some code is the next logical step. Seems to be consensus about taking 
> our known internal dependencies and performing this shade magic."



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18167) OfflineMetaRepair tool may cause HMaster abort always

[Allan Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055043#comment-16055043
 ] 

Allan Yang commented on HBASE-18167:


{quote}
In the current patch I am reusing the same configuration parameter 
"hbase.master.initializationmonitor.timeout" for SSH wait timeout.
I feel it's better we introduce another parameter for this.
What do you say?
{quote}
I think we don't need to introduce a new config. the old one is enough, another 
option is that you can wait forever here, since there is already a timeout 
monitor.

> OfflineMetaRepair tool may cause HMaster abort always
> -
>
> Key: HBASE-18167
> URL: https://issues.apache.org/jira/browse/HBASE-18167
> Project: HBase
>  Issue Type: Bug
>  Components: master
>Affects Versions: 1.4.0, 1.3.1, 1.3.2
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 1.4.0
>
> Attachments: HBASE-18167-branch-1.patch
>
>
> In the production environment, we met a weird scenario where some Meta table 
> HFile blocks were missing due to some reason.
> To recover the environment we tried to rebuild the meta using 
> OfflineMetaRepair tool and restart the cluster, but HMaster couldn't finish 
> it's initialization. It always timed out as namespace table region was never 
> assigned.
> Steps to reproduce
> ==
> 1. Assign meta table region to HMaster (it can be on any RS, just to 
> reproduce the  scenario)
> {noformat}
>   
> hbase.balancer.tablesOnMaster
> hbase:meta
> 
> {noformat}
> 2. Start HMaster and RegionServer
> 2. Create two namespace, say "ns1" & "ns2"
> 3. Create two tables "ns1:t1' & "ns2:t1'
> 4. flush 'hbase:meta"
> 5. Stop HMaster (graceful shutdown)
> 6. Kill -9 RegionServer (Abnormal shutdown)
> 7. Run OfflineMetaRepair as follows,
> {noformat}
>   hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair -fix
> {noformat}
> 8. Restart HMaster and RegionServer
> 9. HMaster will never be able to finish its initialization and abort always 
> with below message,
> {code}
> 2017-06-06 15:11:07,582 FATAL [Hostname:16000.activeMasterManager] 
> master.HMaster: Unhandled exception. Starting shutdown.
> java.io.IOException: Timedout 12ms waiting for namespace table to be 
> assigned
> at 
> org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:98)
> at 
> org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:1054)
> at 
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:848)
> at org.apache.hadoop.hbase.master.HMaster.access$600(HMaster.java:199)
> at org.apache.hadoop.hbase.master.HMaster$2.run(HMaster.java:1871)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Root cause
> ==
> 1. During HM start up AM assumes that it's a failover scenario based on the 
> existing old WAL files, so SSH/SCP will split WAL files and assign the 
> holding regions. 
> 2. During SSH/SCP it retrieves the server holding regions from meta/AM's 
> in-memory-state, but meta only had "regioninfo" entry (as already rebuild by 
> OfflineMetaRepair). So empty region will be returned and it wont trigger any 
> assignment.
> 3. HMaster which is waiting for namespace table to be assigned will timeout 
> and abort always.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18226) Disable reverse DNS lookup at HMaster and use default hostname provided by RegionServer

[Enis Soztutar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055040#comment-16055040
 ] 

Enis Soztutar commented on HBASE-18226:
---

One more thing, we probably want to do this for master as well, no? 

> Disable reverse DNS lookup at HMaster and use default hostname provided by 
> RegionServer
> ---
>
> Key: HBASE-18226
> URL: https://issues.apache.org/jira/browse/HBASE-18226
> Project: HBase
>  Issue Type: Bug
>Reporter: Duo Xu
> Attachments: HBASE-18226.001.patch, HBASE-18226.002.patch, 
> HBASE-18226.003.patch, HBASE-18226.004.patch
>
>
> Description updated:
> In some unusual network environment, forward DNS lookup is supported while 
> reverse DNS lookup may not work properly.
> This JIRA is to address that HMaster uses the hostname passed from RS instead 
> of doing reverse DNS lookup to tells RS which hostname to use during 
> reportForDuty() . This has already been implemented by HBASE-12954 by adding 
> "useThisHostnameInstead" field in RegionServerStatusProtos.
> Currently "useThisHostnameInstead" is optional and RS by default only passes 
> port, server start code and server current time info to HMaster during RS 
> reportForDuty(). In order to use this field, users currently need to specify 
> "hbase.regionserver.hostname" on every regionserver node's hbase-site.xml. 
> This causes some trouble in
> 1. some deployments managed by some management tools like Ambari, which 
> maintains the same copy of hbase-site.xml across all the nodes.
> 2. HBASE-12954 is targeting multihomed hosts, which users want to manually 
> set the hostname value for each node. In the other cases (not multihomed), I 
> just want RS to use the hostname return by the node and set it in 
> useThisHostnameInstead and pass to HMaster during reportForDuty().
> I would like to introduce a setting that if the setting is set to true, 
> "useThisHostnameInstead" will be set to the hostname RS gets from the node. 
> Then HMaster will skip reverse DNS lookup because it sees 
> "useThisHostnameInstead" field is set in the request.
> "hbase.regionserver.hostname.reported.to.master", is it a good name?
> 
> Regarding the hostname returned by the RS node, I read the source code again 
> (including hadoop-common dns.java). By default RS gets hostname by calling 
> InetAddress.getLocalHost().getCanonicalHostName(). If users specify 
> "hbase.regionserver.dns.interface" or "hbase.regionserver.dns.nameserver" or 
> some underlying system configuration changes (eg. modifying 
> /etc/nsswitch.conf), it may first read from DNS or other sources instead of 
> first checking /etc/hosts file.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HBASE-18240) Add hbase-auxillary, a project with hbase utility including an hbase-shaded-thirdparty module with guava, netty, etc.

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack reassigned HBASE-18240:
-

   Assignee: stack
Component/s: dependencies

> Add hbase-auxillary, a project with hbase utility including an 
> hbase-shaded-thirdparty module with guava, netty, etc.
> -
>
> Key: HBASE-18240
> URL: https://issues.apache.org/jira/browse/HBASE-18240
> Project: HBase
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0
>
>
> This issue is about adding a new related project to host hbase auxillary 
> utility. In this new project, the first thing we'd add is a module to host 
> shaded versions of third party libraries.
> This task comes of discussion held here 
> http://apache-hbase.679495.n3.nabble.com/DISCUSS-More-Shading-td4083025.html 
> where one conclusion of the DISCUSSION was "... pushing this part forward 
> with some code is the next logical step. Seems to be consensus about taking 
> our known internal dependencies and performing this shade magic."



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-18240) Add hbase-auxillary, a project with hbase utility including an hbase-shaded-thirdparty module with guava, netty, etc.

[stack (JIRA)]

stack created HBASE-18240:
-

 Summary: Add hbase-auxillary, a project with hbase utility 
including an hbase-shaded-thirdparty module with guava, netty, etc.
 Key: HBASE-18240
 URL: https://issues.apache.org/jira/browse/HBASE-18240
 Project: HBase
  Issue Type: Sub-task
Reporter: stack


This issue is about adding a new related project to host hbase auxillary 
utility. In this new project, the first thing we'd add is a module to host 
shaded versions of third party libraries.

This task comes of discussion held here 
http://apache-hbase.679495.n3.nabble.com/DISCUSS-More-Shading-td4083025.html 
where one conclusion of the DISCUSSION was "... pushing this part forward with 
some code is the next logical step. Seems to be consensus about taking our 
known internal dependencies and performing this shade magic."



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18061) [C++] Fix retry logic in multi-get calls

[Sudeep Sunthankar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudeep Sunthankar updated HBASE-18061:
--
Attachment: HBASE-18061.HBASE-14850.v3.patch

Foll changes included in the patch:-

# Added linking of krb5, ssl, crypto and pthread libraries in Makefile.
# Added exception_class_name in ResponseConverter while storing 
RemoteExceptions.
# Passing list of actions, so that we can fetch original index of the action 
when storing Result/Exception in ResponseConverter.
# Fixed a bug how we store the fetched responses in Result vector as it wasn't 
proper, (Pt. 2 above) + this change fixes the same.
# Added a unit test for for multi-gets with pre region splits and changed the 
number of requests to 10K.
# Changed the number of region servers in clien-test to 10 in mini cluster 
startup

Thanks

> [C++] Fix retry logic in multi-get calls
> 
>
> Key: HBASE-18061
> URL: https://issues.apache.org/jira/browse/HBASE-18061
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Sudeep Sunthankar
> Fix For: HBASE-14850
>
> Attachments: HBASE-18061.HBASE-14850.v1.patch, 
> HBASE-18061.HBASE-14850.v3.patch
>
>
> HBASE-17576 adds multi-gets. There are a couple of todos to fix in the retry 
> logic, and some unit testing to be done for the multi-gets. We'll do these in 
> this issue. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (HBASE-18061) [C++] Fix retry logic in multi-get calls

[Sudeep Sunthankar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054992#comment-16054992
 ] 

Sudeep Sunthankar edited comment on HBASE-18061 at 6/20/17 12:49 AM:
-

Deleted last patch because of some issues in Makefile in the patch



was (Author: sudeeps):
This patch consists of the foll changes:-
# Added exception_class_name in ResponseConverter while storing 
RemoteExceptions.
# Passing list of actions, so that we can fetch original index of the action 
when storing Result/Exception in ResponseConverter.
# Fixed a bug how we store the fetched responses in Result vector as it wasn't 
proper, (Pt. 2 above) + this change fixes the same.
# Added a unit test for for multi-gets with pre region splits and changed the 
number of requests to 10K.
# Changed the number of region servers in clien-test to 10 in mini cluster 
startup
# Added reviews from the last patch

Thanks.

> [C++] Fix retry logic in multi-get calls
> 
>
> Key: HBASE-18061
> URL: https://issues.apache.org/jira/browse/HBASE-18061
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Sudeep Sunthankar
> Fix For: HBASE-14850
>
> Attachments: HBASE-18061.HBASE-14850.v1.patch
>
>
> HBASE-17576 adds multi-gets. There are a couple of todos to fix in the retry 
> logic, and some unit testing to be done for the multi-gets. We'll do these in 
> this issue. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18144) Forward-port the old exclusive row lock; there are scenarios where it performs better

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054993#comment-16054993
 ] 

Jerry He commented on HBASE-18144:
--

The  write-lock attempt is blocked (not able to acquire the lock), and this 
attempt also blocks other read attempts on the same row it looks like.

> Forward-port the old exclusive row lock; there are scenarios where it 
> performs better
> -
>
> Key: HBASE-18144
> URL: https://issues.apache.org/jira/browse/HBASE-18144
> Project: HBase
>  Issue Type: Bug
>  Components: Increment
>Affects Versions: 1.2.5
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.3.2, 1.2.7
>
> Attachments: DisorderedBatchAndIncrementUT.patch, 
> HBASE-18144.master.001.patch
>
>
> Description to follow.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18061) [C++] Fix retry logic in multi-get calls

[Sudeep Sunthankar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudeep Sunthankar updated HBASE-18061:
--
Attachment: (was: HBASE-18061.HBASE-14850.v3.patch)

> [C++] Fix retry logic in multi-get calls
> 
>
> Key: HBASE-18061
> URL: https://issues.apache.org/jira/browse/HBASE-18061
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Sudeep Sunthankar
> Fix For: HBASE-14850
>
> Attachments: HBASE-18061.HBASE-14850.v1.patch
>
>
> HBASE-17576 adds multi-gets. There are a couple of todos to fix in the retry 
> logic, and some unit testing to be done for the multi-gets. We'll do these in 
> this issue. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18061) [C++] Fix retry logic in multi-get calls

[Sudeep Sunthankar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudeep Sunthankar updated HBASE-18061:
--
Status: Open  (was: Patch Available)

> [C++] Fix retry logic in multi-get calls
> 
>
> Key: HBASE-18061
> URL: https://issues.apache.org/jira/browse/HBASE-18061
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Sudeep Sunthankar
> Fix For: HBASE-14850
>
> Attachments: HBASE-18061.HBASE-14850.v1.patch
>
>
> HBASE-17576 adds multi-gets. There are a couple of todos to fix in the retry 
> logic, and some unit testing to be done for the multi-gets. We'll do these in 
> this issue. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18061) [C++] Fix retry logic in multi-get calls

[Sudeep Sunthankar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudeep Sunthankar updated HBASE-18061:
--
Status: Patch Available  (was: In Progress)

> [C++] Fix retry logic in multi-get calls
> 
>
> Key: HBASE-18061
> URL: https://issues.apache.org/jira/browse/HBASE-18061
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Sudeep Sunthankar
> Fix For: HBASE-14850
>
> Attachments: HBASE-18061.HBASE-14850.v1.patch, 
> HBASE-18061.HBASE-14850.v3.patch
>
>
> HBASE-17576 adds multi-gets. There are a couple of todos to fix in the retry 
> logic, and some unit testing to be done for the multi-gets. We'll do these in 
> this issue. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18061) [C++] Fix retry logic in multi-get calls

[Sudeep Sunthankar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudeep Sunthankar updated HBASE-18061:
--
Attachment: HBASE-18061.HBASE-14850.v3.patch

This patch consists of the foll changes:-
# Added exception_class_name in ResponseConverter while storing 
RemoteExceptions.
# Passing list of actions, so that we can fetch original index of the action 
when storing Result/Exception in ResponseConverter.
# Fixed a bug how we store the fetched responses in Result vector as it wasn't 
proper, (Pt. 2 above) + this change fixes the same.
# Added a unit test for for multi-gets with pre region splits and changed the 
number of requests to 10K.
# Changed the number of region servers in clien-test to 10 in mini cluster 
startup
# Added reviews from the last patch

Thanks.

> [C++] Fix retry logic in multi-get calls
> 
>
> Key: HBASE-18061
> URL: https://issues.apache.org/jira/browse/HBASE-18061
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Sudeep Sunthankar
> Fix For: HBASE-14850
>
> Attachments: HBASE-18061.HBASE-14850.v1.patch, 
> HBASE-18061.HBASE-14850.v3.patch
>
>
> HBASE-17576 adds multi-gets. There are a couple of todos to fix in the retry 
> logic, and some unit testing to be done for the multi-gets. We'll do these in 
> this issue. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Work started] (HBASE-18061) [C++] Fix retry logic in multi-get calls

[Sudeep Sunthankar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HBASE-18061 started by Sudeep Sunthankar.
-
> [C++] Fix retry logic in multi-get calls
> 
>
> Key: HBASE-18061
> URL: https://issues.apache.org/jira/browse/HBASE-18061
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Sudeep Sunthankar
> Fix For: HBASE-14850
>
> Attachments: HBASE-18061.HBASE-14850.v1.patch, 
> HBASE-18061.HBASE-14850.v3.patch
>
>
> HBASE-17576 adds multi-gets. There are a couple of todos to fix in the retry 
> logic, and some unit testing to be done for the multi-gets. We'll do these in 
> this issue. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18144) Forward-port the old exclusive row lock; there are scenarios where it performs better

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054981#comment-16054981
 ] 

stack commented on HBASE-18144:
---

Can we write a test that demonstrates the deadlock? The scenario described 
seems clean enough to repro?

[~jerryhe] When you say 'wedged', you mean blocked until the reads complete? A 
'wedge' is 'less' than the deadlock where we cannot progress until the try 
expires?

Thanks for taking a look sir.

> Forward-port the old exclusive row lock; there are scenarios where it 
> performs better
> -
>
> Key: HBASE-18144
> URL: https://issues.apache.org/jira/browse/HBASE-18144
> Project: HBase
>  Issue Type: Bug
>  Components: Increment
>Affects Versions: 1.2.5
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.3.2, 1.2.7
>
> Attachments: DisorderedBatchAndIncrementUT.patch, 
> HBASE-18144.master.001.patch
>
>
> Description to follow.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18144) Forward-port the old exclusive row lock; there are scenarios where it performs better

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054963#comment-16054963
 ] 

Jerry He commented on HBASE-18144:
--

Trying to catch up with all the good findings and discussion here. 

This is a good example. 
{quote}
handler 1 is doing a batch put of row(1,2,3,4,5,6,7,8,9). At the same time, 
handler 4 is doing a batch put but with reversed keys(9,8,7,6,5,4,3,2,1). 
1. handler 1 have got readlock for row 1,2,3,4,5,6,7, and going to try row 8's 
readlock
2. handler 4 have got readlock for row 9,8,7,6,5,4,3, and going to try row 2's 
readlock
3. At the same time, handler 0 is serving a request to increment row 2, it need 
to get the writelock of row 2, but it have to wait since handler 1 have already 
got row 2's readlock (handler 0 blocked)
4. since handler 0 is trying to get row 2' writelock, handler 4's attempt to 
try row 2's readlock need to wait(handler 4 blocked)
5. At the same time,handler 3 is serving a request to increment row 8, it need 
to get the writelock of row 8, but it have to wait since handler 4 have already 
got row 8's readlock (handler 3 blocked)
6. since handler 3 is trying to get row 8' writelock, handler 1's attempt to 
try row 8's readlock need to wait(handler 1 blocked)
{quote}
bq. 3. At the same time, handler 0 is serving a request to increment row 2, it 
need to get the writelock of row 2, but it have to wait since handler 1 have 
already got row 2's readlock (handler 0 blocked)
   ok, This is expected.
   
bq. 4. since handler 0 is trying to get row 2' writelock, handler 4's attempt 
to try row 2's readlock need to wait(handler 4 blocked)
   ReentrantReadWriteLock does not seem to be good here. 
   handler 4 is only attempting a readlock on row 2, which handler 1 alreadly 
has. 
   Any writelock request (increment) is like a wedge in the middle.  Read 
concurrency is compromised.
   
bq. 6. since handler 3 is trying to get row 8' writelock, handler 1's attempt 
to try row 8's readlock need to wait(handler 1 blocked)
   Same here. handler 1 is only attempting a readlock on row 8, which handler 4 
already has. 
   A writelock request arriving in the middle is like a wedge again, 
compromising read concurrency.
   
HBASE-18233 will help solve the deallock.

How sorting will add additional benefit is hard to see.  For example, there are 
two batch puts of rows (1, 2, 3, 4, 5, 6, 7, 8).
The first thread has acquired locks on batch 1,2,3,4,5,6,7.  Then if a 
writelock request arrives anywhere in the middle, the second batch puts from 
another handler is wedged anyways.

> Forward-port the old exclusive row lock; there are scenarios where it 
> performs better
> -
>
> Key: HBASE-18144
> URL: https://issues.apache.org/jira/browse/HBASE-18144
> Project: HBase
>  Issue Type: Bug
>  Components: Increment
>Affects Versions: 1.2.5
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.3.2, 1.2.7
>
> Attachments: DisorderedBatchAndIncrementUT.patch, 
> HBASE-18144.master.001.patch
>
>
> Description to follow.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-16242) Upgrade Avro to 1.7.7

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054957#comment-16054957
 ] 

Hudson commented on HBASE-16242:


SUCCESS: Integrated in Jenkins build HBase-2.0 #73 (See 
[https://builds.apache.org/job/HBase-2.0/73/])
HBASE-16242 Manually resolve transitive Apache Avro dependency to (busbey: rev 
d3ba357c7c4d594fd8e75f9631fa4510512989b5)
* (edit) hbase-common/pom.xml
* (edit) pom.xml
* (edit) hbase-spark/pom.xml


> Upgrade Avro to 1.7.7
> -
>
> Key: HBASE-16242
> URL: https://issues.apache.org/jira/browse/HBASE-16242
> Project: HBase
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Ben McCann
>Assignee: Sean Busbey
> Fix For: 3.0.0, 1.4.0, 2.0.0-alpha-2
>
> Attachments: HBASE-16242.0.patch
>
>
> I'd like to see Avro upgraded to 1.8.1 or at the very least 1.7.7



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-16242) Upgrade Avro to 1.7.7

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054941#comment-16054941
 ] 

Hudson commented on HBASE-16242:


SUCCESS: Integrated in Jenkins build HBase-1.4 #778 (See 
[https://builds.apache.org/job/HBase-1.4/778/])
HBASE-16242 Manually resolve transitive Apache Avro dependency to (busbey: rev 
c6e587c5ac24ce29d6d90d2b75ac1c943eb7f38f)
* (edit) hbase-common/pom.xml
* (edit) pom.xml


> Upgrade Avro to 1.7.7
> -
>
> Key: HBASE-16242
> URL: https://issues.apache.org/jira/browse/HBASE-16242
> Project: HBase
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Ben McCann
>Assignee: Sean Busbey
> Fix For: 3.0.0, 1.4.0, 2.0.0-alpha-2
>
> Attachments: HBASE-16242.0.patch
>
>
> I'd like to see Avro upgraded to 1.8.1 or at the very least 1.7.7



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (HBASE-18214) Replace the folly::AtomicHashMap usage in the RPC layer

[Devaraj Das (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054931#comment-16054931
 ] 

Devaraj Das edited comment on HBASE-18214 at 6/19/17 11:23 PM:
---

bq. Instead of  std::unique_ptr mutex_; use 
std::shared_timed_mutex mutex_;
The issue is the shared_timed_mutex is (rightfully) not copyable. So one would 
have to define explicit copy constructors in the classes where this is used... 
It's not clear as to what such a copy constructor would do anyway (mutexes are 
fundamentally not copyable safely). Treating it as a pointer in class 
declarations works around such things since all the instances in question would 
refer to the same underlying mutex. But maybe, we should declare it to be 
shared_ptr as opposed to a unique_ptr. In the usecase we have, we probably will 
not run into issues with either anyway I guess.


was (Author: devaraj):
bq. Instead of  std::unique_ptr mutex_; use 
std::shared_timed_mutex mutex_;
The issue is the shared_timed_mutex is (rightfully) not copyable. So one would 
have to define explicit copy constructors in the classes where this is used... 
It's not clear as to what such a copy constructor would do anyway (mutexes are 
fundamentally not copyable safely). Treating it as a pointer in class 
declarations works around such things since all the instances in question would 
refer to the same underlying mutex. But maybe, we should declare it to be 
shared_ptr as opposed to a shared_ptr. In the usecase we have, we probably will 
not run into issues with either anyway I guess.

> Replace the folly::AtomicHashMap usage in the RPC layer
> ---
>
> Key: HBASE-18214
> URL: https://issues.apache.org/jira/browse/HBASE-18214
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Devaraj Das
>Assignee: Devaraj Das
> Attachments: 18214-1-1.txt, 18214-1-2.txt
>
>
> In my tests, I saw that folly::AtomicHashMap usage is not appropriate for 
> one, rather common use case. It'd become sort of unusable (inserts would 
> hang) after a bunch of inserts and erases. This hashmap is used to keep track 
> of call-Id after a connection is set up in the RPC layer (insert a 
> call-id/msg pair when an RPC is sent, and erase the pair when the 
> corresponding response is received). Here is a simple program that will 
> demonstrate the issue:
> {code}
> folly::AtomicHashMap f(100);
> int i = 0;
> while (i < 1) {
> try {
>   f.insert(i,100);
>   LOG(INFO) << "Inserted " << i << "  " << f.size();
>   f.erase(i);
>   LOG(INFO) << "Deleted " << i << "  " << f.size();
>   i++;
> } catch (const std::exception ) {
>   LOG(INFO) << "Exception " << e.what();
>   break;
> }
> }
> {code}
> After poking around a little bit, it is indeed called out as a limitation 
> here 
> https://github.com/facebook/folly/blob/master/folly/docs/AtomicHashMap.md 
> (grep for 'erase'). Proposal is to replace this with something that will fit 
> in in the above usecase (thinking of using std::unordered_map).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18214) Replace the folly::AtomicHashMap usage in the RPC layer

[Devaraj Das (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054931#comment-16054931
 ] 

Devaraj Das commented on HBASE-18214:
-

bq. Instead of  std::unique_ptr mutex_; use 
std::shared_timed_mutex mutex_;
The issue is the shared_timed_mutex is (rightfully) not copyable. So one would 
have to define explicit copy constructors in the classes where this is used... 
It's not clear as to what such a copy constructor would do anyway (mutexes are 
fundamentally not copyable safely). Treating it as a pointer in class 
declarations works around such things since all the instances in question would 
refer to the same underlying mutex. But maybe, we should declare it to be 
shared_ptr as opposed to a shared_ptr. In the usecase we have, we probably will 
not run into issues with either anyway I guess.

> Replace the folly::AtomicHashMap usage in the RPC layer
> ---
>
> Key: HBASE-18214
> URL: https://issues.apache.org/jira/browse/HBASE-18214
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Devaraj Das
>Assignee: Devaraj Das
> Attachments: 18214-1-1.txt, 18214-1-2.txt
>
>
> In my tests, I saw that folly::AtomicHashMap usage is not appropriate for 
> one, rather common use case. It'd become sort of unusable (inserts would 
> hang) after a bunch of inserts and erases. This hashmap is used to keep track 
> of call-Id after a connection is set up in the RPC layer (insert a 
> call-id/msg pair when an RPC is sent, and erase the pair when the 
> corresponding response is received). Here is a simple program that will 
> demonstrate the issue:
> {code}
> folly::AtomicHashMap f(100);
> int i = 0;
> while (i < 1) {
> try {
>   f.insert(i,100);
>   LOG(INFO) << "Inserted " << i << "  " << f.size();
>   f.erase(i);
>   LOG(INFO) << "Deleted " << i << "  " << f.size();
>   i++;
> } catch (const std::exception ) {
>   LOG(INFO) << "Exception " << e.what();
>   break;
> }
> }
> {code}
> After poking around a little bit, it is indeed called out as a limitation 
> here 
> https://github.com/facebook/folly/blob/master/folly/docs/AtomicHashMap.md 
> (grep for 'erase'). Proposal is to replace this with something that will fit 
> in in the above usecase (thinking of using std::unordered_map).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-16242) Upgrade Avro to 1.7.7

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054925#comment-16054925
 ] 

Hudson commented on HBASE-16242:


FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3225 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/3225/])
HBASE-16242 Manually resolve transitive Apache Avro dependency to (busbey: rev 
2263190361e26cfb85b0bbfd82cf56f5a6f85ca8)
* (edit) pom.xml
* (edit) hbase-common/pom.xml
* (edit) hbase-spark/pom.xml


> Upgrade Avro to 1.7.7
> -
>
> Key: HBASE-16242
> URL: https://issues.apache.org/jira/browse/HBASE-16242
> Project: HBase
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Ben McCann
>Assignee: Sean Busbey
> Fix For: 3.0.0, 1.4.0, 2.0.0-alpha-2
>
> Attachments: HBASE-16242.0.patch
>
>
> I'd like to see Avro upgraded to 1.8.1 or at the very least 1.7.7



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18214) Replace the folly::AtomicHashMap usage in the RPC layer

[Devaraj Das (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Devaraj Das updated HBASE-18214:

Attachment: 18214-1-2.txt

Attached patch makes the unordered_map usage slightly abstract since there are 
two places where similar functionality is required. A wrapper class MapUtil 
encapsulates the behavior we need. The thing doesn't compile yet. C++ template 
syntax needs to be dealt with.

> Replace the folly::AtomicHashMap usage in the RPC layer
> ---
>
> Key: HBASE-18214
> URL: https://issues.apache.org/jira/browse/HBASE-18214
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Devaraj Das
>Assignee: Devaraj Das
> Attachments: 18214-1-1.txt, 18214-1-2.txt
>
>
> In my tests, I saw that folly::AtomicHashMap usage is not appropriate for 
> one, rather common use case. It'd become sort of unusable (inserts would 
> hang) after a bunch of inserts and erases. This hashmap is used to keep track 
> of call-Id after a connection is set up in the RPC layer (insert a 
> call-id/msg pair when an RPC is sent, and erase the pair when the 
> corresponding response is received). Here is a simple program that will 
> demonstrate the issue:
> {code}
> folly::AtomicHashMap f(100);
> int i = 0;
> while (i < 1) {
> try {
>   f.insert(i,100);
>   LOG(INFO) << "Inserted " << i << "  " << f.size();
>   f.erase(i);
>   LOG(INFO) << "Deleted " << i << "  " << f.size();
>   i++;
> } catch (const std::exception ) {
>   LOG(INFO) << "Exception " << e.what();
>   break;
> }
> }
> {code}
> After poking around a little bit, it is indeed called out as a limitation 
> here 
> https://github.com/facebook/folly/blob/master/folly/docs/AtomicHashMap.md 
> (grep for 'erase'). Proposal is to replace this with something that will fit 
> in in the above usecase (thinking of using std::unordered_map).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-17954) Switch findbugs implementation to spotbugs

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054881#comment-16054881
 ] 

Sean Busbey commented on HBASE-17954:
-

+1 pushing as soon as my batch of test runs finish.

> Switch findbugs implementation to spotbugs
> --
>
> Key: HBASE-17954
> URL: https://issues.apache.org/jira/browse/HBASE-17954
> Project: HBase
>  Issue Type: Task
>  Components: build, community, test
>Reporter: Sean Busbey
>Assignee: Jan Hentschel
> Fix For: 2.0.0
>
> Attachments: HBASE-17954.master.001.patch
>
>
> The Hadoop folks got some nice finds out of switching their findbugs plugin 
> to use the new spotbugs fork instead, let's try it out too. (ref  
> HADOOP-14316 for details)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-17988) get-active-master.rb and draining_servers.rb no longer work

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054882#comment-16054882
 ] 

Sean Busbey commented on HBASE-17988:
-

+1 pushing as soon as my batch of test runs finish.

> get-active-master.rb and draining_servers.rb no longer work
> ---
>
> Key: HBASE-17988
> URL: https://issues.apache.org/jira/browse/HBASE-17988
> Project: HBase
>  Issue Type: Bug
>  Components: scripts
>Affects Versions: 2.0.0
>Reporter: Mike Drob
>Assignee: Chinmay Kulkarni
>Priority: Critical
> Fix For: 2.0.0, 3.0.0
>
> Attachments: HBASE-17988.002.patch, HBASE-17988.patch
>
>
> The scripts {{bin/get-active-master.rb}} and {{bin/draining_servers.rb}} no 
> longer work on current master branch. Here is an example error message:
> {noformat}
> $ bin/hbase-jruby bin/get-active-master.rb 
> NoMethodError: undefined method `masterAddressZNode' for 
> #
>at bin/get-active-master.rb:35
> {noformat}
> My initial probing suggests that this is likely due to movement that happened 
> in HBASE-16690. Perhaps instead of reworking the ruby, there is similar Java 
> functionality already existing somewhere.
> Putting priority at critical since it's impossible to know whether users rely 
> on the scripts.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-16351) do dependency license check via enforcer plugin

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054880#comment-16054880
 ] 

Sean Busbey commented on HBASE-16351:
-

+1 pushing as soon as my batch of test runs finish.

> do dependency license check via enforcer plugin
> ---
>
> Key: HBASE-16351
> URL: https://issues.apache.org/jira/browse/HBASE-16351
> Project: HBase
>  Issue Type: Improvement
>  Components: build, dependencies
>Reporter: Sean Busbey
>Assignee: Mike Drob
> Fix For: 1.4.0, 2.0.0-alpha-2
>
> Attachments: HBASE-16351.patch
>
>
> As a stop-gap measure we've made our velocity template fail things when we 
> attempt to bundle a cat-x dependency (see HBASE-16318). Unfortunately, the 
> error messages in this case are non-obvious and digging to find the culprit 
> in a partially rendered LICENSE file leaves a lot to be desired.
> The maven enforcer plugin should allow us to fail more gracefully, with 
> output given on the maven console.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18230) Generated LICENSE file includes unsubstituted Velocity variables

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054883#comment-16054883
 ] 

Sean Busbey commented on HBASE-18230:
-

+1 pushing as soon as my batch of test runs finish.

> Generated LICENSE file includes unsubstituted Velocity variables
> 
>
> Key: HBASE-18230
> URL: https://issues.apache.org/jira/browse/HBASE-18230
> Project: HBase
>  Issue Type: Bug
>  Components: build
>Affects Versions: 2.0.0-alpha-1
>Reporter: Mike Drob
>Assignee: Mike Drob
> Fix For: 1.4.0, 2.0.0-alpha-2
>
> Attachments: HBASE-18230.patch
>
>
> From the release vote:
> {quote}
> we have a ton of places where we have velocity variables instead of
> copyright years, but IIRC that's a problem on branch-1 right now too.
> {quote}
> This is referring to lines like these:
> {noformat}
>   * javax.annotation-api, ${dep.licenses[0].comments}
>   * javax.servlet-api, ${dep.licenses[0].comments}
>   * jetty-schemas, ${dep.licenses[0].comments}
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18212) In Standalone mode with local filesystem HBase logs Warning message:Failed to invoke 'unbuffer' method in class class org.apache.hadoop.fs.FSDataInputStream

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054878#comment-16054878
 ] 

Sean Busbey commented on HBASE-18212:
-

+1 pushing as soon as my batch of test runs finish.

> In Standalone mode with local filesystem HBase logs Warning message:Failed to 
> invoke 'unbuffer' method in class class org.apache.hadoop.fs.FSDataInputStream
> 
>
> Key: HBASE-18212
> URL: https://issues.apache.org/jira/browse/HBASE-18212
> Project: HBase
>  Issue Type: Bug
>  Components: Operability
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Umesh Agashe
>Assignee: Ashish Singhi
>Priority: Minor
> Attachments: HBASE-18212.patch
>
>
> New users may get nervous after seeing following warning level log messages 
> (considering new users will most likely run HBase in Standalone mode first):
> {code}
> WARN  [MemStoreFlusher.1] io.FSDataInputStreamWrapper: Failed to invoke 
> 'unbuffer' method in class class org.apache.hadoop.fs.FSDataInputStream . So 
> there may be a TCP socket connection left open in CLOSE_WAIT state.
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (HBASE-15737) Remove use of Guava Stopwatch

[Martin Serrano (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054865#comment-16054865
 ] 

Martin Serrano edited comment on HBASE-15737 at 6/19/17 10:32 PM:
--

Use of {{StopWatch.elapsed()}} instead of {{StopWatch.ellapsedMillis()}} is 
compatible with guava 14.0 and later.  {{StopWatch.ellapsedMillis()}} went away 
in 16.0.



was (Author: mserrano):
Use of `StopWatch.elapsed()` instead of `StopWatch.ellapsedMillis()` is 
compatible with guava 14.0 and later.  `StopWatch.ellapsedMillis()` went away 
in 16.0.


> Remove use of Guava Stopwatch
> -
>
> Key: HBASE-15737
> URL: https://issues.apache.org/jira/browse/HBASE-15737
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Attachments: 15737.v1.txt, 15737.v2.txt, 15737.v3.txt
>
>
> HBASE-14963 removed reference to Guava Stopwatch from hbase-client module.
> However, there're still 3 classes referring to Guava Stopwatch :
> hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestClientNoCluster.java:import
>  com.google.common.base.Stopwatch;
> hbase-server/src/main/java/org/apache/hadoop/hbase/util/JvmPauseMonitor.java:import
>  com.google.common.base.Stopwatch;
> hbase-server/src/test/java/org/apache/hadoop/hbase/ScanPerformanceEvaluation.java:import
>  com.google.common.base.Stopwatch;
> We should remove reference to Guava Stopwatch.
> hadoop is no longer referencing Guava Stopwatch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-15737) Remove use of Guava Stopwatch

[Martin Serrano (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054865#comment-16054865
 ] 

Martin Serrano commented on HBASE-15737:


Use of `StopWatch.elapsed()` instead of `StopWatch.ellapsedMillis()` is 
compatible with guava 14.0 and later.  `StopWatch.ellapsedMillis()` went away 
in 16.0.


> Remove use of Guava Stopwatch
> -
>
> Key: HBASE-15737
> URL: https://issues.apache.org/jira/browse/HBASE-15737
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Attachments: 15737.v1.txt, 15737.v2.txt, 15737.v3.txt
>
>
> HBASE-14963 removed reference to Guava Stopwatch from hbase-client module.
> However, there're still 3 classes referring to Guava Stopwatch :
> hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestClientNoCluster.java:import
>  com.google.common.base.Stopwatch;
> hbase-server/src/main/java/org/apache/hadoop/hbase/util/JvmPauseMonitor.java:import
>  com.google.common.base.Stopwatch;
> hbase-server/src/test/java/org/apache/hadoop/hbase/ScanPerformanceEvaluation.java:import
>  com.google.common.base.Stopwatch;
> We should remove reference to Guava Stopwatch.
> hadoop is no longer referencing Guava Stopwatch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-15749) Shade guava dependency

[Martin Serrano (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054861#comment-16054861
 ] 

Martin Serrano commented on HBASE-15749:


Although this is closed now, here is some context for future reference.

* LocalHBaseCluster is used for testing
* JvmPauseMonitor is used by LocalHBaseCluster
* My environment must now use a guava that is compatible with HBase server 
dependencies in order to run client tests.
* :(





> Shade guava dependency
> --
>
> Key: HBASE-15749
> URL: https://issues.apache.org/jira/browse/HBASE-15749
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>
> HBase codebase uses Guava library extensively.
> There have been JIRAs such as HBASE-14963 which tried to make compatibility 
> story around Guava better.
> Long term fix, as suggested over in HBASE-14963, is to shade Guava dependency.
> Future use of Guava in HBase would be more secure once shading is done.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-17924) Consider sorting the row order when processing multi() ops before taking rowlocks

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054860#comment-16054860
 ] 

Jerry He commented on HBASE-17924:
--

Thanks for the pointer. I will be reading thru the latest goodies on 
HBASE-18144.

> Consider sorting the row order when processing multi() ops before taking 
> rowlocks
> -
>
> Key: HBASE-17924
> URL: https://issues.apache.org/jira/browse/HBASE-17924
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0, 1.1.8
>Reporter: Andrew Purtell
>Assignee: Allan Yang
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17924.patch, HBASE-17924.v0.patch, 
> HBASE-17924.v2.patch, HBASE-17924.v3.patch, HBASE-17924.v4.patch, 
> HBASE-17924.v5.patch
>
>
> When processing a batch mutation, we take row locks in whatever order the 
> mutations were added to the multi op by the client.
>  
> {noformat}
> RSRpcServices#multi -> RSRpcServices#mutateRows -> HRegion#mutateRow -> 
> HRegion#mutateRowsWithLocks -> HRegion#processRowsWithLocks
> {noformat}
> Or
> {noformat}
> RSRpcServices#multi -> RSRpcServices#doNonAtomicRegionMutation ->
>   HRegion#get 
> | HRegion#append 
> | HRegion#increment 
> | HRegionServer#doBatchOp -> HRegion#batchMutate -> 
> HRegion#doMiniBatchMutation
> {noformat}
>  
> multi() is fed by client APIs that accept a RowMutations object containing 
> actions for multiple rows. The container for ops inside RowMutations is an 
> ArrayList, which doesn't change the ordering of objects added to it. The 
> protobuf implementation of the messages for multi ops do not reorder the list 
> of actions. When processing multi ops we iterate over the actions in the 
> order rehydrated from protobuf.
> We should discuss sorting the order of ops by row key when processing multi() 
> ops before taking row locks. Does this make lock ordering more predictable 
> for server side operations? Yes, but potentially surprising for the client, 
> right? Is there any legitimate reason we should take locks out of row key 
> sorted order because the client has structured the request as such?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-16415) Replication in different namespace

[Guanghao Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054827#comment-16054827
 ] 

Guanghao Zhang commented on HBASE-16415:


Why not extends HBaseInterClusterReplicationEndpoint and only override 
createBatches method?

bq. It seems option 3 is better.
redirect() is only a behavior of RedirectingInterClusterReplicationEndpoint. So 
it is not good to add it to the parent class? Thanks.

> Replication in different namespace
> --
>
> Key: HBASE-16415
> URL: https://issues.apache.org/jira/browse/HBASE-16415
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: Christian Guegi
>Assignee: Jan Kunigk
>
> It would be nice to replicate tables from one namespace to another namespace.
> Example:
> Master cluster, namespace=default, table=bar
> Slave cluster, namespace=dr, table=bar
> Replication happens in class ReplicationSink:
>   public void replicateEntries(List entries, final CellScanner 
> cells, ...){
> ...
> TableName table = 
> TableName.valueOf(entry.getKey().getTableName().toByteArray());
> ...
> addToHashMultiMap(rowMap, table, clusterIds, m);
> ...
> for (Entry> entry : 
> rowMap.entrySet()) {
>   batch(entry.getKey(), entry.getValue().values());
> }
>}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-17931) Assign system tables to servers with highest version

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-17931:

Component/s: Region Assignment

> Assign system tables to servers with highest version
> 
>
> Key: HBASE-17931
> URL: https://issues.apache.org/jira/browse/HBASE-17931
> Project: HBase
>  Issue Type: Bug
>  Components: Region Assignment, scan
>Reporter: Phil Yang
>Assignee: Phil Yang
>Priority: Blocker
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17931.branch-1.v01.patch, 
> HBASE-17931.branch-1.v02.patch, HBASE-17931.v01.patch, HBASE-17931.v02.patch, 
> HBASE-17931.v03.patch
>
>
> In branch-1 and master we have some improvement and new features on scanning 
> which is not compatible.
> A client of old version to a server of new version is compatible (must be a 
> bug if not, maybe need some test?). 
> A client of new version may not be able to read from a server of old version 
> correctly (because of scan limit, moreResults flag, etc), which is ok for 
> major/minor upgrade and we can tell users to upgrade server before upgrading 
> client. But RS also use scan to read meta. If meta table is in RS of old 
> version, all RSs of new version may have trouble while scanning meta table.
> So we should make sure meta table always in servers of new version. Force 
> meta table in Master and upgrade Master first, or assign meta table in region 
> servers with latest version?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-17988) get-active-master.rb and draining_servers.rb no longer work

[Mike Drob (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054812#comment-16054812
 ] 

Mike Drob commented on HBASE-17988:
---

Yea. We can take care of any lint issues on HBASE-18238

> get-active-master.rb and draining_servers.rb no longer work
> ---
>
> Key: HBASE-17988
> URL: https://issues.apache.org/jira/browse/HBASE-17988
> Project: HBase
>  Issue Type: Bug
>  Components: scripts
>Affects Versions: 2.0.0
>Reporter: Mike Drob
>Assignee: Chinmay Kulkarni
>Priority: Critical
> Fix For: 2.0.0, 3.0.0
>
> Attachments: HBASE-17988.002.patch, HBASE-17988.patch
>
>
> The scripts {{bin/get-active-master.rb}} and {{bin/draining_servers.rb}} no 
> longer work on current master branch. Here is an example error message:
> {noformat}
> $ bin/hbase-jruby bin/get-active-master.rb 
> NoMethodError: undefined method `masterAddressZNode' for 
> #
>at bin/get-active-master.rb:35
> {noformat}
> My initial probing suggests that this is likely due to movement that happened 
> in HBASE-16690. Perhaps instead of reworking the ruby, there is similar Java 
> functionality already existing somewhere.
> Putting priority at critical since it's impossible to know whether users rely 
> on the scripts.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-17766) Generate Javadoc for hbase-spark module

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-17766:

Status: In Progress  (was: Patch Available)

moving out of patch available pending finding on scaladoc

> Generate Javadoc for hbase-spark module 
> 
>
> Key: HBASE-17766
> URL: https://issues.apache.org/jira/browse/HBASE-17766
> Project: HBase
>  Issue Type: Bug
>  Components: documentation, spark
>Reporter: Yi Liang
>Assignee: Yi Liang
> Fix For: 2.0.0
>
> Attachments: HBASE-17766-V1.patch, spark-api.jpg, user-api.jpg
>
>
>  Scala classes in hbase-spark module aren't showing up in our API docs nor 
> our internal API docs. see https://hbase.apache.org/apidocs/ 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18212) In Standalone mode with local filesystem HBase logs Warning message:Failed to invoke 'unbuffer' method in class class org.apache.hadoop.fs.FSDataInputStream

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-18212:

Component/s: Operability

> In Standalone mode with local filesystem HBase logs Warning message:Failed to 
> invoke 'unbuffer' method in class class org.apache.hadoop.fs.FSDataInputStream
> 
>
> Key: HBASE-18212
> URL: https://issues.apache.org/jira/browse/HBASE-18212
> Project: HBase
>  Issue Type: Bug
>  Components: Operability
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Umesh Agashe
>Assignee: Ashish Singhi
>Priority: Minor
> Attachments: HBASE-18212.patch
>
>
> New users may get nervous after seeing following warning level log messages 
> (considering new users will most likely run HBase in Standalone mode first):
> {code}
> WARN  [MemStoreFlusher.1] io.FSDataInputStreamWrapper: Failed to invoke 
> 'unbuffer' method in class class org.apache.hadoop.fs.FSDataInputStream . So 
> there may be a TCP socket connection left open in CLOSE_WAIT state.
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18212) In Standalone mode with local filesystem HBase logs Warning message:Failed to invoke 'unbuffer' method in class class org.apache.hadoop.fs.FSDataInputStream

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-18212:

Priority: Minor  (was: Major)

> In Standalone mode with local filesystem HBase logs Warning message:Failed to 
> invoke 'unbuffer' method in class class org.apache.hadoop.fs.FSDataInputStream
> 
>
> Key: HBASE-18212
> URL: https://issues.apache.org/jira/browse/HBASE-18212
> Project: HBase
>  Issue Type: Bug
>  Components: Operability
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Umesh Agashe
>Assignee: Ashish Singhi
>Priority: Minor
> Attachments: HBASE-18212.patch
>
>
> New users may get nervous after seeing following warning level log messages 
> (considering new users will most likely run HBase in Standalone mode first):
> {code}
> WARN  [MemStoreFlusher.1] io.FSDataInputStreamWrapper: Failed to invoke 
> 'unbuffer' method in class class org.apache.hadoop.fs.FSDataInputStream . So 
> there may be a TCP socket connection left open in CLOSE_WAIT state.
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HBASE-18212) In Standalone mode with local filesystem HBase logs Warning message:Failed to invoke 'unbuffer' method in class class org.apache.hadoop.fs.FSDataInputStream

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey reassigned HBASE-18212:
---

Assignee: Ashish Singhi

> In Standalone mode with local filesystem HBase logs Warning message:Failed to 
> invoke 'unbuffer' method in class class org.apache.hadoop.fs.FSDataInputStream
> 
>
> Key: HBASE-18212
> URL: https://issues.apache.org/jira/browse/HBASE-18212
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Umesh Agashe
>Assignee: Ashish Singhi
> Attachments: HBASE-18212.patch
>
>
> New users may get nervous after seeing following warning level log messages 
> (considering new users will most likely run HBase in Standalone mode first):
> {code}
> WARN  [MemStoreFlusher.1] io.FSDataInputStreamWrapper: Failed to invoke 
> 'unbuffer' method in class class org.apache.hadoop.fs.FSDataInputStream . So 
> there may be a TCP socket connection left open in CLOSE_WAIT state.
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18197) import.java, job output is printing two times.

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054799#comment-16054799
 ] 

Hadoop QA commented on HBASE-18197:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} docker {color} | {color:blue} 0m 5s 
{color} | {color:blue} Dockerfile 
'/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/docker/Dockerfile'
 not found, falling back to built-in. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 4m 21s 
{color} | {color:red} Docker failed to build yetus/hbase:date2017-06-19. 
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12872457/HBASE-18197.branch-1.0.001.patch
 |
| JIRA Issue | HBASE-18197 |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/7239/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> import.java, job output is printing two times.
> --
>
> Key: HBASE-18197
> URL: https://issues.apache.org/jira/browse/HBASE-18197
> Project: HBase
>  Issue Type: Bug
>  Components: hbase
>Affects Versions: 1.0.2, 1.2.0, 1.4.0
>Reporter: Chandra Sekhar
>Assignee: Jan Hentschel
>Priority: Trivial
> Attachments: HBASE-18197.branch-1.0.001.patch, 
> HBASE-18197.branch-1.2.001.patch
>
>
> import.java, job output is printing two times.
> {quote}
> after job completed, job.waitForCompletion(true) is calling two times.
> {quote}
> {code}
> Job job = createSubmittableJob(conf, otherArgs);
> boolean isJobSuccessful = job.waitForCompletion(true);
> if(isJobSuccessful){
>   // Flush all the regions of the table
>   flushRegionsIfNecessary(conf);
> }
> long inputRecords = 
> job.getCounters().findCounter(TaskCounter.MAP_INPUT_RECORDS).getValue();
> long outputRecords = 
> job.getCounters().findCounter(TaskCounter.MAP_OUTPUT_RECORDS).getValue();
> if (outputRecords < inputRecords) {
>   System.err.println("Warning, not all records were imported (maybe 
> filtered out).");
>   if (outputRecords == 0) {
> System.err.println("If the data was exported from HBase 0.94 "+
> "consider using -Dhbase.import.version=0.94.");
>   }
> }
> System.exit(job.waitForCompletion(true) ? 0 : 1);
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18023) Log multi-* requests for more than threshold number of rows

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-18023:

Status: In Progress  (was: Patch Available)

moving out of patch available pending update for review feedback.

> Log multi-* requests for more than threshold number of rows
> ---
>
> Key: HBASE-18023
> URL: https://issues.apache.org/jira/browse/HBASE-18023
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Reporter: Clay B.
>Assignee: David Harju
>Priority: Minor
> Attachments: HBASE-18023.master.001.patch
>
>
> Today, if a user happens to do something like a large multi-put, they can get 
> through request throttling (e.g. it is one request) but still crash a region 
> server with a garbage storm. We have seen regionservers hit this issue and it 
> is silent and deadly. The RS will report nothing more than a mysterious 
> garbage collection and exit out.
> Ideally, we could report a large multi-* request before starting it, in case 
> it happens to be deadly. Knowing the client, user and how many rows are 
> affected would be a good start to tracking down painful users.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-17988) get-active-master.rb and draining_servers.rb no longer work

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054795#comment-16054795
 ] 

Sean Busbey commented on HBASE-17988:
-

[~mdrob] you still +1?

> get-active-master.rb and draining_servers.rb no longer work
> ---
>
> Key: HBASE-17988
> URL: https://issues.apache.org/jira/browse/HBASE-17988
> Project: HBase
>  Issue Type: Bug
>  Components: scripts
>Affects Versions: 2.0.0
>Reporter: Mike Drob
>Assignee: Chinmay Kulkarni
>Priority: Critical
> Fix For: 2.0.0, 3.0.0
>
> Attachments: HBASE-17988.002.patch, HBASE-17988.patch
>
>
> The scripts {{bin/get-active-master.rb}} and {{bin/draining_servers.rb}} no 
> longer work on current master branch. Here is an example error message:
> {noformat}
> $ bin/hbase-jruby bin/get-active-master.rb 
> NoMethodError: undefined method `masterAddressZNode' for 
> #
>at bin/get-active-master.rb:35
> {noformat}
> My initial probing suggests that this is likely due to movement that happened 
> in HBASE-16690. Perhaps instead of reworking the ruby, there is similar Java 
> functionality already existing somewhere.
> Putting priority at critical since it's impossible to know whether users rely 
> on the scripts.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18226) Disable reverse DNS lookup at HMaster and use default hostname provided by RegionServer

[Duo Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054796#comment-16054796
 ] 

Duo Xu commented on HBASE-18226:


Failed tests are not related to the patch. My replaced jar also worked in the 
target cluster.

Now we need to come up with a good name for the configuration. How about 
"hbase.regionserver.hostname.disable.master.reversedns"?

> Disable reverse DNS lookup at HMaster and use default hostname provided by 
> RegionServer
> ---
>
> Key: HBASE-18226
> URL: https://issues.apache.org/jira/browse/HBASE-18226
> Project: HBase
>  Issue Type: Bug
>Reporter: Duo Xu
> Attachments: HBASE-18226.001.patch, HBASE-18226.002.patch, 
> HBASE-18226.003.patch, HBASE-18226.004.patch
>
>
> Description updated:
> In some unusual network environment, forward DNS lookup is supported while 
> reverse DNS lookup may not work properly.
> This JIRA is to address that HMaster uses the hostname passed from RS instead 
> of doing reverse DNS lookup to tells RS which hostname to use during 
> reportForDuty() . This has already been implemented by HBASE-12954 by adding 
> "useThisHostnameInstead" field in RegionServerStatusProtos.
> Currently "useThisHostnameInstead" is optional and RS by default only passes 
> port, server start code and server current time info to HMaster during RS 
> reportForDuty(). In order to use this field, users currently need to specify 
> "hbase.regionserver.hostname" on every regionserver node's hbase-site.xml. 
> This causes some trouble in
> 1. some deployments managed by some management tools like Ambari, which 
> maintains the same copy of hbase-site.xml across all the nodes.
> 2. HBASE-12954 is targeting multihomed hosts, which users want to manually 
> set the hostname value for each node. In the other cases (not multihomed), I 
> just want RS to use the hostname return by the node and set it in 
> useThisHostnameInstead and pass to HMaster during reportForDuty().
> I would like to introduce a setting that if the setting is set to true, 
> "useThisHostnameInstead" will be set to the hostname RS gets from the node. 
> Then HMaster will skip reverse DNS lookup because it sees 
> "useThisHostnameInstead" field is set in the request.
> "hbase.regionserver.hostname.reported.to.master", is it a good name?
> 
> Regarding the hostname returned by the RS node, I read the source code again 
> (including hadoop-common dns.java). By default RS gets hostname by calling 
> InetAddress.getLocalHost().getCanonicalHostName(). If users specify 
> "hbase.regionserver.dns.interface" or "hbase.regionserver.dns.nameserver" or 
> some underlying system configuration changes (eg. modifying 
> /etc/nsswitch.conf), it may first read from DNS or other sources instead of 
> first checking /etc/hosts file.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18185) IntegrationTestTimeBoundedRequestsWithRegionReplicas unbalanced tests fails with AssertionError

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-18185:

Status: In Progress  (was: Patch Available)

moving out of patch available pending update for review

> IntegrationTestTimeBoundedRequestsWithRegionReplicas unbalanced tests fails 
> with AssertionError
> ---
>
> Key: HBASE-18185
> URL: https://issues.apache.org/jira/browse/HBASE-18185
> Project: HBase
>  Issue Type: Bug
>  Components: integration tests
>Affects Versions: 2.0.0
>Reporter: Balazs Meszaros
>Assignee: Balazs Meszaros
>Priority: Minor
> Fix For: 2.0.0, 1.4.0, 1.3.2, 1.2.7
>
> Attachments: HBASE-18185-BM-0001.patch, HBASE-18185-BM-0002.patch
>
>
> We got the following error:
> Exception in thread "main" java.lang.AssertionError: Verification failed with 
> error code 1
> at org.junit.Assert.fail(Assert.java:88)
> at 
> org.apache.hadoop.hbase.test.IntegrationTestTimeBoundedRequestsWithRegionReplicas.runIngestTest(IntegrationTestTimeBoundedRequestsWithRegionReplicas.java:217)
> at 
> org.apache.hadoop.hbase.IntegrationTestIngest.internalRunIngestTest(IntegrationTestIngest.java:123)
> at 
> org.apache.hadoop.hbase.IntegrationTestIngest.runTestFromCommandLine(IntegrationTestIngest.java:106)
> at 
> org.apache.hadoop.hbase.IntegrationTestBase.doWork(IntegrationTestBase.java:123)
> at 
> org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:112)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at 
> org.apache.hadoop.hbase.test.IntegrationTestTimeBoundedRequestsWithRegionReplicas.main(IntegrationTestTimeBoundedRequestsWithRegionReplicas.java:362)
> The reason why we got it because another assertion fails in 
> UnbalanceKillAndRebalanceAction:
> Exception in thread "Thread-57" java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at 
> org.apache.hadoop.hbase.chaos.actions.UnbalanceKillAndRebalanceAction.perform(UnbalanceKillAndRebalanceAction.java:60)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-17546) Incorrect syntax at HBase-Spark Module Examples

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-17546:

   Resolution: Fixed
Fix Version/s: (was: 1.1.8)
   2.0.0-alpha-2
   3.0.0
 Tags:   (was: #HBasespark)
   Status: Resolved  (was: Patch Available)

pushed to master and branch-2. Thanks for the contribution [~chetkhatri] and 
sorry for the lag!

> Incorrect syntax at HBase-Spark Module Examples
> ---
>
> Key: HBASE-17546
> URL: https://issues.apache.org/jira/browse/HBASE-17546
> Project: HBase
>  Issue Type: Bug
>  Components: spark
>Affects Versions: 1.1.8
>Reporter: Chetan Khatri
>Assignee: Chetan Khatri
>Priority: Minor
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-17546.master.001.patch
>
>
> Corrected : Syntax errors at HBase-Spark module
> Description:
>  - Syntax errors has been correct at HBase-Spark module examples.
> - Spark Transformation : somewhere show() and somewhere show only, corrected 
> to show().



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18226) Disable reverse DNS lookup at HMaster and use default hostname provided by RegionServer

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054769#comment-16054769
 ] 

Hadoop QA commented on HBASE-18226:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m 48s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 29s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 
6s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 12s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
4s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 12s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 12s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s 
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
33m 40s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha3. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 8s 
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 158m 19s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
41s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 232m 22s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.wal.TestAsyncLogRolling |
| Timed out junit tests | 
org.apache.hadoop.hbase.replication.multiwal.TestReplicationKillMasterRSCompressedWithMultipleAsyncWAL
 |
|   | org.apache.hadoop.hbase.TestRegionLoad |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.10.1 Server=1.10.1 Image:yetus/hbase:757bf37 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12873525/HBASE-18226.004.patch 
|
| JIRA Issue | HBASE-18226 |
| Optional Tests |  asflicense  javac  javadoc  unit  xml  findbugs  
hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux 1638afb24744 3.19.0-25-generic #26~14.04.1-Ubuntu SMP Fri Jul 
24 21:16:20 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / ce1ce72 |
| 

[jira] [Updated] (HBASE-17546) Incorrect syntax at HBase-Spark Module Examples

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-17546:

Priority: Minor  (was: Major)

> Incorrect syntax at HBase-Spark Module Examples
> ---
>
> Key: HBASE-17546
> URL: https://issues.apache.org/jira/browse/HBASE-17546
> Project: HBase
>  Issue Type: Bug
>  Components: spark
>Affects Versions: 1.1.8
>Reporter: Chetan Khatri
>Assignee: Chetan Khatri
>Priority: Minor
> Fix For: 1.1.8
>
> Attachments: HBASE-17546.master.001.patch
>
>
> Corrected : Syntax errors at HBase-Spark module
> Description:
>  - Syntax errors has been correct at HBase-Spark module examples.
> - Spark Transformation : somewhere show() and somewhere show only, corrected 
> to show().



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (HBASE-16246) Ensure avro version in hbase-spark matches that shipped in our default hadoop profile

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey resolved HBASE-16246.
-
   Resolution: Duplicate
Fix Version/s: (was: 2.0.0)

closing this out as a dup of HBASE-16242, since that forces the same version 
everywhere.

> Ensure avro version in hbase-spark matches that shipped in our default hadoop 
> profile
> -
>
> Key: HBASE-16246
> URL: https://issues.apache.org/jira/browse/HBASE-16246
> Project: HBase
>  Issue Type: Bug
>  Components: spark
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Priority: Blocker
>
> currently, the hbase-spark module pulls in avro 1.7.6. We should keep this 
> matching the version in our default hadoop to keep things consistent (as of 
> Hadoop 2.7.1 that's avro 1.7.4).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-16242) Upgrade Avro

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-16242:

   Resolution: Fixed
Fix Version/s: 1.4.0
   Status: Resolved  (was: Patch Available)

> Upgrade Avro
> 
>
> Key: HBASE-16242
> URL: https://issues.apache.org/jira/browse/HBASE-16242
> Project: HBase
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Ben McCann
>Assignee: Sean Busbey
> Fix For: 3.0.0, 1.4.0, 2.0.0-alpha-2
>
> Attachments: HBASE-16242.0.patch
>
>
> I'd like to see Avro upgraded to 1.8.1 or at the very least 1.7.7



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-16242) Upgrade Avro to 1.7.7

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-16242:

Summary: Upgrade Avro to 1.7.7  (was: Upgrade Avro)

> Upgrade Avro to 1.7.7
> -
>
> Key: HBASE-16242
> URL: https://issues.apache.org/jira/browse/HBASE-16242
> Project: HBase
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Ben McCann
>Assignee: Sean Busbey
> Fix For: 3.0.0, 1.4.0, 2.0.0-alpha-2
>
> Attachments: HBASE-16242.0.patch
>
>
> I'd like to see Avro upgraded to 1.8.1 or at the very least 1.7.7



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18238) Address ruby static analysis for bin directory

[Mike Drob (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mike Drob updated HBASE-18238:
--
Attachment: report.txt

> Address ruby static analysis for bin directory
> --
>
> Key: HBASE-18238
> URL: https://issues.apache.org/jira/browse/HBASE-18238
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Mike Drob
> Fix For: 2.0.0
>
> Attachments: report.txt
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18237) Address ruby static analysis reports

[Mike Drob (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054732#comment-16054732
 ] 

Mike Drob commented on HBASE-18237:
---

Even though the static analysis doesn't work through Yetus, folks can still run 
it locally and check the results.

> Address ruby static analysis reports
> 
>
> Key: HBASE-18237
> URL: https://issues.apache.org/jira/browse/HBASE-18237
> Project: HBase
>  Issue Type: Improvement
>Reporter: Mike Drob
> Fix For: 2.0.0
>
>
> Parent issue for addressing the static analysis findings for our ruby sources.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-18239) Address ruby static analysis for shell module

[Mike Drob (JIRA)]

Mike Drob created HBASE-18239:
-

 Summary: Address ruby static analysis for shell module
 Key: HBASE-18239
 URL: https://issues.apache.org/jira/browse/HBASE-18239
 Project: HBase
  Issue Type: Sub-task
Reporter: Mike Drob






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-18238) Address ruby static analysis for bin directory

[Mike Drob (JIRA)]

Mike Drob created HBASE-18238:
-

 Summary: Address ruby static analysis for bin directory
 Key: HBASE-18238
 URL: https://issues.apache.org/jira/browse/HBASE-18238
 Project: HBase
  Issue Type: Sub-task
Reporter: Mike Drob






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-18237) Address ruby static analysis reports

[Mike Drob (JIRA)]

Mike Drob created HBASE-18237:
-

 Summary: Address ruby static analysis reports
 Key: HBASE-18237
 URL: https://issues.apache.org/jira/browse/HBASE-18237
 Project: HBase
  Issue Type: Improvement
Reporter: Mike Drob
 Fix For: 2.0.0


Parent issue for addressing the static analysis findings for our ruby sources.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-16148) Hybrid Logical Clocks(placeholder for running tests)

[Amit Patel (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Patel updated HBASE-16148:
---
Attachment: HBASE-16148.master.008.patch

> Hybrid Logical Clocks(placeholder for running tests)
> 
>
> Key: HBASE-16148
> URL: https://issues.apache.org/jira/browse/HBASE-16148
> Project: HBase
>  Issue Type: Sub-task
>  Components: API
>Reporter: Sai Teja Ranuva
>Assignee: Sai Teja Ranuva
>Priority: Minor
>  Labels: test-patch
> Attachments: HBASE-16148.master.001.patch, 
> HBASE-16148.master.002.patch, HBASE-16148.master.003.patch, 
> HBASE-16148.master.004.patch, HBASE-16148.master.005.patch, 
> HBASE-16148.master.006.patch, HBASE-16148.master.007.patch, 
> HBASE-16148.master.008.patch, HBASE-16148.master.6.patch, 
> HBASE-16148.master.test.1.patch, HBASE-16148.master.test.2.patch, 
> HBASE-16148.master.test.3.patch, HBASE-16148.master.test.4.patch, 
> HBASE-16148.master.test.5.patch, HLC.10.1.patch, HLC.10.2.patch, 
> HLC.10.3.patch, HLC.10.4.patch, HLC.10.5.patch, HLC.10.6.patch, 
> HLC.10.7.patch, HLC.10.patch, HLC.1.patch, HLC.2.patch, HLC.3.patch, 
> HLC.4.patch, HLC.5.patch, HLC.6.patch, HLC.8.patch, HLC.9.patch, HLC.patch
>
>
> This JIRA is just a placeholder to test Hybrid Logical Clocks code.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-16148) Hybrid Logical Clocks(placeholder for running tests)

[Amit Patel (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054710#comment-16054710
 ] 

Amit Patel commented on HBASE-16148:


v Test with only meta table as HLC (removed server-side check for client 
setting timestamp in PUTs, included mapping of time ranges in GETs with respect 
to clock type). Does not yet include updating of clock for open/close region 
(will be added in subsequent post). ADDENDUM: ignored TestLockProcedure

> Hybrid Logical Clocks(placeholder for running tests)
> 
>
> Key: HBASE-16148
> URL: https://issues.apache.org/jira/browse/HBASE-16148
> Project: HBase
>  Issue Type: Sub-task
>  Components: API
>Reporter: Sai Teja Ranuva
>Assignee: Sai Teja Ranuva
>Priority: Minor
>  Labels: test-patch
> Attachments: HBASE-16148.master.001.patch, 
> HBASE-16148.master.002.patch, HBASE-16148.master.003.patch, 
> HBASE-16148.master.004.patch, HBASE-16148.master.005.patch, 
> HBASE-16148.master.006.patch, HBASE-16148.master.007.patch, 
> HBASE-16148.master.6.patch, HBASE-16148.master.test.1.patch, 
> HBASE-16148.master.test.2.patch, HBASE-16148.master.test.3.patch, 
> HBASE-16148.master.test.4.patch, HBASE-16148.master.test.5.patch, 
> HLC.10.1.patch, HLC.10.2.patch, HLC.10.3.patch, HLC.10.4.patch, 
> HLC.10.5.patch, HLC.10.6.patch, HLC.10.7.patch, HLC.10.patch, HLC.1.patch, 
> HLC.2.patch, HLC.3.patch, HLC.4.patch, HLC.5.patch, HLC.6.patch, HLC.8.patch, 
> HLC.9.patch, HLC.patch
>
>
> This JIRA is just a placeholder to test Hybrid Logical Clocks code.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HBASE-18236) [C++] Add batching and reporting to simple-client

[Enis Soztutar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Enis Soztutar updated HBASE-18236:
--
Attachment: hbase-simple-client-batch.patch

> [C++] Add batching and reporting to simple-client
> -
>
> Key: HBASE-18236
> URL: https://issues.apache.org/jira/browse/HBASE-18236
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: HBASE-14850
>
> Attachments: hbase-simple-client-batch.patch
>
>
> Simple change to add some batching configuration to simple-client to help 
> testing multi-get and multi-puts with large number of rows. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HBASE-18236) [C++] Add batching and reporting to simple-client

[Enis Soztutar (JIRA)]

Enis Soztutar created HBASE-18236:
-

 Summary: [C++] Add batching and reporting to simple-client
 Key: HBASE-18236
 URL: https://issues.apache.org/jira/browse/HBASE-18236
 Project: HBase
  Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Fix For: HBASE-14850


Simple change to add some batching configuration to simple-client to help 
testing multi-get and multi-puts with large number of rows. 





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (HBASE-18226) Disable reverse DNS lookup at HMaster and use default hostname provided by RegionServer

[Duo Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054692#comment-16054692
 ] 

Duo Xu edited comment on HBASE-18226 at 6/19/17 8:37 PM:
-

Regarding why we don't add a custom DNS server into our unsecure deployment, 
this is a decision made by the HDInsight deployment team due to some history. I 
can confirm that the current default HDInsight clusters does not support rDNS, 
but in custom virtual network where customers deploy the cluster, customers can 
deploy their own custom DNS server and enable rDNS for the cluster.


was (Author: onpduo):
Regarding why we don't add a custom DNS server into our unsecure deployment, 
this is a decision made by the HDInsight deployment team due to some history. I 
can confirm that the current default HDInsight clusters does not support rDNS, 
but in custom virtual network where they deploy the cluster, customers can 
deploy their own custom DNS and enable rDNS for the cluster.

> Disable reverse DNS lookup at HMaster and use default hostname provided by 
> RegionServer
> ---
>
> Key: HBASE-18226
> URL: https://issues.apache.org/jira/browse/HBASE-18226
> Project: HBase
>  Issue Type: Bug
>Reporter: Duo Xu
> Attachments: HBASE-18226.001.patch, HBASE-18226.002.patch, 
> HBASE-18226.003.patch, HBASE-18226.004.patch
>
>
> Description updated:
> In some unusual network environment, forward DNS lookup is supported while 
> reverse DNS lookup may not work properly.
> This JIRA is to address that HMaster uses the hostname passed from RS instead 
> of doing reverse DNS lookup to tells RS which hostname to use during 
> reportForDuty() . This has already been implemented by HBASE-12954 by adding 
> "useThisHostnameInstead" field in RegionServerStatusProtos.
> Currently "useThisHostnameInstead" is optional and RS by default only passes 
> port, server start code and server current time info to HMaster during RS 
> reportForDuty(). In order to use this field, users currently need to specify 
> "hbase.regionserver.hostname" on every regionserver node's hbase-site.xml. 
> This causes some trouble in
> 1. some deployments managed by some management tools like Ambari, which 
> maintains the same copy of hbase-site.xml across all the nodes.
> 2. HBASE-12954 is targeting multihomed hosts, which users want to manually 
> set the hostname value for each node. In the other cases (not multihomed), I 
> just want RS to use the hostname return by the node and set it in 
> useThisHostnameInstead and pass to HMaster during reportForDuty().
> I would like to introduce a setting that if the setting is set to true, 
> "useThisHostnameInstead" will be set to the hostname RS gets from the node. 
> Then HMaster will skip reverse DNS lookup because it sees 
> "useThisHostnameInstead" field is set in the request.
> "hbase.regionserver.hostname.reported.to.master", is it a good name?
> 
> Regarding the hostname returned by the RS node, I read the source code again 
> (including hadoop-common dns.java). By default RS gets hostname by calling 
> InetAddress.getLocalHost().getCanonicalHostName(). If users specify 
> "hbase.regionserver.dns.interface" or "hbase.regionserver.dns.nameserver" or 
> some underlying system configuration changes (eg. modifying 
> /etc/nsswitch.conf), it may first read from DNS or other sources instead of 
> first checking /etc/hosts file.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18226) Disable reverse DNS lookup at HMaster and use default hostname provided by RegionServer

[Duo Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054692#comment-16054692
 ] 

Duo Xu commented on HBASE-18226:


Regarding why we don't add a custom DNS server into our unsecure deployment, 
this is a decision made by the HDInsight deployment team due to some history. I 
can confirm that the current default HDInsight clusters does not support rDNS, 
but in custom virtual network where they deploy the cluster, customers can 
deploy their own custom DNS and enable rDNS for the cluster.

> Disable reverse DNS lookup at HMaster and use default hostname provided by 
> RegionServer
> ---
>
> Key: HBASE-18226
> URL: https://issues.apache.org/jira/browse/HBASE-18226
> Project: HBase
>  Issue Type: Bug
>Reporter: Duo Xu
> Attachments: HBASE-18226.001.patch, HBASE-18226.002.patch, 
> HBASE-18226.003.patch, HBASE-18226.004.patch
>
>
> Description updated:
> In some unusual network environment, forward DNS lookup is supported while 
> reverse DNS lookup may not work properly.
> This JIRA is to address that HMaster uses the hostname passed from RS instead 
> of doing reverse DNS lookup to tells RS which hostname to use during 
> reportForDuty() . This has already been implemented by HBASE-12954 by adding 
> "useThisHostnameInstead" field in RegionServerStatusProtos.
> Currently "useThisHostnameInstead" is optional and RS by default only passes 
> port, server start code and server current time info to HMaster during RS 
> reportForDuty(). In order to use this field, users currently need to specify 
> "hbase.regionserver.hostname" on every regionserver node's hbase-site.xml. 
> This causes some trouble in
> 1. some deployments managed by some management tools like Ambari, which 
> maintains the same copy of hbase-site.xml across all the nodes.
> 2. HBASE-12954 is targeting multihomed hosts, which users want to manually 
> set the hostname value for each node. In the other cases (not multihomed), I 
> just want RS to use the hostname return by the node and set it in 
> useThisHostnameInstead and pass to HMaster during reportForDuty().
> I would like to introduce a setting that if the setting is set to true, 
> "useThisHostnameInstead" will be set to the hostname RS gets from the node. 
> Then HMaster will skip reverse DNS lookup because it sees 
> "useThisHostnameInstead" field is set in the request.
> "hbase.regionserver.hostname.reported.to.master", is it a good name?
> 
> Regarding the hostname returned by the RS node, I read the source code again 
> (including hadoop-common dns.java). By default RS gets hostname by calling 
> InetAddress.getLocalHost().getCanonicalHostName(). If users specify 
> "hbase.regionserver.dns.interface" or "hbase.regionserver.dns.nameserver" or 
> some underlying system configuration changes (eg. modifying 
> /etc/nsswitch.conf), it may first read from DNS or other sources instead of 
> first checking /etc/hosts file.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18214) Replace the folly::AtomicHashMap usage in the RPC layer

[Devaraj Das (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054658#comment-16054658
 ] 

Devaraj Das commented on HBASE-18214:
-

Yeah you are right [~enis]. I need to also change similarly for the {{requests_ 
}}.. Patch coming up in 5-10.

> Replace the folly::AtomicHashMap usage in the RPC layer
> ---
>
> Key: HBASE-18214
> URL: https://issues.apache.org/jira/browse/HBASE-18214
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Devaraj Das
>Assignee: Devaraj Das
> Attachments: 18214-1-1.txt
>
>
> In my tests, I saw that folly::AtomicHashMap usage is not appropriate for 
> one, rather common use case. It'd become sort of unusable (inserts would 
> hang) after a bunch of inserts and erases. This hashmap is used to keep track 
> of call-Id after a connection is set up in the RPC layer (insert a 
> call-id/msg pair when an RPC is sent, and erase the pair when the 
> corresponding response is received). Here is a simple program that will 
> demonstrate the issue:
> {code}
> folly::AtomicHashMap f(100);
> int i = 0;
> while (i < 1) {
> try {
>   f.insert(i,100);
>   LOG(INFO) << "Inserted " << i << "  " << f.size();
>   f.erase(i);
>   LOG(INFO) << "Deleted " << i << "  " << f.size();
>   i++;
> } catch (const std::exception ) {
>   LOG(INFO) << "Exception " << e.what();
>   break;
> }
> }
> {code}
> After poking around a little bit, it is indeed called out as a limitation 
> here 
> https://github.com/facebook/folly/blob/master/folly/docs/AtomicHashMap.md 
> (grep for 'erase'). Proposal is to replace this with something that will fit 
> in in the above usecase (thinking of using std::unordered_map).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18214) Replace the folly::AtomicHashMap usage in the RPC layer

[Enis Soztutar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054652#comment-16054652
 ] 

Enis Soztutar commented on HBASE-18214:
---

Sorry I was looking at the wrong place. My hacked-up patch actually removes 
another AtomicHashMap, but this {{ClientDispatcher::requests_}}. So in this 
patch, I think we should replace both {{ClientDispatcher::requests_}} and 
{{ClientHandler::resp_msgs_}}. 

I was testing with the simple-client, and indeed after about 100K RPCs, the 
client just retries the same RPC because of the exception from these maps. 

> Replace the folly::AtomicHashMap usage in the RPC layer
> ---
>
> Key: HBASE-18214
> URL: https://issues.apache.org/jira/browse/HBASE-18214
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Devaraj Das
>Assignee: Devaraj Das
> Attachments: 18214-1-1.txt
>
>
> In my tests, I saw that folly::AtomicHashMap usage is not appropriate for 
> one, rather common use case. It'd become sort of unusable (inserts would 
> hang) after a bunch of inserts and erases. This hashmap is used to keep track 
> of call-Id after a connection is set up in the RPC layer (insert a 
> call-id/msg pair when an RPC is sent, and erase the pair when the 
> corresponding response is received). Here is a simple program that will 
> demonstrate the issue:
> {code}
> folly::AtomicHashMap f(100);
> int i = 0;
> while (i < 1) {
> try {
>   f.insert(i,100);
>   LOG(INFO) << "Inserted " << i << "  " << f.size();
>   f.erase(i);
>   LOG(INFO) << "Deleted " << i << "  " << f.size();
>   i++;
> } catch (const std::exception ) {
>   LOG(INFO) << "Exception " << e.what();
>   break;
> }
> }
> {code}
> After poking around a little bit, it is indeed called out as a limitation 
> here 
> https://github.com/facebook/folly/blob/master/folly/docs/AtomicHashMap.md 
> (grep for 'erase'). Proposal is to replace this with something that will fit 
> in in the above usecase (thinking of using std::unordered_map).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (HBASE-18226) Disable reverse DNS lookup at HMaster and use default hostname provided by RegionServer

[Duo Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054618#comment-16054618
 ] 

Duo Xu edited comment on HBASE-18226 at 6/19/17 7:46 PM:
-

[~elserj]

Yeah, it is hard to come up with an appropriate name because the setting is in 
RS code but the reverse dns lookup happens on master code :)

Yes, nothing changes actually. All the protocol or major changes have been 
implemented in HBASE-12954. This JIRA is reusing the "useThisHostnameInstead" 
field in RegionServerStatusProtos and adding a new configuration to turn it 
on/off. By default it is off, so it should not break anything.

Regarding how it works in Azure, see the table I replied above.

|| ||non-secure (default Azure HDInsight environment)|| secure||
|reverse DNS|not supported|supported by adding a customized DNS server during 
our deployment|
|forward DNS| supported|supported|

The symptom you see currently in our non-secure clusters, all the nodes are 
using IP as the servername, that is due to rDNS is not supported.


was (Author: onpduo):
[~elserj]

Yeah, it is hard to come up with an appropriate name because the setting is in 
RS code but the reverse dns lookup happens on master code :)

Yes, nothing changes actually. All the protocol or major changes have been 
implemented in HBASE-12954. This JIRA is reusing the "useThisHostnameInstead" 
field in RegionServerStatusProtos and adding a new configuration to turn it 
on/off. By default it is off, so it should not break anything.

Regarding how it works in Azure, see the table I replied above.

|| ||non-secure (default Azure environment)|| secure||
|reverse DNS|not supported|supported by adding a customized DNS server during 
our deployment|
|forward DNS| supported|supported|

The symptom you see currently in our non-secure clusters, all the nodes are 
using IP as the servername, that is due to rDNS is not supported.

> Disable reverse DNS lookup at HMaster and use default hostname provided by 
> RegionServer
> ---
>
> Key: HBASE-18226
> URL: https://issues.apache.org/jira/browse/HBASE-18226
> Project: HBase
>  Issue Type: Bug
>Reporter: Duo Xu
> Attachments: HBASE-18226.001.patch, HBASE-18226.002.patch, 
> HBASE-18226.003.patch, HBASE-18226.004.patch
>
>
> Description updated:
> In some unusual network environment, forward DNS lookup is supported while 
> reverse DNS lookup may not work properly.
> This JIRA is to address that HMaster uses the hostname passed from RS instead 
> of doing reverse DNS lookup to tells RS which hostname to use during 
> reportForDuty() . This has already been implemented by HBASE-12954 by adding 
> "useThisHostnameInstead" field in RegionServerStatusProtos.
> Currently "useThisHostnameInstead" is optional and RS by default only passes 
> port, server start code and server current time info to HMaster during RS 
> reportForDuty(). In order to use this field, users currently need to specify 
> "hbase.regionserver.hostname" on every regionserver node's hbase-site.xml. 
> This causes some trouble in
> 1. some deployments managed by some management tools like Ambari, which 
> maintains the same copy of hbase-site.xml across all the nodes.
> 2. HBASE-12954 is targeting multihomed hosts, which users want to manually 
> set the hostname value for each node. In the other cases (not multihomed), I 
> just want RS to use the hostname return by the node and set it in 
> useThisHostnameInstead and pass to HMaster during reportForDuty().
> I would like to introduce a setting that if the setting is set to true, 
> "useThisHostnameInstead" will be set to the hostname RS gets from the node. 
> Then HMaster will skip reverse DNS lookup because it sees 
> "useThisHostnameInstead" field is set in the request.
> "hbase.regionserver.hostname.reported.to.master", is it a good name?
> 
> Regarding the hostname returned by the RS node, I read the source code again 
> (including hadoop-common dns.java). By default RS gets hostname by calling 
> InetAddress.getLocalHost().getCanonicalHostName(). If users specify 
> "hbase.regionserver.dns.interface" or "hbase.regionserver.dns.nameserver" or 
> some underlying system configuration changes (eg. modifying 
> /etc/nsswitch.conf), it may first read from DNS or other sources instead of 
> first checking /etc/hosts file.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18221) Switch from pread to stream should happen under HStore's reentrant lock

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054639#comment-16054639
 ] 

Ted Yu commented on HBASE-18221:


{code}
+  public final int getCompactedfilesCount() {
{code}
getCompactedfilesCount -> getCompactedFilesCount

The method can be package private.
{code}
+  Collection getCompactedfiles();
{code}
getCompactedfiles -> getCompactedFiles



> Switch from pread to stream should happen under HStore's reentrant lock
> ---
>
> Key: HBASE-18221
> URL: https://issues.apache.org/jira/browse/HBASE-18221
> Project: HBase
>  Issue Type: Sub-task
>  Components: Scanners
>Affects Versions: 2.0.0, 3.0.0, 2.0.0-alpha-1
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-18221_2_fortestcasefailure.patch, 
> HBASE-18221_2.patch
>
>
> Found this while debugging HBASE-18186. When we try to reopen the scanners on 
> the storefiles while trying to switch over from pread to stream, we do not 
> use the HStore's reentrant lock to get the current Storefiles from the 
> StoreFileManager. All the scan APIs are guarded under that and we must do it 
> here also other wise the CompactedHfileDischarger may cause race issues with 
> the HStore's datastructures like here
> {code}
> 2017-06-14 18:16:17,223 WARN  
> [RpcServer.default.FPBQ.Fifo.handler=23,queue=1,port=16020] 
> regionserver.StoreScanner: failed to switch to stream read
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.getScannersForStoreFiles(StoreFileScanner.java:133)
> at 
> org.apache.hadoop.hbase.regionserver.HStore.getScanners(HStore.java:1221)
> at 
> org.apache.hadoop.hbase.regionserver.StoreScanner.trySwitchToStreamRead(StoreScanner.java:997)
> at 
> org.apache.hadoop.hbase.regionserver.StoreScanner.shipped(StoreScanner.java:1134)
> at 
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.shipped(KeyValueHeap.java:445)
> at 
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.shipped(HRegion.java:6459)
> at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices$RegionScannerShippedCallBack.run(RSRpcServices.java:339)
> at 
> org.apache.hadoop.hbase.ipc.ServerCall.setResponse(ServerCall.java:252)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:166)
> at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278)
> at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258)
> {code}
> I have a working patch fixing this problem. Will do some more testing and try 
> to upload the patch after I write a test case for this. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-18164) Much faster locality cost function and candidate generator

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054635#comment-16054635
 ] 

Hadoop QA commented on HBASE-18164:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 44s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
37s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
25m 17s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha3. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 111m 13s 
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
15s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 148m 59s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:757bf37 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12873520/HBASE-18164-05.patch |
| JIRA Issue | HBASE-18164 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 73c32f90f01e 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / ce1ce72 |
| Default Java | 1.8.0_131 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/7235/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/7235/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> Much faster locality cost function and candidate generator
> --
>
> Key: HBASE-18164
> URL: https://issues.apache.org/jira/browse/HBASE-18164
> Project: HBase
>  Issue Type: Improvement
>  Components: Balancer
>Reporter: Kahlil Oppenheimer
>Assignee: Kahlil Oppenheimer
>Priority: Critical
> Attachments: 

[jira] [Commented] (HBASE-18221) Switch from pread to stream should happen under HStore's reentrant lock

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054628#comment-16054628
 ] 

Hadoop QA commented on HBASE-18221:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
53s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 8s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s 
{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 29s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 30s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 30s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
16s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 23s 
{color} | {color:red} The patch causes 14 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m 44s 
{color} | {color:red} The patch causes 14 errors with Hadoop v2.6.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m 7s 
{color} | {color:red} The patch causes 14 errors with Hadoop v2.6.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m 25s 
{color} | {color:red} The patch causes 14 errors with Hadoop v2.6.4. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m 44s 
{color} | {color:red} The patch causes 14 errors with Hadoop v2.6.5. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 8m 3s 
{color} | {color:red} The patch causes 14 errors with Hadoop v2.7.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 9m 27s 
{color} | {color:red} The patch causes 14 errors with Hadoop v2.7.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 10m 48s 
{color} | {color:red} The patch causes 14 errors with Hadoop v2.7.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 12m 7s 
{color} | {color:red} The patch causes 14 errors with Hadoop v3.0.0-alpha3. 
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 19s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 29s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
8s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 42s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:757bf37 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12873540/HBASE-18221_2.patch |
| JIRA Issue | HBASE-18221 |
| Optional Tests |  asflicense  

[jira] [Commented] (HBASE-18226) Disable reverse DNS lookup at HMaster and use default hostname provided by RegionServer

[Duo Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054618#comment-16054618
 ] 

Duo Xu commented on HBASE-18226:


[~elserj]

Yeah, it is hard to come up with an appropriate name because the setting is in 
RS code but the reverse dns lookup happens on master code :)

Yes, nothing changed actually. All the protocol or major changes have been 
implemented in HBASE-12954. This JIRA is reusing the "useThisHostnameInstead" 
field in RegionServerStatusProtos and adding a new configuration to turn it 
on/off. By default it is off, so it should not break anything.

Regarding how it works in Azure, see the table I replied above.

|| ||non-secure (default Azure environment)|| secure||
|reverse DNS|not supported|supported by adding a customized DNS server during 
our deployment|
|forward DNS| supported|supported|

The symptom you see currently in our non-secure clusters, all the nodes are 
using IP as the servername, that is due to rDNS is not supported.

> Disable reverse DNS lookup at HMaster and use default hostname provided by 
> RegionServer
> ---
>
> Key: HBASE-18226
> URL: https://issues.apache.org/jira/browse/HBASE-18226
> Project: HBase
>  Issue Type: Bug
>Reporter: Duo Xu
> Attachments: HBASE-18226.001.patch, HBASE-18226.002.patch, 
> HBASE-18226.003.patch, HBASE-18226.004.patch
>
>
> Description updated:
> In some unusual network environment, forward DNS lookup is supported while 
> reverse DNS lookup may not work properly.
> This JIRA is to address that HMaster uses the hostname passed from RS instead 
> of doing reverse DNS lookup to tells RS which hostname to use during 
> reportForDuty() . This has already been implemented by HBASE-12954 by adding 
> "useThisHostnameInstead" field in RegionServerStatusProtos.
> Currently "useThisHostnameInstead" is optional and RS by default only passes 
> port, server start code and server current time info to HMaster during RS 
> reportForDuty(). In order to use this field, users currently need to specify 
> "hbase.regionserver.hostname" on every regionserver node's hbase-site.xml. 
> This causes some trouble in
> 1. some deployments managed by some management tools like Ambari, which 
> maintains the same copy of hbase-site.xml across all the nodes.
> 2. HBASE-12954 is targeting multihomed hosts, which users want to manually 
> set the hostname value for each node. In the other cases (not multihomed), I 
> just want RS to use the hostname return by the node and set it in 
> useThisHostnameInstead and pass to HMaster during reportForDuty().
> I would like to introduce a setting that if the setting is set to true, 
> "useThisHostnameInstead" will be set to the hostname RS gets from the node. 
> Then HMaster will skip reverse DNS lookup because it sees 
> "useThisHostnameInstead" field is set in the request.
> "hbase.regionserver.hostname.reported.to.master", is it a good name?
> 
> Regarding the hostname returned by the RS node, I read the source code again 
> (including hadoop-common dns.java). By default RS gets hostname by calling 
> InetAddress.getLocalHost().getCanonicalHostName(). If users specify 
> "hbase.regionserver.dns.interface" or "hbase.regionserver.dns.nameserver" or 
> some underlying system configuration changes (eg. modifying 
> /etc/nsswitch.conf), it may first read from DNS or other sources instead of 
> first checking /etc/hosts file.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (HBASE-18178) [C++] Retrying meta location lookup and zookeeper connection

[Enis Soztutar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Enis Soztutar resolved HBASE-18178.
---
   Resolution: Fixed
Fix Version/s: HBASE-14850

Pushed this to the branch. 

> [C++] Retrying meta location lookup and zookeeper connection 
> -
>
> Key: HBASE-18178
> URL: https://issues.apache.org/jira/browse/HBASE-18178
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: HBASE-14850
>
> Attachments: hbase-18178-v1.patch
>
>
> Currently location-cache can only do a single lookup to meta. If meta 
> location changes or we have zookeeper connection problems, we never retry. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (HBASE-18226) Disable reverse DNS lookup at HMaster and use default hostname provided by RegionServer

[Duo Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-18226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054618#comment-16054618
 ] 

Duo Xu edited comment on HBASE-18226 at 6/19/17 7:19 PM:
-

[~elserj]

Yeah, it is hard to come up with an appropriate name because the setting is in 
RS code but the reverse dns lookup happens on master code :)

Yes, nothing changes actually. All the protocol or major changes have been 
implemented in HBASE-12954. This JIRA is reusing the "useThisHostnameInstead" 
field in RegionServerStatusProtos and adding a new configuration to turn it 
on/off. By default it is off, so it should not break anything.

Regarding how it works in Azure, see the table I replied above.

|| ||non-secure (default Azure environment)|| secure||
|reverse DNS|not supported|supported by adding a customized DNS server during 
our deployment|
|forward DNS| supported|supported|

The symptom you see currently in our non-secure clusters, all the nodes are 
using IP as the servername, that is due to rDNS is not supported.


was (Author: onpduo):
[~elserj]

Yeah, it is hard to come up with an appropriate name because the setting is in 
RS code but the reverse dns lookup happens on master code :)

Yes, nothing changed actually. All the protocol or major changes have been 
implemented in HBASE-12954. This JIRA is reusing the "useThisHostnameInstead" 
field in RegionServerStatusProtos and adding a new configuration to turn it 
on/off. By default it is off, so it should not break anything.

Regarding how it works in Azure, see the table I replied above.

|| ||non-secure (default Azure environment)|| secure||
|reverse DNS|not supported|supported by adding a customized DNS server during 
our deployment|
|forward DNS| supported|supported|

The symptom you see currently in our non-secure clusters, all the nodes are 
using IP as the servername, that is due to rDNS is not supported.

> Disable reverse DNS lookup at HMaster and use default hostname provided by 
> RegionServer
> ---
>
> Key: HBASE-18226
> URL: https://issues.apache.org/jira/browse/HBASE-18226
> Project: HBase
>  Issue Type: Bug
>Reporter: Duo Xu
> Attachments: HBASE-18226.001.patch, HBASE-18226.002.patch, 
> HBASE-18226.003.patch, HBASE-18226.004.patch
>
>
> Description updated:
> In some unusual network environment, forward DNS lookup is supported while 
> reverse DNS lookup may not work properly.
> This JIRA is to address that HMaster uses the hostname passed from RS instead 
> of doing reverse DNS lookup to tells RS which hostname to use during 
> reportForDuty() . This has already been implemented by HBASE-12954 by adding 
> "useThisHostnameInstead" field in RegionServerStatusProtos.
> Currently "useThisHostnameInstead" is optional and RS by default only passes 
> port, server start code and server current time info to HMaster during RS 
> reportForDuty(). In order to use this field, users currently need to specify 
> "hbase.regionserver.hostname" on every regionserver node's hbase-site.xml. 
> This causes some trouble in
> 1. some deployments managed by some management tools like Ambari, which 
> maintains the same copy of hbase-site.xml across all the nodes.
> 2. HBASE-12954 is targeting multihomed hosts, which users want to manually 
> set the hostname value for each node. In the other cases (not multihomed), I 
> just want RS to use the hostname return by the node and set it in 
> useThisHostnameInstead and pass to HMaster during reportForDuty().
> I would like to introduce a setting that if the setting is set to true, 
> "useThisHostnameInstead" will be set to the hostname RS gets from the node. 
> Then HMaster will skip reverse DNS lookup because it sees 
> "useThisHostnameInstead" field is set in the request.
> "hbase.regionserver.hostname.reported.to.master", is it a good name?
> 
> Regarding the hostname returned by the RS node, I read the source code again 
> (including hadoop-common dns.java). By default RS gets hostname by calling 
> InetAddress.getLocalHost().getCanonicalHostName(). If users specify 
> "hbase.regionserver.dns.interface" or "hbase.regionserver.dns.nameserver" or 
> some underlying system configuration changes (eg. modifying 
> /etc/nsswitch.conf), it may first read from DNS or other sources instead of 
> first checking /etc/hosts file.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)