[jira] [Created] (HBASE-14676) HBaseTestCase clean out: Purge Incommon Interface and Table and Region implementations

2015-10-22 Thread stack (JIRA) 
stack created HBASE-14676:
-

 Summary: HBaseTestCase clean out: Purge Incommon Interface and 
Table and Region implementations
 Key: HBASE-14676
 URL: https://issues.apache.org/jira/browse/HBASE-14676
 Project: HBase
  Issue Type: Bug
Reporter: stack


As part of the hollowing out of the old HBaseTestCase in preparation for 
removal, this patch purges the old Incommon trick that made it so you could 
pass an Interface to a method to do loading or testing and the implementation 
could be an HTable or a Region.  Instead, replace it with a RegionTable, a 
Regoin that has the Table Interface as a decoration so you can use same loading 
code and same test both on top of the Region or on other side of network via 
HTable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14675) Excise deprecated Put#add(...) and replace with Put#addColumn(...)

2015-10-22 Thread Jonathan Hsieh (JIRA) 
Jonathan Hsieh created HBASE-14675:
--

 Summary: Excise deprecated Put#add(...) and replace with 
Put#addColumn(...)
 Key: HBASE-14675
 URL: https://issues.apache.org/jira/browse/HBASE-14675
 Project: HBase
  Issue Type: Improvement
Affects Versions: 2.0.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Fix For: 2.0.0


The Put API changed from #add(...) to #addColumn(...).  This updates all 
instances of it and removes it from the Put (which was added for hbase 1.0.0)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Could all tests run MR without MiniMapReduceCluster?

2015-10-22 Thread 陈恒 
When i dig in HBASE-14659,  i found in testcase TestHFileOutputFormat, some 
tests run MR job
without MiniMRCluster and some run with it. 


The ones with no MiniMRCluster run much faster!
Do we need to 'test' clustered MR especially in a unit test? 


As stack suggestion, i post this topic here for discussion.

[jira] [Created] (HBASE-14677) BucketAllocator freeBlock is the bottleneck in bulk block cache evictions

2015-10-22 Thread Vladimir Rodionov (JIRA) 
Vladimir Rodionov created HBASE-14677:
-

 Summary: BucketAllocator freeBlock is the bottleneck in bulk block 
cache evictions
 Key: HBASE-14677
 URL: https://issues.apache.org/jira/browse/HBASE-14677
 Project: HBase
  Issue Type: Bug
  Components: BlockCache
Affects Versions: 0.98.15, 1.0.1.1, 1.1.1, 1.0.2
Reporter: Vladimir Rodionov


BucketCache implementation of freeBlock has a bad performance / concurrency and 
affects bulk block evictions from cache on a file close (after split, 
compactions or region moves).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-14677) BucketAllocator freeBlock is the bottleneck in bulk block cache evictions

2015-10-22 Thread Vladimir Rodionov (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-14677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Rodionov resolved HBASE-14677.
---
Resolution: Duplicate

Duplicate of HBASE-14624

> BucketAllocator freeBlock is the bottleneck in bulk block cache evictions
> -
>
> Key: HBASE-14677
> URL: https://issues.apache.org/jira/browse/HBASE-14677
> Project: HBase
>  Issue Type: Bug
>  Components: BlockCache
>Affects Versions: 1.0.2, 1.1.1, 1.0.1.1, 0.98.15
>Reporter: Vladimir Rodionov
>
> BucketCache implementation of freeBlock has a bad performance / concurrency 
> and affects bulk block evictions from cache on a file close (after split, 
> compactions or region moves).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14678) Experiment: Temporarily disable balancer and a few others to see if root of crashed/timedout JVMs

2015-10-22 Thread stack (JIRA) 
stack created HBASE-14678:
-

 Summary: Experiment: Temporarily disable balancer and a few others 
to see if root of crashed/timedout JVMs
 Key: HBASE-14678
 URL: https://issues.apache.org/jira/browse/HBASE-14678
 Project: HBase
  Issue Type: Sub-task
Reporter: stack


Looking at recent builds of 1.2, I see a few of the runs finishing with kills 
and notice that a JVM exited without reporting back state. Running the hanging 
test finder, I can see at least that in one case that the balancer tests seem 
to be outstanding; looking in test output, seems to be still going on A few 
others are reported as hung but they look like they have just started running 
and are just killed by surefire.

This issue is about trying to disable a few of the problematics like balancer 
tests to see if our overall stability improves. If so, I can concentrate on 
stabilizing these few tests. Else will just undo the experiment and put the 
tests back on line.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Could all tests run MR without MiniMapReduceCluster?

2015-10-22 Thread Elliott Clark 
Do we have any tests that test the locality feature for map task -> machine?

That's the only thing I can think of that would 100% require an mr
mincluster.

I'd be +1 on removing most of the uses of mr minicluster.

On Wed, Oct 21, 2015 at 10:01 PM, 陈恒  wrote:

> When i dig in HBASE-14659,  i found in testcase TestHFileOutputFormat,
> some tests run MR job
> without MiniMRCluster and some run with it.
>
>
> The ones with no MiniMRCluster run much faster!
> Do we need to 'test' clustered MR especially in a unit test?
>
>
> As stack suggestion, i post this topic here for discussion.

Re: Could all tests run MR without MiniMapReduceCluster?

2015-10-22 Thread Stack 
On Wed, Oct 21, 2015 at 10:01 PM, 陈恒  wrote:

> When i dig in HBASE-14659,  i found in testcase TestHFileOutputFormat,
> some tests run MR job
> without MiniMRCluster and some run with it.
>
>
> The ones with no MiniMRCluster run much faster!
> Do we need to 'test' clustered MR especially in a unit test?
>
>
>
We have some fat unit tests that spin up loads of mini clusters. For MR,
the same code base is exercised whether cluster or not so if no MR cluster
means less resources used by test and all runs faster, I'd be +1 in favor
of undoing cluster (for all but the cases where cluster needed to prove
some code as per example Elliott raises above).

Thanks  陈恒
St.Ack

Re: add msg in disable table interface

2015-10-22 Thread Ted Yu 
Isn't such information already available in master log (though not on one
line) ?

bq. a process need to add cp so disable

Using online table alter feature, there is no need to disable the whole
table.

Cheers

On Thu, Oct 22, 2015 at 3:07 AM, wangyongqiang0...@163.com <
wangyongqiang0...@163.com> wrote:

> we can find some tables are disabled sometimes, and we want to chcek out
> why.
> when we check master log, we can see logs are as follows:
> Client=root//"ip_address disable" "table_name"
>
> this log is from hbase0.98.10, line 2213
>
> can we add a parameter named message to the interface disableTable,  we
> can add some explain-msg, such as
> hbase-shell disable table(we can know this table is disabled by hand)
> a process need to add cp so disable
> a process need to drop table so disable
>
> we can more easily know why this table is disabled
>
>
>
> wangyongqiang0...@163.com
>

[jira] [Created] (HBASE-14679) Separate security shell tests from other shell tests

2015-10-22 Thread Ted Yu (JIRA) 
Ted Yu created HBASE-14679:
--

 Summary: Separate security shell tests from other shell tests
 Key: HBASE-14679
 URL: https://issues.apache.org/jira/browse/HBASE-14679
 Project: HBase
  Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Minor


TestShell takes a long time to run, sometimes timing out (especially on 
branch-1).

This issue proposes the following changes:

separate security shell tests into their own test class
use unique table names for different tests



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14681) Upgrade Checkstyle plugin to 2.15 to avoid MCHECKSTYLE-288

2015-10-22 Thread Misty Stanley-Jones (JIRA) 
Misty Stanley-Jones created HBASE-14681:
---

 Summary: Upgrade Checkstyle plugin to 2.15 to avoid MCHECKSTYLE-288
 Key: HBASE-14681
 URL: https://issues.apache.org/jira/browse/HBASE-14681
 Project: HBase
  Issue Type: Bug
  Components: build
Affects Versions: 2.0.0
Reporter: Misty Stanley-Jones
Assignee: Misty Stanley-Jones
 Fix For: 2.0.0


We are getting a NPE in checkstyle when running mvn:site. It seems to be 
MCHECKSTYLE-288. Updating maven-checkstyle-plugin from 2.13 to 2.15 seems to 
fix it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14682) CM restore functionality for regionservers is broken

2015-10-22 Thread Enis Soztutar (JIRA) 
Enis Soztutar created HBASE-14682:
-

 Summary: CM restore functionality for regionservers is broken
 Key: HBASE-14682
 URL: https://issues.apache.org/jira/browse/HBASE-14682
 Project: HBase
  Issue Type: Bug
Reporter: Enis Soztutar
Assignee: Enis Soztutar


In the {{DistributedHBaseCluster.restoreClusterStatus()}}, we are starting a 
regionserver for backup masters. Seems to be a copy-paste error from 
HBASE-12429. 

This is causing further issues on our test rig since we are starting a 
regionserver where only a backup master were before:
{code}
INFO  [main] hbase.HBaseClusterManager: Executing remote command: ps aux | grep 
proc_master | grep -v grep | tr -s ' ' | cut -d ' ' -f2 , hostname:10.0.0.99
INFO  [main] util.Shell: Executing full command [/usr/bin/ssh  -o 
StrictHostKeyChecking=no 10.0.0.99 "sudo su - hbase -c \"ps aux | grep 
proc_master | grep -v grep | tr -s ' ' | cut -d ' ' -f2\""]
INFO  [main] hbase.HBaseClusterManager: Executed remote command, exit code:0 , 
output:13244

INFO  [main] hbase.HBaseClusterManager: Executing remote command: ps aux | grep 
proc_regionserver | grep -v grep | tr -s ' ' | cut -d ' ' -f2 , 
hostname:10.0.0.99
INFO  [main] util.Shell: Executing full command [/usr/bin/ssh  -o 
StrictHostKeyChecking=no 10.0.0.99 "sudo su - hbase -c \"ps aux | grep 
proc_regionserver | grep -v grep | tr -s ' ' | cut -d ' ' -f2\""]
INFO  [main] hbase.HBaseClusterManager: Executed remote command, exit code:0 , 
output:
INFO  [main] hbase.HBaseCluster: Restoring cluster - starting initial region 
server: 10.0.0.99:16000
INFO  [main] hbase.HBaseCluster: Starting RS on: 10.0.0.99
INFO  [main] hbase.HBaseClusterManager: Executing remote command: 
/usr/hdp/2.3.3.0-2971/hbase/bin/hbase-daemon.sh --config /tmp/hbaseConf start 
regionserver , hostname:10.0.0.99
INFO  [main] util.Shell: Executing full command [/usr/bin/ssh  -o 
StrictHostKeyChecking=no 10.0.0.99 "sudo su - hbase -c 
\"/usr/hdp/2.3.3.0-2971/hbase/bin/hbase-daemon.sh --config /tmp/hbaseConf start 
reg
INFO  [main] hbase.HBaseClusterManager: Executed remote command, exit code:0 , 
output:starting regionserver, logging to 
/var/log/hbase/hbase-hbase-regionserver-zk0-hb23-h.out

INFO  [main] hbase.HBaseClusterManager: Executing remote command: ps aux | grep 
proc_regionserver | grep -v grep | tr -s ' ' | cut -d ' ' -f2 , 
hostname:10.0.0.126
INFO  [main] util.Shell: Executing full command [/usr/bin/ssh  -o 
StrictHostKeyChecking=no 10.0.0.126 "sudo su - hbase -c \"ps aux | grep 
proc_regionserver | grep -v grep | tr -s ' ' | cut -d ' ' -f2\""]
INFO  [main] hbase.HBaseClusterManager: Executed remote command, exit code:0 , 
output:
INFO  [main] hbase.HBaseCluster: Added new HBaseAdmin
INFO  [main] hbase.HBaseCluster: Restoring cluster - done
DEBUG [main] hbase.IntegrationTestBase: Done restoring the cluster
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-14681) Upgrade Checkstyle plugin to 2.16

2015-10-22 Thread Misty Stanley-Jones (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HBASE-14681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Misty Stanley-Jones resolved HBASE-14681.
-
  Resolution: Fixed
Hadoop Flags: Reviewed

> Upgrade Checkstyle plugin to 2.16
> -
>
> Key: HBASE-14681
> URL: https://issues.apache.org/jira/browse/HBASE-14681
> Project: HBase
>  Issue Type: Bug
>  Components: build
>Affects Versions: 2.0.0
>Reporter: Misty Stanley-Jones
>Assignee: Misty Stanley-Jones
> Fix For: 2.0.0
>
> Attachments: 2.13_fail.log, 2.15_success.log.zip, HBASE-14681.patch
>
>
> We are getting a NPE in checkstyle when running mvn:site. It seems to be 
> MCHECKSTYLE-288 or MCHECKSTYLE-250. Updating maven-checkstyle-plugin from 
> 2.13 to 2.16 seems to fix it.
> Or maybe not.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

add msg in disable table interface

2015-10-22 Thread wangyongqiang0...@163.com 
we can find some tables are disabled sometimes, and we want to chcek out why.
when we check master log, we can see logs are as follows:
Client=root//"ip_address disable" "table_name"

this log is from hbase0.98.10, line 2213

can we add a parameter named message to the interface disableTable,  we can add 
some explain-msg, such as 
hbase-shell disable table(we can know this table is disabled by hand)
a process need to add cp so disable
a process need to drop table so disable

we can more easily know why this table is disabled



wangyongqiang0...@163.com

Upgrading Apache HTrace integration to 4.x branch

2015-10-22 Thread Andrey Redko 
Hey guys,

I am wondering if there are there any plans on upgrading Apache HTrace
version used in HBase client / server from 3.x to 4.x release branch?
Unfortunately, HTrace  3.x and 4.x versions are not compatible with respect
to integration modules (zipkin, htraced) and cause nasty classpath
conflicts.

Thanks!

Best Regards,
Andriy Redko

[jira] [Created] (HBASE-14683) Batching in buffered mutator is awful when adding lists of mutations.

2015-10-22 Thread Elliott Clark (JIRA) 
Elliott Clark created HBASE-14683:
-

 Summary: Batching in buffered mutator is awful when adding lists 
of mutations.
 Key: HBASE-14683
 URL: https://issues.apache.org/jira/browse/HBASE-14683
 Project: HBase
  Issue Type: Bug
Reporter: Elliott Clark
Assignee: Elliott Clark






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Could all tests run MR without MiniMapReduceCluster?

2015-10-22 Thread Elliott Clark 
On Thu, Oct 22, 2015 at 7:41 PM, Heng Chen  wrote:

> IMO we can remove them firstly,  and see what QA says.
>

+1

Opinions wanted: new site skin

2015-10-22 Thread Misty Stanley-Jones 
Hi all,

We are currently using the reFlow Maven site skin. I went looking around
and found Fluido, which seems to be a bit more extensible. I built and
staged a version of the site at
http://mstanleyjones.github.io/hbase/index.html. Note the Github ribbon and
the Google site search. I'm curious to know what you think.

I also put the 0.94 docs menu as a submenu of the Documentation menu, to
see how it looked.

Thanks,
Misty

Re: Could all tests run MR without MiniMapReduceCluster?

2015-10-22 Thread Stack 
What Elliott said.
St.Ack

On Thu, Oct 22, 2015 at 10:09 PM, Elliott Clark  wrote:

> On Thu, Oct 22, 2015 at 7:41 PM, Heng Chen 
> wrote:
>
> > IMO we can remove them firstly,  and see what QA says.
> >
>
> +1
>

Re: Opinions wanted: new site skin

2015-10-22 Thread Stack 
I think it looks pretty amazing. Fresh!  I like the menu of menu reports.
It shines light on mess we've shoved under the carpet for ages now
(dependencies, being public w/ what the licensing of our depenencies
are...).  Can we change the google search to be search-hadoop.com setup for
hbase searches?

+1

St.Ack

On Thu, Oct 22, 2015 at 10:17 PM, Misty Stanley-Jones <
mstanleyjo...@cloudera.com> wrote:

> Hi all,
>
> We are currently using the reFlow Maven site skin. I went looking around
> and found Fluido, which seems to be a bit more extensible. I built and
> staged a version of the site at
> http://mstanleyjones.github.io/hbase/index.html. Note the Github ribbon
> and
> the Google site search. I'm curious to know what you think.
>
> I also put the 0.94 docs menu as a submenu of the Documentation menu, to
> see how it looked.
>
> Thanks,
> Misty
>

Re: Upgrading Apache HTrace integration to 4.x branch

2015-10-22 Thread Dave Latham 
Looks like there's work to do that happening at
https://issues.apache.org/jira/browse/HBASE-14451

On Thu, Oct 22, 2015 at 7:47 AM, Andrey Redko  wrote:

> Hey guys,
>
> I am wondering if there are there any plans on upgrading Apache HTrace
> version used in HBase client / server from 3.x to 4.x release branch?
> Unfortunately, HTrace  3.x and 4.x versions are not compatible with respect
> to integration modules (zipkin, htraced) and cause nasty classpath
> conflicts.
>
> Thanks!
>
> Best Regards,
> Andriy Redko
>

Re: Upgrading Apache HTrace integration to 4.x branch

2015-10-22 Thread Stack 
What Dave said. The posted patch will put us up on htrace 4. Lacking is the
work to make it so traces tell a good story, that all the previous trace
points are properly connected and perhaps adding more. Should get to it
soon.

Thanks for asking,
St.Ack

On Thu, Oct 22, 2015 at 8:49 AM, Dave Latham  wrote:

> Looks like there's work to do that happening at
> https://issues.apache.org/jira/browse/HBASE-14451
>
> On Thu, Oct 22, 2015 at 7:47 AM, Andrey Redko  wrote:
>
> > Hey guys,
> >
> > I am wondering if there are there any plans on upgrading Apache HTrace
> > version used in HBase client / server from 3.x to 4.x release branch?
> > Unfortunately, HTrace  3.x and 4.x versions are not compatible with
> respect
> > to integration modules (zipkin, htraced) and cause nasty classpath
> > conflicts.
> >
> > Thanks!
> >
> > Best Regards,
> > Andriy Redko
> >
>