date:20141015

[jira] [Updated] (HBASE-12259) Bring quorum based write ahead log into HBase

2014-10-15 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-12259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-12259:

Issue Type: Improvement  (was: Bug)

 Bring quorum based write ahead log into HBase
 -

 Key: HBASE-12259
 URL: https://issues.apache.org/jira/browse/HBASE-12259
 Project: HBase
  Issue Type: Improvement
Affects Versions: 2.0.0
Reporter: Elliott Clark

 HydraBase ( 
 https://code.facebook.com/posts/32638043166/hydrabase-the-evolution-of-hbase-facebook/
  ) Facebook's implementation of HBase with Raft for consensus will be going 
 open source shortly. We should pull in the parts of that fb-0.89 based 
 implementation, and offer it as a feature in whatever next major release is 
 next up. Right now the Hydrabase code base isn't ready to be released into 
 the wild; it should be ready soon ( for some definition of soon).
 Since Hydrabase is based upon 0.89 most of the code is not directly 
 applicable. So lots of work will probably need to be done in a feature branch 
 before a merge vote.
 Is this something that's wanted?
 Is there anything clean up that needs to be done before the log 
 implementation is able to be replaced like this?
 What's our story with upgrading to this? Are we ok with requiring down time ?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12259) Bring quorum based write ahead log into HBase

2014-10-15 Thread Sean Busbey (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-12259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172097#comment-14172097
]

Sean Busbey commented on HBASE-12259:
-

The refactoring working in HBASE-10378 should get finished up first. It
obviates some other WAL clean up tickets and generally provides us with a
cleaner separation of concerns than the current HLog.

There's a simplified roadmap around WAL improvements on that ticket and a patch
from a bit ago. Both should get updated in the next day or so with a version
that I think is ready as a first pass implementation. One of the follow-ons is
getting the WAL related code all into its own module, which I think will help a
lot in getting the recovery side of things better isolated.

Bring quorum based write ahead log into HBase
-

Key: HBASE-12259
URL: https://issues.apache.org/jira/browse/HBASE-12259
Project: HBase
Issue Type: Bug
Affects Versions: 2.0.0
Reporter: Elliott Clark

HydraBase (
https://code.facebook.com/posts/32638043166/hydrabase-the-evolution-of-hbase-facebook/
) Facebook's implementation of HBase with Raft for consensus will be going
open source shortly. We should pull in the parts of that fb-0.89 based
implementation, and offer it as a feature in whatever next major release is
next up. Right now the Hydrabase code base isn't ready to be released into
the wild; it should be ready soon ( for some definition of soon).
Since Hydrabase is based upon 0.89 most of the code is not directly
applicable. So lots of work will probably need to be done in a feature branch
before a merge vote.
Is this something that's wanted?
Is there anything clean up that needs to be done before the log
implementation is able to be replaced like this?
What's our story with upgrading to this? Are we ok with requiring down time ?

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12238) A few ugly exceptions on startup

2014-10-15 Thread Hadoop QA (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-12238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172099#comment-14172099
]

Hadoop QA commented on HBASE-12238:
---

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12674928/12238.txt
against trunk revision .
ATTACHMENT ID: 12674928

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.

{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.

{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.

{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.

{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100

{color:green}+1 site{color}. The mvn site goal succeeds with this patch.

{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.procedure.TestZKProcedure

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/11350//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11350//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11350//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11350//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11350//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11350//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11350//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11350//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11350//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11350//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11350//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11350//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11350//console

This message is automatically generated.

A few ugly exceptions on startup

Key: HBASE-12238
URL: https://issues.apache.org/jira/browse/HBASE-12238
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.1
Reporter: stack
Assignee: stack
Priority: Minor
Fix For: 2.0.0, 0.99.2

Attachments: 12238.txt

Let me fix a few innocuous exceptions that show on startup (saw testing
0.99.1), even when regular -- will throw people off.
Here is one:
{code}
2014-10-12 19:07:15,251 INFO [c2020:16020.activeMasterManager]
zookeeper.MetaTableLocator: Failed verification of hbase:meta,,1 at
address=c2021.halxg.cloudera.com,16020,1413165899611,
exception=org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: Region hbase:meta,,1 is
not online on c2021.halxg.cloudera.com,16020,1413166029547
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2677)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:838)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionInfo(RSRpcServices.java:1110)
at
org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20158)
at

[jira] [Updated] (HBASE-12259) Bring quorum based write ahead log into HBase

2014-10-15 Thread Sean Busbey (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-12259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-12259:

Component/s: wal

 Bring quorum based write ahead log into HBase
 -

 Key: HBASE-12259
 URL: https://issues.apache.org/jira/browse/HBASE-12259
 Project: HBase
  Issue Type: Improvement
  Components: wal
Affects Versions: 2.0.0
Reporter: Elliott Clark

 HydraBase ( 
 https://code.facebook.com/posts/32638043166/hydrabase-the-evolution-of-hbase-facebook/
  ) Facebook's implementation of HBase with Raft for consensus will be going 
 open source shortly. We should pull in the parts of that fb-0.89 based 
 implementation, and offer it as a feature in whatever next major release is 
 next up. Right now the Hydrabase code base isn't ready to be released into 
 the wild; it should be ready soon ( for some definition of soon).
 Since Hydrabase is based upon 0.89 most of the code is not directly 
 applicable. So lots of work will probably need to be done in a feature branch 
 before a merge vote.
 Is this something that's wanted?
 Is there anything clean up that needs to be done before the log 
 implementation is able to be replaced like this?
 What's our story with upgrading to this? Are we ok with requiring down time ?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12233) Bring back root table

2014-10-15 Thread Francis Liu (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-12233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172123#comment-14172123
 ] 

Francis Liu commented on HBASE-12233:
-

Thanks for the feedback guys. Just to reiterate, we do have a real need to 
scale and we will improve/fix what's necessary be it on hbase, hdfs, or infra 
to meet that need. So far based on our experimentation and experience from 
actual large cluster deployments, it is hbase that is preventing us from 
scaling to our expected needs and splitting meta is the clear solution. 

{quote}
We could try this just with a 0.98 based feature branch and determine if it's 
worth pursuing further and elsewhere.
{quote}
[~apurtell] Thanks for volunteering. Is this a feature branch used to determine 
whether it is worthwhile to bring back root (and split meta) or a feature 
branch to determine whether it is feasible to backport it to 0.98? What would 
be the measure of success? 

IMHO a 0.98 feature branch should be enough as a proving ground for either 
scenario.




 Bring back root table
 -

 Key: HBASE-12233
 URL: https://issues.apache.org/jira/browse/HBASE-12233
 Project: HBase
  Issue Type: Sub-task
Reporter: Virag Kothari
Assignee: Virag Kothari
 Attachments: HBASE-12233.patch


 First step towards splitting meta.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-12264) ImportTsv should fail fast if output is not specified and table does not exist

2014-10-15 Thread Ashish Singhi (JIRA)

Ashish Singhi created HBASE-12264:
-

 Summary: ImportTsv should fail fast if output is not specified and 
table does not exist
 Key: HBASE-12264
 URL: https://issues.apache.org/jira/browse/HBASE-12264
 Project: HBase
  Issue Type: Improvement
  Components: mapreduce
Affects Versions: 0.98.5
Reporter: Ashish Singhi
Assignee: Ashish Singhi
Priority: Minor


ImportTsv should fail fast if {{importtsv.bulk.output}} is not specified and 
the specified table also does not exist



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11992) Backport HBASE-11367 (Pluggable replication endpoint) to 0.98

2014-10-15 Thread ramkrishna.s.vasudevan (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172145#comment-14172145
]

ramkrishna.s.vasudevan commented on HBASE-11992:

bq.Would it be possible for you to test again with two clusters where at least
two RSes are replicating in each? Only upgrade the RSes, one at a time.
I did this test as suggested.
1) Started two clusters with one master and 2 RSes each.
2) Created 3 tables in each cluster with REPLICATION_SCOPE and ensured that
atleast one table is present in each RS.
3) Inserted data into all tables
4) Scanned the peer cluster and read back all the data from all the tables.
5) Stopped one RS in the main cluster.
6) Applied the patch and issued balance such that the newly started RS also has
some table.
7) Inserted data into that table
8) Scan the peer cluster to read back all the data
9) Stop the RS in the peer cluster.
10) Apply the patch in the peer cluster for one RS. Read back all the data and
do balancing such that we have tables in this RS also
11) Insert new data from the main cluster.
12) Scan the data in the peer cluster and ensure all the new data is also
available
13) Apply patch on the other RS in the main cluster.
14) Balance the cluster and add new data to all the tables.
15) Scan the peer clusters and ensure all the data is available.
I think the basic scenarios are working fine. What do you think
[~apurtell],[~enis]?

Backport HBASE-11367 (Pluggable replication endpoint) to 0.98
-

Key: HBASE-11992
URL: https://issues.apache.org/jira/browse/HBASE-11992
Project: HBase
Issue Type: Task
Reporter: Andrew Purtell
Assignee: ramkrishna.s.vasudevan
Attachments: HBASE-11992_0.98_1.patch, hbase-11367_0.98.patch

ReplicationSource tails the logs for each peer. HBASE-11367 introduces
ReplicationEndpoint which is customizable per peer. ReplicationEndpoint is
run in the same RS process and instantiated per replication peer per region
server. Implementations of this interface handle the actual shipping of WAL
edits to the remote cluster.
This issue is for backporting HBASE-11367 to 0.98.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-12264) ImportTsv should fail fast if output is not specified and table does not exist

2014-10-15 Thread Ashish Singhi (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-12264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-12264:
--
Attachment: HBase-12264.patch

 ImportTsv should fail fast if output is not specified and table does not exist
 --

 Key: HBASE-12264
 URL: https://issues.apache.org/jira/browse/HBASE-12264
 Project: HBase
  Issue Type: Improvement
  Components: mapreduce
Affects Versions: 0.98.5
Reporter: Ashish Singhi
Assignee: Ashish Singhi
Priority: Minor
 Attachments: HBase-12264.patch


 ImportTsv should fail fast if {{importtsv.bulk.output}} is not specified and 
 the specified table also does not exist



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-12264) ImportTsv should fail fast if output is not specified and table does not exist

2014-10-15 Thread Ashish Singhi (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-12264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-12264:
--
Status: Patch Available  (was: Open)

Patch for master branch.
Please some one review.

 ImportTsv should fail fast if output is not specified and table does not exist
 --

 Key: HBASE-12264
 URL: https://issues.apache.org/jira/browse/HBASE-12264
 Project: HBase
  Issue Type: Improvement
  Components: mapreduce
Affects Versions: 0.98.5
Reporter: Ashish Singhi
Assignee: Ashish Singhi
Priority: Minor
 Attachments: HBase-12264.patch


 ImportTsv should fail fast if {{importtsv.bulk.output}} is not specified and 
 the specified table also does not exist



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12255) NPE in OpenRegionHandler after restart hdfs without stop hbase

2014-10-15 Thread Junyong Li (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-12255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172161#comment-14172161
 ] 

Junyong Li commented on HBASE-12255:


The log only has a single line java.lang.NullPointerException, no more than 
stacktrace

 NPE in OpenRegionHandler after restart hdfs without stop hbase
 --

 Key: HBASE-12255
 URL: https://issues.apache.org/jira/browse/HBASE-12255
 Project: HBase
  Issue Type: Bug
 Environment: hadoop-2.5.1
 hbase-0.98
 phonenix-4.1.0
Reporter: Junyong Li

 I have a phoenix table 'EVENT', and the table have a index 'IDX_DATE_HOUR_X'.
 I restarted hdfs without stop hbase, after then, the hbase table cann't be 
 scaned.
 I try to restart hbase, the all hbase table still cann't be scaned.
 the regionserver log have many exception like this:
 2014-10-13 19:33:05,287 INFO  
 [A01101303447-V1,60020,1413199890407-recovery-writer--pool4-t3] 
 client.AsyncProcess: #4, waiting for some tasks to finish. Expected max=0, 
 tasksSent=9, tasksDone=8, currentTasksDone=8, retries=8 hasError=fal
 se, tableName=IDX_DATE_HOUR_X
 2014-10-13 19:33:05,298 INFO  
 [A01101303447-V1,60020,1413199890407-recovery-writer--pool4-t2] 
 client.AsyncProcess: #5, waiting for some tasks to finish. Expected max=0, 
 tasksSent=9, tasksDone=8, currentTasksDone=8, retries=8 hasError=fal
 se, tableName=IDX_DATE_HOUR_X
 2014-10-13 19:33:05,311 INFO  
 [A01101303447-V1,60020,1413199890407-recovery-writer--pool4-t1] 
 client.AsyncProcess: #6, waiting for some tasks to finish. Expected max=0, 
 tasksSent=9, tasksDone=8, currentTasksDone=8, retries=8 hasError=fal
 se, tableName=IDX_DATE_HOUR_X
 2014-10-13 19:33:06,452 INFO  [ReplicationExecutor-0] 
 replication.ReplicationQueuesZKImpl: Moving 
 A01101303447-V1,60020,1413199414409's hlogs to my queue
 2014-10-13 19:33:15,325 INFO  
 [A01101303447-V1,60020,1413199890407-recovery-writer--pool4-t1] 
 client.AsyncProcess: #6, waiting for some tasks to finish. Expected max=0, 
 tasksSent=10, tasksDone=9, currentTasksDone=9, retries=9 hasError=fa
 lse, tableName=IDX_DATE_HOUR_X
 2014-10-13 19:33:15,333 INFO  [htable-pool6-t2] client.AsyncProcess: #6, 
 table=IDX_DATE_HOUR_X, attempt=10/350 failed 12 ops, last exception: 
 org.apache.hadoop.hbase.exceptions.RegionOpeningException: 
 org.apache.hadoop.hbase.exceptions.R
 egionOpeningException: Region 
 IDX_DATE_HOUR_X,t\x00\x00\x00\x00\x00,1413186874829.9a92abb84768b129df3faedb877f7bea.
  is opening on A01101303447-V1,60020,1413199890407
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2759)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:4213)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3437)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29593)
 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027)
 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
 at 
 org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
 at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
 at java.lang.Thread.run(Thread.java:744)
 ---
 After two days's try, i found that:
 If idisable 'EVENT', other tables can be scanned, then i enable 'EVENT' 
 manually, the region log show that NullPointExceptin has occur then replaying 
 WAL, the following is log:
 2014-10-13 19:25:21,043 INFO  [RS_OPEN_REGION-A01101303447-V1:60020-1] 
 regionserver.HRegion: Replaying edits from 
 hdfs://localhost/hbase-0.98/data/default/EVENT/def4a581d4ad963cbb8cad32cbfbab2e/recovered.edits/002
 2014-10-13 19:25:21,048 INFO  
 [A01101303447-V1,60020,1413199414409-recovery-writer--pool17-t2-SendThread(localhost:2182)]
  zookeeper.ClientCnxn: Opening socket connection to server 
 localhost/0:0:0:0:0:0:0:1:2182. Will not attempt to authe
 nticate using SASL (unknown error)
 2014-10-13 19:25:21,049 INFO  
 [A01101303447-V1,60020,1413199414409-recovery-writer--pool17-t2-SendThread(localhost:2182)]
  zookeeper.ClientCnxn: Socket connection established to 
 localhost/0:0:0:0:0:0:0:1:2182, initiating session
 2014-10-13 19:25:21,051 INFO  
 [A01101303447-V1,60020,1413199414409-recovery-writer--pool17-t3] 
 zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x199c8484 
 connecting to ZooKeeper ensemble=localhost:2182
 2014-10-13 19:25:21,051 INFO  
 [A01101303447-V1,60020,1413199414409-recovery-writer--pool17-t3] 
 zookeeper.ZooKeeper: Initiating client connection, 
 connectString=localhost:2182 sessionTimeout=9 
 watcher=hconnection-0x199c8484, quorum=lo
 calhost:2182,

[jira] [Created] (HBASE-12265) HBase shell 'show_filters' points to internal Facebook URL

2014-10-15 Thread Niels Basjes (JIRA)

Niels Basjes created HBASE-12265:


 Summary: HBase shell 'show_filters' points to internal Facebook URL
 Key: HBASE-12265
 URL: https://issues.apache.org/jira/browse/HBASE-12265
 Project: HBase
  Issue Type: Bug
  Components: shell
Reporter: Niels Basjes
Priority: Minor


In the HBase shell the output of the show_filters command starts with this:
{code}
hbase(main):001:0 show_filters
Documentation on filters mentioned below can be found at: 
https://our.intern.facebook.com/intern/wiki/index.php/HBase/Filter_Language
ColumnPrefixFilter
TimestampsFilter
...
{code}

This documentation link cannot be reached by me. It seems to be an internal 
link to a wiki that can only be reached by Facebook employees.

This link is in this file in two places 
hbase-shell/src/main/ruby/shell/commands/show_filters.rb

So far I have not been able to find the 'right' page to point to (I did a quick 
check of the apache wiki and the hbase book).

So either remove the link or add a section to the hbase book and point there. I 
think the latter (creating documentation) is the best solution.

Perhaps Facebook is willing to donate the apparently existing pages?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12202) Support DirectByteBuffer usage in HFileBlock

2014-10-15 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-12202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172180#comment-14172180
 ] 

ramkrishna.s.vasudevan commented on HBASE-12202:


Pls give me a day's time. I will check this and get back on it.

 Support DirectByteBuffer usage in HFileBlock
 

 Key: HBASE-12202
 URL: https://issues.apache.org/jira/browse/HBASE-12202
 Project: HBase
  Issue Type: Sub-task
  Components: regionserver, Scanners
Reporter: Anoop Sam John
Assignee: Anoop Sam John
 Fix For: 2.0.0, 0.99.2

 Attachments: HBASE-12202.patch






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-12266) Slow Scan can cause dead loop in ClientScanner

2014-10-15 Thread Qiang Tian (JIRA)

Qiang Tian created HBASE-12266:
--

 Summary: Slow Scan can cause dead loop in ClientScanner 
 Key: HBASE-12266
 URL: https://issues.apache.org/jira/browse/HBASE-12266
 Project: HBase
  Issue Type: Bug
  Components: Scanners
Affects Versions: 0.96.0
Reporter: Qiang Tian
Priority: Minor


see http://search-hadoop.com/m/DHED45SVsC1.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-12266) Slow Scan can cause dead loop in ClientScanner

2014-10-15 Thread Qiang Tian (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-12266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Qiang Tian updated HBASE-12266:
---
Attachment: HBASE-12266-master.patch

any particular purpose to set it to true there?
thanks.

 Slow Scan can cause dead loop in ClientScanner 
 ---

 Key: HBASE-12266
 URL: https://issues.apache.org/jira/browse/HBASE-12266
 Project: HBase
  Issue Type: Bug
  Components: Scanners
Affects Versions: 0.96.0
Reporter: Qiang Tian
Priority: Minor
 Attachments: HBASE-12266-master.patch


 see http://search-hadoop.com/m/DHED45SVsC1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-12266) Slow Scan can cause dead loop in ClientScanner

2014-10-15 Thread Qiang Tian (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-12266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Qiang Tian updated HBASE-12266:
---
Status: Patch Available  (was: Open)

 Slow Scan can cause dead loop in ClientScanner 
 ---

 Key: HBASE-12266
 URL: https://issues.apache.org/jira/browse/HBASE-12266
 Project: HBase
  Issue Type: Bug
  Components: Scanners
Affects Versions: 0.96.0
Reporter: Qiang Tian
Priority: Minor
 Attachments: HBASE-12266-master.patch


 see http://search-hadoop.com/m/DHED45SVsC1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12264) ImportTsv should fail fast if output is not specified and table does not exist

2014-10-15 Thread Hadoop QA (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-12264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172240#comment-14172240
]

Hadoop QA commented on HBASE-12264:
---

{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12674957/HBase-12264.patch
against trunk revision .
ATTACHMENT ID: 12674957

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.

{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.

{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.

{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.

{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100

{color:green}+1 site{color}. The mvn site goal succeeds with this patch.

{color:green}+1 core tests{color}. The patch passed unit tests in .

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/11351//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11351//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11351//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11351//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11351//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11351//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11351//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11351//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11351//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11351//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11351//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11351//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11351//console

This message is automatically generated.

ImportTsv should fail fast if output is not specified and table does not exist
--

Key: HBASE-12264
URL: https://issues.apache.org/jira/browse/HBASE-12264
Project: HBase
Issue Type: Improvement
Components: mapreduce
Affects Versions: 0.98.5
Reporter: Ashish Singhi
Assignee: Ashish Singhi
Priority: Minor
Attachments: HBase-12264.patch

ImportTsv should fail fast if {{importtsv.bulk.output}} is not specified and
the specified table also does not exist

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-10201) Port 'Make flush decisions per column family' to trunk

2014-10-15 Thread zhangduo (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-10201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172250#comment-14172250
 ] 

zhangduo commented on HBASE-10201:
--

Run the same benchmark on a 3 regionservers cluster(2 * Xeon E5-2650 2.6G, 3T * 
11 sata), the result is smililar.

Without per CF flush:
metric_storeCount: 3,
metric_storeFileCount: 9,
metric_memStoreSize: 39965016,
metric_storeFileSize: 4460709275,
metric_compactionsCompletedCount: 46,
metric_numBytesCompactedCount: 11030906070,
metric_numFilesCompactedCount: 145,
Write amplification: 2.47

With per CF flush:
metric_storeCount: 3,
metric_storeFileCount: 7,
metric_memStoreSize: 110195648,
metric_storeFileSize: 4369570622,
metric_compactionsCompletedCount: 27,
metric_numBytesCompactedCount: 10353718691,
metric_numFilesCompactedCount: 89,
Write amplification: 2.37

The patch has a big impact on compactionsCompletedCount, but a small impact on 
numBytesCompactedCount. This is reasonable, the patch only prevent flushing 
small files of small CFs and reduce its compaction number, but most 
numBytesCompactedCount is contributed by large CFs which is not effected(or at 
least, very small) by this patch. So we only get a small improvement of write 
amplification(5%~10%).


 Port 'Make flush decisions per column family' to trunk
 --

 Key: HBASE-10201
 URL: https://issues.apache.org/jira/browse/HBASE-10201
 Project: HBase
  Issue Type: Improvement
Reporter: Ted Yu
 Attachments: 3149-trunk-v1.txt, HBASE-10201-0.98.patch, 
 HBASE-10201-0.98_1.patch, HBASE-10201-0.98_2.patch


 Currently the flush decision is made using the aggregate size of all column 
 families. When large and small column families co-exist, this causes many 
 small flushes of the smaller CF. We need to make per-CF flush decisions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-12266) Slow Scan can cause dead loop in ClientScanner

2014-10-15 Thread Hadoop QA (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-12266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172281#comment-14172281
]

Hadoop QA commented on HBASE-12266:
---

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment

http://issues.apache.org/jira/secure/attachment/12674963/HBASE-12266-master.patch
against trunk revision .
ATTACHMENT ID: 12674963