date:20131216

[jira] [Commented] (HBASE-10169) Batch coprocessor

2013-12-16 Thread Hadoop QA (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-10169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848909#comment-13848909
]

Hadoop QA commented on HBASE-10169:
---

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment

http://issues.apache.org/jira/secure/attachment/12618869/Batch%20Coprocessor%20Design%20Document.docx
against trunk revision .

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.

{color:red}-1 patch{color}. The patch command could not apply the patch.

Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8175//console

This message is automatically generated.

Batch coprocessor
-

Key: HBASE-10169
URL: https://issues.apache.org/jira/browse/HBASE-10169
Project: HBase
Issue Type: Sub-task
Components: Coprocessors
Affects Versions: 0.99.0
Reporter: Jingcheng Du
Assignee: Jingcheng Du
Attachments: Batch Coprocessor Design Document.docx, HBASE-10169.patch

This is designed to improve the coprocessor invocation in the client side.
Currently the coprocessor invocation is to send a call to each region. If
there’s one region server, and 100 regions are located in this server, each
coprocessor invocation will send 100 calls, each call uses a single thread in
the client side. The threads will run out soon when the coprocessor
invocations are heavy.
In this design, all the calls to the same region server will be grouped into
one in a single coprocessor invocation. This call will be spread into each
region in the server side, and the results will be merged ahead in the server
side before being returned to the client.

--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HBASE-10137) GeneralBulkAssigner with retain assignment plan can be used in EnableTableHandler to bulk assign the regions

2013-12-16 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-10137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13849025#comment-13849025
 ] 

ramkrishna.s.vasudevan commented on HBASE-10137:


As this is fixed in Trunk this change should be fine.  
+1 for being only on trunk.
One quesiton is 
bq.In case of any failures retrying assignments and wait in 
GeneralBulkAssigner#waitUntilDone.
Yes this would happen now.
But after disabled had happened before enabling, if a  REgion server goes down 
and going with retainAssignment? how would this work?  
META may have the dead RS entry but the servermanager won't be giving the old 
ones.  Will this be handled here?

 GeneralBulkAssigner with retain assignment plan can be used in 
 EnableTableHandler to bulk assign the regions
 

 Key: HBASE-10137
 URL: https://issues.apache.org/jira/browse/HBASE-10137
 Project: HBase
  Issue Type: Bug
  Components: Region Assignment
Affects Versions: 0.96.0, 0.94.14
Reporter: rajeshbabu
Assignee: rajeshbabu
 Fix For: 0.98.0, 0.96.2, 0.99.0

 Attachments: HBASE-10137.patch


 Current in BulkEnabler we are assigning one region at a time, instead we can 
 use GeneralBulkAssigner to bulk assign multiple regions at a time.
 {code}
   for (HRegionInfo region : regions) {
 if (assignmentManager.getRegionStates()
 .isRegionInTransition(region)) {
   continue;
 }
 final HRegionInfo hri = region;
 pool.execute(Trace.wrap(BulkEnabler.populatePool,new Runnable() {
   public void run() {
 assignmentManager.assign(hri, true);
   }
 }));
   }
 {code}



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HBASE-10137) GeneralBulkAssigner with retain assignment plan can be used in EnableTableHandler to bulk assign the regions

2013-12-16 Thread rajeshbabu (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-10137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13849040#comment-13849040
 ] 

rajeshbabu commented on HBASE-10137:


@Ram,
bq. But after disabled had happened before enabling, if a REgion server goes 
down and going with retainAssignment? how would this work? META may have the 
dead RS entry but the servermanager won't be giving the old ones. Will this be 
handled here?
If a region server is dead and no new RS started in the same host, then 
randomly one server from online servers(list from server manager) will be 
selected for the region. Its fine only.
{code}
  if (localServers.isEmpty()) {
// No servers on the new cluster match up with this hostname,
// assign randomly.
ServerName randomServer = servers.get(RANDOM.nextInt(servers.size()));
assignments.get(randomServer).add(region);
numRandomAssignments++;
{code}

 GeneralBulkAssigner with retain assignment plan can be used in 
 EnableTableHandler to bulk assign the regions
 

 Key: HBASE-10137
 URL: https://issues.apache.org/jira/browse/HBASE-10137
 Project: HBase
  Issue Type: Bug
  Components: Region Assignment
Affects Versions: 0.96.0, 0.94.14
Reporter: rajeshbabu
Assignee: rajeshbabu
 Fix For: 0.98.0, 0.96.2, 0.99.0

 Attachments: HBASE-10137.patch


 Current in BulkEnabler we are assigning one region at a time, instead we can 
 use GeneralBulkAssigner to bulk assign multiple regions at a time.
 {code}
   for (HRegionInfo region : regions) {
 if (assignmentManager.getRegionStates()
 .isRegionInTransition(region)) {
   continue;
 }
 final HRegionInfo hri = region;
 pool.execute(Trace.wrap(BulkEnabler.populatePool,new Runnable() {
   public void run() {
 assignmentManager.assign(hri, true);
   }
 }));
   }
 {code}



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HBASE-10137) GeneralBulkAssigner with retain assignment plan can be used in EnableTableHandler to bulk assign the regions

2013-12-16 Thread ramkrishna.s.vasudevan (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-10137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13849045#comment-13849045
 ] 

ramkrishna.s.vasudevan commented on HBASE-10137:


+1 on commit then.  
[~jxiang]
Want to take a look before commit?

 GeneralBulkAssigner with retain assignment plan can be used in 
 EnableTableHandler to bulk assign the regions
 

 Key: HBASE-10137
 URL: https://issues.apache.org/jira/browse/HBASE-10137
 Project: HBase
  Issue Type: Bug
  Components: Region Assignment
Affects Versions: 0.96.0, 0.94.14
Reporter: rajeshbabu
Assignee: rajeshbabu
 Fix For: 0.98.0, 0.96.2, 0.99.0

 Attachments: HBASE-10137.patch


 Current in BulkEnabler we are assigning one region at a time, instead we can 
 use GeneralBulkAssigner to bulk assign multiple regions at a time.
 {code}
   for (HRegionInfo region : regions) {
 if (assignmentManager.getRegionStates()
 .isRegionInTransition(region)) {
   continue;
 }
 final HRegionInfo hri = region;
 pool.execute(Trace.wrap(BulkEnabler.populatePool,new Runnable() {
   public void run() {
 assignmentManager.assign(hri, true);
   }
 }));
   }
 {code}



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Re: HBase upgrade error

2013-12-16 Thread Ted Yu

The exception happened in protobuf

Can you upgrade hadoop to release 2.2 ?

Cheers

On Dec 15, 2013, at 11:58 PM, hzwangxx whwangxingx...@163.com wrote:

 Hi，
I want to upgrade hbase from 0.94 to 0.96. According to 
 http://hbase.apache.org/book/upgrade0.96.html. I shutdown 0.94 cluster, use 
 bin/hbase upgrade -check, I encountered a problem as follows:
 
 13/12/16 15:43:08 INFO util.HFileV1Detector: Target dir is: 
 hdfs://node01:9000/hbase
 13/12/16 15:43:08 ERROR util.HFileV1Detector: 
 java.lang.UnsupportedOperationException: This is supposed to be overridden by 
 subclasses.
 13/12/16 15:43:08 WARN migration.UpgradeTo96: There are some HFileV1, or 
 corrupt files (files with incorrect major version).
 
 and use bin/hbase upgrade -execute can list the trackers:
 13/12/16 15:45:56 INFO zookeeper.ClientCnxn: EventThread shut down
 13/12/16 15:45:56 INFO zookeeper.ZooKeeper: Session: 0x142f975e49e0003 closed
 13/12/16 15:45:56 INFO migration.UpgradeTo96: Starting Namespace upgrade
 13/12/16 15:45:57 WARN conf.Configuration: fs.default.name is deprecated. 
 Instead, use fs.defaultFS
 Exception in thread main java.lang.UnsupportedOperationException: This is 
 supposed to be overridden by subclasses.
at 
 com.google.protobuf.GeneratedMessage.getUnknownFields(GeneratedMessage.java:180)
at 
 org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$GetListingRequestProto.getSerializedSize(ClientNamenodeProtocolProtos.java:18005)
at 
 com.google.protobuf.AbstractMessageLite.toByteString(AbstractMessageLite.java:49)
at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.constructRpcRequest(ProtobufRpcEngine.java:149)
at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:193)
at sun.proxy.$Proxy10.getListing(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at 
 org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
at 
 org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
at sun.proxy.$Proxy10.getListing(Unknown Source)
at 
 org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:440)
at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1526)
at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1509)
at 
 org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:407)
at org.apache.hadoop.hbase.util.FSUtils.getVersion(FSUtils.java:473)
at 
 org.apache.hadoop.hbase.migration.NamespaceUpgrade.verifyNSUpgrade(NamespaceUpgrade.java:488)
at 
 org.apache.hadoop.hbase.migration.NamespaceUpgrade.upgradeTableDirs(NamespaceUpgrade.java:127)
at 
 org.apache.hadoop.hbase.migration.NamespaceUpgrade.run(NamespaceUpgrade.java:502)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at 
 org.apache.hadoop.hbase.migration.UpgradeTo96.executeTool(UpgradeTo96.java:222)
at 
 org.apache.hadoop.hbase.migration.UpgradeTo96.executeUpgrade(UpgradeTo96.java:212)
at 
 org.apache.hadoop.hbase.migration.UpgradeTo96.run(UpgradeTo96.java:134)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at 
 org.apache.hadoop.hbase.migration.UpgradeTo96.main(UpgradeTo96.java:258)
 
 my hadoop version is: hadoop-2.0.0-cdh4.2.1 and hbase version is 
 0.94.0-cdh4.2.1.
 
 Waiting for your help. Thanks~
 Best Wishes!
 hzwangxx

[jira] [Commented] (HBASE-10087) Store should be locked during a memstore snapshot

2013-12-16 Thread Nicolas Liochon (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-10087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13849215#comment-13849215
 ] 

Nicolas Liochon commented on HBASE-10087:
-

Sorry for the delay.

Theoretically, we have an issue:
{code}
@Override
public void prepare() {
  memstore.snapshot();
  this.snapshot = memstore.getSnapshot();
  this.snapshotTimeRangeTracker = memstore.getSnapshotTimeRangeTracker(); 
}
{code}
w/o a lock, we could have an inconsistency between the snapshot and the 
snapshotTimeRangeTracker.
As well, we could have an inconsistency in memstore.snapshot();

But it seems it cannot happen, because: 
Store#snaphot is called only in the tests.
Store#prepare is called with the write lock on the HRegion.
The functions that will modify the store also have the read lock on HRegion, so 
we can't be inconsistent in practice.

This said, adding the lock in prepare() would be more consistent imho. I would 
propose to add it on trunk.




 Store should be locked during a memstore snapshot
 -

 Key: HBASE-10087
 URL: https://issues.apache.org/jira/browse/HBASE-10087
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.98.0, 0.96.1, 0.94.14
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
 Fix For: 0.98.0, 0.96.1, 0.94.15

 Attachments: 10079.v1.patch


 regression from HBASE-9963, found while looking at HBASE-10079.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

1 2 3 >

1 - 100 of 297 matches

Mail list logo