[jira] [Commented] (HBASE-5074) support checksums in HBase block cache
[ https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13218003#comment-13218003 ] Zhihong Yu commented on HBASE-5074: --- I wish I had been more prudent before making the previous comments. support checksums in HBase block cache -- Key: HBASE-5074 URL: https://issues.apache.org/jira/browse/HBASE-5074 Project: HBase Issue Type: Improvement Components: regionserver Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: D1521.1.patch, D1521.1.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.2.patch, D1521.2.patch, D1521.3.patch, D1521.3.patch, D1521.4.patch, D1521.4.patch, D1521.5.patch, D1521.5.patch, D1521.6.patch, D1521.6.patch, D1521.7.patch, D1521.7.patch, D1521.8.patch, D1521.8.patch, D1521.9.patch, D1521.9.patch The current implementation of HDFS stores the data in one block file and the metadata(checksum) in another block file. This means that every read into the HBase block cache actually consumes two disk iops, one to the datafile and one to the checksum file. This is a major problem for scaling HBase, because HBase is usually bottlenecked on the number of random disk iops that the storage-hardware offers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5482) In 0.90, balancer algo leading to same region balanced twice and picking same region with Src and Destination as same RS.
[
https://issues.apache.org/jira/browse/HBASE-5482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-5482:
--
Attachment: HBASE-5482_1.patch
Patch for 0.90.
In 0.90, balancer algo leading to same region balanced twice and picking same
region with Src and Destination as same RS.
-
Key: HBASE-5482
URL: https://issues.apache.org/jira/browse/HBASE-5482
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.90.5
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.90.7
Attachments: HBASE-5482_1.patch
There are possibility of 2 problems
- When we populate regionsToMove while iterating the serverinfo in
descending manner there is a chance that the same region can be added twice.
Because in the first loop we do a randomization of the regions.
Where as when we get we have neededRegions!= 0 we just get the region in the
index and add it again . This may lead to have same region in the
regionsToMove list.
- Another problem is
when the problem in the first point happens then there is a chance that
the regionToMove can have the same src and destination and the same region
can be picked every 5 mins.
{code}
for(Map.EntryHServerInfo, ListHRegionInfo server :
serversByLoad.descendingMap().entrySet()) {
BalanceInfo balanceInfo = serverBalanceInfo.get(server.getKey());
int idx =
balanceInfo == null ? 0 : balanceInfo.getNextRegionForUnload();
if (idx = server.getValue().size()) break;
HRegionInfo region = server.getValue().get(idx);
if (region.isMetaRegion()) continue; // Don't move meta regions.
regionsToMove.add(new RegionPlan(region, server.getKey(), null));
if(--neededRegions == 0) {
// No more regions needed, done shedding
break;
}
}
{code}
If i have meta and root in the top two loaded region server(totally 3 RS), we
just skip the regions in those region server and populate the region from the
least loaded RS.
Then in the next loop we iterate from the least loaded server and populate
the destination as also the same server.
This is leading to a condition where every 5 min balancing happens and also
the server is same for src and dest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5482) In 0.90, balancer algo leading to same region balanced twice and picking same region with Src and Destination as same RS.
[
https://issues.apache.org/jira/browse/HBASE-5482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13218246#comment-13218246
]
ramkrishna.s.vasudevan commented on HBASE-5482:
---
Checking if the patch similar to this is needed in trunk. Once confirmed will
prepare a patch for 0.92.
In 0.90, balancer algo leading to same region balanced twice and picking same
region with Src and Destination as same RS.
-
Key: HBASE-5482
URL: https://issues.apache.org/jira/browse/HBASE-5482
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.90.5
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.90.7
Attachments: HBASE-5482_1.patch
There are possibility of 2 problems
- When we populate regionsToMove while iterating the serverinfo in
descending manner there is a chance that the same region can be added twice.
Because in the first loop we do a randomization of the regions.
Where as when we get we have neededRegions!= 0 we just get the region in the
index and add it again . This may lead to have same region in the
regionsToMove list.
- Another problem is
when the problem in the first point happens then there is a chance that
the regionToMove can have the same src and destination and the same region
can be picked every 5 mins.
{code}
for(Map.EntryHServerInfo, ListHRegionInfo server :
serversByLoad.descendingMap().entrySet()) {
BalanceInfo balanceInfo = serverBalanceInfo.get(server.getKey());
int idx =
balanceInfo == null ? 0 : balanceInfo.getNextRegionForUnload();
if (idx = server.getValue().size()) break;
HRegionInfo region = server.getValue().get(idx);
if (region.isMetaRegion()) continue; // Don't move meta regions.
regionsToMove.add(new RegionPlan(region, server.getKey(), null));
if(--neededRegions == 0) {
// No more regions needed, done shedding
break;
}
}
{code}
If i have meta and root in the top two loaded region server(totally 3 RS), we
just skip the regions in those region server and populate the region from the
least loaded RS.
Then in the next loop we iterate from the least loaded server and populate
the destination as also the same server.
This is leading to a condition where every 5 min balancing happens and also
the server is same for src and dest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5399) Cut the link between the client and the zookeeper ensemble
[
https://issues.apache.org/jira/browse/HBASE-5399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13218257#comment-13218257
]
nkeywal commented on HBASE-5399:
If we really want it, I found 3 options, and tried 2.
1) Adding 'close' to the HMasterInterface
After looking at it, I don't think it's a good option: HMasterInterface is an
interface shared between the client the server. So adding a close function to
it would mean the server must implement it, while it's a client side function
only. I believe that's the reason why there is already a function 'stopProxy'
in the RPCENgine instead of a close function.
2) Adding the possibility to have a delayed close in RPCENgine
Instead of doing it for HMasterInterface in Connection only, we could do it all
proxies and code this in RPCENgine.
There is already a reference counting in the hbase RPCENgine. So we could add
here some code to allow a delayed close. I don't see why it would not be
possible, all the code seems to be in the hbase package (and not hadoop). This
would require smart convention to make it configurable on a per proxy basis,
but it should work.
3) Add an class with a delegation
So I've got this
{noformat}
public interface SharedMaster extends HMasterInterface, Closeable {}
{noformat}
With this in HConnection
{noformat}
public interface HConnection extends Abortable, Closeable {
public SharedMaster getSharedMaster()
}
{noformat}
Then the client writes
{noformat}
SharedMaster master = connection.getSharedMaster();
try {
master.move(encodedRegionName, destServerName);
} finally {
master.close();
}
{noformat}
With a java proxy to manage the delegation for us:
{noformat}
private static class SharedMasterHandler implements InvocationHandler {
private HConnectionImplementation connection;
private HMasterInterface master;
SharedMasterHandler(HConnectionImplementation connection,
HMasterInterface master){
this.connection = connection;
this.master = master;
}
@Override
public Object invoke(Object proxy, Method method, Object[] args) throws
Throwable {
if (method.getName().equals(close)){
connection.releaseSharedMaster((HMasterInterface)master);
return null;
} else {
return method.invoke(master, args);
}
}
}
{noformat}
releaseSharedMaster is private in this solution.
It was not really my first idea, but it's a reasonable way to get to the
objective. The reflective delegation is not fast, obviously, but it doesn't
matter here as there is much more expensive remote call just after...
I'am currently testing it, it seems to work.
Cut the link between the client and the zookeeper ensemble
--
Key: HBASE-5399
URL: https://issues.apache.org/jira/browse/HBASE-5399
Project: HBase
Issue Type: Improvement
Components: client
Affects Versions: 0.94.0
Environment: all
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
Attachments: 5399_inprogress.patch, 5399_inprogress.v3.patch,
5399_inprogress.v9.patch
The link is often considered as an issue, for various reasons. One of them
being that there is a limit on the number of connection that ZK can manage.
Stack was suggesting as well to remove the link to master from HConnection.
There are choices to be made considering the existing API (that we don't want
to break).
The first patches I will submit on hadoop-qa should not be committed: they
are here to show the progress on the direction taken.
ZooKeeper is used for:
- public getter, to let the client do whatever he wants, and close ZooKeeper
when closing the connection = we have to deprecate this but keep it.
- read get master address to create a master = now done with a temporary
zookeeper connection
- read root location = now done with a temporary zookeeper connection, but
questionable. Used in public function locateRegion. To be reworked.
- read cluster id = now done once with a temporary zookeeper connection.
- check if base done is available = now done once with a zookeeper
connection given as a parameter
- isTableDisabled/isTableAvailable = public functions, now done with a
temporary zookeeper connection.
- Called internally from HBaseAdmin and HTable
- getCurrentNrHRS(): public function to get the number of region servers and
create a pool of thread = now done with a temporary zookeeper connection
-
Master is used for:
- getMaster public getter, as for ZooKeeper = we have to deprecate this but
keep it.
- isMasterRunning(): public function, used internally by HMerge HBaseAdmin
- getHTableDescriptor*: public functions offering access to the master. =
we could make them using a
[jira] [Created] (HBASE-5486) Warn message in HTable: Stringify the byte[]
Warn message in HTable: Stringify the byte[]
Key: HBASE-5486
URL: https://issues.apache.org/jira/browse/HBASE-5486
Project: HBase
Issue Type: Bug
Components: client
Reporter: Himanshu Vashishtha
Priority: Trivial
The warn message in the method getStartEndKeys() in HTable can be improved by
stringifying the byte array for Regions.Qualifier
Currently, a sample message is like :
12/01/17 16:36:34 WARN client.HTable: Null [B@552c8fa8 cell in
keyvalues={test5,\xC9\xA2\x00\x00\x00\x00\x00\x00/00_0,1326642537734.dbc62b2765529a9ad2ddcf8eb58cb2dc./info:server/1326750341579/Put/vlen=28,
test5,\xC9\xA2\x00\x00\x00\x00\x00\x00/00_0,1326642537734.dbc62b2765529a9ad2ddcf8eb58cb2dc./info:serverstartcode/1326750341579/Put/vlen=8}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5486) Warn message in HTable: Stringify the byte[]
[
https://issues.apache.org/jira/browse/HBASE-5486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Himanshu Vashishtha updated HBASE-5486:
---
Assignee: Himanshu Vashishtha
Affects Version/s: 0.92.0
Status: Patch Available (was: Open)
Warn message in HTable: Stringify the byte[]
Key: HBASE-5486
URL: https://issues.apache.org/jira/browse/HBASE-5486
Project: HBase
Issue Type: Bug
Components: client
Affects Versions: 0.92.0
Reporter: Himanshu Vashishtha
Assignee: Himanshu Vashishtha
Priority: Trivial
Labels: noob
Attachments: 5486.patch
The warn message in the method getStartEndKeys() in HTable can be improved by
stringifying the byte array for Regions.Qualifier
Currently, a sample message is like :
12/01/17 16:36:34 WARN client.HTable: Null [B@552c8fa8 cell in
keyvalues={test5,\xC9\xA2\x00\x00\x00\x00\x00\x00/00_0,1326642537734.dbc62b2765529a9ad2ddcf8eb58cb2dc./info:server/1326750341579/Put/vlen=28,
test5,\xC9\xA2\x00\x00\x00\x00\x00\x00/00_0,1326642537734.dbc62b2765529a9ad2ddcf8eb58cb2dc./info:serverstartcode/1326750341579/Put/vlen=8}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5486) Warn message in HTable: Stringify the byte[]
[
https://issues.apache.org/jira/browse/HBASE-5486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Himanshu Vashishtha updated HBASE-5486:
---
Attachment: 5486.patch
Warn message in HTable: Stringify the byte[]
Key: HBASE-5486
URL: https://issues.apache.org/jira/browse/HBASE-5486
Project: HBase
Issue Type: Bug
Components: client
Affects Versions: 0.92.0
Reporter: Himanshu Vashishtha
Assignee: Himanshu Vashishtha
Priority: Trivial
Labels: noob
Attachments: 5486.patch
The warn message in the method getStartEndKeys() in HTable can be improved by
stringifying the byte array for Regions.Qualifier
Currently, a sample message is like :
12/01/17 16:36:34 WARN client.HTable: Null [B@552c8fa8 cell in
keyvalues={test5,\xC9\xA2\x00\x00\x00\x00\x00\x00/00_0,1326642537734.dbc62b2765529a9ad2ddcf8eb58cb2dc./info:server/1326750341579/Put/vlen=28,
test5,\xC9\xA2\x00\x00\x00\x00\x00\x00/00_0,1326642537734.dbc62b2765529a9ad2ddcf8eb58cb2dc./info:serverstartcode/1326750341579/Put/vlen=8}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5486) Warn message in HTable: Stringify the byte[]
[
https://issues.apache.org/jira/browse/HBASE-5486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13218358#comment-13218358
]
Hadoop QA commented on HBASE-5486:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12516355/5486.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
-1 patch. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1058//console
This message is automatically generated.
Warn message in HTable: Stringify the byte[]
Key: HBASE-5486
URL: https://issues.apache.org/jira/browse/HBASE-5486
Project: HBase
Issue Type: Bug
Components: client
Affects Versions: 0.92.0
Reporter: Himanshu Vashishtha
Assignee: Himanshu Vashishtha
Priority: Trivial
Labels: noob
Attachments: 5486.patch
The warn message in the method getStartEndKeys() in HTable can be improved by
stringifying the byte array for Regions.Qualifier
Currently, a sample message is like :
12/01/17 16:36:34 WARN client.HTable: Null [B@552c8fa8 cell in
keyvalues={test5,\xC9\xA2\x00\x00\x00\x00\x00\x00/00_0,1326642537734.dbc62b2765529a9ad2ddcf8eb58cb2dc./info:server/1326750341579/Put/vlen=28,
test5,\xC9\xA2\x00\x00\x00\x00\x00\x00/00_0,1326642537734.dbc62b2765529a9ad2ddcf8eb58cb2dc./info:serverstartcode/1326750341579/Put/vlen=8}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5486) Warn message in HTable: Stringify the byte[]
[
https://issues.apache.org/jira/browse/HBASE-5486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13218361#comment-13218361
]
Himanshu Vashishtha commented on HBASE-5486:
This is converting a static byte array into a String, I don't think testing
will add anything. Please let me know if you have other thoughts.
Warn message in HTable: Stringify the byte[]
Key: HBASE-5486
URL: https://issues.apache.org/jira/browse/HBASE-5486
Project: HBase
Issue Type: Bug
Components: client
Affects Versions: 0.92.0
Reporter: Himanshu Vashishtha
Assignee: Himanshu Vashishtha
Priority: Trivial
Labels: noob
Attachments: 5486.patch
The warn message in the method getStartEndKeys() in HTable can be improved by
stringifying the byte array for Regions.Qualifier
Currently, a sample message is like :
12/01/17 16:36:34 WARN client.HTable: Null [B@552c8fa8 cell in
keyvalues={test5,\xC9\xA2\x00\x00\x00\x00\x00\x00/00_0,1326642537734.dbc62b2765529a9ad2ddcf8eb58cb2dc./info:server/1326750341579/Put/vlen=28,
test5,\xC9\xA2\x00\x00\x00\x00\x00\x00/00_0,1326642537734.dbc62b2765529a9ad2ddcf8eb58cb2dc./info:serverstartcode/1326750341579/Put/vlen=8}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5120) Timeout monitor races with table disable handler
[
https://issues.apache.org/jira/browse/HBASE-5120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-5120:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Closed. Reopen if I have it wrong Ram.
Timeout monitor races with table disable handler
Key: HBASE-5120
URL: https://issues.apache.org/jira/browse/HBASE-5120
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Zhihong Yu
Assignee: ramkrishna.s.vasudevan
Priority: Blocker
Fix For: 0.94.0
Attachments: HBASE-5120.patch, HBASE-5120_1.patch,
HBASE-5120_2.patch, HBASE-5120_3.patch, HBASE-5120_4.patch,
HBASE-5120_5.patch, HBASE-5120_5.patch
Here is what J-D described here:
https://issues.apache.org/jira/browse/HBASE-5119?focusedCommentId=13179176page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13179176
I think I will retract from my statement that it used to be extremely racy
and caused more troubles than it fixed, on my first test I got a stuck
region in transition instead of being able to recover. The timeout was set to
2 minutes to be sure I hit it.
First the region gets closed
{quote}
2012-01-04 00:16:25,811 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Sent CLOSE to
sv4r5s38,62023,1325635980913 for region
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
{quote}
2 minutes later it times out:
{quote}
2012-01-04 00:18:30,026 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
out: test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
state=PENDING_CLOSE, ts=1325636185810, server=null
2012-01-04 00:18:30,026 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Region has been
PENDING_CLOSE for too long, running forced unassign again on
region=test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
2012-01-04 00:18:30,027 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
region test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
(offlining)
{quote}
100ms later the master finally gets the event:
{quote}
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_CLOSED, server=sv4r5s38,62023,1325635980913,
region=1a4b111bcc228043e89f59c4c3f6a791, which is more than 15 seconds late
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED
event for 1a4b111bcc228043e89f59c4c3f6a791
2012-01-04 00:18:30,129 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Table being disabled so
deleting ZK node and removing from regions in transition, skipping assignment
of region test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
2012-01-04 00:18:30,129 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Deleting existing unassigned node for
1a4b111bcc228043e89f59c4c3f6a791 that is in expected state RS_ZK_REGION_CLOSED
2012-01-04 00:18:30,166 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Successfully deleted unassigned node for
region 1a4b111bcc228043e89f59c4c3f6a791 in expected state RS_ZK_REGION_CLOSED
{quote}
At this point everything is fine, the region was processed as closed. But
wait, remember that line where it said it was going to force an unassign?
{quote}
2012-01-04 00:18:30,322 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:62003-0x134589d3db03587 Creating unassigned node for
1a4b111bcc228043e89f59c4c3f6a791 in a CLOSING state
2012-01-04 00:18:30,328 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Server null returned
java.lang.NullPointerException: Passed server is null for
1a4b111bcc228043e89f59c4c3f6a791
{quote}
Now the master is confused, it recreated the RIT znode but the region doesn't
even exist anymore. It even tries to shut it down but is blocked by NPEs. Now
this is what's going on.
The late ZK notification that the znode was deleted (but it got recreated
after):
{quote}
2012-01-04 00:19:33,285 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: The znode of region
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791. has been
deleted.
{quote}
Then it prints this, and much later tries to unassign it again:
{quote}
2012-01-04 00:19:46,607 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on region
to clear regions in transition;
test1,089cd0c9,1325635015491.1a4b111bcc228043e89f59c4c3f6a791.
state=PENDING_CLOSE, ts=1325636310328, server=null
...
2012-01-04 00:20:39,623 DEBUG
[jira] [Commented] (HBASE-5270) Handle potential data loss due to concurrent processing of processFaileOver and ServerShutdownHandler
[ https://issues.apache.org/jira/browse/HBASE-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13218407#comment-13218407 ] Prakash Khemani commented on HBASE-5270: Assuming that the master uses the saved region-server list in joinCluster, can you then please outline the scenario where problems can still happen? There is some handling of META and ROOT not being available in ServerShutdownHandler and I am wondering why that is not sufficient. On 2/27/12 11:17 PM, chunhui shen (Commented) (JIRA) j...@apache.org Handle potential data loss due to concurrent processing of processFaileOver and ServerShutdownHandler - Key: HBASE-5270 URL: https://issues.apache.org/jira/browse/HBASE-5270 Project: HBase Issue Type: Sub-task Components: master Reporter: Zhihong Yu Assignee: chunhui shen Fix For: 0.92.1, 0.94.0 Attachments: 5270-90-testcase.patch, 5270-90-testcasev2.patch, 5270-90.patch, 5270-90v2.patch, 5270-90v3.patch, 5270-testcase.patch, 5270-testcasev2.patch, hbase-5270.patch, hbase-5270v2.patch, hbase-5270v4.patch, hbase-5270v5.patch, hbase-5270v6.patch, hbase-5270v7.patch, hbase-5270v8.patch, sampletest.txt This JIRA continues the effort from HBASE-5179. Starting with Stack's comments about patches for 0.92 and TRUNK: Reviewing 0.92v17 isDeadServerInProgress is a new public method in ServerManager but it does not seem to be used anywhere. Does isDeadRootServerInProgress need to be public? Ditto for meta version. This method param names are not right 'definitiveRootServer'; what is meant by definitive? Do they need this qualifier? Is there anything in place to stop us expiring a server twice if its carrying root and meta? What is difference between asking assignment manager isCarryingRoot and this variable that is passed in? Should be doc'd at least. Ditto for meta. I think I've asked for this a few times - onlineServers needs to be explained... either in javadoc or in comment. This is the param passed into joinCluster. How does it arise? I think I know but am unsure. God love the poor noob that comes awandering this code trying to make sense of it all. It looks like we get the list by trawling zk for regionserver znodes that have not checked in. Don't we do this operation earlier in master setup? Are we doing it again here? Though distributed split log is configured, we will do in master single process splitting under some conditions with this patch. Its not explained in code why we would do this. Why do we think master log splitting 'high priority' when it could very well be slower. Should we only go this route if distributed splitting is not going on. Do we know if concurrent distributed log splitting and master splitting works? Why would we have dead servers in progress here in master startup? Because a servershutdownhandler fired? This patch is different to the patch for 0.90. Should go into trunk first with tests, then 0.92. Should it be in this issue? This issue is really hard to follow now. Maybe this issue is for 0.90.x and new issue for more work on this trunk patch? This patch needs to have the v18 differences applied. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5484) Spelling mistake in error message in HMasterCommandLine
[ https://issues.apache.org/jira/browse/HBASE-5484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David S. Wang updated HBASE-5484: - Attachment: diff1.txt Spelling mistake in error message in HMasterCommandLine --- Key: HBASE-5484 URL: https://issues.apache.org/jira/browse/HBASE-5484 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.90.5, 0.92.0 Reporter: David S. Wang Assignee: David S. Wang Priority: Trivial Labels: newbie, noob Fix For: 0.94.0 Attachments: diff1.txt If hadoop-zookeeper-server is installed and started, starting hbase-master in standalone mode will display this error message which has some typos. $ sudo /etc/init.d/hadoop-hbase-master start Starting Hadoop HBase master daemon: starting master, logging to /usr/lib/hbase/logs/hbase-hbase-master/cloudera-vm.out Couldnt start ZK at requested address of 2181, instead got: 2182. Aborting. Why? Because clients (eg shell) wont be able to find this ZK quorum hbase-master. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5484) Spelling mistake in error message in HMasterCommandLine
[ https://issues.apache.org/jira/browse/HBASE-5484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David S. Wang updated HBASE-5484: - Status: Patch Available (was: In Progress) Unit tested: Ran test suite without any new errors. Unit tested by starting HBase in standalone mode and setting the ZK port to a TCP port that was already bound to, to hit this message: java.io.IOException: Could not start ZK at requested port of 56071. ZK was started at port: 56074. Aborting as clients (e.g. shell) will not be able to find this ZK quorum. Ran test-patch.sh with the following results: -1 overall. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -131 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version ) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. Spelling mistake in error message in HMasterCommandLine --- Key: HBASE-5484 URL: https://issues.apache.org/jira/browse/HBASE-5484 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.92.0, 0.90.5 Reporter: David S. Wang Assignee: David S. Wang Priority: Trivial Labels: newbie, noob Fix For: 0.94.0 Attachments: diff1.txt If hadoop-zookeeper-server is installed and started, starting hbase-master in standalone mode will display this error message which has some typos. $ sudo /etc/init.d/hadoop-hbase-master start Starting Hadoop HBase master daemon: starting master, logging to /usr/lib/hbase/logs/hbase-hbase-master/cloudera-vm.out Couldnt start ZK at requested address of 2181, instead got: 2182. Aborting. Why? Because clients (eg shell) wont be able to find this ZK quorum hbase-master. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5074) support checksums in HBase block cache
[ https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13218423#comment-13218423 ] dhruba borthakur commented on HBASE-5074: - @Stack: I am pretty sure that the feature is on by default (but let me check and get back to you). Regarding the exception message about CRC32C, the Enum is trying to create this object but failing to do so because the Hadoop library in Hadoop 1.0 does not have support for this one (Hadop 2.0 supports CRC32C). The reason I kept that is because people who might already be experimenting with Hadoop 2.0 will get this support out-of-the-box. But I agree that it will be good to get rid of this exception message at startup. Do you have any suggestions on this one? @Todd: will take your excellent suggestion and make the majorVersion inside HFileBlock as a static. Thanks. @Ted: Thanks for your comments. Will try to gather metrics in my cluster and post to this JIRA. support checksums in HBase block cache -- Key: HBASE-5074 URL: https://issues.apache.org/jira/browse/HBASE-5074 Project: HBase Issue Type: Improvement Components: regionserver Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: D1521.1.patch, D1521.1.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.2.patch, D1521.2.patch, D1521.3.patch, D1521.3.patch, D1521.4.patch, D1521.4.patch, D1521.5.patch, D1521.5.patch, D1521.6.patch, D1521.6.patch, D1521.7.patch, D1521.7.patch, D1521.8.patch, D1521.8.patch, D1521.9.patch, D1521.9.patch The current implementation of HDFS stores the data in one block file and the metadata(checksum) in another block file. This means that every read into the HBase block cache actually consumes two disk iops, one to the datafile and one to the checksum file. This is a major problem for scaling HBase, because HBase is usually bottlenecked on the number of random disk iops that the storage-hardware offers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5074) support checksums in HBase block cache
[ https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13218470#comment-13218470 ] stack commented on HBASE-5074: -- @Dhruba Its good trying for PureJavaCrc32 first. Get rid of the WARN w/ thread dump I'd say especially as is where it comes after reporting we're not going to use PureJavaCrc32. The feature does seem to be on by default but it would be nice to know it w/o having to go to ganglia graphs to figure my i/o loading to see whether or not this feature is enabled -- going to ganglia would be useless anyways in case where I've no history w/ an hbase read load -- so some kind of log output might be useful? Good on you D. support checksums in HBase block cache -- Key: HBASE-5074 URL: https://issues.apache.org/jira/browse/HBASE-5074 Project: HBase Issue Type: Improvement Components: regionserver Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: D1521.1.patch, D1521.1.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.2.patch, D1521.2.patch, D1521.3.patch, D1521.3.patch, D1521.4.patch, D1521.4.patch, D1521.5.patch, D1521.5.patch, D1521.6.patch, D1521.6.patch, D1521.7.patch, D1521.7.patch, D1521.8.patch, D1521.8.patch, D1521.9.patch, D1521.9.patch The current implementation of HDFS stores the data in one block file and the metadata(checksum) in another block file. This means that every read into the HBase block cache actually consumes two disk iops, one to the datafile and one to the checksum file. This is a major problem for scaling HBase, because HBase is usually bottlenecked on the number of random disk iops that the storage-hardware offers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5484) Spelling mistake in error message in HMasterCommandLine
[ https://issues.apache.org/jira/browse/HBASE-5484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5484: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed trunk and to 0.94 branch. Spelling mistake in error message in HMasterCommandLine --- Key: HBASE-5484 URL: https://issues.apache.org/jira/browse/HBASE-5484 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.90.5, 0.92.0 Reporter: David S. Wang Assignee: David S. Wang Priority: Trivial Labels: newbie, noob Fix For: 0.94.0 Attachments: diff1.txt If hadoop-zookeeper-server is installed and started, starting hbase-master in standalone mode will display this error message which has some typos. $ sudo /etc/init.d/hadoop-hbase-master start Starting Hadoop HBase master daemon: starting master, logging to /usr/lib/hbase/logs/hbase-hbase-master/cloudera-vm.out Couldnt start ZK at requested address of 2181, instead got: 2182. Aborting. Why? Because clients (eg shell) wont be able to find this ZK quorum hbase-master. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5486) Warn message in HTable: Stringify the byte[]
[
https://issues.apache.org/jira/browse/HBASE-5486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13218484#comment-13218484
]
stack commented on HBASE-5486:
--
Himanshu The patch build failed because you need to use --no-prefix on the git
patches you attach here. Do that the next time. Let me commit this.
Warn message in HTable: Stringify the byte[]
Key: HBASE-5486
URL: https://issues.apache.org/jira/browse/HBASE-5486
Project: HBase
Issue Type: Bug
Components: client
Affects Versions: 0.92.0
Reporter: Himanshu Vashishtha
Assignee: Himanshu Vashishtha
Priority: Trivial
Labels: noob
Attachments: 5486.patch
The warn message in the method getStartEndKeys() in HTable can be improved by
stringifying the byte array for Regions.Qualifier
Currently, a sample message is like :
12/01/17 16:36:34 WARN client.HTable: Null [B@552c8fa8 cell in
keyvalues={test5,\xC9\xA2\x00\x00\x00\x00\x00\x00/00_0,1326642537734.dbc62b2765529a9ad2ddcf8eb58cb2dc./info:server/1326750341579/Put/vlen=28,
test5,\xC9\xA2\x00\x00\x00\x00\x00\x00/00_0,1326642537734.dbc62b2765529a9ad2ddcf8eb58cb2dc./info:serverstartcode/1326750341579/Put/vlen=8}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5486) Warn message in HTable: Stringify the byte[]
[
https://issues.apache.org/jira/browse/HBASE-5486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13218489#comment-13218489
]
stack commented on HBASE-5486:
--
Hmm.. shouldn't this toString be a static itself in HConstants rather than make
it each time? Want to have another go at it Himanshu? Thanks.
Warn message in HTable: Stringify the byte[]
Key: HBASE-5486
URL: https://issues.apache.org/jira/browse/HBASE-5486
Project: HBase
Issue Type: Bug
Components: client
Affects Versions: 0.92.0
Reporter: Himanshu Vashishtha
Assignee: Himanshu Vashishtha
Priority: Trivial
Labels: noob
Attachments: 5486.patch
The warn message in the method getStartEndKeys() in HTable can be improved by
stringifying the byte array for Regions.Qualifier
Currently, a sample message is like :
12/01/17 16:36:34 WARN client.HTable: Null [B@552c8fa8 cell in
keyvalues={test5,\xC9\xA2\x00\x00\x00\x00\x00\x00/00_0,1326642537734.dbc62b2765529a9ad2ddcf8eb58cb2dc./info:server/1326750341579/Put/vlen=28,
test5,\xC9\xA2\x00\x00\x00\x00\x00\x00/00_0,1326642537734.dbc62b2765529a9ad2ddcf8eb58cb2dc./info:serverstartcode/1326750341579/Put/vlen=8}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4907) Port 89-fb changes to trunk
[ https://issues.apache.org/jira/browse/HBASE-4907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-4907: - Priority: Blocker (was: Minor) Making this a blocker on 0.94 (undo it if you think otherwise Lars -- making it blocker so we don't forget it... we can punt later if it doesn't make sense). Port 89-fb changes to trunk --- Key: HBASE-4907 URL: https://issues.apache.org/jira/browse/HBASE-4907 Project: HBase Issue Type: Improvement Components: client, regionserver Reporter: Nicolas Spiegelberg Assignee: Nicolas Spiegelberg Priority: Blocker Fix For: 0.94.0 A super task to track the progress of porting 89-fb functionality fixes to trunk. Note that these tasks are focused on RegionServer functionality. 89-specific master functionality doesn't have a time frame for porting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5484) Spelling mistake in error message in HMasterCommandLine
[ https://issues.apache.org/jira/browse/HBASE-5484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13218493#comment-13218493 ] Hadoop QA commented on HBASE-5484: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12516365/diff1.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -131 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 155 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestImportTsv Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1059//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1059//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1059//console This message is automatically generated. Spelling mistake in error message in HMasterCommandLine --- Key: HBASE-5484 URL: https://issues.apache.org/jira/browse/HBASE-5484 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.90.5, 0.92.0 Reporter: David S. Wang Assignee: David S. Wang Priority: Trivial Labels: newbie, noob Fix For: 0.94.0 Attachments: diff1.txt If hadoop-zookeeper-server is installed and started, starting hbase-master in standalone mode will display this error message which has some typos. $ sudo /etc/init.d/hadoop-hbase-master start Starting Hadoop HBase master daemon: starting master, logging to /usr/lib/hbase/logs/hbase-hbase-master/cloudera-vm.out Couldnt start ZK at requested address of 2181, instead got: 2182. Aborting. Why? Because clients (eg shell) wont be able to find this ZK quorum hbase-master. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4907) Port 89-fb changes to trunk
[ https://issues.apache.org/jira/browse/HBASE-4907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13218503#comment-13218503 ] Nicolas Spiegelberg commented on HBASE-4907: @stack: I'd make it a blocker for 0.96 or 1.0. I'd say the ones to mark blocker are HBASE-3967, HBASE-3776, and maybe HBASE-4914. We're actually pretty far along on this task. A lot of the other jiras are either bug fixes (which we can put in post-release, if necessary) or utilities (which shouldn't destabilize nor prevent the release). Port 89-fb changes to trunk --- Key: HBASE-4907 URL: https://issues.apache.org/jira/browse/HBASE-4907 Project: HBase Issue Type: Improvement Components: client, regionserver Reporter: Nicolas Spiegelberg Assignee: Nicolas Spiegelberg Priority: Blocker Fix For: 0.94.0 A super task to track the progress of porting 89-fb functionality fixes to trunk. Note that these tasks are focused on RegionServer functionality. 89-specific master functionality doesn't have a time frame for porting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4907) Port 89-fb changes to trunk
[ https://issues.apache.org/jira/browse/HBASE-4907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13218505#comment-13218505 ] Nicolas Spiegelberg commented on HBASE-4907: I think something that we should consider as a blocker for 0.96 is explicit region placement data locality checking for the new master. Port 89-fb changes to trunk --- Key: HBASE-4907 URL: https://issues.apache.org/jira/browse/HBASE-4907 Project: HBase Issue Type: Improvement Components: client, regionserver Reporter: Nicolas Spiegelberg Assignee: Nicolas Spiegelberg Priority: Blocker Fix For: 0.94.0 A super task to track the progress of porting 89-fb functionality fixes to trunk. Note that these tasks are focused on RegionServer functionality. 89-specific master functionality doesn't have a time frame for porting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5399) Cut the link between the client and the zookeeper ensemble
[ https://issues.apache.org/jira/browse/HBASE-5399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13218514#comment-13218514 ] stack commented on HBASE-5399: -- On 1., yeah, the close should close the connection -- a client-side thing On 2., not so mad about it. On 3., you obtain the objective it seems but the solution does seem convoluted (more indirection in the client makes it yet more obtuse). Put up a patch I'd say. Lets have a look. SharedMaster is probably not the right name for the Interface? CloseableMaster or MasterConnection and doc that the close applies to the closing of the client connection to master only. Good on you N Cut the link between the client and the zookeeper ensemble -- Key: HBASE-5399 URL: https://issues.apache.org/jira/browse/HBASE-5399 Project: HBase Issue Type: Improvement Components: client Affects Versions: 0.94.0 Environment: all Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 5399_inprogress.patch, 5399_inprogress.v3.patch, 5399_inprogress.v9.patch The link is often considered as an issue, for various reasons. One of them being that there is a limit on the number of connection that ZK can manage. Stack was suggesting as well to remove the link to master from HConnection. There are choices to be made considering the existing API (that we don't want to break). The first patches I will submit on hadoop-qa should not be committed: they are here to show the progress on the direction taken. ZooKeeper is used for: - public getter, to let the client do whatever he wants, and close ZooKeeper when closing the connection = we have to deprecate this but keep it. - read get master address to create a master = now done with a temporary zookeeper connection - read root location = now done with a temporary zookeeper connection, but questionable. Used in public function locateRegion. To be reworked. - read cluster id = now done once with a temporary zookeeper connection. - check if base done is available = now done once with a zookeeper connection given as a parameter - isTableDisabled/isTableAvailable = public functions, now done with a temporary zookeeper connection. - Called internally from HBaseAdmin and HTable - getCurrentNrHRS(): public function to get the number of region servers and create a pool of thread = now done with a temporary zookeeper connection - Master is used for: - getMaster public getter, as for ZooKeeper = we have to deprecate this but keep it. - isMasterRunning(): public function, used internally by HMerge HBaseAdmin - getHTableDescriptor*: public functions offering access to the master. = we could make them using a temporary master connection as well. Main points are: - hbase class for ZooKeeper; ZooKeeperWatcher is really designed for a strongly coupled architecture ;-). This can be changed, but requires a lot of modifications in these classes (likely adding a class in the middle of the hierarchy, something like that). Anyway, non connected client will always be really slower, because it's a tcp connection, and establishing a tcp connection is slow. - having a link between ZK and all the client seems to make sense for some Use Cases. However, it won't scale if a TCP connection is required for every client - if we move the table descriptor part away from the client, we need to find a new place for it. - we will have the same issue if HBaseAdmin (for both ZK Master), may be we can put a timeout on the connection. That would make the whole system less deterministic however. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3967) Support deletes in HFileOutputFormat based bulk import mechanism
[ https://issues.apache.org/jira/browse/HBASE-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-3967: - Priority: Blocker (was: Major) Fix Version/s: 0.96.0 Making blocker on 0.96 at Nicolas's suggestion Support deletes in HFileOutputFormat based bulk import mechanism Key: HBASE-3967 URL: https://issues.apache.org/jira/browse/HBASE-3967 Project: HBase Issue Type: Sub-task Reporter: Kannan Muthukkaruppan Priority: Blocker Fix For: 0.96.0 Attachments: diff.patch During bulk imports, it'll be useful to be able to do delete mutations (either to delete data that already exists in HBase or was inserted earlier during this run of the import). For example, we have a use case, where we are processing a log of data which may have both inserts and deletes in the mix and we want to upload that into HBase using the bulk import mechanism. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3776) Add Bloom Filter Support to HFileOutputFormat
[ https://issues.apache.org/jira/browse/HBASE-3776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-3776: - Priority: Blocker (was: Minor) Fix Version/s: 0.96.0 Making blocker on 0.96 at Nicolas's suggestion Add Bloom Filter Support to HFileOutputFormat - Key: HBASE-3776 URL: https://issues.apache.org/jira/browse/HBASE-3776 Project: HBase Issue Type: Sub-task Reporter: Nicolas Spiegelberg Assignee: Jerry Chen Priority: Blocker Labels: hbase Fix For: 0.96.0 Add Bloom Filter support for bulk imports. Lacking a bloom filter, even on a single imported file, can cause perf degradation. Since we now set our compression type based on the HBase CF configuration, it would be good to follow this path for the bloom filter addition. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3967) Support deletes in HFileOutputFormat based bulk import mechanism
[ https://issues.apache.org/jira/browse/HBASE-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-3967: - Fix Version/s: (was: 0.96.0) 0.94.0 Misread. N said 0.94 blocker. Support deletes in HFileOutputFormat based bulk import mechanism Key: HBASE-3967 URL: https://issues.apache.org/jira/browse/HBASE-3967 Project: HBase Issue Type: Sub-task Reporter: Kannan Muthukkaruppan Priority: Blocker Fix For: 0.94.0 Attachments: diff.patch During bulk imports, it'll be useful to be able to do delete mutations (either to delete data that already exists in HBase or was inserted earlier during this run of the import). For example, we have a use case, where we are processing a log of data which may have both inserts and deletes in the mix and we want to upload that into HBase using the bulk import mechanism. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4907) Port 89-fb changes to trunk
[ https://issues.apache.org/jira/browse/HBASE-4907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-4907: - Fix Version/s: (was: 0.94.0) 0.96.0 Making blocker on 0.96.0 at Nicolas's suggestion. Also made the suggested 0.94.0 blockers (Kick them out Lars if they don't align w/ what you want in 0.94.0). Port 89-fb changes to trunk --- Key: HBASE-4907 URL: https://issues.apache.org/jira/browse/HBASE-4907 Project: HBase Issue Type: Improvement Components: client, regionserver Reporter: Nicolas Spiegelberg Assignee: Nicolas Spiegelberg Priority: Blocker Fix For: 0.96.0 A super task to track the progress of porting 89-fb functionality fixes to trunk. Note that these tasks are focused on RegionServer functionality. 89-specific master functionality doesn't have a time frame for porting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4324) Single unassigned directory is very slow when there are many unassigned nodes
[ https://issues.apache.org/jira/browse/HBASE-4324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13218529#comment-13218529 ] stack commented on HBASE-4324: -- Yeah, should still be an issue. Probably better to have it in 0.96, the singularity, since will necessitate change in layout up in zk. Single unassigned directory is very slow when there are many unassigned nodes - Key: HBASE-4324 URL: https://issues.apache.org/jira/browse/HBASE-4324 Project: HBase Issue Type: Bug Components: zookeeper Affects Versions: 0.90.4 Reporter: Todd Lipcon Fix For: 0.96.0 Because we use a single znode for /unassigned, and we re-list it every time its contents change, assignment speed per region is O(number of unassigned regions) rather than O(1). Every time something changes about one unassigned region, the master has to re-list the entire contents of the directory inside of AssignmentManager.nodeChildrenChanged(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5485) LogCleaner refers to non-existant SnapshotLogCleaner
[ https://issues.apache.org/jira/browse/HBASE-5485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5485: - Resolution: Fixed Fix Version/s: 0.94.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed trunk and 0.94 branch. Thanks for patch Gregory (and review Ted). LogCleaner refers to non-existant SnapshotLogCleaner Key: HBASE-5485 URL: https://issues.apache.org/jira/browse/HBASE-5485 Project: HBase Issue Type: Bug Components: documentation, wal Affects Versions: 0.94.0 Reporter: Gregory Chanan Assignee: Gregory Chanan Priority: Minor Fix For: 0.94.0 Attachments: HBASE-5485.patch, HBASE-5485.patch LogCleaner code refers to SnapshotLogCleaner, but no such class exists. Perhaps it refers to work done in HBASE-50 that was never checked in. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5484) Spelling mistake in error message in HMasterCommandLine
[ https://issues.apache.org/jira/browse/HBASE-5484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13218533#comment-13218533 ] Hudson commented on HBASE-5484: --- Integrated in HBase-0.94 #2 (See [https://builds.apache.org/job/HBase-0.94/2/]) HBASE-5484 Spelling mistake in error message in HMasterCommandLine (Revision 1294789) Result = SUCCESS stack : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/HMasterCommandLine.java Spelling mistake in error message in HMasterCommandLine --- Key: HBASE-5484 URL: https://issues.apache.org/jira/browse/HBASE-5484 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.90.5, 0.92.0 Reporter: David S. Wang Assignee: David S. Wang Priority: Trivial Labels: newbie, noob Fix For: 0.94.0 Attachments: diff1.txt If hadoop-zookeeper-server is installed and started, starting hbase-master in standalone mode will display this error message which has some typos. $ sudo /etc/init.d/hadoop-hbase-master start Starting Hadoop HBase master daemon: starting master, logging to /usr/lib/hbase/logs/hbase-hbase-master/cloudera-vm.out Couldnt start ZK at requested address of 2181, instead got: 2182. Aborting. Why? Because clients (eg shell) wont be able to find this ZK quorum hbase-master. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5074) support checksums in HBase block cache
[ https://issues.apache.org/jira/browse/HBASE-5074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13218546#comment-13218546 ] Zhihong Yu commented on HBASE-5074: --- I first mentioned porting PureJavaCrc32C to HBase here: https://issues.apache.org/jira/browse/HBASE-5074?focusedCommentId=13202490page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13202490 Is that something worth trying ? support checksums in HBase block cache -- Key: HBASE-5074 URL: https://issues.apache.org/jira/browse/HBASE-5074 Project: HBase Issue Type: Improvement Components: regionserver Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: D1521.1.patch, D1521.1.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.10.patch, D1521.2.patch, D1521.2.patch, D1521.3.patch, D1521.3.patch, D1521.4.patch, D1521.4.patch, D1521.5.patch, D1521.5.patch, D1521.6.patch, D1521.6.patch, D1521.7.patch, D1521.7.patch, D1521.8.patch, D1521.8.patch, D1521.9.patch, D1521.9.patch The current implementation of HDFS stores the data in one block file and the metadata(checksum) in another block file. This means that every read into the HBase block cache actually consumes two disk iops, one to the datafile and one to the checksum file. This is a major problem for scaling HBase, because HBase is usually bottlenecked on the number of random disk iops that the storage-hardware offers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-5487) Generic framework for Master-coordinated tasks
Generic framework for Master-coordinated tasks -- Key: HBASE-5487 URL: https://issues.apache.org/jira/browse/HBASE-5487 Project: HBase Issue Type: New Feature Components: master, regionserver, zookeeper Affects Versions: 0.94.0 Reporter: Mubarak Seyed Need a framework to execute master-coordinated tasks in a fault-tolerant manner. Master-coordinated tasks such as online-scheme change and delete-range (deleting region(s) based on start/end key) can make use of this framework. The advantages of framework are 1. Eliminate repeated code in Master, ZooKeeper tracker and Region-server for master-coordinated tasks 2. Ability to abstract the common functions across Master - ZK and RS - ZK 3. Easy to plugin new master-coordinated tasks without adding code to core components -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5485) LogCleaner refers to non-existant SnapshotLogCleaner
[ https://issues.apache.org/jira/browse/HBASE-5485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13218598#comment-13218598 ] Hudson commented on HBASE-5485: --- Integrated in HBase-0.94 #3 (See [https://builds.apache.org/job/HBase-0.94/3/]) HBASE-5485 LogCleaner refers to non-existant SnapshotLogCleaner (Revision 1294800) Result = SUCCESS stack : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/LogCleaner.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/LogCleanerDelegate.java LogCleaner refers to non-existant SnapshotLogCleaner Key: HBASE-5485 URL: https://issues.apache.org/jira/browse/HBASE-5485 Project: HBase Issue Type: Bug Components: documentation, wal Affects Versions: 0.94.0 Reporter: Gregory Chanan Assignee: Gregory Chanan Priority: Minor Fix For: 0.94.0 Attachments: HBASE-5485.patch, HBASE-5485.patch LogCleaner code refers to SnapshotLogCleaner, but no such class exists. Perhaps it refers to work done in HBASE-50 that was never checked in. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5443) Add PB-based calls to HRegionInterface
[ https://issues.apache.org/jira/browse/HBASE-5443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13218601#comment-13218601 ] jirapos...@reviews.apache.org commented on HBASE-5443: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4054/#review5407 --- High-level, do we need to split the Interface more -- into admin vs user protos? Would love to get rid of the plethora of methods but probably not a task to be done as part of this issue? pom.xml https://reviews.apache.org/r/4054/#comment11742 There are prereqs for this to work? I suppose compile-proto.sh checks and its failing it seems fails the build.. thats good. pom.xml https://reviews.apache.org/r/4054/#comment11741 Do we do this elsewhere in the pom? If so, should set a boolean once? src/main/proto/HRegionProtocol.proto https://reviews.apache.org/r/4054/#comment11743 So, all protobuf files go here? We need to ensure this the case w/ all patches (I think the rpc patch was putting them elsewhere.) src/main/proto/HRegionProtocol.proto https://reviews.apache.org/r/4054/#comment11744 Elsewhere in hbase the convention is 'what' and then the generated classes are in a 'generated' subpackage as in there is a thrift subpackage and under there are thrift utils and then a 'generated' subpackage. Ditto avro. This is different in that we have a generated top level subpackage w/ proto as a subpackage. Should we flip it? src/main/proto/HRegionProtocol.proto https://reviews.apache.org/r/4054/#comment11745 Should the suffix be Proto -- our convention for Protobuf classes (?) -- or Protos? Why the plural? Perhaps this a container for a bunch of Proto? src/main/proto/HRegionProtocol.proto https://reviews.apache.org/r/4054/#comment11755 This class seems to have more than HR stuff in it. Should we make more protos? A client proto? src/main/proto/HRegionProtocol.proto https://reviews.apache.org/r/4054/#comment11751 Mutation (Probably already used) src/main/proto/HRegionProtocol.proto https://reviews.apache.org/r/4054/#comment11752 What is this? For filters? src/main/proto/HRegionProtocol.proto https://reviews.apache.org/r/4054/#comment11753 Mutate is not deprecated? We don't have deprecated stuff in this proto file? This is doing what from current API? src/main/proto/HRegionProtocol.proto https://reviews.apache.org/r/4054/#comment11756 Are these client datastructures? Or they go w/ the RS proto and are used by clients? src/main/proto/HRegionProtocol.proto https://reviews.apache.org/r/4054/#comment11754 Where does this come from in current API? src/main/proto/HRegionProtocol.proto https://reviews.apache.org/r/4054/#comment11758 Doesn't HRI have a tablename in it? So maybe this should have a HRI? What is this LogKeyProto modeling? Should be WALKeyProto? src/main/proto/HRegionProtocol.proto https://reviews.apache.org/r/4054/#comment11759 WALEntryProto? src/main/proto/HRegionProtocol.proto https://reviews.apache.org/r/4054/#comment11760 Should there be more in here? src/main/proto/HRegionProtocol.proto https://reviews.apache.org/r/4054/#comment11761 Is this a class or method name? If method name is convention to capitalize? src/main/proto/HRegionProtocol.proto https://reviews.apache.org/r/4054/#comment11762 So, even if method returns a HRegionInfo, we return a GetRegionInfoResponseProto in case the response changes? src/main/proto/HRegionProtocol.proto https://reviews.apache.org/r/4054/#comment11763 regionName and encodedName and HRegionInfo class should we standardize? src/main/proto/HRegionProtocol.proto https://reviews.apache.org/r/4054/#comment11764 Excellent. Later we need to return first set of results in here rather than have to have client go back to do a next. src/main/proto/HRegionProtocol.proto https://reviews.apache.org/r/4054/#comment11765 Maybe NextOnScannerRequestProto so relates better to our current API src/main/proto/HRegionProtocol.proto https://reviews.apache.org/r/4054/#comment11766 We should make this optional some day.. that you have to call a close. src/main/proto/HRegionProtocol.proto https://reviews.apache.org/r/4054/#comment11767 Man. Anyone use this thing? src/main/proto/HRegionProtocol.proto https://reviews.apache.org/r/4054/#comment11768 We should have one or the other. Can have one call through to the other (thats implementation?) src/main/proto/HRegionProtocol.proto https://reviews.apache.org/r/4054/#comment11769 Oh, I see how method naming and classes goes src/main/proto/HRegionProtocol.proto https://reviews.apache.org/r/4054/#comment11770 I
[jira] [Commented] (HBASE-5443) Add PB-based calls to HRegionInterface
[ https://issues.apache.org/jira/browse/HBASE-5443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13218700#comment-13218700 ] jirapos...@reviews.apache.org commented on HBASE-5443: -- bq. On 2012-02-28 20:44:26, Michael Stack wrote: bq. src/main/proto/HRegionProtocol.proto, line 118 bq. https://reviews.apache.org/r/4054/diff/1/?file=86003#file86003line118 bq. bq. Where does this come from in current API? bq. bq. Jimmy Xiang wrote: bq. Yes. It is from the current API. It seems nobody is using it. I will get rid of it. - Jimmy --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4054/#review5407 --- On 2012-02-27 18:54:31, Jimmy Xiang wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4054/ bq. --- bq. bq. (Updated 2012-02-27 18:54:31) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. This is the first draft of the ProtoBuff HRegionProtocol. The corresponding java vs pb method mapping is attached to the jira: https://issues.apache.org/jira/browse/HBASE-5443 bq. bq. Please review. I'd like to move ahead after we get to some agreement. bq. bq. bq. This addresses bug HBASE-5443. bq. https://issues.apache.org/jira/browse/HBASE-5443 bq. bq. bq. Diffs bq. - bq. bq.pom.xml 066c027 bq.src/main/proto/HRegionProtocol.proto PRE-CREATION bq.src/main/proto/hbase.proto PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/4054/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Jimmy bq. bq. Add PB-based calls to HRegionInterface -- Key: HBASE-5443 URL: https://issues.apache.org/jira/browse/HBASE-5443 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Todd Lipcon Assignee: Jimmy Xiang Fix For: 0.96.0 Attachments: region_java-proto-mapping.pdf -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5443) Add PB-based calls to HRegionInterface
[ https://issues.apache.org/jira/browse/HBASE-5443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13218698#comment-13218698 ] jirapos...@reviews.apache.org commented on HBASE-5443: -- bq. On 2012-02-28 23:28:19, Shaneal Manek wrote: bq. pom.xml, line 761 bq. https://reviews.apache.org/r/4054/diff/1/?file=86002#file86002line761 bq. bq. It's not too bad - a bunch of hadoop-common stuff already does. (e.g. https://github.com/apache/hadoop-common/blob/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/pom.xml) You are right. I copied over from there. - Jimmy --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4054/#review5421 --- On 2012-02-27 18:54:31, Jimmy Xiang wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4054/ bq. --- bq. bq. (Updated 2012-02-27 18:54:31) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. This is the first draft of the ProtoBuff HRegionProtocol. The corresponding java vs pb method mapping is attached to the jira: https://issues.apache.org/jira/browse/HBASE-5443 bq. bq. Please review. I'd like to move ahead after we get to some agreement. bq. bq. bq. This addresses bug HBASE-5443. bq. https://issues.apache.org/jira/browse/HBASE-5443 bq. bq. bq. Diffs bq. - bq. bq.pom.xml 066c027 bq.src/main/proto/HRegionProtocol.proto PRE-CREATION bq.src/main/proto/hbase.proto PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/4054/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Jimmy bq. bq. Add PB-based calls to HRegionInterface -- Key: HBASE-5443 URL: https://issues.apache.org/jira/browse/HBASE-5443 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Todd Lipcon Assignee: Jimmy Xiang Fix For: 0.96.0 Attachments: region_java-proto-mapping.pdf -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5443) Add PB-based calls to HRegionInterface
[ https://issues.apache.org/jira/browse/HBASE-5443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13218699#comment-13218699 ] jirapos...@reviews.apache.org commented on HBASE-5443: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4054/#review5416 --- pom.xml https://reviews.apache.org/r/4054/#comment11806 does it make sense to assume protoc is in path? src/main/proto/HRegionProtocol.proto https://reviews.apache.org/r/4054/#comment11807 HRI does have the tablename. Correct. src/main/proto/HRegionProtocol.proto https://reviews.apache.org/r/4054/#comment11808 if the region is foo,bar,1234.4567. Then the encoded name is 4567? no? src/main/proto/HRegionProtocol.proto https://reviews.apache.org/r/4054/#comment11809 Why would you give a regionname, as opposed to a tablename? - Alex On 2012-02-27 18:54:31, Jimmy Xiang wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4054/ bq. --- bq. bq. (Updated 2012-02-27 18:54:31) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. This is the first draft of the ProtoBuff HRegionProtocol. The corresponding java vs pb method mapping is attached to the jira: https://issues.apache.org/jira/browse/HBASE-5443 bq. bq. Please review. I'd like to move ahead after we get to some agreement. bq. bq. bq. This addresses bug HBASE-5443. bq. https://issues.apache.org/jira/browse/HBASE-5443 bq. bq. bq. Diffs bq. - bq. bq.pom.xml 066c027 bq.src/main/proto/HRegionProtocol.proto PRE-CREATION bq.src/main/proto/hbase.proto PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/4054/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Jimmy bq. bq. Add PB-based calls to HRegionInterface -- Key: HBASE-5443 URL: https://issues.apache.org/jira/browse/HBASE-5443 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Todd Lipcon Assignee: Jimmy Xiang Fix For: 0.96.0 Attachments: region_java-proto-mapping.pdf -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5443) Add PB-based calls to HRegionInterface
[ https://issues.apache.org/jira/browse/HBASE-5443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13218701#comment-13218701 ] jirapos...@reviews.apache.org commented on HBASE-5443: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4054/#review5421 --- pom.xml https://reviews.apache.org/r/4054/#comment11821 It's not too bad - a bunch of hadoop-common stuff already does. (e.g. https://github.com/apache/hadoop-common/blob/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/pom.xml) - Shaneal On 2012-02-27 18:54:31, Jimmy Xiang wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4054/ bq. --- bq. bq. (Updated 2012-02-27 18:54:31) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. This is the first draft of the ProtoBuff HRegionProtocol. The corresponding java vs pb method mapping is attached to the jira: https://issues.apache.org/jira/browse/HBASE-5443 bq. bq. Please review. I'd like to move ahead after we get to some agreement. bq. bq. bq. This addresses bug HBASE-5443. bq. https://issues.apache.org/jira/browse/HBASE-5443 bq. bq. bq. Diffs bq. - bq. bq.pom.xml 066c027 bq.src/main/proto/HRegionProtocol.proto PRE-CREATION bq.src/main/proto/hbase.proto PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/4054/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Jimmy bq. bq. Add PB-based calls to HRegionInterface -- Key: HBASE-5443 URL: https://issues.apache.org/jira/browse/HBASE-5443 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Todd Lipcon Assignee: Jimmy Xiang Fix For: 0.96.0 Attachments: region_java-proto-mapping.pdf -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5443) Add PB-based calls to HRegionInterface
[ https://issues.apache.org/jira/browse/HBASE-5443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13218706#comment-13218706 ] jirapos...@reviews.apache.org commented on HBASE-5443: -- bq. On 2012-02-28 20:44:26, Michael Stack wrote: bq. High-level, do we need to split the Interface more -- into admin vs user protos? Would love to get rid of the plethora of methods but probably not a task to be done as part of this issue? Yes. Todd mentioned that too. Will do. bq. On 2012-02-28 20:44:26, Michael Stack wrote: bq. src/main/proto/hbase.proto, line 36 bq. https://reviews.apache.org/r/4054/diff/1/?file=86004#file86004line36 bq. bq. My guess is that there is no uint32? There is uint32. Should we use uint32 for timestamp? If it is in second, it is ok for many years. If it is in millisecond, we should use uint64. bq. On 2012-02-28 20:44:26, Michael Stack wrote: bq. src/main/proto/hbase.proto, line 42 bq. https://reviews.apache.org/r/4054/diff/1/?file=86004#file86004line42 bq. bq. Whats a 'name'? Is it a region name? It is for a generic NameValue pair. bq. On 2012-02-28 20:44:26, Michael Stack wrote: bq. pom.xml, line 749 bq. https://reviews.apache.org/r/4054/diff/1/?file=86002#file86002line749 bq. bq. Do we do this elsewhere in the pom? If so, should set a boolean once? This is the only place. bq. On 2012-02-28 20:44:26, Michael Stack wrote: bq. src/main/proto/HRegionProtocol.proto, line 1 bq. https://reviews.apache.org/r/4054/diff/1/?file=86003#file86003line1 bq. bq. So, all protobuf files go here? We need to ensure this the case w/ all patches (I think the rpc patch was putting them elsewhere.) All proto files should go here, while Java files go to other places like other Java classes. bq. On 2012-02-28 20:44:26, Michael Stack wrote: bq. src/main/proto/HRegionProtocol.proto, line 21 bq. https://reviews.apache.org/r/4054/diff/1/?file=86003#file86003line21 bq. bq. Elsewhere in hbase the convention is 'what' and then the generated classes are in a 'generated' subpackage as in there is a thrift subpackage and under there are thrift utils and then a 'generated' subpackage. Ditto avro. This is different in that we have a generated top level subpackage w/ proto as a subpackage. Should we flip it? Sure, will do. bq. On 2012-02-28 20:44:26, Michael Stack wrote: bq. src/main/proto/HRegionProtocol.proto, line 22 bq. https://reviews.apache.org/r/4054/diff/1/?file=86003#file86003line22 bq. bq. Should the suffix be Proto -- our convention for Protobuf classes (?) -- or Protos? Why the plural? Perhaps this a container for a bunch of Proto? Yes, there are a bunch of messages. Each message is a Proto. bq. On 2012-02-28 20:44:26, Michael Stack wrote: bq. src/main/proto/HRegionProtocol.proto, line 23 bq. https://reviews.apache.org/r/4054/diff/1/?file=86003#file86003line23 bq. bq. This class seems to have more than HR stuff in it. Should we make more protos? A client proto? Ok. bq. On 2012-02-28 20:44:26, Michael Stack wrote: bq. src/main/proto/HRegionProtocol.proto, line 62 bq. https://reviews.apache.org/r/4054/diff/1/?file=86003#file86003line62 bq. bq. What is this? For filters? For checkAndPut, checkAndDelete, this is condition to check. bq. On 2012-02-28 20:44:26, Michael Stack wrote: bq. src/main/proto/HRegionProtocol.proto, line 88 bq. https://reviews.apache.org/r/4054/diff/1/?file=86003#file86003line88 bq. bq. Mutate is not deprecated? We don't have deprecated stuff in this proto file? This is doing what from current API? This Mutate is different from the mutateRow() in HRegionInterface. It is for append() and increment(). Any suggestion for a better naming? bq. On 2012-02-28 20:44:26, Michael Stack wrote: bq. src/main/proto/HRegionProtocol.proto, line 105 bq. https://reviews.apache.org/r/4054/diff/1/?file=86003#file86003line105 bq. bq. Are these client datastructures? Or they go w/ the RS proto and are used by clients? For now, I prefer not to change the client data structures. So these go w/ the RS proto only and are not used by clients directly. bq. On 2012-02-28 20:44:26, Michael Stack wrote: bq. src/main/proto/HRegionProtocol.proto, line 118 bq. https://reviews.apache.org/r/4054/diff/1/?file=86003#file86003line118 bq. bq. Where does this come from in current API? Yes. bq. On 2012-02-28 20:44:26, Michael Stack wrote: bq. src/main/proto/HRegionProtocol.proto, line 148 bq. https://reviews.apache.org/r/4054/diff/1/?file=86003#file86003line148 bq. bq. WALEntryProto? Ok. bq. On 2012-02-28 20:44:26, Michael Stack wrote: bq. src/main/proto/HRegionProtocol.proto, line 126 bq. https://reviews.apache.org/r/4054/diff/1/?file=86003#file86003line126 bq. bq. Doesn't HRI have a
[jira] [Commented] (HBASE-5443) Add PB-based calls to HRegionInterface
[ https://issues.apache.org/jira/browse/HBASE-5443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13218714#comment-13218714 ] jirapos...@reviews.apache.org commented on HBASE-5443: -- bq. On 2012-02-28 23:08:15, Alex Newman wrote: bq. pom.xml, line 761 bq. https://reviews.apache.org/r/4054/diff/1/?file=86002#file86002line761 bq. bq. does it make sense to assume protoc is in path? Hadoop requires that. I think HBase can do the same thing too. bq. On 2012-02-28 23:08:15, Alex Newman wrote: bq. src/main/proto/HRegionProtocol.proto, line 265 bq. https://reviews.apache.org/r/4054/diff/1/?file=86003#file86003line265 bq. bq. Why would you give a regionname, as opposed to a tablename? The current HRegionInterface uses regionName. It is to lock a row in a region. Otherwise, we need to find which region is the row in. bq. On 2012-02-28 23:08:15, Alex Newman wrote: bq. src/main/proto/HRegionProtocol.proto, line 174 bq. https://reviews.apache.org/r/4054/diff/1/?file=86003#file86003line174 bq. bq. if the region is bq. bq. foo,bar,1234.4567. bq. bq. Then the encoded name is 4567? no? That's right. The encoded name is the hash. - Jimmy --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4054/#review5416 --- On 2012-02-27 18:54:31, Jimmy Xiang wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4054/ bq. --- bq. bq. (Updated 2012-02-27 18:54:31) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. This is the first draft of the ProtoBuff HRegionProtocol. The corresponding java vs pb method mapping is attached to the jira: https://issues.apache.org/jira/browse/HBASE-5443 bq. bq. Please review. I'd like to move ahead after we get to some agreement. bq. bq. bq. This addresses bug HBASE-5443. bq. https://issues.apache.org/jira/browse/HBASE-5443 bq. bq. bq. Diffs bq. - bq. bq.pom.xml 066c027 bq.src/main/proto/HRegionProtocol.proto PRE-CREATION bq.src/main/proto/hbase.proto PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/4054/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Jimmy bq. bq. Add PB-based calls to HRegionInterface -- Key: HBASE-5443 URL: https://issues.apache.org/jira/browse/HBASE-5443 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Todd Lipcon Assignee: Jimmy Xiang Fix For: 0.96.0 Attachments: region_java-proto-mapping.pdf -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5451) Switch RPC call envelope/headers to PBs
[ https://issues.apache.org/jira/browse/HBASE-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13218721#comment-13218721 ] jirapos...@reviews.apache.org commented on HBASE-5451: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4096/ --- Review request for hbase. Summary --- Switch RPC call envelope/headers to PBs This addresses bug HBASE-5451. https://issues.apache.org/jira/browse/HBASE-5451 Diffs - http://svn.apache.org/repos/asf/hbase/trunk/pom.xml 1293032 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/DataOutputOutputStream.java 1293032 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 1293032 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 1293032 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/User.java 1293032 http://svn.apache.org/repos/asf/hbase/trunk/src/main/proto/RPCMessageProto.proto PRE-CREATION Diff: https://reviews.apache.org/r/4096/diff Testing --- Thanks, Devaraj Switch RPC call envelope/headers to PBs --- Key: HBASE-5451 URL: https://issues.apache.org/jira/browse/HBASE-5451 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Todd Lipcon Assignee: Devaraj Das Attachments: rpc-proto.patch.1_2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5443) Add PB-based calls to HRegionInterface
[ https://issues.apache.org/jira/browse/HBASE-5443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13218726#comment-13218726 ] jirapos...@reviews.apache.org commented on HBASE-5443: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4054/#review5415 --- src/main/proto/HRegionProtocol.proto https://reviews.apache.org/r/4054/#comment11804 Why is this called 'ColumnFamilyMapProto' (as opposed to just, say, 'ColumnProto' or 'ColumnNameProto')? src/main/proto/HRegionProtocol.proto https://reviews.apache.org/r/4054/#comment11805 What is this 'value' field? What is its serialization format? - Shaneal On 2012-02-27 18:54:31, Jimmy Xiang wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4054/ bq. --- bq. bq. (Updated 2012-02-27 18:54:31) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. This is the first draft of the ProtoBuff HRegionProtocol. The corresponding java vs pb method mapping is attached to the jira: https://issues.apache.org/jira/browse/HBASE-5443 bq. bq. Please review. I'd like to move ahead after we get to some agreement. bq. bq. bq. This addresses bug HBASE-5443. bq. https://issues.apache.org/jira/browse/HBASE-5443 bq. bq. bq. Diffs bq. - bq. bq.pom.xml 066c027 bq.src/main/proto/HRegionProtocol.proto PRE-CREATION bq.src/main/proto/hbase.proto PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/4054/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Jimmy bq. bq. Add PB-based calls to HRegionInterface -- Key: HBASE-5443 URL: https://issues.apache.org/jira/browse/HBASE-5443 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Todd Lipcon Assignee: Jimmy Xiang Fix For: 0.96.0 Attachments: region_java-proto-mapping.pdf -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5443) Add PB-based calls to HRegionInterface
[ https://issues.apache.org/jira/browse/HBASE-5443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13218738#comment-13218738 ] jirapos...@reviews.apache.org commented on HBASE-5443: -- bq. On 2012-02-28 22:59:39, Shaneal Manek wrote: bq. src/main/proto/HRegionProtocol.proto, line 84 bq. https://reviews.apache.org/r/4054/diff/1/?file=86003#file86003line84 bq. bq. What is this 'value' field? What is its serialization format? This value field is the value used to compare. It is a plain byte array. bq. On 2012-02-28 22:59:39, Shaneal Manek wrote: bq. src/main/proto/HRegionProtocol.proto, line 28 bq. https://reviews.apache.org/r/4054/diff/1/?file=86003#file86003line28 bq. bq. Why is this called 'ColumnFamilyMapProto' (as opposed to just, say, 'ColumnProto' or 'ColumnNameProto')? It is a column family and a list of its columns. ColumnListProto, ColumnsProto? - Jimmy --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4054/#review5415 --- On 2012-02-27 18:54:31, Jimmy Xiang wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4054/ bq. --- bq. bq. (Updated 2012-02-27 18:54:31) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. This is the first draft of the ProtoBuff HRegionProtocol. The corresponding java vs pb method mapping is attached to the jira: https://issues.apache.org/jira/browse/HBASE-5443 bq. bq. Please review. I'd like to move ahead after we get to some agreement. bq. bq. bq. This addresses bug HBASE-5443. bq. https://issues.apache.org/jira/browse/HBASE-5443 bq. bq. bq. Diffs bq. - bq. bq.pom.xml 066c027 bq.src/main/proto/HRegionProtocol.proto PRE-CREATION bq.src/main/proto/hbase.proto PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/4054/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Jimmy bq. bq. Add PB-based calls to HRegionInterface -- Key: HBASE-5443 URL: https://issues.apache.org/jira/browse/HBASE-5443 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Todd Lipcon Assignee: Jimmy Xiang Fix For: 0.96.0 Attachments: region_java-proto-mapping.pdf -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5445) Add PB-based calls to HMasterInterface
[ https://issues.apache.org/jira/browse/HBASE-5445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan updated HBASE-5445: -- Fix Version/s: 0.96.0 Add PB-based calls to HMasterInterface -- Key: HBASE-5445 URL: https://issues.apache.org/jira/browse/HBASE-5445 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Todd Lipcon Assignee: Gregory Chanan Fix For: 0.96.0 Attachments: HMasterProtocol.proto, hbase.proto -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5399) Cut the link between the client and the zookeeper ensemble
[ https://issues.apache.org/jira/browse/HBASE-5399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-5399: --- Attachment: 5399_inprogress.v14.patch Yes, that's why I rejected it initially. But I can't find a better one. Moreover, I can't share the code with ZooKeeperWatcher, but they can have the same interface. I can simplify the internal code, but the indirection will remain. On Tue, Feb 28, 2012 at 8:27 PM, stack (Commented) (JIRA) Cut the link between the client and the zookeeper ensemble -- Key: HBASE-5399 URL: https://issues.apache.org/jira/browse/HBASE-5399 Project: HBase Issue Type: Improvement Components: client Affects Versions: 0.94.0 Environment: all Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 5399_inprogress.patch, 5399_inprogress.v14.patch, 5399_inprogress.v3.patch, 5399_inprogress.v9.patch The link is often considered as an issue, for various reasons. One of them being that there is a limit on the number of connection that ZK can manage. Stack was suggesting as well to remove the link to master from HConnection. There are choices to be made considering the existing API (that we don't want to break). The first patches I will submit on hadoop-qa should not be committed: they are here to show the progress on the direction taken. ZooKeeper is used for: - public getter, to let the client do whatever he wants, and close ZooKeeper when closing the connection = we have to deprecate this but keep it. - read get master address to create a master = now done with a temporary zookeeper connection - read root location = now done with a temporary zookeeper connection, but questionable. Used in public function locateRegion. To be reworked. - read cluster id = now done once with a temporary zookeeper connection. - check if base done is available = now done once with a zookeeper connection given as a parameter - isTableDisabled/isTableAvailable = public functions, now done with a temporary zookeeper connection. - Called internally from HBaseAdmin and HTable - getCurrentNrHRS(): public function to get the number of region servers and create a pool of thread = now done with a temporary zookeeper connection - Master is used for: - getMaster public getter, as for ZooKeeper = we have to deprecate this but keep it. - isMasterRunning(): public function, used internally by HMerge HBaseAdmin - getHTableDescriptor*: public functions offering access to the master. = we could make them using a temporary master connection as well. Main points are: - hbase class for ZooKeeper; ZooKeeperWatcher is really designed for a strongly coupled architecture ;-). This can be changed, but requires a lot of modifications in these classes (likely adding a class in the middle of the hierarchy, something like that). Anyway, non connected client will always be really slower, because it's a tcp connection, and establishing a tcp connection is slow. - having a link between ZK and all the client seems to make sense for some Use Cases. However, it won't scale if a TCP connection is required for every client - if we move the table descriptor part away from the client, we need to find a new place for it. - we will have the same issue if HBaseAdmin (for both ZK Master), may be we can put a timeout on the connection. That would make the whole system less deterministic however. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5487) Generic framework for Master-coordinated tasks
[
https://issues.apache.org/jira/browse/HBASE-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13218785#comment-13218785
]
stack commented on HBASE-5487:
--
I took a look at FATE over in accumulo. Its some nice generic primitives for
running a suite of idempotent operations (even if operation only part
completes, if its run again, it should clean up and continue). There is a
notion of locking on a table (so can stop it transiting I suppose; there are
read/write locks), a stack for operations (ops are pushed and popped off the
stack), operations can respond done, failed, or even w/ a new set of operations
to do first (This basic can be used to step through a number of tasks one after
the other). All is persisted up in zk run by the master; if master dies, a new
master can pick up the half-done task and finish it. Clients can watch zk to
see if task is done. There ain't too much to the fate package; there is fate
class itself, an admin, a 'store' interface of which there is a zk
implementation. We should for sure take inspiration at least from the work
already done.
Here are the ops they do via fate:
{code}
fate.seedTransaction(opid, new TraceRepoMaster(new
CreateTable(c.user, tableName, timeType, options)), autoCleanup);
fate.seedTransaction(opid, new TraceRepoMaster(new
RenameTable(tableId, oldTableName, newTableName)), autoCleanup);
fate.seedTransaction(opid, new TraceRepoMaster(new
CloneTable(c.user, srcTableId, tableName, propertiesToSet,
propertiesToExclude)), autoCleanup);
fate.seedTransaction(opid, new TraceRepoMaster(new
DeleteTable(tableId)), autoCleanup);
fate.seedTransaction(opid, new TraceRepoMaster(new
ChangeTableState(tableId, TableOperation.ONLINE)), autoCleanup);
fate.seedTransaction(opid, new TraceRepoMaster(new
ChangeTableState(tableId, TableOperation.OFFLINE)), autoCleanup);
fate.seedTransaction(opid, new TraceRepoMaster(new
TableRangeOp(MergeInfo.Operation.MERGE, tableId, startRow, endRow)),
autoCleanup);
fate.seedTransaction(opid, new TraceRepoMaster(new
TableRangeOp(MergeInfo.Operation.DELETE, tableId, startRow, endRow)),
autoCleanup);
fate.seedTransaction(opid, new TraceRepoMaster(new
BulkImport(tableId, dir, failDir, setTime)), autoCleanup);
fate.seedTransaction(opid, new TraceRepoMaster(new
CompactRange(tableId, startRow, endRow)), autoCleanup);{code}
{code}
CompactRange is their term for merge. It takes a key range span, figures the
tablets involved and runs the compact/merge. We want that and then something
to do the remove or regions too?
Generic framework for Master-coordinated tasks
--
Key: HBASE-5487
URL: https://issues.apache.org/jira/browse/HBASE-5487
Project: HBase
Issue Type: New Feature
Components: master, regionserver, zookeeper
Affects Versions: 0.94.0
Reporter: Mubarak Seyed
Labels: noob
Need a framework to execute master-coordinated tasks in a fault-tolerant
manner.
Master-coordinated tasks such as online-scheme change and delete-range
(deleting region(s) based on start/end key) can make use of this framework.
The advantages of framework are
1. Eliminate repeated code in Master, ZooKeeper tracker and Region-server for
master-coordinated tasks
2. Ability to abstract the common functions across Master - ZK and RS - ZK
3. Easy to plugin new master-coordinated tasks without adding code to core
components
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5270) Handle potential data loss due to concurrent processing of processFaileOver and ServerShutdownHandler
[ https://issues.apache.org/jira/browse/HBASE-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13218820#comment-13218820 ] chunhui shen commented on HBASE-5270: - @Prakash In a live cluster, do the following step 1.kill the master; 1.start the master, and master is initializing； 3.master complete splitLog 4.kill the META server 5.master start assigning ROOT and META 6.Now meta region data will loss， because we can't ensure META server's log is split caused by step 4. Handle potential data loss due to concurrent processing of processFaileOver and ServerShutdownHandler - Key: HBASE-5270 URL: https://issues.apache.org/jira/browse/HBASE-5270 Project: HBase Issue Type: Sub-task Components: master Reporter: Zhihong Yu Assignee: chunhui shen Fix For: 0.92.1, 0.94.0 Attachments: 5270-90-testcase.patch, 5270-90-testcasev2.patch, 5270-90.patch, 5270-90v2.patch, 5270-90v3.patch, 5270-testcase.patch, 5270-testcasev2.patch, hbase-5270.patch, hbase-5270v2.patch, hbase-5270v4.patch, hbase-5270v5.patch, hbase-5270v6.patch, hbase-5270v7.patch, hbase-5270v8.patch, sampletest.txt This JIRA continues the effort from HBASE-5179. Starting with Stack's comments about patches for 0.92 and TRUNK: Reviewing 0.92v17 isDeadServerInProgress is a new public method in ServerManager but it does not seem to be used anywhere. Does isDeadRootServerInProgress need to be public? Ditto for meta version. This method param names are not right 'definitiveRootServer'; what is meant by definitive? Do they need this qualifier? Is there anything in place to stop us expiring a server twice if its carrying root and meta? What is difference between asking assignment manager isCarryingRoot and this variable that is passed in? Should be doc'd at least. Ditto for meta. I think I've asked for this a few times - onlineServers needs to be explained... either in javadoc or in comment. This is the param passed into joinCluster. How does it arise? I think I know but am unsure. God love the poor noob that comes awandering this code trying to make sense of it all. It looks like we get the list by trawling zk for regionserver znodes that have not checked in. Don't we do this operation earlier in master setup? Are we doing it again here? Though distributed split log is configured, we will do in master single process splitting under some conditions with this patch. Its not explained in code why we would do this. Why do we think master log splitting 'high priority' when it could very well be slower. Should we only go this route if distributed splitting is not going on. Do we know if concurrent distributed log splitting and master splitting works? Why would we have dead servers in progress here in master startup? Because a servershutdownhandler fired? This patch is different to the patch for 0.90. Should go into trunk first with tests, then 0.92. Should it be in this issue? This issue is really hard to follow now. Maybe this issue is for 0.90.x and new issue for more work on this trunk patch? This patch needs to have the v18 differences applied. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-5488) Fixed OfflineMetaRepair bug
Fixed OfflineMetaRepair bug Key: HBASE-5488 URL: https://issues.apache.org/jira/browse/HBASE-5488 Project: HBase Issue Type: Bug Affects Versions: 0.90.6 Reporter: gaojinchao Priority: Minor Fix For: 0.90.7 I want to use OfflineMetaRepair tools and found onbody fix this bugs. I will make a patch. 12/01/05 23:23:30 ERROR util.HBaseFsck: Bailed out due to: java.lang.IllegalArgumentException: Wrong FS: hdfs:// us01-ciqps1-name01.carrieriq.com:9000/hbase/M2M-INTEGRATION-MM_TION-13 25190318714/0003d2ede27668737e192d8430dbe5d0/.regioninfo, expected: file:/// at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:352) at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:47) at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:368) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251) at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.init(ChecksumFileSystem.java:126) at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:284) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:398) at org.apache.hadoop.hbase.util.HBaseFsck.loadMetaEntry(HBaseFsck.java:256) at org.apache.hadoop.hbase.util.HBaseFsck.loadTableInfo(HBaseFsck.java:284) at org.apache.hadoop.hbase.util.HBaseFsck.rebuildMeta(HBaseFsck.java:402) at org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair.main(OfflineMetaRe -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-5488) Fixed OfflineMetaRepair bug
[ https://issues.apache.org/jira/browse/HBASE-5488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] gaojinchao reassigned HBASE-5488: - Assignee: gaojinchao Fixed OfflineMetaRepair bug Key: HBASE-5488 URL: https://issues.apache.org/jira/browse/HBASE-5488 Project: HBase Issue Type: Bug Affects Versions: 0.90.6 Reporter: gaojinchao Assignee: gaojinchao Priority: Minor Fix For: 0.90.7 I want to use OfflineMetaRepair tools and found onbody fix this bugs. I will make a patch. 12/01/05 23:23:30 ERROR util.HBaseFsck: Bailed out due to: java.lang.IllegalArgumentException: Wrong FS: hdfs:// us01-ciqps1-name01.carrieriq.com:9000/hbase/M2M-INTEGRATION-MM_TION-13 25190318714/0003d2ede27668737e192d8430dbe5d0/.regioninfo, expected: file:/// at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:352) at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:47) at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:368) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251) at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.init(ChecksumFileSystem.java:126) at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:284) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:398) at org.apache.hadoop.hbase.util.HBaseFsck.loadMetaEntry(HBaseFsck.java:256) at org.apache.hadoop.hbase.util.HBaseFsck.loadTableInfo(HBaseFsck.java:284) at org.apache.hadoop.hbase.util.HBaseFsck.rebuildMeta(HBaseFsck.java:402) at org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair.main(OfflineMetaRe -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5488) Fixed OfflineMetaRepair bug
[ https://issues.apache.org/jira/browse/HBASE-5488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] gaojinchao updated HBASE-5488: -- Attachment: HBASE-5488_branch90.txt Fixed OfflineMetaRepair bug Key: HBASE-5488 URL: https://issues.apache.org/jira/browse/HBASE-5488 Project: HBase Issue Type: Bug Affects Versions: 0.90.6 Reporter: gaojinchao Assignee: gaojinchao Priority: Minor Fix For: 0.90.7 Attachments: HBASE-5488_branch90.txt I want to use OfflineMetaRepair tools and found onbody fix this bugs. I will make a patch. 12/01/05 23:23:30 ERROR util.HBaseFsck: Bailed out due to: java.lang.IllegalArgumentException: Wrong FS: hdfs:// us01-ciqps1-name01.carrieriq.com:9000/hbase/M2M-INTEGRATION-MM_TION-13 25190318714/0003d2ede27668737e192d8430dbe5d0/.regioninfo, expected: file:/// at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:352) at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:47) at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:368) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251) at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.init(ChecksumFileSystem.java:126) at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:284) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:398) at org.apache.hadoop.hbase.util.HBaseFsck.loadMetaEntry(HBaseFsck.java:256) at org.apache.hadoop.hbase.util.HBaseFsck.loadTableInfo(HBaseFsck.java:284) at org.apache.hadoop.hbase.util.HBaseFsck.rebuildMeta(HBaseFsck.java:402) at org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair.main(OfflineMetaRe -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-5489) Add HTable accessor to get regions for a key range
Add HTable accessor to get regions for a key range -- Key: HBASE-5489 URL: https://issues.apache.org/jira/browse/HBASE-5489 Project: HBase Issue Type: Improvement Components: client Reporter: David S. Wang Assignee: David S. Wang Priority: Minor It would be nice to have an accessor to find all regions that overlap with a particular range of keys. Right now, the only way to accomplish that is to call HTable.getStartEndKeys(), then follow that with calls to getRegionLocation() for the range of keys you are interested in. This algorithm has 2 drawbacks: * It returns more keys than is necessary most of the time. This is especially evident if there are a lot of regions comprising the table and the range of keys is small. * It always does a scan of .META. via MetaScannerVisitor for at least HTable.getStartEndKeys(), and perhaps for HRegionLocations that are not already cached by the client. An accessor that limited its scans to a specified range could avoid scanning .META. at all if the HRegionLocations being fetched were already cached by the client, thereby potentially making this operation faster in common cases. Here's a proposal for the accessor: /** * Get the corresponding regions for an arbitrary range of keys. * p * @param startRow Starting row in range, inclusive * @param endRow Ending row in range, inclusive * @return A list of HRegionLocations corresponding to the regions that * contain the specified range * @throws IOException if a remote or network exception occurs */ public ListHRegionLocation getRegionsInRange(final byte [] startKey, final byte [] endKey) throws IOException -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5485) LogCleaner refers to non-existant SnapshotLogCleaner
[ https://issues.apache.org/jira/browse/HBASE-5485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13218933#comment-13218933 ] Hudson commented on HBASE-5485: --- Integrated in HBase-TRUNK-security #124 (See [https://builds.apache.org/job/HBase-TRUNK-security/124/]) HBASE-5485 LogCleaner refers to non-existant SnapshotLogCleaner (Revision 1294799) Result = SUCCESS stack : Files : * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/LogCleaner.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/LogCleanerDelegate.java LogCleaner refers to non-existant SnapshotLogCleaner Key: HBASE-5485 URL: https://issues.apache.org/jira/browse/HBASE-5485 Project: HBase Issue Type: Bug Components: documentation, wal Affects Versions: 0.94.0 Reporter: Gregory Chanan Assignee: Gregory Chanan Priority: Minor Fix For: 0.94.0 Attachments: HBASE-5485.patch, HBASE-5485.patch LogCleaner code refers to SnapshotLogCleaner, but no such class exists. Perhaps it refers to work done in HBASE-50 that was never checked in. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5484) Spelling mistake in error message in HMasterCommandLine
[ https://issues.apache.org/jira/browse/HBASE-5484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13218934#comment-13218934 ] Hudson commented on HBASE-5484: --- Integrated in HBase-TRUNK-security #124 (See [https://builds.apache.org/job/HBase-TRUNK-security/124/]) HBASE-5484 Spelling mistake in error message in HMasterCommandLine (Revision 1294790) Result = SUCCESS stack : Files : * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMasterCommandLine.java Spelling mistake in error message in HMasterCommandLine --- Key: HBASE-5484 URL: https://issues.apache.org/jira/browse/HBASE-5484 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.90.5, 0.92.0 Reporter: David S. Wang Assignee: David S. Wang Priority: Trivial Labels: newbie, noob Fix For: 0.94.0 Attachments: diff1.txt If hadoop-zookeeper-server is installed and started, starting hbase-master in standalone mode will display this error message which has some typos. $ sudo /etc/init.d/hadoop-hbase-master start Starting Hadoop HBase master daemon: starting master, logging to /usr/lib/hbase/logs/hbase-hbase-master/cloudera-vm.out Couldnt start ZK at requested address of 2181, instead got: 2182. Aborting. Why? Because clients (eg shell) wont be able to find this ZK quorum hbase-master. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-2445) Clean up client retry policies
[ https://issues.apache.org/jira/browse/HBASE-2445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-2445: - Fix Version/s: (was: 0.94.0) 0.96.0 I don't see anybody working on this for 0.94. Pull back if you feel otherwise. Clean up client retry policies -- Key: HBASE-2445 URL: https://issues.apache.org/jira/browse/HBASE-2445 Project: HBase Issue Type: Improvement Components: client Reporter: Todd Lipcon Priority: Critical Labels: moved_from_0_20_5 Fix For: 0.96.0 Right now almost all retry behavior is governed by a single parameter that determines the number of retries. In a few places, there are also conf for the number of millis to sleep between retries. This isn't quite flexible enough. If we can refactor some of the retry logic into a RetryPolicy class, we could introduce exponential backoff where appropriate, clean up some of the config, etc -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
